HADOOP: Difference between revisions
Jump to navigation
Jump to search
| Line 19: | Line 19: | ||
<pre> | <pre> | ||
ln -s /home/olchansk/sysadm/hadoop/conf.daq_test /etc/hadoop-0.20 | ln -s /home/olchansk/sysadm/hadoop/conf.daq_test /etc/hadoop-0.20 | ||
alternatives --install /etc/hadoop-0.20/conf hadoop-0.20-conf /etc/hadoop-0.20/conf.daq_test 50 | |||
alternatives --display hadoop-0.20-conf | |||
mkdir /data8/hdfs_data | |||
chown -R hdfs.hdfs /data8/hdfs_data | |||
(add /data8/hdfs_data to /home/olchansk/sysadm/hadoop/conf.daq_test/hdfs-site.xml) | |||
service hadoop-0.20-datanode start | |||
tail -100 /usr/lib/hadoop-0.20/logs/hadoop-hadoop-datanode-ladd08.triumf.ca.log | |||
</pre> | </pre> | ||
* watch name node: http://ladd12.triumf.ca:50070 | |||
* watch data node: http://ladd08.triumf.ca: | |||
Revision as of 09:23, 2 January 2012
HADOOP
Create a data node
- install data node software
cd /triumfcs/trshare/olchansk/linux/hadoop/ rpm --import RPM-GPG-KEY-cloudera rpm -vh --install cdh3-repository-1.0-1.noarch.rpm-SL5 (or -SL6) (cd $HOME; sh /triumfcs/trshare/olchansk/linux/hadoop/jdk-6u30-linux-x64-rpm.bin) cd ~ yum install hadoop"*"datanode hadoop"*"fuse hadoop"*"native chkconfig hadoop-0.20-datanode off
- FIXME: adjust hadoop UID/GID somehow - it is different on every machine! wrong, wrong, wrong!
- configure data node
ln -s /home/olchansk/sysadm/hadoop/conf.daq_test /etc/hadoop-0.20 alternatives --install /etc/hadoop-0.20/conf hadoop-0.20-conf /etc/hadoop-0.20/conf.daq_test 50 alternatives --display hadoop-0.20-conf mkdir /data8/hdfs_data chown -R hdfs.hdfs /data8/hdfs_data (add /data8/hdfs_data to /home/olchansk/sysadm/hadoop/conf.daq_test/hdfs-site.xml) service hadoop-0.20-datanode start tail -100 /usr/lib/hadoop-0.20/logs/hadoop-hadoop-datanode-ladd08.triumf.ca.log
- watch name node: http://ladd12.triumf.ca:50070
- watch data node: http://ladd08.triumf.ca: