HADOOP

HADOOP

Create a data node

• install data node software

cd /triumfcs/trshare/olchansk/linux/hadoop/
rpm --import RPM-GPG-KEY-cloudera 
rpm -vh --install cdh3-repository-1.0-1.noarch.rpm-SL5 (or -SL6)
(cd $HOME; sh /triumfcs/trshare/olchansk/linux/hadoop/jdk-6u30-linux-x64-rpm.bin)
cd ~
yum install hadoop"*"datanode hadoop"*"fuse hadoop"*"native
chkconfig hadoop-0.20-datanode off

• FIXME: adjust hadoop UID/GID somehow - it is different on every machine! wrong, wrong, wrong!

• configure data node

ln -s /home/olchansk/sysadm/hadoop/conf.daq_test /etc/hadoop-0.20
alternatives --install /etc/hadoop-0.20/conf hadoop-0.20-conf /etc/hadoop-0.20/conf.daq_test 50
alternatives --display hadoop-0.20-conf
mkdir /data8/hdfs_data
chown -R hdfs.hdfs /data8/hdfs_data
(add /data8/hdfs_data to /home/olchansk/sysadm/hadoop/conf.daq_test/hdfs-site.xml)
service hadoop-0.20-datanode start
tail -100 /usr/lib/hadoop-0.20/logs/hadoop-hadoop-datanode-ladd08.triumf.ca.log

• mount hdfs: unset LD_LIBRARY_PATH; hadoop-fuse-dfs dfs://ladd12:8020 /mnt/xxx
• watch name node: http://ladd12.triumf.ca:50070
• watch data node: http://ladd08.triumf.ca:50075

Performance

aaa