HADOOP

From DaqWiki
Jump to navigation Jump to search

HADOOP

Links

Management

Complete removal

  638  xemacs -nw /etc/rc.local --- comment out all hdfs stuff
  639  umount /hdfs
  642  service hadoop-hdfs-datanode stop
  643  chkconfig --list | grep hdfs
  644  chkconfig --list | grep had
  650  rm /datah
  652  cd /iris_data1/ --- this is different on every data node
  656  rm -rf hdfs_data/
  659  rpm -qa | grep -i hadoop
  660  yum erase hadoop
  666  rm -rf /etc/hadoop/
  667  rm /etc/alternatives/hadoop-conf 
  668  rm /etc/yum.repos.d/cloudera-cdh4.repo
  672  userdel hdfs
  673  grep hdfs /etc/passwd
  674  sync

Base install

cd /triumfcs/trshare/olchansk/linux/hadoop/
rpm --import RPM-GPG-KEY-cloudera
rpm -vh --install cloudera-cdh-4-0.noarch.rpm
yum install hadoop-hdfs hadoop-hdfs-fuse
ln -s /home/olchansk/sysadm/hadoop/conf.daq_test /etc/hadoop/
alternatives --install /etc/hadoop/conf hadoop-conf /etc/hadoop/conf.daq_test 50
alternatives --display hadoop-conf

Create a name node

  • yum install hadoop-hdfs-namenode
  • chkconfig hadoop-hdfs-namenode off
  • TBW
  • add to /etc/rc.local: firewall rules and service startup
iptables -F hadoop-namenode
iptables -N hadoop-namenode
iptables -A hadoop-namenode -p tcp -s 142.90.0.0/16 --dport 8020 -j ACCEPT
iptables -A hadoop-namenode -p tcp -s 142.90.0.0/16 --dport 50070 -j ACCEPT
iptables -A hadoop-namenode -p tcp --dport 8020 -j REJECT
iptables -A hadoop-namenode -p tcp --dport 50070 -j REJECT
iptables -A INPUT -j hadoop-namenode
iptables -L -v
service hadoop-hdfs-namenode restart

Create a data node

(only SL6 is supported. SL5 has problems with dead /hdfs mounts)

  • install data node software
yum install hadoop-hdfs-datanode
chkconfig hadoop-hdfs-datanode off
  • FIXME: adjust hadoop UID/GID somehow - it is different on every machine! wrong, wrong, wrong!
  • configure data node (dataNNN is the local data disk)
mkdir /dataNNN/hdfs_data
chown -R hdfs.hdfs /dataNNN/hdfs_data
ln -s /dataNNN/hdfs_data /datah
  • login as root@nameserver (ladd12)
    • cd /etc/hadoop/conf
    • add new node to hosts
    • add new node to topology.perl
    • su - hdfs, run: hadoop dfsadmin -refreshNodes
  • login as root@ladd00
    • add new node to /etc/httpd/conf.d/ssl.conf around "hadoop-50075"
    • service httpd restart
  • observe data node is listed as dead on the nameserver web page
  • add to /etc/rc.local: firewall rules, service startup and hdfs mount
iptables -F hadoop
iptables -N hadoop
iptables -A hadoop -p tcp -s 142.90.0.0/16 --dport 50010 -j ACCEPT
iptables -A hadoop -p tcp -s 142.90.0.0/16 --dport 50020 -j ACCEPT
iptables -A hadoop -p tcp -s 142.90.0.0/16 --dport 50075 -j ACCEPT
iptables -A hadoop -p tcp --dport 50010 -j REJECT
iptables -A hadoop -p tcp --dport 50020 -j REJECT
iptables -A hadoop -p tcp --dport 50075 -j REJECT
iptables -A INPUT -j hadoop
iptables -L -v
service hadoop-hdfs-datanode restart

mkdir /hdfs
hadoop-fuse-dfs dfs://ladd12:8020 /hdfs
  • start the data node: run /etc/rc.local
  • observe data node is now listed as live on the nameserver web page

decommission a data node

  • add node name to "hosts.exclude", should be typed exactly as it is entered in "hosts"
  • login hdfs@namenode
  • hdfs dfsadmin -refreshNodes
  • go to the hdfs web interface "Live Nodes" page, observe the data node status is "decommission in progress"
  • go to the hdfs web interface "Decommissioning Nodes", observe node is listed, observe "Under Replicated Blocks" count down to zero
  • after the node no longer listed in "Decommissioning Nodes" and is listed as "Decommissioned" in "Live Nodes", it can be shut down.

create user directory

  • login hdfs@namenode
  • hadoop fs -mkdir /users/trinat
  • hadoop fs -chown trinat /users/trinat

rebalance the data

(does not work with)

  • login hdfs@namenode
  • hdfs balancer
  • watch the grass grow

Setup quotas

  • ssh hdfs@namenode
  • hdfs dfsadmin -setSpaceQuota 600g /users/trinat

note: the quota applies to the size of files and their replicas. If default replication count is 3, the user can create 600g/3 = 200g worth of files.

Report quotas

  • (as any user)
  • hdfs dfs -count -q /users/"*"

The reported numbers are not obvious, for example: 

ladd12:fs$ hdfs dfs -count -q /users/"*"
        none             inf            none             inf            1            5            7809498 /users/lindner
        none             inf            none             inf            1            1          130360111 /users/olchansk
        none             inf    600000000000    208986905727            2          146       130337698091 /users/trinat

Here:

  • 600000000000 is the value set by dfsadmin -setSpaceQuota
  • 130337698091 is the size of all files not counting replicas (same value as reported by "du -ks /hdfs/users/trinat")
  • 208986905727 = 600000000000 - 3*130337698091 is the remaining space couning replicas
  • actual remaining space is 130337698091/3 = 69662301909 (about 69 Gbytes) assuming replication count of 3.

debug functions

  • start the data node: service hadoop-hdfs-datanode restart
  • look at the log file: tail -100 /var/log/hadoop-hdfs/hadoop-hdfs-datanode-ladd12.triumf.ca.log
  • mount hdfs: unset LD_LIBRARY_PATH; hadoop-fuse-dfs dfs://ladd12:8020 /mnt/xxx
  • watch name node: http://ladd12.triumf.ca:50070
  • watch data node: http://ladd08.triumf.ca:50075
  • report status and configuration: hdfs dfsadmin -report
  • report disk usage: hdfs dfs -count -q /users/"*"
  • report location of nodes: hdfs dfsadmin -printTopology
  • report everything about every file: hdfs fsck / -openforwrite -files -blocks -locations -racks

Performance

  • cluster with 3 data nodes: ladd12 (quad-core i7-860), ladd08 (dual opteron 2GHz), ladd05 (dual-core Athlon 2.6 GHz)
  • hdfs mounted on ladd12
  • write benchmark:
ladd12:olchansk$ /usr/bin/time dd if=/dev/zero of=xxx bs=1024k count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 31.3406 s, 33.5 MB/s
0.00user 0.18system 0:31.43elapsed 0%CPU (0avgtext+0avgdata 7104maxresident)k
0inputs+0outputs (0major+483minor)pagefaults 0swaps
ladd12:olchansk$ /usr/bin/time dd if=/dev/zero of=xxx2 bs=1024k count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 436.875 s, 24.0 MB/s
0.00user 1.72system 7:16.97elapsed 0%CPU (0avgtext+0avgdata 7104maxresident)k
0inputs+0outputs (0major+483minor)pagefaults 0swaps

ganglia reports peak network use 30 Mbytes/sec, ladd12: load 1, cpu use 10%, disk use 15%+15%busy; ladd05: load 2, cpu use 30%sys, 30%wait, disk use 15%+20%busy; ladd08: load 6, cpu use 30%sys, 70%wait, disk use 40%+40%busy.

  • read benchmark:
ladd12:olchansk$ ls -l
total 11391305
-rw-r--r-- 1 olchansk nobody   130360111 Dec 28 18:36 run21600sub00000.mid.gz
-rw-r--r-- 1 olchansk nobody  1048576000 Jan  2 10:35 xxx
-rw-r--r-- 1 olchansk nobody 10485760000 Jan  2 10:37 xxx2
ladd12:olchansk$ /usr/bin/time dd if=run21600sub00000.mid.gz of=/dev/null
254609+1 records in
254609+1 records out
130360111 bytes (130 MB) copied, 1.55497 s, 83.8 MB/s
0.02user 0.08system 0:01.69elapsed 6%CPU (0avgtext+0avgdata 3024maxresident)k
254632inputs+0outputs (1major+227minor)pagefaults 0swaps
ladd12:olchansk$ 
ladd12:olchansk$ /usr/bin/time dd if=xxx of=/dev/null
2048000+0 records in
2048000+0 records out
1048576000 bytes (1.0 GB) copied, 11.4278 s, 91.8 MB/s
0.18user 0.69system 0:11.43elapsed 7%CPU (0avgtext+0avgdata 3008maxresident)k
2048000inputs+0outputs (0major+227minor)pagefaults 0swaps
ladd12:olchansk$ 
ladd12:olchansk$ /usr/bin/time dd if=xxx2 of=/dev/null
20480000+0 records in
20480000+0 records out
10485760000 bytes (10 GB) copied, 130.188 s, 80.5 MB/s
1.68user 7.10system 2:10.19elapsed 6%CPU (0avgtext+0avgdata 3024maxresident)k
20480000inputs+0outputs (0major+228minor)pagefaults 0swaps

no network use, cpu use ladd12 is 10%+30%wait, disk use is 0% because all data was cached in memory: 10GB file, 16GB RAM.


Network ports

  • data node ports:
[root@ladd08 ~]# diff xxx1 xxx2 | grep LISTEN
> tcp        0      0 :::50020                    :::*                        LISTEN      
> tcp        0      0 :::47860                    :::*                        LISTEN      
> tcp        0      0 :::50010                    :::*                        LISTEN      
> tcp        0      0 :::50075                    :::*                        LISTEN

port 47860 is the java rmi server port, it is different on different machines and sometimes changes after restart. how to pin it to a fixed port number so I can firewall it?!?

  • name node:
[root@ladd12 ~]# netstat -an | grep LISTEN | grep tcp > xxx2
[root@ladd12 ~]# diff xxx1 xxx2
< tcp        0      0 ::ffff:142.90.119.126:8020  :::*                        LISTEN      
< tcp        0      0 :::50070                    :::*                        LISTEN      
< tcp        0      0 :::52745                    :::*                        LISTEN

port 52745 is probably also the java rmi server port...

  • name node:
> lsof -P | grep LISTEN | grep java
java      18096      hdfs  144u     IPv4         1464177661      0t0        TCP *:44898 (LISTEN)
java      18096      hdfs  155u     IPv4         1464179424      0t0        TCP ladd12.triumf.ca:8020 (LISTEN)
java      18096      hdfs  165u     IPv4         1464180124      0t0        TCP *:50070 (LISTEN)
java      18332      hdfs  144u     IPv4         1464181247      0t0        TCP *:59445 (LISTEN)
java      18332      hdfs  151u     IPv4         1464182326      0t0        TCP *:50010 (LISTEN)
java      18332      hdfs  152u     IPv4         1464182357      0t0        TCP *:50075 (LISTEN)
java      18332      hdfs  157u     IPv4         1464182653      0t0        TCP *:50020 (LISTEN)

port 44898 and 59445 are not documented. probably java rpm server ports

Retrieved from "https://daq00.triumf.ca/DaqWiki/index.php?title=HADOOP&oldid=5287"