DEAP
Jump to navigation
Jump to search
Links
- https://deap06.triumf.ca/ MIDAS status page
- https://deap06.triumf.ca/elog/ ELOG
- https://deap06.triumf.ca/ganglia/ GANGLIA system monitoring
- https://deap06.triumf.ca/nodeinfo/config.html computer configuration and status
- https://deap06.triumf.ca/vme01/ VME crate 1
- https://deap06.triumf.ca/vme02/ VME crate 2
- https://deap06.triumf.ca/vme03/ VME crate 3
- https://deap06.triumf.ca:8443/ UPS
- https://deap06.triumf.ca:8444/ Power Distribution Unit
- https://deap06.triumf.ca:8445/ KVM (deapkvm8)
- https://deapkvm8 (192.168.1.17) (deapkvm8) ATEN IP KVM (only works from deap06 gateway machine) (deap, deap)
DAQ machines
- deap00: main daq machine (storage, home directories, central services, etc)
- deap01..05: A3818 daq machines
- deap06.triumf.ca: temporary network gateway
- deap07: spare A3818 daq machine (old deap00) (used for PCIe ADC DAQ)
- deap08: spare deap00 machine
- lxdeap01: VME daq machine
- deapvme01..03: VME crate power supplies
- deapups: DAQ UPS unit
- deapcdu: DAQ power distribution unit
- deapkvm8: 8-port IP KVM
- mscb520: MSCB-ETH bridge
deapups connections
- C13(f) : Switched Load 1 - Network switch SW-DEAP-DAQ
- C13(f) : Switched Load 2 - N/C
- C13(f) : Switched Load 3 - N/C
- C13(f) : Unswitched Load 4 - Rack Fan Left
- C13(f) : Unswitched Load 4 - Rack Fan Centre
- C13(f) : Unswitched Load 4 - Rack Fan Right
- C13(f) : Unswitched Load 4 - DEAPKVM
- C19(f) : Unswitched Load 4 - CDU
deapcdu connections
1 : DEAP01 9 : n/c 2 : DEAP02 10 : SCB2 3 : DEAP03 11 : SCB1 4 : DEAP04 12 : n/c 5 : DEAP05 13 : n/c 6 : LXDEAP01 14 : DEAP08 7 : N/C 15 : DEAPMPOD 8 : DEAP00 16 : DEAPKVM8
Network configuration (TRIUMF)
DEAP DAQ machines are on the private network 192.168.1.x (netmask 255.255.255.0)
Gateway to TRIUMF network is 1U machine deap06.triumf.ca connected to the LADD-NIS cluster (deap account on ladd00).
Gateway services running on the gateway:
- DHCP server for the 192.168.1.x network (/etc/hosts, /etc/dhcp/dhcpd.conf)
- apache SSL/https proxy for MIDAS status page, ELOG, ganglia and nodeinfo (/etc/httpd/conf.d/ssl.conf, /etc/httpd/htpasswd)
- NAT proxy from private network to the TRIUMF network (/etc/rc.local). Makes the internet accessible from deapNN machines.
Network configuration (DEAP)
The DEAP DAQ cluster is configured for standalone running with or without an internet connection.
(NB: Some internet functions are required: access to NTP for time synchronization and access to Linux package repositories to install packages, etc)
Usernames, passwords and hostnames are distributed using NIS:
- domain name: DEAP-NIS
- deap00 is the master server
- there are no secondary servers
- hostnames are distributed using NIS (from deap00:/etc/hosts, MUST MATCH deap06:/etc/hosts!)
- to solve chicken-and-egg problem deap00 IP address has to be listed in each machine /etc/hosts (MUST MATCH deap06 and deap00 /etc/hosts!) (SL6.2+ NIS broadcast does not work so deap00 has to be listed in each machine /etc/yp.conf, also NFS filesystems are mounted before NIS is started).
- also NIS has to be listed in front of DNS in the "hosts:" entry of /etc/nsswitch.conf
DNS kludge:
- normally DNS would be used to distribute IP addresses and hostnames to the DHCP server, to deap00 and to other deap machines. But we do not have a private DNS server and the TRIUMF DNS server has the wrong IP addresses for deap machines (142.90.x.x).
- deap06 DHCP is telling all machines to use the TRIUMF DNS server (to resolve internet addresses - google, etc). To avoid confusion between local deap00, etc hostnames and deap00, etc hostnames from TRIUMF, /etc/nsswitch.conf "hosts:" entry has to list "nis" before "dns".
- hopefully the deap00, etc hostnames will be resolved correctly by the SNOlab DNS servers and all this kludging can go away.
System monitoring tools:
- ganglia
- triumf_nodeinfo
- konstantin's ganglia packages (monitor_nfs, ganglia sensors, top, etc) - To install/update: yum --disablerepo="*" --enablerepo=konstantin update
- diskscrub
Backups
- backups of Linux images:
- backups of linux images are done to deap00:/data/root/backups using cron job on deap00:/etc/cron.d/backup.lxdaq.cron and deap00:~root/backup.lxdaq
- backups of home directories: NONE
- backups of data disks: NONE
Creating boot disks for deap01..deap05
mirrored 16GB USB Flash disks
go here Cloning_raid1_boot_disks
single 8GB/16GB USB Flash disks
The V7865 VME processors use single USB flash disks. To create the boot disks, follow instructions for #64GB_SSD_boot_disks, but clone "lxdeap01" instead of "deap01".
64GB SSD boot disks
- attach SSD disk to any of the deap01..deap05 machines (SATA+power)
- login as root to that machine
- "fdisk -l" to identify which /dev/sdX disk it is
- cd /data/root/backups
- ./clone.perl ./deap01 /dev/sdX
- observe script completes sucessfully and prints "Done. You can remove /dev/sdX and try to boot from it."
- disconnect the disk
- connect to new machine, try to boot from it