- deap00 links:
- https://deapdaqgw.snolab.ca/ - MIDAS status page
- https://deapdaqgw.snolab.ca/elog/ - DEAP00 ELOG
- https://deapdaqgw.snolab.ca/ganglia/ - GANGLIA system monitoring
- https://deapdaqgw.snolab.ca/nodeinfo/config.html - computer configuration and status
- http://dug1.snolab.ca/ - data server
- https://deapdaqgw.snolab.ca/check/ - deapgw environment monitoring MIDAS status page
- (obsolete) https://deapdaqgw.snolab.ca/quotareport/quota.html - report deapdaqgw disk quota
- https://deapdaqgw.snolab.ca/zfsquotareport/zfsquota.html - report deapdaqgw disk use
- https://deapdaqgw.snolab.ca/pingmon/deapgw.html - deapgw network monitor
- https://deapdaqgw.snolab.ca/pingmon_deap00/deap00.html - deap00 network monitor
- TWiki links:
- https://www.snolab.ca/deap/private/TWiki/bin/view SNOLAB Wiki
- https://www.snolab.ca/deap/private/TWiki/bin/view/Main/DeapDAQ - main documenation page
- Direct links to DAQ hardware:
- https://deapdaqgw.snolab.ca/vme01/ VME crate 1 (all 3 VME crates: does not work from google-chrome, firefox is ok, safari is ok. after 10 sec reloads to the midas status page)
- https://deapdaqgw.snolab.ca/vme02/ VME crate 2
- https://deapdaqgw.snolab.ca/vme03/ VME crate 3
- https://deapdaqgw.snolab.ca:8443/ UPS (deapups, guest/guest)
- https://deapdaqgw.snolab.ca:8444/index.html Sentry Power Distribution Unit (deapcdu, deap/deap)
- https://deapdaqgw.snolab.ca:8445/ KVM (deapkvm8) (this proxy does not work)
- https://deapkvm8 (192.168.1.17) (deapkvm8) ATEN IP KVM (only works from firefox on deapdaqgw gateway machine - follow VNC instructions below for offsite access) (deapuser, deapuser)
- https://deapdaqgw.snolab.ca/deapcam01/ (deap, deapCAM3600, select "mobile mode sign-in")
- deapcam02 ssh tunnel: from local computer start tunnel: "ssh -v email@example.com -L localhost:8080:deapcam02:80" (you can omit "-v), then on a local web browser, open "http://localhost:8080" ("8080" is the number from the ssh command, you can use some other number if port 8080 is already in use).
Fingerprints for the deapdaqgw.snolab.ca SSL certificate:
- SHA1: D3 06 C6 80 40 0D 0D 9F 83 E5 AD CD EB F2 BE 7A F2 E4 4A 67
- MD5: 7D 9E AD 4F 1B 03 D9 63 10 E5 3F 1E ED 40 A1 EF
- SHA-256: 2A ED 88 3E 38 27 25 D0 1E 4D 48 A1 78 FC E0 0B 6E 58 00 FA A6 53 B1 FE 50 3D 91 CE C9 AB 08 19
- SHA-1: 1A FE 53 23 0A 22 F3 35 C8 20 CD 0E 0E F0 3F A7 72 B7 2F 29
Power up sequence
- power up and turn on all 3 UPSes
- network switch, CDU and gateway machine are connected to non-switchable UPS ports, they should power up and boot
- one should be able to ping the gateway machine
- ssh deapgw@deapdaqgw, ping deapups and deapcdu
- open the UPS and CDU web pages through the HTTPS proxy
- from the CDU web page (outlet control), turn on deap00. If deap00 is off (0 power use) but outlet is "on", use the "reboot" action.
- wait for deap00 to boot (ping deap00)
- mhttpd and elog should start automatically
- open the DEAP MIDAS status page
- now one can ssh deap@deapdaqgw then ssh deap00
- on the MIDAS status page, start the slow controls frontends: UPS, CDU, VME crates and NutUps, clear all alarms. Do not start MPOD and SCB yet.
- all frontends should start "green", except for "vme02" (and "mpod") should report "communication problem"
- on the CDU web page, turn on all power outlets (use "global control action" - "on" - "apply")
- wait for VME02 (and MPOD) to boot: their frontend status should turn "green"
- wait for deap01..deap05 to boot (no simple indication, but they should ping from deap00)
- from the MIDAS "VME" slow controls pages, turn on all 3 VME crates
- from the MIDAS "programs" page, start mpod and scb frontends. They status should show "green"
- from the MIDAS "MPOD_HV" page, turn on the MPOD ("main switch ON", if ready to ramp up the voltages, "output ON")
- from the MIDAS "SCB" page, turn on all SCBs ("ON" button)
- wait for lxdeap01 to boot (should be able to ping from deap00)
- from the MIDAS "programs" page, start all daq frontends
- clear all MIDAS alarms
- start a run, wait for a bit, stop the run (to confirm all frontends are happy)
- DAQ is now ready to take data
Tripp-lite management software
ssh root@deap00 /opt/nut/bin/upsdrvctl stop (unplug USB cable from UPS, wait 10 sec, plug it back in) service pald restart /var/tripplite/poweralert/console/pal_console.sh ### restore NUT monitoring service pald stop ### this takes a few minutes /opt/nut/bin/upsdrvctl start
- lsusb -v | grep -i product
- lsusb -v | grep -i serial
NUT UPS configuration
NB - UPS names are tied to the UPS serial numbers via the NUT config file!
- ssh root@deap00
- (initial software checkout) cd ~; svn checkout svn://anonscm.debian.org/nut/trunk nut
- (update) cd ~/nut; svn update
- (build, install) cd ~/nut; ./autogen.sh; ./configure --prefix=/opt/nut; make -j6 -k; make -k; make -k install
- config file: /opt/nut/etc/ups.conf
[ups1] driver = usbhid-ups port = auto desc = "ups1" serial = "2231ELCPS720300082" [ups2] driver = usbhid-ups port = auto desc = "ups2" serial = "2211KW0PS733900093" [ups3] driver = usbhid-ups port = auto desc = "ups3" serial = "2231ELCPS720300090"
- restart drivers: /opt/nut/bin/upsdrvctl start
- reload upsd: /opt/nut/sbin/upsd -c reload
- see ups status: /opt/nut/bin/upsc ups1
- snmpwalk -v 2c -M +/home/deap/online/slow/fesnmp -m +Sentry3-MIB -c public deapcdu sentry3
- snmpset -v 2c -M +/home/deap/online/slow/fesnmp -m +Sentry3-MIB -c write deapcdu outletControlAction.1.1.1 i 1 ### turn on outlet 1
- snmpset -v 2c -M +/home/deap/online/slow/fesnmp -m +Sentry3-MIB -c write deapcdu outletControlAction.1.1.3 i 2 ### turn off outlet 3
VME and MPOD snmp
- snmpwalk -v 2c -M +/home/deap/online/slow/fewiener -m +WIENER-CRATE-MIB -c guru deapvme01 crate
- snmpset -v 2c -M +/home/deap/online/slow/fewiener -m +WIENER-CRATE-MIB -c guru deapvme01 sysMainSwitch.0 i 1 ### turn crate on
- snmpset -v 2c -M +/home/deap/online/slow/fewiener -m +WIENER-CRATE-MIB -c guru deapvme01 sysMainSwitch.0 i 0 ### turn crate off
Network configuration (DEAP)
The DEAP DAQ cluster is configured for standalone running with or without an internet connection.
(NB: Some internet functions are required: access to NTP for time synchronization and access to Linux package repositories to install packages, etc)
deapdaqgw: enp4s0 - eth0 - uplink (dhcp) enp5s0 - eth1 - daq private network deap00 mobo, e1000e driver - eth0 - daq private network (dhcp) eth1 - deap05e deap00 pcie nic, igb driver - (top to bottom) eth2 - deap01a eth3 - deap02b eth4 - deap03c eth5 - deap04d deap00 10gige nic, sfc driver - eth6 - n/c eth7 - dug1 (10gige) deap01..deap05: eth0 - daq private network eth1 - secondary network direct link to deap00 lxdeap01: eth0 or eth1 - daq private network (use either port)
Main DHCP server is running on the gateway machine. It provides IP addresses to all devices on the main private network. apt install isc-dhcp-server, systemctl enable isc-dhcp-server
Additional DHCP server is running on deap00. It provides IP addresses to the secondary network links to deap00 (deap01a..deap04d)
See following sections for more details.
DEAP network nodes with statically configured IP addresses:
- deapmod : Wiener MPOD firmware does not support DHCP
deapdaqgw is the gateway machine that provides internet access to the DEAP DAQ cluster.
- NAT ("network address translation", see /etc/rc.local)
- IP address assignement via /etc/hosts
- DNS via dnsmasq serving contents of /etc/hosts and bridge to upstream DNS (configured in /etc/resolv.conf by upstream DHCP). apt install dnsmasq, systemctl enable dnsmasq, see Ubuntu instructions.
- DHCP for all machines via /etc/dhcpd/dhcpd.conf, Special DHCP settings:
- "option routers" sets the "default route" through the gateway machine itself
- "option domain-name-servers" sets the DNS server in /etc/resolv.conf to dnsmasq on the gateway machine
- "option ntp-servers" specifies the time servers, (but not used by any hosts?)
- "option domain-name" is not specified, leaving the "domain" and "search" entries of /etc/resolv.conf blank (actually the entries are not there)
- unknown clients are assigned IP addresses in the range 192.168.x.200 through .250.
- MSCB nodes are assigned "infinite" leases by avoid a bug in MSCB firmware
- remember to "service dhcpd restart" after editing /etc/dhcp/dhcpd.conf
- HTTPS proxy for midas, elog, and other web-connected devices (see links above). Edit /etc/httpd/conf.d/*.conf
- cron job: zfsquotareport
- cron job: pingmon network monitor
- MIDAS experiment "check" (user deapgw)
deap00 is the main machine for the DEAP DAQ cluster.
- DHCP for secondary network links to frontend machines, remember to "service dhcpd restart" after editing /etc/dhcp/dhcpd.conf
- NIS master
- NFS export of home disks, data disks (NFS exports list: edit /etc/netgroup, run "make -C /var/yp")
- httpd for ganglia, nodeinfo, etc
- xinetd, tftpd for booting deap01..05, lxdeap01
network port assignments
- eth0: main connection to the local network, IP address is assigned by DHCP from the gateway machine
- eth1: connected to deap05
- eth2..eth5: Intel 4-port card, ports are numbered from the top, connected to deap01..deap04 in order.
- eth6..eth7: 10GigE network card
- disk list: 2x120GB SSD, 2x3TB + 2x4TB HDD:
[root@deap00 ~]# date Fri Apr 10 18:00:12 EDT 2015 [root@deap00 ~]# ./smart-status.perl Disk model serial temperature realloc pending uncorr CRC err RRER /dev/sda WDC WD40EZRX-00SPEB0 WD-WCC4E0555417 41 0 0 0 0 0 /dev/sdb WDC WD40EZRX-00SPEB0 WD-WCC4E0602954 39 0 0 0 0 0 /dev/sdc ST3000DM001-9YN166 W1F0SG0W 31 0 0 0 6 - /dev/sdd ST3000DM001-9YN166 W1F0THDE 33 0 0 0 0 - /dev/sde KINGSTON SV300S37A120G 50026B7744027BCB 33 0 ? ? ? - /dev/sdf KINGSTON SV300S37A120G 50026B774909A85D 30 2 ? ? ? - [root@deap00 ~]#
- spare disk: the 1st 4TB HDD is the hot spare. It is partitioned to be compatible with replacement in case of failure of any other disks. Only /dev/sda1 is used (as a bootable partition).
- "/" is /dev/md4 RAID1 of sde1, sdf1, sdb1(W), sda1(W). Special note "W" means "write mostly" - means avoid reading from these disks - means read from SSD but write to both SSD and HDD (SSD, SSD, 4TB, 4TB)
- "/home" is /dev/md5 RAID5 of sdb1, sdc1, sdd1 (4TB, 3TB, 3TB)
- swap is /dev/md6 RAID5 sdb2, sdc2, sdd2 (4TB, 3TB, 3TB)
- "/data" is /dev/md7 RAID5 sdb3, sdc3, sdd3 (4TB, 3TB, 3TB)
- note how 4TB /dev/sda is partitioned same as /dev/sdb, but only /dev/sda1 is used - the rest of the disk as a "hot spare"
[root@deap00 ~]# date Mon Apr 13 14:49:03 EDT 2015 [root@deap00 ~]# cat /proc/mdstat | grep active | sort md4 : active raid1 sda1(W) sdb1(W) sde1 sdf1 md5 : active raid5 sdb3 sdc1 sdd1 md6 : active raid5 sdb2 sdc2 sdd2 md7 : active raid5 sdb4 sdc3 sdd3 [root@deap00 ~]# date Fri Apr 10 18:01:45 EDT 2015 [root@deap00 ~]# df -kl Filesystem 1K-blocks Used Available Use% Mounted on /dev/md4 115246492 52353428 57033052 48% / tmpfs 16414940 1894852 14520088 12% /dev/shm /dev/md5 403039088 240822952 141736292 63% /home /dev/md7 5236247032 1709964112 3260290168 35% /data [root@deap00 ~]#
Usernames, passwords and hostnames are distributed using NIS:
- domain name: DEAP-NIS
- deap00 is the master server
- there are no secondary servers
- deapdaqgw time is configured in /etc/ntp.conf (currently triumf, dsurface, ca.pool.ntp.org)
- deapdaqgw DHCP configuration file /etc/dhcp/dhcpd.conf line "option ntp-servers" provides time servers to all dhcp clients (currently deapdaqgw, except deap00, deap01..05, lxdeap01: "option ntp-servers 0.0.0.0")
- deap00 and frontend machines:
- /etc/ntp.conf specifies: "server deapdaqgw iburst prefer" options are: iburst=quick sync, prefer=prefer to use this server
- in addition, local dhcp client writes the dhcp ntp-servers "0.0.0.0" into /etc/ntp.conf which seems to be benign but we do now know how turn this off.
deapdaqgw is used as the time master instead of deap00 to make the configuration more symmetric: deap00 and all frontend machines have the same time configuration instead of deap00 being different.
System monitoring tools
- konstantin's ganglia packages (monitor_nfs, ganglia sensors, top, etc) - To install/update: see TRIUMF SL install instructions.
VNC access to the KVM
Generic VNC instructions: VNC
DEAP specific VNC instruction:
- start local VNC client in "listen mode" as described at VNC (usually: vncviewer -listen 5500)
- ssh deap@deapdaqgw
- rm ~/.vnc/passwd
- vncserver -geometry 1600x1200 ### watch the output line: "desktop is deapdaqgw:2" <--- remember the value ":2" printed (it will be different each time)
- vncconfig -display localhost:2 -connect send.triumf.ca:5500 <--- localhost:2 uses the ":2" from the vncserver output, "send.triumf.ca:5500" is the location and port of your VNC client.
- inside VNC:
- open a terminal. We are deap@deapdaqgw
- inside terminal start: firefox
- if firefox complains about running elsewhere, find where it is running and kill it
- open https://deapkvm8
- login as administrator or deapuser (password deapuser)
- go to a console port, click "connect". If default configuration is correct, the java program for the KVM console window will open
- it may prompt for permission to run ATEN Java application
- it may prompt for which application to use to open "java web start" files. Select: Downloads/javaws (symlink to /usr/java/jre1.8.0_31/bin/javaws)
- NB: KVM firmware loads a 32-bit library libikvmlib which requires 32-bit java which requires "yum install gtk2.i686"
- the java application with the KVM console window should open
- if java refuses to start the applet due to "strict permissions", add an exception to this: run jcontrol, in the "security" tab, "edit site list...", add "https://deapkvm8"
Backups of all system disks (SSD and USB flash media) are done to the deap00 data disk. This includes deapdaqgw and deap00 SSDs:
- deap00 cron job /etc/cron.d/backup.lxdaq.cron
- runs script deap00:~root/backup.lxdaq
- writes backups to deap00 data disk: deap00:/data/root/backups:
[root@deap00 ~]# ls -l /data/root/backups/ total 56 -rwxr-xr-x 1 root root 4208 Dec 7 2012 clone.perl dr-xr-xr-x 31 root root 4096 Jul 3 13:28 deap00 <--- backup of deap00 "/" (raid1 mirrored SSDs) dr-xr-xr-x 28 root root 4096 Jul 3 13:32 deap01 <--- backups turned off dr-xr-xr-x 28 root root 4096 Jul 3 13:32 deap02 <--- same dr-xr-xr-x 28 root root 4096 Apr 19 18:44 deap03 <--- same dr-xr-xr-x 28 root root 4096 Jul 3 13:32 deap04 <--- same dr-xr-xr-x 28 root root 4096 Jul 3 13:32 deap05 <--- same dr-xr-xr-x 28 root root 4096 Mar 2 21:32 deap07 <--- same dr-xr-xr-x 28 root root 4096 Mar 2 21:32 deap08 <--- same dr-xr-xr-x 29 root root 4096 Jul 3 11:10 deapdaqgw <--- backup of deapdaqgw single SSD dr-xr-xr-x 26 root root 4096 Jul 3 13:38 lxdeap01 <--- backup of lxdeap01 USB flash disk -rwxr-xr-x 1 root root 7672 Dec 7 2012 uuidfix.perl [root@deap00 ~]#
- clone.perl and uuidfix.perl are the scripts for writing these backups back to bootable media (to 16GB USB flash disk or to 30/60GB SSD)
Backups of deap00 home disk, deapdaqgw and of the backups of deap00, deap01 and lxdeap01 system disks are done to TRIUMF ladd00 data disk.
- ladd00 cron job /etc/cron.d/backup.os.cron
- runs script /root/backup.os.all
- runs script "/root/backup.os deapdaqgw.snolab.ca" writes to /ladd/data0/backup.os/deapdaqgw.snolab.ca
- runs script /root/backup.deap:
cd /ladd/data0/backup.os rsync -avx --delete-after deapdaqgw.snolab.ca:/data/root/backups/deap00 deap/deap00 >> $lastlog 2>&1 rsync -avx --delete-after deapdaqgw.snolab.ca:/data/root/backups/deap01 deap/deap01 >> $lastlog 2>&1 rsync -avx --delete-after deapdaqgw.snolab.ca:/data/root/backups/lxdeap01 deap/lxdeap01 >> $lastlog 2>&1 rsync -avx --delete-after --max-size=100000000 deapdaqgw.snolab.ca:/home deap/home >> $lastlog 2>&1
[root@ladd00 ~]# ls -l /ladd/data0/backup.os/ /ladd/data0/backup.os/deap /ladd/data0/backup.os/: ... drwxr-xr-x 6 root root 4096 Jun 17 15:34 deap dr-xr-xr-x 29 root root 4096 Jul 3 08:10 deapdaqgw.snolab.ca -rw-r--r-- 1 root root 2414 Jul 8 15:01 deapdaqgw.snolab.ca.last.log -rw-r--r-- 1 root root 113454 Jul 8 15:24 deap.last.log ... /ladd/data0/backup.os/deap: total 16 drwxr-xr-x 3 root root 4096 Jun 12 12:53 deap00 drwxr-xr-x 3 root root 4096 Jun 12 12:55 deap01 drwxr-xr-x 3 root root 4096 Jun 17 15:34 home drwxr-xr-x 3 root root 4096 Jun 12 12:56 lxdeap01 [root@ladd00 ~]#
There are no backups of the deap00 data disks.