TRB3 Centos-7 instructions: Difference between revisions
(One intermediate revision by one other user not shown) | |||
Line 279: | Line 279: | ||
The MIDAS frontend is setup to read events directly from the UDP buffer. You can see a running MIDAS TRB3 frontend here: | The MIDAS frontend is setup to read events directly from the UDP buffer. You can see a running MIDAS TRB3 frontend here: | ||
=== Setting up for MIDAS data taking === | === Setting up for MIDAS data taking === |
Latest revision as of 06:09, 14 June 2021
Introduction
See main TRB3 document for introductory material.
Centos-7 network setup
Install a switch of second network port. Do the following to configure DHCP on second network port, to assign IP to the TRB3. This example uses 192.168.1.X addresses; the Centos-7 machine is 192.168.1.253, the TRB3 is meant to be assigned 192.1681.1.1. This example uses TRB #171; adapt the instructions to your TRB number.
1) Assign a static 192.168.1.253 IP to your second network interface, using nm-connection-editor. This is what the configuration looks like afterwards for me
[root@daq11 tmp]# cat /etc/sysconfig/network-scripts/ifcfg-static_trb3 HWADDR=30:85:A9:A9:17:83 TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=none IPADDR=192.168.1.253 PREFIX=24 GATEWAY=192.168.1.253 DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_PRIVACY=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=static_trb3 UUID=89793cba-7645-42cf-ac24-f7e48bd08948 ONBOOT=yes [root@daq11 tmp]# ifconfig enp12s0 enp12s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.1.253 netmask 255.255.255.0 broadcast 192.168.1.255 inet6 fe80::f3e:f38e:d9ba:d654 prefixlen 64 scopeid 0x20<link> ether 30:85:a9:a9:17:83 txqueuelen 1000 (Ethernet) RX packets 52604116955 bytes 23147929921965 (21.0 TiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 13999033 bytes 1196314909 (1.1 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 19 memory 0xfb100000-fb120000
2) Install dhcp
yum -y install dhcp
3) Configure DHCP to look for the expected IP address for TRB3
[root@daq11 tmp]# cat /etc/dhcp/dhcpd.conf allow booting; allow bootp; ignore unknown-clients; #ddns-update-style ad-hoc; # set lease time to 3 days default-lease-time 259200; max-lease-time 259200; # Define the new subnet subnet 192.168.1.0 netmask 255.255.255.0 { interface enp12s0; range 192.168.1.100 192.168.1.250; not authoritative; deny unknown-clients; } # Define IP for TRB3 group { default-lease-time infinite; max-lease-time infinite; host trb171 { fixed-address 192.168.1.1; hardware ethernet da:7a:36:e9:37:ac; } }
4) Start DHCP and see if it assigns correct IP to TRB3
[root@daq11 tmp]# systemctl enable dhcpd Created symlink from /etc/systemd/system/multi-user.target.wants/dhcpd.service to /usr/lib/systemd/system/dhcpd.service. [root@daq11 tmp]# systemctl restart dhcpd; <power cycle TRB3> <check ping works> [root@daq11 tmp]# ping 192.168.1.1 PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data. 64 bytes from 192.168.1.1: icmp_seq=2 ttl=255 time=0.087 ms 64 bytes from 192.168.1.1: icmp_seq=3 ttl=255 time=0.144 ms [root@daq11 tmp]# systemctl status dhcpd ● dhcpd.service - DHCPv4 Server Daemon Loaded: loaded (/usr/lib/systemd/system/dhcpd.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2018-04-09 16:20:35 PDT; 18s ago Docs: man:dhcpd(8) man:dhcpd.conf(5) Main PID: 10650 (dhcpd) Status: "Dispatching packets..." CGroup: /system.slice/dhcpd.service └─10650 /usr/sbin/dhcpd -f -cf /etc/dhcp/dhcpd.conf -user dhcpd -group dhcpd --no-pid Apr 09 16:20:35 daq11.triumf.ca dhcpd[10650]: Wrote 0 new dynamic host decls to leases file. Apr 09 16:20:35 daq11.triumf.ca dhcpd[10650]: Wrote 0 leases to leases file. Apr 09 16:20:35 daq11.triumf.ca dhcpd[10650]: Listening on LPF/enp12s0/30:85:a9:a9:17:83/192.168.1.0/24 Apr 09 16:20:35 daq11.triumf.ca dhcpd[10650]: Sending on LPF/enp12s0/30:85:a9:a9:17:83/192.168.1.0/24 Apr 09 16:20:35 daq11.triumf.ca dhcpd[10650]: Sending on Socket/fallback/fallback-net Apr 09 16:20:35 daq11.triumf.ca systemd[1]: Started DHCPv4 Server Daemon. Apr 09 16:20:49 daq11.triumf.ca dhcpd[10650]: DHCPDISCOVER from da:7a:36:e9:37:ac via enp12s0 Apr 09 16:20:49 daq11.triumf.ca dhcpd[10650]: DHCPOFFER on 192.168.1.1 to da:7a:36:e9:37:ac via enp12s0 Apr 09 16:20:49 daq11.triumf.ca dhcpd[10650]: DHCPREQUEST for 192.168.1.1 (192.168.1.253) from da:7a:36:e9:37:ac via enp12s0 Apr 09 16:20:49 daq11.triumf.ca dhcpd[10650]: DHCPACK on 192.168.1.1 to da:7a:36:e9:37:ac via enp12s0
5) Assign hostname to this IP
[root@daq11 tmp]# more /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.1.1 trb171
6) Open a hole in firewall to the TRB3
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.168.1.1/32" port protocol="tcp" port="0-65535" accept" firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.168.1.1/32" port protocol="udp" port="0-65535" accept" firewall-cmd --add-port=5071/udp --permanent firewall-cmd --reload
Centos-7 Software Setup
The TRB3 software and associated analysis packages are developed on newer operating systems than Centos-7 (namely Opensuse). So a certain amount of work is needed to get the software to work on Centos-7.
1) I started by copying over the software in trbsoft from an existing installation.
scp trbsoft_backup.tar.gz trb3@ladd00:/data0/trb3/.
2) The main problem is with the TRB3 code requiring a newer version of perl. I used perlbrew to setup a newer version of perl than Centos-7 supports:
Following the instructions on website:
source ~/perl5/perlbrew/etc/bashrc perlbrew init perlbrew install perl-5.24.0
and then add this line to login script
rm -f /home/trb3/.perlbrew/init source ~/perl5/perlbrew/etc/bashrc perlbrew switch perl-5.24.0
3) Rebuild libtrbnet with different perl
cd ~/trbsoft/trbnettools/libtrbnet_perl perl Makefile.PL cd .. make TRB3=1 clean make TRB3=1 make TRB3=1 install
4) Modify
~/trbsoft/daqtools/merge_serial_address.pl and ~/trbsoft/daqtools/web/cts_gui
to use the perlbrew version of perl, rather than the system default.
5) Rebuild DABC (optional: this step is not necessary if we use the UDP version of the frontend, which is default now)
Make sure you have ROOT (ie ROOTSYS) setup properly
svn co https://subversion.gsi.de/dabc/trb3 trb3 cd trb3 make
This seems to build dabc and stream successfully. go4 doesn't compile cleanly, but this doesn't matter for the MIDAS frontend readout.
6) Modify the IP and MAC address in the file trbsoft/daqtools/users/triumf_trb171/db/register_configgbe_ip.db. This should be the IP and MAC address of the private IP interface on the server computer; so the IP is 192.168.1.253 in our case; use ipconfig to check the MAC.
7) Setup your login script (BASH in this case):
DAQ_TOOLS_PATH=~/trbsoft/daqtools USER_DIR=~/trbsoft/daqtools/users/triumf_trb171 TRB_WEB_DIR=$DAQ_TOOLS_PATH/web export PATH=$PATH:$DAQ_TOOLS_PATH:$HOME/trbsoft/trbnettools/bin export PATH=$PATH:$DAQ_TOOLS_PATH/tools export PATH=$PATH:$USER_DIR export PATH=$PATH:$HOME/trbsoft/trb3/dabc/bin/ export LD_LIBRARY_PATH=$HOME/trbsoft/trb3/dabc/lib/ # Use perl 5.24 rm -f /home/trb3/.perlbrew/init source ~/perl5/perlbrew/etc/bashrc perlbrew switch perl-5.24.0 ### TL ### This part restarts the trbnetd server; ### this server takes instructions from trbcmd executable and ### passes them actually to the TRB. export TRB3_SERVER=trb171:26000 export TRBNETDPID=$(pgrep -f "trbnetd -i 171") export DAQOPSERVER=localhost:171 export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/home/trb3/trbsoft/trbnettools/libtrbnet test -s ~/.alias && . ~/.alias || true # PATH="/home/trb3/perl5/bin${PATH:+:${PATH}}"; export PATH; PERL5LIB="/home/trb3/perl5/perlbrew/perls/perl-5.24.0/lib/5.24.0/:/home/trb3/trbsoft/daqtools/web/include:/home/trb3/trbsoft/trbnettools/libtrbnet_perl/blib/lib:/usr/share/perl5/vendor_perl"; export PERL5LIB; #PERL_LOCAL_LIB_ROOT="/home/trb3/perl5${PERL_LOCAL_LIB_ROOT:+:${PERL_LOCAL_LIB_ROOT}}"; export PERL_LOCAL_LIB_ROOT; #PERL_MB_OPT="--install_base \"/home/trb3/perl5\""; export PERL_MB_OPT; #PERL_MM_OPT="INSTALL_BASE=/home/trb3/perl5"; export PERL_MM_OPT; export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/home/trb3/trbsoft/trbnettools/libtrbnet_perl/blib/arch/auto/HADES/TrbNet/ # setup ROOT source ~/packages/root/bin/thisroot.sh # setup MIDAS export MIDASSYS=${HOME}/packages/midas export PATH=${PATH}:${MIDASSYS}/linux/bin export MIDAS_EXPTAB=${HOME}/online/exptab export ROOTANASYS=${HOME}/packages/rootana
8) Now start the trb3 program and initialize the TRB3
cd ~/trbsoft/daqtools/users/triumf_trb171 source startup.sh
9) If this is a new TRB3, need to modify the following files,
trbcmd i 0xffff # check open to figure out the serial number of new TRB3 FPGAs # use this info to modify following files: vi db/addresses_trb3.db vi $DAQ_TOOLS_PATH/base/serials_trb3.db vi db/register_configtdc.db
TRB3 MIDAS implementation
We have a working MIDAS readout of the TRB3 TDC data. Currently this readout uses a lot of the original GSI tools for setting up the TRB3 before the readout can start. At this time you need to do steps 1 and 2 described in "Start up instructions" in order to setup the TRB3 before starting the MIDAS frontend. Eventually we will try to remove as many as possible of the original GSI tools.
MIDAS frontend code is available here:
https://bitbucket.org/ttriumfdaq/trb3_frontend
The MIDAS frontend is setup to read events directly from the UDP buffer. You can see a running MIDAS TRB3 frontend here:
Setting up for MIDAS data taking
We are using the account trb3@daq11.triumf.ca for running this DAQ. If you need to restart the DAQ system, do the following
- Run the setup program that will initialize the TRB3 and set the 10kHz software trigger
cd /home/trb3/trbsoft/daqtools/users/triumf_trb171 source startup.sh
- (Optional) We can also start a webgui that allows us to check/configure the TRB3 separate from MIDAS; this is sometimes useful for monitoring. To do this, do
cd /home/trb3/trbsoft/daqtools/web export PERL5LIB=/home/trb3/perl5/perlbrew/perls/perl-5.24.0/lib/5.24.0/:/home/trb3/trbsoft/daqtools/web/include:/home/trb3/trbsoft/trbnettools/libtrbnet_perl/bli b/lib:/usr/share/perl5/vendor_perl ./cts_gui --noopenxterm --port=1234 --endpoint=0xc001
This allows access to GSI web GUI.
- Start the MIDAS TRB3 readout frontend, if not running:
/home/trb3/online/trb3_frontend/fetrb3UDP.exe -D
Actually, the default fetrb3UDP program does not create all the ODB settings that it should. It misses one setting, which specifies the name of the bank for a particular address. Create this variable after running the program the first time, using odbedit:
[local:trb3:R]Settings> cd /Equipment/TRB3_UDP/Settings [local:trb3:S]Settings>create STRING trb765 String length [32]: [local:trb3:S]Settings>move trb765 1 [local:trb3:S]Settings>set trb765 TRBA
As noted, we currently (April 9) we have the TRB3 setup with a 10kHz software trigger.
Running the analyzer
I added simple decoder and histograming classes to rootana. Currently I have only implemented the crude TDC calibration into the TRB3 bank decoders. In the long run we may want to integrate the fine TDC calibration directly into the frontend.
To look at the analyzer display, run the following
/home/trb3/online/trb3_frontend/analyzer/anaDisplay.exe
The display shows different histograms, arranged by FPGA and channel number. Histograms are
- The TDC times for each channel, with crude calibration applied
- The TDC raw fine times; ie, the raw counts for the sub-5ns part of the TDCs
- TDC time difference histograms; for this canvas you can set the reference FPGA and channel
Analyzer notes:
- On the physical TRB3 board the FPGAs are numbered as being 1-4, but the analyzer converts these to the more normal 0-3 indexing.
- Channel 0 is also the TDC associated with the actual trigger; in our case, the time of the software trigger
- For FPGA 0 and 1 the firmware is configured so we are reading both leading and falling TDC edges; so for these FPGAs channel 1 is the leading edge of the first input and channel 2 is the trailing edge of the first input, and so on. For FPGAs 2 and 3 we are only reading the leading edge of the TDCs.
To dump data from MIDAS file, do following
cd /home/trb3/online/trb3_frontend/analyzer ./ana.exe /home/trb3/online/run00026.mid.gz
Modify ana.cxx to analyze the TDC data and type 'make' to recompile ana.exe.