TRB3 Centos-7 instructions

Introduction

See main TRB3 document for introductory material.

Centos-7 network setup

Install a switch of second network port. Do the following to configure DHCP on second network port, to assign IP to the TRB3. This example uses 192.168.1.X addresses; the Centos-7 machine is 192.168.1.253, the TRB3 is meant to be assigned 192.1681.1.1. This example uses TRB #171; adapt the instructions to your TRB number.

1) Assign a static 192.168.1.253 IP to your second network interface, using nm-connection-editor. This is what the configuration looks like afterwards for me

[root@daq11 tmp]# cat /etc/sysconfig/network-scripts/ifcfg-static_trb3 
HWADDR=30:85:A9:A9:17:83
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
IPADDR=192.168.1.253
PREFIX=24
GATEWAY=192.168.1.253
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_PRIVACY=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=static_trb3
UUID=89793cba-7645-42cf-ac24-f7e48bd08948
ONBOOT=yes

[root@daq11 tmp]# ifconfig  enp12s0
enp12s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.253  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::f3e:f38e:d9ba:d654  prefixlen 64  scopeid 0x20<link>
        ether 30:85:a9:a9:17:83  txqueuelen 1000  (Ethernet)
        RX packets 52604116955  bytes 23147929921965 (21.0 TiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 13999033  bytes 1196314909 (1.1 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 19  memory 0xfb100000-fb120000  

2) Install dhcp

yum -y install dhcp 

3) Configure DHCP to look for the expected IP address for TRB3

[root@daq11 tmp]# cat /etc/dhcp/dhcpd.conf

allow booting;
allow bootp;
ignore unknown-clients;
#ddns-update-style ad-hoc;

# set lease time to 3 days
default-lease-time 259200;
max-lease-time 259200;

# Define the new subnet
subnet 192.168.1.0 netmask 255.255.255.0 {
    interface enp12s0;
    range  192.168.1.100 192.168.1.250; 
    not authoritative;
    deny unknown-clients;
} 

# Define IP for TRB3
group {
      default-lease-time infinite;
      max-lease-time infinite;
      host trb171 { fixed-address 192.168.1.1; hardware ethernet da:7a:36:e9:37:ac; }
}

4) Start DHCP and see if it assigns correct IP to TRB3

[root@daq11 tmp]# systemctl enable dhcpd
Created symlink from /etc/systemd/system/multi-user.target.wants/dhcpd.service to /usr/lib/systemd/system/dhcpd.service.
[root@daq11 tmp]#  systemctl restart dhcpd;

<power cycle TRB3>
<check ping works>
[root@daq11 tmp]# ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=2 ttl=255 time=0.087 ms
64 bytes from 192.168.1.1: icmp_seq=3 ttl=255 time=0.144 ms

[root@daq11 tmp]# systemctl status dhcpd
● dhcpd.service - DHCPv4 Server Daemon
   Loaded: loaded (/usr/lib/systemd/system/dhcpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2018-04-09 16:20:35 PDT; 18s ago
     Docs: man:dhcpd(8)
           man:dhcpd.conf(5)
 Main PID: 10650 (dhcpd)
   Status: "Dispatching packets..."
   CGroup: /system.slice/dhcpd.service
           └─10650 /usr/sbin/dhcpd -f -cf /etc/dhcp/dhcpd.conf -user dhcpd -group dhcpd --no-pid

Apr 09 16:20:35 daq11.triumf.ca dhcpd[10650]: Wrote 0 new dynamic host decls to leases file.
Apr 09 16:20:35 daq11.triumf.ca dhcpd[10650]: Wrote 0 leases to leases file.
Apr 09 16:20:35 daq11.triumf.ca dhcpd[10650]: Listening on LPF/enp12s0/30:85:a9:a9:17:83/192.168.1.0/24
Apr 09 16:20:35 daq11.triumf.ca dhcpd[10650]: Sending on   LPF/enp12s0/30:85:a9:a9:17:83/192.168.1.0/24
Apr 09 16:20:35 daq11.triumf.ca dhcpd[10650]: Sending on   Socket/fallback/fallback-net
Apr 09 16:20:35 daq11.triumf.ca systemd[1]: Started DHCPv4 Server Daemon.
Apr 09 16:20:49 daq11.triumf.ca dhcpd[10650]: DHCPDISCOVER from da:7a:36:e9:37:ac via enp12s0
Apr 09 16:20:49 daq11.triumf.ca dhcpd[10650]: DHCPOFFER on 192.168.1.1 to da:7a:36:e9:37:ac via enp12s0
Apr 09 16:20:49 daq11.triumf.ca dhcpd[10650]: DHCPREQUEST for 192.168.1.1 (192.168.1.253) from da:7a:36:e9:37:ac via enp12s0
Apr 09 16:20:49 daq11.triumf.ca dhcpd[10650]: DHCPACK on 192.168.1.1 to da:7a:36:e9:37:ac via enp12s0

5) Assign hostname to this IP

[root@daq11 tmp]# more /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.1     trb171

6) Open a hole in firewall to the TRB3

firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.168.1.1/32" port protocol="tcp" port="0-65535" accept"
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.168.1.1/32" port protocol="udp" port="0-65535" accept"
firewall-cmd --add-port=5071/udp --permanent
firewall-cmd --reload

Centos-7 Software Setup

The TRB3 software and associated analysis packages are developed on newer operating systems than Centos-7 (namely Opensuse). So a certain amount of work is needed to get the software to work on Centos-7.

1) I started by copying over the software in trbsoft from an existing installation.

scp trbsoft_backup.tar.gz trb3@ladd00:/data0/trb3/.

2) The main problem is with the TRB3 code requiring a newer version of perl. I used perlbrew to setup a newer version of perl than Centos-7 supports:

Following the instructions on website:

https://perlbrew.pl/

source ~/perl5/perlbrew/etc/bashrc
perlbrew init
perlbrew install perl-5.24.0

and then add this line to login script

rm -f /home/trb3/.perlbrew/init
source ~/perl5/perlbrew/etc/bashrc
perlbrew switch perl-5.24.0 

3) Rebuild libtrbnet with different perl

cd ~/trbsoft/trbnettools/libtrbnet_perl
perl Makefile.PL
cd ..
make TRB3=1 clean
make TRB3=1
make TRB3=1 install

4) Modify

~/trbsoft/daqtools/merge_serial_address.pl
and
~/trbsoft/daqtools/web/cts_gui

to use the perlbrew version of perl, rather than the system default.

5) Rebuild DABC (optional: this step is not necessary if we use the UDP version of the frontend, which is default now)

Make sure you have ROOT (ie ROOTSYS) setup properly

 svn co https://subversion.gsi.de/dabc/trb3 trb3
cd trb3
make

This seems to build dabc and stream successfully. go4 doesn't compile cleanly, but this doesn't matter for the MIDAS frontend readout.

6) Modify the IP and MAC address in the file trbsoft/daqtools/users/triumf_trb171/db/register_configgbe_ip.db. This should be the IP and MAC address of the private IP interface on the server computer; so the IP is 192.168.1.253 in our case; use ipconfig to check the MAC.

7) Setup your login script (BASH in this case):

DAQ_TOOLS_PATH=~/trbsoft/daqtools
USER_DIR=~/trbsoft/daqtools/users/triumf_trb171
TRB_WEB_DIR=$DAQ_TOOLS_PATH/web

export PATH=$PATH:$DAQ_TOOLS_PATH:$HOME/trbsoft/trbnettools/bin
export PATH=$PATH:$DAQ_TOOLS_PATH/tools
export PATH=$PATH:$USER_DIR
export PATH=$PATH:$HOME/trbsoft/trb3/dabc/bin/
export LD_LIBRARY_PATH=$HOME/trbsoft/trb3/dabc/lib/

# Use perl 5.24
rm -f /home/trb3/.perlbrew/init
source ~/perl5/perlbrew/etc/bashrc
perlbrew switch perl-5.24.0 

### TL
### This part restarts the trbnetd server;
### this server takes instructions from trbcmd executable and
### passes them actually to the TRB.
export TRB3_SERVER=trb171:26000
export TRBNETDPID=$(pgrep -f "trbnetd -i 171")

export DAQOPSERVER=localhost:171

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/home/trb3/trbsoft/trbnettools/libtrbnet

test -s ~/.alias && . ~/.alias || true

# PATH="/home/trb3/perl5/bin${PATH:+:${PATH}}"; export PATH;
PERL5LIB="/home/trb3/perl5/perlbrew/perls/perl-5.24.0/lib/5.24.0/:/home/trb3/trbsoft/daqtools/web/include:/home/trb3/trbsoft/trbnettools/libtrbnet_perl/blib/lib:/usr/share/perl5/vendor_perl"; export PERL5LIB;
#PERL_LOCAL_LIB_ROOT="/home/trb3/perl5${PERL_LOCAL_LIB_ROOT:+:${PERL_LOCAL_LIB_ROOT}}"; export PERL_LOCAL_LIB_ROOT;
#PERL_MB_OPT="--install_base \"/home/trb3/perl5\""; export PERL_MB_OPT;
#PERL_MM_OPT="INSTALL_BASE=/home/trb3/perl5"; export PERL_MM_OPT;
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/home/trb3/trbsoft/trbnettools/libtrbnet_perl/blib/arch/auto/HADES/TrbNet/


# setup ROOT 
source ~/packages/root/bin/thisroot.sh 

# setup MIDAS
export MIDASSYS=${HOME}/packages/midas
export PATH=${PATH}:${MIDASSYS}/linux/bin
export MIDAS_EXPTAB=${HOME}/online/exptab
export ROOTANASYS=${HOME}/packages/rootana

8) Now start the trb3 program and initialize the TRB3

cd ~/trbsoft/daqtools/users/triumf_trb171
source startup.sh

9) If this is a new TRB3, need to modify the following files,

trbcmd i 0xffff  
# check open to figure out the serial number of new TRB3 FPGAs
# use this info to modify following files:
vi db/addresses_trb3.db
vi $DAQ_TOOLS_PATH/base/serials_trb3.db
vi db/register_configtdc.db

TRB3 MIDAS implementation

We have a working MIDAS readout of the TRB3 TDC data. Currently this readout uses a lot of the original GSI tools for setting up the TRB3 before the readout can start. At this time you need to do steps 1 and 2 described in "Start up instructions" in order to setup the TRB3 before starting the MIDAS frontend. Eventually we will try to remove as many as possible of the original GSI tools.

MIDAS frontend code is available here:

https://bitbucket.org/ttriumfdaq/trb3_frontend

The MIDAS frontend is setup to read events directly from the UDP buffer. You can see a running MIDAS TRB3 frontend here:

Setting up for MIDAS data taking

We are using the account trb3@daq11.triumf.ca for running this DAQ. If you need to restart the DAQ system, do the following

• Run the setup program that will initialize the TRB3 and set the 10kHz software trigger

 
cd /home/trb3/trbsoft/daqtools/users/triumf_trb171
source startup.sh

• (Optional) We can also start a webgui that allows us to check/configure the TRB3 separate from MIDAS; this is sometimes useful for monitoring. To do this, do

cd /home/trb3/trbsoft/daqtools/web
export PERL5LIB=/home/trb3/perl5/perlbrew/perls/perl-5.24.0/lib/5.24.0/:/home/trb3/trbsoft/daqtools/web/include:/home/trb3/trbsoft/trbnettools/libtrbnet_perl/bli b/lib:/usr/share/perl5/vendor_perl
./cts_gui --noopenxterm --port=1234 --endpoint=0xc001

This allows access to GSI web GUI.

• Start the MIDAS TRB3 readout frontend, if not running:

/home/trb3/online/trb3_frontend/fetrb3UDP.exe -D

Actually, the default fetrb3UDP program does not create all the ODB settings that it should. It misses one setting, which specifies the name of the bank for a particular address. Create this variable after running the program the first time, using odbedit:

[local:trb3:R]Settings> cd /Equipment/TRB3_UDP/Settings
[local:trb3:S]Settings>create STRING trb765          
String length [32]:
[local:trb3:S]Settings>move trb765 1
[local:trb3:S]Settings>set trb765 TRBA

As noted, we currently (April 9) we have the TRB3 setup with a 10kHz software trigger.

Running the analyzer

I added simple decoder and histograming classes to rootana. Currently I have only implemented the crude TDC calibration into the TRB3 bank decoders. In the long run we may want to integrate the fine TDC calibration directly into the frontend.

To look at the analyzer display, run the following

/home/trb3/online/trb3_frontend/analyzer/anaDisplay.exe

The display shows different histograms, arranged by FPGA and channel number. Histograms are

1. The TDC times for each channel, with crude calibration applied
2. The TDC raw fine times; ie, the raw counts for the sub-5ns part of the TDCs
3. TDC time difference histograms; for this canvas you can set the reference FPGA and channel

Analyzer notes:

• On the physical TRB3 board the FPGAs are numbered as being 1-4, but the analyzer converts these to the more normal 0-3 indexing.
• Channel 0 is also the TDC associated with the actual trigger; in our case, the time of the software trigger
• For FPGA 0 and 1 the firmware is configured so we are reading both leading and falling TDC edges; so for these FPGAs channel 1 is the leading edge of the first input and channel 2 is the trailing edge of the first input, and so on. For FPGAs 2 and 3 we are only reading the leading edge of the TDCs.

To dump data from MIDAS file, do following

cd /home/trb3/online/trb3_frontend/analyzer
./ana.exe /home/trb3/online/run00026.mid.gz

Modify ana.cxx to analyze the TDC data and type 'make' to recompile ana.exe.