Setup MIDAS experiment at TRIUMF

From MidasWiki
Jump to navigation Jump to search

Prepare the user account

  • Setup the user account for running this instance of midas. For machines part of the LADD cluster, follow these <a href="http://daq-plone.triumf.ca/SM/docs/local/NewLaddUser">instructions</a>
  • check that the account is using the /bin/tcsh shell
  • make $HOME/.cshrc look like this:
#!/bin/echo You must source

setenv LANG C
setenv SVN_EDITOR "emacs -nw"
setenv GIT_EDITOR "emacs -nw"
setenv CVS_RSH ssh
setenv MIDASSYS $HOME/packages/midas
setenv ROOTSYS  $HOME/packages/root
setenv MIDAS_EXPTAB $HOME/online/exptab

# setup the MIDAS mserver

switch (`hostname`)
case ladd05*:
    unsetenv MIDAS_SERVER_HOST
    breaksw
default:
    setenv MIDAS_SERVER_HOST ladd05.triumf.ca:7071
endsw

# select 64-bit or 32-bit MIDAS and ROOT

switch (`uname -i`)
case i386:
    #export ROOTSYS=/triumfcs/trshare/olchansk/root/root_v5.20.00_SL45_32
    setenv ROOTSYS /triumfcs/trshare/olchansk/root/root_v5.28.00_SL55_32
    setenv PATH .:$MIDASSYS/linux-m32/bin:$PATH
    breaksw
default:
    #export ROOTSYS=/triumfcs/trshare/olchansk/root/root_v5.26.00b_SL54_64
    #setenv ROOTSYS /triumfcs/trshare/olchansk/root/root_v5.28.00_SL55_64
    setenv ROOTSYS $HOME/packages/root
    setenv PATH .:$MIDASSYS/linux/bin:$PATH
endsw

setenv PATH .:$HOME/online/bin:$HOME/packages/roody/bin:$ROOTSYS/bin:$PATH

#end
  • mkdir $HOME/packages
  • Logout and login again, for .cshrc changes to take effect

Install ROOT

  • Identify the Linux version: RH9 (Red Hat Linux 9), FC3 (Fedora Core 3), RHEL4/SL4 (Red Hat Enterprise LInux 4/Scientific Linux 4), SL5, SL6: more /etc/redhat-release
  • Decide to use 32-bit or 64-bit ROOT
  • cd $HOME/packages
  • ls -l /triumfcs/trshare/olchansk/root/ ### to see all available ROOT packages
  • ln -s /triumfcs/trshare/olchansk/root/root_vNNN_VVV_BB root, where NNN is the latest available version of ROOT ("ls -l /triumfcs/trshare/olchansk/root"), VVV is the Linux version code (RH9, FC3, SL4, etc) and BB is "_32" or "_64" for 32-bit or 64-bit ROOT. For example: /triumfcs/trshare/olchansk/root/root_v5.10.00_SL40
  • Check that ROOT works: "echo $ROOTSYS", "$ROOTSYS/bin/root"

Install MIDAS

Install ROOTANA

Install ROODY

Prepare VME hardware

Hardware check list:

Install Universe-II VME driver (V7648, V7750, V7805, V7851)

  • login as root (ssh root@localhost)
  • get latest version of vmic driver from ladd00: scp username@ladd00:/home/olchansk/daq/vmisft-7433-NNN-KOMMM.tar.gz .
  • tar xzvf vmisft-7433-3.5-KO2.tar.gz
  • chown -R root.root vmisft-7433-3.5-KO2
  • cd vmisft-7433-3.5-KO2
  • cd vme_universe
  • make
  • make install
  • cd ..
  • make
  • edit /etc/rc.local, add these lines: 
    modprobe vme_universe
mkdir -p /dev/bus/vme
mknod /dev/bus/vme/ctl c 221 8
chmod a+wr /dev/bus/vme/ctl
  • run "modprobe vme_universe", run "lsmod" to check that the vme_universe module was loaded
  • run "ls -l /dev/bus/vme/ctl" to check that the VME device file exists, it should be "crw-rw-rw- 1 root root 221, 8 Feb 17 15:47 /dev/bus/vme/ctl"

 

Install Tsi-148 VME driver (V7865)

  • login as root (ssh root@localhost)
  • yum install kernel-devel
  • get latest version of the driver from ladd00: scp username@ladd00:/home/olchansk/daq/v7865/v7865-sdk-linux-R01.00-KONNN.tar.gz
  • tar xzvf v7865-sdk-linux-R01.00-KO6.tar.gz
  • chown -R root.root v7865-sdk-linux-R01.00-KO6
  • cd v7865-sdk-linux-R01.00-KO6/gefvme/module
  • make
  • make install
  • cd $HOME
  • edit /etc/rc.local, verify that it has these 2 lines:
    modprobe gefvme
sh /root/gefvme-makedevs
  • run "modprobe gefvme", run "lsmod" to check that the gefvme module was loaded
  • run "/home/olchansk/daq/vme/vmescan_gef.exe" to check that vme access works.

Setup the experiment environement

  • decide which computer will host MIDAS (where MIDAS shared memory buffers will reside). This computer will run the mserver, mlogger and mhttpd.
  • in .cshrc put the name of this computer into the section for setting MIDAS_SERVER_HOST. Note that multiple experiments can run on the same computer by using different mserver ports (7071, mhttpd ports (8081) and roody ports (9091)
  • mkdir $HOME/online
  • cd $HOME/online
  • mkdir bin src elog history
  • mkdir -p /ladd/data1/t2kvme5/data; ln -s /ladd/data1/t2kvme5/data $HOME/online
  • create the exptab file "$HOME/online/exptab" following the example below. The first entry (exptname) is the name if the DAQ system (MIDAS experiment name), the second entry (/home/USER/online) is the location of MIDAS shared memory buffers (by convention, $HOME/online), the third entry (kopio03) is your username.
exptname /home/kopio03/online kopio03
  • logout and login again for all changes to take effect

Setup experiment startup scripts

  • login to the experiment host computer
  • echo $MIDAS_SERVER_HOST ### to check correct value - should be blank
  • create $HOME/online/bin/start_daq.sh, replacing XXX with the hostname of the machine running the experiment (and changing the mserver and mhttpd ports, as needed), replacing machine names in the access control list for mhttpd and mserver as needed. (NOTE: mhttpd ladd00 permission is for password-protecting the experiment via the password protected SSL proxy at https://ladd00.triumf.ca/expt/dragon/, see ladd00:/etc/httpd/conf.d/ssl.conf)
#!/bin/sh

cd $HOME/online

case `hostname` in
XXX*)
    echo "Good, we are on XXX!"
    ;;
*)
    echo "The start_daq script should be executed on XXX"
    exit 1
    ;;
esac

odbedit -c clean

mhttpd  -p 8081 -D -a ladd00.triumf.ca -a localhost -a XXX.triumf.ca
mserver -p 7071 -D -a localhost -a lxdragon01.triumf.ca -a lxdragon02.triumf.ca -a XXX.triumf.ca
mlogger -D

#end file

Setup experiment database (ODB)

  • run $HOME/online/bin/start_daq.sh
  • open the midas status page at http://localhost:8081 (you will see most stuff "red" as nothing is running yet)
  • create a frontend program (mlogger will not run without at least one equipment)
    • cd $HOME/online/src, copy Makefile, fevme.cxx
    • make (creates fevme.exe or fevme_gef.exe executable)
  • run ./fevme.exe (on the computer with the VME interface, could be different from computer hosting the experiment), observe that corresponding equipments have been created
  • odbedit, run these commands: (replace user names and directory names)
set "/Logger/Message file" "/home/kopio03/online/midas.log"
set "/Logger/Data Dir"     "/home/kopio03/online/data"
create STRING "/Logger/History dir"
set "/Logger/History dir"  "/home/kopio03/online/history"
create STRING "/Logger/Elog dir"
set "/Logger/Elog dir"     "/home/kopio03/online/elog"
exit
  • run $HOME/online/bin/start_daq.sh (observe that mlogger has started)
  • odbedit, run these commands: (replace user names and directory names)
set "/Logger/ODB dump file" "/home/kopio03/online/history/run%05d.xml"
set "/Logger/ODB dump" "y"
set "/Logger/Channels/0/Settings/Filename" "run%05dsub%03d.mid.gz"
set "/Logger/Channels/0/Settings/Subrun byte limit" "1000000000"
set "/Logger/Channels/0/Settings/Compression"   1
set "/Logger/Channels/0/Settings/ODB Dump" "y"
set "/Programs/Logger/Required" y
set "/Programs/Logger/Start command" "mlogger -D"
set "/Programs/fevme/Required" "y"
set "/Programs/fevme/Start command" "ssh -n lxdaq09 $HOME/online/src/fevme_gef.exe -O"
exit
  • open web browser: firefox http://localhost:8081
  • save the url bookmark to the "personal toolbar"
  • go to the Programs page, stop mlogger, stop fevme, start mlogger, start fevme
  • go to the Status page, start run, stop run
  • go back to the Status page, everything should be green
  • start a run
  • send signals to the ADC gate
  • you should be getting events
  • to look at data, proceed with setting up the <a href="../../../SR/rootana">ROOT analyzer</a>

Start DAQ programs at boot time

  • add this to /etc/rc.local (replace username and location of the start_daq script)
su - alpha -c /home/alpha/online/bin/start_daq.sh

Setup local software version control

Version control for experiment source code is setup using "git" (http://git-scm.com/)

  • cd $HOME/online
  • git init
  • git add exptab
  • git add bin/start_daq.sh
  • git add .gitignore ### contents can be
*~
*.o
*.exe
  • git add src/Makefile src/*.cxx ...
  • git commit -a

Adjust MIDAS buffer sizes

Default MIDAS SYSTEM buffer size is 8 Mbytes, fairly small for high-data-rate experiments. The rule of thum is to have at least a few seconds worth of buffer space available. For example, if event size is 10 Kbytes and the event rate is 1 kHz, data rate is 10*10^3*1*10^3 = 10 Mbytes/sec. To buffer 10 seconds of data we need 100 Mbytes of buffer space.

To resize the MIDAS event buffers (SYSTEM, etc) do this:

  • stop all frontends, stop mlogger
  • start odbedit:
    • cd "/Experiment/Buffer sizes"
    • set SYSTEM 100000000
  • run "mdump -z SYSTEM"
  • if mdump complains about the size of .SYSTEM.SHM, remove it, try again.
  • ls -l /dev/shm ### to observe that the size of shared memory is correct


Secure MIDAS and Elog (PRELIMINARY)

xxx

HOWTO setup midas and elog with a password protected SSL proxy

 

Introduction

 

Often, access to midas and elog do not have good password protection because they use the un-encrypted http:// access method, where passwords are sent in clear text and can be easily sniffed as they travel over the network.

 

One way to password-protect midas/mhttpd and elog is through a password-protected SSL proxy, for example, using apache httpd. In this configuration, one uses the Linux stock httpd that accepts encrypted https:// connections and forwards them to mhttpd and elogd. Instead of using mhttpd and elogd passwords, one configures password protection in httpd via the regular apache httpd mechanisms.

 

Recent versions of elogd do support SSL https:// connections, but since one has to run an SSL proxy for securing access to mhttpd anyway, it is simpler to run both through the same SSL proxy using the same SSL host certificate and the same httpd password file.

 

It is up to the local administrator whether to enable bypasses of the SSL proxy for local non-password protected access to mhttpd and elog. For mhttpd, this is done using the "-a hostname" switch. Normally there will be only "-a localhost" switch, enabling access only for the local machine (where the SSL proxy is running). Additional "-a hostname" switches enable access from listed local machines. No "-a xxx" enables access from everywhere (defeating the purpose of the SSL proxy, unless access controls are enforced elsewhere, i.e. by a site firewall or by local firewall rules). For elogd, this is done using the "-n localhost" switch with enables only access from the same machine if present, or access from anywhere is absent (defeating the purpose of the SSL proxy, unless access controls are enforced elsewhere).

 

It is recommended to run elogd from the same user as the main daq user and to keep elogd.cfg and all logbooks in the home directory of this user, where they are captured by the normal site backup system.

 

Follow these easy steps:

 

  • Install Elog
    • install the elog rpm (from https://midas.psi.ch/elog/download/RPMS)
    • cd $HOME/elog
    • cp -rpv /usr/local/elog/* .
    • (to import elogs from mhttpd elog: cd logbooks; ln -s /home/t2km11/online/elog midas; cd midas; /usr/local/bin/elconv)
    • edit start_elogd to read:
#!/bin/sh

killall elogd
sleep 1
killall -KILL elogd
sleep 1
/usr/local/sbin/elogd -n localhost -x -c $HOME/elog/elogd.cfg -p 8082

#end
    • edit elogd.cfg to read:
[global]
port = 8082
usr = t2km11
grp = t2km11
SMTP host = smtp.triumf.ca
URL = https://xxx/elog/
#URL = http://xxx:8082

Reverse sort = 1
Display Mode = full

List Menu commands = New, Find, Admin, Help
Menu commands = New, Edit, Reply, Find, Duplicate, Help

Entries Per Page = 30
Supress Email on edit = 1
Default encoding = 1
Page title = T2K M11 ELOG
Resolve host names = 1

Logfile = /home/t2km11/elog/elogd.log
#Logging level = 3

[midas]

List page Title = T2K M11 MIDAS ELOG
Comment = T2K M11 MIDAS ELOG
Page Title = T2K M11 MIDAS ELOG
RSS Title = [$logbook - $type - $system] $subject, posted by $author

Attributes = Author, Subject, Run, Type, System
Show Attributes Edit = Run, Author, Subject, Type, System
Required Attributes = Author, Type, System, Subject

Options Type = Routine, Reply, Shift Summary, Modification, Question, Info, Problem
Options System = General, DAQ, Beamline

Preset Run = $shell(MIDASSYS=. /home/t2km11/packages/midas/linux/bin/odbedit -d Runinfo -c 'ls -v \"run number\"')

Preset On Reply Type = Reply
Preset On Reply Run = $shell(MIDASSYS=. /home/t2km11/packages/midas/linux/bin/odbedit -d Runinfo -c 'ls -v \"run number\"')

List Display = Date, Subject, Type, System, Author, ID
Quick Filter = Date, Type, ID

Remove on reply = Author
Quote on reply = 1

Use lock = 1

************* Email Functionality ****************

Use Email Subject = [T2KM11 - $System] $Subject
Omit Email To = 1

Email System General = xxx
  • Install SSL proxy
    • login as root to the SSL Proxy machine.
    • create SSL certificate (see https://trmail.triumf.ca/CA/server.html)
      • cd ~root
      • openssl req -new -nodes -out ladd09.csr -keyout ladd09.key (answer: CA, BC, Vancouver, TRIUMF, DAQ, *.triumf.ca, email@email.com
      • Mail -s "Andrew, please sign and return to email@email.com" andrew@email.com < ladd09.csr
      • (for now) self-sign the certificate: openssl x509 -req -days 365 -in ladd09.csr -signkey ladd09.key -out ladd09.crt
    • yum install mod_ssl

      • edit /etc/httpd/conf.d/ssl.conf to read (add at the very bottom, right before the "</VirtualHost>" entry. This assumes mhttpd is running on port 8081, elogd is running on port 8082.

    SSLCertificateFile /root/ladd09.crt
SSLCertificateKeyFile /root/ladd09.key
ProxyPass /elog/ http://localhost:8082/ retry=1
ProxyPass /      http://localhost:8081/ retry=1
      • comment out duplicate "SSLCertificateFile" and "SSLCertificateKeyFile" elsewhere in the file
      • add password protection: again, right before "</VirtualHost>", add this:
    <Location />
SSLRequireSSL
AuthType Basic
AuthName "password protected site"
Require valid-user

# create password file: touch /etc/httpd/htpasswd
# to add new user or change password: htpasswd /etc/httpd/htpasswd username
AuthUserFile /etc/httpd/htpasswd

</Location>
    • service httpd restart
    • Test stuff:
    • Configure MIDAS:

      • in ODB, set "/Elog/URL" to "https://host/elog/midas", set "/Elog/Logbook Dir" to "/home/t2km11/elog/logbooks/midas" (or History "make Elog" would not work).

      • now from the midas status page, the "Elog" button should take us to the https Elog URL. From the History panel, "Create Elog" should yield a "New entry edit page", with the history image already attached and visible.
    • Troubleshoot:

     

      K.O.

       

      Retrieved from "https://daq00.triumf.ca/MidasWiki/index.php?title=Setup_MIDAS_experiment_at_TRIUMF&oldid=231"