Setup MIDAS experiment at TRIUMF: Difference between revisions

From MidasWiki
Jump to navigation Jump to search
No edit summary
Line 1: Line 1:
{{Pagelinks}}
{{Pagelinks}}
=== Standard layout of MIDAS experiment ===
== Standard layout of MIDAS experiment ==


<pre>
<pre>
Line 25: Line 25:
</pre>
</pre>


=== Prepare the user account ===
== Prepare the user account ==
<div id="NOTES"></div>
<div id="NOTES"></div>
;NOTES
;NOTES
* MIDAS versions August 2015 and later have enhanced [[Security]] and use different port numbers. Instructions for older versions differ slightly.
* MIDAS versions August 2015 and later have enhanced [[Security]] and the default port numbers are different. Instructions for older versions differ slightly.
** follow the <span style="color:green">green</span> instructions for new (August 2015 or later) versions
** follow the <span style="color:green">green</span> instructions for new (August 2015 or later) versions
** follow the  <span style="color:orange">orange</span> instructions for older versions
** follow the  <span style="color:orange">orange</span> instructions for older versions


* The '''Default Ports''' are different between the two versions:
* The '''Default Ports''' are different between the two versions:
** <span style="color:green">default ports (since August 2015) are mserver (1175), mhttpd (8443) and roody (9091) </span>
** <span style="color:green">default ports (since August 2015) are mserver (1175), mhttpd (8080 and 8443) </span>
** <span style="color:orange">default ports are mserver (7071), mhttpd (8081) and roody (9091) </span>
** <span style="color:orange">default ports are mserver (7071), mhttpd (8081) </span>


The default port for ROODY (9091) is unchanged.


* Setup the user account for running this instance of midas. For machines part of the LADD cluster, follow these  [http://daq-plone.triumf.ca/SM/docs/local/NewLaddUser] instructions.
* Setup the user account for running this instance of midas. For machines part of the LADD cluster, follow these  [http://daq-plone.triumf.ca/SM/docs/local/NewLaddUser] instructions.
Line 85: Line 86:
* Logout and login again, for .cshrc changes to take effect
* Logout and login again, for .cshrc changes to take effect


=== Install ROOT ===
== Install ROOT ==


* Identify the Linux version: RH9 (Red Hat Linux 9), FC3 (Fedora Core 3), RHEL4/SL4 (Red Hat Enterprise LInux 4/Scientific Linux 4), SL5, SL6: more /etc/redhat-release
* Identify the Linux version: RH9 (Red Hat Linux 9), FC3 (Fedora Core 3), RHEL4/SL4 (Red Hat Enterprise LInux 4/Scientific Linux 4), SL5, SL6: more /etc/redhat-release
Line 94: Line 95:
* Check that ROOT works: "echo $ROOTSYS", "$ROOTSYS/bin/root"
* Check that ROOT works: "echo $ROOTSYS", "$ROOTSYS/bin/root"


=== Install MIDAS ===
== Install MIDAS ==


* cd $HOME/packages
* cd $HOME/packages
Line 110: Line 111:
You can see a list of other installation problems at [[Common problems & Debugging recipes]].
You can see a list of other installation problems at [[Common problems & Debugging recipes]].


=== Install ROOTANA ===
== Install ROOTANA ==


* cd $HOME/packages
* cd $HOME/packages
Line 117: Line 118:
* make
* make


=== Install ROODY ===
== Install ROODY ==


* cd $HOME/packages
* cd $HOME/packages
Line 125: Line 126:
* $HOME/packages/roody/bin/roody, run the program
* $HOME/packages/roody/bin/roody, run the program


=== Prepare VME hardware ===
== Prepare VME hardware ==


Hardware check list:
Hardware check list:
Line 138: Line 139:
** ./vmescan.exe (or _gef.exe, depending on the VME driver in use)
** ./vmescan.exe (or _gef.exe, depending on the VME driver in use)


==== Install Universe-II VME driver (V7648, V7750, V7805, V7851) ====
== Install Universe-II VME driver (V7648, V7750, V7805, V7851) ==


* login as root (ssh root@localhost)
* login as root (ssh root@localhost)
Line 159: Line 160:
* cd ~/packages/vme; vmescan.exe
* cd ~/packages/vme; vmescan.exe


==== Install Tsi-148 VME driver (V7865) ====
== Install Tsi-148 VME driver (V7865) ==


* login as root (ssh root@localhost)
* login as root (ssh root@localhost)
Line 176: Line 177:
* cd ~/packages/vme; vmescan_gef.exe
* cd ~/packages/vme; vmescan_gef.exe


=== Setup the experiment environment ===
== Setup the experiment environment ==


* decide which computer will host MIDAS (where MIDAS shared memory buffers will reside). This computer will run the mserver, mlogger and mhttpd.
* decide which computer will host MIDAS (where MIDAS shared memory buffers will reside). This computer will run the mserver, mlogger and mhttpd.
Line 189: Line 190:
* logout and login again for all changes to take effect
* logout and login again for all changes to take effect


=== Setup experiment startup scripts ===
== Setup experiment startup scripts ==
* login to the experiment host computer
* login to the experiment host computer
* echo $MIDAS_SERVER_HOST ### to check correct value - should be blank
* echo $MIDAS_SERVER_HOST ### to check correct value - should be blank
Line 227: Line 228:
  #end file
  #end file


=== Setup experiment database (ODB) ===
== Setup experiment database (ODB) ==


* <span style="color:green"> Start mhttpd once on command line. You will get an error saying you need to create a mongoose password file with command htdigest; see [[mhttpd]] for details. </span>
* <span style="color:green"> Start mhttpd once on command line.  
  You will get an error saying you need to create a mongoose password file with command htdigest; see [[mhttpd]] for details. </span>
* run $HOME/online/bin/start_daq.sh
* run $HOME/online/bin/start_daq.sh
* open the midas status page at either (see [[#NOTES]])
* open the midas status page at either (see [[#NOTES]])
Line 271: Line 273:
* to look at data, proceed with setting up the <a href="../../../SR/rootana">ROOT analyzer</a>
* to look at data, proceed with setting up the <a href="../../../SR/rootana">ROOT analyzer</a>


=== Start DAQ programs at boot time ===
== Start DAQ programs at boot time ==


* add this to /etc/rc.local (replace username and location of the start_daq script)
* add this to /etc/rc.local (replace username and location of the start_daq script)
Line 278: Line 280:
</pre>
</pre>


=== Setup local software version control ===
== Setup local software version control ==


Version control for experiment source code is setup using "git" (http://git-scm.com/)
Version control for experiment source code is setup using "git" (http://git-scm.com/)
Line 295: Line 297:
* git commit -a
* git commit -a


=== Adjust MIDAS buffer sizes ===
== Adjust MIDAS buffer sizes ==


Default MIDAS SYSTEM buffer size is 8 Mbytes, fairly small for high-data-rate experiments. The rule of thumb is to have at least a few seconds worth of buffer space available. For example, if event size is 10 Kbytes and the event rate is 1 kHz, data rate is 10*10^3*1*10^3 = 10 Mbytes/sec. To buffer 10 seconds of data we need 100 Mbytes of buffer space.
Default MIDAS SYSTEM buffer size is 8 Mbytes, fairly small for high-data-rate experiments. The rule of thumb is to have at least a few seconds worth of buffer space available. For example, if event size is 10 Kbytes and the event rate is 1 kHz, data rate is 10*10^3*1*10^3 = 10 Mbytes/sec. To buffer 10 seconds of data we need 100 Mbytes of buffer space.
Line 308: Line 310:
* ls -l /dev/shm ### to observe that the size of shared memory is correct
* ls -l /dev/shm ### to observe that the size of shared memory is correct


== Secure MIDAS and Elog Web access  ==
== Secure MIDAS and ELOG Web access  ==
In versions prior to May 2015, the default web access to MIDAS and ELOG uses the "http:" protocol which is insecure. In this case, all information is transmitted as clear text meaning that secret, confidential and sensitive information (such as the MIDAS and ELOG passwords and usernames) can be stolen "easily". This means that even "password protected" MIDAS and ELOG pages are not really protected if accessed using the "http" method.
Better security for HTTP is gained by using a password protected '''SSL (https) proxy'''. (It does not provide absolute security because of remaining problems with the security of SSL certificates, security of passwords, etc). Setting up an SSL (https) proxy is described [[#Setting up an HTTP proxy|below]].
 
Since May 2015, an ''alternative secure option'' to setting up an HTTP proxy is available to users of MIDAS. Recent versions of elogd (ELOG) do support SSL https:// connections, and [[#mhttpd with HTTPS/SSL server (Mongoose)]] is now available.  This option provides a similar level of security to an HTTP proxy.
 
=== Which security option to choose? ===
For a small operation that does not require root access to the host computer, mhttpd with mongoose is a good light weight solution. It is ideal for small experiments that until now have been using the "http:" protocol for mhttpd.
 
For a more elaborate setup with private networks etc., APACHE https proxy is probably better. For big experiments, resources like webcams,ganglia, couchdb, etc. also need password protection and APACHE https can provide all this.
=== Which one is easier to setup? ===
 
* mhttpd + mongoose is fairly simple - you have to create a password file and (optional) a properly signed https certificate.
* apache httpd is fairly straightforward if you follow well written instructions (such as we provide for using it with midas), but you do need root access and you do have to edit a good number of config files.
 


=== mhttpd with HTTPS/SSL server (Mongoose) ===
=== mhttpd with HTTPS/SSL server (Mongoose) ===


Since May 2015 the MIDAS web server [[mhttpd]] is explicitly linked with OpenSSL to provide secure HTTPS connections via the [https://bitbucket.org/tmidas/midas/src/ecb9a8537448a8a43f7f9a2bfdb82e578208cde3/doc/mongoose/?at=develop Mongoose] web server (see [[mhttpd]]). With this version, default web access to MIDAS uses the "https" protocol. Web access to {{Utility|name=mhttpd}} can be restricted by using the [[#Access Control List|Access Control List]].  The first time {{Utility|name=mhttpd}} is run, an SSL certificate and a password file must be created. See [[mhttpd#HTTPS/SSL server (Mongoose)|HTTPS/SSL server (Mongoose)]] for instructions. Recent versions of elogd  also support SSL https:// connections. This means it is no longer
Since May 2015 the MIDAS web server [[mhttpd]] is explicitly linked with OpenSSL to provide secure HTTPS connections via the [https://bitbucket.org/tmidas/midas/src/ecb9a8537448a8a43f7f9a2bfdb82e578208cde3/doc/mongoose/?at=develop Mongoose] web server (see [[mhttpd]]). With this version, default web access to MIDAS uses the "https" protocol. Web access to {{Utility|name=mhttpd}} can be restricted by using the [[#Access Control List|Access Control List]].  The first time {{Utility|name=mhttpd}} is run, a default SSL certificate is used, and a password file must be created. For additional security, the user should create and use their own SSL certificate. See [[mhttpd#HTTPS/SSL server (Mongoose)|HTTPS/SSL server (Mongoose)]] for instructions.
necessary to set up a SSL (https) proxy as described below.




=== mhttpd using an HTTPS/SSL proxy ===
An [[#Setting up an HTTP proxy|HTTP proxy]] must be set up. This is the only way of securing older version of [[mhttpd]] (pre August 2015). Older versions of mhttpd are started using the -p port option  e.g.
* <span style="color:orange"> mhttpd  -D -p 8080  </span>


== Insecure MIDAS and Elog Web access ==
To run a new version of mhttpd using an HTTP proxy, use the options provided to run the old (non-Mongoose) webserver on a given port, i.e.
* <span style="color:green">  mhttpd --oldserver 8080 --nomg  -D </span>


=== Old versions of mhttpd with HTTP ===
When using an SSL proxy, only access from the SSL proxy (and maybe some special trusted machines) should be permitted.  
In versions prior to May 2015, the default web access to MIDAS and ELOG uses the "http:" protocol which is insecure. All information is transmitted as clear text meaning that secret, confidential and sensitive information (such as the MIDAS and ELOG passwords and usernames) can be stolen "easily". This means that even "password protected" MIDAS and ELOG pages are not really protected if accessed using the "http" method.  For this reason, it is recommended that users update to [[#mhttpd with HTTPS/SSL server (Mongoose)]].
This is done using the "-a hostname" switch of [[mhttpd]]. Normally there will be only "-a localhost" switch, enabling access only for the local machine (where the SSL proxy is running). Additional "-a hostname" switches enable access from listed local machines. No "-a xxx" enables access from everywhere (defeating the purpose of the SSL proxy, unless access controls are enforced elsewhere, i.e. by a site firewall or by local firewall rules).
 
If this is not possible, somewhat better security for HTTP is gained by using a password protected '''SSL (https) proxy'''. (It does not provide absolute security because of remaining problems with the security of SSL certificates, security of passwords, etc).


== Setting up an HTTP proxy ==
In this example, we use APACHE HTTPD to password-protect a typical midas/mhttpd and elog installation.
In this example, we use APACHE HTTPD to password-protect a typical midas/mhttpd and elog installation.


In this configuration, one uses the Linux stock httpd that accepts encrypted https:// connections and forwards them to mhttpd and elogd. Instead of (or in addition to) using mhttpd and elogd passwords, one configures password protection in httpd via the regular apache httpd password mechanisms (htpasswd, etc).
In this configuration, one uses the Linux stock httpd that accepts encrypted https:// connections and forwards them to mhttpd and elogd. Instead of (or in addition to) using mhttpd and elogd passwords, one configures password protection in httpd via the regular apache httpd password mechanisms (htpasswd, etc).


Recent versions of elogd do support SSL https:// connections, but if one has to run an SSL proxy for securing access to an old version mhttpd anyway,  
Recent versions of elogd do support SSL https:// connections, but if one is running an SSL proxy for anyway, it is simpler to run both through the same SSL proxy using the same SSL host certificate and the same httpd password file.
it is simpler to run both through the same SSL proxy using the same SSL host certificate and the same httpd password file.
 
==== Restricting http: access to old versions of mhttpd ====
 
By default, versions of mhttpd prior to August 2015 accept http connections from anybody. If it's not possible to [[#mhttpd with HTTPS/SSL server (Mongoose)|upgrade]], an SSL Proxy may be used to restrict access.  When using an SSL proxy, only access from the SSL proxy (and maybe some special trusted machines) should be permitted. This is done using the "-a hostname" switch. Normally there will be only "-a localhost" switch, enabling access only for the local machine (where the SSL proxy is running). Additional "-a hostname" switches enable access from listed local machines. No "-a xxx" enables access from everywhere (defeating the purpose of the SSL proxy, unless access controls are enforced elsewhere, i.e. by a site firewall or by local firewall rules).


=== Restricting http: access to elogd ===
=== Restricting http: access to elogd ===
;Note
;Note
:Recent versions of elogd and [[mhttpd]] do support SSL https:// connections and are recommended. The following information is for those still using older versions of mhttpd.
:Recent versions of elogd do support SSL https:// connections. The following information is for those using an HTTP proxy (see above).


For elogd, this is done using the "-n localhost" switch with enables only access from the same machine if present, or access from anywhere is absent (defeating the purpose of the SSL proxy, unless access controls are enforced elsewhere).
For elogd, this is done using the "-n localhost" switch with enables only access from the same machine if present, or access from anywhere is absent (defeating the purpose of the SSL proxy, unless access controls are enforced elsewhere).
Line 344: Line 358:


=== Instructions for installing elogd ===
=== Instructions for installing elogd ===
==== Install Elog ====
<ul>
<ul>
<ul><li>install the elog rpm (from https://midas.psi.ch/elog/download/RPMS)</li><li>cd $HOME/elog</li><li>cp -rpv /usr/local/elog/* .</li><li>(to import elogs from mhttpd elog: cd logbooks; ln -s /home/t2km11/online/elog midas; cd midas; /usr/local/bin/elconv)<br /></li><li>edit start_elogd to read:</li></ul>
<ul><li>install the elog rpm (from https://midas.psi.ch/elog/download/RPMS)</li><li>cd $HOME/elog</li><li>cp -rpv /usr/local/elog/* .</li><li>(to import elogs from mhttpd elog: cd logbooks; ln -s /home/t2km11/online/elog midas; cd midas; /usr/local/bin/elconv)<br /></li><li>edit start_elogd to read:</li></ul>
Line 472: Line 484:
* now from the midas status page, the "Elog" button should take us to the https Elog URL
* now from the midas status page, the "Elog" button should take us to the https Elog URL


== Setup the history mhttpd for faster access to history plots ==
=== Setup the history mhttpd for faster access to history plots ===
When running an SSL proxy,
* start the main mhttpd (<span style="color:orange">orange</span> command for old mhttpd, <span style="color:green">green</span> for new mhttpd with Mongoose(post August2015):
** <span style="color:orange">"mhttpd -p 8071 -D" </span>  or 
** <span style="color:green"> "mhttpd  -D --oldserver 8071 --nomg" </span>


* start the main mhttpd: mhttpd -p 8071 -D"
* start the history mhttpd
* start the history mhttpd: mhttpd -p 8072 -D -H"
** <span style="color:orange"> "mhttpd -p 8072 -D -H"</span> or
** <span style="color:green"> "mhttpd -D -H --oldserver 8072 --nomg" </span>
* set ODB /History/URL to "http://alphacpc09.cern.ch:8072/HS/"
* set ODB /History/URL to "http://alphacpc09.cern.ch:8072/HS/"
* open the MIDAS status page
* open the MIDAS status page
Line 490: Line 507:
** apxs -c -I. -I/usr/include/libxml2 -i mod_xml2enc.c
** apxs -c -I. -I/usr/include/libxml2 -i mod_xml2enc.c
** cd /etc/httpd/conf.d, add this to ssl.conf:
** cd /etc/httpd/conf.d, add this to ssl.conf:
*** before the ProxyPass statements:
 
;before the ProxyPass statements:
<pre>
<pre>
# proxy the MIDAS web servers   
# proxy the MIDAS web servers   
Line 501: Line 519:
ProxyRequests off   
ProxyRequests off   
</pre>
</pre>
*** after the ProxyPass statements:
;after the ProxyPass statements:
<pre>
<pre>
# ALPHA1 history access  
# ALPHA1 history access  
Line 510: Line 528:
ProxyHTMLURLMap http://alphacpc09.cern.ch:8072/HS/ /alpha1/history/  
ProxyHTMLURLMap http://alphacpc09.cern.ch:8072/HS/ /alpha1/history/  
</pre>
</pre>
*** adjust:
;adjust:
**** "alpha1" is the experiment name
*"alpha1" is the experiment name
**** "alphacpc09.cern.ch" is the machine running mhttpd
*"alphacpc09.cern.ch" is the machine running mhttpd
**** "8071" is the port number of the main mhttpd
*"8071" is the port number of the main mhttpd  
**** "8072" is the port number of the history mhttpd (mhttpd -p 8072 -D -H")
** <span style="color:orange">"mhttpd -p 8071 -D" </span>  or 
** <span style="color:green"> "mhttpd  -D --oldserver 8071 --nomg" </span>
* "8072" is the port number of the history mhttpd  
** <span style="color:orange">"mhttpd -p 8072 -D -H" </span>  or 
** <span style="color:green"> "mhttpd  -D -H --oldserver 8072 --nomg" </span>
 
 


[[Category:Contents]] [[Category:Buffer]] [[Category:Driver]]
[[Category:Contents]] [[Category:Buffer]] [[Category:Driver]]

Revision as of 18:10, 6 November 2015


Standard layout of MIDAS experiment

/home/exptuser/
  packages/
    root <---- ROOT (64-bit or 32-bit)
    mxml
    midas <---- MIDAS
      linux/{lib,bin}  <---- binaries matching the selected 64-bit/32-bit flavour of ROOT
      linux-m32/{lib,bin}   <---- limited function 32-bit binaries for 32-bit frontend machines, build by "make linux32"
      linux-m64/{lib,bin}   <---- limited function 64-bit binaries (only needed if ROOT and linux/bin are 32-bit)
      linux-crosscompile/{lib,bin}  <---- cross-compiled limited function binaries for PPC and ARM frontends (see Makefile)
    rootana <---- ROOT analyzer
    roody <---- graphical online histogram viewer for MIDAS and ROOTANA
  online/
    exptab <---- experiment definition
    {.ODB,.SYSTEM,.SYSMSG,etc}.SHM <---- MIDAS shared memory save files
    src <---- experiment frontend sources
    bin,scripts
    elog <---- MIDAS elog
    history <---- MIDAS history
    data -> /data/exptname/current <---- symlink to the data directory
/data/exptname/current <---- experiment data directory with ODB save files and MIDAS .mid/.mid.gz data files

Prepare the user account

NOTES
  • MIDAS versions August 2015 and later have enhanced Security and the default port numbers are different. Instructions for older versions differ slightly.
    • follow the green instructions for new (August 2015 or later) versions
    • follow the orange instructions for older versions
  • The Default Ports are different between the two versions:
    • default ports (since August 2015) are mserver (1175), mhttpd (8080 and 8443)
    • default ports are mserver (7071), mhttpd (8081)

The default port for ROODY (9091) is unchanged.

  • Setup the user account for running this instance of midas. For machines part of the LADD cluster, follow these [1] instructions.
  • check that the account is using the /bin/tcsh shell
  • make $HOME/.cshrc look like this:
#!/bin/echo You must source
#
setenv LANG C
setenv SVN_EDITOR "emacs -nw"
setenv GIT_EDITOR "emacs -nw"
setenv CVS_RSH ssh
setenv MIDASSYS $HOME/packages/midas
setenv ROOTSYS  $HOME/packages/root
setenv ROOTANASYS $HOME/packages/rootana
setenv MIDAS_EXPTAB $HOME/online/exptab
#
# setup the MIDAS mserver
#
switch (`hostname`)
case ladd05*:
   unsetenv MIDAS_SERVER_HOST
   breaksw
default:
setenv MIDAS_SERVER_HOST ladd05.triumf.ca:1175  # latest MIDAS version see #NOTES
# or setenv MIDAS_SERVER_HOST ladd05.triumf.ca:7071   # older MIDAS version see #NOTES
endsw
#
# select 64-bit or 32-bit MIDAS and ROOT
#
switch (`uname -i`)
case i386:
   #export ROOTSYS=/triumfcs/trshare/olchansk/root/root_v5.20.00_SL45_32
   setenv ROOTSYS /triumfcs/trshare/olchansk/root/root_v5.28.00_SL55_32
   setenv PATH .:$MIDASSYS/linux-m32/bin:$PATH
   breaksw
default:
   #export ROOTSYS=/triumfcs/trshare/olchansk/root/root_v5.26.00b_SL54_64
   #setenv ROOTSYS /triumfcs/trshare/olchansk/root/root_v5.28.00_SL55_64
   setenv ROOTSYS $HOME/packages/root
   setenv PATH .:$MIDASSYS/linux/bin:$PATH
endsw
#
setenv PATH .:$HOME/online/bin:$HOME/packages/roody/bin:$ROOTSYS/bin:$PATH
#
#end
  • mkdir $HOME/packages
  • Logout and login again, for .cshrc changes to take effect

Install ROOT

  • Identify the Linux version: RH9 (Red Hat Linux 9), FC3 (Fedora Core 3), RHEL4/SL4 (Red Hat Enterprise LInux 4/Scientific Linux 4), SL5, SL6: more /etc/redhat-release
  • Decide to use 32-bit or 64-bit ROOT ('uname -a')
  • cd $HOME/packages
  • ls -l /daq/daqshare/olchansk/root/ ### to see all available ROOT packages
  • ln -s /daq/daqshare/olchansk/root/root_vNNN_VVV_BB root, where NNN is the latest available version of ROOT ("ls -l /daq/daqshare/olchansk/root"), VVV is the Linux version code (RH9, FC3, SL4, etc) and BB is "_32" or "_64" for 32-bit or 64-bit ROOT. For example: /daq/daqshare/olchansk/root/root_v5.10.00_SL40
  • Check that ROOT works: "echo $ROOTSYS", "$ROOTSYS/bin/root"

Install MIDAS

You can see a list of other installation problems at Common problems & Debugging recipes.

Install ROOTANA

Install ROODY

Prepare VME hardware

Hardware check list:

Install Universe-II VME driver (V7648, V7750, V7805, V7851)

  • login as root (ssh root@localhost)
  • get latest version of vmic driver from ladd00: scp username@ladd00:/home/olchansk/daq/v7805/vmisft-7433-NNN-KOMMM.tar.gz .
  • tar xzvf vmisft-7433-3.5-KO2.tar.gz
  • chown -R root.root vmisft-7433-3.5-KO2
  • cd vmisft-7433-3.5-KO2
  • cd vme_universe
  • make
  • make install
  • cd ..
  • make
  • edit /etc/rc.local, add these lines:
modprobe vme_universe
mkdir -p /dev/bus/vme
mknod /dev/bus/vme/ctl c 221 8
chmod a+wr /dev/bus/vme/ctl
  • run "modprobe vme_universe", run "lsmod" to check that the vme_universe module was loaded
  • run "ls -l /dev/bus/vme/ctl" to check that the VME device file exists, it should be "crw-rw-rw- 1 root root 221, 8 Feb 17 15:47 /dev/bus/vme/ctl"
  • cd ~/packages/vme; vmescan.exe

Install Tsi-148 VME driver (V7865)

  • login as root (ssh root@localhost)
  • yum install kernel-devel
  • get latest version of the driver from ladd00: scp username@ladd00:/home/olchansk/daq/v7865/v7865-sdk-linux-R01.00-KONNN.tar.gz
  • tar xzvf v7865-sdk-linux-R01.00-KO6.tar.gz
  • chown -R root.root v7865-sdk-linux-R01.00-KO6
  • cd v7865-sdk-linux-R01.00-KO6/gefvme/module
  • make
  • make install
  • cd $HOME
  • edit /etc/rc.local, verify that it has these 2 lines:
modprobe gefvme
sh /root/gefvme-makedevs
  • run "modprobe gefvme", run "lsmod" to check that the gefvme module was loaded
  • cd ~/packages/vme; vmescan_gef.exe

Setup the experiment environment

  • decide which computer will host MIDAS (where MIDAS shared memory buffers will reside). This computer will run the mserver, mlogger and mhttpd.
  • IMPORTANT: unless running an experiment where all programs run on one machine, you will now (since August 2015) have to allow access for MIDAS programs running on remote machines - see #NOTES
  • in .cshrc put the name of this computer into the section for setting MIDAS_SERVER_HOST. Note that multiple experiments can run on the same computer by using different ports.
  • mkdir $HOME/online
  • cd $HOME/online
  • create directories for local programs, sources, elog and history: mkdir bin src elog history
  • create data directory: mkdir -p /ladd/data1/t2kvme5/data; ln -s /ladd/data1/t2kvme5/data $HOME/online
  • create the exptab file "$HOME/online/exptab" following the example below. The first entry (exptname) is the name if the DAQ system (MIDAS experiment name), the second entry (/home/USER/online) is the location of MIDAS shared memory buffers (by convention, $HOME/online), the third entry (kopio03) is your username.
exptname /home/kopio03/online kopio03
  • logout and login again for all changes to take effect

Setup experiment startup scripts

  • login to the experiment host computer
  • echo $MIDAS_SERVER_HOST ### to check correct value - should be blank
  • - see #NOTES
  • - see #NOTES
    • OR optionally restrict access to specified hosts using the -a argument
  • create $HOME/online/bin/start_daq.sh, replacing XXX with the hostname of the machine running the experiment (and changing the mserver and mhttpd ports, as needed).
#!/bin/sh
cd $HOME/online
#
case `hostname` in XXX*)
   echo "Good, we are on XXX!"
   ;;
*)
   echo "The start_daq script should be executed on XXX"
   exit 1
   ;;
esac
#
odbedit -c clean

# see #NOTES
# start mhttpd on default port. (https version). Use argument --mg to use a different port)
mhttpd  -D  
# start mserver on default port (use argument -p to use a different port) 
mserver -D 


# OR (see #NOTES)
# mhttpd  -p 8081 -D -a localhost -a XXX.triumf.ca
# mserver -p 7071 -D -a localhost -a lxdragon01.triumf.ca -a lxdragon02.triumf.ca -a XXX.triumf.ca

#
mlogger -D
#end file

Setup experiment database (ODB)

  • Start mhttpd once on command line.
 You will get an error saying you need to create a mongoose password file with command htdigest; see mhttpd for details. 
  • run $HOME/online/bin/start_daq.sh
  • open the midas status page at either (see #NOTES)
  • DON'T DO THIS YET! run ./fevme.exe (on the computer with the VME interface, could be different from computer hosting the experiment), observe that corresponding equipments have been created
  • odbedit, run these commands: (replace user names and directory names)
set "/Logger/Message file" "/home/kopio03/online/midas.log"
set "/Logger/Data Dir"     "/home/kopio03/online/data"
create STRING "/Logger/History dir"
set "/Logger/History dir"  "/home/kopio03/online/history"
create STRING "/Logger/Elog dir"
set "/Logger/Elog dir"     "/home/kopio03/online/elog"
exit
  • odbedit, run these commands: (replace user names and directory names)
set "/Logger/ODB dump file" "/home/kopio03/online/history/run%05d.xml"
set "/Logger/ODB dump" "y"
set "/Logger/Channels/0/Settings/Filename" "run%05dsub%03d.mid.gz"
set "/Logger/Channels/0/Settings/Subrun byte limit" "1000000000"
set "/Logger/Channels/0/Settings/Compression"   1
set "/Logger/Channels/0/Settings/ODB Dump" "y"
set "/Programs/Logger/Required" y
set "/Programs/Logger/Start command" "mlogger -D"
set "/Programs/fevme/Required" "y"
set "/Programs/fevme/Start command" "ssh -n lxdaq09 $HOME/online/src/fevme_gef.exe -O"
exit
  • open web browser e.g. firefox. Point to either ( see #NOTES)
  • save the url bookmark to the "personal toolbar"
  • go to the Programs page, stop mlogger, stop fevme, start mlogger, start fevme
  • go to the Status page, start run, stop run
  • go back to the Status page, everything should be green
  • start a run
  • send signals to the ADC gate
  • you should be getting events
  • to look at data, proceed with setting up the <a href="../../../SR/rootana">ROOT analyzer</a>

Start DAQ programs at boot time

  • add this to /etc/rc.local (replace username and location of the start_daq script)
su - alpha -c /home/alpha/online/bin/start_daq.sh

Setup local software version control

Version control for experiment source code is setup using "git" (http://git-scm.com/)

  • cd $HOME/online
  • git init
  • git add exptab
  • git add bin/start_daq.sh
  • git add .gitignore ### contents can be
*~
*.o
*.exe
  • git add src/Makefile src/*.cxx ...
  • git commit -a

Adjust MIDAS buffer sizes

Default MIDAS SYSTEM buffer size is 8 Mbytes, fairly small for high-data-rate experiments. The rule of thumb is to have at least a few seconds worth of buffer space available. For example, if event size is 10 Kbytes and the event rate is 1 kHz, data rate is 10*10^3*1*10^3 = 10 Mbytes/sec. To buffer 10 seconds of data we need 100 Mbytes of buffer space.

To resize the MIDAS event buffers (SYSTEM, etc) do this:

  • stop all frontends, stop mlogger
  • start odbedit:
    • cd "/Experiment/Buffer sizes"
    • set SYSTEM 100000000
  • run "mdump -z SYSTEM"
  • if mdump complains about the size of .SYSTEM.SHM, remove it, try again.
  • ls -l /dev/shm ### to observe that the size of shared memory is correct

Secure MIDAS and ELOG Web access

In versions prior to May 2015, the default web access to MIDAS and ELOG uses the "http:" protocol which is insecure. In this case, all information is transmitted as clear text meaning that secret, confidential and sensitive information (such as the MIDAS and ELOG passwords and usernames) can be stolen "easily". This means that even "password protected" MIDAS and ELOG pages are not really protected if accessed using the "http" method.

Better security for HTTP is gained by using a password protected SSL (https) proxy. (It does not provide absolute security because of remaining problems with the security of SSL certificates, security of passwords, etc). Setting up an SSL (https) proxy is described below.

Since May 2015, an alternative secure option to setting up an HTTP proxy is available to users of MIDAS. Recent versions of elogd (ELOG) do support SSL https:// connections, and #mhttpd with HTTPS/SSL server (Mongoose) is now available. This option provides a similar level of security to an HTTP proxy.

Which security option to choose?

For a small operation that does not require root access to the host computer, mhttpd with mongoose is a good light weight solution. It is ideal for small experiments that until now have been using the "http:" protocol for mhttpd.

For a more elaborate setup with private networks etc., APACHE https proxy is probably better. For big experiments, resources like webcams,ganglia, couchdb, etc. also need password protection and APACHE https can provide all this.

Which one is easier to setup?

  • mhttpd + mongoose is fairly simple - you have to create a password file and (optional) a properly signed https certificate.
  • apache httpd is fairly straightforward if you follow well written instructions (such as we provide for using it with midas), but you do need root access and you do have to edit a good number of config files.


mhttpd with HTTPS/SSL server (Mongoose)

Since May 2015 the MIDAS web server mhttpd is explicitly linked with OpenSSL to provide secure HTTPS connections via the Mongoose web server (see mhttpd). With this version, default web access to MIDAS uses the "https" protocol. Web access to mhttpd can be restricted by using the Access Control List. The first time mhttpd is run, a default SSL certificate is used, and a password file must be created. For additional security, the user should create and use their own SSL certificate. See HTTPS/SSL server (Mongoose) for instructions.


mhttpd using an HTTPS/SSL proxy

An HTTP proxy must be set up. This is the only way of securing older version of mhttpd (pre August 2015). Older versions of mhttpd are started using the -p port option e.g.

  • mhttpd -D -p 8080

To run a new version of mhttpd using an HTTP proxy, use the options provided to run the old (non-Mongoose) webserver on a given port, i.e.

  • mhttpd --oldserver 8080 --nomg -D

When using an SSL proxy, only access from the SSL proxy (and maybe some special trusted machines) should be permitted. This is done using the "-a hostname" switch of mhttpd. Normally there will be only "-a localhost" switch, enabling access only for the local machine (where the SSL proxy is running). Additional "-a hostname" switches enable access from listed local machines. No "-a xxx" enables access from everywhere (defeating the purpose of the SSL proxy, unless access controls are enforced elsewhere, i.e. by a site firewall or by local firewall rules).

Setting up an HTTP proxy

In this example, we use APACHE HTTPD to password-protect a typical midas/mhttpd and elog installation.

In this configuration, one uses the Linux stock httpd that accepts encrypted https:// connections and forwards them to mhttpd and elogd. Instead of (or in addition to) using mhttpd and elogd passwords, one configures password protection in httpd via the regular apache httpd password mechanisms (htpasswd, etc).

Recent versions of elogd do support SSL https:// connections, but if one is running an SSL proxy for anyway, it is simpler to run both through the same SSL proxy using the same SSL host certificate and the same httpd password file.

Restricting http: access to elogd

Note
Recent versions of elogd do support SSL https:// connections. The following information is for those using an HTTP proxy (see above).

For elogd, this is done using the "-n localhost" switch with enables only access from the same machine if present, or access from anywhere is absent (defeating the purpose of the SSL proxy, unless access controls are enforced elsewhere).

(It is recommended to run elogd from the same user as the main daq user and to keep elogd.cfg and all logbooks in the home directory of this user, where they are captured by the normal site backup system)

Instructions for installing elogd

    • install the elog rpm (from https://midas.psi.ch/elog/download/RPMS)
    • cd $HOME/elog
    • cp -rpv /usr/local/elog/* .
    • (to import elogs from mhttpd elog: cd logbooks; ln -s /home/t2km11/online/elog midas; cd midas; /usr/local/bin/elconv)
    • edit start_elogd to read:
#!/bin/sh

killall elogd
sleep 1
killall -KILL elogd
sleep 1
/usr/local/sbin/elogd -n localhost -x -c $HOME/elog/elogd.cfg -p 8082

#end
    • edit elogd.cfg to read:
[global]
port = 8082
usr = t2km11
grp = t2km11
SMTP host = smtp.triumf.ca
URL = https://xxx/elog/
#URL = http://xxx:8082

Reverse sort = 1
Display Mode = full

List Menu commands = New, Find, Admin, Help
Menu commands = New, Edit, Reply, Find, Duplicate, Help

Entries Per Page = 30
Supress Email on edit = 1
Default encoding = 1
Page title = T2K M11 ELOG
Resolve host names = 1

Logfile = /home/t2km11/elog/elogd.log
#Logging level = 3

[midas]

List page Title = T2K M11 MIDAS ELOG
Comment = T2K M11 MIDAS ELOG
Page Title = T2K M11 MIDAS ELOG
RSS Title = [$logbook - $type - $system] $subject, posted by $author

Attributes = Author, Subject, Run, Type, System
Show Attributes Edit = Run, Author, Subject, Type, System
Required Attributes = Author, Type, System, Subject

Options Type = Routine, Reply, Shift Summary, Modification, Question, Info, Problem
Options System = General, DAQ, Beamline

Preset Run = $shell(MIDASSYS=. /home/t2km11/packages/midas/linux/bin/odbedit -d Runinfo -c 'ls -v \"run number\"')

Preset On Reply Type = Reply
Preset On Reply Run = $shell(MIDASSYS=. /home/t2km11/packages/midas/linux/bin/odbedit -d Runinfo -c 'ls -v \"run number\"')

List Display = Date, Subject, Type, System, Author, ID
Quick Filter = Date, Type, ID

Remove on reply = Author
Quote on reply = 1

Use lock = 1

************* Email Functionality ****************

Use Email Subject = [T2KM11 - $System] $Subject
Omit Email To = 1

Email System General = xxx

Install SSL proxy

  • login as root to the SSL Proxy machine
  • cd ~root
  • create a certificate request (replace ladd09 with your hostname): openssl req -new -nodes -newkey rsa:2048 -sha256 -out ladd09.csr -keyout ladd09.key (answer: CA, BC, Vancouver, TRIUMF, DAQ, ladd09.triumf.ca, email@email.com
  • (optionally) sign it by TRIUMF: Mail -s "Andrew, please sign and return to email@email.com" andrew@email.com < ladd09.csr
  • sign it yourself: openssl x509 -req -days 365 -sha256 -in ladd09.csr -signkey ladd09.key -out ladd09.crt
  • if the certificate expires, renew it by signing it again
  • explore the private key: openssl pkey -in ladd09.key -text -noout
  • explore the certificate request: openssl req -in ladd00.csr -text -noout
  • explore the certificate: openssl x509 -in ladd09.crt -noout -text
  • yum install mod_ssl
  • edit /etc/httpd/conf.d/ssl.conf, make it read (add at the very bottom, right before the "</VirtualHost>" entry at the end of the file. This assumes mhttpd is running on port 8081, elogd is running on port 8082, both on localhost.
...
SSLCertificateFile /root/ladd09.crt
SSLCertificateKeyFile /root/ladd09.key
ProxyPass /elog/ http://localhost:8082/ retry=1
ProxyPass /      http://localhost:8081/ retry=1
</VirtualHost>
  • comment out duplicate "SSLCertificateFile" and "SSLCertificateKeyFile" elsewhere in the file
  • (optionally) If you got a certificate that is signed by DigiCert or RapidSSL then you'll need to add a line specifying the certificate chain file to ssl.conf:
...
SSLCertificateChainFile /root/certificate/DigiCertCA.crt
...
  • add password protection: again, right before "</VirtualHost>" at the end of the file, add this:
...
<Location />
SSLRequireSSL
AuthType Basic
AuthName "password protected site"
Require valid-user

# create password file: touch /etc/httpd/htpasswd
# to add new user or change password: htpasswd /etc/httpd/htpasswd username
AuthUserFile /etc/httpd/htpasswd

</Location>
</VirtualHost>
  • chkconfig httpd on
  • service httpd restart
  • test it
  • in ODB, set "/Elog/URL" to "https://host/elog/"
  • now from the midas status page, the "Elog" button should take us to the https Elog URL

Setup the history mhttpd for faster access to history plots

When running an SSL proxy,

  • start the main mhttpd (orange command for old mhttpd, green for new mhttpd with Mongoose(post August2015):
    • "mhttpd -p 8071 -D" or
    • "mhttpd -D --oldserver 8071 --nomg"
  • start the history mhttpd
    • "mhttpd -p 8072 -D -H" or
    • "mhttpd -D -H --oldserver 8072 --nomg"
  • set ODB /History/URL to "http://alphacpc09.cern.ch:8072/HS/"
  • open the MIDAS status page
  • go to the history section, try to open any history plot, observe that the history plot gif image loads correctly, inspect it's URL (use "copy image URL" or "view source", etc), it should point to port 8072 causing connection to the history mhttpd.
  • continue with these instructions to setup history mhttpd access through an SSL proxy:
  • setup SSL proxy access (required mod_proxy_html)
    • login as root to the SSL proxy machine
    • on SL5, install the missing mod_proxy_html httpd module:
    • yum install httpd-devel libxml2-devel
    • wget http://apache.webthing.com/mod_proxy_html/mod_proxy_html.tar.bz2
    • tar xjvf mod_proxy_html.tar.bz2
    • cd mod_proxy_html
    • apxs -c -I. -I/usr/include/libxml2 -i mod_proxy_html.c
    • apxs -c -I. -I/usr/include/libxml2 -i mod_xml2enc.c
    • cd /etc/httpd/conf.d, add this to ssl.conf:
before the ProxyPass statements
# proxy the MIDAS web servers  
LoadModule  xml2enc_module       modules/mod_xml2enc.so  
LoadModule  proxy_html_module    modules/mod_proxy_html.so  
ProxyHTMLLinks  a               href  
ProxyHTMLLinks  link            href  
ProxyHTMLLinks  img             src  
#ProxyHTMLEnable On  
ProxyRequests off  
after the ProxyPass statements
# ALPHA1 history access 
ProxyPass /alpha1/history/   http://alphacpc09.cern.ch:8072/HS/  retry=1 
ProxyPass /alpha1/           http://alphacpc09.cern.ch:8071/     retry=1 
 
ProxyHTMLEnable On  
ProxyHTMLURLMap http://alphacpc09.cern.ch:8072/HS/ /alpha1/history/ 
adjust
  • "alpha1" is the experiment name
  • "alphacpc09.cern.ch" is the machine running mhttpd
  • "8071" is the port number of the main mhttpd
    • "mhttpd -p 8071 -D" or
    • "mhttpd -D --oldserver 8071 --nomg"
  • "8072" is the port number of the history mhttpd
    • "mhttpd -p 8072 -D -H" or
    • "mhttpd -D -H --oldserver 8072 --nomg"