MidasWiki - User contributions [en]

Setup MIDAS experiment at TRIUMF

2026-03-12T20:50:34Z

Dfujimoto: /* Setup experiment database (ODB) */

{{Pagelinks}}
== Introduction ==
This page describes setting up a MIDAS experiment at TRIUMF. This information can be adapted for other sites.

== Environment variables ==
* '''MIDASSYS''' Base directory of the MIDAS package, midas and mxml should be at the same level.
* '''MIDAS_EXPTAB''' Experiment definition file
* '''MIDAS_SERVER_HOST''' MIDAS host server name for remote midas connections.
* '''MIDAS_EXPT_NAME''' Experiment name

== Standard layout of MIDAS experiment ==
The following shows the directory layout of a standard MIDAS experiment:

/home/exptuser/
packages/
root <---- ROOT
mxml
mscb
midas/ <---- MIDAS
linux/{lib,bin} <---- binaries matching the selected 64-bit/32-bit flavour of ROOT
linux-m32/{lib,bin} <---- limited function 32-bit binaries for 32-bit frontend machines, build by "make linux32"
linux-m64/{lib,bin} <---- limited function 64-bit binaries (only needed if ROOT and linux/bin are 32-bit)
linux-arm/{lib,bin} <---- full function ARM cross-compiled using "make linuxarm"
linux-crosscompile/{lib,bin} <---- cross-compiled limited function binaries for PPC and ARM frontends (see Makefile)
rootana <---- ROOT analyzer
roody <---- graphical online histogram viewer for MIDAS and ROOTANA
online/
exptab <---- experiment definition
{.ODB,.SYSTEM,.SYSMSG,etc}.SHM <---- MIDAS shared memory save files
src <---- experiment frontend sources
bin,scripts
elog <---- MIDAS elog
history <---- MIDAS history
data -> /data/exptname/current <---- symlink to the data directory
/data/exptname/current <---- experiment data directory with ODB save files and MIDAS .mid/.mid.gz data files

== Prepare computers ==

On some operating systems, several MIDAS functions require administrator access:

* on el7 linux (SL7/CC7/CentOS7/RHEL7) - access to mhttpd port 8443 requires special firewall rules, see here: https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29
* on el7 linux - access to mserver to run frontends and other programs on some other computer requires special firewall rules, see here: https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29
* on el7 linux - on the frontend machines (and other machines that will connect to the mserver, the same firewall rule needs to be created (use the IP address of the machine running the mserver)

== Prepare the user account ==
<div id="NOTES"></div>

* Setup the user account for running this instance of midas. For machines part of the LADD cluster, follow these [http://daq-plone.triumf.ca/SM/docs/local/NewLaddUser] instructions.
* check that the account is using the /bin/bash shell
* make $HOME/.profile look like this:

<pre>
#!/bin/echo You must source

export SVN_EDITOR="emacs -nw"
export GIT_EDITOR="emacs -nw"
export MIDASSYS=$HOME/packages/midas
export ROOTANASYS=$HOME/packages/rootana
export MIDAS_EXPTAB=$HOME/online/exptab
#
# setup the MIDAS mserver
#
case `hostname` in
daq07*)
unset MIDAS_SERVER_HOST
;;
*)
export MIDAS_SERVER_HOST=daq07.triumf.ca:7070
;;
esac
#
# select 64-bit or 32-bit MIDAS and ROOT
#
case `uname -i` in
i386)
source /daq/daqshare/olchansk/root/root_v5.34.01_SL62_32/bin/thisroot.sh
export PATH=.:$MIDASSYS/linux-m32/bin:$PATH
;;
*)
#source /daq/daqshare/olchansk/root/root_v5.34.34_SL67_64/bin/thisroot.sh
source $HOME/packages/root/bin/thisroot.sh
export PATH=.:$MIDASSYS/linux/bin:$PATH
;;
esac
#
export PATH=.:$HOME/online/bin:$HOME/packages/roody/bin:$PATH
#
#end
</pre>

* mkdir $HOME/packages
* Logout and login again, for .cshrc changes to take effect

== Install ROOT ==

* Identify the Linux version: RH9 (Red Hat Linux 9), FC3 (Fedora Core 3), RHEL4/SL4 (Red Hat Enterprise LInux 4/Scientific Linux 4), SL5x, SL6x, (CentOS/CC/SL) el7x: more /etc/redhat-release
* Decide to use 32-bit or 64-bit ROOT ('uname -a')
* cd $HOME/packages
* ls -l /daq/daqshare/olchansk/root/ ### to see all available ROOT packages
* ln -s /daq/daqshare/olchansk/root/root_vNNN_VVV_BB root, where NNN is the latest available version of ROOT ("ls -l /daq/daqshare/olchansk/root"), VVV is the Linux version code (RH9, FC3, SL4, etc) and BB is "_32" or "_64" for 32-bit or 64-bit ROOT. For example: /daq/daqshare/olchansk/root/root_v5.10.00_SL40
* for example: ln -s /daq/daqshare/olchansk/root/root_v5.34.34_el72_64 $HOME/packages/root
* Check that ROOT works: "source $HOME/packages/root/bin/thisroot.sh; root". Type ".q" to exit root.

== Install MIDAS ==

* cd $HOME/packages
* (OBSOLETE) svn co svn+ssh://svn@savannah.psi.ch/repos/meg/midas/trunk midas, password "svn". (password has to be entered twice)
* (OBSOLETE) svn co svn+ssh://svn@savannah.psi.ch/repos/meg/mxml/trunk mxml
* git clone https://bitbucket.org/tmidas/midas --recursive
* (OBSOLETE) git clone https://bitbucket.org/tmidas/mxml
* (OBSOLETE) git clone https://bitbucket.org/tmidas/mscb
* (IF BITBUCKET IS DOWN) git clone -v --progress https://daq.triumf.ca/~daqweb/git/mxml.git
* (IF BITBUCKET IS DOWN) git clone -v --progress https://daq.triumf.ca/~daqweb/git/mscb.git
* (IF BITBUCKET IS DOWN) git clone -v --progress https://daq.triumf.ca/~daqweb/git/midas.git
* cd midas
* make
* (only if needed) make linux32 ### build the 32-bit MIDAS libraries
* ls -l linux/bin/odbedit ### check that odbedit has been created (do not run it yet)

You can see a list of other installation problems at [[Common problems & Debugging recipes]].

;NOTE 1
: Optional features in MIDAS can be explicitly disabled if desired when making MIDAS using the NO_xxx feature (NO_ROOT,NO_MYSQL,NO_ODBC,NO_SQLITE,NO_MSCB), e.g. "make NO_ROOT=1" to disable ROOT. These NO_xxx Makefile variables are only used to control autodetection.

;NOTE 2
: Since June 2019 the mxml and mscb packages are submodules of the midas package, so no need to clone them separately. If you have an existing clone of midas but not yet the submodules, you need

$ git submodule update --init --recursive

: To update both midas and the submodules, you need

$ git pull --recurse-submodules

== Install ROOTANA ==

* cd $HOME/packages
* git clone https://bitbucket.org/tmidas/rootana
* cd rootana
* make

== Install ROODY ==

* cd $HOME/packages
* git clone https://bitbucket.org/tmidas/roody
* cd roody
* make
* $HOME/packages/roody/bin/roody, run the program

== Install additional additional packages ==

* cd $HOME/packages
* git clone https://bitbucket.org/ttriumfdaq/vme
* git clone https://bitbucket.org/ttriumfdaq/frontends

== Build special versions of MIDAS ==

Build special versions of MIDAS for the case when some MIDAS programs, such as VME frontends, will run on a different computer that may have a different flavour of operating system, i.e. 32-bit linux or an older version of Scientific Linux.

* login to the computer where the frontends will run and:
* if it is a 32-bit linux: cd $HOME/packages/midas; make linux32
* if it is a 64-bit linux: cd $HOME/packages/midas; make linux64

* login to the host computer to cross-compile ARM code:
* if it is an ARM linux: cd $HOME/packages/midas; make linuxarm # may need to install ARM cross compilers

== Prepare VME hardware ==

Hardware check list:
* VME crate
* VME processor (supported are V77xx, V7805, V7865)
* On all VME modules, set the VME address jumpers as described here: http://daq-plone.triumf.ca/SM/docs/local/vme_jumpers
* run vmescan to confirm correct VME addresses
** cd $HOME/packages
** svn checkout https://ladd00.triumf.ca/svn/daqsvn/trunk/vme
** cd vme
** make
** ./vmescan.exe (or _gef.exe, depending on the VME driver in use)

== Install Universe-II VME driver (V7648, V7750, V7805, V7851) ==

* login as root (ssh root@localhost)
* follow instructions: https://www.triumf.info/wiki/DAQwiki/index.php/VME-CPU#V7648.2C_V7750.2C_V7805.2C_V7851_:_Setup_vme_universe_VME_drivers
* cd ~/packages/vme; vmescan.exe

== Install Tsi-148 VME driver (V7865) ==

* login as root (ssh root@localhost)
* follow instructions: https://www.triumf.info/wiki/DAQwiki/index.php/VME-CPU#V7865_and_XVB-602_:_Setup_gefvme.2Ftsi148_VME_drivers
* cd ~/packages/vme; vmescan_gef.exe

== Setup the experiment environment ==

* Decide which computer will host MIDAS (where MIDAS shared memory buffers will reside).
: This computer will run the [[mserver]], [[mlogger]] and [[mhttpd]] applications. (It is usually the machine where the MIDAS,ROOT etc. packages have been downloaded). It will be referred to as the host machine (localhost).

The environment is slightly different depending on whether all programs run on the host machine, or whether some programs run on remote host(s) :
=== ALL programs run on localhost ===
:If all programs run on the host machine (localhost), it is not necessary to run [[mserver]]. [[Environment Variables#MIDAS_SERVER_HOST|MIDAS_SERVER_HOST]] will not be assigned (see example .cshrc [[#Prepare the user account|above]]).

=== Some programs run on REMOTE host(s) ===
: '''IMPORTANT:'''
# Since August 2015 '''you must explicitly allow access for clients running on remote machines'''. To do this, follow the '''[[Security#MIDAS programs on remote machines|instructions here]]'''.
# The example code .cshrc ([[#Prepare the user account|see above]]) should be present on both host and remote machine(s). This will ensure that [[Environment Variables#MIDAS_SERVER_HOST|MIDAS_SERVER_HOST]] will NOT be set for the host machine (localhost), but on a remote machine, MIDAS_SERVER_HOST will be set to the MIDAS host machine.
# The client [[mserver]] must be started on the MIDAS host machine. Note that multiple experiments can run on the same host machine by starting several instances of [[mserver]] (one for each experiment) running with different ports (and .cshrc would be edited so that MIDAS_SERVER_HOST is set to the appropriate port for the experiment).

 
On the host machine:
* mkdir $HOME/online
* cd $HOME/online
* create directories for local programs, sources, elog and history: mkdir bin src elog history
* create data directory: mkdir -p /ladd/data1/t2kvme5/data; ln -s /ladd/data1/t2kvme5/data $HOME/online
* create the exptab file "$HOME/online/exptab" following the example below. The first entry (exptname) is the name if the DAQ system (MIDAS experiment name), the second entry (/home/USER/online) is the location of MIDAS shared memory buffers (by convention, $HOME/online), the third entry (kopio03) is your username.
<pre>exptname /home/kopio03/online kopio03</pre>
* logout and login again for all changes to take effect

== Setup experiment startup scripts ==
* login to the experiment host computer
* echo $MIDAS_SERVER_HOST ### to check correct value - should be blank
* create $HOME/online/bin/start_daq.sh, replacing XXX with the hostname of the machine running the experiment (and changing the mserver and mhttpd ports, as needed).

#!/bin/sh
# start_daq.sh
cd $HOME/online
#
case `hostname` in XXX*)
echo "Good, we are on XXX!"
;;
*)
echo "The start_daq script should be executed on XXX"
exit 1
;;
esac
#
odbedit -c clean
# start [[mhttpd]] on default port. (Mongoose https version - see [[mhttpd]] for other options)
mhttpd -D -a localhost -a XXX.triumf.ca # optionally restrict access to specified hosts
#
# start [[mserver]] on default port (use argument -p to use a different port)
mserver -D # access must now be specifically allowed - see [[#Setup the experiment environment|above]]

# OR ([[#NOTES|older MIDAS versions]])
# mhttpd -p 8081 -D -a localhost -a XXX.triumf.ca # optionally restrict access to specified hosts
# mserver -p 7071 -D -a localhost -a lxdragon01.triumf.ca -a lxdragon02.triumf.ca -a XXX.triumf.ca # optionally restrict access to specified hosts

#
mlogger -D
#end file

== Run the MIDAS Web Server ==
Let's start the MIDAS webserver for the first time:

Start [[mhttpd]] on the ''experiment host'' (localhost) like this:
[mhostpc] mhttpd
You will get the following messages:
[mhttpd,INFO] ODB subtree /Runinfo corrected successfully
Mongoose web server will listen on ports "8080r,8443s"
[mhttpd,ERROR] [mhttpd.cxx:17892:mongoose,ERROR] cannot find SSL certificate file "/home/agdaq/online/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17893:mongoose,ERROR] please create SSL certificate file: openssl req -new -nodes -newkey rsa:2048 -sha256 -out ssl_cert.csr -keyout ssl_cert.key; openssl x509 -req -days 365 -sha256 -in ssl_cert.csr -signkey ssl_cert.key -out ssl_cert.pem; cat ssl_cert.key >> ssl_cert.pem
could not start the mongoose web server, see messages and midas.log, bye!

Create a self-signed certificate suitable for initial testing by executing the command printed by mhttpd:


[mhostpc] openssl req -new -nodes -newkey rsa:2048 -sha256 -out ssl_cert.csr -keyout ssl_cert.key; openssl x509 -req -days 365 -sha256 -in ssl_cert.csr -signkey ssl_cert.key -out ssl_cert.pem; cat ssl_cert.key >> ssl_cert.pem

For production use, you should create a properly signed certificate, see [[Mhttpd#Create an SSL certificate|create your own SSL certificate]] or you should run mhttpd behind an SSL proxy.

Run mhttpd again.

You will get the following messages:
[mhttpd,INFO] ODB subtree /Runinfo corrected successfully
Mongoose web server will listen on ports "8080r,8443s"
Mongoose web server will use SSL certificate file "/home/johnfoo/packages/midas/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17633:mongoose,ERROR] mongoose web server cannot find password file "/home/johnfoo/online/htpasswd.txt"
[mhttpd,ERROR] [mhttpd.cxx:17634:mongoose,ERROR] please create password file: htdigest -c /home/johnfoo/online/htpasswd.txt Default midas
could not start the mongoose web server, see messages and midas.log, bye!

Create the password file by following the instructions printed by mhttpd. The http digest domain name is the experiment name, suggested default user name is "midas". You will be asked to type in a password


[mhostpc] htdigest -c /home/johnfoo/online/htpasswd.txt exptname midas
Adding password for midas in realm exptname.
New password:
Re-type new password:

It is a good idea to set the password file {{Filepath|path=htpasswd.txt}} readable and writable by owner only.

Now restart {{Utility|name=mhttpd}}
[mhostpc] mhttpd
Mongoose web server will listen on ports "8080r,8443s" **see note
Mongoose web server will use SSL certificate file "/home/suz/packages/midas/ssl_cert.pem"
Mongoose web server will use authentication realm "Default", password file "./htpasswd.txt"

Now point a web browser running on the same host computer (localhost) to https://localhost:8443
If the web browser is running on a different computer, go to URL of the form

https://mhostpc.triumf.ca:8443 (substitute your host machine name and domain for "mhostpc.triumf.ca")

If you are using the default SSL certificate you will probably get a message: "This Connection is Untrusted". Click "I understand the risks" and add an exception. This is because the test certificate is self-signed. Then confirm an exception.

If instead you get a "connection refused" error, the midas host pc may have the firewall enabled. To make a firewall exception for MIDAS, follow instructions here http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29

You should then see an authentication box asking you for the user name and password. The user name is "midas". Enter the password you just created. The Midas [[Status Page]] should appear with multiple buttons for run control as well as equipment listing (no equipments will be listed as yet) and application listings. Please refer to [[mhttpd]] (the MIDAS Web-based Run Control utility) for further information. You can start and stop runs from the main status page, and use the [[ODB Page]] to access the database (ODB).

; Note
: Default ports of 8080 and 8443 are used by [[mhttpd]]. If these ports are in use on your machine, start mhttpd with alternative ports, e.g.
[mhostpc] mhttpd --https 8448 --http 8089
: or see [[Mhttpd#Usage]] to change the default ports.

== Setup experiment database (ODB) ==

* run $HOME/online/bin/start_daq.sh

* odbedit, run these commands: (replace user names and directory names)
<pre>
# set "/Logger/Message file" "/home/kopio03/online/midas.log" # obsolete
set "/Logger/Data Dir" "/home/kopio03/online/data"
set "/Logger/History dir" "/home/kopio03/online/history"
create STRING "/Logger/Elog dir"
set "/Logger/Elog dir" "/home/kopio03/online/elog"
set "/Logger/ODB dump file" "/home/kopio03/online/history/run%05d.xml"
set "/Logger/ODB dump" "y"
set "/Logger/Channels/0/Settings/Filename" "run%05dsub%03d.mid.gz"
set "/Logger/Channels/0/Settings/Subrun byte limit" "1000000000"
set "/Logger/Channels/0/Settings/Compression" 1
set "/Logger/Channels/0/Settings/ODB Dump" "y"
set "/Programs/Logger/Required" y
set "/Programs/Logger/Start command" "mlogger -D"
set "/Programs/fevme/Required" "y"
set "/Programs/fevme/Start command" "ssh -n lxdaq09 $HOME/online/src/fevme_gef.exe -O"
exit
</pre>
* open web browser e.g. firefox.
* go to the midas status page at https://localhost:8443 (default port).
** if running [[mhttpd]] with Mongoose HTTPS/OpenSSL (the default) for the first time, you will need to create a password file. Follow the instructions (see [[mhttpd#HTTPS/SSL server (Mongoose)]] for details).
** For other options (i.e. HTTPS/SSL proxy) see [[#Secure MIDAS and ELOG Web access]]
*OR open the midas status page at http://localhost:8081 ([[#NOTES|older MIDAS versions]])
* midas status page will show most stuff "red" as nothing is running yet
* DON'T DO THIS YET run ./fevme.exe (on the computer with the VME interface, could be different from computer hosting the experiment), observe that corresponding equipments have been created
* save the url bookmark to the "personal toolbar"
* go to the Programs page, stop mlogger, stop fevme, start mlogger, start fevme
* go to the Status page, start run, stop run
* go back to the Status page, everything should be green
* start a run
* send signals to the ADC gate
* you should be getting events
* to look at data, proceed with setting up the [[ROOTANA|ROOT Analyzer]].

== Start DAQ programs at boot time ==

* add this to /etc/rc.local (replace username and location of the start_daq script)
<pre>
su - alpha -c /home/alpha/online/bin/start_daq.sh
</pre>

== Setup local software version control ==

Version control for experiment source code is setup using "git" (http://git-scm.com/)

* cd $HOME/online
* git init
* git add exptab
* git add bin/start_daq.sh
* git add .gitignore ### contents can be
<pre>
*~
*.o
*.exe
</pre>
* git add src/Makefile src/*.cxx ...
* git commit -a

== Adjust MIDAS buffer sizes ==

Default MIDAS SYSTEM buffer size is 8 Mbytes, fairly small for high-data-rate experiments. The rule of thumb is to have at least a few seconds worth of buffer space available. For example, if event size is 10 Kbytes and the event rate is 1 kHz, data rate is 10*10^3*1*10^3 = 10 Mbytes/sec. To buffer 10 seconds of data we need 100 Mbytes of buffer space.

To resize the MIDAS event buffers (SYSTEM, etc) do this:
* stop all frontends, stop mlogger
* start odbedit:
** cd "/Experiment/Buffer sizes"
** set SYSTEM 100000000
* run "mdump -z SYSTEM"
* if mdump complains about the size of .SYSTEM.SHM, remove it, try again.
* ls -l /dev/shm ### to observe that the size of shared memory is correct

== Secure MIDAS and ELOG Web access ==
In versions prior to May 2015, the default web access to MIDAS and ELOG uses the "http:" protocol which is insecure. In this case, all information is transmitted as clear text meaning that secret, confidential and sensitive information (such as the MIDAS and ELOG passwords and usernames) can be stolen "easily". This means that even "password protected" MIDAS and ELOG pages are not really protected if accessed using the "http" method.

Better security for HTTP is gained by using a password protected '''SSL (https) proxy'''. (It does not provide absolute security because of remaining problems with the security of SSL certificates, security of passwords, etc). Setting up an SSL (https) proxy is described [[#Setting up an HTTP proxy|below]].

Since May 2015, an ''alternative secure option'' to setting up an HTTP proxy is available to users of MIDAS. Recent versions of elogd (ELOG) do support SSL https:// connections, and [[#mhttpd with HTTPS/SSL server (Mongoose)]] is now available. This option is the default, and provides a similar level of security to an HTTP proxy.

See [[Security#Web Access]] for a comparison of these two secure options.

=== mhttpd with HTTPS/SSL server (Mongoose) ===

Since May 2015 the MIDAS web server [[mhttpd]] is explicitly linked with OpenSSL to provide secure HTTPS connections via the [https://bitbucket.org/tmidas/midas/src/ecb9a8537448a8a43f7f9a2bfdb82e578208cde3/doc/mongoose/?at=develop Mongoose] web server (see [[mhttpd]]). With this version, default web access to MIDAS uses the "https" protocol. Web access to {{Utility|name=mhttpd}} can be restricted by using the "-a hostname" switch of [[mhttpd]]. The first time {{Utility|name=mhttpd}} is run, a password file must be created. An SSL certificate is also required. See [[mhttpd#HTTPS/SSL server (Mongoose)|HTTPS/SSL server (Mongoose)]] for instructions.

=== mhttpd using an HTTPS/SSL proxy ===

THESE INSTRUCTIONS ARE WRONG, DO NOT DO THIS. See instead [https://daq00.triumf.ca/MidasWiki/index.php/Setup_MIDAS_experiment_at_TRIUMF#Install_https_proxy this section].

An [[#Setting up an HTTP proxy|HTTP proxy]] must be set up. This is the only way of securing older version of [[mhttpd]] (pre August 2015). Older versions of mhttpd are started using the -p port option e.g.
* mhttpd -D -p 8080 

To run a new version of mhttpd using an HTTP proxy, use the options provided to run the old (non-Mongoose) webserver on a given port, i.e.
* mhttpd --oldserver 8080 --nomg -D 

When using an SSL proxy, only access from the SSL proxy (and maybe some special trusted machines) should be permitted.
This is done using the "-a hostname" switch of [[mhttpd]]. Normally there will be only "-a localhost" switch, enabling access only for the local machine (where the SSL proxy is running). Additional "-a hostname" switches enable access from listed local machines. No "-a xxx" enables access from everywhere (defeating the purpose of the SSL proxy, unless access controls are enforced elsewhere, i.e. by a site firewall or by local firewall rules).

== Setting up an HTTP proxy ==
In this example, we use APACHE HTTPD to password-protect a typical midas/mhttpd and elog installation.

In this configuration, one uses the Linux stock httpd that accepts encrypted https:// connections and forwards them to mhttpd and elogd. Instead of (or in addition to) using mhttpd and elogd passwords, one configures password protection in httpd via the regular apache httpd password mechanisms (htpasswd, etc).

Recent versions of elogd do support SSL https:// connections, but if one is running an SSL proxy for anyway, it is simpler to run both through the same SSL proxy using the same SSL host certificate and the same httpd password file.

=== Restricting http: access to elogd ===
;Note
:Recent versions of elogd do support SSL https:// connections. The following information is for those using an HTTP proxy (see above).

For elogd, this is done using the "-n localhost" switch with enables only access from the same machine if present, or access from anywhere is absent (defeating the purpose of the SSL proxy, unless access controls are enforced elsewhere).

(It is recommended to run elogd from the same user as the main daq user and to keep elogd.cfg and all logbooks in the home directory of this user, where they are captured by the normal site backup system)

== Install standalone elog ==

* login into the user account that will run the elog
* cd $HOME/packages
* git clone https://bitbucket.org/ritt/elog
* cd elog
* make
* create new file start_elogd with this contents:
<pre>
#!/bin/sh

killall elogd
sleep 1
killall -KILL elogd
sleep 1
$HOME/packages/elog/elogd -n localhost -x -c $HOME/packages/elog/elogd.cfg -p 8082 -D

#end
</pre>
* chmod a+x start_elogd
* edit elogd.cfg to read:
<pre>
[global]
port = 8082
SMTP host = smtp.triumf.ca
URL = https://titan00.triumf.ca/elog/

Reverse sort = 1
Display Mode = full

#List Menu commands = New, Find, Admin, Help
#Menu commands = New, Edit, Reply, Find, Duplicate, Help

Entries Per Page = 30
Supress Email on edit = 1
Default encoding = 1
Page title = TITAN ELOG
Resolve host names = 1

Logfile = /home/titan/packages/elog/elogd.log
#Logging level = 3

[midas]

List page Title = T2K M11 MIDAS ELOG
Comment = T2K M11 MIDAS ELOG
Page Title = T2K M11 MIDAS ELOG
RSS Title = [$logbook - $type - $system] $subject, posted by $author

Attributes = Author, Subject, Run, Type, System
Show Attributes Edit = Run, Author, Subject, Type, System
Required Attributes = Author, Type, System, Subject

Options Type = Routine, Reply, Shift Summary, Modification, Question, Info, Problem
Options System = General, DAQ, Beamline

Preset Run = $shell(MIDASSYS=. /home/t2km11/packages/midas/linux/bin/odbedit -d Runinfo -c 'ls -v \"run number\"')

Preset On Reply Type = Reply
Preset On Reply Run = $shell(MIDASSYS=. /home/t2km11/packages/midas/linux/bin/odbedit -d Runinfo -c 'ls -v \"run number\"')

List Display = Date, Subject, Type, System, Author, ID
Quick Filter = Date, Type, ID

Remove on reply = Author
Quote on reply = 1

Use lock = 1

************* Email Functionality ****************

Use Email Subject = [T2KM11 - $System] $Subject
Omit Email To = 1

Email System General = xxx
</pre>
* ./start_elogd &
* firefox http://localhost:8082 # hould show the elog message index

To start elogd automatically when the machine is rebooted, login as root and
* add this text to /etc/rc.local:
<pre>
su - titan -c "/home/titan/packages/elog/start_elogd"
</pre>
* chmod a+x /etc/rc.local
* systemctl start rc-local

To import elog entries from the mhttpd elog, do this:

* cd ~/packages/elog/logbooks
* ln -s /home/t2km11/online/elog midas
* cd midas
* ~/packages/elog/elconv)

== Install https proxy ==

THESE INSTRUCTIONS ARE OBSOLETE, INSTEAD,
* GO HERE: https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Install_apache_httpd_proxy_for_midas_and_elog
* AND GO HERE: https://midas.triumf.ca/MidasWiki/index.php/Quickstart_Linux#Run_the_MIDAS_Web_Server

FOLLOWING INSTRUCTIONS ARE OBSOLETE...

* login as root to the https proxy machine
* cd ~root
* yum install mod_ssl
* yum install crypto-utils # see http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_monitoring_of_HTTPS_certificates
* create a certificate request (replace ladd09 with your hostname): openssl req -new -nodes -newkey rsa:2048 -sha256 -out ladd09.csr -keyout ladd09.key (answer: CA, BC, Vancouver, TRIUMF, DAQ, ladd09.triumf.ca, email@email.com
* sign it by TRIUMF:
** mail -s "Certificate request" yourself@email.com < ladd09.csr
** forward this request to Andrew Daviel
** he will email the signed crt file, copy it to this system as ladd09.crt
* sign it yourself: openssl x509 -req -days 365 -sha256 -in ladd09.csr -signkey ladd09.key -out ladd09.crt
* (if the certificate expires, renew it by signing it again)
* Additional commands for working with certificates:
** explore the private key: openssl pkey -in ladd09.key -text -noout
** explore the certificate request: openssl req -in ladd00.csr -text -noout
** explore the certificate: openssl x509 -in ladd09.crt -noout -text
* move certificate files to proper system locations:
* mv ladd09.key /etc/pki/tls/private/
* mv ladd09.crt /etc/pki/tls/certs/
* if selinux is enabled, do this:
** restorecon -Rv /etc/pki/tls/certs/
** restorecon -Rv /etc/pki/tls/private/
** /usr/sbin/setsebool -P httpd_can_network_connect 1
* open /etc/httpd/conf.d/ssl.conf in a text editor, go to the very bottom and right before the "</VirtualHost>" entry, add following text:
<pre>
...
SSLCertificateFile /etc/pki/tls/certs/ladd09.crt
SSLCertificateKeyFile /etc/pki/tls/private/ladd09.key

ProxyPass /elog/ http://localhost:8082/ retry=1
ProxyPass / http://localhost:8080/ retry=1

<Location />

SSLRequireSSL
AuthType Basic
AuthName "password protected site"
Require valid-user

# create password file: touch /etc/httpd/htpasswd
# to add new user or change password: htpasswd /etc/httpd/htpasswd username
AuthUserFile /etc/httpd/htpasswd

</Location>
</VirtualHost>
...
</pre>
* comment out duplicate "SSLCertificateFile" and "SSLCertificateKeyFile" elsewhere in the file
* (optionally) If you got a certificate that is signed by DigiCert or RapidSSL then you'll need to add a line specifying the certificate chain file:
<pre>
...
SSLCertificateChainFile /etc/pki/tls/certs/DigiCertCA.crt
...
</pre>
* touch /etc/httpd/htpasswd
* htpasswd /etc/httpd/htpasswd midas # enter password midas
* chkconfig httpd on
* service httpd restart
* firewall-cmd --add-port=443/tcp --permanent
* firewall-cmd --reload
* firewall-cmd --list-all
* test it
** test the SSL proxy: https://host/ should yield the midas status page, https://host/elog/ should yield the elog message index
* in ODB, set "/Elog/URL" to "https://host/elog/"
* now from the midas status page, the "Elog" button should take us to the https Elog URL

In needed, enable user directories: https://blah/~user in ~user/public_html

* edit /etc/httpd/conf.d/userdir.conf, replace "UserDir disabled" with "UserDir enabled"
* setsebool -P httpd_enable_homedirs true
* systemctl restart httpd

== Setup the history mhttpd for faster access to history plots ==
When running an SSL proxy,
* start the main mhttpd (orange command for old mhttpd, green for new mhttpd with Mongoose(post August2015):
** "mhttpd -p 8071 -D" or
** "mhttpd -D --oldserver 8071 --nomg" 

* start the history mhttpd
** "mhttpd -p 8072 -D -H" or
** "mhttpd -D -H --oldserver 8072 --nomg" 
* set ODB /History/URL to "http://alphacpc09.cern.ch:8072/HS/"
* open the MIDAS status page
* go to the history section, try to open any history plot, observe that the history plot gif image loads correctly, inspect it's URL (use "copy image URL" or "view source", etc), it should point to port 8072 causing connection to the history mhttpd.
* continue with these instructions to setup history mhttpd access through an SSL proxy:
* setup SSL proxy access (required mod_proxy_html)
** login as root to the SSL proxy machine
** on SL5, install the missing mod_proxy_html httpd module:
** yum install httpd-devel libxml2-devel
** wget http://apache.webthing.com/mod_proxy_html/mod_proxy_html.tar.bz2
** tar xjvf mod_proxy_html.tar.bz2
** cd mod_proxy_html
** apxs -c -I. -I/usr/include/libxml2 -i mod_proxy_html.c
** apxs -c -I. -I/usr/include/libxml2 -i mod_xml2enc.c
** cd /etc/httpd/conf.d, add this to ssl.conf:

;before the ProxyPass statements:
<pre>
# proxy the MIDAS web servers
LoadModule xml2enc_module modules/mod_xml2enc.so
LoadModule proxy_html_module modules/mod_proxy_html.so
ProxyHTMLLinks a href
ProxyHTMLLinks link href
ProxyHTMLLinks img src
#ProxyHTMLEnable On
ProxyRequests off
</pre>
;after the ProxyPass statements:
<pre>
# ALPHA1 history access
ProxyPass /alpha1/history/ http://alphacpc09.cern.ch:8072/HS/ retry=1
ProxyPass /alpha1/ http://alphacpc09.cern.ch:8071/ retry=1

ProxyHTMLEnable On
ProxyHTMLURLMap http://alphacpc09.cern.ch:8072/HS/ /alpha1/history/
</pre>
;adjust:
*"alpha1" is the experiment name
*"alphacpc09.cern.ch" is the machine running mhttpd
*"8071" is the port number of the main mhttpd
** "mhttpd -p 8071 -D" or
** "mhttpd -D --oldserver 8071 --nomg" 
* "8072" is the port number of the history mhttpd
** "mhttpd -p 8072 -D -H" or
** "mhttpd -D -H --oldserver 8072 --nomg" 

[[Category:Installation]] [[Category:Buffer]] [[Category:Driver]]

Setup MIDAS experiment at TRIUMF

2026-03-12T20:46:59Z

Dfujimoto: /* Setup experiment database (ODB) */

{{Pagelinks}}
== Introduction ==
This page describes setting up a MIDAS experiment at TRIUMF. This information can be adapted for other sites.

== Environment variables ==
* '''MIDASSYS''' Base directory of the MIDAS package, midas and mxml should be at the same level.
* '''MIDAS_EXPTAB''' Experiment definition file
* '''MIDAS_SERVER_HOST''' MIDAS host server name for remote midas connections.
* '''MIDAS_EXPT_NAME''' Experiment name

== Standard layout of MIDAS experiment ==
The following shows the directory layout of a standard MIDAS experiment:

/home/exptuser/
packages/
root <---- ROOT
mxml
mscb
midas/ <---- MIDAS
linux/{lib,bin} <---- binaries matching the selected 64-bit/32-bit flavour of ROOT
linux-m32/{lib,bin} <---- limited function 32-bit binaries for 32-bit frontend machines, build by "make linux32"
linux-m64/{lib,bin} <---- limited function 64-bit binaries (only needed if ROOT and linux/bin are 32-bit)
linux-arm/{lib,bin} <---- full function ARM cross-compiled using "make linuxarm"
linux-crosscompile/{lib,bin} <---- cross-compiled limited function binaries for PPC and ARM frontends (see Makefile)
rootana <---- ROOT analyzer
roody <---- graphical online histogram viewer for MIDAS and ROOTANA
online/
exptab <---- experiment definition
{.ODB,.SYSTEM,.SYSMSG,etc}.SHM <---- MIDAS shared memory save files
src <---- experiment frontend sources
bin,scripts
elog <---- MIDAS elog
history <---- MIDAS history
data -> /data/exptname/current <---- symlink to the data directory
/data/exptname/current <---- experiment data directory with ODB save files and MIDAS .mid/.mid.gz data files

== Prepare computers ==

On some operating systems, several MIDAS functions require administrator access:

* on el7 linux (SL7/CC7/CentOS7/RHEL7) - access to mhttpd port 8443 requires special firewall rules, see here: https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29
* on el7 linux - access to mserver to run frontends and other programs on some other computer requires special firewall rules, see here: https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29
* on el7 linux - on the frontend machines (and other machines that will connect to the mserver, the same firewall rule needs to be created (use the IP address of the machine running the mserver)

== Prepare the user account ==
<div id="NOTES"></div>

* Setup the user account for running this instance of midas. For machines part of the LADD cluster, follow these [http://daq-plone.triumf.ca/SM/docs/local/NewLaddUser] instructions.
* check that the account is using the /bin/bash shell
* make $HOME/.profile look like this:

<pre>
#!/bin/echo You must source

export SVN_EDITOR="emacs -nw"
export GIT_EDITOR="emacs -nw"
export MIDASSYS=$HOME/packages/midas
export ROOTANASYS=$HOME/packages/rootana
export MIDAS_EXPTAB=$HOME/online/exptab
#
# setup the MIDAS mserver
#
case `hostname` in
daq07*)
unset MIDAS_SERVER_HOST
;;
*)
export MIDAS_SERVER_HOST=daq07.triumf.ca:7070
;;
esac
#
# select 64-bit or 32-bit MIDAS and ROOT
#
case `uname -i` in
i386)
source /daq/daqshare/olchansk/root/root_v5.34.01_SL62_32/bin/thisroot.sh
export PATH=.:$MIDASSYS/linux-m32/bin:$PATH
;;
*)
#source /daq/daqshare/olchansk/root/root_v5.34.34_SL67_64/bin/thisroot.sh
source $HOME/packages/root/bin/thisroot.sh
export PATH=.:$MIDASSYS/linux/bin:$PATH
;;
esac
#
export PATH=.:$HOME/online/bin:$HOME/packages/roody/bin:$PATH
#
#end
</pre>

* mkdir $HOME/packages
* Logout and login again, for .cshrc changes to take effect

== Install ROOT ==

* Identify the Linux version: RH9 (Red Hat Linux 9), FC3 (Fedora Core 3), RHEL4/SL4 (Red Hat Enterprise LInux 4/Scientific Linux 4), SL5x, SL6x, (CentOS/CC/SL) el7x: more /etc/redhat-release
* Decide to use 32-bit or 64-bit ROOT ('uname -a')
* cd $HOME/packages
* ls -l /daq/daqshare/olchansk/root/ ### to see all available ROOT packages
* ln -s /daq/daqshare/olchansk/root/root_vNNN_VVV_BB root, where NNN is the latest available version of ROOT ("ls -l /daq/daqshare/olchansk/root"), VVV is the Linux version code (RH9, FC3, SL4, etc) and BB is "_32" or "_64" for 32-bit or 64-bit ROOT. For example: /daq/daqshare/olchansk/root/root_v5.10.00_SL40
* for example: ln -s /daq/daqshare/olchansk/root/root_v5.34.34_el72_64 $HOME/packages/root
* Check that ROOT works: "source $HOME/packages/root/bin/thisroot.sh; root". Type ".q" to exit root.

== Install MIDAS ==

* cd $HOME/packages
* (OBSOLETE) svn co svn+ssh://svn@savannah.psi.ch/repos/meg/midas/trunk midas, password "svn". (password has to be entered twice)
* (OBSOLETE) svn co svn+ssh://svn@savannah.psi.ch/repos/meg/mxml/trunk mxml
* git clone https://bitbucket.org/tmidas/midas --recursive
* (OBSOLETE) git clone https://bitbucket.org/tmidas/mxml
* (OBSOLETE) git clone https://bitbucket.org/tmidas/mscb
* (IF BITBUCKET IS DOWN) git clone -v --progress https://daq.triumf.ca/~daqweb/git/mxml.git
* (IF BITBUCKET IS DOWN) git clone -v --progress https://daq.triumf.ca/~daqweb/git/mscb.git
* (IF BITBUCKET IS DOWN) git clone -v --progress https://daq.triumf.ca/~daqweb/git/midas.git
* cd midas
* make
* (only if needed) make linux32 ### build the 32-bit MIDAS libraries
* ls -l linux/bin/odbedit ### check that odbedit has been created (do not run it yet)

You can see a list of other installation problems at [[Common problems & Debugging recipes]].

;NOTE 1
: Optional features in MIDAS can be explicitly disabled if desired when making MIDAS using the NO_xxx feature (NO_ROOT,NO_MYSQL,NO_ODBC,NO_SQLITE,NO_MSCB), e.g. "make NO_ROOT=1" to disable ROOT. These NO_xxx Makefile variables are only used to control autodetection.

;NOTE 2
: Since June 2019 the mxml and mscb packages are submodules of the midas package, so no need to clone them separately. If you have an existing clone of midas but not yet the submodules, you need

$ git submodule update --init --recursive

: To update both midas and the submodules, you need

$ git pull --recurse-submodules

== Install ROOTANA ==

* cd $HOME/packages
* git clone https://bitbucket.org/tmidas/rootana
* cd rootana
* make

== Install ROODY ==

* cd $HOME/packages
* git clone https://bitbucket.org/tmidas/roody
* cd roody
* make
* $HOME/packages/roody/bin/roody, run the program

== Install additional additional packages ==

* cd $HOME/packages
* git clone https://bitbucket.org/ttriumfdaq/vme
* git clone https://bitbucket.org/ttriumfdaq/frontends

== Build special versions of MIDAS ==

Build special versions of MIDAS for the case when some MIDAS programs, such as VME frontends, will run on a different computer that may have a different flavour of operating system, i.e. 32-bit linux or an older version of Scientific Linux.

* login to the computer where the frontends will run and:
* if it is a 32-bit linux: cd $HOME/packages/midas; make linux32
* if it is a 64-bit linux: cd $HOME/packages/midas; make linux64

* login to the host computer to cross-compile ARM code:
* if it is an ARM linux: cd $HOME/packages/midas; make linuxarm # may need to install ARM cross compilers

== Prepare VME hardware ==

Hardware check list:
* VME crate
* VME processor (supported are V77xx, V7805, V7865)
* On all VME modules, set the VME address jumpers as described here: http://daq-plone.triumf.ca/SM/docs/local/vme_jumpers
* run vmescan to confirm correct VME addresses
** cd $HOME/packages
** svn checkout https://ladd00.triumf.ca/svn/daqsvn/trunk/vme
** cd vme
** make
** ./vmescan.exe (or _gef.exe, depending on the VME driver in use)

== Install Universe-II VME driver (V7648, V7750, V7805, V7851) ==

* login as root (ssh root@localhost)
* follow instructions: https://www.triumf.info/wiki/DAQwiki/index.php/VME-CPU#V7648.2C_V7750.2C_V7805.2C_V7851_:_Setup_vme_universe_VME_drivers
* cd ~/packages/vme; vmescan.exe

== Install Tsi-148 VME driver (V7865) ==

* login as root (ssh root@localhost)
* follow instructions: https://www.triumf.info/wiki/DAQwiki/index.php/VME-CPU#V7865_and_XVB-602_:_Setup_gefvme.2Ftsi148_VME_drivers
* cd ~/packages/vme; vmescan_gef.exe

== Setup the experiment environment ==

* Decide which computer will host MIDAS (where MIDAS shared memory buffers will reside).
: This computer will run the [[mserver]], [[mlogger]] and [[mhttpd]] applications. (It is usually the machine where the MIDAS,ROOT etc. packages have been downloaded). It will be referred to as the host machine (localhost).

The environment is slightly different depending on whether all programs run on the host machine, or whether some programs run on remote host(s) :
=== ALL programs run on localhost ===
:If all programs run on the host machine (localhost), it is not necessary to run [[mserver]]. [[Environment Variables#MIDAS_SERVER_HOST|MIDAS_SERVER_HOST]] will not be assigned (see example .cshrc [[#Prepare the user account|above]]).

=== Some programs run on REMOTE host(s) ===
: '''IMPORTANT:'''
# Since August 2015 '''you must explicitly allow access for clients running on remote machines'''. To do this, follow the '''[[Security#MIDAS programs on remote machines|instructions here]]'''.
# The example code .cshrc ([[#Prepare the user account|see above]]) should be present on both host and remote machine(s). This will ensure that [[Environment Variables#MIDAS_SERVER_HOST|MIDAS_SERVER_HOST]] will NOT be set for the host machine (localhost), but on a remote machine, MIDAS_SERVER_HOST will be set to the MIDAS host machine.
# The client [[mserver]] must be started on the MIDAS host machine. Note that multiple experiments can run on the same host machine by starting several instances of [[mserver]] (one for each experiment) running with different ports (and .cshrc would be edited so that MIDAS_SERVER_HOST is set to the appropriate port for the experiment).

 
On the host machine:
* mkdir $HOME/online
* cd $HOME/online
* create directories for local programs, sources, elog and history: mkdir bin src elog history
* create data directory: mkdir -p /ladd/data1/t2kvme5/data; ln -s /ladd/data1/t2kvme5/data $HOME/online
* create the exptab file "$HOME/online/exptab" following the example below. The first entry (exptname) is the name if the DAQ system (MIDAS experiment name), the second entry (/home/USER/online) is the location of MIDAS shared memory buffers (by convention, $HOME/online), the third entry (kopio03) is your username.
<pre>exptname /home/kopio03/online kopio03</pre>
* logout and login again for all changes to take effect

== Setup experiment startup scripts ==
* login to the experiment host computer
* echo $MIDAS_SERVER_HOST ### to check correct value - should be blank
* create $HOME/online/bin/start_daq.sh, replacing XXX with the hostname of the machine running the experiment (and changing the mserver and mhttpd ports, as needed).

#!/bin/sh
# start_daq.sh
cd $HOME/online
#
case `hostname` in XXX*)
echo "Good, we are on XXX!"
;;
*)
echo "The start_daq script should be executed on XXX"
exit 1
;;
esac
#
odbedit -c clean
# start [[mhttpd]] on default port. (Mongoose https version - see [[mhttpd]] for other options)
mhttpd -D -a localhost -a XXX.triumf.ca # optionally restrict access to specified hosts
#
# start [[mserver]] on default port (use argument -p to use a different port)
mserver -D # access must now be specifically allowed - see [[#Setup the experiment environment|above]]

# OR ([[#NOTES|older MIDAS versions]])
# mhttpd -p 8081 -D -a localhost -a XXX.triumf.ca # optionally restrict access to specified hosts
# mserver -p 7071 -D -a localhost -a lxdragon01.triumf.ca -a lxdragon02.triumf.ca -a XXX.triumf.ca # optionally restrict access to specified hosts

#
mlogger -D
#end file

== Run the MIDAS Web Server ==
Let's start the MIDAS webserver for the first time:

Start [[mhttpd]] on the ''experiment host'' (localhost) like this:
[mhostpc] mhttpd
You will get the following messages:
[mhttpd,INFO] ODB subtree /Runinfo corrected successfully
Mongoose web server will listen on ports "8080r,8443s"
[mhttpd,ERROR] [mhttpd.cxx:17892:mongoose,ERROR] cannot find SSL certificate file "/home/agdaq/online/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17893:mongoose,ERROR] please create SSL certificate file: openssl req -new -nodes -newkey rsa:2048 -sha256 -out ssl_cert.csr -keyout ssl_cert.key; openssl x509 -req -days 365 -sha256 -in ssl_cert.csr -signkey ssl_cert.key -out ssl_cert.pem; cat ssl_cert.key >> ssl_cert.pem
could not start the mongoose web server, see messages and midas.log, bye!

Create a self-signed certificate suitable for initial testing by executing the command printed by mhttpd:


[mhostpc] openssl req -new -nodes -newkey rsa:2048 -sha256 -out ssl_cert.csr -keyout ssl_cert.key; openssl x509 -req -days 365 -sha256 -in ssl_cert.csr -signkey ssl_cert.key -out ssl_cert.pem; cat ssl_cert.key >> ssl_cert.pem

For production use, you should create a properly signed certificate, see [[Mhttpd#Create an SSL certificate|create your own SSL certificate]] or you should run mhttpd behind an SSL proxy.

Run mhttpd again.

You will get the following messages:
[mhttpd,INFO] ODB subtree /Runinfo corrected successfully
Mongoose web server will listen on ports "8080r,8443s"
Mongoose web server will use SSL certificate file "/home/johnfoo/packages/midas/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17633:mongoose,ERROR] mongoose web server cannot find password file "/home/johnfoo/online/htpasswd.txt"
[mhttpd,ERROR] [mhttpd.cxx:17634:mongoose,ERROR] please create password file: htdigest -c /home/johnfoo/online/htpasswd.txt Default midas
could not start the mongoose web server, see messages and midas.log, bye!

Create the password file by following the instructions printed by mhttpd. The http digest domain name is the experiment name, suggested default user name is "midas". You will be asked to type in a password


[mhostpc] htdigest -c /home/johnfoo/online/htpasswd.txt exptname midas
Adding password for midas in realm exptname.
New password:
Re-type new password:

It is a good idea to set the password file {{Filepath|path=htpasswd.txt}} readable and writable by owner only.

Now restart {{Utility|name=mhttpd}}
[mhostpc] mhttpd
Mongoose web server will listen on ports "8080r,8443s" **see note
Mongoose web server will use SSL certificate file "/home/suz/packages/midas/ssl_cert.pem"
Mongoose web server will use authentication realm "Default", password file "./htpasswd.txt"

Now point a web browser running on the same host computer (localhost) to https://localhost:8443
If the web browser is running on a different computer, go to URL of the form

https://mhostpc.triumf.ca:8443 (substitute your host machine name and domain for "mhostpc.triumf.ca")

If you are using the default SSL certificate you will probably get a message: "This Connection is Untrusted". Click "I understand the risks" and add an exception. This is because the test certificate is self-signed. Then confirm an exception.

If instead you get a "connection refused" error, the midas host pc may have the firewall enabled. To make a firewall exception for MIDAS, follow instructions here http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29

You should then see an authentication box asking you for the user name and password. The user name is "midas". Enter the password you just created. The Midas [[Status Page]] should appear with multiple buttons for run control as well as equipment listing (no equipments will be listed as yet) and application listings. Please refer to [[mhttpd]] (the MIDAS Web-based Run Control utility) for further information. You can start and stop runs from the main status page, and use the [[ODB Page]] to access the database (ODB).

; Note
: Default ports of 8080 and 8443 are used by [[mhttpd]]. If these ports are in use on your machine, start mhttpd with alternative ports, e.g.
[mhostpc] mhttpd --https 8448 --http 8089
: or see [[Mhttpd#Usage]] to change the default ports.

== Setup experiment database (ODB) ==

* run $HOME/online/bin/start_daq.sh

* odbedit, run these commands: (replace user names and directory names)
<pre>
# set "/Logger/Message file" "/home/kopio03/online/midas.log" # obsolete
set "/Logger/Data Dir" "/home/kopio03/online/data"
create STRING "/Logger/History dir"
set "/Logger/History dir" "/home/kopio03/online/history"
create STRING "/Logger/Elog dir"
set "/Logger/Elog dir" "/home/kopio03/online/elog"
set "/Logger/ODB dump file" "/home/kopio03/online/history/run%05d.xml"
set "/Logger/ODB dump" "y"
set "/Logger/Channels/0/Settings/Filename" "run%05dsub%03d.mid.gz"
set "/Logger/Channels/0/Settings/Subrun byte limit" "1000000000"
set "/Logger/Channels/0/Settings/Compression" 1
set "/Logger/Channels/0/Settings/ODB Dump" "y"
set "/Programs/Logger/Required" y
set "/Programs/Logger/Start command" "mlogger -D"
set "/Programs/fevme/Required" "y"
set "/Programs/fevme/Start command" "ssh -n lxdaq09 $HOME/online/src/fevme_gef.exe -O"
exit
</pre>
* open web browser e.g. firefox.
* go to the midas status page at https://localhost:8443 (default port).
** if running [[mhttpd]] with Mongoose HTTPS/OpenSSL (the default) for the first time, you will need to create a password file. Follow the instructions (see [[mhttpd#HTTPS/SSL server (Mongoose)]] for details).
** For other options (i.e. HTTPS/SSL proxy) see [[#Secure MIDAS and ELOG Web access]]
*OR open the midas status page at http://localhost:8081 ([[#NOTES|older MIDAS versions]])
* midas status page will show most stuff "red" as nothing is running yet
* DON'T DO THIS YET run ./fevme.exe (on the computer with the VME interface, could be different from computer hosting the experiment), observe that corresponding equipments have been created
* save the url bookmark to the "personal toolbar"
* go to the Programs page, stop mlogger, stop fevme, start mlogger, start fevme
* go to the Status page, start run, stop run
* go back to the Status page, everything should be green
* start a run
* send signals to the ADC gate
* you should be getting events
* to look at data, proceed with setting up the [[ROOTANA|ROOT Analyzer]].

== Start DAQ programs at boot time ==

* add this to /etc/rc.local (replace username and location of the start_daq script)
<pre>
su - alpha -c /home/alpha/online/bin/start_daq.sh
</pre>

== Setup local software version control ==

Version control for experiment source code is setup using "git" (http://git-scm.com/)

* cd $HOME/online
* git init
* git add exptab
* git add bin/start_daq.sh
* git add .gitignore ### contents can be
<pre>
*~
*.o
*.exe
</pre>
* git add src/Makefile src/*.cxx ...
* git commit -a

== Adjust MIDAS buffer sizes ==

Default MIDAS SYSTEM buffer size is 8 Mbytes, fairly small for high-data-rate experiments. The rule of thumb is to have at least a few seconds worth of buffer space available. For example, if event size is 10 Kbytes and the event rate is 1 kHz, data rate is 10*10^3*1*10^3 = 10 Mbytes/sec. To buffer 10 seconds of data we need 100 Mbytes of buffer space.

To resize the MIDAS event buffers (SYSTEM, etc) do this:
* stop all frontends, stop mlogger
* start odbedit:
** cd "/Experiment/Buffer sizes"
** set SYSTEM 100000000
* run "mdump -z SYSTEM"
* if mdump complains about the size of .SYSTEM.SHM, remove it, try again.
* ls -l /dev/shm ### to observe that the size of shared memory is correct

== Secure MIDAS and ELOG Web access ==
In versions prior to May 2015, the default web access to MIDAS and ELOG uses the "http:" protocol which is insecure. In this case, all information is transmitted as clear text meaning that secret, confidential and sensitive information (such as the MIDAS and ELOG passwords and usernames) can be stolen "easily". This means that even "password protected" MIDAS and ELOG pages are not really protected if accessed using the "http" method.

Better security for HTTP is gained by using a password protected '''SSL (https) proxy'''. (It does not provide absolute security because of remaining problems with the security of SSL certificates, security of passwords, etc). Setting up an SSL (https) proxy is described [[#Setting up an HTTP proxy|below]].

Since May 2015, an ''alternative secure option'' to setting up an HTTP proxy is available to users of MIDAS. Recent versions of elogd (ELOG) do support SSL https:// connections, and [[#mhttpd with HTTPS/SSL server (Mongoose)]] is now available. This option is the default, and provides a similar level of security to an HTTP proxy.

See [[Security#Web Access]] for a comparison of these two secure options.

=== mhttpd with HTTPS/SSL server (Mongoose) ===

Since May 2015 the MIDAS web server [[mhttpd]] is explicitly linked with OpenSSL to provide secure HTTPS connections via the [https://bitbucket.org/tmidas/midas/src/ecb9a8537448a8a43f7f9a2bfdb82e578208cde3/doc/mongoose/?at=develop Mongoose] web server (see [[mhttpd]]). With this version, default web access to MIDAS uses the "https" protocol. Web access to {{Utility|name=mhttpd}} can be restricted by using the "-a hostname" switch of [[mhttpd]]. The first time {{Utility|name=mhttpd}} is run, a password file must be created. An SSL certificate is also required. See [[mhttpd#HTTPS/SSL server (Mongoose)|HTTPS/SSL server (Mongoose)]] for instructions.

=== mhttpd using an HTTPS/SSL proxy ===

THESE INSTRUCTIONS ARE WRONG, DO NOT DO THIS. See instead [https://daq00.triumf.ca/MidasWiki/index.php/Setup_MIDAS_experiment_at_TRIUMF#Install_https_proxy this section].

An [[#Setting up an HTTP proxy|HTTP proxy]] must be set up. This is the only way of securing older version of [[mhttpd]] (pre August 2015). Older versions of mhttpd are started using the -p port option e.g.
* mhttpd -D -p 8080 

To run a new version of mhttpd using an HTTP proxy, use the options provided to run the old (non-Mongoose) webserver on a given port, i.e.
* mhttpd --oldserver 8080 --nomg -D 

When using an SSL proxy, only access from the SSL proxy (and maybe some special trusted machines) should be permitted.
This is done using the "-a hostname" switch of [[mhttpd]]. Normally there will be only "-a localhost" switch, enabling access only for the local machine (where the SSL proxy is running). Additional "-a hostname" switches enable access from listed local machines. No "-a xxx" enables access from everywhere (defeating the purpose of the SSL proxy, unless access controls are enforced elsewhere, i.e. by a site firewall or by local firewall rules).

== Setting up an HTTP proxy ==
In this example, we use APACHE HTTPD to password-protect a typical midas/mhttpd and elog installation.

In this configuration, one uses the Linux stock httpd that accepts encrypted https:// connections and forwards them to mhttpd and elogd. Instead of (or in addition to) using mhttpd and elogd passwords, one configures password protection in httpd via the regular apache httpd password mechanisms (htpasswd, etc).

Recent versions of elogd do support SSL https:// connections, but if one is running an SSL proxy for anyway, it is simpler to run both through the same SSL proxy using the same SSL host certificate and the same httpd password file.

=== Restricting http: access to elogd ===
;Note
:Recent versions of elogd do support SSL https:// connections. The following information is for those using an HTTP proxy (see above).

For elogd, this is done using the "-n localhost" switch with enables only access from the same machine if present, or access from anywhere is absent (defeating the purpose of the SSL proxy, unless access controls are enforced elsewhere).

(It is recommended to run elogd from the same user as the main daq user and to keep elogd.cfg and all logbooks in the home directory of this user, where they are captured by the normal site backup system)

== Install standalone elog ==

* login into the user account that will run the elog
* cd $HOME/packages
* git clone https://bitbucket.org/ritt/elog
* cd elog
* make
* create new file start_elogd with this contents:
<pre>
#!/bin/sh

killall elogd
sleep 1
killall -KILL elogd
sleep 1
$HOME/packages/elog/elogd -n localhost -x -c $HOME/packages/elog/elogd.cfg -p 8082 -D

#end
</pre>
* chmod a+x start_elogd
* edit elogd.cfg to read:
<pre>
[global]
port = 8082
SMTP host = smtp.triumf.ca
URL = https://titan00.triumf.ca/elog/

Reverse sort = 1
Display Mode = full

#List Menu commands = New, Find, Admin, Help
#Menu commands = New, Edit, Reply, Find, Duplicate, Help

Entries Per Page = 30
Supress Email on edit = 1
Default encoding = 1
Page title = TITAN ELOG
Resolve host names = 1

Logfile = /home/titan/packages/elog/elogd.log
#Logging level = 3

[midas]

List page Title = T2K M11 MIDAS ELOG
Comment = T2K M11 MIDAS ELOG
Page Title = T2K M11 MIDAS ELOG
RSS Title = [$logbook - $type - $system] $subject, posted by $author

Attributes = Author, Subject, Run, Type, System
Show Attributes Edit = Run, Author, Subject, Type, System
Required Attributes = Author, Type, System, Subject

Options Type = Routine, Reply, Shift Summary, Modification, Question, Info, Problem
Options System = General, DAQ, Beamline

Preset Run = $shell(MIDASSYS=. /home/t2km11/packages/midas/linux/bin/odbedit -d Runinfo -c 'ls -v \"run number\"')

Preset On Reply Type = Reply
Preset On Reply Run = $shell(MIDASSYS=. /home/t2km11/packages/midas/linux/bin/odbedit -d Runinfo -c 'ls -v \"run number\"')

List Display = Date, Subject, Type, System, Author, ID
Quick Filter = Date, Type, ID

Remove on reply = Author
Quote on reply = 1

Use lock = 1

************* Email Functionality ****************

Use Email Subject = [T2KM11 - $System] $Subject
Omit Email To = 1

Email System General = xxx
</pre>
* ./start_elogd &
* firefox http://localhost:8082 # hould show the elog message index

To start elogd automatically when the machine is rebooted, login as root and
* add this text to /etc/rc.local:
<pre>
su - titan -c "/home/titan/packages/elog/start_elogd"
</pre>
* chmod a+x /etc/rc.local
* systemctl start rc-local

To import elog entries from the mhttpd elog, do this:

* cd ~/packages/elog/logbooks
* ln -s /home/t2km11/online/elog midas
* cd midas
* ~/packages/elog/elconv)

== Install https proxy ==

THESE INSTRUCTIONS ARE OBSOLETE, INSTEAD,
* GO HERE: https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Install_apache_httpd_proxy_for_midas_and_elog
* AND GO HERE: https://midas.triumf.ca/MidasWiki/index.php/Quickstart_Linux#Run_the_MIDAS_Web_Server

FOLLOWING INSTRUCTIONS ARE OBSOLETE...

* login as root to the https proxy machine
* cd ~root
* yum install mod_ssl
* yum install crypto-utils # see http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_monitoring_of_HTTPS_certificates
* create a certificate request (replace ladd09 with your hostname): openssl req -new -nodes -newkey rsa:2048 -sha256 -out ladd09.csr -keyout ladd09.key (answer: CA, BC, Vancouver, TRIUMF, DAQ, ladd09.triumf.ca, email@email.com
* sign it by TRIUMF:
** mail -s "Certificate request" yourself@email.com < ladd09.csr
** forward this request to Andrew Daviel
** he will email the signed crt file, copy it to this system as ladd09.crt
* sign it yourself: openssl x509 -req -days 365 -sha256 -in ladd09.csr -signkey ladd09.key -out ladd09.crt
* (if the certificate expires, renew it by signing it again)
* Additional commands for working with certificates:
** explore the private key: openssl pkey -in ladd09.key -text -noout
** explore the certificate request: openssl req -in ladd00.csr -text -noout
** explore the certificate: openssl x509 -in ladd09.crt -noout -text
* move certificate files to proper system locations:
* mv ladd09.key /etc/pki/tls/private/
* mv ladd09.crt /etc/pki/tls/certs/
* if selinux is enabled, do this:
** restorecon -Rv /etc/pki/tls/certs/
** restorecon -Rv /etc/pki/tls/private/
** /usr/sbin/setsebool -P httpd_can_network_connect 1
* open /etc/httpd/conf.d/ssl.conf in a text editor, go to the very bottom and right before the "</VirtualHost>" entry, add following text:
<pre>
...
SSLCertificateFile /etc/pki/tls/certs/ladd09.crt
SSLCertificateKeyFile /etc/pki/tls/private/ladd09.key

ProxyPass /elog/ http://localhost:8082/ retry=1
ProxyPass / http://localhost:8080/ retry=1

<Location />

SSLRequireSSL
AuthType Basic
AuthName "password protected site"
Require valid-user

# create password file: touch /etc/httpd/htpasswd
# to add new user or change password: htpasswd /etc/httpd/htpasswd username
AuthUserFile /etc/httpd/htpasswd

</Location>
</VirtualHost>
...
</pre>
* comment out duplicate "SSLCertificateFile" and "SSLCertificateKeyFile" elsewhere in the file
* (optionally) If you got a certificate that is signed by DigiCert or RapidSSL then you'll need to add a line specifying the certificate chain file:
<pre>
...
SSLCertificateChainFile /etc/pki/tls/certs/DigiCertCA.crt
...
</pre>
* touch /etc/httpd/htpasswd
* htpasswd /etc/httpd/htpasswd midas # enter password midas
* chkconfig httpd on
* service httpd restart
* firewall-cmd --add-port=443/tcp --permanent
* firewall-cmd --reload
* firewall-cmd --list-all
* test it
** test the SSL proxy: https://host/ should yield the midas status page, https://host/elog/ should yield the elog message index
* in ODB, set "/Elog/URL" to "https://host/elog/"
* now from the midas status page, the "Elog" button should take us to the https Elog URL

In needed, enable user directories: https://blah/~user in ~user/public_html

* edit /etc/httpd/conf.d/userdir.conf, replace "UserDir disabled" with "UserDir enabled"
* setsebool -P httpd_enable_homedirs true
* systemctl restart httpd

== Setup the history mhttpd for faster access to history plots ==
When running an SSL proxy,
* start the main mhttpd (orange command for old mhttpd, green for new mhttpd with Mongoose(post August2015):
** "mhttpd -p 8071 -D" or
** "mhttpd -D --oldserver 8071 --nomg" 

* start the history mhttpd
** "mhttpd -p 8072 -D -H" or
** "mhttpd -D -H --oldserver 8072 --nomg" 
* set ODB /History/URL to "http://alphacpc09.cern.ch:8072/HS/"
* open the MIDAS status page
* go to the history section, try to open any history plot, observe that the history plot gif image loads correctly, inspect it's URL (use "copy image URL" or "view source", etc), it should point to port 8072 causing connection to the history mhttpd.
* continue with these instructions to setup history mhttpd access through an SSL proxy:
* setup SSL proxy access (required mod_proxy_html)
** login as root to the SSL proxy machine
** on SL5, install the missing mod_proxy_html httpd module:
** yum install httpd-devel libxml2-devel
** wget http://apache.webthing.com/mod_proxy_html/mod_proxy_html.tar.bz2
** tar xjvf mod_proxy_html.tar.bz2
** cd mod_proxy_html
** apxs -c -I. -I/usr/include/libxml2 -i mod_proxy_html.c
** apxs -c -I. -I/usr/include/libxml2 -i mod_xml2enc.c
** cd /etc/httpd/conf.d, add this to ssl.conf:

;before the ProxyPass statements:
<pre>
# proxy the MIDAS web servers
LoadModule xml2enc_module modules/mod_xml2enc.so
LoadModule proxy_html_module modules/mod_proxy_html.so
ProxyHTMLLinks a href
ProxyHTMLLinks link href
ProxyHTMLLinks img src
#ProxyHTMLEnable On
ProxyRequests off
</pre>
;after the ProxyPass statements:
<pre>
# ALPHA1 history access
ProxyPass /alpha1/history/ http://alphacpc09.cern.ch:8072/HS/ retry=1
ProxyPass /alpha1/ http://alphacpc09.cern.ch:8071/ retry=1

ProxyHTMLEnable On
ProxyHTMLURLMap http://alphacpc09.cern.ch:8072/HS/ /alpha1/history/
</pre>
;adjust:
*"alpha1" is the experiment name
*"alphacpc09.cern.ch" is the machine running mhttpd
*"8071" is the port number of the main mhttpd
** "mhttpd -p 8071 -D" or
** "mhttpd -D --oldserver 8071 --nomg" 
* "8072" is the port number of the history mhttpd
** "mhttpd -p 8072 -D -H" or
** "mhttpd -D -H --oldserver 8072 --nomg" 

[[Category:Installation]] [[Category:Buffer]] [[Category:Driver]]

Setup MIDAS experiment at TRIUMF

2026-02-03T17:32:31Z

Dfujimoto: /* mhttpd using an HTTPS/SSL proxy */

{{Pagelinks}}
== Introduction ==
This page describes setting up a MIDAS experiment at TRIUMF. This information can be adapted for other sites.

== Environment variables ==
* '''MIDASSYS''' Base directory of the MIDAS package, midas and mxml should be at the same level.
* '''MIDAS_EXPTAB''' Experiment definition file
* '''MIDAS_SERVER_HOST''' MIDAS host server name for remote midas connections.
* '''MIDAS_EXPT_NAME''' Experiment name

== Standard layout of MIDAS experiment ==
The following shows the directory layout of a standard MIDAS experiment:

/home/exptuser/
packages/
root <---- ROOT
mxml
mscb
midas/ <---- MIDAS
linux/{lib,bin} <---- binaries matching the selected 64-bit/32-bit flavour of ROOT
linux-m32/{lib,bin} <---- limited function 32-bit binaries for 32-bit frontend machines, build by "make linux32"
linux-m64/{lib,bin} <---- limited function 64-bit binaries (only needed if ROOT and linux/bin are 32-bit)
linux-arm/{lib,bin} <---- full function ARM cross-compiled using "make linuxarm"
linux-crosscompile/{lib,bin} <---- cross-compiled limited function binaries for PPC and ARM frontends (see Makefile)
rootana <---- ROOT analyzer
roody <---- graphical online histogram viewer for MIDAS and ROOTANA
online/
exptab <---- experiment definition
{.ODB,.SYSTEM,.SYSMSG,etc}.SHM <---- MIDAS shared memory save files
src <---- experiment frontend sources
bin,scripts
elog <---- MIDAS elog
history <---- MIDAS history
data -> /data/exptname/current <---- symlink to the data directory
/data/exptname/current <---- experiment data directory with ODB save files and MIDAS .mid/.mid.gz data files

== Prepare computers ==

On some operating systems, several MIDAS functions require administrator access:

* on el7 linux (SL7/CC7/CentOS7/RHEL7) - access to mhttpd port 8443 requires special firewall rules, see here: https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29
* on el7 linux - access to mserver to run frontends and other programs on some other computer requires special firewall rules, see here: https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29
* on el7 linux - on the frontend machines (and other machines that will connect to the mserver, the same firewall rule needs to be created (use the IP address of the machine running the mserver)

== Prepare the user account ==
<div id="NOTES"></div>

* Setup the user account for running this instance of midas. For machines part of the LADD cluster, follow these [http://daq-plone.triumf.ca/SM/docs/local/NewLaddUser] instructions.
* check that the account is using the /bin/bash shell
* make $HOME/.profile look like this:

<pre>
#!/bin/echo You must source

export SVN_EDITOR="emacs -nw"
export GIT_EDITOR="emacs -nw"
export MIDASSYS=$HOME/packages/midas
export ROOTANASYS=$HOME/packages/rootana
export MIDAS_EXPTAB=$HOME/online/exptab
#
# setup the MIDAS mserver
#
case `hostname` in
daq07*)
unset MIDAS_SERVER_HOST
;;
*)
export MIDAS_SERVER_HOST=daq07.triumf.ca:7070
;;
esac
#
# select 64-bit or 32-bit MIDAS and ROOT
#
case `uname -i` in
i386)
source /daq/daqshare/olchansk/root/root_v5.34.01_SL62_32/bin/thisroot.sh
export PATH=.:$MIDASSYS/linux-m32/bin:$PATH
;;
*)
#source /daq/daqshare/olchansk/root/root_v5.34.34_SL67_64/bin/thisroot.sh
source $HOME/packages/root/bin/thisroot.sh
export PATH=.:$MIDASSYS/linux/bin:$PATH
;;
esac
#
export PATH=.:$HOME/online/bin:$HOME/packages/roody/bin:$PATH
#
#end
</pre>

* mkdir $HOME/packages
* Logout and login again, for .cshrc changes to take effect

== Install ROOT ==

* Identify the Linux version: RH9 (Red Hat Linux 9), FC3 (Fedora Core 3), RHEL4/SL4 (Red Hat Enterprise LInux 4/Scientific Linux 4), SL5x, SL6x, (CentOS/CC/SL) el7x: more /etc/redhat-release
* Decide to use 32-bit or 64-bit ROOT ('uname -a')
* cd $HOME/packages
* ls -l /daq/daqshare/olchansk/root/ ### to see all available ROOT packages
* ln -s /daq/daqshare/olchansk/root/root_vNNN_VVV_BB root, where NNN is the latest available version of ROOT ("ls -l /daq/daqshare/olchansk/root"), VVV is the Linux version code (RH9, FC3, SL4, etc) and BB is "_32" or "_64" for 32-bit or 64-bit ROOT. For example: /daq/daqshare/olchansk/root/root_v5.10.00_SL40
* for example: ln -s /daq/daqshare/olchansk/root/root_v5.34.34_el72_64 $HOME/packages/root
* Check that ROOT works: "source $HOME/packages/root/bin/thisroot.sh; root". Type ".q" to exit root.

== Install MIDAS ==

* cd $HOME/packages
* (OBSOLETE) svn co svn+ssh://svn@savannah.psi.ch/repos/meg/midas/trunk midas, password "svn". (password has to be entered twice)
* (OBSOLETE) svn co svn+ssh://svn@savannah.psi.ch/repos/meg/mxml/trunk mxml
* git clone https://bitbucket.org/tmidas/midas --recursive
* (OBSOLETE) git clone https://bitbucket.org/tmidas/mxml
* (OBSOLETE) git clone https://bitbucket.org/tmidas/mscb
* (IF BITBUCKET IS DOWN) git clone -v --progress https://daq.triumf.ca/~daqweb/git/mxml.git
* (IF BITBUCKET IS DOWN) git clone -v --progress https://daq.triumf.ca/~daqweb/git/mscb.git
* (IF BITBUCKET IS DOWN) git clone -v --progress https://daq.triumf.ca/~daqweb/git/midas.git
* cd midas
* make
* (only if needed) make linux32 ### build the 32-bit MIDAS libraries
* ls -l linux/bin/odbedit ### check that odbedit has been created (do not run it yet)

You can see a list of other installation problems at [[Common problems & Debugging recipes]].

;NOTE 1
: Optional features in MIDAS can be explicitly disabled if desired when making MIDAS using the NO_xxx feature (NO_ROOT,NO_MYSQL,NO_ODBC,NO_SQLITE,NO_MSCB), e.g. "make NO_ROOT=1" to disable ROOT. These NO_xxx Makefile variables are only used to control autodetection.

;NOTE 2
: Since June 2019 the mxml and mscb packages are submodules of the midas package, so no need to clone them separately. If you have an existing clone of midas but not yet the submodules, you need

$ git submodule update --init --recursive

: To update both midas and the submodules, you need

$ git pull --recurse-submodules

== Install ROOTANA ==

* cd $HOME/packages
* git clone https://bitbucket.org/tmidas/rootana
* cd rootana
* make

== Install ROODY ==

* cd $HOME/packages
* git clone https://bitbucket.org/tmidas/roody
* cd roody
* make
* $HOME/packages/roody/bin/roody, run the program

== Install additional additional packages ==

* cd $HOME/packages
* git clone https://bitbucket.org/ttriumfdaq/vme
* git clone https://bitbucket.org/ttriumfdaq/frontends

== Build special versions of MIDAS ==

Build special versions of MIDAS for the case when some MIDAS programs, such as VME frontends, will run on a different computer that may have a different flavour of operating system, i.e. 32-bit linux or an older version of Scientific Linux.

* login to the computer where the frontends will run and:
* if it is a 32-bit linux: cd $HOME/packages/midas; make linux32
* if it is a 64-bit linux: cd $HOME/packages/midas; make linux64

* login to the host computer to cross-compile ARM code:
* if it is an ARM linux: cd $HOME/packages/midas; make linuxarm # may need to install ARM cross compilers

== Prepare VME hardware ==

Hardware check list:
* VME crate
* VME processor (supported are V77xx, V7805, V7865)
* On all VME modules, set the VME address jumpers as described here: http://daq-plone.triumf.ca/SM/docs/local/vme_jumpers
* run vmescan to confirm correct VME addresses
** cd $HOME/packages
** svn checkout https://ladd00.triumf.ca/svn/daqsvn/trunk/vme
** cd vme
** make
** ./vmescan.exe (or _gef.exe, depending on the VME driver in use)

== Install Universe-II VME driver (V7648, V7750, V7805, V7851) ==

* login as root (ssh root@localhost)
* follow instructions: https://www.triumf.info/wiki/DAQwiki/index.php/VME-CPU#V7648.2C_V7750.2C_V7805.2C_V7851_:_Setup_vme_universe_VME_drivers
* cd ~/packages/vme; vmescan.exe

== Install Tsi-148 VME driver (V7865) ==

* login as root (ssh root@localhost)
* follow instructions: https://www.triumf.info/wiki/DAQwiki/index.php/VME-CPU#V7865_and_XVB-602_:_Setup_gefvme.2Ftsi148_VME_drivers
* cd ~/packages/vme; vmescan_gef.exe

== Setup the experiment environment ==

* Decide which computer will host MIDAS (where MIDAS shared memory buffers will reside).
: This computer will run the [[mserver]], [[mlogger]] and [[mhttpd]] applications. (It is usually the machine where the MIDAS,ROOT etc. packages have been downloaded). It will be referred to as the host machine (localhost).

The environment is slightly different depending on whether all programs run on the host machine, or whether some programs run on remote host(s) :
=== ALL programs run on localhost ===
:If all programs run on the host machine (localhost), it is not necessary to run [[mserver]]. [[Environment Variables#MIDAS_SERVER_HOST|MIDAS_SERVER_HOST]] will not be assigned (see example .cshrc [[#Prepare the user account|above]]).

=== Some programs run on REMOTE host(s) ===
: '''IMPORTANT:'''
# Since August 2015 '''you must explicitly allow access for clients running on remote machines'''. To do this, follow the '''[[Security#MIDAS programs on remote machines|instructions here]]'''.
# The example code .cshrc ([[#Prepare the user account|see above]]) should be present on both host and remote machine(s). This will ensure that [[Environment Variables#MIDAS_SERVER_HOST|MIDAS_SERVER_HOST]] will NOT be set for the host machine (localhost), but on a remote machine, MIDAS_SERVER_HOST will be set to the MIDAS host machine.
# The client [[mserver]] must be started on the MIDAS host machine. Note that multiple experiments can run on the same host machine by starting several instances of [[mserver]] (one for each experiment) running with different ports (and .cshrc would be edited so that MIDAS_SERVER_HOST is set to the appropriate port for the experiment).

 
On the host machine:
* mkdir $HOME/online
* cd $HOME/online
* create directories for local programs, sources, elog and history: mkdir bin src elog history
* create data directory: mkdir -p /ladd/data1/t2kvme5/data; ln -s /ladd/data1/t2kvme5/data $HOME/online
* create the exptab file "$HOME/online/exptab" following the example below. The first entry (exptname) is the name if the DAQ system (MIDAS experiment name), the second entry (/home/USER/online) is the location of MIDAS shared memory buffers (by convention, $HOME/online), the third entry (kopio03) is your username.
<pre>exptname /home/kopio03/online kopio03</pre>
* logout and login again for all changes to take effect

== Setup experiment startup scripts ==
* login to the experiment host computer
* echo $MIDAS_SERVER_HOST ### to check correct value - should be blank
* create $HOME/online/bin/start_daq.sh, replacing XXX with the hostname of the machine running the experiment (and changing the mserver and mhttpd ports, as needed).

#!/bin/sh
# start_daq.sh
cd $HOME/online
#
case `hostname` in XXX*)
echo "Good, we are on XXX!"
;;
*)
echo "The start_daq script should be executed on XXX"
exit 1
;;
esac
#
odbedit -c clean
# start [[mhttpd]] on default port. (Mongoose https version - see [[mhttpd]] for other options)
mhttpd -D -a localhost -a XXX.triumf.ca # optionally restrict access to specified hosts
#
# start [[mserver]] on default port (use argument -p to use a different port)
mserver -D # access must now be specifically allowed - see [[#Setup the experiment environment|above]]

# OR ([[#NOTES|older MIDAS versions]])
# mhttpd -p 8081 -D -a localhost -a XXX.triumf.ca # optionally restrict access to specified hosts
# mserver -p 7071 -D -a localhost -a lxdragon01.triumf.ca -a lxdragon02.triumf.ca -a XXX.triumf.ca # optionally restrict access to specified hosts

#
mlogger -D
#end file

== Run the MIDAS Web Server ==
Let's start the MIDAS webserver for the first time:

Start [[mhttpd]] on the ''experiment host'' (localhost) like this:
[mhostpc] mhttpd
You will get the following messages:
[mhttpd,INFO] ODB subtree /Runinfo corrected successfully
Mongoose web server will listen on ports "8080r,8443s"
[mhttpd,ERROR] [mhttpd.cxx:17892:mongoose,ERROR] cannot find SSL certificate file "/home/agdaq/online/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17893:mongoose,ERROR] please create SSL certificate file: openssl req -new -nodes -newkey rsa:2048 -sha256 -out ssl_cert.csr -keyout ssl_cert.key; openssl x509 -req -days 365 -sha256 -in ssl_cert.csr -signkey ssl_cert.key -out ssl_cert.pem; cat ssl_cert.key >> ssl_cert.pem
could not start the mongoose web server, see messages and midas.log, bye!

Create a self-signed certificate suitable for initial testing by executing the command printed by mhttpd:


[mhostpc] openssl req -new -nodes -newkey rsa:2048 -sha256 -out ssl_cert.csr -keyout ssl_cert.key; openssl x509 -req -days 365 -sha256 -in ssl_cert.csr -signkey ssl_cert.key -out ssl_cert.pem; cat ssl_cert.key >> ssl_cert.pem

For production use, you should create a properly signed certificate, see [[Mhttpd#Create an SSL certificate|create your own SSL certificate]] or you should run mhttpd behind an SSL proxy.

Run mhttpd again.

You will get the following messages:
[mhttpd,INFO] ODB subtree /Runinfo corrected successfully
Mongoose web server will listen on ports "8080r,8443s"
Mongoose web server will use SSL certificate file "/home/johnfoo/packages/midas/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17633:mongoose,ERROR] mongoose web server cannot find password file "/home/johnfoo/online/htpasswd.txt"
[mhttpd,ERROR] [mhttpd.cxx:17634:mongoose,ERROR] please create password file: htdigest -c /home/johnfoo/online/htpasswd.txt Default midas
could not start the mongoose web server, see messages and midas.log, bye!

Create the password file by following the instructions printed by mhttpd. The http digest domain name is the experiment name, suggested default user name is "midas". You will be asked to type in a password


[mhostpc] htdigest -c /home/johnfoo/online/htpasswd.txt exptname midas
Adding password for midas in realm exptname.
New password:
Re-type new password:

It is a good idea to set the password file {{Filepath|path=htpasswd.txt}} readable and writable by owner only.

Now restart {{Utility|name=mhttpd}}
[mhostpc] mhttpd
Mongoose web server will listen on ports "8080r,8443s" **see note
Mongoose web server will use SSL certificate file "/home/suz/packages/midas/ssl_cert.pem"
Mongoose web server will use authentication realm "Default", password file "./htpasswd.txt"

Now point a web browser running on the same host computer (localhost) to https://localhost:8443
If the web browser is running on a different computer, go to URL of the form

https://mhostpc.triumf.ca:8443 (substitute your host machine name and domain for "mhostpc.triumf.ca")

If you are using the default SSL certificate you will probably get a message: "This Connection is Untrusted". Click "I understand the risks" and add an exception. This is because the test certificate is self-signed. Then confirm an exception.

If instead you get a "connection refused" error, the midas host pc may have the firewall enabled. To make a firewall exception for MIDAS, follow instructions here http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29

You should then see an authentication box asking you for the user name and password. The user name is "midas". Enter the password you just created. The Midas [[Status Page]] should appear with multiple buttons for run control as well as equipment listing (no equipments will be listed as yet) and application listings. Please refer to [[mhttpd]] (the MIDAS Web-based Run Control utility) for further information. You can start and stop runs from the main status page, and use the [[ODB Page]] to access the database (ODB).

; Note
: Default ports of 8080 and 8443 are used by [[mhttpd]]. If these ports are in use on your machine, start mhttpd with alternative ports, e.g.
[mhostpc] mhttpd --https 8448 --http 8089
: or see [[Mhttpd#Usage]] to change the default ports.

== Setup experiment database (ODB) ==

* run $HOME/online/bin/start_daq.sh

* odbedit, run these commands: (replace user names and directory names)
<pre>
set "/Logger/Message file" "/home/kopio03/online/midas.log"
set "/Logger/Data Dir" "/home/kopio03/online/data"
create STRING "/Logger/History dir"
set "/Logger/History dir" "/home/kopio03/online/history"
create STRING "/Logger/Elog dir"
set "/Logger/Elog dir" "/home/kopio03/online/elog"
set "/Logger/ODB dump file" "/home/kopio03/online/history/run%05d.xml"
set "/Logger/ODB dump" "y"
set "/Logger/Channels/0/Settings/Filename" "run%05dsub%03d.mid.gz"
set "/Logger/Channels/0/Settings/Subrun byte limit" "1000000000"
set "/Logger/Channels/0/Settings/Compression" 1
set "/Logger/Channels/0/Settings/ODB Dump" "y"
set "/Programs/Logger/Required" y
set "/Programs/Logger/Start command" "mlogger -D"
set "/Programs/fevme/Required" "y"
set "/Programs/fevme/Start command" "ssh -n lxdaq09 $HOME/online/src/fevme_gef.exe -O"
exit
</pre>
* open web browser e.g. firefox.
* go to the midas status page at https://localhost:8443 (default port).
** if running [[mhttpd]] with Mongoose HTTPS/OpenSSL (the default) for the first time, you will need to create a password file. Follow the instructions (see [[mhttpd#HTTPS/SSL server (Mongoose)]] for details).
** For other options (i.e. HTTPS/SSL proxy) see [[#Secure MIDAS and ELOG Web access]]
*OR open the midas status page at http://localhost:8081 ([[#NOTES|older MIDAS versions]])
* midas status page will show most stuff "red" as nothing is running yet
* DON'T DO THIS YET run ./fevme.exe (on the computer with the VME interface, could be different from computer hosting the experiment), observe that corresponding equipments have been created
* save the url bookmark to the "personal toolbar"
* go to the Programs page, stop mlogger, stop fevme, start mlogger, start fevme
* go to the Status page, start run, stop run
* go back to the Status page, everything should be green
* start a run
* send signals to the ADC gate
* you should be getting events
* to look at data, proceed with setting up the [[ROOTANA|ROOT Analyzer]].

== Start DAQ programs at boot time ==

* add this to /etc/rc.local (replace username and location of the start_daq script)
<pre>
su - alpha -c /home/alpha/online/bin/start_daq.sh
</pre>

== Setup local software version control ==

Version control for experiment source code is setup using "git" (http://git-scm.com/)

* cd $HOME/online
* git init
* git add exptab
* git add bin/start_daq.sh
* git add .gitignore ### contents can be
<pre>
*~
*.o
*.exe
</pre>
* git add src/Makefile src/*.cxx ...
* git commit -a

== Adjust MIDAS buffer sizes ==

Default MIDAS SYSTEM buffer size is 8 Mbytes, fairly small for high-data-rate experiments. The rule of thumb is to have at least a few seconds worth of buffer space available. For example, if event size is 10 Kbytes and the event rate is 1 kHz, data rate is 10*10^3*1*10^3 = 10 Mbytes/sec. To buffer 10 seconds of data we need 100 Mbytes of buffer space.

To resize the MIDAS event buffers (SYSTEM, etc) do this:
* stop all frontends, stop mlogger
* start odbedit:
** cd "/Experiment/Buffer sizes"
** set SYSTEM 100000000
* run "mdump -z SYSTEM"
* if mdump complains about the size of .SYSTEM.SHM, remove it, try again.
* ls -l /dev/shm ### to observe that the size of shared memory is correct

== Secure MIDAS and ELOG Web access ==
In versions prior to May 2015, the default web access to MIDAS and ELOG uses the "http:" protocol which is insecure. In this case, all information is transmitted as clear text meaning that secret, confidential and sensitive information (such as the MIDAS and ELOG passwords and usernames) can be stolen "easily". This means that even "password protected" MIDAS and ELOG pages are not really protected if accessed using the "http" method.

Better security for HTTP is gained by using a password protected '''SSL (https) proxy'''. (It does not provide absolute security because of remaining problems with the security of SSL certificates, security of passwords, etc). Setting up an SSL (https) proxy is described [[#Setting up an HTTP proxy|below]].

Since May 2015, an ''alternative secure option'' to setting up an HTTP proxy is available to users of MIDAS. Recent versions of elogd (ELOG) do support SSL https:// connections, and [[#mhttpd with HTTPS/SSL server (Mongoose)]] is now available. This option is the default, and provides a similar level of security to an HTTP proxy.

See [[Security#Web Access]] for a comparison of these two secure options.

=== mhttpd with HTTPS/SSL server (Mongoose) ===

Since May 2015 the MIDAS web server [[mhttpd]] is explicitly linked with OpenSSL to provide secure HTTPS connections via the [https://bitbucket.org/tmidas/midas/src/ecb9a8537448a8a43f7f9a2bfdb82e578208cde3/doc/mongoose/?at=develop Mongoose] web server (see [[mhttpd]]). With this version, default web access to MIDAS uses the "https" protocol. Web access to {{Utility|name=mhttpd}} can be restricted by using the "-a hostname" switch of [[mhttpd]]. The first time {{Utility|name=mhttpd}} is run, a password file must be created. An SSL certificate is also required. See [[mhttpd#HTTPS/SSL server (Mongoose)|HTTPS/SSL server (Mongoose)]] for instructions.

=== mhttpd using an HTTPS/SSL proxy ===

THESE INSTRUCTIONS ARE WRONG, DO NOT DO THIS. See instead [https://daq00.triumf.ca/MidasWiki/index.php/Setup_MIDAS_experiment_at_TRIUMF#Install_https_proxy this section].

An [[#Setting up an HTTP proxy|HTTP proxy]] must be set up. This is the only way of securing older version of [[mhttpd]] (pre August 2015). Older versions of mhttpd are started using the -p port option e.g.
* mhttpd -D -p 8080 

To run a new version of mhttpd using an HTTP proxy, use the options provided to run the old (non-Mongoose) webserver on a given port, i.e.
* mhttpd --oldserver 8080 --nomg -D 

When using an SSL proxy, only access from the SSL proxy (and maybe some special trusted machines) should be permitted.
This is done using the "-a hostname" switch of [[mhttpd]]. Normally there will be only "-a localhost" switch, enabling access only for the local machine (where the SSL proxy is running). Additional "-a hostname" switches enable access from listed local machines. No "-a xxx" enables access from everywhere (defeating the purpose of the SSL proxy, unless access controls are enforced elsewhere, i.e. by a site firewall or by local firewall rules).

== Setting up an HTTP proxy ==
In this example, we use APACHE HTTPD to password-protect a typical midas/mhttpd and elog installation.

In this configuration, one uses the Linux stock httpd that accepts encrypted https:// connections and forwards them to mhttpd and elogd. Instead of (or in addition to) using mhttpd and elogd passwords, one configures password protection in httpd via the regular apache httpd password mechanisms (htpasswd, etc).

Recent versions of elogd do support SSL https:// connections, but if one is running an SSL proxy for anyway, it is simpler to run both through the same SSL proxy using the same SSL host certificate and the same httpd password file.

=== Restricting http: access to elogd ===
;Note
:Recent versions of elogd do support SSL https:// connections. The following information is for those using an HTTP proxy (see above).

For elogd, this is done using the "-n localhost" switch with enables only access from the same machine if present, or access from anywhere is absent (defeating the purpose of the SSL proxy, unless access controls are enforced elsewhere).

(It is recommended to run elogd from the same user as the main daq user and to keep elogd.cfg and all logbooks in the home directory of this user, where they are captured by the normal site backup system)

== Install standalone elog ==

* login into the user account that will run the elog
* cd $HOME/packages
* git clone https://bitbucket.org/ritt/elog
* cd elog
* make
* create new file start_elogd with this contents:
<pre>
#!/bin/sh

killall elogd
sleep 1
killall -KILL elogd
sleep 1
$HOME/packages/elog/elogd -n localhost -x -c $HOME/packages/elog/elogd.cfg -p 8082 -D

#end
</pre>
* chmod a+x start_elogd
* edit elogd.cfg to read:
<pre>
[global]
port = 8082
SMTP host = smtp.triumf.ca
URL = https://titan00.triumf.ca/elog/

Reverse sort = 1
Display Mode = full

#List Menu commands = New, Find, Admin, Help
#Menu commands = New, Edit, Reply, Find, Duplicate, Help

Entries Per Page = 30
Supress Email on edit = 1
Default encoding = 1
Page title = TITAN ELOG
Resolve host names = 1

Logfile = /home/titan/packages/elog/elogd.log
#Logging level = 3

[midas]

List page Title = T2K M11 MIDAS ELOG
Comment = T2K M11 MIDAS ELOG
Page Title = T2K M11 MIDAS ELOG
RSS Title = [$logbook - $type - $system] $subject, posted by $author

Attributes = Author, Subject, Run, Type, System
Show Attributes Edit = Run, Author, Subject, Type, System
Required Attributes = Author, Type, System, Subject

Options Type = Routine, Reply, Shift Summary, Modification, Question, Info, Problem
Options System = General, DAQ, Beamline

Preset Run = $shell(MIDASSYS=. /home/t2km11/packages/midas/linux/bin/odbedit -d Runinfo -c 'ls -v \"run number\"')

Preset On Reply Type = Reply
Preset On Reply Run = $shell(MIDASSYS=. /home/t2km11/packages/midas/linux/bin/odbedit -d Runinfo -c 'ls -v \"run number\"')

List Display = Date, Subject, Type, System, Author, ID
Quick Filter = Date, Type, ID

Remove on reply = Author
Quote on reply = 1

Use lock = 1

************* Email Functionality ****************

Use Email Subject = [T2KM11 - $System] $Subject
Omit Email To = 1

Email System General = xxx
</pre>
* ./start_elogd &
* firefox http://localhost:8082 # hould show the elog message index

To start elogd automatically when the machine is rebooted, login as root and
* add this text to /etc/rc.local:
<pre>
su - titan -c "/home/titan/packages/elog/start_elogd"
</pre>
* chmod a+x /etc/rc.local
* systemctl start rc-local

To import elog entries from the mhttpd elog, do this:

* cd ~/packages/elog/logbooks
* ln -s /home/t2km11/online/elog midas
* cd midas
* ~/packages/elog/elconv)

== Install https proxy ==

THESE INSTRUCTIONS ARE OBSOLETE, INSTEAD,
* GO HERE: https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Install_apache_httpd_proxy_for_midas_and_elog
* AND GO HERE: https://midas.triumf.ca/MidasWiki/index.php/Quickstart_Linux#Run_the_MIDAS_Web_Server

FOLLOWING INSTRUCTIONS ARE OBSOLETE...

* login as root to the https proxy machine
* cd ~root
* yum install mod_ssl
* yum install crypto-utils # see http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_monitoring_of_HTTPS_certificates
* create a certificate request (replace ladd09 with your hostname): openssl req -new -nodes -newkey rsa:2048 -sha256 -out ladd09.csr -keyout ladd09.key (answer: CA, BC, Vancouver, TRIUMF, DAQ, ladd09.triumf.ca, email@email.com
* sign it by TRIUMF:
** mail -s "Certificate request" yourself@email.com < ladd09.csr
** forward this request to Andrew Daviel
** he will email the signed crt file, copy it to this system as ladd09.crt
* sign it yourself: openssl x509 -req -days 365 -sha256 -in ladd09.csr -signkey ladd09.key -out ladd09.crt
* (if the certificate expires, renew it by signing it again)
* Additional commands for working with certificates:
** explore the private key: openssl pkey -in ladd09.key -text -noout
** explore the certificate request: openssl req -in ladd00.csr -text -noout
** explore the certificate: openssl x509 -in ladd09.crt -noout -text
* move certificate files to proper system locations:
* mv ladd09.key /etc/pki/tls/private/
* mv ladd09.crt /etc/pki/tls/certs/
* if selinux is enabled, do this:
** restorecon -Rv /etc/pki/tls/certs/
** restorecon -Rv /etc/pki/tls/private/
** /usr/sbin/setsebool -P httpd_can_network_connect 1
* open /etc/httpd/conf.d/ssl.conf in a text editor, go to the very bottom and right before the "</VirtualHost>" entry, add following text:
<pre>
...
SSLCertificateFile /etc/pki/tls/certs/ladd09.crt
SSLCertificateKeyFile /etc/pki/tls/private/ladd09.key

ProxyPass /elog/ http://localhost:8082/ retry=1
ProxyPass / http://localhost:8080/ retry=1

<Location />

SSLRequireSSL
AuthType Basic
AuthName "password protected site"
Require valid-user

# create password file: touch /etc/httpd/htpasswd
# to add new user or change password: htpasswd /etc/httpd/htpasswd username
AuthUserFile /etc/httpd/htpasswd

</Location>
</VirtualHost>
...
</pre>
* comment out duplicate "SSLCertificateFile" and "SSLCertificateKeyFile" elsewhere in the file
* (optionally) If you got a certificate that is signed by DigiCert or RapidSSL then you'll need to add a line specifying the certificate chain file:
<pre>
...
SSLCertificateChainFile /etc/pki/tls/certs/DigiCertCA.crt
...
</pre>
* touch /etc/httpd/htpasswd
* htpasswd /etc/httpd/htpasswd midas # enter password midas
* chkconfig httpd on
* service httpd restart
* firewall-cmd --add-port=443/tcp --permanent
* firewall-cmd --reload
* firewall-cmd --list-all
* test it
** test the SSL proxy: https://host/ should yield the midas status page, https://host/elog/ should yield the elog message index
* in ODB, set "/Elog/URL" to "https://host/elog/"
* now from the midas status page, the "Elog" button should take us to the https Elog URL

In needed, enable user directories: https://blah/~user in ~user/public_html

* edit /etc/httpd/conf.d/userdir.conf, replace "UserDir disabled" with "UserDir enabled"
* setsebool -P httpd_enable_homedirs true
* systemctl restart httpd

== Setup the history mhttpd for faster access to history plots ==
When running an SSL proxy,
* start the main mhttpd (orange command for old mhttpd, green for new mhttpd with Mongoose(post August2015):
** "mhttpd -p 8071 -D" or
** "mhttpd -D --oldserver 8071 --nomg" 

* start the history mhttpd
** "mhttpd -p 8072 -D -H" or
** "mhttpd -D -H --oldserver 8072 --nomg" 
* set ODB /History/URL to "http://alphacpc09.cern.ch:8072/HS/"
* open the MIDAS status page
* go to the history section, try to open any history plot, observe that the history plot gif image loads correctly, inspect it's URL (use "copy image URL" or "view source", etc), it should point to port 8072 causing connection to the history mhttpd.
* continue with these instructions to setup history mhttpd access through an SSL proxy:
* setup SSL proxy access (required mod_proxy_html)
** login as root to the SSL proxy machine
** on SL5, install the missing mod_proxy_html httpd module:
** yum install httpd-devel libxml2-devel
** wget http://apache.webthing.com/mod_proxy_html/mod_proxy_html.tar.bz2
** tar xjvf mod_proxy_html.tar.bz2
** cd mod_proxy_html
** apxs -c -I. -I/usr/include/libxml2 -i mod_proxy_html.c
** apxs -c -I. -I/usr/include/libxml2 -i mod_xml2enc.c
** cd /etc/httpd/conf.d, add this to ssl.conf:

;before the ProxyPass statements:
<pre>
# proxy the MIDAS web servers
LoadModule xml2enc_module modules/mod_xml2enc.so
LoadModule proxy_html_module modules/mod_proxy_html.so
ProxyHTMLLinks a href
ProxyHTMLLinks link href
ProxyHTMLLinks img src
#ProxyHTMLEnable On
ProxyRequests off
</pre>
;after the ProxyPass statements:
<pre>
# ALPHA1 history access
ProxyPass /alpha1/history/ http://alphacpc09.cern.ch:8072/HS/ retry=1
ProxyPass /alpha1/ http://alphacpc09.cern.ch:8071/ retry=1

ProxyHTMLEnable On
ProxyHTMLURLMap http://alphacpc09.cern.ch:8072/HS/ /alpha1/history/
</pre>
;adjust:
*"alpha1" is the experiment name
*"alphacpc09.cern.ch" is the machine running mhttpd
*"8071" is the port number of the main mhttpd
** "mhttpd -p 8071 -D" or
** "mhttpd -D --oldserver 8071 --nomg" 
* "8072" is the port number of the history mhttpd
** "mhttpd -p 8072 -D -H" or
** "mhttpd -D -H --oldserver 8072 --nomg" 

[[Category:Installation]] [[Category:Buffer]] [[Category:Driver]]

Setup MIDAS experiment at TRIUMF

2026-02-03T17:30:46Z

Dfujimoto: /* Install https proxy */

{{Pagelinks}}
== Introduction ==
This page describes setting up a MIDAS experiment at TRIUMF. This information can be adapted for other sites.

== Environment variables ==
* '''MIDASSYS''' Base directory of the MIDAS package, midas and mxml should be at the same level.
* '''MIDAS_EXPTAB''' Experiment definition file
* '''MIDAS_SERVER_HOST''' MIDAS host server name for remote midas connections.
* '''MIDAS_EXPT_NAME''' Experiment name

== Standard layout of MIDAS experiment ==
The following shows the directory layout of a standard MIDAS experiment:

/home/exptuser/
packages/
root <---- ROOT
mxml
mscb
midas/ <---- MIDAS
linux/{lib,bin} <---- binaries matching the selected 64-bit/32-bit flavour of ROOT
linux-m32/{lib,bin} <---- limited function 32-bit binaries for 32-bit frontend machines, build by "make linux32"
linux-m64/{lib,bin} <---- limited function 64-bit binaries (only needed if ROOT and linux/bin are 32-bit)
linux-arm/{lib,bin} <---- full function ARM cross-compiled using "make linuxarm"
linux-crosscompile/{lib,bin} <---- cross-compiled limited function binaries for PPC and ARM frontends (see Makefile)
rootana <---- ROOT analyzer
roody <---- graphical online histogram viewer for MIDAS and ROOTANA
online/
exptab <---- experiment definition
{.ODB,.SYSTEM,.SYSMSG,etc}.SHM <---- MIDAS shared memory save files
src <---- experiment frontend sources
bin,scripts
elog <---- MIDAS elog
history <---- MIDAS history
data -> /data/exptname/current <---- symlink to the data directory
/data/exptname/current <---- experiment data directory with ODB save files and MIDAS .mid/.mid.gz data files

== Prepare computers ==

On some operating systems, several MIDAS functions require administrator access:

* on el7 linux (SL7/CC7/CentOS7/RHEL7) - access to mhttpd port 8443 requires special firewall rules, see here: https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29
* on el7 linux - access to mserver to run frontends and other programs on some other computer requires special firewall rules, see here: https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29
* on el7 linux - on the frontend machines (and other machines that will connect to the mserver, the same firewall rule needs to be created (use the IP address of the machine running the mserver)

== Prepare the user account ==
<div id="NOTES"></div>

* Setup the user account for running this instance of midas. For machines part of the LADD cluster, follow these [http://daq-plone.triumf.ca/SM/docs/local/NewLaddUser] instructions.
* check that the account is using the /bin/bash shell
* make $HOME/.profile look like this:

<pre>
#!/bin/echo You must source

export SVN_EDITOR="emacs -nw"
export GIT_EDITOR="emacs -nw"
export MIDASSYS=$HOME/packages/midas
export ROOTANASYS=$HOME/packages/rootana
export MIDAS_EXPTAB=$HOME/online/exptab
#
# setup the MIDAS mserver
#
case `hostname` in
daq07*)
unset MIDAS_SERVER_HOST
;;
*)
export MIDAS_SERVER_HOST=daq07.triumf.ca:7070
;;
esac
#
# select 64-bit or 32-bit MIDAS and ROOT
#
case `uname -i` in
i386)
source /daq/daqshare/olchansk/root/root_v5.34.01_SL62_32/bin/thisroot.sh
export PATH=.:$MIDASSYS/linux-m32/bin:$PATH
;;
*)
#source /daq/daqshare/olchansk/root/root_v5.34.34_SL67_64/bin/thisroot.sh
source $HOME/packages/root/bin/thisroot.sh
export PATH=.:$MIDASSYS/linux/bin:$PATH
;;
esac
#
export PATH=.:$HOME/online/bin:$HOME/packages/roody/bin:$PATH
#
#end
</pre>

* mkdir $HOME/packages
* Logout and login again, for .cshrc changes to take effect

== Install ROOT ==

* Identify the Linux version: RH9 (Red Hat Linux 9), FC3 (Fedora Core 3), RHEL4/SL4 (Red Hat Enterprise LInux 4/Scientific Linux 4), SL5x, SL6x, (CentOS/CC/SL) el7x: more /etc/redhat-release
* Decide to use 32-bit or 64-bit ROOT ('uname -a')
* cd $HOME/packages
* ls -l /daq/daqshare/olchansk/root/ ### to see all available ROOT packages
* ln -s /daq/daqshare/olchansk/root/root_vNNN_VVV_BB root, where NNN is the latest available version of ROOT ("ls -l /daq/daqshare/olchansk/root"), VVV is the Linux version code (RH9, FC3, SL4, etc) and BB is "_32" or "_64" for 32-bit or 64-bit ROOT. For example: /daq/daqshare/olchansk/root/root_v5.10.00_SL40
* for example: ln -s /daq/daqshare/olchansk/root/root_v5.34.34_el72_64 $HOME/packages/root
* Check that ROOT works: "source $HOME/packages/root/bin/thisroot.sh; root". Type ".q" to exit root.

== Install MIDAS ==

* cd $HOME/packages
* (OBSOLETE) svn co svn+ssh://svn@savannah.psi.ch/repos/meg/midas/trunk midas, password "svn". (password has to be entered twice)
* (OBSOLETE) svn co svn+ssh://svn@savannah.psi.ch/repos/meg/mxml/trunk mxml
* git clone https://bitbucket.org/tmidas/midas --recursive
* (OBSOLETE) git clone https://bitbucket.org/tmidas/mxml
* (OBSOLETE) git clone https://bitbucket.org/tmidas/mscb
* (IF BITBUCKET IS DOWN) git clone -v --progress https://daq.triumf.ca/~daqweb/git/mxml.git
* (IF BITBUCKET IS DOWN) git clone -v --progress https://daq.triumf.ca/~daqweb/git/mscb.git
* (IF BITBUCKET IS DOWN) git clone -v --progress https://daq.triumf.ca/~daqweb/git/midas.git
* cd midas
* make
* (only if needed) make linux32 ### build the 32-bit MIDAS libraries
* ls -l linux/bin/odbedit ### check that odbedit has been created (do not run it yet)

You can see a list of other installation problems at [[Common problems & Debugging recipes]].

;NOTE 1
: Optional features in MIDAS can be explicitly disabled if desired when making MIDAS using the NO_xxx feature (NO_ROOT,NO_MYSQL,NO_ODBC,NO_SQLITE,NO_MSCB), e.g. "make NO_ROOT=1" to disable ROOT. These NO_xxx Makefile variables are only used to control autodetection.

;NOTE 2
: Since June 2019 the mxml and mscb packages are submodules of the midas package, so no need to clone them separately. If you have an existing clone of midas but not yet the submodules, you need

$ git submodule update --init --recursive

: To update both midas and the submodules, you need

$ git pull --recurse-submodules

== Install ROOTANA ==

* cd $HOME/packages
* git clone https://bitbucket.org/tmidas/rootana
* cd rootana
* make

== Install ROODY ==

* cd $HOME/packages
* git clone https://bitbucket.org/tmidas/roody
* cd roody
* make
* $HOME/packages/roody/bin/roody, run the program

== Install additional additional packages ==

* cd $HOME/packages
* git clone https://bitbucket.org/ttriumfdaq/vme
* git clone https://bitbucket.org/ttriumfdaq/frontends

== Build special versions of MIDAS ==

Build special versions of MIDAS for the case when some MIDAS programs, such as VME frontends, will run on a different computer that may have a different flavour of operating system, i.e. 32-bit linux or an older version of Scientific Linux.

* login to the computer where the frontends will run and:
* if it is a 32-bit linux: cd $HOME/packages/midas; make linux32
* if it is a 64-bit linux: cd $HOME/packages/midas; make linux64

* login to the host computer to cross-compile ARM code:
* if it is an ARM linux: cd $HOME/packages/midas; make linuxarm # may need to install ARM cross compilers

== Prepare VME hardware ==

Hardware check list:
* VME crate
* VME processor (supported are V77xx, V7805, V7865)
* On all VME modules, set the VME address jumpers as described here: http://daq-plone.triumf.ca/SM/docs/local/vme_jumpers
* run vmescan to confirm correct VME addresses
** cd $HOME/packages
** svn checkout https://ladd00.triumf.ca/svn/daqsvn/trunk/vme
** cd vme
** make
** ./vmescan.exe (or _gef.exe, depending on the VME driver in use)

== Install Universe-II VME driver (V7648, V7750, V7805, V7851) ==

* login as root (ssh root@localhost)
* follow instructions: https://www.triumf.info/wiki/DAQwiki/index.php/VME-CPU#V7648.2C_V7750.2C_V7805.2C_V7851_:_Setup_vme_universe_VME_drivers
* cd ~/packages/vme; vmescan.exe

== Install Tsi-148 VME driver (V7865) ==

* login as root (ssh root@localhost)
* follow instructions: https://www.triumf.info/wiki/DAQwiki/index.php/VME-CPU#V7865_and_XVB-602_:_Setup_gefvme.2Ftsi148_VME_drivers
* cd ~/packages/vme; vmescan_gef.exe

== Setup the experiment environment ==

* Decide which computer will host MIDAS (where MIDAS shared memory buffers will reside).
: This computer will run the [[mserver]], [[mlogger]] and [[mhttpd]] applications. (It is usually the machine where the MIDAS,ROOT etc. packages have been downloaded). It will be referred to as the host machine (localhost).

The environment is slightly different depending on whether all programs run on the host machine, or whether some programs run on remote host(s) :
=== ALL programs run on localhost ===
:If all programs run on the host machine (localhost), it is not necessary to run [[mserver]]. [[Environment Variables#MIDAS_SERVER_HOST|MIDAS_SERVER_HOST]] will not be assigned (see example .cshrc [[#Prepare the user account|above]]).

=== Some programs run on REMOTE host(s) ===
: '''IMPORTANT:'''
# Since August 2015 '''you must explicitly allow access for clients running on remote machines'''. To do this, follow the '''[[Security#MIDAS programs on remote machines|instructions here]]'''.
# The example code .cshrc ([[#Prepare the user account|see above]]) should be present on both host and remote machine(s). This will ensure that [[Environment Variables#MIDAS_SERVER_HOST|MIDAS_SERVER_HOST]] will NOT be set for the host machine (localhost), but on a remote machine, MIDAS_SERVER_HOST will be set to the MIDAS host machine.
# The client [[mserver]] must be started on the MIDAS host machine. Note that multiple experiments can run on the same host machine by starting several instances of [[mserver]] (one for each experiment) running with different ports (and .cshrc would be edited so that MIDAS_SERVER_HOST is set to the appropriate port for the experiment).

 
On the host machine:
* mkdir $HOME/online
* cd $HOME/online
* create directories for local programs, sources, elog and history: mkdir bin src elog history
* create data directory: mkdir -p /ladd/data1/t2kvme5/data; ln -s /ladd/data1/t2kvme5/data $HOME/online
* create the exptab file "$HOME/online/exptab" following the example below. The first entry (exptname) is the name if the DAQ system (MIDAS experiment name), the second entry (/home/USER/online) is the location of MIDAS shared memory buffers (by convention, $HOME/online), the third entry (kopio03) is your username.
<pre>exptname /home/kopio03/online kopio03</pre>
* logout and login again for all changes to take effect

== Setup experiment startup scripts ==
* login to the experiment host computer
* echo $MIDAS_SERVER_HOST ### to check correct value - should be blank
* create $HOME/online/bin/start_daq.sh, replacing XXX with the hostname of the machine running the experiment (and changing the mserver and mhttpd ports, as needed).

#!/bin/sh
# start_daq.sh
cd $HOME/online
#
case `hostname` in XXX*)
echo "Good, we are on XXX!"
;;
*)
echo "The start_daq script should be executed on XXX"
exit 1
;;
esac
#
odbedit -c clean
# start [[mhttpd]] on default port. (Mongoose https version - see [[mhttpd]] for other options)
mhttpd -D -a localhost -a XXX.triumf.ca # optionally restrict access to specified hosts
#
# start [[mserver]] on default port (use argument -p to use a different port)
mserver -D # access must now be specifically allowed - see [[#Setup the experiment environment|above]]

# OR ([[#NOTES|older MIDAS versions]])
# mhttpd -p 8081 -D -a localhost -a XXX.triumf.ca # optionally restrict access to specified hosts
# mserver -p 7071 -D -a localhost -a lxdragon01.triumf.ca -a lxdragon02.triumf.ca -a XXX.triumf.ca # optionally restrict access to specified hosts

#
mlogger -D
#end file

== Run the MIDAS Web Server ==
Let's start the MIDAS webserver for the first time:

Start [[mhttpd]] on the ''experiment host'' (localhost) like this:
[mhostpc] mhttpd
You will get the following messages:
[mhttpd,INFO] ODB subtree /Runinfo corrected successfully
Mongoose web server will listen on ports "8080r,8443s"
[mhttpd,ERROR] [mhttpd.cxx:17892:mongoose,ERROR] cannot find SSL certificate file "/home/agdaq/online/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17893:mongoose,ERROR] please create SSL certificate file: openssl req -new -nodes -newkey rsa:2048 -sha256 -out ssl_cert.csr -keyout ssl_cert.key; openssl x509 -req -days 365 -sha256 -in ssl_cert.csr -signkey ssl_cert.key -out ssl_cert.pem; cat ssl_cert.key >> ssl_cert.pem
could not start the mongoose web server, see messages and midas.log, bye!

Create a self-signed certificate suitable for initial testing by executing the command printed by mhttpd:


[mhostpc] openssl req -new -nodes -newkey rsa:2048 -sha256 -out ssl_cert.csr -keyout ssl_cert.key; openssl x509 -req -days 365 -sha256 -in ssl_cert.csr -signkey ssl_cert.key -out ssl_cert.pem; cat ssl_cert.key >> ssl_cert.pem

For production use, you should create a properly signed certificate, see [[Mhttpd#Create an SSL certificate|create your own SSL certificate]] or you should run mhttpd behind an SSL proxy.

Run mhttpd again.

You will get the following messages:
[mhttpd,INFO] ODB subtree /Runinfo corrected successfully
Mongoose web server will listen on ports "8080r,8443s"
Mongoose web server will use SSL certificate file "/home/johnfoo/packages/midas/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17633:mongoose,ERROR] mongoose web server cannot find password file "/home/johnfoo/online/htpasswd.txt"
[mhttpd,ERROR] [mhttpd.cxx:17634:mongoose,ERROR] please create password file: htdigest -c /home/johnfoo/online/htpasswd.txt Default midas
could not start the mongoose web server, see messages and midas.log, bye!

Create the password file by following the instructions printed by mhttpd. The http digest domain name is the experiment name, suggested default user name is "midas". You will be asked to type in a password


[mhostpc] htdigest -c /home/johnfoo/online/htpasswd.txt exptname midas
Adding password for midas in realm exptname.
New password:
Re-type new password:

It is a good idea to set the password file {{Filepath|path=htpasswd.txt}} readable and writable by owner only.

Now restart {{Utility|name=mhttpd}}
[mhostpc] mhttpd
Mongoose web server will listen on ports "8080r,8443s" **see note
Mongoose web server will use SSL certificate file "/home/suz/packages/midas/ssl_cert.pem"
Mongoose web server will use authentication realm "Default", password file "./htpasswd.txt"

Now point a web browser running on the same host computer (localhost) to https://localhost:8443
If the web browser is running on a different computer, go to URL of the form

https://mhostpc.triumf.ca:8443 (substitute your host machine name and domain for "mhostpc.triumf.ca")

If you are using the default SSL certificate you will probably get a message: "This Connection is Untrusted". Click "I understand the risks" and add an exception. This is because the test certificate is self-signed. Then confirm an exception.

If instead you get a "connection refused" error, the midas host pc may have the firewall enabled. To make a firewall exception for MIDAS, follow instructions here http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29

You should then see an authentication box asking you for the user name and password. The user name is "midas". Enter the password you just created. The Midas [[Status Page]] should appear with multiple buttons for run control as well as equipment listing (no equipments will be listed as yet) and application listings. Please refer to [[mhttpd]] (the MIDAS Web-based Run Control utility) for further information. You can start and stop runs from the main status page, and use the [[ODB Page]] to access the database (ODB).

; Note
: Default ports of 8080 and 8443 are used by [[mhttpd]]. If these ports are in use on your machine, start mhttpd with alternative ports, e.g.
[mhostpc] mhttpd --https 8448 --http 8089
: or see [[Mhttpd#Usage]] to change the default ports.

== Setup experiment database (ODB) ==

* run $HOME/online/bin/start_daq.sh

* odbedit, run these commands: (replace user names and directory names)
<pre>
set "/Logger/Message file" "/home/kopio03/online/midas.log"
set "/Logger/Data Dir" "/home/kopio03/online/data"
create STRING "/Logger/History dir"
set "/Logger/History dir" "/home/kopio03/online/history"
create STRING "/Logger/Elog dir"
set "/Logger/Elog dir" "/home/kopio03/online/elog"
set "/Logger/ODB dump file" "/home/kopio03/online/history/run%05d.xml"
set "/Logger/ODB dump" "y"
set "/Logger/Channels/0/Settings/Filename" "run%05dsub%03d.mid.gz"
set "/Logger/Channels/0/Settings/Subrun byte limit" "1000000000"
set "/Logger/Channels/0/Settings/Compression" 1
set "/Logger/Channels/0/Settings/ODB Dump" "y"
set "/Programs/Logger/Required" y
set "/Programs/Logger/Start command" "mlogger -D"
set "/Programs/fevme/Required" "y"
set "/Programs/fevme/Start command" "ssh -n lxdaq09 $HOME/online/src/fevme_gef.exe -O"
exit
</pre>
* open web browser e.g. firefox.
* go to the midas status page at https://localhost:8443 (default port).
** if running [[mhttpd]] with Mongoose HTTPS/OpenSSL (the default) for the first time, you will need to create a password file. Follow the instructions (see [[mhttpd#HTTPS/SSL server (Mongoose)]] for details).
** For other options (i.e. HTTPS/SSL proxy) see [[#Secure MIDAS and ELOG Web access]]
*OR open the midas status page at http://localhost:8081 ([[#NOTES|older MIDAS versions]])
* midas status page will show most stuff "red" as nothing is running yet
* DON'T DO THIS YET run ./fevme.exe (on the computer with the VME interface, could be different from computer hosting the experiment), observe that corresponding equipments have been created
* save the url bookmark to the "personal toolbar"
* go to the Programs page, stop mlogger, stop fevme, start mlogger, start fevme
* go to the Status page, start run, stop run
* go back to the Status page, everything should be green
* start a run
* send signals to the ADC gate
* you should be getting events
* to look at data, proceed with setting up the [[ROOTANA|ROOT Analyzer]].

== Start DAQ programs at boot time ==

* add this to /etc/rc.local (replace username and location of the start_daq script)
<pre>
su - alpha -c /home/alpha/online/bin/start_daq.sh
</pre>

== Setup local software version control ==

Version control for experiment source code is setup using "git" (http://git-scm.com/)

* cd $HOME/online
* git init
* git add exptab
* git add bin/start_daq.sh
* git add .gitignore ### contents can be
<pre>
*~
*.o
*.exe
</pre>
* git add src/Makefile src/*.cxx ...
* git commit -a

== Adjust MIDAS buffer sizes ==

Default MIDAS SYSTEM buffer size is 8 Mbytes, fairly small for high-data-rate experiments. The rule of thumb is to have at least a few seconds worth of buffer space available. For example, if event size is 10 Kbytes and the event rate is 1 kHz, data rate is 10*10^3*1*10^3 = 10 Mbytes/sec. To buffer 10 seconds of data we need 100 Mbytes of buffer space.

To resize the MIDAS event buffers (SYSTEM, etc) do this:
* stop all frontends, stop mlogger
* start odbedit:
** cd "/Experiment/Buffer sizes"
** set SYSTEM 100000000
* run "mdump -z SYSTEM"
* if mdump complains about the size of .SYSTEM.SHM, remove it, try again.
* ls -l /dev/shm ### to observe that the size of shared memory is correct

== Secure MIDAS and ELOG Web access ==
In versions prior to May 2015, the default web access to MIDAS and ELOG uses the "http:" protocol which is insecure. In this case, all information is transmitted as clear text meaning that secret, confidential and sensitive information (such as the MIDAS and ELOG passwords and usernames) can be stolen "easily". This means that even "password protected" MIDAS and ELOG pages are not really protected if accessed using the "http" method.

Better security for HTTP is gained by using a password protected '''SSL (https) proxy'''. (It does not provide absolute security because of remaining problems with the security of SSL certificates, security of passwords, etc). Setting up an SSL (https) proxy is described [[#Setting up an HTTP proxy|below]].

Since May 2015, an ''alternative secure option'' to setting up an HTTP proxy is available to users of MIDAS. Recent versions of elogd (ELOG) do support SSL https:// connections, and [[#mhttpd with HTTPS/SSL server (Mongoose)]] is now available. This option is the default, and provides a similar level of security to an HTTP proxy.

See [[Security#Web Access]] for a comparison of these two secure options.

=== mhttpd with HTTPS/SSL server (Mongoose) ===

Since May 2015 the MIDAS web server [[mhttpd]] is explicitly linked with OpenSSL to provide secure HTTPS connections via the [https://bitbucket.org/tmidas/midas/src/ecb9a8537448a8a43f7f9a2bfdb82e578208cde3/doc/mongoose/?at=develop Mongoose] web server (see [[mhttpd]]). With this version, default web access to MIDAS uses the "https" protocol. Web access to {{Utility|name=mhttpd}} can be restricted by using the "-a hostname" switch of [[mhttpd]]. The first time {{Utility|name=mhttpd}} is run, a password file must be created. An SSL certificate is also required. See [[mhttpd#HTTPS/SSL server (Mongoose)|HTTPS/SSL server (Mongoose)]] for instructions.

=== mhttpd using an HTTPS/SSL proxy ===

THESE INSTRUCTIONS ARE WRONG, DO NOT DO THIS. See instead the [https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Install_apache_httpd_proxy_for_midas_and_elog TRIUMF DAQ site].

An [[#Setting up an HTTP proxy|HTTP proxy]] must be set up. This is the only way of securing older version of [[mhttpd]] (pre August 2015). Older versions of mhttpd are started using the -p port option e.g.
* mhttpd -D -p 8080 

To run a new version of mhttpd using an HTTP proxy, use the options provided to run the old (non-Mongoose) webserver on a given port, i.e.
* mhttpd --oldserver 8080 --nomg -D 

When using an SSL proxy, only access from the SSL proxy (and maybe some special trusted machines) should be permitted.
This is done using the "-a hostname" switch of [[mhttpd]]. Normally there will be only "-a localhost" switch, enabling access only for the local machine (where the SSL proxy is running). Additional "-a hostname" switches enable access from listed local machines. No "-a xxx" enables access from everywhere (defeating the purpose of the SSL proxy, unless access controls are enforced elsewhere, i.e. by a site firewall or by local firewall rules).

== Setting up an HTTP proxy ==
In this example, we use APACHE HTTPD to password-protect a typical midas/mhttpd and elog installation.

In this configuration, one uses the Linux stock httpd that accepts encrypted https:// connections and forwards them to mhttpd and elogd. Instead of (or in addition to) using mhttpd and elogd passwords, one configures password protection in httpd via the regular apache httpd password mechanisms (htpasswd, etc).

Recent versions of elogd do support SSL https:// connections, but if one is running an SSL proxy for anyway, it is simpler to run both through the same SSL proxy using the same SSL host certificate and the same httpd password file.

=== Restricting http: access to elogd ===
;Note
:Recent versions of elogd do support SSL https:// connections. The following information is for those using an HTTP proxy (see above).

For elogd, this is done using the "-n localhost" switch with enables only access from the same machine if present, or access from anywhere is absent (defeating the purpose of the SSL proxy, unless access controls are enforced elsewhere).

(It is recommended to run elogd from the same user as the main daq user and to keep elogd.cfg and all logbooks in the home directory of this user, where they are captured by the normal site backup system)

== Install standalone elog ==

* login into the user account that will run the elog
* cd $HOME/packages
* git clone https://bitbucket.org/ritt/elog
* cd elog
* make
* create new file start_elogd with this contents:
<pre>
#!/bin/sh

killall elogd
sleep 1
killall -KILL elogd
sleep 1
$HOME/packages/elog/elogd -n localhost -x -c $HOME/packages/elog/elogd.cfg -p 8082 -D

#end
</pre>
* chmod a+x start_elogd
* edit elogd.cfg to read:
<pre>
[global]
port = 8082
SMTP host = smtp.triumf.ca
URL = https://titan00.triumf.ca/elog/

Reverse sort = 1
Display Mode = full

#List Menu commands = New, Find, Admin, Help
#Menu commands = New, Edit, Reply, Find, Duplicate, Help

Entries Per Page = 30
Supress Email on edit = 1
Default encoding = 1
Page title = TITAN ELOG
Resolve host names = 1

Logfile = /home/titan/packages/elog/elogd.log
#Logging level = 3

[midas]

List page Title = T2K M11 MIDAS ELOG
Comment = T2K M11 MIDAS ELOG
Page Title = T2K M11 MIDAS ELOG
RSS Title = [$logbook - $type - $system] $subject, posted by $author

Attributes = Author, Subject, Run, Type, System
Show Attributes Edit = Run, Author, Subject, Type, System
Required Attributes = Author, Type, System, Subject

Options Type = Routine, Reply, Shift Summary, Modification, Question, Info, Problem
Options System = General, DAQ, Beamline

Preset Run = $shell(MIDASSYS=. /home/t2km11/packages/midas/linux/bin/odbedit -d Runinfo -c 'ls -v \"run number\"')

Preset On Reply Type = Reply
Preset On Reply Run = $shell(MIDASSYS=. /home/t2km11/packages/midas/linux/bin/odbedit -d Runinfo -c 'ls -v \"run number\"')

List Display = Date, Subject, Type, System, Author, ID
Quick Filter = Date, Type, ID

Remove on reply = Author
Quote on reply = 1

Use lock = 1

************* Email Functionality ****************

Use Email Subject = [T2KM11 - $System] $Subject
Omit Email To = 1

Email System General = xxx
</pre>
* ./start_elogd &
* firefox http://localhost:8082 # hould show the elog message index

To start elogd automatically when the machine is rebooted, login as root and
* add this text to /etc/rc.local:
<pre>
su - titan -c "/home/titan/packages/elog/start_elogd"
</pre>
* chmod a+x /etc/rc.local
* systemctl start rc-local

To import elog entries from the mhttpd elog, do this:

* cd ~/packages/elog/logbooks
* ln -s /home/t2km11/online/elog midas
* cd midas
* ~/packages/elog/elconv)

== Install https proxy ==

THESE INSTRUCTIONS ARE OBSOLETE, INSTEAD,
* GO HERE: https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Install_apache_httpd_proxy_for_midas_and_elog
* AND GO HERE: https://midas.triumf.ca/MidasWiki/index.php/Quickstart_Linux#Run_the_MIDAS_Web_Server

FOLLOWING INSTRUCTIONS ARE OBSOLETE...

* login as root to the https proxy machine
* cd ~root
* yum install mod_ssl
* yum install crypto-utils # see http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_monitoring_of_HTTPS_certificates
* create a certificate request (replace ladd09 with your hostname): openssl req -new -nodes -newkey rsa:2048 -sha256 -out ladd09.csr -keyout ladd09.key (answer: CA, BC, Vancouver, TRIUMF, DAQ, ladd09.triumf.ca, email@email.com
* sign it by TRIUMF:
** mail -s "Certificate request" yourself@email.com < ladd09.csr
** forward this request to Andrew Daviel
** he will email the signed crt file, copy it to this system as ladd09.crt
* sign it yourself: openssl x509 -req -days 365 -sha256 -in ladd09.csr -signkey ladd09.key -out ladd09.crt
* (if the certificate expires, renew it by signing it again)
* Additional commands for working with certificates:
** explore the private key: openssl pkey -in ladd09.key -text -noout
** explore the certificate request: openssl req -in ladd00.csr -text -noout
** explore the certificate: openssl x509 -in ladd09.crt -noout -text
* move certificate files to proper system locations:
* mv ladd09.key /etc/pki/tls/private/
* mv ladd09.crt /etc/pki/tls/certs/
* if selinux is enabled, do this:
** restorecon -Rv /etc/pki/tls/certs/
** restorecon -Rv /etc/pki/tls/private/
** /usr/sbin/setsebool -P httpd_can_network_connect 1
* open /etc/httpd/conf.d/ssl.conf in a text editor, go to the very bottom and right before the "</VirtualHost>" entry, add following text:
<pre>
...
SSLCertificateFile /etc/pki/tls/certs/ladd09.crt
SSLCertificateKeyFile /etc/pki/tls/private/ladd09.key

ProxyPass /elog/ http://localhost:8082/ retry=1
ProxyPass / http://localhost:8080/ retry=1

<Location />

SSLRequireSSL
AuthType Basic
AuthName "password protected site"
Require valid-user

# create password file: touch /etc/httpd/htpasswd
# to add new user or change password: htpasswd /etc/httpd/htpasswd username
AuthUserFile /etc/httpd/htpasswd

</Location>
</VirtualHost>
...
</pre>
* comment out duplicate "SSLCertificateFile" and "SSLCertificateKeyFile" elsewhere in the file
* (optionally) If you got a certificate that is signed by DigiCert or RapidSSL then you'll need to add a line specifying the certificate chain file:
<pre>
...
SSLCertificateChainFile /etc/pki/tls/certs/DigiCertCA.crt
...
</pre>
* touch /etc/httpd/htpasswd
* htpasswd /etc/httpd/htpasswd midas # enter password midas
* chkconfig httpd on
* service httpd restart
* firewall-cmd --add-port=443/tcp --permanent
* firewall-cmd --reload
* firewall-cmd --list-all
* test it
** test the SSL proxy: https://host/ should yield the midas status page, https://host/elog/ should yield the elog message index
* in ODB, set "/Elog/URL" to "https://host/elog/"
* now from the midas status page, the "Elog" button should take us to the https Elog URL

In needed, enable user directories: https://blah/~user in ~user/public_html

* edit /etc/httpd/conf.d/userdir.conf, replace "UserDir disabled" with "UserDir enabled"
* setsebool -P httpd_enable_homedirs true
* systemctl restart httpd

== Setup the history mhttpd for faster access to history plots ==
When running an SSL proxy,
* start the main mhttpd (orange command for old mhttpd, green for new mhttpd with Mongoose(post August2015):
** "mhttpd -p 8071 -D" or
** "mhttpd -D --oldserver 8071 --nomg" 

* start the history mhttpd
** "mhttpd -p 8072 -D -H" or
** "mhttpd -D -H --oldserver 8072 --nomg" 
* set ODB /History/URL to "http://alphacpc09.cern.ch:8072/HS/"
* open the MIDAS status page
* go to the history section, try to open any history plot, observe that the history plot gif image loads correctly, inspect it's URL (use "copy image URL" or "view source", etc), it should point to port 8072 causing connection to the history mhttpd.
* continue with these instructions to setup history mhttpd access through an SSL proxy:
* setup SSL proxy access (required mod_proxy_html)
** login as root to the SSL proxy machine
** on SL5, install the missing mod_proxy_html httpd module:
** yum install httpd-devel libxml2-devel
** wget http://apache.webthing.com/mod_proxy_html/mod_proxy_html.tar.bz2
** tar xjvf mod_proxy_html.tar.bz2
** cd mod_proxy_html
** apxs -c -I. -I/usr/include/libxml2 -i mod_proxy_html.c
** apxs -c -I. -I/usr/include/libxml2 -i mod_xml2enc.c
** cd /etc/httpd/conf.d, add this to ssl.conf:

;before the ProxyPass statements:
<pre>
# proxy the MIDAS web servers
LoadModule xml2enc_module modules/mod_xml2enc.so
LoadModule proxy_html_module modules/mod_proxy_html.so
ProxyHTMLLinks a href
ProxyHTMLLinks link href
ProxyHTMLLinks img src
#ProxyHTMLEnable On
ProxyRequests off
</pre>
;after the ProxyPass statements:
<pre>
# ALPHA1 history access
ProxyPass /alpha1/history/ http://alphacpc09.cern.ch:8072/HS/ retry=1
ProxyPass /alpha1/ http://alphacpc09.cern.ch:8071/ retry=1

ProxyHTMLEnable On
ProxyHTMLURLMap http://alphacpc09.cern.ch:8072/HS/ /alpha1/history/
</pre>
;adjust:
*"alpha1" is the experiment name
*"alphacpc09.cern.ch" is the machine running mhttpd
*"8071" is the port number of the main mhttpd
** "mhttpd -p 8071 -D" or
** "mhttpd -D --oldserver 8071 --nomg" 
* "8072" is the port number of the history mhttpd
** "mhttpd -p 8072 -D -H" or
** "mhttpd -D -H --oldserver 8072 --nomg" 

[[Category:Installation]] [[Category:Buffer]] [[Category:Driver]]

Setup MIDAS experiment at TRIUMF

2026-02-03T17:30:09Z

Dfujimoto: /* Secure MIDAS and ELOG Web access */

{{Pagelinks}}
== Introduction ==
This page describes setting up a MIDAS experiment at TRIUMF. This information can be adapted for other sites.

== Environment variables ==
* '''MIDASSYS''' Base directory of the MIDAS package, midas and mxml should be at the same level.
* '''MIDAS_EXPTAB''' Experiment definition file
* '''MIDAS_SERVER_HOST''' MIDAS host server name for remote midas connections.
* '''MIDAS_EXPT_NAME''' Experiment name

== Standard layout of MIDAS experiment ==
The following shows the directory layout of a standard MIDAS experiment:

/home/exptuser/
packages/
root <---- ROOT
mxml
mscb
midas/ <---- MIDAS
linux/{lib,bin} <---- binaries matching the selected 64-bit/32-bit flavour of ROOT
linux-m32/{lib,bin} <---- limited function 32-bit binaries for 32-bit frontend machines, build by "make linux32"
linux-m64/{lib,bin} <---- limited function 64-bit binaries (only needed if ROOT and linux/bin are 32-bit)
linux-arm/{lib,bin} <---- full function ARM cross-compiled using "make linuxarm"
linux-crosscompile/{lib,bin} <---- cross-compiled limited function binaries for PPC and ARM frontends (see Makefile)
rootana <---- ROOT analyzer
roody <---- graphical online histogram viewer for MIDAS and ROOTANA
online/
exptab <---- experiment definition
{.ODB,.SYSTEM,.SYSMSG,etc}.SHM <---- MIDAS shared memory save files
src <---- experiment frontend sources
bin,scripts
elog <---- MIDAS elog
history <---- MIDAS history
data -> /data/exptname/current <---- symlink to the data directory
/data/exptname/current <---- experiment data directory with ODB save files and MIDAS .mid/.mid.gz data files

== Prepare computers ==

On some operating systems, several MIDAS functions require administrator access:

* on el7 linux (SL7/CC7/CentOS7/RHEL7) - access to mhttpd port 8443 requires special firewall rules, see here: https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29
* on el7 linux - access to mserver to run frontends and other programs on some other computer requires special firewall rules, see here: https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29
* on el7 linux - on the frontend machines (and other machines that will connect to the mserver, the same firewall rule needs to be created (use the IP address of the machine running the mserver)

== Prepare the user account ==
<div id="NOTES"></div>

* Setup the user account for running this instance of midas. For machines part of the LADD cluster, follow these [http://daq-plone.triumf.ca/SM/docs/local/NewLaddUser] instructions.
* check that the account is using the /bin/bash shell
* make $HOME/.profile look like this:

<pre>
#!/bin/echo You must source

export SVN_EDITOR="emacs -nw"
export GIT_EDITOR="emacs -nw"
export MIDASSYS=$HOME/packages/midas
export ROOTANASYS=$HOME/packages/rootana
export MIDAS_EXPTAB=$HOME/online/exptab
#
# setup the MIDAS mserver
#
case `hostname` in
daq07*)
unset MIDAS_SERVER_HOST
;;
*)
export MIDAS_SERVER_HOST=daq07.triumf.ca:7070
;;
esac
#
# select 64-bit or 32-bit MIDAS and ROOT
#
case `uname -i` in
i386)
source /daq/daqshare/olchansk/root/root_v5.34.01_SL62_32/bin/thisroot.sh
export PATH=.:$MIDASSYS/linux-m32/bin:$PATH
;;
*)
#source /daq/daqshare/olchansk/root/root_v5.34.34_SL67_64/bin/thisroot.sh
source $HOME/packages/root/bin/thisroot.sh
export PATH=.:$MIDASSYS/linux/bin:$PATH
;;
esac
#
export PATH=.:$HOME/online/bin:$HOME/packages/roody/bin:$PATH
#
#end
</pre>

* mkdir $HOME/packages
* Logout and login again, for .cshrc changes to take effect

== Install ROOT ==

* Identify the Linux version: RH9 (Red Hat Linux 9), FC3 (Fedora Core 3), RHEL4/SL4 (Red Hat Enterprise LInux 4/Scientific Linux 4), SL5x, SL6x, (CentOS/CC/SL) el7x: more /etc/redhat-release
* Decide to use 32-bit or 64-bit ROOT ('uname -a')
* cd $HOME/packages
* ls -l /daq/daqshare/olchansk/root/ ### to see all available ROOT packages
* ln -s /daq/daqshare/olchansk/root/root_vNNN_VVV_BB root, where NNN is the latest available version of ROOT ("ls -l /daq/daqshare/olchansk/root"), VVV is the Linux version code (RH9, FC3, SL4, etc) and BB is "_32" or "_64" for 32-bit or 64-bit ROOT. For example: /daq/daqshare/olchansk/root/root_v5.10.00_SL40
* for example: ln -s /daq/daqshare/olchansk/root/root_v5.34.34_el72_64 $HOME/packages/root
* Check that ROOT works: "source $HOME/packages/root/bin/thisroot.sh; root". Type ".q" to exit root.

== Install MIDAS ==

* cd $HOME/packages
* (OBSOLETE) svn co svn+ssh://svn@savannah.psi.ch/repos/meg/midas/trunk midas, password "svn". (password has to be entered twice)
* (OBSOLETE) svn co svn+ssh://svn@savannah.psi.ch/repos/meg/mxml/trunk mxml
* git clone https://bitbucket.org/tmidas/midas --recursive
* (OBSOLETE) git clone https://bitbucket.org/tmidas/mxml
* (OBSOLETE) git clone https://bitbucket.org/tmidas/mscb
* (IF BITBUCKET IS DOWN) git clone -v --progress https://daq.triumf.ca/~daqweb/git/mxml.git
* (IF BITBUCKET IS DOWN) git clone -v --progress https://daq.triumf.ca/~daqweb/git/mscb.git
* (IF BITBUCKET IS DOWN) git clone -v --progress https://daq.triumf.ca/~daqweb/git/midas.git
* cd midas
* make
* (only if needed) make linux32 ### build the 32-bit MIDAS libraries
* ls -l linux/bin/odbedit ### check that odbedit has been created (do not run it yet)

You can see a list of other installation problems at [[Common problems & Debugging recipes]].

;NOTE 1
: Optional features in MIDAS can be explicitly disabled if desired when making MIDAS using the NO_xxx feature (NO_ROOT,NO_MYSQL,NO_ODBC,NO_SQLITE,NO_MSCB), e.g. "make NO_ROOT=1" to disable ROOT. These NO_xxx Makefile variables are only used to control autodetection.

;NOTE 2
: Since June 2019 the mxml and mscb packages are submodules of the midas package, so no need to clone them separately. If you have an existing clone of midas but not yet the submodules, you need

$ git submodule update --init --recursive

: To update both midas and the submodules, you need

$ git pull --recurse-submodules

== Install ROOTANA ==

* cd $HOME/packages
* git clone https://bitbucket.org/tmidas/rootana
* cd rootana
* make

== Install ROODY ==

* cd $HOME/packages
* git clone https://bitbucket.org/tmidas/roody
* cd roody
* make
* $HOME/packages/roody/bin/roody, run the program

== Install additional additional packages ==

* cd $HOME/packages
* git clone https://bitbucket.org/ttriumfdaq/vme
* git clone https://bitbucket.org/ttriumfdaq/frontends

== Build special versions of MIDAS ==

Build special versions of MIDAS for the case when some MIDAS programs, such as VME frontends, will run on a different computer that may have a different flavour of operating system, i.e. 32-bit linux or an older version of Scientific Linux.

* login to the computer where the frontends will run and:
* if it is a 32-bit linux: cd $HOME/packages/midas; make linux32
* if it is a 64-bit linux: cd $HOME/packages/midas; make linux64

* login to the host computer to cross-compile ARM code:
* if it is an ARM linux: cd $HOME/packages/midas; make linuxarm # may need to install ARM cross compilers

== Prepare VME hardware ==

Hardware check list:
* VME crate
* VME processor (supported are V77xx, V7805, V7865)
* On all VME modules, set the VME address jumpers as described here: http://daq-plone.triumf.ca/SM/docs/local/vme_jumpers
* run vmescan to confirm correct VME addresses
** cd $HOME/packages
** svn checkout https://ladd00.triumf.ca/svn/daqsvn/trunk/vme
** cd vme
** make
** ./vmescan.exe (or _gef.exe, depending on the VME driver in use)

== Install Universe-II VME driver (V7648, V7750, V7805, V7851) ==

* login as root (ssh root@localhost)
* follow instructions: https://www.triumf.info/wiki/DAQwiki/index.php/VME-CPU#V7648.2C_V7750.2C_V7805.2C_V7851_:_Setup_vme_universe_VME_drivers
* cd ~/packages/vme; vmescan.exe

== Install Tsi-148 VME driver (V7865) ==

* login as root (ssh root@localhost)
* follow instructions: https://www.triumf.info/wiki/DAQwiki/index.php/VME-CPU#V7865_and_XVB-602_:_Setup_gefvme.2Ftsi148_VME_drivers
* cd ~/packages/vme; vmescan_gef.exe

== Setup the experiment environment ==

* Decide which computer will host MIDAS (where MIDAS shared memory buffers will reside).
: This computer will run the [[mserver]], [[mlogger]] and [[mhttpd]] applications. (It is usually the machine where the MIDAS,ROOT etc. packages have been downloaded). It will be referred to as the host machine (localhost).

The environment is slightly different depending on whether all programs run on the host machine, or whether some programs run on remote host(s) :
=== ALL programs run on localhost ===
:If all programs run on the host machine (localhost), it is not necessary to run [[mserver]]. [[Environment Variables#MIDAS_SERVER_HOST|MIDAS_SERVER_HOST]] will not be assigned (see example .cshrc [[#Prepare the user account|above]]).

=== Some programs run on REMOTE host(s) ===
: '''IMPORTANT:'''
# Since August 2015 '''you must explicitly allow access for clients running on remote machines'''. To do this, follow the '''[[Security#MIDAS programs on remote machines|instructions here]]'''.
# The example code .cshrc ([[#Prepare the user account|see above]]) should be present on both host and remote machine(s). This will ensure that [[Environment Variables#MIDAS_SERVER_HOST|MIDAS_SERVER_HOST]] will NOT be set for the host machine (localhost), but on a remote machine, MIDAS_SERVER_HOST will be set to the MIDAS host machine.
# The client [[mserver]] must be started on the MIDAS host machine. Note that multiple experiments can run on the same host machine by starting several instances of [[mserver]] (one for each experiment) running with different ports (and .cshrc would be edited so that MIDAS_SERVER_HOST is set to the appropriate port for the experiment).

 
On the host machine:
* mkdir $HOME/online
* cd $HOME/online
* create directories for local programs, sources, elog and history: mkdir bin src elog history
* create data directory: mkdir -p /ladd/data1/t2kvme5/data; ln -s /ladd/data1/t2kvme5/data $HOME/online
* create the exptab file "$HOME/online/exptab" following the example below. The first entry (exptname) is the name if the DAQ system (MIDAS experiment name), the second entry (/home/USER/online) is the location of MIDAS shared memory buffers (by convention, $HOME/online), the third entry (kopio03) is your username.
<pre>exptname /home/kopio03/online kopio03</pre>
* logout and login again for all changes to take effect

== Setup experiment startup scripts ==
* login to the experiment host computer
* echo $MIDAS_SERVER_HOST ### to check correct value - should be blank
* create $HOME/online/bin/start_daq.sh, replacing XXX with the hostname of the machine running the experiment (and changing the mserver and mhttpd ports, as needed).

#!/bin/sh
# start_daq.sh
cd $HOME/online
#
case `hostname` in XXX*)
echo "Good, we are on XXX!"
;;
*)
echo "The start_daq script should be executed on XXX"
exit 1
;;
esac
#
odbedit -c clean
# start [[mhttpd]] on default port. (Mongoose https version - see [[mhttpd]] for other options)
mhttpd -D -a localhost -a XXX.triumf.ca # optionally restrict access to specified hosts
#
# start [[mserver]] on default port (use argument -p to use a different port)
mserver -D # access must now be specifically allowed - see [[#Setup the experiment environment|above]]

# OR ([[#NOTES|older MIDAS versions]])
# mhttpd -p 8081 -D -a localhost -a XXX.triumf.ca # optionally restrict access to specified hosts
# mserver -p 7071 -D -a localhost -a lxdragon01.triumf.ca -a lxdragon02.triumf.ca -a XXX.triumf.ca # optionally restrict access to specified hosts

#
mlogger -D
#end file

== Run the MIDAS Web Server ==
Let's start the MIDAS webserver for the first time:

Start [[mhttpd]] on the ''experiment host'' (localhost) like this:
[mhostpc] mhttpd
You will get the following messages:
[mhttpd,INFO] ODB subtree /Runinfo corrected successfully
Mongoose web server will listen on ports "8080r,8443s"
[mhttpd,ERROR] [mhttpd.cxx:17892:mongoose,ERROR] cannot find SSL certificate file "/home/agdaq/online/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17893:mongoose,ERROR] please create SSL certificate file: openssl req -new -nodes -newkey rsa:2048 -sha256 -out ssl_cert.csr -keyout ssl_cert.key; openssl x509 -req -days 365 -sha256 -in ssl_cert.csr -signkey ssl_cert.key -out ssl_cert.pem; cat ssl_cert.key >> ssl_cert.pem
could not start the mongoose web server, see messages and midas.log, bye!

Create a self-signed certificate suitable for initial testing by executing the command printed by mhttpd:


[mhostpc] openssl req -new -nodes -newkey rsa:2048 -sha256 -out ssl_cert.csr -keyout ssl_cert.key; openssl x509 -req -days 365 -sha256 -in ssl_cert.csr -signkey ssl_cert.key -out ssl_cert.pem; cat ssl_cert.key >> ssl_cert.pem

For production use, you should create a properly signed certificate, see [[Mhttpd#Create an SSL certificate|create your own SSL certificate]] or you should run mhttpd behind an SSL proxy.

Run mhttpd again.

You will get the following messages:
[mhttpd,INFO] ODB subtree /Runinfo corrected successfully
Mongoose web server will listen on ports "8080r,8443s"
Mongoose web server will use SSL certificate file "/home/johnfoo/packages/midas/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17633:mongoose,ERROR] mongoose web server cannot find password file "/home/johnfoo/online/htpasswd.txt"
[mhttpd,ERROR] [mhttpd.cxx:17634:mongoose,ERROR] please create password file: htdigest -c /home/johnfoo/online/htpasswd.txt Default midas
could not start the mongoose web server, see messages and midas.log, bye!

Create the password file by following the instructions printed by mhttpd. The http digest domain name is the experiment name, suggested default user name is "midas". You will be asked to type in a password


[mhostpc] htdigest -c /home/johnfoo/online/htpasswd.txt exptname midas
Adding password for midas in realm exptname.
New password:
Re-type new password:

It is a good idea to set the password file {{Filepath|path=htpasswd.txt}} readable and writable by owner only.

Now restart {{Utility|name=mhttpd}}
[mhostpc] mhttpd
Mongoose web server will listen on ports "8080r,8443s" **see note
Mongoose web server will use SSL certificate file "/home/suz/packages/midas/ssl_cert.pem"
Mongoose web server will use authentication realm "Default", password file "./htpasswd.txt"

Now point a web browser running on the same host computer (localhost) to https://localhost:8443
If the web browser is running on a different computer, go to URL of the form

https://mhostpc.triumf.ca:8443 (substitute your host machine name and domain for "mhostpc.triumf.ca")

If you are using the default SSL certificate you will probably get a message: "This Connection is Untrusted". Click "I understand the risks" and add an exception. This is because the test certificate is self-signed. Then confirm an exception.

If instead you get a "connection refused" error, the midas host pc may have the firewall enabled. To make a firewall exception for MIDAS, follow instructions here http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29

You should then see an authentication box asking you for the user name and password. The user name is "midas". Enter the password you just created. The Midas [[Status Page]] should appear with multiple buttons for run control as well as equipment listing (no equipments will be listed as yet) and application listings. Please refer to [[mhttpd]] (the MIDAS Web-based Run Control utility) for further information. You can start and stop runs from the main status page, and use the [[ODB Page]] to access the database (ODB).

; Note
: Default ports of 8080 and 8443 are used by [[mhttpd]]. If these ports are in use on your machine, start mhttpd with alternative ports, e.g.
[mhostpc] mhttpd --https 8448 --http 8089
: or see [[Mhttpd#Usage]] to change the default ports.

== Setup experiment database (ODB) ==

* run $HOME/online/bin/start_daq.sh

* odbedit, run these commands: (replace user names and directory names)
<pre>
set "/Logger/Message file" "/home/kopio03/online/midas.log"
set "/Logger/Data Dir" "/home/kopio03/online/data"
create STRING "/Logger/History dir"
set "/Logger/History dir" "/home/kopio03/online/history"
create STRING "/Logger/Elog dir"
set "/Logger/Elog dir" "/home/kopio03/online/elog"
set "/Logger/ODB dump file" "/home/kopio03/online/history/run%05d.xml"
set "/Logger/ODB dump" "y"
set "/Logger/Channels/0/Settings/Filename" "run%05dsub%03d.mid.gz"
set "/Logger/Channels/0/Settings/Subrun byte limit" "1000000000"
set "/Logger/Channels/0/Settings/Compression" 1
set "/Logger/Channels/0/Settings/ODB Dump" "y"
set "/Programs/Logger/Required" y
set "/Programs/Logger/Start command" "mlogger -D"
set "/Programs/fevme/Required" "y"
set "/Programs/fevme/Start command" "ssh -n lxdaq09 $HOME/online/src/fevme_gef.exe -O"
exit
</pre>
* open web browser e.g. firefox.
* go to the midas status page at https://localhost:8443 (default port).
** if running [[mhttpd]] with Mongoose HTTPS/OpenSSL (the default) for the first time, you will need to create a password file. Follow the instructions (see [[mhttpd#HTTPS/SSL server (Mongoose)]] for details).
** For other options (i.e. HTTPS/SSL proxy) see [[#Secure MIDAS and ELOG Web access]]
*OR open the midas status page at http://localhost:8081 ([[#NOTES|older MIDAS versions]])
* midas status page will show most stuff "red" as nothing is running yet
* DON'T DO THIS YET run ./fevme.exe (on the computer with the VME interface, could be different from computer hosting the experiment), observe that corresponding equipments have been created
* save the url bookmark to the "personal toolbar"
* go to the Programs page, stop mlogger, stop fevme, start mlogger, start fevme
* go to the Status page, start run, stop run
* go back to the Status page, everything should be green
* start a run
* send signals to the ADC gate
* you should be getting events
* to look at data, proceed with setting up the [[ROOTANA|ROOT Analyzer]].

== Start DAQ programs at boot time ==

* add this to /etc/rc.local (replace username and location of the start_daq script)
<pre>
su - alpha -c /home/alpha/online/bin/start_daq.sh
</pre>

== Setup local software version control ==

Version control for experiment source code is setup using "git" (http://git-scm.com/)

* cd $HOME/online
* git init
* git add exptab
* git add bin/start_daq.sh
* git add .gitignore ### contents can be
<pre>
*~
*.o
*.exe
</pre>
* git add src/Makefile src/*.cxx ...
* git commit -a

== Adjust MIDAS buffer sizes ==

Default MIDAS SYSTEM buffer size is 8 Mbytes, fairly small for high-data-rate experiments. The rule of thumb is to have at least a few seconds worth of buffer space available. For example, if event size is 10 Kbytes and the event rate is 1 kHz, data rate is 10*10^3*1*10^3 = 10 Mbytes/sec. To buffer 10 seconds of data we need 100 Mbytes of buffer space.

To resize the MIDAS event buffers (SYSTEM, etc) do this:
* stop all frontends, stop mlogger
* start odbedit:
** cd "/Experiment/Buffer sizes"
** set SYSTEM 100000000
* run "mdump -z SYSTEM"
* if mdump complains about the size of .SYSTEM.SHM, remove it, try again.
* ls -l /dev/shm ### to observe that the size of shared memory is correct

== Secure MIDAS and ELOG Web access ==
In versions prior to May 2015, the default web access to MIDAS and ELOG uses the "http:" protocol which is insecure. In this case, all information is transmitted as clear text meaning that secret, confidential and sensitive information (such as the MIDAS and ELOG passwords and usernames) can be stolen "easily". This means that even "password protected" MIDAS and ELOG pages are not really protected if accessed using the "http" method.

Better security for HTTP is gained by using a password protected '''SSL (https) proxy'''. (It does not provide absolute security because of remaining problems with the security of SSL certificates, security of passwords, etc). Setting up an SSL (https) proxy is described [[#Setting up an HTTP proxy|below]].

Since May 2015, an ''alternative secure option'' to setting up an HTTP proxy is available to users of MIDAS. Recent versions of elogd (ELOG) do support SSL https:// connections, and [[#mhttpd with HTTPS/SSL server (Mongoose)]] is now available. This option is the default, and provides a similar level of security to an HTTP proxy.

See [[Security#Web Access]] for a comparison of these two secure options.

=== mhttpd with HTTPS/SSL server (Mongoose) ===

Since May 2015 the MIDAS web server [[mhttpd]] is explicitly linked with OpenSSL to provide secure HTTPS connections via the [https://bitbucket.org/tmidas/midas/src/ecb9a8537448a8a43f7f9a2bfdb82e578208cde3/doc/mongoose/?at=develop Mongoose] web server (see [[mhttpd]]). With this version, default web access to MIDAS uses the "https" protocol. Web access to {{Utility|name=mhttpd}} can be restricted by using the "-a hostname" switch of [[mhttpd]]. The first time {{Utility|name=mhttpd}} is run, a password file must be created. An SSL certificate is also required. See [[mhttpd#HTTPS/SSL server (Mongoose)|HTTPS/SSL server (Mongoose)]] for instructions.

=== mhttpd using an HTTPS/SSL proxy ===

THESE INSTRUCTIONS ARE WRONG, DO NOT DO THIS. See instead the [https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Install_apache_httpd_proxy_for_midas_and_elog TRIUMF DAQ site].

An [[#Setting up an HTTP proxy|HTTP proxy]] must be set up. This is the only way of securing older version of [[mhttpd]] (pre August 2015). Older versions of mhttpd are started using the -p port option e.g.
* mhttpd -D -p 8080 

To run a new version of mhttpd using an HTTP proxy, use the options provided to run the old (non-Mongoose) webserver on a given port, i.e.
* mhttpd --oldserver 8080 --nomg -D 

When using an SSL proxy, only access from the SSL proxy (and maybe some special trusted machines) should be permitted.
This is done using the "-a hostname" switch of [[mhttpd]]. Normally there will be only "-a localhost" switch, enabling access only for the local machine (where the SSL proxy is running). Additional "-a hostname" switches enable access from listed local machines. No "-a xxx" enables access from everywhere (defeating the purpose of the SSL proxy, unless access controls are enforced elsewhere, i.e. by a site firewall or by local firewall rules).

== Setting up an HTTP proxy ==
In this example, we use APACHE HTTPD to password-protect a typical midas/mhttpd and elog installation.

In this configuration, one uses the Linux stock httpd that accepts encrypted https:// connections and forwards them to mhttpd and elogd. Instead of (or in addition to) using mhttpd and elogd passwords, one configures password protection in httpd via the regular apache httpd password mechanisms (htpasswd, etc).

Recent versions of elogd do support SSL https:// connections, but if one is running an SSL proxy for anyway, it is simpler to run both through the same SSL proxy using the same SSL host certificate and the same httpd password file.

=== Restricting http: access to elogd ===
;Note
:Recent versions of elogd do support SSL https:// connections. The following information is for those using an HTTP proxy (see above).

For elogd, this is done using the "-n localhost" switch with enables only access from the same machine if present, or access from anywhere is absent (defeating the purpose of the SSL proxy, unless access controls are enforced elsewhere).

(It is recommended to run elogd from the same user as the main daq user and to keep elogd.cfg and all logbooks in the home directory of this user, where they are captured by the normal site backup system)

== Install standalone elog ==

* login into the user account that will run the elog
* cd $HOME/packages
* git clone https://bitbucket.org/ritt/elog
* cd elog
* make
* create new file start_elogd with this contents:
<pre>
#!/bin/sh

killall elogd
sleep 1
killall -KILL elogd
sleep 1
$HOME/packages/elog/elogd -n localhost -x -c $HOME/packages/elog/elogd.cfg -p 8082 -D

#end
</pre>
* chmod a+x start_elogd
* edit elogd.cfg to read:
<pre>
[global]
port = 8082
SMTP host = smtp.triumf.ca
URL = https://titan00.triumf.ca/elog/

Reverse sort = 1
Display Mode = full

#List Menu commands = New, Find, Admin, Help
#Menu commands = New, Edit, Reply, Find, Duplicate, Help

Entries Per Page = 30
Supress Email on edit = 1
Default encoding = 1
Page title = TITAN ELOG
Resolve host names = 1

Logfile = /home/titan/packages/elog/elogd.log
#Logging level = 3

[midas]

List page Title = T2K M11 MIDAS ELOG
Comment = T2K M11 MIDAS ELOG
Page Title = T2K M11 MIDAS ELOG
RSS Title = [$logbook - $type - $system] $subject, posted by $author

Attributes = Author, Subject, Run, Type, System
Show Attributes Edit = Run, Author, Subject, Type, System
Required Attributes = Author, Type, System, Subject

Options Type = Routine, Reply, Shift Summary, Modification, Question, Info, Problem
Options System = General, DAQ, Beamline

Preset Run = $shell(MIDASSYS=. /home/t2km11/packages/midas/linux/bin/odbedit -d Runinfo -c 'ls -v \"run number\"')

Preset On Reply Type = Reply
Preset On Reply Run = $shell(MIDASSYS=. /home/t2km11/packages/midas/linux/bin/odbedit -d Runinfo -c 'ls -v \"run number\"')

List Display = Date, Subject, Type, System, Author, ID
Quick Filter = Date, Type, ID

Remove on reply = Author
Quote on reply = 1

Use lock = 1

************* Email Functionality ****************

Use Email Subject = [T2KM11 - $System] $Subject
Omit Email To = 1

Email System General = xxx
</pre>
* ./start_elogd &
* firefox http://localhost:8082 # hould show the elog message index

To start elogd automatically when the machine is rebooted, login as root and
* add this text to /etc/rc.local:
<pre>
su - titan -c "/home/titan/packages/elog/start_elogd"
</pre>
* chmod a+x /etc/rc.local
* systemctl start rc-local

To import elog entries from the mhttpd elog, do this:

* cd ~/packages/elog/logbooks
* ln -s /home/t2km11/online/elog midas
* cd midas
* ~/packages/elog/elconv)

== Install https proxy ==

THESE INSTRUCTIONS ARE OBSOLETE, INSTEAD,
* GO HERE: https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Configure_HTTPS_server_.28CentOS7.29
* AND GO HERE: https://midas.triumf.ca/MidasWiki/index.php/Quickstart_Linux#Run_the_MIDAS_Web_Server

FOLLOWING INSTRUCTIONS ARE OBSOLETE...

* login as root to the https proxy machine
* cd ~root
* yum install mod_ssl
* yum install crypto-utils # see http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_monitoring_of_HTTPS_certificates
* create a certificate request (replace ladd09 with your hostname): openssl req -new -nodes -newkey rsa:2048 -sha256 -out ladd09.csr -keyout ladd09.key (answer: CA, BC, Vancouver, TRIUMF, DAQ, ladd09.triumf.ca, email@email.com
* sign it by TRIUMF:
** mail -s "Certificate request" yourself@email.com < ladd09.csr
** forward this request to Andrew Daviel
** he will email the signed crt file, copy it to this system as ladd09.crt
* sign it yourself: openssl x509 -req -days 365 -sha256 -in ladd09.csr -signkey ladd09.key -out ladd09.crt
* (if the certificate expires, renew it by signing it again)
* Additional commands for working with certificates:
** explore the private key: openssl pkey -in ladd09.key -text -noout
** explore the certificate request: openssl req -in ladd00.csr -text -noout
** explore the certificate: openssl x509 -in ladd09.crt -noout -text
* move certificate files to proper system locations:
* mv ladd09.key /etc/pki/tls/private/
* mv ladd09.crt /etc/pki/tls/certs/
* if selinux is enabled, do this:
** restorecon -Rv /etc/pki/tls/certs/
** restorecon -Rv /etc/pki/tls/private/
** /usr/sbin/setsebool -P httpd_can_network_connect 1
* open /etc/httpd/conf.d/ssl.conf in a text editor, go to the very bottom and right before the "</VirtualHost>" entry, add following text:
<pre>
...
SSLCertificateFile /etc/pki/tls/certs/ladd09.crt
SSLCertificateKeyFile /etc/pki/tls/private/ladd09.key

ProxyPass /elog/ http://localhost:8082/ retry=1
ProxyPass / http://localhost:8080/ retry=1

<Location />

SSLRequireSSL
AuthType Basic
AuthName "password protected site"
Require valid-user

# create password file: touch /etc/httpd/htpasswd
# to add new user or change password: htpasswd /etc/httpd/htpasswd username
AuthUserFile /etc/httpd/htpasswd

</Location>
</VirtualHost>
...
</pre>
* comment out duplicate "SSLCertificateFile" and "SSLCertificateKeyFile" elsewhere in the file
* (optionally) If you got a certificate that is signed by DigiCert or RapidSSL then you'll need to add a line specifying the certificate chain file:
<pre>
...
SSLCertificateChainFile /etc/pki/tls/certs/DigiCertCA.crt
...
</pre>
* touch /etc/httpd/htpasswd
* htpasswd /etc/httpd/htpasswd midas # enter password midas
* chkconfig httpd on
* service httpd restart
* firewall-cmd --add-port=443/tcp --permanent
* firewall-cmd --reload
* firewall-cmd --list-all
* test it
** test the SSL proxy: https://host/ should yield the midas status page, https://host/elog/ should yield the elog message index
* in ODB, set "/Elog/URL" to "https://host/elog/"
* now from the midas status page, the "Elog" button should take us to the https Elog URL

In needed, enable user directories: https://blah/~user in ~user/public_html

* edit /etc/httpd/conf.d/userdir.conf, replace "UserDir disabled" with "UserDir enabled"
* setsebool -P httpd_enable_homedirs true
* systemctl restart httpd

== Setup the history mhttpd for faster access to history plots ==
When running an SSL proxy,
* start the main mhttpd (orange command for old mhttpd, green for new mhttpd with Mongoose(post August2015):
** "mhttpd -p 8071 -D" or
** "mhttpd -D --oldserver 8071 --nomg" 

* start the history mhttpd
** "mhttpd -p 8072 -D -H" or
** "mhttpd -D -H --oldserver 8072 --nomg" 
* set ODB /History/URL to "http://alphacpc09.cern.ch:8072/HS/"
* open the MIDAS status page
* go to the history section, try to open any history plot, observe that the history plot gif image loads correctly, inspect it's URL (use "copy image URL" or "view source", etc), it should point to port 8072 causing connection to the history mhttpd.
* continue with these instructions to setup history mhttpd access through an SSL proxy:
* setup SSL proxy access (required mod_proxy_html)
** login as root to the SSL proxy machine
** on SL5, install the missing mod_proxy_html httpd module:
** yum install httpd-devel libxml2-devel
** wget http://apache.webthing.com/mod_proxy_html/mod_proxy_html.tar.bz2
** tar xjvf mod_proxy_html.tar.bz2
** cd mod_proxy_html
** apxs -c -I. -I/usr/include/libxml2 -i mod_proxy_html.c
** apxs -c -I. -I/usr/include/libxml2 -i mod_xml2enc.c
** cd /etc/httpd/conf.d, add this to ssl.conf:

;before the ProxyPass statements:
<pre>
# proxy the MIDAS web servers
LoadModule xml2enc_module modules/mod_xml2enc.so
LoadModule proxy_html_module modules/mod_proxy_html.so
ProxyHTMLLinks a href
ProxyHTMLLinks link href
ProxyHTMLLinks img src
#ProxyHTMLEnable On
ProxyRequests off
</pre>
;after the ProxyPass statements:
<pre>
# ALPHA1 history access
ProxyPass /alpha1/history/ http://alphacpc09.cern.ch:8072/HS/ retry=1
ProxyPass /alpha1/ http://alphacpc09.cern.ch:8071/ retry=1

ProxyHTMLEnable On
ProxyHTMLURLMap http://alphacpc09.cern.ch:8072/HS/ /alpha1/history/
</pre>
;adjust:
*"alpha1" is the experiment name
*"alphacpc09.cern.ch" is the machine running mhttpd
*"8071" is the port number of the main mhttpd
** "mhttpd -p 8071 -D" or
** "mhttpd -D --oldserver 8071 --nomg" 
* "8072" is the port number of the history mhttpd
** "mhttpd -p 8072 -D -H" or
** "mhttpd -D -H --oldserver 8072 --nomg" 

[[Category:Installation]] [[Category:Buffer]] [[Category:Driver]]

Setup MySQL database for MIDAS

2026-02-03T01:37:26Z

Dfujimoto: /* Setting up MySQL on Ubuntu 24.04 LTS */

This page provides instructions for setting up a MySQL database to use with the MIDAS history logging.

Setting up the MySQL database is highly dependent on what OS and OS version you are using. Most of the setup work is just MySQL related. For this reason we provide below instructions for the MySQL database setup on a couple different OS. You will probably need to modify these instructions for your OS. Here's the some installation instructions:

* [[#Setting up MySQL on Ubuntu 24.04 LTS | Setting up MySQL on Ubuntu 24.04 LTS ]]
* [[#Setting up MySQL using mariadb on macbook | Setting up MySQL using mariadb on macbook]]

== Setting up MySQL on Ubuntu 24.04 LTS ==

This instructions assume you have root privileges and install MySQL as ROOT.

=== Do these steps as root user ===

Install the MySQL server software.

<pre>
apt install mysql-server
</pre>

You will find mysql has started after installation

<pre>
root@mpmt-daq03:/home1/wheel# service mysql status
mysql.service - MySQL Community Server
Loaded: loaded (/usr/lib/systemd/system/mysql.service; enabled; preset: enabled)
Active: active (running) since Fri 2025-04-25 15:12:58 PDT; 4s ago
Process: 153129 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0>
Main PID: 153137 (mysqld)
Status: "Server is operational"
</pre>

Default configuration only allows MySQL connections to localhost. Good
<pre>
root@mpmt-daq03:/home1/wheel# grep bind-add /etc/mysql/mysql.conf.d/mysqld.cnf
bind-address = 127.0.0.1
mysqlx-bind-address = 127.0.0.1
</pre>

Login to MySQL as root and create the MySQL accounts

<pre>
mysql -h localhost
...
mysql> CREATE DATABASE IF NOT EXISTS history;
mysql> CREATE USER history_reader@'localhost' IDENTIFIED BY 'CHANGE_THIS_PASSWORD!';
mysql> GRANT SELECT ON history.* TO history_reader@'localhost';
mysql> CREATE USER 'history_writer'@'localhost' IDENTIFIED BY 'CHANGE_THIS_PASSWORD!';
mysql> GRANT SELECT,INSERT,CREATE,ALTER,INDEX ON history.* TO 'history_writer'@'localhost';
mysql> flush privileges;
</pre>

ATTENTION: change the account password listed above to something different!

Check the socket to connect to:

<pre>
root@mpmt-daq03:/home1/wheel# grep sock /etc/mysql/my.cnf
# Port or socket location where to connect
socket = /run/mysqld/mysqld.sock
</pre>

=== Do these steps as the experiment user ===

Rebuild MIDAS with support for MySQL. If this works correctly the build will notice MySQL is installed. For instance, with cmake we will see

<pre>
-- MIDAS: Found MySQL version 8.0.41
</pre>

Go to online directory and create files so MIDAS knows accounts to talk to ODB

<pre>

testmysql@mpmt-daq03:~/online$ cd ~/online
testmysql@mpmt-daq03:~/online$ emacs -nw mysql_reader.txt
testmysql@mpmt-daq03:~/online$ emacs -nw mysql_writer.txt
testmysql@mpmt-daq03:~/online$ more mysql_*.txt
::::::::::::::
mysql_reader.txt
::::::::::::::
database=history
socket=/run/mysqld/mysqld.sock
user=history_reader
password=CHANGE_THIS_PASSWORD!
::::::::::::::
mysql_writer.txt
::::::::::::::
database=history
socket=/run/mysqld/mysqld.sock
user=history_writer
password=CHANGE_THIS_PASSWORD!
</pre>

Again, set the correct password in these files.

Change the mlogger settings so we use the MySQL for the history logging:
* disable the previous history logger
* enable the MySQL history logging:
<pre>
set "/Logger/History/MYSQL/Active" y
</pre>

Restart the mlogger on the command line and see that some MySQL tables are created and columns added:

<pre>
testmysql@mpmt-daq03:~/online$ mlogger
[Logger,INFO] Per-variable history is enabled
[Logger,ERROR] [history_schema.cxx:1258:Mysql::Prepare,ERROR] mysql_query(SELECT event_name, table_name, itimestamp FROM _history_index WHERE table_name!='';) error 1146 (Table 'history._history_index' doesn't exist)
[Logger,INFO] Adding SQL table "myperiodicequipment_pydt"
[Logger,INFO] Adding SQL table "_history_index"
[Logger,INFO] Adding column "event_name" to SQL table "_history_index", status 1
[Logger,INFO] Adding column "table_name" to SQL table "_history_index", status 1
...
[Logger,INFO] Adding column "fluxgate0" to SQL table "myperiodicequipment_pydt", status 1
[Logger,INFO] Adding column "fluxgate1" to SQL table "myperiodicequipment_pydt", status 1
[Logger,INFO] Adding column "valvex" to SQL table "myperiodicequipment_pydt", status 1
[Logger,INFO] Adding column "valvey" to SQL table "myperiodicequipment_pydt", status 1
[Logger,INFO] Adding column "t5" to SQL table "myperiodicequipment_pydt", status 1
...
</pre>

Login directly to the MySQL database and make sure the data is actually being logged

<pre>
mysql> select _i_time,fluxgate0 from myperiodicequipment_pydt where _i_time <1745620795;
+------------+-----------+
| _i_time | fluxgate0 |
+------------+-----------+
| 1745620782 | 6.20209 |
| 1745620783 | 6.3519 |
| 1745620784 | 6.5677 |
| 1745620785 | 7.81413 |
| 1745620786 | 5.41127 |
| 1745620787 | 7.31112 |
| 1745620788 | 6.14285 |
| 1745620789 | 7.20425 |
| 1745620790 | 6.9303 |
| 1745620791 | 7.08178 |
| 1745620792 | 7.44422 |
| 1745620794 | 5.32677 |
+------------+-----------+
12 rows in set (0.00 sec)
</pre>

Restart mhttpd and you will see the history plots starting to get data from MySQL database.

=== Change database location ===

To move the database to a different location on file do the following as root ([https://tecadmin.net/change-default-mysql-data-directory-in-linux/ source]):
<pre>
systemctl stop mysql
cp -a /var/lib/mysql /new/data/directory
</pre>
Then update the MySQL configuration file /etc/mysql/my.cnf
<pre>
[mysqld]
...
datadir=/new/data/directory
...
</pre>
You will need to either (a) adjust your apparmor settings, or (b) fully [https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#disable_apparmor disable apparmor].
Finally, restart the server:
<pre>
systemctl start mysql
</pre>
You can now delete the original database directory
<pre>
rm -rf /var/lib/mysql
</pre>

== Setting up MySQL using mariadb on macbook ==

* install mariadb-10.4
<pre>
macos:
sudo port install mariadb-10.4
ln -s ln -fs /opt/local/lib/mariadb-10.4/bin/mysql_config ~/bin/
mysql_config --version
10.4.13
</pre>
* rebuild midas, check the cmake finds the correct mysql: MIDAS: Found MySQL version 10.4.13
* create $HOME/.my.cnf
<pre>
[mysqld]
#port=3306
socket=/Users/olchansk/mysql/mysql.sock
datadir=/Users/olchansk/mysql/db
pid-file=/Users/olchansk/mysql/mysqld.pid

[client]
#port=3306
socket=/Users/olchansk/mysql/mysql.sock
</pre>
* mkdir $HOME/mysql
* start mysql:
<pre>
macos: /opt/local/lib/mariadb-10.4/bin/mysqld
</pre>
* configure the database: start the "mysql" utility (/opt/local/lib/mariadb-10.4/bin/mysql)
<pre>
NOTE: instructions thanks to Ben S.
NOTE: use some other passwords instead of the example "reader_password" and "writer_password"

MariaDB [(none)]> CREATE DATABASE IF NOT EXISTS history;
Query OK, 1 row affected (0.003 sec)

MariaDB [(none)]> CREATE USER history_reader@'localhost' IDENTIFIED BY 'reader_password';
Query OK, 0 rows affected (0.027 sec)

MariaDB [(none)]> GRANT SELECT ON history.* TO history_reader@'localhost';
Query OK, 0 rows affected (0.016 sec)

MariaDB [(none)]> CREATE USER 'history_writer'@'localhost' IDENTIFIED BY 'writer_password';
Query OK, 0 rows affected (0.014 sec)

MariaDB [(none)]> GRANT SELECT,INSERT,CREATE,ALTER,INDEX ON history.* TO 'history_writer'@'localhost';
Query OK, 0 rows affected (0.017 sec)

MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.001 sec)
</pre>
* try connecting to both users:
<pre>
/opt/local/lib/mariadb-10.4/bin/mysql -u history_reader -p ### should ask for reader password
/opt/local/lib/mariadb-10.4/bin/mysql -u history_writer -p ### should ask for writer password
</pre>
* create midas config files:
** mysql_writer.txt
<pre>
#server=localhost
#port=xxx
database=history
socket=/Users/olchansk/mysql/mysql.sock
user=history_writer
password=writer_password ### change this!!!
#buffer=1000
</pre>
** mysql_reader.txt
<pre>
#server=localhost
#port=xxx
database=history
socket=/Users/olchansk/mysql/mysql.sock
user=history_reader
password=reader_password ### change this!!!
</pre>
* test MIDAS connection
<pre>
mh2sql --mysql mysql_writer.txt
(there should be no errors printed)
</pre>
* if desired, import midas history from .hst files:
<pre>
mh2sql --mysql mysql_writer.txt *.hst
Reading 130813.hst
[mh2sql,ERROR] [history_schema.cxx:1165:Mysql::Prepare,ERROR] mysql_query(SELECT event_name, table_name, itimestamp FROM _history_index WHERE table_name!='';) error 1146 (Table 'history._history_index' doesn't exist)
[mh2sql,INFO] Adding SQL table "rpcexample", status 311
[mh2sql,INFO] Adding SQL table "rpcexample_1596127680", status 1
[mh2sql,INFO] Adding SQL table "_history_index", status 1
[mh2sql,INFO] Adding column "event_name" to SQL table "_history_index", status 1
...
[mh2sql,INFO] Adding SQL table "run_transitions", status 311
[mh2sql,INFO] Adding SQL table "run_transitions_1596127681", status 1
[mh2sql,INFO] Adding column "state" to SQL table "run_transitions_1596127681", status 1
[mh2sql,INFO] Adding column "run_number" to SQL table "run_transitions_1596127681", status 1
Reading 130815.hst
</pre>

** change ODB /Logger/History/MYSQL/Active to "y", "mysql writer" and "mysql reader" to location of these files (leave the default values if these files are in the experiment directory).
* enable MYSQL history in mhttpd:
** change ODB /History/LoggerHistoryChannel to "MYSQL"

Message System

2026-01-15T20:32:58Z

Dfujimoto:

{{Pagelinks}}
<div id="midas.log"></div>
== MIDAS Log file ==
MIDAS provides a general MIDAS log file for recording system and user messages across the different components of the data acquisition clients. The default name of this file is '''''midas.log'''''. It is not necessary for [[mlogger]] to be running for the MIDAS message logging system to work. As soon as any MIDAS application is started, the MIDAS log file is produced.
The [[#Example MIDAS Log file|MIDAS log file]] contains system and user messages generated by any application connected to the given experiment.

Note that MIDAS messages may alternatively be sent into the data stream for logging.

== Viewing MIDAS messages ==
All the MIDAS messages can be found in '''''midas.log'''''.
More convenient is the [[mhttpd]] [[Message Page]] where the latest MIDAS messages are displayed. The very latest midas message can also be found on the [[mhttpd]] [[Status Page]]. Midas messages are also sent to any clients running in an xterm (e.g. [[odbedit]]).

<div id="midasloglocation"></div>
== Location of the MIDAS log file midas.log ==
The location of this file is dependent on how the experiment has been setup, i.e.

; with the [[/Logger ODB tree]] :
If the ''/Logger'' ODB tree exists (i.e. has been created by [[mlogger]])
* The message file will be located in the defined directory specified by the key [[Keys in the ODB /Logger tree#Data dir|/Logger/Data dir]].
* The name of the message file may be changed using the ODB key [[Keys in the ODB /Logger tree#Message File|/Logger/Message File]].

;without the [[/Logger ODB tree]] :
If the ''/Logger'' ODB tree does NOT exist , i.e. the MIDAS standard logger [[mlogger]] is not being used,
* the location of the log file will be in the experiment-specific directory defined by either
** the Environment Variable named MIDAS_DIR , or if this not defined,
** the exptab file .

----
== Generating messages for the MIDAS log ==
Any client can produce status or error messages to be sent to the [[#MIDAS Log file]] with a single call to
[http://ladd00.triumf.ca/~daqweb/doc/midas-devel/doc/html/group__msgfunctionc.html#gaac032ca2438c47466bfc9722de6746ea cm_msg()] using the MIDAS library, e.g.

status = db_find_key(hDB, 0, "/Equipment/Cycle_scalers/Settings/",&hKey);
if(status != DB_SUCCESS && status != DB_NO_KEY)
{
cm_msg(MERROR, "begin_of_run", "error accessing \"/Equipment/Cycle_scalers/Settings/\" (%d)",status);
return status;
}

Depending on the [[#message types|message type]], these messages are forwarded to any other clients who may be available to receive these messages, as well as to the central MIDAS Log file
midas.log.
The message system is based on the [[Midas Core#The Buffer Manager|buffer manager]] scheme, but with a dedicated header to identify the type of message. A dedicated buffer (i.e. shared
memory) .SYSMSG.SHM is used to receive and distribute messages.

<div id="Message Macros"></div>

== Message Types ==

The predefined '''MIDAS Message macros''' contained in the MIDAS library provide a list of defined messages types.
These Macros compact the 3 first arguments of the cm_msg() call.
The Macro replaces the type of message, the routine name and the line number in the C-code. See
[http://ladd00.triumf.ca/~daqweb/doc/midas-devel/doc/html/group__msgfunctionc.html#gaac032ca2438c47466bfc9722de6746ea example] in cm_msg().

The available message types ( defined in midas.h ) for use with cm_msg() are:

;MT_ERROR
:Error message, to be displayed in red
:<code>(MT_ERROR, __FILE__, __LINE__)</code>
;MT_INFO
:Info or status message
:<code>(MT_INFO, __FILE__, __LINE__)</code>
;MT_DEBUG
:Only sent to SYSMSG buffer, not to midas.log file. Handy if you produce lots of message and don't want to flood the message file. Plus it does not change the timing of your app, since the SYSMSG buffer is much faster than writing to a file.
:<code>(MT_DEBUG, __FILE__, __LINE__)</code>
;MT_USER
:Message generated interactively by a user, like in the chat window or via the odbedit "msg" command
:<code>(MT_USER, __FILE__, __LINE__)</code>
;MT_LOG
:Messages with are only logged but not put into the SYSMSG buffer
:<code>(MT_LOG, __FILE__, __LINE__)</code>
;MT_TALK
:Messages which should go through the speech synthesis in the browser and are "spoken"
:<code>(MT_TALK, __FILE__, __LINE__)</code>
;MT_CALL
:Message which would be forwarded to the user via a messaging app (historically this was an actual analog telephone call via a modem)
:<code>(MT_CALL, __FILE__, __LINE__)</code>

== Example MIDAS Log file ==

Fri Mar 24 10:48:40 2000 [CHAOS] Run 8362 started
Fri Mar 24 10:48:40 2000 [Logger] Run #8362 started
Fri Mar 24 10:55:04 2000 [Lazy_Tape] cni-043[10] (cp:383.6s) /dev/nst0/run08360.ybs 849.896MB file NEW
Fri Mar 24 11:24:03 2000 [MStatus] Program MStatus on host umelba started
Fri Mar 24 11:24:03 2000 [MStatus] Program MStatus on host umelba stopped
Fri Mar 24 11:27:02 2000 [Logger] stopping run after having received 1200000 events
Fri Mar 24 11:27:03 2000 [CHAOS] Run 8362 stopped
Fri Mar 24 11:27:03 2000 [SUSI] saving info in run log
Fri Mar 24 11:27:03 2000 [Logger] Run #8362 stopped
Fri Mar 24 11:27:13 2000 [Logger] starting new run
Fri Mar 24 11:27:14 2000 [CHAOS] Run 8363 started
Fri Mar 24 11:27:14 2000 [CHAOS] odb_access_file -I- /Equipment/kos_trigger/Dump not found
Fri Mar 24 11:27:14 2000 [Logger] Run #8363 started
Fri Mar 24 11:33:47 2000 [Lazy_Tape] cni-043[11] (cp:391.8s) /dev/nst0/run08361.ybs 850.209MB file NEW
Fri Mar 24 11:42:35 2000 [CHAOS] Run 8363 stopped
Fri Mar 24 11:42:40 2000 [SUSI] saving info in run log
Fri Mar 24 11:42:41 2000 [ODBEdit] Run #8363 stopped
Fri Mar 24 12:19:57 2000 [MChart] client [umelba.Triumf.CA]MChart failed watchdog test after 10 sec
Fri Mar 24 12:19:57 2000 [MChart] Program MChart on host koslx0 stopped

[[Category:Messages]] [[Category:Macros]]

Alarm System

2025-10-31T17:16:50Z

Dfujimoto: Added messagebird voice calling instructions

{{Pagelinks}}

= Links =
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[/Alarms ODB tree]]
* [[Alarms Page]]
* [[Mhttpd|mhttpd MIDAS web server]]
</div>

= Introduction =
MIDAS provides an alarm system, which by default is turned off. When the alarm system is [[/Alarms ODB tree#Alarm system active|activated]] and an alarm condition is detected, alarm messages are sent by the system which appear as an alarm banner on the [[Status Page|mhttpd status page]], and as a message on any windows running [[odbedit]] clients. The alarm system is flexible and can be extensively customized for each experiment
using the [[Alarms Page|mhttpd Alarms Page]] or [[odbedit]].

The alarm system is built-in and part of the main experiment scheduler. This means no separate task is necessary to benefit from the alarm system. Its setup and activation is done through the '''[[/Alarms ODB tree]]'''. The alarm system includes several other features such as sequencing and control of the experiment. The alarm capabilities are:
* Alarm setting on any ODB variable against a threshold parameter.
* Alarm triggered by evaluated condition
* Selection of Alarm check frequency
* Selection of Alarm trigger frequency
* Customization alarm scheme; under this scheme multiple choices of alarm type can be selected
* Selection of alarm message destination (to system message log or to elog)
* email or SMS alerts can be sent
* Alarm triggered when a Program is not running

= Implementation of the MIDAS Alarm System =

The alarm system source code is [http://ladd00.triumf.ca/~daqweb/doc/midas-devel/doc/html/alarm_8c_source.html alarm.c].
Alarms are checked inside [http://ladd00.triumf.ca/~daqweb/doc/midas-devel/doc/html/group__alfunctioncode.html alarm.c::al_check()]. This function is called by cm_yield() every 10 seconds and by rpc_server_thread(), also every 10 seconds. For remote MIDAS clients, their al_check() issues an RPC_AL_CHECK RPC call into the MIDAS server utility [[mserver]], where rpc_server_dispatch() calls the local al_check(). As result, all alarm checks run inside a process directly attached to the local MIDAS shared memory (inside a local client or inside an mserver process for a remote client). Each and every MIDAS client runs the alarm checks. To prevent race conditions between different MIDAS clients, access to al_check() is serialized using the ALARM semaphore. Inside al_check(), alarms are triggered using
[http://ladd00.triumf.ca/~daqweb/doc/midas-devel/doc/html/group__alfunctioncode.html al_trigger_alarm()], which in turn calls al_trigger_class(). Inside al_trigger_class(), the alarm is recorded into an elog or into [[Message System|midas.log]] using cm_msg(MTALK).

Special note should be made of the ODB setting [[/Alarms ODB tree#System message interval|system message interval]], which has a surprising effect - after an alarm is recorded into system messages (using cm_msg(MTALK)), no record is made of any subsequent alarms until the time interval set by this variable elapses. With default value of 60 seconds, after one alarm, no more alarms are recorded for 60 seconds. Also, because all the alarms are checked at the same time, '''only the first''' triggered alarm will be recorded.

As of alarm.c rev 4683, {{Odbpath|path=/Alarms/System message interval}} is set to 0 ensures that every alarm is recorded into the [[Message System#MIDAS Log file|MIDAS log file]]. (In previous revisions, this setting may still miss some alarms).

<div id="Alarm class"></div>
= Alarms structure =
The [[/Alarms ODB tree]] structure is split into 2 sections:
*"Alarms" which define the condition to be tested. The user can create as many [[/Alarms ODB tree#Alarms subtree|Alarms]] as desired, but each must be one of the four defined [[#Alarm Types|Alarm types]] .
*"Classes" which define the action to be taken when the alarm occurs. Two Classes (Alarm and Warning) are defined by default. The user can add more [[/Alarms ODB tree#Classes subtree|Classes]] as desired.
 

In order to make the system flexible, each alarm class may perform different actions when an alarm is given. For example, it may
* write a system message (see [[/Alarms ODB tree#Write System Message|Write System Message]])
* write to the elog (see [[/Alarms ODB tree#Write Elog Message|Write Elog Message]])
* stop the run (see [[/Alarms ODB tree#Stop run|Stop run]])
* spawn a detached script listed in the ODB variable [[/Alarms ODB tree#Execute command|Execute command]]. This feature is used when an Alarm triggers Email or SMS alerts (see [[#Alarm triggering Email or SMS alerts|example]]).
 

= Alarm Types =

The four available Alarm Types are shown in Table 1. They are defined in [http://ladd00.triumf.ca/~daqweb/doc/midas-devel/html/midas_8h.html&nbsp midas.h]. The alarm type is entered into the [[/Alarms ODB tree#Type|Type]] key.

{| style="text-align: left; width: 100%; background-color: white;" border="3" cellpadding="2" cellspacing="2"
|+ Table 1 : Defined Alarm Types
|-
| colspan="2" rowspan="1" style="vertical-align: top; background-color: lavender; font-weight: bold;" | Alarm Type
| colspan="1" rowspan="1" style="vertical-align: top; background-color: lavender; font-weight: bold;" | INT value
| colspan="1" rowspan="1" style="vertical-align: top; background-color: lavender; font-weight: bold;" | Explanation

|-
| colspan="1" rowspan="1" style="vertical-align: top; background-color: white; font-weight: bold;" |Internal alarms
| colspan="1" rowspan="1" style="vertical-align: top; background-color: white; font-weight: normal;" | AT_INTERNAL
|1
|Trigger on internal (program) alarm setting through the use of the al_...() functions.

|-
| style="vertical-align: top; background-color: white; font-weight: bold;" |Program alarms
|AT_PROGRAM
|2
|Triggered on condition of the state of the defined task (i.e. program not running)

|-
| style="vertical-align: top; background-color: white; font-weight: bold;" |Evaluated alarms
|AT_EVALUATED
|3
|Triggered by ODB value on given arithmetical condition.

|-
| style="vertical-align: top; background-color: white; font-weight: bold;" |Periodic alarms
|AT_PERIODIC
|4
|Triggered by timeout condition defined in the alarm setting.

|}

== Program Alarm ==

Program (or rather "Program not running") alarms, when enabled, warn the user when a program is not running.

Program alarms are enabled by setting the ODB key [[/Programs ODB tree#Alarm class|/Programs/<client-name>/Alarm class]] to a valid Alarm class specified in the [[/Alarms ODB tree]]. The first time the alarm is triggered, an /Alarms/Alarms/<client-name> subtree will be created automatically. The program alarm will not be visible in the [[Alarms Page]] until the alarm has triggered, and the subtree created.

The alarm system periodically calls [http://ladd00.triumf.ca/~daqweb/doc/midas-devel/doc/html/group__alfunctioncode.html al_check()]. This causes every client listed in the {{Odbpath|path=/Programs}} ODB tree to be tested using [http://ladd00.triumf.ca/~daqweb/doc/midas-devel/doc/html/group__cmfunctionc.html cm_exist()] to see if it is running. If the client is not running, the time of first failure is recorded in the ODB key [[/Programs ODB tree#First failed|/Programs/<client-name>/First failed]].

If the client has not been running for longer than the time set in ODB key [[/Programs ODB tree#Check interval|/Programs/<client-name>/Check interval]], a "Program not running" alarm is triggered (if enabled by [[/Programs ODB tree#Alarm class|Alarm class]]) and the program is restarted (if enabled by [[/Programs ODB tree#Auto restart|/Programs/<client-name>/Auto restart]] and a valid [[/Programs ODB tree#Start command|Start command]] is supplied).

The "not running" condition is tested every 10 seconds (each time al_check() is called), but the frequency of ''Program not running'' alarms can be reduced by increasing the value of the ODB key
[[/Programs ODB tree#Check interval|/Programs/<client-name>/Check interval]]
(default value 60 seconds). This can be useful if [[/Alarms ODB tree#System message interval|System message interval]] in the specified alarm class subtree is set to zero.

== Periodic Alarm ==
The periodic alarm is activated periodically according to the time in [[/Programs ODB tree#Check interval|/Programs/<client-name>/Check interval]]. An example of a periodic alarm is "Demo Periodic" in the [[/Alarms ODB tree#Example|example]].

== Evaluated Alarm ==

Evaluated alarms require an ''alarm condition'' which is entered into the ODB key [[/Alarms ODB tree#Condition|Condition]] in the [[/Alarms ODB tree#<alarm_name> subtree|<alarm_name> subtree]].
The condition may be simply a '''comparison''' between any ODB variable and a threshold parameter, e.g.

/Runinfo/Run number > 100
or it may be an '''evaluated condition'''. One can write conditions like

/Equipment/HV/Variables/Input[*] > 100
or

/Equipment/HV/Variables/Input[2-3] > 100
or

/Equipment/HV/Variables/Input[1,4,5-8,10] > 100

to check all or certain values from an array. A dash means a range including both indices. If one array element fulfills the alarm condition, the alarm is triggered. In addition, bit-wise alarm conditions are possible, e.g.

/Equipment/Environment/Variables/Input[0] & 8
The alarm is triggered if bit #3 is set in Input[0].

The value of an evaluated alarm is computed using al_evaluate_condition() in [http://ladd00.triumf.ca/~daqweb/doc/midas-devel/html/alarm_8c_source.html alarm.c].

Sometimes alarm trigger kind of accidentally on some analog input voltage if they are noise. In order to avoid this, the "Trigger count required" can be used. If this value contains a non-zero value of N, then the alarm system requires the alarm condition to be met N consecutive times until the alarm to be triggered.

== Internal Alarm ==
These are triggered in a program using a call to
[https://daq.triumf.ca/~daqweb/doc/midas-devel/html/group__alfunctioncode.html al_trigger_alarm()]. See also description of al_trigger_alarm() sequence [[#Implementation of the MIDAS Alarm System|above]].

There is nothing surprising in these alarms. Each alarm is checked with a time period set by ODB key [[ /Alarms ODB tree#Check interval|Check interval]] in the [[/Alarms ODB tree]].

= Alarm triggering Email or SMS alerts =

It is possible to have the MIDAS alarm system send email or SMS alerts to cell phones when alarms are triggered. This can be configured by defining an ODB alarm on a critical ODB parameter, e.g.

/Alarms/Alarms/Liquid Level
Active y
Triggered 0 (0x0)
Type 3 (0x3)
Check interval 60 (0x3C)
Checked last 1227690148 (0x492D10A4)
Trigger count 0 (0x0)
Trigger count required 0 (0x0)
Time triggered first (empty)
Time triggered last (empty)
Condition /Equipment/Environment/Variables/Input[0] < 10
Alarm Class Level Alarm
Alarm Message Liquid Level is only %s

In this example, the alarm triggers an alarm of class "Level Alarm". This alarm class is defined as follows:

/Alarms/Classes/Level Alarm
Write system message y
Write Elog message n
System message interval 600 (0x258)
System message last 0 (0x0)
Execute command /home/midas/level_alarm '%s'
Execute interval 1800 (0x708)
Execute last 0 (0x0)
Stop run n
Display BGColor red
Display FGColor black

The key here is to call a script "level_alarm", which can send emails. Use something like:

#/bin/csh
echo $1 | mail -s \"Level Alarm\" your.name@domain.edu
odbedit -c 'msg 2 level_alarm \"Alarm was sent to your.name@domain.edu\"'

Ensure the shell script is executable (e.g., <code>chmod +x script.sh</code>).

The second command just generates a MIDAS system message for confirmation. Most cell phones (depends on the provider) have an email address. If you send an email there, it will be translated into a SMS message.

The script file above can of course be more complicated. A perl script could be used that parses an address list, so other interested parties can register by adding his/her email address to that list. The script may also collects some other slow control variables (like pressure, temperature) and combine them into the SMS message.

For very sensitive systems, having an alarm via SMS may not be sufficient, since the alarm system could be down (e.g. computer crash, network failure). In this case 'negative alarms' can be used. For example, every 30 minutes the system may send an SMS with the current parameter values. If the expected message is not received, it may indicate that something in the MIDAS system is wrong.

= Alarm triggering Slack notifications =

A more modern way for notification is to use messengers apps such as Slack. To send alarms to Slack, do the following:

* Go to https://api.slack.com/apps
* Create an App "MIDAS alarms", select "From scratch"
* Name it "MIDAS alarm" or similar, select your workspace
* Click on "Add features and functionality"
* Select "Incoming Webhooks"
* Activate Incoming Webhooks
* Click on "Add New Webhook to Workspace"
* Select channel where alarms get posted, allow access
* Copy sample curl request and replace "Hello World" by "$1", such as:

curl -X POST -H 'Content-type: application/json' --data "{\"text\":\"$1\"}' https://hooks.slack.com/services/[xxx]/[yyy]/[zzz]

(leave they keys xxx, yyy, zzz as shown on the web page).

* Call the curl command from the alarm system by putting the above command under <code>/Alarms/Classes/All/Execute command</code> into the ODB.
* Ensure the shell script is executable (e.g., <code>chmod +x script.sh</code>)

= Alarm triggering Telegram notifications =

A telegram bot needs to be created to post Midas alarms into telegram channels.

* Go to https://web.telegram.org and login by scanning the QR-code with the telegram app on your phone
* Go to https://t.me/botfather
* Click on "open in web"
* A chat window will open.
* Send "/newbot" into the chat and assign name and username to your bot
* At the end of the bot creation a access token will be displayed. Save that access token.
* Post "/mybots" into the chat
* Select the bot that was just created
* Click on "Bot settings"
* Click on "Allow Groups"
* Enable groups for the bot
* Click on "back to settings"
* Click on Channel Admin Rights
* Click on "Manage channel" (a checkmark should appear next to the text)
* Close this chat.

* Add the created bot to a telegram group chat via searching for the bot name in "add members" (same as adding new People to the group)
* Now you need the chat_id of the same group chat. When you open the group chat in a webbrowser the URL should look like this https://web.telegram.org/a/#-1700000000 (where 1700000000 will be a different number for you)
* The group chat_id is then -1001700000000 (add -100 in front of the number in your URL)
* With the chat_id from above and the access token from the bot creation process you can post to the telegram group like this: (replace <chat_id> and <accessToken> with your numbers)

curl -s -X POST "https://api.telegram.org/bot<accessToken>/sendMessage" -d chat_id=<chat_id> -d text="🚨 Midas alarm triggered with message = $1"

* Create a shell script with this command
* Ensure the shell script is executable (e.g., <code>chmod +x script.sh</code>)
* Call the shell script from the alarm system by putting <code>path/to/script.sh '%1$s'</code> under <code>/Alarms/Classes/All/Execute command</code> into the ODB.
* Should you want to post the same alarm to multiple systems you can just separate scripts in <code>/Alarms/Classes/All/Execute command</code> with a semicolon <code>path/to/script.sh '%1$s';path/to/script2.sh '%1$s'</code>

= Alarm triggering Mattermost notifications =

First a mattermost bot needs to be created. To do this you need to be an Admin of the Team that the bot should post to.

* In the drop down menu on the top left corner of the mattermost page click on "Integrations" (note: this field will be missing if you are not an Admin of the selected mattermost team)
* Select "Incoming Webhooks"
* Click on "Add Incoming Webhook"
* Give the bot a title and username (e.g. midasalarms) and select the channel to post to, click "save"
* Now a URL for the incoming webhook should be displayed (https://mattermost.gitlab.rlp.net/hooks/<token>). Copy that URL into the curl command in the following script:

#!/bin/bash
alarm_message=$1
generate_post_data()
{
cat <<EOF
{
"text": "@all Midas alarm triggered with message = $alarm_message",
"icon_emoji":":rotating_light:",
"username":"Midas"}
EOF
}
curl -i -X POST -H 'Content-Type: application/jsoni' --data "$(generate_post_data)" https://mattermost.gitlab.rlp.net/hooks/<token>

- Call the shell script from the alarm system by putting <code>path/to/script.sh '%1$s'</code> under <code>/Alarms/Classes/All/Execute command</code> into the ODB.
Ensure the shell script is executable (e.g., <code>chmod +x script.sh</code>).

= Alarm triggering MessageBird Voice Call =

We will use a text-to-voice to read out MIDAS alarms in a phone call to shifters. The service is called bird (formerly messagebird). Note that bird hosts two platforms: "engagement" and "connectivity". Each has its own API, documentation, and web interface. We will use the engagement platform. Be sure to look at the [https://docs.bird.com correct documentation].

== MessageBird Setup ==

# Make an account: https://app.bird.com/
# Add some funds to your main wallet: https://app.bird.com/settings/billing/overview/wallets/
# Buy a number. This number will be the where the call originates from, so you should pick something local to your area.
## In the left-hand navigation side panel: "Developer" -> "Numbers" -> "Numbers & Senders"
## Top-right: "Browse Inventory"
## Make sure your number is capable of voice calling (it seems like most of them are)
## Follow the on-screen prompts to finish your purchase and activate the number. You will have to create a UseCase. Either "Mixed" or "Security Alert" should do.
# Ensure compliance with local regulations
## Go back to the number summary: "Developer" -> "Numbers" -> "Numbers & Senders". You should now have you number listed. Click on it.
## At the top there is a menu, select "Voice"
## You should see some links to set up brand identification and compliance (what will you be using the number for, the company associated with the number, etc). Follow the on-screen instructions. Once compliance has been requested it may take a few days to be approved.
## While you wait, you also need to create a Voice Channel. In this same menu, click on "+ Create channel". You can see your channels from the side bar navigation: "Developer" -> "Voice" -> "Voice Setup".

== Information Needed to Make Calls ==

You need a few pieces of info:

* API Access Key
** In the side bar navigation: "Developer" -> "API Access Keys" (at the bottom)
** Make a new key. Keep it somewhere safe.
* Workspace ID
** In the side bar navigation: "Settings" (at the bottom) -> "Workspaces" -> Click on your workspace
* Channel ID
** In the side bar navigation: "Developer" -> "Voice" -> "Voice Setup" -> Click on your voice channel -> Settings (top right gear)
* Call ID
** Only available once the call is placed. See the script below.
* Outgoing number
** In the side bar navigation: "Developer" -> "Numbers" -> "Numbers & Senders"

== Placing a Call ==

You can follow the documentation here: https://docs.bird.com/api/voice-api/voice-calls-api. Here is a python example script:
<pre>
import json, requests, time

# settings
workspaceId = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
channelId = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
APIKEY = 'YourAPIKey'
outgoing = '16042221047' # outgoing number with country code
targetnumber = 'YourPurcahsedPhoneNumber' # who to call
message = "Hello, world! This is my first message."

# place a call
# https://docs.bird.com/api/voice-api/voice-calls-api/initiate-an-outbound-call
def place_call(number):

# place the call
response = requests.post(
f"https://api.bird.com/workspaces/{workspaceId}/channels/{channelId}/calls",
headers={"Authorization":f"AccessKey {APIKEY}","Content-Type":"application/json"},
data=json.dumps({
"from": outgoing,
"to": number,
"ringTimeout": 60,
"maxDuration": 120,
"record": False,
})
)

# call id is needed for the next steps
data = response.json()
callId = data['id']
return callId

# get the call status
# https://docs.bird.com/api/voice-api/voice-calls-api/get-a-call-insights
def get_call_status(callId):
response = requests.get(
f"https://api.bird.com/workspaces/{workspaceId}/channels/{channelId}/calls/{callId}/insights",
headers={"Authorization":f"AccessKey {APIKEY}","Accept":"*/*"},
)
data = response.json()
return data['status']

# text-to-voice: play message in the call
# https://docs.bird.com/api/voice-api/voice-calls-api/say-text-to-speech-tts
def play_message(callId, message):
response = requests.post(
f"https://api.bird.com/workspaces/{workspaceId}/channels/{channelId}/calls/{callId}/say",
headers={"Authorization":f"AccessKey {APIKEY}","Content-Type":"application/json"},
data=json.dumps({
"text": message,
"locale": "en-US",
"voice": "text",
"loop": 10, # repeat the message this many times
"timeout": 30
})
)
return response.json()

# RUN =========================

# place call
callId = place_call(targetnumber)

# wait until call is answered, or until 30s has passed
t0 = time.time()
while time.time() - t0 < 30:
status = get_call_status(callId)
print(f'Call status: "{status}"')

# call was answered
if status == 'ongoing':
break

# check again later
time.sleep(1)

# play the message
time.sleep(0.5)
play_message(callId, message+',') # comma after message introduces a pause between loops
</pre>

[[Category:Alarms]]