Quickstart Linux: Difference between revisions

From MidasWiki
Jump to navigation Jump to search
(Created page with "{{Pagelinks}} ==Installation== Decide on a username to run the experiment, and log in as this username on the host machine. This machine will host the MIDAS experiment, i.e....")
 
 
(47 intermediate revisions by 5 users not shown)
Line 1: Line 1:
{{Pagelinks}}
{{Pagelinks}}
== Introduction ==
This quickstart shows you how to install the MIDAS packages on a linux box and create a '''MIDAS experiment'''.
The linux box will be called the MIDAS '''''experiment host'''''.


==Installation==
==Installation==
Decide on a username to run the experiment, and log in as this username on the host machine. This machine will host
Log on to the ''[[#Introduction|Experiment Host]]'' and decide on a name for the MIDAS experiment (the ''experiment name'').
the MIDAS experiment, i.e. where the ODB and other shared memory buffers will reside. This will also be where the MIDAS and other packages will be installed, and will be called the MIDAS host (localhost). Decide on a name for the experiment (i.e. MIDAS experiment name).
In these instructions, the MIDAS packages and experiment will be installed under the username "johnfoo".  
The ODB and other shared memory buffers for the MIDAS experiment will reside on the ''experiment host'' (localhost).
In these instructions, the username is "johnfoo" and the MIDAS and other packages will be installed under $HOME/packages/ . The MIDAS experiment directory will be $HOME/online and the MIDAS experiment name will be "e777". The MIDAS host name will be "mhostpc" The user will substitute appropriate host name, username, MIDAS experiment name and directories for his/her own setup.
MIDAS and the other required packages will be installed under directory  {{Filepath|path=$HOME/packages}}. The MIDAS ''experiment directory'' will be {{Filepath|path=$HOME/online}} and the MIDAS ''experiment name'' will be "e777". The name of the ''experiment host'' will be "mhostpc".
 
;NOTE
: The user should substitute directory paths for the MIDAS packages and experiment directory as required, and names for ''experiment host'', username, ''experiment name'' appropriate to his/her own setup.


== Experimental setup ==
== Experimental setup ==
These instructions describe setting up an experiment that will run ONLY on the MIDAS host computer (i.e. '''localhost only'''). In this case, the MIDAS host has access to any hardware required, and all frontends, logger, analyzer etc. run on the one computer.  
These instructions describe setting up an experiment that will run ONLY on the MIDAS ''experiment host'' computer (i.e. '''localhost only'''). In this case, the MIDAS ''experiment host'' has access to any hardware required, and all frontends, logger, analyzer etc. run on the one computer.  


Modifications to run an experiment with additional frontend(s) running remotely (i.e. on remote cpu(s)) will be described
Modifications to run an experiment with additional frontend(s) running '''remotely''' (i.e. on remote cpu(s)) will also be described.
below. In that case, the remote cpu(s) typically have access to some or all the hardware. They might be cpus running in VME crates for example, connected to the MIDAS Host via Ethernet.  They read data from the hardware and send it back to the MIDAS Host to be analyzed and/or logged.
Follow the instructions for setting up a localhost-only experiment until indicated.  
   
   
In both cases, the main MIDAS applications (e.g. [[mlogger]], [[mhttpd]]) run on the MIDAS host.
In both cases, the main MIDAS applications (e.g. [[mlogger]], [[mhttpd]]) run on the MIDAS ''experiment host''.


==Environment Variables==
==Environment Variables==
The following Environment variables should to be added to the $HOME/.cshrc (or .bash) file depending on your shell so that they will then be defined at login. Substitute appropriate values for your own setup.
The following Environment variables should to be added to the {{File|name=.cshrc}} or {{File|name=.bashrc}} file (depending on your shell) in the {{Filepath|path=$HOME}} directory on the ''experiment host'',  so that they will be defined at login. '''Substitute appropriate values for your own setup'''.  
<small>
{| class="wikitable"
{| class="wikitable"
|-
|-
! csh  !! bash !! comment
! csh  !! bash !! comment
|-
|  setenv LANG C ||  export LANG=C || ||
|-
| setenv GIT_EDITOR "emacs -nw" || export GIT_EDITOR="emacs -nw" || ||
|-
|-
| setenv MIDASSYS $HOME/packages/midas ||  export MIDASSYS=$HOME/packages/midas ||Base directory of the MIDAS package
| setenv MIDASSYS $HOME/packages/midas ||  export MIDASSYS=$HOME/packages/midas ||Base directory of the MIDAS package
|-
| setenv ROOTSYS  $HOME/packages/root  || export ROOTSYS=$HOME/packages/root|| setup ROOTSYS only if using ROOT
|-
|-
| setenv MIDAS_EXPTAB $HOME/online/exptab || export MIDAS_EXPTAB=$HOME/online/exptab  || MIDAS experiment table  
| setenv MIDAS_EXPTAB $HOME/online/exptab || export MIDAS_EXPTAB=$HOME/online/exptab  || MIDAS experiment table  
|-
|-
|-
| setenv MIDAS_EXPT_NAME e777  || export MIDAS_EXPT_NAME=e777  || MIDAS experiment name
| setenv MIDAS_EXPT_NAME e777  || export MIDAS_EXPT_NAME=e777  || MIDAS experiment name
|-
|-
| setenv PATH .:$MIDASSYS/linux/bin:$PATH  || export PATH=$PATH:$MIDASSYS/linux/bin || path
| setenv PATH .:$MIDASSYS/bin:$PATH  || export PATH=$PATH:$MIDASSYS/bin || path
|-
| setenv PATH .:$HOME/online/bin:$ROOTSYS/bin:$PATH|| export PATH=$PATH:$HOME/online/bin:$ROOTSYS/bin  ||if using ROOT
|}
|}
</small>
Logout and login again, or source .cshrc (source .bashrc) for the changes to take effect.
Logout and login again, or source {{File|name=.cshrc}} (source {{File|name=.bashrc}}) for the changes to take effect.
;NOTE
: See [[MIDAS environment variables]] for a list of environment variables used by MIDAS.


== ROOT Package Installation ==
== ROOT Package Installation ==
For full MIDAS operation, ROOT is needed for the data logging and analysis packages. However, ROOT is not essential to run MIDAS. If ROOT is NOT installed, the environment variable ROOTSYS should be undefined.
MIDAS has optional integration with [https://root.cern.ch ROOT]. ROOT is not required for acquiring data or storing it in midas files, but is used by some of the analysis tools (including the example analyzer that will be built later in this Quickstart). It is also possible to write midas data directly to ROOT files if desired.


If ROOT is needed, see [[Setup MIDAS experiment#Install ROOT|Install ROOT]].
If you require ROOT integration, follow the installation instructions on the [https://root.cern.ch ROOT website] and set the ROOTSYS environment variable.
 
If you do not require ROOT integration, leave the ROOTSYS variable undefined.
 
== Adding SSL support ==
 
We recommend running mhttpd (the MIDAS web server) behind an industry-grade web proxy like apache httpd or nginx. httpd/nginx will then handle the SSL security. It is however also possible to build mhttpd with built-in SSL support. See [[Mhttpd|the mhttpd documentation]] for more details about web security.
 
If you wish to enable SSL support in mhttpd itself:
 
<small>
  [mhostpc] cd $MIDASSYS
  [mhostpc] make mbedtls
</small>


== MIDAS Package Installation ==
== MIDAS Package Installation ==
Installed the MIDAS and other package(s) from the MIDAS git repository as follows:
The MIDAS package will be installed on the ''experiment host'' in the directory given by MIDASSYS - see [[#Environment Variables]].
    mkdir $HOME/packages
    cd $HOME/packages
    git clone https://bitbucket.org/tmidas/midas
    git clone https://bitbucket.org/tmidas/mxml


Install the MSCB  package only if you will be using the MSCB (see[[MSCB Page]]) system for slow-controls:
Install the MIDAS and other package(s) from the MIDAS git repository in an xterm as follows:
<small>
    git clone https://bitbucket.org/tmidas/mscb    ## MSCB package
  [mhostpc] mkdir $HOME/packages
    cd mscb
  [mhostpc] cd $HOME/packages
    make        ## Build the MSCB package
  [mhostpc] git clone https://bitbucket.org/tmidas/midas --recurse-submodules
    cd ../
</small>
 
If you already cloned MIDAS and forgot the "--recurse-submodules", you can do a "git submodule update --init --recursive" later. If you want to pull MIDAS later to update to the current version, you need either "git pull --recurse-submodules". To tell git to always recurse, you can set a global flag via "git config submodule.recurse true".


The MIDAS Makefile will auto-detect whether MSCB, ROOT, MYSQL, SQLITE, ODBC etc. are available (or have been installed). If you need these features, make sure they are available before building the MIDAS package.
The MIDAS Makefile will auto-detect whether MSCB, ROOT, MYSQL, SQLITE, ODBC etc. are available (or have been installed). If you need these features, make sure they are available before building the MIDAS package.
<small>
  [mhostpc] cd $MIDASSYS
  [mhostpc] mkdir build
  [mhostpc] cd build
  [mhostpc] cmake ..                  ### configure build process (see note below about cmake/cmake3)
  [mhostpc] make install              ### build and install midas to bin/ and lib/ directories
  [mhostpc] cd ..
  [mhostpc] ls -l bin/odbedit        ### check that odbedit has been created (do not run it yet)
</small> 


    cd midas
MIDAS requires cmake version 3.10 or later. On some systems, you have to call "cmake3 ..", on others just "cmake ..". Some systems might require you to install cmake first (see the [https://cmake.org/download/ cmake website; installation is very simple).
    make          ## Build the MIDAS package
    ls -l linux/bin/odbedit      ### check that odbedit has been created (do not run it yet)
 
In case of problems see [[Installation#Installing MIDAS|bitbucket down]] or [[Installation/Compilation problems#SSL certificate errors|SSL certificate errors]].


In case of problems see
* [[Installation#Installing MIDAS|bitbucket is down]] or
* [[Installation/Compilation problems#SSL certificate errors|SSL certificate errors]].


==ROOTANA Package Installation==
==ROOTANA Package Installation==
If ROOTANA is needed,
 
     cd $HOME/packages
rootana is an optional framework for analyzing midas data in realtime and displaying plots (either by a graphical interface or via webpages). Most of the features are now available in manalyzer, which is included in the base midas install.
     git clone https://bitbucket.org/tmidas/rootana
 
     cd rootana
If you require the legacy ROOTANA package, you can install it with:
     make
<small>
     [mhostpc] cd $HOME/packages
     [mhostpc] git clone https://bitbucket.org/tmidas/rootana
     [mhostpc] cd rootana
     [mhostpc] make
</small>
 
add ROOTANASYS=$HOME/packages/rootana to the login file.
 
Install the JSROOT package:
<pre>
cd $HOME/packages
git clone https://github.com/linev/jsroot.git
</pre>
 
add JSROOTSYS=$HOME/packages/jsroot to the login file.


==ROODY Package Installation==
==ROODY Package Installation==
ROODY is a legacy package for displaying live ROOT histograms. It is generally not used by new experiments.
If ROODY is needed,
If ROODY is needed,
     cd $HOME/packages
<small>
     git clone https://bitbucket.org/tmidas/roody
     [mhostpc] cd $HOME/packages
     cd roody
     [mhostpc] git clone https://bitbucket.org/tmidas/roody
     make
     [mhostpc] cd roody
   
     [mhostpc] make
</small>


== Create exptab ==
== Create the Experiment file exptab ==
In this example, the experimental directory is $HOME/online.
In this example, the experimental directory is {{Filepath|path=$HOME/online}}. Create this directory on the ''experiment host'' (localhost):
mkdir $HOME/online
<small>
cd $HOME/online
  [mhostpc] mkdir $HOME/online
Create the [[exptab]] file "$HOME/online/exptab" containing the three parameters
  [mhostpc] cd $HOME/online</small>
Then create the [[exptab]] file {{Filepath|path=$HOME/online/exptab}} containing the three parameters:
# the experiment name  
# the experiment name  
# location of MIDAS shared memory buffers
# location of MIDAS shared memory buffers
# username
# username
as follows:
as follows (using parameters for your own experiment):
    cat > exptab
<small>
     e777 /home/johnfoo/online johnfoo
  [mhostpc] cat > exptab
     e777 /home/johnfoo/online johnfoo</small>
 
The path of the  {{File|name=exptab}} file is given by MIDAS_EXPTAB - see [[#Environment Variables]].


== Create shared memory files  ==
== Create shared memory files  ==
At this point you should be able to run [[odbedit]] on localhost. The first time odbedit is run, it will create the required .*.SHM files  in the MIDAS experiment directory $HOME/online.  
At this point you should be able to run [[odbedit]] on the ''experiment host'' (localhost). The first time {{Utility|name=odbedit}} is run, it will create the required {{File|name=.*.SHM}} files  in the MIDAS experiment directory {{Filepath|path=$HOME/online}}. These are the saved files for the MIDAS shared memory.


The default directory $HOME/online will contain the MIDAS messages file (midas.log) and
The default directory {{Filepath|path=$HOME/online}} will contain the MIDAS messages file ({{File|name=midas.log}}) and
any data files you may create. If you want an alternative location for the data files see [[#Customizing your experiment]].
any data files you may create. If you want an alternative location for the data files see [[#Customizing your experiment]].
    
    
Now run odbedit, and type the command "ls" to list the default directories.  
Run {{Utility|name=odbedit}}, and type {{Odbedit cmd|cmd=ls}} to list the default directories.  
  odbedit>
  <small>[mhostpc] odbedit
  [local:e777:S]/>ls
  [local:e777:S]/>ls
  Experiment
  Experiment
Line 110: Line 152:
  Logger
  Logger
  Runinfo
  Runinfo
  Alarms  
  Alarms</small>
   
   


== Run the MIDAS logger [[mlogger]] ==
== Run the MIDAS Web Server ==
Start the MIDAS logger
 
$ mlogger
=== Using apache ===
This should start without error at this point. It is usually run as a daemon, however in this case it is run in a terminal to check for errors.
 
Recommended for production systems with maximum security is to run mhttpd behind an industry-standard SSL/https proxy with password protection. We document instructions for apache here, but you could nginx instead.
 
To do this using apache httpd on CentOS-7, do this:
* configure apache httpd per instructions for [https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Install_apache_httpd_proxy_for_midas_and_elog Ubuntu] or [https://daq00.triumf.ca/DaqWiki/index.php/SLinstall#Configure_HTTPS_server_.28CentOS7.29 CentOS 7]
* run mhttpd
* use web browser to connect to https://mhostpc/
* enter the user name and password configured in apache httpd
* the midas status page should open
 
=== Standalone (less secure) ===
 
For simplified installation, one can use standalone mhttpd with built-in https and password protection. This configuration
is only as secure as the underlying mongoose and OpenSSL web server and encryption packages. Use the odbedit tool to edit settings in the [[Webserver_ODB_tree|Webserver ODB tree]]:
 
* Set "Enable https port" to "y"
* Set "https port passwords" to "y"
 
Many current operating systems implement a built-in firewall which will block outside access to mhttpd. See instructions for [https://daq00.triumf.ca/DaqWiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29 CentOS 7].
 
Start [[mhttpd]] on the ''experiment host'' (localhost) like this:
<small>[mhostpc] mhttpd</small>
 
You will get the following messages:
<small>
  [mhttpd,ERROR] [mhttpd.cxx:13880:mongoose,ERROR] mongoose web server cannot find password file "/home/johnfoo/online/htpasswd.txt"
  [mhttpd,ERROR] [mhttpd.cxx:13881:mongoose,ERROR] please create password file: touch /home/johnfoo/online/htpasswd.txt
  [mhttpd,ERROR] [mhttpd.cxx:13818:mongoose,ERROR] mongoose web server password file "/home/johnfoo/online/htpasswd.txt" has no passwords for realm "exptname"
  [mhttpd,ERROR] [mhttpd.cxx:13819:mongoose,ERROR] please add passwords by running: htdigest /home/johnfoo/online/htpasswd.txt exptname midas
  mongoose_init: Error: Cannot initialize authorization object!
  Error: Could not start the mongoose web server, see messages and midas.log, bye!
</small>
 
Create the password file by following the instructions printed by mhttpd. The http digest domain name is the experiment name, suggested default user name is "midas". You will be asked to type in a password. (htdigest is part of the httpd-tools package)
 
<small>
[mhostpc] htdigest -c /home/johnfoo/online/htpasswd.txt exptname midas
Adding password for midas in realm exptname.
New password:
Re-type new password:
</small>
It is a good idea to set the password file {{Filepath|path=htpasswd.txt}} readable and writable by owner only.


== Run the MIDAS Web Server mhttpd ==
If you are running behind a firewall or plan to set up an HTTPS/SSL proxy, follow the instructions
[[Setup MIDAS experiment#mhttpd using an HTTPS/SSL proxy|mhttpd using an HTTPS/SSL proxy]].


Otherwise, the default HTTPS/SSL web server (Mongoose) will be used.
Run {{Utility|name=mhttpd}} again.
Now try to start [[mhttpd]] like this:
$ mhttpd
You will get the following messages:
[mhttpd,INFO] ODB subtree /Runinfo corrected successfully
Mongoose web server will listen on ports "8080r,8443s"
Mongoose web server will use SSL certificate file "/home/johnfoo/packages/midas/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17633:mongoose,ERROR] mongoose web server cannot find password file "/home/johnfoo/online/htpasswd.txt"
[mhttpd,ERROR] [mhttpd.cxx:17634:mongoose,ERROR] please create password file: htdigest -c /home/johnfoo/online/htpasswd.txt Default midas
could not start the mongoose web server, see messages and midas.log, bye!


For testing, use the SSL certificate file provided in the MIDAS package (unless your site provides one, or you wish to create one of your own, see [[Mhttpd#Create an SSL certificate|Create an SSL certificate]]).  
If you get error messages like <small><code>https port "[::]:8443" requested, but mhttpd compiled without MG_ENABLE_SSL</code></small> then you need to re-compile MIDAS after [[Mhttpd#Build_mhttpd_with_SSL_support|adding SSL support]].


However, you do need to create a password file by following the above instructions. You will be asked to type in a password
Otherwise, you will get the following messages:
<small>
  HTTP Digest authentication with realm "exptname" and password file "/home/johnfoo/online/htpasswd.txt"
  Hostlist off, connections from anywhere will be accepted
  Listening on "http://localhost:8080", passwords OFF, hostlist OFF
  Listening on "http://[::1]:8080", passwords OFF, hostlist OFF
  [mhttpd,ERROR] [mhttpd.cxx:15739:mongoose_listen,ERROR] cannot find SSL certificate file "/home/johnfoo/online/ssl_cert.pem"
  [mhttpd,ERROR] [mhttpd.cxx:15740:mongoose_listen,ERROR] please create SSL certificate file using openssl: cd $MIDASSYS; openssl req -new -nodes -newkey rsa:2048 -sha256 -out ssl_cert.csr -keyout ssl_cert.key -subj "/C=/ST=/L=/O=midas/OU=mhttpd/CN=localhost"; openssl x509 -req -days 365 -sha256 -in ssl_cert.csr -signkey ssl_cert.key -out ssl_cert.pem; cat ssl_cert.key >> ssl_cert.pem
  [mhttpd,ERROR] [mhttpd.cxx:15741:mongoose_listen,ERROR] or using certbot (recommened): setup certbot per Let's Encrypt instructions, certificates are typically saved in /etc/letsencrypt/live/$HOSTNAME/, copy fullchain.pem and privkey.pem to $MIDASSYS; cd $MIDASSYS; cat fullchain.pem privkey.pem > ssl_cert.pem
  [mhttpd,ERROR] [mhttpd.cxx:15918:mongoose_init,ERROR] Failed to listen on a TCP port enabled in ODB /WebServer
  Error: Could not start the mongoose web server, see messages and midas.log, bye!
</small>


$  htdigest -c /home/johnfoo/online/htpasswd.txt Default midas
Create a self-signed certificate suitable for initial testing by executing the commands printed by mhttpd.
Adding password for midas in realm Default.
New password:
Re-type new password:


Now restart mhttpd
For production use, you should create a properly signed certificate, see [[Mhttpd#Create an SSL certificate|create your own SSL certificate]] or you should run mhttpd behind an SSL/https proxy (apache httpd).
$ mhttpd
Mongoose web server will listen on ports "8080r,8443s"
Mongoose web server will use SSL certificate file "/home/suz/packages/midas/ssl_cert.pem"
Mongoose web server will use authentication realm "Default", password file "./htpasswd.txt"


Now point a web browser running on the same host computer (localhost) to https://localhost:8443/
(If the web browser is running on a different computer, go to URL
https://mhostpc.triumf.ca:8443  where you substitute your host machine name for "mhostpc.triumf.ca").


If you are using the default SSL certificate you will probably get a "This Connection is Untrusted" messageClick "I understand the risks" and add an exception. This is because the test certificate is self-signed. Then confirm an exception.
Now restart  {{Utility|name=mhttpd}}
<small>[mhostpc] mhttpd
HTTP Digest authentication with realm "test_pol" and password file "/home/johnfoo/online/htpasswd.txt"
Hostlist off, connections from anywhere will be accepted
Listening on "http://localhost:8080", passwords OFF, hostlist OFF
Listening on "http://[::1]:8080", passwords OFF, hostlist OFF
Mongoose web server will use https certificate file "./ssl_cert.pem"
  Listening on "https://[::]:8443", passwords enabled, hostlist OFF
Mongoose web server is using 1 threads
Mongoose web server is using 2 threads
Mongoose web server is using 3 threads
Mongoose web server is using 4 threads</small>


You should then see an authentication box asking you for the user name and password. The user name is "midas". Enter the password you just created. Then you should see the MIDAS [[Status Page]] for your experiment.
Now point a web browser running on the same host computer (localhost) to https://localhost:8443.
If the web browser is running on a different computer, go to URL of the form
<small>
https://mhostpc.triumf.ca:8443  (substitute your host machine name and domain for "mhostpc.triumf.ca")
</small>
If you are using the default SSL certificate you will probably get a message: "This Connection is Untrusted".  Click "I understand the risks" and add an exception. This is because the test certificate is self-signed. Then confirm an exception.  


Note that the default ports of 8080 and 8443 are used by [[mhttpd]]. If these ports are in use on your machine, start mhttpd with alternative ports, e.g.
You should then see an authentication box asking you for the user name and password. The user name is "midas". Enter the password you just created. The Midas [[Status Page]] should appear with multiple buttons for run control as well as equipment listing (no equipments will be listed as yet) and application listings. Please refer to [[mhttpd]] (the MIDAS Web-based Run Control utility) for further information. You can start and stop runs from the main status page, and use the [[ODB Page]] to access the database (ODB).
$ mhttpd --https 8448 --http 8089
or see [[Mhttpd#Usage]] to change the default ports.


; Note
: Default ports of 8080 and 8443 are used by [[mhttpd]]. To use some other ports, change the relevant values in the [[Webserver_ODB_tree|Webserver ODB tree]].


== Create a start_daq script ==
== Run the MIDAS logger ==
Create $HOME/online/bin/start_daq.sh as follows, (supply the mhttpd ports if default is not used).
Start the MIDAS logger [[mlogger]] on the ''experiment host'' (localhost) :
<small>[mhostpc] mlogger</small>
This should start without error. It is usually run as a daemon, however in this case it is run in a terminal to check for errors. Starting the midas logger [[mlogger]] will automatically create more keys in the {{Odbpath|path=/Logger}} ODB tree.


  #!/bin/sh
From the MIDAS status page, go to the "Programs" page, then the "Logger" page and set ODB variables:
* Required -> set to "y"
* Start command -> set to "mlogger -D"
* Alarm Class -> set to "Alarm"
 
To test this, go to the "Programs" page, press "stop logger", then "start logger".
 
Start a run, wait 10 seconds, stop the run.
 
From the MIDAS "Status" page, go to the "logging channels -> channel #0" and set ODB variables:
* Output -> "FILE"
 
Start a run, go to the "Status" page, observe that the output file is written with lz4 compression and has the "mid.lz4" extension.
 
Stop the run.
 
== Run test frontend ==
 
Run <code>fetest</code>
 
On the MIDAS "Status" page there will be 5 new equipment entries:
 
{| class="wikitable"
|+ Equipment created by the fetest program
|-
! Equipment name !! Data it sends !! How often data is sent
|-
| test_random || Random data banks || 1Hz
|-
| test_slow || Example slow control data for testing the midas history plots || 1Hz, even when a run isn't in progress
|-
| test_rare || Data that look like a sine wave || 0.1Hz, even when a run isn't in progress
|-
| test_bulk || Large events || 1Hz
|-
| test_rpc || None. It shows an example of a "remote procedure call" (RPC) handler. Javascript/C++/Python code can connect to the program, and it will respond with the current time (in this toy example) || never
|}
 
Note the event counters for test_slow and test_rare are already incrementing.
 
Start a run.
 
Observe that the event counters for test_bulk and test_random are also incrementing now, and that the output data file size is growing.
 
Click on the "Event dump" page and then the "Run" button. You'll see the raw data that is being written to disk. (The data from most of the equipment is double-precision floating point numbers; the data from the "bulk" equipment is just a bunch of zeros).
 
Return the status page and stop the run.
 
== Create a history plot ==
 
Go to the MIDAS "History" page.
 
Say "new", panel name "test". In the history editor select event "test_slow", and click the "save" button (you can ignore the rest of the options for now). If the "test_slow" event isn't listed, make sure that the mlogger program is running, as described above.
 
History plot "test" should open with a sine-wave plot. Data on the plot should update as long as "fetest" is running (there is no need to start a run).
 
== Clients run on Localhost only ==
;NOTE:
* If creating a MIDAS experiment with a '''REMOTE''' frontend, '''continue by following the instructions''' [[#Running with one or more REMOTE frontends]].
* If all clients are running on the Experiment Host (i.e. '''localhost'''), continue with the following instructions:
 
=== Frontend and Analyzer (localhost) ===
 
THIS SECTION IS VERY OBSOLETE
 
There are several examples of [[Frontend user code]] ({{File|name=frontend.c}}) in the MIDAS package available under {{Filepath|path=$MIDASSYS/examples/}}. Choose a suitable example from this directory that you can later modify for your own particular setup (e.g. for a slow control, choose the {{Filepath|path=../slowcont/}} example).
 
A '''frontend''' is a program that usually reads data from the hardware and sends it to a buffer to be logged and analyzed. You can find documentation about the frontend structure under [[Frontend Application]], [[Frontend Operation]] and [[Frontend user code]].
 
The example chosen here is from {{Filepath|path=$MIDASSYS/examples/experiment/}}. It does '''not need any hardware''' and produces simulated events. 
 
On the MIDAS ''experiment host'' (localhost), copy the example to the experiment directory and build it.
<small>
      [mhostpc] cd $HOME/online
      [mhostpc] cp $MIDASSYS/examples/experiment/* .
      [mhostpc] make</small>
The analyzer will only build if ROOT has previously been installed.
At this point the frontend and the analyzer should be ready if no errors were generated during the build. Try the frontend and analyzer by starting them in xterms.
<small>
      [mhostpc] frontend
      [mhostpc] analyzer</small>
 
=== Equipment ODB tree ===
Starting the frontend will automatically create the  {{Odbpath|path=/Equipment}} ODB tree and one or more equipments. In this case, the equipments "trigger" and "scaler" have been defined by code in  {{File|name=frontend.c}} and created the first time frontend runs.
<small>
    [mhostpc] odbedit
    [local:e777:S]/>ls /equipment
    Trigger                       
    Scaler  </small>
View the MIDAS main status web page (by pointing your browser to https://localhost:8443/ or appropriate port if not using the default).
Observe on the [[Status Page]] that the two Equipments (Trigger, Scaler) have appeared in the Equipment section (Figure 1) and are coloured green.
<br>
'''Click thumbnail to enlarge'''
[[File:quickstart_status_page.png|thumb|left|Figure 1: web server status page showing example frontend equipments]]
<div style="clear: both"></div>  <!-- clear wraparound after thumbnail -->
 
Rather than using odbedit,  you can view the contents of these equipments by clicking on the ODB button, then on {{Odbpath|path=/Equipment}}, then {{Odbpath|path=Trigger}} or {{Odbpath|path=Scaler}}. See [[ODB Page]] for more information.
 
=== Start a run ===
Start a run by pressing the Start button on the [[Status Page]].
The Equipment display on the Status Page will now show some event statistics (Figure 2)
'''Click thumbnail to enlarge'''
[[File:sample_frontend_status.png|thumb|left|Figure 2: Equipments shown on Status Page showing event statistics]]
<div style="clear: both"></div>  <!-- clear wraparound after thumbnail -->
 
 
 
The frontend will show an event display that will update when a run is started, e.g.
 
Sample Frontend connected to <local>. Press "!" to exit      18:48:2546:08
==========================================================================
Run status:  Running    Run number 2                                  |/  /
===========================================================================
Equipment    Status    Events    Events/sec Rate[B/s]  ODB->FE FE->OD
---------------------------------------------------------------------------
Trigger      OK        13948      99.0      5413.0    0          140       
Scaler        OK        15        0.3        15.4      0          15   
 
The [[Status Page]] will also show events being generated (click the refresh button if necessary).
 
=== Event Dump ===
While a run is in progress, the midas application [[mdump]] will provide you an event dump of the collected data from the running frontend, e.g.
<small>
  mhostpc> mdump -l 50
<span style="color:whitesmoke">blank</span>
- MIDAS revision: Mon Nov 2 11:50:51 2015 -0800 - 3b66779 -- Enter <!> to Exit ------- Midas Dump ---
------------------------ Event# 1 ------------------------
Evid:0001- Mask:0000- Serial:0- Time:0x567a0c5d- Dsize:40/0x28
#banks:2 - Bank list:-ADC0TDC0-
<span style="color:whitesmoke">blank</span>
Bank:ADC0 Length: 8(I*1)/2(I*4)/4(Type) Type:Unsigned Integer*2
  1-> 0x0167 0x03c6 0x0069 0x0073
  <span style="color:whitesmoke">blank</span>
Bank:TDC0 Length: 8(I*1)/2(I*4)/4(Type) Type:Unsigned Integer*2
  1-> 0x0051 0x00ff 0x004a 0x00ec
</small>
For further data processing/analysis, either the midas analyzer or the rootana can used for data display as well.
 
=== Create a startup script (localhost) ===
It is useful to create a script that will automatically start all the required clients for the experiment. This is called a startup script. Here, the startup script will be  {{File|name=start_daq.sh}} (any name can be used). Create  {{Filepath|path=$HOME/online/bin/start_daq.sh}} on the ''experiment host'' as follows (supply the [[mhttpd]] ports if default is not used):
 
<small>#!/bin/sh
  # start_daq.sh
  # start_daq.sh
  cd $HOME/online
  cd $HOME/online
  odbedit -c clean
  odbedit -c clean
  #  start mhttpd on default port. (Mongoose https version)
  #  start mhttpd on default port. (Mongoose https version)
  mhttpd  -D  # optionally restrict access to localhost and/or other specified hosts - see [[mhttpd]]
  mhttpd  -D  # you can optionally restrict access to localhost and other specified hosts - see [[mhttpd]]
  xterm -e ./frontend &
  xterm -e ./frontend &
  xterm -e ./analyzer &
  xterm -e ./analyzer &
  mlogger -D
  mlogger -D
  #end file
  #end file</small>


Before running this script, you will need to shutdown mhttpd and mlogger if they are running. This script will start them as daemons. You will also need to [[#Create a frontend and analyzer]] if desired.  
Before running this script, you will need to shutdown any running clients:
<small>mhostpc> odbedit -c "sh all"</small>
or you can use the  {{Utility|name=mhttpd}} [[Programs Page]] to shut them down.


== Create a frontend and analyzer ==
The script will start them as daemons.
There are several examples available under $MIDASSYS/examples/. Choose a suitable example from this directory that you can modify later for your own particular setup (e.g. for a slow control, choose the slowcont/ example). The example chosen here is $MIDASSYS/examples/experiment/ .
By running the startup script start_daq.sh, several MIDAS applications will be started in sequence:
# Cleanup previous MIDAS application (if any).
# Start the MIDAS web server [mhttpd]
# Start the frontend application in its own xterm (for debugging purpose).
# Start the analyzer application in its own xterm (for debugging purpose).
# Start the MIDAS Data logger [mlogger]
  <small>[mhostpc] sh ./start_daq</small>


On the MIDAS host (localhost), copy the example to the experiment directory and build it.
You may wish to modify the script to restart missing clients only.  
      cd $HOME/online
      cp $MIDASSYS/examples/experiment/* .
      make
The analyzer will build properly if ROOT has been previously installed.
    At this point the frontend and the analyzer should be ready if no errors were generated during the build. You can try the frontend and analyzer by starting them by hand, or run the script start_daq.sh (see [[#Create a start_daq script]]).


By running the script start_daq.sh, several midas applications will be started in sequence.
Next [[#Customizing your experiment|customize your experiment]].
        Cleanup previous midas application (if any).
        Start the midas web server [mhttpd]
        Start the frontend application in its own xterm (for debugging purpose).
        Start the analyzer application in its own xterm (for debugging purpose).
        Start the Midas Data logger [mlogger]


  $ sh ./start_daq
== Running with one or more REMOTE frontends ==
These instructions assume you have already followed the instructions to setup an experiment running on the MIDAS experiment host (localhost), i.e. you have setup the MIDAS Environment variables, downloaded MIDAS packages, setup the [[mhttpd]] password and have {{Utility|name=mhttpd}} running successfully.


Starting a frontend will automatically create the /Equipment ODB tree and one or more equipments. Starting the midas logger [[mlogger]] will automatically create more keys in the /Logger ODB tree.
In the case of '''remote''' frontend(s), the remote cpu(s) typically have access to some or all of the hardware. They might be VMIC cpus running in VME crates for example, connected to the MIDAS Host via Ethernet.  They read data from the hardware and send it back to the MIDAS ''experiment host'' to be logged and analyzed.


View the MIDAS main status web page by pointing your browser to https://localhost:8443/ (or appropriate port if not using the default).
=== Experimental setup (Remote) ===
The Midas run status page should appear with multiple buttons for run control as well as equipment listing and application listings. Please refer to [[mhttpd]] (the MIDAS Web-based Run Control utility) for further information. You can start and stop runs from the main status page, and use the [[ODB Page]] to access the database (ODB). You will want to customize the ODB for your experiment.
These instructions assume that the MIDAS ''experiment host'' computer is 64-bit (if not see [[#Build 32-bit MIDAS libraries|note below]]), and there is one ''remote host'', a 32-bit machine named "rlxhost".  


While a run is in progress, the midas application [[mdump]] will provide you an event dump of the collected data from the running frontend. For further data processing/analysis, either the midas analyzer or the rootana can used for data display as well.
The ''remote host'' mounts the {{Filepath|path=/home}} disk of the ''experiment host'', so that it '''has access to the MIDAS packages''', the {{Filepath|path=$HOME/online}} directory and shares the .cshrc (or .bashrc) script. It is convenient to set up ssh keys (use ssh-keygen) so that the ''remote host'' can be accessed without supplying the password from the ''experiment host''.


   
If your setup is different, you will have to make changes as appropriate. 


== Running with one or more REMOTE frontend(s) ==
=== Build 32-bit MIDAS libraries ===
These instructions assume you have already setup an experiment running on localhost, i.e. you have setup the MIDAS Environment variables, downloaded MIDAS packages, setup the mhttpd password etc.
NOTE
<p style="margin-left:40px;margin-right:40px;">
If the MIDAS ''experiment host'' computer is 32-bit (i.e. the same as the ''remote host''), which is a simpler setup, you will have built the 32-bit MIDAS libraries already, and can skip the rest of this section. Note however, that in the .cshrc file in the  [[#Modify .cshrc (.bashrc) for Remote host|following section]] you will not need the lines starting "# select 64-bit or 32-bit MIDAS and ROOT".  The PATH should contain <span style="color:#214200; font-weight:normal; font-style:italic">$MIDASSYS/bin</span> on both experiment host and remote host. This will point to the 32-bit MIDAS libraries.  
------------------------------
</p>
<br>
Assuming the MIDAS host computer is 64-bit, you will have already built the 64-bit MIDAS libraries.
In this case, if the remote frontend is 32-bit, you will need to build the 32-bit MIDAS libraries on the 64-bit machine, i.e.
<small>
    [mhostpc] cd /home/packages/midas
    [mhostpc] make linux32                ### build the 32-bit MIDAS libraries on 64-bit machine
    [mhostpc] ls -l linux-m32/bin/odbedit ### check that the 32-bit odbedit has been created </small>
Do not run  {{Utility|name=odbedit}} yet on the ''remote host''!


Follow these instructions to make the modifications needed to run an experiment with remote frontends.
If the "make linux32" command fails, you may need to install 32bit support for your compiler (e.g. "apt install g++-multilib").


These instructions also assume that there is one '''remote''' cpu, which is a 32-bit machine which mounts the /home disk of the localhost machine, so that it has access to the MIDAS packages, the $HOME/online directory and shares the .cshrc (or .bash) script. The REMOTE Host will be called "rlxhost" in these instructions.
=== Modify .cshrc (.bashrc) for Remote host ===
Add the following to the {{File|name=.cshrc}} (modified appropriately if using {{File|name=.bashrc}}). When running with a REMOTE Host, the application [[mserver]] will be started (using a particular port) on the MIDAS ''experiment host'' machine only.  Note that the environment variable MIDAS_SERVER_HOST is defined on the remote machine(s) but NOT on the ''experiment host''.  


If the MIDAS host computer (localhost) is 64-bit, and the frontend is 32-bit, you will need to build the 32-bit MIDAS libraries on the 64-bit machine, i.e.
If ROOT has been installed properly and the script <ROOT installation directory>/bin/thisroot.sh is called from .cshrc or .bashrc, the ROOTSYS environment variable should be defined and the ROOT bin directory should be in the PATH environment variable. Make sure that this is the case.
    cd /home/packages/midas
    make linux32                ### build the 32-bit MIDAS libraries on 64-bit machine
    ls -l linux-m32/bin/odbedit ### check that the 32-bit odbedit has been created (do not run it)


=== Modify .cshrc (.bash) ===
<small>
Add the following to the .cshrc (modified appropriately for .bash). When running with a REMOTE Host, the application [[mserver]] will be started (using a particular port) on the MIDAS host machine only.  Note that the environment variable MIDAS_SERVER_HOST is defined on the remote machine(s) but not on the MIDAS host.
  # setenv MIDAS_HOST mhostpc  # substitute your experiment host name
# setenv MIDAS_HOST mhostpc  # substitute your midas host name
# setenv REMOTE_HOST rlxhost # substitute your remote host name
# setenv REMOTE_HOST rlxhost   # substitute your remote host name (add more if > 1)
# setup the MIDAS mserver host
# setup the MIDAS mserver host
#
#
switch (`hostname`)
switch (`hostname`)
case $MIDAS_HOST*:                 
case $MIDAS_HOST*:                 
   unsetenv MIDAS_SERVER_HOST    ## MIDAS_SERVER_HOST not defined
   unsetenv MIDAS_SERVER_HOST    ## MIDAS_SERVER_HOST not defined
   breaksw
   breaksw
default:
default:
setenv MIDAS_SERVER_HOST $MIDAS_HOST.triumf.ca:1175  ## for remote host, define as hostname + domain name + default mserver port
setenv MIDAS_SERVER_HOST $MIDAS_HOST.triumf.ca:1175  ## for remote host, define MIDAS_SERVER_HOST, use your domain here
                                                ## port must match that of [[mserver]] used in start_daq.sh  
                                                      ## as hostname + domain name + default mserver port
endsw
                                                      ## port must match that of [[mserver]] used in start_daq.sh  
#
endsw
# select 64-bit or 32-bit MIDAS and ROOT
#
#
# select 64-bit or 32-bit MIDAS
switch (`uname -i`)
#
case i386:
switch (`uname -i`)
  setenv ROOTSYS /triumfcs/trshare/olchansk/root/root_v5.28.00_SL55_32
case i386:
   setenv PATH .:$MIDASSYS/linux-m32/bin:$PATH
   setenv PATH .:$MIDASSYS/linux-m32/bin:$PATH
   breaksw
   breaksw
default:
default:
  setenv ROOTSYS $HOME/packages/root
   setenv PATH .:$MIDASSYS/linux/bin:$PATH
   setenv PATH .:$MIDASSYS/linux/bin:$PATH
endsw
endsw
#
#
setenv PATH .:$HOME/online/bin:$HOME/packages/roody/bin:$ROOTSYS/bin:$PATH
setenv PATH .:$HOME/online/bin:$HOME/packages/roody/bin:$PATH
#
#
</small>
After executing {{File|name=.cshrc}} (or {{File|name=.bashrc}}) or logging out, on the MIDAS host (mhostpc) make sure environment variable MIDAS_SERVER_HOST is NOT defined
<small>[mhostpc] echo $MIDAS_SERVER_HOST
[mhostpc] MIDAS_SERVER_HOST: Undefined variable.</small>


After executing .cshrc (or .bash) or logging out, on the MIDAS host (mhostpc) make sure MIDAS_SERVER_HOST is NOT defined
and on the '''''remote host''''', make sure MIDAS_SERVER_HOST IS defined:
echo $MIDAS_SERVER_HOST
  <span style="color:green; font-size:80%">[rlxhost] echo $MIDAS_SERVER_HOST
MIDAS_SERVER_HOST: Undefined variable.
  [rlxhost] MIDAS_SERVER_HOST</span>


and on the REMOTE Host, make sure MIDAS_SERVER_HOST IS defined:  
Also on the ''remote host'', make sure that the correct (32-bit)  {{Utility|name=odbedit}} will be used (do not run it yet!)
  echo $MIDAS_SERVER_HOST
<span style="color:green; font-size:80%">[rlxhost] which odbedit
  MIDAS_SERVER_HOST
/home/johnfoo/packages/midas/linux-m32/bin/odbedit</span>


Also on the REMOTE Host, make sure that the correct (32-bit) odbedit will be used (do not run it yet!)
=== Grant Remote host access permission ===
[johnfoo@rlxhost e777]$ which odbedit
Give permission for the remote host(s) to access the experiment by following the instructions to [[Security#MIDAS programs on remote machines|allow MIDAS programs on remote machines]].  If the MIDAS web server [[mhttpd]] is running, use [[ODB Page]] to edit the ODB keys listed, otherwise use [[odbedit]] on the ''experiment host'' machine.
/home/johnfoo/packages/midas/linux-m32/bin/odbedit
=== Start mserver on Experiment Host ===
The application [[mserver]] is not required when running an experiment on the ''experiment host'' (localhost) only, but it '''IS required''' for access from a '''REMOTE''' frontend.
Start [[mserver]] on MIDAS host (default port 1175) or use the "-p" option for a different port.
The port must match the port defined for environment variable MIDAS_SERVER_HOST in {{File|name=.cshrc}} (or {{File|name=.bashrc}}) [[#Modify .cshrc (.bashrc) for Remote host|above]].
  <small>[mhostpc] mserver -D</small>


=== Grant REMOTE host access permission ===  
The environment variables on the REMOTE host should be:
Give permission for the remote host(s) to access the experiment. Follow the instructions [[Security#MIDAS programs on remote machines||Allow MIDAS programs on remote machines]].  If the MIDAS web server is running, use [[ODB Page]] to edit the ODB keys listed, otherwise use [[odbedit]] on the MIDAS host machine.
<span style="color:green; font-size:80%">
    [rlxhost] printenv | grep MIDAS
=== Modify experiment startup script ===
    MIDASSYS=/home/e777/packages/midas
Modify the experiment startup script start_daq.sh.
    MIDAS_EXPT_NAME=e777
The application [[mserver]] is not required when running on localhost only, but it IS required for access from a REMOTE frontend.
    MIDAS_SERVER_HOST=mhostpc.triumf.ca:1175</span>
#!/bin/sh
(a different port may be used - see above)
# start_daq.sh
 
cd $HOME/online
=== Check access to ODB from Remote host ===
#
Run {{Utility|name=odbedit}} on the remote host to check access to ODB.
switch (`hostname`)
<span style="color:green; font-size:80%">[rlxhost] odbedit
case $MIDAS_HOST*:
[mhostpc.triumf.ca:1175:e777:Stopped]/>ls </span>
    echo "Good, we are on $MIDAS_HOST"
If it does not work correctly, check that
    breaksw
* MIDAS_SERVER_HOST is defined on REMOTE host with same port as [[mserver]]
case $REMOTE_HOST*:
* <span style="color:darkcyan;font-style:italic">mserver</span> is running on MIDAS host with the correct port
    echo "start_daq script should be executed on $MIDAS_HOST"
* MIDAS_SERVER_HOST is NOT defined on MIDAS host
endsw
* remote access permission to REMOTE host has been granted as described above.
#
 
odbedit -c clean
=== Build the analyzer on Experiment Host ===
#  start mhttpd on default port (Mongoose https version)
The analyzer will be built and run on the MIDAS ''experiment host'' machine.
mhttpd  -D  # optionally restrict access to localhost and/or other specified hosts - see [[mhttpd]]
Select an example from $MIDASSYS/examples/ (see [[#Frontend and Analyzer (localhost)]]) and copy it to $HOME/online on the MIDAS host machine. Since we are assuming a 64-bit MIDAS host and a 32-bit remote host, edit the example Makefile so only the analyzer will be built.
#  start mserver on default port (use argument -p to use a different port)  
 
mserver -D  # start mserver on default port (1175) or a different port using -p option - see [[mserver]]
=== Build the frontend (32bit) for Remote host ===
            # the port must match that of MIDAS_SERVER_HOST in .cshrc (.bash) - see above
The frontend runs on a 32-bit remote host in this example. It therefore must either
            # access must be specifically allowed - see above
* be built 32-bit on the 64-bit MIDAS host with the "-m32" flag
xterm -e ./analyzer &
* or built on the REMOTE host
## start frontend on remote host
and in either case the [[#Build 32-bit MIDAS libraries|32-bit MIDAS libraries]] used for linking.
#ssh $REMOTE_HOST $HOME/online/bin/start_frontend -O >& $HOME/online/bin/start_frontend.log  &
 
mlogger -D
To avoid confusion, create a different subdirectory for the frontend, e.g.
#end file
  <span style="color:green; font-size:80%">[rlxhost] mkdir $HOME/online/fe_32</span>
Copy the Makefile and frontend code from the same example to this directory.
Modify the Makefile as needed, and only build the frontend. 
 
Once built, start the frontend in an xterm on the REMOTE host.  
  <span style="color:green; font-size:80%">[rlxhost] $HOME/online/fe_32/frontend</span>
Check that it connects successfully to the experiment and the Equipments appear on the [[Status Page]] as described under [[#Frontend and Analyzer (localhost)]].  The Equipment Status (Figure 2) should show that the Equipments are now running on the remote cpu, i.e. "SampleFrontend@rlxhost.triumf.ca" rather than "Sample Frontend@localhost".
 
=== Run the Analyzer on Experiment host ===
Start the analyzer in an xterm on the MIDAS ''experiment host''
<small>[johnfoo@mhostpc e777] $HOME/online/analyzer</small>
 
Continue by starting a run and dumping the data as described under [[#Frontend and Analyzer (localhost)]].
 
=== Create experiment startup script ===
It is useful to create a script that will automatically start all the required clients for the experiment. This is called a startup script (any name can be used). Create  {{Filepath|path=$HOME/online/bin/start_daq.sh}} on the ''experiment host'' as follows, (supply the [[mhttpd]] ports if default is not used) :
<small>
#!/bin/sh
# start_daq.sh
cd $HOME/online
#
switch (`hostname`)
  case $MIDAS_HOST*:   ## MIDAS_HOST is assigned in .cshrc (.bashrc)
      echo "Good, we are on $MIDAS_HOST"
      breaksw
  case $REMOTE_HOST*:   ## REMOTE_HOST is assigned in .cshrc (.bashrc)
      echo "startup script should be executed on $MIDAS_HOST"
endsw
#
odbedit -c clean
#  start mhttpd on default port (Mongoose https version)
mhttpd  -D  # you can optionally restrict access to localhost and/or other specified hosts - see [[mhttpd]]
#  start mserver on default port (use argument -p to use a different port)  
mserver -D  # start mserver on default port (1175) or a different port using -p option - see [[mserver]]
            # the port must match that of MIDAS_SERVER_HOST in .cshrc (.bashrc) - see above
            # access must be specifically allowed - see [[#Grant Remote host access permission||above]]
xterm -e ./analyzer &
## start frontend on remote host
#ssh $REMOTE_HOST $HOME/online/bin/start_frontend -O >& $HOME/online/bin/start_frontend.log  &
mlogger -D
#end file
</small>
Note that command to start the frontend on the remote host is commented out. It can be uncommented once the start_frontend script is written and tested.
 
Shut down any clients that are running already by running odbedit on the MIDAS ''experiment host'' and execute the new startup script  {{File|name=start_daq.sh}} (make it executable if desired)
<small>
  [mhostpc] odbedit
  [local:e777:S]/>sh all
  [mhostpc] ./start_daq.sh </small>
and check that {{Utility|name=odbedit}} still works on the ''REMOTE host''
<span style="color:green; font-size:80%">
    [rlxhost] odbedit
    [mhostpc.triumf.ca:1175:e777:Stopped]/>ls </span>
 
You may wish to modify the script to restart missing clients only.
 
 
=== Create script to start frontend === 
The startup file  {{File|name=start_daq.sh}}  invoked a script  {{File|name=start_frontend}} to start the remote frontend. This line was commented out (see [[#Create experiment startup script|above]]). Once the frontend is working, create a script to start the frontend (e.g. file {{Filepath|path=$HOME/online/bin/start_frontend}}).
<small>
#!/bin/tcsh
# Script to start frontend running on $REMOTE_HOST
#
# This script runs on the REMOTE HOST
#
echo "starting frontend for experiment $MIDAS_EXPT_NAME "
xterm -geometry 150x50+800+0 -fg white -bg blue -title "Frontend" -e  "$HOME/online/fe_32/frontend" &
echo "Frontend task has been started"
exit
</small>
Check that this script works by running it on the REMOTE host. If successful,
run the commented command in an xterm on the ''experiment host''
    <small>[mhostpc] ssh $REMOTE_HOST $HOME/online/bin/start_frontend -O >& $HOME/online/bin/start_frontend.log  & </small>
This command relies on password-less access set up with SSH keys (see [[#Experimental setup (Remote)]]).
If successful, uncomment line in {{File|name=start_daq.sh}}, and run the script to check that it can start the remote client(s) from the MIDAS ''experiment host'' (first shutting down all running clients) :
<small>
    [mhostpc] odbedit -c "sh all"
    [mhostpc] ./start_daq.sh </small>
 
Next [[#Customizing your experiment|customize your experiment]].
 
 
== Customizing your experiment ==
 
Here are some of the more common options provided for users to customize their experiments according to their own preferences:
 
; Write frontend(s) to read out and control your hardware
: Modify the example frontend and Makefile (from $MIDASSYS/examples/) you have already copied and built so it accesses your hardware.
 
: The example Frontend.c is an example of [[Frontend user code]]. You can find documentation about the frontend structure under [[Frontend Application]], [[Frontend Operation]] and [[Frontend user code]]. You may need to set up some [[Event Notification (Hot-Link)|Hot-Links]] in the frontend. The MIDAS package includes device drivers for some of the more common hardware. See [[MIDAS Driver Library]] for more information. 
 
; Modify analyzer(s) for your data
: Modify the example analyzer code as necessary to analyze the data from your frontends.
 
; Slow Controls
: Control your slow-controls equipment with slow-control frontend(s) optionally using the [[#MIDAS Package Installation|MSCB]] system.
 
; Data logger
: Set up the data logger [[mlogger]] to log data to a storage device. Many options are available - see [[Data Logger]], [[mlogger]] and [[/Logger ODB tree]] for instructions.
 
; Allow programs to be restarted from the [[Programs Page]]
: See mhttpd [[Programs Page]] and [[/Programs ODB tree]]
 
; Write scripts which run at start and end of run
: See [[/Programs ODB tree#Execute on start run]] and  [[/Programs ODB tree#Execute on stop run]]


Note that command to start the frontend on the remote host is commented out. It can be uncommented once the start_frontend script is
; Create a shutdown script
written and tested.
: Create a script similar to the startup script to shutdown the experiment


Shut down any clients that are running already by running odbedit on the MIDAS HOST
; Lazy logger
  odbedit>
: Set up the [[lazylogger]] for archiving data files for storage.
[local:e777:S]/>sh all


Execute the new startup script start_daq.sh and check that odbedit now works on the REMOTE host
; Create edit-on-start parameters
: See [[Edit-on-start Parameters]]


[johnfoo@rlxhost e777]$ odbedit
; Set up the alarm system
[mhostpc.triumf.ca:1177:e777:Stopped]/>ls
: See [[Alarm System]] and [[Alarms Page]]


If it does not, check that
; Set up history system
* MIDAS_SERVER_HOST is setup correctly
: See [[History System]] and [[History Page]]
* mserver is running with the correct port
* remote access permission to REMOTE host has been granted
See above for details.


=== Build the frontend ===
; Set up the Electronic logbook (Elog)  
The frontend must be built on the REMOTE Host (a 32-bit machine in this example). To avoid confusion, create a different subdirectory
: See [[Elog]] for details
for the frontend running on the remote host, e.g.
  mkdir $HOME/online/fe32
Create or modify an example frontend in this directory, and modify a Makefile to compile and build it with the 32-bit MIDAS libraries.
Once built, start the frontend by hand (while logged on to the REMOTE host). Check that it connects successfully to the experiment (e.g. with mhttpd Status Page).


=== Create start_frontend script === 
; Set up the Sequencer
Create start_frontend script (e.g. file $HOME/online/bin/start_frontend)
: The [[Sequencer Page#Sequencer]] can be used to run a sequence of runs, changing parameters as needed.


#!/bin/tcsh
; Write a webserver Custom page
# Script to start frontend running on $REMOTE_HOST
: To better control and monitor your experiment, you may want to write a [[Custom Page]].
#
# This script runs on the REMOTE HOST
#
echo "starting frontend for experiment $MIDAS_EXPT_NAME "
xterm -geometry 150x50+800+0 -fg white -bg blue -title "Frontend" -e  "$HOME/$ONLINE/fe32/frontend" &
echo "Frontend task has been started"
exit


Check it works by running it on the REMOTE host.
== Start MIDAS on boot ==
Then uncomment line in start_daq.sh, and check start_daq.sh can start all the clients from the MIDAS host.


To start MIDAS mhttpd, mlogger and elogd at boot time, add following cron entries:
<pre>
[iris@iris00 ~]$ crontab -l
@reboot /home/iris/packages/midas/bin/mhttpd -D
@reboot /home/iris/packages/midas/bin/mlogger -D
@reboot /home/iris/packages/elog/elogd -c /home/iris/packages/elog/elogd.cfgiris -D
[iris@iris00 ~]$
</pre>


== Customize the experiment ==
[[Category:Installation]]
=== data directory ===
To change where data files are stored and the message file midas.log is located away from the default,
create a data directory, e.g.
mkdir /data1/e777/; mkdir /data1/e777/data
[local:e777:S]/>ls /logger  ## list the contents of the /logger tree
Data dir                        /home/johnfoo/online/         
Message file                    midas.log
If desired, set the location for the midas.log and any data files, e.g.
[local:e777:S]/>set "/logger/data dir" /data1/e777/data/
[local:Default:S]/>exit

Latest revision as of 03:18, 3 January 2024


Introduction

This quickstart shows you how to install the MIDAS packages on a linux box and create a MIDAS experiment. The linux box will be called the MIDAS experiment host.

Installation

Log on to the Experiment Host and decide on a name for the MIDAS experiment (the experiment name). In these instructions, the MIDAS packages and experiment will be installed under the username "johnfoo". The ODB and other shared memory buffers for the MIDAS experiment will reside on the experiment host (localhost). MIDAS and the other required packages will be installed under directory $HOME/packages. The MIDAS experiment directory will be $HOME/online and the MIDAS experiment name will be "e777". The name of the experiment host will be "mhostpc".

NOTE
The user should substitute directory paths for the MIDAS packages and experiment directory as required, and names for experiment host, username, experiment name appropriate to his/her own setup.

Experimental setup

These instructions describe setting up an experiment that will run ONLY on the MIDAS experiment host computer (i.e. localhost only). In this case, the MIDAS experiment host has access to any hardware required, and all frontends, logger, analyzer etc. run on the one computer.

Modifications to run an experiment with additional frontend(s) running remotely (i.e. on remote cpu(s)) will also be described. Follow the instructions for setting up a localhost-only experiment until indicated.

In both cases, the main MIDAS applications (e.g. mlogger, mhttpd) run on the MIDAS experiment host.

Environment Variables

The following Environment variables should to be added to the .cshrc or .bashrc file (depending on your shell) in the $HOME directory on the experiment host, so that they will be defined at login. Substitute appropriate values for your own setup.

csh bash comment
setenv MIDASSYS $HOME/packages/midas export MIDASSYS=$HOME/packages/midas Base directory of the MIDAS package
setenv MIDAS_EXPTAB $HOME/online/exptab export MIDAS_EXPTAB=$HOME/online/exptab MIDAS experiment table
setenv MIDAS_EXPT_NAME e777 export MIDAS_EXPT_NAME=e777 MIDAS experiment name
setenv PATH .:$MIDASSYS/bin:$PATH export PATH=$PATH:$MIDASSYS/bin path

Logout and login again, or source .cshrc (source .bashrc) for the changes to take effect.

NOTE
See MIDAS environment variables for a list of environment variables used by MIDAS.

ROOT Package Installation

MIDAS has optional integration with ROOT. ROOT is not required for acquiring data or storing it in midas files, but is used by some of the analysis tools (including the example analyzer that will be built later in this Quickstart). It is also possible to write midas data directly to ROOT files if desired.

If you require ROOT integration, follow the installation instructions on the ROOT website and set the ROOTSYS environment variable.

If you do not require ROOT integration, leave the ROOTSYS variable undefined.

Adding SSL support

We recommend running mhttpd (the MIDAS web server) behind an industry-grade web proxy like apache httpd or nginx. httpd/nginx will then handle the SSL security. It is however also possible to build mhttpd with built-in SSL support. See the mhttpd documentation for more details about web security.

If you wish to enable SSL support in mhttpd itself:

 [mhostpc] cd $MIDASSYS
 [mhostpc] make mbedtls

MIDAS Package Installation

The MIDAS package will be installed on the experiment host in the directory given by MIDASSYS - see #Environment Variables.

Install the MIDAS and other package(s) from the MIDAS git repository in an xterm as follows:

  [mhostpc] mkdir $HOME/packages
  [mhostpc] cd $HOME/packages
  [mhostpc] git clone https://bitbucket.org/tmidas/midas --recurse-submodules

If you already cloned MIDAS and forgot the "--recurse-submodules", you can do a "git submodule update --init --recursive" later. If you want to pull MIDAS later to update to the current version, you need either "git pull --recurse-submodules". To tell git to always recurse, you can set a global flag via "git config submodule.recurse true".

The MIDAS Makefile will auto-detect whether MSCB, ROOT, MYSQL, SQLITE, ODBC etc. are available (or have been installed). If you need these features, make sure they are available before building the MIDAS package.

  [mhostpc] cd $MIDASSYS
  [mhostpc] mkdir build
  [mhostpc] cd build
  [mhostpc] cmake ..                  ### configure build process (see note below about cmake/cmake3)
  [mhostpc] make install              ### build and install midas to bin/ and lib/ directories
  [mhostpc] cd ..
  [mhostpc] ls -l bin/odbedit         ### check that odbedit has been created (do not run it yet)

MIDAS requires cmake version 3.10 or later. On some systems, you have to call "cmake3 ..", on others just "cmake ..". Some systems might require you to install cmake first (see the [https://cmake.org/download/ cmake website; installation is very simple).

In case of problems see

ROOTANA Package Installation

rootana is an optional framework for analyzing midas data in realtime and displaying plots (either by a graphical interface or via webpages). Most of the features are now available in manalyzer, which is included in the base midas install.

If you require the legacy ROOTANA package, you can install it with:

   [mhostpc] cd $HOME/packages
   [mhostpc] git clone https://bitbucket.org/tmidas/rootana
   [mhostpc] cd rootana
   [mhostpc] make

add ROOTANASYS=$HOME/packages/rootana to the login file.

Install the JSROOT package:

cd $HOME/packages
git clone https://github.com/linev/jsroot.git

add JSROOTSYS=$HOME/packages/jsroot to the login file.

ROODY Package Installation

ROODY is a legacy package for displaying live ROOT histograms. It is generally not used by new experiments.

If ROODY is needed,

   [mhostpc] cd $HOME/packages
   [mhostpc] git clone https://bitbucket.org/tmidas/roody
   [mhostpc] cd roody
   [mhostpc] make

Create the Experiment file exptab

In this example, the experimental directory is $HOME/online. Create this directory on the experiment host (localhost):

  [mhostpc] mkdir $HOME/online
  [mhostpc] cd $HOME/online

Then create the exptab file $HOME/online/exptab containing the three parameters:

  1. the experiment name
  2. location of MIDAS shared memory buffers
  3. username

as follows (using parameters for your own experiment):

  [mhostpc] cat > exptab
   e777 /home/johnfoo/online johnfoo

The path of the exptab file is given by MIDAS_EXPTAB - see #Environment Variables.

Create shared memory files

At this point you should be able to run odbedit on the experiment host (localhost). The first time odbedit is run, it will create the required .*.SHM files in the MIDAS experiment directory $HOME/online. These are the saved files for the MIDAS shared memory.

The default directory $HOME/online will contain the MIDAS messages file ( midas.log) and any data files you may create. If you want an alternative location for the data files see #Customizing your experiment.

Run odbedit, and type odbedit command ls to list the default directories.

[mhostpc] odbedit
[local:e777:S]/>ls
Experiment
System
Programs
Logger
Runinfo
Alarms 

Run the MIDAS Web Server

Using apache

Recommended for production systems with maximum security is to run mhttpd behind an industry-standard SSL/https proxy with password protection. We document instructions for apache here, but you could nginx instead.

To do this using apache httpd on CentOS-7, do this:

  • configure apache httpd per instructions for Ubuntu or CentOS 7
  • run mhttpd
  • use web browser to connect to https://mhostpc/
  • enter the user name and password configured in apache httpd
  • the midas status page should open

Standalone (less secure)

For simplified installation, one can use standalone mhttpd with built-in https and password protection. This configuration is only as secure as the underlying mongoose and OpenSSL web server and encryption packages. Use the odbedit tool to edit settings in the Webserver ODB tree:

  • Set "Enable https port" to "y"
  • Set "https port passwords" to "y"

Many current operating systems implement a built-in firewall which will block outside access to mhttpd. See instructions for CentOS 7.

Start mhttpd on the experiment host (localhost) like this:

[mhostpc] mhttpd

You will get the following messages:


 [mhttpd,ERROR] [mhttpd.cxx:13880:mongoose,ERROR] mongoose web server cannot find password file "/home/johnfoo/online/htpasswd.txt"
 [mhttpd,ERROR] [mhttpd.cxx:13881:mongoose,ERROR] please create password file: touch /home/johnfoo/online/htpasswd.txt
 [mhttpd,ERROR] [mhttpd.cxx:13818:mongoose,ERROR] mongoose web server password file "/home/johnfoo/online/htpasswd.txt" has no passwords for realm "exptname"
 [mhttpd,ERROR] [mhttpd.cxx:13819:mongoose,ERROR] please add passwords by running: htdigest /home/johnfoo/online/htpasswd.txt exptname midas
 mongoose_init: Error: Cannot initialize authorization object!
 Error: Could not start the mongoose web server, see messages and midas.log, bye!

Create the password file by following the instructions printed by mhttpd. The http digest domain name is the experiment name, suggested default user name is "midas". You will be asked to type in a password. (htdigest is part of the httpd-tools package)

[mhostpc] htdigest -c /home/johnfoo/online/htpasswd.txt exptname midas
Adding password for midas in realm exptname.
New password:
Re-type new password:

It is a good idea to set the password file htpasswd.txt readable and writable by owner only.


Run mhttpd again.

If you get error messages like https port "[::]:8443" requested, but mhttpd compiled without MG_ENABLE_SSL then you need to re-compile MIDAS after adding SSL support.

Otherwise, you will get the following messages:


 HTTP Digest authentication with realm "exptname" and password file "/home/johnfoo/online/htpasswd.txt"
 Hostlist off, connections from anywhere will be accepted
 Listening on "http://localhost:8080", passwords OFF, hostlist OFF
 Listening on "http://[::1]:8080", passwords OFF, hostlist OFF
 [mhttpd,ERROR] [mhttpd.cxx:15739:mongoose_listen,ERROR] cannot find SSL certificate file "/home/johnfoo/online/ssl_cert.pem"
 [mhttpd,ERROR] [mhttpd.cxx:15740:mongoose_listen,ERROR] please create SSL certificate file using openssl: cd $MIDASSYS; openssl req -new -nodes -newkey rsa:2048 -sha256 -out ssl_cert.csr -keyout ssl_cert.key -subj "/C=/ST=/L=/O=midas/OU=mhttpd/CN=localhost"; openssl x509 -req -days 365 -sha256 -in ssl_cert.csr -signkey ssl_cert.key -out ssl_cert.pem; cat ssl_cert.key >> ssl_cert.pem
 [mhttpd,ERROR] [mhttpd.cxx:15741:mongoose_listen,ERROR] or using certbot (recommened): setup certbot per Let's Encrypt instructions, certificates are typically saved in /etc/letsencrypt/live/$HOSTNAME/, copy fullchain.pem and privkey.pem to $MIDASSYS; cd $MIDASSYS; cat fullchain.pem privkey.pem > ssl_cert.pem
 [mhttpd,ERROR] [mhttpd.cxx:15918:mongoose_init,ERROR] Failed to listen on a TCP port enabled in ODB /WebServer
 Error: Could not start the mongoose web server, see messages and midas.log, bye!

Create a self-signed certificate suitable for initial testing by executing the commands printed by mhttpd.

For production use, you should create a properly signed certificate, see create your own SSL certificate or you should run mhttpd behind an SSL/https proxy (apache httpd).


Now restart mhttpd

[mhostpc] mhttpd
HTTP Digest authentication with realm "test_pol" and password file "/home/johnfoo/online/htpasswd.txt"
Hostlist off, connections from anywhere will be accepted
Listening on "http://localhost:8080", passwords OFF, hostlist OFF
Listening on "http://[::1]:8080", passwords OFF, hostlist OFF
Mongoose web server will use https certificate file "./ssl_cert.pem"
Listening on "https://[::]:8443", passwords enabled, hostlist OFF
Mongoose web server is using 1 threads
Mongoose web server is using 2 threads
Mongoose web server is using 3 threads
Mongoose web server is using 4 threads

Now point a web browser running on the same host computer (localhost) to https://localhost:8443. If the web browser is running on a different computer, go to URL of the form

https://mhostpc.triumf.ca:8443  (substitute your host machine name and domain for "mhostpc.triumf.ca")

If you are using the default SSL certificate you will probably get a message: "This Connection is Untrusted". Click "I understand the risks" and add an exception. This is because the test certificate is self-signed. Then confirm an exception.

You should then see an authentication box asking you for the user name and password. The user name is "midas". Enter the password you just created. The Midas Status Page should appear with multiple buttons for run control as well as equipment listing (no equipments will be listed as yet) and application listings. Please refer to mhttpd (the MIDAS Web-based Run Control utility) for further information. You can start and stop runs from the main status page, and use the ODB Page to access the database (ODB).

Note
Default ports of 8080 and 8443 are used by mhttpd. To use some other ports, change the relevant values in the Webserver ODB tree.

Run the MIDAS logger

Start the MIDAS logger mlogger on the experiment host (localhost) :

[mhostpc] mlogger 

This should start without error. It is usually run as a daemon, however in this case it is run in a terminal to check for errors. Starting the midas logger mlogger will automatically create more keys in the /Logger ODB tree.

From the MIDAS status page, go to the "Programs" page, then the "Logger" page and set ODB variables:

  • Required -> set to "y"
  • Start command -> set to "mlogger -D"
  • Alarm Class -> set to "Alarm"

To test this, go to the "Programs" page, press "stop logger", then "start logger".

Start a run, wait 10 seconds, stop the run.

From the MIDAS "Status" page, go to the "logging channels -> channel #0" and set ODB variables:

  • Output -> "FILE"

Start a run, go to the "Status" page, observe that the output file is written with lz4 compression and has the "mid.lz4" extension.

Stop the run.

Run test frontend

Run fetest

On the MIDAS "Status" page there will be 5 new equipment entries:

Equipment created by the fetest program
Equipment name Data it sends How often data is sent
test_random Random data banks 1Hz
test_slow Example slow control data for testing the midas history plots 1Hz, even when a run isn't in progress
test_rare Data that look like a sine wave 0.1Hz, even when a run isn't in progress
test_bulk Large events 1Hz
test_rpc None. It shows an example of a "remote procedure call" (RPC) handler. Javascript/C++/Python code can connect to the program, and it will respond with the current time (in this toy example) never

Note the event counters for test_slow and test_rare are already incrementing.

Start a run.

Observe that the event counters for test_bulk and test_random are also incrementing now, and that the output data file size is growing.

Click on the "Event dump" page and then the "Run" button. You'll see the raw data that is being written to disk. (The data from most of the equipment is double-precision floating point numbers; the data from the "bulk" equipment is just a bunch of zeros).

Return the status page and stop the run.

Create a history plot

Go to the MIDAS "History" page.

Say "new", panel name "test". In the history editor select event "test_slow", and click the "save" button (you can ignore the rest of the options for now). If the "test_slow" event isn't listed, make sure that the mlogger program is running, as described above.

History plot "test" should open with a sine-wave plot. Data on the plot should update as long as "fetest" is running (there is no need to start a run).

Clients run on Localhost only

NOTE
  • If creating a MIDAS experiment with a REMOTE frontend, continue by following the instructions #Running with one or more REMOTE frontends.
  • If all clients are running on the Experiment Host (i.e. localhost), continue with the following instructions:

Frontend and Analyzer (localhost)

THIS SECTION IS VERY OBSOLETE

There are several examples of Frontend user code ( frontend.c) in the MIDAS package available under $MIDASSYS/examples/. Choose a suitable example from this directory that you can later modify for your own particular setup (e.g. for a slow control, choose the ../slowcont/ example).

A frontend is a program that usually reads data from the hardware and sends it to a buffer to be logged and analyzed. You can find documentation about the frontend structure under Frontend Application, Frontend Operation and Frontend user code.

The example chosen here is from $MIDASSYS/examples/experiment/. It does not need any hardware and produces simulated events.

On the MIDAS experiment host (localhost), copy the example to the experiment directory and build it.

     [mhostpc] cd $HOME/online
     [mhostpc] cp $MIDASSYS/examples/experiment/* .
     [mhostpc] make

The analyzer will only build if ROOT has previously been installed. At this point the frontend and the analyzer should be ready if no errors were generated during the build. Try the frontend and analyzer by starting them in xterms.

     [mhostpc] frontend
     [mhostpc] analyzer

Equipment ODB tree

Starting the frontend will automatically create the /Equipment ODB tree and one or more equipments. In this case, the equipments "trigger" and "scaler" have been defined by code in frontend.c and created the first time frontend runs.

    [mhostpc] odbedit
    [local:e777:S]/>ls /equipment
    Trigger                         
    Scaler  

View the MIDAS main status web page (by pointing your browser to https://localhost:8443/ or appropriate port if not using the default). Observe on the Status Page that the two Equipments (Trigger, Scaler) have appeared in the Equipment section (Figure 1) and are coloured green.
Click thumbnail to enlarge

Figure 1: web server status page showing example frontend equipments

Rather than using odbedit, you can view the contents of these equipments by clicking on the ODB button, then on /Equipment, then Trigger or Scaler. See ODB Page for more information.

Start a run

Start a run by pressing the Start button on the Status Page. The Equipment display on the Status Page will now show some event statistics (Figure 2) Click thumbnail to enlarge

Figure 2: Equipments shown on Status Page showing event statistics


The frontend will show an event display that will update when a run is started, e.g.

Sample Frontend connected to <local>. Press "!" to exit      18:48:2546:08
==========================================================================
Run status:   Running    Run number 2                                   |/   /
===========================================================================
Equipment     Status     Events     Events/sec Rate[B/s]  ODB->FE FE->OD
---------------------------------------------------------------------------
Trigger       OK         13948      99.0       5413.0     0          140         
Scaler        OK         15         0.3         15.4      0          15    

The Status Page will also show events being generated (click the refresh button if necessary).

Event Dump

While a run is in progress, the midas application mdump will provide you an event dump of the collected data from the running frontend, e.g.

 mhostpc> mdump -l 50
blank
- MIDAS revision: Mon Nov 2 11:50:51 2015 -0800 - 3b66779 -- Enter <!> to Exit ------- Midas Dump ---
------------------------ Event# 1 ------------------------
Evid:0001- Mask:0000- Serial:0- Time:0x567a0c5d- Dsize:40/0x28
#banks:2 - Bank list:-ADC0TDC0-
blank
Bank:ADC0 Length: 8(I*1)/2(I*4)/4(Type) Type:Unsigned Integer*2
  1-> 0x0167 0x03c6 0x0069 0x0073 
 blank
Bank:TDC0 Length: 8(I*1)/2(I*4)/4(Type) Type:Unsigned Integer*2
  1-> 0x0051 0x00ff 0x004a 0x00ec 

For further data processing/analysis, either the midas analyzer or the rootana can used for data display as well.

Create a startup script (localhost)

It is useful to create a script that will automatically start all the required clients for the experiment. This is called a startup script. Here, the startup script will be start_daq.sh (any name can be used). Create $HOME/online/bin/start_daq.sh on the experiment host as follows (supply the mhttpd ports if default is not used):

#!/bin/sh
# start_daq.sh
cd $HOME/online
odbedit -c clean
#   start mhttpd on default port. (Mongoose https version)
mhttpd  -D  # you can optionally restrict access to localhost and other specified hosts - see mhttpd
xterm -e ./frontend &
xterm -e ./analyzer &
mlogger -D
#end file

Before running this script, you will need to shutdown any running clients:

mhostpc> odbedit -c "sh all"

or you can use the mhttpd Programs Page to shut them down.

The script will start them as daemons. By running the startup script start_daq.sh, several MIDAS applications will be started in sequence:

  1. Cleanup previous MIDAS application (if any).
  2. Start the MIDAS web server [mhttpd]
  3. Start the frontend application in its own xterm (for debugging purpose).
  4. Start the analyzer application in its own xterm (for debugging purpose).
  5. Start the MIDAS Data logger [mlogger]
 [mhostpc] sh ./start_daq

You may wish to modify the script to restart missing clients only.

Next customize your experiment.

Running with one or more REMOTE frontends

These instructions assume you have already followed the instructions to setup an experiment running on the MIDAS experiment host (localhost), i.e. you have setup the MIDAS Environment variables, downloaded MIDAS packages, setup the mhttpd password and have mhttpd running successfully.

In the case of remote frontend(s), the remote cpu(s) typically have access to some or all of the hardware. They might be VMIC cpus running in VME crates for example, connected to the MIDAS Host via Ethernet. They read data from the hardware and send it back to the MIDAS experiment host to be logged and analyzed.

Experimental setup (Remote)

These instructions assume that the MIDAS experiment host computer is 64-bit (if not see note below), and there is one remote host, a 32-bit machine named "rlxhost".

The remote host mounts the /home disk of the experiment host, so that it has access to the MIDAS packages, the $HOME/online directory and shares the .cshrc (or .bashrc) script. It is convenient to set up ssh keys (use ssh-keygen) so that the remote host can be accessed without supplying the password from the experiment host.

If your setup is different, you will have to make changes as appropriate.

Build 32-bit MIDAS libraries

NOTE

If the MIDAS experiment host computer is 32-bit (i.e. the same as the remote host), which is a simpler setup, you will have built the 32-bit MIDAS libraries already, and can skip the rest of this section. Note however, that in the .cshrc file in the following section you will not need the lines starting "# select 64-bit or 32-bit MIDAS and ROOT". The PATH should contain $MIDASSYS/bin on both experiment host and remote host. This will point to the 32-bit MIDAS libraries.



Assuming the MIDAS host computer is 64-bit, you will have already built the 64-bit MIDAS libraries. In this case, if the remote frontend is 32-bit, you will need to build the 32-bit MIDAS libraries on the 64-bit machine, i.e.

   [mhostpc] cd /home/packages/midas
   [mhostpc] make linux32                ### build the 32-bit MIDAS libraries on 64-bit machine
   [mhostpc] ls -l linux-m32/bin/odbedit ### check that the 32-bit odbedit has been created 

Do not run odbedit yet on the remote host!

If the "make linux32" command fails, you may need to install 32bit support for your compiler (e.g. "apt install g++-multilib").

Modify .cshrc (.bashrc) for Remote host

Add the following to the .cshrc (modified appropriately if using .bashrc). When running with a REMOTE Host, the application mserver will be started (using a particular port) on the MIDAS experiment host machine only. Note that the environment variable MIDAS_SERVER_HOST is defined on the remote machine(s) but NOT on the experiment host.

If ROOT has been installed properly and the script <ROOT installation directory>/bin/thisroot.sh is called from .cshrc or .bashrc, the ROOTSYS environment variable should be defined and the ROOT bin directory should be in the PATH environment variable. Make sure that this is the case.

# setenv MIDAS_HOST mhostpc   # substitute your experiment host name
# setenv REMOTE_HOST rlxhost  # substitute your remote host name
# setup the MIDAS mserver host
#
switch (`hostname`)
case $MIDAS_HOST*:                
  unsetenv MIDAS_SERVER_HOST     ## MIDAS_SERVER_HOST not defined
  breaksw
default:
setenv MIDAS_SERVER_HOST $MIDAS_HOST.triumf.ca:1175  ## for remote host, define MIDAS_SERVER_HOST, use your domain here
                                                     ## as hostname + domain name + default mserver port
                                                     ## port must match that of mserver used in start_daq.sh 
endsw
#
# select 64-bit or 32-bit MIDAS
#
switch (`uname -i`)
case i386:
  setenv PATH .:$MIDASSYS/linux-m32/bin:$PATH
  breaksw
default:
  setenv PATH .:$MIDASSYS/linux/bin:$PATH
endsw
#
setenv PATH .:$HOME/online/bin:$HOME/packages/roody/bin:$PATH
#

After executing .cshrc (or .bashrc) or logging out, on the MIDAS host (mhostpc) make sure environment variable MIDAS_SERVER_HOST is NOT defined

[mhostpc] echo $MIDAS_SERVER_HOST
[mhostpc] MIDAS_SERVER_HOST: Undefined variable.

and on the remote host, make sure MIDAS_SERVER_HOST IS defined:

 [rlxhost] echo $MIDAS_SERVER_HOST
 [rlxhost] MIDAS_SERVER_HOST

Also on the remote host, make sure that the correct (32-bit) odbedit will be used (do not run it yet!)

[rlxhost] which odbedit
/home/johnfoo/packages/midas/linux-m32/bin/odbedit

Grant Remote host access permission

Give permission for the remote host(s) to access the experiment by following the instructions to allow MIDAS programs on remote machines. If the MIDAS web server mhttpd is running, use ODB Page to edit the ODB keys listed, otherwise use odbedit on the experiment host machine.

Start mserver on Experiment Host

The application mserver is not required when running an experiment on the experiment host (localhost) only, but it IS required for access from a REMOTE frontend. Start mserver on MIDAS host (default port 1175) or use the "-p" option for a different port. The port must match the port defined for environment variable MIDAS_SERVER_HOST in .cshrc (or .bashrc) above.

 [mhostpc] mserver -D

The environment variables on the REMOTE host should be:


   [rlxhost] printenv | grep MIDAS
   MIDASSYS=/home/e777/packages/midas
   MIDAS_EXPT_NAME=e777
   MIDAS_SERVER_HOST=mhostpc.triumf.ca:1175

(a different port may be used - see above)

Check access to ODB from Remote host

Run odbedit on the remote host to check access to ODB.

[rlxhost] odbedit
[mhostpc.triumf.ca:1175:e777:Stopped]/>ls 

If it does not work correctly, check that

  • MIDAS_SERVER_HOST is defined on REMOTE host with same port as mserver
  • mserver is running on MIDAS host with the correct port
  • MIDAS_SERVER_HOST is NOT defined on MIDAS host
  • remote access permission to REMOTE host has been granted as described above.

Build the analyzer on Experiment Host

The analyzer will be built and run on the MIDAS experiment host machine. Select an example from $MIDASSYS/examples/ (see #Frontend and Analyzer (localhost)) and copy it to $HOME/online on the MIDAS host machine. Since we are assuming a 64-bit MIDAS host and a 32-bit remote host, edit the example Makefile so only the analyzer will be built.

Build the frontend (32bit) for Remote host

The frontend runs on a 32-bit remote host in this example. It therefore must either

  • be built 32-bit on the 64-bit MIDAS host with the "-m32" flag
  • or built on the REMOTE host

and in either case the 32-bit MIDAS libraries used for linking.

To avoid confusion, create a different subdirectory for the frontend, e.g.

 [rlxhost] mkdir $HOME/online/fe_32

Copy the Makefile and frontend code from the same example to this directory. Modify the Makefile as needed, and only build the frontend.

Once built, start the frontend in an xterm on the REMOTE host.

[rlxhost] $HOME/online/fe_32/frontend

Check that it connects successfully to the experiment and the Equipments appear on the Status Page as described under #Frontend and Analyzer (localhost). The Equipment Status (Figure 2) should show that the Equipments are now running on the remote cpu, i.e. "SampleFrontend@rlxhost.triumf.ca" rather than "Sample Frontend@localhost".

Run the Analyzer on Experiment host

Start the analyzer in an xterm on the MIDAS experiment host

[johnfoo@mhostpc e777] $HOME/online/analyzer

Continue by starting a run and dumping the data as described under #Frontend and Analyzer (localhost).

Create experiment startup script

It is useful to create a script that will automatically start all the required clients for the experiment. This is called a startup script (any name can be used). Create $HOME/online/bin/start_daq.sh on the experiment host as follows, (supply the mhttpd ports if default is not used) :

#!/bin/sh
# start_daq.sh
cd $HOME/online
#
switch (`hostname`)
  case $MIDAS_HOST*:    ## MIDAS_HOST is assigned in .cshrc (.bashrc)
     echo "Good, we are on $MIDAS_HOST"
     breaksw
  case $REMOTE_HOST*:   ## REMOTE_HOST is assigned in .cshrc (.bashrc)
     echo "startup script should be executed on $MIDAS_HOST"
endsw
#
odbedit -c clean
#   start mhttpd on default port (Mongoose https version)
mhttpd  -D   # you can optionally restrict access to localhost and/or other specified hosts - see mhttpd
#   start mserver on default port (use argument -p to use a different port) 
mserver -D  # start mserver on default port (1175) or a different port using -p option - see mserver
            # the port must match that of MIDAS_SERVER_HOST in .cshrc (.bashrc) - see above
            # access must be specifically allowed - see |above
xterm -e ./analyzer &
## start frontend on remote host
#ssh $REMOTE_HOST $HOME/online/bin/start_frontend -O >& $HOME/online/bin/start_frontend.log  &
mlogger -D
#end file

Note that command to start the frontend on the remote host is commented out. It can be uncommented once the start_frontend script is written and tested.

Shut down any clients that are running already by running odbedit on the MIDAS experiment host and execute the new startup script start_daq.sh (make it executable if desired)


  [mhostpc] odbedit
  [local:e777:S]/>sh all
  [mhostpc] ./start_daq.sh 

and check that odbedit still works on the REMOTE host


   [rlxhost] odbedit 
   [mhostpc.triumf.ca:1175:e777:Stopped]/>ls 

You may wish to modify the script to restart missing clients only.


Create script to start frontend

The startup file start_daq.sh invoked a script start_frontend to start the remote frontend. This line was commented out (see above). Once the frontend is working, create a script to start the frontend (e.g. file $HOME/online/bin/start_frontend).

#!/bin/tcsh
# Script to start frontend running on $REMOTE_HOST
#
# This script runs on the REMOTE HOST
#
echo "starting frontend for experiment $MIDAS_EXPT_NAME "
xterm -geometry 150x50+800+0 -fg white -bg blue -title "Frontend" -e  "$HOME/online/fe_32/frontend" &
echo "Frontend task has been started"
exit

Check that this script works by running it on the REMOTE host. If successful, run the commented command in an xterm on the experiment host

   [mhostpc] ssh $REMOTE_HOST $HOME/online/bin/start_frontend -O >& $HOME/online/bin/start_frontend.log  & 

This command relies on password-less access set up with SSH keys (see #Experimental setup (Remote)). If successful, uncomment line in start_daq.sh, and run the script to check that it can start the remote client(s) from the MIDAS experiment host (first shutting down all running clients) :

   [mhostpc] odbedit -c "sh all"
   [mhostpc] ./start_daq.sh 

Next customize your experiment.


Customizing your experiment

Here are some of the more common options provided for users to customize their experiments according to their own preferences:

Write frontend(s) to read out and control your hardware
Modify the example frontend and Makefile (from $MIDASSYS/examples/) you have already copied and built so it accesses your hardware.
The example Frontend.c is an example of Frontend user code. You can find documentation about the frontend structure under Frontend Application, Frontend Operation and Frontend user code. You may need to set up some Hot-Links in the frontend. The MIDAS package includes device drivers for some of the more common hardware. See MIDAS Driver Library for more information.
Modify analyzer(s) for your data
Modify the example analyzer code as necessary to analyze the data from your frontends.
Slow Controls
Control your slow-controls equipment with slow-control frontend(s) optionally using the MSCB system.
Data logger
Set up the data logger mlogger to log data to a storage device. Many options are available - see Data Logger, mlogger and /Logger ODB tree for instructions.
Allow programs to be restarted from the Programs Page
See mhttpd Programs Page and /Programs ODB tree
Write scripts which run at start and end of run
See /Programs ODB tree#Execute on start run and /Programs ODB tree#Execute on stop run
Create a shutdown script
Create a script similar to the startup script to shutdown the experiment
Lazy logger
Set up the lazylogger for archiving data files for storage.
Create edit-on-start parameters
See Edit-on-start Parameters
Set up the alarm system
See Alarm System and Alarms Page
Set up history system
See History System and History Page
Set up the Electronic logbook (Elog)
See Elog for details
Set up the Sequencer
The Sequencer Page#Sequencer can be used to run a sequence of runs, changing parameters as needed.
Write a webserver Custom page
To better control and monitor your experiment, you may want to write a Custom Page.

Start MIDAS on boot

To start MIDAS mhttpd, mlogger and elogd at boot time, add following cron entries:

[iris@iris00 ~]$ crontab -l
@reboot /home/iris/packages/midas/bin/mhttpd -D
@reboot /home/iris/packages/midas/bin/mlogger -D
@reboot /home/iris/packages/elog/elogd -c /home/iris/packages/elog/elogd.cfgiris -D
[iris@iris00 ~]$