MidasWiki - User contributions [en]

Electronic Logbook (ELOG)

2023-09-15T19:33:31Z

Olchansk: /* Configuration file elogd.cfg */

{{Pagelinks}}
= Links =
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[/Elog ODB tree]]
* [[mhttpd]]
* mhttpd [[Elog Page]]
* [https://midas.psi.ch/elog/ '''Elog''' ]
* [[Melog]] Utility
* [[Elog|Elog Utility]]
</div>

= Introduction =
The '''Electronic LogBook''' (ELOG) is a utility built-in the [[mhttpd|web server]]. It is available and operational once the web-server is running. A dedicated button "ELOG" will be present on the main
Midas status page for creating new entries. This mode is referred as the [[#Built-in Elog|Built-in Elog]].

This tool can replace the experimental logbook for daily entries. The main advantage of the Elog over a paper logbook is the possiblity to access it remotely, and provide a general knowledge of the experiment.

More recently, the independent [https://elog.psi.ch/elog/ '''Elog''' ] package can be linked to a Midas experiment. Its advantage over the built-in version is its flexibility. This package is used worldwide and improvements are constantly being made. A full-features documentation and standalone installation can be found at the Elog web site.

== Built-in Elog ==
The ELOG page (accessed by clicking the Elog button if present on the main status page ) provides access to an electronic logbook. the Elog is not limited strictly to experiments. Worldwide Elog implementations can be found on the internet. This internal implementation doesn't requires any setup to be operational. The Elog is customized through the ODB /Elog tree.

== External Elog ==
The external implementation requires a proper Elog installation which is fully described on the Elog web site. The External Elog implementation also requires a dedicated entry in the [[/Elog ODB tree]] as shown in the code below. It also requires the package Elog to be already installed, and properly configured. Once the ODB entry is present, the internal ELOG is disabled.

The External Elog is customized through the configuration file elogd.cfg (see [[#Installation|Installation]] below).

= Installation of External Elog =
Installation requires requires several steps described below.

== Build Elog package ==

On Linux and Mac, we recommend building elog from git sources:
<pre>
mkdir -p $HOME/packages
cd $HOME/packages
git clone https://bitbucket.org/ritt/elog --recursive
cd elog
git pull ### update to latest version
make -j
ls -l elogd
</pre>

== Installation ==

Elog will run from it's build directory, an "install" step is not needed.

We will use Elog in conjunction with MIDAS mhttpd and Apache httpd. mhttpd will run on TCP port 8080, elogd on port 8082, apache httpd will provide HTTPS encryption and password protection, see https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Install_apache_httpd_proxy_for_midas_and_elog

Make sure "ProxyPass /elog/" is uncommented in apache config file, restart apache after making this change.

Try https://yourhostname/elog/, after entering correct password, error should be: "Service Unavailable"

== Configuration file elogd.cfg ==
* copy the default midas/src/elogd.cfg from the MIDAS distrbution to your operating directory.
* modify the elogd.cfg to reflect your configuration

<pre>
# This is a simple elogd configuration file to work with MIDAS
#
[global]
; port under which elogd should run
port = 8082
; password file, created under 'logbook dir'
password file = elog.pwd
; directory under which elog was installed (themes etc.)
resource dir = /home/exptuser/packages/elog
; directory where the password file will end up
logbook dir = /myexpt/logbook
; anyone can create it's own account
self register = 1
; this is the URL of the elogd https proxy
url = https://myexpt.triumf.ca/elog/
; the "main" tab will bring you back to mhttpd
main tab = Xenon
; this is the URL of mhttpd https proxy
main tab url = https://myexpt.triumf.ca
; email server for sending email notifications, contact your campus network administrator for correct hostname
smtp host = smtp.triumf.ca
; Define one logbook for online use. Severl logbooks can be defined here
[MyOnline]
; directory where the logfiles will be written to
Data dir = /myexpt/logbook
Comment = My MIDAS Experiment Electronic Logbook
; mimic old mhttpd behaviour
Attributes = Run number, Author, Type, System, Subject
Options Type = Routine, Shift Summary, Minor Error, Severe Error, Fix, Question, Info, Modification, Alarm, Test, Other,
Options System = General, DAQ, Detector, Electronics, Target, Beamline
Extendable Options = Type, System
; This substitution will enter the current run number
Preset Run number = $shell(odbedit -e myexpt -h NodeA -d Runinfo -c 'ls -v \"run number\"')
Preset Author = $long_name
Required Attributes = Type, Subject
; Run number and Author cannot be changed
Locked Attributes = Run number, Author
Page Title = ELOG - $subject
Reverse sort = 1
Quick filter = Date, Type, Author
; Don't send any emails
Suppress email to users = 1
</pre>

Konstantin's most recently used config file:

<pre>
[global]
port = 9085
URL = https://daq00.triumf.ca/elog-dl
SMTP host = smtp.triumf.ca
Logfile = elogd.log
Time format = %d %b %Y %H:%M
Default encoding = 1
Allowed encoding = 1
Password file = elogd-dl.passwd
Admin User = Olchansk
Menu commands = List, New, Edit, Delete, Reply, Duplicate, Find, Config, Help
Self register = 3
Message Width = 80
Preset Author = $long_name
Preset on reply Author = $long_name
Locked Attributes = Author
Max content length = 100000000

[DarkLight]
Theme = default
Comment = DarkLight elog
Attributes = Author, Type, Category, Subject
Options Type = Routine, Problem Report, Problem Fixed
Options Category = General, Software, Hardware
Extendable Options = Category, Type
Required Attributes = Author, Type
Page Title = DL - $subject
Reverse sort = 1
Quick filter = Date, Type
Use Email Subject = [DL] $subject
Use email from = noreply@daq00.triumf.ca
</pre>

== Start the Elog daemon ==
* start the elog daemon
* -x is for the shell substitution of the command Preset Run number = $shell(...)
* if elog is not running on same machine as midas (mhttpd, mlogger), change the odbedit command in the config file to have correct -e and -h arguments.
<pre>
$HOME/packages/elog/elogd -c dldaq.cfg -x
</pre>

== Activate External Elog ==

At this point the Elog from the MIDAS web page is accessing the [[[#Built-in Elog|Built-in]]] (internal) Elog. To activate the external Elog you need to edit two ODB variables (which mhttpd automatically created):

* the ODB string "/Elog/URL" needs to be set to the URL of the external ELOG.
* the ODB bool "/Elog/External Elog" needs to be set to 'y'

Note that after changing the ODB variable 'External Elog' from 'n' to 'y' you will need to reload the elog page in order to get the external elog.

Confirm proper operation of the external Elog by creating an entry. You will be prompted for a username and password. Click on New registration. Full control of these features are described in the Elog documentation.
Stop and restart the Elogd in the background.

NodeB:~>/installation_elog_dir/elogd -c elogd.cfg -x -D

= How to convert Built-in Elog files to External files =
In the event you had a previous entry under the internal elog, you can convert the internal to external using the elconv tool.

NodeB:~> cp internal/elog_logbook/*.log /myexpt/logbook/.
NodeB:~> cd /myexpt/logbook
NodeB:~> /installation_elog_dir/elconv

= How to send entries to the Elog from a script =
Utilities exist to send elog entries automatically (e.g. from a script run at begin-of-run). Use [[Melog|melog]] to send an
entry to a Built-in elog, or [[Elog|elog]] to send an entry to an '''external''' elog.

[[Category:ELOG]]

Electronic Logbook (ELOG)

2023-09-15T18:19:42Z

Olchansk: /* Start the Elog daemon */

{{Pagelinks}}
= Links =
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[/Elog ODB tree]]
* [[mhttpd]]
* mhttpd [[Elog Page]]
* [https://midas.psi.ch/elog/ '''Elog''' ]
* [[Melog]] Utility
* [[Elog|Elog Utility]]
</div>

= Introduction =
The '''Electronic LogBook''' (ELOG) is a utility built-in the [[mhttpd|web server]]. It is available and operational once the web-server is running. A dedicated button "ELOG" will be present on the main
Midas status page for creating new entries. This mode is referred as the [[#Built-in Elog|Built-in Elog]].

This tool can replace the experimental logbook for daily entries. The main advantage of the Elog over a paper logbook is the possiblity to access it remotely, and provide a general knowledge of the experiment.

More recently, the independent [https://elog.psi.ch/elog/ '''Elog''' ] package can be linked to a Midas experiment. Its advantage over the built-in version is its flexibility. This package is used worldwide and improvements are constantly being made. A full-features documentation and standalone installation can be found at the Elog web site.

== Built-in Elog ==
The ELOG page (accessed by clicking the Elog button if present on the main status page ) provides access to an electronic logbook. the Elog is not limited strictly to experiments. Worldwide Elog implementations can be found on the internet. This internal implementation doesn't requires any setup to be operational. The Elog is customized through the ODB /Elog tree.

== External Elog ==
The external implementation requires a proper Elog installation which is fully described on the Elog web site. The External Elog implementation also requires a dedicated entry in the [[/Elog ODB tree]] as shown in the code below. It also requires the package Elog to be already installed, and properly configured. Once the ODB entry is present, the internal ELOG is disabled.

The External Elog is customized through the configuration file elogd.cfg (see [[#Installation|Installation]] below).

= Installation of External Elog =
Installation requires requires several steps described below.

== Build Elog package ==

On Linux and Mac, we recommend building elog from git sources:
<pre>
mkdir -p $HOME/packages
cd $HOME/packages
git clone https://bitbucket.org/ritt/elog --recursive
cd elog
git pull ### update to latest version
make -j
ls -l elogd
</pre>

== Installation ==

Elog will run from it's build directory, an "install" step is not needed.

We will use Elog in conjunction with MIDAS mhttpd and Apache httpd. mhttpd will run on TCP port 8080, elogd on port 8082, apache httpd will provide HTTPS encryption and password protection, see https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Install_apache_httpd_proxy_for_midas_and_elog

Make sure "ProxyPass /elog/" is uncommented in apache config file, restart apache after making this change.

Try https://yourhostname/elog/, after entering correct password, error should be: "Service Unavailable"

== Configuration file elogd.cfg ==
* copy the default midas/src/elogd.cfg from the MIDAS distrbution to your operating directory.
* modify the elogd.cfg to reflect your configuration

# This is a simple elogd configuration file to work with MIDAS
#
[global]
; port under which elogd should run
port = 8082
; password file, created under 'logbook dir'
password file = elog.pwd
; directory under which elog was installed (themes etc.)
resource dir = /home/exptuser/packages/elog
; directory where the password file will end up
logbook dir = /myexpt/logbook
; anyone can create it's own account
self register = 1
; this is the URL of the elogd https proxy
url = https://myexpt.triumf.ca/elog/
; the "main" tab will bring you back to mhttpd
main tab = Xenon
; this is the URL of mhttpd https proxy
main tab url = https://myexpt.triumf.ca
; email server for sending email notifications, contact your campus network administrator for correct hostname
smtp host = smtp.triumf.ca
; Define one logbook for online use. Severl logbooks can be defined here
[MyOnline]
; directory where the logfiles will be written to
Data dir = /myexpt/logbook
Comment = My MIDAS Experiment Electronic Logbook
; mimic old mhttpd behaviour
Attributes = Run number, Author, Type, System, Subject
Options Type = Routine, Shift Summary, Minor Error, Severe Error, Fix, Question, Info, Modification, Alarm, Test, Other,
Options System = General, DAQ, Detector, Electronics, Target, Beamline
Extendable Options = Type, System
; This substitution will enter the current run number
Preset Run number = $shell(odbedit -e myexpt -h NodeA -d Runinfo -c 'ls -v \"run number\"')
Preset Author = $long_name
Required Attributes = Type, Subject
; Run number and Author cannot be changed
Locked Attributes = Run number, Author
Page Title = ELOG - $subject
Reverse sort = 1
Quick filter = Date, Type, Author
; Don't send any emails
Suppress email to users = 1

== Start the Elog daemon ==
* start the elog daemon
* -x is for the shell substitution of the command Preset Run number = $shell(...)
* if elog is not running on same machine as midas (mhttpd, mlogger), change the odbedit command in the config file to have correct -e and -h arguments.
<pre>
$HOME/packages/elog/elogd -c dldaq.cfg -x
</pre>

== Activate External Elog ==

At this point the Elog from the MIDAS web page is accessing the [[[#Built-in Elog|Built-in]]] (internal) Elog. To activate the external Elog you need to edit two ODB variables (which mhttpd automatically created):

* the ODB string "/Elog/URL" needs to be set to the URL of the external ELOG.
* the ODB bool "/Elog/External Elog" needs to be set to 'y'

Note that after changing the ODB variable 'External Elog' from 'n' to 'y' you will need to reload the elog page in order to get the external elog.

Confirm proper operation of the external Elog by creating an entry. You will be prompted for a username and password. Click on New registration. Full control of these features are described in the Elog documentation.
Stop and restart the Elogd in the background.

NodeB:~>/installation_elog_dir/elogd -c elogd.cfg -x -D

= How to convert Built-in Elog files to External files =
In the event you had a previous entry under the internal elog, you can convert the internal to external using the elconv tool.

NodeB:~> cp internal/elog_logbook/*.log /myexpt/logbook/.
NodeB:~> cd /myexpt/logbook
NodeB:~> /installation_elog_dir/elconv

= How to send entries to the Elog from a script =
Utilities exist to send elog entries automatically (e.g. from a script run at begin-of-run). Use [[Melog|melog]] to send an
entry to a Built-in elog, or [[Elog|elog]] to send an entry to an '''external''' elog.

[[Category:ELOG]]

Electronic Logbook (ELOG)

2023-09-15T18:11:37Z

Olchansk: /* Configuration file elogd.cfg */

{{Pagelinks}}
= Links =
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[/Elog ODB tree]]
* [[mhttpd]]
* mhttpd [[Elog Page]]
* [https://midas.psi.ch/elog/ '''Elog''' ]
* [[Melog]] Utility
* [[Elog|Elog Utility]]
</div>

= Introduction =
The '''Electronic LogBook''' (ELOG) is a utility built-in the [[mhttpd|web server]]. It is available and operational once the web-server is running. A dedicated button "ELOG" will be present on the main
Midas status page for creating new entries. This mode is referred as the [[#Built-in Elog|Built-in Elog]].

This tool can replace the experimental logbook for daily entries. The main advantage of the Elog over a paper logbook is the possiblity to access it remotely, and provide a general knowledge of the experiment.

More recently, the independent [https://elog.psi.ch/elog/ '''Elog''' ] package can be linked to a Midas experiment. Its advantage over the built-in version is its flexibility. This package is used worldwide and improvements are constantly being made. A full-features documentation and standalone installation can be found at the Elog web site.

== Built-in Elog ==
The ELOG page (accessed by clicking the Elog button if present on the main status page ) provides access to an electronic logbook. the Elog is not limited strictly to experiments. Worldwide Elog implementations can be found on the internet. This internal implementation doesn't requires any setup to be operational. The Elog is customized through the ODB /Elog tree.

== External Elog ==
The external implementation requires a proper Elog installation which is fully described on the Elog web site. The External Elog implementation also requires a dedicated entry in the [[/Elog ODB tree]] as shown in the code below. It also requires the package Elog to be already installed, and properly configured. Once the ODB entry is present, the internal ELOG is disabled.

The External Elog is customized through the configuration file elogd.cfg (see [[#Installation|Installation]] below).

= Installation of External Elog =
Installation requires requires several steps described below.

== Build Elog package ==

On Linux and Mac, we recommend building elog from git sources:
<pre>
mkdir -p $HOME/packages
cd $HOME/packages
git clone https://bitbucket.org/ritt/elog --recursive
cd elog
git pull ### update to latest version
make -j
ls -l elogd
</pre>

== Installation ==

Elog will run from it's build directory, an "install" step is not needed.

We will use Elog in conjunction with MIDAS mhttpd and Apache httpd. mhttpd will run on TCP port 8080, elogd on port 8082, apache httpd will provide HTTPS encryption and password protection, see https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Install_apache_httpd_proxy_for_midas_and_elog

Make sure "ProxyPass /elog/" is uncommented in apache config file, restart apache after making this change.

Try https://yourhostname/elog/, after entering correct password, error should be: "Service Unavailable"

== Configuration file elogd.cfg ==
* copy the default midas/src/elogd.cfg from the MIDAS distrbution to your operating directory.
* modify the elogd.cfg to reflect your configuration

# This is a simple elogd configuration file to work with MIDAS
#
[global]
; port under which elogd should run
port = 8082
; password file, created under 'logbook dir'
password file = elog.pwd
; directory under which elog was installed (themes etc.)
resource dir = /home/exptuser/packages/elog
; directory where the password file will end up
logbook dir = /myexpt/logbook
; anyone can create it's own account
self register = 1
; this is the URL of the elogd https proxy
url = https://myexpt.triumf.ca/elog/
; the "main" tab will bring you back to mhttpd
main tab = Xenon
; this is the URL of mhttpd https proxy
main tab url = https://myexpt.triumf.ca
; email server for sending email notifications, contact your campus network administrator for correct hostname
smtp host = smtp.triumf.ca
; Define one logbook for online use. Severl logbooks can be defined here
[MyOnline]
; directory where the logfiles will be written to
Data dir = /myexpt/logbook
Comment = My MIDAS Experiment Electronic Logbook
; mimic old mhttpd behaviour
Attributes = Run number, Author, Type, System, Subject
Options Type = Routine, Shift Summary, Minor Error, Severe Error, Fix, Question, Info, Modification, Alarm, Test, Other,
Options System = General, DAQ, Detector, Electronics, Target, Beamline
Extendable Options = Type, System
; This substitution will enter the current run number
Preset Run number = $shell(odbedit -e myexpt -h NodeA -d Runinfo -c 'ls -v \"run number\"')
Preset Author = $long_name
Required Attributes = Type, Subject
; Run number and Author cannot be changed
Locked Attributes = Run number, Author
Page Title = ELOG - $subject
Reverse sort = 1
Quick filter = Date, Type, Author
; Don't send any emails
Suppress email to users = 1

== Start the Elog daemon ==
* start the elog daemon. -x is for the shell substitution of the command Preset Run number = $shell(...)
* The argument invokes the odbedit remotely if needed to retrieve the current run number. You will have to ensure the proper path to the odbedit and the proper -e, -h arguments for the experiment and host.

You may want to verify this command from the console.

NodeB:~>/installation_elog_dir/elogd -c elogd.cfg -x<cr> starts the mhttpd at its correct port and possibly in the daemon form.
NodeA:~>mhttpd -p 8080 -D

== Activate External Elog ==

At this point the Elog from the MIDAS web page is accessing the [[[#Built-in Elog|Built-in]]] (internal) Elog. To activate the external Elog you need to edit two ODB variables (which mhttpd automatically created):

* the ODB string "/Elog/URL" needs to be set to the URL of the external ELOG.
* the ODB bool "/Elog/External Elog" needs to be set to 'y'

Note that after changing the ODB variable 'External Elog' from 'n' to 'y' you will need to reload the elog page in order to get the external elog.

Confirm proper operation of the external Elog by creating an entry. You will be prompted for a username and password. Click on New registration. Full control of these features are described in the Elog documentation.
Stop and restart the Elogd in the background.

NodeB:~>/installation_elog_dir/elogd -c elogd.cfg -x -D

= How to convert Built-in Elog files to External files =
In the event you had a previous entry under the internal elog, you can convert the internal to external using the elconv tool.

NodeB:~> cp internal/elog_logbook/*.log /myexpt/logbook/.
NodeB:~> cd /myexpt/logbook
NodeB:~> /installation_elog_dir/elconv

= How to send entries to the Elog from a script =
Utilities exist to send elog entries automatically (e.g. from a script run at begin-of-run). Use [[Melog|melog]] to send an
entry to a Built-in elog, or [[Elog|elog]] to send an entry to an '''external''' elog.

[[Category:ELOG]]

Electronic Logbook (ELOG)

2023-09-13T21:18:41Z

Olchansk: /* Installation considerations */

{{Pagelinks}}
= Links =
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[/Elog ODB tree]]
* [[mhttpd]]
* mhttpd [[Elog Page]]
* [https://midas.psi.ch/elog/ '''Elog''' ]
* [[Melog]] Utility
* [[Elog|Elog Utility]]
</div>

= Introduction =
The '''Electronic LogBook''' (ELOG) is a utility built-in the [[mhttpd|web server]]. It is available and operational once the web-server is running. A dedicated button "ELOG" will be present on the main
Midas status page for creating new entries. This mode is referred as the [[#Built-in Elog|Built-in Elog]].

This tool can replace the experimental logbook for daily entries. The main advantage of the Elog over a paper logbook is the possiblity to access it remotely, and provide a general knowledge of the experiment.

More recently, the independent [https://elog.psi.ch/elog/ '''Elog''' ] package can be linked to a Midas experiment. Its advantage over the built-in version is its flexibility. This package is used worldwide and improvements are constantly being made. A full-features documentation and standalone installation can be found at the Elog web site.

== Built-in Elog ==
The ELOG page (accessed by clicking the Elog button if present on the main status page ) provides access to an electronic logbook. the Elog is not limited strictly to experiments. Worldwide Elog implementations can be found on the internet. This internal implementation doesn't requires any setup to be operational. The Elog is customized through the ODB /Elog tree.

== External Elog ==
The external implementation requires a proper Elog installation which is fully described on the Elog web site. The External Elog implementation also requires a dedicated entry in the [[/Elog ODB tree]] as shown in the code below. It also requires the package Elog to be already installed, and properly configured. Once the ODB entry is present, the internal ELOG is disabled.

The External Elog is customized through the configuration file elogd.cfg (see [[#Installation|Installation]] below).

= Installation of External Elog =
Installation requires requires several steps described below.

== Build Elog package ==

On Linux and Mac, we recommend building elog from git sources:
<pre>
mkdir -p $HOME/packages
cd $HOME/packages
git clone https://bitbucket.org/ritt/elog --recursive
cd elog
git pull ### update to latest version
make -j
ls -l elogd
</pre>

== Installation ==

Elog will run from it's build directory, an "install" step is not needed.

We will use Elog in conjunction with MIDAS mhttpd and Apache httpd. mhttpd will run on TCP port 8080, elogd on port 8082, apache httpd will provide HTTPS encryption and password protection, see https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Install_apache_httpd_proxy_for_midas_and_elog

Make sure "ProxyPass /elog/" is uncommented in apache config file, restart apache after making this change.

Try https://yourhostname/elog/, after entering correct password, error should be: "Service Unavailable"

== Configuration file elogd.cfg ==
* copy the default midas/src/elogd.cfg from the MIDAS distrbution to your operating directory.
* modify the elogd.cfg to reflect your configuration

# This is a simple elogd configuration file to work with MIDAS
#
[global]
; port under which elogd should run
port = 8081
; password file, created under 'logbook dir'
password file = elog.pwd
; directory under which elog was installed (themes etc.)
resource dir = /elog_installation_dir
; directory where the password file will end up
logbook dir = /myexpt/logbook
; anyone can create it's own account
self register = 1
; URL under which elogd is accessible
url = http://NodeB.triumf.ca:8081
; the "main" tab will bring you back to mhttpd
main tab = Xenon
; this is the URL of mhttpd which must run on a different port
main tab url = http://NodeA:8080
; only needed for email notifications
smtp host = your.smtp.host
; Define one logbook for online use. Severl logbooks can be defined here
[MyOnline]
; directory where the logfiles will be written to
Data dir = /myexpt/logbook
Comment = My MIDAS Experiment Electronic Logbook
; mimic old mhttpd behaviour
Attributes = Run number, Author, Type, System, Subject
Options Type = Routine, Shift Summary, Minor Error, Severe Error, Fix, Question, Info, Modification, Alarm, Test, Other,
Options System = General, DAQ, Detector, Electronics, Target, Beamline
Extendable Options = Type, System
; This substitution will enter the current run number
Preset Run number = $shell(odbedit -e myexpt -h NodeA -d Runinfo -c 'ls -v \"run number\"')
Preset Author = $long_name
Required Attributes = Type, Subject
; Run number and Author cannot be changed
Locked Attributes = Run number, Author
Page Title = ELOG - $subject
Reverse sort = 1
Quick filter = Date, Type, Author
; Don't send any emails
Suppress email to users = 1

== Start the Elog daemon ==
* start the elog daemon. -x is for the shell substitution of the command Preset Run number = $shell(...)
* The argument invokes the odbedit remotely if needed to retrieve the current run number. You will have to ensure the proper path to the odbedit and the proper -e, -h arguments for the experiment and host.

You may want to verify this command from the console.

NodeB:~>/installation_elog_dir/elogd -c elogd.cfg -x<cr> starts the mhttpd at its correct port and possibly in the daemon form.
NodeA:~>mhttpd -p 8080 -D

== Activate External Elog ==

At this point the Elog from the MIDAS web page is accessing the [[[#Built-in Elog|Built-in]]] (internal) Elog. To activate the external Elog you need to edit two ODB variables (which mhttpd automatically created):

* the ODB string "/Elog/URL" needs to be set to the URL of the external ELOG.
* the ODB bool "/Elog/External Elog" needs to be set to 'y'

Note that after changing the ODB variable 'External Elog' from 'n' to 'y' you will need to reload the elog page in order to get the external elog.

Confirm proper operation of the external Elog by creating an entry. You will be prompted for a username and password. Click on New registration. Full control of these features are described in the Elog documentation.
Stop and restart the Elogd in the background.

NodeB:~>/installation_elog_dir/elogd -c elogd.cfg -x -D

= How to convert Built-in Elog files to External files =
In the event you had a previous entry under the internal elog, you can convert the internal to external using the elconv tool.

NodeB:~> cp internal/elog_logbook/*.log /myexpt/logbook/.
NodeB:~> cd /myexpt/logbook
NodeB:~> /installation_elog_dir/elconv

= How to send entries to the Elog from a script =
Utilities exist to send elog entries automatically (e.g. from a script run at begin-of-run). Use [[Melog|melog]] to send an
entry to a Built-in elog, or [[Elog|elog]] to send an entry to an '''external''' elog.

[[Category:ELOG]]

Electronic Logbook (ELOG)

2023-09-13T21:12:47Z

Olchansk: /* Download Elog package */

{{Pagelinks}}
= Links =
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[/Elog ODB tree]]
* [[mhttpd]]
* mhttpd [[Elog Page]]
* [https://midas.psi.ch/elog/ '''Elog''' ]
* [[Melog]] Utility
* [[Elog|Elog Utility]]
</div>

= Introduction =
The '''Electronic LogBook''' (ELOG) is a utility built-in the [[mhttpd|web server]]. It is available and operational once the web-server is running. A dedicated button "ELOG" will be present on the main
Midas status page for creating new entries. This mode is referred as the [[#Built-in Elog|Built-in Elog]].

This tool can replace the experimental logbook for daily entries. The main advantage of the Elog over a paper logbook is the possiblity to access it remotely, and provide a general knowledge of the experiment.

More recently, the independent [https://elog.psi.ch/elog/ '''Elog''' ] package can be linked to a Midas experiment. Its advantage over the built-in version is its flexibility. This package is used worldwide and improvements are constantly being made. A full-features documentation and standalone installation can be found at the Elog web site.

== Built-in Elog ==
The ELOG page (accessed by clicking the Elog button if present on the main status page ) provides access to an electronic logbook. the Elog is not limited strictly to experiments. Worldwide Elog implementations can be found on the internet. This internal implementation doesn't requires any setup to be operational. The Elog is customized through the ODB /Elog tree.

== External Elog ==
The external implementation requires a proper Elog installation which is fully described on the Elog web site. The External Elog implementation also requires a dedicated entry in the [[/Elog ODB tree]] as shown in the code below. It also requires the package Elog to be already installed, and properly configured. Once the ODB entry is present, the internal ELOG is disabled.

The External Elog is customized through the configuration file elogd.cfg (see [[#Installation|Installation]] below).

= Installation of External Elog =
Installation requires requires several steps described below.

== Build Elog package ==

On Linux and Mac, we recommend building elog from git sources:
<pre>
mkdir -p $HOME/packages
cd $HOME/packages
git clone https://bitbucket.org/ritt/elog --recursive
cd elog
git pull ### update to latest version
make -j
ls -l elogd
</pre>

== Installation considerations ==
You need to take note of several considerations for its installation. Several locations are required for the different files that elog deals with.
* elog resource directory ( e.g. /elog_installation_dir where elog is installed)
* logbook directory (ex: /myexpt/logbook where the pwd and elog entries are stored).
* The pwd file uses encryption for the user password.

* As this Elog installation is tailored towards an experiment, a restriction applies i.e. ensure that the mhttpd and elog applications shares at least the same file system.
** This means that either both applications runs on the same machine or a nsf mount provides file sharing.
** You need to know the node and ports for both applications. Like mhttpd, elogd also requires a port number for communication through the web (e.g. NodeA:mhttpd -p 8080, NodeB:elogd -p 8081).

== Configuration file elogd.cfg ==
* copy the default midas/src/elogd.cfg from the MIDAS distrbution to your operating directory.
* modify the elogd.cfg to reflect your configuration

# This is a simple elogd configuration file to work with MIDAS
#
[global]
; port under which elogd should run
port = 8081
; password file, created under 'logbook dir'
password file = elog.pwd
; directory under which elog was installed (themes etc.)
resource dir = /elog_installation_dir
; directory where the password file will end up
logbook dir = /myexpt/logbook
; anyone can create it's own account
self register = 1
; URL under which elogd is accessible
url = http://NodeB.triumf.ca:8081
; the "main" tab will bring you back to mhttpd
main tab = Xenon
; this is the URL of mhttpd which must run on a different port
main tab url = http://NodeA:8080
; only needed for email notifications
smtp host = your.smtp.host
; Define one logbook for online use. Severl logbooks can be defined here
[MyOnline]
; directory where the logfiles will be written to
Data dir = /myexpt/logbook
Comment = My MIDAS Experiment Electronic Logbook
; mimic old mhttpd behaviour
Attributes = Run number, Author, Type, System, Subject
Options Type = Routine, Shift Summary, Minor Error, Severe Error, Fix, Question, Info, Modification, Alarm, Test, Other,
Options System = General, DAQ, Detector, Electronics, Target, Beamline
Extendable Options = Type, System
; This substitution will enter the current run number
Preset Run number = $shell(odbedit -e myexpt -h NodeA -d Runinfo -c 'ls -v \"run number\"')
Preset Author = $long_name
Required Attributes = Type, Subject
; Run number and Author cannot be changed
Locked Attributes = Run number, Author
Page Title = ELOG - $subject
Reverse sort = 1
Quick filter = Date, Type, Author
; Don't send any emails
Suppress email to users = 1

== Start the Elog daemon ==
* start the elog daemon. -x is for the shell substitution of the command Preset Run number = $shell(...)
* The argument invokes the odbedit remotely if needed to retrieve the current run number. You will have to ensure the proper path to the odbedit and the proper -e, -h arguments for the experiment and host.

You may want to verify this command from the console.

NodeB:~>/installation_elog_dir/elogd -c elogd.cfg -x<cr> starts the mhttpd at its correct port and possibly in the daemon form.
NodeA:~>mhttpd -p 8080 -D

== Activate External Elog ==

At this point the Elog from the MIDAS web page is accessing the [[[#Built-in Elog|Built-in]]] (internal) Elog. To activate the external Elog you need to edit two ODB variables (which mhttpd automatically created):

* the ODB string "/Elog/URL" needs to be set to the URL of the external ELOG.
* the ODB bool "/Elog/External Elog" needs to be set to 'y'

Note that after changing the ODB variable 'External Elog' from 'n' to 'y' you will need to reload the elog page in order to get the external elog.

Confirm proper operation of the external Elog by creating an entry. You will be prompted for a username and password. Click on New registration. Full control of these features are described in the Elog documentation.
Stop and restart the Elogd in the background.

NodeB:~>/installation_elog_dir/elogd -c elogd.cfg -x -D

= How to convert Built-in Elog files to External files =
In the event you had a previous entry under the internal elog, you can convert the internal to external using the elconv tool.

NodeB:~> cp internal/elog_logbook/*.log /myexpt/logbook/.
NodeB:~> cd /myexpt/logbook
NodeB:~> /installation_elog_dir/elconv

= How to send entries to the Elog from a script =
Utilities exist to send elog entries automatically (e.g. from a script run at begin-of-run). Use [[Melog|melog]] to send an
entry to a Built-in elog, or [[Elog|elog]] to send an entry to an '''external''' elog.

[[Category:ELOG]]

Electronic Logbook (ELOG)

2023-09-13T21:09:26Z

Olchansk: /* Build Elog package */

{{Pagelinks}}
= Links =
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[/Elog ODB tree]]
* [[mhttpd]]
* mhttpd [[Elog Page]]
* [https://midas.psi.ch/elog/ '''Elog''' ]
* [[Melog]] Utility
* [[Elog|Elog Utility]]
</div>

= Introduction =
The '''Electronic LogBook''' (ELOG) is a utility built-in the [[mhttpd|web server]]. It is available and operational once the web-server is running. A dedicated button "ELOG" will be present on the main
Midas status page for creating new entries. This mode is referred as the [[#Built-in Elog|Built-in Elog]].

This tool can replace the experimental logbook for daily entries. The main advantage of the Elog over a paper logbook is the possiblity to access it remotely, and provide a general knowledge of the experiment.

More recently, the independent [https://elog.psi.ch/elog/ '''Elog''' ] package can be linked to a Midas experiment. Its advantage over the built-in version is its flexibility. This package is used worldwide and improvements are constantly being made. A full-features documentation and standalone installation can be found at the Elog web site.

== Built-in Elog ==
The ELOG page (accessed by clicking the Elog button if present on the main status page ) provides access to an electronic logbook. the Elog is not limited strictly to experiments. Worldwide Elog implementations can be found on the internet. This internal implementation doesn't requires any setup to be operational. The Elog is customized through the ODB /Elog tree.

== External Elog ==
The external implementation requires a proper Elog installation which is fully described on the Elog web site. The External Elog implementation also requires a dedicated entry in the [[/Elog ODB tree]] as shown in the code below. It also requires the package Elog to be already installed, and properly configured. Once the ODB entry is present, the internal ELOG is disabled.

The External Elog is customized through the configuration file elogd.cfg (see [[#Installation|Installation]] below).

= Installation of External Elog =
Installation requires requires several steps described below.

== Build Elog package ==

On Linux and Mac, we recommend building elog from git sources:
<pre>
mkdir -p $HOME/packages
cd $HOME/packages
git clone https://bitbucket.org/ritt/elog --recursive
cd elog
git pull ### update to latest version
make -j
ls -l elogd
</pre>

== Download Elog package ==

* Download the Elog package from the web site mentioned above.
** Windows, Linux, Mac version can be found there. Simple installation procedures are also described. Its installation can be done at the system level or at the user level. The Elog can service multiple Electronic logbooks in parallel and therefore an extra entry in its configuration file can provide specific experimental elog in a similar fashion as the internal one.

== Installation considerations ==
You need to take note of several considerations for its installation. Several locations are required for the different files that elog deals with.
* elog resource directory ( e.g. /elog_installation_dir where elog is installed)
* logbook directory (ex: /myexpt/logbook where the pwd and elog entries are stored).
* The pwd file uses encryption for the user password.

* As this Elog installation is tailored towards an experiment, a restriction applies i.e. ensure that the mhttpd and elog applications shares at least the same file system.
** This means that either both applications runs on the same machine or a nsf mount provides file sharing.
** You need to know the node and ports for both applications. Like mhttpd, elogd also requires a port number for communication through the web (e.g. NodeA:mhttpd -p 8080, NodeB:elogd -p 8081).

== Configuration file elogd.cfg ==
* copy the default midas/src/elogd.cfg from the MIDAS distrbution to your operating directory.
* modify the elogd.cfg to reflect your configuration

# This is a simple elogd configuration file to work with MIDAS
#
[global]
; port under which elogd should run
port = 8081
; password file, created under 'logbook dir'
password file = elog.pwd
; directory under which elog was installed (themes etc.)
resource dir = /elog_installation_dir
; directory where the password file will end up
logbook dir = /myexpt/logbook
; anyone can create it's own account
self register = 1
; URL under which elogd is accessible
url = http://NodeB.triumf.ca:8081
; the "main" tab will bring you back to mhttpd
main tab = Xenon
; this is the URL of mhttpd which must run on a different port
main tab url = http://NodeA:8080
; only needed for email notifications
smtp host = your.smtp.host
; Define one logbook for online use. Severl logbooks can be defined here
[MyOnline]
; directory where the logfiles will be written to
Data dir = /myexpt/logbook
Comment = My MIDAS Experiment Electronic Logbook
; mimic old mhttpd behaviour
Attributes = Run number, Author, Type, System, Subject
Options Type = Routine, Shift Summary, Minor Error, Severe Error, Fix, Question, Info, Modification, Alarm, Test, Other,
Options System = General, DAQ, Detector, Electronics, Target, Beamline
Extendable Options = Type, System
; This substitution will enter the current run number
Preset Run number = $shell(odbedit -e myexpt -h NodeA -d Runinfo -c 'ls -v \"run number\"')
Preset Author = $long_name
Required Attributes = Type, Subject
; Run number and Author cannot be changed
Locked Attributes = Run number, Author
Page Title = ELOG - $subject
Reverse sort = 1
Quick filter = Date, Type, Author
; Don't send any emails
Suppress email to users = 1

== Start the Elog daemon ==
* start the elog daemon. -x is for the shell substitution of the command Preset Run number = $shell(...)
* The argument invokes the odbedit remotely if needed to retrieve the current run number. You will have to ensure the proper path to the odbedit and the proper -e, -h arguments for the experiment and host.

You may want to verify this command from the console.

NodeB:~>/installation_elog_dir/elogd -c elogd.cfg -x<cr> starts the mhttpd at its correct port and possibly in the daemon form.
NodeA:~>mhttpd -p 8080 -D

== Activate External Elog ==

At this point the Elog from the MIDAS web page is accessing the [[[#Built-in Elog|Built-in]]] (internal) Elog. To activate the external Elog you need to edit two ODB variables (which mhttpd automatically created):

* the ODB string "/Elog/URL" needs to be set to the URL of the external ELOG.
* the ODB bool "/Elog/External Elog" needs to be set to 'y'

Note that after changing the ODB variable 'External Elog' from 'n' to 'y' you will need to reload the elog page in order to get the external elog.

Confirm proper operation of the external Elog by creating an entry. You will be prompted for a username and password. Click on New registration. Full control of these features are described in the Elog documentation.
Stop and restart the Elogd in the background.

NodeB:~>/installation_elog_dir/elogd -c elogd.cfg -x -D

= How to convert Built-in Elog files to External files =
In the event you had a previous entry under the internal elog, you can convert the internal to external using the elconv tool.

NodeB:~> cp internal/elog_logbook/*.log /myexpt/logbook/.
NodeB:~> cd /myexpt/logbook
NodeB:~> /installation_elog_dir/elconv

= How to send entries to the Elog from a script =
Utilities exist to send elog entries automatically (e.g. from a script run at begin-of-run). Use [[Melog|melog]] to send an
entry to a Built-in elog, or [[Elog|elog]] to send an entry to an '''external''' elog.

[[Category:ELOG]]

Electronic Logbook (ELOG)

2023-09-13T21:08:52Z

Olchansk: /* Download Elog package */

{{Pagelinks}}
= Links =
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[/Elog ODB tree]]
* [[mhttpd]]
* mhttpd [[Elog Page]]
* [https://midas.psi.ch/elog/ '''Elog''' ]
* [[Melog]] Utility
* [[Elog|Elog Utility]]
</div>

= Introduction =
The '''Electronic LogBook''' (ELOG) is a utility built-in the [[mhttpd|web server]]. It is available and operational once the web-server is running. A dedicated button "ELOG" will be present on the main
Midas status page for creating new entries. This mode is referred as the [[#Built-in Elog|Built-in Elog]].

This tool can replace the experimental logbook for daily entries. The main advantage of the Elog over a paper logbook is the possiblity to access it remotely, and provide a general knowledge of the experiment.

More recently, the independent [https://elog.psi.ch/elog/ '''Elog''' ] package can be linked to a Midas experiment. Its advantage over the built-in version is its flexibility. This package is used worldwide and improvements are constantly being made. A full-features documentation and standalone installation can be found at the Elog web site.

== Built-in Elog ==
The ELOG page (accessed by clicking the Elog button if present on the main status page ) provides access to an electronic logbook. the Elog is not limited strictly to experiments. Worldwide Elog implementations can be found on the internet. This internal implementation doesn't requires any setup to be operational. The Elog is customized through the ODB /Elog tree.

== External Elog ==
The external implementation requires a proper Elog installation which is fully described on the Elog web site. The External Elog implementation also requires a dedicated entry in the [[/Elog ODB tree]] as shown in the code below. It also requires the package Elog to be already installed, and properly configured. Once the ODB entry is present, the internal ELOG is disabled.

The External Elog is customized through the configuration file elogd.cfg (see [[#Installation|Installation]] below).

= Installation of External Elog =
Installation requires requires several steps described below.

== Build Elog package ==

On Linux and Mac, we recommend building elog from git sources:
<pre>
mkdir -p $HOME/packages
cd $HOME/packages
git clone https://bitbucket.org/ritt/elog --recursive
make -j
ls -l elogd
</pre>

== Download Elog package ==

* Download the Elog package from the web site mentioned above.
** Windows, Linux, Mac version can be found there. Simple installation procedures are also described. Its installation can be done at the system level or at the user level. The Elog can service multiple Electronic logbooks in parallel and therefore an extra entry in its configuration file can provide specific experimental elog in a similar fashion as the internal one.

== Installation considerations ==
You need to take note of several considerations for its installation. Several locations are required for the different files that elog deals with.
* elog resource directory ( e.g. /elog_installation_dir where elog is installed)
* logbook directory (ex: /myexpt/logbook where the pwd and elog entries are stored).
* The pwd file uses encryption for the user password.

* As this Elog installation is tailored towards an experiment, a restriction applies i.e. ensure that the mhttpd and elog applications shares at least the same file system.
** This means that either both applications runs on the same machine or a nsf mount provides file sharing.
** You need to know the node and ports for both applications. Like mhttpd, elogd also requires a port number for communication through the web (e.g. NodeA:mhttpd -p 8080, NodeB:elogd -p 8081).

== Configuration file elogd.cfg ==
* copy the default midas/src/elogd.cfg from the MIDAS distrbution to your operating directory.
* modify the elogd.cfg to reflect your configuration

# This is a simple elogd configuration file to work with MIDAS
#
[global]
; port under which elogd should run
port = 8081
; password file, created under 'logbook dir'
password file = elog.pwd
; directory under which elog was installed (themes etc.)
resource dir = /elog_installation_dir
; directory where the password file will end up
logbook dir = /myexpt/logbook
; anyone can create it's own account
self register = 1
; URL under which elogd is accessible
url = http://NodeB.triumf.ca:8081
; the "main" tab will bring you back to mhttpd
main tab = Xenon
; this is the URL of mhttpd which must run on a different port
main tab url = http://NodeA:8080
; only needed for email notifications
smtp host = your.smtp.host
; Define one logbook for online use. Severl logbooks can be defined here
[MyOnline]
; directory where the logfiles will be written to
Data dir = /myexpt/logbook
Comment = My MIDAS Experiment Electronic Logbook
; mimic old mhttpd behaviour
Attributes = Run number, Author, Type, System, Subject
Options Type = Routine, Shift Summary, Minor Error, Severe Error, Fix, Question, Info, Modification, Alarm, Test, Other,
Options System = General, DAQ, Detector, Electronics, Target, Beamline
Extendable Options = Type, System
; This substitution will enter the current run number
Preset Run number = $shell(odbedit -e myexpt -h NodeA -d Runinfo -c 'ls -v \"run number\"')
Preset Author = $long_name
Required Attributes = Type, Subject
; Run number and Author cannot be changed
Locked Attributes = Run number, Author
Page Title = ELOG - $subject
Reverse sort = 1
Quick filter = Date, Type, Author
; Don't send any emails
Suppress email to users = 1

== Start the Elog daemon ==
* start the elog daemon. -x is for the shell substitution of the command Preset Run number = $shell(...)
* The argument invokes the odbedit remotely if needed to retrieve the current run number. You will have to ensure the proper path to the odbedit and the proper -e, -h arguments for the experiment and host.

You may want to verify this command from the console.

NodeB:~>/installation_elog_dir/elogd -c elogd.cfg -x<cr> starts the mhttpd at its correct port and possibly in the daemon form.
NodeA:~>mhttpd -p 8080 -D

== Activate External Elog ==

At this point the Elog from the MIDAS web page is accessing the [[[#Built-in Elog|Built-in]]] (internal) Elog. To activate the external Elog you need to edit two ODB variables (which mhttpd automatically created):

* the ODB string "/Elog/URL" needs to be set to the URL of the external ELOG.
* the ODB bool "/Elog/External Elog" needs to be set to 'y'

Note that after changing the ODB variable 'External Elog' from 'n' to 'y' you will need to reload the elog page in order to get the external elog.

Confirm proper operation of the external Elog by creating an entry. You will be prompted for a username and password. Click on New registration. Full control of these features are described in the Elog documentation.
Stop and restart the Elogd in the background.

NodeB:~>/installation_elog_dir/elogd -c elogd.cfg -x -D

= How to convert Built-in Elog files to External files =
In the event you had a previous entry under the internal elog, you can convert the internal to external using the elconv tool.

NodeB:~> cp internal/elog_logbook/*.log /myexpt/logbook/.
NodeB:~> cd /myexpt/logbook
NodeB:~> /installation_elog_dir/elconv

= How to send entries to the Elog from a script =
Utilities exist to send elog entries automatically (e.g. from a script run at begin-of-run). Use [[Melog|melog]] to send an
entry to a Built-in elog, or [[Elog|elog]] to send an entry to an '''external''' elog.

[[Category:ELOG]]

Electronic Logbook (ELOG)

2023-09-13T21:05:01Z

Olchansk: /* Introduction */

{{Pagelinks}}
= Links =
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[/Elog ODB tree]]
* [[mhttpd]]
* mhttpd [[Elog Page]]
* [https://midas.psi.ch/elog/ '''Elog''' ]
* [[Melog]] Utility
* [[Elog|Elog Utility]]
</div>

= Introduction =
The '''Electronic LogBook''' (ELOG) is a utility built-in the [[mhttpd|web server]]. It is available and operational once the web-server is running. A dedicated button "ELOG" will be present on the main
Midas status page for creating new entries. This mode is referred as the [[#Built-in Elog|Built-in Elog]].

This tool can replace the experimental logbook for daily entries. The main advantage of the Elog over a paper logbook is the possiblity to access it remotely, and provide a general knowledge of the experiment.

More recently, the independent [https://elog.psi.ch/elog/ '''Elog''' ] package can be linked to a Midas experiment. Its advantage over the built-in version is its flexibility. This package is used worldwide and improvements are constantly being made. A full-features documentation and standalone installation can be found at the Elog web site.

== Built-in Elog ==
The ELOG page (accessed by clicking the Elog button if present on the main status page ) provides access to an electronic logbook. the Elog is not limited strictly to experiments. Worldwide Elog implementations can be found on the internet. This internal implementation doesn't requires any setup to be operational. The Elog is customized through the ODB /Elog tree.

== External Elog ==
The external implementation requires a proper Elog installation which is fully described on the Elog web site. The External Elog implementation also requires a dedicated entry in the [[/Elog ODB tree]] as shown in the code below. It also requires the package Elog to be already installed, and properly configured. Once the ODB entry is present, the internal ELOG is disabled.

The External Elog is customized through the configuration file elogd.cfg (see [[#Installation|Installation]] below).

= Installation of External Elog =
Installation requires requires several steps described below.

== Download Elog package ==
* Download the Elog package from the web site mentioned above.
** Windows, Linux, Mac version can be found there. Simple installation procedures are also described. Its installation can be done at the system level or at the user level. The Elog can service multiple Electronic logbooks in parallel and therefore an extra entry in its configuration file can provide specific experimental elog in a similar fashion as the internal one.

== Installation considerations ==
You need to take note of several considerations for its installation. Several locations are required for the different files that elog deals with.
* elog resource directory ( e.g. /elog_installation_dir where elog is installed)
* logbook directory (ex: /myexpt/logbook where the pwd and elog entries are stored).
* The pwd file uses encryption for the user password.

* As this Elog installation is tailored towards an experiment, a restriction applies i.e. ensure that the mhttpd and elog applications shares at least the same file system.
** This means that either both applications runs on the same machine or a nsf mount provides file sharing.
** You need to know the node and ports for both applications. Like mhttpd, elogd also requires a port number for communication through the web (e.g. NodeA:mhttpd -p 8080, NodeB:elogd -p 8081).

== Configuration file elogd.cfg ==
* copy the default midas/src/elogd.cfg from the MIDAS distrbution to your operating directory.
* modify the elogd.cfg to reflect your configuration

# This is a simple elogd configuration file to work with MIDAS
#
[global]
; port under which elogd should run
port = 8081
; password file, created under 'logbook dir'
password file = elog.pwd
; directory under which elog was installed (themes etc.)
resource dir = /elog_installation_dir
; directory where the password file will end up
logbook dir = /myexpt/logbook
; anyone can create it's own account
self register = 1
; URL under which elogd is accessible
url = http://NodeB.triumf.ca:8081
; the "main" tab will bring you back to mhttpd
main tab = Xenon
; this is the URL of mhttpd which must run on a different port
main tab url = http://NodeA:8080
; only needed for email notifications
smtp host = your.smtp.host
; Define one logbook for online use. Severl logbooks can be defined here
[MyOnline]
; directory where the logfiles will be written to
Data dir = /myexpt/logbook
Comment = My MIDAS Experiment Electronic Logbook
; mimic old mhttpd behaviour
Attributes = Run number, Author, Type, System, Subject
Options Type = Routine, Shift Summary, Minor Error, Severe Error, Fix, Question, Info, Modification, Alarm, Test, Other,
Options System = General, DAQ, Detector, Electronics, Target, Beamline
Extendable Options = Type, System
; This substitution will enter the current run number
Preset Run number = $shell(odbedit -e myexpt -h NodeA -d Runinfo -c 'ls -v \"run number\"')
Preset Author = $long_name
Required Attributes = Type, Subject
; Run number and Author cannot be changed
Locked Attributes = Run number, Author
Page Title = ELOG - $subject
Reverse sort = 1
Quick filter = Date, Type, Author
; Don't send any emails
Suppress email to users = 1

== Start the Elog daemon ==
* start the elog daemon. -x is for the shell substitution of the command Preset Run number = $shell(...)
* The argument invokes the odbedit remotely if needed to retrieve the current run number. You will have to ensure the proper path to the odbedit and the proper -e, -h arguments for the experiment and host.

You may want to verify this command from the console.

NodeB:~>/installation_elog_dir/elogd -c elogd.cfg -x<cr> starts the mhttpd at its correct port and possibly in the daemon form.
NodeA:~>mhttpd -p 8080 -D

== Activate External Elog ==

At this point the Elog from the MIDAS web page is accessing the [[[#Built-in Elog|Built-in]]] (internal) Elog. To activate the external Elog you need to edit two ODB variables (which mhttpd automatically created):

* the ODB string "/Elog/URL" needs to be set to the URL of the external ELOG.
* the ODB bool "/Elog/External Elog" needs to be set to 'y'

Note that after changing the ODB variable 'External Elog' from 'n' to 'y' you will need to reload the elog page in order to get the external elog.

Confirm proper operation of the external Elog by creating an entry. You will be prompted for a username and password. Click on New registration. Full control of these features are described in the Elog documentation.
Stop and restart the Elogd in the background.

NodeB:~>/installation_elog_dir/elogd -c elogd.cfg -x -D

= How to convert Built-in Elog files to External files =
In the event you had a previous entry under the internal elog, you can convert the internal to external using the elconv tool.

NodeB:~> cp internal/elog_logbook/*.log /myexpt/logbook/.
NodeB:~> cd /myexpt/logbook
NodeB:~> /installation_elog_dir/elconv

= How to send entries to the Elog from a script =
Utilities exist to send elog entries automatically (e.g. from a script run at begin-of-run). Use [[Melog|melog]] to send an
entry to a Built-in elog, or [[Elog|elog]] to send an entry to an '''external''' elog.

[[Category:ELOG]]

User talk:Zaher

2023-08-17T22:24:24Z

Olchansk: Welcome!

'''Welcome to ''MidasWiki''!'''
We hope you will contribute much and well.
You will probably want to read the [https://www.mediawiki.org/wiki/Special:MyLanguage/Help:Contents help pages].
Again, welcome and have fun! [[User:Olchansk|Olchansk]] ([[User talk:Olchansk|talk]]) 15:24, 17 August 2023 (PDT)

Main Page

2023-02-17T23:04:25Z

Olchansk: /* Documentation */

= Introduction =

MIDAS is a modern data acquisition system developed at [http://www.psi.ch PSI] and [http://www.triumf.ca TRIUMF]. Supported hardware includes VME, Fastbus, CAMAC, RS232, GPIB, USB, ethernet, fiber optic and MSCB attached data acquisition devices. MIDAS is written in C/C++ and runs on Linux, MacOS and MS Windows. It is licensed under the [http://www.gnu.org/copyleft/gpl.html GNU Public License].

= Documentation =

* [[Midas documentation| Main Documentation]]
* [[Installation | Installation Instructions]]
* [[Changelog | Changelog / Upgrade Instructions]]
* [https://daq.triumf.ca/~daqweb/doc/midas-develop/html/modules.html Doxygen] Code structure, functions, etc...
* [https://bitbucket.com/tmidas/midas git repository on bitbucket] Source code, bug tracking, etc
* [https://midas.triumf.ca/forum MIDAS forum] Discussion, announcements, problem reports, requests for improvements

= News and Support=

For news about bug fixes and new releases, go to the
[https://midas.triumf.ca/forum MIDAS forum] hosted by TRIUMF using
the [http://elog.psi.ch/elog ELOG] system. Users can register in this system to be notified automatically via
E-mail when new entries are submitted.

Another source of information is the [https://bitbucket.com/tmidas/midas Bitbucket Repository] of MIDAS, where one can see the latest changes to the software.

Finally, you can read presentations from
* [https://indico.psi.ch/conferenceDisplay.py?ovw=True&confId=3793 2015 MIDAS Workshop]
* [https://indico.triumf.ca/conferenceDisplay.py?confId=2342 2017 MIDAS Workshop]
* [https://indico.triumf.ca/conferenceDisplay.py?confId=2438 2019 MIDAS Workshop]

= Related packages =

* [http://elog.psi.ch/elog ELOG] - Electronics Logbook from PSI
* [[ROOT]] - data analysis package from CERN
* [[ROOTANA]] - ROOT-based analyzer for MIDAS
* [[ROODY]] - viewer for online histograms
* http://elog.psi.ch/rome/ - MEG/PSI data analysis package

History System

2020-07-30T16:56:35Z

Olchansk: /* Write MYSQL-history events */

{{Pagelinks}}

== Links ==
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[/History ODB tree]]
* [[History Page]]
* [[Mhformat|MIDAS history file format]]
* [[Keys in the ODB /Logger tree|/Logger ODB tree]]
* [[Keys in the ODB /Logger/History subtree|/Logger/History subtree]]
* [[Image History]]
</div>

<div id="history file"></div>
= Introduction =

The history system is an add-on capability built into the MIDAS data logger [[mlogger]] to record useful information in parallel to the data logging.
This information is recorded with a time stamp and saved into a '''history file''' (see [[#History drivers|History drivers]]) for later retrieval. One set of files is created per day containing all the requested history events. The history data may be displayed in graphical form using the [[mhttpd]] MIDAS webserver utility, giving the user an easy way of seeing how experimental variables have changed with time.
The history logging will be in action only if the MIDAS data logger is running, but it is not necessary to have any [[Keys in the ODB /Logger tree#/Logger/Channels subtree|data logging channel]] enabled.

Since June 2020, it is possible to also use the midas history system to record images (png/jpeg etc) periodically, for example to later produce a timelapse from a webcam - see the [[Image History]] page for more details.

= MidasHistoryInterface =

The new history configuration defines history channels similar to logger channels. Each history channel creates a history interface class (class MidasHistoryInterface). The [[mlogger|data logger]] can use this interface to write data into the history, mhttpd, mhist & co can use this interface to read data from the history. The interface is defined in history.h

* http://ladd00.triumf.ca/~daqweb/doc/midas/include/history.h

= History drivers =

Presently, the MIDAS distribution contains these drivers:

; MIDAS-History
: this is the traditional history driver, with data stored in the .hst binary files. (format is documented here [[mhformat]], see also [[mhist]] and [[mhdump]])
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#MIDAS subtree |"MIDAS"]]''' is set up to write MIDAS-history.

; ODBC-History
: stores history data in SQL database using the ODBC abstraction layer. See [[#MIDAS ODBC SQL History system|ODBC SQL History system]].

: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#ODBC subtree|"ODBC"]]''' is set up to write ODBC-history.

; SQLITE-History
: stores data in SQL database as SQLITE3 .sqlite3 files. The SQL table format is similar to ODBC-history, performance is similar, no database server required.
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#SQLITE subtree|"SQLITE"]]''' is set up to write SQLITE-history.

; MYSQL-History
:
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#MYSQL subtree|"MYSQL"]]''' is set up to write MYSQL-history.

; FILE-History
: stores each variable in its own file, making it much easier to trace the history of a particular variable.
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#FILE subtree|FILE]]''' is set up to write FILE-history.

More drivers can be easily added by the user (create history_xxx.cxx, add constructor to hs_get_history(), create the ODB subtree /Logger/History/xxx/channel).

= Location of History Files =
;MIDAS-history
:By default, the traditional MIDAS-history is enabled, writing history files into the MIDAS data directory path given by the ODB key [[Keys in the ODB /Logger tree#Data dir|Data dir]].
:If history data is important, it is recommended to write it to a directory that is backed up or archived. (The MIDAS data directory is usually located on a scratch data disk without backups).

:The location of the MIDAS-history data files can be changed by use of the ODB Key [[Keys in the ODB /Logger tree#History dir|History dir]].

;ODBC-History
:The location of ODBC-History files is ...

;SQLITE-History
:The location of SQLITE-History files is the directory path given by the ODB key [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]].

;MYSQL-History
:The location of MYSQL-History files is ...

;FILE-History
:The location of FILE-History files is the directory path given by the ODB key [[Keys in the ODB /Logger/History subtree#History dir|History dir]].

= Types of History Events =

There are two basic types of history events, which are defined in different ways:

* "[[#Frontend History Event|Frontend]]" (or "Equipment") History event composed in the frontend. See [[#Frontend History Event|Frontend History Event]] for details.
* "Virtual" (or "Links") History event composed within the ODB under the [[/History ODB tree#Links subtree|/History/Links subtree]]. See [[#Virtual History Event|Virtual History Event]] for details.

Both these history event definitions take effect when the data logger gets a start run [[Run States and Transitions|transition]]. Any modifications made during the run are not applied until the start of the next run.

== Virtual History Event ==
The history of any ODB variable can be recorded in a "virtual" history event by creating a link to that variable in the [[/History ODB tree#Links subtree|/History/Links subtree]]. This is used for ODB variables that are not in an Equipment.

; To enable Virtual History Events
: Create links as desired in [[/History ODB tree#Links subtree|/History/Links subtree]].

History events are created for each entry under ODB subdirectory {{Odbpath|path=/History/Links}}.

Two types of links are permitted:
<dl>
<dt> ODB key {{Odbpath|path=/history/links/aaa}} is a link to a subdirectory</dt>
<dd><ol>
<li> db_watch() is setup to watch this subdirectory</li>
<li> tags are created for each subdirectory entry (1 tag per entry)</li>
<li> There is no possibility for naming array elements, so 1 tag per array, regardless of the number of elements.</li> </ol></dd>
<dt>ODB key {{Odbpath|path=/history/links/bbb}} is a subdirectory with links to ODB values</dt>
<dd> <ol>
<li>db_watch is setup to watch each link target</li>
<li> tags are created for each link (1 tag per link)</li>
<li> tag name is the link name (NOT the target name). There is no possibility for naming array elements</li>
</ol></dd>
</dl>

Mixing links and subdirectories is not permitted.

The update period of history events created for {{Odbpath|path=/history/links}} is controlled by entries in {{Odbpath|path=/history/links periods}}. Numeric values of periods are same as for a [[#Frontend History Event|Frontend History event]]. Numeric value 0 disables the history for a particular event.

See [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087] for more information.

== Frontend History Event ==

Each [[Frontend Operation#Equipments and Events|Equipment]] has the capability to generate "history data".

; To enable Frontend History Events

: Set the [[/Equipment ODB tree#Log history|Log history]] ODB key of each equipment to be logged '''non-zero'''. (See [[Equipment List Parameters#Log History|Log History parameter]]). (You may need to restart the frontend.)

This causes the event contents to be copied to the ODB [[/Equipment ODB tree#Variables subtree|/Equipment/''<equipment-name>''/Variables subtree]] for the use of the History system. history events are created by parsing the content of {{Odbpath|path=/Equipment/<equipment-name>/variables}}.

For each equipment, history is controlled by the value of {{Odbpath|path=/Equipment/<equipment-name>/common/period}} as follows:
* 0 = history disabled
* 1 = history is enabled
* >1 = history is enabled, throttled down

The throttling is implemented in log_history()/watch_history() by this algorithm:
the very first history event is recorded, then all changed to the data are ignored until "period" seconds has elapsed. Then the next history event will be recorded, and following
changes will be ignored until "period" second elapses, and so forth. Period value "1" has special meaning - there is no throttling, all history events are logged.

This information is from [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087].

=== Sequence for frontend History Event ===
The main steps for the frontend History Event is as follows:

* The user code in the frontend equipment Event Readout routines reads the data, placing it into a MIDAS data bank
* In mfe.c, if the key [[/Equipment ODB tree#Log history|Log history]] > 0 (i.e. the history system is enabled for this equipment), this data bank is written into ODB (by update_odb()).
* This ODB write triggers an ODB hot-link into mlogger
* The hot-link calls mlogger.c::log_history(), which calls hs_write() to write the data into the history file. The frequency of the history writes is specified by the number of seconds between writes stored in the ODB key [[/Equipment ODB tree#Log history|Log history]].
* history.c::hs_write() or history_odbc::hs_write_odbc() writes the data into a history file or into an SQL database.

= How to Write History events =
* Enable desired [[#To enable Frontend History Events|Frontend]] and/or [[#To enable Virtual History Events|Virtual]] history events
* [[#Select the history driver|Select the history driver]]
* Follow instructions below for selected driver:
** [[#Write MIDAS-History events|Write MIDAS-history events]]
** [[#Write ODBC-History events|Write ODBC-history events]]
** [[#Write SQLITE-History events|Write SQLITE-history events]]
** [[#Write MYSQL-History events|Write MYSQL-history events]]
** [[#Write FILE-History events|Write FILE-history events]]

== Select the history driver ==

* Select the '''[[#History drivers|history driver]]''' and therefore the [[Keys in the ODB /Logger/History subtree#<history-logging-channel> subtrees|history-logging-channel]] to use to write the MIDAS-history files.
* set [[Keys in the ODB /Logger/History subtree#Active|Active]] ODB key in the selected [[Keys in the ODB /Logger/History subtree#<history-logging-channel> subtrees|history-logging-channel]] to "y". All other logging channels should be disabled.
* set [[/History ODB tree#LoggerHistoryChannel|LoggerHistoryChannel]] ODB key to the selected [[#History drivers|History channel]].

== Write MIDAS-History events ==
* Enable desired [[#To enable Frontend History Events|Frontend]] and/or [[#To enable Virtual History Events|Virtual]] history events
* Perform steps in [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "MIDAS".
* Restart [[mhttpd]] and [[mlogger]], start the run.

MIDAS-history files will be saved into the [[#Location of History Files|MIDAS-history data directory]].

== Write ODBC-History events ==

See [[#MIDAS ODBC SQL History system|MIDAS ODBC SQL History system]].
* If necessary, [[#Set up a MySQL database|set up a MySQL database]].
* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps in [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "ODBC".
* Check ODB Key [[Keys in the ODB /Logger/History subtree#Writer_ODBC_DSN|Writer_ODBC_DSN]] is set to "history writer" and Key [[Keys in the ODB /Logger/History subtree#Reader_ODBC_DSN|Reader_ODBC_DSN]] is set to "history reader" (the default values)
* set [[Keys in the ODB /Logger/History subtree#Debug|Debug]] to 1 if desired
* restart [[mlogger]] in verbose mode (i.e. mlogger -v), and observe how it issues SQL commands to create the tables and columns corresponding to MIDAS-history events and tags.
* restart [[mhttpd]]

'''Notes'''
# [[mlogger]] is programmed to raise alarms if connection to SQL database is interrupted or if some events cannot be written into the database (i.e. data type mismatch, SQL syntax errors, etc). The traditional MIDAS-history never raised alarms because it "never failed" - other than from "disk full" errors, which are immediately obvious.
# With ODBC History enabled, [[mhttpd]] will only use history information from the SQL database to make history plots and to extract history variable names for the history plot editor. The ODB {{Odbpath|path=/History}} subtrees[[/History ODB tree#Tags subtree|Tags]] and [[/History ODB tree#Events subtree|Events]] are not used.
# variable names presented to the user may change from MIDAS names to SQL names (use the history panel "Label" text fields to create permanent plot labels).

== Write SQLITE-history events ==

No special configuration required.

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "SQLITE".

* Create a directory for storage of sqlite history files
* save its path in the ODB key [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]]
* restart [[mhttpd]] and [[mlogger]]

In the history data directory (in [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]]) you should see files with name "mh_xxx.sqlite3" corresponding to your equipment and variable names.

== Write MYSQL-history events ==

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "MYSQL".
* setup the MySQL database:
* install mariadb-10.4
<pre>
macos:
sudo port install mariadb-10.4
ln -s ln -fs /opt/local/lib/mariadb-10.4/bin/mysql_config ~/bin/
mysql_config --version
10.4.13
</pre>
* rebuild midas, check the cmake finds the correct mysql: MIDAS: Found MySQL version 10.4.13
* create $HOME/.my.cnf
<pre>
[mysqld]
#port=3306
socket=/Users/olchansk/mysql/mysql.sock
datadir=/Users/olchansk/mysql/db
pid-file=/Users/olchansk/mysql/mysqld.pid

[client]
#port=3306
socket=/Users/olchansk/mysql/mysql.sock
</pre>
* mkdir $HOME/mysql
* start mysql:
<pre>
macos: /opt/local/lib/mariadb-10.4/bin/mysqld
</pre>
* configure the database: start the "mysql" utility (/opt/local/lib/mariadb-10.4/bin/mysql)
<pre>
NOTE: instructions thanks to Ben S.
NOTE: use some other passwords instead of the example "reader_password" and "writer_password"

MariaDB [(none)]> CREATE DATABASE IF NOT EXISTS history;
Query OK, 1 row affected (0.003 sec)

MariaDB [(none)]> CREATE USER history_reader@'localhost' IDENTIFIED BY 'reader_password';
Query OK, 0 rows affected (0.027 sec)

MariaDB [(none)]> GRANT SELECT ON history.* TO history_reader@'localhost';
Query OK, 0 rows affected (0.016 sec)

MariaDB [(none)]> CREATE USER 'history_writer'@'localhost' IDENTIFIED BY 'writer_password';
Query OK, 0 rows affected (0.014 sec)

MariaDB [(none)]> GRANT SELECT,INSERT,CREATE,ALTER,INDEX ON history.* TO 'history_writer'@'localhost';
Query OK, 0 rows affected (0.017 sec)

MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.001 sec)
</pre>
* try connecting to both users:
<pre>
/opt/local/lib/mariadb-10.4/bin/mysql -u history_reader -p ### should ask for reader password
/opt/local/lib/mariadb-10.4/bin/mysql -u history_writer -p ### should ask for writer password
</pre>
* create midas config files:
** mysql_writer.txt
<pre>
#server=localhost
#port=xxx
database=history
socket=/Users/olchansk/mysql/mysql.sock
user=history_writer
password=writer_password ### change this!!!
#buffer=1000
</pre>
** mysql_reader.txt
<pre>
#server=localhost
#port=xxx
database=history
socket=/Users/olchansk/mysql/mysql.sock
user=history_reader
password=reader_password ### change this!!!
</pre>
* test MIDAS connection
<pre>
mh2sql --mysql mysql_writer.txt
(there should be no errors printed)
</pre>
* if desired, import midas history from .hst files:
<pre>
mh2sql --mysql mysql_writer.txt *.hst
Reading 130813.hst
[mh2sql,ERROR] [history_schema.cxx:1165:Mysql::Prepare,ERROR] mysql_query(SELECT event_name, table_name, itimestamp FROM _history_index WHERE table_name!='';) error 1146 (Table 'history._history_index' doesn't exist)
[mh2sql,INFO] Adding SQL table "rpcexample", status 311
[mh2sql,INFO] Adding SQL table "rpcexample_1596127680", status 1
[mh2sql,INFO] Adding SQL table "_history_index", status 1
[mh2sql,INFO] Adding column "event_name" to SQL table "_history_index", status 1
...
[mh2sql,INFO] Adding SQL table "run_transitions", status 311
[mh2sql,INFO] Adding SQL table "run_transitions_1596127681", status 1
[mh2sql,INFO] Adding column "state" to SQL table "run_transitions_1596127681", status 1
[mh2sql,INFO] Adding column "run_number" to SQL table "run_transitions_1596127681", status 1
Reading 130815.hst
...
</pre>
* enable MYSQL history in mlogger:
** change ODB /Logger/History/MYSQL/Active to "y", "mysql writer" and "mysql reader" to location of these files (leave the default values if these files are in the experiment directory).
* enable MYSQL history in mhttpd:
** change ODB /History/LoggerHistoryChannel to "MYSQL"
* restart mlogger, restart mhttpd
...........

== Write FILE-history events ==

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "SQLITE".
* Assign directory path for saved files using the ODB key [[Keys in the ODB /Logger/History subtree#History dir|History dir]].
...........

 
---------------
 

= History Events =
The following information is derived from [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087].

A "history event" is a history atomic unit of data. Associated with each history event is a timestamp (unix time),
a name (limited to NAME_LENGTH in the old history) and a list of history tags that describe the individual data
values inside the history event.

When making history plots in [[mhttpd]] (see [[History Page]]) for each curve on the plot, one selects a history event (from the list
of currently active events, recently active events or the list of all events that ever existed), then from the list of tags
inside the history event one selects the particular variable that will be plotted.

== Old MIDAS history events ==
In the old MIDAS history, all history events are written into one history file (.hst file + optional .def and .idx event definition and time index files
which can be/are regenerated automatically from the .hst file). History events are identified by 16-bit history event IDs, the persistent mapping
from history event names and the 16-bit history event IDs is stored in ODB subtree {{Odbpath|path=/History/Events}}. In addition the list of all known history event tags is
stored in ODB subtree {{Odbpath|path=/History/Tags}}. For per-equipment history, the 16-bit history event ID is the value of ODB key [[/Equipment ODB tree#Event ID|/Equipment/<equipment-name>/Common/Event ID]].

== SQL history events ==
In the SQL history (MySQL, SQLITE, etc), each history event is an SQL table. The history event tags are the SQL table columns.

== FILE history events ==
In the new FILE history, each history event is written into a separate file, tag definition are recorded in text format in the file header, history event
data is appended to the file in binary format (fixed record size). If the history event definition is changed, a new file will be started.

= History Event Construction =
The [[mlogger]] creates [[#Frontend history event]]s in open_history() by parsing the ODB subtree {{Odbpath|path=/equipment/<equipment-name>/variables}}. Each ODB entry under the {{Odbpath|path=variables/}} subtree is referred to as a "variable". Each variable can be a single ODB value, an array of ODB values, or a subdirectory (corresponding to TID_STRUCT structured data banks). As each variable is processed, one or more tags are created to describe it.

Single ODB values will generally produce a single tag, while arrays can produce (depending whether the array is "[[#Named Arrays|Named]]" or not)
* multiple tags - one per array element ("Named" array) or
* one single tag - describing the whole array

The code can generate two types of history:
<dl>
<dt> "per-equipment" history</dt>
<dd> will have the tags for all variables concatenated together into one single history event</dd>
<dt> "per-variable" history</dt>
<dd> will have one history event defined for each variable. Inside could be one tag (for single odb values and unnamed array) or multiple tags (for named arrays and structured data
banks).</dd>
</dl>

Per-equipment history is the original MIDAS history implementation. Per-variable history was added to permit efficient data storage in SQL tables. Its initial implementation used 1 ODB hotlink for each variable and it was easy to exceed the maximum permitted number of ODB hotlinks (db_open_record()). To reduce consumption of hotlinks, db_watch() has been implemented and now per-variable history only uses 1 ODB hotlink per equipment.

;NOTE
: With db_watch, per-equipment history is no longer available. '''Per-variable history is the new default''' (and the only option).

[https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087]

= Construction of history event tags =
Currently,
* single ODB values are treated as arrays of length "1"
* string ODB entries are not permitted

History event tags are constructed in the [[mlogger]] add_equipment(). Presently there are some quirks with the generation of history event tags (see [https://midas.triumf.ca/elog/Midas/1088]),
e.g. single odb values are treated as arrays of length "1" and "string" ODB entries are not permitted.

For variables of type TID_KEY (subdirectories, corresponding to TID_STRUCT structured banks), one tag is generated for each subdirectory entry. Tag names for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa/bbb}} will be "aaa_bbb" (with an underscore). Subdirectory entries of type TID_KEY and TID_LINK are currently not supported.

For "[[#Named Arrays|Named Arrays]]", individual tags are generated for each array element. Tag names are taken from the ''Names'' array. For empty tag names (empty names array), tags are "aaa_0", "aaa_1", etc (for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}}).

For "[[#Named Arrays|Single Names Arrays]]", tag names have the variable name appended (with a space), for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}} and an empty names array, tags will be "aaa_0 aaa", "aaa_1 aaa", etc. For a populated names array, the tags will be "name0 aaa", "name1 aaa", etc.

For unnamed arrays and single odb variables (in ODB, single odb variables are arrays of length 1), a single tag is generated.

== Named Arrays ==

'''Named Arrays''' have a "Names" entry in {{Odbpath|path=/equipment/<equipment-name>/settings}}.

For example, to add names to {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}}, create a string array {{Odbpath|path=/equipment/<equipment-name>/settings/names aaa}}. The names array should be at least as long as the corresponding data array. Individual entries in the names array can be left blank (tag names will be "aaa_0", "aaa_1", etc). Duplicate tag names are not permitted.

A single "Names" entry can be created to name all arrays in variables with the same names ('''"Single Names"''').

Create {{Odbpath|path=/equipment/<equipment-name>/settings/names}} and arrays {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}} and {{Odbpath|path=/equipment/<equipment-name>/variables/bbb}} will have
history tags "name0 aaa", "name1 aaa", "name0 bbb", "name1 bbb", etc. If "names" are left blank, tag names will be "aaa_0 aaa", "aaa_1 aaa", "bbb_0 bbb", "bbb_1 bbb", etc.

In the [[mhttpd]] web server [[History Page]] variables viewer, "single name" arrays are displayed in a 2D table.

This information is from [https://midas.triumf.ca/elog/Midas/1088].

= Historical note on MIDAS-History Tag names =
Tags are only used for MIDAS-history events.

The mlogger/mhttpd implementation of /History/Tags has proved troublesome and we are moving away from it.

;During the transition period:
: Implementations starting with Version 4435 of [[mlogger]] and [[mhttpd]] (i.e. "new" versions) will now work '''without''' /History/Tags. The history tags are read directly from the history files themselves. To run in this mode, set ODB key [[/History ODB tree#DisableTags|/History/DisableTags]] to "y". Existing [[/History ODB tree#Tags subtree|/History/Tags subtree]] will be automatically deleted.

Two downsides to running like this are
* it is slower and tags become non-persistent
* if some frontends have not been running for a while, their variables may vanish from the history panel editor.

== New-style History Tags ==
For the above two reasons, using /History/Tags is still recommended.
However, the format of the tags is now changed to simplify management and reduce ODB size.

; For NEW implementations
: the key [[/History ODB tree#CreateOldTags|CreateOldTags]] will not be present. "New-style" history tags will be created automatically in the [[/History ODB tree#Tags subtree|Tags subtree]].

;If you have old-style tags
: [[mlogger]] will automatically convert the old tags to this new format (this is why you should make a backup of your ODB).

** using "old" (i.e. pre Version 4435) mlogger with "new" mhttpd is OK: new mhttpd understands both formats of /History/Tags.
** using "old" mhttpd with "new" mlogger is OK provided the ODB key [[/History ODB tree#CreateOldTags|/History/CreateOldTags]] is set to "y" before starting mlogger.

== Old-style History Tags ==
Prior to versions 4435 of [[mlogger]] and [[mhttpd]], old-style history tags were generated.
See
* http://ladd00.triumf.ca/~daqweb/doc/midas-old/html/F_History_logging.html
for details.

 
---------------
 

= MIDAS ODBC SQL History system =

This section describes the internal workings of the MIDAS SQL history system.

The SQL history system implements the MIDAS history using a SQL database for data storage instead of flat binary files.

The SQL history code lives in a separate files history.h and history_sql.cxx. The present implementation uses the ODBC API for accessing SQL databases.
MySQL is fully supported, Pgsql support is partially implemented, support for arbitrary ODBC drivers should be easy to add in history_sql and most SQL operations are done using an interface class.

Preliminary testing with MySQL and PgSQL indicates that disk space requirements are the same for both storage mechanisms using flat binary files or either of the 2 SQL databases. All 3 tested storage mechanisms store the data uncompressed. (Observed gzip-1 compression ratios from CERN-ALPHA history files are 50% or better).

Preliminary testing also indicates that MySQL database is "not too slow" for use by T2K/ND280 slow control system. Using MySQL, performance seems to improve somewhat compared to traditional MIDAS-history because each SQL table is stored into a separate file, compared to everything-in-one-file in traditional MIDAS-history.

== Set up a MySQL database ==

In order to setup a MySQL database, the following is needed:

* the server hostname and port number
* database name
* user name and password.

It is recommended to create 3 different users:

* a "root" user with full priveleges,
* a "reader" user with read-only priveleges
* and a "writer" user with "insert, create table, and add column" priveleges (no permission to drop table, remove columns, etc)

=== Example ===
#Setup a MySQL database (RHEL/SL 5)

ssh root@localhost
service mysqld start
/usr/bin/mysql_secure_installation (enter current password for root: press <enter>, then accept default answer "Y" to all questions)
mysqladmin -p create history
using mysql-administrator, create 2 users:
history_writer, with database "history" schema privileges: select, insert, create, alter, index
history_reader, with database "history" schema privileges: select

#Setup $HOME/.odbc.ini file:

cat $HOME/.odbc.ini
[history_writer]
Description = history_writer
Driver = /usr/lib64/libmyodbc3.so
SERVER = ladd05.triumf.ca
PORT =
Database = history
OPTION = 3
SOCKET =
User = history_writer
Password = ******
[history_reader]
Description = history_reader
Driver = /usr/lib64/libmyodbc3.so
SERVER = ladd05.triumf.ca
PORT =
Database = history
OPTION = 3
SOCKET =
User = history_reader
Password =

Note: if you are using mhist you will need a separate .odbc.ini file for each user that you have connecting to the database (through MIDAS).

== Layout of SQL tables ==

* One table is created for each history event. Table name is the same as the history event name (as reported by [[mhdump]]).
* one column is created for each history tag (history variable). Column name is the same as the tag name (as reported by mhdump). For array tags, one column is created for each array element: array a[3] will produce columns a_0, a_1 and a_2.
* two special columns are always created: _i_time and _t_time are the timestamps in the UNIX-time integer format and in the SQL "time" format. (A priori, it is not clear which timestamp format is more useful for end users. mhttpd uses the _i_time integer time format). Note that this timestamp is the time when mlogger receives the hot-link signal (see above) and it can be a few seconds behind the time when the user placed the data into the MIDAS bank. For high precision (and sub-second time resolution) time stamps, users should generate their own timestamps and include them as part of the data itself.
* a special table called _history_index is created to remember the mapping between SQL names and MIDAS names for event names, variable name and tag names.

=== Schema Evolution ===

* when new data fields are added, new SQL columns are created automatically
* existing SQL columns are never deleted
* arrays are expanded automatically, and arrays never shrink.
* if the type of a data field changes, a conflict may occur with existing SQL columns, for example if previous type was "FLOAT" and new type is "DOUBLE", the existing SQL columns would have the SQL data type "float", too narrow to store new values. mlogger will complain about this. Such conflicts may be resolved by changing the type of the SQL column using SQL tools. In general, SQL names are created from MIDAS names by replacing all non-alphanumerical characters to an underscore "_". If this procedure creates duplicate column names, the column names and made unique by appending them with a random number, i.e. MIDAS names "A+B" and "A-B" will be translated to "A_B" and "A_B_12345" (random integer). True MIDAS names are saved into the _history_index.

[[Category:history]]

History System

2020-07-30T16:56:10Z

Olchansk: /* Write MYSQL-history events */

{{Pagelinks}}

== Links ==
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[/History ODB tree]]
* [[History Page]]
* [[Mhformat|MIDAS history file format]]
* [[Keys in the ODB /Logger tree|/Logger ODB tree]]
* [[Keys in the ODB /Logger/History subtree|/Logger/History subtree]]
* [[Image History]]
</div>

<div id="history file"></div>
= Introduction =

The history system is an add-on capability built into the MIDAS data logger [[mlogger]] to record useful information in parallel to the data logging.
This information is recorded with a time stamp and saved into a '''history file''' (see [[#History drivers|History drivers]]) for later retrieval. One set of files is created per day containing all the requested history events. The history data may be displayed in graphical form using the [[mhttpd]] MIDAS webserver utility, giving the user an easy way of seeing how experimental variables have changed with time.
The history logging will be in action only if the MIDAS data logger is running, but it is not necessary to have any [[Keys in the ODB /Logger tree#/Logger/Channels subtree|data logging channel]] enabled.

Since June 2020, it is possible to also use the midas history system to record images (png/jpeg etc) periodically, for example to later produce a timelapse from a webcam - see the [[Image History]] page for more details.

= MidasHistoryInterface =

The new history configuration defines history channels similar to logger channels. Each history channel creates a history interface class (class MidasHistoryInterface). The [[mlogger|data logger]] can use this interface to write data into the history, mhttpd, mhist & co can use this interface to read data from the history. The interface is defined in history.h

* http://ladd00.triumf.ca/~daqweb/doc/midas/include/history.h

= History drivers =

Presently, the MIDAS distribution contains these drivers:

; MIDAS-History
: this is the traditional history driver, with data stored in the .hst binary files. (format is documented here [[mhformat]], see also [[mhist]] and [[mhdump]])
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#MIDAS subtree |"MIDAS"]]''' is set up to write MIDAS-history.

; ODBC-History
: stores history data in SQL database using the ODBC abstraction layer. See [[#MIDAS ODBC SQL History system|ODBC SQL History system]].

: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#ODBC subtree|"ODBC"]]''' is set up to write ODBC-history.

; SQLITE-History
: stores data in SQL database as SQLITE3 .sqlite3 files. The SQL table format is similar to ODBC-history, performance is similar, no database server required.
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#SQLITE subtree|"SQLITE"]]''' is set up to write SQLITE-history.

; MYSQL-History
:
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#MYSQL subtree|"MYSQL"]]''' is set up to write MYSQL-history.

; FILE-History
: stores each variable in its own file, making it much easier to trace the history of a particular variable.
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#FILE subtree|FILE]]''' is set up to write FILE-history.

More drivers can be easily added by the user (create history_xxx.cxx, add constructor to hs_get_history(), create the ODB subtree /Logger/History/xxx/channel).

= Location of History Files =
;MIDAS-history
:By default, the traditional MIDAS-history is enabled, writing history files into the MIDAS data directory path given by the ODB key [[Keys in the ODB /Logger tree#Data dir|Data dir]].
:If history data is important, it is recommended to write it to a directory that is backed up or archived. (The MIDAS data directory is usually located on a scratch data disk without backups).

:The location of the MIDAS-history data files can be changed by use of the ODB Key [[Keys in the ODB /Logger tree#History dir|History dir]].

;ODBC-History
:The location of ODBC-History files is ...

;SQLITE-History
:The location of SQLITE-History files is the directory path given by the ODB key [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]].

;MYSQL-History
:The location of MYSQL-History files is ...

;FILE-History
:The location of FILE-History files is the directory path given by the ODB key [[Keys in the ODB /Logger/History subtree#History dir|History dir]].

= Types of History Events =

There are two basic types of history events, which are defined in different ways:

* "[[#Frontend History Event|Frontend]]" (or "Equipment") History event composed in the frontend. See [[#Frontend History Event|Frontend History Event]] for details.
* "Virtual" (or "Links") History event composed within the ODB under the [[/History ODB tree#Links subtree|/History/Links subtree]]. See [[#Virtual History Event|Virtual History Event]] for details.

Both these history event definitions take effect when the data logger gets a start run [[Run States and Transitions|transition]]. Any modifications made during the run are not applied until the start of the next run.

== Virtual History Event ==
The history of any ODB variable can be recorded in a "virtual" history event by creating a link to that variable in the [[/History ODB tree#Links subtree|/History/Links subtree]]. This is used for ODB variables that are not in an Equipment.

; To enable Virtual History Events
: Create links as desired in [[/History ODB tree#Links subtree|/History/Links subtree]].

History events are created for each entry under ODB subdirectory {{Odbpath|path=/History/Links}}.

Two types of links are permitted:
<dl>
<dt> ODB key {{Odbpath|path=/history/links/aaa}} is a link to a subdirectory</dt>
<dd><ol>
<li> db_watch() is setup to watch this subdirectory</li>
<li> tags are created for each subdirectory entry (1 tag per entry)</li>
<li> There is no possibility for naming array elements, so 1 tag per array, regardless of the number of elements.</li> </ol></dd>
<dt>ODB key {{Odbpath|path=/history/links/bbb}} is a subdirectory with links to ODB values</dt>
<dd> <ol>
<li>db_watch is setup to watch each link target</li>
<li> tags are created for each link (1 tag per link)</li>
<li> tag name is the link name (NOT the target name). There is no possibility for naming array elements</li>
</ol></dd>
</dl>

Mixing links and subdirectories is not permitted.

The update period of history events created for {{Odbpath|path=/history/links}} is controlled by entries in {{Odbpath|path=/history/links periods}}. Numeric values of periods are same as for a [[#Frontend History Event|Frontend History event]]. Numeric value 0 disables the history for a particular event.

See [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087] for more information.

== Frontend History Event ==

Each [[Frontend Operation#Equipments and Events|Equipment]] has the capability to generate "history data".

; To enable Frontend History Events

: Set the [[/Equipment ODB tree#Log history|Log history]] ODB key of each equipment to be logged '''non-zero'''. (See [[Equipment List Parameters#Log History|Log History parameter]]). (You may need to restart the frontend.)

This causes the event contents to be copied to the ODB [[/Equipment ODB tree#Variables subtree|/Equipment/''<equipment-name>''/Variables subtree]] for the use of the History system. history events are created by parsing the content of {{Odbpath|path=/Equipment/<equipment-name>/variables}}.

For each equipment, history is controlled by the value of {{Odbpath|path=/Equipment/<equipment-name>/common/period}} as follows:
* 0 = history disabled
* 1 = history is enabled
* >1 = history is enabled, throttled down

The throttling is implemented in log_history()/watch_history() by this algorithm:
the very first history event is recorded, then all changed to the data are ignored until "period" seconds has elapsed. Then the next history event will be recorded, and following
changes will be ignored until "period" second elapses, and so forth. Period value "1" has special meaning - there is no throttling, all history events are logged.

This information is from [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087].

=== Sequence for frontend History Event ===
The main steps for the frontend History Event is as follows:

* The user code in the frontend equipment Event Readout routines reads the data, placing it into a MIDAS data bank
* In mfe.c, if the key [[/Equipment ODB tree#Log history|Log history]] > 0 (i.e. the history system is enabled for this equipment), this data bank is written into ODB (by update_odb()).
* This ODB write triggers an ODB hot-link into mlogger
* The hot-link calls mlogger.c::log_history(), which calls hs_write() to write the data into the history file. The frequency of the history writes is specified by the number of seconds between writes stored in the ODB key [[/Equipment ODB tree#Log history|Log history]].
* history.c::hs_write() or history_odbc::hs_write_odbc() writes the data into a history file or into an SQL database.

= How to Write History events =
* Enable desired [[#To enable Frontend History Events|Frontend]] and/or [[#To enable Virtual History Events|Virtual]] history events
* [[#Select the history driver|Select the history driver]]
* Follow instructions below for selected driver:
** [[#Write MIDAS-History events|Write MIDAS-history events]]
** [[#Write ODBC-History events|Write ODBC-history events]]
** [[#Write SQLITE-History events|Write SQLITE-history events]]
** [[#Write MYSQL-History events|Write MYSQL-history events]]
** [[#Write FILE-History events|Write FILE-history events]]

== Select the history driver ==

* Select the '''[[#History drivers|history driver]]''' and therefore the [[Keys in the ODB /Logger/History subtree#<history-logging-channel> subtrees|history-logging-channel]] to use to write the MIDAS-history files.
* set [[Keys in the ODB /Logger/History subtree#Active|Active]] ODB key in the selected [[Keys in the ODB /Logger/History subtree#<history-logging-channel> subtrees|history-logging-channel]] to "y". All other logging channels should be disabled.
* set [[/History ODB tree#LoggerHistoryChannel|LoggerHistoryChannel]] ODB key to the selected [[#History drivers|History channel]].

== Write MIDAS-History events ==
* Enable desired [[#To enable Frontend History Events|Frontend]] and/or [[#To enable Virtual History Events|Virtual]] history events
* Perform steps in [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "MIDAS".
* Restart [[mhttpd]] and [[mlogger]], start the run.

MIDAS-history files will be saved into the [[#Location of History Files|MIDAS-history data directory]].

== Write ODBC-History events ==

See [[#MIDAS ODBC SQL History system|MIDAS ODBC SQL History system]].
* If necessary, [[#Set up a MySQL database|set up a MySQL database]].
* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps in [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "ODBC".
* Check ODB Key [[Keys in the ODB /Logger/History subtree#Writer_ODBC_DSN|Writer_ODBC_DSN]] is set to "history writer" and Key [[Keys in the ODB /Logger/History subtree#Reader_ODBC_DSN|Reader_ODBC_DSN]] is set to "history reader" (the default values)
* set [[Keys in the ODB /Logger/History subtree#Debug|Debug]] to 1 if desired
* restart [[mlogger]] in verbose mode (i.e. mlogger -v), and observe how it issues SQL commands to create the tables and columns corresponding to MIDAS-history events and tags.
* restart [[mhttpd]]

'''Notes'''
# [[mlogger]] is programmed to raise alarms if connection to SQL database is interrupted or if some events cannot be written into the database (i.e. data type mismatch, SQL syntax errors, etc). The traditional MIDAS-history never raised alarms because it "never failed" - other than from "disk full" errors, which are immediately obvious.
# With ODBC History enabled, [[mhttpd]] will only use history information from the SQL database to make history plots and to extract history variable names for the history plot editor. The ODB {{Odbpath|path=/History}} subtrees[[/History ODB tree#Tags subtree|Tags]] and [[/History ODB tree#Events subtree|Events]] are not used.
# variable names presented to the user may change from MIDAS names to SQL names (use the history panel "Label" text fields to create permanent plot labels).

== Write SQLITE-history events ==

No special configuration required.

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "SQLITE".

* Create a directory for storage of sqlite history files
* save its path in the ODB key [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]]
* restart [[mhttpd]] and [[mlogger]]

In the history data directory (in [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]]) you should see files with name "mh_xxx.sqlite3" corresponding to your equipment and variable names.

== Write MYSQL-history events ==

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "MYSQL".
* setup the MySQL database:
* install mariadb-10.4
<pre>
macos:
sudo port install mariadb-10.4
ln -s ln -fs /opt/local/lib/mariadb-10.4/bin/mysql_config ~/bin/
mysql_config --version
10.4.13
</pre>
* rebuild midas, check the cmake finds the correct mysql: MIDAS: Found MySQL version 10.4.13
* create $HOME/.my.cnf
<pre>
[mysqld]
#port=3306
socket=/Users/olchansk/mysql/mysql.sock
datadir=/Users/olchansk/mysql/db
pid-file=/Users/olchansk/mysql/mysqld.pid

[client]
#port=3306
socket=/Users/olchansk/mysql/mysql.sock
</pre>
* mkdir $HOME/mysql
* start mysql:
<pre>
macos: /opt/local/lib/mariadb-10.4/bin/mysqld
</pre>
* configure the database: start the "mysql" utility (/opt/local/lib/mariadb-10.4/bin/mysql)
<pre>
NOTE: instructions thanks to Ben S.
NOTE: use some other passwords instead of the example "reader_password" and "writer_password"

MariaDB [(none)]> CREATE DATABASE IF NOT EXISTS history;
Query OK, 1 row affected (0.003 sec)

MariaDB [(none)]> CREATE USER history_reader@'localhost' IDENTIFIED BY 'reader_password';
Query OK, 0 rows affected (0.027 sec)

MariaDB [(none)]> GRANT SELECT ON history.* TO history_reader@'localhost';
Query OK, 0 rows affected (0.016 sec)

MariaDB [(none)]> CREATE USER 'history_writer'@'localhost' IDENTIFIED BY 'writer_password';
Query OK, 0 rows affected (0.014 sec)

MariaDB [(none)]> GRANT SELECT,INSERT,CREATE,ALTER,INDEX ON history.* TO 'history_writer'@'localhost';
Query OK, 0 rows affected (0.017 sec)

MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.001 sec)
</pre>
* try connecting to both users:
<pre>
/opt/local/lib/mariadb-10.4/bin/mysql -u history_reader -p ### should ask for reader password
/opt/local/lib/mariadb-10.4/bin/mysql -u history_writer -p ### should ask for writer password
</pre>
* create midas config files:
** mysql_writer.txt
<pre>
#server=localhost
#port=xxx
database=history
socket=/Users/olchansk/mysql/mysql.sock
user=history_writer
password=writer_password ### change this!!!
#buffer=1000
</pre>
** mysql_reader.txt
<pre>
#server=localhost
#port=xxx
database=history
socket=/Users/olchansk/mysql/mysql.sock
user=history_reader
password=reader_password ### change this!!!
</pre>
* test MIDAS connection
<pre>
mh2sql --mysql mysql_writer.txt
(there should be no errors printed)
</pre>
* if desired, import midas history from .hst files:
<pre>
mh2sql --mysql mysql_writer.txt *.hst
Reading 130813.hst
[mh2sql,ERROR] [history_schema.cxx:1165:Mysql::Prepare,ERROR] mysql_query(SELECT event_name, table_name, itimestamp FROM _history_index WHERE table_name!='';) error 1146 (Table 'history._history_index' doesn't exist)
[mh2sql,INFO] Adding SQL table "rpcexample", status 311
[mh2sql,INFO] Adding SQL table "rpcexample_1596127680", status 1
[mh2sql,INFO] Adding SQL table "_history_index", status 1
[mh2sql,INFO] Adding column "event_name" to SQL table "_history_index", status 1
...
[mh2sql,INFO] Adding SQL table "run_transitions", status 311
[mh2sql,INFO] Adding SQL table "run_transitions_1596127681", status 1
[mh2sql,INFO] Adding column "state" to SQL table "run_transitions_1596127681", status 1
[mh2sql,INFO] Adding column "run_number" to SQL table "run_transitions_1596127681", status 1
Reading 130815.hst
...
</re>
* enable MYSQL history in mlogger:
** change ODB /Logger/History/MYSQL/Active to "y", "mysql writer" and "mysql reader" to location of these files (leave the default values if these files are in the experiment directory).
* enable MYSQL history in mhttpd:
** change ODB /History/LoggerHistoryChannel to "MYSQL"
* restart mlogger, restart mhttpd
...........

== Write FILE-history events ==

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "SQLITE".
* Assign directory path for saved files using the ODB key [[Keys in the ODB /Logger/History subtree#History dir|History dir]].
...........

 
---------------
 

= History Events =
The following information is derived from [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087].

A "history event" is a history atomic unit of data. Associated with each history event is a timestamp (unix time),
a name (limited to NAME_LENGTH in the old history) and a list of history tags that describe the individual data
values inside the history event.

When making history plots in [[mhttpd]] (see [[History Page]]) for each curve on the plot, one selects a history event (from the list
of currently active events, recently active events or the list of all events that ever existed), then from the list of tags
inside the history event one selects the particular variable that will be plotted.

== Old MIDAS history events ==
In the old MIDAS history, all history events are written into one history file (.hst file + optional .def and .idx event definition and time index files
which can be/are regenerated automatically from the .hst file). History events are identified by 16-bit history event IDs, the persistent mapping
from history event names and the 16-bit history event IDs is stored in ODB subtree {{Odbpath|path=/History/Events}}. In addition the list of all known history event tags is
stored in ODB subtree {{Odbpath|path=/History/Tags}}. For per-equipment history, the 16-bit history event ID is the value of ODB key [[/Equipment ODB tree#Event ID|/Equipment/<equipment-name>/Common/Event ID]].

== SQL history events ==
In the SQL history (MySQL, SQLITE, etc), each history event is an SQL table. The history event tags are the SQL table columns.

== FILE history events ==
In the new FILE history, each history event is written into a separate file, tag definition are recorded in text format in the file header, history event
data is appended to the file in binary format (fixed record size). If the history event definition is changed, a new file will be started.

= History Event Construction =
The [[mlogger]] creates [[#Frontend history event]]s in open_history() by parsing the ODB subtree {{Odbpath|path=/equipment/<equipment-name>/variables}}. Each ODB entry under the {{Odbpath|path=variables/}} subtree is referred to as a "variable". Each variable can be a single ODB value, an array of ODB values, or a subdirectory (corresponding to TID_STRUCT structured data banks). As each variable is processed, one or more tags are created to describe it.

Single ODB values will generally produce a single tag, while arrays can produce (depending whether the array is "[[#Named Arrays|Named]]" or not)
* multiple tags - one per array element ("Named" array) or
* one single tag - describing the whole array

The code can generate two types of history:
<dl>
<dt> "per-equipment" history</dt>
<dd> will have the tags for all variables concatenated together into one single history event</dd>
<dt> "per-variable" history</dt>
<dd> will have one history event defined for each variable. Inside could be one tag (for single odb values and unnamed array) or multiple tags (for named arrays and structured data
banks).</dd>
</dl>

Per-equipment history is the original MIDAS history implementation. Per-variable history was added to permit efficient data storage in SQL tables. Its initial implementation used 1 ODB hotlink for each variable and it was easy to exceed the maximum permitted number of ODB hotlinks (db_open_record()). To reduce consumption of hotlinks, db_watch() has been implemented and now per-variable history only uses 1 ODB hotlink per equipment.

;NOTE
: With db_watch, per-equipment history is no longer available. '''Per-variable history is the new default''' (and the only option).

[https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087]

= Construction of history event tags =
Currently,
* single ODB values are treated as arrays of length "1"
* string ODB entries are not permitted

History event tags are constructed in the [[mlogger]] add_equipment(). Presently there are some quirks with the generation of history event tags (see [https://midas.triumf.ca/elog/Midas/1088]),
e.g. single odb values are treated as arrays of length "1" and "string" ODB entries are not permitted.

For variables of type TID_KEY (subdirectories, corresponding to TID_STRUCT structured banks), one tag is generated for each subdirectory entry. Tag names for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa/bbb}} will be "aaa_bbb" (with an underscore). Subdirectory entries of type TID_KEY and TID_LINK are currently not supported.

For "[[#Named Arrays|Named Arrays]]", individual tags are generated for each array element. Tag names are taken from the ''Names'' array. For empty tag names (empty names array), tags are "aaa_0", "aaa_1", etc (for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}}).

For "[[#Named Arrays|Single Names Arrays]]", tag names have the variable name appended (with a space), for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}} and an empty names array, tags will be "aaa_0 aaa", "aaa_1 aaa", etc. For a populated names array, the tags will be "name0 aaa", "name1 aaa", etc.

For unnamed arrays and single odb variables (in ODB, single odb variables are arrays of length 1), a single tag is generated.

== Named Arrays ==

'''Named Arrays''' have a "Names" entry in {{Odbpath|path=/equipment/<equipment-name>/settings}}.

For example, to add names to {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}}, create a string array {{Odbpath|path=/equipment/<equipment-name>/settings/names aaa}}. The names array should be at least as long as the corresponding data array. Individual entries in the names array can be left blank (tag names will be "aaa_0", "aaa_1", etc). Duplicate tag names are not permitted.

A single "Names" entry can be created to name all arrays in variables with the same names ('''"Single Names"''').

Create {{Odbpath|path=/equipment/<equipment-name>/settings/names}} and arrays {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}} and {{Odbpath|path=/equipment/<equipment-name>/variables/bbb}} will have
history tags "name0 aaa", "name1 aaa", "name0 bbb", "name1 bbb", etc. If "names" are left blank, tag names will be "aaa_0 aaa", "aaa_1 aaa", "bbb_0 bbb", "bbb_1 bbb", etc.

In the [[mhttpd]] web server [[History Page]] variables viewer, "single name" arrays are displayed in a 2D table.

This information is from [https://midas.triumf.ca/elog/Midas/1088].

= Historical note on MIDAS-History Tag names =
Tags are only used for MIDAS-history events.

The mlogger/mhttpd implementation of /History/Tags has proved troublesome and we are moving away from it.

;During the transition period:
: Implementations starting with Version 4435 of [[mlogger]] and [[mhttpd]] (i.e. "new" versions) will now work '''without''' /History/Tags. The history tags are read directly from the history files themselves. To run in this mode, set ODB key [[/History ODB tree#DisableTags|/History/DisableTags]] to "y". Existing [[/History ODB tree#Tags subtree|/History/Tags subtree]] will be automatically deleted.

Two downsides to running like this are
* it is slower and tags become non-persistent
* if some frontends have not been running for a while, their variables may vanish from the history panel editor.

== New-style History Tags ==
For the above two reasons, using /History/Tags is still recommended.
However, the format of the tags is now changed to simplify management and reduce ODB size.

; For NEW implementations
: the key [[/History ODB tree#CreateOldTags|CreateOldTags]] will not be present. "New-style" history tags will be created automatically in the [[/History ODB tree#Tags subtree|Tags subtree]].

;If you have old-style tags
: [[mlogger]] will automatically convert the old tags to this new format (this is why you should make a backup of your ODB).

** using "old" (i.e. pre Version 4435) mlogger with "new" mhttpd is OK: new mhttpd understands both formats of /History/Tags.
** using "old" mhttpd with "new" mlogger is OK provided the ODB key [[/History ODB tree#CreateOldTags|/History/CreateOldTags]] is set to "y" before starting mlogger.

== Old-style History Tags ==
Prior to versions 4435 of [[mlogger]] and [[mhttpd]], old-style history tags were generated.
See
* http://ladd00.triumf.ca/~daqweb/doc/midas-old/html/F_History_logging.html
for details.

 
---------------
 

= MIDAS ODBC SQL History system =

This section describes the internal workings of the MIDAS SQL history system.

The SQL history system implements the MIDAS history using a SQL database for data storage instead of flat binary files.

The SQL history code lives in a separate files history.h and history_sql.cxx. The present implementation uses the ODBC API for accessing SQL databases.
MySQL is fully supported, Pgsql support is partially implemented, support for arbitrary ODBC drivers should be easy to add in history_sql and most SQL operations are done using an interface class.

Preliminary testing with MySQL and PgSQL indicates that disk space requirements are the same for both storage mechanisms using flat binary files or either of the 2 SQL databases. All 3 tested storage mechanisms store the data uncompressed. (Observed gzip-1 compression ratios from CERN-ALPHA history files are 50% or better).

Preliminary testing also indicates that MySQL database is "not too slow" for use by T2K/ND280 slow control system. Using MySQL, performance seems to improve somewhat compared to traditional MIDAS-history because each SQL table is stored into a separate file, compared to everything-in-one-file in traditional MIDAS-history.

== Set up a MySQL database ==

In order to setup a MySQL database, the following is needed:

* the server hostname and port number
* database name
* user name and password.

It is recommended to create 3 different users:

* a "root" user with full priveleges,
* a "reader" user with read-only priveleges
* and a "writer" user with "insert, create table, and add column" priveleges (no permission to drop table, remove columns, etc)

=== Example ===
#Setup a MySQL database (RHEL/SL 5)

ssh root@localhost
service mysqld start
/usr/bin/mysql_secure_installation (enter current password for root: press <enter>, then accept default answer "Y" to all questions)
mysqladmin -p create history
using mysql-administrator, create 2 users:
history_writer, with database "history" schema privileges: select, insert, create, alter, index
history_reader, with database "history" schema privileges: select

#Setup $HOME/.odbc.ini file:

cat $HOME/.odbc.ini
[history_writer]
Description = history_writer
Driver = /usr/lib64/libmyodbc3.so
SERVER = ladd05.triumf.ca
PORT =
Database = history
OPTION = 3
SOCKET =
User = history_writer
Password = ******
[history_reader]
Description = history_reader
Driver = /usr/lib64/libmyodbc3.so
SERVER = ladd05.triumf.ca
PORT =
Database = history
OPTION = 3
SOCKET =
User = history_reader
Password =

Note: if you are using mhist you will need a separate .odbc.ini file for each user that you have connecting to the database (through MIDAS).

== Layout of SQL tables ==

* One table is created for each history event. Table name is the same as the history event name (as reported by [[mhdump]]).
* one column is created for each history tag (history variable). Column name is the same as the tag name (as reported by mhdump). For array tags, one column is created for each array element: array a[3] will produce columns a_0, a_1 and a_2.
* two special columns are always created: _i_time and _t_time are the timestamps in the UNIX-time integer format and in the SQL "time" format. (A priori, it is not clear which timestamp format is more useful for end users. mhttpd uses the _i_time integer time format). Note that this timestamp is the time when mlogger receives the hot-link signal (see above) and it can be a few seconds behind the time when the user placed the data into the MIDAS bank. For high precision (and sub-second time resolution) time stamps, users should generate their own timestamps and include them as part of the data itself.
* a special table called _history_index is created to remember the mapping between SQL names and MIDAS names for event names, variable name and tag names.

=== Schema Evolution ===

* when new data fields are added, new SQL columns are created automatically
* existing SQL columns are never deleted
* arrays are expanded automatically, and arrays never shrink.
* if the type of a data field changes, a conflict may occur with existing SQL columns, for example if previous type was "FLOAT" and new type is "DOUBLE", the existing SQL columns would have the SQL data type "float", too narrow to store new values. mlogger will complain about this. Such conflicts may be resolved by changing the type of the SQL column using SQL tools. In general, SQL names are created from MIDAS names by replacing all non-alphanumerical characters to an underscore "_". If this procedure creates duplicate column names, the column names and made unique by appending them with a random number, i.e. MIDAS names "A+B" and "A-B" will be translated to "A_B" and "A_B_12345" (random integer). True MIDAS names are saved into the _history_index.

[[Category:history]]

History System

2020-07-30T16:47:53Z

Olchansk: /* Write MYSQL-history events */

{{Pagelinks}}

== Links ==
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[/History ODB tree]]
* [[History Page]]
* [[Mhformat|MIDAS history file format]]
* [[Keys in the ODB /Logger tree|/Logger ODB tree]]
* [[Keys in the ODB /Logger/History subtree|/Logger/History subtree]]
* [[Image History]]
</div>

<div id="history file"></div>
= Introduction =

The history system is an add-on capability built into the MIDAS data logger [[mlogger]] to record useful information in parallel to the data logging.
This information is recorded with a time stamp and saved into a '''history file''' (see [[#History drivers|History drivers]]) for later retrieval. One set of files is created per day containing all the requested history events. The history data may be displayed in graphical form using the [[mhttpd]] MIDAS webserver utility, giving the user an easy way of seeing how experimental variables have changed with time.
The history logging will be in action only if the MIDAS data logger is running, but it is not necessary to have any [[Keys in the ODB /Logger tree#/Logger/Channels subtree|data logging channel]] enabled.

Since June 2020, it is possible to also use the midas history system to record images (png/jpeg etc) periodically, for example to later produce a timelapse from a webcam - see the [[Image History]] page for more details.

= MidasHistoryInterface =

The new history configuration defines history channels similar to logger channels. Each history channel creates a history interface class (class MidasHistoryInterface). The [[mlogger|data logger]] can use this interface to write data into the history, mhttpd, mhist & co can use this interface to read data from the history. The interface is defined in history.h

* http://ladd00.triumf.ca/~daqweb/doc/midas/include/history.h

= History drivers =

Presently, the MIDAS distribution contains these drivers:

; MIDAS-History
: this is the traditional history driver, with data stored in the .hst binary files. (format is documented here [[mhformat]], see also [[mhist]] and [[mhdump]])
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#MIDAS subtree |"MIDAS"]]''' is set up to write MIDAS-history.

; ODBC-History
: stores history data in SQL database using the ODBC abstraction layer. See [[#MIDAS ODBC SQL History system|ODBC SQL History system]].

: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#ODBC subtree|"ODBC"]]''' is set up to write ODBC-history.

; SQLITE-History
: stores data in SQL database as SQLITE3 .sqlite3 files. The SQL table format is similar to ODBC-history, performance is similar, no database server required.
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#SQLITE subtree|"SQLITE"]]''' is set up to write SQLITE-history.

; MYSQL-History
:
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#MYSQL subtree|"MYSQL"]]''' is set up to write MYSQL-history.

; FILE-History
: stores each variable in its own file, making it much easier to trace the history of a particular variable.
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#FILE subtree|FILE]]''' is set up to write FILE-history.

More drivers can be easily added by the user (create history_xxx.cxx, add constructor to hs_get_history(), create the ODB subtree /Logger/History/xxx/channel).

= Location of History Files =
;MIDAS-history
:By default, the traditional MIDAS-history is enabled, writing history files into the MIDAS data directory path given by the ODB key [[Keys in the ODB /Logger tree#Data dir|Data dir]].
:If history data is important, it is recommended to write it to a directory that is backed up or archived. (The MIDAS data directory is usually located on a scratch data disk without backups).

:The location of the MIDAS-history data files can be changed by use of the ODB Key [[Keys in the ODB /Logger tree#History dir|History dir]].

;ODBC-History
:The location of ODBC-History files is ...

;SQLITE-History
:The location of SQLITE-History files is the directory path given by the ODB key [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]].

;MYSQL-History
:The location of MYSQL-History files is ...

;FILE-History
:The location of FILE-History files is the directory path given by the ODB key [[Keys in the ODB /Logger/History subtree#History dir|History dir]].

= Types of History Events =

There are two basic types of history events, which are defined in different ways:

* "[[#Frontend History Event|Frontend]]" (or "Equipment") History event composed in the frontend. See [[#Frontend History Event|Frontend History Event]] for details.
* "Virtual" (or "Links") History event composed within the ODB under the [[/History ODB tree#Links subtree|/History/Links subtree]]. See [[#Virtual History Event|Virtual History Event]] for details.

Both these history event definitions take effect when the data logger gets a start run [[Run States and Transitions|transition]]. Any modifications made during the run are not applied until the start of the next run.

== Virtual History Event ==
The history of any ODB variable can be recorded in a "virtual" history event by creating a link to that variable in the [[/History ODB tree#Links subtree|/History/Links subtree]]. This is used for ODB variables that are not in an Equipment.

; To enable Virtual History Events
: Create links as desired in [[/History ODB tree#Links subtree|/History/Links subtree]].

History events are created for each entry under ODB subdirectory {{Odbpath|path=/History/Links}}.

Two types of links are permitted:
<dl>
<dt> ODB key {{Odbpath|path=/history/links/aaa}} is a link to a subdirectory</dt>
<dd><ol>
<li> db_watch() is setup to watch this subdirectory</li>
<li> tags are created for each subdirectory entry (1 tag per entry)</li>
<li> There is no possibility for naming array elements, so 1 tag per array, regardless of the number of elements.</li> </ol></dd>
<dt>ODB key {{Odbpath|path=/history/links/bbb}} is a subdirectory with links to ODB values</dt>
<dd> <ol>
<li>db_watch is setup to watch each link target</li>
<li> tags are created for each link (1 tag per link)</li>
<li> tag name is the link name (NOT the target name). There is no possibility for naming array elements</li>
</ol></dd>
</dl>

Mixing links and subdirectories is not permitted.

The update period of history events created for {{Odbpath|path=/history/links}} is controlled by entries in {{Odbpath|path=/history/links periods}}. Numeric values of periods are same as for a [[#Frontend History Event|Frontend History event]]. Numeric value 0 disables the history for a particular event.

See [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087] for more information.

== Frontend History Event ==

Each [[Frontend Operation#Equipments and Events|Equipment]] has the capability to generate "history data".

; To enable Frontend History Events

: Set the [[/Equipment ODB tree#Log history|Log history]] ODB key of each equipment to be logged '''non-zero'''. (See [[Equipment List Parameters#Log History|Log History parameter]]). (You may need to restart the frontend.)

This causes the event contents to be copied to the ODB [[/Equipment ODB tree#Variables subtree|/Equipment/''<equipment-name>''/Variables subtree]] for the use of the History system. history events are created by parsing the content of {{Odbpath|path=/Equipment/<equipment-name>/variables}}.

For each equipment, history is controlled by the value of {{Odbpath|path=/Equipment/<equipment-name>/common/period}} as follows:
* 0 = history disabled
* 1 = history is enabled
* >1 = history is enabled, throttled down

The throttling is implemented in log_history()/watch_history() by this algorithm:
the very first history event is recorded, then all changed to the data are ignored until "period" seconds has elapsed. Then the next history event will be recorded, and following
changes will be ignored until "period" second elapses, and so forth. Period value "1" has special meaning - there is no throttling, all history events are logged.

This information is from [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087].

=== Sequence for frontend History Event ===
The main steps for the frontend History Event is as follows:

* The user code in the frontend equipment Event Readout routines reads the data, placing it into a MIDAS data bank
* In mfe.c, if the key [[/Equipment ODB tree#Log history|Log history]] > 0 (i.e. the history system is enabled for this equipment), this data bank is written into ODB (by update_odb()).
* This ODB write triggers an ODB hot-link into mlogger
* The hot-link calls mlogger.c::log_history(), which calls hs_write() to write the data into the history file. The frequency of the history writes is specified by the number of seconds between writes stored in the ODB key [[/Equipment ODB tree#Log history|Log history]].
* history.c::hs_write() or history_odbc::hs_write_odbc() writes the data into a history file or into an SQL database.

= How to Write History events =
* Enable desired [[#To enable Frontend History Events|Frontend]] and/or [[#To enable Virtual History Events|Virtual]] history events
* [[#Select the history driver|Select the history driver]]
* Follow instructions below for selected driver:
** [[#Write MIDAS-History events|Write MIDAS-history events]]
** [[#Write ODBC-History events|Write ODBC-history events]]
** [[#Write SQLITE-History events|Write SQLITE-history events]]
** [[#Write MYSQL-History events|Write MYSQL-history events]]
** [[#Write FILE-History events|Write FILE-history events]]

== Select the history driver ==

* Select the '''[[#History drivers|history driver]]''' and therefore the [[Keys in the ODB /Logger/History subtree#<history-logging-channel> subtrees|history-logging-channel]] to use to write the MIDAS-history files.
* set [[Keys in the ODB /Logger/History subtree#Active|Active]] ODB key in the selected [[Keys in the ODB /Logger/History subtree#<history-logging-channel> subtrees|history-logging-channel]] to "y". All other logging channels should be disabled.
* set [[/History ODB tree#LoggerHistoryChannel|LoggerHistoryChannel]] ODB key to the selected [[#History drivers|History channel]].

== Write MIDAS-History events ==
* Enable desired [[#To enable Frontend History Events|Frontend]] and/or [[#To enable Virtual History Events|Virtual]] history events
* Perform steps in [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "MIDAS".
* Restart [[mhttpd]] and [[mlogger]], start the run.

MIDAS-history files will be saved into the [[#Location of History Files|MIDAS-history data directory]].

== Write ODBC-History events ==

See [[#MIDAS ODBC SQL History system|MIDAS ODBC SQL History system]].
* If necessary, [[#Set up a MySQL database|set up a MySQL database]].
* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps in [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "ODBC".
* Check ODB Key [[Keys in the ODB /Logger/History subtree#Writer_ODBC_DSN|Writer_ODBC_DSN]] is set to "history writer" and Key [[Keys in the ODB /Logger/History subtree#Reader_ODBC_DSN|Reader_ODBC_DSN]] is set to "history reader" (the default values)
* set [[Keys in the ODB /Logger/History subtree#Debug|Debug]] to 1 if desired
* restart [[mlogger]] in verbose mode (i.e. mlogger -v), and observe how it issues SQL commands to create the tables and columns corresponding to MIDAS-history events and tags.
* restart [[mhttpd]]

'''Notes'''
# [[mlogger]] is programmed to raise alarms if connection to SQL database is interrupted or if some events cannot be written into the database (i.e. data type mismatch, SQL syntax errors, etc). The traditional MIDAS-history never raised alarms because it "never failed" - other than from "disk full" errors, which are immediately obvious.
# With ODBC History enabled, [[mhttpd]] will only use history information from the SQL database to make history plots and to extract history variable names for the history plot editor. The ODB {{Odbpath|path=/History}} subtrees[[/History ODB tree#Tags subtree|Tags]] and [[/History ODB tree#Events subtree|Events]] are not used.
# variable names presented to the user may change from MIDAS names to SQL names (use the history panel "Label" text fields to create permanent plot labels).

== Write SQLITE-history events ==

No special configuration required.

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "SQLITE".

* Create a directory for storage of sqlite history files
* save its path in the ODB key [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]]
* restart [[mhttpd]] and [[mlogger]]

In the history data directory (in [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]]) you should see files with name "mh_xxx.sqlite3" corresponding to your equipment and variable names.

== Write MYSQL-history events ==

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "MYSQL".
* setup the MySQL database:
* install mariadb-10.4
<pre>
macos:
sudo port install mariadb-10.4
ln -s ln -fs /opt/local/lib/mariadb-10.4/bin/mysql_config ~/bin/
mysql_config --version
10.4.13
</pre>
* rebuild midas, check the cmake finds the correct mysql: MIDAS: Found MySQL version 10.4.13
* create $HOME/.my.cnf
<pre>
[mysqld]
#port=3306
socket=/Users/olchansk/mysql/mysql.sock
datadir=/Users/olchansk/mysql/db
pid-file=/Users/olchansk/mysql/mysqld.pid

[client]
#port=3306
socket=/Users/olchansk/mysql/mysql.sock
</pre>
* mkdir $HOME/mysql
* start mysql:
<pre>
macos: /opt/local/lib/mariadb-10.4/bin/mysqld
</pre>
* configure the database: start the "mysql" utility (/opt/local/lib/mariadb-10.4/bin/mysql)
<pre>
NOTE: instructions thanks to Ben S.
NOTE: use some other passwords instead of the example "reader_password" and "writer_password"

MariaDB [(none)]> CREATE DATABASE IF NOT EXISTS history;
Query OK, 1 row affected (0.003 sec)

MariaDB [(none)]> CREATE USER history_reader@'localhost' IDENTIFIED BY 'reader_password';
Query OK, 0 rows affected (0.027 sec)

MariaDB [(none)]> GRANT SELECT ON history.* TO history_reader@'localhost';
Query OK, 0 rows affected (0.016 sec)

MariaDB [(none)]> CREATE USER 'history_writer'@'localhost' IDENTIFIED BY 'writer_password';
Query OK, 0 rows affected (0.014 sec)

MariaDB [(none)]> GRANT SELECT,INSERT,CREATE,ALTER,INDEX ON history.* TO 'history_writer'@'localhost';
Query OK, 0 rows affected (0.017 sec)

MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.001 sec)
</pre>
* try connecting to both users:
<pre>
/opt/local/lib/mariadb-10.4/bin/mysql -u history_reader -p ### should ask for reader password
/opt/local/lib/mariadb-10.4/bin/mysql -u history_writer -p ### should ask for writer password
</pre>
* create midas config files:
** mysql_writer.txt
<pre>
#server=localhost
#port=xxx
database=history
socket=/Users/olchansk/mysql/mysql.sock
user=history_writer
password=writer_password ### change this!!!
#buffer=1000
</pre>
** mysql_reader.txt
<pre>
#server=localhost
#port=xxx
database=history
socket=/Users/olchansk/mysql/mysql.sock
user=history_reader
password=reader_password ### change this!!!
</pre>
* enable MYSQL history in mlogger:
** change ODB /Logger/History/MYSQL/Active to "y", "mysql writer" and "mysql reader" to location of these files (leave the default values if these files are in the experiment directory).
* enable MYSQL history in mhttpd:
** change ODB /History/LoggerHistoryChannel to "MYSQL"
* restart mlogger, restart mhttpd
...........

== Write FILE-history events ==

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "SQLITE".
* Assign directory path for saved files using the ODB key [[Keys in the ODB /Logger/History subtree#History dir|History dir]].
...........

 
---------------
 

= History Events =
The following information is derived from [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087].

A "history event" is a history atomic unit of data. Associated with each history event is a timestamp (unix time),
a name (limited to NAME_LENGTH in the old history) and a list of history tags that describe the individual data
values inside the history event.

When making history plots in [[mhttpd]] (see [[History Page]]) for each curve on the plot, one selects a history event (from the list
of currently active events, recently active events or the list of all events that ever existed), then from the list of tags
inside the history event one selects the particular variable that will be plotted.

== Old MIDAS history events ==
In the old MIDAS history, all history events are written into one history file (.hst file + optional .def and .idx event definition and time index files
which can be/are regenerated automatically from the .hst file). History events are identified by 16-bit history event IDs, the persistent mapping
from history event names and the 16-bit history event IDs is stored in ODB subtree {{Odbpath|path=/History/Events}}. In addition the list of all known history event tags is
stored in ODB subtree {{Odbpath|path=/History/Tags}}. For per-equipment history, the 16-bit history event ID is the value of ODB key [[/Equipment ODB tree#Event ID|/Equipment/<equipment-name>/Common/Event ID]].

== SQL history events ==
In the SQL history (MySQL, SQLITE, etc), each history event is an SQL table. The history event tags are the SQL table columns.

== FILE history events ==
In the new FILE history, each history event is written into a separate file, tag definition are recorded in text format in the file header, history event
data is appended to the file in binary format (fixed record size). If the history event definition is changed, a new file will be started.

= History Event Construction =
The [[mlogger]] creates [[#Frontend history event]]s in open_history() by parsing the ODB subtree {{Odbpath|path=/equipment/<equipment-name>/variables}}. Each ODB entry under the {{Odbpath|path=variables/}} subtree is referred to as a "variable". Each variable can be a single ODB value, an array of ODB values, or a subdirectory (corresponding to TID_STRUCT structured data banks). As each variable is processed, one or more tags are created to describe it.

Single ODB values will generally produce a single tag, while arrays can produce (depending whether the array is "[[#Named Arrays|Named]]" or not)
* multiple tags - one per array element ("Named" array) or
* one single tag - describing the whole array

The code can generate two types of history:
<dl>
<dt> "per-equipment" history</dt>
<dd> will have the tags for all variables concatenated together into one single history event</dd>
<dt> "per-variable" history</dt>
<dd> will have one history event defined for each variable. Inside could be one tag (for single odb values and unnamed array) or multiple tags (for named arrays and structured data
banks).</dd>
</dl>

Per-equipment history is the original MIDAS history implementation. Per-variable history was added to permit efficient data storage in SQL tables. Its initial implementation used 1 ODB hotlink for each variable and it was easy to exceed the maximum permitted number of ODB hotlinks (db_open_record()). To reduce consumption of hotlinks, db_watch() has been implemented and now per-variable history only uses 1 ODB hotlink per equipment.

;NOTE
: With db_watch, per-equipment history is no longer available. '''Per-variable history is the new default''' (and the only option).

[https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087]

= Construction of history event tags =
Currently,
* single ODB values are treated as arrays of length "1"
* string ODB entries are not permitted

History event tags are constructed in the [[mlogger]] add_equipment(). Presently there are some quirks with the generation of history event tags (see [https://midas.triumf.ca/elog/Midas/1088]),
e.g. single odb values are treated as arrays of length "1" and "string" ODB entries are not permitted.

For variables of type TID_KEY (subdirectories, corresponding to TID_STRUCT structured banks), one tag is generated for each subdirectory entry. Tag names for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa/bbb}} will be "aaa_bbb" (with an underscore). Subdirectory entries of type TID_KEY and TID_LINK are currently not supported.

For "[[#Named Arrays|Named Arrays]]", individual tags are generated for each array element. Tag names are taken from the ''Names'' array. For empty tag names (empty names array), tags are "aaa_0", "aaa_1", etc (for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}}).

For "[[#Named Arrays|Single Names Arrays]]", tag names have the variable name appended (with a space), for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}} and an empty names array, tags will be "aaa_0 aaa", "aaa_1 aaa", etc. For a populated names array, the tags will be "name0 aaa", "name1 aaa", etc.

For unnamed arrays and single odb variables (in ODB, single odb variables are arrays of length 1), a single tag is generated.

== Named Arrays ==

'''Named Arrays''' have a "Names" entry in {{Odbpath|path=/equipment/<equipment-name>/settings}}.

For example, to add names to {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}}, create a string array {{Odbpath|path=/equipment/<equipment-name>/settings/names aaa}}. The names array should be at least as long as the corresponding data array. Individual entries in the names array can be left blank (tag names will be "aaa_0", "aaa_1", etc). Duplicate tag names are not permitted.

A single "Names" entry can be created to name all arrays in variables with the same names ('''"Single Names"''').

Create {{Odbpath|path=/equipment/<equipment-name>/settings/names}} and arrays {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}} and {{Odbpath|path=/equipment/<equipment-name>/variables/bbb}} will have
history tags "name0 aaa", "name1 aaa", "name0 bbb", "name1 bbb", etc. If "names" are left blank, tag names will be "aaa_0 aaa", "aaa_1 aaa", "bbb_0 bbb", "bbb_1 bbb", etc.

In the [[mhttpd]] web server [[History Page]] variables viewer, "single name" arrays are displayed in a 2D table.

This information is from [https://midas.triumf.ca/elog/Midas/1088].

= Historical note on MIDAS-History Tag names =
Tags are only used for MIDAS-history events.

The mlogger/mhttpd implementation of /History/Tags has proved troublesome and we are moving away from it.

;During the transition period:
: Implementations starting with Version 4435 of [[mlogger]] and [[mhttpd]] (i.e. "new" versions) will now work '''without''' /History/Tags. The history tags are read directly from the history files themselves. To run in this mode, set ODB key [[/History ODB tree#DisableTags|/History/DisableTags]] to "y". Existing [[/History ODB tree#Tags subtree|/History/Tags subtree]] will be automatically deleted.

Two downsides to running like this are
* it is slower and tags become non-persistent
* if some frontends have not been running for a while, their variables may vanish from the history panel editor.

== New-style History Tags ==
For the above two reasons, using /History/Tags is still recommended.
However, the format of the tags is now changed to simplify management and reduce ODB size.

; For NEW implementations
: the key [[/History ODB tree#CreateOldTags|CreateOldTags]] will not be present. "New-style" history tags will be created automatically in the [[/History ODB tree#Tags subtree|Tags subtree]].

;If you have old-style tags
: [[mlogger]] will automatically convert the old tags to this new format (this is why you should make a backup of your ODB).

** using "old" (i.e. pre Version 4435) mlogger with "new" mhttpd is OK: new mhttpd understands both formats of /History/Tags.
** using "old" mhttpd with "new" mlogger is OK provided the ODB key [[/History ODB tree#CreateOldTags|/History/CreateOldTags]] is set to "y" before starting mlogger.

== Old-style History Tags ==
Prior to versions 4435 of [[mlogger]] and [[mhttpd]], old-style history tags were generated.
See
* http://ladd00.triumf.ca/~daqweb/doc/midas-old/html/F_History_logging.html
for details.

 
---------------
 

= MIDAS ODBC SQL History system =

This section describes the internal workings of the MIDAS SQL history system.

The SQL history system implements the MIDAS history using a SQL database for data storage instead of flat binary files.

The SQL history code lives in a separate files history.h and history_sql.cxx. The present implementation uses the ODBC API for accessing SQL databases.
MySQL is fully supported, Pgsql support is partially implemented, support for arbitrary ODBC drivers should be easy to add in history_sql and most SQL operations are done using an interface class.

Preliminary testing with MySQL and PgSQL indicates that disk space requirements are the same for both storage mechanisms using flat binary files or either of the 2 SQL databases. All 3 tested storage mechanisms store the data uncompressed. (Observed gzip-1 compression ratios from CERN-ALPHA history files are 50% or better).

Preliminary testing also indicates that MySQL database is "not too slow" for use by T2K/ND280 slow control system. Using MySQL, performance seems to improve somewhat compared to traditional MIDAS-history because each SQL table is stored into a separate file, compared to everything-in-one-file in traditional MIDAS-history.

== Set up a MySQL database ==

In order to setup a MySQL database, the following is needed:

* the server hostname and port number
* database name
* user name and password.

It is recommended to create 3 different users:

* a "root" user with full priveleges,
* a "reader" user with read-only priveleges
* and a "writer" user with "insert, create table, and add column" priveleges (no permission to drop table, remove columns, etc)

=== Example ===
#Setup a MySQL database (RHEL/SL 5)

ssh root@localhost
service mysqld start
/usr/bin/mysql_secure_installation (enter current password for root: press <enter>, then accept default answer "Y" to all questions)
mysqladmin -p create history
using mysql-administrator, create 2 users:
history_writer, with database "history" schema privileges: select, insert, create, alter, index
history_reader, with database "history" schema privileges: select

#Setup $HOME/.odbc.ini file:

cat $HOME/.odbc.ini
[history_writer]
Description = history_writer
Driver = /usr/lib64/libmyodbc3.so
SERVER = ladd05.triumf.ca
PORT =
Database = history
OPTION = 3
SOCKET =
User = history_writer
Password = ******
[history_reader]
Description = history_reader
Driver = /usr/lib64/libmyodbc3.so
SERVER = ladd05.triumf.ca
PORT =
Database = history
OPTION = 3
SOCKET =
User = history_reader
Password =

Note: if you are using mhist you will need a separate .odbc.ini file for each user that you have connecting to the database (through MIDAS).

== Layout of SQL tables ==

* One table is created for each history event. Table name is the same as the history event name (as reported by [[mhdump]]).
* one column is created for each history tag (history variable). Column name is the same as the tag name (as reported by mhdump). For array tags, one column is created for each array element: array a[3] will produce columns a_0, a_1 and a_2.
* two special columns are always created: _i_time and _t_time are the timestamps in the UNIX-time integer format and in the SQL "time" format. (A priori, it is not clear which timestamp format is more useful for end users. mhttpd uses the _i_time integer time format). Note that this timestamp is the time when mlogger receives the hot-link signal (see above) and it can be a few seconds behind the time when the user placed the data into the MIDAS bank. For high precision (and sub-second time resolution) time stamps, users should generate their own timestamps and include them as part of the data itself.
* a special table called _history_index is created to remember the mapping between SQL names and MIDAS names for event names, variable name and tag names.

=== Schema Evolution ===

* when new data fields are added, new SQL columns are created automatically
* existing SQL columns are never deleted
* arrays are expanded automatically, and arrays never shrink.
* if the type of a data field changes, a conflict may occur with existing SQL columns, for example if previous type was "FLOAT" and new type is "DOUBLE", the existing SQL columns would have the SQL data type "float", too narrow to store new values. mlogger will complain about this. Such conflicts may be resolved by changing the type of the SQL column using SQL tools. In general, SQL names are created from MIDAS names by replacing all non-alphanumerical characters to an underscore "_". If this procedure creates duplicate column names, the column names and made unique by appending them with a random number, i.e. MIDAS names "A+B" and "A-B" will be translated to "A_B" and "A_B_12345" (random integer). True MIDAS names are saved into the _history_index.

[[Category:history]]

History System

2020-07-30T16:46:12Z

Olchansk: /* Write MYSQL-history events */

{{Pagelinks}}

== Links ==
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[/History ODB tree]]
* [[History Page]]
* [[Mhformat|MIDAS history file format]]
* [[Keys in the ODB /Logger tree|/Logger ODB tree]]
* [[Keys in the ODB /Logger/History subtree|/Logger/History subtree]]
* [[Image History]]
</div>

<div id="history file"></div>
= Introduction =

The history system is an add-on capability built into the MIDAS data logger [[mlogger]] to record useful information in parallel to the data logging.
This information is recorded with a time stamp and saved into a '''history file''' (see [[#History drivers|History drivers]]) for later retrieval. One set of files is created per day containing all the requested history events. The history data may be displayed in graphical form using the [[mhttpd]] MIDAS webserver utility, giving the user an easy way of seeing how experimental variables have changed with time.
The history logging will be in action only if the MIDAS data logger is running, but it is not necessary to have any [[Keys in the ODB /Logger tree#/Logger/Channels subtree|data logging channel]] enabled.

Since June 2020, it is possible to also use the midas history system to record images (png/jpeg etc) periodically, for example to later produce a timelapse from a webcam - see the [[Image History]] page for more details.

= MidasHistoryInterface =

The new history configuration defines history channels similar to logger channels. Each history channel creates a history interface class (class MidasHistoryInterface). The [[mlogger|data logger]] can use this interface to write data into the history, mhttpd, mhist & co can use this interface to read data from the history. The interface is defined in history.h

* http://ladd00.triumf.ca/~daqweb/doc/midas/include/history.h

= History drivers =

Presently, the MIDAS distribution contains these drivers:

; MIDAS-History
: this is the traditional history driver, with data stored in the .hst binary files. (format is documented here [[mhformat]], see also [[mhist]] and [[mhdump]])
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#MIDAS subtree |"MIDAS"]]''' is set up to write MIDAS-history.

; ODBC-History
: stores history data in SQL database using the ODBC abstraction layer. See [[#MIDAS ODBC SQL History system|ODBC SQL History system]].

: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#ODBC subtree|"ODBC"]]''' is set up to write ODBC-history.

; SQLITE-History
: stores data in SQL database as SQLITE3 .sqlite3 files. The SQL table format is similar to ODBC-history, performance is similar, no database server required.
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#SQLITE subtree|"SQLITE"]]''' is set up to write SQLITE-history.

; MYSQL-History
:
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#MYSQL subtree|"MYSQL"]]''' is set up to write MYSQL-history.

; FILE-History
: stores each variable in its own file, making it much easier to trace the history of a particular variable.
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#FILE subtree|FILE]]''' is set up to write FILE-history.

More drivers can be easily added by the user (create history_xxx.cxx, add constructor to hs_get_history(), create the ODB subtree /Logger/History/xxx/channel).

= Location of History Files =
;MIDAS-history
:By default, the traditional MIDAS-history is enabled, writing history files into the MIDAS data directory path given by the ODB key [[Keys in the ODB /Logger tree#Data dir|Data dir]].
:If history data is important, it is recommended to write it to a directory that is backed up or archived. (The MIDAS data directory is usually located on a scratch data disk without backups).

:The location of the MIDAS-history data files can be changed by use of the ODB Key [[Keys in the ODB /Logger tree#History dir|History dir]].

;ODBC-History
:The location of ODBC-History files is ...

;SQLITE-History
:The location of SQLITE-History files is the directory path given by the ODB key [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]].

;MYSQL-History
:The location of MYSQL-History files is ...

;FILE-History
:The location of FILE-History files is the directory path given by the ODB key [[Keys in the ODB /Logger/History subtree#History dir|History dir]].

= Types of History Events =

There are two basic types of history events, which are defined in different ways:

* "[[#Frontend History Event|Frontend]]" (or "Equipment") History event composed in the frontend. See [[#Frontend History Event|Frontend History Event]] for details.
* "Virtual" (or "Links") History event composed within the ODB under the [[/History ODB tree#Links subtree|/History/Links subtree]]. See [[#Virtual History Event|Virtual History Event]] for details.

Both these history event definitions take effect when the data logger gets a start run [[Run States and Transitions|transition]]. Any modifications made during the run are not applied until the start of the next run.

== Virtual History Event ==
The history of any ODB variable can be recorded in a "virtual" history event by creating a link to that variable in the [[/History ODB tree#Links subtree|/History/Links subtree]]. This is used for ODB variables that are not in an Equipment.

; To enable Virtual History Events
: Create links as desired in [[/History ODB tree#Links subtree|/History/Links subtree]].

History events are created for each entry under ODB subdirectory {{Odbpath|path=/History/Links}}.

Two types of links are permitted:
<dl>
<dt> ODB key {{Odbpath|path=/history/links/aaa}} is a link to a subdirectory</dt>
<dd><ol>
<li> db_watch() is setup to watch this subdirectory</li>
<li> tags are created for each subdirectory entry (1 tag per entry)</li>
<li> There is no possibility for naming array elements, so 1 tag per array, regardless of the number of elements.</li> </ol></dd>
<dt>ODB key {{Odbpath|path=/history/links/bbb}} is a subdirectory with links to ODB values</dt>
<dd> <ol>
<li>db_watch is setup to watch each link target</li>
<li> tags are created for each link (1 tag per link)</li>
<li> tag name is the link name (NOT the target name). There is no possibility for naming array elements</li>
</ol></dd>
</dl>

Mixing links and subdirectories is not permitted.

The update period of history events created for {{Odbpath|path=/history/links}} is controlled by entries in {{Odbpath|path=/history/links periods}}. Numeric values of periods are same as for a [[#Frontend History Event|Frontend History event]]. Numeric value 0 disables the history for a particular event.

See [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087] for more information.

== Frontend History Event ==

Each [[Frontend Operation#Equipments and Events|Equipment]] has the capability to generate "history data".

; To enable Frontend History Events

: Set the [[/Equipment ODB tree#Log history|Log history]] ODB key of each equipment to be logged '''non-zero'''. (See [[Equipment List Parameters#Log History|Log History parameter]]). (You may need to restart the frontend.)

This causes the event contents to be copied to the ODB [[/Equipment ODB tree#Variables subtree|/Equipment/''<equipment-name>''/Variables subtree]] for the use of the History system. history events are created by parsing the content of {{Odbpath|path=/Equipment/<equipment-name>/variables}}.

For each equipment, history is controlled by the value of {{Odbpath|path=/Equipment/<equipment-name>/common/period}} as follows:
* 0 = history disabled
* 1 = history is enabled
* >1 = history is enabled, throttled down

The throttling is implemented in log_history()/watch_history() by this algorithm:
the very first history event is recorded, then all changed to the data are ignored until "period" seconds has elapsed. Then the next history event will be recorded, and following
changes will be ignored until "period" second elapses, and so forth. Period value "1" has special meaning - there is no throttling, all history events are logged.

This information is from [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087].

=== Sequence for frontend History Event ===
The main steps for the frontend History Event is as follows:

* The user code in the frontend equipment Event Readout routines reads the data, placing it into a MIDAS data bank
* In mfe.c, if the key [[/Equipment ODB tree#Log history|Log history]] > 0 (i.e. the history system is enabled for this equipment), this data bank is written into ODB (by update_odb()).
* This ODB write triggers an ODB hot-link into mlogger
* The hot-link calls mlogger.c::log_history(), which calls hs_write() to write the data into the history file. The frequency of the history writes is specified by the number of seconds between writes stored in the ODB key [[/Equipment ODB tree#Log history|Log history]].
* history.c::hs_write() or history_odbc::hs_write_odbc() writes the data into a history file or into an SQL database.

= How to Write History events =
* Enable desired [[#To enable Frontend History Events|Frontend]] and/or [[#To enable Virtual History Events|Virtual]] history events
* [[#Select the history driver|Select the history driver]]
* Follow instructions below for selected driver:
** [[#Write MIDAS-History events|Write MIDAS-history events]]
** [[#Write ODBC-History events|Write ODBC-history events]]
** [[#Write SQLITE-History events|Write SQLITE-history events]]
** [[#Write MYSQL-History events|Write MYSQL-history events]]
** [[#Write FILE-History events|Write FILE-history events]]

== Select the history driver ==

* Select the '''[[#History drivers|history driver]]''' and therefore the [[Keys in the ODB /Logger/History subtree#<history-logging-channel> subtrees|history-logging-channel]] to use to write the MIDAS-history files.
* set [[Keys in the ODB /Logger/History subtree#Active|Active]] ODB key in the selected [[Keys in the ODB /Logger/History subtree#<history-logging-channel> subtrees|history-logging-channel]] to "y". All other logging channels should be disabled.
* set [[/History ODB tree#LoggerHistoryChannel|LoggerHistoryChannel]] ODB key to the selected [[#History drivers|History channel]].

== Write MIDAS-History events ==
* Enable desired [[#To enable Frontend History Events|Frontend]] and/or [[#To enable Virtual History Events|Virtual]] history events
* Perform steps in [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "MIDAS".
* Restart [[mhttpd]] and [[mlogger]], start the run.

MIDAS-history files will be saved into the [[#Location of History Files|MIDAS-history data directory]].

== Write ODBC-History events ==

See [[#MIDAS ODBC SQL History system|MIDAS ODBC SQL History system]].
* If necessary, [[#Set up a MySQL database|set up a MySQL database]].
* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps in [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "ODBC".
* Check ODB Key [[Keys in the ODB /Logger/History subtree#Writer_ODBC_DSN|Writer_ODBC_DSN]] is set to "history writer" and Key [[Keys in the ODB /Logger/History subtree#Reader_ODBC_DSN|Reader_ODBC_DSN]] is set to "history reader" (the default values)
* set [[Keys in the ODB /Logger/History subtree#Debug|Debug]] to 1 if desired
* restart [[mlogger]] in verbose mode (i.e. mlogger -v), and observe how it issues SQL commands to create the tables and columns corresponding to MIDAS-history events and tags.
* restart [[mhttpd]]

'''Notes'''
# [[mlogger]] is programmed to raise alarms if connection to SQL database is interrupted or if some events cannot be written into the database (i.e. data type mismatch, SQL syntax errors, etc). The traditional MIDAS-history never raised alarms because it "never failed" - other than from "disk full" errors, which are immediately obvious.
# With ODBC History enabled, [[mhttpd]] will only use history information from the SQL database to make history plots and to extract history variable names for the history plot editor. The ODB {{Odbpath|path=/History}} subtrees[[/History ODB tree#Tags subtree|Tags]] and [[/History ODB tree#Events subtree|Events]] are not used.
# variable names presented to the user may change from MIDAS names to SQL names (use the history panel "Label" text fields to create permanent plot labels).

== Write SQLITE-history events ==

No special configuration required.

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "SQLITE".

* Create a directory for storage of sqlite history files
* save its path in the ODB key [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]]
* restart [[mhttpd]] and [[mlogger]]

In the history data directory (in [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]]) you should see files with name "mh_xxx.sqlite3" corresponding to your equipment and variable names.

== Write MYSQL-history events ==

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "MYSQL".
* setup the MySQL database:
* install mariadb-10.4
<pre>
macos:
sudo port install mariadb-10.4
ln -s ln -fs /opt/local/lib/mariadb-10.4/bin/mysql_config ~/bin/
mysql_config --version
10.4.13
</pre>
* rebuild midas, check the cmake finds the correct mysql: MIDAS: Found MySQL version 10.4.13
* create $HOME/.my.cnf
<pre>
[mysqld]
#port=3306
socket=/Users/olchansk/mysql/mysql.sock
datadir=/Users/olchansk/mysql/db
pid-file=/Users/olchansk/mysql/mysqld.pid

[client]
#port=3306
socket=/Users/olchansk/mysql/mysql.sock
</pre>
* mkdir $HOME/mysql
* start mysql:
<pre>
macos: /opt/local/lib/mariadb-10.4/bin/mysqld
</pre>
* configure the database: start the "mysql" utility (/opt/local/lib/mariadb-10.4/bin/mysql)
<pre>
NOTE: instructions thanks to Ben S.
NOTE: use some other passwords instead of the example "reader_password" and "writer_password"

MariaDB [(none)]> CREATE DATABASE IF NOT EXISTS history;
Query OK, 1 row affected (0.003 sec)

MariaDB [(none)]> CREATE USER history_reader@'localhost' IDENTIFIED BY 'reader_password';
Query OK, 0 rows affected (0.027 sec)

MariaDB [(none)]> GRANT SELECT ON history.* TO history_reader@'localhost';
Query OK, 0 rows affected (0.016 sec)

MariaDB [(none)]> CREATE USER 'history_writer'@'localhost' IDENTIFIED BY 'writer_password';
Query OK, 0 rows affected (0.014 sec)

MariaDB [(none)]> GRANT SELECT,INSERT,CREATE,ALTER,INDEX ON history.* TO 'history_writer'@'localhost';
Query OK, 0 rows affected (0.017 sec)
</pre>
* try connecting to both users:
<pre>
/opt/local/lib/mariadb-10.4/bin/mysql -u history_reader -p ### should ask for reader password
/opt/local/lib/mariadb-10.4/bin/mysql -u history_writer -p ### should ask for writer password
</pre>
* create midas config files:
** mysql_writer.txt
<pre>
#server=localhost
#port=xxx
database=history
socket=/Users/olchansk/mysql/mysql.sock
user=history_writer
password=writer_password ### change this!!!
#buffer=1000
</pre>
** mysql_reader.txt
<pre>
#server=localhost
#port=xxx
database=history
socket=/Users/olchansk/mysql/mysql.sock
user=history_reader
password=reader_password ### change this!!!
</pre>
* enable MYSQL history in mlogger:
** change ODB /Logger/History/MYSQL/Active to "y", "mysql writer" and "mysql reader" to location of these files (leave the default values if these files are in the experiment directory).
* enable MYSQL history in mhttpd:
** change ODB /History/LoggerHistoryChannel to "MYSQL"
* restart mlogger, restart mhttpd
...........

== Write FILE-history events ==

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "SQLITE".
* Assign directory path for saved files using the ODB key [[Keys in the ODB /Logger/History subtree#History dir|History dir]].
...........

 
---------------
 

= History Events =
The following information is derived from [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087].

A "history event" is a history atomic unit of data. Associated with each history event is a timestamp (unix time),
a name (limited to NAME_LENGTH in the old history) and a list of history tags that describe the individual data
values inside the history event.

When making history plots in [[mhttpd]] (see [[History Page]]) for each curve on the plot, one selects a history event (from the list
of currently active events, recently active events or the list of all events that ever existed), then from the list of tags
inside the history event one selects the particular variable that will be plotted.

== Old MIDAS history events ==
In the old MIDAS history, all history events are written into one history file (.hst file + optional .def and .idx event definition and time index files
which can be/are regenerated automatically from the .hst file). History events are identified by 16-bit history event IDs, the persistent mapping
from history event names and the 16-bit history event IDs is stored in ODB subtree {{Odbpath|path=/History/Events}}. In addition the list of all known history event tags is
stored in ODB subtree {{Odbpath|path=/History/Tags}}. For per-equipment history, the 16-bit history event ID is the value of ODB key [[/Equipment ODB tree#Event ID|/Equipment/<equipment-name>/Common/Event ID]].

== SQL history events ==
In the SQL history (MySQL, SQLITE, etc), each history event is an SQL table. The history event tags are the SQL table columns.

== FILE history events ==
In the new FILE history, each history event is written into a separate file, tag definition are recorded in text format in the file header, history event
data is appended to the file in binary format (fixed record size). If the history event definition is changed, a new file will be started.

= History Event Construction =
The [[mlogger]] creates [[#Frontend history event]]s in open_history() by parsing the ODB subtree {{Odbpath|path=/equipment/<equipment-name>/variables}}. Each ODB entry under the {{Odbpath|path=variables/}} subtree is referred to as a "variable". Each variable can be a single ODB value, an array of ODB values, or a subdirectory (corresponding to TID_STRUCT structured data banks). As each variable is processed, one or more tags are created to describe it.

Single ODB values will generally produce a single tag, while arrays can produce (depending whether the array is "[[#Named Arrays|Named]]" or not)
* multiple tags - one per array element ("Named" array) or
* one single tag - describing the whole array

The code can generate two types of history:
<dl>
<dt> "per-equipment" history</dt>
<dd> will have the tags for all variables concatenated together into one single history event</dd>
<dt> "per-variable" history</dt>
<dd> will have one history event defined for each variable. Inside could be one tag (for single odb values and unnamed array) or multiple tags (for named arrays and structured data
banks).</dd>
</dl>

Per-equipment history is the original MIDAS history implementation. Per-variable history was added to permit efficient data storage in SQL tables. Its initial implementation used 1 ODB hotlink for each variable and it was easy to exceed the maximum permitted number of ODB hotlinks (db_open_record()). To reduce consumption of hotlinks, db_watch() has been implemented and now per-variable history only uses 1 ODB hotlink per equipment.

;NOTE
: With db_watch, per-equipment history is no longer available. '''Per-variable history is the new default''' (and the only option).

[https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087]

= Construction of history event tags =
Currently,
* single ODB values are treated as arrays of length "1"
* string ODB entries are not permitted

History event tags are constructed in the [[mlogger]] add_equipment(). Presently there are some quirks with the generation of history event tags (see [https://midas.triumf.ca/elog/Midas/1088]),
e.g. single odb values are treated as arrays of length "1" and "string" ODB entries are not permitted.

For variables of type TID_KEY (subdirectories, corresponding to TID_STRUCT structured banks), one tag is generated for each subdirectory entry. Tag names for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa/bbb}} will be "aaa_bbb" (with an underscore). Subdirectory entries of type TID_KEY and TID_LINK are currently not supported.

For "[[#Named Arrays|Named Arrays]]", individual tags are generated for each array element. Tag names are taken from the ''Names'' array. For empty tag names (empty names array), tags are "aaa_0", "aaa_1", etc (for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}}).

For "[[#Named Arrays|Single Names Arrays]]", tag names have the variable name appended (with a space), for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}} and an empty names array, tags will be "aaa_0 aaa", "aaa_1 aaa", etc. For a populated names array, the tags will be "name0 aaa", "name1 aaa", etc.

For unnamed arrays and single odb variables (in ODB, single odb variables are arrays of length 1), a single tag is generated.

== Named Arrays ==

'''Named Arrays''' have a "Names" entry in {{Odbpath|path=/equipment/<equipment-name>/settings}}.

For example, to add names to {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}}, create a string array {{Odbpath|path=/equipment/<equipment-name>/settings/names aaa}}. The names array should be at least as long as the corresponding data array. Individual entries in the names array can be left blank (tag names will be "aaa_0", "aaa_1", etc). Duplicate tag names are not permitted.

A single "Names" entry can be created to name all arrays in variables with the same names ('''"Single Names"''').

Create {{Odbpath|path=/equipment/<equipment-name>/settings/names}} and arrays {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}} and {{Odbpath|path=/equipment/<equipment-name>/variables/bbb}} will have
history tags "name0 aaa", "name1 aaa", "name0 bbb", "name1 bbb", etc. If "names" are left blank, tag names will be "aaa_0 aaa", "aaa_1 aaa", "bbb_0 bbb", "bbb_1 bbb", etc.

In the [[mhttpd]] web server [[History Page]] variables viewer, "single name" arrays are displayed in a 2D table.

This information is from [https://midas.triumf.ca/elog/Midas/1088].

= Historical note on MIDAS-History Tag names =
Tags are only used for MIDAS-history events.

The mlogger/mhttpd implementation of /History/Tags has proved troublesome and we are moving away from it.

;During the transition period:
: Implementations starting with Version 4435 of [[mlogger]] and [[mhttpd]] (i.e. "new" versions) will now work '''without''' /History/Tags. The history tags are read directly from the history files themselves. To run in this mode, set ODB key [[/History ODB tree#DisableTags|/History/DisableTags]] to "y". Existing [[/History ODB tree#Tags subtree|/History/Tags subtree]] will be automatically deleted.

Two downsides to running like this are
* it is slower and tags become non-persistent
* if some frontends have not been running for a while, their variables may vanish from the history panel editor.

== New-style History Tags ==
For the above two reasons, using /History/Tags is still recommended.
However, the format of the tags is now changed to simplify management and reduce ODB size.

; For NEW implementations
: the key [[/History ODB tree#CreateOldTags|CreateOldTags]] will not be present. "New-style" history tags will be created automatically in the [[/History ODB tree#Tags subtree|Tags subtree]].

;If you have old-style tags
: [[mlogger]] will automatically convert the old tags to this new format (this is why you should make a backup of your ODB).

** using "old" (i.e. pre Version 4435) mlogger with "new" mhttpd is OK: new mhttpd understands both formats of /History/Tags.
** using "old" mhttpd with "new" mlogger is OK provided the ODB key [[/History ODB tree#CreateOldTags|/History/CreateOldTags]] is set to "y" before starting mlogger.

== Old-style History Tags ==
Prior to versions 4435 of [[mlogger]] and [[mhttpd]], old-style history tags were generated.
See
* http://ladd00.triumf.ca/~daqweb/doc/midas-old/html/F_History_logging.html
for details.

 
---------------
 

= MIDAS ODBC SQL History system =

This section describes the internal workings of the MIDAS SQL history system.

The SQL history system implements the MIDAS history using a SQL database for data storage instead of flat binary files.

The SQL history code lives in a separate files history.h and history_sql.cxx. The present implementation uses the ODBC API for accessing SQL databases.
MySQL is fully supported, Pgsql support is partially implemented, support for arbitrary ODBC drivers should be easy to add in history_sql and most SQL operations are done using an interface class.

Preliminary testing with MySQL and PgSQL indicates that disk space requirements are the same for both storage mechanisms using flat binary files or either of the 2 SQL databases. All 3 tested storage mechanisms store the data uncompressed. (Observed gzip-1 compression ratios from CERN-ALPHA history files are 50% or better).

Preliminary testing also indicates that MySQL database is "not too slow" for use by T2K/ND280 slow control system. Using MySQL, performance seems to improve somewhat compared to traditional MIDAS-history because each SQL table is stored into a separate file, compared to everything-in-one-file in traditional MIDAS-history.

== Set up a MySQL database ==

In order to setup a MySQL database, the following is needed:

* the server hostname and port number
* database name
* user name and password.

It is recommended to create 3 different users:

* a "root" user with full priveleges,
* a "reader" user with read-only priveleges
* and a "writer" user with "insert, create table, and add column" priveleges (no permission to drop table, remove columns, etc)

=== Example ===
#Setup a MySQL database (RHEL/SL 5)

ssh root@localhost
service mysqld start
/usr/bin/mysql_secure_installation (enter current password for root: press <enter>, then accept default answer "Y" to all questions)
mysqladmin -p create history
using mysql-administrator, create 2 users:
history_writer, with database "history" schema privileges: select, insert, create, alter, index
history_reader, with database "history" schema privileges: select

#Setup $HOME/.odbc.ini file:

cat $HOME/.odbc.ini
[history_writer]
Description = history_writer
Driver = /usr/lib64/libmyodbc3.so
SERVER = ladd05.triumf.ca
PORT =
Database = history
OPTION = 3
SOCKET =
User = history_writer
Password = ******
[history_reader]
Description = history_reader
Driver = /usr/lib64/libmyodbc3.so
SERVER = ladd05.triumf.ca
PORT =
Database = history
OPTION = 3
SOCKET =
User = history_reader
Password =

Note: if you are using mhist you will need a separate .odbc.ini file for each user that you have connecting to the database (through MIDAS).

== Layout of SQL tables ==

* One table is created for each history event. Table name is the same as the history event name (as reported by [[mhdump]]).
* one column is created for each history tag (history variable). Column name is the same as the tag name (as reported by mhdump). For array tags, one column is created for each array element: array a[3] will produce columns a_0, a_1 and a_2.
* two special columns are always created: _i_time and _t_time are the timestamps in the UNIX-time integer format and in the SQL "time" format. (A priori, it is not clear which timestamp format is more useful for end users. mhttpd uses the _i_time integer time format). Note that this timestamp is the time when mlogger receives the hot-link signal (see above) and it can be a few seconds behind the time when the user placed the data into the MIDAS bank. For high precision (and sub-second time resolution) time stamps, users should generate their own timestamps and include them as part of the data itself.
* a special table called _history_index is created to remember the mapping between SQL names and MIDAS names for event names, variable name and tag names.

=== Schema Evolution ===

* when new data fields are added, new SQL columns are created automatically
* existing SQL columns are never deleted
* arrays are expanded automatically, and arrays never shrink.
* if the type of a data field changes, a conflict may occur with existing SQL columns, for example if previous type was "FLOAT" and new type is "DOUBLE", the existing SQL columns would have the SQL data type "float", too narrow to store new values. mlogger will complain about this. Such conflicts may be resolved by changing the type of the SQL column using SQL tools. In general, SQL names are created from MIDAS names by replacing all non-alphanumerical characters to an underscore "_". If this procedure creates duplicate column names, the column names and made unique by appending them with a random number, i.e. MIDAS names "A+B" and "A-B" will be translated to "A_B" and "A_B_12345" (random integer). True MIDAS names are saved into the _history_index.

[[Category:history]]

History System

2020-07-30T16:43:10Z

Olchansk: /* Write MYSQL-history events */

{{Pagelinks}}

== Links ==
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[/History ODB tree]]
* [[History Page]]
* [[Mhformat|MIDAS history file format]]
* [[Keys in the ODB /Logger tree|/Logger ODB tree]]
* [[Keys in the ODB /Logger/History subtree|/Logger/History subtree]]
* [[Image History]]
</div>

<div id="history file"></div>
= Introduction =

The history system is an add-on capability built into the MIDAS data logger [[mlogger]] to record useful information in parallel to the data logging.
This information is recorded with a time stamp and saved into a '''history file''' (see [[#History drivers|History drivers]]) for later retrieval. One set of files is created per day containing all the requested history events. The history data may be displayed in graphical form using the [[mhttpd]] MIDAS webserver utility, giving the user an easy way of seeing how experimental variables have changed with time.
The history logging will be in action only if the MIDAS data logger is running, but it is not necessary to have any [[Keys in the ODB /Logger tree#/Logger/Channels subtree|data logging channel]] enabled.

Since June 2020, it is possible to also use the midas history system to record images (png/jpeg etc) periodically, for example to later produce a timelapse from a webcam - see the [[Image History]] page for more details.

= MidasHistoryInterface =

The new history configuration defines history channels similar to logger channels. Each history channel creates a history interface class (class MidasHistoryInterface). The [[mlogger|data logger]] can use this interface to write data into the history, mhttpd, mhist & co can use this interface to read data from the history. The interface is defined in history.h

* http://ladd00.triumf.ca/~daqweb/doc/midas/include/history.h

= History drivers =

Presently, the MIDAS distribution contains these drivers:

; MIDAS-History
: this is the traditional history driver, with data stored in the .hst binary files. (format is documented here [[mhformat]], see also [[mhist]] and [[mhdump]])
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#MIDAS subtree |"MIDAS"]]''' is set up to write MIDAS-history.

; ODBC-History
: stores history data in SQL database using the ODBC abstraction layer. See [[#MIDAS ODBC SQL History system|ODBC SQL History system]].

: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#ODBC subtree|"ODBC"]]''' is set up to write ODBC-history.

; SQLITE-History
: stores data in SQL database as SQLITE3 .sqlite3 files. The SQL table format is similar to ODBC-history, performance is similar, no database server required.
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#SQLITE subtree|"SQLITE"]]''' is set up to write SQLITE-history.

; MYSQL-History
:
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#MYSQL subtree|"MYSQL"]]''' is set up to write MYSQL-history.

; FILE-History
: stores each variable in its own file, making it much easier to trace the history of a particular variable.
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#FILE subtree|FILE]]''' is set up to write FILE-history.

More drivers can be easily added by the user (create history_xxx.cxx, add constructor to hs_get_history(), create the ODB subtree /Logger/History/xxx/channel).

= Location of History Files =
;MIDAS-history
:By default, the traditional MIDAS-history is enabled, writing history files into the MIDAS data directory path given by the ODB key [[Keys in the ODB /Logger tree#Data dir|Data dir]].
:If history data is important, it is recommended to write it to a directory that is backed up or archived. (The MIDAS data directory is usually located on a scratch data disk without backups).

:The location of the MIDAS-history data files can be changed by use of the ODB Key [[Keys in the ODB /Logger tree#History dir|History dir]].

;ODBC-History
:The location of ODBC-History files is ...

;SQLITE-History
:The location of SQLITE-History files is the directory path given by the ODB key [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]].

;MYSQL-History
:The location of MYSQL-History files is ...

;FILE-History
:The location of FILE-History files is the directory path given by the ODB key [[Keys in the ODB /Logger/History subtree#History dir|History dir]].

= Types of History Events =

There are two basic types of history events, which are defined in different ways:

* "[[#Frontend History Event|Frontend]]" (or "Equipment") History event composed in the frontend. See [[#Frontend History Event|Frontend History Event]] for details.
* "Virtual" (or "Links") History event composed within the ODB under the [[/History ODB tree#Links subtree|/History/Links subtree]]. See [[#Virtual History Event|Virtual History Event]] for details.

Both these history event definitions take effect when the data logger gets a start run [[Run States and Transitions|transition]]. Any modifications made during the run are not applied until the start of the next run.

== Virtual History Event ==
The history of any ODB variable can be recorded in a "virtual" history event by creating a link to that variable in the [[/History ODB tree#Links subtree|/History/Links subtree]]. This is used for ODB variables that are not in an Equipment.

; To enable Virtual History Events
: Create links as desired in [[/History ODB tree#Links subtree|/History/Links subtree]].

History events are created for each entry under ODB subdirectory {{Odbpath|path=/History/Links}}.

Two types of links are permitted:
<dl>
<dt> ODB key {{Odbpath|path=/history/links/aaa}} is a link to a subdirectory</dt>
<dd><ol>
<li> db_watch() is setup to watch this subdirectory</li>
<li> tags are created for each subdirectory entry (1 tag per entry)</li>
<li> There is no possibility for naming array elements, so 1 tag per array, regardless of the number of elements.</li> </ol></dd>
<dt>ODB key {{Odbpath|path=/history/links/bbb}} is a subdirectory with links to ODB values</dt>
<dd> <ol>
<li>db_watch is setup to watch each link target</li>
<li> tags are created for each link (1 tag per link)</li>
<li> tag name is the link name (NOT the target name). There is no possibility for naming array elements</li>
</ol></dd>
</dl>

Mixing links and subdirectories is not permitted.

The update period of history events created for {{Odbpath|path=/history/links}} is controlled by entries in {{Odbpath|path=/history/links periods}}. Numeric values of periods are same as for a [[#Frontend History Event|Frontend History event]]. Numeric value 0 disables the history for a particular event.

See [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087] for more information.

== Frontend History Event ==

Each [[Frontend Operation#Equipments and Events|Equipment]] has the capability to generate "history data".

; To enable Frontend History Events

: Set the [[/Equipment ODB tree#Log history|Log history]] ODB key of each equipment to be logged '''non-zero'''. (See [[Equipment List Parameters#Log History|Log History parameter]]). (You may need to restart the frontend.)

This causes the event contents to be copied to the ODB [[/Equipment ODB tree#Variables subtree|/Equipment/''<equipment-name>''/Variables subtree]] for the use of the History system. history events are created by parsing the content of {{Odbpath|path=/Equipment/<equipment-name>/variables}}.

For each equipment, history is controlled by the value of {{Odbpath|path=/Equipment/<equipment-name>/common/period}} as follows:
* 0 = history disabled
* 1 = history is enabled
* >1 = history is enabled, throttled down

The throttling is implemented in log_history()/watch_history() by this algorithm:
the very first history event is recorded, then all changed to the data are ignored until "period" seconds has elapsed. Then the next history event will be recorded, and following
changes will be ignored until "period" second elapses, and so forth. Period value "1" has special meaning - there is no throttling, all history events are logged.

This information is from [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087].

=== Sequence for frontend History Event ===
The main steps for the frontend History Event is as follows:

* The user code in the frontend equipment Event Readout routines reads the data, placing it into a MIDAS data bank
* In mfe.c, if the key [[/Equipment ODB tree#Log history|Log history]] > 0 (i.e. the history system is enabled for this equipment), this data bank is written into ODB (by update_odb()).
* This ODB write triggers an ODB hot-link into mlogger
* The hot-link calls mlogger.c::log_history(), which calls hs_write() to write the data into the history file. The frequency of the history writes is specified by the number of seconds between writes stored in the ODB key [[/Equipment ODB tree#Log history|Log history]].
* history.c::hs_write() or history_odbc::hs_write_odbc() writes the data into a history file or into an SQL database.

= How to Write History events =
* Enable desired [[#To enable Frontend History Events|Frontend]] and/or [[#To enable Virtual History Events|Virtual]] history events
* [[#Select the history driver|Select the history driver]]
* Follow instructions below for selected driver:
** [[#Write MIDAS-History events|Write MIDAS-history events]]
** [[#Write ODBC-History events|Write ODBC-history events]]
** [[#Write SQLITE-History events|Write SQLITE-history events]]
** [[#Write MYSQL-History events|Write MYSQL-history events]]
** [[#Write FILE-History events|Write FILE-history events]]

== Select the history driver ==

* Select the '''[[#History drivers|history driver]]''' and therefore the [[Keys in the ODB /Logger/History subtree#<history-logging-channel> subtrees|history-logging-channel]] to use to write the MIDAS-history files.
* set [[Keys in the ODB /Logger/History subtree#Active|Active]] ODB key in the selected [[Keys in the ODB /Logger/History subtree#<history-logging-channel> subtrees|history-logging-channel]] to "y". All other logging channels should be disabled.
* set [[/History ODB tree#LoggerHistoryChannel|LoggerHistoryChannel]] ODB key to the selected [[#History drivers|History channel]].

== Write MIDAS-History events ==
* Enable desired [[#To enable Frontend History Events|Frontend]] and/or [[#To enable Virtual History Events|Virtual]] history events
* Perform steps in [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "MIDAS".
* Restart [[mhttpd]] and [[mlogger]], start the run.

MIDAS-history files will be saved into the [[#Location of History Files|MIDAS-history data directory]].

== Write ODBC-History events ==

See [[#MIDAS ODBC SQL History system|MIDAS ODBC SQL History system]].
* If necessary, [[#Set up a MySQL database|set up a MySQL database]].
* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps in [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "ODBC".
* Check ODB Key [[Keys in the ODB /Logger/History subtree#Writer_ODBC_DSN|Writer_ODBC_DSN]] is set to "history writer" and Key [[Keys in the ODB /Logger/History subtree#Reader_ODBC_DSN|Reader_ODBC_DSN]] is set to "history reader" (the default values)
* set [[Keys in the ODB /Logger/History subtree#Debug|Debug]] to 1 if desired
* restart [[mlogger]] in verbose mode (i.e. mlogger -v), and observe how it issues SQL commands to create the tables and columns corresponding to MIDAS-history events and tags.
* restart [[mhttpd]]

'''Notes'''
# [[mlogger]] is programmed to raise alarms if connection to SQL database is interrupted or if some events cannot be written into the database (i.e. data type mismatch, SQL syntax errors, etc). The traditional MIDAS-history never raised alarms because it "never failed" - other than from "disk full" errors, which are immediately obvious.
# With ODBC History enabled, [[mhttpd]] will only use history information from the SQL database to make history plots and to extract history variable names for the history plot editor. The ODB {{Odbpath|path=/History}} subtrees[[/History ODB tree#Tags subtree|Tags]] and [[/History ODB tree#Events subtree|Events]] are not used.
# variable names presented to the user may change from MIDAS names to SQL names (use the history panel "Label" text fields to create permanent plot labels).

== Write SQLITE-history events ==

No special configuration required.

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "SQLITE".

* Create a directory for storage of sqlite history files
* save its path in the ODB key [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]]
* restart [[mhttpd]] and [[mlogger]]

In the history data directory (in [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]]) you should see files with name "mh_xxx.sqlite3" corresponding to your equipment and variable names.

== Write MYSQL-history events ==

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "MYSQL".
* setup the MySQL database:
* install mariadb-10.4
<pre>
macos:
sudo port install mariadb-10.4
ln -s ln -fs /opt/local/lib/mariadb-10.4/bin/mysql_config ~/bin/
mysql_config --version
10.4.13
</pre>
* rebuild midas, check the cmake finds the correct mysql: MIDAS: Found MySQL version 10.4.13
* create $HOME/.my.cnf
<pre>
[mysqld]
#port=3306
socket=/Users/olchansk/mysql/mysql.sock
datadir=/Users/olchansk/mysql/db
pid-file=/Users/olchansk/mysql/mysqld.pid

[client]
#port=3306
socket=/Users/olchansk/mysql/mysql.sock
</pre>
* mkdir $HOME/mysql
* start mysql:
<pre>
macos: /opt/local/lib/mariadb-10.4/bin/mysqld
</pre>
* configure the database: start the "mysql" utility (/opt/local/lib/mariadb-10.4/bin/mysql)
<pre>
NOTE: instructions thanks to Ben S.
NOTE: use some other passwords instead of the example "reader_password" and "writer_password"

MariaDB [(none)]> CREATE DATABASE IF NOT EXISTS history;
Query OK, 1 row affected (0.003 sec)

MariaDB [(none)]> CREATE USER history_reader@'localhost' IDENTIFIED BY 'reader_password';
Query OK, 0 rows affected (0.027 sec)

MariaDB [(none)]> GRANT SELECT ON history.* TO history_reader@'localhost';
Query OK, 0 rows affected (0.016 sec)

MariaDB [(none)]> CREATE USER 'history_writer'@'localhost' IDENTIFIED BY 'writer_password';
Query OK, 0 rows affected (0.014 sec)

MariaDB [(none)]> GRANT SELECT,INSERT,CREATE,ALTER ON history.* TO 'history_writer'@'localhost';
Query OK, 0 rows affected (0.017 sec)
</pre>
* try connecting to both users:
<pre>
/opt/local/lib/mariadb-10.4/bin/mysql -u history_reader -p ### should ask for reader password
/opt/local/lib/mariadb-10.4/bin/mysql -u history_writer -p ### should ask for writer password
</pre>
* create midas config files:
** mysql_writer.txt
<pre>
#server=localhost
#port=xxx
database=history
socket=/Users/olchansk/mysql/mysql.sock
user=history_writer
password=writer_password ### change this!!!
#buffer=1000
</pre>
** mysql_reader.txt
<pre>
#server=localhost
#port=xxx
database=history
socket=/Users/olchansk/mysql/mysql.sock
user=history_reader
password=reader_password ### change this!!!
</pre>
* enable MYSQL history in mlogger:
** change ODB /Logger/History/MYSQL/Active to "y", "mysql writer" and "mysql reader" to location of these files (leave the default values if these files are in the experiment directory).
* enable MYSQL history in mhttpd:
** change ODB /History/LoggerHistoryChannel to "MYSQL"
* restart mlogger, restart mhttpd
...........

== Write FILE-history events ==

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "SQLITE".
* Assign directory path for saved files using the ODB key [[Keys in the ODB /Logger/History subtree#History dir|History dir]].
...........

 
---------------
 

= History Events =
The following information is derived from [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087].

A "history event" is a history atomic unit of data. Associated with each history event is a timestamp (unix time),
a name (limited to NAME_LENGTH in the old history) and a list of history tags that describe the individual data
values inside the history event.

When making history plots in [[mhttpd]] (see [[History Page]]) for each curve on the plot, one selects a history event (from the list
of currently active events, recently active events or the list of all events that ever existed), then from the list of tags
inside the history event one selects the particular variable that will be plotted.

== Old MIDAS history events ==
In the old MIDAS history, all history events are written into one history file (.hst file + optional .def and .idx event definition and time index files
which can be/are regenerated automatically from the .hst file). History events are identified by 16-bit history event IDs, the persistent mapping
from history event names and the 16-bit history event IDs is stored in ODB subtree {{Odbpath|path=/History/Events}}. In addition the list of all known history event tags is
stored in ODB subtree {{Odbpath|path=/History/Tags}}. For per-equipment history, the 16-bit history event ID is the value of ODB key [[/Equipment ODB tree#Event ID|/Equipment/<equipment-name>/Common/Event ID]].

== SQL history events ==
In the SQL history (MySQL, SQLITE, etc), each history event is an SQL table. The history event tags are the SQL table columns.

== FILE history events ==
In the new FILE history, each history event is written into a separate file, tag definition are recorded in text format in the file header, history event
data is appended to the file in binary format (fixed record size). If the history event definition is changed, a new file will be started.

= History Event Construction =
The [[mlogger]] creates [[#Frontend history event]]s in open_history() by parsing the ODB subtree {{Odbpath|path=/equipment/<equipment-name>/variables}}. Each ODB entry under the {{Odbpath|path=variables/}} subtree is referred to as a "variable". Each variable can be a single ODB value, an array of ODB values, or a subdirectory (corresponding to TID_STRUCT structured data banks). As each variable is processed, one or more tags are created to describe it.

Single ODB values will generally produce a single tag, while arrays can produce (depending whether the array is "[[#Named Arrays|Named]]" or not)
* multiple tags - one per array element ("Named" array) or
* one single tag - describing the whole array

The code can generate two types of history:
<dl>
<dt> "per-equipment" history</dt>
<dd> will have the tags for all variables concatenated together into one single history event</dd>
<dt> "per-variable" history</dt>
<dd> will have one history event defined for each variable. Inside could be one tag (for single odb values and unnamed array) or multiple tags (for named arrays and structured data
banks).</dd>
</dl>

Per-equipment history is the original MIDAS history implementation. Per-variable history was added to permit efficient data storage in SQL tables. Its initial implementation used 1 ODB hotlink for each variable and it was easy to exceed the maximum permitted number of ODB hotlinks (db_open_record()). To reduce consumption of hotlinks, db_watch() has been implemented and now per-variable history only uses 1 ODB hotlink per equipment.

;NOTE
: With db_watch, per-equipment history is no longer available. '''Per-variable history is the new default''' (and the only option).

[https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087]

= Construction of history event tags =
Currently,
* single ODB values are treated as arrays of length "1"
* string ODB entries are not permitted

History event tags are constructed in the [[mlogger]] add_equipment(). Presently there are some quirks with the generation of history event tags (see [https://midas.triumf.ca/elog/Midas/1088]),
e.g. single odb values are treated as arrays of length "1" and "string" ODB entries are not permitted.

For variables of type TID_KEY (subdirectories, corresponding to TID_STRUCT structured banks), one tag is generated for each subdirectory entry. Tag names for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa/bbb}} will be "aaa_bbb" (with an underscore). Subdirectory entries of type TID_KEY and TID_LINK are currently not supported.

For "[[#Named Arrays|Named Arrays]]", individual tags are generated for each array element. Tag names are taken from the ''Names'' array. For empty tag names (empty names array), tags are "aaa_0", "aaa_1", etc (for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}}).

For "[[#Named Arrays|Single Names Arrays]]", tag names have the variable name appended (with a space), for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}} and an empty names array, tags will be "aaa_0 aaa", "aaa_1 aaa", etc. For a populated names array, the tags will be "name0 aaa", "name1 aaa", etc.

For unnamed arrays and single odb variables (in ODB, single odb variables are arrays of length 1), a single tag is generated.

== Named Arrays ==

'''Named Arrays''' have a "Names" entry in {{Odbpath|path=/equipment/<equipment-name>/settings}}.

For example, to add names to {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}}, create a string array {{Odbpath|path=/equipment/<equipment-name>/settings/names aaa}}. The names array should be at least as long as the corresponding data array. Individual entries in the names array can be left blank (tag names will be "aaa_0", "aaa_1", etc). Duplicate tag names are not permitted.

A single "Names" entry can be created to name all arrays in variables with the same names ('''"Single Names"''').

Create {{Odbpath|path=/equipment/<equipment-name>/settings/names}} and arrays {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}} and {{Odbpath|path=/equipment/<equipment-name>/variables/bbb}} will have
history tags "name0 aaa", "name1 aaa", "name0 bbb", "name1 bbb", etc. If "names" are left blank, tag names will be "aaa_0 aaa", "aaa_1 aaa", "bbb_0 bbb", "bbb_1 bbb", etc.

In the [[mhttpd]] web server [[History Page]] variables viewer, "single name" arrays are displayed in a 2D table.

This information is from [https://midas.triumf.ca/elog/Midas/1088].

= Historical note on MIDAS-History Tag names =
Tags are only used for MIDAS-history events.

The mlogger/mhttpd implementation of /History/Tags has proved troublesome and we are moving away from it.

;During the transition period:
: Implementations starting with Version 4435 of [[mlogger]] and [[mhttpd]] (i.e. "new" versions) will now work '''without''' /History/Tags. The history tags are read directly from the history files themselves. To run in this mode, set ODB key [[/History ODB tree#DisableTags|/History/DisableTags]] to "y". Existing [[/History ODB tree#Tags subtree|/History/Tags subtree]] will be automatically deleted.

Two downsides to running like this are
* it is slower and tags become non-persistent
* if some frontends have not been running for a while, their variables may vanish from the history panel editor.

== New-style History Tags ==
For the above two reasons, using /History/Tags is still recommended.
However, the format of the tags is now changed to simplify management and reduce ODB size.

; For NEW implementations
: the key [[/History ODB tree#CreateOldTags|CreateOldTags]] will not be present. "New-style" history tags will be created automatically in the [[/History ODB tree#Tags subtree|Tags subtree]].

;If you have old-style tags
: [[mlogger]] will automatically convert the old tags to this new format (this is why you should make a backup of your ODB).

** using "old" (i.e. pre Version 4435) mlogger with "new" mhttpd is OK: new mhttpd understands both formats of /History/Tags.
** using "old" mhttpd with "new" mlogger is OK provided the ODB key [[/History ODB tree#CreateOldTags|/History/CreateOldTags]] is set to "y" before starting mlogger.

== Old-style History Tags ==
Prior to versions 4435 of [[mlogger]] and [[mhttpd]], old-style history tags were generated.
See
* http://ladd00.triumf.ca/~daqweb/doc/midas-old/html/F_History_logging.html
for details.

 
---------------
 

= MIDAS ODBC SQL History system =

This section describes the internal workings of the MIDAS SQL history system.

The SQL history system implements the MIDAS history using a SQL database for data storage instead of flat binary files.

The SQL history code lives in a separate files history.h and history_sql.cxx. The present implementation uses the ODBC API for accessing SQL databases.
MySQL is fully supported, Pgsql support is partially implemented, support for arbitrary ODBC drivers should be easy to add in history_sql and most SQL operations are done using an interface class.

Preliminary testing with MySQL and PgSQL indicates that disk space requirements are the same for both storage mechanisms using flat binary files or either of the 2 SQL databases. All 3 tested storage mechanisms store the data uncompressed. (Observed gzip-1 compression ratios from CERN-ALPHA history files are 50% or better).

Preliminary testing also indicates that MySQL database is "not too slow" for use by T2K/ND280 slow control system. Using MySQL, performance seems to improve somewhat compared to traditional MIDAS-history because each SQL table is stored into a separate file, compared to everything-in-one-file in traditional MIDAS-history.

== Set up a MySQL database ==

In order to setup a MySQL database, the following is needed:

* the server hostname and port number
* database name
* user name and password.

It is recommended to create 3 different users:

* a "root" user with full priveleges,
* a "reader" user with read-only priveleges
* and a "writer" user with "insert, create table, and add column" priveleges (no permission to drop table, remove columns, etc)

=== Example ===
#Setup a MySQL database (RHEL/SL 5)

ssh root@localhost
service mysqld start
/usr/bin/mysql_secure_installation (enter current password for root: press <enter>, then accept default answer "Y" to all questions)
mysqladmin -p create history
using mysql-administrator, create 2 users:
history_writer, with database "history" schema privileges: select, insert, create, alter, index
history_reader, with database "history" schema privileges: select

#Setup $HOME/.odbc.ini file:

cat $HOME/.odbc.ini
[history_writer]
Description = history_writer
Driver = /usr/lib64/libmyodbc3.so
SERVER = ladd05.triumf.ca
PORT =
Database = history
OPTION = 3
SOCKET =
User = history_writer
Password = ******
[history_reader]
Description = history_reader
Driver = /usr/lib64/libmyodbc3.so
SERVER = ladd05.triumf.ca
PORT =
Database = history
OPTION = 3
SOCKET =
User = history_reader
Password =

Note: if you are using mhist you will need a separate .odbc.ini file for each user that you have connecting to the database (through MIDAS).

== Layout of SQL tables ==

* One table is created for each history event. Table name is the same as the history event name (as reported by [[mhdump]]).
* one column is created for each history tag (history variable). Column name is the same as the tag name (as reported by mhdump). For array tags, one column is created for each array element: array a[3] will produce columns a_0, a_1 and a_2.
* two special columns are always created: _i_time and _t_time are the timestamps in the UNIX-time integer format and in the SQL "time" format. (A priori, it is not clear which timestamp format is more useful for end users. mhttpd uses the _i_time integer time format). Note that this timestamp is the time when mlogger receives the hot-link signal (see above) and it can be a few seconds behind the time when the user placed the data into the MIDAS bank. For high precision (and sub-second time resolution) time stamps, users should generate their own timestamps and include them as part of the data itself.
* a special table called _history_index is created to remember the mapping between SQL names and MIDAS names for event names, variable name and tag names.

=== Schema Evolution ===

* when new data fields are added, new SQL columns are created automatically
* existing SQL columns are never deleted
* arrays are expanded automatically, and arrays never shrink.
* if the type of a data field changes, a conflict may occur with existing SQL columns, for example if previous type was "FLOAT" and new type is "DOUBLE", the existing SQL columns would have the SQL data type "float", too narrow to store new values. mlogger will complain about this. Such conflicts may be resolved by changing the type of the SQL column using SQL tools. In general, SQL names are created from MIDAS names by replacing all non-alphanumerical characters to an underscore "_". If this procedure creates duplicate column names, the column names and made unique by appending them with a random number, i.e. MIDAS names "A+B" and "A-B" will be translated to "A_B" and "A_B_12345" (random integer). True MIDAS names are saved into the _history_index.

[[Category:history]]

History System

2020-07-30T16:20:37Z

Olchansk: /* Write MYSQL-history events */

{{Pagelinks}}

== Links ==
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[/History ODB tree]]
* [[History Page]]
* [[Mhformat|MIDAS history file format]]
* [[Keys in the ODB /Logger tree|/Logger ODB tree]]
* [[Keys in the ODB /Logger/History subtree|/Logger/History subtree]]
* [[Image History]]
</div>

<div id="history file"></div>
= Introduction =

The history system is an add-on capability built into the MIDAS data logger [[mlogger]] to record useful information in parallel to the data logging.
This information is recorded with a time stamp and saved into a '''history file''' (see [[#History drivers|History drivers]]) for later retrieval. One set of files is created per day containing all the requested history events. The history data may be displayed in graphical form using the [[mhttpd]] MIDAS webserver utility, giving the user an easy way of seeing how experimental variables have changed with time.
The history logging will be in action only if the MIDAS data logger is running, but it is not necessary to have any [[Keys in the ODB /Logger tree#/Logger/Channels subtree|data logging channel]] enabled.

Since June 2020, it is possible to also use the midas history system to record images (png/jpeg etc) periodically, for example to later produce a timelapse from a webcam - see the [[Image History]] page for more details.

= MidasHistoryInterface =

The new history configuration defines history channels similar to logger channels. Each history channel creates a history interface class (class MidasHistoryInterface). The [[mlogger|data logger]] can use this interface to write data into the history, mhttpd, mhist & co can use this interface to read data from the history. The interface is defined in history.h

* http://ladd00.triumf.ca/~daqweb/doc/midas/include/history.h

= History drivers =

Presently, the MIDAS distribution contains these drivers:

; MIDAS-History
: this is the traditional history driver, with data stored in the .hst binary files. (format is documented here [[mhformat]], see also [[mhist]] and [[mhdump]])
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#MIDAS subtree |"MIDAS"]]''' is set up to write MIDAS-history.

; ODBC-History
: stores history data in SQL database using the ODBC abstraction layer. See [[#MIDAS ODBC SQL History system|ODBC SQL History system]].

: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#ODBC subtree|"ODBC"]]''' is set up to write ODBC-history.

; SQLITE-History
: stores data in SQL database as SQLITE3 .sqlite3 files. The SQL table format is similar to ODBC-history, performance is similar, no database server required.
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#SQLITE subtree|"SQLITE"]]''' is set up to write SQLITE-history.

; MYSQL-History
:
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#MYSQL subtree|"MYSQL"]]''' is set up to write MYSQL-history.

; FILE-History
: stores each variable in its own file, making it much easier to trace the history of a particular variable.
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#FILE subtree|FILE]]''' is set up to write FILE-history.

More drivers can be easily added by the user (create history_xxx.cxx, add constructor to hs_get_history(), create the ODB subtree /Logger/History/xxx/channel).

= Location of History Files =
;MIDAS-history
:By default, the traditional MIDAS-history is enabled, writing history files into the MIDAS data directory path given by the ODB key [[Keys in the ODB /Logger tree#Data dir|Data dir]].
:If history data is important, it is recommended to write it to a directory that is backed up or archived. (The MIDAS data directory is usually located on a scratch data disk without backups).

:The location of the MIDAS-history data files can be changed by use of the ODB Key [[Keys in the ODB /Logger tree#History dir|History dir]].

;ODBC-History
:The location of ODBC-History files is ...

;SQLITE-History
:The location of SQLITE-History files is the directory path given by the ODB key [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]].

;MYSQL-History
:The location of MYSQL-History files is ...

;FILE-History
:The location of FILE-History files is the directory path given by the ODB key [[Keys in the ODB /Logger/History subtree#History dir|History dir]].

= Types of History Events =

There are two basic types of history events, which are defined in different ways:

* "[[#Frontend History Event|Frontend]]" (or "Equipment") History event composed in the frontend. See [[#Frontend History Event|Frontend History Event]] for details.
* "Virtual" (or "Links") History event composed within the ODB under the [[/History ODB tree#Links subtree|/History/Links subtree]]. See [[#Virtual History Event|Virtual History Event]] for details.

Both these history event definitions take effect when the data logger gets a start run [[Run States and Transitions|transition]]. Any modifications made during the run are not applied until the start of the next run.

== Virtual History Event ==
The history of any ODB variable can be recorded in a "virtual" history event by creating a link to that variable in the [[/History ODB tree#Links subtree|/History/Links subtree]]. This is used for ODB variables that are not in an Equipment.

; To enable Virtual History Events
: Create links as desired in [[/History ODB tree#Links subtree|/History/Links subtree]].

History events are created for each entry under ODB subdirectory {{Odbpath|path=/History/Links}}.

Two types of links are permitted:
<dl>
<dt> ODB key {{Odbpath|path=/history/links/aaa}} is a link to a subdirectory</dt>
<dd><ol>
<li> db_watch() is setup to watch this subdirectory</li>
<li> tags are created for each subdirectory entry (1 tag per entry)</li>
<li> There is no possibility for naming array elements, so 1 tag per array, regardless of the number of elements.</li> </ol></dd>
<dt>ODB key {{Odbpath|path=/history/links/bbb}} is a subdirectory with links to ODB values</dt>
<dd> <ol>
<li>db_watch is setup to watch each link target</li>
<li> tags are created for each link (1 tag per link)</li>
<li> tag name is the link name (NOT the target name). There is no possibility for naming array elements</li>
</ol></dd>
</dl>

Mixing links and subdirectories is not permitted.

The update period of history events created for {{Odbpath|path=/history/links}} is controlled by entries in {{Odbpath|path=/history/links periods}}. Numeric values of periods are same as for a [[#Frontend History Event|Frontend History event]]. Numeric value 0 disables the history for a particular event.

See [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087] for more information.

== Frontend History Event ==

Each [[Frontend Operation#Equipments and Events|Equipment]] has the capability to generate "history data".

; To enable Frontend History Events

: Set the [[/Equipment ODB tree#Log history|Log history]] ODB key of each equipment to be logged '''non-zero'''. (See [[Equipment List Parameters#Log History|Log History parameter]]). (You may need to restart the frontend.)

This causes the event contents to be copied to the ODB [[/Equipment ODB tree#Variables subtree|/Equipment/''<equipment-name>''/Variables subtree]] for the use of the History system. history events are created by parsing the content of {{Odbpath|path=/Equipment/<equipment-name>/variables}}.

For each equipment, history is controlled by the value of {{Odbpath|path=/Equipment/<equipment-name>/common/period}} as follows:
* 0 = history disabled
* 1 = history is enabled
* >1 = history is enabled, throttled down

The throttling is implemented in log_history()/watch_history() by this algorithm:
the very first history event is recorded, then all changed to the data are ignored until "period" seconds has elapsed. Then the next history event will be recorded, and following
changes will be ignored until "period" second elapses, and so forth. Period value "1" has special meaning - there is no throttling, all history events are logged.

This information is from [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087].

=== Sequence for frontend History Event ===
The main steps for the frontend History Event is as follows:

* The user code in the frontend equipment Event Readout routines reads the data, placing it into a MIDAS data bank
* In mfe.c, if the key [[/Equipment ODB tree#Log history|Log history]] > 0 (i.e. the history system is enabled for this equipment), this data bank is written into ODB (by update_odb()).
* This ODB write triggers an ODB hot-link into mlogger
* The hot-link calls mlogger.c::log_history(), which calls hs_write() to write the data into the history file. The frequency of the history writes is specified by the number of seconds between writes stored in the ODB key [[/Equipment ODB tree#Log history|Log history]].
* history.c::hs_write() or history_odbc::hs_write_odbc() writes the data into a history file or into an SQL database.

= How to Write History events =
* Enable desired [[#To enable Frontend History Events|Frontend]] and/or [[#To enable Virtual History Events|Virtual]] history events
* [[#Select the history driver|Select the history driver]]
* Follow instructions below for selected driver:
** [[#Write MIDAS-History events|Write MIDAS-history events]]
** [[#Write ODBC-History events|Write ODBC-history events]]
** [[#Write SQLITE-History events|Write SQLITE-history events]]
** [[#Write MYSQL-History events|Write MYSQL-history events]]
** [[#Write FILE-History events|Write FILE-history events]]

== Select the history driver ==

* Select the '''[[#History drivers|history driver]]''' and therefore the [[Keys in the ODB /Logger/History subtree#<history-logging-channel> subtrees|history-logging-channel]] to use to write the MIDAS-history files.
* set [[Keys in the ODB /Logger/History subtree#Active|Active]] ODB key in the selected [[Keys in the ODB /Logger/History subtree#<history-logging-channel> subtrees|history-logging-channel]] to "y". All other logging channels should be disabled.
* set [[/History ODB tree#LoggerHistoryChannel|LoggerHistoryChannel]] ODB key to the selected [[#History drivers|History channel]].

== Write MIDAS-History events ==
* Enable desired [[#To enable Frontend History Events|Frontend]] and/or [[#To enable Virtual History Events|Virtual]] history events
* Perform steps in [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "MIDAS".
* Restart [[mhttpd]] and [[mlogger]], start the run.

MIDAS-history files will be saved into the [[#Location of History Files|MIDAS-history data directory]].

== Write ODBC-History events ==

See [[#MIDAS ODBC SQL History system|MIDAS ODBC SQL History system]].
* If necessary, [[#Set up a MySQL database|set up a MySQL database]].
* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps in [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "ODBC".
* Check ODB Key [[Keys in the ODB /Logger/History subtree#Writer_ODBC_DSN|Writer_ODBC_DSN]] is set to "history writer" and Key [[Keys in the ODB /Logger/History subtree#Reader_ODBC_DSN|Reader_ODBC_DSN]] is set to "history reader" (the default values)
* set [[Keys in the ODB /Logger/History subtree#Debug|Debug]] to 1 if desired
* restart [[mlogger]] in verbose mode (i.e. mlogger -v), and observe how it issues SQL commands to create the tables and columns corresponding to MIDAS-history events and tags.
* restart [[mhttpd]]

'''Notes'''
# [[mlogger]] is programmed to raise alarms if connection to SQL database is interrupted or if some events cannot be written into the database (i.e. data type mismatch, SQL syntax errors, etc). The traditional MIDAS-history never raised alarms because it "never failed" - other than from "disk full" errors, which are immediately obvious.
# With ODBC History enabled, [[mhttpd]] will only use history information from the SQL database to make history plots and to extract history variable names for the history plot editor. The ODB {{Odbpath|path=/History}} subtrees[[/History ODB tree#Tags subtree|Tags]] and [[/History ODB tree#Events subtree|Events]] are not used.
# variable names presented to the user may change from MIDAS names to SQL names (use the history panel "Label" text fields to create permanent plot labels).

== Write SQLITE-history events ==

No special configuration required.

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "SQLITE".

* Create a directory for storage of sqlite history files
* save its path in the ODB key [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]]
* restart [[mhttpd]] and [[mlogger]]

In the history data directory (in [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]]) you should see files with name "mh_xxx.sqlite3" corresponding to your equipment and variable names.

== Write MYSQL-history events ==

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "MYSQL".
* setup the MySQL database:
* install mariadb-10.4
<pre>
macos:
sudo port install mariadb-10.4
ln -s ln -fs /opt/local/lib/mariadb-10.4/bin/mysql_config ~/bin/
mysql_config --version
10.4.13
</pre>
* rebuild midas, check the cmake finds the correct mysql: MIDAS: Found MySQL version 10.4.13
* create $HOME/.my.cnf
<pre>
[mysqld]
#port=3306
socket=/Users/olchansk/mysql/mysql.sock
datadir=/Users/olchansk/mysql/db
pid-file=/Users/olchansk/mysql/mysqld.pid

[client]
#port=3306
socket=/Users/olchansk/mysql/mysql.sock
</pre>
* mkdir $HOME/mysql
* start mysql:
<pre>
macos: /opt/local/lib/mariadb-10.4/bin/mysqld
</pre>
* configure the database: start the "mysql" utility (/opt/local/lib/mariadb-10.4/bin/mysql)
<pre>
NOTE: instructions thanks to Ben S.
NOTE: use some other passwords instead of the example "reader_password" and "writer_password"

MariaDB [(none)]> CREATE DATABASE IF NOT EXISTS history;
Query OK, 1 row affected (0.003 sec)

MariaDB [(none)]> CREATE USER 'history_reader'@'localhost' IDENTIFIED BY 'reader_password';
Query OK, 0 rows affected (0.027 sec)

MariaDB [(none)]> GRANT SELECT ON history.* TO history_reader;
Query OK, 0 rows affected (0.016 sec)

MariaDB [(none)]> CREATE USER 'history_writer'@'localhost' IDENTIFIED BY 'writer_password';
Query OK, 0 rows affected (0.014 sec)

MariaDB [(none)]> GRANT SELECT,INSERT,CREATE,ALTER ON history.* TO 'history_writer'@'localhost';
Query OK, 0 rows affected (0.017 sec)
</pre>
* try connecting to both users:
<pre>
/opt/local/lib/mariadb-10.4/bin/mysql -u history_reader ### should connect without a password
/opt/local/lib/mariadb-10.4/bin/mysql -u history_writer -p ### should ask for password
</pre>
* create midas config files:
** mysql_writer.txt
<pre>
#server=localhost
#port=xxx
database=history
socket=/Users/olchansk/mysql/mysql.sock
user=history_writer
password=writer_password ### change this!!!
#buffer=1000
</pre>
** mysql_reader.txt
<pre>
#server=localhost
#port=xxx
database=history
socket=/Users/olchansk/mysql/mysql.sock
user=history_reader
password=reader_password ### change this!!!
</pre>
* enable MYSQL history in mlogger:
** change ODB /Logger/History/MYSQL/Active to "y", "mysql writer" and "mysql reader" to location of these files (leave the default values if these files are in the experiment directory).
* enable MYSQL history in mhttpd:
** change ODB /History/LoggerHistoryChannel to "MYSQL"
* restart mlogger, restart mhttpd
...........

== Write FILE-history events ==

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "SQLITE".
* Assign directory path for saved files using the ODB key [[Keys in the ODB /Logger/History subtree#History dir|History dir]].
...........

 
---------------
 

= History Events =
The following information is derived from [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087].

A "history event" is a history atomic unit of data. Associated with each history event is a timestamp (unix time),
a name (limited to NAME_LENGTH in the old history) and a list of history tags that describe the individual data
values inside the history event.

When making history plots in [[mhttpd]] (see [[History Page]]) for each curve on the plot, one selects a history event (from the list
of currently active events, recently active events or the list of all events that ever existed), then from the list of tags
inside the history event one selects the particular variable that will be plotted.

== Old MIDAS history events ==
In the old MIDAS history, all history events are written into one history file (.hst file + optional .def and .idx event definition and time index files
which can be/are regenerated automatically from the .hst file). History events are identified by 16-bit history event IDs, the persistent mapping
from history event names and the 16-bit history event IDs is stored in ODB subtree {{Odbpath|path=/History/Events}}. In addition the list of all known history event tags is
stored in ODB subtree {{Odbpath|path=/History/Tags}}. For per-equipment history, the 16-bit history event ID is the value of ODB key [[/Equipment ODB tree#Event ID|/Equipment/<equipment-name>/Common/Event ID]].

== SQL history events ==
In the SQL history (MySQL, SQLITE, etc), each history event is an SQL table. The history event tags are the SQL table columns.

== FILE history events ==
In the new FILE history, each history event is written into a separate file, tag definition are recorded in text format in the file header, history event
data is appended to the file in binary format (fixed record size). If the history event definition is changed, a new file will be started.

= History Event Construction =
The [[mlogger]] creates [[#Frontend history event]]s in open_history() by parsing the ODB subtree {{Odbpath|path=/equipment/<equipment-name>/variables}}. Each ODB entry under the {{Odbpath|path=variables/}} subtree is referred to as a "variable". Each variable can be a single ODB value, an array of ODB values, or a subdirectory (corresponding to TID_STRUCT structured data banks). As each variable is processed, one or more tags are created to describe it.

Single ODB values will generally produce a single tag, while arrays can produce (depending whether the array is "[[#Named Arrays|Named]]" or not)
* multiple tags - one per array element ("Named" array) or
* one single tag - describing the whole array

The code can generate two types of history:
<dl>
<dt> "per-equipment" history</dt>
<dd> will have the tags for all variables concatenated together into one single history event</dd>
<dt> "per-variable" history</dt>
<dd> will have one history event defined for each variable. Inside could be one tag (for single odb values and unnamed array) or multiple tags (for named arrays and structured data
banks).</dd>
</dl>

Per-equipment history is the original MIDAS history implementation. Per-variable history was added to permit efficient data storage in SQL tables. Its initial implementation used 1 ODB hotlink for each variable and it was easy to exceed the maximum permitted number of ODB hotlinks (db_open_record()). To reduce consumption of hotlinks, db_watch() has been implemented and now per-variable history only uses 1 ODB hotlink per equipment.

;NOTE
: With db_watch, per-equipment history is no longer available. '''Per-variable history is the new default''' (and the only option).

[https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087]

= Construction of history event tags =
Currently,
* single ODB values are treated as arrays of length "1"
* string ODB entries are not permitted

History event tags are constructed in the [[mlogger]] add_equipment(). Presently there are some quirks with the generation of history event tags (see [https://midas.triumf.ca/elog/Midas/1088]),
e.g. single odb values are treated as arrays of length "1" and "string" ODB entries are not permitted.

For variables of type TID_KEY (subdirectories, corresponding to TID_STRUCT structured banks), one tag is generated for each subdirectory entry. Tag names for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa/bbb}} will be "aaa_bbb" (with an underscore). Subdirectory entries of type TID_KEY and TID_LINK are currently not supported.

For "[[#Named Arrays|Named Arrays]]", individual tags are generated for each array element. Tag names are taken from the ''Names'' array. For empty tag names (empty names array), tags are "aaa_0", "aaa_1", etc (for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}}).

For "[[#Named Arrays|Single Names Arrays]]", tag names have the variable name appended (with a space), for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}} and an empty names array, tags will be "aaa_0 aaa", "aaa_1 aaa", etc. For a populated names array, the tags will be "name0 aaa", "name1 aaa", etc.

For unnamed arrays and single odb variables (in ODB, single odb variables are arrays of length 1), a single tag is generated.

== Named Arrays ==

'''Named Arrays''' have a "Names" entry in {{Odbpath|path=/equipment/<equipment-name>/settings}}.

For example, to add names to {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}}, create a string array {{Odbpath|path=/equipment/<equipment-name>/settings/names aaa}}. The names array should be at least as long as the corresponding data array. Individual entries in the names array can be left blank (tag names will be "aaa_0", "aaa_1", etc). Duplicate tag names are not permitted.

A single "Names" entry can be created to name all arrays in variables with the same names ('''"Single Names"''').

Create {{Odbpath|path=/equipment/<equipment-name>/settings/names}} and arrays {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}} and {{Odbpath|path=/equipment/<equipment-name>/variables/bbb}} will have
history tags "name0 aaa", "name1 aaa", "name0 bbb", "name1 bbb", etc. If "names" are left blank, tag names will be "aaa_0 aaa", "aaa_1 aaa", "bbb_0 bbb", "bbb_1 bbb", etc.

In the [[mhttpd]] web server [[History Page]] variables viewer, "single name" arrays are displayed in a 2D table.

This information is from [https://midas.triumf.ca/elog/Midas/1088].

= Historical note on MIDAS-History Tag names =
Tags are only used for MIDAS-history events.

The mlogger/mhttpd implementation of /History/Tags has proved troublesome and we are moving away from it.

;During the transition period:
: Implementations starting with Version 4435 of [[mlogger]] and [[mhttpd]] (i.e. "new" versions) will now work '''without''' /History/Tags. The history tags are read directly from the history files themselves. To run in this mode, set ODB key [[/History ODB tree#DisableTags|/History/DisableTags]] to "y". Existing [[/History ODB tree#Tags subtree|/History/Tags subtree]] will be automatically deleted.

Two downsides to running like this are
* it is slower and tags become non-persistent
* if some frontends have not been running for a while, their variables may vanish from the history panel editor.

== New-style History Tags ==
For the above two reasons, using /History/Tags is still recommended.
However, the format of the tags is now changed to simplify management and reduce ODB size.

; For NEW implementations
: the key [[/History ODB tree#CreateOldTags|CreateOldTags]] will not be present. "New-style" history tags will be created automatically in the [[/History ODB tree#Tags subtree|Tags subtree]].

;If you have old-style tags
: [[mlogger]] will automatically convert the old tags to this new format (this is why you should make a backup of your ODB).

** using "old" (i.e. pre Version 4435) mlogger with "new" mhttpd is OK: new mhttpd understands both formats of /History/Tags.
** using "old" mhttpd with "new" mlogger is OK provided the ODB key [[/History ODB tree#CreateOldTags|/History/CreateOldTags]] is set to "y" before starting mlogger.

== Old-style History Tags ==
Prior to versions 4435 of [[mlogger]] and [[mhttpd]], old-style history tags were generated.
See
* http://ladd00.triumf.ca/~daqweb/doc/midas-old/html/F_History_logging.html
for details.

 
---------------
 

= MIDAS ODBC SQL History system =

This section describes the internal workings of the MIDAS SQL history system.

The SQL history system implements the MIDAS history using a SQL database for data storage instead of flat binary files.

The SQL history code lives in a separate files history.h and history_sql.cxx. The present implementation uses the ODBC API for accessing SQL databases.
MySQL is fully supported, Pgsql support is partially implemented, support for arbitrary ODBC drivers should be easy to add in history_sql and most SQL operations are done using an interface class.

Preliminary testing with MySQL and PgSQL indicates that disk space requirements are the same for both storage mechanisms using flat binary files or either of the 2 SQL databases. All 3 tested storage mechanisms store the data uncompressed. (Observed gzip-1 compression ratios from CERN-ALPHA history files are 50% or better).

Preliminary testing also indicates that MySQL database is "not too slow" for use by T2K/ND280 slow control system. Using MySQL, performance seems to improve somewhat compared to traditional MIDAS-history because each SQL table is stored into a separate file, compared to everything-in-one-file in traditional MIDAS-history.

== Set up a MySQL database ==

In order to setup a MySQL database, the following is needed:

* the server hostname and port number
* database name
* user name and password.

It is recommended to create 3 different users:

* a "root" user with full priveleges,
* a "reader" user with read-only priveleges
* and a "writer" user with "insert, create table, and add column" priveleges (no permission to drop table, remove columns, etc)

=== Example ===
#Setup a MySQL database (RHEL/SL 5)

ssh root@localhost
service mysqld start
/usr/bin/mysql_secure_installation (enter current password for root: press <enter>, then accept default answer "Y" to all questions)
mysqladmin -p create history
using mysql-administrator, create 2 users:
history_writer, with database "history" schema privileges: select, insert, create, alter, index
history_reader, with database "history" schema privileges: select

#Setup $HOME/.odbc.ini file:

cat $HOME/.odbc.ini
[history_writer]
Description = history_writer
Driver = /usr/lib64/libmyodbc3.so
SERVER = ladd05.triumf.ca
PORT =
Database = history
OPTION = 3
SOCKET =
User = history_writer
Password = ******
[history_reader]
Description = history_reader
Driver = /usr/lib64/libmyodbc3.so
SERVER = ladd05.triumf.ca
PORT =
Database = history
OPTION = 3
SOCKET =
User = history_reader
Password =

Note: if you are using mhist you will need a separate .odbc.ini file for each user that you have connecting to the database (through MIDAS).

== Layout of SQL tables ==

* One table is created for each history event. Table name is the same as the history event name (as reported by [[mhdump]]).
* one column is created for each history tag (history variable). Column name is the same as the tag name (as reported by mhdump). For array tags, one column is created for each array element: array a[3] will produce columns a_0, a_1 and a_2.
* two special columns are always created: _i_time and _t_time are the timestamps in the UNIX-time integer format and in the SQL "time" format. (A priori, it is not clear which timestamp format is more useful for end users. mhttpd uses the _i_time integer time format). Note that this timestamp is the time when mlogger receives the hot-link signal (see above) and it can be a few seconds behind the time when the user placed the data into the MIDAS bank. For high precision (and sub-second time resolution) time stamps, users should generate their own timestamps and include them as part of the data itself.
* a special table called _history_index is created to remember the mapping between SQL names and MIDAS names for event names, variable name and tag names.

=== Schema Evolution ===

* when new data fields are added, new SQL columns are created automatically
* existing SQL columns are never deleted
* arrays are expanded automatically, and arrays never shrink.
* if the type of a data field changes, a conflict may occur with existing SQL columns, for example if previous type was "FLOAT" and new type is "DOUBLE", the existing SQL columns would have the SQL data type "float", too narrow to store new values. mlogger will complain about this. Such conflicts may be resolved by changing the type of the SQL column using SQL tools. In general, SQL names are created from MIDAS names by replacing all non-alphanumerical characters to an underscore "_". If this procedure creates duplicate column names, the column names and made unique by appending them with a random number, i.e. MIDAS names "A+B" and "A-B" will be translated to "A_B" and "A_B_12345" (random integer). True MIDAS names are saved into the _history_index.

[[Category:history]]

History System

2020-07-30T16:16:29Z

Olchansk: /* Write MYSQL-history events */

{{Pagelinks}}

== Links ==
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[/History ODB tree]]
* [[History Page]]
* [[Mhformat|MIDAS history file format]]
* [[Keys in the ODB /Logger tree|/Logger ODB tree]]
* [[Keys in the ODB /Logger/History subtree|/Logger/History subtree]]
* [[Image History]]
</div>

<div id="history file"></div>
= Introduction =

The history system is an add-on capability built into the MIDAS data logger [[mlogger]] to record useful information in parallel to the data logging.
This information is recorded with a time stamp and saved into a '''history file''' (see [[#History drivers|History drivers]]) for later retrieval. One set of files is created per day containing all the requested history events. The history data may be displayed in graphical form using the [[mhttpd]] MIDAS webserver utility, giving the user an easy way of seeing how experimental variables have changed with time.
The history logging will be in action only if the MIDAS data logger is running, but it is not necessary to have any [[Keys in the ODB /Logger tree#/Logger/Channels subtree|data logging channel]] enabled.

Since June 2020, it is possible to also use the midas history system to record images (png/jpeg etc) periodically, for example to later produce a timelapse from a webcam - see the [[Image History]] page for more details.

= MidasHistoryInterface =

The new history configuration defines history channels similar to logger channels. Each history channel creates a history interface class (class MidasHistoryInterface). The [[mlogger|data logger]] can use this interface to write data into the history, mhttpd, mhist & co can use this interface to read data from the history. The interface is defined in history.h

* http://ladd00.triumf.ca/~daqweb/doc/midas/include/history.h

= History drivers =

Presently, the MIDAS distribution contains these drivers:

; MIDAS-History
: this is the traditional history driver, with data stored in the .hst binary files. (format is documented here [[mhformat]], see also [[mhist]] and [[mhdump]])
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#MIDAS subtree |"MIDAS"]]''' is set up to write MIDAS-history.

; ODBC-History
: stores history data in SQL database using the ODBC abstraction layer. See [[#MIDAS ODBC SQL History system|ODBC SQL History system]].

: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#ODBC subtree|"ODBC"]]''' is set up to write ODBC-history.

; SQLITE-History
: stores data in SQL database as SQLITE3 .sqlite3 files. The SQL table format is similar to ODBC-history, performance is similar, no database server required.
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#SQLITE subtree|"SQLITE"]]''' is set up to write SQLITE-history.

; MYSQL-History
:
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#MYSQL subtree|"MYSQL"]]''' is set up to write MYSQL-history.

; FILE-History
: stores each variable in its own file, making it much easier to trace the history of a particular variable.
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#FILE subtree|FILE]]''' is set up to write FILE-history.

More drivers can be easily added by the user (create history_xxx.cxx, add constructor to hs_get_history(), create the ODB subtree /Logger/History/xxx/channel).

= Location of History Files =
;MIDAS-history
:By default, the traditional MIDAS-history is enabled, writing history files into the MIDAS data directory path given by the ODB key [[Keys in the ODB /Logger tree#Data dir|Data dir]].
:If history data is important, it is recommended to write it to a directory that is backed up or archived. (The MIDAS data directory is usually located on a scratch data disk without backups).

:The location of the MIDAS-history data files can be changed by use of the ODB Key [[Keys in the ODB /Logger tree#History dir|History dir]].

;ODBC-History
:The location of ODBC-History files is ...

;SQLITE-History
:The location of SQLITE-History files is the directory path given by the ODB key [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]].

;MYSQL-History
:The location of MYSQL-History files is ...

;FILE-History
:The location of FILE-History files is the directory path given by the ODB key [[Keys in the ODB /Logger/History subtree#History dir|History dir]].

= Types of History Events =

There are two basic types of history events, which are defined in different ways:

* "[[#Frontend History Event|Frontend]]" (or "Equipment") History event composed in the frontend. See [[#Frontend History Event|Frontend History Event]] for details.
* "Virtual" (or "Links") History event composed within the ODB under the [[/History ODB tree#Links subtree|/History/Links subtree]]. See [[#Virtual History Event|Virtual History Event]] for details.

Both these history event definitions take effect when the data logger gets a start run [[Run States and Transitions|transition]]. Any modifications made during the run are not applied until the start of the next run.

== Virtual History Event ==
The history of any ODB variable can be recorded in a "virtual" history event by creating a link to that variable in the [[/History ODB tree#Links subtree|/History/Links subtree]]. This is used for ODB variables that are not in an Equipment.

; To enable Virtual History Events
: Create links as desired in [[/History ODB tree#Links subtree|/History/Links subtree]].

History events are created for each entry under ODB subdirectory {{Odbpath|path=/History/Links}}.

Two types of links are permitted:
<dl>
<dt> ODB key {{Odbpath|path=/history/links/aaa}} is a link to a subdirectory</dt>
<dd><ol>
<li> db_watch() is setup to watch this subdirectory</li>
<li> tags are created for each subdirectory entry (1 tag per entry)</li>
<li> There is no possibility for naming array elements, so 1 tag per array, regardless of the number of elements.</li> </ol></dd>
<dt>ODB key {{Odbpath|path=/history/links/bbb}} is a subdirectory with links to ODB values</dt>
<dd> <ol>
<li>db_watch is setup to watch each link target</li>
<li> tags are created for each link (1 tag per link)</li>
<li> tag name is the link name (NOT the target name). There is no possibility for naming array elements</li>
</ol></dd>
</dl>

Mixing links and subdirectories is not permitted.

The update period of history events created for {{Odbpath|path=/history/links}} is controlled by entries in {{Odbpath|path=/history/links periods}}. Numeric values of periods are same as for a [[#Frontend History Event|Frontend History event]]. Numeric value 0 disables the history for a particular event.

See [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087] for more information.

== Frontend History Event ==

Each [[Frontend Operation#Equipments and Events|Equipment]] has the capability to generate "history data".

; To enable Frontend History Events

: Set the [[/Equipment ODB tree#Log history|Log history]] ODB key of each equipment to be logged '''non-zero'''. (See [[Equipment List Parameters#Log History|Log History parameter]]). (You may need to restart the frontend.)

This causes the event contents to be copied to the ODB [[/Equipment ODB tree#Variables subtree|/Equipment/''<equipment-name>''/Variables subtree]] for the use of the History system. history events are created by parsing the content of {{Odbpath|path=/Equipment/<equipment-name>/variables}}.

For each equipment, history is controlled by the value of {{Odbpath|path=/Equipment/<equipment-name>/common/period}} as follows:
* 0 = history disabled
* 1 = history is enabled
* >1 = history is enabled, throttled down

The throttling is implemented in log_history()/watch_history() by this algorithm:
the very first history event is recorded, then all changed to the data are ignored until "period" seconds has elapsed. Then the next history event will be recorded, and following
changes will be ignored until "period" second elapses, and so forth. Period value "1" has special meaning - there is no throttling, all history events are logged.

This information is from [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087].

=== Sequence for frontend History Event ===
The main steps for the frontend History Event is as follows:

* The user code in the frontend equipment Event Readout routines reads the data, placing it into a MIDAS data bank
* In mfe.c, if the key [[/Equipment ODB tree#Log history|Log history]] > 0 (i.e. the history system is enabled for this equipment), this data bank is written into ODB (by update_odb()).
* This ODB write triggers an ODB hot-link into mlogger
* The hot-link calls mlogger.c::log_history(), which calls hs_write() to write the data into the history file. The frequency of the history writes is specified by the number of seconds between writes stored in the ODB key [[/Equipment ODB tree#Log history|Log history]].
* history.c::hs_write() or history_odbc::hs_write_odbc() writes the data into a history file or into an SQL database.

= How to Write History events =
* Enable desired [[#To enable Frontend History Events|Frontend]] and/or [[#To enable Virtual History Events|Virtual]] history events
* [[#Select the history driver|Select the history driver]]
* Follow instructions below for selected driver:
** [[#Write MIDAS-History events|Write MIDAS-history events]]
** [[#Write ODBC-History events|Write ODBC-history events]]
** [[#Write SQLITE-History events|Write SQLITE-history events]]
** [[#Write MYSQL-History events|Write MYSQL-history events]]
** [[#Write FILE-History events|Write FILE-history events]]

== Select the history driver ==

* Select the '''[[#History drivers|history driver]]''' and therefore the [[Keys in the ODB /Logger/History subtree#<history-logging-channel> subtrees|history-logging-channel]] to use to write the MIDAS-history files.
* set [[Keys in the ODB /Logger/History subtree#Active|Active]] ODB key in the selected [[Keys in the ODB /Logger/History subtree#<history-logging-channel> subtrees|history-logging-channel]] to "y". All other logging channels should be disabled.
* set [[/History ODB tree#LoggerHistoryChannel|LoggerHistoryChannel]] ODB key to the selected [[#History drivers|History channel]].

== Write MIDAS-History events ==
* Enable desired [[#To enable Frontend History Events|Frontend]] and/or [[#To enable Virtual History Events|Virtual]] history events
* Perform steps in [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "MIDAS".
* Restart [[mhttpd]] and [[mlogger]], start the run.

MIDAS-history files will be saved into the [[#Location of History Files|MIDAS-history data directory]].

== Write ODBC-History events ==

See [[#MIDAS ODBC SQL History system|MIDAS ODBC SQL History system]].
* If necessary, [[#Set up a MySQL database|set up a MySQL database]].
* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps in [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "ODBC".
* Check ODB Key [[Keys in the ODB /Logger/History subtree#Writer_ODBC_DSN|Writer_ODBC_DSN]] is set to "history writer" and Key [[Keys in the ODB /Logger/History subtree#Reader_ODBC_DSN|Reader_ODBC_DSN]] is set to "history reader" (the default values)
* set [[Keys in the ODB /Logger/History subtree#Debug|Debug]] to 1 if desired
* restart [[mlogger]] in verbose mode (i.e. mlogger -v), and observe how it issues SQL commands to create the tables and columns corresponding to MIDAS-history events and tags.
* restart [[mhttpd]]

'''Notes'''
# [[mlogger]] is programmed to raise alarms if connection to SQL database is interrupted or if some events cannot be written into the database (i.e. data type mismatch, SQL syntax errors, etc). The traditional MIDAS-history never raised alarms because it "never failed" - other than from "disk full" errors, which are immediately obvious.
# With ODBC History enabled, [[mhttpd]] will only use history information from the SQL database to make history plots and to extract history variable names for the history plot editor. The ODB {{Odbpath|path=/History}} subtrees[[/History ODB tree#Tags subtree|Tags]] and [[/History ODB tree#Events subtree|Events]] are not used.
# variable names presented to the user may change from MIDAS names to SQL names (use the history panel "Label" text fields to create permanent plot labels).

== Write SQLITE-history events ==

No special configuration required.

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "SQLITE".

* Create a directory for storage of sqlite history files
* save its path in the ODB key [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]]
* restart [[mhttpd]] and [[mlogger]]

In the history data directory (in [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]]) you should see files with name "mh_xxx.sqlite3" corresponding to your equipment and variable names.

== Write MYSQL-history events ==

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "MYSQL".
* setup the MySQL database:
* install mariadb-10.4
<pre>
macos:
sudo port install mariadb-10.4
ln -s ln -fs /opt/local/lib/mariadb-10.4/bin/mysql_config ~/bin/
mysql_config --version
10.4.13
</pre>
* rebuild midas, check the cmake finds the correct mysql: MIDAS: Found MySQL version 10.4.13
* create $HOME/.my.cnf
<pre>
[mysqld]
#port=3306
socket=/Users/olchansk/mysql/mysql.sock
datadir=/Users/olchansk/mysql/db
pid-file=/Users/olchansk/mysql/mysqld.pid

[client]
#port=3306
socket=/Users/olchansk/mysql/mysql.sock
</pre>
* mkdir $HOME/mysql
* start mysql:
<pre>
macos: /opt/local/lib/mariadb-10.4/bin/mysqld
</pre>
* configure the database: start the "mysql" utility (/opt/local/lib/mariadb-10.4/bin/mysql)
<pre>
NOTE: instructions thanks to Ben S.
NOTE: use your own password instead of "writer_password" in this example

MariaDB [(none)]> CREATE DATABASE IF NOT EXISTS history;
Query OK, 1 row affected (0.003 sec)

MariaDB [(none)]> CREATE USER history_reader;
Query OK, 0 rows affected (0.027 sec)

MariaDB [(none)]> GRANT SELECT ON history.* TO history_reader;
Query OK, 0 rows affected (0.016 sec)

MariaDB [(none)]> CREATE USER 'history_writer'@'localhost' IDENTIFIED BY 'writer_password';
Query OK, 0 rows affected (0.014 sec)

MariaDB [(none)]> GRANT SELECT,INSERT,CREATE,ALTER ON history.* TO 'history_writer'@'localhost';
Query OK, 0 rows affected (0.017 sec)
</pre>
* try connecting to both users:
<pre>
/opt/local/lib/mariadb-10.4/bin/mysql -u history_reader ### should connect without a password
/opt/local/lib/mariadb-10.4/bin/mysql -u history_writer -p ### should ask for password
</pre>
* create midas config files:
** mysql_writer.txt
<pre>
#server=localhost
#port=xxx
database=history
socket=/Users/olchansk/mysql/mysql.sock
user=history_writer
password=writer_password ### change this!!!
#buffer=1000
</pre>
** mysql_reader.txt
<pre>
#server=localhost
#port=xxx
database=history
socket=/Users/olchansk/mysql/mysql.sock
user=history_reader
password=reader_password ### change this!!!
</pre>
* enable MYSQL history in mlogger:
** change ODB /Logger/History/MYSQL/Active to "y", "mysql writer" and "mysql reader" to location of these files (leave the default values if these files are in the experiment directory).
* enable MYSQL history in mhttpd:
** change ODB /History/LoggerHistoryChannel to "MYSQL"
* restart mlogger, restart mhttpd
...........

== Write FILE-history events ==

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "SQLITE".
* Assign directory path for saved files using the ODB key [[Keys in the ODB /Logger/History subtree#History dir|History dir]].
...........

 
---------------
 

= History Events =
The following information is derived from [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087].

A "history event" is a history atomic unit of data. Associated with each history event is a timestamp (unix time),
a name (limited to NAME_LENGTH in the old history) and a list of history tags that describe the individual data
values inside the history event.

When making history plots in [[mhttpd]] (see [[History Page]]) for each curve on the plot, one selects a history event (from the list
of currently active events, recently active events or the list of all events that ever existed), then from the list of tags
inside the history event one selects the particular variable that will be plotted.

== Old MIDAS history events ==
In the old MIDAS history, all history events are written into one history file (.hst file + optional .def and .idx event definition and time index files
which can be/are regenerated automatically from the .hst file). History events are identified by 16-bit history event IDs, the persistent mapping
from history event names and the 16-bit history event IDs is stored in ODB subtree {{Odbpath|path=/History/Events}}. In addition the list of all known history event tags is
stored in ODB subtree {{Odbpath|path=/History/Tags}}. For per-equipment history, the 16-bit history event ID is the value of ODB key [[/Equipment ODB tree#Event ID|/Equipment/<equipment-name>/Common/Event ID]].

== SQL history events ==
In the SQL history (MySQL, SQLITE, etc), each history event is an SQL table. The history event tags are the SQL table columns.

== FILE history events ==
In the new FILE history, each history event is written into a separate file, tag definition are recorded in text format in the file header, history event
data is appended to the file in binary format (fixed record size). If the history event definition is changed, a new file will be started.

= History Event Construction =
The [[mlogger]] creates [[#Frontend history event]]s in open_history() by parsing the ODB subtree {{Odbpath|path=/equipment/<equipment-name>/variables}}. Each ODB entry under the {{Odbpath|path=variables/}} subtree is referred to as a "variable". Each variable can be a single ODB value, an array of ODB values, or a subdirectory (corresponding to TID_STRUCT structured data banks). As each variable is processed, one or more tags are created to describe it.

Single ODB values will generally produce a single tag, while arrays can produce (depending whether the array is "[[#Named Arrays|Named]]" or not)
* multiple tags - one per array element ("Named" array) or
* one single tag - describing the whole array

The code can generate two types of history:
<dl>
<dt> "per-equipment" history</dt>
<dd> will have the tags for all variables concatenated together into one single history event</dd>
<dt> "per-variable" history</dt>
<dd> will have one history event defined for each variable. Inside could be one tag (for single odb values and unnamed array) or multiple tags (for named arrays and structured data
banks).</dd>
</dl>

Per-equipment history is the original MIDAS history implementation. Per-variable history was added to permit efficient data storage in SQL tables. Its initial implementation used 1 ODB hotlink for each variable and it was easy to exceed the maximum permitted number of ODB hotlinks (db_open_record()). To reduce consumption of hotlinks, db_watch() has been implemented and now per-variable history only uses 1 ODB hotlink per equipment.

;NOTE
: With db_watch, per-equipment history is no longer available. '''Per-variable history is the new default''' (and the only option).

[https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087]

= Construction of history event tags =
Currently,
* single ODB values are treated as arrays of length "1"
* string ODB entries are not permitted

History event tags are constructed in the [[mlogger]] add_equipment(). Presently there are some quirks with the generation of history event tags (see [https://midas.triumf.ca/elog/Midas/1088]),
e.g. single odb values are treated as arrays of length "1" and "string" ODB entries are not permitted.

For variables of type TID_KEY (subdirectories, corresponding to TID_STRUCT structured banks), one tag is generated for each subdirectory entry. Tag names for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa/bbb}} will be "aaa_bbb" (with an underscore). Subdirectory entries of type TID_KEY and TID_LINK are currently not supported.

For "[[#Named Arrays|Named Arrays]]", individual tags are generated for each array element. Tag names are taken from the ''Names'' array. For empty tag names (empty names array), tags are "aaa_0", "aaa_1", etc (for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}}).

For "[[#Named Arrays|Single Names Arrays]]", tag names have the variable name appended (with a space), for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}} and an empty names array, tags will be "aaa_0 aaa", "aaa_1 aaa", etc. For a populated names array, the tags will be "name0 aaa", "name1 aaa", etc.

For unnamed arrays and single odb variables (in ODB, single odb variables are arrays of length 1), a single tag is generated.

== Named Arrays ==

'''Named Arrays''' have a "Names" entry in {{Odbpath|path=/equipment/<equipment-name>/settings}}.

For example, to add names to {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}}, create a string array {{Odbpath|path=/equipment/<equipment-name>/settings/names aaa}}. The names array should be at least as long as the corresponding data array. Individual entries in the names array can be left blank (tag names will be "aaa_0", "aaa_1", etc). Duplicate tag names are not permitted.

A single "Names" entry can be created to name all arrays in variables with the same names ('''"Single Names"''').

Create {{Odbpath|path=/equipment/<equipment-name>/settings/names}} and arrays {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}} and {{Odbpath|path=/equipment/<equipment-name>/variables/bbb}} will have
history tags "name0 aaa", "name1 aaa", "name0 bbb", "name1 bbb", etc. If "names" are left blank, tag names will be "aaa_0 aaa", "aaa_1 aaa", "bbb_0 bbb", "bbb_1 bbb", etc.

In the [[mhttpd]] web server [[History Page]] variables viewer, "single name" arrays are displayed in a 2D table.

This information is from [https://midas.triumf.ca/elog/Midas/1088].

= Historical note on MIDAS-History Tag names =
Tags are only used for MIDAS-history events.

The mlogger/mhttpd implementation of /History/Tags has proved troublesome and we are moving away from it.

;During the transition period:
: Implementations starting with Version 4435 of [[mlogger]] and [[mhttpd]] (i.e. "new" versions) will now work '''without''' /History/Tags. The history tags are read directly from the history files themselves. To run in this mode, set ODB key [[/History ODB tree#DisableTags|/History/DisableTags]] to "y". Existing [[/History ODB tree#Tags subtree|/History/Tags subtree]] will be automatically deleted.

Two downsides to running like this are
* it is slower and tags become non-persistent
* if some frontends have not been running for a while, their variables may vanish from the history panel editor.

== New-style History Tags ==
For the above two reasons, using /History/Tags is still recommended.
However, the format of the tags is now changed to simplify management and reduce ODB size.

; For NEW implementations
: the key [[/History ODB tree#CreateOldTags|CreateOldTags]] will not be present. "New-style" history tags will be created automatically in the [[/History ODB tree#Tags subtree|Tags subtree]].

;If you have old-style tags
: [[mlogger]] will automatically convert the old tags to this new format (this is why you should make a backup of your ODB).

** using "old" (i.e. pre Version 4435) mlogger with "new" mhttpd is OK: new mhttpd understands both formats of /History/Tags.
** using "old" mhttpd with "new" mlogger is OK provided the ODB key [[/History ODB tree#CreateOldTags|/History/CreateOldTags]] is set to "y" before starting mlogger.

== Old-style History Tags ==
Prior to versions 4435 of [[mlogger]] and [[mhttpd]], old-style history tags were generated.
See
* http://ladd00.triumf.ca/~daqweb/doc/midas-old/html/F_History_logging.html
for details.

 
---------------
 

= MIDAS ODBC SQL History system =

This section describes the internal workings of the MIDAS SQL history system.

The SQL history system implements the MIDAS history using a SQL database for data storage instead of flat binary files.

The SQL history code lives in a separate files history.h and history_sql.cxx. The present implementation uses the ODBC API for accessing SQL databases.
MySQL is fully supported, Pgsql support is partially implemented, support for arbitrary ODBC drivers should be easy to add in history_sql and most SQL operations are done using an interface class.

Preliminary testing with MySQL and PgSQL indicates that disk space requirements are the same for both storage mechanisms using flat binary files or either of the 2 SQL databases. All 3 tested storage mechanisms store the data uncompressed. (Observed gzip-1 compression ratios from CERN-ALPHA history files are 50% or better).

Preliminary testing also indicates that MySQL database is "not too slow" for use by T2K/ND280 slow control system. Using MySQL, performance seems to improve somewhat compared to traditional MIDAS-history because each SQL table is stored into a separate file, compared to everything-in-one-file in traditional MIDAS-history.

== Set up a MySQL database ==

In order to setup a MySQL database, the following is needed:

* the server hostname and port number
* database name
* user name and password.

It is recommended to create 3 different users:

* a "root" user with full priveleges,
* a "reader" user with read-only priveleges
* and a "writer" user with "insert, create table, and add column" priveleges (no permission to drop table, remove columns, etc)

=== Example ===
#Setup a MySQL database (RHEL/SL 5)

ssh root@localhost
service mysqld start
/usr/bin/mysql_secure_installation (enter current password for root: press <enter>, then accept default answer "Y" to all questions)
mysqladmin -p create history
using mysql-administrator, create 2 users:
history_writer, with database "history" schema privileges: select, insert, create, alter, index
history_reader, with database "history" schema privileges: select

#Setup $HOME/.odbc.ini file:

cat $HOME/.odbc.ini
[history_writer]
Description = history_writer
Driver = /usr/lib64/libmyodbc3.so
SERVER = ladd05.triumf.ca
PORT =
Database = history
OPTION = 3
SOCKET =
User = history_writer
Password = ******
[history_reader]
Description = history_reader
Driver = /usr/lib64/libmyodbc3.so
SERVER = ladd05.triumf.ca
PORT =
Database = history
OPTION = 3
SOCKET =
User = history_reader
Password =

Note: if you are using mhist you will need a separate .odbc.ini file for each user that you have connecting to the database (through MIDAS).

== Layout of SQL tables ==

* One table is created for each history event. Table name is the same as the history event name (as reported by [[mhdump]]).
* one column is created for each history tag (history variable). Column name is the same as the tag name (as reported by mhdump). For array tags, one column is created for each array element: array a[3] will produce columns a_0, a_1 and a_2.
* two special columns are always created: _i_time and _t_time are the timestamps in the UNIX-time integer format and in the SQL "time" format. (A priori, it is not clear which timestamp format is more useful for end users. mhttpd uses the _i_time integer time format). Note that this timestamp is the time when mlogger receives the hot-link signal (see above) and it can be a few seconds behind the time when the user placed the data into the MIDAS bank. For high precision (and sub-second time resolution) time stamps, users should generate their own timestamps and include them as part of the data itself.
* a special table called _history_index is created to remember the mapping between SQL names and MIDAS names for event names, variable name and tag names.

=== Schema Evolution ===

* when new data fields are added, new SQL columns are created automatically
* existing SQL columns are never deleted
* arrays are expanded automatically, and arrays never shrink.
* if the type of a data field changes, a conflict may occur with existing SQL columns, for example if previous type was "FLOAT" and new type is "DOUBLE", the existing SQL columns would have the SQL data type "float", too narrow to store new values. mlogger will complain about this. Such conflicts may be resolved by changing the type of the SQL column using SQL tools. In general, SQL names are created from MIDAS names by replacing all non-alphanumerical characters to an underscore "_". If this procedure creates duplicate column names, the column names and made unique by appending them with a random number, i.e. MIDAS names "A+B" and "A-B" will be translated to "A_B" and "A_B_12345" (random integer). True MIDAS names are saved into the _history_index.

[[Category:history]]

History System

2020-07-30T16:15:57Z

Olchansk: /* Write MYSQL-history events */

{{Pagelinks}}

== Links ==
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[/History ODB tree]]
* [[History Page]]
* [[Mhformat|MIDAS history file format]]
* [[Keys in the ODB /Logger tree|/Logger ODB tree]]
* [[Keys in the ODB /Logger/History subtree|/Logger/History subtree]]
* [[Image History]]
</div>

<div id="history file"></div>
= Introduction =

The history system is an add-on capability built into the MIDAS data logger [[mlogger]] to record useful information in parallel to the data logging.
This information is recorded with a time stamp and saved into a '''history file''' (see [[#History drivers|History drivers]]) for later retrieval. One set of files is created per day containing all the requested history events. The history data may be displayed in graphical form using the [[mhttpd]] MIDAS webserver utility, giving the user an easy way of seeing how experimental variables have changed with time.
The history logging will be in action only if the MIDAS data logger is running, but it is not necessary to have any [[Keys in the ODB /Logger tree#/Logger/Channels subtree|data logging channel]] enabled.

Since June 2020, it is possible to also use the midas history system to record images (png/jpeg etc) periodically, for example to later produce a timelapse from a webcam - see the [[Image History]] page for more details.

= MidasHistoryInterface =

The new history configuration defines history channels similar to logger channels. Each history channel creates a history interface class (class MidasHistoryInterface). The [[mlogger|data logger]] can use this interface to write data into the history, mhttpd, mhist & co can use this interface to read data from the history. The interface is defined in history.h

* http://ladd00.triumf.ca/~daqweb/doc/midas/include/history.h

= History drivers =

Presently, the MIDAS distribution contains these drivers:

; MIDAS-History
: this is the traditional history driver, with data stored in the .hst binary files. (format is documented here [[mhformat]], see also [[mhist]] and [[mhdump]])
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#MIDAS subtree |"MIDAS"]]''' is set up to write MIDAS-history.

; ODBC-History
: stores history data in SQL database using the ODBC abstraction layer. See [[#MIDAS ODBC SQL History system|ODBC SQL History system]].

: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#ODBC subtree|"ODBC"]]''' is set up to write ODBC-history.

; SQLITE-History
: stores data in SQL database as SQLITE3 .sqlite3 files. The SQL table format is similar to ODBC-history, performance is similar, no database server required.
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#SQLITE subtree|"SQLITE"]]''' is set up to write SQLITE-history.

; MYSQL-History
:
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#MYSQL subtree|"MYSQL"]]''' is set up to write MYSQL-history.

; FILE-History
: stores each variable in its own file, making it much easier to trace the history of a particular variable.
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#FILE subtree|FILE]]''' is set up to write FILE-history.

More drivers can be easily added by the user (create history_xxx.cxx, add constructor to hs_get_history(), create the ODB subtree /Logger/History/xxx/channel).

= Location of History Files =
;MIDAS-history
:By default, the traditional MIDAS-history is enabled, writing history files into the MIDAS data directory path given by the ODB key [[Keys in the ODB /Logger tree#Data dir|Data dir]].
:If history data is important, it is recommended to write it to a directory that is backed up or archived. (The MIDAS data directory is usually located on a scratch data disk without backups).

:The location of the MIDAS-history data files can be changed by use of the ODB Key [[Keys in the ODB /Logger tree#History dir|History dir]].

;ODBC-History
:The location of ODBC-History files is ...

;SQLITE-History
:The location of SQLITE-History files is the directory path given by the ODB key [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]].

;MYSQL-History
:The location of MYSQL-History files is ...

;FILE-History
:The location of FILE-History files is the directory path given by the ODB key [[Keys in the ODB /Logger/History subtree#History dir|History dir]].

= Types of History Events =

There are two basic types of history events, which are defined in different ways:

* "[[#Frontend History Event|Frontend]]" (or "Equipment") History event composed in the frontend. See [[#Frontend History Event|Frontend History Event]] for details.
* "Virtual" (or "Links") History event composed within the ODB under the [[/History ODB tree#Links subtree|/History/Links subtree]]. See [[#Virtual History Event|Virtual History Event]] for details.

Both these history event definitions take effect when the data logger gets a start run [[Run States and Transitions|transition]]. Any modifications made during the run are not applied until the start of the next run.

== Virtual History Event ==
The history of any ODB variable can be recorded in a "virtual" history event by creating a link to that variable in the [[/History ODB tree#Links subtree|/History/Links subtree]]. This is used for ODB variables that are not in an Equipment.

; To enable Virtual History Events
: Create links as desired in [[/History ODB tree#Links subtree|/History/Links subtree]].

History events are created for each entry under ODB subdirectory {{Odbpath|path=/History/Links}}.

Two types of links are permitted:
<dl>
<dt> ODB key {{Odbpath|path=/history/links/aaa}} is a link to a subdirectory</dt>
<dd><ol>
<li> db_watch() is setup to watch this subdirectory</li>
<li> tags are created for each subdirectory entry (1 tag per entry)</li>
<li> There is no possibility for naming array elements, so 1 tag per array, regardless of the number of elements.</li> </ol></dd>
<dt>ODB key {{Odbpath|path=/history/links/bbb}} is a subdirectory with links to ODB values</dt>
<dd> <ol>
<li>db_watch is setup to watch each link target</li>
<li> tags are created for each link (1 tag per link)</li>
<li> tag name is the link name (NOT the target name). There is no possibility for naming array elements</li>
</ol></dd>
</dl>

Mixing links and subdirectories is not permitted.

The update period of history events created for {{Odbpath|path=/history/links}} is controlled by entries in {{Odbpath|path=/history/links periods}}. Numeric values of periods are same as for a [[#Frontend History Event|Frontend History event]]. Numeric value 0 disables the history for a particular event.

See [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087] for more information.

== Frontend History Event ==

Each [[Frontend Operation#Equipments and Events|Equipment]] has the capability to generate "history data".

; To enable Frontend History Events

: Set the [[/Equipment ODB tree#Log history|Log history]] ODB key of each equipment to be logged '''non-zero'''. (See [[Equipment List Parameters#Log History|Log History parameter]]). (You may need to restart the frontend.)

This causes the event contents to be copied to the ODB [[/Equipment ODB tree#Variables subtree|/Equipment/''<equipment-name>''/Variables subtree]] for the use of the History system. history events are created by parsing the content of {{Odbpath|path=/Equipment/<equipment-name>/variables}}.

For each equipment, history is controlled by the value of {{Odbpath|path=/Equipment/<equipment-name>/common/period}} as follows:
* 0 = history disabled
* 1 = history is enabled
* >1 = history is enabled, throttled down

The throttling is implemented in log_history()/watch_history() by this algorithm:
the very first history event is recorded, then all changed to the data are ignored until "period" seconds has elapsed. Then the next history event will be recorded, and following
changes will be ignored until "period" second elapses, and so forth. Period value "1" has special meaning - there is no throttling, all history events are logged.

This information is from [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087].

=== Sequence for frontend History Event ===
The main steps for the frontend History Event is as follows:

* The user code in the frontend equipment Event Readout routines reads the data, placing it into a MIDAS data bank
* In mfe.c, if the key [[/Equipment ODB tree#Log history|Log history]] > 0 (i.e. the history system is enabled for this equipment), this data bank is written into ODB (by update_odb()).
* This ODB write triggers an ODB hot-link into mlogger
* The hot-link calls mlogger.c::log_history(), which calls hs_write() to write the data into the history file. The frequency of the history writes is specified by the number of seconds between writes stored in the ODB key [[/Equipment ODB tree#Log history|Log history]].
* history.c::hs_write() or history_odbc::hs_write_odbc() writes the data into a history file or into an SQL database.

= How to Write History events =
* Enable desired [[#To enable Frontend History Events|Frontend]] and/or [[#To enable Virtual History Events|Virtual]] history events
* [[#Select the history driver|Select the history driver]]
* Follow instructions below for selected driver:
** [[#Write MIDAS-History events|Write MIDAS-history events]]
** [[#Write ODBC-History events|Write ODBC-history events]]
** [[#Write SQLITE-History events|Write SQLITE-history events]]
** [[#Write MYSQL-History events|Write MYSQL-history events]]
** [[#Write FILE-History events|Write FILE-history events]]

== Select the history driver ==

* Select the '''[[#History drivers|history driver]]''' and therefore the [[Keys in the ODB /Logger/History subtree#<history-logging-channel> subtrees|history-logging-channel]] to use to write the MIDAS-history files.
* set [[Keys in the ODB /Logger/History subtree#Active|Active]] ODB key in the selected [[Keys in the ODB /Logger/History subtree#<history-logging-channel> subtrees|history-logging-channel]] to "y". All other logging channels should be disabled.
* set [[/History ODB tree#LoggerHistoryChannel|LoggerHistoryChannel]] ODB key to the selected [[#History drivers|History channel]].

== Write MIDAS-History events ==
* Enable desired [[#To enable Frontend History Events|Frontend]] and/or [[#To enable Virtual History Events|Virtual]] history events
* Perform steps in [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "MIDAS".
* Restart [[mhttpd]] and [[mlogger]], start the run.

MIDAS-history files will be saved into the [[#Location of History Files|MIDAS-history data directory]].

== Write ODBC-History events ==

See [[#MIDAS ODBC SQL History system|MIDAS ODBC SQL History system]].
* If necessary, [[#Set up a MySQL database|set up a MySQL database]].
* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps in [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "ODBC".
* Check ODB Key [[Keys in the ODB /Logger/History subtree#Writer_ODBC_DSN|Writer_ODBC_DSN]] is set to "history writer" and Key [[Keys in the ODB /Logger/History subtree#Reader_ODBC_DSN|Reader_ODBC_DSN]] is set to "history reader" (the default values)
* set [[Keys in the ODB /Logger/History subtree#Debug|Debug]] to 1 if desired
* restart [[mlogger]] in verbose mode (i.e. mlogger -v), and observe how it issues SQL commands to create the tables and columns corresponding to MIDAS-history events and tags.
* restart [[mhttpd]]

'''Notes'''
# [[mlogger]] is programmed to raise alarms if connection to SQL database is interrupted or if some events cannot be written into the database (i.e. data type mismatch, SQL syntax errors, etc). The traditional MIDAS-history never raised alarms because it "never failed" - other than from "disk full" errors, which are immediately obvious.
# With ODBC History enabled, [[mhttpd]] will only use history information from the SQL database to make history plots and to extract history variable names for the history plot editor. The ODB {{Odbpath|path=/History}} subtrees[[/History ODB tree#Tags subtree|Tags]] and [[/History ODB tree#Events subtree|Events]] are not used.
# variable names presented to the user may change from MIDAS names to SQL names (use the history panel "Label" text fields to create permanent plot labels).

== Write SQLITE-history events ==

No special configuration required.

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "SQLITE".

* Create a directory for storage of sqlite history files
* save its path in the ODB key [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]]
* restart [[mhttpd]] and [[mlogger]]

In the history data directory (in [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]]) you should see files with name "mh_xxx.sqlite3" corresponding to your equipment and variable names.

== Write MYSQL-history events ==

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "MYSQL".
* setup the MySQL database:
* install mariadb-10.4
<pre>
macos:
sudo port install mariadb-10.4
ln -s ln -fs /opt/local/lib/mariadb-10.4/bin/mysql_config ~/bin/
mysql_config --version
10.4.13
</pre>
* rebuild midas, check the cmake finds the correct mysql: MIDAS: Found MySQL version 10.4.13
* create $HOME/.my.cnf
<pre>
[mysqld]
#port=3306
socket=/Users/olchansk/mysql/mysql.sock
datadir=/Users/olchansk/mysql/db
pid-file=/Users/olchansk/mysql/mysqld.pid

[client]
#port=3306
socket=/Users/olchansk/mysql/mysql.sock
</pre>
* mkdir $HOME/mysql
* start mysql:
<pre>
macos: /opt/local/lib/mariadb-10.4/bin/mysqld
</pre>
* configure the database: start the "mysql" utility (/opt/local/lib/mariadb-10.4/bin/mysql)
<pre>
NOTE: instructions thanks to Ben S.
NOTE: use your own password instead of "writer_password" in this example

MariaDB [(none)]> CREATE DATABASE IF NOT EXISTS history;
Query OK, 1 row affected (0.003 sec)

MariaDB [(none)]> CREATE USER history_reader;
Query OK, 0 rows affected (0.027 sec)

MariaDB [(none)]> GRANT SELECT ON history.* TO history_reader;
Query OK, 0 rows affected (0.016 sec)

MariaDB [(none)]> CREATE USER 'history_writer'@'localhost' IDENTIFIED BY 'writer_password';
Query OK, 0 rows affected (0.014 sec)

MariaDB [(none)]> GRANT SELECT,INSERT,CREATE,ALTER ON history.* TO 'history_writer'@'localhost';
Query OK, 0 rows affected (0.017 sec)
</pre>
* try connecting to both users:
<pre>
/opt/local/lib/mariadb-10.4/bin/mysql -u history_reader ### should connect without a password
/opt/local/lib/mariadb-10.4/bin/mysql -u history_writer -p ### should ask for password
</pre>
* create midas config files:
** mysql_writer.txt
<pre>
#server=localhost
#port=xxx
database=history
socket=/Users/olchansk/mysql/mysql.sock
user=writer
password=writer_password ### change this!!!
#buffer=1000
</pre>
** mysql_reader.txt
<pre>
#server=localhost
#port=xxx
database=history
socket=/Users/olchansk/mysql/mysql.sock
user=reader
password=reader_password ### change this!!!
</pre>
* enable MYSQL history in mlogger:
** change ODB /Logger/History/MYSQL/Active to "y", "mysql writer" and "mysql reader" to location of these files (leave the default values if these files are in the experiment directory).
* enable MYSQL history in mhttpd:
** change ODB /History/LoggerHistoryChannel to "MYSQL"
* restart mlogger, restart mhttpd
...........

== Write FILE-history events ==

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "SQLITE".
* Assign directory path for saved files using the ODB key [[Keys in the ODB /Logger/History subtree#History dir|History dir]].
...........

 
---------------
 

= History Events =
The following information is derived from [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087].

A "history event" is a history atomic unit of data. Associated with each history event is a timestamp (unix time),
a name (limited to NAME_LENGTH in the old history) and a list of history tags that describe the individual data
values inside the history event.

When making history plots in [[mhttpd]] (see [[History Page]]) for each curve on the plot, one selects a history event (from the list
of currently active events, recently active events or the list of all events that ever existed), then from the list of tags
inside the history event one selects the particular variable that will be plotted.

== Old MIDAS history events ==
In the old MIDAS history, all history events are written into one history file (.hst file + optional .def and .idx event definition and time index files
which can be/are regenerated automatically from the .hst file). History events are identified by 16-bit history event IDs, the persistent mapping
from history event names and the 16-bit history event IDs is stored in ODB subtree {{Odbpath|path=/History/Events}}. In addition the list of all known history event tags is
stored in ODB subtree {{Odbpath|path=/History/Tags}}. For per-equipment history, the 16-bit history event ID is the value of ODB key [[/Equipment ODB tree#Event ID|/Equipment/<equipment-name>/Common/Event ID]].

== SQL history events ==
In the SQL history (MySQL, SQLITE, etc), each history event is an SQL table. The history event tags are the SQL table columns.

== FILE history events ==
In the new FILE history, each history event is written into a separate file, tag definition are recorded in text format in the file header, history event
data is appended to the file in binary format (fixed record size). If the history event definition is changed, a new file will be started.

= History Event Construction =
The [[mlogger]] creates [[#Frontend history event]]s in open_history() by parsing the ODB subtree {{Odbpath|path=/equipment/<equipment-name>/variables}}. Each ODB entry under the {{Odbpath|path=variables/}} subtree is referred to as a "variable". Each variable can be a single ODB value, an array of ODB values, or a subdirectory (corresponding to TID_STRUCT structured data banks). As each variable is processed, one or more tags are created to describe it.

Single ODB values will generally produce a single tag, while arrays can produce (depending whether the array is "[[#Named Arrays|Named]]" or not)
* multiple tags - one per array element ("Named" array) or
* one single tag - describing the whole array

The code can generate two types of history:
<dl>
<dt> "per-equipment" history</dt>
<dd> will have the tags for all variables concatenated together into one single history event</dd>
<dt> "per-variable" history</dt>
<dd> will have one history event defined for each variable. Inside could be one tag (for single odb values and unnamed array) or multiple tags (for named arrays and structured data
banks).</dd>
</dl>

Per-equipment history is the original MIDAS history implementation. Per-variable history was added to permit efficient data storage in SQL tables. Its initial implementation used 1 ODB hotlink for each variable and it was easy to exceed the maximum permitted number of ODB hotlinks (db_open_record()). To reduce consumption of hotlinks, db_watch() has been implemented and now per-variable history only uses 1 ODB hotlink per equipment.

;NOTE
: With db_watch, per-equipment history is no longer available. '''Per-variable history is the new default''' (and the only option).

[https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087]

= Construction of history event tags =
Currently,
* single ODB values are treated as arrays of length "1"
* string ODB entries are not permitted

History event tags are constructed in the [[mlogger]] add_equipment(). Presently there are some quirks with the generation of history event tags (see [https://midas.triumf.ca/elog/Midas/1088]),
e.g. single odb values are treated as arrays of length "1" and "string" ODB entries are not permitted.

For variables of type TID_KEY (subdirectories, corresponding to TID_STRUCT structured banks), one tag is generated for each subdirectory entry. Tag names for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa/bbb}} will be "aaa_bbb" (with an underscore). Subdirectory entries of type TID_KEY and TID_LINK are currently not supported.

For "[[#Named Arrays|Named Arrays]]", individual tags are generated for each array element. Tag names are taken from the ''Names'' array. For empty tag names (empty names array), tags are "aaa_0", "aaa_1", etc (for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}}).

For "[[#Named Arrays|Single Names Arrays]]", tag names have the variable name appended (with a space), for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}} and an empty names array, tags will be "aaa_0 aaa", "aaa_1 aaa", etc. For a populated names array, the tags will be "name0 aaa", "name1 aaa", etc.

For unnamed arrays and single odb variables (in ODB, single odb variables are arrays of length 1), a single tag is generated.

== Named Arrays ==

'''Named Arrays''' have a "Names" entry in {{Odbpath|path=/equipment/<equipment-name>/settings}}.

For example, to add names to {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}}, create a string array {{Odbpath|path=/equipment/<equipment-name>/settings/names aaa}}. The names array should be at least as long as the corresponding data array. Individual entries in the names array can be left blank (tag names will be "aaa_0", "aaa_1", etc). Duplicate tag names are not permitted.

A single "Names" entry can be created to name all arrays in variables with the same names ('''"Single Names"''').

Create {{Odbpath|path=/equipment/<equipment-name>/settings/names}} and arrays {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}} and {{Odbpath|path=/equipment/<equipment-name>/variables/bbb}} will have
history tags "name0 aaa", "name1 aaa", "name0 bbb", "name1 bbb", etc. If "names" are left blank, tag names will be "aaa_0 aaa", "aaa_1 aaa", "bbb_0 bbb", "bbb_1 bbb", etc.

In the [[mhttpd]] web server [[History Page]] variables viewer, "single name" arrays are displayed in a 2D table.

This information is from [https://midas.triumf.ca/elog/Midas/1088].

= Historical note on MIDAS-History Tag names =
Tags are only used for MIDAS-history events.

The mlogger/mhttpd implementation of /History/Tags has proved troublesome and we are moving away from it.

;During the transition period:
: Implementations starting with Version 4435 of [[mlogger]] and [[mhttpd]] (i.e. "new" versions) will now work '''without''' /History/Tags. The history tags are read directly from the history files themselves. To run in this mode, set ODB key [[/History ODB tree#DisableTags|/History/DisableTags]] to "y". Existing [[/History ODB tree#Tags subtree|/History/Tags subtree]] will be automatically deleted.

Two downsides to running like this are
* it is slower and tags become non-persistent
* if some frontends have not been running for a while, their variables may vanish from the history panel editor.

== New-style History Tags ==
For the above two reasons, using /History/Tags is still recommended.
However, the format of the tags is now changed to simplify management and reduce ODB size.

; For NEW implementations
: the key [[/History ODB tree#CreateOldTags|CreateOldTags]] will not be present. "New-style" history tags will be created automatically in the [[/History ODB tree#Tags subtree|Tags subtree]].

;If you have old-style tags
: [[mlogger]] will automatically convert the old tags to this new format (this is why you should make a backup of your ODB).

** using "old" (i.e. pre Version 4435) mlogger with "new" mhttpd is OK: new mhttpd understands both formats of /History/Tags.
** using "old" mhttpd with "new" mlogger is OK provided the ODB key [[/History ODB tree#CreateOldTags|/History/CreateOldTags]] is set to "y" before starting mlogger.

== Old-style History Tags ==
Prior to versions 4435 of [[mlogger]] and [[mhttpd]], old-style history tags were generated.
See
* http://ladd00.triumf.ca/~daqweb/doc/midas-old/html/F_History_logging.html
for details.

 
---------------
 

= MIDAS ODBC SQL History system =

This section describes the internal workings of the MIDAS SQL history system.

The SQL history system implements the MIDAS history using a SQL database for data storage instead of flat binary files.

The SQL history code lives in a separate files history.h and history_sql.cxx. The present implementation uses the ODBC API for accessing SQL databases.
MySQL is fully supported, Pgsql support is partially implemented, support for arbitrary ODBC drivers should be easy to add in history_sql and most SQL operations are done using an interface class.

Preliminary testing with MySQL and PgSQL indicates that disk space requirements are the same for both storage mechanisms using flat binary files or either of the 2 SQL databases. All 3 tested storage mechanisms store the data uncompressed. (Observed gzip-1 compression ratios from CERN-ALPHA history files are 50% or better).

Preliminary testing also indicates that MySQL database is "not too slow" for use by T2K/ND280 slow control system. Using MySQL, performance seems to improve somewhat compared to traditional MIDAS-history because each SQL table is stored into a separate file, compared to everything-in-one-file in traditional MIDAS-history.

== Set up a MySQL database ==

In order to setup a MySQL database, the following is needed:

* the server hostname and port number
* database name
* user name and password.

It is recommended to create 3 different users:

* a "root" user with full priveleges,
* a "reader" user with read-only priveleges
* and a "writer" user with "insert, create table, and add column" priveleges (no permission to drop table, remove columns, etc)

=== Example ===
#Setup a MySQL database (RHEL/SL 5)

ssh root@localhost
service mysqld start
/usr/bin/mysql_secure_installation (enter current password for root: press <enter>, then accept default answer "Y" to all questions)
mysqladmin -p create history
using mysql-administrator, create 2 users:
history_writer, with database "history" schema privileges: select, insert, create, alter, index
history_reader, with database "history" schema privileges: select

#Setup $HOME/.odbc.ini file:

cat $HOME/.odbc.ini
[history_writer]
Description = history_writer
Driver = /usr/lib64/libmyodbc3.so
SERVER = ladd05.triumf.ca
PORT =
Database = history
OPTION = 3
SOCKET =
User = history_writer
Password = ******
[history_reader]
Description = history_reader
Driver = /usr/lib64/libmyodbc3.so
SERVER = ladd05.triumf.ca
PORT =
Database = history
OPTION = 3
SOCKET =
User = history_reader
Password =

Note: if you are using mhist you will need a separate .odbc.ini file for each user that you have connecting to the database (through MIDAS).

== Layout of SQL tables ==

* One table is created for each history event. Table name is the same as the history event name (as reported by [[mhdump]]).
* one column is created for each history tag (history variable). Column name is the same as the tag name (as reported by mhdump). For array tags, one column is created for each array element: array a[3] will produce columns a_0, a_1 and a_2.
* two special columns are always created: _i_time and _t_time are the timestamps in the UNIX-time integer format and in the SQL "time" format. (A priori, it is not clear which timestamp format is more useful for end users. mhttpd uses the _i_time integer time format). Note that this timestamp is the time when mlogger receives the hot-link signal (see above) and it can be a few seconds behind the time when the user placed the data into the MIDAS bank. For high precision (and sub-second time resolution) time stamps, users should generate their own timestamps and include them as part of the data itself.
* a special table called _history_index is created to remember the mapping between SQL names and MIDAS names for event names, variable name and tag names.

=== Schema Evolution ===

* when new data fields are added, new SQL columns are created automatically
* existing SQL columns are never deleted
* arrays are expanded automatically, and arrays never shrink.
* if the type of a data field changes, a conflict may occur with existing SQL columns, for example if previous type was "FLOAT" and new type is "DOUBLE", the existing SQL columns would have the SQL data type "float", too narrow to store new values. mlogger will complain about this. Such conflicts may be resolved by changing the type of the SQL column using SQL tools. In general, SQL names are created from MIDAS names by replacing all non-alphanumerical characters to an underscore "_". If this procedure creates duplicate column names, the column names and made unique by appending them with a random number, i.e. MIDAS names "A+B" and "A-B" will be translated to "A_B" and "A_B_12345" (random integer). True MIDAS names are saved into the _history_index.

[[Category:history]]

Keys in the ODB /Logger/History subtree

2020-07-30T16:08:16Z

Olchansk: /* MYSQL Reader */

== Links ==
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[/Logger ODB tree| /Logger tree]]
* [[/History ODB tree| /History tree]]
* [[History System]]
* [[Logging to a mySQL database]]
* [[Keys in the ODB /Logger/Channels subtree| /Logger/Channels subtree]]
* [[Keys in the ODB /Logger/SQL subtree| /Logger/SQL subtree]]
* [[mlogger |MIDAS logger (mlogger)]]
</div>
 
---------------
 

=''/Logger/History'' subtree =

The ODB ''/Logger/History'' subtree is a subdirectory of the [[Keys in the ODB /Logger tree | ODB /Logger tree]], created
when the MIDAS logger utility [[mlogger]] is started for the first time. The keys will be filled with default values which users may change as required to customize their experiments.

When [[mlogger]] is started for the first time, it now creates '''five''' /Logger/History/<''history-logging-channel''> ODB subtrees to control the five default history-logging-channels'', named for each of the default
[[History System#History drivers|History drivers]].

The keys present in each history-logging-channel are described below.

 
---------------
 

= Example =
The following example shows the /Logger/History ODB subtree after creation by [[mlogger]] (obtained using the [[odbedit]] '''ls''' command).
See below for information on individual keys.

<pre>
[local:midas:S]/>ls -r /Logger/History
History
MIDAS
Type MIDAS
Active y
Debug 0
PerVariableHistory 0
ODBC
Type ODBC
Active n
Debug 0
Writer_ODBC_DSN history_writer
Reader_ODBC_DSN history_reader
SQLITE
Type SQLITE
Active n
Debug 0
Sqlite dir
MYSQL
Type MYSQL
Active n
Debug 0
MYSQL Writer mysql_writer.txt
MYSQL Reader mysql_reader.txt
FILE
Type FILE
Active n
Debug 0
History dir
</pre>

 
-----
 

= ''<history-logging-channel>'' subtrees =

Each <''history-logging-channel''> subtree in the [[#top|Logger/History subtree]]
is created (by [[mlogger]]) to control one of the [[History System#History drivers|history drivers]] available, and named for that history driver.
 Presently the five following history-logging-channel subtrees are created :
* MIDAS
* ODBC
* SQLITE
* MYSQL
* FILE
(see [[#Example|Example]] above).

By default, only the MIDAS-history logging channel is [[#active|Active]].

;NOTE
: In older MIDAS versions, the history-logging-channel subtrees were named 0,1,2... respectively

 
-----
 

== Keys common to all history logging channel subtrees ==
The following three keys are common to all [[#<history-logging-channel> subtrees|history logging channel subtrees]] :

=== ''Type''===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:''' [[History System#History drivers|History driver]] name
</div>

Each [[#<history-logging-channel> subtrees |history-logging-channel subtree]] contains a ''Type'' key which specifies the type of
[[History System#History drivers|History driver]] for this channel.

When the subtree is created, the ''Type'' key is set to the respective History driver name, e.g.
: ODB key /Logger/History/'''MIDAS'''/Type will be set to '''"MIDAS"'''
: ODB key /Logger/History/'''ODBC'''/Type will be set to '''"ODBC"'''

and so on (see [[#Example|Example]] above for all five logging channels).

 
---------------
 
=== ''Active'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' BOOL
* '''Default:''' "n"
</div>

Each [[#<history-logging-channel> subtrees |history-logging-channel subtree]] contains a ''Active'' key
which enables or disables history logging in this channel.

By default, only the MIDAS-history logging channel will be set active, i.e. ODB key /Logger/History/'''MIDAS'''/Active will be set to "y".

 
;Note
: This key in the [[#MIDAS subtree|MIDAS subtree]] replaces obsolete key [[Keys in the ODB /Logger tree #/Logger/WriteFileHistory|/Logger/WriteFileHistory]]

 
--------------
 

=== ''Debug'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' INT
* '''Default:''' 0
</div>

Each [[#<history-logging-channel> subtrees |history-logging-channel subtree]] contains a ''Debug'' key
which controls the debug level for this channel. If set to 0, debugging is disabled.

;Note
:This key in the [[#ODBC subtree|ODBC subtree]] replaces obsolete key [[Keys in the ODB /Logger tree #/Logger/ODBC_Debug |/Logger/ODBC_Debug]]

 
--------------
 

== ''MIDAS'' subtree ==

This is the [[#<history-logging-channel> subtrees| history-logging-channel subtree]] that controls [[History System#History drivers|MIDAS-history]] logging.

For those Keys common to all history-logging-channels see
*[[#Type|Type]]
*[[#Active|Active]]
*[[#Debug|Debug]]

;Note
: By default, this logging channel will be set active.

The following section(s) describe keys unique to the ''MIDAS'' subtree :

=== ''PerVariableHistory'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' INT
* '''Default:''' 0
</div>

This key is present in the [[#MIDAS subtree|MIDAS subtree]] (whose [[#Type|Type]] key set to "MIDAS").

The default value of 0 is intended to operate the MIDAS-history in "traditional" mode. In this mode:
* there is one history record for each equipment
* history record id is equal to the equipment id
* Subtrees /History/Events and /History/Tags are not required and can be safely deleted

;Note
: this key replaces obsolete key /History/PerVariableHistory

 
--------------
 

== ''ODBC'' subtree ==

This is the [[#<history-logging-channel> subtrees| history-logging-channel subtree]] that controls [[History System#History drivers|ODBC-history]] logging.

For those Keys common to all history-logging-channels see
*[[#Type|Type]]
*[[#Active|Active]]
*[[#Debug|Debug]]

;Note
:[[#Debug|Debug]] key in this subtree replaces obsolete key [[Keys in the ODB /Logger tree #/Logger/ODBC_Debug |/Logger/ODBC_Debug]].

The following section(s) describe keys unique to the ODBC subtree:

=== ''Writer_ODBC_DSN'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:''' "history_writer"
</div>

This key is present in the [[#ODBC subtree|ODBC]] subtree (whose [[#Type|Type]] key is set to "ODBC").

It contains the ODBC Data Source Name for writing and is used by [[mlogger]].

;Note
: this key replaces obsolete key [[Keys in the ODB /Logger tree #/Logger/ODBC DSN |/Logger/ODBC DSN]]
 
--------------
 

=== ''Reader_ODBC_DSN'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:''' "history_reader"
</div>

This key is present in the [[#ODBC subtree|ODBC]] subtree (whose [[#Type|Type]] key is set to "ODBC").

It contains the ODBC Data Source Name for reading, and is used by [[mhttpd]] and [[mhist]] utilities.

;Note
: this key replaces obsolete key /history/ODB_DSN

 
--------------
 

== ''SQLITE'' subtree ==

This is the [[#<history-logging-channel> subtrees| history-logging-channel subtree]] that controls [[History System#History drivers|SQLITE-history]] logging.

For those Keys common to all history-logging-channels see
*[[#Type|Type]]
*[[#Active|Active]]
*[[#Debug|Debug]]

The following section(s) describe keys unique to the SQLITE subtree :

=== ''Sqlite dir'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:'''
</div>

This key is present in the [[#SQLITE subtree|SQLITE subtree]] (whose [[#Type|Type]] key is set to "SQLITE").

It contains the location of SQLITE database files, and is of the form "/file/system/path".

 
--------------
 

== ''MYSQL'' subtree ==

This is the [[#<history-logging-channel> subtrees| history-logging-channel subtree]] that controls [[History System#History drivers|MYSQL-history]] logging.

For those Keys common to all history-logging-channels see
*[[#Type|Type]]
*[[#Active|Active]]
*[[#Debug|Debug]]

The following section(s) describe keys unique to the MYSQL subtree :

=== ''MYSQL Writer'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:''' mysql_writer.txt
</div>

This key is present in the [[#MYSQL subtree|MYSQL subtree]] (whose [[#Type|Type]] key is set to "MYSQL").

It contains the name of the config file for the MYSQL database and the history writer account. This file should look similar to the following: see [[History_System#Write_MYSQL-history_events]]
 
--------------
 

=== ''MYSQL Reader'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:''' mysql_reader.txt
</div>

This key is present in the [[#MYSQL subtree|MYSQL subtree]] (whose [[#Type|Type]] key is set to "MYSQL").

It contains the name of the config file for the MYSQL database and the history reader account. This file should look similar to the following: see [[History_System#Write_MYSQL-history_events]]
 
--------------
 

== ''FILE'' subtree ==

This is the [[#<history-logging-channel> subtrees| history-logging-channel subtree]] that controls [[History System#History drivers|FILE-history]] logging.

For those Keys common to all history-logging-channels see
*[[#Type|Type]]
*[[#Active|Active]]
*[[#Debug|Debug]]

The following section(s) describe keys unique to the FILE subtree :

=== ''History dir'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:''' ""
</div>

This key is present in the [[#FILE subtree|FILE subtree]] (whose [[#Type|Type]] key is set to "FILE").

It contains the name of the directory in which the FILE-history files are to be stored.

 
--------------
 

[[Category: Logger Tree]] [[Category:History]]

Keys in the ODB /Logger/History subtree

2020-07-30T16:07:50Z

Olchansk: /* MYSQL Reader */

== Links ==
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[/Logger ODB tree| /Logger tree]]
* [[/History ODB tree| /History tree]]
* [[History System]]
* [[Logging to a mySQL database]]
* [[Keys in the ODB /Logger/Channels subtree| /Logger/Channels subtree]]
* [[Keys in the ODB /Logger/SQL subtree| /Logger/SQL subtree]]
* [[mlogger |MIDAS logger (mlogger)]]
</div>
 
---------------
 

=''/Logger/History'' subtree =

The ODB ''/Logger/History'' subtree is a subdirectory of the [[Keys in the ODB /Logger tree | ODB /Logger tree]], created
when the MIDAS logger utility [[mlogger]] is started for the first time. The keys will be filled with default values which users may change as required to customize their experiments.

When [[mlogger]] is started for the first time, it now creates '''five''' /Logger/History/<''history-logging-channel''> ODB subtrees to control the five default history-logging-channels'', named for each of the default
[[History System#History drivers|History drivers]].

The keys present in each history-logging-channel are described below.

 
---------------
 

= Example =
The following example shows the /Logger/History ODB subtree after creation by [[mlogger]] (obtained using the [[odbedit]] '''ls''' command).
See below for information on individual keys.

<pre>
[local:midas:S]/>ls -r /Logger/History
History
MIDAS
Type MIDAS
Active y
Debug 0
PerVariableHistory 0
ODBC
Type ODBC
Active n
Debug 0
Writer_ODBC_DSN history_writer
Reader_ODBC_DSN history_reader
SQLITE
Type SQLITE
Active n
Debug 0
Sqlite dir
MYSQL
Type MYSQL
Active n
Debug 0
MYSQL Writer mysql_writer.txt
MYSQL Reader mysql_reader.txt
FILE
Type FILE
Active n
Debug 0
History dir
</pre>

 
-----
 

= ''<history-logging-channel>'' subtrees =

Each <''history-logging-channel''> subtree in the [[#top|Logger/History subtree]]
is created (by [[mlogger]]) to control one of the [[History System#History drivers|history drivers]] available, and named for that history driver.
 Presently the five following history-logging-channel subtrees are created :
* MIDAS
* ODBC
* SQLITE
* MYSQL
* FILE
(see [[#Example|Example]] above).

By default, only the MIDAS-history logging channel is [[#active|Active]].

;NOTE
: In older MIDAS versions, the history-logging-channel subtrees were named 0,1,2... respectively

 
-----
 

== Keys common to all history logging channel subtrees ==
The following three keys are common to all [[#<history-logging-channel> subtrees|history logging channel subtrees]] :

=== ''Type''===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:''' [[History System#History drivers|History driver]] name
</div>

Each [[#<history-logging-channel> subtrees |history-logging-channel subtree]] contains a ''Type'' key which specifies the type of
[[History System#History drivers|History driver]] for this channel.

When the subtree is created, the ''Type'' key is set to the respective History driver name, e.g.
: ODB key /Logger/History/'''MIDAS'''/Type will be set to '''"MIDAS"'''
: ODB key /Logger/History/'''ODBC'''/Type will be set to '''"ODBC"'''

and so on (see [[#Example|Example]] above for all five logging channels).

 
---------------
 
=== ''Active'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' BOOL
* '''Default:''' "n"
</div>

Each [[#<history-logging-channel> subtrees |history-logging-channel subtree]] contains a ''Active'' key
which enables or disables history logging in this channel.

By default, only the MIDAS-history logging channel will be set active, i.e. ODB key /Logger/History/'''MIDAS'''/Active will be set to "y".

 
;Note
: This key in the [[#MIDAS subtree|MIDAS subtree]] replaces obsolete key [[Keys in the ODB /Logger tree #/Logger/WriteFileHistory|/Logger/WriteFileHistory]]

 
--------------
 

=== ''Debug'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' INT
* '''Default:''' 0
</div>

Each [[#<history-logging-channel> subtrees |history-logging-channel subtree]] contains a ''Debug'' key
which controls the debug level for this channel. If set to 0, debugging is disabled.

;Note
:This key in the [[#ODBC subtree|ODBC subtree]] replaces obsolete key [[Keys in the ODB /Logger tree #/Logger/ODBC_Debug |/Logger/ODBC_Debug]]

 
--------------
 

== ''MIDAS'' subtree ==

This is the [[#<history-logging-channel> subtrees| history-logging-channel subtree]] that controls [[History System#History drivers|MIDAS-history]] logging.

For those Keys common to all history-logging-channels see
*[[#Type|Type]]
*[[#Active|Active]]
*[[#Debug|Debug]]

;Note
: By default, this logging channel will be set active.

The following section(s) describe keys unique to the ''MIDAS'' subtree :

=== ''PerVariableHistory'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' INT
* '''Default:''' 0
</div>

This key is present in the [[#MIDAS subtree|MIDAS subtree]] (whose [[#Type|Type]] key set to "MIDAS").

The default value of 0 is intended to operate the MIDAS-history in "traditional" mode. In this mode:
* there is one history record for each equipment
* history record id is equal to the equipment id
* Subtrees /History/Events and /History/Tags are not required and can be safely deleted

;Note
: this key replaces obsolete key /History/PerVariableHistory

 
--------------
 

== ''ODBC'' subtree ==

This is the [[#<history-logging-channel> subtrees| history-logging-channel subtree]] that controls [[History System#History drivers|ODBC-history]] logging.

For those Keys common to all history-logging-channels see
*[[#Type|Type]]
*[[#Active|Active]]
*[[#Debug|Debug]]

;Note
:[[#Debug|Debug]] key in this subtree replaces obsolete key [[Keys in the ODB /Logger tree #/Logger/ODBC_Debug |/Logger/ODBC_Debug]].

The following section(s) describe keys unique to the ODBC subtree:

=== ''Writer_ODBC_DSN'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:''' "history_writer"
</div>

This key is present in the [[#ODBC subtree|ODBC]] subtree (whose [[#Type|Type]] key is set to "ODBC").

It contains the ODBC Data Source Name for writing and is used by [[mlogger]].

;Note
: this key replaces obsolete key [[Keys in the ODB /Logger tree #/Logger/ODBC DSN |/Logger/ODBC DSN]]
 
--------------
 

=== ''Reader_ODBC_DSN'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:''' "history_reader"
</div>

This key is present in the [[#ODBC subtree|ODBC]] subtree (whose [[#Type|Type]] key is set to "ODBC").

It contains the ODBC Data Source Name for reading, and is used by [[mhttpd]] and [[mhist]] utilities.

;Note
: this key replaces obsolete key /history/ODB_DSN

 
--------------
 

== ''SQLITE'' subtree ==

This is the [[#<history-logging-channel> subtrees| history-logging-channel subtree]] that controls [[History System#History drivers|SQLITE-history]] logging.

For those Keys common to all history-logging-channels see
*[[#Type|Type]]
*[[#Active|Active]]
*[[#Debug|Debug]]

The following section(s) describe keys unique to the SQLITE subtree :

=== ''Sqlite dir'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:'''
</div>

This key is present in the [[#SQLITE subtree|SQLITE subtree]] (whose [[#Type|Type]] key is set to "SQLITE").

It contains the location of SQLITE database files, and is of the form "/file/system/path".

 
--------------
 

== ''MYSQL'' subtree ==

This is the [[#<history-logging-channel> subtrees| history-logging-channel subtree]] that controls [[History System#History drivers|MYSQL-history]] logging.

For those Keys common to all history-logging-channels see
*[[#Type|Type]]
*[[#Active|Active]]
*[[#Debug|Debug]]

The following section(s) describe keys unique to the MYSQL subtree :

=== ''MYSQL Writer'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:''' mysql_writer.txt
</div>

This key is present in the [[#MYSQL subtree|MYSQL subtree]] (whose [[#Type|Type]] key is set to "MYSQL").

It contains the name of the config file for the MYSQL database and the history writer account. This file should look similar to the following: see [[History_System#Write_MYSQL-history_events]]
 
--------------
 

=== ''MYSQL Reader'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:''' mysql_reader.txt
</div>

This key is present in the [[#MYSQL subtree|MYSQL subtree]] (whose [[#Type|Type]] key is set to "MYSQL").

It contains the name of the config file for the MYSQL database and the history reader account. This file should look similar to the following: [[History_System#Write_MYSQL-history_events]]
 
--------------
 

== ''FILE'' subtree ==

This is the [[#<history-logging-channel> subtrees| history-logging-channel subtree]] that controls [[History System#History drivers|FILE-history]] logging.

For those Keys common to all history-logging-channels see
*[[#Type|Type]]
*[[#Active|Active]]
*[[#Debug|Debug]]

The following section(s) describe keys unique to the FILE subtree :

=== ''History dir'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:''' ""
</div>

This key is present in the [[#FILE subtree|FILE subtree]] (whose [[#Type|Type]] key is set to "FILE").

It contains the name of the directory in which the FILE-history files are to be stored.

 
--------------
 

[[Category: Logger Tree]] [[Category:History]]

Keys in the ODB /Logger/History subtree

2020-07-30T16:07:25Z

Olchansk: /* MYSQL Writer */

== Links ==
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[/Logger ODB tree| /Logger tree]]
* [[/History ODB tree| /History tree]]
* [[History System]]
* [[Logging to a mySQL database]]
* [[Keys in the ODB /Logger/Channels subtree| /Logger/Channels subtree]]
* [[Keys in the ODB /Logger/SQL subtree| /Logger/SQL subtree]]
* [[mlogger |MIDAS logger (mlogger)]]
</div>
 
---------------
 

=''/Logger/History'' subtree =

The ODB ''/Logger/History'' subtree is a subdirectory of the [[Keys in the ODB /Logger tree | ODB /Logger tree]], created
when the MIDAS logger utility [[mlogger]] is started for the first time. The keys will be filled with default values which users may change as required to customize their experiments.

When [[mlogger]] is started for the first time, it now creates '''five''' /Logger/History/<''history-logging-channel''> ODB subtrees to control the five default history-logging-channels'', named for each of the default
[[History System#History drivers|History drivers]].

The keys present in each history-logging-channel are described below.

 
---------------
 

= Example =
The following example shows the /Logger/History ODB subtree after creation by [[mlogger]] (obtained using the [[odbedit]] '''ls''' command).
See below for information on individual keys.

<pre>
[local:midas:S]/>ls -r /Logger/History
History
MIDAS
Type MIDAS
Active y
Debug 0
PerVariableHistory 0
ODBC
Type ODBC
Active n
Debug 0
Writer_ODBC_DSN history_writer
Reader_ODBC_DSN history_reader
SQLITE
Type SQLITE
Active n
Debug 0
Sqlite dir
MYSQL
Type MYSQL
Active n
Debug 0
MYSQL Writer mysql_writer.txt
MYSQL Reader mysql_reader.txt
FILE
Type FILE
Active n
Debug 0
History dir
</pre>

 
-----
 

= ''<history-logging-channel>'' subtrees =

Each <''history-logging-channel''> subtree in the [[#top|Logger/History subtree]]
is created (by [[mlogger]]) to control one of the [[History System#History drivers|history drivers]] available, and named for that history driver.
 Presently the five following history-logging-channel subtrees are created :
* MIDAS
* ODBC
* SQLITE
* MYSQL
* FILE
(see [[#Example|Example]] above).

By default, only the MIDAS-history logging channel is [[#active|Active]].

;NOTE
: In older MIDAS versions, the history-logging-channel subtrees were named 0,1,2... respectively

 
-----
 

== Keys common to all history logging channel subtrees ==
The following three keys are common to all [[#<history-logging-channel> subtrees|history logging channel subtrees]] :

=== ''Type''===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:''' [[History System#History drivers|History driver]] name
</div>

Each [[#<history-logging-channel> subtrees |history-logging-channel subtree]] contains a ''Type'' key which specifies the type of
[[History System#History drivers|History driver]] for this channel.

When the subtree is created, the ''Type'' key is set to the respective History driver name, e.g.
: ODB key /Logger/History/'''MIDAS'''/Type will be set to '''"MIDAS"'''
: ODB key /Logger/History/'''ODBC'''/Type will be set to '''"ODBC"'''

and so on (see [[#Example|Example]] above for all five logging channels).

 
---------------
 
=== ''Active'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' BOOL
* '''Default:''' "n"
</div>

Each [[#<history-logging-channel> subtrees |history-logging-channel subtree]] contains a ''Active'' key
which enables or disables history logging in this channel.

By default, only the MIDAS-history logging channel will be set active, i.e. ODB key /Logger/History/'''MIDAS'''/Active will be set to "y".

 
;Note
: This key in the [[#MIDAS subtree|MIDAS subtree]] replaces obsolete key [[Keys in the ODB /Logger tree #/Logger/WriteFileHistory|/Logger/WriteFileHistory]]

 
--------------
 

=== ''Debug'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' INT
* '''Default:''' 0
</div>

Each [[#<history-logging-channel> subtrees |history-logging-channel subtree]] contains a ''Debug'' key
which controls the debug level for this channel. If set to 0, debugging is disabled.

;Note
:This key in the [[#ODBC subtree|ODBC subtree]] replaces obsolete key [[Keys in the ODB /Logger tree #/Logger/ODBC_Debug |/Logger/ODBC_Debug]]

 
--------------
 

== ''MIDAS'' subtree ==

This is the [[#<history-logging-channel> subtrees| history-logging-channel subtree]] that controls [[History System#History drivers|MIDAS-history]] logging.

For those Keys common to all history-logging-channels see
*[[#Type|Type]]
*[[#Active|Active]]
*[[#Debug|Debug]]

;Note
: By default, this logging channel will be set active.

The following section(s) describe keys unique to the ''MIDAS'' subtree :

=== ''PerVariableHistory'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' INT
* '''Default:''' 0
</div>

This key is present in the [[#MIDAS subtree|MIDAS subtree]] (whose [[#Type|Type]] key set to "MIDAS").

The default value of 0 is intended to operate the MIDAS-history in "traditional" mode. In this mode:
* there is one history record for each equipment
* history record id is equal to the equipment id
* Subtrees /History/Events and /History/Tags are not required and can be safely deleted

;Note
: this key replaces obsolete key /History/PerVariableHistory

 
--------------
 

== ''ODBC'' subtree ==

This is the [[#<history-logging-channel> subtrees| history-logging-channel subtree]] that controls [[History System#History drivers|ODBC-history]] logging.

For those Keys common to all history-logging-channels see
*[[#Type|Type]]
*[[#Active|Active]]
*[[#Debug|Debug]]

;Note
:[[#Debug|Debug]] key in this subtree replaces obsolete key [[Keys in the ODB /Logger tree #/Logger/ODBC_Debug |/Logger/ODBC_Debug]].

The following section(s) describe keys unique to the ODBC subtree:

=== ''Writer_ODBC_DSN'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:''' "history_writer"
</div>

This key is present in the [[#ODBC subtree|ODBC]] subtree (whose [[#Type|Type]] key is set to "ODBC").

It contains the ODBC Data Source Name for writing and is used by [[mlogger]].

;Note
: this key replaces obsolete key [[Keys in the ODB /Logger tree #/Logger/ODBC DSN |/Logger/ODBC DSN]]
 
--------------
 

=== ''Reader_ODBC_DSN'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:''' "history_reader"
</div>

This key is present in the [[#ODBC subtree|ODBC]] subtree (whose [[#Type|Type]] key is set to "ODBC").

It contains the ODBC Data Source Name for reading, and is used by [[mhttpd]] and [[mhist]] utilities.

;Note
: this key replaces obsolete key /history/ODB_DSN

 
--------------
 

== ''SQLITE'' subtree ==

This is the [[#<history-logging-channel> subtrees| history-logging-channel subtree]] that controls [[History System#History drivers|SQLITE-history]] logging.

For those Keys common to all history-logging-channels see
*[[#Type|Type]]
*[[#Active|Active]]
*[[#Debug|Debug]]

The following section(s) describe keys unique to the SQLITE subtree :

=== ''Sqlite dir'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:'''
</div>

This key is present in the [[#SQLITE subtree|SQLITE subtree]] (whose [[#Type|Type]] key is set to "SQLITE").

It contains the location of SQLITE database files, and is of the form "/file/system/path".

 
--------------
 

== ''MYSQL'' subtree ==

This is the [[#<history-logging-channel> subtrees| history-logging-channel subtree]] that controls [[History System#History drivers|MYSQL-history]] logging.

For those Keys common to all history-logging-channels see
*[[#Type|Type]]
*[[#Active|Active]]
*[[#Debug|Debug]]

The following section(s) describe keys unique to the MYSQL subtree :

=== ''MYSQL Writer'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:''' mysql_writer.txt
</div>

This key is present in the [[#MYSQL subtree|MYSQL subtree]] (whose [[#Type|Type]] key is set to "MYSQL").

It contains the name of the config file for the MYSQL database and the history writer account. This file should look similar to the following: see [[History_System#Write_MYSQL-history_events]]
 
--------------
 

=== ''MYSQL Reader'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:''' mysql_reader.txt
</div>

This key is present in the [[#MYSQL subtree|MYSQL subtree]] (whose [[#Type|Type]] key is set to "MYSQL").

It contains the name of the config file for the MYSQL database and the history reader account. This file should look similar to the following:
<pre>
server=localhost
port=3306
database=history
socket=/var/lib/mysql/mysql.sock
user=history_reader
password=[password of mysql user]
buffer=10000
</pre>

 
--------------
 

== ''FILE'' subtree ==

This is the [[#<history-logging-channel> subtrees| history-logging-channel subtree]] that controls [[History System#History drivers|FILE-history]] logging.

For those Keys common to all history-logging-channels see
*[[#Type|Type]]
*[[#Active|Active]]
*[[#Debug|Debug]]

The following section(s) describe keys unique to the FILE subtree :

=== ''History dir'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:''' ""
</div>

This key is present in the [[#FILE subtree|FILE subtree]] (whose [[#Type|Type]] key is set to "FILE").

It contains the name of the directory in which the FILE-history files are to be stored.

 
--------------
 

[[Category: Logger Tree]] [[Category:History]]

Keys in the ODB /Logger/History subtree

2020-07-30T16:07:04Z

Olchansk: /* MYSQL Writer */

== Links ==
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[/Logger ODB tree| /Logger tree]]
* [[/History ODB tree| /History tree]]
* [[History System]]
* [[Logging to a mySQL database]]
* [[Keys in the ODB /Logger/Channels subtree| /Logger/Channels subtree]]
* [[Keys in the ODB /Logger/SQL subtree| /Logger/SQL subtree]]
* [[mlogger |MIDAS logger (mlogger)]]
</div>
 
---------------
 

=''/Logger/History'' subtree =

The ODB ''/Logger/History'' subtree is a subdirectory of the [[Keys in the ODB /Logger tree | ODB /Logger tree]], created
when the MIDAS logger utility [[mlogger]] is started for the first time. The keys will be filled with default values which users may change as required to customize their experiments.

When [[mlogger]] is started for the first time, it now creates '''five''' /Logger/History/<''history-logging-channel''> ODB subtrees to control the five default history-logging-channels'', named for each of the default
[[History System#History drivers|History drivers]].

The keys present in each history-logging-channel are described below.

 
---------------
 

= Example =
The following example shows the /Logger/History ODB subtree after creation by [[mlogger]] (obtained using the [[odbedit]] '''ls''' command).
See below for information on individual keys.

<pre>
[local:midas:S]/>ls -r /Logger/History
History
MIDAS
Type MIDAS
Active y
Debug 0
PerVariableHistory 0
ODBC
Type ODBC
Active n
Debug 0
Writer_ODBC_DSN history_writer
Reader_ODBC_DSN history_reader
SQLITE
Type SQLITE
Active n
Debug 0
Sqlite dir
MYSQL
Type MYSQL
Active n
Debug 0
MYSQL Writer mysql_writer.txt
MYSQL Reader mysql_reader.txt
FILE
Type FILE
Active n
Debug 0
History dir
</pre>

 
-----
 

= ''<history-logging-channel>'' subtrees =

Each <''history-logging-channel''> subtree in the [[#top|Logger/History subtree]]
is created (by [[mlogger]]) to control one of the [[History System#History drivers|history drivers]] available, and named for that history driver.
 Presently the five following history-logging-channel subtrees are created :
* MIDAS
* ODBC
* SQLITE
* MYSQL
* FILE
(see [[#Example|Example]] above).

By default, only the MIDAS-history logging channel is [[#active|Active]].

;NOTE
: In older MIDAS versions, the history-logging-channel subtrees were named 0,1,2... respectively

 
-----
 

== Keys common to all history logging channel subtrees ==
The following three keys are common to all [[#<history-logging-channel> subtrees|history logging channel subtrees]] :

=== ''Type''===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:''' [[History System#History drivers|History driver]] name
</div>

Each [[#<history-logging-channel> subtrees |history-logging-channel subtree]] contains a ''Type'' key which specifies the type of
[[History System#History drivers|History driver]] for this channel.

When the subtree is created, the ''Type'' key is set to the respective History driver name, e.g.
: ODB key /Logger/History/'''MIDAS'''/Type will be set to '''"MIDAS"'''
: ODB key /Logger/History/'''ODBC'''/Type will be set to '''"ODBC"'''

and so on (see [[#Example|Example]] above for all five logging channels).

 
---------------
 
=== ''Active'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' BOOL
* '''Default:''' "n"
</div>

Each [[#<history-logging-channel> subtrees |history-logging-channel subtree]] contains a ''Active'' key
which enables or disables history logging in this channel.

By default, only the MIDAS-history logging channel will be set active, i.e. ODB key /Logger/History/'''MIDAS'''/Active will be set to "y".

 
;Note
: This key in the [[#MIDAS subtree|MIDAS subtree]] replaces obsolete key [[Keys in the ODB /Logger tree #/Logger/WriteFileHistory|/Logger/WriteFileHistory]]

 
--------------
 

=== ''Debug'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' INT
* '''Default:''' 0
</div>

Each [[#<history-logging-channel> subtrees |history-logging-channel subtree]] contains a ''Debug'' key
which controls the debug level for this channel. If set to 0, debugging is disabled.

;Note
:This key in the [[#ODBC subtree|ODBC subtree]] replaces obsolete key [[Keys in the ODB /Logger tree #/Logger/ODBC_Debug |/Logger/ODBC_Debug]]

 
--------------
 

== ''MIDAS'' subtree ==

This is the [[#<history-logging-channel> subtrees| history-logging-channel subtree]] that controls [[History System#History drivers|MIDAS-history]] logging.

For those Keys common to all history-logging-channels see
*[[#Type|Type]]
*[[#Active|Active]]
*[[#Debug|Debug]]

;Note
: By default, this logging channel will be set active.

The following section(s) describe keys unique to the ''MIDAS'' subtree :

=== ''PerVariableHistory'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' INT
* '''Default:''' 0
</div>

This key is present in the [[#MIDAS subtree|MIDAS subtree]] (whose [[#Type|Type]] key set to "MIDAS").

The default value of 0 is intended to operate the MIDAS-history in "traditional" mode. In this mode:
* there is one history record for each equipment
* history record id is equal to the equipment id
* Subtrees /History/Events and /History/Tags are not required and can be safely deleted

;Note
: this key replaces obsolete key /History/PerVariableHistory

 
--------------
 

== ''ODBC'' subtree ==

This is the [[#<history-logging-channel> subtrees| history-logging-channel subtree]] that controls [[History System#History drivers|ODBC-history]] logging.

For those Keys common to all history-logging-channels see
*[[#Type|Type]]
*[[#Active|Active]]
*[[#Debug|Debug]]

;Note
:[[#Debug|Debug]] key in this subtree replaces obsolete key [[Keys in the ODB /Logger tree #/Logger/ODBC_Debug |/Logger/ODBC_Debug]].

The following section(s) describe keys unique to the ODBC subtree:

=== ''Writer_ODBC_DSN'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:''' "history_writer"
</div>

This key is present in the [[#ODBC subtree|ODBC]] subtree (whose [[#Type|Type]] key is set to "ODBC").

It contains the ODBC Data Source Name for writing and is used by [[mlogger]].

;Note
: this key replaces obsolete key [[Keys in the ODB /Logger tree #/Logger/ODBC DSN |/Logger/ODBC DSN]]
 
--------------
 

=== ''Reader_ODBC_DSN'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:''' "history_reader"
</div>

This key is present in the [[#ODBC subtree|ODBC]] subtree (whose [[#Type|Type]] key is set to "ODBC").

It contains the ODBC Data Source Name for reading, and is used by [[mhttpd]] and [[mhist]] utilities.

;Note
: this key replaces obsolete key /history/ODB_DSN

 
--------------
 

== ''SQLITE'' subtree ==

This is the [[#<history-logging-channel> subtrees| history-logging-channel subtree]] that controls [[History System#History drivers|SQLITE-history]] logging.

For those Keys common to all history-logging-channels see
*[[#Type|Type]]
*[[#Active|Active]]
*[[#Debug|Debug]]

The following section(s) describe keys unique to the SQLITE subtree :

=== ''Sqlite dir'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:'''
</div>

This key is present in the [[#SQLITE subtree|SQLITE subtree]] (whose [[#Type|Type]] key is set to "SQLITE").

It contains the location of SQLITE database files, and is of the form "/file/system/path".

 
--------------
 

== ''MYSQL'' subtree ==

This is the [[#<history-logging-channel> subtrees| history-logging-channel subtree]] that controls [[History System#History drivers|MYSQL-history]] logging.

For those Keys common to all history-logging-channels see
*[[#Type|Type]]
*[[#Active|Active]]
*[[#Debug|Debug]]

The following section(s) describe keys unique to the MYSQL subtree :

=== ''MYSQL Writer'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:''' mysql_writer.txt
</div>

This key is present in the [[#MYSQL subtree|MYSQL subtree]] (whose [[#Type|Type]] key is set to "MYSQL").

It contains the name of the config file for the MYSQL database and the history writer account. This file should look similar to the following: see [[History_System#Write_MYSQL-history_events]]
<pre>
server=localhost
port=3306
database=history
socket=/var/lib/mysql/mysql.sock
user=history_writer
password=[password of mysql user]
buffer=10000
</pre>

 
--------------
 

=== ''MYSQL Reader'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:''' mysql_reader.txt
</div>

This key is present in the [[#MYSQL subtree|MYSQL subtree]] (whose [[#Type|Type]] key is set to "MYSQL").

It contains the name of the config file for the MYSQL database and the history reader account. This file should look similar to the following:
<pre>
server=localhost
port=3306
database=history
socket=/var/lib/mysql/mysql.sock
user=history_reader
password=[password of mysql user]
buffer=10000
</pre>

 
--------------
 

== ''FILE'' subtree ==

This is the [[#<history-logging-channel> subtrees| history-logging-channel subtree]] that controls [[History System#History drivers|FILE-history]] logging.

For those Keys common to all history-logging-channels see
*[[#Type|Type]]
*[[#Active|Active]]
*[[#Debug|Debug]]

The following section(s) describe keys unique to the FILE subtree :

=== ''History dir'' ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:''' ""
</div>

This key is present in the [[#FILE subtree|FILE subtree]] (whose [[#Type|Type]] key is set to "FILE").

It contains the name of the directory in which the FILE-history files are to be stored.

 
--------------
 

[[Category: Logger Tree]] [[Category:History]]

History System

2020-07-30T16:05:51Z

Olchansk: /* Write MYSQL-history events */

{{Pagelinks}}

== Links ==
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[/History ODB tree]]
* [[History Page]]
* [[Mhformat|MIDAS history file format]]
* [[Keys in the ODB /Logger tree|/Logger ODB tree]]
* [[Keys in the ODB /Logger/History subtree|/Logger/History subtree]]
* [[Image History]]
</div>

<div id="history file"></div>
= Introduction =

The history system is an add-on capability built into the MIDAS data logger [[mlogger]] to record useful information in parallel to the data logging.
This information is recorded with a time stamp and saved into a '''history file''' (see [[#History drivers|History drivers]]) for later retrieval. One set of files is created per day containing all the requested history events. The history data may be displayed in graphical form using the [[mhttpd]] MIDAS webserver utility, giving the user an easy way of seeing how experimental variables have changed with time.
The history logging will be in action only if the MIDAS data logger is running, but it is not necessary to have any [[Keys in the ODB /Logger tree#/Logger/Channels subtree|data logging channel]] enabled.

Since June 2020, it is possible to also use the midas history system to record images (png/jpeg etc) periodically, for example to later produce a timelapse from a webcam - see the [[Image History]] page for more details.

= MidasHistoryInterface =

The new history configuration defines history channels similar to logger channels. Each history channel creates a history interface class (class MidasHistoryInterface). The [[mlogger|data logger]] can use this interface to write data into the history, mhttpd, mhist & co can use this interface to read data from the history. The interface is defined in history.h

* http://ladd00.triumf.ca/~daqweb/doc/midas/include/history.h

= History drivers =

Presently, the MIDAS distribution contains these drivers:

; MIDAS-History
: this is the traditional history driver, with data stored in the .hst binary files. (format is documented here [[mhformat]], see also [[mhist]] and [[mhdump]])
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#MIDAS subtree |"MIDAS"]]''' is set up to write MIDAS-history.

; ODBC-History
: stores history data in SQL database using the ODBC abstraction layer. See [[#MIDAS ODBC SQL History system|ODBC SQL History system]].

: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#ODBC subtree|"ODBC"]]''' is set up to write ODBC-history.

; SQLITE-History
: stores data in SQL database as SQLITE3 .sqlite3 files. The SQL table format is similar to ODBC-history, performance is similar, no database server required.
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#SQLITE subtree|"SQLITE"]]''' is set up to write SQLITE-history.

; MYSQL-History
:
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#MYSQL subtree|"MYSQL"]]''' is set up to write MYSQL-history.

; FILE-History
: stores each variable in its own file, making it much easier to trace the history of a particular variable.
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#FILE subtree|FILE]]''' is set up to write FILE-history.

More drivers can be easily added by the user (create history_xxx.cxx, add constructor to hs_get_history(), create the ODB subtree /Logger/History/xxx/channel).

= Location of History Files =
;MIDAS-history
:By default, the traditional MIDAS-history is enabled, writing history files into the MIDAS data directory path given by the ODB key [[Keys in the ODB /Logger tree#Data dir|Data dir]].
:If history data is important, it is recommended to write it to a directory that is backed up or archived. (The MIDAS data directory is usually located on a scratch data disk without backups).

:The location of the MIDAS-history data files can be changed by use of the ODB Key [[Keys in the ODB /Logger tree#History dir|History dir]].

;ODBC-History
:The location of ODBC-History files is ...

;SQLITE-History
:The location of SQLITE-History files is the directory path given by the ODB key [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]].

;MYSQL-History
:The location of MYSQL-History files is ...

;FILE-History
:The location of FILE-History files is the directory path given by the ODB key [[Keys in the ODB /Logger/History subtree#History dir|History dir]].

= Types of History Events =

There are two basic types of history events, which are defined in different ways:

* "[[#Frontend History Event|Frontend]]" (or "Equipment") History event composed in the frontend. See [[#Frontend History Event|Frontend History Event]] for details.
* "Virtual" (or "Links") History event composed within the ODB under the [[/History ODB tree#Links subtree|/History/Links subtree]]. See [[#Virtual History Event|Virtual History Event]] for details.

Both these history event definitions take effect when the data logger gets a start run [[Run States and Transitions|transition]]. Any modifications made during the run are not applied until the start of the next run.

== Virtual History Event ==
The history of any ODB variable can be recorded in a "virtual" history event by creating a link to that variable in the [[/History ODB tree#Links subtree|/History/Links subtree]]. This is used for ODB variables that are not in an Equipment.

; To enable Virtual History Events
: Create links as desired in [[/History ODB tree#Links subtree|/History/Links subtree]].

History events are created for each entry under ODB subdirectory {{Odbpath|path=/History/Links}}.

Two types of links are permitted:
<dl>
<dt> ODB key {{Odbpath|path=/history/links/aaa}} is a link to a subdirectory</dt>
<dd><ol>
<li> db_watch() is setup to watch this subdirectory</li>
<li> tags are created for each subdirectory entry (1 tag per entry)</li>
<li> There is no possibility for naming array elements, so 1 tag per array, regardless of the number of elements.</li> </ol></dd>
<dt>ODB key {{Odbpath|path=/history/links/bbb}} is a subdirectory with links to ODB values</dt>
<dd> <ol>
<li>db_watch is setup to watch each link target</li>
<li> tags are created for each link (1 tag per link)</li>
<li> tag name is the link name (NOT the target name). There is no possibility for naming array elements</li>
</ol></dd>
</dl>

Mixing links and subdirectories is not permitted.

The update period of history events created for {{Odbpath|path=/history/links}} is controlled by entries in {{Odbpath|path=/history/links periods}}. Numeric values of periods are same as for a [[#Frontend History Event|Frontend History event]]. Numeric value 0 disables the history for a particular event.

See [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087] for more information.

== Frontend History Event ==

Each [[Frontend Operation#Equipments and Events|Equipment]] has the capability to generate "history data".

; To enable Frontend History Events

: Set the [[/Equipment ODB tree#Log history|Log history]] ODB key of each equipment to be logged '''non-zero'''. (See [[Equipment List Parameters#Log History|Log History parameter]]). (You may need to restart the frontend.)

This causes the event contents to be copied to the ODB [[/Equipment ODB tree#Variables subtree|/Equipment/''<equipment-name>''/Variables subtree]] for the use of the History system. history events are created by parsing the content of {{Odbpath|path=/Equipment/<equipment-name>/variables}}.

For each equipment, history is controlled by the value of {{Odbpath|path=/Equipment/<equipment-name>/common/period}} as follows:
* 0 = history disabled
* 1 = history is enabled
* >1 = history is enabled, throttled down

The throttling is implemented in log_history()/watch_history() by this algorithm:
the very first history event is recorded, then all changed to the data are ignored until "period" seconds has elapsed. Then the next history event will be recorded, and following
changes will be ignored until "period" second elapses, and so forth. Period value "1" has special meaning - there is no throttling, all history events are logged.

This information is from [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087].

=== Sequence for frontend History Event ===
The main steps for the frontend History Event is as follows:

* The user code in the frontend equipment Event Readout routines reads the data, placing it into a MIDAS data bank
* In mfe.c, if the key [[/Equipment ODB tree#Log history|Log history]] > 0 (i.e. the history system is enabled for this equipment), this data bank is written into ODB (by update_odb()).
* This ODB write triggers an ODB hot-link into mlogger
* The hot-link calls mlogger.c::log_history(), which calls hs_write() to write the data into the history file. The frequency of the history writes is specified by the number of seconds between writes stored in the ODB key [[/Equipment ODB tree#Log history|Log history]].
* history.c::hs_write() or history_odbc::hs_write_odbc() writes the data into a history file or into an SQL database.

= How to Write History events =
* Enable desired [[#To enable Frontend History Events|Frontend]] and/or [[#To enable Virtual History Events|Virtual]] history events
* [[#Select the history driver|Select the history driver]]
* Follow instructions below for selected driver:
** [[#Write MIDAS-History events|Write MIDAS-history events]]
** [[#Write ODBC-History events|Write ODBC-history events]]
** [[#Write SQLITE-History events|Write SQLITE-history events]]
** [[#Write MYSQL-History events|Write MYSQL-history events]]
** [[#Write FILE-History events|Write FILE-history events]]

== Select the history driver ==

* Select the '''[[#History drivers|history driver]]''' and therefore the [[Keys in the ODB /Logger/History subtree#<history-logging-channel> subtrees|history-logging-channel]] to use to write the MIDAS-history files.
* set [[Keys in the ODB /Logger/History subtree#Active|Active]] ODB key in the selected [[Keys in the ODB /Logger/History subtree#<history-logging-channel> subtrees|history-logging-channel]] to "y". All other logging channels should be disabled.
* set [[/History ODB tree#LoggerHistoryChannel|LoggerHistoryChannel]] ODB key to the selected [[#History drivers|History channel]].

== Write MIDAS-History events ==
* Enable desired [[#To enable Frontend History Events|Frontend]] and/or [[#To enable Virtual History Events|Virtual]] history events
* Perform steps in [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "MIDAS".
* Restart [[mhttpd]] and [[mlogger]], start the run.

MIDAS-history files will be saved into the [[#Location of History Files|MIDAS-history data directory]].

== Write ODBC-History events ==

See [[#MIDAS ODBC SQL History system|MIDAS ODBC SQL History system]].
* If necessary, [[#Set up a MySQL database|set up a MySQL database]].
* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps in [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "ODBC".
* Check ODB Key [[Keys in the ODB /Logger/History subtree#Writer_ODBC_DSN|Writer_ODBC_DSN]] is set to "history writer" and Key [[Keys in the ODB /Logger/History subtree#Reader_ODBC_DSN|Reader_ODBC_DSN]] is set to "history reader" (the default values)
* set [[Keys in the ODB /Logger/History subtree#Debug|Debug]] to 1 if desired
* restart [[mlogger]] in verbose mode (i.e. mlogger -v), and observe how it issues SQL commands to create the tables and columns corresponding to MIDAS-history events and tags.
* restart [[mhttpd]]

'''Notes'''
# [[mlogger]] is programmed to raise alarms if connection to SQL database is interrupted or if some events cannot be written into the database (i.e. data type mismatch, SQL syntax errors, etc). The traditional MIDAS-history never raised alarms because it "never failed" - other than from "disk full" errors, which are immediately obvious.
# With ODBC History enabled, [[mhttpd]] will only use history information from the SQL database to make history plots and to extract history variable names for the history plot editor. The ODB {{Odbpath|path=/History}} subtrees[[/History ODB tree#Tags subtree|Tags]] and [[/History ODB tree#Events subtree|Events]] are not used.
# variable names presented to the user may change from MIDAS names to SQL names (use the history panel "Label" text fields to create permanent plot labels).

== Write SQLITE-history events ==

No special configuration required.

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "SQLITE".

* Create a directory for storage of sqlite history files
* save its path in the ODB key [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]]
* restart [[mhttpd]] and [[mlogger]]

In the history data directory (in [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]]) you should see files with name "mh_xxx.sqlite3" corresponding to your equipment and variable names.

== Write MYSQL-history events ==

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "MYSQL".
* setup the MySQL database:
* install mariadb-10.4
<pre>
macos:
sudo port install mariadb-10.4
ln -s ln -fs /opt/local/lib/mariadb-10.4/bin/mysql_config ~/bin/
mysql_config --version
10.4.13
</pre>
* rebuild midas, check the cmake finds the correct mysql: MIDAS: Found MySQL version 10.4.13
* create $HOME/.my.cnf
<pre>
[mysqld]
#port=3306
socket=/Users/olchansk/mysql/mysql.sock
datadir=/Users/olchansk/mysql/db
pid-file=/Users/olchansk/mysql/mysqld.pid

[client]
#port=3306
socket=/Users/olchansk/mysql/mysql.sock
</pre>
* mkdir $HOME/mysql
* start mysql:
<pre>
macos: /opt/local/lib/mariadb-10.4/bin/mysqld
</pre>
* configure the database: start the "mysql" utility (/opt/local/lib/mariadb-10.4/bin/mysql)
<pre>
NOTE: instructions thanks to Ben S.
NOTE: use your own password instead of "writer_password" in this example

MariaDB [(none)]> CREATE DATABASE IF NOT EXISTS history;
Query OK, 1 row affected (0.003 sec)

MariaDB [(none)]> CREATE USER history_reader;
Query OK, 0 rows affected (0.027 sec)

MariaDB [(none)]> GRANT SELECT ON history.* TO history_reader;
Query OK, 0 rows affected (0.016 sec)

MariaDB [(none)]> CREATE USER 'history_writer'@'localhost' IDENTIFIED BY 'writer_password';
Query OK, 0 rows affected (0.014 sec)

MariaDB [(none)]> GRANT SELECT,INSERT,CREATE,ALTER ON history.* TO 'history_writer'@'localhost';
Query OK, 0 rows affected (0.017 sec)
</pre>
* try connecting to both users:
<pre>
/opt/local/lib/mariadb-10.4/bin/mysql -u history_reader ### should connect without a password
/opt/local/lib/mariadb-10.4/bin/mysql -u history_writer -p ### should ask for password
</pre>
* create midas config files:
** mysql_writer.txt
<pre>
#server=localhost
#port=xxx
database=history
socket=/Users/olchansk/mysql/mysql.sock
user=writer
password=writer_password ### change this!!!
#buffer=1000
</pre>
** mysql_reader.txt
<pre>
#server=localhost
#port=xxx
database=history
socket=/Users/olchansk/mysql/mysql.sock
user=reader
#password=reader
</pre>

...........

== Write FILE-history events ==

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "SQLITE".
* Assign directory path for saved files using the ODB key [[Keys in the ODB /Logger/History subtree#History dir|History dir]].
...........

 
---------------
 

= History Events =
The following information is derived from [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087].

A "history event" is a history atomic unit of data. Associated with each history event is a timestamp (unix time),
a name (limited to NAME_LENGTH in the old history) and a list of history tags that describe the individual data
values inside the history event.

When making history plots in [[mhttpd]] (see [[History Page]]) for each curve on the plot, one selects a history event (from the list
of currently active events, recently active events or the list of all events that ever existed), then from the list of tags
inside the history event one selects the particular variable that will be plotted.

== Old MIDAS history events ==
In the old MIDAS history, all history events are written into one history file (.hst file + optional .def and .idx event definition and time index files
which can be/are regenerated automatically from the .hst file). History events are identified by 16-bit history event IDs, the persistent mapping
from history event names and the 16-bit history event IDs is stored in ODB subtree {{Odbpath|path=/History/Events}}. In addition the list of all known history event tags is
stored in ODB subtree {{Odbpath|path=/History/Tags}}. For per-equipment history, the 16-bit history event ID is the value of ODB key [[/Equipment ODB tree#Event ID|/Equipment/<equipment-name>/Common/Event ID]].

== SQL history events ==
In the SQL history (MySQL, SQLITE, etc), each history event is an SQL table. The history event tags are the SQL table columns.

== FILE history events ==
In the new FILE history, each history event is written into a separate file, tag definition are recorded in text format in the file header, history event
data is appended to the file in binary format (fixed record size). If the history event definition is changed, a new file will be started.

= History Event Construction =
The [[mlogger]] creates [[#Frontend history event]]s in open_history() by parsing the ODB subtree {{Odbpath|path=/equipment/<equipment-name>/variables}}. Each ODB entry under the {{Odbpath|path=variables/}} subtree is referred to as a "variable". Each variable can be a single ODB value, an array of ODB values, or a subdirectory (corresponding to TID_STRUCT structured data banks). As each variable is processed, one or more tags are created to describe it.

Single ODB values will generally produce a single tag, while arrays can produce (depending whether the array is "[[#Named Arrays|Named]]" or not)
* multiple tags - one per array element ("Named" array) or
* one single tag - describing the whole array

The code can generate two types of history:
<dl>
<dt> "per-equipment" history</dt>
<dd> will have the tags for all variables concatenated together into one single history event</dd>
<dt> "per-variable" history</dt>
<dd> will have one history event defined for each variable. Inside could be one tag (for single odb values and unnamed array) or multiple tags (for named arrays and structured data
banks).</dd>
</dl>

Per-equipment history is the original MIDAS history implementation. Per-variable history was added to permit efficient data storage in SQL tables. Its initial implementation used 1 ODB hotlink for each variable and it was easy to exceed the maximum permitted number of ODB hotlinks (db_open_record()). To reduce consumption of hotlinks, db_watch() has been implemented and now per-variable history only uses 1 ODB hotlink per equipment.

;NOTE
: With db_watch, per-equipment history is no longer available. '''Per-variable history is the new default''' (and the only option).

[https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087]

= Construction of history event tags =
Currently,
* single ODB values are treated as arrays of length "1"
* string ODB entries are not permitted

History event tags are constructed in the [[mlogger]] add_equipment(). Presently there are some quirks with the generation of history event tags (see [https://midas.triumf.ca/elog/Midas/1088]),
e.g. single odb values are treated as arrays of length "1" and "string" ODB entries are not permitted.

For variables of type TID_KEY (subdirectories, corresponding to TID_STRUCT structured banks), one tag is generated for each subdirectory entry. Tag names for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa/bbb}} will be "aaa_bbb" (with an underscore). Subdirectory entries of type TID_KEY and TID_LINK are currently not supported.

For "[[#Named Arrays|Named Arrays]]", individual tags are generated for each array element. Tag names are taken from the ''Names'' array. For empty tag names (empty names array), tags are "aaa_0", "aaa_1", etc (for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}}).

For "[[#Named Arrays|Single Names Arrays]]", tag names have the variable name appended (with a space), for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}} and an empty names array, tags will be "aaa_0 aaa", "aaa_1 aaa", etc. For a populated names array, the tags will be "name0 aaa", "name1 aaa", etc.

For unnamed arrays and single odb variables (in ODB, single odb variables are arrays of length 1), a single tag is generated.

== Named Arrays ==

'''Named Arrays''' have a "Names" entry in {{Odbpath|path=/equipment/<equipment-name>/settings}}.

For example, to add names to {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}}, create a string array {{Odbpath|path=/equipment/<equipment-name>/settings/names aaa}}. The names array should be at least as long as the corresponding data array. Individual entries in the names array can be left blank (tag names will be "aaa_0", "aaa_1", etc). Duplicate tag names are not permitted.

A single "Names" entry can be created to name all arrays in variables with the same names ('''"Single Names"''').

Create {{Odbpath|path=/equipment/<equipment-name>/settings/names}} and arrays {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}} and {{Odbpath|path=/equipment/<equipment-name>/variables/bbb}} will have
history tags "name0 aaa", "name1 aaa", "name0 bbb", "name1 bbb", etc. If "names" are left blank, tag names will be "aaa_0 aaa", "aaa_1 aaa", "bbb_0 bbb", "bbb_1 bbb", etc.

In the [[mhttpd]] web server [[History Page]] variables viewer, "single name" arrays are displayed in a 2D table.

This information is from [https://midas.triumf.ca/elog/Midas/1088].

= Historical note on MIDAS-History Tag names =
Tags are only used for MIDAS-history events.

The mlogger/mhttpd implementation of /History/Tags has proved troublesome and we are moving away from it.

;During the transition period:
: Implementations starting with Version 4435 of [[mlogger]] and [[mhttpd]] (i.e. "new" versions) will now work '''without''' /History/Tags. The history tags are read directly from the history files themselves. To run in this mode, set ODB key [[/History ODB tree#DisableTags|/History/DisableTags]] to "y". Existing [[/History ODB tree#Tags subtree|/History/Tags subtree]] will be automatically deleted.

Two downsides to running like this are
* it is slower and tags become non-persistent
* if some frontends have not been running for a while, their variables may vanish from the history panel editor.

== New-style History Tags ==
For the above two reasons, using /History/Tags is still recommended.
However, the format of the tags is now changed to simplify management and reduce ODB size.

; For NEW implementations
: the key [[/History ODB tree#CreateOldTags|CreateOldTags]] will not be present. "New-style" history tags will be created automatically in the [[/History ODB tree#Tags subtree|Tags subtree]].

;If you have old-style tags
: [[mlogger]] will automatically convert the old tags to this new format (this is why you should make a backup of your ODB).

** using "old" (i.e. pre Version 4435) mlogger with "new" mhttpd is OK: new mhttpd understands both formats of /History/Tags.
** using "old" mhttpd with "new" mlogger is OK provided the ODB key [[/History ODB tree#CreateOldTags|/History/CreateOldTags]] is set to "y" before starting mlogger.

== Old-style History Tags ==
Prior to versions 4435 of [[mlogger]] and [[mhttpd]], old-style history tags were generated.
See
* http://ladd00.triumf.ca/~daqweb/doc/midas-old/html/F_History_logging.html
for details.

 
---------------
 

= MIDAS ODBC SQL History system =

This section describes the internal workings of the MIDAS SQL history system.

The SQL history system implements the MIDAS history using a SQL database for data storage instead of flat binary files.

The SQL history code lives in a separate files history.h and history_sql.cxx. The present implementation uses the ODBC API for accessing SQL databases.
MySQL is fully supported, Pgsql support is partially implemented, support for arbitrary ODBC drivers should be easy to add in history_sql and most SQL operations are done using an interface class.

Preliminary testing with MySQL and PgSQL indicates that disk space requirements are the same for both storage mechanisms using flat binary files or either of the 2 SQL databases. All 3 tested storage mechanisms store the data uncompressed. (Observed gzip-1 compression ratios from CERN-ALPHA history files are 50% or better).

Preliminary testing also indicates that MySQL database is "not too slow" for use by T2K/ND280 slow control system. Using MySQL, performance seems to improve somewhat compared to traditional MIDAS-history because each SQL table is stored into a separate file, compared to everything-in-one-file in traditional MIDAS-history.

== Set up a MySQL database ==

In order to setup a MySQL database, the following is needed:

* the server hostname and port number
* database name
* user name and password.

It is recommended to create 3 different users:

* a "root" user with full priveleges,
* a "reader" user with read-only priveleges
* and a "writer" user with "insert, create table, and add column" priveleges (no permission to drop table, remove columns, etc)

=== Example ===
#Setup a MySQL database (RHEL/SL 5)

ssh root@localhost
service mysqld start
/usr/bin/mysql_secure_installation (enter current password for root: press <enter>, then accept default answer "Y" to all questions)
mysqladmin -p create history
using mysql-administrator, create 2 users:
history_writer, with database "history" schema privileges: select, insert, create, alter, index
history_reader, with database "history" schema privileges: select

#Setup $HOME/.odbc.ini file:

cat $HOME/.odbc.ini
[history_writer]
Description = history_writer
Driver = /usr/lib64/libmyodbc3.so
SERVER = ladd05.triumf.ca
PORT =
Database = history
OPTION = 3
SOCKET =
User = history_writer
Password = ******
[history_reader]
Description = history_reader
Driver = /usr/lib64/libmyodbc3.so
SERVER = ladd05.triumf.ca
PORT =
Database = history
OPTION = 3
SOCKET =
User = history_reader
Password =

Note: if you are using mhist you will need a separate .odbc.ini file for each user that you have connecting to the database (through MIDAS).

== Layout of SQL tables ==

* One table is created for each history event. Table name is the same as the history event name (as reported by [[mhdump]]).
* one column is created for each history tag (history variable). Column name is the same as the tag name (as reported by mhdump). For array tags, one column is created for each array element: array a[3] will produce columns a_0, a_1 and a_2.
* two special columns are always created: _i_time and _t_time are the timestamps in the UNIX-time integer format and in the SQL "time" format. (A priori, it is not clear which timestamp format is more useful for end users. mhttpd uses the _i_time integer time format). Note that this timestamp is the time when mlogger receives the hot-link signal (see above) and it can be a few seconds behind the time when the user placed the data into the MIDAS bank. For high precision (and sub-second time resolution) time stamps, users should generate their own timestamps and include them as part of the data itself.
* a special table called _history_index is created to remember the mapping between SQL names and MIDAS names for event names, variable name and tag names.

=== Schema Evolution ===

* when new data fields are added, new SQL columns are created automatically
* existing SQL columns are never deleted
* arrays are expanded automatically, and arrays never shrink.
* if the type of a data field changes, a conflict may occur with existing SQL columns, for example if previous type was "FLOAT" and new type is "DOUBLE", the existing SQL columns would have the SQL data type "float", too narrow to store new values. mlogger will complain about this. Such conflicts may be resolved by changing the type of the SQL column using SQL tools. In general, SQL names are created from MIDAS names by replacing all non-alphanumerical characters to an underscore "_". If this procedure creates duplicate column names, the column names and made unique by appending them with a random number, i.e. MIDAS names "A+B" and "A-B" will be translated to "A_B" and "A_B_12345" (random integer). True MIDAS names are saved into the _history_index.

[[Category:history]]

History System

2020-07-30T16:04:45Z

Olchansk: /* Write MYSQL-history events */

{{Pagelinks}}

== Links ==
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[/History ODB tree]]
* [[History Page]]
* [[Mhformat|MIDAS history file format]]
* [[Keys in the ODB /Logger tree|/Logger ODB tree]]
* [[Keys in the ODB /Logger/History subtree|/Logger/History subtree]]
* [[Image History]]
</div>

<div id="history file"></div>
= Introduction =

The history system is an add-on capability built into the MIDAS data logger [[mlogger]] to record useful information in parallel to the data logging.
This information is recorded with a time stamp and saved into a '''history file''' (see [[#History drivers|History drivers]]) for later retrieval. One set of files is created per day containing all the requested history events. The history data may be displayed in graphical form using the [[mhttpd]] MIDAS webserver utility, giving the user an easy way of seeing how experimental variables have changed with time.
The history logging will be in action only if the MIDAS data logger is running, but it is not necessary to have any [[Keys in the ODB /Logger tree#/Logger/Channels subtree|data logging channel]] enabled.

Since June 2020, it is possible to also use the midas history system to record images (png/jpeg etc) periodically, for example to later produce a timelapse from a webcam - see the [[Image History]] page for more details.

= MidasHistoryInterface =

The new history configuration defines history channels similar to logger channels. Each history channel creates a history interface class (class MidasHistoryInterface). The [[mlogger|data logger]] can use this interface to write data into the history, mhttpd, mhist & co can use this interface to read data from the history. The interface is defined in history.h

* http://ladd00.triumf.ca/~daqweb/doc/midas/include/history.h

= History drivers =

Presently, the MIDAS distribution contains these drivers:

; MIDAS-History
: this is the traditional history driver, with data stored in the .hst binary files. (format is documented here [[mhformat]], see also [[mhist]] and [[mhdump]])
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#MIDAS subtree |"MIDAS"]]''' is set up to write MIDAS-history.

; ODBC-History
: stores history data in SQL database using the ODBC abstraction layer. See [[#MIDAS ODBC SQL History system|ODBC SQL History system]].

: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#ODBC subtree|"ODBC"]]''' is set up to write ODBC-history.

; SQLITE-History
: stores data in SQL database as SQLITE3 .sqlite3 files. The SQL table format is similar to ODBC-history, performance is similar, no database server required.
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#SQLITE subtree|"SQLITE"]]''' is set up to write SQLITE-history.

; MYSQL-History
:
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#MYSQL subtree|"MYSQL"]]''' is set up to write MYSQL-history.

; FILE-History
: stores each variable in its own file, making it much easier to trace the history of a particular variable.
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#FILE subtree|FILE]]''' is set up to write FILE-history.

More drivers can be easily added by the user (create history_xxx.cxx, add constructor to hs_get_history(), create the ODB subtree /Logger/History/xxx/channel).

= Location of History Files =
;MIDAS-history
:By default, the traditional MIDAS-history is enabled, writing history files into the MIDAS data directory path given by the ODB key [[Keys in the ODB /Logger tree#Data dir|Data dir]].
:If history data is important, it is recommended to write it to a directory that is backed up or archived. (The MIDAS data directory is usually located on a scratch data disk without backups).

:The location of the MIDAS-history data files can be changed by use of the ODB Key [[Keys in the ODB /Logger tree#History dir|History dir]].

;ODBC-History
:The location of ODBC-History files is ...

;SQLITE-History
:The location of SQLITE-History files is the directory path given by the ODB key [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]].

;MYSQL-History
:The location of MYSQL-History files is ...

;FILE-History
:The location of FILE-History files is the directory path given by the ODB key [[Keys in the ODB /Logger/History subtree#History dir|History dir]].

= Types of History Events =

There are two basic types of history events, which are defined in different ways:

* "[[#Frontend History Event|Frontend]]" (or "Equipment") History event composed in the frontend. See [[#Frontend History Event|Frontend History Event]] for details.
* "Virtual" (or "Links") History event composed within the ODB under the [[/History ODB tree#Links subtree|/History/Links subtree]]. See [[#Virtual History Event|Virtual History Event]] for details.

Both these history event definitions take effect when the data logger gets a start run [[Run States and Transitions|transition]]. Any modifications made during the run are not applied until the start of the next run.

== Virtual History Event ==
The history of any ODB variable can be recorded in a "virtual" history event by creating a link to that variable in the [[/History ODB tree#Links subtree|/History/Links subtree]]. This is used for ODB variables that are not in an Equipment.

; To enable Virtual History Events
: Create links as desired in [[/History ODB tree#Links subtree|/History/Links subtree]].

History events are created for each entry under ODB subdirectory {{Odbpath|path=/History/Links}}.

Two types of links are permitted:
<dl>
<dt> ODB key {{Odbpath|path=/history/links/aaa}} is a link to a subdirectory</dt>
<dd><ol>
<li> db_watch() is setup to watch this subdirectory</li>
<li> tags are created for each subdirectory entry (1 tag per entry)</li>
<li> There is no possibility for naming array elements, so 1 tag per array, regardless of the number of elements.</li> </ol></dd>
<dt>ODB key {{Odbpath|path=/history/links/bbb}} is a subdirectory with links to ODB values</dt>
<dd> <ol>
<li>db_watch is setup to watch each link target</li>
<li> tags are created for each link (1 tag per link)</li>
<li> tag name is the link name (NOT the target name). There is no possibility for naming array elements</li>
</ol></dd>
</dl>

Mixing links and subdirectories is not permitted.

The update period of history events created for {{Odbpath|path=/history/links}} is controlled by entries in {{Odbpath|path=/history/links periods}}. Numeric values of periods are same as for a [[#Frontend History Event|Frontend History event]]. Numeric value 0 disables the history for a particular event.

See [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087] for more information.

== Frontend History Event ==

Each [[Frontend Operation#Equipments and Events|Equipment]] has the capability to generate "history data".

; To enable Frontend History Events

: Set the [[/Equipment ODB tree#Log history|Log history]] ODB key of each equipment to be logged '''non-zero'''. (See [[Equipment List Parameters#Log History|Log History parameter]]). (You may need to restart the frontend.)

This causes the event contents to be copied to the ODB [[/Equipment ODB tree#Variables subtree|/Equipment/''<equipment-name>''/Variables subtree]] for the use of the History system. history events are created by parsing the content of {{Odbpath|path=/Equipment/<equipment-name>/variables}}.

For each equipment, history is controlled by the value of {{Odbpath|path=/Equipment/<equipment-name>/common/period}} as follows:
* 0 = history disabled
* 1 = history is enabled
* >1 = history is enabled, throttled down

The throttling is implemented in log_history()/watch_history() by this algorithm:
the very first history event is recorded, then all changed to the data are ignored until "period" seconds has elapsed. Then the next history event will be recorded, and following
changes will be ignored until "period" second elapses, and so forth. Period value "1" has special meaning - there is no throttling, all history events are logged.

This information is from [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087].

=== Sequence for frontend History Event ===
The main steps for the frontend History Event is as follows:

* The user code in the frontend equipment Event Readout routines reads the data, placing it into a MIDAS data bank
* In mfe.c, if the key [[/Equipment ODB tree#Log history|Log history]] > 0 (i.e. the history system is enabled for this equipment), this data bank is written into ODB (by update_odb()).
* This ODB write triggers an ODB hot-link into mlogger
* The hot-link calls mlogger.c::log_history(), which calls hs_write() to write the data into the history file. The frequency of the history writes is specified by the number of seconds between writes stored in the ODB key [[/Equipment ODB tree#Log history|Log history]].
* history.c::hs_write() or history_odbc::hs_write_odbc() writes the data into a history file or into an SQL database.

= How to Write History events =
* Enable desired [[#To enable Frontend History Events|Frontend]] and/or [[#To enable Virtual History Events|Virtual]] history events
* [[#Select the history driver|Select the history driver]]
* Follow instructions below for selected driver:
** [[#Write MIDAS-History events|Write MIDAS-history events]]
** [[#Write ODBC-History events|Write ODBC-history events]]
** [[#Write SQLITE-History events|Write SQLITE-history events]]
** [[#Write MYSQL-History events|Write MYSQL-history events]]
** [[#Write FILE-History events|Write FILE-history events]]

== Select the history driver ==

* Select the '''[[#History drivers|history driver]]''' and therefore the [[Keys in the ODB /Logger/History subtree#<history-logging-channel> subtrees|history-logging-channel]] to use to write the MIDAS-history files.
* set [[Keys in the ODB /Logger/History subtree#Active|Active]] ODB key in the selected [[Keys in the ODB /Logger/History subtree#<history-logging-channel> subtrees|history-logging-channel]] to "y". All other logging channels should be disabled.
* set [[/History ODB tree#LoggerHistoryChannel|LoggerHistoryChannel]] ODB key to the selected [[#History drivers|History channel]].

== Write MIDAS-History events ==
* Enable desired [[#To enable Frontend History Events|Frontend]] and/or [[#To enable Virtual History Events|Virtual]] history events
* Perform steps in [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "MIDAS".
* Restart [[mhttpd]] and [[mlogger]], start the run.

MIDAS-history files will be saved into the [[#Location of History Files|MIDAS-history data directory]].

== Write ODBC-History events ==

See [[#MIDAS ODBC SQL History system|MIDAS ODBC SQL History system]].
* If necessary, [[#Set up a MySQL database|set up a MySQL database]].
* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps in [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "ODBC".
* Check ODB Key [[Keys in the ODB /Logger/History subtree#Writer_ODBC_DSN|Writer_ODBC_DSN]] is set to "history writer" and Key [[Keys in the ODB /Logger/History subtree#Reader_ODBC_DSN|Reader_ODBC_DSN]] is set to "history reader" (the default values)
* set [[Keys in the ODB /Logger/History subtree#Debug|Debug]] to 1 if desired
* restart [[mlogger]] in verbose mode (i.e. mlogger -v), and observe how it issues SQL commands to create the tables and columns corresponding to MIDAS-history events and tags.
* restart [[mhttpd]]

'''Notes'''
# [[mlogger]] is programmed to raise alarms if connection to SQL database is interrupted or if some events cannot be written into the database (i.e. data type mismatch, SQL syntax errors, etc). The traditional MIDAS-history never raised alarms because it "never failed" - other than from "disk full" errors, which are immediately obvious.
# With ODBC History enabled, [[mhttpd]] will only use history information from the SQL database to make history plots and to extract history variable names for the history plot editor. The ODB {{Odbpath|path=/History}} subtrees[[/History ODB tree#Tags subtree|Tags]] and [[/History ODB tree#Events subtree|Events]] are not used.
# variable names presented to the user may change from MIDAS names to SQL names (use the history panel "Label" text fields to create permanent plot labels).

== Write SQLITE-history events ==

No special configuration required.

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "SQLITE".

* Create a directory for storage of sqlite history files
* save its path in the ODB key [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]]
* restart [[mhttpd]] and [[mlogger]]

In the history data directory (in [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]]) you should see files with name "mh_xxx.sqlite3" corresponding to your equipment and variable names.

== Write MYSQL-history events ==

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "MYSQL".
* setup the MySQL database:
* install mariadb-10.4
<pre>
macos:
sudo port install mariadb-10.4
ln -s ln -fs /opt/local/lib/mariadb-10.4/bin/mysql_config ~/bin/
mysql_config --version
10.4.13
</pre>
* rebuild midas, check the cmake finds the correct mysql: MIDAS: Found MySQL version 10.4.13
* create $HOME/.my.cnf
<pre>
[mysqld]
#port=3306
socket=/Users/olchansk/mysql/mysql.sock
datadir=/Users/olchansk/mysql/db
pid-file=/Users/olchansk/mysql/mysqld.pid

[client]
#port=3306
socket=/Users/olchansk/mysql/mysql.sock
</pre>
* mkdir $HOME/mysql
* start mysql:
<pre>
macos: d
</pre>
* configure the database: start the "mysql" utility (/opt/local/lib/mariadb-10.4/bin/mysql)
<pre>
NOTE: instructions thanks to Ben S.
NOTE: use your own password instead of "writer_password" in this example

MariaDB [(none)]> CREATE DATABASE IF NOT EXISTS history;
Query OK, 1 row affected (0.003 sec)

MariaDB [(none)]> CREATE USER history_reader;
Query OK, 0 rows affected (0.027 sec)

MariaDB [(none)]> GRANT SELECT ON history.* TO history_reader;
Query OK, 0 rows affected (0.016 sec)

MariaDB [(none)]> CREATE USER 'history_writer'@'localhost' IDENTIFIED BY 'writer_password';
Query OK, 0 rows affected (0.014 sec)

MariaDB [(none)]> GRANT SELECT,INSERT,CREATE,ALTER ON history.* TO 'history_writer'@'localhost';
Query OK, 0 rows affected (0.017 sec)
</pre>
* try connecting to both users:
<pre>
/opt/local/lib/mariadb-10.4/bin/mysql -u history_reader ### should connect without a password
/opt/local/lib/mariadb-10.4/bin/mysql -u history_writer -p ### should ask for password
</pre>
* create midas config files:
** mysql_writer.txt
<pre>
#server=localhost
#port=xxx
database=history
socket=/Users/olchansk/mysql/mysql.sock
user=writer
password=writer_password ### change this!!!
#buffer=1000
</pre>
** mysql_reader.txt
<pre>
#server=localhost
#port=xxx
database=history
socket=/Users/olchansk/mysql/mysql.sock
user=reader
#password=reader
</pre>

...........

== Write FILE-history events ==

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "SQLITE".
* Assign directory path for saved files using the ODB key [[Keys in the ODB /Logger/History subtree#History dir|History dir]].
...........

 
---------------
 

= History Events =
The following information is derived from [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087].

A "history event" is a history atomic unit of data. Associated with each history event is a timestamp (unix time),
a name (limited to NAME_LENGTH in the old history) and a list of history tags that describe the individual data
values inside the history event.

When making history plots in [[mhttpd]] (see [[History Page]]) for each curve on the plot, one selects a history event (from the list
of currently active events, recently active events or the list of all events that ever existed), then from the list of tags
inside the history event one selects the particular variable that will be plotted.

== Old MIDAS history events ==
In the old MIDAS history, all history events are written into one history file (.hst file + optional .def and .idx event definition and time index files
which can be/are regenerated automatically from the .hst file). History events are identified by 16-bit history event IDs, the persistent mapping
from history event names and the 16-bit history event IDs is stored in ODB subtree {{Odbpath|path=/History/Events}}. In addition the list of all known history event tags is
stored in ODB subtree {{Odbpath|path=/History/Tags}}. For per-equipment history, the 16-bit history event ID is the value of ODB key [[/Equipment ODB tree#Event ID|/Equipment/<equipment-name>/Common/Event ID]].

== SQL history events ==
In the SQL history (MySQL, SQLITE, etc), each history event is an SQL table. The history event tags are the SQL table columns.

== FILE history events ==
In the new FILE history, each history event is written into a separate file, tag definition are recorded in text format in the file header, history event
data is appended to the file in binary format (fixed record size). If the history event definition is changed, a new file will be started.

= History Event Construction =
The [[mlogger]] creates [[#Frontend history event]]s in open_history() by parsing the ODB subtree {{Odbpath|path=/equipment/<equipment-name>/variables}}. Each ODB entry under the {{Odbpath|path=variables/}} subtree is referred to as a "variable". Each variable can be a single ODB value, an array of ODB values, or a subdirectory (corresponding to TID_STRUCT structured data banks). As each variable is processed, one or more tags are created to describe it.

Single ODB values will generally produce a single tag, while arrays can produce (depending whether the array is "[[#Named Arrays|Named]]" or not)
* multiple tags - one per array element ("Named" array) or
* one single tag - describing the whole array

The code can generate two types of history:
<dl>
<dt> "per-equipment" history</dt>
<dd> will have the tags for all variables concatenated together into one single history event</dd>
<dt> "per-variable" history</dt>
<dd> will have one history event defined for each variable. Inside could be one tag (for single odb values and unnamed array) or multiple tags (for named arrays and structured data
banks).</dd>
</dl>

Per-equipment history is the original MIDAS history implementation. Per-variable history was added to permit efficient data storage in SQL tables. Its initial implementation used 1 ODB hotlink for each variable and it was easy to exceed the maximum permitted number of ODB hotlinks (db_open_record()). To reduce consumption of hotlinks, db_watch() has been implemented and now per-variable history only uses 1 ODB hotlink per equipment.

;NOTE
: With db_watch, per-equipment history is no longer available. '''Per-variable history is the new default''' (and the only option).

[https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087]

= Construction of history event tags =
Currently,
* single ODB values are treated as arrays of length "1"
* string ODB entries are not permitted

History event tags are constructed in the [[mlogger]] add_equipment(). Presently there are some quirks with the generation of history event tags (see [https://midas.triumf.ca/elog/Midas/1088]),
e.g. single odb values are treated as arrays of length "1" and "string" ODB entries are not permitted.

For variables of type TID_KEY (subdirectories, corresponding to TID_STRUCT structured banks), one tag is generated for each subdirectory entry. Tag names for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa/bbb}} will be "aaa_bbb" (with an underscore). Subdirectory entries of type TID_KEY and TID_LINK are currently not supported.

For "[[#Named Arrays|Named Arrays]]", individual tags are generated for each array element. Tag names are taken from the ''Names'' array. For empty tag names (empty names array), tags are "aaa_0", "aaa_1", etc (for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}}).

For "[[#Named Arrays|Single Names Arrays]]", tag names have the variable name appended (with a space), for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}} and an empty names array, tags will be "aaa_0 aaa", "aaa_1 aaa", etc. For a populated names array, the tags will be "name0 aaa", "name1 aaa", etc.

For unnamed arrays and single odb variables (in ODB, single odb variables are arrays of length 1), a single tag is generated.

== Named Arrays ==

'''Named Arrays''' have a "Names" entry in {{Odbpath|path=/equipment/<equipment-name>/settings}}.

For example, to add names to {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}}, create a string array {{Odbpath|path=/equipment/<equipment-name>/settings/names aaa}}. The names array should be at least as long as the corresponding data array. Individual entries in the names array can be left blank (tag names will be "aaa_0", "aaa_1", etc). Duplicate tag names are not permitted.

A single "Names" entry can be created to name all arrays in variables with the same names ('''"Single Names"''').

Create {{Odbpath|path=/equipment/<equipment-name>/settings/names}} and arrays {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}} and {{Odbpath|path=/equipment/<equipment-name>/variables/bbb}} will have
history tags "name0 aaa", "name1 aaa", "name0 bbb", "name1 bbb", etc. If "names" are left blank, tag names will be "aaa_0 aaa", "aaa_1 aaa", "bbb_0 bbb", "bbb_1 bbb", etc.

In the [[mhttpd]] web server [[History Page]] variables viewer, "single name" arrays are displayed in a 2D table.

This information is from [https://midas.triumf.ca/elog/Midas/1088].

= Historical note on MIDAS-History Tag names =
Tags are only used for MIDAS-history events.

The mlogger/mhttpd implementation of /History/Tags has proved troublesome and we are moving away from it.

;During the transition period:
: Implementations starting with Version 4435 of [[mlogger]] and [[mhttpd]] (i.e. "new" versions) will now work '''without''' /History/Tags. The history tags are read directly from the history files themselves. To run in this mode, set ODB key [[/History ODB tree#DisableTags|/History/DisableTags]] to "y". Existing [[/History ODB tree#Tags subtree|/History/Tags subtree]] will be automatically deleted.

Two downsides to running like this are
* it is slower and tags become non-persistent
* if some frontends have not been running for a while, their variables may vanish from the history panel editor.

== New-style History Tags ==
For the above two reasons, using /History/Tags is still recommended.
However, the format of the tags is now changed to simplify management and reduce ODB size.

; For NEW implementations
: the key [[/History ODB tree#CreateOldTags|CreateOldTags]] will not be present. "New-style" history tags will be created automatically in the [[/History ODB tree#Tags subtree|Tags subtree]].

;If you have old-style tags
: [[mlogger]] will automatically convert the old tags to this new format (this is why you should make a backup of your ODB).

** using "old" (i.e. pre Version 4435) mlogger with "new" mhttpd is OK: new mhttpd understands both formats of /History/Tags.
** using "old" mhttpd with "new" mlogger is OK provided the ODB key [[/History ODB tree#CreateOldTags|/History/CreateOldTags]] is set to "y" before starting mlogger.

== Old-style History Tags ==
Prior to versions 4435 of [[mlogger]] and [[mhttpd]], old-style history tags were generated.
See
* http://ladd00.triumf.ca/~daqweb/doc/midas-old/html/F_History_logging.html
for details.

 
---------------
 

= MIDAS ODBC SQL History system =

This section describes the internal workings of the MIDAS SQL history system.

The SQL history system implements the MIDAS history using a SQL database for data storage instead of flat binary files.

The SQL history code lives in a separate files history.h and history_sql.cxx. The present implementation uses the ODBC API for accessing SQL databases.
MySQL is fully supported, Pgsql support is partially implemented, support for arbitrary ODBC drivers should be easy to add in history_sql and most SQL operations are done using an interface class.

Preliminary testing with MySQL and PgSQL indicates that disk space requirements are the same for both storage mechanisms using flat binary files or either of the 2 SQL databases. All 3 tested storage mechanisms store the data uncompressed. (Observed gzip-1 compression ratios from CERN-ALPHA history files are 50% or better).

Preliminary testing also indicates that MySQL database is "not too slow" for use by T2K/ND280 slow control system. Using MySQL, performance seems to improve somewhat compared to traditional MIDAS-history because each SQL table is stored into a separate file, compared to everything-in-one-file in traditional MIDAS-history.

== Set up a MySQL database ==

In order to setup a MySQL database, the following is needed:

* the server hostname and port number
* database name
* user name and password.

It is recommended to create 3 different users:

* a "root" user with full priveleges,
* a "reader" user with read-only priveleges
* and a "writer" user with "insert, create table, and add column" priveleges (no permission to drop table, remove columns, etc)

=== Example ===
#Setup a MySQL database (RHEL/SL 5)

ssh root@localhost
service mysqld start
/usr/bin/mysql_secure_installation (enter current password for root: press <enter>, then accept default answer "Y" to all questions)
mysqladmin -p create history
using mysql-administrator, create 2 users:
history_writer, with database "history" schema privileges: select, insert, create, alter, index
history_reader, with database "history" schema privileges: select

#Setup $HOME/.odbc.ini file:

cat $HOME/.odbc.ini
[history_writer]
Description = history_writer
Driver = /usr/lib64/libmyodbc3.so
SERVER = ladd05.triumf.ca
PORT =
Database = history
OPTION = 3
SOCKET =
User = history_writer
Password = ******
[history_reader]
Description = history_reader
Driver = /usr/lib64/libmyodbc3.so
SERVER = ladd05.triumf.ca
PORT =
Database = history
OPTION = 3
SOCKET =
User = history_reader
Password =

Note: if you are using mhist you will need a separate .odbc.ini file for each user that you have connecting to the database (through MIDAS).

== Layout of SQL tables ==

* One table is created for each history event. Table name is the same as the history event name (as reported by [[mhdump]]).
* one column is created for each history tag (history variable). Column name is the same as the tag name (as reported by mhdump). For array tags, one column is created for each array element: array a[3] will produce columns a_0, a_1 and a_2.
* two special columns are always created: _i_time and _t_time are the timestamps in the UNIX-time integer format and in the SQL "time" format. (A priori, it is not clear which timestamp format is more useful for end users. mhttpd uses the _i_time integer time format). Note that this timestamp is the time when mlogger receives the hot-link signal (see above) and it can be a few seconds behind the time when the user placed the data into the MIDAS bank. For high precision (and sub-second time resolution) time stamps, users should generate their own timestamps and include them as part of the data itself.
* a special table called _history_index is created to remember the mapping between SQL names and MIDAS names for event names, variable name and tag names.

=== Schema Evolution ===

* when new data fields are added, new SQL columns are created automatically
* existing SQL columns are never deleted
* arrays are expanded automatically, and arrays never shrink.
* if the type of a data field changes, a conflict may occur with existing SQL columns, for example if previous type was "FLOAT" and new type is "DOUBLE", the existing SQL columns would have the SQL data type "float", too narrow to store new values. mlogger will complain about this. Such conflicts may be resolved by changing the type of the SQL column using SQL tools. In general, SQL names are created from MIDAS names by replacing all non-alphanumerical characters to an underscore "_". If this procedure creates duplicate column names, the column names and made unique by appending them with a random number, i.e. MIDAS names "A+B" and "A-B" will be translated to "A_B" and "A_B_12345" (random integer). True MIDAS names are saved into the _history_index.

[[Category:history]]

History System

2020-07-30T15:45:52Z

Olchansk: /* Write MYSQL-history events */

{{Pagelinks}}

== Links ==
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[/History ODB tree]]
* [[History Page]]
* [[Mhformat|MIDAS history file format]]
* [[Keys in the ODB /Logger tree|/Logger ODB tree]]
* [[Keys in the ODB /Logger/History subtree|/Logger/History subtree]]
* [[Image History]]
</div>

<div id="history file"></div>
= Introduction =

The history system is an add-on capability built into the MIDAS data logger [[mlogger]] to record useful information in parallel to the data logging.
This information is recorded with a time stamp and saved into a '''history file''' (see [[#History drivers|History drivers]]) for later retrieval. One set of files is created per day containing all the requested history events. The history data may be displayed in graphical form using the [[mhttpd]] MIDAS webserver utility, giving the user an easy way of seeing how experimental variables have changed with time.
The history logging will be in action only if the MIDAS data logger is running, but it is not necessary to have any [[Keys in the ODB /Logger tree#/Logger/Channels subtree|data logging channel]] enabled.

Since June 2020, it is possible to also use the midas history system to record images (png/jpeg etc) periodically, for example to later produce a timelapse from a webcam - see the [[Image History]] page for more details.

= MidasHistoryInterface =

The new history configuration defines history channels similar to logger channels. Each history channel creates a history interface class (class MidasHistoryInterface). The [[mlogger|data logger]] can use this interface to write data into the history, mhttpd, mhist & co can use this interface to read data from the history. The interface is defined in history.h

* http://ladd00.triumf.ca/~daqweb/doc/midas/include/history.h

= History drivers =

Presently, the MIDAS distribution contains these drivers:

; MIDAS-History
: this is the traditional history driver, with data stored in the .hst binary files. (format is documented here [[mhformat]], see also [[mhist]] and [[mhdump]])
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#MIDAS subtree |"MIDAS"]]''' is set up to write MIDAS-history.

; ODBC-History
: stores history data in SQL database using the ODBC abstraction layer. See [[#MIDAS ODBC SQL History system|ODBC SQL History system]].

: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#ODBC subtree|"ODBC"]]''' is set up to write ODBC-history.

; SQLITE-History
: stores data in SQL database as SQLITE3 .sqlite3 files. The SQL table format is similar to ODBC-history, performance is similar, no database server required.
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#SQLITE subtree|"SQLITE"]]''' is set up to write SQLITE-history.

; MYSQL-History
:
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#MYSQL subtree|"MYSQL"]]''' is set up to write MYSQL-history.

; FILE-History
: stores each variable in its own file, making it much easier to trace the history of a particular variable.
: By default ODB subtree '''[[Keys in the ODB /Logger/History subtree#FILE subtree|FILE]]''' is set up to write FILE-history.

More drivers can be easily added by the user (create history_xxx.cxx, add constructor to hs_get_history(), create the ODB subtree /Logger/History/xxx/channel).

= Location of History Files =
;MIDAS-history
:By default, the traditional MIDAS-history is enabled, writing history files into the MIDAS data directory path given by the ODB key [[Keys in the ODB /Logger tree#Data dir|Data dir]].
:If history data is important, it is recommended to write it to a directory that is backed up or archived. (The MIDAS data directory is usually located on a scratch data disk without backups).

:The location of the MIDAS-history data files can be changed by use of the ODB Key [[Keys in the ODB /Logger tree#History dir|History dir]].

;ODBC-History
:The location of ODBC-History files is ...

;SQLITE-History
:The location of SQLITE-History files is the directory path given by the ODB key [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]].

;MYSQL-History
:The location of MYSQL-History files is ...

;FILE-History
:The location of FILE-History files is the directory path given by the ODB key [[Keys in the ODB /Logger/History subtree#History dir|History dir]].

= Types of History Events =

There are two basic types of history events, which are defined in different ways:

* "[[#Frontend History Event|Frontend]]" (or "Equipment") History event composed in the frontend. See [[#Frontend History Event|Frontend History Event]] for details.
* "Virtual" (or "Links") History event composed within the ODB under the [[/History ODB tree#Links subtree|/History/Links subtree]]. See [[#Virtual History Event|Virtual History Event]] for details.

Both these history event definitions take effect when the data logger gets a start run [[Run States and Transitions|transition]]. Any modifications made during the run are not applied until the start of the next run.

== Virtual History Event ==
The history of any ODB variable can be recorded in a "virtual" history event by creating a link to that variable in the [[/History ODB tree#Links subtree|/History/Links subtree]]. This is used for ODB variables that are not in an Equipment.

; To enable Virtual History Events
: Create links as desired in [[/History ODB tree#Links subtree|/History/Links subtree]].

History events are created for each entry under ODB subdirectory {{Odbpath|path=/History/Links}}.

Two types of links are permitted:
<dl>
<dt> ODB key {{Odbpath|path=/history/links/aaa}} is a link to a subdirectory</dt>
<dd><ol>
<li> db_watch() is setup to watch this subdirectory</li>
<li> tags are created for each subdirectory entry (1 tag per entry)</li>
<li> There is no possibility for naming array elements, so 1 tag per array, regardless of the number of elements.</li> </ol></dd>
<dt>ODB key {{Odbpath|path=/history/links/bbb}} is a subdirectory with links to ODB values</dt>
<dd> <ol>
<li>db_watch is setup to watch each link target</li>
<li> tags are created for each link (1 tag per link)</li>
<li> tag name is the link name (NOT the target name). There is no possibility for naming array elements</li>
</ol></dd>
</dl>

Mixing links and subdirectories is not permitted.

The update period of history events created for {{Odbpath|path=/history/links}} is controlled by entries in {{Odbpath|path=/history/links periods}}. Numeric values of periods are same as for a [[#Frontend History Event|Frontend History event]]. Numeric value 0 disables the history for a particular event.

See [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087] for more information.

== Frontend History Event ==

Each [[Frontend Operation#Equipments and Events|Equipment]] has the capability to generate "history data".

; To enable Frontend History Events

: Set the [[/Equipment ODB tree#Log history|Log history]] ODB key of each equipment to be logged '''non-zero'''. (See [[Equipment List Parameters#Log History|Log History parameter]]). (You may need to restart the frontend.)

This causes the event contents to be copied to the ODB [[/Equipment ODB tree#Variables subtree|/Equipment/''<equipment-name>''/Variables subtree]] for the use of the History system. history events are created by parsing the content of {{Odbpath|path=/Equipment/<equipment-name>/variables}}.

For each equipment, history is controlled by the value of {{Odbpath|path=/Equipment/<equipment-name>/common/period}} as follows:
* 0 = history disabled
* 1 = history is enabled
* >1 = history is enabled, throttled down

The throttling is implemented in log_history()/watch_history() by this algorithm:
the very first history event is recorded, then all changed to the data are ignored until "period" seconds has elapsed. Then the next history event will be recorded, and following
changes will be ignored until "period" second elapses, and so forth. Period value "1" has special meaning - there is no throttling, all history events are logged.

This information is from [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087].

=== Sequence for frontend History Event ===
The main steps for the frontend History Event is as follows:

* The user code in the frontend equipment Event Readout routines reads the data, placing it into a MIDAS data bank
* In mfe.c, if the key [[/Equipment ODB tree#Log history|Log history]] > 0 (i.e. the history system is enabled for this equipment), this data bank is written into ODB (by update_odb()).
* This ODB write triggers an ODB hot-link into mlogger
* The hot-link calls mlogger.c::log_history(), which calls hs_write() to write the data into the history file. The frequency of the history writes is specified by the number of seconds between writes stored in the ODB key [[/Equipment ODB tree#Log history|Log history]].
* history.c::hs_write() or history_odbc::hs_write_odbc() writes the data into a history file or into an SQL database.

= How to Write History events =
* Enable desired [[#To enable Frontend History Events|Frontend]] and/or [[#To enable Virtual History Events|Virtual]] history events
* [[#Select the history driver|Select the history driver]]
* Follow instructions below for selected driver:
** [[#Write MIDAS-History events|Write MIDAS-history events]]
** [[#Write ODBC-History events|Write ODBC-history events]]
** [[#Write SQLITE-History events|Write SQLITE-history events]]
** [[#Write MYSQL-History events|Write MYSQL-history events]]
** [[#Write FILE-History events|Write FILE-history events]]

== Select the history driver ==

* Select the '''[[#History drivers|history driver]]''' and therefore the [[Keys in the ODB /Logger/History subtree#<history-logging-channel> subtrees|history-logging-channel]] to use to write the MIDAS-history files.
* set [[Keys in the ODB /Logger/History subtree#Active|Active]] ODB key in the selected [[Keys in the ODB /Logger/History subtree#<history-logging-channel> subtrees|history-logging-channel]] to "y". All other logging channels should be disabled.
* set [[/History ODB tree#LoggerHistoryChannel|LoggerHistoryChannel]] ODB key to the selected [[#History drivers|History channel]].

== Write MIDAS-History events ==
* Enable desired [[#To enable Frontend History Events|Frontend]] and/or [[#To enable Virtual History Events|Virtual]] history events
* Perform steps in [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "MIDAS".
* Restart [[mhttpd]] and [[mlogger]], start the run.

MIDAS-history files will be saved into the [[#Location of History Files|MIDAS-history data directory]].

== Write ODBC-History events ==

See [[#MIDAS ODBC SQL History system|MIDAS ODBC SQL History system]].
* If necessary, [[#Set up a MySQL database|set up a MySQL database]].
* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps in [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "ODBC".
* Check ODB Key [[Keys in the ODB /Logger/History subtree#Writer_ODBC_DSN|Writer_ODBC_DSN]] is set to "history writer" and Key [[Keys in the ODB /Logger/History subtree#Reader_ODBC_DSN|Reader_ODBC_DSN]] is set to "history reader" (the default values)
* set [[Keys in the ODB /Logger/History subtree#Debug|Debug]] to 1 if desired
* restart [[mlogger]] in verbose mode (i.e. mlogger -v), and observe how it issues SQL commands to create the tables and columns corresponding to MIDAS-history events and tags.
* restart [[mhttpd]]

'''Notes'''
# [[mlogger]] is programmed to raise alarms if connection to SQL database is interrupted or if some events cannot be written into the database (i.e. data type mismatch, SQL syntax errors, etc). The traditional MIDAS-history never raised alarms because it "never failed" - other than from "disk full" errors, which are immediately obvious.
# With ODBC History enabled, [[mhttpd]] will only use history information from the SQL database to make history plots and to extract history variable names for the history plot editor. The ODB {{Odbpath|path=/History}} subtrees[[/History ODB tree#Tags subtree|Tags]] and [[/History ODB tree#Events subtree|Events]] are not used.
# variable names presented to the user may change from MIDAS names to SQL names (use the history panel "Label" text fields to create permanent plot labels).

== Write SQLITE-history events ==

No special configuration required.

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "SQLITE".

* Create a directory for storage of sqlite history files
* save its path in the ODB key [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]]
* restart [[mhttpd]] and [[mlogger]]

In the history data directory (in [[Keys in the ODB /Logger/History subtree#Sqlite dir|Sqlite dir]]) you should see files with name "mh_xxx.sqlite3" corresponding to your equipment and variable names.

== Write MYSQL-history events ==

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "MYSQL".
...........

== Write FILE-history events ==

* Enable desired virtual and frontend history events (see [[#Virtual History Event|Virtual history events]] and [[#Frontend History Event|Frontend history events]]).
* Perform steps to [[#Select the history driver|Select History Driver]] with [[#History drivers|history driver]] "SQLITE".
* Assign directory path for saved files using the ODB key [[Keys in the ODB /Logger/History subtree#History dir|History dir]].
...........

 
---------------
 

= History Events =
The following information is derived from [https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087].

A "history event" is a history atomic unit of data. Associated with each history event is a timestamp (unix time),
a name (limited to NAME_LENGTH in the old history) and a list of history tags that describe the individual data
values inside the history event.

When making history plots in [[mhttpd]] (see [[History Page]]) for each curve on the plot, one selects a history event (from the list
of currently active events, recently active events or the list of all events that ever existed), then from the list of tags
inside the history event one selects the particular variable that will be plotted.

== Old MIDAS history events ==
In the old MIDAS history, all history events are written into one history file (.hst file + optional .def and .idx event definition and time index files
which can be/are regenerated automatically from the .hst file). History events are identified by 16-bit history event IDs, the persistent mapping
from history event names and the 16-bit history event IDs is stored in ODB subtree {{Odbpath|path=/History/Events}}. In addition the list of all known history event tags is
stored in ODB subtree {{Odbpath|path=/History/Tags}}. For per-equipment history, the 16-bit history event ID is the value of ODB key [[/Equipment ODB tree#Event ID|/Equipment/<equipment-name>/Common/Event ID]].

== SQL history events ==
In the SQL history (MySQL, SQLITE, etc), each history event is an SQL table. The history event tags are the SQL table columns.

== FILE history events ==
In the new FILE history, each history event is written into a separate file, tag definition are recorded in text format in the file header, history event
data is appended to the file in binary format (fixed record size). If the history event definition is changed, a new file will be started.

= History Event Construction =
The [[mlogger]] creates [[#Frontend history event]]s in open_history() by parsing the ODB subtree {{Odbpath|path=/equipment/<equipment-name>/variables}}. Each ODB entry under the {{Odbpath|path=variables/}} subtree is referred to as a "variable". Each variable can be a single ODB value, an array of ODB values, or a subdirectory (corresponding to TID_STRUCT structured data banks). As each variable is processed, one or more tags are created to describe it.

Single ODB values will generally produce a single tag, while arrays can produce (depending whether the array is "[[#Named Arrays|Named]]" or not)
* multiple tags - one per array element ("Named" array) or
* one single tag - describing the whole array

The code can generate two types of history:
<dl>
<dt> "per-equipment" history</dt>
<dd> will have the tags for all variables concatenated together into one single history event</dd>
<dt> "per-variable" history</dt>
<dd> will have one history event defined for each variable. Inside could be one tag (for single odb values and unnamed array) or multiple tags (for named arrays and structured data
banks).</dd>
</dl>

Per-equipment history is the original MIDAS history implementation. Per-variable history was added to permit efficient data storage in SQL tables. Its initial implementation used 1 ODB hotlink for each variable and it was easy to exceed the maximum permitted number of ODB hotlinks (db_open_record()). To reduce consumption of hotlinks, db_watch() has been implemented and now per-variable history only uses 1 ODB hotlink per equipment.

;NOTE
: With db_watch, per-equipment history is no longer available. '''Per-variable history is the new default''' (and the only option).

[https://midas.triumf.ca/elog/Midas/1087 MidasElog#1087]

= Construction of history event tags =
Currently,
* single ODB values are treated as arrays of length "1"
* string ODB entries are not permitted

History event tags are constructed in the [[mlogger]] add_equipment(). Presently there are some quirks with the generation of history event tags (see [https://midas.triumf.ca/elog/Midas/1088]),
e.g. single odb values are treated as arrays of length "1" and "string" ODB entries are not permitted.

For variables of type TID_KEY (subdirectories, corresponding to TID_STRUCT structured banks), one tag is generated for each subdirectory entry. Tag names for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa/bbb}} will be "aaa_bbb" (with an underscore). Subdirectory entries of type TID_KEY and TID_LINK are currently not supported.

For "[[#Named Arrays|Named Arrays]]", individual tags are generated for each array element. Tag names are taken from the ''Names'' array. For empty tag names (empty names array), tags are "aaa_0", "aaa_1", etc (for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}}).

For "[[#Named Arrays|Single Names Arrays]]", tag names have the variable name appended (with a space), for {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}} and an empty names array, tags will be "aaa_0 aaa", "aaa_1 aaa", etc. For a populated names array, the tags will be "name0 aaa", "name1 aaa", etc.

For unnamed arrays and single odb variables (in ODB, single odb variables are arrays of length 1), a single tag is generated.

== Named Arrays ==

'''Named Arrays''' have a "Names" entry in {{Odbpath|path=/equipment/<equipment-name>/settings}}.

For example, to add names to {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}}, create a string array {{Odbpath|path=/equipment/<equipment-name>/settings/names aaa}}. The names array should be at least as long as the corresponding data array. Individual entries in the names array can be left blank (tag names will be "aaa_0", "aaa_1", etc). Duplicate tag names are not permitted.

A single "Names" entry can be created to name all arrays in variables with the same names ('''"Single Names"''').

Create {{Odbpath|path=/equipment/<equipment-name>/settings/names}} and arrays {{Odbpath|path=/equipment/<equipment-name>/variables/aaa}} and {{Odbpath|path=/equipment/<equipment-name>/variables/bbb}} will have
history tags "name0 aaa", "name1 aaa", "name0 bbb", "name1 bbb", etc. If "names" are left blank, tag names will be "aaa_0 aaa", "aaa_1 aaa", "bbb_0 bbb", "bbb_1 bbb", etc.

In the [[mhttpd]] web server [[History Page]] variables viewer, "single name" arrays are displayed in a 2D table.

This information is from [https://midas.triumf.ca/elog/Midas/1088].

= Historical note on MIDAS-History Tag names =
Tags are only used for MIDAS-history events.

The mlogger/mhttpd implementation of /History/Tags has proved troublesome and we are moving away from it.

;During the transition period:
: Implementations starting with Version 4435 of [[mlogger]] and [[mhttpd]] (i.e. "new" versions) will now work '''without''' /History/Tags. The history tags are read directly from the history files themselves. To run in this mode, set ODB key [[/History ODB tree#DisableTags|/History/DisableTags]] to "y". Existing [[/History ODB tree#Tags subtree|/History/Tags subtree]] will be automatically deleted.

Two downsides to running like this are
* it is slower and tags become non-persistent
* if some frontends have not been running for a while, their variables may vanish from the history panel editor.

== New-style History Tags ==
For the above two reasons, using /History/Tags is still recommended.
However, the format of the tags is now changed to simplify management and reduce ODB size.

; For NEW implementations
: the key [[/History ODB tree#CreateOldTags|CreateOldTags]] will not be present. "New-style" history tags will be created automatically in the [[/History ODB tree#Tags subtree|Tags subtree]].

;If you have old-style tags
: [[mlogger]] will automatically convert the old tags to this new format (this is why you should make a backup of your ODB).

** using "old" (i.e. pre Version 4435) mlogger with "new" mhttpd is OK: new mhttpd understands both formats of /History/Tags.
** using "old" mhttpd with "new" mlogger is OK provided the ODB key [[/History ODB tree#CreateOldTags|/History/CreateOldTags]] is set to "y" before starting mlogger.

== Old-style History Tags ==
Prior to versions 4435 of [[mlogger]] and [[mhttpd]], old-style history tags were generated.
See
* http://ladd00.triumf.ca/~daqweb/doc/midas-old/html/F_History_logging.html
for details.

 
---------------
 

= MIDAS ODBC SQL History system =

This section describes the internal workings of the MIDAS SQL history system.

The SQL history system implements the MIDAS history using a SQL database for data storage instead of flat binary files.

The SQL history code lives in a separate files history.h and history_sql.cxx. The present implementation uses the ODBC API for accessing SQL databases.
MySQL is fully supported, Pgsql support is partially implemented, support for arbitrary ODBC drivers should be easy to add in history_sql and most SQL operations are done using an interface class.

Preliminary testing with MySQL and PgSQL indicates that disk space requirements are the same for both storage mechanisms using flat binary files or either of the 2 SQL databases. All 3 tested storage mechanisms store the data uncompressed. (Observed gzip-1 compression ratios from CERN-ALPHA history files are 50% or better).

Preliminary testing also indicates that MySQL database is "not too slow" for use by T2K/ND280 slow control system. Using MySQL, performance seems to improve somewhat compared to traditional MIDAS-history because each SQL table is stored into a separate file, compared to everything-in-one-file in traditional MIDAS-history.

== Set up a MySQL database ==

In order to setup a MySQL database, the following is needed:

* the server hostname and port number
* database name
* user name and password.

It is recommended to create 3 different users:

* a "root" user with full priveleges,
* a "reader" user with read-only priveleges
* and a "writer" user with "insert, create table, and add column" priveleges (no permission to drop table, remove columns, etc)

=== Example ===
#Setup a MySQL database (RHEL/SL 5)

ssh root@localhost
service mysqld start
/usr/bin/mysql_secure_installation (enter current password for root: press <enter>, then accept default answer "Y" to all questions)
mysqladmin -p create history
using mysql-administrator, create 2 users:
history_writer, with database "history" schema privileges: select, insert, create, alter, index
history_reader, with database "history" schema privileges: select

#Setup $HOME/.odbc.ini file:

cat $HOME/.odbc.ini
[history_writer]
Description = history_writer
Driver = /usr/lib64/libmyodbc3.so
SERVER = ladd05.triumf.ca
PORT =
Database = history
OPTION = 3
SOCKET =
User = history_writer
Password = ******
[history_reader]
Description = history_reader
Driver = /usr/lib64/libmyodbc3.so
SERVER = ladd05.triumf.ca
PORT =
Database = history
OPTION = 3
SOCKET =
User = history_reader
Password =

Note: if you are using mhist you will need a separate .odbc.ini file for each user that you have connecting to the database (through MIDAS).

== Layout of SQL tables ==

* One table is created for each history event. Table name is the same as the history event name (as reported by [[mhdump]]).
* one column is created for each history tag (history variable). Column name is the same as the tag name (as reported by mhdump). For array tags, one column is created for each array element: array a[3] will produce columns a_0, a_1 and a_2.
* two special columns are always created: _i_time and _t_time are the timestamps in the UNIX-time integer format and in the SQL "time" format. (A priori, it is not clear which timestamp format is more useful for end users. mhttpd uses the _i_time integer time format). Note that this timestamp is the time when mlogger receives the hot-link signal (see above) and it can be a few seconds behind the time when the user placed the data into the MIDAS bank. For high precision (and sub-second time resolution) time stamps, users should generate their own timestamps and include them as part of the data itself.
* a special table called _history_index is created to remember the mapping between SQL names and MIDAS names for event names, variable name and tag names.

=== Schema Evolution ===

* when new data fields are added, new SQL columns are created automatically
* existing SQL columns are never deleted
* arrays are expanded automatically, and arrays never shrink.
* if the type of a data field changes, a conflict may occur with existing SQL columns, for example if previous type was "FLOAT" and new type is "DOUBLE", the existing SQL columns would have the SQL data type "float", too narrow to store new values. mlogger will complain about this. Such conflicts may be resolved by changing the type of the SQL column using SQL tools. In general, SQL names are created from MIDAS names by replacing all non-alphanumerical characters to an underscore "_". If this procedure creates duplicate column names, the column names and made unique by appending them with a random number, i.e. MIDAS names "A+B" and "A-B" will be translated to "A_B" and "A_B_12345" (random integer). True MIDAS names are saved into the _history_index.

[[Category:history]]

Main Page

2020-07-11T15:23:49Z

Olchansk: /* Related packages */

= Introduction =

MIDAS is a modern data acquisition system developed at [http://www.psi.ch PSI] and [http://www.triumf.ca TRIUMF]. Supported hardware includes VME, Fastbus, CAMAC, RS232, GPIB, USB, ethernet, fiber optic and MSCB attached data acquisition devices. MIDAS is written in C/C++ and runs on Linux, MacOS and MS Windows. It is licensed under the [http://www.gnu.org/copyleft/gpl.html GNU Public License].

= Documentation =

* [[Midas documentation| Main Documentation]]
* [[Installation | Installation Instructions]]
* [[Changelog | Changelog / Upgrade Instructions]]
* [https://daq.triumf.ca/~daqweb/doc/midas-devel/html/modules.html Doxygen] Code structure, functions, etc...
* [https://bitbucket.com/tmidas/midas git repository on bitbucket] Source code, bug tracking, etc
* [https://midas.triumf.ca/forum MIDAS forum] Discussion, announcements, problem reports, requests for improvements

= News and Support=

For news about bug fixes and new releases, go to the
[https://midas.triumf.ca/forum MIDAS forum] hosted by TRIUMF using
the [http://elog.psi.ch/elog ELOG] system. Users can register in this system to be notified automatically via
E-mail when new entries are submitted.

Another source of information is the [https://bitbucket.com/tmidas/midas Bitbucket Repository] of MIDAS, where one can see the latest changes to the software.

Finally, you can read presentations from
* [https://indico.psi.ch/conferenceDisplay.py?ovw=True&confId=3793 2015 MIDAS Workshop]
* [https://indico.triumf.ca/conferenceDisplay.py?confId=2342 2017 MIDAS Workshop]
* [https://indico.triumf.ca/conferenceDisplay.py?confId=2438 2019 MIDAS Workshop]

= Related packages =

* [http://elog.psi.ch/elog ELOG] - Electronics Logbook from PSI
* [[ROOT]] - data analysis package from CERN
* [[ROOTANA]] - ROOT-based analyzer for MIDAS
* [[ROODY]] - viewer for online histograms
* http://elog.psi.ch/rome/ - MEG/PSI data analysis package

Main Page

2020-04-26T14:54:57Z

Olchansk: /* News and Support */

= Introduction =

MIDAS is a modern data acquisition system developed at [http://www.psi.ch PSI] and [http://www.triumf.ca TRIUMF]. Supported hardware includes VME, Fastbus, CAMAC, RS232, GPIB, USB, ethernet, fiber optic and MSCB attached data acquisition devices. MIDAS is written in C/C++ and runs on Linux, MacOS and MS Windows. It is licensed under the [http://www.gnu.org/copyleft/gpl.html GNU Public License].

= Documentation =

* [[Midas documentation| Main Documentation]]
* [[Installation | Installation Instructions]]
* [[Changelog | Changelog / Upgrade Instructions]]
* [https://daq.triumf.ca/~daqweb/doc/midas-devel/html/modules.html Doxygen] Code structure, functions, etc...
* [https://bitbucket.com/tmidas/midas git repository on bitbucket] Source code, bug tracking, etc
* [https://midas.triumf.ca/forum MIDAS forum] Discussion, announcements, problem reports, requests for improvements

= News and Support=

For news about bug fixes and new releases, go to the
[https://midas.triumf.ca/forum MIDAS forum] hosted by TRIUMF using
the [http://elog.psi.ch/elog ELOG] system. Users can register in this system to be notified automatically via
E-mail when new entries are submitted.

Another source of information is the [https://bitbucket.com/tmidas/midas Bitbucket Repository] of MIDAS, where one can see the latest changes to the software.

Finally, you can read presentations from
* [https://indico.psi.ch/conferenceDisplay.py?ovw=True&confId=3793 2015 MIDAS Workshop]
* [https://indico.triumf.ca/conferenceDisplay.py?confId=2342 2017 MIDAS Workshop]
* [https://indico.triumf.ca/conferenceDisplay.py?confId=2438 2019 MIDAS Workshop]

= Related packages =

* [http://midas.psi.ch/elog ELOG] - Electronics Logbook from PSI
* [[ROOT]] - data analysis package from CERN
* [[ROOTANA]] - ROOT-based analyzer for MIDAS
* [[ROODY]] - viewer for online histograms
* http://elog.psi.ch/rome/ - MEG/PSI data analysis package

Main Page

2020-04-26T14:54:31Z

Olchansk: /* News and Support */

= Introduction =

MIDAS is a modern data acquisition system developed at [http://www.psi.ch PSI] and [http://www.triumf.ca TRIUMF]. Supported hardware includes VME, Fastbus, CAMAC, RS232, GPIB, USB, ethernet, fiber optic and MSCB attached data acquisition devices. MIDAS is written in C/C++ and runs on Linux, MacOS and MS Windows. It is licensed under the [http://www.gnu.org/copyleft/gpl.html GNU Public License].

= Documentation =

* [[Midas documentation| Main Documentation]]
* [[Installation | Installation Instructions]]
* [[Changelog | Changelog / Upgrade Instructions]]
* [https://daq.triumf.ca/~daqweb/doc/midas-devel/html/modules.html Doxygen] Code structure, functions, etc...
* [https://bitbucket.com/tmidas/midas git repository on bitbucket] Source code, bug tracking, etc
* [https://midas.triumf.ca/forum MIDAS forum] Discussion, announcements, problem reports, requests for improvements

= News and Support=

For news about bug fixes and new releases of Midas go to the
[https://midas.triumf.ca/forum Midas forum] hosted by TRIUMF using
the [http://elog.psi.ch/elog ELOG] system. Users can register in this system to be notified automatically via
E-mail when new entries are submitted.

Another source of information is the [https://bitbucket.com/tmidas/midas Bitbucket Repository] of MIDAS, where one can see the latest changes to the software.

Finally, you can read presentations from
* [https://indico.psi.ch/conferenceDisplay.py?ovw=True&confId=3793 2015 MIDAS Workshop]
* [https://indico.triumf.ca/conferenceDisplay.py?confId=2342 2017 MIDAS Workshop]
* [https://indico.triumf.ca/conferenceDisplay.py?confId=2438 2019 MIDAS Workshop]

= Related packages =

* [http://midas.psi.ch/elog ELOG] - Electronics Logbook from PSI
* [[ROOT]] - data analysis package from CERN
* [[ROOTANA]] - ROOT-based analyzer for MIDAS
* [[ROODY]] - viewer for online histograms
* http://elog.psi.ch/rome/ - MEG/PSI data analysis package

Main Page

2020-04-26T14:52:48Z

Olchansk: /* Documentation */

= Introduction =

MIDAS is a modern data acquisition system developed at [http://www.psi.ch PSI] and [http://www.triumf.ca TRIUMF]. Supported hardware includes VME, Fastbus, CAMAC, RS232, GPIB, USB, ethernet, fiber optic and MSCB attached data acquisition devices. MIDAS is written in C/C++ and runs on Linux, MacOS and MS Windows. It is licensed under the [http://www.gnu.org/copyleft/gpl.html GNU Public License].

= Documentation =

* [[Midas documentation| Main Documentation]]
* [[Installation | Installation Instructions]]
* [[Changelog | Changelog / Upgrade Instructions]]
* [https://daq.triumf.ca/~daqweb/doc/midas-devel/html/modules.html Doxygen] Code structure, functions, etc...
* [https://bitbucket.com/tmidas/midas git repository on bitbucket] Source code, bug tracking, etc
* [https://midas.triumf.ca/forum MIDAS forum] Discussion, announcements, problem reports, requests for improvements

= News and Support=

News about bug fixes and new releases of Midas can be obtained from the
[https://midas.triumf.ca/forum Midas news group] located at
TRIUMF which uses the [http://elog.psi.ch/elog ELOG]
system. Users can register in this system to be notified automatically via
E-mail when new entries are submitted.

Another source of information is the [https://bitbucket.com/tmidas/midas Bitbucket Repository] of MIDAS, where one can see the latest changes to the software.

Finally, you can read presentations from
* [https://indico.psi.ch/conferenceDisplay.py?ovw=True&confId=3793 2015 MIDAS Workshop]
* [https://indico.triumf.ca/conferenceDisplay.py?confId=2342 2017 MIDAS Workshop]
* [https://indico.triumf.ca/conferenceDisplay.py?confId=2438 2019 MIDAS Workshop]

= Related packages =

* [http://midas.psi.ch/elog ELOG] - Electronics Logbook from PSI
* [[ROOT]] - data analysis package from CERN
* [[ROOTANA]] - ROOT-based analyzer for MIDAS
* [[ROODY]] - viewer for online histograms
* http://elog.psi.ch/rome/ - MEG/PSI data analysis package

Main Page

2020-04-26T14:51:40Z

Olchansk: /* Documentation */

= Introduction =

MIDAS is a modern data acquisition system developed at [http://www.psi.ch PSI] and [http://www.triumf.ca TRIUMF]. Supported hardware includes VME, Fastbus, CAMAC, RS232, GPIB, USB, ethernet, fiber optic and MSCB attached data acquisition devices. MIDAS is written in C/C++ and runs on Linux, MacOS and MS Windows. It is licensed under the [http://www.gnu.org/copyleft/gpl.html GNU Public License].

= Documentation =

* [[Midas documentation| Main Documentation]]
* [[Installation | Installation Instructions]]
* [[Changelog | Changelog / Upgrade Instructions]]
* [https://daq.triumf.ca/~daqweb/doc/midas-devel/html/modules.html Doxygen] Code structure, functions, etc...
* [https://bitbucket.com/tmidas/midas git repository on bitbucket]
* [https://midas.triumf.ca/forum MIDAS forum]

= News and Support=

News about bug fixes and new releases of Midas can be obtained from the
[https://midas.triumf.ca/forum Midas news group] located at
TRIUMF which uses the [http://elog.psi.ch/elog ELOG]
system. Users can register in this system to be notified automatically via
E-mail when new entries are submitted.

Another source of information is the [https://bitbucket.com/tmidas/midas Bitbucket Repository] of MIDAS, where one can see the latest changes to the software.

Finally, you can read presentations from
* [https://indico.psi.ch/conferenceDisplay.py?ovw=True&confId=3793 2015 MIDAS Workshop]
* [https://indico.triumf.ca/conferenceDisplay.py?confId=2342 2017 MIDAS Workshop]
* [https://indico.triumf.ca/conferenceDisplay.py?confId=2438 2019 MIDAS Workshop]

= Related packages =

* [http://midas.psi.ch/elog ELOG] - Electronics Logbook from PSI
* [[ROOT]] - data analysis package from CERN
* [[ROOTANA]] - ROOT-based analyzer for MIDAS
* [[ROODY]] - viewer for online histograms
* http://elog.psi.ch/rome/ - MEG/PSI data analysis package

Main Page

2020-04-26T14:50:15Z

Olchansk: /* Documentation */

= Introduction =

MIDAS is a modern data acquisition system developed at [http://www.psi.ch PSI] and [http://www.triumf.ca TRIUMF]. Supported hardware includes VME, Fastbus, CAMAC, RS232, GPIB, USB, ethernet, fiber optic and MSCB attached data acquisition devices. MIDAS is written in C/C++ and runs on Linux, MacOS and MS Windows. It is licensed under the [http://www.gnu.org/copyleft/gpl.html GNU Public License].

= Documentation =

* [[Midas documentation| Main Documentation]]
* [[Installation | Installation Instructions]]
* [[Changelog | Changelog / Upgrade Instructions]]
* [https://daq.triumf.ca/~daqweb/doc/midas-devel/html/modules.html Doxygen] Code structure, functions, etc...
* [https://bitbucket.com/tmidas/midas git repository on bitbucket]

= News and Support=

News about bug fixes and new releases of Midas can be obtained from the
[https://midas.triumf.ca/forum Midas news group] located at
TRIUMF which uses the [http://elog.psi.ch/elog ELOG]
system. Users can register in this system to be notified automatically via
E-mail when new entries are submitted.

Another source of information is the [https://bitbucket.com/tmidas/midas Bitbucket Repository] of MIDAS, where one can see the latest changes to the software.

Finally, you can read presentations from
* [https://indico.psi.ch/conferenceDisplay.py?ovw=True&confId=3793 2015 MIDAS Workshop]
* [https://indico.triumf.ca/conferenceDisplay.py?confId=2342 2017 MIDAS Workshop]
* [https://indico.triumf.ca/conferenceDisplay.py?confId=2438 2019 MIDAS Workshop]

= Related packages =

* [http://midas.psi.ch/elog ELOG] - Electronics Logbook from PSI
* [[ROOT]] - data analysis package from CERN
* [[ROOTANA]] - ROOT-based analyzer for MIDAS
* [[ROODY]] - viewer for online histograms
* http://elog.psi.ch/rome/ - MEG/PSI data analysis package

User talk:Marco.francesconi

2020-03-31T15:46:49Z

Olchansk: Welcome!

User talk:Jmckenna

2019-12-02T22:34:56Z

Olchansk: Welcome!

User talk:Bsmith

2019-06-14T18:39:59Z

Olchansk: Welcome!

User talk:Lmartin

2019-06-03T17:07:52Z

Olchansk: Welcome!

User talk:Gerritzen

2019-06-03T17:06:45Z

Olchansk: Welcome!

/Alias ODB tree

2019-02-11T19:03:54Z

Olchansk: /* Symbolic Links */

<div style="column-count:4;-moz-column-count:4;-webkit-column-count:4">
* [[Midas_documentation|Midas Documentation]]
* [[Feature_listing|Feature Listing]]
* [[Application_listing|Application Listing]]
* [[Online_Database|Online Database]]
</div>

== Purpose ==
The ODB /Alias tree provides the user with a way to access other webpages via buttons placed on the [[mhttpd]] [[Status Page]]. It can provide links to external webpages, or symbolic links (short-cuts) to ODB sub-trees.

== Creating the /Alias tree ==
The /Alias tree is optional, and is not present until [[ODB#Creating ODB keys|created by the user]].

<div id="Alias-Button"></div>
== Keys in the /Alias tree ==
Any key created under /Alias will appear as a button (i.e. '''alias-button''') on the [[Status Page]], with the same name as the key (except as noted below).

By default, the clicking of an alias-button in the web interface will spawn a '''new frame'''. To force the display of the alias page in the '''same frame''', an "&" has to be appended to the name of the alias key. The "&" is stripped off the alias name when it appears as an alias-button (see [[#Symbolic Links|example]]).

=== External links ===
The following [[odbedit]] commands demonstrate how to make a link to an external webpage:

[local:t2kgas:S] cd /Alias
[local:t2kgas:S]create string triumf
String length [32]:
[local:t2kgas:S] set triumf "http://triumf.ca"

This will cause a button <span style="color: #444444; background-color: #CCCCCC; font-style:italic; font-size: 90; padding:0.25em;
padding-left: 0.5em;padding-right: 0.5em;border:1px solid #808080;border-radius: 5px;margin-bottom:1px;">triumf
on the status page, Clicking on this alias-button will show the contents of the link in a new frame.

=== Links to ODB ===
The /Alias tree may also contain a list of symbolic links to any desired ODB location.

Create a string under /alias with this contents:
<pre>
?cmd=odb&odb_path=/Programs
</pre>

[[Category:ODB Tree]]

Setup MIDAS experiment at TRIUMF

2017-06-26T17:01:16Z

Olchansk:

{{Pagelinks}}
== Introduction ==
This page describes setting up a MIDAS experiment at TRIUMF. This information can be adapted for other sites.

== Environment variables ==
* '''MIDASSYS''' Base directory of the MIDAS package, midas and mxml should be at the same level.
* '''MIDAS_EXPTAB''' Experiment definition file
* '''MIDAS_SERVER_HOST''' MIDAS host server name for remote midas connections.
* '''MIDAS_EXPT_NAME''' Experiment name

== Standard layout of MIDAS experiment ==
The following shows the directory layout of a standard MIDAS experiment:

/home/exptuser/
packages/
root <---- ROOT
mxml
mscb
midas/ <---- MIDAS
linux/{lib,bin} <---- binaries matching the selected 64-bit/32-bit flavour of ROOT
linux-m32/{lib,bin} <---- limited function 32-bit binaries for 32-bit frontend machines, build by "make linux32"
linux-m64/{lib,bin} <---- limited function 64-bit binaries (only needed if ROOT and linux/bin are 32-bit)
linux-arm/{lib,bin} <---- full function ARM cross-compiled using "make linuxarm"
linux-crosscompile/{lib,bin} <---- cross-compiled limited function binaries for PPC and ARM frontends (see Makefile)
rootana <---- ROOT analyzer
roody <---- graphical online histogram viewer for MIDAS and ROOTANA
online/
exptab <---- experiment definition
{.ODB,.SYSTEM,.SYSMSG,etc}.SHM <---- MIDAS shared memory save files
src <---- experiment frontend sources
bin,scripts
elog <---- MIDAS elog
history <---- MIDAS history
data -> /data/exptname/current <---- symlink to the data directory
/data/exptname/current <---- experiment data directory with ODB save files and MIDAS .mid/.mid.gz data files

== Prepare computers ==

On some operating systems, several MIDAS functions require administrator access:

* on el7 linux (SL7/CC7/CentOS7/RHEL7) - access to mhttpd port 8443 requires special firewall rules, see here: https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29
* on el7 linux - access to mserver to run frontends and other programs on some other computer requires special firewall rules, see here: https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29
* on el7 linux - on the frontend machines (and other machines that will connect to the mserver, the same firewall rule needs to be created (use the IP address of the machine running the mserver)

== Prepare the user account ==
<div id="NOTES"></div>

* Setup the user account for running this instance of midas. For machines part of the LADD cluster, follow these [http://daq-plone.triumf.ca/SM/docs/local/NewLaddUser] instructions.
* check that the account is using the /bin/bash shell
* make $HOME/.profile look like this:

<pre>
#!/bin/echo You must source

export SVN_EDITOR="emacs -nw"
export GIT_EDITOR="emacs -nw"
export MIDASSYS=$HOME/packages/midas
export ROOTANASYS=$HOME/packages/rootana
export MIDAS_EXPTAB=$HOME/online/exptab
#
# setup the MIDAS mserver
#
case `hostname` in
daq07*)
unset MIDAS_SERVER_HOST
;;
*)
export MIDAS_SERVER_HOST=daq07.triumf.ca:7070
;;
esac
#
# select 64-bit or 32-bit MIDAS and ROOT
#
case `uname -i` in
i386)
source /daq/daqshare/olchansk/root/root_v5.34.01_SL62_32/bin/thisroot.sh
export PATH=.:$MIDASSYS/linux-m32/bin:$PATH
;;
*)
#source /daq/daqshare/olchansk/root/root_v5.34.34_SL67_64/bin/thisroot.sh
source $HOME/packages/root/bin/thisroot.sh
export PATH=.:$MIDASSYS/linux/bin:$PATH
;;
esac
#
export PATH=.:$HOME/online/bin:$HOME/packages/roody/bin:$PATH
#
#end
</pre>

* mkdir $HOME/packages
* Logout and login again, for .cshrc changes to take effect

== Install ROOT ==

* Identify the Linux version: RH9 (Red Hat Linux 9), FC3 (Fedora Core 3), RHEL4/SL4 (Red Hat Enterprise LInux 4/Scientific Linux 4), SL5x, SL6x, (CentOS/CC/SL) el7x: more /etc/redhat-release
* Decide to use 32-bit or 64-bit ROOT ('uname -a')
* cd $HOME/packages
* ls -l /daq/daqshare/olchansk/root/ ### to see all available ROOT packages
* ln -s /daq/daqshare/olchansk/root/root_vNNN_VVV_BB root, where NNN is the latest available version of ROOT ("ls -l /daq/daqshare/olchansk/root"), VVV is the Linux version code (RH9, FC3, SL4, etc) and BB is "_32" or "_64" for 32-bit or 64-bit ROOT. For example: /daq/daqshare/olchansk/root/root_v5.10.00_SL40
* for example: ln -s /daq/daqshare/olchansk/root/root_v5.34.34_el72_64 $HOME/packages/root
* Check that ROOT works: "source $HOME/packages/root/bin/thisroot.sh; root". Type ".q" to exit root.

== Install MIDAS ==

* cd $HOME/packages
* (OBSOLETE) svn co svn+ssh://svn@savannah.psi.ch/repos/meg/midas/trunk midas, password "svn". (password has to be entered twice)
* (OBSOLETE) svn co svn+ssh://svn@savannah.psi.ch/repos/meg/mxml/trunk mxml
* git clone https://bitbucket.org/tmidas/midas
* git clone https://bitbucket.org/tmidas/mxml
* git clone https://bitbucket.org/tmidas/mscb
* (IF BITBUCKET IS DOWN) git clone -v --progress https://daq.triumf.ca/~daqweb/git/mxml.git
* (IF BITBUCKET IS DOWN) git clone -v --progress https://daq.triumf.ca/~daqweb/git/mscb.git
* (IF BITBUCKET IS DOWN) git clone -v --progress https://daq.triumf.ca/~daqweb/git/midas.git
* cd midas
* make
* (only if needed) make linux32 ### build the 32-bit MIDAS libraries
* ls -l linux/bin/odbedit ### check that odbedit has been created (do not run it yet)

You can see a list of other installation problems at [[Common problems & Debugging recipes]].

;NOTE
: Optional features in MIDAS can be explicitly disabled if desired when making MIDAS using the NO_xxx feature (NO_ROOT,NO_MYSQL,NO_ODBC,NO_SQLITE,NO_MSCB), e.g. "make NO_ROOT=1" to disable ROOT. These NO_xxx Makefile variables are only used to control autodetection.

== Install ROOTANA ==

* cd $HOME/packages
* git clone https://bitbucket.org/tmidas/rootana
* cd rootana
* make

== Install ROODY ==

* cd $HOME/packages
* git clone https://bitbucket.org/tmidas/roody
* cd roody
* make
* $HOME/packages/roody/bin/roody, run the program

== Install additional additional packages ==

* cd $HOME/packages
* git clone https://bitbucket.org/ttriumfdaq/vme
* git clone https://bitbucket.org/ttriumfdaq/frontends

== Build special versions of MIDAS ==

Build special versions of MIDAS for the case when some MIDAS programs, such as VME frontends, will run on a different computer that may have a different flavour of operating system, i.e. 32-bit linux or an older version of Scientific Linux.

* login to the computer where the frontends will run and:
* if it is a 32-bit linux: cd $HOME/packages/midas; make linux32
* if it is a 64-bit linux: cd $HOME/packages/midas; make linux64

* login to the host computer to cross-compile ARM code:
* if it is an ARM linux: cd $HOME/packages/midas; make linuxarm # may need to install ARM cross compilers

== Prepare VME hardware ==

Hardware check list:
* VME crate
* VME processor (supported are V77xx, V7805, V7865)
* On all VME modules, set the VME address jumpers as described here: http://daq-plone.triumf.ca/SM/docs/local/vme_jumpers
* run vmescan to confirm correct VME addresses
** cd $HOME/packages
** svn checkout https://ladd00.triumf.ca/svn/daqsvn/trunk/vme
** cd vme
** make
** ./vmescan.exe (or _gef.exe, depending on the VME driver in use)

== Install Universe-II VME driver (V7648, V7750, V7805, V7851) ==

* login as root (ssh root@localhost)
* follow instructions: https://www.triumf.info/wiki/DAQwiki/index.php/VME-CPU#V7648.2C_V7750.2C_V7805.2C_V7851_:_Setup_vme_universe_VME_drivers
* cd ~/packages/vme; vmescan.exe

== Install Tsi-148 VME driver (V7865) ==

* login as root (ssh root@localhost)
* follow instructions: https://www.triumf.info/wiki/DAQwiki/index.php/VME-CPU#V7865_and_XVB-602_:_Setup_gefvme.2Ftsi148_VME_drivers
* cd ~/packages/vme; vmescan_gef.exe

== Setup the experiment environment ==

* Decide which computer will host MIDAS (where MIDAS shared memory buffers will reside).
: This computer will run the [[mserver]], [[mlogger]] and [[mhttpd]] applications. (It is usually the machine where the MIDAS,ROOT etc. packages have been downloaded). It will be referred to as the host machine (localhost).

The environment is slightly different depending on whether all programs run on the host machine, or whether some programs run on remote host(s) :
=== ALL programs run on localhost ===
:If all programs run on the host machine (localhost), it is not necessary to run [[mserver]]. [[Environment Variables#MIDAS_SERVER_HOST|MIDAS_SERVER_HOST]] will not be assigned (see example .cshrc [[#Prepare the user account|above]]).

=== Some programs run on REMOTE host(s) ===
: '''IMPORTANT:'''
# Since August 2015 '''you must explicitly allow access for clients running on remote machines'''. To do this, follow the '''[[Security#MIDAS programs on remote machines|instructions here]]'''.
# The example code .cshrc ([[#Prepare the user account|see above]]) should be present on both host and remote machine(s). This will ensure that [[Environment Variables#MIDAS_SERVER_HOST|MIDAS_SERVER_HOST]] will NOT be set for the host machine (localhost), but on a remote machine, MIDAS_SERVER_HOST will be set to the MIDAS host machine.
# The client [[mserver]] must be started on the MIDAS host machine. Note that multiple experiments can run on the same host machine by starting several instances of [[mserver]] (one for each experiment) running with different ports (and .cshrc would be edited so that MIDAS_SERVER_HOST is set to the appropriate port for the experiment).

 
On the host machine:
* mkdir $HOME/online
* cd $HOME/online
* create directories for local programs, sources, elog and history: mkdir bin src elog history
* create data directory: mkdir -p /ladd/data1/t2kvme5/data; ln -s /ladd/data1/t2kvme5/data $HOME/online
* create the exptab file "$HOME/online/exptab" following the example below. The first entry (exptname) is the name if the DAQ system (MIDAS experiment name), the second entry (/home/USER/online) is the location of MIDAS shared memory buffers (by convention, $HOME/online), the third entry (kopio03) is your username.
<pre>exptname /home/kopio03/online kopio03</pre>
* logout and login again for all changes to take effect

== Setup experiment startup scripts ==
* login to the experiment host computer
* echo $MIDAS_SERVER_HOST ### to check correct value - should be blank
* create $HOME/online/bin/start_daq.sh, replacing XXX with the hostname of the machine running the experiment (and changing the mserver and mhttpd ports, as needed).

#!/bin/sh
# start_daq.sh
cd $HOME/online
#
case `hostname` in XXX*)
echo "Good, we are on XXX!"
;;
*)
echo "The start_daq script should be executed on XXX"
exit 1
;;
esac
#
odbedit -c clean
# start [[mhttpd]] on default port. (Mongoose https version - see [[mhttpd]] for other options)
mhttpd -D -a localhost -a XXX.triumf.ca # optionally restrict access to specified hosts
#
# start [[mserver]] on default port (use argument -p to use a different port)
mserver -D # access must now be specifically allowed - see [[#Setup the experiment environment|above]]

# OR ([[#NOTES|older MIDAS versions]])
# mhttpd -p 8081 -D -a localhost -a XXX.triumf.ca # optionally restrict access to specified hosts
# mserver -p 7071 -D -a localhost -a lxdragon01.triumf.ca -a lxdragon02.triumf.ca -a XXX.triumf.ca # optionally restrict access to specified hosts

#
mlogger -D
#end file

== Run the MIDAS Web Server ==
Let's start the MIDAS webserver for the first time:

Start [[mhttpd]] on the ''experiment host'' (localhost) like this:
[mhostpc] mhttpd
You will get the following messages:
[mhttpd,INFO] ODB subtree /Runinfo corrected successfully
Mongoose web server will listen on ports "8080r,8443s"
[mhttpd,ERROR] [mhttpd.cxx:17892:mongoose,ERROR] cannot find SSL certificate file "/home/agdaq/online/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17893:mongoose,ERROR] please create SSL certificate file: openssl req -new -nodes -newkey rsa:2048 -sha256 -out ssl_cert.csr -keyout ssl_cert.key; openssl x509 -req -days 365 -sha256 -in ssl_cert.csr -signkey ssl_cert.key -out ssl_cert.pem; cat ssl_cert.key >> ssl_cert.pem
could not start the mongoose web server, see messages and midas.log, bye!

Create a self-signed certificate suitable for initial testing by executing the command printed by mhttpd:


[mhostpc] openssl req -new -nodes -newkey rsa:2048 -sha256 -out ssl_cert.csr -keyout ssl_cert.key; openssl x509 -req -days 365 -sha256 -in ssl_cert.csr -signkey ssl_cert.key -out ssl_cert.pem; cat ssl_cert.key >> ssl_cert.pem

For production use, you should create a properly signed certificate, see [[Mhttpd#Create an SSL certificate|create your own SSL certificate]] or you should run mhttpd behind an SSL proxy.

Run mhttpd again.

You will get the following messages:
[mhttpd,INFO] ODB subtree /Runinfo corrected successfully
Mongoose web server will listen on ports "8080r,8443s"
Mongoose web server will use SSL certificate file "/home/johnfoo/packages/midas/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17633:mongoose,ERROR] mongoose web server cannot find password file "/home/johnfoo/online/htpasswd.txt"
[mhttpd,ERROR] [mhttpd.cxx:17634:mongoose,ERROR] please create password file: htdigest -c /home/johnfoo/online/htpasswd.txt Default midas
could not start the mongoose web server, see messages and midas.log, bye!

Create the password file by following the instructions printed by mhttpd. The http digest domain name is the experiment name, suggested default user name is "midas". You will be asked to type in a password


[mhostpc] htdigest -c /home/johnfoo/online/htpasswd.txt exptname midas
Adding password for midas in realm exptname.
New password:
Re-type new password:

It is a good idea to set the password file {{Filepath|path=htpasswd.txt}} readable and writable by owner only.

Now restart {{Utility|name=mhttpd}}
[mhostpc] mhttpd
Mongoose web server will listen on ports "8080r,8443s" **see note
Mongoose web server will use SSL certificate file "/home/suz/packages/midas/ssl_cert.pem"
Mongoose web server will use authentication realm "Default", password file "./htpasswd.txt"

Now point a web browser running on the same host computer (localhost) to https://localhost:8443
If the web browser is running on a different computer, go to URL of the form

https://mhostpc.triumf.ca:8443 (substitute your host machine name and domain for "mhostpc.triumf.ca")

If you are using the default SSL certificate you will probably get a message: "This Connection is Untrusted". Click "I understand the risks" and add an exception. This is because the test certificate is self-signed. Then confirm an exception.

If instead you get a "connection refused" error, the midas host pc may have the firewall enabled. To make a firewall exception for MIDAS, follow instructions here http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29

You should then see an authentication box asking you for the user name and password. The user name is "midas". Enter the password you just created. The Midas [[Status Page]] should appear with multiple buttons for run control as well as equipment listing (no equipments will be listed as yet) and application listings. Please refer to [[mhttpd]] (the MIDAS Web-based Run Control utility) for further information. You can start and stop runs from the main status page, and use the [[ODB Page]] to access the database (ODB).

; Note
: Default ports of 8080 and 8443 are used by [[mhttpd]]. If these ports are in use on your machine, start mhttpd with alternative ports, e.g.
[mhostpc] mhttpd --https 8448 --http 8089
: or see [[Mhttpd#Usage]] to change the default ports.

== Setup experiment database (ODB) ==

* run $HOME/online/bin/start_daq.sh

* odbedit, run these commands: (replace user names and directory names)
<pre>
set "/Logger/Message file" "/home/kopio03/online/midas.log"
set "/Logger/Data Dir" "/home/kopio03/online/data"
create STRING "/Logger/History dir"
set "/Logger/History dir" "/home/kopio03/online/history"
create STRING "/Logger/Elog dir"
set "/Logger/Elog dir" "/home/kopio03/online/elog"
set "/Logger/ODB dump file" "/home/kopio03/online/history/run%05d.xml"
set "/Logger/ODB dump" "y"
set "/Logger/Channels/0/Settings/Filename" "run%05dsub%03d.mid.gz"
set "/Logger/Channels/0/Settings/Subrun byte limit" "1000000000"
set "/Logger/Channels/0/Settings/Compression" 1
set "/Logger/Channels/0/Settings/ODB Dump" "y"
set "/Programs/Logger/Required" y
set "/Programs/Logger/Start command" "mlogger -D"
set "/Programs/fevme/Required" "y"
set "/Programs/fevme/Start command" "ssh -n lxdaq09 $HOME/online/src/fevme_gef.exe -O"
exit
</pre>
* open web browser e.g. firefox.
* go to the midas status page at https://localhost:8443 (default port).
** if running [[mhttpd]] with Mongoose HTTPS/OpenSSL (the default) for the first time, you will need to create a password file. Follow the instructions (see [[mhttpd#HTTPS/SSL server (Mongoose)]] for details).
** For other options (i.e. HTTPS/SSL proxy) see [[#Secure MIDAS and ELOG Web access]]
*OR open the midas status page at http://localhost:8081 ([[#NOTES|older MIDAS versions]])
* midas status page will show most stuff "red" as nothing is running yet
* DON'T DO THIS YET run ./fevme.exe (on the computer with the VME interface, could be different from computer hosting the experiment), observe that corresponding equipments have been created
* save the url bookmark to the "personal toolbar"
* go to the Programs page, stop mlogger, stop fevme, start mlogger, start fevme
* go to the Status page, start run, stop run
* go back to the Status page, everything should be green
* start a run
* send signals to the ADC gate
* you should be getting events
* to look at data, proceed with setting up the [[ROOTANA|ROOT Analyzer]].

== Start DAQ programs at boot time ==

* add this to /etc/rc.local (replace username and location of the start_daq script)
<pre>
su - alpha -c /home/alpha/online/bin/start_daq.sh
</pre>

== Setup local software version control ==

Version control for experiment source code is setup using "git" (http://git-scm.com/)

* cd $HOME/online
* git init
* git add exptab
* git add bin/start_daq.sh
* git add .gitignore ### contents can be
<pre>
*~
*.o
*.exe
</pre>
* git add src/Makefile src/*.cxx ...
* git commit -a

== Adjust MIDAS buffer sizes ==

Default MIDAS SYSTEM buffer size is 8 Mbytes, fairly small for high-data-rate experiments. The rule of thumb is to have at least a few seconds worth of buffer space available. For example, if event size is 10 Kbytes and the event rate is 1 kHz, data rate is 10*10^3*1*10^3 = 10 Mbytes/sec. To buffer 10 seconds of data we need 100 Mbytes of buffer space.

To resize the MIDAS event buffers (SYSTEM, etc) do this:
* stop all frontends, stop mlogger
* start odbedit:
** cd "/Experiment/Buffer sizes"
** set SYSTEM 100000000
* run "mdump -z SYSTEM"
* if mdump complains about the size of .SYSTEM.SHM, remove it, try again.
* ls -l /dev/shm ### to observe that the size of shared memory is correct

== Secure MIDAS and ELOG Web access ==
In versions prior to May 2015, the default web access to MIDAS and ELOG uses the "http:" protocol which is insecure. In this case, all information is transmitted as clear text meaning that secret, confidential and sensitive information (such as the MIDAS and ELOG passwords and usernames) can be stolen "easily". This means that even "password protected" MIDAS and ELOG pages are not really protected if accessed using the "http" method.

Better security for HTTP is gained by using a password protected '''SSL (https) proxy'''. (It does not provide absolute security because of remaining problems with the security of SSL certificates, security of passwords, etc). Setting up an SSL (https) proxy is described [[#Setting up an HTTP proxy|below]].

Since May 2015, an ''alternative secure option'' to setting up an HTTP proxy is available to users of MIDAS. Recent versions of elogd (ELOG) do support SSL https:// connections, and [[#mhttpd with HTTPS/SSL server (Mongoose)]] is now available. This option is the default, and provides a similar level of security to an HTTP proxy.

See [[Security#Web Access]] for a comparison of these two secure options.

=== mhttpd with HTTPS/SSL server (Mongoose) ===

Since May 2015 the MIDAS web server [[mhttpd]] is explicitly linked with OpenSSL to provide secure HTTPS connections via the [https://bitbucket.org/tmidas/midas/src/ecb9a8537448a8a43f7f9a2bfdb82e578208cde3/doc/mongoose/?at=develop Mongoose] web server (see [[mhttpd]]). With this version, default web access to MIDAS uses the "https" protocol. Web access to {{Utility|name=mhttpd}} can be restricted by using the "-a hostname" switch of [[mhttpd]]. The first time {{Utility|name=mhttpd}} is run, a password file must be created. An SSL certificate is also required. See [[mhttpd#HTTPS/SSL server (Mongoose)|HTTPS/SSL server (Mongoose)]] for instructions.

=== mhttpd using an HTTPS/SSL proxy ===

THESE INSTRUCTIONS ARE WRONG, DO NOT DO THIS.

An [[#Setting up an HTTP proxy|HTTP proxy]] must be set up. This is the only way of securing older version of [[mhttpd]] (pre August 2015). Older versions of mhttpd are started using the -p port option e.g.
* mhttpd -D -p 8080 

To run a new version of mhttpd using an HTTP proxy, use the options provided to run the old (non-Mongoose) webserver on a given port, i.e.
* mhttpd --oldserver 8080 --nomg -D 

When using an SSL proxy, only access from the SSL proxy (and maybe some special trusted machines) should be permitted.
This is done using the "-a hostname" switch of [[mhttpd]]. Normally there will be only "-a localhost" switch, enabling access only for the local machine (where the SSL proxy is running). Additional "-a hostname" switches enable access from listed local machines. No "-a xxx" enables access from everywhere (defeating the purpose of the SSL proxy, unless access controls are enforced elsewhere, i.e. by a site firewall or by local firewall rules).

== Setting up an HTTP proxy ==
In this example, we use APACHE HTTPD to password-protect a typical midas/mhttpd and elog installation.

In this configuration, one uses the Linux stock httpd that accepts encrypted https:// connections and forwards them to mhttpd and elogd. Instead of (or in addition to) using mhttpd and elogd passwords, one configures password protection in httpd via the regular apache httpd password mechanisms (htpasswd, etc).

Recent versions of elogd do support SSL https:// connections, but if one is running an SSL proxy for anyway, it is simpler to run both through the same SSL proxy using the same SSL host certificate and the same httpd password file.

=== Restricting http: access to elogd ===
;Note
:Recent versions of elogd do support SSL https:// connections. The following information is for those using an HTTP proxy (see above).

For elogd, this is done using the "-n localhost" switch with enables only access from the same machine if present, or access from anywhere is absent (defeating the purpose of the SSL proxy, unless access controls are enforced elsewhere).

(It is recommended to run elogd from the same user as the main daq user and to keep elogd.cfg and all logbooks in the home directory of this user, where they are captured by the normal site backup system)

== Install standalone elog ==

* login into the user account that will run the elog
* cd $HOME/packages
* git clone https://bitbucket.org/ritt/elog
* cd elog
* make
* create new file start_elogd with this contents:
<pre>
#!/bin/sh

killall elogd
sleep 1
killall -KILL elogd
sleep 1
$HOME/packages/elog/elogd -n localhost -x -c $HOME/packages/elog/elogd.cfg -p 8082 -D

#end
</pre>
* chmod a+x start_elogd
* edit elogd.cfg to read:
<pre>
[global]
port = 8082
SMTP host = smtp.triumf.ca
URL = https://titan00.triumf.ca/elog/

Reverse sort = 1
Display Mode = full

#List Menu commands = New, Find, Admin, Help
#Menu commands = New, Edit, Reply, Find, Duplicate, Help

Entries Per Page = 30
Supress Email on edit = 1
Default encoding = 1
Page title = TITAN ELOG
Resolve host names = 1

Logfile = /home/titan/packages/elog/elogd.log
#Logging level = 3

[midas]

List page Title = T2K M11 MIDAS ELOG
Comment = T2K M11 MIDAS ELOG
Page Title = T2K M11 MIDAS ELOG
RSS Title = [$logbook - $type - $system] $subject, posted by $author

Attributes = Author, Subject, Run, Type, System
Show Attributes Edit = Run, Author, Subject, Type, System
Required Attributes = Author, Type, System, Subject

Options Type = Routine, Reply, Shift Summary, Modification, Question, Info, Problem
Options System = General, DAQ, Beamline

Preset Run = $shell(MIDASSYS=. /home/t2km11/packages/midas/linux/bin/odbedit -d Runinfo -c 'ls -v \"run number\"')

Preset On Reply Type = Reply
Preset On Reply Run = $shell(MIDASSYS=. /home/t2km11/packages/midas/linux/bin/odbedit -d Runinfo -c 'ls -v \"run number\"')

List Display = Date, Subject, Type, System, Author, ID
Quick Filter = Date, Type, ID

Remove on reply = Author
Quote on reply = 1

Use lock = 1

************* Email Functionality ****************

Use Email Subject = [T2KM11 - $System] $Subject
Omit Email To = 1

Email System General = xxx
</pre>
* ./start_elogd &
* firefox http://localhost:8082 # hould show the elog message index

To start elogd automatically when the machine is rebooted, login as root and
* add this text to /etc/rc.local:
<pre>
su - titan -c "/home/titan/packages/elog/start_elogd"
</pre>
* chmod a+x /etc/rc.local
* systemctl start rc-local

To import elog entries from the mhttpd elog, do this:

* cd ~/packages/elog/logbooks
* ln -s /home/t2km11/online/elog midas
* cd midas
* ~/packages/elog/elconv)

== Install https proxy ==

THESE INSTRUCTIONS ARE OBSOLETE, INSTEAD,
* GO HERE: https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Configure_HTTPS_server_.28CentOS7.29
* AND GO HERE: https://midas.triumf.ca/MidasWiki/index.php/Quickstart_Linux#Run_the_MIDAS_Web_Server

FOLLOWING INSTRUCTIONS ARE OBSOLETE...

* login as root to the https proxy machine
* cd ~root
* yum install mod_ssl
* yum install crypto-utils # see http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_monitoring_of_HTTPS_certificates
* create a certificate request (replace ladd09 with your hostname): openssl req -new -nodes -newkey rsa:2048 -sha256 -out ladd09.csr -keyout ladd09.key (answer: CA, BC, Vancouver, TRIUMF, DAQ, ladd09.triumf.ca, email@email.com
* sign it by TRIUMF:
** mail -s "Certificate request" yourself@email.com < ladd09.csr
** forward this request to Andrew Daviel
** he will email the signed crt file, copy it to this system as ladd09.crt
* sign it yourself: openssl x509 -req -days 365 -sha256 -in ladd09.csr -signkey ladd09.key -out ladd09.crt
* (if the certificate expires, renew it by signing it again)
* Additional commands for working with certificates:
** explore the private key: openssl pkey -in ladd09.key -text -noout
** explore the certificate request: openssl req -in ladd00.csr -text -noout
** explore the certificate: openssl x509 -in ladd09.crt -noout -text
* move certificate files to proper system locations:
* mv ladd09.key /etc/pki/tls/private/
* mv ladd09.crt /etc/pki/tls/certs/
* if selinux is enabled, do this:
** restorecon -Rv /etc/pki/tls/certs/
** restorecon -Rv /etc/pki/tls/private/
** /usr/sbin/setsebool -P httpd_can_network_connect 1
* open /etc/httpd/conf.d/ssl.conf in a text editor, go to the very bottom and right before the "</VirtualHost>" entry, add following text:
<pre>
...
SSLCertificateFile /etc/pki/tls/certs/ladd09.crt
SSLCertificateKeyFile /etc/pki/tls/private/ladd09.key

ProxyPass /elog/ http://localhost:8082/ retry=1
ProxyPass / http://localhost:8080/ retry=1

<Location />

SSLRequireSSL
AuthType Basic
AuthName "password protected site"
Require valid-user

# create password file: touch /etc/httpd/htpasswd
# to add new user or change password: htpasswd /etc/httpd/htpasswd username
AuthUserFile /etc/httpd/htpasswd

</Location>
</VirtualHost>
...
</pre>
* comment out duplicate "SSLCertificateFile" and "SSLCertificateKeyFile" elsewhere in the file
* (optionally) If you got a certificate that is signed by DigiCert or RapidSSL then you'll need to add a line specifying the certificate chain file:
<pre>
...
SSLCertificateChainFile /etc/pki/tls/certs/DigiCertCA.crt
...
</pre>
* touch /etc/httpd/htpasswd
* htpasswd /etc/httpd/htpasswd midas # enter password midas
* chkconfig httpd on
* service httpd restart
* firewall-cmd --add-port=443/tcp --permanent
* firewall-cmd --reload
* firewall-cmd --list-all
* test it
** test the SSL proxy: https://host/ should yield the midas status page, https://host/elog/ should yield the elog message index
* in ODB, set "/Elog/URL" to "https://host/elog/"
* now from the midas status page, the "Elog" button should take us to the https Elog URL

In needed, enable user directories: https://blah/~user in ~user/public_html

* edit /etc/httpd/conf.d/userdir.conf, replace "UserDir disabled" with "UserDir enabled"
* setsebool -P httpd_enable_homedirs true
* systemctl restart httpd

== Setup the history mhttpd for faster access to history plots ==
When running an SSL proxy,
* start the main mhttpd (orange command for old mhttpd, green for new mhttpd with Mongoose(post August2015):
** "mhttpd -p 8071 -D" or
** "mhttpd -D --oldserver 8071 --nomg" 

* start the history mhttpd
** "mhttpd -p 8072 -D -H" or
** "mhttpd -D -H --oldserver 8072 --nomg" 
* set ODB /History/URL to "http://alphacpc09.cern.ch:8072/HS/"
* open the MIDAS status page
* go to the history section, try to open any history plot, observe that the history plot gif image loads correctly, inspect it's URL (use "copy image URL" or "view source", etc), it should point to port 8072 causing connection to the history mhttpd.
* continue with these instructions to setup history mhttpd access through an SSL proxy:
* setup SSL proxy access (required mod_proxy_html)
** login as root to the SSL proxy machine
** on SL5, install the missing mod_proxy_html httpd module:
** yum install httpd-devel libxml2-devel
** wget http://apache.webthing.com/mod_proxy_html/mod_proxy_html.tar.bz2
** tar xjvf mod_proxy_html.tar.bz2
** cd mod_proxy_html
** apxs -c -I. -I/usr/include/libxml2 -i mod_proxy_html.c
** apxs -c -I. -I/usr/include/libxml2 -i mod_xml2enc.c
** cd /etc/httpd/conf.d, add this to ssl.conf:

;before the ProxyPass statements:
<pre>
# proxy the MIDAS web servers
LoadModule xml2enc_module modules/mod_xml2enc.so
LoadModule proxy_html_module modules/mod_proxy_html.so
ProxyHTMLLinks a href
ProxyHTMLLinks link href
ProxyHTMLLinks img src
#ProxyHTMLEnable On
ProxyRequests off
</pre>
;after the ProxyPass statements:
<pre>
# ALPHA1 history access
ProxyPass /alpha1/history/ http://alphacpc09.cern.ch:8072/HS/ retry=1
ProxyPass /alpha1/ http://alphacpc09.cern.ch:8071/ retry=1

ProxyHTMLEnable On
ProxyHTMLURLMap http://alphacpc09.cern.ch:8072/HS/ /alpha1/history/
</pre>
;adjust:
*"alpha1" is the experiment name
*"alphacpc09.cern.ch" is the machine running mhttpd
*"8071" is the port number of the main mhttpd
** "mhttpd -p 8071 -D" or
** "mhttpd -D --oldserver 8071 --nomg" 
* "8072" is the port number of the history mhttpd
** "mhttpd -p 8072 -D -H" or
** "mhttpd -D -H --oldserver 8072 --nomg" 

[[Category:Installation]] [[Category:Buffer]] [[Category:Driver]]

Quickstart Linux

2017-03-02T00:21:28Z

Olchansk: /* Start MIDAS on boot */

{{Pagelinks}}
== Introduction ==
This quickstart shows you how to install the MIDAS packages on a linux box and create a '''MIDAS experiment'''.
The linux box will be called the MIDAS '''''experiment host'''''.

==Installation==
Log on to the ''[[#Introduction|Experiment Host]]'' and decide on a name for the MIDAS experiment (the ''experiment name'').
In these instructions, the MIDAS packages and experiment will be installed under the username "johnfoo".
The ODB and other shared memory buffers for the MIDAS experiment will reside on the ''experiment host'' (localhost).
MIDAS and the other required packages will be installed under directory {{Filepath|path=$HOME/packages}}. The MIDAS ''experiment directory'' will be {{Filepath|path=$HOME/online}} and the MIDAS ''experiment name'' will be "e777". The name of the ''experiment host'' will be "mhostpc".

;NOTE
: The user should substitute directory paths for the MIDAS packages and experiment directory as required, and names for ''experiment host'', username, ''experiment name'' appropriate to his/her own setup.

== Experimental setup ==
These instructions describe setting up an experiment that will run ONLY on the MIDAS ''experiment host'' computer (i.e. '''localhost only'''). In this case, the MIDAS ''experiment host'' has access to any hardware required, and all frontends, logger, analyzer etc. run on the one computer.

Modifications to run an experiment with additional frontend(s) running '''remotely''' (i.e. on remote cpu(s)) will also be described.
Follow the instructions for setting up a localhost-only experiment until indicated.

In both cases, the main MIDAS applications (e.g. [[mlogger]], [[mhttpd]]) run on the MIDAS ''experiment host''.

==Environment Variables==
The following Environment variables should to be added to the {{File|name=.cshrc}} or {{File|name=.bashrc}} file (depending on your shell) in the {{Filepath|path=$HOME}} directory on the ''experiment host'', so that they will be defined at login. '''Substitute appropriate values for your own setup'''.

{| class="wikitable"
|-
! csh !! bash !! comment
|-
| setenv GIT_EDITOR "emacs -nw" || export GIT_EDITOR="emacs -nw" ||
|-
| setenv MIDASSYS $HOME/packages/midas || export MIDASSYS=$HOME/packages/midas ||Base directory of the MIDAS package
|-
| setenv MIDAS_EXPTAB $HOME/online/exptab || export MIDAS_EXPTAB=$HOME/online/exptab || MIDAS experiment table
|-
|-
| setenv MIDAS_EXPT_NAME e777 || export MIDAS_EXPT_NAME=e777 || MIDAS experiment name
|-
| setenv PATH .:$MIDASSYS/linux/bin:$PATH || export PATH=$PATH:$MIDASSYS/linux/bin || path
|}

Logout and login again, or source {{File|name=.cshrc}} (source {{File|name=.bashrc}}) for the changes to take effect.
;NOTE
: See [[MIDAS environment variables]] for a list of environment variables used by MIDAS.

== ROOT Package Installation ==
For full MIDAS operation, ROOT is needed for the data logging and analysis packages. It needed to build the example analyzer in this Quickstart. However, ROOT is not essential to run MIDAS. If ROOT is NOT installed, the environment variable ROOTSYS should be undefined.

If ROOT is needed, see [[Setup MIDAS experiment#Install ROOT|Install ROOT]].

== MIDAS Package Installation ==
The MIDAS package will be installed on the ''experiment host'' in the directory given by MIDASSYS - see [[#Environment Variables]].

Install the MIDAS and other package(s) from the MIDAS git repository in an xterm as follows:

[mhostpc] mkdir $HOME/packages
[mhostpc] cd $HOME/packages
[mhostpc] git clone https://bitbucket.org/tmidas/midas
[mhostpc] git clone https://bitbucket.org/tmidas/mxml

Install the MSCB package '''only''' if you will be using the MSCB system for slow-controls (see [[MSCB Page]]):

[mhostpc] git clone https://bitbucket.org/tmidas/mscb ## MSCB package
[mhostpc] cd mscb
[mhostpc] make ## Build the MSCB package
[mhostpc] cd ../

The MIDAS Makefile will auto-detect whether MSCB, ROOT, MYSQL, SQLITE, ODBC etc. are available (or have been installed). If you need these features, make sure they are available before building the MIDAS package.

[mhostpc] cd midas
[mhostpc] make ## Build the MIDAS package
[mhostpc] ls -l linux/bin/odbedit ### check that odbedit has been created (do not run it yet)


In case of problems see
* [[Installation#Installing MIDAS|bitbucket is down]] or
* [[Installation/Compilation problems#SSL certificate errors|SSL certificate errors]].

==ROOTANA Package Installation==

Install the ROOT analyzer package:

[mhostpc] cd $HOME/packages
[mhostpc] git clone https://bitbucket.org/tmidas/rootana
[mhostpc] cd rootana
[mhostpc] make


add ROOTANASYS=$HOME/packages/rootana to the login file.

Install the JSROOT package:
<pre>
cd $HOME/packages
git clone https://github.com/linev/jsroot.git
</pre>

add JSROOTSYS=$HOME/packages/jsroot to the login file.

==ROODY Package Installation==
If ROODY is needed,

[mhostpc] cd $HOME/packages
[mhostpc] git clone https://bitbucket.org/tmidas/roody
[mhostpc] cd roody
[mhostpc] make


== Create the Experiment file exptab ==
In this example, the experimental directory is {{Filepath|path=$HOME/online}}. Create this directory on the ''experiment host'' (localhost):

[mhostpc] mkdir $HOME/online
[mhostpc] cd $HOME/online
Then create the [[exptab]] file {{Filepath|path=$HOME/online/exptab}} containing the three parameters:
# the experiment name
# location of MIDAS shared memory buffers
# username
as follows (using parameters for your own experiment):

[mhostpc] cat > exptab
e777 /home/johnfoo/online johnfoo

The path of the {{File|name=exptab}} file is given by MIDAS_EXPTAB - see [[#Environment Variables]].

== Create shared memory files ==
At this point you should be able to run [[odbedit]] on the ''experiment host'' (localhost). The first time {{Utility|name=odbedit}} is run, it will create the required {{File|name=.*.SHM}} files in the MIDAS experiment directory {{Filepath|path=$HOME/online}}. These are the saved files for the MIDAS shared memory.

The default directory {{Filepath|path=$HOME/online}} will contain the MIDAS messages file ({{File|name=midas.log}}) and
any data files you may create. If you want an alternative location for the data files see [[#Customizing your experiment]].

Run {{Utility|name=odbedit}}, and type {{Odbedit cmd|cmd=ls}} to list the default directories.
[mhostpc] odbedit
[local:e777:S]/>ls
Experiment
System
Programs
Logger
Runinfo
Alarms

== Run the MIDAS Web Server ==

Recommended for production systems with maximum security is to run mhttpd behind an industry-standard SSL/https proxy with password protection.

To do this using apache httpd on CentOS-7, do this:
* configure apache httpd per instructions at https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Configure_HTTPS_server_.28CentOS7.29
* run odbedit, change values of
** "/experiment/http redirect to https" set to "n"
** "/experiment/midas https port" set to 0
* run mhttpd
* use web browser to connect to https://mhostpc/
* enter the user name and password configured in apache httpd
* the midas status page should open

For simplified installation, one can use standalone mhttpd with built-in https and password protection. This configuration
is only as secure as the underlying mongoose and OpenSSL web server and encryption packages.

Many current operating systems implement a built-in firewall which will block outside access to mhttpd.

On CentOS-7, so this: https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29

Start [[mhttpd]] on the ''experiment host'' (localhost) like this:
[mhostpc] mhttpd
You will get the following messages:

Mongoose web server will listen on ports "8080r,8443s"
[mhttpd,ERROR] [mhttpd.cxx:17892:mongoose,ERROR] cannot find SSL certificate file "/home/agdaq/online/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17893:mongoose,ERROR] please create SSL certificate file: openssl req -new -nodes -newkey rsa:2048 -sha256 -out ssl_cert.csr -keyout ssl_cert.key; openssl x509 -req -days 365 -sha256 -in ssl_cert.csr -signkey ssl_cert.key -out ssl_cert.pem; cat ssl_cert.key >> ssl_cert.pem
could not start the mongoose web server, see messages and midas.log, bye!

Create a self-signed certificate suitable for initial testing by executing the commands printed by mhttpd.

For production use, you should create a properly signed certificate, see [[Mhttpd#Create an SSL certificate|create your own SSL certificate]] or you should run mhttpd behind an SSL/https proxy (apache httpd).

Run mhttpd again.

You will get the following messages:

Mongoose web server will listen on ports "8080r,8443s"
Mongoose web server will use SSL certificate file "/home/johnfoo/packages/midas/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17633:mongoose,ERROR] mongoose web server cannot find password file "/home/johnfoo/online/htpasswd.txt"
[mhttpd,ERROR] [mhttpd.cxx:17634:mongoose,ERROR] please create password file: htdigest -c /home/johnfoo/online/htpasswd.txt Default midas
could not start the mongoose web server, see messages and midas.log, bye!

Create the password file by following the instructions printed by mhttpd. The http digest domain name is the experiment name, suggested default user name is "midas". You will be asked to type in a password. (htdigest is part of the httpd-tools package)


[mhostpc] htdigest -c /home/johnfoo/online/htpasswd.txt exptname midas
Adding password for midas in realm exptname.
New password:
Re-type new password:

It is a good idea to set the password file {{Filepath|path=htpasswd.txt}} readable and writable by owner only.

Now restart {{Utility|name=mhttpd}}
[mhostpc] mhttpd
Mongoose web server will listen on ports "8080r,8443s" **see note
Mongoose web server will use SSL certificate file "/home/suz/packages/midas/ssl_cert.pem"
Mongoose web server will use authentication realm "Default", password file "./htpasswd.txt"

Now point a web browser running on the same host computer (localhost) to https://localhost:8443
If the web browser is running on a different computer, go to URL of the form

https://mhostpc.triumf.ca:8443 (substitute your host machine name and domain for "mhostpc.triumf.ca")

If you are using the default SSL certificate you will probably get a message: "This Connection is Untrusted". Click "I understand the risks" and add an exception. This is because the test certificate is self-signed. Then confirm an exception.

If instead you get a "connection refused" error, the midas host pc may have the firewall enabled. To make a firewall exception for MIDAS, follow instructions here http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29

You should then see an authentication box asking you for the user name and password. The user name is "midas". Enter the password you just created. The Midas [[Status Page]] should appear with multiple buttons for run control as well as equipment listing (no equipments will be listed as yet) and application listings. Please refer to [[mhttpd]] (the MIDAS Web-based Run Control utility) for further information. You can start and stop runs from the main status page, and use the [[ODB Page]] to access the database (ODB).

; Note
: Default ports of 8080 and 8443 are used by [[mhttpd]]. To use some other ports, change the values of ODB "/experiment/midas http port" and "midas https port".

== Run the MIDAS logger ==
Start the MIDAS logger [[mlogger]] on the ''experiment host'' (localhost) :
[mhostpc] mlogger
This should start without error. It is usually run as a daemon, however in this case it is run in a terminal to check for errors. Starting the midas logger [[mlogger]] will automatically create more keys in the {{Odbpath|path=/Logger}} ODB tree.

From the MIDAS status page, go to the "Programs" page, then the "Logger" page and set ODB variables:
* Required -> set to "y"
* Start command -> set to "mlogger -D"
* Alarm Class -> set to "Alarm"

To test this, go to the "Programs" page, press "stop logger", then "start logger".

Start a run, wait 10 seconds, stop the run.

From the MIDAS "Status" page, go to the "logging channels -> channel #0" and set ODB variables:
* Output -> "FILE"

Start a run, go to the "Status" page, observe that the output file is written with gzip compression and has the "mid.gz" extension.

Stop the run.

== Run test frontend ==

Run fetest

On the MIDAS "Status" page there will be 3 new equipment entries:
* test -> empty data bank for daq speed benchmarking
* slow -> example slow controls data for testing the midas history plots
* random -> example collection of data banks for testing the analyzer

Start a run.

Observe event counters are incrementing, output data file size is growing.

Stop the run.

== Create a history plot ==

Go to the MIDAS "History" page.

Say "new", panel name "test". In the history editor select event "slow", say "save".

History plot "test" should open with a sine-wave plot. Data on the plot should update as long as "fetest" is running. (there is no need to start a run).

Press web browser "reload" to update the plot.

== Clients run on Localhost only ==
;NOTE:
* If creating a MIDAS experiment with a '''REMOTE''' frontend, '''continue by following the instructions''' [[#Running with one or more REMOTE frontends]].
* If all clients are running on the Experiment Host (i.e. '''localhost'''), continue with the following instructions:

=== Frontend and Analyzer (localhost) ===

THIS SECTION IS VERY OBSOLETE

There are several examples of [[Frontend user code]] ({{File|name=frontend.c}}) in the MIDAS package available under {{Filepath|path=$MIDASSYS/examples/}}. Choose a suitable example from this directory that you can later modify for your own particular setup (e.g. for a slow control, choose the {{Filepath|path=../slowcont/}} example).

A '''frontend''' is a program that usually reads data from the hardware and sends it to a buffer to be logged and analyzed. You can find documentation about the frontend structure under [[Frontend Application]], [[Frontend Operation]] and [[Frontend user code]].

The example chosen here is from {{Filepath|path=$MIDASSYS/examples/experiment/}}. It does '''not need any hardware''' and produces simulated events.

On the MIDAS ''experiment host'' (localhost), copy the example to the experiment directory and build it.

[mhostpc] cd $HOME/online
[mhostpc] cp $MIDASSYS/examples/experiment/* .
[mhostpc] make
The analyzer will only build if ROOT has previously been installed.
At this point the frontend and the analyzer should be ready if no errors were generated during the build. Try the frontend and analyzer by starting them in xterms.

[mhostpc] frontend
[mhostpc] analyzer

=== Equipment ODB tree ===
Starting the frontend will automatically create the {{Odbpath|path=/Equipment}} ODB tree and one or more equipments. In this case, the equipments "trigger" and "scaler" have been defined by code in {{File|name=frontend.c}} and created the first time frontend runs.

[mhostpc] odbedit
[local:e777:S]/>ls /equipment
Trigger
Scaler 

View the MIDAS main status web page (by pointing your browser to https://localhost:8443/ or appropriate port if not using the default).
Observe on the [[Status Page]] that the two Equipments (Trigger, Scaler) have appeared in the Equipment section (Figure 1) and are coloured green.
 
'''Click thumbnail to enlarge'''
[[File:quickstart_status_page.png|thumb|left|Figure 1: web server status page showing example frontend equipments]]
<div style="clear: both"></div> 

Rather than using odbedit, you can view the contents of these equipments by clicking on the ODB button, then on {{Odbpath|path=/Equipment}}, then {{Odbpath|path=Trigger}} or {{Odbpath|path=Scaler}}. See [[ODB Page]] for more information.

=== Start a run ===
Start a run by pressing the Start button on the [[Status Page]].
The Equipment display on the Status Page will now show some event statistics (Figure 2)
'''Click thumbnail to enlarge'''
[[File:sample_frontend_status.png|thumb|left|Figure 2: Equipments shown on Status Page showing event statistics]]
<div style="clear: both"></div> 

The frontend will show an event display that will update when a run is started, e.g.

Sample Frontend connected to <local>. Press "!" to exit 18:48:2546:08
==========================================================================
Run status: Running Run number 2 |/ /
===========================================================================
Equipment Status Events Events/sec Rate[B/s] ODB->FE FE->OD
---------------------------------------------------------------------------
Trigger OK 13948 99.0 5413.0 0 140
Scaler OK 15 0.3 15.4 0 15

The [[Status Page]] will also show events being generated (click the refresh button if necessary).

=== Event Dump ===
While a run is in progress, the midas application [[mdump]] will provide you an event dump of the collected data from the running frontend, e.g.

mhostpc> mdump -l 50
blank
- MIDAS revision: Mon Nov 2 11:50:51 2015 -0800 - 3b66779 -- Enter <!> to Exit ------- Midas Dump ---
------------------------ Event# 1 ------------------------
Evid:0001- Mask:0000- Serial:0- Time:0x567a0c5d- Dsize:40/0x28
#banks:2 - Bank list:-ADC0TDC0-
blank
Bank:ADC0 Length: 8(I*1)/2(I*4)/4(Type) Type:Unsigned Integer*2
1-> 0x0167 0x03c6 0x0069 0x0073
blank
Bank:TDC0 Length: 8(I*1)/2(I*4)/4(Type) Type:Unsigned Integer*2
1-> 0x0051 0x00ff 0x004a 0x00ec

For further data processing/analysis, either the midas analyzer or the rootana can used for data display as well.

=== Create a startup script (localhost) ===
It is useful to create a script that will automatically start all the required clients for the experiment. This is called a startup script. Here, the startup script will be {{File|name=start_daq.sh}} (any name can be used). Create {{Filepath|path=$HOME/online/bin/start_daq.sh}} on the ''experiment host'' as follows (supply the [[mhttpd]] ports if default is not used):

#!/bin/sh
# start_daq.sh
cd $HOME/online
odbedit -c clean
# start mhttpd on default port. (Mongoose https version)
mhttpd -D # you can optionally restrict access to localhost and other specified hosts - see [[mhttpd]]
xterm -e ./frontend &
xterm -e ./analyzer &
mlogger -D
#end file

Before running this script, you will need to shutdown any running clients:
mhostpc> odbedit -c "sh all"
or you can use the {{Utility|name=mhttpd}} [[Programs Page]] to shut them down.

The script will start them as daemons.
By running the startup script start_daq.sh, several MIDAS applications will be started in sequence:
# Cleanup previous MIDAS application (if any).
# Start the MIDAS web server [mhttpd]
# Start the frontend application in its own xterm (for debugging purpose).
# Start the analyzer application in its own xterm (for debugging purpose).
# Start the MIDAS Data logger [mlogger]
[mhostpc] sh ./start_daq

You may wish to modify the script to restart missing clients only.

Next [[#Customizing your experiment|customize your experiment]].

== Running with one or more REMOTE frontends ==
These instructions assume you have already followed the instructions to setup an experiment running on the MIDAS experiment host (localhost), i.e. you have setup the MIDAS Environment variables, downloaded MIDAS packages, setup the [[mhttpd]] password and have {{Utility|name=mhttpd}} running successfully.

In the case of '''remote''' frontend(s), the remote cpu(s) typically have access to some or all of the hardware. They might be VMIC cpus running in VME crates for example, connected to the MIDAS Host via Ethernet. They read data from the hardware and send it back to the MIDAS ''experiment host'' to be logged and analyzed.

=== Experimental setup (Remote) ===
These instructions assume that the MIDAS ''experiment host'' computer is 64-bit (if not see [[#Build 32-bit MIDAS libraries|note below]]), and there is one ''remote host'', a 32-bit machine named "rlxhost".

The ''remote host'' mounts the {{Filepath|path=/home}} disk of the ''experiment host'', so that it '''has access to the MIDAS packages''', the {{Filepath|path=$HOME/online}} directory and shares the .cshrc (or .bashrc) script. It is convenient to set up ssh keys (use ssh-keygen) so that the ''remote host'' can be accessed without supplying the password from the ''experiment host''.

If your setup is different, you will have to make changes as appropriate.

=== Build 32-bit MIDAS libraries ===
NOTE

If the MIDAS ''experiment host'' computer is 32-bit (i.e. the same as the ''remote host''), which is a simpler setup, you will have built the 32-bit MIDAS libraries already, and can skip the rest of this section. Note however, that in the .cshrc file in the [[#Modify .cshrc (.bashrc) for Remote host|following section]] you will not need the lines starting "# select 64-bit or 32-bit MIDAS and ROOT". The PATH should contain $MIDASSYS/linux/bin on both experiment host and remote host. This will point to the 32-bit MIDAS libraries.
------------------------------

 
Assuming the MIDAS host computer is 64-bit, you will have already built the 64-bit MIDAS libraries.
In this case, if the remote frontend is 32-bit, you will need to build the 32-bit MIDAS libraries on the 64-bit machine, i.e.

[mhostpc] cd /home/packages/midas
[mhostpc] make linux32 ### build the 32-bit MIDAS libraries on 64-bit machine
[mhostpc] ls -l linux-m32/bin/odbedit ### check that the 32-bit odbedit has been created 
Do not run {{Utility|name=odbedit}} yet on the ''remote host''!

=== Modify .cshrc (.bashrc) for Remote host ===
Add the following to the {{File|name=.cshrc}} (modified appropriately if using {{File|name=.bashrc}}). When running with a REMOTE Host, the application [[mserver]] will be started (using a particular port) on the MIDAS ''experiment host'' machine only. Note that the environment variable MIDAS_SERVER_HOST is defined on the remote machine(s) but NOT on the ''experiment host''.

# setenv MIDAS_HOST mhostpc # substitute your experiment host name
# setenv REMOTE_HOST rlxhost # substitute your remote host name
# setup the MIDAS mserver host
#
switch (`hostname`)
case $MIDAS_HOST*:
unsetenv MIDAS_SERVER_HOST ## MIDAS_SERVER_HOST not defined
breaksw
default:
setenv MIDAS_SERVER_HOST $MIDAS_HOST.triumf.ca:1175 ## for remote host, define MIDAS_SERVER_HOST
## as hostname + domain name + default mserver port
## port must match that of [[mserver]] used in start_daq.sh
endsw
#
# select 64-bit or 32-bit MIDAS and ROOT
#
switch (`uname -i`)
case i386:
setenv ROOTSYS /triumfcs/trshare/olchansk/root/root_v5.28.00_SL55_32 ## or appropriate path
setenv PATH .:$MIDASSYS/linux-m32/bin:$PATH
breaksw
default:
setenv ROOTSYS $HOME/packages/root
setenv PATH .:$MIDASSYS/linux/bin:$PATH
endsw
#
setenv PATH .:$HOME/online/bin:$HOME/packages/roody/bin:$ROOTSYS/bin:$PATH
#

After executing {{File|name=.cshrc}} (or {{File|name=.bashrc}}) or logging out, on the MIDAS host (mhostpc) make sure environment variable MIDAS_SERVER_HOST is NOT defined
[mhostpc] echo $MIDAS_SERVER_HOST
[mhostpc] MIDAS_SERVER_HOST: Undefined variable.

and on the '''''remote host''''', make sure MIDAS_SERVER_HOST IS defined:
[rlxhost] echo $MIDAS_SERVER_HOST
[rlxhost] MIDAS_SERVER_HOST

Also on the ''remote host'', make sure that the correct (32-bit) {{Utility|name=odbedit}} will be used (do not run it yet!)
[rlxhost] which odbedit
/home/johnfoo/packages/midas/linux-m32/bin/odbedit

=== Grant Remote host access permission ===
Give permission for the remote host(s) to access the experiment by following the instructions to [[Security#MIDAS programs on remote machines|allow MIDAS programs on remote machines]]. If the MIDAS web server [[mhttpd]] is running, use [[ODB Page]] to edit the ODB keys listed, otherwise use [[odbedit]] on the ''experiment host'' machine.

=== Start mserver on Experiment Host ===
The application [[mserver]] is not required when running an experiment on the ''experiment host'' (localhost) only, but it '''IS required''' for access from a '''REMOTE''' frontend.
Start [[mserver]] on MIDAS host (default port 1175) or use the "-p" option for a different port.
The port must match the port defined for environment variable MIDAS_SERVER_HOST in {{File|name=.cshrc}} (or {{File|name=.bashrc}}) [[#Modify .cshrc (.bashrc) for Remote host|above]].
[mhostpc] mserver -D

The environment variables on the REMOTE host should be:

[rlxhost] printenv | grep MIDAS
MIDASSYS=/home/e777/packages/midas
MIDAS_EXPT_NAME=e777
MIDAS_SERVER_HOST=mhostpc.triumf.ca:1175
(a different port may be used - see above)

=== Check access to ODB from Remote host ===
Run {{Utility|name=odbedit}} on the remote host to check access to ODB.
[rlxhost] odbedit
[mhostpc.triumf.ca:1175:e777:Stopped]/>ls 
If it does not work correctly, check that
* MIDAS_SERVER_HOST is defined on REMOTE host with same port as [[mserver]]
* mserver is running on MIDAS host with the correct port
* MIDAS_SERVER_HOST is NOT defined on MIDAS host
* remote access permission to REMOTE host has been granted as described above.

=== Build the analyzer on Experiment Host ===
The analyzer will be built and run on the MIDAS ''experiment host'' machine.
Select an example from $MIDASSYS/examples/ (see [[#Frontend and Analyzer (localhost)]]) and copy it to $HOME/online on the MIDAS host machine. Since we are assuming a 64-bit MIDAS host and a 32-bit remote host, edit the example Makefile so only the analyzer will be built.

=== Build the frontend (32bit) for Remote host ===
The frontend runs on a 32-bit remote host in this example. It therefore must either
* be built 32-bit on the 64-bit MIDAS host with the "-m32" flag
* or built on the REMOTE host
and in either case the [[#Build 32-bit MIDAS libraries|32-bit MIDAS libraries]] used for linking.

To avoid confusion, create a different subdirectory for the frontend, e.g.
[rlxhost] mkdir $HOME/online/fe_32
Copy the Makefile and frontend code from the same example to this directory.
Modify the Makefile as needed, and only build the frontend.

Once built, start the frontend in an xterm on the REMOTE host.
[rlxhost] $HOME/online/fe_32/frontend
Check that it connects successfully to the experiment and the Equipments appear on the [[Status Page]] as described under [[#Frontend and Analyzer (localhost)]]. The Equipment Status (Figure 2) should show that the Equipments are now running on the remote cpu, i.e. "SampleFrontend@rlxhost.triumf.ca" rather than "Sample Frontend@localhost".

=== Run the Analyzer on Experiment host ===
Start the analyzer in an xterm on the MIDAS ''experiment host''
[johnfoo@mhostpc e777] $HOME/online/analyzer

Continue by starting a run and dumping the data as described under [[#Frontend and Analyzer (localhost)]].

=== Create experiment startup script ===
It is useful to create a script that will automatically start all the required clients for the experiment. This is called a startup script (any name can be used). Create {{Filepath|path=$HOME/online/bin/start_daq.sh}} on the ''experiment host'' as follows, (supply the [[mhttpd]] ports if default is not used) :

#!/bin/sh
# start_daq.sh
cd $HOME/online
#
switch (`hostname`)
case $MIDAS_HOST*: ## MIDAS_HOST is assigned in .cshrc (.bashrc)
echo "Good, we are on $MIDAS_HOST"
breaksw
case $REMOTE_HOST*: ## REMOTE_HOST is assigned in .cshrc (.bashrc)
echo "startup script should be executed on $MIDAS_HOST"
endsw
#
odbedit -c clean
# start mhttpd on default port (Mongoose https version)
mhttpd -D # you can optionally restrict access to localhost and/or other specified hosts - see [[mhttpd]]
# start mserver on default port (use argument -p to use a different port)
mserver -D # start mserver on default port (1175) or a different port using -p option - see [[mserver]]
# the port must match that of MIDAS_SERVER_HOST in .cshrc (.bashrc) - see above
# access must be specifically allowed - see [[#Grant Remote host access permission||above]]
xterm -e ./analyzer &
## start frontend on remote host
#ssh $REMOTE_HOST $HOME/online/bin/start_frontend -O >& $HOME/online/bin/start_frontend.log &
mlogger -D
#end file

Note that command to start the frontend on the remote host is commented out. It can be uncommented once the start_frontend script is written and tested.

Shut down any clients that are running already by running odbedit on the MIDAS ''experiment host'' and execute the new startup script {{File|name=start_daq.sh}} (make it executable if desired)

[mhostpc] odbedit
[local:e777:S]/>sh all
[mhostpc] ./start_daq.sh 
and check that {{Utility|name=odbedit}} still works on the ''REMOTE host''

[rlxhost] odbedit
[mhostpc.triumf.ca:1175:e777:Stopped]/>ls 

You may wish to modify the script to restart missing clients only.

=== Create script to start frontend ===
The startup file {{File|name=start_daq.sh}} invoked a script {{File|name=start_frontend}} to start the remote frontend. This line was commented out (see [[#Create experiment startup script|above]]). Once the frontend is working, create a script to start the frontend (e.g. file {{Filepath|path=$HOME/online/bin/start_frontend}}).

#!/bin/tcsh
# Script to start frontend running on $REMOTE_HOST
#
# This script runs on the REMOTE HOST
#
echo "starting frontend for experiment $MIDAS_EXPT_NAME "
xterm -geometry 150x50+800+0 -fg white -bg blue -title "Frontend" -e "$HOME/online/fe_32/frontend" &
echo "Frontend task has been started"
exit

Check that this script works by running it on the REMOTE host. If successful,
run the commented command in an xterm on the ''experiment host''
[mhostpc] ssh $REMOTE_HOST $HOME/online/bin/start_frontend -O >& $HOME/online/bin/start_frontend.log & 
This command relies on password-less access set up with SSH keys (see [[#Experimental setup (Remote)]]).
If successful, uncomment line in {{File|name=start_daq.sh}}, and run the script to check that it can start the remote client(s) from the MIDAS ''experiment host'' (first shutting down all running clients) :

[mhostpc] odbedit -c "sh all"
[mhostpc] ./start_daq.sh 

Next [[#Customizing your experiment|customize your experiment]].

== Customizing your experiment ==

Here are some of the more common options provided for users to customize their experiments according to their own preferences:

; Write frontend(s) to read out and control your hardware
: Modify the example frontend and Makefile (from $MIDASSYS/examples/) you have already copied and built so it accesses your hardware.

: The example Frontend.c is an example of [[Frontend user code]]. You can find documentation about the frontend structure under [[Frontend Application]], [[Frontend Operation]] and [[Frontend user code]]. You may need to set up some [[Event Notification (Hot-Link)|Hot-Links]] in the frontend. The MIDAS package includes device drivers for some of the more common hardware. See [[MIDAS Driver Library]] for more information.

; Modify analyzer(s) for your data
: Modify the example analyzer code as necessary to analyze the data from your frontends.

; Slow Controls
: Control your slow-controls equipment with slow-control frontend(s) optionally using the [[#MIDAS Package Installation|MSCB]] system.

; Data logger
: Set up the data logger [[mlogger]] to log data to a storage device. Many options are available - see [[Data Logger]], [[mlogger]] and [[/Logger ODB tree]] for instructions.

; Allow programs to be restarted from the [[Programs Page]]
: See mhttpd [[Programs Page]] and [[/Programs ODB tree]]

; Write scripts which run at start and end of run
: See [[/Programs ODB tree#Execute on start run]] and [[/Programs ODB tree#Execute on stop run]]

; Create a shutdown script
: Create a script similar to the startup script to shutdown the experiment

; Lazy logger
: Set up the [[lazylogger]] for archiving data files for storage.

; Create edit-on-start parameters
: See [[Edit-on-start Parameters]]

; Set up the alarm system
: See [[Alarm System]] and [[Alarms Page]]

; Set up history system
: See [[History System]] and [[History Page]]

; Set up the Electronic logbook (Elog)
: See [[Elog]] for details

; Set up the Sequencer
: The [[Sequencer Page#Sequencer]] can be used to run a sequence of runs, changing parameters as needed.

; Write a webserver Custom page
: To better control and monitor your experiment, you may want to write a [[Custom Page]].

== Start MIDAS on boot ==

To start MIDAS mhttpd, mlogger and elogd at boot time, add following cron entries:
<pre>
[iris@iris00 ~]$ crontab -l
@reboot /home/iris/packages/midas/linux/bin/mhttpd -D
@reboot /home/iris/packages/midas/linux/bin/mlogger -D
@reboot /home/iris/packages/elog/elogd -c /home/iris/packages/elog/elogd.cfgiris -D
[iris@iris00 ~]$
</pre>

[[Category:Installation]]

Quickstart Linux

2017-03-02T00:19:30Z

Olchansk: /* Customizing your experiment */

Security

2017-03-01T23:51:30Z

Olchansk: /* MIDAS programs on remote machines */

{{Pagelinks}}
= Links =
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[mhttpd]] web server
* [[odbedit]]
* [[/Experiment ODB tree #/Experiment/Security |/Experiment/Security]] ODB subtree
* [[mserver]]
</div>
= Access Control to a MIDAS experiment =

Historically, by default there has been no restriction for any user to connect locally or remotely to a given experiment. This has now changed (August 2015).
The Web Server [[mhttpd]] by default now requires an SSL Certificate and a username and password (see [[#Web Access]]).

Other network security issues have also been addressed (see [https://midas.triumf.ca/elog/Midas/1088 Note 1088]). Changes to [[mserver]] (August 2015) make the default behaviour to reject all external network connections ([https://midas.triumf.ca/elog/Midas/1090 Note 1090]), so that only programs running on the localhost will be allowed. To allow access from remote machines, the user will have to take action as described [[#MIDAS programs on remote machines|below]].

== Web Access ==
In versions prior to May 2015, the default web access to MIDAS and ELOG uses the "http:" protocol which is insecure. In this case, all information is transmitted as clear text meaning that secret, confidential and sensitive information (such as the MIDAS and ELOG passwords and usernames) can be stolen "easily". This means that even "password protected" MIDAS and ELOG pages are not adequately protected if accessed using the "http" method. Until now, an SSL (https) proxy had to be set up to provide reasonable security for experiments.

Recent versions of elogd (ELOG) do support SSL https:// connections, and
the Web Server [[mhttpd]] has recently (May 2015) been updated to use OpenSSL to provide secure HTTPS connections via the [https://bitbucket.org/tmidas/midas/src/ecb9a8537448a8a43f7f9a2bfdb82e578208cde3/doc/mongoose/?at=develop Mongoose] web server. This limits access by default, by requiring a username and password - see [[mhttpd#HTTPS/SSL server (Mongoose)|HTTPS/SSL server (Mongoose)]] for more information. This option provides a similar level of security to an HTTP proxy.

=== Which security option to choose? ===

For a small operation that does not require root access to the host computer, mhttpd with mongoose is a good light weight solution. It is ideal for small experiments that until now have been using the "http:" protocol for mhttpd.

For a more elaborate setup with private networks etc., APACHE https proxy is probably better. For big experiments, resources like webcams,ganglia, couchdb, etc. also need password protection and APACHE https can provide all this.

=== Which one is easier to setup? ===

* mhttpd with mongoose is fairly simple - you have to create a password file and provide a properly signed https certificate - see [[mhttpd#HTTPS/SSL server (Mongoose)|HTTPS/SSL server (Mongoose)]]
* apache httpd is fairly straightforward if you follow well written instructions (such as we provide for using it with midas), but you do need root access and you do have to edit a good number of config files. Note that mhttpd with mongoose can be run with an SSL proxy - see [[Setup MIDAS experiment#using an HTTPS/SSL proxy]]

== MIDAS programs running on localhost ==

Out-of-the-box MIDAS is now secure (August 2015). By default, connections from the outside are not possible. MIDAS RPC TCP ports are bound to the localhost interface.
This configuration is suitable for testing MIDAS on a laptop and for running a simple experiment where all programs run on one machine.
MIDAS ports (except for the mhttpd web ports) do not show up on network port scans.

The change in binding UDP ports is generally incompatible with previous versions of MIDAS, so all MIDAS programs should be rebuild and restarted. If rebuilding all MIDAS programs is
impossible see [https://midas.triumf.ca/elog/Midas/1090 Note 1090]).

[[mserver]] will still work in this localhost-restricted configuration - one should use "[[odbedit]] -h localhost" to connect. Multiple mserver instances on
the same machine - using different TCP ports via "-p" and ODB [[/Experiment ODB tree#midas server port|/Experiment/midas server port]] - are still supported.

== MIDAS programs on remote machines ==

To run MIDAS programs on remote machines the following is now required:
# change the ODB setting [[/Experiment ODB tree#Enable non-localhost RPC|/Experiment/Security/Enable non-localhost RPC]] to "yes" and restart [[mserver]]
# add the hostnames of all remote machines that will run MIDAS programs to the MIDAS RPC access control list in ODB key [[/Experiment ODB tree#Allowed hosts|/Experiment/Security/RPC hosts/Allowed hosts]].
# if needed, add firewall rules on both the MIDAS host and on all the MIDAS client machines - each machine running MIDAS programs must have permission to connect to all other machines running MIDAS programs.
<pre>
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="x.y.z.w/32" port protocol="tcp" port="0-65535" accept"
... repeat for the midas host and for all client machines, replace x.y.z.w with the actual IP addresses.
firewall-cmd --reload
firewall-cmd --list-all
</pre>

To avoid "guessing" the host names expected by MIDAS, follow the following procedure:
* On the local machine ("daq06") set ODB key "enable non-localhost rpc" to "yes" and restart the [[mserver]] (step 1 above)
* go to the remote machine ("ladd21") and try to start the MIDAS program, i.e. "[[odbedit]] -h daq06". This will bomb and there will be a message in the [[Message System#midas.log|Midas log file]] rejecting the connection from unallowed host 'ladd21.triumf.ca'.
* Add this host to /Experiment/Security/RPC hosts/Allowed hosts.
* After you add this hostname to RPC hosts, you should see messages in the [[Message System#midas.log|Midas log file]] about reloading the access control list
* try connecting again, it should work now.

;NOTE
If MIDAS clients have to connect from random hosts (i.e. dynamically assigned random DHCP addresses), one can disable the host name checks by setting ODB key [[/Experiment ODB tree#Disable RPC hosts check|/experiment/security/Disable RPC hosts check]] to "yes". This configuration is insecure and should only be done on a private network behind a firewall.

= ODB access control =
Note that '''other forms of ODB access control''' independent of these security features is also available which can protect the experiment from accidental access by authorized users:
* Write access can be restricted while a run is in progress (see [[Lock when running]] )
* Individual keys or subtrees in the experiment's ODB can be set "read only" with the [[odbedit]] command chmod.
* [[Custom Page|Custom web pages]] can provide experimenters with access to a subset of ODB keys necessary for the experiment. By [[Status Page#suppressed|hiding]] the [[ODB Page]] access button, the ODB can be protected from non-expert access via the web server.

 
------------
 

<div style="background-color:whitesmoke">

= Security on OLDER (pre-August 2015) versions of MIDAS =

Versions of MIDAS older than August 2015 are vulnerable to malicious/unauthorized access. Network security on old versions can be enhanced by implementing a '''firewall''' and/or restrictions on off-site access. This kind of security can be provided by [[Setup MIDAS experiment#Setting up an HTTP proxy|setting up an HTTP proxy]] to [[mhttpd]] .

== Protect experiment from accidental access (old MIDAS versions) ==

The MIDAS system provides a means to setup access restrictions using the ODB in order to protect the experiment from accidental or unauthorized access. '''This will not stop malicious or determined hackers''' but may prevent mistakes by authorized users. These security features are still available, but their usefulness has largely been superceded by the improved security features implemented August 2015 (see [[#Access Control to a MIDAS experiment]]);

There are two levels of access restriction available each of which can be enabled independently:

* To restrict write access via the web by requiring a password before any parameter can be changed.
* To require a password before MIDAS clients can start running on the host.

The user can select either or both of these security features.

== Restrict user access (old MIDAS versions) ==

'''NOTE: this is not proof against malicious access.''' See [[#Access Control to a MIDAS experiment]].

The ODB [[/Experiment ODB tree #/Experiment/Security |/Experiment/Security]] subtree can also be used to restrict access to the experiment via the Web. This subtree is automatically created (if not already present) when the {{Odbedit cmd|cmd=webpasswd}} is issued as follows:

C:\online>odbedit
[local:Default:S]/>cd Experiment/
[local]/>webpasswd
Password:<xxxx>
Retype password:<xxxx>

After running the {{Odbedit cmd|cmd=webpasswd}}, a new ODB key i.e. [[/Experiment ODB tree #/Experiment/Security/Web Password |/Experiment/Security/Web Password]] will
be present containing the encrypted web password.

If this web access restriction is set up, the user will be requested to provide the "Web Password" when accessing the requested experiment in "Write Access" mode. The "Read Only Access" mode is still available to all users of older versions of [[mhttpd]], or those who supply the username and password now required by [[mhttpd]] (May 2015).

[local:bnqr:S]/Experiment>ls Security/
Web Password pon4@#@%SSDF2

== Restrict client access (old MIDAS versions) ==
'''NOTE: this is not proof against malicious access.''' See [[#Access Control to a MIDAS experiment]].

In order to restrict accidental access to the experiment, a password mechanism can to be defined. This is provided by the [[/Experiment ODB tree #/Experiment/Security |/Experiment/Security]] subtree in odb. This subtree is automatically created (if not already present) when the [[odbedit]] command passwd is issued as follows:

C:\online>odbedit
[local:Default:S]/>cd Experiment/
[local]/>passwd
Password:<xxxx>
Retype password:<xxxx>

After running the {{Odbedit cmd|cmd=passwd}}, the {{Odbpath|path=/Experiment/Security}} subtree will be present.

=== Allowing access without password (old MIDAS versions) ===
NOTE: This feature has been superceded by [[#Access Control to a MIDAS experiment]] for [[mhttpd]] versions May 2015 or later.

While [[#Restrict client access|restricting access]] can make sense to deny access to outsider to a given experiment, it can be annoying for the people working directly at the back-end computer or for an automatic frontend reloading mechanism. To address this problem, specific hosts and clients can be exempt from having to supply a password before being granted full access.

==== Allowed hosts (old MIDAS versions) ====
NOTE: This feature has been superceded by [[#Access Control to a MIDAS experiment]] for [[mhttpd]] versions May 2015 or later.

Hostnames to be allowed full access to the ODB are listed in the [[/Experiment ODB tree #/Experiment/Security/Allowed hosts subtree |/Experiment/Security/Allowed hosts]] subtree, e.g.

[local]/>cd "/Experiment/Security/Allowed hosts"
[local]rhosts>create int myhost.domain
[local]rhosts>

where <myHost.domain> is to be replaced by the full IP address of the host requesting full clearance, e.g "pierre.triumf.ca".

==== Allowed programs (old MIDAS versions) ====
NOTE: This feature has been superceded by [[#Access Control to a MIDAS experiment]] for [[mhttpd]] versions May 2015 or later.

Programs (i.e. clients) to be allowed full access to the ODB (regardless of the node on which they are running) can be listed in the
[[/Experiment ODB tree #/Experiment/Security/Allowed programs subtree |/Experiment/Security/Allowed programs]] subtree,

[local]/>cd "/Experiment/Security/Allowed programs"
[local]:S>create int mstat
[local]:S>

</div> 

[[Category:Security]] [[Category:Experiment]]

Security

2017-03-01T23:50:00Z

Olchansk: /* MIDAS programs on remote machines */

{{Pagelinks}}
= Links =
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[mhttpd]] web server
* [[odbedit]]
* [[/Experiment ODB tree #/Experiment/Security |/Experiment/Security]] ODB subtree
* [[mserver]]
</div>
= Access Control to a MIDAS experiment =

Historically, by default there has been no restriction for any user to connect locally or remotely to a given experiment. This has now changed (August 2015).
The Web Server [[mhttpd]] by default now requires an SSL Certificate and a username and password (see [[#Web Access]]).

Other network security issues have also been addressed (see [https://midas.triumf.ca/elog/Midas/1088 Note 1088]). Changes to [[mserver]] (August 2015) make the default behaviour to reject all external network connections ([https://midas.triumf.ca/elog/Midas/1090 Note 1090]), so that only programs running on the localhost will be allowed. To allow access from remote machines, the user will have to take action as described [[#MIDAS programs on remote machines|below]].

== Web Access ==
In versions prior to May 2015, the default web access to MIDAS and ELOG uses the "http:" protocol which is insecure. In this case, all information is transmitted as clear text meaning that secret, confidential and sensitive information (such as the MIDAS and ELOG passwords and usernames) can be stolen "easily". This means that even "password protected" MIDAS and ELOG pages are not adequately protected if accessed using the "http" method. Until now, an SSL (https) proxy had to be set up to provide reasonable security for experiments.

Recent versions of elogd (ELOG) do support SSL https:// connections, and
the Web Server [[mhttpd]] has recently (May 2015) been updated to use OpenSSL to provide secure HTTPS connections via the [https://bitbucket.org/tmidas/midas/src/ecb9a8537448a8a43f7f9a2bfdb82e578208cde3/doc/mongoose/?at=develop Mongoose] web server. This limits access by default, by requiring a username and password - see [[mhttpd#HTTPS/SSL server (Mongoose)|HTTPS/SSL server (Mongoose)]] for more information. This option provides a similar level of security to an HTTP proxy.

=== Which security option to choose? ===

For a small operation that does not require root access to the host computer, mhttpd with mongoose is a good light weight solution. It is ideal for small experiments that until now have been using the "http:" protocol for mhttpd.

For a more elaborate setup with private networks etc., APACHE https proxy is probably better. For big experiments, resources like webcams,ganglia, couchdb, etc. also need password protection and APACHE https can provide all this.

=== Which one is easier to setup? ===

* mhttpd with mongoose is fairly simple - you have to create a password file and provide a properly signed https certificate - see [[mhttpd#HTTPS/SSL server (Mongoose)|HTTPS/SSL server (Mongoose)]]
* apache httpd is fairly straightforward if you follow well written instructions (such as we provide for using it with midas), but you do need root access and you do have to edit a good number of config files. Note that mhttpd with mongoose can be run with an SSL proxy - see [[Setup MIDAS experiment#using an HTTPS/SSL proxy]]

== MIDAS programs running on localhost ==

Out-of-the-box MIDAS is now secure (August 2015). By default, connections from the outside are not possible. MIDAS RPC TCP ports are bound to the localhost interface.
This configuration is suitable for testing MIDAS on a laptop and for running a simple experiment where all programs run on one machine.
MIDAS ports (except for the mhttpd web ports) do not show up on network port scans.

The change in binding UDP ports is generally incompatible with previous versions of MIDAS, so all MIDAS programs should be rebuild and restarted. If rebuilding all MIDAS programs is
impossible see [https://midas.triumf.ca/elog/Midas/1090 Note 1090]).

[[mserver]] will still work in this localhost-restricted configuration - one should use "[[odbedit]] -h localhost" to connect. Multiple mserver instances on
the same machine - using different TCP ports via "-p" and ODB [[/Experiment ODB tree#midas server port|/Experiment/midas server port]] - are still supported.

== MIDAS programs on remote machines ==

To run MIDAS programs on remote machines the following is now required:
# change the ODB setting [[/Experiment ODB tree#Enable non-localhost RPC|/Experiment/Security/Enable non-localhost RPC]] to "yes" and restart [[mserver]]
# add the hostnames of all remote machines that will run MIDAS programs to the MIDAS RPC access control list in ODB key [[/Experiment ODB tree#Allowed hosts|/Experiment/Security/RPC hosts/Allowed hosts]].
# if needed, add firewall rules on both the MIDAS host and on all the MIDAS client machines:
<pre>
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="x.y.z.w/32" port protocol="tcp" port="0-65535" accept"
... repeat for all client machines
firewall-cmd --reload
firewall-cmd --list-all
</pre>

To avoid "guessing" the host names expected by MIDAS, follow the following procedure:
* On the local machine ("daq06") set ODB key "enable non-localhost rpc" to "yes" and restart the [[mserver]] (step 1 above)
* go to the remote machine ("ladd21") and try to start the MIDAS program, i.e. "[[odbedit]] -h daq06". This will bomb and there will be a message in the [[Message System#midas.log|Midas log file]] rejecting the connection from unallowed host 'ladd21.triumf.ca'.
* Add this host to /Experiment/Security/RPC hosts/Allowed hosts.
* After you add this hostname to RPC hosts, you should see messages in the [[Message System#midas.log|Midas log file]] about reloading the access control list
* try connecting again, it should work now.

;NOTE
If MIDAS clients have to connect from random hosts (i.e. dynamically assigned random DHCP addresses), one can disable the host name checks by setting ODB key [[/Experiment ODB tree#Disable RPC hosts check|/experiment/security/Disable RPC hosts check]] to "yes". This configuration is insecure and should only be done on a private network behind a firewall.

= ODB access control =
Note that '''other forms of ODB access control''' independent of these security features is also available which can protect the experiment from accidental access by authorized users:
* Write access can be restricted while a run is in progress (see [[Lock when running]] )
* Individual keys or subtrees in the experiment's ODB can be set "read only" with the [[odbedit]] command chmod.
* [[Custom Page|Custom web pages]] can provide experimenters with access to a subset of ODB keys necessary for the experiment. By [[Status Page#suppressed|hiding]] the [[ODB Page]] access button, the ODB can be protected from non-expert access via the web server.

 
------------
 

<div style="background-color:whitesmoke">

= Security on OLDER (pre-August 2015) versions of MIDAS =

Versions of MIDAS older than August 2015 are vulnerable to malicious/unauthorized access. Network security on old versions can be enhanced by implementing a '''firewall''' and/or restrictions on off-site access. This kind of security can be provided by [[Setup MIDAS experiment#Setting up an HTTP proxy|setting up an HTTP proxy]] to [[mhttpd]] .

== Protect experiment from accidental access (old MIDAS versions) ==

The MIDAS system provides a means to setup access restrictions using the ODB in order to protect the experiment from accidental or unauthorized access. '''This will not stop malicious or determined hackers''' but may prevent mistakes by authorized users. These security features are still available, but their usefulness has largely been superceded by the improved security features implemented August 2015 (see [[#Access Control to a MIDAS experiment]]);

There are two levels of access restriction available each of which can be enabled independently:

* To restrict write access via the web by requiring a password before any parameter can be changed.
* To require a password before MIDAS clients can start running on the host.

The user can select either or both of these security features.

== Restrict user access (old MIDAS versions) ==

'''NOTE: this is not proof against malicious access.''' See [[#Access Control to a MIDAS experiment]].

The ODB [[/Experiment ODB tree #/Experiment/Security |/Experiment/Security]] subtree can also be used to restrict access to the experiment via the Web. This subtree is automatically created (if not already present) when the {{Odbedit cmd|cmd=webpasswd}} is issued as follows:

C:\online>odbedit
[local:Default:S]/>cd Experiment/
[local]/>webpasswd
Password:<xxxx>
Retype password:<xxxx>

After running the {{Odbedit cmd|cmd=webpasswd}}, a new ODB key i.e. [[/Experiment ODB tree #/Experiment/Security/Web Password |/Experiment/Security/Web Password]] will
be present containing the encrypted web password.

If this web access restriction is set up, the user will be requested to provide the "Web Password" when accessing the requested experiment in "Write Access" mode. The "Read Only Access" mode is still available to all users of older versions of [[mhttpd]], or those who supply the username and password now required by [[mhttpd]] (May 2015).

[local:bnqr:S]/Experiment>ls Security/
Web Password pon4@#@%SSDF2

== Restrict client access (old MIDAS versions) ==
'''NOTE: this is not proof against malicious access.''' See [[#Access Control to a MIDAS experiment]].

In order to restrict accidental access to the experiment, a password mechanism can to be defined. This is provided by the [[/Experiment ODB tree #/Experiment/Security |/Experiment/Security]] subtree in odb. This subtree is automatically created (if not already present) when the [[odbedit]] command passwd is issued as follows:

C:\online>odbedit
[local:Default:S]/>cd Experiment/
[local]/>passwd
Password:<xxxx>
Retype password:<xxxx>

After running the {{Odbedit cmd|cmd=passwd}}, the {{Odbpath|path=/Experiment/Security}} subtree will be present.

=== Allowing access without password (old MIDAS versions) ===
NOTE: This feature has been superceded by [[#Access Control to a MIDAS experiment]] for [[mhttpd]] versions May 2015 or later.

While [[#Restrict client access|restricting access]] can make sense to deny access to outsider to a given experiment, it can be annoying for the people working directly at the back-end computer or for an automatic frontend reloading mechanism. To address this problem, specific hosts and clients can be exempt from having to supply a password before being granted full access.

==== Allowed hosts (old MIDAS versions) ====
NOTE: This feature has been superceded by [[#Access Control to a MIDAS experiment]] for [[mhttpd]] versions May 2015 or later.

Hostnames to be allowed full access to the ODB are listed in the [[/Experiment ODB tree #/Experiment/Security/Allowed hosts subtree |/Experiment/Security/Allowed hosts]] subtree, e.g.

[local]/>cd "/Experiment/Security/Allowed hosts"
[local]rhosts>create int myhost.domain
[local]rhosts>

where <myHost.domain> is to be replaced by the full IP address of the host requesting full clearance, e.g "pierre.triumf.ca".

==== Allowed programs (old MIDAS versions) ====
NOTE: This feature has been superceded by [[#Access Control to a MIDAS experiment]] for [[mhttpd]] versions May 2015 or later.

Programs (i.e. clients) to be allowed full access to the ODB (regardless of the node on which they are running) can be listed in the
[[/Experiment ODB tree #/Experiment/Security/Allowed programs subtree |/Experiment/Security/Allowed programs]] subtree,

[local]/>cd "/Experiment/Security/Allowed programs"
[local]:S>create int mstat
[local]:S>

</div> 

[[Category:Security]] [[Category:Experiment]]

Setup MIDAS experiment at TRIUMF

2016-11-21T21:44:12Z

Olchansk: /* Prepare the user account */

{{Pagelinks}}
== Introduction ==
This page describes setting up a MIDAS experiment at TRIUMF. This information can be adapted for other sites.

== Environment variables ==
* '''MIDASSYS''' Base directory of the MIDAS package, midas and mxml should be at the same level.
* '''MIDAS_EXPTAB''' Experiment definition file
* '''MIDAS_SERVER_HOST''' MIDAS host server name for remote midas connections.
* '''MIDAS_EXPT_NAME''' Experiment name

== Standard layout of MIDAS experiment ==
The following shows the directory layout of a standard MIDAS experiment:

/home/exptuser/
packages/
root <---- ROOT
mxml
mscb
midas/ <---- MIDAS
linux/{lib,bin} <---- binaries matching the selected 64-bit/32-bit flavour of ROOT
linux-m32/{lib,bin} <---- limited function 32-bit binaries for 32-bit frontend machines, build by "make linux32"
linux-m64/{lib,bin} <---- limited function 64-bit binaries (only needed if ROOT and linux/bin are 32-bit)
linux-arm/{lib,bin} <---- full function ARM cross-compiled using "make linuxarm"
linux-crosscompile/{lib,bin} <---- cross-compiled limited function binaries for PPC and ARM frontends (see Makefile)
rootana <---- ROOT analyzer
roody <---- graphical online histogram viewer for MIDAS and ROOTANA
online/
exptab <---- experiment definition
{.ODB,.SYSTEM,.SYSMSG,etc}.SHM <---- MIDAS shared memory save files
src <---- experiment frontend sources
bin,scripts
elog <---- MIDAS elog
history <---- MIDAS history
data -> /data/exptname/current <---- symlink to the data directory
/data/exptname/current <---- experiment data directory with ODB save files and MIDAS .mid/.mid.gz data files

== Prepare computers ==

On some operating systems, several MIDAS functions require administrator access:

* on el7 linux (SL7/CC7/CentOS7/RHEL7) - access to mhttpd port 8443 requires special firewall rules, see here: https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29
* on el7 linux - access to mserver to run frontends and other programs on some other computer requires special firewall rules, see here: https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29
* on el7 linux - on the frontend machines (and other machines that will connect to the mserver, the same firewall rule needs to be created (use the IP address of the machine running the mserver)

== Prepare the user account ==
<div id="NOTES"></div>

* Setup the user account for running this instance of midas. For machines part of the LADD cluster, follow these [http://daq-plone.triumf.ca/SM/docs/local/NewLaddUser] instructions.
* check that the account is using the /bin/bash shell
* make $HOME/.profile look like this:

<pre>
#!/bin/echo You must source

export SVN_EDITOR="emacs -nw"
export GIT_EDITOR="emacs -nw"
export MIDASSYS=$HOME/packages/midas
export ROOTANASYS=$HOME/packages/rootana
export MIDAS_EXPTAB=$HOME/online/exptab
#
# setup the MIDAS mserver
#
case `hostname` in
daq07*)
unset MIDAS_SERVER_HOST
;;
*)
export MIDAS_SERVER_HOST=daq07.triumf.ca:7070
;;
esac
#
# select 64-bit or 32-bit MIDAS and ROOT
#
case `uname -i` in
i386)
source /daq/daqshare/olchansk/root/root_v5.34.01_SL62_32/bin/thisroot.sh
export PATH=.:$MIDASSYS/linux-m32/bin:$PATH
;;
*)
#source /daq/daqshare/olchansk/root/root_v5.34.34_SL67_64/bin/thisroot.sh
source $HOME/packages/root/bin/thisroot.sh
export PATH=.:$MIDASSYS/linux/bin:$PATH
;;
esac
#
export PATH=.:$HOME/online/bin:$HOME/packages/roody/bin:$PATH
#
#end
</pre>

* mkdir $HOME/packages
* Logout and login again, for .cshrc changes to take effect

== Install ROOT ==

* Identify the Linux version: RH9 (Red Hat Linux 9), FC3 (Fedora Core 3), RHEL4/SL4 (Red Hat Enterprise LInux 4/Scientific Linux 4), SL5x, SL6x, (CentOS/CC/SL) el7x: more /etc/redhat-release
* Decide to use 32-bit or 64-bit ROOT ('uname -a')
* cd $HOME/packages
* ls -l /daq/daqshare/olchansk/root/ ### to see all available ROOT packages
* ln -s /daq/daqshare/olchansk/root/root_vNNN_VVV_BB root, where NNN is the latest available version of ROOT ("ls -l /daq/daqshare/olchansk/root"), VVV is the Linux version code (RH9, FC3, SL4, etc) and BB is "_32" or "_64" for 32-bit or 64-bit ROOT. For example: /daq/daqshare/olchansk/root/root_v5.10.00_SL40
* for example: ln -s /daq/daqshare/olchansk/root/root_v5.34.34_el72_64 $HOME/packages/root
* Check that ROOT works: "source $HOME/packages/root/bin/thisroot.sh; root". Type ".q" to exit root.

== Install MIDAS ==

* cd $HOME/packages
* (OBSOLETE) svn co svn+ssh://svn@savannah.psi.ch/repos/meg/midas/trunk midas, password "svn". (password has to be entered twice)
* (OBSOLETE) svn co svn+ssh://svn@savannah.psi.ch/repos/meg/mxml/trunk mxml
* git clone https://bitbucket.org/tmidas/midas
* git clone https://bitbucket.org/tmidas/mxml
* git clone https://bitbucket.org/tmidas/mscb
* (IF BITBUCKET IS DOWN) git clone -v --progress https://daq.triumf.ca/~daqweb/git/mxml.git
* (IF BITBUCKET IS DOWN) git clone -v --progress https://daq.triumf.ca/~daqweb/git/mscb.git
* (IF BITBUCKET IS DOWN) git clone -v --progress https://daq.triumf.ca/~daqweb/git/midas.git
* cd midas
* make
* (only if needed) make linux32 ### build the 32-bit MIDAS libraries
* ls -l linux/bin/odbedit ### check that odbedit has been created (do not run it yet)

You can see a list of other installation problems at [[Common problems & Debugging recipes]].

;NOTE
: Optional features in MIDAS can be explicitly disabled if desired when making MIDAS using the NO_xxx feature (NO_ROOT,NO_MYSQL,NO_ODBC,NO_SQLITE,NO_MSCB), e.g. "make NO_ROOT=1" to disable ROOT. These NO_xxx Makefile variables are only used to control autodetection.

== Install ROOTANA ==

* cd $HOME/packages
* git clone https://bitbucket.org/tmidas/rootana
* cd rootana
* make

== Install ROODY ==

* cd $HOME/packages
* git clone https://bitbucket.org/tmidas/roody
* cd roody
* make
* $HOME/packages/roody/bin/roody, run the program

== Build special versions of MIDAS ==

Build special versions of MIDAS for the case when some MIDAS programs, such as VME frontends, will run on a different computer that may have a different flavour of operating system, i.e. 32-bit linux or an older version of Scientific Linux.

* login to the computer where the frontends will run and:
* if it is a 32-bit linux: cd $HOME/packages/midas; make linux32
* if it is a 64-bit linux: cd $HOME/packages/midas; make linux64

* login to the host computer to cross-compile ARM code:
* if it is an ARM linux: cd $HOME/packages/midas; make linuxarm # may need to install ARM cross compilers

== Prepare VME hardware ==

Hardware check list:
* VME crate
* VME processor (supported are V77xx, V7805, V7865)
* On all VME modules, set the VME address jumpers as described here: http://daq-plone.triumf.ca/SM/docs/local/vme_jumpers
* run vmescan to confirm correct VME addresses
** cd $HOME/packages
** svn checkout https://ladd00.triumf.ca/svn/daqsvn/trunk/vme
** cd vme
** make
** ./vmescan.exe (or _gef.exe, depending on the VME driver in use)

== Install Universe-II VME driver (V7648, V7750, V7805, V7851) ==

* login as root (ssh root@localhost)
* follow instructions: https://www.triumf.info/wiki/DAQwiki/index.php/VME-CPU#V7648.2C_V7750.2C_V7805.2C_V7851_:_Setup_vme_universe_VME_drivers
* cd ~/packages/vme; vmescan.exe

== Install Tsi-148 VME driver (V7865) ==

* login as root (ssh root@localhost)
* follow instructions: https://www.triumf.info/wiki/DAQwiki/index.php/VME-CPU#V7865_and_XVB-602_:_Setup_gefvme.2Ftsi148_VME_drivers
* cd ~/packages/vme; vmescan_gef.exe

== Setup the experiment environment ==

* Decide which computer will host MIDAS (where MIDAS shared memory buffers will reside).
: This computer will run the [[mserver]], [[mlogger]] and [[mhttpd]] applications. (It is usually the machine where the MIDAS,ROOT etc. packages have been downloaded). It will be referred to as the host machine (localhost).

The environment is slightly different depending on whether all programs run on the host machine, or whether some programs run on remote host(s) :
=== ALL programs run on localhost ===
:If all programs run on the host machine (localhost), it is not necessary to run [[mserver]]. [[Environment Variables#MIDAS_SERVER_HOST|MIDAS_SERVER_HOST]] will not be assigned (see example .cshrc [[#Prepare the user account|above]]).

=== Some programs run on REMOTE host(s) ===
: '''IMPORTANT:'''
# Since August 2015 '''you must explicitly allow access for clients running on remote machines'''. To do this, follow the '''[[Security#MIDAS programs on remote machines|instructions here]]'''.
# The example code .cshrc ([[#Prepare the user account|see above]]) should be present on both host and remote machine(s). This will ensure that [[Environment Variables#MIDAS_SERVER_HOST|MIDAS_SERVER_HOST]] will NOT be set for the host machine (localhost), but on a remote machine, MIDAS_SERVER_HOST will be set to the MIDAS host machine.
# The client [[mserver]] must be started on the MIDAS host machine. Note that multiple experiments can run on the same host machine by starting several instances of [[mserver]] (one for each experiment) running with different ports (and .cshrc would be edited so that MIDAS_SERVER_HOST is set to the appropriate port for the experiment).

 
On the host machine:
* mkdir $HOME/online
* cd $HOME/online
* create directories for local programs, sources, elog and history: mkdir bin src elog history
* create data directory: mkdir -p /ladd/data1/t2kvme5/data; ln -s /ladd/data1/t2kvme5/data $HOME/online
* create the exptab file "$HOME/online/exptab" following the example below. The first entry (exptname) is the name if the DAQ system (MIDAS experiment name), the second entry (/home/USER/online) is the location of MIDAS shared memory buffers (by convention, $HOME/online), the third entry (kopio03) is your username.
<pre>exptname /home/kopio03/online kopio03</pre>
* logout and login again for all changes to take effect

== Setup experiment startup scripts ==
* login to the experiment host computer
* echo $MIDAS_SERVER_HOST ### to check correct value - should be blank
* create $HOME/online/bin/start_daq.sh, replacing XXX with the hostname of the machine running the experiment (and changing the mserver and mhttpd ports, as needed).

#!/bin/sh
# start_daq.sh
cd $HOME/online
#
case `hostname` in XXX*)
echo "Good, we are on XXX!"
;;
*)
echo "The start_daq script should be executed on XXX"
exit 1
;;
esac
#
odbedit -c clean
# start [[mhttpd]] on default port. (Mongoose https version - see [[mhttpd]] for other options)
mhttpd -D -a localhost -a XXX.triumf.ca # optionally restrict access to specified hosts
#
# start [[mserver]] on default port (use argument -p to use a different port)
mserver -D # access must now be specifically allowed - see [[#Setup the experiment environment|above]]

# OR ([[#NOTES|older MIDAS versions]])
# mhttpd -p 8081 -D -a localhost -a XXX.triumf.ca # optionally restrict access to specified hosts
# mserver -p 7071 -D -a localhost -a lxdragon01.triumf.ca -a lxdragon02.triumf.ca -a XXX.triumf.ca # optionally restrict access to specified hosts

#
mlogger -D
#end file

== Run the MIDAS Web Server ==
Let's start the MIDAS webserver for the first time:

Start [[mhttpd]] on the ''experiment host'' (localhost) like this:
[mhostpc] mhttpd
You will get the following messages:
[mhttpd,INFO] ODB subtree /Runinfo corrected successfully
Mongoose web server will listen on ports "8080r,8443s"
[mhttpd,ERROR] [mhttpd.cxx:17892:mongoose,ERROR] cannot find SSL certificate file "/home/agdaq/online/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17893:mongoose,ERROR] please create SSL certificate file: openssl req -new -nodes -newkey rsa:2048 -sha256 -out ssl_cert.csr -keyout ssl_cert.key; openssl x509 -req -days 365 -sha256 -in ssl_cert.csr -signkey ssl_cert.key -out ssl_cert.pem; cat ssl_cert.key >> ssl_cert.pem
could not start the mongoose web server, see messages and midas.log, bye!

Create a self-signed certificate suitable for initial testing by executing the command printed by mhttpd:


[mhostpc] openssl req -new -nodes -newkey rsa:2048 -sha256 -out ssl_cert.csr -keyout ssl_cert.key; openssl x509 -req -days 365 -sha256 -in ssl_cert.csr -signkey ssl_cert.key -out ssl_cert.pem; cat ssl_cert.key >> ssl_cert.pem

For production use, you should create a properly signed certificate, see [[Mhttpd#Create an SSL certificate|create your own SSL certificate]] or you should run mhttpd behind an SSL proxy.

Run mhttpd again.

You will get the following messages:
[mhttpd,INFO] ODB subtree /Runinfo corrected successfully
Mongoose web server will listen on ports "8080r,8443s"
Mongoose web server will use SSL certificate file "/home/johnfoo/packages/midas/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17633:mongoose,ERROR] mongoose web server cannot find password file "/home/johnfoo/online/htpasswd.txt"
[mhttpd,ERROR] [mhttpd.cxx:17634:mongoose,ERROR] please create password file: htdigest -c /home/johnfoo/online/htpasswd.txt Default midas
could not start the mongoose web server, see messages and midas.log, bye!

Create the password file by following the instructions printed by mhttpd. The http digest domain name is the experiment name, suggested default user name is "midas". You will be asked to type in a password


[mhostpc] htdigest -c /home/johnfoo/online/htpasswd.txt exptname midas
Adding password for midas in realm exptname.
New password:
Re-type new password:

It is a good idea to set the password file {{Filepath|path=htpasswd.txt}} readable and writable by owner only.

Now restart {{Utility|name=mhttpd}}
[mhostpc] mhttpd
Mongoose web server will listen on ports "8080r,8443s" **see note
Mongoose web server will use SSL certificate file "/home/suz/packages/midas/ssl_cert.pem"
Mongoose web server will use authentication realm "Default", password file "./htpasswd.txt"

Now point a web browser running on the same host computer (localhost) to https://localhost:8443
If the web browser is running on a different computer, go to URL of the form

https://mhostpc.triumf.ca:8443 (substitute your host machine name and domain for "mhostpc.triumf.ca")

If you are using the default SSL certificate you will probably get a message: "This Connection is Untrusted". Click "I understand the risks" and add an exception. This is because the test certificate is self-signed. Then confirm an exception.

If instead you get a "connection refused" error, the midas host pc may have the firewall enabled. To make a firewall exception for MIDAS, follow instructions here http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29

You should then see an authentication box asking you for the user name and password. The user name is "midas". Enter the password you just created. The Midas [[Status Page]] should appear with multiple buttons for run control as well as equipment listing (no equipments will be listed as yet) and application listings. Please refer to [[mhttpd]] (the MIDAS Web-based Run Control utility) for further information. You can start and stop runs from the main status page, and use the [[ODB Page]] to access the database (ODB).

; Note
: Default ports of 8080 and 8443 are used by [[mhttpd]]. If these ports are in use on your machine, start mhttpd with alternative ports, e.g.
[mhostpc] mhttpd --https 8448 --http 8089
: or see [[Mhttpd#Usage]] to change the default ports.

== Setup experiment database (ODB) ==

* run $HOME/online/bin/start_daq.sh

* odbedit, run these commands: (replace user names and directory names)
<pre>
set "/Logger/Message file" "/home/kopio03/online/midas.log"
set "/Logger/Data Dir" "/home/kopio03/online/data"
create STRING "/Logger/History dir"
set "/Logger/History dir" "/home/kopio03/online/history"
create STRING "/Logger/Elog dir"
set "/Logger/Elog dir" "/home/kopio03/online/elog"
set "/Logger/ODB dump file" "/home/kopio03/online/history/run%05d.xml"
set "/Logger/ODB dump" "y"
set "/Logger/Channels/0/Settings/Filename" "run%05dsub%03d.mid.gz"
set "/Logger/Channels/0/Settings/Subrun byte limit" "1000000000"
set "/Logger/Channels/0/Settings/Compression" 1
set "/Logger/Channels/0/Settings/ODB Dump" "y"
set "/Programs/Logger/Required" y
set "/Programs/Logger/Start command" "mlogger -D"
set "/Programs/fevme/Required" "y"
set "/Programs/fevme/Start command" "ssh -n lxdaq09 $HOME/online/src/fevme_gef.exe -O"
exit
</pre>
* open web browser e.g. firefox.
* go to the midas status page at https://localhost:8443 (default port).
** if running [[mhttpd]] with Mongoose HTTPS/OpenSSL (the default) for the first time, you will need to create a password file. Follow the instructions (see [[mhttpd#HTTPS/SSL server (Mongoose)]] for details).
** For other options (i.e. HTTPS/SSL proxy) see [[#Secure MIDAS and ELOG Web access]]
*OR open the midas status page at http://localhost:8081 ([[#NOTES|older MIDAS versions]])
* midas status page will show most stuff "red" as nothing is running yet
* DON'T DO THIS YET run ./fevme.exe (on the computer with the VME interface, could be different from computer hosting the experiment), observe that corresponding equipments have been created
* save the url bookmark to the "personal toolbar"
* go to the Programs page, stop mlogger, stop fevme, start mlogger, start fevme
* go to the Status page, start run, stop run
* go back to the Status page, everything should be green
* start a run
* send signals to the ADC gate
* you should be getting events
* to look at data, proceed with setting up the [[ROOTANA|ROOT Analyzer]].

== Start DAQ programs at boot time ==

* add this to /etc/rc.local (replace username and location of the start_daq script)
<pre>
su - alpha -c /home/alpha/online/bin/start_daq.sh
</pre>

== Setup local software version control ==

Version control for experiment source code is setup using "git" (http://git-scm.com/)

* cd $HOME/online
* git init
* git add exptab
* git add bin/start_daq.sh
* git add .gitignore ### contents can be
<pre>
*~
*.o
*.exe
</pre>
* git add src/Makefile src/*.cxx ...
* git commit -a

== Adjust MIDAS buffer sizes ==

Default MIDAS SYSTEM buffer size is 8 Mbytes, fairly small for high-data-rate experiments. The rule of thumb is to have at least a few seconds worth of buffer space available. For example, if event size is 10 Kbytes and the event rate is 1 kHz, data rate is 10*10^3*1*10^3 = 10 Mbytes/sec. To buffer 10 seconds of data we need 100 Mbytes of buffer space.

To resize the MIDAS event buffers (SYSTEM, etc) do this:
* stop all frontends, stop mlogger
* start odbedit:
** cd "/Experiment/Buffer sizes"
** set SYSTEM 100000000
* run "mdump -z SYSTEM"
* if mdump complains about the size of .SYSTEM.SHM, remove it, try again.
* ls -l /dev/shm ### to observe that the size of shared memory is correct

== Secure MIDAS and ELOG Web access ==
In versions prior to May 2015, the default web access to MIDAS and ELOG uses the "http:" protocol which is insecure. In this case, all information is transmitted as clear text meaning that secret, confidential and sensitive information (such as the MIDAS and ELOG passwords and usernames) can be stolen "easily". This means that even "password protected" MIDAS and ELOG pages are not really protected if accessed using the "http" method.

Better security for HTTP is gained by using a password protected '''SSL (https) proxy'''. (It does not provide absolute security because of remaining problems with the security of SSL certificates, security of passwords, etc). Setting up an SSL (https) proxy is described [[#Setting up an HTTP proxy|below]].

Since May 2015, an ''alternative secure option'' to setting up an HTTP proxy is available to users of MIDAS. Recent versions of elogd (ELOG) do support SSL https:// connections, and [[#mhttpd with HTTPS/SSL server (Mongoose)]] is now available. This option is the default, and provides a similar level of security to an HTTP proxy.

See [[Security#Web Access]] for a comparison of these two secure options.

=== mhttpd with HTTPS/SSL server (Mongoose) ===

Since May 2015 the MIDAS web server [[mhttpd]] is explicitly linked with OpenSSL to provide secure HTTPS connections via the [https://bitbucket.org/tmidas/midas/src/ecb9a8537448a8a43f7f9a2bfdb82e578208cde3/doc/mongoose/?at=develop Mongoose] web server (see [[mhttpd]]). With this version, default web access to MIDAS uses the "https" protocol. Web access to {{Utility|name=mhttpd}} can be restricted by using the "-a hostname" switch of [[mhttpd]]. The first time {{Utility|name=mhttpd}} is run, a password file must be created. An SSL certificate is also required. See [[mhttpd#HTTPS/SSL server (Mongoose)|HTTPS/SSL server (Mongoose)]] for instructions.

=== mhttpd using an HTTPS/SSL proxy ===

THESE INSTRUCTIONS ARE WRONG, DO NOT DO THIS.

An [[#Setting up an HTTP proxy|HTTP proxy]] must be set up. This is the only way of securing older version of [[mhttpd]] (pre August 2015). Older versions of mhttpd are started using the -p port option e.g.
* mhttpd -D -p 8080 

To run a new version of mhttpd using an HTTP proxy, use the options provided to run the old (non-Mongoose) webserver on a given port, i.e.
* mhttpd --oldserver 8080 --nomg -D 

When using an SSL proxy, only access from the SSL proxy (and maybe some special trusted machines) should be permitted.
This is done using the "-a hostname" switch of [[mhttpd]]. Normally there will be only "-a localhost" switch, enabling access only for the local machine (where the SSL proxy is running). Additional "-a hostname" switches enable access from listed local machines. No "-a xxx" enables access from everywhere (defeating the purpose of the SSL proxy, unless access controls are enforced elsewhere, i.e. by a site firewall or by local firewall rules).

== Setting up an HTTP proxy ==
In this example, we use APACHE HTTPD to password-protect a typical midas/mhttpd and elog installation.

In this configuration, one uses the Linux stock httpd that accepts encrypted https:// connections and forwards them to mhttpd and elogd. Instead of (or in addition to) using mhttpd and elogd passwords, one configures password protection in httpd via the regular apache httpd password mechanisms (htpasswd, etc).

Recent versions of elogd do support SSL https:// connections, but if one is running an SSL proxy for anyway, it is simpler to run both through the same SSL proxy using the same SSL host certificate and the same httpd password file.

=== Restricting http: access to elogd ===
;Note
:Recent versions of elogd do support SSL https:// connections. The following information is for those using an HTTP proxy (see above).

For elogd, this is done using the "-n localhost" switch with enables only access from the same machine if present, or access from anywhere is absent (defeating the purpose of the SSL proxy, unless access controls are enforced elsewhere).

(It is recommended to run elogd from the same user as the main daq user and to keep elogd.cfg and all logbooks in the home directory of this user, where they are captured by the normal site backup system)

== Install standalone elog ==

* login into the user account that will run the elog
* cd $HOME/packages
* git clone https://bitbucket.org/ritt/elog
* cd elog
* make
* create new file start_elogd with this contents:
<pre>
#!/bin/sh

killall elogd
sleep 1
killall -KILL elogd
sleep 1
$HOME/packages/elog/elogd -n localhost -x -c $HOME/packages/elog/elogd.cfg -p 8082 -D

#end
</pre>
* chmod a+x start_elogd
* edit elogd.cfg to read:
<pre>
[global]
port = 8082
SMTP host = smtp.triumf.ca
URL = https://titan00.triumf.ca/elog/

Reverse sort = 1
Display Mode = full

#List Menu commands = New, Find, Admin, Help
#Menu commands = New, Edit, Reply, Find, Duplicate, Help

Entries Per Page = 30
Supress Email on edit = 1
Default encoding = 1
Page title = TITAN ELOG
Resolve host names = 1

Logfile = /home/titan/packages/elog/elogd.log
#Logging level = 3

[midas]

List page Title = T2K M11 MIDAS ELOG
Comment = T2K M11 MIDAS ELOG
Page Title = T2K M11 MIDAS ELOG
RSS Title = [$logbook - $type - $system] $subject, posted by $author

Attributes = Author, Subject, Run, Type, System
Show Attributes Edit = Run, Author, Subject, Type, System
Required Attributes = Author, Type, System, Subject

Options Type = Routine, Reply, Shift Summary, Modification, Question, Info, Problem
Options System = General, DAQ, Beamline

Preset Run = $shell(MIDASSYS=. /home/t2km11/packages/midas/linux/bin/odbedit -d Runinfo -c 'ls -v \"run number\"')

Preset On Reply Type = Reply
Preset On Reply Run = $shell(MIDASSYS=. /home/t2km11/packages/midas/linux/bin/odbedit -d Runinfo -c 'ls -v \"run number\"')

List Display = Date, Subject, Type, System, Author, ID
Quick Filter = Date, Type, ID

Remove on reply = Author
Quote on reply = 1

Use lock = 1

************* Email Functionality ****************

Use Email Subject = [T2KM11 - $System] $Subject
Omit Email To = 1

Email System General = xxx
</pre>
* ./start_elogd &
* firefox http://localhost:8082 # hould show the elog message index

To start elogd automatically when the machine is rebooted, login as root and
* add this text to /etc/rc.local:
<pre>
su - titan -c "/home/titan/packages/elog/start_elogd"
</pre>
* chmod a+x /etc/rc.local
* systemctl start rc-local

To import elog entries from the mhttpd elog, do this:

* cd ~/packages/elog/logbooks
* ln -s /home/t2km11/online/elog midas
* cd midas
* ~/packages/elog/elconv)

== Install https proxy ==

THESE INSTRUCTIONS ARE OBSOLETE, INSTEAD,
* GO HERE: https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Configure_HTTPS_server_.28CentOS7.29
* AND GO HERE: https://midas.triumf.ca/MidasWiki/index.php/Quickstart_Linux#Run_the_MIDAS_Web_Server

FOLLOWING INSTRUCTIONS ARE OBSOLETE...

* login as root to the https proxy machine
* cd ~root
* yum install mod_ssl
* yum install crypto-utils # see http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_monitoring_of_HTTPS_certificates
* create a certificate request (replace ladd09 with your hostname): openssl req -new -nodes -newkey rsa:2048 -sha256 -out ladd09.csr -keyout ladd09.key (answer: CA, BC, Vancouver, TRIUMF, DAQ, ladd09.triumf.ca, email@email.com
* sign it by TRIUMF:
** mail -s "Certificate request" yourself@email.com < ladd09.csr
** forward this request to Andrew Daviel
** he will email the signed crt file, copy it to this system as ladd09.crt
* sign it yourself: openssl x509 -req -days 365 -sha256 -in ladd09.csr -signkey ladd09.key -out ladd09.crt
* (if the certificate expires, renew it by signing it again)
* Additional commands for working with certificates:
** explore the private key: openssl pkey -in ladd09.key -text -noout
** explore the certificate request: openssl req -in ladd00.csr -text -noout
** explore the certificate: openssl x509 -in ladd09.crt -noout -text
* move certificate files to proper system locations:
* mv ladd09.key /etc/pki/tls/private/
* mv ladd09.crt /etc/pki/tls/certs/
* if selinux is enabled, do this:
** restorecon -Rv /etc/pki/tls/certs/
** restorecon -Rv /etc/pki/tls/private/
** /usr/sbin/setsebool -P httpd_can_network_connect 1
* open /etc/httpd/conf.d/ssl.conf in a text editor, go to the very bottom and right before the "</VirtualHost>" entry, add following text:
<pre>
...
SSLCertificateFile /etc/pki/tls/certs/ladd09.crt
SSLCertificateKeyFile /etc/pki/tls/private/ladd09.key

ProxyPass /elog/ http://localhost:8082/ retry=1
ProxyPass / http://localhost:8080/ retry=1

<Location />

SSLRequireSSL
AuthType Basic
AuthName "password protected site"
Require valid-user

# create password file: touch /etc/httpd/htpasswd
# to add new user or change password: htpasswd /etc/httpd/htpasswd username
AuthUserFile /etc/httpd/htpasswd

</Location>
</VirtualHost>
...
</pre>
* comment out duplicate "SSLCertificateFile" and "SSLCertificateKeyFile" elsewhere in the file
* (optionally) If you got a certificate that is signed by DigiCert or RapidSSL then you'll need to add a line specifying the certificate chain file:
<pre>
...
SSLCertificateChainFile /etc/pki/tls/certs/DigiCertCA.crt
...
</pre>
* touch /etc/httpd/htpasswd
* htpasswd /etc/httpd/htpasswd midas # enter password midas
* chkconfig httpd on
* service httpd restart
* firewall-cmd --add-port=443/tcp --permanent
* firewall-cmd --reload
* firewall-cmd --list-all
* test it
** test the SSL proxy: https://host/ should yield the midas status page, https://host/elog/ should yield the elog message index
* in ODB, set "/Elog/URL" to "https://host/elog/"
* now from the midas status page, the "Elog" button should take us to the https Elog URL

In needed, enable user directories: https://blah/~user in ~user/public_html

* edit /etc/httpd/conf.d/userdir.conf, replace "UserDir disabled" with "UserDir enabled"
* setsebool -P httpd_enable_homedirs true
* systemctl restart httpd

== Setup the history mhttpd for faster access to history plots ==
When running an SSL proxy,
* start the main mhttpd (orange command for old mhttpd, green for new mhttpd with Mongoose(post August2015):
** "mhttpd -p 8071 -D" or
** "mhttpd -D --oldserver 8071 --nomg" 

* start the history mhttpd
** "mhttpd -p 8072 -D -H" or
** "mhttpd -D -H --oldserver 8072 --nomg" 
* set ODB /History/URL to "http://alphacpc09.cern.ch:8072/HS/"
* open the MIDAS status page
* go to the history section, try to open any history plot, observe that the history plot gif image loads correctly, inspect it's URL (use "copy image URL" or "view source", etc), it should point to port 8072 causing connection to the history mhttpd.
* continue with these instructions to setup history mhttpd access through an SSL proxy:
* setup SSL proxy access (required mod_proxy_html)
** login as root to the SSL proxy machine
** on SL5, install the missing mod_proxy_html httpd module:
** yum install httpd-devel libxml2-devel
** wget http://apache.webthing.com/mod_proxy_html/mod_proxy_html.tar.bz2
** tar xjvf mod_proxy_html.tar.bz2
** cd mod_proxy_html
** apxs -c -I. -I/usr/include/libxml2 -i mod_proxy_html.c
** apxs -c -I. -I/usr/include/libxml2 -i mod_xml2enc.c
** cd /etc/httpd/conf.d, add this to ssl.conf:

;before the ProxyPass statements:
<pre>
# proxy the MIDAS web servers
LoadModule xml2enc_module modules/mod_xml2enc.so
LoadModule proxy_html_module modules/mod_proxy_html.so
ProxyHTMLLinks a href
ProxyHTMLLinks link href
ProxyHTMLLinks img src
#ProxyHTMLEnable On
ProxyRequests off
</pre>
;after the ProxyPass statements:
<pre>
# ALPHA1 history access
ProxyPass /alpha1/history/ http://alphacpc09.cern.ch:8072/HS/ retry=1
ProxyPass /alpha1/ http://alphacpc09.cern.ch:8071/ retry=1

ProxyHTMLEnable On
ProxyHTMLURLMap http://alphacpc09.cern.ch:8072/HS/ /alpha1/history/
</pre>
;adjust:
*"alpha1" is the experiment name
*"alphacpc09.cern.ch" is the machine running mhttpd
*"8071" is the port number of the main mhttpd
** "mhttpd -p 8071 -D" or
** "mhttpd -D --oldserver 8071 --nomg" 
* "8072" is the port number of the history mhttpd
** "mhttpd -p 8072 -D -H" or
** "mhttpd -D -H --oldserver 8072 --nomg" 

[[Category:Installation]] [[Category:Buffer]] [[Category:Driver]]

Setup MIDAS experiment at TRIUMF

2016-11-21T21:43:50Z

Olchansk: /* Prepare the user account */

{{Pagelinks}}
== Introduction ==
This page describes setting up a MIDAS experiment at TRIUMF. This information can be adapted for other sites.

== Environment variables ==
* '''MIDASSYS''' Base directory of the MIDAS package, midas and mxml should be at the same level.
* '''MIDAS_EXPTAB''' Experiment definition file
* '''MIDAS_SERVER_HOST''' MIDAS host server name for remote midas connections.
* '''MIDAS_EXPT_NAME''' Experiment name

== Standard layout of MIDAS experiment ==
The following shows the directory layout of a standard MIDAS experiment:

/home/exptuser/
packages/
root <---- ROOT
mxml
mscb
midas/ <---- MIDAS
linux/{lib,bin} <---- binaries matching the selected 64-bit/32-bit flavour of ROOT
linux-m32/{lib,bin} <---- limited function 32-bit binaries for 32-bit frontend machines, build by "make linux32"
linux-m64/{lib,bin} <---- limited function 64-bit binaries (only needed if ROOT and linux/bin are 32-bit)
linux-arm/{lib,bin} <---- full function ARM cross-compiled using "make linuxarm"
linux-crosscompile/{lib,bin} <---- cross-compiled limited function binaries for PPC and ARM frontends (see Makefile)
rootana <---- ROOT analyzer
roody <---- graphical online histogram viewer for MIDAS and ROOTANA
online/
exptab <---- experiment definition
{.ODB,.SYSTEM,.SYSMSG,etc}.SHM <---- MIDAS shared memory save files
src <---- experiment frontend sources
bin,scripts
elog <---- MIDAS elog
history <---- MIDAS history
data -> /data/exptname/current <---- symlink to the data directory
/data/exptname/current <---- experiment data directory with ODB save files and MIDAS .mid/.mid.gz data files

== Prepare computers ==

On some operating systems, several MIDAS functions require administrator access:

* on el7 linux (SL7/CC7/CentOS7/RHEL7) - access to mhttpd port 8443 requires special firewall rules, see here: https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29
* on el7 linux - access to mserver to run frontends and other programs on some other computer requires special firewall rules, see here: https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29
* on el7 linux - on the frontend machines (and other machines that will connect to the mserver, the same firewall rule needs to be created (use the IP address of the machine running the mserver)

== Prepare the user account ==
<div id="NOTES"></div>
;NOTES
* Instructions are given for recent MIDAS versions (August 2015 and later) which have enhanced [[Security]]. Instructions for older versions differ slightly.
** follow the orange instructions for older versions

* Setup the user account for running this instance of midas. For machines part of the LADD cluster, follow these [http://daq-plone.triumf.ca/SM/docs/local/NewLaddUser] instructions.
* check that the account is using the /bin/bash shell
* make $HOME/.profile look like this:

<pre>
#!/bin/echo You must source

export SVN_EDITOR="emacs -nw"
export GIT_EDITOR="emacs -nw"
export MIDASSYS=$HOME/packages/midas
export ROOTANASYS=$HOME/packages/rootana
export MIDAS_EXPTAB=$HOME/online/exptab
#
# setup the MIDAS mserver
#
case `hostname` in
daq07*)
unset MIDAS_SERVER_HOST
;;
*)
export MIDAS_SERVER_HOST=daq07.triumf.ca:7070
;;
esac
#
# select 64-bit or 32-bit MIDAS and ROOT
#
case `uname -i` in
i386)
source /daq/daqshare/olchansk/root/root_v5.34.01_SL62_32/bin/thisroot.sh
export PATH=.:$MIDASSYS/linux-m32/bin:$PATH
;;
*)
#source /daq/daqshare/olchansk/root/root_v5.34.34_SL67_64/bin/thisroot.sh
source $HOME/packages/root/bin/thisroot.sh
export PATH=.:$MIDASSYS/linux/bin:$PATH
;;
esac
#
export PATH=.:$HOME/online/bin:$HOME/packages/roody/bin:$PATH
#
#end
</pre>

* mkdir $HOME/packages
* Logout and login again, for .cshrc changes to take effect

== Install ROOT ==

* Identify the Linux version: RH9 (Red Hat Linux 9), FC3 (Fedora Core 3), RHEL4/SL4 (Red Hat Enterprise LInux 4/Scientific Linux 4), SL5x, SL6x, (CentOS/CC/SL) el7x: more /etc/redhat-release
* Decide to use 32-bit or 64-bit ROOT ('uname -a')
* cd $HOME/packages
* ls -l /daq/daqshare/olchansk/root/ ### to see all available ROOT packages
* ln -s /daq/daqshare/olchansk/root/root_vNNN_VVV_BB root, where NNN is the latest available version of ROOT ("ls -l /daq/daqshare/olchansk/root"), VVV is the Linux version code (RH9, FC3, SL4, etc) and BB is "_32" or "_64" for 32-bit or 64-bit ROOT. For example: /daq/daqshare/olchansk/root/root_v5.10.00_SL40
* for example: ln -s /daq/daqshare/olchansk/root/root_v5.34.34_el72_64 $HOME/packages/root
* Check that ROOT works: "source $HOME/packages/root/bin/thisroot.sh; root". Type ".q" to exit root.

== Install MIDAS ==

* cd $HOME/packages
* (OBSOLETE) svn co svn+ssh://svn@savannah.psi.ch/repos/meg/midas/trunk midas, password "svn". (password has to be entered twice)
* (OBSOLETE) svn co svn+ssh://svn@savannah.psi.ch/repos/meg/mxml/trunk mxml
* git clone https://bitbucket.org/tmidas/midas
* git clone https://bitbucket.org/tmidas/mxml
* git clone https://bitbucket.org/tmidas/mscb
* (IF BITBUCKET IS DOWN) git clone -v --progress https://daq.triumf.ca/~daqweb/git/mxml.git
* (IF BITBUCKET IS DOWN) git clone -v --progress https://daq.triumf.ca/~daqweb/git/mscb.git
* (IF BITBUCKET IS DOWN) git clone -v --progress https://daq.triumf.ca/~daqweb/git/midas.git
* cd midas
* make
* (only if needed) make linux32 ### build the 32-bit MIDAS libraries
* ls -l linux/bin/odbedit ### check that odbedit has been created (do not run it yet)

You can see a list of other installation problems at [[Common problems & Debugging recipes]].

;NOTE
: Optional features in MIDAS can be explicitly disabled if desired when making MIDAS using the NO_xxx feature (NO_ROOT,NO_MYSQL,NO_ODBC,NO_SQLITE,NO_MSCB), e.g. "make NO_ROOT=1" to disable ROOT. These NO_xxx Makefile variables are only used to control autodetection.

== Install ROOTANA ==

* cd $HOME/packages
* git clone https://bitbucket.org/tmidas/rootana
* cd rootana
* make

== Install ROODY ==

* cd $HOME/packages
* git clone https://bitbucket.org/tmidas/roody
* cd roody
* make
* $HOME/packages/roody/bin/roody, run the program

== Build special versions of MIDAS ==

Build special versions of MIDAS for the case when some MIDAS programs, such as VME frontends, will run on a different computer that may have a different flavour of operating system, i.e. 32-bit linux or an older version of Scientific Linux.

* login to the computer where the frontends will run and:
* if it is a 32-bit linux: cd $HOME/packages/midas; make linux32
* if it is a 64-bit linux: cd $HOME/packages/midas; make linux64

* login to the host computer to cross-compile ARM code:
* if it is an ARM linux: cd $HOME/packages/midas; make linuxarm # may need to install ARM cross compilers

== Prepare VME hardware ==

Hardware check list:
* VME crate
* VME processor (supported are V77xx, V7805, V7865)
* On all VME modules, set the VME address jumpers as described here: http://daq-plone.triumf.ca/SM/docs/local/vme_jumpers
* run vmescan to confirm correct VME addresses
** cd $HOME/packages
** svn checkout https://ladd00.triumf.ca/svn/daqsvn/trunk/vme
** cd vme
** make
** ./vmescan.exe (or _gef.exe, depending on the VME driver in use)

== Install Universe-II VME driver (V7648, V7750, V7805, V7851) ==

* login as root (ssh root@localhost)
* follow instructions: https://www.triumf.info/wiki/DAQwiki/index.php/VME-CPU#V7648.2C_V7750.2C_V7805.2C_V7851_:_Setup_vme_universe_VME_drivers
* cd ~/packages/vme; vmescan.exe

== Install Tsi-148 VME driver (V7865) ==

* login as root (ssh root@localhost)
* follow instructions: https://www.triumf.info/wiki/DAQwiki/index.php/VME-CPU#V7865_and_XVB-602_:_Setup_gefvme.2Ftsi148_VME_drivers
* cd ~/packages/vme; vmescan_gef.exe

== Setup the experiment environment ==

* Decide which computer will host MIDAS (where MIDAS shared memory buffers will reside).
: This computer will run the [[mserver]], [[mlogger]] and [[mhttpd]] applications. (It is usually the machine where the MIDAS,ROOT etc. packages have been downloaded). It will be referred to as the host machine (localhost).

The environment is slightly different depending on whether all programs run on the host machine, or whether some programs run on remote host(s) :
=== ALL programs run on localhost ===
:If all programs run on the host machine (localhost), it is not necessary to run [[mserver]]. [[Environment Variables#MIDAS_SERVER_HOST|MIDAS_SERVER_HOST]] will not be assigned (see example .cshrc [[#Prepare the user account|above]]).

=== Some programs run on REMOTE host(s) ===
: '''IMPORTANT:'''
# Since August 2015 '''you must explicitly allow access for clients running on remote machines'''. To do this, follow the '''[[Security#MIDAS programs on remote machines|instructions here]]'''.
# The example code .cshrc ([[#Prepare the user account|see above]]) should be present on both host and remote machine(s). This will ensure that [[Environment Variables#MIDAS_SERVER_HOST|MIDAS_SERVER_HOST]] will NOT be set for the host machine (localhost), but on a remote machine, MIDAS_SERVER_HOST will be set to the MIDAS host machine.
# The client [[mserver]] must be started on the MIDAS host machine. Note that multiple experiments can run on the same host machine by starting several instances of [[mserver]] (one for each experiment) running with different ports (and .cshrc would be edited so that MIDAS_SERVER_HOST is set to the appropriate port for the experiment).

 
On the host machine:
* mkdir $HOME/online
* cd $HOME/online
* create directories for local programs, sources, elog and history: mkdir bin src elog history
* create data directory: mkdir -p /ladd/data1/t2kvme5/data; ln -s /ladd/data1/t2kvme5/data $HOME/online
* create the exptab file "$HOME/online/exptab" following the example below. The first entry (exptname) is the name if the DAQ system (MIDAS experiment name), the second entry (/home/USER/online) is the location of MIDAS shared memory buffers (by convention, $HOME/online), the third entry (kopio03) is your username.
<pre>exptname /home/kopio03/online kopio03</pre>
* logout and login again for all changes to take effect

== Setup experiment startup scripts ==
* login to the experiment host computer
* echo $MIDAS_SERVER_HOST ### to check correct value - should be blank
* create $HOME/online/bin/start_daq.sh, replacing XXX with the hostname of the machine running the experiment (and changing the mserver and mhttpd ports, as needed).

#!/bin/sh
# start_daq.sh
cd $HOME/online
#
case `hostname` in XXX*)
echo "Good, we are on XXX!"
;;
*)
echo "The start_daq script should be executed on XXX"
exit 1
;;
esac
#
odbedit -c clean
# start [[mhttpd]] on default port. (Mongoose https version - see [[mhttpd]] for other options)
mhttpd -D -a localhost -a XXX.triumf.ca # optionally restrict access to specified hosts
#
# start [[mserver]] on default port (use argument -p to use a different port)
mserver -D # access must now be specifically allowed - see [[#Setup the experiment environment|above]]

# OR ([[#NOTES|older MIDAS versions]])
# mhttpd -p 8081 -D -a localhost -a XXX.triumf.ca # optionally restrict access to specified hosts
# mserver -p 7071 -D -a localhost -a lxdragon01.triumf.ca -a lxdragon02.triumf.ca -a XXX.triumf.ca # optionally restrict access to specified hosts

#
mlogger -D
#end file

== Run the MIDAS Web Server ==
Let's start the MIDAS webserver for the first time:

Start [[mhttpd]] on the ''experiment host'' (localhost) like this:
[mhostpc] mhttpd
You will get the following messages:
[mhttpd,INFO] ODB subtree /Runinfo corrected successfully
Mongoose web server will listen on ports "8080r,8443s"
[mhttpd,ERROR] [mhttpd.cxx:17892:mongoose,ERROR] cannot find SSL certificate file "/home/agdaq/online/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17893:mongoose,ERROR] please create SSL certificate file: openssl req -new -nodes -newkey rsa:2048 -sha256 -out ssl_cert.csr -keyout ssl_cert.key; openssl x509 -req -days 365 -sha256 -in ssl_cert.csr -signkey ssl_cert.key -out ssl_cert.pem; cat ssl_cert.key >> ssl_cert.pem
could not start the mongoose web server, see messages and midas.log, bye!

Create a self-signed certificate suitable for initial testing by executing the command printed by mhttpd:


[mhostpc] openssl req -new -nodes -newkey rsa:2048 -sha256 -out ssl_cert.csr -keyout ssl_cert.key; openssl x509 -req -days 365 -sha256 -in ssl_cert.csr -signkey ssl_cert.key -out ssl_cert.pem; cat ssl_cert.key >> ssl_cert.pem

For production use, you should create a properly signed certificate, see [[Mhttpd#Create an SSL certificate|create your own SSL certificate]] or you should run mhttpd behind an SSL proxy.

Run mhttpd again.

You will get the following messages:
[mhttpd,INFO] ODB subtree /Runinfo corrected successfully
Mongoose web server will listen on ports "8080r,8443s"
Mongoose web server will use SSL certificate file "/home/johnfoo/packages/midas/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17633:mongoose,ERROR] mongoose web server cannot find password file "/home/johnfoo/online/htpasswd.txt"
[mhttpd,ERROR] [mhttpd.cxx:17634:mongoose,ERROR] please create password file: htdigest -c /home/johnfoo/online/htpasswd.txt Default midas
could not start the mongoose web server, see messages and midas.log, bye!

Create the password file by following the instructions printed by mhttpd. The http digest domain name is the experiment name, suggested default user name is "midas". You will be asked to type in a password


[mhostpc] htdigest -c /home/johnfoo/online/htpasswd.txt exptname midas
Adding password for midas in realm exptname.
New password:
Re-type new password:

It is a good idea to set the password file {{Filepath|path=htpasswd.txt}} readable and writable by owner only.

Now restart {{Utility|name=mhttpd}}
[mhostpc] mhttpd
Mongoose web server will listen on ports "8080r,8443s" **see note
Mongoose web server will use SSL certificate file "/home/suz/packages/midas/ssl_cert.pem"
Mongoose web server will use authentication realm "Default", password file "./htpasswd.txt"

Now point a web browser running on the same host computer (localhost) to https://localhost:8443
If the web browser is running on a different computer, go to URL of the form

https://mhostpc.triumf.ca:8443 (substitute your host machine name and domain for "mhostpc.triumf.ca")

If you are using the default SSL certificate you will probably get a message: "This Connection is Untrusted". Click "I understand the risks" and add an exception. This is because the test certificate is self-signed. Then confirm an exception.

If instead you get a "connection refused" error, the midas host pc may have the firewall enabled. To make a firewall exception for MIDAS, follow instructions here http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29

You should then see an authentication box asking you for the user name and password. The user name is "midas". Enter the password you just created. The Midas [[Status Page]] should appear with multiple buttons for run control as well as equipment listing (no equipments will be listed as yet) and application listings. Please refer to [[mhttpd]] (the MIDAS Web-based Run Control utility) for further information. You can start and stop runs from the main status page, and use the [[ODB Page]] to access the database (ODB).

; Note
: Default ports of 8080 and 8443 are used by [[mhttpd]]. If these ports are in use on your machine, start mhttpd with alternative ports, e.g.
[mhostpc] mhttpd --https 8448 --http 8089
: or see [[Mhttpd#Usage]] to change the default ports.

== Setup experiment database (ODB) ==

* run $HOME/online/bin/start_daq.sh

* odbedit, run these commands: (replace user names and directory names)
<pre>
set "/Logger/Message file" "/home/kopio03/online/midas.log"
set "/Logger/Data Dir" "/home/kopio03/online/data"
create STRING "/Logger/History dir"
set "/Logger/History dir" "/home/kopio03/online/history"
create STRING "/Logger/Elog dir"
set "/Logger/Elog dir" "/home/kopio03/online/elog"
set "/Logger/ODB dump file" "/home/kopio03/online/history/run%05d.xml"
set "/Logger/ODB dump" "y"
set "/Logger/Channels/0/Settings/Filename" "run%05dsub%03d.mid.gz"
set "/Logger/Channels/0/Settings/Subrun byte limit" "1000000000"
set "/Logger/Channels/0/Settings/Compression" 1
set "/Logger/Channels/0/Settings/ODB Dump" "y"
set "/Programs/Logger/Required" y
set "/Programs/Logger/Start command" "mlogger -D"
set "/Programs/fevme/Required" "y"
set "/Programs/fevme/Start command" "ssh -n lxdaq09 $HOME/online/src/fevme_gef.exe -O"
exit
</pre>
* open web browser e.g. firefox.
* go to the midas status page at https://localhost:8443 (default port).
** if running [[mhttpd]] with Mongoose HTTPS/OpenSSL (the default) for the first time, you will need to create a password file. Follow the instructions (see [[mhttpd#HTTPS/SSL server (Mongoose)]] for details).
** For other options (i.e. HTTPS/SSL proxy) see [[#Secure MIDAS and ELOG Web access]]
*OR open the midas status page at http://localhost:8081 ([[#NOTES|older MIDAS versions]])
* midas status page will show most stuff "red" as nothing is running yet
* DON'T DO THIS YET run ./fevme.exe (on the computer with the VME interface, could be different from computer hosting the experiment), observe that corresponding equipments have been created
* save the url bookmark to the "personal toolbar"
* go to the Programs page, stop mlogger, stop fevme, start mlogger, start fevme
* go to the Status page, start run, stop run
* go back to the Status page, everything should be green
* start a run
* send signals to the ADC gate
* you should be getting events
* to look at data, proceed with setting up the [[ROOTANA|ROOT Analyzer]].

== Start DAQ programs at boot time ==

* add this to /etc/rc.local (replace username and location of the start_daq script)
<pre>
su - alpha -c /home/alpha/online/bin/start_daq.sh
</pre>

== Setup local software version control ==

Version control for experiment source code is setup using "git" (http://git-scm.com/)

* cd $HOME/online
* git init
* git add exptab
* git add bin/start_daq.sh
* git add .gitignore ### contents can be
<pre>
*~
*.o
*.exe
</pre>
* git add src/Makefile src/*.cxx ...
* git commit -a

== Adjust MIDAS buffer sizes ==

Default MIDAS SYSTEM buffer size is 8 Mbytes, fairly small for high-data-rate experiments. The rule of thumb is to have at least a few seconds worth of buffer space available. For example, if event size is 10 Kbytes and the event rate is 1 kHz, data rate is 10*10^3*1*10^3 = 10 Mbytes/sec. To buffer 10 seconds of data we need 100 Mbytes of buffer space.

To resize the MIDAS event buffers (SYSTEM, etc) do this:
* stop all frontends, stop mlogger
* start odbedit:
** cd "/Experiment/Buffer sizes"
** set SYSTEM 100000000
* run "mdump -z SYSTEM"
* if mdump complains about the size of .SYSTEM.SHM, remove it, try again.
* ls -l /dev/shm ### to observe that the size of shared memory is correct

== Secure MIDAS and ELOG Web access ==
In versions prior to May 2015, the default web access to MIDAS and ELOG uses the "http:" protocol which is insecure. In this case, all information is transmitted as clear text meaning that secret, confidential and sensitive information (such as the MIDAS and ELOG passwords and usernames) can be stolen "easily". This means that even "password protected" MIDAS and ELOG pages are not really protected if accessed using the "http" method.

Better security for HTTP is gained by using a password protected '''SSL (https) proxy'''. (It does not provide absolute security because of remaining problems with the security of SSL certificates, security of passwords, etc). Setting up an SSL (https) proxy is described [[#Setting up an HTTP proxy|below]].

Since May 2015, an ''alternative secure option'' to setting up an HTTP proxy is available to users of MIDAS. Recent versions of elogd (ELOG) do support SSL https:// connections, and [[#mhttpd with HTTPS/SSL server (Mongoose)]] is now available. This option is the default, and provides a similar level of security to an HTTP proxy.

See [[Security#Web Access]] for a comparison of these two secure options.

=== mhttpd with HTTPS/SSL server (Mongoose) ===

Since May 2015 the MIDAS web server [[mhttpd]] is explicitly linked with OpenSSL to provide secure HTTPS connections via the [https://bitbucket.org/tmidas/midas/src/ecb9a8537448a8a43f7f9a2bfdb82e578208cde3/doc/mongoose/?at=develop Mongoose] web server (see [[mhttpd]]). With this version, default web access to MIDAS uses the "https" protocol. Web access to {{Utility|name=mhttpd}} can be restricted by using the "-a hostname" switch of [[mhttpd]]. The first time {{Utility|name=mhttpd}} is run, a password file must be created. An SSL certificate is also required. See [[mhttpd#HTTPS/SSL server (Mongoose)|HTTPS/SSL server (Mongoose)]] for instructions.

=== mhttpd using an HTTPS/SSL proxy ===

THESE INSTRUCTIONS ARE WRONG, DO NOT DO THIS.

An [[#Setting up an HTTP proxy|HTTP proxy]] must be set up. This is the only way of securing older version of [[mhttpd]] (pre August 2015). Older versions of mhttpd are started using the -p port option e.g.
* mhttpd -D -p 8080 

To run a new version of mhttpd using an HTTP proxy, use the options provided to run the old (non-Mongoose) webserver on a given port, i.e.
* mhttpd --oldserver 8080 --nomg -D 

When using an SSL proxy, only access from the SSL proxy (and maybe some special trusted machines) should be permitted.
This is done using the "-a hostname" switch of [[mhttpd]]. Normally there will be only "-a localhost" switch, enabling access only for the local machine (where the SSL proxy is running). Additional "-a hostname" switches enable access from listed local machines. No "-a xxx" enables access from everywhere (defeating the purpose of the SSL proxy, unless access controls are enforced elsewhere, i.e. by a site firewall or by local firewall rules).

== Setting up an HTTP proxy ==
In this example, we use APACHE HTTPD to password-protect a typical midas/mhttpd and elog installation.

In this configuration, one uses the Linux stock httpd that accepts encrypted https:// connections and forwards them to mhttpd and elogd. Instead of (or in addition to) using mhttpd and elogd passwords, one configures password protection in httpd via the regular apache httpd password mechanisms (htpasswd, etc).

Recent versions of elogd do support SSL https:// connections, but if one is running an SSL proxy for anyway, it is simpler to run both through the same SSL proxy using the same SSL host certificate and the same httpd password file.

=== Restricting http: access to elogd ===
;Note
:Recent versions of elogd do support SSL https:// connections. The following information is for those using an HTTP proxy (see above).

For elogd, this is done using the "-n localhost" switch with enables only access from the same machine if present, or access from anywhere is absent (defeating the purpose of the SSL proxy, unless access controls are enforced elsewhere).

(It is recommended to run elogd from the same user as the main daq user and to keep elogd.cfg and all logbooks in the home directory of this user, where they are captured by the normal site backup system)

== Install standalone elog ==

* login into the user account that will run the elog
* cd $HOME/packages
* git clone https://bitbucket.org/ritt/elog
* cd elog
* make
* create new file start_elogd with this contents:
<pre>
#!/bin/sh

killall elogd
sleep 1
killall -KILL elogd
sleep 1
$HOME/packages/elog/elogd -n localhost -x -c $HOME/packages/elog/elogd.cfg -p 8082 -D

#end
</pre>
* chmod a+x start_elogd
* edit elogd.cfg to read:
<pre>
[global]
port = 8082
SMTP host = smtp.triumf.ca
URL = https://titan00.triumf.ca/elog/

Reverse sort = 1
Display Mode = full

#List Menu commands = New, Find, Admin, Help
#Menu commands = New, Edit, Reply, Find, Duplicate, Help

Entries Per Page = 30
Supress Email on edit = 1
Default encoding = 1
Page title = TITAN ELOG
Resolve host names = 1

Logfile = /home/titan/packages/elog/elogd.log
#Logging level = 3

[midas]

List page Title = T2K M11 MIDAS ELOG
Comment = T2K M11 MIDAS ELOG
Page Title = T2K M11 MIDAS ELOG
RSS Title = [$logbook - $type - $system] $subject, posted by $author

Attributes = Author, Subject, Run, Type, System
Show Attributes Edit = Run, Author, Subject, Type, System
Required Attributes = Author, Type, System, Subject

Options Type = Routine, Reply, Shift Summary, Modification, Question, Info, Problem
Options System = General, DAQ, Beamline

Preset Run = $shell(MIDASSYS=. /home/t2km11/packages/midas/linux/bin/odbedit -d Runinfo -c 'ls -v \"run number\"')

Preset On Reply Type = Reply
Preset On Reply Run = $shell(MIDASSYS=. /home/t2km11/packages/midas/linux/bin/odbedit -d Runinfo -c 'ls -v \"run number\"')

List Display = Date, Subject, Type, System, Author, ID
Quick Filter = Date, Type, ID

Remove on reply = Author
Quote on reply = 1

Use lock = 1

************* Email Functionality ****************

Use Email Subject = [T2KM11 - $System] $Subject
Omit Email To = 1

Email System General = xxx
</pre>
* ./start_elogd &
* firefox http://localhost:8082 # hould show the elog message index

To start elogd automatically when the machine is rebooted, login as root and
* add this text to /etc/rc.local:
<pre>
su - titan -c "/home/titan/packages/elog/start_elogd"
</pre>
* chmod a+x /etc/rc.local
* systemctl start rc-local

To import elog entries from the mhttpd elog, do this:

* cd ~/packages/elog/logbooks
* ln -s /home/t2km11/online/elog midas
* cd midas
* ~/packages/elog/elconv)

== Install https proxy ==

THESE INSTRUCTIONS ARE OBSOLETE, INSTEAD,
* GO HERE: https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Configure_HTTPS_server_.28CentOS7.29
* AND GO HERE: https://midas.triumf.ca/MidasWiki/index.php/Quickstart_Linux#Run_the_MIDAS_Web_Server

FOLLOWING INSTRUCTIONS ARE OBSOLETE...

* login as root to the https proxy machine
* cd ~root
* yum install mod_ssl
* yum install crypto-utils # see http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_monitoring_of_HTTPS_certificates
* create a certificate request (replace ladd09 with your hostname): openssl req -new -nodes -newkey rsa:2048 -sha256 -out ladd09.csr -keyout ladd09.key (answer: CA, BC, Vancouver, TRIUMF, DAQ, ladd09.triumf.ca, email@email.com
* sign it by TRIUMF:
** mail -s "Certificate request" yourself@email.com < ladd09.csr
** forward this request to Andrew Daviel
** he will email the signed crt file, copy it to this system as ladd09.crt
* sign it yourself: openssl x509 -req -days 365 -sha256 -in ladd09.csr -signkey ladd09.key -out ladd09.crt
* (if the certificate expires, renew it by signing it again)
* Additional commands for working with certificates:
** explore the private key: openssl pkey -in ladd09.key -text -noout
** explore the certificate request: openssl req -in ladd00.csr -text -noout
** explore the certificate: openssl x509 -in ladd09.crt -noout -text
* move certificate files to proper system locations:
* mv ladd09.key /etc/pki/tls/private/
* mv ladd09.crt /etc/pki/tls/certs/
* if selinux is enabled, do this:
** restorecon -Rv /etc/pki/tls/certs/
** restorecon -Rv /etc/pki/tls/private/
** /usr/sbin/setsebool -P httpd_can_network_connect 1
* open /etc/httpd/conf.d/ssl.conf in a text editor, go to the very bottom and right before the "</VirtualHost>" entry, add following text:
<pre>
...
SSLCertificateFile /etc/pki/tls/certs/ladd09.crt
SSLCertificateKeyFile /etc/pki/tls/private/ladd09.key

ProxyPass /elog/ http://localhost:8082/ retry=1
ProxyPass / http://localhost:8080/ retry=1

<Location />

SSLRequireSSL
AuthType Basic
AuthName "password protected site"
Require valid-user

# create password file: touch /etc/httpd/htpasswd
# to add new user or change password: htpasswd /etc/httpd/htpasswd username
AuthUserFile /etc/httpd/htpasswd

</Location>
</VirtualHost>
...
</pre>
* comment out duplicate "SSLCertificateFile" and "SSLCertificateKeyFile" elsewhere in the file
* (optionally) If you got a certificate that is signed by DigiCert or RapidSSL then you'll need to add a line specifying the certificate chain file:
<pre>
...
SSLCertificateChainFile /etc/pki/tls/certs/DigiCertCA.crt
...
</pre>
* touch /etc/httpd/htpasswd
* htpasswd /etc/httpd/htpasswd midas # enter password midas
* chkconfig httpd on
* service httpd restart
* firewall-cmd --add-port=443/tcp --permanent
* firewall-cmd --reload
* firewall-cmd --list-all
* test it
** test the SSL proxy: https://host/ should yield the midas status page, https://host/elog/ should yield the elog message index
* in ODB, set "/Elog/URL" to "https://host/elog/"
* now from the midas status page, the "Elog" button should take us to the https Elog URL

In needed, enable user directories: https://blah/~user in ~user/public_html

* edit /etc/httpd/conf.d/userdir.conf, replace "UserDir disabled" with "UserDir enabled"
* setsebool -P httpd_enable_homedirs true
* systemctl restart httpd

== Setup the history mhttpd for faster access to history plots ==
When running an SSL proxy,
* start the main mhttpd (orange command for old mhttpd, green for new mhttpd with Mongoose(post August2015):
** "mhttpd -p 8071 -D" or
** "mhttpd -D --oldserver 8071 --nomg" 

* start the history mhttpd
** "mhttpd -p 8072 -D -H" or
** "mhttpd -D -H --oldserver 8072 --nomg" 
* set ODB /History/URL to "http://alphacpc09.cern.ch:8072/HS/"
* open the MIDAS status page
* go to the history section, try to open any history plot, observe that the history plot gif image loads correctly, inspect it's URL (use "copy image URL" or "view source", etc), it should point to port 8072 causing connection to the history mhttpd.
* continue with these instructions to setup history mhttpd access through an SSL proxy:
* setup SSL proxy access (required mod_proxy_html)
** login as root to the SSL proxy machine
** on SL5, install the missing mod_proxy_html httpd module:
** yum install httpd-devel libxml2-devel
** wget http://apache.webthing.com/mod_proxy_html/mod_proxy_html.tar.bz2
** tar xjvf mod_proxy_html.tar.bz2
** cd mod_proxy_html
** apxs -c -I. -I/usr/include/libxml2 -i mod_proxy_html.c
** apxs -c -I. -I/usr/include/libxml2 -i mod_xml2enc.c
** cd /etc/httpd/conf.d, add this to ssl.conf:

;before the ProxyPass statements:
<pre>
# proxy the MIDAS web servers
LoadModule xml2enc_module modules/mod_xml2enc.so
LoadModule proxy_html_module modules/mod_proxy_html.so
ProxyHTMLLinks a href
ProxyHTMLLinks link href
ProxyHTMLLinks img src
#ProxyHTMLEnable On
ProxyRequests off
</pre>
;after the ProxyPass statements:
<pre>
# ALPHA1 history access
ProxyPass /alpha1/history/ http://alphacpc09.cern.ch:8072/HS/ retry=1
ProxyPass /alpha1/ http://alphacpc09.cern.ch:8071/ retry=1

ProxyHTMLEnable On
ProxyHTMLURLMap http://alphacpc09.cern.ch:8072/HS/ /alpha1/history/
</pre>
;adjust:
*"alpha1" is the experiment name
*"alphacpc09.cern.ch" is the machine running mhttpd
*"8071" is the port number of the main mhttpd
** "mhttpd -p 8071 -D" or
** "mhttpd -D --oldserver 8071 --nomg" 
* "8072" is the port number of the history mhttpd
** "mhttpd -p 8072 -D -H" or
** "mhttpd -D -H --oldserver 8072 --nomg" 

[[Category:Installation]] [[Category:Buffer]] [[Category:Driver]]

Quickstart Linux

2016-11-16T23:41:26Z

Olchansk: /* ROOTANA Package Installation */

Quickstart Linux

2016-11-08T01:28:22Z

Olchansk: /* Run test frontend */

{{Pagelinks}}
== Introduction ==
This quickstart shows you how to install the MIDAS packages on a linux box and create a '''MIDAS experiment'''.
The linux box will be called the MIDAS '''''experiment host'''''.

==Installation==
Log on to the ''[[#Introduction|Experiment Host]]'' and decide on a name for the MIDAS experiment (the ''experiment name'').
In these instructions, the MIDAS packages and experiment will be installed under the username "johnfoo".
The ODB and other shared memory buffers for the MIDAS experiment will reside on the ''experiment host'' (localhost).
MIDAS and the other required packages will be installed under directory {{Filepath|path=$HOME/packages}}. The MIDAS ''experiment directory'' will be {{Filepath|path=$HOME/online}} and the MIDAS ''experiment name'' will be "e777". The name of the ''experiment host'' will be "mhostpc".

;NOTE
: The user should substitute directory paths for the MIDAS packages and experiment directory as required, and names for ''experiment host'', username, ''experiment name'' appropriate to his/her own setup.

== Experimental setup ==
These instructions describe setting up an experiment that will run ONLY on the MIDAS ''experiment host'' computer (i.e. '''localhost only'''). In this case, the MIDAS ''experiment host'' has access to any hardware required, and all frontends, logger, analyzer etc. run on the one computer.

Modifications to run an experiment with additional frontend(s) running '''remotely''' (i.e. on remote cpu(s)) will also be described.
Follow the instructions for setting up a localhost-only experiment until indicated.

In both cases, the main MIDAS applications (e.g. [[mlogger]], [[mhttpd]]) run on the MIDAS ''experiment host''.

==Environment Variables==
The following Environment variables should to be added to the {{File|name=.cshrc}} or {{File|name=.bashrc}} file (depending on your shell) in the {{Filepath|path=$HOME}} directory on the ''experiment host'', so that they will be defined at login. '''Substitute appropriate values for your own setup'''.

{| class="wikitable"
|-
! csh !! bash !! comment
|-
| setenv GIT_EDITOR "emacs -nw" || export GIT_EDITOR="emacs -nw" ||
|-
| setenv MIDASSYS $HOME/packages/midas || export MIDASSYS=$HOME/packages/midas ||Base directory of the MIDAS package
|-
| setenv MIDAS_EXPTAB $HOME/online/exptab || export MIDAS_EXPTAB=$HOME/online/exptab || MIDAS experiment table
|-
|-
| setenv MIDAS_EXPT_NAME e777 || export MIDAS_EXPT_NAME=e777 || MIDAS experiment name
|-
| setenv PATH .:$MIDASSYS/linux/bin:$PATH || export PATH=$PATH:$MIDASSYS/linux/bin || path
|}

Logout and login again, or source {{File|name=.cshrc}} (source {{File|name=.bashrc}}) for the changes to take effect.
;NOTE
: See [[MIDAS environment variables]] for a list of environment variables used by MIDAS.

== ROOT Package Installation ==
For full MIDAS operation, ROOT is needed for the data logging and analysis packages. It needed to build the example analyzer in this Quickstart. However, ROOT is not essential to run MIDAS. If ROOT is NOT installed, the environment variable ROOTSYS should be undefined.

If ROOT is needed, see [[Setup MIDAS experiment#Install ROOT|Install ROOT]].

== MIDAS Package Installation ==
The MIDAS package will be installed on the ''experiment host'' in the directory given by MIDASSYS - see [[#Environment Variables]].

Install the MIDAS and other package(s) from the MIDAS git repository in an xterm as follows:

[mhostpc] mkdir $HOME/packages
[mhostpc] cd $HOME/packages
[mhostpc] git clone https://bitbucket.org/tmidas/midas
[mhostpc] git clone https://bitbucket.org/tmidas/mxml

Install the MSCB package '''only''' if you will be using the MSCB system for slow-controls (see [[MSCB Page]]):

[mhostpc] git clone https://bitbucket.org/tmidas/mscb ## MSCB package
[mhostpc] cd mscb
[mhostpc] make ## Build the MSCB package
[mhostpc] cd ../

The MIDAS Makefile will auto-detect whether MSCB, ROOT, MYSQL, SQLITE, ODBC etc. are available (or have been installed). If you need these features, make sure they are available before building the MIDAS package.

[mhostpc] cd midas
[mhostpc] make ## Build the MIDAS package
[mhostpc] ls -l linux/bin/odbedit ### check that odbedit has been created (do not run it yet)


In case of problems see
* [[Installation#Installing MIDAS|bitbucket is down]] or
* [[Installation/Compilation problems#SSL certificate errors|SSL certificate errors]].

==ROOTANA Package Installation==
If ROOTANA is needed,

[mhostpc] cd $HOME/packages
[mhostpc] git clone https://bitbucket.org/tmidas/rootana
[mhostpc] cd rootana
[mhostpc] make

==ROODY Package Installation==
If ROODY is needed,

[mhostpc] cd $HOME/packages
[mhostpc] git clone https://bitbucket.org/tmidas/roody
[mhostpc] cd roody
[mhostpc] make


== Create the Experiment file exptab ==
In this example, the experimental directory is {{Filepath|path=$HOME/online}}. Create this directory on the ''experiment host'' (localhost):

[mhostpc] mkdir $HOME/online
[mhostpc] cd $HOME/online
Then create the [[exptab]] file {{Filepath|path=$HOME/online/exptab}} containing the three parameters:
# the experiment name
# location of MIDAS shared memory buffers
# username
as follows (using parameters for your own experiment):

[mhostpc] cat > exptab
e777 /home/johnfoo/online johnfoo

The path of the {{File|name=exptab}} file is given by MIDAS_EXPTAB - see [[#Environment Variables]].

== Create shared memory files ==
At this point you should be able to run [[odbedit]] on the ''experiment host'' (localhost). The first time {{Utility|name=odbedit}} is run, it will create the required {{File|name=.*.SHM}} files in the MIDAS experiment directory {{Filepath|path=$HOME/online}}. These are the saved files for the MIDAS shared memory.

The default directory {{Filepath|path=$HOME/online}} will contain the MIDAS messages file ({{File|name=midas.log}}) and
any data files you may create. If you want an alternative location for the data files see [[#Customizing your experiment]].

Run {{Utility|name=odbedit}}, and type {{Odbedit cmd|cmd=ls}} to list the default directories.
[mhostpc] odbedit
[local:e777:S]/>ls
Experiment
System
Programs
Logger
Runinfo
Alarms

== Run the MIDAS Web Server ==

Recommended for production systems with maximum security is to run mhttpd behind an industry-standard SSL/https proxy with password protection.

To do this using apache httpd on CentOS-7, do this:
* configure apache httpd per instructions at https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Configure_HTTPS_server_.28CentOS7.29
* run odbedit, change values of
** "/experiment/http redirect to https" set to "n"
** "/experiment/midas https port" set to 0
* run mhttpd
* use web browser to connect to https://mhostpc/
* enter the user name and password configured in apache httpd
* the midas status page should open

For simplified installation, one can use standalone mhttpd with built-in https and password protection. This configuration
is only as secure as the underlying mongoose and OpenSSL web server and encryption packages.

Many current operating systems implement a built-in firewall which will block outside access to mhttpd.

On CentOS-7, so this: https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29

Start [[mhttpd]] on the ''experiment host'' (localhost) like this:
[mhostpc] mhttpd
You will get the following messages:

Mongoose web server will listen on ports "8080r,8443s"
[mhttpd,ERROR] [mhttpd.cxx:17892:mongoose,ERROR] cannot find SSL certificate file "/home/agdaq/online/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17893:mongoose,ERROR] please create SSL certificate file: openssl req -new -nodes -newkey rsa:2048 -sha256 -out ssl_cert.csr -keyout ssl_cert.key; openssl x509 -req -days 365 -sha256 -in ssl_cert.csr -signkey ssl_cert.key -out ssl_cert.pem; cat ssl_cert.key >> ssl_cert.pem
could not start the mongoose web server, see messages and midas.log, bye!

Create a self-signed certificate suitable for initial testing by executing the commands printed by mhttpd.

For production use, you should create a properly signed certificate, see [[Mhttpd#Create an SSL certificate|create your own SSL certificate]] or you should run mhttpd behind an SSL/https proxy (apache httpd).

Run mhttpd again.

You will get the following messages:

Mongoose web server will listen on ports "8080r,8443s"
Mongoose web server will use SSL certificate file "/home/johnfoo/packages/midas/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17633:mongoose,ERROR] mongoose web server cannot find password file "/home/johnfoo/online/htpasswd.txt"
[mhttpd,ERROR] [mhttpd.cxx:17634:mongoose,ERROR] please create password file: htdigest -c /home/johnfoo/online/htpasswd.txt Default midas
could not start the mongoose web server, see messages and midas.log, bye!

Create the password file by following the instructions printed by mhttpd. The http digest domain name is the experiment name, suggested default user name is "midas". You will be asked to type in a password. (htdigest is part of the httpd-tools package)


[mhostpc] htdigest -c /home/johnfoo/online/htpasswd.txt exptname midas
Adding password for midas in realm exptname.
New password:
Re-type new password:

It is a good idea to set the password file {{Filepath|path=htpasswd.txt}} readable and writable by owner only.

Now restart {{Utility|name=mhttpd}}
[mhostpc] mhttpd
Mongoose web server will listen on ports "8080r,8443s" **see note
Mongoose web server will use SSL certificate file "/home/suz/packages/midas/ssl_cert.pem"
Mongoose web server will use authentication realm "Default", password file "./htpasswd.txt"

Now point a web browser running on the same host computer (localhost) to https://localhost:8443
If the web browser is running on a different computer, go to URL of the form

https://mhostpc.triumf.ca:8443 (substitute your host machine name and domain for "mhostpc.triumf.ca")

If you are using the default SSL certificate you will probably get a message: "This Connection is Untrusted". Click "I understand the risks" and add an exception. This is because the test certificate is self-signed. Then confirm an exception.

If instead you get a "connection refused" error, the midas host pc may have the firewall enabled. To make a firewall exception for MIDAS, follow instructions here http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29

You should then see an authentication box asking you for the user name and password. The user name is "midas". Enter the password you just created. The Midas [[Status Page]] should appear with multiple buttons for run control as well as equipment listing (no equipments will be listed as yet) and application listings. Please refer to [[mhttpd]] (the MIDAS Web-based Run Control utility) for further information. You can start and stop runs from the main status page, and use the [[ODB Page]] to access the database (ODB).

; Note
: Default ports of 8080 and 8443 are used by [[mhttpd]]. To use some other ports, change the values of ODB "/experiment/midas http port" and "midas https port".

== Run the MIDAS logger ==
Start the MIDAS logger [[mlogger]] on the ''experiment host'' (localhost) :
[mhostpc] mlogger
This should start without error. It is usually run as a daemon, however in this case it is run in a terminal to check for errors. Starting the midas logger [[mlogger]] will automatically create more keys in the {{Odbpath|path=/Logger}} ODB tree.

From the MIDAS status page, go to the "Programs" page, then the "Logger" page and set ODB variables:
* Required -> set to "y"
* Start command -> set to "mlogger -D"
* Alarm Class -> set to "Alarm"

To test this, go to the "Programs" page, press "stop logger", then "start logger".

Start a run, wait 10 seconds, stop the run.

From the MIDAS "Status" page, go to the "logging channels -> channel #0" and set ODB variables:
* Output -> "FILE"

Start a run, go to the "Status" page, observe that the output file is written with gzip compression and has the "mid.gz" extension.

Stop the run.

== Run test frontend ==

Run fetest

On the MIDAS "Status" page there will be 3 new equipment entries:
* test -> empty data bank for daq speed benchmarking
* slow -> example slow controls data for testing the midas history plots
* random -> example collection of data banks for testing the analyzer

Start a run.

Observe event counters are incrementing, output data file size is growing.

Stop the run.

== Create a history plot ==

Go to the MIDAS "History" page.

Say "new", panel name "test". In the history editor select event "slow", say "save".

History plot "test" should open with a sine-wave plot. Data on the plot should update as long as "fetest" is running. (there is no need to start a run).

Press web browser "reload" to update the plot.

== Clients run on Localhost only ==
;NOTE:
* If creating a MIDAS experiment with a '''REMOTE''' frontend, '''continue by following the instructions''' [[#Running with one or more REMOTE frontends]].
* If all clients are running on the Experiment Host (i.e. '''localhost'''), continue with the following instructions:

=== Frontend and Analyzer (localhost) ===

THIS SECTION IS VERY OBSOLETE

There are several examples of [[Frontend user code]] ({{File|name=frontend.c}}) in the MIDAS package available under {{Filepath|path=$MIDASSYS/examples/}}. Choose a suitable example from this directory that you can later modify for your own particular setup (e.g. for a slow control, choose the {{Filepath|path=../slowcont/}} example).

A '''frontend''' is a program that usually reads data from the hardware and sends it to a buffer to be logged and analyzed. You can find documentation about the frontend structure under [[Frontend Application]], [[Frontend Operation]] and [[Frontend user code]].

The example chosen here is from {{Filepath|path=$MIDASSYS/examples/experiment/}}. It does '''not need any hardware''' and produces simulated events.

On the MIDAS ''experiment host'' (localhost), copy the example to the experiment directory and build it.

[mhostpc] cd $HOME/online
[mhostpc] cp $MIDASSYS/examples/experiment/* .
[mhostpc] make
The analyzer will only build if ROOT has previously been installed.
At this point the frontend and the analyzer should be ready if no errors were generated during the build. Try the frontend and analyzer by starting them in xterms.

[mhostpc] frontend
[mhostpc] analyzer

=== Equipment ODB tree ===
Starting the frontend will automatically create the {{Odbpath|path=/Equipment}} ODB tree and one or more equipments. In this case, the equipments "trigger" and "scaler" have been defined by code in {{File|name=frontend.c}} and created the first time frontend runs.

[mhostpc] odbedit
[local:e777:S]/>ls /equipment
Trigger
Scaler 

View the MIDAS main status web page (by pointing your browser to https://localhost:8443/ or appropriate port if not using the default).
Observe on the [[Status Page]] that the two Equipments (Trigger, Scaler) have appeared in the Equipment section (Figure 1) and are coloured green.
 
'''Click thumbnail to enlarge'''
[[File:quickstart_status_page.png|thumb|left|Figure 1: web server status page showing example frontend equipments]]
<div style="clear: both"></div> 

Rather than using odbedit, you can view the contents of these equipments by clicking on the ODB button, then on {{Odbpath|path=/Equipment}}, then {{Odbpath|path=Trigger}} or {{Odbpath|path=Scaler}}. See [[ODB Page]] for more information.

=== Start a run ===
Start a run by pressing the Start button on the [[Status Page]].
The Equipment display on the Status Page will now show some event statistics (Figure 2)
'''Click thumbnail to enlarge'''
[[File:sample_frontend_status.png|thumb|left|Figure 2: Equipments shown on Status Page showing event statistics]]
<div style="clear: both"></div> 

The frontend will show an event display that will update when a run is started, e.g.

Sample Frontend connected to <local>. Press "!" to exit 18:48:2546:08
==========================================================================
Run status: Running Run number 2 |/ /
===========================================================================
Equipment Status Events Events/sec Rate[B/s] ODB->FE FE->OD
---------------------------------------------------------------------------
Trigger OK 13948 99.0 5413.0 0 140
Scaler OK 15 0.3 15.4 0 15

The [[Status Page]] will also show events being generated (click the refresh button if necessary).

=== Event Dump ===
While a run is in progress, the midas application [[mdump]] will provide you an event dump of the collected data from the running frontend, e.g.

mhostpc> mdump -l 50
blank
- MIDAS revision: Mon Nov 2 11:50:51 2015 -0800 - 3b66779 -- Enter <!> to Exit ------- Midas Dump ---
------------------------ Event# 1 ------------------------
Evid:0001- Mask:0000- Serial:0- Time:0x567a0c5d- Dsize:40/0x28
#banks:2 - Bank list:-ADC0TDC0-
blank
Bank:ADC0 Length: 8(I*1)/2(I*4)/4(Type) Type:Unsigned Integer*2
1-> 0x0167 0x03c6 0x0069 0x0073
blank
Bank:TDC0 Length: 8(I*1)/2(I*4)/4(Type) Type:Unsigned Integer*2
1-> 0x0051 0x00ff 0x004a 0x00ec

For further data processing/analysis, either the midas analyzer or the rootana can used for data display as well.

=== Create a startup script (localhost) ===
It is useful to create a script that will automatically start all the required clients for the experiment. This is called a startup script. Here, the startup script will be {{File|name=start_daq.sh}} (any name can be used). Create {{Filepath|path=$HOME/online/bin/start_daq.sh}} on the ''experiment host'' as follows (supply the [[mhttpd]] ports if default is not used):

#!/bin/sh
# start_daq.sh
cd $HOME/online
odbedit -c clean
# start mhttpd on default port. (Mongoose https version)
mhttpd -D # you can optionally restrict access to localhost and other specified hosts - see [[mhttpd]]
xterm -e ./frontend &
xterm -e ./analyzer &
mlogger -D
#end file

Before running this script, you will need to shutdown any running clients:
mhostpc> odbedit -c "sh all"
or you can use the {{Utility|name=mhttpd}} [[Programs Page]] to shut them down.

The script will start them as daemons.
By running the startup script start_daq.sh, several MIDAS applications will be started in sequence:
# Cleanup previous MIDAS application (if any).
# Start the MIDAS web server [mhttpd]
# Start the frontend application in its own xterm (for debugging purpose).
# Start the analyzer application in its own xterm (for debugging purpose).
# Start the MIDAS Data logger [mlogger]
[mhostpc] sh ./start_daq

You may wish to modify the script to restart missing clients only.

Next [[#Customizing your experiment|customize your experiment]].

== Running with one or more REMOTE frontends ==
These instructions assume you have already followed the instructions to setup an experiment running on the MIDAS experiment host (localhost), i.e. you have setup the MIDAS Environment variables, downloaded MIDAS packages, setup the [[mhttpd]] password and have {{Utility|name=mhttpd}} running successfully.

In the case of '''remote''' frontend(s), the remote cpu(s) typically have access to some or all of the hardware. They might be VMIC cpus running in VME crates for example, connected to the MIDAS Host via Ethernet. They read data from the hardware and send it back to the MIDAS ''experiment host'' to be logged and analyzed.

=== Experimental setup (Remote) ===
These instructions assume that the MIDAS ''experiment host'' computer is 64-bit (if not see [[#Build 32-bit MIDAS libraries|note below]]), and there is one ''remote host'', a 32-bit machine named "rlxhost".

The ''remote host'' mounts the {{Filepath|path=/home}} disk of the ''experiment host'', so that it '''has access to the MIDAS packages''', the {{Filepath|path=$HOME/online}} directory and shares the .cshrc (or .bashrc) script. It is convenient to set up ssh keys (use ssh-keygen) so that the ''remote host'' can be accessed without supplying the password from the ''experiment host''.

If your setup is different, you will have to make changes as appropriate.

=== Build 32-bit MIDAS libraries ===
NOTE

If the MIDAS ''experiment host'' computer is 32-bit (i.e. the same as the ''remote host''), which is a simpler setup, you will have built the 32-bit MIDAS libraries already, and can skip the rest of this section. Note however, that in the .cshrc file in the [[#Modify .cshrc (.bashrc) for Remote host|following section]] you will not need the lines starting "# select 64-bit or 32-bit MIDAS and ROOT". The PATH should contain $MIDASSYS/linux/bin on both experiment host and remote host. This will point to the 32-bit MIDAS libraries.
------------------------------

 
Assuming the MIDAS host computer is 64-bit, you will have already built the 64-bit MIDAS libraries.
In this case, if the remote frontend is 32-bit, you will need to build the 32-bit MIDAS libraries on the 64-bit machine, i.e.

[mhostpc] cd /home/packages/midas
[mhostpc] make linux32 ### build the 32-bit MIDAS libraries on 64-bit machine
[mhostpc] ls -l linux-m32/bin/odbedit ### check that the 32-bit odbedit has been created 
Do not run {{Utility|name=odbedit}} yet on the ''remote host''!

=== Modify .cshrc (.bashrc) for Remote host ===
Add the following to the {{File|name=.cshrc}} (modified appropriately if using {{File|name=.bashrc}}). When running with a REMOTE Host, the application [[mserver]] will be started (using a particular port) on the MIDAS ''experiment host'' machine only. Note that the environment variable MIDAS_SERVER_HOST is defined on the remote machine(s) but NOT on the ''experiment host''.

# setenv MIDAS_HOST mhostpc # substitute your experiment host name
# setenv REMOTE_HOST rlxhost # substitute your remote host name
# setup the MIDAS mserver host
#
switch (`hostname`)
case $MIDAS_HOST*:
unsetenv MIDAS_SERVER_HOST ## MIDAS_SERVER_HOST not defined
breaksw
default:
setenv MIDAS_SERVER_HOST $MIDAS_HOST.triumf.ca:1175 ## for remote host, define MIDAS_SERVER_HOST
## as hostname + domain name + default mserver port
## port must match that of [[mserver]] used in start_daq.sh
endsw
#
# select 64-bit or 32-bit MIDAS and ROOT
#
switch (`uname -i`)
case i386:
setenv ROOTSYS /triumfcs/trshare/olchansk/root/root_v5.28.00_SL55_32 ## or appropriate path
setenv PATH .:$MIDASSYS/linux-m32/bin:$PATH
breaksw
default:
setenv ROOTSYS $HOME/packages/root
setenv PATH .:$MIDASSYS/linux/bin:$PATH
endsw
#
setenv PATH .:$HOME/online/bin:$HOME/packages/roody/bin:$ROOTSYS/bin:$PATH
#

After executing {{File|name=.cshrc}} (or {{File|name=.bashrc}}) or logging out, on the MIDAS host (mhostpc) make sure environment variable MIDAS_SERVER_HOST is NOT defined
[mhostpc] echo $MIDAS_SERVER_HOST
[mhostpc] MIDAS_SERVER_HOST: Undefined variable.

and on the '''''remote host''''', make sure MIDAS_SERVER_HOST IS defined:
[rlxhost] echo $MIDAS_SERVER_HOST
[rlxhost] MIDAS_SERVER_HOST

Also on the ''remote host'', make sure that the correct (32-bit) {{Utility|name=odbedit}} will be used (do not run it yet!)
[rlxhost] which odbedit
/home/johnfoo/packages/midas/linux-m32/bin/odbedit

=== Grant Remote host access permission ===
Give permission for the remote host(s) to access the experiment by following the instructions to [[Security#MIDAS programs on remote machines|allow MIDAS programs on remote machines]]. If the MIDAS web server [[mhttpd]] is running, use [[ODB Page]] to edit the ODB keys listed, otherwise use [[odbedit]] on the ''experiment host'' machine.

=== Start mserver on Experiment Host ===
The application [[mserver]] is not required when running an experiment on the ''experiment host'' (localhost) only, but it '''IS required''' for access from a '''REMOTE''' frontend.
Start [[mserver]] on MIDAS host (default port 1175) or use the "-p" option for a different port.
The port must match the port defined for environment variable MIDAS_SERVER_HOST in {{File|name=.cshrc}} (or {{File|name=.bashrc}}) [[#Modify .cshrc (.bashrc) for Remote host|above]].
[mhostpc] mserver -D

The environment variables on the REMOTE host should be:

[rlxhost] printenv | grep MIDAS
MIDASSYS=/home/e777/packages/midas
MIDAS_EXPT_NAME=e777
MIDAS_SERVER_HOST=mhostpc.triumf.ca:1175
(a different port may be used - see above)

=== Check access to ODB from Remote host ===
Run {{Utility|name=odbedit}} on the remote host to check access to ODB.
[rlxhost] odbedit
[mhostpc.triumf.ca:1175:e777:Stopped]/>ls 
If it does not work correctly, check that
* MIDAS_SERVER_HOST is defined on REMOTE host with same port as [[mserver]]
* mserver is running on MIDAS host with the correct port
* MIDAS_SERVER_HOST is NOT defined on MIDAS host
* remote access permission to REMOTE host has been granted as described above.

=== Build the analyzer on Experiment Host ===
The analyzer will be built and run on the MIDAS ''experiment host'' machine.
Select an example from $MIDASSYS/examples/ (see [[#Frontend and Analyzer (localhost)]]) and copy it to $HOME/online on the MIDAS host machine. Since we are assuming a 64-bit MIDAS host and a 32-bit remote host, edit the example Makefile so only the analyzer will be built.

=== Build the frontend (32bit) for Remote host ===
The frontend runs on a 32-bit remote host in this example. It therefore must either
* be built 32-bit on the 64-bit MIDAS host with the "-m32" flag
* or built on the REMOTE host
and in either case the [[#Build 32-bit MIDAS libraries|32-bit MIDAS libraries]] used for linking.

To avoid confusion, create a different subdirectory for the frontend, e.g.
[rlxhost] mkdir $HOME/online/fe_32
Copy the Makefile and frontend code from the same example to this directory.
Modify the Makefile as needed, and only build the frontend.

Once built, start the frontend in an xterm on the REMOTE host.
[rlxhost] $HOME/online/fe_32/frontend
Check that it connects successfully to the experiment and the Equipments appear on the [[Status Page]] as described under [[#Frontend and Analyzer (localhost)]]. The Equipment Status (Figure 2) should show that the Equipments are now running on the remote cpu, i.e. "SampleFrontend@rlxhost.triumf.ca" rather than "Sample Frontend@localhost".

=== Run the Analyzer on Experiment host ===
Start the analyzer in an xterm on the MIDAS ''experiment host''
[johnfoo@mhostpc e777] $HOME/online/analyzer

Continue by starting a run and dumping the data as described under [[#Frontend and Analyzer (localhost)]].

=== Create experiment startup script ===
It is useful to create a script that will automatically start all the required clients for the experiment. This is called a startup script (any name can be used). Create {{Filepath|path=$HOME/online/bin/start_daq.sh}} on the ''experiment host'' as follows, (supply the [[mhttpd]] ports if default is not used) :

#!/bin/sh
# start_daq.sh
cd $HOME/online
#
switch (`hostname`)
case $MIDAS_HOST*: ## MIDAS_HOST is assigned in .cshrc (.bashrc)
echo "Good, we are on $MIDAS_HOST"
breaksw
case $REMOTE_HOST*: ## REMOTE_HOST is assigned in .cshrc (.bashrc)
echo "startup script should be executed on $MIDAS_HOST"
endsw
#
odbedit -c clean
# start mhttpd on default port (Mongoose https version)
mhttpd -D # you can optionally restrict access to localhost and/or other specified hosts - see [[mhttpd]]
# start mserver on default port (use argument -p to use a different port)
mserver -D # start mserver on default port (1175) or a different port using -p option - see [[mserver]]
# the port must match that of MIDAS_SERVER_HOST in .cshrc (.bashrc) - see above
# access must be specifically allowed - see [[#Grant Remote host access permission||above]]
xterm -e ./analyzer &
## start frontend on remote host
#ssh $REMOTE_HOST $HOME/online/bin/start_frontend -O >& $HOME/online/bin/start_frontend.log &
mlogger -D
#end file

Note that command to start the frontend on the remote host is commented out. It can be uncommented once the start_frontend script is written and tested.

Shut down any clients that are running already by running odbedit on the MIDAS ''experiment host'' and execute the new startup script {{File|name=start_daq.sh}} (make it executable if desired)

[mhostpc] odbedit
[local:e777:S]/>sh all
[mhostpc] ./start_daq.sh 
and check that {{Utility|name=odbedit}} still works on the ''REMOTE host''

[rlxhost] odbedit
[mhostpc.triumf.ca:1175:e777:Stopped]/>ls 

You may wish to modify the script to restart missing clients only.

=== Create script to start frontend ===
The startup file {{File|name=start_daq.sh}} invoked a script {{File|name=start_frontend}} to start the remote frontend. This line was commented out (see [[#Create experiment startup script|above]]). Once the frontend is working, create a script to start the frontend (e.g. file {{Filepath|path=$HOME/online/bin/start_frontend}}).

#!/bin/tcsh
# Script to start frontend running on $REMOTE_HOST
#
# This script runs on the REMOTE HOST
#
echo "starting frontend for experiment $MIDAS_EXPT_NAME "
xterm -geometry 150x50+800+0 -fg white -bg blue -title "Frontend" -e "$HOME/online/fe_32/frontend" &
echo "Frontend task has been started"
exit

Check that this script works by running it on the REMOTE host. If successful,
run the commented command in an xterm on the ''experiment host''
[mhostpc] ssh $REMOTE_HOST $HOME/online/bin/start_frontend -O >& $HOME/online/bin/start_frontend.log & 
This command relies on password-less access set up with SSH keys (see [[#Experimental setup (Remote)]]).
If successful, uncomment line in {{File|name=start_daq.sh}}, and run the script to check that it can start the remote client(s) from the MIDAS ''experiment host'' (first shutting down all running clients) :

[mhostpc] odbedit -c "sh all"
[mhostpc] ./start_daq.sh 

Next [[#Customizing your experiment|customize your experiment]].

== Customizing your experiment ==

Here are some of the more common options provided for users to customize their experiments according to their own preferences:

; Write frontend(s) to read out and control your hardware
: Modify the example frontend and Makefile (from $MIDASSYS/examples/) you have already copied and built so it accesses your hardware.

: The example Frontend.c is an example of [[Frontend user code]]. You can find documentation about the frontend structure under [[Frontend Application]], [[Frontend Operation]] and [[Frontend user code]]. You may need to set up some [[Event Notification (Hot-Link)|Hot-Links]] in the frontend. The MIDAS package includes device drivers for some of the more common hardware. See [[MIDAS Driver Library]] for more information.

; Modify analyzer(s) for your data
: Modify the example analyzer code as necessary to analyze the data from your frontends.

; Slow Controls
: Control your slow-controls equipment with slow-control frontend(s) optionally using the [[#MIDAS Package Installation|MSCB]] system.

; Data logger
: Set up the data logger [[mlogger]] to log data to a storage device. Many options are available - see [[Data Logger]], [[mlogger]] and [[/Logger ODB tree]] for instructions.

; Allow programs to be restarted from the [[Programs Page]]
: See mhttpd [[Programs Page]] and [[/Programs ODB tree]]

; Write scripts which run at start and end of run
: See [[/Programs ODB tree#Execute on start run]] and [[/Programs ODB tree#Execute on stop run]]

; Create a shutdown script
: Create a script similar to the startup script to shutdown the experiment

; Lazy logger
: Set up the [[lazylogger]] for archiving data files for storage.

; Create edit-on-start parameters
: See [[Edit-on-start Parameters]]

; Set up the alarm system
: See [[Alarm System]] and [[Alarms Page]]

; Set up history system
: See [[History System]] and [[History Page]]

; Set up the Electronic logbook (Elog)
: See [[Elog]] for details

; Set up the Sequencer
: The [[Sequencer Page#Sequencer]] can be used to run a sequence of runs, changing parameters as needed.

; Write a webserver Custom page
: To better control and monitor your experiment, you may want to write a [[Custom Page]].

[[Category:Installation]]

Quickstart Linux

2016-11-08T01:24:44Z

Olchansk: /* Run test frontend */

{{Pagelinks}}
== Introduction ==
This quickstart shows you how to install the MIDAS packages on a linux box and create a '''MIDAS experiment'''.
The linux box will be called the MIDAS '''''experiment host'''''.

==Installation==
Log on to the ''[[#Introduction|Experiment Host]]'' and decide on a name for the MIDAS experiment (the ''experiment name'').
In these instructions, the MIDAS packages and experiment will be installed under the username "johnfoo".
The ODB and other shared memory buffers for the MIDAS experiment will reside on the ''experiment host'' (localhost).
MIDAS and the other required packages will be installed under directory {{Filepath|path=$HOME/packages}}. The MIDAS ''experiment directory'' will be {{Filepath|path=$HOME/online}} and the MIDAS ''experiment name'' will be "e777". The name of the ''experiment host'' will be "mhostpc".

;NOTE
: The user should substitute directory paths for the MIDAS packages and experiment directory as required, and names for ''experiment host'', username, ''experiment name'' appropriate to his/her own setup.

== Experimental setup ==
These instructions describe setting up an experiment that will run ONLY on the MIDAS ''experiment host'' computer (i.e. '''localhost only'''). In this case, the MIDAS ''experiment host'' has access to any hardware required, and all frontends, logger, analyzer etc. run on the one computer.

Modifications to run an experiment with additional frontend(s) running '''remotely''' (i.e. on remote cpu(s)) will also be described.
Follow the instructions for setting up a localhost-only experiment until indicated.

In both cases, the main MIDAS applications (e.g. [[mlogger]], [[mhttpd]]) run on the MIDAS ''experiment host''.

==Environment Variables==
The following Environment variables should to be added to the {{File|name=.cshrc}} or {{File|name=.bashrc}} file (depending on your shell) in the {{Filepath|path=$HOME}} directory on the ''experiment host'', so that they will be defined at login. '''Substitute appropriate values for your own setup'''.

{| class="wikitable"
|-
! csh !! bash !! comment
|-
| setenv GIT_EDITOR "emacs -nw" || export GIT_EDITOR="emacs -nw" ||
|-
| setenv MIDASSYS $HOME/packages/midas || export MIDASSYS=$HOME/packages/midas ||Base directory of the MIDAS package
|-
| setenv MIDAS_EXPTAB $HOME/online/exptab || export MIDAS_EXPTAB=$HOME/online/exptab || MIDAS experiment table
|-
|-
| setenv MIDAS_EXPT_NAME e777 || export MIDAS_EXPT_NAME=e777 || MIDAS experiment name
|-
| setenv PATH .:$MIDASSYS/linux/bin:$PATH || export PATH=$PATH:$MIDASSYS/linux/bin || path
|}

Logout and login again, or source {{File|name=.cshrc}} (source {{File|name=.bashrc}}) for the changes to take effect.
;NOTE
: See [[MIDAS environment variables]] for a list of environment variables used by MIDAS.

== ROOT Package Installation ==
For full MIDAS operation, ROOT is needed for the data logging and analysis packages. It needed to build the example analyzer in this Quickstart. However, ROOT is not essential to run MIDAS. If ROOT is NOT installed, the environment variable ROOTSYS should be undefined.

If ROOT is needed, see [[Setup MIDAS experiment#Install ROOT|Install ROOT]].

== MIDAS Package Installation ==
The MIDAS package will be installed on the ''experiment host'' in the directory given by MIDASSYS - see [[#Environment Variables]].

Install the MIDAS and other package(s) from the MIDAS git repository in an xterm as follows:

[mhostpc] mkdir $HOME/packages
[mhostpc] cd $HOME/packages
[mhostpc] git clone https://bitbucket.org/tmidas/midas
[mhostpc] git clone https://bitbucket.org/tmidas/mxml

Install the MSCB package '''only''' if you will be using the MSCB system for slow-controls (see [[MSCB Page]]):

[mhostpc] git clone https://bitbucket.org/tmidas/mscb ## MSCB package
[mhostpc] cd mscb
[mhostpc] make ## Build the MSCB package
[mhostpc] cd ../

The MIDAS Makefile will auto-detect whether MSCB, ROOT, MYSQL, SQLITE, ODBC etc. are available (or have been installed). If you need these features, make sure they are available before building the MIDAS package.

[mhostpc] cd midas
[mhostpc] make ## Build the MIDAS package
[mhostpc] ls -l linux/bin/odbedit ### check that odbedit has been created (do not run it yet)


In case of problems see
* [[Installation#Installing MIDAS|bitbucket is down]] or
* [[Installation/Compilation problems#SSL certificate errors|SSL certificate errors]].

==ROOTANA Package Installation==
If ROOTANA is needed,

[mhostpc] cd $HOME/packages
[mhostpc] git clone https://bitbucket.org/tmidas/rootana
[mhostpc] cd rootana
[mhostpc] make

==ROODY Package Installation==
If ROODY is needed,

[mhostpc] cd $HOME/packages
[mhostpc] git clone https://bitbucket.org/tmidas/roody
[mhostpc] cd roody
[mhostpc] make


== Create the Experiment file exptab ==
In this example, the experimental directory is {{Filepath|path=$HOME/online}}. Create this directory on the ''experiment host'' (localhost):

[mhostpc] mkdir $HOME/online
[mhostpc] cd $HOME/online
Then create the [[exptab]] file {{Filepath|path=$HOME/online/exptab}} containing the three parameters:
# the experiment name
# location of MIDAS shared memory buffers
# username
as follows (using parameters for your own experiment):

[mhostpc] cat > exptab
e777 /home/johnfoo/online johnfoo

The path of the {{File|name=exptab}} file is given by MIDAS_EXPTAB - see [[#Environment Variables]].

== Create shared memory files ==
At this point you should be able to run [[odbedit]] on the ''experiment host'' (localhost). The first time {{Utility|name=odbedit}} is run, it will create the required {{File|name=.*.SHM}} files in the MIDAS experiment directory {{Filepath|path=$HOME/online}}. These are the saved files for the MIDAS shared memory.

The default directory {{Filepath|path=$HOME/online}} will contain the MIDAS messages file ({{File|name=midas.log}}) and
any data files you may create. If you want an alternative location for the data files see [[#Customizing your experiment]].

Run {{Utility|name=odbedit}}, and type {{Odbedit cmd|cmd=ls}} to list the default directories.
[mhostpc] odbedit
[local:e777:S]/>ls
Experiment
System
Programs
Logger
Runinfo
Alarms

== Run the MIDAS Web Server ==

Recommended for production systems with maximum security is to run mhttpd behind an industry-standard SSL/https proxy with password protection.

To do this using apache httpd on CentOS-7, do this:
* configure apache httpd per instructions at https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Configure_HTTPS_server_.28CentOS7.29
* run odbedit, change values of
** "/experiment/http redirect to https" set to "n"
** "/experiment/midas https port" set to 0
* run mhttpd
* use web browser to connect to https://mhostpc/
* enter the user name and password configured in apache httpd
* the midas status page should open

For simplified installation, one can use standalone mhttpd with built-in https and password protection. This configuration
is only as secure as the underlying mongoose and OpenSSL web server and encryption packages.

Many current operating systems implement a built-in firewall which will block outside access to mhttpd.

On CentOS-7, so this: https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29

Start [[mhttpd]] on the ''experiment host'' (localhost) like this:
[mhostpc] mhttpd
You will get the following messages:

Mongoose web server will listen on ports "8080r,8443s"
[mhttpd,ERROR] [mhttpd.cxx:17892:mongoose,ERROR] cannot find SSL certificate file "/home/agdaq/online/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17893:mongoose,ERROR] please create SSL certificate file: openssl req -new -nodes -newkey rsa:2048 -sha256 -out ssl_cert.csr -keyout ssl_cert.key; openssl x509 -req -days 365 -sha256 -in ssl_cert.csr -signkey ssl_cert.key -out ssl_cert.pem; cat ssl_cert.key >> ssl_cert.pem
could not start the mongoose web server, see messages and midas.log, bye!

Create a self-signed certificate suitable for initial testing by executing the commands printed by mhttpd.

For production use, you should create a properly signed certificate, see [[Mhttpd#Create an SSL certificate|create your own SSL certificate]] or you should run mhttpd behind an SSL/https proxy (apache httpd).

Run mhttpd again.

You will get the following messages:

Mongoose web server will listen on ports "8080r,8443s"
Mongoose web server will use SSL certificate file "/home/johnfoo/packages/midas/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17633:mongoose,ERROR] mongoose web server cannot find password file "/home/johnfoo/online/htpasswd.txt"
[mhttpd,ERROR] [mhttpd.cxx:17634:mongoose,ERROR] please create password file: htdigest -c /home/johnfoo/online/htpasswd.txt Default midas
could not start the mongoose web server, see messages and midas.log, bye!

Create the password file by following the instructions printed by mhttpd. The http digest domain name is the experiment name, suggested default user name is "midas". You will be asked to type in a password. (htdigest is part of the httpd-tools package)


[mhostpc] htdigest -c /home/johnfoo/online/htpasswd.txt exptname midas
Adding password for midas in realm exptname.
New password:
Re-type new password:

It is a good idea to set the password file {{Filepath|path=htpasswd.txt}} readable and writable by owner only.

Now restart {{Utility|name=mhttpd}}
[mhostpc] mhttpd
Mongoose web server will listen on ports "8080r,8443s" **see note
Mongoose web server will use SSL certificate file "/home/suz/packages/midas/ssl_cert.pem"
Mongoose web server will use authentication realm "Default", password file "./htpasswd.txt"

Now point a web browser running on the same host computer (localhost) to https://localhost:8443
If the web browser is running on a different computer, go to URL of the form

https://mhostpc.triumf.ca:8443 (substitute your host machine name and domain for "mhostpc.triumf.ca")

If you are using the default SSL certificate you will probably get a message: "This Connection is Untrusted". Click "I understand the risks" and add an exception. This is because the test certificate is self-signed. Then confirm an exception.

If instead you get a "connection refused" error, the midas host pc may have the firewall enabled. To make a firewall exception for MIDAS, follow instructions here http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29

You should then see an authentication box asking you for the user name and password. The user name is "midas". Enter the password you just created. The Midas [[Status Page]] should appear with multiple buttons for run control as well as equipment listing (no equipments will be listed as yet) and application listings. Please refer to [[mhttpd]] (the MIDAS Web-based Run Control utility) for further information. You can start and stop runs from the main status page, and use the [[ODB Page]] to access the database (ODB).

; Note
: Default ports of 8080 and 8443 are used by [[mhttpd]]. To use some other ports, change the values of ODB "/experiment/midas http port" and "midas https port".

== Run the MIDAS logger ==
Start the MIDAS logger [[mlogger]] on the ''experiment host'' (localhost) :
[mhostpc] mlogger
This should start without error. It is usually run as a daemon, however in this case it is run in a terminal to check for errors. Starting the midas logger [[mlogger]] will automatically create more keys in the {{Odbpath|path=/Logger}} ODB tree.

From the MIDAS status page, go to the "Programs" page, then the "Logger" page and set ODB variables:
* Required -> set to "y"
* Start command -> set to "mlogger -D"
* Alarm Class -> set to "Alarm"

To test this, go to the "Programs" page, press "stop logger", then "start logger".

Start a run, wait 10 seconds, stop the run.

From the MIDAS "Status" page, go to the "logging channels -> channel #0" and set ODB variables:
* Output -> "FILE"

Start a run, go to the "Status" page, observe that the output file is written with gzip compression and has the "mid.gz" extension.

Stop the run.

== Run test frontend ==

Run fetest

On the MIDAS "Status" page there will be 3 new equipment entries:
* test -> empty data bank for daq speed benchmarking
* slow -> example slow controls data for testing the midas history plots
* random -> example collection of data banks for testing the analyzer

Start a run.

Observe event counters are incrementing, output data file size is growing.

Stop the run.

== Clients run on Localhost only ==
;NOTE:
* If creating a MIDAS experiment with a '''REMOTE''' frontend, '''continue by following the instructions''' [[#Running with one or more REMOTE frontends]].
* If all clients are running on the Experiment Host (i.e. '''localhost'''), continue with the following instructions:

=== Frontend and Analyzer (localhost) ===

THIS SECTION IS VERY OBSOLETE

There are several examples of [[Frontend user code]] ({{File|name=frontend.c}}) in the MIDAS package available under {{Filepath|path=$MIDASSYS/examples/}}. Choose a suitable example from this directory that you can later modify for your own particular setup (e.g. for a slow control, choose the {{Filepath|path=../slowcont/}} example).

A '''frontend''' is a program that usually reads data from the hardware and sends it to a buffer to be logged and analyzed. You can find documentation about the frontend structure under [[Frontend Application]], [[Frontend Operation]] and [[Frontend user code]].

The example chosen here is from {{Filepath|path=$MIDASSYS/examples/experiment/}}. It does '''not need any hardware''' and produces simulated events.

On the MIDAS ''experiment host'' (localhost), copy the example to the experiment directory and build it.

[mhostpc] cd $HOME/online
[mhostpc] cp $MIDASSYS/examples/experiment/* .
[mhostpc] make
The analyzer will only build if ROOT has previously been installed.
At this point the frontend and the analyzer should be ready if no errors were generated during the build. Try the frontend and analyzer by starting them in xterms.

[mhostpc] frontend
[mhostpc] analyzer

=== Equipment ODB tree ===
Starting the frontend will automatically create the {{Odbpath|path=/Equipment}} ODB tree and one or more equipments. In this case, the equipments "trigger" and "scaler" have been defined by code in {{File|name=frontend.c}} and created the first time frontend runs.

[mhostpc] odbedit
[local:e777:S]/>ls /equipment
Trigger
Scaler 

View the MIDAS main status web page (by pointing your browser to https://localhost:8443/ or appropriate port if not using the default).
Observe on the [[Status Page]] that the two Equipments (Trigger, Scaler) have appeared in the Equipment section (Figure 1) and are coloured green.
 
'''Click thumbnail to enlarge'''
[[File:quickstart_status_page.png|thumb|left|Figure 1: web server status page showing example frontend equipments]]
<div style="clear: both"></div> 

Rather than using odbedit, you can view the contents of these equipments by clicking on the ODB button, then on {{Odbpath|path=/Equipment}}, then {{Odbpath|path=Trigger}} or {{Odbpath|path=Scaler}}. See [[ODB Page]] for more information.

=== Start a run ===
Start a run by pressing the Start button on the [[Status Page]].
The Equipment display on the Status Page will now show some event statistics (Figure 2)
'''Click thumbnail to enlarge'''
[[File:sample_frontend_status.png|thumb|left|Figure 2: Equipments shown on Status Page showing event statistics]]
<div style="clear: both"></div> 

The frontend will show an event display that will update when a run is started, e.g.

Sample Frontend connected to <local>. Press "!" to exit 18:48:2546:08
==========================================================================
Run status: Running Run number 2 |/ /
===========================================================================
Equipment Status Events Events/sec Rate[B/s] ODB->FE FE->OD
---------------------------------------------------------------------------
Trigger OK 13948 99.0 5413.0 0 140
Scaler OK 15 0.3 15.4 0 15

The [[Status Page]] will also show events being generated (click the refresh button if necessary).

=== Event Dump ===
While a run is in progress, the midas application [[mdump]] will provide you an event dump of the collected data from the running frontend, e.g.

mhostpc> mdump -l 50
blank
- MIDAS revision: Mon Nov 2 11:50:51 2015 -0800 - 3b66779 -- Enter <!> to Exit ------- Midas Dump ---
------------------------ Event# 1 ------------------------
Evid:0001- Mask:0000- Serial:0- Time:0x567a0c5d- Dsize:40/0x28
#banks:2 - Bank list:-ADC0TDC0-
blank
Bank:ADC0 Length: 8(I*1)/2(I*4)/4(Type) Type:Unsigned Integer*2
1-> 0x0167 0x03c6 0x0069 0x0073
blank
Bank:TDC0 Length: 8(I*1)/2(I*4)/4(Type) Type:Unsigned Integer*2
1-> 0x0051 0x00ff 0x004a 0x00ec

For further data processing/analysis, either the midas analyzer or the rootana can used for data display as well.

=== Create a startup script (localhost) ===
It is useful to create a script that will automatically start all the required clients for the experiment. This is called a startup script. Here, the startup script will be {{File|name=start_daq.sh}} (any name can be used). Create {{Filepath|path=$HOME/online/bin/start_daq.sh}} on the ''experiment host'' as follows (supply the [[mhttpd]] ports if default is not used):

#!/bin/sh
# start_daq.sh
cd $HOME/online
odbedit -c clean
# start mhttpd on default port. (Mongoose https version)
mhttpd -D # you can optionally restrict access to localhost and other specified hosts - see [[mhttpd]]
xterm -e ./frontend &
xterm -e ./analyzer &
mlogger -D
#end file

Before running this script, you will need to shutdown any running clients:
mhostpc> odbedit -c "sh all"
or you can use the {{Utility|name=mhttpd}} [[Programs Page]] to shut them down.

The script will start them as daemons.
By running the startup script start_daq.sh, several MIDAS applications will be started in sequence:
# Cleanup previous MIDAS application (if any).
# Start the MIDAS web server [mhttpd]
# Start the frontend application in its own xterm (for debugging purpose).
# Start the analyzer application in its own xterm (for debugging purpose).
# Start the MIDAS Data logger [mlogger]
[mhostpc] sh ./start_daq

You may wish to modify the script to restart missing clients only.

Next [[#Customizing your experiment|customize your experiment]].

== Running with one or more REMOTE frontends ==
These instructions assume you have already followed the instructions to setup an experiment running on the MIDAS experiment host (localhost), i.e. you have setup the MIDAS Environment variables, downloaded MIDAS packages, setup the [[mhttpd]] password and have {{Utility|name=mhttpd}} running successfully.

In the case of '''remote''' frontend(s), the remote cpu(s) typically have access to some or all of the hardware. They might be VMIC cpus running in VME crates for example, connected to the MIDAS Host via Ethernet. They read data from the hardware and send it back to the MIDAS ''experiment host'' to be logged and analyzed.

=== Experimental setup (Remote) ===
These instructions assume that the MIDAS ''experiment host'' computer is 64-bit (if not see [[#Build 32-bit MIDAS libraries|note below]]), and there is one ''remote host'', a 32-bit machine named "rlxhost".

The ''remote host'' mounts the {{Filepath|path=/home}} disk of the ''experiment host'', so that it '''has access to the MIDAS packages''', the {{Filepath|path=$HOME/online}} directory and shares the .cshrc (or .bashrc) script. It is convenient to set up ssh keys (use ssh-keygen) so that the ''remote host'' can be accessed without supplying the password from the ''experiment host''.

If your setup is different, you will have to make changes as appropriate.

=== Build 32-bit MIDAS libraries ===
NOTE

If the MIDAS ''experiment host'' computer is 32-bit (i.e. the same as the ''remote host''), which is a simpler setup, you will have built the 32-bit MIDAS libraries already, and can skip the rest of this section. Note however, that in the .cshrc file in the [[#Modify .cshrc (.bashrc) for Remote host|following section]] you will not need the lines starting "# select 64-bit or 32-bit MIDAS and ROOT". The PATH should contain $MIDASSYS/linux/bin on both experiment host and remote host. This will point to the 32-bit MIDAS libraries.
------------------------------

 
Assuming the MIDAS host computer is 64-bit, you will have already built the 64-bit MIDAS libraries.
In this case, if the remote frontend is 32-bit, you will need to build the 32-bit MIDAS libraries on the 64-bit machine, i.e.

[mhostpc] cd /home/packages/midas
[mhostpc] make linux32 ### build the 32-bit MIDAS libraries on 64-bit machine
[mhostpc] ls -l linux-m32/bin/odbedit ### check that the 32-bit odbedit has been created 
Do not run {{Utility|name=odbedit}} yet on the ''remote host''!

=== Modify .cshrc (.bashrc) for Remote host ===
Add the following to the {{File|name=.cshrc}} (modified appropriately if using {{File|name=.bashrc}}). When running with a REMOTE Host, the application [[mserver]] will be started (using a particular port) on the MIDAS ''experiment host'' machine only. Note that the environment variable MIDAS_SERVER_HOST is defined on the remote machine(s) but NOT on the ''experiment host''.

# setenv MIDAS_HOST mhostpc # substitute your experiment host name
# setenv REMOTE_HOST rlxhost # substitute your remote host name
# setup the MIDAS mserver host
#
switch (`hostname`)
case $MIDAS_HOST*:
unsetenv MIDAS_SERVER_HOST ## MIDAS_SERVER_HOST not defined
breaksw
default:
setenv MIDAS_SERVER_HOST $MIDAS_HOST.triumf.ca:1175 ## for remote host, define MIDAS_SERVER_HOST
## as hostname + domain name + default mserver port
## port must match that of [[mserver]] used in start_daq.sh
endsw
#
# select 64-bit or 32-bit MIDAS and ROOT
#
switch (`uname -i`)
case i386:
setenv ROOTSYS /triumfcs/trshare/olchansk/root/root_v5.28.00_SL55_32 ## or appropriate path
setenv PATH .:$MIDASSYS/linux-m32/bin:$PATH
breaksw
default:
setenv ROOTSYS $HOME/packages/root
setenv PATH .:$MIDASSYS/linux/bin:$PATH
endsw
#
setenv PATH .:$HOME/online/bin:$HOME/packages/roody/bin:$ROOTSYS/bin:$PATH
#

After executing {{File|name=.cshrc}} (or {{File|name=.bashrc}}) or logging out, on the MIDAS host (mhostpc) make sure environment variable MIDAS_SERVER_HOST is NOT defined
[mhostpc] echo $MIDAS_SERVER_HOST
[mhostpc] MIDAS_SERVER_HOST: Undefined variable.

and on the '''''remote host''''', make sure MIDAS_SERVER_HOST IS defined:
[rlxhost] echo $MIDAS_SERVER_HOST
[rlxhost] MIDAS_SERVER_HOST

Also on the ''remote host'', make sure that the correct (32-bit) {{Utility|name=odbedit}} will be used (do not run it yet!)
[rlxhost] which odbedit
/home/johnfoo/packages/midas/linux-m32/bin/odbedit

=== Grant Remote host access permission ===
Give permission for the remote host(s) to access the experiment by following the instructions to [[Security#MIDAS programs on remote machines|allow MIDAS programs on remote machines]]. If the MIDAS web server [[mhttpd]] is running, use [[ODB Page]] to edit the ODB keys listed, otherwise use [[odbedit]] on the ''experiment host'' machine.

=== Start mserver on Experiment Host ===
The application [[mserver]] is not required when running an experiment on the ''experiment host'' (localhost) only, but it '''IS required''' for access from a '''REMOTE''' frontend.
Start [[mserver]] on MIDAS host (default port 1175) or use the "-p" option for a different port.
The port must match the port defined for environment variable MIDAS_SERVER_HOST in {{File|name=.cshrc}} (or {{File|name=.bashrc}}) [[#Modify .cshrc (.bashrc) for Remote host|above]].
[mhostpc] mserver -D

The environment variables on the REMOTE host should be:

[rlxhost] printenv | grep MIDAS
MIDASSYS=/home/e777/packages/midas
MIDAS_EXPT_NAME=e777
MIDAS_SERVER_HOST=mhostpc.triumf.ca:1175
(a different port may be used - see above)

=== Check access to ODB from Remote host ===
Run {{Utility|name=odbedit}} on the remote host to check access to ODB.
[rlxhost] odbedit
[mhostpc.triumf.ca:1175:e777:Stopped]/>ls 
If it does not work correctly, check that
* MIDAS_SERVER_HOST is defined on REMOTE host with same port as [[mserver]]
* mserver is running on MIDAS host with the correct port
* MIDAS_SERVER_HOST is NOT defined on MIDAS host
* remote access permission to REMOTE host has been granted as described above.

=== Build the analyzer on Experiment Host ===
The analyzer will be built and run on the MIDAS ''experiment host'' machine.
Select an example from $MIDASSYS/examples/ (see [[#Frontend and Analyzer (localhost)]]) and copy it to $HOME/online on the MIDAS host machine. Since we are assuming a 64-bit MIDAS host and a 32-bit remote host, edit the example Makefile so only the analyzer will be built.

=== Build the frontend (32bit) for Remote host ===
The frontend runs on a 32-bit remote host in this example. It therefore must either
* be built 32-bit on the 64-bit MIDAS host with the "-m32" flag
* or built on the REMOTE host
and in either case the [[#Build 32-bit MIDAS libraries|32-bit MIDAS libraries]] used for linking.

To avoid confusion, create a different subdirectory for the frontend, e.g.
[rlxhost] mkdir $HOME/online/fe_32
Copy the Makefile and frontend code from the same example to this directory.
Modify the Makefile as needed, and only build the frontend.

Once built, start the frontend in an xterm on the REMOTE host.
[rlxhost] $HOME/online/fe_32/frontend
Check that it connects successfully to the experiment and the Equipments appear on the [[Status Page]] as described under [[#Frontend and Analyzer (localhost)]]. The Equipment Status (Figure 2) should show that the Equipments are now running on the remote cpu, i.e. "SampleFrontend@rlxhost.triumf.ca" rather than "Sample Frontend@localhost".

=== Run the Analyzer on Experiment host ===
Start the analyzer in an xterm on the MIDAS ''experiment host''
[johnfoo@mhostpc e777] $HOME/online/analyzer

Continue by starting a run and dumping the data as described under [[#Frontend and Analyzer (localhost)]].

=== Create experiment startup script ===
It is useful to create a script that will automatically start all the required clients for the experiment. This is called a startup script (any name can be used). Create {{Filepath|path=$HOME/online/bin/start_daq.sh}} on the ''experiment host'' as follows, (supply the [[mhttpd]] ports if default is not used) :

#!/bin/sh
# start_daq.sh
cd $HOME/online
#
switch (`hostname`)
case $MIDAS_HOST*: ## MIDAS_HOST is assigned in .cshrc (.bashrc)
echo "Good, we are on $MIDAS_HOST"
breaksw
case $REMOTE_HOST*: ## REMOTE_HOST is assigned in .cshrc (.bashrc)
echo "startup script should be executed on $MIDAS_HOST"
endsw
#
odbedit -c clean
# start mhttpd on default port (Mongoose https version)
mhttpd -D # you can optionally restrict access to localhost and/or other specified hosts - see [[mhttpd]]
# start mserver on default port (use argument -p to use a different port)
mserver -D # start mserver on default port (1175) or a different port using -p option - see [[mserver]]
# the port must match that of MIDAS_SERVER_HOST in .cshrc (.bashrc) - see above
# access must be specifically allowed - see [[#Grant Remote host access permission||above]]
xterm -e ./analyzer &
## start frontend on remote host
#ssh $REMOTE_HOST $HOME/online/bin/start_frontend -O >& $HOME/online/bin/start_frontend.log &
mlogger -D
#end file

Note that command to start the frontend on the remote host is commented out. It can be uncommented once the start_frontend script is written and tested.

Shut down any clients that are running already by running odbedit on the MIDAS ''experiment host'' and execute the new startup script {{File|name=start_daq.sh}} (make it executable if desired)

[mhostpc] odbedit
[local:e777:S]/>sh all
[mhostpc] ./start_daq.sh 
and check that {{Utility|name=odbedit}} still works on the ''REMOTE host''

[rlxhost] odbedit
[mhostpc.triumf.ca:1175:e777:Stopped]/>ls 

You may wish to modify the script to restart missing clients only.

=== Create script to start frontend ===
The startup file {{File|name=start_daq.sh}} invoked a script {{File|name=start_frontend}} to start the remote frontend. This line was commented out (see [[#Create experiment startup script|above]]). Once the frontend is working, create a script to start the frontend (e.g. file {{Filepath|path=$HOME/online/bin/start_frontend}}).

#!/bin/tcsh
# Script to start frontend running on $REMOTE_HOST
#
# This script runs on the REMOTE HOST
#
echo "starting frontend for experiment $MIDAS_EXPT_NAME "
xterm -geometry 150x50+800+0 -fg white -bg blue -title "Frontend" -e "$HOME/online/fe_32/frontend" &
echo "Frontend task has been started"
exit

Check that this script works by running it on the REMOTE host. If successful,
run the commented command in an xterm on the ''experiment host''
[mhostpc] ssh $REMOTE_HOST $HOME/online/bin/start_frontend -O >& $HOME/online/bin/start_frontend.log & 
This command relies on password-less access set up with SSH keys (see [[#Experimental setup (Remote)]]).
If successful, uncomment line in {{File|name=start_daq.sh}}, and run the script to check that it can start the remote client(s) from the MIDAS ''experiment host'' (first shutting down all running clients) :

[mhostpc] odbedit -c "sh all"
[mhostpc] ./start_daq.sh 

Next [[#Customizing your experiment|customize your experiment]].

== Customizing your experiment ==

Here are some of the more common options provided for users to customize their experiments according to their own preferences:

; Write frontend(s) to read out and control your hardware
: Modify the example frontend and Makefile (from $MIDASSYS/examples/) you have already copied and built so it accesses your hardware.

: The example Frontend.c is an example of [[Frontend user code]]. You can find documentation about the frontend structure under [[Frontend Application]], [[Frontend Operation]] and [[Frontend user code]]. You may need to set up some [[Event Notification (Hot-Link)|Hot-Links]] in the frontend. The MIDAS package includes device drivers for some of the more common hardware. See [[MIDAS Driver Library]] for more information.

; Modify analyzer(s) for your data
: Modify the example analyzer code as necessary to analyze the data from your frontends.

; Slow Controls
: Control your slow-controls equipment with slow-control frontend(s) optionally using the [[#MIDAS Package Installation|MSCB]] system.

; Data logger
: Set up the data logger [[mlogger]] to log data to a storage device. Many options are available - see [[Data Logger]], [[mlogger]] and [[/Logger ODB tree]] for instructions.

; Allow programs to be restarted from the [[Programs Page]]
: See mhttpd [[Programs Page]] and [[/Programs ODB tree]]

; Write scripts which run at start and end of run
: See [[/Programs ODB tree#Execute on start run]] and [[/Programs ODB tree#Execute on stop run]]

; Create a shutdown script
: Create a script similar to the startup script to shutdown the experiment

; Lazy logger
: Set up the [[lazylogger]] for archiving data files for storage.

; Create edit-on-start parameters
: See [[Edit-on-start Parameters]]

; Set up the alarm system
: See [[Alarm System]] and [[Alarms Page]]

; Set up history system
: See [[History System]] and [[History Page]]

; Set up the Electronic logbook (Elog)
: See [[Elog]] for details

; Set up the Sequencer
: The [[Sequencer Page#Sequencer]] can be used to run a sequence of runs, changing parameters as needed.

; Write a webserver Custom page
: To better control and monitor your experiment, you may want to write a [[Custom Page]].

[[Category:Installation]]

Quickstart Linux

2016-11-08T01:21:06Z

Olchansk: /* Clients run on Localhost only */

{{Pagelinks}}
== Introduction ==
This quickstart shows you how to install the MIDAS packages on a linux box and create a '''MIDAS experiment'''.
The linux box will be called the MIDAS '''''experiment host'''''.

==Installation==
Log on to the ''[[#Introduction|Experiment Host]]'' and decide on a name for the MIDAS experiment (the ''experiment name'').
In these instructions, the MIDAS packages and experiment will be installed under the username "johnfoo".
The ODB and other shared memory buffers for the MIDAS experiment will reside on the ''experiment host'' (localhost).
MIDAS and the other required packages will be installed under directory {{Filepath|path=$HOME/packages}}. The MIDAS ''experiment directory'' will be {{Filepath|path=$HOME/online}} and the MIDAS ''experiment name'' will be "e777". The name of the ''experiment host'' will be "mhostpc".

;NOTE
: The user should substitute directory paths for the MIDAS packages and experiment directory as required, and names for ''experiment host'', username, ''experiment name'' appropriate to his/her own setup.

== Experimental setup ==
These instructions describe setting up an experiment that will run ONLY on the MIDAS ''experiment host'' computer (i.e. '''localhost only'''). In this case, the MIDAS ''experiment host'' has access to any hardware required, and all frontends, logger, analyzer etc. run on the one computer.

Modifications to run an experiment with additional frontend(s) running '''remotely''' (i.e. on remote cpu(s)) will also be described.
Follow the instructions for setting up a localhost-only experiment until indicated.

In both cases, the main MIDAS applications (e.g. [[mlogger]], [[mhttpd]]) run on the MIDAS ''experiment host''.

==Environment Variables==
The following Environment variables should to be added to the {{File|name=.cshrc}} or {{File|name=.bashrc}} file (depending on your shell) in the {{Filepath|path=$HOME}} directory on the ''experiment host'', so that they will be defined at login. '''Substitute appropriate values for your own setup'''.

{| class="wikitable"
|-
! csh !! bash !! comment
|-
| setenv GIT_EDITOR "emacs -nw" || export GIT_EDITOR="emacs -nw" ||
|-
| setenv MIDASSYS $HOME/packages/midas || export MIDASSYS=$HOME/packages/midas ||Base directory of the MIDAS package
|-
| setenv MIDAS_EXPTAB $HOME/online/exptab || export MIDAS_EXPTAB=$HOME/online/exptab || MIDAS experiment table
|-
|-
| setenv MIDAS_EXPT_NAME e777 || export MIDAS_EXPT_NAME=e777 || MIDAS experiment name
|-
| setenv PATH .:$MIDASSYS/linux/bin:$PATH || export PATH=$PATH:$MIDASSYS/linux/bin || path
|}

Logout and login again, or source {{File|name=.cshrc}} (source {{File|name=.bashrc}}) for the changes to take effect.
;NOTE
: See [[MIDAS environment variables]] for a list of environment variables used by MIDAS.

== ROOT Package Installation ==
For full MIDAS operation, ROOT is needed for the data logging and analysis packages. It needed to build the example analyzer in this Quickstart. However, ROOT is not essential to run MIDAS. If ROOT is NOT installed, the environment variable ROOTSYS should be undefined.

If ROOT is needed, see [[Setup MIDAS experiment#Install ROOT|Install ROOT]].

== MIDAS Package Installation ==
The MIDAS package will be installed on the ''experiment host'' in the directory given by MIDASSYS - see [[#Environment Variables]].

Install the MIDAS and other package(s) from the MIDAS git repository in an xterm as follows:

[mhostpc] mkdir $HOME/packages
[mhostpc] cd $HOME/packages
[mhostpc] git clone https://bitbucket.org/tmidas/midas
[mhostpc] git clone https://bitbucket.org/tmidas/mxml

Install the MSCB package '''only''' if you will be using the MSCB system for slow-controls (see [[MSCB Page]]):

[mhostpc] git clone https://bitbucket.org/tmidas/mscb ## MSCB package
[mhostpc] cd mscb
[mhostpc] make ## Build the MSCB package
[mhostpc] cd ../

The MIDAS Makefile will auto-detect whether MSCB, ROOT, MYSQL, SQLITE, ODBC etc. are available (or have been installed). If you need these features, make sure they are available before building the MIDAS package.

[mhostpc] cd midas
[mhostpc] make ## Build the MIDAS package
[mhostpc] ls -l linux/bin/odbedit ### check that odbedit has been created (do not run it yet)


In case of problems see
* [[Installation#Installing MIDAS|bitbucket is down]] or
* [[Installation/Compilation problems#SSL certificate errors|SSL certificate errors]].

==ROOTANA Package Installation==
If ROOTANA is needed,

[mhostpc] cd $HOME/packages
[mhostpc] git clone https://bitbucket.org/tmidas/rootana
[mhostpc] cd rootana
[mhostpc] make

==ROODY Package Installation==
If ROODY is needed,

[mhostpc] cd $HOME/packages
[mhostpc] git clone https://bitbucket.org/tmidas/roody
[mhostpc] cd roody
[mhostpc] make


== Create the Experiment file exptab ==
In this example, the experimental directory is {{Filepath|path=$HOME/online}}. Create this directory on the ''experiment host'' (localhost):

[mhostpc] mkdir $HOME/online
[mhostpc] cd $HOME/online
Then create the [[exptab]] file {{Filepath|path=$HOME/online/exptab}} containing the three parameters:
# the experiment name
# location of MIDAS shared memory buffers
# username
as follows (using parameters for your own experiment):

[mhostpc] cat > exptab
e777 /home/johnfoo/online johnfoo

The path of the {{File|name=exptab}} file is given by MIDAS_EXPTAB - see [[#Environment Variables]].

== Create shared memory files ==
At this point you should be able to run [[odbedit]] on the ''experiment host'' (localhost). The first time {{Utility|name=odbedit}} is run, it will create the required {{File|name=.*.SHM}} files in the MIDAS experiment directory {{Filepath|path=$HOME/online}}. These are the saved files for the MIDAS shared memory.

The default directory {{Filepath|path=$HOME/online}} will contain the MIDAS messages file ({{File|name=midas.log}}) and
any data files you may create. If you want an alternative location for the data files see [[#Customizing your experiment]].

Run {{Utility|name=odbedit}}, and type {{Odbedit cmd|cmd=ls}} to list the default directories.
[mhostpc] odbedit
[local:e777:S]/>ls
Experiment
System
Programs
Logger
Runinfo
Alarms

== Run the MIDAS Web Server ==

Recommended for production systems with maximum security is to run mhttpd behind an industry-standard SSL/https proxy with password protection.

To do this using apache httpd on CentOS-7, do this:
* configure apache httpd per instructions at https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Configure_HTTPS_server_.28CentOS7.29
* run odbedit, change values of
** "/experiment/http redirect to https" set to "n"
** "/experiment/midas https port" set to 0
* run mhttpd
* use web browser to connect to https://mhostpc/
* enter the user name and password configured in apache httpd
* the midas status page should open

For simplified installation, one can use standalone mhttpd with built-in https and password protection. This configuration
is only as secure as the underlying mongoose and OpenSSL web server and encryption packages.

Many current operating systems implement a built-in firewall which will block outside access to mhttpd.

On CentOS-7, so this: https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29

Start [[mhttpd]] on the ''experiment host'' (localhost) like this:
[mhostpc] mhttpd
You will get the following messages:

Mongoose web server will listen on ports "8080r,8443s"
[mhttpd,ERROR] [mhttpd.cxx:17892:mongoose,ERROR] cannot find SSL certificate file "/home/agdaq/online/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17893:mongoose,ERROR] please create SSL certificate file: openssl req -new -nodes -newkey rsa:2048 -sha256 -out ssl_cert.csr -keyout ssl_cert.key; openssl x509 -req -days 365 -sha256 -in ssl_cert.csr -signkey ssl_cert.key -out ssl_cert.pem; cat ssl_cert.key >> ssl_cert.pem
could not start the mongoose web server, see messages and midas.log, bye!

Create a self-signed certificate suitable for initial testing by executing the commands printed by mhttpd.

For production use, you should create a properly signed certificate, see [[Mhttpd#Create an SSL certificate|create your own SSL certificate]] or you should run mhttpd behind an SSL/https proxy (apache httpd).

Run mhttpd again.

You will get the following messages:

Mongoose web server will listen on ports "8080r,8443s"
Mongoose web server will use SSL certificate file "/home/johnfoo/packages/midas/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17633:mongoose,ERROR] mongoose web server cannot find password file "/home/johnfoo/online/htpasswd.txt"
[mhttpd,ERROR] [mhttpd.cxx:17634:mongoose,ERROR] please create password file: htdigest -c /home/johnfoo/online/htpasswd.txt Default midas
could not start the mongoose web server, see messages and midas.log, bye!

Create the password file by following the instructions printed by mhttpd. The http digest domain name is the experiment name, suggested default user name is "midas". You will be asked to type in a password. (htdigest is part of the httpd-tools package)


[mhostpc] htdigest -c /home/johnfoo/online/htpasswd.txt exptname midas
Adding password for midas in realm exptname.
New password:
Re-type new password:

It is a good idea to set the password file {{Filepath|path=htpasswd.txt}} readable and writable by owner only.

Now restart {{Utility|name=mhttpd}}
[mhostpc] mhttpd
Mongoose web server will listen on ports "8080r,8443s" **see note
Mongoose web server will use SSL certificate file "/home/suz/packages/midas/ssl_cert.pem"
Mongoose web server will use authentication realm "Default", password file "./htpasswd.txt"

Now point a web browser running on the same host computer (localhost) to https://localhost:8443
If the web browser is running on a different computer, go to URL of the form

https://mhostpc.triumf.ca:8443 (substitute your host machine name and domain for "mhostpc.triumf.ca")

If you are using the default SSL certificate you will probably get a message: "This Connection is Untrusted". Click "I understand the risks" and add an exception. This is because the test certificate is self-signed. Then confirm an exception.

If instead you get a "connection refused" error, the midas host pc may have the firewall enabled. To make a firewall exception for MIDAS, follow instructions here http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29

You should then see an authentication box asking you for the user name and password. The user name is "midas". Enter the password you just created. The Midas [[Status Page]] should appear with multiple buttons for run control as well as equipment listing (no equipments will be listed as yet) and application listings. Please refer to [[mhttpd]] (the MIDAS Web-based Run Control utility) for further information. You can start and stop runs from the main status page, and use the [[ODB Page]] to access the database (ODB).

; Note
: Default ports of 8080 and 8443 are used by [[mhttpd]]. To use some other ports, change the values of ODB "/experiment/midas http port" and "midas https port".

== Run the MIDAS logger ==
Start the MIDAS logger [[mlogger]] on the ''experiment host'' (localhost) :
[mhostpc] mlogger
This should start without error. It is usually run as a daemon, however in this case it is run in a terminal to check for errors. Starting the midas logger [[mlogger]] will automatically create more keys in the {{Odbpath|path=/Logger}} ODB tree.

From the MIDAS status page, go to the "Programs" page, then the "Logger" page and set ODB variables:
* Required -> set to "y"
* Start command -> set to "mlogger -D"
* Alarm Class -> set to "Alarm"

To test this, go to the "Programs" page, press "stop logger", then "start logger".

Start a run, wait 10 seconds, stop the run.

From the MIDAS "Status" page, go to the "logging channels -> channel #0" and set ODB variables:
* Output -> "FILE"

Start a run, go to the "Status" page, observe that the output file is written with gzip compression and has the "mid.gz" extension.

Stop the run.

== Run test frontend ==

Run fetest

On the MIDAS "Status" page...

== Clients run on Localhost only ==
;NOTE:
* If creating a MIDAS experiment with a '''REMOTE''' frontend, '''continue by following the instructions''' [[#Running with one or more REMOTE frontends]].
* If all clients are running on the Experiment Host (i.e. '''localhost'''), continue with the following instructions:

=== Frontend and Analyzer (localhost) ===

THIS SECTION IS VERY OBSOLETE

There are several examples of [[Frontend user code]] ({{File|name=frontend.c}}) in the MIDAS package available under {{Filepath|path=$MIDASSYS/examples/}}. Choose a suitable example from this directory that you can later modify for your own particular setup (e.g. for a slow control, choose the {{Filepath|path=../slowcont/}} example).

A '''frontend''' is a program that usually reads data from the hardware and sends it to a buffer to be logged and analyzed. You can find documentation about the frontend structure under [[Frontend Application]], [[Frontend Operation]] and [[Frontend user code]].

The example chosen here is from {{Filepath|path=$MIDASSYS/examples/experiment/}}. It does '''not need any hardware''' and produces simulated events.

On the MIDAS ''experiment host'' (localhost), copy the example to the experiment directory and build it.

[mhostpc] cd $HOME/online
[mhostpc] cp $MIDASSYS/examples/experiment/* .
[mhostpc] make
The analyzer will only build if ROOT has previously been installed.
At this point the frontend and the analyzer should be ready if no errors were generated during the build. Try the frontend and analyzer by starting them in xterms.

[mhostpc] frontend
[mhostpc] analyzer

=== Equipment ODB tree ===
Starting the frontend will automatically create the {{Odbpath|path=/Equipment}} ODB tree and one or more equipments. In this case, the equipments "trigger" and "scaler" have been defined by code in {{File|name=frontend.c}} and created the first time frontend runs.

[mhostpc] odbedit
[local:e777:S]/>ls /equipment
Trigger
Scaler 

View the MIDAS main status web page (by pointing your browser to https://localhost:8443/ or appropriate port if not using the default).
Observe on the [[Status Page]] that the two Equipments (Trigger, Scaler) have appeared in the Equipment section (Figure 1) and are coloured green.
 
'''Click thumbnail to enlarge'''
[[File:quickstart_status_page.png|thumb|left|Figure 1: web server status page showing example frontend equipments]]
<div style="clear: both"></div> 

Rather than using odbedit, you can view the contents of these equipments by clicking on the ODB button, then on {{Odbpath|path=/Equipment}}, then {{Odbpath|path=Trigger}} or {{Odbpath|path=Scaler}}. See [[ODB Page]] for more information.

=== Start a run ===
Start a run by pressing the Start button on the [[Status Page]].
The Equipment display on the Status Page will now show some event statistics (Figure 2)
'''Click thumbnail to enlarge'''
[[File:sample_frontend_status.png|thumb|left|Figure 2: Equipments shown on Status Page showing event statistics]]
<div style="clear: both"></div> 

The frontend will show an event display that will update when a run is started, e.g.

Sample Frontend connected to <local>. Press "!" to exit 18:48:2546:08
==========================================================================
Run status: Running Run number 2 |/ /
===========================================================================
Equipment Status Events Events/sec Rate[B/s] ODB->FE FE->OD
---------------------------------------------------------------------------
Trigger OK 13948 99.0 5413.0 0 140
Scaler OK 15 0.3 15.4 0 15

The [[Status Page]] will also show events being generated (click the refresh button if necessary).

=== Event Dump ===
While a run is in progress, the midas application [[mdump]] will provide you an event dump of the collected data from the running frontend, e.g.

mhostpc> mdump -l 50
blank
- MIDAS revision: Mon Nov 2 11:50:51 2015 -0800 - 3b66779 -- Enter <!> to Exit ------- Midas Dump ---
------------------------ Event# 1 ------------------------
Evid:0001- Mask:0000- Serial:0- Time:0x567a0c5d- Dsize:40/0x28
#banks:2 - Bank list:-ADC0TDC0-
blank
Bank:ADC0 Length: 8(I*1)/2(I*4)/4(Type) Type:Unsigned Integer*2
1-> 0x0167 0x03c6 0x0069 0x0073
blank
Bank:TDC0 Length: 8(I*1)/2(I*4)/4(Type) Type:Unsigned Integer*2
1-> 0x0051 0x00ff 0x004a 0x00ec

For further data processing/analysis, either the midas analyzer or the rootana can used for data display as well.

=== Create a startup script (localhost) ===
It is useful to create a script that will automatically start all the required clients for the experiment. This is called a startup script. Here, the startup script will be {{File|name=start_daq.sh}} (any name can be used). Create {{Filepath|path=$HOME/online/bin/start_daq.sh}} on the ''experiment host'' as follows (supply the [[mhttpd]] ports if default is not used):

#!/bin/sh
# start_daq.sh
cd $HOME/online
odbedit -c clean
# start mhttpd on default port. (Mongoose https version)
mhttpd -D # you can optionally restrict access to localhost and other specified hosts - see [[mhttpd]]
xterm -e ./frontend &
xterm -e ./analyzer &
mlogger -D
#end file

Before running this script, you will need to shutdown any running clients:
mhostpc> odbedit -c "sh all"
or you can use the {{Utility|name=mhttpd}} [[Programs Page]] to shut them down.

The script will start them as daemons.
By running the startup script start_daq.sh, several MIDAS applications will be started in sequence:
# Cleanup previous MIDAS application (if any).
# Start the MIDAS web server [mhttpd]
# Start the frontend application in its own xterm (for debugging purpose).
# Start the analyzer application in its own xterm (for debugging purpose).
# Start the MIDAS Data logger [mlogger]
[mhostpc] sh ./start_daq

You may wish to modify the script to restart missing clients only.

Next [[#Customizing your experiment|customize your experiment]].

== Running with one or more REMOTE frontends ==
These instructions assume you have already followed the instructions to setup an experiment running on the MIDAS experiment host (localhost), i.e. you have setup the MIDAS Environment variables, downloaded MIDAS packages, setup the [[mhttpd]] password and have {{Utility|name=mhttpd}} running successfully.

In the case of '''remote''' frontend(s), the remote cpu(s) typically have access to some or all of the hardware. They might be VMIC cpus running in VME crates for example, connected to the MIDAS Host via Ethernet. They read data from the hardware and send it back to the MIDAS ''experiment host'' to be logged and analyzed.

=== Experimental setup (Remote) ===
These instructions assume that the MIDAS ''experiment host'' computer is 64-bit (if not see [[#Build 32-bit MIDAS libraries|note below]]), and there is one ''remote host'', a 32-bit machine named "rlxhost".

The ''remote host'' mounts the {{Filepath|path=/home}} disk of the ''experiment host'', so that it '''has access to the MIDAS packages''', the {{Filepath|path=$HOME/online}} directory and shares the .cshrc (or .bashrc) script. It is convenient to set up ssh keys (use ssh-keygen) so that the ''remote host'' can be accessed without supplying the password from the ''experiment host''.

If your setup is different, you will have to make changes as appropriate.

=== Build 32-bit MIDAS libraries ===
NOTE

If the MIDAS ''experiment host'' computer is 32-bit (i.e. the same as the ''remote host''), which is a simpler setup, you will have built the 32-bit MIDAS libraries already, and can skip the rest of this section. Note however, that in the .cshrc file in the [[#Modify .cshrc (.bashrc) for Remote host|following section]] you will not need the lines starting "# select 64-bit or 32-bit MIDAS and ROOT". The PATH should contain $MIDASSYS/linux/bin on both experiment host and remote host. This will point to the 32-bit MIDAS libraries.
------------------------------

 
Assuming the MIDAS host computer is 64-bit, you will have already built the 64-bit MIDAS libraries.
In this case, if the remote frontend is 32-bit, you will need to build the 32-bit MIDAS libraries on the 64-bit machine, i.e.

[mhostpc] cd /home/packages/midas
[mhostpc] make linux32 ### build the 32-bit MIDAS libraries on 64-bit machine
[mhostpc] ls -l linux-m32/bin/odbedit ### check that the 32-bit odbedit has been created 
Do not run {{Utility|name=odbedit}} yet on the ''remote host''!

=== Modify .cshrc (.bashrc) for Remote host ===
Add the following to the {{File|name=.cshrc}} (modified appropriately if using {{File|name=.bashrc}}). When running with a REMOTE Host, the application [[mserver]] will be started (using a particular port) on the MIDAS ''experiment host'' machine only. Note that the environment variable MIDAS_SERVER_HOST is defined on the remote machine(s) but NOT on the ''experiment host''.

# setenv MIDAS_HOST mhostpc # substitute your experiment host name
# setenv REMOTE_HOST rlxhost # substitute your remote host name
# setup the MIDAS mserver host
#
switch (`hostname`)
case $MIDAS_HOST*:
unsetenv MIDAS_SERVER_HOST ## MIDAS_SERVER_HOST not defined
breaksw
default:
setenv MIDAS_SERVER_HOST $MIDAS_HOST.triumf.ca:1175 ## for remote host, define MIDAS_SERVER_HOST
## as hostname + domain name + default mserver port
## port must match that of [[mserver]] used in start_daq.sh
endsw
#
# select 64-bit or 32-bit MIDAS and ROOT
#
switch (`uname -i`)
case i386:
setenv ROOTSYS /triumfcs/trshare/olchansk/root/root_v5.28.00_SL55_32 ## or appropriate path
setenv PATH .:$MIDASSYS/linux-m32/bin:$PATH
breaksw
default:
setenv ROOTSYS $HOME/packages/root
setenv PATH .:$MIDASSYS/linux/bin:$PATH
endsw
#
setenv PATH .:$HOME/online/bin:$HOME/packages/roody/bin:$ROOTSYS/bin:$PATH
#

After executing {{File|name=.cshrc}} (or {{File|name=.bashrc}}) or logging out, on the MIDAS host (mhostpc) make sure environment variable MIDAS_SERVER_HOST is NOT defined
[mhostpc] echo $MIDAS_SERVER_HOST
[mhostpc] MIDAS_SERVER_HOST: Undefined variable.

and on the '''''remote host''''', make sure MIDAS_SERVER_HOST IS defined:
[rlxhost] echo $MIDAS_SERVER_HOST
[rlxhost] MIDAS_SERVER_HOST

Also on the ''remote host'', make sure that the correct (32-bit) {{Utility|name=odbedit}} will be used (do not run it yet!)
[rlxhost] which odbedit
/home/johnfoo/packages/midas/linux-m32/bin/odbedit

=== Grant Remote host access permission ===
Give permission for the remote host(s) to access the experiment by following the instructions to [[Security#MIDAS programs on remote machines|allow MIDAS programs on remote machines]]. If the MIDAS web server [[mhttpd]] is running, use [[ODB Page]] to edit the ODB keys listed, otherwise use [[odbedit]] on the ''experiment host'' machine.

=== Start mserver on Experiment Host ===
The application [[mserver]] is not required when running an experiment on the ''experiment host'' (localhost) only, but it '''IS required''' for access from a '''REMOTE''' frontend.
Start [[mserver]] on MIDAS host (default port 1175) or use the "-p" option for a different port.
The port must match the port defined for environment variable MIDAS_SERVER_HOST in {{File|name=.cshrc}} (or {{File|name=.bashrc}}) [[#Modify .cshrc (.bashrc) for Remote host|above]].
[mhostpc] mserver -D

The environment variables on the REMOTE host should be:

[rlxhost] printenv | grep MIDAS
MIDASSYS=/home/e777/packages/midas
MIDAS_EXPT_NAME=e777
MIDAS_SERVER_HOST=mhostpc.triumf.ca:1175
(a different port may be used - see above)

=== Check access to ODB from Remote host ===
Run {{Utility|name=odbedit}} on the remote host to check access to ODB.
[rlxhost] odbedit
[mhostpc.triumf.ca:1175:e777:Stopped]/>ls 
If it does not work correctly, check that
* MIDAS_SERVER_HOST is defined on REMOTE host with same port as [[mserver]]
* mserver is running on MIDAS host with the correct port
* MIDAS_SERVER_HOST is NOT defined on MIDAS host
* remote access permission to REMOTE host has been granted as described above.

=== Build the analyzer on Experiment Host ===
The analyzer will be built and run on the MIDAS ''experiment host'' machine.
Select an example from $MIDASSYS/examples/ (see [[#Frontend and Analyzer (localhost)]]) and copy it to $HOME/online on the MIDAS host machine. Since we are assuming a 64-bit MIDAS host and a 32-bit remote host, edit the example Makefile so only the analyzer will be built.

=== Build the frontend (32bit) for Remote host ===
The frontend runs on a 32-bit remote host in this example. It therefore must either
* be built 32-bit on the 64-bit MIDAS host with the "-m32" flag
* or built on the REMOTE host
and in either case the [[#Build 32-bit MIDAS libraries|32-bit MIDAS libraries]] used for linking.

To avoid confusion, create a different subdirectory for the frontend, e.g.
[rlxhost] mkdir $HOME/online/fe_32
Copy the Makefile and frontend code from the same example to this directory.
Modify the Makefile as needed, and only build the frontend.

Once built, start the frontend in an xterm on the REMOTE host.
[rlxhost] $HOME/online/fe_32/frontend
Check that it connects successfully to the experiment and the Equipments appear on the [[Status Page]] as described under [[#Frontend and Analyzer (localhost)]]. The Equipment Status (Figure 2) should show that the Equipments are now running on the remote cpu, i.e. "SampleFrontend@rlxhost.triumf.ca" rather than "Sample Frontend@localhost".

=== Run the Analyzer on Experiment host ===
Start the analyzer in an xterm on the MIDAS ''experiment host''
[johnfoo@mhostpc e777] $HOME/online/analyzer

Continue by starting a run and dumping the data as described under [[#Frontend and Analyzer (localhost)]].

=== Create experiment startup script ===
It is useful to create a script that will automatically start all the required clients for the experiment. This is called a startup script (any name can be used). Create {{Filepath|path=$HOME/online/bin/start_daq.sh}} on the ''experiment host'' as follows, (supply the [[mhttpd]] ports if default is not used) :

#!/bin/sh
# start_daq.sh
cd $HOME/online
#
switch (`hostname`)
case $MIDAS_HOST*: ## MIDAS_HOST is assigned in .cshrc (.bashrc)
echo "Good, we are on $MIDAS_HOST"
breaksw
case $REMOTE_HOST*: ## REMOTE_HOST is assigned in .cshrc (.bashrc)
echo "startup script should be executed on $MIDAS_HOST"
endsw
#
odbedit -c clean
# start mhttpd on default port (Mongoose https version)
mhttpd -D # you can optionally restrict access to localhost and/or other specified hosts - see [[mhttpd]]
# start mserver on default port (use argument -p to use a different port)
mserver -D # start mserver on default port (1175) or a different port using -p option - see [[mserver]]
# the port must match that of MIDAS_SERVER_HOST in .cshrc (.bashrc) - see above
# access must be specifically allowed - see [[#Grant Remote host access permission||above]]
xterm -e ./analyzer &
## start frontend on remote host
#ssh $REMOTE_HOST $HOME/online/bin/start_frontend -O >& $HOME/online/bin/start_frontend.log &
mlogger -D
#end file

Note that command to start the frontend on the remote host is commented out. It can be uncommented once the start_frontend script is written and tested.

Shut down any clients that are running already by running odbedit on the MIDAS ''experiment host'' and execute the new startup script {{File|name=start_daq.sh}} (make it executable if desired)

[mhostpc] odbedit
[local:e777:S]/>sh all
[mhostpc] ./start_daq.sh 
and check that {{Utility|name=odbedit}} still works on the ''REMOTE host''

[rlxhost] odbedit
[mhostpc.triumf.ca:1175:e777:Stopped]/>ls 

You may wish to modify the script to restart missing clients only.

=== Create script to start frontend ===
The startup file {{File|name=start_daq.sh}} invoked a script {{File|name=start_frontend}} to start the remote frontend. This line was commented out (see [[#Create experiment startup script|above]]). Once the frontend is working, create a script to start the frontend (e.g. file {{Filepath|path=$HOME/online/bin/start_frontend}}).

#!/bin/tcsh
# Script to start frontend running on $REMOTE_HOST
#
# This script runs on the REMOTE HOST
#
echo "starting frontend for experiment $MIDAS_EXPT_NAME "
xterm -geometry 150x50+800+0 -fg white -bg blue -title "Frontend" -e "$HOME/online/fe_32/frontend" &
echo "Frontend task has been started"
exit

Check that this script works by running it on the REMOTE host. If successful,
run the commented command in an xterm on the ''experiment host''
[mhostpc] ssh $REMOTE_HOST $HOME/online/bin/start_frontend -O >& $HOME/online/bin/start_frontend.log & 
This command relies on password-less access set up with SSH keys (see [[#Experimental setup (Remote)]]).
If successful, uncomment line in {{File|name=start_daq.sh}}, and run the script to check that it can start the remote client(s) from the MIDAS ''experiment host'' (first shutting down all running clients) :

[mhostpc] odbedit -c "sh all"
[mhostpc] ./start_daq.sh 

Next [[#Customizing your experiment|customize your experiment]].

== Customizing your experiment ==

Here are some of the more common options provided for users to customize their experiments according to their own preferences:

; Write frontend(s) to read out and control your hardware
: Modify the example frontend and Makefile (from $MIDASSYS/examples/) you have already copied and built so it accesses your hardware.

: The example Frontend.c is an example of [[Frontend user code]]. You can find documentation about the frontend structure under [[Frontend Application]], [[Frontend Operation]] and [[Frontend user code]]. You may need to set up some [[Event Notification (Hot-Link)|Hot-Links]] in the frontend. The MIDAS package includes device drivers for some of the more common hardware. See [[MIDAS Driver Library]] for more information.

; Modify analyzer(s) for your data
: Modify the example analyzer code as necessary to analyze the data from your frontends.

; Slow Controls
: Control your slow-controls equipment with slow-control frontend(s) optionally using the [[#MIDAS Package Installation|MSCB]] system.

; Data logger
: Set up the data logger [[mlogger]] to log data to a storage device. Many options are available - see [[Data Logger]], [[mlogger]] and [[/Logger ODB tree]] for instructions.

; Allow programs to be restarted from the [[Programs Page]]
: See mhttpd [[Programs Page]] and [[/Programs ODB tree]]

; Write scripts which run at start and end of run
: See [[/Programs ODB tree#Execute on start run]] and [[/Programs ODB tree#Execute on stop run]]

; Create a shutdown script
: Create a script similar to the startup script to shutdown the experiment

; Lazy logger
: Set up the [[lazylogger]] for archiving data files for storage.

; Create edit-on-start parameters
: See [[Edit-on-start Parameters]]

; Set up the alarm system
: See [[Alarm System]] and [[Alarms Page]]

; Set up history system
: See [[History System]] and [[History Page]]

; Set up the Electronic logbook (Elog)
: See [[Elog]] for details

; Set up the Sequencer
: The [[Sequencer Page#Sequencer]] can be used to run a sequence of runs, changing parameters as needed.

; Write a webserver Custom page
: To better control and monitor your experiment, you may want to write a [[Custom Page]].

[[Category:Installation]]

Quickstart Linux

2016-11-08T01:16:33Z

Olchansk: /* Run the MIDAS logger */

{{Pagelinks}}
== Introduction ==
This quickstart shows you how to install the MIDAS packages on a linux box and create a '''MIDAS experiment'''.
The linux box will be called the MIDAS '''''experiment host'''''.

==Installation==
Log on to the ''[[#Introduction|Experiment Host]]'' and decide on a name for the MIDAS experiment (the ''experiment name'').
In these instructions, the MIDAS packages and experiment will be installed under the username "johnfoo".
The ODB and other shared memory buffers for the MIDAS experiment will reside on the ''experiment host'' (localhost).
MIDAS and the other required packages will be installed under directory {{Filepath|path=$HOME/packages}}. The MIDAS ''experiment directory'' will be {{Filepath|path=$HOME/online}} and the MIDAS ''experiment name'' will be "e777". The name of the ''experiment host'' will be "mhostpc".

;NOTE
: The user should substitute directory paths for the MIDAS packages and experiment directory as required, and names for ''experiment host'', username, ''experiment name'' appropriate to his/her own setup.

== Experimental setup ==
These instructions describe setting up an experiment that will run ONLY on the MIDAS ''experiment host'' computer (i.e. '''localhost only'''). In this case, the MIDAS ''experiment host'' has access to any hardware required, and all frontends, logger, analyzer etc. run on the one computer.

Modifications to run an experiment with additional frontend(s) running '''remotely''' (i.e. on remote cpu(s)) will also be described.
Follow the instructions for setting up a localhost-only experiment until indicated.

In both cases, the main MIDAS applications (e.g. [[mlogger]], [[mhttpd]]) run on the MIDAS ''experiment host''.

==Environment Variables==
The following Environment variables should to be added to the {{File|name=.cshrc}} or {{File|name=.bashrc}} file (depending on your shell) in the {{Filepath|path=$HOME}} directory on the ''experiment host'', so that they will be defined at login. '''Substitute appropriate values for your own setup'''.

{| class="wikitable"
|-
! csh !! bash !! comment
|-
| setenv GIT_EDITOR "emacs -nw" || export GIT_EDITOR="emacs -nw" ||
|-
| setenv MIDASSYS $HOME/packages/midas || export MIDASSYS=$HOME/packages/midas ||Base directory of the MIDAS package
|-
| setenv MIDAS_EXPTAB $HOME/online/exptab || export MIDAS_EXPTAB=$HOME/online/exptab || MIDAS experiment table
|-
|-
| setenv MIDAS_EXPT_NAME e777 || export MIDAS_EXPT_NAME=e777 || MIDAS experiment name
|-
| setenv PATH .:$MIDASSYS/linux/bin:$PATH || export PATH=$PATH:$MIDASSYS/linux/bin || path
|}

Logout and login again, or source {{File|name=.cshrc}} (source {{File|name=.bashrc}}) for the changes to take effect.
;NOTE
: See [[MIDAS environment variables]] for a list of environment variables used by MIDAS.

== ROOT Package Installation ==
For full MIDAS operation, ROOT is needed for the data logging and analysis packages. It needed to build the example analyzer in this Quickstart. However, ROOT is not essential to run MIDAS. If ROOT is NOT installed, the environment variable ROOTSYS should be undefined.

If ROOT is needed, see [[Setup MIDAS experiment#Install ROOT|Install ROOT]].

== MIDAS Package Installation ==
The MIDAS package will be installed on the ''experiment host'' in the directory given by MIDASSYS - see [[#Environment Variables]].

Install the MIDAS and other package(s) from the MIDAS git repository in an xterm as follows:

[mhostpc] mkdir $HOME/packages
[mhostpc] cd $HOME/packages
[mhostpc] git clone https://bitbucket.org/tmidas/midas
[mhostpc] git clone https://bitbucket.org/tmidas/mxml

Install the MSCB package '''only''' if you will be using the MSCB system for slow-controls (see [[MSCB Page]]):

[mhostpc] git clone https://bitbucket.org/tmidas/mscb ## MSCB package
[mhostpc] cd mscb
[mhostpc] make ## Build the MSCB package
[mhostpc] cd ../

The MIDAS Makefile will auto-detect whether MSCB, ROOT, MYSQL, SQLITE, ODBC etc. are available (or have been installed). If you need these features, make sure they are available before building the MIDAS package.

[mhostpc] cd midas
[mhostpc] make ## Build the MIDAS package
[mhostpc] ls -l linux/bin/odbedit ### check that odbedit has been created (do not run it yet)


In case of problems see
* [[Installation#Installing MIDAS|bitbucket is down]] or
* [[Installation/Compilation problems#SSL certificate errors|SSL certificate errors]].

==ROOTANA Package Installation==
If ROOTANA is needed,

[mhostpc] cd $HOME/packages
[mhostpc] git clone https://bitbucket.org/tmidas/rootana
[mhostpc] cd rootana
[mhostpc] make

==ROODY Package Installation==
If ROODY is needed,

[mhostpc] cd $HOME/packages
[mhostpc] git clone https://bitbucket.org/tmidas/roody
[mhostpc] cd roody
[mhostpc] make


== Create the Experiment file exptab ==
In this example, the experimental directory is {{Filepath|path=$HOME/online}}. Create this directory on the ''experiment host'' (localhost):

[mhostpc] mkdir $HOME/online
[mhostpc] cd $HOME/online
Then create the [[exptab]] file {{Filepath|path=$HOME/online/exptab}} containing the three parameters:
# the experiment name
# location of MIDAS shared memory buffers
# username
as follows (using parameters for your own experiment):

[mhostpc] cat > exptab
e777 /home/johnfoo/online johnfoo

The path of the {{File|name=exptab}} file is given by MIDAS_EXPTAB - see [[#Environment Variables]].

== Create shared memory files ==
At this point you should be able to run [[odbedit]] on the ''experiment host'' (localhost). The first time {{Utility|name=odbedit}} is run, it will create the required {{File|name=.*.SHM}} files in the MIDAS experiment directory {{Filepath|path=$HOME/online}}. These are the saved files for the MIDAS shared memory.

The default directory {{Filepath|path=$HOME/online}} will contain the MIDAS messages file ({{File|name=midas.log}}) and
any data files you may create. If you want an alternative location for the data files see [[#Customizing your experiment]].

Run {{Utility|name=odbedit}}, and type {{Odbedit cmd|cmd=ls}} to list the default directories.
[mhostpc] odbedit
[local:e777:S]/>ls
Experiment
System
Programs
Logger
Runinfo
Alarms

== Run the MIDAS Web Server ==

Recommended for production systems with maximum security is to run mhttpd behind an industry-standard SSL/https proxy with password protection.

To do this using apache httpd on CentOS-7, do this:
* configure apache httpd per instructions at https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Configure_HTTPS_server_.28CentOS7.29
* run odbedit, change values of
** "/experiment/http redirect to https" set to "n"
** "/experiment/midas https port" set to 0
* run mhttpd
* use web browser to connect to https://mhostpc/
* enter the user name and password configured in apache httpd
* the midas status page should open

For simplified installation, one can use standalone mhttpd with built-in https and password protection. This configuration
is only as secure as the underlying mongoose and OpenSSL web server and encryption packages.

Many current operating systems implement a built-in firewall which will block outside access to mhttpd.

On CentOS-7, so this: https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29

Start [[mhttpd]] on the ''experiment host'' (localhost) like this:
[mhostpc] mhttpd
You will get the following messages:

Mongoose web server will listen on ports "8080r,8443s"
[mhttpd,ERROR] [mhttpd.cxx:17892:mongoose,ERROR] cannot find SSL certificate file "/home/agdaq/online/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17893:mongoose,ERROR] please create SSL certificate file: openssl req -new -nodes -newkey rsa:2048 -sha256 -out ssl_cert.csr -keyout ssl_cert.key; openssl x509 -req -days 365 -sha256 -in ssl_cert.csr -signkey ssl_cert.key -out ssl_cert.pem; cat ssl_cert.key >> ssl_cert.pem
could not start the mongoose web server, see messages and midas.log, bye!

Create a self-signed certificate suitable for initial testing by executing the commands printed by mhttpd.

For production use, you should create a properly signed certificate, see [[Mhttpd#Create an SSL certificate|create your own SSL certificate]] or you should run mhttpd behind an SSL/https proxy (apache httpd).

Run mhttpd again.

You will get the following messages:

Mongoose web server will listen on ports "8080r,8443s"
Mongoose web server will use SSL certificate file "/home/johnfoo/packages/midas/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17633:mongoose,ERROR] mongoose web server cannot find password file "/home/johnfoo/online/htpasswd.txt"
[mhttpd,ERROR] [mhttpd.cxx:17634:mongoose,ERROR] please create password file: htdigest -c /home/johnfoo/online/htpasswd.txt Default midas
could not start the mongoose web server, see messages and midas.log, bye!

Create the password file by following the instructions printed by mhttpd. The http digest domain name is the experiment name, suggested default user name is "midas". You will be asked to type in a password. (htdigest is part of the httpd-tools package)


[mhostpc] htdigest -c /home/johnfoo/online/htpasswd.txt exptname midas
Adding password for midas in realm exptname.
New password:
Re-type new password:

It is a good idea to set the password file {{Filepath|path=htpasswd.txt}} readable and writable by owner only.

Now restart {{Utility|name=mhttpd}}
[mhostpc] mhttpd
Mongoose web server will listen on ports "8080r,8443s" **see note
Mongoose web server will use SSL certificate file "/home/suz/packages/midas/ssl_cert.pem"
Mongoose web server will use authentication realm "Default", password file "./htpasswd.txt"

Now point a web browser running on the same host computer (localhost) to https://localhost:8443
If the web browser is running on a different computer, go to URL of the form

https://mhostpc.triumf.ca:8443 (substitute your host machine name and domain for "mhostpc.triumf.ca")

If you are using the default SSL certificate you will probably get a message: "This Connection is Untrusted". Click "I understand the risks" and add an exception. This is because the test certificate is self-signed. Then confirm an exception.

If instead you get a "connection refused" error, the midas host pc may have the firewall enabled. To make a firewall exception for MIDAS, follow instructions here http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29

You should then see an authentication box asking you for the user name and password. The user name is "midas". Enter the password you just created. The Midas [[Status Page]] should appear with multiple buttons for run control as well as equipment listing (no equipments will be listed as yet) and application listings. Please refer to [[mhttpd]] (the MIDAS Web-based Run Control utility) for further information. You can start and stop runs from the main status page, and use the [[ODB Page]] to access the database (ODB).

; Note
: Default ports of 8080 and 8443 are used by [[mhttpd]]. To use some other ports, change the values of ODB "/experiment/midas http port" and "midas https port".

== Run the MIDAS logger ==
Start the MIDAS logger [[mlogger]] on the ''experiment host'' (localhost) :
[mhostpc] mlogger
This should start without error. It is usually run as a daemon, however in this case it is run in a terminal to check for errors. Starting the midas logger [[mlogger]] will automatically create more keys in the {{Odbpath|path=/Logger}} ODB tree.

From the MIDAS status page, go to the "Programs" page, then the "Logger" page and set ODB variables:
* Required -> set to "y"
* Start command -> set to "mlogger -D"
* Alarm Class -> set to "Alarm"

To test this, go to the "Programs" page, press "stop logger", then "start logger".

Start a run, wait 10 seconds, stop the run.

From the MIDAS "Status" page, go to the "logging channels -> channel #0" and set ODB variables:
* Output -> "FILE"

Start a run, go to the "Status" page, observe that the output file is written with gzip compression and has the "mid.gz" extension.

Stop the run.

== Clients run on Localhost only ==
;NOTE:
* If creating a MIDAS experiment with a '''REMOTE''' frontend, '''continue by following the instructions''' [[#Running with one or more REMOTE frontends]].
* If all clients are running on the Experiment Host (i.e. '''localhost'''), continue with the following instructions:

=== Frontend and Analyzer (localhost) ===

THIS SECTION IS VERY OBSOLETE

There are several examples of [[Frontend user code]] ({{File|name=frontend.c}}) in the MIDAS package available under {{Filepath|path=$MIDASSYS/examples/}}. Choose a suitable example from this directory that you can later modify for your own particular setup (e.g. for a slow control, choose the {{Filepath|path=../slowcont/}} example).

A '''frontend''' is a program that usually reads data from the hardware and sends it to a buffer to be logged and analyzed. You can find documentation about the frontend structure under [[Frontend Application]], [[Frontend Operation]] and [[Frontend user code]].

The example chosen here is from {{Filepath|path=$MIDASSYS/examples/experiment/}}. It does '''not need any hardware''' and produces simulated events.

On the MIDAS ''experiment host'' (localhost), copy the example to the experiment directory and build it.

[mhostpc] cd $HOME/online
[mhostpc] cp $MIDASSYS/examples/experiment/* .
[mhostpc] make
The analyzer will only build if ROOT has previously been installed.
At this point the frontend and the analyzer should be ready if no errors were generated during the build. Try the frontend and analyzer by starting them in xterms.

[mhostpc] frontend
[mhostpc] analyzer

=== Equipment ODB tree ===
Starting the frontend will automatically create the {{Odbpath|path=/Equipment}} ODB tree and one or more equipments. In this case, the equipments "trigger" and "scaler" have been defined by code in {{File|name=frontend.c}} and created the first time frontend runs.

[mhostpc] odbedit
[local:e777:S]/>ls /equipment
Trigger
Scaler 

View the MIDAS main status web page (by pointing your browser to https://localhost:8443/ or appropriate port if not using the default).
Observe on the [[Status Page]] that the two Equipments (Trigger, Scaler) have appeared in the Equipment section (Figure 1) and are coloured green.
 
'''Click thumbnail to enlarge'''
[[File:quickstart_status_page.png|thumb|left|Figure 1: web server status page showing example frontend equipments]]
<div style="clear: both"></div> 

Rather than using odbedit, you can view the contents of these equipments by clicking on the ODB button, then on {{Odbpath|path=/Equipment}}, then {{Odbpath|path=Trigger}} or {{Odbpath|path=Scaler}}. See [[ODB Page]] for more information.

=== Start a run ===
Start a run by pressing the Start button on the [[Status Page]].
The Equipment display on the Status Page will now show some event statistics (Figure 2)
'''Click thumbnail to enlarge'''
[[File:sample_frontend_status.png|thumb|left|Figure 2: Equipments shown on Status Page showing event statistics]]
<div style="clear: both"></div> 

The frontend will show an event display that will update when a run is started, e.g.

Sample Frontend connected to <local>. Press "!" to exit 18:48:2546:08
==========================================================================
Run status: Running Run number 2 |/ /
===========================================================================
Equipment Status Events Events/sec Rate[B/s] ODB->FE FE->OD
---------------------------------------------------------------------------
Trigger OK 13948 99.0 5413.0 0 140
Scaler OK 15 0.3 15.4 0 15

The [[Status Page]] will also show events being generated (click the refresh button if necessary).

=== Event Dump ===
While a run is in progress, the midas application [[mdump]] will provide you an event dump of the collected data from the running frontend, e.g.

mhostpc> mdump -l 50
blank
- MIDAS revision: Mon Nov 2 11:50:51 2015 -0800 - 3b66779 -- Enter <!> to Exit ------- Midas Dump ---
------------------------ Event# 1 ------------------------
Evid:0001- Mask:0000- Serial:0- Time:0x567a0c5d- Dsize:40/0x28
#banks:2 - Bank list:-ADC0TDC0-
blank
Bank:ADC0 Length: 8(I*1)/2(I*4)/4(Type) Type:Unsigned Integer*2
1-> 0x0167 0x03c6 0x0069 0x0073
blank
Bank:TDC0 Length: 8(I*1)/2(I*4)/4(Type) Type:Unsigned Integer*2
1-> 0x0051 0x00ff 0x004a 0x00ec

For further data processing/analysis, either the midas analyzer or the rootana can used for data display as well.

=== Create a startup script (localhost) ===
It is useful to create a script that will automatically start all the required clients for the experiment. This is called a startup script. Here, the startup script will be {{File|name=start_daq.sh}} (any name can be used). Create {{Filepath|path=$HOME/online/bin/start_daq.sh}} on the ''experiment host'' as follows (supply the [[mhttpd]] ports if default is not used):

#!/bin/sh
# start_daq.sh
cd $HOME/online
odbedit -c clean
# start mhttpd on default port. (Mongoose https version)
mhttpd -D # you can optionally restrict access to localhost and other specified hosts - see [[mhttpd]]
xterm -e ./frontend &
xterm -e ./analyzer &
mlogger -D
#end file

Before running this script, you will need to shutdown any running clients:
mhostpc> odbedit -c "sh all"
or you can use the {{Utility|name=mhttpd}} [[Programs Page]] to shut them down.

The script will start them as daemons.
By running the startup script start_daq.sh, several MIDAS applications will be started in sequence:
# Cleanup previous MIDAS application (if any).
# Start the MIDAS web server [mhttpd]
# Start the frontend application in its own xterm (for debugging purpose).
# Start the analyzer application in its own xterm (for debugging purpose).
# Start the MIDAS Data logger [mlogger]
[mhostpc] sh ./start_daq

You may wish to modify the script to restart missing clients only.

Next [[#Customizing your experiment|customize your experiment]].

== Running with one or more REMOTE frontends ==
These instructions assume you have already followed the instructions to setup an experiment running on the MIDAS experiment host (localhost), i.e. you have setup the MIDAS Environment variables, downloaded MIDAS packages, setup the [[mhttpd]] password and have {{Utility|name=mhttpd}} running successfully.

In the case of '''remote''' frontend(s), the remote cpu(s) typically have access to some or all of the hardware. They might be VMIC cpus running in VME crates for example, connected to the MIDAS Host via Ethernet. They read data from the hardware and send it back to the MIDAS ''experiment host'' to be logged and analyzed.

=== Experimental setup (Remote) ===
These instructions assume that the MIDAS ''experiment host'' computer is 64-bit (if not see [[#Build 32-bit MIDAS libraries|note below]]), and there is one ''remote host'', a 32-bit machine named "rlxhost".

The ''remote host'' mounts the {{Filepath|path=/home}} disk of the ''experiment host'', so that it '''has access to the MIDAS packages''', the {{Filepath|path=$HOME/online}} directory and shares the .cshrc (or .bashrc) script. It is convenient to set up ssh keys (use ssh-keygen) so that the ''remote host'' can be accessed without supplying the password from the ''experiment host''.

If your setup is different, you will have to make changes as appropriate.

=== Build 32-bit MIDAS libraries ===
NOTE

If the MIDAS ''experiment host'' computer is 32-bit (i.e. the same as the ''remote host''), which is a simpler setup, you will have built the 32-bit MIDAS libraries already, and can skip the rest of this section. Note however, that in the .cshrc file in the [[#Modify .cshrc (.bashrc) for Remote host|following section]] you will not need the lines starting "# select 64-bit or 32-bit MIDAS and ROOT". The PATH should contain $MIDASSYS/linux/bin on both experiment host and remote host. This will point to the 32-bit MIDAS libraries.
------------------------------

 
Assuming the MIDAS host computer is 64-bit, you will have already built the 64-bit MIDAS libraries.
In this case, if the remote frontend is 32-bit, you will need to build the 32-bit MIDAS libraries on the 64-bit machine, i.e.

[mhostpc] cd /home/packages/midas
[mhostpc] make linux32 ### build the 32-bit MIDAS libraries on 64-bit machine
[mhostpc] ls -l linux-m32/bin/odbedit ### check that the 32-bit odbedit has been created 
Do not run {{Utility|name=odbedit}} yet on the ''remote host''!

=== Modify .cshrc (.bashrc) for Remote host ===
Add the following to the {{File|name=.cshrc}} (modified appropriately if using {{File|name=.bashrc}}). When running with a REMOTE Host, the application [[mserver]] will be started (using a particular port) on the MIDAS ''experiment host'' machine only. Note that the environment variable MIDAS_SERVER_HOST is defined on the remote machine(s) but NOT on the ''experiment host''.

# setenv MIDAS_HOST mhostpc # substitute your experiment host name
# setenv REMOTE_HOST rlxhost # substitute your remote host name
# setup the MIDAS mserver host
#
switch (`hostname`)
case $MIDAS_HOST*:
unsetenv MIDAS_SERVER_HOST ## MIDAS_SERVER_HOST not defined
breaksw
default:
setenv MIDAS_SERVER_HOST $MIDAS_HOST.triumf.ca:1175 ## for remote host, define MIDAS_SERVER_HOST
## as hostname + domain name + default mserver port
## port must match that of [[mserver]] used in start_daq.sh
endsw
#
# select 64-bit or 32-bit MIDAS and ROOT
#
switch (`uname -i`)
case i386:
setenv ROOTSYS /triumfcs/trshare/olchansk/root/root_v5.28.00_SL55_32 ## or appropriate path
setenv PATH .:$MIDASSYS/linux-m32/bin:$PATH
breaksw
default:
setenv ROOTSYS $HOME/packages/root
setenv PATH .:$MIDASSYS/linux/bin:$PATH
endsw
#
setenv PATH .:$HOME/online/bin:$HOME/packages/roody/bin:$ROOTSYS/bin:$PATH
#

After executing {{File|name=.cshrc}} (or {{File|name=.bashrc}}) or logging out, on the MIDAS host (mhostpc) make sure environment variable MIDAS_SERVER_HOST is NOT defined
[mhostpc] echo $MIDAS_SERVER_HOST
[mhostpc] MIDAS_SERVER_HOST: Undefined variable.

and on the '''''remote host''''', make sure MIDAS_SERVER_HOST IS defined:
[rlxhost] echo $MIDAS_SERVER_HOST
[rlxhost] MIDAS_SERVER_HOST

Also on the ''remote host'', make sure that the correct (32-bit) {{Utility|name=odbedit}} will be used (do not run it yet!)
[rlxhost] which odbedit
/home/johnfoo/packages/midas/linux-m32/bin/odbedit

=== Grant Remote host access permission ===
Give permission for the remote host(s) to access the experiment by following the instructions to [[Security#MIDAS programs on remote machines|allow MIDAS programs on remote machines]]. If the MIDAS web server [[mhttpd]] is running, use [[ODB Page]] to edit the ODB keys listed, otherwise use [[odbedit]] on the ''experiment host'' machine.

=== Start mserver on Experiment Host ===
The application [[mserver]] is not required when running an experiment on the ''experiment host'' (localhost) only, but it '''IS required''' for access from a '''REMOTE''' frontend.
Start [[mserver]] on MIDAS host (default port 1175) or use the "-p" option for a different port.
The port must match the port defined for environment variable MIDAS_SERVER_HOST in {{File|name=.cshrc}} (or {{File|name=.bashrc}}) [[#Modify .cshrc (.bashrc) for Remote host|above]].
[mhostpc] mserver -D

The environment variables on the REMOTE host should be:

[rlxhost] printenv | grep MIDAS
MIDASSYS=/home/e777/packages/midas
MIDAS_EXPT_NAME=e777
MIDAS_SERVER_HOST=mhostpc.triumf.ca:1175
(a different port may be used - see above)

=== Check access to ODB from Remote host ===
Run {{Utility|name=odbedit}} on the remote host to check access to ODB.
[rlxhost] odbedit
[mhostpc.triumf.ca:1175:e777:Stopped]/>ls 
If it does not work correctly, check that
* MIDAS_SERVER_HOST is defined on REMOTE host with same port as [[mserver]]
* mserver is running on MIDAS host with the correct port
* MIDAS_SERVER_HOST is NOT defined on MIDAS host
* remote access permission to REMOTE host has been granted as described above.

=== Build the analyzer on Experiment Host ===
The analyzer will be built and run on the MIDAS ''experiment host'' machine.
Select an example from $MIDASSYS/examples/ (see [[#Frontend and Analyzer (localhost)]]) and copy it to $HOME/online on the MIDAS host machine. Since we are assuming a 64-bit MIDAS host and a 32-bit remote host, edit the example Makefile so only the analyzer will be built.

=== Build the frontend (32bit) for Remote host ===
The frontend runs on a 32-bit remote host in this example. It therefore must either
* be built 32-bit on the 64-bit MIDAS host with the "-m32" flag
* or built on the REMOTE host
and in either case the [[#Build 32-bit MIDAS libraries|32-bit MIDAS libraries]] used for linking.

To avoid confusion, create a different subdirectory for the frontend, e.g.
[rlxhost] mkdir $HOME/online/fe_32
Copy the Makefile and frontend code from the same example to this directory.
Modify the Makefile as needed, and only build the frontend.

Once built, start the frontend in an xterm on the REMOTE host.
[rlxhost] $HOME/online/fe_32/frontend
Check that it connects successfully to the experiment and the Equipments appear on the [[Status Page]] as described under [[#Frontend and Analyzer (localhost)]]. The Equipment Status (Figure 2) should show that the Equipments are now running on the remote cpu, i.e. "SampleFrontend@rlxhost.triumf.ca" rather than "Sample Frontend@localhost".

=== Run the Analyzer on Experiment host ===
Start the analyzer in an xterm on the MIDAS ''experiment host''
[johnfoo@mhostpc e777] $HOME/online/analyzer

Continue by starting a run and dumping the data as described under [[#Frontend and Analyzer (localhost)]].

=== Create experiment startup script ===
It is useful to create a script that will automatically start all the required clients for the experiment. This is called a startup script (any name can be used). Create {{Filepath|path=$HOME/online/bin/start_daq.sh}} on the ''experiment host'' as follows, (supply the [[mhttpd]] ports if default is not used) :

#!/bin/sh
# start_daq.sh
cd $HOME/online
#
switch (`hostname`)
case $MIDAS_HOST*: ## MIDAS_HOST is assigned in .cshrc (.bashrc)
echo "Good, we are on $MIDAS_HOST"
breaksw
case $REMOTE_HOST*: ## REMOTE_HOST is assigned in .cshrc (.bashrc)
echo "startup script should be executed on $MIDAS_HOST"
endsw
#
odbedit -c clean
# start mhttpd on default port (Mongoose https version)
mhttpd -D # you can optionally restrict access to localhost and/or other specified hosts - see [[mhttpd]]
# start mserver on default port (use argument -p to use a different port)
mserver -D # start mserver on default port (1175) or a different port using -p option - see [[mserver]]
# the port must match that of MIDAS_SERVER_HOST in .cshrc (.bashrc) - see above
# access must be specifically allowed - see [[#Grant Remote host access permission||above]]
xterm -e ./analyzer &
## start frontend on remote host
#ssh $REMOTE_HOST $HOME/online/bin/start_frontend -O >& $HOME/online/bin/start_frontend.log &
mlogger -D
#end file

Note that command to start the frontend on the remote host is commented out. It can be uncommented once the start_frontend script is written and tested.

Shut down any clients that are running already by running odbedit on the MIDAS ''experiment host'' and execute the new startup script {{File|name=start_daq.sh}} (make it executable if desired)

[mhostpc] odbedit
[local:e777:S]/>sh all
[mhostpc] ./start_daq.sh 
and check that {{Utility|name=odbedit}} still works on the ''REMOTE host''

[rlxhost] odbedit
[mhostpc.triumf.ca:1175:e777:Stopped]/>ls 

You may wish to modify the script to restart missing clients only.

=== Create script to start frontend ===
The startup file {{File|name=start_daq.sh}} invoked a script {{File|name=start_frontend}} to start the remote frontend. This line was commented out (see [[#Create experiment startup script|above]]). Once the frontend is working, create a script to start the frontend (e.g. file {{Filepath|path=$HOME/online/bin/start_frontend}}).

#!/bin/tcsh
# Script to start frontend running on $REMOTE_HOST
#
# This script runs on the REMOTE HOST
#
echo "starting frontend for experiment $MIDAS_EXPT_NAME "
xterm -geometry 150x50+800+0 -fg white -bg blue -title "Frontend" -e "$HOME/online/fe_32/frontend" &
echo "Frontend task has been started"
exit

Check that this script works by running it on the REMOTE host. If successful,
run the commented command in an xterm on the ''experiment host''
[mhostpc] ssh $REMOTE_HOST $HOME/online/bin/start_frontend -O >& $HOME/online/bin/start_frontend.log & 
This command relies on password-less access set up with SSH keys (see [[#Experimental setup (Remote)]]).
If successful, uncomment line in {{File|name=start_daq.sh}}, and run the script to check that it can start the remote client(s) from the MIDAS ''experiment host'' (first shutting down all running clients) :

[mhostpc] odbedit -c "sh all"
[mhostpc] ./start_daq.sh 

Next [[#Customizing your experiment|customize your experiment]].

== Customizing your experiment ==

Here are some of the more common options provided for users to customize their experiments according to their own preferences:

; Write frontend(s) to read out and control your hardware
: Modify the example frontend and Makefile (from $MIDASSYS/examples/) you have already copied and built so it accesses your hardware.

: The example Frontend.c is an example of [[Frontend user code]]. You can find documentation about the frontend structure under [[Frontend Application]], [[Frontend Operation]] and [[Frontend user code]]. You may need to set up some [[Event Notification (Hot-Link)|Hot-Links]] in the frontend. The MIDAS package includes device drivers for some of the more common hardware. See [[MIDAS Driver Library]] for more information.

; Modify analyzer(s) for your data
: Modify the example analyzer code as necessary to analyze the data from your frontends.

; Slow Controls
: Control your slow-controls equipment with slow-control frontend(s) optionally using the [[#MIDAS Package Installation|MSCB]] system.

; Data logger
: Set up the data logger [[mlogger]] to log data to a storage device. Many options are available - see [[Data Logger]], [[mlogger]] and [[/Logger ODB tree]] for instructions.

; Allow programs to be restarted from the [[Programs Page]]
: See mhttpd [[Programs Page]] and [[/Programs ODB tree]]

; Write scripts which run at start and end of run
: See [[/Programs ODB tree#Execute on start run]] and [[/Programs ODB tree#Execute on stop run]]

; Create a shutdown script
: Create a script similar to the startup script to shutdown the experiment

; Lazy logger
: Set up the [[lazylogger]] for archiving data files for storage.

; Create edit-on-start parameters
: See [[Edit-on-start Parameters]]

; Set up the alarm system
: See [[Alarm System]] and [[Alarms Page]]

; Set up history system
: See [[History System]] and [[History Page]]

; Set up the Electronic logbook (Elog)
: See [[Elog]] for details

; Set up the Sequencer
: The [[Sequencer Page#Sequencer]] can be used to run a sequence of runs, changing parameters as needed.

; Write a webserver Custom page
: To better control and monitor your experiment, you may want to write a [[Custom Page]].

[[Category:Installation]]

Quickstart Linux

2016-11-08T01:12:32Z

Olchansk: /* Run the MIDAS logger */

{{Pagelinks}}
== Introduction ==
This quickstart shows you how to install the MIDAS packages on a linux box and create a '''MIDAS experiment'''.
The linux box will be called the MIDAS '''''experiment host'''''.

==Installation==
Log on to the ''[[#Introduction|Experiment Host]]'' and decide on a name for the MIDAS experiment (the ''experiment name'').
In these instructions, the MIDAS packages and experiment will be installed under the username "johnfoo".
The ODB and other shared memory buffers for the MIDAS experiment will reside on the ''experiment host'' (localhost).
MIDAS and the other required packages will be installed under directory {{Filepath|path=$HOME/packages}}. The MIDAS ''experiment directory'' will be {{Filepath|path=$HOME/online}} and the MIDAS ''experiment name'' will be "e777". The name of the ''experiment host'' will be "mhostpc".

;NOTE
: The user should substitute directory paths for the MIDAS packages and experiment directory as required, and names for ''experiment host'', username, ''experiment name'' appropriate to his/her own setup.

== Experimental setup ==
These instructions describe setting up an experiment that will run ONLY on the MIDAS ''experiment host'' computer (i.e. '''localhost only'''). In this case, the MIDAS ''experiment host'' has access to any hardware required, and all frontends, logger, analyzer etc. run on the one computer.

Modifications to run an experiment with additional frontend(s) running '''remotely''' (i.e. on remote cpu(s)) will also be described.
Follow the instructions for setting up a localhost-only experiment until indicated.

In both cases, the main MIDAS applications (e.g. [[mlogger]], [[mhttpd]]) run on the MIDAS ''experiment host''.

==Environment Variables==
The following Environment variables should to be added to the {{File|name=.cshrc}} or {{File|name=.bashrc}} file (depending on your shell) in the {{Filepath|path=$HOME}} directory on the ''experiment host'', so that they will be defined at login. '''Substitute appropriate values for your own setup'''.

{| class="wikitable"
|-
! csh !! bash !! comment
|-
| setenv GIT_EDITOR "emacs -nw" || export GIT_EDITOR="emacs -nw" ||
|-
| setenv MIDASSYS $HOME/packages/midas || export MIDASSYS=$HOME/packages/midas ||Base directory of the MIDAS package
|-
| setenv MIDAS_EXPTAB $HOME/online/exptab || export MIDAS_EXPTAB=$HOME/online/exptab || MIDAS experiment table
|-
|-
| setenv MIDAS_EXPT_NAME e777 || export MIDAS_EXPT_NAME=e777 || MIDAS experiment name
|-
| setenv PATH .:$MIDASSYS/linux/bin:$PATH || export PATH=$PATH:$MIDASSYS/linux/bin || path
|}

Logout and login again, or source {{File|name=.cshrc}} (source {{File|name=.bashrc}}) for the changes to take effect.
;NOTE
: See [[MIDAS environment variables]] for a list of environment variables used by MIDAS.

== ROOT Package Installation ==
For full MIDAS operation, ROOT is needed for the data logging and analysis packages. It needed to build the example analyzer in this Quickstart. However, ROOT is not essential to run MIDAS. If ROOT is NOT installed, the environment variable ROOTSYS should be undefined.

If ROOT is needed, see [[Setup MIDAS experiment#Install ROOT|Install ROOT]].

== MIDAS Package Installation ==
The MIDAS package will be installed on the ''experiment host'' in the directory given by MIDASSYS - see [[#Environment Variables]].

Install the MIDAS and other package(s) from the MIDAS git repository in an xterm as follows:

[mhostpc] mkdir $HOME/packages
[mhostpc] cd $HOME/packages
[mhostpc] git clone https://bitbucket.org/tmidas/midas
[mhostpc] git clone https://bitbucket.org/tmidas/mxml

Install the MSCB package '''only''' if you will be using the MSCB system for slow-controls (see [[MSCB Page]]):

[mhostpc] git clone https://bitbucket.org/tmidas/mscb ## MSCB package
[mhostpc] cd mscb
[mhostpc] make ## Build the MSCB package
[mhostpc] cd ../

The MIDAS Makefile will auto-detect whether MSCB, ROOT, MYSQL, SQLITE, ODBC etc. are available (or have been installed). If you need these features, make sure they are available before building the MIDAS package.

[mhostpc] cd midas
[mhostpc] make ## Build the MIDAS package
[mhostpc] ls -l linux/bin/odbedit ### check that odbedit has been created (do not run it yet)


In case of problems see
* [[Installation#Installing MIDAS|bitbucket is down]] or
* [[Installation/Compilation problems#SSL certificate errors|SSL certificate errors]].

==ROOTANA Package Installation==
If ROOTANA is needed,

[mhostpc] cd $HOME/packages
[mhostpc] git clone https://bitbucket.org/tmidas/rootana
[mhostpc] cd rootana
[mhostpc] make

==ROODY Package Installation==
If ROODY is needed,

[mhostpc] cd $HOME/packages
[mhostpc] git clone https://bitbucket.org/tmidas/roody
[mhostpc] cd roody
[mhostpc] make


== Create the Experiment file exptab ==
In this example, the experimental directory is {{Filepath|path=$HOME/online}}. Create this directory on the ''experiment host'' (localhost):

[mhostpc] mkdir $HOME/online
[mhostpc] cd $HOME/online
Then create the [[exptab]] file {{Filepath|path=$HOME/online/exptab}} containing the three parameters:
# the experiment name
# location of MIDAS shared memory buffers
# username
as follows (using parameters for your own experiment):

[mhostpc] cat > exptab
e777 /home/johnfoo/online johnfoo

The path of the {{File|name=exptab}} file is given by MIDAS_EXPTAB - see [[#Environment Variables]].

== Create shared memory files ==
At this point you should be able to run [[odbedit]] on the ''experiment host'' (localhost). The first time {{Utility|name=odbedit}} is run, it will create the required {{File|name=.*.SHM}} files in the MIDAS experiment directory {{Filepath|path=$HOME/online}}. These are the saved files for the MIDAS shared memory.

The default directory {{Filepath|path=$HOME/online}} will contain the MIDAS messages file ({{File|name=midas.log}}) and
any data files you may create. If you want an alternative location for the data files see [[#Customizing your experiment]].

Run {{Utility|name=odbedit}}, and type {{Odbedit cmd|cmd=ls}} to list the default directories.
[mhostpc] odbedit
[local:e777:S]/>ls
Experiment
System
Programs
Logger
Runinfo
Alarms

== Run the MIDAS Web Server ==

Recommended for production systems with maximum security is to run mhttpd behind an industry-standard SSL/https proxy with password protection.

To do this using apache httpd on CentOS-7, do this:
* configure apache httpd per instructions at https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Configure_HTTPS_server_.28CentOS7.29
* run odbedit, change values of
** "/experiment/http redirect to https" set to "n"
** "/experiment/midas https port" set to 0
* run mhttpd
* use web browser to connect to https://mhostpc/
* enter the user name and password configured in apache httpd
* the midas status page should open

For simplified installation, one can use standalone mhttpd with built-in https and password protection. This configuration
is only as secure as the underlying mongoose and OpenSSL web server and encryption packages.

Many current operating systems implement a built-in firewall which will block outside access to mhttpd.

On CentOS-7, so this: https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29

Start [[mhttpd]] on the ''experiment host'' (localhost) like this:
[mhostpc] mhttpd
You will get the following messages:

Mongoose web server will listen on ports "8080r,8443s"
[mhttpd,ERROR] [mhttpd.cxx:17892:mongoose,ERROR] cannot find SSL certificate file "/home/agdaq/online/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17893:mongoose,ERROR] please create SSL certificate file: openssl req -new -nodes -newkey rsa:2048 -sha256 -out ssl_cert.csr -keyout ssl_cert.key; openssl x509 -req -days 365 -sha256 -in ssl_cert.csr -signkey ssl_cert.key -out ssl_cert.pem; cat ssl_cert.key >> ssl_cert.pem
could not start the mongoose web server, see messages and midas.log, bye!

Create a self-signed certificate suitable for initial testing by executing the commands printed by mhttpd.

For production use, you should create a properly signed certificate, see [[Mhttpd#Create an SSL certificate|create your own SSL certificate]] or you should run mhttpd behind an SSL/https proxy (apache httpd).

Run mhttpd again.

You will get the following messages:

Mongoose web server will listen on ports "8080r,8443s"
Mongoose web server will use SSL certificate file "/home/johnfoo/packages/midas/ssl_cert.pem"
[mhttpd,ERROR] [mhttpd.cxx:17633:mongoose,ERROR] mongoose web server cannot find password file "/home/johnfoo/online/htpasswd.txt"
[mhttpd,ERROR] [mhttpd.cxx:17634:mongoose,ERROR] please create password file: htdigest -c /home/johnfoo/online/htpasswd.txt Default midas
could not start the mongoose web server, see messages and midas.log, bye!

Create the password file by following the instructions printed by mhttpd. The http digest domain name is the experiment name, suggested default user name is "midas". You will be asked to type in a password. (htdigest is part of the httpd-tools package)


[mhostpc] htdigest -c /home/johnfoo/online/htpasswd.txt exptname midas
Adding password for midas in realm exptname.
New password:
Re-type new password:

It is a good idea to set the password file {{Filepath|path=htpasswd.txt}} readable and writable by owner only.

Now restart {{Utility|name=mhttpd}}
[mhostpc] mhttpd
Mongoose web server will listen on ports "8080r,8443s" **see note
Mongoose web server will use SSL certificate file "/home/suz/packages/midas/ssl_cert.pem"
Mongoose web server will use authentication realm "Default", password file "./htpasswd.txt"

Now point a web browser running on the same host computer (localhost) to https://localhost:8443
If the web browser is running on a different computer, go to URL of the form

https://mhostpc.triumf.ca:8443 (substitute your host machine name and domain for "mhostpc.triumf.ca")

If you are using the default SSL certificate you will probably get a message: "This Connection is Untrusted". Click "I understand the risks" and add an exception. This is because the test certificate is self-signed. Then confirm an exception.

If instead you get a "connection refused" error, the midas host pc may have the firewall enabled. To make a firewall exception for MIDAS, follow instructions here http://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Enable_firewall_for_MIDAS_.28CentOS7.29

You should then see an authentication box asking you for the user name and password. The user name is "midas". Enter the password you just created. The Midas [[Status Page]] should appear with multiple buttons for run control as well as equipment listing (no equipments will be listed as yet) and application listings. Please refer to [[mhttpd]] (the MIDAS Web-based Run Control utility) for further information. You can start and stop runs from the main status page, and use the [[ODB Page]] to access the database (ODB).

; Note
: Default ports of 8080 and 8443 are used by [[mhttpd]]. To use some other ports, change the values of ODB "/experiment/midas http port" and "midas https port".

== Run the MIDAS logger ==
Start the MIDAS logger [[mlogger]] on the ''experiment host'' (localhost) :
[mhostpc] mlogger
This should start without error. It is usually run as a daemon, however in this case it is run in a terminal to check for errors. Starting the midas logger [[mlogger]] will automatically create more keys in the {{Odbpath|path=/Logger}} ODB tree.

From the MIDAS status page, go to the "Programs" page, then the "Logger" page and set ODB variables:
* Required -> set to "y"
* Start command -> set to "mlogger -D"
* Alarm Class -> set to "Alarm"

To test this, go to the "Programs" page, press "stop logger", then "start logger".

== Clients run on Localhost only ==
;NOTE:
* If creating a MIDAS experiment with a '''REMOTE''' frontend, '''continue by following the instructions''' [[#Running with one or more REMOTE frontends]].
* If all clients are running on the Experiment Host (i.e. '''localhost'''), continue with the following instructions:

=== Frontend and Analyzer (localhost) ===

THIS SECTION IS VERY OBSOLETE

There are several examples of [[Frontend user code]] ({{File|name=frontend.c}}) in the MIDAS package available under {{Filepath|path=$MIDASSYS/examples/}}. Choose a suitable example from this directory that you can later modify for your own particular setup (e.g. for a slow control, choose the {{Filepath|path=../slowcont/}} example).

A '''frontend''' is a program that usually reads data from the hardware and sends it to a buffer to be logged and analyzed. You can find documentation about the frontend structure under [[Frontend Application]], [[Frontend Operation]] and [[Frontend user code]].

The example chosen here is from {{Filepath|path=$MIDASSYS/examples/experiment/}}. It does '''not need any hardware''' and produces simulated events.

On the MIDAS ''experiment host'' (localhost), copy the example to the experiment directory and build it.

[mhostpc] cd $HOME/online
[mhostpc] cp $MIDASSYS/examples/experiment/* .
[mhostpc] make
The analyzer will only build if ROOT has previously been installed.
At this point the frontend and the analyzer should be ready if no errors were generated during the build. Try the frontend and analyzer by starting them in xterms.

[mhostpc] frontend
[mhostpc] analyzer

=== Equipment ODB tree ===
Starting the frontend will automatically create the {{Odbpath|path=/Equipment}} ODB tree and one or more equipments. In this case, the equipments "trigger" and "scaler" have been defined by code in {{File|name=frontend.c}} and created the first time frontend runs.

[mhostpc] odbedit
[local:e777:S]/>ls /equipment
Trigger
Scaler 

View the MIDAS main status web page (by pointing your browser to https://localhost:8443/ or appropriate port if not using the default).
Observe on the [[Status Page]] that the two Equipments (Trigger, Scaler) have appeared in the Equipment section (Figure 1) and are coloured green.
 
'''Click thumbnail to enlarge'''
[[File:quickstart_status_page.png|thumb|left|Figure 1: web server status page showing example frontend equipments]]
<div style="clear: both"></div> 

Rather than using odbedit, you can view the contents of these equipments by clicking on the ODB button, then on {{Odbpath|path=/Equipment}}, then {{Odbpath|path=Trigger}} or {{Odbpath|path=Scaler}}. See [[ODB Page]] for more information.

=== Start a run ===
Start a run by pressing the Start button on the [[Status Page]].
The Equipment display on the Status Page will now show some event statistics (Figure 2)
'''Click thumbnail to enlarge'''
[[File:sample_frontend_status.png|thumb|left|Figure 2: Equipments shown on Status Page showing event statistics]]
<div style="clear: both"></div> 

The frontend will show an event display that will update when a run is started, e.g.

Sample Frontend connected to <local>. Press "!" to exit 18:48:2546:08
==========================================================================
Run status: Running Run number 2 |/ /
===========================================================================
Equipment Status Events Events/sec Rate[B/s] ODB->FE FE->OD
---------------------------------------------------------------------------
Trigger OK 13948 99.0 5413.0 0 140
Scaler OK 15 0.3 15.4 0 15

The [[Status Page]] will also show events being generated (click the refresh button if necessary).

=== Event Dump ===
While a run is in progress, the midas application [[mdump]] will provide you an event dump of the collected data from the running frontend, e.g.

mhostpc> mdump -l 50
blank
- MIDAS revision: Mon Nov 2 11:50:51 2015 -0800 - 3b66779 -- Enter <!> to Exit ------- Midas Dump ---
------------------------ Event# 1 ------------------------
Evid:0001- Mask:0000- Serial:0- Time:0x567a0c5d- Dsize:40/0x28
#banks:2 - Bank list:-ADC0TDC0-
blank
Bank:ADC0 Length: 8(I*1)/2(I*4)/4(Type) Type:Unsigned Integer*2
1-> 0x0167 0x03c6 0x0069 0x0073
blank
Bank:TDC0 Length: 8(I*1)/2(I*4)/4(Type) Type:Unsigned Integer*2
1-> 0x0051 0x00ff 0x004a 0x00ec

For further data processing/analysis, either the midas analyzer or the rootana can used for data display as well.

=== Create a startup script (localhost) ===
It is useful to create a script that will automatically start all the required clients for the experiment. This is called a startup script. Here, the startup script will be {{File|name=start_daq.sh}} (any name can be used). Create {{Filepath|path=$HOME/online/bin/start_daq.sh}} on the ''experiment host'' as follows (supply the [[mhttpd]] ports if default is not used):

#!/bin/sh
# start_daq.sh
cd $HOME/online
odbedit -c clean
# start mhttpd on default port. (Mongoose https version)
mhttpd -D # you can optionally restrict access to localhost and other specified hosts - see [[mhttpd]]
xterm -e ./frontend &
xterm -e ./analyzer &
mlogger -D
#end file

Before running this script, you will need to shutdown any running clients:
mhostpc> odbedit -c "sh all"
or you can use the {{Utility|name=mhttpd}} [[Programs Page]] to shut them down.

The script will start them as daemons.
By running the startup script start_daq.sh, several MIDAS applications will be started in sequence:
# Cleanup previous MIDAS application (if any).
# Start the MIDAS web server [mhttpd]
# Start the frontend application in its own xterm (for debugging purpose).
# Start the analyzer application in its own xterm (for debugging purpose).
# Start the MIDAS Data logger [mlogger]
[mhostpc] sh ./start_daq

You may wish to modify the script to restart missing clients only.

Next [[#Customizing your experiment|customize your experiment]].

== Running with one or more REMOTE frontends ==
These instructions assume you have already followed the instructions to setup an experiment running on the MIDAS experiment host (localhost), i.e. you have setup the MIDAS Environment variables, downloaded MIDAS packages, setup the [[mhttpd]] password and have {{Utility|name=mhttpd}} running successfully.

In the case of '''remote''' frontend(s), the remote cpu(s) typically have access to some or all of the hardware. They might be VMIC cpus running in VME crates for example, connected to the MIDAS Host via Ethernet. They read data from the hardware and send it back to the MIDAS ''experiment host'' to be logged and analyzed.

=== Experimental setup (Remote) ===
These instructions assume that the MIDAS ''experiment host'' computer is 64-bit (if not see [[#Build 32-bit MIDAS libraries|note below]]), and there is one ''remote host'', a 32-bit machine named "rlxhost".

The ''remote host'' mounts the {{Filepath|path=/home}} disk of the ''experiment host'', so that it '''has access to the MIDAS packages''', the {{Filepath|path=$HOME/online}} directory and shares the .cshrc (or .bashrc) script. It is convenient to set up ssh keys (use ssh-keygen) so that the ''remote host'' can be accessed without supplying the password from the ''experiment host''.

If your setup is different, you will have to make changes as appropriate.

=== Build 32-bit MIDAS libraries ===
NOTE

If the MIDAS ''experiment host'' computer is 32-bit (i.e. the same as the ''remote host''), which is a simpler setup, you will have built the 32-bit MIDAS libraries already, and can skip the rest of this section. Note however, that in the .cshrc file in the [[#Modify .cshrc (.bashrc) for Remote host|following section]] you will not need the lines starting "# select 64-bit or 32-bit MIDAS and ROOT". The PATH should contain $MIDASSYS/linux/bin on both experiment host and remote host. This will point to the 32-bit MIDAS libraries.
------------------------------

 
Assuming the MIDAS host computer is 64-bit, you will have already built the 64-bit MIDAS libraries.
In this case, if the remote frontend is 32-bit, you will need to build the 32-bit MIDAS libraries on the 64-bit machine, i.e.

[mhostpc] cd /home/packages/midas
[mhostpc] make linux32 ### build the 32-bit MIDAS libraries on 64-bit machine
[mhostpc] ls -l linux-m32/bin/odbedit ### check that the 32-bit odbedit has been created 
Do not run {{Utility|name=odbedit}} yet on the ''remote host''!

=== Modify .cshrc (.bashrc) for Remote host ===
Add the following to the {{File|name=.cshrc}} (modified appropriately if using {{File|name=.bashrc}}). When running with a REMOTE Host, the application [[mserver]] will be started (using a particular port) on the MIDAS ''experiment host'' machine only. Note that the environment variable MIDAS_SERVER_HOST is defined on the remote machine(s) but NOT on the ''experiment host''.

# setenv MIDAS_HOST mhostpc # substitute your experiment host name
# setenv REMOTE_HOST rlxhost # substitute your remote host name
# setup the MIDAS mserver host
#
switch (`hostname`)
case $MIDAS_HOST*:
unsetenv MIDAS_SERVER_HOST ## MIDAS_SERVER_HOST not defined
breaksw
default:
setenv MIDAS_SERVER_HOST $MIDAS_HOST.triumf.ca:1175 ## for remote host, define MIDAS_SERVER_HOST
## as hostname + domain name + default mserver port
## port must match that of [[mserver]] used in start_daq.sh
endsw
#
# select 64-bit or 32-bit MIDAS and ROOT
#
switch (`uname -i`)
case i386:
setenv ROOTSYS /triumfcs/trshare/olchansk/root/root_v5.28.00_SL55_32 ## or appropriate path
setenv PATH .:$MIDASSYS/linux-m32/bin:$PATH
breaksw
default:
setenv ROOTSYS $HOME/packages/root
setenv PATH .:$MIDASSYS/linux/bin:$PATH
endsw
#
setenv PATH .:$HOME/online/bin:$HOME/packages/roody/bin:$ROOTSYS/bin:$PATH
#

After executing {{File|name=.cshrc}} (or {{File|name=.bashrc}}) or logging out, on the MIDAS host (mhostpc) make sure environment variable MIDAS_SERVER_HOST is NOT defined
[mhostpc] echo $MIDAS_SERVER_HOST
[mhostpc] MIDAS_SERVER_HOST: Undefined variable.

and on the '''''remote host''''', make sure MIDAS_SERVER_HOST IS defined:
[rlxhost] echo $MIDAS_SERVER_HOST
[rlxhost] MIDAS_SERVER_HOST

Also on the ''remote host'', make sure that the correct (32-bit) {{Utility|name=odbedit}} will be used (do not run it yet!)
[rlxhost] which odbedit
/home/johnfoo/packages/midas/linux-m32/bin/odbedit

=== Grant Remote host access permission ===
Give permission for the remote host(s) to access the experiment by following the instructions to [[Security#MIDAS programs on remote machines|allow MIDAS programs on remote machines]]. If the MIDAS web server [[mhttpd]] is running, use [[ODB Page]] to edit the ODB keys listed, otherwise use [[odbedit]] on the ''experiment host'' machine.

=== Start mserver on Experiment Host ===
The application [[mserver]] is not required when running an experiment on the ''experiment host'' (localhost) only, but it '''IS required''' for access from a '''REMOTE''' frontend.
Start [[mserver]] on MIDAS host (default port 1175) or use the "-p" option for a different port.
The port must match the port defined for environment variable MIDAS_SERVER_HOST in {{File|name=.cshrc}} (or {{File|name=.bashrc}}) [[#Modify .cshrc (.bashrc) for Remote host|above]].
[mhostpc] mserver -D

The environment variables on the REMOTE host should be:

[rlxhost] printenv | grep MIDAS
MIDASSYS=/home/e777/packages/midas
MIDAS_EXPT_NAME=e777
MIDAS_SERVER_HOST=mhostpc.triumf.ca:1175
(a different port may be used - see above)

=== Check access to ODB from Remote host ===
Run {{Utility|name=odbedit}} on the remote host to check access to ODB.
[rlxhost] odbedit
[mhostpc.triumf.ca:1175:e777:Stopped]/>ls 
If it does not work correctly, check that
* MIDAS_SERVER_HOST is defined on REMOTE host with same port as [[mserver]]
* mserver is running on MIDAS host with the correct port
* MIDAS_SERVER_HOST is NOT defined on MIDAS host
* remote access permission to REMOTE host has been granted as described above.

=== Build the analyzer on Experiment Host ===
The analyzer will be built and run on the MIDAS ''experiment host'' machine.
Select an example from $MIDASSYS/examples/ (see [[#Frontend and Analyzer (localhost)]]) and copy it to $HOME/online on the MIDAS host machine. Since we are assuming a 64-bit MIDAS host and a 32-bit remote host, edit the example Makefile so only the analyzer will be built.

=== Build the frontend (32bit) for Remote host ===
The frontend runs on a 32-bit remote host in this example. It therefore must either
* be built 32-bit on the 64-bit MIDAS host with the "-m32" flag
* or built on the REMOTE host
and in either case the [[#Build 32-bit MIDAS libraries|32-bit MIDAS libraries]] used for linking.

To avoid confusion, create a different subdirectory for the frontend, e.g.
[rlxhost] mkdir $HOME/online/fe_32
Copy the Makefile and frontend code from the same example to this directory.
Modify the Makefile as needed, and only build the frontend.

Once built, start the frontend in an xterm on the REMOTE host.
[rlxhost] $HOME/online/fe_32/frontend
Check that it connects successfully to the experiment and the Equipments appear on the [[Status Page]] as described under [[#Frontend and Analyzer (localhost)]]. The Equipment Status (Figure 2) should show that the Equipments are now running on the remote cpu, i.e. "SampleFrontend@rlxhost.triumf.ca" rather than "Sample Frontend@localhost".

=== Run the Analyzer on Experiment host ===
Start the analyzer in an xterm on the MIDAS ''experiment host''
[johnfoo@mhostpc e777] $HOME/online/analyzer

Continue by starting a run and dumping the data as described under [[#Frontend and Analyzer (localhost)]].

=== Create experiment startup script ===
It is useful to create a script that will automatically start all the required clients for the experiment. This is called a startup script (any name can be used). Create {{Filepath|path=$HOME/online/bin/start_daq.sh}} on the ''experiment host'' as follows, (supply the [[mhttpd]] ports if default is not used) :

#!/bin/sh
# start_daq.sh
cd $HOME/online
#
switch (`hostname`)
case $MIDAS_HOST*: ## MIDAS_HOST is assigned in .cshrc (.bashrc)
echo "Good, we are on $MIDAS_HOST"
breaksw
case $REMOTE_HOST*: ## REMOTE_HOST is assigned in .cshrc (.bashrc)
echo "startup script should be executed on $MIDAS_HOST"
endsw
#
odbedit -c clean
# start mhttpd on default port (Mongoose https version)
mhttpd -D # you can optionally restrict access to localhost and/or other specified hosts - see [[mhttpd]]
# start mserver on default port (use argument -p to use a different port)
mserver -D # start mserver on default port (1175) or a different port using -p option - see [[mserver]]
# the port must match that of MIDAS_SERVER_HOST in .cshrc (.bashrc) - see above
# access must be specifically allowed - see [[#Grant Remote host access permission||above]]
xterm -e ./analyzer &
## start frontend on remote host
#ssh $REMOTE_HOST $HOME/online/bin/start_frontend -O >& $HOME/online/bin/start_frontend.log &
mlogger -D
#end file

Note that command to start the frontend on the remote host is commented out. It can be uncommented once the start_frontend script is written and tested.

Shut down any clients that are running already by running odbedit on the MIDAS ''experiment host'' and execute the new startup script {{File|name=start_daq.sh}} (make it executable if desired)

[mhostpc] odbedit
[local:e777:S]/>sh all
[mhostpc] ./start_daq.sh 
and check that {{Utility|name=odbedit}} still works on the ''REMOTE host''

[rlxhost] odbedit
[mhostpc.triumf.ca:1175:e777:Stopped]/>ls 

You may wish to modify the script to restart missing clients only.

=== Create script to start frontend ===
The startup file {{File|name=start_daq.sh}} invoked a script {{File|name=start_frontend}} to start the remote frontend. This line was commented out (see [[#Create experiment startup script|above]]). Once the frontend is working, create a script to start the frontend (e.g. file {{Filepath|path=$HOME/online/bin/start_frontend}}).

#!/bin/tcsh
# Script to start frontend running on $REMOTE_HOST
#
# This script runs on the REMOTE HOST
#
echo "starting frontend for experiment $MIDAS_EXPT_NAME "
xterm -geometry 150x50+800+0 -fg white -bg blue -title "Frontend" -e "$HOME/online/fe_32/frontend" &
echo "Frontend task has been started"
exit

Check that this script works by running it on the REMOTE host. If successful,
run the commented command in an xterm on the ''experiment host''
[mhostpc] ssh $REMOTE_HOST $HOME/online/bin/start_frontend -O >& $HOME/online/bin/start_frontend.log & 
This command relies on password-less access set up with SSH keys (see [[#Experimental setup (Remote)]]).
If successful, uncomment line in {{File|name=start_daq.sh}}, and run the script to check that it can start the remote client(s) from the MIDAS ''experiment host'' (first shutting down all running clients) :

[mhostpc] odbedit -c "sh all"
[mhostpc] ./start_daq.sh 

Next [[#Customizing your experiment|customize your experiment]].

== Customizing your experiment ==

Here are some of the more common options provided for users to customize their experiments according to their own preferences:

; Write frontend(s) to read out and control your hardware
: Modify the example frontend and Makefile (from $MIDASSYS/examples/) you have already copied and built so it accesses your hardware.

: The example Frontend.c is an example of [[Frontend user code]]. You can find documentation about the frontend structure under [[Frontend Application]], [[Frontend Operation]] and [[Frontend user code]]. You may need to set up some [[Event Notification (Hot-Link)|Hot-Links]] in the frontend. The MIDAS package includes device drivers for some of the more common hardware. See [[MIDAS Driver Library]] for more information.

; Modify analyzer(s) for your data
: Modify the example analyzer code as necessary to analyze the data from your frontends.

; Slow Controls
: Control your slow-controls equipment with slow-control frontend(s) optionally using the [[#MIDAS Package Installation|MSCB]] system.

; Data logger
: Set up the data logger [[mlogger]] to log data to a storage device. Many options are available - see [[Data Logger]], [[mlogger]] and [[/Logger ODB tree]] for instructions.

; Allow programs to be restarted from the [[Programs Page]]
: See mhttpd [[Programs Page]] and [[/Programs ODB tree]]

; Write scripts which run at start and end of run
: See [[/Programs ODB tree#Execute on start run]] and [[/Programs ODB tree#Execute on stop run]]

; Create a shutdown script
: Create a script similar to the startup script to shutdown the experiment

; Lazy logger
: Set up the [[lazylogger]] for archiving data files for storage.

; Create edit-on-start parameters
: See [[Edit-on-start Parameters]]

; Set up the alarm system
: See [[Alarm System]] and [[Alarms Page]]

; Set up history system
: See [[History System]] and [[History Page]]

; Set up the Electronic logbook (Elog)
: See [[Elog]] for details

; Set up the Sequencer
: The [[Sequencer Page#Sequencer]] can be used to run a sequence of runs, changing parameters as needed.

; Write a webserver Custom page
: To better control and monitor your experiment, you may want to write a [[Custom Page]].

[[Category:Installation]]