ODB Access and Use

• Midas Doc
• Feature Listing
• Application Listing
• Online Database
• FAQ
• Index

Links

• odbedit
• Odbedit command list
• mhttpd ODB page
• ODB structure

Purpose

This page gives details on how users may access and use the ODB.

Introduction

The MIDAS Online Data Base (ODB) is the main communication platform between MIDAS applications. It contains all the information needed to run an experiment. A description of the ODB Structure and a list of all the ODB trees used by the system can be found in Online Database.

ODB Keys

The ODB is hierarchically structured, similar to a file system, with directories and sub-directories (or trees and subtrees). The data are stored in key/data pairs, similar to the Windows NT registry. ODB keys can be dynamically created and deleted. The data associated with a key can be of different types such as: byte, words, double words, float, strings, etc. or arrays of any of those.

A key can also be a directory or a symbolic link. ODB Key names are case-independent, and may contain spaces.

As of Jan 2017, ODB key names are now checked to ensure they are UTF-8 compliant. This check will show up in (at least) two ways:

1. Attempts to create a new ODB variable if the ODB key is not UTF-8 compliant.
2. When a program first connects to the ODB, it runs a check to ensure that the ODB is valid. This will now include a check that all key names are UTF-8 compliant. Any non-UTF8 compliant key names will be replaced by a string of the pointer to the key.

This behaviour (checking UTF-8 compatibility and automatically fixing ODB names) can be disabled by setting an environment variable

MIDAS_INVALID_STRING_IS_OK

It doesn't matter what the environment variable is set to; it just needs to be set. Note also that this variable is only checked once, when a program starts.

Accessing the ODB

The MIDAS ODB can be accessed interactively in two ways:

• odbedit application command line.
• Midas server mhttpd ODB page

For non-interactive access,

• The MIDAS library provides a complete set of C/C++ functions to manage and operate on ODB keys. Examples of many of these routines in use can be found in Frontend user code. For the full list, see the `db_*` set of functions in midas.h / midas.cxx.
• As of May 2020, an object-oriented C++ interface to the ODB is provided, which provides a "magic" C++ map/dictionary that auto-syncs with the ODB. See Odbxx.
• A set of Javascript functions are provided for web-based access in Custom Pages.
• A Python library is provided that provides a very pythonic interface to the ODB.
• If you're not using python/C++/javascript, your scripts can call the odbedit command with -c argument. Examples of this can be found in Script Access

Creating the ODB

After installation of MIDAS, before any other applications are started, the ODB must be created by the user (see creating the ODB ). Running odbedit to create the ODB also causes the trees /Runinfo, /Experiment, /System to be created and filled with default values. Other system applications will automatically create their own initial ODB structure filled with default values (for example, the data logger creates the /Logger ODB tree. The user may then modify these configurations to fit his/her requirements (see customizing). A list of the trees in the ODB used by the system can be found here.

Shared-memory files

Creating the ODB also creates all the shared-memory files needed for the experiment (see list below). By default, these files will be created in the area indicated in the exptab file for your experiment.

.ALARM.SHM  .ELOG.SHM  .HISTORY.SHM  .MSG.SHM  .ODB.SHM  .SYSMSG.SHM  .SYSTEM.SHM

Customizing the ODB

The user customizes the ODB for their experiment by modifying the values of "system" parameters (e.g. Keys in the ODB /Logger tree, or adding extra keys under existing trees as needed, e.g. "Edit on Start" parameters under the /Experiment ODB tree. The user may creates "optional" trees e.g. (/Script ODB tree, /Custom ODB tree) populating them with keys as required. The user is also free to create his/her own tree structure under root (/) for his/her own purposes.

Creating ODB keys

The easiest way to create, set (and delete if necessary) ODB keys is to use the ODB Page of the web server mhttpd. This also includes creating subdirectories and symbolic links. See ODB Page#Create an ODB key for details.

Alternatively, odbedit can be used as demonstrated in the following example. Odbedit is more powerful, also allowing the user to for example re-order, copy or rename the keys (see odbedit command list).

Once a directory is created, keys can be created of the types supported by MIDAS (see Midas Data Types), e.g. INT DWORD BOOL FLOAT DOUBLE STRING. Arrays of all these types can also be created. Values are then assigned to the keys using the odbedit command set.

Example

This example shows how to create an ODB tree (/Custom) using odbedit. Creating the same tree using the ODB Page is easy (see ODB Page#Create an ODB key for more information).

The odbedit commands are shown in red. Comments are shown in green and preceded by a "#".

$ odbedit   
[local:midas:S]/>pwd    # show current directory (tree) 
/
[local:midas:S]/>mkdir custom/images    # make directory "custom" and subdirectory "images"
[local:midas:S]/>ls  # list all keys
System
Programs
Experiment
Runinfo
Alarms
Custom
[local:midas:S]/>cd custom   # change directory to "/custom" 
[local:midas:S]/custom>ls
images
[local:midas:S]/custom>create string "custom page&"  # Create a key of type STRING. Note use of quotes as key name contains a space 
String length [32]: 128
[local:midas:S]/custom>set "custom page&" /home/midas/custom/custom.html  # Set key to a value. 
#             Could type instead "set cus" followed by tab to demonstrate tab-completion 
[local:midas:S]/custom>ls -r  # list recursively 
custom
   images
   custom page&           /home/midas/custom/custom.html

Creating arrays

Creating arrays in the ODB is simple using the ODB Page (see ODB Page#Create an ODB key for more information).

The following example demonstrates the use of odbedit to create an array, set its elements to various value, and expand it. An array can be truncated using the odbedit trunc command.

[local:pol:S]/test>create int fred[5]
[local:pol:S]/test>set fred[*] 5
[local:pol:S]/test>set fred[1..3] 6
[local:pol:S]/test>set fred[8] 9
[local:pol:S]/test>ls
fred
                               5
                               6
                               6
                               6
                               5
                               0
                               0
                               0
                               9

Reordering ODB keys

Sometimes it may be desired to reorder the ODB keys once created. This must be done using odbedit.

The odbedit command move is used for this purpose as in the following example:

The key "custom" can be moved to the top (or bottom) of the list, e.g.

[local:midas:S]/>ls
System
Programs
Experiment
Runinfo
Alarms
Custom

The key "Custom" can be moved to the top

[local:midas:S]/>move custom top
[local:midas:S]/>ls
Custom
System
Programs
Experiment
Runinfo
Alarms

or to any position, e.g.

[local:midas:Stopped]/>move custom 1
[local:midas:Stopped]/>ls
System
Custom
Programs
Experiment
Runinfo
Alarms

Create a subdirectory in the ODB

Creating a subdirectory is very easy to do using the ODB Page. See Create an ODB key.

Using odbedit, select the desired directory using odbedit command cd then make the directory with odbedit command mkdir e.g.

[local:test:S]cd /Experiment
[local:test:S]mkdir "edit on start"

Setting multiple keys at once

There are occasions where multiple keys of different subfolder need to be modified; an Equipment may have multiple boards each one with its own configuration. For this reason the set command can expand paths to match multiple keys with a single command. An example using the standard ODB:

[local:test:S]set /Alarms/Alarms/*/Active n

will deactivate all alarms.

Create a symbolic link in the ODB

Creating a symbolic link is easy to do using the ODB Page. See Create an ODB key.

Creating the same symbolic link with odbedit

[local:test:S]cd /Experiment/edit on start
[local:test:S]Edit on Start> ln "/logger/write data" "write data"
[local:test:S]Edit on Start> ls
write data -> /logger/write data
                               y

Limiting ODB values to specific options

If you have an ODB key that should only accept certain values (e.g. you want the user to specify a file-type, but you only support "json" and "xml"), you can enforce this through the web interface. Note that there is no low-level enforcement - it is purely a convenience for webpage use! Power-users are still able to set "unapproved" values via C++/Python/Javascript/odbedit...

To enable this for ODB key x, create an ODB key called Options x in the same directory. This "options" key should be a list of the valid values. For example:

[local:test:S]Edit on Start> ls -l
Key name                        Type    #Val  Size  Last Opn Mode Value
---------------------------------------------------------------------------
experiment number               INT32   1     4     21m  0   RWD  9998
Options experiment number       INT32   3     4     21m  0   RWD
                                       [0]             1234
                                       [1]             9998
                                       [2]             9999

In the regular midas ODB webpage, "experiment number" will be shown as a dropdown (select) element rather than a free-form text input (in this case the valid options would be 1234/9998/9999). In custom pages, you could create an modbselect element and specify data-auto-options="1" to populate these options. The "start a run" page also supports this feature.

Again, there is no low-level enforcement of these options; it is purely a user-level convenience for the web interfaces.

Save and reload the ODB

The ODB can be saved at its current position in several formats (ascii, xml as a c structure) using the odbedit save command, and reloaded from a saved file with the odbedit load command.

NOTE

JSON format has been added (January 2016). See #JSON.

e.g. to save the ODB into a file in the local directory in ASCII format :

[local:exp:Stopped]/>save my_exp.odb 

and to load the ODB from a previously saved file in the local directory :

[local:npet:Stopped]/>load saved.odb

Since the ODB may become corrupted, a copy is saved automatically at the beginning of each run. Since this file is overwritten each time, users often also save a copy (that will not be overwritten) at the end of each run (see ODB Dump) along with the data for that run. (The ODB contents can also be sent to the logging channel and saved with the run data - see ODB Dump in /Logger/Channels subtree).

Users can then return their ODB to the state it was in for any particular run.

The entire database need not be loaded. Saved ASCII files can be made of just a part of the database, and these can be reloaded into the database. Since the full path is given in the saved file, the file can be loaded from any position in the database.

The saved ASCII file may of course be edited prior to loading, if keynames or values need to be changed. If the keys in the load file do not exist, they will be created. If they do exist, the values from the file will be loaded.

experim.h include file

The odbedit make command creates "experim.h", a file containing a C structure and ascii representation of part of the ODB. The file is created in the current directory. e.g.

[local:midas:S]/>make
Analyzer "Analyzer" not found in ODB, skipping analyzer parameters.
"experim.h" has been written to /home/midas/online

The file experim.h contains C structures for the ODB trees /Experiment, /Analyzer/Parameters, and /Equipment/<equipment-name>/Settings if these trees are present. All C structures are accompanied by a string representation which can be used in the db_create_record function to setup an ODB structure which matches the required C structure, ensuring that the required keys are present in the ODB (see Frontend or setting up a hot link).

If the name of the analyzer is not the default (i.e. "Analyzer"), then in order to include the analyzer section, the ODB key /<Analyzer-name>/Parameters has to be present, where <Analyzer-name> is the name of the analyzer.

The command used is then "make <Analyzer-name> ".

An example experim.h can be found at [1].

JSON

New commands have been added (January 2016) which correspond to JSON encoding of ODB data see mjsonrpc#JSON encoding of ODB data. The command odbedit command json encodes all ODB data and metadata. ODB can be fully reloaded or restored from ODB JSON save files.

$ odbedit
[local:js:S]>cd /Equipment/rpcexample/variables
[local:js:S]Variables>ls
SLOW
                                1273
                                1376610644
                                74.31448279461522
[local:js:S]Variables>json
status: 1, json: {
  "SLOW/key" : { "type" : 10, "num_values" : 3, "access_mode" : 7, "last_written" : 1448506616 },
  "SLOW" : [ 1273, 1376610644, 7.4314482794615216e+01 ]
}

The odbedit command jsls encodes a single ODB directory and returns the full information printed by the odbedit command ls -l.

[local:js:S]Variables>jsls
jsls "/Equipment/RpcExample/Variables", status: 1, json: {
  "SLOW/key" : { "type" : 10, "num_values" : 3, "access_mode" : 7, "last_written" : 1448506616 },
  "SLOW" : [ 1273, 1376610644, 7.4314482794615216e+01 ]
}

The odbedit command jsvalues list the data in a format for exporting ODB data to web pages: links are followed to their final values and ODB key names are converted to lower case for use with case-sensitive languages such as Javascript.

[local:js:S]Variables>jsvalues
status: 1, json: {
  "slow/name" : "SLOW",
  "slow/last_written" : 1448506616,
  "slow" : [ 1273, 1376610644, 7.4314482794615216e+01 ]
}

Corrupted ODB

If the ODB becomes corrupted, odbedit may no longer work, and other clients will also fail to open the database. In this case, the old ODB should be deleted and a new one created. The contents of the ODB can be reloaded from a saved ODB file. Follow the instructions in Recovery from Corrupted ODB.

Accessing the ODB from a script

The external command feature of odbedit (see Odbedit#Using an external command) allows for sophisticated scripts to be created that can manipulate the ODB. Such scripts can for example

• check ODB parameters prior to beginning of run
• send run parameters to the electronic logbook
• set some ODB parameters when a script button on the mhttpd main status page is pressed
• load a particular saved odb file when a button on a custom page is pressed (e.g. load a "tune")

Examples of shell scripts that may send odbedit commands :

• script run at end-of-run or begin-of-run (see below)
• script run when a script-button or Customscript button is pressed

Script writers may find it useful to [#Message Page and message log|send messages to the MIDAS message log]] to indicate progress or error.

Example end-of-run script

The following is part of a script run at end-of-run, where it reads some parameters from the odb and saves them in a temporary file to be sent to the elog.

To make the script run at end of run, the name of the script is entered in the ODB key /Programs/Execute on stop run (see /Programs ODB tree#Execute on stop run). Similarly, a script may be made to run at the beginning of run by entering its name in the ODB key /Programs/Execute on start run (see /Programs ODB tree#Execute on start run).

#!/bin/tcsh
# This script is started at the end of each run. It takes some parameters
# from the odb and creates an entry in the elog 
.....
# Start collecting information from ODB first
set Run_number = `odb -e $MIDAS_EXPT_NAME -c 'ls "/Runinfo/Run number"'`
set number = `echo $Run_number | awk '{print $3}'`
set sample = `odb -e $MIDAS_EXPT_NAME -c 'ls "/Experiment/Edit on Start/sample"'`
set Sample = `echo $sample | awk '{print $2}'`
.....
# Now create the temporary file to be sent to the elog
echo "Run # $number" >> $fin
odb -e $MIDAS_EXPT_NAME -c 'ls "/Runinfo/Start time"' >> $fin
odb -e $MIDAS_EXPT_NAME -c 'ls "/Runinfo/Stop time"' >> $fin
echo "$Sample at T = $T K, H = $H T and RF = $RF mW">> $fin
.....

Chat command

The odbedit command chat can be used to communicate between users, each running odbedit clients. The chat messages are also (sing Aug 2016) sent to the mhttpd Chat Page. The odbedit command msg can be also be used interactively to generate chat messages. If chat messages do not appear in the Chat Page see #Troubleshooting.

[local:js:S]/>chat
Your name> george
Exit chat mode with empty line.
> Are you there?
[george,USER] Are you there?
19:33:59 [fred,USER] Hallo George

Msg command - send a message

The parameters of the odbedit command msg have been changed (August 2016). The odbedit command msg now works differently interactively than from the command line. A hidden parameter "facility" has been added, which is either set to "chat" or "midas".

To web server Chat page

Interactively, the odbedit command msg works like the #Chat command, sending a chat message which appears on the web server Chat Page and on any other odbedit session for that experiment. The chat message does NOT appear in the MIDAS message log or Message Page. In this case, the "facility" is hard-coded as "chat" and there are two parameters, "user" and "message", e.g.

[local:js:S]/>msg fred "hi, are you there?"
[fred,USER] hi, are you there?

To web server Message Page and message log

To send a message to the webserver Message Page and the message logfile ( midas.log) use the command line with the odbedit command msg. In this case, the "facility" is hard-coded as "midas", and there are three parameters, two of which are optional:

type (optional)

Default is 8 (MT_USER). If supplied, it must be set to one of the numerical values 1 (MT_ERROR), 2 (MT_INFO), 4 (MT_DEBUG) or 8 (MT_USER), as defined in midas.h.

name (optional)

Ignored unless the type is MT_ERROR. Default is "script".

message

the message to be sent, enclosed in quotes if it includes spaces

See Table 1 below for examples.

Table 1 : examples of odbedit command msg

odbedit -c 'msg command'	Type		Name	Resulting Message	Explanation
'msg 2 "testing 123" '	2	MT_INFO		[ODBEdit,INFO] testing 123	Message type supplied. Program name set to "ODBEdit".
'msg "testing 123" '	8	MT_USER		[script,USER] testing 123	Message type omitted, defaults to MT_USER which sets program name to "script".
'msg 1 my_name "error message" '	1	MT_ERROR	my_name	[ODBEdit,ERROR] [odbedit.cxx:2490:my_name,ERROR] error message	Name parameter is supplied. Error message appears with a red background.
'msg 1 "error message" '	1	MT_ERROR		[ODBEdit,ERROR] [odbedit.cxx:2490:script,ERROR] error message	Name parameter omitted, default name is "script"

Troubleshooting

If the message does not appear on the Chat Page (interactive) or Message Page (command line), check that the ODB key /Logger/Data dir is set to a full directory path, such as /home/user/online/js, not a relative path such as the current directory "./". If set to a full directory path, all clients will use the same midas.log file, regardless of which directory they were started from. With a relative path, mhttpd and odbedit must be started in the same directory for messages/chat to appear in the appropriate mhttpd page.