Midas Core: Difference between revisions

From MidasWiki
Jump to navigation Jump to search
(The midas documentation has moved to https://midas.triumf.ca/docs/)
Tag: Replaced
 
(One intermediate revision by one other user not shown)
Line 1: Line 1:
{{Pagelinks}}
The midas documentation has moved. See [https://midas.triumf.ca/docs/develop/getting_started/what_is_midas/#midas-101 Midas 101] on the What is midas page and the [https://midas.triumf.ca/docs/develop/features/ Feature guide] for the new documentation about the main components of midas.
==Midas components==
The main elements of the MIDAS package are listed below with a short description of their function. Please refer to the [[Overall Midas system diagram]] to see how these elements interact to form the MIDAS system.
 
*The Buffer Manager - ''Handles the experimental data transfer from the frontend to the backend'' -
*Message System - ''Dedicated buffer system to handle Midas internal messages'' -
*Online Database (ODB) - ''Dynamic Database holding all user information related to a given experiment'' -
*Frontend Acquisition code - ''User code defining what is to be acquired/control over time during an active experiment'' -
*MIDAS Server Remote access server (RPC server) - ''Interface connecting remote Midas client to your local experiment'' -
*Data Logger Data storage - ''Midas client handling the recording of the collected data to physical storage media'' -
*Analyzer Data analyzer - ''Midas client able to connect to a Midas data stream or to a saved Midas data file for data analysis'' -
*Run Control Data flow control -
*Slow Control system Device monitoring and control.
*History system Event history storage and retrieval.
*Alarm System Overall system and user alarm.
*Electronic Logbook Online User Logbook.
*Run Sequencer
 
===The Buffer Manager===
The "buffer manager" consists of a set of library functions for event collection and distribution. A buffer is a shared memory region in RAM, which can be accessed by several processes, called "clients". Processes sending events to a buffer are called "producers". Processes reading events from the buffer are called "consumers".
 
A buffer is organized as a FIFO (First-In-First-Out) memory. Consumers can specify which type of events they want to receive from a buffer. For this purpose each event contains a MIDAS header with an event ID and other pertinent information.
 
Buffers can be accessed locally through the shared memory or remotely via the MIDAS server [[mserver]]  acting as an interface to that same shared memory.
 
A common problem in DAQ systems is the possible crash of a client, such as a user analyzer. This can cause the whole system to hang up, and may require a restart of the DAQ causing a loss of both time and, eventually, precious data. In order to address this problem, a special watchdog scheme has been implemented. Each client attached to the buffer manager signals its presence periodically by storing a time-stamp in the shared memory. Every other client connected to the same buffer manager can then check if the other parties are still alive. If not, proper action is taken consisting in removing the dead client hooks from the system, leaving the system in a working condition.
 
===Message System===
Any client can produce status or error messages with a single call using the MIDAS library. These messages are then forwarded to any other clients who may be available to receive these messages, as well as to a central log file system. The [[Message System]] is based on the buffer manager scheme, but with a dedicated header to identify the type of message. A dedicated buffer (i.e. shared memory) is used to receive and distribute messages. Predefined message types contained in the MIDAS library cover most of the message requirements. See [[mlogger#logging in MIDAS]] for more details.
 
===Online Database (ODB)===
In a distributed DAQ environment, configuration data is usually stored in several files on different computers. MIDAS, however, uses a different approach: all relevant data for a given experiment are stored in a central database called the "Online DataBase" (ODB). This database contains run parameters; logging channel information; condition parameters for front-ends and analyzers; slow control values; status and performance data; and any information defined by the user.
 
The main advantage of this concept is that all programs participating in an experiment have full access to these data without having to contact different computers. A possible disadvantage could be the extra load put on the particular host serving the ODB. As the access to such a database can be remote, the connection is performed through an RPC layer. MIDAS includes its own RPC which has been optimized for speed. Byte ordering (i.e. big/little endian) is taken care of, such that cross-platform database access is possible, with the advantage that the RPC doesn't define a byte ordering. Instead it uses the transmitter type, and converts to the required byte ordering only if needed by the receiver. Measurement shows that up to 50,000 accesses per second with a local connection, and around 500 accesses per second remotely over the MIDAS server, can be obtained (numbers from 1990).
 
The ODB is hierarchically structured, similar to a file system, with directories and sub-directories (see [[Online Database#The ODB Structure|ODB Structure]]) . The data are stored in key/data pairs, similar to the Windows NT registry. Keys can be dynamically created and deleted
(see [[odbedit]] or {{Utility|name=mhttpd}} Web Server [[ODB Page]]). The data associated with a key can be of different types such as: byte, words, double words, float, strings, etc. or arrays of any of those. A key can also be a directory or a symbolic link (c.f. Unix).
 
<div id="MidasCoreHere"></div>
 
The MIDAS library provides a complete set of functions [http://ladd00.triumf.ca/~daqweb/doc/midas-devel/doc/html/group__odbfunctionc.html] to manage and operate on these keys.
Any ODB client can register a  "hot-link" between a local C-structure and any element of the ODB. The hot-link mechanism ensures that whenever a client (program) changes a value in this ODB sub-tree, the local C-structure automatically receives an update of the changed data. Additionally, a client can register a callback function which will be executed as soon as the hot-link's update has been received. For more information see [[Event Notification (Hot-Link)]] .
 
===MIDAS Server===
For remote access to a MIDAS experiment, a remote procedure call (RPC) server is available [[mserver]]. It uses an optimized MIDAS RPC scheme for improved access speed. The server can be started manually or via inetd (UNIX) or as a service under Windows NT. For each incoming connection it creates a new sub-process which serves this connection over a TCP link. The MIDAS server not only serves client connections to a given experiment, but takes the experiment's name as a parameter meaning that only one MIDAS server is necessary to manage several experiments on the same node.
 
===Frontend===
The frontend program refers to a task running on a particular computer which has access to hardware equipment. Several frontends can be attached simultaneously to a given experiment. Each frontend can be composed of multiple Equipments. The term "EQUIPMENT" refers to a single or a collection of sub-task(s) meant to collect and regroup logical or physical data under a single and uniquely identified event.
 
The frontend program is composed of a general framework which is experiment-independent, and a set of template routines for the user to fill in. This program will:
* Register the given Equipment(s) list to a specific MIDAS experiment.
* Provide the means of collecting data from hardware sources defined by each Equipment Read function.
* Gather these data in a known format (e.g. Fixed, MIDAS) for each equipment.
* Send these data to the buffer manager either locally or remotely.
* Periodically collect statistics of the acquisition task, and send them to the Online Database.
 
The frontend framework sends events to the buffer manager and optionally a copy to the ODB. A "Data cache" in the frontend and on the server side reduces the amount of network operations, pushing the transfer speed closer to the physical limit of the network configuration.
 
The data collection in the frontend framework can be triggered by several mechanisms. Currently the frontend supports the following type:
 
* Periodic events: scheduled event based on a fixed time interval. They can be used to read information such as scaler values, temperatures etc.
* Polled events: hardware trigger information read continuously which in turns if the signal is asserted will trigger the equipment readout.
* LAM events: generated only when pre-defined LAM is asserted (CAMAC).
* Interrupt events: generated by particular hardware device supporting interrupt mode.
* Slow Control events: special class of events that are used in the slow control system.
 
Each of these types of trigger can be enabled/activated for a particular experimental State, Transition State, or a combination of any of them. Examples such as "read scaler event only when running" or "read periodic event if the run state is not paused and on all transitions" are possible.
 
Dedicated header and library files for hardware access to CAMAC, VME, Fastbus, GPIB and RS232 are part of the MIDAS distribution set. For full details see [[Frontend Operation]].
 
===Data Logger===
The data logger [[mlogger]] is a client running on the backend computer receiving events from the buffer manager and saving them onto disk, tape or via FTP to a remote computer. It supports several parallel logging channels with individual event selection criteria. Data can currently be written in five different formats: MIDAS binary, ASCII, ROOT and DUMP (see [[Event Structure]]).
 
Basic functionality of the logger includes:
 
Run Control based on:
* event limit not reached yet.
* recorded byte limit not reached yet.
* logging device not full.
* logging selection of particular events based on Event Identifier.
* auto restart feature allowing logging of several runs of a given size or duration without user intervention.
* recording of ODB values to a so-called MIDAS History System
* recording of the ODB to all or individual logging channels at the begin-of-run and end-of-run states, as well as to a separate disk file in XML or ASCII format.
For more information see [[mlogger|Logging in MIDAS]] .
 
===Analyzer===
The Analyzer is a backend task (as opposed to the frontend). As in the front-end section, the analyzer provided by MIDAS is a framework on which the user can develop his/her own applications. This framework can be built for private analysis (no external analyzer hooks) or specific analysis packages such as HBOOK, ROOT from the CERN (none of those libraries are included in the MIDAS distribution). See SECTION 7: Data Analysis for more information.
 
The analyzer takes care of receiving events (a few lines of code are necessary to receive events from the buffer manager); initializing the HBOOK or ROOT system; and automatically booking N-tuples/TTree for all events. Interface to user routines for event analysis is provided.
 
The analyzer is structured into "stages", where each stage analyses a subset of the event data. Low level stages can perform ADC and TDC calibration, while high level stages can calculate "physics" results. The same analyzer executable can be used to run online (where events are received from the buffer manager) and off-line (where events are read from file). When running online, generated N-tuples/TTree are stored in a ring-buffer in shared memory. They can be analysed with PAW without stopping the run.
 
When running off-line, the analyzer can read MIDAS binary files, analyse the events, add calculated data for each event and produce a HBOOK RZ output file which can be read in by PAW later. The analyzer framework also supports analyzer parameters. It automatically maps C-structures used in the analyzer to ODB records via Event Notification (Hot-Link). To control the analyzer, only the values in the ODB have to be changed, which are automatically propagated to the analyzer parameters. If analysis software has been already developed, MIDAS provides the functionality necessary to interface the analyzer code to the MIDAS data channel. Support for languages such as C, C++ is available.
 
===Run Control===
As mentioned earlier, the Online Database (ODB) contains all the pertinent information regarding an experiment. For that reason a run control program requires only to access the ODB. A basic program supplied in the package called [[odbedit]] provides a simple and safe means of interacting with the ODB. However, to access all the MIDAS capabilities, the MIDAS Web-based Run Control utility [[mhttpd]] should be used.
 
Three "Run States" define the state of the MIDAS data acquisition system: Stopped, Paused, and Running. In order to change from one state to another, MIDAS provides four basic "Transition" functions: TR_START, TR_PAUSE, TR_RESUME, and TR_STOP. During these transition periods, any MIDAS client registered to receive notification of such a transition will be able to perform dedicated tasks in either synchronized or asynchronized mode, within the overall run control of the experiment.
 
In order to provide more flexibility to the transition sequence of all the MIDAS clients connected to a given experiment, each transition function has a transition sequence number attached to it. This transition sequence is used to establish within a given transition the order of the invocation of the MIDAS clients (from the lowest sequence number to the highest). See  [[Run States and Transitions#Run Transition Priority]] for details.
 
===Slow Control===
The [[Slow Control System]] is a special front-end equipment or program dedicated to the control of hardware modules based on user parameters. It takes advantage of the Online Database and its hot-link capability. Demand and Measured values from Slow Control system equipment like high voltage power supplies or beam line magnets are stored directly in the ODB.
 
To control a device it is then enough to modify the demand values in the database. The modified value is automatically propagated to the slow control system, which in turn uses the specific device driver to apply the change to the particular hardware. Measured values from the hardware are periodically sent back to the ODB to reflect the current status of the sub-system.
 
The Slow Control system is organized in a Object Oriented way with 3 levels of calls such Class, Device and Bus drivers. Each Class driver refers to a particular set of functionality of that class i.e. High-Voltage, Temperature, General I/O, Magnet etc. The implementation of the device-specific calls is done in a second stage "Device Driver" while the actual hardware implementation is done in a third layer "Bus Driver". The current MIDAS distribution already has some device driver for general I/O and commercial High Voltage power supply systems (see supported hardware ). The necessary code composing the hardware device driver is kept simple by only requiring a "set channel value" and "read channel value" for example. For the High Voltage class driver, a GUI (graphical user interface) under Windows or Qt is already available. It can set, load and print high voltages for any devices of that class.
 
===History system===
The MIDAS [[History System]] is a recording function embedded in the MIDAS logger [[mlogger]]. Parallel to its main data logging function of defined channels, the MIDAS logger can store slow control data and/or periodic events on disk file. Each history entry consists of the time-stamp at which the event has occurred, and the value[s] of the parameter to be recorded.
 
The activation of a recording is not controlled by the history function, but by the actual equipment (see MIDAS History System). This permits a higher flexibility of the history system such as dynamic modification of the event structure without restarting the MIDAS logger. At any given time, data-over-time relations can be displayed through the web with the [[History Page]] of the MIDAS Web-based Run Control utility [[mhttpd]], or queried from the disk file through the MIDAS [[mhist]] utility,
 
The history data extraction from the disk file is done using low level file functions giving similar results as a standard database mechanism but with faster access time. Due to its simple use and good display quality, this section has been reworked to be able to handle larger number of parameters (see [[History System]] ).
 
===Alarm System===
The MIDAS [[Alarm System]] is a built-in feature of the MIDAS server. It acts upon the description of the required alarm defined in the Online Database (ODB). Currently the internal alarms supports the following mechanism:
 
ODB value over fixed threshold at regular time interval, a pre-defined ODB value will be compared to a fixed value.
MIDAS client control During Run State transitions, pre-defined MIDAS client names will be checked if currently present.
General C-code alarm setting Alarm C function permitting to issue user defined alarm.
The action triggered by the alarm is left to the user through the means of a detached script. But basic alarm report is available such as:
 
* Logging the alarm message to the experiment log file.
* Sending an "Electronic Log message" (see Electronic logbook (Elog)).
* Interrupt data acquisition. For more information see MIDAS [[Alarm System]] and [[/Alarms ODB tree]] .
 
===Electronic Logbook===
The Electronic logbook is a feature which provides the experimenter an alternative way of logging his/her own information related to the current experiment. This electronic logbook may supplement or complement the standard paper logbook and in the mean time allow "web publishing" of this information. The internal electronic logbook is a built-in feature of MIDAS, and the electronic logbook information is accessible from any web browser as long as the MIDAS web server ( [[mhttpd]]) is running in the background of the system. The external electronic log [http://midas.psi.ch/elog/] can be used instead.
In this case the ELOG runs on a separate port but is still accessible from the [[Elog Page]]. For more information see  {{Utility|name=mhttpd}} [[Elog Page]].
 
 
===Run Sequencer===
A [[Sequencer]] for starting and stopping runs is available ( Jun 2011). This allows the user to program a set of runs to be performed automatically. Conditions may be changed between runs, and each run may be stopped after a time or when a certain condition is reached.
 
[[Category:Contents]] [[Category:Buffer]] [[Category:Messages]] [[Category:ODB]] [[Category:History]] ]]Category:Alarms]] [[Category:Frontend]]
[[Category:Analyzer]] [[Category:Data Logging]] [[Category:Run control]] [[Category:Slow Control]] [[Category:Sequencer]]  [[Category:Driver]]
[[Category:Remote Access Server]]

Latest revision as of 22:01, 30 October 2026

The midas documentation has moved. See Midas 101 on the What is midas page and the Feature guide for the new documentation about the main components of midas.

Retrieved from "https://daq00.triumf.ca/MidasWiki/index.php?title=Midas_Core&oldid=3769"