Event Structure: Difference between revisions

From MidasWiki
Jump to navigation Jump to search
No edit summary
(The midas documentation has moved to https://midas.triumf.ca/docs/)
Tag: Replaced
 
Line 1: Line 1:
{{Pagelinks}}
The midas documentation has moved. See [https://midas.triumf.ca/docs/develop/features/acquiring_data/data_format/ Event data format] for the new documentation about the structure of midas events.
== Links ==
<div style="column-count:4;-moz-column-count:4;-webkit-column-count:4">
* [[MIDAS Event Construction]]
</div>
 
 
== Introduction ==
A midas data file contains midas events. The structure of these of events is detailed on this page. The same structure is used both for passing events between different programs (using [[Event_Buffer|event buffers]]) and for writing the data to disk.
 
Midas data files are written by the [[Mlogger|mlogger]] program and end in the .mid extension. Files can optionally be [[Keys_in_the_ODB_/Logger/Channels_subtree#Compress|compressed]] using gzip, lz4, bzip2 or pbzip2 (thus ending in .mid.gz etc).
 
Two event formats are supported by the frontends, although the vast majority of cases use the "MIDAS" format:
* [[#MIDAS Format Event|"MIDAS"]] 
* [[#FIXED Format Event|"FIXED"]]
 
Note that a frontend cannot write data directly into ROOT format. A conversion to ROOT may be done (e.g. by the data logger [[Mlogger]]) from one of the [[Equipment List Parameters#Format|supported formats]].
 
== Overall File Format ==
 
A midas file is simply a concatenation of midas events. There are no special headers or lookup tables, but it may contain a couple of special events (e.g. a dump of the ODB content). See [[#Special Events|special events]] for more details.
 
== MIDAS Format Event ==
The MIDAS event format is a variable length event format. It uses "banks" as subsets of an event. A bank is composed of a bank header followed by the data.
 
<span  style="color: black; background-color: mintcream; font-size: 110%; font-style:bold;>
Figure 1:  Structure of MIDAS event showing Event and Bank headers with data banks.</span>
<br>[[File:MEHBH.jpg]]
 
=== Event Header ===
Each event carries a 16-byte header, generated by the  [[Frontend_Operation#Frontend|Frontend]] with the
''bm_compose_event()'' routine and is used by consumers to distinguish between
different events.
 
The event header is defined in the EVENT_HEADER structure in the file ''midas.h'' in the midas package.
It has following structure:
 
; EventID : the [[Equipment_List_Parameters#EventID|EventID]] identifies the event by number. Usually 1 is used for triggered events, 2 for scaler events, 3 for HV events etc.
; TriggerMask : the  [[Equipment List Parameters# TriggerMask|trigger mask]] can be used to describe the sub-type of an event. A trigger event can have different trigger sources like "physics event", "calibration event", "clock event". These trigger sources are usually read in by the front-end in a pattern unit. Consumers can request events with a specific triggering mask.
; Serial number : The serial number starts at 0 and is incremented by the front-end for each event.
;Time Stamp : the time stamp is written by the front-end before an event is read out. It uses the time() function which returns the time in seconds since 1.1.1970 00:00:00 UTC.
;Event Data Size : The event data size contains the size of the event in bytes excluding the header.
 
<div id="byte-ordering"></div>
Event headers are always kept in the <b>byte ordering</b> of the local machine. If events are sent over the network between computers with different byte
ordering, the event header is swapped automatically, but not the event contents.
 
If the byte ordering of the contents of a complete event has to be swapped,
the routine ''bk_swap()'' can be used.
 
=== Data Area ===
The <b>data area</b> of the event can contain information in any user format (integer, real etc.), although only certain formats are supported when events are copied to the ODB or written by the logger in ASCII format.
 
The Data Area of a MIDAS event consists of a global Bank Header followed by one or more MIDAS Data Banks.
 
A Data Bank is a substructure of an event and can contain only one type of data, either a single value or an array of values.
Each Data Bank has an individual bank header containing a name of exactly four characters, which is treated as a bank ID.
 
==== Bank Header (global) ====
There is one global Bank Header per event, defined in the BANK_HEADER structure in the file ''midas.h''.
It has the following structure:
 
; All Bank Size : Size in bytes of the following data banks including their bank names
; Flags : The four LSB 0:3 indicate the version of the bank structures. This is currently "1" and used for endian detection (byte ordering). The 5th bit indicates that the banks are 32-bit banks (Bank Size being 4 bytes long) and the 6th bit indicates that the banks are 64-bit aligned using the BANK32A bank header.
 
The global Bank Header is initialized by the [[MIDAS_Event_Construction|bk_init()]], [[MIDAS_Event_Construction|bk_init32()]] or [[MIDAS_Event_Construction|bk_init32a()]] calls.
 
==== MIDAS Data Bank ====
Each data bank contains a header defined by the BANK (or BANK32) structures in the file ''midas.h''.
It has the following structure:
; Bank name :  four characters for the name of each bank. Each bank in an event must have a unique name.
; Bank type :  one of the  [[Midas Data Types]] TID_xxx values  to encode the data type
; Bank length :  size in bytes of the following data.
 
Following the header is the data in the format designated by the Bank type parameter (e.g. integer or float).
A separate MIDAS bank must be created for each data type needed. 
 
==== Bank Alignment ====
 
The data area of a bank must have a size of multiples of 8 Bytes. This is required to align the next bank header on a 8-Byte boundary in memory which allows faster access by the CPU. If a bank size is a few bytes short of a multiple of 8 bytes, the size is rounded up to the next multiple of 8 bytes and the remaining bytes in the bank may contain random data. Since midas event headers, bank headers (except the BANK32 bank header) and banks are all multiples of 8 bytes long, all data structures inside a midas event is 8-byte aligned for optimal CPU access. The BANK32 header is 12 Bytes long, causing the following Data area not being 8-byte aligned. If 8-byte alignment is important, the BANK32A header must be used instead. This can be done by initializing the bank structure with the [[MIDAS_Event_Construction|bk_init32a()]] call.
 
=== Example of MIDAS event ===
 
Figure 2 shows a MIDAS event containing banks coloured to match the structure in Figure 1. This has been
obtained from a MIDAS data file using the [[mdump]] application.
 
<span  style="color: black; background-color: mintcream;  font-size: 110%; font-style:bold;> Figure  2: Example of MIDAS banks dumped by  <span style="color:darkcyan;">''mdump''</span>.</span>
 
------------------------ Event# 2 --------------------------------<br>
<span style="background-color: rgb(204, 204, 255);">Evid:000d- Mask:0000- Serial:0- Time:0x4c7a6869- Dsize:48/0x30</span><br>
<span style="background-color: lightgreen;">\#banks:1 - Bank list:-SDAS-</span><br>
<br>
<span style="background-color: orange;">Bank:SDAS Length: 32(I*1)/8(I*4)/8(Type)Type:Real*4 (FMT machine dependent)</span><br>
<span style="background-color: pink;">  1-> 4.000e+00 1.000e+01 1.000e+00 3.400e+00 3.400e+00 3.400e+00 3.400e+00 3.400e+00</span><br>
<br>
------------------------ Event# 3 --------------------------------<br>
<span style="background-color: rgb(204, 204, 255);">Evid:0001- Mask:0000- Serial:0- Time:0x4c7a686b- Dsize:344/0x158</span><br>
<span style="background-color: lightgreen;">\#banks:2 - Bank list:-MPETMCPP-</span><br>
<br>
<span style="background-color: orange;">Bank:MPET Length: 304(I*1)/76(I*4)/76(Type) Type:Unsigned Integer*4</span><br>
<span style="background-color: pink;">    1-> 0x80010000 0x00000002 0x10010000 0x00004e21 0x80020000 0x00000002 0x20020000 0x000015f4<br>
  9-> 0x20020000 0x00001660 0x20020000 0x0000185f 0x20020000 0x0000191e 0x20020000 0x000019d6<br>
  17-> 0x40020000 0x00001a37 0x20020000 0x00001a77 0x20020000 0x00001ba2 0x10020000 0x00004e22<br>
  25-> 0x80030000 0x00000002 0x20030000 0x00001637 0x20030000 0x000018d1 0x20030000 0x000019bc<br>
  33-> 0x20030000 0x00001b35 0x20030000 0x00001bb2 0x10030000 0x00004e21 0x80040000 0x00000002<br>
  41-> 0x10040000 0x00004e22 0x80050000 0x00000002 0x20050000 0x000013c5 0x20050000 0x000017f2<br>
  49-> 0x20050000 0x0000185f 0x20050000 0x00001976 0x20050000 0x00001aa8 0x10050000 0x00004e21<br>
  57-> 0x80060000 0x00000002 0x20060000 0x000015c3 0x20060000 0x000018d8 0x20060000 0x0000198d<br>
  65-> 0x20060000 0x00001ac4 0x10060000 0x00004e22 0x80070000 0x00000002 0x20070000 0x00001747<br>
  73-> 0x20070000 0x000019ae 0x10070000 0x00004e21</span><br>
<br>
<span style="background-color: orange;">Bank:MCPP Length: 16(I*1)/4(I*4)/4(Type) Type:Unsigned Integer*4</span><br>
<span style="background-color: pink;">  1-> 0x00005e4c 0x0000352d 0x00006453 0x00006d5b</span><br>
<br>
--------------------------------------End of MIDAS Format ---------------------------------------------------------
 
 
== FIXED Format Event ==
The  "FIXED" format event is the simplest event format. The event length is fixed and is mapped to a C structure that is filled by the readout routine. Since the standard MIDAS analyzer cannot work with this format, it is only recommended for an experiment which uses its own analyzer and wants to avoid the overhead of a bank structure, or for monitoring purposes in the ODB.
 
The structure has to be defined twice: once for the compiler in the form of a C structure, and once for the ODB in form of an ASCII representation. There are several ways of doing this. The ASCII string may be supplied to the system as the [[Equipment List Parameters#Bank Definition/Init String|init string]] in the equipment list as follows:
[[File:FixedFormat.png|Example of definition of FIXED format using Equipment field "init string" ]]
 
Alternatively, the structure may be defined first in the ODB, under <span style="color: purple; font-style:italic;">/Equipment/<eqp_name>/Variables</span>,
and an ''experim.h'' file generated (see [[odbedit|experim.h]]. The structure is then supplied to the readout routine by the frontend program including "experim.h", as follows:
 
[[File:FixedFormatExperim.png|Example of structure from experim.h for a fixed event]]
 
To select "FIXED" format, the [[Equipment List Parameters|Equipment Definition]] must have the [[Equipment List Parameters#Format|format parameter]] set to "FIXED".
The Equipment definition for this fixed event might be:
 
{ "Info ODB",    /* equipment name */
    10, 0,        /* event ID, trigger mask */
    "",            /* no banks sent */
    EQ_PERIODIC,  /* equipment type */
    0,            /* interrupt source */
    "FIXED",      /* format */
    TRUE,          /* enabled */
    RO_RUNNING | RO_ODB |
          RO_EOR,  /* read when running;
                      send to odb */
    500,          /* polling period */
    0,            /* event limit */
    0,            /* number of sub-events */
    0,            /* log history */
    "", "", "",
    info_odb,      /* readout routine */
    NULL,NULL,NULL,
  },
 
It is a good idea to check the record size and/or create the record in the ODB when using C structures from experim.h, to make sure that the structures in the ODB and in experim.h
are identical.  The frontend code might look like this:
 
/* frontend.c */
....
#include <span style="color:red">experim.h</span>
//
// <span style="color:red">INFO_ODB_EVENT</span> and  <span style="color:red">INFO_ODB_EVENT_STR</span> are defined in experim.h
INFO_ODB_EVENT cyinfo;
INFO_ODB_EVENT_STR(info_odb_event_str);
HNDLE hInfo;
INT status, size;
char  str_set[256];
....
sprintf(str_set,"/Equipment/INFO ODB/Variables");
//
/* create record /Equipment/INFO ODB/Variables to make sure it exists  */
/* find the key for info odb */
  status = db_find_key(hDB, 0, str_set, &hInfo);
  if (status != DB_SUCCESS)
    {
      printf( "Key %s not found; creating record for info odb\n",str_set);
      status = db_create_record(hDB, 0, str_set, strcomb(info_odb_event_str));
    }
  /* check the record size */
  status = db_get_record_size(hDB, hInfo, 0, &size);
  if (sizeof(INFO_ODB_EVENT) != size)
    {
        cm_msg(MERROR, "bnmr_init", "error; record sizes do not match");
        return DB_TYPE_MISMATCH;
    }
  .......
 
A '''readout routine''' for this fixed event is as follows:
 
INT info_odb(char * pevent, INT off)
/* - periodic equipment updating the ODB ONLY
  - no event generation for the data stream.
*/
{
  //
  /* fill various values */
  cyinfo.helicity = gbl_ppg_hel;
  cyinfo.current_cycle = gbl_CYCLE_N;
  cyinfo.current_scan = gbl_SCAN_N;
  cyinfo.epicsdev_set_v_ = epics_params.Epics_val;
  cyinfo.epicsdev_read_v_ = epics_params.Epics_read;
  cyinfo.campdev_set = 0; 
  cyinfo.campdev_read = 0;
  // 
  memcpy(pevent, (char *)&(cyinfo.helicity), sizeof(cyinfo));
  pevent += sizeof(cyinfo);
  logMsg ("info_odb %d size:%d\n",gbl_CYCLE_N,sizeof(cyinfo),0,0,0,0);
  return sizeof(cyinfo);
}
 
More examples of FIXED events can be found in the slow controls device drivers, for example
'''../examples/slowcont/frontend.c''' and '''../drivers/class/hv.c'''
 
 
== Special Events ==
 
The three types of special events are begin-of-run, end-of-run and message events.
 
In all cases, the events contain an EVENT_HEADER, and then proceed directly to the data payload. There are no banks or BANK_HEADER.
 
=== Begin-of-run and End-of-run Events ===
 
If [[Keys_in_the_ODB_/Logger/Channels_subtree#ODB_Dump|file-level ODB dumps]] are enabled, two extra events will be added to each midas file (as the first and last events, respectively):
 
* The begin-of-run (BOR) event has an EventID of 0x8000.
* The end-of-run (EOR) event has an EventID of 0x8001.
 
In both cases, the data area of the event contains a JSON or XML representation of the ODB at the time the event was written. The format (json or xml) is specified using the [[Keys_in_the_ODB_/Logger/Channels_subtree#ODB_dump_format|ODB dump format]] setting.
 
Note that although the events are called "begin-of-run" and "end-of-run", they are actually written for every [[Subruns|subrun]]! (Subruns let you split the data for a run across multiple files, e.g. each of 1GB in size). This is so that analysis code can find any relevant ODB settings in the same midas file as the data it is analysing.
 
The BOR event has a trigger mask equal to 0x494d (aka 'MI'), and a serial number equal to the current run number.
 
=== Message Events ===
 
If [[Keys_in_the_ODB_/Logger/Channels_subtree#Log_messages|log messages]] is enabled, any messages written using cm_msg() will be stored in the file.
 
A message event has an EventID of 0x8002, and the message is contained in the data area as an ASCII string.
 
== Tape Format ==
<div style="background-color:mistyrose;">
;Warning : Writing to tape as described below is not maintained. Files are usually written to disk, and may later be archived onto tape or other storage devices using normal file transfer methods.
</div>
 
Events are written to disk files without any reformatting.
For tapes, however, a fixed block size is used. The block size TAPE_BUFFER_SIZE is defined in ''midas.h'' and usually 32kB.
 
A tape can therefore be identified as a MIDAS formatted tape.
 
The routine ''tape_copy()'' in the utility mtape.c is an example of how to read a tape in MIDAS format.
 
 
[[Category:Event]] [[Category:Appendices]]

Latest revision as of 22:00, 30 October 2026

The midas documentation has moved. See Event data format for the new documentation about the structure of midas events.

Retrieved from "https://daq00.triumf.ca/MidasWiki/index.php?title=Event_Structure&oldid=3706"