Data format

previous.gif Utilities - Top - Supported hardware next.gif

Midas supports two differents data format so far. A possible new candidate would be the NeXus format, but presently no implementation has been developed.


Midas format

Special formats are used in MIDAS for the event header, banks and when writing to disk or tape. This appendix explains these formats in detail. Each event carries a 16-byte header. The header is generated by the front-end with the bm_compose_event() routine and is used by consumers to distinguish between different events. The header is defined in the EVENT_HEADER structure in midas.h. It has following structure:

Event and bank headers with data block.
mEHBH.jpg

The event ID describes the type of event. Usually 1 is used for triggered events, 2 for scaler events, 3 for HV events etc. The trigger mask can be used to describe the sub-type of an event. A trigger event can have different trigger sources like "physics event", "calibration event", "clock event". These trigger sources are usually read in by the front-end in a pattern unit. Consumers can request events with a specific triggering mask. The serial number starts at one and is incremented by the front-end for each event. The time stamp is written by the front-end before an event is read out. It uses the time() function which returns the time in seconds since 1.1.1970 00:00:00 UTC. The data size contains the number of bytes that follows the event header. The data area of the event can contain information in any user format, although only certain formats are supported when events are copied to the ODB or written by the logger in ASCII format. Event headers are always kept in the byte ordering of the local machine. If events are sent over the network between computers with different byte ordering, the event header is swapped automatically, but not the event contents.


YBOS format

As mentioned earlier the YBOS documentation is available at the following URL address: Ybos site Originally YBOS is a collection of FORTRAN functions which facilitate the manipulation of group of data. It also describes a mode of encoding/storing data in an organized way. YBOS defines specific ways for:

YBOS is organized on a 4-byte alignment structure.

The YBOS library function provides all the tools for manipulation of the above mentioned elements in a independent Operating System like. But the implementation of the YBOS part in Midas does not use any reference to the YBOS library code. Instead only the strict necessary functions have to be re-written in C and incorporated into the Midas package. This has been motivated by the fact that only a sub-set of function is essential to the operation of:

This Midas/YBOS implementation restricts the user to a subset of the YBOS package only for the front-end part. It doesn't prevent him/her to use the full YBOS library for stand alone program accessing data file written by Midas.

The YBOS implementation under Midas has the following restrictions:

YBOS uses "double word" unit for all length references.

Ybos Event and bank headers with data block.
yEHBH.jpg

The bank length parameter corresponds to the size of the data section in double word count + 1. The supported bank type are defined in the ybos.h file see YBOS Bank Types.

Ybos Physical record structure with data block.
yPH.jpg

The Offset is computed with the following rules:


History format

MIDAS history data are written to MIDAS .hst files. For a complete working example of reading these files, look at mhdump.cxx.

This is the internal format of MIDAS .hst files (reverse engineered by K.Olchanski).

History file consists of definition records and data records. Definition records for each history event always preceed data records for this event. Other than that, definition and data records are present in the file in no particular order. If data definition changes at some point, the new definition record is written to the file and new data records follow using the new format.

The data is written in native-endian format and big-endian history files (i.e. written on PowerPC-based computer) would not read on little-endian computers (i.e. Intel/AMD CPUs).

When decoding history data records, be aware that the data is written on the format returned by db_get_data(), and follow certain alignement rules. Misaligned data may contain empty padding bytes.

definition record:
   struct HIST_RECORD:
      uint32_t record_type = 0x46445348 (RT_DEF)
      uint32_t event_id    = history event id
      uint32_t time        = unix time
      uint32_t def_offset  = 0
      uint32_t data_size   = size in bytes of following data = ntags*sizeof(TAG)
   event name:
      char name[NAME_LENGTH] = name of this event (fixed size, 32 bytes)
   struct TAG [0]
      char name[NAME_LENGTH] = tag name (fixed size, 32 bytes)
      uint32_t type          = type of the data (TID_DOUBLE, etc)
      uint32_t n_data        = 1 for single variables, number of array elements for arrays
   struct TAG [1]
      ...
   struct TAG [ntags-1]
      ...

data record:
   struct HIST_RECORD: (same as above)
      uint32_t record_type = 0x41445348 (RT_DATA)
      uint32_t event_id    = history event id
      uint32_t time        = unix time
      uint32_t def_offset  = ???
      uint32_t data_size   = size in bytes of following data
   event data
      char[data_size]      = history data for this event (see mhdump.cxx and examples below on decoding this data

To decode the "event data" blob in a data record, one should parse the data definition for this event and remember the contents of all tag definitions. One way to parse the data is by computing the "offset" for each tag by adding up the tag size for each tag. For each tag, the corresponding data size is the product of n_data and the size of the MIDAS type, given in this table (extracted from midas.c):

/* data type sizes */
int tid_size[] = {
   0,                           /* tid == 0 not defined                               */
   1,                           /* TID_BYTE      unsigned byte         0       255    */
   1,                           /* TID_SBYTE     signed byte         -128      127    */
   1,                           /* TID_CHAR      single character      0       255    */
   2,                           /* TID_WORD      two bytes             0      65535   */
   2,                           /* TID_SHORT     signed word        -32768    32767   */
   4,                           /* TID_DWORD     four bytes            0      2^32-1  */
   4,                           /* TID_INT       signed dword        -2^31    2^31-1  */
   4,                           /* TID_BOOL      four bytes bool       0        1     */
   4,                           /* TID_FLOAT     4 Byte float format                  */
   8,                           /* TID_DOUBLE    8 Byte float format                  */
   1,                           /* TID_BITFIELD  8 Bits Bitfield    00000000 11111111 */
   0,                           /* TID_STRING    zero terminated string               */
   0,                           /* TID_ARRAY     variable length array of unkown type */
   0,                           /* TID_STRUCT    C structure                          */
   0,                           /* TID_KEY       key in online database               */
   0                            /* TID_LINK      link in online database              */
};

Example code for computing tag offsets:

int offset = 0;
for (int itag=0; itag<ntags; itag++) {
   int size = tags[itag].n_data * tid_size[tags[itag].type];
   type_for_tag[itag] = tags[itag].type;
   offset_for_tag[itag] = offset;
   offset += size;
}

Example code for decoding the event data blob:

char *buf = new char[data_size];
fread(buf, 1, data_size, f); // read data blob from file

int itag = ...; // the tag we want to look at
int j    = ...; // array index inside the tag

int offset = offset_for_tag[itag];
void* ptr = (void*)(buf+offset);

switch (type_for_tag[itag]) {
   default:
      printf("unknownType%d ",type_for_tag[itag]);
      break;
   case 6: /* DWORD */
      printf("%u ",((uint32_t*)ptr)[j]);
      break;
}

previous.gif Utilities - Top - Supported hardware next.gif


Midas DOC Version 3.0.0 ---- PSI Stefan Ritt ----
Contributions: Pierre-Andre Amaudruz - Sergio Ballestrero - Suzannah Daviel - Doxygen - Peter Green - Qing Gu - Greg Hackman - Gertjan Hofman - Paul Knowles - Exaos Lee - Rudi Meier - Glenn Moloney - Dave Morris - John M O'Donnell - Konstantin Olchanski - Renee Poutissou - Tamsen Schurman - Andreas Suter - Jan M.Wouters - Piotr Adam Zolnierczuk