BNMR: Data Logging

• BNMR overview
• Getting started
• Troubleshooting
• All BNMR pages
• Midas

MUD files

bnmr and bnqr do not save their data as Midas files, but use MUD files instead (.msr extension).

Therefore we only use the Midas mlogger for recording history data, not the regular event data.

The key feature to note is that MUD files are not event-based. One can save statistics and histograms, but not complex per-event data. Histograms may be used to a single number per event (the X-axis is event number) or the total counts over an entire run (the X-axis is time-of-flight in ns, for example). Data saved as statistics can include the mean/min/max/std.dev. and more information.

Users analyze MUD files using:

• the mudpy and bdata python packages (with some GUIs built on top of them)
• the MUSR/BNMR analyzer program physica
• the MUD utility program mud_util

A couple of python utilities based on bdata are provided in the bnmr repository, which may be useful when adding new features to the data logger:

• bnxr_common/tests/dump_mud_file.py dumps the content of a MUD file to screen
• bnxr_common/tests/diff_mud_files.py compares the content of 2 MUD files

DAQ experts may enable dumping of Midas banks to a Midas file for debugging purposes. The banks written will come from the BNMR:_frontend frontend program only, and will not contain any EPICS variables, CAMP variables, or any manipulation of histograms performed in the bnxr_logger.exe program.

bnxr_logger.exe introduction

The bnxr_logger.exe program is responsible for collecting data and writing to MUD files.

Most of the parameters controlling the data logging are found in the ODB at /MUD Logging. Most settings are configured manually, though some are set/computed by the RF calculator program and the beginning of each run.

The program gets data from the following sources:

• Histograms - read from midas banks sent by the frontend
• Camp - read directly from CAMP
• EPICS - read directly from EPICS
• ODB variables - read directly from the ODB
• Run comments - read from a text file that is edited via a web interface

More details on each of these is presented further down this page.

Saved data files

The bnmr and bnqr experiments use the MUSR-style Run Numbering scheme. The data files are named according to the run number. Real data files are archived in the MUSR Data Archive at the end of the run.

During a run, several versions of the .msr (or MUD) data file are retained with a version number appended (in case of any problems with the most recent saved data). A symlink to the most recent version is updated during the run, so analysis programs do not have to concern themselves with the versioning information.

For example, for BNMR run 40362, while the run is in progress, the saved data directory would contain files such as

lrwxrwxrwx    1 bnmr     bnmr           48   Feb 26 12:09 040362.msr -> /isdaq/data1/bnmr/dlog/current/040362.msr_v27
-rw-r--r--    1 bnmr     bnmr         4403   Feb 26 12:09 040362.msr_v27
-rw-r--r--    1 bnmr     bnmr         4403   Feb 26 12:09 040362.msr_v26
-rw-r--r--    1 bnmr     bnmr         4403   Feb 26 12:09 040362.msr_v25
-rw-r--r--    1 bnmr     bnmr         4403   Feb 26 12:08 040362.msr_v24

At end-of-run, normally all earlier versions are purged and the final data file is renamed without the appended version number, and the file archived to the MUSR Data Archive. These steps can be suppressed by setting /MUD Logging/Global/Settings/Purge and archive at end of run to false, for example if the user decides it would be better to keep the version 26 of a file instead of the most recent version 27.

Using the same example as before, after the run the saved data directory would contain:

-rw-r--r--    1 bnmr     bnmr         4399 Feb 26 12:10 040362.msr

Archiving

The ODB keys Archiver task and Archiver dir in /MUD Logging/Global/Settings determine where files for "real" runs get copied to after a run is complete. This is normally just an scp to a directory on cmms.triumf.ca.

Nuking a run

If the user decides that a run is irredeemable, they may choose to "nuke" the run and start again. This involves deleting ALL the files that have been written, pretending that the run never happened. The same run number will then be used for the next run. Users appreciate this feature as it allows analysis to occur on contiguous ranges of run numbers.

Indicating that a run should be nuked is done with the /MUD Logging/Global/Settings/Nuke this run flag in the ODB, which is generally achieved by the user clicking the "Nuke ⚠️" button on the Status page

Cleaning up if end-of-run fails

The logger_cleanup.exe program can be used if bnxr_logger.exe crashes during a run and fails to perform the standard cleanup/archive duties at the end of a run.

It effectively just runs the same code as the main MUD logger runs at the end of the run, but allows the user to specify any run number in the past if they wish.

It will:

• Delete any temporary files that are no longer needed
• Archive the final file if this was a "real" run (not a "test" run)

Usage is:

      logger_cleanup.exe
           [-r Run number]
                        Run number of run to be cleaned up and/or archived. Defaults to the most recent run.

           [-h Hostname  ]
                      hostname is supplied only for a remote host

           [-e Experiment]
                       experiment name (default is given by environment variable MIDAS_EXPT_NAME)

bnxr_logger.exe details

The code for the MUD logger is found in the bnmr repository at bnxr_common/logging. bnmr and bnqr are very similar, so the executable is almost identical for both (just an #ifdef for the default ODB settings to use). The main configuration is all done through ODB keys.

The program is built as part of the main build (cd build; make install from the bnmr repository).

The general functions performed by the program are:

• Read data sent by the frontend and process it. This includes both the data from the scalers as well as metadata like the current cycle number.
• Read data from EPICS and CAMP, either periodically or each time we get data from the frontend.
• Write data to a MUD file periodically during the run.

Histograms

TODO

CAMP

CAMP data is logged in two separate sections of the MUD file - in the run description and as independent variables. The bnxr_logger.exe program connects directly to the CAMP server and reads the values from there.

Run description

The MUD run description section has entries for the magnetic field strength and temperature of the sample. This data can be exposed to users when searching for data files taken previously.

ODB settings in /MUD Logging/Run description/Settings specify whether to read the magnetic field and/or temperature from CAMP, and if so, which instrument to read from. CAMP can automatically compute the standard deviation of variables throughout a run, and the user can choose to record this value in the file too.

Only a single number gets written to the run description for each variable - the average value throughout the run.

NOTE

Automatic CAMP variables to be logged must be set loggable in the CAMP CUI - see #Set CAMP variable loggable.

Independent variables

Other CAMP variables can be logged during a run too.

ODB settings in /MUD Logging/Camp/Settings define whether CAMP variables should be written as histograms of readings or just average values. The general configuration is to log a per-cycle histograms for T1 (TI) modes and just average values for T2 (TD) modes.

The user does not list each individual CAMP variable to be logged in the ODB, but instead lists which "categories" of variables should be included in the MUD file (e.g. log_mdarc). The mapping between CAMP variable and "category" is done via CAMP itself (see next section).

This CAMP data is stored as "independent variables" in the MUD file, keyed by the CAMP path (of format like /Magnet/mag_field).

Set CAMP variable loggable

All CAMP variables to be logged must be set loggable in the CAMP user interface. This includes the parameters to be used in the run description.

The loggable categories are shown in Table 1.

Table 1 : CAMP logging for Type 1 and Type 2 runs

Set logging	Variable Logged for
	Type1 (I-MUSR)	Type2 (TD-MUSR)
log_mdarc	Y	Y
log_mdarc_td	N	Y
log_td_musr
log_mdarc_ti	Y	N
log_i_musr

Checking which variables will be logged

The "Logging" custom webpage includes a feature that allows the user to see which CAMP variables will be logged given the current ODB and CAMP settings. This is implemented using the midas JRPC system to talk to the bnxr_logger.exe program, which returns the list of variables.

CAMP error file

CAMP keeps a log file for errors. This can be found in camp_srv.log at path /home/musrdaq/musr/camp/[camphost]/log/ where camphost is bnmrvw for bnmr or polvw for bnqr. Error counts can also be found on the CAMP CUI under R.Errors and W.Errors in Figure 1 above.

e.g.

 tail -f ~musrdaq/musr/camp/bnmrvw/log/camp_srv.log
[JUN 26 17:46:41:2018] camp_srv: instrument 'biasV' not online in insIfWriteVerify /biasV | failed 2 attempts
[JUN 26 17:46:43:2018] camp_srv: instrument 'biasV' not online in insIfWriteVerify /biasV | failed 2 attempts
[JUN 26 17:46:45:2018] camp_srv: instrument 'biasV' not online in insIfWriteVerify /biasV | failed 2 attempts

EPICS

See the EPICS page for an overview of EPICS in bnmr and bnqr.

Users directly specify which EPICS variables should be logged in /MUD Logging/Epics/Settings (or by using the relevant bit of the Custom Logging page). The bnxr_logger.exe program connects directly to EPICS and reads the values itself.

The code allows for deciding whether to store a histogram of readings throughout the run or just summary statistics. In general, a histogram is written for TI runs, where the scaler data is being store in a per-cycle style; the histograms for the EPICS data and scaler data will be the same length, with one entry per PPG cycle. To avoid overloading the EPICS server, we can read from it periodically rather than every time a new PPG cycle starts. Each time we read a value, we will apply it to all the cycles that happened since we last read.

EPICS data is stored as "independent variables" in the MUD file, keyed by the PV name (of format like ILE2:BIAS15:RDVOL).

Run comments

When a run is in progress, the user is presented with a textarea element on the status page. This is hooked up to the run_comment_editor.py program using Midas JRPC. The python program just writes a text file on disk in the directory specified by /Logger/Data dir. Other settings for the python program are in /Equipment/RunComments/Settings.

When it is time for the bnxr_logger.exe program to write a new MUD file, it just reads the file that was written by the python program. There are a couple of extra settings in /MUD Logging/TextFileComments/Settings.

MUD files have a section dedicated to storing run comments.

ODB variables

TODO