BNMR: Data Logging

• BNMR overview
• Getting started
• Troubleshooting
• All BNMR pages
• Midas

MUD files

bnmr and bnqr do not save their data as Midas files, but use MUD files instead (.msr extension).

Therefore we only use the Midas mlogger for recording history data, not the regular event data.

The key feature to note is that MUD files are not event-based. One can save statistics and histograms, but not complex per-event data. Histograms may be used to a single number per event (the X-axis is event number) or the total counts over an entire run (the X-axis is time-of-flight in ns, for example). Data saved as statistics can include the mean/min/max/std.dev. and more information.

Users analyze MUD files using:

• the mudpy and bdata python packages (with some GUIs built on top of them)
• the MUSR/BNMR analyzer program physica
• the MUD utility program mud_util

A couple of python utilities based on bdata are provided in the bnmr repository, which may be useful when adding new features to the data logger:

• bnxr_common/tests/dump_mud_file.py dumps the content of a MUD file to screen
• bnxr_common/tests/diff_mud_files.py compares the content of 2 MUD files

DAQ experts may enable dumping of Midas banks to a Midas file for debugging purposes. The banks written will come from the BNMR:_frontend frontend program only, and will not contain any EPICS variables, CAMP variables, or any manipulation of histograms performed in the bnxr_logger.exe program.

bnxr_logger.exe introduction

The bnxr_logger.exe program is responsible for collecting data and writing to MUD files.

Most of the parameters controlling the data logging are found in the ODB at /MUD Logging. Most settings are configured manually, though some are set/computed by the RF calculator program and the beginning of each run.

The program gets data from the following sources:

• Histograms - read from midas banks sent by the frontend
• Camp - read directly from CAMP
• EPICS - read directly from EPICS
• ODB variables - read directly from the ODB
• Run comments - read from a text file that is edited via a web interface

More details on each of these is presented further down this page.

Saved data files

The bnmr and bnqr experiments use the MUSR-style Run Numbering scheme. The data files are named according to the run number. Real data files are archived in the MUSR Data Archive at the end of the run.

During a run, several versions of the .msr (or MUD) data file are retained with a version number appended (in case of any problems with the most recent saved data). A symlink to the most recent version is updated during the run, so analysis programs do not have to concern themselves with the versioning information.

For example, for BNMR run 40362, while the run is in progress, the saved data directory would contain files such as

lrwxrwxrwx    1 bnmr     bnmr           48   Feb 26 12:09 040362.msr -> /isdaq/data1/bnmr/dlog/current/040362.msr_v27
-rw-r--r--    1 bnmr     bnmr         4403   Feb 26 12:09 040362.msr_v27
-rw-r--r--    1 bnmr     bnmr         4403   Feb 26 12:09 040362.msr_v26
-rw-r--r--    1 bnmr     bnmr         4403   Feb 26 12:09 040362.msr_v25
-rw-r--r--    1 bnmr     bnmr         4403   Feb 26 12:08 040362.msr_v24

At end-of-run, normally all earlier versions are purged and the final data file is renamed without the appended version number, and the file archived to the MUSR Data Archive. These steps can be suppressed by setting /MUD Logging/Global/Settings/Purge and archive at end of run to false, for example if the user decides it would be better to keep the version 26 of a file instead of the most recent version 27.

Using the same example as before, after the run the saved data directory would contain:

-rw-r--r--    1 bnmr     bnmr         4399 Feb 26 12:10 040362.msr

Archiving

The ODB keys Archiver task and Archiver dir in /MUD Logging/Global/Settings determine where files for "real" runs get copied to after a run is complete. This is normally just an scp to a directory on cmms.triumf.ca.

Nuking a run

If the user decides that a run is irredeemable, they may choose to "nuke" the run and start again. This involves deleting ALL the files that have been written, pretending that the run never happened. The same run number will then be used for the next run. Users appreciate this feature as it allows analysis to occur on contiguous ranges of run numbers.

Indicating that a run should be nuked is done with the /MUD Logging/Global/Settings/Nuke this run flag in the ODB, which is generally achieved by the user clicking the "Nuke ⚠️" button on the Status page

Cleaning up if end-of-run fails

The logger_cleanup.exe program can be used if bnxr_logger.exe crashes during a run and fails to perform the standard cleanup/archive duties at the end of a run.

It effectively just runs the same code as the main MUD logger runs at the end of the run, but allows the user to specify any run number in the past if they wish.

It will:

• Delete any temporary files that are no longer needed
• Archive the final file if this was a "real" run (not a "test" run)

Usage is:

      logger_cleanup.exe
           [-r Run number]
                        Run number of run to be cleaned up and/or archived. Defaults to the most recent run.

           [-h Hostname  ]
                      hostname is supplied only for a remote host

           [-e Experiment]
                       experiment name (default is given by environment variable MIDAS_EXPT_NAME)

Resetting the run numbers

BNMR and BNQR reuse run numbers each year (i.e. you need to specify both the year and run number to identify a dataset).

At the start of each year, you need to change where the data is being written to. This will automatically reset the run numbers to the new values for the year as well

cd /isdaq/data1/<expt>/dlog
mkdir <year>
rm current
ln -s <year> current
# Use the web interface to toggle between real and test modes (without starting a run), and verify that the run numbers are 30000/40000 (for BNMR) and 30000/45000 (for BNQR).

bnxr_logger.exe details

The code for the MUD logger is found in the bnmr repository at bnxr_common/logging. bnmr and bnqr are very similar, so the executable is almost identical for both (just an #ifdef for the default ODB settings to use). The main configuration is all done through ODB keys.

The program is built as part of the main build (cd build; make install from the bnmr repository).

The general functions performed by the program are:

• Read data sent by the frontend and process it. This includes both the data from the scalers as well as metadata like the current cycle number.
• Read data from EPICS and CAMP, either periodically or each time we get data from the frontend.
• Write data to a MUD file periodically during the run.

Histograms

See the histograms and scalers page for more details.

Settings are stored in the ODB at /MUD Logging/Histos/Settings.

CAMP

CAMP data is logged in two separate sections of the MUD file - in the run description and as independent variables. The bnxr_logger.exe program connects directly to the CAMP server and reads the values from there.

Run description

The MUD run description section has entries for the magnetic field strength and temperature of the sample. This data can be exposed to users when searching for data files taken previously.

ODB settings in /MUD Logging/Run description/Settings specify whether to read the magnetic field and/or temperature from CAMP, and if so, which instrument to read from. CAMP can automatically compute the standard deviation of variables throughout a run, and the user can choose to record this value in the file too.

Only a single number gets written to the run description for each variable - the average value throughout the run.

NOTE: Automatic CAMP variables to be logged must be set loggable in the CAMP CUI - see #Set CAMP variable loggable.

Independent variables

Other CAMP variables can be logged during a run too.

ODB settings in /MUD Logging/Camp/Settings define whether CAMP variables should be written as histograms of readings or just average values. The general configuration is to log a per-cycle histograms for T1 (TI) modes and just average values for T2 (TD) modes.

The user does not list each individual CAMP variable to be logged in the ODB, but instead lists which "categories" of variables should be included in the MUD file (e.g. log_mdarc). The mapping between CAMP variable and "category" is done via CAMP itself (see next section).

This CAMP data is stored as "independent variables" in the MUD file, keyed by the CAMP path (of format like /Magnet/mag_field).

Set CAMP variable loggable

All CAMP variables to be logged must be set loggable in the CAMP user interface. This includes the parameters to be used in the run description.

The loggable categories are shown in Table 1.

Table 1 : CAMP logging for Type 1 and Type 2 runs

Set logging	Variable Logged for
	Type1 (I-MUSR)	Type2 (TD-MUSR)
log_mdarc	Y	Y
log_mdarc_td	N	Y
log_td_musr
log_mdarc_ti	Y	N
log_i_musr

Checking which variables will be logged

The "Logging" custom webpage includes a feature that allows the user to see which CAMP variables will be logged given the current ODB and CAMP settings. This is implemented using the midas JRPC system to talk to the bnxr_logger.exe program, which returns the list of variables.

CAMP error file

CAMP keeps a log file for errors. This can be found in camp_srv.log at path /home/musrdaq/musr/camp/[camphost]/log/ where camphost is bnmrvw for bnmr or polvw for bnqr. Error counts can also be found on the CAMP CUI under R.Errors and W.Errors in Figure 1 above.

e.g.

 tail -f ~musrdaq/musr/camp/bnmrvw/log/camp_srv.log
[JUN 26 17:46:41:2018] camp_srv: instrument 'biasV' not online in insIfWriteVerify /biasV | failed 2 attempts
[JUN 26 17:46:43:2018] camp_srv: instrument 'biasV' not online in insIfWriteVerify /biasV | failed 2 attempts
[JUN 26 17:46:45:2018] camp_srv: instrument 'biasV' not online in insIfWriteVerify /biasV | failed 2 attempts

EPICS

See the EPICS page for an overview of EPICS in bnmr and bnqr.

Users directly specify which EPICS variables should be logged in /MUD Logging/Epics/Settings (or by using the relevant bit of the Custom Logging page). The bnxr_logger.exe program connects directly to EPICS and reads the values itself.

The code allows for deciding whether to store a histogram of readings throughout the run or just summary statistics. In general, a histogram is written for TI runs, where the scaler data is being store in a per-cycle style; the histograms for the EPICS data and scaler data will be the same length, with one entry per PPG cycle. To avoid overloading the EPICS server, we can read from it periodically rather than every time a new PPG cycle starts. Each time we read a value, we will apply it to all the cycles that happened since we last read.

EPICS data is stored as "independent variables" in the MUD file, keyed by the PV name (of format like ILE2:BIAS15:RDVOL).

Run comments

When a run is in progress, the user is presented with a textarea element on the status page. This allows the user to note any critical information about the run that should be stored in the file, not just in a separate logbook. The textarea is hooked up to the run_comment_editor.py program using Midas JRPC. The python program just writes a text file on disk in the directory specified by /Logger/Data dir. Other settings for the python program are in /Equipment/RunComments/Settings.

When it is time for the bnxr_logger.exe program to write a new MUD file, it just reads the file that was written by the python program. There are a couple of extra settings in /MUD Logging/TextFileComments/Settings.

MUD files have a section dedicated to storing run comments.

ODB variables

Independent variables

ODB variables specifying the configuration of the run are stored as independent variables in the MUD file. The user does not specify directly which variables log for each mode; the decision is made by the RF calculator program.

In particular, the variables to be saved are stored as ODB links in the /Equipment/RFCalculator/Computed/Mud links/PPG/PPG20 directory (for mode 20; similar names for other modes). The definition of the links is in mud_ind_var_links.py, which is included by rf_calculator_fe.py. This keeps all the mode-specific complexity in one place, and in a language that is more expressive. The C++ code in bnxr_logger.exe simply has to iterate over all the links in the relevant directory.

Note the following conversions between C++/ODB data types and MUD independent variables:

• Arrays/vectors are not supported (in principle we could store a histogram, but users have not requested that yet)
• Numbers are stored in the mean member of the MUD variable.
• Strings are stored in the units member of the MUD variable.
• Booleans are stored with 1/0 in the mean member and yes/no in the units member of the MUD variable.
• If the ODB link name contains parentheses, the string between the parenthesis is stored in the units member of the MUD variable (e.g. a link called Frequency (Hz) will end up with Hz as the units).
• If the ODB link name does not contain parentheses (e.g. Frequency), but there is another ODB key called Frequency ((UNITS)), the string in the latter key is stored as the units for the former. This is for backwards-compatibility with older files that were not consistent about whether MUD variables contain units in the name, or only in the units.

ODB settings are stored as "independent variables" in the MUD file, keyed by the ODB link name (of format like PPG/PPG20/Enable RF to be backwards-compatible with older files). To be clear, the name of the link itself is used, not the name of the target of the link.

See the mode changer page for details about how the user may load settings from a MUD file to replicate a previous experimental configuration.

Run description

Some ODB variables are also used to build the run description in the MUD file (along with any CAMP variables being used for the run description).

The variables used are:

• /Experiment/Edit on start/experiment number for the exptNumber
• /Experiment/Edit on start/experimenter for the experimenter
• /Experiment/Edit on start/run_title for the run_title
• /Experiment/Edit on start/sample for the sample
• /Experiment/Edit on start/orientation for the orient
• /Experiment/Name for the area and apparatus,
• /CurrentMode/Name for the insert (hijacking a field used by MUSR but not bnmr or bnqr)
• /Runinfo/Start time binary for the timeBegin
• /Runinfo/Stop time binary for the timeEnd
• /MUD Logging/Run description/Settings/Manual field value if not reading from CAMP
• /MUD Logging/Run description/Settings/Manual temperature value if not reading from CAMP

See edit-on-start parameters for more details on those variables.