BNMR: EPICS

• BNMR overview
• Getting started
• Troubleshooting
• All BNMR pages
• Midas

Purpose

This page gives bnmr or bnqr DAQ users information about the ISAC EPICS Control Pages and the Access required for running bnmr or bnqr experiments.

Introduction

The EPICS system is used at TRIUMF for the control system for beamlines etc. Experimenters will need to display the Epics Control Pages in order to run the bnmr or bnqr experiments. They must also ensure that BNMR has been granted write access to the required Epics parameters as described below, or they will not be able to change the relevant beamline controls.

Display ISAC EPICS control pages

Switch onto a different desktop on linux box isdaq01 (or isdaq06) so the Epics Control page does not clutter the desktop used for running the experiment. Access the epics control pages on isacepics1 with ssh from an xterm window, as follows (if you don't know the password, ask a group member).

ssh bnmr@isacepics1
( Type in the password )

You should see the main ISAC EPICS control page come up after a few seconds (Figure 1).

Figure 1: ISAC EPICS control page

Note

There is no bnqr user account on isacepics1. The bnmr account is used for both experiments and the same EPICS access permission (i.e. BNMR) applies to both.

Write access to EPICS polarizer parameters

Most Epics parameters can be read by anyone, but write access has to be granted by the Controls Group. Several experiments have been granted write access to the same polarizer IOCs, and to avoid conflict, only one of these experiments is granted overall Access. During the beam period, and for testing the helicity switching etc, access must be granted to expbnmr.

To check whether expbnmr has access, display the main EPICS control pages.

On the main EPICS Control Page (cf Figure 1) click the button labelled "Access". A page similar to Figure 2 will be displayed.

Figure 2: ISAC Access Control

Clicking on the View access button (Figure 2) will show the current access. This should be set to expbnmr as in Figure 3.

Figure 3: Polarizer Access set to expbnmr

If the Access is not set to expbnmr, ask the operators to switch the access by pressing the button Polarizer Access for bNMR/bNQR (Figure 2).

Even when "Access" is granted to BNMR, only those EPICS IOCs to which BNMR has been granted write access can be changed by the experimenters from the EPICS Control pages, e.g. Helicity, NaCell Voltage, and all BNMR/BNQR parameters for single/dual channel mode control.

Some of these parameters are also writable by the bnmr and bnqr BNMR and BNQR VMICs (lxbnmr,lxbnqr). They have been granted write access by the Controls Group to allow the frontend programs to flip helicity and turn the beam on/off via PPG signals. The frontends also use EPICS to scan the Rb cell voltage, for example.

WARNING

There is NO protection against bnmr and bnqr experimenters changing Epics parameters via the Epics Control pages or the frontend programs while an experimental run is in progress on either bnmr or bnqr. When running in single channel mode, there is also no protection against each experiment interfering with each other by both flipping the helicity for example. Users must ensure that they know what they are doing!

Usage in software (frontend and logger)

EPICS is used by:

• the frontend to scan the Rb cell voltage, laser intensity etc in certain modes
• the frontend to report some histogram totals to operators
• the frontend to sanity-check the dual channel mode and helicity settings
• the frontend to decide when to start the PPG in dual channel mode (optionally - see dual channel mode page for more details about when we use EPICS status vs a hardware signal to start the PPG)
• the logger to read any variables the user wants logging in the MUD file

UDP broadcast issue

The simplest way to use EPICS is to set the environment variable EPICS_CA_AUTO_ADDR_LIST=YES. This means that EPICS will do a UDP broadcast to find which IOC server is handling a given Process Variable, and you do not need to worry about the specifics of each PV. If you add new PVs you can just let EPICS find the correct IOC to talk to.

However, the TRIUMF firewall machines sometimes get into a state where lots of UDP broadcast packets get dropped (technical details according to Steve M: the data and control planes get split across different nodes). This prevents EPICS from reliably finding the correct IOC for a PV, and will report that many PVs cannot be found. For that reason, we can instead set EPICS_CA_AUTO_ADDR_LIST=NO and specify the IOCs of interest in the EPICS_CA_ADDR_LIST environment variable. This setup means that we do not need any UDP broadcast packets, and our EPICS communications work well even if the firewalls are behaving badly.

However, when adding a new EPICS PV to be scanned or sanity-checked, you must find out which IOC they are controlled by! This can be done either by emailing the controls group, or running the cainfo command - e.g. the following command shows that ile2ioc.triumf.ca (142.90.132.2) controls ILE2:BIAS15:VOL:

[bnmr@isdaq01 ~]$ cainfo ILE2:BIAS15:VOL
ILE2:BIAS15:VOL
    State:            connected
    Host:             ile2ioc.triumf.ca:9004
    Access:           read, write
    Native data type: DBF_DOUBLE
    Request type:     DBR_DOUBLE
    Element count:    1

Using EPICS_CA_AUTO_ADDR_LIST=NO is a much worse experience for shifters. If they want to log/control a PV that isn't controlled by an IOC that is already in EPICS_CA_ADDR_LIST, they must:

• Find out which IOC controls the PV using cainfo (or by emailing the Controls group if the network is really bad)
• Edit ~/.bashrc file to add the IOC to EPICS_CA_ADDR_LIST
• Source the updated ~/.bashrc file to pick up the change
• Restart all the DAQ programs with kill-all; start-all

Shifters often have many terminals open, and they must remember to source the new environment in each one before running start-all; otherwise it is very easy to restart programs with the wrong environment! Using EPICS_CA_AUTO_ADDR_LIST=YES is far superior unless the network is having issues.

EPICS environment variables

Currently (July 2022) bnmr and bnqr experiments are using epics-baseR3.14.12.

$ ls -lt /home/midas/packages/epics-base
 /home/midas/packages/epics-base -> epics-baseR3.14.12/

We have 3 common settings that may be used based on whether different parts of the network are having issues. You can change between them using the bash functions use_epics_gw etc.

Table 1: EPICS Environment Variables

	Normal usage - connect directly to IOCs of interest	Connect to gateway server (readonly)	Connect to broadcast address (vulnerable to UDP issues)
Bash function	use_epics_direct	use_epics_gw	use_epics_broadcast
Settings	EPICS_CA_ADDR_LIST=142.90.132.2 142.90.132.10 EPICS_CA_AUTO_ADDR_LIST=NO EPICS_CA_REPEATER_PORT=9003 EPICS_CA_SERVER_PORT=9004	EPICS_CA_ADDR_LIST=142.90.132.2 142.90.132.10 142.90.132.20 142.90.132.92 142.90.133.57 142.90.133.68 142.90.133.84 EPICS_CA_AUTO_ADDR_LIST=NO EPICS_CA_REPEATER_PORT=9093 EPICS_CA_SERVER_PORT=9094	EPICS_CA_ADDR_LIST=142.90.133.255 EPICS_CA_AUTO_ADDR_LIST=YES EPICS_CA_REPEATER_PORT=9003 EPICS_CA_SERVER_PORT=9004

NOTE: If you want to change which EPICS variables are used, you must change the ~/.bashrc file, source the updated version, then run kill-all; start-all.

EPICS Troubleshooting

All EPICS access depends on the EPICS Environment variables being set up correctly, and linking with the correct version of the EPICS software in epics-base. Check the #EPICS_environment_variables are set correctly.

Try reading using caget

When having trouble accessing EPICS, first try to read one of the IOCs, e.g. the Na/Rb cell voltage with caget, e.g.

 caget ILE2:BIAS15:RDVOL
 ILE2:BIAS15:RDVOL              262.074

If instead the command times out

 caget ILE2:BIAS15:RDVOL
 Channel connect timed out: 'ILE2:BIAS15:RDVOL' not found

this indicates a problem with the EPICS environment variable settings or a firewall issue. EPICS controls are firewalled. Try the caget command several times, as there may be a packet-loss problem where caget only works some of the time (though most likely your environment variables are incorrect, or you're connecting from the wrong machine or account).

From isdaq01, reading any EPICS variable should work.

Check access using cainfo

First see write access to EPICS Polarizer parameters. If access is not granted, writing to the Polarizer parameters will not be successful.

In order to write to an EPICS variable, permission must be granted by the EPICS group. This is done by user and cpu. For example, users bnmr and bnqr on isdaq01 are granted permission to write to the EPICS Experiment variables for bnmr and bnqr respectively. Writing to the Na/Rb cell voltage IOC 'ILE2:BIAS15:RDVOL' is granted to users bnmr and bnqr on lxbnmr and lxbnqr respectively.

Run cainfo to check for write permission, e.g.

[bnqr@isdaq01 bnqr]$ cainfo BNQR:VAR1
BNQR:VAR1
    State:            connected
    Host:             ile2ioc.triumf.ca:9004
    Access:           read, write
    Native data type: DBF_DOUBLE
    Request type:     DBR_DOUBLE
    Element count:    1

Try writing using caput

caput can be used to write to an IOC, e.g.

[bnmr@isdaq01 ~]$  caput BNMR:VAR1 5
Old : BNMR:VAR1                      1.9869e+06
New : BNMR:VAR1                      5

If write access is false, the value will not change, but no error message will be output.

Try reading/writing using epics_test.exe

The frontend and logger use a more user-friendly C++ wrapper around the native EPICS API. You can test this wrapper using the epics_test.exe and epics_test_32bit.exe programs that are built alongside the frontend/logger.

Usage is like:

epics_test.exe get BNQR:VAR1       # Connect to EPICS then read PV as a number
epics_test.exe getstr BNQR:VAR1    # Connect to EPICS then read PV as a string
epics_test.exe set BNQR:VAR1 2.5   # Connect to EPICS then set PV value

EPICS checking

• The frontend performs sanity-check of some EPICS variables at the start of the run - see the dual channel mode page for details.
• The frontend checks the state of the helicity waveplate during a run - see the helicity page for details.