BNMR: EPICS
Purpose
This page gives bnmr or bnqr DAQ users information about the ISAC EPICS Control Pages and the Access required for running bnmr or bnqr experiments.
Introduction
The EPICS system is used at TRIUMF for the control system for beamlines etc. Experimenters will need to display the Epics Control Pages in order to run the bnmr or bnqr experiments. They must also ensure that BNMR has been granted write access to the required Epics parameters as described below, or they will not be able to change the relevant beamline controls.
Display ISAC EPICS control pages
Switch onto a different desktop on linux box isdaq01 (or isdaq06) so the Epics Control page does not clutter the desktop used for running the experiment. Access the epics control pages on isacepics1 with ssh from an xterm window, as follows (if you don't know the password, ask a group member).
ssh bnmr@isacepics1 ( Type in the password )
You should see the main ISAC EPICS control page come up after a few seconds (Figure 1).
- Note
- There is no bnqr user account on isacepics1. The bnmr account is used for both experiments and the same EPICS access permission (i.e. BNMR) applies to both.
Write access to EPICS polarizer parameters
Most Epics parameters can be read by anyone, but write access has to be granted by the Controls Group. Several experiments have been granted write access to the same polarizer IOCs, and to avoid conflict, only one of these experiments is granted overall Access. During the beam period, and for testing the helicity switching etc, access must be granted to expbnmr.
To check whether expbnmr has access, display the main EPICS control pages.
On the main EPICS Control Page (cf Figure 1) click the button labelled "Access". A page similar to Figure 2 will be displayed.
Clicking on the View access button (Figure 2) will show the current access. This should be set to expbnmr as in Figure 3.
If the Access is not set to expbnmr, ask the operators to switch the access by pressing the button Polarizer Access for bNMR/bNQR (Figure 2).
Even when "Access" is granted to BNMR, only those EPICS IOCs to which BNMR has been granted write access can be changed by the experimenters from the EPICS Control pages, e.g. Helicity, NaCell Voltage, and all BNMR/BNQR parameters for single/dual channel mode control.
Some of these parameters are also writable by the bnmr and bnqr BNMR and BNQR VMICs (lxbnmr,lxbnqr). They have been granted write access by the Controls Group to allow the frontend programs to flip helicity and turn the beam on/off via PPG signals. The frontends also use EPICS to scan the Rb cell voltage, for example.
WARNING
There is NO protection against bnmr and bnqr experimenters changing Epics parameters via the Epics Control pages or the frontend programs while an experimental run is in progress on either bnmr or bnqr. When running in single channel mode, there is also no protection against each experiment interfering with each other by both flipping the helicity for example. Users must ensure that they know what they are doing!
Usage in software (frontend and logger)
EPICS is used by:
- the frontend to scan the Rb cell voltage, laser intensity etc in certain modes
- the frontend to report some histogram totals to operators
- the frontend to sanity-check the dual channel mode and helicity settings
- the frontend to decide when to start the PPG in dual channel mode (optionally - see dual channel mode page for more details about when we use EPICS status vs a hardware signal to start the PPG)
- the logger to read any variables the user wants logging in the MUD file
UDP broadcast issue
The simplest way to use EPICS is to set the environment variable EPICS_CA_AUTO_ADDR_LIST=YES. This means that EPICS will do a UDP broadcast to find which IOC server is handling a given Process Variable, and you do not need to worry about the specifics of each PV. If you add new PVs you can just let EPICS find the correct IOC to talk to.
However, the TRIUMF firewall machines sometimes get into a state where lots of UDP broadcast packets get dropped (technical details according to Steve M: the data and control planes get split across different nodes). This prevents EPICS from reliably finding the correct IOC for a PV, and will report that many PVs cannot be found. For that reason, we can instead set EPICS_CA_AUTO_ADDR_LIST=NO and specify the IOCs of interest in the EPICS_CA_ADDR_LIST environment variable. This setup means that we do not need any UDP broadcast packets, and our EPICS communications work well even if the firewalls are behaving badly.
However, when adding a new EPICS PV to be scanned or sanity-checked, you must find out which IOC they are controlled by! This can be done either by emailing the controls group, or running the cainfo command - e.g. the following command shows that ile2ioc.triumf.ca (142.90.132.2) controls ILE2:BIAS15:VOL:
[bnmr@isdaq01 ~]$ cainfo ILE2:BIAS15:VOL
ILE2:BIAS15:VOL
State: connected
Host: ile2ioc.triumf.ca:9004
Access: read, write
Native data type: DBF_DOUBLE
Request type: DBR_DOUBLE
Element count: 1
Using EPICS_CA_AUTO_ADDR_LIST=NO is a much worse experience for shifters. If they want to log/control a PV that isn't controlled by an IOC that is already in EPICS_CA_ADDR_LIST, they must:
- Find out which IOC controls the PV using cainfo (or by emailing the Controls group if the network is really bad)
- Edit ~/.bashrc file to add the IOC to
EPICS_CA_ADDR_LIST - Source the updated ~/.bashrc file to pick up the change
- Restart all the DAQ programs with
kill-all; start-all
Shifters often have many terminals open, and they must remember to source the new environment in each one before running start-all; otherwise it is very easy to restart programs with the wrong environment! Using EPICS_CA_AUTO_ADDR_LIST=YES is far superior unless the network is having issues.
EPICS environment variables
Currently (July 2022) bnmr and bnqr experiments are using epics-baseR3.14.12.
$ ls -lt /home/midas/packages/epics-base /home/midas/packages/epics-base -> epics-baseR3.14.12/
We have 3 common settings that may be used based on whether different parts of the network are having issues. You can change between them using the bash functions use_epics_gw etc.
| Normal usage - connect directly to IOCs of interest | Connect to gateway server (readonly) | Connect to broadcast address (vulnerable to UDP issues) | |
|---|---|---|---|
| Bash function | use_epics_direct | use_epics_gw | use_epics_broadcast |
| Settings |
EPICS_CA_ADDR_LIST=142.90.132.2 142.90.132.10 EPICS_CA_AUTO_ADDR_LIST=NO EPICS_CA_REPEATER_PORT=9003 EPICS_CA_SERVER_PORT=9004 |
EPICS_CA_ADDR_LIST=142.90.132.2 142.90.132.10 142.90.132.20 142.90.132.92 142.90.133.57 142.90.133.68 142.90.133.84 EPICS_CA_AUTO_ADDR_LIST=NO EPICS_CA_REPEATER_PORT=9093 EPICS_CA_SERVER_PORT=9094 |
EPICS_CA_ADDR_LIST=142.90.133.255 EPICS_CA_AUTO_ADDR_LIST=YES EPICS_CA_REPEATER_PORT=9003 EPICS_CA_SERVER_PORT=9004 |
NOTE: If you want to change which EPICS variables are used, you must change the ~/.bashrc file, source the updated version, then run kill-all; start-all.
EPICS Troubleshooting
All EPICS access depends on the EPICS Environment variables being set up correctly, and linking with the correct version of the EPICS software in epics-base. Check the #EPICS_environment_variables are set correctly.
Try reading using caget
When having trouble accessing EPICS, first try to read one of the IOCs, e.g. the Na/Rb cell voltage with caget, e.g.
caget ILE2:BIAS15:RDVOL ILE2:BIAS15:RDVOL 262.074
If instead the command times out
caget ILE2:BIAS15:RDVOL Channel connect timed out: 'ILE2:BIAS15:RDVOL' not found
this indicates a problem with the EPICS environment variable settings or a firewall issue. EPICS controls are firewalled. Try the caget command several times, as there may be a packet-loss problem where caget only works some of the time (though most likely your environment variables are incorrect, or you're connecting from the wrong machine or account).
From isdaq01, reading any EPICS variable should work.
Check access using cainfo
First see write access to EPICS Polarizer parameters. If access is not granted, writing to the Polarizer parameters will not be successful.
In order to write to an EPICS variable, permission must be granted by the EPICS group. This is done by user and cpu. For example, users bnmr and bnqr on isdaq01 are granted permission to write to the EPICS Experiment variables for bnmr and bnqr respectively. Writing to the Na/Rb cell voltage IOC 'ILE2:BIAS15:RDVOL' is granted to users bnmr and bnqr on lxbnmr and lxbnqr respectively.
Run cainfo to check for write permission, e.g.
[bnqr@isdaq01 bnqr]$ cainfo BNQR:VAR1
BNQR:VAR1
State: connected
Host: ile2ioc.triumf.ca:9004
Access: read, write
Native data type: DBF_DOUBLE
Request type: DBR_DOUBLE
Element count: 1
Try writing using caput
caput can be used to write to an IOC, e.g.
[bnmr@isdaq01 ~]$ caput BNMR:VAR1 5 Old : BNMR:VAR1 1.9869e+06 New : BNMR:VAR1 5
If write access is false, the value will not change, but no error message will be output.
Try reading/writing using epics_test.exe
The frontend and logger use a more user-friendly C++ wrapper around the native EPICS API. You can test this wrapper using the epics_test.exe and epics_test_32bit.exe programs that are built alongside the frontend/logger.
Usage is like:
epics_test.exe get BNQR:VAR1 # Connect to EPICS then read PV as a number epics_test.exe getstr BNQR:VAR1 # Connect to EPICS then read PV as a string epics_test.exe set BNQR:VAR1 2.5 # Connect to EPICS then set PV value
EPICS checking
- The frontend performs sanity-check of some EPICS variables at the start of the run - see the dual channel mode page for details.
- The frontend checks the state of the helicity waveplate during a run - see the helicity page for details.