BNMR: EPICS: Difference between revisions

From DaqWiki
Jump to navigation Jump to search
en>Suz
m (EPICS page moved to BNMR: EPICS page: Decided to prefix all BNMR pages with "BNMR:")
 
(37 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{Pagelinks}}
{{Pagelinks}}


= Links =
== Purpose ==
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
This page gives  {{bnmqr|join=or}} DAQ users information about the '''ISAC EPICS Control Pages''' and the '''Access''' required for running  {{bnmqr|join=or}}  experiments.
* [[BNMR]]
*
*
* [[Getting Started]]
</div>


== Introduction ==
== Introduction ==
Experimenters will need to display the Epics Control Pages in order to run the {{bnmqr}} experiment. They must also ensure that bnmr has been given [[#write access to Epics Parameters]], or they will not be able to change the relevant beamline controls (see [[#note]]).
The [http://www.aps.anl.gov/epics/ EPICS] system is used at TRIUMF for the control system for beamlines etc.
Experimenters will need to display the Epics Control Pages in order to run the {{bnmqr|join=or}} experiments. They must also ensure that ''BNMR'' has been granted write access to the required Epics parameters as described [[#Write_access_to_EPICS_parameters|below]], or they will not be able to change the relevant beamline controls.


== Display Epics Control Pages ==
== Display ISAC EPICS control pages ==
Switch onto a different desktop on {{linux|box=isdaq01}} (or isdaq06) so the Epics Control page does not clutter the desktop used for running the experiment. Access the epics control pages on isacepics1 with ssh from an xterm window, as follows (if you don't know the password, ask a bnmr group member).
Switch onto a different desktop on linux box isdaq01 (or isdaq06) so the Epics Control page does not clutter the desktop used for running the experiment. Access the epics control pages on isacepics1 with ssh from an xterm window, as follows (if you don't know the password, ask a group member).


  ssh bnmr@isacepics1
  ssh bnmr@isacepics1
  ( Type in the password )
  ( Type in the password )


You should see the main ISAC EPICS control page come up after a few seconds.
You should see the main ISAC EPICS control page come up after a few seconds (Figure 1).
 
[[Image:Isac_epics_main.png|center|thumb|600px|Figure 1: ISAC EPICS control page]]
<br clear=all>
 
 
<div id=note></div>  
<div id=note></div>  
;Note
;Note
:There is no bnqr account on isacepics1. The bnmr account is used for both bnmr/bnqr experiments and the same EPICS access permission (i.e. bnmr) applies to both experiments.
:There is no ''bnqr'' user account on isacepics1. The ''bnmr'' account is used for both experiments and the same EPICS access permission (i.e. ''BNMR'') applies to both.
 
== Write access to EPICS polarizer parameters ==
Most Epics parameters can be read by anyone, but write access has to be granted by the Controls Group.
Several experiments have been granted write access to the same polarizer IOCs, and to avoid conflict, only one of these experiments is granted overall Access.  During the beam period, and for testing the helicity switching etc, access must be granted to <span style=color:blue>expbnmr</span>.
 
To check whether <span style=color:blue>expbnmr</span> has access, display the main EPICS control pages.
 
On the main [[#Display_ISAC_EPICS_control_pages|EPICS Control Page]] (cf Figure 1) click the button labelled "Access". A page similar to Figure 2 will be displayed.
 
[[Image:Isac_access_control.png|center|thumb|600px|Figure 2: ISAC Access Control]]
<br clear=all>
Clicking on the '''View access''' button (Figure 2) will show the current access. This should be set to <span style=color:blue>expbnmr</span> as in Figure 3.
 
[[Image:Isac_polarizer_access.png|center|thumb|600px|Figure 3: Polarizer Access set to expbnmr]]
<br clear=all>
 
If the Access is not set to <span style=color:blue>expbnmr</span>, ask the operators to switch the access by pressing the button ''Polarizer Access for bNMR/bNQR'' (Figure 2).
 
Even when "Access" is granted to ''BNMR'', only those EPICS IOCs to which BNMR has been granted write access can be changed by the experimenters from the EPICS Control pages, e.g. Helicity, NaCell Voltage, and all BNMR/BNQR parameters for single/dual channel mode control.
 
Some of these parameters are also writable by the {{bnmqr|join=and}} BNMR and BNQR VMICs (lxbnmr,lxbnqr). They have been granted write access by the Controls Group to allow the frontend programs to flip helicity and turn the beam on/off via PPG signals. The frontends also use EPICS to scan the Rb cell voltage, for example.
 
=== WARNING ===
There is NO protection against  {{bnmqr|join=and}} experimenters changing Epics parameters via the Epics Control pages or the frontend programs while an experimental run is in progress on either {{bnmqr|join=or}}. When running in [[BNMR:_Dual_Channel_Mode#Single_Channel_Mode|single channel mode]], there is also no protection against each experiment interfering with each other by both flipping the helicity for example.  Users must ensure that they know what they are doing!
 
== Usage in software (frontend and logger) ==
 
EPICS is used by:
* the frontend to scan the Rb cell voltage, laser intensity etc in certain modes
* the frontend to report some histogram totals to operators
* the frontend to [[#EPICS_checking|sanity-check]] the dual channel mode and helicity settings
* the frontend to decide when to start the PPG in dual channel mode (optionally - see dual channel mode page for more details about when we use EPICS status vs a hardware signal to start the PPG)
* the logger to read any variables the user wants logging in the MUD file
 
=== UDP broadcast issue ===
 
The simplest way to use EPICS is to set the environment variable <code>EPICS_CA_AUTO_ADDR_LIST=YES</code>. This means that EPICS will do a UDP broadcast to find which IOC server is handling a given Process Variable, and you do not need to worry about the specifics of each PV. If you add new PVs you can just let EPICS find the correct IOC to talk to.
 
However, the TRIUMF firewall machines sometimes get into a state where lots of UDP broadcast packets get dropped (technical details according to Steve M: the data and control planes get split across different nodes). This prevents EPICS from reliably finding the correct IOC for a PV, and will report that many PVs cannot be found. For that reason, we can instead set <code>EPICS_CA_AUTO_ADDR_LIST=NO</code> and specify the IOCs of interest in the <code>EPICS_CA_ADDR_LIST</code> environment variable. This setup means that we do not need any UDP broadcast packets, and our EPICS communications work well even if the firewalls are behaving badly.
 
However, when adding a new EPICS PV to be scanned or sanity-checked, you must find out which IOC they are controlled by! This can be done either by emailing the controls group, or running the <code>cainfo</code> command - e.g. the following command shows that ile2ioc.triumf.ca (142.90.132.2) controls ILE2:BIAS15:VOL:
 
<pre>
[bnmr@isdaq01 ~]$ cainfo ILE2:BIAS15:VOL
ILE2:BIAS15:VOL
    State:            connected
    Host:            ile2ioc.triumf.ca:9004
    Access:          read, write
    Native data type: DBF_DOUBLE
    Request type:    DBR_DOUBLE
    Element count:    1
</pre>
 
Using <code>EPICS_CA_AUTO_ADDR_LIST=NO</code> is a much worse experience for shifters. If they want to log/control a PV that isn't controlled by an IOC that is already in <code>EPICS_CA_ADDR_LIST</code>, they must:
 
* Find out which IOC controls the PV using cainfo (or by emailing the Controls group if the network is really bad)
* Edit ~/.bashrc file to add the IOC to <code>EPICS_CA_ADDR_LIST</code>
* Source the updated ~/.bashrc file to pick up the change
* Restart all the DAQ programs with <code>kill-all; start-all</code>
 
Shifters often have many terminals open, and they must remember to source the new environment in each one before running <code>start-all</code>; otherwise it is very easy to restart programs with the wrong environment! Using <code>EPICS_CA_AUTO_ADDR_LIST=YES</code> is '''far''' superior unless the network is having issues.
 
=== EPICS environment variables ===
Currently (July 2022)  {{bnmqr|join=and}} experiments are using epics-baseR3.14.12.
<pre>
$ ls -lt /home/midas/packages/epics-base
/home/midas/packages/epics-base -> epics-baseR3.14.12/
</pre>
 
We have 3 common settings that may be used based on whether different parts of the network are having issues. You can change between them using the bash functions <code>use_epics_gw</code> etc.
 
{| border="1" style="border-collapse:collapse; color:black; background-color:aliceblue"  "
|+ Table 1: EPICS Environment Variables
!  !! Normal usage - connect directly to IOCs of interest !! Connect to gateway server (readonly) !! Connect to broadcast address (vulnerable to UDP issues)
|-
| '''Bash function''' || use_epics_direct || use_epics_gw || use_epics_broadcast
|-
| '''Settings'''
|
<pre>
EPICS_CA_ADDR_LIST=142.90.132.2 142.90.132.10
EPICS_CA_AUTO_ADDR_LIST=NO
EPICS_CA_REPEATER_PORT=9003
EPICS_CA_SERVER_PORT=9004
</pre>
|
<pre>
EPICS_CA_ADDR_LIST=142.90.132.2 142.90.132.10 142.90.132.20 142.90.132.92 142.90.133.57 142.90.133.68 142.90.133.84
EPICS_CA_AUTO_ADDR_LIST=NO
EPICS_CA_REPEATER_PORT=9093
EPICS_CA_SERVER_PORT=9094
</pre>
|
<pre>
EPICS_CA_ADDR_LIST=142.90.133.255
EPICS_CA_AUTO_ADDR_LIST=YES
EPICS_CA_REPEATER_PORT=9003
EPICS_CA_SERVER_PORT=9004
</pre>
|}
 
NOTE: If you want to change which EPICS variables are used, you must change the ~/.bashrc file, source the updated version, then run <code>kill-all; start-all</code>.
 
=== EPICS Troubleshooting ===
All EPICS access depends on the EPICS Environment variables being set up correctly, and linking with the correct version of the EPICS software in epics-base.
Check the [[#EPICS_environment_variables]] are set correctly.
 
==== Try reading using caget ====
 
When having trouble accessing EPICS, first try to read one of the IOCs, e.g. the Na/Rb cell voltage with '''caget''', e.g.
<pre>
caget ILE2:BIAS15:RDVOL
ILE2:BIAS15:RDVOL              262.074
</pre>
 
If instead the command times out
<pre>
caget ILE2:BIAS15:RDVOL
Channel connect timed out: 'ILE2:BIAS15:RDVOL' not found
</pre>
this indicates a problem with the EPICS environment variable settings or a firewall issue. EPICS controls are firewalled. Try the caget command several times, as there may be a packet-loss problem where caget only works some of the time (though most likely your environment variables are incorrect, or you're connecting from the wrong machine or account).
 
From isdaq01, reading any EPICS variable should work.
 
==== Check access using cainfo ====
First see [[#Write_access_to_EPICS_polarizer_parameters|write access to EPICS Polarizer parameters]]. If access is not granted, writing to the Polarizer parameters will not be successful.
 
In order to write to an EPICS variable, permission must be granted by the EPICS group. This is done by user and cpu. For example, users ''bnmr'' and ''bnqr'' on '''isdaq01''' are granted permission to write to the EPICS Experiment variables for bnmr and bnqr respectively. Writing to the Na/Rb cell voltage IOC 'ILE2:BIAS15:RDVOL' is granted to users ''bnmr'' and ''bnqr'' on '''lxbnmr''' and '''lxbnqr''' respectively.
 
Run '''cainfo''' to check for write permission, e.g.
<pre>
[bnqr@isdaq01 bnqr]$ cainfo BNQR:VAR1
BNQR:VAR1
    State:            connected
    Host:            ile2ioc.triumf.ca:9004
    Access:          read, write
    Native data type: DBF_DOUBLE
    Request type:    DBR_DOUBLE
    Element count:    1
</pre>


==== Try writing using caput ====


== Write access to Epics Parameters ==
'''caput''' can be used to write to an IOC, e.g.  
Most Epics parameters can be read by anyone, but write access has to be granted by the Controls Group. Only those EPICS parameters to which users bnmr and bnqr have been granted write access can be changed by the experimenters, and only if "Access" is granted to the bnmr experiment. As far as EPICS write access is concerned, if access is granted to bnmr, then bnqr also has access.
<pre>
[bnmr@isdaq01 ~]$  caput BNMR:VAR1 5
Old : BNMR:VAR1                      1.9869e+06
New : BNMR:VAR1                      5
</pre>
If write access is false, the value will not change, but no error message will be output.


At the beginning of the bnmr/bnqr beam period, it is therefore essential to ensure that the "Access" parameter on the main ISAC Epics controls page is set to "BNMR".
==== Try reading/writing using epics_test.exe ====


;If "Access" is '''NOT''' set to "BNMR", telephone the Control Room and ask them to change the Access.
The frontend and logger use a more user-friendly C++ wrapper around the native EPICS API. You can test this wrapper using the <code>epics_test.exe</code> and <code>epics_test_32bit.exe</code> programs that are built alongside the frontend/logger.


Examples of EPICS parameters to which the controls group have granted bnmr  write access are Helicity, NaCell voltage, and all BNMR/BNQR parameters for single/dual channel mode control.
Usage is like:
<pre>
epics_test.exe get BNQR:VAR1      # Connect to EPICS then read PV as a number
epics_test.exe getstr BNQR:VAR1    # Connect to EPICS then read PV as a string
epics_test.exe set BNQR:VAR1 2.5  # Connect to EPICS then set PV value
</pre>


Some of these parameters are also writable by the BNMR and BNQR frontends (PPCs), e.g. Helicity and Sodium Cell voltage. In some modes, an Epics parameter such as the Sodium Cell voltage is scanned. In all modes,  the frontend PPC is responsible for flipping the helicity each cycle ( by hardware via a PPG signal to the Epics Helicity switch).
=== EPICS checking ===
* The [[BNMR: frontend|frontend]] performs sanity-check of some EPICS variables at the start of the run - see the [[BNMR:_Dual_Channel_Mode#Frontend_sanity-checks|dual channel mode]] page for details.
* The [[BNMR: frontend|frontend]] checks the state of the helicity waveplate during a run - see the [[BNMR:_Helicity#EPICS_Helicity_checks|helicity]] page for details.


WARNING:
[[Category:BNMR]] [[Category:EPICS]]
There is NO protection against users changing Epics parameters via the Epics Control pages while an experimental run is in progress on either bnmr or bnqr DAQs.  Users must ensure that they know what they are doing!

Latest revision as of 10:24, 18 July 2022

Purpose

This page gives bnmr or bnqr DAQ users information about the ISAC EPICS Control Pages and the Access required for running bnmr or bnqr experiments.

Introduction

The EPICS system is used at TRIUMF for the control system for beamlines etc. Experimenters will need to display the Epics Control Pages in order to run the bnmr or bnqr experiments. They must also ensure that BNMR has been granted write access to the required Epics parameters as described below, or they will not be able to change the relevant beamline controls.

Display ISAC EPICS control pages

Switch onto a different desktop on linux box isdaq01 (or isdaq06) so the Epics Control page does not clutter the desktop used for running the experiment. Access the epics control pages on isacepics1 with ssh from an xterm window, as follows (if you don't know the password, ask a group member).

ssh bnmr@isacepics1
( Type in the password )

You should see the main ISAC EPICS control page come up after a few seconds (Figure 1).

Figure 1: ISAC EPICS control page



Note
There is no bnqr user account on isacepics1. The bnmr account is used for both experiments and the same EPICS access permission (i.e. BNMR) applies to both.

Write access to EPICS polarizer parameters

Most Epics parameters can be read by anyone, but write access has to be granted by the Controls Group. Several experiments have been granted write access to the same polarizer IOCs, and to avoid conflict, only one of these experiments is granted overall Access. During the beam period, and for testing the helicity switching etc, access must be granted to expbnmr.

To check whether expbnmr has access, display the main EPICS control pages.

On the main EPICS Control Page (cf Figure 1) click the button labelled "Access". A page similar to Figure 2 will be displayed.

Figure 2: ISAC Access Control


Clicking on the View access button (Figure 2) will show the current access. This should be set to expbnmr as in Figure 3.

Figure 3: Polarizer Access set to expbnmr


If the Access is not set to expbnmr, ask the operators to switch the access by pressing the button Polarizer Access for bNMR/bNQR (Figure 2).

Even when "Access" is granted to BNMR, only those EPICS IOCs to which BNMR has been granted write access can be changed by the experimenters from the EPICS Control pages, e.g. Helicity, NaCell Voltage, and all BNMR/BNQR parameters for single/dual channel mode control.

Some of these parameters are also writable by the bnmr and bnqr BNMR and BNQR VMICs (lxbnmr,lxbnqr). They have been granted write access by the Controls Group to allow the frontend programs to flip helicity and turn the beam on/off via PPG signals. The frontends also use EPICS to scan the Rb cell voltage, for example.

WARNING

There is NO protection against bnmr and bnqr experimenters changing Epics parameters via the Epics Control pages or the frontend programs while an experimental run is in progress on either bnmr or bnqr. When running in single channel mode, there is also no protection against each experiment interfering with each other by both flipping the helicity for example. Users must ensure that they know what they are doing!

Usage in software (frontend and logger)

EPICS is used by:

  • the frontend to scan the Rb cell voltage, laser intensity etc in certain modes
  • the frontend to report some histogram totals to operators
  • the frontend to sanity-check the dual channel mode and helicity settings
  • the frontend to decide when to start the PPG in dual channel mode (optionally - see dual channel mode page for more details about when we use EPICS status vs a hardware signal to start the PPG)
  • the logger to read any variables the user wants logging in the MUD file

UDP broadcast issue

The simplest way to use EPICS is to set the environment variable EPICS_CA_AUTO_ADDR_LIST=YES. This means that EPICS will do a UDP broadcast to find which IOC server is handling a given Process Variable, and you do not need to worry about the specifics of each PV. If you add new PVs you can just let EPICS find the correct IOC to talk to.

However, the TRIUMF firewall machines sometimes get into a state where lots of UDP broadcast packets get dropped (technical details according to Steve M: the data and control planes get split across different nodes). This prevents EPICS from reliably finding the correct IOC for a PV, and will report that many PVs cannot be found. For that reason, we can instead set EPICS_CA_AUTO_ADDR_LIST=NO and specify the IOCs of interest in the EPICS_CA_ADDR_LIST environment variable. This setup means that we do not need any UDP broadcast packets, and our EPICS communications work well even if the firewalls are behaving badly.

However, when adding a new EPICS PV to be scanned or sanity-checked, you must find out which IOC they are controlled by! This can be done either by emailing the controls group, or running the cainfo command - e.g. the following command shows that ile2ioc.triumf.ca (142.90.132.2) controls ILE2:BIAS15:VOL:

[bnmr@isdaq01 ~]$ cainfo ILE2:BIAS15:VOL
ILE2:BIAS15:VOL
    State:            connected
    Host:             ile2ioc.triumf.ca:9004
    Access:           read, write
    Native data type: DBF_DOUBLE
    Request type:     DBR_DOUBLE
    Element count:    1

Using EPICS_CA_AUTO_ADDR_LIST=NO is a much worse experience for shifters. If they want to log/control a PV that isn't controlled by an IOC that is already in EPICS_CA_ADDR_LIST, they must:

  • Find out which IOC controls the PV using cainfo (or by emailing the Controls group if the network is really bad)
  • Edit ~/.bashrc file to add the IOC to EPICS_CA_ADDR_LIST
  • Source the updated ~/.bashrc file to pick up the change
  • Restart all the DAQ programs with kill-all; start-all

Shifters often have many terminals open, and they must remember to source the new environment in each one before running start-all; otherwise it is very easy to restart programs with the wrong environment! Using EPICS_CA_AUTO_ADDR_LIST=YES is far superior unless the network is having issues.

EPICS environment variables

Currently (July 2022) bnmr and bnqr experiments are using epics-baseR3.14.12.

$ ls -lt /home/midas/packages/epics-base
 /home/midas/packages/epics-base -> epics-baseR3.14.12/

We have 3 common settings that may be used based on whether different parts of the network are having issues. You can change between them using the bash functions use_epics_gw etc.

Table 1: EPICS Environment Variables
Normal usage - connect directly to IOCs of interest Connect to gateway server (readonly) Connect to broadcast address (vulnerable to UDP issues)
Bash function use_epics_direct use_epics_gw use_epics_broadcast
Settings
EPICS_CA_ADDR_LIST=142.90.132.2 142.90.132.10
EPICS_CA_AUTO_ADDR_LIST=NO
EPICS_CA_REPEATER_PORT=9003
EPICS_CA_SERVER_PORT=9004
EPICS_CA_ADDR_LIST=142.90.132.2 142.90.132.10 142.90.132.20 142.90.132.92 142.90.133.57 142.90.133.68 142.90.133.84
EPICS_CA_AUTO_ADDR_LIST=NO
EPICS_CA_REPEATER_PORT=9093
EPICS_CA_SERVER_PORT=9094
EPICS_CA_ADDR_LIST=142.90.133.255
EPICS_CA_AUTO_ADDR_LIST=YES
EPICS_CA_REPEATER_PORT=9003
EPICS_CA_SERVER_PORT=9004

NOTE: If you want to change which EPICS variables are used, you must change the ~/.bashrc file, source the updated version, then run kill-all; start-all.

EPICS Troubleshooting

All EPICS access depends on the EPICS Environment variables being set up correctly, and linking with the correct version of the EPICS software in epics-base. Check the #EPICS_environment_variables are set correctly.

Try reading using caget

When having trouble accessing EPICS, first try to read one of the IOCs, e.g. the Na/Rb cell voltage with caget, e.g.

 caget ILE2:BIAS15:RDVOL
 ILE2:BIAS15:RDVOL              262.074

If instead the command times out

 caget ILE2:BIAS15:RDVOL
 Channel connect timed out: 'ILE2:BIAS15:RDVOL' not found

this indicates a problem with the EPICS environment variable settings or a firewall issue. EPICS controls are firewalled. Try the caget command several times, as there may be a packet-loss problem where caget only works some of the time (though most likely your environment variables are incorrect, or you're connecting from the wrong machine or account).

From isdaq01, reading any EPICS variable should work.

Check access using cainfo

First see write access to EPICS Polarizer parameters. If access is not granted, writing to the Polarizer parameters will not be successful.

In order to write to an EPICS variable, permission must be granted by the EPICS group. This is done by user and cpu. For example, users bnmr and bnqr on isdaq01 are granted permission to write to the EPICS Experiment variables for bnmr and bnqr respectively. Writing to the Na/Rb cell voltage IOC 'ILE2:BIAS15:RDVOL' is granted to users bnmr and bnqr on lxbnmr and lxbnqr respectively.

Run cainfo to check for write permission, e.g.

[bnqr@isdaq01 bnqr]$ cainfo BNQR:VAR1
BNQR:VAR1
    State:            connected
    Host:             ile2ioc.triumf.ca:9004
    Access:           read, write
    Native data type: DBF_DOUBLE
    Request type:     DBR_DOUBLE
    Element count:    1

Try writing using caput

caput can be used to write to an IOC, e.g.

[bnmr@isdaq01 ~]$  caput BNMR:VAR1 5
Old : BNMR:VAR1                      1.9869e+06
New : BNMR:VAR1                      5

If write access is false, the value will not change, but no error message will be output.

Try reading/writing using epics_test.exe

The frontend and logger use a more user-friendly C++ wrapper around the native EPICS API. You can test this wrapper using the epics_test.exe and epics_test_32bit.exe programs that are built alongside the frontend/logger.

Usage is like:

epics_test.exe get BNQR:VAR1       # Connect to EPICS then read PV as a number
epics_test.exe getstr BNQR:VAR1    # Connect to EPICS then read PV as a string
epics_test.exe set BNQR:VAR1 2.5   # Connect to EPICS then set PV value

EPICS checking

  • The frontend performs sanity-check of some EPICS variables at the start of the run - see the dual channel mode page for details.
  • The frontend checks the state of the helicity waveplate during a run - see the helicity page for details.