POL: Difference between revisions

From DaqWiki
Jump to navigation Jump to search
No edit summary
No edit summary
 
(6 intermediate revisions by the same user not shown)
Line 6: Line 6:
</div>
</div>


== DAQ VERSIONS ==
=== Starting the DAQ ===


<span style="color:red">There are currently two versions of the DAQ. This wiki documents the new version, written in 2021/2022. Documentation of the old DAQ can be found on the [http://www.triumf.info/wiki/DAQwiki/index.php/POL triumf.info wiki].</span>
ssh pol@midpol
start-all


To use the new DAQ, type <code>bash</code> after logging in to pol@isdaq01.
Then go to https://midpol.triumf.ca to see the DAQ webpages.
 
Data is written to <code>/data1/pol/data/current</code>.


== DAQ machines ==
== DAQ machines ==


* isdaq01 - main daq computer, located in ISAC counting room, shared with bnmr/bnqr experiments
* midpol - main daq computer, located on ISAC-1 floor behind the BNMR cage
* lxpol - VME interface
* lxpol - VME controller in the VME rack near midpol
* isdaq04 - ISAC1 floor terminal


Main account is pol@isdaq01.
Main account is pol@midpol.


== The POL DAQ System ==
== The POL DAQ System ==
Line 29: Line 31:


All the other software runs on the [[POL: Hardware and software#How to switch the Host Computer|host computer]]
All the other software runs on the [[POL: Hardware and software#How to switch the Host Computer|host computer]]
(currently ''isdaq01''). Both ''lxpol'' and ''isdaq01'' run linux.
(currently ''midpol''). Both ''lxpol'' and ''midpol'' run linux.


== Running the POL Experiment ==
== Running the POL Experiment ==
=== Starting the POL Experiment from scratch ===
=== Starting the POL Experiment from scratch ===


* Log on to DAQ host ''isdaq01'' with username ''pol''
* Log on to DAQ host ''midpol'' with username ''pol''
* In an terminal, type ''start-all''
* In an terminal, type ''start-all''


More details about ''start-all'' is in the [POL: Hardware and software#Scripts|scripts section].
More details about ''start-all'' is in the [[POL: Hardware and software#Scripts|scripts section]].


=== Viewing the state of the DAQ programs ===
=== Viewing the state of the DAQ programs ===


Most interaction with the DAQ is done through the web interface at https://isdaq01.triumf.ca:8446.
Most interaction with the DAQ is done through the web interface at https://midpol.triumf.ca.
 
However, you can also view the state of the main DAQ program (called the frontend) by typing ''show-windows'' in a terminal when logged in as pol@midpol. (If connecting to remotely via ssh, you'll need to enable X11 forwarding to see the window - e.g. <code>ssh -Y pol@midpol</code>).
 
=== Configuring settings/parameters ===


However, you can also view the state of the main DAQ program (called the frontend) by typing ''show-windows'' in a terminal when logged in as pol@isdaq01. (If connecting to remotely via ssh, you'll need to enable X11 forwarding to see the window - e.g. <code>ssh -Y pol@isdaq01</code>).
See the documentation about the [[POL:_Webpages#POL_Main_Status_Page|main status page]] for settings that users are expected to edit.
 
=== Starting a run ===
 
Click the "Start" button on the [[POL:_Webpages#POL_Main_Status_Page|main status page]], choose whether to write data to disk or not, then confirm that you want to start the run.


=== Shutdown the POL DAQ ===
=== Shutdown the POL DAQ ===
Line 49: Line 59:
To shutdown the POL DAQ software completely:
To shutdown the POL DAQ software completely:


* Log on to DAQ host ''isdaq01'' with username ''pol''
* Log on to DAQ host ''midpol'' with username ''pol''
* In an terminal, type ''kill-all''
* In an terminal, type ''kill-all''


This will shut down all the clients for the POL experiment, including the web server ''mhttpd''.
This will shut down all the clients for the POL experiment, including the web server ''mhttpd''.


== Nomenclature ==
== Nomenclature ==
Line 169: Line 178:


== Troubleshooting ==
== Troubleshooting ==
=== General advice ===
<ol>
<li>See if there's an obvious error message on the "Messages" webpage</li>
<li>Try restarting the DAQ programs</li>
<li>Try power-cycling the VME crate</li>
</ol>


=== The run would not start ===
=== The run would not start ===
Check the Error/Information message log for more information. Reasons for the run not starting include :
Check the "Messages" webpage for more information. Reasons for the run not starting include :
<ul>
<ul>
<li>frontend code is not running</li>
<li>frontend code is not running</li>
<li>error in frontend parameter values</li>
<li>error in frontend parameter values</li>
<li>VME crate powered down
<li>hardware not responding properly</li>
</li>
</ul>
</ul>


=== The run stopped unexpectedly ===
=== The run stopped unexpectedly ===
Check the Error/Information message
Check the "Messages" webpage for more information. Reasons for the run stopping include:
log for more information. Reasons for the run stopping unexpectedly include:
<ul>
<ul>
<li>Run reached number of scans requested (input parameter ''number of scans'' is non-zero)</li>
<li>Run reached number of scans requested (input parameter ''Stop after N loops'' is non-zero)</li>
<li>DAC value did not change after increment</li>
<li>The frontend code detected a fatal error - see message log and frontend xterm</li>
<li>The frontend code detected a fatal error - see message log and frontend xterm</li>
</ul>
</ul>


=== DAC Scan stops with error; cannot step the DAC successfully ===
If the "Messages" webpage shows errors of the form:
<pre>
Scaler is showing 2948 bins at start of cycle rather than expected 0
Data source Histos isn't ready for the start of the next cycle!
</pre>
then the scalers are either configured incorrectly, or are not behaving as expected. You will likely need to use the <code>sis3820_interactive_32bit.exe</code> debugging program on lxpol to view the registers on the scaler and ensure they are as expected (especially the LNE prescale setting and whether it's in arm or enable mode). If the scaler settings look okay, there may be a programming error in the DAQ.


The following refers to check parameters in the [[#Information Table]] i.e. ''DAC check'', ''jitter'': 
=== Run starts, but scan does not progress ===
 
Check the stdout of the frontend program for repeating lines like "Scaler has seen 0/10100 bins".
;Note:
:''DAC check'' should now be disabled in order to run with small DAC increments. ''DAC check'' should only be enabled for testing.
 
If running with ''DAC check'' enabled, check the following parameters in the [[#Information Table]]: 
* check jitter parameter is reasonable (i.e.  &gt;=  [[POL Experiment DAQ documentation##Stepping the DAC| DAC LSB]]) .
 
* If  DAC_increment is very small (i.e. close to LSB or jitter), you will have to run with  ''DAC check'' disabled (i.e. no  [[POL Experiment DAQ documentation#Stepping the DAC" | checks on DAC stepping ]]).
 
=== Run starts but DAC not scanning, no data ===
 
The frontend code is probably not running, and the [[#Alarm_System" | alarm system]] disabled.
If the alarm system is enabled, the run will be prevented from starting, and an alarm banner
will appear to indicate the problem.
 
Otherwise, check that the hardware (Galil Rio, Power supply etc.) is all switched on and
working.  The PPG relies on an external signal from the TOF
to start the next cycle. If this signal is not present, the PPG will run one cycle then stop, waiting for an external start.
 
For testing, can switch PPG external start to internal (under Frontend Flags on the Custom Page) . The PPG should then continue past the first cycle. In this case, investigate connection of End TOF Signal to PPG External Start.


<ul>
<li>If the scaler has seen 0 or -1 bins, check the cabling between the PPG and the scaler, as the PPG probably never started</li>
<li>If scaler has seen 100 bins (or however many time bins you specified for the "Num time bins" setting) and you're using pulsed beam mode, check the cabling between the RFQ and the PPG</li>
<li>If the scaler has seen 10099/10100 bins (or one less than the expected number), there is either a programming error in the DAQ or the scaler has been manually configured incorrectly (it is likely configured for "arm" logic rather than "enable" logic). Try restarting the frontend.</li>
</ul>


=== Alarm banner is present ===
=== Alarm banner is present ===
Line 230: Line 234:


<pre>
<pre>
pol@isdaq01 pol]$ ssh lxpol
[pol@midpol pol]$ ssh lxpol
Last login: Wed Apr  2 16:05:12 2014 from isdaq01.triumf.ca
[pol@lxpol ~]$ bash
 
(new daq) [pol@lxpol ~]$ pol_new/packages/pol/bin/fepol_32bit.exe
[pol@lxpol ~]$ online/pol/pol_fevme.exe
</pre>
</pre>


Line 251: Line 254:
;NOTE
;NOTE
: If you cannot ssh to lxpol, check that the VME crate is powered up and the network is working.
: If you cannot ssh to lxpol, check that the VME crate is powered up and the network is working.


=== How to switch the Host Computer ===
=== How to switch the Host Computer ===
The host computer is the desktop computer that users of the ''pol'' experiment
The host computer is the desktop computer that users of the ''pol'' experiment
log into to run the DAQ. Currently it is '''isdaq01'''.  If isdaq01 dies, the experiment can
log into to run the DAQ. Currently it is '''midpol'''.  If midpol dies, the experiment can
be switched to run on an alternate host that is part of the isdaq00 NIS cluster, e.g. isdaq06.
be switched to run on an alternate host that is part of the isdaq00 NIS cluster, e.g. isdaq06.
The alternate machine also has access to the  {{Filepath|path=/home/pol}} directory.
The alternate machine also has access to the  {{Filepath|path=/home/pol}} directory.


* If the current host computer (isdaq01) is still running, type ''kill-all'' and log out.
* If the current host computer (midpol) is still running, type ''kill-all'' and log out.
* Log onto the alternate host computer (isdaq06)
* Log onto the alternate host computer (isdaq06)
** Edit the definition of <code>$DAQ_HOST</code> in /home/pol/.bashrc
** Edit the definition of <code>$DAQ_HOST</code> in /home/pol/.bashrc
** Remove the SHM host file <code>rm /home/pol/online/pol/.SHM_HOST.TXT</code>
** Edit the SHM host file <code>/home/pol/online/.SHM_HOST.TXT</code>
** Log out and log in again
** Log out and log in again
** Check environment variable DAQ_HOST is now set to isdaq06  
** Check environment variable DAQ_HOST is now set to isdaq06  
Line 270: Line 270:


== POL DAQ updates ==
== POL DAQ updates ==
* February 2023 DAQ Host is now midpol
* January 2022 Rewritten using the cycling framework
* January 2022 Rewritten using the cycling framework
* April 2016 DAQ Host is now isdaq01
* April 2016 DAQ Host is now isdaq01

Latest revision as of 14:51, 17 February 2023

Links

Starting the DAQ

ssh pol@midpol
start-all

Then go to https://midpol.triumf.ca to see the DAQ webpages.

Data is written to /data1/pol/data/current.

DAQ machines

  • midpol - main daq computer, located on ISAC-1 floor behind the BNMR cage
  • lxpol - VME controller in the VME rack near midpol

Main account is pol@midpol.

The POL DAQ System

The DAQ system for the POL experiment uses MIDAS, the standard DAQ system at TRIUMF.

The POL experiment has its own DAQ hardware (i.e. VME crate and DAQ modules). These are controlled and read out by a MIDAS frontend fepol_32bit.exe, which runs on a VMIC VME processor called lxpol.

All the other software runs on the host computer (currently midpol). Both lxpol and midpol run linux.

Running the POL Experiment

Starting the POL Experiment from scratch

  • Log on to DAQ host midpol with username pol
  • In an terminal, type start-all

More details about start-all is in the scripts section.

Viewing the state of the DAQ programs

Most interaction with the DAQ is done through the web interface at https://midpol.triumf.ca.

However, you can also view the state of the main DAQ program (called the frontend) by typing show-windows in a terminal when logged in as pol@midpol. (If connecting to remotely via ssh, you'll need to enable X11 forwarding to see the window - e.g. ssh -Y pol@midpol).

Configuring settings/parameters

See the documentation about the main status page for settings that users are expected to edit.

Starting a run

Click the "Start" button on the main status page, choose whether to write data to disk or not, then confirm that you want to start the run.

Shutdown the POL DAQ

To shutdown the POL DAQ software completely:

  • Log on to DAQ host midpol with username pol
  • In an terminal, type kill-all

This will shut down all the clients for the POL experiment, including the web server mhttpd.

Nomenclature

Bin

see #time bin

Cycle

see #PPG Cycle

DAC

The Galil Rio DAC is scanned during the experiment (see #DAC scan).

Dwell Time

The Dwell Time is the same as the Bin Width. This is the time the Scaler collects the data for each #time bin.

DAC Scan (or Sweep)
  • Users configure the voltage range and number of increments the DAC should step through. Note that if you specify 10 steps, data will be taken at 11 different voltages. This makes the step size calculation more intuitive (0 to 4V in 10 steps ⇒ 0.4V between data points ⇒ 11 data points as we must include both the lower and upper bounds).
  • At begin-of-run, the DAC is set to the value given by DAC Start (V).
  • After one #supercycle (i.e. one or more PPG #cycles), the DAC voltage is incremented by the value given by DAC increment (V)
  • When the requested number of Number of DAC increments are done, one DAC scan has been completed.
  • A number of DAC scans is usually done during each #run.
Galil RIO

DAC/ADC module (see Galil Rio)

Input Parameters

The Input Parameters are parameters whose values are input by the user at the beginning of each run. These parameters control the experiment. The input parameters are available on the POL custom page.

PPG

VME Pulse programmer module (see DAQ Hardware). The PPG outputs a pattern of pulses to control the experiment.


PPG Cycle
  • One PPG Cycle is called a cycle.
  • A PPG Cycle starts when the PPG receives a "start" signal by
  • A PPG Cycle ends when its program finishes.
  • The length of the PPG cycle*is determined by the input parameters.
    • number of time bins and
    • dwell time (ms)
    *assuming that the input parameters DAC sleep time (ms) and TOF pulse width (ms), which are also part of the PPG cycle, are set to 0 (i.e. the minimum delay (30ns)).
Run
  • A run starts when the user starts the DAQ system (by pressing Starton the main status page or Pol custom page)
  • A run ends
    • when the user stops the DAQ system (by pressing Stop on the main status or custom page) or
    • automatically if an error is detected
SIS3820 Scaler/MCS

VME multichannel scaler module (see DAQ Hardware).

Supercycle
  • One or more #ppg cycles at the same DAC voltage is called one supercycle
  • The number of ppg cycles per supercycle is determined by the input parameter Num PPG cycles per DAC increment
  • If the DAC is incremented after each PPG cycle i.e. Num PPG cycles per DAC increment = 1, cycle and supercycle are identical
  • The time bin data from each cycle within one supercycle is summed.
  • At the end of each supercycle, the scaler data are output (as MIDAS Banks) and the DAC is incremented.
Time Bin

The time during the #scaler is enabled to collect data is divided into a number of time bins, each of length given by the dwell time. Both of these values are input parameters used to program the PPG. The scaler advances to the next time bin when it receives the LNE signal from the PPG (see Hardware).


Troubleshooting

General advice

  1. See if there's an obvious error message on the "Messages" webpage
  2. Try restarting the DAQ programs
  3. Try power-cycling the VME crate

The run would not start

Check the "Messages" webpage for more information. Reasons for the run not starting include :

  • frontend code is not running
  • error in frontend parameter values
  • hardware not responding properly

The run stopped unexpectedly

Check the "Messages" webpage for more information. Reasons for the run stopping include:

  • Run reached number of scans requested (input parameter Stop after N loops is non-zero)
  • The frontend code detected a fatal error - see message log and frontend xterm

If the "Messages" webpage shows errors of the form:

Scaler is showing 2948 bins at start of cycle rather than expected 0
Data source Histos isn't ready for the start of the next cycle!

then the scalers are either configured incorrectly, or are not behaving as expected. You will likely need to use the sis3820_interactive_32bit.exe debugging program on lxpol to view the registers on the scaler and ensure they are as expected (especially the LNE prescale setting and whether it's in arm or enable mode). If the scaler settings look okay, there may be a programming error in the DAQ.

Run starts, but scan does not progress

Check the stdout of the frontend program for repeating lines like "Scaler has seen 0/10100 bins".

  • If the scaler has seen 0 or -1 bins, check the cabling between the PPG and the scaler, as the PPG probably never started
  • If scaler has seen 100 bins (or however many time bins you specified for the "Num time bins" setting) and you're using pulsed beam mode, check the cabling between the RFQ and the PPG
  • If the scaler has seen 10099/10100 bins (or one less than the expected number), there is either a programming error in the DAQ or the scaler has been manually configured incorrectly (it is likely configured for "arm" logic rather than "enable" logic). Try restarting the frontend.

Alarm banner is present

This indicates an error that may prevent the run from starting. Fix the problem and clear the alarm.

Frontend code will not run - frontend window disappears immediately

This means that the frontend code has detected an error, and exits. Check for Error Messages. If necessary, start the frontend code in an xterm to get more information, because the window will not disappear on failure.

This error will occur if there is a problem with the VME Crate (e.g. powered down), frontend program not running, Galil Rio DAC/ADC not responding etc.

How to Start frontend code in an xterm

Log onto the VMIC device lxpol, e.g. from an xterm on the host or local computer, and start the frontend code, e.g.

[pol@midpol pol]$ ssh lxpol
[pol@lxpol ~]$ bash
(new daq) [pol@lxpol ~]$ pol_new/packages/pol/bin/fepol_32bit.exe

The frontend will run and go through its initialization procedure.

It may detect an error, e.g.

Connecting to Galil device at IP 192.168.1.100
terminate called after throwing an instance of 'KOsocketException'
Abort

In this case, there is a hardware problem with the Galil Rio DAC and the frontend code is aborted. Check that the Galil Rio DAC is present, powered up and connected to the VMIC via a network connection.

NOTE
If you cannot ssh to lxpol, check that the VME crate is powered up and the network is working.

How to switch the Host Computer

The host computer is the desktop computer that users of the pol experiment log into to run the DAQ. Currently it is midpol. If midpol dies, the experiment can be switched to run on an alternate host that is part of the isdaq00 NIS cluster, e.g. isdaq06. The alternate machine also has access to the /home/pol directory.

  • If the current host computer (midpol) is still running, type kill-all and log out.
  • Log onto the alternate host computer (isdaq06)
    • Edit the definition of $DAQ_HOST in /home/pol/.bashrc
    • Edit the SHM host file /home/pol/online/.SHM_HOST.TXT
    • Log out and log in again
    • Check environment variable DAQ_HOST is now set to isdaq06
    • Execute start-all and point the webserver to isdaq06:8088

POL DAQ updates

  • February 2023 DAQ Host is now midpol
  • January 2022 Rewritten using the cycling framework
  • April 2016 DAQ Host is now isdaq01
  • May 2015 frontend window now created using screen utility
  • July 2014 Galil Rio loaded with internal program (by Donald) to average ADC0-3 values. Averaged values are sent to banks CYCL.
  • May 2014 DAQ frontend code partially rewritten to reduce dead time.
  • April 2014 Galil Rio DAC/ADC replaces GPIB DAC . DAQ host is now isdaq06.
  • 2013 GPIB DAC controlled directly from isdaq01 rather than midtis07
  • April 2011 VMIC- based system for POL (replaces PPC-based system shared with BNQR expt). Pol's new scaler isthe VME scaler (SIS3820)