EBIT DAQ

Links

• TITAN DAQ General Documentation that explains the core principles of the DAQ system
• DAQ status
• Main settings
• Online plots
• titan_data repository for user-friendly python tools for reading the data
• The EBIT DAQ system is based on the MIDAS DAQ package

Hardware

EBIT has 2 main hardware components the PPG and the MCA4.

PPG32 (Pulse Programmer)

The PPG used by EBIT is a PPG32, built at TRIUMF (see manual). The PPG32 is designed to be clocked at 10MHz, and an internal "divide down" is used to produce frequencies of up to 100MHz. Presently only the internal clock is used, and the PPG is clocked at 100MHz. In the future, the PPG may be clocked using the special TITAN 100MHz external clock, in a similar way to the MPET and CPET PPGs.

Due to the requirement that the PPG32 be clocked externally by the special TITAN 100MHz "since wave" pulse generator, the firmware has been modified to allow this with a particular value of "divide downs". This also requires that the module be set to TTL Input Clock and Trigger signals using an jumper on the board. It is planned to change the PPG to a newer version, where the individual PPG Outputs may also be set to NIM or TTL with jumpers.

The PPG32 supports up to 32 Output channels, and the minimal delay is 3 clock cycles. However, the PPG is only fitted with 16 NIM connectors, limiting the device to 16 channels. More NIM connectors may be added if required.

MCA4 / MCS4 (time-of-flight)

The MCA4/MCS4 is a USB-based device that produces time-of-flight histograms. There is one BNC input that starts a clock, and two inputs that can "stop" the clock. The two "stop" channels work independently (i.e. you can measure the "stop" times of both channels even though the "start" signal was only sent once). The bin width of the histograms (aka the dwell time) and the number of bins is configurable. The minimum dwell time is 30ns if only one "stop" channel is being used, or 50ns if both "stop" channels are enabled. The maximum number of bins is 16 million, but reading out that much data would significantly degrade DAQ performance.

If using the MCA4, ensure the "feebit_mca4" program is running (using the Programs webpage)

The MCA4 is programmed to expect TTL (positive) signals!

Feature configuration

The DAQ system is configured on the Scan settings webpage.

Scanning

It is possible to automatically scan different EPICS variables and/or PPG variables (pulse widths and time offsets). See the TITAN_DAQ_General_Documentation#Core_principles for more information about 1D and 2D scans.

Enabling/disabling the PPG program

You can choose whether to run the PPG program or not during the run. Disabling the PPG program still allows you to record MCA4 data, and still allows you to scan EPICS variables and/or toggle the Faraday cup in/out of the beamline if desired.

If the PPG is disable, you must choose how long each "cycle" is (i.e. how long to take data for before moving on to the next step); the default is 10 seconds. (When the PPG is enabled, the cycle duration is dictated by the length of the PPG program.) Even if you aren't scanning/toggling anything, you should still set a reasonable cycle length; data is only written to disk at the end of each cycle, so if your cycle length is set to many minutes you could lose a lot of data if the DAQ crashes during a run.

MCA4 settings

You can specify whether data is read out from the MCA4, and if so, whether to read out data just from STOP1 or also from STOP2. The histogram bin width (dwell time) is the same for both channels, but you may specify a different number of bins for each channel if desired.

Faraday cup readout

You can choose whether to read out the current measured by the TRFCBL:FC0 and ILE2T:FC3 Faraday cups. The values are currently read at the end of each supercycle.

Faraday cup in/out toggling

Three modes are available for controlling the TRFCBL:FC0 Faraday cup:

• Automatically take data with the cup out and then the cup in, before moving on to the next step in the scan.
• Only take data with the cup in one position, and force the cup to that position at the start of the run.
• Only take data with the cup in one position, leaving the cup in its existing position at the start of the run.

ZMQ announcement of the cycle state

You can request that a ZMQ socket is opened, which will broadcast the start/end of each run and the start/end of each PPG cycle. For EBIT, a PUB (publisher) socket will be opened, which you should connect to with a SUB (subscriber) socket. Remember that SUB sockets MUST also subscribe to a topic after connecting (even if the messages being published don't specify a topic).

 import zmq
   
 context = zmq.Context()
 socket = context.socket(zmq.SUB)
 socket.connect("tcp://somewhere.triumf.ca:1234")
 socket.subscribe("")

All the messages are sent as serialized JSON. The exact format of the "new cycle" messages depends on what is being scanned/toggled during a run.

All messages include the midas run number, and the message time as a UNIX timestamp in MILLISECONDS.

Example messages

Begin of run

{
 "command": "begin_of_run",
 "run_number": 147,
 "scan_config": {
   "EPICS": [
     {
       "demand_dev": "A:B:C",
       "end": 600.0,
       "human_name": "Example",
       "measured_dev": "",
       "n_points": 6,
       "scan_dimension": "X",
       "start": 100.0,
       "use_formula": 0,
       "values": [
         100.0,
         200.0,
         300.0,
         400.0,
         500.0,
         600.0
       ]
     }
   ],
   "FC0InOut": [
     "flip_when": "every super-cycle"
   ],
   "PPG": [],
   "nX": 6,
   "nY": 0,
   "nZ": 0
 },
 "timestamp": 1701477018928
}

Start of cycle, EPICS value being scanned and Faraday cup being toggled in/out automatically. Note that multiple EPICS devices can be scanned, so that entry is an array. We include both the value that the user requested (100.0 in this example) and the actual physically measured value (99.97 in this example). There is only one toggle for whether the FC0 cup is in/out of the beamline, so that entry is not an array.

{
 "command": "new_cycle",
 "cycle_number": 0,
 "run_number": 147,
 "scan_settings": {
   "EPICS": [
     {
       "demand_dev": "A:B:C",
       "demand_val": 100.0,
       "human_name": "Example",
       "measured_dev": "",
       "measured_val": 0.0
     }
   ],
   "FC0InOut": {
     "as_boolean": false,
     "as_string": "false"
   },
   "PPG": []
 },
 "timestamp": 1701477018935
}

Start of cycle, 2D scan of EPICS and PPG, but no toggling of Faraday Cup. Again the PPG entry is an array.

{
 "command": "new_cycle",
 "cycle_number": 0,
 "run_number": 152,
 "scan_settings": {
   "EPICS": [
     {
       "demand_dev": "A:B:C",
       "demand_val": 100.0,
       "human_name": "Example",
       "measured_dev": "",
       "measured_val": 0.0
     }
   ],
   "FC0InOut": null,
   "PPG": [
     {
       "odb_path": "/Equipment/PPGCompiler/Programming/Pulse_beam/time offset (ms)",
       "value": 0.0
     }
   ]
 },
 "timestamp": 1701478228205
}

End of cycle

 {'command': 'end_of_cycle', 'cycle_number': 12, 'run_number': 113, 'timestamp': 1624392509388}

End of run

 {'command': 'end_of_run', 'run_number': 113, 'timestamp': 1624392523080}

Full run

 {'command': 'begin_of_run', 'run_number': 113, 'scan_config': {.....}, 'timestamp': 1624392506190}
 {'command': 'new_cycle', 'cycle_number': 0, 'run_number': 113, 'scan_settings': {.....}, 'timestamp': 1624392506385}
 {'command': 'end_of_cycle', 'cycle_number': 0, 'run_number': 113, 'timestamp': 1624392509388}
 {'command': 'new_cycle', 'cycle_number': 1, 'run_number': 113, 'scan_settings': {.....}, 'timestamp': 1624392509399}
 {'command': 'end_of_cycle', 'cycle_number': 1, 'run_number': 113, 'timestamp': 1624392512402}
 {'command': 'new_cycle', 'cycle_number': 2, 'run_number': 113, 'scan_settings': {.....}, 'timestamp': 1624392512413}
 {'command': 'end_of_cycle', 'cycle_number': 2, 'run_number': 113, 'timestamp': 1624392515416}
 {'command': 'new_cycle', 'cycle_number': 3, 'run_number': 113, 'scan_settings': {.....}, 'timestamp': 1624392515427}
 {'command': 'end_of_cycle', 'cycle_number': 3, 'run_number': 113, 'timestamp': 1624392518045}
 {'command': 'end_of_run', 'run_number': 113, 'timestamp': 1624392523080}

Data format

In the final data file, each event contains data from a single scan configuration (i.e. X step, Y step, FC0 in/out). The data from all cycles within a supercycle are combined into a single event. The histograms contain only the data from that scan configuration. Each event also contains data explaining the scan configuration that applied to that event.

If the run continued for many loops, there may be several events with the same scan configuration. E.g.

 Event #1: Loop 0, X step 0, cup out
 Event #2: Loop 0, X step 0, cup in
 Event #3: Loop 0, X step 1, cup out
 Event #4: Loop 0, X step 1, cup in
 Event #5: Loop 1, X step 0, cup out
 Event #6: Loop 1, X step 0, cup in
 Event #7: Loop 1, X step 1, cup out
 Event #8: Loop 1, X step 1, cup in

Offline analysis would need to sum the histograms in events #1 and #5 to get all the data for the "X step 0, cup out" configuration.

Python tools for reading the data in a user-friendly manner can be found in the titan_data repository.

If you are logged-on to titan05 as the ebit user, you can run the ebitdump command to dump data to screen as it is being taken (it's only written at the end of each supercycle). Example output from this tool is:

 Event # 6, at 2020-08-26 16:27:30
   Loop count 0, X point 3, Y point -1, FC0 Out
   PPG scanning:
     /Equipment/PPGCompiler/Programming/del_cycle_end/time offset (ms) => 8.0
   Faraday cup readings:
     ILE2T:FC3:SCALECUR => -9.765624723809263e-13
     TRFCBL:FC0:SCALECUR => 7.019043007407133e-13
   MCA4 channel 1 has 8192 bins. Max bin is 1203
   MCA4 channel 2 has 0 bins. Max bin is None
 Event # 7, at 2020-08-26 16:27:36
   Loop count 0, X point 3, Y point -1, FC0 In
   PPG scanning:
     /Equipment/PPGCompiler/Programming/del_cycle_end/time offset (ms) => 8.0
   Faraday cup readings:
     ILE2T:FC3:SCALECUR => 0.0
     TRFCBL:FC0:SCALECUR => 1.129150375631105e-12
   MCA4 channel 1 has 8192 bins. Max bin is 1205
   MCA4 channel 2 has 0 bins. Max bin is None
 Event # 8, at 2020-08-26 16:27:40
   Loop count 0, X point 4, Y point -1, FC0 Out
   PPG scanning:
     /Equipment/PPGCompiler/Programming/del_cycle_end/time offset (ms) => 9.0
   Faraday cup readings:
     ILE2T:FC3:SCALECUR => 9.765624723809263e-13
     TRFCBL:FC0:SCALECUR => 8.239746233408562e-13
   MCA4 channel 1 has 8192 bins. Max bin is 1341
   MCA4 channel 2 has 0 bins. Max bin is None

See TITAN_DAQ_General_Documentation#CPET.2FMPET.2FEBIT_data_format for the gory details of the file format.

Software

The DAQ system is based on the MIDAS DAQ package. Software written for EBIT is linked with the MIDAS libraries and uses common framework for MPET/CPET/EBIT.

The EBIT software can be found at /home/ebit/packages/ebit on titan05 (note that historical versions of the software were located in /home/ebit/online).

Run control is done using the MIDAS webserver mhttpd with Custom Pages written for ebit users so that they can easily enter the parameters required to program the PPG and to run the DAQ.

Computers

• titan05 is a "normal" PC that runs the main midas programs (mhttpd web server etc).
• lxebit is a PC that lives in the EBIT VME crate. It has direct access to the PPG (which runs a programmable sequence of pulses to other hardware) and any other hardware in the VME crate. lxebit uses an NFS mount and NIS accounts, so sees the same /home/ebit as titan05. This machine runs the "feebit_ppg" program.
• titan-pi04 is a raspberry pi that talks to the Fastcom MCA4. It can only be accessed from within TRIUMF (either on-site or via the TRIUMF VPN). It does not use NFS/NIS, and has its own /home/ebit. This machine runs the "feebit_mca4" program.

Starting and stopping DAQ programs

Programs can be started/stopped from the Midas programs page. All programs that appear on that page should be running (green status).

To start/stop everything (or if the programs page is inaccessible), you can use the following aliases after logging in as the "ebit" user on titan05:

 kill-all (an alias of ~/packages/ebit/bin/kill-all.sh that stops all the DAQ programs)
 start-all (an alias of ~/packages/ebit/bin/start-all.sh that starts all the DAQ programs)

Program listing

In the order that programs appear on the Programs page:

• mhttpd - midas web server
• Logger - midas data logger
• feebit_ppg - main EBIT frontend that talks to the PPG, scans EPICS values etc. Runs on lxebit.
• PPGCompilerFrontend - converts a user-supplied PPG program into bytecode that can be loaded onto the PPG
• saveload - helps save and load scan settings
• mserver - midas RPC server that allows the programs on lxebit and titan-pi04 to talk to the main experiment on titan05
• feebit_mca4 - frontend that interacts with the MCA4 and sends the histogram data to feebit_ppg at the end of each supercycle
• ebitana - python analyzer that creates plots based on live data (plots are viewed via the Web plots page)

Pulling the latest version of the EBIT DAQ software

 # On titan05 (which shares a /home/ebit with lxebit)
 cd ~/packages/ebit
 git pull
 git submodule update

 # On lxebit (not titan05 - we can't cross-compile from titan05 as it doesn't have compatible 32-bit ZMQ libs)
 cd ~/packages/ebit/build
 make install

 # On titan-pi04
 cd ~/packages/ebit
 git pull
 git submodule update
 cd build
 make install

 # On titan05 - restart the DAQ
 kill-all
 start-all

cmake configurations

Different cmake configurations are used on titan05 and titan-pi04 when building the software. If cmake needs to be run again, use:

 # On titan05 - default build creates 32-bit feebit_ppg for lxebit, plus 64-bit analyzer for titan05
 cd ~/packages/ebit/build
 cmake ..

 # On titan-pi04 - build 64-bit feebit_mca4
 cd ~/packages/ebit/build
 cmake -DBUILD_MCA4_FE_ONLY=1 ..

MCA4 frontend design rationale

Fastcom provided USB code that only works on some versions of linux. In particular, it is not possible to reliably talk to the MCA4 from lxebit. Therefore, we use a raspberry pi running Ubuntu 20.04 for the MCA4 communications, and run the rest of the DAQ from lxebit as normal. Ubuntu 20.04 provides very stable interaction with the MCA4.

The MCA4 takes several seconds to handle a "start acquisition" command. If this was not the case, we would stop and restart the MCA4 at the end of each supercycle, so that "fresh" histograms are created for each scan step. However, such high latency is unacceptable, so we leave the MCA4 running all the time, and do calculations in the frontend that subtracts the state of the histogram at the end of the previous supercycle from the current state.

We only write data to disk at the end of each supercycle, so normally users would be left waiting for a long time to see what the data looks like. We therefore have 2 ways to send histogram data from feebit_mca4 to feebit_ppg. At the end of each supercycle, we send data with an event ID of 1 into the BUFMCA4 buffer. Within each cycle (every few hundred ms) we send data with an event ID of 0 into the BUFMCA4 buffer. Only the data from the former events are written to disk, but the latter events can be read by the rootana-based analyzers to show a more responsive visualization of the data. This works very well as the data read from the MCS4 is cumulative, and is only reset at the end of each supercycle.