Daq

Links

• https://alphacpc05.cern.ch/agdaq/ -- midas daq page for the agdaq system at CERN
• https://daq16.triumf.ca -- midas daq page for the agmini system at TRIUMF
• https://daq.triumf.ca/elog-alphag/alphag/ -- elog
• https://daqstore.triumf.ca/AgWiki/index.php/Main_Page -- this wiki
• https://bitbucket.org/expalpha/agdaq - git repository for agdaq
• https://bitbucket.org/expalpha/agana - git repository for agana
• https://bitbucket.org/expalpha -- git repository on bitbucket
• https://edev-group.triumf.ca/fw/exp/alphag -- git repository on edev gitlab

Inventory database

• https://ladd00.triumf.ca/daqinv/frontend/list/170 - PWB inventory database
• https://ladd00.triumf.ca/daqinv/frontend/list/168 - ADC (GRIF-16) inventory, FMC-ADC32rev0,rev1 inventory
• https://ladd00.triumf.ca/daqinv/frontend/list/176 - FMC-ADC32rev1.1 inventory
• https://ladd00.triumf.ca/daqinv/frontend/list/174 - BSC RTM inventory
• https://ladd00.triumf.ca/daqinv/frontend/list/173 - BSC ASD inventory
• https://ladd00.triumf.ca/daqinv/frontend/list/171 - TMB inventory

Hardware manuals

• TRG - trigger board (GRIF-C) manual
• CDM - https://daq.triumf.ca/DaqWiki/index.php/VME-CDM -- clock VME-CDM manual
• ADC - https://daq.triumf.ca/DaqWiki/index.php/VME-GRIF-ADC16-Rev1 -- GRIF-16 ADC manual
• PWB - TPC Pad Wing Board manual
• TDC - https://daq.triumf.ca/DaqWiki/index.php/GSI_TRB3
• chronobox - chronobox manual
• ASD - BSC preamp board
• RTM - BSC VME rear transition board

Performance

Full system: (ADC fw rel-20201112-ko, PWB fw 20201005_ko)

• 1 Hz stable, ADC adc32 9 supp off rate 0.3 M/s, adc16 off, PWB udp_delay 25 supp off active 64 rate 6.6+4+6+4.6=21.2 M/s, TDC off, total rate 21 M/s
• 500 Hz stable, ADC adc32 9 supp 1500 rate 6 M/s, adc16 off, PWB udp_delay 10 supp 100 active 63 rate 3+4+3+3=13 M/s, TDC off, total rate 19 M/s

Single ADC: (fw rel-20201112-ko)

Fully suppressed data, sthreshold 5000

• 1 Hz: stable
• 500 Hz: stable (nominal max trigger rate)
• 6700 Hz: stable, 8 Mbytes/sec
• 6800 Hz: evb eventually confused
• 6850 Hz: evb confused, no lost triggers in the ADC, no crash in the ADC

All data, 510 samples/event, sthreshold 10 (fw rel-20201122-ko)

• 1 Hz: stable
• 100 Hz: stable, 3.4 Mbytes/sec
• 500 Hz: stable, 17 Mbytes/sec (nominal max trigger rate)
• 1000 Hz: stable, 32 Mbytes/sec
• 3100 Hz: stable, 106 M/s (approaching saturation of 1gige ethernet)
• 3200 Hz: stable, 110 M/s - TRG communication is in trouble
• 3300 Hz: stable, 116 M/s
• 3400 Hz: incomplete events, no ADC crash ("dropped due to full" counting, "dropped due to busy" are zero, this is as expected).

no suppression (different fpga data path)

• 1 Hz: stable
• 100 Hz: stable
• 500 Hz: stable, 17 M/s (nominal max trigger rate)
• 1000 Hz: stable, 32 M/s
• 3000 Hz: stable, 100 M/s (approaching saturation of 1gige ethernet)
• 3300 Hz: incomplete events, no ADC crash ("dropped due to full" counting, "dropped due to busy" are zero, this is as expected).

Single PWB, fw 20201005_ko, see https://alphacpc05.cern.ch/elog/Detectors/5563

Fully suppressed data:

• up to 500 Hz, stable

All pads firing (FW pulser)

• 4 SCA: up to 272 Hz, 81.6 Mbytes/sec, at higher rate: event fifo overflow
• 2 SCA: up to 295 Hz, 44.25 Mbytes/sec, at higher rate: event fifo overflow
• 1 SCA: up to 295 Hz, 22.5 Mbytes/sec, at higher rate: event fifo overflow

Conclusion:

• max PWB date rate 80 Mbytes/sec,
• max non-suppressed PWB trigger rate: 295 Hz
• max suppressed PWB trigger rate: 500 Hz (1.6 ms sca readout time)

Network connections

The experiment private network number is: 192.168.1.x, gateway 192.168.1.1, netmask 255.255.255.0

network map:

---- TRIUMF/CERN network --1gige-- alphagdaq --10gige-- juniper switch --1gige-- centrecom switch

alphagdaq network connections

on the rear of the machine:

-----------------------------------------
|   | rj45 | rj45 |         | sfp | sfp |
-----------------------------------------

- left rj45 - copper 1gige - eno1 - spare (inactive)
- right rj45 - copper 1gige - eno2 - dhcp - cern/triumf network
- left sfp - 10gige - enp1s0f1 - spare (inactive)
- right sfp - 10gige - enp1s0f0 - static 192.168.1.1

connections between switches

• alphagdaq to juniper switch: direct attach (DAC) 10gige cable into any port of juniper switch
• juniper switch to centrecom switch: rj45 sfp in any port of juniper switch - cat5e/cat6 cable - to any port of centrecom switch
• cat5e/cat6 cable from any port of centrecom switch to the XXX port on the back of the juniper switch

juniper switch

connected to juniper switch front is:

• everything that sends event data to alphagdaq, specifically:
  • TRG (rj45 sfp)
  • 16x ADC (fiber sfp)
  • 64x PWB (fiber sfp)
• 10gige uplink to alphagdaq (10gige DAC cable)
• 1gige link to centrecom switch (rj45 sfp)
• 40gige ports are not used

connected to the juniper switch rear:

• CON is the serial console via RJ45-to-DB9 to USB-serial adapter to alphagdaq USB port (9600bps, 8N1, no flow control)
• C0 is the management ethernet interface port, connect to any port on the centrecom switch (juniper-private will ping)

centrecom switch

• everything with copper rj45 connections, specifically:
  • CDM boards
  • HV, LV and VME power supplies
  • RaspberryPi3 boards
  • gas handling MFCs (algas)
  • cooling system controller (moxa01)

USB connections

Note:

• alphagdaq has USB2 and USB3 connections. They are not the same, they are not generally interchangeable.
• USB3 connections and cables are generally blue coloured.

List of USB connections from alphagdaq:

• front USB3 port (blue): USB-A-to-B cable to USB blaster connected to the trigger board (front of rack)
• rear USB3 port (blue): USB3 short jumper to USB3 hub
• rear USB2 port (black): USB2 unpowered hub (Logiix) to USB-A-to-B cable to UPS USB connector
• rear USB2 port (black): USB-to-Serial adaper to DB9 to RJ45 adapter to Juniper switch console port

Note: UPS USB connection does not work with alphagdaq USB ports (both usb2 and usb3), but combination using the unpowered USB2 hub seems to work ok.

List of USB connections from the USB hub:

• this is a powered USB3 hub
• USB-A-to-MicroUSB - to lvdb boards
• USB-A-to-Wiener interlock cable

Clock and trigger distribution

Explanation

Trigger is generated by the trigger board (aka TRG, aka GRIF-C), from the TRG eSATA output through the eSATA splitter it is fed into the master CDM. The master CDM outputs the trigger on all MiniSAS output ports. One of the master CDM MiniSAS outputs is connected to the CDM slave eSATA input. The same as the master CDM, the slave CDM outputs the trigger on all MiniSAS output ports.

The master 62.5MHz clock is generated by the master CDM using the internal 10MHz oscillator or the external 10MHz AD clock. Also on MiniSAS port 6 the master CDM generates the 200MHz clock for the TDC. This clock is frequency-locked to the master 62.5MHz clock.

Master CDM clock selection (external CD 10MHz clock or internal oscillator 10MHz clock) is done by a magic esper command (see below).

From the master CDM, the clock is fed to: the slave CDM, the TRG (via the eSATA splitter) and to the TDC (via the RJ45 splitter).

The slave CDM sends the clock and the trigger to the ADCs and PWBs.

Chronobox connections

Chronoboxes are synchronized with the daq using two signals: 10 MHz timestamp clock and TTL sync.

The sync signal used is the trigger signal - timestamps are reset by the 1st event - which is not a physics event, but the first trigger in the trigger sync sequence.

All chronoboxes are configured in "slave" and "chain" mode.

Slave mode: 10 MHz timestamp clock goes into the CLK_IN input (configured as NIM on the first chronobox and TTL on all others), sync signal goes into TTL input LEMO 4. (bank B direction set to "in").

Chain mode: 10 MHz timestamp clock is TTL output LEMO 0, sync is TTL output LEMO 1. (bank A direction set to "out"). These signals feed into CLK_IN (TTL mode) and TTL input LEMO 4 in the next chronobox in the chain.

The first chronobox receives the 10 MHz timestamp clock and sync signals from the master CDM per diagrams below.

• CDM 10 MHz clock output is NIM, feeds directly into the chronobox CLK_IN (NIM mode).
• CDM "trigger" ("sync") output is NIM, has to go through a NIM-to-TTL converter before feeding chronobox TTL input LEMO 4. I use a LeCroy 222 "gate and delay generator" module. NIM input goes into the "Start" input, TTL output is taken from the "TTL" output. Signal width is adjusted to around 100-1000 ns. Any other sutable NIM-to-TTL converter can also be used.
• "sync" output must be enabled in the master CDM:

esper-tool cdm00
cd cdm
write multi_sync 2

Schematic

TRG (GRIF-C)
------------
eSATA <------> eSATA splitter

eSATA splitter
--------------
trigger ---> eSATA ---> CDM-Master eSATA (trigger signal)
clock <--- (eSATA --- MiniSAS) <--- CDM MiniSAS (62.5MHz clock)

CDM-Master
----------
LEMO1B <----- 10 MHz AD clock (NIM)
LEMO2A --> (only if there is no external clock) 4ns lemo cable --> LEMO1B (62.5 MHz frequency monitor)
LEMO2B --> chronobox sync (NIM) --> NIM-to-TTL --> long lemo cable to chronobox sync input (TTL)
LEMO3B --> long lemo cable to chronobox clock input (NIM) (CLK_IN configured for NIM input)
eSATA <--- eSATA splitter <--- eSATA from TRG (trigger signal)
MiniSAS 1 --> (MiniSAS --- eSATA) --> CDM-Slave eSATA (62.5 MHz)
MiniSAS 6 --> (MiniSAS --- eSATA) --> eSATA splitter --> RJ45 --> TDC (200 MHz)

CDM-Slave
---------
LEMO2A --> 4ns lemo cable --> LEMO1B (62.5 MHz frequency monitor)
eSATA <--- trigger and 62.5MHz clock from CDM-Master
MiniSAS 1 --> ADC trigger and 62.5MHz clock
MiniSAS 2 --> same
MiniSAS 3 --> same
MiniSAS 4 --> same
MiniSAS 5 --> PWB trigger and 62.5MHz clock
MiniSAS 6 --> same

Layout of the VME crate at CERN. The ALPHA-T module is in the rightmost slot. The Master CDM is next to it and it's labelled as #02. The Slave CDM is on the lefmost position and labelled by #03.

Setup of CDM boards

ALPHA-g uses the Rev3 CDM boards. The 4 CDM boards that belong to ALPHA-g have serial numbers 0, 1, 2 and 3 and are labeled cdm00, cdm01, cdm02, cdm03.

All the boards can be used as master and slave positions, but have to be configured appropriately.

Slave setup

• install the board
• connect ethernet
• connect esata external clock
• esper-tool cdmNN ### i.e. esper-tool cdm01
• cd template
• write current_setup 4
• cd /cdm
• read
• observe clock frequency counters report correct values. "ext_clk" will only count if the lemo jumper is installed (see below)

11    esata_clk        uint32           R                 [62500246]                      
12    ext_clk          uint32           R                 [62500246]                      

• cd /lmk
• read
• observe both PLL1 and PLL2 are locked: both values should be "1"

144   pll1_ld          uint8            R                 [1]                             
147   pll2_ld          uint8            R                 [1]                             
156   ld1_counter      uint32           RW                [3]                             
157   ld2_counter      uint32           RW                [3]                             

If needed, check correct operation of the pll lock monitoring:

• unplug the esata cable from the master CDM
• "red" light should go off
• read of lmk should report "pll1_ld" value 0 and "ld1_counter" and "ld2_counter" should increment
• reconnect the esata cable
• "red" light should return
• read of lmk should report both pll1_ld and pll2_ld locked (values "1").

Master setup

• install the board
• connect ethernet
• esper-tool cdmNN ### i.e. esper-tool cdm01
• cd template
• write current_setup 3
• cd /lmk
• write clkin_sel_mode 0
• cd /cdm
• read
• observe clock frequency counters report correct values. "ext_clk" will only count if the "clock loopback" lemo jumper is installed (see below)

12    ext_clk          uint32           R                 [62500189]                      

• cd /lmk
• read
• observe both PLL1 and PLL2 are locked: both values should be "1"

144   pll1_ld          uint8            R                 [1]                             
147   pll2_ld          uint8            R                 [1]                             
156   ld1_counter      uint32           RW                [3]                             
157   ld2_counter      uint32           RW                [3]                             

Check the 200MHz clock:

• connect MiniSAS-eSATA splitter cable to MiniSAS output 6, connect eSATA end to CDM eSATA input
• in esper-tool, read cdm: esata_clk should report 200MHz clock

11    esata_clk        uint32           R                 [200000603]                     
12    ext_clk          uint32           R                 [62500188]                      

Master setup with external 10 MHz clock

• setup master CDM as above (using internal clock)
• connect 10MHz external clock to LEMO1A
• esper-tool cdmNN
• cd /cdm
• write sel_nim 1 # if clock is NIM signal
• write sel_nim 0 # if clock is TTL signal (i.e. chronobox LEMO output)
• read ext_clk

[http://cdm00:/cdm]> read ext_clk
9999556

• if ext_clk reads 0 or does not read something close to 10MHz, stop here. debug the signal, try changing sel_nim, etc.
• cd /lmk
• write clkin_sel_mode 2 # select external clock
• read
• observe both PLL1 and PLL2 are locked: both values should be "1"

150   clkin2_sel       bool             R                 [True]                          
144   pll1_ld          uint8            R                 [1]                             
147   pll2_ld          uint8            R                 [1]                             

• on the MIDAS TRG page, observe that TRG measured external clock frequency is close to 62.5 MHz

clkin_sel_mode positions

• 0 = 10MHz internal oscillator (use on master CDM in standalone mode)
• 1 = eSATA clock (use on slave CDM)
• 2 = external clock, should be 10MHz (use on master CDM to run from AD clock)

LEMO connections

LEMO connectors:

|
|LEMO1A|LEMO1B
|LEMO2A|LEMO2B
|LEMO3A|LEMO3B
|eSATA
|RJ45 ETH
|minisas
|

• clock loopback jumper: to read the frequency of the generated 62.5MHz clock, connect LEMO cable from LEMO2B to LEMO1A
• 10MHz internal clock jumper: to feed 10MHz internal oscillator to the external clock, connect LEMO cable from LEMO3A to the LEMO1A
• 10MHz external clock: to feed 10MHz external clock, connect LEMO cable to LEMO1A

MIDAS frontends

UDP

feudp receives udp packets from the ADC and PWB boards. For each udp packet 1 midas event is created, with 1 data bank containing the udp data. Events are sent into the MIDAS event buffer BUFUDP. The names of the data banks are assigned in ODB /eq/feudp/settings.

{ADC,PWB} --> 1gige --> switch --> 10gige --> alphagdaq --> feudp -> BUFUDP

CTRL

fectrl controls all the ADC, PWB and trigger boards. It provides run start and stop sequencing, runs the event synchronization sequence at the begin of the run, and records slow controls data (temperatures, voltages, etc).

fectrl receives UDP data packets from the trigger board and sends them to the MIDAS event buffer BUFUDP.

fectrl receives commands from the control web pages (for ADC, PWB and trigger) to reboot, initialize, etc the boards.

fectrl configures the event builder via odb /eq/fectrl/evbconfig.

ADC <-> http esper <-> fectrl -> slow control data into midas history

PWB <-> http esper <-> fectrl -> slow control data into midas history

TRG <-> udp comm <-> fectrl -> BUFUDP, slow control and counters into midas history

fectrl <-> midas rpc <-> mhttpd <-> json rpc <-> control web pages for ADC, PWB and trigger

EVB

feevb receives udp packet data from the MIDAS event buffer BUFUDP, inspects the packet timestamps and collects the data with matching timestamps into physics events. feevb has provisions to do data suppression, reduction and compression in addition to the data reduction done in the ADC and PWB firmware.

feevb event synchronization relies on the event synchronization sequence done by fectrl when a run is started.

{ADC, PWB, TRG} -> BUFUDP -> feevb -> SYSTEM -> mlogger -> compression -> disk storage

internal structure

• event buffer readers, instances for BUFTRG, BUFTDC, BUFADC, BUFUDP

BufferReader class - reads one midas event buffer:
- BufferReaderThread()
-- in batches of 1000 events
--- lock
--- TickLocked()
----- ReadEvent() -> bm_receive_event_alloc()
----- fHandler->HandleEvent()
-- fHandler->XMaybeFlushBank()
-- sleep 100 usec

• event header decoder and intermediate buffer

Handler class - accumulates events, sends them to the EVB in batches

- HandleEvent() <- BufferReaderThread() - receive incoming events
-- loop over all banks, examine bank name
--- call AddAdcBank(), AddPwbBank(), AddTrgBank(), AddTdcBank() according to bank name
----  AddXxxBank() -> XAddBank()
--- unknown banks go directly to evb->SendQueue
-- check if synchronization has completed

- XAddBank() - save BankBuf objects in a buffer

- XMaybeFlushBank(), XFlushBank() - send all accumulated data to the Evb class
-- evb->lock()
-- for each buffered BankBuf objects,
--- call evb->AddBankLocked()
-- buffer is now empty

• event builder

Evb class - assemble event fragments according to timestamps

- fEvents fifo (deque) - time sorted list of partially assembled events
-- new entries are added by FindEvent()
-- entries are modified by MergeSlot()
-- old entries are removed by GetLocked()

- AddBankLocked() <- XFlushBank() <- BufferReaderThread()
-- call fSync->Add()
-- transform BankBuf to EvbEventBuf
-- push it into per-module buffer fBuf[imodule]

- EvbThread()
-- EvbTickLocked()
--- BuildLocked()
---- in a loop, for no more than 1 sec
----- loop over all per-module buffers fBuf
------ pop first entry, pass it to BuildSlot()
------- call FindEvent()
------- call MergeSlot()
------- call CheckEvent()
--- in a loop
---- get all completed events via GetLocked() pass them to SendQueue->PushEvent()
-- if done nothing, sleep 100 usec, otherwise nominal sleep to free the evb lock

- GetNext() <- GetLocked() <- EvbThread() - get completed events
-- examine oldest EvbEvent entry in fEvents:
-- pop it if it is complete
-- pop it if it is too old (to prevent old incomplete events from stalling the evb) ("pop age" on the evb page)
-- pop it if a following event is complete ("pop following" on the evb page)

• send queue

SendQueue class - prepare events for writing to midas SYSTEM event buffer

- PushEvent() <- EvbThread()
-- push FragmentBuf into the queue

- SendQueueThread()
-- in a batch of 1000 events, call Tick()
--- pop FragmentBuf from the queue, call SendEvent()
--- compose midas event (memcpy()!)
--- TMFE fEq->SendEvent()
--- bm_send_event()
-- sleep 100 usec

• equipment object

EvbEq class - implements the midas equipment
- HandleBeginRun()
- HandleEndRun()
- HandlePeriodic()

multithreading

• BufferReaderThread() one per midas event buffer
  • high CPU use: bm_receive_event(), AddXxxBank()
  • lock contention: midas event buffer lock, evb lock into XFlushBank->AddBankLocked()
• EvbThread()
  • O(size of fEvents): FindEvent(), GetNext()
  • lock contention: evb lock against XFlushBank(), SendQueue lock (1 per output event)
• SendQueueThread()
  • high CPU use: memcpy() in SendEvent(), memcpy() in bm_send_event
  • lock contention: event queue lock against EvbThread, midas event buffer lock
• main thread
  • lock contention: evb lock in ReportEvb() and elsewhere

ODB entries

/Equipment/CTRL/Settings

• TBW

/Equipment/CTRL/Settings/Pulser

• ClockFreqHz - unit of 1/sec (Hz), set to 62500000, the TRG 62.5 MHz clock frequency (16 ns)
• PulseWidthClk - unit of 16 ns clocks, set to 25. (400 ns). width of internal pulser signal (used to go to a TRG external output, absent on GRIF-C boards)
• PulsePeriodClk - unit of 16 ns clocks, pulser in clock periods (+1), set to zero.
• PulseFreqHz - unit of 1/sec (Hz), requested pulser frequency, actual frequency is rounded to 16 ns clock granularity, change this value to select desired clock frequency
• BurstCtrl - enable burst of pulses 0xNNTTTTTT, NN=number of pulses, TTTTTT=period between pulses
• OutputEnable - enable TRG pulser external output (absent on GRIF-C boards)
• Enable - allow the pulser to run (bit 3 in TRG conf_trig_enable register)

/Equipment/CTRL/Settings/TrigSrc - trigger source selection

• TrigPulser - trigger on the pulser
• TrigEsataNimGrandOr - trigger on external NIM inputs of GRIF-16 ADC modules
• TrigAdc16GrandOr - trigger on any adc16 signals (see adc16_masks)
• TrigAdc32GrandOr - trigger on any adc32 signals (see adc32_masks)
• TrigAdcGrandOr - trigger on adc16_grand_or or adc32_grand_or
• (removed from trg firmware) Trig1ormore - trigger on adc16 multiplicity 1 or more
• (removed from trg firmware) Trig2ormore - ditto, 2 or more
• (removed from trg firmware) Trig3ormore - ditto, 3 or more
• (removed from trg firmware) Trig4ormore - ditto, 4 or more
• (removed from trg firmware) TrigAwCoincA - trigger on TPC AW coincidence (see TRG manual)
• (removed from trg firmware) TrigAwCoincB - ditto
• (removed from trg firmware) TrigAwCoincC - ditto
• (removed from trg firmware) TrigAwCoincD - ditto
• (removed from trg firmware) TrigAwCoinc - ditto
• (removed from trg firmware) TrigAw1ormore - trigger on TPC AW per-preamp multiplicity 1 or more (see TRG manual)
• (removed from trg firmware) TrigAw2ormore - ditto, 2 or more
• (removed from trg firmware) TrigAw3ormore - ditto, 3 or more
• (removed from trg firmware) TrigAw4ormore - ditto, 4 or more
• TrigAwMLU - trigger on TPC AW MLU signal (see TRG manual)
• TrigBscGrandOr - trigger on any BSC signal
• TrigBscMult - trigger on predefined BSC multiplicity (see TRG manual)
• TrigCoinc - trigger on predefined coincidence of TPC AW, BSC and external signal (see TRG manual)

/Equipment/CTRL/Settings/TRG - trigger settings

• Enable - if set to "n", fectrl will not do anything with the trigger board. Normal value "y"
• Modules - hostnames of the trigger board. Normal value "alphat01"
• NimMask - 32-bit mask for the 32x NIM inputs (see TRG manual). Normal value 0, unless external trigger is in use.
• EsataMask - 32-bit mask for the 32x eSATA signals (see TRG manual). Normal value 0.
• adc16_mask[16] - 16x 16-bit mask for the 100MHz GRIF-16 onboard ADCs (see TRG manual). Normal value 0xFFFF for all 16 sas links.
• adc32_mask[16] - 16x 32-bit mask for the 62.5MHz FMC-ADC32 ADCs (see TRG manual). Normal value 0xFFFFFFFF for all 16 sas links.
• PassThrough - if set to "y", trigger is passed through the trigger module without generating any events and without causing deadtime.
• AwCoincA, B, C and D - 32-bit masks for the aw16 coincidence trigger (see TRG manual)
• Aw16FromAdc16 - route AW trigger (aw16) from adc16 (100MHz ADCs 0..15)
• Aw16FromAdc32a - route AW trigger (aw16) from adc32 group 15..0 (62.5MHz FMC-ADC32 minisas links 0..7)
• Aw16FromAdc32b - route AW trigger (aw16) from adc32 group 31..16 (62.5MHz FMC-ADC32 minisas links 8..15)
• MluDir, MluFiles - location of aw16 MLU bit patterns
• MluSelectedFile - currently selected/loaded MLU file. 0="grand or", 1="2 or more hit clusters", 2="2 or more hit clusters, with gap 2 or more", 3="3 or more hit clusters"
• MluPrompt - length of aw16_prompt_run gate in units of 8 ns clock, see TRG#aw16_mlu
• MluWait - length of aw_prompt_wait gate in units of 8 ns clock, see TRG#aw16_mlu
• MluTrigDelayClk - "TrigDelay" when triggered of aw16 MLU, in units of 16 ns clock
• (BSC settings, TBW)
• BscFromAdc16a - (at CERN, set to "y") route BSC trigger data (bsc64_bus) from 100 MHz adc16 digitizers minisas links 0..7
• BscFromAdc16b - route BSC trigger data (bsc64_bus) from 100 MHz adc16 digitizers minisas links 8..15
• BscBotOnly - route BSC trigger data from bottom ASD cards, bsc64_bus = bot
• BscTopOnly - route BSC trigger data from top ASD cards, bsc64_bus = top
• BscBotTopOr - (set to "y") use bsc64_bus = (top OR bottom)
• BscBotTopAnd - use bsc64_bus = (top AND bottom)
• BscMultiplicityMin - (set to 2) BSC multiplicity trigger - minimum multiplicity
• BscMultiplicityWindowClk8 - (set to 80) BSC multiplicity window, 8 ns clocks
• BscEmptyWindowClk8 - (set to 80) BSC empty window, 8 ns clocks, see TRG#bsc64_multiplicity_trigger
• BscGrandOrDelayClk16 - (not implemented) trigger delay used by fectrl when TrigSrc/TrigBscGrandOr is selected
• BscMultTrigDelayClk16 - trigger delay used by fectrl when TrigSrc/TrigBscMult is selected
• CoincStart - set same as CoincRequire
• CoincRequire - 0x5: (BSC grand or)*(TPC aw16_grand_or), 0x9: (BSC multiplicity)*(TWC aw16_grand_or), 0x6: (BSC grand or)*(AW16 MLU), see TRG#Coincidence_trigger and TRG#conf_coinc_control bits
• CoincWindow - length of coincidence window in 8ns clocks.
• CoincTrigDelayClk16 - trigger delay used by fectrl when TrigSrc/TrigCoinc is selected

/Equipment/CTRL/Settings/PWB - PWB settings

• Enable - if set to "n", fectrl will not do anything with PWBs. Normal value "y".
• enable_boot_user_page - if set to "n", fectrl will not try to reboot PWBs into the user page firmware. Normal value "y".
• enable_trigger - if set to "n", all PWBs will be set to ignore the trigger. Normal value "y".
• enable_trigger_group_a - field wire pulser defeats the data suppression (every channel has a pulse) resulting in too much data overwhelming the DAQ (lost UDP packets). To permit correct data taking of field wire pulser data, PWBs are split into two groups (a and b), only one of the two groups should be enabled for field wire pulser runs. Both groups should be enabled for normal data taking. Normal value is "y"
• enable_trigger_group_b - see "enable_trigger_group_a". Normal value is "y".
• modules[64] - hostnames of PWB boards
• boot_user_page[64] - if set to "y" and enable_boot_user_page is "y", fectrl will reboot the corresponding PWB to the user page firmware.
• trigger[64] - if set to "y" and enable_trigget is "y", fectrl will set the PWB to accept the trigger.
• ch_enable - if set to "n" disables all PWB channels. Normal value is "y".
• ch_force - is set to "y" disables channel suppression (all PWB channels are read)
• suppress_reset - if set to "y" enables channel suppression for reset channels (threshold has to be set correctly)
• suppress_fpn - if set to "y" enables channel suppression for fpn channels (threshold has to be set correctly)
• suppress_pads - set to "y" enables channel suppression for TPC pad channels (threshold has to be set correctly)
• disable_reset1 - data suppression algorithm does not work for the channel "reset1" and it has to be suppressed explicitly by setting this value to "y". Normal value is "y".
• baseline_{reset,fpn,pads}[64] - waveform baseline for each PWB position
• threshold_{reset,fpn,pads} - waveform suppression threshold, see below:
• ch_threshold - PWB sca_N_ch_threshold, if set to 0, threshold is computed as "baseline_pads[seqpwb]-threshold_pads". Normal value is 0.
• test_mode - enable PWB test mode, the ADC data is replaced with a test pattern, see PWB esper variable signalproc/test_mode

/Equipment/CTRL/Settings/TDC - TDC settings

• Enable - if set to "n", fectrl will tell the event builder that the TDC is not running. Normal value is "y".

/Equipment/CTRL/Variables

• Scalers - TRG scalers and counters. from fectrl.cxx ReadTrgLocked()

         // 0..15 read the 16 base scalers at 0x100
         // 16..31 read the adc16 scalers at 0x430
         // 32..63 read the adc32 scalers at 0x440
         // 64..79 read the 16 additional scalers at 0x110
         // 80..95 read the 16 counters_adc_selected scalers at 0x460
         // 96..159 read the 64 bar scalers at 0x470
         // additional scalers should be appended at the end.

/Equipment/CTRL/EvbConfig - EVB configuration

All arrays have the same size corresponding to the number of EVB slots.

• name[] - slot name for the web page (trg01, adc01, adc01/adc32, pwb31/0, tdc01, etc)
• type[] - slot type: 1=TRG, 2=ADC16 and ADC32, 3=obsolete PWB, 4=obsolete PWB, 5=PWB, 6=TDC
• module[] - module number (adcNN, adcNN+100, pwbNN, etc)
• nbanks[] - number of expected banks (16 for ADC16, 32 for ADC32, not used, set to 1 for all others)
• tsfreq[] - timestamp frequency in Hz.

/Equipment/ADC_UDP - ADC UDP configuration

• Common/Buffer - "BUFADC" - event buffer to use, name is hardwired in feevb.cxx
• Settings/udp_port - 50006 - UDP port number, also used by fectrl to configure the ADC modules
• Settings/rcv_bufsize - 209715200 - udp socket buffer size set by setsockopt(SOL_SOCKET, SO_RCVBUF). must have a corresponding command in /etc/rc.local to increase maximum system socket size: "/usr/sbin/sysctl -w net.core.rmem_max=200000000"
• Settings/packet_size - 1500 - non-jumbo-frame UDP packet maximum size
• Settings/max_buffered_packets - 40000 - maximum size of internal packet fifo. If it overflows, extra packets are discarded.
• Settings/max_packets_per_event - 8000 - how many UDP packets to pack into one midas event==

/Equipment/PWB_X_UDP - PWB UDP configuration

For explanation see ADC_UDP above.

• Common/Buffer - "BUFUDP" - event buffer to use, name is hardwired in feevb.cxx
• Settings/udp_port - 50007, 50008, 50009, 50010 - for A, B, C and D UDP receivers respectively, also used by fectrl to configure the PWB modules
• Settings/rcv_bufsize - 209715200
• Settings/packet_size - 1500
• Settings/max_buffered_packets - 10000
• Settings/max_packets_per_event - 8000

Trigger configuration

These trigger modes are implemented:

• (manual trigger from web page)
• software pulser trigger (from fectrl)
• hardware pulser trigger (from the trigger board)
• NIM and eSATA trigger from ADC front panel inputs
• adc16 and adc32 discriminator triggers:
  • adc16 grand-or trigger
  • adc32 grand-or trigger
  • adc (adc16+adc32) grand-or trigger
  • adc16 multiplicity trigger (1ormore, 2ormore, 3ormore, 4ormore) for use with barrel scintillator
  • anode wire multiplicity trigger (1ormore, 2ormore, 3ormore, 4ormore) (anode wire signals can be fed from adc16 or from adc32 signals)
  • anode wire coincidence trigger
  • anode wire MLU (memory lookup) trigger

NIM trigger from ADC front panel

Trigger path:

NIM signal --> LEMO input on ADC front panel --> data encoder --> data link to trigger board --> data decoder --> mask --> nim-esata-grand-or --> trigger

To enable NIM trigger, do this:

• connect NIM signal to ADC front panel input
  • Note1: NIM signal must be inverted
  • Note2: LEMO connectors should be set to "NIM input" mode by ADC on-board jumpers (see ADC manual)
• observe correct bit goes to zero in the TRG web page: data link high work should change:

0xNF00'0000 (N is the module id) to
0xN700'0000 (left lemo) or
0xNB00'0000 (right lemo)

• Note3: if data link bits do not change, most likely the LEMO connectors are set to "DAC output" mode. Try to use LEMO inputs of a different ADC.
• compute the NIM mask: there are 32 bits, 2 bits per ADC data link. Bit 0x1 is right LEMO, bit 0x2 is left LEMO
• for example, to use left LEMO of ADC data link 12, set nim mask to (0x2)<<(2*12) = 0x02000000
• development branch of the git repository contains scripts/nim_mask.py, which can compute forwards and backwards
• initialize the trigger board (on the TRG web page press button "initialize"), or start a new run
• on the history plot with "NIM grand or" counter, the rate should change from zero
• if a run is active, if one of the ADC LEMO outputs is set to output the trigger signal as a NIM pulse, one should see it fire use a scope. The delay or jitter between NIM trigger in and NIM trigger out should be around 100-200 ns (FIXME - measure this!)
• to set the DAQ to trigger on the NIM signal, on the run start page, check the box "TrigEsataNimGrandOr".
  • Note: remember to set the scaledown to 0 if so desired.

TPC field wire pulser

The TPC field wire pulser input should be connected directly to the DAC output of ADC[0] (right lemo connector).

DAC settings in ODB /Equipment/CTRL/Settings/ADC/DAC/

• for pulsing PWBs, baseline -1V, pulse height +2V (maximum possible pulse height in PWBs. Overdrives the AW ADC waveforms)
  • dac_baseline: -8000
  • dac_amplitude: 8000
  • ramp_up_rate: 128
  • ramp_down_rate: 70
  • ramp_top_len: 128

• for pulsing AW, baseline -0.25V, pulse height +0.7V (AW ADC waveforms are in range. PWB waveforms are too small for goor time resolution)
  • dac_baseline: -2000
  • dac_amplitude: 4000
  • ramp_up_rate: 20
  • ramp_down_rate: 20
  • ramp_top_len: 128

BSC pulser

The BSC pulser input on the BSC VME rear transition board (RTM) should be connected directly to the DAC output of ADC[4,5,6,7] (right lemo connector). (8 RTMs, but 4 ADCs, each signal is split using a LEMO "T" connector).

DAC settings in ODB /Equipment/CTRL/Settings/ADC/DAC/

• for pulsing BSC, baseline -0.950V, pulse height +1.4V
  • dac_baseline: -8000
  • dac_amplitude: 4000
  • ramp_up_rate: 64
  • ramp_down_rate: 64
  • ramp_top_len: 128

NIM output

To configure ADC analog output for NIM compatible pulse:

DAC settings in ODB /Equipment/CTRL/Settings/ADC/DAC/

• for output of NIM pulse: baseline 0V, pulse height -0.9V, pulse width 100 ns
  • dac_baseline: 0
  • dac_amplitude: 8000
  • ramp_up_rate: 128
  • ramp_down_rate: 128
  • ramp_top_len: 4
  • dac_xor: 1 (invert the signal)

TDC connections

• SFP8 - fiber SFP to Juniper switch
• right RJ45 - 200MHz clock
• left RJ45 - external trigger

MIDAS Data banks

• ATAT - fectrl - UDP packets from GRIF-C/TRG trigger board
• TRBA - fetdc - TDC data
• TRTM - fetdc - TDC FPGA temperatures, TID_FLOAT[5].
• AAnn - feudp - UDP packets from GRIF-16 ADC (before the event builder), "nn" is the ADC module number: adc01 is AA01 through adc20 is AA20.
• BBxx - feudp - UDP packets from PWB, obsolete
• Bnnm, Cnnm - feevb - EVB output banks: ADC data, "nn" is the ADC module number (1..20), "m" is the channel number (0..9,A..F), format is same as AAnn banks.
• PAnn - feudp - UDP packets from PWB, "nn" is the PWB module number (00..99), obsolete
• PBnn - feudp - UDP packets from PWB, "nn" is the PWB module number (00..99), PB01 is pwb01, PB78 is pwb78
• PCnn - feevb - EVB output banks: PWB data, "nn" is the PWB module number (00..99), same as above, format is same as PBnn banks

ZZZ

ZZZ