TACTIC

TACTIC

• computer: daq18.triumf.ca
• midas: https://daq18.triumf.ca
• elog: https://elog.triumf.ca/Tactic/General/ (has a "tactic" user account)

Old V1740 DAQ

Overview

Uses 2x V1740 via CAEN USB interface. 64 channels each - can read out 2 of the 8 TACTIC "sectors".

These programs should be running:

• mhttpd (web server)
• Logger (writing data to disk)
• feMKS (gas mixer)
• fetactic (data readout from digitizers)
• ana (online analysis of data)

Wiring diagram and trigger logic

Trigger and v1740 connections:

• both v1740 trigger outputs go to logic OR. pulser goes into the same OR
• from this OR, they fan out to v1740 trigger inputs
• analog output of first v1740 provides the run gate (via polarity inverting kludge)
• analog output of second v1740 provides a "buffer-half-full" trigger veto (via polarity inverting kludge)

Trigger notes: (KO 7-oct-2024)

• trigger signal is not synched with the v1740 clock, actual "adc stop" time will have a jitter between the two v1740 of 1-2 clock periods
• actual event time jitter between the two v1740 is around 2 clock (from looking at event timestamps).
• because v1740 busy are not synced (and busy from first v1740 is missing), the second v1740 often has extra events. sometimes the first v1740 has an extra event. fetactic.exe has workaround against this, mostly by dropping mismatched data.

Analyzer notes

• Code is in /home/tactic/newTACTIC/manalyzer.
• Not source-controlled.
• Contains many hard-coded thresholds/settings/limits. Also includes a hard-coded mapping between digitizer channel number and hardware sector/pad number.
• Used for both online and offline analysis of data.

Re-compilation

# Rebuild fetactic.exe
cd /home/tactic/newTACTIC/v1740mt
make
ls -l fetactic.exe

# Rebuild analyzer
cd /home/tactic/newTACTIC/manalyzer
make
ls -l ./tacticana.exe

New V2745 DAQ

Overview

Uses 6x CAEN V2745 digitizers, with option for reading 8 digitizers if needed. With 8 digitizers can read all 8 sectors from the hardware.

Digitizers are readout over 1G ethernet, and run DPP_ZLE firmware to reduce the data rate significantly. Future versions of TACTIC may run DPP_PHA or custom firmware to further reduce the data rate.

Wiring diagram and trigger logic

The trigger is a global OR of every channel of every digitizer (though we can suppress certain channels if desired).

Digitizers are set to NIM logic levels, and OR is done using standard NIM fan-in/fan-out modules. Trigger-OR is sent to 9 locations (8 digitizers + 1 visual scaler) so we use a 16x FI/FO. Busy-OR is only sent to 8 locations, so can use an 8x FI/FO.

Frontend notes

The main frontend code is shared with the Darkside-20k experiment that uses VX2740/VX2745 digitizers running custom firmware. The repository name (dsproto_vx2740) reflects this heritage, but the code now supports more than just Darkside's usage! The code is located on BitBucket at https://bitbucket.org/ttriumfdaq/dsproto_vx2740/

Digitizer configuration is done via the VX settings webpage. The concept is that the "default" value is applied to all digitizers, unless you specifically override the value for a certain digitizer. This means it's more efficient to change settings, as you only have to change one "default" parameter rather than 8 separate board-specific parameters. Settings that have been overridden for one or more boards appear with a YELLOW background on the webpage.

Settings that users may want to configure

• Readout channel mask - which channels get read out when a trigger is accepted
• Ch over thresh A en mask - which channels contribute to the self-trigger logic
• Chan over thresh thresholds - the thresholds in ADC RELATIVE TO THE BASELINE
• Scope/ZLE waveform length - how long the waveforms should be.
• Pre-trigger for ZLE mode - where in the waveform the trigger pulse should appear
• ZLE threshold (ADC) - how many ADC below baseline a signal must be before it gets read out. Note this is different to the trigger threshold! You could trigger readout if the signal goes 50ADC below baseline, but then read out any data that's 10ADC below baseline.
• ZLE look back (samples) - how many samples to read out before the signal went below the ZLE threshold
• ZLE look forward (samples) - how many samples to read out after the signal goes back above the ZLE threshold
• Ch over thresh A gate width(ns) - set this to roughly how long you expect a "real" event to last. If you set it too short then the NIM OR module will output triggers multiple times for a single event (e.g. for the first channel to get hit, and then 100ns later for another channel, and then 300ns after that for another channel, ....). The digitizers would still only read out data for the first trigger, but the visual scaler will show a much higher rate, which would be very confusing for debugging.
• DC offset (pct) - set the baseline between 0ADC and 65535ADC, as a percentage
• Signal offset (uV) - extra tweak to the input signal - best set using a script
• VGA gain - 0-40dB in 0.5dB increments

Settings that are different for the first digitizer in the chain

• Run start source
  • First board: Start acq on midas run start
  • Other boards: Start acq on encoded CLKIN
• Use external clock
  • First board: True
  • Other boards: False

Settings that users shouldn't touch

• Run start delay (ns) - 0 for the last board, increasing by 48ns for each board in the chain (240, 192, 144, 96, 49, 0).
• Trigger out mode - ITLA - ensure we output the self-trigger signal
• Trigger on external signal - True - ensure we only trigger on the global OR that gets fed back in to the board
• ZLE enable - True for all channels
• Ch over thresh A multiplicity - 1 - in principle you could require 2+ hits simultaneously before we trigger (to reduce triggering on noise/bad topology events), but this setting only applies within a single board. So you'll get weird geometric efficiencies (more likely to trigger on tracks that hit multiple pads in the same sector vs pads in different sectors). If we want to trigger on 2+ hits across multiple boards we can use the DAC or LVDS connectors, and connect them to a board running custom firmware.
• Chan over thresh rising edge - False - TACTIC uses negative pulses
• Veto source / Veto when source is high - Disabled / True
• Use NIM IO - True
• Enable clock out - True
• GPIO mode - Busy
• Busy in source - SIN
• Sync out mode - Run
• Enable DC offsets - True

Analyzer notes

• Code is in /home/tactic/newTACTIC/tacticana on daq18, and https://bitbucket.org/ttriumfdaq/tacticana/src/master/ on bitbucket.
• Uses JSON files to define histogram limits, channel mapping etc. See the README.md file in the repository for details.
• Specify which config file to use via the --config flag.
• It replaces "gaps" in the ZLE waveforms with the baseline value (so the rest of the code sees a "normal" waveform with some very flat sections). This was done to minimise the number of changes needed in the rest of the analysis code that was inherited from the V1740-based DAQ.

# Run online using the v2745.json config file:
/home/tactic/newTACTIC/tacticana/tacticana.exe --midas-progname tacticana -R8081 -- --conf /home/tactic/newTACTIC/tacticana/v2745.json

# Run offline using the v2745.json config file:
/home/tactic/newTACTIC/tacticana/tacticana.exe /path/to/something.mid -- --conf /home/tactic/newTACTIC/tacticana/v2745.json

# NOTE THE --config FLAG MUST COME AFTER A '--'!!!!!

Custom webpages

• The VX settings page lets you configure all the digitizers. As note above, there is a column for the "default" value for all digitizers, plus optional board-specific "override" values.
• The VX waveforms page shows realtime display of waveforms from a handful of channels. It currently only allows you to view 4 channels from a single board. For more advanced options, use the tacticana new webpage.
• The VX rates page shows the readout rate of each board/channel, as well as the overall readout rate over time. The former may be useful for identifying any abnormally "hot"/"cold" channels that have a bad baseline. The latter is useful for tracking stability over time.
• The VME crate page allows you to power-cycle the VME crate remotely. Useful if the digitizers get in a really bad state and can only be fixed by power-cycling (stop the run, stop the VX frontend, power-cycle the crate, start the VX frontend, wait for the status page to show "Initialized", start the run)
• The tacticana new page is a ROOT-based display of histograms created by the analyzer. You will need to refresh the page when a new run starts.

Setting baselines

The V2745s have two settings related to the baseline of each channel.

• DC offset sets the target of where the baseline "should" be. It's specified as a percentage of the full scale 0-65535. For TACTIC with negative pulses, it makes sense to set it to 90% or so (aka 58982 ADC).
• Signal offset is an extra tweak to each channel (in uV) that can be used to ensure a 0V input actually gets to the expected baseline.

ZLE and trigger thresholds are RELATIVE TO THE EXPECTED BASELINE (i.e. 58982 ADC). The digitizers to not monitor the actual data to decide what the "real baseline" is. It is therefore critical that the signal offset is set so that the real baseline matches the expected baseline.

• If the real baseline is lower than expected, you will trigger more often and your data rate will be higher.
• If the real baseline is higher than expected, you will miss triggering on small pulses, and may miss some ZLE chunks that would otherwise be readout.

Note that baselines often drift based on the temperature of the board.

There is a script in the dsproto_vx2740 repository that can be used to set the "signal offset" of each channel so that the real baseline matches the expected baseline. This script is very basic at the moment, but could be made smarter (both in terms of the algorithm for finding the optimal offset, and in automating more of the settings that need to be changed).

# Stop the run
# Ensure there aren't any real signals coming in (i.e. we're just reading noise)
# Set 'ZLE enable' to False for all channels
# Set 'Trigger out mode' to 'TestPulse' for the first board, and 'Disabled' for all other boards

cd ~/newTACTIC/dsproto_vx2740/
python set_offsets.py 

# Set 'ZLE enable' to True for all channels
# Set 'Trigger out mode' to 'ITLA' for all boards

Adding more digitizers to the chain

There are currently 6 digitizers in the chain. 2 more will be delivered soon. To fully add a new digitizer to the chain, do the following steps. You should only connect one "new" board to the ethernet network at a time (as otherwise they will fight over who gets to use the hard-coded default IP address).

• Turn off the VME crate
• Install the new digitizer in the VME crate
• Connect digitizer via ethernet cable to the switch
• Connect CLK-IN of this board to the CLK-OUT of previous board in chain
• Connect LEMO cables to NIM fan-in/fan-out modules (GPIO, S-IN, TRG-IN, TRG-OUT)
• Turn on the VME crate
• On daq18 web browser, navigate to http://192.168.0.254 (this is the default IP address that boards use coming from the factory)
  • On the Firmware page, load the DPP-ZLE firmware
  • On the Network page, set the IP address to 192.168.0.xxx, where xxx isn't already used (e.g. 7 for the 7th board)
• On daq18, as root, edit /etc/hosts to assign a hostname (e.g. vx07 for 192.168.0.7)
• On the midas Programs webpage, stop the VX frontend
• On the midas VX settings webpage, change the number of boards (first row of first table)
• On the midas Programs webpage, start the VX frontend. It will abort immediately, but will have created space in the ODB for you to configure the new board
• On the midas VX settings webpage, set the hostname of the new board and adjust the run start delay by 48ns for all the earlier boards in the chain
• On the midas Programs webpage, start the VX frontend. It should start correctly this time

Re-compilation

# Rebuild vx2740_group_fe.exe
cd /home/tactic/newTACTIC/dsproto_vx2740/build
make install -j

# Rebuild analyzer
cd /home/tactic/newTACTIC/tacticana
make -j

Known issues

Rare bug in CAEN software

There is a rare bug in the CAEN software/firmware that causes the entire frontend to crash on malformed data. This can be quite annoying, and could cause significant downtime if the user doesn't realise that the program has crashed and/or the run has stopped.

midas contains a "sequencer" functionality that can be used to mitigate this. Every 10s the sequencer will check to see if the frontend has crashed, and restart it if needed. It will also start a run if the current one has stopped.

To use this sequencer, go to the PySequencer page and start the script using the "play" button. You'll be prompted for how long you want to acquire data for.

Note that if you want to stop data-taking early, you should use the "stop" (square) button on the PySequencer page, not the regular button on the status page for stopping a run (as the sequencer will just detect that the run has stopped and start a new one for you!).

Click 'Expand' to view the actual sequencer script I implemented ---->

import datetime
import midas

def define_params(seq):
    seq.register_param("duration_hours", "Total acquisition time", 5.0)

def sequence(seq):
    stop_run_if_needed(seq)
    start_time = datetime.datetime.now()
    elapsed_hours = 0.0
        
    while elapsed_hours < seq.get_param("duration_hours"):
        validate_prog_running(seq)
        seq.wait_seconds(10)
        elapsed_hours = round((datetime.datetime.now() - start_time).total_seconds() / 3600, 2)
        
    stop_run_if_needed(seq)
        
def at_exit(seq):
    stop_run_if_needed(seq)
    
def stop_run_if_needed(seq):
    if seq.odb_get("/Runinfo/State") != midas.STATE_STOPPED:
        seq.stop_run()
    
def validate_prog_running(seq):
    prog_name = "VX2740_Group_01"
    if not seq.client_exists(prog_name):
        stop_run_if_needed(seq)
        seq.msg(f"Stopped run as {prog_name} was not running! Restarting program.")
        seq.wait_seconds(5)
        seq.start_client(prog_name)
        seq.wait_clients_running(prog_name)
        
        if not seq.client_exists(prog_name):
            raise RuntimeError(f"Failed to restart {prog_name} - abort sequence!")
            
        seq.wait_odb("/Equipment/VX2740_Data_Group_001/Common/Status", "==", "Initialized")
        
    if seq.odb_get("/Runinfo/State") != midas.STATE_RUNNING:
        seq.start_run()