DS-DM

DS-DM

DarkSide-20k Global and Crate Data Manager board (GDM and CDM).

Global Data Manager (GDM):

• clock distribution to CDM boards (including GPS/atomic clock source)
• collection of trigger data from CDM boards, processing and distribution of trigger decision to CDM boards
• run control
• integration with GPS 10MHz and 1pps clocks and GPS/IRIG date and time information

Crate Data Manager (CDM):

• clock distribution from GDM to CAEN VX digitizers
• receive trigger data from CAEN VX digitizers
• send trigger data to GDM
• run control and dead time control

Links

• https://dsvslice.triumf.ca/vx_napoli - DS vertical slice at TRIUMF
• https://edev-group.triumf.ca/fw/exp/darkside/gcdm - git repository, DS-DM firmware
• https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0 - DS-DM board Rev0
• https://edev-group.triumf.ca/hw/exp/dark-side-20k/cdm-gdm-data-manager-vme-cards/rev1 - DS-DM board Rev1
• https://docs.xilinx.com/v/u/en-US/ds593 - Xilinx Platform Cable USB II
• https://www.enclustra.com/en/products/system-on-chip-modules/mercury-xu8/ - Enclustra
• https://bitbucket.org/team-ds-dm/ds-dm-software - git repository, DS-DM MIDAS frontend
• https://bitbucket.org/team-ds-dm/phasemeasurement - VX Phase measurement script
• https://ladd00.triumf.ca/daqinv/frontend/list/178 - inventory database
• https://daq00.triumf.ca/elog-ds/DS-DAQ - DS-DAQ elog
• https://bitbucket.org/ttriumfdaq/dsproto_vx2740/src/develop/ - VX DSFE MIDAS frontend
• https://dsdaqgw.triumf.ca/vslice/ - vertical slice DAQ
• https://daq00.triumf.ca/DaqWiki/index.php/DarkSide - DS-DAQ Wiki page
• https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0 - GPS and Rb Clock adapter board

Onboard hardware

• jtag chain: arm_dap_0 0x5BA00477, xczu4_1 0x04721093
• Eclustra Mercury+ XU8 module: ME-XU8-4CG-1E-D11E-R2.1
  • Xilinx® Zynq Ultrascale+™ MPSoC XCZU4CG-1FBVB900E
  • DDR4 ECC SDRAM (PS) 2 GB
  • DDR4 SDRAM (PL) 1GB
• ethernet mac chip: AT24MAC402-SSHM-T ("602" chip is wrong)
• USB UART for Enclustra serial console, micro-USB, 115200n8
• clock chip: SI5394A-A-GM and oscillator CS-044-054.0M (54 MHz)
• U23 3.3V current meter and thermometer, LTC2990IMS#TRPBF
• LEDs:
  • LED_FP A/B/C/D 0/1/2/3
  • led1 - 3V3_SW_ON, SOM_POWER_GOOD - Enclustra FPGA module 3.3V power is good
  • led2 - LTM4624 PGOOD
  • led3 - FPGA_DONE - FPGA has booted
  • led4 - TP-S-1, PCLK_P
  • led5 - TP-S-2, PCLK_N
• LEMO connectors (top to bottom)
  • J4 - input (NIM/TTL) (EXT_IN_LV(1), EXT_IN_LV(2))
  • J5 - input (NIM/TTL) (EXT_IN_LV(3), EXT_IN_LV(4))
  • J6 - external clock (GPS 10MHz and PPS)
  • J7 - output (NIM/TTL) (EXT_OUT(1), EXT_OUT(2))
• SMA connectors
  • J9, J10 - CLK_CCA from U6 C.C.
  • J11, J12 - CLK_TP0
• RJ45 ethernet connector (100mbit: green light on, yellow light flashes when there is traffic)
• SFP connector (SFP is FTLF8526P3BNL, 6 Gbit/sec, 850 nm, 300m 50/125um OM3 MMF)
• 4 QSFP connectors (GDM)
• 6 VX connectors (CDM)

Buttons, jumpers and switches

Buttons:

• PB1 - HRST - reboot FPGA (power-on reset)
• PB2 - SRST - (SRSTn) - reboot ARM CPU

Switches:

• SW1 - boot mode BM0, BM1 [-->]
• SW2 - LEMO output NIM<->TTL
• SW3 - LEMO input 1 and 2 NIM/TTL
• SW4 - LEMO input 2 and 4 NIM/TTL
• SW5 - LEMO clock input NIM/TTL
• SW6 - serial console select. [PS<--PL] PS is ARM CPU, PL is FPGA.

Front panel

| top
|
| LED-FP1 | LED_FP(0,1,2,3)
|
| SFP J???
|
| LEMO J4-LEFT, J4-RIGHT | EXT_IN_LV(1), EXT_IN_LV(2)
| LEMO J5-LEFT, J5-RIGHT | EXT_IN_LV(3), EXT_IN_LV(4)
| LEMO J6-LEFT, J6-RIGHT | CLK_EXT1, CLK_EXT0 (125 MHz only) 
| LEMO J7-LEFT, J7-RIGHT | EXT_OUT(1), EXT_OUT(2)
|
| J-VX-1
| J-VX-2 or QSFP-1
| J-VX-3 or QSFP-2
| J-VX-4 or QSFP-3
| J-VX-5 or QSFP-4
| J-VX-6
|
| RJ45 J3 ethernet
|
| bottom

VX adapter board

LVDS I/O connector

34 pin connector: 0|:::: :::: :::: :::: :|15,16 n/c

split-cable connection

0 .. 7 -> N/C

8 -> VX_RX(3) - not used
9 -> VX_RX(2) - busy VX to CDM
10 -> VX_RX(1) - DS20K 125 MHz serial data VX to CDM
11 -> VX_RX(0) - DS20K 62.5MHz clock VX to CDM

12 <- VX_TX(0) - TRG CDM to VX
13 <- VX_TX(1) - TSM CDM to VX, to be VETO CDM to VX
14 <- VX_TX(2) (set by jumper routed here or to VX CLKIN SYNC) - 125 MHz serial data CDM to VX
15 <- n/c (CLK routed to VX CLKIN CLK) - 62.5 MHz clock CDM to VX

16 - n/c

one-to-one connection

0 -> VX2_RX(3)
1 -> VX2_RX(2)
2 -> VX2_RX(0)
3 -> VX2_RX(1)

4 <- CLK
5 <- VX2_TX(0)
6 <- VX2_TX(1)
7 <- VX2_TX(2)

8 -> VX1_RX(3)
9 -> VX1_RX(2)
10 -> VX1_RX(1)
11 -> VX1_RX(0)

12 <- VX1_TX(0)
13 <- VX1_TX(1)
14 <- VX1_TX(2) (set by jumper routed here or to VX CLKIN SYNC)
15 <- n/c (CLK routed to VX CLKIN CLK)

16 - n/c

Board schematics Rev0

• File:SCH-DS-xDM-Rev0.PDF
• note: FPGA pin annotations ("IO", "SCLK", "PCLK", etc) on the schematics are bogus, instead, trace them to the FPGA pins.
• note: ENC A is J800, ENC B is J801, ENC C is J900 (schematic name to enclustra name)
• note: Enclustra special pins: "GC" is "clock capable", "HDGC" is "clock capable", "MGTREFCLK" is MGT reference clocks.
• board modifications:
  • ethernet mac chip
  • NIM output (no U15, etc)
  • RJ45 wrong pinout (board mod or special ethernet cable)
  • 125 MHz clock mods (TBW)
  • disconnect QSFP0_SEL from SFP_RS0 and QSFP1_SEL from SFP_RS1, these signals are not used by modern SFPs
  • provide SFP i2c modsel to allow SFP and QSFP at the same time on address 0x50

Board schematics Rev1

• File:SCH-DS-xDM-Rev1.PDF
• ENC-A is J800, ENC-B is J801, ENC-C is J900
• modifications from Rev0:
• clocks:
  • 125MHz osc: CLK_XO_125, CLK3_XO_125 (CLK2_XO_125 removed)
  • C.C. in2 is CLK_CC_IN from ENC-C142,144
  • C.C. out0 via CLK_A to CLK_CC_OUT0 (ENC-C3,5), CLK_CC_OUT1 (ENC-B3,5), CLK_CC_OUT2 (ENC-C151,153)
• RTC chip: TP_S pins gone, RTC_I2C_SCL, RTC_I2C_SDA, RTC_1PPS and RTC_32KHZ added
  • RTC_1Hz ENC-C160 pull up 10k to 3.3V
  • RTC_32k ENC-B129 pull up 10k to 1.8V
• QSFP, SFP control lines:
  • HW_ID_xDM ENC-B131 gone, reused as SFP_ModPrsN
  • SFP_ModPrsN (MOD_ABS) ENC-B131 input (no pulls)
  • SFP_TX_fault (was QSFP2_SEL) QSFP_ModPrsN ENC-C157 input (no pulls)
  • SFP_TX_disable from QSFP_LPMode ENC-C163 output (pull up 10k to 3.3V)
  • SFP_ModDet n/c (was QSFP_ModPrsN) renamed SFP_ModPrsN input
  • SFP_RX_LOS to QSFP_IntN ENC-C159 input (pull up 2k2 to 3.3V)
  • SFP_RS0, SFP_RS1 n/c, assembly option: floating
  • QSFP_ResetN - FPGA output, pull up 10k to 3.3V
  • QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL - FPGA outputs, pull up 10k to 3.3V
  • QSFP_LPmode - FPGA output, pull up 10k to 3.3V
  • QSFP_ModPrsN - ??? - pull up 2k2 to 3.3V
  • QSFP_IntN - ??? - pull up 2k2 to 3.3V
• VX_TX and VX_RX reassigned for better length matching
• cosmetic changes:
  • front panel USB-C connector (replaces Micro-USB)
  • (reset push button PB1 PORn ENC-A132 aka PS_POR_B aka PS_POR#)
  • (reset push button PB2 SRSTn ENC-A124 aka PS_SRST_B aka PS_SRST#)
  • front panel PB3 HRSTn same as PB1
• Enclustra errata:
  • possible cross talk between pins AA2 (J801-B90, VX9_RX2) and AE3 (J801-B129, RTC_32KHZ). no RTC on the CDM.
  • ethernet link up failure (see Microchip KSZ9031 errata)

FPGA MGT blocks

* SFP - ENC B45,B47 and B48,B50 - FPGA D5,D6 MGTHTX0_D and D1,D2 MGTHRX0_D
* QSFP0 TX0 - ENC C13,C17 - FPGA H5,H6 MGTHTX0_C
* QSFP0 TX1 - ENC C21,C25 - FPGA G7,G8 MGTHTX1_C
* QSFP0 TX2 - ENC C29,C23 - FPGA F5,F6 MGTHTX2_C
* QSFP1 TX0 - ENC C37,C41 - FPGA E7,E8 MGTHTX3_C
* QSFP1 TX1 - ENC C45,C47 - FPGA P5,P6 MGTHTX0_B
* QSFP1 TX2 - ENC C51,C53 - FPGA M5,M6 MGTHTX1_B
* QSFP2 TX0 - ENC C57,C59 - FPGA L3,L4 MGTHRX2_B
* QSFP2 TX1 - ENC C63,C65 - FPGA K5,K6 MGTHTX3_B
* QSFP2 TX2 - ENC C75,C77 - FPGA W3,W4 MGTHTX0_A
* QSFP3 TX0 - ENC C79,C81 - FPGA V5,V6 MGTHTX1_A
* QSFP3 TX1 - ENC C85,C87 - FPGA T5,T6 MGTHRX2_A
* QSFP3 TX2 - ENC C89,C91 - FPGA R3,R4 MGTHTX3_A

Clock distribution Rev0

Simplified:

• 125 MHz osc -> CLK_XO_125 -> MGTREFCLK0_A -> not used
• 125 MHz osc -> CLK3_XO_125 -> MGTREFCLK1_B -> SFP RX ref clock, QSFP RX and TX ref clock (this is not final design!)
• (disconnected) 125 MHz osc -> CLK2_XO_125 -> FPGA AG8,AH8 (GC)
• 125 MHz osc -> C.C. in1

• C.C. in0 <- CLK_EXT1 (10 MHz GPS clock)
• C.C. in1 <- 125 MHz osc
• (disconnected) C.C. in2 <- CLK_CC_IN <- MGTREFCLK0_D <- SFP RX clock (cannot be used because of uncontrollable phase)
• C.C. in2 <- CLK2_XO_125 <- FPGA AG8,AH8 (GC) <- SFP RX recovered 125 MHz clock
• C.C. in3 <- CLK_FB

• C.C. 125 MHz -> CLK_CC_OUT0 -> MGTREFCLK0_B -> QSFP RX and TX ref clock (final design)
• C.C. 125 MHz -> CLK_CC_OUT1 -> MGTREFCLK1_D -> SFP TX clock
• C.C. 62.5 MHz -> VX clock fanout

proposed changes:

• add C.C. 125 MHz -> new CLK_CC_OUT2 (old CLK2_XO_125) -> FPGA AG8,AH8 (GC)
• change C.C. in2 <- new CLK_CC_IN <- FPGA AK8,AK9 (non-GC)

Complete:

125 MHz oscillator - U5 fan out -
  q0 -> CLK_XO_125 -> ENC C72,C74 -> FPGA R7,R8 MGTREFCLK0_A (not used)
  q1 -> U6 C.C. in1
  q2 -> CLK3_XO_125 -> ENC C7,C9 -> FPGA J7,J8 MGTREFCLK1_B -> SFP RX reference clock, QSFP RX and TX reference clocks (not final design!)
  q3 -> disconnected on the board, was CLK2_XO_125 -> ENC C151,C153 -> FPGA AG8,AH8 (GC)

U6 C.C (clock cleaner) -

in0 <- CLK_EXT1 (presumably GPS 10 MHz ref clock)
in1 <- 125 MHz oscillator via U5
in2 <- (was: CLK_CC_IN <- ENC B10,B12 <- FPGA D9,D10 MGTREFCLK0_D <- SFP RX clock, 125 MHz)
in2 <- CLK2_XO_125 <- ENC C151,B153 <- FPGA AG8,AH8 (GC) <- mgt_link_data_to_processing.rx_data_clk (SFP RX data clock, 125 MHz)
in3 <- CLK_FB

out0 -> CLK_CCA -> U12 (125 MHz)
out1 -> CLK_CCB -> VX1..6 (62.5 MHz)
out2 -> CLK_CCC -> VX7..12 (62.5 MHz)
out3 -> CLK_FB into in3

CLK_CCA -> U12 (125 MHz fan out) ->

Q0 -> not used
Q1 -> CLK_CC_OUT0 -> ENC C3-5 -> FPGA L7,L8 MGTREFCLK0_B -> QSFP RX and TX reference clocks (final design)
Q2 -> CLK_CC_OUT1 -> ENC B3-5 -> FPGA B9,B10 MGTREFCLK1_D -> SFP TX clock
Q3 -> not used
Q4 -> not used
Q5 -> SMA J9/J10

CLK_TP0 - FPGA N7,N8 MGTREFCLK1_A - ENC C69,C71 - DS-DM SMA J11, J12 (NOT IN CDM PROJECT)

Notes:
* CLK_XO_125 (125 MHz osc) is not used
* 62.5 MHz VX clock does not go into the FPGA
* CLK_EXT0 going to in1 of U5 cannot be used. only permitted frequency is 125 MHz (it drives the MGT reference clocks) and is this frequency is too high for LEMO cables and connectors

Proposed modifications:
- CLK_CC_IN: from FPGA output pin (ENC C142,C144 - FPGA AK8,AK9) to clock cleaner in2 (instead of CLK2_XO_125 pins)
- CLK_CCA -> U12 -> currently unused out3 -> CLK2_XO_125 FPGA pins
- repurpose CLK_EXT0 at GPS 1pps/IRIG input to FPGA

Clock distribution Rev1

Simplified:

• 125 MHz osc -> CLK_XO_125 -> MGTREFCLK0_A -> not used
• 125 MHz osc -> CLK3_XO_125 -> MGTREFCLK1_B -> SFP RX ref clock
• 125 MHz osc -> C.C. in1

• C.C. in0 <- CLK_EXT1 (10 MHz GPS clock)
• C.C. in1 <- 125 MHz osc
• C.C. in2 <- CLK_CC_IN <- FPGA AK9,AK8 <- SFP RX recovered clock, 125 MHz
• C.C. in3 <- CLK_FB

• C.C. 125 MHz -> CLK_CC_OUT0 -> MGTREFCLK0_B -> QSFP RX and TX ref clock
• C.C. 125 MHz -> CLK_CC_OUT1 -> MGTREFCLK1_D -> SFP TX clock
• C.C. 125 MHz -> CLK_CC_OUT2 -> FPGA AG8,AH8 (GC) -> not used
• C.C. 62.5 MHz -> VX clock fanout

Complete:

125 MHz oscillator - U5 fan out -
  q0 -> CLK_XO_125 -> ENC C72,C74 -> FPGA R8,R7 MGTREFCLK0_A -> XDC CLK_XO_125_P -> VHDL not used
  q1 -> U6 C.C. in1
  q2 -> CLK3_XO_125 -> ENC C7,C9 -> FPGA J8,J7 MGTREFCLK1_B -> XDC GDM missing, CDM CLK3_XO_125_P -> VHDL SFP RX reference clock (mgt_rx_ref_clk)
  q3 -> not used, was CLK2_XO_125

U6 C.C (clock cleaner) -

in0 <- CLK_EXT1 (GPS 10 MHz clock)
in1 <- 125 MHz oscillator via U5
in2 <- CLK_CC_IN <- ENC C142,C144 <- FPGA AK9,AK8 <- XDC GDM missing, CDM CLK_CC_IN1_P <- VHDL rx_clk
in3 <- CLK_FB

out0 -> CLK_CCA -> U12 (125 MHz)
out1 -> CLK_CCB -> VX1..6 (62.5 MHz)
out2 -> CLK_CCC -> VX7..12 (62.5 MHz)
out3 -> CLK_FB into in3

CLK_CCA -> U12 (125 MHz fan out) ->

Q0 -> not used
Q1 -> CLK_CC_OUT0 -> ENC C3,C5 -> FPGA L8,L7 MGTREFCLK0_B -> XDC GDM clk_mgtrefclk0_x0y1_p, CDM CLK_CC_OUT0_P -> VHDL  GDM mgt_b_ref_clk QSFP RX and TX reference clocks, CDM alternate rx_clk via clock mux
Q2 -> CLK_CC_OUT1 -> ENC B3,B5 -> FPGA B10,B9 MGTREFCLK1_D -> XDC CLK_CC_OUT1_P -> VHDL GDM not used, CDM mgt_tx_ref_clk SFP TX clock
Q3 -> CLK_CC_OUT2 -> ENC C151,C153 -> FPGA AG8,AH8 (GC) -> XDC GDM, CDM missing
Q4 -> not used
Q5 -> SMA J9/J10

CLK_TP0 <-> ENC C69,C71 <-> FPGA N8,N7 MGTREFCLK1_A (not used) -> XDC CLK_TP0_P -> VHDL not used (DS-DM SMA J11, J12)

Notes:
* Enclustra ENC-B is J801, ENC-C is J900
* "(GC)" is clock-capable FPGA pin
* CLK_XO_125 (125 MHz osc) is not used in FPGA
* 62.5 MHz VX clock does not go into the FPGA
* CLK_EXT0 going to in1 of U5 cannot be used. only permitted frequency is 125 MHz (it drives the MGT reference clocks) and is this frequency is too high for LEMO cables and connectors

I2C bus

• I2C_SCL is J-ENC-A pin 111 I2C_SCL is FPGA I2C_SCL_PL AB13 (IO_L1N_TOL1D_64) and I2C_SCL_PS F18 (PS_MIO10)
• I2C_SDA is J-ENC-A pin 113 I2C_SDA is FPGA I2C_SDA_PL AH13 (IO_L7N_T1L1Q_AD13N_64) and I2C_SDA_PS G18 (PS_MIO11)

• XU8 secure EEPROM ATSHA204A at 0x64, this is 0110'010X -> linux _011'0010 is 0x32. (but responds to scan and read at 0x33)
• U4 ethernet mac chip, EEPROM at 1010 A2 A1 A0 X and MAC/serial_no at 1011 A2 A1 A0 X. A0=VCC, A1=VCC, A2=GND -> linux _101'0011 and _101'1011 is 0x53 and 0x5B.
• U6 clock chip, address 1101 0 A1 A0 X. A1=VCC, A0=N/C (internal pull-up) -> linux _110'1011 is 0x6b
• U23 voltmeter at 10011 ADR1 ADR0. ADR0=GND, ADR1=VCC -> linux _100'1110 is 0x4e
• SFP, address 1010000X -> linux _101'0000 is 0x50. additional SFP data at 0x51
• QSFP0, QSFP1, QSFP2, QSFP3 (QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL)

I2C clock builder connection

• use Silicon Labs USB "Clock builder pro field programmer", www.silabs.com/CBProgrammer
• connect rainbow jumper cable pins:
  • black - 1-GND to GND on DS-DM
  • white - 3-SCLK to SCL on the DS-DM
  • grey - 7-SDA_SDIO to SDA on the DS-DM
• power up the DS-DM
• plug USB programmer into Windows laptop
• on Windows, run "ClockBuilder Pro"
• it should report "Field programmer detected", press "EVB GUI"
• in EVB GUI, press "Config", set I2C address 0x6B
• press "Scan", it should find Si5394A-A-GM
• select the "Status" tab, should see real-time status of clock chip

GDM MGT configuration

• TX configuration:
• GDM MGT transceivers are configured as "multilane" TX and RX.
• there is 12 TXes ("lanes")
• MGT reference 125 MHz clock goes into [2:0]gtreclk00_in and [11:0]gtrefclk0_in
• one MGT is designated as "master"
• PLL of master MGT converts reference clock into common TX clock and common tx_user_clk2 which becomes tx_data_clk
• common tx_user_clk2 aka tx_data_clk goes into all TXes and clocks tx_data.
• tx_user_clk2 aka tx_data_clk is 125 MHz but not same phase as MGT reference clock.

• RX interim configuration:
• there is 12 RXes ("lanes")
• each RX produces it's own recovered RX clock
• "multilane" configuration assumes all RX recovered clocks run at the same frequency (TX on the other end are driven by common TX clock, see above), but have different phase
• one RX recovered clock is designated as "master" (rx_user_clk2 aka rx_data_clk) and a phase-matching fifo/buffer is used to bring rx_data from all 12 RXes to this common rx_data_clk
• this works because each CDM SFP TX runs on the SFP RX recovered clock which is frequency-locked with the GDM QSFP TX clock.

• RX final configuration:
• MGTs permit using the common TX clock (tx_user_clk2 aka tx_data_clk) as the common rx_data_clk (they are frequency locked through the CDM).
• this permits use of tx_data_clk as the main clock domain in the GDM and removes the need to bring rx_data into the tx_data_clk domain (actually this is done in the MGT RX phase matching fifo/buffer).

Clock path

NOTE: MUST REVIEW!!!

10 MHz ext clock or GDM 125 MHz oscillator
-> GDM QSFP MGT reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz (GDM main clock domain) and TX bit clock 2.5 GHz
-> GDM QSFP optic transmitter
-> CDM SFP optic receiver
-> CDM SFP MGT, RX reference clock is CDM 125 MHz oscillator
-> MGT RX recovered clock 125 MHz (CDM main clock domain)
-> CC_CLK_IN -> CDM C.C. -> CC_CLK_OUT1 -> CDM SFP TX reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz and TX bit clock 2.5 GHz
-> (tx_data phase matching fifo from CDM main clock domain to tx_data_clk)
-> CDM SFP optic transmitter
-> GDM QSFP RX optic receiver (12x)
-> GDM QSFP MGT (RX reference clock is same as TX reference clock)
-> MGT RX recovered clock (12x recovered clocks)
-> in multi-lane configuration, one of them is the "master" recovered clock rx_data_clk
-> (rx_data phase matching fifo from rx_data_clk to GDM main clock domain)

CDM rx_clk mux

when CDM SFP is not connected, there is no SFP recovered clock and a mux is used to switch between clk_cc_out0 (power up default) and rx_clk_mgt (SFP recovered clock)

Test SFP disconnected

note: if I say "--cc-in1", CC seems to lock on the 10 MHz GPS external clock, to prevent this, test sequence includes reloading the CC and the reset of MGT.

./test_cdm_local.exe --write32 0x30 0x0 ### rx_clk mux select CC clock
./test_cdm_local.exe --load-cc
./test_cdm_local.exe --reset-mgt
./test_cdm_local.exe --cdm-clocks
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773594e (125000014) should be ~125 MHz  <=== all 4 clocks wobble close to 125 MHz
0x1034 rx_clk:             0x0773594f (125000015) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x0773594f (125000015) should be ~125 MHz
0x103C tx_clk:             0x0773594f (125000015) should be ~125 MHz
0x1040 clk_50MHz:          0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk:         0x05f5e100 (100000000) should be 100 MHz exactly
./test_cdm_local.exe --write32 0x30 0x1 ### rx_clk mux select SFP recovered clock
./test_cdm_local.exe --cdm-clocks
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773598b (125000075) should be ~125 MHz
0x1034 rx_clk:             0x076d58ec (124606700) should be ~125 MHz  <=== off frequency because there is no valid SFP recovered clock
0x1038 mgt_tx_ref_clk_raw: 0x0773598c (125000076) should be ~125 MHz
0x103C tx_clk:             0x0773598b (125000075) should be ~125 MHz
0x1040 clk_50MHz:          0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk:         0x05f5e100 (100000000) should be 100 MHz exactly

Clock domains

GPS

• no GPS : GDM runs from internal 125 MHz oscillator
• external 10 MHz clock : GDM runs from external 10 MHz clock and optional 1pps signal (use VME-NIMIO32 NIM outputs)
• GPS receiver : GDM runs from GPS 10 MHz clock and GPS IRIG serial data
• LNGS GPS:
  • provides 1pps and serial data over fiber from GPS receiver (master). LNGS xxx box is not used.
  • serial data goes to GDM, decoded, 1pps signal extracted, goes to LEMO output
  • 1pps from GDM is used to train the Rubidium clock which provides a 10 MHz clock
  • 10 MHz output from Rubidium clock goes to GDM 10 MHz external clock input

GDM

• AXI clock (100 MHz) - AXI registers
• 125 MHz oscillator - to clock cleaner
• 10 MHz external clock LEMO input - to clock cleaner
• FPGA 125 MHz clock CLK_CC_IN - to clock cleaner - not used (could be used for cascaded GDMs)
• 125 MHz oscillator - CLK3_XO_125 - mgt_b_ref_clk QSFP MGT reference clock (interim GDM design)
• clock cleaner output 125 MHz fanout:
  • CLK_CC_OUT0 - QSFP MGT reference clock (final design)
  • CLK_CC_OUT1 - not used (CDM SFP reference clock)
  • CLK_CC_OUT2 - not used
• QSFP MGT TX data clock 125 MHz tx_data_clk (main clock domain)
  • QSFP TX data
  • QSFP RX data (in final design, rx_data_clk is same as tx_data_clk. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk == tx_data_clk)
  • ds20k block
• (in interim design, rx_data_clk is disconnected from tx_data_clk. rx_data_clk is the "multilane master clock" which is one of the 12x RX recovered clocks. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk)

note: all these clocks are frequency locked to 125 MHz

CDM

• AXI clock (100 MHz) - AXI registers
• 10 MHz external clock LEMO input - to clock cleaner (not used)
• 125 MHz oscillator to fanout
  • to clock cleaner
  • to SFP MGT RX reference clock: CLK3_XO_125 to mgt_rx_ref_clk to gtrefclk01_in
• SFP MGT RX recovered clock 125 MHz
  • MGT PLL to MGT rx_user_clk2 aka rx_data_clk
  • MGT rxrecclkout_out to mgt_rx_rec_clk to CLK_CC_IN to C.C.
• SFP MGT rx_user_clk2 aka rx_data_clk (250 MHz/8 bit, 125 MHz/16 bit, 62.5 MHz/32 bit data) (main clock domain)
  • SFP RX data
  • ds20k block
  • VX TX clock PLLs
  • VX RX clock PLLs
• C.C. fan out
  • 62.5 MHz VX clocks (12x)
  • CLK_CC_OUT0 (not used, GDM QSFP MGT reference clock)
  • CLK_CC_OUT1 125 MHz to mgt_tx_ref_clk to gtrefclk00_in to SFP MGT TX reference clock
  • CLK_CC_OUT2 (not used)
• SFP MGT tx_user_clk2 aka tx_data_clk
  • SFP TX data
  • TX data phase matching fifo from main clock domain to tx_data_clk
• VX TX clock PLLs
  • 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
  • VX TX data phase matching from main clock domain to VX TX clock (12 total)
  • VX TX serializer
  • VX TX LVDS transmitter
• VX RX clock PLLs
  • 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
  • VX RX LVDS receivers (12 total)
  • VX RX deserializers (12 total)
  • VX TX data phase matching from VX RX clock to main clock domain

VX

• everything runs on the VX main 125 MHz clock
• correct phase of VX to CDM LVDS data is adjusted by scan of CDM VX RX PLL clock phase (VX to CDM link is now established)
• correct phase of CDM to VX LVDS data is adjusted by scan of CDM VX RX PLL clock phase (link is established after VX to CDM idle data pattern changes from "VX RX data bad" to "good".
• after good phases are found by scan, they are not expected to change unless cables are changed, CDM and VX modules are changed or CDM firmware is rebuilt. (rebuild of VX firmware should not affect LVDS data phase).
• if there is excessive link errors, phase scan must be repeated.

Board test plan

To test:

• Enclustra FPGA board
• SFP port - SFP_RS0/RS1 connected to QSFP0_SEL/1_SEL is wrong? our Finisar SFP says RS0, RS1 N/C, so probably okey.
• SW5 CLK_EXT1 NIM works. TTL needs to be tested. flipping SW5 CLK_EXT0 side from NIM to TTL makes CLK_EXT1 go LOS and OOF in the clock chip. R23 and R53 should be removed?
• BOOT_MODE 0 and 1

Partial:

• U23 3.3V current meter and thermometer. V1,V2 is current monitor, same as in application note. V3,V4 is thermometer, same in application note, except capacitor C118 is 0.1u instead of 470pF. Tested ok: Tint, VCC, V1, V2, V1-V2. Test failed: TR2 reads 50-something degC instead of same as Tint.

Done:

• LED_FP1A..D: tested ok. K.O. 15 sep 2022
• USB UART: tested ok. K.O. 15 sep 2022
• J4A, J4B, J5A, J5B LEMO inputs (NIM/TTL) EXT_IN_LV(1..4). TTL threshold 1.7V, NIM threshold -0.3V. 50 Ohm termination. TTL straight, NIM inverted.
• J6A, J6B LEMO clock inputs: 10 MHz TTL from chronobox works, 10 MHz NIM from IO32 works. 18-apr-2023
• J7A, J7B LEMO outputs EXT_OUT(1), EXT_OUT(2) (NIM/TTL) (tested 23nov2022, K.O.). TTL ???, NIM inverted.
  • TTL out no 50 ohm termination: 0=0V, 1=5V, rise and fall time ~5 ns
  • TTL out with 50 ohm termination: 0=0V, 1=2.5V, rise and fall time <2ns
  • NIM out no 50 ohm termination: 0=+50mV, 1=-1.8V, rise and fall time ~3ns
  • NIM out with 50 ohm termination: 0=0V, 1=-0.9V, rise and fall time <2ns
• ethernet MAC i2c chip (K.O. 20sep2022: can read: i2cdump 0 0x5b, nov2022 have u-boot driver)
• CDM VX ports 2x(CLK, 3 tx, 4 rx) tested using test_cdm.exe and LVDS loopback in VX firmware.
• SFP i2c tested KO 22jun2023
• QSFP i2c tested KO 22jun2023
• i2c testing complete 22jun2023
• QSFP rx,tx tested 26june2023. lane0,1,2 ok, lane3 tx not connected, laser is off. qsfp0,1,2,3 all lanes ok.
• SFP rx,tx tested 26june2023. rx and tx okey. LOS ok, mod_absent ok.

Failure:

• ethernet: does not connect to alliedtelesys switch. connects to my USB-eth adapter at 100 Mbit speed. uboot mii status reports connection speed oscillating between 1000, 100 and 10. K.O. 16-sep-2022
• SFP LOS and mod_absent are swapped (in the FPGA pin definitions?)
• QSFP slot numbering is wrong.

Checklist for newly build boards

setup and boot

• put new board on workbench
• check - vme connector present, vme extraction handles present
• check - standoff are removed from all thru-holes
• plug Enclustra module
• check - SW1 both switches are in the "6" and "3" positions
• check - SW6 both switches are in the "PS" position
• Rev0: connect micro-usb cable from linux PC
• Rev1: connect usb-C cable from linux PC (USB C-to-C or C-to-A)
• connect ethernet from 1gige capable network switch
• connect power from lab power supply - GND, +5V and -12V nominal, +5.7V and -12.5V actual
• power up, +5V current 2.10-2.8A, -12V current 0.05A
• on linux PC, open a new terminal, run: minicom -D /dev/ttyACM0 -b 115200
• in minicom window, observe messages about Xilinx first stage boot loader, etc
• on the ethernet switch, observe network link is 1gige speed (not 10mbit, not 100mbit).
• if everything boots okey, there will be a login prompt, login as root, password root.
• alternatively, from root@daq13: ssh root@dsdm
• busybox devmem 0x80010000 # read firmware version number, i.e. 0xEDAD0A77

test i2c

• check i2c: (no SFP, no QSFP plugged in!)

root@dsdm:~# i2cdetect -y -r 0
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e -- 
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- -- 
70: -- -- -- -- -- -- -- --                         
root@dsdm:~# 

• test clock chip:

root@dsdm:~# ./test_cdm_local.exe --load-cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Loading CC registers...
Clock chip data block size 519
root@dsdm:~# ./test_cdm_local.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Polling CC status...
Clock chip state 1, status:  LOS_IN0 OOF_IN0 IN1 IN_SEL_1
Ctrl-C

• test RTC:

root@dsdm:~# ./test_cdm_local.exe --rtc-read
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Read the RTC chip!
reg 0x00:  0 0 1 0 0 1 1 0  0x26
reg 0x01:  0 0 1 1 1 0 0 1  0x39
reg 0x02:  0 0 0 0 0 1 1 1  0x07
reg 0x03:  0 0 0 0 0 0 1 0  0x02
reg 0x04:  0 0 0 0 0 0 1 0  0x02
reg 0x05:  0 0 0 0 0 0 0 1  0x01
reg 0x06:  0 0 0 0 0 0 0 0  0x00
reg 0x07:  0 0 0 0 0 0 0 0  0x00
reg 0x08:  0 0 0 0 0 0 0 0  0x00
reg 0x09:  0 0 0 0 0 0 0 0  0x00
reg 0x10:  0 0 0 0 0 0 0 0  0x00
reg 0x11:  0 0 0 0 0 0 0 0  0x00
reg 0x12:  0 0 0 0 0 0 0 0  0x00
reg 0x13:  0 0 0 0 0 0 0 0  0x00
reg 0x14:  0 0 0 1 1 0 0 0  0x18
reg 0x15:  0 0 0 0 1 0 0 0  0x08
reg 0x16:  0 0 0 0 0 0 0 0  0x00
reg 0x17:  0 0 0 1 0 1 1 1  0x17
reg 0x18:  0 1 0 0 0 0 0 0  0x40
seconds: 26, minutes: 39, hours: 07, day 2, date: 00-01-02, temp 23.25C
root@dsdm:~# 

test clocks

root@dsdm:~# ./test_cdm_local.exe --reset-mgt
...
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735cb9 (125000889) should be ~125 MHz
0x1034 rx_clk:             0x07735cb9 (125000889) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735cb9 (125000889) should be ~125 MHz
0x103C tx_clk:             0x07735cb9 (125000889) should be ~125 MHz
0x1040 clk_50MHz:          0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk:         0x05f5e100 (100000000) should be 100 MHz exactly
^C
root@dsdm:~# ./test_cdm_local.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735c6a (125000810) should be ~125 MHz
0x1034 rx_clk:             0x07735c69 (125000809) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735c69 (125000809) should be ~125 MHz
0x103C tx_clk:             0x07735c6a (125000810) should be ~125 MHz
0x1040 clk_50MHz:          0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk:         0x05f5e100 (100000000) should be 100 MHz exactly
^C
root@dsdm:~# 

test LED, LEMO

• test LEDs, observe all 4 LEDs turn on and off every 1 second

root@dsdm:~# ./test_cdm_local.exe --blink-led
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
LED on
LED off

• TBW - test LEMO outputs

have test firmware lemo outputs are RTC chip 1pps and 48kHz
see them on the scope

• TBW - test LEMO inputs

./test_cdm_local.exe --monreg 6

test VX

• on the CDM, connect all split blue cables to VX modules
• set all VX modules to LVDS loopback mode (in VX settings MIDAS page)
• ssh root@cdm00
• run "./test_cdm.exe --test-vx", it should report:

CDM registers: 0x0000000000000000 0x00000000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok 
CDM registers: 0x0000eeeeeeeeeeee 0x00007700: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok 
CDM registers: 0x0000080808080808 0x00000100: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok 
CDM registers: 0x0000040404040404 0x00000200: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok 
CDM registers: 0x0000020202020202 0x00000400: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok 
CDM registers: 0x0000000000000000 0x00000800: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok 
CDM registers: 0x0000808080808080 0x00001000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok 
CDM registers: 0x0000404040404040 0x00002000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok 
CDM registers: 0x0000202020202020 0x00004000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok 
CDM registers: 0x0000000000000000 0x00008000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok 
all on!
all off!

• on the VX settings MIDAS page LVDS input section, observe line status is cycling on and off.
• note: VX_RX(0) is not tested by this, corresponding VX_TX is used as the VX clock

test SFP (CDM)

• connect SFP module
• ssh root@cdm00
• run "./test_cdm.exe --sfp", it should report:

root@cdm00:~# ./test_cdm.exe --sfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x6a332b58, ds20k version 0x20241209
DS-DM firmware build 0x6a332b58, ds20k version 0x20241209
Polling SFP status...
identifier 0x03
connector  0x07
encoding   0x01
wavelength 0x0352 (850 nm)
vendor_name [FINISAR CORP.   ]
vendor_pn   [FTLF8526P3BNL   ]
vendor_rev  [A   ]
vendor_sn   [N3AB4LV         ]
vendor_date [200319  ]
dm_type    0x68
temp 30.9 C
vcc  3.285 V
tx_bias  7.362 mA
tx_power 475.7 uW
rx_power 859.1 uW
SFP good 1, status: temp 30.9 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 859 uW
SFP good 1, status: temp 30.8 C, tx_bias 7.4 mA, tx_power 475 uW, rx_power 859 uW
SFP good 1, status: temp 30.8 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 859 uW
SFP good 1, status: temp 30.7 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 859 uW
SFP good 1, status: temp 30.8 C, tx_bias 7.4 mA, tx_power 475 uW, rx_power 858 uW
^C
root@cdm00:~# 

• NOTE1: if fiber cable is disconnected or GDM is not running, rx_power would read 0 uW
• NOTE2: better SFP modules have smaller tx_bias and larger tx_power (more bang for the buck)
• NOTE3: bad fiber cable and bad fiber connection will result in smaller rx_power

test QSFP (GDM)

• connect QSFP module (any slot)
• ssh root@gdm01
• run "./test_cdm.exe --qsfp3 --qsfp", try --qsfp0, --qsfp1, --qsfp2, --qsfp3 for the 4 QSFP slots

root@gdm01:~# ./test_cdm.exe --qsfp3 --qsfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x9e30baf9, ds20k version 0x20241209
DS-DM firmware build 0x9e30baf9, ds20k version 0x20241209
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
 gpio-378 (                    |sysfs               ) out hi 
 gpio-379 (                    |sysfs               ) out hi 
 gpio-381 (                    |sysfs               ) out lo 
 gpio-382 (                    |sysfs               ) out hi 
Polling QSFP status...
identifier 0x0d
status     0x02
los        0x89
temp       36.7 C
vcc        3.302 V
rx_power     0.1 413.7 460.3   0.1 uW
tx_bias      7.8   7.9   7.9   0.0 mA
tx_power   806.0 842.9 865.2   0.1 uW
vendor_name [FINISAR CORP    ]
vendor_pn   [FTL410QD4C      ]
vendor_rev  [A ]
wavelength  850
max_temp    70 C
vendor_sn   [X79AC0R         ]
vendor_date [220309  ]
QSFP good 1, status: temp 36.7 C, los 0x89, tx_bias 7.8 7.9 7.9 0.0 mA, tx_power 806 843 865   0 uW, rx_power   0 414 460   0 uW
^C

• NOTE1: 4th tx_bias and tx_power are zero because 4th channel does not transmit any data, QSFP detects this and automatically shuts down the optical transmitter

Serial console

• check that linux computer has correct udev rules to allow access to /dev/ttyACM devices, see https://daq00.triumf.ca/DaqWiki/index.php/SLinstall#Configure_USB_device_permissions and https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Configure_USB_device_permissions
• connect micro-USB cable to connector J-UCB, other end connect to linux computer
• observe /dev/ttyACM0 was created
• run "minicom -D /dev/ttyACM0 -b 115200"
• should have gdm-cdm login
• username root, password root

i2c

ZynqMP> i2c bus
Bus 0:  i2c@ff020000
ZynqMP> i2c dev 0
Setting bus to 0
ZynqMP> i2c probe  
Valid chip addresses: 33 4E 53 5B 6B 77
ZynqMP> i2c md 0x5b 0x98
0098: fc c2 3d 00 00 12 1a 6e 0a 90 85 04 94 10 08 50    ..=....n.......P

root@gdm-cdm:~# i2cdetect 0
Warning: Can't use SMBus Quick Write command, will skip some addresses
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0.
I will probe address range 0x03-0x77.
Continue? [Y/n] 
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                                                 
10:                                                 
20:                                                 
30: -- -- -- 33 -- -- -- --                         
40:                                                 
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- -- 
60:                                                 
70:                                                 
root@gdm-cdm:~# 
root@gdm-cdm:~# i2cdump 0 0x5b
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0, address 0x5b, mode byte
Continue? [Y/n] 
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
80: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00    ???????P?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e    ........??=..??n
a0: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00    ???????P?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e    ........??=..??n
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
root@gdm-cdm:~# 

root@gdm0:~# i2cdetect -r 0
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e -- 
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- -- 
70: -- -- -- -- -- -- -- --                         
root@gdm0:~# 

root@cdm0:~# i2cdetect -r 0
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e -- 
50: 50 51 -- 53 -- -- -- -- -- -- -- 5b -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- -- 
70: -- -- -- -- -- -- -- --                         
root@cdm0:~# 

• 0x33 - XU8 secure EEPROM (should be at 0x32)
• 0x4e - U23 current and temperature monitor
• 0x50, 0x51 - SFP
• 0x50 - QSFP, 4 QSFP modules enabled by GPIO QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL
• 0x53, 0x5b - ethernet mac eeprom
• 0x6b - U6 clock chip

U23

NOTE: Text reads double of Tint. not sure why. K.O. 21-mar-2024.

• internal temperature only

root@gdm0:~# i2cset -y 0 0x4e 0x01 0x00 b  # control register: "repeat mode, internal temperature only"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x03 # "Tint ready" and "busy", "busy is always 1 in repeat mode"
root@gdm0:~# i2cdump -y 0 0x4e b
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 03 00 03 03 81 db 2a ce 2a 8e 00 6f 00 45 20 3f    ?.????*?*?.o.E ?
10: 01 00 01 01 01 db 2a ce 2a 8e 00 6f 00 45 20 3f    ?.????*?*?.o.E ?
...
readback:
reg0 - 03 - Tint ready
reg1 - 00 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB 0x81, bit 0x80 is "DV, data valid", bit 0x40 is "SS, sensor short", 0x20 is "SO, sensor open"
reg5 - Tint LSB 0xDB, Tint = 0x01DB = 475 * 0.0625 degC = 29.6 degC
reg6..F - stale data

• Tint, V1, V2, TR2, VCC

root@gdm0:~# i2cset -y 0 0x4e 0x01 0x18 b  # control register: "repeat mode, V1, V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 7f 18 7f 7f 81 b9 aa d0 aa 8f 83 5e 83 5e a0 41    ???????????^?^?A
10: 01 18 01 01 01 b9 2a d0 2a 8f 03 5e 03 5e 20 41    ??????*?*??^?^ A
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x1b9*0.0625 = 27.5 degC
reg6 - V1 MSB 0xaa, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 LSB 0xd0, V1 = 0x2ad0*305.18/1000000 = 3.3447 V (correct)
reg8 - V2 MSB 0xaa, ditto
reg9 - V2 LSB 0x8f, V2 = 0x2a8f*305.18/1000000 = 3.3249 V (correct, smaller than V1)
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x5e. TR2 = 0x35e*0.0625 = 53.875 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x41, VCC = 2.5+0x2041*305.18/1000000 = 5.019 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.99 A (about right?)

• Tint, V1-V2, TR2, VCC

root@gdm0:~# i2cset -y 0 0x4e 0x01 0x19 b  # control register: "repeat mode, V1-V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 7f 19 7f 7f 81 69 83 f2 83 f2 83 3a 83 3a a0 44    ?????i?????:?:?D
10: 01 19 01 01 01 69 03 f2 03 f2 03 3a 03 3a 20 44    ?????i?????:?: D
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x169*0.0625 = 22.5 degC
reg6 - V1 or V1-V2 MSB 0x83, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 or V1-V2 LSB 0xf2, V1-V2 = 0x3f2*19.42/1000000 = 0.0196142 V (correct, compare with V1 and V2 measured above)
reg8 - V2 or V1-V2 MSB
reg9 - V2 or V1-V2 LSB
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x3a. TR2 = 0x33a*0.0625 = 51.625 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x44, VCC = 2.5+0x2044*305.18/1000000 = 5.021 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.9807 A (about right?)

SFP

root@cdm0:~# i2cdump 0 0x50
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 03 04 07 00 00 00 00 40 40 0c 00 01 3d 00 00 00    ???....@@?.?=...
10: 0c 02 00 1e 46 49 4e 49 53 41 52 20 43 4f 52 50    ??.?FINISAR CORP
20: 2e 20 20 20 00 00 90 65 46 54 4c 46 38 35 32 36    .   ..?eFTLF8526
30: 50 33 42 4e 4c 20 20 20 41 20 20 20 03 52 00 9d    P3BNL   A   ?R.?
40: 00 1a 00 00 4e 33 41 42 34 4c 56 20 20 20 20 20    .?..N3AB4LV     
50: 20 20 20 20 32 30 30 33 31 39 20 20 68 f0 03 de        200319  h???
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
root@cdm0:~# 
root@cdm0:~# i2cdump 0 0x51
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 5a 00 d8 00 55 00 e2 00 90 88 71 48 8c a0 75 30    Z.?.U.?.??qH??u0
10: 21 34 01 f4 1b 58 03 e8 31 2d 04 eb 1f 07 06 31    !4???X??1-?????1
20: 31 2d 00 64 27 10 00 9e 00 00 00 00 00 00 00 00    1-.d'?.?........
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
40: 00 00 00 00 3f 80 00 00 00 00 00 00 01 00 00 00    ....??......?...
50: 01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 b7    ?...?...?......?
60: 19 9f 80 c5 0e 17 12 c4 1f 99 00 00 00 00 30 00    ??????????....0.
70: 00 00 00 00 00 00 00 00 ff ff ff ff ff ff ff 01    ...............?
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
root@cdm0:~# 

QSFP

QSFP i2c enable lines, active low:

QSFP0_SEL - JENC-A 82 - PS-MIO40 - linux gpio 378
QSFP1_SEL - JENC-A 84 - PS-MIO41 - linux gpio 379
QSFP2_SEL - JENC-A 100 - PS-MIO44 - linux gpio 382
QSFP3_SEL - JENC-A 106 - PS-MIO43 - linux gpio 381 (notice 2 and 3 are out of order)

# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
# echo 378 >> /sys/class/gpio/export ### SEL0 338+40
# echo 379 >> /sys/class/gpio/export ### SEL1 338+41
# echo 381 >> /sys/class/gpio/export ### SEL3 338+43
# echo 382 >> /sys/class/gpio/export ### SEL2 338+44
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
 gpio-378 (                    |sysfs               ) in  hi 
 gpio-379 (                    |sysfs               ) in  hi 
 gpio-381 (                    |sysfs               ) in  hi 
 gpio-382 (                    |sysfs               ) in  hi 
root@gdm0:~# echo out >> /sys/class/gpio/gpio381/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio382/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio378/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio379/direction
root@gdm0:~# 
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
 gpio-378 (                    |sysfs               ) out lo 
 gpio-379 (                    |sysfs               ) out lo 
 gpio-381 (                    |sysfs               ) out lo 
 gpio-382 (                    |sysfs               ) out lo 
root@gdm0:~#
echo 1 >> /sys/class/gpio/gpio381/value
echo 1 >> /sys/class/gpio/gpio382/value
echo 1 >> /sys/class/gpio/gpio378/value
echo 1 >> /sys/class/gpio/gpio379/value
cat /sys/kernel/debug/gpio
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
 gpio-378 (                    |sysfs               ) out hi 
 gpio-379 (                    |sysfs               ) out hi 
 gpio-381 (                    |sysfs               ) out hi 
 gpio-382 (                    |sysfs               ) out hi 
root@gdm0:~# 
root@gdm0:~# i2cdetect -y -r 0
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e -- 
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- -- 
70: -- -- -- -- -- -- -- --                         
root@gdm0:~# 

NOTICE NOTHING AT ADDRESS 0x50

ENABLE QSFP0, OBSERVE IT IS AT ADDRESS 0x50

root@gdm0:~# echo 0 >> /sys/class/gpio/gpio378/value
root@gdm0:~# i2cdetect -y -r 0
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e -- 
50: 50 -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- -- 
70: -- -- -- -- -- -- -- --                         
root@gdm0:~# 
root@gdm0:~# i2cdump -y 0 0x50
No size specified (using byte-data access)
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 0d 00 02 8e 00 00 01 00 00 05 55 00 05 00 00 00    ?.??..?..?U.?...
10: 00 00 00 00 00 00 1e b8 00 00 81 a0 00 00 00 00    ......??..??....
20: 00 00 12 12 00 01 00 01 00 01 0e e0 0f 20 0e e0    ..??.?.?.???? ??
30: 00 00 1f a0 1e 3b 1f 72 00 01 00 00 00 00 00 00    ..???;?r.?......
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
60: 00 00 00 00 00 00 00 00 00 00 1f 00 00 00 08 00    ..........?...?.
70: 00 00 00 00 00 00 00 ff ff ff ff ff ff ff ff 00    ................
80: 0d 00 0c 04 00 00 00 40 40 02 d5 05 67 00 00 96    ?.??...@@???g..?
90: 00 00 c8 00 46 49 4e 49 53 41 52 20 43 4f 52 50    ..?.FINISAR CORP
a0: 20 20 20 20 07 00 90 65 46 54 4c 34 31 30 51 44        ?.?eFTL410QD
b0: 34 43 20 20 20 20 20 20 41 20 42 68 07 d0 00 43    4C      A Bh??.C
c0: 00 07 0f de 58 37 39 41 43 30 52 20 20 20 20 20    .???X79AC0R     
d0: 20 20 20 20 32 32 30 33 30 39 20 20 3c 10 00 9e        220309  <?.?
e0: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20                    
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
root@gdm0:~# 

ethernet mac eeprom

• correct chip with 84-bit ethernet mac address

root@cdm1:~# i2cdump 0 0x53
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
root@cdm1:~# 

root@cdm1:~# i2cdump 0 0x5b
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
80: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00    ????4??2?K?.?...
90: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c    ..........??=?Q<
a0: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00    ????4??2?K?.?...
b0: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c    ..........??=?Q<
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
root@cdm1:~# 

• wrong "602" chip with 64-bit IPv6 address

root@cdm0:~# i2cdump 0 0x53
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
root@cdm0:~# i2cdump 0 0x5b
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
80: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00    ???????Q?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e    ........??=..??.
a0: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00    ???????Q?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e    ........??=..??.
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
root@cdm0:~# 

read ethernet mac address from i2c

(this code is copied from uboot command line i2c code)

in uboot sources board/xilinx/common/board.c replace original function with this:

// special code to read ethernet MAC address from the DS-DM-Rev0 board. K.O. Sep-2022

int zynq_board_read_rom_ethaddr(unsigned char *ethaddr)
{
        struct udevice *bus;
	int ret;
        int busnum = 0;

	ret = uclass_get_device_by_seq(UCLASS_I2C, busnum, &bus);
	if (ret) {
           printf("%s: No bus %d\n", __func__, busnum);
           return ret;
	}

        int chip_addr = 0x5B;

        struct udevice *dev;

        ret = i2c_get_chip(bus, chip_addr, 1, &dev);
	if (ret) {
           printf("%s: Bus %d no chip 0x%02x\n", __func__, busnum, chip_addr);
           return ret;
	}

        int dev_addr = 0x98;

        unsigned char data[8];
           
        ret = dm_i2c_read(dev, dev_addr, data, 8);

	if (ret) {
           printf("%s: Bus %d chip 0x%02x read error %d\n", __func__, busnum, chip_addr, ret);
           return ret;
	}

        printf("%s: Bus %d chip 0x%02x addr 0x%02x read: 0x %02x %02x %02x %02x %02x %02x %02x %02x\n", __func__, busnum, chip_addr, dev_addr, data[0], data[1], data[2], data[3], data[4], data[5], data[6], data[7]);

        // see Atmel-8807-SEEPROM-AT24MAC402-602-Datasheet.pdf

        if (data[0] == 0) {
           // eiu-48 chip
           ethaddr[0] = data[2];
           ethaddr[1] = data[3];
           ethaddr[2] = data[4];
           ethaddr[3] = data[5];
           ethaddr[4] = data[6];
           ethaddr[5] = data[7];
        } else {
           // eiu-64 chip
           ethaddr[0] = data[0];
           ethaddr[1] = data[1];
           ethaddr[2] = data[2];
           ethaddr[3] = data[5];
           ethaddr[4] = data[6];
           ethaddr[5] = data[7];
        }

        printf("%s: ethaddr %02x:%02x:%02x:%02x:%02x:%02x\n", __func__, ethaddr[0], ethaddr[1], ethaddr[2], ethaddr[3], ethaddr[4], ethaddr[5]);

        return ret;
}

also this should have worked if i2c_xxx() functions were enabled in uboot:

i2c_set_bus_num(0);
i2c_probe(0x5b);
i2c_read(0x5b, 0x9a, ethaddr, 6);

read ethernet mac address from i2c (SHOULD WORK)

from: https://stackoverflow.com/questions/43637540/is-there-linux-or-u-boot-support-to-read-a-mac-address-from-a-chip-at-startup

#ethernet related setup
setup_eth=run readmac buildmac
#read mac address from eeprom
readmac=i2c dev 0; i2c read 50 FA.1 6 $loadaddr
#build the ethaddr variable
#not very nice, but does the job
buildmac=\
e=" "; sep=" " \
for i in 0 1 2 3 4 5 ; do\
setexpr x $loadaddr + $i\
setexpr.b b *$x\
e="$e$sep$b"\
sep=":"\
done &&\
setenv ethaddr $e

read ethernet mac address from i2c (DOES NOT WORK)

this method does not work: inside board/xilinx/common/board.c:int zynq_board_read_rom_ethaddr(unsigned char *ethaddr), uclass_get_device_by_ofnode(UCLASS_I2C_EEPROM, eeprom, &dev) returns ENODEV and read from i2c does not happen. K.O. Sep-2022

Read:

• https://support.xilinx.com/s/question/0D52E00006hpKlsSAE/uboot-and-ethernet-mac-address-from-eeprom?language=en_US (last answer)
• https://github.com/Xilinx/u-boot-xlnx/blob/master/board/xilinx/common/board.c (zynq_board_read_rom_ethaddr())

Note:

• 0x5B is the i2c chip address
• 0x9A is the data offset inside the chip, see datasheet or i2c read dump above.

Edit:

• emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/u-boot/files/platform-top.h

#include <configs/xilinx_zynqmp.h>
#include <configs/platform-auto.h>
//#define CONFIG_I2C_EEPROM                                                                                                                                                                   
//#define CONFIG_SYS_I2C_EEPROM_ADDR 0x5b                                                                                                                                                     
//#define CONFIG_SYS_I2C_EEPROM_ADDR_OVERFLOW 0x0                                                                                                                                             
#define CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET 0x9A
#error HERE!

• emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi

/include/ "system-conf.dtsi"
/ {
chosen {
   xlnx,eeprom = &eeprom;
};
};

&i2c0 {
eeprom: eeprom@5b { /* u88 */
compatible = "atmel,24mac402";
reg = <0x5b>;
};
};

• components/yocto/workspace/sources/u-boot-xlnx/configs/xilinx_zynqmp_virt_defconfig

CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET=0x9A

RTC chip

• DS3231 RTC chip
• FPGA connections:

I2C SCL <-  J-ENC A85 <-  FPGA E17 <-  XDC TP_S <-  VHDL RTC_I2C_SCL output (10k pull-up to 3.3V)
I2C SDA <-> J-ENC A87 <-> FPGA D17 <-> XDC TP_S <-> VHDL RTC_I2C_SDA bidir (10k pull-up to 3.3V)
RTC_1Hz  -> J-ENC C160 -> FPGA AH12 -> XDC "slow_io"  -> VHDL RTC_1PPS input (10k pull-up to 1.2V)
RTC_32k  -> J-ENC B129 -> FPGA AE3  -> XDC "free pin" -> VHDL RTC_32KHZ input (10k pull-up to 1.8V)

• test notes

FPGA registers:

  rtc_i2c_clock_ie  <= not register_data_in_block_0(14)(0);
  rtc_i2c_clock_out <= register_data_in_block_0(14)(1);
  rtc_i2c_data_ie   <= not register_data_in_block_0(14)(4);
  rtc_i2c_data_out  <= register_data_in_block_0(14)(5);

  ext_out(1) <= rtc_1pps_in;
  ext_out(2) <= rtc_32khz_in;

  register_data_out_block_0(13) <= x"000000" & rtc_i2c_data_ie & rtc_i2c_data_out & rtc_i2c_clock_ie & rtc_i2c_clock_out & rtc_1pps_in & rtc_32khz_in & rtc_i2c_data_in & rtc_i2c_clock_in;

After power us on the RTC chip:
  i2c clock 3.3V
  i2c data  3.3V
  32kHz     0.778V see clock running with amplitude 0->1.8V (FPGA pullup) period about 30 usec (~33 kHz)
  1pps      1.2V (FPGA pullup)

./test1.exe --read32 0x34
dsdm_read32[0x00000034] is 0x000000ab (171) -> 1010'1011 -> data_ie, clock_ie, 1pps, data_in, clock_in
./test1.exe --read32 0x38
dsdm_read32[0x00000038] is 0x00000000 (0)

./test1.exe --write32 0x38 0x01 -> clock oe
./test1.exe --read32 0x34
Ddsdm_read32[0x00000034] is 0x0000008a (138) -> 1000'1010 -> data_ie, 1pps, data_in, clock 0V, data 3.3V

./test1.exe --write32 0x38 0x10 -> data_oe
./test1.exe --read32 0x34
dsdm_read32[0x00000034] is 0x00000029 (41) -> 0010'1001 -> clock_ie, 1pps, clock_in, clock 3.3V, data 0V

root@dsdm:~# ./test_cdm_local.exe --rtc-read
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Read the RTC chip!
reg 0x00:  0 1 0 1 0 1 1 1  0x57
reg 0x01:  0 0 1 0 0 0 1 0  0x22
reg 0x02:  0 0 0 0 0 0 0 0  0x00
reg 0x03:  0 0 0 0 0 0 0 0  0x00
reg 0x04:  0 0 0 0 0 0 0 0  0x00
reg 0x05:  0 0 0 0 0 0 0 0  0x00
reg 0x06:  0 0 0 0 0 0 0 0  0x00
reg 0x07:  0 0 0 0 0 0 0 0  0x00
reg 0x08:  0 0 0 0 0 0 0 0  0x00
reg 0x09:  0 0 0 0 0 0 0 0  0x00
reg 0x10:  0 0 0 0 0 0 0 0  0x00
reg 0x11:  0 0 0 0 0 0 0 0  0x00
reg 0x12:  0 0 0 0 0 0 0 0  0x00
reg 0x13:  0 0 0 0 0 0 0 0  0x00
reg 0x14:  0 0 0 1 1 0 0 0  0x18
reg 0x15:  0 0 0 0 1 0 0 0  0x08
reg 0x16:  0 0 0 0 0 0 0 0  0x00
reg 0x17:  0 0 0 1 0 1 1 1  0x17
reg 0x18:  1 1 0 0 0 0 0 0  0xc0
seconds: 57, minutes: 22, hours: 00, day 0, date: 00-00-00, temp 23.75C
root@dsdm:~# 

clock chip configuration

file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt:

• VCO is 14 GHz
• Tvco is 71.43 ps
• N0 divider is 14, frequency is 1000 MHz
• out0 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz
• out1 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
• out2 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
• out3 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz is the feedback for zero delay

clock chip monitoring

from si5395-94-92-family.pdf:

• reg 0x1: page select, set to 0 or set to 5 to read 0x53F
• reg 0x2: 0x94
• reg 0x3: 0x53 -> device is a si5394
• reg 0xC: LOSXAXB
• reg 0xD: LOS and OOF for the 4 clock inputs
• reg 0xE: LOL and HOLD
• reg 0xF: CAL_PLL
• reg 0x11: sticky bits for reg 0xC
• reg 0x12: sticky bits for reg 0xD
• reg 0x13: sticky bits for reg 0xE
• reg 0x14: sticky bits for reg 0xF
• reg 0x1C: device reset
• reg 0x1E: low power, hard reset, SYNC
• reg 0x507: currently selected input clock
• reg 0x52A: input clock select
• reg 0x535: FORCE_HOLD
• reg 0x53F: HOLD_HIST_VALID and FASTLOCK_STATUS

Install Xilinx tools

• install Vivado 2020.2

login at https://www.xilinx.com/myprofile.html
go to "Downloads"
go to archive,
find 2020.2
download Xilinx_Unified_2020.2_1118_1232_Lin64.bin
sh ./Xilinx_Unified_2020.2_1118_1232_Lin64.bin
banner window should open with spinner "downloading installation data"
"a newer version is available" -> say "continue"
next
"select install type" window:
provide email and password,
select "download image"
select directory /home/olchansk/Xilinx/Downloads/2020.2\
select "linux" and "full image"
next
download summary: space required 38.52 Gbytes
download
installation progress
downloading spinner, 16 M/s 47 minutes...
"download image has been created successfully". Ok.
check contents of /home/olchansk/Xilinx/Downloads/2020.2
ls -l /home/olchansk/Xilinx/Downloads/2020.2
total 67
drwxr-xr-x 2 olchansk users    9 Sep  1 16:22 bin
drwxr-xr-x 3 olchansk users   15 Sep  1 16:23 data
drwxr-xr-x 4 olchansk users    4 Sep  1 16:22 lib
drwxr-xr-x 2 olchansk users  644 Sep  1 16:22 payload
drwxr-xr-x 2 olchansk users    7 Sep  1 16:22 scripts
drwxr-xr-x 4 olchansk users    4 Sep  1 16:22 tps
-rwxr-xr-x 1 olchansk users 3256 Nov 18  2020 xsetup
daq13:2020.2$ 
./xsetup
spinned loading installation data
xilinx design tools 2022.1 now available -> say continue
"welcome" -> next
"select product" -> vivado -> next -> vivado hl system edition -> next
select devices: only zynq ultrascale+ mpsoc -> next
select destination: /opt/Xilinx (as root, mkdir /opt/Xilinx, chmod olchansk.users /opt/Xilinx)
install ...
complete
move /home/olchansk/Xilinx/Downloads/2020.2 to /daq/daqstore/olchansk/Xilinx/Downloads/

• install petalinux 2020.2

./xsetup
"a newer version is available" -> say "continue"
next
"select product to install" -> select Petalinux (Linux only) -> next
"select destination directory" -> select "/opt/Xilinx" (disk space required 2.64 GB) -> next
"summary" -> install ...
error about missing /tmp/tmp-something files
"installation completed successfully" (hard to dismiss, "ok" button is partially cut-off)
done?
I think it failed, /opt/Xilinx/PetaLinux/2020.2/bin is empty except for petalinux-v2020.2-final-installer.run
try to run it by hand, same error about /tmp/tmp-something files. strange...
notice it complains about "truncate", which truncate finds ~/bin/truncate, get rid of it,
try again
now complains about missing texinfo and zlib1g:i386
apt install texinfo -> ok
apt install zlib1g:i386 -> installs bunch of gcc stuff -> ok
try again
reports "already installed" -> delete /opt/Xilinx/.xinstall/PetaLinux_2020.2/, delete entries in ~/.Xilinx/registry/installedSW.xml
try again
success

• install vivado 2022.1 and petalinux 2022.1 - everything is pretty much the same

Petalinux

• cd PetaLinux_GDM_CDM
• petalinux-config
• enable i2c MAC address and DHCP

git clone https://bitbucket.org:/team-ds-dm/ds-dm-u-boot-xlnx.git
cd ds-dm-u-boot-xlnx
git checkout ds-dm-u-boot-xlnx

linux-components ->
uboot -> ext-local-src
external u-boot local source -> ds-dm-u-boot-xlnx (path to the customized uboot git repository)

• enable DHCP

Subsystem AUTO Hardware Settings -> Ethernet Settings
randomize MAC address -> NO
ethernet mac address -> leave empty
obtain ip address automatically -> YES

• set hostname and product names

Firmware Version Configuration ->
Host name -> "ds-dm"
Product name -> "Petalinux_GDM_CDM"

• configure linux kernel

petalinux-config -c kernel

• enable NFS-Root

petalinux-config
Image Packaging Configuration > Root File System Type -> set to NFS
Location of NFS root directory set to "/nfsroot"

petalinux-config -c kernel
Networking support > IP: kernel level configuration
enable DHCP, BOOTP, RARP
File systems > Network file systems > Root file systems on NFS

• manually fix linux kernel command line:

grep nfsroot PetaLinux_GDM_CDM/project-spec/configs/config
edit CONFIG_SUBSYSTEM_BOOTARGS_GENERATED to read
earlycon console=ttyPS0,115200 clk_ignore_unused panic=60 root=/dev/nfs nfsroot=/nfsroot/%s ip=dhcp rw

• check configuration in
  • PetaLinux_GDM_CDM/project-spec/configs/config
  • PetaLinux_GDM_CDM/project-spec/configs/rootfs_config
  • PetaLinux_GDM_CDM/components/plnx_workspace/device-tree/device-tree/system-conf.dtsi

JTAG server

localhost:3121

ds20k block

module ds20k
  (
   //    CLOCK INPUTs
   input wire clk,
   input reset, // pulse for power-up reset
   input wire pll_is_locked, // clock cleaner PLL is locked to selected input clock

   //    REGISTER_DATA
   
   input wire [255:0] [31:0] register_data_in,
   output reg [255:0] [31:0] register_data_out,
   input wire register_write_strobe, // pulse when AXI write transaction puts new data in register_data_in
   input wire register_read_ack, // pulse after AXI read transaction captures data from register_data_out, used to read from FIFO
   
   //    GDM QSFP FIBER LINKS
   output reg [11:0] [15:0] qsfp_tx_data,
   output reg [11:0] [1:0]  qsfp_tx_ctrl,
   input wire [11:0] [15:0] qsfp_rx_data_error, // not sure what this is.
   input wire [11:0] [15:0] qsfp_rx_data,
   input wire [11:0] [1:0]  qsfp_rx_ctrl,
   input wire [11:0] qsfp_rx_is_good, // single bit indicating that RX link is up and data is good.

   // CDM SFP FIBER LINKS
   output reg [15:0] sfp_tx_data,
   output reg [1:0]  sfp_tx_ctrl,
   input wire [15:0] sfp_rx_data,
   input wire [1:0]  sfp_rx_ctrl,
   input wire        sfp_rx_is_good, // single bit indicating that RX link is up and data is good.
      
   //    VX_RXs
   input wire [3:0] vx1_rx,
   input wire [3:0] vx2_rx,  
   input wire [3:0] vx3_rx,  
   input wire [3:0] vx4_rx,  
   input wire [3:0] vx5_rx,  
   input wire [3:0] vx6_rx,  
   input wire [3:0] vx7_rx,  
   input wire [3:0] vx8_rx,
   input wire [3:0] vx9_rx,  
   input wire [3:0] vx10_rx,    
   input wire [3:0] vx11_rx,
   input wire [3:0] vx12_rx,
   
   //    VX_TXs
   output reg [2:0] vx1_tx_out,
   output reg [2:0] vx2_tx_out,
   output reg [2:0] vx3_tx_out,
   output reg [2:0] vx4_tx_out,
   output reg [2:0] vx5_tx_out,
   output reg [2:0] vx6_tx_out,
   output reg [2:0] vx7_tx_out,
   output reg [2:0] vx8_tx_out,
   output reg [2:0] vx9_tx_out,
   output reg [2:0] vx10_tx_out,
   output reg [2:0] vx11_tx_out,
   output reg [2:0] vx12_tx_out,

   // remove input wire gdm_trg,
   // remove input wire gdm_tsm,
   
   //    LEMO INPUTs
   input wire [4:1] ext_in_lv_async, // direct connection to LEMO connectors, not clocked
    
   //    LEMO OUTPUTs
   output reg [2:1] ext_out, // direct connection to LEMO connectors, not clocked
    
   //    FRONT PANEL LEDs
   output reg [3:0] fp_led_out, // direct connection to LEMO connectors, not clocked

   //    trigger and tsm output
   // remove output reg trg_out,
   // remove output reg tsm_out
   );

world view

Main components:

• clock distribution - to ensure all digitizers run synchronously and waveform timestamp are easy to assemble intophysics events.
• global trigger and run control - all digitizers are triggered at the same time to record calibration events, pulser events and the run startup synchronization sequence.
• busy distribution - when some digitizers go busy and stop accepting triggers, physics events may become incomplete, this must be recorded and managed.
• hitmap distribution - to create an online picture of the detector for monitoring and for triggering.

global clock distribution

Clock distribution ensures all digitizers run synchronously (on the same clock) and waveform timestamps are easy to assemble into physics events. The clock source is synchronized with GPS to ensure a metrologically validated clock frequency (tied to the SI definition of 1 second) and date stamp (via GPS IRIG-B date stamp). Individual digitizer waveform timestamp are reset at the begin of run to establish a valid time zero (run start time, date stamped with the GPS IRIG-B date stamp).

Base clock is 125 MHz (8 ns). Individual digitizers are synchronized at initialization (power-up) time. Because of incontrollable delays in the electronics (i.e. variations between individual PLL chips in each digitizer), waveform timestamps between individual digitizers may be off by 1-2 clocks (8-16 ns). After the system is running and is stable (thermally and electrically), timestamp waveform variation and jitter are less than 1 ns.

This is validated by feeding a common calibration pulser signal into analog inputs of each digitizer (at 100 Hz) and measuring (fitting) the pulse time in the waveform. With sufficient statistics, differences between waveform timestamp are measured with picosecond precision.

CDM PLL allows fine phase adjustment of the 62.5 MHz VX2746 clock in groups of 6 clocks (2 groups per CDM) with precision 71.43 ps.

Due to FPGA technology limitations (serial link hard IPs, PLLs, etc) the pattern of 1-2 clock delays between individual digitizers may change/shift from power up to power up. Significant additional work would be required to ensure this pattern is repeatable and stable after every power up and initialization. As mentioned before, after power up and initialization, variation between waveform timestamp is sub-nanosecond as measured by the analog calibration pulser.

Clock chain:

• GPS receiver (1pps clock reference and IRIG-B date stamp)
• LNGS GPS clock distribution (fiber link to converter box at the experiment)
• GPS 1pps to PRS-10 Rb atomic clock (10 MHz clock output)
• 10 MHz clock from PRS-10 to GDM PLL (extra low jitter oscillator, 125 MHz clock output)
• 125 MHz clock to GDM fiber link transmitters (2.5 Gbps line rate)
• CDM fiber link receiver and clock recovery (125 MHz clock)
• 125 MHz recovered clock to CDM PLL (extra low jitter oscillator, 62.5 MHz clock output)
• 62.5 MHz CDM clock to VX2745 PLL via LVDS line
• VX2745 PLL (125 MHz clock output)
• 125 MHz clock to digitizer chips, clock counter and waveform timestamp.

This is the normal mode of operation, with global clock synchronized with the GPS clock (metrologically referenced to SI 1 Hz definition).

If GPS signal is not available, degraded modes of operation are available (in order of worsening performance)

• PRS-10 Rb atomic clock synchronized with a battery-powered time-of-day chip
• PRS-10 Rb atomic clock synchronized with Internet time (NTP) assuming Internet network connection is up
• PRS-10 Rb clock standalone
• GDM internal oscillator

All degraded modes provide high precision low jitter 125 MHz system clock, but long term stability (seconds per day) will vary. (TBM)

In the absence of GPS IRIG-B date stamp, the battery-powered time-of-day chip will be used (TBI) with fallback to Internet time (NTP). Date stamp long term stability is expected to be better than 1 second per day (TBI, TBM).

Note:

• TBM: to be measured
• TBI: to be implemented

Current status (April 2025):

• full clock chain from GDM PLL to VX2745 digitizers is implemented and tested
• waveform timestamp variations are measured (see report from Marek)
• GDM external clock is validated (using VME-GRIFFIN-CDM high quality clock source module)
• PRS-10 10 MHz and 1pps connection to GDM is implemented and tested
• GPS receiver 1pps connection to PRS-10 is implemented and tested
• GPS receiver IRIG-B date stamp connection to GDM tested using a software decoder. Hardware decoder and connection to time slice marker, TBI.
• LNGS date stamp decoder and connection to time slice marker, TBI
• LNGS 1pps connection to PRS-10, not available
• operate vertical slize system from GPS, not available (GPS is in the det lab, vertical slice is in the daq lab).

global trigger distribution

Global trigger distribution provides a synchronous waveform trigger to all VX2745, all CDM and all VX2745 are provided wiht the same data and are all triggered at the same time (without accounting for the 1-2 clock offset described in the clock section).

Global trigger is used for run control: the first trigger of a run resets the waveform timestamp counter, the CDM timestamp counter and the GDM timestamp counter. At the begin of run, GDM generates a sequence of triggers following an easy to identify pattern. If a VX2745 misses a trigger (for any reason), online and offline software can easily identify the tru run start time using this pattern.

After a run has started, global trigger can be generated by a programmable pulser, by an external LEMO signal (i.e. external calibration signal, laser Q-switch, etc) and by the VX2745 trigger hitmap.

Trigger sources (TRG):

• TRG pulser
• LEMO inputs
• hitmap trigger

Time slice marker sources (TSM):

• TSM pulser

Trigger chain, K-code and LVDS:

• TRG/TSM source
• TRG/TSM K-code from GDM to CDM
• CDM K-code to LVDS line 12 and 13 lines (TRG and TSM)

Latency from GDM LEMO input to CDM LVDS out: TBM

Trigger chain, packet trigger:

• GDM TRG/TSM source to TRG/TSM packet generator
• or GDM hitmap group trigger packet generator
• TRG/TSM packet from GDM to CDM
• TRG/TSM packet from CDM to VX2745 via LVDS line, 62.5 MHz, 8b10b encoding.

Latency from GDM LEMO input to VX2745 internal trigger: TBM

Notes: TBM - to be measured TBI - to be implemented

Current status (April 2025):

• LVDS TRG and TSM distribution is done and tested
• packet TRG and TSM distribution from GDM to CDM is done and tested
• packet TRG and TSM from CDM to VX2745 under testing
• GDM hitmap group trigger packet generator, TBI
• interlock between TRG packet, hitmap group trigger packet and TSM, TBI

global busy distribution

If any VX2745 goes busy (for any reason) and stops accepting triggers, physics data may become incomplete. This situation needs to be recorded and managed.

Busy chain:

• VX2745 firmware busy logic (CAEN and Yair) generates the VX busy signal on LVDS line 9 (CDM VX_RX_2)
• CDM monitors VX busy
• CDM records busy transitions (TBI)
• CDM computes the "CDM busy" as grand-OR of all VX busy
• CDM periodically transmits CDM busy status the GDM using special K-codes (NB: confirm transmission period)
• GDM receives CDM busy updates and maintains "CDM busy" for each CDM
• GDM records busy transitions (TBI)
• GDM computes "GDM busy" as grand-OR of all CDM busy
• effectively, GDM busy is a grand-OR of all VX busy (with max delay, TBM)

In the pilot implementation, any VX busy stops the experiment by sending a veto signal to all VX2745 which stops them from accepting triggers.

Veto chain:

• GDM computes "GDM veto" as GDM busy
• GDM periodically transmits GDM veto status to all CDMs using special K-codes (NB: confirm transmission period)
• CDM receive GDM veto updates and maintain the "CDM veto" signal
• CDM computes "VX veto" as CDM veto
• CDM transmits VX veto to all VX2745 on LVDS pair 13 (TBI, pair 13 is currently used for time slice markers)
• VX2745 firmware (Yair) has VX veto block all physics triggers (NB: except for the time slice marker?)

Total delay from VX busy to VX veto (on the LVDS lines), TBM.

Notes:

• TBM - to be measured
• TBI - to be implemented

Current status (April 2025):

• full busy chain from VX to GDM is implemented, testing in progress
• full veto chain from GDM to VX veto is implemented, tested
• CDM veto to VX2745 veto, TBI (TSM must switch to TSM packet and liberate LVDS pair 13)
• full test awaiting for a method of artificially cause a VX2745 busy, TBI

hitmap and hitmap trigger

To monitor the system in real time and to generate group triggers, we use the VX hitmaps.

Each VX2745 has 64 channels, each channel has a trigger logic block (digital discriminator) that generates a waveform self-trigger and sends a hitmap bit to the CDM (64-bits per VX2745) as a data packet. CDM passes hitmap data packets to the GDM and also records non-empty hitmap packets. CDM MIDAS frontend passes non-empty hitmap packets to MIDAS for online monitoring and visualization (TBI). GDM collects all the hitmaps and generates a global trigger. Alternatively, it generates a group trigger (sends a 48-bit bitmap of which VX2745 should trigger a waveform), TBI.

Pilot implementation computes the hitmap trigger as a grand-OR of all hitmaps.

Hitmap chain:

• VX2745 analog inputs to digitizers to digital trigger logic block, 1 bit per channel (CAEN, Yair)
• 64-bits of hitmap (1 bit/channel) are collected and periodically transmitted to the CDM (Sam De Jong) (NB: confirm transmission period)
• VX hitmap packets are transmitted to the CDM on the LVDS line 10 (CDM VX_RX_1) at 62.5 MHz, 8b10b encoded.
• CDM receives hitmap packets (12 input data streams, 12-to-1 packet mux)
• CDM sends all hitmap packets to the GDM at 125 MHz, 16b20b encoded.
• non-empty hitmap packets are written to the CDM-to-CPU data FIFO. (to be read by MIDAS frontend, TBI)
• grand-OR of VX hitmap packets (12*64 bits) can generate a CDM-local VX trigger
• GDM receives hitmap packets, computes "GDM hitmap trigger" (TBI)
• if enabled, GDM hitmap trigger causes GDM trigger, which is processed as described in the global trigger section

Delay, latency and jitter from analog pulse on the VX2745 analog input to VX trigger on LVDS line 12, TBM.

Hitmap group trigger (TBI):

• GDM receives hitmap packets, computes the 48-bit bitmap of which VX2745 should trigger
• how to specify this computation is not clear. a generic 12*64 MLU is impossible, insufficient RAM on the FPGA
• 48-bit bitmap is sent to all CDMs as a data packet
• CDM receives data packets from GDM and retransmits them to the VX2745 (all CDMs and all VXes see the same packets at the same time).
• VX2745 receives the 48-bit bitmap, checks if it's bit is set and triggers waveform readout of all channels (there will be no more than 48 VX2745 in the experiment) (TBI, Yair)

Notes:

• TBM - to be measured
• TBI - to be implemented

Current status (April 2025):

• VX hitmap packet generator, testing in progress (Yair firmware)
• hitmap packet from VX to CDM to GDM, implemented, tested
• CDM hitmap processing, implemented, tested
• GDM hitmap processing, TBI
• hitmap group trigger, TBI (GDM: KO, CDM: done, VX2745: Yair)
• MIDAS frontend readout of hitmap data from FPGA FIFO: TBI (FIFO code: Ian, done; integration and C++ code: KO)
• online monitoring and visualization in MIDAS: TBI

firmware map

Note:

• red lines: clocks
• green lines: AXI/Avalon packet streams
• blue lines: serial data

description

same thing, in words:

detector
digitizer, 125 MHz
digital filter
digital discriminator
hit map, 64 bits at 125 MHz (could be 250 MHz, filter and discriminator clock)
packetizer, 64 bits -> id, timestamp, 8x 8-bit words, eop
8/10 serializer, 12.5 MHz parallel in, 125 MHz serial out
lvds line to CDM vx_rx
BBB: also VX busy to lvds line to CDM vx_rx, do not want to depend on serial comm for vx busy, do not want to inject async data into the hit map packet stream
--- CDM
lvds line capture in IBUF register, vxN_rx clock, 125 MHz clock, from PLL with adjustable phase, scan phase to find sweet spot
vx rx 10/8 deserializer, vxN_rx clock, stobes out parallel data every 10 clocks at 12.5 MHz
vx rx phase transfer from vxN_rx clock to main CDM clock, have 10 clocks for transfer to happen
from here everything is on the CDM main clock
vx rx packet adapter, 12.5 MHz 8-bit packets to 125 MHz 16-bit packets, internal FIFO to avoid data overrun
sfp tx mux, all VX packet streams into one CDM sfp tx packet stream
BBB: capture VX RX busy to CDM main clock (IBUF register), grand-or becomes vx_rx_busy
sfp tx packetizer (data,eop,vx_rx_busy -> data,k)
sfp tx 16/20 serializer, 16 bit at 125 MHz in, 2500 MHz serial out (20 bits at 125 MHz)
sfp tx fiber link to GDM, 2 Gigabits/sec
--- GDM
qsfp rx deserializer, 16 bit at 125 MHz output on main GDM clock
qsfp rx depacketizer (data,k -> packet data,eop; qsfp_rx_busy)
qsfp rx demux, hit map packets routed to GDM trigger logic block
GDM trigger logic block looks at hit map, generates yes/no trigger decision, encodes it as a trigger packet
TSM generator encodes GPS time data as a TSM packet
qsfp tx mux - trigger packets, tsm packets, etc to qsfp tx packet stream
qsfp tx packetizer (data,eop;trg,tsm,bsy -> data,k)
NB: the same tx data is sent to all 12 qsfp tx ports, to make sure we do not accidentally desync the CDMs.
qsfp tx 16/20 serializer, 16-bit at 125 MHz to 20-bit at 250 MHz to 2500 MHz serial out
qsfp tx fiber link to CDM, 2 Gigabits/sec
--- CDM
sfp rx 20/16 deserializer, 16-bit at 125 MHz
sfp rx depacketizer, (data,k -> data,eop;trg_in,tsm_in,bsy_in)
sfp rx demux (in reality, noop, all packets go to same place, vx tx)
vx tx mux (packets from GDM, packets with simulated waveforms from CDM midas frontend via AXI FIFO)
vx tx packet adapter 16-bit at 125 MHz to 8-bit at 12.5 MHz strobed every 10 clocks. NB: most important, this 10-clock strobe runs in sync between all CDMs!
vx tx phase transfer from main CDM clock to vxN_tx clock, have 10 clocks for transfer to happen
from here we run on the vxN_tx clock
vx txN serializer, 8-bit at 12.5 MHz to 125 MHz serial
vx_txN OBUF register, vxN_rx clock, 125 MHz clock, from PLL with adjustable phase, scan phase to find sweet spot
lvds line to vx
BBB: bsy_in from GDM is converted from pulse to level, goes out lvds line to vx, sync to vxN_tx clock
TTT: trg_in from GDM is a pulse, does out lvds line to vx, sync to vxN_tx clock.
NB: tsm is always a packet, bsy is always a signal (no packet), trg can be a packet or signal.
--- VX
lvds data captured by 125 MHz ADC sampling clock (CAEN base firmware logic)
lvds data connected to Yair's block
10/8 deserializer
depacketizer (data,k -> data,eop) to avoid accidental desync, we do not send any no trg, no tsm, no bsy K-codes.
demux
trigger packets go their way (4x 64 bit words of data go to event header: timestamps, hitmap data)
tsm packets go their way (4x 64 bit words of data go to tsm event header: timestamps, GPS time data)
BBB: bsy from lvds line stops waveform acquisition
TTT: trg from lvds line cause waveform acquisition, same as trg packet, but has no timestamp and other data attached to it.

Firmware registers

Block 0

0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2

Block 1

#### Register Block 1 (CDM) 
  Clock Chip and MGT Link, Control and Status Registers 
   | Register | Axi Address | Description | note 
   |:-|:-|:-|:-|             
   | 0 | 0x80011000 | Clock Chip Status and Control | CLK_IN_SEL_LS(1-0)(read-only), CLK_EXT_SEL_LS(2), CLK_RSTn_LS(3), CLK_LOSXTn_LS(4), CLK_LOLn_LS(5), CLK_I\
NTn_LS(6)    
   | 1 | 0x80011004 | free for use | |  
   | 2 | 0x80011008 | MGT Link Reset | mgt_rst(0), Reset for RX Error Latched(1)|    
   | 3 | 0x8001100c | MGT rx slide control | not needed or used |       
   | 4 | 0x80011010 |  link_up_and_running(10) tx_link_up_and_running(9), tx_sending_data(8), tx_link_up(7), rx_link_up_and_running(6), rx_error(5), rx_receivi\
ng_data(4), rx_link_up(3), link_power_good(2), sfp_rx_los(1), sfp_mod_absent(0) | Should be 0x7DC |  
   | 5 | 0x80011014 | reserved debug register | |        
   | 6-11 | ... | free for use | |    
   | 12 | 0x80011030 | mgt_rx_ref_clk_raw | ~0x7735940 (125 MHz) |    
   | 13 | 0x80011034 | rx_clk | ~0x7735940 (125 MHz) |    
   | 14 | 0x80011038 | mgt_tx_ref_clk_raw | ~0x7735940 (125 MHz) |    
   | 15 | 0x8001103c | tx_clk | ~0x7735940 (125 MHz) |    
   | 16 | 0x80011040 | clk_50MHz | 0x2faf080 (50 MHz) |    
   | 17 | 0x80011044 | Register_Block1_clk | 0x5f5e100 (100 MHz) |    
   | 18-20 | ... | free for use | |    
   | 21 | 0x80011054 | free for use | |    

Register 0x1000

            printf("0x1000 SFP c.c. status: 0x%08x\n", reg1000);
            printf("    CLK_IN_SEL_LS   0x%x\n", (reg1000>>0) & 3);
            printf("    CLK_EXT_SEL_LS  %d\n", (reg1000>>2) & 1);
            printf("    CLK_CLK_RSTn_LS %d\n", (reg1000>>3) & 1);
            printf("    CLK_LOSXTn_LS   %d\n", (reg1000>>4) & 1);
            printf("    CLK_LOLn_LS     %d\n", (reg1000>>5) & 1);
            printf("    CLK_INTn_LS     %d\n", (reg1000>>6) & 1);
            printf("0x1008 SFP link reset:  0x%08x\n", dev->dsdm_read32(0x1008));

Register 0x1010

            printf("0x1010 SFP link status: 0x%08x\n", reg1010);
            printf("    sfp_mod_absent_N       %d\n", (reg1010>>0) & 1);
            printf("    sfp_rx_los_N           %d\n", (reg1010>>1) & 1);
            printf("    link_power_good        %d\n", (reg1010>>2) & 1);
            printf("    rx_link_up             %d\n", (reg1010>>3) & 1);
            printf("    rx_receiving_data      %d\n", (reg1010>>4) & 1);
            printf("    rx_error               %d\n", (reg1010>>5) & 1);
            printf("    rx_lnk_up_and_running  %d\n", (reg1010>>6) & 1);
            printf("    tx_link_up             %d\n", (reg1010>>7) & 1);
            printf("    tx_sending_data        %d\n", (reg1010>>8) & 1);
            printf("    tx_link_up_and_running %d\n", (reg1010>>9) & 1);
            printf("    link_up_and_running    %d\n", (reg1010>>10) & 1);

Register 0x1014

            printf("0x1014 SFP link data:   0x%08x\n", reg1014);
            printf("    rx_data     0x%04x\n", (reg1014>>0) & 0xFFFF);
            printf("    k28p1_k28p5 %d\n", (reg1014>>16) & 1);
            printf("    rx_ctrl0    0x%01x\n", (reg1014>>17) & 3);
            printf("    rx_ctrl1    0x%01x\n", (reg1014>>19) & 3);
            printf("    rx_ctrl3    0x%01x\n", (reg1014>>21) & 1);
            printf("    tx_state    0x%01x\n", (reg1014>>22) & 3);
            printf("    rx_state    0x%01x\n", (reg1014>>24) & 0xF);
            printf("    rx_receiving_data %d\n", (reg1014>>28) & 1);
            printf("    rx_link_up        %d\n", (reg1014>>29) & 1);
            printf("    rx_error          %d\n", (reg1014>>30) & 1);
            printf("    rx_link_rst       %d\n", (reg1014>>31) & 1);

• see cdm_link_establishment.vhd
• tx_state:

0 = tx_idle
1 = waiting_for_rx_link
2 = sending_k28p1_k28p5
3 = sending_data

• rx_state:

0 = rx_idle
1 = waiting_for_ks
2 = rx_retry
3 = rx_up_switch_tx_clk_source
4 = waiting_for_data
5 = receiving_data
6 = rx_receive_error

Block 3

DS20k block register map

• busybox devmem 0x80013000 32

Note: R=readable, W=writable, L=latched by CMD_LATCH, B=reset on begin of run

reg | version    | xDM | xx | description
  0 | 0x20230731 | ALL | RO | ds20k version
  0 | 0x20240118 | ALL | RW | ds20k version and command
  1 | 0x20230731 | ALL | RW | scratch read/write register
  2 | 0x20230731 | ALL | RW | configure inputs and outputs
  3 | 0x20230731 | ALL | RW | FP_LED mux
  4 | 0x20230731 | ALL | RW | EXT_OUT mux
  5 | 0x20230731 | ALL | RO | VX_RX state
  6 | 0x20230731 | ALL | RO | VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state
  7 | 0x20230731 | ALL | RW | LED_OUT, EXT_OUT, VX_TX outputs
  8 | 0x20230731 | ALL | RW | VX_TX mux and config
  9 | 0x20230731 | ALL | RW | trigger config
 10 | 0x20231013 | ALL | RO | status register
 11 | 0x20230731 | ALL | RO | trigger counter
 12 | 0x20230731 | ALL | RO | time slice marker counter
 13 | 0x20240814 | ALL | RO | GPS 1pps period, 125 MHz
 14 | 0x20240814 | ALL | RO | Ru clock 1pps period, 125 MHz
 15 | 0x20230811 | CDM | RO | SFP RX status
 16 | 0x20230811 | CDM | RW | SFP TX control
 17 | 0x20230811 | GDM | RO | QSFP RX data 0, 1
 18 |            |.    |.   | 2, 3
 19 |            |.    |.   | 4, 5
 20 |            |.    |.   | 6, 7
 21 |            |     |.   | 8, 9
 22 |            |.    |.   | 10, 11
 23 | 0x20230811 | GDM | RW | QSFP TX control
 24 | 0x20231013 | ALL | RW | trigger pulser period
 25 | 0x20231013 | ALL | RW | trigger pulser burst control
 26 | 0x20231013 | ALL | RW | tsm pulser period
 27 | 0x20231204 | ALL | RW | data fifo CPU to FPGA
 28 | 0x20231204 | ALL | RW | packet loopback control
 28 | 0x20240118 | ALL | RW | data fifo FPGA to CPU
 29 | 0x20231208 | GDM | RW | bitmap of active qsfp ports
 29 | 0x20240118 | ALL | RW | packet loopback control
 29 | 0x20240510 | ALL | RW | packet routing
 30 | 0x20231208 | GDM | RO | qsfp link status ports 0..7
 31 | 0x20231208 | GDM | RO | qsfp link status ports 8..11
 32 | 0x20240118 | GDM | RW | enabled QSFP ports
 33 | 0x20240118 | CDM | RW | enabled VX ports
 34,35 | 0x20240118 | ALL | RO | time stamp 64 bits
 36,37 | 0x20240118 | ALL | RO | old time stamp
 38,39,40 | 0x20240118 | CDM | RO | VX busy counters
 41,42,43 | 0x20240118 | GDM | RO | QSFP busy counters
 44 | 0x20240118 | CDM | RO | cdm_bsy_up_counter and cdm_bsy_pulse_counter
 45 | 0x20240118 | CDM | RO | cdm_veto_pulse_counter and cdm_veto_up_counter
 46 | 0x20240118 | GDM | RO | gdm_bsy_pulse_counter and gdm_bsy_up_counter
 47 | 0x20240118 | GDM | RO | gdm_bsy_refresh_counter
 48 | 0x20240118 | GDM | RO |  gdm_veto_up_counter and gdm_veto_pulse_counter
 49,50,51 | 0x20240424 | CDM | RO | VX RX serial data monitor, 8 bits per VX port
 52,53    | 0x20240424 | CDM | RO | VX serial link status, 4 bits per VX port
 54 | 0x20240424 | CDM | RO | VX TX serial data monitor
 55 | 0x20240430 | ALL | RO | QSFP, SFP and VX link loss counters
 56 | 0x20240510 | ALL | RO | sfp_rx_packet_counter
 57 | same       | ALL | RO | sfp_tx_packet_counter
 58 | same       | ALL | RO | qsfp_rx_packet_counter[0]
 59 | same       | ALL | RO | qsfp_tx_packet_counter
 60 | same       | ALL | RO | vx_rx_packet_counter[0]
 61 | same       | ALL | RO | vx_tx_packet_counter
 62 | 0x20240719 | CDM | RO | cdm_hitmap_period, ports 0, 1
 63 | same       | CDM | RO | ports 2, 3
 64 | same       | CDM | RO | ports 4, 5
 65 | same       | CDM | RO | ports 6, 7
 66 | same       | CDM | RO | ports 8, 9
 67 | same       | CDM | RO | ports 10, 11
 68 | 0x20240814 | ALL | RW | GPS control and status
 69 | 0x20241104 | CDM | RO | vx_tx_trg_packet_counter, counter of TRG packets CDM->VX
 70 | 0x20241104 | CDM | RO | vx_tx_tsm_packet_counter, counter of TSM packets CDM->VX
 71 | 0x20241104 | ALL | RO | packet error bits
 72,73 | 0x20241104 | CDM | ROL | vx_tx_trg_data_latched[63:0] trigger packet data
 74,75 | 0x20241104 | CDM | ROL | vx_tx_tsm_data_latched[63:0] tsm packet data (truncated to 64 bits)
 76 | 0x20241104 | CDM | ROL | cdm_hitmap_trigger_counter_latched, counter of hitmap triggers generated in the CDM
 77,78,79 | 0x20241104 | CDM | ROL | vx_rx_hitmap_data_latched | VX_RX 80 bits of HITMAP packet from VX
 80 | 0x20241209 | CDM | RO | vx_lvds_rx_link_status | VX LVDS link status
 81 | 0x20241209 | CDM | RO | vx_lvds_rx_trg_data | VX LVDS TRG packet first 32 bits
 82 | 0x20241209 | CDM | RO | vx_lvds_rx_tsm_data | VX LVDS TSM packet first 32 bits
 83 | 0x20241209 | CDM | RO | vx_lvds_rx_decoded_packet_counter | VX LVDS decoded packet counter
 84 | 0x20241209 | CDM | RO | vx_lvds_rx_packet_error_counter | VX LVDS packet error counter

Register 0 0x80013000 ds20k version

on read: ds20k version 0xYYYYMMDD

on write:

• 0 - noop - as of version 0x20240118, write a zero after writing a command
• 1 - cmd_reset - reset logic to good state
• 2 - cmd_arm_ts - arm timestamp reset
• 3 - cmd_trg - issue a trigger
• 4 - cmd_tsm - issue a tsm
• 5 - cmd_vx_rx_reset - reset the VX receive path
• 6 - cmd_vx_tx_reset - reset the VX transmit path
• 7 - cmd_hitmap_trg - generate a hitmap trigger and data packet
• 8 - cmd_trg_pulser_reset - reset the trigger pulser
• 9 - cmd_tsm_pulser_reset - reset the tsm pulser
• 10 - cmd_bor_start - start begin-of-run trigger sequence
• 11 - cmd_bor_clear - after run has started, clear begin-of-run status bits
• 12 - cmd_latch - latch counters & etc into AXI registers for coherent readout

Register 1 0x80013004 scratch

scratch read-write register

Register 2 0x80013008 input and output config

bit | version | fpga name       | description
  0 | ALL     | lemo_enable     | enable LEMO input 1
  1 |         |                 | 2
  2 |         |                 | 3
  3 |         |                 | 4
  4 | ALL     | lemo_invert     | invert LEMO input 1
  5 |         |                 | 2
  6 |         |                 | 3
  7 |         |                 | 4
  9 | ALL     | ext_out_disable | disable LEMO output 1
 10 |         |                 | 2
 11 | ALL     | ext_out_invert  | invert LEMO output 1
 12 |         |                 | 2

Register 3 0x8001300C FP_LED control

wire [15:0] led_out_mux_sel  = register_data_in[3][15:0];
wire [3:0]  led_out_invert   = register_data_in[3][19:16];

led_out_mux_sel is 4 groups (one per LED) of 4 bits (choice 0..15):
mux | version    | fpga name       | description
  0 | ALL        |                 | power on default
  1 | ALL        | led_out_reg     | register 7 bits
  2 | 0x20231013 | pll_locked      | clock chip PLL is locked
  3 | 0x20231013 | sfp_link_status | SFP link is good
  3 | 0x20240118 | sfp_link_status or qsfp_tx_link_rx_status | SFP/QSFP link is good
  4 | ALL        | lemo_in_sync[1] | LEMO input 1
  5 | ALL        | lemo_in_sync[2] | LEMO input 2
  6 | ALL        | lemo_in_sync[3] | LEMO input 3
  7 | ALL        | lemo_in_sync[4] | LEMO input 4
  8 | ALL        | ext_out[1]      | LEMO output 1
  9 | ALL        | ext_out[2]      | LEMO output 2
  A | 0x20231013 | trg_in          | trigger
  B | 0x20231013 | tsm_in          | time slice marker
  C | 0x20240118 | gdm_bsy         | GDM busy: OR of all CDM busy
  D | 0x20240118 | cdm_bsy         | CDM busy: OR of all VX busy
  E | 0x20240118 | cdm_veto        | GDM busy -> GDM veto -> CDM veto -> VX trigger veto
  F | ALL        |                 | fixed logic level 1

Register 4 0x80013010 LEMO OUT control

wire [7:0] ext_out_mux_sel = register_data_in[4][7:0];

ext_out_mux_sel is 2 groups (one per LEMO) of 4 bits (choice 0..15):

mux | version    | fpga name       | description
  0 | ALL        |                 | power on default
  1 | ALL        | ext_out_reg     | register 7
  2 | ALL        | trg_pulser      | pulser trigger
  3 | 0x20240724 | vx1_tx_out[2]   | vx1 serial data out
  4 | 0x20240724 | vx1_rx[1]       | vx1 serial data in
  5 | 0x20240724 | vx_rx_iob[0]    | vx1 serial data in captured by IOB register
  6 | ALL        | lemo_in_async[1]| test synchronizer
  7 | ALL        | lemo_in_sync[1] | test synchronizer
  8 | 0x20240724 | trg_in_pulse    | trigger signal
  9 | 0x20240724 | tsm_in_pulse    | time slice marker signal
  A | 0x20240118 | cdm_bsy         | CDM busy from VX
  B | 0x20240118 | gdm_bsy         | GDM busy from CDM
  C | 0x20240118 | cdm_veto        | veto from GDM to CDM to VX
  D | 0x20240118 | vx1_rx[1]       | serial data VX to CDM
  E | not used (sink)
  F | ALL        | 1               | fixed logic level 1

Register 5 0x80013014 VX_RX status

assign register_data_out[5] = {
                vx8_rx[3], vx8_rx[2], vx8_rx[1], vx8_rx[0],
                vx7_rx[3], vx7_rx[2], vx7_rx[1], vx7_rx[0],
                vx6_rx[3], vx6_rx[2], vx6_rx[1], vx6_rx[0],
                vx5_rx[3], vx5_rx[2], vx5_rx[1], vx5_rx[0],
                vx4_rx[3], vx4_rx[2], vx4_rx[1], vx4_rx[0],
                vx3_rx[3], vx3_rx[2], vx3_rx[1], vx3_rx[0],
                vx2_rx[3], vx2_rx[2], vx2_rx[1], vx2_rx[0],
                vx1_rx[3], vx1_rx[2], vx1_rx[1], vx1_rx[0]
                };

Register 6 0x80013018 VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state

assign register_data_out[6] = {
                vx2_tx_out[2], vx2_tx_out[1], vx2_tx_out[0], vx1_tx_out[2],
                vx1_tx_out[1], vx1_tx_out[0], ext_out[2], ext_out[1],
                fp_led_out[3], fp_led_out[2], fp_led_out[1], fp_led_out[0],
                ext_in_lv[4], ext_in_lv[3], ext_in_lv[2], ext_in_lv[1],
                vx12_rx[3], vx12_rx[2], vx12_rx[1], vx12_rx[0],
                vx11_rx[3], vx11_rx[2], vx11_rx[1], vx11_rx[0],
                vx10_rx[3], vx10_rx[2], vx10_rx[1], vx10_rx[0],
                vx9_rx[3], vx9_rx[2], vx9_rx[1], vx9_rx[0]
                };

bit | version | fpga name       | description
  0 | ALL     | vx9_rx          | VX_RX
  1 |         |                 | 
  2 |         |                 | 
  3 |         |                 | 
  4 | ALL     | vx10_rx         | VX_RX
  5 |         |                 | 
  6 |         |                 | 
  7 |         |                 | 
  8 | ALL     | vx11_rx         | VX_RX
  9 |         |                 | 
 10 |         |                 | 
 11 |         |                 | 
 12 | ALL     | vx12_rx         | VX_RX
 13 |         |                 | 
 14 |         |                 | 
 15 |         |                 | 
 16 | ALL     | ext_in_lv       | LEMO inputs
 17 |         |                 | 
 18 |         |                 | 
 19 |         |                 | 
 20 | ALL     | FP_LED          | FP_LEDs
 21 |         |                 | 
 22 |         |                 | 
 23 |         |                 | 
 24 | ALL     | ext_out[1]      | LEMO outputs
 25 |         | ext_out[2]      | 
 26 | ALL     | vx1_tx          | VX1_TX
 27 |         |                 | 
 28 |         |                 |
 29 | ALL     | vx2_tx          | VX2_TX 
 30 |         |                 | 
 31 |         |                 | 

Register 7 0x8001301C LED_OUT, EXT_OUT, VX_TX outputs

   wire [3:0] led_out_reg = register_data_in[7][3:0];
   wire [2:1] ext_out_reg = register_data_in[7][5:4];
   // register_data_in[7][6];
   // register_data_in[7][7];
   wire [7:0] vx_tx_out_reg = register_data_in[7][15:8];

bit | version | fpga name       | description
  0 | ALL     | led_out_reg     | FP_LED 1
  1 |         |                 | 2
  2 |         |                 | 3
  3 |         |                 | 4
  4 | ALL     | ext_out_reg     | LEMO OUT 1
  5 |         |                 | 2
  6 | -       |                 | 
  7 | -       |                 | 
  8 | ALL     | vx_tx_out_reg   | VX1_TX 0
  9 |         |                 | 1
 10 |         |                 | 2
 11 |         |                 | -
 12 | ALL     |                 | VX2_TX 0
 13 |         |                 | 1
 14 |         |                 | 2
 15 |         |                 | -
 16 | -       |                 |
 17 |         |                 | 
 18 |         |                 | 
 19 |         |                 | 
 20 |         |                 |
 21 |         |                 | 
 22 |         |                 | 
 23 |         |                 | 
 24 |         |                 |
 25 |         |                 | 
 26 |         |                 |
 27 |         |                 | 
 28 |         |                 |
 29 |         |                 | 
 30 |         |                 | 
 31 |         |                 | 

Register 8 0x80013020 VX_TX config

wire [3:0] vx_tx_mux_sel = register_data_in[8][3:0];
vx_tx_loopback           = register_data_in[8][31];

vx_tx_mux_sel is 4 bits (choice 0..15):
0 - power on default, control by vx_tx_out_reg
1 - GDM
2 - CDM
3 - pulser loopback test
4 - pulser loopback test
5 - 62.5 MHz output
6 - 125 MHz output
7 - trg, tsm, serial
8 - trg, tsm, lvds serial rx to serial tx loopback
9 - GPS box control (ds20k rev 0x20240814)
10
11
12
13
14
15 - production config: trg, veto, serial

Register 9 trg and tsm source

from version 0x20240724

   wire [15:0] trg_src_mask      = register_data_in[9][15:0];
   wire [31:16] tsm_src_mask      = register_data_in[9][31:16];

   wire [15:0]      trg_src_bits =
                    {
                     1'b0,            // 15
                     1'b0,            // 14
                     vx_tx_tsm_done,  // 13
                     vx_tx_trg_done,  // 12

                     1'b0, // gdm_hitmap_trigger, // 11
                     cdm_hitmap_trigger, // 10
                     sfp_rx_tsm,      // 9
                     sfp_rx_trg,      // 8

                     sfp_rx_data[1],  // 7
                     sfp_rx_data[0],  // 6
                     tsm_pulser,      // 5
                     trg_pulser,      // 4

                     lemo_in_sync[4], // 3
                     lemo_in_sync[3], // 2
                     lemo_in_sync[2], // 1
                     lemo_in_sync[1]  // 0
                     };
   
   wire [15:0]      trg_bits = trg_src_bits & trg_src_mask;
   wire [15:0]      tsm_bits = trg_src_bits & tsm_src_mask;

before that:

   wire [7:0] trg_src_mask      = register_data_in[9][7:0];
   wire [7:0] tsm_src_mask      = register_data_in[9][15:8];
   wire       trg_pulser_enable = register_data_in[9][16];
   wire       tsm_pulser_enable = register_data_in[9][17];
   wire       trg_software      = register_data_in[9][18];
   wire       tsm_software      = register_data_in[9][19];
   // bits 20:31 not used

wire [7:0]       xxx_src_bits =
                    {
                     sfp_rx_data[1],
                     sfp_rx_data[0],
                     tsm_pulser & tsm_pulser_enable,
                     trg_pulser & trg_pulser_enable,
                     lemo_in_sync[4],
                     lemo_in_sync[3],
                     lemo_in_sync[2],
                     lemo_in_sync[1]
                     };

trg_src_mask and tsm_src_mask bits:
0 - LEMO IN 1
1 - LEMO IN 2
2 - LEMO IN 3
3 - LEMO IN 4
4 - trg_pulser
5 - tsm_pulser
6 - sfp_rx_data[0] // to become sfp_trg_in, selected from sfp_rx_data[0], cdm_rx trg_out and gdm trigger packet
7 - sfp_rx_data[1] // to become sfp_tsm_in, selected from sfp_rx_data[1], cdm_rx tsm_out and gdm tsm packet

Register 10 0x80013028 status register

bit | version    | fpga name       | description
  0 | 0x20231013 | pll_locked      | clock chip PLL is locked
  1 | 0x20240118 | ts_reset_armed  | timestamp reset is armed
  2 | 0x20240118 | qsfp_tx_link_rx_status | QSFP link status is good for all enabled ports
  3 | 0x20240424 | vx_tx_link_rx_status   | VX link status is good for all enabled ports
  4 | 0x20240118 | cdm_bsy         | VX busy grand-or
  5 | 0x20240118 | gdm_bsy         | QSFP busy grand-or 
  6 | 0x20240118 | gdm_veto        | gdm_veto = gdm_busy 
  7 | 0x20240118 | cdm_veto        | CDM veto from GDM to VX 
  8 | 0x20240725 | bor_started     | begin-of-run sequence started
  9 | 0x20240725 | bor_finished    | begin-of-run sequence fininished, see commands 10 and 11 
 10 |         |                 | 
 11 |         |                 | 
 12 |         |                 | 
 13 |         |                 | 
 14 |         |                 | 
 15 |         |                 | 
 16 |         |                 | 
 17 |         |                 | 
 18 |         |                 | 
 19 |         |                 | 
 20 |         |                 | 
 21 |         |                 | 
 22 |         |                 | 
 23 |         |                 | 
 24 |         |                 | 
 25 |         |                 | 
 26 |         |                 | 
 27 |         |                 | 
 28 |         |                 |
 29 |         |                 | 
 30 |         |                 | 
 31 |         |                 | 

Register 11 0x8001302C trg_counter

trigger counter

Register 12 0x80013030 tsm_counter

time slice marker counter

Register 13 0x80013034 GPS 1pps period

GPS 1pps period in 8 ns clocks

Register 14 0x80013038 Rb clock 1pps period

PRS-10 Rb clock 1pps output period in 8 ns clocks

Register 15 0x8001303C SFP RX status

bit | ds20k version | fpga signal name | description
  0 | ALL        | sfp_rx_data[15:0]   | cdm sfp received data
 16 | same       | sfp_rx_data_is_k[0] | 
 17 | same       | sfp_rx_data_is_k[1] | 
 18 |            | 0                   | 
 19 | 0x20231204 | sfp_rx_sel_lpb      | sfp tx->rx loopback
 20 | 0x20231013 | sfp_link_status     | sfp link connected, exchanging data
 21 | same       | sfp_link_rx_status  | sfp link receiving correct idle pattern from GDM TX
 22 | same       | sfp_link_error      | sfp link receiver error (badk or overflow)
 23 | same       | sfp_rx_data_error   | sfp transceiver state machine is in error state
 24 |            |                     | 
 25 |            |                     | 
 26 |            |                     | 
 27 |            |                     | 
 28 |            |                     |
 29 |            |                     | 
 30 |            |                     | 
 31 |            |                     | 

Register 16 SFP TX control

   wire [15:0] sfp_tx_data_reg = register_data_in[16][15:0];
   wire [1:0]  sfp_tx_ctrl_reg = register_data_in[16][17:16];
   // 18
   // 19
   // 20:23
   wire qsfp_tx_enable_trg     = register_data_in[23][24]; // enable QSFP TX trg_in_pulse k-code
   wire qsfp_tx_enable_tsm     = register_data_in[23][25]; // enable QSFP TX tsm_in_pulse k-code
   // 26
   // 27
   //wire sfp_rx_sel_loopback    = register_data_in[16][28]; // TX->RX serial loopback
   wire sfp_tx_sel_loopback    = register_data_in[16][29]; // RX->TX serial loopback
   wire sfp_tx_sel_trg         = register_data_in[16][30]; // 16 individual bits
   wire sfp_tx_sel_reg         = register_data_in[16][31]; // from register

Register 17-22 QSFP RX data

QSFP RX data links 0..11

Register 23 QSFP TX control

   wire [15:0] qsfp_tx_data_reg = register_data_in[23][15:0];
   wire [1:0]  qsfp_tx_ctrl_reg = register_data_in[23][17:16];
   wire qsfp_rx_sel_lpb         = register_data_in[23][28]; // TX->RX loopback
   wire qsfp_tx_sel_lpb         = register_data_in[23][29]; // RX->TX loopback
   wire qsfp_tx_sel_trg         = register_data_in[23][30]; // 16 individual bits
   wire qsfp_tx_sel_reg         = register_data_in[23][31]; // data from register

Register 24 0x80013060 trigger pulser period

trigger pulser period in units of 8 ns (125 MHz clock)

Register 25 0x80013064 trigger burst pulser

   wire [7:0]         conf_burst_count  = conf_pulser_burst_ctrl[31:24];
   wire [23:0]        conf_burst_period = conf_pulser_burst_ctrl[23:0];

Register 26 0x80013068 tsm pulser period

time slice marker period in units of 8 ns (125 MHz clock)

Register 27 0x8001306C data write fifo

   wire        fifo_reset = register_data_in[27][31];
   wire        fifo_to_fpga_wr1 = register_data_in[27][27];
   wire        fifo_to_fpga_wr2 = register_data_in[27][26];
   wire [16:0] fifo_to_fpga_din = register_data_in[27][16:0];

   assign register_data_out[27][31:24] = register_data_in[27][31:24]; // echo write bits
   assign register_data_out[27][23] = fifo_to_fpga_full;
   assign register_data_out[27][22] = fifo_to_fpga_empty;

Register 28 0x80013070 data read fifo

   wire        fifo_reset = register_data_in[28][31];
   wire        fifo_from_fpga_rd1 = register_data_in[28][25];
   wire        fifo_from_fpga_rd2 = register_data_in[28][24];

   assign register_data_out[28][31:24] = register_data_in[28][31:24]; // echo write bits
   assign register_data_out[28][21] = fifo_from_fpga_full;
   assign register_data_out[28][20] = fifo_from_fpga_empty;
   assign register_data_out[28][16:0] = fifo_from_fpga_dout;

Register 29 0x80013074 packet routing

Control packet routing and loopbacks:

   wire [3:0]  dn_route_ctrl             = register_data_in[29][3:0];
   wire [3:0]  up_route_ctrl             = register_data_in[29][7:4];
   wire [3:0]  fifo_to_fpga_route_ctrl   = register_data_in[29][11:8];
   //wire [3:0]  spare_route_ctrl        = register_data_in[29][15:12];

   wire        dn_mux_trg_enable         = register_data_in[29][16];
   wire        dn_mux_tsm_enable         = register_data_in[29][17];
   wire        dn_mux_sfp_rx_fifo_enable = register_data_in[29][18];
   // 19
   // 20..23

   wire        up_mux_vx_rx_enable       = register_data_in[29][24];
   // 25
   // 26
   // 27
   wire        fifo_from_fpga_hitmap_enable       = register_data_in[29][28];
   // 29..31

fifo_to_fpga output routing:

• 0 - to down packet mux
• 1 - to up packet mux
• 2 - to fifo_from_fpga mux
• 3 - not used

down packet mux inputs:

• fifo_to_fpga_0
• trg_pkt16 enabled by dn_mux_trg_enable
• tsm_pkt16 enabled by dn_mux_tsm_enable
• sfp_rx_fifo_pkt16 enabled by dn_mux_sfp_rx_fifo_enable
• up_pkt16_2 loopback from up packet mux

down packet mux output routing:

• 0 - to vx_tx_pkt16 to VX TX (vx_link_tx and vx_ser_tx)
• 1 - to fifo_from_fpga mux
• 2 - to up packet mux loopback
• 3 - to qsfp_tx_pkt16 to GDM QSFP TX (cdm_link_tx)

up packet mux inputs:

• vx_rx_pkt16 enabled by up_mux_vx_rx_enable data from 12 VX RX links
• fifo_to_fpga_1
• dn_pkt16_2 loopback from down packet mux

up packet mux output routing:

• 0 - to sfp_tx_pkt16 to CDM SFP TX (VX data to GDM)
• 1 - to fifo_from_fpga mux
• 2 - to down packet mux loop loopback
• 3 - not used

fifo_from_fpga mux inputs:

• fifo_to_fpga_2
• dn_pkt16_1 from down packet mux
• up_pkt16_1 from up packet mux
• qsfp_tx_pkt16 from GDM QSFP link 0 (there is no GDM QSFP 12-to-1 mux)
• hitmap_pkt16 enabled by fifo_from_fpga_hitmap_enable in register 29

Register 30 0x80013078 qsfp link status ports 0..7

assign register_data_out[30] = 
     {
       qsfp_rx_data_error[7], qsfp_link_error[7], qsfp_link_status[7], qsfp_link_rx_status[7], // 7
       qsfp_rx_data_error[6], qsfp_link_error[6], qsfp_link_status[6], qsfp_link_rx_status[6], // 6
       qsfp_rx_data_error[5], qsfp_link_error[5], qsfp_link_status[5], qsfp_link_rx_status[5], // 5
       qsfp_rx_data_error[4], qsfp_link_error[4], qsfp_link_status[4], qsfp_link_rx_status[4], // 4
       qsfp_rx_data_error[3], qsfp_link_error[3], qsfp_link_status[3], qsfp_link_rx_status[3], // 3
       qsfp_rx_data_error[2], qsfp_link_error[2], qsfp_link_status[2], qsfp_link_rx_status[2], // 2
       qsfp_rx_data_error[1], qsfp_link_error[1], qsfp_link_status[1], qsfp_link_rx_status[1], // 1
       qsfp_rx_data_error[0], qsfp_link_error[0], qsfp_link_status[0], qsfp_link_rx_status[0]  // 0
      };

Register 31 0x8001307C qsfp link status ports 8..11

assign register_data_out[31] = 
     {
       4'b0000,
       4'b0000,
       4'b0000,
       4'b0000,
       qsfp_rx_data_error[11], qsfp_link_error[11], qsfp_link_status[11], qsfp_link_rx_status[11], // 11
       qsfp_rx_data_error[10], qsfp_link_error[10], qsfp_link_status[10], qsfp_link_rx_status[10], // 10
       qsfp_rx_data_error[9],  qsfp_link_error[9],  qsfp_link_status[9],  qsfp_link_rx_status[9],  // 9
       qsfp_rx_data_error[8],  qsfp_link_error[8],  qsfp_link_status[8],  qsfp_link_rx_status[8]   // 8
      };

Register 32 0x80013080 bitmap of enabled qsfp ports

   wire [11:0]  qsfp_mask       = register_data_in[32][11:0];
   wire         qsfp_bsy_force  = register_data_in[32][12];
   // not used                  = register_data_in[32][15:13];
   wire [15:0]  gdm_veto_extend = register_data_in[32][31:16];

On the GDM:

qsfp_rx_bsy[11..0] are pulses received from the CDMs

qsfp_rx_bsy_or is the grand-or of qsfp_rx_bsy masked by qsfp_mask

if qsfp_rx_bsy_or is high, gdm_bsy goes up and stays up for vx_bsy_extend*2 clocks

gdm_veto = gdm_bsy (bsy of any one VX causes trigger veto to all of them)

gdm_veto transition 0->1 causes gdm_veto_pulse. as long as gdm_veto is high, gdm_veto_pulse is generated every gdm_veto_extend clocks

gdm_veto_pulse is sent to all CDMs.

For this to work right, cdm_bsy_extend should not be bigger than gdm_veto_extend.

Register 33 0x80013084 bitmap of enabled VX ports

   wire [11:0]  vx_mask       = register_data_in[33][11:0];
   wire         vx_bsy_force  = register_data_in[33][12];
   // not used                = register_data_in[33][15:13];
   wire [15:0]  vx_bsy_extend = register_data_in[33][31:16];

On the CDM:

cdm_bsy is a grand-or of all vx_bsy masked by vx_mask (list of active VXes).

vx_bsy_extend controls how often state of cdm_bsy is sent to the GDM. when cdm_bsy goes 0->1, we send a cdm_bsy_pulse and keep resending it every vx_bsy_extend clocks as long as cdm_bsy stays high.

cdm_bsy_pulse is sent to the GDM.

sfp_rx_veto is the received from the GDM

if sfp_rx_veto goes up, cdm_veto goes up and stays up for gdm_veto_extend clocks.

for this to work right, CDM gdm_veto_extend must be bigger than GDM gdm_veto_extend.

cdm_veto goes to VXes on v1_tx_out[1] which is LVDS input 13.

Register 34, 35 0x80013088, 8C current timestamp

current 64-bit timestamp, 125 MHz

Register 36, 37 0x80013090, 94 old timestamp

old 64-bit timestamp, 125 MHz. timestamp saved at run start when it is reset to 0.

Register 38, 39, 40 0x80013098, 9C, A0 VX busy counters

8 bits per VX port, counters overflow to 255, reset at run start.

Register 41, 42, 43 0x800130A4, A8, AC QSFP busy counters

8 bits per QSFP port, counters overflow to 255, reset at run start.

Register 44, 45, 46, 47, 48 0x800130B0, B4, B8, BC, C0 CDM and GDM busy and veto counters

• cdm_busy = grand-or of all VX busy for enabled VXes
• gdm_busy = grand-or of all CDM busy for enabled CDMs
• gdm_veto = gdm_busy

 44 | lo 16 bits | cdm_bsy_up_counter      | CDM busy, increments when cdm_busy goes 0->1
 44 | hi 16 bits | cdm_bsy_pulse_counter   | CDM busy to GDM, increments for each cdm_bsy_pulse sent to the GDM

 45 | lo 16 bits | cdm_veto_pulse_counter  | CDM veto from GDM, increments for each sfp_rx_veto received from the GDM
 45 | hi 16 bits | cdm_veto_up_counter     | CDM veto to VX, increments each time cdm_veto is set to 1.

 46 | lo 16 bits | gdm_bsy_pulse_counter   | GDM busy from CDM, increments for each qsfp_rx_busy received from CDMs (unless they overlap)
 46 | hi 16 bits | gdm_bsy_up_counter      | GDM busy, increments each time gdm_bsy goes 0->1

 47 | lo 16 bits | gdm_bsy_refresh_counter | GDM busy, increments each time gdm_bsy is extended by new qsfp_rx_busy
 47 | hi 16 bits | spare                   |

 48 | lo 16 bits | gdm_veto_up_counter     | GDM veto, increments each time gdm_veto goes 0->1
 48 | hi 16 bits | gdm_veto_pulse_counter  | GDM veto to CDM, increments for each gdm_veto_pulse sent to the CDM

Register 49, 50, 51 0x800130xx VX RX serial data monitor

VX RX data, 8-bit per VX channel. k-bit is omitted.

   assign register_data_out[49][7:0]   = vx_rx_data[0]; // vx1
   assign register_data_out[49][15:8]  = vx_rx_data[1]; // vx2
   assign register_data_out[49][23:16] = vx_rx_data[2]; // vx3
   assign register_data_out[49][31:24] = vx_rx_data[3]; // vx4

   assign register_data_out[50][7:0]   = vx_rx_data[4]; // vx5
   assign register_data_out[50][15:8]  = vx_rx_data[5]; // vx6
   assign register_data_out[50][23:16] = vx_rx_data[6]; // vx7
   assign register_data_out[50][31:24] = vx_rx_data[7]; // vx8

   assign register_data_out[51][7:0]   = vx_rx_data[8]; // vx9
   assign register_data_out[51][15:8]  = vx_rx_data[9]; // vx10
   assign register_data_out[51][23:16] = vx_rx_data[10]; // vx11
   assign register_data_out[51][31:24] = vx_rx_data[11]; // vx12

Register 52, 53 VX link status

   assign register_data_out[52] = 
     {
       vx_rx_error[7], vx_link_error[7], vx_link_status[7], vx_link_rx_status[7], // 7
       vx_rx_error[6], vx_link_error[6], vx_link_status[6], vx_link_rx_status[6], // 6
       vx_rx_error[5], vx_link_error[5], vx_link_status[5], vx_link_rx_status[5], // 5
       vx_rx_error[4], vx_link_error[4], vx_link_status[4], vx_link_rx_status[4], // 4
       vx_rx_error[3], vx_link_error[3], vx_link_status[3], vx_link_rx_status[3], // 3
       vx_rx_error[2], vx_link_error[2], vx_link_status[2], vx_link_rx_status[2], // 2
       vx_rx_error[1], vx_link_error[1], vx_link_status[1], vx_link_rx_status[1], // 1
       vx_rx_error[0], vx_link_error[0], vx_link_status[0], vx_link_rx_status[0]  // 0
      };

   assign register_data_out[53] = 
     {
       //4'b0000,
       1'b0, ~vx_rx_deser_rdy[0], vx_rx_code_err[0], vx_rx_disp_err[0],
       //4'b0000,
       //4'b0000,
       //4'b0000,
       vx_rx_monitor[0],
       vx_rx_error[11], vx_link_error[11], vx_link_status[11], vx_link_rx_status[11], // 11
       vx_rx_error[10], vx_link_error[10], vx_link_status[10], vx_link_rx_status[10], // 10
       vx_rx_error[9],  vx_link_error[9],  vx_link_status[9],  vx_link_rx_status[9],  //  9
       vx_rx_error[8],  vx_link_error[8],  vx_link_status[8],  vx_link_rx_status[8]   //  8
      };

vx_rx_monitor (12-bit) is from deserializer_10b.sv:

   assign monitor_out[9:0] = lastByte[9:0];
   assign monitor_out[10]  = comma;
   assign monitor_out[11]  = ready;

Register 54 VX TX serial data monitor

• contents of vx_tx_monitor from vx_ser_tx.sv:

   assign monitor_out[8:0]   = data_to_encoder; // 8-bit + k
   assign monitor_out[9]     = valid;
   assign monitor_out[15:10] = 0;
   assign monitor_out[25:16] = encoded_data; // 10-bit
   assign monitor_out[26]    = encoded_valid;
   assign monitor_out[27]    = 0;
   assign monitor_out[31:28] = 0;

Register 55 QSFP, SFP, VX link loss counters

assign register_data_out[55] = {
  vx_rx_error_counter,
  qsfp_link_rx_status_drop_counter,
  sfp_link_rx_status_drop_counter,
  vx_link_rx_status_drop_counter
};

24..31 - CDM VX RX error counters, count any errors in the VX RX data path (bad serial data, fifo overflow, etc)
16..23 - GDM QSFP link loss counter, increments on qsfp_tx_link_rx_status 1->0 (reg10)
 8..15 - CDM SFP  link loss counter, increments on sfp_link_rx_status     1->0 (reg15)
 0...7 - CDM VX   link loss counter, increments on vx_tx_link_rx_status   1->0 (reg10)

Register 68 GPS control and status

assign register_data_out[68] =
     {
      8'b00000000,      // 23+8
      rb_1pps_counter,  // 16+8 bits
      gps_1pps_counter, // 8+8 bits
      1'b0,
      1'b0,
      rb_ser_in,   // 5
      gps_data_in, // 4
      gps_aux_out, // 3
      gps_aux_in,  // 2
      rb_1pps_in,  // 1
      gps_1pps_in  // 0
      };

Register 71 packet error bits

assign register_data_out[71] =
     {
      8'b00000000,  // 23+8 bits
      8'b00000000,  // 16+8 bits
      8'b00000000,  //  8+8 bits
      1'b0, // 7
      1'b0, // 6
      1'b0, // 5
      1'b0, // 4
      cdm_hitmap_encode_error_latch, // 3
      cdm_hitmap_pkt16_error_latch,  // 2
      vx_tx_tsm_pkt8_error_latch,    // 1
      vx_tx_trg_pkt8_error_latch     // 0
      };

Firmware registers branch develop_ko

Register map

  0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
  1 | ALL | ALL | RW | read write scratch register
  2 | ALL | CDM | ?? | MGT not used
  3 | ALL | CDM | RO | MGT debug_data
  4 | ALL | CDM | RW | clk_config_vec
  5 | ALL | CDM | ?? | not used
  6 | ALL | CDM | RO | CDM_link_data_processing:o_error_time
  7 | ALL | CDM | RO | CDM_link_data_processing:o_error_count

register 0 0x80010000

GDM:

0 - gdm_link_interface:i_mgt_rst
2 - gdm_link_interface:i_link_down_latched_rst
8 - GDM_link_data_processing:i_rst
10..9 - GDM_link_data_processing:i_data_mode

CDM:

0 - cdm_link_interface:i_mgt_rst
2 - cdm_link_interface:i_link_down_latched_rst
8 - CDM_link_data_processing:i_rst
10..9 - CDM_link_data_processing:i_data_mode

register 1 0x80010004

GDM:

nlinks-1..0 - gdm_link_interface:i_rx_slide_trigger

CDM:

nlinks-1..0 - cdm_link_interface:i_rx_slide_trigger

register 2 0x80010008

GDM:

nlinks-1..0 - gdm_link_interface:o_link_power_good
nlinks+15..16 - gdm_link_interface:o_link_status

CDM:

nlinks-1..0 - cdm_link_interface:o_link_power_good
nlinks+15..16 - cdm_link_interface:o_link_status

register 3 0x8001000c

GDM: simple loopback register

CDM:

31..0 - debug_data - cdm_link_interface:o_debug

o_debug:

rx_link_rst & rx_error & rx_link_up & rx_receiving_data &
std_logic_vector(rx_state_count) & tx_state_count_on_rx_clk & i_rx_ctrl3(0) &
i_rx_ctrl1(1 downto 0) & i_rx_ctrl0(1 downto 0) &
rx_data_is_k28p1_k28p5 &
i_rx_data;

register 4 0x80010010

GDM write:

0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS

GDM read:

0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS
4 - clk_config_vec(4) - CLK_LOSXTn_LS
5 - clk_config_vec(5) - CLK_LOLn_LS
6 - clk_config_vec(6) - CLK_INTn_LS
7 - constant 1
31..8 - constant 0

register 5 0x80010014

not used

register 6 0x80010018

GDM:

3..0 - GDM_link_data_processing:i_status_select

CDM:

31..0 - CDM_link_data_processing:o_error_count

register 7 0x8001001c

GDM:

31..0 - GDM_link_data_processing:o_status_vector

CDM:

31..0 - CDM_link_data_processing:o_error_count

GDM, CDM, VX packet communications

• timestamp math

1 clock is 8 ns is 125 MHz
8 bits of clocks is 256 clocks is 2048 ns is ~2 usec
16 bits of clocks is ~500 usec is 0.5 msec
24 bits of clocks is ~134 msec
32 bits of clocks is ~34 sec
40 bits of clocks is ~8.7 ksec is 2.4 hours
48 bits of clocks is ~625 hours is ~26 days
56 bits of clocks is ~6.6 kdays is ~18 kyears
62 bits of clocks is ~10 Mhours is 427 kdays is ~1.1 kyears
64 bits of clocks is ~4.4 kyears

• 0x02 - TRG packet, 8 bytes, 80 adc clocks, 640 ns on lvds link

0 - 0x02
1 - trg_counter[7:0]
2 - ts64 low byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 high byte 3
6 - trg_in_latch[7:0]
7 - trg_in_latch[15:8]

• 0x03 - HITMAP_TRG packet, 12 bytes, 120 adc clocks, 960 ns on lvds link

0 - 0x03
1 - trg_counter[7:0]
2 - ts64 low byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 high byte 3
6 - vx_bitmap[7:0]
7 - vx_bitmap[15:8]
8 - vx_bitmap[23:16]
9 - vx_bitmap[31:24]
10 - vx_bitmap[39:32]
11 - vx_bitmap[47:40]

• 0x10 - TSM packet, 26 bytes, 260 adc clocks, 2080 ns on lvds link

0 - 0x10
1 - tsm_counter[7:0]
2 - gdm_ts64 low byte 0
3 - 1
4 - 2
5 - 3
6 - 4
7 - 5
8 - 6
9 - gdm_ts64 high byte 7
10 - gps_ts64 low byte 0
11 - 1
12 - 2
13 - 3
14 - 4
15 - 5
16 - 6
17 - gps_ts64 high byte 7
18 - gps_data64 low byte 0
19 - 1
20 - 2
21 - 3
22 - 4
23 - 5
24 - 6
25 - gps_data64 high byte 7

• 0x81 - VX hitmap packet, 10 bytes, 100 adc clocks, 800 ns on lvds link, 48 ns on fiber link

0 - 0x81
1 - VX ID
2 - hitmap low byte, nits 7:0
3 - 15:8
4 - 23:16
5 - 31:24
6 - ...:32
7 - ...
8 - ...
9 - hitmap low byte, bits 63:...

• 0x82 - CDM hitmap packet, 108 bytes, not sent on lvds link, not sent on fiber link

0 - 0x82
1 - cdm_hitmap_trigger_counter[7:0]
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - cdm_hitmap_or12 byte 0 (7:0)
11 - cdm_hitmap_or12 byte 1 (11:8) plus 4 bits: 12=0, 13=0, 14=0, 15=cdm_hitmap_grand_or
12 - cdm_hitmap_data, low byte, 12*64 bits = 768 bits = 48 words = 96 bytes
...
107 - cdm_hitmap_data, high byte

AXI bus timing

• AXI 100 MHz clock, 10 ns, 32-bit data
• AXI single-dword read: 36 clock repeat rate, 360 ns is 2.777 MHz, 4 bytes per transfer is 11.11 Mbytes/sec
• AXI single-qword read: 13 clock repeat rate, 130 ns is 7.7 MHz, 4 bytes per transfer is 30 Mbytes/sec, not accounting for the gap
• AXI memcpy read: 4 transfers at 13 clocks, gap, 4 transfers at 13 clocks, gap, etc. below 30 Mbytes/sec.

• AXI single-dword write: 20 clocks repeat rate, 200 ns is 5 MHz, 4 bytes per transfer is 20 Mbytes/sec
• AXI single-qword write: 13+20 clocks repeat rate, 330 ns is 3 MHz, 16 bytes per burst is 48 Mbytes/sec

AXI bus addresses

• see AXI/AMBA addresses assigned inthe FPGA project: (s_axi/reg0 is the DS-DM AXI registers)

daq00:ds-dm-gcdm$ grep assign_bd_address scripts/GDM_CDM_XU8_bd.tcl
  assign_bd_address -offset 0x80010000 -range 0x00004000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs axi_register_interfa_0/s_axi/reg0] -force
  assign_bd_address -offset 0x000400000000 -range 0x40000000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs ddr4/C0_DDR4_MEMORY_MAP/C0_DDR4_ADDRESS_BLOCK] -force
  assign_bd_address -offset 0x80000000 -range 0x00010000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs system_management_wiz/S_AXI_LITE/Reg] -force
daq00:ds-dm-gcdm$ 

• see AXI/AMBA addresses exported from FPGA project to Linux kernel: (uio for debug bridge should say "debug bridge")

root@gdm0:~# cat /sys/class/uio/uio*/name
axi-pmon
axi-pmon
axi-pmon
axi-pmon
root@gdm0:~# 
root@gdm0:~# ls -l /sys/class/uio/
total 0
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio0 -> ../../devices/platform/amba/ffa00000.perf-monitor/uio/uio0
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio1 -> ../../devices/platform/amba/fd0b0000.perf-monitor/uio/uio1
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio2 -> ../../devices/platform/amba/fd490000.perf-monitor/uio/uio2
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio3 -> ../../devices/platform/amba/ffa10000.perf-monitor/uio/uio3
root@gdm0:~# ls -l /sys/class/uio/../../devices/platform/amba/
total 0
-rw-r--r-- 1 root root 4096 Oct 18 19:37 driver_override
drwxr-xr-x 3 root root    0 Oct 18 01:36 fd070000.memory-controller
drwxr-xr-x 4 root root    0 Oct 18 01:36 fd0b0000.perf-monitor
drwxr-xr-x 4 root root    0 Oct 18 01:36 fd400000.zynqmp_phy
drwxr-xr-x 4 root root    0 Oct 18 01:36 fd490000.perf-monitor
drwxr-xr-x 4 root root    0 Oct 18 01:36 fd500000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 fd510000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 fd520000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 fd530000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 fd540000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 fd550000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 fd560000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 fd570000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 fd6e0000.cci
drwxr-xr-x 5 root root    0 Oct 18 01:36 ff000000.serial
drwxr-xr-x 4 root root    0 Oct 18 01:36 ff020000.i2c
drwxr-xr-x 6 root root    0 Oct 18 01:36 ff0a0000.gpio
drwxr-xr-x 4 root root    0 Oct 18 01:36 ff0b0000.ethernet
drwxr-xr-x 4 root root    0 Oct 18 01:36 ff0e0000.ethernet
drwxr-xr-x 4 root root    0 Oct 18 01:36 ff0f0000.spi
drwxr-xr-x 5 root root    0 Oct 18 01:36 ff160000.mmc
drwxr-xr-x 5 root root    0 Oct 18 01:36 ff170000.mmc
drwxr-xr-x 3 root root    0 Oct 18 01:36 ff960000.memory-controller
drwxr-xr-x 4 root root    0 Oct 18 01:36 ff9d0000.usb0
drwxr-xr-x 4 root root    0 Oct 18 01:36 ffa00000.perf-monitor
drwxr-xr-x 4 root root    0 Oct 18 01:36 ffa10000.perf-monitor
drwxr-xr-x 4 root root    0 Oct 18 01:36 ffa50000.ams
drwxr-xr-x 5 root root    0 Oct 18 01:36 ffa60000.rtc
drwxr-xr-x 4 root root    0 Oct 18 01:36 ffa80000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 ffa90000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 ffaa0000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 ffab0000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 ffac0000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 ffad0000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 ffae0000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 ffaf0000.dma
-r--r--r-- 1 root root 4096 Oct 18 19:37 modalias
lrwxrwxrwx 1 root root    0 Oct 18 19:37 of_node -> ../../../firmware/devicetree/base/amba
drwxr-xr-x 2 root root    0 Oct 18 19:37 power
lrwxrwxrwx 1 root root    0 Oct 18 01:36 subsystem -> ../../../bus/platform
-rw-r--r-- 1 root root 4096 Oct 18 01:36 uevent
root@gdm0:~# 

Build firmware

Build from git clone

THESE ARE K.O.'s NOTES FOR CREATING THE PETALINUX DIRECTORY.

THEY DO NOT WORK!

COPY PETALINUX FROM A WORKING PROJECT AND USE "make gdm" and "make cdm" AS DESCRIBED BELOW.

• git clone git@edev-group.triumf.ca:fw/exp/darkside/gcdm.git
• #Makefile change VIVADO_SETTINGS_SCRIPT := /opt/Xilinx/Vivado/2022.1/settings64.sh
• #. /opt/Xilinx/Vivado/2022.1/settings64.sh
• . /opt/Xilinx/Vivado/2020.2/settings64.sh
• make clean
• make all_from_scratch
• . /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
• make petalinux_create
• make petalinux_rebuild_new_hw_des
• bomb out: The TMPDIR: /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp can't be located on nfs.
• mkdir /tmp/build_tmp
• rm -rf /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp/
• ln -s /tmp/build_tmp /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp
• try again
• grinds, loads a whole bunch of packages...
• finishes with desire to copy things to /tftpboot
• make sdcard_cp_to wants to copy files from PetaLinux_GDM_CDM/images/linux/ to SD card

Build firmware

NOTE: directory Petalinux_GDM_CDM should already exist!

#. /opt/Xilinx/Vivado/2020.2/settings64.sh
. /opt/Xilinx/Vivado/2022.2/settings64.sh
. /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
make clean_gdm   # remove gdm build tree
make gdm         # build or rebuild GDM
make copy_gdm    # copy to gdm0
make clean_cdm   # remove cdm build tree
make cdm         # build or rebuild CDM
make copy_cdm    # copy to cdm0 and cdm1

copy to SD card:

open a root shell
format 16 GB Sd card per above
cd .../ds-dm-gcdm
make copy

build times

CDM 12-june-2023 69aabc1c25130d970bc375aca684bd68849e6685
daq13 AMD-5700G 1688.61user 399.33system 23:28.84elapsed 148%CPU
dsdaqgw AMD-7700 1090.55user 247.34system 16:03.55elapsed 138%CPU
dsdaqgw AMD-7700 CDM incremental 196.68user 67.62system 7:35.42elapsed 58%CPU
dsdaqgw AMD-7700 CDM incremental 684.72user 94.17system 7:30.17elapsed 173%CPU
dsdaqgw AMD-7700 GDM incremental 849.84user 99.79system 9:04.56elapsed 174%CPU

make bootable sd card

format the sd card

this only needs to be done once

• become root@dsdaqgw
• cd ~olchansk/git/ds-dm-gcdm
• use "lsblk" to identify the SD card (should show as 8/16/32 GB block device)/ /dev/sdd in this case
• make sdcard_format SDCARD_DEVICE=/dev/sdd
• disconnect sd card, reconnect the sd card (to detect new partition tables, etc)

copy boot files to the sd card

• as root: identify partition labels, run "blkid", should say "BOOT", "rootfs" and "data"
• mount

mkdir -p /mnt/BOOT
mount -L BOOT /mnt/BOOT
cp PetaLinux_GDM_CDM/images/linux/BOOT_{GDM,CDM}.BIN  /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/boot.scr            /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/image.ub            /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/uboot.env           /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/uboot-redund.env    /mnt/BOOT/
umount /mnt/BOOT
eject /dev/sdX

boot messages

Xilinx Zynq MP First Stage Boot Loader 
Release 2020.2   Sep 24 2022  -  13:29:15
NOTICE:  ATF running on XCZU4CG/silicon v4/RTL5.1 at 0xfffea000
NOTICE:  BL31: v2.2(release):xlnx_rebase_v2.2_2020.3
NOTICE:  BL31: Built : 18:02:46, Sep 28 2022


U-Boot 2020.01 (Sep 28 2022 - 18:03:39 +0000)

Model: DarkSide 20k DM
Board: Xilinx ZynqMP
DRAM:  2 GiB
usb dr_mode not found
PMUFW:  v1.1
EL Level:       EL2
Chip ID:        zu4
NAND:  0 MiB
MMC:   mmc@ff160000: 0, mmc@ff170000: 1
In:    serial@ff000000
Out:   serial@ff000000
Err:   serial@ff000000
Bootmode: SD_MODE1
Reset reason:   SOFT 
Net:   
ZYNQ GEM: ff0b0000, mdio bus ff0b0000, phyaddr -1, interface rgmii-id

Warning: ethernet@ff0b0000 (eth0) using random MAC address - d6:62:5f:13:00:44
eth0: ethernet@ff0b0000
ZYNQ GEM: ff0e0000, mdio bus ff0e0000, phyaddr -1, interface rgmii-id
Could not get PHY for eth1: addr -1

Hit any key to stop autoboot:  0 
ZynqMP> 
CTRL-A Z for help | 115200 8N1 | NOR | Minicom 2.7.1 | VT102 | Online 122:2 | ttyACM0                                       

load FPGA from u-boot

• https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/124682257/U-Boot+FPGA+Driver

ZynqMP> fpga info
Xilinx Device
Descriptor @ 0x000000007fddb2c0
Family:         ZynqMP PL
Interface type: csu_dma configuration interface (ZynqMP)
Device Size:    1 bytes
Cookie:         0x0 (0)
Device name:    zu4
Device Function Table @ 0x000000007fda5fe8
PCAP status     0xa0002fde
ZynqMP> 

• cp CDM_XU8_top.bit /tftpboot/fpga.bit

dhcp
tftpb 0x10000000 fpga.bit
fpga loadb 0 0x10000000 ${filesize}

ZynqMP> dhcp
BOOTP broadcast 1
DHCP client bound to address 192.168.0.100 (1 ms)
*** Warning: no boot file name; using 'C0A80064.img'
Using ethernet@ff0b0000 device
TFTP from server 192.168.0.1; our IP address is 192.168.0.100
Filename 'C0A80064.img'.
Load address: 0x8000000
Loading: *
TFTP error: 'file /tftpboot/C0A80064.img not found for 192.168.0.100' (1)
Not retrying...
ZynqMP> tftpb 0x10000000 fpga.bit
Using ethernet@ff0b0000 device
TFTP from server 192.168.0.1; our IP address is 192.168.0.100
Filename 'fpga.bit'.
Load address: 0x10000000
Loading: #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         ############
         6.2 MiB/s
done
Bytes transferred = 7797807 (76fc2f hex)
ZynqMP> fpga loadb 0 0x10000000 ${filesize}
  design filename = "CDM_XU8_top;UserID=0XFFFFFFFF;Version=2022.2"
  part number = "xczu4cg-fbvb900-1-e"
  date = "2024/08/14"
  time = "14:18:22"
  bytes in bitstream = 7797692
zynqmp_align_dma_buffer: Align buffer at 0000000010000073 to 000000000fffff80(swap 0)
ZynqMP> 

load FPGA from Linux

• https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841847/Solution+ZynqMP+PL+Programming

this will reset the CPU

cp fpga.bit /lib/firmware/
echo fpga.bit > /sys/class/fpga_manager/fpga0/firmware

• make .bin file:

bootgen -image CDM_XU8_top.bif -arch zynqmp -o ./fpga.bin -w

• cat CDM_XU8_top.bif

dsdaqgw:ds-dm-gcdm$ cat CDM_XU8_top.bif
all:
{
        [destination_device = pl] ./Vivado_CDM_XU8/CDM_XU8.runs/impl_1/CDM_XU8_top.bit
}
dsdaqgw:ds-dm-gcdm$ 

this will reset the CPU

cp fpga.bin /lib/firmware/
echo fpga.bin > /sys/class/fpga_manager/fpga0/firmware

this will reset the CPU

root@dsdm:~# ./fpgautil -b fpga.bin -f Full

this will reset the CPU

DTSO file from here: https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841847/Solution+ZynqMP+PL+Programming
echo 0 > /sys/class/fpga_manager/fpga0/flags
mount -t configfs configfs /configfs
root@dsdm:~# dtc -O dtb -o fpga.dtbo -b 0 -@ fpga.dtso
root@dsdm:~# cp fpga.dtbo /lib/firmware/
root@dsdm:~# cp fpga.bit /lib/firmware/
root@dsdm:~# rmdir /configfs/device-tree/overlays/fpga
root@dsdm:~# mkdir /configfs/device-tree/overlays/fpga
root@dsdm:~# echo -n "fpga.dtbo" > /configfs/device-tree/overlays/fpga/path 

fpgautil

• https://github.com/Xilinx/meta-xilinx/blob/master/meta-xilinx-core/recipes-bsp/fpga-manager-script/files/fpgautil.c

git clone https://github.com/Xilinx/meta-xilinx.git
cd meta-xilinx/meta-xilinx-core/recipes-bsp/fpga-manager-script/files/
scp fpgautil.c root@dsdm:
ssh root@dsdm
make fpgautil
ls -l ./fpgautil

root@dsdm:~# ls -l ./fpgautil
-rwxr-xr-x 1 root root 72256 Aug 16 00:15 ./fpgautil
root@dsdm:~# ./fpgautil 

fpgautil: FPGA Utility for Loading/reading PL Configuration

Usage:	fpgautil -b <bin file path> -o <dtbo file path>

Options: -b <binfile>		(Bin file path)
         -o <dtbofile>		(DTBO file path)
         -f <flags>		Optional: <Bitstream type flags>
				   f := <Full | Partial > 
         -n <Fpga region info>  FPGA Regions represent FPGA's
                                and partial reconfiguration
                                regions of FPGA's in the
                                Device Tree
				Default: <full>
	  -s <secure flags>	Optional: <Secure flags>
				   s := <AuthDDR | AuthOCM | EnUsrKey | EnDevKey | AuthEnUsrKeyDDR | AuthEnUsrKeyOCM | AuthEnDevKeyDDR | AuthEnDevKeyOCM>
	  -k <AesKey>		Optional: <AES User Key>
	  -r <Readback> 	Optional: <file name>
				Default: By default Read back contents will be stored in readback.bin file
	  -t			Optional: <Readback Type>
				   0 - Configuration Register readback
				   1 - Configuration Data Frames readback
				Default: 0 (Configuration register readback)
	  -R 			Optional: Remove overlay from a live tree
 
Examples:
(Load Full bitstream using Overlay)
fpgautil -b top.bit.bin -o can.dtbo -f Full -n full 
(Load Partial bitstream using Overlay)
fpgautil -b rm0.bit.bin -o rm0.dtbo -f Partial -n PR0
(Load Full bitstream using sysfs interface)
fpgautil -b top.bit.bin -f Full
(Load Partial bitstream using sysfs interface)
fpgautil -b rm0.bit.bin -f Partial
(Load Authenticated bitstream through the sysfs interface)
fpgautil -b top.bit.bin -f Full -s AuthDDR 
(Load Parital Encrypted Userkey bitstream using Overlay)
fpgautil -b top.bit.bin -o pl.dtbo -f Partial -s EnUsrKey -k <32byte key value>
(Read PL Configuration Registers)
fpgautil -b top.bit.bin -r
(Remove Partial Overlay)
fpgautil -R -n PR0
(Remove Full Overlay)
fpgautil -R -n full
Note: fpgautil -R is responsible for only removing the dtbo file from the livetree. it will not remove the PL logic from the FPGA region.
 
root@dsdm:~# 

fw_printenv

to access u-boot environment from Linux:

• apt install -y libubootenv-tool
• create /etc/fw_env.config

/media/BOOT/uboot.env 0 0x40000
/media/BOOT/uboot-redund.env 0 0x40000

• if uboot.env files do not exist, run "saveenv" from u-boot command prompt
• fw_printenv and fw_setenv should work

Boot from network

u-boot

ZynqMP> setenv bootcmd run bootcmd_dhcp
ZynqMP> saveenv
ZynqMP> reset

boot.scr

# boot.scr
# mkimage -C none -A arm -T script -d boot.scr boot.scr.uimg
echo Loading FPGA!
#tftpb 0x10000000 fpga.bit
tftpb 0x10000000 {ipaddr}.bit
fpga loadb 0 0x10000000 ${filesize}
echo Booting Linux!
run bootcmd_pxe
echo Done!

mkimage -C none -A arm -T script -d boot.scr boot.scr.uimg

tftpboot

cp /home/olchansk/git/ds-dm-gcdm/boot.scr.uimg /tftpboot
ln -s /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/images/linux /tftpboot/xilinx-dsdm
mkdir /tftpboot/pxelinux.cfg
cat > /tftpboot/pxelinux.cfg/default-arm-zynqmp <<EOF
LABEL Linux
   KERNEL xilinx-dsdm/Image
   FDT xilinx-dsdm/system.dtb
   #INITRD rootfs.cpio.gz.u-boot
EOF

boot sequence

• xilinx magic load BOOT.BIN from SD card
• load FPGA form BOOT.BIN
• load and run u-boot from BOOT.BIN or from image.ub
• u-boot load environment from ??? probably SD card uboot-redund.env, this includes our bootcmd
• run bootcmd which run bootcmd_dhcp which does:
• from /tftpboot:
• load and run boot.scr.uimg which does:
• load FPGA image xilinx-dsdm/${ipaddr}.bit
• run bootcmd_pxe which does:
• load pxelinux.cfg/default-arm-zynqmp which does:
• load xilinx-dsdm/Image ### this is the linux kernel
• load xilinx-dsdm/system.dtb ### this is the device tree
• start linux kernel
• linux kernel does dhcp
• linux kernel does nfs mount /nfsroot/%s,vers=3,tcp ### %s is replaced by the hostname supplied by DHCP
• userland starts and runs to console and ssh login.

Xilinx ILA

References:

• https://github.com/Xilinx/XilinxVirtualCable/tree/master
• https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/644579329/Xilinx+Virtual+Cable
• https://support.xilinx.com/s/article/974879?language=en_US
• https://docs.xilinx.com/v/u/en-US/pg245-debug-bridge
• https://docs.xilinx.com/r/2020.2-English/ug908-vivado-programming-debugging/Initializing-Vivado-IDE-hw_server
• (we do not use this one) https://github.com/paulscherrerinstitute/xvcSupport/blob/master/README.md

Build xvcserver_cdm.exe: (it is built as a static executable, can be copied and run anywhere)

ssh dsdaq@gdm0
cd /home/dsdaq/online/ds-dm-software
git pull ### get latest version
make xvcserver_cdm.exe
ssh root@gdm0
/home/dsdaq/online/ds-dm-software/xvcserver_cdm.exe
INFO: To connect to this xvcServer instance, use url: TCP:gdm0:2542

To activate and use the vivado logic analyzer:

• data path: vivado -> hw_server -> xvcserver -> mmap axi bus -> debug bridge -> jtag -> ILA
• define ILAs in the code
• instantiate the xilinx debug bridge at AXI bus address 0x80020000 (FIXME!!! this collides with Ian's AXI addresses)
• build and boot the new FPGA firmware. updating the linux kernel is not necessary.
• login root@gdm0, run: /home/dsdaq/online/ds-dm-software/xvcserver_cdm.exe -v ### with "-v" for the first time to see that vivado does connect to it, without "-v", normally.
• login dsdaqgw, run: hw_server -s tcp:localhost:3121 -e "set auto-open-servers xilinx-xvc:gdm0:2542" ### tells us to connect to port localhost:3121
• login dsdaqgw, run vivado, open project, open hardware manager, open target, open new target, "connect to remote server", hostname "localhost", port "3121", next (bombs, try again, 3 times), popup add virtual cable, enter hostname "gdm0" port "2542", "ok", it shows in "hardware targets", "next", "finish", error popup "[Common 17-163] Missing value for option 'objects', please type 'set_property -help' for usage info", ignore it, in "hardware", right click the "gdm0" one, open target, under "hardware" and "debug bridge" we should see all the ILAs, under "hardware device properties", the "probes file" should have the ".ltx" file generated by vivado "Vivado_GDM_XU8/GDM_XU8.runs/impl_1/debug_nets.ltx", click on an ILA, a waveform should open.

Software

• ssh cdm0 # or gdm0
• sudo apt install i2c-tools libi2c-dev
• git clone https://bitbucket.org/team-ds-dm/ds-dm-software
• cd ds-dm-software
• make

test_cdm.exe

CDM SFP status

# /home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0xbb2f0ae7
CDM firmware 0xbb2f0ae7
arg 1: [--sfp]
Polling SFP status...
identifier 0x03
connector  0x07
encoding   0x01
wavelength 0x0352 (850 nm)
vendor_name [FINISAR CORP.   ]
vendor_pn   [FTLF8526P3BNL   ]
vendor_rev  [A   ]
vendor_sn   [N3AB9M8         ]
vendor_date [200319  ]
dm_type    0x68
temp 29.0 C
vcc  3.323 V
tx_bias  7.250 mA
tx_power 478.4 uW
rx_power 2.3 uW
SFP good 1, status: temp 30.6 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 818 uW
...

GDM QSFP status

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --qsfp3 --qsfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x53aee418
CDM firmware 0x53aee418
arg 1: [--qsfp3]
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
 gpio-378 (                    |sysfs               ) out hi 
 gpio-379 (                    |sysfs               ) out hi 
 gpio-381 (                    |sysfs               ) out lo 
 gpio-382 (                    |sysfs               ) out hi 
arg 2: [--qsfp]
Polling QSFP status...
identifier 0x0d
status     0x02
los        0x8f
temp       28.2 C
vcc        3.323 V
rx_power     0.1   0.1   0.1   0.1 uW
tx_bias      7.6   7.6   7.6   0.0 mA
tx_power   792.2 773.8 823.0   0.1 uW
vendor_name [FINISAR CORP    ]
vendor_pn   [FTL410QD4C      ]
vendor_rev  [A ]
wavelength  850
max_temp    70 C
vendor_sn   [X79AC0R         ]
vendor_date [220309  ]
QSFP good 1, status: temp 27.7 C, los 0x8b, tx_bias 7.5 7.6 7.6 0.0 mA, tx_power 792 772 821   0 uW, rx_power   0   0 466   0 uW

GDM clock status

/home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks

• clock chip not loaded, not running:

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x0f 0x16 0x04 0x00 0x01 0x68 0x09 0x00 0x02 0xf2 0x00 0x1f 0xf0 0x22 0xf2, 0x507: 0x00, 0x52A: 0x00, 0x53F: 0x04
Clock chip state 0, status:  SYSINCAL XAXB_ERR LOL CAL_PLL IN0 IN_SEL_0 FASTLOCK_STATUS

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x1f7caf52 (528265042) should be ~125 MHz
0x1034 rx_clk:             0x00000000 (0) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x1f7caf52 (528265042) should be ~125 MHz
0x103C tx_clk:             0x00000000 (0) should be ~125 MHz
0x1040 clk_50MHz:          0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk:         0x05f5e100 (100000000) should be 100 MHz exactly

• clock chip good (IN0 - external 10 MHz clock)

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x44 0x00 0xd0 0x01 0x1f 0xff 0x22 0xf2, 0x507: 0x3f, 0x52A: 0x01, 0x53F: 0x00
Clock chip state 1, status:  LOS_IN2 OOF_IN2 IN0 IN_SEL_REGCTRL IN_SEL_0

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735a3b (125000251) should be ~125 MHz
0x1034 rx_clk:             0x07735a3c (125000252) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a3b (125000251) should be ~125 MHz
0x103C tx_clk:             0x07735a3c (125000252) should be ~125 MHz
0x1040 clk_50MHz:          0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk:         0x05f5e100 (100000000) should be 100 MHz exactly

CDM clock status

/home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks

• clock chip not loaded, not running:

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x0f 0x16 0x04 0x00 0x01 0x68 0x19 0x00 0x02 0xf2 0x00 0x1f 0xf0 0x22 0xf2, 0x507: 0x00, 0x52A: 0x00, 0x53F: 0x04
Clock chip state 0, status:  SYSINCAL XAXB_ERR LOL CAL_PLL IN0 IN_SEL_0 FASTLOCK_STATUS

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735851 (124999761) should be ~125 MHz
0x1034 rx_clk:             0x00000000 (0) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x0127fefa (19398394) should be ~125 MHz
0x103C tx_clk:             0x00000000 (0) should be ~125 MHz
0x1040 clk_50MHz:          0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk:         0x05f5e100 (100000000) should be 100 MHz exactly

• clock chip uses internal clock (IN1 - internal oscillator), observe rx_clk frequency is not same as others

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0x7f, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status:  IN1 IN_SEL_1 HOLD_HIST_VALID

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735853 (124999763) should be ~125 MHz
0x1034 rx_clk:             0x07735b0a (125000458) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735853 (124999763) should be ~125 MHz
0x103C tx_clk:             0x07735852 (124999762) should be ~125 MHz
0x1040 clk_50MHz:          0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk:         0x05f5e100 (100000000) should be 100 MHz exactly

• clock chip uses SFP recovered clock (IN2 - sfp rx recovered clock), observe mgt_rx_ref_clk_raw (CDM 125 MHz oscillator) is different from others (SFP RX recovered clock)

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0xbf, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status:  IN2 IN_SEL_1 HOLD_HIST_VALID
^C
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773581b (124999707) should be ~125 MHz
0x1034 rx_clk:             0x07735ad6 (125000406) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735ad7 (125000407) should be ~125 MHz
0x103C tx_clk:             0x07735ad6 (125000406) should be ~125 MHz
0x1040 clk_50MHz:          0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk:         0x05f5e100 (100000000) should be 100 MHz exactly
^C

CDM link status, PRBS test mode

# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2

• fiber disconnected, no link:

CDM firmware:    0xbb2f0ae7
0x1000 SFP c.c. status: 0x00000031
    CLK_IN_SEL_LS   0x1
    CLK_EXT_SEL_LS  0
    CLK_CLK_RSTn_LS 0
    CLK_LOSXTn_LS   1
    CLK_LOLn_LS     1
    CLK_INTn_LS     0
0x1008 SFP link reset:  0x00000000
0x1010 SFP link status: 0x00000025
    sfp_mod_absent_N       1
    sfp_rx_los_N           0
    link_power_good        1
    rx_link_up             0
    rx_receiving_data      0
    rx_error               1
    rx_lnk_up_and_running  0
    tx_link_up             0
    tx_sending_data        0
    tx_link_up_and_running 0
    link_up_and_running    0
0x1014 SFP link data:   0x466a8187
    rx_data     0x8187
    k28p1_k28p5 0
    rx_ctrl0    0x1
    rx_ctrl1    0x1
    rx_ctrl3    0x1
    tx_state    0x1
    rx_state    0x6
    rx_receiving_data 0
    rx_link_up        0
    rx_error          1
    rx_link_rst       0
0x2000 link test mode: 0x00000200, seconds: 0x00079093, errors: 0xffffffff

• fiber connected, good link:

CDM firmware:    0xbb2f0ae7
0x1000 SFP c.c. status: 0x000000b2
    CLK_IN_SEL_LS   0x2
    CLK_EXT_SEL_LS  0
    CLK_CLK_RSTn_LS 0
    CLK_LOSXTn_LS   1
    CLK_LOLn_LS     1
    CLK_INTn_LS     0
0x1008 SFP link reset:  0x00000000
0x1010 SFP link status: 0x000007dc
    sfp_mod_absent_N       0
    sfp_rx_los_N           0
    link_power_good        1
    rx_link_up             1
    rx_receiving_data      1
    rx_error               0
    rx_lnk_up_and_running  1
    tx_link_up             1
    tx_sending_data        1
    tx_link_up_and_running 1
    link_up_and_running    1
0x1014 SFP link data:   0x35c02774
    rx_data     0x2774
    k28p1_k28p5 0
    rx_ctrl0    0x0
    rx_ctrl1    0x0
    rx_ctrl3    0x0
    tx_state    0x3
    rx_state    0x5
    rx_receiving_data 1
    rx_link_up        1
    rx_error          0
    rx_link_rst       0
0x2000 link test mode: 0x00000200, seconds: 0x00078a8c, errors: 0x00000000

GDM link status, PRBS test mode

1 link connected, no errors:
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
GDM firmware:    0x6b2ee010
0x1014: 0x00000008, 18: 0x00000008, 1C: 0x00000000, 24: 0x00000fff
0x2000: 0x00000200, time: 0x00078aa4, errors:
0xffffffff 0xffffffff 0xffffffff
0x00000000 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff

CDM link status

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test0

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0xbf, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status:  IN2 IN_SEL_1 HOLD_HIST_VALID

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x077357a0 (124999584) should be ~125 MHz
0x1034 rx_clk:             0x07735a5c (125000284) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a5b (125000283) should be ~125 MHz
0x103C tx_clk:             0x07735a5b (125000283) should be ~125 MHz
0x1040 clk_50MHz:          0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk:         0x05f5e100 (100000000) should be 100 MHz exactly

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM firmware:    0x6d2ef81a
0x1000 SFP c.c. status: 0x000000b2
    CLK_IN_SEL_LS   0x2
    CLK_EXT_SEL_LS  0
    CLK_CLK_RSTn_LS 0
    CLK_LOSXTn_LS   1
    CLK_LOLn_LS     1
    CLK_INTn_LS     0
0x1008 SFP link reset:  0x00000000
0x1010 SFP link status: 0x000007dc
    sfp_mod_absent_N       0
    sfp_rx_los_N           0
    link_power_good        1
    rx_link_up             1
    rx_receiving_data      1
    rx_error               0
    rx_lnk_up_and_running  1
    tx_link_up             1
    tx_sending_data        1
    tx_link_up_and_running 1
    link_up_and_running    1
0x1014 SFP link data:   0x35c6bcbc
    rx_data     0xbcbc
    k28p1_k28p5 0
    rx_ctrl0    0x3
    rx_ctrl1    0x0
    rx_ctrl3    0x0
    tx_state    0x3
    rx_state    0x5
    rx_receiving_data 1
    rx_link_up        1
    rx_error          0
    rx_link_rst       0
0x2000 link test mode: 0x00000000, seconds: 0x00001671, errors: 0xffffffff

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 15
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
reg[15] is 0x0033bcbc (3390652)

GDM link status

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test0

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x44 0x00 0xd0 0x01 0x1f 0xff 0x22 0xf2, 0x507: 0x3f, 0x52A: 0x01, 0x53F: 0x02
Clock chip state 1, status:  LOS_IN2 OOF_IN2 IN0 IN_SEL_REGCTRL IN_SEL_0 HOLD_HIST_VALID

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735a0a (125000202) should be ~125 MHz
0x1034 rx_clk:             0x07735a0a (125000202) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a0a (125000202) should be ~125 MHz
0x103C tx_clk:             0x07735a0a (125000202) should be ~125 MHz
0x1040 clk_50MHz:          0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk:         0x05f5e100 (100000000) should be 100 MHz exactly

register 0x1018 bit 0x800

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM firmware:    0x6d2ebce6
0x1014: 0x00000800, 18: 0x00000800, 1C: 0x00000000, 24: 0x00000fff
0x2000: 0x00000000, time: 0x00003d2f, errors:
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff

link data alternates 0xbcbc and 0x1cbc

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 22
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
reg[22] is 0xbcbc93ab (-1128492117)

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 22
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
reg[22] is 0x1cbc1aaf (482089647)
root@gdm0:~# 

Run trg and tsm

on the GDM:

ssh root@gdm00
./test_cdm.exe --gdm-clocks
./test_cdm.exe --load-cc
./test_cdm.exe --cc
./test_cdm.exe --reset-mgt
./test_cdm.exe --gdm-clocks
./test_cdm.exe --writereg 2 0xff # enable LEMO NIM inputs
./test_cdm.exe --writereg 3 0xba54 # LEDs: lemo1, lemo2, trg, tsm
./test_cdm.exe --writereg 4 0x99 $ # LEMO out is trg_in_pulse
./test_cdm.exe --writereg 9 0x32010 # trg and tsm from trg_pulser and tsm_pulse
./test_cdm.exe --writereg 24 1250000 # trg pulser 100 Hz
./test_cdm.exe --writereg 26 125000000 # tsm pulser 1 Hz
./test_cdm.exe --writereg 23 0x40000000 # route trg_in and tsm_in to qsfp tx bits 0 and 1
./test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz

on the CDM:

ssh root@cdm01
./test_cdm.exe --cdm-clocks
./test_cdm.exe --load-cc
./test_cdm.exe --reset-mgt
./test_cdm.exe --cdm-clocks
./test_cdm.exe --cdm-link # issue --reset-mgt on CDM and GDM until link is good
./test_cdm.exe --writereg 2 0xff # enable LEMO NIM inputs
./test_cdm.exe --writereg 3 0xba54 # enable LEDs: lemo1, lemo2, trg, tsm
./test_cdm.exe --writereg 4 0x99 # enable LEMO output trg_in
./test_cdm.exe --writereg 9 0x8040 # enable trg_in and tsm_in from sfp rx bits 0 and 1
./test_cdm.exe --writereg 7 0xff00 # drive VX LVDS lines to logic level 0
./test_cdm.exe --writereg 8 0x2 # VX LVDS with trg and tsm, misrouted in second VX
#./test_cdm.exe --writereg 8 0x7 # VX LVDS with tsm
./test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz

switch GDM and CDM to packetizer trg and tsm:

on the GDM:

/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x00000000

on the CDM:

/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x0804
/home/dsdaq/online/ds-dm-software/test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz

Run packet loopback

GDM CPU -> fifo_to_fpga -> GDM QSFP -> CDM SFP -> fifo_from_fpga -> CDM CPU

On the GDM: (CDM is connected to first QSFP port)

/home/dsdaq/online/ds-dm-software/test_cdm.exe --load-cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --reset-mgt
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 32 1 ### tell GDM to use first QSFP port
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link ### confirm link status is "3"
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 29 2 ### enable GDM packet data injection
/home/dsdaq/online/ds-dm-software/test_cdm.exe --test-fifo-write-loop

One the CDM:

/home/dsdaq/online/ds-dm-software/test_cdm.exe --load-cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --reset-mgt
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 29 3 ### enable CDM packet data injection
/home/dsdaq/online/ds-dm-software/test_cdm.exe --test-fifo-read

dsvslice integration

VX setup

• general
  • Start acq from user code = y, all others = n (as of Mar 2023: will start when first trigger received)
  • Use NIM IO = y
  • Use external clock = y
  • LVDS quartet is input = n, y, n, y
  • LVDS quartet mode = User, User, User, User

• trigger from front panel NIM:
  • Trigger on external signal = y, all others = n
  • connect CDM EXT_OUT(2) to VX "TrigIn"

• trigger from LVDS "Sync" mode
  • Trigger on LVDS Sync signal = y, all others = n
  • LVDS quartet mode = User, Sync, User, User

• trigger from LVDS "User" mode
  • Trigger on LVDS pair 12 signal = y, all others = n
  • LVDS quartet is input = n, y, n, y
  • LVDS quartet mode = User, User, User, User

GDM setup

• GDM is gdm0
• set inputs to NIM mode
• set outputs to TTL mode (this GDM has wrong NIM output circuit)
• use top QSFP slot, connect split cable 1 into CDM01, cable 2 into CDM02
• connect non-inverted NIM trigger signal to top-LEMO-left EXT_IN_LV(1)
• connect non-inverted NIM TSM signal to top-LEME-right EXT_IN_LV(2)
• GDM LEDs: TRIG, TSM, trigger enabled, trigger_out
• GDM LEMO_OUT: trigger, trigger

CDM setup

• set CDM LEMO inputs to NIM
• set CDM LEMO outputs to NIM
• CDM01 is cdm0
• CDM02 is cdm1
• connect GDM fiber links to SFP port
• connect 1st VX port of CDM01 to VX1
• connect 1st VX port of CDM02 to VX2
• connect LEMO EXT_OUT(2) to VX "TrigIn", CDM01 to VX1, CDM02 to VX2
• power up
• CDM LEDs: GDM TRIG, GDM TSM, trigger enabled, trigger out
• CDM LEMO_OUT: gdm_trg, trigger

After power up

• start the CDM frontend from the MIDAS "Programs" page. To start manually, see the Start Command on the Programs page.
• CDM frontend should enable the VX clock, disable the trigger
• from the MIDAS status page, goto the CDM page
• outdated: in the CDMx data tables, the 2nd number should read 0x35c08008, if it does not and the last 4 digits randomly change, reset the GDM links
• if the FEs complain - do in order: for GDM, CDM01, CDM02, ..., press "reset mgt" of each board, then press "unreset mgt", if it does not help, STOP HERE
• start a run
• CDM frontend will enable the trigger
• GDM frontend will enable the trigger
• LEDs on the GDM should flash, LEDs on the CDM should flash, TrigIn and TrigOut of the VX should flash
• stop a run
• GDM frontend will disable the trigger
• CDM frontend will disable the trigger

Phase measurement

• pip3 install matplotlib
• pip3 install scipy
• export PYTHONPATH=$HOME/packages/midas/python
• #git clone https://github.com/J033X071C/PhaseMeasurement
• git clone https://bitbucket.org/team-ds-dm/phasemeasurement.git
• cd phasemeasurement
• python3 ./phaseMeasurement.py --help

daq00:PhaseMeasurement$ python3 ./phaseMeasurement.py --help
usage: phaseMeasurement.py [-h] fileName numberEvents numberVX sizeEvents stopEvent minHist maxHist numberBin writeToTXT saveAsPDF

Read data from midas file (in .lz4 format) to calculate phase between the clock of VX1 and VX2

positional arguments:
  fileName      Name of the file we want to read data from (Example: run00389.mid.lz4)
  numberEvents  Number of events recorded in the file
  numberVX      Number of VX used in this run (usually 2...)
  sizeEvents    Number of points per event
  stopEvent     Number of events you want to go through to calculate phase
  minHist       Minimal value for the x axis of the phase measurement histogram (in ns)
  maxHist       Maximal value for the x axis of the phase measurement histogram (in ns)
  numberBin     Number of bins wanted for the generated histogram
  writeToTXT    Write argument as yes to generate text file with results of calculation
  saveAsPDF     Save generated plots to PDF files

optional arguments:
  -h, --help    show this help message and exit
daq00:PhaseMeasurement$ 

• try an old file with
• python3 ./phaseMeasurement.py run00877.mid.lz4 10000 2 10000 500 -20 20 81 yes yes

bin size = 0.494 ns
num_events =  347
mean = -1.705 ns
rms = 3.087 ns
mean_error = 0.166 ns
centroid = -1.706 ns.
width (sigma) = 0.363 ns.
error on the centroid = 0.016558 ns.

• ls -l *.txt *.pdf

dsdaq@dsvslice:~/online/PhaseMeasurement$ ls -l *.txt *.pdf
-rw-rw-r-- 1 dsdaq dsdaq 64728 Dec 14 16:56 run00877.mid.lz4_Plots.pdf
-rw-rw-r-- 1 dsdaq dsdaq   274 Dec 14 16:56 run00877.mid.lz4.txt
dsdaq@dsvslice:~/online/PhaseMeasurement$ 

• scope settings (from email message)

From fcote-lortie@triumf.ca  Thu Dec 15 17:11:46 2022
From: Francis Cote-Lortie <fcote-lortie@triumf.ca>
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: Re: How to use scope as a waveform generator
Date: Fri, 16 Dec 2022 01:11:44 +0000

  1.  Turn the power on (bottom left of the scope)
  2.  Access the waveform generator display by pressing the Gen button (bottom right of the scope)
  3.  The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
  4.  Turn the output on (first option on the display) by pressing on the button. It will go from 0 to 1.

     The settings that we are using right now are:
      Type of waveform: Sine wave
      Offset: 0 V
      Amplitude: 1 Vpp
      Frequency: 50 kHz
      Noise: 0 V
________________________________
From: Francis Cote-Lortie <fcote-lortie@triumf.ca>
Sent: Thursday, December 15, 2022 4:58 PM
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: Re: How to use scope as a waveform generator


  1.  Turn the power on (bottom left of the scope)
  2.  Access the waveform generator display by pressing the Gen button (bottom right of the scope)
  3.  The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
  4.  Turn the output on (first option on the display)

     The settings that we are using right now are:
      Type of waveform: Sine wave
      Offset: 0 V
      Amplitude: 1 Vpp
      Frequency: 50 kHz
      Noise: 0 V
________________________________
From: Francis Cote-Lortie
Sent: Thursday, December 15, 2022 4:54 PM
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: How to use scope as a waveform generator


  1.  Turn the power on (bottom left of the scope)
  2.  Access the waveform generator display by pressing the Gen button (bottom right of the scope)
  3.  The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
  4.  Turn the output on (first option on the display)

Standalone link test

CDM: program clock chip
busybox devmem 0x80011000 32 0x8
busybox devmem 0x80011000 32 0x0
/home/dsdaq/si5394-i2c-file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt  0 0x6b

GDM, CDM: link reset
busybox devmem 0x80011008 32 1

GDM, CDM: release reset
busybox devmem 0x80011008 32 0

CDM: link status (NOTE: SFP LOS and mod_absent are swapped!!!)
busybox devmem 0x80011010 32
0x00000024 <- fiber plugged
0x00000025 <- fiber unplugged
0x00000027 <- SFP unplugged
0x000007DC <- successful link with GDM

CDM: link state machine and data
busybox devmem 0x80011014 32
0x35C06FF6

CDM: set link to counting mode
busybox devmem 0x80012000 32 0x101
busybox devmem 0x80012000 32 0x100

CDM: time counter and error counter
root@cdm1:~# busybox devmem 0x80012000 32
0x00000100 <--- link mode
root@cdm1:~# busybox devmem 0x80012004 32
0x0000058C <--- seconds counter
root@cdm1:~# busybox devmem 0x80012004 32
0x0000058D
root@cdm1:~# busybox devmem 0x80012008 32
0x00000000 <--- error counter

GDM: no link
root@gdm0:~# busybox devmem 0x80011014 32
0x00000000
root@gdm0:~# busybox devmem 0x80011018 32
0x00000000
root@gdm0:~# busybox devmem 0x8001101c 32
0x00000000
root@gdm0:~# busybox devmem 0x80011024 32
0x00000FFF
root@gdm0:~# 

GDM: good link channel 10, counting mode
root@gdm0:~# busybox devmem 0x80012000 32 0x101
root@gdm0:~# busybox devmem 0x80012000 32 0x100
root@gdm0:~# busybox devmem 0x80012008 32
0x3A8B68C2
root@gdm0:~# busybox devmem 0x80012008 32
0x42E03BEF
root@gdm0:~# busybox devmem 0x8001200c 32
0xDA090972
root@gdm0:~# busybox devmem 0x8001200c 32
0xDE6F22E9
root@gdm0:~# busybox devmem 0x80012019 32
Bus error
root@gdm0:~# busybox devmem 0x80012010 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012014 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012018 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x8001201c 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012020 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012024 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012028 32
0x00000000
root@gdm0:~# busybox devmem 0x8001202c 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012030 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012034 32
0xFFFFFFFF
root@gdm0:~# 

mapping of link channels:

qsfp0 lane0 - 0x0100 - link 8
qsfp0 lane1 - 0x0200 - link 9
qsfp0 lane2 - 0x0400 - link 10
qsfp0 lane3 - n/c
qsfp1 lane0 - 0x0800 - link 11
qsfp1 lane1 - 0x0010 - link 4
qsfp1 lane2 - 0x0020 - link 5
qsfp1 lane3 - n/c
qsfp2 lane0 - 0x0040 - link 6
qsfp2 lane1 - 0x0080 - link 7
qsfp2 lane2 - 0x0001 - link 0
qsfp2 lane3 - n/c
qsfp3 lane0 - 0x0002 - link 1
qsfp3 lane1 - 0x0004 - link 2
qsfp3 lane2 - 0x0008 - link 3
qsfp3 lane3 - n/c

script to start the test with 2 CDMs:

ssh dsdaq@dsvslice
ssh root@gdm0 busybox devmem 0x80011008 32 1
ssh root@cdm0 busybox devmem 0x80011008 32 1
ssh root@cdm1 busybox devmem 0x80011008 32 1
ssh root@gdm0 busybox devmem 0x80011008 32 0
ssh root@cdm1 busybox devmem 0x80011008 32 0
ssh root@cdm0 busybox devmem 0x80011008 32 0
ssh root@gdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
ssh root@cdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
ssh root@cdm1 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2

LEMO trigger GDM to CDM to VX

on the GDM:

/home/dsdaq/si5394-i2c-file /home/dsdaq/GDM_v1.0_IN0_EXT1_and_IN1_fixed_Si5394-RevA-Registers.txt 0 0x6b
busybox devmem 0x80011008 32 0x1
busybox devmem 0x80011008 32 0x0
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x80001230 # QSFP TX fixed pattern
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 2 0xff # enable LEMO inputs
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 3 0x7654 # enable LED, one per LEMO input
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x0F0F # enable LEMO to trg_in and tsm_in
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x40001230 # enable trg_in and tsm_in output to QSFP

on the CDM:

/home/dsdaq/si5394-i2c-file /home/dsdaq/CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
/home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp
busybox devmem 0x80011008 32 1
busybox devmem 0x80011008 32 0
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x8040 # trg_in from sfp[0], tsm_in form sfp[1]
/home/dsdaq/online/ds-dm-software/test_cdm.exe 11 # trg_in counter
/home/dsdaq/online/ds-dm-software/test_cdm.exe 12 # tsm_in counter

GPS receiver VCL-2705

• Valiant VCL-2705 GPS receiver
• https://www.valiantcom.com/time-distribution/gps-receiver-irig-b.html
• USB connection is /dev/ttyACM0, 115200 bps, "GNSSAUX" prompt
• minicom -D /dev/ttyACM0 -b 115200
• user manual download instructoins - see sheet of paper with user name and password in the shipping box
• usb commands:

gnss-help
gnss-showver -> F/W: Ver 1.7 Feb 17 2020 16:20:43

gnss-showselftest -> no antenna connected
Overall   : FAIL
EPROM Test: PASS
Antenna   : NOT DETECTED
GNSS      : COMMUNICATION OK

gnss-showsettings
GNSS NMEA BAUDRATE        :115200
GNSS ANTENNA LENGTH       :30 meters
GNSS USER CONFIGURED DELAY:-65 nanoseconds
GNSS 1PPS PULSE WIDTH     :200 milliseconds
GNSS MODE                 :GPS
GNSS STATUS               :STATIONARY

GNSSAUX> gnss-showserial
SERIAL :2704H01V17MAX310

GNSSAUX> gnss-showmode
GNSS MODE : GPS

GNSSAUX> gnss-showstatus
GNSS STATUS: STATIONARY

GNSSAUX> gnss-showalarms
CURRENT ALARMS GNSS
ANTENNA   : NOT DETECTED

GNSSAUX> gnss-showerrors
CURRENT ALARMS GNSS
ANTENNA : **NOT DETECTED
ERROR STATISTICS GNSS
RMC GOOD DURATION  : Secs 0
RMC BAD DURATION   : Secs 0
LOCK GOOD SECS DURATION  : Secs 0
LOCK BAD SECS DURATION   : Secs 0
SATINFO GOOD ITERATIONS   :0
SATINFO REJECT ITERATIONS :0
SATINFO NOTALKER ITERATIONS   :0
CURRENT MONITOR STATE GNSS
STATE :Phase-1 HUNTING ANTENNA DETECT

GNSSAUX> gnss-showsatinfo
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-showsats
Total Sats: 0

GNSSAUX> gnss-showmyloc
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-show1ppsstate 
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-showjamstatus
Not Available !

GNSSAUX> gnss-showspoofstatus
Not Available !

--- antenna connected, can see the sky ---

GNSSAUX> gnss-showselftest
Overall   : PASS
EPROM Test: PASS
Antenna   : DETECTED
GNSS      : COMMUNICATION OK

GNSSAUX> gnss-showalarms
CURRENT ALARMS GNSS
ANTENNA   : DETECTED
GNSS LOCK : AVAILABLE

GNSSAUX> gnss-showerrors
CURRENT ALARMS GNSS
ANTENNA : DETECTED
GNSS LOCK : AVAILABLE
ERROR STATISTICS GNSS
RMC GOOD DURATION  : Mins 1,Secs 34
RMC BAD DURATION   : Secs 55
LOCK GOOD SECS DURATION  : Mins 1,Secs 34
LOCK BAD SECS DURATION   : Secs 55
SATINFO GOOD ITERATIONS   :3
SATINFO REJECT ITERATIONS :0
SATINFO NOTALKER ITERATIONS   :0
CURRENT MONITOR STATE GNSS
STATE :Phase-4 NORMAL OPERATION, Monitoring GNSSLOCK

GNSSAUX> gnss-showsatinfo
GNSS MODE :GPS
SATELLITE INFORMATION  TALKER:GPS
NMEA ID : 01-32
NO OF SATELLITES IN VIEW: 08
NO OF XXGSV MSGS        : 03
SatNo   PRN NO (SV ID)      ELEVATION (degs)    AZIMUTH (degs)      C/No (SNR)
1       0                   0                   0                   0                   
2       0                   0                   0                   0                   
3       0                   0                   0                   0                   
4       0                   0                   0                   0                   
5       0                   0                   0                   0                   
6       0                   0                   0                   0                   
7       0                   0                   0                   0                   
8       0                   0                   0                   0                   
<<<<<< End of Sat Info >>>>>>

GNSSAUX> gnss-showsats
Total Sats: 8

GNSSAUX> gnss-showmyloc
GNSS RECEIVER LOCATION:
Latitude : 4914.81911
Longitude: 12313.69595

GNSSAUX> gnss-show1ppsstate
GPS 1PPS STATE: LOCKED

GNSSAUX> gnss-showmyloc        

GNSS RECEIVER LOCATION:
Latitude : 4914.80688
Longitude: 12313.69531

enter into google maps search box as: 49 14.80688, -123 13.69531, observe the space, the moved dot and the minus.

GNSSAUX> gnss-resetgnss

Executing....Please Wait....
$$$$$END

• 1PPS BNC output: period 1 sec, pulse width 200 ms, 3.3V into 1MOhm, 1.38V into 50Ohm.
• IRIG-B BNC output: 5.6V into 1MOhm, 2.4V into 50Ohm.
• IRIG-B format selector: default is all up.

Rb clock PRS10

• https://www.thinksrs.com/products/prs10.html
• 10 MHz output is sine wave around 5V peak to peak
• 1PPS BNC output is 10 usec pulse, 5V into 1MHohm.
• RS232 connection: minicom -D /dev/ttyUSB0 -b 9600
• ser2net config: localhost,3001:raw:600:usb-5-2-1.0:9600 -XONXOFF -RTSCTS LOCAL
• ssh daq13, cd /home/olchansk/git/ser2net, ./ser2net -c ~/daq/ds/ser2net.conf -d
• ssh daq13, cd ~/daq/ds, python3 prs10.py
• RS232 commands:

ID?
PRS10_3.56_SN_105719
VB1
SN?
RS1 -- reset
ST? -- status
FC? -- 10MHz OCXO drive voltage DAC settings
DS? -- "detected signals"
GA? -- gain of frequency lock loop between ovenized oscillator and Rb cell, 0=use ovenized oscillator only
MO? -- magnetic offset of the Rb cell, range 2300..3600, if out of range, unit must be set to different operating mode, see prs10m.pdf
MR? -- magnetic read
TT? -- time-tag, time in ns between 1PPS out and 1PPS in
TS? -- time slope, ???
TO? -- time offset, ???
PS? -- pulse slope, ???
PL? -- 0=phase lock off, 1=phase lock on, lock to 1PPS input
PT? -- phase lock integrator time constant, PT8 is integrator time constant 18.2 hours, natural time constant 2.25 hours
PF? -- phase lock stability factor, PF2 is "1"
PI? -- phase lock integrator

Analog to digital 12 bit ADC, values 0.000 to 4.998

AD0? -- Spare (J204)
AD1? -- +24V(heater supply) divided by 10.
AD2? -- +24V(electronics supply) divided by 10
AD3? -- Drain voltage to lamp FET divided by 10
AD4? -- Gate voltage to lamp FET divided by 10
AD5? -- Crystal heater control voltage
AD6? -- Resonance cell heater control voltage
AD7? -- Discharge lamp heater control voltage
AD8? -- Amplified ac photosignal
AD9? -- Photocell’s I/V converter voltage divided by 4
AD10? -- Case temperature (10 mV/°C)
AD11? -- Crystal thermistors
AD12? -- Cell thermistors
AD13? -- Lamp thermistors
AD14? -- Frequency calibration pot / external calibration voltage
AD15? -- Analog ground

A/D via CPU E-port:

AD16? -- Varactor voltage for 22.48 MHz VCXO (inside RF synthesizer) / 4
AD17? -- Varactor voltage for 360 MHz VCO (output of RF synthesizer) / 4
AD18? -- Gain control voltage for amplifier which drives frequency multiplier / 4
AD19? -- RF synthesizer’s lock indicator voltage (nominally 4.8 V when locked )

ST?

ST1 : Power supplies and Discharge Lamp
ST1 bit, Condition which sets bit, Corrective Action
0 -- +24 for electronic < +22 Vdc
1 -- +24 for electronics > +30 Vdc
2 -- +24 for heaters <+22 Vdc
3 -- +24 for heaters > +30 Vdc
4 -- Lamp light level too low
5 -- Lamp light level too high
6 -- Gate voltage too low
7 -- Gate voltage too high

ST2: RF Synthesizer
ST2 bit, Condition which sets bit, Corrective Action
0 -- RF synthesizer PLL unlocked
1 -- RF crystal varactor too low
2 -- RF crystal varactor too high
3 -- RF VCO control too low
4 -- RF VCO control too high
5 -- RF AGC control too low
6 -- RF AGC control too high
7 -- Bad PLL parameter

ST3: Temperature Controllers
ST3 bit, Condition which sets bit
0 -- Lamp temp below set point
1 -- Lamp temp above set point
2 -- Crystal temp below set point
3 -- Crystal temp above set point
4 -- Cell temp below set point
5 -- Cell temp above set point
6 -- Case temperature too low
7 -- Case temperature too high

ST4: Frequency Lock-Loop Control
ST4 bit, Condition which sets bit
0 -- Frequency lock control is off
1 -- Frequency lock is disabled
2 -- 10 MHz EFC is too high
3 -- 10 MHz EFC is too low
4 -- Analog cal voltage > 4.9 V
5 -- Analog cal voltage < 0.1
6 -- not used
7 -- not used

ST5: Frequency Lock to External 1pps
ST5 bit, Condition which sets bit
0 -- PLL disabled
1 -- < 256 good 1pps inputs
2 -- PLL active
3 -- > 256 bad 1pps inputs
4 -- Excessive time interval
5 -- PLL restarted
6 -- f control saturated
7 -- No 1pps input

ST6: System Level Events
ST6 bit and Condition which sets bit
0 Lamp restart
1 Watchdog time-out and reset
2 Bad interrupt vector
3 EEPROM write failure
4 EEPROM data corruption
5 Bad command syntax
6 Bad command parameter
7 Unit has been reset

• ST? on warm start

received:  PRS_10
received:  255,255,255,243,34,255
received:  0,0,0,1,34,0
...
received:  0,0,0,1,34,0
received:  0,0,0,0,2,0
...
received:  0,0,0,0,2,0
received:  0,0,0,0,4,0

• ST? on loss of external 1PPS

...
received:  0,0,0,0,4,0
disconnect 1PPS input
received:  0,0,0,0,132,0
...
reconnect 1PPS input
received:  0,0,0,0,132,0
received:  0,0,0,0,4,0
...

• ST? on coldish start

daq13:ds$ python3 prs10.py
Connected
received [  ] old [ b'' ] counter:  0
received [ PRS_10 ] old [  ] counter:  0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter:  0
received [ 80,0,0,1,34,1 ] old [ 255,255,255,243,34,255 ] counter:  0
received [ 0,0,1,1,34,0 ] old [ 80,0,0,1,34,1 ] counter:  0
received [ 0,0,0,1,34,0 ] old [ 0,0,1,1,34,0 ] counter:  0
received [ 0,0,16,1,34,0 ] old [ 0,0,0,1,34,0 ] counter:  6
received [ 0,0,20,1,34,0 ] old [ 0,0,16,1,34,0 ] counter:  5
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter:  8
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter:  3
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter:  49
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter:  250
...

• ST? on cold start, note: no bump in the 10MHz clock as reported by DS-DM clock chip

Connected
received [  ] old [ b'' ] counter:  0
received [ PRS_10 ] old [  ] counter:  0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter:  0
received [ 80,0,21,1,34,1 ] old [ 255,255,255,243,34,255 ] counter:  0
received [ 64,0,21,1,34,0 ] old [ 80,0,21,1,34,1 ] counter:  0
received [ 0,0,21,1,34,0 ] old [ 64,0,21,1,34,0 ] counter:  0
received [ 0,0,20,1,34,0 ] old [ 0,0,21,1,34,0 ] counter:  158
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter:  23
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter:  47
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter:  40
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter:  249

det fac integration test

Connections:

• GPS receiver "IRIG-B SEL" both switches "up" - both "on", IRIG-B format B004
• GPS receiver USB-B -> long cable -> daq13 USB-A
• GPS receiver "1PPS out" -> long BNC cable -> BNC-T -> scope (5V, no 50ohm) and Rb clock BNC "1PPS in"
• GPS receiver "IRIG-B 50ohms" -> long BNC cable -> BNC-T -> scope (5V, no 50ohm) and DS-DM LEMO input 1 (TTL mode)
• Rb clock RS232 -> RS232 straight cable -> RS232-to-USB adapter -> daq13 USB-A
• Rb clock "1PPS out" BNC -> scope (5V, no 50ohm, trig threshold rising edge 2V)
• Rb clock "10MHz output 50 Ohm" BNC -> lemo -> lemo-T -> scope (sine wave, 5V, no 50ohm) and DS-DM clock input.

Programs to run:

• on daq13: cd /home/olchansk/git/ser2net, ./ser2net -c ~/daq/ds/ser2net.conf -d

localhost,3001:raw:600:usb-5-2-1.0:9600  -XONXOFF -RTSCTS LOCAL

• on daq13: cd /home/olchansk/daq/ds, python3 prs10.py ### connects to ser2net

daq13:ds$ python3 prs10.py
Connected
received [  ] old [ b'' ] counter:  0
received [ PRS_10 ] old [  ] counter:  0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter:  0
received [ 80,0,21,1,34,1 ] old [ 255,255,255,243,34,255 ] counter:  0
received [ 64,0,21,1,34,0 ] old [ 80,0,21,1,34,1 ] counter:  0
received [ 0,0,21,1,34,0 ] old [ 64,0,21,1,34,0 ] counter:  0
received [ 0,0,20,1,34,0 ] old [ 0,0,21,1,34,0 ] counter:  158
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter:  23
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter:  47
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter:  40
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter:  249
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  12096
received [ 0,0,0,0,20,0 ] old [ 0,0,0,0,132,0 ] counter:  2
received [ 0,0,0,0,148,0 ] old [ 0,0,0,0,20,0 ] counter:  0
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,148,0 ] counter:  575
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  11449
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  2755
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  34386
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  41035
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  113401
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  33375
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  54767
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  85059
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  33222
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  119234
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  121990
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  128184
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  56002
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  428237
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  8250
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  1388
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  30506
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  142704
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  179451
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  106182
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  68747
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  65424
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  157587
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  6932
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  1388
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  20255
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  4
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  225941
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  72183
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  26970
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  1
39287

• on ds-dm: ./test_cdm.exe --irigb ### note sbs mismatch is because I should wrap around at 16 bits

dataframe: S00000000S100001010S...S001010101S000001000S, sec: 00, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16724 should be 82260
dataframe: S10000000S100001010S...S101010101S000001000S, sec: 01, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16725 should be 82261
dataframe: S01000000S100001010S...S011010101S000001000S, sec: 02, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16726 should be 82262
dataframe: S11000000S100001010S...S111010101S000001000S, sec: 03, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16727 should be 82263
dataframe: S00100000S100001010S...S000110101S000001000S, sec: 04, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16728 should be 82264
dataframe: S10100000S100001010S...S100110101S000001000S, sec: 05, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16729 should be 82265

DS-IOGC GPS interface board

• Rev0 git repository: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0/-/tree/main?ref_type=heads
• Rev0 schematics: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0/-/blob/main/Altium/Project%20Outputs%20for%20DS-IOGC-Rev0/SCH-DS-IOGC-Rev0.pdf?ref_type=heads
• Rev0 schematics: SCH-DS-IOGC-Rev0
• Rev1 git repository: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev1/
• Rev1 schematics: SCH-DS-IOGC-Rev1

Changes Rev0 to Rev1

From: Peter Margetak <pmargetak@triumf.ca>
Subject: IOGC REV1 review
Date: Wed, 4 Sep 2024 07:31:19 +0000

Hi Konstantin,
Pls have a look at SCH for new rev. I'd like to send it to mfr next week so if you can comment by early next week. Meanwhile I work on layout and other stuff.

Changes:

New ICs - all powered +5V
U20 - inverters for  RUclk RX/TX
U21 - non inverting line driver for RU-1pps-out (so you don't have to route if via GDM to see it on scope)
U22 - non inverting buffer for ext 1pps input

All Lemo connectors have the same position but they are double lemos now => new panel needed
@Marek Walczak<mailto:mwalczak@triumf.ca> you can print it ahead once pcb is done + update IOGC docs and panel description

J2A/B - Test ports for RU-1pps in and out
J5A/B - inputs for external GPS data and External source of 1pps
J6A/B  - aux in/out for GDM

SW1 - no change - select RX/TX  USB/GDM
SW2 - select latch sensitivity for rising/falling edge
SW3 - select source of GPS data (opto or ext)  AND select source of 1pps input (latch or ext)

p.

PRS-10 Rb clock device

The Rb clock PRS-10 device provides these connections:

RS232 RX input - serial communication, non-standard RS232
RS232 TX output - serial communication, non-standard RS232
10 MHz clock output - coax 50 Ohm high resolution 10 MHz clock
1pps output - 1 Hz clock corresponding to the 10 MHz clock
1pps input - 1pps signal from GPS receiver

Mode of operation:

• 10 MHz clock is always running
• 1pps output is always running
• if 1pps input from GPS received is present, after 256 pulses PRS-10 will sync it's 1pps output with the 1pps input by adjusting the frequency of the 10 MHz clock
• when unlocked: 1pps output and 1pps input unrelated
• when locked to GPS: 1pps output and 1pps input always go up and down at the same time

Theory of operation:

• 10 MHz clock is produced by a high-quality crystal (stable on the scale of seconds)
• crystal oscillator is synchronized to a Rb cell (stable on the scale of hours and days)
• Rb cell resonant frequency is synchronized to the GPS 1pps signal (stable on the scale of months and years)

Rev1 connections

• LEMO connectors (front panel)

LEMO J2A output - Rb clock 1pps in monitor
LEMO J2B output - Rb clock 1pps out monitor
LEMO J5A input  - GPS IRIG-B from GPS receiver to FPGA (VCL-2705)
LEMO J5B input  - GPS 1pps from GPS receiver to PRS-10 (VCL-2705)
LEMO J6A output - AUX-OUT from FPGA (dual-LEMO PCB-side)
LEMO J6B input  - AUX-IN to FPGA (dual-LEMO away-from -PCB)

• SMB connectors (back)

SMB J3 output - GPS 1pps loopback to LNGS
SMB J4 input - LNGS GPS data input

• LEDs

D1 - same as LEMO J2A out (Rb clock 1pps in)
D2 - same as SMB J3 out (GPS 1pps from LNGS or from a GPS receiver)
D5 - controlled by FPGA-OUT-LED1
D6 - controlled by FPGA-OUT-LED2
D7 - PRS-10 24V power ok

• switches

SW1A and SW1B - route PRS-10 RS232 to USB or to FPGA
SW2A - route PRS-10 1pps input from SMB J4 (LNGS) or from LEMO J5B (GPS receiver 1pps)
SW2B - route FPGA-IN-GPSDATA input from SMB J4 (LNGS) or from LEMO J5A (GPS receiver IRIG-B data)
SW3 - LNGS 1pps from rising edge or from falling edge of SMB J4 (LNGS)

Rb clock cable

Rb clock ----- DS-IOGC side, pin numbers are as labeled on the cable connectors

1 - 1pps out - 8 - 1pps out                    --- correct
2 - nc
3 - nc
4 - TXD      - 3  - RU-DATA-OUT - USB-RX input --- correct
5 - 1pps in  - 2  - 1pps in                    --- correct
6 - +24V     - 10 - +24V                       --- should by pin 1 to use both +24V pins?
7 - RXD      - 15 - RU-DATA-IN - USB-TX output --- correct
8 - nc
9 - +24V     - 10 - +24V --- correct
10 - GND     - 9  - GND  --- correct

VX connections

VXA_TX0 - FPGA-OUT-LED2 - D6 LED ("10 MHz clock")
VXA_TX1 - FPGA-OUT-LED1 - D5 LED ("GPS DATA")
VXA_TX2 - FPGA-OUT-RU1PPS-EN - enable 1pps to Rb clock
VXA_TX3 - not used (62.5 Hz clock)

VXB_TX0 - FPGA-TX - PRS-10 RS-232 out
VXB_TX1 - FPGA-OUT-AUX - 5V TTL J5 LEMO out
VXB_TX2 - FPGA-OUT-OPTO1PPS-EN - PRS-10 power enable
VXB_TX3 - not used (62.5 MHz clock)

VXA_RX0 - FPGA-IN-OPTO1PPS - 1pps from GPS
VXA_RX1 - FPGA-IN-GPSDATA - J4 GPS data
VXA_RX2 - n/c
VXA_RX3 - n/c

VXB_RX0 - n/c
VXB_RX1 - FPGA-IN-AUX - J5 LEMO TTL input
VXB_RX2 - FPGA-RX - PRS-10 RS-232 in
VXB_RX3 - FPGA-IN_RU1PPS - PRS-10 1pps output

test sequence

• ./test_cdm_local.exe --writereg 7 0 ### clear reg 7
• ./test_cdm_local.exe --writereg 8 0 ### set vx_tx mux to vx_tx control from reg 7
• connect blue cable to GDM port 6 (next to the ethernet connector)
• ./test_cdm_local.exe --writereg 7 0x4000 ### power up
• test LEDs:
• ./test_cdm_local.exe --writereg 7 0x4100 ### right LED2 D6
• ./test_cdm_local.exe --writereg 7 0x4200 ### left LED1 D5
• test AUX-IN and AUX-OUT:
• ./test_cdm_local.exe --writereg 7 0x6000 ### J6A LEMO (PCB-side) measure +5VDC
• install LEMO jumper between LEMO J6A and J6B
• write 0x4000 then ./test_cdm_local.exe 6 ### reads 0x5C, bit 5 0x20 reads 0
• write 0x6000 then ./test_cdm_local.exe 6 ### reads 0x7C, bit 5 0x20 reads 1
• write 0x4000 to clear all bits
• test the 10 MHz clock:
• init the c.c: ./test_cdm_local.exe --load-cc
• connect 10 MHz clock output to DS-DM LEMO J6-LEFT (CLK_EXT1)
• 10 MHz output ok - IN0 is good: ./test_cdm_local.exe --cc

Clock chip state 1, status:  OOF_IN2 IN1 IN_SEL_1 HOLD_HIST_VALID

• without 10 MHz reports IN0 LOS (no signal) and OOF (wrong frequency)

Clock chip state 1, status:  LOS_IN0 OOF_IN0 OOF_IN2 IN1 IN_SEL_1 HOLD_HIST_VALID

• test GPS IRIG-B data:
• connect GPS IRIG-B signal to J6B (dual-LEMO away from PCB)
• IRIG-B via AUX-IN is ok: ./test_cdm_local.exe --irigb

dataframe: S00100110S111001000S...S011110000S010000000S, sec: 34, min: 17, hrs: 00, day of year: 223, year: 2024, date: 10 aug, sbs:  1054 should be  1054

• test IRIG-B via J5A GPS-EXT:
• connect GPS IRIG-B signal to J5A (dual-LEMO next to PCB)
• set SW2B: switch on the side of SMB connectors, slide direction of 24V power supply connector
• ./test_cdm_local.exe --irigb built for VXA_RX1 should work
• test GPS 1pps signal:
• connect GPS 1pps to J5B (dual-LEMO away from PCB)
• nothing happens, no output from U22
• remove R51
• set SW2A: switch on the side away from SMB connectors, slide direction of 24V power supply connector
• observe LEDs D1 and D2 blink at 1Hz
• SKIP THIS connect GPS 1pps to J4 using LEMO-SMB adapter
• SKIP THIS nothing happens, nLE is always 3.2V, both settings of SW2
• SKIP THIS !!!failed here!!!
• ./test_cdm_local.exe --writereg 8 9 ### set vx_tx mux to GPS control, hard enables PRS-10 power and PRS-10 1pps in
• observe 1pps counters in reg 68 count at 1 Hz: rb_1pps 0x51->0x52, gps_1pps 0xd5->oxd6

root@dsdm:~# ./test_cdm_local.exe 68
ds20k_reg[68] is 0x0051d520 (5362976)
ds20k_reg[68] is 0x0052d624 (5428772)
...

• observe PRS-10 can see the 1pps signal "130" changes to "2" after 243 seconds to "4"
• observe period of GPS and PRS-10 1pps is identical (plus/minus 1 clock)

root@dsdm:~# ./test_cdm_local.exe 13 14
ds20k_reg[13] is 0x077356d4 (124999380)
ds20k_reg[14] is 0x077356d4 (124999380)

• observe GDM/CDM clocks are exactly 125 MHz, we are running on DS-DM internal oscillator

root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735943 (125000003) should be ~125 MHz
0x1034 rx_clk:             0x07735943 (125000003) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735943 (125000003) should be ~125 MHz
0x103C tx_clk:             0x07735943 (125000003) should be ~125 MHz
0x1040 clk_50MHz:          0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk:         0x05f5e100 (100000000) should be 100 MHz exactly

• if tx_clk is missing, reset the mgt: ./test_cdm_local.exe --reset-mgt
• switch CC to external clock:

root@dsdm:~# ./test_cdm_local.exe --cc-in0
CC use clock input 0: 10 MHz LEMO external clock
root@dsdm:~# ./test_cdm_local.exe --cc
Polling CC status...
Clock chip state 1, status:  IN0 IN_SEL_REGCTRL IN_SEL_0 HOLD_HIST_VALID

• observe rx_clk and tx_clk are now slightly off: they run from PRS-10 10 MHz clock and are measured against the DS-DM internal oscillator.

root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x94b12519, ds20k version 0x20240814
DS-DM firmware build 0x94b12519, ds20k version 0x20240814
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x077358e1 (124999905) should be ~125 MHz
0x1034 rx_clk:             0x07735b49 (125000521) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735b49 (125000521) should be ~125 MHz
0x103C tx_clk:             0x07735b49 (125000521) should be ~125 MHz
0x1040 clk_50MHz:          0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk:         0x05f5e100 (100000000) should be 100 MHz exactly

• observe GPS and PRS-10 1pps period is now exactly 125*10^6 of 8ns clocks (equal to 1 second)

root@dsdm:~# ./test_cdm_local.exe 13 14
ds20k_reg[13] is 0x0773593f (124999999)
ds20k_reg[14] is 0x0773593f (124999999)

• look at them repeatedly, observe reg 13 "GPS 1pps period" has some wobble, reg 14 "Rb clock 1pps period" is steady. this is as expected: if GPS 1pps was steady, we do not need to Rb clock. PRS-10 1pps is derived from the PRS-10 10 MHz clock and is measured against the 125 MHz clock derived from the same 10 MHz clock in the GDM FPGA.

Rev0 Test status:

• GPS 1pps to SMB-in ok (LED flashes)
• GPS 1pps to FPGA ok
• GPS data to FPGA ok
• GPS 1pps to PRS-10 enabled from FPGA ok
• PRS-10 Rb clock 1pps out to FPGA ok
• PRS-10 Rb clock 1pps out to FPGA to LEMO AUX out ok
• can see GPS 1pps, IRIG-B, PRS-10 1pps out, 10 MHz on the scope, ok
• PRS-10 syncs on leading edge (0->1) of GPS 1pps signal, ok
• reg 13 and 14 1pps periods are identical, ok
• CC locks on PRS-10 10 MHz clock, DS-DM runs on PRS-10 clock, ok
• NOT TESTED - smb output
• NOT TESTED - optical converter fiber to SMB
• NOT TESTED - optical converter SMB to fiber
• NOT TESTED - SMB loopback
• NOT TESTED - fiber loopback

Rev1 Test status: (21 Nov 2024)

• power up ok
• LEDs ok
• LEMO AUX-IN, AUX-out ok
• GPS IRIG-B to FPGA via J6B AUX-IN ok
• GPS IRIG-B to FPGA via J5A GPS-EXT ok
• PRS-10 Rb clock 1pps out to FPGA ok
• PRS-10 Rb clock 10 MHz to C.C. ok
• PRS-10 RS-232 ok
• J5B 1pps input fail. input signal is 3.3V without termination, 1.5V when plugged into IOGC (on R51), no output from U22.

DS-20K DAQ

Overview

DS-DM, GDM and CDM are key parts of the DS-20K DAQ system:

• common clock distribution from external clock (atomic clock, GPS) to GDM to per-quadrant CDMs to VX digitizers
• common trigger distribution from GDM internal algorithm or external input to all VX digitizers
• run control: GDM, CDM, VX all start recording data at the same time (clock and timestamp reset)
• collection of trigger data from VX digitizers to per-quadrant CDMs to GDM

Deliverables

• hardware and firmware for GDM to CDM clock distribution
• hardware and firmware for CDM to VX clock distribution
• hardware and firmware for GDM external clock input (atomic clock or GPS)
• hardware and firmware for CDM and VX serial communications (VX LVDS I/O connector)
• firmware for run control (timestamp reset and sync): GDM to CDM to VX
• firmware for common trigger distribution: GDM to CDM to VX
• firmware for trigger data flow: VX to CDM to GDM
• firmware for busy control: VX to CDM to GDM back to CDM to VX
• firmware for flow control: FEP to GDM MIDAS frontend to GDM to CDM to VX
• GDM MIDAS frontend: clock selector and monitoring, trigger and run control, busy and flow control, GDM housekeeping
• CDM MIDAS frontend: clock monitoring, CDM housekeeping

specific performance:

• GDM external clock: 10 MHz GPS clock
• GDM to CDM fiber link:
  • clock XXX MHz
  • link data rate: XXX Gbit/sec
  • CDM recovered clock: XXX MHz
  • CDM recovered clock jitter: XXX ns
  • phase alignment between CDMs: XXX ns
  • phase alignment between CDMs persists across reboots, power cycles, firmware updates
  • phase alignment between CDMs should be easy to measure
  • phase alignment between CDMs should be easy to recalibrate if hardware parts are replaced (DS-DM boards, fiber transceivers, fiber cables, etc)
  • data packet bandwidth: XXX Mbytes/sec
  • data packet latency: XXX clocks
  • data packet skew between CDMs: XXX clocks
• CDM to VX clock:
  • clock: XXX MHz
  • jitter, all CDM clock outputs: XXX MHz
  • phase alignment between all CDM clock outputs: XXX ns
• CDM to VX trigger:
  • TBD (use the VX "sync" input or VX LVDS I/O line or VX serial link packet)
• CDM to VX serial link:
  • clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
  • bit rate: XXX bits/sec
  • latency: XXX link clocks
  • maximum skew between VXes: XXX ns
• VX to CDM serial link:
  • clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
  • bit rate: XXX bits/sec
  • latency: XXX link clocks
  • maximum skew between VXes: XXX ns
• timestamp reset:
  • maximum skew between VXes: XXX ns
• busy round trip time: XXX ns (VX to CDM to GDM back to CDM to VX)
• flow control latency: XXX ns (FEP software to GDM MIDAS frontend to GDM to CDM to VX)

Technical risk items

this refers to unexpected behaviour and performance of system components, causes big difficulty in implementing the system, prevents delivery of deliverables, and prevents or negatively affects operation of the DS-20K DAQ or of the whole experiment.

(14-sep-2022, list is not sorted by any criterial: severity, probability, ease of investigation)

(stability of course is long term stability, across hours, days, weeks, months, years)

• stability of Enclustra FPGA modules (crashes/year, failures to boot/year, flash corruption/year)
• stability of GDM external clock PLL (lock loss/year)
• stability of CDM recovered clock (lock loss/year, unexpected phase drifts, etc)
• unexpected failures or bit error rates in GDM-CDM fiber links
• stability of CDM VX clock outputs (stability of clock cleaner chip)
• stability of VX internal clock distribution (VX PLL lock loss events)
• stability of VX CAEN base firmware (different versions of CAEN base firmware have different clock distribution behaviour)
• strange things in CAEN base firmware (unexpected clocking of LVDS I/O, unexpected phase shifts between clocks, etc)
• DS-DM and VX hardware problems (incompatible LVSD I/O, incompatible clock signals, etc)

Milestones

(14-sep-2022: at this stage of the project, priority must be given to identifying and retiring (so called) technical risk factors. it is not good to build the complete system only to discover that (for example) some Enclustra FPGA modules require 5 attempts to boot and erase their flash memory contents once a month. Both example are real-life actual problems that caused big difficulties in GRIFFIN/TIGRESS and ALPHA-g experiments).

Development and testing milestones in time reversed order:

• full DAQ data challenge: all VXes, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
• one quadrant data challenge: 1 VME crate of VX, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
• vertical slice data challenge: 1 VME crate, 2 VX, 2 CDM (1 VX per CDM), GDM, etc operate as designed
• GDM-CDM link finalized (data rate frozen, data packet format frozen, data content permitted to change)
• CDM-VX serial link finalized (data rate frozen, data packet format frozen, data content permitted to change)
• run control (timestamp reset) and trigger distribution design agreed upon, frozen (list of possible triggers permitted to change)
• VX to CDM to GDM data flow design agreed upon, frozen (data contents permitted to change)
• major technical risk items retired (all hardware and firmware is working as expecred without mysteries and surprises, all problems are identified, investigated, resolved, solutions tested)
• stable operation of CDM-VX serial links in vertical slice system
• stable operation of GDM to CDM clock in vertical slice system
• stable operation of CDM to VX clock in vertical slice system
• vertical slice system assembled (1 VME crate, 2 VX, 2 CDM, 1 GDM, network, FEP, TSP, MIDAS)

ZZZ

ZZZ