DS-DM: Difference between revisions

From DaqWiki
Jump to navigation Jump to search
 
(20 intermediate revisions by the same user not shown)
Line 189: Line 189:
</pre>
</pre>


= Clock distribution =
= Clock distribution Rev0 =


Simplified:
Simplified:
Line 256: Line 256:
</pre>
</pre>


= I2C bus =
= Clock distribution Rev1 =
 
Simplified:
 
* 125 MHz osc -> CLK_XO_125 -> MGTREFCLK0_A -> not used
* 125 MHz osc -> CLK3_XO_125 -> MGTREFCLK1_B -> SFP RX ref clock
* 125 MHz osc -> C.C. in1
 
* C.C. in0 <- CLK_EXT1 (10 MHz GPS clock)
* C.C. in1 <- 125 MHz osc
* C.C. in2 <- CLK_CC_IN <- FPGA AK9,AK8 <- SFP RX recovered clock, 125 MHz
* C.C. in3 <- CLK_FB
 
* C.C. 125 MHz -> CLK_CC_OUT0 -> MGTREFCLK0_B -> QSFP RX and TX ref clock
* C.C. 125 MHz -> CLK_CC_OUT1 -> MGTREFCLK1_D -> SFP TX clock
* C.C. 125 MHz -> CLK_CC_OUT2 -> FPGA AG8,AH8 (GC) -> not used
* C.C. 62.5 MHz -> VX clock fanout
 
Complete:
 
<pre>
125 MHz oscillator - U5 fan out -
  q0 -> CLK_XO_125 -> ENC C72,C74 -> FPGA R8,R7 MGTREFCLK0_A -> XDC CLK_XO_125_P -> VHDL not used
  q1 -> U6 C.C. in1
  q2 -> CLK3_XO_125 -> ENC C7,C9 -> FPGA J8,J7 MGTREFCLK1_B -> XDC GDM missing, CDM CLK3_XO_125_P -> VHDL SFP RX reference clock (mgt_rx_ref_clk)
  q3 -> not used, was CLK2_XO_125
 
U6 C.C (clock cleaner) -
 
in0 <- CLK_EXT1 (GPS 10 MHz clock)
in1 <- 125 MHz oscillator via U5
in2 <- CLK_CC_IN <- ENC C142,C144 <- FPGA AK9,AK8 <- XDC GDM missing, CDM CLK_CC_IN1_P <- VHDL rx_clk
in3 <- CLK_FB
 
out0 -> CLK_CCA -> U12 (125 MHz)
out1 -> CLK_CCB -> VX1..6 (62.5 MHz)
out2 -> CLK_CCC -> VX7..12 (62.5 MHz)
out3 -> CLK_FB into in3
 
CLK_CCA -> U12 (125 MHz fan out) ->
 
Q0 -> not used
Q1 -> CLK_CC_OUT0 -> ENC C3,C5 -> FPGA L8,L7 MGTREFCLK0_B -> XDC GDM clk_mgtrefclk0_x0y1_p, CDM CLK_CC_OUT0_P -> VHDL  GDM mgt_b_ref_clk QSFP RX and TX reference clocks, CDM alternate rx_clk via clock mux
Q2 -> CLK_CC_OUT1 -> ENC B3,B5 -> FPGA B10,B9 MGTREFCLK1_D -> XDC CLK_CC_OUT1_P -> VHDL GDM not used, CDM mgt_tx_ref_clk SFP TX clock
Q3 -> CLK_CC_OUT2 -> ENC C151,C153 -> FPGA AG8,AH8 (GC) -> XDC GDM, CDM missing
Q4 -> not used
Q5 -> SMA J9/J10
 
CLK_TP0 <-> ENC C69,C71 <-> FPGA N8,N7 MGTREFCLK1_A (not used) -> XDC CLK_TP0_P -> VHDL not used (DS-DM SMA J11, J12)
 
Notes:
* Enclustra ENC-B is J801, ENC-C is J900
* "(GC)" is clock-capable FPGA pin
* CLK_XO_125 (125 MHz osc) is not used in FPGA
* 62.5 MHz VX clock does not go into the FPGA
* CLK_EXT0 going to in1 of U5 cannot be used. only permitted frequency is 125 MHz (it drives the MGT reference clocks) and is this frequency is too high for LEMO cables and connectors
</pre>
 
= I2C bus =
 
* I2C_SCL is J-ENC-A pin 111 I2C_SCL is FPGA I2C_SCL_PL AB13 (IO_L1N_TOL1D_64) and I2C_SCL_PS F18 (PS_MIO10)
* I2C_SDA is J-ENC-A pin 113 I2C_SDA is FPGA I2C_SDA_PL AH13 (IO_L7N_T1L1Q_AD13N_64) and I2C_SDA_PS G18 (PS_MIO11)
 
* XU8 secure EEPROM ATSHA204A at 0x64, this is 0110'010X -> linux _011'0010 is 0x32. (but responds to scan and read at 0x33)
* U4 ethernet mac chip, EEPROM at 1010 A2 A1 A0 X and MAC/serial_no at 1011 A2 A1 A0 X. A0=VCC, A1=VCC, A2=GND -> linux _101'0011 and _101'1011 is 0x53 and 0x5B.
* U6 clock chip, address 1101 0 A1 A0 X. A1=VCC, A0=N/C (internal pull-up) -> linux _110'1011 is 0x6b
* U23 voltmeter at 10011 ADR1 ADR0. ADR0=GND, ADR1=VCC -> linux _100'1110 is 0x4e
* SFP, address 1010000X -> linux _101'0000 is 0x50. additional SFP data at 0x51
* QSFP0, QSFP1, QSFP2, QSFP3 (QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL)
 
= I2C clock builder connection =
 
* use Silicon Labs USB "Clock builder pro field programmer", www.silabs.com/CBProgrammer
* connect rainbow jumper cable pins:
** black - 1-GND to GND on DS-DM
** white - 3-SCLK to SCL on the DS-DM
** grey - 7-SDA_SDIO to SDA on the DS-DM
* power up the DS-DM
* plug USB programmer into Windows laptop
* on Windows, run "ClockBuilder Pro"
* it should report "Field programmer detected", press "EVB GUI"
* in EVB GUI, press "Config", set I2C address 0x6B
* press "Scan", it should find Si5394A-A-GM
* select the "Status" tab, should see real-time status of clock chip
 
= GDM MGT configuration =
 
* TX configuration:
* GDM MGT transceivers are configured as "multilane" TX and RX.
* there is 12 TXes ("lanes")
* MGT reference 125 MHz clock goes into [2:0]gtreclk00_in and [11:0]gtrefclk0_in
* one MGT is designated as "master"
* PLL of master MGT converts reference clock into common TX clock and common tx_user_clk2 which becomes tx_data_clk
* common tx_user_clk2 aka tx_data_clk goes into all TXes and clocks tx_data.
* tx_user_clk2 aka tx_data_clk is 125 MHz but not same phase as MGT reference clock.
 
* RX interim configuration:
* there is 12 RXes ("lanes")
* each RX produces it's own recovered RX clock
* "multilane" configuration assumes all RX recovered clocks run at the same frequency (TX on the other end are driven by common TX clock, see above), but have different phase
* one RX recovered clock is designated as "master" (rx_user_clk2 aka rx_data_clk) and a phase-matching fifo/buffer is used to bring rx_data from all 12 RXes to this common rx_data_clk
* this works because each CDM SFP TX runs on the SFP RX recovered clock which is frequency-locked with the GDM QSFP TX clock.


* I2C_SCL is J-ENC-A pin 111 I2C_SCL is FPGA I2C_SCL_PL AB13 (IO_L1N_TOL1D_64) and I2C_SCL_PS F18 (PS_MIO10)
* RX final configuration:
* I2C_SDA is J-ENC-A pin 113 I2C_SDA is FPGA I2C_SDA_PL AH13 (IO_L7N_T1L1Q_AD13N_64) and I2C_SDA_PS G18 (PS_MIO11)
* MGTs permit using the common TX clock (tx_user_clk2 aka tx_data_clk) as the common rx_data_clk (they are frequency locked through the CDM).
* this permits use of tx_data_clk as the main clock domain in the GDM and removes the need to bring rx_data into the tx_data_clk domain (actually this is done in the MGT RX phase matching fifo/buffer).


* XU8 secure EEPROM ATSHA204A at 0x64, this is 0110'010X -> linux _011'0010 is 0x32. (but responds to scan and read at 0x33)
= Clock path =
* U4 ethernet mac chip, EEPROM at 1010 A2 A1 A0 X and MAC/serial_no at 1011 A2 A1 A0 X. A0=VCC, A1=VCC, A2=GND -> linux _101'0011 and _101'1011 is 0x53 and 0x5B.
* U6 clock chip, address 1101 0 A1 A0 X. A1=VCC, A0=N/C (internal pull-up) -> linux _110'1011 is 0x6b
* U23 voltmeter at 10011 ADR1 ADR0. ADR0=GND, ADR1=VCC -> linux _100'1110 is 0x4e
* SFP, address 1010000X -> linux _101'0000 is 0x50. additional SFP data at 0x51
* QSFP0, QSFP1, QSFP2, QSFP3 (QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL)


= I2C clock builder connection =
NOTE: MUST REVIEW!!!


* use Silicon Labs USB "Clock builder pro field programmer", www.silabs.com/CBProgrammer
<pre>
* connect rainbow jumper cable pins:
10 MHz ext clock or GDM 125 MHz oscillator
** black - 1-GND to GND on DS-DM
-> GDM QSFP MGT reference clock 125 MHz
** white - 3-SCLK to SCL on the DS-DM
-> MGT PLL -> tx_data_clk 125 MHz (GDM main clock domain) and TX bit clock 2.5 GHz
** grey - 7-SDA_SDIO to SDA on the DS-DM
-> GDM QSFP optic transmitter
* power up the DS-DM
-> CDM SFP optic receiver
* plug USB programmer into Windows laptop
-> CDM SFP MGT, RX reference clock is CDM 125 MHz oscillator
* on Windows, run "ClockBuilder Pro"
-> MGT RX recovered clock 125 MHz (CDM main clock domain)
* it should report "Field programmer detected", press "EVB GUI"
-> CC_CLK_IN -> CDM C.C. -> CC_CLK_OUT1 -> CDM SFP TX reference clock 125 MHz
* in EVB GUI, press "Config", set I2C address 0x6B
* press "Scan", it should find Si5394A-A-GM
* select the "Status" tab, should see real-time status of clock chip
 
= GDM MGT configuration =
 
* TX configuration:
* GDM MGT transceivers are configured as "multilane" TX and RX.
* there is 12 TXes ("lanes")
* MGT reference 125 MHz clock goes into [2:0]gtreclk00_in and [11:0]gtrefclk0_in
* one MGT is designated as "master"
* PLL of master MGT converts reference clock into common TX clock and common tx_user_clk2 which becomes tx_data_clk
* common tx_user_clk2 aka tx_data_clk goes into all TXes and clocks tx_data.
* tx_user_clk2 aka tx_data_clk is 125 MHz but not same phase as MGT reference clock.
 
* RX interim configuration:
* there is 12 RXes ("lanes")
* each RX produces it's own recovered RX clock
* "multilane" configuration assumes all RX recovered clocks run at the same frequency (TX on the other end are driven by common TX clock, see above), but have different phase
* one RX recovered clock is designated as "master" (rx_user_clk2 aka rx_data_clk) and a phase-matching fifo/buffer is used to bring rx_data from all 12 RXes to this common rx_data_clk
* this works because each CDM SFP TX runs on the SFP RX recovered clock which is frequency-locked with the GDM QSFP TX clock.
 
* RX final configuration:
* MGTs permit using the common TX clock (tx_user_clk2 aka tx_data_clk) as the common rx_data_clk (they are frequency locked through the CDM).
* this permits use of tx_data_clk as the main clock domain in the GDM and removes the need to bring rx_data into the tx_data_clk domain (actually this is done in the MGT RX phase matching fifo/buffer).
 
= Clock path =
 
<pre>
10 MHz ext clock or GDM 125 MHz oscillator
-> GDM QSFP MGT reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz (GDM main clock domain) and TX bit clock 2.5 GHz
-> GDM QSFP optic transmitter
-> CDM SFP optic receiver
-> CDM SFP MGT, RX reference clock is CDM 125 MHz oscillator
-> MGT RX recovered clock 125 MHz (CDM main clock domain)
-> CC_CLK_IN -> CDM C.C. -> CC_CLK_OUT1 -> CDM SFP TX reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz and TX bit clock 2.5 GHz
-> MGT PLL -> tx_data_clk 125 MHz and TX bit clock 2.5 GHz
-> (tx_data phase matching fifo from CDM main clock domain to tx_data_clk)
-> (tx_data phase matching fifo from CDM main clock domain to tx_data_clk)
Line 1,047: Line 1,107:
<pre>
<pre>
CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET=0x9A
CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET=0x9A
</pre>
= RTC chip =
* DS3231 RTC chip
* FPGA connections:
<pre>
I2C SCL <- J-ENC A85 <- FPGA E17 <- XDC TP_S <- VHDL TP_S is output
I2C SDA <-> J-ENC A87 <-> FPGA D17 <-> XDC TP_S <- VHDL TP_S is output
1pps -> J-ENC B129 -> FPGA AE3 -> XDC "free pin"
1pps -> J-ENC C160 -> FPGA AH12 -> XDC "slow_io"
</pre>
</pre>


Line 1,538: Line 1,609:


on write:
on write:
* 0x00 - noop - as of version 0x20240118, write a zero after writing a command
* 0 - noop - as of version 0x20240118, write a zero after writing a command
* 0x01 - CMD_reset - reset logic to good state
* 1 - cmd_reset - reset logic to good state
* 0x02 - CMD_arm_ts - arm timestamp reset
* 2 - cmd_arm_ts - arm timestamp reset
* 0x03 - CMD_trg - issue a trigger
* 3 - cmd_trg - issue a trigger
* 0x04 - CMD_tsm - issue a tsm
* 4 - cmd_tsm - issue a tsm
* 0x05 - CMD_vx_rx_reset - reset the VX receive path
* 5 - cmd_vx_rx_reset - reset the VX receive path
* 0x06 - CMD_vx_tx_reset - reset the VX transmit path
* 6 - cmd_vx_tx_reset - reset the VX transmit path
* 0x07 - CMD_hitmap_trg - generate a hitmap trigger and data packet
* 7 - cmd_hitmap_trg - generate a hitmap trigger and data packet
* 0x08 - CMD_trg_pulser_reset - reset the trigger pulser
* 8 - cmd_trg_pulser_reset - reset the trigger pulser
* 0x09 - CMD_tsm_pulser_reset - reset the tsm pulser
* 9 - cmd_tsm_pulser_reset - reset the tsm pulser
* 10 - cmd_bor_start - start begin-of-run trigger sequence
* 11 - cmd_bor_clear - after run has started, clear begin-of-run status bits
* 12 - cmd_latch - latch counters & etc into AXI registers for coherent readout


=== Register 1 0x80013004 scratch ===
=== Register 1 0x80013004 scratch ===
Line 1,757: Line 1,831:
6 - 125 MHz output
6 - 125 MHz output
7 - trg, tsm, serial
7 - trg, tsm, serial
8 - trg, veto, serial
8 - trg, tsm, lvds serial rx to serial tx loopback
9 - GPS box control (ds20k rev 0x20240814)
9 - GPS box control (ds20k rev 0x20240814)
10
10
Line 1,767: Line 1,841:
</pre>
</pre>


=== Register 9 0x80013024 trigger control ===
=== Register 9 trg and tsm source ===


from version 0x20240724
from version 0x20240724
Line 1,779: Line 1,853:
   wire [15:0]      trg_src_bits =
   wire [15:0]      trg_src_bits =
                     {
                     {
                     1'b0,
                     1'b0,           // 15
                     1'b0,
                     1'b0,           // 14
                     1'b0,
                     vx_tx_tsm_done, // 13
                     1'b0,
                     vx_tx_trg_done, // 12


                     1'b0, // gdm_hitmap_trigger,
                     1'b0, // gdm_hitmap_trigger, // 11
                     cdm_hitmap_trigger,
                     cdm_hitmap_trigger, // 10
                     sfp_rx_tsm,
                     sfp_rx_tsm,     // 9
                     sfp_rx_trg,
                     sfp_rx_trg,     // 8


                     sfp_rx_data[1],
                     sfp_rx_data[1], // 7
                     sfp_rx_data[0],
                     sfp_rx_data[0], // 6
                     tsm_pulser,
                     tsm_pulser,     // 5
                     trg_pulser,
                     trg_pulser,     // 4


                     lemo_in_sync[4],
                     lemo_in_sync[4], // 3
                     lemo_in_sync[3],
                     lemo_in_sync[3], // 2
                     lemo_in_sync[2],
                     lemo_in_sync[2], // 1
                     lemo_in_sync[1]
                     lemo_in_sync[1] // 0
                     };
                     };
    
    
Line 1,919: Line 1,993:
</pre>
</pre>


=== Register 16 0x80013040 SFP TX control ===
=== Register 16 SFP TX control ===


<pre>
<pre>
   wire [15:0] sfp_tx_data_reg = register_data_in[16][15:0];
   wire [15:0] sfp_tx_data_reg = register_data_in[16][15:0];
   wire [1:0]  sfp_tx_ctrl_reg = register_data_in[16][17:16];
   wire [1:0]  sfp_tx_ctrl_reg = register_data_in[16][17:16];
   wire sfp_rx_sel_lpb        = register_data_in[16][28]; // TX->RX serial loopback
   // 18
   wire sfp_tx_sel_lpb        = register_data_in[16][29]; // RX->TX serial loopback
  // 19
  // 20:23
  wire qsfp_tx_enable_trg    = register_data_in[23][24]; // enable QSFP TX trg_in_pulse k-code
  wire qsfp_tx_enable_tsm    = register_data_in[23][25]; // enable QSFP TX tsm_in_pulse k-code
  // 26
  // 27
  //wire sfp_rx_sel_loopback    = register_data_in[16][28]; // TX->RX serial loopback
   wire sfp_tx_sel_loopback    = register_data_in[16][29]; // RX->TX serial loopback
   wire sfp_tx_sel_trg        = register_data_in[16][30]; // 16 individual bits
   wire sfp_tx_sel_trg        = register_data_in[16][30]; // 16 individual bits
   wire sfp_tx_sel_reg        = register_data_in[16][31]; // from register
   wire sfp_tx_sel_reg        = register_data_in[16][31]; // from register
</pre>
</pre>


=== Register 17-22 0x80013044 QSFP RX data ===
=== Register 17-22 QSFP RX data ===


QSFP RX data links 0..11
QSFP RX data links 0..11


=== Register 23 0x8001305C QSFP TX control ===
=== Register 23 QSFP TX control ===


<pre>
<pre>
Line 2,278: Line 2,359:
       rb_1pps_in,  // 1
       rb_1pps_in,  // 1
       gps_1pps_in  // 0
       gps_1pps_in  // 0
      };
</pre>
=== Register 71 packet error bits ===
<pre>
assign register_data_out[71] =
    {
      8'b00000000,  // 23+8 bits
      8'b00000000,  // 16+8 bits
      8'b00000000,  //  8+8 bits
      1'b0,
      1'b0,
      1'b0,
      1'b0,
      1'b0,
      cdm_hitmap_pkt16_error_latch,
      vx_tx_tsm_pkt8_error_latch,  // 1
      vx_tx_trg_pkt8_error_latch  // 0
       };
       };
</pre>
</pre>

Latest revision as of 17:20, 13 November 2024

DS-DM

DarkSide-20k Global and Crate Data Manager board (GDM and CDM).

Global Data Manager (GDM):

  • clock distribution to CDM boards (including GPS/atomic clock source)
  • collection of trigger data from CDM boards, processing and distribution of trigger decision to CDM boards
  • run control
  • integration with GPS 10MHz and 1pps clocks and GPS/IRIG date and time information

Crate Data Manager (CDM):

  • clock distribution from GDM to CAEN VX digitizers
  • receive trigger data from CAEN VX digitizers
  • send trigger data to GDM
  • run control and dead time control

Links

Onboard hardware

  • jtag chain: arm_dap_0 0x5BA00477, xczu4_1 0x04721093
  • Eclustra Mercury+ XU8 module: ME-XU8-4CG-1E-D11E-R2.1
    • Xilinx® Zynq Ultrascale+™ MPSoC XCZU4CG-1FBVB900E
    • DDR4 ECC SDRAM (PS) 2 GB
    • DDR4 SDRAM (PL) 1GB
  • ethernet mac chip: AT24MAC402-SSHM-T ("602" chip is wrong)
  • USB UART for Enclustra serial console, micro-USB, 115200n8
  • clock chip: SI5394A-A-GM and oscillator CS-044-054.0M (54 MHz)
  • U23 3.3V current meter and thermometer, LTC2990IMS#TRPBF
  • LEDs:
    • LED_FP A/B/C/D 0/1/2/3
    • led1 - 3V3_SW_ON, SOM_POWER_GOOD - Enclustra FPGA module 3.3V power is good
    • led2 - LTM4624 PGOOD
    • led3 - FPGA_DONE - FPGA has booted
    • led4 - TP-S-1, PCLK_P
    • led5 - TP-S-2, PCLK_N
  • LEMO connectors (top to bottom)
    • J4 - input (NIM/TTL) (EXT_IN_LV(1), EXT_IN_LV(2))
    • J5 - input (NIM/TTL) (EXT_IN_LV(3), EXT_IN_LV(4))
    • J6 - external clock (GPS 10MHz and PPS)
    • J7 - output (NIM/TTL) (EXT_OUT(1), EXT_OUT(2))
  • SMA connectors
    • J9, J10 - CLK_CCA from U6 C.C.
    • J11, J12 - CLK_TP0
  • RJ45 ethernet connector (100mbit: green light on, yellow light flashes when there is traffic)
  • SFP connector (SFP is FTLF8526P3BNL, 6 Gbit/sec, 850 nm, 300m 50/125um OM3 MMF)
  • 4 QSFP connectors (GDM)
  • 6 VX connectors (CDM)

Buttons, jumpers and switches

Buttons:

  • PB1 - HRST - reboot FPGA (power-on reset)
  • PB2 - SRST - (SRSTn) - reboot ARM CPU

Switches:

  • SW1 - boot mode BM0, BM1 [-->]
  • SW2 - LEMO output NIM<->TTL
  • SW3 - LEMO input 1 and 2 NIM/TTL
  • SW4 - LEMO input 2 and 4 NIM/TTL
  • SW5 - LEMO clock input NIM/TTL
  • SW6 - serial console select. [PS<--PL] PS is ARM CPU, PL is FPGA.

Front panel

| top
|
| LED-FP1 | LED_FP(0,1,2,3)
|
| SFP J???
|
| LEMO J4-LEFT, J4-RIGHT | EXT_IN_LV(1), EXT_IN_LV(2)
| LEMO J5-LEFT, J5-RIGHT | EXT_IN_LV(3), EXT_IN_LV(4)
| LEMO J6-LEFT, J6-RIGHT | CLK_EXT1, CLK_EXT0 (125 MHz only) 
| LEMO J7-LEFT, J7-RIGHT | EXT_OUT(1), EXT_OUT(2)
|
| J-VX-1
| J-VX-2 or QSFP-1
| J-VX-3 or QSFP-2
| J-VX-4 or QSFP-3
| J-VX-5 or QSFP-4
| J-VX-6
|
| RJ45 J3 ethernet
|
| bottom

VX adapter board

LVDS I/O connector 

34 pin connector: 0|:::: :::: :::: :::: :|15,16 n/c

split-cable connection

0 .. 7 -> N/C

8 -> VX_RX(3) - not used
9 -> VX_RX(2) - busy VX to CDM
10 -> VX_RX(1) - DS20K 125 MHz serial data VX to CDM
11 -> VX_RX(0) - DS20K 62.5MHz clock VX to CDM

12 <- VX_TX(0) - TRG CDM to VX
13 <- VX_TX(1) - TSM CDM to VX, to be VETO CDM to VX
14 <- VX_TX(2) (set by jumper routed here or to VX CLKIN SYNC) - 125 MHz serial data CDM to VX
15 <- n/c (CLK routed to VX CLKIN CLK) - 62.5 MHz clock CDM to VX

16 - n/c

one-to-one connection

0 -> VX2_RX(3)
1 -> VX2_RX(2)
2 -> VX2_RX(0)
3 -> VX2_RX(1)

4 <- CLK
5 <- VX2_TX(0)
6 <- VX2_TX(1)
7 <- VX2_TX(2)

8 -> VX1_RX(3)
9 -> VX1_RX(2)
10 -> VX1_RX(1)
11 -> VX1_RX(0)

12 <- VX1_TX(0)
13 <- VX1_TX(1)
14 <- VX1_TX(2) (set by jumper routed here or to VX CLKIN SYNC)
15 <- n/c (CLK routed to VX CLKIN CLK)

16 - n/c

Board schematics

  • File:SCH-DS-xDM-Rev0.PDF
  • note: FPGA pin annotations ("IO", "SCLK", "PCLK", etc) on the schematics are bogus, instead, trace them to the FPGA pins.
  • note: ENC A is J800, ENC B is J801, ENC C is J900 (schematic name to enclustra name)
  • note: Enclustra special pins: "GC" is "clock capable", "HDGC" is "clock capable", "MGTREFCLK" is MGT reference clocks.
  • board modifications:
    • ethernet mac chip
    • NIM output (no U15, etc)
    • RJ45 wrong pinout (board mod or special ethernet cable)
    • 125 MHz clock mods (TBW)
    • disconnect QSFP0_SEL from SFP_RS0 and QSFP1_SEL from SFP_RS1, these signals are not used by modern SFPs
    • provide SFP i2c modsel to allow SFP and QSFP at the same time on address 0x50

FPGA MGT blocks

* SFP - ENC B45,B47 and B48,B50 - FPGA D5,D6 MGTHTX0_D and D1,D2 MGTHRX0_D
* QSFP0 TX0 - ENC C13,C17 - FPGA H5,H6 MGTHTX0_C
* QSFP0 TX1 - ENC C21,C25 - FPGA G7,G8 MGTHTX1_C
* QSFP0 TX2 - ENC C29,C23 - FPGA F5,F6 MGTHTX2_C
* QSFP1 TX0 - ENC C37,C41 - FPGA E7,E8 MGTHTX3_C
* QSFP1 TX1 - ENC C45,C47 - FPGA P5,P6 MGTHTX0_B
* QSFP1 TX2 - ENC C51,C53 - FPGA M5,M6 MGTHTX1_B
* QSFP2 TX0 - ENC C57,C59 - FPGA L3,L4 MGTHRX2_B
* QSFP2 TX1 - ENC C63,C65 - FPGA K5,K6 MGTHTX3_B
* QSFP2 TX2 - ENC C75,C77 - FPGA W3,W4 MGTHTX0_A
* QSFP3 TX0 - ENC C79,C81 - FPGA V5,V6 MGTHTX1_A
* QSFP3 TX1 - ENC C85,C87 - FPGA T5,T6 MGTHRX2_A
* QSFP3 TX2 - ENC C89,C91 - FPGA R3,R4 MGTHTX3_A

Clock distribution Rev0

Simplified:

  • 125 MHz osc -> CLK_XO_125 -> MGTREFCLK0_A -> not used
  • 125 MHz osc -> CLK3_XO_125 -> MGTREFCLK1_B -> SFP RX ref clock, QSFP RX and TX ref clock (this is not final design!)
  • (disconnected) 125 MHz osc -> CLK2_XO_125 -> FPGA AG8,AH8 (GC)
  • 125 MHz osc -> C.C. in1
  • C.C. in0 <- CLK_EXT1 (10 MHz GPS clock)
  • C.C. in1 <- 125 MHz osc
  • (disconnected) C.C. in2 <- CLK_CC_IN <- MGTREFCLK0_D <- SFP RX clock (cannot be used because of uncontrollable phase)
  • C.C. in2 <- CLK2_XO_125 <- FPGA AG8,AH8 (GC) <- SFP RX recovered 125 MHz clock
  • C.C. in3 <- CLK_FB
  • C.C. 125 MHz -> CLK_CC_OUT0 -> MGTREFCLK0_B -> QSFP RX and TX ref clock (final design)
  • C.C. 125 MHz -> CLK_CC_OUT1 -> MGTREFCLK1_D -> SFP TX clock
  • C.C. 62.5 MHz -> VX clock fanout

proposed changes:

  • add C.C. 125 MHz -> new CLK_CC_OUT2 (old CLK2_XO_125) -> FPGA AG8,AH8 (GC)
  • change C.C. in2 <- new CLK_CC_IN <- FPGA AK8,AK9 (non-GC)

Complete: 

125 MHz oscillator - U5 fan out -
  q0 -> CLK_XO_125 -> ENC C72,C74 -> FPGA R7,R8 MGTREFCLK0_A (not used)
  q1 -> U6 C.C. in1
  q2 -> CLK3_XO_125 -> ENC C7,C9 -> FPGA J7,J8 MGTREFCLK1_B -> SFP RX reference clock, QSFP RX and TX reference clocks (not final design!)
  q3 -> disconnected on the board, was CLK2_XO_125 -> ENC C151,C153 -> FPGA AG8,AH8 (GC)

U6 C.C (clock cleaner) -

in0 <- CLK_EXT1 (presumably GPS 10 MHz ref clock)
in1 <- 125 MHz oscillator via U5
in2 <- (was: CLK_CC_IN <- ENC B10,B12 <- FPGA D9,D10 MGTREFCLK0_D <- SFP RX clock, 125 MHz)
in2 <- CLK2_XO_125 <- ENC C151,B153 <- FPGA AG8,AH8 (GC) <- mgt_link_data_to_processing.rx_data_clk (SFP RX data clock, 125 MHz)
in3 <- CLK_FB

out0 -> CLK_CCA -> U12 (125 MHz)
out1 -> CLK_CCB -> VX1..6 (62.5 MHz)
out2 -> CLK_CCC -> VX7..12 (62.5 MHz)
out3 -> CLK_FB into in3

CLK_CCA -> U12 (125 MHz fan out) ->

Q0 -> not used
Q1 -> CLK_CC_OUT0 -> ENC C3-5 -> FPGA L7,L8 MGTREFCLK0_B -> QSFP RX and TX reference clocks (final design)
Q2 -> CLK_CC_OUT1 -> ENC B3-5 -> FPGA B9,B10 MGTREFCLK1_D -> SFP TX clock
Q3 -> not used
Q4 -> not used
Q5 -> SMA J9/J10

CLK_TP0 - FPGA N7,N8 MGTREFCLK1_A - ENC C69,C71 - DS-DM SMA J11, J12 (NOT IN CDM PROJECT)

Notes:
* CLK_XO_125 (125 MHz osc) is not used
* 62.5 MHz VX clock does not go into the FPGA
* CLK_EXT0 going to in1 of U5 cannot be used. only permitted frequency is 125 MHz (it drives the MGT reference clocks) and is this frequency is too high for LEMO cables and connectors

Proposed modifications:
- CLK_CC_IN: from FPGA output pin (ENC C142,C144 - FPGA AK8,AK9) to clock cleaner in2 (instead of CLK2_XO_125 pins)
- CLK_CCA -> U12 -> currently unused out3 -> CLK2_XO_125 FPGA pins
- repurpose CLK_EXT0 at GPS 1pps/IRIG input to FPGA

Clock distribution Rev1

Simplified:

  • 125 MHz osc -> CLK_XO_125 -> MGTREFCLK0_A -> not used
  • 125 MHz osc -> CLK3_XO_125 -> MGTREFCLK1_B -> SFP RX ref clock
  • 125 MHz osc -> C.C. in1
  • C.C. in0 <- CLK_EXT1 (10 MHz GPS clock)
  • C.C. in1 <- 125 MHz osc
  • C.C. in2 <- CLK_CC_IN <- FPGA AK9,AK8 <- SFP RX recovered clock, 125 MHz
  • C.C. in3 <- CLK_FB
  • C.C. 125 MHz -> CLK_CC_OUT0 -> MGTREFCLK0_B -> QSFP RX and TX ref clock
  • C.C. 125 MHz -> CLK_CC_OUT1 -> MGTREFCLK1_D -> SFP TX clock
  • C.C. 125 MHz -> CLK_CC_OUT2 -> FPGA AG8,AH8 (GC) -> not used
  • C.C. 62.5 MHz -> VX clock fanout

Complete: 

125 MHz oscillator - U5 fan out -
  q0 -> CLK_XO_125 -> ENC C72,C74 -> FPGA R8,R7 MGTREFCLK0_A -> XDC CLK_XO_125_P -> VHDL not used
  q1 -> U6 C.C. in1
  q2 -> CLK3_XO_125 -> ENC C7,C9 -> FPGA J8,J7 MGTREFCLK1_B -> XDC GDM missing, CDM CLK3_XO_125_P -> VHDL SFP RX reference clock (mgt_rx_ref_clk)
  q3 -> not used, was CLK2_XO_125

U6 C.C (clock cleaner) -

in0 <- CLK_EXT1 (GPS 10 MHz clock)
in1 <- 125 MHz oscillator via U5
in2 <- CLK_CC_IN <- ENC C142,C144 <- FPGA AK9,AK8 <- XDC GDM missing, CDM CLK_CC_IN1_P <- VHDL rx_clk
in3 <- CLK_FB

out0 -> CLK_CCA -> U12 (125 MHz)
out1 -> CLK_CCB -> VX1..6 (62.5 MHz)
out2 -> CLK_CCC -> VX7..12 (62.5 MHz)
out3 -> CLK_FB into in3

CLK_CCA -> U12 (125 MHz fan out) ->

Q0 -> not used
Q1 -> CLK_CC_OUT0 -> ENC C3,C5 -> FPGA L8,L7 MGTREFCLK0_B -> XDC GDM clk_mgtrefclk0_x0y1_p, CDM CLK_CC_OUT0_P -> VHDL  GDM mgt_b_ref_clk QSFP RX and TX reference clocks, CDM alternate rx_clk via clock mux
Q2 -> CLK_CC_OUT1 -> ENC B3,B5 -> FPGA B10,B9 MGTREFCLK1_D -> XDC CLK_CC_OUT1_P -> VHDL GDM not used, CDM mgt_tx_ref_clk SFP TX clock
Q3 -> CLK_CC_OUT2 -> ENC C151,C153 -> FPGA AG8,AH8 (GC) -> XDC GDM, CDM missing
Q4 -> not used
Q5 -> SMA J9/J10

CLK_TP0 <-> ENC C69,C71 <-> FPGA N8,N7 MGTREFCLK1_A (not used) -> XDC CLK_TP0_P -> VHDL not used (DS-DM SMA J11, J12)

Notes:
* Enclustra ENC-B is J801, ENC-C is J900
* "(GC)" is clock-capable FPGA pin
* CLK_XO_125 (125 MHz osc) is not used in FPGA
* 62.5 MHz VX clock does not go into the FPGA
* CLK_EXT0 going to in1 of U5 cannot be used. only permitted frequency is 125 MHz (it drives the MGT reference clocks) and is this frequency is too high for LEMO cables and connectors

I2C bus

  • I2C_SCL is J-ENC-A pin 111 I2C_SCL is FPGA I2C_SCL_PL AB13 (IO_L1N_TOL1D_64) and I2C_SCL_PS F18 (PS_MIO10)
  • I2C_SDA is J-ENC-A pin 113 I2C_SDA is FPGA I2C_SDA_PL AH13 (IO_L7N_T1L1Q_AD13N_64) and I2C_SDA_PS G18 (PS_MIO11)
  • XU8 secure EEPROM ATSHA204A at 0x64, this is 0110'010X -> linux _011'0010 is 0x32. (but responds to scan and read at 0x33)
  • U4 ethernet mac chip, EEPROM at 1010 A2 A1 A0 X and MAC/serial_no at 1011 A2 A1 A0 X. A0=VCC, A1=VCC, A2=GND -> linux _101'0011 and _101'1011 is 0x53 and 0x5B.
  • U6 clock chip, address 1101 0 A1 A0 X. A1=VCC, A0=N/C (internal pull-up) -> linux _110'1011 is 0x6b
  • U23 voltmeter at 10011 ADR1 ADR0. ADR0=GND, ADR1=VCC -> linux _100'1110 is 0x4e
  • SFP, address 1010000X -> linux _101'0000 is 0x50. additional SFP data at 0x51
  • QSFP0, QSFP1, QSFP2, QSFP3 (QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL)

I2C clock builder connection

  • use Silicon Labs USB "Clock builder pro field programmer", www.silabs.com/CBProgrammer
  • connect rainbow jumper cable pins:
    • black - 1-GND to GND on DS-DM
    • white - 3-SCLK to SCL on the DS-DM
    • grey - 7-SDA_SDIO to SDA on the DS-DM
  • power up the DS-DM
  • plug USB programmer into Windows laptop
  • on Windows, run "ClockBuilder Pro"
  • it should report "Field programmer detected", press "EVB GUI"
  • in EVB GUI, press "Config", set I2C address 0x6B
  • press "Scan", it should find Si5394A-A-GM
  • select the "Status" tab, should see real-time status of clock chip

GDM MGT configuration

  • TX configuration:
  • GDM MGT transceivers are configured as "multilane" TX and RX.
  • there is 12 TXes ("lanes")
  • MGT reference 125 MHz clock goes into [2:0]gtreclk00_in and [11:0]gtrefclk0_in
  • one MGT is designated as "master"
  • PLL of master MGT converts reference clock into common TX clock and common tx_user_clk2 which becomes tx_data_clk
  • common tx_user_clk2 aka tx_data_clk goes into all TXes and clocks tx_data.
  • tx_user_clk2 aka tx_data_clk is 125 MHz but not same phase as MGT reference clock.
  • RX interim configuration:
  • there is 12 RXes ("lanes")
  • each RX produces it's own recovered RX clock
  • "multilane" configuration assumes all RX recovered clocks run at the same frequency (TX on the other end are driven by common TX clock, see above), but have different phase
  • one RX recovered clock is designated as "master" (rx_user_clk2 aka rx_data_clk) and a phase-matching fifo/buffer is used to bring rx_data from all 12 RXes to this common rx_data_clk
  • this works because each CDM SFP TX runs on the SFP RX recovered clock which is frequency-locked with the GDM QSFP TX clock.
  • RX final configuration:
  • MGTs permit using the common TX clock (tx_user_clk2 aka tx_data_clk) as the common rx_data_clk (they are frequency locked through the CDM).
  • this permits use of tx_data_clk as the main clock domain in the GDM and removes the need to bring rx_data into the tx_data_clk domain (actually this is done in the MGT RX phase matching fifo/buffer).

Clock path

NOTE: MUST REVIEW!!! 

10 MHz ext clock or GDM 125 MHz oscillator
-> GDM QSFP MGT reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz (GDM main clock domain) and TX bit clock 2.5 GHz
-> GDM QSFP optic transmitter
-> CDM SFP optic receiver
-> CDM SFP MGT, RX reference clock is CDM 125 MHz oscillator
-> MGT RX recovered clock 125 MHz (CDM main clock domain)
-> CC_CLK_IN -> CDM C.C. -> CC_CLK_OUT1 -> CDM SFP TX reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz and TX bit clock 2.5 GHz
-> (tx_data phase matching fifo from CDM main clock domain to tx_data_clk)
-> CDM SFP optic transmitter
-> GDM QSFP RX optic receiver (12x)
-> GDM QSFP MGT (RX reference clock is same as TX reference clock)
-> MGT RX recovered clock (12x recovered clocks)
-> in multi-lane configuration, one of them is the "master" recovered clock rx_data_clk
-> (rx_data phase matching fifo from rx_data_clk to GDM main clock domain)

CDM rx_clk mux

when CDM SFP is not connected, there is no SFP recovered clock and a mux is used to switch between clk_cc_out0 (power up default) and rx_clk_mgt (SFP recovered clock)

Test SFP disconnected

note: if I say "--cc-in1", CC seems to lock on the 10 MHz GPS external clock, to prevent this, test sequence includes reloading the CC and the reset of MGT. 

./test_cdm_local.exe --write32 0x30 0x0 ### rx_clk mux select CC clock
./test_cdm_local.exe --load-cc
./test_cdm_local.exe --reset-mgt
./test_cdm_local.exe --cdm-clocks
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773594e (125000014) should be ~125 MHz  <=== all 4 clocks wobble close to 125 MHz
0x1034 rx_clk:             0x0773594f (125000015) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x0773594f (125000015) should be ~125 MHz
0x103C tx_clk:             0x0773594f (125000015) should be ~125 MHz
0x1040 clk_50MHz:          0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk:         0x05f5e100 (100000000) should be 100 MHz exactly
./test_cdm_local.exe --write32 0x30 0x1 ### rx_clk mux select SFP recovered clock
./test_cdm_local.exe --cdm-clocks
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773598b (125000075) should be ~125 MHz
0x1034 rx_clk:             0x076d58ec (124606700) should be ~125 MHz  <=== off frequency because there is no valid SFP recovered clock
0x1038 mgt_tx_ref_clk_raw: 0x0773598c (125000076) should be ~125 MHz
0x103C tx_clk:             0x0773598b (125000075) should be ~125 MHz
0x1040 clk_50MHz:          0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk:         0x05f5e100 (100000000) should be 100 MHz exactly

Clock domains

GPS

  • no GPS : GDM runs from internal 125 MHz oscillator
  • external 10 MHz clock : GDM runs from external 10 MHz clock and optional 1pps signal (use VME-NIMIO32 NIM outputs)
  • GPS receiver : GDM runs from GPS 10 MHz clock and GPS IRIG serial data
  • LNGS GPS:
    • provides 1pps and serial data over fiber from GPS receiver (master). LNGS xxx box is not used.
    • serial data goes to GDM, decoded, 1pps signal extracted, goes to LEMO output
    • 1pps from GDM is used to train the Rubidium clock which provides a 10 MHz clock
    • 10 MHz output from Rubidium clock goes to GDM 10 MHz external clock input

GDM

  • AXI clock (100 MHz) - AXI registers
  • 125 MHz oscillator - to clock cleaner
  • 10 MHz external clock LEMO input - to clock cleaner
  • FPGA 125 MHz clock CLK_CC_IN - to clock cleaner - not used (could be used for cascaded GDMs)
  • 125 MHz oscillator - CLK3_XO_125 - mgt_b_ref_clk QSFP MGT reference clock (interim GDM design)
  • clock cleaner output 125 MHz fanout:
    • CLK_CC_OUT0 - QSFP MGT reference clock (final design)
    • CLK_CC_OUT1 - not used (CDM SFP reference clock)
    • CLK_CC_OUT2 - not used
  • QSFP MGT TX data clock 125 MHz tx_data_clk (main clock domain)
    • QSFP TX data
    • QSFP RX data (in final design, rx_data_clk is same as tx_data_clk. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk == tx_data_clk)
    • ds20k block
  • (in interim design, rx_data_clk is disconnected from tx_data_clk. rx_data_clk is the "multilane master clock" which is one of the 12x RX recovered clocks. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk)

note: all these clocks are frequency locked to 125 MHz

CDM

  • AXI clock (100 MHz) - AXI registers
  • 10 MHz external clock LEMO input - to clock cleaner (not used)
  • 125 MHz oscillator to fanout
    • to clock cleaner
    • to SFP MGT RX reference clock: CLK3_XO_125 to mgt_rx_ref_clk to gtrefclk01_in
  • SFP MGT RX recovered clock 125 MHz
    • MGT PLL to MGT rx_user_clk2 aka rx_data_clk
    • MGT rxrecclkout_out to mgt_rx_rec_clk to CLK_CC_IN to C.C.
  • SFP MGT rx_user_clk2 aka rx_data_clk (250 MHz/8 bit, 125 MHz/16 bit, 62.5 MHz/32 bit data) (main clock domain)
    • SFP RX data
    • ds20k block
    • VX TX clock PLLs
    • VX RX clock PLLs
  • C.C. fan out
    • 62.5 MHz VX clocks (12x)
    • CLK_CC_OUT0 (not used, GDM QSFP MGT reference clock)
    • CLK_CC_OUT1 125 MHz to mgt_tx_ref_clk to gtrefclk00_in to SFP MGT TX reference clock
    • CLK_CC_OUT2 (not used)
  • SFP MGT tx_user_clk2 aka tx_data_clk
    • SFP TX data
    • TX data phase matching fifo from main clock domain to tx_data_clk
  • VX TX clock PLLs
    • 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
    • VX TX data phase matching from main clock domain to VX TX clock (12 total)
    • VX TX serializer
    • VX TX LVDS transmitter
  • VX RX clock PLLs
    • 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
    • VX RX LVDS receivers (12 total)
    • VX RX deserializers (12 total)
    • VX TX data phase matching from VX RX clock to main clock domain

VX

  • everything runs on the VX main 125 MHz clock
  • correct phase of VX to CDM LVDS data is adjusted by scan of CDM VX RX PLL clock phase (VX to CDM link is now established)
  • correct phase of CDM to VX LVDS data is adjusted by scan of CDM VX RX PLL clock phase (link is established after VX to CDM idle data pattern changes from "VX RX data bad" to "good".
  • after good phases are found by scan, they are not expected to change unless cables are changed, CDM and VX modules are changed or CDM firmware is rebuilt. (rebuild of VX firmware should not affect LVDS data phase).
  • if there is excessive link errors, phase scan must be repeated.

Board test plan

To test:

  • Enclustra FPGA board
  • SFP port - SFP_RS0/RS1 connected to QSFP0_SEL/1_SEL is wrong? our Finisar SFP says RS0, RS1 N/C, so probably okey.
  • SW5 CLK_EXT1 NIM works. TTL needs to be tested. flipping SW5 CLK_EXT0 side from NIM to TTL makes CLK_EXT1 go LOS and OOF in the clock chip. R23 and R53 should be removed?
  • BOOT_MODE 0 and 1

Partial:

  • U23 3.3V current meter and thermometer. V1,V2 is current monitor, same as in application note. V3,V4 is thermometer, same in application note, except capacitor C118 is 0.1u instead of 470pF. Tested ok: Tint, VCC, V1, V2, V1-V2. Test failed: TR2 reads 50-something degC instead of same as Tint.

Done:

  • LED_FP1A..D: tested ok. K.O. 15 sep 2022
  • USB UART: tested ok. K.O. 15 sep 2022
  • J4A, J4B, J5A, J5B LEMO inputs (NIM/TTL) EXT_IN_LV(1..4). TTL threshold 1.7V, NIM threshold -0.3V. 50 Ohm termination. TTL straight, NIM inverted.
  • J6A, J6B LEMO clock inputs: 10 MHz TTL from chronobox works, 10 MHz NIM from IO32 works. 18-apr-2023
  • J7A, J7B LEMO outputs EXT_OUT(1), EXT_OUT(2) (NIM/TTL) (tested 23nov2022, K.O.). TTL ???, NIM inverted.
    • TTL out no 50 ohm termination: 0=0V, 1=5V, rise and fall time ~5 ns
    • TTL out with 50 ohm termination: 0=0V, 1=2.5V, rise and fall time <2ns
    • NIM out no 50 ohm termination: 0=+50mV, 1=-1.8V, rise and fall time ~3ns
    • NIM out with 50 ohm termination: 0=0V, 1=-0.9V, rise and fall time <2ns
  • ethernet MAC i2c chip (K.O. 20sep2022: can read: i2cdump 0 0x5b, nov2022 have u-boot driver)
  • CDM VX ports 2x(CLK, 3 tx, 4 rx) tested using test_cdm.exe and LVDS loopback in VX firmware.
  • SFP i2c tested KO 22jun2023
  • QSFP i2c tested KO 22jun2023
  • i2c testing complete 22jun2023
  • QSFP rx,tx tested 26june2023. lane0,1,2 ok, lane3 tx not connected, laser is off. qsfp0,1,2,3 all lanes ok.
  • SFP rx,tx tested 26june2023. rx and tx okey. LOS ok, mod_absent ok.

Failure:

  • ethernet: does not connect to alliedtelesys switch. connects to my USB-eth adapter at 100 Mbit speed. uboot mii status reports connection speed oscillating between 1000, 100 and 10. K.O. 16-sep-2022
  • SFP LOS and mod_absent are swapped (in the FPGA pin definitions?)
  • QSFP slot numbering is wrong.

Checklist for newly build boards

  • put new board on workbench
  • check - vme connector present, vme extraction handles present
  • check - standoff are removed from all thru-holes
  • plug Enclustra module
  • check - SW6 both switches are in the "PS" position
  • connect micro-usb cable from linux PC
  • connect ethernet from 1gige capable network switch
  • connect power from lab power supply - GND, +5V and -12V nominal, +5.7V and -12.5V actual
  • power up, +5V current 2.10-2.8A, -12V current 0.05A
  • on linux PC, open a new terminal, run: minicom -D /dev/ttyACM0
  • in minicom window, observe messages about Xilinx first stage boot loader, etc
  • on the ethernet switch, observe network link is 1gige speed (not 10mbit, not 100mbit).
  • if everything boots okey, there will be a login prompt, login as root, password root.
  • busybox devmem 0x80010000 # read firmware version number, i.e. 0xEDAD0A77
  • TBW - test LEMO inputs
  • TBW - test LEMO outputs
  • TBW - test VX connectors
  • TBW - test SFP connector
  • TBW - test QSFP connector

Serial console

i2c

ZynqMP> i2c bus
Bus 0:  i2c@ff020000
ZynqMP> i2c dev 0
Setting bus to 0
ZynqMP> i2c probe  
Valid chip addresses: 33 4E 53 5B 6B 77
ZynqMP> i2c md 0x5b 0x98
0098: fc c2 3d 00 00 12 1a 6e 0a 90 85 04 94 10 08 50    ..=....n.......P

root@gdm-cdm:~# i2cdetect 0
Warning: Can't use SMBus Quick Write command, will skip some addresses
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0.
I will probe address range 0x03-0x77.
Continue? [Y/n] 
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                                                 
10:                                                 
20:                                                 
30: -- -- -- 33 -- -- -- --                         
40:                                                 
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- -- 
60:                                                 
70:                                                 
root@gdm-cdm:~# 
root@gdm-cdm:~# i2cdump 0 0x5b
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0, address 0x5b, mode byte
Continue? [Y/n] 
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
80: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00    ???????P?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e    ........??=..??n
a0: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00    ???????P?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e    ........??=..??n
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
root@gdm-cdm:~# 

root@gdm0:~# i2cdetect -r 0
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e -- 
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- -- 
70: -- -- -- -- -- -- -- --                         
root@gdm0:~# 

root@cdm0:~# i2cdetect -r 0
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e -- 
50: 50 51 -- 53 -- -- -- -- -- -- -- 5b -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- -- 
70: -- -- -- -- -- -- -- --                         
root@cdm0:~#
  • 0x33 - XU8 secure EEPROM (should be at 0x32)
  • 0x4e - U23 current and temperature monitor
  • 0x50, 0x51 - SFP
  • 0x50 - QSFP, 4 QSFP modules enabled by GPIO QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL
  • 0x53, 0x5b - ethernet mac eeprom
  • 0x6b - U6 clock chip

U23

NOTE: Text reads double of Tint. not sure why. K.O. 21-mar-2024.

  • internal temperature only
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x00 b  # control register: "repeat mode, internal temperature only"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x03 # "Tint ready" and "busy", "busy is always 1 in repeat mode"
root@gdm0:~# i2cdump -y 0 0x4e b
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 03 00 03 03 81 db 2a ce 2a 8e 00 6f 00 45 20 3f    ?.????*?*?.o.E ?
10: 01 00 01 01 01 db 2a ce 2a 8e 00 6f 00 45 20 3f    ?.????*?*?.o.E ?
...
readback:
reg0 - 03 - Tint ready
reg1 - 00 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB 0x81, bit 0x80 is "DV, data valid", bit 0x40 is "SS, sensor short", 0x20 is "SO, sensor open"
reg5 - Tint LSB 0xDB, Tint = 0x01DB = 475 * 0.0625 degC = 29.6 degC
reg6..F - stale data
  • Tint, V1, V2, TR2, VCC
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x18 b  # control register: "repeat mode, V1, V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 7f 18 7f 7f 81 b9 aa d0 aa 8f 83 5e 83 5e a0 41    ???????????^?^?A
10: 01 18 01 01 01 b9 2a d0 2a 8f 03 5e 03 5e 20 41    ??????*?*??^?^ A
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x1b9*0.0625 = 27.5 degC
reg6 - V1 MSB 0xaa, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 LSB 0xd0, V1 = 0x2ad0*305.18/1000000 = 3.3447 V (correct)
reg8 - V2 MSB 0xaa, ditto
reg9 - V2 LSB 0x8f, V2 = 0x2a8f*305.18/1000000 = 3.3249 V (correct, smaller than V1)
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x5e. TR2 = 0x35e*0.0625 = 53.875 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x41, VCC = 2.5+0x2041*305.18/1000000 = 5.019 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.99 A (about right?)
  • Tint, V1-V2, TR2, VCC
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x19 b  # control register: "repeat mode, V1-V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 7f 19 7f 7f 81 69 83 f2 83 f2 83 3a 83 3a a0 44    ?????i?????:?:?D
10: 01 19 01 01 01 69 03 f2 03 f2 03 3a 03 3a 20 44    ?????i?????:?: D
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x169*0.0625 = 22.5 degC
reg6 - V1 or V1-V2 MSB 0x83, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 or V1-V2 LSB 0xf2, V1-V2 = 0x3f2*19.42/1000000 = 0.0196142 V (correct, compare with V1 and V2 measured above)
reg8 - V2 or V1-V2 MSB
reg9 - V2 or V1-V2 LSB
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x3a. TR2 = 0x33a*0.0625 = 51.625 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x44, VCC = 2.5+0x2044*305.18/1000000 = 5.021 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.9807 A (about right?)

SFP

root@cdm0:~# i2cdump 0 0x50
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 03 04 07 00 00 00 00 40 40 0c 00 01 3d 00 00 00    ???....@@?.?=...
10: 0c 02 00 1e 46 49 4e 49 53 41 52 20 43 4f 52 50    ??.?FINISAR CORP
20: 2e 20 20 20 00 00 90 65 46 54 4c 46 38 35 32 36    .   ..?eFTLF8526
30: 50 33 42 4e 4c 20 20 20 41 20 20 20 03 52 00 9d    P3BNL   A   ?R.?
40: 00 1a 00 00 4e 33 41 42 34 4c 56 20 20 20 20 20    .?..N3AB4LV     
50: 20 20 20 20 32 30 30 33 31 39 20 20 68 f0 03 de        200319  h???
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
root@cdm0:~# 
root@cdm0:~# i2cdump 0 0x51
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 5a 00 d8 00 55 00 e2 00 90 88 71 48 8c a0 75 30    Z.?.U.?.??qH??u0
10: 21 34 01 f4 1b 58 03 e8 31 2d 04 eb 1f 07 06 31    !4???X??1-?????1
20: 31 2d 00 64 27 10 00 9e 00 00 00 00 00 00 00 00    1-.d'?.?........
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
40: 00 00 00 00 3f 80 00 00 00 00 00 00 01 00 00 00    ....??......?...
50: 01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 b7    ?...?...?......?
60: 19 9f 80 c5 0e 17 12 c4 1f 99 00 00 00 00 30 00    ??????????....0.
70: 00 00 00 00 00 00 00 00 ff ff ff ff ff ff ff 01    ...............?
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
root@cdm0:~#

QSFP

QSFP i2c enable lines, active low:

QSFP0_SEL - JENC-A 82 - PS-MIO40 - linux gpio 378
QSFP1_SEL - JENC-A 84 - PS-MIO41 - linux gpio 379
QSFP2_SEL - JENC-A 100 - PS-MIO44 - linux gpio 382
QSFP3_SEL - JENC-A 106 - PS-MIO43 - linux gpio 381 (notice 2 and 3 are out of order)

# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
# echo 378 >> /sys/class/gpio/export ### SEL0 338+40
# echo 379 >> /sys/class/gpio/export ### SEL1 338+41
# echo 381 >> /sys/class/gpio/export ### SEL3 338+43
# echo 382 >> /sys/class/gpio/export ### SEL2 338+44
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
 gpio-378 (                    |sysfs               ) in  hi 
 gpio-379 (                    |sysfs               ) in  hi 
 gpio-381 (                    |sysfs               ) in  hi 
 gpio-382 (                    |sysfs               ) in  hi 
root@gdm0:~# echo out >> /sys/class/gpio/gpio381/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio382/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio378/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio379/direction
root@gdm0:~# 
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
 gpio-378 (                    |sysfs               ) out lo 
 gpio-379 (                    |sysfs               ) out lo 
 gpio-381 (                    |sysfs               ) out lo 
 gpio-382 (                    |sysfs               ) out lo 
root@gdm0:~#
echo 1 >> /sys/class/gpio/gpio381/value
echo 1 >> /sys/class/gpio/gpio382/value
echo 1 >> /sys/class/gpio/gpio378/value
echo 1 >> /sys/class/gpio/gpio379/value
cat /sys/kernel/debug/gpio
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
 gpio-378 (                    |sysfs               ) out hi 
 gpio-379 (                    |sysfs               ) out hi 
 gpio-381 (                    |sysfs               ) out hi 
 gpio-382 (                    |sysfs               ) out hi 
root@gdm0:~# 
root@gdm0:~# i2cdetect -y -r 0
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e -- 
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- -- 
70: -- -- -- -- -- -- -- --                         
root@gdm0:~# 

NOTICE NOTHING AT ADDRESS 0x50

ENABLE QSFP0, OBSERVE IT IS AT ADDRESS 0x50

root@gdm0:~# echo 0 >> /sys/class/gpio/gpio378/value
root@gdm0:~# i2cdetect -y -r 0
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e -- 
50: 50 -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- -- 
70: -- -- -- -- -- -- -- --                         
root@gdm0:~# 
root@gdm0:~# i2cdump -y 0 0x50
No size specified (using byte-data access)
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 0d 00 02 8e 00 00 01 00 00 05 55 00 05 00 00 00    ?.??..?..?U.?...
10: 00 00 00 00 00 00 1e b8 00 00 81 a0 00 00 00 00    ......??..??....
20: 00 00 12 12 00 01 00 01 00 01 0e e0 0f 20 0e e0    ..??.?.?.???? ??
30: 00 00 1f a0 1e 3b 1f 72 00 01 00 00 00 00 00 00    ..???;?r.?......
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
60: 00 00 00 00 00 00 00 00 00 00 1f 00 00 00 08 00    ..........?...?.
70: 00 00 00 00 00 00 00 ff ff ff ff ff ff ff ff 00    ................
80: 0d 00 0c 04 00 00 00 40 40 02 d5 05 67 00 00 96    ?.??...@@???g..?
90: 00 00 c8 00 46 49 4e 49 53 41 52 20 43 4f 52 50    ..?.FINISAR CORP
a0: 20 20 20 20 07 00 90 65 46 54 4c 34 31 30 51 44        ?.?eFTL410QD
b0: 34 43 20 20 20 20 20 20 41 20 42 68 07 d0 00 43    4C      A Bh??.C
c0: 00 07 0f de 58 37 39 41 43 30 52 20 20 20 20 20    .???X79AC0R     
d0: 20 20 20 20 32 32 30 33 30 39 20 20 3c 10 00 9e        220309  <?.?
e0: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20                    
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
root@gdm0:~#

ethernet mac eeprom

  • correct chip with 84-bit ethernet mac address
root@cdm1:~# i2cdump 0 0x53
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
root@cdm1:~# 

root@cdm1:~# i2cdump 0 0x5b
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
80: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00    ????4??2?K?.?...
90: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c    ..........??=?Q<
a0: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00    ????4??2?K?.?...
b0: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c    ..........??=?Q<
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
root@cdm1:~#
  • wrong "602" chip with 64-bit IPv6 address
root@cdm0:~# i2cdump 0 0x53
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
root@cdm0:~# i2cdump 0 0x5b
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
80: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00    ???????Q?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e    ........??=..??.
a0: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00    ???????Q?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e    ........??=..??.
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
root@cdm0:~#

read ethernet mac address from i2c

(this code is copied from uboot command line i2c code)

in uboot sources board/xilinx/common/board.c replace original function with this: 

// special code to read ethernet MAC address from the DS-DM-Rev0 board. K.O. Sep-2022

int zynq_board_read_rom_ethaddr(unsigned char *ethaddr)
{
        struct udevice *bus;
	int ret;
        int busnum = 0;

	ret = uclass_get_device_by_seq(UCLASS_I2C, busnum, &bus);
	if (ret) {
           printf("%s: No bus %d\n", __func__, busnum);
           return ret;
	}

        int chip_addr = 0x5B;

        struct udevice *dev;

        ret = i2c_get_chip(bus, chip_addr, 1, &dev);
	if (ret) {
           printf("%s: Bus %d no chip 0x%02x\n", __func__, busnum, chip_addr);
           return ret;
	}

        int dev_addr = 0x98;

        unsigned char data[8];
           
        ret = dm_i2c_read(dev, dev_addr, data, 8);

	if (ret) {
           printf("%s: Bus %d chip 0x%02x read error %d\n", __func__, busnum, chip_addr, ret);
           return ret;
	}

        printf("%s: Bus %d chip 0x%02x addr 0x%02x read: 0x %02x %02x %02x %02x %02x %02x %02x %02x\n", __func__, busnum, chip_addr, dev_addr, data[0], data[1], data[2], data[3], data[4], data[5], data[6], data[7]);

        // see Atmel-8807-SEEPROM-AT24MAC402-602-Datasheet.pdf

        if (data[0] == 0) {
           // eiu-48 chip
           ethaddr[0] = data[2];
           ethaddr[1] = data[3];
           ethaddr[2] = data[4];
           ethaddr[3] = data[5];
           ethaddr[4] = data[6];
           ethaddr[5] = data[7];
        } else {
           // eiu-64 chip
           ethaddr[0] = data[0];
           ethaddr[1] = data[1];
           ethaddr[2] = data[2];
           ethaddr[3] = data[5];
           ethaddr[4] = data[6];
           ethaddr[5] = data[7];
        }

        printf("%s: ethaddr %02x:%02x:%02x:%02x:%02x:%02x\n", __func__, ethaddr[0], ethaddr[1], ethaddr[2], ethaddr[3], ethaddr[4], ethaddr[5]);

        return ret;
}

also this should have worked if i2c_xxx() functions were enabled in uboot: 

i2c_set_bus_num(0);
i2c_probe(0x5b);
i2c_read(0x5b, 0x9a, ethaddr, 6);

read ethernet mac address from i2c (SHOULD WORK)

from: https://stackoverflow.com/questions/43637540/is-there-linux-or-u-boot-support-to-read-a-mac-address-from-a-chip-at-startup 

#ethernet related setup
setup_eth=run readmac buildmac
#read mac address from eeprom
readmac=i2c dev 0; i2c read 50 FA.1 6 $loadaddr
#build the ethaddr variable
#not very nice, but does the job
buildmac=\
e=" "; sep=" " \
for i in 0 1 2 3 4 5 ; do\
setexpr x $loadaddr + $i\
setexpr.b b *$x\
e="$e$sep$b"\
sep=":"\
done &&\
setenv ethaddr $e

read ethernet mac address from i2c (DOES NOT WORK)

this method does not work: inside board/xilinx/common/board.c:int zynq_board_read_rom_ethaddr(unsigned char *ethaddr), uclass_get_device_by_ofnode(UCLASS_I2C_EEPROM, eeprom, &dev) returns ENODEV and read from i2c does not happen. K.O. Sep-2022

Read:

Note:

  • 0x5B is the i2c chip address
  • 0x9A is the data offset inside the chip, see datasheet or i2c read dump above.

Edit:

  • emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/u-boot/files/platform-top.h
#include <configs/xilinx_zynqmp.h>
#include <configs/platform-auto.h>
//#define CONFIG_I2C_EEPROM                                                                                                                                                                   
//#define CONFIG_SYS_I2C_EEPROM_ADDR 0x5b                                                                                                                                                     
//#define CONFIG_SYS_I2C_EEPROM_ADDR_OVERFLOW 0x0                                                                                                                                             
#define CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET 0x9A
#error HERE!
  • emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi
/include/ "system-conf.dtsi"
/ {
chosen {
   xlnx,eeprom = &eeprom;
};
};

&i2c0 {
eeprom: eeprom@5b { /* u88 */
compatible = "atmel,24mac402";
reg = <0x5b>;
};
};
  • components/yocto/workspace/sources/u-boot-xlnx/configs/xilinx_zynqmp_virt_defconfig
CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET=0x9A

RTC chip

  • DS3231 RTC chip
  • FPGA connections:
I2C SCL <- J-ENC A85 <- FPGA E17 <- XDC TP_S <- VHDL TP_S is output
I2C SDA <-> J-ENC A87 <-> FPGA D17 <-> XDC TP_S <- VHDL TP_S is output
1pps -> J-ENC B129 -> FPGA AE3 -> XDC "free pin"
1pps -> J-ENC C160 -> FPGA AH12 -> XDC "slow_io"

enable VX clock

devmem 0x80010010 32 0x8; sleep 1;  devmem 0x80010010 32 0x0;
si5394-i2c-file /media/sd-mmcblk1p1/00_freerun.txt 0 0x6b

clock chip configuration

file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt:

  • VCO is 14 GHz
  • Tvco is 71.43 ps
  • N0 divider is 14, frequency is 1000 MHz
  • out0 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz
  • out1 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
  • out2 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
  • out3 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz is the feedback for zero delay

clock chip monitoring

from si5395-94-92-family.pdf:

  • reg 0x1: page select, set to 0 or set to 5 to read 0x53F
  • reg 0x2: 0x94
  • reg 0x3: 0x53 -> device is a si5394
  • reg 0xC: LOSXAXB
  • reg 0xD: LOS and OOF for the 4 clock inputs
  • reg 0xE: LOL and HOLD
  • reg 0xF: CAL_PLL
  • reg 0x11: sticky bits for reg 0xC
  • reg 0x12: sticky bits for reg 0xD
  • reg 0x13: sticky bits for reg 0xE
  • reg 0x14: sticky bits for reg 0xF
  • reg 0x1C: device reset
  • reg 0x1E: low power, hard reset, SYNC
  • reg 0x507: currently selected input clock
  • reg 0x52A: input clock select
  • reg 0x535: FORCE_HOLD
  • reg 0x53F: HOLD_HIST_VALID and FASTLOCK_STATUS

arm and linux benchmark

memory benchmark: 

daq13$ arm-linux-gnueabi-gcc -o memcpy.armv7 memcpy.cc -march=armv7 -static -O2
scp memcpy.armv7 to ...
root@gdm-cdm:~# ./memcpy.armv7 
memcpy       1 KiBytes:   1288 MB/sec
memcpy       2 KiBytes:   1924 MB/sec
memcpy       4 KiBytes:   2554 MB/sec
memcpy       8 KiBytes:   3054 MB/sec
memcpy      16 KiBytes:   3262 MB/sec
memcpy      32 KiBytes:   3250 MB/sec
memcpy      64 KiBytes:   3456 MB/sec
memcpy     128 KiBytes:   3556 MB/sec
memcpy     256 KiBytes:   3780 MB/sec
memcpy     512 KiBytes:   3795 MB/sec
memcpy    1024 KiBytes:   3789 MB/sec
memcpy    2048 KiBytes:   3729 MB/sec
memcpy    4096 KiBytes:   3717 MB/sec
memcpy    8192 KiBytes:   3687 MB/sec
memcpy   16384 KiBytes:   3632 MB/sec
memcpy   32768 KiBytes:   3529 MB/sec
memcpy   65536 KiBytes:   3318 MB/sec
memcpy  131072 KiBytes:   2893 MB/sec
root@gdm-cdm:~#

ethernet receive: 

daq13:bin$ ./ttcp -t -s -n 100000 10.0.0.24
ttcp-t: buflen=8192, nbuf=100000, align=16384/0, port=5001  tcp  -> 10.0.0.24
ttcp-t: socket
ttcp-t: connect
ttcp-t: 819200000 bytes in 7.25 real seconds = 110358.39 KB/sec +++
ttcp-t: 100000 I/O calls, msec/call = 0.07, calls/sec = 13794.80
ttcp-t: 0.0user 0.2sys 0:07real 3% 0i+0d 760maxrss 0+2pf 1461+31csw
daq13:bin$ 

root@gdm-cdm:~# ./ttcp.armv7 -r -s
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001  tcp
ttcp-r: socket
ttcp-r: accept from 10.0.0.25
ttcp-r: 819200000 bytes in 7.27 real seconds = 110098.22 KB/sec +++
ttcp-r: 212040 I/O calls, msec/call = 0.04, calls/sec = 29181.53
ttcp-r: 0.1user 5.7sys 0:07real 81% 0i+0d 584maxrss 0+2pf 125601+2699csw
root@gdm-cdm:~#

ethernet transmit: 

root@gdm-cdm:~# ./ttcp.armv7 -t -s -n 100000 10.0.0.25
ttcp-t: buflen=8192, nbuf=100000, align=16384/0, port=5001  tcp  -> 10.0.0.25
ttcp-t: socket
ttcp-t: connect
ttcp-t: 819200000 bytes in 6.95 real seconds = 115078.69 KB/sec +++
ttcp-t: 100000 I/O calls, msec/call = 0.07, calls/sec = 14384.84
ttcp-t: 0.0user 0.7sys 0:06real 11% 0i+0d 584maxrss 0+2pf 1162+1017csw
root@gdm-cdm:~# 

daq13:bin$ ./ttcp -r -s
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001  tcp
ttcp-r: socket
ttcp-r: accept from 10.0.0.24
ttcp-r: 819200000 bytes in 6.97 real seconds = 114841.84 KB/sec +++
ttcp-r: 161335 I/O calls, msec/call = 0.04, calls/sec = 23160.01
ttcp-r: 0.0user 1.9sys 0:06real 28% 0i+0d 760maxrss 0+2pf 80646+51csw
daq13:bin$

Install Xilinx tools

  • install Vivado 2020.2
login at https://www.xilinx.com/myprofile.html
go to "Downloads"
go to archive,
find 2020.2
download Xilinx_Unified_2020.2_1118_1232_Lin64.bin
sh ./Xilinx_Unified_2020.2_1118_1232_Lin64.bin
banner window should open with spinner "downloading installation data"
"a newer version is available" -> say "continue"
next
"select install type" window:
provide email and password,
select "download image"
select directory /home/olchansk/Xilinx/Downloads/2020.2\
select "linux" and "full image"
next
download summary: space required 38.52 Gbytes
download
installation progress
downloading spinner, 16 M/s 47 minutes...
"download image has been created successfully". Ok.
check contents of /home/olchansk/Xilinx/Downloads/2020.2
ls -l /home/olchansk/Xilinx/Downloads/2020.2
total 67
drwxr-xr-x 2 olchansk users    9 Sep  1 16:22 bin
drwxr-xr-x 3 olchansk users   15 Sep  1 16:23 data
drwxr-xr-x 4 olchansk users    4 Sep  1 16:22 lib
drwxr-xr-x 2 olchansk users  644 Sep  1 16:22 payload
drwxr-xr-x 2 olchansk users    7 Sep  1 16:22 scripts
drwxr-xr-x 4 olchansk users    4 Sep  1 16:22 tps
-rwxr-xr-x 1 olchansk users 3256 Nov 18  2020 xsetup
daq13:2020.2$ 
./xsetup
spinned loading installation data
xilinx design tools 2022.1 now available -> say continue
"welcome" -> next
"select product" -> vivado -> next -> vivado hl system edition -> next
select devices: only zynq ultrascale+ mpsoc -> next
select destination: /opt/Xilinx (as root, mkdir /opt/Xilinx, chmod olchansk.users /opt/Xilinx)
install ...
complete
move /home/olchansk/Xilinx/Downloads/2020.2 to /daq/daqstore/olchansk/Xilinx/Downloads/
  • install petalinux 2020.2
./xsetup
"a newer version is available" -> say "continue"
next
"select product to install" -> select Petalinux (Linux only) -> next
"select destination directory" -> select "/opt/Xilinx" (disk space required 2.64 GB) -> next
"summary" -> install ...
error about missing /tmp/tmp-something files
"installation completed successfully" (hard to dismiss, "ok" button is partially cut-off)
done?
I think it failed, /opt/Xilinx/PetaLinux/2020.2/bin is empty except for petalinux-v2020.2-final-installer.run
try to run it by hand, same error about /tmp/tmp-something files. strange...
notice it complains about "truncate", which truncate finds ~/bin/truncate, get rid of it,
try again
now complains about missing texinfo and zlib1g:i386
apt install texinfo -> ok
apt install zlib1g:i386 -> installs bunch of gcc stuff -> ok
try again
reports "already installed" -> delete /opt/Xilinx/.xinstall/PetaLinux_2020.2/, delete entries in ~/.Xilinx/registry/installedSW.xml
try again
success
  • install vivado 2022.1 and petalinux 2022.1 - everything is pretty much the same

Petalinux

  • cd PetaLinux_GDM_CDM
  • petalinux-config
  • enable i2c MAC address and DHCP
git clone https://bitbucket.org:/team-ds-dm/ds-dm-u-boot-xlnx.git
cd ds-dm-u-boot-xlnx
git checkout ds-dm-u-boot-xlnx

linux-components ->
uboot -> ext-local-src
external u-boot local source -> ds-dm-u-boot-xlnx (path to the customized uboot git repository)
  • enable DHCP
Subsystem AUTO Hardware Settings -> Ethernet Settings
randomize MAC address -> NO
ethernet mac address -> leave empty
obtain ip address automatically -> YES
  • set hostname and product names
Firmware Version Configuration ->
Host name -> "ds-dm"
Product name -> "Petalinux_GDM_CDM"
  • configure linux kernel
petalinux-config -c kernel
  • enable NFS-Root
petalinux-config
Image Packaging Configuration > Root File System Type -> set to NFS
Location of NFS root directory set to "/nfsroot"

petalinux-config -c kernel
Networking support > IP: kernel level configuration
enable DHCP, BOOTP, RARP
File systems > Network file systems > Root file systems on NFS
  • manually fix linux kernel command line:
grep nfsroot PetaLinux_GDM_CDM/project-spec/configs/config
edit CONFIG_SUBSYSTEM_BOOTARGS_GENERATED to read
earlycon console=ttyPS0,115200 clk_ignore_unused panic=60 root=/dev/nfs nfsroot=/nfsroot/%s ip=dhcp rw
  • check configuration in
    • PetaLinux_GDM_CDM/project-spec/configs/config
    • PetaLinux_GDM_CDM/project-spec/configs/rootfs_config
    • PetaLinux_GDM_CDM/components/plnx_workspace/device-tree/device-tree/system-conf.dtsi

JTAG server

localhost:3121

ds20k block

module ds20k
  (
   //    CLOCK INPUTs
   input wire clk,
   input reset, // pulse for power-up reset
   input wire pll_is_locked, // clock cleaner PLL is locked to selected input clock

   //    REGISTER_DATA
   
   input wire [255:0] [31:0] register_data_in,
   output reg [255:0] [31:0] register_data_out,
   input wire register_write_strobe, // pulse when AXI write transaction puts new data in register_data_in
   input wire register_read_ack, // pulse after AXI read transaction captures data from register_data_out, used to read from FIFO
   
   //    GDM QSFP FIBER LINKS
   output reg [11:0] [15:0] qsfp_tx_data,
   output reg [11:0] [1:0]  qsfp_tx_ctrl,
   input wire [11:0] [15:0] qsfp_rx_data_error, // not sure what this is.
   input wire [11:0] [15:0] qsfp_rx_data,
   input wire [11:0] [1:0]  qsfp_rx_ctrl,
   input wire [11:0] qsfp_rx_is_good, // single bit indicating that RX link is up and data is good.

   // CDM SFP FIBER LINKS
   output reg [15:0] sfp_tx_data,
   output reg [1:0]  sfp_tx_ctrl,
   input wire [15:0] sfp_rx_data,
   input wire [1:0]  sfp_rx_ctrl,
   input wire        sfp_rx_is_good, // single bit indicating that RX link is up and data is good.
      
   //    VX_RXs
   input wire [3:0] vx1_rx,
   input wire [3:0] vx2_rx,  
   input wire [3:0] vx3_rx,  
   input wire [3:0] vx4_rx,  
   input wire [3:0] vx5_rx,  
   input wire [3:0] vx6_rx,  
   input wire [3:0] vx7_rx,  
   input wire [3:0] vx8_rx,
   input wire [3:0] vx9_rx,  
   input wire [3:0] vx10_rx,    
   input wire [3:0] vx11_rx,
   input wire [3:0] vx12_rx,
   
   //    VX_TXs
   output reg [2:0] vx1_tx_out,
   output reg [2:0] vx2_tx_out,
   output reg [2:0] vx3_tx_out,
   output reg [2:0] vx4_tx_out,
   output reg [2:0] vx5_tx_out,
   output reg [2:0] vx6_tx_out,
   output reg [2:0] vx7_tx_out,
   output reg [2:0] vx8_tx_out,
   output reg [2:0] vx9_tx_out,
   output reg [2:0] vx10_tx_out,
   output reg [2:0] vx11_tx_out,
   output reg [2:0] vx12_tx_out,

   // remove input wire gdm_trg,
   // remove input wire gdm_tsm,
   
   //    LEMO INPUTs
   input wire [4:1] ext_in_lv_async, // direct connection to LEMO connectors, not clocked
    
   //    LEMO OUTPUTs
   output reg [2:1] ext_out, // direct connection to LEMO connectors, not clocked
    
   //    FRONT PANEL LEDs
   output reg [3:0] fp_led_out, // direct connection to LEMO connectors, not clocked

   //    trigger and tsm output
   // remove output reg trg_out,
   // remove output reg tsm_out
   );

world view

Note:

  • red lines: clocks
  • green lines: AXI/Avalon packet streams
  • blue lines: serial data

GDM-CDM-VX.drawio.svg

description

same thing, in words: 

detector
digitizer, 125 MHz
digital filter
digital discriminator
hit map, 64 bits at 125 MHz (could be 250 MHz, filter and discriminator clock)
packetizer, 64 bits -> id, timestamp, 8x 8-bit words, eop
8/10 serializer, 12.5 MHz parallel in, 125 MHz serial out
lvds line to CDM vx_rx
BBB: also VX busy to lvds line to CDM vx_rx, do not want to depend on serial comm for vx busy, do not want to inject async data into the hit map packet stream
--- CDM
lvds line capture in IBUF register, vxN_rx clock, 125 MHz clock, from PLL with adjustable phase, scan phase to find sweet spot
vx rx 10/8 deserializer, vxN_rx clock, stobes out parallel data every 10 clocks at 12.5 MHz
vx rx phase transfer from vxN_rx clock to main CDM clock, have 10 clocks for transfer to happen
from here everything is on the CDM main clock
vx rx packet adapter, 12.5 MHz 8-bit packets to 125 MHz 16-bit packets, internal FIFO to avoid data overrun
sfp tx mux, all VX packet streams into one CDM sfp tx packet stream
BBB: capture VX RX busy to CDM main clock (IBUF register), grand-or becomes vx_rx_busy
sfp tx packetizer (data,eop,vx_rx_busy -> data,k)
sfp tx 16/20 serializer, 16 bit at 125 MHz in, 2500 MHz serial out (20 bits at 125 MHz)
sfp tx fiber link to GDM, 2 Gigabits/sec
--- GDM
qsfp rx deserializer, 16 bit at 125 MHz output on main GDM clock
qsfp rx depacketizer (data,k -> packet data,eop; qsfp_rx_busy)
qsfp rx demux, hit map packets routed to GDM trigger logic block
GDM trigger logic block looks at hit map, generates yes/no trigger decision, encodes it as a trigger packet
TSM generator encodes GPS time data as a TSM packet
qsfp tx mux - trigger packets, tsm packets, etc to qsfp tx packet stream
qsfp tx packetizer (data,eop;trg,tsm,bsy -> data,k)
NB: the same tx data is sent to all 12 qsfp tx ports, to make sure we do not accidentally desync the CDMs.
qsfp tx 16/20 serializer, 16-bit at 125 MHz to 20-bit at 250 MHz to 2500 MHz serial out
qsfp tx fiber link to CDM, 2 Gigabits/sec
--- CDM
sfp rx 20/16 deserializer, 16-bit at 125 MHz
sfp rx depacketizer, (data,k -> data,eop;trg_in,tsm_in,bsy_in)
sfp rx demux (in reality, noop, all packets go to same place, vx tx)
vx tx mux (packets from GDM, packets with simulated waveforms from CDM midas frontend via AXI FIFO)
vx tx packet adapter 16-bit at 125 MHz to 8-bit at 12.5 MHz strobed every 10 clocks. NB: most important, this 10-clock strobe runs in sync between all CDMs!
vx tx phase transfer from main CDM clock to vxN_tx clock, have 10 clocks for transfer to happen
from here we run on the vxN_tx clock
vx txN serializer, 8-bit at 12.5 MHz to 125 MHz serial
vx_txN OBUF register, vxN_rx clock, 125 MHz clock, from PLL with adjustable phase, scan phase to find sweet spot
lvds line to vx
BBB: bsy_in from GDM is converted from pulse to level, goes out lvds line to vx, sync to vxN_tx clock
TTT: trg_in from GDM is a pulse, does out lvds line to vx, sync to vxN_tx clock.
NB: tsm is always a packet, bsy is always a signal (no packet), trg can be a packet or signal.
--- VX
lvds data captured by 125 MHz ADC sampling clock (CAEN base firmware logic)
lvds data connected to Yair's block
10/8 deserializer
depacketizer (data,k -> data,eop) to avoid accidental desync, we do not send any no trg, no tsm, no bsy K-codes.
demux
trigger packets go their way (4x 64 bit words of data go to event header: timestamps, hitmap data)
tsm packets go their way (4x 64 bit words of data go to tsm event header: timestamps, GPS time data)
BBB: bsy from lvds line stops waveform acquisition
TTT: trg from lvds line cause waveform acquisition, same as trg packet, but has no timestamp and other data attached to it.

Firmware registers

Block 0

0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2

Block 1

Block 2

Block 3

DS20k block register map

  • busybox devmem 0x80013000 32

Note: R=readable, W=writable, L=latched by CMD_LATCH, B=reset on begin of run 

reg | version    | xDM | xx | description
  0 | 0x20230731 | ALL | RO | ds20k version
  0 | 0x20240118 | ALL | RW | ds20k version and command
  1 | 0x20230731 | ALL | RW | scratch read/write register
  2 | 0x20230731 | ALL | RW | configure inputs and outputs
  3 | 0x20230731 | ALL | RW | FP_LED mux
  4 | 0x20230731 | ALL | RW | EXT_OUT mux
  5 | 0x20230731 | ALL | RO | VX_RX state
  6 | 0x20230731 | ALL | RO | VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state
  7 | 0x20230731 | ALL | RW | LED_OUT, EXT_OUT, VX_TX outputs
  8 | 0x20230731 | ALL | RW | VX_TX mux and config
  9 | 0x20230731 | ALL | RW | trigger config
 10 | 0x20231013 | ALL | RO | status register
 11 | 0x20230731 | ALL | RO | trigger counter
 12 | 0x20230731 | ALL | RO | time slice marker counter
 13 | 0x20240814 | ALL | RO | GPS 1pps period, 125 MHz
 14 | 0x20240814 | ALL | RO | Ru clock 1pps period, 125 MHz
 15 | 0x20230811 | CDM | RO | SFP RX status
 16 | 0x20230811 | CDM | RW | SFP TX control
 17 | 0x20230811 | GDM | RO | QSFP RX data 0, 1
 18 |            |.    |.   | 2, 3
 19 |            |.    |.   | 4, 5
 20 |            |.    |.   | 6, 7
 21 |            |     |.   | 8, 9
 22 |            |.    |.   | 10, 11
 23 | 0x20230811 | GDM | RW | QSFP TX control
 24 | 0x20231013 | ALL | RW | trigger pulser period
 25 | 0x20231013 | ALL | RW | trigger pulser burst control
 26 | 0x20231013 | ALL | RW | tsm pulser period
 27 | 0x20231204 | ALL | RW | data fifo CPU to FPGA
 28 | 0x20231204 | ALL | RW | packet loopback control
 28 | 0x20240118 | ALL | RW | data fifo FPGA to CPU
 29 | 0x20231208 | GDM | RW | bitmap of active qsfp ports
 29 | 0x20240118 | ALL | RW | packet loopback control
 29 | 0x20240510 | ALL | RW | packet routing
 30 | 0x20231208 | GDM | RO | qsfp link status ports 0..7
 31 | 0x20231208 | GDM | RO | qsfp link status ports 8..11
 32 | 0x20240118 | GDM | RW | enabled QSFP ports
 33 | 0x20240118 | CDM | RW | enabled VX ports
 34,35 | 0x20240118 | ALL | RO | time stamp 64 bits
 36,37 | 0x20240118 | ALL | RO | old time stamp
 38,39,40 | 0x20240118 | CDM | RO | VX busy counters
 41,42,43 | 0x20240118 | GDM | RO | QSFP busy counters
 44 | 0x20240118 | CDM | RO | cdm_bsy_up_counter and cdm_bsy_pulse_counter
 45 | 0x20240118 | CDM | RO | cdm_veto_pulse_counter and cdm_veto_up_counter
 46 | 0x20240118 | GDM | RO | gdm_bsy_pulse_counter and gdm_bsy_up_counter
 47 | 0x20240118 | GDM | RO | gdm_bsy_refresh_counter
 48 | 0x20240118 | GDM | RO |  gdm_veto_up_counter and gdm_veto_pulse_counter
 49,50,51 | 0x20240424 | CDM | RO | VX RX serial data monitor, 8 bits per VX port
 52,53    | 0x20240424 | CDM | RO | VX serial link status, 4 bits per VX port
 54 | 0x20240424 | CDM | RO | VX TX serial data monitor
 55 | 0x20240430 | ALL | RO | QSFP, SFP and VX link loss counters
 56 | 0x20240510 | ALL | RO | sfp_rx_packet_counter
 57 | same       | ALL | RO | sfp_tx_packet_counter
 58 | same       | ALL | RO | qsfp_rx_packet_counter[0]
 59 | same       | ALL | RO | qsfp_tx_packet_counter
 60 | same       | ALL | RO | vx_rx_packet_counter[0]
 61 | same       | ALL | RO | vx_tx_packet_counter
 62 | 0x20240719 | CDM | RO | cdm_hitmap_period, ports 0, 1
 63 | same       | CDM | RO | ports 2, 3
 64 | same       | CDM | RO | ports 4, 5
 65 | same       | CDM | RO | ports 6, 7
 66 | same       | CDM | RO | ports 8, 9
 67 | same       | CDM | RO | ports 10, 11
 68 | 0x20240814 | ALL | RW | GPS control and status
 69 | 0x20241104 | CDM | RO | vx_tx_trg_packet_counter, counter of TRG packets CDM->VX
 70 | 0x20241104 | CDM | RO | vx_tx_tsm_packet_counter, counter of TSM packets CDM->VX
 71 | 0x20241104 | ALL | RO | packet error bits
 72,73 | 0x20241104 | CDM | ROL | vx_tx_trg_data_latched[63:0] trigger packet data
 74,75 | 0x20241104 | CDM | ROL | vx_tx_tsm_data_latched[63:0] tsm packet data (truncated to 64 bits)
 76 | 0x20241104 | CDM | ROL | cdm_hitmap_trigger_counter_latched, counter of hitmap triggers generated in the CDM

Register 0 0x80013000 ds20k version

on read: ds20k version 0xYYYYMMDD

on write:

  • 0 - noop - as of version 0x20240118, write a zero after writing a command
  • 1 - cmd_reset - reset logic to good state
  • 2 - cmd_arm_ts - arm timestamp reset
  • 3 - cmd_trg - issue a trigger
  • 4 - cmd_tsm - issue a tsm
  • 5 - cmd_vx_rx_reset - reset the VX receive path
  • 6 - cmd_vx_tx_reset - reset the VX transmit path
  • 7 - cmd_hitmap_trg - generate a hitmap trigger and data packet
  • 8 - cmd_trg_pulser_reset - reset the trigger pulser
  • 9 - cmd_tsm_pulser_reset - reset the tsm pulser
  • 10 - cmd_bor_start - start begin-of-run trigger sequence
  • 11 - cmd_bor_clear - after run has started, clear begin-of-run status bits
  • 12 - cmd_latch - latch counters & etc into AXI registers for coherent readout

Register 1 0x80013004 scratch

scratch read-write register

Register 2 0x80013008 input and output config

bit | version | fpga name       | description
  0 | ALL     | lemo_enable     | enable LEMO input 1
  1 |         |                 | 2
  2 |         |                 | 3
  3 |         |                 | 4
  4 | ALL     | lemo_invert     | invert LEMO input 1
  5 |         |                 | 2
  6 |         |                 | 3
  7 |         |                 | 4
  9 | ALL     | ext_out_disable | disable LEMO output 1
 10 |         |                 | 2
 11 | ALL     | ext_out_invert  | invert LEMO output 1
 12 |         |                 | 2

Register 3 0x8001300C FP_LED control

wire [15:0] led_out_mux_sel  = register_data_in[3][15:0];
wire [3:0]  led_out_invert   = register_data_in[3][19:16];

led_out_mux_sel is 4 groups (one per LED) of 4 bits (choice 0..15):
mux | version    | fpga name       | description
  0 | ALL        |                 | power on default
  1 | ALL        | led_out_reg     | register 7 bits
  2 | 0x20231013 | pll_locked      | clock chip PLL is locked
  3 | 0x20231013 | sfp_link_status | SFP link is good
  3 | 0x20240118 | sfp_link_status or qsfp_tx_link_rx_status | SFP/QSFP link is good
  4 | ALL        | lemo_in_sync[1] | LEMO input 1
  5 | ALL        | lemo_in_sync[2] | LEMO input 2
  6 | ALL        | lemo_in_sync[3] | LEMO input 3
  7 | ALL        | lemo_in_sync[4] | LEMO input 4
  8 | ALL        | ext_out[1]      | LEMO output 1
  9 | ALL        | ext_out[2]      | LEMO output 2
  A | 0x20231013 | trg_in          | trigger
  B | 0x20231013 | tsm_in          | time slice marker
  C | 0x20240118 | gdm_bsy         | GDM busy: OR of all CDM busy
  D | 0x20240118 | cdm_bsy         | CDM busy: OR of all VX busy
  E | 0x20240118 | cdm_veto        | GDM busy -> GDM veto -> CDM veto -> VX trigger veto
  F | ALL        |                 | fixed logic level 1

Register 4 0x80013010 LEMO OUT control

wire [7:0] ext_out_mux_sel = register_data_in[4][7:0];

ext_out_mux_sel is 2 groups (one per LEMO) of 4 bits (choice 0..15):

mux | version    | fpga name       | description
  0 | ALL        |                 | power on default
  1 | ALL        | ext_out_reg     | register 7
  2 | ALL        | trg_pulser      | pulser trigger
  3 | 0x20240724 | vx1_tx_out[2]   | vx1 serial data out
  4 | 0x20240724 | vx1_rx[1]       | vx1 serial data in
  5 | 0x20240724 | vx_rx_iob[0]    | vx1 serial data in captured by IOB register
  6 | ALL        | lemo_in_async[1]| test synchronizer
  7 | ALL        | lemo_in_sync[1] | test synchronizer
  8 | 0x20240724 | trg_in_pulse    | trigger signal
  9 | 0x20240724 | tsm_in_pulse    | time slice marker signal
  A | 0x20240118 | cdm_bsy         | CDM busy from VX
  B | 0x20240118 | gdm_bsy         | GDM busy from CDM
  C | 0x20240118 | cdm_veto        | veto from GDM to CDM to VX
  D | 0x20240118 | vx1_rx[1]       | serial data VX to CDM
  E | not used (sink)
  F | ALL        | 1               | fixed logic level 1

Register 5 0x80013014 VX_RX status

assign register_data_out[5] = {
                vx8_rx[3], vx8_rx[2], vx8_rx[1], vx8_rx[0],
                vx7_rx[3], vx7_rx[2], vx7_rx[1], vx7_rx[0],
                vx6_rx[3], vx6_rx[2], vx6_rx[1], vx6_rx[0],
                vx5_rx[3], vx5_rx[2], vx5_rx[1], vx5_rx[0],
                vx4_rx[3], vx4_rx[2], vx4_rx[1], vx4_rx[0],
                vx3_rx[3], vx3_rx[2], vx3_rx[1], vx3_rx[0],
                vx2_rx[3], vx2_rx[2], vx2_rx[1], vx2_rx[0],
                vx1_rx[3], vx1_rx[2], vx1_rx[1], vx1_rx[0]
                };

Register 6 0x80013018 VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state

assign register_data_out[6] = {
                vx2_tx_out[2], vx2_tx_out[1], vx2_tx_out[0], vx1_tx_out[2],
                vx1_tx_out[1], vx1_tx_out[0], ext_out[2], ext_out[1],
                fp_led_out[3], fp_led_out[2], fp_led_out[1], fp_led_out[0],
                ext_in_lv[4], ext_in_lv[3], ext_in_lv[2], ext_in_lv[1],
                vx12_rx[3], vx12_rx[2], vx12_rx[1], vx12_rx[0],
                vx11_rx[3], vx11_rx[2], vx11_rx[1], vx11_rx[0],
                vx10_rx[3], vx10_rx[2], vx10_rx[1], vx10_rx[0],
                vx9_rx[3], vx9_rx[2], vx9_rx[1], vx9_rx[0]
                };

bit | version | fpga name       | description
  0 | ALL     | vx9_rx          | VX_RX
  1 |         |                 | 
  2 |         |                 | 
  3 |         |                 | 
  4 | ALL     | vx10_rx         | VX_RX
  5 |         |                 | 
  6 |         |                 | 
  7 |         |                 | 
  8 | ALL     | vx11_rx         | VX_RX
  9 |         |                 | 
 10 |         |                 | 
 11 |         |                 | 
 12 | ALL     | vx12_rx         | VX_RX
 13 |         |                 | 
 14 |         |                 | 
 15 |         |                 | 
 16 | ALL     | ext_in_lv       | LEMO inputs
 17 |         |                 | 
 18 |         |                 | 
 19 |         |                 | 
 20 | ALL     | FP_LED          | FP_LEDs
 21 |         |                 | 
 22 |         |                 | 
 23 |         |                 | 
 24 | ALL     | ext_out[1]      | LEMO outputs
 25 |         | ext_out[2]      | 
 26 | ALL     | vx1_tx          | VX1_TX
 27 |         |                 | 
 28 |         |                 |
 29 | ALL     | vx2_tx          | VX2_TX 
 30 |         |                 | 
 31 |         |                 |

Register 7 0x8001301C LED_OUT, EXT_OUT, VX_TX outputs

   wire [3:0] led_out_reg = register_data_in[7][3:0];
   wire [2:1] ext_out_reg = register_data_in[7][5:4];
   // register_data_in[7][6];
   // register_data_in[7][7];
   wire [7:0] vx_tx_out_reg = register_data_in[7][15:8];

bit | version | fpga name       | description
  0 | ALL     | led_out_reg     | FP_LED 1
  1 |         |                 | 2
  2 |         |                 | 3
  3 |         |                 | 4
  4 | ALL     | ext_out_reg     | LEMO OUT 1
  5 |         |                 | 2
  6 | -       |                 | 
  7 | -       |                 | 
  8 | ALL     | vx_tx_out_reg   | VX1_TX 0
  9 |         |                 | 1
 10 |         |                 | 2
 11 |         |                 | -
 12 | ALL     |                 | VX2_TX 0
 13 |         |                 | 1
 14 |         |                 | 2
 15 |         |                 | -
 16 | -       |                 |
 17 |         |                 | 
 18 |         |                 | 
 19 |         |                 | 
 20 |         |                 |
 21 |         |                 | 
 22 |         |                 | 
 23 |         |                 | 
 24 |         |                 |
 25 |         |                 | 
 26 |         |                 |
 27 |         |                 | 
 28 |         |                 |
 29 |         |                 | 
 30 |         |                 | 
 31 |         |                 |

Register 8 0x80013020 VX_TX config

wire [3:0] vx_tx_mux_sel = register_data_in[8][3:0];
vx_tx_loopback           = register_data_in[8][31];

vx_tx_mux_sel is 4 bits (choice 0..15):
0 - power on default, control by vx_tx_out_reg
1 - GDM
2 - CDM
3 - pulser loopback test
4 - pulser loopback test
5 - 62.5 MHz output
6 - 125 MHz output
7 - trg, tsm, serial
8 - trg, tsm, lvds serial rx to serial tx loopback
9 - GPS box control (ds20k rev 0x20240814)
10
11
12
13
14
15 - production config: trg, veto, serial

Register 9 trg and tsm source

from version 0x20240724 

   wire [15:0] trg_src_mask      = register_data_in[9][15:0];
   wire [31:16] tsm_src_mask      = register_data_in[9][31:16];

   wire [15:0]      trg_src_bits =
                    {
                     1'b0,            // 15
                     1'b0,            // 14
                     vx_tx_tsm_done,  // 13
                     vx_tx_trg_done,  // 12

                     1'b0, // gdm_hitmap_trigger, // 11
                     cdm_hitmap_trigger, // 10
                     sfp_rx_tsm,      // 9
                     sfp_rx_trg,      // 8

                     sfp_rx_data[1],  // 7
                     sfp_rx_data[0],  // 6
                     tsm_pulser,      // 5
                     trg_pulser,      // 4

                     lemo_in_sync[4], // 3
                     lemo_in_sync[3], // 2
                     lemo_in_sync[2], // 1
                     lemo_in_sync[1]  // 0
                     };
   
   wire [15:0]      trg_bits = trg_src_bits & trg_src_mask;
   wire [15:0]      tsm_bits = trg_src_bits & tsm_src_mask;

before that: 

   wire [7:0] trg_src_mask      = register_data_in[9][7:0];
   wire [7:0] tsm_src_mask      = register_data_in[9][15:8];
   wire       trg_pulser_enable = register_data_in[9][16];
   wire       tsm_pulser_enable = register_data_in[9][17];
   wire       trg_software      = register_data_in[9][18];
   wire       tsm_software      = register_data_in[9][19];
   // bits 20:31 not used

wire [7:0]       xxx_src_bits =
                    {
                     sfp_rx_data[1],
                     sfp_rx_data[0],
                     tsm_pulser & tsm_pulser_enable,
                     trg_pulser & trg_pulser_enable,
                     lemo_in_sync[4],
                     lemo_in_sync[3],
                     lemo_in_sync[2],
                     lemo_in_sync[1]
                     };

trg_src_mask and tsm_src_mask bits:
0 - LEMO IN 1
1 - LEMO IN 2
2 - LEMO IN 3
3 - LEMO IN 4
4 - trg_pulser
5 - tsm_pulser
6 - sfp_rx_data[0] // to become sfp_trg_in, selected from sfp_rx_data[0], cdm_rx trg_out and gdm trigger packet
7 - sfp_rx_data[1] // to become sfp_tsm_in, selected from sfp_rx_data[1], cdm_rx tsm_out and gdm tsm packet

Register 10 0x80013028 status register

bit | version    | fpga name       | description
  0 | 0x20231013 | pll_locked      | clock chip PLL is locked
  1 | 0x20240118 | ts_reset_armed  | timestamp reset is armed
  2 | 0x20240118 | qsfp_tx_link_rx_status | QSFP link status is good for all enabled ports
  3 | 0x20240424 | vx_tx_link_rx_status   | VX link status is good for all enabled ports
  4 | 0x20240118 | cdm_bsy         | VX busy grand-or
  5 | 0x20240118 | gdm_bsy         | QSFP busy grand-or 
  6 | 0x20240118 | gdm_veto        | gdm_veto = gdm_busy 
  7 | 0x20240118 | cdm_veto        | CDM veto from GDM to VX 
  8 | 0x20240725 | bor_started     | begin-of-run sequence started
  9 | 0x20240725 | bor_finished    | begin-of-run sequence fininished, see commands 10 and 11 
 10 |         |                 | 
 11 |         |                 | 
 12 |         |                 | 
 13 |         |                 | 
 14 |         |                 | 
 15 |         |                 | 
 16 |         |                 | 
 17 |         |                 | 
 18 |         |                 | 
 19 |         |                 | 
 20 |         |                 | 
 21 |         |                 | 
 22 |         |                 | 
 23 |         |                 | 
 24 |         |                 | 
 25 |         |                 | 
 26 |         |                 | 
 27 |         |                 | 
 28 |         |                 |
 29 |         |                 | 
 30 |         |                 | 
 31 |         |                 |

Register 11 0x8001302C trg_counter

trigger counter

Register 12 0x80013030 tsm_counter

time slice marker counter

Register 13 0x80013034 GPS 1pps period

GPS 1pps period in 8 ns clocks

Register 14 0x80013038 Rb clock 1pps period

PRS-10 Rb clock 1pps output period in 8 ns clocks

Register 15 0x8001303C SFP RX status

bit | ds20k version | fpga signal name | description
  0 | ALL        | sfp_rx_data[15:0]   | cdm sfp received data
 16 | same       | sfp_rx_data_is_k[0] | 
 17 | same       | sfp_rx_data_is_k[1] | 
 18 |            | 0                   | 
 19 | 0x20231204 | sfp_rx_sel_lpb      | sfp tx->rx loopback
 20 | 0x20231013 | sfp_link_status     | sfp link connected, exchanging data
 21 | same       | sfp_link_rx_status  | sfp link receiving correct idle pattern from GDM TX
 22 | same       | sfp_link_error      | sfp link receiver error (badk or overflow)
 23 | same       | sfp_rx_data_error   | sfp transceiver state machine is in error state
 24 |            |                     | 
 25 |            |                     | 
 26 |            |                     | 
 27 |            |                     | 
 28 |            |                     |
 29 |            |                     | 
 30 |            |                     | 
 31 |            |                     |

Register 16 SFP TX control

   wire [15:0] sfp_tx_data_reg = register_data_in[16][15:0];
   wire [1:0]  sfp_tx_ctrl_reg = register_data_in[16][17:16];
   // 18
   // 19
   // 20:23
   wire qsfp_tx_enable_trg     = register_data_in[23][24]; // enable QSFP TX trg_in_pulse k-code
   wire qsfp_tx_enable_tsm     = register_data_in[23][25]; // enable QSFP TX tsm_in_pulse k-code
   // 26
   // 27
   //wire sfp_rx_sel_loopback    = register_data_in[16][28]; // TX->RX serial loopback
   wire sfp_tx_sel_loopback    = register_data_in[16][29]; // RX->TX serial loopback
   wire sfp_tx_sel_trg         = register_data_in[16][30]; // 16 individual bits
   wire sfp_tx_sel_reg         = register_data_in[16][31]; // from register

Register 17-22 QSFP RX data

QSFP RX data links 0..11

Register 23 QSFP TX control

   wire [15:0] qsfp_tx_data_reg = register_data_in[23][15:0];
   wire [1:0]  qsfp_tx_ctrl_reg = register_data_in[23][17:16];
   wire qsfp_rx_sel_lpb         = register_data_in[23][28]; // TX->RX loopback
   wire qsfp_tx_sel_lpb         = register_data_in[23][29]; // RX->TX loopback
   wire qsfp_tx_sel_trg         = register_data_in[23][30]; // 16 individual bits
   wire qsfp_tx_sel_reg         = register_data_in[23][31]; // data from register

Register 24 0x80013060 trigger pulser period

trigger pulser period in units of 8 ns (125 MHz clock)

Register 25 0x80013064 trigger burst pulser

   wire [7:0]         conf_burst_count  = conf_pulser_burst_ctrl[31:24];
   wire [23:0]        conf_burst_period = conf_pulser_burst_ctrl[23:0];

Register 26 0x80013068 tsm pulser period

time slice marker period in units of 8 ns (125 MHz clock)

Register 27 0x8001306C data write fifo

   wire        fifo_reset = register_data_in[27][31];
   wire        fifo_to_fpga_wr1 = register_data_in[27][27];
   wire        fifo_to_fpga_wr2 = register_data_in[27][26];
   wire [16:0] fifo_to_fpga_din = register_data_in[27][16:0];

   assign register_data_out[27][31:24] = register_data_in[27][31:24]; // echo write bits
   assign register_data_out[27][23] = fifo_to_fpga_full;
   assign register_data_out[27][22] = fifo_to_fpga_empty;

Register 28 0x80013070 data read fifo

   wire        fifo_reset = register_data_in[28][31];
   wire        fifo_from_fpga_rd1 = register_data_in[28][25];
   wire        fifo_from_fpga_rd2 = register_data_in[28][24];

   assign register_data_out[28][31:24] = register_data_in[28][31:24]; // echo write bits
   assign register_data_out[28][21] = fifo_from_fpga_full;
   assign register_data_out[28][20] = fifo_from_fpga_empty;
   assign register_data_out[28][16:0] = fifo_from_fpga_dout;

Register 29 0x80013074 packet routing

Control packet routing and loopbacks: 

   wire [3:0]  dn_route_ctrl             = register_data_in[29][3:0];
   wire [3:0]  up_route_ctrl             = register_data_in[29][7:4];
   wire [3:0]  fifo_to_fpga_route_ctrl   = register_data_in[29][11:8];
   //wire [3:0]  spare_route_ctrl        = register_data_in[29][15:12];

   wire        dn_mux_trg_enable         = register_data_in[29][16];
   wire        dn_mux_tsm_enable         = register_data_in[29][17];
   wire        dn_mux_sfp_rx_fifo_enable = register_data_in[29][18];
   // 19
   // 20..23

   wire        up_mux_vx_rx_enable       = register_data_in[29][24];
   // 25
   // 26
   // 27
   wire        fifo_from_fpga_hitmap_enable       = register_data_in[29][28];
   // 29..31

fifo_to_fpga output routing:

  • 0 - to down packet mux
  • 1 - to up packet mux
  • 2 - to fifo_from_fpga mux
  • 3 - not used

down packet mux inputs:

  • fifo_to_fpga_0
  • trg_pkt16 enabled by dn_mux_trg_enable
  • tsm_pkt16 enabled by dn_mux_tsm_enable
  • sfp_rx_fifo_pkt16 enabled by dn_mux_sfp_rx_fifo_enable
  • up_pkt16_2 loopback from up packet mux

down packet mux output routing:

  • 0 - to vx_tx_pkt16 to VX TX (vx_link_tx and vx_ser_tx)
  • 1 - to fifo_from_fpga mux
  • 2 - to up packet mux loopback
  • 3 - to qsfp_tx_pkt16 to GDM QSFP TX (cdm_link_tx)

up packet mux inputs:

  • vx_rx_pkt16 enabled by up_mux_vx_rx_enable data from 12 VX RX links
  • fifo_to_fpga_1
  • dn_pkt16_2 loopback from down packet mux

up packet mux output routing:

  • 0 - to sfp_tx_pkt16 to CDM SFP TX (VX data to GDM)
  • 1 - to fifo_from_fpga mux
  • 2 - to down packet mux loop loopback
  • 3 - not used

fifo_from_fpga mux inputs:

  • fifo_to_fpga_2
  • dn_pkt16_1 from down packet mux
  • up_pkt16_1 from up packet mux
  • qsfp_tx_pkt16 from GDM QSFP link 0 (there is no GDM QSFP 12-to-1 mux)
  • hitmap_pkt16 enabled by fifo_from_fpga_hitmap_enable in register 29

Register 30 0x80013078 qsfp link status ports 0..7

assign register_data_out[30] = 
     {
       qsfp_rx_data_error[7], qsfp_link_error[7], qsfp_link_status[7], qsfp_link_rx_status[7], // 7
       qsfp_rx_data_error[6], qsfp_link_error[6], qsfp_link_status[6], qsfp_link_rx_status[6], // 6
       qsfp_rx_data_error[5], qsfp_link_error[5], qsfp_link_status[5], qsfp_link_rx_status[5], // 5
       qsfp_rx_data_error[4], qsfp_link_error[4], qsfp_link_status[4], qsfp_link_rx_status[4], // 4
       qsfp_rx_data_error[3], qsfp_link_error[3], qsfp_link_status[3], qsfp_link_rx_status[3], // 3
       qsfp_rx_data_error[2], qsfp_link_error[2], qsfp_link_status[2], qsfp_link_rx_status[2], // 2
       qsfp_rx_data_error[1], qsfp_link_error[1], qsfp_link_status[1], qsfp_link_rx_status[1], // 1
       qsfp_rx_data_error[0], qsfp_link_error[0], qsfp_link_status[0], qsfp_link_rx_status[0]  // 0
      };

Register 31 0x8001307C qsfp link status ports 8..11

assign register_data_out[31] = 
     {
       4'b0000,
       4'b0000,
       4'b0000,
       4'b0000,
       qsfp_rx_data_error[11], qsfp_link_error[11], qsfp_link_status[11], qsfp_link_rx_status[11], // 11
       qsfp_rx_data_error[10], qsfp_link_error[10], qsfp_link_status[10], qsfp_link_rx_status[10], // 10
       qsfp_rx_data_error[9],  qsfp_link_error[9],  qsfp_link_status[9],  qsfp_link_rx_status[9],  // 9
       qsfp_rx_data_error[8],  qsfp_link_error[8],  qsfp_link_status[8],  qsfp_link_rx_status[8]   // 8
      };

Register 32 0x80013080 bitmap of enabled qsfp ports

   wire [11:0]  qsfp_mask       = register_data_in[32][11:0];
   wire         qsfp_bsy_force  = register_data_in[32][12];
   // not used                  = register_data_in[32][15:13];
   wire [15:0]  gdm_veto_extend = register_data_in[32][31:16];

On the GDM:

qsfp_rx_bsy[11..0] are pulses received from the CDMs

qsfp_rx_bsy_or is the grand-or of qsfp_rx_bsy masked by qsfp_mask

if qsfp_rx_bsy_or is high, gdm_bsy goes up and stays up for vx_bsy_extend*2 clocks

gdm_veto = gdm_bsy (bsy of any one VX causes trigger veto to all of them)

gdm_veto transition 0->1 causes gdm_veto_pulse. as long as gdm_veto is high, gdm_veto_pulse is generated every gdm_veto_extend clocks

gdm_veto_pulse is sent to all CDMs.

For this to work right, cdm_bsy_extend should not be bigger than gdm_veto_extend.

Register 33 0x80013084 bitmap of enabled VX ports

   wire [11:0]  vx_mask       = register_data_in[33][11:0];
   wire         vx_bsy_force  = register_data_in[33][12];
   // not used                = register_data_in[33][15:13];
   wire [15:0]  vx_bsy_extend = register_data_in[33][31:16];

On the CDM:

cdm_bsy is a grand-or of all vx_bsy masked by vx_mask (list of active VXes).

vx_bsy_extend controls how often state of cdm_bsy is sent to the GDM. when cdm_bsy goes 0->1, we send a cdm_bsy_pulse and keep resending it every vx_bsy_extend clocks as long as cdm_bsy stays high.

cdm_bsy_pulse is sent to the GDM.

sfp_rx_veto is the received from the GDM

if sfp_rx_veto goes up, cdm_veto goes up and stays up for gdm_veto_extend clocks.

for this to work right, CDM gdm_veto_extend must be bigger than GDM gdm_veto_extend.

cdm_veto goes to VXes on v1_tx_out[1] which is LVDS input 13.

Register 34, 35 0x80013088, 8C current timestamp

current 64-bit timestamp, 125 MHz

Register 36, 37 0x80013090, 94 old timestamp

old 64-bit timestamp, 125 MHz. timestamp saved at run start when it is reset to 0.

Register 38, 39, 40 0x80013098, 9C, A0 VX busy counters

8 bits per VX port, counters overflow to 255, reset at run start.

Register 41, 42, 43 0x800130A4, A8, AC QSFP busy counters

8 bits per QSFP port, counters overflow to 255, reset at run start.

Register 44, 45, 46, 47, 48 0x800130B0, B4, B8, BC, C0 CDM and GDM busy and veto counters

  • cdm_busy = grand-or of all VX busy for enabled VXes
  • gdm_busy = grand-or of all CDM busy for enabled CDMs
  • gdm_veto = gdm_busy
 44 | lo 16 bits | cdm_bsy_up_counter      | CDM busy, increments when cdm_busy goes 0->1
 44 | hi 16 bits | cdm_bsy_pulse_counter   | CDM busy to GDM, increments for each cdm_bsy_pulse sent to the GDM

 45 | lo 16 bits | cdm_veto_pulse_counter  | CDM veto from GDM, increments for each sfp_rx_veto received from the GDM
 45 | hi 16 bits | cdm_veto_up_counter     | CDM veto to VX, increments each time cdm_veto is set to 1.

 46 | lo 16 bits | gdm_bsy_pulse_counter   | GDM busy from CDM, increments for each qsfp_rx_busy received from CDMs (unless they overlap)
 46 | hi 16 bits | gdm_bsy_up_counter      | GDM busy, increments each time gdm_bsy goes 0->1

 47 | lo 16 bits | gdm_bsy_refresh_counter | GDM busy, increments each time gdm_bsy is extended by new qsfp_rx_busy
 47 | hi 16 bits | spare                   |

 48 | lo 16 bits | gdm_veto_up_counter     | GDM veto, increments each time gdm_veto goes 0->1
 48 | hi 16 bits | gdm_veto_pulse_counter  | GDM veto to CDM, increments for each gdm_veto_pulse sent to the CDM

Register 49, 50, 51 0x800130xx VX RX serial data monitor

VX RX data, 8-bit per VX channel. k-bit is omitted. 

   assign register_data_out[49][7:0]   = vx_rx_data[0]; // vx1
   assign register_data_out[49][15:8]  = vx_rx_data[1]; // vx2
   assign register_data_out[49][23:16] = vx_rx_data[2]; // vx3
   assign register_data_out[49][31:24] = vx_rx_data[3]; // vx4

   assign register_data_out[50][7:0]   = vx_rx_data[4]; // vx5
   assign register_data_out[50][15:8]  = vx_rx_data[5]; // vx6
   assign register_data_out[50][23:16] = vx_rx_data[6]; // vx7
   assign register_data_out[50][31:24] = vx_rx_data[7]; // vx8

   assign register_data_out[51][7:0]   = vx_rx_data[8]; // vx9
   assign register_data_out[51][15:8]  = vx_rx_data[9]; // vx10
   assign register_data_out[51][23:16] = vx_rx_data[10]; // vx11
   assign register_data_out[51][31:24] = vx_rx_data[11]; // vx12

Register 52, 53 VX link status

   assign register_data_out[52] = 
     {
       vx_rx_error[7], vx_link_error[7], vx_link_status[7], vx_link_rx_status[7], // 7
       vx_rx_error[6], vx_link_error[6], vx_link_status[6], vx_link_rx_status[6], // 6
       vx_rx_error[5], vx_link_error[5], vx_link_status[5], vx_link_rx_status[5], // 5
       vx_rx_error[4], vx_link_error[4], vx_link_status[4], vx_link_rx_status[4], // 4
       vx_rx_error[3], vx_link_error[3], vx_link_status[3], vx_link_rx_status[3], // 3
       vx_rx_error[2], vx_link_error[2], vx_link_status[2], vx_link_rx_status[2], // 2
       vx_rx_error[1], vx_link_error[1], vx_link_status[1], vx_link_rx_status[1], // 1
       vx_rx_error[0], vx_link_error[0], vx_link_status[0], vx_link_rx_status[0]  // 0
      };

   assign register_data_out[53] = 
     {
       //4'b0000,
       1'b0, ~vx_rx_deser_rdy[0], vx_rx_code_err[0], vx_rx_disp_err[0],
       //4'b0000,
       //4'b0000,
       //4'b0000,
       vx_rx_monitor[0],
       vx_rx_error[11], vx_link_error[11], vx_link_status[11], vx_link_rx_status[11], // 11
       vx_rx_error[10], vx_link_error[10], vx_link_status[10], vx_link_rx_status[10], // 10
       vx_rx_error[9],  vx_link_error[9],  vx_link_status[9],  vx_link_rx_status[9],  //  9
       vx_rx_error[8],  vx_link_error[8],  vx_link_status[8],  vx_link_rx_status[8]   //  8
      };

vx_rx_monitor (12-bit) is from deserializer_10b.sv: 

   assign monitor_out[9:0] = lastByte[9:0];
   assign monitor_out[10]  = comma;
   assign monitor_out[11]  = ready;

Register 54 VX TX serial data monitor

  • contents of vx_tx_monitor from vx_ser_tx.sv:
   assign monitor_out[8:0]   = data_to_encoder; // 8-bit + k
   assign monitor_out[9]     = valid;
   assign monitor_out[15:10] = 0;
   assign monitor_out[25:16] = encoded_data; // 10-bit
   assign monitor_out[26]    = encoded_valid;
   assign monitor_out[27]    = 0;
   assign monitor_out[31:28] = 0;

Register 55 QSFP, SFP, VX link loss counters

assign register_data_out[55] = {
  vx_rx_error_counter,
  qsfp_link_rx_status_drop_counter,
  sfp_link_rx_status_drop_counter,
  vx_link_rx_status_drop_counter
};

24..31 - CDM VX RX error counters, count any errors in the VX RX data path (bad serial data, fifo overflow, etc)
16..23 - GDM QSFP link loss counter, increments on qsfp_tx_link_rx_status 1->0 (reg10)
 8..15 - CDM SFP  link loss counter, increments on sfp_link_rx_status     1->0 (reg15)
 0...7 - CDM VX   link loss counter, increments on vx_tx_link_rx_status   1->0 (reg10)

Register 68 GPS control and status

assign register_data_out[68] =
     {
      8'b00000000,      // 23+8
      rb_1pps_counter,  // 16+8 bits
      gps_1pps_counter, // 8+8 bits
      1'b0,
      1'b0,
      rb_ser_in,   // 5
      gps_data_in, // 4
      gps_aux_out, // 3
      gps_aux_in,  // 2
      rb_1pps_in,  // 1
      gps_1pps_in  // 0
      };

Register 71 packet error bits

assign register_data_out[71] =
     {
      8'b00000000,  // 23+8 bits
      8'b00000000,  // 16+8 bits
      8'b00000000,  //  8+8 bits
      1'b0,
      1'b0,
      1'b0,
      1'b0,
      1'b0,
      cdm_hitmap_pkt16_error_latch,
      vx_tx_tsm_pkt8_error_latch,  // 1
      vx_tx_trg_pkt8_error_latch   // 0
      };

Firmware registers branch develop_ko

Register map

  0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
  1 | ALL | ALL | RW | read write scratch register
  2 | ALL | CDM | ?? | MGT not used
  3 | ALL | CDM | RO | MGT debug_data
  4 | ALL | CDM | RW | clk_config_vec
  5 | ALL | CDM | ?? | not used
  6 | ALL | CDM | RO | CDM_link_data_processing:o_error_time
  7 | ALL | CDM | RO | CDM_link_data_processing:o_error_count

register 0 0x80010000

GDM: 

0 - gdm_link_interface:i_mgt_rst
2 - gdm_link_interface:i_link_down_latched_rst
8 - GDM_link_data_processing:i_rst
10..9 - GDM_link_data_processing:i_data_mode

CDM: 

0 - cdm_link_interface:i_mgt_rst
2 - cdm_link_interface:i_link_down_latched_rst
8 - CDM_link_data_processing:i_rst
10..9 - CDM_link_data_processing:i_data_mode

register 1 0x80010004

GDM: 

nlinks-1..0 - gdm_link_interface:i_rx_slide_trigger

CDM: 

nlinks-1..0 - cdm_link_interface:i_rx_slide_trigger

register 2 0x80010008

GDM: 

nlinks-1..0 - gdm_link_interface:o_link_power_good
nlinks+15..16 - gdm_link_interface:o_link_status

CDM: 

nlinks-1..0 - cdm_link_interface:o_link_power_good
nlinks+15..16 - cdm_link_interface:o_link_status

register 3 0x8001000c

GDM: simple loopback register

CDM: 

31..0 - debug_data - cdm_link_interface:o_debug

o_debug: 

rx_link_rst & rx_error & rx_link_up & rx_receiving_data &
std_logic_vector(rx_state_count) & tx_state_count_on_rx_clk & i_rx_ctrl3(0) &
i_rx_ctrl1(1 downto 0) & i_rx_ctrl0(1 downto 0) &
rx_data_is_k28p1_k28p5 &
i_rx_data;

register 4 0x80010010

GDM write: 

0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS

GDM read: 

0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS
4 - clk_config_vec(4) - CLK_LOSXTn_LS
5 - clk_config_vec(5) - CLK_LOLn_LS
6 - clk_config_vec(6) - CLK_INTn_LS
7 - constant 1
31..8 - constant 0

register 5 0x80010014

not used

register 6 0x80010018

GDM: 

3..0 - GDM_link_data_processing:i_status_select

CDM: 

31..0 - CDM_link_data_processing:o_error_count

register 7 0x8001001c

GDM: 

31..0 - GDM_link_data_processing:o_status_vector

CDM: 

31..0 - CDM_link_data_processing:o_error_count

GDM, CDM, VX packet communications

  • timestamp math
1 clock is 8 ns is 125 MHz
8 bits of clocks is 256 clocks is 2048 ns is ~2 usec
16 bits of clocks is ~500 usec is 0.5 msec
24 bits of clocks is ~134 msec
32 bits of clocks is ~34 sec
40 bits of clocks is ~8.7 ksec is 2.4 hours
48 bits of clocks is ~625 hours is ~26 days
56 bits of clocks is ~6.6 kdays is ~18 kyears
62 bits of clocks is ~10 Mhours is 427 kdays is ~1.1 kyears
64 bits of clocks is ~4.4 kyears
  • 0x02 - TRG packet, 8 bytes, 80 adc clocks, 640 ns on lvds link
0 - 0x02
1 - trg_counter[7:0]
2 - ts64 low byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 high byte 3
6 - trg_in_latch[7:0]
7 - trg_in_latch[15:8]
  • 0x03 - HITMAP_TRG packet, 12 bytes, 120 adc clocks, 960 ns on lvds link
0 - 0x03
1 - trg_counter[7:0]
2 - ts64 low byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 high byte 3
6 - vx_bitmap[7:0]
7 - vx_bitmap[15:8]
8 - vx_bitmap[23:16]
9 - vx_bitmap[31:24]
10 - vx_bitmap[39:32]
11 - vx_bitmap[47:40]
  • 0x10 - TSM packet, 26 bytes, 260 adc clocks, 2080 ns on lvds link
0 - 0x10
1 - tsm_counter[7:0]
2 - gdm_ts64 low byte 0
3 - 1
4 - 2
5 - 3
6 - 4
7 - 5
8 - 6
9 - gdm_ts64 high byte 7
10 - gps_ts64 low byte 0
11 - 1
12 - 2
13 - 3
14 - 4
15 - 5
16 - 6
17 - gps_ts64 high byte 7
18 - gps_data64 low byte 0
19 - 1
20 - 2
21 - 3
22 - 4
23 - 5
24 - 6
25 - gps_data64 high byte 7
  • 0x81 - VX hitmap packet, 10 bytes, 100 adc clocks, 800 ns on lvds link, 48 ns on fiber link
0 - 0x81
1 - VX ID
2 - hitmap low byte, nits 7:0
3 - 15:8
4 - 23:16
5 - 31:24
6 - ...:32
7 - ...
8 - ...
9 - hitmap low byte, bits 63:...
  • 0x82 - CDM hitmap packet, 108 bytes, not sent on lvds link, not sent on fiber link
0 - 0x82
1 - cdm_hitmap_trigger_counter[7:0]
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - cdm_hitmap_or12 byte 0 (7:0)
11 - cdm_hitmap_or12 byte 1 (11:8) plus 4 bits: 12=0, 13=0, 14=0, 15=cdm_hitmap_grand_or
12 - cdm_hitmap_data, low byte, 12*64 bits = 768 bits = 48 words = 96 bytes
...
107 - cdm_hitmap_data, high byte

AXI bus timing

  • AXI 100 MHz clock, 10 ns, 32-bit data
  • AXI single-dword read: 36 clock repeat rate, 360 ns is 2.777 MHz, 4 bytes per transfer is 11.11 Mbytes/sec
  • AXI single-qword read: 13 clock repeat rate, 130 ns is 7.7 MHz, 4 bytes per transfer is 30 Mbytes/sec, not accounting for the gap
  • AXI memcpy read: 4 transfers at 13 clocks, gap, 4 transfers at 13 clocks, gap, etc. below 30 Mbytes/sec.

Ds-dm-axi-read.png Ds-dm-axi-read-64.png Ds-dm-axi-read-memcpy.png

  • AXI single-dword write: 20 clocks repeat rate, 200 ns is 5 MHz, 4 bytes per transfer is 20 Mbytes/sec
  • AXI single-qword write: 13+20 clocks repeat rate, 330 ns is 3 MHz, 16 bytes per burst is 48 Mbytes/sec

Ds-dm-axi-write.png Ds-dm-axi-write-64.png

AXI bus addresses

  • see AXI/AMBA addresses assigned inthe FPGA project: (s_axi/reg0 is the DS-DM AXI registers)
daq00:ds-dm-gcdm$ grep assign_bd_address scripts/GDM_CDM_XU8_bd.tcl
  assign_bd_address -offset 0x80010000 -range 0x00004000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs axi_register_interfa_0/s_axi/reg0] -force
  assign_bd_address -offset 0x000400000000 -range 0x40000000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs ddr4/C0_DDR4_MEMORY_MAP/C0_DDR4_ADDRESS_BLOCK] -force
  assign_bd_address -offset 0x80000000 -range 0x00010000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs system_management_wiz/S_AXI_LITE/Reg] -force
daq00:ds-dm-gcdm$
  • see AXI/AMBA addresses exported from FPGA project to Linux kernel: (uio for debug bridge should say "debug bridge")
root@gdm0:~# cat /sys/class/uio/uio*/name
axi-pmon
axi-pmon
axi-pmon
axi-pmon
root@gdm0:~# 
root@gdm0:~# ls -l /sys/class/uio/
total 0
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio0 -> ../../devices/platform/amba/ffa00000.perf-monitor/uio/uio0
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio1 -> ../../devices/platform/amba/fd0b0000.perf-monitor/uio/uio1
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio2 -> ../../devices/platform/amba/fd490000.perf-monitor/uio/uio2
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio3 -> ../../devices/platform/amba/ffa10000.perf-monitor/uio/uio3
root@gdm0:~# ls -l /sys/class/uio/../../devices/platform/amba/
total 0
-rw-r--r-- 1 root root 4096 Oct 18 19:37 driver_override
drwxr-xr-x 3 root root    0 Oct 18 01:36 fd070000.memory-controller
drwxr-xr-x 4 root root    0 Oct 18 01:36 fd0b0000.perf-monitor
drwxr-xr-x 4 root root    0 Oct 18 01:36 fd400000.zynqmp_phy
drwxr-xr-x 4 root root    0 Oct 18 01:36 fd490000.perf-monitor
drwxr-xr-x 4 root root    0 Oct 18 01:36 fd500000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 fd510000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 fd520000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 fd530000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 fd540000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 fd550000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 fd560000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 fd570000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 fd6e0000.cci
drwxr-xr-x 5 root root    0 Oct 18 01:36 ff000000.serial
drwxr-xr-x 4 root root    0 Oct 18 01:36 ff020000.i2c
drwxr-xr-x 6 root root    0 Oct 18 01:36 ff0a0000.gpio
drwxr-xr-x 4 root root    0 Oct 18 01:36 ff0b0000.ethernet
drwxr-xr-x 4 root root    0 Oct 18 01:36 ff0e0000.ethernet
drwxr-xr-x 4 root root    0 Oct 18 01:36 ff0f0000.spi
drwxr-xr-x 5 root root    0 Oct 18 01:36 ff160000.mmc
drwxr-xr-x 5 root root    0 Oct 18 01:36 ff170000.mmc
drwxr-xr-x 3 root root    0 Oct 18 01:36 ff960000.memory-controller
drwxr-xr-x 4 root root    0 Oct 18 01:36 ff9d0000.usb0
drwxr-xr-x 4 root root    0 Oct 18 01:36 ffa00000.perf-monitor
drwxr-xr-x 4 root root    0 Oct 18 01:36 ffa10000.perf-monitor
drwxr-xr-x 4 root root    0 Oct 18 01:36 ffa50000.ams
drwxr-xr-x 5 root root    0 Oct 18 01:36 ffa60000.rtc
drwxr-xr-x 4 root root    0 Oct 18 01:36 ffa80000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 ffa90000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 ffaa0000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 ffab0000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 ffac0000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 ffad0000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 ffae0000.dma
drwxr-xr-x 4 root root    0 Oct 18 01:36 ffaf0000.dma
-r--r--r-- 1 root root 4096 Oct 18 19:37 modalias
lrwxrwxrwx 1 root root    0 Oct 18 19:37 of_node -> ../../../firmware/devicetree/base/amba
drwxr-xr-x 2 root root    0 Oct 18 19:37 power
lrwxrwxrwx 1 root root    0 Oct 18 01:36 subsystem -> ../../../bus/platform
-rw-r--r-- 1 root root 4096 Oct 18 01:36 uevent
root@gdm0:~#

Build firmware

Build from git clone

THESE ARE K.O.'s NOTES FOR CREATING THE PETALINUX DIRECTORY.

THEY DO NOT WORK!

COPY PETALINUX FROM A WORKING PROJECT AND USE "make gdm" and "make cdm" AS DESCRIBED BELOW.

  • git clone git@edev-group.triumf.ca:fw/exp/darkside/gcdm.git
  • #Makefile change VIVADO_SETTINGS_SCRIPT := /opt/Xilinx/Vivado/2022.1/settings64.sh
  • #. /opt/Xilinx/Vivado/2022.1/settings64.sh
  • . /opt/Xilinx/Vivado/2020.2/settings64.sh
  • make clean
  • make all_from_scratch
  • . /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
  • make petalinux_create
  • make petalinux_rebuild_new_hw_des
  • bomb out: The TMPDIR: /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp can't be located on nfs.
  • mkdir /tmp/build_tmp
  • rm -rf /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp/
  • ln -s /tmp/build_tmp /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp
  • try again
  • grinds, loads a whole bunch of packages...
  • finishes with desire to copy things to /tftpboot
  • make sdcard_cp_to wants to copy files from PetaLinux_GDM_CDM/images/linux/ to SD card

Build firmware

NOTE: directory Petalinux_GDM_CDM should already exist! 

#. /opt/Xilinx/Vivado/2020.2/settings64.sh
. /opt/Xilinx/Vivado/2022.2/settings64.sh
. /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
make clean_gdm   # remove gdm build tree
make gdm         # build or rebuild GDM
make copy_gdm    # copy to gdm0
make clean_cdm   # remove cdm build tree
make cdm         # build or rebuild CDM
make copy_cdm    # copy to cdm0 and cdm1

copy to SD card: 

open a root shell
format 16 GB Sd card per above
cd .../ds-dm-gcdm
make copy

build times

CDM 12-june-2023 69aabc1c25130d970bc375aca684bd68849e6685
daq13 AMD-5700G 1688.61user 399.33system 23:28.84elapsed 148%CPU
dsdaqgw AMD-7700 1090.55user 247.34system 16:03.55elapsed 138%CPU
dsdaqgw AMD-7700 CDM incremental 196.68user 67.62system 7:35.42elapsed 58%CPU
dsdaqgw AMD-7700 CDM incremental 684.72user 94.17system 7:30.17elapsed 173%CPU
dsdaqgw AMD-7700 GDM incremental 849.84user 99.79system 9:04.56elapsed 174%CPU

prepare bootable sd card

format the sd card

this only needs to be done once

  • become root
  • cd ~olchansk/git/ds-dm-gcdm
  • use "lsblk" to identify the SD card (should show as 8/16/32 GB block device)/ /dev/sdd in this case
  • make sdcard_format SDCARD_DEVICE=/dev/sdd
  • disconnect sd card, reconnect the sd card (to detect new partition tables, etc)

copy CDM boot files

cd /home/dsdmdev/git/ds-dm-gcdm
make copy

copy boot files to the sd card

  • as root: identify partition labels, run "blkid", should say "BOOT", "rootfs" and "data"
  • mount
mkdir /media/olchansk/BOOT
mkdir /media/olchansk/rootfs
mkdir /media/olchansk/data
mount -L BOOT /media/olchansk/BOOT
mount -L rootfs /media/olchansk/rootfs
mount -L data /media/olchansk/data
cp PetaLinux_GDM_CDM/images/linux/BOOT.BIN /media/olchansk/BOOT/
cp PetaLinux_GDM_CDM/images/linux/boot.scr /media/olchansk/BOOT/
cp PetaLinux_GDM_CDM/images/linux/image.ub /media/olchansk/BOOT/
umount /media/olchansk/BOOT
umount /media/olchansk/rootfs
umount /media/olchansk/data
eject /dev/sdd

boot messages

Xilinx Zynq MP First Stage Boot Loader 
Release 2020.2   Sep 24 2022  -  13:29:15
NOTICE:  ATF running on XCZU4CG/silicon v4/RTL5.1 at 0xfffea000
NOTICE:  BL31: v2.2(release):xlnx_rebase_v2.2_2020.3
NOTICE:  BL31: Built : 18:02:46, Sep 28 2022


U-Boot 2020.01 (Sep 28 2022 - 18:03:39 +0000)

Model: DarkSide 20k DM
Board: Xilinx ZynqMP
DRAM:  2 GiB
usb dr_mode not found
PMUFW:  v1.1
EL Level:       EL2
Chip ID:        zu4
NAND:  0 MiB
MMC:   mmc@ff160000: 0, mmc@ff170000: 1
In:    serial@ff000000
Out:   serial@ff000000
Err:   serial@ff000000
Bootmode: SD_MODE1
Reset reason:   SOFT 
Net:   
ZYNQ GEM: ff0b0000, mdio bus ff0b0000, phyaddr -1, interface rgmii-id

Warning: ethernet@ff0b0000 (eth0) using random MAC address - d6:62:5f:13:00:44
eth0: ethernet@ff0b0000
ZYNQ GEM: ff0e0000, mdio bus ff0e0000, phyaddr -1, interface rgmii-id
Could not get PHY for eth1: addr -1

Hit any key to stop autoboot:  0 
ZynqMP> 
CTRL-A Z for help | 115200 8N1 | NOR | Minicom 2.7.1 | VT102 | Online 122:2 | ttyACM0

load FPGA from u-boot

ZynqMP> fpga info
Xilinx Device
Descriptor @ 0x000000007fddb2c0
Family:         ZynqMP PL
Interface type: csu_dma configuration interface (ZynqMP)
Device Size:    1 bytes
Cookie:         0x0 (0)
Device name:    zu4
Device Function Table @ 0x000000007fda5fe8
PCAP status     0xa0002fde
ZynqMP>
  • cp CDM_XU8_top.bit /tftpboot/fpga.bit
dhcp
tftpb 0x10000000 fpga.bit
fpga loadb 0 0x10000000 ${filesize}

ZynqMP> dhcp
BOOTP broadcast 1
DHCP client bound to address 192.168.0.100 (1 ms)
*** Warning: no boot file name; using 'C0A80064.img'
Using ethernet@ff0b0000 device
TFTP from server 192.168.0.1; our IP address is 192.168.0.100
Filename 'C0A80064.img'.
Load address: 0x8000000
Loading: *
TFTP error: 'file /tftpboot/C0A80064.img not found for 192.168.0.100' (1)
Not retrying...
ZynqMP> tftpb 0x10000000 fpga.bit
Using ethernet@ff0b0000 device
TFTP from server 192.168.0.1; our IP address is 192.168.0.100
Filename 'fpga.bit'.
Load address: 0x10000000
Loading: #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         ############
         6.2 MiB/s
done
Bytes transferred = 7797807 (76fc2f hex)
ZynqMP> fpga loadb 0 0x10000000 ${filesize}
  design filename = "CDM_XU8_top;UserID=0XFFFFFFFF;Version=2022.2"
  part number = "xczu4cg-fbvb900-1-e"
  date = "2024/08/14"
  time = "14:18:22"
  bytes in bitstream = 7797692
zynqmp_align_dma_buffer: Align buffer at 0000000010000073 to 000000000fffff80(swap 0)
ZynqMP>

load FPGA from Linux

this will reset the CPU

cp fpga.bit /lib/firmware/
echo fpga.bit > /sys/class/fpga_manager/fpga0/firmware
  • make .bin file:
bootgen -image CDM_XU8_top.bif -arch zynqmp -o ./fpga.bin -w
  • cat CDM_XU8_top.bif
dsdaqgw:ds-dm-gcdm$ cat CDM_XU8_top.bif
all:
{
        [destination_device = pl] ./Vivado_CDM_XU8/CDM_XU8.runs/impl_1/CDM_XU8_top.bit
}
dsdaqgw:ds-dm-gcdm$

this will reset the CPU

cp fpga.bin /lib/firmware/
echo fpga.bin > /sys/class/fpga_manager/fpga0/firmware

this will reset the CPU

root@dsdm:~# ./fpgautil -b fpga.bin -f Full

this will reset the CPU

DTSO file from here: https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841847/Solution+ZynqMP+PL+Programming
echo 0 > /sys/class/fpga_manager/fpga0/flags
mount -t configfs configfs /configfs
root@dsdm:~# dtc -O dtb -o fpga.dtbo -b 0 -@ fpga.dtso
root@dsdm:~# cp fpga.dtbo /lib/firmware/
root@dsdm:~# cp fpga.bit /lib/firmware/
root@dsdm:~# rmdir /configfs/device-tree/overlays/fpga
root@dsdm:~# mkdir /configfs/device-tree/overlays/fpga
root@dsdm:~# echo -n "fpga.dtbo" > /configfs/device-tree/overlays/fpga/path

fpgautil

git clone https://github.com/Xilinx/meta-xilinx.git
cd meta-xilinx/meta-xilinx-core/recipes-bsp/fpga-manager-script/files/
scp fpgautil.c root@dsdm:
ssh root@dsdm
make fpgautil
ls -l ./fpgautil

root@dsdm:~# ls -l ./fpgautil
-rwxr-xr-x 1 root root 72256 Aug 16 00:15 ./fpgautil
root@dsdm:~# ./fpgautil 

fpgautil: FPGA Utility for Loading/reading PL Configuration

Usage:	fpgautil -b <bin file path> -o <dtbo file path>

Options: -b <binfile>		(Bin file path)
         -o <dtbofile>		(DTBO file path)
         -f <flags>		Optional: <Bitstream type flags>
				   f := <Full | Partial > 
         -n <Fpga region info>  FPGA Regions represent FPGA's
                                and partial reconfiguration
                                regions of FPGA's in the
                                Device Tree
				Default: <full>
	  -s <secure flags>	Optional: <Secure flags>
				   s := <AuthDDR | AuthOCM | EnUsrKey | EnDevKey | AuthEnUsrKeyDDR | AuthEnUsrKeyOCM | AuthEnDevKeyDDR | AuthEnDevKeyOCM>
	  -k <AesKey>		Optional: <AES User Key>
	  -r <Readback> 	Optional: <file name>
				Default: By default Read back contents will be stored in readback.bin file
	  -t			Optional: <Readback Type>
				   0 - Configuration Register readback
				   1 - Configuration Data Frames readback
				Default: 0 (Configuration register readback)
	  -R 			Optional: Remove overlay from a live tree
 
Examples:
(Load Full bitstream using Overlay)
fpgautil -b top.bit.bin -o can.dtbo -f Full -n full 
(Load Partial bitstream using Overlay)
fpgautil -b rm0.bit.bin -o rm0.dtbo -f Partial -n PR0
(Load Full bitstream using sysfs interface)
fpgautil -b top.bit.bin -f Full
(Load Partial bitstream using sysfs interface)
fpgautil -b rm0.bit.bin -f Partial
(Load Authenticated bitstream through the sysfs interface)
fpgautil -b top.bit.bin -f Full -s AuthDDR 
(Load Parital Encrypted Userkey bitstream using Overlay)
fpgautil -b top.bit.bin -o pl.dtbo -f Partial -s EnUsrKey -k <32byte key value>
(Read PL Configuration Registers)
fpgautil -b top.bit.bin -r
(Remove Partial Overlay)
fpgautil -R -n PR0
(Remove Full Overlay)
fpgautil -R -n full
Note: fpgautil -R is responsible for only removing the dtbo file from the livetree. it will not remove the PL logic from the FPGA region.
 
root@dsdm:~#

fw_printenv

to access u-boot environment from Linux:

  • apt install -y libubootenv-tool
  • create /etc/fw_env.config
/media/BOOT/uboot.env 0 0x40000
/media/BOOT/uboot-redund.env 0 0x40000
  • if uboot.env files do not exist, run "saveenv" from u-boot command prompt
  • fw_printenv and fw_setenv should work

Boot from network

u-boot

ZynqMP> setenv bootcmd run bootcmd_dhcp
ZynqMP> saveenv
ZynqMP> reset

boot.scr

# boot.scr
# mkimage -C none -A arm -T script -d boot.scr boot.scr.uimg
echo Loading FPGA!
#tftpb 0x10000000 fpga.bit
tftpb 0x10000000 {ipaddr}.bit
fpga loadb 0 0x10000000 ${filesize}
echo Booting Linux!
run bootcmd_pxe
echo Done!

mkimage -C none -A arm -T script -d boot.scr boot.scr.uimg

tftpboot

cp /home/olchansk/git/ds-dm-gcdm/boot.scr.uimg /tftpboot
ln -s /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/images/linux /tftpboot/xilinx-dsdm
mkdir /tftpboot/pxelinux.cfg
cat > /tftpboot/pxelinux.cfg/default-arm-zynqmp <<EOF
LABEL Linux
   KERNEL xilinx-dsdm/Image
   FDT xilinx-dsdm/system.dtb
   #INITRD rootfs.cpio.gz.u-boot
EOF

boot sequence

  • xilinx magic load BOOT.BIN from SD card
  • load FPGA form BOOT.BIN
  • load and run u-boot from BOOT.BIN or from image.ub
  • u-boot load environment from ??? probably SD card uboot-redund.env, this includes our bootcmd
  • run bootcmd which run bootcmd_dhcp which does:
  • from /tftpboot:
  • load and run boot.scr.uimg which does:
  • load FPGA image xilinx-dsdm/${ipaddr}.bit
  • run bootcmd_pxe which does:
  • load pxelinux.cfg/default-arm-zynqmp which does:
  • load xilinx-dsdm/Image ### this is the linux kernel
  • load xilinx-dsdm/system.dtb ### this is the device tree
  • start linux kernel
  • linux kernel does dhcp
  • linux kernel does nfs mount /nfsroot/%s,vers=3,tcp ### %s is replaced by the hostname supplied by DHCP
  • userland starts and runs to console and ssh login.

Xilinx ILA

References:

Build xvcserver_cdm.exe: (it is built as a static executable, can be copied and run anywhere) 

ssh dsdaq@gdm0
cd /home/dsdaq/online/ds-dm-software
git pull ### get latest version
make xvcserver_cdm.exe
ssh root@gdm0
/home/dsdaq/online/ds-dm-software/xvcserver_cdm.exe
INFO: To connect to this xvcServer instance, use url: TCP:gdm0:2542

To activate and use the vivado logic analyzer:

  • data path: vivado -> hw_server -> xvcserver -> mmap axi bus -> debug bridge -> jtag -> ILA
  • define ILAs in the code
  • instantiate the xilinx debug bridge at AXI bus address 0x80020000 (FIXME!!! this collides with Ian's AXI addresses)
  • build and boot the new FPGA firmware. updating the linux kernel is not necessary.
  • login root@gdm0, run: /home/dsdaq/online/ds-dm-software/xvcserver_cdm.exe -v ### with "-v" for the first time to see that vivado does connect to it, without "-v", normally.
  • login dsdaqgw, run: hw_server -s tcp:localhost:3121 -e "set auto-open-servers xilinx-xvc:gdm0:2542" ### tells us to connect to port localhost:3121
  • login dsdaqgw, run vivado, open project, open hardware manager, open target, open new target, "connect to remote server", hostname "localhost", port "3121", next (bombs, try again, 3 times), popup add virtual cable, enter hostname "gdm0" port "2542", "ok", it shows in "hardware targets", "next", "finish", error popup "[Common 17-163] Missing value for option 'objects', please type 'set_property -help' for usage info", ignore it, in "hardware", right click the "gdm0" one, open target, under "hardware" and "debug bridge" we should see all the ILAs, under "hardware device properties", the "probes file" should have the ".ltx" file generated by vivado "Vivado_GDM_XU8/GDM_XU8.runs/impl_1/debug_nets.ltx", click on an ILA, a waveform should open.

Software

test_cdm.exe

CDM SFP status

# /home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0xbb2f0ae7
CDM firmware 0xbb2f0ae7
arg 1: [--sfp]
Polling SFP status...
identifier 0x03
connector  0x07
encoding   0x01
wavelength 0x0352 (850 nm)
vendor_name [FINISAR CORP.   ]
vendor_pn   [FTLF8526P3BNL   ]
vendor_rev  [A   ]
vendor_sn   [N3AB9M8         ]
vendor_date [200319  ]
dm_type    0x68
temp 29.0 C
vcc  3.323 V
tx_bias  7.250 mA
tx_power 478.4 uW
rx_power 2.3 uW
SFP good 1, status: temp 30.6 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 818 uW
...

GDM QSFP status

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --qsfp3 --qsfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x53aee418
CDM firmware 0x53aee418
arg 1: [--qsfp3]
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
 gpio-378 (                    |sysfs               ) out hi 
 gpio-379 (                    |sysfs               ) out hi 
 gpio-381 (                    |sysfs               ) out lo 
 gpio-382 (                    |sysfs               ) out hi 
arg 2: [--qsfp]
Polling QSFP status...
identifier 0x0d
status     0x02
los        0x8f
temp       28.2 C
vcc        3.323 V
rx_power     0.1   0.1   0.1   0.1 uW
tx_bias      7.6   7.6   7.6   0.0 mA
tx_power   792.2 773.8 823.0   0.1 uW
vendor_name [FINISAR CORP    ]
vendor_pn   [FTL410QD4C      ]
vendor_rev  [A ]
wavelength  850
max_temp    70 C
vendor_sn   [X79AC0R         ]
vendor_date [220309  ]
QSFP good 1, status: temp 27.7 C, los 0x8b, tx_bias 7.5 7.6 7.6 0.0 mA, tx_power 792 772 821   0 uW, rx_power   0   0 466   0 uW

GDM clock status

/home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
  • clock chip not loaded, not running:
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x0f 0x16 0x04 0x00 0x01 0x68 0x09 0x00 0x02 0xf2 0x00 0x1f 0xf0 0x22 0xf2, 0x507: 0x00, 0x52A: 0x00, 0x53F: 0x04
Clock chip state 0, status:  SYSINCAL XAXB_ERR LOL CAL_PLL IN0 IN_SEL_0 FASTLOCK_STATUS

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x1f7caf52 (528265042) should be ~125 MHz
0x1034 rx_clk:             0x00000000 (0) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x1f7caf52 (528265042) should be ~125 MHz
0x103C tx_clk:             0x00000000 (0) should be ~125 MHz
0x1040 clk_50MHz:          0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk:         0x05f5e100 (100000000) should be 100 MHz exactly
  • clock chip good (IN0 - external 10 MHz clock)
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x44 0x00 0xd0 0x01 0x1f 0xff 0x22 0xf2, 0x507: 0x3f, 0x52A: 0x01, 0x53F: 0x00
Clock chip state 1, status:  LOS_IN2 OOF_IN2 IN0 IN_SEL_REGCTRL IN_SEL_0

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735a3b (125000251) should be ~125 MHz
0x1034 rx_clk:             0x07735a3c (125000252) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a3b (125000251) should be ~125 MHz
0x103C tx_clk:             0x07735a3c (125000252) should be ~125 MHz
0x1040 clk_50MHz:          0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk:         0x05f5e100 (100000000) should be 100 MHz exactly

CDM clock status

/home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
  • clock chip not loaded, not running:
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x0f 0x16 0x04 0x00 0x01 0x68 0x19 0x00 0x02 0xf2 0x00 0x1f 0xf0 0x22 0xf2, 0x507: 0x00, 0x52A: 0x00, 0x53F: 0x04
Clock chip state 0, status:  SYSINCAL XAXB_ERR LOL CAL_PLL IN0 IN_SEL_0 FASTLOCK_STATUS

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735851 (124999761) should be ~125 MHz
0x1034 rx_clk:             0x00000000 (0) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x0127fefa (19398394) should be ~125 MHz
0x103C tx_clk:             0x00000000 (0) should be ~125 MHz
0x1040 clk_50MHz:          0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk:         0x05f5e100 (100000000) should be 100 MHz exactly
  • clock chip uses internal clock (IN1 - internal oscillator), observe rx_clk frequency is not same as others
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0x7f, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status:  IN1 IN_SEL_1 HOLD_HIST_VALID

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735853 (124999763) should be ~125 MHz
0x1034 rx_clk:             0x07735b0a (125000458) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735853 (124999763) should be ~125 MHz
0x103C tx_clk:             0x07735852 (124999762) should be ~125 MHz
0x1040 clk_50MHz:          0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk:         0x05f5e100 (100000000) should be 100 MHz exactly
  • clock chip uses SFP recovered clock (IN2 - sfp rx recovered clock), observe mgt_rx_ref_clk_raw (CDM 125 MHz oscillator) is different from others (SFP RX recovered clock)
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0xbf, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status:  IN2 IN_SEL_1 HOLD_HIST_VALID
^C
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773581b (124999707) should be ~125 MHz
0x1034 rx_clk:             0x07735ad6 (125000406) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735ad7 (125000407) should be ~125 MHz
0x103C tx_clk:             0x07735ad6 (125000406) should be ~125 MHz
0x1040 clk_50MHz:          0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk:         0x05f5e100 (100000000) should be 100 MHz exactly
^C

CDM link status, PRBS test mode

# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
  • fiber disconnected, no link:
CDM firmware:    0xbb2f0ae7
0x1000 SFP c.c. status: 0x00000031
    CLK_IN_SEL_LS   0x1
    CLK_EXT_SEL_LS  0
    CLK_CLK_RSTn_LS 0
    CLK_LOSXTn_LS   1
    CLK_LOLn_LS     1
    CLK_INTn_LS     0
0x1008 SFP link reset:  0x00000000
0x1010 SFP link status: 0x00000025
    sfp_mod_absent_N       1
    sfp_rx_los_N           0
    link_power_good        1
    rx_link_up             0
    rx_receiving_data      0
    rx_error               1
    rx_lnk_up_and_running  0
    tx_link_up             0
    tx_sending_data        0
    tx_link_up_and_running 0
    link_up_and_running    0
0x1014 SFP link data:   0x466a8187
    rx_data     0x8187
    k28p1_k28p5 0
    rx_ctrl0    0x1
    rx_ctrl1    0x1
    rx_ctrl3    0x1
    tx_state    0x1
    rx_state    0x6
    rx_receiving_data 0
    rx_link_up        0
    rx_error          1
    rx_link_rst       0
0x2000 link test mode: 0x00000200, seconds: 0x00079093, errors: 0xffffffff
  • fiber connected, good link:
CDM firmware:    0xbb2f0ae7
0x1000 SFP c.c. status: 0x000000b2
    CLK_IN_SEL_LS   0x2
    CLK_EXT_SEL_LS  0
    CLK_CLK_RSTn_LS 0
    CLK_LOSXTn_LS   1
    CLK_LOLn_LS     1
    CLK_INTn_LS     0
0x1008 SFP link reset:  0x00000000
0x1010 SFP link status: 0x000007dc
    sfp_mod_absent_N       0
    sfp_rx_los_N           0
    link_power_good        1
    rx_link_up             1
    rx_receiving_data      1
    rx_error               0
    rx_lnk_up_and_running  1
    tx_link_up             1
    tx_sending_data        1
    tx_link_up_and_running 1
    link_up_and_running    1
0x1014 SFP link data:   0x35c02774
    rx_data     0x2774
    k28p1_k28p5 0
    rx_ctrl0    0x0
    rx_ctrl1    0x0
    rx_ctrl3    0x0
    tx_state    0x3
    rx_state    0x5
    rx_receiving_data 1
    rx_link_up        1
    rx_error          0
    rx_link_rst       0
0x2000 link test mode: 0x00000200, seconds: 0x00078a8c, errors: 0x00000000

GDM link status, PRBS test mode

1 link connected, no errors:
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
GDM firmware:    0x6b2ee010
0x1014: 0x00000008, 18: 0x00000008, 1C: 0x00000000, 24: 0x00000fff
0x2000: 0x00000200, time: 0x00078aa4, errors:
0xffffffff 0xffffffff 0xffffffff
0x00000000 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff

CDM link status

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test0

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0xbf, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status:  IN2 IN_SEL_1 HOLD_HIST_VALID

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x077357a0 (124999584) should be ~125 MHz
0x1034 rx_clk:             0x07735a5c (125000284) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a5b (125000283) should be ~125 MHz
0x103C tx_clk:             0x07735a5b (125000283) should be ~125 MHz
0x1040 clk_50MHz:          0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk:         0x05f5e100 (100000000) should be 100 MHz exactly

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM firmware:    0x6d2ef81a
0x1000 SFP c.c. status: 0x000000b2
    CLK_IN_SEL_LS   0x2
    CLK_EXT_SEL_LS  0
    CLK_CLK_RSTn_LS 0
    CLK_LOSXTn_LS   1
    CLK_LOLn_LS     1
    CLK_INTn_LS     0
0x1008 SFP link reset:  0x00000000
0x1010 SFP link status: 0x000007dc
    sfp_mod_absent_N       0
    sfp_rx_los_N           0
    link_power_good        1
    rx_link_up             1
    rx_receiving_data      1
    rx_error               0
    rx_lnk_up_and_running  1
    tx_link_up             1
    tx_sending_data        1
    tx_link_up_and_running 1
    link_up_and_running    1
0x1014 SFP link data:   0x35c6bcbc
    rx_data     0xbcbc
    k28p1_k28p5 0
    rx_ctrl0    0x3
    rx_ctrl1    0x0
    rx_ctrl3    0x0
    tx_state    0x3
    rx_state    0x5
    rx_receiving_data 1
    rx_link_up        1
    rx_error          0
    rx_link_rst       0
0x2000 link test mode: 0x00000000, seconds: 0x00001671, errors: 0xffffffff

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 15
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
reg[15] is 0x0033bcbc (3390652)

GDM link status

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test0

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x44 0x00 0xd0 0x01 0x1f 0xff 0x22 0xf2, 0x507: 0x3f, 0x52A: 0x01, 0x53F: 0x02
Clock chip state 1, status:  LOS_IN2 OOF_IN2 IN0 IN_SEL_REGCTRL IN_SEL_0 HOLD_HIST_VALID

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735a0a (125000202) should be ~125 MHz
0x1034 rx_clk:             0x07735a0a (125000202) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a0a (125000202) should be ~125 MHz
0x103C tx_clk:             0x07735a0a (125000202) should be ~125 MHz
0x1040 clk_50MHz:          0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk:         0x05f5e100 (100000000) should be 100 MHz exactly

register 0x1018 bit 0x800

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM firmware:    0x6d2ebce6
0x1014: 0x00000800, 18: 0x00000800, 1C: 0x00000000, 24: 0x00000fff
0x2000: 0x00000000, time: 0x00003d2f, errors:
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff

link data alternates 0xbcbc and 0x1cbc

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 22
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
reg[22] is 0xbcbc93ab (-1128492117)

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 22
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
reg[22] is 0x1cbc1aaf (482089647)
root@gdm0:~#

Run trg and tsm

on the GDM:

ssh root@gdm00
./test_cdm.exe --gdm-clocks
./test_cdm.exe --load-cc
./test_cdm.exe --cc
./test_cdm.exe --reset-mgt
./test_cdm.exe --gdm-clocks
./test_cdm.exe --writereg 2 0xff # enable LEMO NIM inputs
./test_cdm.exe --writereg 3 0xba54 # LEDs: lemo1, lemo2, trg, tsm
./test_cdm.exe --writereg 4 0x99 $ # LEMO out is trg_in_pulse
./test_cdm.exe --writereg 9 0x32010 # trg and tsm from trg_pulser and tsm_pulse
./test_cdm.exe --writereg 24 1250000 # trg pulser 100 Hz
./test_cdm.exe --writereg 26 125000000 # tsm pulser 1 Hz
./test_cdm.exe --writereg 23 0x40000000 # route trg_in and tsm_in to qsfp tx bits 0 and 1
./test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz

on the CDM:

ssh root@cdm01
./test_cdm.exe --cdm-clocks
./test_cdm.exe --load-cc
./test_cdm.exe --reset-mgt
./test_cdm.exe --cdm-clocks
./test_cdm.exe --cdm-link # issue --reset-mgt on CDM and GDM until link is good
./test_cdm.exe --writereg 2 0xff # enable LEMO NIM inputs
./test_cdm.exe --writereg 3 0xba54 # enable LEDs: lemo1, lemo2, trg, tsm
./test_cdm.exe --writereg 4 0x99 # enable LEMO output trg_in
./test_cdm.exe --writereg 9 0x8040 # enable trg_in and tsm_in from sfp rx bits 0 and 1
./test_cdm.exe --writereg 7 0xff00 # drive VX LVDS lines to logic level 0
./test_cdm.exe --writereg 8 0x2 # VX LVDS with trg and tsm, misrouted in second VX
#./test_cdm.exe --writereg 8 0x7 # VX LVDS with tsm
./test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz

switch GDM and CDM to packetizer trg and tsm: 

on the GDM:

/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x00000000

on the CDM:

/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x0804
/home/dsdaq/online/ds-dm-software/test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz

Run packet loopback

GDM CPU -> fifo_to_fpga -> GDM QSFP -> CDM SFP -> fifo_from_fpga -> CDM CPU

On the GDM: (CDM is connected to first QSFP port) 

/home/dsdaq/online/ds-dm-software/test_cdm.exe --load-cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --reset-mgt
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 32 1 ### tell GDM to use first QSFP port
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link ### confirm link status is "3"
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 29 2 ### enable GDM packet data injection
/home/dsdaq/online/ds-dm-software/test_cdm.exe --test-fifo-write-loop

One the CDM: 

/home/dsdaq/online/ds-dm-software/test_cdm.exe --load-cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --reset-mgt
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 29 3 ### enable CDM packet data injection
/home/dsdaq/online/ds-dm-software/test_cdm.exe --test-fifo-read

dsvslice integration

VX setup

  • general
    • Start acq from user code = y, all others = n (as of Mar 2023: will start when first trigger received)
    • Use NIM IO = y
    • Use external clock = y
    • LVDS quartet is input = n, y, n, y
    • LVDS quartet mode = User, User, User, User
  • trigger from front panel NIM:
    • Trigger on external signal = y, all others = n
    • connect CDM EXT_OUT(2) to VX "TrigIn"
  • trigger from LVDS "Sync" mode
    • Trigger on LVDS Sync signal = y, all others = n
    • LVDS quartet mode = User, Sync, User, User
  • trigger from LVDS "User" mode
    • Trigger on LVDS pair 12 signal = y, all others = n
    • LVDS quartet is input = n, y, n, y
    • LVDS quartet mode = User, User, User, User

GDM setup

  • GDM is gdm0
  • set inputs to NIM mode
  • set outputs to TTL mode (this GDM has wrong NIM output circuit)
  • use top QSFP slot, connect split cable 1 into CDM01, cable 2 into CDM02
  • connect non-inverted NIM trigger signal to top-LEMO-left EXT_IN_LV(1)
  • connect non-inverted NIM TSM signal to top-LEME-right EXT_IN_LV(2)
  • GDM LEDs: TRIG, TSM, trigger enabled, trigger_out
  • GDM LEMO_OUT: trigger, trigger

CDM setup

  • set CDM LEMO inputs to NIM
  • set CDM LEMO outputs to NIM
  • CDM01 is cdm0
  • CDM02 is cdm1
  • connect GDM fiber links to SFP port
  • connect 1st VX port of CDM01 to VX1
  • connect 1st VX port of CDM02 to VX2
  • connect LEMO EXT_OUT(2) to VX "TrigIn", CDM01 to VX1, CDM02 to VX2
  • power up
  • CDM LEDs: GDM TRIG, GDM TSM, trigger enabled, trigger out
  • CDM LEMO_OUT: gdm_trg, trigger

After power up

  • start the CDM frontend from the MIDAS "Programs" page. To start manually, see the Start Command on the Programs page.
  • CDM frontend should enable the VX clock, disable the trigger
  • from the MIDAS status page, goto the CDM page
  • outdated: in the CDMx data tables, the 2nd number should read 0x35c08008, if it does not and the last 4 digits randomly change, reset the GDM links
  • if the FEs complain - do in order: for GDM, CDM01, CDM02, ..., press "reset mgt" of each board, then press "unreset mgt", if it does not help, STOP HERE
  • start a run
  • CDM frontend will enable the trigger
  • GDM frontend will enable the trigger
  • LEDs on the GDM should flash, LEDs on the CDM should flash, TrigIn and TrigOut of the VX should flash
  • stop a run
  • GDM frontend will disable the trigger
  • CDM frontend will disable the trigger

Phase measurement

daq00:PhaseMeasurement$ python3 ./phaseMeasurement.py --help
usage: phaseMeasurement.py [-h] fileName numberEvents numberVX sizeEvents stopEvent minHist maxHist numberBin writeToTXT saveAsPDF

Read data from midas file (in .lz4 format) to calculate phase between the clock of VX1 and VX2

positional arguments:
  fileName      Name of the file we want to read data from (Example: run00389.mid.lz4)
  numberEvents  Number of events recorded in the file
  numberVX      Number of VX used in this run (usually 2...)
  sizeEvents    Number of points per event
  stopEvent     Number of events you want to go through to calculate phase
  minHist       Minimal value for the x axis of the phase measurement histogram (in ns)
  maxHist       Maximal value for the x axis of the phase measurement histogram (in ns)
  numberBin     Number of bins wanted for the generated histogram
  writeToTXT    Write argument as yes to generate text file with results of calculation
  saveAsPDF     Save generated plots to PDF files

optional arguments:
  -h, --help    show this help message and exit
daq00:PhaseMeasurement$
  • try an old file with
  • python3 ./phaseMeasurement.py run00877.mid.lz4 10000 2 10000 500 -20 20 81 yes yes
bin size = 0.494 ns
num_events =  347
mean = -1.705 ns
rms = 3.087 ns
mean_error = 0.166 ns
centroid = -1.706 ns.
width (sigma) = 0.363 ns.
error on the centroid = 0.016558 ns.
  • ls -l *.txt *.pdf
dsdaq@dsvslice:~/online/PhaseMeasurement$ ls -l *.txt *.pdf
-rw-rw-r-- 1 dsdaq dsdaq 64728 Dec 14 16:56 run00877.mid.lz4_Plots.pdf
-rw-rw-r-- 1 dsdaq dsdaq   274 Dec 14 16:56 run00877.mid.lz4.txt
dsdaq@dsvslice:~/online/PhaseMeasurement$
  • scope settings (from email message)
From fcote-lortie@triumf.ca  Thu Dec 15 17:11:46 2022
From: Francis Cote-Lortie <fcote-lortie@triumf.ca>
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: Re: How to use scope as a waveform generator
Date: Fri, 16 Dec 2022 01:11:44 +0000

  1.  Turn the power on (bottom left of the scope)
  2.  Access the waveform generator display by pressing the Gen button (bottom right of the scope)
  3.  The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
  4.  Turn the output on (first option on the display) by pressing on the button. It will go from 0 to 1.

     The settings that we are using right now are:
      Type of waveform: Sine wave
      Offset: 0 V
      Amplitude: 1 Vpp
      Frequency: 50 kHz
      Noise: 0 V
________________________________
From: Francis Cote-Lortie <fcote-lortie@triumf.ca>
Sent: Thursday, December 15, 2022 4:58 PM
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: Re: How to use scope as a waveform generator


  1.  Turn the power on (bottom left of the scope)
  2.  Access the waveform generator display by pressing the Gen button (bottom right of the scope)
  3.  The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
  4.  Turn the output on (first option on the display)

     The settings that we are using right now are:
      Type of waveform: Sine wave
      Offset: 0 V
      Amplitude: 1 Vpp
      Frequency: 50 kHz
      Noise: 0 V
________________________________
From: Francis Cote-Lortie
Sent: Thursday, December 15, 2022 4:54 PM
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: How to use scope as a waveform generator


  1.  Turn the power on (bottom left of the scope)
  2.  Access the waveform generator display by pressing the Gen button (bottom right of the scope)
  3.  The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
  4.  Turn the output on (first option on the display)

Standalone link test

CDM: program clock chip
busybox devmem 0x80011000 32 0x8
busybox devmem 0x80011000 32 0x0
/home/dsdaq/si5394-i2c-file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt  0 0x6b

GDM, CDM: link reset
busybox devmem 0x80011008 32 1

GDM, CDM: release reset
busybox devmem 0x80011008 32 0

CDM: link status (NOTE: SFP LOS and mod_absent are swapped!!!)
busybox devmem 0x80011010 32
0x00000024 <- fiber plugged
0x00000025 <- fiber unplugged
0x00000027 <- SFP unplugged
0x000007DC <- successful link with GDM

CDM: link state machine and data
busybox devmem 0x80011014 32
0x35C06FF6

CDM: set link to counting mode
busybox devmem 0x80012000 32 0x101
busybox devmem 0x80012000 32 0x100

CDM: time counter and error counter
root@cdm1:~# busybox devmem 0x80012000 32
0x00000100 <--- link mode
root@cdm1:~# busybox devmem 0x80012004 32
0x0000058C <--- seconds counter
root@cdm1:~# busybox devmem 0x80012004 32
0x0000058D
root@cdm1:~# busybox devmem 0x80012008 32
0x00000000 <--- error counter

GDM: no link
root@gdm0:~# busybox devmem 0x80011014 32
0x00000000
root@gdm0:~# busybox devmem 0x80011018 32
0x00000000
root@gdm0:~# busybox devmem 0x8001101c 32
0x00000000
root@gdm0:~# busybox devmem 0x80011024 32
0x00000FFF
root@gdm0:~# 

GDM: good link channel 10, counting mode
root@gdm0:~# busybox devmem 0x80012000 32 0x101
root@gdm0:~# busybox devmem 0x80012000 32 0x100
root@gdm0:~# busybox devmem 0x80012008 32
0x3A8B68C2
root@gdm0:~# busybox devmem 0x80012008 32
0x42E03BEF
root@gdm0:~# busybox devmem 0x8001200c 32
0xDA090972
root@gdm0:~# busybox devmem 0x8001200c 32
0xDE6F22E9
root@gdm0:~# busybox devmem 0x80012019 32
Bus error
root@gdm0:~# busybox devmem 0x80012010 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012014 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012018 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x8001201c 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012020 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012024 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012028 32
0x00000000
root@gdm0:~# busybox devmem 0x8001202c 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012030 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012034 32
0xFFFFFFFF
root@gdm0:~#

mapping of link channels: 

qsfp0 lane0 - 0x0100 - link 8
qsfp0 lane1 - 0x0200 - link 9
qsfp0 lane2 - 0x0400 - link 10
qsfp0 lane3 - n/c
qsfp1 lane0 - 0x0800 - link 11
qsfp1 lane1 - 0x0010 - link 4
qsfp1 lane2 - 0x0020 - link 5
qsfp1 lane3 - n/c
qsfp2 lane0 - 0x0040 - link 6
qsfp2 lane1 - 0x0080 - link 7
qsfp2 lane2 - 0x0001 - link 0
qsfp2 lane3 - n/c
qsfp3 lane0 - 0x0002 - link 1
qsfp3 lane1 - 0x0004 - link 2
qsfp3 lane2 - 0x0008 - link 3
qsfp3 lane3 - n/c

script to start the test with 2 CDMs: 

ssh dsdaq@dsvslice
ssh root@gdm0 busybox devmem 0x80011008 32 1
ssh root@cdm0 busybox devmem 0x80011008 32 1
ssh root@cdm1 busybox devmem 0x80011008 32 1
ssh root@gdm0 busybox devmem 0x80011008 32 0
ssh root@cdm1 busybox devmem 0x80011008 32 0
ssh root@cdm0 busybox devmem 0x80011008 32 0
ssh root@gdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
ssh root@cdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
ssh root@cdm1 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2

LEMO trigger GDM to CDM to VX

on the GDM: 

/home/dsdaq/si5394-i2c-file /home/dsdaq/GDM_v1.0_IN0_EXT1_and_IN1_fixed_Si5394-RevA-Registers.txt 0 0x6b
busybox devmem 0x80011008 32 0x1
busybox devmem 0x80011008 32 0x0
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x80001230 # QSFP TX fixed pattern
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 2 0xff # enable LEMO inputs
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 3 0x7654 # enable LED, one per LEMO input
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x0F0F # enable LEMO to trg_in and tsm_in
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x40001230 # enable trg_in and tsm_in output to QSFP

on the CDM: 

/home/dsdaq/si5394-i2c-file /home/dsdaq/CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
/home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp
busybox devmem 0x80011008 32 1
busybox devmem 0x80011008 32 0
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x8040 # trg_in from sfp[0], tsm_in form sfp[1]
/home/dsdaq/online/ds-dm-software/test_cdm.exe 11 # trg_in counter
/home/dsdaq/online/ds-dm-software/test_cdm.exe 12 # tsm_in counter

GPS receiver VCL-2705

gnss-help
gnss-showver -> F/W: Ver 1.7 Feb 17 2020 16:20:43

gnss-showselftest -> no antenna connected
Overall   : FAIL
EPROM Test: PASS
Antenna   : NOT DETECTED
GNSS      : COMMUNICATION OK

gnss-showsettings
GNSS NMEA BAUDRATE        :115200
GNSS ANTENNA LENGTH       :30 meters
GNSS USER CONFIGURED DELAY:-65 nanoseconds
GNSS 1PPS PULSE WIDTH     :200 milliseconds
GNSS MODE                 :GPS
GNSS STATUS               :STATIONARY

GNSSAUX> gnss-showserial
SERIAL :2704H01V17MAX310

GNSSAUX> gnss-showmode
GNSS MODE : GPS

GNSSAUX> gnss-showstatus
GNSS STATUS: STATIONARY

GNSSAUX> gnss-showalarms
CURRENT ALARMS GNSS
ANTENNA   : NOT DETECTED

GNSSAUX> gnss-showerrors
CURRENT ALARMS GNSS
ANTENNA : **NOT DETECTED
ERROR STATISTICS GNSS
RMC GOOD DURATION  : Secs 0
RMC BAD DURATION   : Secs 0
LOCK GOOD SECS DURATION  : Secs 0
LOCK BAD SECS DURATION   : Secs 0
SATINFO GOOD ITERATIONS   :0
SATINFO REJECT ITERATIONS :0
SATINFO NOTALKER ITERATIONS   :0
CURRENT MONITOR STATE GNSS
STATE :Phase-1 HUNTING ANTENNA DETECT

GNSSAUX> gnss-showsatinfo
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-showsats
Total Sats: 0

GNSSAUX> gnss-showmyloc
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-show1ppsstate 
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-showjamstatus
Not Available !

GNSSAUX> gnss-showspoofstatus
Not Available !

--- antenna connected, can see the sky ---

GNSSAUX> gnss-showselftest
Overall   : PASS
EPROM Test: PASS
Antenna   : DETECTED
GNSS      : COMMUNICATION OK

GNSSAUX> gnss-showalarms
CURRENT ALARMS GNSS
ANTENNA   : DETECTED
GNSS LOCK : AVAILABLE

GNSSAUX> gnss-showerrors
CURRENT ALARMS GNSS
ANTENNA : DETECTED
GNSS LOCK : AVAILABLE
ERROR STATISTICS GNSS
RMC GOOD DURATION  : Mins 1,Secs 34
RMC BAD DURATION   : Secs 55
LOCK GOOD SECS DURATION  : Mins 1,Secs 34
LOCK BAD SECS DURATION   : Secs 55
SATINFO GOOD ITERATIONS   :3
SATINFO REJECT ITERATIONS :0
SATINFO NOTALKER ITERATIONS   :0
CURRENT MONITOR STATE GNSS
STATE :Phase-4 NORMAL OPERATION, Monitoring GNSSLOCK

GNSSAUX> gnss-showsatinfo
GNSS MODE :GPS
SATELLITE INFORMATION  TALKER:GPS
NMEA ID : 01-32
NO OF SATELLITES IN VIEW: 08
NO OF XXGSV MSGS        : 03
SatNo   PRN NO (SV ID)      ELEVATION (degs)    AZIMUTH (degs)      C/No (SNR)
1       0                   0                   0                   0                   
2       0                   0                   0                   0                   
3       0                   0                   0                   0                   
4       0                   0                   0                   0                   
5       0                   0                   0                   0                   
6       0                   0                   0                   0                   
7       0                   0                   0                   0                   
8       0                   0                   0                   0                   
<<<<<< End of Sat Info >>>>>>

GNSSAUX> gnss-showsats
Total Sats: 8

GNSSAUX> gnss-showmyloc
GNSS RECEIVER LOCATION:
Latitude : 4914.81911
Longitude: 12313.69595

GNSSAUX> gnss-show1ppsstate
GPS 1PPS STATE: LOCKED

GNSSAUX> gnss-showmyloc        

GNSS RECEIVER LOCATION:
Latitude : 4914.80688
Longitude: 12313.69531

enter into google maps search box as: 49 14.80688, -123 13.69531, observe the space, the moved dot and the minus.

GNSSAUX> gnss-resetgnss

Executing....Please Wait....
$$$$$END
  • 1PPS BNC output: period 1 sec, pulse width 200 ms, 3.3V into 1MOhm, 1.38V into 50Ohm.
  • IRIG-B BNC output: 5.6V into 1MOhm, 2.4V into 50Ohm.
  • IRIG-B format selector: default is all up.

Rb clock PRS10

  • https://www.thinksrs.com/products/prs10.html
  • 10 MHz output is sine wave around 5V peak to peak
  • 1PPS BNC output is 10 usec pulse, 5V into 1MHohm.
  • RS232 connection: minicom -D /dev/ttyUSB0 -b 9600
  • ser2net config: localhost,3001:raw:600:usb-5-2-1.0:9600 -XONXOFF -RTSCTS LOCAL
  • ssh daq13, cd /home/olchansk/git/ser2net, ./ser2net -c ~/daq/ds/ser2net.conf -d
  • ssh daq13, cd ~/daq/ds, python3 prs10.py
  • RS232 commands:
ID?
PRS10_3.56_SN_105719
VB1
SN?
RS1 -- reset
ST? -- status
FC? -- 10MHz OCXO drive voltage DAC settings
DS? -- "detected signals"
GA? -- gain of frequency lock loop between ovenized oscillator and Rb cell, 0=use ovenized oscillator only
MO? -- magnetic offset of the Rb cell, range 2300..3600, if out of range, unit must be set to different operating mode, see prs10m.pdf
MR? -- magnetic read
TT? -- time-tag, time in ns between 1PPS out and 1PPS in
TS? -- time slope, ???
TO? -- time offset, ???
PS? -- pulse slope, ???
PL? -- 0=phase lock off, 1=phase lock on, lock to 1PPS input
PT? -- phase lock integrator time constant, PT8 is integrator time constant 18.2 hours, natural time constant 2.25 hours
PF? -- phase lock stability factor, PF2 is "1"
PI? -- phase lock integrator

Analog to digital 12 bit ADC, values 0.000 to 4.998

AD0? -- Spare (J204)
AD1? -- +24V(heater supply) divided by 10.
AD2? -- +24V(electronics supply) divided by 10
AD3? -- Drain voltage to lamp FET divided by 10
AD4? -- Gate voltage to lamp FET divided by 10
AD5? -- Crystal heater control voltage
AD6? -- Resonance cell heater control voltage
AD7? -- Discharge lamp heater control voltage
AD8? -- Amplified ac photosignal
AD9? -- Photocell’s I/V converter voltage divided by 4
AD10? -- Case temperature (10 mV/°C)
AD11? -- Crystal thermistors
AD12? -- Cell thermistors
AD13? -- Lamp thermistors
AD14? -- Frequency calibration pot / external calibration voltage
AD15? -- Analog ground

A/D via CPU E-port:

AD16? -- Varactor voltage for 22.48 MHz VCXO (inside RF synthesizer) / 4
AD17? -- Varactor voltage for 360 MHz VCO (output of RF synthesizer) / 4
AD18? -- Gain control voltage for amplifier which drives frequency multiplier / 4
AD19? -- RF synthesizer’s lock indicator voltage (nominally 4.8 V when locked )

ST?

ST1 : Power supplies and Discharge Lamp
ST1 bit, Condition which sets bit, Corrective Action
0 -- +24 for electronic < +22 Vdc
1 -- +24 for electronics > +30 Vdc
2 -- +24 for heaters <+22 Vdc
3 -- +24 for heaters > +30 Vdc
4 -- Lamp light level too low
5 -- Lamp light level too high
6 -- Gate voltage too low
7 -- Gate voltage too high

ST2: RF Synthesizer
ST2 bit, Condition which sets bit, Corrective Action
0 -- RF synthesizer PLL unlocked
1 -- RF crystal varactor too low
2 -- RF crystal varactor too high
3 -- RF VCO control too low
4 -- RF VCO control too high
5 -- RF AGC control too low
6 -- RF AGC control too high
7 -- Bad PLL parameter

ST3: Temperature Controllers
ST3 bit, Condition which sets bit
0 -- Lamp temp below set point
1 -- Lamp temp above set point
2 -- Crystal temp below set point
3 -- Crystal temp above set point
4 -- Cell temp below set point
5 -- Cell temp above set point
6 -- Case temperature too low
7 -- Case temperature too high

ST4: Frequency Lock-Loop Control
ST4 bit, Condition which sets bit
0 -- Frequency lock control is off
1 -- Frequency lock is disabled
2 -- 10 MHz EFC is too high
3 -- 10 MHz EFC is too low
4 -- Analog cal voltage > 4.9 V
5 -- Analog cal voltage < 0.1
6 -- not used
7 -- not used

ST5: Frequency Lock to External 1pps
ST5 bit, Condition which sets bit
0 -- PLL disabled
1 -- < 256 good 1pps inputs
2 -- PLL active
3 -- > 256 bad 1pps inputs
4 -- Excessive time interval
5 -- PLL restarted
6 -- f control saturated
7 -- No 1pps input

ST6: System Level Events
ST6 bit and Condition which sets bit
0 Lamp restart
1 Watchdog time-out and reset
2 Bad interrupt vector
3 EEPROM write failure
4 EEPROM data corruption
5 Bad command syntax
6 Bad command parameter
7 Unit has been reset
  • ST? on warm start
received:  PRS_10
received:  255,255,255,243,34,255
received:  0,0,0,1,34,0
...
received:  0,0,0,1,34,0
received:  0,0,0,0,2,0
...
received:  0,0,0,0,2,0
received:  0,0,0,0,4,0
  • ST? on loss of external 1PPS
...
received:  0,0,0,0,4,0
disconnect 1PPS input
received:  0,0,0,0,132,0
...
reconnect 1PPS input
received:  0,0,0,0,132,0
received:  0,0,0,0,4,0
...
  • ST? on coldish start
daq13:ds$ python3 prs10.py
Connected
received [  ] old [ b'' ] counter:  0
received [ PRS_10 ] old [  ] counter:  0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter:  0
received [ 80,0,0,1,34,1 ] old [ 255,255,255,243,34,255 ] counter:  0
received [ 0,0,1,1,34,0 ] old [ 80,0,0,1,34,1 ] counter:  0
received [ 0,0,0,1,34,0 ] old [ 0,0,1,1,34,0 ] counter:  0
received [ 0,0,16,1,34,0 ] old [ 0,0,0,1,34,0 ] counter:  6
received [ 0,0,20,1,34,0 ] old [ 0,0,16,1,34,0 ] counter:  5
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter:  8
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter:  3
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter:  49
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter:  250
...
  • ST? on cold start, note: no bump in the 10MHz clock as reported by DS-DM clock chip
Connected
received [  ] old [ b'' ] counter:  0
received [ PRS_10 ] old [  ] counter:  0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter:  0
received [ 80,0,21,1,34,1 ] old [ 255,255,255,243,34,255 ] counter:  0
received [ 64,0,21,1,34,0 ] old [ 80,0,21,1,34,1 ] counter:  0
received [ 0,0,21,1,34,0 ] old [ 64,0,21,1,34,0 ] counter:  0
received [ 0,0,20,1,34,0 ] old [ 0,0,21,1,34,0 ] counter:  158
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter:  23
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter:  47
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter:  40
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter:  249

det fac integration test

Connections:

  • GPS receiver "IRIG-B SEL" both switches "up" - both "on", IRIG-B format B004
  • GPS receiver USB-B -> long cable -> daq13 USB-A
  • GPS receiver "1PPS out" -> long BNC cable -> BNC-T -> scope (5V, no 50ohm) and Rb clock BNC "1PPS in"
  • GPS receiver "IRIG-B 50ohms" -> long BNC cable -> BNC-T -> scope (5V, no 50ohm) and DS-DM LEMO input 1 (TTL mode)
  • Rb clock RS232 -> RS232 straight cable -> RS232-to-USB adapter -> daq13 USB-A
  • Rb clock "1PPS out" BNC -> scope (5V, no 50ohm, trig threshold rising edge 2V)
  • Rb clock "10MHz output 50 Ohm" BNC -> lemo -> lemo-T -> scope (sine wave, 5V, no 50ohm) and DS-DM clock input.

Programs to run:

  • on daq13: cd /home/olchansk/git/ser2net, ./ser2net -c ~/daq/ds/ser2net.conf -d
localhost,3001:raw:600:usb-5-2-1.0:9600  -XONXOFF -RTSCTS LOCAL
  • on daq13: cd /home/olchansk/daq/ds, python3 prs10.py ### connects to ser2net
daq13:ds$ python3 prs10.py
Connected
received [  ] old [ b'' ] counter:  0
received [ PRS_10 ] old [  ] counter:  0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter:  0
received [ 80,0,21,1,34,1 ] old [ 255,255,255,243,34,255 ] counter:  0
received [ 64,0,21,1,34,0 ] old [ 80,0,21,1,34,1 ] counter:  0
received [ 0,0,21,1,34,0 ] old [ 64,0,21,1,34,0 ] counter:  0
received [ 0,0,20,1,34,0 ] old [ 0,0,21,1,34,0 ] counter:  158
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter:  23
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter:  47
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter:  40
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter:  249
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  12096
received [ 0,0,0,0,20,0 ] old [ 0,0,0,0,132,0 ] counter:  2
received [ 0,0,0,0,148,0 ] old [ 0,0,0,0,20,0 ] counter:  0
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,148,0 ] counter:  575
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  11449
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  2755
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  34386
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  41035
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  113401
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  33375
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  54767
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  85059
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  33222
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  119234
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  121990
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  128184
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  56002
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  428237
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  8250
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  1388
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  30506
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  142704
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  179451
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  106182
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  68747
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  65424
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  157587
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  6932
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  1388
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  20255
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  4
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  225941
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  72183
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter:  26970
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter:  1
39287
  • on ds-dm: ./test_cdm.exe --irigb ### note sbs mismatch is because I should wrap around at 16 bits
dataframe: S00000000S100001010S...S001010101S000001000S, sec: 00, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16724 should be 82260
dataframe: S10000000S100001010S...S101010101S000001000S, sec: 01, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16725 should be 82261
dataframe: S01000000S100001010S...S011010101S000001000S, sec: 02, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16726 should be 82262
dataframe: S11000000S100001010S...S111010101S000001000S, sec: 03, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16727 should be 82263
dataframe: S00100000S100001010S...S000110101S000001000S, sec: 04, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16728 should be 82264
dataframe: S10100000S100001010S...S100110101S000001000S, sec: 05, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16729 should be 82265

DS-IOGC GPS interface board

Changes Rev0 to Rev1

From: Peter Margetak <pmargetak@triumf.ca>
Subject: IOGC REV1 review
Date: Wed, 4 Sep 2024 07:31:19 +0000

Hi Konstantin,
Pls have a look at SCH for new rev. I'd like to send it to mfr next week so if you can comment by early next week. Meanwhile I work on layout and other stuff.

Changes:

New ICs - all powered +5V
U20 - inverters for  RUclk RX/TX
U21 - non inverting line driver for RU-1pps-out (so you don't have to route if via GDM to see it on scope)
U22 - non inverting buffer for ext 1pps input

All Lemo connectors have the same position but they are double lemos now => new panel needed
@Marek Walczak<mailto:mwalczak@triumf.ca> you can print it ahead once pcb is done + update IOGC docs and panel description

J2A/B - Test ports for RU-1pps in and out
J5A/B - inputs for external GPS data and External source of 1pps
J6A/B  - aux in/out for GDM

SW1 - no change - select RX/TX  USB/GDM
SW2 - select latch sensitivity for rising/falling edge
SW3 - select source of GPS data (opto or ext)  AND select source of 1pps input (latch or ext)

p.

PRS-10 Rb clock device

The Rb clock PRS-10 device provides these connections: 

RS232 RX input - serial communication, non-standard RS232
RS232 TX output - serial communication, non-standard RS232
10 MHz clock output - coax 50 Ohm high resolution 10 MHz clock
1pps output - 1 Hz clock corresponding to the 10 MHz clock
1pps input - 1pps signal from GPS receiver

Mode of operation:

  • 10 MHz clock is always running
  • 1pps output is always running
  • if 1pps input from GPS received is present, after 256 pulses PRS-10 will sync it's 1pps output with the 1pps input by adjusting the frequency of the 10 MHz clock
  • when unlocked: 1pps output and 1pps input unrelated
  • when locked to GPS: 1pps output and 1pps input always go up and down at the same time

Theory of operation:

  • 10 MHz clock is produced by a high-quality crystal (stable on the scale of seconds)
  • crystal oscillator is synchronized to a Rb cell (stable on the scale of hours and days)
  • Rb cell resonant frequency is synchronized to the GPS 1pps signal (stable on the scale of months and years)

Rev1 connections

  • LEMO connectors (front panel)
LEMO J2A output - Rb clock 1pps in monitor
LEMO J2B output - Rb clock 1pps out monitor
LEMO J5A input - GPS IRIG-B from GPS receiver to FPGA (VCL-2705)
LEMO J5B input - GPS 1pps from GPS receiver to PRS-10 (VCL-2705)
LEMO J6A input - AUX-IN to FPGA
LEMO J6B output - AUX-OUT from FPGA
  • SMB connectors (back)
SMB J3 output - GPS 1pps loopback to LNGS
SMB J4 input - LNGS GPS data input
  • LEDs
D1 - same as LEMO J2A out (Rb clock 1pps in)
D2 - same as SMB J3 out (GPS 1pps from LNGS or from a GPS receiver)
D5 - controlled by FPGA-OUT-LED1
D6 - controlled by FPGA-OUT-LED2
D7 - PRS-10 24V power ok
  • switches
SW1A and SW1B - route PRS-10 RS232 to USB or to FPGA
SW2A - route PRS-10 1pps input from SMB J4 (LNGS) or from LEMO J5B (GPS receiver 1pps)
SW2B - route FPGA-IN-GPSDATA input from SMB J4 (LNGS) or from LEMO J5A (GPS receiver IRIG-B data)
SW3 - LNGS 1pps from rising edge or from falling edge of SMB J4 (LNGS)

Rb clock cable

Rb clock ----- DS-IOGC side, pin numbers are as labeled on the cable connectors

1 - 1pps out - 8 - 1pps out                    --- correct
2 - nc
3 - nc
4 - TXD      - 3  - RU-DATA-OUT - USB-RX input --- correct
5 - 1pps in  - 2  - 1pps in                    --- correct
6 - +24V     - 10 - +24V                       --- should by pin 1 to use both +24V pins?
7 - RXD      - 15 - RU-DATA-IN - USB-TX output --- correct
8 - nc
9 - +24V     - 10 - +24V --- correct
10 - GND     - 9  - GND  --- correct

VX connections

VXA_TX0 - FPGA-OUT-LED2 - D6 LED ("10 MHz clock")
VXA_TX1 - FPGA-OUT-LED1 - D5 LED ("GPS DATA")
VXA_TX2 - FPGA-OUT-RU1PPS-EN - enable 1pps to Rb clock
VXA_TX3 - not used (62.5 Hz clock)

VXB_TX0 - FPGA-TX - PRS-10 RS-232 out
VXB_TX1 - FPGA-OUT-AUX - 5V TTL J5 LEMO out
VXB_TX2 - FPGA-OUT-OPTO1PPS-EN - PRS-10 power enable
VXB_TX3 - not used (62.5 MHz clock)

VXA_RX0 - FPGA-IN-OPTO1PPS - 1pps from GPS
VXA_RX1 - FPGA-IN-GPSDATA - J4 GPS data
VXA_RX2 - n/c
VXA_RX3 - n/c

VXB_RX0 - n/c
VXB_RX1 - FPGA-IN-AUX - J5 LEMO TTL input
VXB_RX2 - FPGA-RX - PRS-10 RS-232 in
VXB_RX3 - FPGA-IN_RU1PPS - PRS-10 1pps output

test sequence

  • ./test_cdm_local.exe --writereg 7 0 ### clear reg 7
  • ./test_cdm_local.exe --writereg 8 0 ### set vx_tx mux to vx_tx control from reg 7
  • connect blue cable to GDM port 6 (next to the ethernet connector)
  • ./test_cdm_local.exe --writereg 7 0x4000 ### power up
  • ./test_cdm_local.exe --writereg 7 0x4100 ### right LED D6
  • ./test_cdm_local.exe --writereg 7 0x4200 ### left LED D5
  • ./test_cdm_local.exe --writereg 7 0x6000 ### J5 LEMO measure +5VDC
  • install LEMO jumper between LEMO J5 and J6
  • write 0x4000 then ./test_cdm_local.exe 6 ### reads 0x5C, bit 5 0x20 reads 0
  • write 0x6000 then ./test_cdm_local.exe 6 ### reads 0x7C, bit 5 0x20 reads 1
  • write 0x4000 to clear all bits
  • 10 MHz output ok - IN0 is good: ./test_cdm_local.exe --cc
Clock chip state 1, status:  OOF_IN2 IN1 IN_SEL_1 HOLD_HIST_VALID
  • without 10 MHz reports IN0 LOS (no signal) and OOF (wrong frequency)
Clock chip state 1, status:  LOS_IN0 OOF_IN0 OOF_IN2 IN1 IN_SEL_1 HOLD_HIST_VALID
  • IRIG-B via AUX-IN is ok: ./test_cdm_local.exe --irigb
dataframe: S00100110S111001000S...S011110000S010000000S, sec: 34, min: 17, hrs: 00, day of year: 223, year: 2024, date: 10 aug, sbs:  1054 should be  1054
  • ./test_cdm_local.exe --writereg 8 9 ### set vx_tx mux to GPS control, hard enables PRS-10 power and PRS-10 1pps in
  • observe 1pps counters in reg 68 count at 1 Hz: rb_1pps 0x51->0x52, gps_1pps 0xd5->oxd6
root@dsdm:~# ./test_cdm_local.exe 68
ds20k_reg[68] is 0x0051d520 (5362976)
ds20k_reg[68] is 0x0052d624 (5428772)
...
  • observe PRS-10 can see the 1pps signal "130" changes to "2" after 243 seconds to "4"
  • observe period of GPS and PRS-10 1pps is identical (plus/minus 1 clock)
root@dsdm:~# ./test_cdm_local.exe 13 14
ds20k_reg[13] is 0x077356d4 (124999380)
ds20k_reg[14] is 0x077356d4 (124999380)
  • observe GDM/CDM clocks are exactly 125 MHz, we are running on DS-DM internal oscillator
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735943 (125000003) should be ~125 MHz
0x1034 rx_clk:             0x07735943 (125000003) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735943 (125000003) should be ~125 MHz
0x103C tx_clk:             0x07735943 (125000003) should be ~125 MHz
0x1040 clk_50MHz:          0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk:         0x05f5e100 (100000000) should be 100 MHz exactly
  • switch CC to external clock:
root@dsdm:~# ./test_cdm_local.exe --cc-in0
CC use clock input 0: 10 MHz LEMO external clock
root@dsdm:~# ./test_cdm_local.exe --cc
Polling CC status...
Clock chip state 1, status:  IN0 IN_SEL_REGCTRL IN_SEL_0 HOLD_HIST_VALID
  • observe rx_clk and tx_clk are now slightly off: they run from PRS-10 10 MHz clock and are measured against the DS-DM internal oscillator.
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x94b12519, ds20k version 0x20240814
DS-DM firmware build 0x94b12519, ds20k version 0x20240814
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x077358e1 (124999905) should be ~125 MHz
0x1034 rx_clk:             0x07735b49 (125000521) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735b49 (125000521) should be ~125 MHz
0x103C tx_clk:             0x07735b49 (125000521) should be ~125 MHz
0x1040 clk_50MHz:          0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk:         0x05f5e100 (100000000) should be 100 MHz exactly
  • observe GPS and PRS-10 1pps period is now exactly 125*10^6 of 8ns clocks (equal to 1 second)
root@dsdm:~# ./test_cdm_local.exe 13 14
ds20k_reg[13] is 0x0773593f (124999999)
ds20k_reg[14] is 0x0773593f (124999999)
  • look at them repeatedly, observe reg 13 "GPS 1pps period" has some wobble, reg 14 "Rb clock 1pps period" is steady. this is as expected: if GPS 1pps was steady, we do not need to Rb clock. PRS-10 1pps is derived from the PRS-10 10 MHz clock and is measured against the 125 MHz clock derived from the same 10 MHz clock in the GDM FPGA.

Test status:

  • GPS 1pps to SMB-in ok (LED flashes)
  • GPS 1pps to FPGA ok
  • GPS data to FPGA ok
  • GPS 1pps to PRS-10 enabled from FPGA ok
  • PRS-10 Rb clock 1pps out to FPGA ok
  • PRS-10 Rb clock 1pps out to FPGA to LEMO AUX out ok
  • can see GPS 1pps, IRIG-B, PRS-10 1pps out, 10 MHz on the scope, ok
  • PRS-10 syncs on leading edge (0->1) of GPS 1pps signal, ok
  • reg 13 and 14 1pps periods are identical, ok
  • CC locks on PRS-10 10 MHz clock, DS-DM runs on PRS-10 clock, ok
  • NOT TESTED - smb output
  • NOT TESTED - optical converter fiber to SMB
  • NOT TESTED - optical converter SMB to fiber
  • NOT TESTED - SMB loopback
  • NOT TESTED - fiber loopback

VX busy logic

DS-20K DAQ

Overview

DS-DM, GDM and CDM are key parts of the DS-20K DAQ system:

  • common clock distribution from external clock (atomic clock, GPS) to GDM to per-quadrant CDMs to VX digitizers
  • common trigger distribution from GDM internal algorithm or external input to all VX digitizers
  • run control: GDM, CDM, VX all start recording data at the same time (clock and timestamp reset)
  • collection of trigger data from VX digitizers to per-quadrant CDMs to GDM

Deliverables

  • hardware and firmware for GDM to CDM clock distribution
  • hardware and firmware for CDM to VX clock distribution
  • hardware and firmware for GDM external clock input (atomic clock or GPS)
  • hardware and firmware for CDM and VX serial communications (VX LVDS I/O connector)
  • firmware for run control (timestamp reset and sync): GDM to CDM to VX
  • firmware for common trigger distribution: GDM to CDM to VX
  • firmware for trigger data flow: VX to CDM to GDM
  • firmware for busy control: VX to CDM to GDM back to CDM to VX
  • firmware for flow control: FEP to GDM MIDAS frontend to GDM to CDM to VX
  • GDM MIDAS frontend: clock selector and monitoring, trigger and run control, busy and flow control, GDM housekeeping
  • CDM MIDAS frontend: clock monitoring, CDM housekeeping

specific performance:

  • GDM external clock: 10 MHz GPS clock
  • GDM to CDM fiber link:
    • clock XXX MHz
    • link data rate: XXX Gbit/sec
    • CDM recovered clock: XXX MHz
    • CDM recovered clock jitter: XXX ns
    • phase alignment between CDMs: XXX ns
    • phase alignment between CDMs persists across reboots, power cycles, firmware updates
    • phase alignment between CDMs should be easy to measure
    • phase alignment between CDMs should be easy to recalibrate if hardware parts are replaced (DS-DM boards, fiber transceivers, fiber cables, etc)
    • data packet bandwidth: XXX Mbytes/sec
    • data packet latency: XXX clocks
    • data packet skew between CDMs: XXX clocks
  • CDM to VX clock:
    • clock: XXX MHz
    • jitter, all CDM clock outputs: XXX MHz
    • phase alignment between all CDM clock outputs: XXX ns
  • CDM to VX trigger:
    • TBD (use the VX "sync" input or VX LVDS I/O line or VX serial link packet)
  • CDM to VX serial link:
    • clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
    • bit rate: XXX bits/sec
    • latency: XXX link clocks
    • maximum skew between VXes: XXX ns
  • VX to CDM serial link:
    • clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
    • bit rate: XXX bits/sec
    • latency: XXX link clocks
    • maximum skew between VXes: XXX ns
  • timestamp reset:
    • maximum skew between VXes: XXX ns
  • busy round trip time: XXX ns (VX to CDM to GDM back to CDM to VX)
  • flow control latency: XXX ns (FEP software to GDM MIDAS frontend to GDM to CDM to VX)

Technical risk items

this refers to unexpected behaviour and performance of system components, causes big difficulty in implementing the system, prevents delivery of deliverables, and prevents or negatively affects operation of the DS-20K DAQ or of the whole experiment.

(14-sep-2022, list is not sorted by any criterial: severity, probability, ease of investigation)

(stability of course is long term stability, across hours, days, weeks, months, years)

  • stability of Enclustra FPGA modules (crashes/year, failures to boot/year, flash corruption/year)
  • stability of GDM external clock PLL (lock loss/year)
  • stability of CDM recovered clock (lock loss/year, unexpected phase drifts, etc)
  • unexpected failures or bit error rates in GDM-CDM fiber links
  • stability of CDM VX clock outputs (stability of clock cleaner chip)
  • stability of VX internal clock distribution (VX PLL lock loss events)
  • stability of VX CAEN base firmware (different versions of CAEN base firmware have different clock distribution behaviour)
  • strange things in CAEN base firmware (unexpected clocking of LVDS I/O, unexpected phase shifts between clocks, etc)
  • DS-DM and VX hardware problems (incompatible LVSD I/O, incompatible clock signals, etc)

Milestones

(14-sep-2022: at this stage of the project, priority must be given to identifying and retiring (so called) technical risk factors. it is not good to build the complete system only to discover that (for example) some Enclustra FPGA modules require 5 attempts to boot and erase their flash memory contents once a month. Both example are real-life actual problems that caused big difficulties in GRIFFIN/TIGRESS and ALPHA-g experiments).

Development and testing milestones in time reversed order:

  • full DAQ data challenge: all VXes, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
  • one quadrant data challenge: 1 VME crate of VX, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
  • vertical slice data challenge: 1 VME crate, 2 VX, 2 CDM (1 VX per CDM), GDM, etc operate as designed
  • GDM-CDM link finalized (data rate frozen, data packet format frozen, data content permitted to change)
  • CDM-VX serial link finalized (data rate frozen, data packet format frozen, data content permitted to change)
  • run control (timestamp reset) and trigger distribution design agreed upon, frozen (list of possible triggers permitted to change)
  • VX to CDM to GDM data flow design agreed upon, frozen (data contents permitted to change)
  • major technical risk items retired (all hardware and firmware is working as expecred without mysteries and surprises, all problems are identified, investigated, resolved, solutions tested)
  • stable operation of CDM-VX serial links in vertical slice system
  • stable operation of GDM to CDM clock in vertical slice system
  • stable operation of CDM to VX clock in vertical slice system
  • vertical slice system assembled (1 VME crate, 2 VX, 2 CDM, 1 GDM, network, FEP, TSP, MIDAS)

ZZZ

ZZZ

Retrieved from "https://daq00.triumf.ca/DaqWiki/index.php?title=DS-DM&oldid=8043"