Revision as of 12:39, 31 July 2023

DS-DM

DarkSide-20k Global and Crate Data Manager board (GDM and CDM).

Global Data Manager (GDM):

clock distribution to CDM boards (including GPS/atomic clock source)
collection of trigger data from CDM boards, processing and distribution of trigger decision to CDM boards
run control
integration with GPS 10MHz and 1pps clocks and GPS/IRIG date and time information

Crate Data Manager (CDM):

clock distribution from GDM to CAEN VX digitizers
receive trigger data from CAEN VX digitizers
send trigger data to GDM
run control and dead time control

Links

https://dsvslice.triumf.ca/vx_napoli - DS vertical slice at TRIUMF
https://edev-group.triumf.ca/fw/exp/darkside/gcdm - git repository, DS-DM firmware
https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0 - git repository, DS-DM board
https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0/-/blob/master/Altium/Project%20Outputs%20for%20DS-DM-Rev0/SCH-DS-xDM-Rev0.PDF - DS-DM schematics
https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0/-/blob/master/Testing/Clk3_XO_125_to_fpgaIN_recoveredMGTclk_to_IN2_Si5394-RevA-Project.slabtimeproj
https://docs.xilinx.com/v/u/en-US/ds593 - Xilinx Platform Cable USB II
https://www.enclustra.com/en/products/system-on-chip-modules/mercury-xu8/ - Enclustra
https://bitbucket.org/team-ds-dm/ds-dm-software - git repository, DS-DM MIDAS frontend
https://bitbucket.org/team-ds-dm/phasemeasurement - VX Phase measurement script
https://ladd00.triumf.ca/daqinv/frontend/list/178 - inventory database

Onboard hardware

jtag chain: arm_dap_0 0x5BA00477, xczu4_1 0x04721093
Eclustra Mercury+ XU8 module: ME-XU8-4CG-1E-D11E-R2.1
- Xilinx® Zynq Ultrascale+™ MPSoC XCZU4CG-1FBVB900E
- DDR4 ECC SDRAM (PS) 2 GB
- DDR4 SDRAM (PL) 1GB
ethernet mac chip: AT24MAC402-SSHM-T ("602" chip is wrong)
USB UART for Enclustra serial console, micro-USB, 115200n8
clock chip: SI5394A-A-GM and oscillator CS-044-054.0M (54 MHz)
U23 3.3V current meter and thermometer
LEDs:
- LED_FP A/B/C/D 0/1/2/3
- led1 - 3V3_SW_ON, SOM_POWER_GOOD - Enclustra FPGA module 3.3V power is good
- led2 - LTM4624 PGOOD
- led3 - FPGA_DONE - FPGA has booted
- led4 - TP-S-1, PCLK_P
- led5 - TP-S-2, PCLK_N
LEMO connectors (top to bottom)
- J4 - input (NIM/TTL) (EXT_IN_LV(1), EXT_IN_LV(2))
- J5 - input (NIM/TTL) (EXT_IN_LV(3), EXT_IN_LV(4))
- J6 - external clock (GPS 10MHz and PPS)
- J7 - output (NIM/TTL) (EXT_OUT(1), EXT_OUT(2))
SMA connectors
- J9, J10 - CLK_CCA from U6 C.C.
- J11, J12 - CLK_TP0
RJ45 ethernet connector (100mbit: green light on, yellow light flashes when there is traffic)
SFP connector (SFP is FTLF8526P3BNL, 6 Gbit/sec, 850 nm, 300m 50/125um OM3 MMF)
4 QSFP connectors (GDM)
6 VX connectors (CDM)

Buttons, jumpers and switches

Buttons:

PB1 - HRST - reboot FPGA (power-on reset)
PB2 - SRST - (SRSTn) - reboot ARM CPU

Switches:

SW1 - boot mode BM0, BM1 [-->]
SW2 - LEMO output NIM<->TTL
SW3 - LEMO input 1 and 2 NIM/TTL
SW4 - LEMO input 2 and 4 NIM/TTL
SW5 - LEMO clock input NIM/TTL
SW6 - serial console select. [PS<--PL] PS is ARM CPU, PL is FPGA.

Front panel

| top
|
| LED-FP1 | LED_FP(0,1,2,3)
|
| SFP J???
|
| LEMO J4-LEFT, J4-RIGHT | EXT_IN_LV(1), EXT_IN_LV(2)
| LEMO J5-LEFT, J5-RIGHT | EXT_IN_LV(3), EXT_IN_LV(4)
| LEMO J6-LEFT, J6-RIGHT | CLK_EXT1, CLK_EXT0 (125 MHz only) 
| LEMO J7-LEFT, J7-RIGHT | EXT_OUT(1), EXT_OUT(2)
|
| J-VX-1
| J-VX-2 or QSFP-1
| J-VX-3 or QSFP-2
| J-VX-4 or QSFP-3
| J-VX-5 or QSFP-4
| J-VX-6
|
| RJ45 J3 ethernet
|
| bottom

VX adapter board

LVDS I/O connector

34 pin connector: 0|:::: :::: :::: :::: :|15,16 n/c

split-cable connection

0 .. 7 -> N/C

8 -> VX_RX(3)
9 -> VX_RX(2)
10 -> VX_RX(1)
11 -> VX_RX(0)

12 <- VX_TX(0)
13 <- VX_TX(1)
14 <- VX_TX(2) (set by jumper routed here or to VX CLKIN SYNC)
15 <- n/c (CLK routed to VX CLKIN CLK)

16 - n/c

one-to-one connection

0 -> VX2_RX(3)
1 -> VX2_RX(2)
2 -> VX2_RX(0)
3 -> VX2_RX(1)

4 <- CLK
5 <- VX2_TX(0)
6 <- VX2_TX(1)
7 <- VX2_TX(2)

8 -> VX1_RX(3)
9 -> VX1_RX(2)
10 -> VX1_RX(1)
11 -> VX1_RX(0)

12 <- VX1_TX(0)
13 <- VX1_TX(1)
14 <- VX1_TX(2) (set by jumper routed here or to VX CLKIN SYNC)
15 <- n/c (CLK routed to VX CLKIN CLK)

16 - n/c

Board schematics

File:SCH-DS-xDM-Rev0.PDF
note: FPGA pin annotations ("IO", "SCLK", "PCLK", etc) on the schematics are bogus, instead, trace them to the FPGA pins.
note: ENC A is J800, ENC B is J801, ENC C is J900 (schematic name to enclustra name)
note: Enclustra special pins: "GC" is "clock capable", "HDGC" is "clock capable", "MGTREFCLK" is MGT reference clocks.
board modifications:
- ethernet mac chip
- NIM output (no U15, etc)
- RJ45 wrong pinout (board mod or special ethernet cable)
- 125 MHz clock mods (TBW)
- disconnect QSFP0_SEL from SFP_RS0 and QSFP1_SEL from SFP_RS1, these signals are not used by modern SFPs
- provide SFP i2c modsel to allow SFP and QSFP at the same time on address 0x50

FPGA MGT blocks

* SFP - ENC B45,B47 and B48,B50 - FPGA D5,D6 MGTHTX0_D and D1,D2 MGTHRX0_D
* QSFP0 TX0 - ENC C13,C17 - FPGA H5,H6 MGTHTX0_C
* QSFP0 TX1 - ENC C21,C25 - FPGA G7,G8 MGTHTX1_C
* QSFP0 TX2 - ENC C29,C23 - FPGA F5,F6 MGTHTX2_C
* QSFP1 TX0 - ENC C37,C41 - FPGA E7,E8 MGTHTX3_C
* QSFP1 TX1 - ENC C45,C47 - FPGA P5,P6 MGTHTX0_B
* QSFP1 TX2 - ENC C51,C53 - FPGA M5,M6 MGTHTX1_B
* QSFP2 TX0 - ENC C57,C59 - FPGA L3,L4 MGTHRX2_B
* QSFP2 TX1 - ENC C63,C65 - FPGA K5,K6 MGTHTX3_B
* QSFP2 TX2 - ENC C75,C77 - FPGA W3,W4 MGTHTX0_A
* QSFP3 TX0 - ENC C79,C81 - FPGA V5,V6 MGTHTX1_A
* QSFP3 TX1 - ENC C85,C87 - FPGA T5,T6 MGTHRX2_A
* QSFP3 TX2 - ENC C89,C91 - FPGA R3,R4 MGTHTX3_A

Clock distribution

Simplified:

125 MHz osc -> CLK_XO_125 -> MGTREFCLK0_A -> not used
125 MHz osc -> CLK3_XO_125 -> MGTREFCLK1_B -> SFP RX ref clock, QSFP RX and TX ref clock (this is not final design!)
(disconnected) 125 MHz osc -> CLK2_XO_125 -> FPGA AG8,AH8 (GC)
125 MHz osc -> C.C. in1

C.C. in0 <- CLK_EXT1 (10 MHz GPS clock)
C.C. in1 <- 125 MHz osc
(disconnected) C.C. in2 <- CLK_CC_IN <- MGTREFCLK0_D <- SFP RX clock (cannot be used because of uncontrollable phase)
C.C. in2 <- CLK2_XO_125 <- FPGA AG8,AH8 (GC) <- SFP RX recovered 125 MHz clock
C.C. in3 <- CLK_FB

C.C. 125 MHz -> CLK_CC_OUT0 -> MGTREFCLK0_B -> QSFP RX and TX ref clock (final design)
C.C. 125 MHz -> CLK_CC_OUT1 -> MGTREFCLK1_D -> SFP TX clock
C.C. 62.5 MHz -> VX clock fanout

proposed changes:

add C.C. 125 MHz -> new CLK_CC_OUT2 (old CLK2_XO_125) -> FPGA AG8,AH8 (GC)
change C.C. in2 <- new CLK_CC_IN <- FPGA AK8,AK9 (non-GC)

Complete:

125 MHz oscillator - U5 fan out -
  q0 -> CLK_XO_125 -> ENC C72,C74 -> FPGA R7,R8 MGTREFCLK0_A (not used)
  q1 -> U6 C.C. in1
  q2 -> CLK3_XO_125 -> ENC C7,C9 -> FPGA J7,J8 MGTREFCLK1_B -> SFP RX reference clock, QSFP RX and TX reference clocks (not final design!)
  q3 -> disconnected on the board, was CLK2_XO_125 -> ENC C151,C153 -> FPGA AG8,AH8 (GC)

U6 C.C (clock cleaner) -

in0 <- CLK_EXT1 (presumably GPS 10 MHz ref clock)
in1 <- 125 MHz oscillator via U5
in2 <- (was: CLK_CC_IN <- ENC B10,B12 <- FPGA D9,D10 MGTREFCLK0_D <- SFP RX clock, 125 MHz)
in2 <- CLK2_XO_125 <- ENC C151,B153 <- FPGA AG8,AH8 (GC) <- mgt_link_data_to_processing.rx_data_clk (SFP RX data clock, 125 MHz)
in3 <- CLK_FB

out0 -> CLK_CCA -> U12 (125 MHz)
out1 -> CLK_CCB -> VX1..6 (62.5 MHz)
out2 -> CLK_CCC -> VX7..12 (62.5 MHz)
out3 -> CLK_FB into in3

CLK_CCA -> U12 (125 MHz fan out) ->

Q0 -> not used
Q1 -> CLK_CC_OUT0 -> ENC C3-5 -> FPGA L7,L8 MGTREFCLK0_B -> QSFP RX and TX reference clocks (final design)
Q2 -> CLK_CC_OUT1 -> ENC B3-5 -> FPGA B9,B10 MGTREFCLK1_D -> SFP TX clock
Q3 -> not used
Q4 -> not used
Q5 -> SMA J9/J10

CLK_TP0 - FPGA N7,N8 MGTREFCLK1_A - ENC C69,C71 - DS-DM SMA J11, J12 (NOT IN CDM PROJECT)

Notes:
* CLK_XO_125 (125 MHz osc) is not used
* 62.5 MHz VX clock does not go into the FPGA
* CLK_EXT0 going to in1 of U5 cannot be used. only permitted frequency is 125 MHz (it drives the MGT reference clocks) and is this frequency is too high for LEMO cables and connectors

Proposed modifications:
- CLK_CC_IN: from FPGA output pin (ENC C142,C144 - FPGA AK8,AK9) to clock cleaner in2 (instead of CLK2_XO_125 pins)
- CLK_CCA -> U12 -> currently unused out3 -> CLK2_XO_125 FPGA pins
- repurpose CLK_EXT0 at GPS 1pps/IRIG input to FPGA

I2C bus

I2C_SCL is J-ENC-A pin 111 I2C_SCL is FPGA I2C_SCL_PL AB13 (IO_L1N_TOL1D_64) and I2C_SCL_PS F18 (PS_MIO10)
I2C_SDA is J-ENC-A pin 113 I2C_SDA is FPGA I2C_SDA_PL AH13 (IO_L7N_T1L1Q_AD13N_64) and I2C_SDA_PS G18 (PS_MIO11)

XU8 secure EEPROM ATSHA204A at 0x64, this is 0110'010X -> linux _011'0010 is 0x32. (but responds to scan and read at 0x33)
U4 ethernet mac chip, EEPROM at 1010 A2 A1 A0 X and MAC/serial_no at 1011 A2 A1 A0 X. A0=VCC, A1=VCC, A2=GND -> linux _101'0011 and _101'1011 is 0x53 and 0x5B.
U6 clock chip, address 1101 0 A1 A0 X. A1=VCC, A0=N/C (internal pull-up) -> linux _110'1011 is 0x6b
U23 voltmeter at 10011 ADR1 ADR0. ADR0=GND, ADR1=VCC -> linux _100'1110 is 0x4e
SFP, address 1010000X -> linux _101'0000 is 0x50. additional SFP data at 0x51
QSFP0, QSFP1, QSFP2, QSFP3 (QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL)

I2C clock builder connection

use Silicon Labs USB "Clock builder pro field programmer", www.silabs.com/CBProgrammer
connect rainbow jumper cable pins:
- black - 1-GND to GND on DS-DM
- white - 3-SCLK to SCL on the DS-DM
- grey - 7-SDA_SDIO to SDA on the DS-DM
power up the DS-DM
plug USB programmer into Windows laptop
on Windows, run "ClockBuilder Pro"
it should report "Field programmer detected", press "EVB GUI"
in EVB GUI, press "Config", set I2C address 0x6B
press "Scan", it should find Si5394A-A-GM
select the "Status" tab, should see real-time status of clock chip

GDM MGT configuration

TX configuration:
GDM MGT transceivers are configured as "multilane" TX and RX.
there is 12 TXes ("lanes")
MGT reference 125 MHz clock goes into [2:0]gtreclk00_in and [11:0]gtrefclk0_in
one MGT is designated as "master"
PLL of master MGT converts reference clock into common TX clock and common tx_user_clk2 which becomes tx_data_clk
common tx_user_clk2 aka tx_data_clk goes into all TXes and clocks tx_data.
tx_user_clk2 aka tx_data_clk is 125 MHz but not same phase as MGT reference clock.

RX interim configuration:
there is 12 RXes ("lanes")
each RX produces it's own recovered RX clock
"multilane" configuration assumes all RX recovered clocks run at the same frequency (TX on the other end are driven by common TX clock, see above), but have different phase
one RX recovered clock is designated as "master" (rx_user_clk2 aka rx_data_clk) and a phase-matching fifo/buffer is used to bring rx_data from all 12 RXes to this common rx_data_clk
this works because each CDM SFP TX runs on the SFP RX recovered clock which is frequency-locked with the GDM QSFP TX clock.

RX final configuration:
MGTs permit using the common TX clock (tx_user_clk2 aka tx_data_clk) as the common rx_data_clk (they are frequency locked through the CDM).
this permits use of tx_data_clk as the main clock domain in the GDM and removes the need to bring rx_data into the tx_data_clk domain (actually this is done in the MGT RX phase matching fifo/buffer).

Clock path

10 MHz ext clock or GDM 125 MHz oscillator
-> GDM QSFP MGT reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz (GDM main clock domain) and TX bit clock 2.5 GHz
-> GDM QSFP optic transmitter
-> CDM SFP optic receiver
-> CDM SFP MGT, RX reference clock is CDM 125 MHz oscillator
-> MGT RX recovered clock 125 MHz (CDM main clock domain)
-> CC_CLK_IN -> CDM C.C. -> CC_CLK_OUT1 -> CDM SFP TX reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz and TX bit clock 2.5 GHz
-> (tx_data phase matching fifo from CDM main clock domain to tx_data_clk)
-> CDM SFP optic transmitter
-> GDM QSFP RX optic receiver (12x)
-> GDM QSFP MGT (RX reference clock is same as TX reference clock)
-> MGT RX recovered clock (12x recovered clocks)
-> in multi-lane configuration, one of them is the "master" recovered clock rx_data_clk
-> (rx_data phase matching fifo from rx_data_clk to GDM main clock domain)

Clock domains

GPS

no GPS : GDM runs from internal 125 MHz oscillator
external 10 MHz clock : GDM runs from external 10 MHz clock and optional 1pps signal (use VME-NIMIO32 NIM outputs)
GPS receiver : GDM runs from GPS 10 MHz clock and GPS IRIG serial data
LNGS GPS:
- provides 1pps and serial data over fiber from GPS receiver (master). LNGS xxx box is not used.
- serial data goes to GDM, decoded, 1pps signal extracted, goes to LEMO output
- 1pps from GDM is used to train the Rubidium clock which provides a 10 MHz clock
- 10 MHz output from Rubidium clock goes to GDM 10 MHz external clock input

GDM

AXI clock (100 MHz) - AXI registers
125 MHz oscillator - to clock cleaner
10 MHz external clock LEMO input - to clock cleaner
FPGA 125 MHz clock CLK_CC_IN - to clock cleaner - not used (could be used for cascaded GDMs)
125 MHz oscillator - CLK3_XO_125 - mgt_b_ref_clk QSFP MGT reference clock (interim GDM design)
clock cleaner output 125 MHz fanout:
- CLK_CC_OUT0 - QSFP MGT reference clock (final design)
- CLK_CC_OUT1 - not used (CDM SFP reference clock)
- CLK_CC_OUT2 - not used
QSFP MGT TX data clock 125 MHz tx_data_clk (main clock domain)
- QSFP TX data
- QSFP RX data (in final design, rx_data_clk is same as tx_data_clk. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk == tx_data_clk)
- ds20k block
(in interim design, rx_data_clk is disconnected from tx_data_clk. rx_data_clk is the "multilane master clock" which is one of the 12x RX recovered clocks. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk)

note: all these clocks are frequency locked to 125 MHz

CDM

AXI clock (100 MHz) - AXI registers
10 MHz external clock LEMO input - to clock cleaner (not used)
125 MHz oscillator to fanout
- to clock cleaner
- to SFP MGT RX reference clock: CLK3_XO_125 to mgt_rx_ref_clk to gtrefclk01_in
SFP MGT RX recovered clock 125 MHz
- MGT PLL to MGT rx_user_clk2 aka rx_data_clk
- MGT rxrecclkout_out to mgt_rx_rec_clk to CLK_CC_IN to C.C.
SFP MGT rx_user_clk2 aka rx_data_clk (250 MHz/8 bit, 125 MHz/16 bit, 62.5 MHz/32 bit data) (main clock domain)
- SFP RX data
- ds20k block
- VX TX clock PLLs
- VX RX clock PLLs
C.C. fan out
- 62.5 MHz VX clocks (12x)
- CLK_CC_OUT0 (not used, GDM QSFP MGT reference clock)
- CLK_CC_OUT1 125 MHz to mgt_tx_ref_clk to gtrefclk00_in to SFP MGT TX reference clock
- CLK_CC_OUT2 (not used)
SFP MGT tx_user_clk2 aka tx_data_clk
- SFP TX data
- TX data phase matching fifo from main clock domain to tx_data_clk
VX TX clock PLLs
- 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
- VX TX data phase matching from main clock domain to VX TX clock (12 total)
- VX TX serializer
- VX TX LVDS transmitter
VX RX clock PLLs
- 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
- VX RX LVDS receivers (12 total)
- VX RX deserializers (12 total)
- VX TX data phase matching from VX RX clock to main clock domain

VX

everything runs on the VX main 125 MHz clock
correct phase of VX to CDM LVDS data is adjusted by scan of CDM VX RX PLL clock phase (VX to CDM link is now established)
correct phase of CDM to VX LVDS data is adjusted by scan of CDM VX RX PLL clock phase (link is established after VX to CDM idle data pattern changes from "VX RX data bad" to "good".
after good phases are found by scan, they are not expected to change unless cables are changed, CDM and VX modules are changed or CDM firmware is rebuilt. (rebuild of VX firmware should not affect LVDS data phase).
if there is excessive link errors, phase scan must be repeated.

Board test plan

To test:

Enclustra FPGA board
SFP port - SFP_RS0/RS1 connected to QSFP0_SEL/1_SEL is wrong? our Finisar SFP says RS0, RS1 N/C, so probably okey.
SW5 CLK_EXT1 NIM works. TTL needs to be tested. flipping SW5 CLK_EXT0 side from NIM to TTL makes CLK_EXT1 go LOS and OOF in the clock chip. R23 and R53 should be removed?
BOOT_MODE 0 and 1

Partial:

U23 3.3V current meter and thermometer. V1,V2 is current monitor, same as in application note. V3,V4 is thermometer, same in application note, except capacitor C118 is 0.1u instead of 470pF. Tested ok: Tint, VCC, V1, V2, V1-V2. Test failed: TR2 reads 50-something degC instead of same as Tint.

Done:

LED_FP1A..D: tested ok. K.O. 15 sep 2022
USB UART: tested ok. K.O. 15 sep 2022
J4A, J4B, J5A, J5B LEMO inputs (NIM/TTL) EXT_IN_LV(1..4). TTL threshold 1.7V, NIM threshold -0.3V. 50 Ohm termination. TTL straight, NIM inverted.
J6A, J6B LEMO clock inputs: 10 MHz TTL from chronobox works, 10 MHz NIM from IO32 works. 18-apr-2023
J7A, J7B LEMO outputs EXT_OUT(1), EXT_OUT(2) (NIM/TTL) (tested 23nov2022, K.O.). TTL ???, NIM inverted.
- TTL out no 50 ohm termination: 0=0V, 1=5V, rise and fall time ~5 ns
- TTL out with 50 ohm termination: 0=0V, 1=2.5V, rise and fall time <2ns
- NIM out no 50 ohm termination: 0=+50mV, 1=-1.8V, rise and fall time ~3ns
- NIM out with 50 ohm termination: 0=0V, 1=-0.9V, rise and fall time <2ns
ethernet MAC i2c chip (K.O. 20sep2022: can read: i2cdump 0 0x5b, nov2022 have u-boot driver)
CDM VX ports 2x(CLK, 3 tx, 4 rx) tested using test_cdm.exe and LVDS loopback in VX firmware.
SFP i2c tested KO 22jun2023
QSFP i2c tested KO 22jun2023
i2c testing complete 22jun2023
QSFP rx,tx tested 26june2023. lane0,1,2 ok, lane3 tx not connected, laser is off. qsfp0,1,2,3 all lanes ok.
SFP rx,tx tested 26june2023. rx and tx okey. LOS ok, mod_absent ok.

Failure:

ethernet: does not connect to alliedtelesys switch. connects to my USB-eth adapter at 100 Mbit speed. uboot mii status reports connection speed oscillating between 1000, 100 and 10. K.O. 16-sep-2022
SFP LOS and mod_absent are swapped (in the FPGA pin definitions?)
QSFP slot numbering is wrong.

Checklist for newly build boards

put new board on workbench
check - vme connector present, vme extraction handles present
check - standoff are removed from all thru-holes
plug Enclustra module
check - SW6 both switches are in the "PS" position
connect micro-usb cable from linux PC
connect ethernet from 1gige capable network switch
connect power from lab power supply - GND, +5V and -12V nominal, +5.7V and -12.5V actual
power up, +5V current 2.10-2.8A, -12V current 0.05A
on linux PC, open a new terminal, run: minicom -D /dev/ttyACM0
in minicom window, observe messages about Xilinx first stage boot loader, etc
on the ethernet switch, observe network link is 1gige speed (not 10mbit, not 100mbit).
if everything boots okey, there will be a login prompt, login as root, password root.
busybox devmem 0x80010000 # read firmware version number, i.e. 0xEDAD0A77
TBW - test LEMO inputs
TBW - test LEMO outputs
TBW - test VX connectors
TBW - test SFP connector
TBW - test QSFP connector

Serial console

check that linux computer has correct udev rules to allow access to /dev/ttyACM devices, see https://daq00.triumf.ca/DaqWiki/index.php/SLinstall#Configure_USB_device_permissions and https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Configure_USB_device_permissions
connect micro-USB cable to connector J-UCB, other end connect to linux computer
observe /dev/ttyACM0 was created
run "minicom -D /dev/ttyACM0" (default serial settings are ok, otherwise, 115200n8)
should have gdm-cdm login
username root, password root

i2c

ZynqMP> i2c bus
Bus 0:  i2c@ff020000
ZynqMP> i2c dev 0
Setting bus to 0
ZynqMP> i2c probe  
Valid chip addresses: 33 4E 53 5B 6B 77
ZynqMP> i2c md 0x5b 0x98
0098: fc c2 3d 00 00 12 1a 6e 0a 90 85 04 94 10 08 50    ..=....n.......P

root@gdm-cdm:~# i2cdetect 0
Warning: Can't use SMBus Quick Write command, will skip some addresses
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0.
I will probe address range 0x03-0x77.
Continue? [Y/n] 
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                                                 
10:                                                 
20:                                                 
30: -- -- -- 33 -- -- -- --                         
40:                                                 
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- -- 
60:                                                 
70:                                                 
root@gdm-cdm:~# 
root@gdm-cdm:~# i2cdump 0 0x5b
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0, address 0x5b, mode byte
Continue? [Y/n] 
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
80: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00    ???????P?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e    ........??=..??n
a0: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00    ???????P?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e    ........??=..??n
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
root@gdm-cdm:~#

root@gdm0:~# i2cdetect -r 0
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e -- 
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- -- 
70: -- -- -- -- -- -- -- --                         
root@gdm0:~#

root@cdm0:~# i2cdetect -r 0
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e -- 
50: 50 51 -- 53 -- -- -- -- -- -- -- 5b -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- -- 
70: -- -- -- -- -- -- -- --                         
root@cdm0:~#

0x33 - XU8 secure EEPROM (should be at 0x32)
0x4e - U23 current and temperature monitor
0x50, 0x51 - SFP
0x50 - QSFP, 4 QSFP modules enabled by GPIO QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL
0x53, 0x5b - ethernet mac eeprom
0x6b - U6 clock chip

U23

internal temperature only

root@gdm0:~# i2cset -y 0 0x4e 0x01 0x00 b  # control register: "repeat mode, internal temperature only"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x03 # "Tint ready" and "busy", "busy is always 1 in repeat mode"
root@gdm0:~# i2cdump -y 0 0x4e b
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 03 00 03 03 81 db 2a ce 2a 8e 00 6f 00 45 20 3f    ?.????*?*?.o.E ?
10: 01 00 01 01 01 db 2a ce 2a 8e 00 6f 00 45 20 3f    ?.????*?*?.o.E ?
...
readback:
reg0 - 03 - Tint ready
reg1 - 00 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB 0x81, bit 0x80 is "DV, data valid", bit 0x40 is "SS, sensor short", 0x20 is "SO, sensor open"
reg5 - Tint LSB 0xDB, Tint = 0x01DB = 475 * 0.0625 degC = 29.6 degC
reg6..F - stale data

Tint, V1, V2, TR2, VCC

root@gdm0:~# i2cset -y 0 0x4e 0x01 0x18 b  # control register: "repeat mode, V1, V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 7f 18 7f 7f 81 b9 aa d0 aa 8f 83 5e 83 5e a0 41    ???????????^?^?A
10: 01 18 01 01 01 b9 2a d0 2a 8f 03 5e 03 5e 20 41    ??????*?*??^?^ A
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x1b9*0.0625 = 27.5 degC
reg6 - V1 MSB 0xaa, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 LSB 0xd0, V1 = 0x2ad0*305.18/1000000 = 3.3447 V (correct)
reg8 - V2 MSB 0xaa, ditto
reg9 - V2 LSB 0x8f, V2 = 0x2a8f*305.18/1000000 = 3.3249 V (correct, smaller than V1)
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x5e. TR2 = 0x35e*0.0625 = 53.875 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x41, VCC = 2.5+0x2041*305.18/1000000 = 5.019 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.99 A (about right?)

Tint, V1-V2, TR2, VCC

root@gdm0:~# i2cset -y 0 0x4e 0x01 0x19 b  # control register: "repeat mode, V1-V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 7f 19 7f 7f 81 69 83 f2 83 f2 83 3a 83 3a a0 44    ?????i?????:?:?D
10: 01 19 01 01 01 69 03 f2 03 f2 03 3a 03 3a 20 44    ?????i?????:?: D
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x169*0.0625 = 22.5 degC
reg6 - V1 or V1-V2 MSB 0x83, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 or V1-V2 LSB 0xf2, V1-V2 = 0x3f2*19.42/1000000 = 0.0196142 V (correct, compare with V1 and V2 measured above)
reg8 - V2 or V1-V2 MSB
reg9 - V2 or V1-V2 LSB
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x3a. TR2 = 0x33a*0.0625 = 51.625 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x44, VCC = 2.5+0x2044*305.18/1000000 = 5.021 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.9807 A (about right?)

SFP

root@cdm0:~# i2cdump 0 0x50
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 03 04 07 00 00 00 00 40 40 0c 00 01 3d 00 00 00    ???....@@?.?=...
10: 0c 02 00 1e 46 49 4e 49 53 41 52 20 43 4f 52 50    ??.?FINISAR CORP
20: 2e 20 20 20 00 00 90 65 46 54 4c 46 38 35 32 36    .   ..?eFTLF8526
30: 50 33 42 4e 4c 20 20 20 41 20 20 20 03 52 00 9d    P3BNL   A   ?R.?
40: 00 1a 00 00 4e 33 41 42 34 4c 56 20 20 20 20 20    .?..N3AB4LV     
50: 20 20 20 20 32 30 30 33 31 39 20 20 68 f0 03 de        200319  h???
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
root@cdm0:~# 
root@cdm0:~# i2cdump 0 0x51
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 5a 00 d8 00 55 00 e2 00 90 88 71 48 8c a0 75 30    Z.?.U.?.??qH??u0
10: 21 34 01 f4 1b 58 03 e8 31 2d 04 eb 1f 07 06 31    !4???X??1-?????1
20: 31 2d 00 64 27 10 00 9e 00 00 00 00 00 00 00 00    1-.d'?.?........
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
40: 00 00 00 00 3f 80 00 00 00 00 00 00 01 00 00 00    ....??......?...
50: 01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 b7    ?...?...?......?
60: 19 9f 80 c5 0e 17 12 c4 1f 99 00 00 00 00 30 00    ??????????....0.
70: 00 00 00 00 00 00 00 00 ff ff ff ff ff ff ff 01    ...............?
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
root@cdm0:~#

QSFP

QSFP i2c enable lines, active low:

QSFP0_SEL - JENC-A 82 - PS-MIO40 - linux gpio 378
QSFP1_SEL - JENC-A 84 - PS-MIO41 - linux gpio 379
QSFP2_SEL - JENC-A 100 - PS-MIO44 - linux gpio 382
QSFP3_SEL - JENC-A 106 - PS-MIO43 - linux gpio 381 (notice 2 and 3 are out of order)

# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
# echo 378 >> /sys/class/gpio/export ### SEL0 338+40
# echo 379 >> /sys/class/gpio/export ### SEL1 338+41
# echo 381 >> /sys/class/gpio/export ### SEL3 338+43
# echo 382 >> /sys/class/gpio/export ### SEL2 338+44
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
 gpio-378 (                    |sysfs               ) in  hi 
 gpio-379 (                    |sysfs               ) in  hi 
 gpio-381 (                    |sysfs               ) in  hi 
 gpio-382 (                    |sysfs               ) in  hi 
root@gdm0:~# echo out >> /sys/class/gpio/gpio381/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio382/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio378/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio379/direction
root@gdm0:~# 
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
 gpio-378 (                    |sysfs               ) out lo 
 gpio-379 (                    |sysfs               ) out lo 
 gpio-381 (                    |sysfs               ) out lo 
 gpio-382 (                    |sysfs               ) out lo 
root@gdm0:~#
echo 1 >> /sys/class/gpio/gpio381/value
echo 1 >> /sys/class/gpio/gpio382/value
echo 1 >> /sys/class/gpio/gpio378/value
echo 1 >> /sys/class/gpio/gpio379/value
cat /sys/kernel/debug/gpio
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
 gpio-378 (                    |sysfs               ) out hi 
 gpio-379 (                    |sysfs               ) out hi 
 gpio-381 (                    |sysfs               ) out hi 
 gpio-382 (                    |sysfs               ) out hi 
root@gdm0:~# 
root@gdm0:~# i2cdetect -y -r 0
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e -- 
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- -- 
70: -- -- -- -- -- -- -- --                         
root@gdm0:~# 

NOTICE NOTHING AT ADDRESS 0x50

ENABLE QSFP0, OBSERVE IT IS AT ADDRESS 0x50

root@gdm0:~# echo 0 >> /sys/class/gpio/gpio378/value
root@gdm0:~# i2cdetect -y -r 0
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e -- 
50: 50 -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- -- 
70: -- -- -- -- -- -- -- --                         
root@gdm0:~# 
root@gdm0:~# i2cdump -y 0 0x50
No size specified (using byte-data access)
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 0d 00 02 8e 00 00 01 00 00 05 55 00 05 00 00 00    ?.??..?..?U.?...
10: 00 00 00 00 00 00 1e b8 00 00 81 a0 00 00 00 00    ......??..??....
20: 00 00 12 12 00 01 00 01 00 01 0e e0 0f 20 0e e0    ..??.?.?.???? ??
30: 00 00 1f a0 1e 3b 1f 72 00 01 00 00 00 00 00 00    ..???;?r.?......
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
60: 00 00 00 00 00 00 00 00 00 00 1f 00 00 00 08 00    ..........?...?.
70: 00 00 00 00 00 00 00 ff ff ff ff ff ff ff ff 00    ................
80: 0d 00 0c 04 00 00 00 40 40 02 d5 05 67 00 00 96    ?.??...@@???g..?
90: 00 00 c8 00 46 49 4e 49 53 41 52 20 43 4f 52 50    ..?.FINISAR CORP
a0: 20 20 20 20 07 00 90 65 46 54 4c 34 31 30 51 44        ?.?eFTL410QD
b0: 34 43 20 20 20 20 20 20 41 20 42 68 07 d0 00 43    4C      A Bh??.C
c0: 00 07 0f de 58 37 39 41 43 30 52 20 20 20 20 20    .???X79AC0R     
d0: 20 20 20 20 32 32 30 33 30 39 20 20 3c 10 00 9e        220309  <?.?
e0: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20                    
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
root@gdm0:~#

ethernet mac eeprom

correct chip with 84-bit ethernet mac address

root@cdm1:~# i2cdump 0 0x53
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
root@cdm1:~# 

root@cdm1:~# i2cdump 0 0x5b
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
80: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00    ????4??2?K?.?...
90: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c    ..........??=?Q<
a0: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00    ????4??2?K?.?...
b0: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c    ..........??=?Q<
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
root@cdm1:~#

wrong "602" chip with 64-bit IPv6 address

root@cdm0:~# i2cdump 0 0x53
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
root@cdm0:~# i2cdump 0 0x5b
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
80: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00    ???????Q?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e    ........??=..??.
a0: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00    ???????Q?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e    ........??=..??.
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
root@cdm0:~#

read ethernet mac address from i2c

(this code is copied from uboot command line i2c code)

in uboot sources board/xilinx/common/board.c replace original function with this:

// special code to read ethernet MAC address from the DS-DM-Rev0 board. K.O. Sep-2022

int zynq_board_read_rom_ethaddr(unsigned char *ethaddr)
{
        struct udevice *bus;
	int ret;
        int busnum = 0;

	ret = uclass_get_device_by_seq(UCLASS_I2C, busnum, &bus);
	if (ret) {
           printf("%s: No bus %d\n", __func__, busnum);
           return ret;
	}

        int chip_addr = 0x5B;

        struct udevice *dev;

        ret = i2c_get_chip(bus, chip_addr, 1, &dev);
	if (ret) {
           printf("%s: Bus %d no chip 0x%02x\n", __func__, busnum, chip_addr);
           return ret;
	}

        int dev_addr = 0x98;

        unsigned char data[8];
           
        ret = dm_i2c_read(dev, dev_addr, data, 8);

	if (ret) {
           printf("%s: Bus %d chip 0x%02x read error %d\n", __func__, busnum, chip_addr, ret);
           return ret;
	}

        printf("%s: Bus %d chip 0x%02x addr 0x%02x read: 0x %02x %02x %02x %02x %02x %02x %02x %02x\n", __func__, busnum, chip_addr, dev_addr, data[0], data[1], data[2], data[3], data[4], data[5], data[6], data[7]);

        // see Atmel-8807-SEEPROM-AT24MAC402-602-Datasheet.pdf

        if (data[0] == 0) {
           // eiu-48 chip
           ethaddr[0] = data[2];
           ethaddr[1] = data[3];
           ethaddr[2] = data[4];
           ethaddr[3] = data[5];
           ethaddr[4] = data[6];
           ethaddr[5] = data[7];
        } else {
           // eiu-64 chip
           ethaddr[0] = data[0];
           ethaddr[1] = data[1];
           ethaddr[2] = data[2];
           ethaddr[3] = data[5];
           ethaddr[4] = data[6];
           ethaddr[5] = data[7];
        }

        printf("%s: ethaddr %02x:%02x:%02x:%02x:%02x:%02x\n", __func__, ethaddr[0], ethaddr[1], ethaddr[2], ethaddr[3], ethaddr[4], ethaddr[5]);

        return ret;
}

also this should have worked if i2c_xxx() functions were enabled in uboot:

i2c_set_bus_num(0);
i2c_probe(0x5b);
i2c_read(0x5b, 0x9a, ethaddr, 6);

read ethernet mac address from i2c (SHOULD WORK)

from: https://stackoverflow.com/questions/43637540/is-there-linux-or-u-boot-support-to-read-a-mac-address-from-a-chip-at-startup

#ethernet related setup
setup_eth=run readmac buildmac
#read mac address from eeprom
readmac=i2c dev 0; i2c read 50 FA.1 6 $loadaddr
#build the ethaddr variable
#not very nice, but does the job
buildmac=\
e=" "; sep=" " \
for i in 0 1 2 3 4 5 ; do\
setexpr x $loadaddr + $i\
setexpr.b b *$x\
e="$e$sep$b"\
sep=":"\
done &&\
setenv ethaddr $e

read ethernet mac address from i2c (DOES NOT WORK)

this method does not work: inside board/xilinx/common/board.c:int zynq_board_read_rom_ethaddr(unsigned char *ethaddr), uclass_get_device_by_ofnode(UCLASS_I2C_EEPROM, eeprom, &dev) returns ENODEV and read from i2c does not happen. K.O. Sep-2022

Read:

https://support.xilinx.com/s/question/0D52E00006hpKlsSAE/uboot-and-ethernet-mac-address-from-eeprom?language=en_US (last answer)
https://github.com/Xilinx/u-boot-xlnx/blob/master/board/xilinx/common/board.c (zynq_board_read_rom_ethaddr())

Note:

0x5B is the i2c chip address
0x9A is the data offset inside the chip, see datasheet or i2c read dump above.

Edit:

emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/u-boot/files/platform-top.h

#include <configs/xilinx_zynqmp.h>
#include <configs/platform-auto.h>
//#define CONFIG_I2C_EEPROM                                                                                                                                                                   
//#define CONFIG_SYS_I2C_EEPROM_ADDR 0x5b                                                                                                                                                     
//#define CONFIG_SYS_I2C_EEPROM_ADDR_OVERFLOW 0x0                                                                                                                                             
#define CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET 0x9A
#error HERE!

emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi

/include/ "system-conf.dtsi"
/ {
chosen {
   xlnx,eeprom = &eeprom;
};
};

&i2c0 {
eeprom: eeprom@5b { /* u88 */
compatible = "atmel,24mac402";
reg = <0x5b>;
};
};

components/yocto/workspace/sources/u-boot-xlnx/configs/xilinx_zynqmp_virt_defconfig

CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET=0x9A

enable VX clock

devmem 0x80010010 32 0x8; sleep 1;  devmem 0x80010010 32 0x0;
si5394-i2c-file /media/sd-mmcblk1p1/00_freerun.txt 0 0x6b

clock chip configuration

file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt:

VCO is 14 GHz
Tvco is 71.43 ps
N0 divider is 14, frequency is 1000 MHz
out0 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz
out1 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
out2 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
out3 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz is the feedback for zero delay

clock chip monitoring

from si5395-94-92-family.pdf:

reg 0x1: page select, set to 0 or set to 5 to read 0x53F
reg 0x2: 0x94
reg 0x3: 0x53 -> device is a si5394
reg 0xC: LOSXAXB
reg 0xD: LOS and OOF for the 4 clock inputs
reg 0xE: LOL and HOLD
reg 0xF: CAL_PLL
reg 0x11: sticky bits for reg 0xC
reg 0x12: sticky bits for reg 0xD
reg 0x13: sticky bits for reg 0xE
reg 0x14: sticky bits for reg 0xF
reg 0x1C: device reset
reg 0x1E: low power, hard reset, SYNC
reg 0x507: currently selected input clock
reg 0x52A: input clock select
reg 0x53F: HOLD_HIST_VALID and FASTLOCK_STATUS

arm and linux benchmark

memory benchmark:

daq13$ arm-linux-gnueabi-gcc -o memcpy.armv7 memcpy.cc -march=armv7 -static -O2
scp memcpy.armv7 to ...
root@gdm-cdm:~# ./memcpy.armv7 
memcpy       1 KiBytes:   1288 MB/sec
memcpy       2 KiBytes:   1924 MB/sec
memcpy       4 KiBytes:   2554 MB/sec
memcpy       8 KiBytes:   3054 MB/sec
memcpy      16 KiBytes:   3262 MB/sec
memcpy      32 KiBytes:   3250 MB/sec
memcpy      64 KiBytes:   3456 MB/sec
memcpy     128 KiBytes:   3556 MB/sec
memcpy     256 KiBytes:   3780 MB/sec
memcpy     512 KiBytes:   3795 MB/sec
memcpy    1024 KiBytes:   3789 MB/sec
memcpy    2048 KiBytes:   3729 MB/sec
memcpy    4096 KiBytes:   3717 MB/sec
memcpy    8192 KiBytes:   3687 MB/sec
memcpy   16384 KiBytes:   3632 MB/sec
memcpy   32768 KiBytes:   3529 MB/sec
memcpy   65536 KiBytes:   3318 MB/sec
memcpy  131072 KiBytes:   2893 MB/sec
root@gdm-cdm:~#

ethernet receive:

daq13:bin$ ./ttcp -t -s -n 100000 10.0.0.24
ttcp-t: buflen=8192, nbuf=100000, align=16384/0, port=5001  tcp  -> 10.0.0.24
ttcp-t: socket
ttcp-t: connect
ttcp-t: 819200000 bytes in 7.25 real seconds = 110358.39 KB/sec +++
ttcp-t: 100000 I/O calls, msec/call = 0.07, calls/sec = 13794.80
ttcp-t: 0.0user 0.2sys 0:07real 3% 0i+0d 760maxrss 0+2pf 1461+31csw
daq13:bin$ 

root@gdm-cdm:~# ./ttcp.armv7 -r -s
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001  tcp
ttcp-r: socket
ttcp-r: accept from 10.0.0.25
ttcp-r: 819200000 bytes in 7.27 real seconds = 110098.22 KB/sec +++
ttcp-r: 212040 I/O calls, msec/call = 0.04, calls/sec = 29181.53
ttcp-r: 0.1user 5.7sys 0:07real 81% 0i+0d 584maxrss 0+2pf 125601+2699csw
root@gdm-cdm:~#

ethernet transmit:

root@gdm-cdm:~# ./ttcp.armv7 -t -s -n 100000 10.0.0.25
ttcp-t: buflen=8192, nbuf=100000, align=16384/0, port=5001  tcp  -> 10.0.0.25
ttcp-t: socket
ttcp-t: connect
ttcp-t: 819200000 bytes in 6.95 real seconds = 115078.69 KB/sec +++
ttcp-t: 100000 I/O calls, msec/call = 0.07, calls/sec = 14384.84
ttcp-t: 0.0user 0.7sys 0:06real 11% 0i+0d 584maxrss 0+2pf 1162+1017csw
root@gdm-cdm:~# 

daq13:bin$ ./ttcp -r -s
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001  tcp
ttcp-r: socket
ttcp-r: accept from 10.0.0.24
ttcp-r: 819200000 bytes in 6.97 real seconds = 114841.84 KB/sec +++
ttcp-r: 161335 I/O calls, msec/call = 0.04, calls/sec = 23160.01
ttcp-r: 0.0user 1.9sys 0:06real 28% 0i+0d 760maxrss 0+2pf 80646+51csw
daq13:bin$

Install Xilinx tools

install Vivado 2020.2

login at https://www.xilinx.com/myprofile.html
go to "Downloads"
go to archive,
find 2020.2
download Xilinx_Unified_2020.2_1118_1232_Lin64.bin
sh ./Xilinx_Unified_2020.2_1118_1232_Lin64.bin
banner window should open with spinner "downloading installation data"
"a newer version is available" -> say "continue"
next
"select install type" window:
provide email and password,
select "download image"
select directory /home/olchansk/Xilinx/Downloads/2020.2\
select "linux" and "full image"
next
download summary: space required 38.52 Gbytes
download
installation progress
downloading spinner, 16 M/s 47 minutes...
"download image has been created successfully". Ok.
check contents of /home/olchansk/Xilinx/Downloads/2020.2
ls -l /home/olchansk/Xilinx/Downloads/2020.2
total 67
drwxr-xr-x 2 olchansk users    9 Sep  1 16:22 bin
drwxr-xr-x 3 olchansk users   15 Sep  1 16:23 data
drwxr-xr-x 4 olchansk users    4 Sep  1 16:22 lib
drwxr-xr-x 2 olchansk users  644 Sep  1 16:22 payload
drwxr-xr-x 2 olchansk users    7 Sep  1 16:22 scripts
drwxr-xr-x 4 olchansk users    4 Sep  1 16:22 tps
-rwxr-xr-x 1 olchansk users 3256 Nov 18  2020 xsetup
daq13:2020.2$ 
./xsetup
spinned loading installation data
xilinx design tools 2022.1 now available -> say continue
"welcome" -> next
"select product" -> vivado -> next -> vivado hl system edition -> next
select devices: only zynq ultrascale+ mpsoc -> next
select destination: /opt/Xilinx (as root, mkdir /opt/Xilinx, chmod olchansk.users /opt/Xilinx)
install ...
complete
move /home/olchansk/Xilinx/Downloads/2020.2 to /daq/daqstore/olchansk/Xilinx/Downloads/

install petalinux 2020.2

./xsetup
"a newer version is available" -> say "continue"
next
"select product to install" -> select Petalinux (Linux only) -> next
"select destination directory" -> select "/opt/Xilinx" (disk space required 2.64 GB) -> next
"summary" -> install ...
error about missing /tmp/tmp-something files
"installation completed successfully" (hard to dismiss, "ok" button is partially cut-off)
done?
I think it failed, /opt/Xilinx/PetaLinux/2020.2/bin is empty except for petalinux-v2020.2-final-installer.run
try to run it by hand, same error about /tmp/tmp-something files. strange...
notice it complains about "truncate", which truncate finds ~/bin/truncate, get rid of it,
try again
now complains about missing texinfo and zlib1g:i386
apt install texinfo -> ok
apt install zlib1g:i386 -> installs bunch of gcc stuff -> ok
try again
reports "already installed" -> delete /opt/Xilinx/.xinstall/PetaLinux_2020.2/, delete entries in ~/.Xilinx/registry/installedSW.xml
try again
success

install vivado 2022.1 and petalinux 2022.1 - everything is pretty much the same

Petalinux

cd PetaLinux_GDM_CDM
petalinux-config
enable i2c MAC address and DHCP

git clone https://bitbucket.org:/team-ds-dm/ds-dm-u-boot-xlnx.git
cd ds-dm-u-boot-xlnx
git checkout ds-dm-u-boot-xlnx

linux-components ->
uboot -> ext-local-src
external u-boot local source -> ds-dm-u-boot-xlnx (path to the customized uboot git repository)

enable DHCP

Subsystem AUTO Hardware Settings -> Ethernet Settings
randomize MAC address -> NO
ethernet mac address -> leave empty
obtain ip address automatically -> YES

set hostname and product names

Firmware Version Configuration ->
Host name -> "ds-dm"
Product name -> "Petalinux_GDM_CDM"

configure linux kernel

petalinux-config -c kernel

enable NFS-Root

petalinux-config
Image Packaging Configuration > Root File System Type -> set to NFS
Location of NFS root directory set to "/nfsroot"

petalinux-config -c kernel
Networking support > IP: kernel level configuration
enable DHCP, BOOTP, RARP
File systems > Network file systems > Root file systems on NFS

manually fix linux kernel command line:

grep nfsroot PetaLinux_GDM_CDM/project-spec/configs/config
edit CONFIG_SUBSYSTEM_BOOTARGS_GENERATED to read
earlycon console=ttyPS0,115200 clk_ignore_unused panic=60 root=/dev/nfs nfsroot=/nfsroot/%s ip=dhcp rw

check configuration in
- PetaLinux_GDM_CDM/project-spec/configs/config
- PetaLinux_GDM_CDM/project-spec/configs/rootfs_config
- PetaLinux_GDM_CDM/components/plnx_workspace/device-tree/device-tree/system-conf.dtsi

JTAG server

localhost:3121

Firmware registers

AAA

Firmware registers branch develop_ko

Register map

  0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
  1 | ALL | ALL | RW | read write scratch register
  2 | ALL | CDM | ?? | MGT not used
  3 | ALL | CDM | RO | MGT debug_data
  4 | ALL | CDM | RW | clk_config_vec
  5 | ALL | CDM | ?? | not used
  6 | ALL | CDM | RO | CDM_link_data_processing:o_error_time
  7 | ALL | CDM | RO | CDM_link_data_processing:o_error_count

255 | ALL | ALL | RW | control register

register 0 0x80010000

GDM:

0 - gdm_link_interface:i_mgt_rst
2 - gdm_link_interface:i_link_down_latched_rst
8 - GDM_link_data_processing:i_rst
10..9 - GDM_link_data_processing:i_data_mode

CDM:

0 - cdm_link_interface:i_mgt_rst
2 - cdm_link_interface:i_link_down_latched_rst
8 - CDM_link_data_processing:i_rst
10..9 - CDM_link_data_processing:i_data_mode

register 1 0x80010004

GDM:

nlinks-1..0 - gdm_link_interface:i_rx_slide_trigger

CDM:

nlinks-1..0 - cdm_link_interface:i_rx_slide_trigger

register 2 0x80010008

GDM:

nlinks-1..0 - gdm_link_interface:o_link_power_good
nlinks+15..16 - gdm_link_interface:o_link_status

CDM:

nlinks-1..0 - cdm_link_interface:o_link_power_good
nlinks+15..16 - cdm_link_interface:o_link_status

register 3 0x8001000c

GDM: simple loopback register

CDM:

31..0 - debug_data - cdm_link_interface:o_debug

o_debug:

rx_link_rst & rx_error & rx_link_up & rx_receiving_data &
std_logic_vector(rx_state_count) & tx_state_count_on_rx_clk & i_rx_ctrl3(0) &
i_rx_ctrl1(1 downto 0) & i_rx_ctrl0(1 downto 0) &
rx_data_is_k28p1_k28p5 &
i_rx_data;

register 4 0x80010010

GDM write:

0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS

GDM read:

0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS
4 - clk_config_vec(4) - CLK_LOSXTn_LS
5 - clk_config_vec(5) - CLK_LOLn_LS
6 - clk_config_vec(6) - CLK_INTn_LS
7 - constant 1
31..8 - constant 0

register 5 0x80010014

not used

register 6 0x80010018

GDM:

3..0 - GDM_link_data_processing:i_status_select

CDM:

31..0 - CDM_link_data_processing:o_error_count

register 7 0x8001001c

GDM:

31..0 - GDM_link_data_processing:o_status_vector

CDM:

31..0 - CDM_link_data_processing:o_error_count

register 246 0x800103d8

VX_RX

register 247 0x800103dc

VX_RX, LEMO_IN, FP_LED, LEMO_OUT, VX_TX

control register 255 0x800103fc

31..28 | ALL | function mux, 4 bits: 0 - outputs driven by control register, 1 - dsvslice GDM production, 2 - dsvslice CDM production, 3, 4, 5 - LVDS loopback, see source code
27 | ALL | lemo_in(4) invert
26 | ALL | lemo_in(3) invert
25 | ALL | lemo_in(2) invert
24 | ALL | lemo_in(1) invert
23 | ALL | lemo_in(4) enable
22 | ALL | lemo_in(3) enable
21 | ALL | lemo_in(2) enable
20 | ALL | lemo_in(1) enable
19 | ALL | software_trigger
18 | ALL | lemo_trigger_enable
17 | ALL | pulser_trigger_enable
16 | ALL | trigger_enable
15 | ALL | gdm_trigger_enable
14
13 | ALL | drive ext_out(2) LEMO output
12 | ALL | drive ext_out(1) LEMO output
11 | ALL | fp_led_out(3) front panel LEDs
10 | ALL | fp_led_out(2)
 9 | ALL | fp_led_out(1)
 8 | ALL | fp_led_out(0)
 7 | ALL | not used
 6 | ALL | vx2_tx_out(2)
 5 | ALL | vx2_tx_out(1)
 4 | ALL | vx2_tx_out(0)
 3 | ALL | not used
 2 | ALL | vx1_tx_out(2)
 1 | ALL | vx1_tx_out(1)
 0 | ALL | vx1_tx_out(0)

Power up sequence

on the CDM:

busybox devmem 0x80010010 32 0x8
sleep 1;
busybox devmem 0x80010010 32 0x0
#/home/dsdaq/si5394-i2c-file /home/dsdaq/CDM_v2.0_IN1_fixed_and_IN2_RX_Recovered_Si5394-RevA-Registers.txt  0 0x6b
/home/dsdaq/si5394-i2c-file /home/dsdaq/CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b
busybox devmem 0x80010004 32 0x1;  # link startup mode reset until phase-aligned
busybox devmem 0x80010000 32 0x5;  # reset link and link status registers
sleep 1;   # just holds reset high
busybox devmem 0x80010000 32 0x0; # lower reset and let links lock
sleep 2;   # give links time to lock
busybox devmem 0x80010008 32;  # read status 0x00000003 means locked and phase aligned
busybox devmem 0x8001000c 32   # read data, 0x1B1C 3CBC lower 16 bits are K-Codes K.28.1 (3C) and K.28.5 (BC) when locked and phase aligned

Build firmware

Build from git clone

git clone git@edev-group.triumf.ca:fw/exp/darkside/gcdm.git
#Makefile change VIVADO_SETTINGS_SCRIPT := /opt/Xilinx/Vivado/2022.1/settings64.sh
#. /opt/Xilinx/Vivado/2022.1/settings64.sh
. /opt/Xilinx/Vivado/2020.2/settings64.sh
make clean
make all_from_scratch
. /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
make petalinux_create
make petalinux_rebuild_new_hw_des
bomb out: The TMPDIR: /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp can't be located on nfs.
mkdir /tmp/build_tmp
rm -rf /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp/
ln -s /tmp/build_tmp /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp
try again
grinds, loads a whole bunch of packages...
finishes with desire to copy things to /tftpboot
make sdcard_cp_to wants to copy files from PetaLinux_GDM_CDM/images/linux/ to SD card

Build branch develop_ko

. /opt/Xilinx/Vivado/2020.2/settings64.sh
. /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
make clean_gdm   # remove gdm build tree
make gdm         # build or rebuild GDM
make copy_gdm    # copy to gdm0
make clean_cdm   # remove cdm build tree
make cdm         # build or rebuild CDM
make copy_cdm    # copy to cdm0 and cdm1

Build in a fresh account

ssh dsdmdev@daq13
mkdir git
cd git
git clone /home/olchansk/git/ds-dm-gcdm
cd ds-dm-gcdm
. /opt/Xilinx/Vivado/2020.2/settings64.sh
. /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
/usr/bin/time make all_from_scratch BOARD=CDM
(sdcard_cp_to bombs out! FIXED!)
#/usr/bin/time make petalinux_build BOARD=CDM
#/usr/bin/time make petalinux_repackage BOARD=CDM

clean GDM:

rm -rf Vivado_GDM_XU8
/usr/bin/time make vivado_create_and_build BOARD=GDM
/usr/bin/time make petalinux_repackage BOARD=GDM

rebuild GDM:

/usr/bin/time make vivado_rebuild_fw_hw_des BOARD=GDM
/usr/bin/time make petalinux_repackage BOARD=GDM

copy to running GDM or CDM:

scp PetaLinux_GDM_CDM/images/linux/BOOT.BIN dsdaq@dsvslice:
#ssh dsdaq@dsvslice
#scp BOOT.BIN root@gdm0:/media/sd-mmcblk1p1/BOOT.BIN
#ssh root@gdm0 -> shutdown -r now
ssh dsdaq@dsvslice scp BOOT.BIN root@xdmn:/media/BOOT/BOOT.BIN
ssh dsdaq@dsvslice ssh root@xdmn shutdown -r now

copy to SD card:

open a root shell
format 16 GB Sd card per above
cd /home/dsdmdev/git/ds-dm-gcdm
make copy
(cannot confirm it worked, DS-DM CDM sn 6 serial console is broken)

build times

CDM 12-june-2023 69aabc1c25130d970bc375aca684bd68849e6685
daq13 AMD-5700G 1688.61user 399.33system 23:28.84elapsed 148%CPU
dsdaqgw AMD-7700 1090.55user 247.34system 16:03.55elapsed 138%CPU

prepare bootable sd card

format the sd card

this only needs to be done once

become root
cd ~olchansk/git/ds-dm-gcdm
use "lsblk" to identify the SD card (should show as 8/16/32 GB block device)/ /dev/sdd in this case
make sdcard_format SDCARD_DEVICE=/dev/sdd
disconnect sd card, reconnect the sd card (to detect new partition tables, etc)

copy CDM boot files

cd /home/dsdmdev/git/ds-dm-gcdm
make copy

copy boot files to the sd card

as root: identify partition labels, run "blkid", should say "BOOT", "rootfs" and "data"
mount

mkdir /media/olchansk/BOOT
mkdir /media/olchansk/rootfs
mkdir /media/olchansk/data
mount -L BOOT /media/olchansk/BOOT
mount -L rootfs /media/olchansk/rootfs
mount -L data /media/olchansk/data
cp PetaLinux_GDM_CDM/images/linux/BOOT.BIN /media/olchansk/BOOT/
cp PetaLinux_GDM_CDM/images/linux/boot.scr /media/olchansk/BOOT/
cp PetaLinux_GDM_CDM/images/linux/image.ub /media/olchansk/BOOT/
umount /media/olchansk/BOOT
umount /media/olchansk/rootfs
umount /media/olchansk/data
eject /dev/sdd

boot messages

Xilinx Zynq MP First Stage Boot Loader 
Release 2020.2   Sep 24 2022  -  13:29:15
NOTICE:  ATF running on XCZU4CG/silicon v4/RTL5.1 at 0xfffea000
NOTICE:  BL31: v2.2(release):xlnx_rebase_v2.2_2020.3
NOTICE:  BL31: Built : 18:02:46, Sep 28 2022


U-Boot 2020.01 (Sep 28 2022 - 18:03:39 +0000)

Model: DarkSide 20k DM
Board: Xilinx ZynqMP
DRAM:  2 GiB
usb dr_mode not found
PMUFW:  v1.1
EL Level:       EL2
Chip ID:        zu4
NAND:  0 MiB
MMC:   mmc@ff160000: 0, mmc@ff170000: 1
In:    serial@ff000000
Out:   serial@ff000000
Err:   serial@ff000000
Bootmode: SD_MODE1
Reset reason:   SOFT 
Net:   
ZYNQ GEM: ff0b0000, mdio bus ff0b0000, phyaddr -1, interface rgmii-id

Warning: ethernet@ff0b0000 (eth0) using random MAC address - d6:62:5f:13:00:44
eth0: ethernet@ff0b0000
ZYNQ GEM: ff0e0000, mdio bus ff0e0000, phyaddr -1, interface rgmii-id
Could not get PHY for eth1: addr -1

Hit any key to stop autoboot:  0 
ZynqMP> 
CTRL-A Z for help | 115200 8N1 | NOR | Minicom 2.7.1 | VT102 | Online 122:2 | ttyACM0

Software

ssh cdm0 # or gdm0
sudo apt install i2c-tools libi2c-dev
git clone https://bitbucket.org/team-ds-dm/ds-dm-software
cd ds-dm-software
make

test_cdm.exe

CDM SFP status: /home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp

DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0xbb2f0ae7
CDM firmware 0xbb2f0ae7
arg 1: [--sfp]
Polling SFP status...
identifier 0x03
connector  0x07
encoding   0x01
wavelength 0x0352 (850 nm)
vendor_name [FINISAR CORP.   ]
vendor_pn   [FTLF8526P3BNL   ]
vendor_rev  [A   ]
vendor_sn   [N3AB9M8         ]
vendor_date [200319  ]
dm_type    0x68
temp 29.0 C
vcc  3.323 V
tx_bias  7.250 mA
tx_power 478.4 uW
rx_power 2.3 uW
SFP good 1, status: temp 30.6 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 818 uW
...

GDM QSFP status: /home/dsdaq/online/ds-dm-software/test_cdm.exe --qsfp3 --qsfp

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --qsfp3 --qsfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x53aee418
CDM firmware 0x53aee418
arg 1: [--qsfp3]
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
 gpio-378 (                    |sysfs               ) out hi 
 gpio-379 (                    |sysfs               ) out hi 
 gpio-381 (                    |sysfs               ) out lo 
 gpio-382 (                    |sysfs               ) out hi 
arg 2: [--qsfp]
Polling QSFP status...
identifier 0x0d
status     0x02
los        0x8f
temp       28.2 C
vcc        3.323 V
rx_power     0.1   0.1   0.1   0.1 uW
tx_bias      7.6   7.6   7.6   0.0 mA
tx_power   792.2 773.8 823.0   0.1 uW
vendor_name [FINISAR CORP    ]
vendor_pn   [FTL410QD4C      ]
vendor_rev  [A ]
wavelength  850
max_temp    70 C
vendor_sn   [X79AC0R         ]
vendor_date [220309  ]
QSFP good 1, status: temp 27.7 C, los 0x8b, tx_bias 7.5 7.6 7.6 0.0 mA, tx_power 792 772 821   0 uW, rx_power   0   0 466   0 uW

CDM link status: /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
- fiber disconnected, no link:

CDM firmware:    0xbb2f0ae7
0x1000 SFP c.c. status: 0x00000031
    CLK_IN_SEL_LS   0x1
    CLK_EXT_SEL_LS  0
    CLK_CLK_RSTn_LS 0
    CLK_LOSXTn_LS   1
    CLK_LOLn_LS     1
    CLK_INTn_LS     0
0x1008 SFP link reset:  0x00000000
0x1010 SFP link status: 0x00000025
    sfp_mod_absent_N       1
    sfp_rx_los_N           0
    link_power_good        1
    rx_link_up             0
    rx_receiving_data      0
    rx_error               1
    rx_lnk_up_and_running  0
    tx_link_up             0
    tx_sending_data        0
    tx_link_up_and_running 0
    link_up_and_running    0
0x1014 SFP link data:   0x466a8187
    rx_data     0x8187
    k28p1_k28p5 0
    rx_ctrl0    0x1
    rx_ctrl1    0x1
    rx_ctrl3    0x1
    tx_state    0x1
    rx_state    0x6
    rx_receiving_data 0
    rx_link_up        0
    rx_error          1
    rx_link_rst       0
0x2000 link test mode: 0x00000200, seconds: 0x00079093, errors: 0xffffffff

- fiber connected, good link:

CDM firmware:    0xbb2f0ae7
0x1000 SFP c.c. status: 0x000000b2
    CLK_IN_SEL_LS   0x2
    CLK_EXT_SEL_LS  0
    CLK_CLK_RSTn_LS 0
    CLK_LOSXTn_LS   1
    CLK_LOLn_LS     1
    CLK_INTn_LS     0
0x1008 SFP link reset:  0x00000000
0x1010 SFP link status: 0x000007dc
    sfp_mod_absent_N       0
    sfp_rx_los_N           0
    link_power_good        1
    rx_link_up             1
    rx_receiving_data      1
    rx_error               0
    rx_lnk_up_and_running  1
    tx_link_up             1
    tx_sending_data        1
    tx_link_up_and_running 1
    link_up_and_running    1
0x1014 SFP link data:   0x35c02774
    rx_data     0x2774
    k28p1_k28p5 0
    rx_ctrl0    0x0
    rx_ctrl1    0x0
    rx_ctrl3    0x0
    tx_state    0x3
    rx_state    0x5
    rx_receiving_data 1
    rx_link_up        1
    rx_error          0
    rx_link_rst       0
0x2000 link test mode: 0x00000200, seconds: 0x00078a8c, errors: 0x00000000

GDM link status, 1 link connected, no errors: /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link

GDM firmware:    0x6b2ee010
0x1014: 0x00000008, 18: 0x00000008, 1C: 0x00000000, 24: 0x00000fff
0x2000: 0x00000200, time: 0x00078aa4, errors:
0xffffffff 0xffffffff 0xffffffff
0x00000000 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff

dsvslice integration

VX setup

general
- Start acq from user code = y, all others = n (as of Mar 2023: will start when first trigger received)
- Use NIM IO = y
- Use external clock = y
- LVDS quartet is input = n, y, n, y
- LVDS quartet mode = User, User, User, User

trigger from front panel NIM:
- Trigger on external signal = y, all others = n
- connect CDM EXT_OUT(2) to VX "TrigIn"

trigger from LVDS "Sync" mode
- Trigger on LVDS Sync signal = y, all others = n
- LVDS quartet mode = User, Sync, User, User

trigger from LVDS "User" mode
- Trigger on LVDS pair 12 signal = y, all others = n
- LVDS quartet is input = n, y, n, y
- LVDS quartet mode = User, User, User, User

GDM setup

GDM is gdm0
set inputs to NIM mode
set outputs to TTL mode (this GDM has wrong NIM output circuit)
use top QSFP slot, connect split cable 1 into CDM01, cable 2 into CDM02
connect non-inverted NIM trigger signal to top-LEMO-left EXT_IN_LV(1)
connect non-inverted NIM TSM signal to top-LEME-right EXT_IN_LV(2)
GDM LEDs: TRIG, TSM, trigger enabled, trigger_out
GDM LEMO_OUT: trigger, trigger

CDM setup

set CDM LEMO inputs to NIM
set CDM LEMO outputs to NIM
CDM01 is cdm0
CDM02 is cdm1
connect GDM fiber links to SFP port
connect 1st VX port of CDM01 to VX1
connect 1st VX port of CDM02 to VX2
connect LEMO EXT_OUT(2) to VX "TrigIn", CDM01 to VX1, CDM02 to VX2
power up
CDM LEDs: GDM TRIG, GDM TSM, trigger enabled, trigger out
CDM LEMO_OUT: gdm_trg, trigger

After power up

start the CDM frontend from the MIDAS "Programs" page. To start manually, see the Start Command on the Programs page.
CDM frontend should enable the VX clock, disable the trigger
from the MIDAS status page, goto the CDM page
in the CDM01, CDM02 data table, the 2nd number should read 0x35c08008, if it does not and the last 4 digits randomly change, reset the GDM links:
- in order GDM, CDM01, CDM02, press "reset1" of each board, then press "reset4", then press "reset0", 2nd number should read as above. if it does not, STOP HERE.
start a run
CDM frontend will enable the trigger
GDM frontend will enable the trigger
LEDs on the GDM should flash, LEDs on the CDM should flash, TrigIn and TrigOut of the VX should flash
stop a run
GDM frontend will disable the trigger
CDM frontend will disable the trigger

Phase measurement

pip3 install matplotlib
pip3 install scipy
export PYTHONPATH=$HOME/packages/midas/python
#git clone https://github.com/J033X071C/PhaseMeasurement
git clone https://bitbucket.org/team-ds-dm/phasemeasurement.git
cd phasemeasurement
python3 ./phaseMeasurement.py --help

daq00:PhaseMeasurement$ python3 ./phaseMeasurement.py --help
usage: phaseMeasurement.py [-h] fileName numberEvents numberVX sizeEvents stopEvent minHist maxHist numberBin writeToTXT saveAsPDF

Read data from midas file (in .lz4 format) to calculate phase between the clock of VX1 and VX2

positional arguments:
  fileName      Name of the file we want to read data from (Example: run00389.mid.lz4)
  numberEvents  Number of events recorded in the file
  numberVX      Number of VX used in this run (usually 2...)
  sizeEvents    Number of points per event
  stopEvent     Number of events you want to go through to calculate phase
  minHist       Minimal value for the x axis of the phase measurement histogram (in ns)
  maxHist       Maximal value for the x axis of the phase measurement histogram (in ns)
  numberBin     Number of bins wanted for the generated histogram
  writeToTXT    Write argument as yes to generate text file with results of calculation
  saveAsPDF     Save generated plots to PDF files

optional arguments:
  -h, --help    show this help message and exit
daq00:PhaseMeasurement$

try an old file with
python3 ./phaseMeasurement.py run00877.mid.lz4 10000 2 10000 500 -20 20 81 yes yes

bin size = 0.494 ns
num_events =  347
mean = -1.705 ns
rms = 3.087 ns
mean_error = 0.166 ns
centroid = -1.706 ns.
width (sigma) = 0.363 ns.
error on the centroid = 0.016558 ns.

ls -l *.txt *.pdf

dsdaq@dsvslice:~/online/PhaseMeasurement$ ls -l *.txt *.pdf
-rw-rw-r-- 1 dsdaq dsdaq 64728 Dec 14 16:56 run00877.mid.lz4_Plots.pdf
-rw-rw-r-- 1 dsdaq dsdaq   274 Dec 14 16:56 run00877.mid.lz4.txt
dsdaq@dsvslice:~/online/PhaseMeasurement$

Standalone link test

CDM: program clock chip
busybox devmem 0x80011000 32 0x8
busybox devmem 0x80011000 32 0x0
/home/dsdaq/si5394-i2c-file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt  0 0x6b

GDM, CDM: link reset
busybox devmem 0x80011008 32 1

GDM, CDM: release reset
busybox devmem 0x80011008 32 0

CDM: link status (NOTE: SFP LOS and mod_absent are swapped!!!)
busybox devmem 0x80011010 32
0x00000024 <- fiber plugged
0x00000025 <- fiber unplugged
0x00000027 <- SFP unplugged
0x000007DC <- successful link with GDM

CDM: link state machine and data
busybox devmem 0x80011014 32
0x35C06FF6

CDM: set link to counting mode
busybox devmem 0x80012000 32 0x101
busybox devmem 0x80012000 32 0x100

CDM: time counter and error counter
root@cdm1:~# busybox devmem 0x80012000 32
0x00000100 <--- link mode
root@cdm1:~# busybox devmem 0x80012004 32
0x0000058C <--- seconds counter
root@cdm1:~# busybox devmem 0x80012004 32
0x0000058D
root@cdm1:~# busybox devmem 0x80012008 32
0x00000000 <--- error counter

GDM: no link
root@gdm0:~# busybox devmem 0x80011014 32
0x00000000
root@gdm0:~# busybox devmem 0x80011018 32
0x00000000
root@gdm0:~# busybox devmem 0x8001101c 32
0x00000000
root@gdm0:~# busybox devmem 0x80011024 32
0x00000FFF
root@gdm0:~# 

GDM: good link channel 10, counting mode
root@gdm0:~# busybox devmem 0x80012000 32 0x101
root@gdm0:~# busybox devmem 0x80012000 32 0x100
root@gdm0:~# busybox devmem 0x80012008 32
0x3A8B68C2
root@gdm0:~# busybox devmem 0x80012008 32
0x42E03BEF
root@gdm0:~# busybox devmem 0x8001200c 32
0xDA090972
root@gdm0:~# busybox devmem 0x8001200c 32
0xDE6F22E9
root@gdm0:~# busybox devmem 0x80012019 32
Bus error
root@gdm0:~# busybox devmem 0x80012010 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012014 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012018 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x8001201c 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012020 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012024 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012028 32
0x00000000
root@gdm0:~# busybox devmem 0x8001202c 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012030 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012034 32
0xFFFFFFFF
root@gdm0:~#

mapping of link channels:

qsfp0 lane0 - 0x0100 - link 8
qsfp0 lane1 - 0x0200 - link 9
qsfp0 lane2 - 0x0400 - link 10
qsfp0 lane3 - n/c
qsfp1 lane0 - 0x0800 - link 11
qsfp1 lane1 - 0x0010 - link 4
qsfp1 lane2 - 0x0020 - link 5
qsfp1 lane3 - n/c
qsfp2 lane0 - 0x0040 - link 6
qsfp2 lane1 - 0x0080 - link 7
qsfp2 lane2 - 0x0001 - link 0
qsfp2 lane3 - n/c
qsfp3 lane0 - 0x0002 - link 1
qsfp3 lane1 - 0x0004 - link 2
qsfp3 lane2 - 0x0008 - link 3
qsfp3 lane3 - n/c

script to start the test with 2 CDMs:

ssh dsdaq@dsvslice
ssh root@gdm0 busybox devmem 0x80011008 32 1
ssh root@cdm0 busybox devmem 0x80011008 32 1
ssh root@cdm1 busybox devmem 0x80011008 32 1
ssh root@gdm0 busybox devmem 0x80011008 32 0
ssh root@cdm1 busybox devmem 0x80011008 32 0
ssh root@cdm0 busybox devmem 0x80011008 32 0
ssh root@gdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
ssh root@cdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
ssh root@cdm1 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2

DS-20K DAQ

Overview

DS-DM, GDM and CDM are key parts of the DS-20K DAQ system:

common clock distribution from external clock (atomic clock, GPS) to GDM to per-quadrant CDMs to VX digitizers
common trigger distribution from GDM internal algorithm or external input to all VX digitizers
run control: GDM, CDM, VX all start recording data at the same time (clock and timestamp reset)
collection of trigger data from VX digitizers to per-quadrant CDMs to GDM

Deliverables

hardware and firmware for GDM to CDM clock distribution
hardware and firmware for CDM to VX clock distribution
hardware and firmware for GDM external clock input (atomic clock or GPS)
hardware and firmware for CDM and VX serial communications (VX LVDS I/O connector)
firmware for run control (timestamp reset and sync): GDM to CDM to VX
firmware for common trigger distribution: GDM to CDM to VX
firmware for trigger data flow: VX to CDM to GDM
firmware for busy control: VX to CDM to GDM back to CDM to VX
firmware for flow control: FEP to GDM MIDAS frontend to GDM to CDM to VX
GDM MIDAS frontend: clock selector and monitoring, trigger and run control, busy and flow control, GDM housekeeping
CDM MIDAS frontend: clock monitoring, CDM housekeeping

specific performance:

GDM external clock: 10 MHz GPS clock
GDM to CDM fiber link:
- clock XXX MHz
- link data rate: XXX Gbit/sec
- CDM recovered clock: XXX MHz
- CDM recovered clock jitter: XXX ns
- phase alignment between CDMs: XXX ns
- phase alignment between CDMs persists across reboots, power cycles, firmware updates
- phase alignment between CDMs should be easy to measure
- phase alignment between CDMs should be easy to recalibrate if hardware parts are replaced (DS-DM boards, fiber transceivers, fiber cables, etc)
- data packet bandwidth: XXX Mbytes/sec
- data packet latency: XXX clocks
- data packet skew between CDMs: XXX clocks
CDM to VX clock:
- clock: XXX MHz
- jitter, all CDM clock outputs: XXX MHz
- phase alignment between all CDM clock outputs: XXX ns
CDM to VX trigger:
- TBD (use the VX "sync" input or VX LVDS I/O line or VX serial link packet)
CDM to VX serial link:
- clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
- bit rate: XXX bits/sec
- latency: XXX link clocks
- maximum skew between VXes: XXX ns
VX to CDM serial link:
- clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
- bit rate: XXX bits/sec
- latency: XXX link clocks
- maximum skew between VXes: XXX ns
timestamp reset:
- maximum skew between VXes: XXX ns
busy round trip time: XXX ns (VX to CDM to GDM back to CDM to VX)
flow control latency: XXX ns (FEP software to GDM MIDAS frontend to GDM to CDM to VX)

Technical risk items

this refers to unexpected behaviour and performance of system components, causes big difficulty in implementing the system, prevents delivery of deliverables, and prevents or negatively affects operation of the DS-20K DAQ or of the whole experiment.

(14-sep-2022, list is not sorted by any criterial: severity, probability, ease of investigation)

(stability of course is long term stability, across hours, days, weeks, months, years)

stability of Enclustra FPGA modules (crashes/year, failures to boot/year, flash corruption/year)
stability of GDM external clock PLL (lock loss/year)
stability of CDM recovered clock (lock loss/year, unexpected phase drifts, etc)
unexpected failures or bit error rates in GDM-CDM fiber links
stability of CDM VX clock outputs (stability of clock cleaner chip)
stability of VX internal clock distribution (VX PLL lock loss events)
stability of VX CAEN base firmware (different versions of CAEN base firmware have different clock distribution behaviour)
strange things in CAEN base firmware (unexpected clocking of LVDS I/O, unexpected phase shifts between clocks, etc)
DS-DM and VX hardware problems (incompatible LVSD I/O, incompatible clock signals, etc)

Milestones

(14-sep-2022: at this stage of the project, priority must be given to identifying and retiring (so called) technical risk factors. it is not good to build the complete system only to discover that (for example) some Enclustra FPGA modules require 5 attempts to boot and erase their flash memory contents once a month. Both example are real-life actual problems that caused big difficulties in GRIFFIN/TIGRESS and ALPHA-g experiments).

Development and testing milestones in time reversed order:

full DAQ data challenge: all VXes, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
one quadrant data challenge: 1 VME crate of VX, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
vertical slice data challenge: 1 VME crate, 2 VX, 2 CDM (1 VX per CDM), GDM, etc operate as designed
GDM-CDM link finalized (data rate frozen, data packet format frozen, data content permitted to change)
CDM-VX serial link finalized (data rate frozen, data packet format frozen, data content permitted to change)
run control (timestamp reset) and trigger distribution design agreed upon, frozen (list of possible triggers permitted to change)
VX to CDM to GDM data flow design agreed upon, frozen (data contents permitted to change)
major technical risk items retired (all hardware and firmware is working as expecred without mysteries and surprises, all problems are identified, investigated, resolved, solutions tested)
stable operation of CDM-VX serial links in vertical slice system
stable operation of GDM to CDM clock in vertical slice system
stable operation of CDM to VX clock in vertical slice system
vertical slice system assembled (1 VME crate, 2 VX, 2 CDM, 1 GDM, network, FEP, TSP, MIDAS)

ZZZ

@@ Line 1,251: / Line 1,251: @@
 = Firmware registers =
+AAA
+= Firmware registers branch develop_ko =
 == Register map ==

DS-DM: Difference between revisions

Revision as of 12:39, 31 July 2023

DS-DM

Links

Onboard hardware

Buttons, jumpers and switches

Front panel

VX adapter board

split-cable connection

one-to-one connection

Board schematics

FPGA MGT blocks

Clock distribution

I2C bus

I2C clock builder connection

GDM MGT configuration

Clock path

Clock domains

GPS

GDM

CDM

VX

Board test plan

Checklist for newly build boards

Serial console

i2c

U23

SFP

QSFP

ethernet mac eeprom

read ethernet mac address from i2c

read ethernet mac address from i2c (SHOULD WORK)

read ethernet mac address from i2c (DOES NOT WORK)

enable VX clock

clock chip configuration

clock chip monitoring

arm and linux benchmark

Install Xilinx tools

Petalinux

JTAG server

Firmware registers

Firmware registers branch develop_ko

Register map

register 0 0x80010000

register 1 0x80010004

register 2 0x80010008

register 3 0x8001000c

register 4 0x80010010

register 5 0x80010014

register 6 0x80010018

register 7 0x8001001c

register 246 0x800103d8

register 247 0x800103dc

control register 255 0x800103fc

Power up sequence

Build firmware

Build from git clone

Build branch develop_ko

Build in a fresh account

build times

prepare bootable sd card

format the sd card

copy CDM boot files

copy boot files to the sd card

boot messages

Software

test_cdm.exe

dsvslice integration

VX setup

GDM setup

CDM setup

After power up

Phase measurement

Standalone link test

DS-20K DAQ

Overview

Deliverables

Technical risk items

Milestones

ZZZ