DS-DM
Jump to navigation
Jump to search
DS-DM
DarkSide-20k Global and Crate Data Manager board (GDM and CDM).
Global Data Manager (GDM):
- clock distribution to CDM boards (including GPS/atomic clock source)
- collection of trigger data from CDM boards, processing and distribution of trigger decision to CDM boards
- run control
- integration with GPS 10MHz and 1pps clocks and GPS/IRIG date and time information
Crate Data Manager (CDM):
- clock distribution from GDM to CAEN VX digitizers
- receive trigger data from CAEN VX digitizers
- send trigger data to GDM
- run control and dead time control
Links
- https://dsvslice.triumf.ca/vx_napoli - DS vertical slice at TRIUMF
- https://edev-group.triumf.ca/fw/exp/darkside/gcdm - git repository, DS-DM firmware
- https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0 - git repository, DS-DM board
- https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0/-/blob/master/Altium/Project%20Outputs%20for%20DS-DM-Rev0/SCH-DS-xDM-Rev0.PDF - DS-DM schematics
- https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0/-/blob/master/Testing/Clk3_XO_125_to_fpgaIN_recoveredMGTclk_to_IN2_Si5394-RevA-Project.slabtimeproj
- https://docs.xilinx.com/v/u/en-US/ds593 - Xilinx Platform Cable USB II
- https://www.enclustra.com/en/products/system-on-chip-modules/mercury-xu8/ - Enclustra
- https://bitbucket.org/team-ds-dm/ds-dm-software - git repository, DS-DM MIDAS frontend
- https://bitbucket.org/team-ds-dm/phasemeasurement - VX Phase measurement script
- https://ladd00.triumf.ca/daqinv/frontend/list/178 - inventory database
Onboard hardware
- jtag chain: arm_dap_0 0x5BA00477, xczu4_1 0x04721093
- Eclustra Mercury+ XU8 module: ME-XU8-4CG-1E-D11E-R2.1
- Xilinx® Zynq Ultrascale+™ MPSoC XCZU4CG-1FBVB900E
- DDR4 ECC SDRAM (PS) 2 GB
- DDR4 SDRAM (PL) 1GB
- ethernet mac chip: AT24MAC402-SSHM-T ("602" chip is wrong)
- USB UART for Enclustra serial console, micro-USB, 115200n8
- clock chip: SI5394A-A-GM and oscillator CS-044-054.0M (54 MHz)
- U23 3.3V current meter and thermometer
- LEDs:
- LED_FP A/B/C/D 0/1/2/3
- led1 - 3V3_SW_ON, SOM_POWER_GOOD - Enclustra FPGA module 3.3V power is good
- led2 - LTM4624 PGOOD
- led3 - FPGA_DONE - FPGA has booted
- led4 - TP-S-1, PCLK_P
- led5 - TP-S-2, PCLK_N
- LEMO connectors (top to bottom)
- J4 - input (NIM/TTL) (EXT_IN_LV(1), EXT_IN_LV(2))
- J5 - input (NIM/TTL) (EXT_IN_LV(3), EXT_IN_LV(4))
- J6 - external clock (GPS 10MHz and PPS)
- J7 - output (NIM/TTL) (EXT_OUT(1), EXT_OUT(2))
- SMA connectors
- J9, J10 - CLK_CCA from U6 C.C.
- J11, J12 - CLK_TP0
- RJ45 ethernet connector (100mbit: green light on, yellow light flashes when there is traffic)
- SFP connector (SFP is FTLF8526P3BNL, 6 Gbit/sec, 850 nm, 300m 50/125um OM3 MMF)
- 4 QSFP connectors (GDM)
- 6 VX connectors (CDM)
Buttons, jumpers and switches
Buttons:
- PB1 - HRST - reboot FPGA (power-on reset)
- PB2 - SRST - (SRSTn) - reboot ARM CPU
Switches:
- SW1 - boot mode BM0, BM1 [-->]
- SW2 - LEMO output NIM<->TTL
- SW3 - LEMO input 1 and 2 NIM/TTL
- SW4 - LEMO input 2 and 4 NIM/TTL
- SW5 - LEMO clock input NIM/TTL
- SW6 - serial console select. [PS<--PL] PS is ARM CPU, PL is FPGA.
Front panel
| top | | LED-FP1 | LED_FP(0,1,2,3) | | SFP J??? | | LEMO J4-LEFT, J4-RIGHT | EXT_IN_LV(1), EXT_IN_LV(2) | LEMO J5-LEFT, J5-RIGHT | EXT_IN_LV(3), EXT_IN_LV(4) | LEMO J6-LEFT, J6-RIGHT | CLK_EXT1, CLK_EXT0 (125 MHz only) | LEMO J7-LEFT, J7-RIGHT | EXT_OUT(1), EXT_OUT(2) | | J-VX-1 | J-VX-2 or QSFP-1 | J-VX-3 or QSFP-2 | J-VX-4 or QSFP-3 | J-VX-5 or QSFP-4 | J-VX-6 | | RJ45 J3 ethernet | | bottom
VX adapter board
LVDS I/O connector
34 pin connector: 0|:::: :::: :::: :::: :|15,16 n/c
split-cable connection
0 .. 7 -> N/C 8 -> VX_RX(3) 9 -> VX_RX(2) 10 -> VX_RX(1) 11 -> VX_RX(0) 12 <- VX_TX(0) 13 <- VX_TX(1) 14 <- VX_TX(2) (set by jumper routed here or to VX CLKIN SYNC) 15 <- n/c (CLK routed to VX CLKIN CLK) 16 - n/c
one-to-one connection
0 -> VX2_RX(3) 1 -> VX2_RX(2) 2 -> VX2_RX(0) 3 -> VX2_RX(1) 4 <- CLK 5 <- VX2_TX(0) 6 <- VX2_TX(1) 7 <- VX2_TX(2) 8 -> VX1_RX(3) 9 -> VX1_RX(2) 10 -> VX1_RX(1) 11 -> VX1_RX(0) 12 <- VX1_TX(0) 13 <- VX1_TX(1) 14 <- VX1_TX(2) (set by jumper routed here or to VX CLKIN SYNC) 15 <- n/c (CLK routed to VX CLKIN CLK) 16 - n/c
Board schematics
- File:SCH-DS-xDM-Rev0.PDF
- note: FPGA pin annotations ("IO", "SCLK", "PCLK", etc) on the schematics are bogus, instead, trace them to the FPGA pins.
- note: ENC A is J800, ENC B is J801, ENC C is J900 (schematic name to enclustra name)
- note: Enclustra special pins: "GC" is "clock capable", "HDGC" is "clock capable", "MGTREFCLK" is MGT reference clocks.
- board modifications:
- ethernet mac chip
- NIM output (no U15, etc)
- RJ45 wrong pinout (board mod or special ethernet cable)
- 125 MHz clock mods (TBW)
- disconnect QSFP0_SEL from SFP_RS0 and QSFP1_SEL from SFP_RS1, these signals are not used by modern SFPs
- provide SFP i2c modsel to allow SFP and QSFP at the same time on address 0x50
FPGA MGT blocks
* SFP - ENC B45,B47 and B48,B50 - FPGA D5,D6 MGTHTX0_D and D1,D2 MGTHRX0_D * QSFP0 TX0 - ENC C13,C17 - FPGA H5,H6 MGTHTX0_C * QSFP0 TX1 - ENC C21,C25 - FPGA G7,G8 MGTHTX1_C * QSFP0 TX2 - ENC C29,C23 - FPGA F5,F6 MGTHTX2_C * QSFP1 TX0 - ENC C37,C41 - FPGA E7,E8 MGTHTX3_C * QSFP1 TX1 - ENC C45,C47 - FPGA P5,P6 MGTHTX0_B * QSFP1 TX2 - ENC C51,C53 - FPGA M5,M6 MGTHTX1_B * QSFP2 TX0 - ENC C57,C59 - FPGA L3,L4 MGTHRX2_B * QSFP2 TX1 - ENC C63,C65 - FPGA K5,K6 MGTHTX3_B * QSFP2 TX2 - ENC C75,C77 - FPGA W3,W4 MGTHTX0_A * QSFP3 TX0 - ENC C79,C81 - FPGA V5,V6 MGTHTX1_A * QSFP3 TX1 - ENC C85,C87 - FPGA T5,T6 MGTHRX2_A * QSFP3 TX2 - ENC C89,C91 - FPGA R3,R4 MGTHTX3_A
Clock distribution
Simplified:
- 125 MHz osc -> CLK_XO_125 -> MGTREFCLK0_A -> not used
- 125 MHz osc -> CLK3_XO_125 -> MGTREFCLK1_B -> SFP RX ref clock, QSFP RX and TX ref clock (this is not final design!)
- (disconnected) 125 MHz osc -> CLK2_XO_125 -> FPGA AG8,AH8 (GC)
- 125 MHz osc -> C.C. in1
- C.C. in0 <- CLK_EXT1 (10 MHz GPS clock)
- C.C. in1 <- 125 MHz osc
- (disconnected) C.C. in2 <- CLK_CC_IN <- MGTREFCLK0_D <- SFP RX clock (cannot be used because of uncontrollable phase)
- C.C. in2 <- CLK2_XO_125 <- FPGA AG8,AH8 (GC) <- SFP RX recovered 125 MHz clock
- C.C. in3 <- CLK_FB
- C.C. 125 MHz -> CLK_CC_OUT0 -> MGTREFCLK0_B -> QSFP RX and TX ref clock (final design)
- C.C. 125 MHz -> CLK_CC_OUT1 -> MGTREFCLK1_D -> SFP TX clock
- C.C. 62.5 MHz -> VX clock fanout
proposed changes:
- add C.C. 125 MHz -> new CLK_CC_OUT2 (old CLK2_XO_125) -> FPGA AG8,AH8 (GC)
- change C.C. in2 <- new CLK_CC_IN <- FPGA AK8,AK9 (non-GC)
Complete:
125 MHz oscillator - U5 fan out - q0 -> CLK_XO_125 -> ENC C72,C74 -> FPGA R7,R8 MGTREFCLK0_A (not used) q1 -> U6 C.C. in1 q2 -> CLK3_XO_125 -> ENC C7,C9 -> FPGA J7,J8 MGTREFCLK1_B -> SFP RX reference clock, QSFP RX and TX reference clocks (not final design!) q3 -> disconnected on the board, was CLK2_XO_125 -> ENC C151,C153 -> FPGA AG8,AH8 (GC) U6 C.C (clock cleaner) - in0 <- CLK_EXT1 (presumably GPS 10 MHz ref clock) in1 <- 125 MHz oscillator via U5 in2 <- (was: CLK_CC_IN <- ENC B10,B12 <- FPGA D9,D10 MGTREFCLK0_D <- SFP RX clock, 125 MHz) in2 <- CLK2_XO_125 <- ENC C151,B153 <- FPGA AG8,AH8 (GC) <- mgt_link_data_to_processing.rx_data_clk (SFP RX data clock, 125 MHz) in3 <- CLK_FB out0 -> CLK_CCA -> U12 (125 MHz) out1 -> CLK_CCB -> VX1..6 (62.5 MHz) out2 -> CLK_CCC -> VX7..12 (62.5 MHz) out3 -> CLK_FB into in3 CLK_CCA -> U12 (125 MHz fan out) -> Q0 -> not used Q1 -> CLK_CC_OUT0 -> ENC C3-5 -> FPGA L7,L8 MGTREFCLK0_B -> QSFP RX and TX reference clocks (final design) Q2 -> CLK_CC_OUT1 -> ENC B3-5 -> FPGA B9,B10 MGTREFCLK1_D -> SFP TX clock Q3 -> not used Q4 -> not used Q5 -> SMA J9/J10 CLK_TP0 - FPGA N7,N8 MGTREFCLK1_A - ENC C69,C71 - DS-DM SMA J11, J12 (NOT IN CDM PROJECT) Notes: * CLK_XO_125 (125 MHz osc) is not used * 62.5 MHz VX clock does not go into the FPGA * CLK_EXT0 going to in1 of U5 cannot be used. only permitted frequency is 125 MHz (it drives the MGT reference clocks) and is this frequency is too high for LEMO cables and connectors Proposed modifications: - CLK_CC_IN: from FPGA output pin (ENC C142,C144 - FPGA AK8,AK9) to clock cleaner in2 (instead of CLK2_XO_125 pins) - CLK_CCA -> U12 -> currently unused out3 -> CLK2_XO_125 FPGA pins - repurpose CLK_EXT0 at GPS 1pps/IRIG input to FPGA
I2C bus
- I2C_SCL is J-ENC-A pin 111 I2C_SCL is FPGA I2C_SCL_PL AB13 (IO_L1N_TOL1D_64) and I2C_SCL_PS F18 (PS_MIO10)
- I2C_SDA is J-ENC-A pin 113 I2C_SDA is FPGA I2C_SDA_PL AH13 (IO_L7N_T1L1Q_AD13N_64) and I2C_SDA_PS G18 (PS_MIO11)
- XU8 secure EEPROM ATSHA204A at 0x64, this is 0110'010X -> linux _011'0010 is 0x32. (but responds to scan and read at 0x33)
- U4 ethernet mac chip, EEPROM at 1010 A2 A1 A0 X and MAC/serial_no at 1011 A2 A1 A0 X. A0=VCC, A1=VCC, A2=GND -> linux _101'0011 and _101'1011 is 0x53 and 0x5B.
- U6 clock chip, address 1101 0 A1 A0 X. A1=VCC, A0=N/C (internal pull-up) -> linux _110'1011 is 0x6b
- U23 voltmeter at 10011 ADR1 ADR0. ADR0=GND, ADR1=VCC -> linux _100'1110 is 0x4e
- SFP, address 1010000X -> linux _101'0000 is 0x50. additional SFP data at 0x51
- QSFP0, QSFP1, QSFP2, QSFP3 (QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL)
I2C clock builder connection
- use Silicon Labs USB "Clock builder pro field programmer", www.silabs.com/CBProgrammer
- connect rainbow jumper cable pins:
- black - 1-GND to GND on DS-DM
- white - 3-SCLK to SCL on the DS-DM
- grey - 7-SDA_SDIO to SDA on the DS-DM
- power up the DS-DM
- plug USB programmer into Windows laptop
- on Windows, run "ClockBuilder Pro"
- it should report "Field programmer detected", press "EVB GUI"
- in EVB GUI, press "Config", set I2C address 0x6B
- press "Scan", it should find Si5394A-A-GM
- select the "Status" tab, should see real-time status of clock chip
GDM MGT configuration
- TX configuration:
- GDM MGT transceivers are configured as "multilane" TX and RX.
- there is 12 TXes ("lanes")
- MGT reference 125 MHz clock goes into [2:0]gtreclk00_in and [11:0]gtrefclk0_in
- one MGT is designated as "master"
- PLL of master MGT converts reference clock into common TX clock and common tx_user_clk2 which becomes tx_data_clk
- common tx_user_clk2 aka tx_data_clk goes into all TXes and clocks tx_data.
- tx_user_clk2 aka tx_data_clk is 125 MHz but not same phase as MGT reference clock.
- RX interim configuration:
- there is 12 RXes ("lanes")
- each RX produces it's own recovered RX clock
- "multilane" configuration assumes all RX recovered clocks run at the same frequency (TX on the other end are driven by common TX clock, see above), but have different phase
- one RX recovered clock is designated as "master" (rx_user_clk2 aka rx_data_clk) and a phase-matching fifo/buffer is used to bring rx_data from all 12 RXes to this common rx_data_clk
- this works because each CDM SFP TX runs on the SFP RX recovered clock which is frequency-locked with the GDM QSFP TX clock.
- RX final configuration:
- MGTs permit using the common TX clock (tx_user_clk2 aka tx_data_clk) as the common rx_data_clk (they are frequency locked through the CDM).
- this permits use of tx_data_clk as the main clock domain in the GDM and removes the need to bring rx_data into the tx_data_clk domain (actually this is done in the MGT RX phase matching fifo/buffer).
Clock path
10 MHz ext clock or GDM 125 MHz oscillator -> GDM QSFP MGT reference clock 125 MHz -> MGT PLL -> tx_data_clk 125 MHz (GDM main clock domain) and TX bit clock 2.5 GHz -> GDM QSFP optic transmitter -> CDM SFP optic receiver -> CDM SFP MGT, RX reference clock is CDM 125 MHz oscillator -> MGT RX recovered clock 125 MHz (CDM main clock domain) -> CC_CLK_IN -> CDM C.C. -> CC_CLK_OUT1 -> CDM SFP TX reference clock 125 MHz -> MGT PLL -> tx_data_clk 125 MHz and TX bit clock 2.5 GHz -> (tx_data phase matching fifo from CDM main clock domain to tx_data_clk) -> CDM SFP optic transmitter -> GDM QSFP RX optic receiver (12x) -> GDM QSFP MGT (RX reference clock is same as TX reference clock) -> MGT RX recovered clock (12x recovered clocks) -> in multi-lane configuration, one of them is the "master" recovered clock rx_data_clk -> (rx_data phase matching fifo from rx_data_clk to GDM main clock domain)
Clock domains
GPS
- no GPS : GDM runs from internal 125 MHz oscillator
- external 10 MHz clock : GDM runs from external 10 MHz clock and optional 1pps signal (use VME-NIMIO32 NIM outputs)
- GPS receiver : GDM runs from GPS 10 MHz clock and GPS IRIG serial data
- LNGS GPS:
- provides 1pps and serial data over fiber from GPS receiver (master). LNGS xxx box is not used.
- serial data goes to GDM, decoded, 1pps signal extracted, goes to LEMO output
- 1pps from GDM is used to train the Rubidium clock which provides a 10 MHz clock
- 10 MHz output from Rubidium clock goes to GDM 10 MHz external clock input
GDM
- AXI clock (100 MHz) - AXI registers
- 125 MHz oscillator - to clock cleaner
- 10 MHz external clock LEMO input - to clock cleaner
- FPGA 125 MHz clock CLK_CC_IN - to clock cleaner - not used (could be used for cascaded GDMs)
- 125 MHz oscillator - CLK3_XO_125 - mgt_b_ref_clk QSFP MGT reference clock (interim GDM design)
- clock cleaner output 125 MHz fanout:
- CLK_CC_OUT0 - QSFP MGT reference clock (final design)
- CLK_CC_OUT1 - not used (CDM SFP reference clock)
- CLK_CC_OUT2 - not used
- QSFP MGT TX data clock 125 MHz tx_data_clk (main clock domain)
- QSFP TX data
- QSFP RX data (in final design, rx_data_clk is same as tx_data_clk. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk == tx_data_clk)
- ds20k block
- (in interim design, rx_data_clk is disconnected from tx_data_clk. rx_data_clk is the "multilane master clock" which is one of the 12x RX recovered clocks. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk)
note: all these clocks are frequency locked to 125 MHz
CDM
- AXI clock (100 MHz) - AXI registers
- 10 MHz external clock LEMO input - to clock cleaner (not used)
- 125 MHz oscillator to fanout
- to clock cleaner
- to SFP MGT RX reference clock: CLK3_XO_125 to mgt_rx_ref_clk to gtrefclk01_in
- SFP MGT RX recovered clock 125 MHz
- MGT PLL to MGT rx_user_clk2 aka rx_data_clk
- MGT rxrecclkout_out to mgt_rx_rec_clk to CLK_CC_IN to C.C.
- SFP MGT rx_user_clk2 aka rx_data_clk (250 MHz/8 bit, 125 MHz/16 bit, 62.5 MHz/32 bit data) (main clock domain)
- SFP RX data
- ds20k block
- VX TX clock PLLs
- VX RX clock PLLs
- C.C. fan out
- 62.5 MHz VX clocks (12x)
- CLK_CC_OUT0 (not used, GDM QSFP MGT reference clock)
- CLK_CC_OUT1 125 MHz to mgt_tx_ref_clk to gtrefclk00_in to SFP MGT TX reference clock
- CLK_CC_OUT2 (not used)
- SFP MGT tx_user_clk2 aka tx_data_clk
- SFP TX data
- TX data phase matching fifo from main clock domain to tx_data_clk
- VX TX clock PLLs
- 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
- VX TX data phase matching from main clock domain to VX TX clock (12 total)
- VX TX serializer
- VX TX LVDS transmitter
- VX RX clock PLLs
- 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
- VX RX LVDS receivers (12 total)
- VX RX deserializers (12 total)
- VX TX data phase matching from VX RX clock to main clock domain
VX
- everything runs on the VX main 125 MHz clock
- correct phase of VX to CDM LVDS data is adjusted by scan of CDM VX RX PLL clock phase (VX to CDM link is now established)
- correct phase of CDM to VX LVDS data is adjusted by scan of CDM VX RX PLL clock phase (link is established after VX to CDM idle data pattern changes from "VX RX data bad" to "good".
- after good phases are found by scan, they are not expected to change unless cables are changed, CDM and VX modules are changed or CDM firmware is rebuilt. (rebuild of VX firmware should not affect LVDS data phase).
- if there is excessive link errors, phase scan must be repeated.
Board test plan
To test:
- Enclustra FPGA board
- SFP port - SFP_RS0/RS1 connected to QSFP0_SEL/1_SEL is wrong? our Finisar SFP says RS0, RS1 N/C, so probably okey.
- SW5 CLK_EXT1 NIM works. TTL needs to be tested. flipping SW5 CLK_EXT0 side from NIM to TTL makes CLK_EXT1 go LOS and OOF in the clock chip. R23 and R53 should be removed?
- BOOT_MODE 0 and 1
Partial:
- U23 3.3V current meter and thermometer. V1,V2 is current monitor, same as in application note. V3,V4 is thermometer, same in application note, except capacitor C118 is 0.1u instead of 470pF. Tested ok: Tint, VCC, V1, V2, V1-V2. Test failed: TR2 reads 50-something degC instead of same as Tint.
Done:
- LED_FP1A..D: tested ok. K.O. 15 sep 2022
- USB UART: tested ok. K.O. 15 sep 2022
- J4A, J4B, J5A, J5B LEMO inputs (NIM/TTL) EXT_IN_LV(1..4). TTL threshold 1.7V, NIM threshold -0.3V. 50 Ohm termination. TTL straight, NIM inverted.
- J6A, J6B LEMO clock inputs: 10 MHz TTL from chronobox works, 10 MHz NIM from IO32 works. 18-apr-2023
- J7A, J7B LEMO outputs EXT_OUT(1), EXT_OUT(2) (NIM/TTL) (tested 23nov2022, K.O.). TTL ???, NIM inverted.
- TTL out no 50 ohm termination: 0=0V, 1=5V, rise and fall time ~5 ns
- TTL out with 50 ohm termination: 0=0V, 1=2.5V, rise and fall time <2ns
- NIM out no 50 ohm termination: 0=+50mV, 1=-1.8V, rise and fall time ~3ns
- NIM out with 50 ohm termination: 0=0V, 1=-0.9V, rise and fall time <2ns
- ethernet MAC i2c chip (K.O. 20sep2022: can read: i2cdump 0 0x5b, nov2022 have u-boot driver)
- CDM VX ports 2x(CLK, 3 tx, 4 rx) tested using test_cdm.exe and LVDS loopback in VX firmware.
- SFP i2c tested KO 22jun2023
- QSFP i2c tested KO 22jun2023
- i2c testing complete 22jun2023
- QSFP rx,tx tested 26june2023. lane0,1,2 ok, lane3 tx not connected, laser is off. qsfp0,1,2,3 all lanes ok.
- SFP rx,tx tested 26june2023. rx and tx okey. LOS ok, mod_absent ok.
Failure:
- ethernet: does not connect to alliedtelesys switch. connects to my USB-eth adapter at 100 Mbit speed. uboot mii status reports connection speed oscillating between 1000, 100 and 10. K.O. 16-sep-2022
- SFP LOS and mod_absent are swapped (in the FPGA pin definitions?)
- QSFP slot numbering is wrong.
Checklist for newly build boards
- put new board on workbench
- check - vme connector present, vme extraction handles present
- check - standoff are removed from all thru-holes
- plug Enclustra module
- check - SW6 both switches are in the "PS" position
- connect micro-usb cable from linux PC
- connect ethernet from 1gige capable network switch
- connect power from lab power supply - GND, +5V and -12V nominal, +5.7V and -12.5V actual
- power up, +5V current 2.10-2.8A, -12V current 0.05A
- on linux PC, open a new terminal, run: minicom -D /dev/ttyACM0
- in minicom window, observe messages about Xilinx first stage boot loader, etc
- on the ethernet switch, observe network link is 1gige speed (not 10mbit, not 100mbit).
- if everything boots okey, there will be a login prompt, login as root, password root.
- busybox devmem 0x80010000 # read firmware version number, i.e. 0xEDAD0A77
- TBW - test LEMO inputs
- TBW - test LEMO outputs
- TBW - test VX connectors
- TBW - test SFP connector
- TBW - test QSFP connector
Serial console
- check that linux computer has correct udev rules to allow access to /dev/ttyACM devices, see https://daq00.triumf.ca/DaqWiki/index.php/SLinstall#Configure_USB_device_permissions and https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Configure_USB_device_permissions
- connect micro-USB cable to connector J-UCB, other end connect to linux computer
- observe /dev/ttyACM0 was created
- run "minicom -D /dev/ttyACM0" (default serial settings are ok, otherwise, 115200n8)
- should have gdm-cdm login
- username root, password root
i2c
ZynqMP> i2c bus Bus 0: i2c@ff020000 ZynqMP> i2c dev 0 Setting bus to 0 ZynqMP> i2c probe Valid chip addresses: 33 4E 53 5B 6B 77 ZynqMP> i2c md 0x5b 0x98 0098: fc c2 3d 00 00 12 1a 6e 0a 90 85 04 94 10 08 50 ..=....n.......P
root@gdm-cdm:~# i2cdetect 0
Warning: Can't use SMBus Quick Write command, will skip some addresses
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0.
I will probe address range 0x03-0x77.
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f
00:
10:
20:
30: -- -- -- 33 -- -- -- --
40:
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60:
70:
root@gdm-cdm:~#
root@gdm-cdm:~# i2cdump 0 0x5b
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0, address 0x5b, mode byte
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00 ???????P?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e ........??=..??n
a0: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00 ???????P?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e ........??=..??n
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@gdm-cdm:~#
root@gdm0:~# i2cdetect -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#
root@cdm0:~# i2cdetect -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: 50 51 -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@cdm0:~#
- 0x33 - XU8 secure EEPROM (should be at 0x32)
- 0x4e - U23 current and temperature monitor
- 0x50, 0x51 - SFP
- 0x50 - QSFP, 4 QSFP modules enabled by GPIO QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL
- 0x53, 0x5b - ethernet mac eeprom
- 0x6b - U6 clock chip
U23
- internal temperature only
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x00 b # control register: "repeat mode, internal temperature only"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x03 # "Tint ready" and "busy", "busy is always 1 in repeat mode"
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 00 03 03 81 db 2a ce 2a 8e 00 6f 00 45 20 3f ?.????*?*?.o.E ?
10: 01 00 01 01 01 db 2a ce 2a 8e 00 6f 00 45 20 3f ?.????*?*?.o.E ?
...
readback:
reg0 - 03 - Tint ready
reg1 - 00 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB 0x81, bit 0x80 is "DV, data valid", bit 0x40 is "SS, sensor short", 0x20 is "SO, sensor open"
reg5 - Tint LSB 0xDB, Tint = 0x01DB = 475 * 0.0625 degC = 29.6 degC
reg6..F - stale data
- Tint, V1, V2, TR2, VCC
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x18 b # control register: "repeat mode, V1, V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 7f 18 7f 7f 81 b9 aa d0 aa 8f 83 5e 83 5e a0 41 ???????????^?^?A
10: 01 18 01 01 01 b9 2a d0 2a 8f 03 5e 03 5e 20 41 ??????*?*??^?^ A
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x1b9*0.0625 = 27.5 degC
reg6 - V1 MSB 0xaa, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 LSB 0xd0, V1 = 0x2ad0*305.18/1000000 = 3.3447 V (correct)
reg8 - V2 MSB 0xaa, ditto
reg9 - V2 LSB 0x8f, V2 = 0x2a8f*305.18/1000000 = 3.3249 V (correct, smaller than V1)
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x5e. TR2 = 0x35e*0.0625 = 53.875 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x41, VCC = 2.5+0x2041*305.18/1000000 = 5.019 V (correct, VCC is +5V)
3V3_SW current is (V2-V1)/0.020 = 0.99 A (about right?)
- Tint, V1-V2, TR2, VCC
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x19 b # control register: "repeat mode, V1-V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 7f 19 7f 7f 81 69 83 f2 83 f2 83 3a 83 3a a0 44 ?????i?????:?:?D
10: 01 19 01 01 01 69 03 f2 03 f2 03 3a 03 3a 20 44 ?????i?????:?: D
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x169*0.0625 = 22.5 degC
reg6 - V1 or V1-V2 MSB 0x83, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 or V1-V2 LSB 0xf2, V1-V2 = 0x3f2*19.42/1000000 = 0.0196142 V (correct, compare with V1 and V2 measured above)
reg8 - V2 or V1-V2 MSB
reg9 - V2 or V1-V2 LSB
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x3a. TR2 = 0x33a*0.0625 = 51.625 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x44, VCC = 2.5+0x2044*305.18/1000000 = 5.021 V (correct, VCC is +5V)
3V3_SW current is (V2-V1)/0.020 = 0.9807 A (about right?)
SFP
root@cdm0:~# i2cdump 0 0x50
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 04 07 00 00 00 00 40 40 0c 00 01 3d 00 00 00 ???....@@?.?=...
10: 0c 02 00 1e 46 49 4e 49 53 41 52 20 43 4f 52 50 ??.?FINISAR CORP
20: 2e 20 20 20 00 00 90 65 46 54 4c 46 38 35 32 36 . ..?eFTLF8526
30: 50 33 42 4e 4c 20 20 20 41 20 20 20 03 52 00 9d P3BNL A ?R.?
40: 00 1a 00 00 4e 33 41 42 34 4c 56 20 20 20 20 20 .?..N3AB4LV
50: 20 20 20 20 32 30 30 33 31 39 20 20 68 f0 03 de 200319 h???
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@cdm0:~#
root@cdm0:~# i2cdump 0 0x51
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 5a 00 d8 00 55 00 e2 00 90 88 71 48 8c a0 75 30 Z.?.U.?.??qH??u0
10: 21 34 01 f4 1b 58 03 e8 31 2d 04 eb 1f 07 06 31 !4???X??1-?????1
20: 31 2d 00 64 27 10 00 9e 00 00 00 00 00 00 00 00 1-.d'?.?........
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
40: 00 00 00 00 3f 80 00 00 00 00 00 00 01 00 00 00 ....??......?...
50: 01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 b7 ?...?...?......?
60: 19 9f 80 c5 0e 17 12 c4 1f 99 00 00 00 00 30 00 ??????????....0.
70: 00 00 00 00 00 00 00 00 ff ff ff ff ff ff ff 01 ...............?
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@cdm0:~#
QSFP
QSFP i2c enable lines, active low: QSFP0_SEL - JENC-A 82 - PS-MIO40 - linux gpio 378 QSFP1_SEL - JENC-A 84 - PS-MIO41 - linux gpio 379 QSFP2_SEL - JENC-A 100 - PS-MIO44 - linux gpio 382 QSFP3_SEL - JENC-A 106 - PS-MIO43 - linux gpio 381 (notice 2 and 3 are out of order)
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
# echo 378 >> /sys/class/gpio/export ### SEL0 338+40
# echo 379 >> /sys/class/gpio/export ### SEL1 338+41
# echo 381 >> /sys/class/gpio/export ### SEL3 338+43
# echo 382 >> /sys/class/gpio/export ### SEL2 338+44
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) in hi
gpio-379 ( |sysfs ) in hi
gpio-381 ( |sysfs ) in hi
gpio-382 ( |sysfs ) in hi
root@gdm0:~# echo out >> /sys/class/gpio/gpio381/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio382/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio378/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio379/direction
root@gdm0:~#
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out lo
gpio-379 ( |sysfs ) out lo
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out lo
root@gdm0:~#
echo 1 >> /sys/class/gpio/gpio381/value
echo 1 >> /sys/class/gpio/gpio382/value
echo 1 >> /sys/class/gpio/gpio378/value
echo 1 >> /sys/class/gpio/gpio379/value
cat /sys/kernel/debug/gpio
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out hi
gpio-382 ( |sysfs ) out hi
root@gdm0:~#
root@gdm0:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#
NOTICE NOTHING AT ADDRESS 0x50
ENABLE QSFP0, OBSERVE IT IS AT ADDRESS 0x50
root@gdm0:~# echo 0 >> /sys/class/gpio/gpio378/value
root@gdm0:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: 50 -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#
root@gdm0:~# i2cdump -y 0 0x50
No size specified (using byte-data access)
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 0d 00 02 8e 00 00 01 00 00 05 55 00 05 00 00 00 ?.??..?..?U.?...
10: 00 00 00 00 00 00 1e b8 00 00 81 a0 00 00 00 00 ......??..??....
20: 00 00 12 12 00 01 00 01 00 01 0e e0 0f 20 0e e0 ..??.?.?.???? ??
30: 00 00 1f a0 1e 3b 1f 72 00 01 00 00 00 00 00 00 ..???;?r.?......
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
60: 00 00 00 00 00 00 00 00 00 00 1f 00 00 00 08 00 ..........?...?.
70: 00 00 00 00 00 00 00 ff ff ff ff ff ff ff ff 00 ................
80: 0d 00 0c 04 00 00 00 40 40 02 d5 05 67 00 00 96 ?.??...@@???g..?
90: 00 00 c8 00 46 49 4e 49 53 41 52 20 43 4f 52 50 ..?.FINISAR CORP
a0: 20 20 20 20 07 00 90 65 46 54 4c 34 31 30 51 44 ?.?eFTL410QD
b0: 34 43 20 20 20 20 20 20 41 20 42 68 07 d0 00 43 4C A Bh??.C
c0: 00 07 0f de 58 37 39 41 43 30 52 20 20 20 20 20 .???X79AC0R
d0: 20 20 20 20 32 32 30 33 30 39 20 20 3c 10 00 9e 220309 <?.?
e0: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@gdm0:~#
ethernet mac eeprom
- correct chip with 84-bit ethernet mac address
root@cdm1:~# i2cdump 0 0x53
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
root@cdm1:~#
root@cdm1:~# i2cdump 0 0x5b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00 ????4??2?K?.?...
90: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c ..........??=?Q<
a0: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00 ????4??2?K?.?...
b0: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c ..........??=?Q<
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@cdm1:~#
- wrong "602" chip with 64-bit IPv6 address
root@cdm0:~# i2cdump 0 0x53
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
root@cdm0:~# i2cdump 0 0x5b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00 ???????Q?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e ........??=..??.
a0: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00 ???????Q?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e ........??=..??.
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@cdm0:~#
read ethernet mac address from i2c
(this code is copied from uboot command line i2c code)
in uboot sources board/xilinx/common/board.c replace original function with this:
// special code to read ethernet MAC address from the DS-DM-Rev0 board. K.O. Sep-2022
int zynq_board_read_rom_ethaddr(unsigned char *ethaddr)
{
struct udevice *bus;
int ret;
int busnum = 0;
ret = uclass_get_device_by_seq(UCLASS_I2C, busnum, &bus);
if (ret) {
printf("%s: No bus %d\n", __func__, busnum);
return ret;
}
int chip_addr = 0x5B;
struct udevice *dev;
ret = i2c_get_chip(bus, chip_addr, 1, &dev);
if (ret) {
printf("%s: Bus %d no chip 0x%02x\n", __func__, busnum, chip_addr);
return ret;
}
int dev_addr = 0x98;
unsigned char data[8];
ret = dm_i2c_read(dev, dev_addr, data, 8);
if (ret) {
printf("%s: Bus %d chip 0x%02x read error %d\n", __func__, busnum, chip_addr, ret);
return ret;
}
printf("%s: Bus %d chip 0x%02x addr 0x%02x read: 0x %02x %02x %02x %02x %02x %02x %02x %02x\n", __func__, busnum, chip_addr, dev_addr, data[0], data[1], data[2], data[3], data[4], data[5], data[6], data[7]);
// see Atmel-8807-SEEPROM-AT24MAC402-602-Datasheet.pdf
if (data[0] == 0) {
// eiu-48 chip
ethaddr[0] = data[2];
ethaddr[1] = data[3];
ethaddr[2] = data[4];
ethaddr[3] = data[5];
ethaddr[4] = data[6];
ethaddr[5] = data[7];
} else {
// eiu-64 chip
ethaddr[0] = data[0];
ethaddr[1] = data[1];
ethaddr[2] = data[2];
ethaddr[3] = data[5];
ethaddr[4] = data[6];
ethaddr[5] = data[7];
}
printf("%s: ethaddr %02x:%02x:%02x:%02x:%02x:%02x\n", __func__, ethaddr[0], ethaddr[1], ethaddr[2], ethaddr[3], ethaddr[4], ethaddr[5]);
return ret;
}
also this should have worked if i2c_xxx() functions were enabled in uboot:
i2c_set_bus_num(0); i2c_probe(0x5b); i2c_read(0x5b, 0x9a, ethaddr, 6);
read ethernet mac address from i2c (SHOULD WORK)
#ethernet related setup setup_eth=run readmac buildmac #read mac address from eeprom readmac=i2c dev 0; i2c read 50 FA.1 6 $loadaddr #build the ethaddr variable #not very nice, but does the job buildmac=\ e=" "; sep=" " \ for i in 0 1 2 3 4 5 ; do\ setexpr x $loadaddr + $i\ setexpr.b b *$x\ e="$e$sep$b"\ sep=":"\ done &&\ setenv ethaddr $e
read ethernet mac address from i2c (DOES NOT WORK)
this method does not work: inside board/xilinx/common/board.c:int zynq_board_read_rom_ethaddr(unsigned char *ethaddr), uclass_get_device_by_ofnode(UCLASS_I2C_EEPROM, eeprom, &dev) returns ENODEV and read from i2c does not happen. K.O. Sep-2022
Read:
- https://support.xilinx.com/s/question/0D52E00006hpKlsSAE/uboot-and-ethernet-mac-address-from-eeprom?language=en_US (last answer)
- https://github.com/Xilinx/u-boot-xlnx/blob/master/board/xilinx/common/board.c (zynq_board_read_rom_ethaddr())
Note:
- 0x5B is the i2c chip address
- 0x9A is the data offset inside the chip, see datasheet or i2c read dump above.
Edit:
- emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/u-boot/files/platform-top.h
#include <configs/xilinx_zynqmp.h> #include <configs/platform-auto.h> //#define CONFIG_I2C_EEPROM //#define CONFIG_SYS_I2C_EEPROM_ADDR 0x5b //#define CONFIG_SYS_I2C_EEPROM_ADDR_OVERFLOW 0x0 #define CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET 0x9A #error HERE!
- emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi
/include/ "system-conf.dtsi"
/ {
chosen {
xlnx,eeprom = &eeprom;
};
};
&i2c0 {
eeprom: eeprom@5b { /* u88 */
compatible = "atmel,24mac402";
reg = <0x5b>;
};
};
- components/yocto/workspace/sources/u-boot-xlnx/configs/xilinx_zynqmp_virt_defconfig
CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET=0x9A
enable VX clock
devmem 0x80010010 32 0x8; sleep 1; devmem 0x80010010 32 0x0; si5394-i2c-file /media/sd-mmcblk1p1/00_freerun.txt 0 0x6b
clock chip configuration
file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt:
- VCO is 14 GHz
- Tvco is 71.43 ps
- N0 divider is 14, frequency is 1000 MHz
- out0 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz
- out1 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
- out2 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
- out3 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz is the feedback for zero delay
clock chip monitoring
from si5395-94-92-family.pdf:
- reg 0x1: page select, set to 0 or set to 5 to read 0x53F
- reg 0x2: 0x94
- reg 0x3: 0x53 -> device is a si5394
- reg 0xC: LOSXAXB
- reg 0xD: LOS and OOF for the 4 clock inputs
- reg 0xE: LOL and HOLD
- reg 0xF: CAL_PLL
- reg 0x11: sticky bits for reg 0xC
- reg 0x12: sticky bits for reg 0xD
- reg 0x13: sticky bits for reg 0xE
- reg 0x14: sticky bits for reg 0xF
- reg 0x1C: device reset
- reg 0x1E: low power, hard reset, SYNC
- reg 0x507: currently selected input clock
- reg 0x52A: input clock select
- reg 0x53F: HOLD_HIST_VALID and FASTLOCK_STATUS
arm and linux benchmark
memory benchmark:
daq13$ arm-linux-gnueabi-gcc -o memcpy.armv7 memcpy.cc -march=armv7 -static -O2 scp memcpy.armv7 to ... root@gdm-cdm:~# ./memcpy.armv7 memcpy 1 KiBytes: 1288 MB/sec memcpy 2 KiBytes: 1924 MB/sec memcpy 4 KiBytes: 2554 MB/sec memcpy 8 KiBytes: 3054 MB/sec memcpy 16 KiBytes: 3262 MB/sec memcpy 32 KiBytes: 3250 MB/sec memcpy 64 KiBytes: 3456 MB/sec memcpy 128 KiBytes: 3556 MB/sec memcpy 256 KiBytes: 3780 MB/sec memcpy 512 KiBytes: 3795 MB/sec memcpy 1024 KiBytes: 3789 MB/sec memcpy 2048 KiBytes: 3729 MB/sec memcpy 4096 KiBytes: 3717 MB/sec memcpy 8192 KiBytes: 3687 MB/sec memcpy 16384 KiBytes: 3632 MB/sec memcpy 32768 KiBytes: 3529 MB/sec memcpy 65536 KiBytes: 3318 MB/sec memcpy 131072 KiBytes: 2893 MB/sec root@gdm-cdm:~#
ethernet receive:
daq13:bin$ ./ttcp -t -s -n 100000 10.0.0.24 ttcp-t: buflen=8192, nbuf=100000, align=16384/0, port=5001 tcp -> 10.0.0.24 ttcp-t: socket ttcp-t: connect ttcp-t: 819200000 bytes in 7.25 real seconds = 110358.39 KB/sec +++ ttcp-t: 100000 I/O calls, msec/call = 0.07, calls/sec = 13794.80 ttcp-t: 0.0user 0.2sys 0:07real 3% 0i+0d 760maxrss 0+2pf 1461+31csw daq13:bin$ root@gdm-cdm:~# ./ttcp.armv7 -r -s ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp ttcp-r: socket ttcp-r: accept from 10.0.0.25 ttcp-r: 819200000 bytes in 7.27 real seconds = 110098.22 KB/sec +++ ttcp-r: 212040 I/O calls, msec/call = 0.04, calls/sec = 29181.53 ttcp-r: 0.1user 5.7sys 0:07real 81% 0i+0d 584maxrss 0+2pf 125601+2699csw root@gdm-cdm:~#
ethernet transmit:
root@gdm-cdm:~# ./ttcp.armv7 -t -s -n 100000 10.0.0.25 ttcp-t: buflen=8192, nbuf=100000, align=16384/0, port=5001 tcp -> 10.0.0.25 ttcp-t: socket ttcp-t: connect ttcp-t: 819200000 bytes in 6.95 real seconds = 115078.69 KB/sec +++ ttcp-t: 100000 I/O calls, msec/call = 0.07, calls/sec = 14384.84 ttcp-t: 0.0user 0.7sys 0:06real 11% 0i+0d 584maxrss 0+2pf 1162+1017csw root@gdm-cdm:~# daq13:bin$ ./ttcp -r -s ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp ttcp-r: socket ttcp-r: accept from 10.0.0.24 ttcp-r: 819200000 bytes in 6.97 real seconds = 114841.84 KB/sec +++ ttcp-r: 161335 I/O calls, msec/call = 0.04, calls/sec = 23160.01 ttcp-r: 0.0user 1.9sys 0:06real 28% 0i+0d 760maxrss 0+2pf 80646+51csw daq13:bin$
Install Xilinx tools
- install Vivado 2020.2
login at https://www.xilinx.com/myprofile.html go to "Downloads" go to archive, find 2020.2 download Xilinx_Unified_2020.2_1118_1232_Lin64.bin sh ./Xilinx_Unified_2020.2_1118_1232_Lin64.bin banner window should open with spinner "downloading installation data" "a newer version is available" -> say "continue" next "select install type" window: provide email and password, select "download image" select directory /home/olchansk/Xilinx/Downloads/2020.2\ select "linux" and "full image" next download summary: space required 38.52 Gbytes download installation progress downloading spinner, 16 M/s 47 minutes... "download image has been created successfully". Ok. check contents of /home/olchansk/Xilinx/Downloads/2020.2 ls -l /home/olchansk/Xilinx/Downloads/2020.2 total 67 drwxr-xr-x 2 olchansk users 9 Sep 1 16:22 bin drwxr-xr-x 3 olchansk users 15 Sep 1 16:23 data drwxr-xr-x 4 olchansk users 4 Sep 1 16:22 lib drwxr-xr-x 2 olchansk users 644 Sep 1 16:22 payload drwxr-xr-x 2 olchansk users 7 Sep 1 16:22 scripts drwxr-xr-x 4 olchansk users 4 Sep 1 16:22 tps -rwxr-xr-x 1 olchansk users 3256 Nov 18 2020 xsetup daq13:2020.2$ ./xsetup spinned loading installation data xilinx design tools 2022.1 now available -> say continue "welcome" -> next "select product" -> vivado -> next -> vivado hl system edition -> next select devices: only zynq ultrascale+ mpsoc -> next select destination: /opt/Xilinx (as root, mkdir /opt/Xilinx, chmod olchansk.users /opt/Xilinx) install ... complete move /home/olchansk/Xilinx/Downloads/2020.2 to /daq/daqstore/olchansk/Xilinx/Downloads/
- install petalinux 2020.2
./xsetup "a newer version is available" -> say "continue" next "select product to install" -> select Petalinux (Linux only) -> next "select destination directory" -> select "/opt/Xilinx" (disk space required 2.64 GB) -> next "summary" -> install ... error about missing /tmp/tmp-something files "installation completed successfully" (hard to dismiss, "ok" button is partially cut-off) done? I think it failed, /opt/Xilinx/PetaLinux/2020.2/bin is empty except for petalinux-v2020.2-final-installer.run try to run it by hand, same error about /tmp/tmp-something files. strange... notice it complains about "truncate", which truncate finds ~/bin/truncate, get rid of it, try again now complains about missing texinfo and zlib1g:i386 apt install texinfo -> ok apt install zlib1g:i386 -> installs bunch of gcc stuff -> ok try again reports "already installed" -> delete /opt/Xilinx/.xinstall/PetaLinux_2020.2/, delete entries in ~/.Xilinx/registry/installedSW.xml try again success
- install vivado 2022.1 and petalinux 2022.1 - everything is pretty much the same
Petalinux
- cd PetaLinux_GDM_CDM
- petalinux-config
- enable i2c MAC address and DHCP
git clone https://bitbucket.org:/team-ds-dm/ds-dm-u-boot-xlnx.git cd ds-dm-u-boot-xlnx git checkout ds-dm-u-boot-xlnx linux-components -> uboot -> ext-local-src external u-boot local source -> ds-dm-u-boot-xlnx (path to the customized uboot git repository)
- enable DHCP
Subsystem AUTO Hardware Settings -> Ethernet Settings randomize MAC address -> NO ethernet mac address -> leave empty obtain ip address automatically -> YES
- set hostname and product names
Firmware Version Configuration -> Host name -> "ds-dm" Product name -> "Petalinux_GDM_CDM"
- configure linux kernel
petalinux-config -c kernel
- enable NFS-Root
petalinux-config Image Packaging Configuration > Root File System Type -> set to NFS Location of NFS root directory set to "/nfsroot" petalinux-config -c kernel Networking support > IP: kernel level configuration enable DHCP, BOOTP, RARP File systems > Network file systems > Root file systems on NFS
- manually fix linux kernel command line:
grep nfsroot PetaLinux_GDM_CDM/project-spec/configs/config edit CONFIG_SUBSYSTEM_BOOTARGS_GENERATED to read earlycon console=ttyPS0,115200 clk_ignore_unused panic=60 root=/dev/nfs nfsroot=/nfsroot/%s ip=dhcp rw
- check configuration in
- PetaLinux_GDM_CDM/project-spec/configs/config
- PetaLinux_GDM_CDM/project-spec/configs/rootfs_config
- PetaLinux_GDM_CDM/components/plnx_workspace/device-tree/device-tree/system-conf.dtsi
JTAG server
localhost:3121
Firmware registers
Block 0
0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
Block 1
Block 2
Block 3
DS20k registers:
- busybox devmem 0x80013000 32
reg | version | xDM | xx | description 0 | 0x20230731 | ALL | RO | ds20k version 1 | 0x20230731 | ALL | RW | scratch read/write register 2 | 0x20230731 | ALL | RW | configure inputs and outputs 3 | 0x20230731 | ALL | RW | FP_LED mux 4 | 0x20230731 | ALL | RW | EXT_OUT mux 5 | 0x20230731 | ALL | RO | VX_RX state 6 | 0x20230731 | ALL | RO | VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state 7 | 0x20230731 | ALL | RW | LED_OUT, EXT_OUT, VX_TX outputs 8 | 0x20230731 | ALL | RW | VX_TX mux and config 9 | 0x20230731 | ALL | RW | trigger config 10 | - | ALL | RO | status register 11 | 0x20230731 | ALL | RO | trigger counter 12 | 0x20230731 | ALL | RO | time slice marker counter xx | - | GDM | RO | GPS 1pps period, 125 MHz xx | - | GDM | RO | Ru clock 1pps period, 125 MHz 255 | - | ALL | RW | old control register
Register 0 0x80013000 ds20k version
ds20k version 0xYYYYMMDD
Register 1 0x80013004 scratch
scratch read-write register
Register 2 0x80013008 input and output config
bit | version | fpga name | description 0 | ALL | lemo_enable | enable LEMO input 1 1 |. | | 2 2 | |. | 3 3 | | | 4 4 | ALL. | lemo_invert | invert LEMO input 1 5 | | | 2 6 | | | 3 7 | | | 4 9 | ALL | ext_out_disable | disable LEMO output 1 10 | | | 2 11 | ALL. | ext_out_invert | invert LEMO output 1 12 | | | 2
Register 3 0x8001300C FP_LED control
wire [15:0] led_out_mux_sel = register_data_in[3][15:0]; wire [3:0] led_out_invert = register_data_in[2][19:16];
led_out_mux_sel is 4 groups (one per LED) of 4 bits (choice 0..15): 0 - power on default 1 - led_out_reg register 7 2 - reserved for "PLL locked" 3 - reserved for "SFP link is good" 4 - LEMO input 1 5 - LEMO input 2 6 - LEMO input 3 7 - LEMO input 4 8 - LEMO output 1 9 - LEMO output 2 10 11 12 13 14 15 - fixed logic level 1
Register 4 0x80013010 LEMO OUT control
wire [7:0] ext_out_mux_sel = register_data_in[4][7:0];
ext_out_mux_sel is 2 groups (one per LEMO) of 4 bits (choice 0..15): 0 - power on default 1 - ext_out_reg register 7 2 - trg_pulser 3 - vx1_tx_out[0] for VX loopback test 4 - vx1_rx[3] for VX loopback test 5 - vx1_rx3_sync for VX loopback test 6 7 8 9 10 11 12 13 14 15 - fixed logic level 1
Register 5 0x80013014 VX_RX status
assign register_data_out[5] = {
vx8_rx[3], vx8_rx[2], vx8_rx[1], vx8_rx[0],
vx7_rx[3], vx7_rx[2], vx7_rx[1], vx7_rx[0],
vx6_rx[3], vx6_rx[2], vx6_rx[1], vx6_rx[0],
vx5_rx[3], vx5_rx[2], vx5_rx[1], vx5_rx[0],
vx4_rx[3], vx4_rx[2], vx4_rx[1], vx4_rx[0],
vx3_rx[3], vx3_rx[2], vx3_rx[1], vx3_rx[0],
vx2_rx[3], vx2_rx[2], vx2_rx[1], vx2_rx[0],
vx1_rx[3], vx1_rx[2], vx1_rx[1], vx1_rx[0]
};
Register 6 0x80013018 VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state
assign register_data_out[6] = {
vx2_tx_out[2], vx2_tx_out[1], vx2_tx_out[0], vx1_tx_out[2],
vx1_tx_out[1], vx1_tx_out[0], ext_out[2], ext_out[1],
fp_led_out[3], fp_led_out[2], fp_led_out[1], fp_led_out[0],
ext_in_lv[4], ext_in_lv[3], ext_in_lv[2], ext_in_lv[1],
vx12_rx[3], vx12_rx[2], vx12_rx[1], vx12_rx[0],
vx11_rx[3], vx11_rx[2], vx11_rx[1], vx11_rx[0],
vx10_rx[3], vx10_rx[2], vx10_rx[1], vx10_rx[0],
vx9_rx[3], vx9_rx[2], vx9_rx[1], vx9_rx[0]
};
bit | version | fpga name | description 0 | ALL | vx9_rx | VX_RX 1 | | | 2 | | | 3 | | | 4 | ALL | vx10_rx | VX_RX 5 | | | 6 | | | 7 | | | 8 | ALL | vx11_rx | VX_RX 9 | | | 10 | | | 11 | | | 12 | ALL | vx12_rx | VX_RX 13 | | | 14 | | | 15 | | | 16 | ALL | ext_in_lv | LEMO inputs 17 | | | 18 | | | 19 | | | 20 | ALL | FP_LED | FP_LEDs 21 | | | 22 | | | 23 | | | 24 | ALL | ext_out[1] | LEMO outputs 25 | | ext_out[2] | 26 | ALL | vx1_tx | VX1_TX 27 | | | 28 | | | 29 | ALL | vx2_tx | VX2_TX 30 | | | 31 | | |
Register 7 0x8001301C LED_OUT, EXT_OUT, VX_TX ouputs
wire [3:0] led_out_reg = register_data_in[7][3:0]; wire [2:1] ext_out_reg = register_data_in[7][5:4]; // register_data_in[7][6]; // register_data_in[7][7]; wire [7:0] vx_tx_out_reg = register_data_in[7][15:8];
bit | version | fpga name | description 0 | ALL | led_out_reg | FP_LED 1 1 | | | 2 2 | | | 3 3 | | | 4 4 | ALL | ext_out_reg | LEMO OUT 1 5 | | | 2 6 | - | | 7 | - | | 8 | ALL | vx_tx_out_reg | VX1_TX 0 9 | | | 1 10 | | | 2 11 | | | - 12 | ALL | | VX2_TX 0 13 | | | 1 14 | | | 2 15 | | | - 16 | - | | 17 | | | 18 | | | 19 | | | 20 | | | 21 | | | 22 | | | 23 | | | 24 | | | 25 | | | 26 | | | 27 | | | 28 | | | 29 | | | 30 | | | 31 | | |
Register 8 0x80013020 VX_TX config
wire [3:0] vx_tx_mux_sel = register_data_in[8][3:0];
vx_tx_mux_sel is 4 bits (choice 0..15): 0 - power on default, control by vx_tx_out_reg 1 - GDM 2 - CDM 3 - pulser loopback test 4 - pulser loopback test 5 - 62.5 MHz output 6 - 125 MHz output 7 8 9 10 11 12 13 14 15
Register 9 0x80013024 trigger control
wire [7:0] trg_src_mask = register_data_in[9][7:0]; wire [7:0] tsm_src_mask = register_data_in[9][15:8]; wire trg_pulser_enable = register_data_in[9][16]; wire tsm_pulser_enable = register_data_in[9][17]; wire trg_software = register_data_in[9][18]; wire tsm_software = register_data_in[9][19]; // bits 20:31 not used
wire [7:0] xxx_src_bits =
{
sfp_rx[1],
sfp_rx[0],
tsm_pulser & tsm_pulser_enable,
trg_pulser & trg_pulser_enable,
lemo_in_sync[4],
lemo_in_sync[3],
lemo_in_sync[2],
lemo_in_sync[1]
};
trg_src_mask and tsm_src_mask bits: 0 - LEMO IN 1 1 - LEMO IN 2 2 - LEMO IN 3 3 - LEMO IN 4 4 - trg_pulser 5 - tsm_pulser 6 - sfp_rx[0] 7 - sfp_rx[1]
Register 10 0x80013028 status register
bit | version | fpga name | description 0 | | | 1 | | | 2 | | | 3 | | | 4 | | | 5 | | | 6 | | | 7 | | | 8 | | | 9 | | | 10 | | | 11 | | | 12 | | | 13 | | | 14 | | | 15 | | | 16 | | | 17 | | | 18 | | | 19 | | | 20 | | | 21 | | | 22 | | | 23 | | | 24 | | | 25 | | | 26 | | | 27 | | | 28 | | | 29 | | | 30 | | | 31 | | |
Register 11 0x8001302C trg_counter
trigger counter
Register 12 0x80013030 tsm_counter
time slice marker counter
Firmware registers branch develop_ko
Register map
0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2 1 | ALL | ALL | RW | read write scratch register 2 | ALL | CDM | ?? | MGT not used 3 | ALL | CDM | RO | MGT debug_data 4 | ALL | CDM | RW | clk_config_vec 5 | ALL | CDM | ?? | not used 6 | ALL | CDM | RO | CDM_link_data_processing:o_error_time 7 | ALL | CDM | RO | CDM_link_data_processing:o_error_count 255 | ALL | ALL | RW | control register
register 0 0x80010000
GDM:
0 - gdm_link_interface:i_mgt_rst 2 - gdm_link_interface:i_link_down_latched_rst 8 - GDM_link_data_processing:i_rst 10..9 - GDM_link_data_processing:i_data_mode
CDM:
0 - cdm_link_interface:i_mgt_rst 2 - cdm_link_interface:i_link_down_latched_rst 8 - CDM_link_data_processing:i_rst 10..9 - CDM_link_data_processing:i_data_mode
register 1 0x80010004
GDM:
nlinks-1..0 - gdm_link_interface:i_rx_slide_trigger
CDM:
nlinks-1..0 - cdm_link_interface:i_rx_slide_trigger
register 2 0x80010008
GDM:
nlinks-1..0 - gdm_link_interface:o_link_power_good nlinks+15..16 - gdm_link_interface:o_link_status
CDM:
nlinks-1..0 - cdm_link_interface:o_link_power_good nlinks+15..16 - cdm_link_interface:o_link_status
register 3 0x8001000c
GDM: simple loopback register
CDM:
31..0 - debug_data - cdm_link_interface:o_debug
o_debug:
rx_link_rst & rx_error & rx_link_up & rx_receiving_data & std_logic_vector(rx_state_count) & tx_state_count_on_rx_clk & i_rx_ctrl3(0) & i_rx_ctrl1(1 downto 0) & i_rx_ctrl0(1 downto 0) & rx_data_is_k28p1_k28p5 & i_rx_data;
register 4 0x80010010
GDM write:
0 - clk_config_vec(0) - CLK_IN_SEL_LS(0) 1 - clk_config_vec(1) - CLK_IN_SEL_LS(1) 2 - clk_config_vec(2) - CLK_EXT_SEL_LS 3 - clk_config_vec(3) - CLK_RSTn_LS
GDM read:
0 - clk_config_vec(0) - CLK_IN_SEL_LS(0) 1 - clk_config_vec(1) - CLK_IN_SEL_LS(1) 2 - clk_config_vec(2) - CLK_EXT_SEL_LS 3 - clk_config_vec(3) - CLK_RSTn_LS 4 - clk_config_vec(4) - CLK_LOSXTn_LS 5 - clk_config_vec(5) - CLK_LOLn_LS 6 - clk_config_vec(6) - CLK_INTn_LS 7 - constant 1 31..8 - constant 0
register 5 0x80010014
not used
register 6 0x80010018
GDM:
3..0 - GDM_link_data_processing:i_status_select
CDM:
31..0 - CDM_link_data_processing:o_error_count
register 7 0x8001001c
GDM:
31..0 - GDM_link_data_processing:o_status_vector
CDM:
31..0 - CDM_link_data_processing:o_error_count
register 246 0x800103d8
VX_RX
register 247 0x800103dc
VX_RX, LEMO_IN, FP_LED, LEMO_OUT, VX_TX
control register 255 0x800103fc
31..28 | ALL | function mux, 4 bits: 0 - outputs driven by control register, 1 - dsvslice GDM production, 2 - dsvslice CDM production, 3, 4, 5 - LVDS loopback, see source code 27 | ALL | lemo_in(4) invert 26 | ALL | lemo_in(3) invert 25 | ALL | lemo_in(2) invert 24 | ALL | lemo_in(1) invert 23 | ALL | lemo_in(4) enable 22 | ALL | lemo_in(3) enable 21 | ALL | lemo_in(2) enable 20 | ALL | lemo_in(1) enable 19 | ALL | software_trigger 18 | ALL | lemo_trigger_enable 17 | ALL | pulser_trigger_enable 16 | ALL | trigger_enable 15 | ALL | gdm_trigger_enable 14 13 | ALL | drive ext_out(2) LEMO output 12 | ALL | drive ext_out(1) LEMO output 11 | ALL | fp_led_out(3) front panel LEDs 10 | ALL | fp_led_out(2) 9 | ALL | fp_led_out(1) 8 | ALL | fp_led_out(0) 7 | ALL | not used 6 | ALL | vx2_tx_out(2) 5 | ALL | vx2_tx_out(1) 4 | ALL | vx2_tx_out(0) 3 | ALL | not used 2 | ALL | vx1_tx_out(2) 1 | ALL | vx1_tx_out(1) 0 | ALL | vx1_tx_out(0)
Power up sequence
- on the CDM:
busybox devmem 0x80010010 32 0x8 sleep 1; busybox devmem 0x80010010 32 0x0 #/home/dsdaq/si5394-i2c-file /home/dsdaq/CDM_v2.0_IN1_fixed_and_IN2_RX_Recovered_Si5394-RevA-Registers.txt 0 0x6b /home/dsdaq/si5394-i2c-file /home/dsdaq/CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b busybox devmem 0x80010004 32 0x1; # link startup mode reset until phase-aligned busybox devmem 0x80010000 32 0x5; # reset link and link status registers sleep 1; # just holds reset high busybox devmem 0x80010000 32 0x0; # lower reset and let links lock sleep 2; # give links time to lock busybox devmem 0x80010008 32; # read status 0x00000003 means locked and phase aligned busybox devmem 0x8001000c 32 # read data, 0x1B1C 3CBC lower 16 bits are K-Codes K.28.1 (3C) and K.28.5 (BC) when locked and phase aligned
Build firmware
Build from git clone
THESE ARE K.O.'s NOTES FOR CREATING THE PETALINUX DIRECTORY.
THEY DO NOT WORK!
COPY PETALINUX FROM A WORKING PROJECT AND USE "make gdm" and "make cdm" AS DESCRIBED BELOW.
- git clone git@edev-group.triumf.ca:fw/exp/darkside/gcdm.git
- #Makefile change VIVADO_SETTINGS_SCRIPT := /opt/Xilinx/Vivado/2022.1/settings64.sh
- #. /opt/Xilinx/Vivado/2022.1/settings64.sh
- . /opt/Xilinx/Vivado/2020.2/settings64.sh
- make clean
- make all_from_scratch
- . /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
- make petalinux_create
- make petalinux_rebuild_new_hw_des
- bomb out: The TMPDIR: /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp can't be located on nfs.
- mkdir /tmp/build_tmp
- rm -rf /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp/
- ln -s /tmp/build_tmp /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp
- try again
- grinds, loads a whole bunch of packages...
- finishes with desire to copy things to /tftpboot
- make sdcard_cp_to wants to copy files from PetaLinux_GDM_CDM/images/linux/ to SD card
Build firmware
NOTE: directory Petalinux_GDM_CDM should already exist!
. /opt/Xilinx/Vivado/2020.2/settings64.sh . /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh make clean_gdm # remove gdm build tree make gdm # build or rebuild GDM make copy_gdm # copy to gdm0 make clean_cdm # remove cdm build tree make cdm # build or rebuild CDM make copy_cdm # copy to cdm0 and cdm1
copy to SD card:
open a root shell format 16 GB Sd card per above cd .../ds-dm-gcdm make copy
build times
CDM 12-june-2023 69aabc1c25130d970bc375aca684bd68849e6685 daq13 AMD-5700G 1688.61user 399.33system 23:28.84elapsed 148%CPU dsdaqgw AMD-7700 1090.55user 247.34system 16:03.55elapsed 138%CPU
prepare bootable sd card
format the sd card
this only needs to be done once
- become root
- cd ~olchansk/git/ds-dm-gcdm
- use "lsblk" to identify the SD card (should show as 8/16/32 GB block device)/ /dev/sdd in this case
- make sdcard_format SDCARD_DEVICE=/dev/sdd
- disconnect sd card, reconnect the sd card (to detect new partition tables, etc)
copy CDM boot files
cd /home/dsdmdev/git/ds-dm-gcdm make copy
copy boot files to the sd card
- as root: identify partition labels, run "blkid", should say "BOOT", "rootfs" and "data"
- mount
mkdir /media/olchansk/BOOT mkdir /media/olchansk/rootfs mkdir /media/olchansk/data mount -L BOOT /media/olchansk/BOOT mount -L rootfs /media/olchansk/rootfs mount -L data /media/olchansk/data cp PetaLinux_GDM_CDM/images/linux/BOOT.BIN /media/olchansk/BOOT/ cp PetaLinux_GDM_CDM/images/linux/boot.scr /media/olchansk/BOOT/ cp PetaLinux_GDM_CDM/images/linux/image.ub /media/olchansk/BOOT/ umount /media/olchansk/BOOT umount /media/olchansk/rootfs umount /media/olchansk/data eject /dev/sdd
boot messages
Xilinx Zynq MP First Stage Boot Loader Release 2020.2 Sep 24 2022 - 13:29:15 NOTICE: ATF running on XCZU4CG/silicon v4/RTL5.1 at 0xfffea000 NOTICE: BL31: v2.2(release):xlnx_rebase_v2.2_2020.3 NOTICE: BL31: Built : 18:02:46, Sep 28 2022 U-Boot 2020.01 (Sep 28 2022 - 18:03:39 +0000) Model: DarkSide 20k DM Board: Xilinx ZynqMP DRAM: 2 GiB usb dr_mode not found PMUFW: v1.1 EL Level: EL2 Chip ID: zu4 NAND: 0 MiB MMC: mmc@ff160000: 0, mmc@ff170000: 1 In: serial@ff000000 Out: serial@ff000000 Err: serial@ff000000 Bootmode: SD_MODE1 Reset reason: SOFT Net: ZYNQ GEM: ff0b0000, mdio bus ff0b0000, phyaddr -1, interface rgmii-id Warning: ethernet@ff0b0000 (eth0) using random MAC address - d6:62:5f:13:00:44 eth0: ethernet@ff0b0000 ZYNQ GEM: ff0e0000, mdio bus ff0e0000, phyaddr -1, interface rgmii-id Could not get PHY for eth1: addr -1 Hit any key to stop autoboot: 0 ZynqMP> CTRL-A Z for help | 115200 8N1 | NOR | Minicom 2.7.1 | VT102 | Online 122:2 | ttyACM0
Software
- ssh cdm0 # or gdm0
- sudo apt install i2c-tools libi2c-dev
- git clone https://bitbucket.org/team-ds-dm/ds-dm-software
- cd ds-dm-software
- make
test_cdm.exe
- CDM SFP status: /home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp
DS-DM mapping /dev/mem at 0x80010000 DS-DM FPGA firmware revision 0xbb2f0ae7 CDM firmware 0xbb2f0ae7 arg 1: [--sfp] Polling SFP status... identifier 0x03 connector 0x07 encoding 0x01 wavelength 0x0352 (850 nm) vendor_name [FINISAR CORP. ] vendor_pn [FTLF8526P3BNL ] vendor_rev [A ] vendor_sn [N3AB9M8 ] vendor_date [200319 ] dm_type 0x68 temp 29.0 C vcc 3.323 V tx_bias 7.250 mA tx_power 478.4 uW rx_power 2.3 uW SFP good 1, status: temp 30.6 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 818 uW ...
- GDM QSFP status: /home/dsdaq/online/ds-dm-software/test_cdm.exe --qsfp3 --qsfp
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --qsfp3 --qsfp DS-DM mapping /dev/mem at 0x80010000 DS-DM FPGA firmware revision 0x53aee418 CDM firmware 0x53aee418 arg 1: [--qsfp3] gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio: gpio-378 ( |sysfs ) out hi gpio-379 ( |sysfs ) out hi gpio-381 ( |sysfs ) out lo gpio-382 ( |sysfs ) out hi arg 2: [--qsfp] Polling QSFP status... identifier 0x0d status 0x02 los 0x8f temp 28.2 C vcc 3.323 V rx_power 0.1 0.1 0.1 0.1 uW tx_bias 7.6 7.6 7.6 0.0 mA tx_power 792.2 773.8 823.0 0.1 uW vendor_name [FINISAR CORP ] vendor_pn [FTL410QD4C ] vendor_rev [A ] wavelength 850 max_temp 70 C vendor_sn [X79AC0R ] vendor_date [220309 ] QSFP good 1, status: temp 27.7 C, los 0x8b, tx_bias 7.5 7.6 7.6 0.0 mA, tx_power 792 772 821 0 uW, rx_power 0 0 466 0 uW
- CDM link status: /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
- fiber disconnected, no link:
CDM firmware: 0xbb2f0ae7
0x1000 SFP c.c. status: 0x00000031
CLK_IN_SEL_LS 0x1
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x00000025
sfp_mod_absent_N 1
sfp_rx_los_N 0
link_power_good 1
rx_link_up 0
rx_receiving_data 0
rx_error 1
rx_lnk_up_and_running 0
tx_link_up 0
tx_sending_data 0
tx_link_up_and_running 0
link_up_and_running 0
0x1014 SFP link data: 0x466a8187
rx_data 0x8187
k28p1_k28p5 0
rx_ctrl0 0x1
rx_ctrl1 0x1
rx_ctrl3 0x1
tx_state 0x1
rx_state 0x6
rx_receiving_data 0
rx_link_up 0
rx_error 1
rx_link_rst 0
0x2000 link test mode: 0x00000200, seconds: 0x00079093, errors: 0xffffffff
- fiber connected, good link:
CDM firmware: 0xbb2f0ae7
0x1000 SFP c.c. status: 0x000000b2
CLK_IN_SEL_LS 0x2
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x000007dc
sfp_mod_absent_N 0
sfp_rx_los_N 0
link_power_good 1
rx_link_up 1
rx_receiving_data 1
rx_error 0
rx_lnk_up_and_running 1
tx_link_up 1
tx_sending_data 1
tx_link_up_and_running 1
link_up_and_running 1
0x1014 SFP link data: 0x35c02774
rx_data 0x2774
k28p1_k28p5 0
rx_ctrl0 0x0
rx_ctrl1 0x0
rx_ctrl3 0x0
tx_state 0x3
rx_state 0x5
rx_receiving_data 1
rx_link_up 1
rx_error 0
rx_link_rst 0
0x2000 link test mode: 0x00000200, seconds: 0x00078a8c, errors: 0x00000000
- GDM link status, 1 link connected, no errors: /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
GDM firmware: 0x6b2ee010 0x1014: 0x00000008, 18: 0x00000008, 1C: 0x00000000, 24: 0x00000fff 0x2000: 0x00000200, time: 0x00078aa4, errors: 0xffffffff 0xffffffff 0xffffffff 0x00000000 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff
dsvslice integration
VX setup
- general
- Start acq from user code = y, all others = n (as of Mar 2023: will start when first trigger received)
- Use NIM IO = y
- Use external clock = y
- LVDS quartet is input = n, y, n, y
- LVDS quartet mode = User, User, User, User
- trigger from front panel NIM:
- Trigger on external signal = y, all others = n
- connect CDM EXT_OUT(2) to VX "TrigIn"
- trigger from LVDS "Sync" mode
- Trigger on LVDS Sync signal = y, all others = n
- LVDS quartet mode = User, Sync, User, User
- trigger from LVDS "User" mode
- Trigger on LVDS pair 12 signal = y, all others = n
- LVDS quartet is input = n, y, n, y
- LVDS quartet mode = User, User, User, User
GDM setup
- GDM is gdm0
- set inputs to NIM mode
- set outputs to TTL mode (this GDM has wrong NIM output circuit)
- use top QSFP slot, connect split cable 1 into CDM01, cable 2 into CDM02
- connect non-inverted NIM trigger signal to top-LEMO-left EXT_IN_LV(1)
- connect non-inverted NIM TSM signal to top-LEME-right EXT_IN_LV(2)
- GDM LEDs: TRIG, TSM, trigger enabled, trigger_out
- GDM LEMO_OUT: trigger, trigger
CDM setup
- set CDM LEMO inputs to NIM
- set CDM LEMO outputs to NIM
- CDM01 is cdm0
- CDM02 is cdm1
- connect GDM fiber links to SFP port
- connect 1st VX port of CDM01 to VX1
- connect 1st VX port of CDM02 to VX2
- connect LEMO EXT_OUT(2) to VX "TrigIn", CDM01 to VX1, CDM02 to VX2
- power up
- CDM LEDs: GDM TRIG, GDM TSM, trigger enabled, trigger out
- CDM LEMO_OUT: gdm_trg, trigger
After power up
- start the CDM frontend from the MIDAS "Programs" page. To start manually, see the Start Command on the Programs page.
- CDM frontend should enable the VX clock, disable the trigger
- from the MIDAS status page, goto the CDM page
- in the CDM01, CDM02 data table, the 2nd number should read 0x35c08008, if it does not and the last 4 digits randomly change, reset the GDM links:
- in order GDM, CDM01, CDM02, press "reset1" of each board, then press "reset4", then press "reset0", 2nd number should read as above. if it does not, STOP HERE.
- start a run
- CDM frontend will enable the trigger
- GDM frontend will enable the trigger
- LEDs on the GDM should flash, LEDs on the CDM should flash, TrigIn and TrigOut of the VX should flash
- stop a run
- GDM frontend will disable the trigger
- CDM frontend will disable the trigger
Phase measurement
- pip3 install matplotlib
- pip3 install scipy
- export PYTHONPATH=$HOME/packages/midas/python
- #git clone https://github.com/J033X071C/PhaseMeasurement
- git clone https://bitbucket.org/team-ds-dm/phasemeasurement.git
- cd phasemeasurement
- python3 ./phaseMeasurement.py --help
daq00:PhaseMeasurement$ python3 ./phaseMeasurement.py --help usage: phaseMeasurement.py [-h] fileName numberEvents numberVX sizeEvents stopEvent minHist maxHist numberBin writeToTXT saveAsPDF Read data from midas file (in .lz4 format) to calculate phase between the clock of VX1 and VX2 positional arguments: fileName Name of the file we want to read data from (Example: run00389.mid.lz4) numberEvents Number of events recorded in the file numberVX Number of VX used in this run (usually 2...) sizeEvents Number of points per event stopEvent Number of events you want to go through to calculate phase minHist Minimal value for the x axis of the phase measurement histogram (in ns) maxHist Maximal value for the x axis of the phase measurement histogram (in ns) numberBin Number of bins wanted for the generated histogram writeToTXT Write argument as yes to generate text file with results of calculation saveAsPDF Save generated plots to PDF files optional arguments: -h, --help show this help message and exit daq00:PhaseMeasurement$
- try an old file with
- python3 ./phaseMeasurement.py run00877.mid.lz4 10000 2 10000 500 -20 20 81 yes yes
bin size = 0.494 ns num_events = 347 mean = -1.705 ns rms = 3.087 ns mean_error = 0.166 ns centroid = -1.706 ns. width (sigma) = 0.363 ns. error on the centroid = 0.016558 ns.
- ls -l *.txt *.pdf
dsdaq@dsvslice:~/online/PhaseMeasurement$ ls -l *.txt *.pdf -rw-rw-r-- 1 dsdaq dsdaq 64728 Dec 14 16:56 run00877.mid.lz4_Plots.pdf -rw-rw-r-- 1 dsdaq dsdaq 274 Dec 14 16:56 run00877.mid.lz4.txt dsdaq@dsvslice:~/online/PhaseMeasurement$
Standalone link test
CDM: program clock chip busybox devmem 0x80011000 32 0x8 busybox devmem 0x80011000 32 0x0 /home/dsdaq/si5394-i2c-file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b GDM, CDM: link reset busybox devmem 0x80011008 32 1 GDM, CDM: release reset busybox devmem 0x80011008 32 0 CDM: link status (NOTE: SFP LOS and mod_absent are swapped!!!) busybox devmem 0x80011010 32 0x00000024 <- fiber plugged 0x00000025 <- fiber unplugged 0x00000027 <- SFP unplugged 0x000007DC <- successful link with GDM CDM: link state machine and data busybox devmem 0x80011014 32 0x35C06FF6 CDM: set link to counting mode busybox devmem 0x80012000 32 0x101 busybox devmem 0x80012000 32 0x100 CDM: time counter and error counter root@cdm1:~# busybox devmem 0x80012000 32 0x00000100 <--- link mode root@cdm1:~# busybox devmem 0x80012004 32 0x0000058C <--- seconds counter root@cdm1:~# busybox devmem 0x80012004 32 0x0000058D root@cdm1:~# busybox devmem 0x80012008 32 0x00000000 <--- error counter GDM: no link root@gdm0:~# busybox devmem 0x80011014 32 0x00000000 root@gdm0:~# busybox devmem 0x80011018 32 0x00000000 root@gdm0:~# busybox devmem 0x8001101c 32 0x00000000 root@gdm0:~# busybox devmem 0x80011024 32 0x00000FFF root@gdm0:~# GDM: good link channel 10, counting mode root@gdm0:~# busybox devmem 0x80012000 32 0x101 root@gdm0:~# busybox devmem 0x80012000 32 0x100 root@gdm0:~# busybox devmem 0x80012008 32 0x3A8B68C2 root@gdm0:~# busybox devmem 0x80012008 32 0x42E03BEF root@gdm0:~# busybox devmem 0x8001200c 32 0xDA090972 root@gdm0:~# busybox devmem 0x8001200c 32 0xDE6F22E9 root@gdm0:~# busybox devmem 0x80012019 32 Bus error root@gdm0:~# busybox devmem 0x80012010 32 0xFFFFFFFF root@gdm0:~# busybox devmem 0x80012014 32 0xFFFFFFFF root@gdm0:~# busybox devmem 0x80012018 32 0xFFFFFFFF root@gdm0:~# busybox devmem 0x8001201c 32 0xFFFFFFFF root@gdm0:~# busybox devmem 0x80012020 32 0xFFFFFFFF root@gdm0:~# busybox devmem 0x80012024 32 0xFFFFFFFF root@gdm0:~# busybox devmem 0x80012028 32 0x00000000 root@gdm0:~# busybox devmem 0x8001202c 32 0xFFFFFFFF root@gdm0:~# busybox devmem 0x80012030 32 0xFFFFFFFF root@gdm0:~# busybox devmem 0x80012034 32 0xFFFFFFFF root@gdm0:~#
mapping of link channels:
qsfp0 lane0 - 0x0100 - link 8 qsfp0 lane1 - 0x0200 - link 9 qsfp0 lane2 - 0x0400 - link 10 qsfp0 lane3 - n/c qsfp1 lane0 - 0x0800 - link 11 qsfp1 lane1 - 0x0010 - link 4 qsfp1 lane2 - 0x0020 - link 5 qsfp1 lane3 - n/c qsfp2 lane0 - 0x0040 - link 6 qsfp2 lane1 - 0x0080 - link 7 qsfp2 lane2 - 0x0001 - link 0 qsfp2 lane3 - n/c qsfp3 lane0 - 0x0002 - link 1 qsfp3 lane1 - 0x0004 - link 2 qsfp3 lane2 - 0x0008 - link 3 qsfp3 lane3 - n/c
script to start the test with 2 CDMs:
ssh dsdaq@dsvslice ssh root@gdm0 busybox devmem 0x80011008 32 1 ssh root@cdm0 busybox devmem 0x80011008 32 1 ssh root@cdm1 busybox devmem 0x80011008 32 1 ssh root@gdm0 busybox devmem 0x80011008 32 0 ssh root@cdm1 busybox devmem 0x80011008 32 0 ssh root@cdm0 busybox devmem 0x80011008 32 0 ssh root@gdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2 ssh root@cdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2 ssh root@cdm1 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
DS-20K DAQ
Overview
DS-DM, GDM and CDM are key parts of the DS-20K DAQ system:
- common clock distribution from external clock (atomic clock, GPS) to GDM to per-quadrant CDMs to VX digitizers
- common trigger distribution from GDM internal algorithm or external input to all VX digitizers
- run control: GDM, CDM, VX all start recording data at the same time (clock and timestamp reset)
- collection of trigger data from VX digitizers to per-quadrant CDMs to GDM
Deliverables
- hardware and firmware for GDM to CDM clock distribution
- hardware and firmware for CDM to VX clock distribution
- hardware and firmware for GDM external clock input (atomic clock or GPS)
- hardware and firmware for CDM and VX serial communications (VX LVDS I/O connector)
- firmware for run control (timestamp reset and sync): GDM to CDM to VX
- firmware for common trigger distribution: GDM to CDM to VX
- firmware for trigger data flow: VX to CDM to GDM
- firmware for busy control: VX to CDM to GDM back to CDM to VX
- firmware for flow control: FEP to GDM MIDAS frontend to GDM to CDM to VX
- GDM MIDAS frontend: clock selector and monitoring, trigger and run control, busy and flow control, GDM housekeeping
- CDM MIDAS frontend: clock monitoring, CDM housekeeping
specific performance:
- GDM external clock: 10 MHz GPS clock
- GDM to CDM fiber link:
- clock XXX MHz
- link data rate: XXX Gbit/sec
- CDM recovered clock: XXX MHz
- CDM recovered clock jitter: XXX ns
- phase alignment between CDMs: XXX ns
- phase alignment between CDMs persists across reboots, power cycles, firmware updates
- phase alignment between CDMs should be easy to measure
- phase alignment between CDMs should be easy to recalibrate if hardware parts are replaced (DS-DM boards, fiber transceivers, fiber cables, etc)
- data packet bandwidth: XXX Mbytes/sec
- data packet latency: XXX clocks
- data packet skew between CDMs: XXX clocks
- CDM to VX clock:
- clock: XXX MHz
- jitter, all CDM clock outputs: XXX MHz
- phase alignment between all CDM clock outputs: XXX ns
- CDM to VX trigger:
- TBD (use the VX "sync" input or VX LVDS I/O line or VX serial link packet)
- CDM to VX serial link:
- clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
- bit rate: XXX bits/sec
- latency: XXX link clocks
- maximum skew between VXes: XXX ns
- VX to CDM serial link:
- clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
- bit rate: XXX bits/sec
- latency: XXX link clocks
- maximum skew between VXes: XXX ns
- timestamp reset:
- maximum skew between VXes: XXX ns
- busy round trip time: XXX ns (VX to CDM to GDM back to CDM to VX)
- flow control latency: XXX ns (FEP software to GDM MIDAS frontend to GDM to CDM to VX)
Technical risk items
this refers to unexpected behaviour and performance of system components, causes big difficulty in implementing the system, prevents delivery of deliverables, and prevents or negatively affects operation of the DS-20K DAQ or of the whole experiment.
(14-sep-2022, list is not sorted by any criterial: severity, probability, ease of investigation)
(stability of course is long term stability, across hours, days, weeks, months, years)
- stability of Enclustra FPGA modules (crashes/year, failures to boot/year, flash corruption/year)
- stability of GDM external clock PLL (lock loss/year)
- stability of CDM recovered clock (lock loss/year, unexpected phase drifts, etc)
- unexpected failures or bit error rates in GDM-CDM fiber links
- stability of CDM VX clock outputs (stability of clock cleaner chip)
- stability of VX internal clock distribution (VX PLL lock loss events)
- stability of VX CAEN base firmware (different versions of CAEN base firmware have different clock distribution behaviour)
- strange things in CAEN base firmware (unexpected clocking of LVDS I/O, unexpected phase shifts between clocks, etc)
- DS-DM and VX hardware problems (incompatible LVSD I/O, incompatible clock signals, etc)
Milestones
(14-sep-2022: at this stage of the project, priority must be given to identifying and retiring (so called) technical risk factors. it is not good to build the complete system only to discover that (for example) some Enclustra FPGA modules require 5 attempts to boot and erase their flash memory contents once a month. Both example are real-life actual problems that caused big difficulties in GRIFFIN/TIGRESS and ALPHA-g experiments).
Development and testing milestones in time reversed order:
- full DAQ data challenge: all VXes, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
- one quadrant data challenge: 1 VME crate of VX, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
- vertical slice data challenge: 1 VME crate, 2 VX, 2 CDM (1 VX per CDM), GDM, etc operate as designed
- GDM-CDM link finalized (data rate frozen, data packet format frozen, data content permitted to change)
- CDM-VX serial link finalized (data rate frozen, data packet format frozen, data content permitted to change)
- run control (timestamp reset) and trigger distribution design agreed upon, frozen (list of possible triggers permitted to change)
- VX to CDM to GDM data flow design agreed upon, frozen (data contents permitted to change)
- major technical risk items retired (all hardware and firmware is working as expecred without mysteries and surprises, all problems are identified, investigated, resolved, solutions tested)
- stable operation of CDM-VX serial links in vertical slice system
- stable operation of GDM to CDM clock in vertical slice system
- stable operation of CDM to VX clock in vertical slice system
- vertical slice system assembled (1 VME crate, 2 VX, 2 CDM, 1 GDM, network, FEP, TSP, MIDAS)
ZZZ
ZZZ