DS-DM: Difference between revisions
Jump to navigation
Jump to search
| Line 3,575: | Line 3,575: | ||
* from the MIDAS status page, goto the CDM page | * from the MIDAS status page, goto the CDM page | ||
* outdated: in the CDMx data tables, the 2nd number should read 0x35c08008, if it does not and the last 4 digits randomly change, reset the GDM links | * outdated: in the CDMx data tables, the 2nd number should read 0x35c08008, if it does not and the last 4 digits randomly change, reset the GDM links | ||
*if the | *if the FEs complain - do in order: for GDM, CDM01, CDM02, ..., press "reset mgt" of each board, then press "unreset mgt", if it does not help, STOP HERE | ||
* start a run | * start a run | ||
* CDM frontend will enable the trigger | * CDM frontend will enable the trigger | ||
Revision as of 14:59, 5 November 2024
DS-DM
DarkSide-20k Global and Crate Data Manager board (GDM and CDM).
Global Data Manager (GDM):
- clock distribution to CDM boards (including GPS/atomic clock source)
- collection of trigger data from CDM boards, processing and distribution of trigger decision to CDM boards
- run control
- integration with GPS 10MHz and 1pps clocks and GPS/IRIG date and time information
Crate Data Manager (CDM):
- clock distribution from GDM to CAEN VX digitizers
- receive trigger data from CAEN VX digitizers
- send trigger data to GDM
- run control and dead time control
Links
- https://dsvslice.triumf.ca/vx_napoli - DS vertical slice at TRIUMF
- https://edev-group.triumf.ca/fw/exp/darkside/gcdm - git repository, DS-DM firmware
- https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0 - git repository, DS-DM board
- https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0/-/blob/master/Altium/Project%20Outputs%20for%20DS-DM-Rev0/SCH-DS-xDM-Rev0.PDF - DS-DM schematics
- https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0/-/blob/master/Testing/Clk3_XO_125_to_fpgaIN_recoveredMGTclk_to_IN2_Si5394-RevA-Project.slabtimeproj
- https://docs.xilinx.com/v/u/en-US/ds593 - Xilinx Platform Cable USB II
- https://www.enclustra.com/en/products/system-on-chip-modules/mercury-xu8/ - Enclustra
- https://bitbucket.org/team-ds-dm/ds-dm-software - git repository, DS-DM MIDAS frontend
- https://bitbucket.org/team-ds-dm/phasemeasurement - VX Phase measurement script
- https://ladd00.triumf.ca/daqinv/frontend/list/178 - inventory database
- https://daq00.triumf.ca/elog-ds/DS-DAQ - DS-DAQ elog
- https://bitbucket.org/ttriumfdaq/dsproto_vx2740/src/develop/ - VX DSFE MIDAS frontend
- https://dsdaqgw.triumf.ca/vslice/ - vertical slice DAQ
- https://daq00.triumf.ca/DaqWiki/index.php/DarkSide - DS-DAQ Wiki page
- https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0 - GPS and Rb Clock adapter board
Onboard hardware
- jtag chain: arm_dap_0 0x5BA00477, xczu4_1 0x04721093
- Eclustra Mercury+ XU8 module: ME-XU8-4CG-1E-D11E-R2.1
- Xilinx® Zynq Ultrascale+™ MPSoC XCZU4CG-1FBVB900E
- DDR4 ECC SDRAM (PS) 2 GB
- DDR4 SDRAM (PL) 1GB
- ethernet mac chip: AT24MAC402-SSHM-T ("602" chip is wrong)
- USB UART for Enclustra serial console, micro-USB, 115200n8
- clock chip: SI5394A-A-GM and oscillator CS-044-054.0M (54 MHz)
- U23 3.3V current meter and thermometer, LTC2990IMS#TRPBF
- LEDs:
- LED_FP A/B/C/D 0/1/2/3
- led1 - 3V3_SW_ON, SOM_POWER_GOOD - Enclustra FPGA module 3.3V power is good
- led2 - LTM4624 PGOOD
- led3 - FPGA_DONE - FPGA has booted
- led4 - TP-S-1, PCLK_P
- led5 - TP-S-2, PCLK_N
- LEMO connectors (top to bottom)
- J4 - input (NIM/TTL) (EXT_IN_LV(1), EXT_IN_LV(2))
- J5 - input (NIM/TTL) (EXT_IN_LV(3), EXT_IN_LV(4))
- J6 - external clock (GPS 10MHz and PPS)
- J7 - output (NIM/TTL) (EXT_OUT(1), EXT_OUT(2))
- SMA connectors
- J9, J10 - CLK_CCA from U6 C.C.
- J11, J12 - CLK_TP0
- RJ45 ethernet connector (100mbit: green light on, yellow light flashes when there is traffic)
- SFP connector (SFP is FTLF8526P3BNL, 6 Gbit/sec, 850 nm, 300m 50/125um OM3 MMF)
- 4 QSFP connectors (GDM)
- 6 VX connectors (CDM)
Buttons, jumpers and switches
Buttons:
- PB1 - HRST - reboot FPGA (power-on reset)
- PB2 - SRST - (SRSTn) - reboot ARM CPU
Switches:
- SW1 - boot mode BM0, BM1 [-->]
- SW2 - LEMO output NIM<->TTL
- SW3 - LEMO input 1 and 2 NIM/TTL
- SW4 - LEMO input 2 and 4 NIM/TTL
- SW5 - LEMO clock input NIM/TTL
- SW6 - serial console select. [PS<--PL] PS is ARM CPU, PL is FPGA.
Front panel
| top | | LED-FP1 | LED_FP(0,1,2,3) | | SFP J??? | | LEMO J4-LEFT, J4-RIGHT | EXT_IN_LV(1), EXT_IN_LV(2) | LEMO J5-LEFT, J5-RIGHT | EXT_IN_LV(3), EXT_IN_LV(4) | LEMO J6-LEFT, J6-RIGHT | CLK_EXT1, CLK_EXT0 (125 MHz only) | LEMO J7-LEFT, J7-RIGHT | EXT_OUT(1), EXT_OUT(2) | | J-VX-1 | J-VX-2 or QSFP-1 | J-VX-3 or QSFP-2 | J-VX-4 or QSFP-3 | J-VX-5 or QSFP-4 | J-VX-6 | | RJ45 J3 ethernet | | bottom
VX adapter board
LVDS I/O connector
34 pin connector: 0|:::: :::: :::: :::: :|15,16 n/c
split-cable connection
0 .. 7 -> N/C 8 -> VX_RX(3) - not used 9 -> VX_RX(2) - busy VX to CDM 10 -> VX_RX(1) - DS20K 125 MHz serial data VX to CDM 11 -> VX_RX(0) - DS20K 62.5MHz clock VX to CDM 12 <- VX_TX(0) - TRG CDM to VX 13 <- VX_TX(1) - TSM CDM to VX, to be VETO CDM to VX 14 <- VX_TX(2) (set by jumper routed here or to VX CLKIN SYNC) - 125 MHz serial data CDM to VX 15 <- n/c (CLK routed to VX CLKIN CLK) - 62.5 MHz clock CDM to VX 16 - n/c
one-to-one connection
0 -> VX2_RX(3) 1 -> VX2_RX(2) 2 -> VX2_RX(0) 3 -> VX2_RX(1) 4 <- CLK 5 <- VX2_TX(0) 6 <- VX2_TX(1) 7 <- VX2_TX(2) 8 -> VX1_RX(3) 9 -> VX1_RX(2) 10 -> VX1_RX(1) 11 -> VX1_RX(0) 12 <- VX1_TX(0) 13 <- VX1_TX(1) 14 <- VX1_TX(2) (set by jumper routed here or to VX CLKIN SYNC) 15 <- n/c (CLK routed to VX CLKIN CLK) 16 - n/c
Board schematics
- File:SCH-DS-xDM-Rev0.PDF
- note: FPGA pin annotations ("IO", "SCLK", "PCLK", etc) on the schematics are bogus, instead, trace them to the FPGA pins.
- note: ENC A is J800, ENC B is J801, ENC C is J900 (schematic name to enclustra name)
- note: Enclustra special pins: "GC" is "clock capable", "HDGC" is "clock capable", "MGTREFCLK" is MGT reference clocks.
- board modifications:
- ethernet mac chip
- NIM output (no U15, etc)
- RJ45 wrong pinout (board mod or special ethernet cable)
- 125 MHz clock mods (TBW)
- disconnect QSFP0_SEL from SFP_RS0 and QSFP1_SEL from SFP_RS1, these signals are not used by modern SFPs
- provide SFP i2c modsel to allow SFP and QSFP at the same time on address 0x50
FPGA MGT blocks
* SFP - ENC B45,B47 and B48,B50 - FPGA D5,D6 MGTHTX0_D and D1,D2 MGTHRX0_D * QSFP0 TX0 - ENC C13,C17 - FPGA H5,H6 MGTHTX0_C * QSFP0 TX1 - ENC C21,C25 - FPGA G7,G8 MGTHTX1_C * QSFP0 TX2 - ENC C29,C23 - FPGA F5,F6 MGTHTX2_C * QSFP1 TX0 - ENC C37,C41 - FPGA E7,E8 MGTHTX3_C * QSFP1 TX1 - ENC C45,C47 - FPGA P5,P6 MGTHTX0_B * QSFP1 TX2 - ENC C51,C53 - FPGA M5,M6 MGTHTX1_B * QSFP2 TX0 - ENC C57,C59 - FPGA L3,L4 MGTHRX2_B * QSFP2 TX1 - ENC C63,C65 - FPGA K5,K6 MGTHTX3_B * QSFP2 TX2 - ENC C75,C77 - FPGA W3,W4 MGTHTX0_A * QSFP3 TX0 - ENC C79,C81 - FPGA V5,V6 MGTHTX1_A * QSFP3 TX1 - ENC C85,C87 - FPGA T5,T6 MGTHRX2_A * QSFP3 TX2 - ENC C89,C91 - FPGA R3,R4 MGTHTX3_A
Clock distribution
Simplified:
- 125 MHz osc -> CLK_XO_125 -> MGTREFCLK0_A -> not used
- 125 MHz osc -> CLK3_XO_125 -> MGTREFCLK1_B -> SFP RX ref clock, QSFP RX and TX ref clock (this is not final design!)
- (disconnected) 125 MHz osc -> CLK2_XO_125 -> FPGA AG8,AH8 (GC)
- 125 MHz osc -> C.C. in1
- C.C. in0 <- CLK_EXT1 (10 MHz GPS clock)
- C.C. in1 <- 125 MHz osc
- (disconnected) C.C. in2 <- CLK_CC_IN <- MGTREFCLK0_D <- SFP RX clock (cannot be used because of uncontrollable phase)
- C.C. in2 <- CLK2_XO_125 <- FPGA AG8,AH8 (GC) <- SFP RX recovered 125 MHz clock
- C.C. in3 <- CLK_FB
- C.C. 125 MHz -> CLK_CC_OUT0 -> MGTREFCLK0_B -> QSFP RX and TX ref clock (final design)
- C.C. 125 MHz -> CLK_CC_OUT1 -> MGTREFCLK1_D -> SFP TX clock
- C.C. 62.5 MHz -> VX clock fanout
proposed changes:
- add C.C. 125 MHz -> new CLK_CC_OUT2 (old CLK2_XO_125) -> FPGA AG8,AH8 (GC)
- change C.C. in2 <- new CLK_CC_IN <- FPGA AK8,AK9 (non-GC)
Complete:
125 MHz oscillator - U5 fan out - q0 -> CLK_XO_125 -> ENC C72,C74 -> FPGA R7,R8 MGTREFCLK0_A (not used) q1 -> U6 C.C. in1 q2 -> CLK3_XO_125 -> ENC C7,C9 -> FPGA J7,J8 MGTREFCLK1_B -> SFP RX reference clock, QSFP RX and TX reference clocks (not final design!) q3 -> disconnected on the board, was CLK2_XO_125 -> ENC C151,C153 -> FPGA AG8,AH8 (GC) U6 C.C (clock cleaner) - in0 <- CLK_EXT1 (presumably GPS 10 MHz ref clock) in1 <- 125 MHz oscillator via U5 in2 <- (was: CLK_CC_IN <- ENC B10,B12 <- FPGA D9,D10 MGTREFCLK0_D <- SFP RX clock, 125 MHz) in2 <- CLK2_XO_125 <- ENC C151,B153 <- FPGA AG8,AH8 (GC) <- mgt_link_data_to_processing.rx_data_clk (SFP RX data clock, 125 MHz) in3 <- CLK_FB out0 -> CLK_CCA -> U12 (125 MHz) out1 -> CLK_CCB -> VX1..6 (62.5 MHz) out2 -> CLK_CCC -> VX7..12 (62.5 MHz) out3 -> CLK_FB into in3 CLK_CCA -> U12 (125 MHz fan out) -> Q0 -> not used Q1 -> CLK_CC_OUT0 -> ENC C3-5 -> FPGA L7,L8 MGTREFCLK0_B -> QSFP RX and TX reference clocks (final design) Q2 -> CLK_CC_OUT1 -> ENC B3-5 -> FPGA B9,B10 MGTREFCLK1_D -> SFP TX clock Q3 -> not used Q4 -> not used Q5 -> SMA J9/J10 CLK_TP0 - FPGA N7,N8 MGTREFCLK1_A - ENC C69,C71 - DS-DM SMA J11, J12 (NOT IN CDM PROJECT) Notes: * CLK_XO_125 (125 MHz osc) is not used * 62.5 MHz VX clock does not go into the FPGA * CLK_EXT0 going to in1 of U5 cannot be used. only permitted frequency is 125 MHz (it drives the MGT reference clocks) and is this frequency is too high for LEMO cables and connectors Proposed modifications: - CLK_CC_IN: from FPGA output pin (ENC C142,C144 - FPGA AK8,AK9) to clock cleaner in2 (instead of CLK2_XO_125 pins) - CLK_CCA -> U12 -> currently unused out3 -> CLK2_XO_125 FPGA pins - repurpose CLK_EXT0 at GPS 1pps/IRIG input to FPGA
I2C bus
- I2C_SCL is J-ENC-A pin 111 I2C_SCL is FPGA I2C_SCL_PL AB13 (IO_L1N_TOL1D_64) and I2C_SCL_PS F18 (PS_MIO10)
- I2C_SDA is J-ENC-A pin 113 I2C_SDA is FPGA I2C_SDA_PL AH13 (IO_L7N_T1L1Q_AD13N_64) and I2C_SDA_PS G18 (PS_MIO11)
- XU8 secure EEPROM ATSHA204A at 0x64, this is 0110'010X -> linux _011'0010 is 0x32. (but responds to scan and read at 0x33)
- U4 ethernet mac chip, EEPROM at 1010 A2 A1 A0 X and MAC/serial_no at 1011 A2 A1 A0 X. A0=VCC, A1=VCC, A2=GND -> linux _101'0011 and _101'1011 is 0x53 and 0x5B.
- U6 clock chip, address 1101 0 A1 A0 X. A1=VCC, A0=N/C (internal pull-up) -> linux _110'1011 is 0x6b
- U23 voltmeter at 10011 ADR1 ADR0. ADR0=GND, ADR1=VCC -> linux _100'1110 is 0x4e
- SFP, address 1010000X -> linux _101'0000 is 0x50. additional SFP data at 0x51
- QSFP0, QSFP1, QSFP2, QSFP3 (QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL)
I2C clock builder connection
- use Silicon Labs USB "Clock builder pro field programmer", www.silabs.com/CBProgrammer
- connect rainbow jumper cable pins:
- black - 1-GND to GND on DS-DM
- white - 3-SCLK to SCL on the DS-DM
- grey - 7-SDA_SDIO to SDA on the DS-DM
- power up the DS-DM
- plug USB programmer into Windows laptop
- on Windows, run "ClockBuilder Pro"
- it should report "Field programmer detected", press "EVB GUI"
- in EVB GUI, press "Config", set I2C address 0x6B
- press "Scan", it should find Si5394A-A-GM
- select the "Status" tab, should see real-time status of clock chip
GDM MGT configuration
- TX configuration:
- GDM MGT transceivers are configured as "multilane" TX and RX.
- there is 12 TXes ("lanes")
- MGT reference 125 MHz clock goes into [2:0]gtreclk00_in and [11:0]gtrefclk0_in
- one MGT is designated as "master"
- PLL of master MGT converts reference clock into common TX clock and common tx_user_clk2 which becomes tx_data_clk
- common tx_user_clk2 aka tx_data_clk goes into all TXes and clocks tx_data.
- tx_user_clk2 aka tx_data_clk is 125 MHz but not same phase as MGT reference clock.
- RX interim configuration:
- there is 12 RXes ("lanes")
- each RX produces it's own recovered RX clock
- "multilane" configuration assumes all RX recovered clocks run at the same frequency (TX on the other end are driven by common TX clock, see above), but have different phase
- one RX recovered clock is designated as "master" (rx_user_clk2 aka rx_data_clk) and a phase-matching fifo/buffer is used to bring rx_data from all 12 RXes to this common rx_data_clk
- this works because each CDM SFP TX runs on the SFP RX recovered clock which is frequency-locked with the GDM QSFP TX clock.
- RX final configuration:
- MGTs permit using the common TX clock (tx_user_clk2 aka tx_data_clk) as the common rx_data_clk (they are frequency locked through the CDM).
- this permits use of tx_data_clk as the main clock domain in the GDM and removes the need to bring rx_data into the tx_data_clk domain (actually this is done in the MGT RX phase matching fifo/buffer).
Clock path
10 MHz ext clock or GDM 125 MHz oscillator -> GDM QSFP MGT reference clock 125 MHz -> MGT PLL -> tx_data_clk 125 MHz (GDM main clock domain) and TX bit clock 2.5 GHz -> GDM QSFP optic transmitter -> CDM SFP optic receiver -> CDM SFP MGT, RX reference clock is CDM 125 MHz oscillator -> MGT RX recovered clock 125 MHz (CDM main clock domain) -> CC_CLK_IN -> CDM C.C. -> CC_CLK_OUT1 -> CDM SFP TX reference clock 125 MHz -> MGT PLL -> tx_data_clk 125 MHz and TX bit clock 2.5 GHz -> (tx_data phase matching fifo from CDM main clock domain to tx_data_clk) -> CDM SFP optic transmitter -> GDM QSFP RX optic receiver (12x) -> GDM QSFP MGT (RX reference clock is same as TX reference clock) -> MGT RX recovered clock (12x recovered clocks) -> in multi-lane configuration, one of them is the "master" recovered clock rx_data_clk -> (rx_data phase matching fifo from rx_data_clk to GDM main clock domain)
CDM rx_clk mux
when CDM SFP is not connected, there is no SFP recovered clock and a mux is used to switch between clk_cc_out0 (power up default) and rx_clk_mgt (SFP recovered clock)
Test SFP disconnected
note: if I say "--cc-in1", CC seems to lock on the 10 MHz GPS external clock, to prevent this, test sequence includes reloading the CC and the reset of MGT.
./test_cdm_local.exe --write32 0x30 0x0 ### rx_clk mux select CC clock ./test_cdm_local.exe --load-cc ./test_cdm_local.exe --reset-mgt ./test_cdm_local.exe --cdm-clocks CDM clock frequency counters: 0x1030 mgt_rx_ref_clk_raw: 0x0773594e (125000014) should be ~125 MHz <=== all 4 clocks wobble close to 125 MHz 0x1034 rx_clk: 0x0773594f (125000015) should be ~125 MHz 0x1038 mgt_tx_ref_clk_raw: 0x0773594f (125000015) should be ~125 MHz 0x103C tx_clk: 0x0773594f (125000015) should be ~125 MHz 0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly 0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly ./test_cdm_local.exe --write32 0x30 0x1 ### rx_clk mux select SFP recovered clock ./test_cdm_local.exe --cdm-clocks CDM clock frequency counters: 0x1030 mgt_rx_ref_clk_raw: 0x0773598b (125000075) should be ~125 MHz 0x1034 rx_clk: 0x076d58ec (124606700) should be ~125 MHz <=== off frequency because there is no valid SFP recovered clock 0x1038 mgt_tx_ref_clk_raw: 0x0773598c (125000076) should be ~125 MHz 0x103C tx_clk: 0x0773598b (125000075) should be ~125 MHz 0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly 0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
Clock domains
GPS
- no GPS : GDM runs from internal 125 MHz oscillator
- external 10 MHz clock : GDM runs from external 10 MHz clock and optional 1pps signal (use VME-NIMIO32 NIM outputs)
- GPS receiver : GDM runs from GPS 10 MHz clock and GPS IRIG serial data
- LNGS GPS:
- provides 1pps and serial data over fiber from GPS receiver (master). LNGS xxx box is not used.
- serial data goes to GDM, decoded, 1pps signal extracted, goes to LEMO output
- 1pps from GDM is used to train the Rubidium clock which provides a 10 MHz clock
- 10 MHz output from Rubidium clock goes to GDM 10 MHz external clock input
GDM
- AXI clock (100 MHz) - AXI registers
- 125 MHz oscillator - to clock cleaner
- 10 MHz external clock LEMO input - to clock cleaner
- FPGA 125 MHz clock CLK_CC_IN - to clock cleaner - not used (could be used for cascaded GDMs)
- 125 MHz oscillator - CLK3_XO_125 - mgt_b_ref_clk QSFP MGT reference clock (interim GDM design)
- clock cleaner output 125 MHz fanout:
- CLK_CC_OUT0 - QSFP MGT reference clock (final design)
- CLK_CC_OUT1 - not used (CDM SFP reference clock)
- CLK_CC_OUT2 - not used
- QSFP MGT TX data clock 125 MHz tx_data_clk (main clock domain)
- QSFP TX data
- QSFP RX data (in final design, rx_data_clk is same as tx_data_clk. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk == tx_data_clk)
- ds20k block
- (in interim design, rx_data_clk is disconnected from tx_data_clk. rx_data_clk is the "multilane master clock" which is one of the 12x RX recovered clocks. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk)
note: all these clocks are frequency locked to 125 MHz
CDM
- AXI clock (100 MHz) - AXI registers
- 10 MHz external clock LEMO input - to clock cleaner (not used)
- 125 MHz oscillator to fanout
- to clock cleaner
- to SFP MGT RX reference clock: CLK3_XO_125 to mgt_rx_ref_clk to gtrefclk01_in
- SFP MGT RX recovered clock 125 MHz
- MGT PLL to MGT rx_user_clk2 aka rx_data_clk
- MGT rxrecclkout_out to mgt_rx_rec_clk to CLK_CC_IN to C.C.
- SFP MGT rx_user_clk2 aka rx_data_clk (250 MHz/8 bit, 125 MHz/16 bit, 62.5 MHz/32 bit data) (main clock domain)
- SFP RX data
- ds20k block
- VX TX clock PLLs
- VX RX clock PLLs
- C.C. fan out
- 62.5 MHz VX clocks (12x)
- CLK_CC_OUT0 (not used, GDM QSFP MGT reference clock)
- CLK_CC_OUT1 125 MHz to mgt_tx_ref_clk to gtrefclk00_in to SFP MGT TX reference clock
- CLK_CC_OUT2 (not used)
- SFP MGT tx_user_clk2 aka tx_data_clk
- SFP TX data
- TX data phase matching fifo from main clock domain to tx_data_clk
- VX TX clock PLLs
- 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
- VX TX data phase matching from main clock domain to VX TX clock (12 total)
- VX TX serializer
- VX TX LVDS transmitter
- VX RX clock PLLs
- 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
- VX RX LVDS receivers (12 total)
- VX RX deserializers (12 total)
- VX TX data phase matching from VX RX clock to main clock domain
VX
- everything runs on the VX main 125 MHz clock
- correct phase of VX to CDM LVDS data is adjusted by scan of CDM VX RX PLL clock phase (VX to CDM link is now established)
- correct phase of CDM to VX LVDS data is adjusted by scan of CDM VX RX PLL clock phase (link is established after VX to CDM idle data pattern changes from "VX RX data bad" to "good".
- after good phases are found by scan, they are not expected to change unless cables are changed, CDM and VX modules are changed or CDM firmware is rebuilt. (rebuild of VX firmware should not affect LVDS data phase).
- if there is excessive link errors, phase scan must be repeated.
Board test plan
To test:
- Enclustra FPGA board
- SFP port - SFP_RS0/RS1 connected to QSFP0_SEL/1_SEL is wrong? our Finisar SFP says RS0, RS1 N/C, so probably okey.
- SW5 CLK_EXT1 NIM works. TTL needs to be tested. flipping SW5 CLK_EXT0 side from NIM to TTL makes CLK_EXT1 go LOS and OOF in the clock chip. R23 and R53 should be removed?
- BOOT_MODE 0 and 1
Partial:
- U23 3.3V current meter and thermometer. V1,V2 is current monitor, same as in application note. V3,V4 is thermometer, same in application note, except capacitor C118 is 0.1u instead of 470pF. Tested ok: Tint, VCC, V1, V2, V1-V2. Test failed: TR2 reads 50-something degC instead of same as Tint.
Done:
- LED_FP1A..D: tested ok. K.O. 15 sep 2022
- USB UART: tested ok. K.O. 15 sep 2022
- J4A, J4B, J5A, J5B LEMO inputs (NIM/TTL) EXT_IN_LV(1..4). TTL threshold 1.7V, NIM threshold -0.3V. 50 Ohm termination. TTL straight, NIM inverted.
- J6A, J6B LEMO clock inputs: 10 MHz TTL from chronobox works, 10 MHz NIM from IO32 works. 18-apr-2023
- J7A, J7B LEMO outputs EXT_OUT(1), EXT_OUT(2) (NIM/TTL) (tested 23nov2022, K.O.). TTL ???, NIM inverted.
- TTL out no 50 ohm termination: 0=0V, 1=5V, rise and fall time ~5 ns
- TTL out with 50 ohm termination: 0=0V, 1=2.5V, rise and fall time <2ns
- NIM out no 50 ohm termination: 0=+50mV, 1=-1.8V, rise and fall time ~3ns
- NIM out with 50 ohm termination: 0=0V, 1=-0.9V, rise and fall time <2ns
- ethernet MAC i2c chip (K.O. 20sep2022: can read: i2cdump 0 0x5b, nov2022 have u-boot driver)
- CDM VX ports 2x(CLK, 3 tx, 4 rx) tested using test_cdm.exe and LVDS loopback in VX firmware.
- SFP i2c tested KO 22jun2023
- QSFP i2c tested KO 22jun2023
- i2c testing complete 22jun2023
- QSFP rx,tx tested 26june2023. lane0,1,2 ok, lane3 tx not connected, laser is off. qsfp0,1,2,3 all lanes ok.
- SFP rx,tx tested 26june2023. rx and tx okey. LOS ok, mod_absent ok.
Failure:
- ethernet: does not connect to alliedtelesys switch. connects to my USB-eth adapter at 100 Mbit speed. uboot mii status reports connection speed oscillating between 1000, 100 and 10. K.O. 16-sep-2022
- SFP LOS and mod_absent are swapped (in the FPGA pin definitions?)
- QSFP slot numbering is wrong.
Checklist for newly build boards
- put new board on workbench
- check - vme connector present, vme extraction handles present
- check - standoff are removed from all thru-holes
- plug Enclustra module
- check - SW6 both switches are in the "PS" position
- connect micro-usb cable from linux PC
- connect ethernet from 1gige capable network switch
- connect power from lab power supply - GND, +5V and -12V nominal, +5.7V and -12.5V actual
- power up, +5V current 2.10-2.8A, -12V current 0.05A
- on linux PC, open a new terminal, run: minicom -D /dev/ttyACM0
- in minicom window, observe messages about Xilinx first stage boot loader, etc
- on the ethernet switch, observe network link is 1gige speed (not 10mbit, not 100mbit).
- if everything boots okey, there will be a login prompt, login as root, password root.
- busybox devmem 0x80010000 # read firmware version number, i.e. 0xEDAD0A77
- TBW - test LEMO inputs
- TBW - test LEMO outputs
- TBW - test VX connectors
- TBW - test SFP connector
- TBW - test QSFP connector
Serial console
- check that linux computer has correct udev rules to allow access to /dev/ttyACM devices, see https://daq00.triumf.ca/DaqWiki/index.php/SLinstall#Configure_USB_device_permissions and https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Configure_USB_device_permissions
- connect micro-USB cable to connector J-UCB, other end connect to linux computer
- observe /dev/ttyACM0 was created
- run "minicom -D /dev/ttyACM0" (default serial settings are ok, otherwise, 115200n8)
- should have gdm-cdm login
- username root, password root
i2c
ZynqMP> i2c bus Bus 0: i2c@ff020000 ZynqMP> i2c dev 0 Setting bus to 0 ZynqMP> i2c probe Valid chip addresses: 33 4E 53 5B 6B 77 ZynqMP> i2c md 0x5b 0x98 0098: fc c2 3d 00 00 12 1a 6e 0a 90 85 04 94 10 08 50 ..=....n.......P
root@gdm-cdm:~# i2cdetect 0
Warning: Can't use SMBus Quick Write command, will skip some addresses
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0.
I will probe address range 0x03-0x77.
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f
00:
10:
20:
30: -- -- -- 33 -- -- -- --
40:
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60:
70:
root@gdm-cdm:~#
root@gdm-cdm:~# i2cdump 0 0x5b
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0, address 0x5b, mode byte
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00 ???????P?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e ........??=..??n
a0: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00 ???????P?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e ........??=..??n
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@gdm-cdm:~#
root@gdm0:~# i2cdetect -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#
root@cdm0:~# i2cdetect -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: 50 51 -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@cdm0:~#
- 0x33 - XU8 secure EEPROM (should be at 0x32)
- 0x4e - U23 current and temperature monitor
- 0x50, 0x51 - SFP
- 0x50 - QSFP, 4 QSFP modules enabled by GPIO QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL
- 0x53, 0x5b - ethernet mac eeprom
- 0x6b - U6 clock chip
U23
NOTE: Text reads double of Tint. not sure why. K.O. 21-mar-2024.
- internal temperature only
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x00 b # control register: "repeat mode, internal temperature only"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x03 # "Tint ready" and "busy", "busy is always 1 in repeat mode"
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 00 03 03 81 db 2a ce 2a 8e 00 6f 00 45 20 3f ?.????*?*?.o.E ?
10: 01 00 01 01 01 db 2a ce 2a 8e 00 6f 00 45 20 3f ?.????*?*?.o.E ?
...
readback:
reg0 - 03 - Tint ready
reg1 - 00 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB 0x81, bit 0x80 is "DV, data valid", bit 0x40 is "SS, sensor short", 0x20 is "SO, sensor open"
reg5 - Tint LSB 0xDB, Tint = 0x01DB = 475 * 0.0625 degC = 29.6 degC
reg6..F - stale data
- Tint, V1, V2, TR2, VCC
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x18 b # control register: "repeat mode, V1, V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 7f 18 7f 7f 81 b9 aa d0 aa 8f 83 5e 83 5e a0 41 ???????????^?^?A
10: 01 18 01 01 01 b9 2a d0 2a 8f 03 5e 03 5e 20 41 ??????*?*??^?^ A
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x1b9*0.0625 = 27.5 degC
reg6 - V1 MSB 0xaa, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 LSB 0xd0, V1 = 0x2ad0*305.18/1000000 = 3.3447 V (correct)
reg8 - V2 MSB 0xaa, ditto
reg9 - V2 LSB 0x8f, V2 = 0x2a8f*305.18/1000000 = 3.3249 V (correct, smaller than V1)
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x5e. TR2 = 0x35e*0.0625 = 53.875 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x41, VCC = 2.5+0x2041*305.18/1000000 = 5.019 V (correct, VCC is +5V)
3V3_SW current is (V2-V1)/0.020 = 0.99 A (about right?)
- Tint, V1-V2, TR2, VCC
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x19 b # control register: "repeat mode, V1-V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 7f 19 7f 7f 81 69 83 f2 83 f2 83 3a 83 3a a0 44 ?????i?????:?:?D
10: 01 19 01 01 01 69 03 f2 03 f2 03 3a 03 3a 20 44 ?????i?????:?: D
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x169*0.0625 = 22.5 degC
reg6 - V1 or V1-V2 MSB 0x83, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 or V1-V2 LSB 0xf2, V1-V2 = 0x3f2*19.42/1000000 = 0.0196142 V (correct, compare with V1 and V2 measured above)
reg8 - V2 or V1-V2 MSB
reg9 - V2 or V1-V2 LSB
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x3a. TR2 = 0x33a*0.0625 = 51.625 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x44, VCC = 2.5+0x2044*305.18/1000000 = 5.021 V (correct, VCC is +5V)
3V3_SW current is (V2-V1)/0.020 = 0.9807 A (about right?)
SFP
root@cdm0:~# i2cdump 0 0x50
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 04 07 00 00 00 00 40 40 0c 00 01 3d 00 00 00 ???....@@?.?=...
10: 0c 02 00 1e 46 49 4e 49 53 41 52 20 43 4f 52 50 ??.?FINISAR CORP
20: 2e 20 20 20 00 00 90 65 46 54 4c 46 38 35 32 36 . ..?eFTLF8526
30: 50 33 42 4e 4c 20 20 20 41 20 20 20 03 52 00 9d P3BNL A ?R.?
40: 00 1a 00 00 4e 33 41 42 34 4c 56 20 20 20 20 20 .?..N3AB4LV
50: 20 20 20 20 32 30 30 33 31 39 20 20 68 f0 03 de 200319 h???
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@cdm0:~#
root@cdm0:~# i2cdump 0 0x51
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 5a 00 d8 00 55 00 e2 00 90 88 71 48 8c a0 75 30 Z.?.U.?.??qH??u0
10: 21 34 01 f4 1b 58 03 e8 31 2d 04 eb 1f 07 06 31 !4???X??1-?????1
20: 31 2d 00 64 27 10 00 9e 00 00 00 00 00 00 00 00 1-.d'?.?........
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
40: 00 00 00 00 3f 80 00 00 00 00 00 00 01 00 00 00 ....??......?...
50: 01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 b7 ?...?...?......?
60: 19 9f 80 c5 0e 17 12 c4 1f 99 00 00 00 00 30 00 ??????????....0.
70: 00 00 00 00 00 00 00 00 ff ff ff ff ff ff ff 01 ...............?
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@cdm0:~#
QSFP
QSFP i2c enable lines, active low: QSFP0_SEL - JENC-A 82 - PS-MIO40 - linux gpio 378 QSFP1_SEL - JENC-A 84 - PS-MIO41 - linux gpio 379 QSFP2_SEL - JENC-A 100 - PS-MIO44 - linux gpio 382 QSFP3_SEL - JENC-A 106 - PS-MIO43 - linux gpio 381 (notice 2 and 3 are out of order)
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
# echo 378 >> /sys/class/gpio/export ### SEL0 338+40
# echo 379 >> /sys/class/gpio/export ### SEL1 338+41
# echo 381 >> /sys/class/gpio/export ### SEL3 338+43
# echo 382 >> /sys/class/gpio/export ### SEL2 338+44
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) in hi
gpio-379 ( |sysfs ) in hi
gpio-381 ( |sysfs ) in hi
gpio-382 ( |sysfs ) in hi
root@gdm0:~# echo out >> /sys/class/gpio/gpio381/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio382/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio378/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio379/direction
root@gdm0:~#
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out lo
gpio-379 ( |sysfs ) out lo
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out lo
root@gdm0:~#
echo 1 >> /sys/class/gpio/gpio381/value
echo 1 >> /sys/class/gpio/gpio382/value
echo 1 >> /sys/class/gpio/gpio378/value
echo 1 >> /sys/class/gpio/gpio379/value
cat /sys/kernel/debug/gpio
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out hi
gpio-382 ( |sysfs ) out hi
root@gdm0:~#
root@gdm0:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#
NOTICE NOTHING AT ADDRESS 0x50
ENABLE QSFP0, OBSERVE IT IS AT ADDRESS 0x50
root@gdm0:~# echo 0 >> /sys/class/gpio/gpio378/value
root@gdm0:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: 50 -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#
root@gdm0:~# i2cdump -y 0 0x50
No size specified (using byte-data access)
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 0d 00 02 8e 00 00 01 00 00 05 55 00 05 00 00 00 ?.??..?..?U.?...
10: 00 00 00 00 00 00 1e b8 00 00 81 a0 00 00 00 00 ......??..??....
20: 00 00 12 12 00 01 00 01 00 01 0e e0 0f 20 0e e0 ..??.?.?.???? ??
30: 00 00 1f a0 1e 3b 1f 72 00 01 00 00 00 00 00 00 ..???;?r.?......
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
60: 00 00 00 00 00 00 00 00 00 00 1f 00 00 00 08 00 ..........?...?.
70: 00 00 00 00 00 00 00 ff ff ff ff ff ff ff ff 00 ................
80: 0d 00 0c 04 00 00 00 40 40 02 d5 05 67 00 00 96 ?.??...@@???g..?
90: 00 00 c8 00 46 49 4e 49 53 41 52 20 43 4f 52 50 ..?.FINISAR CORP
a0: 20 20 20 20 07 00 90 65 46 54 4c 34 31 30 51 44 ?.?eFTL410QD
b0: 34 43 20 20 20 20 20 20 41 20 42 68 07 d0 00 43 4C A Bh??.C
c0: 00 07 0f de 58 37 39 41 43 30 52 20 20 20 20 20 .???X79AC0R
d0: 20 20 20 20 32 32 30 33 30 39 20 20 3c 10 00 9e 220309 <?.?
e0: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@gdm0:~#
ethernet mac eeprom
- correct chip with 84-bit ethernet mac address
root@cdm1:~# i2cdump 0 0x53
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
root@cdm1:~#
root@cdm1:~# i2cdump 0 0x5b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00 ????4??2?K?.?...
90: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c ..........??=?Q<
a0: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00 ????4??2?K?.?...
b0: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c ..........??=?Q<
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@cdm1:~#
- wrong "602" chip with 64-bit IPv6 address
root@cdm0:~# i2cdump 0 0x53
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
root@cdm0:~# i2cdump 0 0x5b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00 ???????Q?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e ........??=..??.
a0: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00 ???????Q?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e ........??=..??.
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@cdm0:~#
read ethernet mac address from i2c
(this code is copied from uboot command line i2c code)
in uboot sources board/xilinx/common/board.c replace original function with this:
// special code to read ethernet MAC address from the DS-DM-Rev0 board. K.O. Sep-2022
int zynq_board_read_rom_ethaddr(unsigned char *ethaddr)
{
struct udevice *bus;
int ret;
int busnum = 0;
ret = uclass_get_device_by_seq(UCLASS_I2C, busnum, &bus);
if (ret) {
printf("%s: No bus %d\n", __func__, busnum);
return ret;
}
int chip_addr = 0x5B;
struct udevice *dev;
ret = i2c_get_chip(bus, chip_addr, 1, &dev);
if (ret) {
printf("%s: Bus %d no chip 0x%02x\n", __func__, busnum, chip_addr);
return ret;
}
int dev_addr = 0x98;
unsigned char data[8];
ret = dm_i2c_read(dev, dev_addr, data, 8);
if (ret) {
printf("%s: Bus %d chip 0x%02x read error %d\n", __func__, busnum, chip_addr, ret);
return ret;
}
printf("%s: Bus %d chip 0x%02x addr 0x%02x read: 0x %02x %02x %02x %02x %02x %02x %02x %02x\n", __func__, busnum, chip_addr, dev_addr, data[0], data[1], data[2], data[3], data[4], data[5], data[6], data[7]);
// see Atmel-8807-SEEPROM-AT24MAC402-602-Datasheet.pdf
if (data[0] == 0) {
// eiu-48 chip
ethaddr[0] = data[2];
ethaddr[1] = data[3];
ethaddr[2] = data[4];
ethaddr[3] = data[5];
ethaddr[4] = data[6];
ethaddr[5] = data[7];
} else {
// eiu-64 chip
ethaddr[0] = data[0];
ethaddr[1] = data[1];
ethaddr[2] = data[2];
ethaddr[3] = data[5];
ethaddr[4] = data[6];
ethaddr[5] = data[7];
}
printf("%s: ethaddr %02x:%02x:%02x:%02x:%02x:%02x\n", __func__, ethaddr[0], ethaddr[1], ethaddr[2], ethaddr[3], ethaddr[4], ethaddr[5]);
return ret;
}
also this should have worked if i2c_xxx() functions were enabled in uboot:
i2c_set_bus_num(0); i2c_probe(0x5b); i2c_read(0x5b, 0x9a, ethaddr, 6);
read ethernet mac address from i2c (SHOULD WORK)
#ethernet related setup setup_eth=run readmac buildmac #read mac address from eeprom readmac=i2c dev 0; i2c read 50 FA.1 6 $loadaddr #build the ethaddr variable #not very nice, but does the job buildmac=\ e=" "; sep=" " \ for i in 0 1 2 3 4 5 ; do\ setexpr x $loadaddr + $i\ setexpr.b b *$x\ e="$e$sep$b"\ sep=":"\ done &&\ setenv ethaddr $e
read ethernet mac address from i2c (DOES NOT WORK)
this method does not work: inside board/xilinx/common/board.c:int zynq_board_read_rom_ethaddr(unsigned char *ethaddr), uclass_get_device_by_ofnode(UCLASS_I2C_EEPROM, eeprom, &dev) returns ENODEV and read from i2c does not happen. K.O. Sep-2022
Read:
- https://support.xilinx.com/s/question/0D52E00006hpKlsSAE/uboot-and-ethernet-mac-address-from-eeprom?language=en_US (last answer)
- https://github.com/Xilinx/u-boot-xlnx/blob/master/board/xilinx/common/board.c (zynq_board_read_rom_ethaddr())
Note:
- 0x5B is the i2c chip address
- 0x9A is the data offset inside the chip, see datasheet or i2c read dump above.
Edit:
- emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/u-boot/files/platform-top.h
#include <configs/xilinx_zynqmp.h> #include <configs/platform-auto.h> //#define CONFIG_I2C_EEPROM //#define CONFIG_SYS_I2C_EEPROM_ADDR 0x5b //#define CONFIG_SYS_I2C_EEPROM_ADDR_OVERFLOW 0x0 #define CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET 0x9A #error HERE!
- emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi
/include/ "system-conf.dtsi"
/ {
chosen {
xlnx,eeprom = &eeprom;
};
};
&i2c0 {
eeprom: eeprom@5b { /* u88 */
compatible = "atmel,24mac402";
reg = <0x5b>;
};
};
- components/yocto/workspace/sources/u-boot-xlnx/configs/xilinx_zynqmp_virt_defconfig
CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET=0x9A
enable VX clock
devmem 0x80010010 32 0x8; sleep 1; devmem 0x80010010 32 0x0; si5394-i2c-file /media/sd-mmcblk1p1/00_freerun.txt 0 0x6b
clock chip configuration
file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt:
- VCO is 14 GHz
- Tvco is 71.43 ps
- N0 divider is 14, frequency is 1000 MHz
- out0 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz
- out1 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
- out2 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
- out3 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz is the feedback for zero delay
clock chip monitoring
from si5395-94-92-family.pdf:
- reg 0x1: page select, set to 0 or set to 5 to read 0x53F
- reg 0x2: 0x94
- reg 0x3: 0x53 -> device is a si5394
- reg 0xC: LOSXAXB
- reg 0xD: LOS and OOF for the 4 clock inputs
- reg 0xE: LOL and HOLD
- reg 0xF: CAL_PLL
- reg 0x11: sticky bits for reg 0xC
- reg 0x12: sticky bits for reg 0xD
- reg 0x13: sticky bits for reg 0xE
- reg 0x14: sticky bits for reg 0xF
- reg 0x1C: device reset
- reg 0x1E: low power, hard reset, SYNC
- reg 0x507: currently selected input clock
- reg 0x52A: input clock select
- reg 0x535: FORCE_HOLD
- reg 0x53F: HOLD_HIST_VALID and FASTLOCK_STATUS
arm and linux benchmark
memory benchmark:
daq13$ arm-linux-gnueabi-gcc -o memcpy.armv7 memcpy.cc -march=armv7 -static -O2 scp memcpy.armv7 to ... root@gdm-cdm:~# ./memcpy.armv7 memcpy 1 KiBytes: 1288 MB/sec memcpy 2 KiBytes: 1924 MB/sec memcpy 4 KiBytes: 2554 MB/sec memcpy 8 KiBytes: 3054 MB/sec memcpy 16 KiBytes: 3262 MB/sec memcpy 32 KiBytes: 3250 MB/sec memcpy 64 KiBytes: 3456 MB/sec memcpy 128 KiBytes: 3556 MB/sec memcpy 256 KiBytes: 3780 MB/sec memcpy 512 KiBytes: 3795 MB/sec memcpy 1024 KiBytes: 3789 MB/sec memcpy 2048 KiBytes: 3729 MB/sec memcpy 4096 KiBytes: 3717 MB/sec memcpy 8192 KiBytes: 3687 MB/sec memcpy 16384 KiBytes: 3632 MB/sec memcpy 32768 KiBytes: 3529 MB/sec memcpy 65536 KiBytes: 3318 MB/sec memcpy 131072 KiBytes: 2893 MB/sec root@gdm-cdm:~#
ethernet receive:
daq13:bin$ ./ttcp -t -s -n 100000 10.0.0.24 ttcp-t: buflen=8192, nbuf=100000, align=16384/0, port=5001 tcp -> 10.0.0.24 ttcp-t: socket ttcp-t: connect ttcp-t: 819200000 bytes in 7.25 real seconds = 110358.39 KB/sec +++ ttcp-t: 100000 I/O calls, msec/call = 0.07, calls/sec = 13794.80 ttcp-t: 0.0user 0.2sys 0:07real 3% 0i+0d 760maxrss 0+2pf 1461+31csw daq13:bin$ root@gdm-cdm:~# ./ttcp.armv7 -r -s ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp ttcp-r: socket ttcp-r: accept from 10.0.0.25 ttcp-r: 819200000 bytes in 7.27 real seconds = 110098.22 KB/sec +++ ttcp-r: 212040 I/O calls, msec/call = 0.04, calls/sec = 29181.53 ttcp-r: 0.1user 5.7sys 0:07real 81% 0i+0d 584maxrss 0+2pf 125601+2699csw root@gdm-cdm:~#
ethernet transmit:
root@gdm-cdm:~# ./ttcp.armv7 -t -s -n 100000 10.0.0.25 ttcp-t: buflen=8192, nbuf=100000, align=16384/0, port=5001 tcp -> 10.0.0.25 ttcp-t: socket ttcp-t: connect ttcp-t: 819200000 bytes in 6.95 real seconds = 115078.69 KB/sec +++ ttcp-t: 100000 I/O calls, msec/call = 0.07, calls/sec = 14384.84 ttcp-t: 0.0user 0.7sys 0:06real 11% 0i+0d 584maxrss 0+2pf 1162+1017csw root@gdm-cdm:~# daq13:bin$ ./ttcp -r -s ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp ttcp-r: socket ttcp-r: accept from 10.0.0.24 ttcp-r: 819200000 bytes in 6.97 real seconds = 114841.84 KB/sec +++ ttcp-r: 161335 I/O calls, msec/call = 0.04, calls/sec = 23160.01 ttcp-r: 0.0user 1.9sys 0:06real 28% 0i+0d 760maxrss 0+2pf 80646+51csw daq13:bin$
Install Xilinx tools
- install Vivado 2020.2
login at https://www.xilinx.com/myprofile.html go to "Downloads" go to archive, find 2020.2 download Xilinx_Unified_2020.2_1118_1232_Lin64.bin sh ./Xilinx_Unified_2020.2_1118_1232_Lin64.bin banner window should open with spinner "downloading installation data" "a newer version is available" -> say "continue" next "select install type" window: provide email and password, select "download image" select directory /home/olchansk/Xilinx/Downloads/2020.2\ select "linux" and "full image" next download summary: space required 38.52 Gbytes download installation progress downloading spinner, 16 M/s 47 minutes... "download image has been created successfully". Ok. check contents of /home/olchansk/Xilinx/Downloads/2020.2 ls -l /home/olchansk/Xilinx/Downloads/2020.2 total 67 drwxr-xr-x 2 olchansk users 9 Sep 1 16:22 bin drwxr-xr-x 3 olchansk users 15 Sep 1 16:23 data drwxr-xr-x 4 olchansk users 4 Sep 1 16:22 lib drwxr-xr-x 2 olchansk users 644 Sep 1 16:22 payload drwxr-xr-x 2 olchansk users 7 Sep 1 16:22 scripts drwxr-xr-x 4 olchansk users 4 Sep 1 16:22 tps -rwxr-xr-x 1 olchansk users 3256 Nov 18 2020 xsetup daq13:2020.2$ ./xsetup spinned loading installation data xilinx design tools 2022.1 now available -> say continue "welcome" -> next "select product" -> vivado -> next -> vivado hl system edition -> next select devices: only zynq ultrascale+ mpsoc -> next select destination: /opt/Xilinx (as root, mkdir /opt/Xilinx, chmod olchansk.users /opt/Xilinx) install ... complete move /home/olchansk/Xilinx/Downloads/2020.2 to /daq/daqstore/olchansk/Xilinx/Downloads/
- install petalinux 2020.2
./xsetup "a newer version is available" -> say "continue" next "select product to install" -> select Petalinux (Linux only) -> next "select destination directory" -> select "/opt/Xilinx" (disk space required 2.64 GB) -> next "summary" -> install ... error about missing /tmp/tmp-something files "installation completed successfully" (hard to dismiss, "ok" button is partially cut-off) done? I think it failed, /opt/Xilinx/PetaLinux/2020.2/bin is empty except for petalinux-v2020.2-final-installer.run try to run it by hand, same error about /tmp/tmp-something files. strange... notice it complains about "truncate", which truncate finds ~/bin/truncate, get rid of it, try again now complains about missing texinfo and zlib1g:i386 apt install texinfo -> ok apt install zlib1g:i386 -> installs bunch of gcc stuff -> ok try again reports "already installed" -> delete /opt/Xilinx/.xinstall/PetaLinux_2020.2/, delete entries in ~/.Xilinx/registry/installedSW.xml try again success
- install vivado 2022.1 and petalinux 2022.1 - everything is pretty much the same
Petalinux
- cd PetaLinux_GDM_CDM
- petalinux-config
- enable i2c MAC address and DHCP
git clone https://bitbucket.org:/team-ds-dm/ds-dm-u-boot-xlnx.git cd ds-dm-u-boot-xlnx git checkout ds-dm-u-boot-xlnx linux-components -> uboot -> ext-local-src external u-boot local source -> ds-dm-u-boot-xlnx (path to the customized uboot git repository)
- enable DHCP
Subsystem AUTO Hardware Settings -> Ethernet Settings randomize MAC address -> NO ethernet mac address -> leave empty obtain ip address automatically -> YES
- set hostname and product names
Firmware Version Configuration -> Host name -> "ds-dm" Product name -> "Petalinux_GDM_CDM"
- configure linux kernel
petalinux-config -c kernel
- enable NFS-Root
petalinux-config Image Packaging Configuration > Root File System Type -> set to NFS Location of NFS root directory set to "/nfsroot" petalinux-config -c kernel Networking support > IP: kernel level configuration enable DHCP, BOOTP, RARP File systems > Network file systems > Root file systems on NFS
- manually fix linux kernel command line:
grep nfsroot PetaLinux_GDM_CDM/project-spec/configs/config edit CONFIG_SUBSYSTEM_BOOTARGS_GENERATED to read earlycon console=ttyPS0,115200 clk_ignore_unused panic=60 root=/dev/nfs nfsroot=/nfsroot/%s ip=dhcp rw
- check configuration in
- PetaLinux_GDM_CDM/project-spec/configs/config
- PetaLinux_GDM_CDM/project-spec/configs/rootfs_config
- PetaLinux_GDM_CDM/components/plnx_workspace/device-tree/device-tree/system-conf.dtsi
JTAG server
localhost:3121
ds20k block
module ds20k
(
// CLOCK INPUTs
input wire clk,
input reset, // pulse for power-up reset
input wire pll_is_locked, // clock cleaner PLL is locked to selected input clock
// REGISTER_DATA
input wire [255:0] [31:0] register_data_in,
output reg [255:0] [31:0] register_data_out,
input wire register_write_strobe, // pulse when AXI write transaction puts new data in register_data_in
input wire register_read_ack, // pulse after AXI read transaction captures data from register_data_out, used to read from FIFO
// GDM QSFP FIBER LINKS
output reg [11:0] [15:0] qsfp_tx_data,
output reg [11:0] [1:0] qsfp_tx_ctrl,
input wire [11:0] [15:0] qsfp_rx_data_error, // not sure what this is.
input wire [11:0] [15:0] qsfp_rx_data,
input wire [11:0] [1:0] qsfp_rx_ctrl,
input wire [11:0] qsfp_rx_is_good, // single bit indicating that RX link is up and data is good.
// CDM SFP FIBER LINKS
output reg [15:0] sfp_tx_data,
output reg [1:0] sfp_tx_ctrl,
input wire [15:0] sfp_rx_data,
input wire [1:0] sfp_rx_ctrl,
input wire sfp_rx_is_good, // single bit indicating that RX link is up and data is good.
// VX_RXs
input wire [3:0] vx1_rx,
input wire [3:0] vx2_rx,
input wire [3:0] vx3_rx,
input wire [3:0] vx4_rx,
input wire [3:0] vx5_rx,
input wire [3:0] vx6_rx,
input wire [3:0] vx7_rx,
input wire [3:0] vx8_rx,
input wire [3:0] vx9_rx,
input wire [3:0] vx10_rx,
input wire [3:0] vx11_rx,
input wire [3:0] vx12_rx,
// VX_TXs
output reg [2:0] vx1_tx_out,
output reg [2:0] vx2_tx_out,
output reg [2:0] vx3_tx_out,
output reg [2:0] vx4_tx_out,
output reg [2:0] vx5_tx_out,
output reg [2:0] vx6_tx_out,
output reg [2:0] vx7_tx_out,
output reg [2:0] vx8_tx_out,
output reg [2:0] vx9_tx_out,
output reg [2:0] vx10_tx_out,
output reg [2:0] vx11_tx_out,
output reg [2:0] vx12_tx_out,
// remove input wire gdm_trg,
// remove input wire gdm_tsm,
// LEMO INPUTs
input wire [4:1] ext_in_lv_async, // direct connection to LEMO connectors, not clocked
// LEMO OUTPUTs
output reg [2:1] ext_out, // direct connection to LEMO connectors, not clocked
// FRONT PANEL LEDs
output reg [3:0] fp_led_out, // direct connection to LEMO connectors, not clocked
// trigger and tsm output
// remove output reg trg_out,
// remove output reg tsm_out
);
world view
Note:
- red lines: clocks
- green lines: AXI/Avalon packet streams
- blue lines: serial data
description
same thing, in words:
detector digitizer, 125 MHz digital filter digital discriminator hit map, 64 bits at 125 MHz (could be 250 MHz, filter and discriminator clock) packetizer, 64 bits -> id, timestamp, 8x 8-bit words, eop 8/10 serializer, 12.5 MHz parallel in, 125 MHz serial out lvds line to CDM vx_rx BBB: also VX busy to lvds line to CDM vx_rx, do not want to depend on serial comm for vx busy, do not want to inject async data into the hit map packet stream --- CDM lvds line capture in IBUF register, vxN_rx clock, 125 MHz clock, from PLL with adjustable phase, scan phase to find sweet spot vx rx 10/8 deserializer, vxN_rx clock, stobes out parallel data every 10 clocks at 12.5 MHz vx rx phase transfer from vxN_rx clock to main CDM clock, have 10 clocks for transfer to happen from here everything is on the CDM main clock vx rx packet adapter, 12.5 MHz 8-bit packets to 125 MHz 16-bit packets, internal FIFO to avoid data overrun sfp tx mux, all VX packet streams into one CDM sfp tx packet stream BBB: capture VX RX busy to CDM main clock (IBUF register), grand-or becomes vx_rx_busy sfp tx packetizer (data,eop,vx_rx_busy -> data,k) sfp tx 16/20 serializer, 16 bit at 125 MHz in, 2500 MHz serial out (20 bits at 125 MHz) sfp tx fiber link to GDM, 2 Gigabits/sec --- GDM qsfp rx deserializer, 16 bit at 125 MHz output on main GDM clock qsfp rx depacketizer (data,k -> packet data,eop; qsfp_rx_busy) qsfp rx demux, hit map packets routed to GDM trigger logic block GDM trigger logic block looks at hit map, generates yes/no trigger decision, encodes it as a trigger packet TSM generator encodes GPS time data as a TSM packet qsfp tx mux - trigger packets, tsm packets, etc to qsfp tx packet stream qsfp tx packetizer (data,eop;trg,tsm,bsy -> data,k) NB: the same tx data is sent to all 12 qsfp tx ports, to make sure we do not accidentally desync the CDMs. qsfp tx 16/20 serializer, 16-bit at 125 MHz to 20-bit at 250 MHz to 2500 MHz serial out qsfp tx fiber link to CDM, 2 Gigabits/sec --- CDM sfp rx 20/16 deserializer, 16-bit at 125 MHz sfp rx depacketizer, (data,k -> data,eop;trg_in,tsm_in,bsy_in) sfp rx demux (in reality, noop, all packets go to same place, vx tx) vx tx mux (packets from GDM, packets with simulated waveforms from CDM midas frontend via AXI FIFO) vx tx packet adapter 16-bit at 125 MHz to 8-bit at 12.5 MHz strobed every 10 clocks. NB: most important, this 10-clock strobe runs in sync between all CDMs! vx tx phase transfer from main CDM clock to vxN_tx clock, have 10 clocks for transfer to happen from here we run on the vxN_tx clock vx txN serializer, 8-bit at 12.5 MHz to 125 MHz serial vx_txN OBUF register, vxN_rx clock, 125 MHz clock, from PLL with adjustable phase, scan phase to find sweet spot lvds line to vx BBB: bsy_in from GDM is converted from pulse to level, goes out lvds line to vx, sync to vxN_tx clock TTT: trg_in from GDM is a pulse, does out lvds line to vx, sync to vxN_tx clock. NB: tsm is always a packet, bsy is always a signal (no packet), trg can be a packet or signal. --- VX lvds data captured by 125 MHz ADC sampling clock (CAEN base firmware logic) lvds data connected to Yair's block 10/8 deserializer depacketizer (data,k -> data,eop) to avoid accidental desync, we do not send any no trg, no tsm, no bsy K-codes. demux trigger packets go their way (4x 64 bit words of data go to event header: timestamps, hitmap data) tsm packets go their way (4x 64 bit words of data go to tsm event header: timestamps, GPS time data) BBB: bsy from lvds line stops waveform acquisition TTT: trg from lvds line cause waveform acquisition, same as trg packet, but has no timestamp and other data attached to it.
Firmware registers
Block 0
0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
Block 1
Block 2
Block 3
DS20k block register map
- busybox devmem 0x80013000 32
reg | version | xDM | xx | description 0 | 0x20230731 | ALL | RO | ds20k version 0 | 0x20240118 | ALL | RW | ds20k version and command 1 | 0x20230731 | ALL | RW | scratch read/write register 2 | 0x20230731 | ALL | RW | configure inputs and outputs 3 | 0x20230731 | ALL | RW | FP_LED mux 4 | 0x20230731 | ALL | RW | EXT_OUT mux 5 | 0x20230731 | ALL | RO | VX_RX state 6 | 0x20230731 | ALL | RO | VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state 7 | 0x20230731 | ALL | RW | LED_OUT, EXT_OUT, VX_TX outputs 8 | 0x20230731 | ALL | RW | VX_TX mux and config 9 | 0x20230731 | ALL | RW | trigger config 10 | 0x20231013 | ALL | RO | status register 11 | 0x20230731 | ALL | RO | trigger counter 12 | 0x20230731 | ALL | RO | time slice marker counter 13 | 0x20240814 | ALL | RO | GPS 1pps period, 125 MHz 14 | 0x20240814 | ALL | RO | Ru clock 1pps period, 125 MHz 15 | 0x20230811 | CDM | RO | SFP RX status 16 | 0x20230811 | CDM | RW | SFP TX control 17 | 0x20230811 | GDM | RO | QSFP RX data 0, 1 18 | |. |. | 2, 3 19 | |. |. | 4, 5 20 | |. |. | 6, 7 21 | | |. | 8, 9 22 | |. |. | 10, 11 23 | 0x20230811 | GDM | RW | QSFP TX control 24 | 0x20231013 | ALL | RW | trigger pulser period 25 | 0x20231013 | ALL | RW | trigger pulser burst control 26 | 0x20231013 | ALL | RW | tsm pulser period 27 | 0x20231204 | ALL | RW | data fifo CPU to FPGA 28 | 0x20231204 | ALL | RW | packet loopback control 28 | 0x20240118 | ALL | RW | data fifo FPGA to CPU 29 | 0x20231208 | GDM | RW | bitmap of active qsfp ports 29 | 0x20240118 | ALL | RW | packet loopback control 29 | 0x20240510 | ALL | RW | packet routing 30 | 0x20231208 | GDM | RO | qsfp link status ports 0..7 31 | 0x20231208 | GDM | RO | qsfp link status ports 8..11 32 | 0x20240118 | GDM | RW | enabled QSFP ports 33 | 0x20240118 | CDM | RW | enabled VX ports 34,35 | 0x20240118 | ALL | RO | time stamp 64 bits 36,37 | 0x20240118 | ALL | RO | old time stamp 38,39,40 | 0x20240118 | CDM | RO | VX busy counters 41,42,43 | 0x20240118 | GDM | RO | QSFP busy counters 44 | 0x20240118 | CDM | RO | cdm_bsy_up_counter and cdm_bsy_pulse_counter 45 | 0x20240118 | CDM | RO | cdm_veto_pulse_counter and cdm_veto_up_counter 46 | 0x20240118 | GDM | RO | gdm_bsy_pulse_counter and gdm_bsy_up_counter 47 | 0x20240118 | GDM | RO | gdm_bsy_refresh_counter 48 | 0x20240118 | GDM | RO | gdm_veto_up_counter and gdm_veto_pulse_counter 49,50,51 | 0x20240424 | CDM | RO | VX RX serial data monitor, 8 bits per VX port 52,53 | 0x20240424 | CDM | RO | VX serial link status, 4 bits per VX port 54 | 0x20240424 | CDM | RO | VX TX serial data monitor 55 | 0x20240430 | ALL | RO | QSFP, SFP and VX link loss counters 56 | 0x20240510 | ALL | RO | sfp_rx_packet_counter 57 | same | ALL | RO | sfp_tx_packet_counter 58 | same | ALL | RO | qsfp_rx_packet_counter[0] 59 | same | ALL | RO | qsfp_tx_packet_counter 60 | same | ALL | RO | vx_rx_packet_counter[0] 61 | same | ALL | RO | vx_tx_packet_counter 62 | 0x20240719 | CDM | RO | cdm_hitmap_period, ports 0, 1 63 | same | CDM | RO | ports 2, 3 64 | same | CDM | RO | ports 4, 5 65 | same | CDM | RO | ports 6, 7 66 | same | CDM | RO | ports 8, 9 67 | same | CDM | RO | ports 10, 11 68 | 0x20240814 | ALL | RW | GPS control and status
Register 0 0x80013000 ds20k version
on read: ds20k version 0xYYYYMMDD
on write:
- 0x00 - noop - as of version 0x20240118, write a zero after writing a command
- 0x01 - CMD_reset - reset logic to good state
- 0x02 - CMD_arm_ts - arm timestamp reset
- 0x03 - CMD_trg - issue a trigger
- 0x04 - CMD_tsm - issue a tsm
- 0x05 - CMD_vx_rx_reset - reset the VX receive path
- 0x06 - CMD_vx_tx_reset - reset the VX transmit path
- 0x07 - CMD_hitmap_trg - generate a hitmap trigger and data packet
- 0x08 - CMD_trg_pulser_reset - reset the trigger pulser
- 0x09 - CMD_tsm_pulser_reset - reset the tsm pulser
Register 1 0x80013004 scratch
scratch read-write register
Register 2 0x80013008 input and output config
bit | version | fpga name | description 0 | ALL | lemo_enable | enable LEMO input 1 1 | | | 2 2 | | | 3 3 | | | 4 4 | ALL | lemo_invert | invert LEMO input 1 5 | | | 2 6 | | | 3 7 | | | 4 9 | ALL | ext_out_disable | disable LEMO output 1 10 | | | 2 11 | ALL | ext_out_invert | invert LEMO output 1 12 | | | 2
Register 3 0x8001300C FP_LED control
wire [15:0] led_out_mux_sel = register_data_in[3][15:0]; wire [3:0] led_out_invert = register_data_in[3][19:16];
led_out_mux_sel is 4 groups (one per LED) of 4 bits (choice 0..15): mux | version | fpga name | description 0 | ALL | | power on default 1 | ALL | led_out_reg | register 7 bits 2 | 0x20231013 | pll_locked | clock chip PLL is locked 3 | 0x20231013 | sfp_link_status | SFP link is good 3 | 0x20240118 | sfp_link_status or qsfp_tx_link_rx_status | SFP/QSFP link is good 4 | ALL | lemo_in_sync[1] | LEMO input 1 5 | ALL | lemo_in_sync[2] | LEMO input 2 6 | ALL | lemo_in_sync[3] | LEMO input 3 7 | ALL | lemo_in_sync[4] | LEMO input 4 8 | ALL | ext_out[1] | LEMO output 1 9 | ALL | ext_out[2] | LEMO output 2 A | 0x20231013 | trg_in | trigger B | 0x20231013 | tsm_in | time slice marker C | 0x20240118 | gdm_bsy | GDM busy: OR of all CDM busy D | 0x20240118 | cdm_bsy | CDM busy: OR of all VX busy E | 0x20240118 | cdm_veto | GDM busy -> GDM veto -> CDM veto -> VX trigger veto F | ALL | | fixed logic level 1
Register 4 0x80013010 LEMO OUT control
wire [7:0] ext_out_mux_sel = register_data_in[4][7:0];
ext_out_mux_sel is 2 groups (one per LEMO) of 4 bits (choice 0..15): mux | version | fpga name | description 0 | ALL | | power on default 1 | ALL | ext_out_reg | register 7 2 | ALL | trg_pulser | pulser trigger 3 | 0x20240724 | vx1_tx_out[2] | vx1 serial data out 4 | 0x20240724 | vx1_rx[1] | vx1 serial data in 5 | 0x20240724 | vx_rx_iob[0] | vx1 serial data in captured by IOB register 6 | ALL | lemo_in_async[1]| test synchronizer 7 | ALL | lemo_in_sync[1] | test synchronizer 8 | 0x20240724 | trg_in_pulse | trigger signal 9 | 0x20240724 | tsm_in_pulse | time slice marker signal A | 0x20240118 | cdm_bsy | CDM busy from VX B | 0x20240118 | gdm_bsy | GDM busy from CDM C | 0x20240118 | cdm_veto | veto from GDM to CDM to VX D | 0x20240118 | vx1_rx[1] | serial data VX to CDM E | not used (sink) F | ALL | 1 | fixed logic level 1
Register 5 0x80013014 VX_RX status
assign register_data_out[5] = {
vx8_rx[3], vx8_rx[2], vx8_rx[1], vx8_rx[0],
vx7_rx[3], vx7_rx[2], vx7_rx[1], vx7_rx[0],
vx6_rx[3], vx6_rx[2], vx6_rx[1], vx6_rx[0],
vx5_rx[3], vx5_rx[2], vx5_rx[1], vx5_rx[0],
vx4_rx[3], vx4_rx[2], vx4_rx[1], vx4_rx[0],
vx3_rx[3], vx3_rx[2], vx3_rx[1], vx3_rx[0],
vx2_rx[3], vx2_rx[2], vx2_rx[1], vx2_rx[0],
vx1_rx[3], vx1_rx[2], vx1_rx[1], vx1_rx[0]
};
Register 6 0x80013018 VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state
assign register_data_out[6] = {
vx2_tx_out[2], vx2_tx_out[1], vx2_tx_out[0], vx1_tx_out[2],
vx1_tx_out[1], vx1_tx_out[0], ext_out[2], ext_out[1],
fp_led_out[3], fp_led_out[2], fp_led_out[1], fp_led_out[0],
ext_in_lv[4], ext_in_lv[3], ext_in_lv[2], ext_in_lv[1],
vx12_rx[3], vx12_rx[2], vx12_rx[1], vx12_rx[0],
vx11_rx[3], vx11_rx[2], vx11_rx[1], vx11_rx[0],
vx10_rx[3], vx10_rx[2], vx10_rx[1], vx10_rx[0],
vx9_rx[3], vx9_rx[2], vx9_rx[1], vx9_rx[0]
};
bit | version | fpga name | description 0 | ALL | vx9_rx | VX_RX 1 | | | 2 | | | 3 | | | 4 | ALL | vx10_rx | VX_RX 5 | | | 6 | | | 7 | | | 8 | ALL | vx11_rx | VX_RX 9 | | | 10 | | | 11 | | | 12 | ALL | vx12_rx | VX_RX 13 | | | 14 | | | 15 | | | 16 | ALL | ext_in_lv | LEMO inputs 17 | | | 18 | | | 19 | | | 20 | ALL | FP_LED | FP_LEDs 21 | | | 22 | | | 23 | | | 24 | ALL | ext_out[1] | LEMO outputs 25 | | ext_out[2] | 26 | ALL | vx1_tx | VX1_TX 27 | | | 28 | | | 29 | ALL | vx2_tx | VX2_TX 30 | | | 31 | | |
Register 7 0x8001301C LED_OUT, EXT_OUT, VX_TX outputs
wire [3:0] led_out_reg = register_data_in[7][3:0]; wire [2:1] ext_out_reg = register_data_in[7][5:4]; // register_data_in[7][6]; // register_data_in[7][7]; wire [7:0] vx_tx_out_reg = register_data_in[7][15:8];
bit | version | fpga name | description 0 | ALL | led_out_reg | FP_LED 1 1 | | | 2 2 | | | 3 3 | | | 4 4 | ALL | ext_out_reg | LEMO OUT 1 5 | | | 2 6 | - | | 7 | - | | 8 | ALL | vx_tx_out_reg | VX1_TX 0 9 | | | 1 10 | | | 2 11 | | | - 12 | ALL | | VX2_TX 0 13 | | | 1 14 | | | 2 15 | | | - 16 | - | | 17 | | | 18 | | | 19 | | | 20 | | | 21 | | | 22 | | | 23 | | | 24 | | | 25 | | | 26 | | | 27 | | | 28 | | | 29 | | | 30 | | | 31 | | |
Register 8 0x80013020 VX_TX config
wire [3:0] vx_tx_mux_sel = register_data_in[8][3:0]; vx_tx_loopback = register_data_in[8][31];
vx_tx_mux_sel is 4 bits (choice 0..15): 0 - power on default, control by vx_tx_out_reg 1 - GDM 2 - CDM 3 - pulser loopback test 4 - pulser loopback test 5 - 62.5 MHz output 6 - 125 MHz output 7 - trg, tsm, serial 8 - trg, veto, serial 9 - GPS box control (ds20k rev 0x20240814) 10 11 12 13 14 15 - production config: trg, veto, serial
Register 9 0x80013024 trigger control
from version 0x20240724
wire [15:0] trg_src_mask = register_data_in[9][15:0]; wire [31:16] tsm_src_mask = register_data_in[9][31:16];
wire [15:0] trg_src_bits =
{
1'b0,
1'b0,
1'b0,
1'b0,
1'b0, // gdm_hitmap_trigger,
cdm_hitmap_trigger,
sfp_rx_tsm,
sfp_rx_trg,
sfp_rx_data[1],
sfp_rx_data[0],
tsm_pulser,
trg_pulser,
lemo_in_sync[4],
lemo_in_sync[3],
lemo_in_sync[2],
lemo_in_sync[1]
};
wire [15:0] trg_bits = trg_src_bits & trg_src_mask;
wire [15:0] tsm_bits = trg_src_bits & tsm_src_mask;
before that:
wire [7:0] trg_src_mask = register_data_in[9][7:0]; wire [7:0] tsm_src_mask = register_data_in[9][15:8]; wire trg_pulser_enable = register_data_in[9][16]; wire tsm_pulser_enable = register_data_in[9][17]; wire trg_software = register_data_in[9][18]; wire tsm_software = register_data_in[9][19]; // bits 20:31 not used
wire [7:0] xxx_src_bits =
{
sfp_rx_data[1],
sfp_rx_data[0],
tsm_pulser & tsm_pulser_enable,
trg_pulser & trg_pulser_enable,
lemo_in_sync[4],
lemo_in_sync[3],
lemo_in_sync[2],
lemo_in_sync[1]
};
trg_src_mask and tsm_src_mask bits: 0 - LEMO IN 1 1 - LEMO IN 2 2 - LEMO IN 3 3 - LEMO IN 4 4 - trg_pulser 5 - tsm_pulser 6 - sfp_rx_data[0] // to become sfp_trg_in, selected from sfp_rx_data[0], cdm_rx trg_out and gdm trigger packet 7 - sfp_rx_data[1] // to become sfp_tsm_in, selected from sfp_rx_data[1], cdm_rx tsm_out and gdm tsm packet
Register 10 0x80013028 status register
bit | version | fpga name | description 0 | 0x20231013 | pll_locked | clock chip PLL is locked 1 | 0x20240118 | ts_reset_armed | timestamp reset is armed 2 | 0x20240118 | qsfp_tx_link_rx_status | QSFP link status is good for all enabled ports 3 | 0x20240424 | vx_tx_link_rx_status | VX link status is good for all enabled ports 4 | 0x20240118 | cdm_bsy | VX busy grand-or 5 | 0x20240118 | gdm_bsy | QSFP busy grand-or 6 | 0x20240118 | gdm_veto | gdm_veto = gdm_busy 7 | 0x20240118 | cdm_veto | CDM veto from GDM to VX 8 | 0x20240725 | bor_started | begin-of-run sequence started 9 | 0x20240725 | bor_finished | begin-of-run sequence fininished, see commands 10 and 11 10 | | | 11 | | | 12 | | | 13 | | | 14 | | | 15 | | | 16 | | | 17 | | | 18 | | | 19 | | | 20 | | | 21 | | | 22 | | | 23 | | | 24 | | | 25 | | | 26 | | | 27 | | | 28 | | | 29 | | | 30 | | | 31 | | |
Register 11 0x8001302C trg_counter
trigger counter
Register 12 0x80013030 tsm_counter
time slice marker counter
Register 13 0x80013034 GPS 1pps period
GPS 1pps period in 8 ns clocks
Register 14 0x80013038 Rb clock 1pps period
PRS-10 Rb clock 1pps output period in 8 ns clocks
Register 15 0x8001303C SFP RX status
bit | ds20k version | fpga signal name | description 0 | ALL | sfp_rx_data[15:0] | cdm sfp received data 16 | same | sfp_rx_data_is_k[0] | 17 | same | sfp_rx_data_is_k[1] | 18 | | 0 | 19 | 0x20231204 | sfp_rx_sel_lpb | sfp tx->rx loopback 20 | 0x20231013 | sfp_link_status | sfp link connected, exchanging data 21 | same | sfp_link_rx_status | sfp link receiving correct idle pattern from GDM TX 22 | same | sfp_link_error | sfp link receiver error (badk or overflow) 23 | same | sfp_rx_data_error | sfp transceiver state machine is in error state 24 | | | 25 | | | 26 | | | 27 | | | 28 | | | 29 | | | 30 | | | 31 | | |
Register 16 0x80013040 SFP TX control
wire [15:0] sfp_tx_data_reg = register_data_in[16][15:0]; wire [1:0] sfp_tx_ctrl_reg = register_data_in[16][17:16]; wire sfp_rx_sel_lpb = register_data_in[16][28]; // TX->RX serial loopback wire sfp_tx_sel_lpb = register_data_in[16][29]; // RX->TX serial loopback wire sfp_tx_sel_trg = register_data_in[16][30]; // 16 individual bits wire sfp_tx_sel_reg = register_data_in[16][31]; // from register
Register 17-22 0x80013044 QSFP RX data
QSFP RX data links 0..11
Register 23 0x8001305C QSFP TX control
wire [15:0] qsfp_tx_data_reg = register_data_in[23][15:0]; wire [1:0] qsfp_tx_ctrl_reg = register_data_in[23][17:16]; wire qsfp_rx_sel_lpb = register_data_in[23][28]; // TX->RX loopback wire qsfp_tx_sel_lpb = register_data_in[23][29]; // RX->TX loopback wire qsfp_tx_sel_trg = register_data_in[23][30]; // 16 individual bits wire qsfp_tx_sel_reg = register_data_in[23][31]; // data from register
Register 24 0x80013060 trigger pulser period
trigger pulser period in units of 8 ns (125 MHz clock)
Register 25 0x80013064 trigger burst pulser
wire [7:0] conf_burst_count = conf_pulser_burst_ctrl[31:24]; wire [23:0] conf_burst_period = conf_pulser_burst_ctrl[23:0];
Register 26 0x80013068 tsm pulser period
time slice marker period in units of 8 ns (125 MHz clock)
Register 27 0x8001306C data write fifo
wire fifo_reset = register_data_in[27][31]; wire fifo_to_fpga_wr1 = register_data_in[27][27]; wire fifo_to_fpga_wr2 = register_data_in[27][26]; wire [16:0] fifo_to_fpga_din = register_data_in[27][16:0];
assign register_data_out[27][31:24] = register_data_in[27][31:24]; // echo write bits assign register_data_out[27][23] = fifo_to_fpga_full; assign register_data_out[27][22] = fifo_to_fpga_empty;
Register 28 0x80013070 data read fifo
wire fifo_reset = register_data_in[28][31]; wire fifo_from_fpga_rd1 = register_data_in[28][25]; wire fifo_from_fpga_rd2 = register_data_in[28][24];
assign register_data_out[28][31:24] = register_data_in[28][31:24]; // echo write bits assign register_data_out[28][21] = fifo_from_fpga_full; assign register_data_out[28][20] = fifo_from_fpga_empty; assign register_data_out[28][16:0] = fifo_from_fpga_dout;
Register 29 0x80013074 packet routing
Control packet routing and loopbacks:
wire [3:0] dn_route_ctrl = register_data_in[29][3:0]; wire [3:0] up_route_ctrl = register_data_in[29][7:4]; wire [3:0] fifo_to_fpga_route_ctrl = register_data_in[29][11:8]; //wire [3:0] spare_route_ctrl = register_data_in[29][15:12]; wire dn_mux_trg_enable = register_data_in[29][16]; wire dn_mux_tsm_enable = register_data_in[29][17]; wire dn_mux_sfp_rx_fifo_enable = register_data_in[29][18]; // 19 // 20..23 wire up_mux_vx_rx_enable = register_data_in[29][24]; // 25 // 26 // 27 wire fifo_from_fpga_hitmap_enable = register_data_in[29][28]; // 29..31
fifo_to_fpga output routing:
- 0 - to down packet mux
- 1 - to up packet mux
- 2 - to fifo_from_fpga mux
- 3 - not used
down packet mux inputs:
- fifo_to_fpga_0
- trg_pkt16 enabled by dn_mux_trg_enable
- tsm_pkt16 enabled by dn_mux_tsm_enable
- sfp_rx_fifo_pkt16 enabled by dn_mux_sfp_rx_fifo_enable
- up_pkt16_2 loopback from up packet mux
down packet mux output routing:
- 0 - to vx_tx_pkt16 to VX TX (vx_link_tx and vx_ser_tx)
- 1 - to fifo_from_fpga mux
- 2 - to up packet mux loopback
- 3 - to qsfp_tx_pkt16 to GDM QSFP TX (cdm_link_tx)
up packet mux inputs:
- vx_rx_pkt16 enabled by up_mux_vx_rx_enable data from 12 VX RX links
- fifo_to_fpga_1
- dn_pkt16_2 loopback from down packet mux
up packet mux output routing:
- 0 - to sfp_tx_pkt16 to CDM SFP TX (VX data to GDM)
- 1 - to fifo_from_fpga mux
- 2 - to down packet mux loop loopback
- 3 - not used
fifo_from_fpga mux inputs:
- fifo_to_fpga_2
- dn_pkt16_1 from down packet mux
- up_pkt16_1 from up packet mux
- qsfp_tx_pkt16 from GDM QSFP link 0 (there is no GDM QSFP 12-to-1 mux)
- hitmap_pkt16 enabled by fifo_from_fpga_hitmap_enable in register 29
Register 30 0x80013078 qsfp link status ports 0..7
assign register_data_out[30] =
{
qsfp_rx_data_error[7], qsfp_link_error[7], qsfp_link_status[7], qsfp_link_rx_status[7], // 7
qsfp_rx_data_error[6], qsfp_link_error[6], qsfp_link_status[6], qsfp_link_rx_status[6], // 6
qsfp_rx_data_error[5], qsfp_link_error[5], qsfp_link_status[5], qsfp_link_rx_status[5], // 5
qsfp_rx_data_error[4], qsfp_link_error[4], qsfp_link_status[4], qsfp_link_rx_status[4], // 4
qsfp_rx_data_error[3], qsfp_link_error[3], qsfp_link_status[3], qsfp_link_rx_status[3], // 3
qsfp_rx_data_error[2], qsfp_link_error[2], qsfp_link_status[2], qsfp_link_rx_status[2], // 2
qsfp_rx_data_error[1], qsfp_link_error[1], qsfp_link_status[1], qsfp_link_rx_status[1], // 1
qsfp_rx_data_error[0], qsfp_link_error[0], qsfp_link_status[0], qsfp_link_rx_status[0] // 0
};
Register 31 0x8001307C qsfp link status ports 8..11
assign register_data_out[31] =
{
4'b0000,
4'b0000,
4'b0000,
4'b0000,
qsfp_rx_data_error[11], qsfp_link_error[11], qsfp_link_status[11], qsfp_link_rx_status[11], // 11
qsfp_rx_data_error[10], qsfp_link_error[10], qsfp_link_status[10], qsfp_link_rx_status[10], // 10
qsfp_rx_data_error[9], qsfp_link_error[9], qsfp_link_status[9], qsfp_link_rx_status[9], // 9
qsfp_rx_data_error[8], qsfp_link_error[8], qsfp_link_status[8], qsfp_link_rx_status[8] // 8
};
Register 32 0x80013080 bitmap of enabled qsfp ports
wire [11:0] qsfp_mask = register_data_in[32][11:0]; wire qsfp_bsy_force = register_data_in[32][12]; // not used = register_data_in[32][15:13]; wire [15:0] gdm_veto_extend = register_data_in[32][31:16];
On the GDM:
qsfp_rx_bsy[11..0] are pulses received from the CDMs
qsfp_rx_bsy_or is the grand-or of qsfp_rx_bsy masked by qsfp_mask
if qsfp_rx_bsy_or is high, gdm_bsy goes up and stays up for vx_bsy_extend*2 clocks
gdm_veto = gdm_bsy (bsy of any one VX causes trigger veto to all of them)
gdm_veto transition 0->1 causes gdm_veto_pulse. as long as gdm_veto is high, gdm_veto_pulse is generated every gdm_veto_extend clocks
gdm_veto_pulse is sent to all CDMs.
For this to work right, cdm_bsy_extend should not be bigger than gdm_veto_extend.
Register 33 0x80013084 bitmap of enabled VX ports
wire [11:0] vx_mask = register_data_in[33][11:0]; wire vx_bsy_force = register_data_in[33][12]; // not used = register_data_in[33][15:13]; wire [15:0] vx_bsy_extend = register_data_in[33][31:16];
On the CDM:
cdm_bsy is a grand-or of all vx_bsy masked by vx_mask (list of active VXes).
vx_bsy_extend controls how often state of cdm_bsy is sent to the GDM. when cdm_bsy goes 0->1, we send a cdm_bsy_pulse and keep resending it every vx_bsy_extend clocks as long as cdm_bsy stays high.
cdm_bsy_pulse is sent to the GDM.
sfp_rx_veto is the received from the GDM
if sfp_rx_veto goes up, cdm_veto goes up and stays up for gdm_veto_extend clocks.
for this to work right, CDM gdm_veto_extend must be bigger than GDM gdm_veto_extend.
cdm_veto goes to VXes on v1_tx_out[1] which is LVDS input 13.
Register 34, 35 0x80013088, 8C current timestamp
current 64-bit timestamp, 125 MHz
Register 36, 37 0x80013090, 94 old timestamp
old 64-bit timestamp, 125 MHz. timestamp saved at run start when it is reset to 0.
Register 38, 39, 40 0x80013098, 9C, A0 VX busy counters
8 bits per VX port, counters overflow to 255, reset at run start.
Register 41, 42, 43 0x800130A4, A8, AC QSFP busy counters
8 bits per QSFP port, counters overflow to 255, reset at run start.
Register 44, 45, 46, 47, 48 0x800130B0, B4, B8, BC, C0 CDM and GDM busy and veto counters
- cdm_busy = grand-or of all VX busy for enabled VXes
- gdm_busy = grand-or of all CDM busy for enabled CDMs
- gdm_veto = gdm_busy
44 | lo 16 bits | cdm_bsy_up_counter | CDM busy, increments when cdm_busy goes 0->1 44 | hi 16 bits | cdm_bsy_pulse_counter | CDM busy to GDM, increments for each cdm_bsy_pulse sent to the GDM 45 | lo 16 bits | cdm_veto_pulse_counter | CDM veto from GDM, increments for each sfp_rx_veto received from the GDM 45 | hi 16 bits | cdm_veto_up_counter | CDM veto to VX, increments each time cdm_veto is set to 1. 46 | lo 16 bits | gdm_bsy_pulse_counter | GDM busy from CDM, increments for each qsfp_rx_busy received from CDMs (unless they overlap) 46 | hi 16 bits | gdm_bsy_up_counter | GDM busy, increments each time gdm_bsy goes 0->1 47 | lo 16 bits | gdm_bsy_refresh_counter | GDM busy, increments each time gdm_bsy is extended by new qsfp_rx_busy 47 | hi 16 bits | spare | 48 | lo 16 bits | gdm_veto_up_counter | GDM veto, increments each time gdm_veto goes 0->1 48 | hi 16 bits | gdm_veto_pulse_counter | GDM veto to CDM, increments for each gdm_veto_pulse sent to the CDM
Register 49, 50, 51 0x800130xx VX RX serial data monitor
VX RX data, 8-bit per VX channel. k-bit is omitted.
assign register_data_out[49][7:0] = vx_rx_data[0]; // vx1 assign register_data_out[49][15:8] = vx_rx_data[1]; // vx2 assign register_data_out[49][23:16] = vx_rx_data[2]; // vx3 assign register_data_out[49][31:24] = vx_rx_data[3]; // vx4 assign register_data_out[50][7:0] = vx_rx_data[4]; // vx5 assign register_data_out[50][15:8] = vx_rx_data[5]; // vx6 assign register_data_out[50][23:16] = vx_rx_data[6]; // vx7 assign register_data_out[50][31:24] = vx_rx_data[7]; // vx8 assign register_data_out[51][7:0] = vx_rx_data[8]; // vx9 assign register_data_out[51][15:8] = vx_rx_data[9]; // vx10 assign register_data_out[51][23:16] = vx_rx_data[10]; // vx11 assign register_data_out[51][31:24] = vx_rx_data[11]; // vx12
Register 52, 53 VX link status
assign register_data_out[52] =
{
vx_rx_error[7], vx_link_error[7], vx_link_status[7], vx_link_rx_status[7], // 7
vx_rx_error[6], vx_link_error[6], vx_link_status[6], vx_link_rx_status[6], // 6
vx_rx_error[5], vx_link_error[5], vx_link_status[5], vx_link_rx_status[5], // 5
vx_rx_error[4], vx_link_error[4], vx_link_status[4], vx_link_rx_status[4], // 4
vx_rx_error[3], vx_link_error[3], vx_link_status[3], vx_link_rx_status[3], // 3
vx_rx_error[2], vx_link_error[2], vx_link_status[2], vx_link_rx_status[2], // 2
vx_rx_error[1], vx_link_error[1], vx_link_status[1], vx_link_rx_status[1], // 1
vx_rx_error[0], vx_link_error[0], vx_link_status[0], vx_link_rx_status[0] // 0
};
assign register_data_out[53] =
{
//4'b0000,
1'b0, ~vx_rx_deser_rdy[0], vx_rx_code_err[0], vx_rx_disp_err[0],
//4'b0000,
//4'b0000,
//4'b0000,
vx_rx_monitor[0],
vx_rx_error[11], vx_link_error[11], vx_link_status[11], vx_link_rx_status[11], // 11
vx_rx_error[10], vx_link_error[10], vx_link_status[10], vx_link_rx_status[10], // 10
vx_rx_error[9], vx_link_error[9], vx_link_status[9], vx_link_rx_status[9], // 9
vx_rx_error[8], vx_link_error[8], vx_link_status[8], vx_link_rx_status[8] // 8
};
vx_rx_monitor (12-bit) is from deserializer_10b.sv:
assign monitor_out[9:0] = lastByte[9:0]; assign monitor_out[10] = comma; assign monitor_out[11] = ready;
Register 54 VX TX serial data monitor
- contents of vx_tx_monitor from vx_ser_tx.sv:
assign monitor_out[8:0] = data_to_encoder; // 8-bit + k assign monitor_out[9] = valid; assign monitor_out[15:10] = 0; assign monitor_out[25:16] = encoded_data; // 10-bit assign monitor_out[26] = encoded_valid; assign monitor_out[27] = 0; assign monitor_out[31:28] = 0;
Register 55 QSFP, SFP, VX link loss counters
assign register_data_out[55] = {
vx_rx_error_counter,
qsfp_link_rx_status_drop_counter,
sfp_link_rx_status_drop_counter,
vx_link_rx_status_drop_counter
};
24..31 - CDM VX RX error counters, count any errors in the VX RX data path (bad serial data, fifo overflow, etc) 16..23 - GDM QSFP link loss counter, increments on qsfp_tx_link_rx_status 1->0 (reg10) 8..15 - CDM SFP link loss counter, increments on sfp_link_rx_status 1->0 (reg15) 0...7 - CDM VX link loss counter, increments on vx_tx_link_rx_status 1->0 (reg10)
Register 68 GPS control and status
assign register_data_out[68] =
{
8'b00000000, // 23+8
rb_1pps_counter, // 16+8 bits
gps_1pps_counter, // 8+8 bits
1'b0,
1'b0,
rb_ser_in, // 5
gps_data_in, // 4
gps_aux_out, // 3
gps_aux_in, // 2
rb_1pps_in, // 1
gps_1pps_in // 0
};
Firmware registers branch develop_ko
Register map
0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2 1 | ALL | ALL | RW | read write scratch register 2 | ALL | CDM | ?? | MGT not used 3 | ALL | CDM | RO | MGT debug_data 4 | ALL | CDM | RW | clk_config_vec 5 | ALL | CDM | ?? | not used 6 | ALL | CDM | RO | CDM_link_data_processing:o_error_time 7 | ALL | CDM | RO | CDM_link_data_processing:o_error_count
register 0 0x80010000
GDM:
0 - gdm_link_interface:i_mgt_rst 2 - gdm_link_interface:i_link_down_latched_rst 8 - GDM_link_data_processing:i_rst 10..9 - GDM_link_data_processing:i_data_mode
CDM:
0 - cdm_link_interface:i_mgt_rst 2 - cdm_link_interface:i_link_down_latched_rst 8 - CDM_link_data_processing:i_rst 10..9 - CDM_link_data_processing:i_data_mode
register 1 0x80010004
GDM:
nlinks-1..0 - gdm_link_interface:i_rx_slide_trigger
CDM:
nlinks-1..0 - cdm_link_interface:i_rx_slide_trigger
register 2 0x80010008
GDM:
nlinks-1..0 - gdm_link_interface:o_link_power_good nlinks+15..16 - gdm_link_interface:o_link_status
CDM:
nlinks-1..0 - cdm_link_interface:o_link_power_good nlinks+15..16 - cdm_link_interface:o_link_status
register 3 0x8001000c
GDM: simple loopback register
CDM:
31..0 - debug_data - cdm_link_interface:o_debug
o_debug:
rx_link_rst & rx_error & rx_link_up & rx_receiving_data & std_logic_vector(rx_state_count) & tx_state_count_on_rx_clk & i_rx_ctrl3(0) & i_rx_ctrl1(1 downto 0) & i_rx_ctrl0(1 downto 0) & rx_data_is_k28p1_k28p5 & i_rx_data;
register 4 0x80010010
GDM write:
0 - clk_config_vec(0) - CLK_IN_SEL_LS(0) 1 - clk_config_vec(1) - CLK_IN_SEL_LS(1) 2 - clk_config_vec(2) - CLK_EXT_SEL_LS 3 - clk_config_vec(3) - CLK_RSTn_LS
GDM read:
0 - clk_config_vec(0) - CLK_IN_SEL_LS(0) 1 - clk_config_vec(1) - CLK_IN_SEL_LS(1) 2 - clk_config_vec(2) - CLK_EXT_SEL_LS 3 - clk_config_vec(3) - CLK_RSTn_LS 4 - clk_config_vec(4) - CLK_LOSXTn_LS 5 - clk_config_vec(5) - CLK_LOLn_LS 6 - clk_config_vec(6) - CLK_INTn_LS 7 - constant 1 31..8 - constant 0
register 5 0x80010014
not used
register 6 0x80010018
GDM:
3..0 - GDM_link_data_processing:i_status_select
CDM:
31..0 - CDM_link_data_processing:o_error_count
register 7 0x8001001c
GDM:
31..0 - GDM_link_data_processing:o_status_vector
CDM:
31..0 - CDM_link_data_processing:o_error_count
GDM, CDM, VX packet communications
- timestamp math
1 clock is 8 ns is 125 MHz 8 bits of clocks is 256 clocks is 2048 ns is ~2 usec 16 bits of clocks is ~500 usec is 0.5 msec 24 bits of clocks is ~134 msec 32 bits of clocks is ~34 sec 40 bits of clocks is ~8.7 ksec is 2.4 hours 48 bits of clocks is ~625 hours is ~26 days 56 bits of clocks is ~6.6 kdays is ~18 kyears 62 bits of clocks is ~10 Mhours is 427 kdays is ~1.1 kyears 64 bits of clocks is ~4.4 kyears
- 0x02 - TRG packet, 8 bytes, 80 adc clocks, 640 ns on lvds link
0 - 0x02 1 - trg_counter[7:0] 2 - ts64 low byte 0 3 - ts64 byte 1 4 - ts64 byte 2 5 - ts64 high byte 3 6 - trg_in_latch[7:0] 7 - trg_in_latch[15:8]
- 0x03 - HITMAP_TRG packet, 12 bytes, 120 adc clocks, 960 ns on lvds link
0 - 0x03 1 - trg_counter[7:0] 2 - ts64 low byte 0 3 - ts64 byte 1 4 - ts64 byte 2 5 - ts64 high byte 3 6 - vx_bitmap[7:0] 7 - vx_bitmap[15:8] 8 - vx_bitmap[23:16] 9 - vx_bitmap[31:24] 10 - vx_bitmap[39:32] 11 - vx_bitmap[47:40]
- 0x10 - TSM packet, 26 bytes, 260 adc clocks, 2080 ns on lvds link
0 - 0x10 1 - tsm_counter[7:0] 2 - gdm_ts64 low byte 0 3 - 1 4 - 2 5 - 3 6 - 4 7 - 5 8 - 6 9 - gdm_ts64 high byte 7 10 - gps_ts64 low byte 0 11 - 1 12 - 2 13 - 3 14 - 4 15 - 5 16 - 6 17 - gps_ts64 high byte 7 18 - gps_data64 low byte 0 19 - 1 20 - 2 21 - 3 22 - 4 23 - 5 24 - 6 25 - gps_data64 high byte 7
- 0x81 - VX hitmap packet, 10 bytes, 100 adc clocks, 800 ns on lvds link, 48 ns on fiber link
0 - 0x81 1 - VX ID 2 - hitmap low byte, nits 7:0 3 - 15:8 4 - 23:16 5 - 31:24 6 - ...:32 7 - ... 8 - ... 9 - hitmap low byte, bits 63:...
- 0x82 - CDM hitmap packet, 108 bytes, not sent on lvds link, not sent on fiber link
0 - 0x82 1 - cdm_hitmap_trigger_counter[7:0] 2 - ts64 byte 0 3 - ts64 byte 1 4 - ts64 byte 2 5 - ts64 byte 3 6 - ts64 byte 4 7 - ts64 byte 5 8 - ts64 byte 6 9 - ts64 byte 7 10 - cdm_hitmap_or12 byte 0 (7:0) 11 - cdm_hitmap_or12 byte 1 (11:8) plus 4 bits: 12=0, 13=0, 14=0, 15=cdm_hitmap_grand_or 12 - cdm_hitmap_data, low byte, 12*64 bits = 768 bits = 48 words = 96 bytes ... 107 - cdm_hitmap_data, high byte
AXI bus timing
- AXI 100 MHz clock, 10 ns, 32-bit data
- AXI single-dword read: 36 clock repeat rate, 360 ns is 2.777 MHz, 4 bytes per transfer is 11.11 Mbytes/sec
- AXI single-qword read: 13 clock repeat rate, 130 ns is 7.7 MHz, 4 bytes per transfer is 30 Mbytes/sec, not accounting for the gap
- AXI memcpy read: 4 transfers at 13 clocks, gap, 4 transfers at 13 clocks, gap, etc. below 30 Mbytes/sec.
- AXI single-dword write: 20 clocks repeat rate, 200 ns is 5 MHz, 4 bytes per transfer is 20 Mbytes/sec
- AXI single-qword write: 13+20 clocks repeat rate, 330 ns is 3 MHz, 16 bytes per burst is 48 Mbytes/sec
AXI bus addresses
- see AXI/AMBA addresses assigned inthe FPGA project: (s_axi/reg0 is the DS-DM AXI registers)
daq00:ds-dm-gcdm$ grep assign_bd_address scripts/GDM_CDM_XU8_bd.tcl assign_bd_address -offset 0x80010000 -range 0x00004000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs axi_register_interfa_0/s_axi/reg0] -force assign_bd_address -offset 0x000400000000 -range 0x40000000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs ddr4/C0_DDR4_MEMORY_MAP/C0_DDR4_ADDRESS_BLOCK] -force assign_bd_address -offset 0x80000000 -range 0x00010000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs system_management_wiz/S_AXI_LITE/Reg] -force daq00:ds-dm-gcdm$
- see AXI/AMBA addresses exported from FPGA project to Linux kernel: (uio for debug bridge should say "debug bridge")
root@gdm0:~# cat /sys/class/uio/uio*/name axi-pmon axi-pmon axi-pmon axi-pmon root@gdm0:~# root@gdm0:~# ls -l /sys/class/uio/ total 0 lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio0 -> ../../devices/platform/amba/ffa00000.perf-monitor/uio/uio0 lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio1 -> ../../devices/platform/amba/fd0b0000.perf-monitor/uio/uio1 lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio2 -> ../../devices/platform/amba/fd490000.perf-monitor/uio/uio2 lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio3 -> ../../devices/platform/amba/ffa10000.perf-monitor/uio/uio3 root@gdm0:~# ls -l /sys/class/uio/../../devices/platform/amba/ total 0 -rw-r--r-- 1 root root 4096 Oct 18 19:37 driver_override drwxr-xr-x 3 root root 0 Oct 18 01:36 fd070000.memory-controller drwxr-xr-x 4 root root 0 Oct 18 01:36 fd0b0000.perf-monitor drwxr-xr-x 4 root root 0 Oct 18 01:36 fd400000.zynqmp_phy drwxr-xr-x 4 root root 0 Oct 18 01:36 fd490000.perf-monitor drwxr-xr-x 4 root root 0 Oct 18 01:36 fd500000.dma drwxr-xr-x 4 root root 0 Oct 18 01:36 fd510000.dma drwxr-xr-x 4 root root 0 Oct 18 01:36 fd520000.dma drwxr-xr-x 4 root root 0 Oct 18 01:36 fd530000.dma drwxr-xr-x 4 root root 0 Oct 18 01:36 fd540000.dma drwxr-xr-x 4 root root 0 Oct 18 01:36 fd550000.dma drwxr-xr-x 4 root root 0 Oct 18 01:36 fd560000.dma drwxr-xr-x 4 root root 0 Oct 18 01:36 fd570000.dma drwxr-xr-x 4 root root 0 Oct 18 01:36 fd6e0000.cci drwxr-xr-x 5 root root 0 Oct 18 01:36 ff000000.serial drwxr-xr-x 4 root root 0 Oct 18 01:36 ff020000.i2c drwxr-xr-x 6 root root 0 Oct 18 01:36 ff0a0000.gpio drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0b0000.ethernet drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0e0000.ethernet drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0f0000.spi drwxr-xr-x 5 root root 0 Oct 18 01:36 ff160000.mmc drwxr-xr-x 5 root root 0 Oct 18 01:36 ff170000.mmc drwxr-xr-x 3 root root 0 Oct 18 01:36 ff960000.memory-controller drwxr-xr-x 4 root root 0 Oct 18 01:36 ff9d0000.usb0 drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa00000.perf-monitor drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa10000.perf-monitor drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa50000.ams drwxr-xr-x 5 root root 0 Oct 18 01:36 ffa60000.rtc drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa80000.dma drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa90000.dma drwxr-xr-x 4 root root 0 Oct 18 01:36 ffaa0000.dma drwxr-xr-x 4 root root 0 Oct 18 01:36 ffab0000.dma drwxr-xr-x 4 root root 0 Oct 18 01:36 ffac0000.dma drwxr-xr-x 4 root root 0 Oct 18 01:36 ffad0000.dma drwxr-xr-x 4 root root 0 Oct 18 01:36 ffae0000.dma drwxr-xr-x 4 root root 0 Oct 18 01:36 ffaf0000.dma -r--r--r-- 1 root root 4096 Oct 18 19:37 modalias lrwxrwxrwx 1 root root 0 Oct 18 19:37 of_node -> ../../../firmware/devicetree/base/amba drwxr-xr-x 2 root root 0 Oct 18 19:37 power lrwxrwxrwx 1 root root 0 Oct 18 01:36 subsystem -> ../../../bus/platform -rw-r--r-- 1 root root 4096 Oct 18 01:36 uevent root@gdm0:~#
Build firmware
Build from git clone
THESE ARE K.O.'s NOTES FOR CREATING THE PETALINUX DIRECTORY.
THEY DO NOT WORK!
COPY PETALINUX FROM A WORKING PROJECT AND USE "make gdm" and "make cdm" AS DESCRIBED BELOW.
- git clone git@edev-group.triumf.ca:fw/exp/darkside/gcdm.git
- #Makefile change VIVADO_SETTINGS_SCRIPT := /opt/Xilinx/Vivado/2022.1/settings64.sh
- #. /opt/Xilinx/Vivado/2022.1/settings64.sh
- . /opt/Xilinx/Vivado/2020.2/settings64.sh
- make clean
- make all_from_scratch
- . /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
- make petalinux_create
- make petalinux_rebuild_new_hw_des
- bomb out: The TMPDIR: /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp can't be located on nfs.
- mkdir /tmp/build_tmp
- rm -rf /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp/
- ln -s /tmp/build_tmp /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp
- try again
- grinds, loads a whole bunch of packages...
- finishes with desire to copy things to /tftpboot
- make sdcard_cp_to wants to copy files from PetaLinux_GDM_CDM/images/linux/ to SD card
Build firmware
NOTE: directory Petalinux_GDM_CDM should already exist!
#. /opt/Xilinx/Vivado/2020.2/settings64.sh . /opt/Xilinx/Vivado/2022.2/settings64.sh . /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh make clean_gdm # remove gdm build tree make gdm # build or rebuild GDM make copy_gdm # copy to gdm0 make clean_cdm # remove cdm build tree make cdm # build or rebuild CDM make copy_cdm # copy to cdm0 and cdm1
copy to SD card:
open a root shell format 16 GB Sd card per above cd .../ds-dm-gcdm make copy
build times
CDM 12-june-2023 69aabc1c25130d970bc375aca684bd68849e6685 daq13 AMD-5700G 1688.61user 399.33system 23:28.84elapsed 148%CPU dsdaqgw AMD-7700 1090.55user 247.34system 16:03.55elapsed 138%CPU dsdaqgw AMD-7700 CDM incremental 196.68user 67.62system 7:35.42elapsed 58%CPU dsdaqgw AMD-7700 CDM incremental 684.72user 94.17system 7:30.17elapsed 173%CPU dsdaqgw AMD-7700 GDM incremental 849.84user 99.79system 9:04.56elapsed 174%CPU
prepare bootable sd card
format the sd card
this only needs to be done once
- become root
- cd ~olchansk/git/ds-dm-gcdm
- use "lsblk" to identify the SD card (should show as 8/16/32 GB block device)/ /dev/sdd in this case
- make sdcard_format SDCARD_DEVICE=/dev/sdd
- disconnect sd card, reconnect the sd card (to detect new partition tables, etc)
copy CDM boot files
cd /home/dsdmdev/git/ds-dm-gcdm make copy
copy boot files to the sd card
- as root: identify partition labels, run "blkid", should say "BOOT", "rootfs" and "data"
- mount
mkdir /media/olchansk/BOOT mkdir /media/olchansk/rootfs mkdir /media/olchansk/data mount -L BOOT /media/olchansk/BOOT mount -L rootfs /media/olchansk/rootfs mount -L data /media/olchansk/data cp PetaLinux_GDM_CDM/images/linux/BOOT.BIN /media/olchansk/BOOT/ cp PetaLinux_GDM_CDM/images/linux/boot.scr /media/olchansk/BOOT/ cp PetaLinux_GDM_CDM/images/linux/image.ub /media/olchansk/BOOT/ umount /media/olchansk/BOOT umount /media/olchansk/rootfs umount /media/olchansk/data eject /dev/sdd
boot messages
Xilinx Zynq MP First Stage Boot Loader Release 2020.2 Sep 24 2022 - 13:29:15 NOTICE: ATF running on XCZU4CG/silicon v4/RTL5.1 at 0xfffea000 NOTICE: BL31: v2.2(release):xlnx_rebase_v2.2_2020.3 NOTICE: BL31: Built : 18:02:46, Sep 28 2022 U-Boot 2020.01 (Sep 28 2022 - 18:03:39 +0000) Model: DarkSide 20k DM Board: Xilinx ZynqMP DRAM: 2 GiB usb dr_mode not found PMUFW: v1.1 EL Level: EL2 Chip ID: zu4 NAND: 0 MiB MMC: mmc@ff160000: 0, mmc@ff170000: 1 In: serial@ff000000 Out: serial@ff000000 Err: serial@ff000000 Bootmode: SD_MODE1 Reset reason: SOFT Net: ZYNQ GEM: ff0b0000, mdio bus ff0b0000, phyaddr -1, interface rgmii-id Warning: ethernet@ff0b0000 (eth0) using random MAC address - d6:62:5f:13:00:44 eth0: ethernet@ff0b0000 ZYNQ GEM: ff0e0000, mdio bus ff0e0000, phyaddr -1, interface rgmii-id Could not get PHY for eth1: addr -1 Hit any key to stop autoboot: 0 ZynqMP> CTRL-A Z for help | 115200 8N1 | NOR | Minicom 2.7.1 | VT102 | Online 122:2 | ttyACM0
load FPGA from u-boot
ZynqMP> fpga info Xilinx Device Descriptor @ 0x000000007fddb2c0 Family: ZynqMP PL Interface type: csu_dma configuration interface (ZynqMP) Device Size: 1 bytes Cookie: 0x0 (0) Device name: zu4 Device Function Table @ 0x000000007fda5fe8 PCAP status 0xa0002fde ZynqMP>
- cp CDM_XU8_top.bit /tftpboot/fpga.bit
dhcp
tftpb 0x10000000 fpga.bit
fpga loadb 0 0x10000000 ${filesize}
ZynqMP> dhcp
BOOTP broadcast 1
DHCP client bound to address 192.168.0.100 (1 ms)
*** Warning: no boot file name; using 'C0A80064.img'
Using ethernet@ff0b0000 device
TFTP from server 192.168.0.1; our IP address is 192.168.0.100
Filename 'C0A80064.img'.
Load address: 0x8000000
Loading: *
TFTP error: 'file /tftpboot/C0A80064.img not found for 192.168.0.100' (1)
Not retrying...
ZynqMP> tftpb 0x10000000 fpga.bit
Using ethernet@ff0b0000 device
TFTP from server 192.168.0.1; our IP address is 192.168.0.100
Filename 'fpga.bit'.
Load address: 0x10000000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
############
6.2 MiB/s
done
Bytes transferred = 7797807 (76fc2f hex)
ZynqMP> fpga loadb 0 0x10000000 ${filesize}
design filename = "CDM_XU8_top;UserID=0XFFFFFFFF;Version=2022.2"
part number = "xczu4cg-fbvb900-1-e"
date = "2024/08/14"
time = "14:18:22"
bytes in bitstream = 7797692
zynqmp_align_dma_buffer: Align buffer at 0000000010000073 to 000000000fffff80(swap 0)
ZynqMP>
load FPGA from Linux
this will reset the CPU
cp fpga.bit /lib/firmware/ echo fpga.bit > /sys/class/fpga_manager/fpga0/firmware
- make .bin file:
bootgen -image CDM_XU8_top.bif -arch zynqmp -o ./fpga.bin -w
- cat CDM_XU8_top.bif
dsdaqgw:ds-dm-gcdm$ cat CDM_XU8_top.bif
all:
{
[destination_device = pl] ./Vivado_CDM_XU8/CDM_XU8.runs/impl_1/CDM_XU8_top.bit
}
dsdaqgw:ds-dm-gcdm$
this will reset the CPU
cp fpga.bin /lib/firmware/ echo fpga.bin > /sys/class/fpga_manager/fpga0/firmware
this will reset the CPU
root@dsdm:~# ./fpgautil -b fpga.bin -f Full
this will reset the CPU
DTSO file from here: https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841847/Solution+ZynqMP+PL+Programming echo 0 > /sys/class/fpga_manager/fpga0/flags mount -t configfs configfs /configfs root@dsdm:~# dtc -O dtb -o fpga.dtbo -b 0 -@ fpga.dtso root@dsdm:~# cp fpga.dtbo /lib/firmware/ root@dsdm:~# cp fpga.bit /lib/firmware/ root@dsdm:~# rmdir /configfs/device-tree/overlays/fpga root@dsdm:~# mkdir /configfs/device-tree/overlays/fpga root@dsdm:~# echo -n "fpga.dtbo" > /configfs/device-tree/overlays/fpga/path
fpgautil
git clone https://github.com/Xilinx/meta-xilinx.git cd meta-xilinx/meta-xilinx-core/recipes-bsp/fpga-manager-script/files/ scp fpgautil.c root@dsdm: ssh root@dsdm make fpgautil ls -l ./fpgautil
root@dsdm:~# ls -l ./fpgautil
-rwxr-xr-x 1 root root 72256 Aug 16 00:15 ./fpgautil
root@dsdm:~# ./fpgautil
fpgautil: FPGA Utility for Loading/reading PL Configuration
Usage: fpgautil -b <bin file path> -o <dtbo file path>
Options: -b <binfile> (Bin file path)
-o <dtbofile> (DTBO file path)
-f <flags> Optional: <Bitstream type flags>
f := <Full | Partial >
-n <Fpga region info> FPGA Regions represent FPGA's
and partial reconfiguration
regions of FPGA's in the
Device Tree
Default: <full>
-s <secure flags> Optional: <Secure flags>
s := <AuthDDR | AuthOCM | EnUsrKey | EnDevKey | AuthEnUsrKeyDDR | AuthEnUsrKeyOCM | AuthEnDevKeyDDR | AuthEnDevKeyOCM>
-k <AesKey> Optional: <AES User Key>
-r <Readback> Optional: <file name>
Default: By default Read back contents will be stored in readback.bin file
-t Optional: <Readback Type>
0 - Configuration Register readback
1 - Configuration Data Frames readback
Default: 0 (Configuration register readback)
-R Optional: Remove overlay from a live tree
Examples:
(Load Full bitstream using Overlay)
fpgautil -b top.bit.bin -o can.dtbo -f Full -n full
(Load Partial bitstream using Overlay)
fpgautil -b rm0.bit.bin -o rm0.dtbo -f Partial -n PR0
(Load Full bitstream using sysfs interface)
fpgautil -b top.bit.bin -f Full
(Load Partial bitstream using sysfs interface)
fpgautil -b rm0.bit.bin -f Partial
(Load Authenticated bitstream through the sysfs interface)
fpgautil -b top.bit.bin -f Full -s AuthDDR
(Load Parital Encrypted Userkey bitstream using Overlay)
fpgautil -b top.bit.bin -o pl.dtbo -f Partial -s EnUsrKey -k <32byte key value>
(Read PL Configuration Registers)
fpgautil -b top.bit.bin -r
(Remove Partial Overlay)
fpgautil -R -n PR0
(Remove Full Overlay)
fpgautil -R -n full
Note: fpgautil -R is responsible for only removing the dtbo file from the livetree. it will not remove the PL logic from the FPGA region.
root@dsdm:~#
fw_printenv
to access u-boot environment from Linux:
- apt install -y libubootenv-tool
- create /etc/fw_env.config
/media/BOOT/uboot.env 0 0x40000 /media/BOOT/uboot-redund.env 0 0x40000
- if uboot.env files do not exist, run "saveenv" from u-boot command prompt
- fw_printenv and fw_setenv should work
Boot from network
u-boot
ZynqMP> setenv bootcmd run bootcmd_dhcp ZynqMP> saveenv ZynqMP> reset
boot.scr
# boot.scr
# mkimage -C none -A arm -T script -d boot.scr boot.scr.uimg
echo Loading FPGA!
#tftpb 0x10000000 fpga.bit
tftpb 0x10000000 {ipaddr}.bit
fpga loadb 0 0x10000000 ${filesize}
echo Booting Linux!
run bootcmd_pxe
echo Done!
mkimage -C none -A arm -T script -d boot.scr boot.scr.uimg
tftpboot
cp /home/olchansk/git/ds-dm-gcdm/boot.scr.uimg /tftpboot ln -s /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/images/linux /tftpboot/xilinx-dsdm mkdir /tftpboot/pxelinux.cfg cat > /tftpboot/pxelinux.cfg/default-arm-zynqmp <<EOF LABEL Linux KERNEL xilinx-dsdm/Image FDT xilinx-dsdm/system.dtb #INITRD rootfs.cpio.gz.u-boot EOF
boot sequence
- xilinx magic load BOOT.BIN from SD card
- load FPGA form BOOT.BIN
- load and run u-boot from BOOT.BIN or from image.ub
- u-boot load environment from ??? probably SD card uboot-redund.env, this includes our bootcmd
- run bootcmd which run bootcmd_dhcp which does:
- from /tftpboot:
- load and run boot.scr.uimg which does:
- load FPGA image xilinx-dsdm/${ipaddr}.bit
- run bootcmd_pxe which does:
- load pxelinux.cfg/default-arm-zynqmp which does:
- load xilinx-dsdm/Image ### this is the linux kernel
- load xilinx-dsdm/system.dtb ### this is the device tree
- start linux kernel
- linux kernel does dhcp
- linux kernel does nfs mount /nfsroot/%s,vers=3,tcp ### %s is replaced by the hostname supplied by DHCP
- userland starts and runs to console and ssh login.
Xilinx ILA
References:
- https://github.com/Xilinx/XilinxVirtualCable/tree/master
- https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/644579329/Xilinx+Virtual+Cable
- https://support.xilinx.com/s/article/974879?language=en_US
- https://docs.xilinx.com/v/u/en-US/pg245-debug-bridge
- https://docs.xilinx.com/r/2020.2-English/ug908-vivado-programming-debugging/Initializing-Vivado-IDE-hw_server
- (we do not use this one) https://github.com/paulscherrerinstitute/xvcSupport/blob/master/README.md
Build xvcserver_cdm.exe: (it is built as a static executable, can be copied and run anywhere)
ssh dsdaq@gdm0 cd /home/dsdaq/online/ds-dm-software git pull ### get latest version make xvcserver_cdm.exe ssh root@gdm0 /home/dsdaq/online/ds-dm-software/xvcserver_cdm.exe INFO: To connect to this xvcServer instance, use url: TCP:gdm0:2542
To activate and use the vivado logic analyzer:
- data path: vivado -> hw_server -> xvcserver -> mmap axi bus -> debug bridge -> jtag -> ILA
- define ILAs in the code
- instantiate the xilinx debug bridge at AXI bus address 0x80020000 (FIXME!!! this collides with Ian's AXI addresses)
- build and boot the new FPGA firmware. updating the linux kernel is not necessary.
- login root@gdm0, run: /home/dsdaq/online/ds-dm-software/xvcserver_cdm.exe -v ### with "-v" for the first time to see that vivado does connect to it, without "-v", normally.
- login dsdaqgw, run: hw_server -s tcp:localhost:3121 -e "set auto-open-servers xilinx-xvc:gdm0:2542" ### tells us to connect to port localhost:3121
- login dsdaqgw, run vivado, open project, open hardware manager, open target, open new target, "connect to remote server", hostname "localhost", port "3121", next (bombs, try again, 3 times), popup add virtual cable, enter hostname "gdm0" port "2542", "ok", it shows in "hardware targets", "next", "finish", error popup "[Common 17-163] Missing value for option 'objects', please type 'set_property -help' for usage info", ignore it, in "hardware", right click the "gdm0" one, open target, under "hardware" and "debug bridge" we should see all the ILAs, under "hardware device properties", the "probes file" should have the ".ltx" file generated by vivado "Vivado_GDM_XU8/GDM_XU8.runs/impl_1/debug_nets.ltx", click on an ILA, a waveform should open.
Software
- ssh cdm0 # or gdm0
- sudo apt install i2c-tools libi2c-dev
- git clone https://bitbucket.org/team-ds-dm/ds-dm-software
- cd ds-dm-software
- make
test_cdm.exe
CDM SFP status
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp DS-DM mapping /dev/mem at 0x80010000 DS-DM FPGA firmware revision 0xbb2f0ae7 CDM firmware 0xbb2f0ae7 arg 1: [--sfp] Polling SFP status... identifier 0x03 connector 0x07 encoding 0x01 wavelength 0x0352 (850 nm) vendor_name [FINISAR CORP. ] vendor_pn [FTLF8526P3BNL ] vendor_rev [A ] vendor_sn [N3AB9M8 ] vendor_date [200319 ] dm_type 0x68 temp 29.0 C vcc 3.323 V tx_bias 7.250 mA tx_power 478.4 uW rx_power 2.3 uW SFP good 1, status: temp 30.6 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 818 uW ...
GDM QSFP status
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --qsfp3 --qsfp DS-DM mapping /dev/mem at 0x80010000 DS-DM FPGA firmware revision 0x53aee418 CDM firmware 0x53aee418 arg 1: [--qsfp3] gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio: gpio-378 ( |sysfs ) out hi gpio-379 ( |sysfs ) out hi gpio-381 ( |sysfs ) out lo gpio-382 ( |sysfs ) out hi arg 2: [--qsfp] Polling QSFP status... identifier 0x0d status 0x02 los 0x8f temp 28.2 C vcc 3.323 V rx_power 0.1 0.1 0.1 0.1 uW tx_bias 7.6 7.6 7.6 0.0 mA tx_power 792.2 773.8 823.0 0.1 uW vendor_name [FINISAR CORP ] vendor_pn [FTL410QD4C ] vendor_rev [A ] wavelength 850 max_temp 70 C vendor_sn [X79AC0R ] vendor_date [220309 ] QSFP good 1, status: temp 27.7 C, los 0x8b, tx_bias 7.5 7.6 7.6 0.0 mA, tx_power 792 772 821 0 uW, rx_power 0 0 466 0 uW
GDM clock status
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cc /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
- clock chip not loaded, not running:
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc DS-DM mapping /dev/mem at 0x80010000 DS-DM FPGA firmware revision 0x6d2ebce6 CDM firmware 0x6d2ebce6 Polling CC status... Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x0f 0x16 0x04 0x00 0x01 0x68 0x09 0x00 0x02 0xf2 0x00 0x1f 0xf0 0x22 0xf2, 0x507: 0x00, 0x52A: 0x00, 0x53F: 0x04 Clock chip state 0, status: SYSINCAL XAXB_ERR LOL CAL_PLL IN0 IN_SEL_0 FASTLOCK_STATUS root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks DS-DM mapping /dev/mem at 0x80010000 DS-DM FPGA firmware revision 0x6d2ebce6 CDM firmware 0x6d2ebce6 GDM clock frequency counters: 0x1030 mgt_rx_ref_clk_raw: 0x1f7caf52 (528265042) should be ~125 MHz 0x1034 rx_clk: 0x00000000 (0) should be ~125 MHz 0x1038 mgt_tx_ref_clk_raw: 0x1f7caf52 (528265042) should be ~125 MHz 0x103C tx_clk: 0x00000000 (0) should be ~125 MHz 0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly 0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
- clock chip good (IN0 - external 10 MHz clock)
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc DS-DM mapping /dev/mem at 0x80010000 DS-DM FPGA firmware revision 0x6d2ebce6 CDM firmware 0x6d2ebce6 Polling CC status... Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x44 0x00 0xd0 0x01 0x1f 0xff 0x22 0xf2, 0x507: 0x3f, 0x52A: 0x01, 0x53F: 0x00 Clock chip state 1, status: LOS_IN2 OOF_IN2 IN0 IN_SEL_REGCTRL IN_SEL_0 root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks DS-DM mapping /dev/mem at 0x80010000 DS-DM FPGA firmware revision 0x6d2ebce6 CDM firmware 0x6d2ebce6 GDM clock frequency counters: 0x1030 mgt_rx_ref_clk_raw: 0x07735a3b (125000251) should be ~125 MHz 0x1034 rx_clk: 0x07735a3c (125000252) should be ~125 MHz 0x1038 mgt_tx_ref_clk_raw: 0x07735a3b (125000251) should be ~125 MHz 0x103C tx_clk: 0x07735a3c (125000252) should be ~125 MHz 0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly 0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
CDM clock status
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cc /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
- clock chip not loaded, not running:
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc DS-DM mapping /dev/mem at 0x80010000 DS-DM FPGA firmware revision 0x6d2ebec9 CDM firmware 0x6d2ebec9 Polling CC status... Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x0f 0x16 0x04 0x00 0x01 0x68 0x19 0x00 0x02 0xf2 0x00 0x1f 0xf0 0x22 0xf2, 0x507: 0x00, 0x52A: 0x00, 0x53F: 0x04 Clock chip state 0, status: SYSINCAL XAXB_ERR LOL CAL_PLL IN0 IN_SEL_0 FASTLOCK_STATUS root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks DS-DM mapping /dev/mem at 0x80010000 DS-DM FPGA firmware revision 0x6d2ebec9 CDM firmware 0x6d2ebec9 CDM clock frequency counters: 0x1030 mgt_rx_ref_clk_raw: 0x07735851 (124999761) should be ~125 MHz 0x1034 rx_clk: 0x00000000 (0) should be ~125 MHz 0x1038 mgt_tx_ref_clk_raw: 0x0127fefa (19398394) should be ~125 MHz 0x103C tx_clk: 0x00000000 (0) should be ~125 MHz 0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly 0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
- clock chip uses internal clock (IN1 - internal oscillator), observe rx_clk frequency is not same as others
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc DS-DM mapping /dev/mem at 0x80010000 DS-DM FPGA firmware revision 0x6d2ebec9 CDM firmware 0x6d2ebec9 Polling CC status... Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0x7f, 0x52A: 0x02, 0x53F: 0x02 Clock chip state 1, status: IN1 IN_SEL_1 HOLD_HIST_VALID root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks DS-DM mapping /dev/mem at 0x80010000 DS-DM FPGA firmware revision 0x6d2ebec9 CDM firmware 0x6d2ebec9 CDM clock frequency counters: 0x1030 mgt_rx_ref_clk_raw: 0x07735853 (124999763) should be ~125 MHz 0x1034 rx_clk: 0x07735b0a (125000458) should be ~125 MHz 0x1038 mgt_tx_ref_clk_raw: 0x07735853 (124999763) should be ~125 MHz 0x103C tx_clk: 0x07735852 (124999762) should be ~125 MHz 0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly 0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
- clock chip uses SFP recovered clock (IN2 - sfp rx recovered clock), observe mgt_rx_ref_clk_raw (CDM 125 MHz oscillator) is different from others (SFP RX recovered clock)
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc DS-DM mapping /dev/mem at 0x80010000 DS-DM FPGA firmware revision 0x6d2ef81a CDM firmware 0x6d2ef81a Polling CC status... Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0xbf, 0x52A: 0x02, 0x53F: 0x02 Clock chip state 1, status: IN2 IN_SEL_1 HOLD_HIST_VALID ^C root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks DS-DM mapping /dev/mem at 0x80010000 DS-DM FPGA firmware revision 0x6d2ef81a CDM firmware 0x6d2ef81a CDM clock frequency counters: 0x1030 mgt_rx_ref_clk_raw: 0x0773581b (124999707) should be ~125 MHz 0x1034 rx_clk: 0x07735ad6 (125000406) should be ~125 MHz 0x1038 mgt_tx_ref_clk_raw: 0x07735ad7 (125000407) should be ~125 MHz 0x103C tx_clk: 0x07735ad6 (125000406) should be ~125 MHz 0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly 0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly ^C
CDM link status, PRBS test mode
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link # /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
- fiber disconnected, no link:
CDM firmware: 0xbb2f0ae7
0x1000 SFP c.c. status: 0x00000031
CLK_IN_SEL_LS 0x1
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x00000025
sfp_mod_absent_N 1
sfp_rx_los_N 0
link_power_good 1
rx_link_up 0
rx_receiving_data 0
rx_error 1
rx_lnk_up_and_running 0
tx_link_up 0
tx_sending_data 0
tx_link_up_and_running 0
link_up_and_running 0
0x1014 SFP link data: 0x466a8187
rx_data 0x8187
k28p1_k28p5 0
rx_ctrl0 0x1
rx_ctrl1 0x1
rx_ctrl3 0x1
tx_state 0x1
rx_state 0x6
rx_receiving_data 0
rx_link_up 0
rx_error 1
rx_link_rst 0
0x2000 link test mode: 0x00000200, seconds: 0x00079093, errors: 0xffffffff
- fiber connected, good link:
CDM firmware: 0xbb2f0ae7
0x1000 SFP c.c. status: 0x000000b2
CLK_IN_SEL_LS 0x2
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x000007dc
sfp_mod_absent_N 0
sfp_rx_los_N 0
link_power_good 1
rx_link_up 1
rx_receiving_data 1
rx_error 0
rx_lnk_up_and_running 1
tx_link_up 1
tx_sending_data 1
tx_link_up_and_running 1
link_up_and_running 1
0x1014 SFP link data: 0x35c02774
rx_data 0x2774
k28p1_k28p5 0
rx_ctrl0 0x0
rx_ctrl1 0x0
rx_ctrl3 0x0
tx_state 0x3
rx_state 0x5
rx_receiving_data 1
rx_link_up 1
rx_error 0
rx_link_rst 0
0x2000 link test mode: 0x00000200, seconds: 0x00078a8c, errors: 0x00000000
GDM link status, PRBS test mode
1 link connected, no errors: # /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link # /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2 GDM firmware: 0x6b2ee010 0x1014: 0x00000008, 18: 0x00000008, 1C: 0x00000000, 24: 0x00000fff 0x2000: 0x00000200, time: 0x00078aa4, errors: 0xffffffff 0xffffffff 0xffffffff 0x00000000 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff
CDM link status
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test0
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0xbf, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN2 IN_SEL_1 HOLD_HIST_VALID
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x077357a0 (124999584) should be ~125 MHz
0x1034 rx_clk: 0x07735a5c (125000284) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a5b (125000283) should be ~125 MHz
0x103C tx_clk: 0x07735a5b (125000283) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM firmware: 0x6d2ef81a
0x1000 SFP c.c. status: 0x000000b2
CLK_IN_SEL_LS 0x2
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x000007dc
sfp_mod_absent_N 0
sfp_rx_los_N 0
link_power_good 1
rx_link_up 1
rx_receiving_data 1
rx_error 0
rx_lnk_up_and_running 1
tx_link_up 1
tx_sending_data 1
tx_link_up_and_running 1
link_up_and_running 1
0x1014 SFP link data: 0x35c6bcbc
rx_data 0xbcbc
k28p1_k28p5 0
rx_ctrl0 0x3
rx_ctrl1 0x0
rx_ctrl3 0x0
tx_state 0x3
rx_state 0x5
rx_receiving_data 1
rx_link_up 1
rx_error 0
rx_link_rst 0
0x2000 link test mode: 0x00000000, seconds: 0x00001671, errors: 0xffffffff
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 15
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
reg[15] is 0x0033bcbc (3390652)
GDM link status
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test0 root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc DS-DM mapping /dev/mem at 0x80010000 DS-DM FPGA firmware revision 0x6d2ebce6 CDM firmware 0x6d2ebce6 Polling CC status... Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x44 0x00 0xd0 0x01 0x1f 0xff 0x22 0xf2, 0x507: 0x3f, 0x52A: 0x01, 0x53F: 0x02 Clock chip state 1, status: LOS_IN2 OOF_IN2 IN0 IN_SEL_REGCTRL IN_SEL_0 HOLD_HIST_VALID root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks DS-DM mapping /dev/mem at 0x80010000 DS-DM FPGA firmware revision 0x6d2ebce6 CDM firmware 0x6d2ebce6 GDM clock frequency counters: 0x1030 mgt_rx_ref_clk_raw: 0x07735a0a (125000202) should be ~125 MHz 0x1034 rx_clk: 0x07735a0a (125000202) should be ~125 MHz 0x1038 mgt_tx_ref_clk_raw: 0x07735a0a (125000202) should be ~125 MHz 0x103C tx_clk: 0x07735a0a (125000202) should be ~125 MHz 0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly 0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly register 0x1018 bit 0x800 root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link DS-DM mapping /dev/mem at 0x80010000 DS-DM FPGA firmware revision 0x6d2ebce6 CDM firmware 0x6d2ebce6 GDM firmware: 0x6d2ebce6 0x1014: 0x00000800, 18: 0x00000800, 1C: 0x00000000, 24: 0x00000fff 0x2000: 0x00000000, time: 0x00003d2f, errors: 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff 0xffffffff link data alternates 0xbcbc and 0x1cbc root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 22 DS-DM mapping /dev/mem at 0x80010000 DS-DM FPGA firmware revision 0x6d2ebce6 CDM firmware 0x6d2ebce6 reg[22] is 0xbcbc93ab (-1128492117) root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 22 DS-DM mapping /dev/mem at 0x80010000 DS-DM FPGA firmware revision 0x6d2ebce6 CDM firmware 0x6d2ebce6 reg[22] is 0x1cbc1aaf (482089647) root@gdm0:~#
Run trg and tsm
on the GDM: ssh root@gdm00 ./test_cdm.exe --gdm-clocks ./test_cdm.exe --load-cc ./test_cdm.exe --cc ./test_cdm.exe --reset-mgt ./test_cdm.exe --gdm-clocks ./test_cdm.exe --writereg 2 0xff # enable LEMO NIM inputs ./test_cdm.exe --writereg 3 0xba54 # LEDs: lemo1, lemo2, trg, tsm ./test_cdm.exe --writereg 4 0x99 $ # LEMO out is trg_in_pulse ./test_cdm.exe --writereg 9 0x32010 # trg and tsm from trg_pulser and tsm_pulse ./test_cdm.exe --writereg 24 1250000 # trg pulser 100 Hz ./test_cdm.exe --writereg 26 125000000 # tsm pulser 1 Hz ./test_cdm.exe --writereg 23 0x40000000 # route trg_in and tsm_in to qsfp tx bits 0 and 1 ./test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz on the CDM: ssh root@cdm01 ./test_cdm.exe --cdm-clocks ./test_cdm.exe --load-cc ./test_cdm.exe --reset-mgt ./test_cdm.exe --cdm-clocks ./test_cdm.exe --cdm-link # issue --reset-mgt on CDM and GDM until link is good ./test_cdm.exe --writereg 2 0xff # enable LEMO NIM inputs ./test_cdm.exe --writereg 3 0xba54 # enable LEDs: lemo1, lemo2, trg, tsm ./test_cdm.exe --writereg 4 0x99 # enable LEMO output trg_in ./test_cdm.exe --writereg 9 0x8040 # enable trg_in and tsm_in from sfp rx bits 0 and 1 ./test_cdm.exe --writereg 7 0xff00 # drive VX LVDS lines to logic level 0 ./test_cdm.exe --writereg 8 0x2 # VX LVDS with trg and tsm, misrouted in second VX #./test_cdm.exe --writereg 8 0x7 # VX LVDS with tsm ./test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz
switch GDM and CDM to packetizer trg and tsm:
on the GDM: /home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x00000000 on the CDM: /home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x0804 /home/dsdaq/online/ds-dm-software/test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz
Run packet loopback
GDM CPU -> fifo_to_fpga -> GDM QSFP -> CDM SFP -> fifo_from_fpga -> CDM CPU
On the GDM: (CDM is connected to first QSFP port)
/home/dsdaq/online/ds-dm-software/test_cdm.exe --load-cc /home/dsdaq/online/ds-dm-software/test_cdm.exe --reset-mgt /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link /home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 32 1 ### tell GDM to use first QSFP port /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link ### confirm link status is "3" /home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 29 2 ### enable GDM packet data injection /home/dsdaq/online/ds-dm-software/test_cdm.exe --test-fifo-write-loop
One the CDM:
/home/dsdaq/online/ds-dm-software/test_cdm.exe --load-cc /home/dsdaq/online/ds-dm-software/test_cdm.exe --reset-mgt /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link /home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 29 3 ### enable CDM packet data injection /home/dsdaq/online/ds-dm-software/test_cdm.exe --test-fifo-read
dsvslice integration
VX setup
- general
- Start acq from user code = y, all others = n (as of Mar 2023: will start when first trigger received)
- Use NIM IO = y
- Use external clock = y
- LVDS quartet is input = n, y, n, y
- LVDS quartet mode = User, User, User, User
- trigger from front panel NIM:
- Trigger on external signal = y, all others = n
- connect CDM EXT_OUT(2) to VX "TrigIn"
- trigger from LVDS "Sync" mode
- Trigger on LVDS Sync signal = y, all others = n
- LVDS quartet mode = User, Sync, User, User
- trigger from LVDS "User" mode
- Trigger on LVDS pair 12 signal = y, all others = n
- LVDS quartet is input = n, y, n, y
- LVDS quartet mode = User, User, User, User
GDM setup
- GDM is gdm0
- set inputs to NIM mode
- set outputs to TTL mode (this GDM has wrong NIM output circuit)
- use top QSFP slot, connect split cable 1 into CDM01, cable 2 into CDM02
- connect non-inverted NIM trigger signal to top-LEMO-left EXT_IN_LV(1)
- connect non-inverted NIM TSM signal to top-LEME-right EXT_IN_LV(2)
- GDM LEDs: TRIG, TSM, trigger enabled, trigger_out
- GDM LEMO_OUT: trigger, trigger
CDM setup
- set CDM LEMO inputs to NIM
- set CDM LEMO outputs to NIM
- CDM01 is cdm0
- CDM02 is cdm1
- connect GDM fiber links to SFP port
- connect 1st VX port of CDM01 to VX1
- connect 1st VX port of CDM02 to VX2
- connect LEMO EXT_OUT(2) to VX "TrigIn", CDM01 to VX1, CDM02 to VX2
- power up
- CDM LEDs: GDM TRIG, GDM TSM, trigger enabled, trigger out
- CDM LEMO_OUT: gdm_trg, trigger
After power up
- start the CDM frontend from the MIDAS "Programs" page. To start manually, see the Start Command on the Programs page.
- CDM frontend should enable the VX clock, disable the trigger
- from the MIDAS status page, goto the CDM page
- outdated: in the CDMx data tables, the 2nd number should read 0x35c08008, if it does not and the last 4 digits randomly change, reset the GDM links
- if the FEs complain - do in order: for GDM, CDM01, CDM02, ..., press "reset mgt" of each board, then press "unreset mgt", if it does not help, STOP HERE
- start a run
- CDM frontend will enable the trigger
- GDM frontend will enable the trigger
- LEDs on the GDM should flash, LEDs on the CDM should flash, TrigIn and TrigOut of the VX should flash
- stop a run
- GDM frontend will disable the trigger
- CDM frontend will disable the trigger
Phase measurement
- pip3 install matplotlib
- pip3 install scipy
- export PYTHONPATH=$HOME/packages/midas/python
- #git clone https://github.com/J033X071C/PhaseMeasurement
- git clone https://bitbucket.org/team-ds-dm/phasemeasurement.git
- cd phasemeasurement
- python3 ./phaseMeasurement.py --help
daq00:PhaseMeasurement$ python3 ./phaseMeasurement.py --help usage: phaseMeasurement.py [-h] fileName numberEvents numberVX sizeEvents stopEvent minHist maxHist numberBin writeToTXT saveAsPDF Read data from midas file (in .lz4 format) to calculate phase between the clock of VX1 and VX2 positional arguments: fileName Name of the file we want to read data from (Example: run00389.mid.lz4) numberEvents Number of events recorded in the file numberVX Number of VX used in this run (usually 2...) sizeEvents Number of points per event stopEvent Number of events you want to go through to calculate phase minHist Minimal value for the x axis of the phase measurement histogram (in ns) maxHist Maximal value for the x axis of the phase measurement histogram (in ns) numberBin Number of bins wanted for the generated histogram writeToTXT Write argument as yes to generate text file with results of calculation saveAsPDF Save generated plots to PDF files optional arguments: -h, --help show this help message and exit daq00:PhaseMeasurement$
- try an old file with
- python3 ./phaseMeasurement.py run00877.mid.lz4 10000 2 10000 500 -20 20 81 yes yes
bin size = 0.494 ns num_events = 347 mean = -1.705 ns rms = 3.087 ns mean_error = 0.166 ns centroid = -1.706 ns. width (sigma) = 0.363 ns. error on the centroid = 0.016558 ns.
- ls -l *.txt *.pdf
dsdaq@dsvslice:~/online/PhaseMeasurement$ ls -l *.txt *.pdf -rw-rw-r-- 1 dsdaq dsdaq 64728 Dec 14 16:56 run00877.mid.lz4_Plots.pdf -rw-rw-r-- 1 dsdaq dsdaq 274 Dec 14 16:56 run00877.mid.lz4.txt dsdaq@dsvslice:~/online/PhaseMeasurement$
- scope settings (from email message)
From fcote-lortie@triumf.ca Thu Dec 15 17:11:46 2022
From: Francis Cote-Lortie <fcote-lortie@triumf.ca>
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: Re: How to use scope as a waveform generator
Date: Fri, 16 Dec 2022 01:11:44 +0000
1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display) by pressing on the button. It will go from 0 to 1.
The settings that we are using right now are:
Type of waveform: Sine wave
Offset: 0 V
Amplitude: 1 Vpp
Frequency: 50 kHz
Noise: 0 V
________________________________
From: Francis Cote-Lortie <fcote-lortie@triumf.ca>
Sent: Thursday, December 15, 2022 4:58 PM
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: Re: How to use scope as a waveform generator
1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display)
The settings that we are using right now are:
Type of waveform: Sine wave
Offset: 0 V
Amplitude: 1 Vpp
Frequency: 50 kHz
Noise: 0 V
________________________________
From: Francis Cote-Lortie
Sent: Thursday, December 15, 2022 4:54 PM
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: How to use scope as a waveform generator
1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display)
Standalone link test
CDM: program clock chip busybox devmem 0x80011000 32 0x8 busybox devmem 0x80011000 32 0x0 /home/dsdaq/si5394-i2c-file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b GDM, CDM: link reset busybox devmem 0x80011008 32 1 GDM, CDM: release reset busybox devmem 0x80011008 32 0 CDM: link status (NOTE: SFP LOS and mod_absent are swapped!!!) busybox devmem 0x80011010 32 0x00000024 <- fiber plugged 0x00000025 <- fiber unplugged 0x00000027 <- SFP unplugged 0x000007DC <- successful link with GDM CDM: link state machine and data busybox devmem 0x80011014 32 0x35C06FF6 CDM: set link to counting mode busybox devmem 0x80012000 32 0x101 busybox devmem 0x80012000 32 0x100 CDM: time counter and error counter root@cdm1:~# busybox devmem 0x80012000 32 0x00000100 <--- link mode root@cdm1:~# busybox devmem 0x80012004 32 0x0000058C <--- seconds counter root@cdm1:~# busybox devmem 0x80012004 32 0x0000058D root@cdm1:~# busybox devmem 0x80012008 32 0x00000000 <--- error counter GDM: no link root@gdm0:~# busybox devmem 0x80011014 32 0x00000000 root@gdm0:~# busybox devmem 0x80011018 32 0x00000000 root@gdm0:~# busybox devmem 0x8001101c 32 0x00000000 root@gdm0:~# busybox devmem 0x80011024 32 0x00000FFF root@gdm0:~# GDM: good link channel 10, counting mode root@gdm0:~# busybox devmem 0x80012000 32 0x101 root@gdm0:~# busybox devmem 0x80012000 32 0x100 root@gdm0:~# busybox devmem 0x80012008 32 0x3A8B68C2 root@gdm0:~# busybox devmem 0x80012008 32 0x42E03BEF root@gdm0:~# busybox devmem 0x8001200c 32 0xDA090972 root@gdm0:~# busybox devmem 0x8001200c 32 0xDE6F22E9 root@gdm0:~# busybox devmem 0x80012019 32 Bus error root@gdm0:~# busybox devmem 0x80012010 32 0xFFFFFFFF root@gdm0:~# busybox devmem 0x80012014 32 0xFFFFFFFF root@gdm0:~# busybox devmem 0x80012018 32 0xFFFFFFFF root@gdm0:~# busybox devmem 0x8001201c 32 0xFFFFFFFF root@gdm0:~# busybox devmem 0x80012020 32 0xFFFFFFFF root@gdm0:~# busybox devmem 0x80012024 32 0xFFFFFFFF root@gdm0:~# busybox devmem 0x80012028 32 0x00000000 root@gdm0:~# busybox devmem 0x8001202c 32 0xFFFFFFFF root@gdm0:~# busybox devmem 0x80012030 32 0xFFFFFFFF root@gdm0:~# busybox devmem 0x80012034 32 0xFFFFFFFF root@gdm0:~#
mapping of link channels:
qsfp0 lane0 - 0x0100 - link 8 qsfp0 lane1 - 0x0200 - link 9 qsfp0 lane2 - 0x0400 - link 10 qsfp0 lane3 - n/c qsfp1 lane0 - 0x0800 - link 11 qsfp1 lane1 - 0x0010 - link 4 qsfp1 lane2 - 0x0020 - link 5 qsfp1 lane3 - n/c qsfp2 lane0 - 0x0040 - link 6 qsfp2 lane1 - 0x0080 - link 7 qsfp2 lane2 - 0x0001 - link 0 qsfp2 lane3 - n/c qsfp3 lane0 - 0x0002 - link 1 qsfp3 lane1 - 0x0004 - link 2 qsfp3 lane2 - 0x0008 - link 3 qsfp3 lane3 - n/c
script to start the test with 2 CDMs:
ssh dsdaq@dsvslice ssh root@gdm0 busybox devmem 0x80011008 32 1 ssh root@cdm0 busybox devmem 0x80011008 32 1 ssh root@cdm1 busybox devmem 0x80011008 32 1 ssh root@gdm0 busybox devmem 0x80011008 32 0 ssh root@cdm1 busybox devmem 0x80011008 32 0 ssh root@cdm0 busybox devmem 0x80011008 32 0 ssh root@gdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2 ssh root@cdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2 ssh root@cdm1 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
LEMO trigger GDM to CDM to VX
on the GDM:
/home/dsdaq/si5394-i2c-file /home/dsdaq/GDM_v1.0_IN0_EXT1_and_IN1_fixed_Si5394-RevA-Registers.txt 0 0x6b busybox devmem 0x80011008 32 0x1 busybox devmem 0x80011008 32 0x0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x80001230 # QSFP TX fixed pattern /home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 2 0xff # enable LEMO inputs /home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 3 0x7654 # enable LED, one per LEMO input /home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x0F0F # enable LEMO to trg_in and tsm_in /home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x40001230 # enable trg_in and tsm_in output to QSFP
on the CDM:
/home/dsdaq/si5394-i2c-file /home/dsdaq/CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks /home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp busybox devmem 0x80011008 32 1 busybox devmem 0x80011008 32 0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link /home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x8040 # trg_in from sfp[0], tsm_in form sfp[1] /home/dsdaq/online/ds-dm-software/test_cdm.exe 11 # trg_in counter /home/dsdaq/online/ds-dm-software/test_cdm.exe 12 # tsm_in counter
GPS receiver VCL-2705
- Valiant VCL-2705 GPS receiver
- https://www.valiantcom.com/time-distribution/gps-receiver-irig-b.html
- USB connection is /dev/ttyACM0, 115200 bps, "GNSSAUX" prompt
- minicom -D /dev/ttyACM0 -b 115200
- user manual download instructoins - see sheet of paper with user name and password in the shipping box
- usb commands:
gnss-help gnss-showver -> F/W: Ver 1.7 Feb 17 2020 16:20:43 gnss-showselftest -> no antenna connected Overall : FAIL EPROM Test: PASS Antenna : NOT DETECTED GNSS : COMMUNICATION OK gnss-showsettings GNSS NMEA BAUDRATE :115200 GNSS ANTENNA LENGTH :30 meters GNSS USER CONFIGURED DELAY:-65 nanoseconds GNSS 1PPS PULSE WIDTH :200 milliseconds GNSS MODE :GPS GNSS STATUS :STATIONARY GNSSAUX> gnss-showserial SERIAL :2704H01V17MAX310 GNSSAUX> gnss-showmode GNSS MODE : GPS GNSSAUX> gnss-showstatus GNSS STATUS: STATIONARY GNSSAUX> gnss-showalarms CURRENT ALARMS GNSS ANTENNA : NOT DETECTED GNSSAUX> gnss-showerrors CURRENT ALARMS GNSS ANTENNA : **NOT DETECTED ERROR STATISTICS GNSS RMC GOOD DURATION : Secs 0 RMC BAD DURATION : Secs 0 LOCK GOOD SECS DURATION : Secs 0 LOCK BAD SECS DURATION : Secs 0 SATINFO GOOD ITERATIONS :0 SATINFO REJECT ITERATIONS :0 SATINFO NOTALKER ITERATIONS :0 CURRENT MONITOR STATE GNSS STATE :Phase-1 HUNTING ANTENNA DETECT GNSSAUX> gnss-showsatinfo GNSS RECEIVER ANTENNA Not Detected ! GNSSAUX> gnss-showsats Total Sats: 0 GNSSAUX> gnss-showmyloc GNSS RECEIVER ANTENNA Not Detected ! GNSSAUX> gnss-show1ppsstate GNSS RECEIVER ANTENNA Not Detected ! GNSSAUX> gnss-showjamstatus Not Available ! GNSSAUX> gnss-showspoofstatus Not Available ! --- antenna connected, can see the sky --- GNSSAUX> gnss-showselftest Overall : PASS EPROM Test: PASS Antenna : DETECTED GNSS : COMMUNICATION OK GNSSAUX> gnss-showalarms CURRENT ALARMS GNSS ANTENNA : DETECTED GNSS LOCK : AVAILABLE GNSSAUX> gnss-showerrors CURRENT ALARMS GNSS ANTENNA : DETECTED GNSS LOCK : AVAILABLE ERROR STATISTICS GNSS RMC GOOD DURATION : Mins 1,Secs 34 RMC BAD DURATION : Secs 55 LOCK GOOD SECS DURATION : Mins 1,Secs 34 LOCK BAD SECS DURATION : Secs 55 SATINFO GOOD ITERATIONS :3 SATINFO REJECT ITERATIONS :0 SATINFO NOTALKER ITERATIONS :0 CURRENT MONITOR STATE GNSS STATE :Phase-4 NORMAL OPERATION, Monitoring GNSSLOCK GNSSAUX> gnss-showsatinfo GNSS MODE :GPS SATELLITE INFORMATION TALKER:GPS NMEA ID : 01-32 NO OF SATELLITES IN VIEW: 08 NO OF XXGSV MSGS : 03 SatNo PRN NO (SV ID) ELEVATION (degs) AZIMUTH (degs) C/No (SNR) 1 0 0 0 0 2 0 0 0 0 3 0 0 0 0 4 0 0 0 0 5 0 0 0 0 6 0 0 0 0 7 0 0 0 0 8 0 0 0 0 <<<<<< End of Sat Info >>>>>> GNSSAUX> gnss-showsats Total Sats: 8 GNSSAUX> gnss-showmyloc GNSS RECEIVER LOCATION: Latitude : 4914.81911 Longitude: 12313.69595 GNSSAUX> gnss-show1ppsstate GPS 1PPS STATE: LOCKED GNSSAUX> gnss-showmyloc GNSS RECEIVER LOCATION: Latitude : 4914.80688 Longitude: 12313.69531 enter into google maps search box as: 49 14.80688, -123 13.69531, observe the space, the moved dot and the minus. GNSSAUX> gnss-resetgnss Executing....Please Wait.... $$$$$END
- 1PPS BNC output: period 1 sec, pulse width 200 ms, 3.3V into 1MOhm, 1.38V into 50Ohm.
- IRIG-B BNC output: 5.6V into 1MOhm, 2.4V into 50Ohm.
- IRIG-B format selector: default is all up.
Rb clock PRS10
- https://www.thinksrs.com/products/prs10.html
- 10 MHz output is sine wave around 5V peak to peak
- 1PPS BNC output is 10 usec pulse, 5V into 1MHohm.
- RS232 connection: minicom -D /dev/ttyUSB0 -b 9600
- ser2net config: localhost,3001:raw:600:usb-5-2-1.0:9600 -XONXOFF -RTSCTS LOCAL
- ssh daq13, cd /home/olchansk/git/ser2net, ./ser2net -c ~/daq/ds/ser2net.conf -d
- ssh daq13, cd ~/daq/ds, python3 prs10.py
- RS232 commands:
ID? PRS10_3.56_SN_105719 VB1 SN? RS1 -- reset ST? -- status FC? -- 10MHz OCXO drive voltage DAC settings DS? -- "detected signals" GA? -- gain of frequency lock loop between ovenized oscillator and Rb cell, 0=use ovenized oscillator only MO? -- magnetic offset of the Rb cell, range 2300..3600, if out of range, unit must be set to different operating mode, see prs10m.pdf MR? -- magnetic read TT? -- time-tag, time in ns between 1PPS out and 1PPS in TS? -- time slope, ??? TO? -- time offset, ??? PS? -- pulse slope, ??? PL? -- 0=phase lock off, 1=phase lock on, lock to 1PPS input PT? -- phase lock integrator time constant, PT8 is integrator time constant 18.2 hours, natural time constant 2.25 hours PF? -- phase lock stability factor, PF2 is "1" PI? -- phase lock integrator Analog to digital 12 bit ADC, values 0.000 to 4.998 AD0? -- Spare (J204) AD1? -- +24V(heater supply) divided by 10. AD2? -- +24V(electronics supply) divided by 10 AD3? -- Drain voltage to lamp FET divided by 10 AD4? -- Gate voltage to lamp FET divided by 10 AD5? -- Crystal heater control voltage AD6? -- Resonance cell heater control voltage AD7? -- Discharge lamp heater control voltage AD8? -- Amplified ac photosignal AD9? -- Photocell’s I/V converter voltage divided by 4 AD10? -- Case temperature (10 mV/°C) AD11? -- Crystal thermistors AD12? -- Cell thermistors AD13? -- Lamp thermistors AD14? -- Frequency calibration pot / external calibration voltage AD15? -- Analog ground A/D via CPU E-port: AD16? -- Varactor voltage for 22.48 MHz VCXO (inside RF synthesizer) / 4 AD17? -- Varactor voltage for 360 MHz VCO (output of RF synthesizer) / 4 AD18? -- Gain control voltage for amplifier which drives frequency multiplier / 4 AD19? -- RF synthesizer’s lock indicator voltage (nominally 4.8 V when locked ) ST? ST1 : Power supplies and Discharge Lamp ST1 bit, Condition which sets bit, Corrective Action 0 -- +24 for electronic < +22 Vdc 1 -- +24 for electronics > +30 Vdc 2 -- +24 for heaters <+22 Vdc 3 -- +24 for heaters > +30 Vdc 4 -- Lamp light level too low 5 -- Lamp light level too high 6 -- Gate voltage too low 7 -- Gate voltage too high ST2: RF Synthesizer ST2 bit, Condition which sets bit, Corrective Action 0 -- RF synthesizer PLL unlocked 1 -- RF crystal varactor too low 2 -- RF crystal varactor too high 3 -- RF VCO control too low 4 -- RF VCO control too high 5 -- RF AGC control too low 6 -- RF AGC control too high 7 -- Bad PLL parameter ST3: Temperature Controllers ST3 bit, Condition which sets bit 0 -- Lamp temp below set point 1 -- Lamp temp above set point 2 -- Crystal temp below set point 3 -- Crystal temp above set point 4 -- Cell temp below set point 5 -- Cell temp above set point 6 -- Case temperature too low 7 -- Case temperature too high ST4: Frequency Lock-Loop Control ST4 bit, Condition which sets bit 0 -- Frequency lock control is off 1 -- Frequency lock is disabled 2 -- 10 MHz EFC is too high 3 -- 10 MHz EFC is too low 4 -- Analog cal voltage > 4.9 V 5 -- Analog cal voltage < 0.1 6 -- not used 7 -- not used ST5: Frequency Lock to External 1pps ST5 bit, Condition which sets bit 0 -- PLL disabled 1 -- < 256 good 1pps inputs 2 -- PLL active 3 -- > 256 bad 1pps inputs 4 -- Excessive time interval 5 -- PLL restarted 6 -- f control saturated 7 -- No 1pps input ST6: System Level Events ST6 bit and Condition which sets bit 0 Lamp restart 1 Watchdog time-out and reset 2 Bad interrupt vector 3 EEPROM write failure 4 EEPROM data corruption 5 Bad command syntax 6 Bad command parameter 7 Unit has been reset
- ST? on warm start
received: PRS_10 received: 255,255,255,243,34,255 received: 0,0,0,1,34,0 ... received: 0,0,0,1,34,0 received: 0,0,0,0,2,0 ... received: 0,0,0,0,2,0 received: 0,0,0,0,4,0
- ST? on loss of external 1PPS
... received: 0,0,0,0,4,0 disconnect 1PPS input received: 0,0,0,0,132,0 ... reconnect 1PPS input received: 0,0,0,0,132,0 received: 0,0,0,0,4,0 ...
- ST? on coldish start
daq13:ds$ python3 prs10.py Connected received [ ] old [ b'' ] counter: 0 received [ PRS_10 ] old [ ] counter: 0 received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0 received [ 80,0,0,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0 received [ 0,0,1,1,34,0 ] old [ 80,0,0,1,34,1 ] counter: 0 received [ 0,0,0,1,34,0 ] old [ 0,0,1,1,34,0 ] counter: 0 received [ 0,0,16,1,34,0 ] old [ 0,0,0,1,34,0 ] counter: 6 received [ 0,0,20,1,34,0 ] old [ 0,0,16,1,34,0 ] counter: 5 received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 8 received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 3 received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 49 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 250 ...
- ST? on cold start, note: no bump in the 10MHz clock as reported by DS-DM clock chip
Connected received [ ] old [ b'' ] counter: 0 received [ PRS_10 ] old [ ] counter: 0 received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0 received [ 80,0,21,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0 received [ 64,0,21,1,34,0 ] old [ 80,0,21,1,34,1 ] counter: 0 received [ 0,0,21,1,34,0 ] old [ 64,0,21,1,34,0 ] counter: 0 received [ 0,0,20,1,34,0 ] old [ 0,0,21,1,34,0 ] counter: 158 received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 23 received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 47 received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 40 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 249
det fac integration test
Connections:
- GPS receiver "IRIG-B SEL" both switches "up" - both "on", IRIG-B format B004
- GPS receiver USB-B -> long cable -> daq13 USB-A
- GPS receiver "1PPS out" -> long BNC cable -> BNC-T -> scope (5V, no 50ohm) and Rb clock BNC "1PPS in"
- GPS receiver "IRIG-B 50ohms" -> long BNC cable -> BNC-T -> scope (5V, no 50ohm) and DS-DM LEMO input 1 (TTL mode)
- Rb clock RS232 -> RS232 straight cable -> RS232-to-USB adapter -> daq13 USB-A
- Rb clock "1PPS out" BNC -> scope (5V, no 50ohm, trig threshold rising edge 2V)
- Rb clock "10MHz output 50 Ohm" BNC -> lemo -> lemo-T -> scope (sine wave, 5V, no 50ohm) and DS-DM clock input.
Programs to run:
- on daq13: cd /home/olchansk/git/ser2net, ./ser2net -c ~/daq/ds/ser2net.conf -d
localhost,3001:raw:600:usb-5-2-1.0:9600 -XONXOFF -RTSCTS LOCAL
- on daq13: cd /home/olchansk/daq/ds, python3 prs10.py ### connects to ser2net
daq13:ds$ python3 prs10.py Connected received [ ] old [ b'' ] counter: 0 received [ PRS_10 ] old [ ] counter: 0 received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0 received [ 80,0,21,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0 received [ 64,0,21,1,34,0 ] old [ 80,0,21,1,34,1 ] counter: 0 received [ 0,0,21,1,34,0 ] old [ 64,0,21,1,34,0 ] counter: 0 received [ 0,0,20,1,34,0 ] old [ 0,0,21,1,34,0 ] counter: 158 received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 23 received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 47 received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 40 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 249 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 12096 received [ 0,0,0,0,20,0 ] old [ 0,0,0,0,132,0 ] counter: 2 received [ 0,0,0,0,148,0 ] old [ 0,0,0,0,20,0 ] counter: 0 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,148,0 ] counter: 575 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 11449 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 2755 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 34386 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 41035 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 113401 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 33375 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 54767 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 85059 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 33222 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 119234 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 121990 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 128184 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 56002 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 428237 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 8250 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 1388 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 30506 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 142704 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 179451 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 106182 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 68747 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 65424 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 157587 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 6932 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 1388 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 20255 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 4 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 225941 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 72183 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0 received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 26970 received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 1 39287
- on ds-dm: ./test_cdm.exe --irigb ### note sbs mismatch is because I should wrap around at 16 bits
dataframe: S00000000S100001010S...S001010101S000001000S, sec: 00, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16724 should be 82260 dataframe: S10000000S100001010S...S101010101S000001000S, sec: 01, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16725 should be 82261 dataframe: S01000000S100001010S...S011010101S000001000S, sec: 02, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16726 should be 82262 dataframe: S11000000S100001010S...S111010101S000001000S, sec: 03, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16727 should be 82263 dataframe: S00100000S100001010S...S000110101S000001000S, sec: 04, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16728 should be 82264 dataframe: S10100000S100001010S...S100110101S000001000S, sec: 05, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16729 should be 82265
DS-IOGC GPS interface board
- Rev0 git repository: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0/-/tree/main?ref_type=heads
- Rev0 schematics: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0/-/blob/main/Altium/Project%20Outputs%20for%20DS-IOGC-Rev0/SCH-DS-IOGC-Rev0.pdf?ref_type=heads
- Rev0 schematics: SCH-DS-IOGC-Rev0
- Rev1 git repository: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev1/
- Rev1 schematics: SCH-DS-IOGC-Rev1
Changes Rev0 to Rev1
From: Peter Margetak <pmargetak@triumf.ca> Subject: IOGC REV1 review Date: Wed, 4 Sep 2024 07:31:19 +0000 Hi Konstantin, Pls have a look at SCH for new rev. I'd like to send it to mfr next week so if you can comment by early next week. Meanwhile I work on layout and other stuff. Changes: New ICs - all powered +5V U20 - inverters for RUclk RX/TX U21 - non inverting line driver for RU-1pps-out (so you don't have to route if via GDM to see it on scope) U22 - non inverting buffer for ext 1pps input All Lemo connectors have the same position but they are double lemos now => new panel needed @Marek Walczak<mailto:mwalczak@triumf.ca> you can print it ahead once pcb is done + update IOGC docs and panel description J2A/B - Test ports for RU-1pps in and out J5A/B - inputs for external GPS data and External source of 1pps J6A/B - aux in/out for GDM SW1 - no change - select RX/TX USB/GDM SW2 - select latch sensitivity for rising/falling edge SW3 - select source of GPS data (opto or ext) AND select source of 1pps input (latch or ext) p.
PRS-10 Rb clock device
The Rb clock PRS-10 device provides these connections:
RS232 RX input - serial communication, non-standard RS232 RS232 TX output - serial communication, non-standard RS232 10 MHz clock output - coax 50 Ohm high resolution 10 MHz clock 1pps output - 1 Hz clock corresponding to the 10 MHz clock 1pps input - 1pps signal from GPS receiver
Mode of operation:
- 10 MHz clock is always running
- 1pps output is always running
- if 1pps input from GPS received is present, after 256 pulses PRS-10 will sync it's 1pps output with the 1pps input by adjusting the frequency of the 10 MHz clock
- when unlocked: 1pps output and 1pps input unrelated
- when locked to GPS: 1pps output and 1pps input always go up and down at the same time
Theory of operation:
- 10 MHz clock is produced by a high-quality crystal (stable on the scale of seconds)
- crystal oscillator is synchronized to a Rb cell (stable on the scale of hours and days)
- Rb cell resonant frequency is synchronized to the GPS 1pps signal (stable on the scale of months and years)
Rev1 connections
- LEMO connectors (front panel)
LEMO J2A output - Rb clock 1pps in monitor LEMO J2B output - Rb clock 1pps out monitor LEMO J5A input - GPS IRIG-B from GPS receiver to FPGA (VCL-2705) LEMO J5B input - GPS 1pps from GPS receiver to PRS-10 (VCL-2705) LEMO J6A input - AUX-IN to FPGA LEMO J6B output - AUX-OUT from FPGA
- SMB connectors (back)
SMB J3 output - GPS 1pps loopback to LNGS SMB J4 input - LNGS GPS data input
- LEDs
D1 - same as LEMO J2A out (Rb clock 1pps in) D2 - same as SMB J3 out (GPS 1pps from LNGS or from a GPS receiver) D5 - controlled by FPGA-OUT-LED1 D6 - controlled by FPGA-OUT-LED2 D7 - PRS-10 24V power ok
- switches
SW1A and SW1B - route PRS-10 RS232 to USB or to FPGA SW2A - route PRS-10 1pps input from SMB J4 (LNGS) or from LEMO J5B (GPS receiver 1pps) SW2B - route FPGA-IN-GPSDATA input from SMB J4 (LNGS) or from LEMO J5A (GPS receiver IRIG-B data) SW3 - LNGS 1pps from rising edge or from falling edge of SMB J4 (LNGS)
Rb clock cable
Rb clock ----- DS-IOGC side, pin numbers are as labeled on the cable connectors 1 - 1pps out - 8 - 1pps out --- correct 2 - nc 3 - nc 4 - TXD - 3 - RU-DATA-OUT - USB-RX input --- correct 5 - 1pps in - 2 - 1pps in --- correct 6 - +24V - 10 - +24V --- should by pin 1 to use both +24V pins? 7 - RXD - 15 - RU-DATA-IN - USB-TX output --- correct 8 - nc 9 - +24V - 10 - +24V --- correct 10 - GND - 9 - GND --- correct
VX connections
VXA_TX0 - FPGA-OUT-LED2 - D6 LED ("10 MHz clock")
VXA_TX1 - FPGA-OUT-LED1 - D5 LED ("GPS DATA")
VXA_TX2 - FPGA-OUT-RU1PPS-EN - enable 1pps to Rb clock
VXA_TX3 - not used (62.5 Hz clock)
VXB_TX0 - FPGA-TX - PRS-10 RS-232 out
VXB_TX1 - FPGA-OUT-AUX - 5V TTL J5 LEMO out
VXB_TX2 - FPGA-OUT-OPTO1PPS-EN - PRS-10 power enable
VXB_TX3 - not used (62.5 MHz clock)
VXA_RX0 - FPGA-IN-OPTO1PPS - 1pps from GPS
VXA_RX1 - FPGA-IN-GPSDATA - J4 GPS data
VXA_RX2 - n/c
VXA_RX3 - n/c
VXB_RX0 - n/c
VXB_RX1 - FPGA-IN-AUX - J5 LEMO TTL input
VXB_RX2 - FPGA-RX - PRS-10 RS-232 in
VXB_RX3 - FPGA-IN_RU1PPS - PRS-10 1pps output
test sequence
- ./test_cdm_local.exe --writereg 7 0 ### clear reg 7
- ./test_cdm_local.exe --writereg 8 0 ### set vx_tx mux to vx_tx control from reg 7
- connect blue cable to GDM port 6 (next to the ethernet connector)
- ./test_cdm_local.exe --writereg 7 0x4000 ### power up
- ./test_cdm_local.exe --writereg 7 0x4100 ### right LED D6
- ./test_cdm_local.exe --writereg 7 0x4200 ### left LED D5
- ./test_cdm_local.exe --writereg 7 0x6000 ### J5 LEMO measure +5VDC
- install LEMO jumper between LEMO J5 and J6
- write 0x4000 then ./test_cdm_local.exe 6 ### reads 0x5C, bit 5 0x20 reads 0
- write 0x6000 then ./test_cdm_local.exe 6 ### reads 0x7C, bit 5 0x20 reads 1
- write 0x4000 to clear all bits
- 10 MHz output ok - IN0 is good: ./test_cdm_local.exe --cc
Clock chip state 1, status: OOF_IN2 IN1 IN_SEL_1 HOLD_HIST_VALID
- without 10 MHz reports IN0 LOS (no signal) and OOF (wrong frequency)
Clock chip state 1, status: LOS_IN0 OOF_IN0 OOF_IN2 IN1 IN_SEL_1 HOLD_HIST_VALID
- IRIG-B via AUX-IN is ok: ./test_cdm_local.exe --irigb
dataframe: S00100110S111001000S...S011110000S010000000S, sec: 34, min: 17, hrs: 00, day of year: 223, year: 2024, date: 10 aug, sbs: 1054 should be 1054
- ./test_cdm_local.exe --writereg 8 9 ### set vx_tx mux to GPS control, hard enables PRS-10 power and PRS-10 1pps in
- observe 1pps counters in reg 68 count at 1 Hz: rb_1pps 0x51->0x52, gps_1pps 0xd5->oxd6
root@dsdm:~# ./test_cdm_local.exe 68 ds20k_reg[68] is 0x0051d520 (5362976) ds20k_reg[68] is 0x0052d624 (5428772) ...
- observe PRS-10 can see the 1pps signal "130" changes to "2" after 243 seconds to "4"
- observe period of GPS and PRS-10 1pps is identical (plus/minus 1 clock)
root@dsdm:~# ./test_cdm_local.exe 13 14 ds20k_reg[13] is 0x077356d4 (124999380) ds20k_reg[14] is 0x077356d4 (124999380)
- observe GDM/CDM clocks are exactly 125 MHz, we are running on DS-DM internal oscillator
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks GDM clock frequency counters: 0x1030 mgt_rx_ref_clk_raw: 0x07735943 (125000003) should be ~125 MHz 0x1034 rx_clk: 0x07735943 (125000003) should be ~125 MHz 0x1038 mgt_tx_ref_clk_raw: 0x07735943 (125000003) should be ~125 MHz 0x103C tx_clk: 0x07735943 (125000003) should be ~125 MHz 0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly 0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
- switch CC to external clock:
root@dsdm:~# ./test_cdm_local.exe --cc-in0 CC use clock input 0: 10 MHz LEMO external clock root@dsdm:~# ./test_cdm_local.exe --cc Polling CC status... Clock chip state 1, status: IN0 IN_SEL_REGCTRL IN_SEL_0 HOLD_HIST_VALID
- observe rx_clk and tx_clk are now slightly off: they run from PRS-10 10 MHz clock and are measured against the DS-DM internal oscillator.
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks DS-DM mapping /dev/mem at 0x80010000 DS-DM FPGA firmware build 0x94b12519, ds20k version 0x20240814 DS-DM firmware build 0x94b12519, ds20k version 0x20240814 GDM clock frequency counters: 0x1030 mgt_rx_ref_clk_raw: 0x077358e1 (124999905) should be ~125 MHz 0x1034 rx_clk: 0x07735b49 (125000521) should be ~125 MHz 0x1038 mgt_tx_ref_clk_raw: 0x07735b49 (125000521) should be ~125 MHz 0x103C tx_clk: 0x07735b49 (125000521) should be ~125 MHz 0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly 0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
- observe GPS and PRS-10 1pps period is now exactly 125*10^6 of 8ns clocks (equal to 1 second)
root@dsdm:~# ./test_cdm_local.exe 13 14 ds20k_reg[13] is 0x0773593f (124999999) ds20k_reg[14] is 0x0773593f (124999999)
- look at them repeatedly, observe reg 13 "GPS 1pps period" has some wobble, reg 14 "Rb clock 1pps period" is steady. this is as expected: if GPS 1pps was steady, we do not need to Rb clock. PRS-10 1pps is derived from the PRS-10 10 MHz clock and is measured against the 125 MHz clock derived from the same 10 MHz clock in the GDM FPGA.
Test status:
- GPS 1pps to SMB-in ok (LED flashes)
- GPS 1pps to FPGA ok
- GPS data to FPGA ok
- GPS 1pps to PRS-10 enabled from FPGA ok
- PRS-10 Rb clock 1pps out to FPGA ok
- PRS-10 Rb clock 1pps out to FPGA to LEMO AUX out ok
- can see GPS 1pps, IRIG-B, PRS-10 1pps out, 10 MHz on the scope, ok
- PRS-10 syncs on leading edge (0->1) of GPS 1pps signal, ok
- reg 13 and 14 1pps periods are identical, ok
- CC locks on PRS-10 10 MHz clock, DS-DM runs on PRS-10 clock, ok
- NOT TESTED - smb output
- NOT TESTED - optical converter fiber to SMB
- NOT TESTED - optical converter SMB to fiber
- NOT TESTED - SMB loopback
- NOT TESTED - fiber loopback
VX busy logic
DS-20K DAQ
Overview
DS-DM, GDM and CDM are key parts of the DS-20K DAQ system:
- common clock distribution from external clock (atomic clock, GPS) to GDM to per-quadrant CDMs to VX digitizers
- common trigger distribution from GDM internal algorithm or external input to all VX digitizers
- run control: GDM, CDM, VX all start recording data at the same time (clock and timestamp reset)
- collection of trigger data from VX digitizers to per-quadrant CDMs to GDM
Deliverables
- hardware and firmware for GDM to CDM clock distribution
- hardware and firmware for CDM to VX clock distribution
- hardware and firmware for GDM external clock input (atomic clock or GPS)
- hardware and firmware for CDM and VX serial communications (VX LVDS I/O connector)
- firmware for run control (timestamp reset and sync): GDM to CDM to VX
- firmware for common trigger distribution: GDM to CDM to VX
- firmware for trigger data flow: VX to CDM to GDM
- firmware for busy control: VX to CDM to GDM back to CDM to VX
- firmware for flow control: FEP to GDM MIDAS frontend to GDM to CDM to VX
- GDM MIDAS frontend: clock selector and monitoring, trigger and run control, busy and flow control, GDM housekeeping
- CDM MIDAS frontend: clock monitoring, CDM housekeeping
specific performance:
- GDM external clock: 10 MHz GPS clock
- GDM to CDM fiber link:
- clock XXX MHz
- link data rate: XXX Gbit/sec
- CDM recovered clock: XXX MHz
- CDM recovered clock jitter: XXX ns
- phase alignment between CDMs: XXX ns
- phase alignment between CDMs persists across reboots, power cycles, firmware updates
- phase alignment between CDMs should be easy to measure
- phase alignment between CDMs should be easy to recalibrate if hardware parts are replaced (DS-DM boards, fiber transceivers, fiber cables, etc)
- data packet bandwidth: XXX Mbytes/sec
- data packet latency: XXX clocks
- data packet skew between CDMs: XXX clocks
- CDM to VX clock:
- clock: XXX MHz
- jitter, all CDM clock outputs: XXX MHz
- phase alignment between all CDM clock outputs: XXX ns
- CDM to VX trigger:
- TBD (use the VX "sync" input or VX LVDS I/O line or VX serial link packet)
- CDM to VX serial link:
- clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
- bit rate: XXX bits/sec
- latency: XXX link clocks
- maximum skew between VXes: XXX ns
- VX to CDM serial link:
- clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
- bit rate: XXX bits/sec
- latency: XXX link clocks
- maximum skew between VXes: XXX ns
- timestamp reset:
- maximum skew between VXes: XXX ns
- busy round trip time: XXX ns (VX to CDM to GDM back to CDM to VX)
- flow control latency: XXX ns (FEP software to GDM MIDAS frontend to GDM to CDM to VX)
Technical risk items
this refers to unexpected behaviour and performance of system components, causes big difficulty in implementing the system, prevents delivery of deliverables, and prevents or negatively affects operation of the DS-20K DAQ or of the whole experiment.
(14-sep-2022, list is not sorted by any criterial: severity, probability, ease of investigation)
(stability of course is long term stability, across hours, days, weeks, months, years)
- stability of Enclustra FPGA modules (crashes/year, failures to boot/year, flash corruption/year)
- stability of GDM external clock PLL (lock loss/year)
- stability of CDM recovered clock (lock loss/year, unexpected phase drifts, etc)
- unexpected failures or bit error rates in GDM-CDM fiber links
- stability of CDM VX clock outputs (stability of clock cleaner chip)
- stability of VX internal clock distribution (VX PLL lock loss events)
- stability of VX CAEN base firmware (different versions of CAEN base firmware have different clock distribution behaviour)
- strange things in CAEN base firmware (unexpected clocking of LVDS I/O, unexpected phase shifts between clocks, etc)
- DS-DM and VX hardware problems (incompatible LVSD I/O, incompatible clock signals, etc)
Milestones
(14-sep-2022: at this stage of the project, priority must be given to identifying and retiring (so called) technical risk factors. it is not good to build the complete system only to discover that (for example) some Enclustra FPGA modules require 5 attempts to boot and erase their flash memory contents once a month. Both example are real-life actual problems that caused big difficulties in GRIFFIN/TIGRESS and ALPHA-g experiments).
Development and testing milestones in time reversed order:
- full DAQ data challenge: all VXes, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
- one quadrant data challenge: 1 VME crate of VX, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
- vertical slice data challenge: 1 VME crate, 2 VX, 2 CDM (1 VX per CDM), GDM, etc operate as designed
- GDM-CDM link finalized (data rate frozen, data packet format frozen, data content permitted to change)
- CDM-VX serial link finalized (data rate frozen, data packet format frozen, data content permitted to change)
- run control (timestamp reset) and trigger distribution design agreed upon, frozen (list of possible triggers permitted to change)
- VX to CDM to GDM data flow design agreed upon, frozen (data contents permitted to change)
- major technical risk items retired (all hardware and firmware is working as expecred without mysteries and surprises, all problems are identified, investigated, resolved, solutions tested)
- stable operation of CDM-VX serial links in vertical slice system
- stable operation of GDM to CDM clock in vertical slice system
- stable operation of CDM to VX clock in vertical slice system
- vertical slice system assembled (1 VME crate, 2 VX, 2 CDM, 1 GDM, network, FEP, TSP, MIDAS)
ZZZ
ZZZ