DS-DM

DarkSide-20k Global and Crate Data Manager board (GDM and CDM).

Global Data Manager (GDM):

clock distribution to CDM boards (including GPS/atomic clock source)
collection of trigger data from CDM boards, processing and distribution of trigger decision to CDM boards
run control
integration with GPS 10MHz and 1pps clocks and GPS/IRIG date and time information

Crate Data Manager (CDM):

clock distribution from GDM to CAEN VX digitizers
receive trigger data from CAEN VX digitizers
send trigger data to GDM
run control and dead time control

Links

https://dsvslice.triumf.ca/vx_napoli - DS vertical slice at TRIUMF
https://edev-group.triumf.ca/fw/exp/darkside/gcdm - git repository, DS-DM firmware
https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0 - git repository, DS-DM board
https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0/-/blob/master/Altium/Project%20Outputs%20for%20DS-DM-Rev0/SCH-DS-xDM-Rev0.PDF - DS-DM schematics
https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0/-/blob/master/Testing/Clk3_XO_125_to_fpgaIN_recoveredMGTclk_to_IN2_Si5394-RevA-Project.slabtimeproj
https://docs.xilinx.com/v/u/en-US/ds593 - Xilinx Platform Cable USB II
https://www.enclustra.com/en/products/system-on-chip-modules/mercury-xu8/ - Enclustra
https://bitbucket.org/team-ds-dm/ds-dm-software - git repository, DS-DM MIDAS frontend
https://bitbucket.org/team-ds-dm/phasemeasurement - VX Phase measurement script
https://ladd00.triumf.ca/daqinv/frontend/list/178 - inventory database

Onboard hardware

jtag chain: arm_dap_0 0x5BA00477, xczu4_1 0x04721093
Eclustra Mercury+ XU8 module: ME-XU8-4CG-1E-D11E-R2.1
- Xilinx® Zynq Ultrascale+™ MPSoC XCZU4CG-1FBVB900E
- DDR4 ECC SDRAM (PS) 2 GB
- DDR4 SDRAM (PL) 1GB
ethernet mac chip: AT24MAC402-SSHM-T ("602" chip is wrong)
USB UART for Enclustra serial console, micro-USB, 115200n8
clock chip: SI5394A-A-GM and oscillator CS-044-054.0M (54 MHz)
U23 3.3V current meter and thermometer
LEDs:
- LED_FP A/B/C/D 0/1/2/3
- led1 - 3V3_SW_ON, SOM_POWER_GOOD - Enclustra FPGA module 3.3V power is good
- led2 - LTM4624 PGOOD
- led3 - FPGA_DONE - FPGA has booted
- led4 - TP-S-1, PCLK_P
- led5 - TP-S-2, PCLK_N
LEMO connectors (top to bottom)
- J4 - input (NIM/TTL) (EXT_IN_LV(1), EXT_IN_LV(2))
- J5 - input (NIM/TTL) (EXT_IN_LV(3), EXT_IN_LV(4))
- J6 - external clock (GPS 10MHz and PPS)
- J7 - output (NIM/TTL) (EXT_OUT(1), EXT_OUT(2))
SMA connectors
- J9, J10 - CLK_CCA from U6 C.C.
- J11, J12 - CLK_TP0
RJ45 ethernet connector (100mbit: green light on, yellow light flashes when there is traffic)
SFP connector (SFP is FTLF8526P3BNL, 6 Gbit/sec, 850 nm, 300m 50/125um OM3 MMF)
4 QSFP connectors (GDM)
6 VX connectors (CDM)

Buttons, jumpers and switches

Buttons:

PB1 - HRST - reboot FPGA (power-on reset)
PB2 - SRST - (SRSTn) - reboot ARM CPU

Switches:

SW1 - boot mode BM0, BM1 [-->]
SW2 - LEMO output NIM<->TTL
SW3 - LEMO input 1 and 2 NIM/TTL
SW4 - LEMO input 2 and 4 NIM/TTL
SW5 - LEMO clock input NIM/TTL
SW6 - serial console select. [PS<--PL] PS is ARM CPU, PL is FPGA.

Front panel

| top
|
| LED-FP1 | LED_FP(0,1,2,3)
|
| SFP J???
|
| LEMO J4-LEFT, J4-RIGHT | EXT_IN_LV(1), EXT_IN_LV(2)
| LEMO J5-LEFT, J5-RIGHT | EXT_IN_LV(3), EXT_IN_LV(4)
| LEMO J6-LEFT, J6-RIGHT | CLK_EXT1, CLK_EXT0 (125 MHz only) 
| LEMO J7-LEFT, J7-RIGHT | EXT_OUT(1), EXT_OUT(2)
|
| J-VX-1
| J-VX-2 or QSFP
| J-VX-3 or QSFP
| J-VX-4 or QSFP
| J-VX-5 or QSFP
| J-VX-6
|
| RJ45 J3 ethernet
|
| bottom

VX adapter board

LVDS I/O connector

34 pin connector: 0|:::: :::: :::: :::: :|15,16 n/c

split-cable connection

0 .. 7 -> N/C

8 -> VX_RX(3)
9 -> VX_RX(2)
10 -> VX_RX(1)
11 -> VX_RX(0)

12 <- VX_TX(0)
13 <- VX_TX(1)
14 <- VX_TX(2) (set by jumper routed here or to VX CLKIN SYNC)
15 <- n/c (CLK routed to VX CLKIN CLK)

16 - n/c

one-to-one connection

0 -> VX2_RX(3)
1 -> VX2_RX(2)
2 -> VX2_RX(0)
3 -> VX2_RX(1)

4 <- CLK
5 <- VX2_TX(0)
6 <- VX2_TX(1)
7 <- VX2_TX(2)

8 -> VX1_RX(3)
9 -> VX1_RX(2)
10 -> VX1_RX(1)
11 -> VX1_RX(0)

12 <- VX1_TX(0)
13 <- VX1_TX(1)
14 <- VX1_TX(2) (set by jumper routed here or to VX CLKIN SYNC)
15 <- n/c (CLK routed to VX CLKIN CLK)

16 - n/c

Board schematics

File:SCH-DS-xDM-Rev0.PDF
note: FPGA pin annotations ("IO", "SCLK", "PCLK", etc) on the schematics are bogus, instead, trace them to the FPGA pins.
note: ENC A is J800, ENC B is J801, ENC C is J900 (schematic name to enclustra name)

FPGA MGT blocks

* SFP - ENC B45,B47 and B48,B50 - FPGA D5,D6 MGTHTX0_D and D1,D2 MGTHRX0_D
* QSFP0 TX0 - ENC C13,C17 - FPGA H5,H6 MGTHTX0_C
* QSFP0 TX1 - ENC C21,C25 - FPGA G7,G8 MGTHTX1_C
* QSFP0 TX2 - ENC C29,C23 - FPGA F5,F6 MGTHTX2_C
* QSFP1 TX0 - ENC C37,C41 - FPGA E7,E8 MGTHTX3_C
* QSFP1 TX1 - ENC C45,C47 - FPGA P5,P6 MGTHTX0_B
* QSFP1 TX2 - ENC C51,C53 - FPGA M5,M6 MGTHTX1_B
* QSFP2 TX0 - ENC C57,C59 - FPGA L3,L4 MGTHRX2_B
* QSFP2 TX1 - ENC C63,C65 - FPGA K5,K6 MGTHTX3_B
* QSFP2 TX2 - ENC C75,C77 - FPGA W3,W4 MGTHTX0_A
* QSFP3 TX0 - ENC C79,C81 - FPGA V5,V6 MGTHTX1_A
* QSFP3 TX1 - ENC C85,C87 - FPGA T5,T6 MGTHRX2_A
* QSFP3 TX2 - ENC C89,C91 - FPGA R3,R4 MGTHTX3_A

Clock distribution

Simplified:

125 MHz osc -> CLK_XO_125 -> MGTREFCLK0_A -> not used
125 MHz osc -> CLK3_XO_125 -> MGTREFCLK1_B -> SFP RX ref clock, QSFP RX and TX ref clock (this is not final design!)
(disconnected) 125 MHz osc -> CLK2_XO_125 -> FPGA AG8,AH8
125 MHz osc -> C.C. in1

C.C. in0 <- CLK_EXT1 (10 MHz GPS clock)
C.C. in1 <- 125 MHz osc
(disconnected) C.C. in2 <- CLK_CC_IN <- MGTREFCLK0_D <- SFP RX clock (cannot be used because of uncontrollable phase)
C.C. in2 <- CLK2_XO_125 <- FPGA AG8,AH8 <- SFP RX recovered 125 MHz clock
C.C. in3 <- CLK_FB

C.C. 125 MHz -> CLK_CC_OUT0 -> MGTREFCLK0_B -> QSFP RX and TX ref clock (final design)
C.C. 125 MHz -> CLK_CC_OUT1 -> MGTREFCLK1_D -> SFP TX clock
C.C. 62.5 MHz -> VX clock fanout

proposed changes:

add C.C. 125 MHz -> new CLK_CC_OUT2 (old CLK2_XO_125) -> FPGA AG8,AH8
change C.C. in2 <- new CLK_CC_IN <- FPGA AK8,AK9

Complete:

125 MHz oscillator - U5 fan out -
  q0 -> CLK_XO_125 -> ENC C72,C74 -> FPGA R7,R8 MGTREFCLK0_A (not used)
  q1 -> U6 C.C. in1
  q2 -> CLK3_XO_125 -> ENC C7,C9 -> FPGA J7,J8 MGTREFCLK1_B -> SFP RX reference clock, QSFP RX and TX reference clocks (not final design!)
  q3 -> disconnected on the board, was CLK2_XO_125 -> ENC C151,C153 -> FPGA AG8,AH8

U6 C.C (clock cleaner) -

in0 <- CLK_EXT1 (presumably GPS 10 MHz ref clock)
in1 <- 125 MHz oscillator via U5
in2 <- (was: CLK_CC_IN <- ENC B10,B12 <- FPGA D9,D10 MGTREFCLK0_D <- SFP RX clock, 125 MHz)
in2 <- CLK2_XO_125 <- ENC C151,B153 <- FPGA AG8,AH8 <- mgt_link_data_to_processing.rx_data_clk (SFP RX data clock, 125 MHz)
in3 <- CLK_FB

out0 -> CLK_CCA -> U12 (125 MHz)
out1 -> CLK_CCB -> VX1..6 (62.5 MHz)
out2 -> CLK_CCC -> VX7..12 (62.5 MHz)
out3 -> CLK_FB into in3

CLK_CCA -> U12 (125 MHz fan out) ->

Q0 -> not used
Q1 -> CLK_CC_OUT0 -> ENC C3-5 -> FPGA L7,L8 MGTREFCLK0_B -> QSFP RX and TX reference clocks (final design)
Q2 -> CLK_CC_OUT1 -> ENC B3-5 -> FPGA B9,B10 MGTREFCLK1_D -> SFP TX clock
Q3 -> not used
Q4 -> not used
Q5 -> SMA J9/J10

CLK_TP0 - FPGA N7,N8 MGTREFCLK1_A - ENC C69,C71 - DS-DM SMA J11, J12 (NOT IN CDM PROJECT)

We have 3 questions:
1) CLK_XO_125 (125 MHz osc) is not in the CDM project, why?
2) CLK_CC_OUT0 (U6 C.C. 125 MHz) is not in the CDM project, why? -> it is an MGT clock, connected now.
3) 62.5 MHz VX clock does not go into the FPGA, please confirm? -> confirmed

> And 1 comment:
>
> a) CLK_EXT0 going to in1 of U5 does not seem to be useful. only permitted frequency is 125 MHz and we normally do not distribute 125 MHz clocks on LEMO connectors and LEMO cables. (we use LVDS signals over SATA/eSATA cables and connectors). We remove it in the next revision of the board?

Proposed modifications:
- CLK_CC_IN: from FPGA output pin (ENC C142,C144 - FPGA AK8,AK9) to clock cleaner in2 (instead of CLK2_XO_125 pins)
- CLK_CCA -> U12 -> currently unused out3 -> CLK2_XO_125 FPGA pins

I2C bus

I2C_SCL is J-ENC-A pin 111 I2C_SCL is FPGA I2C_SCL_PL AB13 (IO_L1N_TOL1D_64) and I2C_SCL_PS F18 (PS_MIO10)
I2C_SDA is J-ENC-A pin 113 I2C_SDA is FPGA I2C_SDA_PL AH13 (IO_L7N_T1L1Q_AD13N_64) and I2C_SDA_PS G18 (PS_MIO11)

XU8 secure EEPROM ATSHA204A at 0x64, this is 0110'010X -> linux _011'0010 is 0x32. (but responds to scan and read at 0x33)
U4 ethernet mac chip, EEPROM at 1010 A2 A1 A0 X and MAC/serial_no at 1011 A2 A1 A0 X. A0=VCC, A1=VCC, A2=GND -> linux _101'0011 and _101'1011 is 0x53 and 0x5B.
U6 clock chip, address 1101 0 A1 A0 X. A1=VCC, A0=N/C (internal pull-up) -> linux _110'1011 is 0x6b
U23 voltmeter at 10011 ADR1 ADR0. ADR0=GND, ADR1=VCC -> linux _100'1110 is 0x4e
SFP, address 1010000X -> linux _101'0000 is 0x50. also 0x51?
QSFP0, QSFP1, QSFP2, QSFP3 (QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL)

I2C clock builder connection

use Silicon Labs USB "Clock builder pro field programmer", www.silabs.com/CBProgrammer
connect rainbow jumper cable pins:
- black - 1-GND to GND on DS-DM
- white - 3-SCLK to SCL on the DS-DM
- grey - 7-SDA_SDIO to SDA on the DS-DM
power up the DS-DM
plug USB programmer into Windows laptop
on Windows, run "ClockBuilder Pro"
it should report "Field programmer detected", press "EVB GUI"
in EVB GUI, press "Config", set I2C address 0x6B
press "Scan", it should find Si5394A-A-GM
select the "Status" tab, should see real-time status of clock chip

Clock domains

GPS

no GPS : GDM runs from internal 125 MHz oscillator
external 10 MHz clock : GDM runs from external 10 MHz clock and optional 1pps signal (use VME-NIMIO32 NIM outputs)
GPS receiver : GDM runs from GPS 10 MHz clock and GPS IRIG serial data
LNGS GPS repeater:
- provides 1pps and serial data
- serial data goes to GDM, decoded, 1pps signal extracted, goes to LEMO output
- 1pps from GDM is used to train the Rubidium clock which provides a 10 MHz clock
- 10 MHz output from Rubidium clock goes to GDM 10 MHz external clock input

GDM

AXI clock (100 MHz) - AXI registers
125 MHz oscillator - to clock cleaner
10 MHz external clock LEMO input - to clock cleaner
FPGA 125 MHz clock CLK_CC_IN - to clock cleaner - not used (would be used for cascaded GDMs)
clock cleaner output 125 MHz fanout:
- QSFP MGT RX reference clock CLK_CC_OUT0
- QSFP MGT TX reference clock CLK_CC_OUT0
clock cleaner output 125 MHz CLK_CC_OUT2: (main clock domain)
- QSFP TX data clock
- ds20k block
QSFP RX recovered clock (per link, 12 clocks)
- QSFP RX data to phase matching fifo to main clock domain to trigger logic (MGT or PL FIFO, TBD)

CDM

AXI clock (100 MHz) - AXI registers
125 MHz oscillator to clock cleaner, to SFP MGT RX reference clock pin via CLK3_XO_125
SFP RX data clock ("recovered clock") (250 MHz/8 bit, 125 MHz/16 bit, 62.5 MHz/32 bit data)
- to clock cleaner CLK_CC_IN (to VX 62.5 MHz clock and to CLK_CC_OUT0,1,2)
- SFP RX data
- SFP RX data decoder
- (SFP RX MGT clock 125 MHz not used because of uncontrollable phase)
- SFP TX reference clock MGT clock 125 MHz via clock cleaner CLK_CC_OUT1
CLK_CC_OUT0,1,2 (main clock domain)
- SFP TX data clock
- SFP TX data encoder
- trigger logic
- trigger logic to SFP TX
- trigger logic to VX TX
- VX TX data serializer
- VX TX LVDS clock and data
- inside the VX firmware: VX RX clock and data, data deserializer, phase matching fifo to VX trigger logic
CLK_XO_125 125 MHz from oscillator
- not used
CLK_CC_OUT0 125 MHz from clock cleaner
- not used
VX RX receive clock (per link, 12 clocks)
- VX RX clock and data
- VX RX data deserializer
- phase matching fifo to main clock domain trigger logic

VX

VX 62.5 MHz clock from CDM
VX 125 MHz clock
- drives Yair's trigger logic
- trigger logic to LVDS TX
- LVDS TX serializer
- LVDS TX clock and data
LVDS RX clock (125 MHz from CDM)
- LVDS RX clock and data
- LVDS RX data deserializer
- phase matching fifo to VX 125 MHz clock domain

Board test plan

To test:

Enclustra FPGA board
SFP port - RX data (ok), TX data (untested), LOS (untested), i2c (ok), SFP_RS0 connected to QSFP0_SEL is wrong? our Finisar SFP says RS0, RS1 N/C, so probably okey.
GDM QSFP ports (lanes 0,1,2. lane 3 n/c). RX data, TX data, i2c
SW5 CLK_EXT1 NIM works. TTL needs to be tested. flipping SW5 CLK_EXT0 side from NIM to TTL makes CLK_EXT1 go LOS and OOF in the clock chip. R23 and R53 should be removed?
BOOT_MODE 0 and 1
I2C bus. tested all devices except for QSFP.

Partial:

U23 3.3V current meter and thermometer. V1,V2 is current monitor, same as in application note. V3,V4 is thermometer, same in application note, except capacitor C118 is 0.1u instead of 470pF. Tested ok: Tint, VCC, V1, V2, V1-V2. Test failed: TR2 reads 50-something degC instead of same as Tint.

Done:

LED_FP1A..D: tested ok. K.O. 15 sep 2022
USB UART: tested ok. K.O. 15 sep 2022
J4A, J4B, J5A, J5B LEMO inputs (NIM/TTL) EXT_IN_LV(1..4). TTL threshold 1.7V, NIM threshold -0.3V. 50 Ohm termination. TTL straight, NIM inverted.
J6A, J6B LEMO clock inputs: 10 MHz TTL from chronobox works, 10 MHz NIM from IO32 works. 18-apr-2023
J7A, J7B LEMO outputs EXT_OUT(1), EXT_OUT(2) (NIM/TTL) (tested 23nov2022, K.O.). TTL ???, NIM inverted.
- TTL out no 50 ohm termination: 0=0V, 1=5V, rise and fall time ~5 ns
- TTL out with 50 ohm termination: 0=0V, 1=2.5V, rise and fall time <2ns
- NIM out no 50 ohm termination: 0=+50mV, 1=-1.8V, rise and fall time ~3ns
- NIM out with 50 ohm termination: 0=0V, 1=-0.9V, rise and fall time <2ns
ethernet MAC i2c chip (K.O. 20sep2022: can read: i2cdump 0 0x5b, nov2022 have u-boot driver)
CDM VX ports 2x(CLK, 3 tx, 4 rx) tested using test_cdm.exe and LVDS loopback in VX firmware.

Failure:

ethernet: does not connect to alliedtelesys switch. connects to my USB-eth adapter at 100 Mbit speed. uboot mii status reports connection speed oscillating between 1000, 100 and 10. K.O. 16-sep-2022

Checklist for newly build boards

put new board on workbench
check - vme connector present, vme extraction handles present
check - standoff are removed from all thru-holes
plug Enclustra module
check - SW6 both switches are in the "PS" position
connect micro-usb cable from linux PC
connect ethernet from 1gige capable network switch
connect power from lab power supply - GND, +5V and -12V nominal, +5.7V and -12.5V actual
power up, +5V current 2.10-2.8A, -12V current 0.05A
on linux PC, open a new terminal, run: minicom -D /dev/ttyACM0
in minicom window, observe messages about Xilinx first stage boot loader, etc
on the ethernet switch, observe network link is 1gige speed (not 10mbit, not 100mbit).
if everything boots okey, there will be a login prompt, login as root, password root.
busybox devmem 0x80010000 # read firmware version number, i.e. 0xEDAD0A77
TBW - test LEMO inputs
TBW - test LEMO outputs
TBW - test VX connectors
TBW - test SFP connector
TBW - test QSFP connector

Serial console

check that linux computer has correct udev rules to allow access to /dev/ttyACM devices, see https://daq00.triumf.ca/DaqWiki/index.php/SLinstall#Configure_USB_device_permissions and https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Configure_USB_device_permissions
connect micro-USB cable to connector J-UCB, other end connect to linux computer
observe /dev/ttyACM0 was created
run "minicom -D /dev/ttyACM0" (default serial settings are ok, otherwise, 115200n8)
should have gdm-cdm login
username root, password root

i2c

ZynqMP> i2c bus
Bus 0:  i2c@ff020000
ZynqMP> i2c dev 0
Setting bus to 0
ZynqMP> i2c probe  
Valid chip addresses: 33 4E 53 5B 6B 77
ZynqMP> i2c md 0x5b 0x98
0098: fc c2 3d 00 00 12 1a 6e 0a 90 85 04 94 10 08 50    ..=....n.......P

root@gdm-cdm:~# i2cdetect 0
Warning: Can't use SMBus Quick Write command, will skip some addresses
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0.
I will probe address range 0x03-0x77.
Continue? [Y/n] 
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                                                 
10:                                                 
20:                                                 
30: -- -- -- 33 -- -- -- --                         
40:                                                 
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- -- 
60:                                                 
70:                                                 
root@gdm-cdm:~# 
root@gdm-cdm:~# i2cdump 0 0x5b
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0, address 0x5b, mode byte
Continue? [Y/n] 
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
80: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00    ???????P?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e    ........??=..??n
a0: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00    ???????P?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e    ........??=..??n
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
root@gdm-cdm:~#

root@gdm0:~# i2cdetect -r 0
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e -- 
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- -- 
70: -- -- -- -- -- -- -- --                         
root@gdm0:~#

root@cdm0:~# i2cdetect -r 0
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e -- 
50: 50 51 -- 53 -- -- -- -- -- -- -- 5b -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- -- 
70: -- -- -- -- -- -- -- --                         
root@cdm0:~#

0x33 - XU8 secure EEPROM (should be at 0x32)
0x4e - U23 current and temperature monitor
0x50, 0x51 - SFP
0x53, 0x5b - ethernet mac eeprom
0x6b - U6 clock chip

U23

internal temperature only

root@gdm0:~# i2cset -y 0 0x4e 0x01 0x00 b  # control register: "repeat mode, internal temperature only"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x03 # "Tint ready" and "busy", "busy is always 1 in repeat mode"
root@gdm0:~# i2cdump -y 0 0x4e b
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 03 00 03 03 81 db 2a ce 2a 8e 00 6f 00 45 20 3f    ?.????*?*?.o.E ?
10: 01 00 01 01 01 db 2a ce 2a 8e 00 6f 00 45 20 3f    ?.????*?*?.o.E ?
...
readback:
reg0 - 03 - Tint ready
reg1 - 00 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB 0x81, bit 0x80 is "DV, data valid", bit 0x40 is "SS, sensor short", 0x20 is "SO, sensor open"
reg5 - Tint LSB 0xDB, Tint = 0x01DB = 475 * 0.0625 degC = 29.6 degC
reg6..F - stale data

Tint, V1, V2, TR2, VCC

root@gdm0:~# i2cset -y 0 0x4e 0x01 0x18 b  # control register: "repeat mode, V1, V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 7f 18 7f 7f 81 b9 aa d0 aa 8f 83 5e 83 5e a0 41    ???????????^?^?A
10: 01 18 01 01 01 b9 2a d0 2a 8f 03 5e 03 5e 20 41    ??????*?*??^?^ A
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x1b9*0.0625 = 27.5 degC
reg6 - V1 MSB 0xaa, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 LSB 0xd0, V1 = 0x2ad0*305.18/1000000 = 3.3447 V (correct)
reg8 - V2 MSB 0xaa, ditto
reg9 - V2 LSB 0x8f, V2 = 0x2a8f*305.18/1000000 = 3.3249 V (correct, smaller than V1)
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x5e. TR2 = 0x35e*0.0625 = 53.875 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x41, VCC = 2.5+0x2041*305.18/1000000 = 5.019 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.99 A (about right?)

Tint, V1-V2, TR2, VCC

root@gdm0:~# i2cset -y 0 0x4e 0x01 0x19 b  # control register: "repeat mode, V1-V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 7f 19 7f 7f 81 69 83 f2 83 f2 83 3a 83 3a a0 44    ?????i?????:?:?D
10: 01 19 01 01 01 69 03 f2 03 f2 03 3a 03 3a 20 44    ?????i?????:?: D
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x169*0.0625 = 22.5 degC
reg6 - V1 or V1-V2 MSB 0x83, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 or V1-V2 LSB 0xf2, V1-V2 = 0x3f2*19.42/1000000 = 0.0196142 V (correct, compare with V1 and V2 measured above)
reg8 - V2 or V1-V2 MSB
reg9 - V2 or V1-V2 LSB
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x3a. TR2 = 0x33a*0.0625 = 51.625 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x44, VCC = 2.5+0x2044*305.18/1000000 = 5.021 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.9807 A (about right?)

SFP

root@cdm0:~# i2cdump 0 0x50
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 03 04 07 00 00 00 00 40 40 0c 00 01 3d 00 00 00    ???....@@?.?=...
10: 0c 02 00 1e 46 49 4e 49 53 41 52 20 43 4f 52 50    ??.?FINISAR CORP
20: 2e 20 20 20 00 00 90 65 46 54 4c 46 38 35 32 36    .   ..?eFTLF8526
30: 50 33 42 4e 4c 20 20 20 41 20 20 20 03 52 00 9d    P3BNL   A   ?R.?
40: 00 1a 00 00 4e 33 41 42 34 4c 56 20 20 20 20 20    .?..N3AB4LV     
50: 20 20 20 20 32 30 30 33 31 39 20 20 68 f0 03 de        200319  h???
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
root@cdm0:~# 
root@cdm0:~# i2cdump 0 0x51
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 5a 00 d8 00 55 00 e2 00 90 88 71 48 8c a0 75 30    Z.?.U.?.??qH??u0
10: 21 34 01 f4 1b 58 03 e8 31 2d 04 eb 1f 07 06 31    !4???X??1-?????1
20: 31 2d 00 64 27 10 00 9e 00 00 00 00 00 00 00 00    1-.d'?.?........
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
40: 00 00 00 00 3f 80 00 00 00 00 00 00 01 00 00 00    ....??......?...
50: 01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 b7    ?...?...?......?
60: 19 9f 80 c5 0e 17 12 c4 1f 99 00 00 00 00 30 00    ??????????....0.
70: 00 00 00 00 00 00 00 00 ff ff ff ff ff ff ff 01    ...............?
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
root@cdm0:~#

ethernet mac eeprom

correct chip with 84-bit ethernet mac address

root@cdm1:~# i2cdump 0 0x53
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
root@cdm1:~# 

root@cdm1:~# i2cdump 0 0x5b
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
80: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00    ????4??2?K?.?...
90: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c    ..........??=?Q<
a0: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00    ????4??2?K?.?...
b0: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c    ..........??=?Q<
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
root@cdm1:~#

wrong "602" chip with 64-bit IPv6 address

root@cdm0:~# i2cdump 0 0x53
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
root@cdm0:~# i2cdump 0 0x5b
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
80: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00    ???????Q?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e    ........??=..??.
a0: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00    ???????Q?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e    ........??=..??.
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
root@cdm0:~#

read ethernet mac address from i2c

(this code is copied from uboot command line i2c code)

in uboot sources board/xilinx/common/board.c replace original function with this:

// special code to read ethernet MAC address from the DS-DM-Rev0 board. K.O. Sep-2022

int zynq_board_read_rom_ethaddr(unsigned char *ethaddr)
{
        struct udevice *bus;
	int ret;
        int busnum = 0;

	ret = uclass_get_device_by_seq(UCLASS_I2C, busnum, &bus);
	if (ret) {
           printf("%s: No bus %d\n", __func__, busnum);
           return ret;
	}

        int chip_addr = 0x5B;

        struct udevice *dev;

        ret = i2c_get_chip(bus, chip_addr, 1, &dev);
	if (ret) {
           printf("%s: Bus %d no chip 0x%02x\n", __func__, busnum, chip_addr);
           return ret;
	}

        int dev_addr = 0x98;

        unsigned char data[8];
           
        ret = dm_i2c_read(dev, dev_addr, data, 8);

	if (ret) {
           printf("%s: Bus %d chip 0x%02x read error %d\n", __func__, busnum, chip_addr, ret);
           return ret;
	}

        printf("%s: Bus %d chip 0x%02x addr 0x%02x read: 0x %02x %02x %02x %02x %02x %02x %02x %02x\n", __func__, busnum, chip_addr, dev_addr, data[0], data[1], data[2], data[3], data[4], data[5], data[6], data[7]);

        // see Atmel-8807-SEEPROM-AT24MAC402-602-Datasheet.pdf

        if (data[0] == 0) {
           // eiu-48 chip
           ethaddr[0] = data[2];
           ethaddr[1] = data[3];
           ethaddr[2] = data[4];
           ethaddr[3] = data[5];
           ethaddr[4] = data[6];
           ethaddr[5] = data[7];
        } else {
           // eiu-64 chip
           ethaddr[0] = data[0];
           ethaddr[1] = data[1];
           ethaddr[2] = data[2];
           ethaddr[3] = data[5];
           ethaddr[4] = data[6];
           ethaddr[5] = data[7];
        }

        printf("%s: ethaddr %02x:%02x:%02x:%02x:%02x:%02x\n", __func__, ethaddr[0], ethaddr[1], ethaddr[2], ethaddr[3], ethaddr[4], ethaddr[5]);

        return ret;
}

also this should have worked if i2c_xxx() functions were enabled in uboot:

i2c_set_bus_num(0);
i2c_probe(0x5b);
i2c_read(0x5b, 0x9a, ethaddr, 6);

read ethernet mac address from i2c (SHOULD WORK)

from: https://stackoverflow.com/questions/43637540/is-there-linux-or-u-boot-support-to-read-a-mac-address-from-a-chip-at-startup

#ethernet related setup
setup_eth=run readmac buildmac
#read mac address from eeprom
readmac=i2c dev 0; i2c read 50 FA.1 6 $loadaddr
#build the ethaddr variable
#not very nice, but does the job
buildmac=\
e=" "; sep=" " \
for i in 0 1 2 3 4 5 ; do\
setexpr x $loadaddr + $i\
setexpr.b b *$x\
e="$e$sep$b"\
sep=":"\
done &&\
setenv ethaddr $e

read ethernet mac address from i2c (DOES NOT WORK)

this method does not work: inside board/xilinx/common/board.c:int zynq_board_read_rom_ethaddr(unsigned char *ethaddr), uclass_get_device_by_ofnode(UCLASS_I2C_EEPROM, eeprom, &dev) returns ENODEV and read from i2c does not happen. K.O. Sep-2022

Read:

https://support.xilinx.com/s/question/0D52E00006hpKlsSAE/uboot-and-ethernet-mac-address-from-eeprom?language=en_US (last answer)
https://github.com/Xilinx/u-boot-xlnx/blob/master/board/xilinx/common/board.c (zynq_board_read_rom_ethaddr())

Note:

0x5B is the i2c chip address
0x9A is the data offset inside the chip, see datasheet or i2c read dump above.

Edit:

emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/u-boot/files/platform-top.h

#include <configs/xilinx_zynqmp.h>
#include <configs/platform-auto.h>
//#define CONFIG_I2C_EEPROM                                                                                                                                                                   
//#define CONFIG_SYS_I2C_EEPROM_ADDR 0x5b                                                                                                                                                     
//#define CONFIG_SYS_I2C_EEPROM_ADDR_OVERFLOW 0x0                                                                                                                                             
#define CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET 0x9A
#error HERE!

emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi

/include/ "system-conf.dtsi"
/ {
chosen {
   xlnx,eeprom = &eeprom;
};
};

&i2c0 {
eeprom: eeprom@5b { /* u88 */
compatible = "atmel,24mac402";
reg = <0x5b>;
};
};

components/yocto/workspace/sources/u-boot-xlnx/configs/xilinx_zynqmp_virt_defconfig

CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET=0x9A

enable VX clock

devmem 0x80010010 32 0x8; sleep 1;  devmem 0x80010010 32 0x0;
si5394-i2c-file /media/sd-mmcblk1p1/00_freerun.txt 0 0x6b

clock chip configuration

file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt:

VCO is 14 GHz
Tvco is 71.43 ps
N0 divider is 14, frequency is 1000 MHz
out0 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz
out1 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
out2 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
out3 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz is the feedback for zero delay

arm and linux benchmark

memory benchmark:

daq13$ arm-linux-gnueabi-gcc -o memcpy.armv7 memcpy.cc -march=armv7 -static -O2
scp memcpy.armv7 to ...
root@gdm-cdm:~# ./memcpy.armv7 
memcpy       1 KiBytes:   1288 MB/sec
memcpy       2 KiBytes:   1924 MB/sec
memcpy       4 KiBytes:   2554 MB/sec
memcpy       8 KiBytes:   3054 MB/sec
memcpy      16 KiBytes:   3262 MB/sec
memcpy      32 KiBytes:   3250 MB/sec
memcpy      64 KiBytes:   3456 MB/sec
memcpy     128 KiBytes:   3556 MB/sec
memcpy     256 KiBytes:   3780 MB/sec
memcpy     512 KiBytes:   3795 MB/sec
memcpy    1024 KiBytes:   3789 MB/sec
memcpy    2048 KiBytes:   3729 MB/sec
memcpy    4096 KiBytes:   3717 MB/sec
memcpy    8192 KiBytes:   3687 MB/sec
memcpy   16384 KiBytes:   3632 MB/sec
memcpy   32768 KiBytes:   3529 MB/sec
memcpy   65536 KiBytes:   3318 MB/sec
memcpy  131072 KiBytes:   2893 MB/sec
root@gdm-cdm:~#

ethernet receive:

daq13:bin$ ./ttcp -t -s -n 100000 10.0.0.24
ttcp-t: buflen=8192, nbuf=100000, align=16384/0, port=5001  tcp  -> 10.0.0.24
ttcp-t: socket
ttcp-t: connect
ttcp-t: 819200000 bytes in 7.25 real seconds = 110358.39 KB/sec +++
ttcp-t: 100000 I/O calls, msec/call = 0.07, calls/sec = 13794.80
ttcp-t: 0.0user 0.2sys 0:07real 3% 0i+0d 760maxrss 0+2pf 1461+31csw
daq13:bin$ 

root@gdm-cdm:~# ./ttcp.armv7 -r -s
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001  tcp
ttcp-r: socket
ttcp-r: accept from 10.0.0.25
ttcp-r: 819200000 bytes in 7.27 real seconds = 110098.22 KB/sec +++
ttcp-r: 212040 I/O calls, msec/call = 0.04, calls/sec = 29181.53
ttcp-r: 0.1user 5.7sys 0:07real 81% 0i+0d 584maxrss 0+2pf 125601+2699csw
root@gdm-cdm:~#

ethernet transmit:

root@gdm-cdm:~# ./ttcp.armv7 -t -s -n 100000 10.0.0.25
ttcp-t: buflen=8192, nbuf=100000, align=16384/0, port=5001  tcp  -> 10.0.0.25
ttcp-t: socket
ttcp-t: connect
ttcp-t: 819200000 bytes in 6.95 real seconds = 115078.69 KB/sec +++
ttcp-t: 100000 I/O calls, msec/call = 0.07, calls/sec = 14384.84
ttcp-t: 0.0user 0.7sys 0:06real 11% 0i+0d 584maxrss 0+2pf 1162+1017csw
root@gdm-cdm:~# 

daq13:bin$ ./ttcp -r -s
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001  tcp
ttcp-r: socket
ttcp-r: accept from 10.0.0.24
ttcp-r: 819200000 bytes in 6.97 real seconds = 114841.84 KB/sec +++
ttcp-r: 161335 I/O calls, msec/call = 0.04, calls/sec = 23160.01
ttcp-r: 0.0user 1.9sys 0:06real 28% 0i+0d 760maxrss 0+2pf 80646+51csw
daq13:bin$

Install Xilinx tools

install Vivado 2020.2

login at https://www.xilinx.com/myprofile.html
go to "Downloads"
go to archive,
find 2020.2
download Xilinx_Unified_2020.2_1118_1232_Lin64.bin
sh ./Xilinx_Unified_2020.2_1118_1232_Lin64.bin
banner window should open with spinner "downloading installation data"
"a newer version is available" -> say "continue"
next
"select install type" window:
provide email and password,
select "download image"
select directory /home/olchansk/Xilinx/Downloads/2020.2\
select "linux" and "full image"
next
download summary: space required 38.52 Gbytes
download
installation progress
downloading spinner, 16 M/s 47 minutes...
"download image has been created successfully". Ok.
check contents of /home/olchansk/Xilinx/Downloads/2020.2
ls -l /home/olchansk/Xilinx/Downloads/2020.2
total 67
drwxr-xr-x 2 olchansk users    9 Sep  1 16:22 bin
drwxr-xr-x 3 olchansk users   15 Sep  1 16:23 data
drwxr-xr-x 4 olchansk users    4 Sep  1 16:22 lib
drwxr-xr-x 2 olchansk users  644 Sep  1 16:22 payload
drwxr-xr-x 2 olchansk users    7 Sep  1 16:22 scripts
drwxr-xr-x 4 olchansk users    4 Sep  1 16:22 tps
-rwxr-xr-x 1 olchansk users 3256 Nov 18  2020 xsetup
daq13:2020.2$ 
./xsetup
spinned loading installation data
xilinx design tools 2022.1 now available -> say continue
"welcome" -> next
"select product" -> vivado -> next -> vivado hl system edition -> next
select devices: only zynq ultrascale+ mpsoc -> next
select destination: /opt/Xilinx (as root, mkdir /opt/Xilinx, chmod olchansk.users /opt/Xilinx)
install ...
complete
move /home/olchansk/Xilinx/Downloads/2020.2 to /daq/daqstore/olchansk/Xilinx/Downloads/

install petalinux 2020.2

./xsetup
"a newer version is available" -> say "continue"
next
"select product to install" -> select Petalinux (Linux only) -> next
"select destination directory" -> select "/opt/Xilinx" (disk space required 2.64 GB) -> next
"summary" -> install ...
error about missing /tmp/tmp-something files
"installation completed successfully" (hard to dismiss, "ok" button is partially cut-off)
done?
I think it failed, /opt/Xilinx/PetaLinux/2020.2/bin is empty except for petalinux-v2020.2-final-installer.run
try to run it by hand, same error about /tmp/tmp-something files. strange...
notice it complains about "truncate", which truncate finds ~/bin/truncate, get rid of it,
try again
now complains about missing texinfo and zlib1g:i386
apt install texinfo -> ok
apt install zlib1g:i386 -> installs bunch of gcc stuff -> ok
try again
reports "already installed" -> delete /opt/Xilinx/.xinstall/PetaLinux_2020.2/, delete entries in ~/.Xilinx/registry/installedSW.xml
try again
success

install vivado 2022.1 and petalinux 2022.1 - everything is pretty much the same

Petalinux

cd PetaLinux_GDM_CDM
petalinux-config
enable i2c MAC address and DHCP

git clone https://bitbucket.org:/team-ds-dm/ds-dm-u-boot-xlnx.git
cd ds-dm-u-boot-xlnx
git checkout ds-dm-u-boot-xlnx

linux-components ->
uboot -> ext-local-src
external u-boot local source -> ds-dm-u-boot-xlnx (path to the customized uboot git repository)

enable DHCP

Subsystem AUTO Hardware Settings -> Ethernet Settings
randomize MAC address -> NO
ethernet mac address -> leave empty
obtain ip address automatically -> YES

set hostname and product names

Firmware Version Configuration ->
Host name -> "ds-dm"
Product name -> "Petalinux_GDM_CDM"

configure linux kernel

petalinux-config -c kernel

enable NFS-Root

petalinux-config
Image Packaging Configuration > Root File System Type -> set to NFS
Location of NFS root directory set to "/nfsroot"

petalinux-config -c kernel
Networking support > IP: kernel level configuration
enable DHCP, BOOTP, RARP
File systems > Network file systems > Root file systems on NFS

manually fix linux kernel command line:

grep nfsroot PetaLinux_GDM_CDM/project-spec/configs/config
edit CONFIG_SUBSYSTEM_BOOTARGS_GENERATED to read
earlycon console=ttyPS0,115200 clk_ignore_unused panic=60 root=/dev/nfs nfsroot=/nfsroot/%s ip=dhcp rw

check configuration in
- PetaLinux_GDM_CDM/project-spec/configs/config
- PetaLinux_GDM_CDM/project-spec/configs/rootfs_config
- PetaLinux_GDM_CDM/components/plnx_workspace/device-tree/device-tree/system-conf.dtsi

JTAG server

localhost:3121

Firmware registers

Register map

  0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
  1 | ALL | ALL | RW | read write scratch register
  2 | ALL | CDM | ?? | MGT not used
  3 | ALL | CDM | RO | MGT debug_data
  4 | ALL | CDM | RW | clk_config_vec
  5 | ALL | CDM | ?? | not used
  6 | ALL | CDM | RO | CDM_link_data_processing:o_error_time
  7 | ALL | CDM | RO | CDM_link_data_processing:o_error_count

255 | ALL | ALL | RW | control register

register 0 0x80010000

GDM:

0 - gdm_link_interface:i_mgt_rst
2 - gdm_link_interface:i_link_down_latched_rst
8 - GDM_link_data_processing:i_rst
10..9 - GDM_link_data_processing:i_data_mode

CDM:

0 - cdm_link_interface:i_mgt_rst
2 - cdm_link_interface:i_link_down_latched_rst
8 - CDM_link_data_processing:i_rst
10..9 - CDM_link_data_processing:i_data_mode

register 1 0x80010004

GDM:

nlinks-1..0 - gdm_link_interface:i_rx_slide_trigger

CDM:

nlinks-1..0 - cdm_link_interface:i_rx_slide_trigger

register 2 0x80010008

GDM:

nlinks-1..0 - gdm_link_interface:o_link_power_good
nlinks+15..16 - gdm_link_interface:o_link_status

CDM:

nlinks-1..0 - cdm_link_interface:o_link_power_good
nlinks+15..16 - cdm_link_interface:o_link_status

register 3 0x8001000c

GDM: simple loopback register

CDM:

31..0 - debug_data - cdm_link_interface:o_debug

o_debug:

rx_link_rst & rx_error & rx_link_up & rx_receiving_data &
std_logic_vector(rx_state_count) & tx_state_count_on_rx_clk & i_rx_ctrl3(0) &
i_rx_ctrl1(1 downto 0) & i_rx_ctrl0(1 downto 0) &
rx_data_is_k28p1_k28p5 &
i_rx_data;

register 4 0x80010010

GDM write:

0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS

GDM read:

0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS
4 - clk_config_vec(4) - CLK_LOSXTn_LS
5 - clk_config_vec(5) - CLK_LOLn_LS
6 - clk_config_vec(6) - CLK_INTn_LS
7 - constant 1
31..8 - constant 0

register 5 0x80010014

not used

register 6 0x80010018

GDM:

3..0 - GDM_link_data_processing:i_status_select

CDM:

31..0 - CDM_link_data_processing:o_error_count

register 7 0x8001001c

GDM:

31..0 - GDM_link_data_processing:o_status_vector

CDM:

31..0 - CDM_link_data_processing:o_error_count

register 246 0x800103d8

VX_RX

register 247 0x800103dc

VX_RX, LEMO_IN, FP, LEMO_OUT, VX_TX

control register 255 0x800103fc

31..28 | ALL | function mux, 4 bits: 0 - outputs driven by control register, 1 - dsvslice GDM production, 2 - dsvslice CDM production, 3, 4, 5 - LVDS loopback, see source code
27 | ALL | lemo_in(4) invert
26 | ALL | lemo_in(3) invert
25 | ALL | lemo_in(2) invert
24 | ALL | lemo_in(1) invert
23 | ALL | lemo_in(4) enable
22 | ALL | lemo_in(3) enable
21 | ALL | lemo_in(2) enable
20 | ALL | lemo_in(1) enable
19 | ALL | software_trigger
18 | ALL | lemo_trigger_enable
17 | ALL | pulser_trigger_enable
16 | ALL | trigger_enable
15 | ALL | gdm_trigger_enable
14
13 | ALL | drive ext_out(2) LEMO output
12 | ALL | drive ext_out(1) LEMO output
11 | ALL | fp_led_out(3) front panel LEDs
10 | ALL | fp_led_out(2)
 9 | ALL | fp_led_out(1)
 8 | ALL | fp_led_out(0)
 7 | ALL | not used
 6 | ALL | vx2_tx_out(2)
 5 | ALL | vx2_tx_out(1)
 4 | ALL | vx2_tx_out(0)
 3 | ALL | not used
 2 | ALL | vx1_tx_out(2)
 1 | ALL | vx1_tx_out(1)
 0 | ALL | vx1_tx_out(0)

Power up sequence

on the CDM:

busybox devmem 0x80010010 32 0x8
sleep 1;
busybox devmem 0x80010010 32 0x0
#/home/dsdaq/si5394-i2c-file /home/dsdaq/CDM_v2.0_IN1_fixed_and_IN2_RX_Recovered_Si5394-RevA-Registers.txt  0 0x6b
/home/dsdaq/si5394-i2c-file /home/dsdaq/CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b
busybox devmem 0x80010004 32 0x1;  # link startup mode reset until phase-aligned
busybox devmem 0x80010000 32 0x5;  # reset link and link status registers
sleep 1;   # just holds reset high
busybox devmem 0x80010000 32 0x0; # lower reset and let links lock
sleep 2;   # give links time to lock
busybox devmem 0x80010008 32;  # read status 0x00000003 means locked and phase aligned
busybox devmem 0x8001000c 32   # read data, 0x1B1C 3CBC lower 16 bits are K-Codes K.28.1 (3C) and K.28.5 (BC) when locked and phase aligned

Build firmware

Build from git clone

git clone git@edev-group.triumf.ca:fw/exp/darkside/gcdm.git
#Makefile change VIVADO_SETTINGS_SCRIPT := /opt/Xilinx/Vivado/2022.1/settings64.sh
#. /opt/Xilinx/Vivado/2022.1/settings64.sh
. /opt/Xilinx/Vivado/2020.2/settings64.sh
make clean
make all_from_scratch
. /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
make petalinux_create
make petalinux_rebuild_new_hw_des
bomb out: The TMPDIR: /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp can't be located on nfs.
mkdir /tmp/build_tmp
rm -rf /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp/
ln -s /tmp/build_tmp /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp
try again
grinds, loads a whole bunch of packages...
finishes with desire to copy things to /tftpboot
make sdcard_cp_to wants to copy files from PetaLinux_GDM_CDM/images/linux/ to SD card

Build branch develop_ko

. /opt/Xilinx/Vivado/2020.2/settings64.sh
. /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
make clean_gdm   # remove gdm build tree
make gdm         # build or rebuild GDM
make copy_gdm.   # copy to gdm0
make clean_cdm   # remove cdm build tree
make cdm         # build or rebuild CDM
make copy_cdm    # copy to cdm0 and cdm1

Build in a fresh account

ssh dsdmdev@daq13
mkdir git
cd git
git clone /home/olchansk/git/ds-dm-gcdm
cd ds-dm-gcdm
. /opt/Xilinx/Vivado/2020.2/settings64.sh
. /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
/usr/bin/time make all_from_scratch BOARD=CDM
(sdcard_cp_to bombs out! FIXED!)
#/usr/bin/time make petalinux_build BOARD=CDM
#/usr/bin/time make petalinux_repackage BOARD=CDM

clean GDM:

rm -rf Vivado_GDM_XU8
/usr/bin/time make vivado_create_and_build BOARD=GDM
/usr/bin/time make petalinux_repackage BOARD=GDM

rebuild GDM:

/usr/bin/time make vivado_rebuild_fw_hw_des BOARD=GDM
/usr/bin/time make petalinux_repackage BOARD=GDM

copy to running GDM or CDM:

scp PetaLinux_GDM_CDM/images/linux/BOOT.BIN dsdaq@dsvslice:
#ssh dsdaq@dsvslice
#scp BOOT.BIN root@gdm0:/media/sd-mmcblk1p1/BOOT.BIN
#ssh root@gdm0 -> shutdown -r now
ssh dsdaq@dsvslice scp BOOT.BIN root@xdmn:/media/BOOT/BOOT.BIN
ssh dsdaq@dsvslice ssh root@xdmn shutdown -r now

copy to SD card:

open a root shell
format 16 GB Sd card per above
cd /home/dsdmdev/git/ds-dm-gcdm
make copy
(cannot confirm it worked, DS-DM CDM sn 6 serial console is broken)

prepare bootable sd card

format the sd card

this only needs to be done once

become root
cd ~olchansk/git/ds-dm-gcdm
use "lsblk" to identify the SD card (should show as 8/16/32 GB block device)/ /dev/sdd in this case
make sdcard_format SDCARD_DEVICE=/dev/sdd
disconnect sd card, reconnect the sd card (to detect new partition tables, etc)

copy CDM boot files

cd /home/dsdmdev/git/ds-dm-gcdm
make copy

copy boot files to the sd card

as root: identify partition labels, run "blkid", should say "BOOT", "rootfs" and "data"
mount

mkdir /media/olchansk/BOOT
mkdir /media/olchansk/rootfs
mkdir /media/olchansk/data
mount -L BOOT /media/olchansk/BOOT
mount -L rootfs /media/olchansk/rootfs
mount -L data /media/olchansk/data
cp PetaLinux_GDM_CDM/images/linux/BOOT.BIN /media/olchansk/BOOT/
cp PetaLinux_GDM_CDM/images/linux/boot.scr /media/olchansk/BOOT/
cp PetaLinux_GDM_CDM/images/linux/image.ub /media/olchansk/BOOT/
umount /media/olchansk/BOOT
umount /media/olchansk/rootfs
umount /media/olchansk/data
eject /dev/sdd

boot messages

Xilinx Zynq MP First Stage Boot Loader 
Release 2020.2   Sep 24 2022  -  13:29:15
NOTICE:  ATF running on XCZU4CG/silicon v4/RTL5.1 at 0xfffea000
NOTICE:  BL31: v2.2(release):xlnx_rebase_v2.2_2020.3
NOTICE:  BL31: Built : 18:02:46, Sep 28 2022


U-Boot 2020.01 (Sep 28 2022 - 18:03:39 +0000)

Model: DarkSide 20k DM
Board: Xilinx ZynqMP
DRAM:  2 GiB
usb dr_mode not found
PMUFW:  v1.1
EL Level:       EL2
Chip ID:        zu4
NAND:  0 MiB
MMC:   mmc@ff160000: 0, mmc@ff170000: 1
In:    serial@ff000000
Out:   serial@ff000000
Err:   serial@ff000000
Bootmode: SD_MODE1
Reset reason:   SOFT 
Net:   
ZYNQ GEM: ff0b0000, mdio bus ff0b0000, phyaddr -1, interface rgmii-id

Warning: ethernet@ff0b0000 (eth0) using random MAC address - d6:62:5f:13:00:44
eth0: ethernet@ff0b0000
ZYNQ GEM: ff0e0000, mdio bus ff0e0000, phyaddr -1, interface rgmii-id
Could not get PHY for eth1: addr -1

Hit any key to stop autoboot:  0 
ZynqMP> 
CTRL-A Z for help | 115200 8N1 | NOR | Minicom 2.7.1 | VT102 | Online 122:2 | ttyACM0

Software

ssh cdm0 # or gdm0
git clone https://bitbucket.org/team-ds-dm/ds-dm-software
cd ds-dm-software
make

dsvslice integration

VX setup

general
- Start acq from user code = y, all others = n (as of Mar 2023: will start when first trigger received)
- Use NIM IO = y
- Use external clock = y
- LVDS quartet is input = n, y, n, y
- LVDS quartet mode = User, User, User, User

trigger from front panel NIM:
- Trigger on external signal = y, all others = n
- connect CDM EXT_OUT(2) to VX "TrigIn"

trigger from LVDS "Sync" mode
- Trigger on LVDS Sync signal = y, all others = n
- LVDS quartet mode = User, Sync, User, User

trigger from LVDS "User" mode
- Trigger on LVDS pair 12 signal = y, all others = n
- LVDS quartet is input = n, y, n, y
- LVDS quartet mode = User, User, User, User

GDM setup

GDM is gdm0
set inputs to NIM mode
set outputs to TTL mode (this GDM has wrong NIM output circuit)
use top QSFP slot, connect split cable 1 into CDM01, cable 2 into CDM02
connect non-inverted NIM trigger signal to top-LEMO-left EXT_IN_LV(1)
connect non-inverted NIM TSM signal to top-LEME-right EXT_IN_LV(2)
GDM LEDs: TRIG, TSM, trigger enabled, trigger_out
GDM LEMO_OUT: trigger, trigger

CDM setup

set CDM LEMO inputs to NIM
set CDM LEMO outputs to NIM
CDM01 is cdm0
CDM02 is cdm1
connect GDM fiber links to SFP port
connect 1st VX port of CDM01 to VX1
connect 1st VX port of CDM02 to VX2
connect LEMO EXT_OUT(2) to VX "TrigIn", CDM01 to VX1, CDM02 to VX2
power up
CDM LEDs: GDM TRIG, GDM TSM, trigger enabled, trigger out
CDM LEMO_OUT: gdm_trg, trigger

After power up

start the CDM frontend from the MIDAS "Programs" page. To start manually, see the Start Command on the Programs page.
CDM frontend should enable the VX clock, disable the trigger
from the MIDAS status page, goto the CDM page
in the CDM01, CDM02 data table, the 2nd number should read 0x35c08008, if it does not and the last 4 digits randomly change, reset the GDM links:
- in order GDM, CDM01, CDM02, press "reset1" of each board, then press "reset4", then press "reset0", 2nd number should read as above. if it does not, STOP HERE.
start a run
CDM frontend will enable the trigger
GDM frontend will enable the trigger
LEDs on the GDM should flash, LEDs on the CDM should flash, TrigIn and TrigOut of the VX should flash
stop a run
GDM frontend will disable the trigger
CDM frontend will disable the trigger

Phase measurement

pip3 install matplotlib
pip3 install scipy
export PYTHONPATH=$HOME/packages/midas/python
#git clone https://github.com/J033X071C/PhaseMeasurement
git clone https://bitbucket.org/team-ds-dm/phasemeasurement.git
cd phasemeasurement
python3 ./phaseMeasurement.py --help

daq00:PhaseMeasurement$ python3 ./phaseMeasurement.py --help
usage: phaseMeasurement.py [-h] fileName numberEvents numberVX sizeEvents stopEvent minHist maxHist numberBin writeToTXT saveAsPDF

Read data from midas file (in .lz4 format) to calculate phase between the clock of VX1 and VX2

positional arguments:
  fileName      Name of the file we want to read data from (Example: run00389.mid.lz4)
  numberEvents  Number of events recorded in the file
  numberVX      Number of VX used in this run (usually 2...)
  sizeEvents    Number of points per event
  stopEvent     Number of events you want to go through to calculate phase
  minHist       Minimal value for the x axis of the phase measurement histogram (in ns)
  maxHist       Maximal value for the x axis of the phase measurement histogram (in ns)
  numberBin     Number of bins wanted for the generated histogram
  writeToTXT    Write argument as yes to generate text file with results of calculation
  saveAsPDF     Save generated plots to PDF files

optional arguments:
  -h, --help    show this help message and exit
daq00:PhaseMeasurement$

try an old file with
python3 ./phaseMeasurement.py run00877.mid.lz4 10000 2 10000 500 -20 20 81 yes yes

bin size = 0.494 ns
num_events =  347
mean = -1.705 ns
rms = 3.087 ns
mean_error = 0.166 ns
centroid = -1.706 ns.
width (sigma) = 0.363 ns.
error on the centroid = 0.016558 ns.

ls -l *.txt *.pdf

dsdaq@dsvslice:~/online/PhaseMeasurement$ ls -l *.txt *.pdf
-rw-rw-r-- 1 dsdaq dsdaq 64728 Dec 14 16:56 run00877.mid.lz4_Plots.pdf
-rw-rw-r-- 1 dsdaq dsdaq   274 Dec 14 16:56 run00877.mid.lz4.txt
dsdaq@dsvslice:~/online/PhaseMeasurement$

DS-20K DAQ

Overview

DS-DM, GDM and CDM are key parts of the DS-20K DAQ system:

common clock distribution from external clock (atomic clock, GPS) to GDM to per-quadrant CDMs to VX digitizers
common trigger distribution from GDM internal algorithm or external input to all VX digitizers
run control: GDM, CDM, VX all start recording data at the same time (clock and timestamp reset)
collection of trigger data from VX digitizers to per-quadrant CDMs to GDM

Deliverables

hardware and firmware for GDM to CDM clock distribution
hardware and firmware for CDM to VX clock distribution
hardware and firmware for GDM external clock input (atomic clock or GPS)
hardware and firmware for CDM and VX serial communications (VX LVDS I/O connector)
firmware for run control (timestamp reset and sync): GDM to CDM to VX
firmware for common trigger distribution: GDM to CDM to VX
firmware for trigger data flow: VX to CDM to GDM
firmware for busy control: VX to CDM to GDM back to CDM to VX
firmware for flow control: FEP to GDM MIDAS frontend to GDM to CDM to VX
GDM MIDAS frontend: clock selector and monitoring, trigger and run control, busy and flow control, GDM housekeeping
CDM MIDAS frontend: clock monitoring, CDM housekeeping

specific performance:

GDM external clock: 10 MHz GPS clock
GDM to CDM fiber link:
- clock XXX MHz
- link data rate: XXX Gbit/sec
- CDM recovered clock: XXX MHz
- CDM recovered clock jitter: XXX ns
- phase alignment between CDMs: XXX ns
- phase alignment between CDMs persists across reboots, power cycles, firmware updates
- phase alignment between CDMs should be easy to measure
- phase alignment between CDMs should be easy to recalibrate if hardware parts are replaced (DS-DM boards, fiber transceivers, fiber cables, etc)
- data packet bandwidth: XXX Mbytes/sec
- data packet latency: XXX clocks
- data packet skew between CDMs: XXX clocks
CDM to VX clock:
- clock: XXX MHz
- jitter, all CDM clock outputs: XXX MHz
- phase alignment between all CDM clock outputs: XXX ns
CDM to VX trigger:
- TBD (use the VX "sync" input or VX LVDS I/O line or VX serial link packet)
CDM to VX serial link:
- clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
- bit rate: XXX bits/sec
- latency: XXX link clocks
- maximum skew between VXes: XXX ns
VX to CDM serial link:
- clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
- bit rate: XXX bits/sec
- latency: XXX link clocks
- maximum skew between VXes: XXX ns
timestamp reset:
- maximum skew between VXes: XXX ns
busy round trip time: XXX ns (VX to CDM to GDM back to CDM to VX)
flow control latency: XXX ns (FEP software to GDM MIDAS frontend to GDM to CDM to VX)

Technical risk items

this refers to unexpected behaviour and performance of system components, causes big difficulty in implementing the system, prevents delivery of deliverables, and prevents or negatively affects operation of the DS-20K DAQ or of the whole experiment.

(14-sep-2022, list is not sorted by any criterial: severity, probability, ease of investigation)

(stability of course is long term stability, across hours, days, weeks, months, years)

stability of Enclustra FPGA modules (crashes/year, failures to boot/year, flash corruption/year)
stability of GDM external clock PLL (lock loss/year)
stability of CDM recovered clock (lock loss/year, unexpected phase drifts, etc)
unexpected failures or bit error rates in GDM-CDM fiber links
stability of CDM VX clock outputs (stability of clock cleaner chip)
stability of VX internal clock distribution (VX PLL lock loss events)
stability of VX CAEN base firmware (different versions of CAEN base firmware have different clock distribution behaviour)
strange things in CAEN base firmware (unexpected clocking of LVDS I/O, unexpected phase shifts between clocks, etc)
DS-DM and VX hardware problems (incompatible LVSD I/O, incompatible clock signals, etc)

Milestones

(14-sep-2022: at this stage of the project, priority must be given to identifying and retiring (so called) technical risk factors. it is not good to build the complete system only to discover that (for example) some Enclustra FPGA modules require 5 attempts to boot and erase their flash memory contents once a month. Both example are real-life actual problems that caused big difficulties in GRIFFIN/TIGRESS and ALPHA-g experiments).

Development and testing milestones in time reversed order:

full DAQ data challenge: all VXes, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
one quadrant data challenge: 1 VME crate of VX, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
vertical slice data challenge: 1 VME crate, 2 VX, 2 CDM (1 VX per CDM), GDM, etc operate as designed
GDM-CDM link finalized (data rate frozen, data packet format frozen, data content permitted to change)
CDM-VX serial link finalized (data rate frozen, data packet format frozen, data content permitted to change)
run control (timestamp reset) and trigger distribution design agreed upon, frozen (list of possible triggers permitted to change)
VX to CDM to GDM data flow design agreed upon, frozen (data contents permitted to change)
major technical risk items retired (all hardware and firmware is working as expecred without mysteries and surprises, all problems are identified, investigated, resolved, solutions tested)
stable operation of CDM-VX serial links in vertical slice system
stable operation of GDM to CDM clock in vertical slice system
stable operation of CDM to VX clock in vertical slice system
vertical slice system assembled (1 VME crate, 2 VX, 2 CDM, 1 GDM, network, FEP, TSP, MIDAS)

ZZZ

DS-DM

DS-DM

Links

Onboard hardware

Buttons, jumpers and switches

Front panel

VX adapter board

split-cable connection

one-to-one connection

Board schematics

FPGA MGT blocks

Clock distribution

I2C bus

I2C clock builder connection

Clock domains

GPS

GDM

CDM

VX

Board test plan

Checklist for newly build boards

Serial console

i2c

U23

SFP

ethernet mac eeprom

read ethernet mac address from i2c

read ethernet mac address from i2c (SHOULD WORK)

read ethernet mac address from i2c (DOES NOT WORK)

enable VX clock

clock chip configuration

arm and linux benchmark

Install Xilinx tools

Petalinux

JTAG server

Firmware registers

Register map

register 0 0x80010000

register 1 0x80010004

register 2 0x80010008

register 3 0x8001000c

register 4 0x80010010

register 5 0x80010014

register 6 0x80010018

register 7 0x8001001c

register 246 0x800103d8

register 247 0x800103dc

control register 255 0x800103fc

Power up sequence

Build firmware

Build from git clone

Build branch develop_ko

Build in a fresh account

prepare bootable sd card

format the sd card

copy CDM boot files

copy boot files to the sd card

boot messages

Software

dsvslice integration

VX setup

GDM setup

CDM setup

After power up

Phase measurement

DS-20K DAQ

Overview

Deliverables

Technical risk items

Milestones

ZZZ

Navigation menu

Search