DS-DM

From DaqWiki
Jump to navigation Jump to search

DS-DM

DarkSide-20k Global and Crate Data Manager board (GDM and CDM).

Global Data Manager (GDM):

  • clock distribution to CDM boards (including GPS/atomic clock source)
  • collection of trigger data from CDM boards, processing and distribution of trigger decision to CDM boards
  • run control

Crate Data Manager (CDM):

  • clock distribution from GDM to CAEN digitizers
  • receive trigger data from CAEN digitizers
  • send trigger data to GDM
  • run control and dead time control

Links

edev links:

Xilinx links:

Enclustra links:

Onboard hardware

  • jtag chain: arm_dap_0 0x5BA00477, xczu4_1 0x04721093
  • Eclustra Mercury+ XU8 module: ME-XU8-4CG-1E-D11E-R2.1
    • Xilinx® Zynq Ultrascale+™ MPSoC XCZU4CG-1FBVB900E
    • DDR4 ECC SDRAM (PS) 2 GB
    • DDR4 SDRAM (PL) 1GB
  • ethernet mac chip: AT24MAC602-SSHM-T
  • USB UART for Enclustra serial console, micro-USB, 115200n8
  • LEDs:
    • LED_FP A/B/C/D 0/1/2/3
    • led1 - 3V3_SW_ON, SOM_POWER_GOOD - Enclustra FPGA module 3.3V power is good
    • led2 - LTM4624 PGOOD
    • led3 - FPGA_DONE - FPGA has booted
    • led4 - TP-S-1, PCLK_P
    • led5 - TP-S-2, PCLK_N
  • LEMO connectors (top to bottom)
    • J4 - input (NIM/TTL) (EXT_IN_LV(1), EXT_IN_LV(2))
    • J5 - input (NIM/TTL) (EXT_IN_LV(3), EXT_IN_LV(4))
    • J6 - external clock (GPS 10MHz and PPS)
    • J7 - output (NIM/TTL) (EXT_OUT(1), EXT_OUT(2))
  • SMA connectors
    • J9, J10 - CLK_CCA from U6 C.C.
    • J11, J12 - CLK_TP0
  • RJ45 ethernet connector (100mbit: green light on, yellow light flashes when there is traffic)
  • SFP connector
  • 4 QSFP connectors (GDM)
  • 6 VX connectors (CDM)

Buttons, jumpers and switches

Buttons:

  • PB1 - HRST - reboot FPGA (power-on reset)
  • PB2 - SRST - (SRSTn) - reboot ARM CPU

Switches:

  • SW1 - boot mode BM0, BM1 [-->]
  • SW2 - LEMO output NIM<->TTL
  • SW3 - LEMO input 1 and 2 NIM/TTL
  • SW4 - LEMO input 2 and 4 NIM/TTL
  • SW5 - ???
  • SW6 - serial console select. [PS<--PL] PS is ARM CPU, PL is FPGA.

Front panel

| top
|
| LED-FP1 | LED_FP(0,1,2,3)
|
| SFP J???
|
| LEMO J4-LEFT, J4-RIGHT | EXT_IN_LV(1), EXT_IN_LV(2)
| LEMO J5-LEFT, J5-RIGHT | EXT_IN_LV(3), EXT_IN_LV(4)
| LEMO J6-LEFT, J6-RIGHT |
| LEMO J7-LEFT, J7-RIGHT | EXT_OUT(1), EXT_OUT(2)
|
| J-VX-1
| J-VX-2 or QSFP
| J-VX-3 or QSFP
| J-VX-4 or QSFP
| J-VX-5 or QSFP
| J-VX-6
|
| RJ45 J3 ethernet
|
| bottom

VX adapter board

LVDS I/O connector

34 pin connector: 0|:::: :::: :::: :::: :|15,16 n/c

0 -> VX2_RX(3)
1 -> VX2_RX(2)
2 -> VX2_RX(0)
3 -> VX2_RX(1)

4 <- CLK
5 <- VX2_TX(0)
6 <- VX2_TX(1)
7 <- VX2_TX(2)

8 -> VX1_RX(3)
9 -> VX1_RX(2)
10 -> VX1_RX(1)
11 -> VX1_RX(0)

12 <- VX1_TX(0)
13 <- VX1_TX(1)
14 <- VX1_TX(2) (set by jumper routed here or to VX CLKIN SYNC)
15 <- n/c (CLK routed to VX CLKIN CLK)

16 - n/c

Board schematics

Clock distribution

125 MHz oscillator - U5 fan out -
  q0 -> CLK_XO_125 - FPGA PCLK pin - (NOT IN CDM PROJECT)
  q1 -> U6 C.C. in1
  q2 -> CLK3_XO_125 -> FPGA IO pin -> SFP RX reference clock
  q3 -> CLK2_XO_125 -> disconnected on the board (was -> FPGA SCLK pin)

U6 C.C (clock cleaner) -

in0 <- CLK_EXT1 (presumably GPS 10 MHz ref clock)
in1 <- 125 MHz oscillator via U5
in2 <- (was: CLK_CC_IN <- FPGA MGT_CLK pin <- SFP RX clock, 125 MHz)
in2 <- CLK2_XO_125 <- FPGA SCLK pin <- mgt_link_data_to_processing.rx_data_clk (SFP RX data clock, 125 MHz)
in3 <- CLK_FB

out0 -> CLK_CCA -> U12 (125 MHz)
out1 -> CLK_CCB -> VX1..6 (62.5 MHz)
out2 -> CLK_CCC -> VX7..12 (62.5 MHz)
out3 -> CLK_FB into in3

CLK_CCA -> U12 (125 MHz fan out)
 -> SMA J9/J10
 -> CLK_CC_OUT0 -> FPGA IO pin (-> MGT clock input ???)
 -> CLK_CC_OUT1 -> FPGA SCLK pin -> SFP TX clock (MGT clock input 125 MHz)
 -> unused output

CLK_TP0 - FPGA N7, N8 - Enclustra C69, C71 - DS-DM SMA J11, J12 (NOT IN CDM PROJECT)

We have 3 questions:
1) CLK_XO_125 (125 MHz osc) is not in the CDM project, why?
2) CLK_CC_OUT0 (U6 C.C. 125 MHz) is not in the CDM project, why? -> it is an MGT clock, connected now.
3) 62.5 MHz VX clock does not go into the FPGA, please confirm? -> confirmed

> And 1 comment:
>
> a) CLK_EXT0 going to in1 of U5 does not seem to be useful. only permitted frequency is 125 MHz and we normally do not distribute 125 MHz clocks on LEMO connectors and LEMO cables. (we use LVDS signals over SATA/eSATA cables and connectors). We remove it in the next revision of the board?

Proposed modifications:
- CLK_CC_IN: from FPGA output pin (J-ENC-C 142/144) to clock cleaner in2 (instead of CLK2_XO_!25 pins)
- CLK_CCA -> U12 -> currently unused out3 -> CLK2_XO_125 (FPGA SCLK pins)

Clock domains

GDM

  • AXI clock (100 MHz) - AXI registers
  • 10 MHz GPS clock - to clock cleaner
  • 125 MHz oscillator - to clock cleaner, to QSFP MGT RX reference clock pins
  • clock cleaner 125 MHz clock: (main clock domain)
    • QSFP MGT TX clock pins (via ...)
    • QSFP TX data clock
    • QSFP TX data
    • trigger logic
    • trigger logic output to QSFP TX data
  • QSFP RX recovered clock (per link, 12 clocks)
    • QSFP RX data to phase matching fifo to main clock domain to trigger logic

CDM

  • AXI clock (100 MHz) - AXI registers
  • 125 MHz oscillator to clock cleaner, to SFP MGT RX reference clock pin via CLK3_XO_125
  • SFP RX data clock ("recovered clock") (250 MHz/8 bit, 125 MHz/16 bit, 62.5 MHz/32 bit data) (main clock domain)
    • to clock cleaner to VX 62.5 MHz clock
    • SFP RX data
    • SFP RX data decoder
    • (SFP RX MGT clock 125 MHz not used because of uncontrollable phase)
    • SFP TX MGT clock 125 MHz via clock cleaner to CLK_CC_OUT1
    • SFP TX data clock
    • SFP TX data encoder
    • trigger logic
    • trigger logic to SFP TX
    • trigger logic to VX TX
    • VX TX data serializer
    • VX TX LVDS clock and data
    • inside the VX firmware: VX RX clock and data, data deserializer, phase matching fifo to VX trigger logic
  • CLK_XO_125 125 MHz from oscillator
    • not used
  • CLK_CC_OUT0 125 MHz from clock cleaner
    • not used
  • VX RX receive clock (per link, 12 clocks)
    • VX RX clock and data
    • VX RX data deserializer
    • phase matching fifo to main clock domain trigger logic

VX

  • VX 62.5 MHz clock from CDM
  • VX 125 MHz clock
    • drives Yair's trigger logic
    • trigger logic to LVDS TX
    • LVDS TX serializer
    • LVDS TX clock and data
  • LVDS RX clock (125 MHz from CDM)
    • LVDS RX clock and data
    • LVDS RX data deserializer
    • phase matching fifo to VX 125 MHz clock domain

Board test plan

To test:

  • Enclustra FPGA board
  • SFP port
  • CDM VX ports 2x(CLK, 3 tx, 4 rx)
devmem 0x80010014 32 # read VX1_RX
devmem 0x80010018 32 # read VX2_RX
devmem 0x8001001C 32 0x21 # write VX1_TX=1, VX2_TX=2
  • GDM QSFP ports (lanes 0,1,2. lane 3 n/c)
  • J6A, J6B external clock CLK_EXT1, CLK_EXT0 (sw5 is missing how is this supposed to work?). 50 ohm termination. CLK_EXT0 expects 125 MHz ?!?
  • BOOT_MODE 0 and 1

Done:

  • LED_FP1A..D: tested ok. K.O. 15 sep 2022
  • USB UART: tested ok. K.O. 15 sep 2022
  • J4A, J4B, J5A, J5B LEMO inputs (NIM/TTL) EXT_IN_LV(1..4). TTL threshold 1.7V, NIM threshold -0.3V. 50 Ohm termination.
  • J7A, J7B LEMO outputs EXT_OUT(1), EXT_OUT(2) (NIM/TTL) (tested 23nov2022, K.O.)
    • TTL out no 50 ohm termination: 0=0V, 1=5V, rise and fall time ~5 ns
    • TTL out with 50 ohm termination: 0=0V, 1=2.5V, rise and fall time <2ns
    • NIM out no 50 ohm termination: 0=+50mV, 1=-1.8V, rise and fall time ~3ns
    • NIM out with 50 ohm termination: 0=0V, 1=-0.9V, rise and fall time <2ns
  • ethernet MAC i2c chip (K.O. 20sep2022: can read: i2cdump 0 0x5b, nov2022 have u-boot driver)

Failure:

  • ethernet: does not connect to alliedtelesys switch. connects to my USB-eth adapter at 100 Mbit speed. uboot mii status reports connection speed oscillating between 1000, 100 and 10. K.O. 16-sep-2022

Checklist for newly build boards

  • put new board on workbench
  • check - vme connector present, vme extraction handles present
  • check - standoff are removed from all thru-holes
  • plug Enclustra module
  • check - SW6 both switches are in the "PS" position
  • connect micro-usb cable from linux PC
  • connect ethernet from 1gige capable network switch
  • connect power from lab power supply - GND, +5V and -12V nominal, +5.7V and -12.5V actual
  • power up, +5V current 2.10-2.8A, -12V current 0.05A
  • on linux PC, open a new terminal, run: minicom -D /dev/ttyACM0
  • in minicom window, observe messages about Xilinx first stage boot loader, etc
  • on the ethernet switch, observe network link is 1gige speed (not 10mbit, not 100mbit).
  • if everything boots okey, there will be a login prompt, login as root, password root.
  • busybox devmem 0x80010000 # read firmware version number, i.e. 0xEDAD0A77
  • TBW - test LEMO inputs
  • TBW - test LEMO outputs
  • TBW - test VX connectors
  • TBW - test SFP connector
  • TBW - test QSFP connector

Serial console

i2c

ZynqMP> i2c bus
Bus 0:  i2c@ff020000
ZynqMP> i2c dev 0
Setting bus to 0
ZynqMP> i2c probe  
Valid chip addresses: 33 4E 53 5B 6B 77
ZynqMP> i2c md 0x5b 0x98
0098: fc c2 3d 00 00 12 1a 6e 0a 90 85 04 94 10 08 50    ..=....n.......P
root@gdm-cdm:~# i2cdetect 0
Warning: Can't use SMBus Quick Write command, will skip some addresses
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0.
I will probe address range 0x03-0x77.
Continue? [Y/n] 
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                                                 
10:                                                 
20:                                                 
30: -- -- -- 33 -- -- -- --                         
40:                                                 
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- -- 
60:                                                 
70:                                                 
root@gdm-cdm:~# 
root@gdm-cdm:~# i2cdump 0 0x5b
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0, address 0x5b, mode byte
Continue? [Y/n] 
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
80: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00    ???????P?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e    ........??=..??n
a0: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00    ???????P?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e    ........??=..??n
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
root@gdm-cdm:~# 

read ethernet mac address from i2c

(this code is copied from uboot command line i2c code)

in uboot sources board/xilinx/common/board.c replace original function with this:

// special code to read ethernet MAC address from the DS-DM-Rev0 board. K.O. Sep-2022

int zynq_board_read_rom_ethaddr(unsigned char *ethaddr)
{
        struct udevice *bus;
	int ret;
        int busnum = 0;

	ret = uclass_get_device_by_seq(UCLASS_I2C, busnum, &bus);
	if (ret) {
           printf("%s: No bus %d\n", __func__, busnum);
           return ret;
	}

        int chip_addr = 0x5B;

        struct udevice *dev;

        ret = i2c_get_chip(bus, chip_addr, 1, &dev);
	if (ret) {
           printf("%s: Bus %d no chip 0x%02x\n", __func__, busnum, chip_addr);
           return ret;
	}

        int dev_addr = 0x98;

        unsigned char data[8];
           
        ret = dm_i2c_read(dev, dev_addr, data, 8);

	if (ret) {
           printf("%s: Bus %d chip 0x%02x read error %d\n", __func__, busnum, chip_addr, ret);
           return ret;
	}

        printf("%s: Bus %d chip 0x%02x addr 0x%02x read: 0x %02x %02x %02x %02x %02x %02x %02x %02x\n", __func__, busnum, chip_addr, dev_addr, data[0], data[1], data[2], data[3], data[4], data[5], data[6], data[7]);

        // see Atmel-8807-SEEPROM-AT24MAC402-602-Datasheet.pdf

        if (data[0] == 0) {
           // eiu-48 chip
           ethaddr[0] = data[2];
           ethaddr[1] = data[3];
           ethaddr[2] = data[4];
           ethaddr[3] = data[5];
           ethaddr[4] = data[6];
           ethaddr[5] = data[7];
        } else {
           // eiu-64 chip
           ethaddr[0] = data[0];
           ethaddr[1] = data[1];
           ethaddr[2] = data[2];
           ethaddr[3] = data[5];
           ethaddr[4] = data[6];
           ethaddr[5] = data[7];
        }

        printf("%s: ethaddr %02x:%02x:%02x:%02x:%02x:%02x\n", __func__, ethaddr[0], ethaddr[1], ethaddr[2], ethaddr[3], ethaddr[4], ethaddr[5]);

        return ret;
}

also this should have worked if i2c_xxx() functions were enabled in uboot:

i2c_set_bus_num(0);
i2c_probe(0x5b);
i2c_read(0x5b, 0x9a, ethaddr, 6);

read ethernet mac address from i2c (SHOULD WORK)

from: https://stackoverflow.com/questions/43637540/is-there-linux-or-u-boot-support-to-read-a-mac-address-from-a-chip-at-startup

#ethernet related setup
setup_eth=run readmac buildmac
#read mac address from eeprom
readmac=i2c dev 0; i2c read 50 FA.1 6 $loadaddr
#build the ethaddr variable
#not very nice, but does the job
buildmac=\
e=" "; sep=" " \
for i in 0 1 2 3 4 5 ; do\
setexpr x $loadaddr + $i\
setexpr.b b *$x\
e="$e$sep$b"\
sep=":"\
done &&\
setenv ethaddr $e

read ethernet mac address from i2c (DOES NOT WORK)

this method does not work: inside board/xilinx/common/board.c:int zynq_board_read_rom_ethaddr(unsigned char *ethaddr), uclass_get_device_by_ofnode(UCLASS_I2C_EEPROM, eeprom, &dev) returns ENODEV and read from i2c does not happen. K.O. Sep-2022

Read:

Note:

  • 0x5B is the i2c chip address
  • 0x9A is the data offset inside the chip, see datasheet or i2c read dump above.

Edit:

  • emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/u-boot/files/platform-top.h
#include <configs/xilinx_zynqmp.h>
#include <configs/platform-auto.h>
//#define CONFIG_I2C_EEPROM                                                                                                                                                                   
//#define CONFIG_SYS_I2C_EEPROM_ADDR 0x5b                                                                                                                                                     
//#define CONFIG_SYS_I2C_EEPROM_ADDR_OVERFLOW 0x0                                                                                                                                             
#define CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET 0x9A
#error HERE!
  • emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi
/include/ "system-conf.dtsi"
/ {
chosen {
   xlnx,eeprom = &eeprom;
};
};

&i2c0 {
eeprom: eeprom@5b { /* u88 */
compatible = "atmel,24mac402";
reg = <0x5b>;
};
};
  • components/yocto/workspace/sources/u-boot-xlnx/configs/xilinx_zynqmp_virt_defconfig
CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET=0x9A

enable VX clock

devmem 0x80010010 32 0x8; sleep 1;  devmem 0x80010010 32 0x0;
si5394-i2c-file /media/sd-mmcblk1p1/00_freerun.txt 0 0x6b

arm and linux benchmark

memory benchmark:

daq13$ arm-linux-gnueabi-gcc -o memcpy.armv7 memcpy.cc -march=armv7 -static -O2
scp memcpy.armv7 to ...
root@gdm-cdm:~# ./memcpy.armv7 
memcpy       1 KiBytes:   1288 MB/sec
memcpy       2 KiBytes:   1924 MB/sec
memcpy       4 KiBytes:   2554 MB/sec
memcpy       8 KiBytes:   3054 MB/sec
memcpy      16 KiBytes:   3262 MB/sec
memcpy      32 KiBytes:   3250 MB/sec
memcpy      64 KiBytes:   3456 MB/sec
memcpy     128 KiBytes:   3556 MB/sec
memcpy     256 KiBytes:   3780 MB/sec
memcpy     512 KiBytes:   3795 MB/sec
memcpy    1024 KiBytes:   3789 MB/sec
memcpy    2048 KiBytes:   3729 MB/sec
memcpy    4096 KiBytes:   3717 MB/sec
memcpy    8192 KiBytes:   3687 MB/sec
memcpy   16384 KiBytes:   3632 MB/sec
memcpy   32768 KiBytes:   3529 MB/sec
memcpy   65536 KiBytes:   3318 MB/sec
memcpy  131072 KiBytes:   2893 MB/sec
root@gdm-cdm:~# 

ethernet receive:

daq13:bin$ ./ttcp -t -s -n 100000 10.0.0.24
ttcp-t: buflen=8192, nbuf=100000, align=16384/0, port=5001  tcp  -> 10.0.0.24
ttcp-t: socket
ttcp-t: connect
ttcp-t: 819200000 bytes in 7.25 real seconds = 110358.39 KB/sec +++
ttcp-t: 100000 I/O calls, msec/call = 0.07, calls/sec = 13794.80
ttcp-t: 0.0user 0.2sys 0:07real 3% 0i+0d 760maxrss 0+2pf 1461+31csw
daq13:bin$ 

root@gdm-cdm:~# ./ttcp.armv7 -r -s
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001  tcp
ttcp-r: socket
ttcp-r: accept from 10.0.0.25
ttcp-r: 819200000 bytes in 7.27 real seconds = 110098.22 KB/sec +++
ttcp-r: 212040 I/O calls, msec/call = 0.04, calls/sec = 29181.53
ttcp-r: 0.1user 5.7sys 0:07real 81% 0i+0d 584maxrss 0+2pf 125601+2699csw
root@gdm-cdm:~# 

ethernet transmit:

root@gdm-cdm:~# ./ttcp.armv7 -t -s -n 100000 10.0.0.25
ttcp-t: buflen=8192, nbuf=100000, align=16384/0, port=5001  tcp  -> 10.0.0.25
ttcp-t: socket
ttcp-t: connect
ttcp-t: 819200000 bytes in 6.95 real seconds = 115078.69 KB/sec +++
ttcp-t: 100000 I/O calls, msec/call = 0.07, calls/sec = 14384.84
ttcp-t: 0.0user 0.7sys 0:06real 11% 0i+0d 584maxrss 0+2pf 1162+1017csw
root@gdm-cdm:~# 

daq13:bin$ ./ttcp -r -s
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001  tcp
ttcp-r: socket
ttcp-r: accept from 10.0.0.24
ttcp-r: 819200000 bytes in 6.97 real seconds = 114841.84 KB/sec +++
ttcp-r: 161335 I/O calls, msec/call = 0.04, calls/sec = 23160.01
ttcp-r: 0.0user 1.9sys 0:06real 28% 0i+0d 760maxrss 0+2pf 80646+51csw
daq13:bin$ 

Install Xilinx tools

  • install Vivado 2020.2
login at https://www.xilinx.com/myprofile.html
go to "Downloads"
go to archive,
find 2020.2
download Xilinx_Unified_2020.2_1118_1232_Lin64.bin
sh ./Xilinx_Unified_2020.2_1118_1232_Lin64.bin
banner window should open with spinner "downloading installation data"
"a newer version is available" -> say "continue"
next
"select install type" window:
provide email and password,
select "download image"
select directory /home/olchansk/Xilinx/Downloads/2020.2\
select "linux" and "full image"
next
download summary: space required 38.52 Gbytes
download
installation progress
downloading spinner, 16 M/s 47 minutes...
"download image has been created successfully". Ok.
check contents of /home/olchansk/Xilinx/Downloads/2020.2
ls -l /home/olchansk/Xilinx/Downloads/2020.2
total 67
drwxr-xr-x 2 olchansk users    9 Sep  1 16:22 bin
drwxr-xr-x 3 olchansk users   15 Sep  1 16:23 data
drwxr-xr-x 4 olchansk users    4 Sep  1 16:22 lib
drwxr-xr-x 2 olchansk users  644 Sep  1 16:22 payload
drwxr-xr-x 2 olchansk users    7 Sep  1 16:22 scripts
drwxr-xr-x 4 olchansk users    4 Sep  1 16:22 tps
-rwxr-xr-x 1 olchansk users 3256 Nov 18  2020 xsetup
daq13:2020.2$ 
./xsetup
spinned loading installation data
xilinx design tools 2022.1 now available -> say continue
"welcome" -> next
"select product" -> vivado -> next -> vivado hl system edition -> next
select devices: only zynq ultrascale+ mpsoc -> next
select destination: /opt/Xilinx (as root, mkdir /opt/Xilinx, chmod olchansk.users /opt/Xilinx)
install ...
complete
move /home/olchansk/Xilinx/Downloads/2020.2 to /daq/daqstore/olchansk/Xilinx/Downloads/
  • install petalinux 2020.2
./xsetup
"a newer version is available" -> say "continue"
next
"select product to install" -> select Petalinux (Linux only) -> next
"select destination directory" -> select "/opt/Xilinx" (disk space required 2.64 GB) -> next
"summary" -> install ...
error about missing /tmp/tmp-something files
"installation completed successfully" (hard to dismiss, "ok" button is partially cut-off)
done?
I think it failed, /opt/Xilinx/PetaLinux/2020.2/bin is empty except for petalinux-v2020.2-final-installer.run
try to run it by hand, same error about /tmp/tmp-something files. strange...
notice it complains about "truncate", which truncate finds ~/bin/truncate, get rid of it,
try again
now complains about missing texinfo and zlib1g:i386
apt install texinfo -> ok
apt install zlib1g:i386 -> installs bunch of gcc stuff -> ok
try again
reports "already installed" -> delete /opt/Xilinx/.xinstall/PetaLinux_2020.2/, delete entries in ~/.Xilinx/registry/installedSW.xml
try again
success
  • install vivado 2022.1 and petalinux 2022.1 - everything is pretty much the same

Petalinux

  • cd PetaLinux_GDM_CDM
  • petalinux-config
  • enable i2c MAC address and DHCP
git clone https://bitbucket.org:/team-ds-dm/ds-dm-u-boot-xlnx.git
cd ds-dm-u-boot-xlnx
git checkout ds-dm-u-boot-xlnx

linux-components ->
uboot -> ext-local-src
external u-boot local source -> ds-dm-u-boot-xlnx (path to the customized uboot git repository)
  • enable DHCP
Subsystem AUTO Hardware Settings -> Ethernet Settings
randomize MAC address -> NO
ethernet mac address -> leave empty
obtain ip address automatically -> YES
  • set hostname and product names
Firmware Version Configuration ->
Host name -> "ds-dm"
Product name -> "Petalinux_GDM_CDM"
  • configure linux kernel
petalinux-config -c kernel
  • enable NFS-Root
petalinux-config
Image Packaging Configuration > Root File System Type -> set to NFS
Location of NFS root directory set to "/nfsroot"

petalinux-config -c kernel
Networking support > IP: kernel level configuration
enable DHCP, BOOTP, RARP
File systems > Network file systems > Root file systems on NFS
  • manually fix linux kernel command line:
grep nfsroot PetaLinux_GDM_CDM/project-spec/configs/config
edit CONFIG_SUBSYSTEM_BOOTARGS_GENERATED to read
earlycon console=ttyPS0,115200 clk_ignore_unused panic=60 root=/dev/nfs nfsroot=/nfsroot/%s ip=dhcp rw
  • check configuration in
    • PetaLinux_GDM_CDM/project-spec/configs/config
    • PetaLinux_GDM_CDM/project-spec/configs/rootfs_config
    • PetaLinux_GDM_CDM/components/plnx_workspace/device-tree/device-tree/system-conf.dtsi

JTAG server

localhost:3121

Firmware registers

Register map

  0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
  1 | ALL | ALL | RW | read write scratch register
  2 | ALL | CDM | ?? | MGT not used
  3 | ALL | CDM | RO | MGT debug_data
  4 | ALL | CDM | RW | clk_config_vec
  5 | ALL | CDM | ?? | not used
  6 | ALL | CDM | RO | CDM_link_data_processing:o_error_time
  7 | ALL | CDM | RO | CDM_link_data_processing:o_error_count

255 | ALL | ALL | RW | control register

register 0 0x80010000

GDM:

0 - gdm_link_interface:i_mgt_rst
2 - gdm_link_interface:i_link_down_latched_rst
8 - GDM_link_data_processing:i_rst
10..9 - GDM_link_data_processing:i_data_mode

CDM:

0 - cdm_link_interface:i_mgt_rst
2 - cdm_link_interface:i_link_down_latched_rst
8 - CDM_link_data_processing:i_rst
10..9 - CDM_link_data_processing:i_data_mode

register 1 0x80010004

GDM:

nlinks-1..0 - gdm_link_interface:i_rx_slide_trigger

CDM:

nlinks-1..0 - cdm_link_interface:i_rx_slide_trigger

register 2 0x80010008

GDM:

nlinks-1..0 - gdm_link_interface:o_link_power_good
nlinks+15..16 - gdm_link_interface:o_link_status

CDM:

nlinks-1..0 - cdm_link_interface:o_link_power_good
nlinks+15..16 - cdm_link_interface:o_link_status

register 3 0x8001000c

GDM: simple loopback register

CDM: not connected

register 4 0x80010010

register 5 0x80010014

register 6 0x80010018

register 7 0x8001001c

control register 255

31..28 | ALL | function mux, 4 bits: 0 - outputs driven by control register, 1 - not used, 2 - dsvslice production, 3, 4, 5 - LVDS loopback, see source code
27 | ALL | lemo_in(4) invert
26 | ALL | lemo_in(3) invert
25 | ALL | lemo_in(2) invert
24 | ALL | lemo_in(1) invert
23 | ALL | lemo_in(4) enable
22 | ALL | lemo_in(3) enable
21 | ALL | lemo_in(2) enable
20 | ALL | lemo_in(1) enable
19
18
17
16 | ALL | trigger_enable
15
14
13 | ALL | drive ext_out(2) LEMO output
12 | ALL | drive ext_out(1) LEMO output
11 | ALL | fp_led_out(3) front panel LEDs
10 | ALL | fp_led_out(2)
 9 | ALL | fp_led_out(1)
 8 | ALL | fp_led_out(0)
 7 | ALL | not used
 6 | ALL | vx2_tx_out(2)
 5 | ALL | vx2_tx_out(1)
 4 | ALL | vx2_tx_out(0)
 3 | ALL | not used
 2 | ALL | vx1_tx_out(2)
 1 | ALL | vx1_tx_out(1)
 0 | ALL | vx1_tx_out(0)

Power up sequence

  • from README file:
busybox devmem 0x80010010 32 0x8
sleep 1;
busybox devmem 0x80010010 32 0x0
/home/dsdaq/si5394-i2c-file /home/dsdaq/CDM_v2.0_IN1_fixed_and_IN2_RX_Recovered_Si5394-RevA-Registers.txt  0 0x6b
busybox devmem 0x80010004 32 0x1;  # link startup mode reset until phase-aligned
busybox devmem 0x80010000 32 0x5;  # reset link and link status registers
sleep 1;   # just holds reset high
busybox devmem 0x80010000 32 0x0; # lower reset and let links lock
sleep 2;   # give links time to lock
busybox devmem 0x80010008 32;  # read status 0x00000003 means locked and phase aligned
busybox devmem 0x8001000c 32   # read data, 0x1B1C 3CBC lower 16 bits are K-Codes K.28.1 (3C) and K.28.5 (BC) when locked and phase aligned

Build firmware

Build from git clone

  • git clone git@edev-group.triumf.ca:fw/exp/darkside/gcdm.git
  • #Makefile change VIVADO_SETTINGS_SCRIPT := /opt/Xilinx/Vivado/2022.1/settings64.sh
  • #. /opt/Xilinx/Vivado/2022.1/settings64.sh
  • . /opt/Xilinx/Vivado/2020.2/settings64.sh
  • make clean
  • make all_from_scratch
  • . /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
  • make petalinux_create
  • make petalinux_rebuild_new_hw_des
  • bomb out: The TMPDIR: /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp can't be located on nfs.
  • mkdir /tmp/build_tmp
  • rm -rf /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp/
  • ln -s /tmp/build_tmp /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp
  • try again
  • grinds, loads a whole bunch of packages...
  • finishes with desire to copy things to /tftpboot
  • make sdcard_cp_to wants to copy files from PetaLinux_GDM_CDM/images/linux/ to SD card

Rebuild

. /opt/Xilinx/Vivado/2020.2/settings64.sh
. /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
/usr/bin/time make vivado_rebuild_fw_hw_des
# to see errors:
more Vivado_GDM_XU8/GDM_XU8.runs/synth_1/runme.log
# if successful, updates GDM_XU8_top.bit
ls -ltr Vivado_GDM_XU8/GDM_XU8.runs/impl_1/GDM_XU8_top.bit
#-rw-r--r-- 1 olchansk users 7797807 Sep 15 13:30 Vivado_GDM_XU8/GDM_XU8.runs/impl_1/GDM_XU8_top.bit
# update sdcard files
/usr/bin/time make petalinux_repackage
# updates BOOT.BIN
ls -l PetaLinux_GDM_CDM/images/linux/BOOT.BIN 
#-rw-r--r-- 1 olchansk users 9228720 Sep 15 13:34 PetaLinux_GDM_CDM/images/linux/BOOT.BIN

Rebuild branch develop_ko

make petalinux_build
make petalinux_repackage

in a different window, become root, go to the gcdm tree, insert the sd card:

make copy # will mount the sd card, copy xilinx boot files, unmount, eject

move sd card to the DS-DM board, reset DS-DM board by SRST button

Build in a fresh account

ssh dsdmdev@daq13
mkdir git
cd git
git clone /home/olchansk/git/ds-dm-gcdm
cd ds-dm-gcdm
. /opt/Xilinx/Vivado/2020.2/settings64.sh
. /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
/usr/bin/time make all_from_scratch BOARD=CDM
(sdcard_cp_to bombs out! FIXED!)
#/usr/bin/time make petalinux_build BOARD=CDM
#/usr/bin/time make petalinux_repackage BOARD=CDM

clean and build GDM:

rm -rf Vivado_GDM_XU8
/usr/bin/time make vivado_create_and_build BOARD=GDM
/usr/bin/time make petalinux_repackage BOARD=GDM

rebuild GDM:

/usr/bin/time make vivado_rebuild_fw_hw_des BOARD=GDM
/usr/bin/time make petalinux_repackage BOARD=GDM

copy to running GDM or CDM:

scp PetaLinux_GDM_CDM/images/linux/BOOT.BIN dsdaq@dsvslice:
#ssh dsdaq@dsvslice
#scp BOOT.BIN root@gdm0:/media/sd-mmcblk1p1/BOOT.BIN
#ssh root@gdm0 -> shutdown -r now
ssh dsdaq@dsvslice scp BOOT.BIN root@xdmn:/media/BOOT/BOOT.BIN
ssh dsdaq@dsvslice ssh root@xdmn shutdown -r now

copy to SD card:

open a root shell
format 16 GB Sd card per above
cd /home/dsdmdev/git/ds-dm-gcdm
make copy
(cannot confirm it worked, DS-DM CDM sn 6 serial console is broken)

prepare bootable sd card

format the sd card

this only needs to be done once

  • become root
  • cd ~olchansk/git/ds-dm-gcdm
  • use "lsblk" to identify the SD card (should show as 8/16/32 GB block device)/ /dev/sdd in this case
  • make sdcard_format SDCARD_DEVICE=/dev/sdd
  • disconnect sd card, reconnect the sd card (to detect new partition tables, etc)

copy CDM boot files

cd /home/dsdmdev/git/ds-dm-gcdm
make copy

copy boot files to the sd card

  • as root: identify partition labels, run "blkid", should say "BOOT", "rootfs" and "data"
  • mount
mkdir /media/olchansk/BOOT
mkdir /media/olchansk/rootfs
mkdir /media/olchansk/data
mount -L BOOT /media/olchansk/BOOT
mount -L rootfs /media/olchansk/rootfs
mount -L data /media/olchansk/data
cp PetaLinux_GDM_CDM/images/linux/BOOT.BIN /media/olchansk/BOOT/
cp PetaLinux_GDM_CDM/images/linux/boot.scr /media/olchansk/BOOT/
cp PetaLinux_GDM_CDM/images/linux/image.ub /media/olchansk/BOOT/
umount /media/olchansk/BOOT
umount /media/olchansk/rootfs
umount /media/olchansk/data
eject /dev/sdd

boot messages

Xilinx Zynq MP First Stage Boot Loader 
Release 2020.2   Sep 24 2022  -  13:29:15
NOTICE:  ATF running on XCZU4CG/silicon v4/RTL5.1 at 0xfffea000
NOTICE:  BL31: v2.2(release):xlnx_rebase_v2.2_2020.3
NOTICE:  BL31: Built : 18:02:46, Sep 28 2022


U-Boot 2020.01 (Sep 28 2022 - 18:03:39 +0000)

Model: DarkSide 20k DM
Board: Xilinx ZynqMP
DRAM:  2 GiB
usb dr_mode not found
PMUFW:  v1.1
EL Level:       EL2
Chip ID:        zu4
NAND:  0 MiB
MMC:   mmc@ff160000: 0, mmc@ff170000: 1
In:    serial@ff000000
Out:   serial@ff000000
Err:   serial@ff000000
Bootmode: SD_MODE1
Reset reason:   SOFT 
Net:   
ZYNQ GEM: ff0b0000, mdio bus ff0b0000, phyaddr -1, interface rgmii-id

Warning: ethernet@ff0b0000 (eth0) using random MAC address - d6:62:5f:13:00:44
eth0: ethernet@ff0b0000
ZYNQ GEM: ff0e0000, mdio bus ff0e0000, phyaddr -1, interface rgmii-id
Could not get PHY for eth1: addr -1

Hit any key to stop autoboot:  0 
ZynqMP> 
CTRL-A Z for help | 115200 8N1 | NOR | Minicom 2.7.1 | VT102 | Online 122:2 | ttyACM0                                       

Software

dsvslice integration

  • GDM is gdm0
  • CDM is cdm0
  • connect 1st VX port of cdm0 to VX1, 2nd port of cdm0 to VX2
  • power up
  • start the CDM frontend:
#ssh dsdaq@dsvslice
#scp fecdm.exe root@cdm0:
#ssh root@cdm0
#./fecdm.exe -h 192.168.0.251:1176
ssh dsdaq@dsvslice
ssh root@cdm0
/home/dsdaq/online/ds-dm-software/fecdm.exe -h 192.168.0.251:1176
  • CDM frontend should enable the VX clock, disable the trigger
  • start a run
  • CDM frontend will enable the trigger
  • stop a run
  • CDM frontend will disable the trigger

DS-20K DAQ

Overview

DS-DM, GDM and CDM are key parts of the DS-20K DAQ system:

  • common clock distribution from external clock (atomic clock, GPS) to GDM to per-quadrant CDMs to VX digitizers
  • common trigger distribution from GDM internal algorithm or external input to all VX digitizers
  • run control: GDM, CDM, VX all start recording data at the same time (clock and timestamp reset)
  • collection of trigger data from VX digitizers to per-quadrant CDMs to GDM

Deliverables

  • hardware and firmware for GDM to CDM clock distribution
  • hardware and firmware for CDM to VX clock distribution
  • hardware and firmware for GDM external clock input (atomic clock or GPS)
  • hardware and firmware for CDM and VX serial communications (VX LVDS I/O connector)
  • firmware for run control (timestamp reset and sync): GDM to CDM to VX
  • firmware for common trigger distribution: GDM to CDM to VX
  • firmware for trigger data flow: VX to CDM to GDM
  • firmware for busy control: VX to CDM to GDM back to CDM to VX
  • firmware for flow control: FEP to GDM MIDAS frontend to GDM to CDM to VX
  • GDM MIDAS frontend: clock selector and monitoring, trigger and run control, busy and flow control, GDM housekeeping
  • CDM MIDAS frontend: clock monitoring, CDM housekeeping

specific performance:

  • GDM external clock: 10 MHz GPS clock
  • GDM to CDM fiber link:
    • clock XXX MHz
    • link data rate: XXX Gbit/sec
    • CDM recovered clock: XXX MHz
    • CDM recovered clock jitter: XXX ns
    • phase alignment between CDMs: XXX ns
    • phase alignment between CDMs persists across reboots, power cycles, firmware updates
    • phase alignment between CDMs should be easy to measure
    • phase alignment between CDMs should be easy to recalibrate if hardware parts are replaced (DS-DM boards, fiber transceivers, fiber cables, etc)
    • data packet bandwidth: XXX Mbytes/sec
    • data packet latency: XXX clocks
    • data packet skew between CDMs: XXX clocks
  • CDM to VX clock:
    • clock: XXX MHz
    • jitter, all CDM clock outputs: XXX MHz
    • phase alignment between all CDM clock outputs: XXX ns
  • CDM to VX trigger:
    • TBD (use the VX "sync" input or VX LVDS I/O line or VX serial link packet)
  • CDM to VX serial link:
    • clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
    • bit rate: XXX bits/sec
    • latency: XXX link clocks
    • maximum skew between VXes: XXX ns
  • VX to CDM serial link:
    • clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
    • bit rate: XXX bits/sec
    • latency: XXX link clocks
    • maximum skew between VXes: XXX ns
  • timestamp reset:
    • maximum skew between VXes: XXX ns
  • busy round trip time: XXX ns (VX to CDM to GDM back to CDM to VX)
  • flow control latency: XXX ns (FEP software to GDM MIDAS frontend to GDM to CDM to VX)

Technical risk items

this refers to unexpected behaviour and performance of system components, causes big difficulty in implementing the system, prevents delivery of deliverables, and prevents or negatively affects operation of the DS-20K DAQ or of the whole experiment.

(14-sep-2022, list is not sorted by any criterial: severity, probability, ease of investigation)

(stability of course is long term stability, across hours, days, weeks, months, years)

  • stability of Enclustra FPGA modules (crashes/year, failures to boot/year, flash corruption/year)
  • stability of GDM external clock PLL (lock loss/year)
  • stability of CDM recovered clock (lock loss/year, unexpected phase drifts, etc)
  • unexpected failures or bit error rates in GDM-CDM fiber links
  • stability of CDM VX clock outputs (stability of clock cleaner chip)
  • stability of VX internal clock distribution (VX PLL lock loss events)
  • stability of VX CAEN base firmware (different versions of CAEN base firmware have different clock distribution behaviour)
  • strange things in CAEN base firmware (unexpected clocking of LVDS I/O, unexpected phase shifts between clocks, etc)
  • DS-DM and VX hardware problems (incompatible LVSD I/O, incompatible clock signals, etc)

Milestones

(14-sep-2022: at this stage of the project, priority must be given to identifying and retiring (so called) technical risk factors. it is not good to build the complete system only to discover that (for example) some Enclustra FPGA modules require 5 attempts to boot and erase their flash memory contents once a month. Both example are real-life actual problems that caused big difficulties in GRIFFIN/TIGRESS and ALPHA-g experiments).

Development and testing milestones in time reversed order:

  • full DAQ data challenge: all VXes, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
  • one quadrant data challenge: 1 VME crate of VX, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
  • vertical slice data challenge: 1 VME crate, 2 VX, 2 CDM (1 VX per CDM), GDM, etc operate as designed
  • GDM-CDM link finalized (data rate frozen, data packet format frozen, data content permitted to change)
  • CDM-VX serial link finalized (data rate frozen, data packet format frozen, data content permitted to change)
  • run control (timestamp reset) and trigger distribution design agreed upon, frozen (list of possible triggers permitted to change)
  • VX to CDM to GDM data flow design agreed upon, frozen (data contents permitted to change)
  • major technical risk items retired (all hardware and firmware is working as expecred without mysteries and surprises, all problems are identified, investigated, resolved, solutions tested)
  • stable operation of CDM-VX serial links in vertical slice system
  • stable operation of GDM to CDM clock in vertical slice system
  • stable operation of CDM to VX clock in vertical slice system
  • vertical slice system assembled (1 VME crate, 2 VX, 2 CDM, 1 GDM, network, FEP, TSP, MIDAS)

ZZZ

ZZZ