DS-DM: Difference between revisions
Jump to navigation
Jump to search
(→GDM) |
(→GDM) |
||
| Line 408: | Line 408: | ||
CLK_RSTn_LS <= not clk_config_vec(3); | CLK_RSTn_LS <= not clk_config_vec(3); | ||
register_data_out(4)(31 downto 8) <= | register_data_out(4)(31 downto 8) <= | ||
EXT_OUT & EXT_OUT & qsfp_lp_mode & qsfp_reset_n & | 31, 30, 29, 28: EXT_OUT & EXT_OUT & | ||
QSFP_IntN_LS_SFP_RX_LOS & QSFP_ModPrsN_LS_SFP_ModDet & CLK2_XO_125 & CLK2_XO_125 & | 27, 26, 25, 24: qsfp_lp_mode & qsfp_reset_n & QSFP_IntN_LS_SFP_RX_LOS & QSFP_ModPrsN_LS_SFP_ModDet & | ||
EXT_IN_LV & PL_UART_RX & PL_UART_TX & | 2 bits: CLK2_XO_125 & CLK2_XO_125 & | ||
'0' & '0' & '0' & '0' & | 4 bits: EXT_IN_LV & | ||
test_points & HW_ID_xDM & '1'; | 2 bits: PL_UART_RX & PL_UART_TX & | ||
4 bits: '0' & '0' & '0' & '0' & | |||
4 bits: test_points & HW_ID_xDM & '1'; | |||
register_data_out(5) <= VX1_RX & vx_tx(0) & VX2_RX & vx_tx(1) & VX3_RX & vx_tx(2) & VX4_RX & vx_tx(3) & level_shifted_io; | register_data_out(5) <= VX1_RX & vx_tx(0) & VX2_RX & vx_tx(1) & VX3_RX & vx_tx(2) & VX4_RX & vx_tx(3) & level_shifted_io; | ||
register_data_out(6) <= VX5_RX & vx_tx(4) & VX6_RX & vx_tx(5) & VX7_RX & vx_tx(6) & VX8_RX & vx_tx(7) & "1111"; | register_data_out(6) <= VX5_RX & vx_tx(4) & VX6_RX & vx_tx(5) & VX7_RX & vx_tx(6) & VX8_RX & vx_tx(7) & "1111"; | ||
Revision as of 17:55, 30 September 2022
DS-DM
DarkSide-20k Global and Crate Data Manager board (GDM and CDM).
Global Data Manager (GDM):
- clock distribution to CDM boards (including GPS/atomic clock source)
- collection of trigger data from CDM boards, processing and distribution of trigger decision to CDM boards
- run control
Crate Data Manager (CDM):
- clock distribution from GDM to CAEN digitizers
- receive trigger data from CAEN digitizers
- send trigger data to GDM
- run control and dead time control
Links
edev links:
- https://edev-group.triumf.ca/fw/exp/darkside/gcdm
- https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0
- https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0/-/blob/master/Altium/Project%20Outputs%20for%20DS-DM-Rev0/SCH-DS-xDM-Rev0.PDF
- https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0/-/blob/master/Testing/Clk3_XO_125_to_fpgaIN_recoveredMGTclk_to_IN2_Si5394-RevA-Project.slabtimeproj
Xilinx links:
- Platform Cable USB II: https://docs.xilinx.com/v/u/en-US/ds593
Enclustra links:
Onboard hardware
- jtag chain: arm_dap_0 0x5BA00477, xczu4_1 0x04721093
- Eclustra Mercury+ XU8 module: ME-XU8-4CG-1E-D11E-R2.1
- Xilinx® Zynq Ultrascale+™ MPSoC XCZU4CG-1FBVB900E
- DDR4 ECC SDRAM (PS) 2 GB
- DDR4 SDRAM (PL) 1GB
- ethernet mac chip: AT24MAC602-SSHM-T
- USB UART for Enclustra serial console, micro-USB, 115200n8
- LEDs:
- LED_FP A/B/C/D 0/1/2/3
- led1 - 3V3_SW_ON, SOM_POWER_GOOD - Enclustra FPGA module 3.3V power is good
- led2 - LTM4624 PGOOD
- led3 - FPGA_DONE - FPGA has booted
- led4 - TP-S-1, PCLK_P
- led5 - TP-S-2, PCLK_N
- LEMO connectors (top to bottom)
- J4 - input (NIM/TTL)
- J5 - input (NIM/TTL)
- J6 - external clock (GPS 10MHz and PPS)
- J7 - output (NIM/TTL)
- RJ45 ethernet connector (100mbit: green light on, yellow light flashes when there is traffic)
- SFP connector
- 4 QSFP connectors (GDM)
- 6 VX connectors (CDM)
Buttons, jumpers and switches
Buttons:
- PB1 - HRST - reboot FPGA (power-on reset)
- PB2 - SRST - (SRSTn) - reboot ARM CPU
Switches:
- SW1 - boot mode BM0, BM1 [-->]
- SW2 - LEMO output NIM<->TTL
- SW3 - LEMO input 1 and 2 NIM/TTL
- SW4 - LEMO input 2 and 4 NIM/TTL
- SW5 - ???
- SW6 - serial console select. [PS<--PL] PS is ARM CPU, PL is FPGA.
Board schematics
Board test plan
To test:
- Enclustra FPGA board
- SFP port
- CDM VX ports 2x(CLK, 3 tx, 4 rx)
- GDM QSFP ports (lanes 0,1,2. lane 3 n/c)
- J4A, J4B, J5A, J5B external inputs (NIM/TTL) EXT_IN1..4_LV. TTL threshold 1.7V, NIM threshold -0.3V. 50 Ohm termination.
- J6A, J6B external clock CLK_EXT1, CLK_EXT0 (sw5 is missing how is this supposed to work?). 50 ohm termination.
- ethernet MAC i2c chip (K.O. 20sep2022: can read: i2cdump 0 0x5b)
- BOOT_MODE 0 and 1
Done:
- LED_FP1A..D: tested ok. K.O. 15 sep 2022
- USB UART: tested ok. K.O. 15 sep 2022
- J7A, J7B external outputs EXT_OUT1, EXT_OUT2 (NIM/TTL)
- TTL out tested ok, 0=0V, 1=5V, rise time 2 ns, K.O. 15 sep 2022
Failure:
- ethernet: does not connect to alliedtelesys switch. connects to my USB-eth adapter at 100 Mbit speed. uboot mii status reports connection speed oscillating between 1000, 100 and 10. K.O. 16-sep-2022
- J7A, J7B external outputs EXT_OUT1, EXT_OUT2 (NIM/TTL)
- NIM problematic, 0=0V, without 50 ohm termination: 1=-5V, rise time 50 and 150 ns (the two outputs are not the same), drop time 100 ns. with 50 ohm termination, 1=-1V, rise time 20 ns, drop time 10-20ns, the two channels are not the same.
Checklist for newly build boards
AAA
Serial console
- check that linux computer has correct udev rules to allow access to /dev/ttyACM devices, see https://daq00.triumf.ca/DaqWiki/index.php/SLinstall#Configure_USB_device_permissions and https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Configure_USB_device_permissions
- connect micro-USB cable to connector J-UCB, other end connect to linux computer
- observe /dev/ttyACM0 was created
- run "minicom -D /dev/ttyACM0" (default serial settings are ok, otherwise, 115200n8)
- should have gdm-cdm login
- username root, password root
i2c
ZynqMP> i2c bus Bus 0: i2c@ff020000 ZynqMP> i2c dev 0 Setting bus to 0 ZynqMP> i2c probe Valid chip addresses: 33 4E 53 5B 6B 77 ZynqMP> i2c md 0x5b 0x98 0098: fc c2 3d 00 00 12 1a 6e 0a 90 85 04 94 10 08 50 ..=....n.......P
root@gdm-cdm:~# i2cdetect 0
Warning: Can't use SMBus Quick Write command, will skip some addresses
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0.
I will probe address range 0x03-0x77.
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f
00:
10:
20:
30: -- -- -- 33 -- -- -- --
40:
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60:
70:
root@gdm-cdm:~#
root@gdm-cdm:~# i2cdump 0 0x5b
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0, address 0x5b, mode byte
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00 ???????P?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e ........??=..??n
a0: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00 ???????P?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e ........??=..??n
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@gdm-cdm:~#
read ethernet mac address from i2c
(this code is copied from uboot command line i2c code)
in uboot sources board/xillinx/common/board.c replace original function with this:
int zynq_board_read_rom_ethaddr(unsigned char *ethaddr)
{
struct udevice *bus;
int ret;
int busnum = 0;
ret = uclass_get_device_by_seq(UCLASS_I2C, busnum, &bus);
if (ret) {
printf("%s: No bus %d\n", __func__, busnum);
return ret;
}
int chip_addr = 0x5B;
struct udevice *dev;
ret = i2c_get_chip(bus, chip_addr, 1, &dev);
if (ret) {
printf("%s: Bus %d no chip 0x%02x\n", __func__, busnum, chip_addr);
return ret;
}
int dev_addr = 0x98;
unsigned char data[8];
ret = dm_i2c_read(dev, dev_addr, data, 8);
if (ret) {
printf("%s: Bus %d chip 0x%02x read error %d\n", __func__, busnum, chip_addr, ret);
return ret;
}
printf("%s: Bus %d chip 0x%02x addr 0x%02x read: 0x %02x %02x %02x %02x %02x %02x %02x %02x\n", __func__, busnum, chip_addr, dev_addr, data[0], data[1], data[2], data[3], data[4], data[5], data[6], data[7]);
// see Atmel-8807-SEEPROM-AT24MAC402-602-Datasheet.pdf
if (data[0] == 0) {
// eiu-48 chip
ethaddr[0] = data[2];
ethaddr[1] = data[3];
ethaddr[2] = data[4];
ethaddr[3] = data[5];
ethaddr[4] = data[6];
ethaddr[5] = data[7];
} else {
// eiu-64 chip
ethaddr[0] = data[0];
ethaddr[1] = data[1];
ethaddr[2] = data[2];
ethaddr[3] = data[5];
ethaddr[4] = data[6];
ethaddr[5] = data[7];
}
printf("%s: ethaddr %02x:%02x:%02x:%02x:%02x:%02x\n", __func__, ethaddr[0], ethaddr[1], ethaddr[2], ethaddr[3], ethaddr[4], ethaddr[5]);
return ret;
}
also this should have worked if i2c_xxx() functions were enabled in uboot:
i2c_set_bus_num(0); i2c_probe(0x5b); i2c_read(0x5b, 0x9a, ethaddr, 6);
read ethernet mac address from i2c (SHOULD WORK)
#ethernet related setup setup_eth=run readmac buildmac #read mac address from eeprom readmac=i2c dev 0; i2c read 50 FA.1 6 $loadaddr #build the ethaddr variable #not very nice, but does the job buildmac=\ e=" "; sep=" " \ for i in 0 1 2 3 4 5 ; do\ setexpr x $loadaddr + $i\ setexpr.b b *$x\ e="$e$sep$b"\ sep=":"\ done &&\ setenv ethaddr $e
read ethernet mac address from i2c (DOES NOT WORK)
this method does not work: inside board/xilinx/common/board.c:int zynq_board_read_rom_ethaddr(unsigned char *ethaddr), uclass_get_device_by_ofnode(UCLASS_I2C_EEPROM, eeprom, &dev) returns ENODEV and read from i2c does not happen. K.O. Sep-2022
Read:
- https://support.xilinx.com/s/question/0D52E00006hpKlsSAE/uboot-and-ethernet-mac-address-from-eeprom?language=en_US (last answer)
- https://github.com/Xilinx/u-boot-xlnx/blob/master/board/xilinx/common/board.c (zynq_board_read_rom_ethaddr())
Note:
- 0x5B is the i2c chip address
- 0x9A is the data offset inside the chip, see datasheet or i2c read dump above.
Edit:
- emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/u-boot/files/platform-top.h
#include <configs/xilinx_zynqmp.h> #include <configs/platform-auto.h> //#define CONFIG_I2C_EEPROM //#define CONFIG_SYS_I2C_EEPROM_ADDR 0x5b //#define CONFIG_SYS_I2C_EEPROM_ADDR_OVERFLOW 0x0 #define CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET 0x9A #error HERE!
- emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi
/include/ "system-conf.dtsi"
/ {
chosen {
xlnx,eeprom = &eeprom;
};
};
&i2c0 {
eeprom: eeprom@5b { /* u88 */
compatible = "atmel,24mac402";
reg = <0x5b>;
};
};
- components/yocto/workspace/sources/u-boot-xlnx/configs/xilinx_zynqmp_virt_defconfig
CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET=0x9A
Install Xilinx tools
- install Vivado 2020.2
login at https://www.xilinx.com/myprofile.html go to "Downloads" go to archive, find 2020.2 download Xilinx_Unified_2020.2_1118_1232_Lin64.bin sh ./Xilinx_Unified_2020.2_1118_1232_Lin64.bin banner window should open with spinner "downloading installation data" "a newer version is available" -> say "continue" next "select install type" window: provide email and password, select "download image" select directory /home/olchansk/Xilinx/Downloads/2020.2\ select "linux" and "full image" next download summary: space required 38.52 Gbytes download installation progress downloading spinner, 16 M/s 47 minutes... "download image has been created successfully". Ok. check contents of /home/olchansk/Xilinx/Downloads/2020.2 ls -l /home/olchansk/Xilinx/Downloads/2020.2 total 67 drwxr-xr-x 2 olchansk users 9 Sep 1 16:22 bin drwxr-xr-x 3 olchansk users 15 Sep 1 16:23 data drwxr-xr-x 4 olchansk users 4 Sep 1 16:22 lib drwxr-xr-x 2 olchansk users 644 Sep 1 16:22 payload drwxr-xr-x 2 olchansk users 7 Sep 1 16:22 scripts drwxr-xr-x 4 olchansk users 4 Sep 1 16:22 tps -rwxr-xr-x 1 olchansk users 3256 Nov 18 2020 xsetup daq13:2020.2$ ./xsetup spinned loading installation data xilinx design tools 2022.1 now available -> say continue "welcome" -> next "select product" -> vivado -> next -> vivado hl system edition -> next select devices: only zynq ultrascale+ mpsoc -> next select destination: /opt/Xilinx (as root, mkdir /opt/Xilinx, chmod olchansk.users /opt/Xilinx) install ... complete move /home/olchansk/Xilinx/Downloads/2020.2 to /daq/daqstore/olchansk/Xilinx/Downloads/
- install petalinux 2020.2
./xsetup "a newer version is available" -> say "continue" next "select product to install" -> select Petalinux (Linux only) -> next "select destination directory" -> select "/opt/Xilinx" (disk space required 2.64 GB) -> next "summary" -> install ... error about missing /tmp/tmp-something files "installation completed successfully" (hard to dismiss, "ok" button is partially cut-off) done? I think it failed, /opt/Xilinx/PetaLinux/2020.2/bin is empty except for petalinux-v2020.2-final-installer.run try to run it by hand, same error about /tmp/tmp-something files. strange... notice it complains about "truncate", which truncate finds ~/bin/truncate, get rid of it, try again now complains about missing texinfo and zlib1g:i386 apt install texinfo -> ok apt install zlib1g:i386 -> installs bunch of gcc stuff -> ok try again reports "already installed" -> delete /opt/Xilinx/.xinstall/PetaLinux_2020.2/, delete entries in ~/.Xilinx/registry/installedSW.xml try again success
- install vivado 2022.1 and petalinux 2022.1 - everything is pretty much the same
JTAG server
localhost:3121
Firmware registers
GDM
register_data_out(0) <= register_data_in(0); register_data_out(1) <= register_data_in(1); --register_data_out(2) <= register_data_in(2); --register_data_out(3) <= register_data_in(3); register_data_out(2)(15 downto nlinks) <= (others => '0'); register_data_out(2)(31 downto nlinks+16) <= (others => '0'); --register_data_out(2)(31 downto nlinks+16) <= (others => '0'); register_data_out(3) <= debug_data(31 downto 0); register_data_out(4)( 7 downto 0) <= clk_config_vec(7 downto 0); clk_config_vec(3 downto 0) <= register_data_in(4)(3 downto 0); clk_config_vec(4) <= CLK_LOSXTn_LS; clk_config_vec(5) <= CLK_LOLn_LS; clk_config_vec(6) <= CLK_INTn_LS; clk_config_vec(7) <= '1'; CLK_IN_SEL_LS <= clk_config_vec(1 downto 0); CLK_EXT_SEL_LS <= clk_config_vec(2); CLK_RSTn_LS <= not clk_config_vec(3); register_data_out(4)(31 downto 8) <= 31, 30, 29, 28: EXT_OUT & EXT_OUT & 27, 26, 25, 24: qsfp_lp_mode & qsfp_reset_n & QSFP_IntN_LS_SFP_RX_LOS & QSFP_ModPrsN_LS_SFP_ModDet & 2 bits: CLK2_XO_125 & CLK2_XO_125 & 4 bits: EXT_IN_LV & 2 bits: PL_UART_RX & PL_UART_TX & 4 bits: '0' & '0' & '0' & '0' & 4 bits: test_points & HW_ID_xDM & '1'; register_data_out(5) <= VX1_RX & vx_tx(0) & VX2_RX & vx_tx(1) & VX3_RX & vx_tx(2) & VX4_RX & vx_tx(3) & level_shifted_io; register_data_out(6) <= VX5_RX & vx_tx(4) & VX6_RX & vx_tx(5) & VX7_RX & vx_tx(6) & VX8_RX & vx_tx(7) & "1111"; register_data_out(7) <= VX9_RX & vx_tx(8) & VX10_RX & vx_tx(9) & VX11_RX & vx_tx(10) & VX12_RX & vx_tx(11) & "1111"; also i_mgt_rst => register_data_in(0)(0), i_rx_slide_trigger => register_data_in(1)(nlinks-1 downto 0), o_link_power_good => register_data_out(2)(nlinks-1 downto 0), o_link_status => register_data_out(2)(nlinks+15 downto 16), i_link_down_latched_rst => register_data_in(0)(2),
Build firmware
Build from git clone
- git clone git@edev-group.triumf.ca:fw/exp/darkside/gcdm.git
- #Makefile change VIVADO_SETTINGS_SCRIPT := /opt/Xilinx/Vivado/2022.1/settings64.sh
- #. /opt/Xilinx/Vivado/2022.1/settings64.sh
- . /opt/Xilinx/Vivado/2020.2/settings64.sh
- make clean
- make all_from_scratch
- . /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
- make petalinux_create
- make petalinux_rebuild_new_hw_des
- bomb out: The TMPDIR: /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp can't be located on nfs.
- mkdir /tmp/build_tmp
- rm -rf /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp/
- ln -s /tmp/build_tmp /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp
- try again
- grinds, loads a whole bunch of packages...
- finishes with desire to copy things to /tftpboot
- make sdcard_cp_to wants to copy files from PetaLinux_GDM_CDM/images/linux/ to SD card
Rebuild
. /opt/Xilinx/Vivado/2020.2/settings64.sh . /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh /usr/bin/time make vivado_rebuild_fw_hw_des # to see errors: more Vivado_GDM_XU8/GDM_XU8.runs/synth_1/runme.log # if successful, updates GDM_XU8_top.bit ls -ltr Vivado_GDM_XU8/GDM_XU8.runs/impl_1/GDM_XU8_top.bit #-rw-r--r-- 1 olchansk users 7797807 Sep 15 13:30 Vivado_GDM_XU8/GDM_XU8.runs/impl_1/GDM_XU8_top.bit # update sdcard files /usr/bin/time make petalinux_repackage # updates BOOT.BIN ls -l PetaLinux_GDM_CDM/images/linux/BOOT.BIN #-rw-r--r-- 1 olchansk users 9228720 Sep 15 13:34 PetaLinux_GDM_CDM/images/linux/BOOT.BIN
Rebuild branch develop_ko
make petalinux_build make petalinux_repackage
in a different window, become root, go to the gcdm tree, insert the sd card:
make copy # will mount the sd card, copy xilinx boot files, unmount, eject
move sd card to the DS-DM board, reset DS-DM board by SRST button
Build in a fresh account
ssh dsdmdev@daq13 mkdir git cd git git clone /home/olchansk/git/ds-dm-gcdm cd ds-dm-gcdm . /opt/Xilinx/Vivado/2020.2/settings64.sh . /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh /usr/bin/time make all_from_scratch BOARD=CDM (sdcard_cp_to bombs out! FIXED!) #/usr/bin/time make petalinux_build BOARD=CDM #/usr/bin/time make petalinux_repackage BOARD=CDM
open a root shell format 16 GB Sd card per above cd /home/dsdmdev/git/ds-dm-gcdm make copy (cannot confirm it worked, DS-DM CDM sn 6 serial console is broken)
prepare bootable sd card
format the sd card
this only needs to be done once
- become root
- cd ~olchansk/git/ds-dm-gcdm
- use "lsblk" to identify the SD card (should show as 8/16/32 GB block device)/ /dev/sdd in this case
- make sdcard_format SDCARD_DEVICE=/dev/sdd
- disconnect sd card, reconnect the sd card (to detect new partition tables, etc)
copy boot files to the sd card
- as root: identify partition labels, run "blkid", should say "BOOT", "rootfs" and "data"
- mount
mkdir /media/olchansk/BOOT mkdir /media/olchansk/rootfs mkdir /media/olchansk/data mount -L BOOT /media/olchansk/BOOT mount -L rootfs /media/olchansk/rootfs mount -L data /media/olchansk/data cp PetaLinux_GDM_CDM/images/linux/BOOT.BIN /media/olchansk/BOOT/ cp PetaLinux_GDM_CDM/images/linux/boot.scr /media/olchansk/BOOT/ cp PetaLinux_GDM_CDM/images/linux/image.ub /media/olchansk/BOOT/ umount /media/olchansk/BOOT umount /media/olchansk/rootfs umount /media/olchansk/data eject /dev/sdd
DS-20K DAQ
Overview
DS-DM, GDM and CDM are key parts of the DS-20K DAQ system:
- common clock distribution from external clock (atomic clock, GPS) to GDM to per-quadrant CDMs to VX digitizers
- common trigger distribution from GDM internal algorithm or external input to all VX digitizers
- run control: GDM, CDM, VX all start recording data at the same time (clock and timestamp reset)
- collection of trigger data from VX digitizers to per-quadrant CDMs to GDM
Deliverables
- hardware and firmware for GDM to CDM clock distribution
- hardware and firmware for CDM to VX clock distribution
- hardware and firmware for GDM external clock input (atomic clock or GPS)
- hardware and firmware for CDM and VX serial communications (VX LVDS I/O connector)
- firmware for run control (timestamp reset and sync): GDM to CDM to VX
- firmware for common trigger distribution: GDM to CDM to VX
- firmware for trigger data flow: VX to CDM to GDM
- firmware for busy control: VX to CDM to GDM back to CDM to VX
- firmware for flow control: FEP to GDM MIDAS frontend to GDM to CDM to VX
- GDM MIDAS frontend: clock selector and monitoring, trigger and run control, busy and flow control, GDM housekeeping
- CDM MIDAS frontend: clock monitoring, CDM housekeeping
specific performance:
- GDM external clock: 10 MHz GPS clock
- GDM to CDM fiber link:
- clock XXX MHz
- link data rate: XXX Gbit/sec
- CDM recovered clock: XXX MHz
- CDM recovered clock jitter: XXX ns
- phase alignment between CDMs: XXX ns
- phase alignment between CDMs persists across reboots, power cycles, firmware updates
- phase alignment between CDMs should be easy to measure
- phase alignment between CDMs should be easy to recalibrate if hardware parts are replaced (DS-DM boards, fiber transceivers, fiber cables, etc)
- data packet bandwidth: XXX Mbytes/sec
- data packet latency: XXX clocks
- data packet skew between CDMs: XXX clocks
- CDM to VX clock:
- clock: XXX MHz
- jitter, all CDM clock outputs: XXX MHz
- phase alignment between all CDM clock outputs: XXX ns
- CDM to VX trigger:
- TBD (use the VX "sync" input or VX LVDS I/O line or VX serial link packet)
- CDM to VX serial link:
- clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
- bit rate: XXX bits/sec
- latency: XXX link clocks
- maximum skew between VXes: XXX ns
- VX to CDM serial link:
- clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
- bit rate: XXX bits/sec
- latency: XXX link clocks
- maximum skew between VXes: XXX ns
- timestamp reset:
- maximum skew between VXes: XXX ns
- busy round trip time: XXX ns (VX to CDM to GDM back to CDM to VX)
- flow control latency: XXX ns (FEP software to GDM MIDAS frontend to GDM to CDM to VX)
Technical risk items
this refers to unexpected behaviour and performance of system components, causes big difficulty in implementing the system, prevents delivery of deliverables, and prevents or negatively affects operation of the DS-20K DAQ or of the whole experiment.
(14-sep-2022, list is not sorted by any criterial: severity, probability, ease of investigation)
(stability of course is long term stability, across hours, days, weeks, months, years)
- stability of Enclustra FPGA modules (crashes/year, failures to boot/year, flash corruption/year)
- stability of GDM external clock PLL (lock loss/year)
- stability of CDM recovered clock (lock loss/year, unexpected phase drifts, etc)
- unexpected failures or bit error rates in GDM-CDM fiber links
- stability of CDM VX clock outputs (stability of clock cleaner chip)
- stability of VX internal clock distribution (VX PLL lock loss events)
- stability of VX CAEN base firmware (different versions of CAEN base firmware have different clock distribution behaviour)
- strange things in CAEN base firmware (unexpected clocking of LVDS I/O, unexpected phase shifts between clocks, etc)
- DS-DM and VX hardware problems (incompatible LVSD I/O, incompatible clock signals, etc)
Milestones
(14-sep-2022: at this stage of the project, priority must be given to identifying and retiring (so called) technical risk factors. it is not good to build the complete system only to discover that (for example) some Enclustra FPGA modules require 5 attempts to boot and erase their flash memory contents once a month. Both example are real-life actual problems that caused big difficulties in GRIFFIN/TIGRESS and ALPHA-g experiments).
Development and testing milestones in time reversed order:
- full DAQ data challenge: all VXes, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
- one quadrant data challenge: 1 VME crate of VX, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
- vertical slice data challenge: 1 VME crate, 2 VX, 2 CDM (1 VX per CDM), GDM, etc operate as designed
- GDM-CDM link finalized (data rate frozen, data packet format frozen, data content permitted to change)
- CDM-VX serial link finalized (data rate frozen, data packet format frozen, data content permitted to change)
- run control (timestamp reset) and trigger distribution design agreed upon, frozen (list of possible triggers permitted to change)
- VX to CDM to GDM data flow design agreed upon, frozen (data contents permitted to change)
- major technical risk items retired (all hardware and firmware is working as expecred without mysteries and surprises, all problems are identified, investigated, resolved, solutions tested)
- stable operation of CDM-VX serial links in vertical slice system
- stable operation of GDM to CDM clock in vertical slice system
- stable operation of CDM to VX clock in vertical slice system
- vertical slice system assembled (1 VME crate, 2 VX, 2 CDM, 1 GDM, network, FEP, TSP, MIDAS)
ZZZ
ZZZ