DS-DM

From DaqWiki
Jump to navigation Jump to search

DS-DM

DarkSide-20k Global and Crate Data Manager board (GDM and CDM).

Global Data Manager (GDM):

  • clock distribution to CDM boards (including GPS/atomic clock source)
  • collection of trigger data from CDM boards, processing and distribution of trigger decision to CDM boards
  • run control

Crate Data Manager (CDM):

  • clock distribution from GDM to CAEN digitizers
  • receive trigger data from CAEN digitizers
  • send trigger data to GDM
  • run control and dead time control

Links

edev links:

Xilinx links:

Enclustra links:

Onboard hardware

  • jtag chain: arm_dap_0 0x5BA00477, xczu4_1 0x04721093
  • Mercury+ XU8 module: ME-XU8-4CG-1E-D11E-R2.1
    • Xilinx® Zynq Ultrascale+™ MPSoC XCZU4CG-1FBVB900E
    • DDR4 ECC SDRAM (PS) 2 GB
    • DDR4 SDRAM (PL) 1GB

Board schematics

Install Xilinx tools

  • install Vivado 2020.2
login at https://www.xilinx.com/myprofile.html
go to "Downloads"
go to archive,
find 2020.2
download Xilinx_Unified_2020.2_1118_1232_Lin64.bin
sh ./Xilinx_Unified_2020.2_1118_1232_Lin64.bin
banner window should open with spinner "downloading installation data"
"a newer version is available" -> say "continue"
next
"select install type" window:
provide email and password,
select "download image"
select directory /home/olchansk/Xilinx/Downloads/2020.2\
select "linux" and "full image"
next
download summary: space required 38.52 Gbytes
download
installation progress
downloading spinner, 16 M/s 47 minutes...
"download image has been created successfully". Ok.
check contents of /home/olchansk/Xilinx/Downloads/2020.2
ls -l /home/olchansk/Xilinx/Downloads/2020.2
total 67
drwxr-xr-x 2 olchansk users    9 Sep  1 16:22 bin
drwxr-xr-x 3 olchansk users   15 Sep  1 16:23 data
drwxr-xr-x 4 olchansk users    4 Sep  1 16:22 lib
drwxr-xr-x 2 olchansk users  644 Sep  1 16:22 payload
drwxr-xr-x 2 olchansk users    7 Sep  1 16:22 scripts
drwxr-xr-x 4 olchansk users    4 Sep  1 16:22 tps
-rwxr-xr-x 1 olchansk users 3256 Nov 18  2020 xsetup
daq13:2020.2$ 
./xsetup
spinned loading installation data
xilinx design tools 2022.1 now available -> say continue
"welcome" -> next
"select product" -> vivado -> next -> vivado hl system edition -> next
select devices: only zynq ultrascale+ mpsoc -> next
select destination: /opt/Xilinx (as root, mkdir /opt/Xilinx, chmod olchansk.users /opt/Xilinx)
install ...
complete
move /home/olchansk/Xilinx/Downloads/2020.2 to /daq/daqstore/olchansk/Xilinx/Downloads/
  • install petalinux 2020.2
./xsetup
"a newer version is available" -> say "continue"
next
"select product to install" -> select Petalinux (Linux only) -> next
"select destination directory" -> select "/opt/Xilinx" (disk space required 2.64 GB) -> next
"summary" -> install ...
error about missing /tmp/tmp-something files
"installation completed successfully" (hard to dismiss, "ok" button is partially cut-off)
done?
I think it failed, /opt/Xilinx/PetaLinux/2020.2/bin is empty except for petalinux-v2020.2-final-installer.run
try to run it by hand, same error about /tmp/tmp-something files. strange...
notice it complains about "truncate", which truncate finds ~/bin/truncate, get rid of it,
try again
now complains about missing texinfo and zlib1g:i386
apt install texinfo -> ok
apt install zlib1g:i386 -> installs bunch of gcc stuff -> ok
try again
reports "already installed" -> delete /opt/Xilinx/.xinstall/PetaLinux_2020.2/, delete entries in ~/.Xilinx/registry/installedSW.xml
try again
success
  • install vivado 2022.1 and petalinux 2022.1 - everything is pretty much the same

JTAG server

localhost:3121

Build firmware

  • git clone git@edev-group.triumf.ca:fw/exp/darkside/gcdm.git
  • #Makefile change VIVADO_SETTINGS_SCRIPT := /opt/Xilinx/Vivado/2022.1/settings64.sh
  • #. /opt/Xilinx/Vivado/2022.1/settings64.sh
  • . /opt/Xilinx/Vivado/2020.2/settings64.sh
  • make clean
  • make all_from_scratch
  • . /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
  • make petalinux_create
  • make petalinux_rebuild_new_hw_des
  • bomb out: The TMPDIR: /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp can't be located on nfs.
  • mkdir /tmp/build_tmp
  • rm -rf /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp/
  • ln -s /tmp/build_tmp /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp
  • try again
  • grinds, loads a whole bunch of packages...
  • finishes with desire to copy things to /tftpboot
  • make sdcard_cp_to wants to copy files from PetaLinux_GDM_CDM/images/linux/ to SD card

prepare bootable sd card

format the sd card

this only needs to be done once

  • become root
  • cd ~olchansk/git/ds-dm-gcdm
  • use "lsblk" to identify the SD card (should show as 8/16/32 GB block device)/ /dev/sdd in this case
  • make sdcard_format SDCARD_DEVICE=/dev/sdd
  • disconnect sd card, reconnect the sd card (to detect new partition tables, etc)

copy boot files to the sd card

  • as root: identify partition labels, run "blkid", should say "BOOT", "rootfs" and "data"
  • mount
mkdir /media/olchansk/BOOT
mkdir /media/olchansk/rootfs
mkdir /media/olchansk/data
mount -L BOOT /media/olchansk/BOOT
mount -L rootfs /media/olchansk/rootfs
mount -L data /media/olchansk/data
cp PetaLinux_GDM_CDM/images/linux/BOOT.BIN /media/olchansk/BOOT/
cp PetaLinux_GDM_CDM/images/linux/boot.scr /media/olchansk/BOOT/
cp PetaLinux_GDM_CDM/images/linux/image.ub /media/olchansk/BOOT/
umount /media/olchansk/BOOT
umount /media/olchansk/rootfs
umount /media/olchansk/data
eject /dev/sdd

DS-20K DAQ

Overview

DS-DM, GDM and CDM are key parts of the DS-20K DAQ system:

  • common clock distribution from external clock (atomic clock, GPS) to GDM to per-quadrant CDMs to VX digitizers
  • common trigger distribution from GDM internal algorithm or external input to all VX digitizers
  • run control: GDM, CDM, VX all start recording data at the same time (clock and timestamp reset)
  • collection of trigger data from VX digitizers to per-quadrant CDMs to GDM

Deliverables

  • hardware and firmware for GDM to CDM clock distribution
  • hardware and firmware for CDM to VX clock distribution
  • hardware and firmware for GDM external clock input (atomic clock or GPS)
  • hardware and firmware for CDM and VX serial communications (VX LVDS I/O connector)
  • firmware for run control (timestamp reset and sync): GDM to CDM to VX
  • firmware for common trigger distribution: GDM to CDM to VX
  • firmware for trigger data flow: VX to CDM to GDM
  • GDM MIDAS frontend: clock selector and monitoring, trigger and run control, GDM housekeeping
  • CDM MIDAS frontend: clock monitoring, CDM housekeeping

specific performance:

  • GDM external clock: 10 MHz GPS clock
  • GDM to CDM fiber link:
    • clock XXX MHz
    • link data rate: XXX Gbit/sec
    • CDM recovered clock: XXX MHz
    • CDM recovered clock jitter: XXX ns
    • phase alignment between CDMs: XXX ns
    • phase alignment between CDMs persists across reboots, power cycles, firmware updates
    • phase alignment between CDMs should be easy to measure
    • phase alignment between CDMs should be easy to recalibrate if hardware parts are replaced (DS-DM boards, fiber transceivers, fiber cables, etc)
    • data packet bandwidth: XXX Mbytes/sec
    • data packet latency: XXX clocks
    • data packet skew between CDMs: XXX clocks
  • CDM to VX clock:
    • clock: XXX MHz
    • jitter, all CDM clock outputs: XXX MHz
    • phase alignment between all CDM clock outputs: XXX ns
  • CDM to VX trigger:
    • TBD (use the VX "sync" input or VX LVDS I/O line or VX serial link packet)
  • CDM to VX serial link:
    • clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
    • bit rate: XXX bits/sec
    • latency: XXX link clocks
    • maximum skew between VXes: XXX ns
  • VX to CDM serial link:
    • clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
    • bit rate: XXX bits/sec
    • latency: XXX link clocks
    • maximum skew between VXes: XXX ns
  • timestamp reset:
    • maximum skew between VXes: XXX ns

Technical risk items

this refers to unexpected behaviour and performance of system components that prevents delivery of deliverables and prevents or negatively affects operation of the DS-20K DAQ or of the whole experiment.

(14-sep-2022, list is not sorted by any criterial: severity, probability, ease of investigation)

(stability of course is long term stability, across hours, days, weeks, months, years)

  • stability of Enclustra FPGA modules (crashes/year, failures to boot/year, flash corruption/year)
  • stability of GDM external clock PLL
  • stability of CDM recovered clock
  • unexpected failures or bit error rates in GDM-CDM fiber links
  • stability of CDM VX clock outputs (stability of clock cleaner chip)
  • stability of VX internal clock distribution (VX PLL lock loss events)
  • stability of VX CAEN base firmware (different versions of CAEN base firmware have different clock distribution behaviour)
  • strange things in CAEN base firmware (unexpected clocking of LVDS I/O, unexpected phase shifts between clocks, etc)

Milestones

AAA

ZZZ

ZZZ

Retrieved from "https://daq00.triumf.ca/DaqWiki/index.php?title=DS-DM&oldid=6591"