DaqWiki - User contributions [en]

Ubuntu

2024-11-05T23:24:56Z

Mwalczak: /* Ubuntu installer */

= Prerequisites =

* before setting up new machine run memory test
* prepare flash drive with free version of memtest86: https://www.memtest86.com
* test boot from flash drive, test takes ~ few hours
* test will end with summary page, if passed continue with Ubuntu
* number that might be worth noting is memory latency

= Ubuntu version =

<pre>
lsb_release -a
uname -a
</pre>

= Ubuntu installer =

* updated for Ububtu LTS 20.04.01, 22.04.1, 24.04 (only minor differences)

* download the latest Ubuntu LTS desktop installer iso image
* dd the image to a USB key
* power down, disconnect all disks (all HDDs, all SSDs, all M.2)
* connect the SSD to be used as system disk
* if system will use mirrored SSDs (using ZFS mirror), leave second SSD disconnected, we will activate it later
* power up
* boot from USB key in legacy mode or UEFI mode (select this in the BIOS boot menu - F8 for ASUS, F11 for Supermicro)
* follow the instruction:
* "try ubuntu or install ubuntu" - choose "install"
* select language - accept default
* "updates and other software" - accept default settings ("normal install")
* "installation type" - select "advanced features" and "experimental: use ZFS"
* accept partition choice
* "where are you?" - select "Vancouver" (PST time zone)
* "who are you?" - leave all fields blank, except "username" set to "wheel", "password" set to the root password. hostname will be set later after configuring the network
* don't install third party sw
* installation runs in a few minutes, when finished, reboot
* login as user wheel
* answer annouying questions:
* "livepatch" - say "next"
* "help improve" - select "do not send", say "next"
* "privacy" - leave "location" as "off", say "next"
* "ready to go", say "done"
* right-click on the desktop, say "open in terminal", a shell will open
* say "sudo /bin/bash", enter the root password, you now have the root shell
* run nm-connection-editor to configure the network. use netmask 255.255.224.0, gateway 142.90.100.18, DNS 142.90.100.19, search path "triumf.ca"
* after network is up (can ping ladd00), continue with post-installation steps below

= Install instructions =

== prepare ==
<pre>
apt update
apt upgrade
</pre>

== install ssh ==
<pre>
apt install ssh
</pre>

== install git/scripts ==
<pre>
apt -y install git
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd scripts
git pull
</pre>

* if needed, update git/scripts repository from ladd00 to daq00:
* git remote -v ### if it says daq00, we are good
* git remote set-url origin https://daq00.triumf.ca/~olchansk/git/scripts.git
* git pull ### check that it works

== configure hostname ==
<pre>
vi /etc/hostname
</pre>

== disable swap ==

ubuntu installer creates a 2 GB swap partition, not useful
on 32-64 GB machine, disable it:
<pre>
vi /etc/fstab ### comment out the "swap" line
</pre>

== maybe reboot ==

this is a good point to reboot the machine to boot
the latest kernel and to set the correct hostname

== install etckeeper ==

keep contents of /etc in a git repository:

<pre>
apt -y install etckeeper
</pre>

== set timezone ==
<pre>
timedatectl list-timezones | grep -i vancouver
timedatectl set-timezone America/Vancouver
</pre>

== install time synchronization ==
<pre>
apt -y install chrony
#echo server time1.triumf.ca iburst >> /etc/chrony/chrony.conf
#echo server time2.triumf.ca iburst >> /etc/chrony/chrony.conf
#echo server time3.triumf.ca iburst >> /etc/chrony/chrony.conf
cd ~/git/scripts
git pull
cd ~
cp ~/git/scripts/etc/triumf.sources /etc/chrony/sources.d/
systemctl disable systemd-timesyncd.service
systemctl stop systemd-timesyncd.service
systemctl disable ntp
systemctl stop ntp
systemctl enable chrony
systemctl restart chrony
chronyc sources
chronyc tracking
</pre>

NOTE1: if time1, time2, time3 are already listed in /etc/crony/chrony.conf, please remove them and restart chrony.

NOTE2: if time1, time2, time3 are not listed in "chronyc tracking" or if they are not selected by "chronyc tracking", check that /etc/crony/chrony.conf contains "sourcedir /etc/chrony/sources.d", see NOTE4.

NOTE3: read https://chrony-project.org/faq.html#_should_i_prefer_chrony_over_timesyncd_if_i_do_not_need_to_run_a_server

NOTE4: update very old chrony config file, remove chrony, than install it from scratch as above
<pre>
grep sourcedir /etc/chrony/chrony.conf ### if we have it, we are good
apt remove chrony
apt purge chrony
</pre>

== reenable systemd-timesyncd ==

ONLY IF CHRONY DOES NOT WORK

To configure systemd-timesyncd, set "NTP=" in /etc/systemd/timesyncd.conf

<pre>
apt remove chrony
cat /etc/systemd/timesyncd.conf
systemctl enable systemd-timesyncd.service
systemctl restart systemd-timesyncd.service
systemctl status systemd-timesyncd.service
timedatectl status
timedatectl timesync-status
</pre>

== enable outgoing email (debian 11) ==

this is different from ubuntu 20. it uses /etc/mailname and it hardwires the hostname into main.cf.

== enable outgoing email ==

we have an unusual email configuration. outgoing email should work to deliver error messages, notices, etc. incoming email is disabled, we do not receive email for local users.

this causes problems with TRIUMF smtp server. if our message cannot be delivered (wrong email address or receipient computer is turned off), TRIUMF smtp server will generate a delivery failure notification email and try to send it to the "from" address of the failed message. but the "from" address does not receive any email, so another delivery failure notification email is generated and an attempt to deliver it. which again fails, rinse and repeat.

as solution, kray created a special rule, email from scrap.triumf.ca does not generate delivery failure notices. failed messages sit in the queue for 5 days, then they are deleted. (K.O. - confirmed with kray 3jan2024).

to make this work we use the msmtp MTA package.

<pre>
cd ~
apt -y remove postfix
apt -y purge postfix # remove old config files
apt -y install mailutils msmtp msmtp-mta # say "no" to apparmor support
apt -y install bsd-mailx
cd ~/git/scripts/etc
git pull
/bin/cp -fv aliases /etc/aliases
/bin/cp -fv msmtprc /etc/msmtprc
/bin/rm -vf ~root/.forward
/bin/rm -vf /etc/mailname
Mail root
Subject: test
test
^D
CC: <CR>
</pre>

== enable outgoing email (postfix) ==

THIS IS OBSOLETE!!!

* TRIUMF: use smtp.triumf.ca
* CERN: use cernmx.cern.ch

<pre>
apt install postfix ### select "satellite system", enter full hostname "xxx.triumf.ca", enter "smtp.triumf.ca"
apt install mailutils
dpkg-reconfigure postfix ### (if postfix already installed)
</pre>
<pre>
echo olchansk@triumf.ca lindner@triumf.ca bsmith@triumf.ca >> ~root/.forward
mailx root
test
^D
</pre>

== enable ping for all users (debian 11) ==

Without this tweak, Debian will report "operation not permitted" if a user tries to ping somewhere.

<pre>
echo 'net.ipv4.ping_group_range = 0 1000' > /etc/sysctl.d/99-ping.conf
</pre>

== disable apparmor ==

On NFS-Root network booted machines!

If "man man" returns "permission denied" and syslog reports apparmor "sendmsg DENIED" errors, disable apparmor. This is supposedly fixed in kernel 6.0 and later (to be confirmed), see https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/1784499

Disable apparmor, see https://ubuntu.com/server/docs/security-apparmor

This takes effect after a reboot.

<pre>
systemctl stop apparmor.service
systemctl disable apparmor.service
</pre>

== install missing packages ==

(apt eats terminal input, even the "yes |" trick does not quite work,
repeat the following commands until they report that everything
is installed)

<pre>
yes | apt -y install ssh tcsh ethtool ncat rsync strace net-tools traceroute time minicom screen git lsof debsums tmux iptables telnet
yes | apt -y install sysstat smartmontools lm-sensors
yes | apt -y install lsb-release
apt -y install vim # in addition to default vim-tiny, requested by IRIS
apt -y install tcl
apt -y install pax rpm alien ### package converter tools
yes | apt -y install flex bison
yes | apt -y install neofetch
yes | apt -y install snmp snmp-mibs-downloader
yes | apt -y install git subversion g++ gfortran cmake doxygen
yes | apt -y install curl libcurl4 libcurl4-openssl-dev
yes | apt -y install mariadb-client ### mysql client
yes | apt -y install libz-dev libzstd-dev sqlite3 libsqlite3-dev unixodbc-dev
yes | apt -y install libssl-dev
yes | apt -y install emacs xemacs21 joe
yes | apt -y install gnuplot dos2unix
yes | apt -y install mutt bsd-mailx # email clients
yes | apt -y install liblz4-tool pbzip2
yes | apt -y install libc6-dev-i386 # otherwise no /usr/include/sys/types.h
yes | apt -y install libreadline-dev
yes | apt -y install ubuntu-mate-themes
yes | apt -y install libmotif-dev libxmu-dev
yes | apt -y install libusb-dev libusb-1.0-0-dev
yes | apt -y install i2c-tools libi2c-dev libi2c0
yes | apt -y install xfig gsfonts-x11 gsfonts-other # install fonts for xfig
yes | apt -y install libjson-perl
yes | apt -y install libgsl-dev # additional GNU Scientific Library
yes | apt -y install qt5-default # Qt development
yes | apt -y install python3-full python3-dev python3-dbg python3-pip ### for pyROOT
yes | apt -y install imagemagick imagemagick-common ckeditor # for elog
yes | apt -y install libjpeg-dev libjpeg-progs libjpeg-tools
yes | apt -y install linux-tools-common linux-tools-generic # cpupower frequency-info
yes | apt -y install rdesktop remmina remmina-plugin"*" # requested by POL
yes | apt -y install nlohmann-json3-dev # required to build MIDAS with ROOT 6.30 on Ubuntu-22
apt -y install dpkg-dev cmake g++ gcc binutils libx11-dev libxpm-dev libxft-dev libxext-dev python3 libssl-dev libafterimage0 # from https://root.cern/install/dependencies/
apt -y install gfortran libpcre3-dev xlibmesa-glu-dev libglew-dev libftgl-dev libmysqlclient-dev libfftw3-dev libcfitsio-dev graphviz-dev libldap2-dev python3-dev python3-numpy libxml2-dev libkrb5-dev libgsl0-dev qtwebengine5-dev nlohmann-json3-dev libtbb-dev libavahi-compat-libdnssd-dev # from https://root.cern/install/dependencies/
apt -y install libvdt-dev # for ROOT 6.32 on Ubuntu-24
apt -y install u-boot-tools # for Xilinx petalinux
#apt -y install linux-headers-generic # to build linux kernel drivers
</pre>

Ubuntu LTS 20.04:
<pre>
yes | apt -y install linux-image-generic-hwe-20.04 linux-tools-virtual-hwe-20.04 # enable linux 5.11 series kernel
</pre>

Ubuntu LTS 22.04:
<pre>
apt -y install linux-generic-hwe-22.04 # enable linux 6.2.0 series kernel
</pre>

Ubuntu LTS 24.04:
<pre>
apt -y install linux-generic-hwe-24.04 # enable linux 6.8.0 series kernel
</pre>

== disable swap (debian 11) ==

* on 64 GB RAM machines swap is not useful
* on machines booted from network (NFS-ROOT), swap does not work
* on machines running from flash (RPi, etc), flash is too slow for useful swap
* swap configured by linux installers invariably has wrong size and is not useful

<pre>
systemctl disable dphys-swapfile
systemctl stop dphys-swapfile
dphys-swapfile uninstall
</pre>

== configure DNS ==

<pre>
cd ~/git/scripts
git pull
mkdir /etc/systemd/resolved.conf.d
cp etc/resolved-triumf.conf /etc/systemd/resolved.conf.d/
systemctl restart systemd-resolved
resolvectl
#systemd-analyze cat-config systemd/resolved.conf
</pre>

== install ganglia ==

<pre>
apt -y install ganglia-monitor
cd ~root/git/scripts/ganglia
git pull
make install
./ganglia-all.perl
</pre>

fix gmond start before network is ready:

<pre>
mkdir /etc/systemd/system/ganglia-monitor.service.d
echo -e "[Unit]\nAfter=network-online.target\n" > /etc/systemd/system/ganglia-monitor.service.d/local.conf
systemctl daemon-reload
systemctl cat ganglia-monitor.service
</pre>

== install gonodeinfo ==

* go to https://bitbucket.org/dd1/gonodeinfo follow instructions:
<pre>
yes | apt-get -y install golang
mkdir ~/git
cd ~/git
#git clone https://bitbucket.org/dd1/gonodeinfo.git
git clone https://daq00.triumf.ca/~olchansk/git/gonodeinfo.git
cd gonodeinfo
git remote set-url origin https://daq00.triumf.ca/~olchansk/git/gonodeinfo.git
git pull
make
make install # install gonodeinfo agent
cd ~ # this is important
</pre>
* edit /etc/gonodeinfo.conf
* change "Description", "Location", "User" and "Administrator" as appropriate (or delete them)
* change "Servers" to read: Servers: daq00.triumf.ca:8601
* run "gonodeinfo -v"
* if error is "connection refused". go to the nodeinfo server to add this client to the access control list:
* on the gonodeinfo server: run /opt/gonodeinfo/gonodereceive.exe -a daq13
* try gonodeinfo again, there should be no error
* on the gonodeinfo server: run gonodereport, look at the web pages, the new machine should be listed now

== install fonts for EPICS ==

* apt install xfonts-100dpi xfonts-75dpi
* restart Xorg (i.e. "killall Xorg", this will log you out from the console)
* xlsfonts | grep -i helvetica ### should show fonts with different sizes, not just size 0 (scalable)

== install libz.so.1 for CentOS compatibility ==

KO - confirm which versions on quartus need this.

<pre>
yes | apt-get -y install zlib1g
yes | apt-get -y install zlib1g:i386 libc6:i386 libgcc1:i386 gcc-6-base:i386
</pre>

== install libpng12.so.0 for Quartus compatibility ==

(does not work anymore!!!)

<pre>
wget http://ftp.ca.debian.org/debian/pool/main/libp/libpng/libpng12-0_1.2.50-2+deb8u2_amd64.deb
dpkg --install libpng12-0_1.2.50-2+deb8u2_amd64.deb
</pre>

== install libpng12.so.0 for Quartus 13.0sp1 ==

<pre>
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0.50.0
/bin/cp -pv libpng12.so.0 libpng12.so.0.50.0 /lib/x86_64-linux-gnu/
</pre>

== install packages for Xilinx ==

ubuntu LTS 22.04 vivado 2020.1

<pre>
apt install autoconf libtool
apt install libtinfo5
apt install texinfo
apt install zlib1g:i386
</pre>

== install packages for building ROOT ==

<pre>
apt -y install libx11-dev libxpm-dev libxft-dev libxext-dev libpng-dev libjpeg-dev xlibmesa-glu-dev libxml2-dev libgsl-dev cmake
</pre>

== install 32-bit libraries for PHYSICA ==

these instructions are for running 32-bit physica executable built for SL6 on ubuntu LTS 20.04

install physica sources (cannot build, do not have g77)

<pre>
cd ~/packages
git clone https://bitbucket.org/ttriumfdaq/physica.git
</pre>

install 32-bit libraries using ubuntu package manager:

<pre>
apt install lib32z1 # libz.so
</pre>

copy 32-bit SL6 shared libraries to /lib32

<pre>
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libX11.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libgd.so.2 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libpng12.so.0 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libreadline.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libncurses.so.5 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libg2c.so.0 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libxcb.so.1 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libXpm.so.4 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libjpeg.so.62 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libfontconfig.so.1 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libfreetype.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libtinfo.so.5 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libXau.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libexpat.so.1 /lib32/
</pre>

ldd should report:

<pre>
trinatdaq:trinat> ldd /usr/local/physica/physica.exe
linux-gate.so.1 (0xf7fa2000)
libX11.so.6 => /lib32/libX11.so.6 (0xf7e43000)
libgd.so.2 => /lib32/libgd.so.2 (0xf7dfe000)
libpng12.so.0 => /lib32/libpng12.so.0 (0xf7dd6000)
libz.so.1 => /lib32/libz.so.1 (0xf7db8000)
libreadline.so.6 => /lib32/libreadline.so.6 (0xf7d7e000)
libncurses.so.5 => /lib32/libncurses.so.5 (0xf7d5b000)
libg2c.so.0 => /lib32/libg2c.so.0 (0xf7d3d000)
libm.so.6 => /lib32/libm.so.6 (0xf7c39000)
libgcc_s.so.1 => /lib32/libgcc_s.so.1 (0xf7c1a000)
libc.so.6 => /lib32/libc.so.6 (0xf7a2f000)
libxcb.so.1 => /lib32/libxcb.so.1 (0xf7a05000)
libdl.so.2 => /lib32/libdl.so.2 (0xf79ff000)
libXpm.so.4 => /lib32/libXpm.so.4 (0xf79ee000)
libjpeg.so.62 => /lib32/libjpeg.so.62 (0xf7997000)
libfontconfig.so.1 => /lib32/libfontconfig.so.1 (0xf7962000)
libfreetype.so.6 => /lib32/libfreetype.so.6 (0xf78c9000)
libtinfo.so.5 => /lib32/libtinfo.so.5 (0xf78b0000)
/lib/ld-linux.so.2 (0xf7fa4000)
libXau.so.6 => /lib32/libXau.so.6 (0xf78ad000)
libexpat.so.1 => /lib32/libexpat.so.1 (0xf7885000)
trinatdaq:trinat>
</pre>

set login environment:

<pre>
setenv TRIUMF_FONTS $HOME/packages/physica/fonts
setenv PHYSICA_DIR $HOME/packages/physica
alias physica $PHYSICA_DIR/physica-SL6-32
</pre>

test:

<pre>
cd ~/packages/physica
physica
@rangauss.pcm
</pre>

== install wine ==

As far as I know, only needed for BNMR/BNQR

<pre>
apt install wine winetricks
</pre>

== install lightdm ==

unlike the default gdm login manager, lightdm shows the machine hostname and does not require an extra mouse click to swicth from screen saver to login mode.

<pre>
apt -y install lightdm
# select lightdm
</pre>

== install desktop environments ==

note: default display manager and default desktop are deficient, please do not skip this step.

note: if apt asks to choose the display manager, select "lightdm"

note: KO - I recommend the "MATE" desktop.

note: you will have to cut-and-paste this several times because "apt" eats commands, even with "-y" and even piped from "yes".

<pre>
# install MATE desktop
DEBIAN_FRONTEND=noninteractive apt -y install ubuntu-mate-core ubuntu-mate-desktop ubuntu-mate-themes
# install Cinnamon desktop
DEBIAN_FRONTEND=noninteractive apt -y install cinnamon
# install KDE desktop
DEBIAN_FRONTEND=noninteractive apt -y install kubuntu-desktop
# install Lxqt desktop
DEBIAN_FRONTEND=noninteractive apt -y install lxqt
# install Xfce4 desktop
DEBIAN_FRONTEND=noninteractive apt -y install xfce4
</pre>

== install ROOT ==

Please install ROOT per instructions at https://root.cern.ch.

NOTE1: The ROOT package available from Ubuntu repositories is severely out of date and cannot be used with MIDAS and ROOTANA. ### DO NOT DO THIS! apt-get install root-system

NOTE2: as of 2017-Jan-09, ROOT binary kits for Ubuntu do not work (use GCC 5 instead of GCC6), build from source instead.

== Install x2go ==

<pre>
apt-get update
apt-get install x2goserver x2goserver-xsession
</pre>

== enable root login from ladd00/daq00 ==
<pre>
ssh localhost
CTRL-C
/bin/cp ~root/git/scripts/etc/authorized_keys ~root/.ssh/
</pre>

== disable ssh access from outside of TRIUMF ==

to stop ssh login spam, disable ssh access from outside of TRIUMF. this can be done by requesting a firewall block through the helpdesk or by local firewall rule:

<pre>
echo iptables -I INPUT ! -s 142.90.0.0/255.255.0.0 -p tcp --dport 22 -j REJECT >> /etc/rc.local
/etc/rc.local
</pre>

== install smart-status ==
<pre>
ln -s ~/git/scripts/smart-status/smart-status.perl ~root/
</pre>

== enable boot menu and boot messages ==

This will enable the grub menu (with a 10 sec timeout) and
replace black screen with exciting linux boot messages.

* emacs -nw /etc/default/grub
<pre>
GRUB_DEFAULT=0
#GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="vga=769 video=640x480"
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX=""
#GRUB_GFXMODE=640x480
</pre>
* update grub config:
<pre>
grub-mkconfig -o /boot/grub/grub.cfg
</pre>

== reboot ==

this completes installation of the base system.

following sections modify basic ubuntu to fix known problems and to enable special stuff.

= Enable automatic updates =

<pre>
apt install unattended-upgrades
cd ~/git/scripts
git pull
/bin/cp -v etc/99apt-conf-ko /etc/apt/apt.conf.d/
apt-config dump | grep Unattended
</pre>

Following is obsolete:

* emacs -nw /etc/apt/apt.conf.d/50unattended-upgrades
** uncomment in Allowed-Origins "-security" and "-updates"
** add in Allowed-Origins: "Google LLC:stable";
** uncomment/add: "Unattended-Upgrade::Mail "root";
* emacs -nw /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
</pre>
* test: unattended-upgrade --dry-run -v

NOTE: update-on-shutdown is disabled.

NOTE: there is no update-on-boot, but:

NOTE: if machine was off for a long time, the systemd update timer would have expired and it will fire soon after reboot, causing an automatic update run. this is unwanted, and there is no fix or workaround for it. K.O. June-2023.

= Fix bpool is full (obsolete) =

THIS IS CAUSED BY OBSOLETE PACKAGE zsys. PLEASE: apt remove zsys

!!! only if ROOT on ZFS !!!

There is an error in the zsys package that causes bpool to run out of space,
see [[#Ubuntu zsys]] for more details.

To fix:
<pre>
cd ~/git/scripts
git pull
cp etc/zsys.conf /etc/
zsysctl service reload
zsysctl service gc
zpool list bpool
zfs list bpool
df /boot
</pre>

= IPMI instructions =

IPMI is the board management hardware on Supermicro and other server motherboards. This includes hardware sensors - fan rotation speed, temperatures and power supply voltages.

<pre>
apt-get install ipmitool
systemctl enable ipmievd
systemctl restart ipmievd
</pre>

Run:
* ipmitool sel list ### event list
* ipmitool sel elist ### event list
* ipmitool sel clear ### clear event list (if it becomes full)
* ipmitool sensor ### report hardware sensors

= move /home/wheel =

note: this MUST be done if ZFS root and NIS/autofs with /home.

Default location of wheel's home directory will collide with autofs /home, it has to be moved,
for example to /wheel.

<pre>
# logout from the wheel user
# go to another computer
ssh root@daqubuntuxxx
zfs list | grep wheel ### identify zfs name wheel_xxxxxx
#zfs set mountpoint=/wheel rpool/USERDATA/wheel_hm8fzh
zfs set mountpoint=/wheel `zfs list | grep wheel | cut -f1 -d" "`
zfs list | grep wheel
emacs -nw /etc/passwd ### change wheel's home directory from /home/wheel to /wheel
su - wheel ### check that user wheel still works
</pre>

This will break wheel's ability to run snap programs, such as firefox, install chrome as listed below.

= enable NIS (ubuntu 22.04, 24.04, debian 11, 12) =

<pre>
apt -y install rpcbind nis
echo DAQ-NIS >> /etc/defaultdomain
echo ypserver daq00.triumf.ca >> /etc/yp.conf
systemctl enable ypbind.service
systemctl restart ypbind.service
systemctl status ypbind.service
ypwhich -m
</pre>

enable ypserv:

<pre>
sed -i s/NISSERVER=false/NISSERVER=slave/ /etc/default/nis
/usr/lib/yp/ypinit -s daq00
echo ypserver localhost >> /etc/yp.conf
sed -i "s/ypserver .*/ypserver localhost/" /etc/yp.conf
systemctl enable ypserv
systemctl restart ypserv
systemctl restart ypbind
</pre>

edit /etc/nsswitch.conf to read:

<pre>
# begin get data from nis
passwd: files nis
group: files nis
shadow: files nis
automount: files nis
netgroup: files nis
# end get data from nis

#passwd: ...
#group: ...
#shadow: ...

#netgroup: ...
#automount: ...
</pre>

enable hourly update of nis maps:

<pre>
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>

If this is a new machine, then on the master NIS node (daq00), add this new node to /etc/netgroup, and update NIS maps (cd /var/yp; make)

= enable NIS (ubuntu 20.04) =

* apt-get -y install portmap nis ### will ask for NIS domain (DAQ-NIS)
* dpkg-reconfigure nis ### reconfigure if already installed
* ypwhich -m
* edit /etc/default/nis
** set "NISSERVER=slave"
** Ubuntu LTS 20.04, check that "YPBINDARGS=" is blank, remove "-no-dbus" if it is there
* #edit /etc/yp.conf, comment-out everything, add "domain DAQ-NIS server localhost"
* edit /etc/yp.conf, comment-out everything, add "ypserver localhost"
* /usr/lib/yp/ypinit -s daq00
* systemctl enable nis
* systemctl restart nis
* ypwhich
* ypwhich -m
* ypcat -k passwd
* vi /etc/nsswitch.conf ### add the automount line, modify the passwd, group and shadow lines to read this:
<pre>
# begin get data from nis
passwd: files nis
group: files nis
shadow: files nis
automount: files nis
netgroup: files nis
# end get data from nis
</pre>
* enable hourly update of NIS maps
<pre>
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>
* ### NOT NEEDED sudo vi /etc/idmapd.conf ### add line: "Domain = triumf.ca"

= enable autofs =

<pre>
apt -y install autofs
systemctl enable autofs
systemctl restart autofs
ls -l /home/olchansk ### test autofs, check file owner is correct
</pre>

= enable NFS server =

<pre>
apt install nfs-kernel-server
#edit /etc/exports
systemctl enable nfs-server
systemctl restart nfs-server
</pre>

= NIS master =

notes for setting up the NIS master

== wheel user ==

"wheel" is the default administrative user. We do not want it's password exported to NIS (encrypted password hash is world visible) and we do not want it's home directory exported to NFS (~wheel/.ssh is world visible and potentially writable: anybody can change ~wheel/.ssh/authorized_keys).

* move wheel's home directory from /home/wheel to /wheel (see special section about this)
* change wheel's UID and GID from 1000 to a value below MINUID in /var/yp/Makefile

== coherent uids ==

we do not want system accounts defined in /etc/passwd of the NIS master
to be included in the NIS map "passwd". this causes trouble on NIS clients
where newly installed packages fail to create local system users because same
user already exists in NIS.

This is controlled by MINUID in /var/yp/Makefile.

Historical TRIUMF uids start from around 200, but several clusters do not have any historic TRIUMF uids below 500 and MINUID is set to:
* DAQ-NIS: MINUID=200
* ISAC-NIS: MINUID=500
* TITAN-NIS: MINUID=500
* MUSR-NIS: MINUID=500
* TIG-NIS: MINUID=500 (100 on SL6 mother8pi)

Ubuntu 20 has two programs to create users:
* adduser - creates new users with UID 1000 and up as specified in /etc/adduser.conf. No problems here.
* adduser --system - creates new system users with UID 100 and up as specified in /etc/adduser.conf. No problems here.
* useradd - creates new users with UID 1000 and up as specified in /etc/login.defs. No problems here.
* useradd --system - creates new system users with UID 999 and down (read "man useradd", section at the end about SYS_UID_MAX). This collides with NIS MINUID, these system users will be included in the NIS map and cause trouble.

This problem cannot be fixed, SYS_UID_MIN, SYS_UID_MAX and UID_MIN in /etc/login.defs do not seem
to have any effect on UIDs chosen by "useradd --system". (tested on Ubuntu LTS 20.04).

So far only these system accounts seem to be affected by this:
* systemd-coredump
* ganglia

To fix:
* run "sort -r -n -t: -k3 /etc/passwd" to identify the last unused system user uid (range 100..200)
* run "sort -r -n -t: -k3 /etc/group" to identify the last unused system user gid (range 100.200)
* systemd-coredump: manually change UID and GID (package systemd-coredump is usually not installed)
* ganglia: same thing, then change ownership on all ganglia files.

Also read systemd author's opinion on system vs user UIDs:
https://github.com/systemd/systemd/issues/4850#issuecomment-265698275

= Fix systemd-logind NIS breakage =

!!! THIS IS NOT NEEDED FOR UBUNTU LTS 20.04 !!!

there is a delay in ssh logins for normal users. "ssh -v" shows the delay is after "pledge...". this
fix removes the delay.

systemd developers think that we should not use NIS and made sure there are
problems if we do. To give them credit, they do offer a workaround. Read this:
https://github.com/poettering/systemd/commit/695fe4078f0df6564a1be1c4a6a9e8a640d23b67

<pre>
mkdir /etc/systemd/system/systemd-logind.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-logind.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-logind.service
</pre>

= Fix systemd-udevd NIS breakage =

see same problem as above with udev getting stuck. ubuntu lts 20.04.

<pre>
mkdir /etc/systemd/system/systemd-udevd.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-udevd.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-udevd.service
</pre>

= Configure USB device permissions =

Configure USB device permissions for user access to USB-serial devices, Altera USB Blaster, etc.

* create file /etc/udev/rules.d/99-usb-chmod.rules with this contents:

<pre>
emacs -nw /etc/udev/rules.d/99-usb-chmod.rules
ACTION=="add", SUBSYSTEM=="usbmisc", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /dev/%c"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /proc/%c"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVICE}"
ACTION=="add", ENV{PHYSDEVBUS}=="usb-serial", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVPATH}=="/class/tty/ttyS*", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyUSB*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyACM*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyS*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", DEVPATH=="*video*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
</pre>

* reload udev rules: udevadm control --reload-rules
* apply new permissions: udevadm trigger --action=add
* watch udev activity: udevadm monitor -p

= Configure lightdm display manager =

* enable it
<pre>
echo lightdm | dpkg-reconfigure -fteletype lightdm
systemctl disable gdm
systemctl disable sddm
systemctl enable lightdm
</pre>

* make the MATE desktop as default
<pre>
cd ~root/git/scripts/
git pull
/bin/cp -v etc/lightdm_default_mate.conf /etc/lightdm/lightdm.conf.d/
</pre>

* enable login by NIS users
<pre>
/bin/cp -v etc/lightdm_enable_nis_login.conf /etc/lightdm/lightdm.conf.d/
</pre>

* restart lightdm
<pre>
systemctl stop gdm
systemctl restart lightdm
</pre>

= Install libpng12.so.0 =

Quartus 16 needs libpng12:

<pre>
wget http://mirrors.kernel.org/ubuntu/pool/main/libp/libpng/libpng12-0_1.2.54-1ubuntu1_amd64.deb
dpkg --install libpng12-0_1.2.54-1ubuntu1_amd64.deb
</pre>

= Install google-chrome =

<pre>
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
dpkg -i google-chrome-stable_current_amd64.deb
</pre>

confirm autoupdate is enabled, observe dl.google.com is present in the list of repositories:
<pre>
apt update
...
Get:5 https://dl.google.com/linux/chrome/deb stable/main amd64 Packages [1,094 B]
...
</pre>

FOLLOWING IS OBSOLETE:

Instructions from here:
https://www.ubuntuupdates.org/ppa/google_chrome?dist=stable

<pre>
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-tmp.list'
apt update
apt install google-chrome-stable
/bin/rm -f /etc/apt/sources.list.d/google-tmp.list
</pre>

= Install amanda client =

ONLY ONE MACHINES THAT HOST HOME DIRECTORIES

* apt install amanda-client
* edit /etc/amandahosts
<pre>
amanda.triumf.ca amanda amdump
</pre>
* check permissions on /etc/amandahosts:
<pre>
root@daq00:/var/log/amanda# ls -l /etc/amandahosts
-rw------- 1 backup backup 49 Jan 27 10:48 /etc/amandahosts
</pre>
* fix if needed: chown backup.backup /etc/amandahosts; chmod a= /etc/amandahosts; chmod u=wr /etc/amandahosts
* edit /etc/amanda-security.conf, add this line:
<pre>
runtar:gnutar_path=/usr/bin/tar
</pre>

On the amanda machine:

* in amanda disklist, use dump type "bsdtcp-comp-user-tar"
* su - amanda and run amcheck -c daily daq00
<pre>
-bash-4.1$ amcheck -c daily daq00

Amanda Backup Client Hosts Check
--------------------------------
Client check: 1 host checked in 0.092 seconds. 0 problems found.

(brought to you by Amanda 3.3.7p1.git.685ff76d)
</pre>

= Enable rc.local =

For reasons unknown, Ubuntu LTS 20.04 does not enable /etc/rc.local. Do this:

<pre>
cd ~/git/scripts
git pull
cp -n -v etc/rc.local /etc/
chmod a+rx /etc/rc.local
cp etc/rc-local.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable rc-local
systemctl start rc-local
systemctl status rc-local
</pre>

= Remove unwanted packages =

<pre>
apt remove bash-completion # broken, adds unwanted "\" if "ls -l $ROOTSYS/<tab>"
apt remove zsys # broken, do not use
apt remove sddm # login manager
apt remove avahi-daemon avahi-autoipd # not sure what it does, observed using 100% CPU
apt remove modemmanager # probes all serial ports to see if it's a modem
</pre>

= Disable unwanted services =

<pre>
systemctl disable mpd
systemctl disable snapd
systemctl disable ModemManager
systemctl --global mask tracker-extract-3.service
systemctl --global mask tracker-miner-fs-3.service
systemctl daemon-reload
</pre>

= Disable sleep and suspend =

note: we see some computers randomly shutdown or go to sleep, log files indicates the "sleep" or "suspend" button was pushed by user, but no such buttons actually exist. this is the fix for this:

<pre>
systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target systemd-suspend.service systemd-hybrid-sleep.service
</pre>

= Enable crontab @reboot for MIDAS =

startup scripts have a bug - cron @reboot entries for normal users can run before autofs is ready, so if the home directory is on autofs/NFS, it cannot be accessed and the cron job fails. If MIDAS is supposed to be started by cron @reboot, it will not start (there *will* be an error message in /var/log/cron).

<pre>
mkdir /etc/systemd/system/cron.service.d
echo -e "[Unit]\nAfter=ypbind.service autofs.service\n" > /etc/systemd/system/cron.service.d/local.conf
systemctl daemon-reload
systemctl cat cron.service
</pre>

Explore the systemd dependency tree using "systemctl list-dependencies" maybe with "--all".

Visualize the exact boot sequence from previous boot: "systemd-analyze plot > xxx.svg", look at the svg file using a web browser.

Crontab entry to start midas: (install in the midas user crontab, not root crontab)

<pre>
su - midasuser
crontab -l
#@reboot /bin/bash -l -c "/home/trinat/bin/start-daq-applications"
#@reboot /bin/tcsh -c "/home/trinat/bin/start-daq-applications"
</pre>

= Install apache httpd proxy for midas and elog =

This will configure the HTTPS/SSL certificate using "certbot" and "letsencrypt" and configure an HTTPS web server using apache2.

First, configure apache2:

* execute these commands:
<pre>
apt -y install apache2
cd /etc/apache2
</pre>
* create new file conf-available/ssl-daq14.conf # use actual hostname instead of daq14
<pre>
SSLSessionCache shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout 300
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
SSLCryptoDevice builtin
</pre>
* create new file sites-available/daq14-ssl.conf # use actual hostname instead of daq14
<pre>
<IfModule mod_ssl.c>
<VirtualHost *:443>
ServerName daq14.triumf.ca
DocumentRoot /var/www/html
ErrorLog /var/log/apache2/daq14.log
SSLEngine on
# note SSLProtocol, SSLCipherSuite and some other settings are overwritten by /etc/letsencrypt/options-ssl-apache.conf
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SEED:!IDEA:!RC4
## use port specified in elogd.cfg
#ProxyPass /elog/ http://localhost:8082/ retry=1
## use mhttpd port
#ProxyPass / http://localhost:8080/ retry=1
Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"
<Location />
SSLRequireSSL
AuthType Basic
AuthName "DAQ password protected site"
Require valid-user
# create password file: touch /etc/apache2/htpasswd
# to add new user or change password: htpasswd /etc/apache2/htpasswd username
AuthUserFile /etc/apache2/htpasswd
</Location>
</VirtualHost>
</IfModule>
</pre>
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* enable ssl module
* enable new configurations
<pre>
a2enmod ssl
a2enmod headers
a2enmod proxy
a2enmod proxy_http
a2enconf ssl-daq14
a2ensite daq14-ssl
</pre>
* disable default ssl sites
<pre>
a2dissite 000-default-le-ssl
a2dissite 000-default
ls -l /etc/apache2/sites-enabled/ ### should show only daq14-ssl.conf
</pre>
* check that there are no syntax problems
<pre>
apache2ctl configtest
</pre>
* enable and start apache2:
<pre>
systemctl enable apache2
systemctl restart apache2
systemctl status apache2
</pre>
* apache2 may fail to start, look in /var/log/apache2/error.log and /var/log/apache2/daq14.log
* if it says "Failed to configure ... certificate", proceed to the step for setting certbot.
* try to access https://daq14.triumf.ca
** you should see a complaint about self-signed certificate
** you should see a request for password (do not login yet)
** if you get "connection refused", HTTPS port 443 may need to be enabled in the local firewall, look at documentation for ufw.
Second, configure certbot:

(Note: as of 2018-01-18 certbot requires use of http port 80 to get the initial https certificate,
renewal can continue to use the https port 443)

(Note: as of 2019-01-?? certbot requires use of port 80 for renewals)

(Note: unsurprisingly, this requires outside access to connect with letsencrypt, so won't work if PC is only accessible from on-site network)

* check that port 80 is not used by anything:
* netstat -an | grep LISTEN | grep ^tcp | grep 80
* lsof -P | grep -i tcp | grep LISTEN | grep 80
* if lsof reports that apache2 is listening on port 80, follow the apache2 instructions above (remove "listen 80" from apache2.conf

* install certbot (if necessary open tcp port 80 in the firewall, see documentation for ufw):
<pre>
apt install certbot python3-certbot-apache
certbot certonly --standalone --installer apache
</pre>
* then answer questions:
* "activate HTTPS for daq14.triumf.ca" - say ok
* "enter email address" - enter your own email address
* "please read terms..." - read the terms and say "agree"
* it will take a few moments...
* "congratulations..." - say ok.
<pre>
certbot install --apache --cert-name daq14.triumf.ca
</pre>
* then answer questions:
* "choose redirect..." - say "1" (no redirect)
* look inside /etc/apache2/sites-enabled/daq14-ssl.conf to see that SSLCertificateFile & co point to certbot certificates in
/etc/letsencrypt/live/daq14.triumf.ca/
* to check current renewal and to update the certbot config file in /etc/letsencrypt/renewal, run this:
<pre>
certbot renew --standalone --installer apache --force-renewal
</pre>

NOTE: this certificate will expire in 3 months, automatic renewal should work with current version of certbot

Third, activate password protection:

* as shown in the config file above, create password file and initial user: (replace "midas" with specific username)
<pre>
touch /etc/apache2/htpasswd
htpasswd /etc/apache2/htpasswd midas
</pre>

* restart apache2
<pre>
systemctl restart apache2
systemctl status apache2
</pre>

From here:
* enable proxy for MIDAS mhttpd - uncomment redirect in the config file above
* enable proxy for ELOG - ditto
<pre>
a2enmod proxy
a2enmod proxy_http
apache2ctl configtest
systemctl restart apache2
</pre>

* try accessing MIDAS https://daq14.triumf.ca/ (make sure mhttpd is running)
* if it's not working, check odb setting FIXME!
* try accessing ELog https://daq14.triumf.ca/elog/ (make sure elogd is running)
* if it's not working, check elogd.cfg file and make sure
<pre>
SSL = 0
</pre>

NOTE: if certbot fails with errors about 'module' object has no attribute 'pyopenssl',
try this: pip install requests==2.6.0

== generate self-signed certificate ==

<pre>
root@alphacpc05:~# openssl req -nodes -new -x509 -keyout server.key -out server.cert -days 1001
...+....+..+..........+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+..+...+.........+......+.+...+...+.....+...............+.........+...+.+......+...+...........+....+...+..+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+......+.+...+..+.......+..+...+.......+......+...+..+...+......+....+...............+..+...+....+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
......+......+.+..+......+.+......+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.....+......+.+.........+......+.....+.+..+...+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+....+......+.....+...+...+.......+..+.+........+.+...+......+..+..........+..+.+...........+...+.......+......+.....+.......+...+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:CH
State or Province Name (full name) [Some-State]:Geneve
Locality Name (eg, city) []:CERN
Organization Name (eg, company) [Internet Widgits Pty Ltd]:CERN
Organizational Unit Name (eg, section) []:ALPHA experiment
Common Name (e.g. server FQDN or YOUR name) []:alphacpc05.cern.ch
Email Address []:
root@alphacpc05:~#
root@alphacpc05:~#
root@alphacpc05:~# ls -l
-rw-r--r-- 1 root root 1375 juil. 10 21:43 server.cert
-rw------- 1 root root 1708 juil. 10 21:42 server.key
root@alphacpc05:~# systemctl restart apache2
</pre>

= Enable elog PDF preview =

see https://stackoverflow.com/questions/52998331/imagemagick-security-policy-pdf-blocking-conversion

* xemacs -nw /etc/ImageMagick-6/policy.xml
* remove this section at the end:
<pre>

<policy domain="coder" rights="none" pattern="PS" />
<policy domain="coder" rights="none" pattern="PS2" />
<policy domain="coder" rights="none" pattern="PS3" />
<policy domain="coder" rights="none" pattern="EPS" />
<policy domain="coder" rights="none" pattern="PDF" />
<policy domain="coder" rights="none" pattern="XPS" />
</pre>

= Install Jupyter notebook =

<pre>
From https://jupyter.org/install
apt install python3-pip
pip install jupyterlab
pip install notebook
~/.local/bin/jupyter notebook
watch the http://localhost:8888 URL that it printed
say "no" to offer to start firefox (it will not work!)
URL is: http://localhost:8888/tree?token=xxx
from the machine where you are running the web browser (i.e. google-chrome), run (replace trinat@trinatdaq with the username and machine name where you started jupyter)
open a new shell and run: ssh -v trinat@trinatdaq -L 8888:localhost:8888
in the web browser, open http://localhost:8888
this gives us the login page
in the password or token entry field, put the token from the "tree?token=xxx" above (printed by jupyter on startup)
push button "login"
jupyter page should open with the list of files in the trinat home directory
congratulate Brian with full success
</pre>

= Install ZFS quota report =

If there are any ZFS volumes, install script to report disk and quota usage

<pre>
cd ~/git/scripts/quotareport
git pull
mkdir /var/www/html/zfsquotareport
cp -pv ~/git/scripts/quotareport/sorttable.js /var/www/html/zfsquotareport/
ln -s $PWD/zfsquotareport.perl /etc/cron.daily/
touch /etc/crontab
</pre>

If httpd is configured to redirect "/" to MIDAS mhttpd:
* add following to /etc/apache2/sites-enabled/xxx-ssl.conf in front of "ProxyPass / ..."
* run "systemctl reload apache2"
<pre>
## do not proxy zfs quota report directory
ProxyPass /zfsquotareport/ !
</pre>

= Install PHP =

* apt install php libapache2-mod-php
* systemctl restart apache2
* create /var/www/html/info.php
<pre>
<?php

phpinfo();
</pre>
* open https://daq00.triumf.ca/info.php

= Configure TRIUMF printers =

<pre>
systemctl stop cups
systemctl disable cups
systemctl stop snap.cups.cupsd.service
systemctl stop snap.cups.cups-browsed.service
systemctl disable snap.cups.cupsd.service
systemctl disable snap.cups.cups-browsed.service
echo "ServerName printers.triumf.ca" > /etc/cups/client.conf
lpstat -a
</pre>

= Enable core dumps =

By default, Ubuntu LTS 20.04 installs the apport package
which disabled core dumps from user applications. (google it up!).
It is not meant to do this and documentation claims that
it is not installed and not enabled by default. Oh, well...

<pre>
apt remove apport
apt autoremove ### will remove apport-symptoms and a few other packages
</pre>

After this, core dumps are written to file "core" in the current directory.
See /proc/sys/kernel/core_pattern and /proc/sys/kernel/core_uses_pid.

Enable core dump file names to include process id, add following to /etc/rc.local

<pre>
echo 1 > /proc/sys/kernel/core_uses_pid
</pre>

= Enable debugger =

By default, Ubuntu LTS 20.04 does not permit debugger to attach and debug
already running programs. To enable it, add following to /etc/rc.local

<pre>
echo 0 > /proc/sys/kernel/yama/ptrace_scope
</pre>

= Disable Ubuntu Pro nag =

best I can tell, impossible at this time.

== do not do this ==

!!! does nothing !!!

<pre>
pro config set apt_news=false
</pre>

== do not do this ==

!!! breaks automatic updates because 20apt-esm-hook.conf is missing !!!

If "apt upgrade" requests Ubuntu Pro or esm-apps, disable the nag:
<pre>
/bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf
</pre>

== do not do this ==

!!! likely same as above, breaks automatic updates !!!

* comment out /etc/apt/apt.conf.d/20apt-esm-hook.conf

== do not do this ==

!!! removes too many packages !!!

<pre>
apt remove ubuntu-pro-client
</pre>

= Update packages =

* apt-get update # update package list
* apt-get dist-upgrade # install updated packages and update "kept back" packages
* apt-get autoremove # remove packages that apt thinks should be removed

= Finish installation =

Congratulations. There is nothing more to do!

* reboot
<pre>
shutdown -r now
</pre>

= Update to new version of Ubuntu =

<pre>
vi /etc/update-manager/release-upgrades # set "Prompt=normal"
do-release-upgrade
</pre>

Update Ubuntu LTS 20.04 to LTS 22.04:

<pre>
apt remove zsys
</pre>

== daqubuntu ==

<pre>
# reboot to clear out all updates
# vi /etc/update-manager/release-upgrades # set "Prompt=normal"
# do-release-upgrade -c
Checking for a new Ubuntu release
New release '22.04 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
# do-release-upgrade
...
say yes...
...
login.defs, say "Y" (erase local changes, use packaged version)
/etc/systemd/resolved.conf, say "Y" (same as above)
firefox snap, say yes
unable to reach snap store, say "skip"
/etc/gmond.conf, say "Y"
/var/yp/Makefile, say "install the package maintainer's version"
/etc/ypserv.conf, same thing
/etc/ypserv.securenets, same thing
/etc/default/nis, same thing
/etc/speech-dispatcher/modules/mary-generic.conf, same thing
/etc/apt/apt.conf.d/50unattended-upgrades, same thing
...
278 packages are going to be removed, say yes
...
restart required, say yes
...
no ping... yes ping...
...
ssh daqubuntu, ok
apt update, fail, DNS does not work, "host security.ubuntu.com" does not resolve.
fix resolver per https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Disable_NetworkManager
apt update, apt upgrade now works, 0 packages to update
NIS does not work.
</pre>

== midm9a ==

<pre>
login.defs
firefox snap
gmond.conf
ypserv
/etc/default/nis
unattended-upgrades
amanda-security.conf
remove obsolete (no)
reboot
configure dns
reenable nis
</pre>

== daq17 ==

<pre>
firefox snap
imagemagick policy.xml
gmond.conf
chrony.conf
/var/yp/Makefile
ypserv.conf
ypserv.securenets
/etc/default/nis
50unattended-upgrades
</pre>

== daq00 ==

per https://serverpilot.io/docs/how-to-upgrade-ubuntu-20.04-to-22.04/

<pre>
do-release-upgrade -f DistUpgradeViewNonInteractive
</pre>

if it exists "too soon" without doing anything, run it without "-f xxx", most likely it does not like something about this machine. in case of daq00 it did not like how the EFI partitions were mounted. after fixing it, non-interactive upgrade was successful.

== isdaq08 ==

* prepare
<pre>
cd ~/git/scripts
git pull
cd ~
apt -y install debsums
</pre>
* check for modified config files that make upgrade unhappy, deal with all files reported by debsums.
<pre>
root@isdaq08:~# debsums -ce
/etc/ganglia/gmond.conf
/etc/yp.conf
/etc/apt/apt.conf.d/10periodic
root@isdaq08:~#
</pre>
* restore original /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
</pre>
* apt remove ganglia-monitor
* apt remove nis
* "debsums -ce" is now empty

Run the upgrade:

* do-release-upgrade -f DistUpgradeViewNonInteractive

Post upgrade:

* configure DNS
* apt -y install linux-generic-hwe-22.04
* /bin/cp -v ~/git/scripts/etc/99apt-conf-ko /etc/apt/apt.conf.d/ # restore nightly updates
* /bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf # remove the ubuntu-pro nag
* install missing packages
* restore ganglia
* restore nis
* check zpool status, may need zpool upgrade
* reboot

== daqubuntu, U-24 ==

* prepare
<pre>
cd ~/git/scripts
git pull
cd ~
apt -y install debsums
</pre>
* check for modified config files that make upgrade unhappy, deal with all files reported by debsums.
<pre>
root@daqubuntu:~# debsums -ce
/etc/ganglia/gmond.conf
debsums: missing file /etc/init.d/nis (from nis package)
/etc/default/nis
/etc/ypserv.conf
/etc/ypserv.securenets
/var/yp/Makefile
/etc/update-manager/release-upgrades
/etc/apt/apt.conf.d/10periodic
/etc/yp.conf
root@daqubuntu:~#
* restore original /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
</pre>
* apt remove ganglia-monitor
* apt remove nis
* apt autoremove
* restore original release-upgrades: "Prompt: lts"
* "debsums -ce" is now empty

Check for upgrade:

<pre>
root@daqubuntu:~# do-release-upgrade -c
Checking for a new Ubuntu release
There is no development version of an LTS available.
To upgrade to the latest non-LTS development release
set Prompt=normal in /etc/update-manager/release-upgrades.
root@daqubuntu:~#
</pre>

Run the upgrade:

* do-release-upgrade -f DistUpgradeViewNonInteractive

Post upgrade:

* configure DNS
* apt -y install linux-generic-hwe-22.04
* /bin/cp -v ~/git/scripts/etc/99apt-conf-ko /etc/apt/apt.conf.d/ # restore nightly updates
* /bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf # remove the ubuntu-pro nag
* install missing packages
* restore ganglia
* restore nis
* check zpool status, may need zpool upgrade
* reboot

== daq14, U-20-22-24 ==

* apt update, apt upgrade
* apt -y install linux-image-generic-hwe-20.04 linux-tools-virtual-hwe-20.04 ### install kernel 5.15
* shutdown -r now
* stuck waiting for daq14 to shutdown...
* reboot into kernel 5.15
* <pre>
cd ~/git/scripts
git pull
cd ~
apt -y install debsums
</pre>
* debsums -ce
<pre>
/etc/apache2/ports.conf
/etc/dnsmasq.conf
/etc/ganglia/gmond.conf
/etc/yp.conf
/etc/sudoers
</pre>
* apache2 restore original ports.conf, uncomment "Listen 80"
* cp -pv /etc/dnsmasq.conf.dpkg-dist /etc/dnsmasq.conf
* apt remove ganglia-monitor
* edit /etc/yp.conf, remove everything after "# ypserver ypserver.network.com"
* "debsums -ce" is now empty
* do-release-upgrade -f DistUpgradeViewNonInteractive
* runs for a long time
* stuck on "/etc/default/nis", type "Y", press enter, nothing for a bit, then resumes running
* finished
* configure DNS
* reboot
* have kernel 6.8
* apt update; apt upgrade
* apt upgrade guile-2.2-libs ### would not auto-update, "kept back", has to be done by hand
* apt autoremove
* debsums -ce
<pre>
debsums: missing file /etc/init.d/nis (from nis package)
/etc/default/nis
</pre>
* diff /etc/default/nis.dpkg-dist /etc/default/nis
* cp -pv /etc/default/nis.dpkg-dist /etc/default/nis
* debsums -ce
<pre>
debsums: missing file /etc/init.d/nis (from nis package)
</pre>
* we ignore this and run the update
* do-release-upgrade -c
<pre>
Checking for a new Ubuntu release
New release '24.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
</pre>
* do-release-upgrade -f DistUpgradeViewNonInteractive
* bombs out without any error messages
* in /var/log/dist-upgrade/main.log reports "Failed to find a replacement for xapp" and other packages
* apt remove xapp usrmerge ureadahead thunderbird-gnome-support
* no go, complains about even more packages.
* apt list | grep installed | grep -v jammy ### show packages installed from non-ubuntu sources
* remove all packages marked "install,local" ### ubuntu updater does not know where they came from and so cannot update them.
* apt remove desktop-base ### not happy about this package in /var/log/dist-upgrade/apt.log
* apt autoremove
* do-release-upgrade -f DistUpgradeViewNonInteractive
* running for a long time...

= Upgrade to new version of Debian =

https://www.debian.org/releases/bookworm/amd64/release-notes/ch-upgrading.en.html

== 32-bit VME processor Debian 11 to 12 ==

* cd git/scripts; git pull; cd ~
* apt update
* apt upgrade
* edit /etc/apt/sources.list
<pre>
deb http://deb.debian.org/debian/ bookworm main
#deb http://deb.debian.org/debian/ bullseye main
#deb-src http://deb.debian.org/debian/ bullseye main
</pre>
* apt update
* apt upgrade --without-new-pkgs
* apt full-upgrade
* apt list '~c'; apt purge '~c' # purge left-over config files [residual-config]
* reboot

= Ubuntu package manager =

* apt-get install xxx # install package xxx
* apt-get update
* apt-get upgrade
* apt-get dist-upgrade
* apt-get autoremove # remove automatically installed packages required by a removed package
* apt-get remove xxx # remove package xxx
* apt-cache search . # list all available packages
* apt-cache show "." | grep ^Package # list al available packages
* apt-cache madison root-system # show all available versions of package root-system
* apt list # list all installed packages
* dpkg --listfiles libpng16-16 # list all files from this package
* apt list --installed # list all installed packages
* dpkg -S /bin/bash # what package provides this file?
* dpkg -L bash # what files provided by this package?
* debsums -ce # show modified config files
* apt-config dump # show apt configuration

= Ubuntu zsys =

NOTE: DO NOT USE ZSYS, see https://github.com/ubuntu/zsys/issues/218 and https://github.com/ubuntu/zsys/issues/230

* manual removal of old snapshots
<pre>
zsysctl show
zsysctl state remove xy69ye -s
zsysctl state remove xy69ye
zsysctl state remove xy69ye -u wheel
</pre>
* apt remove zsys

NOTE: old zsys snapshots must be cleaned manually, "zsysctl state remove xxx --system" is broken and does not remove user data snapshots

* manages system snapshots
* documentation: https://github.com/ubuntu/zsys
* documentation: (go to next article via link "newer" at the bottom) https://didrocks.fr/2020/05/21/zfs-focus-on-ubuntu-20.04-lts-whats-new/
* ubuntu 20.04 bug, too many snapshots cause /boot to become full and updates fail. https://github.com/ubuntu/zsys/issues/155
* solution: use custom /etc/zsys.conf, limit number of snapshots to 10, see trinatdaq:/etc/zsys.conf
* zsys commands:
<pre>
update-grub # list of all snapshots, errors if some snapshots are broken
zsysctl state remove lnc0k7 --system # remove snapshot
xemacs -nw /etc/zsys.conf; zsysctl service reload; zsysctl service gc # cause gc to run with new settings in zsys.conf
zfs list -r -t snapshot -o name,used,referenced,creation bpool/BOOT # list snapshots
zsysctl show # show snapshots
</pre>

= Ubuntu cloning =

to clone a ubuntu image:

<pre>
cd /nfsroot/lxcpet
emacs -nw etc/hostname ### change hostname
emacs -nw etc/mailname ### change hostname (debian 11)
emacs -nw etc/defaultdomain ### change the NIS domainname
emacs -nw etc/yp.conf ### change the NIS server
cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
emacs -nw root/.ssh/authorized_keys ### update root ssh keys
</pre>

= Ubuntu boot loader =

== maintenance commands ==

* update-initramfs -v -u
* grub-install /dev/sda

= Convert from single to dual mirrored ZFS SSD =

Assuming Ubuntu LTS 22.04 with "instal on ZFS" option, we will
add a second SSD, configure ZFS to use both SSDs in mirrored
configuration and setup grub to boot from either SSD. This
is intended to create a full redundant system where failure
of either SSD does not break the system.

* identify first SSD
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* connect second SSD of identical size
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
/dev/sdb WD Blue SA510 2.5 250GB 22243Z803852 25 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* if second SSD is not autodetected, reboot
* Clone partition table automatically
If both SSDs are identical size, use this simpler method of duplicating the partition table:
<pre>
root@midm9b:~# sfdisk -d /dev/sda > part_table
root@midm9b:~# grep -v ^label-id part_table | sed -e 's/, *uuid=[0-9A-F-]*//' | sfdisk /dev/sdb
</pre>
The grep and sed in the second command are there to prevent disk ID and partition IDs from being cloned. Alternatively the part_table file can be edited manually to remove the label-id line and the uuid entries from the individual partitions.

* Clone partition table manually (e.g. for different size disks)
* list partition table of first SSD:
<pre>
root@midm9b:~# fdisk -l /dev/sda
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>
* create identical partitions on second SSD, use sector numbers from above.
<pre>
root@midm9b:~# gdisk /dev/sdb
GPT fdisk (gdisk) version 1.0.8

Partition table scan:
MBR: not present
BSD: not present
APM: not present
GPT: not present

Creating new GPT entries in memory.

Command (? for help): n
Partition number (1-128, default 1):
First sector (34-488397134, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-488397134, default = 488397134) or {+-}size{KMGTP}: 1050623
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): ef00
Changed type of partition to 'EFI system partition'

Command (? for help): n
Partition number (2-128, default 2):
First sector (34-488397134, default = 1050624) or {+-}size{KMGTP}:
Last sector (1050624-488397134, default = 488397134) or {+-}size{KMGTP}: 5244927
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 8200
Changed type of partition to 'Linux swap'

Command (? for help): n
Partition number (3-128, default 3):
First sector (34-488397134, default = 5244928) or {+-}size{KMGTP}:
Last sector (5244928-488397134, default = 488397134) or {+-}size{KMGTP}: 9439231
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): be00
Changed type of partition to 'Solaris boot'

Command (? for help): n
Partition number (4-128, default 4):
First sector (34-488397134, default = 9439232) or {+-}size{KMGTP}:
Last sector (9439232-488397134, default = 488397134) or {+-}size{KMGTP}:
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): bf00
Changed type of partition to 'Solaris root'

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sdb.
The operation has completed successfully.
root@midm9b:~# fdisk -l /dev/sda /dev/sdb
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root

Disk /dev/sdb: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: EB251739-30C6-422F-A505-5887B5A0B603

Device Start End Sectors Size Type
/dev/sdb1 2048 1050623 1048576 512M EFI System
/dev/sdb2 1050624 5244927 4194304 2G Linux swap
/dev/sdb3 5244928 9439231 4194304 2G Solaris boot
/dev/sdb4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>
* identify second SSD partitions
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part3
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part3 -> ../../sda3
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 -> ../../sdb3
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
</pre>
* convert bpool from single disk to mirrored disk:
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~# zpool attach bpool 99e03dc0-7d4d-f24b-8fa1-f042b9f135db /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3
root@midm9b:~# zpool status bpool
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors
</pre>
* convert rpool
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
root@midm9b:~# zpool attach rpool f6fd54f8-3af7-b943-ae3d-a4e480537fb9 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4
root@midm9b:~# zpool status rpool
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Jan 20 19:40:45 2023
5.83G scanned at 664M/s, 2.92M issued at 332K/s, 9.11G total
0B resilvered, 0.03% done, no estimated completion time
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~#
</pre>
* wait for resilver to complete
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
scan: resilvered 9.65G in 00:00:36 with 0 errors on Fri Jan 20 19:41:21 2023
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
</pre>
* enable booting from second SSD: (instead of /dev/sda1, /dev/sdb1, use UUID=xxx)
<pre>
root@midm9b:~# mkfs.msdos /dev/sdb1
root@midm9b:~# mkdir /boot/efi-sda
root@midm9b:~# mkdir /boot/efi-sdb
root@midm20c:~# blkid | grep vfat ### identify UUID
/dev/sdb1: UUID="DD89-5081" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="d0cb6be4-2f67-5b42-9b26-9e6905e9f774"
/dev/sdc1: UUID="D970-86BA" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="e6d3b5b9-a512-44a2-9205-1a4db06ed2a2"
/dev/sda1: UUID="DDA1-044C" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="6dc9dff0-1c13-8045-a906-7803d3074c70"
root@midm20c:~# cat /etc/fstab | grep vfat ### add mount points with correct UUID
#UUID=D970-86BA /boot/efi vfat umask=0022,fmask=0022,dmask=0022 0 1
UUID=DDA1-044C /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022 0 1
UUID=DD89-5081 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022 0 1
root@midm9b:~# mount -a
root@midm9b:~# df -kl
Filesystem 1K-blocks Used Available Use% Mounted on
...
/dev/sda1 523244 13720 509524 3% /boot/efi
/dev/sdb1 523244 4 523240 1% /boot/efi-sdb
...
root@midm9b:~# rsync -av /boot/efi/ /boot/efi-sdb/
sending incremental file list
EFI/
...
root@midm9b:~# ls -l /boot/efi-sda
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~# ls -l /boot/efi-sdb
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~#
</pre>
* setup script to update grub on second SSD, it must be run manually after every kernel update
<pre>
root@midm9b:~# ln -s ~/git/scripts/etc/update_efi_grub.perl ~/
root@midm9b:~# ~/update_efi_grub.perl -u
EFI dir: /boot/efi-sda
/boot/efi-sda: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sda/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sda: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sda/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
EFI dir: /boot/efi-sdb
/boot/efi-sdb: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sdb/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sdb: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sdb/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
root@midm9b:~#
</pre>

= Disable NetworkManager =

NOTE: THIS IS BROKEN IN UBUNTU LTS 22.04

NetworkManager is useful for configuring dynamic
network interfaces, i.e. laptops that often move
between networks, or connect to multiple choice
of wifi networks, etc.

For machines with statically configured network interfaces,
NetworkManager is not necessary.

As it has been observed to become confused and observed
to malfunction when network links go up and down (it keeps
unnecessarily reconfiguring the ip address, etc), it can
be usefuil to disable it.

* list all network interfaces
<pre>
# /bin/ls -1 /sys/class/net/
enp0s31f6
lo
</pre>
* edit /etc/network/interfaces:
<pre>
rename enp0s31f6=eth0
auto eth0
iface eth0 inet static
address 142.90.120.94/19
gateway 142.90.100.18
</pre>
* statically configure systemd-resolved
** create /etc/systemd/resolved.conf.d/resolved.conf with this contents:
<pre>
[Resolve]
DNS=142.90.100.19
Domains=triumf.ca
</pre>
** systemctl restart systemd-resolved
** resolvectl
** systemd-analyze cat-config systemd/resolved.conf
* disable NetworkManager
<pre>
systemctl disable NetworkManager
</pre>
* reboot

= Configure ECC memory =

== Configure EDAC ==

* apt install edac-utils rasdaemon

=== Intel i3-2120 ===
<pre>
root@musr00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X9SCL/X9SCM
root@musr00:~# edac-ctl --status
edac-ctl: drivers not loaded.
</pre>

=== Intel E-2236 ===
<pre>
root@daq00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SCM-F
root@daq00:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq00:~# edac-util
edac-util: No errors to report.
root@daq00:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
</pre>
* check edac sysfs files (Intel)
<pre>
root@daq00:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_noinfo_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 max_location
-r--r--r-- 1 root root 4096 Jan 25 15:10 mc_name
drwxr-xr-x 2 root root 0 Jan 25 15:10 power
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank0
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank1
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank2
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank3
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank4
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank5
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank6
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank7
--w------- 1 root root 4096 Jan 25 15:10 reset_counters
-r--r--r-- 1 root root 4096 Jan 25 15:10 seconds_since_reset
-r--r--r-- 1 root root 4096 Jan 25 15:10 size_mb
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Jan 25 15:10 uevent
root@daq00:~#
</pre>

=== Intel E3-1270 v6 ===
<pre>
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --status
edac-ctl: drivers are loaded.
root@grsnis01:~# edac-util
edac-util: No errors to report.
root@grsnis01:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@grsnis01:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 max_location
-r--r--r-- 1 root root 4096 Feb 19 12:35 mc_name
drwxr-xr-x 2 root root 0 Feb 19 12:35 power
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank0
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank1
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank2
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank3
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank4
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank5
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank6
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank7
--w------- 1 root root 4096 Feb 19 12:35 reset_counters
-r--r--r-- 1 root root 4096 Feb 19 12:35 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 19 12:35 size_mb
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 19 12:35 uevent
root@grsnis01:~#
</pre>

=== Intel E3-1245 v6 ===

<pre>
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --status
edac-ctl: drivers are loaded.
[root@alphagdaq ~]# edac-util
edac-util: No errors to report.
[root@alphagdaq ~]# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
[root@alphagdaq ~]# ras-mc-ctl --layout
+-----------------------------------------------+
| mc0 |
| csrow0 | csrow1 | csrow2 | csrow3 |
----------+-----------------------------------------------+
channel1: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
----------+-----------------------------------------------+
[root@alphagdaq ~]# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#3channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#3channel#1 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#2channel#0 0 0
[root@alphagdaq ~]# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SSH-F
[root@alphagdaq ~]# ras-mc-ctl --summary
DBD::SQLite::db prepare failed: no such table: mc_event at /usr/sbin/ras-mc-ctl line 1129.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1130.
[root@alphagdaq ~]#
</pre>

=== AMD 3700X ===

(memory is non-ECC)

<pre>
root@daq13:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@daq13:~#
root@daq13:~#
root@daq13:~# edac-ctl --status
edac-ctl: drivers not loaded.
root@daq13:~# edac-util
edac-util: Error: No memory controller data found.
root@daq13:~# edac-util -s
edac-util: EDAC drivers loaded. No memory controllers found
root@daq13:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 2 root root 0 Jan 25 15:26 power
lrwxrwxrwx 1 root root 0 Jan 21 16:16 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Jan 21 16:16 uevent
</pre>

(memory is ECC)

<pre>
root@trinatdaq:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@trinatdaq:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@trinatdaq:~# edac-util
edac-util: No errors to report.
root@trinatdaq:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Dec 15 13:04 mc0
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
lrwxrwxrwx 1 root root 0 Dec 13 18:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dec 13 18:31 uevent
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_noinfo_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 max_location
-r--r--r-- 1 root root 4096 Dec 15 13:04 mc_name
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank4
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank5
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank6
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank7
--w------- 1 root root 4096 Dec 15 13:04 reset_counters
-rw-r--r-- 1 root root 4096 Dec 15 13:04 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Dec 15 13:04 seconds_since_reset
-r--r--r-- 1 root root 4096 Dec 15 13:04 size_mb
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Dec 15 13:04 uevent
root@trinatdaq:~#
</pre>

=== AMD 5000G ===

* no linux driver for AMD 5000-series "G" CPU
* no mention of ECC in the BIOS settings
* unclear status of ECC support in AMD documentation (sais only "pro" "G" CPUs have ECC)
* unclear status of ECC support in ASUS documentation (web page out of date)

=== AMD 5600X ===

<pre>
root@daq17:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-XE GAMING WIFI
root@daq17:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq17:~# edac-util
edac-util: No errors to report.
root@daq17:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@daq17:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Aug 19 19:27 mc0
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
lrwxrwxrwx 1 root root 0 May 10 10:11 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 May 10 10:11 uevent
root@daq17:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_noinfo_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 max_location
-r--r--r-- 1 root root 4096 Aug 19 19:27 mc_name
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank4
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank5
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank6
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank7
--w------- 1 root root 4096 Aug 19 19:27 reset_counters
-rw-r--r-- 1 root root 4096 Aug 19 19:27 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Aug 19 19:27 seconds_since_reset
-r--r--r-- 1 root root 4096 Aug 19 19:27 size_mb
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Aug 19 19:27 uevent
root@daq17:~#
</pre>

=== AMD 3955WX ===

<pre>
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --status
edac-ctl: drivers are loaded.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util
edac-util: No errors to report.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@alphasuperdaq:~/git/scripts/quotareport# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 19 root root 0 Dez 12 04:48 mc0
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
lrwxrwxrwx 1 root root 0 Dez 9 05:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dez 9 05:31 uevent
root@alphasuperdaq:~/git/scripts/quotareport#
root@alphasuperdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 max_location
-r--r--r-- 1 root root 4096 Feb 28 22:19 mc_name
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank0
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank1
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank10
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank11
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank12
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank13
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank14
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank15
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank2
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank3
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank4
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank5
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank6
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank7
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank8
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank9
--w------- 1 root root 4096 Feb 28 22:19 reset_counters
-rw-r--r-- 1 root root 4096 Feb 28 22:19 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Feb 28 22:19 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 28 22:19 size_mb
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 28 22:19 uevent
root@alphasuperdaq:~#
root@alphasuperdaq:~# ras-mc-ctl --layout
Use of uninitialized value $max_pos[3] in modulus (%) at /usr/sbin/ras-mc-ctl line 868.
Use of uninitialized value $d in numeric ge (>=) at /usr/sbin/ras-mc-ctl line 869.
Use of uninitialized value $d in sprintf at /usr/sbin/ras-mc-ctl line 872.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| mc0 |
| csrow0 | csrow1 |
| channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 | channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

0: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
root@alphasuperdaq:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#2 0 0
mc#0csrow#1channel#7 0 0
mc#0csrow#0channel#3 0 0
mc#0csrow#1channel#4 0 0
mc#0csrow#1channel#2 0 0
mc#0csrow#0channel#7 0 0
mc#0csrow#1channel#3 0 0
mc#0csrow#0channel#4 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#5 0 0
mc#0csrow#0channel#6 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#0channel#5 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#6 0 0
root@alphasuperdaq:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: ASUSTeK COMPUTER INC. model Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@alphasuperdaq:~#
</pre>

=== AMD 7700X ===

<pre>
root@dsfe05:~# apt install edac-utils
root@dsfe05:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro H13SAE-MF
root@dsfe05:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@dsfe05:~# edac-util
edac-util: No errors to report.
root@dsfe05:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@dsfe05:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 May 14 09:33 ce_count
-r--r--r-- 1 root root 4096 May 14 09:33 ce_noinfo_count
-r--r--r-- 1 root root 4096 May 14 09:33 max_location
-r--r--r-- 1 root root 4096 May 14 09:33 mc_name
drwxr-xr-x 2 root root 0 May 14 09:33 power
drwxr-xr-x 3 root root 0 May 14 09:33 rank4
drwxr-xr-x 3 root root 0 May 14 09:33 rank5
--w------- 1 root root 4096 May 14 09:33 reset_counters
-r--r--r-- 1 root root 4096 May 14 09:33 seconds_since_reset
-r--r--r-- 1 root root 4096 May 14 09:33 size_mb
-r--r--r-- 1 root root 4096 May 14 09:33 ue_count
-r--r--r-- 1 root root 4096 May 14 09:33 ue_noinfo_count
-rw-r--r-- 1 root root 4096 May 14 09:33 uevent
root@dsfe05:~#
</pre>

== Configure rasdaemon ==

<pre>
apt install rasdaemon
</pre>
<pre>
systemctl enable rasdaemon
systemctl restart rasdaemon
systemctl status rasdaemon
</pre>

<pre>
● rasdaemon.service - RAS daemon to log the RAS events
Loaded: loaded (/lib/systemd/system/rasdaemon.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-01-25 15:16:37 PST; 3min 5s ago
Main PID: 2477175 (rasdaemon)
Tasks: 1 (limit: 76958)
Memory: 17.1M
CGroup: /system.slice/rasdaemon.service
└─2477175 /usr/sbin/rasdaemon -f -r

Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Listening to events for cpus 0 to 11
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mc_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording aer_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording extlog_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mce_record events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording arm_event events
</pre>

== Get reports ==

* Intel 2x32GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --layout
+-------------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-------------------------+
channel1: | 16384 MB | 16384 MB |
channel0: | 16384 MB | 16384 MB |
----------+-------------------------+
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#0channel#1 0 0
root@daq00:~#
</pre>

* Intel 4x16GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#1 0 0
mc#0csrow#2channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#3channel#0 0 0
mc#0csrow#3channel#1 0 0
root@daq00:~#
root@daq00:~# ras-mc-ctl --layout
+-----------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-----------------------+
channel1: | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB |
----------+-----------------------+
root@daq00:~#
root@daq00:~#
root@daq00:~#
root@daq00:~# ras-mc-ctl --print-labels
ras-mc-ctl: Error: No dimm labels for Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@daq00:~#
</pre>

note: ubuntu LTS 22.04 DBD::SQLite::db error is not there.

* AMD 7700 2x32GB DDR5 ECC DIMMs

<pre>
root@dsfe05:~# systemctl status rasdaemon
● rasdaemon.service - RAS daemon to log the RAS events
Loaded: loaded (/lib/systemd/system/rasdaemon.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2024-05-14 09:36:43 PDT; 33ms ago
Process: 4088418 ExecStartPost=/usr/sbin/rasdaemon --enable (code=exited, status=0/SUCCESS)
Main PID: 4088417 (rasdaemon)
Tasks: 1 (limit: 37300)
Memory: 788.0K
CPU: 5ms
CGroup: /system.slice/rasdaemon.service
└─4088417 /usr/sbin/rasdaemon -f -r

May 14 09:36:43 dsfe05 rasdaemon[4088417]: ras:aer_event event enabled
May 14 09:36:43 dsfe05 rasdaemon[4088417]: Enabled event ras:aer_event
May 14 09:36:43 dsfe05 rasdaemon[4088417]: mce:mce_record event enabled
May 14 09:36:43 dsfe05 rasdaemon[4088417]: Enabled event mce:mce_record
May 14 09:36:43 dsfe05 rasdaemon[4088417]: ras:extlog_mem_event event enabled
May 14 09:36:43 dsfe05 rasdaemon[4088417]: Enabled event ras:extlog_mem_event
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording mc_event events
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording aer_event events
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording extlog_event events
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording mce_record events
root@dsfe05:~# ras-mc-ctl --layout
Use of uninitialized value $max_pos[3] in modulus (%) at /usr/sbin/ras-mc-ctl line 907.
Use of uninitialized value $d in numeric ge (>=) at /usr/sbin/ras-mc-ctl line 908.
Use of uninitialized value $d in sprintf at /usr/sbin/ras-mc-ctl line 911.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
+-----------------------------------------------------------------------------------------------+
| mc0 |
| csrow0 | csrow1 | csrow2 | csrow3 |
| channel0 | channel1 | channel0 | channel1 | channel0 | channel1 | channel0 | channel1 |
----+-----------------------------------------------------------------------------------------------+

0: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB |
----+-----------------------------------------------------------------------------------------------+
root@dsfe05:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#2channel#1 0 0
mc#0csrow#2channel#0 0 0
root@dsfe05:~# ras-mc-ctl --print-labels
ras-mc-ctl: Error: No dimm labels for Supermicro model H13SAE-MF
root@dsfe05:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model H13SAE-MF
root@dsfe05:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

No MCE errors.
root@dsfe05:~#
</pre>

= sensors =

== ASUS P7P55D EVO ==

* BIOS version 2101

<pre>
root@iris01:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Core 0: +34.0°C (high = +83.0°C, crit = +99.0°C)
Core 1: +37.0°C (high = +83.0°C, crit = +99.0°C)
Core 2: +38.0°C (high = +83.0°C, crit = +99.0°C)
Core 3: +35.0°C (high = +83.0°C, crit = +99.0°C)

nouveau-pci-0100
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.05 V)
temp1: +46.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

atk0110-acpi-0
Adapter: ACPI interface
Vcore Voltage: 864.00 mV (min = +0.80 V, max = +1.60 V)
+3.3V Voltage: 3.38 V (min = +2.97 V, max = +3.63 V)
+5V Voltage: 5.04 V (min = +4.50 V, max = +5.50 V)
+12V Voltage: 12.15 V (min = +10.20 V, max = +13.80 V)
CPU Fan Speed: 968 RPM (min = 600 RPM, max = 7200 RPM)
Chassis1 Fan Speed: 1288 RPM (min = 600 RPM, max = 7200 RPM)
Chassis2 Fan Speed: 1316 RPM (min = 600 RPM, max = 7200 RPM)
Power Fan Speed: 0 RPM (min = 0 RPM, max = 7200 RPM)
CPU Temperature: +34.0°C (high = +45.0°C, crit = +45.5°C)
MB Temperature: +30.0°C (high = +45.0°C, crit = +46.0°C)

root@iris01:~#
</pre>

== ASUS Z170-DELUXE ==

* BIOS version 3801
* load sensors drivers
<pre>
echo modprobe coretemp >> /etc/rc.local
echo modprobe jc42 >> /etc/rc.local
echo modprobe lm92 >> /etc/rc.local
echo modprobe nct6775 >> /etc/rc.local
</pre>
* in /boot/grub/grub.cfg, add: GRUB_CMDLINE_LINUX_DEFAULT="acpi_enforce_resources=no"
* update grub and reboot: grub-mkconfig -o /boot/grub/grub.cfg

<pre>
root@iris00:~# sensors
nct6793-isa-0290
Adapter: ISA adapter
in0: 600.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 144.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in6: 0.00 V (min = +0.00 V, max = +0.00 V)
in7: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.14 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1000.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in10: 600.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 592.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 968.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1370 RPM (min = 0 RPM)
fan2: 1437 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan6: 0 RPM (min = 0 RPM)
SYSTIN: +32.0°C (high = +98.0°C, hyst = +95.0°C) sensor = thermistor
CPUTIN: +42.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: -128.0°C sensor = thermistor
AUXTIN1: +50.0°C sensor = thermistor
AUXTIN2: +22.0°C sensor = thermistor
AUXTIN3: +28.0°C sensor = thermistor
PECI Agent 0: +50.0°C (high = +98.0°C, hyst = +95.0°C)
(crit = +100.0°C)
PECI Agent 0 Calibration: +42.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
TSI2_TEMP: +3892314.0°C
TSI3_TEMP: +3892314.0°C
TSI4_TEMP: +3892314.0°C
TSI5_TEMP: +3892314.0°C
TSI6_TEMP: +3892314.0°C
TSI7_TEMP: +3892314.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

jc42-i2c-0-1a
Adapter: SMBus I801 adapter at f040
temp1: +36.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

jc42-i2c-0-18
Adapter: SMBus I801 adapter at f040
temp1: +34.8°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

jc42-i2c-0-1b
Adapter: SMBus I801 adapter at f040
temp1: +35.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

jc42-i2c-0-19
Adapter: SMBus I801 adapter at f040
temp1: +36.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +52.0°C (high = +84.0°C, crit = +100.0°C)
Core 0: +52.0°C (high = +84.0°C, crit = +100.0°C)
Core 1: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 2: +48.0°C (high = +84.0°C, crit = +100.0°C)
Core 3: +47.0°C (high = +84.0°C, crit = +100.0°C)

root@iris00:~#
</pre>

== ASUS H110M-A/M.2 ==

* BIOS version 4202
* echo modprobe coretemp >> /etc/rc.local
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@midpol:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +33.0°C (high = +80.0°C, crit = +100.0°C)
Core 0: +33.0°C (high = +80.0°C, crit = +100.0°C)
Core 1: +30.0°C (high = +80.0°C, crit = +100.0°C)

acpitz-acpi-0
Adapter: ACPI interface
temp1: +27.8°C (crit = +119.0°C)
temp2: +29.8°C (crit = +119.0°C)

nct6793-isa-0290
Adapter: ISA adapter
in0: 368.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 152.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in6: 928.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.14 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1000.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in10: 152.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in11: 128.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in12: 136.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in13: 120.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 136.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1004 RPM (min = 0 RPM)
fan2: 1143 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan6: 0 RPM (min = 0 RPM)
SYSTIN: +118.0°C (high = +98.0°C, hyst = +95.0°C) sensor = thermistor
CPUTIN: +29.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +30.0°C sensor = thermistor
AUXTIN1: +112.0°C sensor = thermistor
AUXTIN2: +111.0°C sensor = thermistor
AUXTIN3: +110.0°C sensor = thermistor
PECI Agent 0: +31.0°C (high = +98.0°C, hyst = +95.0°C)
(crit = +100.0°C)
PECI Agent 0 Calibration: +36.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
TSI2_TEMP: +3892314.0°C
TSI3_TEMP: +3892314.0°C
TSI4_TEMP: +3892314.0°C
TSI5_TEMP: +3892314.0°C
TSI6_TEMP: +3892314.0°C
TSI7_TEMP: +3892314.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

root@midpol:~#
</pre>

== ASUS P9X79 WS ==

* https://www.asus.com/supportonly/P9X79%20WS/HelpDesk_Manual/
* BIOS version 4802
* modprobe nct6775
* modprobe coretemp

<pre>
root@daq14:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 0: +29.0°C (high = +82.0°C, crit = +100.0°C)
Core 1: +24.0°C (high = +82.0°C, crit = +100.0°C)
Core 2: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +82.0°C, crit = +100.0°C)

nouveau-pci-0200
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.00 V)
temp1: +39.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

nct6776-isa-0290
Adapter: ISA adapter
Vcore: 1.04 V (min = +0.00 V, max = +1.74 V)
in1: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
AVCC: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
+3.3V: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 2.04 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 904.00 mV (min = +0.00 V, max = +0.00 V) ALARM
3VSB: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
Vbat: 3.30 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1265 RPM (min = 0 RPM)
fan2: 1909 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +34.0°C (high = +0.0°C, hyst = +0.0°C) ALARM sensor = thermistor
CPUTIN: +58.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermal diode
AUXTIN: +31.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
PECI Agent 0: +31.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +96.0°C)
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

root@daq14:~#
</pre>

== ASUS TUF GAMING B550M-PLUS WIFI II ==

* BIOS 2803, 2806
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@midm9a:~# sensors
nct6798-isa-0290
Adapter: ISA adapter
in0: 488.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.82 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in14: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 0 RPM (min = 0 RPM)
fan2: 760 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan7: 1264 RPM (min = 0 RPM)
SYSTIN: +25.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +22.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +95.0°C sensor = thermistor
AUXTIN1: +25.0°C sensor = thermistor
AUXTIN2: +25.0°C sensor = thermistor
AUXTIN3: +25.0°C sensor = thermistor
PECI Agent 0 Calibration: +23.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +32.4°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

amdgpu-pci-0800
Adapter: PCI adapter
vddgfx: 1.45 V
vddnb: 993.00 mV
edge: +28.0°C
PPT: 20.00 W

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +33.4°C

root@midm9a:~#
</pre>

== ASUS ASUS ROG STRIX B550-XE GAMING WIFI ==

* BIOS 2423, 2604
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@daq13:~# sensors
nct6798-isa-0290
Adapter: ISA adapter
in0: 344.00 mV (min = +0.00 V, max = +1.74 V)
in1: 992.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in6: 216.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.30 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.81 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in11: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 280.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 845 RPM (min = 0 RPM)
fan2: 998 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +28.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +27.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +94.0°C sensor = thermistor
AUXTIN1: +28.0°C sensor = thermistor
AUXTIN2: +28.0°C sensor = thermistor
AUXTIN3: +97.0°C sensor = thermistor
PECI Agent 0 Calibration: +27.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +33.6°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

amdgpu-pci-0600
Adapter: PCI adapter
vddgfx: 1.45 V
vddnb: 999.00 mV
edge: +29.0°C
PPT: 14.00 W

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: +30.0°C

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +33.9°C

root@daq13:~#
</pre>

== ASUS ASUS ROG STRIX B550-E GAMING ==

* bios 2803
* echo modprobe jc42 >> /etc/rc.local
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@daq17:~# sensors
jc42-i2c-1-1b
Adapter: SMBus PIIX4 adapter port 0 at 0b00
temp1: +25.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: +28.0°C

nouveau-pci-0800
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.00 V)
temp1: +34.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

nct6798-isa-0290
Adapter: ISA adapter
in0: 288.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 224.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.31 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.79 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 280.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 843 RPM (min = 0 RPM)
fan2: 629 RPM (min = 0 RPM)
fan3: 746 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +22.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +25.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +93.0°C sensor = thermistor
AUXTIN1: +22.0°C sensor = thermistor
AUXTIN2: +22.0°C sensor = thermistor
AUXTIN3: +96.0°C sensor = thermistor
PECI Agent 0 Calibration: +25.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +27.6°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

jc42-i2c-1-1a
Adapter: SMBus PIIX4 adapter port 0 at 0b00
temp1: +23.2°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

asusec-isa-0000
Adapter: ISA adapter
CPU_Opt: 0 RPM
Chipset: +34.0°C
CPU: +25.0°C
Motherboard: +22.0°C
T_Sensor: -40.0°C
VRM: +31.0°C

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +28.0°C
Tccd1: +27.5°C

root@daq17:~#
</pre>

== ASUS PRIME B650-PLUS ==

* BIOS 1811
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@dsdaqgw:~# sensors
amdgpu-pci-0b00
Adapter: PCI adapter
vddgfx: 930.00 mV
vddnb: 1.19 V
edge: +38.0°C
PPT: 25.10 W

nct6799-isa-0290
Adapter: ISA adapter
in0: 920.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.04 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 320.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.28 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.28 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.10 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.04 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 416.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 328.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 0 RPM (min = 0 RPM)
fan2: 1253 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan7: 0 RPM (min = 0 RPM)
SYSTIN: +33.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +35.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +78.0°C sensor = thermistor
AUXTIN1: +11.0°C sensor = thermistor
AUXTIN2: +20.0°C sensor = thermistor
AUXTIN3: +82.0°C sensor = thermistor
PECI Agent 0 Calibration: +35.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +42.6°C
intrusion0: ALARM
intrusion1: OK
beep_enable: disabled

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +42.6°C
Tccd1: +36.4°C

root@dsdaqgw:~#
</pre>

= Enable CPU turbo mode =

* Intel CPU has a nominal CPU frequency (i.e. 3.4GHz) and a turbo-boost CPU frequency (i.e. 4.0GHz). Here we will enable this turbo-boost mode.
* Find out CPU capability
<pre>
root@daq01:~# lscpu | grep Hz
Model name: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
CPU MHz: 3965.803
CPU max MHz: 4000.0000
CPU min MHz: 800.0000
root@daq01:~#
</pre>
* Look up this CPU in the Intel ARK database - google for the CPU model name, i.e.
https://ark.intel.com/content/www/us/en/ark/products/88196/intel-core-i7-6700-processor-8m-cache-up-to-4-00-ghz.html
* Find current frequency settings:
<pre>
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "powersave" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 2.72 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
root@daq01:~#
</pre>
* Note the following:
** current governor is "powersave"
** "performance" governor is available
** "boost state support" is supported and active.
* Confirm CPU frequency governor:
<pre>
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
powersave
powersave
powersave
powersave
powersave
powersave
powersave
powersave
root@daq01:~#
</pre>
* Change governor to "performance":
<pre>
root@daq01:~# cpupower frequency-set --governor performance
Setting cpu: 0
Setting cpu: 1
Setting cpu: 2
Setting cpu: 3
Setting cpu: 4
Setting cpu: 5
Setting cpu: 6
Setting cpu: 7
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
performance
performance
performance
performance
performance
performance
performance
performance
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "performance" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 3.93 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
</pre>
* monitor CPU frequency:
<pre>
root@daq01:~# cpupower monitor
| Nehalem || Mperf || Idle_Stats
CPU| C3 | C6 | PC3 | PC6 || C0 | Cx | Freq || POLL | C1 | C1E | C3 | C6 | C7s | C8
0| 0.00| 0.00| 0.00| 0.00|| 88.80| 11.20| 3973|| 0.00| 0.00| 0.01| 0.02| 0.31| 0.00| 4.25
4| 0.00| 0.00| 0.00| 0.00|| 4.70| 95.30| 3945|| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 95.03
1| 0.73| 3.70| 0.00| 0.00|| 4.52| 95.48| 3864|| 0.00| 0.01| 1.19| 0.44| 2.82| 0.00| 90.23
5| 0.73| 3.70| 0.00| 0.00|| 0.37| 99.63| 3807|| 0.00| 0.00| 0.03| 0.09| 1.70| 0.00| 97.64
2| 2.28| 12.86| 0.00| 0.00|| 1.41| 98.59| 3829|| 0.00| 0.86| 3.17| 0.46| 7.70| 0.00| 85.87
6| 2.28| 12.86| 0.00| 0.00|| 2.88| 97.12| 3856|| 0.00| 0.11| 4.56| 2.15| 10.31| 0.00| 78.99
3| 1.33| 4.81| 0.00| 0.00|| 0.99| 99.01| 3804|| 0.00| 0.49| 0.79| 0.01| 1.03| 0.00| 96.12
7| 1.34| 4.81| 0.00| 0.00|| 1.26| 98.74| 3818|| 0.00| 0.01| 2.32| 0.47| 5.02| 0.00| 90.06
root@daq01:~#
</pre>
* check that the CPU is not overheating:
<pre>
root@daq01:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 1: +38.0°C (high = +84.0°C, crit = +100.0°C)
Core 2: +34.0°C (high = +84.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +84.0°C, crit = +100.0°C)
</pre>
* congratulations, we are running at 4 GHz now!

= Setup ubuntu as gateway to private network =

See also:
* https://daq.triumf.ca/DaqWiki/index.php/VME-CPU#Setup_the_boot_host_computer_.28el7.29
* http://www.triumf.info/wiki/DAQwiki/index.php/Dhcpd_on_eth1

== Steps to do ==

!!! UPDATED 16feb2024 Ubuntu-22.04.03 !!!

* assign network numbers to the private network, i.e. 192.168.1.x, 192.168.2.x, etc
* (on the gateway machine, each private network interface has to have a different network number)
* (each network interface can have multiple networks attached, via VLANs or via eth0:0, eth0:1 constructs)
* assign IP addresses on the private network, save them in /etc/hosts i.e. "hvps 192.168.1.10"
* (for simplicity, assign 192.168.1.1 to the gateway machine itself)
* (IP addresses 192.168.1.0 and 192.168.1.255 are "special", do not use them)
* setup DNS server (dnsmasq) to serve contents of /etc/hosts via DNS (otherwise, many programs will see inconsistent name to IP address mapping)
* setup DHCP server (dnsmasq) to give out the IP addresses
* setup TFTP server (dnsmasq), pxelinux and NFS for diskless booting
* setup time server (chronyd) to provide common time to all devices
* setup NAT so machines on private network can access the internet (to get OS updates, etc)
* setup NIS and NFS so machines on the private network can use common home directories
* setup rsync backup of machines on the private network

== setup hosts ==

* edit /etc/hosts
<pre>
192.168.1.101 dsfe01
... and so forth
</pre>

== setup dns and dhcp ==

!!! updated 16feb2024 for Ubuntu 22.04.3 !!!

!!! note: stock systemd-resolved remains, is configured to forward queries to dnsmasq, configured to forward queries to TRIUMF DNS !!!

!!! note: per authors of systemd, bare hostnames are not permitted, a DNS domain name must always be used. DNS domain name "dsdaq" is used in this example !!!

* apt install dnsmasq
* ensure dnsmasq starts after all interfaces are up (Ubuntu-22)
<pre>
mkdir /etc/systemd/system/dnsmasq.service.d
echo -e "[Unit]\nAfter=network-online.target\n" > /etc/systemd/system/dnsmasq.service.d/local.conf
</pre>
* edit /etc/dnsmasq.conf
<pre>
# /etc/dnsmasq.conf
# DNS settings
#port=0 # disable DNS function
port=53 # enable DNS function
bind-interfaces # do not collide with systemd-resolved, we use 127.0.0.1:53, they use 127.0.0.53:53
domain-needed
bogus-priv
no-resolv
#log-queries # log DNS quesries

# TRIUMF DNS settings

server=142.90.100.19
expand-hosts
domain=dsdaq
local=/dsdaq/
localmx # do not forward MX queries to TRIUMF

# DHCP settings
interface=enp1s0f0 # VX network 192.168.0.x
#interface=missing # FEP and TSP network 192.168.1.x
interface=enp1s0f1 # controls network 192.168.2.x
#dhcp-range=192.168.1.50,192.168.1.150,infinite
dhcp-range=192.168.0.0,static
dhcp-range=192.168.2.0,static
log-dhcp # log DHCP queries
#quiet-dhcp
dhcp-ignore=tag:!known
#dhcp-boot=pxelinux.0

dhcp-option=option:dns-server,192.168.0.248
dhcp-option=option:ntp-server,192.168.0.248

# TFTP settings

enable-tftp
tftp-root=/tftpboot
</pre>
* #mkdir /tftpboot ### per tftp-root (if no ZFS)
* zfs create -o mountpoint=/tftpboot rpool/tftpboot ### (if root is ZFS)
* create resolved-dsdaq.conf with main IP address of dnsmasq
<pre>
[Resolve]
DNS=192.168.0.248
Domains=dsdaq triumf.ca
</pre>
* mkdir -p /etc/systemd/resolved.conf.d/
* /bin/rm -f /etc/systemd/resolved.conf.d/*.conf
* cp resolved-dsdaq.conf /etc/systemd/resolved.conf.d/
* systemctl stop systemd-resolved.service
* systemctl disable systemd-resolved.service
* systemctl enable dnsmasq
* systemctl restart dnsmasq
* try to "ping" or "host" some names from /etc/hosts, it should work
* try to ping daq00, daq00.triumf.ca, all should work
* resolved-dsdaq.conf goes into /etc/systemd/resolved.conf.d/ of all machines on the private network
* if not using systemd-resolved, edit /etc/resolv.conf

== setup chronyd ==

* enable ntp server:
* disable systemd-timesyncd, configure and enable chronyd per instructions above
* create dsdaq.conf
<pre>
# chrony config for dsdaq server

#allow 192.168.0.0
#allow 192.168.1.0
#allow 192.168.2.0
allow all

# end
</pre>
* cp dsdaq.conf /etc/chrony/conf.d/
* systemctl restart chronyd
* chronyc tracking ### wait until time is synchronized (a few seconds)
* create dsdaq.sources # use hostname or IP address of chronyd server
<pre>
# Put this file in /etc/chrony/sources.d
# systemctl restart chrony
# chronyc sources
# chronyc tracking
server dsdaqgw iburst prefer
# end
</pre>
* dsdaq.sources goes to /etc/chrony/sources.d of all machines on the private network

== setup diskless network booting ==

=== setup pxelinux for legacy pxe boot ===

* add bits in dnsmasq.conf
<pre>
dhcp-host=ac:1f:6b:9e:7f:4a,dsfe01,infinite
dhcp-boot=pxelinux.0
dhcp-option=17,"192.168.0.251:/nfsroot/%s,vers=3"
</pre>
* setup pxelinux for Ubuntu-18
<pre>
cd ~
wget https://www.kernel.org/pub/linux/utils/boot/syslinux/4.xx/syslinux-4.03.tar.bz2
tar xjvf syslinux-4.03.tar.bz2
cd syslinux-4.03
cp -pv ./core/pxelinux.0 ./com32/hdt/hdt.c32 ./memdisk/memdisk ./com32/menu/menu.c32 /zssd/tftpboot/
</pre>
* cd /zssd/tftpboot
<pre>
wget http://ladd00.triumf.ca/tftpboot/memtest86+-4.20.iso.zip
wget http://ladd00.triumf.ca/tftpboot/memtest86+-5.01.iso.gz
wget http://ladd00.triumf.ca/tftpboot/modules.alias
wget http://ladd00.triumf.ca/tftpboot/modules.pcimap
wget http://ladd00.triumf.ca/tftpboot/pci.ids
</pre>
* mkdir pxelinux.cfg
* emacs -nw pxelinux.cfg/default
<pre>
default menu.c32
prompt 0

menu title Welcome to the DSVSLICE PXE boot menu

timeout 50

label hdt
kernel hdt.c32

label memtest86+-5.01
kernel memdisk iso initrd=memtest86+-5.01.iso.gz

label memtest86+-4.20
kernel memdisk iso initrd=memtest86+-4.20.iso.zip

label vmlinuz-5.3.0-26-generic
menu default
kernel vmlinuz-5.3.0-26-generic
append initrd=initrd.img-5.3.0-26-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.1.1:/zssd/nfsroot/dsfe01 toram ip=dhcp panic=60 BOOTIF=enp1s0f0

#end
</pre>

=== setup pxelinux for efi pxe boot ===

* https://c-nergy.be/blog/?p=13808
* add dnsmasq.conf bits. note: root-path does not actually work, it is hardwired pxelinux.cfg/default file.
<pre>
# uefi pxe

dhcp-boot=tag:uefipxe,uefi/syslinux.efi
dhcp-option-force=tag:fe01,option:root-path,192.168.0.248:/nfsroot/fe01

# VX network 192.168.0.x

dhcp-host=40:a6:b7:c1:d9:c5,fe01,infinite,set:uefipxe,set:fe01
</pre>
* apt install syslinux pxelinux syslinux-common syslinux-efi syslinux-utils
<pre>
mkdir /tftpboot/uefi
cp /usr/lib/SYSLINUX.EFI/efi64/syslinux.efi /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/ldlinux.e64 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/menu.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/hdt.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libutil.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libmenu.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libcom32.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libgpl.c32 /tftpboot/uefi/
</pre>
* try to boot, it should bomb with "cannot load pxelinux.cfg/default"
* mkdir /tftpboot/uefi/pxelinux.cfg
* create /tftpboot/uefi/pxelinux.cfg/default, note nfsroot path is hardwired, note "http:" is used to load vmlinuz and initrd files (because tftp is super slow)
<pre>
default menu.c32
prompt 0

menu title Welcome to the DSDAQGW UEFI PXE boot menu

timeout 50

label vmlinuz-6.5.0-17-generic
kernel http://192.168.0.248:8088/uefi/vmlinuz-6.5.0-17-generic
append initrd=http://192.168.0.248:8088/uefi/initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=auto rw ip=dhcp panic=60

# append initrd=http://192.168.0.248:8088/uefi/initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.0.248:/nfsroot/fe01 rw ip=dhcp panic=60

# append initrd=initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.0.248:/nfsroot/fe01 rw ip=dhcp panic=60
# append initrd=initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=auto ip=dhcp rw panic=60

#end
</pre>
* try to boot, it will bomb with "cannot load http://...."
* install mini_httpd on port 8088, see https://acme.com/software/mini_httpd/
<pre>
apt install mini-httpd
emacs -nw /etc/default/mini-httpd # set "START=1"
emacs -nw /etc/mini-httpd.conf # set "host=192.168.0.248", "port=8088", "data_dir=/tftpboot"
mkdir /etc/systemd/system/mini-httpd.service.d
echo -e "[Unit]\nAfter=network-online.target\n" > /etc/systemd/system/mini-httpd.service.d/local.conf
systemctl enable mini-httpd
systemctl restart mini-httpd
systemctl status mini-httpd
wget http://192.168.0.248:8088/uefi/syslinux.efi
tail -100 /var/log/mini_httpd.log
</pre>
* fix initramfs bug for "nfsroot=auto", otherwise, "nfsroot=" has to be different for each machine and you have to have separate pxelinux config files for each machine
** emacs -nw /usr/lib/initramfs-tools/etc/dhcp/dhclient-enter-hooks.d/config
** add "echo ROOTPATH=..." if it is missing (Ubuntu LTS 22.04)
<pre>
echo "ROOTSERVER='${new_routers%% *}'"
echo "ROOTPATH='$new_root_path'"
echo "HOSTNAME='$new_host_name'"
</pre>
** regenerate initramfs (be careful you generate it for the right kernel!)
** see https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/2054482
<pre>
mkinitramfs 6.5.0-18-generic
</pre>
* copy linux kernel and initrd
<pre>
cp /boot/vmlinuz-6.5.0-18-generic /tftpboot/uefi/
cp /boot/initrd.img-6.5.0-18-generic /tftpboot/uefi/
chmod a+r /tftpboot/uefi/*
</pre>
* try to boot, should bomb with messages about "trying to mount root filesystem"
* tail /var/log/syslog
<pre>
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 vendor class: PXEClient:Arch:00007:UNDI:003016
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 1:netmask, 2:time-offset, 3:router, 4, 5,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 6:dns-server, 12:hostname, 13:boot-file-size,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 15:domain-name, 17:root-path, 18:extension-path,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 22:max-datagram-reassembly, 23:default-ttl,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 28:broadcast, 40:nis-domain, 41:nis-server,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 42:ntp-server, 43:vendor-encap, 50:requested-address,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 51:lease-time, 54:server-identifier, 58:T1,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 59:T2, 60:vendor-class, 66:tftp-server, 67:bootfile-name,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 97:client-machine-id, 128, 129, 130, 131,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 132, 133, 134, 135
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 next server: 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 broadcast response
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 1 option: 53 message-type 2
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 18 option: 67 bootfile-name uefi/syslinux.efi
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 12 hostname fe01
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 vendor class: PXEClient:Arch:00007:UNDI:003016
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 1:netmask, 2:time-offset, 3:router, 4, 5,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 6:dns-server, 12:hostname, 13:boot-file-size,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 15:domain-name, 17:root-path, 18:extension-path,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 22:max-datagram-reassembly, 23:default-ttl,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 28:broadcast, 40:nis-domain, 41:nis-server,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 42:ntp-server, 43:vendor-encap, 50:requested-address,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 51:lease-time, 54:server-identifier, 58:T1,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 59:T2, 60:vendor-class, 66:tftp-server, 67:bootfile-name,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 97:client-machine-id, 128, 129, 130, 131,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 132, 133, 134, 135
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 next server: 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 broadcast response
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 1 option: 53 message-type 5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 18 option: 67 bootfile-name uefi/syslinux.efi
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 12 hostname fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: error 8 User aborted the transfer received from 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/syslinux.efi to 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/syslinux.efi to 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 requested options: 1:netmask, 3:router, 6:dns-server
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 bootfile name: uefi/syslinux.efi
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 next server: 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 broadcast response
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 1 option: 53 message-type 2
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 requested options: 1:netmask, 3:router, 6:dns-server
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 bootfile name: uefi/syslinux.efi
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 next server: 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 broadcast response
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 1 option: 53 message-type 5
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/ldlinux.e64 to 192.168.0.110
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/01-40-a6-b7-c1-d9-c5 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8006E not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8006 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A800 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A80 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/pxelinux.cfg/default to 192.168.0.110
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/menu.c32 to 192.168.0.110
Feb 16 20:43:10 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/libutil.c32 to 192.168.0.110
Feb 16 20:43:10 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/pxelinux.cfg/default to 192.168.0.110
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 client provides name: dsdaqgw.triumf.ca
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 1:netmask, 28:broadcast, 2:time-offset, 3:router,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 15:domain-name, 6:dns-server, 119:domain-search,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 12:hostname, 44:netbios-ns, 47:netbios-scope,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 26:mtu, 121:classless-static-route, 42:ntp-server
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 bootfile name: uefi/syslinux.efi
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 next server: 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 1 option: 53 message-type 2
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 51 lease-time infinite
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 client provides name: dsdaqgw.triumf.ca
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 1:netmask, 28:broadcast, 2:time-offset, 3:router,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 15:domain-name, 6:dns-server, 119:domain-search,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 12:hostname, 44:netbios-ns, 47:netbios-scope,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 26:mtu, 121:classless-static-route, 42:ntp-server
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 bootfile name: uefi/syslinux.efi
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 next server: 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 1 option: 53 message-type 5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 51 lease-time infinite
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 12 hostname fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw rpc.mountd[3350210]: authenticated mount request from 192.168.0.110:981 for /nfsroot/fe01 (/nfsroot/fe01)
Feb 16 20:45:07 dsdaqgw rpc.mountd[3350210]: authenticated unmount request from 192.168.0.110:859 for /nfsroot/fe01/tmp/autoDY4k5u (/nfsroot/fe01)
</pre>
* tail /var/log/mini_httpd.log
<pre>
192.168.0.110 - - [16/Feb/2024:20:43:15 -0800] "GET /uefi/vmlinuz-6.5.0-17-generic HTTP/1.0" 200 14227944 "" "Syslinux/6.04"
192.168.0.110 - - [16/Feb/2024:20:43:24 -0800] "GET /uefi/initrd.img-6.5.0-17-generic HTTP/1.0" 200 137824833 "" "Syslinux/6.04"
</pre>

=== setup efi http boot ===

https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-deployment-prep-uefi-httpboot.html

=== setup linux kernel ===

* copy the kernel files
<pre>
cd /boot
rsync -av config* initrd* System.map* vmlinuz* /tftpboot/
</pre>
* cd /tftpboot
* chmod a+r *

=== setup nfs ===

* apt-get install nfs-kernel-server
* enable NFS over UDP, edit /etc/nfs.conf add "udp=y":
<pre>
udp=y
</pre>
<pre>
systemctl restart nfs-server.service
</pre>
* emacs -nw /etc/exports
<pre>
/nfsroot/dsfe01 dsfe01(rw,no_root_squash,async,no_subtree_check)
</pre>
* enable services
<pre>
systemctl enable nfs-server
systemctl enable nfs-mountd
systemctl enable nfs-idmapd
systemctl restart nfs-server
systemctl restart nfs-mountd
systemctl restart nfs-idmapd
</pre>
* after editing /etc/exports, run
<pre>
exportfs -av
</pre>

=== setup userland ===

!!! ubuntu-18 version !!!

* zfs create rpool/nfsroot
* zfs set dedup=verify rpool/nfsroot ### enable deduplication to save disk space because most linux images have mostly identical files
* clone ubuntu
<pre>
mkdir /nfsroot/dsfe01
cd /
rsync -avx . /nfsroot/dsfe01
</pre>
* edit config files:
* cd /nfsroot/dsfe01
* emacs -nw etc/hostname ### change to dsfe01
* emacs -nw etc/mailname ### change to dsfe01
* emacs -nw etc/yp.conf ### change daq00.triumf.ca to musr00.triumf.ca
* emacs -nw etc/defaultdomain ### change to MUSR-NIS
* cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
* emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
* emacs -nw root/.ssh/authorized_keys ### update root ssh keys
* emacs -nw etc/fstab ### add this
<pre>
192.168.1.1:/nfsroot/dsfe01 / nfs defaults,nolock 0 0
</pre>
* emacs -nw etc/chrony/chrony.conf
** comment-out all "pool" and "server" entries
** add entry "server 192.168.1.1 iburst"

After dsfe01 is booted:

* disable services:
<pre>
systemctl disable apache2
systemctl disable dnsmasq
systemctl disable zfs-import-cache
</pre>

To setup additional machines, clone dsfe01 instead of cloning the gateway machine

=== Allow manpages to be viewed ===

If <code>/</code> is mounted over NFS, <code>man</code> will report a permission error. Fix it with:

<pre>
ln -s /etc/apparmor.d/usr.bin.man /etc/apparmor.d/disable/
apparmor_parser -R /etc/apparmor.d/usr.bin.man
</pre>

== setup shared home directory ==

=== on the gateway machine ===
* define netgroups
* emacs -nw /etc/netgroup
<pre>
dsfe (dsfe01,,) (dsfe02,,)
</pre>
* emacs -nw /etc/nsswitch.conf ### edit the netgroup line to read:
<pre>
netgroup: files
</pre>
* export the home directories:
* emacs -nw /etc/exports ### add this:
<pre>
/zssd/home1 @dsfe(rw,no_root_squash,async,no_subtree_check)
</pre>
* exportfs -rc

=== on the frontend machine ===

* mkdir /home
* emacs -nw /etc/fstab ### add this:
<pre>
192.168.1.1:/zssd/home1 /home nfs defaults 0 0
</pre>
* mount -a

== setup NAT ==

NAT allows machines on the private network to connect to the internet: https://en.wikipedia.org/wiki/Network_address_translation

In these examples:
* replace "eno1" with name of the outgoing interface (the one connected to the TRIUMF network).
* replace "enp11s0" with name of the private network interface (192.168.1.x network)

* emacs -nw /etc/rc.local ### add this:
<pre>
# /etc/rc.local

# enable NAT

/sbin/iptables -t nat -A POSTROUTING -o eno1 -j MASQUERADE
iptables -L -v

# uncomment following lines if machine has prohibitive FORWARD rules:
#/sbin/iptables -I FORWARD -i eno1 -o enp11s0 -m state --state RELATED,ESTABLISHED -j ACCEPT
#/sbin/iptables -I FORWARD -i enp11s0 -o eno1 -j ACCEPT
#iptables -L -v

iptables -L -v
sysctl -w net.ipv4.ip_forward=1
#sysctl -a | grep forward

sh /etc/firewall-rfc1918.sh

# end
</pre>
* emacs -nw /etc/firewall-rfc1918.sh
<pre>
# firewall-rfc1918.sh

# prevent RFC1918 private network IP addresses from
# going in and out from our uplink.

ETH=eno1

iptables -F in-rfc1918
iptables -N in-rfc1918
iptables -A in-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A in-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A in-rfc1918 --dst 192.168.0.0/16 -j REJECT

iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -I INPUT -j in-rfc1918 -i $ETH

iptables -F out-rfc1918
iptables -N out-rfc1918
iptables -A out-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A out-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A out-rfc1918 --dst 192.168.0.0/16 -j REJECT

iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -I OUTPUT -j out-rfc1918 -o $ETH

iptables -D FORWARD -j out-rfc1918 -o $ETH
iptables -D FORWARD -j out-rfc1918 -o $ETH
iptables -I FORWARD -j out-rfc1918 -o $ETH

# allow TRIUMF-SECURE network

iptables -I in-rfc1918 -s 10.90.0.0/255.255.0.0 -j ACCEPT
iptables -I out-rfc1918 -d 10.90.0.0/255.255.0.0 -j ACCEPT

# show configuration

iptables -L -v

#end
</pre>

= KVM =

<pre>
apt install cpu-checker

root@daq13:~# kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
root@daq13:~#

(if not, shutdown, go into BIOS settings, enable CPU virtualization)

apt install virtinst ### will install many packages
apt install libvirt-clients libvirt-daemon-system-systemd libvirt-daemon qemu qemu-kvm libvirt-daemon-system virtinst bridge-utils

root@daq13:/home1/wheel# virsh list --all
Id Name State
------------------------------
1 ubuntu-guest running

apt install virt-manager

virt-install --name ubuntu-guest --os-variant ubuntu20.04 --vcpus 2 --ram 2048 --location /daq/daqstore/olchansk/linux/Ubuntu/ubuntu-20.04.3-desktop-amd64.iso --network bridge=virbr0,model=virtio --graphics none --extra-args='console=ttyS0,115200n8 serial'

virtual machine will start, boot, etc
to get out of it, CTRL + Shift followed by ]

ssh wheel@daq13
virt-manager

run virt-install again, omit "--graphics none", open graphics console from virt-manager, it booted into ubuntu installer desktop

virt-install --name test10 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --filesystem /kvm_ladd00,/ --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial" --graphics none

virt-install --name test14 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --disk /tmp/xxx/ladd00.img,bus=sata --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial rdshell" --graphics none --check path_in_use=off
</pre>

build image

<pre>
dd if=/dev/zero of=/tmp/xxx/ladd00.img bs=1024M count=20
mkfs.ext3 /tmp/xxx/ladd00.img ### ext4 fails to mount by SL6 kernel, "unknown ext4 options"
cd /kvm_ladd00/
mount -o loop /tmp/xxx/ladd00.img /mnt/tmp
rsync -av . /mnt/tmp/ --delete
umount /mnt/tmp
</pre>

on the guest, configure network: /etc/rc.local
<pre>
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

ifconfig eth2 192.168.122.2
route add -net 0.0.0.0 gw 192.168.122.1
ifconfig -a
netstat -rn

# end
</pre>

= ARM64 cross-compiler =

* arm64, aarch64 are Xilinx FPGA Cortex-A53, RPi4, RPi5 machines
* install packages:
<pre>
apt install g++-12-aarch64-linux-gnu gcc-12-aarch64-linux-gnu-base libstdc++-12-dev-arm64-cross
</pre>
* run:
<pre>
aarch64-linux-gnu-gcc-12 -o ttcp.aarch64 ttcp.c -static
aarch64-linux-gnu-g++-12 -o fecdm.exe -O2 -g -Wall -Wuninitialized -std=c++20 fecdm.o dsdm.o /home/dsdaqdev/packages_common/midas/linux-aarch64-remoteonly/lib/libmidas.a -pthread -lrt -lutil /nfsroot/gdm00/usr/lib/aarch64-linux-gnu/libi2c.a -static
</pre>

= ARM cross-compiler =

NOTE!!!

THIS IS NOT AN AARCH64 (arm64) CROSSCOMPILER!

NOTE!!!

* install packages:
<pre>
apt install libgcc-9-dev-arm64-cross
apt install gcc-arm-linux-gnueabi
apt install gcc-arm-linux-gnueabihf
apt install g++-arm-linux-gnueabihf
apt install g++-arm-linux-gnueabi
</pre>
* find out the correct -march setting, on the target machine, run:
<pre>
root@gdm00:~# g++ -Q --help=target | grep march
-march= armv8-a
</pre>

<pre>
arm-linux-gnueabi-gcc -o ttcp1 ttcp.c -march=armv7 -static
arm-linux-gnueabi-gcc -o memcpy.armv7 memcpy.cc -march=armv7 -static -O2
</pre>

= 32-bit intel cross-compiler =

Ubuntu 22.04:

<pre>
apt install libstdc++-11-dev:i386
apt install zlib1g-dev:i386
</pre>

NOTES:
* "g++ -m32" does not find libstdc++, please use "g++ -m32 -L/usr/lib/gcc/i686-linux-gnu/11/"
* to cross-build 32-bit MIDAS, use "make linux32".
* executables cross-build on Ubuntu-22 do NOT run on 32-bit Debain-11 (GLIBC and GLIBCXX version mismatch)
* executables cross-build on Ubuntu-22 run on 32-bit Debian-12.

= SSH settings for EPICS =

* TRIUMF EPICS runs obsolete version of SSH
* add this to the use .ssh/config
<pre>
Host sbp1*
HostKeyAlgorithms +ssh-rsa
PubKeyAcceptedAlgorithms +ssh-rsa
KexAlgorithms +diffie-hellman-group1-sha1
ForwardX11 yes
ForwardX11Trusted yes
</pre>

= changes for VME processors =

<pre>
apt -y remove sysstat man-db
apt -y purge dkms
apt -y purge mdadm
apt -y autoremove
</pre>

Ubuntu

2024-11-05T23:23:53Z

Mwalczak: /* About Ubuntu */

= Prerequisites =

* before setting up new machine run memory test
* prepare flash drive with free version of memtest86: https://www.memtest86.com
* test boot from flash drive, test takes ~ few hours
* test will end with summary page, if passed continue with Ubuntu
* number that might be worth noting is memory latency

= Ubuntu version =

<pre>
lsb_release -a
uname -a
</pre>

= Ubuntu installer =

* updated for Ububtu LTS 20.04.01, 22.04.1, 24.04 (only minor differences)

* download the latest Ubuntu LTS desktop installer iso image
* dd the image to a USB key
* power down, disconnect all disks (all HDDs, all SSDs, all M.2)
* connect the SSD to be used as system disk
* if system will use mirrored SSDs (using ZFS mirror), leave second SSD disconnected, we will activate it later
* power up
* boot from USB key in legacy mode or UEFI mode (select this in the BIOS boot menu - F8 for ASUS, F11 for Supermicro)
* follow the instruction:
* "try ubuntu or install ubuntu" - choose "install"
* select language - accept default
* "updates and other software" - accept default settings ("normal install")
* "installation type" - select "advanced features" and "experimental: use ZFS"
* accept partition choice
* "where are you?" - select "Vancouver" (PST time zone)
* "who are you?" - leave all fields blank, except "username" set to "wheel", "password" set to the root password. hostname will be set later after configuring the network
* installation runs in a few minutes, when finished, reboot
* login as user wheel
* answer annouying questions:
* "livepatch" - say "next"
* "help improve" - select "do not send", say "next"
* "privacy" - leave "location" as "off", say "next"
* "ready to go", say "done"
* right-click on the desktop, say "open in terminal", a shell will open
* say "sudo /bin/bash", enter the root password, you now have the root shell
* run nm-connection-editor to configure the network. use netmask 255.255.224.0, gateway 142.90.100.18, DNS 142.90.100.19, search path "triumf.ca"
* after network is up (can ping ladd00), continue with post-installation steps below

= Install instructions =

== prepare ==
<pre>
apt update
apt upgrade
</pre>

== install ssh ==
<pre>
apt install ssh
</pre>

== install git/scripts ==
<pre>
apt -y install git
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd scripts
git pull
</pre>

* if needed, update git/scripts repository from ladd00 to daq00:
* git remote -v ### if it says daq00, we are good
* git remote set-url origin https://daq00.triumf.ca/~olchansk/git/scripts.git
* git pull ### check that it works

== configure hostname ==
<pre>
vi /etc/hostname
</pre>

== disable swap ==

ubuntu installer creates a 2 GB swap partition, not useful
on 32-64 GB machine, disable it:
<pre>
vi /etc/fstab ### comment out the "swap" line
</pre>

== maybe reboot ==

this is a good point to reboot the machine to boot
the latest kernel and to set the correct hostname

== install etckeeper ==

keep contents of /etc in a git repository:

<pre>
apt -y install etckeeper
</pre>

== set timezone ==
<pre>
timedatectl list-timezones | grep -i vancouver
timedatectl set-timezone America/Vancouver
</pre>

== install time synchronization ==
<pre>
apt -y install chrony
#echo server time1.triumf.ca iburst >> /etc/chrony/chrony.conf
#echo server time2.triumf.ca iburst >> /etc/chrony/chrony.conf
#echo server time3.triumf.ca iburst >> /etc/chrony/chrony.conf
cd ~/git/scripts
git pull
cd ~
cp ~/git/scripts/etc/triumf.sources /etc/chrony/sources.d/
systemctl disable systemd-timesyncd.service
systemctl stop systemd-timesyncd.service
systemctl disable ntp
systemctl stop ntp
systemctl enable chrony
systemctl restart chrony
chronyc sources
chronyc tracking
</pre>

NOTE1: if time1, time2, time3 are already listed in /etc/crony/chrony.conf, please remove them and restart chrony.

NOTE2: if time1, time2, time3 are not listed in "chronyc tracking" or if they are not selected by "chronyc tracking", check that /etc/crony/chrony.conf contains "sourcedir /etc/chrony/sources.d", see NOTE4.

NOTE3: read https://chrony-project.org/faq.html#_should_i_prefer_chrony_over_timesyncd_if_i_do_not_need_to_run_a_server

NOTE4: update very old chrony config file, remove chrony, than install it from scratch as above
<pre>
grep sourcedir /etc/chrony/chrony.conf ### if we have it, we are good
apt remove chrony
apt purge chrony
</pre>

== reenable systemd-timesyncd ==

ONLY IF CHRONY DOES NOT WORK

To configure systemd-timesyncd, set "NTP=" in /etc/systemd/timesyncd.conf

<pre>
apt remove chrony
cat /etc/systemd/timesyncd.conf
systemctl enable systemd-timesyncd.service
systemctl restart systemd-timesyncd.service
systemctl status systemd-timesyncd.service
timedatectl status
timedatectl timesync-status
</pre>

== enable outgoing email (debian 11) ==

this is different from ubuntu 20. it uses /etc/mailname and it hardwires the hostname into main.cf.

== enable outgoing email ==

we have an unusual email configuration. outgoing email should work to deliver error messages, notices, etc. incoming email is disabled, we do not receive email for local users.

this causes problems with TRIUMF smtp server. if our message cannot be delivered (wrong email address or receipient computer is turned off), TRIUMF smtp server will generate a delivery failure notification email and try to send it to the "from" address of the failed message. but the "from" address does not receive any email, so another delivery failure notification email is generated and an attempt to deliver it. which again fails, rinse and repeat.

as solution, kray created a special rule, email from scrap.triumf.ca does not generate delivery failure notices. failed messages sit in the queue for 5 days, then they are deleted. (K.O. - confirmed with kray 3jan2024).

to make this work we use the msmtp MTA package.

<pre>
cd ~
apt -y remove postfix
apt -y purge postfix # remove old config files
apt -y install mailutils msmtp msmtp-mta # say "no" to apparmor support
apt -y install bsd-mailx
cd ~/git/scripts/etc
git pull
/bin/cp -fv aliases /etc/aliases
/bin/cp -fv msmtprc /etc/msmtprc
/bin/rm -vf ~root/.forward
/bin/rm -vf /etc/mailname
Mail root
Subject: test
test
^D
CC: <CR>
</pre>

== enable outgoing email (postfix) ==

THIS IS OBSOLETE!!!

* TRIUMF: use smtp.triumf.ca
* CERN: use cernmx.cern.ch

<pre>
apt install postfix ### select "satellite system", enter full hostname "xxx.triumf.ca", enter "smtp.triumf.ca"
apt install mailutils
dpkg-reconfigure postfix ### (if postfix already installed)
</pre>
<pre>
echo olchansk@triumf.ca lindner@triumf.ca bsmith@triumf.ca >> ~root/.forward
mailx root
test
^D
</pre>

== enable ping for all users (debian 11) ==

Without this tweak, Debian will report "operation not permitted" if a user tries to ping somewhere.

<pre>
echo 'net.ipv4.ping_group_range = 0 1000' > /etc/sysctl.d/99-ping.conf
</pre>

== disable apparmor ==

On NFS-Root network booted machines!

If "man man" returns "permission denied" and syslog reports apparmor "sendmsg DENIED" errors, disable apparmor. This is supposedly fixed in kernel 6.0 and later (to be confirmed), see https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/1784499

Disable apparmor, see https://ubuntu.com/server/docs/security-apparmor

This takes effect after a reboot.

<pre>
systemctl stop apparmor.service
systemctl disable apparmor.service
</pre>

== install missing packages ==

(apt eats terminal input, even the "yes |" trick does not quite work,
repeat the following commands until they report that everything
is installed)

<pre>
yes | apt -y install ssh tcsh ethtool ncat rsync strace net-tools traceroute time minicom screen git lsof debsums tmux iptables telnet
yes | apt -y install sysstat smartmontools lm-sensors
yes | apt -y install lsb-release
apt -y install vim # in addition to default vim-tiny, requested by IRIS
apt -y install tcl
apt -y install pax rpm alien ### package converter tools
yes | apt -y install flex bison
yes | apt -y install neofetch
yes | apt -y install snmp snmp-mibs-downloader
yes | apt -y install git subversion g++ gfortran cmake doxygen
yes | apt -y install curl libcurl4 libcurl4-openssl-dev
yes | apt -y install mariadb-client ### mysql client
yes | apt -y install libz-dev libzstd-dev sqlite3 libsqlite3-dev unixodbc-dev
yes | apt -y install libssl-dev
yes | apt -y install emacs xemacs21 joe
yes | apt -y install gnuplot dos2unix
yes | apt -y install mutt bsd-mailx # email clients
yes | apt -y install liblz4-tool pbzip2
yes | apt -y install libc6-dev-i386 # otherwise no /usr/include/sys/types.h
yes | apt -y install libreadline-dev
yes | apt -y install ubuntu-mate-themes
yes | apt -y install libmotif-dev libxmu-dev
yes | apt -y install libusb-dev libusb-1.0-0-dev
yes | apt -y install i2c-tools libi2c-dev libi2c0
yes | apt -y install xfig gsfonts-x11 gsfonts-other # install fonts for xfig
yes | apt -y install libjson-perl
yes | apt -y install libgsl-dev # additional GNU Scientific Library
yes | apt -y install qt5-default # Qt development
yes | apt -y install python3-full python3-dev python3-dbg python3-pip ### for pyROOT
yes | apt -y install imagemagick imagemagick-common ckeditor # for elog
yes | apt -y install libjpeg-dev libjpeg-progs libjpeg-tools
yes | apt -y install linux-tools-common linux-tools-generic # cpupower frequency-info
yes | apt -y install rdesktop remmina remmina-plugin"*" # requested by POL
yes | apt -y install nlohmann-json3-dev # required to build MIDAS with ROOT 6.30 on Ubuntu-22
apt -y install dpkg-dev cmake g++ gcc binutils libx11-dev libxpm-dev libxft-dev libxext-dev python3 libssl-dev libafterimage0 # from https://root.cern/install/dependencies/
apt -y install gfortran libpcre3-dev xlibmesa-glu-dev libglew-dev libftgl-dev libmysqlclient-dev libfftw3-dev libcfitsio-dev graphviz-dev libldap2-dev python3-dev python3-numpy libxml2-dev libkrb5-dev libgsl0-dev qtwebengine5-dev nlohmann-json3-dev libtbb-dev libavahi-compat-libdnssd-dev # from https://root.cern/install/dependencies/
apt -y install libvdt-dev # for ROOT 6.32 on Ubuntu-24
apt -y install u-boot-tools # for Xilinx petalinux
#apt -y install linux-headers-generic # to build linux kernel drivers
</pre>

Ubuntu LTS 20.04:
<pre>
yes | apt -y install linux-image-generic-hwe-20.04 linux-tools-virtual-hwe-20.04 # enable linux 5.11 series kernel
</pre>

Ubuntu LTS 22.04:
<pre>
apt -y install linux-generic-hwe-22.04 # enable linux 6.2.0 series kernel
</pre>

Ubuntu LTS 24.04:
<pre>
apt -y install linux-generic-hwe-24.04 # enable linux 6.8.0 series kernel
</pre>

== disable swap (debian 11) ==

* on 64 GB RAM machines swap is not useful
* on machines booted from network (NFS-ROOT), swap does not work
* on machines running from flash (RPi, etc), flash is too slow for useful swap
* swap configured by linux installers invariably has wrong size and is not useful

<pre>
systemctl disable dphys-swapfile
systemctl stop dphys-swapfile
dphys-swapfile uninstall
</pre>

== configure DNS ==

<pre>
cd ~/git/scripts
git pull
mkdir /etc/systemd/resolved.conf.d
cp etc/resolved-triumf.conf /etc/systemd/resolved.conf.d/
systemctl restart systemd-resolved
resolvectl
#systemd-analyze cat-config systemd/resolved.conf
</pre>

== install ganglia ==

<pre>
apt -y install ganglia-monitor
cd ~root/git/scripts/ganglia
git pull
make install
./ganglia-all.perl
</pre>

fix gmond start before network is ready:

<pre>
mkdir /etc/systemd/system/ganglia-monitor.service.d
echo -e "[Unit]\nAfter=network-online.target\n" > /etc/systemd/system/ganglia-monitor.service.d/local.conf
systemctl daemon-reload
systemctl cat ganglia-monitor.service
</pre>

== install gonodeinfo ==

* go to https://bitbucket.org/dd1/gonodeinfo follow instructions:
<pre>
yes | apt-get -y install golang
mkdir ~/git
cd ~/git
#git clone https://bitbucket.org/dd1/gonodeinfo.git
git clone https://daq00.triumf.ca/~olchansk/git/gonodeinfo.git
cd gonodeinfo
git remote set-url origin https://daq00.triumf.ca/~olchansk/git/gonodeinfo.git
git pull
make
make install # install gonodeinfo agent
cd ~ # this is important
</pre>
* edit /etc/gonodeinfo.conf
* change "Description", "Location", "User" and "Administrator" as appropriate (or delete them)
* change "Servers" to read: Servers: daq00.triumf.ca:8601
* run "gonodeinfo -v"
* if error is "connection refused". go to the nodeinfo server to add this client to the access control list:
* on the gonodeinfo server: run /opt/gonodeinfo/gonodereceive.exe -a daq13
* try gonodeinfo again, there should be no error
* on the gonodeinfo server: run gonodereport, look at the web pages, the new machine should be listed now

== install fonts for EPICS ==

* apt install xfonts-100dpi xfonts-75dpi
* restart Xorg (i.e. "killall Xorg", this will log you out from the console)
* xlsfonts | grep -i helvetica ### should show fonts with different sizes, not just size 0 (scalable)

== install libz.so.1 for CentOS compatibility ==

KO - confirm which versions on quartus need this.

<pre>
yes | apt-get -y install zlib1g
yes | apt-get -y install zlib1g:i386 libc6:i386 libgcc1:i386 gcc-6-base:i386
</pre>

== install libpng12.so.0 for Quartus compatibility ==

(does not work anymore!!!)

<pre>
wget http://ftp.ca.debian.org/debian/pool/main/libp/libpng/libpng12-0_1.2.50-2+deb8u2_amd64.deb
dpkg --install libpng12-0_1.2.50-2+deb8u2_amd64.deb
</pre>

== install libpng12.so.0 for Quartus 13.0sp1 ==

<pre>
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0.50.0
/bin/cp -pv libpng12.so.0 libpng12.so.0.50.0 /lib/x86_64-linux-gnu/
</pre>

== install packages for Xilinx ==

ubuntu LTS 22.04 vivado 2020.1

<pre>
apt install autoconf libtool
apt install libtinfo5
apt install texinfo
apt install zlib1g:i386
</pre>

== install packages for building ROOT ==

<pre>
apt -y install libx11-dev libxpm-dev libxft-dev libxext-dev libpng-dev libjpeg-dev xlibmesa-glu-dev libxml2-dev libgsl-dev cmake
</pre>

== install 32-bit libraries for PHYSICA ==

these instructions are for running 32-bit physica executable built for SL6 on ubuntu LTS 20.04

install physica sources (cannot build, do not have g77)

<pre>
cd ~/packages
git clone https://bitbucket.org/ttriumfdaq/physica.git
</pre>

install 32-bit libraries using ubuntu package manager:

<pre>
apt install lib32z1 # libz.so
</pre>

copy 32-bit SL6 shared libraries to /lib32

<pre>
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libX11.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libgd.so.2 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libpng12.so.0 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libreadline.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libncurses.so.5 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libg2c.so.0 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libxcb.so.1 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libXpm.so.4 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libjpeg.so.62 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libfontconfig.so.1 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libfreetype.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libtinfo.so.5 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libXau.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libexpat.so.1 /lib32/
</pre>

ldd should report:

<pre>
trinatdaq:trinat> ldd /usr/local/physica/physica.exe
linux-gate.so.1 (0xf7fa2000)
libX11.so.6 => /lib32/libX11.so.6 (0xf7e43000)
libgd.so.2 => /lib32/libgd.so.2 (0xf7dfe000)
libpng12.so.0 => /lib32/libpng12.so.0 (0xf7dd6000)
libz.so.1 => /lib32/libz.so.1 (0xf7db8000)
libreadline.so.6 => /lib32/libreadline.so.6 (0xf7d7e000)
libncurses.so.5 => /lib32/libncurses.so.5 (0xf7d5b000)
libg2c.so.0 => /lib32/libg2c.so.0 (0xf7d3d000)
libm.so.6 => /lib32/libm.so.6 (0xf7c39000)
libgcc_s.so.1 => /lib32/libgcc_s.so.1 (0xf7c1a000)
libc.so.6 => /lib32/libc.so.6 (0xf7a2f000)
libxcb.so.1 => /lib32/libxcb.so.1 (0xf7a05000)
libdl.so.2 => /lib32/libdl.so.2 (0xf79ff000)
libXpm.so.4 => /lib32/libXpm.so.4 (0xf79ee000)
libjpeg.so.62 => /lib32/libjpeg.so.62 (0xf7997000)
libfontconfig.so.1 => /lib32/libfontconfig.so.1 (0xf7962000)
libfreetype.so.6 => /lib32/libfreetype.so.6 (0xf78c9000)
libtinfo.so.5 => /lib32/libtinfo.so.5 (0xf78b0000)
/lib/ld-linux.so.2 (0xf7fa4000)
libXau.so.6 => /lib32/libXau.so.6 (0xf78ad000)
libexpat.so.1 => /lib32/libexpat.so.1 (0xf7885000)
trinatdaq:trinat>
</pre>

set login environment:

<pre>
setenv TRIUMF_FONTS $HOME/packages/physica/fonts
setenv PHYSICA_DIR $HOME/packages/physica
alias physica $PHYSICA_DIR/physica-SL6-32
</pre>

test:

<pre>
cd ~/packages/physica
physica
@rangauss.pcm
</pre>

== install wine ==

As far as I know, only needed for BNMR/BNQR

<pre>
apt install wine winetricks
</pre>

== install lightdm ==

unlike the default gdm login manager, lightdm shows the machine hostname and does not require an extra mouse click to swicth from screen saver to login mode.

<pre>
apt -y install lightdm
# select lightdm
</pre>

== install desktop environments ==

note: default display manager and default desktop are deficient, please do not skip this step.

note: if apt asks to choose the display manager, select "lightdm"

note: KO - I recommend the "MATE" desktop.

note: you will have to cut-and-paste this several times because "apt" eats commands, even with "-y" and even piped from "yes".

<pre>
# install MATE desktop
DEBIAN_FRONTEND=noninteractive apt -y install ubuntu-mate-core ubuntu-mate-desktop ubuntu-mate-themes
# install Cinnamon desktop
DEBIAN_FRONTEND=noninteractive apt -y install cinnamon
# install KDE desktop
DEBIAN_FRONTEND=noninteractive apt -y install kubuntu-desktop
# install Lxqt desktop
DEBIAN_FRONTEND=noninteractive apt -y install lxqt
# install Xfce4 desktop
DEBIAN_FRONTEND=noninteractive apt -y install xfce4
</pre>

== install ROOT ==

Please install ROOT per instructions at https://root.cern.ch.

NOTE1: The ROOT package available from Ubuntu repositories is severely out of date and cannot be used with MIDAS and ROOTANA. ### DO NOT DO THIS! apt-get install root-system

NOTE2: as of 2017-Jan-09, ROOT binary kits for Ubuntu do not work (use GCC 5 instead of GCC6), build from source instead.

== Install x2go ==

<pre>
apt-get update
apt-get install x2goserver x2goserver-xsession
</pre>

== enable root login from ladd00/daq00 ==
<pre>
ssh localhost
CTRL-C
/bin/cp ~root/git/scripts/etc/authorized_keys ~root/.ssh/
</pre>

== disable ssh access from outside of TRIUMF ==

to stop ssh login spam, disable ssh access from outside of TRIUMF. this can be done by requesting a firewall block through the helpdesk or by local firewall rule:

<pre>
echo iptables -I INPUT ! -s 142.90.0.0/255.255.0.0 -p tcp --dport 22 -j REJECT >> /etc/rc.local
/etc/rc.local
</pre>

== install smart-status ==
<pre>
ln -s ~/git/scripts/smart-status/smart-status.perl ~root/
</pre>

== enable boot menu and boot messages ==

This will enable the grub menu (with a 10 sec timeout) and
replace black screen with exciting linux boot messages.

* emacs -nw /etc/default/grub
<pre>
GRUB_DEFAULT=0
#GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="vga=769 video=640x480"
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX=""
#GRUB_GFXMODE=640x480
</pre>
* update grub config:
<pre>
grub-mkconfig -o /boot/grub/grub.cfg
</pre>

== reboot ==

this completes installation of the base system.

following sections modify basic ubuntu to fix known problems and to enable special stuff.

= Enable automatic updates =

<pre>
apt install unattended-upgrades
cd ~/git/scripts
git pull
/bin/cp -v etc/99apt-conf-ko /etc/apt/apt.conf.d/
apt-config dump | grep Unattended
</pre>

Following is obsolete:

* emacs -nw /etc/apt/apt.conf.d/50unattended-upgrades
** uncomment in Allowed-Origins "-security" and "-updates"
** add in Allowed-Origins: "Google LLC:stable";
** uncomment/add: "Unattended-Upgrade::Mail "root";
* emacs -nw /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
</pre>
* test: unattended-upgrade --dry-run -v

NOTE: update-on-shutdown is disabled.

NOTE: there is no update-on-boot, but:

NOTE: if machine was off for a long time, the systemd update timer would have expired and it will fire soon after reboot, causing an automatic update run. this is unwanted, and there is no fix or workaround for it. K.O. June-2023.

= Fix bpool is full (obsolete) =

THIS IS CAUSED BY OBSOLETE PACKAGE zsys. PLEASE: apt remove zsys

!!! only if ROOT on ZFS !!!

There is an error in the zsys package that causes bpool to run out of space,
see [[#Ubuntu zsys]] for more details.

To fix:
<pre>
cd ~/git/scripts
git pull
cp etc/zsys.conf /etc/
zsysctl service reload
zsysctl service gc
zpool list bpool
zfs list bpool
df /boot
</pre>

= IPMI instructions =

IPMI is the board management hardware on Supermicro and other server motherboards. This includes hardware sensors - fan rotation speed, temperatures and power supply voltages.

<pre>
apt-get install ipmitool
systemctl enable ipmievd
systemctl restart ipmievd
</pre>

Run:
* ipmitool sel list ### event list
* ipmitool sel elist ### event list
* ipmitool sel clear ### clear event list (if it becomes full)
* ipmitool sensor ### report hardware sensors

= move /home/wheel =

note: this MUST be done if ZFS root and NIS/autofs with /home.

Default location of wheel's home directory will collide with autofs /home, it has to be moved,
for example to /wheel.

<pre>
# logout from the wheel user
# go to another computer
ssh root@daqubuntuxxx
zfs list | grep wheel ### identify zfs name wheel_xxxxxx
#zfs set mountpoint=/wheel rpool/USERDATA/wheel_hm8fzh
zfs set mountpoint=/wheel `zfs list | grep wheel | cut -f1 -d" "`
zfs list | grep wheel
emacs -nw /etc/passwd ### change wheel's home directory from /home/wheel to /wheel
su - wheel ### check that user wheel still works
</pre>

This will break wheel's ability to run snap programs, such as firefox, install chrome as listed below.

= enable NIS (ubuntu 22.04, 24.04, debian 11, 12) =

<pre>
apt -y install rpcbind nis
echo DAQ-NIS >> /etc/defaultdomain
echo ypserver daq00.triumf.ca >> /etc/yp.conf
systemctl enable ypbind.service
systemctl restart ypbind.service
systemctl status ypbind.service
ypwhich -m
</pre>

enable ypserv:

<pre>
sed -i s/NISSERVER=false/NISSERVER=slave/ /etc/default/nis
/usr/lib/yp/ypinit -s daq00
echo ypserver localhost >> /etc/yp.conf
sed -i "s/ypserver .*/ypserver localhost/" /etc/yp.conf
systemctl enable ypserv
systemctl restart ypserv
systemctl restart ypbind
</pre>

edit /etc/nsswitch.conf to read:

<pre>
# begin get data from nis
passwd: files nis
group: files nis
shadow: files nis
automount: files nis
netgroup: files nis
# end get data from nis

#passwd: ...
#group: ...
#shadow: ...

#netgroup: ...
#automount: ...
</pre>

enable hourly update of nis maps:

<pre>
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>

If this is a new machine, then on the master NIS node (daq00), add this new node to /etc/netgroup, and update NIS maps (cd /var/yp; make)

= enable NIS (ubuntu 20.04) =

* apt-get -y install portmap nis ### will ask for NIS domain (DAQ-NIS)
* dpkg-reconfigure nis ### reconfigure if already installed
* ypwhich -m
* edit /etc/default/nis
** set "NISSERVER=slave"
** Ubuntu LTS 20.04, check that "YPBINDARGS=" is blank, remove "-no-dbus" if it is there
* #edit /etc/yp.conf, comment-out everything, add "domain DAQ-NIS server localhost"
* edit /etc/yp.conf, comment-out everything, add "ypserver localhost"
* /usr/lib/yp/ypinit -s daq00
* systemctl enable nis
* systemctl restart nis
* ypwhich
* ypwhich -m
* ypcat -k passwd
* vi /etc/nsswitch.conf ### add the automount line, modify the passwd, group and shadow lines to read this:
<pre>
# begin get data from nis
passwd: files nis
group: files nis
shadow: files nis
automount: files nis
netgroup: files nis
# end get data from nis
</pre>
* enable hourly update of NIS maps
<pre>
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>
* ### NOT NEEDED sudo vi /etc/idmapd.conf ### add line: "Domain = triumf.ca"

= enable autofs =

<pre>
apt -y install autofs
systemctl enable autofs
systemctl restart autofs
ls -l /home/olchansk ### test autofs, check file owner is correct
</pre>

= enable NFS server =

<pre>
apt install nfs-kernel-server
#edit /etc/exports
systemctl enable nfs-server
systemctl restart nfs-server
</pre>

= NIS master =

notes for setting up the NIS master

== wheel user ==

"wheel" is the default administrative user. We do not want it's password exported to NIS (encrypted password hash is world visible) and we do not want it's home directory exported to NFS (~wheel/.ssh is world visible and potentially writable: anybody can change ~wheel/.ssh/authorized_keys).

* move wheel's home directory from /home/wheel to /wheel (see special section about this)
* change wheel's UID and GID from 1000 to a value below MINUID in /var/yp/Makefile

== coherent uids ==

we do not want system accounts defined in /etc/passwd of the NIS master
to be included in the NIS map "passwd". this causes trouble on NIS clients
where newly installed packages fail to create local system users because same
user already exists in NIS.

This is controlled by MINUID in /var/yp/Makefile.

Historical TRIUMF uids start from around 200, but several clusters do not have any historic TRIUMF uids below 500 and MINUID is set to:
* DAQ-NIS: MINUID=200
* ISAC-NIS: MINUID=500
* TITAN-NIS: MINUID=500
* MUSR-NIS: MINUID=500
* TIG-NIS: MINUID=500 (100 on SL6 mother8pi)

Ubuntu 20 has two programs to create users:
* adduser - creates new users with UID 1000 and up as specified in /etc/adduser.conf. No problems here.
* adduser --system - creates new system users with UID 100 and up as specified in /etc/adduser.conf. No problems here.
* useradd - creates new users with UID 1000 and up as specified in /etc/login.defs. No problems here.
* useradd --system - creates new system users with UID 999 and down (read "man useradd", section at the end about SYS_UID_MAX). This collides with NIS MINUID, these system users will be included in the NIS map and cause trouble.

This problem cannot be fixed, SYS_UID_MIN, SYS_UID_MAX and UID_MIN in /etc/login.defs do not seem
to have any effect on UIDs chosen by "useradd --system". (tested on Ubuntu LTS 20.04).

So far only these system accounts seem to be affected by this:
* systemd-coredump
* ganglia

To fix:
* run "sort -r -n -t: -k3 /etc/passwd" to identify the last unused system user uid (range 100..200)
* run "sort -r -n -t: -k3 /etc/group" to identify the last unused system user gid (range 100.200)
* systemd-coredump: manually change UID and GID (package systemd-coredump is usually not installed)
* ganglia: same thing, then change ownership on all ganglia files.

Also read systemd author's opinion on system vs user UIDs:
https://github.com/systemd/systemd/issues/4850#issuecomment-265698275

= Fix systemd-logind NIS breakage =

!!! THIS IS NOT NEEDED FOR UBUNTU LTS 20.04 !!!

there is a delay in ssh logins for normal users. "ssh -v" shows the delay is after "pledge...". this
fix removes the delay.

systemd developers think that we should not use NIS and made sure there are
problems if we do. To give them credit, they do offer a workaround. Read this:
https://github.com/poettering/systemd/commit/695fe4078f0df6564a1be1c4a6a9e8a640d23b67

<pre>
mkdir /etc/systemd/system/systemd-logind.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-logind.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-logind.service
</pre>

= Fix systemd-udevd NIS breakage =

see same problem as above with udev getting stuck. ubuntu lts 20.04.

<pre>
mkdir /etc/systemd/system/systemd-udevd.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-udevd.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-udevd.service
</pre>

= Configure USB device permissions =

Configure USB device permissions for user access to USB-serial devices, Altera USB Blaster, etc.

* create file /etc/udev/rules.d/99-usb-chmod.rules with this contents:

<pre>
emacs -nw /etc/udev/rules.d/99-usb-chmod.rules
ACTION=="add", SUBSYSTEM=="usbmisc", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /dev/%c"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /proc/%c"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVICE}"
ACTION=="add", ENV{PHYSDEVBUS}=="usb-serial", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVPATH}=="/class/tty/ttyS*", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyUSB*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyACM*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyS*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", DEVPATH=="*video*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
</pre>

* reload udev rules: udevadm control --reload-rules
* apply new permissions: udevadm trigger --action=add
* watch udev activity: udevadm monitor -p

= Configure lightdm display manager =

* enable it
<pre>
echo lightdm | dpkg-reconfigure -fteletype lightdm
systemctl disable gdm
systemctl disable sddm
systemctl enable lightdm
</pre>

* make the MATE desktop as default
<pre>
cd ~root/git/scripts/
git pull
/bin/cp -v etc/lightdm_default_mate.conf /etc/lightdm/lightdm.conf.d/
</pre>

* enable login by NIS users
<pre>
/bin/cp -v etc/lightdm_enable_nis_login.conf /etc/lightdm/lightdm.conf.d/
</pre>

* restart lightdm
<pre>
systemctl stop gdm
systemctl restart lightdm
</pre>

= Install libpng12.so.0 =

Quartus 16 needs libpng12:

<pre>
wget http://mirrors.kernel.org/ubuntu/pool/main/libp/libpng/libpng12-0_1.2.54-1ubuntu1_amd64.deb
dpkg --install libpng12-0_1.2.54-1ubuntu1_amd64.deb
</pre>

= Install google-chrome =

<pre>
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
dpkg -i google-chrome-stable_current_amd64.deb
</pre>

confirm autoupdate is enabled, observe dl.google.com is present in the list of repositories:
<pre>
apt update
...
Get:5 https://dl.google.com/linux/chrome/deb stable/main amd64 Packages [1,094 B]
...
</pre>

FOLLOWING IS OBSOLETE:

Instructions from here:
https://www.ubuntuupdates.org/ppa/google_chrome?dist=stable

<pre>
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-tmp.list'
apt update
apt install google-chrome-stable
/bin/rm -f /etc/apt/sources.list.d/google-tmp.list
</pre>

= Install amanda client =

ONLY ONE MACHINES THAT HOST HOME DIRECTORIES

* apt install amanda-client
* edit /etc/amandahosts
<pre>
amanda.triumf.ca amanda amdump
</pre>
* check permissions on /etc/amandahosts:
<pre>
root@daq00:/var/log/amanda# ls -l /etc/amandahosts
-rw------- 1 backup backup 49 Jan 27 10:48 /etc/amandahosts
</pre>
* fix if needed: chown backup.backup /etc/amandahosts; chmod a= /etc/amandahosts; chmod u=wr /etc/amandahosts
* edit /etc/amanda-security.conf, add this line:
<pre>
runtar:gnutar_path=/usr/bin/tar
</pre>

On the amanda machine:

* in amanda disklist, use dump type "bsdtcp-comp-user-tar"
* su - amanda and run amcheck -c daily daq00
<pre>
-bash-4.1$ amcheck -c daily daq00

Amanda Backup Client Hosts Check
--------------------------------
Client check: 1 host checked in 0.092 seconds. 0 problems found.

(brought to you by Amanda 3.3.7p1.git.685ff76d)
</pre>

= Enable rc.local =

For reasons unknown, Ubuntu LTS 20.04 does not enable /etc/rc.local. Do this:

<pre>
cd ~/git/scripts
git pull
cp -n -v etc/rc.local /etc/
chmod a+rx /etc/rc.local
cp etc/rc-local.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable rc-local
systemctl start rc-local
systemctl status rc-local
</pre>

= Remove unwanted packages =

<pre>
apt remove bash-completion # broken, adds unwanted "\" if "ls -l $ROOTSYS/<tab>"
apt remove zsys # broken, do not use
apt remove sddm # login manager
apt remove avahi-daemon avahi-autoipd # not sure what it does, observed using 100% CPU
apt remove modemmanager # probes all serial ports to see if it's a modem
</pre>

= Disable unwanted services =

<pre>
systemctl disable mpd
systemctl disable snapd
systemctl disable ModemManager
systemctl --global mask tracker-extract-3.service
systemctl --global mask tracker-miner-fs-3.service
systemctl daemon-reload
</pre>

= Disable sleep and suspend =

note: we see some computers randomly shutdown or go to sleep, log files indicates the "sleep" or "suspend" button was pushed by user, but no such buttons actually exist. this is the fix for this:

<pre>
systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target systemd-suspend.service systemd-hybrid-sleep.service
</pre>

= Enable crontab @reboot for MIDAS =

startup scripts have a bug - cron @reboot entries for normal users can run before autofs is ready, so if the home directory is on autofs/NFS, it cannot be accessed and the cron job fails. If MIDAS is supposed to be started by cron @reboot, it will not start (there *will* be an error message in /var/log/cron).

<pre>
mkdir /etc/systemd/system/cron.service.d
echo -e "[Unit]\nAfter=ypbind.service autofs.service\n" > /etc/systemd/system/cron.service.d/local.conf
systemctl daemon-reload
systemctl cat cron.service
</pre>

Explore the systemd dependency tree using "systemctl list-dependencies" maybe with "--all".

Visualize the exact boot sequence from previous boot: "systemd-analyze plot > xxx.svg", look at the svg file using a web browser.

Crontab entry to start midas: (install in the midas user crontab, not root crontab)

<pre>
su - midasuser
crontab -l
#@reboot /bin/bash -l -c "/home/trinat/bin/start-daq-applications"
#@reboot /bin/tcsh -c "/home/trinat/bin/start-daq-applications"
</pre>

= Install apache httpd proxy for midas and elog =

This will configure the HTTPS/SSL certificate using "certbot" and "letsencrypt" and configure an HTTPS web server using apache2.

First, configure apache2:

* execute these commands:
<pre>
apt -y install apache2
cd /etc/apache2
</pre>
* create new file conf-available/ssl-daq14.conf # use actual hostname instead of daq14
<pre>
SSLSessionCache shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout 300
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
SSLCryptoDevice builtin
</pre>
* create new file sites-available/daq14-ssl.conf # use actual hostname instead of daq14
<pre>
<IfModule mod_ssl.c>
<VirtualHost *:443>
ServerName daq14.triumf.ca
DocumentRoot /var/www/html
ErrorLog /var/log/apache2/daq14.log
SSLEngine on
# note SSLProtocol, SSLCipherSuite and some other settings are overwritten by /etc/letsencrypt/options-ssl-apache.conf
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SEED:!IDEA:!RC4
## use port specified in elogd.cfg
#ProxyPass /elog/ http://localhost:8082/ retry=1
## use mhttpd port
#ProxyPass / http://localhost:8080/ retry=1
Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"
<Location />
SSLRequireSSL
AuthType Basic
AuthName "DAQ password protected site"
Require valid-user
# create password file: touch /etc/apache2/htpasswd
# to add new user or change password: htpasswd /etc/apache2/htpasswd username
AuthUserFile /etc/apache2/htpasswd
</Location>
</VirtualHost>
</IfModule>
</pre>
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* enable ssl module
* enable new configurations
<pre>
a2enmod ssl
a2enmod headers
a2enmod proxy
a2enmod proxy_http
a2enconf ssl-daq14
a2ensite daq14-ssl
</pre>
* disable default ssl sites
<pre>
a2dissite 000-default-le-ssl
a2dissite 000-default
ls -l /etc/apache2/sites-enabled/ ### should show only daq14-ssl.conf
</pre>
* check that there are no syntax problems
<pre>
apache2ctl configtest
</pre>
* enable and start apache2:
<pre>
systemctl enable apache2
systemctl restart apache2
systemctl status apache2
</pre>
* apache2 may fail to start, look in /var/log/apache2/error.log and /var/log/apache2/daq14.log
* if it says "Failed to configure ... certificate", proceed to the step for setting certbot.
* try to access https://daq14.triumf.ca
** you should see a complaint about self-signed certificate
** you should see a request for password (do not login yet)
** if you get "connection refused", HTTPS port 443 may need to be enabled in the local firewall, look at documentation for ufw.
Second, configure certbot:

(Note: as of 2018-01-18 certbot requires use of http port 80 to get the initial https certificate,
renewal can continue to use the https port 443)

(Note: as of 2019-01-?? certbot requires use of port 80 for renewals)

(Note: unsurprisingly, this requires outside access to connect with letsencrypt, so won't work if PC is only accessible from on-site network)

* check that port 80 is not used by anything:
* netstat -an | grep LISTEN | grep ^tcp | grep 80
* lsof -P | grep -i tcp | grep LISTEN | grep 80
* if lsof reports that apache2 is listening on port 80, follow the apache2 instructions above (remove "listen 80" from apache2.conf

* install certbot (if necessary open tcp port 80 in the firewall, see documentation for ufw):
<pre>
apt install certbot python3-certbot-apache
certbot certonly --standalone --installer apache
</pre>
* then answer questions:
* "activate HTTPS for daq14.triumf.ca" - say ok
* "enter email address" - enter your own email address
* "please read terms..." - read the terms and say "agree"
* it will take a few moments...
* "congratulations..." - say ok.
<pre>
certbot install --apache --cert-name daq14.triumf.ca
</pre>
* then answer questions:
* "choose redirect..." - say "1" (no redirect)
* look inside /etc/apache2/sites-enabled/daq14-ssl.conf to see that SSLCertificateFile & co point to certbot certificates in
/etc/letsencrypt/live/daq14.triumf.ca/
* to check current renewal and to update the certbot config file in /etc/letsencrypt/renewal, run this:
<pre>
certbot renew --standalone --installer apache --force-renewal
</pre>

NOTE: this certificate will expire in 3 months, automatic renewal should work with current version of certbot

Third, activate password protection:

* as shown in the config file above, create password file and initial user: (replace "midas" with specific username)
<pre>
touch /etc/apache2/htpasswd
htpasswd /etc/apache2/htpasswd midas
</pre>

* restart apache2
<pre>
systemctl restart apache2
systemctl status apache2
</pre>

From here:
* enable proxy for MIDAS mhttpd - uncomment redirect in the config file above
* enable proxy for ELOG - ditto
<pre>
a2enmod proxy
a2enmod proxy_http
apache2ctl configtest
systemctl restart apache2
</pre>

* try accessing MIDAS https://daq14.triumf.ca/ (make sure mhttpd is running)
* if it's not working, check odb setting FIXME!
* try accessing ELog https://daq14.triumf.ca/elog/ (make sure elogd is running)
* if it's not working, check elogd.cfg file and make sure
<pre>
SSL = 0
</pre>

NOTE: if certbot fails with errors about 'module' object has no attribute 'pyopenssl',
try this: pip install requests==2.6.0

== generate self-signed certificate ==

<pre>
root@alphacpc05:~# openssl req -nodes -new -x509 -keyout server.key -out server.cert -days 1001
...+....+..+..........+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+..+...+.........+......+.+...+...+.....+...............+.........+...+.+......+...+...........+....+...+..+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+......+.+...+..+.......+..+...+.......+......+...+..+...+......+....+...............+..+...+....+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
......+......+.+..+......+.+......+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.....+......+.+.........+......+.....+.+..+...+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+....+......+.....+...+...+.......+..+.+........+.+...+......+..+..........+..+.+...........+...+.......+......+.....+.......+...+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:CH
State or Province Name (full name) [Some-State]:Geneve
Locality Name (eg, city) []:CERN
Organization Name (eg, company) [Internet Widgits Pty Ltd]:CERN
Organizational Unit Name (eg, section) []:ALPHA experiment
Common Name (e.g. server FQDN or YOUR name) []:alphacpc05.cern.ch
Email Address []:
root@alphacpc05:~#
root@alphacpc05:~#
root@alphacpc05:~# ls -l
-rw-r--r-- 1 root root 1375 juil. 10 21:43 server.cert
-rw------- 1 root root 1708 juil. 10 21:42 server.key
root@alphacpc05:~# systemctl restart apache2
</pre>

= Enable elog PDF preview =

see https://stackoverflow.com/questions/52998331/imagemagick-security-policy-pdf-blocking-conversion

* xemacs -nw /etc/ImageMagick-6/policy.xml
* remove this section at the end:
<pre>

<policy domain="coder" rights="none" pattern="PS" />
<policy domain="coder" rights="none" pattern="PS2" />
<policy domain="coder" rights="none" pattern="PS3" />
<policy domain="coder" rights="none" pattern="EPS" />
<policy domain="coder" rights="none" pattern="PDF" />
<policy domain="coder" rights="none" pattern="XPS" />
</pre>

= Install Jupyter notebook =

<pre>
From https://jupyter.org/install
apt install python3-pip
pip install jupyterlab
pip install notebook
~/.local/bin/jupyter notebook
watch the http://localhost:8888 URL that it printed
say "no" to offer to start firefox (it will not work!)
URL is: http://localhost:8888/tree?token=xxx
from the machine where you are running the web browser (i.e. google-chrome), run (replace trinat@trinatdaq with the username and machine name where you started jupyter)
open a new shell and run: ssh -v trinat@trinatdaq -L 8888:localhost:8888
in the web browser, open http://localhost:8888
this gives us the login page
in the password or token entry field, put the token from the "tree?token=xxx" above (printed by jupyter on startup)
push button "login"
jupyter page should open with the list of files in the trinat home directory
congratulate Brian with full success
</pre>

= Install ZFS quota report =

If there are any ZFS volumes, install script to report disk and quota usage

<pre>
cd ~/git/scripts/quotareport
git pull
mkdir /var/www/html/zfsquotareport
cp -pv ~/git/scripts/quotareport/sorttable.js /var/www/html/zfsquotareport/
ln -s $PWD/zfsquotareport.perl /etc/cron.daily/
touch /etc/crontab
</pre>

If httpd is configured to redirect "/" to MIDAS mhttpd:
* add following to /etc/apache2/sites-enabled/xxx-ssl.conf in front of "ProxyPass / ..."
* run "systemctl reload apache2"
<pre>
## do not proxy zfs quota report directory
ProxyPass /zfsquotareport/ !
</pre>

= Install PHP =

* apt install php libapache2-mod-php
* systemctl restart apache2
* create /var/www/html/info.php
<pre>
<?php

phpinfo();
</pre>
* open https://daq00.triumf.ca/info.php

= Configure TRIUMF printers =

<pre>
systemctl stop cups
systemctl disable cups
systemctl stop snap.cups.cupsd.service
systemctl stop snap.cups.cups-browsed.service
systemctl disable snap.cups.cupsd.service
systemctl disable snap.cups.cups-browsed.service
echo "ServerName printers.triumf.ca" > /etc/cups/client.conf
lpstat -a
</pre>

= Enable core dumps =

By default, Ubuntu LTS 20.04 installs the apport package
which disabled core dumps from user applications. (google it up!).
It is not meant to do this and documentation claims that
it is not installed and not enabled by default. Oh, well...

<pre>
apt remove apport
apt autoremove ### will remove apport-symptoms and a few other packages
</pre>

After this, core dumps are written to file "core" in the current directory.
See /proc/sys/kernel/core_pattern and /proc/sys/kernel/core_uses_pid.

Enable core dump file names to include process id, add following to /etc/rc.local

<pre>
echo 1 > /proc/sys/kernel/core_uses_pid
</pre>

= Enable debugger =

By default, Ubuntu LTS 20.04 does not permit debugger to attach and debug
already running programs. To enable it, add following to /etc/rc.local

<pre>
echo 0 > /proc/sys/kernel/yama/ptrace_scope
</pre>

= Disable Ubuntu Pro nag =

best I can tell, impossible at this time.

== do not do this ==

!!! does nothing !!!

<pre>
pro config set apt_news=false
</pre>

== do not do this ==

!!! breaks automatic updates because 20apt-esm-hook.conf is missing !!!

If "apt upgrade" requests Ubuntu Pro or esm-apps, disable the nag:
<pre>
/bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf
</pre>

== do not do this ==

!!! likely same as above, breaks automatic updates !!!

* comment out /etc/apt/apt.conf.d/20apt-esm-hook.conf

== do not do this ==

!!! removes too many packages !!!

<pre>
apt remove ubuntu-pro-client
</pre>

= Update packages =

* apt-get update # update package list
* apt-get dist-upgrade # install updated packages and update "kept back" packages
* apt-get autoremove # remove packages that apt thinks should be removed

= Finish installation =

Congratulations. There is nothing more to do!

* reboot
<pre>
shutdown -r now
</pre>

= Update to new version of Ubuntu =

<pre>
vi /etc/update-manager/release-upgrades # set "Prompt=normal"
do-release-upgrade
</pre>

Update Ubuntu LTS 20.04 to LTS 22.04:

<pre>
apt remove zsys
</pre>

== daqubuntu ==

<pre>
# reboot to clear out all updates
# vi /etc/update-manager/release-upgrades # set "Prompt=normal"
# do-release-upgrade -c
Checking for a new Ubuntu release
New release '22.04 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
# do-release-upgrade
...
say yes...
...
login.defs, say "Y" (erase local changes, use packaged version)
/etc/systemd/resolved.conf, say "Y" (same as above)
firefox snap, say yes
unable to reach snap store, say "skip"
/etc/gmond.conf, say "Y"
/var/yp/Makefile, say "install the package maintainer's version"
/etc/ypserv.conf, same thing
/etc/ypserv.securenets, same thing
/etc/default/nis, same thing
/etc/speech-dispatcher/modules/mary-generic.conf, same thing
/etc/apt/apt.conf.d/50unattended-upgrades, same thing
...
278 packages are going to be removed, say yes
...
restart required, say yes
...
no ping... yes ping...
...
ssh daqubuntu, ok
apt update, fail, DNS does not work, "host security.ubuntu.com" does not resolve.
fix resolver per https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Disable_NetworkManager
apt update, apt upgrade now works, 0 packages to update
NIS does not work.
</pre>

== midm9a ==

<pre>
login.defs
firefox snap
gmond.conf
ypserv
/etc/default/nis
unattended-upgrades
amanda-security.conf
remove obsolete (no)
reboot
configure dns
reenable nis
</pre>

== daq17 ==

<pre>
firefox snap
imagemagick policy.xml
gmond.conf
chrony.conf
/var/yp/Makefile
ypserv.conf
ypserv.securenets
/etc/default/nis
50unattended-upgrades
</pre>

== daq00 ==

per https://serverpilot.io/docs/how-to-upgrade-ubuntu-20.04-to-22.04/

<pre>
do-release-upgrade -f DistUpgradeViewNonInteractive
</pre>

if it exists "too soon" without doing anything, run it without "-f xxx", most likely it does not like something about this machine. in case of daq00 it did not like how the EFI partitions were mounted. after fixing it, non-interactive upgrade was successful.

== isdaq08 ==

* prepare
<pre>
cd ~/git/scripts
git pull
cd ~
apt -y install debsums
</pre>
* check for modified config files that make upgrade unhappy, deal with all files reported by debsums.
<pre>
root@isdaq08:~# debsums -ce
/etc/ganglia/gmond.conf
/etc/yp.conf
/etc/apt/apt.conf.d/10periodic
root@isdaq08:~#
</pre>
* restore original /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
</pre>
* apt remove ganglia-monitor
* apt remove nis
* "debsums -ce" is now empty

Run the upgrade:

* do-release-upgrade -f DistUpgradeViewNonInteractive

Post upgrade:

* configure DNS
* apt -y install linux-generic-hwe-22.04
* /bin/cp -v ~/git/scripts/etc/99apt-conf-ko /etc/apt/apt.conf.d/ # restore nightly updates
* /bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf # remove the ubuntu-pro nag
* install missing packages
* restore ganglia
* restore nis
* check zpool status, may need zpool upgrade
* reboot

== daqubuntu, U-24 ==

* prepare
<pre>
cd ~/git/scripts
git pull
cd ~
apt -y install debsums
</pre>
* check for modified config files that make upgrade unhappy, deal with all files reported by debsums.
<pre>
root@daqubuntu:~# debsums -ce
/etc/ganglia/gmond.conf
debsums: missing file /etc/init.d/nis (from nis package)
/etc/default/nis
/etc/ypserv.conf
/etc/ypserv.securenets
/var/yp/Makefile
/etc/update-manager/release-upgrades
/etc/apt/apt.conf.d/10periodic
/etc/yp.conf
root@daqubuntu:~#
* restore original /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
</pre>
* apt remove ganglia-monitor
* apt remove nis
* apt autoremove
* restore original release-upgrades: "Prompt: lts"
* "debsums -ce" is now empty

Check for upgrade:

<pre>
root@daqubuntu:~# do-release-upgrade -c
Checking for a new Ubuntu release
There is no development version of an LTS available.
To upgrade to the latest non-LTS development release
set Prompt=normal in /etc/update-manager/release-upgrades.
root@daqubuntu:~#
</pre>

Run the upgrade:

* do-release-upgrade -f DistUpgradeViewNonInteractive

Post upgrade:

* configure DNS
* apt -y install linux-generic-hwe-22.04
* /bin/cp -v ~/git/scripts/etc/99apt-conf-ko /etc/apt/apt.conf.d/ # restore nightly updates
* /bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf # remove the ubuntu-pro nag
* install missing packages
* restore ganglia
* restore nis
* check zpool status, may need zpool upgrade
* reboot

== daq14, U-20-22-24 ==

* apt update, apt upgrade
* apt -y install linux-image-generic-hwe-20.04 linux-tools-virtual-hwe-20.04 ### install kernel 5.15
* shutdown -r now
* stuck waiting for daq14 to shutdown...
* reboot into kernel 5.15
* <pre>
cd ~/git/scripts
git pull
cd ~
apt -y install debsums
</pre>
* debsums -ce
<pre>
/etc/apache2/ports.conf
/etc/dnsmasq.conf
/etc/ganglia/gmond.conf
/etc/yp.conf
/etc/sudoers
</pre>
* apache2 restore original ports.conf, uncomment "Listen 80"
* cp -pv /etc/dnsmasq.conf.dpkg-dist /etc/dnsmasq.conf
* apt remove ganglia-monitor
* edit /etc/yp.conf, remove everything after "# ypserver ypserver.network.com"
* "debsums -ce" is now empty
* do-release-upgrade -f DistUpgradeViewNonInteractive
* runs for a long time
* stuck on "/etc/default/nis", type "Y", press enter, nothing for a bit, then resumes running
* finished
* configure DNS
* reboot
* have kernel 6.8
* apt update; apt upgrade
* apt upgrade guile-2.2-libs ### would not auto-update, "kept back", has to be done by hand
* apt autoremove
* debsums -ce
<pre>
debsums: missing file /etc/init.d/nis (from nis package)
/etc/default/nis
</pre>
* diff /etc/default/nis.dpkg-dist /etc/default/nis
* cp -pv /etc/default/nis.dpkg-dist /etc/default/nis
* debsums -ce
<pre>
debsums: missing file /etc/init.d/nis (from nis package)
</pre>
* we ignore this and run the update
* do-release-upgrade -c
<pre>
Checking for a new Ubuntu release
New release '24.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
</pre>
* do-release-upgrade -f DistUpgradeViewNonInteractive
* bombs out without any error messages
* in /var/log/dist-upgrade/main.log reports "Failed to find a replacement for xapp" and other packages
* apt remove xapp usrmerge ureadahead thunderbird-gnome-support
* no go, complains about even more packages.
* apt list | grep installed | grep -v jammy ### show packages installed from non-ubuntu sources
* remove all packages marked "install,local" ### ubuntu updater does not know where they came from and so cannot update them.
* apt remove desktop-base ### not happy about this package in /var/log/dist-upgrade/apt.log
* apt autoremove
* do-release-upgrade -f DistUpgradeViewNonInteractive
* running for a long time...

= Upgrade to new version of Debian =

https://www.debian.org/releases/bookworm/amd64/release-notes/ch-upgrading.en.html

== 32-bit VME processor Debian 11 to 12 ==

* cd git/scripts; git pull; cd ~
* apt update
* apt upgrade
* edit /etc/apt/sources.list
<pre>
deb http://deb.debian.org/debian/ bookworm main
#deb http://deb.debian.org/debian/ bullseye main
#deb-src http://deb.debian.org/debian/ bullseye main
</pre>
* apt update
* apt upgrade --without-new-pkgs
* apt full-upgrade
* apt list '~c'; apt purge '~c' # purge left-over config files [residual-config]
* reboot

= Ubuntu package manager =

* apt-get install xxx # install package xxx
* apt-get update
* apt-get upgrade
* apt-get dist-upgrade
* apt-get autoremove # remove automatically installed packages required by a removed package
* apt-get remove xxx # remove package xxx
* apt-cache search . # list all available packages
* apt-cache show "." | grep ^Package # list al available packages
* apt-cache madison root-system # show all available versions of package root-system
* apt list # list all installed packages
* dpkg --listfiles libpng16-16 # list all files from this package
* apt list --installed # list all installed packages
* dpkg -S /bin/bash # what package provides this file?
* dpkg -L bash # what files provided by this package?
* debsums -ce # show modified config files
* apt-config dump # show apt configuration

= Ubuntu zsys =

NOTE: DO NOT USE ZSYS, see https://github.com/ubuntu/zsys/issues/218 and https://github.com/ubuntu/zsys/issues/230

* manual removal of old snapshots
<pre>
zsysctl show
zsysctl state remove xy69ye -s
zsysctl state remove xy69ye
zsysctl state remove xy69ye -u wheel
</pre>
* apt remove zsys

NOTE: old zsys snapshots must be cleaned manually, "zsysctl state remove xxx --system" is broken and does not remove user data snapshots

* manages system snapshots
* documentation: https://github.com/ubuntu/zsys
* documentation: (go to next article via link "newer" at the bottom) https://didrocks.fr/2020/05/21/zfs-focus-on-ubuntu-20.04-lts-whats-new/
* ubuntu 20.04 bug, too many snapshots cause /boot to become full and updates fail. https://github.com/ubuntu/zsys/issues/155
* solution: use custom /etc/zsys.conf, limit number of snapshots to 10, see trinatdaq:/etc/zsys.conf
* zsys commands:
<pre>
update-grub # list of all snapshots, errors if some snapshots are broken
zsysctl state remove lnc0k7 --system # remove snapshot
xemacs -nw /etc/zsys.conf; zsysctl service reload; zsysctl service gc # cause gc to run with new settings in zsys.conf
zfs list -r -t snapshot -o name,used,referenced,creation bpool/BOOT # list snapshots
zsysctl show # show snapshots
</pre>

= Ubuntu cloning =

to clone a ubuntu image:

<pre>
cd /nfsroot/lxcpet
emacs -nw etc/hostname ### change hostname
emacs -nw etc/mailname ### change hostname (debian 11)
emacs -nw etc/defaultdomain ### change the NIS domainname
emacs -nw etc/yp.conf ### change the NIS server
cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
emacs -nw root/.ssh/authorized_keys ### update root ssh keys
</pre>

= Ubuntu boot loader =

== maintenance commands ==

* update-initramfs -v -u
* grub-install /dev/sda

= Convert from single to dual mirrored ZFS SSD =

Assuming Ubuntu LTS 22.04 with "instal on ZFS" option, we will
add a second SSD, configure ZFS to use both SSDs in mirrored
configuration and setup grub to boot from either SSD. This
is intended to create a full redundant system where failure
of either SSD does not break the system.

* identify first SSD
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* connect second SSD of identical size
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
/dev/sdb WD Blue SA510 2.5 250GB 22243Z803852 25 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* if second SSD is not autodetected, reboot
* Clone partition table automatically
If both SSDs are identical size, use this simpler method of duplicating the partition table:
<pre>
root@midm9b:~# sfdisk -d /dev/sda > part_table
root@midm9b:~# grep -v ^label-id part_table | sed -e 's/, *uuid=[0-9A-F-]*//' | sfdisk /dev/sdb
</pre>
The grep and sed in the second command are there to prevent disk ID and partition IDs from being cloned. Alternatively the part_table file can be edited manually to remove the label-id line and the uuid entries from the individual partitions.

* Clone partition table manually (e.g. for different size disks)
* list partition table of first SSD:
<pre>
root@midm9b:~# fdisk -l /dev/sda
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>
* create identical partitions on second SSD, use sector numbers from above.
<pre>
root@midm9b:~# gdisk /dev/sdb
GPT fdisk (gdisk) version 1.0.8

Partition table scan:
MBR: not present
BSD: not present
APM: not present
GPT: not present

Creating new GPT entries in memory.

Command (? for help): n
Partition number (1-128, default 1):
First sector (34-488397134, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-488397134, default = 488397134) or {+-}size{KMGTP}: 1050623
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): ef00
Changed type of partition to 'EFI system partition'

Command (? for help): n
Partition number (2-128, default 2):
First sector (34-488397134, default = 1050624) or {+-}size{KMGTP}:
Last sector (1050624-488397134, default = 488397134) or {+-}size{KMGTP}: 5244927
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 8200
Changed type of partition to 'Linux swap'

Command (? for help): n
Partition number (3-128, default 3):
First sector (34-488397134, default = 5244928) or {+-}size{KMGTP}:
Last sector (5244928-488397134, default = 488397134) or {+-}size{KMGTP}: 9439231
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): be00
Changed type of partition to 'Solaris boot'

Command (? for help): n
Partition number (4-128, default 4):
First sector (34-488397134, default = 9439232) or {+-}size{KMGTP}:
Last sector (9439232-488397134, default = 488397134) or {+-}size{KMGTP}:
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): bf00
Changed type of partition to 'Solaris root'

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sdb.
The operation has completed successfully.
root@midm9b:~# fdisk -l /dev/sda /dev/sdb
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root

Disk /dev/sdb: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: EB251739-30C6-422F-A505-5887B5A0B603

Device Start End Sectors Size Type
/dev/sdb1 2048 1050623 1048576 512M EFI System
/dev/sdb2 1050624 5244927 4194304 2G Linux swap
/dev/sdb3 5244928 9439231 4194304 2G Solaris boot
/dev/sdb4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>
* identify second SSD partitions
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part3
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part3 -> ../../sda3
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 -> ../../sdb3
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
</pre>
* convert bpool from single disk to mirrored disk:
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~# zpool attach bpool 99e03dc0-7d4d-f24b-8fa1-f042b9f135db /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3
root@midm9b:~# zpool status bpool
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors
</pre>
* convert rpool
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
root@midm9b:~# zpool attach rpool f6fd54f8-3af7-b943-ae3d-a4e480537fb9 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4
root@midm9b:~# zpool status rpool
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Jan 20 19:40:45 2023
5.83G scanned at 664M/s, 2.92M issued at 332K/s, 9.11G total
0B resilvered, 0.03% done, no estimated completion time
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~#
</pre>
* wait for resilver to complete
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
scan: resilvered 9.65G in 00:00:36 with 0 errors on Fri Jan 20 19:41:21 2023
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
</pre>
* enable booting from second SSD: (instead of /dev/sda1, /dev/sdb1, use UUID=xxx)
<pre>
root@midm9b:~# mkfs.msdos /dev/sdb1
root@midm9b:~# mkdir /boot/efi-sda
root@midm9b:~# mkdir /boot/efi-sdb
root@midm20c:~# blkid | grep vfat ### identify UUID
/dev/sdb1: UUID="DD89-5081" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="d0cb6be4-2f67-5b42-9b26-9e6905e9f774"
/dev/sdc1: UUID="D970-86BA" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="e6d3b5b9-a512-44a2-9205-1a4db06ed2a2"
/dev/sda1: UUID="DDA1-044C" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="6dc9dff0-1c13-8045-a906-7803d3074c70"
root@midm20c:~# cat /etc/fstab | grep vfat ### add mount points with correct UUID
#UUID=D970-86BA /boot/efi vfat umask=0022,fmask=0022,dmask=0022 0 1
UUID=DDA1-044C /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022 0 1
UUID=DD89-5081 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022 0 1
root@midm9b:~# mount -a
root@midm9b:~# df -kl
Filesystem 1K-blocks Used Available Use% Mounted on
...
/dev/sda1 523244 13720 509524 3% /boot/efi
/dev/sdb1 523244 4 523240 1% /boot/efi-sdb
...
root@midm9b:~# rsync -av /boot/efi/ /boot/efi-sdb/
sending incremental file list
EFI/
...
root@midm9b:~# ls -l /boot/efi-sda
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~# ls -l /boot/efi-sdb
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~#
</pre>
* setup script to update grub on second SSD, it must be run manually after every kernel update
<pre>
root@midm9b:~# ln -s ~/git/scripts/etc/update_efi_grub.perl ~/
root@midm9b:~# ~/update_efi_grub.perl -u
EFI dir: /boot/efi-sda
/boot/efi-sda: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sda/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sda: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sda/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
EFI dir: /boot/efi-sdb
/boot/efi-sdb: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sdb/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sdb: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sdb/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
root@midm9b:~#
</pre>

= Disable NetworkManager =

NOTE: THIS IS BROKEN IN UBUNTU LTS 22.04

NetworkManager is useful for configuring dynamic
network interfaces, i.e. laptops that often move
between networks, or connect to multiple choice
of wifi networks, etc.

For machines with statically configured network interfaces,
NetworkManager is not necessary.

As it has been observed to become confused and observed
to malfunction when network links go up and down (it keeps
unnecessarily reconfiguring the ip address, etc), it can
be usefuil to disable it.

* list all network interfaces
<pre>
# /bin/ls -1 /sys/class/net/
enp0s31f6
lo
</pre>
* edit /etc/network/interfaces:
<pre>
rename enp0s31f6=eth0
auto eth0
iface eth0 inet static
address 142.90.120.94/19
gateway 142.90.100.18
</pre>
* statically configure systemd-resolved
** create /etc/systemd/resolved.conf.d/resolved.conf with this contents:
<pre>
[Resolve]
DNS=142.90.100.19
Domains=triumf.ca
</pre>
** systemctl restart systemd-resolved
** resolvectl
** systemd-analyze cat-config systemd/resolved.conf
* disable NetworkManager
<pre>
systemctl disable NetworkManager
</pre>
* reboot

= Configure ECC memory =

== Configure EDAC ==

* apt install edac-utils rasdaemon

=== Intel i3-2120 ===
<pre>
root@musr00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X9SCL/X9SCM
root@musr00:~# edac-ctl --status
edac-ctl: drivers not loaded.
</pre>

=== Intel E-2236 ===
<pre>
root@daq00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SCM-F
root@daq00:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq00:~# edac-util
edac-util: No errors to report.
root@daq00:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
</pre>
* check edac sysfs files (Intel)
<pre>
root@daq00:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_noinfo_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 max_location
-r--r--r-- 1 root root 4096 Jan 25 15:10 mc_name
drwxr-xr-x 2 root root 0 Jan 25 15:10 power
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank0
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank1
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank2
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank3
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank4
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank5
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank6
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank7
--w------- 1 root root 4096 Jan 25 15:10 reset_counters
-r--r--r-- 1 root root 4096 Jan 25 15:10 seconds_since_reset
-r--r--r-- 1 root root 4096 Jan 25 15:10 size_mb
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Jan 25 15:10 uevent
root@daq00:~#
</pre>

=== Intel E3-1270 v6 ===
<pre>
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --status
edac-ctl: drivers are loaded.
root@grsnis01:~# edac-util
edac-util: No errors to report.
root@grsnis01:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@grsnis01:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 max_location
-r--r--r-- 1 root root 4096 Feb 19 12:35 mc_name
drwxr-xr-x 2 root root 0 Feb 19 12:35 power
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank0
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank1
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank2
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank3
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank4
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank5
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank6
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank7
--w------- 1 root root 4096 Feb 19 12:35 reset_counters
-r--r--r-- 1 root root 4096 Feb 19 12:35 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 19 12:35 size_mb
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 19 12:35 uevent
root@grsnis01:~#
</pre>

=== Intel E3-1245 v6 ===

<pre>
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --status
edac-ctl: drivers are loaded.
[root@alphagdaq ~]# edac-util
edac-util: No errors to report.
[root@alphagdaq ~]# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
[root@alphagdaq ~]# ras-mc-ctl --layout
+-----------------------------------------------+
| mc0 |
| csrow0 | csrow1 | csrow2 | csrow3 |
----------+-----------------------------------------------+
channel1: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
----------+-----------------------------------------------+
[root@alphagdaq ~]# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#3channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#3channel#1 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#2channel#0 0 0
[root@alphagdaq ~]# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SSH-F
[root@alphagdaq ~]# ras-mc-ctl --summary
DBD::SQLite::db prepare failed: no such table: mc_event at /usr/sbin/ras-mc-ctl line 1129.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1130.
[root@alphagdaq ~]#
</pre>

=== AMD 3700X ===

(memory is non-ECC)

<pre>
root@daq13:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@daq13:~#
root@daq13:~#
root@daq13:~# edac-ctl --status
edac-ctl: drivers not loaded.
root@daq13:~# edac-util
edac-util: Error: No memory controller data found.
root@daq13:~# edac-util -s
edac-util: EDAC drivers loaded. No memory controllers found
root@daq13:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 2 root root 0 Jan 25 15:26 power
lrwxrwxrwx 1 root root 0 Jan 21 16:16 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Jan 21 16:16 uevent
</pre>

(memory is ECC)

<pre>
root@trinatdaq:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@trinatdaq:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@trinatdaq:~# edac-util
edac-util: No errors to report.
root@trinatdaq:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Dec 15 13:04 mc0
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
lrwxrwxrwx 1 root root 0 Dec 13 18:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dec 13 18:31 uevent
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_noinfo_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 max_location
-r--r--r-- 1 root root 4096 Dec 15 13:04 mc_name
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank4
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank5
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank6
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank7
--w------- 1 root root 4096 Dec 15 13:04 reset_counters
-rw-r--r-- 1 root root 4096 Dec 15 13:04 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Dec 15 13:04 seconds_since_reset
-r--r--r-- 1 root root 4096 Dec 15 13:04 size_mb
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Dec 15 13:04 uevent
root@trinatdaq:~#
</pre>

=== AMD 5000G ===

* no linux driver for AMD 5000-series "G" CPU
* no mention of ECC in the BIOS settings
* unclear status of ECC support in AMD documentation (sais only "pro" "G" CPUs have ECC)
* unclear status of ECC support in ASUS documentation (web page out of date)

=== AMD 5600X ===

<pre>
root@daq17:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-XE GAMING WIFI
root@daq17:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq17:~# edac-util
edac-util: No errors to report.
root@daq17:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@daq17:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Aug 19 19:27 mc0
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
lrwxrwxrwx 1 root root 0 May 10 10:11 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 May 10 10:11 uevent
root@daq17:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_noinfo_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 max_location
-r--r--r-- 1 root root 4096 Aug 19 19:27 mc_name
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank4
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank5
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank6
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank7
--w------- 1 root root 4096 Aug 19 19:27 reset_counters
-rw-r--r-- 1 root root 4096 Aug 19 19:27 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Aug 19 19:27 seconds_since_reset
-r--r--r-- 1 root root 4096 Aug 19 19:27 size_mb
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Aug 19 19:27 uevent
root@daq17:~#
</pre>

=== AMD 3955WX ===

<pre>
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --status
edac-ctl: drivers are loaded.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util
edac-util: No errors to report.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@alphasuperdaq:~/git/scripts/quotareport# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 19 root root 0 Dez 12 04:48 mc0
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
lrwxrwxrwx 1 root root 0 Dez 9 05:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dez 9 05:31 uevent
root@alphasuperdaq:~/git/scripts/quotareport#
root@alphasuperdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 max_location
-r--r--r-- 1 root root 4096 Feb 28 22:19 mc_name
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank0
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank1
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank10
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank11
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank12
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank13
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank14
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank15
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank2
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank3
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank4
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank5
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank6
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank7
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank8
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank9
--w------- 1 root root 4096 Feb 28 22:19 reset_counters
-rw-r--r-- 1 root root 4096 Feb 28 22:19 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Feb 28 22:19 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 28 22:19 size_mb
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 28 22:19 uevent
root@alphasuperdaq:~#
root@alphasuperdaq:~# ras-mc-ctl --layout
Use of uninitialized value $max_pos[3] in modulus (%) at /usr/sbin/ras-mc-ctl line 868.
Use of uninitialized value $d in numeric ge (>=) at /usr/sbin/ras-mc-ctl line 869.
Use of uninitialized value $d in sprintf at /usr/sbin/ras-mc-ctl line 872.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| mc0 |
| csrow0 | csrow1 |
| channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 | channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

0: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
root@alphasuperdaq:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#2 0 0
mc#0csrow#1channel#7 0 0
mc#0csrow#0channel#3 0 0
mc#0csrow#1channel#4 0 0
mc#0csrow#1channel#2 0 0
mc#0csrow#0channel#7 0 0
mc#0csrow#1channel#3 0 0
mc#0csrow#0channel#4 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#5 0 0
mc#0csrow#0channel#6 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#0channel#5 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#6 0 0
root@alphasuperdaq:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: ASUSTeK COMPUTER INC. model Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@alphasuperdaq:~#
</pre>

=== AMD 7700X ===

<pre>
root@dsfe05:~# apt install edac-utils
root@dsfe05:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro H13SAE-MF
root@dsfe05:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@dsfe05:~# edac-util
edac-util: No errors to report.
root@dsfe05:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@dsfe05:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 May 14 09:33 ce_count
-r--r--r-- 1 root root 4096 May 14 09:33 ce_noinfo_count
-r--r--r-- 1 root root 4096 May 14 09:33 max_location
-r--r--r-- 1 root root 4096 May 14 09:33 mc_name
drwxr-xr-x 2 root root 0 May 14 09:33 power
drwxr-xr-x 3 root root 0 May 14 09:33 rank4
drwxr-xr-x 3 root root 0 May 14 09:33 rank5
--w------- 1 root root 4096 May 14 09:33 reset_counters
-r--r--r-- 1 root root 4096 May 14 09:33 seconds_since_reset
-r--r--r-- 1 root root 4096 May 14 09:33 size_mb
-r--r--r-- 1 root root 4096 May 14 09:33 ue_count
-r--r--r-- 1 root root 4096 May 14 09:33 ue_noinfo_count
-rw-r--r-- 1 root root 4096 May 14 09:33 uevent
root@dsfe05:~#
</pre>

== Configure rasdaemon ==

<pre>
apt install rasdaemon
</pre>
<pre>
systemctl enable rasdaemon
systemctl restart rasdaemon
systemctl status rasdaemon
</pre>

<pre>
● rasdaemon.service - RAS daemon to log the RAS events
Loaded: loaded (/lib/systemd/system/rasdaemon.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-01-25 15:16:37 PST; 3min 5s ago
Main PID: 2477175 (rasdaemon)
Tasks: 1 (limit: 76958)
Memory: 17.1M
CGroup: /system.slice/rasdaemon.service
└─2477175 /usr/sbin/rasdaemon -f -r

Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Listening to events for cpus 0 to 11
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mc_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording aer_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording extlog_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mce_record events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording arm_event events
</pre>

== Get reports ==

* Intel 2x32GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --layout
+-------------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-------------------------+
channel1: | 16384 MB | 16384 MB |
channel0: | 16384 MB | 16384 MB |
----------+-------------------------+
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#0channel#1 0 0
root@daq00:~#
</pre>

* Intel 4x16GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#1 0 0
mc#0csrow#2channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#3channel#0 0 0
mc#0csrow#3channel#1 0 0
root@daq00:~#
root@daq00:~# ras-mc-ctl --layout
+-----------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-----------------------+
channel1: | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB |
----------+-----------------------+
root@daq00:~#
root@daq00:~#
root@daq00:~#
root@daq00:~# ras-mc-ctl --print-labels
ras-mc-ctl: Error: No dimm labels for Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@daq00:~#
</pre>

note: ubuntu LTS 22.04 DBD::SQLite::db error is not there.

* AMD 7700 2x32GB DDR5 ECC DIMMs

<pre>
root@dsfe05:~# systemctl status rasdaemon
● rasdaemon.service - RAS daemon to log the RAS events
Loaded: loaded (/lib/systemd/system/rasdaemon.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2024-05-14 09:36:43 PDT; 33ms ago
Process: 4088418 ExecStartPost=/usr/sbin/rasdaemon --enable (code=exited, status=0/SUCCESS)
Main PID: 4088417 (rasdaemon)
Tasks: 1 (limit: 37300)
Memory: 788.0K
CPU: 5ms
CGroup: /system.slice/rasdaemon.service
└─4088417 /usr/sbin/rasdaemon -f -r

May 14 09:36:43 dsfe05 rasdaemon[4088417]: ras:aer_event event enabled
May 14 09:36:43 dsfe05 rasdaemon[4088417]: Enabled event ras:aer_event
May 14 09:36:43 dsfe05 rasdaemon[4088417]: mce:mce_record event enabled
May 14 09:36:43 dsfe05 rasdaemon[4088417]: Enabled event mce:mce_record
May 14 09:36:43 dsfe05 rasdaemon[4088417]: ras:extlog_mem_event event enabled
May 14 09:36:43 dsfe05 rasdaemon[4088417]: Enabled event ras:extlog_mem_event
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording mc_event events
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording aer_event events
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording extlog_event events
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording mce_record events
root@dsfe05:~# ras-mc-ctl --layout
Use of uninitialized value $max_pos[3] in modulus (%) at /usr/sbin/ras-mc-ctl line 907.
Use of uninitialized value $d in numeric ge (>=) at /usr/sbin/ras-mc-ctl line 908.
Use of uninitialized value $d in sprintf at /usr/sbin/ras-mc-ctl line 911.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
+-----------------------------------------------------------------------------------------------+
| mc0 |
| csrow0 | csrow1 | csrow2 | csrow3 |
| channel0 | channel1 | channel0 | channel1 | channel0 | channel1 | channel0 | channel1 |
----+-----------------------------------------------------------------------------------------------+

0: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB |
----+-----------------------------------------------------------------------------------------------+
root@dsfe05:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#2channel#1 0 0
mc#0csrow#2channel#0 0 0
root@dsfe05:~# ras-mc-ctl --print-labels
ras-mc-ctl: Error: No dimm labels for Supermicro model H13SAE-MF
root@dsfe05:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model H13SAE-MF
root@dsfe05:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

No MCE errors.
root@dsfe05:~#
</pre>

= sensors =

== ASUS P7P55D EVO ==

* BIOS version 2101

<pre>
root@iris01:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Core 0: +34.0°C (high = +83.0°C, crit = +99.0°C)
Core 1: +37.0°C (high = +83.0°C, crit = +99.0°C)
Core 2: +38.0°C (high = +83.0°C, crit = +99.0°C)
Core 3: +35.0°C (high = +83.0°C, crit = +99.0°C)

nouveau-pci-0100
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.05 V)
temp1: +46.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

atk0110-acpi-0
Adapter: ACPI interface
Vcore Voltage: 864.00 mV (min = +0.80 V, max = +1.60 V)
+3.3V Voltage: 3.38 V (min = +2.97 V, max = +3.63 V)
+5V Voltage: 5.04 V (min = +4.50 V, max = +5.50 V)
+12V Voltage: 12.15 V (min = +10.20 V, max = +13.80 V)
CPU Fan Speed: 968 RPM (min = 600 RPM, max = 7200 RPM)
Chassis1 Fan Speed: 1288 RPM (min = 600 RPM, max = 7200 RPM)
Chassis2 Fan Speed: 1316 RPM (min = 600 RPM, max = 7200 RPM)
Power Fan Speed: 0 RPM (min = 0 RPM, max = 7200 RPM)
CPU Temperature: +34.0°C (high = +45.0°C, crit = +45.5°C)
MB Temperature: +30.0°C (high = +45.0°C, crit = +46.0°C)

root@iris01:~#
</pre>

== ASUS Z170-DELUXE ==

* BIOS version 3801
* load sensors drivers
<pre>
echo modprobe coretemp >> /etc/rc.local
echo modprobe jc42 >> /etc/rc.local
echo modprobe lm92 >> /etc/rc.local
echo modprobe nct6775 >> /etc/rc.local
</pre>
* in /boot/grub/grub.cfg, add: GRUB_CMDLINE_LINUX_DEFAULT="acpi_enforce_resources=no"
* update grub and reboot: grub-mkconfig -o /boot/grub/grub.cfg

<pre>
root@iris00:~# sensors
nct6793-isa-0290
Adapter: ISA adapter
in0: 600.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 144.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in6: 0.00 V (min = +0.00 V, max = +0.00 V)
in7: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.14 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1000.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in10: 600.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 592.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 968.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1370 RPM (min = 0 RPM)
fan2: 1437 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan6: 0 RPM (min = 0 RPM)
SYSTIN: +32.0°C (high = +98.0°C, hyst = +95.0°C) sensor = thermistor
CPUTIN: +42.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: -128.0°C sensor = thermistor
AUXTIN1: +50.0°C sensor = thermistor
AUXTIN2: +22.0°C sensor = thermistor
AUXTIN3: +28.0°C sensor = thermistor
PECI Agent 0: +50.0°C (high = +98.0°C, hyst = +95.0°C)
(crit = +100.0°C)
PECI Agent 0 Calibration: +42.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
TSI2_TEMP: +3892314.0°C
TSI3_TEMP: +3892314.0°C
TSI4_TEMP: +3892314.0°C
TSI5_TEMP: +3892314.0°C
TSI6_TEMP: +3892314.0°C
TSI7_TEMP: +3892314.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

jc42-i2c-0-1a
Adapter: SMBus I801 adapter at f040
temp1: +36.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

jc42-i2c-0-18
Adapter: SMBus I801 adapter at f040
temp1: +34.8°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

jc42-i2c-0-1b
Adapter: SMBus I801 adapter at f040
temp1: +35.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

jc42-i2c-0-19
Adapter: SMBus I801 adapter at f040
temp1: +36.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +52.0°C (high = +84.0°C, crit = +100.0°C)
Core 0: +52.0°C (high = +84.0°C, crit = +100.0°C)
Core 1: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 2: +48.0°C (high = +84.0°C, crit = +100.0°C)
Core 3: +47.0°C (high = +84.0°C, crit = +100.0°C)

root@iris00:~#
</pre>

== ASUS H110M-A/M.2 ==

* BIOS version 4202
* echo modprobe coretemp >> /etc/rc.local
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@midpol:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +33.0°C (high = +80.0°C, crit = +100.0°C)
Core 0: +33.0°C (high = +80.0°C, crit = +100.0°C)
Core 1: +30.0°C (high = +80.0°C, crit = +100.0°C)

acpitz-acpi-0
Adapter: ACPI interface
temp1: +27.8°C (crit = +119.0°C)
temp2: +29.8°C (crit = +119.0°C)

nct6793-isa-0290
Adapter: ISA adapter
in0: 368.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 152.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in6: 928.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.14 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1000.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in10: 152.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in11: 128.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in12: 136.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in13: 120.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 136.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1004 RPM (min = 0 RPM)
fan2: 1143 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan6: 0 RPM (min = 0 RPM)
SYSTIN: +118.0°C (high = +98.0°C, hyst = +95.0°C) sensor = thermistor
CPUTIN: +29.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +30.0°C sensor = thermistor
AUXTIN1: +112.0°C sensor = thermistor
AUXTIN2: +111.0°C sensor = thermistor
AUXTIN3: +110.0°C sensor = thermistor
PECI Agent 0: +31.0°C (high = +98.0°C, hyst = +95.0°C)
(crit = +100.0°C)
PECI Agent 0 Calibration: +36.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
TSI2_TEMP: +3892314.0°C
TSI3_TEMP: +3892314.0°C
TSI4_TEMP: +3892314.0°C
TSI5_TEMP: +3892314.0°C
TSI6_TEMP: +3892314.0°C
TSI7_TEMP: +3892314.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

root@midpol:~#
</pre>

== ASUS P9X79 WS ==

* https://www.asus.com/supportonly/P9X79%20WS/HelpDesk_Manual/
* BIOS version 4802
* modprobe nct6775
* modprobe coretemp

<pre>
root@daq14:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 0: +29.0°C (high = +82.0°C, crit = +100.0°C)
Core 1: +24.0°C (high = +82.0°C, crit = +100.0°C)
Core 2: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +82.0°C, crit = +100.0°C)

nouveau-pci-0200
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.00 V)
temp1: +39.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

nct6776-isa-0290
Adapter: ISA adapter
Vcore: 1.04 V (min = +0.00 V, max = +1.74 V)
in1: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
AVCC: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
+3.3V: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 2.04 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 904.00 mV (min = +0.00 V, max = +0.00 V) ALARM
3VSB: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
Vbat: 3.30 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1265 RPM (min = 0 RPM)
fan2: 1909 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +34.0°C (high = +0.0°C, hyst = +0.0°C) ALARM sensor = thermistor
CPUTIN: +58.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermal diode
AUXTIN: +31.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
PECI Agent 0: +31.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +96.0°C)
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

root@daq14:~#
</pre>

== ASUS TUF GAMING B550M-PLUS WIFI II ==

* BIOS 2803, 2806
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@midm9a:~# sensors
nct6798-isa-0290
Adapter: ISA adapter
in0: 488.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.82 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in14: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 0 RPM (min = 0 RPM)
fan2: 760 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan7: 1264 RPM (min = 0 RPM)
SYSTIN: +25.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +22.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +95.0°C sensor = thermistor
AUXTIN1: +25.0°C sensor = thermistor
AUXTIN2: +25.0°C sensor = thermistor
AUXTIN3: +25.0°C sensor = thermistor
PECI Agent 0 Calibration: +23.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +32.4°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

amdgpu-pci-0800
Adapter: PCI adapter
vddgfx: 1.45 V
vddnb: 993.00 mV
edge: +28.0°C
PPT: 20.00 W

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +33.4°C

root@midm9a:~#
</pre>

== ASUS ASUS ROG STRIX B550-XE GAMING WIFI ==

* BIOS 2423, 2604
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@daq13:~# sensors
nct6798-isa-0290
Adapter: ISA adapter
in0: 344.00 mV (min = +0.00 V, max = +1.74 V)
in1: 992.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in6: 216.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.30 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.81 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in11: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 280.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 845 RPM (min = 0 RPM)
fan2: 998 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +28.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +27.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +94.0°C sensor = thermistor
AUXTIN1: +28.0°C sensor = thermistor
AUXTIN2: +28.0°C sensor = thermistor
AUXTIN3: +97.0°C sensor = thermistor
PECI Agent 0 Calibration: +27.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +33.6°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

amdgpu-pci-0600
Adapter: PCI adapter
vddgfx: 1.45 V
vddnb: 999.00 mV
edge: +29.0°C
PPT: 14.00 W

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: +30.0°C

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +33.9°C

root@daq13:~#
</pre>

== ASUS ASUS ROG STRIX B550-E GAMING ==

* bios 2803
* echo modprobe jc42 >> /etc/rc.local
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@daq17:~# sensors
jc42-i2c-1-1b
Adapter: SMBus PIIX4 adapter port 0 at 0b00
temp1: +25.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: +28.0°C

nouveau-pci-0800
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.00 V)
temp1: +34.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

nct6798-isa-0290
Adapter: ISA adapter
in0: 288.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 224.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.31 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.79 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 280.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 843 RPM (min = 0 RPM)
fan2: 629 RPM (min = 0 RPM)
fan3: 746 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +22.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +25.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +93.0°C sensor = thermistor
AUXTIN1: +22.0°C sensor = thermistor
AUXTIN2: +22.0°C sensor = thermistor
AUXTIN3: +96.0°C sensor = thermistor
PECI Agent 0 Calibration: +25.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +27.6°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

jc42-i2c-1-1a
Adapter: SMBus PIIX4 adapter port 0 at 0b00
temp1: +23.2°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

asusec-isa-0000
Adapter: ISA adapter
CPU_Opt: 0 RPM
Chipset: +34.0°C
CPU: +25.0°C
Motherboard: +22.0°C
T_Sensor: -40.0°C
VRM: +31.0°C

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +28.0°C
Tccd1: +27.5°C

root@daq17:~#
</pre>

== ASUS PRIME B650-PLUS ==

* BIOS 1811
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@dsdaqgw:~# sensors
amdgpu-pci-0b00
Adapter: PCI adapter
vddgfx: 930.00 mV
vddnb: 1.19 V
edge: +38.0°C
PPT: 25.10 W

nct6799-isa-0290
Adapter: ISA adapter
in0: 920.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.04 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 320.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.28 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.28 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.10 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.04 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 416.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 328.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 0 RPM (min = 0 RPM)
fan2: 1253 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan7: 0 RPM (min = 0 RPM)
SYSTIN: +33.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +35.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +78.0°C sensor = thermistor
AUXTIN1: +11.0°C sensor = thermistor
AUXTIN2: +20.0°C sensor = thermistor
AUXTIN3: +82.0°C sensor = thermistor
PECI Agent 0 Calibration: +35.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +42.6°C
intrusion0: ALARM
intrusion1: OK
beep_enable: disabled

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +42.6°C
Tccd1: +36.4°C

root@dsdaqgw:~#
</pre>

= Enable CPU turbo mode =

* Intel CPU has a nominal CPU frequency (i.e. 3.4GHz) and a turbo-boost CPU frequency (i.e. 4.0GHz). Here we will enable this turbo-boost mode.
* Find out CPU capability
<pre>
root@daq01:~# lscpu | grep Hz
Model name: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
CPU MHz: 3965.803
CPU max MHz: 4000.0000
CPU min MHz: 800.0000
root@daq01:~#
</pre>
* Look up this CPU in the Intel ARK database - google for the CPU model name, i.e.
https://ark.intel.com/content/www/us/en/ark/products/88196/intel-core-i7-6700-processor-8m-cache-up-to-4-00-ghz.html
* Find current frequency settings:
<pre>
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "powersave" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 2.72 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
root@daq01:~#
</pre>
* Note the following:
** current governor is "powersave"
** "performance" governor is available
** "boost state support" is supported and active.
* Confirm CPU frequency governor:
<pre>
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
powersave
powersave
powersave
powersave
powersave
powersave
powersave
powersave
root@daq01:~#
</pre>
* Change governor to "performance":
<pre>
root@daq01:~# cpupower frequency-set --governor performance
Setting cpu: 0
Setting cpu: 1
Setting cpu: 2
Setting cpu: 3
Setting cpu: 4
Setting cpu: 5
Setting cpu: 6
Setting cpu: 7
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
performance
performance
performance
performance
performance
performance
performance
performance
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "performance" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 3.93 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
</pre>
* monitor CPU frequency:
<pre>
root@daq01:~# cpupower monitor
| Nehalem || Mperf || Idle_Stats
CPU| C3 | C6 | PC3 | PC6 || C0 | Cx | Freq || POLL | C1 | C1E | C3 | C6 | C7s | C8
0| 0.00| 0.00| 0.00| 0.00|| 88.80| 11.20| 3973|| 0.00| 0.00| 0.01| 0.02| 0.31| 0.00| 4.25
4| 0.00| 0.00| 0.00| 0.00|| 4.70| 95.30| 3945|| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 95.03
1| 0.73| 3.70| 0.00| 0.00|| 4.52| 95.48| 3864|| 0.00| 0.01| 1.19| 0.44| 2.82| 0.00| 90.23
5| 0.73| 3.70| 0.00| 0.00|| 0.37| 99.63| 3807|| 0.00| 0.00| 0.03| 0.09| 1.70| 0.00| 97.64
2| 2.28| 12.86| 0.00| 0.00|| 1.41| 98.59| 3829|| 0.00| 0.86| 3.17| 0.46| 7.70| 0.00| 85.87
6| 2.28| 12.86| 0.00| 0.00|| 2.88| 97.12| 3856|| 0.00| 0.11| 4.56| 2.15| 10.31| 0.00| 78.99
3| 1.33| 4.81| 0.00| 0.00|| 0.99| 99.01| 3804|| 0.00| 0.49| 0.79| 0.01| 1.03| 0.00| 96.12
7| 1.34| 4.81| 0.00| 0.00|| 1.26| 98.74| 3818|| 0.00| 0.01| 2.32| 0.47| 5.02| 0.00| 90.06
root@daq01:~#
</pre>
* check that the CPU is not overheating:
<pre>
root@daq01:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 1: +38.0°C (high = +84.0°C, crit = +100.0°C)
Core 2: +34.0°C (high = +84.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +84.0°C, crit = +100.0°C)
</pre>
* congratulations, we are running at 4 GHz now!

= Setup ubuntu as gateway to private network =

See also:
* https://daq.triumf.ca/DaqWiki/index.php/VME-CPU#Setup_the_boot_host_computer_.28el7.29
* http://www.triumf.info/wiki/DAQwiki/index.php/Dhcpd_on_eth1

== Steps to do ==

!!! UPDATED 16feb2024 Ubuntu-22.04.03 !!!

* assign network numbers to the private network, i.e. 192.168.1.x, 192.168.2.x, etc
* (on the gateway machine, each private network interface has to have a different network number)
* (each network interface can have multiple networks attached, via VLANs or via eth0:0, eth0:1 constructs)
* assign IP addresses on the private network, save them in /etc/hosts i.e. "hvps 192.168.1.10"
* (for simplicity, assign 192.168.1.1 to the gateway machine itself)
* (IP addresses 192.168.1.0 and 192.168.1.255 are "special", do not use them)
* setup DNS server (dnsmasq) to serve contents of /etc/hosts via DNS (otherwise, many programs will see inconsistent name to IP address mapping)
* setup DHCP server (dnsmasq) to give out the IP addresses
* setup TFTP server (dnsmasq), pxelinux and NFS for diskless booting
* setup time server (chronyd) to provide common time to all devices
* setup NAT so machines on private network can access the internet (to get OS updates, etc)
* setup NIS and NFS so machines on the private network can use common home directories
* setup rsync backup of machines on the private network

== setup hosts ==

* edit /etc/hosts
<pre>
192.168.1.101 dsfe01
... and so forth
</pre>

== setup dns and dhcp ==

!!! updated 16feb2024 for Ubuntu 22.04.3 !!!

!!! note: stock systemd-resolved remains, is configured to forward queries to dnsmasq, configured to forward queries to TRIUMF DNS !!!

!!! note: per authors of systemd, bare hostnames are not permitted, a DNS domain name must always be used. DNS domain name "dsdaq" is used in this example !!!

* apt install dnsmasq
* ensure dnsmasq starts after all interfaces are up (Ubuntu-22)
<pre>
mkdir /etc/systemd/system/dnsmasq.service.d
echo -e "[Unit]\nAfter=network-online.target\n" > /etc/systemd/system/dnsmasq.service.d/local.conf
</pre>
* edit /etc/dnsmasq.conf
<pre>
# /etc/dnsmasq.conf
# DNS settings
#port=0 # disable DNS function
port=53 # enable DNS function
bind-interfaces # do not collide with systemd-resolved, we use 127.0.0.1:53, they use 127.0.0.53:53
domain-needed
bogus-priv
no-resolv
#log-queries # log DNS quesries

# TRIUMF DNS settings

server=142.90.100.19
expand-hosts
domain=dsdaq
local=/dsdaq/
localmx # do not forward MX queries to TRIUMF

# DHCP settings
interface=enp1s0f0 # VX network 192.168.0.x
#interface=missing # FEP and TSP network 192.168.1.x
interface=enp1s0f1 # controls network 192.168.2.x
#dhcp-range=192.168.1.50,192.168.1.150,infinite
dhcp-range=192.168.0.0,static
dhcp-range=192.168.2.0,static
log-dhcp # log DHCP queries
#quiet-dhcp
dhcp-ignore=tag:!known
#dhcp-boot=pxelinux.0

dhcp-option=option:dns-server,192.168.0.248
dhcp-option=option:ntp-server,192.168.0.248

# TFTP settings

enable-tftp
tftp-root=/tftpboot
</pre>
* #mkdir /tftpboot ### per tftp-root (if no ZFS)
* zfs create -o mountpoint=/tftpboot rpool/tftpboot ### (if root is ZFS)
* create resolved-dsdaq.conf with main IP address of dnsmasq
<pre>
[Resolve]
DNS=192.168.0.248
Domains=dsdaq triumf.ca
</pre>
* mkdir -p /etc/systemd/resolved.conf.d/
* /bin/rm -f /etc/systemd/resolved.conf.d/*.conf
* cp resolved-dsdaq.conf /etc/systemd/resolved.conf.d/
* systemctl stop systemd-resolved.service
* systemctl disable systemd-resolved.service
* systemctl enable dnsmasq
* systemctl restart dnsmasq
* try to "ping" or "host" some names from /etc/hosts, it should work
* try to ping daq00, daq00.triumf.ca, all should work
* resolved-dsdaq.conf goes into /etc/systemd/resolved.conf.d/ of all machines on the private network
* if not using systemd-resolved, edit /etc/resolv.conf

== setup chronyd ==

* enable ntp server:
* disable systemd-timesyncd, configure and enable chronyd per instructions above
* create dsdaq.conf
<pre>
# chrony config for dsdaq server

#allow 192.168.0.0
#allow 192.168.1.0
#allow 192.168.2.0
allow all

# end
</pre>
* cp dsdaq.conf /etc/chrony/conf.d/
* systemctl restart chronyd
* chronyc tracking ### wait until time is synchronized (a few seconds)
* create dsdaq.sources # use hostname or IP address of chronyd server
<pre>
# Put this file in /etc/chrony/sources.d
# systemctl restart chrony
# chronyc sources
# chronyc tracking
server dsdaqgw iburst prefer
# end
</pre>
* dsdaq.sources goes to /etc/chrony/sources.d of all machines on the private network

== setup diskless network booting ==

=== setup pxelinux for legacy pxe boot ===

* add bits in dnsmasq.conf
<pre>
dhcp-host=ac:1f:6b:9e:7f:4a,dsfe01,infinite
dhcp-boot=pxelinux.0
dhcp-option=17,"192.168.0.251:/nfsroot/%s,vers=3"
</pre>
* setup pxelinux for Ubuntu-18
<pre>
cd ~
wget https://www.kernel.org/pub/linux/utils/boot/syslinux/4.xx/syslinux-4.03.tar.bz2
tar xjvf syslinux-4.03.tar.bz2
cd syslinux-4.03
cp -pv ./core/pxelinux.0 ./com32/hdt/hdt.c32 ./memdisk/memdisk ./com32/menu/menu.c32 /zssd/tftpboot/
</pre>
* cd /zssd/tftpboot
<pre>
wget http://ladd00.triumf.ca/tftpboot/memtest86+-4.20.iso.zip
wget http://ladd00.triumf.ca/tftpboot/memtest86+-5.01.iso.gz
wget http://ladd00.triumf.ca/tftpboot/modules.alias
wget http://ladd00.triumf.ca/tftpboot/modules.pcimap
wget http://ladd00.triumf.ca/tftpboot/pci.ids
</pre>
* mkdir pxelinux.cfg
* emacs -nw pxelinux.cfg/default
<pre>
default menu.c32
prompt 0

menu title Welcome to the DSVSLICE PXE boot menu

timeout 50

label hdt
kernel hdt.c32

label memtest86+-5.01
kernel memdisk iso initrd=memtest86+-5.01.iso.gz

label memtest86+-4.20
kernel memdisk iso initrd=memtest86+-4.20.iso.zip

label vmlinuz-5.3.0-26-generic
menu default
kernel vmlinuz-5.3.0-26-generic
append initrd=initrd.img-5.3.0-26-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.1.1:/zssd/nfsroot/dsfe01 toram ip=dhcp panic=60 BOOTIF=enp1s0f0

#end
</pre>

=== setup pxelinux for efi pxe boot ===

* https://c-nergy.be/blog/?p=13808
* add dnsmasq.conf bits. note: root-path does not actually work, it is hardwired pxelinux.cfg/default file.
<pre>
# uefi pxe

dhcp-boot=tag:uefipxe,uefi/syslinux.efi
dhcp-option-force=tag:fe01,option:root-path,192.168.0.248:/nfsroot/fe01

# VX network 192.168.0.x

dhcp-host=40:a6:b7:c1:d9:c5,fe01,infinite,set:uefipxe,set:fe01
</pre>
* apt install syslinux pxelinux syslinux-common syslinux-efi syslinux-utils
<pre>
mkdir /tftpboot/uefi
cp /usr/lib/SYSLINUX.EFI/efi64/syslinux.efi /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/ldlinux.e64 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/menu.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/hdt.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libutil.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libmenu.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libcom32.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libgpl.c32 /tftpboot/uefi/
</pre>
* try to boot, it should bomb with "cannot load pxelinux.cfg/default"
* mkdir /tftpboot/uefi/pxelinux.cfg
* create /tftpboot/uefi/pxelinux.cfg/default, note nfsroot path is hardwired, note "http:" is used to load vmlinuz and initrd files (because tftp is super slow)
<pre>
default menu.c32
prompt 0

menu title Welcome to the DSDAQGW UEFI PXE boot menu

timeout 50

label vmlinuz-6.5.0-17-generic
kernel http://192.168.0.248:8088/uefi/vmlinuz-6.5.0-17-generic
append initrd=http://192.168.0.248:8088/uefi/initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=auto rw ip=dhcp panic=60

# append initrd=http://192.168.0.248:8088/uefi/initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.0.248:/nfsroot/fe01 rw ip=dhcp panic=60

# append initrd=initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.0.248:/nfsroot/fe01 rw ip=dhcp panic=60
# append initrd=initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=auto ip=dhcp rw panic=60

#end
</pre>
* try to boot, it will bomb with "cannot load http://...."
* install mini_httpd on port 8088, see https://acme.com/software/mini_httpd/
<pre>
apt install mini-httpd
emacs -nw /etc/default/mini-httpd # set "START=1"
emacs -nw /etc/mini-httpd.conf # set "host=192.168.0.248", "port=8088", "data_dir=/tftpboot"
mkdir /etc/systemd/system/mini-httpd.service.d
echo -e "[Unit]\nAfter=network-online.target\n" > /etc/systemd/system/mini-httpd.service.d/local.conf
systemctl enable mini-httpd
systemctl restart mini-httpd
systemctl status mini-httpd
wget http://192.168.0.248:8088/uefi/syslinux.efi
tail -100 /var/log/mini_httpd.log
</pre>
* fix initramfs bug for "nfsroot=auto", otherwise, "nfsroot=" has to be different for each machine and you have to have separate pxelinux config files for each machine
** emacs -nw /usr/lib/initramfs-tools/etc/dhcp/dhclient-enter-hooks.d/config
** add "echo ROOTPATH=..." if it is missing (Ubuntu LTS 22.04)
<pre>
echo "ROOTSERVER='${new_routers%% *}'"
echo "ROOTPATH='$new_root_path'"
echo "HOSTNAME='$new_host_name'"
</pre>
** regenerate initramfs (be careful you generate it for the right kernel!)
** see https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/2054482
<pre>
mkinitramfs 6.5.0-18-generic
</pre>
* copy linux kernel and initrd
<pre>
cp /boot/vmlinuz-6.5.0-18-generic /tftpboot/uefi/
cp /boot/initrd.img-6.5.0-18-generic /tftpboot/uefi/
chmod a+r /tftpboot/uefi/*
</pre>
* try to boot, should bomb with messages about "trying to mount root filesystem"
* tail /var/log/syslog
<pre>
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 vendor class: PXEClient:Arch:00007:UNDI:003016
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 1:netmask, 2:time-offset, 3:router, 4, 5,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 6:dns-server, 12:hostname, 13:boot-file-size,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 15:domain-name, 17:root-path, 18:extension-path,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 22:max-datagram-reassembly, 23:default-ttl,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 28:broadcast, 40:nis-domain, 41:nis-server,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 42:ntp-server, 43:vendor-encap, 50:requested-address,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 51:lease-time, 54:server-identifier, 58:T1,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 59:T2, 60:vendor-class, 66:tftp-server, 67:bootfile-name,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 97:client-machine-id, 128, 129, 130, 131,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 132, 133, 134, 135
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 next server: 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 broadcast response
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 1 option: 53 message-type 2
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 18 option: 67 bootfile-name uefi/syslinux.efi
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 12 hostname fe01
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 vendor class: PXEClient:Arch:00007:UNDI:003016
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 1:netmask, 2:time-offset, 3:router, 4, 5,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 6:dns-server, 12:hostname, 13:boot-file-size,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 15:domain-name, 17:root-path, 18:extension-path,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 22:max-datagram-reassembly, 23:default-ttl,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 28:broadcast, 40:nis-domain, 41:nis-server,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 42:ntp-server, 43:vendor-encap, 50:requested-address,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 51:lease-time, 54:server-identifier, 58:T1,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 59:T2, 60:vendor-class, 66:tftp-server, 67:bootfile-name,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 97:client-machine-id, 128, 129, 130, 131,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 132, 133, 134, 135
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 next server: 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 broadcast response
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 1 option: 53 message-type 5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 18 option: 67 bootfile-name uefi/syslinux.efi
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 12 hostname fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: error 8 User aborted the transfer received from 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/syslinux.efi to 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/syslinux.efi to 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 requested options: 1:netmask, 3:router, 6:dns-server
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 bootfile name: uefi/syslinux.efi
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 next server: 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 broadcast response
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 1 option: 53 message-type 2
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 requested options: 1:netmask, 3:router, 6:dns-server
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 bootfile name: uefi/syslinux.efi
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 next server: 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 broadcast response
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 1 option: 53 message-type 5
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/ldlinux.e64 to 192.168.0.110
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/01-40-a6-b7-c1-d9-c5 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8006E not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8006 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A800 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A80 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/pxelinux.cfg/default to 192.168.0.110
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/menu.c32 to 192.168.0.110
Feb 16 20:43:10 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/libutil.c32 to 192.168.0.110
Feb 16 20:43:10 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/pxelinux.cfg/default to 192.168.0.110
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 client provides name: dsdaqgw.triumf.ca
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 1:netmask, 28:broadcast, 2:time-offset, 3:router,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 15:domain-name, 6:dns-server, 119:domain-search,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 12:hostname, 44:netbios-ns, 47:netbios-scope,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 26:mtu, 121:classless-static-route, 42:ntp-server
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 bootfile name: uefi/syslinux.efi
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 next server: 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 1 option: 53 message-type 2
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 51 lease-time infinite
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 client provides name: dsdaqgw.triumf.ca
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 1:netmask, 28:broadcast, 2:time-offset, 3:router,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 15:domain-name, 6:dns-server, 119:domain-search,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 12:hostname, 44:netbios-ns, 47:netbios-scope,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 26:mtu, 121:classless-static-route, 42:ntp-server
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 bootfile name: uefi/syslinux.efi
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 next server: 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 1 option: 53 message-type 5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 51 lease-time infinite
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 12 hostname fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw rpc.mountd[3350210]: authenticated mount request from 192.168.0.110:981 for /nfsroot/fe01 (/nfsroot/fe01)
Feb 16 20:45:07 dsdaqgw rpc.mountd[3350210]: authenticated unmount request from 192.168.0.110:859 for /nfsroot/fe01/tmp/autoDY4k5u (/nfsroot/fe01)
</pre>
* tail /var/log/mini_httpd.log
<pre>
192.168.0.110 - - [16/Feb/2024:20:43:15 -0800] "GET /uefi/vmlinuz-6.5.0-17-generic HTTP/1.0" 200 14227944 "" "Syslinux/6.04"
192.168.0.110 - - [16/Feb/2024:20:43:24 -0800] "GET /uefi/initrd.img-6.5.0-17-generic HTTP/1.0" 200 137824833 "" "Syslinux/6.04"
</pre>

=== setup efi http boot ===

https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-deployment-prep-uefi-httpboot.html

=== setup linux kernel ===

* copy the kernel files
<pre>
cd /boot
rsync -av config* initrd* System.map* vmlinuz* /tftpboot/
</pre>
* cd /tftpboot
* chmod a+r *

=== setup nfs ===

* apt-get install nfs-kernel-server
* enable NFS over UDP, edit /etc/nfs.conf add "udp=y":
<pre>
udp=y
</pre>
<pre>
systemctl restart nfs-server.service
</pre>
* emacs -nw /etc/exports
<pre>
/nfsroot/dsfe01 dsfe01(rw,no_root_squash,async,no_subtree_check)
</pre>
* enable services
<pre>
systemctl enable nfs-server
systemctl enable nfs-mountd
systemctl enable nfs-idmapd
systemctl restart nfs-server
systemctl restart nfs-mountd
systemctl restart nfs-idmapd
</pre>
* after editing /etc/exports, run
<pre>
exportfs -av
</pre>

=== setup userland ===

!!! ubuntu-18 version !!!

* zfs create rpool/nfsroot
* zfs set dedup=verify rpool/nfsroot ### enable deduplication to save disk space because most linux images have mostly identical files
* clone ubuntu
<pre>
mkdir /nfsroot/dsfe01
cd /
rsync -avx . /nfsroot/dsfe01
</pre>
* edit config files:
* cd /nfsroot/dsfe01
* emacs -nw etc/hostname ### change to dsfe01
* emacs -nw etc/mailname ### change to dsfe01
* emacs -nw etc/yp.conf ### change daq00.triumf.ca to musr00.triumf.ca
* emacs -nw etc/defaultdomain ### change to MUSR-NIS
* cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
* emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
* emacs -nw root/.ssh/authorized_keys ### update root ssh keys
* emacs -nw etc/fstab ### add this
<pre>
192.168.1.1:/nfsroot/dsfe01 / nfs defaults,nolock 0 0
</pre>
* emacs -nw etc/chrony/chrony.conf
** comment-out all "pool" and "server" entries
** add entry "server 192.168.1.1 iburst"

After dsfe01 is booted:

* disable services:
<pre>
systemctl disable apache2
systemctl disable dnsmasq
systemctl disable zfs-import-cache
</pre>

To setup additional machines, clone dsfe01 instead of cloning the gateway machine

=== Allow manpages to be viewed ===

If <code>/</code> is mounted over NFS, <code>man</code> will report a permission error. Fix it with:

<pre>
ln -s /etc/apparmor.d/usr.bin.man /etc/apparmor.d/disable/
apparmor_parser -R /etc/apparmor.d/usr.bin.man
</pre>

== setup shared home directory ==

=== on the gateway machine ===
* define netgroups
* emacs -nw /etc/netgroup
<pre>
dsfe (dsfe01,,) (dsfe02,,)
</pre>
* emacs -nw /etc/nsswitch.conf ### edit the netgroup line to read:
<pre>
netgroup: files
</pre>
* export the home directories:
* emacs -nw /etc/exports ### add this:
<pre>
/zssd/home1 @dsfe(rw,no_root_squash,async,no_subtree_check)
</pre>
* exportfs -rc

=== on the frontend machine ===

* mkdir /home
* emacs -nw /etc/fstab ### add this:
<pre>
192.168.1.1:/zssd/home1 /home nfs defaults 0 0
</pre>
* mount -a

== setup NAT ==

NAT allows machines on the private network to connect to the internet: https://en.wikipedia.org/wiki/Network_address_translation

In these examples:
* replace "eno1" with name of the outgoing interface (the one connected to the TRIUMF network).
* replace "enp11s0" with name of the private network interface (192.168.1.x network)

* emacs -nw /etc/rc.local ### add this:
<pre>
# /etc/rc.local

# enable NAT

/sbin/iptables -t nat -A POSTROUTING -o eno1 -j MASQUERADE
iptables -L -v

# uncomment following lines if machine has prohibitive FORWARD rules:
#/sbin/iptables -I FORWARD -i eno1 -o enp11s0 -m state --state RELATED,ESTABLISHED -j ACCEPT
#/sbin/iptables -I FORWARD -i enp11s0 -o eno1 -j ACCEPT
#iptables -L -v

iptables -L -v
sysctl -w net.ipv4.ip_forward=1
#sysctl -a | grep forward

sh /etc/firewall-rfc1918.sh

# end
</pre>
* emacs -nw /etc/firewall-rfc1918.sh
<pre>
# firewall-rfc1918.sh

# prevent RFC1918 private network IP addresses from
# going in and out from our uplink.

ETH=eno1

iptables -F in-rfc1918
iptables -N in-rfc1918
iptables -A in-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A in-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A in-rfc1918 --dst 192.168.0.0/16 -j REJECT

iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -I INPUT -j in-rfc1918 -i $ETH

iptables -F out-rfc1918
iptables -N out-rfc1918
iptables -A out-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A out-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A out-rfc1918 --dst 192.168.0.0/16 -j REJECT

iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -I OUTPUT -j out-rfc1918 -o $ETH

iptables -D FORWARD -j out-rfc1918 -o $ETH
iptables -D FORWARD -j out-rfc1918 -o $ETH
iptables -I FORWARD -j out-rfc1918 -o $ETH

# allow TRIUMF-SECURE network

iptables -I in-rfc1918 -s 10.90.0.0/255.255.0.0 -j ACCEPT
iptables -I out-rfc1918 -d 10.90.0.0/255.255.0.0 -j ACCEPT

# show configuration

iptables -L -v

#end
</pre>

= KVM =

<pre>
apt install cpu-checker

root@daq13:~# kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
root@daq13:~#

(if not, shutdown, go into BIOS settings, enable CPU virtualization)

apt install virtinst ### will install many packages
apt install libvirt-clients libvirt-daemon-system-systemd libvirt-daemon qemu qemu-kvm libvirt-daemon-system virtinst bridge-utils

root@daq13:/home1/wheel# virsh list --all
Id Name State
------------------------------
1 ubuntu-guest running

apt install virt-manager

virt-install --name ubuntu-guest --os-variant ubuntu20.04 --vcpus 2 --ram 2048 --location /daq/daqstore/olchansk/linux/Ubuntu/ubuntu-20.04.3-desktop-amd64.iso --network bridge=virbr0,model=virtio --graphics none --extra-args='console=ttyS0,115200n8 serial'

virtual machine will start, boot, etc
to get out of it, CTRL + Shift followed by ]

ssh wheel@daq13
virt-manager

run virt-install again, omit "--graphics none", open graphics console from virt-manager, it booted into ubuntu installer desktop

virt-install --name test10 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --filesystem /kvm_ladd00,/ --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial" --graphics none

virt-install --name test14 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --disk /tmp/xxx/ladd00.img,bus=sata --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial rdshell" --graphics none --check path_in_use=off
</pre>

build image

<pre>
dd if=/dev/zero of=/tmp/xxx/ladd00.img bs=1024M count=20
mkfs.ext3 /tmp/xxx/ladd00.img ### ext4 fails to mount by SL6 kernel, "unknown ext4 options"
cd /kvm_ladd00/
mount -o loop /tmp/xxx/ladd00.img /mnt/tmp
rsync -av . /mnt/tmp/ --delete
umount /mnt/tmp
</pre>

on the guest, configure network: /etc/rc.local
<pre>
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

ifconfig eth2 192.168.122.2
route add -net 0.0.0.0 gw 192.168.122.1
ifconfig -a
netstat -rn

# end
</pre>

= ARM64 cross-compiler =

* arm64, aarch64 are Xilinx FPGA Cortex-A53, RPi4, RPi5 machines
* install packages:
<pre>
apt install g++-12-aarch64-linux-gnu gcc-12-aarch64-linux-gnu-base libstdc++-12-dev-arm64-cross
</pre>
* run:
<pre>
aarch64-linux-gnu-gcc-12 -o ttcp.aarch64 ttcp.c -static
aarch64-linux-gnu-g++-12 -o fecdm.exe -O2 -g -Wall -Wuninitialized -std=c++20 fecdm.o dsdm.o /home/dsdaqdev/packages_common/midas/linux-aarch64-remoteonly/lib/libmidas.a -pthread -lrt -lutil /nfsroot/gdm00/usr/lib/aarch64-linux-gnu/libi2c.a -static
</pre>

= ARM cross-compiler =

NOTE!!!

THIS IS NOT AN AARCH64 (arm64) CROSSCOMPILER!

NOTE!!!

* install packages:
<pre>
apt install libgcc-9-dev-arm64-cross
apt install gcc-arm-linux-gnueabi
apt install gcc-arm-linux-gnueabihf
apt install g++-arm-linux-gnueabihf
apt install g++-arm-linux-gnueabi
</pre>
* find out the correct -march setting, on the target machine, run:
<pre>
root@gdm00:~# g++ -Q --help=target | grep march
-march= armv8-a
</pre>

<pre>
arm-linux-gnueabi-gcc -o ttcp1 ttcp.c -march=armv7 -static
arm-linux-gnueabi-gcc -o memcpy.armv7 memcpy.cc -march=armv7 -static -O2
</pre>

= 32-bit intel cross-compiler =

Ubuntu 22.04:

<pre>
apt install libstdc++-11-dev:i386
apt install zlib1g-dev:i386
</pre>

NOTES:
* "g++ -m32" does not find libstdc++, please use "g++ -m32 -L/usr/lib/gcc/i686-linux-gnu/11/"
* to cross-build 32-bit MIDAS, use "make linux32".
* executables cross-build on Ubuntu-22 do NOT run on 32-bit Debain-11 (GLIBC and GLIBCXX version mismatch)
* executables cross-build on Ubuntu-22 run on 32-bit Debian-12.

= SSH settings for EPICS =

* TRIUMF EPICS runs obsolete version of SSH
* add this to the use .ssh/config
<pre>
Host sbp1*
HostKeyAlgorithms +ssh-rsa
PubKeyAcceptedAlgorithms +ssh-rsa
KexAlgorithms +diffie-hellman-group1-sha1
ForwardX11 yes
ForwardX11Trusted yes
</pre>

= changes for VME processors =

<pre>
apt -y remove sysstat man-db
apt -y purge dkms
apt -y purge mdadm
apt -y autoremove
</pre>

DS-DM

2024-11-05T22:59:42Z

Mwalczak: /* After power up */

= DS-DM =

DarkSide-20k Global and Crate Data Manager board (GDM and CDM).

Global Data Manager (GDM):
* clock distribution to CDM boards (including GPS/atomic clock source)
* collection of trigger data from CDM boards, processing and distribution of trigger decision to CDM boards
* run control
* integration with GPS 10MHz and 1pps clocks and GPS/IRIG date and time information

Crate Data Manager (CDM):
* clock distribution from GDM to CAEN VX digitizers
* receive trigger data from CAEN VX digitizers
* send trigger data to GDM
* run control and dead time control

= Links =

* https://dsvslice.triumf.ca/vx_napoli - DS vertical slice at TRIUMF
* https://edev-group.triumf.ca/fw/exp/darkside/gcdm - git repository, DS-DM firmware
* https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0 - git repository, DS-DM board
* https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0/-/blob/master/Altium/Project%20Outputs%20for%20DS-DM-Rev0/SCH-DS-xDM-Rev0.PDF - DS-DM schematics
* https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0/-/blob/master/Testing/Clk3_XO_125_to_fpgaIN_recoveredMGTclk_to_IN2_Si5394-RevA-Project.slabtimeproj
* https://docs.xilinx.com/v/u/en-US/ds593 - Xilinx Platform Cable USB II
* https://www.enclustra.com/en/products/system-on-chip-modules/mercury-xu8/ - Enclustra
* https://bitbucket.org/team-ds-dm/ds-dm-software - git repository, DS-DM MIDAS frontend
* https://bitbucket.org/team-ds-dm/phasemeasurement - VX Phase measurement script
* https://ladd00.triumf.ca/daqinv/frontend/list/178 - inventory database
* https://daq00.triumf.ca/elog-ds/DS-DAQ - DS-DAQ elog
* https://bitbucket.org/ttriumfdaq/dsproto_vx2740/src/develop/ - VX DSFE MIDAS frontend
* https://dsdaqgw.triumf.ca/vslice/ - vertical slice DAQ
* https://daq00.triumf.ca/DaqWiki/index.php/DarkSide - DS-DAQ Wiki page
* https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0 - GPS and Rb Clock adapter board

= Onboard hardware =

* jtag chain: arm_dap_0 0x5BA00477, xczu4_1 0x04721093
* Eclustra Mercury+ XU8 module: ME-XU8-4CG-1E-D11E-R2.1
** Xilinx® Zynq Ultrascale+™ MPSoC XCZU4CG-1FBVB900E
** DDR4 ECC SDRAM (PS) 2 GB
** DDR4 SDRAM (PL) 1GB
* ethernet mac chip: AT24MAC402-SSHM-T ("602" chip is wrong)
* USB UART for Enclustra serial console, micro-USB, 115200n8
* clock chip: SI5394A-A-GM and oscillator CS-044-054.0M (54 MHz)
* U23 3.3V current meter and thermometer, LTC2990IMS#TRPBF
* LEDs:
** LED_FP A/B/C/D 0/1/2/3
** led1 - 3V3_SW_ON, SOM_POWER_GOOD - Enclustra FPGA module 3.3V power is good
** led2 - LTM4624 PGOOD
** led3 - FPGA_DONE - FPGA has booted
** led4 - TP-S-1, PCLK_P
** led5 - TP-S-2, PCLK_N
* LEMO connectors (top to bottom)
** J4 - input (NIM/TTL) (EXT_IN_LV(1), EXT_IN_LV(2))
** J5 - input (NIM/TTL) (EXT_IN_LV(3), EXT_IN_LV(4))
** J6 - external clock (GPS 10MHz and PPS)
** J7 - output (NIM/TTL) (EXT_OUT(1), EXT_OUT(2))
* SMA connectors
** J9, J10 - CLK_CCA from U6 C.C.
** J11, J12 - CLK_TP0
* RJ45 ethernet connector (100mbit: green light on, yellow light flashes when there is traffic)
* SFP connector (SFP is FTLF8526P3BNL, 6 Gbit/sec, 850 nm, 300m 50/125um OM3 MMF)
* 4 QSFP connectors (GDM)
* 6 VX connectors (CDM)

= Buttons, jumpers and switches =

Buttons:

* PB1 - HRST - reboot FPGA (power-on reset)
* PB2 - SRST - (SRSTn) - reboot ARM CPU

Switches:

* SW1 - boot mode BM0, BM1 [-->]
* SW2 - LEMO output NIM<->TTL
* SW3 - LEMO input 1 and 2 NIM/TTL
* SW4 - LEMO input 2 and 4 NIM/TTL
* SW5 - LEMO clock input NIM/TTL
* SW6 - serial console select. [PS<--PL] PS is ARM CPU, PL is FPGA.

= Front panel =

<pre>
| top
|
| LED-FP1 | LED_FP(0,1,2,3)
|
| SFP J???
|
| LEMO J4-LEFT, J4-RIGHT | EXT_IN_LV(1), EXT_IN_LV(2)
| LEMO J5-LEFT, J5-RIGHT | EXT_IN_LV(3), EXT_IN_LV(4)
| LEMO J6-LEFT, J6-RIGHT | CLK_EXT1, CLK_EXT0 (125 MHz only)
| LEMO J7-LEFT, J7-RIGHT | EXT_OUT(1), EXT_OUT(2)
|
| J-VX-1
| J-VX-2 or QSFP-1
| J-VX-3 or QSFP-2
| J-VX-4 or QSFP-3
| J-VX-5 or QSFP-4
| J-VX-6
|
| RJ45 J3 ethernet
|
| bottom
</pre>

= VX adapter board =

LVDS I/O connector
<pre>
34 pin connector: 0|:::: :::: :::: :::: :|15,16 n/c
</pre>

== split-cable connection ==

<pre>
0 .. 7 -> N/C

8 -> VX_RX(3) - not used
9 -> VX_RX(2) - busy VX to CDM
10 -> VX_RX(1) - DS20K 125 MHz serial data VX to CDM
11 -> VX_RX(0) - DS20K 62.5MHz clock VX to CDM

12 <- VX_TX(0) - TRG CDM to VX
13 <- VX_TX(1) - TSM CDM to VX, to be VETO CDM to VX
14 <- VX_TX(2) (set by jumper routed here or to VX CLKIN SYNC) - 125 MHz serial data CDM to VX
15 <- n/c (CLK routed to VX CLKIN CLK) - 62.5 MHz clock CDM to VX

16 - n/c
</pre>

== one-to-one connection ==

<pre>
0 -> VX2_RX(3)
1 -> VX2_RX(2)
2 -> VX2_RX(0)
3 -> VX2_RX(1)

4 <- CLK
5 <- VX2_TX(0)
6 <- VX2_TX(1)
7 <- VX2_TX(2)

8 -> VX1_RX(3)
9 -> VX1_RX(2)
10 -> VX1_RX(1)
11 -> VX1_RX(0)

12 <- VX1_TX(0)
13 <- VX1_TX(1)
14 <- VX1_TX(2) (set by jumper routed here or to VX CLKIN SYNC)
15 <- n/c (CLK routed to VX CLKIN CLK)

16 - n/c
</pre>

= Board schematics =

* [[File:SCH-DS-xDM-Rev0.PDF]]
* note: FPGA pin annotations ("IO", "SCLK", "PCLK", etc) on the schematics are bogus, instead, trace them to the FPGA pins.
* note: ENC A is J800, ENC B is J801, ENC C is J900 (schematic name to enclustra name)
* note: Enclustra special pins: "GC" is "clock capable", "HDGC" is "clock capable", "MGTREFCLK" is MGT reference clocks.
* board modifications:
** ethernet mac chip
** NIM output (no U15, etc)
** RJ45 wrong pinout (board mod or special ethernet cable)
** 125 MHz clock mods (TBW)
** disconnect QSFP0_SEL from SFP_RS0 and QSFP1_SEL from SFP_RS1, these signals are not used by modern SFPs
** provide SFP i2c modsel to allow SFP and QSFP at the same time on address 0x50

= FPGA MGT blocks =

<pre>
* SFP - ENC B45,B47 and B48,B50 - FPGA D5,D6 MGTHTX0_D and D1,D2 MGTHRX0_D
* QSFP0 TX0 - ENC C13,C17 - FPGA H5,H6 MGTHTX0_C
* QSFP0 TX1 - ENC C21,C25 - FPGA G7,G8 MGTHTX1_C
* QSFP0 TX2 - ENC C29,C23 - FPGA F5,F6 MGTHTX2_C
* QSFP1 TX0 - ENC C37,C41 - FPGA E7,E8 MGTHTX3_C
* QSFP1 TX1 - ENC C45,C47 - FPGA P5,P6 MGTHTX0_B
* QSFP1 TX2 - ENC C51,C53 - FPGA M5,M6 MGTHTX1_B
* QSFP2 TX0 - ENC C57,C59 - FPGA L3,L4 MGTHRX2_B
* QSFP2 TX1 - ENC C63,C65 - FPGA K5,K6 MGTHTX3_B
* QSFP2 TX2 - ENC C75,C77 - FPGA W3,W4 MGTHTX0_A
* QSFP3 TX0 - ENC C79,C81 - FPGA V5,V6 MGTHTX1_A
* QSFP3 TX1 - ENC C85,C87 - FPGA T5,T6 MGTHRX2_A
* QSFP3 TX2 - ENC C89,C91 - FPGA R3,R4 MGTHTX3_A
</pre>

= Clock distribution =

Simplified:

* 125 MHz osc -> CLK_XO_125 -> MGTREFCLK0_A -> not used
* 125 MHz osc -> CLK3_XO_125 -> MGTREFCLK1_B -> SFP RX ref clock, QSFP RX and TX ref clock (this is not final design!)
* (disconnected) 125 MHz osc -> CLK2_XO_125 -> FPGA AG8,AH8 (GC)
* 125 MHz osc -> C.C. in1

* C.C. in0 <- CLK_EXT1 (10 MHz GPS clock)
* C.C. in1 <- 125 MHz osc
* (disconnected) C.C. in2 <- CLK_CC_IN <- MGTREFCLK0_D <- SFP RX clock (cannot be used because of uncontrollable phase)
* C.C. in2 <- CLK2_XO_125 <- FPGA AG8,AH8 (GC) <- SFP RX recovered 125 MHz clock
* C.C. in3 <- CLK_FB

* C.C. 125 MHz -> CLK_CC_OUT0 -> MGTREFCLK0_B -> QSFP RX and TX ref clock (final design)
* C.C. 125 MHz -> CLK_CC_OUT1 -> MGTREFCLK1_D -> SFP TX clock
* C.C. 62.5 MHz -> VX clock fanout

proposed changes:
* add C.C. 125 MHz -> new CLK_CC_OUT2 (old CLK2_XO_125) -> FPGA AG8,AH8 (GC)
* change C.C. in2 <- new CLK_CC_IN <- FPGA AK8,AK9 (non-GC)

Complete:

<pre>
125 MHz oscillator - U5 fan out -
q0 -> CLK_XO_125 -> ENC C72,C74 -> FPGA R7,R8 MGTREFCLK0_A (not used)
q1 -> U6 C.C. in1
q2 -> CLK3_XO_125 -> ENC C7,C9 -> FPGA J7,J8 MGTREFCLK1_B -> SFP RX reference clock, QSFP RX and TX reference clocks (not final design!)
q3 -> disconnected on the board, was CLK2_XO_125 -> ENC C151,C153 -> FPGA AG8,AH8 (GC)

U6 C.C (clock cleaner) -

in0 <- CLK_EXT1 (presumably GPS 10 MHz ref clock)
in1 <- 125 MHz oscillator via U5
in2 <- (was: CLK_CC_IN <- ENC B10,B12 <- FPGA D9,D10 MGTREFCLK0_D <- SFP RX clock, 125 MHz)
in2 <- CLK2_XO_125 <- ENC C151,B153 <- FPGA AG8,AH8 (GC) <- mgt_link_data_to_processing.rx_data_clk (SFP RX data clock, 125 MHz)
in3 <- CLK_FB

out0 -> CLK_CCA -> U12 (125 MHz)
out1 -> CLK_CCB -> VX1..6 (62.5 MHz)
out2 -> CLK_CCC -> VX7..12 (62.5 MHz)
out3 -> CLK_FB into in3

CLK_CCA -> U12 (125 MHz fan out) ->

Q0 -> not used
Q1 -> CLK_CC_OUT0 -> ENC C3-5 -> FPGA L7,L8 MGTREFCLK0_B -> QSFP RX and TX reference clocks (final design)
Q2 -> CLK_CC_OUT1 -> ENC B3-5 -> FPGA B9,B10 MGTREFCLK1_D -> SFP TX clock
Q3 -> not used
Q4 -> not used
Q5 -> SMA J9/J10

CLK_TP0 - FPGA N7,N8 MGTREFCLK1_A - ENC C69,C71 - DS-DM SMA J11, J12 (NOT IN CDM PROJECT)

Notes:
* CLK_XO_125 (125 MHz osc) is not used
* 62.5 MHz VX clock does not go into the FPGA
* CLK_EXT0 going to in1 of U5 cannot be used. only permitted frequency is 125 MHz (it drives the MGT reference clocks) and is this frequency is too high for LEMO cables and connectors

Proposed modifications:
- CLK_CC_IN: from FPGA output pin (ENC C142,C144 - FPGA AK8,AK9) to clock cleaner in2 (instead of CLK2_XO_125 pins)
- CLK_CCA -> U12 -> currently unused out3 -> CLK2_XO_125 FPGA pins
- repurpose CLK_EXT0 at GPS 1pps/IRIG input to FPGA
</pre>

= I2C bus =

* I2C_SCL is J-ENC-A pin 111 I2C_SCL is FPGA I2C_SCL_PL AB13 (IO_L1N_TOL1D_64) and I2C_SCL_PS F18 (PS_MIO10)
* I2C_SDA is J-ENC-A pin 113 I2C_SDA is FPGA I2C_SDA_PL AH13 (IO_L7N_T1L1Q_AD13N_64) and I2C_SDA_PS G18 (PS_MIO11)

* XU8 secure EEPROM ATSHA204A at 0x64, this is 0110'010X -> linux _011'0010 is 0x32. (but responds to scan and read at 0x33)
* U4 ethernet mac chip, EEPROM at 1010 A2 A1 A0 X and MAC/serial_no at 1011 A2 A1 A0 X. A0=VCC, A1=VCC, A2=GND -> linux _101'0011 and _101'1011 is 0x53 and 0x5B.
* U6 clock chip, address 1101 0 A1 A0 X. A1=VCC, A0=N/C (internal pull-up) -> linux _110'1011 is 0x6b
* U23 voltmeter at 10011 ADR1 ADR0. ADR0=GND, ADR1=VCC -> linux _100'1110 is 0x4e
* SFP, address 1010000X -> linux _101'0000 is 0x50. additional SFP data at 0x51
* QSFP0, QSFP1, QSFP2, QSFP3 (QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL)

= I2C clock builder connection =

* use Silicon Labs USB "Clock builder pro field programmer", www.silabs.com/CBProgrammer
* connect rainbow jumper cable pins:
** black - 1-GND to GND on DS-DM
** white - 3-SCLK to SCL on the DS-DM
** grey - 7-SDA_SDIO to SDA on the DS-DM
* power up the DS-DM
* plug USB programmer into Windows laptop
* on Windows, run "ClockBuilder Pro"
* it should report "Field programmer detected", press "EVB GUI"
* in EVB GUI, press "Config", set I2C address 0x6B
* press "Scan", it should find Si5394A-A-GM
* select the "Status" tab, should see real-time status of clock chip

= GDM MGT configuration =

* TX configuration:
* GDM MGT transceivers are configured as "multilane" TX and RX.
* there is 12 TXes ("lanes")
* MGT reference 125 MHz clock goes into [2:0]gtreclk00_in and [11:0]gtrefclk0_in
* one MGT is designated as "master"
* PLL of master MGT converts reference clock into common TX clock and common tx_user_clk2 which becomes tx_data_clk
* common tx_user_clk2 aka tx_data_clk goes into all TXes and clocks tx_data.
* tx_user_clk2 aka tx_data_clk is 125 MHz but not same phase as MGT reference clock.

* RX interim configuration:
* there is 12 RXes ("lanes")
* each RX produces it's own recovered RX clock
* "multilane" configuration assumes all RX recovered clocks run at the same frequency (TX on the other end are driven by common TX clock, see above), but have different phase
* one RX recovered clock is designated as "master" (rx_user_clk2 aka rx_data_clk) and a phase-matching fifo/buffer is used to bring rx_data from all 12 RXes to this common rx_data_clk
* this works because each CDM SFP TX runs on the SFP RX recovered clock which is frequency-locked with the GDM QSFP TX clock.

* RX final configuration:
* MGTs permit using the common TX clock (tx_user_clk2 aka tx_data_clk) as the common rx_data_clk (they are frequency locked through the CDM).
* this permits use of tx_data_clk as the main clock domain in the GDM and removes the need to bring rx_data into the tx_data_clk domain (actually this is done in the MGT RX phase matching fifo/buffer).

= Clock path =

<pre>
10 MHz ext clock or GDM 125 MHz oscillator
-> GDM QSFP MGT reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz (GDM main clock domain) and TX bit clock 2.5 GHz
-> GDM QSFP optic transmitter
-> CDM SFP optic receiver
-> CDM SFP MGT, RX reference clock is CDM 125 MHz oscillator
-> MGT RX recovered clock 125 MHz (CDM main clock domain)
-> CC_CLK_IN -> CDM C.C. -> CC_CLK_OUT1 -> CDM SFP TX reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz and TX bit clock 2.5 GHz
-> (tx_data phase matching fifo from CDM main clock domain to tx_data_clk)
-> CDM SFP optic transmitter
-> GDM QSFP RX optic receiver (12x)
-> GDM QSFP MGT (RX reference clock is same as TX reference clock)
-> MGT RX recovered clock (12x recovered clocks)
-> in multi-lane configuration, one of them is the "master" recovered clock rx_data_clk
-> (rx_data phase matching fifo from rx_data_clk to GDM main clock domain)
</pre>

= CDM rx_clk mux =

when CDM SFP is not connected, there is no SFP recovered clock and a mux is used to switch between clk_cc_out0 (power up default) and rx_clk_mgt (SFP recovered clock)

== Test SFP disconnected ==

note: if I say "--cc-in1", CC seems to lock on the 10 MHz GPS external clock,
to prevent this, test sequence includes reloading the CC and the reset of MGT.

<pre>
./test_cdm_local.exe --write32 0x30 0x0 ### rx_clk mux select CC clock
./test_cdm_local.exe --load-cc
./test_cdm_local.exe --reset-mgt
./test_cdm_local.exe --cdm-clocks
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773594e (125000014) should be ~125 MHz <=== all 4 clocks wobble close to 125 MHz
0x1034 rx_clk: 0x0773594f (125000015) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x0773594f (125000015) should be ~125 MHz
0x103C tx_clk: 0x0773594f (125000015) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
./test_cdm_local.exe --write32 0x30 0x1 ### rx_clk mux select SFP recovered clock
./test_cdm_local.exe --cdm-clocks
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773598b (125000075) should be ~125 MHz
0x1034 rx_clk: 0x076d58ec (124606700) should be ~125 MHz <=== off frequency because there is no valid SFP recovered clock
0x1038 mgt_tx_ref_clk_raw: 0x0773598c (125000076) should be ~125 MHz
0x103C tx_clk: 0x0773598b (125000075) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

= Clock domains =

== GPS ==

* no GPS : GDM runs from internal 125 MHz oscillator
* external 10 MHz clock : GDM runs from external 10 MHz clock and optional 1pps signal (use VME-NIMIO32 NIM outputs)
* GPS receiver : GDM runs from GPS 10 MHz clock and GPS IRIG serial data
* LNGS GPS:
** provides 1pps and serial data over fiber from GPS receiver (master). LNGS xxx box is not used.
** serial data goes to GDM, decoded, 1pps signal extracted, goes to LEMO output
** 1pps from GDM is used to train the Rubidium clock which provides a 10 MHz clock
** 10 MHz output from Rubidium clock goes to GDM 10 MHz external clock input

== GDM ==

* AXI clock (100 MHz) - AXI registers
* 125 MHz oscillator - to clock cleaner
* 10 MHz external clock LEMO input - to clock cleaner
* FPGA 125 MHz clock CLK_CC_IN - to clock cleaner - not used (could be used for cascaded GDMs)
* 125 MHz oscillator - CLK3_XO_125 - mgt_b_ref_clk QSFP MGT reference clock (interim GDM design)
* clock cleaner output 125 MHz fanout:
** CLK_CC_OUT0 - QSFP MGT reference clock (final design)
** CLK_CC_OUT1 - not used (CDM SFP reference clock)
** CLK_CC_OUT2 - not used
* QSFP MGT TX data clock 125 MHz tx_data_clk (main clock domain)
** QSFP TX data
** QSFP RX data (in final design, rx_data_clk is same as tx_data_clk. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk == tx_data_clk)
** ds20k block
* (in interim design, rx_data_clk is disconnected from tx_data_clk. rx_data_clk is the "multilane master clock" which is one of the 12x RX recovered clocks. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk)

note: all these clocks are frequency locked to 125 MHz

== CDM ==

* AXI clock (100 MHz) - AXI registers
* 10 MHz external clock LEMO input - to clock cleaner (not used)
* 125 MHz oscillator to fanout
** to clock cleaner
** to SFP MGT RX reference clock: CLK3_XO_125 to mgt_rx_ref_clk to gtrefclk01_in
* SFP MGT RX recovered clock 125 MHz
** MGT PLL to MGT rx_user_clk2 aka rx_data_clk
** MGT rxrecclkout_out to mgt_rx_rec_clk to CLK_CC_IN to C.C.
* SFP MGT rx_user_clk2 aka rx_data_clk (250 MHz/8 bit, 125 MHz/16 bit, 62.5 MHz/32 bit data) (main clock domain)
** SFP RX data
** ds20k block
** VX TX clock PLLs
** VX RX clock PLLs
* C.C. fan out
** 62.5 MHz VX clocks (12x)
** CLK_CC_OUT0 (not used, GDM QSFP MGT reference clock)
** CLK_CC_OUT1 125 MHz to mgt_tx_ref_clk to gtrefclk00_in to SFP MGT TX reference clock
** CLK_CC_OUT2 (not used)
* SFP MGT tx_user_clk2 aka tx_data_clk
** SFP TX data
** TX data phase matching fifo from main clock domain to tx_data_clk
* VX TX clock PLLs
** 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
** VX TX data phase matching from main clock domain to VX TX clock (12 total)
** VX TX serializer
** VX TX LVDS transmitter
* VX RX clock PLLs
** 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
** VX RX LVDS receivers (12 total)
** VX RX deserializers (12 total)
** VX TX data phase matching from VX RX clock to main clock domain

== VX ==

* everything runs on the VX main 125 MHz clock
* correct phase of VX to CDM LVDS data is adjusted by scan of CDM VX RX PLL clock phase (VX to CDM link is now established)
* correct phase of CDM to VX LVDS data is adjusted by scan of CDM VX RX PLL clock phase (link is established after VX to CDM idle data pattern changes from "VX RX data bad" to "good".
* after good phases are found by scan, they are not expected to change unless cables are changed, CDM and VX modules are changed or CDM firmware is rebuilt. (rebuild of VX firmware should not affect LVDS data phase).
* if there is excessive link errors, phase scan must be repeated.

= Board test plan =

To test:
* Enclustra FPGA board
* SFP port - SFP_RS0/RS1 connected to QSFP0_SEL/1_SEL is wrong? our Finisar SFP says RS0, RS1 N/C, so probably okey.
* SW5 CLK_EXT1 NIM works. TTL needs to be tested. flipping SW5 CLK_EXT0 side from NIM to TTL makes CLK_EXT1 go LOS and OOF in the clock chip. R23 and R53 should be removed?
* BOOT_MODE 0 and 1

Partial:
* U23 3.3V current meter and thermometer. V1,V2 is current monitor, same as in application note. V3,V4 is thermometer, same in application note, except capacitor C118 is 0.1u instead of 470pF. Tested ok: Tint, VCC, V1, V2, V1-V2. Test failed: TR2 reads 50-something degC instead of same as Tint.

Done:
* LED_FP1A..D: tested ok. K.O. 15 sep 2022
* USB UART: tested ok. K.O. 15 sep 2022
* J4A, J4B, J5A, J5B LEMO inputs (NIM/TTL) EXT_IN_LV(1..4). TTL threshold 1.7V, NIM threshold -0.3V. 50 Ohm termination. TTL straight, NIM inverted.
* J6A, J6B LEMO clock inputs: 10 MHz TTL from chronobox works, 10 MHz NIM from IO32 works. 18-apr-2023
* J7A, J7B LEMO outputs EXT_OUT(1), EXT_OUT(2) (NIM/TTL) (tested 23nov2022, K.O.). TTL ???, NIM inverted.
** TTL out no 50 ohm termination: 0=0V, 1=5V, rise and fall time ~5 ns
** TTL out with 50 ohm termination: 0=0V, 1=2.5V, rise and fall time <2ns
** NIM out no 50 ohm termination: 0=+50mV, 1=-1.8V, rise and fall time ~3ns
** NIM out with 50 ohm termination: 0=0V, 1=-0.9V, rise and fall time <2ns
* ethernet MAC i2c chip (K.O. 20sep2022: can read: i2cdump 0 0x5b, nov2022 have u-boot driver)
* CDM VX ports 2x(CLK, 3 tx, 4 rx) tested using test_cdm.exe and LVDS loopback in VX firmware.
* SFP i2c tested KO 22jun2023
* QSFP i2c tested KO 22jun2023
* i2c testing complete 22jun2023
* QSFP rx,tx tested 26june2023. lane0,1,2 ok, lane3 tx not connected, laser is off. qsfp0,1,2,3 all lanes ok.
* SFP rx,tx tested 26june2023. rx and tx okey. LOS ok, mod_absent ok.

Failure:
* ethernet: does not connect to alliedtelesys switch. connects to my USB-eth adapter at 100 Mbit speed. uboot mii status reports connection speed oscillating between 1000, 100 and 10. K.O. 16-sep-2022
* SFP LOS and mod_absent are swapped (in the FPGA pin definitions?)
* QSFP slot numbering is wrong.

= Checklist for newly build boards =

* put new board on workbench
* check - vme connector present, vme extraction handles present
* check - standoff are removed from all thru-holes
* plug Enclustra module
* check - SW6 both switches are in the "PS" position
* connect micro-usb cable from linux PC
* connect ethernet from 1gige capable network switch
* connect power from lab power supply - GND, +5V and -12V nominal, +5.7V and -12.5V actual
* power up, +5V current 2.10-2.8A, -12V current 0.05A
* on linux PC, open a new terminal, run: minicom -D /dev/ttyACM0
* in minicom window, observe messages about Xilinx first stage boot loader, etc
* on the ethernet switch, observe network link is 1gige speed (not 10mbit, not 100mbit).
* if everything boots okey, there will be a login prompt, login as root, password root.
* busybox devmem 0x80010000 # read firmware version number, i.e. 0xEDAD0A77
* TBW - test LEMO inputs
* TBW - test LEMO outputs
* TBW - test VX connectors
* TBW - test SFP connector
* TBW - test QSFP connector

= Serial console =

* check that linux computer has correct udev rules to allow access to /dev/ttyACM devices, see https://daq00.triumf.ca/DaqWiki/index.php/SLinstall#Configure_USB_device_permissions and https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Configure_USB_device_permissions
* connect micro-USB cable to connector J-UCB, other end connect to linux computer
* observe /dev/ttyACM0 was created
* run "minicom -D /dev/ttyACM0" (default serial settings are ok, otherwise, 115200n8)
* should have gdm-cdm login
* username root, password root

= i2c =

<pre>
ZynqMP> i2c bus
Bus 0: i2c@ff020000
ZynqMP> i2c dev 0
Setting bus to 0
ZynqMP> i2c probe
Valid chip addresses: 33 4E 53 5B 6B 77
ZynqMP> i2c md 0x5b 0x98
0098: fc c2 3d 00 00 12 1a 6e 0a 90 85 04 94 10 08 50 ..=....n.......P
</pre>

<pre>
root@gdm-cdm:~# i2cdetect 0
Warning: Can't use SMBus Quick Write command, will skip some addresses
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0.
I will probe address range 0x03-0x77.
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f
00:
10:
20:
30: -- -- -- 33 -- -- -- --
40:
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60:
70:
root@gdm-cdm:~#
root@gdm-cdm:~# i2cdump 0 0x5b
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0, address 0x5b, mode byte
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00 ???????P?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e ........??=..??n
a0: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00 ???????P?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e ........??=..??n
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@gdm-cdm:~#
</pre>

<pre>
root@gdm0:~# i2cdetect -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#
</pre>

<pre>
root@cdm0:~# i2cdetect -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: 50 51 -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@cdm0:~#
</pre>

* 0x33 - XU8 secure EEPROM (should be at 0x32)
* 0x4e - U23 current and temperature monitor
* 0x50, 0x51 - SFP
* 0x50 - QSFP, 4 QSFP modules enabled by GPIO QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL
* 0x53, 0x5b - ethernet mac eeprom
* 0x6b - U6 clock chip

== U23 ==

NOTE: Text reads double of Tint. not sure why. K.O. 21-mar-2024.

* internal temperature only

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x00 b # control register: "repeat mode, internal temperature only"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x03 # "Tint ready" and "busy", "busy is always 1 in repeat mode"
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 00 03 03 81 db 2a ce 2a 8e 00 6f 00 45 20 3f ?.????*?*?.o.E ?
10: 01 00 01 01 01 db 2a ce 2a 8e 00 6f 00 45 20 3f ?.????*?*?.o.E ?
...
readback:
reg0 - 03 - Tint ready
reg1 - 00 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB 0x81, bit 0x80 is "DV, data valid", bit 0x40 is "SS, sensor short", 0x20 is "SO, sensor open"
reg5 - Tint LSB 0xDB, Tint = 0x01DB = 475 * 0.0625 degC = 29.6 degC
reg6..F - stale data
</pre>

* Tint, V1, V2, TR2, VCC

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x18 b # control register: "repeat mode, V1, V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 7f 18 7f 7f 81 b9 aa d0 aa 8f 83 5e 83 5e a0 41 ???????????^?^?A
10: 01 18 01 01 01 b9 2a d0 2a 8f 03 5e 03 5e 20 41 ??????*?*??^?^ A
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x1b9*0.0625 = 27.5 degC
reg6 - V1 MSB 0xaa, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 LSB 0xd0, V1 = 0x2ad0*305.18/1000000 = 3.3447 V (correct)
reg8 - V2 MSB 0xaa, ditto
reg9 - V2 LSB 0x8f, V2 = 0x2a8f*305.18/1000000 = 3.3249 V (correct, smaller than V1)
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x5e. TR2 = 0x35e*0.0625 = 53.875 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x41, VCC = 2.5+0x2041*305.18/1000000 = 5.019 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.99 A (about right?)
</pre>

* Tint, V1-V2, TR2, VCC

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x19 b # control register: "repeat mode, V1-V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 7f 19 7f 7f 81 69 83 f2 83 f2 83 3a 83 3a a0 44 ?????i?????:?:?D
10: 01 19 01 01 01 69 03 f2 03 f2 03 3a 03 3a 20 44 ?????i?????:?: D
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x169*0.0625 = 22.5 degC
reg6 - V1 or V1-V2 MSB 0x83, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 or V1-V2 LSB 0xf2, V1-V2 = 0x3f2*19.42/1000000 = 0.0196142 V (correct, compare with V1 and V2 measured above)
reg8 - V2 or V1-V2 MSB
reg9 - V2 or V1-V2 LSB
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x3a. TR2 = 0x33a*0.0625 = 51.625 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x44, VCC = 2.5+0x2044*305.18/1000000 = 5.021 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.9807 A (about right?)
</pre>

== SFP ==

<pre>
root@cdm0:~# i2cdump 0 0x50
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 04 07 00 00 00 00 40 40 0c 00 01 3d 00 00 00 ???....@@?.?=...
10: 0c 02 00 1e 46 49 4e 49 53 41 52 20 43 4f 52 50 ??.?FINISAR CORP
20: 2e 20 20 20 00 00 90 65 46 54 4c 46 38 35 32 36 . ..?eFTLF8526
30: 50 33 42 4e 4c 20 20 20 41 20 20 20 03 52 00 9d P3BNL A ?R.?
40: 00 1a 00 00 4e 33 41 42 34 4c 56 20 20 20 20 20 .?..N3AB4LV
50: 20 20 20 20 32 30 30 33 31 39 20 20 68 f0 03 de 200319 h???
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@cdm0:~#
root@cdm0:~# i2cdump 0 0x51
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 5a 00 d8 00 55 00 e2 00 90 88 71 48 8c a0 75 30 Z.?.U.?.??qH??u0
10: 21 34 01 f4 1b 58 03 e8 31 2d 04 eb 1f 07 06 31 !4???X??1-?????1
20: 31 2d 00 64 27 10 00 9e 00 00 00 00 00 00 00 00 1-.d'?.?........
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
40: 00 00 00 00 3f 80 00 00 00 00 00 00 01 00 00 00 ....??......?...
50: 01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 b7 ?...?...?......?
60: 19 9f 80 c5 0e 17 12 c4 1f 99 00 00 00 00 30 00 ??????????....0.
70: 00 00 00 00 00 00 00 00 ff ff ff ff ff ff ff 01 ...............?
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@cdm0:~#
</pre>

== QSFP ==

<pre>
QSFP i2c enable lines, active low:

QSFP0_SEL - JENC-A 82 - PS-MIO40 - linux gpio 378
QSFP1_SEL - JENC-A 84 - PS-MIO41 - linux gpio 379
QSFP2_SEL - JENC-A 100 - PS-MIO44 - linux gpio 382
QSFP3_SEL - JENC-A 106 - PS-MIO43 - linux gpio 381 (notice 2 and 3 are out of order)
</pre>

<pre>
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
# echo 378 >> /sys/class/gpio/export ### SEL0 338+40
# echo 379 >> /sys/class/gpio/export ### SEL1 338+41
# echo 381 >> /sys/class/gpio/export ### SEL3 338+43
# echo 382 >> /sys/class/gpio/export ### SEL2 338+44
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) in hi
gpio-379 ( |sysfs ) in hi
gpio-381 ( |sysfs ) in hi
gpio-382 ( |sysfs ) in hi
root@gdm0:~# echo out >> /sys/class/gpio/gpio381/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio382/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio378/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio379/direction
root@gdm0:~#
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out lo
gpio-379 ( |sysfs ) out lo
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out lo
root@gdm0:~#
echo 1 >> /sys/class/gpio/gpio381/value
echo 1 >> /sys/class/gpio/gpio382/value
echo 1 >> /sys/class/gpio/gpio378/value
echo 1 >> /sys/class/gpio/gpio379/value
cat /sys/kernel/debug/gpio
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out hi
gpio-382 ( |sysfs ) out hi
root@gdm0:~#
root@gdm0:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#

NOTICE NOTHING AT ADDRESS 0x50
</pre>

<pre>
ENABLE QSFP0, OBSERVE IT IS AT ADDRESS 0x50

root@gdm0:~# echo 0 >> /sys/class/gpio/gpio378/value
root@gdm0:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: 50 -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#
root@gdm0:~# i2cdump -y 0 0x50
No size specified (using byte-data access)
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 0d 00 02 8e 00 00 01 00 00 05 55 00 05 00 00 00 ?.??..?..?U.?...
10: 00 00 00 00 00 00 1e b8 00 00 81 a0 00 00 00 00 ......??..??....
20: 00 00 12 12 00 01 00 01 00 01 0e e0 0f 20 0e e0 ..??.?.?.???? ??
30: 00 00 1f a0 1e 3b 1f 72 00 01 00 00 00 00 00 00 ..???;?r.?......
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
60: 00 00 00 00 00 00 00 00 00 00 1f 00 00 00 08 00 ..........?...?.
70: 00 00 00 00 00 00 00 ff ff ff ff ff ff ff ff 00 ................
80: 0d 00 0c 04 00 00 00 40 40 02 d5 05 67 00 00 96 ?.??...@@???g..?
90: 00 00 c8 00 46 49 4e 49 53 41 52 20 43 4f 52 50 ..?.FINISAR CORP
a0: 20 20 20 20 07 00 90 65 46 54 4c 34 31 30 51 44 ?.?eFTL410QD
b0: 34 43 20 20 20 20 20 20 41 20 42 68 07 d0 00 43 4C A Bh??.C
c0: 00 07 0f de 58 37 39 41 43 30 52 20 20 20 20 20 .???X79AC0R
d0: 20 20 20 20 32 32 30 33 30 39 20 20 3c 10 00 9e 220309 <?.?
e0: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@gdm0:~#
</pre>

== ethernet mac eeprom ==

* correct chip with 84-bit ethernet mac address

<pre>
root@cdm1:~# i2cdump 0 0x53
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
root@cdm1:~#

root@cdm1:~# i2cdump 0 0x5b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00 ????4??2?K?.?...
90: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c ..........??=?Q<
a0: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00 ????4??2?K?.?...
b0: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c ..........??=?Q<
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@cdm1:~#
</pre>

* wrong "602" chip with 64-bit IPv6 address

<pre>
root@cdm0:~# i2cdump 0 0x53
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
root@cdm0:~# i2cdump 0 0x5b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00 ???????Q?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e ........??=..??.
a0: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00 ???????Q?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e ........??=..??.
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@cdm0:~#
</pre>

= read ethernet mac address from i2c =

(this code is copied from uboot command line i2c code)

in uboot sources board/xilinx/common/board.c replace original function with this:

<pre>
// special code to read ethernet MAC address from the DS-DM-Rev0 board. K.O. Sep-2022

int zynq_board_read_rom_ethaddr(unsigned char *ethaddr)
{
struct udevice *bus;
int ret;
int busnum = 0;

ret = uclass_get_device_by_seq(UCLASS_I2C, busnum, &bus);
if (ret) {
printf("%s: No bus %d\n", __func__, busnum);
return ret;
}

int chip_addr = 0x5B;

struct udevice *dev;

ret = i2c_get_chip(bus, chip_addr, 1, &dev);
if (ret) {
printf("%s: Bus %d no chip 0x%02x\n", __func__, busnum, chip_addr);
return ret;
}

int dev_addr = 0x98;

unsigned char data[8];

ret = dm_i2c_read(dev, dev_addr, data, 8);

if (ret) {
printf("%s: Bus %d chip 0x%02x read error %d\n", __func__, busnum, chip_addr, ret);
return ret;
}

printf("%s: Bus %d chip 0x%02x addr 0x%02x read: 0x %02x %02x %02x %02x %02x %02x %02x %02x\n", __func__, busnum, chip_addr, dev_addr, data[0], data[1], data[2], data[3], data[4], data[5], data[6], data[7]);

// see Atmel-8807-SEEPROM-AT24MAC402-602-Datasheet.pdf

if (data[0] == 0) {
// eiu-48 chip
ethaddr[0] = data[2];
ethaddr[1] = data[3];
ethaddr[2] = data[4];
ethaddr[3] = data[5];
ethaddr[4] = data[6];
ethaddr[5] = data[7];
} else {
// eiu-64 chip
ethaddr[0] = data[0];
ethaddr[1] = data[1];
ethaddr[2] = data[2];
ethaddr[3] = data[5];
ethaddr[4] = data[6];
ethaddr[5] = data[7];
}

printf("%s: ethaddr %02x:%02x:%02x:%02x:%02x:%02x\n", __func__, ethaddr[0], ethaddr[1], ethaddr[2], ethaddr[3], ethaddr[4], ethaddr[5]);

return ret;
}
</pre>

also this should have worked if i2c_xxx() functions were enabled in uboot:
<pre>
i2c_set_bus_num(0);
i2c_probe(0x5b);
i2c_read(0x5b, 0x9a, ethaddr, 6);
</pre>

= read ethernet mac address from i2c (SHOULD WORK) =

from: https://stackoverflow.com/questions/43637540/is-there-linux-or-u-boot-support-to-read-a-mac-address-from-a-chip-at-startup

<pre>
#ethernet related setup
setup_eth=run readmac buildmac
#read mac address from eeprom
readmac=i2c dev 0; i2c read 50 FA.1 6 $loadaddr
#build the ethaddr variable
#not very nice, but does the job
buildmac=\
e=" "; sep=" " \
for i in 0 1 2 3 4 5 ; do\
setexpr x $loadaddr + $i\
setexpr.b b *$x\
e="$e$sep$b"\
sep=":"\
done &&\
setenv ethaddr $e
</pre>

= read ethernet mac address from i2c (DOES NOT WORK) =

this method does not work: inside board/xilinx/common/board.c:int zynq_board_read_rom_ethaddr(unsigned char *ethaddr), uclass_get_device_by_ofnode(UCLASS_I2C_EEPROM, eeprom, &dev) returns ENODEV and read from i2c does not happen. K.O. Sep-2022

Read:
* https://support.xilinx.com/s/question/0D52E00006hpKlsSAE/uboot-and-ethernet-mac-address-from-eeprom?language=en_US (last answer)
* https://github.com/Xilinx/u-boot-xlnx/blob/master/board/xilinx/common/board.c (zynq_board_read_rom_ethaddr())

Note:
* 0x5B is the i2c chip address
* 0x9A is the data offset inside the chip, see datasheet or i2c read dump above.

Edit:
* emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/u-boot/files/platform-top.h
<pre>
#include <configs/xilinx_zynqmp.h>
#include <configs/platform-auto.h>
//#define CONFIG_I2C_EEPROM
//#define CONFIG_SYS_I2C_EEPROM_ADDR 0x5b
//#define CONFIG_SYS_I2C_EEPROM_ADDR_OVERFLOW 0x0
#define CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET 0x9A
#error HERE!
</pre>
* emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi
<pre>
/include/ "system-conf.dtsi"
/ {
chosen {
xlnx,eeprom = &eeprom;
};
};

&i2c0 {
eeprom: eeprom@5b { /* u88 */
compatible = "atmel,24mac402";
reg = <0x5b>;
};
};
</pre>
* components/yocto/workspace/sources/u-boot-xlnx/configs/xilinx_zynqmp_virt_defconfig
<pre>
CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET=0x9A
</pre>

= enable VX clock =

<pre>
devmem 0x80010010 32 0x8; sleep 1; devmem 0x80010010 32 0x0;
si5394-i2c-file /media/sd-mmcblk1p1/00_freerun.txt 0 0x6b
</pre>

= clock chip configuration =

file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt:

* VCO is 14 GHz
* Tvco is 71.43 ps
* N0 divider is 14, frequency is 1000 MHz
* out0 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz
* out1 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
* out2 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
* out3 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz is the feedback for zero delay

= clock chip monitoring =

from si5395-94-92-family.pdf:
* reg 0x1: page select, set to 0 or set to 5 to read 0x53F
* reg 0x2: 0x94
* reg 0x3: 0x53 -> device is a si5394
* reg 0xC: LOSXAXB
* reg 0xD: LOS and OOF for the 4 clock inputs
* reg 0xE: LOL and HOLD
* reg 0xF: CAL_PLL
* reg 0x11: sticky bits for reg 0xC
* reg 0x12: sticky bits for reg 0xD
* reg 0x13: sticky bits for reg 0xE
* reg 0x14: sticky bits for reg 0xF
* reg 0x1C: device reset
* reg 0x1E: low power, hard reset, SYNC
* reg 0x507: currently selected input clock
* reg 0x52A: input clock select
* reg 0x535: FORCE_HOLD
* reg 0x53F: HOLD_HIST_VALID and FASTLOCK_STATUS

= arm and linux benchmark =

memory benchmark:

<pre>
daq13$ arm-linux-gnueabi-gcc -o memcpy.armv7 memcpy.cc -march=armv7 -static -O2
scp memcpy.armv7 to ...
root@gdm-cdm:~# ./memcpy.armv7
memcpy 1 KiBytes: 1288 MB/sec
memcpy 2 KiBytes: 1924 MB/sec
memcpy 4 KiBytes: 2554 MB/sec
memcpy 8 KiBytes: 3054 MB/sec
memcpy 16 KiBytes: 3262 MB/sec
memcpy 32 KiBytes: 3250 MB/sec
memcpy 64 KiBytes: 3456 MB/sec
memcpy 128 KiBytes: 3556 MB/sec
memcpy 256 KiBytes: 3780 MB/sec
memcpy 512 KiBytes: 3795 MB/sec
memcpy 1024 KiBytes: 3789 MB/sec
memcpy 2048 KiBytes: 3729 MB/sec
memcpy 4096 KiBytes: 3717 MB/sec
memcpy 8192 KiBytes: 3687 MB/sec
memcpy 16384 KiBytes: 3632 MB/sec
memcpy 32768 KiBytes: 3529 MB/sec
memcpy 65536 KiBytes: 3318 MB/sec
memcpy 131072 KiBytes: 2893 MB/sec
root@gdm-cdm:~#
</pre>

ethernet receive:

<pre>
daq13:bin$ ./ttcp -t -s -n 100000 10.0.0.24
ttcp-t: buflen=8192, nbuf=100000, align=16384/0, port=5001 tcp -> 10.0.0.24
ttcp-t: socket
ttcp-t: connect
ttcp-t: 819200000 bytes in 7.25 real seconds = 110358.39 KB/sec +++
ttcp-t: 100000 I/O calls, msec/call = 0.07, calls/sec = 13794.80
ttcp-t: 0.0user 0.2sys 0:07real 3% 0i+0d 760maxrss 0+2pf 1461+31csw
daq13:bin$

root@gdm-cdm:~# ./ttcp.armv7 -r -s
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 10.0.0.25
ttcp-r: 819200000 bytes in 7.27 real seconds = 110098.22 KB/sec +++
ttcp-r: 212040 I/O calls, msec/call = 0.04, calls/sec = 29181.53
ttcp-r: 0.1user 5.7sys 0:07real 81% 0i+0d 584maxrss 0+2pf 125601+2699csw
root@gdm-cdm:~#
</pre>

ethernet transmit:

<pre>
root@gdm-cdm:~# ./ttcp.armv7 -t -s -n 100000 10.0.0.25
ttcp-t: buflen=8192, nbuf=100000, align=16384/0, port=5001 tcp -> 10.0.0.25
ttcp-t: socket
ttcp-t: connect
ttcp-t: 819200000 bytes in 6.95 real seconds = 115078.69 KB/sec +++
ttcp-t: 100000 I/O calls, msec/call = 0.07, calls/sec = 14384.84
ttcp-t: 0.0user 0.7sys 0:06real 11% 0i+0d 584maxrss 0+2pf 1162+1017csw
root@gdm-cdm:~#

daq13:bin$ ./ttcp -r -s
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 10.0.0.24
ttcp-r: 819200000 bytes in 6.97 real seconds = 114841.84 KB/sec +++
ttcp-r: 161335 I/O calls, msec/call = 0.04, calls/sec = 23160.01
ttcp-r: 0.0user 1.9sys 0:06real 28% 0i+0d 760maxrss 0+2pf 80646+51csw
daq13:bin$
</pre>

= Install Xilinx tools =

* install Vivado 2020.2
<pre>
login at https://www.xilinx.com/myprofile.html
go to "Downloads"
go to archive,
find 2020.2
download Xilinx_Unified_2020.2_1118_1232_Lin64.bin
sh ./Xilinx_Unified_2020.2_1118_1232_Lin64.bin
banner window should open with spinner "downloading installation data"
"a newer version is available" -> say "continue"
next
"select install type" window:
provide email and password,
select "download image"
select directory /home/olchansk/Xilinx/Downloads/2020.2\
select "linux" and "full image"
next
download summary: space required 38.52 Gbytes
download
installation progress
downloading spinner, 16 M/s 47 minutes...
"download image has been created successfully". Ok.
check contents of /home/olchansk/Xilinx/Downloads/2020.2
ls -l /home/olchansk/Xilinx/Downloads/2020.2
total 67
drwxr-xr-x 2 olchansk users 9 Sep 1 16:22 bin
drwxr-xr-x 3 olchansk users 15 Sep 1 16:23 data
drwxr-xr-x 4 olchansk users 4 Sep 1 16:22 lib
drwxr-xr-x 2 olchansk users 644 Sep 1 16:22 payload
drwxr-xr-x 2 olchansk users 7 Sep 1 16:22 scripts
drwxr-xr-x 4 olchansk users 4 Sep 1 16:22 tps
-rwxr-xr-x 1 olchansk users 3256 Nov 18 2020 xsetup
daq13:2020.2$
./xsetup
spinned loading installation data
xilinx design tools 2022.1 now available -> say continue
"welcome" -> next
"select product" -> vivado -> next -> vivado hl system edition -> next
select devices: only zynq ultrascale+ mpsoc -> next
select destination: /opt/Xilinx (as root, mkdir /opt/Xilinx, chmod olchansk.users /opt/Xilinx)
install ...
complete
move /home/olchansk/Xilinx/Downloads/2020.2 to /daq/daqstore/olchansk/Xilinx/Downloads/
</pre>
* install petalinux 2020.2
<pre>
./xsetup
"a newer version is available" -> say "continue"
next
"select product to install" -> select Petalinux (Linux only) -> next
"select destination directory" -> select "/opt/Xilinx" (disk space required 2.64 GB) -> next
"summary" -> install ...
error about missing /tmp/tmp-something files
"installation completed successfully" (hard to dismiss, "ok" button is partially cut-off)
done?
I think it failed, /opt/Xilinx/PetaLinux/2020.2/bin is empty except for petalinux-v2020.2-final-installer.run
try to run it by hand, same error about /tmp/tmp-something files. strange...
notice it complains about "truncate", which truncate finds ~/bin/truncate, get rid of it,
try again
now complains about missing texinfo and zlib1g:i386
apt install texinfo -> ok
apt install zlib1g:i386 -> installs bunch of gcc stuff -> ok
try again
reports "already installed" -> delete /opt/Xilinx/.xinstall/PetaLinux_2020.2/, delete entries in ~/.Xilinx/registry/installedSW.xml
try again
success
</pre>
* install vivado 2022.1 and petalinux 2022.1 - everything is pretty much the same

= Petalinux =

* cd PetaLinux_GDM_CDM
* petalinux-config
* enable i2c MAC address and DHCP
<pre>
git clone https://bitbucket.org:/team-ds-dm/ds-dm-u-boot-xlnx.git
cd ds-dm-u-boot-xlnx
git checkout ds-dm-u-boot-xlnx

linux-components ->
uboot -> ext-local-src
external u-boot local source -> ds-dm-u-boot-xlnx (path to the customized uboot git repository)
</pre>
* enable DHCP
<pre>
Subsystem AUTO Hardware Settings -> Ethernet Settings
randomize MAC address -> NO
ethernet mac address -> leave empty
obtain ip address automatically -> YES
</pre>
* set hostname and product names
<pre>
Firmware Version Configuration ->
Host name -> "ds-dm"
Product name -> "Petalinux_GDM_CDM"
</pre>
* configure linux kernel
<pre>
petalinux-config -c kernel
</pre>
* enable NFS-Root
<pre>
petalinux-config
Image Packaging Configuration > Root File System Type -> set to NFS
Location of NFS root directory set to "/nfsroot"

petalinux-config -c kernel
Networking support > IP: kernel level configuration
enable DHCP, BOOTP, RARP
File systems > Network file systems > Root file systems on NFS
</pre>
* manually fix linux kernel command line:
<pre>
grep nfsroot PetaLinux_GDM_CDM/project-spec/configs/config
edit CONFIG_SUBSYSTEM_BOOTARGS_GENERATED to read
earlycon console=ttyPS0,115200 clk_ignore_unused panic=60 root=/dev/nfs nfsroot=/nfsroot/%s ip=dhcp rw
</pre>
* check configuration in
** PetaLinux_GDM_CDM/project-spec/configs/config
** PetaLinux_GDM_CDM/project-spec/configs/rootfs_config
** PetaLinux_GDM_CDM/components/plnx_workspace/device-tree/device-tree/system-conf.dtsi

= JTAG server =

localhost:3121

= ds20k block =

<pre>
module ds20k
(
// CLOCK INPUTs
input wire clk,
input reset, // pulse for power-up reset
input wire pll_is_locked, // clock cleaner PLL is locked to selected input clock

// REGISTER_DATA

input wire [255:0] [31:0] register_data_in,
output reg [255:0] [31:0] register_data_out,
input wire register_write_strobe, // pulse when AXI write transaction puts new data in register_data_in
input wire register_read_ack, // pulse after AXI read transaction captures data from register_data_out, used to read from FIFO

// GDM QSFP FIBER LINKS
output reg [11:0] [15:0] qsfp_tx_data,
output reg [11:0] [1:0] qsfp_tx_ctrl,
input wire [11:0] [15:0] qsfp_rx_data_error, // not sure what this is.
input wire [11:0] [15:0] qsfp_rx_data,
input wire [11:0] [1:0] qsfp_rx_ctrl,
input wire [11:0] qsfp_rx_is_good, // single bit indicating that RX link is up and data is good.

// CDM SFP FIBER LINKS
output reg [15:0] sfp_tx_data,
output reg [1:0] sfp_tx_ctrl,
input wire [15:0] sfp_rx_data,
input wire [1:0] sfp_rx_ctrl,
input wire sfp_rx_is_good, // single bit indicating that RX link is up and data is good.

// VX_RXs
input wire [3:0] vx1_rx,
input wire [3:0] vx2_rx,
input wire [3:0] vx3_rx,
input wire [3:0] vx4_rx,
input wire [3:0] vx5_rx,
input wire [3:0] vx6_rx,
input wire [3:0] vx7_rx,
input wire [3:0] vx8_rx,
input wire [3:0] vx9_rx,
input wire [3:0] vx10_rx,
input wire [3:0] vx11_rx,
input wire [3:0] vx12_rx,

// VX_TXs
output reg [2:0] vx1_tx_out,
output reg [2:0] vx2_tx_out,
output reg [2:0] vx3_tx_out,
output reg [2:0] vx4_tx_out,
output reg [2:0] vx5_tx_out,
output reg [2:0] vx6_tx_out,
output reg [2:0] vx7_tx_out,
output reg [2:0] vx8_tx_out,
output reg [2:0] vx9_tx_out,
output reg [2:0] vx10_tx_out,
output reg [2:0] vx11_tx_out,
output reg [2:0] vx12_tx_out,

// remove input wire gdm_trg,
// remove input wire gdm_tsm,

// LEMO INPUTs
input wire [4:1] ext_in_lv_async, // direct connection to LEMO connectors, not clocked

// LEMO OUTPUTs
output reg [2:1] ext_out, // direct connection to LEMO connectors, not clocked

// FRONT PANEL LEDs
output reg [3:0] fp_led_out, // direct connection to LEMO connectors, not clocked

// trigger and tsm output
// remove output reg trg_out,
// remove output reg tsm_out
);
</pre>

= world view =

Note:
* red lines: clocks
* green lines: AXI/Avalon packet streams
* blue lines: serial data

[[File:GDM-CDM-VX.drawio.svg]]

== description ==

same thing, in words:

<pre>
detector
digitizer, 125 MHz
digital filter
digital discriminator
hit map, 64 bits at 125 MHz (could be 250 MHz, filter and discriminator clock)
packetizer, 64 bits -> id, timestamp, 8x 8-bit words, eop
8/10 serializer, 12.5 MHz parallel in, 125 MHz serial out
lvds line to CDM vx_rx
BBB: also VX busy to lvds line to CDM vx_rx, do not want to depend on serial comm for vx busy, do not want to inject async data into the hit map packet stream
--- CDM
lvds line capture in IBUF register, vxN_rx clock, 125 MHz clock, from PLL with adjustable phase, scan phase to find sweet spot
vx rx 10/8 deserializer, vxN_rx clock, stobes out parallel data every 10 clocks at 12.5 MHz
vx rx phase transfer from vxN_rx clock to main CDM clock, have 10 clocks for transfer to happen
from here everything is on the CDM main clock
vx rx packet adapter, 12.5 MHz 8-bit packets to 125 MHz 16-bit packets, internal FIFO to avoid data overrun
sfp tx mux, all VX packet streams into one CDM sfp tx packet stream
BBB: capture VX RX busy to CDM main clock (IBUF register), grand-or becomes vx_rx_busy
sfp tx packetizer (data,eop,vx_rx_busy -> data,k)
sfp tx 16/20 serializer, 16 bit at 125 MHz in, 2500 MHz serial out (20 bits at 125 MHz)
sfp tx fiber link to GDM, 2 Gigabits/sec
--- GDM
qsfp rx deserializer, 16 bit at 125 MHz output on main GDM clock
qsfp rx depacketizer (data,k -> packet data,eop; qsfp_rx_busy)
qsfp rx demux, hit map packets routed to GDM trigger logic block
GDM trigger logic block looks at hit map, generates yes/no trigger decision, encodes it as a trigger packet
TSM generator encodes GPS time data as a TSM packet
qsfp tx mux - trigger packets, tsm packets, etc to qsfp tx packet stream
qsfp tx packetizer (data,eop;trg,tsm,bsy -> data,k)
NB: the same tx data is sent to all 12 qsfp tx ports, to make sure we do not accidentally desync the CDMs.
qsfp tx 16/20 serializer, 16-bit at 125 MHz to 20-bit at 250 MHz to 2500 MHz serial out
qsfp tx fiber link to CDM, 2 Gigabits/sec
--- CDM
sfp rx 20/16 deserializer, 16-bit at 125 MHz
sfp rx depacketizer, (data,k -> data,eop;trg_in,tsm_in,bsy_in)
sfp rx demux (in reality, noop, all packets go to same place, vx tx)
vx tx mux (packets from GDM, packets with simulated waveforms from CDM midas frontend via AXI FIFO)
vx tx packet adapter 16-bit at 125 MHz to 8-bit at 12.5 MHz strobed every 10 clocks. NB: most important, this 10-clock strobe runs in sync between all CDMs!
vx tx phase transfer from main CDM clock to vxN_tx clock, have 10 clocks for transfer to happen
from here we run on the vxN_tx clock
vx txN serializer, 8-bit at 12.5 MHz to 125 MHz serial
vx_txN OBUF register, vxN_rx clock, 125 MHz clock, from PLL with adjustable phase, scan phase to find sweet spot
lvds line to vx
BBB: bsy_in from GDM is converted from pulse to level, goes out lvds line to vx, sync to vxN_tx clock
TTT: trg_in from GDM is a pulse, does out lvds line to vx, sync to vxN_tx clock.
NB: tsm is always a packet, bsy is always a signal (no packet), trg can be a packet or signal.
--- VX
lvds data captured by 125 MHz ADC sampling clock (CAEN base firmware logic)
lvds data connected to Yair's block
10/8 deserializer
depacketizer (data,k -> data,eop) to avoid accidental desync, we do not send any no trg, no tsm, no bsy K-codes.
demux
trigger packets go their way (4x 64 bit words of data go to event header: timestamps, hitmap data)
tsm packets go their way (4x 64 bit words of data go to tsm event header: timestamps, GPS time data)
BBB: bsy from lvds line stops waveform acquisition
TTT: trg from lvds line cause waveform acquisition, same as trg packet, but has no timestamp and other data attached to it.
</pre>

= Firmware registers =

== Block 0 ==

<pre>
0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
</pre>

== Block 1 ==

== Block 2 ==

== Block 3 ==

=== DS20k block register map ===

* busybox devmem 0x80013000 32

<pre>
reg | version | xDM | xx | description
0 | 0x20230731 | ALL | RO | ds20k version
0 | 0x20240118 | ALL | RW | ds20k version and command
1 | 0x20230731 | ALL | RW | scratch read/write register
2 | 0x20230731 | ALL | RW | configure inputs and outputs
3 | 0x20230731 | ALL | RW | FP_LED mux
4 | 0x20230731 | ALL | RW | EXT_OUT mux
5 | 0x20230731 | ALL | RO | VX_RX state
6 | 0x20230731 | ALL | RO | VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state
7 | 0x20230731 | ALL | RW | LED_OUT, EXT_OUT, VX_TX outputs
8 | 0x20230731 | ALL | RW | VX_TX mux and config
9 | 0x20230731 | ALL | RW | trigger config
10 | 0x20231013 | ALL | RO | status register
11 | 0x20230731 | ALL | RO | trigger counter
12 | 0x20230731 | ALL | RO | time slice marker counter
13 | 0x20240814 | ALL | RO | GPS 1pps period, 125 MHz
14 | 0x20240814 | ALL | RO | Ru clock 1pps period, 125 MHz
15 | 0x20230811 | CDM | RO | SFP RX status
16 | 0x20230811 | CDM | RW | SFP TX control
17 | 0x20230811 | GDM | RO | QSFP RX data 0, 1
18 | |. |. | 2, 3
19 | |. |. | 4, 5
20 | |. |. | 6, 7
21 | | |. | 8, 9
22 | |. |. | 10, 11
23 | 0x20230811 | GDM | RW | QSFP TX control
24 | 0x20231013 | ALL | RW | trigger pulser period
25 | 0x20231013 | ALL | RW | trigger pulser burst control
26 | 0x20231013 | ALL | RW | tsm pulser period
27 | 0x20231204 | ALL | RW | data fifo CPU to FPGA
28 | 0x20231204 | ALL | RW | packet loopback control
28 | 0x20240118 | ALL | RW | data fifo FPGA to CPU
29 | 0x20231208 | GDM | RW | bitmap of active qsfp ports
29 | 0x20240118 | ALL | RW | packet loopback control
29 | 0x20240510 | ALL | RW | packet routing
30 | 0x20231208 | GDM | RO | qsfp link status ports 0..7
31 | 0x20231208 | GDM | RO | qsfp link status ports 8..11
32 | 0x20240118 | GDM | RW | enabled QSFP ports
33 | 0x20240118 | CDM | RW | enabled VX ports
34,35 | 0x20240118 | ALL | RO | time stamp 64 bits
36,37 | 0x20240118 | ALL | RO | old time stamp
38,39,40 | 0x20240118 | CDM | RO | VX busy counters
41,42,43 | 0x20240118 | GDM | RO | QSFP busy counters
44 | 0x20240118 | CDM | RO | cdm_bsy_up_counter and cdm_bsy_pulse_counter
45 | 0x20240118 | CDM | RO | cdm_veto_pulse_counter and cdm_veto_up_counter
46 | 0x20240118 | GDM | RO | gdm_bsy_pulse_counter and gdm_bsy_up_counter
47 | 0x20240118 | GDM | RO | gdm_bsy_refresh_counter
48 | 0x20240118 | GDM | RO | gdm_veto_up_counter and gdm_veto_pulse_counter
49,50,51 | 0x20240424 | CDM | RO | VX RX serial data monitor, 8 bits per VX port
52,53 | 0x20240424 | CDM | RO | VX serial link status, 4 bits per VX port
54 | 0x20240424 | CDM | RO | VX TX serial data monitor
55 | 0x20240430 | ALL | RO | QSFP, SFP and VX link loss counters
56 | 0x20240510 | ALL | RO | sfp_rx_packet_counter
57 | same | ALL | RO | sfp_tx_packet_counter
58 | same | ALL | RO | qsfp_rx_packet_counter[0]
59 | same | ALL | RO | qsfp_tx_packet_counter
60 | same | ALL | RO | vx_rx_packet_counter[0]
61 | same | ALL | RO | vx_tx_packet_counter
62 | 0x20240719 | CDM | RO | cdm_hitmap_period, ports 0, 1
63 | same | CDM | RO | ports 2, 3
64 | same | CDM | RO | ports 4, 5
65 | same | CDM | RO | ports 6, 7
66 | same | CDM | RO | ports 8, 9
67 | same | CDM | RO | ports 10, 11
68 | 0x20240814 | ALL | RW | GPS control and status
</pre>

=== Register 0 0x80013000 ds20k version ===

on read: ds20k version 0xYYYYMMDD

on write:
* 0x00 - noop - as of version 0x20240118, write a zero after writing a command
* 0x01 - CMD_reset - reset logic to good state
* 0x02 - CMD_arm_ts - arm timestamp reset
* 0x03 - CMD_trg - issue a trigger
* 0x04 - CMD_tsm - issue a tsm
* 0x05 - CMD_vx_rx_reset - reset the VX receive path
* 0x06 - CMD_vx_tx_reset - reset the VX transmit path
* 0x07 - CMD_hitmap_trg - generate a hitmap trigger and data packet
* 0x08 - CMD_trg_pulser_reset - reset the trigger pulser
* 0x09 - CMD_tsm_pulser_reset - reset the tsm pulser

=== Register 1 0x80013004 scratch ===

scratch read-write register

=== Register 2 0x80013008 input and output config ===

<pre>
bit | version | fpga name | description
0 | ALL | lemo_enable | enable LEMO input 1
1 | | | 2
2 | | | 3
3 | | | 4
4 | ALL | lemo_invert | invert LEMO input 1
5 | | | 2
6 | | | 3
7 | | | 4
9 | ALL | ext_out_disable | disable LEMO output 1
10 | | | 2
11 | ALL | ext_out_invert | invert LEMO output 1
12 | | | 2
</pre>

=== Register 3 0x8001300C FP_LED control ===

<pre>
wire [15:0] led_out_mux_sel = register_data_in[3][15:0];
wire [3:0] led_out_invert = register_data_in[3][19:16];
</pre>

<pre>
led_out_mux_sel is 4 groups (one per LED) of 4 bits (choice 0..15):
mux | version | fpga name | description
0 | ALL | | power on default
1 | ALL | led_out_reg | register 7 bits
2 | 0x20231013 | pll_locked | clock chip PLL is locked
3 | 0x20231013 | sfp_link_status | SFP link is good
3 | 0x20240118 | sfp_link_status or qsfp_tx_link_rx_status | SFP/QSFP link is good
4 | ALL | lemo_in_sync[1] | LEMO input 1
5 | ALL | lemo_in_sync[2] | LEMO input 2
6 | ALL | lemo_in_sync[3] | LEMO input 3
7 | ALL | lemo_in_sync[4] | LEMO input 4
8 | ALL | ext_out[1] | LEMO output 1
9 | ALL | ext_out[2] | LEMO output 2
A | 0x20231013 | trg_in | trigger
B | 0x20231013 | tsm_in | time slice marker
C | 0x20240118 | gdm_bsy | GDM busy: OR of all CDM busy
D | 0x20240118 | cdm_bsy | CDM busy: OR of all VX busy
E | 0x20240118 | cdm_veto | GDM busy -> GDM veto -> CDM veto -> VX trigger veto
F | ALL | | fixed logic level 1
</pre>

=== Register 4 0x80013010 LEMO OUT control ===

<pre>
wire [7:0] ext_out_mux_sel = register_data_in[4][7:0];
</pre>

<pre>
ext_out_mux_sel is 2 groups (one per LEMO) of 4 bits (choice 0..15):

mux | version | fpga name | description
0 | ALL | | power on default
1 | ALL | ext_out_reg | register 7
2 | ALL | trg_pulser | pulser trigger
3 | 0x20240724 | vx1_tx_out[2] | vx1 serial data out
4 | 0x20240724 | vx1_rx[1] | vx1 serial data in
5 | 0x20240724 | vx_rx_iob[0] | vx1 serial data in captured by IOB register
6 | ALL | lemo_in_async[1]| test synchronizer
7 | ALL | lemo_in_sync[1] | test synchronizer
8 | 0x20240724 | trg_in_pulse | trigger signal
9 | 0x20240724 | tsm_in_pulse | time slice marker signal
A | 0x20240118 | cdm_bsy | CDM busy from VX
B | 0x20240118 | gdm_bsy | GDM busy from CDM
C | 0x20240118 | cdm_veto | veto from GDM to CDM to VX
D | 0x20240118 | vx1_rx[1] | serial data VX to CDM
E | not used (sink)
F | ALL | 1 | fixed logic level 1
</pre>

=== Register 5 0x80013014 VX_RX status ===

<pre>
assign register_data_out[5] = {
vx8_rx[3], vx8_rx[2], vx8_rx[1], vx8_rx[0],
vx7_rx[3], vx7_rx[2], vx7_rx[1], vx7_rx[0],
vx6_rx[3], vx6_rx[2], vx6_rx[1], vx6_rx[0],
vx5_rx[3], vx5_rx[2], vx5_rx[1], vx5_rx[0],
vx4_rx[3], vx4_rx[2], vx4_rx[1], vx4_rx[0],
vx3_rx[3], vx3_rx[2], vx3_rx[1], vx3_rx[0],
vx2_rx[3], vx2_rx[2], vx2_rx[1], vx2_rx[0],
vx1_rx[3], vx1_rx[2], vx1_rx[1], vx1_rx[0]
};
</pre>

=== Register 6 0x80013018 VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state ===

<pre>
assign register_data_out[6] = {
vx2_tx_out[2], vx2_tx_out[1], vx2_tx_out[0], vx1_tx_out[2],
vx1_tx_out[1], vx1_tx_out[0], ext_out[2], ext_out[1],
fp_led_out[3], fp_led_out[2], fp_led_out[1], fp_led_out[0],
ext_in_lv[4], ext_in_lv[3], ext_in_lv[2], ext_in_lv[1],
vx12_rx[3], vx12_rx[2], vx12_rx[1], vx12_rx[0],
vx11_rx[3], vx11_rx[2], vx11_rx[1], vx11_rx[0],
vx10_rx[3], vx10_rx[2], vx10_rx[1], vx10_rx[0],
vx9_rx[3], vx9_rx[2], vx9_rx[1], vx9_rx[0]
};
</pre>

<pre>
bit | version | fpga name | description
0 | ALL | vx9_rx | VX_RX
1 | | |
2 | | |
3 | | |
4 | ALL | vx10_rx | VX_RX
5 | | |
6 | | |
7 | | |
8 | ALL | vx11_rx | VX_RX
9 | | |
10 | | |
11 | | |
12 | ALL | vx12_rx | VX_RX
13 | | |
14 | | |
15 | | |
16 | ALL | ext_in_lv | LEMO inputs
17 | | |
18 | | |
19 | | |
20 | ALL | FP_LED | FP_LEDs
21 | | |
22 | | |
23 | | |
24 | ALL | ext_out[1] | LEMO outputs
25 | | ext_out[2] |
26 | ALL | vx1_tx | VX1_TX
27 | | |
28 | | |
29 | ALL | vx2_tx | VX2_TX
30 | | |
31 | | |
</pre>

=== Register 7 0x8001301C LED_OUT, EXT_OUT, VX_TX outputs ===

<pre>
wire [3:0] led_out_reg = register_data_in[7][3:0];
wire [2:1] ext_out_reg = register_data_in[7][5:4];
// register_data_in[7][6];
// register_data_in[7][7];
wire [7:0] vx_tx_out_reg = register_data_in[7][15:8];
</pre>

<pre>
bit | version | fpga name | description
0 | ALL | led_out_reg | FP_LED 1
1 | | | 2
2 | | | 3
3 | | | 4
4 | ALL | ext_out_reg | LEMO OUT 1
5 | | | 2
6 | - | |
7 | - | |
8 | ALL | vx_tx_out_reg | VX1_TX 0
9 | | | 1
10 | | | 2
11 | | | -
12 | ALL | | VX2_TX 0
13 | | | 1
14 | | | 2
15 | | | -
16 | - | |
17 | | |
18 | | |
19 | | |
20 | | |
21 | | |
22 | | |
23 | | |
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 8 0x80013020 VX_TX config ===

<pre>
wire [3:0] vx_tx_mux_sel = register_data_in[8][3:0];
vx_tx_loopback = register_data_in[8][31];
</pre>

<pre>
vx_tx_mux_sel is 4 bits (choice 0..15):
0 - power on default, control by vx_tx_out_reg
1 - GDM
2 - CDM
3 - pulser loopback test
4 - pulser loopback test
5 - 62.5 MHz output
6 - 125 MHz output
7 - trg, tsm, serial
8 - trg, veto, serial
9 - GPS box control (ds20k rev 0x20240814)
10
11
12
13
14
15 - production config: trg, veto, serial
</pre>

=== Register 9 0x80013024 trigger control ===

from version 0x20240724

<pre>
wire [15:0] trg_src_mask = register_data_in[9][15:0];
wire [31:16] tsm_src_mask = register_data_in[9][31:16];
</pre>

<pre>
wire [15:0] trg_src_bits =
{
1'b0,
1'b0,
1'b0,
1'b0,

1'b0, // gdm_hitmap_trigger,
cdm_hitmap_trigger,
sfp_rx_tsm,
sfp_rx_trg,

sfp_rx_data[1],
sfp_rx_data[0],
tsm_pulser,
trg_pulser,

lemo_in_sync[4],
lemo_in_sync[3],
lemo_in_sync[2],
lemo_in_sync[1]
};

wire [15:0] trg_bits = trg_src_bits & trg_src_mask;
wire [15:0] tsm_bits = trg_src_bits & tsm_src_mask;
</pre>

before that:

<pre>
wire [7:0] trg_src_mask = register_data_in[9][7:0];
wire [7:0] tsm_src_mask = register_data_in[9][15:8];
wire trg_pulser_enable = register_data_in[9][16];
wire tsm_pulser_enable = register_data_in[9][17];
wire trg_software = register_data_in[9][18];
wire tsm_software = register_data_in[9][19];
// bits 20:31 not used
</pre>

<pre>
wire [7:0] xxx_src_bits =
{
sfp_rx_data[1],
sfp_rx_data[0],
tsm_pulser & tsm_pulser_enable,
trg_pulser & trg_pulser_enable,
lemo_in_sync[4],
lemo_in_sync[3],
lemo_in_sync[2],
lemo_in_sync[1]
};
</pre>

<pre>
trg_src_mask and tsm_src_mask bits:
0 - LEMO IN 1
1 - LEMO IN 2
2 - LEMO IN 3
3 - LEMO IN 4
4 - trg_pulser
5 - tsm_pulser
6 - sfp_rx_data[0] // to become sfp_trg_in, selected from sfp_rx_data[0], cdm_rx trg_out and gdm trigger packet
7 - sfp_rx_data[1] // to become sfp_tsm_in, selected from sfp_rx_data[1], cdm_rx tsm_out and gdm tsm packet
</pre>

=== Register 10 0x80013028 status register ===

<pre>
bit | version | fpga name | description
0 | 0x20231013 | pll_locked | clock chip PLL is locked
1 | 0x20240118 | ts_reset_armed | timestamp reset is armed
2 | 0x20240118 | qsfp_tx_link_rx_status | QSFP link status is good for all enabled ports
3 | 0x20240424 | vx_tx_link_rx_status | VX link status is good for all enabled ports
4 | 0x20240118 | cdm_bsy | VX busy grand-or
5 | 0x20240118 | gdm_bsy | QSFP busy grand-or
6 | 0x20240118 | gdm_veto | gdm_veto = gdm_busy
7 | 0x20240118 | cdm_veto | CDM veto from GDM to VX
8 | 0x20240725 | bor_started | begin-of-run sequence started
9 | 0x20240725 | bor_finished | begin-of-run sequence fininished, see commands 10 and 11
10 | | |
11 | | |
12 | | |
13 | | |
14 | | |
15 | | |
16 | | |
17 | | |
18 | | |
19 | | |
20 | | |
21 | | |
22 | | |
23 | | |
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 11 0x8001302C trg_counter ===

trigger counter

=== Register 12 0x80013030 tsm_counter ===

time slice marker counter

=== Register 13 0x80013034 GPS 1pps period ===

GPS 1pps period in 8 ns clocks

=== Register 14 0x80013038 Rb clock 1pps period ===

PRS-10 Rb clock 1pps output period in 8 ns clocks

=== Register 15 0x8001303C SFP RX status ===

<pre>
bit | ds20k version | fpga signal name | description
0 | ALL | sfp_rx_data[15:0] | cdm sfp received data
16 | same | sfp_rx_data_is_k[0] |
17 | same | sfp_rx_data_is_k[1] |
18 | | 0 |
19 | 0x20231204 | sfp_rx_sel_lpb | sfp tx->rx loopback
20 | 0x20231013 | sfp_link_status | sfp link connected, exchanging data
21 | same | sfp_link_rx_status | sfp link receiving correct idle pattern from GDM TX
22 | same | sfp_link_error | sfp link receiver error (badk or overflow)
23 | same | sfp_rx_data_error | sfp transceiver state machine is in error state
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 16 0x80013040 SFP TX control ===

<pre>
wire [15:0] sfp_tx_data_reg = register_data_in[16][15:0];
wire [1:0] sfp_tx_ctrl_reg = register_data_in[16][17:16];
wire sfp_rx_sel_lpb = register_data_in[16][28]; // TX->RX serial loopback
wire sfp_tx_sel_lpb = register_data_in[16][29]; // RX->TX serial loopback
wire sfp_tx_sel_trg = register_data_in[16][30]; // 16 individual bits
wire sfp_tx_sel_reg = register_data_in[16][31]; // from register
</pre>

=== Register 17-22 0x80013044 QSFP RX data ===

QSFP RX data links 0..11

=== Register 23 0x8001305C QSFP TX control ===

<pre>
wire [15:0] qsfp_tx_data_reg = register_data_in[23][15:0];
wire [1:0] qsfp_tx_ctrl_reg = register_data_in[23][17:16];
wire qsfp_rx_sel_lpb = register_data_in[23][28]; // TX->RX loopback
wire qsfp_tx_sel_lpb = register_data_in[23][29]; // RX->TX loopback
wire qsfp_tx_sel_trg = register_data_in[23][30]; // 16 individual bits
wire qsfp_tx_sel_reg = register_data_in[23][31]; // data from register
</pre>

=== Register 24 0x80013060 trigger pulser period ===

trigger pulser period in units of 8 ns (125 MHz clock)

=== Register 25 0x80013064 trigger burst pulser ===

<pre>
wire [7:0] conf_burst_count = conf_pulser_burst_ctrl[31:24];
wire [23:0] conf_burst_period = conf_pulser_burst_ctrl[23:0];
</pre>

=== Register 26 0x80013068 tsm pulser period ===

time slice marker period in units of 8 ns (125 MHz clock)

=== Register 27 0x8001306C data write fifo ===

<pre>
wire fifo_reset = register_data_in[27][31];
wire fifo_to_fpga_wr1 = register_data_in[27][27];
wire fifo_to_fpga_wr2 = register_data_in[27][26];
wire [16:0] fifo_to_fpga_din = register_data_in[27][16:0];
</pre>

<pre>
assign register_data_out[27][31:24] = register_data_in[27][31:24]; // echo write bits
assign register_data_out[27][23] = fifo_to_fpga_full;
assign register_data_out[27][22] = fifo_to_fpga_empty;
</pre>

=== Register 28 0x80013070 data read fifo ===

<pre>
wire fifo_reset = register_data_in[28][31];
wire fifo_from_fpga_rd1 = register_data_in[28][25];
wire fifo_from_fpga_rd2 = register_data_in[28][24];
</pre>

<pre>
assign register_data_out[28][31:24] = register_data_in[28][31:24]; // echo write bits
assign register_data_out[28][21] = fifo_from_fpga_full;
assign register_data_out[28][20] = fifo_from_fpga_empty;
assign register_data_out[28][16:0] = fifo_from_fpga_dout;
</pre>

=== Register 29 0x80013074 packet routing ===

Control packet routing and loopbacks:

<pre>
wire [3:0] dn_route_ctrl = register_data_in[29][3:0];
wire [3:0] up_route_ctrl = register_data_in[29][7:4];
wire [3:0] fifo_to_fpga_route_ctrl = register_data_in[29][11:8];
//wire [3:0] spare_route_ctrl = register_data_in[29][15:12];

wire dn_mux_trg_enable = register_data_in[29][16];
wire dn_mux_tsm_enable = register_data_in[29][17];
wire dn_mux_sfp_rx_fifo_enable = register_data_in[29][18];
// 19
// 20..23

wire up_mux_vx_rx_enable = register_data_in[29][24];
// 25
// 26
// 27
wire fifo_from_fpga_hitmap_enable = register_data_in[29][28];
// 29..31
</pre>

fifo_to_fpga output routing:
* 0 - to down packet mux
* 1 - to up packet mux
* 2 - to fifo_from_fpga mux
* 3 - not used

down packet mux inputs:
* fifo_to_fpga_0
* trg_pkt16 enabled by dn_mux_trg_enable
* tsm_pkt16 enabled by dn_mux_tsm_enable
* sfp_rx_fifo_pkt16 enabled by dn_mux_sfp_rx_fifo_enable
* up_pkt16_2 loopback from up packet mux

down packet mux output routing:
* 0 - to vx_tx_pkt16 to VX TX (vx_link_tx and vx_ser_tx)
* 1 - to fifo_from_fpga mux
* 2 - to up packet mux loopback
* 3 - to qsfp_tx_pkt16 to GDM QSFP TX (cdm_link_tx)

up packet mux inputs:
* vx_rx_pkt16 enabled by up_mux_vx_rx_enable data from 12 VX RX links
* fifo_to_fpga_1
* dn_pkt16_2 loopback from down packet mux

up packet mux output routing:
* 0 - to sfp_tx_pkt16 to CDM SFP TX (VX data to GDM)
* 1 - to fifo_from_fpga mux
* 2 - to down packet mux loop loopback
* 3 - not used

fifo_from_fpga mux inputs:
* fifo_to_fpga_2
* dn_pkt16_1 from down packet mux
* up_pkt16_1 from up packet mux
* qsfp_tx_pkt16 from GDM QSFP link 0 (there is no GDM QSFP 12-to-1 mux)
* hitmap_pkt16 enabled by fifo_from_fpga_hitmap_enable in register 29

=== Register 30 0x80013078 qsfp link status ports 0..7 ===

<pre>
assign register_data_out[30] =
{
qsfp_rx_data_error[7], qsfp_link_error[7], qsfp_link_status[7], qsfp_link_rx_status[7], // 7
qsfp_rx_data_error[6], qsfp_link_error[6], qsfp_link_status[6], qsfp_link_rx_status[6], // 6
qsfp_rx_data_error[5], qsfp_link_error[5], qsfp_link_status[5], qsfp_link_rx_status[5], // 5
qsfp_rx_data_error[4], qsfp_link_error[4], qsfp_link_status[4], qsfp_link_rx_status[4], // 4
qsfp_rx_data_error[3], qsfp_link_error[3], qsfp_link_status[3], qsfp_link_rx_status[3], // 3
qsfp_rx_data_error[2], qsfp_link_error[2], qsfp_link_status[2], qsfp_link_rx_status[2], // 2
qsfp_rx_data_error[1], qsfp_link_error[1], qsfp_link_status[1], qsfp_link_rx_status[1], // 1
qsfp_rx_data_error[0], qsfp_link_error[0], qsfp_link_status[0], qsfp_link_rx_status[0] // 0
};
</pre>

=== Register 31 0x8001307C qsfp link status ports 8..11 ===

<pre>
assign register_data_out[31] =
{
4'b0000,
4'b0000,
4'b0000,
4'b0000,
qsfp_rx_data_error[11], qsfp_link_error[11], qsfp_link_status[11], qsfp_link_rx_status[11], // 11
qsfp_rx_data_error[10], qsfp_link_error[10], qsfp_link_status[10], qsfp_link_rx_status[10], // 10
qsfp_rx_data_error[9], qsfp_link_error[9], qsfp_link_status[9], qsfp_link_rx_status[9], // 9
qsfp_rx_data_error[8], qsfp_link_error[8], qsfp_link_status[8], qsfp_link_rx_status[8] // 8
};
</pre>

=== Register 32 0x80013080 bitmap of enabled qsfp ports ===

<pre>
wire [11:0] qsfp_mask = register_data_in[32][11:0];
wire qsfp_bsy_force = register_data_in[32][12];
// not used = register_data_in[32][15:13];
wire [15:0] gdm_veto_extend = register_data_in[32][31:16];
</pre>

On the GDM:

qsfp_rx_bsy[11..0] are pulses received from the CDMs

qsfp_rx_bsy_or is the grand-or of qsfp_rx_bsy masked by qsfp_mask

if qsfp_rx_bsy_or is high, gdm_bsy goes up and stays up for vx_bsy_extend*2 clocks

gdm_veto = gdm_bsy (bsy of any one VX causes trigger veto to all of them)

gdm_veto transition 0->1 causes gdm_veto_pulse. as long as gdm_veto is high, gdm_veto_pulse is generated every gdm_veto_extend clocks

gdm_veto_pulse is sent to all CDMs.

For this to work right, cdm_bsy_extend should not be bigger than gdm_veto_extend.

=== Register 33 0x80013084 bitmap of enabled VX ports ===

<pre>
wire [11:0] vx_mask = register_data_in[33][11:0];
wire vx_bsy_force = register_data_in[33][12];
// not used = register_data_in[33][15:13];
wire [15:0] vx_bsy_extend = register_data_in[33][31:16];
</pre>

On the CDM:

cdm_bsy is a grand-or of all vx_bsy masked by vx_mask (list of active VXes).

vx_bsy_extend controls how often state of cdm_bsy is sent to the GDM. when cdm_bsy goes 0->1, we send a cdm_bsy_pulse and keep resending it every vx_bsy_extend clocks as long as cdm_bsy stays high.

cdm_bsy_pulse is sent to the GDM.

sfp_rx_veto is the received from the GDM

if sfp_rx_veto goes up, cdm_veto goes up and stays up for gdm_veto_extend clocks.

for this to work right, CDM gdm_veto_extend must be bigger than GDM gdm_veto_extend.

cdm_veto goes to VXes on v1_tx_out[1] which is LVDS input 13.

=== Register 34, 35 0x80013088, 8C current timestamp ===

current 64-bit timestamp, 125 MHz

=== Register 36, 37 0x80013090, 94 old timestamp ===

old 64-bit timestamp, 125 MHz. timestamp saved at run start when it is reset to 0.

=== Register 38, 39, 40 0x80013098, 9C, A0 VX busy counters ===

8 bits per VX port, counters overflow to 255, reset at run start.

=== Register 41, 42, 43 0x800130A4, A8, AC QSFP busy counters ===

8 bits per QSFP port, counters overflow to 255, reset at run start.

=== Register 44, 45, 46, 47, 48 0x800130B0, B4, B8, BC, C0 CDM and GDM busy and veto counters ===

* cdm_busy = grand-or of all VX busy for enabled VXes
* gdm_busy = grand-or of all CDM busy for enabled CDMs
* gdm_veto = gdm_busy

<pre>
44 | lo 16 bits | cdm_bsy_up_counter | CDM busy, increments when cdm_busy goes 0->1
44 | hi 16 bits | cdm_bsy_pulse_counter | CDM busy to GDM, increments for each cdm_bsy_pulse sent to the GDM

45 | lo 16 bits | cdm_veto_pulse_counter | CDM veto from GDM, increments for each sfp_rx_veto received from the GDM
45 | hi 16 bits | cdm_veto_up_counter | CDM veto to VX, increments each time cdm_veto is set to 1.

46 | lo 16 bits | gdm_bsy_pulse_counter | GDM busy from CDM, increments for each qsfp_rx_busy received from CDMs (unless they overlap)
46 | hi 16 bits | gdm_bsy_up_counter | GDM busy, increments each time gdm_bsy goes 0->1

47 | lo 16 bits | gdm_bsy_refresh_counter | GDM busy, increments each time gdm_bsy is extended by new qsfp_rx_busy
47 | hi 16 bits | spare |

48 | lo 16 bits | gdm_veto_up_counter | GDM veto, increments each time gdm_veto goes 0->1
48 | hi 16 bits | gdm_veto_pulse_counter | GDM veto to CDM, increments for each gdm_veto_pulse sent to the CDM
</pre>

=== Register 49, 50, 51 0x800130xx VX RX serial data monitor ===

VX RX data, 8-bit per VX channel. k-bit is omitted.

<pre>
assign register_data_out[49][7:0] = vx_rx_data[0]; // vx1
assign register_data_out[49][15:8] = vx_rx_data[1]; // vx2
assign register_data_out[49][23:16] = vx_rx_data[2]; // vx3
assign register_data_out[49][31:24] = vx_rx_data[3]; // vx4

assign register_data_out[50][7:0] = vx_rx_data[4]; // vx5
assign register_data_out[50][15:8] = vx_rx_data[5]; // vx6
assign register_data_out[50][23:16] = vx_rx_data[6]; // vx7
assign register_data_out[50][31:24] = vx_rx_data[7]; // vx8

assign register_data_out[51][7:0] = vx_rx_data[8]; // vx9
assign register_data_out[51][15:8] = vx_rx_data[9]; // vx10
assign register_data_out[51][23:16] = vx_rx_data[10]; // vx11
assign register_data_out[51][31:24] = vx_rx_data[11]; // vx12
</pre>

=== Register 52, 53 VX link status ===

<pre>
assign register_data_out[52] =
{
vx_rx_error[7], vx_link_error[7], vx_link_status[7], vx_link_rx_status[7], // 7
vx_rx_error[6], vx_link_error[6], vx_link_status[6], vx_link_rx_status[6], // 6
vx_rx_error[5], vx_link_error[5], vx_link_status[5], vx_link_rx_status[5], // 5
vx_rx_error[4], vx_link_error[4], vx_link_status[4], vx_link_rx_status[4], // 4
vx_rx_error[3], vx_link_error[3], vx_link_status[3], vx_link_rx_status[3], // 3
vx_rx_error[2], vx_link_error[2], vx_link_status[2], vx_link_rx_status[2], // 2
vx_rx_error[1], vx_link_error[1], vx_link_status[1], vx_link_rx_status[1], // 1
vx_rx_error[0], vx_link_error[0], vx_link_status[0], vx_link_rx_status[0] // 0
};

assign register_data_out[53] =
{
//4'b0000,
1'b0, ~vx_rx_deser_rdy[0], vx_rx_code_err[0], vx_rx_disp_err[0],
//4'b0000,
//4'b0000,
//4'b0000,
vx_rx_monitor[0],
vx_rx_error[11], vx_link_error[11], vx_link_status[11], vx_link_rx_status[11], // 11
vx_rx_error[10], vx_link_error[10], vx_link_status[10], vx_link_rx_status[10], // 10
vx_rx_error[9], vx_link_error[9], vx_link_status[9], vx_link_rx_status[9], // 9
vx_rx_error[8], vx_link_error[8], vx_link_status[8], vx_link_rx_status[8] // 8
};
</pre>

vx_rx_monitor (12-bit) is from deserializer_10b.sv:

<pre>
assign monitor_out[9:0] = lastByte[9:0];
assign monitor_out[10] = comma;
assign monitor_out[11] = ready;
</pre>

=== Register 54 VX TX serial data monitor ===

* contents of vx_tx_monitor from vx_ser_tx.sv:
<pre>
assign monitor_out[8:0] = data_to_encoder; // 8-bit + k
assign monitor_out[9] = valid;
assign monitor_out[15:10] = 0;
assign monitor_out[25:16] = encoded_data; // 10-bit
assign monitor_out[26] = encoded_valid;
assign monitor_out[27] = 0;
assign monitor_out[31:28] = 0;
</pre>

=== Register 55 QSFP, SFP, VX link loss counters ===

<pre>
assign register_data_out[55] = {
vx_rx_error_counter,
qsfp_link_rx_status_drop_counter,
sfp_link_rx_status_drop_counter,
vx_link_rx_status_drop_counter
};
</pre>

<pre>
24..31 - CDM VX RX error counters, count any errors in the VX RX data path (bad serial data, fifo overflow, etc)
16..23 - GDM QSFP link loss counter, increments on qsfp_tx_link_rx_status 1->0 (reg10)
8..15 - CDM SFP link loss counter, increments on sfp_link_rx_status 1->0 (reg15)
0...7 - CDM VX link loss counter, increments on vx_tx_link_rx_status 1->0 (reg10)
</pre>

=== Register 68 GPS control and status ===

<pre>
assign register_data_out[68] =
{
8'b00000000, // 23+8
rb_1pps_counter, // 16+8 bits
gps_1pps_counter, // 8+8 bits
1'b0,
1'b0,
rb_ser_in, // 5
gps_data_in, // 4
gps_aux_out, // 3
gps_aux_in, // 2
rb_1pps_in, // 1
gps_1pps_in // 0
};
</pre>

= Firmware registers branch develop_ko =

== Register map ==

<pre>
0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
1 | ALL | ALL | RW | read write scratch register
2 | ALL | CDM | ?? | MGT not used
3 | ALL | CDM | RO | MGT debug_data
4 | ALL | CDM | RW | clk_config_vec
5 | ALL | CDM | ?? | not used
6 | ALL | CDM | RO | CDM_link_data_processing:o_error_time
7 | ALL | CDM | RO | CDM_link_data_processing:o_error_count
</pre>

== register 0 0x80010000 ==

GDM:
<pre>
0 - gdm_link_interface:i_mgt_rst
2 - gdm_link_interface:i_link_down_latched_rst
8 - GDM_link_data_processing:i_rst
10..9 - GDM_link_data_processing:i_data_mode
</pre>

CDM:
<pre>
0 - cdm_link_interface:i_mgt_rst
2 - cdm_link_interface:i_link_down_latched_rst
8 - CDM_link_data_processing:i_rst
10..9 - CDM_link_data_processing:i_data_mode
</pre>

== register 1 0x80010004 ==

GDM:
<pre>
nlinks-1..0 - gdm_link_interface:i_rx_slide_trigger
</pre>

CDM:
<pre>
nlinks-1..0 - cdm_link_interface:i_rx_slide_trigger
</pre>

== register 2 0x80010008 ==

GDM:
<pre>
nlinks-1..0 - gdm_link_interface:o_link_power_good
nlinks+15..16 - gdm_link_interface:o_link_status
</pre>

CDM:
<pre>
nlinks-1..0 - cdm_link_interface:o_link_power_good
nlinks+15..16 - cdm_link_interface:o_link_status
</pre>

== register 3 0x8001000c ==

GDM: simple loopback register

CDM:
<pre>
31..0 - debug_data - cdm_link_interface:o_debug
</pre>

o_debug:
<pre>
rx_link_rst & rx_error & rx_link_up & rx_receiving_data &
std_logic_vector(rx_state_count) & tx_state_count_on_rx_clk & i_rx_ctrl3(0) &
i_rx_ctrl1(1 downto 0) & i_rx_ctrl0(1 downto 0) &
rx_data_is_k28p1_k28p5 &
i_rx_data;
</pre>

== register 4 0x80010010 ==

GDM write:
<pre>
0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS
</pre>

GDM read:
<pre>
0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS
4 - clk_config_vec(4) - CLK_LOSXTn_LS
5 - clk_config_vec(5) - CLK_LOLn_LS
6 - clk_config_vec(6) - CLK_INTn_LS
7 - constant 1
31..8 - constant 0
</pre>

== register 5 0x80010014 ==

not used

== register 6 0x80010018 ==

GDM:
<pre>
3..0 - GDM_link_data_processing:i_status_select
</pre>

CDM:
<pre>
31..0 - CDM_link_data_processing:o_error_count
</pre>

== register 7 0x8001001c ==

GDM:
<pre>
31..0 - GDM_link_data_processing:o_status_vector
</pre>

CDM:
<pre>
31..0 - CDM_link_data_processing:o_error_count
</pre>

= GDM, CDM, VX packet communications =

* timestamp math
<pre>
1 clock is 8 ns is 125 MHz
8 bits of clocks is 256 clocks is 2048 ns is ~2 usec
16 bits of clocks is ~500 usec is 0.5 msec
24 bits of clocks is ~134 msec
32 bits of clocks is ~34 sec
40 bits of clocks is ~8.7 ksec is 2.4 hours
48 bits of clocks is ~625 hours is ~26 days
56 bits of clocks is ~6.6 kdays is ~18 kyears
62 bits of clocks is ~10 Mhours is 427 kdays is ~1.1 kyears
64 bits of clocks is ~4.4 kyears
</pre>
* 0x02 - TRG packet, 8 bytes, 80 adc clocks, 640 ns on lvds link
<pre>
0 - 0x02
1 - trg_counter[7:0]
2 - ts64 low byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 high byte 3
6 - trg_in_latch[7:0]
7 - trg_in_latch[15:8]
</pre>
* 0x03 - HITMAP_TRG packet, 12 bytes, 120 adc clocks, 960 ns on lvds link
<pre>
0 - 0x03
1 - trg_counter[7:0]
2 - ts64 low byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 high byte 3
6 - vx_bitmap[7:0]
7 - vx_bitmap[15:8]
8 - vx_bitmap[23:16]
9 - vx_bitmap[31:24]
10 - vx_bitmap[39:32]
11 - vx_bitmap[47:40]
</pre>
* 0x10 - TSM packet, 26 bytes, 260 adc clocks, 2080 ns on lvds link
<pre>
0 - 0x10
1 - tsm_counter[7:0]
2 - gdm_ts64 low byte 0
3 - 1
4 - 2
5 - 3
6 - 4
7 - 5
8 - 6
9 - gdm_ts64 high byte 7
10 - gps_ts64 low byte 0
11 - 1
12 - 2
13 - 3
14 - 4
15 - 5
16 - 6
17 - gps_ts64 high byte 7
18 - gps_data64 low byte 0
19 - 1
20 - 2
21 - 3
22 - 4
23 - 5
24 - 6
25 - gps_data64 high byte 7
</pre>
* 0x81 - VX hitmap packet, 10 bytes, 100 adc clocks, 800 ns on lvds link, 48 ns on fiber link
<pre>
0 - 0x81
1 - VX ID
2 - hitmap low byte, nits 7:0
3 - 15:8
4 - 23:16
5 - 31:24
6 - ...:32
7 - ...
8 - ...
9 - hitmap low byte, bits 63:...
</pre>
* 0x82 - CDM hitmap packet, 108 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x82
1 - cdm_hitmap_trigger_counter[7:0]
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - cdm_hitmap_or12 byte 0 (7:0)
11 - cdm_hitmap_or12 byte 1 (11:8) plus 4 bits: 12=0, 13=0, 14=0, 15=cdm_hitmap_grand_or
12 - cdm_hitmap_data, low byte, 12*64 bits = 768 bits = 48 words = 96 bytes
...
107 - cdm_hitmap_data, high byte
</pre>

= AXI bus timing =

* AXI 100 MHz clock, 10 ns, 32-bit data
* AXI single-dword read: 36 clock repeat rate, 360 ns is 2.777 MHz, 4 bytes per transfer is 11.11 Mbytes/sec
* AXI single-qword read: 13 clock repeat rate, 130 ns is 7.7 MHz, 4 bytes per transfer is 30 Mbytes/sec, not accounting for the gap
* AXI memcpy read: 4 transfers at 13 clocks, gap, 4 transfers at 13 clocks, gap, etc. below 30 Mbytes/sec.
[[Image:Ds-dm-axi-read.png|100px]]
[[Image:Ds-dm-axi-read-64.png|100px]]
[[Image:Ds-dm-axi-read-memcpy.png|100px]]
* AXI single-dword write: 20 clocks repeat rate, 200 ns is 5 MHz, 4 bytes per transfer is 20 Mbytes/sec
* AXI single-qword write: 13+20 clocks repeat rate, 330 ns is 3 MHz, 16 bytes per burst is 48 Mbytes/sec
[[Image:Ds-dm-axi-write.png|100px]]
[[Image:Ds-dm-axi-write-64.png|100px]]

= AXI bus addresses =

* see AXI/AMBA addresses assigned inthe FPGA project: (s_axi/reg0 is the DS-DM AXI registers)
<pre>
daq00:ds-dm-gcdm$ grep assign_bd_address scripts/GDM_CDM_XU8_bd.tcl
assign_bd_address -offset 0x80010000 -range 0x00004000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs axi_register_interfa_0/s_axi/reg0] -force
assign_bd_address -offset 0x000400000000 -range 0x40000000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs ddr4/C0_DDR4_MEMORY_MAP/C0_DDR4_ADDRESS_BLOCK] -force
assign_bd_address -offset 0x80000000 -range 0x00010000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs system_management_wiz/S_AXI_LITE/Reg] -force
daq00:ds-dm-gcdm$
</pre>

* see AXI/AMBA addresses exported from FPGA project to Linux kernel: (uio for debug bridge should say "debug bridge")
<pre>
root@gdm0:~# cat /sys/class/uio/uio*/name
axi-pmon
axi-pmon
axi-pmon
axi-pmon
root@gdm0:~#
root@gdm0:~# ls -l /sys/class/uio/
total 0
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio0 -> ../../devices/platform/amba/ffa00000.perf-monitor/uio/uio0
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio1 -> ../../devices/platform/amba/fd0b0000.perf-monitor/uio/uio1
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio2 -> ../../devices/platform/amba/fd490000.perf-monitor/uio/uio2
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio3 -> ../../devices/platform/amba/ffa10000.perf-monitor/uio/uio3
root@gdm0:~# ls -l /sys/class/uio/../../devices/platform/amba/
total 0
-rw-r--r-- 1 root root 4096 Oct 18 19:37 driver_override
drwxr-xr-x 3 root root 0 Oct 18 01:36 fd070000.memory-controller
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd0b0000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd400000.zynqmp_phy
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd490000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd500000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd510000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd520000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd530000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd540000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd550000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd560000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd570000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd6e0000.cci
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff000000.serial
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff020000.i2c
drwxr-xr-x 6 root root 0 Oct 18 01:36 ff0a0000.gpio
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0b0000.ethernet
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0e0000.ethernet
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0f0000.spi
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff160000.mmc
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff170000.mmc
drwxr-xr-x 3 root root 0 Oct 18 01:36 ff960000.memory-controller
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff9d0000.usb0
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa00000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa10000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa50000.ams
drwxr-xr-x 5 root root 0 Oct 18 01:36 ffa60000.rtc
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa80000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa90000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffaa0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffab0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffac0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffad0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffae0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffaf0000.dma
-r--r--r-- 1 root root 4096 Oct 18 19:37 modalias
lrwxrwxrwx 1 root root 0 Oct 18 19:37 of_node -> ../../../firmware/devicetree/base/amba
drwxr-xr-x 2 root root 0 Oct 18 19:37 power
lrwxrwxrwx 1 root root 0 Oct 18 01:36 subsystem -> ../../../bus/platform
-rw-r--r-- 1 root root 4096 Oct 18 01:36 uevent
root@gdm0:~#
</pre>

= Build firmware =

== Build from git clone ==

THESE ARE K.O.'s NOTES FOR CREATING THE PETALINUX DIRECTORY.

THEY DO NOT WORK!

COPY PETALINUX FROM A WORKING PROJECT AND USE "make gdm" and "make cdm" AS DESCRIBED BELOW.

* git clone git@edev-group.triumf.ca:fw/exp/darkside/gcdm.git
* #Makefile change VIVADO_SETTINGS_SCRIPT := /opt/Xilinx/Vivado/2022.1/settings64.sh
* #. /opt/Xilinx/Vivado/2022.1/settings64.sh
* . /opt/Xilinx/Vivado/2020.2/settings64.sh
* make clean
* make all_from_scratch
* . /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
* make petalinux_create
* make petalinux_rebuild_new_hw_des
* bomb out: The TMPDIR: /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp can't be located on nfs.
* mkdir /tmp/build_tmp
* rm -rf /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp/
* ln -s /tmp/build_tmp /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp
* try again
* grinds, loads a whole bunch of packages...
* finishes with desire to copy things to /tftpboot
* make sdcard_cp_to wants to copy files from PetaLinux_GDM_CDM/images/linux/ to SD card

== Build firmware ==

NOTE: directory Petalinux_GDM_CDM should already exist!

<pre>
#. /opt/Xilinx/Vivado/2020.2/settings64.sh
. /opt/Xilinx/Vivado/2022.2/settings64.sh
. /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
make clean_gdm # remove gdm build tree
make gdm # build or rebuild GDM
make copy_gdm # copy to gdm0
make clean_cdm # remove cdm build tree
make cdm # build or rebuild CDM
make copy_cdm # copy to cdm0 and cdm1
</pre>

copy to SD card:
<pre>
open a root shell
format 16 GB Sd card per above
cd .../ds-dm-gcdm
make copy
</pre>

= build times =

<pre>
CDM 12-june-2023 69aabc1c25130d970bc375aca684bd68849e6685
daq13 AMD-5700G 1688.61user 399.33system 23:28.84elapsed 148%CPU
dsdaqgw AMD-7700 1090.55user 247.34system 16:03.55elapsed 138%CPU
dsdaqgw AMD-7700 CDM incremental 196.68user 67.62system 7:35.42elapsed 58%CPU
dsdaqgw AMD-7700 CDM incremental 684.72user 94.17system 7:30.17elapsed 173%CPU
dsdaqgw AMD-7700 GDM incremental 849.84user 99.79system 9:04.56elapsed 174%CPU
</pre>

= prepare bootable sd card =

== format the sd card ==

this only needs to be done once

* become root
* cd ~olchansk/git/ds-dm-gcdm
* use "lsblk" to identify the SD card (should show as 8/16/32 GB block device)/ /dev/sdd in this case
* make sdcard_format SDCARD_DEVICE=/dev/sdd
* disconnect sd card, reconnect the sd card (to detect new partition tables, etc)

== copy CDM boot files ==

<pre>
cd /home/dsdmdev/git/ds-dm-gcdm
make copy
</pre>

== copy boot files to the sd card ==

* as root: identify partition labels, run "blkid", should say "BOOT", "rootfs" and "data"
* mount
<pre>
mkdir /media/olchansk/BOOT
mkdir /media/olchansk/rootfs
mkdir /media/olchansk/data
mount -L BOOT /media/olchansk/BOOT
mount -L rootfs /media/olchansk/rootfs
mount -L data /media/olchansk/data
cp PetaLinux_GDM_CDM/images/linux/BOOT.BIN /media/olchansk/BOOT/
cp PetaLinux_GDM_CDM/images/linux/boot.scr /media/olchansk/BOOT/
cp PetaLinux_GDM_CDM/images/linux/image.ub /media/olchansk/BOOT/
umount /media/olchansk/BOOT
umount /media/olchansk/rootfs
umount /media/olchansk/data
eject /dev/sdd
</pre>

= boot messages =

<pre>
Xilinx Zynq MP First Stage Boot Loader
Release 2020.2 Sep 24 2022 - 13:29:15
NOTICE: ATF running on XCZU4CG/silicon v4/RTL5.1 at 0xfffea000
NOTICE: BL31: v2.2(release):xlnx_rebase_v2.2_2020.3
NOTICE: BL31: Built : 18:02:46, Sep 28 2022

U-Boot 2020.01 (Sep 28 2022 - 18:03:39 +0000)

Model: DarkSide 20k DM
Board: Xilinx ZynqMP
DRAM: 2 GiB
usb dr_mode not found
PMUFW: v1.1
EL Level: EL2
Chip ID: zu4
NAND: 0 MiB
MMC: mmc@ff160000: 0, mmc@ff170000: 1
In: serial@ff000000
Out: serial@ff000000
Err: serial@ff000000
Bootmode: SD_MODE1
Reset reason: SOFT
Net:
ZYNQ GEM: ff0b0000, mdio bus ff0b0000, phyaddr -1, interface rgmii-id

Warning: ethernet@ff0b0000 (eth0) using random MAC address - d6:62:5f:13:00:44
eth0: ethernet@ff0b0000
ZYNQ GEM: ff0e0000, mdio bus ff0e0000, phyaddr -1, interface rgmii-id
Could not get PHY for eth1: addr -1

Hit any key to stop autoboot: 0
ZynqMP>
CTRL-A Z for help | 115200 8N1 | NOR | Minicom 2.7.1 | VT102 | Online 122:2 | ttyACM0
</pre>

= load FPGA from u-boot =

* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/124682257/U-Boot+FPGA+Driver

<pre>
ZynqMP> fpga info
Xilinx Device
Descriptor @ 0x000000007fddb2c0
Family: ZynqMP PL
Interface type: csu_dma configuration interface (ZynqMP)
Device Size: 1 bytes
Cookie: 0x0 (0)
Device name: zu4
Device Function Table @ 0x000000007fda5fe8
PCAP status 0xa0002fde
ZynqMP>
</pre>

* cp CDM_XU8_top.bit /tftpboot/fpga.bit

<pre>
dhcp
tftpb 0x10000000 fpga.bit
fpga loadb 0 0x10000000 ${filesize}
</pre>

<pre>
ZynqMP> dhcp
BOOTP broadcast 1
DHCP client bound to address 192.168.0.100 (1 ms)
*** Warning: no boot file name; using 'C0A80064.img'
Using ethernet@ff0b0000 device
TFTP from server 192.168.0.1; our IP address is 192.168.0.100
Filename 'C0A80064.img'.
Load address: 0x8000000
Loading: *
TFTP error: 'file /tftpboot/C0A80064.img not found for 192.168.0.100' (1)
Not retrying...
ZynqMP> tftpb 0x10000000 fpga.bit
Using ethernet@ff0b0000 device
TFTP from server 192.168.0.1; our IP address is 192.168.0.100
Filename 'fpga.bit'.
Load address: 0x10000000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
############
6.2 MiB/s
done
Bytes transferred = 7797807 (76fc2f hex)
ZynqMP> fpga loadb 0 0x10000000 ${filesize}
design filename = "CDM_XU8_top;UserID=0XFFFFFFFF;Version=2022.2"
part number = "xczu4cg-fbvb900-1-e"
date = "2024/08/14"
time = "14:18:22"
bytes in bitstream = 7797692
zynqmp_align_dma_buffer: Align buffer at 0000000010000073 to 000000000fffff80(swap 0)
ZynqMP>
</pre>

= load FPGA from Linux =

* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841847/Solution+ZynqMP+PL+Programming

== this will reset the CPU ==
<pre>
cp fpga.bit /lib/firmware/
echo fpga.bit > /sys/class/fpga_manager/fpga0/firmware
</pre>

* make .bin file:
<pre>
bootgen -image CDM_XU8_top.bif -arch zynqmp -o ./fpga.bin -w
</pre>
* cat CDM_XU8_top.bif
<pre>
dsdaqgw:ds-dm-gcdm$ cat CDM_XU8_top.bif
all:
{
[destination_device = pl] ./Vivado_CDM_XU8/CDM_XU8.runs/impl_1/CDM_XU8_top.bit
}
dsdaqgw:ds-dm-gcdm$
</pre>

== this will reset the CPU ==

<pre>
cp fpga.bin /lib/firmware/
echo fpga.bin > /sys/class/fpga_manager/fpga0/firmware
</pre>

== this will reset the CPU ==

<pre>
root@dsdm:~# ./fpgautil -b fpga.bin -f Full
</pre>

== this will reset the CPU ==

<pre>
DTSO file from here: https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841847/Solution+ZynqMP+PL+Programming
echo 0 > /sys/class/fpga_manager/fpga0/flags
mount -t configfs configfs /configfs
root@dsdm:~# dtc -O dtb -o fpga.dtbo -b 0 -@ fpga.dtso
root@dsdm:~# cp fpga.dtbo /lib/firmware/
root@dsdm:~# cp fpga.bit /lib/firmware/
root@dsdm:~# rmdir /configfs/device-tree/overlays/fpga
root@dsdm:~# mkdir /configfs/device-tree/overlays/fpga
root@dsdm:~# echo -n "fpga.dtbo" > /configfs/device-tree/overlays/fpga/path
</pre>

= fpgautil =

* https://github.com/Xilinx/meta-xilinx/blob/master/meta-xilinx-core/recipes-bsp/fpga-manager-script/files/fpgautil.c

<pre>
git clone https://github.com/Xilinx/meta-xilinx.git
cd meta-xilinx/meta-xilinx-core/recipes-bsp/fpga-manager-script/files/
scp fpgautil.c root@dsdm:
ssh root@dsdm
make fpgautil
ls -l ./fpgautil
</pre>

<pre>
root@dsdm:~# ls -l ./fpgautil
-rwxr-xr-x 1 root root 72256 Aug 16 00:15 ./fpgautil
root@dsdm:~# ./fpgautil

fpgautil: FPGA Utility for Loading/reading PL Configuration

Usage: fpgautil -b <bin file path> -o <dtbo file path>

Options: -b <binfile> (Bin file path)
-o <dtbofile> (DTBO file path)
-f <flags> Optional: <Bitstream type flags>
f := <Full | Partial >
-n <Fpga region info> FPGA Regions represent FPGA's
and partial reconfiguration
regions of FPGA's in the
Device Tree
Default: <full>
-s <secure flags> Optional: <Secure flags>
s := <AuthDDR | AuthOCM | EnUsrKey | EnDevKey | AuthEnUsrKeyDDR | AuthEnUsrKeyOCM | AuthEnDevKeyDDR | AuthEnDevKeyOCM>
-k <AesKey> Optional: <AES User Key>
-r <Readback> Optional: <file name>
Default: By default Read back contents will be stored in readback.bin file
-t Optional: <Readback Type>
0 - Configuration Register readback
1 - Configuration Data Frames readback
Default: 0 (Configuration register readback)
-R Optional: Remove overlay from a live tree

Examples:
(Load Full bitstream using Overlay)
fpgautil -b top.bit.bin -o can.dtbo -f Full -n full
(Load Partial bitstream using Overlay)
fpgautil -b rm0.bit.bin -o rm0.dtbo -f Partial -n PR0
(Load Full bitstream using sysfs interface)
fpgautil -b top.bit.bin -f Full
(Load Partial bitstream using sysfs interface)
fpgautil -b rm0.bit.bin -f Partial
(Load Authenticated bitstream through the sysfs interface)
fpgautil -b top.bit.bin -f Full -s AuthDDR
(Load Parital Encrypted Userkey bitstream using Overlay)
fpgautil -b top.bit.bin -o pl.dtbo -f Partial -s EnUsrKey -k <32byte key value>
(Read PL Configuration Registers)
fpgautil -b top.bit.bin -r
(Remove Partial Overlay)
fpgautil -R -n PR0
(Remove Full Overlay)
fpgautil -R -n full
Note: fpgautil -R is responsible for only removing the dtbo file from the livetree. it will not remove the PL logic from the FPGA region.

root@dsdm:~#
</pre>

= fw_printenv =

to access u-boot environment from Linux:
* apt install -y libubootenv-tool
* create /etc/fw_env.config
<pre>
/media/BOOT/uboot.env 0 0x40000
/media/BOOT/uboot-redund.env 0 0x40000
</pre>
* if uboot.env files do not exist, run "saveenv" from u-boot command prompt
* fw_printenv and fw_setenv should work

= Boot from network =

== u-boot ==

<pre>
ZynqMP> setenv bootcmd run bootcmd_dhcp
ZynqMP> saveenv
ZynqMP> reset
</pre>

== boot.scr ==

<pre>
# boot.scr
# mkimage -C none -A arm -T script -d boot.scr boot.scr.uimg
echo Loading FPGA!
#tftpb 0x10000000 fpga.bit
tftpb 0x10000000 {ipaddr}.bit
fpga loadb 0 0x10000000 ${filesize}
echo Booting Linux!
run bootcmd_pxe
echo Done!
</pre>

<pre>
mkimage -C none -A arm -T script -d boot.scr boot.scr.uimg
</pre>

== tftpboot ==

<pre>
cp /home/olchansk/git/ds-dm-gcdm/boot.scr.uimg /tftpboot
ln -s /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/images/linux /tftpboot/xilinx-dsdm
mkdir /tftpboot/pxelinux.cfg
cat > /tftpboot/pxelinux.cfg/default-arm-zynqmp <<EOF
LABEL Linux
KERNEL xilinx-dsdm/Image
FDT xilinx-dsdm/system.dtb
#INITRD rootfs.cpio.gz.u-boot
EOF
</pre>

== boot sequence ==

* xilinx magic load BOOT.BIN from SD card
* load FPGA form BOOT.BIN
* load and run u-boot from BOOT.BIN or from image.ub
* u-boot load environment from ??? probably SD card uboot-redund.env, this includes our bootcmd
* run bootcmd which run bootcmd_dhcp which does:
* from /tftpboot:
* load and run boot.scr.uimg which does:
* load FPGA image xilinx-dsdm/${ipaddr}.bit
* run bootcmd_pxe which does:
* load pxelinux.cfg/default-arm-zynqmp which does:
* load xilinx-dsdm/Image ### this is the linux kernel
* load xilinx-dsdm/system.dtb ### this is the device tree
* start linux kernel
* linux kernel does dhcp
* linux kernel does nfs mount /nfsroot/%s,vers=3,tcp ### %s is replaced by the hostname supplied by DHCP
* userland starts and runs to console and ssh login.

= Xilinx ILA =

References:
* https://github.com/Xilinx/XilinxVirtualCable/tree/master
* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/644579329/Xilinx+Virtual+Cable
* https://support.xilinx.com/s/article/974879?language=en_US
* https://docs.xilinx.com/v/u/en-US/pg245-debug-bridge
* https://docs.xilinx.com/r/2020.2-English/ug908-vivado-programming-debugging/Initializing-Vivado-IDE-hw_server
* (we do not use this one) https://github.com/paulscherrerinstitute/xvcSupport/blob/master/README.md

Build xvcserver_cdm.exe: (it is built as a static executable, can be copied and run anywhere)
<pre>
ssh dsdaq@gdm0
cd /home/dsdaq/online/ds-dm-software
git pull ### get latest version
make xvcserver_cdm.exe
ssh root@gdm0
/home/dsdaq/online/ds-dm-software/xvcserver_cdm.exe
INFO: To connect to this xvcServer instance, use url: TCP:gdm0:2542
</pre>

To activate and use the vivado logic analyzer:
* data path: vivado -> hw_server -> xvcserver -> mmap axi bus -> debug bridge -> jtag -> ILA
* define ILAs in the code
* instantiate the xilinx debug bridge at AXI bus address 0x80020000 (FIXME!!! this collides with Ian's AXI addresses)
* build and boot the new FPGA firmware. updating the linux kernel is not necessary.
* login root@gdm0, run: /home/dsdaq/online/ds-dm-software/xvcserver_cdm.exe -v ### with "-v" for the first time to see that vivado does connect to it, without "-v", normally.
* login dsdaqgw, run: hw_server -s tcp:localhost:3121 -e "set auto-open-servers xilinx-xvc:gdm0:2542" ### tells us to connect to port localhost:3121
* login dsdaqgw, run vivado, open project, open hardware manager, open target, open new target, "connect to remote server", hostname "localhost", port "3121", next (bombs, try again, 3 times), popup add virtual cable, enter hostname "gdm0" port "2542", "ok", it shows in "hardware targets", "next", "finish", error popup "[Common 17-163] Missing value for option 'objects', please type 'set_property -help' for usage info", ignore it, in "hardware", right click the "gdm0" one, open target, under "hardware" and "debug bridge" we should see all the ILAs, under "hardware device properties", the "probes file" should have the ".ltx" file generated by vivado "Vivado_GDM_XU8/GDM_XU8.runs/impl_1/debug_nets.ltx", click on an ILA, a waveform should open.

= Software =

* ssh cdm0 # or gdm0
* sudo apt install i2c-tools libi2c-dev
* git clone https://bitbucket.org/team-ds-dm/ds-dm-software
* cd ds-dm-software
* make

== test_cdm.exe ==

=== CDM SFP status ===

<pre>
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0xbb2f0ae7
CDM firmware 0xbb2f0ae7
arg 1: [--sfp]
Polling SFP status...
identifier 0x03
connector 0x07
encoding 0x01
wavelength 0x0352 (850 nm)
vendor_name [FINISAR CORP. ]
vendor_pn [FTLF8526P3BNL ]
vendor_rev [A ]
vendor_sn [N3AB9M8 ]
vendor_date [200319 ]
dm_type 0x68
temp 29.0 C
vcc 3.323 V
tx_bias 7.250 mA
tx_power 478.4 uW
rx_power 2.3 uW
SFP good 1, status: temp 30.6 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 818 uW
...
</pre>

=== GDM QSFP status ===

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --qsfp3 --qsfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x53aee418
CDM firmware 0x53aee418
arg 1: [--qsfp3]
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out hi
arg 2: [--qsfp]
Polling QSFP status...
identifier 0x0d
status 0x02
los 0x8f
temp 28.2 C
vcc 3.323 V
rx_power 0.1 0.1 0.1 0.1 uW
tx_bias 7.6 7.6 7.6 0.0 mA
tx_power 792.2 773.8 823.0 0.1 uW
vendor_name [FINISAR CORP ]
vendor_pn [FTL410QD4C ]
vendor_rev [A ]
wavelength 850
max_temp 70 C
vendor_sn [X79AC0R ]
vendor_date [220309 ]
QSFP good 1, status: temp 27.7 C, los 0x8b, tx_bias 7.5 7.6 7.6 0.0 mA, tx_power 792 772 821 0 uW, rx_power 0 0 466 0 uW
</pre>

=== GDM clock status ===

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
</pre>

* clock chip not loaded, not running:

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x0f 0x16 0x04 0x00 0x01 0x68 0x09 0x00 0x02 0xf2 0x00 0x1f 0xf0 0x22 0xf2, 0x507: 0x00, 0x52A: 0x00, 0x53F: 0x04
Clock chip state 0, status: SYSINCAL XAXB_ERR LOL CAL_PLL IN0 IN_SEL_0 FASTLOCK_STATUS

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x1f7caf52 (528265042) should be ~125 MHz
0x1034 rx_clk: 0x00000000 (0) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x1f7caf52 (528265042) should be ~125 MHz
0x103C tx_clk: 0x00000000 (0) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip good (IN0 - external 10 MHz clock)

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x44 0x00 0xd0 0x01 0x1f 0xff 0x22 0xf2, 0x507: 0x3f, 0x52A: 0x01, 0x53F: 0x00
Clock chip state 1, status: LOS_IN2 OOF_IN2 IN0 IN_SEL_REGCTRL IN_SEL_0

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735a3b (125000251) should be ~125 MHz
0x1034 rx_clk: 0x07735a3c (125000252) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a3b (125000251) should be ~125 MHz
0x103C tx_clk: 0x07735a3c (125000252) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

=== CDM clock status ===

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
</pre>

* clock chip not loaded, not running:

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x0f 0x16 0x04 0x00 0x01 0x68 0x19 0x00 0x02 0xf2 0x00 0x1f 0xf0 0x22 0xf2, 0x507: 0x00, 0x52A: 0x00, 0x53F: 0x04
Clock chip state 0, status: SYSINCAL XAXB_ERR LOL CAL_PLL IN0 IN_SEL_0 FASTLOCK_STATUS

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735851 (124999761) should be ~125 MHz
0x1034 rx_clk: 0x00000000 (0) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x0127fefa (19398394) should be ~125 MHz
0x103C tx_clk: 0x00000000 (0) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip uses internal clock (IN1 - internal oscillator), observe rx_clk frequency is not same as others

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0x7f, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN1 IN_SEL_1 HOLD_HIST_VALID

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735853 (124999763) should be ~125 MHz
0x1034 rx_clk: 0x07735b0a (125000458) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735853 (124999763) should be ~125 MHz
0x103C tx_clk: 0x07735852 (124999762) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip uses SFP recovered clock (IN2 - sfp rx recovered clock), observe mgt_rx_ref_clk_raw (CDM 125 MHz oscillator) is different from others (SFP RX recovered clock)

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0xbf, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN2 IN_SEL_1 HOLD_HIST_VALID
^C
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773581b (124999707) should be ~125 MHz
0x1034 rx_clk: 0x07735ad6 (125000406) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735ad7 (125000407) should be ~125 MHz
0x103C tx_clk: 0x07735ad6 (125000406) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
^C
</pre>

=== CDM link status, PRBS test mode ===

<pre>
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
</pre>
* fiber disconnected, no link:
<pre>
CDM firmware: 0xbb2f0ae7
0x1000 SFP c.c. status: 0x00000031
CLK_IN_SEL_LS 0x1
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x00000025
sfp_mod_absent_N 1
sfp_rx_los_N 0
link_power_good 1
rx_link_up 0
rx_receiving_data 0
rx_error 1
rx_lnk_up_and_running 0
tx_link_up 0
tx_sending_data 0
tx_link_up_and_running 0
link_up_and_running 0
0x1014 SFP link data: 0x466a8187
rx_data 0x8187
k28p1_k28p5 0
rx_ctrl0 0x1
rx_ctrl1 0x1
rx_ctrl3 0x1
tx_state 0x1
rx_state 0x6
rx_receiving_data 0
rx_link_up 0
rx_error 1
rx_link_rst 0
0x2000 link test mode: 0x00000200, seconds: 0x00079093, errors: 0xffffffff
</pre>
* fiber connected, good link:
<pre>
CDM firmware: 0xbb2f0ae7
0x1000 SFP c.c. status: 0x000000b2
CLK_IN_SEL_LS 0x2
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x000007dc
sfp_mod_absent_N 0
sfp_rx_los_N 0
link_power_good 1
rx_link_up 1
rx_receiving_data 1
rx_error 0
rx_lnk_up_and_running 1
tx_link_up 1
tx_sending_data 1
tx_link_up_and_running 1
link_up_and_running 1
0x1014 SFP link data: 0x35c02774
rx_data 0x2774
k28p1_k28p5 0
rx_ctrl0 0x0
rx_ctrl1 0x0
rx_ctrl3 0x0
tx_state 0x3
rx_state 0x5
rx_receiving_data 1
rx_link_up 1
rx_error 0
rx_link_rst 0
0x2000 link test mode: 0x00000200, seconds: 0x00078a8c, errors: 0x00000000
</pre>

=== GDM link status, PRBS test mode ===

<pre>
1 link connected, no errors:
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
GDM firmware: 0x6b2ee010
0x1014: 0x00000008, 18: 0x00000008, 1C: 0x00000000, 24: 0x00000fff
0x2000: 0x00000200, time: 0x00078aa4, errors:
0xffffffff 0xffffffff 0xffffffff
0x00000000 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
</pre>

=== CDM link status ===

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test0

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0xbf, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN2 IN_SEL_1 HOLD_HIST_VALID

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x077357a0 (124999584) should be ~125 MHz
0x1034 rx_clk: 0x07735a5c (125000284) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a5b (125000283) should be ~125 MHz
0x103C tx_clk: 0x07735a5b (125000283) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM firmware: 0x6d2ef81a
0x1000 SFP c.c. status: 0x000000b2
CLK_IN_SEL_LS 0x2
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x000007dc
sfp_mod_absent_N 0
sfp_rx_los_N 0
link_power_good 1
rx_link_up 1
rx_receiving_data 1
rx_error 0
rx_lnk_up_and_running 1
tx_link_up 1
tx_sending_data 1
tx_link_up_and_running 1
link_up_and_running 1
0x1014 SFP link data: 0x35c6bcbc
rx_data 0xbcbc
k28p1_k28p5 0
rx_ctrl0 0x3
rx_ctrl1 0x0
rx_ctrl3 0x0
tx_state 0x3
rx_state 0x5
rx_receiving_data 1
rx_link_up 1
rx_error 0
rx_link_rst 0
0x2000 link test mode: 0x00000000, seconds: 0x00001671, errors: 0xffffffff

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 15
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
reg[15] is 0x0033bcbc (3390652)
</pre>

=== GDM link status ===

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test0

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x44 0x00 0xd0 0x01 0x1f 0xff 0x22 0xf2, 0x507: 0x3f, 0x52A: 0x01, 0x53F: 0x02
Clock chip state 1, status: LOS_IN2 OOF_IN2 IN0 IN_SEL_REGCTRL IN_SEL_0 HOLD_HIST_VALID

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735a0a (125000202) should be ~125 MHz
0x1034 rx_clk: 0x07735a0a (125000202) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a0a (125000202) should be ~125 MHz
0x103C tx_clk: 0x07735a0a (125000202) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly

register 0x1018 bit 0x800

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM firmware: 0x6d2ebce6
0x1014: 0x00000800, 18: 0x00000800, 1C: 0x00000000, 24: 0x00000fff
0x2000: 0x00000000, time: 0x00003d2f, errors:
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff

link data alternates 0xbcbc and 0x1cbc

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 22
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
reg[22] is 0xbcbc93ab (-1128492117)

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 22
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
reg[22] is 0x1cbc1aaf (482089647)
root@gdm0:~#

</pre>

=== Run trg and tsm ===

<pre>
on the GDM:

ssh root@gdm00
./test_cdm.exe --gdm-clocks
./test_cdm.exe --load-cc
./test_cdm.exe --cc
./test_cdm.exe --reset-mgt
./test_cdm.exe --gdm-clocks
./test_cdm.exe --writereg 2 0xff # enable LEMO NIM inputs
./test_cdm.exe --writereg 3 0xba54 # LEDs: lemo1, lemo2, trg, tsm
./test_cdm.exe --writereg 4 0x99 $ # LEMO out is trg_in_pulse
./test_cdm.exe --writereg 9 0x32010 # trg and tsm from trg_pulser and tsm_pulse
./test_cdm.exe --writereg 24 1250000 # trg pulser 100 Hz
./test_cdm.exe --writereg 26 125000000 # tsm pulser 1 Hz
./test_cdm.exe --writereg 23 0x40000000 # route trg_in and tsm_in to qsfp tx bits 0 and 1
./test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz

on the CDM:

ssh root@cdm01
./test_cdm.exe --cdm-clocks
./test_cdm.exe --load-cc
./test_cdm.exe --reset-mgt
./test_cdm.exe --cdm-clocks
./test_cdm.exe --cdm-link # issue --reset-mgt on CDM and GDM until link is good
./test_cdm.exe --writereg 2 0xff # enable LEMO NIM inputs
./test_cdm.exe --writereg 3 0xba54 # enable LEDs: lemo1, lemo2, trg, tsm
./test_cdm.exe --writereg 4 0x99 # enable LEMO output trg_in
./test_cdm.exe --writereg 9 0x8040 # enable trg_in and tsm_in from sfp rx bits 0 and 1
./test_cdm.exe --writereg 7 0xff00 # drive VX LVDS lines to logic level 0
./test_cdm.exe --writereg 8 0x2 # VX LVDS with trg and tsm, misrouted in second VX
#./test_cdm.exe --writereg 8 0x7 # VX LVDS with tsm
./test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz
</pre>

switch GDM and CDM to packetizer trg and tsm:

<pre>
on the GDM:

/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x00000000

on the CDM:

/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x0804
/home/dsdaq/online/ds-dm-software/test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz
</pre>

=== Run packet loopback ===

GDM CPU -> fifo_to_fpga -> GDM QSFP -> CDM SFP -> fifo_from_fpga -> CDM CPU

On the GDM: (CDM is connected to first QSFP port)

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --load-cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --reset-mgt
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 32 1 ### tell GDM to use first QSFP port
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link ### confirm link status is "3"
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 29 2 ### enable GDM packet data injection
/home/dsdaq/online/ds-dm-software/test_cdm.exe --test-fifo-write-loop
</pre>

One the CDM:

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --load-cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --reset-mgt
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 29 3 ### enable CDM packet data injection
/home/dsdaq/online/ds-dm-software/test_cdm.exe --test-fifo-read
</pre>

= dsvslice integration =

== VX setup ==

* general
** Start acq from user code = y, all others = n (as of Mar 2023: will start when first trigger received)
** Use NIM IO = y
** Use external clock = y
** LVDS quartet is input = n, y, n, y
** LVDS quartet mode = User, User, User, User

* trigger from front panel NIM:
** Trigger on external signal = y, all others = n
** connect CDM EXT_OUT(2) to VX "TrigIn"

* trigger from LVDS "Sync" mode
** Trigger on LVDS Sync signal = y, all others = n
** LVDS quartet mode = User, Sync, User, User

* trigger from LVDS "User" mode
** Trigger on LVDS pair 12 signal = y, all others = n
** LVDS quartet is input = n, y, n, y
** LVDS quartet mode = User, User, User, User

== GDM setup ==

* GDM is gdm0
* set inputs to NIM mode
* set outputs to TTL mode (this GDM has wrong NIM output circuit)
* use top QSFP slot, connect split cable 1 into CDM01, cable 2 into CDM02
* connect non-inverted NIM trigger signal to top-LEMO-left EXT_IN_LV(1)
* connect non-inverted NIM TSM signal to top-LEME-right EXT_IN_LV(2)
* GDM LEDs: TRIG, TSM, trigger enabled, trigger_out
* GDM LEMO_OUT: trigger, trigger

== CDM setup ==

* set CDM LEMO inputs to NIM
* set CDM LEMO outputs to NIM
* CDM01 is cdm0
* CDM02 is cdm1
* connect GDM fiber links to SFP port
* connect 1st VX port of CDM01 to VX1
* connect 1st VX port of CDM02 to VX2
* connect LEMO EXT_OUT(2) to VX "TrigIn", CDM01 to VX1, CDM02 to VX2
* power up
* CDM LEDs: GDM TRIG, GDM TSM, trigger enabled, trigger out
* CDM LEMO_OUT: gdm_trg, trigger

== After power up ==

* start the CDM frontend from the MIDAS "Programs" page. To start manually, see the Start Command on the Programs page.
* CDM frontend should enable the VX clock, disable the trigger
* from the MIDAS status page, goto the CDM page
* outdated: in the CDMx data tables, the 2nd number should read 0x35c08008, if it does not and the last 4 digits randomly change, reset the GDM links
*if the FEs complain - do in order: for GDM, CDM01, CDM02, ..., press "reset mgt" of each board, then press "unreset mgt", if it does not help, STOP HERE
* start a run
* CDM frontend will enable the trigger
* GDM frontend will enable the trigger
* LEDs on the GDM should flash, LEDs on the CDM should flash, TrigIn and TrigOut of the VX should flash
* stop a run
* GDM frontend will disable the trigger
* CDM frontend will disable the trigger

== Phase measurement ==

* pip3 install matplotlib
* pip3 install scipy
* export PYTHONPATH=$HOME/packages/midas/python
* #git clone https://github.com/J033X071C/PhaseMeasurement
* git clone https://bitbucket.org/team-ds-dm/phasemeasurement.git
* cd phasemeasurement
* python3 ./phaseMeasurement.py --help
<pre>
daq00:PhaseMeasurement$ python3 ./phaseMeasurement.py --help
usage: phaseMeasurement.py [-h] fileName numberEvents numberVX sizeEvents stopEvent minHist maxHist numberBin writeToTXT saveAsPDF

Read data from midas file (in .lz4 format) to calculate phase between the clock of VX1 and VX2

positional arguments:
fileName Name of the file we want to read data from (Example: run00389.mid.lz4)
numberEvents Number of events recorded in the file
numberVX Number of VX used in this run (usually 2...)
sizeEvents Number of points per event
stopEvent Number of events you want to go through to calculate phase
minHist Minimal value for the x axis of the phase measurement histogram (in ns)
maxHist Maximal value for the x axis of the phase measurement histogram (in ns)
numberBin Number of bins wanted for the generated histogram
writeToTXT Write argument as yes to generate text file with results of calculation
saveAsPDF Save generated plots to PDF files

optional arguments:
-h, --help show this help message and exit
daq00:PhaseMeasurement$
</pre>
* try an old file with
* python3 ./phaseMeasurement.py run00877.mid.lz4 10000 2 10000 500 -20 20 81 yes yes
<pre>
bin size = 0.494 ns
num_events = 347
mean = -1.705 ns
rms = 3.087 ns
mean_error = 0.166 ns
centroid = -1.706 ns.
width (sigma) = 0.363 ns.
error on the centroid = 0.016558 ns.
</pre>
* ls -l *.txt *.pdf
<pre>
dsdaq@dsvslice:~/online/PhaseMeasurement$ ls -l *.txt *.pdf
-rw-rw-r-- 1 dsdaq dsdaq 64728 Dec 14 16:56 run00877.mid.lz4_Plots.pdf
-rw-rw-r-- 1 dsdaq dsdaq 274 Dec 14 16:56 run00877.mid.lz4.txt
dsdaq@dsvslice:~/online/PhaseMeasurement$
</pre>
* scope settings (from email message)
<pre>
From fcote-lortie@triumf.ca Thu Dec 15 17:11:46 2022
From: Francis Cote-Lortie <fcote-lortie@triumf.ca>
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: Re: How to use scope as a waveform generator
Date: Fri, 16 Dec 2022 01:11:44 +0000

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display) by pressing on the button. It will go from 0 to 1.

The settings that we are using right now are:
Type of waveform: Sine wave
Offset: 0 V
Amplitude: 1 Vpp
Frequency: 50 kHz
Noise: 0 V
________________________________
From: Francis Cote-Lortie <fcote-lortie@triumf.ca>
Sent: Thursday, December 15, 2022 4:58 PM
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: Re: How to use scope as a waveform generator

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display)

The settings that we are using right now are:
Type of waveform: Sine wave
Offset: 0 V
Amplitude: 1 Vpp
Frequency: 50 kHz
Noise: 0 V
________________________________
From: Francis Cote-Lortie
Sent: Thursday, December 15, 2022 4:54 PM
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: How to use scope as a waveform generator

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display)
</pre>

== Standalone link test ==

<pre>
CDM: program clock chip
busybox devmem 0x80011000 32 0x8
busybox devmem 0x80011000 32 0x0
/home/dsdaq/si5394-i2c-file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b

GDM, CDM: link reset
busybox devmem 0x80011008 32 1

GDM, CDM: release reset
busybox devmem 0x80011008 32 0

CDM: link status (NOTE: SFP LOS and mod_absent are swapped!!!)
busybox devmem 0x80011010 32
0x00000024 <- fiber plugged
0x00000025 <- fiber unplugged
0x00000027 <- SFP unplugged
0x000007DC <- successful link with GDM

CDM: link state machine and data
busybox devmem 0x80011014 32
0x35C06FF6

CDM: set link to counting mode
busybox devmem 0x80012000 32 0x101
busybox devmem 0x80012000 32 0x100

CDM: time counter and error counter
root@cdm1:~# busybox devmem 0x80012000 32
0x00000100 <--- link mode
root@cdm1:~# busybox devmem 0x80012004 32
0x0000058C <--- seconds counter
root@cdm1:~# busybox devmem 0x80012004 32
0x0000058D
root@cdm1:~# busybox devmem 0x80012008 32
0x00000000 <--- error counter

GDM: no link
root@gdm0:~# busybox devmem 0x80011014 32
0x00000000
root@gdm0:~# busybox devmem 0x80011018 32
0x00000000
root@gdm0:~# busybox devmem 0x8001101c 32
0x00000000
root@gdm0:~# busybox devmem 0x80011024 32
0x00000FFF
root@gdm0:~#

GDM: good link channel 10, counting mode
root@gdm0:~# busybox devmem 0x80012000 32 0x101
root@gdm0:~# busybox devmem 0x80012000 32 0x100
root@gdm0:~# busybox devmem 0x80012008 32
0x3A8B68C2
root@gdm0:~# busybox devmem 0x80012008 32
0x42E03BEF
root@gdm0:~# busybox devmem 0x8001200c 32
0xDA090972
root@gdm0:~# busybox devmem 0x8001200c 32
0xDE6F22E9
root@gdm0:~# busybox devmem 0x80012019 32
Bus error
root@gdm0:~# busybox devmem 0x80012010 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012014 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012018 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x8001201c 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012020 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012024 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012028 32
0x00000000
root@gdm0:~# busybox devmem 0x8001202c 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012030 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012034 32
0xFFFFFFFF
root@gdm0:~#
</pre>

mapping of link channels:

<pre>
qsfp0 lane0 - 0x0100 - link 8
qsfp0 lane1 - 0x0200 - link 9
qsfp0 lane2 - 0x0400 - link 10
qsfp0 lane3 - n/c
qsfp1 lane0 - 0x0800 - link 11
qsfp1 lane1 - 0x0010 - link 4
qsfp1 lane2 - 0x0020 - link 5
qsfp1 lane3 - n/c
qsfp2 lane0 - 0x0040 - link 6
qsfp2 lane1 - 0x0080 - link 7
qsfp2 lane2 - 0x0001 - link 0
qsfp2 lane3 - n/c
qsfp3 lane0 - 0x0002 - link 1
qsfp3 lane1 - 0x0004 - link 2
qsfp3 lane2 - 0x0008 - link 3
qsfp3 lane3 - n/c
</pre>

script to start the test with 2 CDMs:

<pre>
ssh dsdaq@dsvslice
ssh root@gdm0 busybox devmem 0x80011008 32 1
ssh root@cdm0 busybox devmem 0x80011008 32 1
ssh root@cdm1 busybox devmem 0x80011008 32 1
ssh root@gdm0 busybox devmem 0x80011008 32 0
ssh root@cdm1 busybox devmem 0x80011008 32 0
ssh root@cdm0 busybox devmem 0x80011008 32 0
ssh root@gdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
ssh root@cdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
ssh root@cdm1 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
</pre>

== LEMO trigger GDM to CDM to VX ==

on the GDM:

<pre>
/home/dsdaq/si5394-i2c-file /home/dsdaq/GDM_v1.0_IN0_EXT1_and_IN1_fixed_Si5394-RevA-Registers.txt 0 0x6b
busybox devmem 0x80011008 32 0x1
busybox devmem 0x80011008 32 0x0
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x80001230 # QSFP TX fixed pattern
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 2 0xff # enable LEMO inputs
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 3 0x7654 # enable LED, one per LEMO input
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x0F0F # enable LEMO to trg_in and tsm_in
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x40001230 # enable trg_in and tsm_in output to QSFP
</pre>

on the CDM:

<pre>
/home/dsdaq/si5394-i2c-file /home/dsdaq/CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
/home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp
busybox devmem 0x80011008 32 1
busybox devmem 0x80011008 32 0
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x8040 # trg_in from sfp[0], tsm_in form sfp[1]
/home/dsdaq/online/ds-dm-software/test_cdm.exe 11 # trg_in counter
/home/dsdaq/online/ds-dm-software/test_cdm.exe 12 # tsm_in counter
</pre>

== GPS receiver VCL-2705 ==

* Valiant VCL-2705 GPS receiver
* https://www.valiantcom.com/time-distribution/gps-receiver-irig-b.html
* USB connection is /dev/ttyACM0, 115200 bps, "GNSSAUX" prompt
* minicom -D /dev/ttyACM0 -b 115200
* user manual download instructoins - see sheet of paper with user name and password in the shipping box
* usb commands:
<pre>
gnss-help
gnss-showver -> F/W: Ver 1.7 Feb 17 2020 16:20:43

gnss-showselftest -> no antenna connected
Overall : FAIL
EPROM Test: PASS
Antenna : NOT DETECTED
GNSS : COMMUNICATION OK

gnss-showsettings
GNSS NMEA BAUDRATE :115200
GNSS ANTENNA LENGTH :30 meters
GNSS USER CONFIGURED DELAY:-65 nanoseconds
GNSS 1PPS PULSE WIDTH :200 milliseconds
GNSS MODE :GPS
GNSS STATUS :STATIONARY

GNSSAUX> gnss-showserial
SERIAL :2704H01V17MAX310

GNSSAUX> gnss-showmode
GNSS MODE : GPS

GNSSAUX> gnss-showstatus
GNSS STATUS: STATIONARY

GNSSAUX> gnss-showalarms
CURRENT ALARMS GNSS
ANTENNA : NOT DETECTED

GNSSAUX> gnss-showerrors
CURRENT ALARMS GNSS
ANTENNA : **NOT DETECTED
ERROR STATISTICS GNSS
RMC GOOD DURATION : Secs 0
RMC BAD DURATION : Secs 0
LOCK GOOD SECS DURATION : Secs 0
LOCK BAD SECS DURATION : Secs 0
SATINFO GOOD ITERATIONS :0
SATINFO REJECT ITERATIONS :0
SATINFO NOTALKER ITERATIONS :0
CURRENT MONITOR STATE GNSS
STATE :Phase-1 HUNTING ANTENNA DETECT

GNSSAUX> gnss-showsatinfo
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-showsats
Total Sats: 0

GNSSAUX> gnss-showmyloc
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-show1ppsstate
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-showjamstatus
Not Available !

GNSSAUX> gnss-showspoofstatus
Not Available !

--- antenna connected, can see the sky ---

GNSSAUX> gnss-showselftest
Overall : PASS
EPROM Test: PASS
Antenna : DETECTED
GNSS : COMMUNICATION OK

GNSSAUX> gnss-showalarms
CURRENT ALARMS GNSS
ANTENNA : DETECTED
GNSS LOCK : AVAILABLE

GNSSAUX> gnss-showerrors
CURRENT ALARMS GNSS
ANTENNA : DETECTED
GNSS LOCK : AVAILABLE
ERROR STATISTICS GNSS
RMC GOOD DURATION : Mins 1,Secs 34
RMC BAD DURATION : Secs 55
LOCK GOOD SECS DURATION : Mins 1,Secs 34
LOCK BAD SECS DURATION : Secs 55
SATINFO GOOD ITERATIONS :3
SATINFO REJECT ITERATIONS :0
SATINFO NOTALKER ITERATIONS :0
CURRENT MONITOR STATE GNSS
STATE :Phase-4 NORMAL OPERATION, Monitoring GNSSLOCK

GNSSAUX> gnss-showsatinfo
GNSS MODE :GPS
SATELLITE INFORMATION TALKER:GPS
NMEA ID : 01-32
NO OF SATELLITES IN VIEW: 08
NO OF XXGSV MSGS : 03
SatNo PRN NO (SV ID) ELEVATION (degs) AZIMUTH (degs) C/No (SNR)
1 0 0 0 0
2 0 0 0 0
3 0 0 0 0
4 0 0 0 0
5 0 0 0 0
6 0 0 0 0
7 0 0 0 0
8 0 0 0 0
<<<<<< End of Sat Info >>>>>>

GNSSAUX> gnss-showsats
Total Sats: 8

GNSSAUX> gnss-showmyloc
GNSS RECEIVER LOCATION:
Latitude : 4914.81911
Longitude: 12313.69595

GNSSAUX> gnss-show1ppsstate
GPS 1PPS STATE: LOCKED

GNSSAUX> gnss-showmyloc

GNSS RECEIVER LOCATION:
Latitude : 4914.80688
Longitude: 12313.69531

enter into google maps search box as: 49 14.80688, -123 13.69531, observe the space, the moved dot and the minus.

GNSSAUX> gnss-resetgnss

Executing....Please Wait....
$$$$$END
</pre>

* 1PPS BNC output: period 1 sec, pulse width 200 ms, 3.3V into 1MOhm, 1.38V into 50Ohm.
* IRIG-B BNC output: 5.6V into 1MOhm, 2.4V into 50Ohm.
* IRIG-B format selector: default is all up.

== Rb clock PRS10 ==

* https://www.thinksrs.com/products/prs10.html
* 10 MHz output is sine wave around 5V peak to peak
* 1PPS BNC output is 10 usec pulse, 5V into 1MHohm.
* RS232 connection: minicom -D /dev/ttyUSB0 -b 9600
* ser2net config: localhost,3001:raw:600:usb-5-2-1.0:9600 -XONXOFF -RTSCTS LOCAL
* ssh daq13, cd /home/olchansk/git/ser2net, ./ser2net -c ~/daq/ds/ser2net.conf -d
* ssh daq13, cd ~/daq/ds, python3 prs10.py
* RS232 commands:
<pre>
ID?
PRS10_3.56_SN_105719
VB1
SN?
RS1 -- reset
ST? -- status
FC? -- 10MHz OCXO drive voltage DAC settings
DS? -- "detected signals"
GA? -- gain of frequency lock loop between ovenized oscillator and Rb cell, 0=use ovenized oscillator only
MO? -- magnetic offset of the Rb cell, range 2300..3600, if out of range, unit must be set to different operating mode, see prs10m.pdf
MR? -- magnetic read
TT? -- time-tag, time in ns between 1PPS out and 1PPS in
TS? -- time slope, ???
TO? -- time offset, ???
PS? -- pulse slope, ???
PL? -- 0=phase lock off, 1=phase lock on, lock to 1PPS input
PT? -- phase lock integrator time constant, PT8 is integrator time constant 18.2 hours, natural time constant 2.25 hours
PF? -- phase lock stability factor, PF2 is "1"
PI? -- phase lock integrator

Analog to digital 12 bit ADC, values 0.000 to 4.998

AD0? -- Spare (J204)
AD1? -- +24V(heater supply) divided by 10.
AD2? -- +24V(electronics supply) divided by 10
AD3? -- Drain voltage to lamp FET divided by 10
AD4? -- Gate voltage to lamp FET divided by 10
AD5? -- Crystal heater control voltage
AD6? -- Resonance cell heater control voltage
AD7? -- Discharge lamp heater control voltage
AD8? -- Amplified ac photosignal
AD9? -- Photocell’s I/V converter voltage divided by 4
AD10? -- Case temperature (10 mV/°C)
AD11? -- Crystal thermistors
AD12? -- Cell thermistors
AD13? -- Lamp thermistors
AD14? -- Frequency calibration pot / external calibration voltage
AD15? -- Analog ground

A/D via CPU E-port:

AD16? -- Varactor voltage for 22.48 MHz VCXO (inside RF synthesizer) / 4
AD17? -- Varactor voltage for 360 MHz VCO (output of RF synthesizer) / 4
AD18? -- Gain control voltage for amplifier which drives frequency multiplier / 4
AD19? -- RF synthesizer’s lock indicator voltage (nominally 4.8 V when locked )

ST?

ST1 : Power supplies and Discharge Lamp
ST1 bit, Condition which sets bit, Corrective Action
0 -- +24 for electronic < +22 Vdc
1 -- +24 for electronics > +30 Vdc
2 -- +24 for heaters <+22 Vdc
3 -- +24 for heaters > +30 Vdc
4 -- Lamp light level too low
5 -- Lamp light level too high
6 -- Gate voltage too low
7 -- Gate voltage too high

ST2: RF Synthesizer
ST2 bit, Condition which sets bit, Corrective Action
0 -- RF synthesizer PLL unlocked
1 -- RF crystal varactor too low
2 -- RF crystal varactor too high
3 -- RF VCO control too low
4 -- RF VCO control too high
5 -- RF AGC control too low
6 -- RF AGC control too high
7 -- Bad PLL parameter

ST3: Temperature Controllers
ST3 bit, Condition which sets bit
0 -- Lamp temp below set point
1 -- Lamp temp above set point
2 -- Crystal temp below set point
3 -- Crystal temp above set point
4 -- Cell temp below set point
5 -- Cell temp above set point
6 -- Case temperature too low
7 -- Case temperature too high

ST4: Frequency Lock-Loop Control
ST4 bit, Condition which sets bit
0 -- Frequency lock control is off
1 -- Frequency lock is disabled
2 -- 10 MHz EFC is too high
3 -- 10 MHz EFC is too low
4 -- Analog cal voltage > 4.9 V
5 -- Analog cal voltage < 0.1
6 -- not used
7 -- not used

ST5: Frequency Lock to External 1pps
ST5 bit, Condition which sets bit
0 -- PLL disabled
1 -- < 256 good 1pps inputs
2 -- PLL active
3 -- > 256 bad 1pps inputs
4 -- Excessive time interval
5 -- PLL restarted
6 -- f control saturated
7 -- No 1pps input

ST6: System Level Events
ST6 bit and Condition which sets bit
0 Lamp restart
1 Watchdog time-out and reset
2 Bad interrupt vector
3 EEPROM write failure
4 EEPROM data corruption
5 Bad command syntax
6 Bad command parameter
7 Unit has been reset
</pre>
* ST? on warm start
<pre>
received: PRS_10
received: 255,255,255,243,34,255
received: 0,0,0,1,34,0
...
received: 0,0,0,1,34,0
received: 0,0,0,0,2,0
...
received: 0,0,0,0,2,0
received: 0,0,0,0,4,0
</pre>
* ST? on loss of external 1PPS
<pre>
...
received: 0,0,0,0,4,0
disconnect 1PPS input
received: 0,0,0,0,132,0
...
reconnect 1PPS input
received: 0,0,0,0,132,0
received: 0,0,0,0,4,0
...
</pre>
* ST? on coldish start
<pre>
daq13:ds$ python3 prs10.py
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,0,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 0,0,1,1,34,0 ] old [ 80,0,0,1,34,1 ] counter: 0
received [ 0,0,0,1,34,0 ] old [ 0,0,1,1,34,0 ] counter: 0
received [ 0,0,16,1,34,0 ] old [ 0,0,0,1,34,0 ] counter: 6
received [ 0,0,20,1,34,0 ] old [ 0,0,16,1,34,0 ] counter: 5
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 8
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 3
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 49
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 250
...
</pre>
* ST? on cold start, note: no bump in the 10MHz clock as reported by DS-DM clock chip
<pre>
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,21,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 64,0,21,1,34,0 ] old [ 80,0,21,1,34,1 ] counter: 0
received [ 0,0,21,1,34,0 ] old [ 64,0,21,1,34,0 ] counter: 0
received [ 0,0,20,1,34,0 ] old [ 0,0,21,1,34,0 ] counter: 158
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 23
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 47
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 40
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 249
</pre>

= det fac integration test =

Connections:
* GPS receiver "IRIG-B SEL" both switches "up" - both "on", IRIG-B format B004
* GPS receiver USB-B -> long cable -> daq13 USB-A
* GPS receiver "1PPS out" -> long BNC cable -> BNC-T -> scope (5V, no 50ohm) and Rb clock BNC "1PPS in"
* GPS receiver "IRIG-B 50ohms" -> long BNC cable -> BNC-T -> scope (5V, no 50ohm) and DS-DM LEMO input 1 (TTL mode)
* Rb clock RS232 -> RS232 straight cable -> RS232-to-USB adapter -> daq13 USB-A
* Rb clock "1PPS out" BNC -> scope (5V, no 50ohm, trig threshold rising edge 2V)
* Rb clock "10MHz output 50 Ohm" BNC -> lemo -> lemo-T -> scope (sine wave, 5V, no 50ohm) and DS-DM clock input.

Programs to run:
* on daq13: cd /home/olchansk/git/ser2net, ./ser2net -c ~/daq/ds/ser2net.conf -d
<pre>
localhost,3001:raw:600:usb-5-2-1.0:9600 -XONXOFF -RTSCTS LOCAL
</pre>
* on daq13: cd /home/olchansk/daq/ds, python3 prs10.py ### connects to ser2net
<pre>
daq13:ds$ python3 prs10.py
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,21,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 64,0,21,1,34,0 ] old [ 80,0,21,1,34,1 ] counter: 0
received [ 0,0,21,1,34,0 ] old [ 64,0,21,1,34,0 ] counter: 0
received [ 0,0,20,1,34,0 ] old [ 0,0,21,1,34,0 ] counter: 158
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 23
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 47
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 40
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 249
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 12096
received [ 0,0,0,0,20,0 ] old [ 0,0,0,0,132,0 ] counter: 2
received [ 0,0,0,0,148,0 ] old [ 0,0,0,0,20,0 ] counter: 0
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,148,0 ] counter: 575
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 11449
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 2755
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 34386
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 41035
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 113401
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 33375
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 54767
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 85059
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 33222
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 119234
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 121990
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 128184
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 56002
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 428237
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 8250
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 1388
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 30506
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 142704
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 179451
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 106182
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 68747
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 65424
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 157587
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 6932
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 1388
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 20255
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 4
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 225941
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 72183
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 26970
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 1
39287
</pre>
* on ds-dm: ./test_cdm.exe --irigb ### note sbs mismatch is because I should wrap around at 16 bits
<pre>
dataframe: S00000000S100001010S...S001010101S000001000S, sec: 00, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16724 should be 82260
dataframe: S10000000S100001010S...S101010101S000001000S, sec: 01, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16725 should be 82261
dataframe: S01000000S100001010S...S011010101S000001000S, sec: 02, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16726 should be 82262
dataframe: S11000000S100001010S...S111010101S000001000S, sec: 03, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16727 should be 82263
dataframe: S00100000S100001010S...S000110101S000001000S, sec: 04, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16728 should be 82264
dataframe: S10100000S100001010S...S100110101S000001000S, sec: 05, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16729 should be 82265
</pre>

= DS-IOGC GPS interface board =

* Rev0 git repository: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0/-/tree/main?ref_type=heads
* Rev0 schematics: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0/-/blob/main/Altium/Project%20Outputs%20for%20DS-IOGC-Rev0/SCH-DS-IOGC-Rev0.pdf?ref_type=heads
* Rev0 schematics: [[:Image:SCH-DS-IOGC-Rev0.pdf|SCH-DS-IOGC-Rev0]]
* Rev1 git repository: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev1/
* Rev1 schematics: [[:Image:SCH-DS-IOGC-Rev1.pdf|SCH-DS-IOGC-Rev1]]

== Changes Rev0 to Rev1 ==

<pre>
From: Peter Margetak <pmargetak@triumf.ca>
Subject: IOGC REV1 review
Date: Wed, 4 Sep 2024 07:31:19 +0000

Hi Konstantin,
Pls have a look at SCH for new rev. I'd like to send it to mfr next week so if you can comment by early next week. Meanwhile I work on layout and other stuff.

Changes:

New ICs - all powered +5V
U20 - inverters for RUclk RX/TX
U21 - non inverting line driver for RU-1pps-out (so you don't have to route if via GDM to see it on scope)
U22 - non inverting buffer for ext 1pps input

All Lemo connectors have the same position but they are double lemos now => new panel needed
@Marek Walczak<mailto:mwalczak@triumf.ca> you can print it ahead once pcb is done + update IOGC docs and panel description

J2A/B - Test ports for RU-1pps in and out
J5A/B - inputs for external GPS data and External source of 1pps
J6A/B - aux in/out for GDM

SW1 - no change - select RX/TX USB/GDM
SW2 - select latch sensitivity for rising/falling edge
SW3 - select source of GPS data (opto or ext) AND select source of 1pps input (latch or ext)

p.
</pre>

== PRS-10 Rb clock device ==

The Rb clock PRS-10 device provides these connections:
<pre>
RS232 RX input - serial communication, non-standard RS232
RS232 TX output - serial communication, non-standard RS232
10 MHz clock output - coax 50 Ohm high resolution 10 MHz clock
1pps output - 1 Hz clock corresponding to the 10 MHz clock
1pps input - 1pps signal from GPS receiver
</pre>

Mode of operation:
* 10 MHz clock is always running
* 1pps output is always running
* if 1pps input from GPS received is present, after 256 pulses PRS-10 will sync it's 1pps output with the 1pps input by adjusting the frequency of the 10 MHz clock
* when unlocked: 1pps output and 1pps input unrelated
* when locked to GPS: 1pps output and 1pps input always go up and down at the same time

Theory of operation:
* 10 MHz clock is produced by a high-quality crystal (stable on the scale of seconds)
* crystal oscillator is synchronized to a Rb cell (stable on the scale of hours and days)
* Rb cell resonant frequency is synchronized to the GPS 1pps signal (stable on the scale of months and years)

== Rev1 connections ==

* LEMO connectors (front panel)
<pre>
LEMO J2A output - Rb clock 1pps in monitor
LEMO J2B output - Rb clock 1pps out monitor
LEMO J5A input - GPS IRIG-B from GPS receiver to FPGA (VCL-2705)
LEMO J5B input - GPS 1pps from GPS receiver to PRS-10 (VCL-2705)
LEMO J6A input - AUX-IN to FPGA
LEMO J6B output - AUX-OUT from FPGA
</pre>

* SMB connectors (back)
<pre>
SMB J3 output - GPS 1pps loopback to LNGS
SMB J4 input - LNGS GPS data input
</pre>

* LEDs
<pre>
D1 - same as LEMO J2A out (Rb clock 1pps in)
D2 - same as SMB J3 out (GPS 1pps from LNGS or from a GPS receiver)
D5 - controlled by FPGA-OUT-LED1
D6 - controlled by FPGA-OUT-LED2
D7 - PRS-10 24V power ok
</pre>

* switches
<pre>
SW1A and SW1B - route PRS-10 RS232 to USB or to FPGA
SW2A - route PRS-10 1pps input from SMB J4 (LNGS) or from LEMO J5B (GPS receiver 1pps)
SW2B - route FPGA-IN-GPSDATA input from SMB J4 (LNGS) or from LEMO J5A (GPS receiver IRIG-B data)
SW3 - LNGS 1pps from rising edge or from falling edge of SMB J4 (LNGS)
</pre>

== Rb clock cable ==
<pre>
Rb clock ----- DS-IOGC side, pin numbers are as labeled on the cable connectors

1 - 1pps out - 8 - 1pps out --- correct
2 - nc
3 - nc
4 - TXD - 3 - RU-DATA-OUT - USB-RX input --- correct
5 - 1pps in - 2 - 1pps in --- correct
6 - +24V - 10 - +24V --- should by pin 1 to use both +24V pins?
7 - RXD - 15 - RU-DATA-IN - USB-TX output --- correct
8 - nc
9 - +24V - 10 - +24V --- correct
10 - GND - 9 - GND --- correct
</pre>

== VX connections ==
<pre>
VXA_TX0 - FPGA-OUT-LED2 - D6 LED ("10 MHz clock")
VXA_TX1 - FPGA-OUT-LED1 - D5 LED ("GPS DATA")
VXA_TX2 - FPGA-OUT-RU1PPS-EN - enable 1pps to Rb clock
VXA_TX3 - not used (62.5 Hz clock)

VXB_TX0 - FPGA-TX - PRS-10 RS-232 out
VXB_TX1 - FPGA-OUT-AUX - 5V TTL J5 LEMO out
VXB_TX2 - FPGA-OUT-OPTO1PPS-EN - PRS-10 power enable
VXB_TX3 - not used (62.5 MHz clock)

VXA_RX0 - FPGA-IN-OPTO1PPS - 1pps from GPS
VXA_RX1 - FPGA-IN-GPSDATA - J4 GPS data
VXA_RX2 - n/c
VXA_RX3 - n/c

VXB_RX0 - n/c
VXB_RX1 - FPGA-IN-AUX - J5 LEMO TTL input
VXB_RX2 - FPGA-RX - PRS-10 RS-232 in
VXB_RX3 - FPGA-IN_RU1PPS - PRS-10 1pps output
</pre>

== test sequence ==

* ./test_cdm_local.exe --writereg 7 0 ### clear reg 7
* ./test_cdm_local.exe --writereg 8 0 ### set vx_tx mux to vx_tx control from reg 7
* connect blue cable to GDM port 6 (next to the ethernet connector)
* ./test_cdm_local.exe --writereg 7 0x4000 ### power up
* ./test_cdm_local.exe --writereg 7 0x4100 ### right LED D6
* ./test_cdm_local.exe --writereg 7 0x4200 ### left LED D5
* ./test_cdm_local.exe --writereg 7 0x6000 ### J5 LEMO measure +5VDC
* install LEMO jumper between LEMO J5 and J6
* write 0x4000 then ./test_cdm_local.exe 6 ### reads 0x5C, bit 5 0x20 reads 0
* write 0x6000 then ./test_cdm_local.exe 6 ### reads 0x7C, bit 5 0x20 reads 1
* write 0x4000 to clear all bits
* 10 MHz output ok - IN0 is good: ./test_cdm_local.exe --cc
<pre>
Clock chip state 1, status: OOF_IN2 IN1 IN_SEL_1 HOLD_HIST_VALID
</pre>
* without 10 MHz reports IN0 LOS (no signal) and OOF (wrong frequency)
<pre>
Clock chip state 1, status: LOS_IN0 OOF_IN0 OOF_IN2 IN1 IN_SEL_1 HOLD_HIST_VALID
</pre>
* IRIG-B via AUX-IN is ok: ./test_cdm_local.exe --irigb
<pre>
dataframe: S00100110S111001000S...S011110000S010000000S, sec: 34, min: 17, hrs: 00, day of year: 223, year: 2024, date: 10 aug, sbs: 1054 should be 1054
</pre>
* ./test_cdm_local.exe --writereg 8 9 ### set vx_tx mux to GPS control, hard enables PRS-10 power and PRS-10 1pps in
* observe 1pps counters in reg 68 count at 1 Hz: rb_1pps 0x51->0x52, gps_1pps 0xd5->oxd6
<pre>
root@dsdm:~# ./test_cdm_local.exe 68
ds20k_reg[68] is 0x0051d520 (5362976)
ds20k_reg[68] is 0x0052d624 (5428772)
...
</pre>
* observe PRS-10 can see the 1pps signal "130" changes to "2" after 243 seconds to "4"
* observe period of GPS and PRS-10 1pps is identical (plus/minus 1 clock)
<pre>
root@dsdm:~# ./test_cdm_local.exe 13 14
ds20k_reg[13] is 0x077356d4 (124999380)
ds20k_reg[14] is 0x077356d4 (124999380)
</pre>
* observe GDM/CDM clocks are exactly 125 MHz, we are running on DS-DM internal oscillator
<pre>
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735943 (125000003) should be ~125 MHz
0x1034 rx_clk: 0x07735943 (125000003) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735943 (125000003) should be ~125 MHz
0x103C tx_clk: 0x07735943 (125000003) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>
* switch CC to external clock:
<pre>
root@dsdm:~# ./test_cdm_local.exe --cc-in0
CC use clock input 0: 10 MHz LEMO external clock
root@dsdm:~# ./test_cdm_local.exe --cc
Polling CC status...
Clock chip state 1, status: IN0 IN_SEL_REGCTRL IN_SEL_0 HOLD_HIST_VALID
</pre>
* observe rx_clk and tx_clk are now slightly off: they run from PRS-10 10 MHz clock and are measured against the DS-DM internal oscillator.
<pre>
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x94b12519, ds20k version 0x20240814
DS-DM firmware build 0x94b12519, ds20k version 0x20240814
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x077358e1 (124999905) should be ~125 MHz
0x1034 rx_clk: 0x07735b49 (125000521) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735b49 (125000521) should be ~125 MHz
0x103C tx_clk: 0x07735b49 (125000521) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>
* observe GPS and PRS-10 1pps period is now exactly 125*10^6 of 8ns clocks (equal to 1 second)
<pre>
root@dsdm:~# ./test_cdm_local.exe 13 14
ds20k_reg[13] is 0x0773593f (124999999)
ds20k_reg[14] is 0x0773593f (124999999)
</pre>
* look at them repeatedly, observe reg 13 "GPS 1pps period" has some wobble, reg 14 "Rb clock 1pps period" is steady. this is as expected: if GPS 1pps was steady, we do not need to Rb clock. PRS-10 1pps is derived from the PRS-10 10 MHz clock and is measured against the 125 MHz clock derived from the same 10 MHz clock in the GDM FPGA.

Test status:

* GPS 1pps to SMB-in ok (LED flashes)
* GPS 1pps to FPGA ok
* GPS data to FPGA ok
* GPS 1pps to PRS-10 enabled from FPGA ok
* PRS-10 Rb clock 1pps out to FPGA ok
* PRS-10 Rb clock 1pps out to FPGA to LEMO AUX out ok
* can see GPS 1pps, IRIG-B, PRS-10 1pps out, 10 MHz on the scope, ok
* PRS-10 syncs on leading edge (0->1) of GPS 1pps signal, ok
* reg 13 and 14 1pps periods are identical, ok
* CC locks on PRS-10 10 MHz clock, DS-DM runs on PRS-10 clock, ok
* NOT TESTED - smb output
* NOT TESTED - optical converter fiber to SMB
* NOT TESTED - optical converter SMB to fiber
* NOT TESTED - SMB loopback
* NOT TESTED - fiber loopback

= VX busy logic =

= DS-20K DAQ =

== Overview ==

DS-DM, GDM and CDM are key parts of the DS-20K DAQ system:
* common clock distribution from external clock (atomic clock, GPS) to GDM to per-quadrant CDMs to VX digitizers
* common trigger distribution from GDM internal algorithm or external input to all VX digitizers
* run control: GDM, CDM, VX all start recording data at the same time (clock and timestamp reset)
* collection of trigger data from VX digitizers to per-quadrant CDMs to GDM

== Deliverables ==

* hardware and firmware for GDM to CDM clock distribution
* hardware and firmware for CDM to VX clock distribution
* hardware and firmware for GDM external clock input (atomic clock or GPS)
* hardware and firmware for CDM and VX serial communications (VX LVDS I/O connector)
* firmware for run control (timestamp reset and sync): GDM to CDM to VX
* firmware for common trigger distribution: GDM to CDM to VX
* firmware for trigger data flow: VX to CDM to GDM
* firmware for busy control: VX to CDM to GDM back to CDM to VX
* firmware for flow control: FEP to GDM MIDAS frontend to GDM to CDM to VX
* GDM MIDAS frontend: clock selector and monitoring, trigger and run control, busy and flow control, GDM housekeeping
* CDM MIDAS frontend: clock monitoring, CDM housekeeping

specific performance:
* GDM external clock: 10 MHz GPS clock
* GDM to CDM fiber link:
** clock XXX MHz
** link data rate: XXX Gbit/sec
** CDM recovered clock: XXX MHz
** CDM recovered clock jitter: XXX ns
** phase alignment between CDMs: XXX ns
** phase alignment between CDMs persists across reboots, power cycles, firmware updates
** phase alignment between CDMs should be easy to measure
** phase alignment between CDMs should be easy to recalibrate if hardware parts are replaced (DS-DM boards, fiber transceivers, fiber cables, etc)
** data packet bandwidth: XXX Mbytes/sec
** data packet latency: XXX clocks
** data packet skew between CDMs: XXX clocks
* CDM to VX clock:
** clock: XXX MHz
** jitter, all CDM clock outputs: XXX MHz
** phase alignment between all CDM clock outputs: XXX ns
* CDM to VX trigger:
** TBD (use the VX "sync" input or VX LVDS I/O line or VX serial link packet)
* CDM to VX serial link:
** clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
** bit rate: XXX bits/sec
** latency: XXX link clocks
** maximum skew between VXes: XXX ns
* VX to CDM serial link:
** clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
** bit rate: XXX bits/sec
** latency: XXX link clocks
** maximum skew between VXes: XXX ns
* timestamp reset:
** maximum skew between VXes: XXX ns
* busy round trip time: XXX ns (VX to CDM to GDM back to CDM to VX)
* flow control latency: XXX ns (FEP software to GDM MIDAS frontend to GDM to CDM to VX)

== Technical risk items ==

this refers to unexpected behaviour and performance of
system components, causes big difficulty in implementing the system,
prevents delivery of deliverables, and prevents or negatively affects operation
of the DS-20K DAQ or of the whole experiment.

(14-sep-2022, list is not sorted by any criterial: severity, probability, ease of investigation)

(stability of course is long term stability, across hours, days, weeks, months, years)

* stability of Enclustra FPGA modules (crashes/year, failures to boot/year, flash corruption/year)
* stability of GDM external clock PLL (lock loss/year)
* stability of CDM recovered clock (lock loss/year, unexpected phase drifts, etc)
* unexpected failures or bit error rates in GDM-CDM fiber links
* stability of CDM VX clock outputs (stability of clock cleaner chip)
* stability of VX internal clock distribution (VX PLL lock loss events)
* stability of VX CAEN base firmware (different versions of CAEN base firmware have different clock distribution behaviour)
* strange things in CAEN base firmware (unexpected clocking of LVDS I/O, unexpected phase shifts between clocks, etc)
* DS-DM and VX hardware problems (incompatible LVSD I/O, incompatible clock signals, etc)

== Milestones ==

(14-sep-2022: at this stage of the project, priority must be given to identifying and retiring (so called) technical risk factors. it is not good to build the complete system only to discover that (for example) some Enclustra FPGA modules require 5 attempts to boot and erase their flash memory contents once a month. Both example are real-life actual problems that caused big difficulties in GRIFFIN/TIGRESS and ALPHA-g experiments).

Development and testing milestones in time reversed order:

* full DAQ data challenge: all VXes, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
* one quadrant data challenge: 1 VME crate of VX, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
* vertical slice data challenge: 1 VME crate, 2 VX, 2 CDM (1 VX per CDM), GDM, etc operate as designed
* GDM-CDM link finalized (data rate frozen, data packet format frozen, data content permitted to change)
* CDM-VX serial link finalized (data rate frozen, data packet format frozen, data content permitted to change)
* run control (timestamp reset) and trigger distribution design agreed upon, frozen (list of possible triggers permitted to change)
* VX to CDM to GDM data flow design agreed upon, frozen (data contents permitted to change)
* major technical risk items retired (all hardware and firmware is working as expecred without mysteries and surprises, all problems are identified, investigated, resolved, solutions tested)
* stable operation of CDM-VX serial links in vertical slice system
* stable operation of GDM to CDM clock in vertical slice system
* stable operation of CDM to VX clock in vertical slice system
* vertical slice system assembled (1 VME crate, 2 VX, 2 CDM, 1 GDM, network, FEP, TSP, MIDAS)

= ZZZ =

ZZZ

DS-DM

2024-11-05T22:46:16Z

Mwalczak: /* After power up */

= DS-DM =

DarkSide-20k Global and Crate Data Manager board (GDM and CDM).

Global Data Manager (GDM):
* clock distribution to CDM boards (including GPS/atomic clock source)
* collection of trigger data from CDM boards, processing and distribution of trigger decision to CDM boards
* run control
* integration with GPS 10MHz and 1pps clocks and GPS/IRIG date and time information

Crate Data Manager (CDM):
* clock distribution from GDM to CAEN VX digitizers
* receive trigger data from CAEN VX digitizers
* send trigger data to GDM
* run control and dead time control

= Links =

* https://dsvslice.triumf.ca/vx_napoli - DS vertical slice at TRIUMF
* https://edev-group.triumf.ca/fw/exp/darkside/gcdm - git repository, DS-DM firmware
* https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0 - git repository, DS-DM board
* https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0/-/blob/master/Altium/Project%20Outputs%20for%20DS-DM-Rev0/SCH-DS-xDM-Rev0.PDF - DS-DM schematics
* https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0/-/blob/master/Testing/Clk3_XO_125_to_fpgaIN_recoveredMGTclk_to_IN2_Si5394-RevA-Project.slabtimeproj
* https://docs.xilinx.com/v/u/en-US/ds593 - Xilinx Platform Cable USB II
* https://www.enclustra.com/en/products/system-on-chip-modules/mercury-xu8/ - Enclustra
* https://bitbucket.org/team-ds-dm/ds-dm-software - git repository, DS-DM MIDAS frontend
* https://bitbucket.org/team-ds-dm/phasemeasurement - VX Phase measurement script
* https://ladd00.triumf.ca/daqinv/frontend/list/178 - inventory database
* https://daq00.triumf.ca/elog-ds/DS-DAQ - DS-DAQ elog
* https://bitbucket.org/ttriumfdaq/dsproto_vx2740/src/develop/ - VX DSFE MIDAS frontend
* https://dsdaqgw.triumf.ca/vslice/ - vertical slice DAQ
* https://daq00.triumf.ca/DaqWiki/index.php/DarkSide - DS-DAQ Wiki page
* https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0 - GPS and Rb Clock adapter board

= Onboard hardware =

* jtag chain: arm_dap_0 0x5BA00477, xczu4_1 0x04721093
* Eclustra Mercury+ XU8 module: ME-XU8-4CG-1E-D11E-R2.1
** Xilinx® Zynq Ultrascale+™ MPSoC XCZU4CG-1FBVB900E
** DDR4 ECC SDRAM (PS) 2 GB
** DDR4 SDRAM (PL) 1GB
* ethernet mac chip: AT24MAC402-SSHM-T ("602" chip is wrong)
* USB UART for Enclustra serial console, micro-USB, 115200n8
* clock chip: SI5394A-A-GM and oscillator CS-044-054.0M (54 MHz)
* U23 3.3V current meter and thermometer, LTC2990IMS#TRPBF
* LEDs:
** LED_FP A/B/C/D 0/1/2/3
** led1 - 3V3_SW_ON, SOM_POWER_GOOD - Enclustra FPGA module 3.3V power is good
** led2 - LTM4624 PGOOD
** led3 - FPGA_DONE - FPGA has booted
** led4 - TP-S-1, PCLK_P
** led5 - TP-S-2, PCLK_N
* LEMO connectors (top to bottom)
** J4 - input (NIM/TTL) (EXT_IN_LV(1), EXT_IN_LV(2))
** J5 - input (NIM/TTL) (EXT_IN_LV(3), EXT_IN_LV(4))
** J6 - external clock (GPS 10MHz and PPS)
** J7 - output (NIM/TTL) (EXT_OUT(1), EXT_OUT(2))
* SMA connectors
** J9, J10 - CLK_CCA from U6 C.C.
** J11, J12 - CLK_TP0
* RJ45 ethernet connector (100mbit: green light on, yellow light flashes when there is traffic)
* SFP connector (SFP is FTLF8526P3BNL, 6 Gbit/sec, 850 nm, 300m 50/125um OM3 MMF)
* 4 QSFP connectors (GDM)
* 6 VX connectors (CDM)

= Buttons, jumpers and switches =

Buttons:

* PB1 - HRST - reboot FPGA (power-on reset)
* PB2 - SRST - (SRSTn) - reboot ARM CPU

Switches:

* SW1 - boot mode BM0, BM1 [-->]
* SW2 - LEMO output NIM<->TTL
* SW3 - LEMO input 1 and 2 NIM/TTL
* SW4 - LEMO input 2 and 4 NIM/TTL
* SW5 - LEMO clock input NIM/TTL
* SW6 - serial console select. [PS<--PL] PS is ARM CPU, PL is FPGA.

= Front panel =

<pre>
| top
|
| LED-FP1 | LED_FP(0,1,2,3)
|
| SFP J???
|
| LEMO J4-LEFT, J4-RIGHT | EXT_IN_LV(1), EXT_IN_LV(2)
| LEMO J5-LEFT, J5-RIGHT | EXT_IN_LV(3), EXT_IN_LV(4)
| LEMO J6-LEFT, J6-RIGHT | CLK_EXT1, CLK_EXT0 (125 MHz only)
| LEMO J7-LEFT, J7-RIGHT | EXT_OUT(1), EXT_OUT(2)
|
| J-VX-1
| J-VX-2 or QSFP-1
| J-VX-3 or QSFP-2
| J-VX-4 or QSFP-3
| J-VX-5 or QSFP-4
| J-VX-6
|
| RJ45 J3 ethernet
|
| bottom
</pre>

= VX adapter board =

LVDS I/O connector
<pre>
34 pin connector: 0|:::: :::: :::: :::: :|15,16 n/c
</pre>

== split-cable connection ==

<pre>
0 .. 7 -> N/C

8 -> VX_RX(3) - not used
9 -> VX_RX(2) - busy VX to CDM
10 -> VX_RX(1) - DS20K 125 MHz serial data VX to CDM
11 -> VX_RX(0) - DS20K 62.5MHz clock VX to CDM

12 <- VX_TX(0) - TRG CDM to VX
13 <- VX_TX(1) - TSM CDM to VX, to be VETO CDM to VX
14 <- VX_TX(2) (set by jumper routed here or to VX CLKIN SYNC) - 125 MHz serial data CDM to VX
15 <- n/c (CLK routed to VX CLKIN CLK) - 62.5 MHz clock CDM to VX

16 - n/c
</pre>

== one-to-one connection ==

<pre>
0 -> VX2_RX(3)
1 -> VX2_RX(2)
2 -> VX2_RX(0)
3 -> VX2_RX(1)

4 <- CLK
5 <- VX2_TX(0)
6 <- VX2_TX(1)
7 <- VX2_TX(2)

8 -> VX1_RX(3)
9 -> VX1_RX(2)
10 -> VX1_RX(1)
11 -> VX1_RX(0)

12 <- VX1_TX(0)
13 <- VX1_TX(1)
14 <- VX1_TX(2) (set by jumper routed here or to VX CLKIN SYNC)
15 <- n/c (CLK routed to VX CLKIN CLK)

16 - n/c
</pre>

= Board schematics =

* [[File:SCH-DS-xDM-Rev0.PDF]]
* note: FPGA pin annotations ("IO", "SCLK", "PCLK", etc) on the schematics are bogus, instead, trace them to the FPGA pins.
* note: ENC A is J800, ENC B is J801, ENC C is J900 (schematic name to enclustra name)
* note: Enclustra special pins: "GC" is "clock capable", "HDGC" is "clock capable", "MGTREFCLK" is MGT reference clocks.
* board modifications:
** ethernet mac chip
** NIM output (no U15, etc)
** RJ45 wrong pinout (board mod or special ethernet cable)
** 125 MHz clock mods (TBW)
** disconnect QSFP0_SEL from SFP_RS0 and QSFP1_SEL from SFP_RS1, these signals are not used by modern SFPs
** provide SFP i2c modsel to allow SFP and QSFP at the same time on address 0x50

= FPGA MGT blocks =

<pre>
* SFP - ENC B45,B47 and B48,B50 - FPGA D5,D6 MGTHTX0_D and D1,D2 MGTHRX0_D
* QSFP0 TX0 - ENC C13,C17 - FPGA H5,H6 MGTHTX0_C
* QSFP0 TX1 - ENC C21,C25 - FPGA G7,G8 MGTHTX1_C
* QSFP0 TX2 - ENC C29,C23 - FPGA F5,F6 MGTHTX2_C
* QSFP1 TX0 - ENC C37,C41 - FPGA E7,E8 MGTHTX3_C
* QSFP1 TX1 - ENC C45,C47 - FPGA P5,P6 MGTHTX0_B
* QSFP1 TX2 - ENC C51,C53 - FPGA M5,M6 MGTHTX1_B
* QSFP2 TX0 - ENC C57,C59 - FPGA L3,L4 MGTHRX2_B
* QSFP2 TX1 - ENC C63,C65 - FPGA K5,K6 MGTHTX3_B
* QSFP2 TX2 - ENC C75,C77 - FPGA W3,W4 MGTHTX0_A
* QSFP3 TX0 - ENC C79,C81 - FPGA V5,V6 MGTHTX1_A
* QSFP3 TX1 - ENC C85,C87 - FPGA T5,T6 MGTHRX2_A
* QSFP3 TX2 - ENC C89,C91 - FPGA R3,R4 MGTHTX3_A
</pre>

= Clock distribution =

Simplified:

* 125 MHz osc -> CLK_XO_125 -> MGTREFCLK0_A -> not used
* 125 MHz osc -> CLK3_XO_125 -> MGTREFCLK1_B -> SFP RX ref clock, QSFP RX and TX ref clock (this is not final design!)
* (disconnected) 125 MHz osc -> CLK2_XO_125 -> FPGA AG8,AH8 (GC)
* 125 MHz osc -> C.C. in1

* C.C. in0 <- CLK_EXT1 (10 MHz GPS clock)
* C.C. in1 <- 125 MHz osc
* (disconnected) C.C. in2 <- CLK_CC_IN <- MGTREFCLK0_D <- SFP RX clock (cannot be used because of uncontrollable phase)
* C.C. in2 <- CLK2_XO_125 <- FPGA AG8,AH8 (GC) <- SFP RX recovered 125 MHz clock
* C.C. in3 <- CLK_FB

* C.C. 125 MHz -> CLK_CC_OUT0 -> MGTREFCLK0_B -> QSFP RX and TX ref clock (final design)
* C.C. 125 MHz -> CLK_CC_OUT1 -> MGTREFCLK1_D -> SFP TX clock
* C.C. 62.5 MHz -> VX clock fanout

proposed changes:
* add C.C. 125 MHz -> new CLK_CC_OUT2 (old CLK2_XO_125) -> FPGA AG8,AH8 (GC)
* change C.C. in2 <- new CLK_CC_IN <- FPGA AK8,AK9 (non-GC)

Complete:

<pre>
125 MHz oscillator - U5 fan out -
q0 -> CLK_XO_125 -> ENC C72,C74 -> FPGA R7,R8 MGTREFCLK0_A (not used)
q1 -> U6 C.C. in1
q2 -> CLK3_XO_125 -> ENC C7,C9 -> FPGA J7,J8 MGTREFCLK1_B -> SFP RX reference clock, QSFP RX and TX reference clocks (not final design!)
q3 -> disconnected on the board, was CLK2_XO_125 -> ENC C151,C153 -> FPGA AG8,AH8 (GC)

U6 C.C (clock cleaner) -

in0 <- CLK_EXT1 (presumably GPS 10 MHz ref clock)
in1 <- 125 MHz oscillator via U5
in2 <- (was: CLK_CC_IN <- ENC B10,B12 <- FPGA D9,D10 MGTREFCLK0_D <- SFP RX clock, 125 MHz)
in2 <- CLK2_XO_125 <- ENC C151,B153 <- FPGA AG8,AH8 (GC) <- mgt_link_data_to_processing.rx_data_clk (SFP RX data clock, 125 MHz)
in3 <- CLK_FB

out0 -> CLK_CCA -> U12 (125 MHz)
out1 -> CLK_CCB -> VX1..6 (62.5 MHz)
out2 -> CLK_CCC -> VX7..12 (62.5 MHz)
out3 -> CLK_FB into in3

CLK_CCA -> U12 (125 MHz fan out) ->

Q0 -> not used
Q1 -> CLK_CC_OUT0 -> ENC C3-5 -> FPGA L7,L8 MGTREFCLK0_B -> QSFP RX and TX reference clocks (final design)
Q2 -> CLK_CC_OUT1 -> ENC B3-5 -> FPGA B9,B10 MGTREFCLK1_D -> SFP TX clock
Q3 -> not used
Q4 -> not used
Q5 -> SMA J9/J10

CLK_TP0 - FPGA N7,N8 MGTREFCLK1_A - ENC C69,C71 - DS-DM SMA J11, J12 (NOT IN CDM PROJECT)

Notes:
* CLK_XO_125 (125 MHz osc) is not used
* 62.5 MHz VX clock does not go into the FPGA
* CLK_EXT0 going to in1 of U5 cannot be used. only permitted frequency is 125 MHz (it drives the MGT reference clocks) and is this frequency is too high for LEMO cables and connectors

Proposed modifications:
- CLK_CC_IN: from FPGA output pin (ENC C142,C144 - FPGA AK8,AK9) to clock cleaner in2 (instead of CLK2_XO_125 pins)
- CLK_CCA -> U12 -> currently unused out3 -> CLK2_XO_125 FPGA pins
- repurpose CLK_EXT0 at GPS 1pps/IRIG input to FPGA
</pre>

= I2C bus =

* I2C_SCL is J-ENC-A pin 111 I2C_SCL is FPGA I2C_SCL_PL AB13 (IO_L1N_TOL1D_64) and I2C_SCL_PS F18 (PS_MIO10)
* I2C_SDA is J-ENC-A pin 113 I2C_SDA is FPGA I2C_SDA_PL AH13 (IO_L7N_T1L1Q_AD13N_64) and I2C_SDA_PS G18 (PS_MIO11)

* XU8 secure EEPROM ATSHA204A at 0x64, this is 0110'010X -> linux _011'0010 is 0x32. (but responds to scan and read at 0x33)
* U4 ethernet mac chip, EEPROM at 1010 A2 A1 A0 X and MAC/serial_no at 1011 A2 A1 A0 X. A0=VCC, A1=VCC, A2=GND -> linux _101'0011 and _101'1011 is 0x53 and 0x5B.
* U6 clock chip, address 1101 0 A1 A0 X. A1=VCC, A0=N/C (internal pull-up) -> linux _110'1011 is 0x6b
* U23 voltmeter at 10011 ADR1 ADR0. ADR0=GND, ADR1=VCC -> linux _100'1110 is 0x4e
* SFP, address 1010000X -> linux _101'0000 is 0x50. additional SFP data at 0x51
* QSFP0, QSFP1, QSFP2, QSFP3 (QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL)

= I2C clock builder connection =

* use Silicon Labs USB "Clock builder pro field programmer", www.silabs.com/CBProgrammer
* connect rainbow jumper cable pins:
** black - 1-GND to GND on DS-DM
** white - 3-SCLK to SCL on the DS-DM
** grey - 7-SDA_SDIO to SDA on the DS-DM
* power up the DS-DM
* plug USB programmer into Windows laptop
* on Windows, run "ClockBuilder Pro"
* it should report "Field programmer detected", press "EVB GUI"
* in EVB GUI, press "Config", set I2C address 0x6B
* press "Scan", it should find Si5394A-A-GM
* select the "Status" tab, should see real-time status of clock chip

= GDM MGT configuration =

* TX configuration:
* GDM MGT transceivers are configured as "multilane" TX and RX.
* there is 12 TXes ("lanes")
* MGT reference 125 MHz clock goes into [2:0]gtreclk00_in and [11:0]gtrefclk0_in
* one MGT is designated as "master"
* PLL of master MGT converts reference clock into common TX clock and common tx_user_clk2 which becomes tx_data_clk
* common tx_user_clk2 aka tx_data_clk goes into all TXes and clocks tx_data.
* tx_user_clk2 aka tx_data_clk is 125 MHz but not same phase as MGT reference clock.

* RX interim configuration:
* there is 12 RXes ("lanes")
* each RX produces it's own recovered RX clock
* "multilane" configuration assumes all RX recovered clocks run at the same frequency (TX on the other end are driven by common TX clock, see above), but have different phase
* one RX recovered clock is designated as "master" (rx_user_clk2 aka rx_data_clk) and a phase-matching fifo/buffer is used to bring rx_data from all 12 RXes to this common rx_data_clk
* this works because each CDM SFP TX runs on the SFP RX recovered clock which is frequency-locked with the GDM QSFP TX clock.

* RX final configuration:
* MGTs permit using the common TX clock (tx_user_clk2 aka tx_data_clk) as the common rx_data_clk (they are frequency locked through the CDM).
* this permits use of tx_data_clk as the main clock domain in the GDM and removes the need to bring rx_data into the tx_data_clk domain (actually this is done in the MGT RX phase matching fifo/buffer).

= Clock path =

<pre>
10 MHz ext clock or GDM 125 MHz oscillator
-> GDM QSFP MGT reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz (GDM main clock domain) and TX bit clock 2.5 GHz
-> GDM QSFP optic transmitter
-> CDM SFP optic receiver
-> CDM SFP MGT, RX reference clock is CDM 125 MHz oscillator
-> MGT RX recovered clock 125 MHz (CDM main clock domain)
-> CC_CLK_IN -> CDM C.C. -> CC_CLK_OUT1 -> CDM SFP TX reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz and TX bit clock 2.5 GHz
-> (tx_data phase matching fifo from CDM main clock domain to tx_data_clk)
-> CDM SFP optic transmitter
-> GDM QSFP RX optic receiver (12x)
-> GDM QSFP MGT (RX reference clock is same as TX reference clock)
-> MGT RX recovered clock (12x recovered clocks)
-> in multi-lane configuration, one of them is the "master" recovered clock rx_data_clk
-> (rx_data phase matching fifo from rx_data_clk to GDM main clock domain)
</pre>

= CDM rx_clk mux =

when CDM SFP is not connected, there is no SFP recovered clock and a mux is used to switch between clk_cc_out0 (power up default) and rx_clk_mgt (SFP recovered clock)

== Test SFP disconnected ==

note: if I say "--cc-in1", CC seems to lock on the 10 MHz GPS external clock,
to prevent this, test sequence includes reloading the CC and the reset of MGT.

<pre>
./test_cdm_local.exe --write32 0x30 0x0 ### rx_clk mux select CC clock
./test_cdm_local.exe --load-cc
./test_cdm_local.exe --reset-mgt
./test_cdm_local.exe --cdm-clocks
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773594e (125000014) should be ~125 MHz <=== all 4 clocks wobble close to 125 MHz
0x1034 rx_clk: 0x0773594f (125000015) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x0773594f (125000015) should be ~125 MHz
0x103C tx_clk: 0x0773594f (125000015) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
./test_cdm_local.exe --write32 0x30 0x1 ### rx_clk mux select SFP recovered clock
./test_cdm_local.exe --cdm-clocks
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773598b (125000075) should be ~125 MHz
0x1034 rx_clk: 0x076d58ec (124606700) should be ~125 MHz <=== off frequency because there is no valid SFP recovered clock
0x1038 mgt_tx_ref_clk_raw: 0x0773598c (125000076) should be ~125 MHz
0x103C tx_clk: 0x0773598b (125000075) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

= Clock domains =

== GPS ==

* no GPS : GDM runs from internal 125 MHz oscillator
* external 10 MHz clock : GDM runs from external 10 MHz clock and optional 1pps signal (use VME-NIMIO32 NIM outputs)
* GPS receiver : GDM runs from GPS 10 MHz clock and GPS IRIG serial data
* LNGS GPS:
** provides 1pps and serial data over fiber from GPS receiver (master). LNGS xxx box is not used.
** serial data goes to GDM, decoded, 1pps signal extracted, goes to LEMO output
** 1pps from GDM is used to train the Rubidium clock which provides a 10 MHz clock
** 10 MHz output from Rubidium clock goes to GDM 10 MHz external clock input

== GDM ==

* AXI clock (100 MHz) - AXI registers
* 125 MHz oscillator - to clock cleaner
* 10 MHz external clock LEMO input - to clock cleaner
* FPGA 125 MHz clock CLK_CC_IN - to clock cleaner - not used (could be used for cascaded GDMs)
* 125 MHz oscillator - CLK3_XO_125 - mgt_b_ref_clk QSFP MGT reference clock (interim GDM design)
* clock cleaner output 125 MHz fanout:
** CLK_CC_OUT0 - QSFP MGT reference clock (final design)
** CLK_CC_OUT1 - not used (CDM SFP reference clock)
** CLK_CC_OUT2 - not used
* QSFP MGT TX data clock 125 MHz tx_data_clk (main clock domain)
** QSFP TX data
** QSFP RX data (in final design, rx_data_clk is same as tx_data_clk. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk == tx_data_clk)
** ds20k block
* (in interim design, rx_data_clk is disconnected from tx_data_clk. rx_data_clk is the "multilane master clock" which is one of the 12x RX recovered clocks. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk)

note: all these clocks are frequency locked to 125 MHz

== CDM ==

* AXI clock (100 MHz) - AXI registers
* 10 MHz external clock LEMO input - to clock cleaner (not used)
* 125 MHz oscillator to fanout
** to clock cleaner
** to SFP MGT RX reference clock: CLK3_XO_125 to mgt_rx_ref_clk to gtrefclk01_in
* SFP MGT RX recovered clock 125 MHz
** MGT PLL to MGT rx_user_clk2 aka rx_data_clk
** MGT rxrecclkout_out to mgt_rx_rec_clk to CLK_CC_IN to C.C.
* SFP MGT rx_user_clk2 aka rx_data_clk (250 MHz/8 bit, 125 MHz/16 bit, 62.5 MHz/32 bit data) (main clock domain)
** SFP RX data
** ds20k block
** VX TX clock PLLs
** VX RX clock PLLs
* C.C. fan out
** 62.5 MHz VX clocks (12x)
** CLK_CC_OUT0 (not used, GDM QSFP MGT reference clock)
** CLK_CC_OUT1 125 MHz to mgt_tx_ref_clk to gtrefclk00_in to SFP MGT TX reference clock
** CLK_CC_OUT2 (not used)
* SFP MGT tx_user_clk2 aka tx_data_clk
** SFP TX data
** TX data phase matching fifo from main clock domain to tx_data_clk
* VX TX clock PLLs
** 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
** VX TX data phase matching from main clock domain to VX TX clock (12 total)
** VX TX serializer
** VX TX LVDS transmitter
* VX RX clock PLLs
** 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
** VX RX LVDS receivers (12 total)
** VX RX deserializers (12 total)
** VX TX data phase matching from VX RX clock to main clock domain

== VX ==

* everything runs on the VX main 125 MHz clock
* correct phase of VX to CDM LVDS data is adjusted by scan of CDM VX RX PLL clock phase (VX to CDM link is now established)
* correct phase of CDM to VX LVDS data is adjusted by scan of CDM VX RX PLL clock phase (link is established after VX to CDM idle data pattern changes from "VX RX data bad" to "good".
* after good phases are found by scan, they are not expected to change unless cables are changed, CDM and VX modules are changed or CDM firmware is rebuilt. (rebuild of VX firmware should not affect LVDS data phase).
* if there is excessive link errors, phase scan must be repeated.

= Board test plan =

To test:
* Enclustra FPGA board
* SFP port - SFP_RS0/RS1 connected to QSFP0_SEL/1_SEL is wrong? our Finisar SFP says RS0, RS1 N/C, so probably okey.
* SW5 CLK_EXT1 NIM works. TTL needs to be tested. flipping SW5 CLK_EXT0 side from NIM to TTL makes CLK_EXT1 go LOS and OOF in the clock chip. R23 and R53 should be removed?
* BOOT_MODE 0 and 1

Partial:
* U23 3.3V current meter and thermometer. V1,V2 is current monitor, same as in application note. V3,V4 is thermometer, same in application note, except capacitor C118 is 0.1u instead of 470pF. Tested ok: Tint, VCC, V1, V2, V1-V2. Test failed: TR2 reads 50-something degC instead of same as Tint.

Done:
* LED_FP1A..D: tested ok. K.O. 15 sep 2022
* USB UART: tested ok. K.O. 15 sep 2022
* J4A, J4B, J5A, J5B LEMO inputs (NIM/TTL) EXT_IN_LV(1..4). TTL threshold 1.7V, NIM threshold -0.3V. 50 Ohm termination. TTL straight, NIM inverted.
* J6A, J6B LEMO clock inputs: 10 MHz TTL from chronobox works, 10 MHz NIM from IO32 works. 18-apr-2023
* J7A, J7B LEMO outputs EXT_OUT(1), EXT_OUT(2) (NIM/TTL) (tested 23nov2022, K.O.). TTL ???, NIM inverted.
** TTL out no 50 ohm termination: 0=0V, 1=5V, rise and fall time ~5 ns
** TTL out with 50 ohm termination: 0=0V, 1=2.5V, rise and fall time <2ns
** NIM out no 50 ohm termination: 0=+50mV, 1=-1.8V, rise and fall time ~3ns
** NIM out with 50 ohm termination: 0=0V, 1=-0.9V, rise and fall time <2ns
* ethernet MAC i2c chip (K.O. 20sep2022: can read: i2cdump 0 0x5b, nov2022 have u-boot driver)
* CDM VX ports 2x(CLK, 3 tx, 4 rx) tested using test_cdm.exe and LVDS loopback in VX firmware.
* SFP i2c tested KO 22jun2023
* QSFP i2c tested KO 22jun2023
* i2c testing complete 22jun2023
* QSFP rx,tx tested 26june2023. lane0,1,2 ok, lane3 tx not connected, laser is off. qsfp0,1,2,3 all lanes ok.
* SFP rx,tx tested 26june2023. rx and tx okey. LOS ok, mod_absent ok.

Failure:
* ethernet: does not connect to alliedtelesys switch. connects to my USB-eth adapter at 100 Mbit speed. uboot mii status reports connection speed oscillating between 1000, 100 and 10. K.O. 16-sep-2022
* SFP LOS and mod_absent are swapped (in the FPGA pin definitions?)
* QSFP slot numbering is wrong.

= Checklist for newly build boards =

* put new board on workbench
* check - vme connector present, vme extraction handles present
* check - standoff are removed from all thru-holes
* plug Enclustra module
* check - SW6 both switches are in the "PS" position
* connect micro-usb cable from linux PC
* connect ethernet from 1gige capable network switch
* connect power from lab power supply - GND, +5V and -12V nominal, +5.7V and -12.5V actual
* power up, +5V current 2.10-2.8A, -12V current 0.05A
* on linux PC, open a new terminal, run: minicom -D /dev/ttyACM0
* in minicom window, observe messages about Xilinx first stage boot loader, etc
* on the ethernet switch, observe network link is 1gige speed (not 10mbit, not 100mbit).
* if everything boots okey, there will be a login prompt, login as root, password root.
* busybox devmem 0x80010000 # read firmware version number, i.e. 0xEDAD0A77
* TBW - test LEMO inputs
* TBW - test LEMO outputs
* TBW - test VX connectors
* TBW - test SFP connector
* TBW - test QSFP connector

= Serial console =

* check that linux computer has correct udev rules to allow access to /dev/ttyACM devices, see https://daq00.triumf.ca/DaqWiki/index.php/SLinstall#Configure_USB_device_permissions and https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Configure_USB_device_permissions
* connect micro-USB cable to connector J-UCB, other end connect to linux computer
* observe /dev/ttyACM0 was created
* run "minicom -D /dev/ttyACM0" (default serial settings are ok, otherwise, 115200n8)
* should have gdm-cdm login
* username root, password root

= i2c =

<pre>
ZynqMP> i2c bus
Bus 0: i2c@ff020000
ZynqMP> i2c dev 0
Setting bus to 0
ZynqMP> i2c probe
Valid chip addresses: 33 4E 53 5B 6B 77
ZynqMP> i2c md 0x5b 0x98
0098: fc c2 3d 00 00 12 1a 6e 0a 90 85 04 94 10 08 50 ..=....n.......P
</pre>

<pre>
root@gdm-cdm:~# i2cdetect 0
Warning: Can't use SMBus Quick Write command, will skip some addresses
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0.
I will probe address range 0x03-0x77.
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f
00:
10:
20:
30: -- -- -- 33 -- -- -- --
40:
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60:
70:
root@gdm-cdm:~#
root@gdm-cdm:~# i2cdump 0 0x5b
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0, address 0x5b, mode byte
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00 ???????P?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e ........??=..??n
a0: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00 ???????P?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e ........??=..??n
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@gdm-cdm:~#
</pre>

<pre>
root@gdm0:~# i2cdetect -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#
</pre>

<pre>
root@cdm0:~# i2cdetect -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: 50 51 -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@cdm0:~#
</pre>

* 0x33 - XU8 secure EEPROM (should be at 0x32)
* 0x4e - U23 current and temperature monitor
* 0x50, 0x51 - SFP
* 0x50 - QSFP, 4 QSFP modules enabled by GPIO QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL
* 0x53, 0x5b - ethernet mac eeprom
* 0x6b - U6 clock chip

== U23 ==

NOTE: Text reads double of Tint. not sure why. K.O. 21-mar-2024.

* internal temperature only

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x00 b # control register: "repeat mode, internal temperature only"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x03 # "Tint ready" and "busy", "busy is always 1 in repeat mode"
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 00 03 03 81 db 2a ce 2a 8e 00 6f 00 45 20 3f ?.????*?*?.o.E ?
10: 01 00 01 01 01 db 2a ce 2a 8e 00 6f 00 45 20 3f ?.????*?*?.o.E ?
...
readback:
reg0 - 03 - Tint ready
reg1 - 00 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB 0x81, bit 0x80 is "DV, data valid", bit 0x40 is "SS, sensor short", 0x20 is "SO, sensor open"
reg5 - Tint LSB 0xDB, Tint = 0x01DB = 475 * 0.0625 degC = 29.6 degC
reg6..F - stale data
</pre>

* Tint, V1, V2, TR2, VCC

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x18 b # control register: "repeat mode, V1, V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 7f 18 7f 7f 81 b9 aa d0 aa 8f 83 5e 83 5e a0 41 ???????????^?^?A
10: 01 18 01 01 01 b9 2a d0 2a 8f 03 5e 03 5e 20 41 ??????*?*??^?^ A
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x1b9*0.0625 = 27.5 degC
reg6 - V1 MSB 0xaa, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 LSB 0xd0, V1 = 0x2ad0*305.18/1000000 = 3.3447 V (correct)
reg8 - V2 MSB 0xaa, ditto
reg9 - V2 LSB 0x8f, V2 = 0x2a8f*305.18/1000000 = 3.3249 V (correct, smaller than V1)
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x5e. TR2 = 0x35e*0.0625 = 53.875 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x41, VCC = 2.5+0x2041*305.18/1000000 = 5.019 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.99 A (about right?)
</pre>

* Tint, V1-V2, TR2, VCC

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x19 b # control register: "repeat mode, V1-V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 7f 19 7f 7f 81 69 83 f2 83 f2 83 3a 83 3a a0 44 ?????i?????:?:?D
10: 01 19 01 01 01 69 03 f2 03 f2 03 3a 03 3a 20 44 ?????i?????:?: D
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x169*0.0625 = 22.5 degC
reg6 - V1 or V1-V2 MSB 0x83, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 or V1-V2 LSB 0xf2, V1-V2 = 0x3f2*19.42/1000000 = 0.0196142 V (correct, compare with V1 and V2 measured above)
reg8 - V2 or V1-V2 MSB
reg9 - V2 or V1-V2 LSB
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x3a. TR2 = 0x33a*0.0625 = 51.625 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x44, VCC = 2.5+0x2044*305.18/1000000 = 5.021 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.9807 A (about right?)
</pre>

== SFP ==

<pre>
root@cdm0:~# i2cdump 0 0x50
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 04 07 00 00 00 00 40 40 0c 00 01 3d 00 00 00 ???....@@?.?=...
10: 0c 02 00 1e 46 49 4e 49 53 41 52 20 43 4f 52 50 ??.?FINISAR CORP
20: 2e 20 20 20 00 00 90 65 46 54 4c 46 38 35 32 36 . ..?eFTLF8526
30: 50 33 42 4e 4c 20 20 20 41 20 20 20 03 52 00 9d P3BNL A ?R.?
40: 00 1a 00 00 4e 33 41 42 34 4c 56 20 20 20 20 20 .?..N3AB4LV
50: 20 20 20 20 32 30 30 33 31 39 20 20 68 f0 03 de 200319 h???
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@cdm0:~#
root@cdm0:~# i2cdump 0 0x51
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 5a 00 d8 00 55 00 e2 00 90 88 71 48 8c a0 75 30 Z.?.U.?.??qH??u0
10: 21 34 01 f4 1b 58 03 e8 31 2d 04 eb 1f 07 06 31 !4???X??1-?????1
20: 31 2d 00 64 27 10 00 9e 00 00 00 00 00 00 00 00 1-.d'?.?........
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
40: 00 00 00 00 3f 80 00 00 00 00 00 00 01 00 00 00 ....??......?...
50: 01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 b7 ?...?...?......?
60: 19 9f 80 c5 0e 17 12 c4 1f 99 00 00 00 00 30 00 ??????????....0.
70: 00 00 00 00 00 00 00 00 ff ff ff ff ff ff ff 01 ...............?
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@cdm0:~#
</pre>

== QSFP ==

<pre>
QSFP i2c enable lines, active low:

QSFP0_SEL - JENC-A 82 - PS-MIO40 - linux gpio 378
QSFP1_SEL - JENC-A 84 - PS-MIO41 - linux gpio 379
QSFP2_SEL - JENC-A 100 - PS-MIO44 - linux gpio 382
QSFP3_SEL - JENC-A 106 - PS-MIO43 - linux gpio 381 (notice 2 and 3 are out of order)
</pre>

<pre>
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
# echo 378 >> /sys/class/gpio/export ### SEL0 338+40
# echo 379 >> /sys/class/gpio/export ### SEL1 338+41
# echo 381 >> /sys/class/gpio/export ### SEL3 338+43
# echo 382 >> /sys/class/gpio/export ### SEL2 338+44
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) in hi
gpio-379 ( |sysfs ) in hi
gpio-381 ( |sysfs ) in hi
gpio-382 ( |sysfs ) in hi
root@gdm0:~# echo out >> /sys/class/gpio/gpio381/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio382/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio378/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio379/direction
root@gdm0:~#
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out lo
gpio-379 ( |sysfs ) out lo
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out lo
root@gdm0:~#
echo 1 >> /sys/class/gpio/gpio381/value
echo 1 >> /sys/class/gpio/gpio382/value
echo 1 >> /sys/class/gpio/gpio378/value
echo 1 >> /sys/class/gpio/gpio379/value
cat /sys/kernel/debug/gpio
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out hi
gpio-382 ( |sysfs ) out hi
root@gdm0:~#
root@gdm0:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#

NOTICE NOTHING AT ADDRESS 0x50
</pre>

<pre>
ENABLE QSFP0, OBSERVE IT IS AT ADDRESS 0x50

root@gdm0:~# echo 0 >> /sys/class/gpio/gpio378/value
root@gdm0:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: 50 -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#
root@gdm0:~# i2cdump -y 0 0x50
No size specified (using byte-data access)
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 0d 00 02 8e 00 00 01 00 00 05 55 00 05 00 00 00 ?.??..?..?U.?...
10: 00 00 00 00 00 00 1e b8 00 00 81 a0 00 00 00 00 ......??..??....
20: 00 00 12 12 00 01 00 01 00 01 0e e0 0f 20 0e e0 ..??.?.?.???? ??
30: 00 00 1f a0 1e 3b 1f 72 00 01 00 00 00 00 00 00 ..???;?r.?......
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
60: 00 00 00 00 00 00 00 00 00 00 1f 00 00 00 08 00 ..........?...?.
70: 00 00 00 00 00 00 00 ff ff ff ff ff ff ff ff 00 ................
80: 0d 00 0c 04 00 00 00 40 40 02 d5 05 67 00 00 96 ?.??...@@???g..?
90: 00 00 c8 00 46 49 4e 49 53 41 52 20 43 4f 52 50 ..?.FINISAR CORP
a0: 20 20 20 20 07 00 90 65 46 54 4c 34 31 30 51 44 ?.?eFTL410QD
b0: 34 43 20 20 20 20 20 20 41 20 42 68 07 d0 00 43 4C A Bh??.C
c0: 00 07 0f de 58 37 39 41 43 30 52 20 20 20 20 20 .???X79AC0R
d0: 20 20 20 20 32 32 30 33 30 39 20 20 3c 10 00 9e 220309 <?.?
e0: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@gdm0:~#
</pre>

== ethernet mac eeprom ==

* correct chip with 84-bit ethernet mac address

<pre>
root@cdm1:~# i2cdump 0 0x53
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
root@cdm1:~#

root@cdm1:~# i2cdump 0 0x5b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00 ????4??2?K?.?...
90: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c ..........??=?Q<
a0: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00 ????4??2?K?.?...
b0: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c ..........??=?Q<
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@cdm1:~#
</pre>

* wrong "602" chip with 64-bit IPv6 address

<pre>
root@cdm0:~# i2cdump 0 0x53
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
root@cdm0:~# i2cdump 0 0x5b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00 ???????Q?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e ........??=..??.
a0: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00 ???????Q?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e ........??=..??.
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@cdm0:~#
</pre>

= read ethernet mac address from i2c =

(this code is copied from uboot command line i2c code)

in uboot sources board/xilinx/common/board.c replace original function with this:

<pre>
// special code to read ethernet MAC address from the DS-DM-Rev0 board. K.O. Sep-2022

int zynq_board_read_rom_ethaddr(unsigned char *ethaddr)
{
struct udevice *bus;
int ret;
int busnum = 0;

ret = uclass_get_device_by_seq(UCLASS_I2C, busnum, &bus);
if (ret) {
printf("%s: No bus %d\n", __func__, busnum);
return ret;
}

int chip_addr = 0x5B;

struct udevice *dev;

ret = i2c_get_chip(bus, chip_addr, 1, &dev);
if (ret) {
printf("%s: Bus %d no chip 0x%02x\n", __func__, busnum, chip_addr);
return ret;
}

int dev_addr = 0x98;

unsigned char data[8];

ret = dm_i2c_read(dev, dev_addr, data, 8);

if (ret) {
printf("%s: Bus %d chip 0x%02x read error %d\n", __func__, busnum, chip_addr, ret);
return ret;
}

printf("%s: Bus %d chip 0x%02x addr 0x%02x read: 0x %02x %02x %02x %02x %02x %02x %02x %02x\n", __func__, busnum, chip_addr, dev_addr, data[0], data[1], data[2], data[3], data[4], data[5], data[6], data[7]);

// see Atmel-8807-SEEPROM-AT24MAC402-602-Datasheet.pdf

if (data[0] == 0) {
// eiu-48 chip
ethaddr[0] = data[2];
ethaddr[1] = data[3];
ethaddr[2] = data[4];
ethaddr[3] = data[5];
ethaddr[4] = data[6];
ethaddr[5] = data[7];
} else {
// eiu-64 chip
ethaddr[0] = data[0];
ethaddr[1] = data[1];
ethaddr[2] = data[2];
ethaddr[3] = data[5];
ethaddr[4] = data[6];
ethaddr[5] = data[7];
}

printf("%s: ethaddr %02x:%02x:%02x:%02x:%02x:%02x\n", __func__, ethaddr[0], ethaddr[1], ethaddr[2], ethaddr[3], ethaddr[4], ethaddr[5]);

return ret;
}
</pre>

also this should have worked if i2c_xxx() functions were enabled in uboot:
<pre>
i2c_set_bus_num(0);
i2c_probe(0x5b);
i2c_read(0x5b, 0x9a, ethaddr, 6);
</pre>

= read ethernet mac address from i2c (SHOULD WORK) =

from: https://stackoverflow.com/questions/43637540/is-there-linux-or-u-boot-support-to-read-a-mac-address-from-a-chip-at-startup

<pre>
#ethernet related setup
setup_eth=run readmac buildmac
#read mac address from eeprom
readmac=i2c dev 0; i2c read 50 FA.1 6 $loadaddr
#build the ethaddr variable
#not very nice, but does the job
buildmac=\
e=" "; sep=" " \
for i in 0 1 2 3 4 5 ; do\
setexpr x $loadaddr + $i\
setexpr.b b *$x\
e="$e$sep$b"\
sep=":"\
done &&\
setenv ethaddr $e
</pre>

= read ethernet mac address from i2c (DOES NOT WORK) =

this method does not work: inside board/xilinx/common/board.c:int zynq_board_read_rom_ethaddr(unsigned char *ethaddr), uclass_get_device_by_ofnode(UCLASS_I2C_EEPROM, eeprom, &dev) returns ENODEV and read from i2c does not happen. K.O. Sep-2022

Read:
* https://support.xilinx.com/s/question/0D52E00006hpKlsSAE/uboot-and-ethernet-mac-address-from-eeprom?language=en_US (last answer)
* https://github.com/Xilinx/u-boot-xlnx/blob/master/board/xilinx/common/board.c (zynq_board_read_rom_ethaddr())

Note:
* 0x5B is the i2c chip address
* 0x9A is the data offset inside the chip, see datasheet or i2c read dump above.

Edit:
* emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/u-boot/files/platform-top.h
<pre>
#include <configs/xilinx_zynqmp.h>
#include <configs/platform-auto.h>
//#define CONFIG_I2C_EEPROM
//#define CONFIG_SYS_I2C_EEPROM_ADDR 0x5b
//#define CONFIG_SYS_I2C_EEPROM_ADDR_OVERFLOW 0x0
#define CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET 0x9A
#error HERE!
</pre>
* emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi
<pre>
/include/ "system-conf.dtsi"
/ {
chosen {
xlnx,eeprom = &eeprom;
};
};

&i2c0 {
eeprom: eeprom@5b { /* u88 */
compatible = "atmel,24mac402";
reg = <0x5b>;
};
};
</pre>
* components/yocto/workspace/sources/u-boot-xlnx/configs/xilinx_zynqmp_virt_defconfig
<pre>
CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET=0x9A
</pre>

= enable VX clock =

<pre>
devmem 0x80010010 32 0x8; sleep 1; devmem 0x80010010 32 0x0;
si5394-i2c-file /media/sd-mmcblk1p1/00_freerun.txt 0 0x6b
</pre>

= clock chip configuration =

file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt:

* VCO is 14 GHz
* Tvco is 71.43 ps
* N0 divider is 14, frequency is 1000 MHz
* out0 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz
* out1 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
* out2 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
* out3 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz is the feedback for zero delay

= clock chip monitoring =

from si5395-94-92-family.pdf:
* reg 0x1: page select, set to 0 or set to 5 to read 0x53F
* reg 0x2: 0x94
* reg 0x3: 0x53 -> device is a si5394
* reg 0xC: LOSXAXB
* reg 0xD: LOS and OOF for the 4 clock inputs
* reg 0xE: LOL and HOLD
* reg 0xF: CAL_PLL
* reg 0x11: sticky bits for reg 0xC
* reg 0x12: sticky bits for reg 0xD
* reg 0x13: sticky bits for reg 0xE
* reg 0x14: sticky bits for reg 0xF
* reg 0x1C: device reset
* reg 0x1E: low power, hard reset, SYNC
* reg 0x507: currently selected input clock
* reg 0x52A: input clock select
* reg 0x535: FORCE_HOLD
* reg 0x53F: HOLD_HIST_VALID and FASTLOCK_STATUS

= arm and linux benchmark =

memory benchmark:

<pre>
daq13$ arm-linux-gnueabi-gcc -o memcpy.armv7 memcpy.cc -march=armv7 -static -O2
scp memcpy.armv7 to ...
root@gdm-cdm:~# ./memcpy.armv7
memcpy 1 KiBytes: 1288 MB/sec
memcpy 2 KiBytes: 1924 MB/sec
memcpy 4 KiBytes: 2554 MB/sec
memcpy 8 KiBytes: 3054 MB/sec
memcpy 16 KiBytes: 3262 MB/sec
memcpy 32 KiBytes: 3250 MB/sec
memcpy 64 KiBytes: 3456 MB/sec
memcpy 128 KiBytes: 3556 MB/sec
memcpy 256 KiBytes: 3780 MB/sec
memcpy 512 KiBytes: 3795 MB/sec
memcpy 1024 KiBytes: 3789 MB/sec
memcpy 2048 KiBytes: 3729 MB/sec
memcpy 4096 KiBytes: 3717 MB/sec
memcpy 8192 KiBytes: 3687 MB/sec
memcpy 16384 KiBytes: 3632 MB/sec
memcpy 32768 KiBytes: 3529 MB/sec
memcpy 65536 KiBytes: 3318 MB/sec
memcpy 131072 KiBytes: 2893 MB/sec
root@gdm-cdm:~#
</pre>

ethernet receive:

<pre>
daq13:bin$ ./ttcp -t -s -n 100000 10.0.0.24
ttcp-t: buflen=8192, nbuf=100000, align=16384/0, port=5001 tcp -> 10.0.0.24
ttcp-t: socket
ttcp-t: connect
ttcp-t: 819200000 bytes in 7.25 real seconds = 110358.39 KB/sec +++
ttcp-t: 100000 I/O calls, msec/call = 0.07, calls/sec = 13794.80
ttcp-t: 0.0user 0.2sys 0:07real 3% 0i+0d 760maxrss 0+2pf 1461+31csw
daq13:bin$

root@gdm-cdm:~# ./ttcp.armv7 -r -s
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 10.0.0.25
ttcp-r: 819200000 bytes in 7.27 real seconds = 110098.22 KB/sec +++
ttcp-r: 212040 I/O calls, msec/call = 0.04, calls/sec = 29181.53
ttcp-r: 0.1user 5.7sys 0:07real 81% 0i+0d 584maxrss 0+2pf 125601+2699csw
root@gdm-cdm:~#
</pre>

ethernet transmit:

<pre>
root@gdm-cdm:~# ./ttcp.armv7 -t -s -n 100000 10.0.0.25
ttcp-t: buflen=8192, nbuf=100000, align=16384/0, port=5001 tcp -> 10.0.0.25
ttcp-t: socket
ttcp-t: connect
ttcp-t: 819200000 bytes in 6.95 real seconds = 115078.69 KB/sec +++
ttcp-t: 100000 I/O calls, msec/call = 0.07, calls/sec = 14384.84
ttcp-t: 0.0user 0.7sys 0:06real 11% 0i+0d 584maxrss 0+2pf 1162+1017csw
root@gdm-cdm:~#

daq13:bin$ ./ttcp -r -s
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 10.0.0.24
ttcp-r: 819200000 bytes in 6.97 real seconds = 114841.84 KB/sec +++
ttcp-r: 161335 I/O calls, msec/call = 0.04, calls/sec = 23160.01
ttcp-r: 0.0user 1.9sys 0:06real 28% 0i+0d 760maxrss 0+2pf 80646+51csw
daq13:bin$
</pre>

= Install Xilinx tools =

* install Vivado 2020.2
<pre>
login at https://www.xilinx.com/myprofile.html
go to "Downloads"
go to archive,
find 2020.2
download Xilinx_Unified_2020.2_1118_1232_Lin64.bin
sh ./Xilinx_Unified_2020.2_1118_1232_Lin64.bin
banner window should open with spinner "downloading installation data"
"a newer version is available" -> say "continue"
next
"select install type" window:
provide email and password,
select "download image"
select directory /home/olchansk/Xilinx/Downloads/2020.2\
select "linux" and "full image"
next
download summary: space required 38.52 Gbytes
download
installation progress
downloading spinner, 16 M/s 47 minutes...
"download image has been created successfully". Ok.
check contents of /home/olchansk/Xilinx/Downloads/2020.2
ls -l /home/olchansk/Xilinx/Downloads/2020.2
total 67
drwxr-xr-x 2 olchansk users 9 Sep 1 16:22 bin
drwxr-xr-x 3 olchansk users 15 Sep 1 16:23 data
drwxr-xr-x 4 olchansk users 4 Sep 1 16:22 lib
drwxr-xr-x 2 olchansk users 644 Sep 1 16:22 payload
drwxr-xr-x 2 olchansk users 7 Sep 1 16:22 scripts
drwxr-xr-x 4 olchansk users 4 Sep 1 16:22 tps
-rwxr-xr-x 1 olchansk users 3256 Nov 18 2020 xsetup
daq13:2020.2$
./xsetup
spinned loading installation data
xilinx design tools 2022.1 now available -> say continue
"welcome" -> next
"select product" -> vivado -> next -> vivado hl system edition -> next
select devices: only zynq ultrascale+ mpsoc -> next
select destination: /opt/Xilinx (as root, mkdir /opt/Xilinx, chmod olchansk.users /opt/Xilinx)
install ...
complete
move /home/olchansk/Xilinx/Downloads/2020.2 to /daq/daqstore/olchansk/Xilinx/Downloads/
</pre>
* install petalinux 2020.2
<pre>
./xsetup
"a newer version is available" -> say "continue"
next
"select product to install" -> select Petalinux (Linux only) -> next
"select destination directory" -> select "/opt/Xilinx" (disk space required 2.64 GB) -> next
"summary" -> install ...
error about missing /tmp/tmp-something files
"installation completed successfully" (hard to dismiss, "ok" button is partially cut-off)
done?
I think it failed, /opt/Xilinx/PetaLinux/2020.2/bin is empty except for petalinux-v2020.2-final-installer.run
try to run it by hand, same error about /tmp/tmp-something files. strange...
notice it complains about "truncate", which truncate finds ~/bin/truncate, get rid of it,
try again
now complains about missing texinfo and zlib1g:i386
apt install texinfo -> ok
apt install zlib1g:i386 -> installs bunch of gcc stuff -> ok
try again
reports "already installed" -> delete /opt/Xilinx/.xinstall/PetaLinux_2020.2/, delete entries in ~/.Xilinx/registry/installedSW.xml
try again
success
</pre>
* install vivado 2022.1 and petalinux 2022.1 - everything is pretty much the same

= Petalinux =

* cd PetaLinux_GDM_CDM
* petalinux-config
* enable i2c MAC address and DHCP
<pre>
git clone https://bitbucket.org:/team-ds-dm/ds-dm-u-boot-xlnx.git
cd ds-dm-u-boot-xlnx
git checkout ds-dm-u-boot-xlnx

linux-components ->
uboot -> ext-local-src
external u-boot local source -> ds-dm-u-boot-xlnx (path to the customized uboot git repository)
</pre>
* enable DHCP
<pre>
Subsystem AUTO Hardware Settings -> Ethernet Settings
randomize MAC address -> NO
ethernet mac address -> leave empty
obtain ip address automatically -> YES
</pre>
* set hostname and product names
<pre>
Firmware Version Configuration ->
Host name -> "ds-dm"
Product name -> "Petalinux_GDM_CDM"
</pre>
* configure linux kernel
<pre>
petalinux-config -c kernel
</pre>
* enable NFS-Root
<pre>
petalinux-config
Image Packaging Configuration > Root File System Type -> set to NFS
Location of NFS root directory set to "/nfsroot"

petalinux-config -c kernel
Networking support > IP: kernel level configuration
enable DHCP, BOOTP, RARP
File systems > Network file systems > Root file systems on NFS
</pre>
* manually fix linux kernel command line:
<pre>
grep nfsroot PetaLinux_GDM_CDM/project-spec/configs/config
edit CONFIG_SUBSYSTEM_BOOTARGS_GENERATED to read
earlycon console=ttyPS0,115200 clk_ignore_unused panic=60 root=/dev/nfs nfsroot=/nfsroot/%s ip=dhcp rw
</pre>
* check configuration in
** PetaLinux_GDM_CDM/project-spec/configs/config
** PetaLinux_GDM_CDM/project-spec/configs/rootfs_config
** PetaLinux_GDM_CDM/components/plnx_workspace/device-tree/device-tree/system-conf.dtsi

= JTAG server =

localhost:3121

= ds20k block =

<pre>
module ds20k
(
// CLOCK INPUTs
input wire clk,
input reset, // pulse for power-up reset
input wire pll_is_locked, // clock cleaner PLL is locked to selected input clock

// REGISTER_DATA

input wire [255:0] [31:0] register_data_in,
output reg [255:0] [31:0] register_data_out,
input wire register_write_strobe, // pulse when AXI write transaction puts new data in register_data_in
input wire register_read_ack, // pulse after AXI read transaction captures data from register_data_out, used to read from FIFO

// GDM QSFP FIBER LINKS
output reg [11:0] [15:0] qsfp_tx_data,
output reg [11:0] [1:0] qsfp_tx_ctrl,
input wire [11:0] [15:0] qsfp_rx_data_error, // not sure what this is.
input wire [11:0] [15:0] qsfp_rx_data,
input wire [11:0] [1:0] qsfp_rx_ctrl,
input wire [11:0] qsfp_rx_is_good, // single bit indicating that RX link is up and data is good.

// CDM SFP FIBER LINKS
output reg [15:0] sfp_tx_data,
output reg [1:0] sfp_tx_ctrl,
input wire [15:0] sfp_rx_data,
input wire [1:0] sfp_rx_ctrl,
input wire sfp_rx_is_good, // single bit indicating that RX link is up and data is good.

// VX_RXs
input wire [3:0] vx1_rx,
input wire [3:0] vx2_rx,
input wire [3:0] vx3_rx,
input wire [3:0] vx4_rx,
input wire [3:0] vx5_rx,
input wire [3:0] vx6_rx,
input wire [3:0] vx7_rx,
input wire [3:0] vx8_rx,
input wire [3:0] vx9_rx,
input wire [3:0] vx10_rx,
input wire [3:0] vx11_rx,
input wire [3:0] vx12_rx,

// VX_TXs
output reg [2:0] vx1_tx_out,
output reg [2:0] vx2_tx_out,
output reg [2:0] vx3_tx_out,
output reg [2:0] vx4_tx_out,
output reg [2:0] vx5_tx_out,
output reg [2:0] vx6_tx_out,
output reg [2:0] vx7_tx_out,
output reg [2:0] vx8_tx_out,
output reg [2:0] vx9_tx_out,
output reg [2:0] vx10_tx_out,
output reg [2:0] vx11_tx_out,
output reg [2:0] vx12_tx_out,

// remove input wire gdm_trg,
// remove input wire gdm_tsm,

// LEMO INPUTs
input wire [4:1] ext_in_lv_async, // direct connection to LEMO connectors, not clocked

// LEMO OUTPUTs
output reg [2:1] ext_out, // direct connection to LEMO connectors, not clocked

// FRONT PANEL LEDs
output reg [3:0] fp_led_out, // direct connection to LEMO connectors, not clocked

// trigger and tsm output
// remove output reg trg_out,
// remove output reg tsm_out
);
</pre>

= world view =

Note:
* red lines: clocks
* green lines: AXI/Avalon packet streams
* blue lines: serial data

[[File:GDM-CDM-VX.drawio.svg]]

== description ==

same thing, in words:

<pre>
detector
digitizer, 125 MHz
digital filter
digital discriminator
hit map, 64 bits at 125 MHz (could be 250 MHz, filter and discriminator clock)
packetizer, 64 bits -> id, timestamp, 8x 8-bit words, eop
8/10 serializer, 12.5 MHz parallel in, 125 MHz serial out
lvds line to CDM vx_rx
BBB: also VX busy to lvds line to CDM vx_rx, do not want to depend on serial comm for vx busy, do not want to inject async data into the hit map packet stream
--- CDM
lvds line capture in IBUF register, vxN_rx clock, 125 MHz clock, from PLL with adjustable phase, scan phase to find sweet spot
vx rx 10/8 deserializer, vxN_rx clock, stobes out parallel data every 10 clocks at 12.5 MHz
vx rx phase transfer from vxN_rx clock to main CDM clock, have 10 clocks for transfer to happen
from here everything is on the CDM main clock
vx rx packet adapter, 12.5 MHz 8-bit packets to 125 MHz 16-bit packets, internal FIFO to avoid data overrun
sfp tx mux, all VX packet streams into one CDM sfp tx packet stream
BBB: capture VX RX busy to CDM main clock (IBUF register), grand-or becomes vx_rx_busy
sfp tx packetizer (data,eop,vx_rx_busy -> data,k)
sfp tx 16/20 serializer, 16 bit at 125 MHz in, 2500 MHz serial out (20 bits at 125 MHz)
sfp tx fiber link to GDM, 2 Gigabits/sec
--- GDM
qsfp rx deserializer, 16 bit at 125 MHz output on main GDM clock
qsfp rx depacketizer (data,k -> packet data,eop; qsfp_rx_busy)
qsfp rx demux, hit map packets routed to GDM trigger logic block
GDM trigger logic block looks at hit map, generates yes/no trigger decision, encodes it as a trigger packet
TSM generator encodes GPS time data as a TSM packet
qsfp tx mux - trigger packets, tsm packets, etc to qsfp tx packet stream
qsfp tx packetizer (data,eop;trg,tsm,bsy -> data,k)
NB: the same tx data is sent to all 12 qsfp tx ports, to make sure we do not accidentally desync the CDMs.
qsfp tx 16/20 serializer, 16-bit at 125 MHz to 20-bit at 250 MHz to 2500 MHz serial out
qsfp tx fiber link to CDM, 2 Gigabits/sec
--- CDM
sfp rx 20/16 deserializer, 16-bit at 125 MHz
sfp rx depacketizer, (data,k -> data,eop;trg_in,tsm_in,bsy_in)
sfp rx demux (in reality, noop, all packets go to same place, vx tx)
vx tx mux (packets from GDM, packets with simulated waveforms from CDM midas frontend via AXI FIFO)
vx tx packet adapter 16-bit at 125 MHz to 8-bit at 12.5 MHz strobed every 10 clocks. NB: most important, this 10-clock strobe runs in sync between all CDMs!
vx tx phase transfer from main CDM clock to vxN_tx clock, have 10 clocks for transfer to happen
from here we run on the vxN_tx clock
vx txN serializer, 8-bit at 12.5 MHz to 125 MHz serial
vx_txN OBUF register, vxN_rx clock, 125 MHz clock, from PLL with adjustable phase, scan phase to find sweet spot
lvds line to vx
BBB: bsy_in from GDM is converted from pulse to level, goes out lvds line to vx, sync to vxN_tx clock
TTT: trg_in from GDM is a pulse, does out lvds line to vx, sync to vxN_tx clock.
NB: tsm is always a packet, bsy is always a signal (no packet), trg can be a packet or signal.
--- VX
lvds data captured by 125 MHz ADC sampling clock (CAEN base firmware logic)
lvds data connected to Yair's block
10/8 deserializer
depacketizer (data,k -> data,eop) to avoid accidental desync, we do not send any no trg, no tsm, no bsy K-codes.
demux
trigger packets go their way (4x 64 bit words of data go to event header: timestamps, hitmap data)
tsm packets go their way (4x 64 bit words of data go to tsm event header: timestamps, GPS time data)
BBB: bsy from lvds line stops waveform acquisition
TTT: trg from lvds line cause waveform acquisition, same as trg packet, but has no timestamp and other data attached to it.
</pre>

= Firmware registers =

== Block 0 ==

<pre>
0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
</pre>

== Block 1 ==

== Block 2 ==

== Block 3 ==

=== DS20k block register map ===

* busybox devmem 0x80013000 32

<pre>
reg | version | xDM | xx | description
0 | 0x20230731 | ALL | RO | ds20k version
0 | 0x20240118 | ALL | RW | ds20k version and command
1 | 0x20230731 | ALL | RW | scratch read/write register
2 | 0x20230731 | ALL | RW | configure inputs and outputs
3 | 0x20230731 | ALL | RW | FP_LED mux
4 | 0x20230731 | ALL | RW | EXT_OUT mux
5 | 0x20230731 | ALL | RO | VX_RX state
6 | 0x20230731 | ALL | RO | VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state
7 | 0x20230731 | ALL | RW | LED_OUT, EXT_OUT, VX_TX outputs
8 | 0x20230731 | ALL | RW | VX_TX mux and config
9 | 0x20230731 | ALL | RW | trigger config
10 | 0x20231013 | ALL | RO | status register
11 | 0x20230731 | ALL | RO | trigger counter
12 | 0x20230731 | ALL | RO | time slice marker counter
13 | 0x20240814 | ALL | RO | GPS 1pps period, 125 MHz
14 | 0x20240814 | ALL | RO | Ru clock 1pps period, 125 MHz
15 | 0x20230811 | CDM | RO | SFP RX status
16 | 0x20230811 | CDM | RW | SFP TX control
17 | 0x20230811 | GDM | RO | QSFP RX data 0, 1
18 | |. |. | 2, 3
19 | |. |. | 4, 5
20 | |. |. | 6, 7
21 | | |. | 8, 9
22 | |. |. | 10, 11
23 | 0x20230811 | GDM | RW | QSFP TX control
24 | 0x20231013 | ALL | RW | trigger pulser period
25 | 0x20231013 | ALL | RW | trigger pulser burst control
26 | 0x20231013 | ALL | RW | tsm pulser period
27 | 0x20231204 | ALL | RW | data fifo CPU to FPGA
28 | 0x20231204 | ALL | RW | packet loopback control
28 | 0x20240118 | ALL | RW | data fifo FPGA to CPU
29 | 0x20231208 | GDM | RW | bitmap of active qsfp ports
29 | 0x20240118 | ALL | RW | packet loopback control
29 | 0x20240510 | ALL | RW | packet routing
30 | 0x20231208 | GDM | RO | qsfp link status ports 0..7
31 | 0x20231208 | GDM | RO | qsfp link status ports 8..11
32 | 0x20240118 | GDM | RW | enabled QSFP ports
33 | 0x20240118 | CDM | RW | enabled VX ports
34,35 | 0x20240118 | ALL | RO | time stamp 64 bits
36,37 | 0x20240118 | ALL | RO | old time stamp
38,39,40 | 0x20240118 | CDM | RO | VX busy counters
41,42,43 | 0x20240118 | GDM | RO | QSFP busy counters
44 | 0x20240118 | CDM | RO | cdm_bsy_up_counter and cdm_bsy_pulse_counter
45 | 0x20240118 | CDM | RO | cdm_veto_pulse_counter and cdm_veto_up_counter
46 | 0x20240118 | GDM | RO | gdm_bsy_pulse_counter and gdm_bsy_up_counter
47 | 0x20240118 | GDM | RO | gdm_bsy_refresh_counter
48 | 0x20240118 | GDM | RO | gdm_veto_up_counter and gdm_veto_pulse_counter
49,50,51 | 0x20240424 | CDM | RO | VX RX serial data monitor, 8 bits per VX port
52,53 | 0x20240424 | CDM | RO | VX serial link status, 4 bits per VX port
54 | 0x20240424 | CDM | RO | VX TX serial data monitor
55 | 0x20240430 | ALL | RO | QSFP, SFP and VX link loss counters
56 | 0x20240510 | ALL | RO | sfp_rx_packet_counter
57 | same | ALL | RO | sfp_tx_packet_counter
58 | same | ALL | RO | qsfp_rx_packet_counter[0]
59 | same | ALL | RO | qsfp_tx_packet_counter
60 | same | ALL | RO | vx_rx_packet_counter[0]
61 | same | ALL | RO | vx_tx_packet_counter
62 | 0x20240719 | CDM | RO | cdm_hitmap_period, ports 0, 1
63 | same | CDM | RO | ports 2, 3
64 | same | CDM | RO | ports 4, 5
65 | same | CDM | RO | ports 6, 7
66 | same | CDM | RO | ports 8, 9
67 | same | CDM | RO | ports 10, 11
68 | 0x20240814 | ALL | RW | GPS control and status
</pre>

=== Register 0 0x80013000 ds20k version ===

on read: ds20k version 0xYYYYMMDD

on write:
* 0x00 - noop - as of version 0x20240118, write a zero after writing a command
* 0x01 - CMD_reset - reset logic to good state
* 0x02 - CMD_arm_ts - arm timestamp reset
* 0x03 - CMD_trg - issue a trigger
* 0x04 - CMD_tsm - issue a tsm
* 0x05 - CMD_vx_rx_reset - reset the VX receive path
* 0x06 - CMD_vx_tx_reset - reset the VX transmit path
* 0x07 - CMD_hitmap_trg - generate a hitmap trigger and data packet
* 0x08 - CMD_trg_pulser_reset - reset the trigger pulser
* 0x09 - CMD_tsm_pulser_reset - reset the tsm pulser

=== Register 1 0x80013004 scratch ===

scratch read-write register

=== Register 2 0x80013008 input and output config ===

<pre>
bit | version | fpga name | description
0 | ALL | lemo_enable | enable LEMO input 1
1 | | | 2
2 | | | 3
3 | | | 4
4 | ALL | lemo_invert | invert LEMO input 1
5 | | | 2
6 | | | 3
7 | | | 4
9 | ALL | ext_out_disable | disable LEMO output 1
10 | | | 2
11 | ALL | ext_out_invert | invert LEMO output 1
12 | | | 2
</pre>

=== Register 3 0x8001300C FP_LED control ===

<pre>
wire [15:0] led_out_mux_sel = register_data_in[3][15:0];
wire [3:0] led_out_invert = register_data_in[3][19:16];
</pre>

<pre>
led_out_mux_sel is 4 groups (one per LED) of 4 bits (choice 0..15):
mux | version | fpga name | description
0 | ALL | | power on default
1 | ALL | led_out_reg | register 7 bits
2 | 0x20231013 | pll_locked | clock chip PLL is locked
3 | 0x20231013 | sfp_link_status | SFP link is good
3 | 0x20240118 | sfp_link_status or qsfp_tx_link_rx_status | SFP/QSFP link is good
4 | ALL | lemo_in_sync[1] | LEMO input 1
5 | ALL | lemo_in_sync[2] | LEMO input 2
6 | ALL | lemo_in_sync[3] | LEMO input 3
7 | ALL | lemo_in_sync[4] | LEMO input 4
8 | ALL | ext_out[1] | LEMO output 1
9 | ALL | ext_out[2] | LEMO output 2
A | 0x20231013 | trg_in | trigger
B | 0x20231013 | tsm_in | time slice marker
C | 0x20240118 | gdm_bsy | GDM busy: OR of all CDM busy
D | 0x20240118 | cdm_bsy | CDM busy: OR of all VX busy
E | 0x20240118 | cdm_veto | GDM busy -> GDM veto -> CDM veto -> VX trigger veto
F | ALL | | fixed logic level 1
</pre>

=== Register 4 0x80013010 LEMO OUT control ===

<pre>
wire [7:0] ext_out_mux_sel = register_data_in[4][7:0];
</pre>

<pre>
ext_out_mux_sel is 2 groups (one per LEMO) of 4 bits (choice 0..15):

mux | version | fpga name | description
0 | ALL | | power on default
1 | ALL | ext_out_reg | register 7
2 | ALL | trg_pulser | pulser trigger
3 | 0x20240724 | vx1_tx_out[2] | vx1 serial data out
4 | 0x20240724 | vx1_rx[1] | vx1 serial data in
5 | 0x20240724 | vx_rx_iob[0] | vx1 serial data in captured by IOB register
6 | ALL | lemo_in_async[1]| test synchronizer
7 | ALL | lemo_in_sync[1] | test synchronizer
8 | 0x20240724 | trg_in_pulse | trigger signal
9 | 0x20240724 | tsm_in_pulse | time slice marker signal
A | 0x20240118 | cdm_bsy | CDM busy from VX
B | 0x20240118 | gdm_bsy | GDM busy from CDM
C | 0x20240118 | cdm_veto | veto from GDM to CDM to VX
D | 0x20240118 | vx1_rx[1] | serial data VX to CDM
E | not used (sink)
F | ALL | 1 | fixed logic level 1
</pre>

=== Register 5 0x80013014 VX_RX status ===

<pre>
assign register_data_out[5] = {
vx8_rx[3], vx8_rx[2], vx8_rx[1], vx8_rx[0],
vx7_rx[3], vx7_rx[2], vx7_rx[1], vx7_rx[0],
vx6_rx[3], vx6_rx[2], vx6_rx[1], vx6_rx[0],
vx5_rx[3], vx5_rx[2], vx5_rx[1], vx5_rx[0],
vx4_rx[3], vx4_rx[2], vx4_rx[1], vx4_rx[0],
vx3_rx[3], vx3_rx[2], vx3_rx[1], vx3_rx[0],
vx2_rx[3], vx2_rx[2], vx2_rx[1], vx2_rx[0],
vx1_rx[3], vx1_rx[2], vx1_rx[1], vx1_rx[0]
};
</pre>

=== Register 6 0x80013018 VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state ===

<pre>
assign register_data_out[6] = {
vx2_tx_out[2], vx2_tx_out[1], vx2_tx_out[0], vx1_tx_out[2],
vx1_tx_out[1], vx1_tx_out[0], ext_out[2], ext_out[1],
fp_led_out[3], fp_led_out[2], fp_led_out[1], fp_led_out[0],
ext_in_lv[4], ext_in_lv[3], ext_in_lv[2], ext_in_lv[1],
vx12_rx[3], vx12_rx[2], vx12_rx[1], vx12_rx[0],
vx11_rx[3], vx11_rx[2], vx11_rx[1], vx11_rx[0],
vx10_rx[3], vx10_rx[2], vx10_rx[1], vx10_rx[0],
vx9_rx[3], vx9_rx[2], vx9_rx[1], vx9_rx[0]
};
</pre>

<pre>
bit | version | fpga name | description
0 | ALL | vx9_rx | VX_RX
1 | | |
2 | | |
3 | | |
4 | ALL | vx10_rx | VX_RX
5 | | |
6 | | |
7 | | |
8 | ALL | vx11_rx | VX_RX
9 | | |
10 | | |
11 | | |
12 | ALL | vx12_rx | VX_RX
13 | | |
14 | | |
15 | | |
16 | ALL | ext_in_lv | LEMO inputs
17 | | |
18 | | |
19 | | |
20 | ALL | FP_LED | FP_LEDs
21 | | |
22 | | |
23 | | |
24 | ALL | ext_out[1] | LEMO outputs
25 | | ext_out[2] |
26 | ALL | vx1_tx | VX1_TX
27 | | |
28 | | |
29 | ALL | vx2_tx | VX2_TX
30 | | |
31 | | |
</pre>

=== Register 7 0x8001301C LED_OUT, EXT_OUT, VX_TX outputs ===

<pre>
wire [3:0] led_out_reg = register_data_in[7][3:0];
wire [2:1] ext_out_reg = register_data_in[7][5:4];
// register_data_in[7][6];
// register_data_in[7][7];
wire [7:0] vx_tx_out_reg = register_data_in[7][15:8];
</pre>

<pre>
bit | version | fpga name | description
0 | ALL | led_out_reg | FP_LED 1
1 | | | 2
2 | | | 3
3 | | | 4
4 | ALL | ext_out_reg | LEMO OUT 1
5 | | | 2
6 | - | |
7 | - | |
8 | ALL | vx_tx_out_reg | VX1_TX 0
9 | | | 1
10 | | | 2
11 | | | -
12 | ALL | | VX2_TX 0
13 | | | 1
14 | | | 2
15 | | | -
16 | - | |
17 | | |
18 | | |
19 | | |
20 | | |
21 | | |
22 | | |
23 | | |
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 8 0x80013020 VX_TX config ===

<pre>
wire [3:0] vx_tx_mux_sel = register_data_in[8][3:0];
vx_tx_loopback = register_data_in[8][31];
</pre>

<pre>
vx_tx_mux_sel is 4 bits (choice 0..15):
0 - power on default, control by vx_tx_out_reg
1 - GDM
2 - CDM
3 - pulser loopback test
4 - pulser loopback test
5 - 62.5 MHz output
6 - 125 MHz output
7 - trg, tsm, serial
8 - trg, veto, serial
9 - GPS box control (ds20k rev 0x20240814)
10
11
12
13
14
15 - production config: trg, veto, serial
</pre>

=== Register 9 0x80013024 trigger control ===

from version 0x20240724

<pre>
wire [15:0] trg_src_mask = register_data_in[9][15:0];
wire [31:16] tsm_src_mask = register_data_in[9][31:16];
</pre>

<pre>
wire [15:0] trg_src_bits =
{
1'b0,
1'b0,
1'b0,
1'b0,

1'b0, // gdm_hitmap_trigger,
cdm_hitmap_trigger,
sfp_rx_tsm,
sfp_rx_trg,

sfp_rx_data[1],
sfp_rx_data[0],
tsm_pulser,
trg_pulser,

lemo_in_sync[4],
lemo_in_sync[3],
lemo_in_sync[2],
lemo_in_sync[1]
};

wire [15:0] trg_bits = trg_src_bits & trg_src_mask;
wire [15:0] tsm_bits = trg_src_bits & tsm_src_mask;
</pre>

before that:

<pre>
wire [7:0] trg_src_mask = register_data_in[9][7:0];
wire [7:0] tsm_src_mask = register_data_in[9][15:8];
wire trg_pulser_enable = register_data_in[9][16];
wire tsm_pulser_enable = register_data_in[9][17];
wire trg_software = register_data_in[9][18];
wire tsm_software = register_data_in[9][19];
// bits 20:31 not used
</pre>

<pre>
wire [7:0] xxx_src_bits =
{
sfp_rx_data[1],
sfp_rx_data[0],
tsm_pulser & tsm_pulser_enable,
trg_pulser & trg_pulser_enable,
lemo_in_sync[4],
lemo_in_sync[3],
lemo_in_sync[2],
lemo_in_sync[1]
};
</pre>

<pre>
trg_src_mask and tsm_src_mask bits:
0 - LEMO IN 1
1 - LEMO IN 2
2 - LEMO IN 3
3 - LEMO IN 4
4 - trg_pulser
5 - tsm_pulser
6 - sfp_rx_data[0] // to become sfp_trg_in, selected from sfp_rx_data[0], cdm_rx trg_out and gdm trigger packet
7 - sfp_rx_data[1] // to become sfp_tsm_in, selected from sfp_rx_data[1], cdm_rx tsm_out and gdm tsm packet
</pre>

=== Register 10 0x80013028 status register ===

<pre>
bit | version | fpga name | description
0 | 0x20231013 | pll_locked | clock chip PLL is locked
1 | 0x20240118 | ts_reset_armed | timestamp reset is armed
2 | 0x20240118 | qsfp_tx_link_rx_status | QSFP link status is good for all enabled ports
3 | 0x20240424 | vx_tx_link_rx_status | VX link status is good for all enabled ports
4 | 0x20240118 | cdm_bsy | VX busy grand-or
5 | 0x20240118 | gdm_bsy | QSFP busy grand-or
6 | 0x20240118 | gdm_veto | gdm_veto = gdm_busy
7 | 0x20240118 | cdm_veto | CDM veto from GDM to VX
8 | 0x20240725 | bor_started | begin-of-run sequence started
9 | 0x20240725 | bor_finished | begin-of-run sequence fininished, see commands 10 and 11
10 | | |
11 | | |
12 | | |
13 | | |
14 | | |
15 | | |
16 | | |
17 | | |
18 | | |
19 | | |
20 | | |
21 | | |
22 | | |
23 | | |
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 11 0x8001302C trg_counter ===

trigger counter

=== Register 12 0x80013030 tsm_counter ===

time slice marker counter

=== Register 13 0x80013034 GPS 1pps period ===

GPS 1pps period in 8 ns clocks

=== Register 14 0x80013038 Rb clock 1pps period ===

PRS-10 Rb clock 1pps output period in 8 ns clocks

=== Register 15 0x8001303C SFP RX status ===

<pre>
bit | ds20k version | fpga signal name | description
0 | ALL | sfp_rx_data[15:0] | cdm sfp received data
16 | same | sfp_rx_data_is_k[0] |
17 | same | sfp_rx_data_is_k[1] |
18 | | 0 |
19 | 0x20231204 | sfp_rx_sel_lpb | sfp tx->rx loopback
20 | 0x20231013 | sfp_link_status | sfp link connected, exchanging data
21 | same | sfp_link_rx_status | sfp link receiving correct idle pattern from GDM TX
22 | same | sfp_link_error | sfp link receiver error (badk or overflow)
23 | same | sfp_rx_data_error | sfp transceiver state machine is in error state
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 16 0x80013040 SFP TX control ===

<pre>
wire [15:0] sfp_tx_data_reg = register_data_in[16][15:0];
wire [1:0] sfp_tx_ctrl_reg = register_data_in[16][17:16];
wire sfp_rx_sel_lpb = register_data_in[16][28]; // TX->RX serial loopback
wire sfp_tx_sel_lpb = register_data_in[16][29]; // RX->TX serial loopback
wire sfp_tx_sel_trg = register_data_in[16][30]; // 16 individual bits
wire sfp_tx_sel_reg = register_data_in[16][31]; // from register
</pre>

=== Register 17-22 0x80013044 QSFP RX data ===

QSFP RX data links 0..11

=== Register 23 0x8001305C QSFP TX control ===

<pre>
wire [15:0] qsfp_tx_data_reg = register_data_in[23][15:0];
wire [1:0] qsfp_tx_ctrl_reg = register_data_in[23][17:16];
wire qsfp_rx_sel_lpb = register_data_in[23][28]; // TX->RX loopback
wire qsfp_tx_sel_lpb = register_data_in[23][29]; // RX->TX loopback
wire qsfp_tx_sel_trg = register_data_in[23][30]; // 16 individual bits
wire qsfp_tx_sel_reg = register_data_in[23][31]; // data from register
</pre>

=== Register 24 0x80013060 trigger pulser period ===

trigger pulser period in units of 8 ns (125 MHz clock)

=== Register 25 0x80013064 trigger burst pulser ===

<pre>
wire [7:0] conf_burst_count = conf_pulser_burst_ctrl[31:24];
wire [23:0] conf_burst_period = conf_pulser_burst_ctrl[23:0];
</pre>

=== Register 26 0x80013068 tsm pulser period ===

time slice marker period in units of 8 ns (125 MHz clock)

=== Register 27 0x8001306C data write fifo ===

<pre>
wire fifo_reset = register_data_in[27][31];
wire fifo_to_fpga_wr1 = register_data_in[27][27];
wire fifo_to_fpga_wr2 = register_data_in[27][26];
wire [16:0] fifo_to_fpga_din = register_data_in[27][16:0];
</pre>

<pre>
assign register_data_out[27][31:24] = register_data_in[27][31:24]; // echo write bits
assign register_data_out[27][23] = fifo_to_fpga_full;
assign register_data_out[27][22] = fifo_to_fpga_empty;
</pre>

=== Register 28 0x80013070 data read fifo ===

<pre>
wire fifo_reset = register_data_in[28][31];
wire fifo_from_fpga_rd1 = register_data_in[28][25];
wire fifo_from_fpga_rd2 = register_data_in[28][24];
</pre>

<pre>
assign register_data_out[28][31:24] = register_data_in[28][31:24]; // echo write bits
assign register_data_out[28][21] = fifo_from_fpga_full;
assign register_data_out[28][20] = fifo_from_fpga_empty;
assign register_data_out[28][16:0] = fifo_from_fpga_dout;
</pre>

=== Register 29 0x80013074 packet routing ===

Control packet routing and loopbacks:

<pre>
wire [3:0] dn_route_ctrl = register_data_in[29][3:0];
wire [3:0] up_route_ctrl = register_data_in[29][7:4];
wire [3:0] fifo_to_fpga_route_ctrl = register_data_in[29][11:8];
//wire [3:0] spare_route_ctrl = register_data_in[29][15:12];

wire dn_mux_trg_enable = register_data_in[29][16];
wire dn_mux_tsm_enable = register_data_in[29][17];
wire dn_mux_sfp_rx_fifo_enable = register_data_in[29][18];
// 19
// 20..23

wire up_mux_vx_rx_enable = register_data_in[29][24];
// 25
// 26
// 27
wire fifo_from_fpga_hitmap_enable = register_data_in[29][28];
// 29..31
</pre>

fifo_to_fpga output routing:
* 0 - to down packet mux
* 1 - to up packet mux
* 2 - to fifo_from_fpga mux
* 3 - not used

down packet mux inputs:
* fifo_to_fpga_0
* trg_pkt16 enabled by dn_mux_trg_enable
* tsm_pkt16 enabled by dn_mux_tsm_enable
* sfp_rx_fifo_pkt16 enabled by dn_mux_sfp_rx_fifo_enable
* up_pkt16_2 loopback from up packet mux

down packet mux output routing:
* 0 - to vx_tx_pkt16 to VX TX (vx_link_tx and vx_ser_tx)
* 1 - to fifo_from_fpga mux
* 2 - to up packet mux loopback
* 3 - to qsfp_tx_pkt16 to GDM QSFP TX (cdm_link_tx)

up packet mux inputs:
* vx_rx_pkt16 enabled by up_mux_vx_rx_enable data from 12 VX RX links
* fifo_to_fpga_1
* dn_pkt16_2 loopback from down packet mux

up packet mux output routing:
* 0 - to sfp_tx_pkt16 to CDM SFP TX (VX data to GDM)
* 1 - to fifo_from_fpga mux
* 2 - to down packet mux loop loopback
* 3 - not used

fifo_from_fpga mux inputs:
* fifo_to_fpga_2
* dn_pkt16_1 from down packet mux
* up_pkt16_1 from up packet mux
* qsfp_tx_pkt16 from GDM QSFP link 0 (there is no GDM QSFP 12-to-1 mux)
* hitmap_pkt16 enabled by fifo_from_fpga_hitmap_enable in register 29

=== Register 30 0x80013078 qsfp link status ports 0..7 ===

<pre>
assign register_data_out[30] =
{
qsfp_rx_data_error[7], qsfp_link_error[7], qsfp_link_status[7], qsfp_link_rx_status[7], // 7
qsfp_rx_data_error[6], qsfp_link_error[6], qsfp_link_status[6], qsfp_link_rx_status[6], // 6
qsfp_rx_data_error[5], qsfp_link_error[5], qsfp_link_status[5], qsfp_link_rx_status[5], // 5
qsfp_rx_data_error[4], qsfp_link_error[4], qsfp_link_status[4], qsfp_link_rx_status[4], // 4
qsfp_rx_data_error[3], qsfp_link_error[3], qsfp_link_status[3], qsfp_link_rx_status[3], // 3
qsfp_rx_data_error[2], qsfp_link_error[2], qsfp_link_status[2], qsfp_link_rx_status[2], // 2
qsfp_rx_data_error[1], qsfp_link_error[1], qsfp_link_status[1], qsfp_link_rx_status[1], // 1
qsfp_rx_data_error[0], qsfp_link_error[0], qsfp_link_status[0], qsfp_link_rx_status[0] // 0
};
</pre>

=== Register 31 0x8001307C qsfp link status ports 8..11 ===

<pre>
assign register_data_out[31] =
{
4'b0000,
4'b0000,
4'b0000,
4'b0000,
qsfp_rx_data_error[11], qsfp_link_error[11], qsfp_link_status[11], qsfp_link_rx_status[11], // 11
qsfp_rx_data_error[10], qsfp_link_error[10], qsfp_link_status[10], qsfp_link_rx_status[10], // 10
qsfp_rx_data_error[9], qsfp_link_error[9], qsfp_link_status[9], qsfp_link_rx_status[9], // 9
qsfp_rx_data_error[8], qsfp_link_error[8], qsfp_link_status[8], qsfp_link_rx_status[8] // 8
};
</pre>

=== Register 32 0x80013080 bitmap of enabled qsfp ports ===

<pre>
wire [11:0] qsfp_mask = register_data_in[32][11:0];
wire qsfp_bsy_force = register_data_in[32][12];
// not used = register_data_in[32][15:13];
wire [15:0] gdm_veto_extend = register_data_in[32][31:16];
</pre>

On the GDM:

qsfp_rx_bsy[11..0] are pulses received from the CDMs

qsfp_rx_bsy_or is the grand-or of qsfp_rx_bsy masked by qsfp_mask

if qsfp_rx_bsy_or is high, gdm_bsy goes up and stays up for vx_bsy_extend*2 clocks

gdm_veto = gdm_bsy (bsy of any one VX causes trigger veto to all of them)

gdm_veto transition 0->1 causes gdm_veto_pulse. as long as gdm_veto is high, gdm_veto_pulse is generated every gdm_veto_extend clocks

gdm_veto_pulse is sent to all CDMs.

For this to work right, cdm_bsy_extend should not be bigger than gdm_veto_extend.

=== Register 33 0x80013084 bitmap of enabled VX ports ===

<pre>
wire [11:0] vx_mask = register_data_in[33][11:0];
wire vx_bsy_force = register_data_in[33][12];
// not used = register_data_in[33][15:13];
wire [15:0] vx_bsy_extend = register_data_in[33][31:16];
</pre>

On the CDM:

cdm_bsy is a grand-or of all vx_bsy masked by vx_mask (list of active VXes).

vx_bsy_extend controls how often state of cdm_bsy is sent to the GDM. when cdm_bsy goes 0->1, we send a cdm_bsy_pulse and keep resending it every vx_bsy_extend clocks as long as cdm_bsy stays high.

cdm_bsy_pulse is sent to the GDM.

sfp_rx_veto is the received from the GDM

if sfp_rx_veto goes up, cdm_veto goes up and stays up for gdm_veto_extend clocks.

for this to work right, CDM gdm_veto_extend must be bigger than GDM gdm_veto_extend.

cdm_veto goes to VXes on v1_tx_out[1] which is LVDS input 13.

=== Register 34, 35 0x80013088, 8C current timestamp ===

current 64-bit timestamp, 125 MHz

=== Register 36, 37 0x80013090, 94 old timestamp ===

old 64-bit timestamp, 125 MHz. timestamp saved at run start when it is reset to 0.

=== Register 38, 39, 40 0x80013098, 9C, A0 VX busy counters ===

8 bits per VX port, counters overflow to 255, reset at run start.

=== Register 41, 42, 43 0x800130A4, A8, AC QSFP busy counters ===

8 bits per QSFP port, counters overflow to 255, reset at run start.

=== Register 44, 45, 46, 47, 48 0x800130B0, B4, B8, BC, C0 CDM and GDM busy and veto counters ===

* cdm_busy = grand-or of all VX busy for enabled VXes
* gdm_busy = grand-or of all CDM busy for enabled CDMs
* gdm_veto = gdm_busy

<pre>
44 | lo 16 bits | cdm_bsy_up_counter | CDM busy, increments when cdm_busy goes 0->1
44 | hi 16 bits | cdm_bsy_pulse_counter | CDM busy to GDM, increments for each cdm_bsy_pulse sent to the GDM

45 | lo 16 bits | cdm_veto_pulse_counter | CDM veto from GDM, increments for each sfp_rx_veto received from the GDM
45 | hi 16 bits | cdm_veto_up_counter | CDM veto to VX, increments each time cdm_veto is set to 1.

46 | lo 16 bits | gdm_bsy_pulse_counter | GDM busy from CDM, increments for each qsfp_rx_busy received from CDMs (unless they overlap)
46 | hi 16 bits | gdm_bsy_up_counter | GDM busy, increments each time gdm_bsy goes 0->1

47 | lo 16 bits | gdm_bsy_refresh_counter | GDM busy, increments each time gdm_bsy is extended by new qsfp_rx_busy
47 | hi 16 bits | spare |

48 | lo 16 bits | gdm_veto_up_counter | GDM veto, increments each time gdm_veto goes 0->1
48 | hi 16 bits | gdm_veto_pulse_counter | GDM veto to CDM, increments for each gdm_veto_pulse sent to the CDM
</pre>

=== Register 49, 50, 51 0x800130xx VX RX serial data monitor ===

VX RX data, 8-bit per VX channel. k-bit is omitted.

<pre>
assign register_data_out[49][7:0] = vx_rx_data[0]; // vx1
assign register_data_out[49][15:8] = vx_rx_data[1]; // vx2
assign register_data_out[49][23:16] = vx_rx_data[2]; // vx3
assign register_data_out[49][31:24] = vx_rx_data[3]; // vx4

assign register_data_out[50][7:0] = vx_rx_data[4]; // vx5
assign register_data_out[50][15:8] = vx_rx_data[5]; // vx6
assign register_data_out[50][23:16] = vx_rx_data[6]; // vx7
assign register_data_out[50][31:24] = vx_rx_data[7]; // vx8

assign register_data_out[51][7:0] = vx_rx_data[8]; // vx9
assign register_data_out[51][15:8] = vx_rx_data[9]; // vx10
assign register_data_out[51][23:16] = vx_rx_data[10]; // vx11
assign register_data_out[51][31:24] = vx_rx_data[11]; // vx12
</pre>

=== Register 52, 53 VX link status ===

<pre>
assign register_data_out[52] =
{
vx_rx_error[7], vx_link_error[7], vx_link_status[7], vx_link_rx_status[7], // 7
vx_rx_error[6], vx_link_error[6], vx_link_status[6], vx_link_rx_status[6], // 6
vx_rx_error[5], vx_link_error[5], vx_link_status[5], vx_link_rx_status[5], // 5
vx_rx_error[4], vx_link_error[4], vx_link_status[4], vx_link_rx_status[4], // 4
vx_rx_error[3], vx_link_error[3], vx_link_status[3], vx_link_rx_status[3], // 3
vx_rx_error[2], vx_link_error[2], vx_link_status[2], vx_link_rx_status[2], // 2
vx_rx_error[1], vx_link_error[1], vx_link_status[1], vx_link_rx_status[1], // 1
vx_rx_error[0], vx_link_error[0], vx_link_status[0], vx_link_rx_status[0] // 0
};

assign register_data_out[53] =
{
//4'b0000,
1'b0, ~vx_rx_deser_rdy[0], vx_rx_code_err[0], vx_rx_disp_err[0],
//4'b0000,
//4'b0000,
//4'b0000,
vx_rx_monitor[0],
vx_rx_error[11], vx_link_error[11], vx_link_status[11], vx_link_rx_status[11], // 11
vx_rx_error[10], vx_link_error[10], vx_link_status[10], vx_link_rx_status[10], // 10
vx_rx_error[9], vx_link_error[9], vx_link_status[9], vx_link_rx_status[9], // 9
vx_rx_error[8], vx_link_error[8], vx_link_status[8], vx_link_rx_status[8] // 8
};
</pre>

vx_rx_monitor (12-bit) is from deserializer_10b.sv:

<pre>
assign monitor_out[9:0] = lastByte[9:0];
assign monitor_out[10] = comma;
assign monitor_out[11] = ready;
</pre>

=== Register 54 VX TX serial data monitor ===

* contents of vx_tx_monitor from vx_ser_tx.sv:
<pre>
assign monitor_out[8:0] = data_to_encoder; // 8-bit + k
assign monitor_out[9] = valid;
assign monitor_out[15:10] = 0;
assign monitor_out[25:16] = encoded_data; // 10-bit
assign monitor_out[26] = encoded_valid;
assign monitor_out[27] = 0;
assign monitor_out[31:28] = 0;
</pre>

=== Register 55 QSFP, SFP, VX link loss counters ===

<pre>
assign register_data_out[55] = {
vx_rx_error_counter,
qsfp_link_rx_status_drop_counter,
sfp_link_rx_status_drop_counter,
vx_link_rx_status_drop_counter
};
</pre>

<pre>
24..31 - CDM VX RX error counters, count any errors in the VX RX data path (bad serial data, fifo overflow, etc)
16..23 - GDM QSFP link loss counter, increments on qsfp_tx_link_rx_status 1->0 (reg10)
8..15 - CDM SFP link loss counter, increments on sfp_link_rx_status 1->0 (reg15)
0...7 - CDM VX link loss counter, increments on vx_tx_link_rx_status 1->0 (reg10)
</pre>

=== Register 68 GPS control and status ===

<pre>
assign register_data_out[68] =
{
8'b00000000, // 23+8
rb_1pps_counter, // 16+8 bits
gps_1pps_counter, // 8+8 bits
1'b0,
1'b0,
rb_ser_in, // 5
gps_data_in, // 4
gps_aux_out, // 3
gps_aux_in, // 2
rb_1pps_in, // 1
gps_1pps_in // 0
};
</pre>

= Firmware registers branch develop_ko =

== Register map ==

<pre>
0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
1 | ALL | ALL | RW | read write scratch register
2 | ALL | CDM | ?? | MGT not used
3 | ALL | CDM | RO | MGT debug_data
4 | ALL | CDM | RW | clk_config_vec
5 | ALL | CDM | ?? | not used
6 | ALL | CDM | RO | CDM_link_data_processing:o_error_time
7 | ALL | CDM | RO | CDM_link_data_processing:o_error_count
</pre>

== register 0 0x80010000 ==

GDM:
<pre>
0 - gdm_link_interface:i_mgt_rst
2 - gdm_link_interface:i_link_down_latched_rst
8 - GDM_link_data_processing:i_rst
10..9 - GDM_link_data_processing:i_data_mode
</pre>

CDM:
<pre>
0 - cdm_link_interface:i_mgt_rst
2 - cdm_link_interface:i_link_down_latched_rst
8 - CDM_link_data_processing:i_rst
10..9 - CDM_link_data_processing:i_data_mode
</pre>

== register 1 0x80010004 ==

GDM:
<pre>
nlinks-1..0 - gdm_link_interface:i_rx_slide_trigger
</pre>

CDM:
<pre>
nlinks-1..0 - cdm_link_interface:i_rx_slide_trigger
</pre>

== register 2 0x80010008 ==

GDM:
<pre>
nlinks-1..0 - gdm_link_interface:o_link_power_good
nlinks+15..16 - gdm_link_interface:o_link_status
</pre>

CDM:
<pre>
nlinks-1..0 - cdm_link_interface:o_link_power_good
nlinks+15..16 - cdm_link_interface:o_link_status
</pre>

== register 3 0x8001000c ==

GDM: simple loopback register

CDM:
<pre>
31..0 - debug_data - cdm_link_interface:o_debug
</pre>

o_debug:
<pre>
rx_link_rst & rx_error & rx_link_up & rx_receiving_data &
std_logic_vector(rx_state_count) & tx_state_count_on_rx_clk & i_rx_ctrl3(0) &
i_rx_ctrl1(1 downto 0) & i_rx_ctrl0(1 downto 0) &
rx_data_is_k28p1_k28p5 &
i_rx_data;
</pre>

== register 4 0x80010010 ==

GDM write:
<pre>
0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS
</pre>

GDM read:
<pre>
0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS
4 - clk_config_vec(4) - CLK_LOSXTn_LS
5 - clk_config_vec(5) - CLK_LOLn_LS
6 - clk_config_vec(6) - CLK_INTn_LS
7 - constant 1
31..8 - constant 0
</pre>

== register 5 0x80010014 ==

not used

== register 6 0x80010018 ==

GDM:
<pre>
3..0 - GDM_link_data_processing:i_status_select
</pre>

CDM:
<pre>
31..0 - CDM_link_data_processing:o_error_count
</pre>

== register 7 0x8001001c ==

GDM:
<pre>
31..0 - GDM_link_data_processing:o_status_vector
</pre>

CDM:
<pre>
31..0 - CDM_link_data_processing:o_error_count
</pre>

= GDM, CDM, VX packet communications =

* timestamp math
<pre>
1 clock is 8 ns is 125 MHz
8 bits of clocks is 256 clocks is 2048 ns is ~2 usec
16 bits of clocks is ~500 usec is 0.5 msec
24 bits of clocks is ~134 msec
32 bits of clocks is ~34 sec
40 bits of clocks is ~8.7 ksec is 2.4 hours
48 bits of clocks is ~625 hours is ~26 days
56 bits of clocks is ~6.6 kdays is ~18 kyears
62 bits of clocks is ~10 Mhours is 427 kdays is ~1.1 kyears
64 bits of clocks is ~4.4 kyears
</pre>
* 0x02 - TRG packet, 8 bytes, 80 adc clocks, 640 ns on lvds link
<pre>
0 - 0x02
1 - trg_counter[7:0]
2 - ts64 low byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 high byte 3
6 - trg_in_latch[7:0]
7 - trg_in_latch[15:8]
</pre>
* 0x03 - HITMAP_TRG packet, 12 bytes, 120 adc clocks, 960 ns on lvds link
<pre>
0 - 0x03
1 - trg_counter[7:0]
2 - ts64 low byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 high byte 3
6 - vx_bitmap[7:0]
7 - vx_bitmap[15:8]
8 - vx_bitmap[23:16]
9 - vx_bitmap[31:24]
10 - vx_bitmap[39:32]
11 - vx_bitmap[47:40]
</pre>
* 0x10 - TSM packet, 26 bytes, 260 adc clocks, 2080 ns on lvds link
<pre>
0 - 0x10
1 - tsm_counter[7:0]
2 - gdm_ts64 low byte 0
3 - 1
4 - 2
5 - 3
6 - 4
7 - 5
8 - 6
9 - gdm_ts64 high byte 7
10 - gps_ts64 low byte 0
11 - 1
12 - 2
13 - 3
14 - 4
15 - 5
16 - 6
17 - gps_ts64 high byte 7
18 - gps_data64 low byte 0
19 - 1
20 - 2
21 - 3
22 - 4
23 - 5
24 - 6
25 - gps_data64 high byte 7
</pre>
* 0x81 - VX hitmap packet, 10 bytes, 100 adc clocks, 800 ns on lvds link, 48 ns on fiber link
<pre>
0 - 0x81
1 - VX ID
2 - hitmap low byte, nits 7:0
3 - 15:8
4 - 23:16
5 - 31:24
6 - ...:32
7 - ...
8 - ...
9 - hitmap low byte, bits 63:...
</pre>
* 0x82 - CDM hitmap packet, 108 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x82
1 - cdm_hitmap_trigger_counter[7:0]
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - cdm_hitmap_or12 byte 0 (7:0)
11 - cdm_hitmap_or12 byte 1 (11:8) plus 4 bits: 12=0, 13=0, 14=0, 15=cdm_hitmap_grand_or
12 - cdm_hitmap_data, low byte, 12*64 bits = 768 bits = 48 words = 96 bytes
...
107 - cdm_hitmap_data, high byte
</pre>

= AXI bus timing =

* AXI 100 MHz clock, 10 ns, 32-bit data
* AXI single-dword read: 36 clock repeat rate, 360 ns is 2.777 MHz, 4 bytes per transfer is 11.11 Mbytes/sec
* AXI single-qword read: 13 clock repeat rate, 130 ns is 7.7 MHz, 4 bytes per transfer is 30 Mbytes/sec, not accounting for the gap
* AXI memcpy read: 4 transfers at 13 clocks, gap, 4 transfers at 13 clocks, gap, etc. below 30 Mbytes/sec.
[[Image:Ds-dm-axi-read.png|100px]]
[[Image:Ds-dm-axi-read-64.png|100px]]
[[Image:Ds-dm-axi-read-memcpy.png|100px]]
* AXI single-dword write: 20 clocks repeat rate, 200 ns is 5 MHz, 4 bytes per transfer is 20 Mbytes/sec
* AXI single-qword write: 13+20 clocks repeat rate, 330 ns is 3 MHz, 16 bytes per burst is 48 Mbytes/sec
[[Image:Ds-dm-axi-write.png|100px]]
[[Image:Ds-dm-axi-write-64.png|100px]]

= AXI bus addresses =

* see AXI/AMBA addresses assigned inthe FPGA project: (s_axi/reg0 is the DS-DM AXI registers)
<pre>
daq00:ds-dm-gcdm$ grep assign_bd_address scripts/GDM_CDM_XU8_bd.tcl
assign_bd_address -offset 0x80010000 -range 0x00004000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs axi_register_interfa_0/s_axi/reg0] -force
assign_bd_address -offset 0x000400000000 -range 0x40000000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs ddr4/C0_DDR4_MEMORY_MAP/C0_DDR4_ADDRESS_BLOCK] -force
assign_bd_address -offset 0x80000000 -range 0x00010000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs system_management_wiz/S_AXI_LITE/Reg] -force
daq00:ds-dm-gcdm$
</pre>

* see AXI/AMBA addresses exported from FPGA project to Linux kernel: (uio for debug bridge should say "debug bridge")
<pre>
root@gdm0:~# cat /sys/class/uio/uio*/name
axi-pmon
axi-pmon
axi-pmon
axi-pmon
root@gdm0:~#
root@gdm0:~# ls -l /sys/class/uio/
total 0
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio0 -> ../../devices/platform/amba/ffa00000.perf-monitor/uio/uio0
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio1 -> ../../devices/platform/amba/fd0b0000.perf-monitor/uio/uio1
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio2 -> ../../devices/platform/amba/fd490000.perf-monitor/uio/uio2
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio3 -> ../../devices/platform/amba/ffa10000.perf-monitor/uio/uio3
root@gdm0:~# ls -l /sys/class/uio/../../devices/platform/amba/
total 0
-rw-r--r-- 1 root root 4096 Oct 18 19:37 driver_override
drwxr-xr-x 3 root root 0 Oct 18 01:36 fd070000.memory-controller
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd0b0000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd400000.zynqmp_phy
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd490000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd500000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd510000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd520000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd530000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd540000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd550000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd560000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd570000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd6e0000.cci
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff000000.serial
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff020000.i2c
drwxr-xr-x 6 root root 0 Oct 18 01:36 ff0a0000.gpio
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0b0000.ethernet
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0e0000.ethernet
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0f0000.spi
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff160000.mmc
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff170000.mmc
drwxr-xr-x 3 root root 0 Oct 18 01:36 ff960000.memory-controller
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff9d0000.usb0
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa00000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa10000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa50000.ams
drwxr-xr-x 5 root root 0 Oct 18 01:36 ffa60000.rtc
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa80000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa90000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffaa0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffab0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffac0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffad0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffae0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffaf0000.dma
-r--r--r-- 1 root root 4096 Oct 18 19:37 modalias
lrwxrwxrwx 1 root root 0 Oct 18 19:37 of_node -> ../../../firmware/devicetree/base/amba
drwxr-xr-x 2 root root 0 Oct 18 19:37 power
lrwxrwxrwx 1 root root 0 Oct 18 01:36 subsystem -> ../../../bus/platform
-rw-r--r-- 1 root root 4096 Oct 18 01:36 uevent
root@gdm0:~#
</pre>

= Build firmware =

== Build from git clone ==

THESE ARE K.O.'s NOTES FOR CREATING THE PETALINUX DIRECTORY.

THEY DO NOT WORK!

COPY PETALINUX FROM A WORKING PROJECT AND USE "make gdm" and "make cdm" AS DESCRIBED BELOW.

* git clone git@edev-group.triumf.ca:fw/exp/darkside/gcdm.git
* #Makefile change VIVADO_SETTINGS_SCRIPT := /opt/Xilinx/Vivado/2022.1/settings64.sh
* #. /opt/Xilinx/Vivado/2022.1/settings64.sh
* . /opt/Xilinx/Vivado/2020.2/settings64.sh
* make clean
* make all_from_scratch
* . /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
* make petalinux_create
* make petalinux_rebuild_new_hw_des
* bomb out: The TMPDIR: /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp can't be located on nfs.
* mkdir /tmp/build_tmp
* rm -rf /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp/
* ln -s /tmp/build_tmp /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp
* try again
* grinds, loads a whole bunch of packages...
* finishes with desire to copy things to /tftpboot
* make sdcard_cp_to wants to copy files from PetaLinux_GDM_CDM/images/linux/ to SD card

== Build firmware ==

NOTE: directory Petalinux_GDM_CDM should already exist!

<pre>
#. /opt/Xilinx/Vivado/2020.2/settings64.sh
. /opt/Xilinx/Vivado/2022.2/settings64.sh
. /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
make clean_gdm # remove gdm build tree
make gdm # build or rebuild GDM
make copy_gdm # copy to gdm0
make clean_cdm # remove cdm build tree
make cdm # build or rebuild CDM
make copy_cdm # copy to cdm0 and cdm1
</pre>

copy to SD card:
<pre>
open a root shell
format 16 GB Sd card per above
cd .../ds-dm-gcdm
make copy
</pre>

= build times =

<pre>
CDM 12-june-2023 69aabc1c25130d970bc375aca684bd68849e6685
daq13 AMD-5700G 1688.61user 399.33system 23:28.84elapsed 148%CPU
dsdaqgw AMD-7700 1090.55user 247.34system 16:03.55elapsed 138%CPU
dsdaqgw AMD-7700 CDM incremental 196.68user 67.62system 7:35.42elapsed 58%CPU
dsdaqgw AMD-7700 CDM incremental 684.72user 94.17system 7:30.17elapsed 173%CPU
dsdaqgw AMD-7700 GDM incremental 849.84user 99.79system 9:04.56elapsed 174%CPU
</pre>

= prepare bootable sd card =

== format the sd card ==

this only needs to be done once

* become root
* cd ~olchansk/git/ds-dm-gcdm
* use "lsblk" to identify the SD card (should show as 8/16/32 GB block device)/ /dev/sdd in this case
* make sdcard_format SDCARD_DEVICE=/dev/sdd
* disconnect sd card, reconnect the sd card (to detect new partition tables, etc)

== copy CDM boot files ==

<pre>
cd /home/dsdmdev/git/ds-dm-gcdm
make copy
</pre>

== copy boot files to the sd card ==

* as root: identify partition labels, run "blkid", should say "BOOT", "rootfs" and "data"
* mount
<pre>
mkdir /media/olchansk/BOOT
mkdir /media/olchansk/rootfs
mkdir /media/olchansk/data
mount -L BOOT /media/olchansk/BOOT
mount -L rootfs /media/olchansk/rootfs
mount -L data /media/olchansk/data
cp PetaLinux_GDM_CDM/images/linux/BOOT.BIN /media/olchansk/BOOT/
cp PetaLinux_GDM_CDM/images/linux/boot.scr /media/olchansk/BOOT/
cp PetaLinux_GDM_CDM/images/linux/image.ub /media/olchansk/BOOT/
umount /media/olchansk/BOOT
umount /media/olchansk/rootfs
umount /media/olchansk/data
eject /dev/sdd
</pre>

= boot messages =

<pre>
Xilinx Zynq MP First Stage Boot Loader
Release 2020.2 Sep 24 2022 - 13:29:15
NOTICE: ATF running on XCZU4CG/silicon v4/RTL5.1 at 0xfffea000
NOTICE: BL31: v2.2(release):xlnx_rebase_v2.2_2020.3
NOTICE: BL31: Built : 18:02:46, Sep 28 2022

U-Boot 2020.01 (Sep 28 2022 - 18:03:39 +0000)

Model: DarkSide 20k DM
Board: Xilinx ZynqMP
DRAM: 2 GiB
usb dr_mode not found
PMUFW: v1.1
EL Level: EL2
Chip ID: zu4
NAND: 0 MiB
MMC: mmc@ff160000: 0, mmc@ff170000: 1
In: serial@ff000000
Out: serial@ff000000
Err: serial@ff000000
Bootmode: SD_MODE1
Reset reason: SOFT
Net:
ZYNQ GEM: ff0b0000, mdio bus ff0b0000, phyaddr -1, interface rgmii-id

Warning: ethernet@ff0b0000 (eth0) using random MAC address - d6:62:5f:13:00:44
eth0: ethernet@ff0b0000
ZYNQ GEM: ff0e0000, mdio bus ff0e0000, phyaddr -1, interface rgmii-id
Could not get PHY for eth1: addr -1

Hit any key to stop autoboot: 0
ZynqMP>
CTRL-A Z for help | 115200 8N1 | NOR | Minicom 2.7.1 | VT102 | Online 122:2 | ttyACM0
</pre>

= load FPGA from u-boot =

* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/124682257/U-Boot+FPGA+Driver

<pre>
ZynqMP> fpga info
Xilinx Device
Descriptor @ 0x000000007fddb2c0
Family: ZynqMP PL
Interface type: csu_dma configuration interface (ZynqMP)
Device Size: 1 bytes
Cookie: 0x0 (0)
Device name: zu4
Device Function Table @ 0x000000007fda5fe8
PCAP status 0xa0002fde
ZynqMP>
</pre>

* cp CDM_XU8_top.bit /tftpboot/fpga.bit

<pre>
dhcp
tftpb 0x10000000 fpga.bit
fpga loadb 0 0x10000000 ${filesize}
</pre>

<pre>
ZynqMP> dhcp
BOOTP broadcast 1
DHCP client bound to address 192.168.0.100 (1 ms)
*** Warning: no boot file name; using 'C0A80064.img'
Using ethernet@ff0b0000 device
TFTP from server 192.168.0.1; our IP address is 192.168.0.100
Filename 'C0A80064.img'.
Load address: 0x8000000
Loading: *
TFTP error: 'file /tftpboot/C0A80064.img not found for 192.168.0.100' (1)
Not retrying...
ZynqMP> tftpb 0x10000000 fpga.bit
Using ethernet@ff0b0000 device
TFTP from server 192.168.0.1; our IP address is 192.168.0.100
Filename 'fpga.bit'.
Load address: 0x10000000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
############
6.2 MiB/s
done
Bytes transferred = 7797807 (76fc2f hex)
ZynqMP> fpga loadb 0 0x10000000 ${filesize}
design filename = "CDM_XU8_top;UserID=0XFFFFFFFF;Version=2022.2"
part number = "xczu4cg-fbvb900-1-e"
date = "2024/08/14"
time = "14:18:22"
bytes in bitstream = 7797692
zynqmp_align_dma_buffer: Align buffer at 0000000010000073 to 000000000fffff80(swap 0)
ZynqMP>
</pre>

= load FPGA from Linux =

* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841847/Solution+ZynqMP+PL+Programming

== this will reset the CPU ==
<pre>
cp fpga.bit /lib/firmware/
echo fpga.bit > /sys/class/fpga_manager/fpga0/firmware
</pre>

* make .bin file:
<pre>
bootgen -image CDM_XU8_top.bif -arch zynqmp -o ./fpga.bin -w
</pre>
* cat CDM_XU8_top.bif
<pre>
dsdaqgw:ds-dm-gcdm$ cat CDM_XU8_top.bif
all:
{
[destination_device = pl] ./Vivado_CDM_XU8/CDM_XU8.runs/impl_1/CDM_XU8_top.bit
}
dsdaqgw:ds-dm-gcdm$
</pre>

== this will reset the CPU ==

<pre>
cp fpga.bin /lib/firmware/
echo fpga.bin > /sys/class/fpga_manager/fpga0/firmware
</pre>

== this will reset the CPU ==

<pre>
root@dsdm:~# ./fpgautil -b fpga.bin -f Full
</pre>

== this will reset the CPU ==

<pre>
DTSO file from here: https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841847/Solution+ZynqMP+PL+Programming
echo 0 > /sys/class/fpga_manager/fpga0/flags
mount -t configfs configfs /configfs
root@dsdm:~# dtc -O dtb -o fpga.dtbo -b 0 -@ fpga.dtso
root@dsdm:~# cp fpga.dtbo /lib/firmware/
root@dsdm:~# cp fpga.bit /lib/firmware/
root@dsdm:~# rmdir /configfs/device-tree/overlays/fpga
root@dsdm:~# mkdir /configfs/device-tree/overlays/fpga
root@dsdm:~# echo -n "fpga.dtbo" > /configfs/device-tree/overlays/fpga/path
</pre>

= fpgautil =

* https://github.com/Xilinx/meta-xilinx/blob/master/meta-xilinx-core/recipes-bsp/fpga-manager-script/files/fpgautil.c

<pre>
git clone https://github.com/Xilinx/meta-xilinx.git
cd meta-xilinx/meta-xilinx-core/recipes-bsp/fpga-manager-script/files/
scp fpgautil.c root@dsdm:
ssh root@dsdm
make fpgautil
ls -l ./fpgautil
</pre>

<pre>
root@dsdm:~# ls -l ./fpgautil
-rwxr-xr-x 1 root root 72256 Aug 16 00:15 ./fpgautil
root@dsdm:~# ./fpgautil

fpgautil: FPGA Utility for Loading/reading PL Configuration

Usage: fpgautil -b <bin file path> -o <dtbo file path>

Options: -b <binfile> (Bin file path)
-o <dtbofile> (DTBO file path)
-f <flags> Optional: <Bitstream type flags>
f := <Full | Partial >
-n <Fpga region info> FPGA Regions represent FPGA's
and partial reconfiguration
regions of FPGA's in the
Device Tree
Default: <full>
-s <secure flags> Optional: <Secure flags>
s := <AuthDDR | AuthOCM | EnUsrKey | EnDevKey | AuthEnUsrKeyDDR | AuthEnUsrKeyOCM | AuthEnDevKeyDDR | AuthEnDevKeyOCM>
-k <AesKey> Optional: <AES User Key>
-r <Readback> Optional: <file name>
Default: By default Read back contents will be stored in readback.bin file
-t Optional: <Readback Type>
0 - Configuration Register readback
1 - Configuration Data Frames readback
Default: 0 (Configuration register readback)
-R Optional: Remove overlay from a live tree

Examples:
(Load Full bitstream using Overlay)
fpgautil -b top.bit.bin -o can.dtbo -f Full -n full
(Load Partial bitstream using Overlay)
fpgautil -b rm0.bit.bin -o rm0.dtbo -f Partial -n PR0
(Load Full bitstream using sysfs interface)
fpgautil -b top.bit.bin -f Full
(Load Partial bitstream using sysfs interface)
fpgautil -b rm0.bit.bin -f Partial
(Load Authenticated bitstream through the sysfs interface)
fpgautil -b top.bit.bin -f Full -s AuthDDR
(Load Parital Encrypted Userkey bitstream using Overlay)
fpgautil -b top.bit.bin -o pl.dtbo -f Partial -s EnUsrKey -k <32byte key value>
(Read PL Configuration Registers)
fpgautil -b top.bit.bin -r
(Remove Partial Overlay)
fpgautil -R -n PR0
(Remove Full Overlay)
fpgautil -R -n full
Note: fpgautil -R is responsible for only removing the dtbo file from the livetree. it will not remove the PL logic from the FPGA region.

root@dsdm:~#
</pre>

= fw_printenv =

to access u-boot environment from Linux:
* apt install -y libubootenv-tool
* create /etc/fw_env.config
<pre>
/media/BOOT/uboot.env 0 0x40000
/media/BOOT/uboot-redund.env 0 0x40000
</pre>
* if uboot.env files do not exist, run "saveenv" from u-boot command prompt
* fw_printenv and fw_setenv should work

= Boot from network =

== u-boot ==

<pre>
ZynqMP> setenv bootcmd run bootcmd_dhcp
ZynqMP> saveenv
ZynqMP> reset
</pre>

== boot.scr ==

<pre>
# boot.scr
# mkimage -C none -A arm -T script -d boot.scr boot.scr.uimg
echo Loading FPGA!
#tftpb 0x10000000 fpga.bit
tftpb 0x10000000 {ipaddr}.bit
fpga loadb 0 0x10000000 ${filesize}
echo Booting Linux!
run bootcmd_pxe
echo Done!
</pre>

<pre>
mkimage -C none -A arm -T script -d boot.scr boot.scr.uimg
</pre>

== tftpboot ==

<pre>
cp /home/olchansk/git/ds-dm-gcdm/boot.scr.uimg /tftpboot
ln -s /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/images/linux /tftpboot/xilinx-dsdm
mkdir /tftpboot/pxelinux.cfg
cat > /tftpboot/pxelinux.cfg/default-arm-zynqmp <<EOF
LABEL Linux
KERNEL xilinx-dsdm/Image
FDT xilinx-dsdm/system.dtb
#INITRD rootfs.cpio.gz.u-boot
EOF
</pre>

== boot sequence ==

* xilinx magic load BOOT.BIN from SD card
* load FPGA form BOOT.BIN
* load and run u-boot from BOOT.BIN or from image.ub
* u-boot load environment from ??? probably SD card uboot-redund.env, this includes our bootcmd
* run bootcmd which run bootcmd_dhcp which does:
* from /tftpboot:
* load and run boot.scr.uimg which does:
* load FPGA image xilinx-dsdm/${ipaddr}.bit
* run bootcmd_pxe which does:
* load pxelinux.cfg/default-arm-zynqmp which does:
* load xilinx-dsdm/Image ### this is the linux kernel
* load xilinx-dsdm/system.dtb ### this is the device tree
* start linux kernel
* linux kernel does dhcp
* linux kernel does nfs mount /nfsroot/%s,vers=3,tcp ### %s is replaced by the hostname supplied by DHCP
* userland starts and runs to console and ssh login.

= Xilinx ILA =

References:
* https://github.com/Xilinx/XilinxVirtualCable/tree/master
* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/644579329/Xilinx+Virtual+Cable
* https://support.xilinx.com/s/article/974879?language=en_US
* https://docs.xilinx.com/v/u/en-US/pg245-debug-bridge
* https://docs.xilinx.com/r/2020.2-English/ug908-vivado-programming-debugging/Initializing-Vivado-IDE-hw_server
* (we do not use this one) https://github.com/paulscherrerinstitute/xvcSupport/blob/master/README.md

Build xvcserver_cdm.exe: (it is built as a static executable, can be copied and run anywhere)
<pre>
ssh dsdaq@gdm0
cd /home/dsdaq/online/ds-dm-software
git pull ### get latest version
make xvcserver_cdm.exe
ssh root@gdm0
/home/dsdaq/online/ds-dm-software/xvcserver_cdm.exe
INFO: To connect to this xvcServer instance, use url: TCP:gdm0:2542
</pre>

To activate and use the vivado logic analyzer:
* data path: vivado -> hw_server -> xvcserver -> mmap axi bus -> debug bridge -> jtag -> ILA
* define ILAs in the code
* instantiate the xilinx debug bridge at AXI bus address 0x80020000 (FIXME!!! this collides with Ian's AXI addresses)
* build and boot the new FPGA firmware. updating the linux kernel is not necessary.
* login root@gdm0, run: /home/dsdaq/online/ds-dm-software/xvcserver_cdm.exe -v ### with "-v" for the first time to see that vivado does connect to it, without "-v", normally.
* login dsdaqgw, run: hw_server -s tcp:localhost:3121 -e "set auto-open-servers xilinx-xvc:gdm0:2542" ### tells us to connect to port localhost:3121
* login dsdaqgw, run vivado, open project, open hardware manager, open target, open new target, "connect to remote server", hostname "localhost", port "3121", next (bombs, try again, 3 times), popup add virtual cable, enter hostname "gdm0" port "2542", "ok", it shows in "hardware targets", "next", "finish", error popup "[Common 17-163] Missing value for option 'objects', please type 'set_property -help' for usage info", ignore it, in "hardware", right click the "gdm0" one, open target, under "hardware" and "debug bridge" we should see all the ILAs, under "hardware device properties", the "probes file" should have the ".ltx" file generated by vivado "Vivado_GDM_XU8/GDM_XU8.runs/impl_1/debug_nets.ltx", click on an ILA, a waveform should open.

= Software =

* ssh cdm0 # or gdm0
* sudo apt install i2c-tools libi2c-dev
* git clone https://bitbucket.org/team-ds-dm/ds-dm-software
* cd ds-dm-software
* make

== test_cdm.exe ==

=== CDM SFP status ===

<pre>
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0xbb2f0ae7
CDM firmware 0xbb2f0ae7
arg 1: [--sfp]
Polling SFP status...
identifier 0x03
connector 0x07
encoding 0x01
wavelength 0x0352 (850 nm)
vendor_name [FINISAR CORP. ]
vendor_pn [FTLF8526P3BNL ]
vendor_rev [A ]
vendor_sn [N3AB9M8 ]
vendor_date [200319 ]
dm_type 0x68
temp 29.0 C
vcc 3.323 V
tx_bias 7.250 mA
tx_power 478.4 uW
rx_power 2.3 uW
SFP good 1, status: temp 30.6 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 818 uW
...
</pre>

=== GDM QSFP status ===

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --qsfp3 --qsfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x53aee418
CDM firmware 0x53aee418
arg 1: [--qsfp3]
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out hi
arg 2: [--qsfp]
Polling QSFP status...
identifier 0x0d
status 0x02
los 0x8f
temp 28.2 C
vcc 3.323 V
rx_power 0.1 0.1 0.1 0.1 uW
tx_bias 7.6 7.6 7.6 0.0 mA
tx_power 792.2 773.8 823.0 0.1 uW
vendor_name [FINISAR CORP ]
vendor_pn [FTL410QD4C ]
vendor_rev [A ]
wavelength 850
max_temp 70 C
vendor_sn [X79AC0R ]
vendor_date [220309 ]
QSFP good 1, status: temp 27.7 C, los 0x8b, tx_bias 7.5 7.6 7.6 0.0 mA, tx_power 792 772 821 0 uW, rx_power 0 0 466 0 uW
</pre>

=== GDM clock status ===

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
</pre>

* clock chip not loaded, not running:

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x0f 0x16 0x04 0x00 0x01 0x68 0x09 0x00 0x02 0xf2 0x00 0x1f 0xf0 0x22 0xf2, 0x507: 0x00, 0x52A: 0x00, 0x53F: 0x04
Clock chip state 0, status: SYSINCAL XAXB_ERR LOL CAL_PLL IN0 IN_SEL_0 FASTLOCK_STATUS

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x1f7caf52 (528265042) should be ~125 MHz
0x1034 rx_clk: 0x00000000 (0) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x1f7caf52 (528265042) should be ~125 MHz
0x103C tx_clk: 0x00000000 (0) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip good (IN0 - external 10 MHz clock)

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x44 0x00 0xd0 0x01 0x1f 0xff 0x22 0xf2, 0x507: 0x3f, 0x52A: 0x01, 0x53F: 0x00
Clock chip state 1, status: LOS_IN2 OOF_IN2 IN0 IN_SEL_REGCTRL IN_SEL_0

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735a3b (125000251) should be ~125 MHz
0x1034 rx_clk: 0x07735a3c (125000252) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a3b (125000251) should be ~125 MHz
0x103C tx_clk: 0x07735a3c (125000252) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

=== CDM clock status ===

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
</pre>

* clock chip not loaded, not running:

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x0f 0x16 0x04 0x00 0x01 0x68 0x19 0x00 0x02 0xf2 0x00 0x1f 0xf0 0x22 0xf2, 0x507: 0x00, 0x52A: 0x00, 0x53F: 0x04
Clock chip state 0, status: SYSINCAL XAXB_ERR LOL CAL_PLL IN0 IN_SEL_0 FASTLOCK_STATUS

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735851 (124999761) should be ~125 MHz
0x1034 rx_clk: 0x00000000 (0) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x0127fefa (19398394) should be ~125 MHz
0x103C tx_clk: 0x00000000 (0) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip uses internal clock (IN1 - internal oscillator), observe rx_clk frequency is not same as others

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0x7f, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN1 IN_SEL_1 HOLD_HIST_VALID

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735853 (124999763) should be ~125 MHz
0x1034 rx_clk: 0x07735b0a (125000458) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735853 (124999763) should be ~125 MHz
0x103C tx_clk: 0x07735852 (124999762) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip uses SFP recovered clock (IN2 - sfp rx recovered clock), observe mgt_rx_ref_clk_raw (CDM 125 MHz oscillator) is different from others (SFP RX recovered clock)

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0xbf, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN2 IN_SEL_1 HOLD_HIST_VALID
^C
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773581b (124999707) should be ~125 MHz
0x1034 rx_clk: 0x07735ad6 (125000406) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735ad7 (125000407) should be ~125 MHz
0x103C tx_clk: 0x07735ad6 (125000406) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
^C
</pre>

=== CDM link status, PRBS test mode ===

<pre>
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
</pre>
* fiber disconnected, no link:
<pre>
CDM firmware: 0xbb2f0ae7
0x1000 SFP c.c. status: 0x00000031
CLK_IN_SEL_LS 0x1
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x00000025
sfp_mod_absent_N 1
sfp_rx_los_N 0
link_power_good 1
rx_link_up 0
rx_receiving_data 0
rx_error 1
rx_lnk_up_and_running 0
tx_link_up 0
tx_sending_data 0
tx_link_up_and_running 0
link_up_and_running 0
0x1014 SFP link data: 0x466a8187
rx_data 0x8187
k28p1_k28p5 0
rx_ctrl0 0x1
rx_ctrl1 0x1
rx_ctrl3 0x1
tx_state 0x1
rx_state 0x6
rx_receiving_data 0
rx_link_up 0
rx_error 1
rx_link_rst 0
0x2000 link test mode: 0x00000200, seconds: 0x00079093, errors: 0xffffffff
</pre>
* fiber connected, good link:
<pre>
CDM firmware: 0xbb2f0ae7
0x1000 SFP c.c. status: 0x000000b2
CLK_IN_SEL_LS 0x2
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x000007dc
sfp_mod_absent_N 0
sfp_rx_los_N 0
link_power_good 1
rx_link_up 1
rx_receiving_data 1
rx_error 0
rx_lnk_up_and_running 1
tx_link_up 1
tx_sending_data 1
tx_link_up_and_running 1
link_up_and_running 1
0x1014 SFP link data: 0x35c02774
rx_data 0x2774
k28p1_k28p5 0
rx_ctrl0 0x0
rx_ctrl1 0x0
rx_ctrl3 0x0
tx_state 0x3
rx_state 0x5
rx_receiving_data 1
rx_link_up 1
rx_error 0
rx_link_rst 0
0x2000 link test mode: 0x00000200, seconds: 0x00078a8c, errors: 0x00000000
</pre>

=== GDM link status, PRBS test mode ===

<pre>
1 link connected, no errors:
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
GDM firmware: 0x6b2ee010
0x1014: 0x00000008, 18: 0x00000008, 1C: 0x00000000, 24: 0x00000fff
0x2000: 0x00000200, time: 0x00078aa4, errors:
0xffffffff 0xffffffff 0xffffffff
0x00000000 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
</pre>

=== CDM link status ===

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test0

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0xbf, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN2 IN_SEL_1 HOLD_HIST_VALID

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x077357a0 (124999584) should be ~125 MHz
0x1034 rx_clk: 0x07735a5c (125000284) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a5b (125000283) should be ~125 MHz
0x103C tx_clk: 0x07735a5b (125000283) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM firmware: 0x6d2ef81a
0x1000 SFP c.c. status: 0x000000b2
CLK_IN_SEL_LS 0x2
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x000007dc
sfp_mod_absent_N 0
sfp_rx_los_N 0
link_power_good 1
rx_link_up 1
rx_receiving_data 1
rx_error 0
rx_lnk_up_and_running 1
tx_link_up 1
tx_sending_data 1
tx_link_up_and_running 1
link_up_and_running 1
0x1014 SFP link data: 0x35c6bcbc
rx_data 0xbcbc
k28p1_k28p5 0
rx_ctrl0 0x3
rx_ctrl1 0x0
rx_ctrl3 0x0
tx_state 0x3
rx_state 0x5
rx_receiving_data 1
rx_link_up 1
rx_error 0
rx_link_rst 0
0x2000 link test mode: 0x00000000, seconds: 0x00001671, errors: 0xffffffff

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 15
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
reg[15] is 0x0033bcbc (3390652)
</pre>

=== GDM link status ===

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test0

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x44 0x00 0xd0 0x01 0x1f 0xff 0x22 0xf2, 0x507: 0x3f, 0x52A: 0x01, 0x53F: 0x02
Clock chip state 1, status: LOS_IN2 OOF_IN2 IN0 IN_SEL_REGCTRL IN_SEL_0 HOLD_HIST_VALID

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735a0a (125000202) should be ~125 MHz
0x1034 rx_clk: 0x07735a0a (125000202) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a0a (125000202) should be ~125 MHz
0x103C tx_clk: 0x07735a0a (125000202) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly

register 0x1018 bit 0x800

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM firmware: 0x6d2ebce6
0x1014: 0x00000800, 18: 0x00000800, 1C: 0x00000000, 24: 0x00000fff
0x2000: 0x00000000, time: 0x00003d2f, errors:
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff

link data alternates 0xbcbc and 0x1cbc

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 22
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
reg[22] is 0xbcbc93ab (-1128492117)

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 22
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
reg[22] is 0x1cbc1aaf (482089647)
root@gdm0:~#

</pre>

=== Run trg and tsm ===

<pre>
on the GDM:

ssh root@gdm00
./test_cdm.exe --gdm-clocks
./test_cdm.exe --load-cc
./test_cdm.exe --cc
./test_cdm.exe --reset-mgt
./test_cdm.exe --gdm-clocks
./test_cdm.exe --writereg 2 0xff # enable LEMO NIM inputs
./test_cdm.exe --writereg 3 0xba54 # LEDs: lemo1, lemo2, trg, tsm
./test_cdm.exe --writereg 4 0x99 $ # LEMO out is trg_in_pulse
./test_cdm.exe --writereg 9 0x32010 # trg and tsm from trg_pulser and tsm_pulse
./test_cdm.exe --writereg 24 1250000 # trg pulser 100 Hz
./test_cdm.exe --writereg 26 125000000 # tsm pulser 1 Hz
./test_cdm.exe --writereg 23 0x40000000 # route trg_in and tsm_in to qsfp tx bits 0 and 1
./test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz

on the CDM:

ssh root@cdm01
./test_cdm.exe --cdm-clocks
./test_cdm.exe --load-cc
./test_cdm.exe --reset-mgt
./test_cdm.exe --cdm-clocks
./test_cdm.exe --cdm-link # issue --reset-mgt on CDM and GDM until link is good
./test_cdm.exe --writereg 2 0xff # enable LEMO NIM inputs
./test_cdm.exe --writereg 3 0xba54 # enable LEDs: lemo1, lemo2, trg, tsm
./test_cdm.exe --writereg 4 0x99 # enable LEMO output trg_in
./test_cdm.exe --writereg 9 0x8040 # enable trg_in and tsm_in from sfp rx bits 0 and 1
./test_cdm.exe --writereg 7 0xff00 # drive VX LVDS lines to logic level 0
./test_cdm.exe --writereg 8 0x2 # VX LVDS with trg and tsm, misrouted in second VX
#./test_cdm.exe --writereg 8 0x7 # VX LVDS with tsm
./test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz
</pre>

switch GDM and CDM to packetizer trg and tsm:

<pre>
on the GDM:

/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x00000000

on the CDM:

/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x0804
/home/dsdaq/online/ds-dm-software/test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz
</pre>

=== Run packet loopback ===

GDM CPU -> fifo_to_fpga -> GDM QSFP -> CDM SFP -> fifo_from_fpga -> CDM CPU

On the GDM: (CDM is connected to first QSFP port)

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --load-cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --reset-mgt
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 32 1 ### tell GDM to use first QSFP port
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link ### confirm link status is "3"
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 29 2 ### enable GDM packet data injection
/home/dsdaq/online/ds-dm-software/test_cdm.exe --test-fifo-write-loop
</pre>

One the CDM:

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --load-cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --reset-mgt
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 29 3 ### enable CDM packet data injection
/home/dsdaq/online/ds-dm-software/test_cdm.exe --test-fifo-read
</pre>

= dsvslice integration =

== VX setup ==

* general
** Start acq from user code = y, all others = n (as of Mar 2023: will start when first trigger received)
** Use NIM IO = y
** Use external clock = y
** LVDS quartet is input = n, y, n, y
** LVDS quartet mode = User, User, User, User

* trigger from front panel NIM:
** Trigger on external signal = y, all others = n
** connect CDM EXT_OUT(2) to VX "TrigIn"

* trigger from LVDS "Sync" mode
** Trigger on LVDS Sync signal = y, all others = n
** LVDS quartet mode = User, Sync, User, User

* trigger from LVDS "User" mode
** Trigger on LVDS pair 12 signal = y, all others = n
** LVDS quartet is input = n, y, n, y
** LVDS quartet mode = User, User, User, User

== GDM setup ==

* GDM is gdm0
* set inputs to NIM mode
* set outputs to TTL mode (this GDM has wrong NIM output circuit)
* use top QSFP slot, connect split cable 1 into CDM01, cable 2 into CDM02
* connect non-inverted NIM trigger signal to top-LEMO-left EXT_IN_LV(1)
* connect non-inverted NIM TSM signal to top-LEME-right EXT_IN_LV(2)
* GDM LEDs: TRIG, TSM, trigger enabled, trigger_out
* GDM LEMO_OUT: trigger, trigger

== CDM setup ==

* set CDM LEMO inputs to NIM
* set CDM LEMO outputs to NIM
* CDM01 is cdm0
* CDM02 is cdm1
* connect GDM fiber links to SFP port
* connect 1st VX port of CDM01 to VX1
* connect 1st VX port of CDM02 to VX2
* connect LEMO EXT_OUT(2) to VX "TrigIn", CDM01 to VX1, CDM02 to VX2
* power up
* CDM LEDs: GDM TRIG, GDM TSM, trigger enabled, trigger out
* CDM LEMO_OUT: gdm_trg, trigger

== After power up ==

* start the CDM frontend from the MIDAS "Programs" page. To start manually, see the Start Command on the Programs page.
* CDM frontend should enable the VX clock, disable the trigger
* from the MIDAS status page, goto the CDM page
* outdated: in the CDMx data tables, the 2nd number should read 0x35c08008, if it does not and the last 4 digits randomly change, reset the GDM links
*if the FE complain - do in order: for GDM, CDM01, CDM02, ..., press "reset mgt" of each board, then press "unreset mgt", if it does not help, STOP HERE
* start a run
* CDM frontend will enable the trigger
* GDM frontend will enable the trigger
* LEDs on the GDM should flash, LEDs on the CDM should flash, TrigIn and TrigOut of the VX should flash
* stop a run
* GDM frontend will disable the trigger
* CDM frontend will disable the trigger

== Phase measurement ==

* pip3 install matplotlib
* pip3 install scipy
* export PYTHONPATH=$HOME/packages/midas/python
* #git clone https://github.com/J033X071C/PhaseMeasurement
* git clone https://bitbucket.org/team-ds-dm/phasemeasurement.git
* cd phasemeasurement
* python3 ./phaseMeasurement.py --help
<pre>
daq00:PhaseMeasurement$ python3 ./phaseMeasurement.py --help
usage: phaseMeasurement.py [-h] fileName numberEvents numberVX sizeEvents stopEvent minHist maxHist numberBin writeToTXT saveAsPDF

Read data from midas file (in .lz4 format) to calculate phase between the clock of VX1 and VX2

positional arguments:
fileName Name of the file we want to read data from (Example: run00389.mid.lz4)
numberEvents Number of events recorded in the file
numberVX Number of VX used in this run (usually 2...)
sizeEvents Number of points per event
stopEvent Number of events you want to go through to calculate phase
minHist Minimal value for the x axis of the phase measurement histogram (in ns)
maxHist Maximal value for the x axis of the phase measurement histogram (in ns)
numberBin Number of bins wanted for the generated histogram
writeToTXT Write argument as yes to generate text file with results of calculation
saveAsPDF Save generated plots to PDF files

optional arguments:
-h, --help show this help message and exit
daq00:PhaseMeasurement$
</pre>
* try an old file with
* python3 ./phaseMeasurement.py run00877.mid.lz4 10000 2 10000 500 -20 20 81 yes yes
<pre>
bin size = 0.494 ns
num_events = 347
mean = -1.705 ns
rms = 3.087 ns
mean_error = 0.166 ns
centroid = -1.706 ns.
width (sigma) = 0.363 ns.
error on the centroid = 0.016558 ns.
</pre>
* ls -l *.txt *.pdf
<pre>
dsdaq@dsvslice:~/online/PhaseMeasurement$ ls -l *.txt *.pdf
-rw-rw-r-- 1 dsdaq dsdaq 64728 Dec 14 16:56 run00877.mid.lz4_Plots.pdf
-rw-rw-r-- 1 dsdaq dsdaq 274 Dec 14 16:56 run00877.mid.lz4.txt
dsdaq@dsvslice:~/online/PhaseMeasurement$
</pre>
* scope settings (from email message)
<pre>
From fcote-lortie@triumf.ca Thu Dec 15 17:11:46 2022
From: Francis Cote-Lortie <fcote-lortie@triumf.ca>
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: Re: How to use scope as a waveform generator
Date: Fri, 16 Dec 2022 01:11:44 +0000

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display) by pressing on the button. It will go from 0 to 1.

The settings that we are using right now are:
Type of waveform: Sine wave
Offset: 0 V
Amplitude: 1 Vpp
Frequency: 50 kHz
Noise: 0 V
________________________________
From: Francis Cote-Lortie <fcote-lortie@triumf.ca>
Sent: Thursday, December 15, 2022 4:58 PM
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: Re: How to use scope as a waveform generator

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display)

The settings that we are using right now are:
Type of waveform: Sine wave
Offset: 0 V
Amplitude: 1 Vpp
Frequency: 50 kHz
Noise: 0 V
________________________________
From: Francis Cote-Lortie
Sent: Thursday, December 15, 2022 4:54 PM
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: How to use scope as a waveform generator

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display)
</pre>

== Standalone link test ==

<pre>
CDM: program clock chip
busybox devmem 0x80011000 32 0x8
busybox devmem 0x80011000 32 0x0
/home/dsdaq/si5394-i2c-file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b

GDM, CDM: link reset
busybox devmem 0x80011008 32 1

GDM, CDM: release reset
busybox devmem 0x80011008 32 0

CDM: link status (NOTE: SFP LOS and mod_absent are swapped!!!)
busybox devmem 0x80011010 32
0x00000024 <- fiber plugged
0x00000025 <- fiber unplugged
0x00000027 <- SFP unplugged
0x000007DC <- successful link with GDM

CDM: link state machine and data
busybox devmem 0x80011014 32
0x35C06FF6

CDM: set link to counting mode
busybox devmem 0x80012000 32 0x101
busybox devmem 0x80012000 32 0x100

CDM: time counter and error counter
root@cdm1:~# busybox devmem 0x80012000 32
0x00000100 <--- link mode
root@cdm1:~# busybox devmem 0x80012004 32
0x0000058C <--- seconds counter
root@cdm1:~# busybox devmem 0x80012004 32
0x0000058D
root@cdm1:~# busybox devmem 0x80012008 32
0x00000000 <--- error counter

GDM: no link
root@gdm0:~# busybox devmem 0x80011014 32
0x00000000
root@gdm0:~# busybox devmem 0x80011018 32
0x00000000
root@gdm0:~# busybox devmem 0x8001101c 32
0x00000000
root@gdm0:~# busybox devmem 0x80011024 32
0x00000FFF
root@gdm0:~#

GDM: good link channel 10, counting mode
root@gdm0:~# busybox devmem 0x80012000 32 0x101
root@gdm0:~# busybox devmem 0x80012000 32 0x100
root@gdm0:~# busybox devmem 0x80012008 32
0x3A8B68C2
root@gdm0:~# busybox devmem 0x80012008 32
0x42E03BEF
root@gdm0:~# busybox devmem 0x8001200c 32
0xDA090972
root@gdm0:~# busybox devmem 0x8001200c 32
0xDE6F22E9
root@gdm0:~# busybox devmem 0x80012019 32
Bus error
root@gdm0:~# busybox devmem 0x80012010 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012014 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012018 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x8001201c 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012020 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012024 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012028 32
0x00000000
root@gdm0:~# busybox devmem 0x8001202c 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012030 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012034 32
0xFFFFFFFF
root@gdm0:~#
</pre>

mapping of link channels:

<pre>
qsfp0 lane0 - 0x0100 - link 8
qsfp0 lane1 - 0x0200 - link 9
qsfp0 lane2 - 0x0400 - link 10
qsfp0 lane3 - n/c
qsfp1 lane0 - 0x0800 - link 11
qsfp1 lane1 - 0x0010 - link 4
qsfp1 lane2 - 0x0020 - link 5
qsfp1 lane3 - n/c
qsfp2 lane0 - 0x0040 - link 6
qsfp2 lane1 - 0x0080 - link 7
qsfp2 lane2 - 0x0001 - link 0
qsfp2 lane3 - n/c
qsfp3 lane0 - 0x0002 - link 1
qsfp3 lane1 - 0x0004 - link 2
qsfp3 lane2 - 0x0008 - link 3
qsfp3 lane3 - n/c
</pre>

script to start the test with 2 CDMs:

<pre>
ssh dsdaq@dsvslice
ssh root@gdm0 busybox devmem 0x80011008 32 1
ssh root@cdm0 busybox devmem 0x80011008 32 1
ssh root@cdm1 busybox devmem 0x80011008 32 1
ssh root@gdm0 busybox devmem 0x80011008 32 0
ssh root@cdm1 busybox devmem 0x80011008 32 0
ssh root@cdm0 busybox devmem 0x80011008 32 0
ssh root@gdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
ssh root@cdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
ssh root@cdm1 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
</pre>

== LEMO trigger GDM to CDM to VX ==

on the GDM:

<pre>
/home/dsdaq/si5394-i2c-file /home/dsdaq/GDM_v1.0_IN0_EXT1_and_IN1_fixed_Si5394-RevA-Registers.txt 0 0x6b
busybox devmem 0x80011008 32 0x1
busybox devmem 0x80011008 32 0x0
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x80001230 # QSFP TX fixed pattern
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 2 0xff # enable LEMO inputs
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 3 0x7654 # enable LED, one per LEMO input
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x0F0F # enable LEMO to trg_in and tsm_in
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x40001230 # enable trg_in and tsm_in output to QSFP
</pre>

on the CDM:

<pre>
/home/dsdaq/si5394-i2c-file /home/dsdaq/CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
/home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp
busybox devmem 0x80011008 32 1
busybox devmem 0x80011008 32 0
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x8040 # trg_in from sfp[0], tsm_in form sfp[1]
/home/dsdaq/online/ds-dm-software/test_cdm.exe 11 # trg_in counter
/home/dsdaq/online/ds-dm-software/test_cdm.exe 12 # tsm_in counter
</pre>

== GPS receiver VCL-2705 ==

* Valiant VCL-2705 GPS receiver
* https://www.valiantcom.com/time-distribution/gps-receiver-irig-b.html
* USB connection is /dev/ttyACM0, 115200 bps, "GNSSAUX" prompt
* minicom -D /dev/ttyACM0 -b 115200
* user manual download instructoins - see sheet of paper with user name and password in the shipping box
* usb commands:
<pre>
gnss-help
gnss-showver -> F/W: Ver 1.7 Feb 17 2020 16:20:43

gnss-showselftest -> no antenna connected
Overall : FAIL
EPROM Test: PASS
Antenna : NOT DETECTED
GNSS : COMMUNICATION OK

gnss-showsettings
GNSS NMEA BAUDRATE :115200
GNSS ANTENNA LENGTH :30 meters
GNSS USER CONFIGURED DELAY:-65 nanoseconds
GNSS 1PPS PULSE WIDTH :200 milliseconds
GNSS MODE :GPS
GNSS STATUS :STATIONARY

GNSSAUX> gnss-showserial
SERIAL :2704H01V17MAX310

GNSSAUX> gnss-showmode
GNSS MODE : GPS

GNSSAUX> gnss-showstatus
GNSS STATUS: STATIONARY

GNSSAUX> gnss-showalarms
CURRENT ALARMS GNSS
ANTENNA : NOT DETECTED

GNSSAUX> gnss-showerrors
CURRENT ALARMS GNSS
ANTENNA : **NOT DETECTED
ERROR STATISTICS GNSS
RMC GOOD DURATION : Secs 0
RMC BAD DURATION : Secs 0
LOCK GOOD SECS DURATION : Secs 0
LOCK BAD SECS DURATION : Secs 0
SATINFO GOOD ITERATIONS :0
SATINFO REJECT ITERATIONS :0
SATINFO NOTALKER ITERATIONS :0
CURRENT MONITOR STATE GNSS
STATE :Phase-1 HUNTING ANTENNA DETECT

GNSSAUX> gnss-showsatinfo
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-showsats
Total Sats: 0

GNSSAUX> gnss-showmyloc
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-show1ppsstate
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-showjamstatus
Not Available !

GNSSAUX> gnss-showspoofstatus
Not Available !

--- antenna connected, can see the sky ---

GNSSAUX> gnss-showselftest
Overall : PASS
EPROM Test: PASS
Antenna : DETECTED
GNSS : COMMUNICATION OK

GNSSAUX> gnss-showalarms
CURRENT ALARMS GNSS
ANTENNA : DETECTED
GNSS LOCK : AVAILABLE

GNSSAUX> gnss-showerrors
CURRENT ALARMS GNSS
ANTENNA : DETECTED
GNSS LOCK : AVAILABLE
ERROR STATISTICS GNSS
RMC GOOD DURATION : Mins 1,Secs 34
RMC BAD DURATION : Secs 55
LOCK GOOD SECS DURATION : Mins 1,Secs 34
LOCK BAD SECS DURATION : Secs 55
SATINFO GOOD ITERATIONS :3
SATINFO REJECT ITERATIONS :0
SATINFO NOTALKER ITERATIONS :0
CURRENT MONITOR STATE GNSS
STATE :Phase-4 NORMAL OPERATION, Monitoring GNSSLOCK

GNSSAUX> gnss-showsatinfo
GNSS MODE :GPS
SATELLITE INFORMATION TALKER:GPS
NMEA ID : 01-32
NO OF SATELLITES IN VIEW: 08
NO OF XXGSV MSGS : 03
SatNo PRN NO (SV ID) ELEVATION (degs) AZIMUTH (degs) C/No (SNR)
1 0 0 0 0
2 0 0 0 0
3 0 0 0 0
4 0 0 0 0
5 0 0 0 0
6 0 0 0 0
7 0 0 0 0
8 0 0 0 0
<<<<<< End of Sat Info >>>>>>

GNSSAUX> gnss-showsats
Total Sats: 8

GNSSAUX> gnss-showmyloc
GNSS RECEIVER LOCATION:
Latitude : 4914.81911
Longitude: 12313.69595

GNSSAUX> gnss-show1ppsstate
GPS 1PPS STATE: LOCKED

GNSSAUX> gnss-showmyloc

GNSS RECEIVER LOCATION:
Latitude : 4914.80688
Longitude: 12313.69531

enter into google maps search box as: 49 14.80688, -123 13.69531, observe the space, the moved dot and the minus.

GNSSAUX> gnss-resetgnss

Executing....Please Wait....
$$$$$END
</pre>

* 1PPS BNC output: period 1 sec, pulse width 200 ms, 3.3V into 1MOhm, 1.38V into 50Ohm.
* IRIG-B BNC output: 5.6V into 1MOhm, 2.4V into 50Ohm.
* IRIG-B format selector: default is all up.

== Rb clock PRS10 ==

* https://www.thinksrs.com/products/prs10.html
* 10 MHz output is sine wave around 5V peak to peak
* 1PPS BNC output is 10 usec pulse, 5V into 1MHohm.
* RS232 connection: minicom -D /dev/ttyUSB0 -b 9600
* ser2net config: localhost,3001:raw:600:usb-5-2-1.0:9600 -XONXOFF -RTSCTS LOCAL
* ssh daq13, cd /home/olchansk/git/ser2net, ./ser2net -c ~/daq/ds/ser2net.conf -d
* ssh daq13, cd ~/daq/ds, python3 prs10.py
* RS232 commands:
<pre>
ID?
PRS10_3.56_SN_105719
VB1
SN?
RS1 -- reset
ST? -- status
FC? -- 10MHz OCXO drive voltage DAC settings
DS? -- "detected signals"
GA? -- gain of frequency lock loop between ovenized oscillator and Rb cell, 0=use ovenized oscillator only
MO? -- magnetic offset of the Rb cell, range 2300..3600, if out of range, unit must be set to different operating mode, see prs10m.pdf
MR? -- magnetic read
TT? -- time-tag, time in ns between 1PPS out and 1PPS in
TS? -- time slope, ???
TO? -- time offset, ???
PS? -- pulse slope, ???
PL? -- 0=phase lock off, 1=phase lock on, lock to 1PPS input
PT? -- phase lock integrator time constant, PT8 is integrator time constant 18.2 hours, natural time constant 2.25 hours
PF? -- phase lock stability factor, PF2 is "1"
PI? -- phase lock integrator

Analog to digital 12 bit ADC, values 0.000 to 4.998

AD0? -- Spare (J204)
AD1? -- +24V(heater supply) divided by 10.
AD2? -- +24V(electronics supply) divided by 10
AD3? -- Drain voltage to lamp FET divided by 10
AD4? -- Gate voltage to lamp FET divided by 10
AD5? -- Crystal heater control voltage
AD6? -- Resonance cell heater control voltage
AD7? -- Discharge lamp heater control voltage
AD8? -- Amplified ac photosignal
AD9? -- Photocell’s I/V converter voltage divided by 4
AD10? -- Case temperature (10 mV/°C)
AD11? -- Crystal thermistors
AD12? -- Cell thermistors
AD13? -- Lamp thermistors
AD14? -- Frequency calibration pot / external calibration voltage
AD15? -- Analog ground

A/D via CPU E-port:

AD16? -- Varactor voltage for 22.48 MHz VCXO (inside RF synthesizer) / 4
AD17? -- Varactor voltage for 360 MHz VCO (output of RF synthesizer) / 4
AD18? -- Gain control voltage for amplifier which drives frequency multiplier / 4
AD19? -- RF synthesizer’s lock indicator voltage (nominally 4.8 V when locked )

ST?

ST1 : Power supplies and Discharge Lamp
ST1 bit, Condition which sets bit, Corrective Action
0 -- +24 for electronic < +22 Vdc
1 -- +24 for electronics > +30 Vdc
2 -- +24 for heaters <+22 Vdc
3 -- +24 for heaters > +30 Vdc
4 -- Lamp light level too low
5 -- Lamp light level too high
6 -- Gate voltage too low
7 -- Gate voltage too high

ST2: RF Synthesizer
ST2 bit, Condition which sets bit, Corrective Action
0 -- RF synthesizer PLL unlocked
1 -- RF crystal varactor too low
2 -- RF crystal varactor too high
3 -- RF VCO control too low
4 -- RF VCO control too high
5 -- RF AGC control too low
6 -- RF AGC control too high
7 -- Bad PLL parameter

ST3: Temperature Controllers
ST3 bit, Condition which sets bit
0 -- Lamp temp below set point
1 -- Lamp temp above set point
2 -- Crystal temp below set point
3 -- Crystal temp above set point
4 -- Cell temp below set point
5 -- Cell temp above set point
6 -- Case temperature too low
7 -- Case temperature too high

ST4: Frequency Lock-Loop Control
ST4 bit, Condition which sets bit
0 -- Frequency lock control is off
1 -- Frequency lock is disabled
2 -- 10 MHz EFC is too high
3 -- 10 MHz EFC is too low
4 -- Analog cal voltage > 4.9 V
5 -- Analog cal voltage < 0.1
6 -- not used
7 -- not used

ST5: Frequency Lock to External 1pps
ST5 bit, Condition which sets bit
0 -- PLL disabled
1 -- < 256 good 1pps inputs
2 -- PLL active
3 -- > 256 bad 1pps inputs
4 -- Excessive time interval
5 -- PLL restarted
6 -- f control saturated
7 -- No 1pps input

ST6: System Level Events
ST6 bit and Condition which sets bit
0 Lamp restart
1 Watchdog time-out and reset
2 Bad interrupt vector
3 EEPROM write failure
4 EEPROM data corruption
5 Bad command syntax
6 Bad command parameter
7 Unit has been reset
</pre>
* ST? on warm start
<pre>
received: PRS_10
received: 255,255,255,243,34,255
received: 0,0,0,1,34,0
...
received: 0,0,0,1,34,0
received: 0,0,0,0,2,0
...
received: 0,0,0,0,2,0
received: 0,0,0,0,4,0
</pre>
* ST? on loss of external 1PPS
<pre>
...
received: 0,0,0,0,4,0
disconnect 1PPS input
received: 0,0,0,0,132,0
...
reconnect 1PPS input
received: 0,0,0,0,132,0
received: 0,0,0,0,4,0
...
</pre>
* ST? on coldish start
<pre>
daq13:ds$ python3 prs10.py
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,0,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 0,0,1,1,34,0 ] old [ 80,0,0,1,34,1 ] counter: 0
received [ 0,0,0,1,34,0 ] old [ 0,0,1,1,34,0 ] counter: 0
received [ 0,0,16,1,34,0 ] old [ 0,0,0,1,34,0 ] counter: 6
received [ 0,0,20,1,34,0 ] old [ 0,0,16,1,34,0 ] counter: 5
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 8
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 3
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 49
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 250
...
</pre>
* ST? on cold start, note: no bump in the 10MHz clock as reported by DS-DM clock chip
<pre>
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,21,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 64,0,21,1,34,0 ] old [ 80,0,21,1,34,1 ] counter: 0
received [ 0,0,21,1,34,0 ] old [ 64,0,21,1,34,0 ] counter: 0
received [ 0,0,20,1,34,0 ] old [ 0,0,21,1,34,0 ] counter: 158
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 23
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 47
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 40
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 249
</pre>

= det fac integration test =

Connections:
* GPS receiver "IRIG-B SEL" both switches "up" - both "on", IRIG-B format B004
* GPS receiver USB-B -> long cable -> daq13 USB-A
* GPS receiver "1PPS out" -> long BNC cable -> BNC-T -> scope (5V, no 50ohm) and Rb clock BNC "1PPS in"
* GPS receiver "IRIG-B 50ohms" -> long BNC cable -> BNC-T -> scope (5V, no 50ohm) and DS-DM LEMO input 1 (TTL mode)
* Rb clock RS232 -> RS232 straight cable -> RS232-to-USB adapter -> daq13 USB-A
* Rb clock "1PPS out" BNC -> scope (5V, no 50ohm, trig threshold rising edge 2V)
* Rb clock "10MHz output 50 Ohm" BNC -> lemo -> lemo-T -> scope (sine wave, 5V, no 50ohm) and DS-DM clock input.

Programs to run:
* on daq13: cd /home/olchansk/git/ser2net, ./ser2net -c ~/daq/ds/ser2net.conf -d
<pre>
localhost,3001:raw:600:usb-5-2-1.0:9600 -XONXOFF -RTSCTS LOCAL
</pre>
* on daq13: cd /home/olchansk/daq/ds, python3 prs10.py ### connects to ser2net
<pre>
daq13:ds$ python3 prs10.py
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,21,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 64,0,21,1,34,0 ] old [ 80,0,21,1,34,1 ] counter: 0
received [ 0,0,21,1,34,0 ] old [ 64,0,21,1,34,0 ] counter: 0
received [ 0,0,20,1,34,0 ] old [ 0,0,21,1,34,0 ] counter: 158
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 23
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 47
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 40
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 249
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 12096
received [ 0,0,0,0,20,0 ] old [ 0,0,0,0,132,0 ] counter: 2
received [ 0,0,0,0,148,0 ] old [ 0,0,0,0,20,0 ] counter: 0
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,148,0 ] counter: 575
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 11449
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 2755
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 34386
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 41035
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 113401
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 33375
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 54767
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 85059
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 33222
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 119234
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 121990
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 128184
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 56002
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 428237
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 8250
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 1388
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 30506
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 142704
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 179451
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 106182
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 68747
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 65424
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 157587
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 6932
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 1388
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 20255
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 4
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 225941
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 72183
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 26970
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 1
39287
</pre>
* on ds-dm: ./test_cdm.exe --irigb ### note sbs mismatch is because I should wrap around at 16 bits
<pre>
dataframe: S00000000S100001010S...S001010101S000001000S, sec: 00, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16724 should be 82260
dataframe: S10000000S100001010S...S101010101S000001000S, sec: 01, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16725 should be 82261
dataframe: S01000000S100001010S...S011010101S000001000S, sec: 02, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16726 should be 82262
dataframe: S11000000S100001010S...S111010101S000001000S, sec: 03, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16727 should be 82263
dataframe: S00100000S100001010S...S000110101S000001000S, sec: 04, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16728 should be 82264
dataframe: S10100000S100001010S...S100110101S000001000S, sec: 05, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16729 should be 82265
</pre>

= DS-IOGC GPS interface board =

* Rev0 git repository: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0/-/tree/main?ref_type=heads
* Rev0 schematics: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0/-/blob/main/Altium/Project%20Outputs%20for%20DS-IOGC-Rev0/SCH-DS-IOGC-Rev0.pdf?ref_type=heads
* Rev0 schematics: [[:Image:SCH-DS-IOGC-Rev0.pdf|SCH-DS-IOGC-Rev0]]
* Rev1 git repository: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev1/
* Rev1 schematics: [[:Image:SCH-DS-IOGC-Rev1.pdf|SCH-DS-IOGC-Rev1]]

== Changes Rev0 to Rev1 ==

<pre>
From: Peter Margetak <pmargetak@triumf.ca>
Subject: IOGC REV1 review
Date: Wed, 4 Sep 2024 07:31:19 +0000

Hi Konstantin,
Pls have a look at SCH for new rev. I'd like to send it to mfr next week so if you can comment by early next week. Meanwhile I work on layout and other stuff.

Changes:

New ICs - all powered +5V
U20 - inverters for RUclk RX/TX
U21 - non inverting line driver for RU-1pps-out (so you don't have to route if via GDM to see it on scope)
U22 - non inverting buffer for ext 1pps input

All Lemo connectors have the same position but they are double lemos now => new panel needed
@Marek Walczak<mailto:mwalczak@triumf.ca> you can print it ahead once pcb is done + update IOGC docs and panel description

J2A/B - Test ports for RU-1pps in and out
J5A/B - inputs for external GPS data and External source of 1pps
J6A/B - aux in/out for GDM

SW1 - no change - select RX/TX USB/GDM
SW2 - select latch sensitivity for rising/falling edge
SW3 - select source of GPS data (opto or ext) AND select source of 1pps input (latch or ext)

p.
</pre>

== PRS-10 Rb clock device ==

The Rb clock PRS-10 device provides these connections:
<pre>
RS232 RX input - serial communication, non-standard RS232
RS232 TX output - serial communication, non-standard RS232
10 MHz clock output - coax 50 Ohm high resolution 10 MHz clock
1pps output - 1 Hz clock corresponding to the 10 MHz clock
1pps input - 1pps signal from GPS receiver
</pre>

Mode of operation:
* 10 MHz clock is always running
* 1pps output is always running
* if 1pps input from GPS received is present, after 256 pulses PRS-10 will sync it's 1pps output with the 1pps input by adjusting the frequency of the 10 MHz clock
* when unlocked: 1pps output and 1pps input unrelated
* when locked to GPS: 1pps output and 1pps input always go up and down at the same time

Theory of operation:
* 10 MHz clock is produced by a high-quality crystal (stable on the scale of seconds)
* crystal oscillator is synchronized to a Rb cell (stable on the scale of hours and days)
* Rb cell resonant frequency is synchronized to the GPS 1pps signal (stable on the scale of months and years)

== Rev1 connections ==

* LEMO connectors (front panel)
<pre>
LEMO J2A output - Rb clock 1pps in monitor
LEMO J2B output - Rb clock 1pps out monitor
LEMO J5A input - GPS IRIG-B from GPS receiver to FPGA (VCL-2705)
LEMO J5B input - GPS 1pps from GPS receiver to PRS-10 (VCL-2705)
LEMO J6A input - AUX-IN to FPGA
LEMO J6B output - AUX-OUT from FPGA
</pre>

* SMB connectors (back)
<pre>
SMB J3 output - GPS 1pps loopback to LNGS
SMB J4 input - LNGS GPS data input
</pre>

* LEDs
<pre>
D1 - same as LEMO J2A out (Rb clock 1pps in)
D2 - same as SMB J3 out (GPS 1pps from LNGS or from a GPS receiver)
D5 - controlled by FPGA-OUT-LED1
D6 - controlled by FPGA-OUT-LED2
D7 - PRS-10 24V power ok
</pre>

* switches
<pre>
SW1A and SW1B - route PRS-10 RS232 to USB or to FPGA
SW2A - route PRS-10 1pps input from SMB J4 (LNGS) or from LEMO J5B (GPS receiver 1pps)
SW2B - route FPGA-IN-GPSDATA input from SMB J4 (LNGS) or from LEMO J5A (GPS receiver IRIG-B data)
SW3 - LNGS 1pps from rising edge or from falling edge of SMB J4 (LNGS)
</pre>

== Rb clock cable ==
<pre>
Rb clock ----- DS-IOGC side, pin numbers are as labeled on the cable connectors

1 - 1pps out - 8 - 1pps out --- correct
2 - nc
3 - nc
4 - TXD - 3 - RU-DATA-OUT - USB-RX input --- correct
5 - 1pps in - 2 - 1pps in --- correct
6 - +24V - 10 - +24V --- should by pin 1 to use both +24V pins?
7 - RXD - 15 - RU-DATA-IN - USB-TX output --- correct
8 - nc
9 - +24V - 10 - +24V --- correct
10 - GND - 9 - GND --- correct
</pre>

== VX connections ==
<pre>
VXA_TX0 - FPGA-OUT-LED2 - D6 LED ("10 MHz clock")
VXA_TX1 - FPGA-OUT-LED1 - D5 LED ("GPS DATA")
VXA_TX2 - FPGA-OUT-RU1PPS-EN - enable 1pps to Rb clock
VXA_TX3 - not used (62.5 Hz clock)

VXB_TX0 - FPGA-TX - PRS-10 RS-232 out
VXB_TX1 - FPGA-OUT-AUX - 5V TTL J5 LEMO out
VXB_TX2 - FPGA-OUT-OPTO1PPS-EN - PRS-10 power enable
VXB_TX3 - not used (62.5 MHz clock)

VXA_RX0 - FPGA-IN-OPTO1PPS - 1pps from GPS
VXA_RX1 - FPGA-IN-GPSDATA - J4 GPS data
VXA_RX2 - n/c
VXA_RX3 - n/c

VXB_RX0 - n/c
VXB_RX1 - FPGA-IN-AUX - J5 LEMO TTL input
VXB_RX2 - FPGA-RX - PRS-10 RS-232 in
VXB_RX3 - FPGA-IN_RU1PPS - PRS-10 1pps output
</pre>

== test sequence ==

* ./test_cdm_local.exe --writereg 7 0 ### clear reg 7
* ./test_cdm_local.exe --writereg 8 0 ### set vx_tx mux to vx_tx control from reg 7
* connect blue cable to GDM port 6 (next to the ethernet connector)
* ./test_cdm_local.exe --writereg 7 0x4000 ### power up
* ./test_cdm_local.exe --writereg 7 0x4100 ### right LED D6
* ./test_cdm_local.exe --writereg 7 0x4200 ### left LED D5
* ./test_cdm_local.exe --writereg 7 0x6000 ### J5 LEMO measure +5VDC
* install LEMO jumper between LEMO J5 and J6
* write 0x4000 then ./test_cdm_local.exe 6 ### reads 0x5C, bit 5 0x20 reads 0
* write 0x6000 then ./test_cdm_local.exe 6 ### reads 0x7C, bit 5 0x20 reads 1
* write 0x4000 to clear all bits
* 10 MHz output ok - IN0 is good: ./test_cdm_local.exe --cc
<pre>
Clock chip state 1, status: OOF_IN2 IN1 IN_SEL_1 HOLD_HIST_VALID
</pre>
* without 10 MHz reports IN0 LOS (no signal) and OOF (wrong frequency)
<pre>
Clock chip state 1, status: LOS_IN0 OOF_IN0 OOF_IN2 IN1 IN_SEL_1 HOLD_HIST_VALID
</pre>
* IRIG-B via AUX-IN is ok: ./test_cdm_local.exe --irigb
<pre>
dataframe: S00100110S111001000S...S011110000S010000000S, sec: 34, min: 17, hrs: 00, day of year: 223, year: 2024, date: 10 aug, sbs: 1054 should be 1054
</pre>
* ./test_cdm_local.exe --writereg 8 9 ### set vx_tx mux to GPS control, hard enables PRS-10 power and PRS-10 1pps in
* observe 1pps counters in reg 68 count at 1 Hz: rb_1pps 0x51->0x52, gps_1pps 0xd5->oxd6
<pre>
root@dsdm:~# ./test_cdm_local.exe 68
ds20k_reg[68] is 0x0051d520 (5362976)
ds20k_reg[68] is 0x0052d624 (5428772)
...
</pre>
* observe PRS-10 can see the 1pps signal "130" changes to "2" after 243 seconds to "4"
* observe period of GPS and PRS-10 1pps is identical (plus/minus 1 clock)
<pre>
root@dsdm:~# ./test_cdm_local.exe 13 14
ds20k_reg[13] is 0x077356d4 (124999380)
ds20k_reg[14] is 0x077356d4 (124999380)
</pre>
* observe GDM/CDM clocks are exactly 125 MHz, we are running on DS-DM internal oscillator
<pre>
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735943 (125000003) should be ~125 MHz
0x1034 rx_clk: 0x07735943 (125000003) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735943 (125000003) should be ~125 MHz
0x103C tx_clk: 0x07735943 (125000003) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>
* switch CC to external clock:
<pre>
root@dsdm:~# ./test_cdm_local.exe --cc-in0
CC use clock input 0: 10 MHz LEMO external clock
root@dsdm:~# ./test_cdm_local.exe --cc
Polling CC status...
Clock chip state 1, status: IN0 IN_SEL_REGCTRL IN_SEL_0 HOLD_HIST_VALID
</pre>
* observe rx_clk and tx_clk are now slightly off: they run from PRS-10 10 MHz clock and are measured against the DS-DM internal oscillator.
<pre>
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x94b12519, ds20k version 0x20240814
DS-DM firmware build 0x94b12519, ds20k version 0x20240814
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x077358e1 (124999905) should be ~125 MHz
0x1034 rx_clk: 0x07735b49 (125000521) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735b49 (125000521) should be ~125 MHz
0x103C tx_clk: 0x07735b49 (125000521) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>
* observe GPS and PRS-10 1pps period is now exactly 125*10^6 of 8ns clocks (equal to 1 second)
<pre>
root@dsdm:~# ./test_cdm_local.exe 13 14
ds20k_reg[13] is 0x0773593f (124999999)
ds20k_reg[14] is 0x0773593f (124999999)
</pre>
* look at them repeatedly, observe reg 13 "GPS 1pps period" has some wobble, reg 14 "Rb clock 1pps period" is steady. this is as expected: if GPS 1pps was steady, we do not need to Rb clock. PRS-10 1pps is derived from the PRS-10 10 MHz clock and is measured against the 125 MHz clock derived from the same 10 MHz clock in the GDM FPGA.

Test status:

* GPS 1pps to SMB-in ok (LED flashes)
* GPS 1pps to FPGA ok
* GPS data to FPGA ok
* GPS 1pps to PRS-10 enabled from FPGA ok
* PRS-10 Rb clock 1pps out to FPGA ok
* PRS-10 Rb clock 1pps out to FPGA to LEMO AUX out ok
* can see GPS 1pps, IRIG-B, PRS-10 1pps out, 10 MHz on the scope, ok
* PRS-10 syncs on leading edge (0->1) of GPS 1pps signal, ok
* reg 13 and 14 1pps periods are identical, ok
* CC locks on PRS-10 10 MHz clock, DS-DM runs on PRS-10 clock, ok
* NOT TESTED - smb output
* NOT TESTED - optical converter fiber to SMB
* NOT TESTED - optical converter SMB to fiber
* NOT TESTED - SMB loopback
* NOT TESTED - fiber loopback

= VX busy logic =

= DS-20K DAQ =

== Overview ==

DS-DM, GDM and CDM are key parts of the DS-20K DAQ system:
* common clock distribution from external clock (atomic clock, GPS) to GDM to per-quadrant CDMs to VX digitizers
* common trigger distribution from GDM internal algorithm or external input to all VX digitizers
* run control: GDM, CDM, VX all start recording data at the same time (clock and timestamp reset)
* collection of trigger data from VX digitizers to per-quadrant CDMs to GDM

== Deliverables ==

* hardware and firmware for GDM to CDM clock distribution
* hardware and firmware for CDM to VX clock distribution
* hardware and firmware for GDM external clock input (atomic clock or GPS)
* hardware and firmware for CDM and VX serial communications (VX LVDS I/O connector)
* firmware for run control (timestamp reset and sync): GDM to CDM to VX
* firmware for common trigger distribution: GDM to CDM to VX
* firmware for trigger data flow: VX to CDM to GDM
* firmware for busy control: VX to CDM to GDM back to CDM to VX
* firmware for flow control: FEP to GDM MIDAS frontend to GDM to CDM to VX
* GDM MIDAS frontend: clock selector and monitoring, trigger and run control, busy and flow control, GDM housekeeping
* CDM MIDAS frontend: clock monitoring, CDM housekeeping

specific performance:
* GDM external clock: 10 MHz GPS clock
* GDM to CDM fiber link:
** clock XXX MHz
** link data rate: XXX Gbit/sec
** CDM recovered clock: XXX MHz
** CDM recovered clock jitter: XXX ns
** phase alignment between CDMs: XXX ns
** phase alignment between CDMs persists across reboots, power cycles, firmware updates
** phase alignment between CDMs should be easy to measure
** phase alignment between CDMs should be easy to recalibrate if hardware parts are replaced (DS-DM boards, fiber transceivers, fiber cables, etc)
** data packet bandwidth: XXX Mbytes/sec
** data packet latency: XXX clocks
** data packet skew between CDMs: XXX clocks
* CDM to VX clock:
** clock: XXX MHz
** jitter, all CDM clock outputs: XXX MHz
** phase alignment between all CDM clock outputs: XXX ns
* CDM to VX trigger:
** TBD (use the VX "sync" input or VX LVDS I/O line or VX serial link packet)
* CDM to VX serial link:
** clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
** bit rate: XXX bits/sec
** latency: XXX link clocks
** maximum skew between VXes: XXX ns
* VX to CDM serial link:
** clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
** bit rate: XXX bits/sec
** latency: XXX link clocks
** maximum skew between VXes: XXX ns
* timestamp reset:
** maximum skew between VXes: XXX ns
* busy round trip time: XXX ns (VX to CDM to GDM back to CDM to VX)
* flow control latency: XXX ns (FEP software to GDM MIDAS frontend to GDM to CDM to VX)

== Technical risk items ==

this refers to unexpected behaviour and performance of
system components, causes big difficulty in implementing the system,
prevents delivery of deliverables, and prevents or negatively affects operation
of the DS-20K DAQ or of the whole experiment.

(14-sep-2022, list is not sorted by any criterial: severity, probability, ease of investigation)

(stability of course is long term stability, across hours, days, weeks, months, years)

* stability of Enclustra FPGA modules (crashes/year, failures to boot/year, flash corruption/year)
* stability of GDM external clock PLL (lock loss/year)
* stability of CDM recovered clock (lock loss/year, unexpected phase drifts, etc)
* unexpected failures or bit error rates in GDM-CDM fiber links
* stability of CDM VX clock outputs (stability of clock cleaner chip)
* stability of VX internal clock distribution (VX PLL lock loss events)
* stability of VX CAEN base firmware (different versions of CAEN base firmware have different clock distribution behaviour)
* strange things in CAEN base firmware (unexpected clocking of LVDS I/O, unexpected phase shifts between clocks, etc)
* DS-DM and VX hardware problems (incompatible LVSD I/O, incompatible clock signals, etc)

== Milestones ==

(14-sep-2022: at this stage of the project, priority must be given to identifying and retiring (so called) technical risk factors. it is not good to build the complete system only to discover that (for example) some Enclustra FPGA modules require 5 attempts to boot and erase their flash memory contents once a month. Both example are real-life actual problems that caused big difficulties in GRIFFIN/TIGRESS and ALPHA-g experiments).

Development and testing milestones in time reversed order:

* full DAQ data challenge: all VXes, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
* one quadrant data challenge: 1 VME crate of VX, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
* vertical slice data challenge: 1 VME crate, 2 VX, 2 CDM (1 VX per CDM), GDM, etc operate as designed
* GDM-CDM link finalized (data rate frozen, data packet format frozen, data content permitted to change)
* CDM-VX serial link finalized (data rate frozen, data packet format frozen, data content permitted to change)
* run control (timestamp reset) and trigger distribution design agreed upon, frozen (list of possible triggers permitted to change)
* VX to CDM to GDM data flow design agreed upon, frozen (data contents permitted to change)
* major technical risk items retired (all hardware and firmware is working as expecred without mysteries and surprises, all problems are identified, investigated, resolved, solutions tested)
* stable operation of CDM-VX serial links in vertical slice system
* stable operation of GDM to CDM clock in vertical slice system
* stable operation of CDM to VX clock in vertical slice system
* vertical slice system assembled (1 VME crate, 2 VX, 2 CDM, 1 GDM, network, FEP, TSP, MIDAS)

= ZZZ =

ZZZ

DS-PC

2024-09-06T23:01:58Z

Mwalczak: adding instructions for DSTS

== DarkSide-20k Front-End (DSFE) machine activation ==

<pre>
ssh wheel@dsdaqgw
psswd: (ask Konstantin or Marek)

sudo /bin/bash
psswd: (ask Konstantin or Marek)

you are root now, see:
id

cd /nfsroot

zfs get all | grep dedup

mkdir dsfeXX

cd dsfe05

rsync -avx . ../dsfeXX/

cd ../dsfeXX/

edit etc/hostname
dsfe05 -> dsfeXX

edit etc/fstab (nothing to change...)

edit etc/rc.local

and comment out:

# ifconfig enp1s0f1 192.168.0.1 netmask 255.255.255.0
# systemctl restart dnsmasq

inspect /tftpboot/uefi/pxelinux.cfg/default
(no action here)

cd /etc
edit /etc/dnsmasq.conf
add dsfeXX under:
# uefi pxe

edit /etc/hosts
add dsfeXX under:
# FEP and TSP network

now we need dsfeXX MAC address

connect 10gige to the right DAC port on the back
power up

tail -f /var/log/syslog
last line (~ 1 min after powering up) will have dsfeXX MAC address
like here:
May 22 16:46:40 dsdaqgw dnsmasq-dhcp[110361]: 1756546062 DHCPDISCOVER(enp1s0f1) 40:a6:b7:c1:d2:8c ignored

edit /etc/dnsmasq.conf
add line below with dsfeXX MAC address below:
# backend network 192.168.4.x

systemctl restart dnsmasq

add dsfeXX to /etc/exports
exportfs -rv

cycle power on dsfeXX

observe /var/log/syslog:
tail -f /var/log/syslog
gets an ip address
loads /tftpboot/uefi/pxelinux.cfg/default
loads uefi/syslinux.efi
nfsroot authenticated mount request from 192.168.4.XXX for /nfsroot/dsfeXX (/nfsroot/dsfeXX)

ping dsfeXX
ssh dsfeXX

edit /etc/gonodeinfo.conf
update it with feXX

top
ifconfig -a
df

observe /var/log/mini_httpd.log
http get /uefi/vmlinuz-6.5.0-18-generic
http get /uefi/initrd.img-6.5.0-18-generic

add network connection to the VX network 192.168.0.x.

in /etc/rc.local:

systemctl stop dnsmasq
ifconfig enp1s0f1 192.168.0.156 netmask 255.255.255.0
sysctl -w net.ipv4.ip_forward=1

192.168.0.156 is dsfe060 (dsfe06, network 0)
192.168.0.157 is dsfe070 and so forth

enp1s0f1 of each machine connected by DAC cable to vlan2 ports on the juniper switch ports 12,13,14,15.

ports 12,13,14,15 switched from 1gige back to 10gige (using the junos "delete" command).

ssh dsvslice4
edit /etc/netgroup
</pre>

== DarkSide-20k Time Slice Processor (DSTS) machine activation ==
<pre>
ssh wheel@dsdaqgw
psswd: (ask Konstantin or Marek)

sudo /bin/bash
psswd: (ask Konstantin or Marek)

you are root now, see:
id

cd /nfsroot

mkdir dsts0X

cd dsts0X

rsync -av ../dsts01/ .

edit etc/hostname
-> dsts0X
edit etc/gonodeinfo.conf
-> dsts0X

edit /etc/exports

add line:

/nfsroot/dsts0X dsts0X(rw,no_root_squash,async,no_subtree_check)

exportfs -rv

edit /etc/dnsmasq.conf

under "pxe boot" add:

dhcp-option-force=tag:dsts0X,option:root-path,192.168.4.1:/nfsroot/dsts0X

under "backend network 192.168.4.x" add

add / comment out

dhcp-host=ac:1f:6b:f8:2f:de,ac:1f:6b:f8:2f:df,dsts0X,infinite,set:pxe,set:dsts0X

remember to 1) chenage "set:pxe" and 2) add proper address (later)

cd ~
systemctl restart dnsmasq

ping dsts0X

ssh dsts0X

ifconfig

enp1s0f0: ether
enp1s0f1: ether

get the addresses and add to /etc/dnsmasq.conf (to the line added before)

systemctl restart dnsmasq

Might happen that the video driver will be unhappy about absent monitor and freeze the machine.
Adding "nomodeset" to the kernel command line (in tftpboot/pxelinux.cfg/default) seems fix it.
</pre>

DS-PC

2024-09-06T22:52:55Z

Mwalczak: DSFE setup info

== DarkSide-20k Front-End (DSFE) machine activation ==

<pre>
ssh wheel@dsdaqgw
psswd:

sudo /bin/bash
psswd:

you are root now, see:
id

cd /nfsroot

zfs get all | grep dedup

mkdir dsfeXX

cd dsfe05

rsync -avx . ../dsfeXX/

cd ../dsfeXX/

edit etc/hostname
dsfe05 -> dsfeXX

edit etc/fstab (nothing to change...)

edit etc/rc.local

and comment out:

# ifconfig enp1s0f1 192.168.0.1 netmask 255.255.255.0
# systemctl restart dnsmasq

inspect /tftpboot/uefi/pxelinux.cfg/default
(no action here)

cd /etc
edit /etc/dnsmasq.conf
add dsfeXX under:
# uefi pxe

edit /etc/hosts
add dsfeXX under:
# FEP and TSP network

now we need dsfeXX MAC address

connect 10gige to the right DAC port on the back
power up

tail -f /var/log/syslog
last line (~ 1 min after powering up) will have dsfeXX MAC address
like here:
May 22 16:46:40 dsdaqgw dnsmasq-dhcp[110361]: 1756546062 DHCPDISCOVER(enp1s0f1) 40:a6:b7:c1:d2:8c ignored

edit /etc/dnsmasq.conf
add line below with dsfeXX MAC address below:
# backend network 192.168.4.x

systemctl restart dnsmasq

add dsfeXX to /etc/exports
exportfs -rv

cycle power on dsfeXX

observe /var/log/syslog:
tail -f /var/log/syslog
gets an ip address
loads /tftpboot/uefi/pxelinux.cfg/default
loads uefi/syslinux.efi
nfsroot authenticated mount request from 192.168.4.XXX for /nfsroot/dsfeXX (/nfsroot/dsfeXX)

ping dsfeXX
ssh dsfeXX

edit /etc/gonodeinfo.conf
update it with feXX

top
ifconfig -a
df

observe /var/log/mini_httpd.log
http get /uefi/vmlinuz-6.5.0-18-generic
http get /uefi/initrd.img-6.5.0-18-generic

add network connection to the VX network 192.168.0.x.

in /etc/rc.local:

systemctl stop dnsmasq
ifconfig enp1s0f1 192.168.0.156 netmask 255.255.255.0
sysctl -w net.ipv4.ip_forward=1

192.168.0.156 is dsfe060 (dsfe06, network 0)
192.168.0.157 is dsfe070 and so forth

enp1s0f1 of each machine connected by DAC cable to vlan2 ports on the juniper switch ports 12,13,14,15.

ports 12,13,14,15 switched from 1gige back to 10gige (using the junos "delete" command).

ssh dsvslice4
edit /etc/netgroup
</pre>

== DarkSide-20k Time Slice Processor (DSTS) machine activation ==

Hardware

2024-09-06T21:56:35Z

Mwalczak: adding "DarkSide-20k PC setup" page

== Hardware information ==

* [http://daq-plone.triumf.ca/HR/ Hardware documentation on the daq-plone site]
* [[PcHardware]] DAQ PC Configurations
* [[VT4_info]] VT4 information
* [[VF48]] VF48 information
* [[VME-ALPHA-TTC]] VME-ALPHA-TTC information
* [[VME-NIMIO32]] information
* [[VME-NIMIO32-Rev3]] information
* [[VME-NEW-IO]] working specs for the new VME I/O board
* [[VME-PPG32]] VME-PPG32 information
* [[VME-CPU]] VMIC/GEFANUC V7648, V7805, V7865 information
* [[VME-CDM]] VME-CDM GRIFFIN Clock Distribution Board
* [[VME-GRIF-ADC16-Rev1]] VME-GRIF-ADC16 Revision 1 as used by ALPHA-h (ALPHA16 firmware)
* [[Phidget]] Phidget devices information
* [[MityARM]] CriticalLink MityARM information
* [[MityCAMAC]] MityARM/MitySoM CAMAC interface
* [[Cyclone5-FPGA-ARM]] Altera Cyclone5 ARM SoC information
* [[BBB]] BeagleBone Black
* [[RPI3]] RaspberryPi3
* [[DE10-Nano]] DE10-Nano
* [https://daqstore.triumf.ca/AgWiki/index.php/Chronobox Chronobox]
* [[HPDVM]] Setup for HP/Agilent DVM
* [[VMIC]] VMIC Family
* [[CAEN]] CAEN Family
* [[SIS]] Family
* [[GSI_TRB3]] GSI TRB3 FPGA-TDC
* [[ESP32]] ESP32 microcontroller
* [[MVM-VC]] MVM VC controller board
* [[DS-DM]] DarkSide-20k Data Manager board
* [[DS-PC]] DarkSide-20k PC setup
* [[DL-TDC]] DarkLight FPGA TDC
* [[DL-DB]] DarkLight Distribution board and SiPM board

File:SCH-DS-IOGC-Rev0.pdf

2024-09-05T21:48:29Z

Mwalczak: Mwalczak reverted File:SCH-DS-IOGC-Rev0.pdf to an old version

File:SCH-DS-IOGC-Rev0.pdf

2024-09-05T21:28:22Z

Mwalczak: Mwalczak reverted File:SCH-DS-IOGC-Rev0.pdf to an old version

DS-DM

2024-09-05T21:27:14Z

Mwalczak: /* DS-IOGC GPS interface board */

= DS-DM =

DarkSide-20k Global and Crate Data Manager board (GDM and CDM).

Global Data Manager (GDM):
* clock distribution to CDM boards (including GPS/atomic clock source)
* collection of trigger data from CDM boards, processing and distribution of trigger decision to CDM boards
* run control
* integration with GPS 10MHz and 1pps clocks and GPS/IRIG date and time information

Crate Data Manager (CDM):
* clock distribution from GDM to CAEN VX digitizers
* receive trigger data from CAEN VX digitizers
* send trigger data to GDM
* run control and dead time control

= Links =

* https://dsvslice.triumf.ca/vx_napoli - DS vertical slice at TRIUMF
* https://edev-group.triumf.ca/fw/exp/darkside/gcdm - git repository, DS-DM firmware
* https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0 - git repository, DS-DM board
* https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0/-/blob/master/Altium/Project%20Outputs%20for%20DS-DM-Rev0/SCH-DS-xDM-Rev0.PDF - DS-DM schematics
* https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0/-/blob/master/Testing/Clk3_XO_125_to_fpgaIN_recoveredMGTclk_to_IN2_Si5394-RevA-Project.slabtimeproj
* https://docs.xilinx.com/v/u/en-US/ds593 - Xilinx Platform Cable USB II
* https://www.enclustra.com/en/products/system-on-chip-modules/mercury-xu8/ - Enclustra
* https://bitbucket.org/team-ds-dm/ds-dm-software - git repository, DS-DM MIDAS frontend
* https://bitbucket.org/team-ds-dm/phasemeasurement - VX Phase measurement script
* https://ladd00.triumf.ca/daqinv/frontend/list/178 - inventory database
* https://daq00.triumf.ca/elog-ds/DS-DAQ - DS-DAQ elog
* https://bitbucket.org/ttriumfdaq/dsproto_vx2740/src/develop/ - VX DSFE MIDAS frontend
* https://dsdaqgw.triumf.ca/vslice/ - vertical slice DAQ
* https://daq00.triumf.ca/DaqWiki/index.php/DarkSide - DS-DAQ Wiki page
* https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0 - GPS and Rb Clock adapter board

= Onboard hardware =

* jtag chain: arm_dap_0 0x5BA00477, xczu4_1 0x04721093
* Eclustra Mercury+ XU8 module: ME-XU8-4CG-1E-D11E-R2.1
** Xilinx® Zynq Ultrascale+™ MPSoC XCZU4CG-1FBVB900E
** DDR4 ECC SDRAM (PS) 2 GB
** DDR4 SDRAM (PL) 1GB
* ethernet mac chip: AT24MAC402-SSHM-T ("602" chip is wrong)
* USB UART for Enclustra serial console, micro-USB, 115200n8
* clock chip: SI5394A-A-GM and oscillator CS-044-054.0M (54 MHz)
* U23 3.3V current meter and thermometer, LTC2990IMS#TRPBF
* LEDs:
** LED_FP A/B/C/D 0/1/2/3
** led1 - 3V3_SW_ON, SOM_POWER_GOOD - Enclustra FPGA module 3.3V power is good
** led2 - LTM4624 PGOOD
** led3 - FPGA_DONE - FPGA has booted
** led4 - TP-S-1, PCLK_P
** led5 - TP-S-2, PCLK_N
* LEMO connectors (top to bottom)
** J4 - input (NIM/TTL) (EXT_IN_LV(1), EXT_IN_LV(2))
** J5 - input (NIM/TTL) (EXT_IN_LV(3), EXT_IN_LV(4))
** J6 - external clock (GPS 10MHz and PPS)
** J7 - output (NIM/TTL) (EXT_OUT(1), EXT_OUT(2))
* SMA connectors
** J9, J10 - CLK_CCA from U6 C.C.
** J11, J12 - CLK_TP0
* RJ45 ethernet connector (100mbit: green light on, yellow light flashes when there is traffic)
* SFP connector (SFP is FTLF8526P3BNL, 6 Gbit/sec, 850 nm, 300m 50/125um OM3 MMF)
* 4 QSFP connectors (GDM)
* 6 VX connectors (CDM)

= Buttons, jumpers and switches =

Buttons:

* PB1 - HRST - reboot FPGA (power-on reset)
* PB2 - SRST - (SRSTn) - reboot ARM CPU

Switches:

* SW1 - boot mode BM0, BM1 [-->]
* SW2 - LEMO output NIM<->TTL
* SW3 - LEMO input 1 and 2 NIM/TTL
* SW4 - LEMO input 2 and 4 NIM/TTL
* SW5 - LEMO clock input NIM/TTL
* SW6 - serial console select. [PS<--PL] PS is ARM CPU, PL is FPGA.

= Front panel =

<pre>
| top
|
| LED-FP1 | LED_FP(0,1,2,3)
|
| SFP J???
|
| LEMO J4-LEFT, J4-RIGHT | EXT_IN_LV(1), EXT_IN_LV(2)
| LEMO J5-LEFT, J5-RIGHT | EXT_IN_LV(3), EXT_IN_LV(4)
| LEMO J6-LEFT, J6-RIGHT | CLK_EXT1, CLK_EXT0 (125 MHz only)
| LEMO J7-LEFT, J7-RIGHT | EXT_OUT(1), EXT_OUT(2)
|
| J-VX-1
| J-VX-2 or QSFP-1
| J-VX-3 or QSFP-2
| J-VX-4 or QSFP-3
| J-VX-5 or QSFP-4
| J-VX-6
|
| RJ45 J3 ethernet
|
| bottom
</pre>

= VX adapter board =

LVDS I/O connector
<pre>
34 pin connector: 0|:::: :::: :::: :::: :|15,16 n/c
</pre>

== split-cable connection ==

<pre>
0 .. 7 -> N/C

8 -> VX_RX(3) - not used
9 -> VX_RX(2) - busy VX to CDM
10 -> VX_RX(1) - DS20K 125 MHz serial data VX to CDM
11 -> VX_RX(0) - DS20K 62.5MHz clock VX to CDM

12 <- VX_TX(0) - TRG CDM to VX
13 <- VX_TX(1) - TSM CDM to VX, to be VETO CDM to VX
14 <- VX_TX(2) (set by jumper routed here or to VX CLKIN SYNC) - 125 MHz serial data CDM to VX
15 <- n/c (CLK routed to VX CLKIN CLK) - 62.5 MHz clock CDM to VX

16 - n/c
</pre>

== one-to-one connection ==

<pre>
0 -> VX2_RX(3)
1 -> VX2_RX(2)
2 -> VX2_RX(0)
3 -> VX2_RX(1)

4 <- CLK
5 <- VX2_TX(0)
6 <- VX2_TX(1)
7 <- VX2_TX(2)

8 -> VX1_RX(3)
9 -> VX1_RX(2)
10 -> VX1_RX(1)
11 -> VX1_RX(0)

12 <- VX1_TX(0)
13 <- VX1_TX(1)
14 <- VX1_TX(2) (set by jumper routed here or to VX CLKIN SYNC)
15 <- n/c (CLK routed to VX CLKIN CLK)

16 - n/c
</pre>

= Board schematics =

* [[File:SCH-DS-xDM-Rev0.PDF]]
* note: FPGA pin annotations ("IO", "SCLK", "PCLK", etc) on the schematics are bogus, instead, trace them to the FPGA pins.
* note: ENC A is J800, ENC B is J801, ENC C is J900 (schematic name to enclustra name)
* note: Enclustra special pins: "GC" is "clock capable", "HDGC" is "clock capable", "MGTREFCLK" is MGT reference clocks.
* board modifications:
** ethernet mac chip
** NIM output (no U15, etc)
** RJ45 wrong pinout (board mod or special ethernet cable)
** 125 MHz clock mods (TBW)
** disconnect QSFP0_SEL from SFP_RS0 and QSFP1_SEL from SFP_RS1, these signals are not used by modern SFPs
** provide SFP i2c modsel to allow SFP and QSFP at the same time on address 0x50

= FPGA MGT blocks =

<pre>
* SFP - ENC B45,B47 and B48,B50 - FPGA D5,D6 MGTHTX0_D and D1,D2 MGTHRX0_D
* QSFP0 TX0 - ENC C13,C17 - FPGA H5,H6 MGTHTX0_C
* QSFP0 TX1 - ENC C21,C25 - FPGA G7,G8 MGTHTX1_C
* QSFP0 TX2 - ENC C29,C23 - FPGA F5,F6 MGTHTX2_C
* QSFP1 TX0 - ENC C37,C41 - FPGA E7,E8 MGTHTX3_C
* QSFP1 TX1 - ENC C45,C47 - FPGA P5,P6 MGTHTX0_B
* QSFP1 TX2 - ENC C51,C53 - FPGA M5,M6 MGTHTX1_B
* QSFP2 TX0 - ENC C57,C59 - FPGA L3,L4 MGTHRX2_B
* QSFP2 TX1 - ENC C63,C65 - FPGA K5,K6 MGTHTX3_B
* QSFP2 TX2 - ENC C75,C77 - FPGA W3,W4 MGTHTX0_A
* QSFP3 TX0 - ENC C79,C81 - FPGA V5,V6 MGTHTX1_A
* QSFP3 TX1 - ENC C85,C87 - FPGA T5,T6 MGTHRX2_A
* QSFP3 TX2 - ENC C89,C91 - FPGA R3,R4 MGTHTX3_A
</pre>

= Clock distribution =

Simplified:

* 125 MHz osc -> CLK_XO_125 -> MGTREFCLK0_A -> not used
* 125 MHz osc -> CLK3_XO_125 -> MGTREFCLK1_B -> SFP RX ref clock, QSFP RX and TX ref clock (this is not final design!)
* (disconnected) 125 MHz osc -> CLK2_XO_125 -> FPGA AG8,AH8 (GC)
* 125 MHz osc -> C.C. in1

* C.C. in0 <- CLK_EXT1 (10 MHz GPS clock)
* C.C. in1 <- 125 MHz osc
* (disconnected) C.C. in2 <- CLK_CC_IN <- MGTREFCLK0_D <- SFP RX clock (cannot be used because of uncontrollable phase)
* C.C. in2 <- CLK2_XO_125 <- FPGA AG8,AH8 (GC) <- SFP RX recovered 125 MHz clock
* C.C. in3 <- CLK_FB

* C.C. 125 MHz -> CLK_CC_OUT0 -> MGTREFCLK0_B -> QSFP RX and TX ref clock (final design)
* C.C. 125 MHz -> CLK_CC_OUT1 -> MGTREFCLK1_D -> SFP TX clock
* C.C. 62.5 MHz -> VX clock fanout

proposed changes:
* add C.C. 125 MHz -> new CLK_CC_OUT2 (old CLK2_XO_125) -> FPGA AG8,AH8 (GC)
* change C.C. in2 <- new CLK_CC_IN <- FPGA AK8,AK9 (non-GC)

Complete:

<pre>
125 MHz oscillator - U5 fan out -
q0 -> CLK_XO_125 -> ENC C72,C74 -> FPGA R7,R8 MGTREFCLK0_A (not used)
q1 -> U6 C.C. in1
q2 -> CLK3_XO_125 -> ENC C7,C9 -> FPGA J7,J8 MGTREFCLK1_B -> SFP RX reference clock, QSFP RX and TX reference clocks (not final design!)
q3 -> disconnected on the board, was CLK2_XO_125 -> ENC C151,C153 -> FPGA AG8,AH8 (GC)

U6 C.C (clock cleaner) -

in0 <- CLK_EXT1 (presumably GPS 10 MHz ref clock)
in1 <- 125 MHz oscillator via U5
in2 <- (was: CLK_CC_IN <- ENC B10,B12 <- FPGA D9,D10 MGTREFCLK0_D <- SFP RX clock, 125 MHz)
in2 <- CLK2_XO_125 <- ENC C151,B153 <- FPGA AG8,AH8 (GC) <- mgt_link_data_to_processing.rx_data_clk (SFP RX data clock, 125 MHz)
in3 <- CLK_FB

out0 -> CLK_CCA -> U12 (125 MHz)
out1 -> CLK_CCB -> VX1..6 (62.5 MHz)
out2 -> CLK_CCC -> VX7..12 (62.5 MHz)
out3 -> CLK_FB into in3

CLK_CCA -> U12 (125 MHz fan out) ->

Q0 -> not used
Q1 -> CLK_CC_OUT0 -> ENC C3-5 -> FPGA L7,L8 MGTREFCLK0_B -> QSFP RX and TX reference clocks (final design)
Q2 -> CLK_CC_OUT1 -> ENC B3-5 -> FPGA B9,B10 MGTREFCLK1_D -> SFP TX clock
Q3 -> not used
Q4 -> not used
Q5 -> SMA J9/J10

CLK_TP0 - FPGA N7,N8 MGTREFCLK1_A - ENC C69,C71 - DS-DM SMA J11, J12 (NOT IN CDM PROJECT)

Notes:
* CLK_XO_125 (125 MHz osc) is not used
* 62.5 MHz VX clock does not go into the FPGA
* CLK_EXT0 going to in1 of U5 cannot be used. only permitted frequency is 125 MHz (it drives the MGT reference clocks) and is this frequency is too high for LEMO cables and connectors

Proposed modifications:
- CLK_CC_IN: from FPGA output pin (ENC C142,C144 - FPGA AK8,AK9) to clock cleaner in2 (instead of CLK2_XO_125 pins)
- CLK_CCA -> U12 -> currently unused out3 -> CLK2_XO_125 FPGA pins
- repurpose CLK_EXT0 at GPS 1pps/IRIG input to FPGA
</pre>

= I2C bus =

* I2C_SCL is J-ENC-A pin 111 I2C_SCL is FPGA I2C_SCL_PL AB13 (IO_L1N_TOL1D_64) and I2C_SCL_PS F18 (PS_MIO10)
* I2C_SDA is J-ENC-A pin 113 I2C_SDA is FPGA I2C_SDA_PL AH13 (IO_L7N_T1L1Q_AD13N_64) and I2C_SDA_PS G18 (PS_MIO11)

* XU8 secure EEPROM ATSHA204A at 0x64, this is 0110'010X -> linux _011'0010 is 0x32. (but responds to scan and read at 0x33)
* U4 ethernet mac chip, EEPROM at 1010 A2 A1 A0 X and MAC/serial_no at 1011 A2 A1 A0 X. A0=VCC, A1=VCC, A2=GND -> linux _101'0011 and _101'1011 is 0x53 and 0x5B.
* U6 clock chip, address 1101 0 A1 A0 X. A1=VCC, A0=N/C (internal pull-up) -> linux _110'1011 is 0x6b
* U23 voltmeter at 10011 ADR1 ADR0. ADR0=GND, ADR1=VCC -> linux _100'1110 is 0x4e
* SFP, address 1010000X -> linux _101'0000 is 0x50. additional SFP data at 0x51
* QSFP0, QSFP1, QSFP2, QSFP3 (QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL)

= I2C clock builder connection =

* use Silicon Labs USB "Clock builder pro field programmer", www.silabs.com/CBProgrammer
* connect rainbow jumper cable pins:
** black - 1-GND to GND on DS-DM
** white - 3-SCLK to SCL on the DS-DM
** grey - 7-SDA_SDIO to SDA on the DS-DM
* power up the DS-DM
* plug USB programmer into Windows laptop
* on Windows, run "ClockBuilder Pro"
* it should report "Field programmer detected", press "EVB GUI"
* in EVB GUI, press "Config", set I2C address 0x6B
* press "Scan", it should find Si5394A-A-GM
* select the "Status" tab, should see real-time status of clock chip

= GDM MGT configuration =

* TX configuration:
* GDM MGT transceivers are configured as "multilane" TX and RX.
* there is 12 TXes ("lanes")
* MGT reference 125 MHz clock goes into [2:0]gtreclk00_in and [11:0]gtrefclk0_in
* one MGT is designated as "master"
* PLL of master MGT converts reference clock into common TX clock and common tx_user_clk2 which becomes tx_data_clk
* common tx_user_clk2 aka tx_data_clk goes into all TXes and clocks tx_data.
* tx_user_clk2 aka tx_data_clk is 125 MHz but not same phase as MGT reference clock.

* RX interim configuration:
* there is 12 RXes ("lanes")
* each RX produces it's own recovered RX clock
* "multilane" configuration assumes all RX recovered clocks run at the same frequency (TX on the other end are driven by common TX clock, see above), but have different phase
* one RX recovered clock is designated as "master" (rx_user_clk2 aka rx_data_clk) and a phase-matching fifo/buffer is used to bring rx_data from all 12 RXes to this common rx_data_clk
* this works because each CDM SFP TX runs on the SFP RX recovered clock which is frequency-locked with the GDM QSFP TX clock.

* RX final configuration:
* MGTs permit using the common TX clock (tx_user_clk2 aka tx_data_clk) as the common rx_data_clk (they are frequency locked through the CDM).
* this permits use of tx_data_clk as the main clock domain in the GDM and removes the need to bring rx_data into the tx_data_clk domain (actually this is done in the MGT RX phase matching fifo/buffer).

= Clock path =

<pre>
10 MHz ext clock or GDM 125 MHz oscillator
-> GDM QSFP MGT reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz (GDM main clock domain) and TX bit clock 2.5 GHz
-> GDM QSFP optic transmitter
-> CDM SFP optic receiver
-> CDM SFP MGT, RX reference clock is CDM 125 MHz oscillator
-> MGT RX recovered clock 125 MHz (CDM main clock domain)
-> CC_CLK_IN -> CDM C.C. -> CC_CLK_OUT1 -> CDM SFP TX reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz and TX bit clock 2.5 GHz
-> (tx_data phase matching fifo from CDM main clock domain to tx_data_clk)
-> CDM SFP optic transmitter
-> GDM QSFP RX optic receiver (12x)
-> GDM QSFP MGT (RX reference clock is same as TX reference clock)
-> MGT RX recovered clock (12x recovered clocks)
-> in multi-lane configuration, one of them is the "master" recovered clock rx_data_clk
-> (rx_data phase matching fifo from rx_data_clk to GDM main clock domain)
</pre>

= Clock domains =

== GPS ==

* no GPS : GDM runs from internal 125 MHz oscillator
* external 10 MHz clock : GDM runs from external 10 MHz clock and optional 1pps signal (use VME-NIMIO32 NIM outputs)
* GPS receiver : GDM runs from GPS 10 MHz clock and GPS IRIG serial data
* LNGS GPS:
** provides 1pps and serial data over fiber from GPS receiver (master). LNGS xxx box is not used.
** serial data goes to GDM, decoded, 1pps signal extracted, goes to LEMO output
** 1pps from GDM is used to train the Rubidium clock which provides a 10 MHz clock
** 10 MHz output from Rubidium clock goes to GDM 10 MHz external clock input

== GDM ==

* AXI clock (100 MHz) - AXI registers
* 125 MHz oscillator - to clock cleaner
* 10 MHz external clock LEMO input - to clock cleaner
* FPGA 125 MHz clock CLK_CC_IN - to clock cleaner - not used (could be used for cascaded GDMs)
* 125 MHz oscillator - CLK3_XO_125 - mgt_b_ref_clk QSFP MGT reference clock (interim GDM design)
* clock cleaner output 125 MHz fanout:
** CLK_CC_OUT0 - QSFP MGT reference clock (final design)
** CLK_CC_OUT1 - not used (CDM SFP reference clock)
** CLK_CC_OUT2 - not used
* QSFP MGT TX data clock 125 MHz tx_data_clk (main clock domain)
** QSFP TX data
** QSFP RX data (in final design, rx_data_clk is same as tx_data_clk. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk == tx_data_clk)
** ds20k block
* (in interim design, rx_data_clk is disconnected from tx_data_clk. rx_data_clk is the "multilane master clock" which is one of the 12x RX recovered clocks. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk)

note: all these clocks are frequency locked to 125 MHz

== CDM ==

* AXI clock (100 MHz) - AXI registers
* 10 MHz external clock LEMO input - to clock cleaner (not used)
* 125 MHz oscillator to fanout
** to clock cleaner
** to SFP MGT RX reference clock: CLK3_XO_125 to mgt_rx_ref_clk to gtrefclk01_in
* SFP MGT RX recovered clock 125 MHz
** MGT PLL to MGT rx_user_clk2 aka rx_data_clk
** MGT rxrecclkout_out to mgt_rx_rec_clk to CLK_CC_IN to C.C.
* SFP MGT rx_user_clk2 aka rx_data_clk (250 MHz/8 bit, 125 MHz/16 bit, 62.5 MHz/32 bit data) (main clock domain)
** SFP RX data
** ds20k block
** VX TX clock PLLs
** VX RX clock PLLs
* C.C. fan out
** 62.5 MHz VX clocks (12x)
** CLK_CC_OUT0 (not used, GDM QSFP MGT reference clock)
** CLK_CC_OUT1 125 MHz to mgt_tx_ref_clk to gtrefclk00_in to SFP MGT TX reference clock
** CLK_CC_OUT2 (not used)
* SFP MGT tx_user_clk2 aka tx_data_clk
** SFP TX data
** TX data phase matching fifo from main clock domain to tx_data_clk
* VX TX clock PLLs
** 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
** VX TX data phase matching from main clock domain to VX TX clock (12 total)
** VX TX serializer
** VX TX LVDS transmitter
* VX RX clock PLLs
** 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
** VX RX LVDS receivers (12 total)
** VX RX deserializers (12 total)
** VX TX data phase matching from VX RX clock to main clock domain

== VX ==

* everything runs on the VX main 125 MHz clock
* correct phase of VX to CDM LVDS data is adjusted by scan of CDM VX RX PLL clock phase (VX to CDM link is now established)
* correct phase of CDM to VX LVDS data is adjusted by scan of CDM VX RX PLL clock phase (link is established after VX to CDM idle data pattern changes from "VX RX data bad" to "good".
* after good phases are found by scan, they are not expected to change unless cables are changed, CDM and VX modules are changed or CDM firmware is rebuilt. (rebuild of VX firmware should not affect LVDS data phase).
* if there is excessive link errors, phase scan must be repeated.

= Board test plan =

To test:
* Enclustra FPGA board
* SFP port - SFP_RS0/RS1 connected to QSFP0_SEL/1_SEL is wrong? our Finisar SFP says RS0, RS1 N/C, so probably okey.
* SW5 CLK_EXT1 NIM works. TTL needs to be tested. flipping SW5 CLK_EXT0 side from NIM to TTL makes CLK_EXT1 go LOS and OOF in the clock chip. R23 and R53 should be removed?
* BOOT_MODE 0 and 1

Partial:
* U23 3.3V current meter and thermometer. V1,V2 is current monitor, same as in application note. V3,V4 is thermometer, same in application note, except capacitor C118 is 0.1u instead of 470pF. Tested ok: Tint, VCC, V1, V2, V1-V2. Test failed: TR2 reads 50-something degC instead of same as Tint.

Done:
* LED_FP1A..D: tested ok. K.O. 15 sep 2022
* USB UART: tested ok. K.O. 15 sep 2022
* J4A, J4B, J5A, J5B LEMO inputs (NIM/TTL) EXT_IN_LV(1..4). TTL threshold 1.7V, NIM threshold -0.3V. 50 Ohm termination. TTL straight, NIM inverted.
* J6A, J6B LEMO clock inputs: 10 MHz TTL from chronobox works, 10 MHz NIM from IO32 works. 18-apr-2023
* J7A, J7B LEMO outputs EXT_OUT(1), EXT_OUT(2) (NIM/TTL) (tested 23nov2022, K.O.). TTL ???, NIM inverted.
** TTL out no 50 ohm termination: 0=0V, 1=5V, rise and fall time ~5 ns
** TTL out with 50 ohm termination: 0=0V, 1=2.5V, rise and fall time <2ns
** NIM out no 50 ohm termination: 0=+50mV, 1=-1.8V, rise and fall time ~3ns
** NIM out with 50 ohm termination: 0=0V, 1=-0.9V, rise and fall time <2ns
* ethernet MAC i2c chip (K.O. 20sep2022: can read: i2cdump 0 0x5b, nov2022 have u-boot driver)
* CDM VX ports 2x(CLK, 3 tx, 4 rx) tested using test_cdm.exe and LVDS loopback in VX firmware.
* SFP i2c tested KO 22jun2023
* QSFP i2c tested KO 22jun2023
* i2c testing complete 22jun2023
* QSFP rx,tx tested 26june2023. lane0,1,2 ok, lane3 tx not connected, laser is off. qsfp0,1,2,3 all lanes ok.
* SFP rx,tx tested 26june2023. rx and tx okey. LOS ok, mod_absent ok.

Failure:
* ethernet: does not connect to alliedtelesys switch. connects to my USB-eth adapter at 100 Mbit speed. uboot mii status reports connection speed oscillating between 1000, 100 and 10. K.O. 16-sep-2022
* SFP LOS and mod_absent are swapped (in the FPGA pin definitions?)
* QSFP slot numbering is wrong.

= Checklist for newly build boards =

* put new board on workbench
* check - vme connector present, vme extraction handles present
* check - standoff are removed from all thru-holes
* plug Enclustra module
* check - SW6 both switches are in the "PS" position
* connect micro-usb cable from linux PC
* connect ethernet from 1gige capable network switch
* connect power from lab power supply - GND, +5V and -12V nominal, +5.7V and -12.5V actual
* power up, +5V current 2.10-2.8A, -12V current 0.05A
* on linux PC, open a new terminal, run: minicom -D /dev/ttyACM0
* in minicom window, observe messages about Xilinx first stage boot loader, etc
* on the ethernet switch, observe network link is 1gige speed (not 10mbit, not 100mbit).
* if everything boots okey, there will be a login prompt, login as root, password root.
* busybox devmem 0x80010000 # read firmware version number, i.e. 0xEDAD0A77
* TBW - test LEMO inputs
* TBW - test LEMO outputs
* TBW - test VX connectors
* TBW - test SFP connector
* TBW - test QSFP connector

= Serial console =

* check that linux computer has correct udev rules to allow access to /dev/ttyACM devices, see https://daq00.triumf.ca/DaqWiki/index.php/SLinstall#Configure_USB_device_permissions and https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Configure_USB_device_permissions
* connect micro-USB cable to connector J-UCB, other end connect to linux computer
* observe /dev/ttyACM0 was created
* run "minicom -D /dev/ttyACM0" (default serial settings are ok, otherwise, 115200n8)
* should have gdm-cdm login
* username root, password root

= i2c =

<pre>
ZynqMP> i2c bus
Bus 0: i2c@ff020000
ZynqMP> i2c dev 0
Setting bus to 0
ZynqMP> i2c probe
Valid chip addresses: 33 4E 53 5B 6B 77
ZynqMP> i2c md 0x5b 0x98
0098: fc c2 3d 00 00 12 1a 6e 0a 90 85 04 94 10 08 50 ..=....n.......P
</pre>

<pre>
root@gdm-cdm:~# i2cdetect 0
Warning: Can't use SMBus Quick Write command, will skip some addresses
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0.
I will probe address range 0x03-0x77.
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f
00:
10:
20:
30: -- -- -- 33 -- -- -- --
40:
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60:
70:
root@gdm-cdm:~#
root@gdm-cdm:~# i2cdump 0 0x5b
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0, address 0x5b, mode byte
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00 ???????P?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e ........??=..??n
a0: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00 ???????P?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e ........??=..??n
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@gdm-cdm:~#
</pre>

<pre>
root@gdm0:~# i2cdetect -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#
</pre>

<pre>
root@cdm0:~# i2cdetect -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: 50 51 -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@cdm0:~#
</pre>

* 0x33 - XU8 secure EEPROM (should be at 0x32)
* 0x4e - U23 current and temperature monitor
* 0x50, 0x51 - SFP
* 0x50 - QSFP, 4 QSFP modules enabled by GPIO QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL
* 0x53, 0x5b - ethernet mac eeprom
* 0x6b - U6 clock chip

== U23 ==

NOTE: Text reads double of Tint. not sure why. K.O. 21-mar-2024.

* internal temperature only

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x00 b # control register: "repeat mode, internal temperature only"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x03 # "Tint ready" and "busy", "busy is always 1 in repeat mode"
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 00 03 03 81 db 2a ce 2a 8e 00 6f 00 45 20 3f ?.????*?*?.o.E ?
10: 01 00 01 01 01 db 2a ce 2a 8e 00 6f 00 45 20 3f ?.????*?*?.o.E ?
...
readback:
reg0 - 03 - Tint ready
reg1 - 00 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB 0x81, bit 0x80 is "DV, data valid", bit 0x40 is "SS, sensor short", 0x20 is "SO, sensor open"
reg5 - Tint LSB 0xDB, Tint = 0x01DB = 475 * 0.0625 degC = 29.6 degC
reg6..F - stale data
</pre>

* Tint, V1, V2, TR2, VCC

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x18 b # control register: "repeat mode, V1, V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 7f 18 7f 7f 81 b9 aa d0 aa 8f 83 5e 83 5e a0 41 ???????????^?^?A
10: 01 18 01 01 01 b9 2a d0 2a 8f 03 5e 03 5e 20 41 ??????*?*??^?^ A
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x1b9*0.0625 = 27.5 degC
reg6 - V1 MSB 0xaa, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 LSB 0xd0, V1 = 0x2ad0*305.18/1000000 = 3.3447 V (correct)
reg8 - V2 MSB 0xaa, ditto
reg9 - V2 LSB 0x8f, V2 = 0x2a8f*305.18/1000000 = 3.3249 V (correct, smaller than V1)
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x5e. TR2 = 0x35e*0.0625 = 53.875 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x41, VCC = 2.5+0x2041*305.18/1000000 = 5.019 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.99 A (about right?)
</pre>

* Tint, V1-V2, TR2, VCC

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x19 b # control register: "repeat mode, V1-V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 7f 19 7f 7f 81 69 83 f2 83 f2 83 3a 83 3a a0 44 ?????i?????:?:?D
10: 01 19 01 01 01 69 03 f2 03 f2 03 3a 03 3a 20 44 ?????i?????:?: D
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x169*0.0625 = 22.5 degC
reg6 - V1 or V1-V2 MSB 0x83, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 or V1-V2 LSB 0xf2, V1-V2 = 0x3f2*19.42/1000000 = 0.0196142 V (correct, compare with V1 and V2 measured above)
reg8 - V2 or V1-V2 MSB
reg9 - V2 or V1-V2 LSB
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x3a. TR2 = 0x33a*0.0625 = 51.625 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x44, VCC = 2.5+0x2044*305.18/1000000 = 5.021 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.9807 A (about right?)
</pre>

== SFP ==

<pre>
root@cdm0:~# i2cdump 0 0x50
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 04 07 00 00 00 00 40 40 0c 00 01 3d 00 00 00 ???....@@?.?=...
10: 0c 02 00 1e 46 49 4e 49 53 41 52 20 43 4f 52 50 ??.?FINISAR CORP
20: 2e 20 20 20 00 00 90 65 46 54 4c 46 38 35 32 36 . ..?eFTLF8526
30: 50 33 42 4e 4c 20 20 20 41 20 20 20 03 52 00 9d P3BNL A ?R.?
40: 00 1a 00 00 4e 33 41 42 34 4c 56 20 20 20 20 20 .?..N3AB4LV
50: 20 20 20 20 32 30 30 33 31 39 20 20 68 f0 03 de 200319 h???
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@cdm0:~#
root@cdm0:~# i2cdump 0 0x51
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 5a 00 d8 00 55 00 e2 00 90 88 71 48 8c a0 75 30 Z.?.U.?.??qH??u0
10: 21 34 01 f4 1b 58 03 e8 31 2d 04 eb 1f 07 06 31 !4???X??1-?????1
20: 31 2d 00 64 27 10 00 9e 00 00 00 00 00 00 00 00 1-.d'?.?........
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
40: 00 00 00 00 3f 80 00 00 00 00 00 00 01 00 00 00 ....??......?...
50: 01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 b7 ?...?...?......?
60: 19 9f 80 c5 0e 17 12 c4 1f 99 00 00 00 00 30 00 ??????????....0.
70: 00 00 00 00 00 00 00 00 ff ff ff ff ff ff ff 01 ...............?
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@cdm0:~#
</pre>

== QSFP ==

<pre>
QSFP i2c enable lines, active low:

QSFP0_SEL - JENC-A 82 - PS-MIO40 - linux gpio 378
QSFP1_SEL - JENC-A 84 - PS-MIO41 - linux gpio 379
QSFP2_SEL - JENC-A 100 - PS-MIO44 - linux gpio 382
QSFP3_SEL - JENC-A 106 - PS-MIO43 - linux gpio 381 (notice 2 and 3 are out of order)
</pre>

<pre>
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
# echo 378 >> /sys/class/gpio/export ### SEL0 338+40
# echo 379 >> /sys/class/gpio/export ### SEL1 338+41
# echo 381 >> /sys/class/gpio/export ### SEL3 338+43
# echo 382 >> /sys/class/gpio/export ### SEL2 338+44
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) in hi
gpio-379 ( |sysfs ) in hi
gpio-381 ( |sysfs ) in hi
gpio-382 ( |sysfs ) in hi
root@gdm0:~# echo out >> /sys/class/gpio/gpio381/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio382/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio378/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio379/direction
root@gdm0:~#
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out lo
gpio-379 ( |sysfs ) out lo
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out lo
root@gdm0:~#
echo 1 >> /sys/class/gpio/gpio381/value
echo 1 >> /sys/class/gpio/gpio382/value
echo 1 >> /sys/class/gpio/gpio378/value
echo 1 >> /sys/class/gpio/gpio379/value
cat /sys/kernel/debug/gpio
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out hi
gpio-382 ( |sysfs ) out hi
root@gdm0:~#
root@gdm0:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#

NOTICE NOTHING AT ADDRESS 0x50
</pre>

<pre>
ENABLE QSFP0, OBSERVE IT IS AT ADDRESS 0x50

root@gdm0:~# echo 0 >> /sys/class/gpio/gpio378/value
root@gdm0:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: 50 -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#
root@gdm0:~# i2cdump -y 0 0x50
No size specified (using byte-data access)
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 0d 00 02 8e 00 00 01 00 00 05 55 00 05 00 00 00 ?.??..?..?U.?...
10: 00 00 00 00 00 00 1e b8 00 00 81 a0 00 00 00 00 ......??..??....
20: 00 00 12 12 00 01 00 01 00 01 0e e0 0f 20 0e e0 ..??.?.?.???? ??
30: 00 00 1f a0 1e 3b 1f 72 00 01 00 00 00 00 00 00 ..???;?r.?......
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
60: 00 00 00 00 00 00 00 00 00 00 1f 00 00 00 08 00 ..........?...?.
70: 00 00 00 00 00 00 00 ff ff ff ff ff ff ff ff 00 ................
80: 0d 00 0c 04 00 00 00 40 40 02 d5 05 67 00 00 96 ?.??...@@???g..?
90: 00 00 c8 00 46 49 4e 49 53 41 52 20 43 4f 52 50 ..?.FINISAR CORP
a0: 20 20 20 20 07 00 90 65 46 54 4c 34 31 30 51 44 ?.?eFTL410QD
b0: 34 43 20 20 20 20 20 20 41 20 42 68 07 d0 00 43 4C A Bh??.C
c0: 00 07 0f de 58 37 39 41 43 30 52 20 20 20 20 20 .???X79AC0R
d0: 20 20 20 20 32 32 30 33 30 39 20 20 3c 10 00 9e 220309 <?.?
e0: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@gdm0:~#
</pre>

== ethernet mac eeprom ==

* correct chip with 84-bit ethernet mac address

<pre>
root@cdm1:~# i2cdump 0 0x53
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
root@cdm1:~#

root@cdm1:~# i2cdump 0 0x5b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00 ????4??2?K?.?...
90: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c ..........??=?Q<
a0: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00 ????4??2?K?.?...
b0: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c ..........??=?Q<
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@cdm1:~#
</pre>

* wrong "602" chip with 64-bit IPv6 address

<pre>
root@cdm0:~# i2cdump 0 0x53
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
root@cdm0:~# i2cdump 0 0x5b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00 ???????Q?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e ........??=..??.
a0: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00 ???????Q?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e ........??=..??.
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@cdm0:~#
</pre>

= read ethernet mac address from i2c =

(this code is copied from uboot command line i2c code)

in uboot sources board/xilinx/common/board.c replace original function with this:

<pre>
// special code to read ethernet MAC address from the DS-DM-Rev0 board. K.O. Sep-2022

int zynq_board_read_rom_ethaddr(unsigned char *ethaddr)
{
struct udevice *bus;
int ret;
int busnum = 0;

ret = uclass_get_device_by_seq(UCLASS_I2C, busnum, &bus);
if (ret) {
printf("%s: No bus %d\n", __func__, busnum);
return ret;
}

int chip_addr = 0x5B;

struct udevice *dev;

ret = i2c_get_chip(bus, chip_addr, 1, &dev);
if (ret) {
printf("%s: Bus %d no chip 0x%02x\n", __func__, busnum, chip_addr);
return ret;
}

int dev_addr = 0x98;

unsigned char data[8];

ret = dm_i2c_read(dev, dev_addr, data, 8);

if (ret) {
printf("%s: Bus %d chip 0x%02x read error %d\n", __func__, busnum, chip_addr, ret);
return ret;
}

printf("%s: Bus %d chip 0x%02x addr 0x%02x read: 0x %02x %02x %02x %02x %02x %02x %02x %02x\n", __func__, busnum, chip_addr, dev_addr, data[0], data[1], data[2], data[3], data[4], data[5], data[6], data[7]);

// see Atmel-8807-SEEPROM-AT24MAC402-602-Datasheet.pdf

if (data[0] == 0) {
// eiu-48 chip
ethaddr[0] = data[2];
ethaddr[1] = data[3];
ethaddr[2] = data[4];
ethaddr[3] = data[5];
ethaddr[4] = data[6];
ethaddr[5] = data[7];
} else {
// eiu-64 chip
ethaddr[0] = data[0];
ethaddr[1] = data[1];
ethaddr[2] = data[2];
ethaddr[3] = data[5];
ethaddr[4] = data[6];
ethaddr[5] = data[7];
}

printf("%s: ethaddr %02x:%02x:%02x:%02x:%02x:%02x\n", __func__, ethaddr[0], ethaddr[1], ethaddr[2], ethaddr[3], ethaddr[4], ethaddr[5]);

return ret;
}
</pre>

also this should have worked if i2c_xxx() functions were enabled in uboot:
<pre>
i2c_set_bus_num(0);
i2c_probe(0x5b);
i2c_read(0x5b, 0x9a, ethaddr, 6);
</pre>

= read ethernet mac address from i2c (SHOULD WORK) =

from: https://stackoverflow.com/questions/43637540/is-there-linux-or-u-boot-support-to-read-a-mac-address-from-a-chip-at-startup

<pre>
#ethernet related setup
setup_eth=run readmac buildmac
#read mac address from eeprom
readmac=i2c dev 0; i2c read 50 FA.1 6 $loadaddr
#build the ethaddr variable
#not very nice, but does the job
buildmac=\
e=" "; sep=" " \
for i in 0 1 2 3 4 5 ; do\
setexpr x $loadaddr + $i\
setexpr.b b *$x\
e="$e$sep$b"\
sep=":"\
done &&\
setenv ethaddr $e
</pre>

= read ethernet mac address from i2c (DOES NOT WORK) =

this method does not work: inside board/xilinx/common/board.c:int zynq_board_read_rom_ethaddr(unsigned char *ethaddr), uclass_get_device_by_ofnode(UCLASS_I2C_EEPROM, eeprom, &dev) returns ENODEV and read from i2c does not happen. K.O. Sep-2022

Read:
* https://support.xilinx.com/s/question/0D52E00006hpKlsSAE/uboot-and-ethernet-mac-address-from-eeprom?language=en_US (last answer)
* https://github.com/Xilinx/u-boot-xlnx/blob/master/board/xilinx/common/board.c (zynq_board_read_rom_ethaddr())

Note:
* 0x5B is the i2c chip address
* 0x9A is the data offset inside the chip, see datasheet or i2c read dump above.

Edit:
* emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/u-boot/files/platform-top.h
<pre>
#include <configs/xilinx_zynqmp.h>
#include <configs/platform-auto.h>
//#define CONFIG_I2C_EEPROM
//#define CONFIG_SYS_I2C_EEPROM_ADDR 0x5b
//#define CONFIG_SYS_I2C_EEPROM_ADDR_OVERFLOW 0x0
#define CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET 0x9A
#error HERE!
</pre>
* emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi
<pre>
/include/ "system-conf.dtsi"
/ {
chosen {
xlnx,eeprom = &eeprom;
};
};

&i2c0 {
eeprom: eeprom@5b { /* u88 */
compatible = "atmel,24mac402";
reg = <0x5b>;
};
};
</pre>
* components/yocto/workspace/sources/u-boot-xlnx/configs/xilinx_zynqmp_virt_defconfig
<pre>
CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET=0x9A
</pre>

= enable VX clock =

<pre>
devmem 0x80010010 32 0x8; sleep 1; devmem 0x80010010 32 0x0;
si5394-i2c-file /media/sd-mmcblk1p1/00_freerun.txt 0 0x6b
</pre>

= clock chip configuration =

file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt:

* VCO is 14 GHz
* Tvco is 71.43 ps
* N0 divider is 14, frequency is 1000 MHz
* out0 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz
* out1 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
* out2 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
* out3 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz is the feedback for zero delay

= clock chip monitoring =

from si5395-94-92-family.pdf:
* reg 0x1: page select, set to 0 or set to 5 to read 0x53F
* reg 0x2: 0x94
* reg 0x3: 0x53 -> device is a si5394
* reg 0xC: LOSXAXB
* reg 0xD: LOS and OOF for the 4 clock inputs
* reg 0xE: LOL and HOLD
* reg 0xF: CAL_PLL
* reg 0x11: sticky bits for reg 0xC
* reg 0x12: sticky bits for reg 0xD
* reg 0x13: sticky bits for reg 0xE
* reg 0x14: sticky bits for reg 0xF
* reg 0x1C: device reset
* reg 0x1E: low power, hard reset, SYNC
* reg 0x507: currently selected input clock
* reg 0x52A: input clock select
* reg 0x535: FORCE_HOLD
* reg 0x53F: HOLD_HIST_VALID and FASTLOCK_STATUS

= arm and linux benchmark =

memory benchmark:

<pre>
daq13$ arm-linux-gnueabi-gcc -o memcpy.armv7 memcpy.cc -march=armv7 -static -O2
scp memcpy.armv7 to ...
root@gdm-cdm:~# ./memcpy.armv7
memcpy 1 KiBytes: 1288 MB/sec
memcpy 2 KiBytes: 1924 MB/sec
memcpy 4 KiBytes: 2554 MB/sec
memcpy 8 KiBytes: 3054 MB/sec
memcpy 16 KiBytes: 3262 MB/sec
memcpy 32 KiBytes: 3250 MB/sec
memcpy 64 KiBytes: 3456 MB/sec
memcpy 128 KiBytes: 3556 MB/sec
memcpy 256 KiBytes: 3780 MB/sec
memcpy 512 KiBytes: 3795 MB/sec
memcpy 1024 KiBytes: 3789 MB/sec
memcpy 2048 KiBytes: 3729 MB/sec
memcpy 4096 KiBytes: 3717 MB/sec
memcpy 8192 KiBytes: 3687 MB/sec
memcpy 16384 KiBytes: 3632 MB/sec
memcpy 32768 KiBytes: 3529 MB/sec
memcpy 65536 KiBytes: 3318 MB/sec
memcpy 131072 KiBytes: 2893 MB/sec
root@gdm-cdm:~#
</pre>

ethernet receive:

<pre>
daq13:bin$ ./ttcp -t -s -n 100000 10.0.0.24
ttcp-t: buflen=8192, nbuf=100000, align=16384/0, port=5001 tcp -> 10.0.0.24
ttcp-t: socket
ttcp-t: connect
ttcp-t: 819200000 bytes in 7.25 real seconds = 110358.39 KB/sec +++
ttcp-t: 100000 I/O calls, msec/call = 0.07, calls/sec = 13794.80
ttcp-t: 0.0user 0.2sys 0:07real 3% 0i+0d 760maxrss 0+2pf 1461+31csw
daq13:bin$

root@gdm-cdm:~# ./ttcp.armv7 -r -s
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 10.0.0.25
ttcp-r: 819200000 bytes in 7.27 real seconds = 110098.22 KB/sec +++
ttcp-r: 212040 I/O calls, msec/call = 0.04, calls/sec = 29181.53
ttcp-r: 0.1user 5.7sys 0:07real 81% 0i+0d 584maxrss 0+2pf 125601+2699csw
root@gdm-cdm:~#
</pre>

ethernet transmit:

<pre>
root@gdm-cdm:~# ./ttcp.armv7 -t -s -n 100000 10.0.0.25
ttcp-t: buflen=8192, nbuf=100000, align=16384/0, port=5001 tcp -> 10.0.0.25
ttcp-t: socket
ttcp-t: connect
ttcp-t: 819200000 bytes in 6.95 real seconds = 115078.69 KB/sec +++
ttcp-t: 100000 I/O calls, msec/call = 0.07, calls/sec = 14384.84
ttcp-t: 0.0user 0.7sys 0:06real 11% 0i+0d 584maxrss 0+2pf 1162+1017csw
root@gdm-cdm:~#

daq13:bin$ ./ttcp -r -s
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 10.0.0.24
ttcp-r: 819200000 bytes in 6.97 real seconds = 114841.84 KB/sec +++
ttcp-r: 161335 I/O calls, msec/call = 0.04, calls/sec = 23160.01
ttcp-r: 0.0user 1.9sys 0:06real 28% 0i+0d 760maxrss 0+2pf 80646+51csw
daq13:bin$
</pre>

= Install Xilinx tools =

* install Vivado 2020.2
<pre>
login at https://www.xilinx.com/myprofile.html
go to "Downloads"
go to archive,
find 2020.2
download Xilinx_Unified_2020.2_1118_1232_Lin64.bin
sh ./Xilinx_Unified_2020.2_1118_1232_Lin64.bin
banner window should open with spinner "downloading installation data"
"a newer version is available" -> say "continue"
next
"select install type" window:
provide email and password,
select "download image"
select directory /home/olchansk/Xilinx/Downloads/2020.2\
select "linux" and "full image"
next
download summary: space required 38.52 Gbytes
download
installation progress
downloading spinner, 16 M/s 47 minutes...
"download image has been created successfully". Ok.
check contents of /home/olchansk/Xilinx/Downloads/2020.2
ls -l /home/olchansk/Xilinx/Downloads/2020.2
total 67
drwxr-xr-x 2 olchansk users 9 Sep 1 16:22 bin
drwxr-xr-x 3 olchansk users 15 Sep 1 16:23 data
drwxr-xr-x 4 olchansk users 4 Sep 1 16:22 lib
drwxr-xr-x 2 olchansk users 644 Sep 1 16:22 payload
drwxr-xr-x 2 olchansk users 7 Sep 1 16:22 scripts
drwxr-xr-x 4 olchansk users 4 Sep 1 16:22 tps
-rwxr-xr-x 1 olchansk users 3256 Nov 18 2020 xsetup
daq13:2020.2$
./xsetup
spinned loading installation data
xilinx design tools 2022.1 now available -> say continue
"welcome" -> next
"select product" -> vivado -> next -> vivado hl system edition -> next
select devices: only zynq ultrascale+ mpsoc -> next
select destination: /opt/Xilinx (as root, mkdir /opt/Xilinx, chmod olchansk.users /opt/Xilinx)
install ...
complete
move /home/olchansk/Xilinx/Downloads/2020.2 to /daq/daqstore/olchansk/Xilinx/Downloads/
</pre>
* install petalinux 2020.2
<pre>
./xsetup
"a newer version is available" -> say "continue"
next
"select product to install" -> select Petalinux (Linux only) -> next
"select destination directory" -> select "/opt/Xilinx" (disk space required 2.64 GB) -> next
"summary" -> install ...
error about missing /tmp/tmp-something files
"installation completed successfully" (hard to dismiss, "ok" button is partially cut-off)
done?
I think it failed, /opt/Xilinx/PetaLinux/2020.2/bin is empty except for petalinux-v2020.2-final-installer.run
try to run it by hand, same error about /tmp/tmp-something files. strange...
notice it complains about "truncate", which truncate finds ~/bin/truncate, get rid of it,
try again
now complains about missing texinfo and zlib1g:i386
apt install texinfo -> ok
apt install zlib1g:i386 -> installs bunch of gcc stuff -> ok
try again
reports "already installed" -> delete /opt/Xilinx/.xinstall/PetaLinux_2020.2/, delete entries in ~/.Xilinx/registry/installedSW.xml
try again
success
</pre>
* install vivado 2022.1 and petalinux 2022.1 - everything is pretty much the same

= Petalinux =

* cd PetaLinux_GDM_CDM
* petalinux-config
* enable i2c MAC address and DHCP
<pre>
git clone https://bitbucket.org:/team-ds-dm/ds-dm-u-boot-xlnx.git
cd ds-dm-u-boot-xlnx
git checkout ds-dm-u-boot-xlnx

linux-components ->
uboot -> ext-local-src
external u-boot local source -> ds-dm-u-boot-xlnx (path to the customized uboot git repository)
</pre>
* enable DHCP
<pre>
Subsystem AUTO Hardware Settings -> Ethernet Settings
randomize MAC address -> NO
ethernet mac address -> leave empty
obtain ip address automatically -> YES
</pre>
* set hostname and product names
<pre>
Firmware Version Configuration ->
Host name -> "ds-dm"
Product name -> "Petalinux_GDM_CDM"
</pre>
* configure linux kernel
<pre>
petalinux-config -c kernel
</pre>
* enable NFS-Root
<pre>
petalinux-config
Image Packaging Configuration > Root File System Type -> set to NFS
Location of NFS root directory set to "/nfsroot"

petalinux-config -c kernel
Networking support > IP: kernel level configuration
enable DHCP, BOOTP, RARP
File systems > Network file systems > Root file systems on NFS
</pre>
* manually fix linux kernel command line:
<pre>
grep nfsroot PetaLinux_GDM_CDM/project-spec/configs/config
edit CONFIG_SUBSYSTEM_BOOTARGS_GENERATED to read
earlycon console=ttyPS0,115200 clk_ignore_unused panic=60 root=/dev/nfs nfsroot=/nfsroot/%s ip=dhcp rw
</pre>
* check configuration in
** PetaLinux_GDM_CDM/project-spec/configs/config
** PetaLinux_GDM_CDM/project-spec/configs/rootfs_config
** PetaLinux_GDM_CDM/components/plnx_workspace/device-tree/device-tree/system-conf.dtsi

= JTAG server =

localhost:3121

= ds20k block =

<pre>
module ds20k
(
// CLOCK INPUTs
input wire clk,
input reset, // pulse for power-up reset
input wire pll_is_locked, // clock cleaner PLL is locked to selected input clock

// REGISTER_DATA

input wire [255:0] [31:0] register_data_in,
output reg [255:0] [31:0] register_data_out,
input wire register_write_strobe, // pulse when AXI write transaction puts new data in register_data_in
input wire register_read_ack, // pulse after AXI read transaction captures data from register_data_out, used to read from FIFO

// GDM QSFP FIBER LINKS
output reg [11:0] [15:0] qsfp_tx_data,
output reg [11:0] [1:0] qsfp_tx_ctrl,
input wire [11:0] [15:0] qsfp_rx_data_error, // not sure what this is.
input wire [11:0] [15:0] qsfp_rx_data,
input wire [11:0] [1:0] qsfp_rx_ctrl,
input wire [11:0] qsfp_rx_is_good, // single bit indicating that RX link is up and data is good.

// CDM SFP FIBER LINKS
output reg [15:0] sfp_tx_data,
output reg [1:0] sfp_tx_ctrl,
input wire [15:0] sfp_rx_data,
input wire [1:0] sfp_rx_ctrl,
input wire sfp_rx_is_good, // single bit indicating that RX link is up and data is good.

// VX_RXs
input wire [3:0] vx1_rx,
input wire [3:0] vx2_rx,
input wire [3:0] vx3_rx,
input wire [3:0] vx4_rx,
input wire [3:0] vx5_rx,
input wire [3:0] vx6_rx,
input wire [3:0] vx7_rx,
input wire [3:0] vx8_rx,
input wire [3:0] vx9_rx,
input wire [3:0] vx10_rx,
input wire [3:0] vx11_rx,
input wire [3:0] vx12_rx,

// VX_TXs
output reg [2:0] vx1_tx_out,
output reg [2:0] vx2_tx_out,
output reg [2:0] vx3_tx_out,
output reg [2:0] vx4_tx_out,
output reg [2:0] vx5_tx_out,
output reg [2:0] vx6_tx_out,
output reg [2:0] vx7_tx_out,
output reg [2:0] vx8_tx_out,
output reg [2:0] vx9_tx_out,
output reg [2:0] vx10_tx_out,
output reg [2:0] vx11_tx_out,
output reg [2:0] vx12_tx_out,

// remove input wire gdm_trg,
// remove input wire gdm_tsm,

// LEMO INPUTs
input wire [4:1] ext_in_lv_async, // direct connection to LEMO connectors, not clocked

// LEMO OUTPUTs
output reg [2:1] ext_out, // direct connection to LEMO connectors, not clocked

// FRONT PANEL LEDs
output reg [3:0] fp_led_out, // direct connection to LEMO connectors, not clocked

// trigger and tsm output
// remove output reg trg_out,
// remove output reg tsm_out
);
</pre>

= world view =

Note:
* red lines: clocks
* green lines: AXI/Avalon packet streams
* blue lines: serial data

[[File:GDM-CDM-VX.drawio.svg]]

== description ==

same thing, in words:

<pre>
detector
digitizer, 125 MHz
digital filter
digital discriminator
hit map, 64 bits at 125 MHz (could be 250 MHz, filter and discriminator clock)
packetizer, 64 bits -> id, timestamp, 8x 8-bit words, eop
8/10 serializer, 12.5 MHz parallel in, 125 MHz serial out
lvds line to CDM vx_rx
BBB: also VX busy to lvds line to CDM vx_rx, do not want to depend on serial comm for vx busy, do not want to inject async data into the hit map packet stream
--- CDM
lvds line capture in IBUF register, vxN_rx clock, 125 MHz clock, from PLL with adjustable phase, scan phase to find sweet spot
vx rx 10/8 deserializer, vxN_rx clock, stobes out parallel data every 10 clocks at 12.5 MHz
vx rx phase transfer from vxN_rx clock to main CDM clock, have 10 clocks for transfer to happen
from here everything is on the CDM main clock
vx rx packet adapter, 12.5 MHz 8-bit packets to 125 MHz 16-bit packets, internal FIFO to avoid data overrun
sfp tx mux, all VX packet streams into one CDM sfp tx packet stream
BBB: capture VX RX busy to CDM main clock (IBUF register), grand-or becomes vx_rx_busy
sfp tx packetizer (data,eop,vx_rx_busy -> data,k)
sfp tx 16/20 serializer, 16 bit at 125 MHz in, 2500 MHz serial out (20 bits at 125 MHz)
sfp tx fiber link to GDM, 2 Gigabits/sec
--- GDM
qsfp rx deserializer, 16 bit at 125 MHz output on main GDM clock
qsfp rx depacketizer (data,k -> packet data,eop; qsfp_rx_busy)
qsfp rx demux, hit map packets routed to GDM trigger logic block
GDM trigger logic block looks at hit map, generates yes/no trigger decision, encodes it as a trigger packet
TSM generator encodes GPS time data as a TSM packet
qsfp tx mux - trigger packets, tsm packets, etc to qsfp tx packet stream
qsfp tx packetizer (data,eop;trg,tsm,bsy -> data,k)
NB: the same tx data is sent to all 12 qsfp tx ports, to make sure we do not accidentally desync the CDMs.
qsfp tx 16/20 serializer, 16-bit at 125 MHz to 20-bit at 250 MHz to 2500 MHz serial out
qsfp tx fiber link to CDM, 2 Gigabits/sec
--- CDM
sfp rx 20/16 deserializer, 16-bit at 125 MHz
sfp rx depacketizer, (data,k -> data,eop;trg_in,tsm_in,bsy_in)
sfp rx demux (in reality, noop, all packets go to same place, vx tx)
vx tx mux (packets from GDM, packets with simulated waveforms from CDM midas frontend via AXI FIFO)
vx tx packet adapter 16-bit at 125 MHz to 8-bit at 12.5 MHz strobed every 10 clocks. NB: most important, this 10-clock strobe runs in sync between all CDMs!
vx tx phase transfer from main CDM clock to vxN_tx clock, have 10 clocks for transfer to happen
from here we run on the vxN_tx clock
vx txN serializer, 8-bit at 12.5 MHz to 125 MHz serial
vx_txN OBUF register, vxN_rx clock, 125 MHz clock, from PLL with adjustable phase, scan phase to find sweet spot
lvds line to vx
BBB: bsy_in from GDM is converted from pulse to level, goes out lvds line to vx, sync to vxN_tx clock
TTT: trg_in from GDM is a pulse, does out lvds line to vx, sync to vxN_tx clock.
NB: tsm is always a packet, bsy is always a signal (no packet), trg can be a packet or signal.
--- VX
lvds data captured by 125 MHz ADC sampling clock (CAEN base firmware logic)
lvds data connected to Yair's block
10/8 deserializer
depacketizer (data,k -> data,eop) to avoid accidental desync, we do not send any no trg, no tsm, no bsy K-codes.
demux
trigger packets go their way (4x 64 bit words of data go to event header: timestamps, hitmap data)
tsm packets go their way (4x 64 bit words of data go to tsm event header: timestamps, GPS time data)
BBB: bsy from lvds line stops waveform acquisition
TTT: trg from lvds line cause waveform acquisition, same as trg packet, but has no timestamp and other data attached to it.
</pre>

= Firmware registers =

== Block 0 ==

<pre>
0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
</pre>

== Block 1 ==

== Block 2 ==

== Block 3 ==

=== DS20k block register map ===

* busybox devmem 0x80013000 32

<pre>
reg | version | xDM | xx | description
0 | 0x20230731 | ALL | RO | ds20k version
0 | 0x20240118 | ALL | RW | ds20k version and command
1 | 0x20230731 | ALL | RW | scratch read/write register
2 | 0x20230731 | ALL | RW | configure inputs and outputs
3 | 0x20230731 | ALL | RW | FP_LED mux
4 | 0x20230731 | ALL | RW | EXT_OUT mux
5 | 0x20230731 | ALL | RO | VX_RX state
6 | 0x20230731 | ALL | RO | VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state
7 | 0x20230731 | ALL | RW | LED_OUT, EXT_OUT, VX_TX outputs
8 | 0x20230731 | ALL | RW | VX_TX mux and config
9 | 0x20230731 | ALL | RW | trigger config
10 | 0x20231013 | ALL | RO | status register
11 | 0x20230731 | ALL | RO | trigger counter
12 | 0x20230731 | ALL | RO | time slice marker counter
13 | 0x20240814 | ALL | RO | GPS 1pps period, 125 MHz
14 | 0x20240814 | ALL | RO | Ru clock 1pps period, 125 MHz
15 | 0x20230811 | CDM | RO | SFP RX status
16 | 0x20230811 | CDM | RW | SFP TX control
17 | 0x20230811 | GDM | RO | QSFP RX data 0, 1
18 | |. |. | 2, 3
19 | |. |. | 4, 5
20 | |. |. | 6, 7
21 | | |. | 8, 9
22 | |. |. | 10, 11
23 | 0x20230811 | GDM | RW | QSFP TX control
24 | 0x20231013 | ALL | RW | trigger pulser period
25 | 0x20231013 | ALL | RW | trigger pulser burst control
26 | 0x20231013 | ALL | RW | tsm pulser period
27 | 0x20231204 | ALL | RW | data fifo CPU to FPGA
28 | 0x20231204 | ALL | RW | packet loopback control
28 | 0x20240118 | ALL | RW | data fifo FPGA to CPU
29 | 0x20231208 | GDM | RW | bitmap of active qsfp ports
29 | 0x20240118 | ALL | RW | packet loopback control
29 | 0x20240510 | ALL | RW | packet routing
30 | 0x20231208 | GDM | RO | qsfp link status ports 0..7
31 | 0x20231208 | GDM | RO | qsfp link status ports 8..11
32 | 0x20240118 | GDM | RW | enabled QSFP ports
33 | 0x20240118 | CDM | RW | enabled VX ports
34,35 | 0x20240118 | ALL | RO | time stamp 64 bits
36,37 | 0x20240118 | ALL | RO | old time stamp
38,39,40 | 0x20240118 | CDM | RO | VX busy counters
41,42,43 | 0x20240118 | GDM | RO | QSFP busy counters
44 | 0x20240118 | CDM | RO | cdm_bsy_up_counter and cdm_bsy_pulse_counter
45 | 0x20240118 | CDM | RO | cdm_veto_pulse_counter and cdm_veto_up_counter
46 | 0x20240118 | GDM | RO | gdm_bsy_pulse_counter and gdm_bsy_up_counter
47 | 0x20240118 | GDM | RO | gdm_bsy_refresh_counter
48 | 0x20240118 | GDM | RO | gdm_veto_up_counter and gdm_veto_pulse_counter
49,50,51 | 0x20240424 | CDM | RO | VX RX serial data monitor, 8 bits per VX port
52,53 | 0x20240424 | CDM | RO | VX serial link status, 4 bits per VX port
54 | 0x20240424 | CDM | RO | VX TX serial data monitor
55 | 0x20240430 | ALL | RO | QSFP, SFP and VX link loss counters
56 | 0x20240510 | ALL | RO | sfp_rx_packet_counter
57 | same | ALL | RO | sfp_tx_packet_counter
58 | same | ALL | RO | qsfp_rx_packet_counter[0]
59 | same | ALL | RO | qsfp_tx_packet_counter
60 | same | ALL | RO | vx_rx_packet_counter[0]
61 | same | ALL | RO | vx_tx_packet_counter
62 | 0x20240719 | CDM | RO | cdm_hitmap_period, ports 0, 1
63 | same | CDM | RO | ports 2, 3
64 | same | CDM | RO | ports 4, 5
65 | same | CDM | RO | ports 6, 7
66 | same | CDM | RO | ports 8, 9
67 | same | CDM | RO | ports 10, 11
68 | 0x20240814 | ALL | RW | GPS control and status
</pre>

=== Register 0 0x80013000 ds20k version ===

on read: ds20k version 0xYYYYMMDD

on write:
* 0x00 - noop - as of version 0x20240118, write a zero after writing a command
* 0x01 - CMD_reset - reset logic to good state
* 0x02 - CMD_arm_ts - arm timestamp reset
* 0x03 - CMD_trg - issue a trigger
* 0x04 - CMD_tsm - issue a tsm
* 0x05 - CMD_vx_rx_reset - reset the VX receive path
* 0x06 - CMD_vx_tx_reset - reset the VX transmit path
* 0x07 - CMD_hitmap_trg - generate a hitmap trigger and data packet
* 0x08 - CMD_trg_pulser_reset - reset the trigger pulser
* 0x09 - CMD_tsm_pulser_reset - reset the tsm pulser

=== Register 1 0x80013004 scratch ===

scratch read-write register

=== Register 2 0x80013008 input and output config ===

<pre>
bit | version | fpga name | description
0 | ALL | lemo_enable | enable LEMO input 1
1 | | | 2
2 | | | 3
3 | | | 4
4 | ALL | lemo_invert | invert LEMO input 1
5 | | | 2
6 | | | 3
7 | | | 4
9 | ALL | ext_out_disable | disable LEMO output 1
10 | | | 2
11 | ALL | ext_out_invert | invert LEMO output 1
12 | | | 2
</pre>

=== Register 3 0x8001300C FP_LED control ===

<pre>
wire [15:0] led_out_mux_sel = register_data_in[3][15:0];
wire [3:0] led_out_invert = register_data_in[3][19:16];
</pre>

<pre>
led_out_mux_sel is 4 groups (one per LED) of 4 bits (choice 0..15):
mux | version | fpga name | description
0 | ALL | | power on default
1 | ALL | led_out_reg | register 7 bits
2 | 0x20231013 | pll_locked | clock chip PLL is locked
3 | 0x20231013 | sfp_link_status | SFP link is good
3 | 0x20240118 | sfp_link_status or qsfp_tx_link_rx_status | SFP/QSFP link is good
4 | ALL | lemo_in_sync[1] | LEMO input 1
5 | ALL | lemo_in_sync[2] | LEMO input 2
6 | ALL | lemo_in_sync[3] | LEMO input 3
7 | ALL | lemo_in_sync[4] | LEMO input 4
8 | ALL | ext_out[1] | LEMO output 1
9 | ALL | ext_out[2] | LEMO output 2
A | 0x20231013 | trg_in | trigger
B | 0x20231013 | tsm_in | time slice marker
C | 0x20240118 | gdm_bsy | GDM busy: OR of all CDM busy
D | 0x20240118 | cdm_bsy | CDM busy: OR of all VX busy
E | 0x20240118 | cdm_veto | GDM busy -> GDM veto -> CDM veto -> VX trigger veto
F | ALL | | fixed logic level 1
</pre>

=== Register 4 0x80013010 LEMO OUT control ===

<pre>
wire [7:0] ext_out_mux_sel = register_data_in[4][7:0];
</pre>

<pre>
ext_out_mux_sel is 2 groups (one per LEMO) of 4 bits (choice 0..15):

mux | version | fpga name | description
0 | ALL | | power on default
1 | ALL | ext_out_reg | register 7
2 | ALL | trg_pulser | pulser trigger
3 | 0x20240724 | vx1_tx_out[2] | vx1 serial data out
4 | 0x20240724 | vx1_rx[1] | vx1 serial data in
5 | 0x20240724 | vx_rx_iob[0] | vx1 serial data in captured by IOB register
6 | ALL | lemo_in_async[1]| test synchronizer
7 | ALL | lemo_in_sync[1] | test synchronizer
8 | 0x20240724 | trg_in_pulse | trigger signal
9 | 0x20240724 | tsm_in_pulse | time slice marker signal
A | 0x20240118 | cdm_bsy | CDM busy from VX
B | 0x20240118 | gdm_bsy | GDM busy from CDM
C | 0x20240118 | cdm_veto | veto from GDM to CDM to VX
D | 0x20240118 | vx1_rx[1] | serial data VX to CDM
E | not used (sink)
F | ALL | 1 | fixed logic level 1
</pre>

=== Register 5 0x80013014 VX_RX status ===

<pre>
assign register_data_out[5] = {
vx8_rx[3], vx8_rx[2], vx8_rx[1], vx8_rx[0],
vx7_rx[3], vx7_rx[2], vx7_rx[1], vx7_rx[0],
vx6_rx[3], vx6_rx[2], vx6_rx[1], vx6_rx[0],
vx5_rx[3], vx5_rx[2], vx5_rx[1], vx5_rx[0],
vx4_rx[3], vx4_rx[2], vx4_rx[1], vx4_rx[0],
vx3_rx[3], vx3_rx[2], vx3_rx[1], vx3_rx[0],
vx2_rx[3], vx2_rx[2], vx2_rx[1], vx2_rx[0],
vx1_rx[3], vx1_rx[2], vx1_rx[1], vx1_rx[0]
};
</pre>

=== Register 6 0x80013018 VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state ===

<pre>
assign register_data_out[6] = {
vx2_tx_out[2], vx2_tx_out[1], vx2_tx_out[0], vx1_tx_out[2],
vx1_tx_out[1], vx1_tx_out[0], ext_out[2], ext_out[1],
fp_led_out[3], fp_led_out[2], fp_led_out[1], fp_led_out[0],
ext_in_lv[4], ext_in_lv[3], ext_in_lv[2], ext_in_lv[1],
vx12_rx[3], vx12_rx[2], vx12_rx[1], vx12_rx[0],
vx11_rx[3], vx11_rx[2], vx11_rx[1], vx11_rx[0],
vx10_rx[3], vx10_rx[2], vx10_rx[1], vx10_rx[0],
vx9_rx[3], vx9_rx[2], vx9_rx[1], vx9_rx[0]
};
</pre>

<pre>
bit | version | fpga name | description
0 | ALL | vx9_rx | VX_RX
1 | | |
2 | | |
3 | | |
4 | ALL | vx10_rx | VX_RX
5 | | |
6 | | |
7 | | |
8 | ALL | vx11_rx | VX_RX
9 | | |
10 | | |
11 | | |
12 | ALL | vx12_rx | VX_RX
13 | | |
14 | | |
15 | | |
16 | ALL | ext_in_lv | LEMO inputs
17 | | |
18 | | |
19 | | |
20 | ALL | FP_LED | FP_LEDs
21 | | |
22 | | |
23 | | |
24 | ALL | ext_out[1] | LEMO outputs
25 | | ext_out[2] |
26 | ALL | vx1_tx | VX1_TX
27 | | |
28 | | |
29 | ALL | vx2_tx | VX2_TX
30 | | |
31 | | |
</pre>

=== Register 7 0x8001301C LED_OUT, EXT_OUT, VX_TX outputs ===

<pre>
wire [3:0] led_out_reg = register_data_in[7][3:0];
wire [2:1] ext_out_reg = register_data_in[7][5:4];
// register_data_in[7][6];
// register_data_in[7][7];
wire [7:0] vx_tx_out_reg = register_data_in[7][15:8];
</pre>

<pre>
bit | version | fpga name | description
0 | ALL | led_out_reg | FP_LED 1
1 | | | 2
2 | | | 3
3 | | | 4
4 | ALL | ext_out_reg | LEMO OUT 1
5 | | | 2
6 | - | |
7 | - | |
8 | ALL | vx_tx_out_reg | VX1_TX 0
9 | | | 1
10 | | | 2
11 | | | -
12 | ALL | | VX2_TX 0
13 | | | 1
14 | | | 2
15 | | | -
16 | - | |
17 | | |
18 | | |
19 | | |
20 | | |
21 | | |
22 | | |
23 | | |
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 8 0x80013020 VX_TX config ===

<pre>
wire [3:0] vx_tx_mux_sel = register_data_in[8][3:0];
vx_tx_loopback = register_data_in[8][31];
</pre>

<pre>
vx_tx_mux_sel is 4 bits (choice 0..15):
0 - power on default, control by vx_tx_out_reg
1 - GDM
2 - CDM
3 - pulser loopback test
4 - pulser loopback test
5 - 62.5 MHz output
6 - 125 MHz output
7 - trg, tsm, serial
8 - trg, veto, serial
9 - GPS box control (ds20k rev 0x20240814)
10
11
12
13
14
15 - production config: trg, veto, serial
</pre>

=== Register 9 0x80013024 trigger control ===

from version 0x20240724

<pre>
wire [15:0] trg_src_mask = register_data_in[9][15:0];
wire [31:16] tsm_src_mask = register_data_in[9][31:16];
</pre>

<pre>
wire [15:0] trg_src_bits =
{
1'b0,
1'b0,
1'b0,
1'b0,

1'b0, // gdm_hitmap_trigger,
cdm_hitmap_trigger,
sfp_rx_tsm,
sfp_rx_trg,

sfp_rx_data[1],
sfp_rx_data[0],
tsm_pulser,
trg_pulser,

lemo_in_sync[4],
lemo_in_sync[3],
lemo_in_sync[2],
lemo_in_sync[1]
};

wire [15:0] trg_bits = trg_src_bits & trg_src_mask;
wire [15:0] tsm_bits = trg_src_bits & tsm_src_mask;
</pre>

before that:

<pre>
wire [7:0] trg_src_mask = register_data_in[9][7:0];
wire [7:0] tsm_src_mask = register_data_in[9][15:8];
wire trg_pulser_enable = register_data_in[9][16];
wire tsm_pulser_enable = register_data_in[9][17];
wire trg_software = register_data_in[9][18];
wire tsm_software = register_data_in[9][19];
// bits 20:31 not used
</pre>

<pre>
wire [7:0] xxx_src_bits =
{
sfp_rx_data[1],
sfp_rx_data[0],
tsm_pulser & tsm_pulser_enable,
trg_pulser & trg_pulser_enable,
lemo_in_sync[4],
lemo_in_sync[3],
lemo_in_sync[2],
lemo_in_sync[1]
};
</pre>

<pre>
trg_src_mask and tsm_src_mask bits:
0 - LEMO IN 1
1 - LEMO IN 2
2 - LEMO IN 3
3 - LEMO IN 4
4 - trg_pulser
5 - tsm_pulser
6 - sfp_rx_data[0] // to become sfp_trg_in, selected from sfp_rx_data[0], cdm_rx trg_out and gdm trigger packet
7 - sfp_rx_data[1] // to become sfp_tsm_in, selected from sfp_rx_data[1], cdm_rx tsm_out and gdm tsm packet
</pre>

=== Register 10 0x80013028 status register ===

<pre>
bit | version | fpga name | description
0 | 0x20231013 | pll_locked | clock chip PLL is locked
1 | 0x20240118 | ts_reset_armed | timestamp reset is armed
2 | 0x20240118 | qsfp_tx_link_rx_status | QSFP link status is good for all enabled ports
3 | 0x20240424 | vx_tx_link_rx_status | VX link status is good for all enabled ports
4 | 0x20240118 | cdm_bsy | VX busy grand-or
5 | 0x20240118 | gdm_bsy | QSFP busy grand-or
6 | 0x20240118 | gdm_veto | gdm_veto = gdm_busy
7 | 0x20240118 | cdm_veto | CDM veto from GDM to VX
8 | 0x20240725 | bor_started | begin-of-run sequence started
9 | 0x20240725 | bor_finished | begin-of-run sequence fininished, see commands 10 and 11
10 | | |
11 | | |
12 | | |
13 | | |
14 | | |
15 | | |
16 | | |
17 | | |
18 | | |
19 | | |
20 | | |
21 | | |
22 | | |
23 | | |
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 11 0x8001302C trg_counter ===

trigger counter

=== Register 12 0x80013030 tsm_counter ===

time slice marker counter

=== Register 13 0x80013034 GPS 1pps period ===

GPS 1pps period in 8 ns clocks

=== Register 14 0x80013038 Rb clock 1pps period ===

PRS-10 Rb clock 1pps output period in 8 ns clocks

=== Register 15 0x8001303C SFP RX status ===

<pre>
bit | ds20k version | fpga signal name | description
0 | ALL | sfp_rx_data[15:0] | cdm sfp received data
16 | same | sfp_rx_data_is_k[0] |
17 | same | sfp_rx_data_is_k[1] |
18 | | 0 |
19 | 0x20231204 | sfp_rx_sel_lpb | sfp tx->rx loopback
20 | 0x20231013 | sfp_link_status | sfp link connected, exchanging data
21 | same | sfp_link_rx_status | sfp link receiving correct idle pattern from GDM TX
22 | same | sfp_link_error | sfp link receiver error (badk or overflow)
23 | same | sfp_rx_data_error | sfp transceiver state machine is in error state
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 16 0x80013040 SFP TX control ===

<pre>
wire [15:0] sfp_tx_data_reg = register_data_in[16][15:0];
wire [1:0] sfp_tx_ctrl_reg = register_data_in[16][17:16];
wire sfp_rx_sel_lpb = register_data_in[16][28]; // TX->RX serial loopback
wire sfp_tx_sel_lpb = register_data_in[16][29]; // RX->TX serial loopback
wire sfp_tx_sel_trg = register_data_in[16][30]; // 16 individual bits
wire sfp_tx_sel_reg = register_data_in[16][31]; // from register
</pre>

=== Register 17-22 0x80013044 QSFP RX data ===

QSFP RX data links 0..11

=== Register 23 0x8001305C QSFP TX control ===

<pre>
wire [15:0] qsfp_tx_data_reg = register_data_in[23][15:0];
wire [1:0] qsfp_tx_ctrl_reg = register_data_in[23][17:16];
wire qsfp_rx_sel_lpb = register_data_in[23][28]; // TX->RX loopback
wire qsfp_tx_sel_lpb = register_data_in[23][29]; // RX->TX loopback
wire qsfp_tx_sel_trg = register_data_in[23][30]; // 16 individual bits
wire qsfp_tx_sel_reg = register_data_in[23][31]; // data from register
</pre>

=== Register 24 0x80013060 trigger pulser period ===

trigger pulser period in units of 8 ns (125 MHz clock)

=== Register 25 0x80013064 trigger burst pulser ===

<pre>
wire [7:0] conf_burst_count = conf_pulser_burst_ctrl[31:24];
wire [23:0] conf_burst_period = conf_pulser_burst_ctrl[23:0];
</pre>

=== Register 26 0x80013068 tsm pulser period ===

time slice marker period in units of 8 ns (125 MHz clock)

=== Register 27 0x8001306C data write fifo ===

<pre>
wire fifo_reset = register_data_in[27][31];
wire fifo_to_fpga_wr1 = register_data_in[27][27];
wire fifo_to_fpga_wr2 = register_data_in[27][26];
wire [16:0] fifo_to_fpga_din = register_data_in[27][16:0];
</pre>

<pre>
assign register_data_out[27][31:24] = register_data_in[27][31:24]; // echo write bits
assign register_data_out[27][23] = fifo_to_fpga_full;
assign register_data_out[27][22] = fifo_to_fpga_empty;
</pre>

=== Register 28 0x80013070 data read fifo ===

<pre>
wire fifo_reset = register_data_in[28][31];
wire fifo_from_fpga_rd1 = register_data_in[28][25];
wire fifo_from_fpga_rd2 = register_data_in[28][24];
</pre>

<pre>
assign register_data_out[28][31:24] = register_data_in[28][31:24]; // echo write bits
assign register_data_out[28][21] = fifo_from_fpga_full;
assign register_data_out[28][20] = fifo_from_fpga_empty;
assign register_data_out[28][16:0] = fifo_from_fpga_dout;
</pre>

=== Register 29 0x80013074 packet routing ===

Control packet routing and loopbacks:

<pre>
wire [3:0] dn_route_ctrl = register_data_in[29][3:0];
wire [3:0] up_route_ctrl = register_data_in[29][7:4];
wire [3:0] fifo_to_fpga_route_ctrl = register_data_in[29][11:8];
//wire [3:0] spare_route_ctrl = register_data_in[29][15:12];

wire dn_mux_trg_enable = register_data_in[29][16];
wire dn_mux_tsm_enable = register_data_in[29][17];
wire dn_mux_sfp_rx_fifo_enable = register_data_in[29][18];
// 19
// 20..23

wire up_mux_vx_rx_enable = register_data_in[29][24];
// 25
// 26
// 27
wire fifo_from_fpga_hitmap_enable = register_data_in[29][28];
// 29..31
</pre>

fifo_to_fpga output routing:
* 0 - to down packet mux
* 1 - to up packet mux
* 2 - to fifo_from_fpga mux
* 3 - not used

down packet mux inputs:
* fifo_to_fpga_0
* trg_pkt16 enabled by dn_mux_trg_enable
* tsm_pkt16 enabled by dn_mux_tsm_enable
* sfp_rx_fifo_pkt16 enabled by dn_mux_sfp_rx_fifo_enable
* up_pkt16_2 loopback from up packet mux

down packet mux output routing:
* 0 - to vx_tx_pkt16 to VX TX (vx_link_tx and vx_ser_tx)
* 1 - to fifo_from_fpga mux
* 2 - to up packet mux loopback
* 3 - to qsfp_tx_pkt16 to GDM QSFP TX (cdm_link_tx)

up packet mux inputs:
* vx_rx_pkt16 enabled by up_mux_vx_rx_enable data from 12 VX RX links
* fifo_to_fpga_1
* dn_pkt16_2 loopback from down packet mux

up packet mux output routing:
* 0 - to sfp_tx_pkt16 to CDM SFP TX (VX data to GDM)
* 1 - to fifo_from_fpga mux
* 2 - to down packet mux loop loopback
* 3 - not used

fifo_from_fpga mux inputs:
* fifo_to_fpga_2
* dn_pkt16_1 from down packet mux
* up_pkt16_1 from up packet mux
* qsfp_tx_pkt16 from GDM QSFP link 0 (there is no GDM QSFP 12-to-1 mux)
* hitmap_pkt16 enabled by fifo_from_fpga_hitmap_enable in register 29

=== Register 30 0x80013078 qsfp link status ports 0..7 ===

<pre>
assign register_data_out[30] =
{
qsfp_rx_data_error[7], qsfp_link_error[7], qsfp_link_status[7], qsfp_link_rx_status[7], // 7
qsfp_rx_data_error[6], qsfp_link_error[6], qsfp_link_status[6], qsfp_link_rx_status[6], // 6
qsfp_rx_data_error[5], qsfp_link_error[5], qsfp_link_status[5], qsfp_link_rx_status[5], // 5
qsfp_rx_data_error[4], qsfp_link_error[4], qsfp_link_status[4], qsfp_link_rx_status[4], // 4
qsfp_rx_data_error[3], qsfp_link_error[3], qsfp_link_status[3], qsfp_link_rx_status[3], // 3
qsfp_rx_data_error[2], qsfp_link_error[2], qsfp_link_status[2], qsfp_link_rx_status[2], // 2
qsfp_rx_data_error[1], qsfp_link_error[1], qsfp_link_status[1], qsfp_link_rx_status[1], // 1
qsfp_rx_data_error[0], qsfp_link_error[0], qsfp_link_status[0], qsfp_link_rx_status[0] // 0
};
</pre>

=== Register 31 0x8001307C qsfp link status ports 8..11 ===

<pre>
assign register_data_out[31] =
{
4'b0000,
4'b0000,
4'b0000,
4'b0000,
qsfp_rx_data_error[11], qsfp_link_error[11], qsfp_link_status[11], qsfp_link_rx_status[11], // 11
qsfp_rx_data_error[10], qsfp_link_error[10], qsfp_link_status[10], qsfp_link_rx_status[10], // 10
qsfp_rx_data_error[9], qsfp_link_error[9], qsfp_link_status[9], qsfp_link_rx_status[9], // 9
qsfp_rx_data_error[8], qsfp_link_error[8], qsfp_link_status[8], qsfp_link_rx_status[8] // 8
};
</pre>

=== Register 32 0x80013080 bitmap of enabled qsfp ports ===

<pre>
wire [11:0] qsfp_mask = register_data_in[32][11:0];
wire qsfp_bsy_force = register_data_in[32][12];
// not used = register_data_in[32][15:13];
wire [15:0] gdm_veto_extend = register_data_in[32][31:16];
</pre>

On the GDM:

qsfp_rx_bsy[11..0] are pulses received from the CDMs

qsfp_rx_bsy_or is the grand-or of qsfp_rx_bsy masked by qsfp_mask

if qsfp_rx_bsy_or is high, gdm_bsy goes up and stays up for vx_bsy_extend*2 clocks

gdm_veto = gdm_bsy (bsy of any one VX causes trigger veto to all of them)

gdm_veto transition 0->1 causes gdm_veto_pulse. as long as gdm_veto is high, gdm_veto_pulse is generated every gdm_veto_extend clocks

gdm_veto_pulse is sent to all CDMs.

For this to work right, cdm_bsy_extend should not be bigger than gdm_veto_extend.

=== Register 33 0x80013084 bitmap of enabled VX ports ===

<pre>
wire [11:0] vx_mask = register_data_in[33][11:0];
wire vx_bsy_force = register_data_in[33][12];
// not used = register_data_in[33][15:13];
wire [15:0] vx_bsy_extend = register_data_in[33][31:16];
</pre>

On the CDM:

cdm_bsy is a grand-or of all vx_bsy masked by vx_mask (list of active VXes).

vx_bsy_extend controls how often state of cdm_bsy is sent to the GDM. when cdm_bsy goes 0->1, we send a cdm_bsy_pulse and keep resending it every vx_bsy_extend clocks as long as cdm_bsy stays high.

cdm_bsy_pulse is sent to the GDM.

sfp_rx_veto is the received from the GDM

if sfp_rx_veto goes up, cdm_veto goes up and stays up for gdm_veto_extend clocks.

for this to work right, CDM gdm_veto_extend must be bigger than GDM gdm_veto_extend.

cdm_veto goes to VXes on v1_tx_out[1] which is LVDS input 13.

=== Register 34, 35 0x80013088, 8C current timestamp ===

current 64-bit timestamp, 125 MHz

=== Register 36, 37 0x80013090, 94 old timestamp ===

old 64-bit timestamp, 125 MHz. timestamp saved at run start when it is reset to 0.

=== Register 38, 39, 40 0x80013098, 9C, A0 VX busy counters ===

8 bits per VX port, counters overflow to 255, reset at run start.

=== Register 41, 42, 43 0x800130A4, A8, AC QSFP busy counters ===

8 bits per QSFP port, counters overflow to 255, reset at run start.

=== Register 44, 45, 46, 47, 48 0x800130B0, B4, B8, BC, C0 CDM and GDM busy and veto counters ===

* cdm_busy = grand-or of all VX busy for enabled VXes
* gdm_busy = grand-or of all CDM busy for enabled CDMs
* gdm_veto = gdm_busy

<pre>
44 | lo 16 bits | cdm_bsy_up_counter | CDM busy, increments when cdm_busy goes 0->1
44 | hi 16 bits | cdm_bsy_pulse_counter | CDM busy to GDM, increments for each cdm_bsy_pulse sent to the GDM

45 | lo 16 bits | cdm_veto_pulse_counter | CDM veto from GDM, increments for each sfp_rx_veto received from the GDM
45 | hi 16 bits | cdm_veto_up_counter | CDM veto to VX, increments each time cdm_veto is set to 1.

46 | lo 16 bits | gdm_bsy_pulse_counter | GDM busy from CDM, increments for each qsfp_rx_busy received from CDMs (unless they overlap)
46 | hi 16 bits | gdm_bsy_up_counter | GDM busy, increments each time gdm_bsy goes 0->1

47 | lo 16 bits | gdm_bsy_refresh_counter | GDM busy, increments each time gdm_bsy is extended by new qsfp_rx_busy
47 | hi 16 bits | spare |

48 | lo 16 bits | gdm_veto_up_counter | GDM veto, increments each time gdm_veto goes 0->1
48 | hi 16 bits | gdm_veto_pulse_counter | GDM veto to CDM, increments for each gdm_veto_pulse sent to the CDM
</pre>

=== Register 49, 50, 51 0x800130xx VX RX serial data monitor ===

VX RX data, 8-bit per VX channel. k-bit is omitted.

<pre>
assign register_data_out[49][7:0] = vx_rx_data[0]; // vx1
assign register_data_out[49][15:8] = vx_rx_data[1]; // vx2
assign register_data_out[49][23:16] = vx_rx_data[2]; // vx3
assign register_data_out[49][31:24] = vx_rx_data[3]; // vx4

assign register_data_out[50][7:0] = vx_rx_data[4]; // vx5
assign register_data_out[50][15:8] = vx_rx_data[5]; // vx6
assign register_data_out[50][23:16] = vx_rx_data[6]; // vx7
assign register_data_out[50][31:24] = vx_rx_data[7]; // vx8

assign register_data_out[51][7:0] = vx_rx_data[8]; // vx9
assign register_data_out[51][15:8] = vx_rx_data[9]; // vx10
assign register_data_out[51][23:16] = vx_rx_data[10]; // vx11
assign register_data_out[51][31:24] = vx_rx_data[11]; // vx12
</pre>

=== Register 52, 53 VX link status ===

<pre>
assign register_data_out[52] =
{
vx_rx_error[7], vx_link_error[7], vx_link_status[7], vx_link_rx_status[7], // 7
vx_rx_error[6], vx_link_error[6], vx_link_status[6], vx_link_rx_status[6], // 6
vx_rx_error[5], vx_link_error[5], vx_link_status[5], vx_link_rx_status[5], // 5
vx_rx_error[4], vx_link_error[4], vx_link_status[4], vx_link_rx_status[4], // 4
vx_rx_error[3], vx_link_error[3], vx_link_status[3], vx_link_rx_status[3], // 3
vx_rx_error[2], vx_link_error[2], vx_link_status[2], vx_link_rx_status[2], // 2
vx_rx_error[1], vx_link_error[1], vx_link_status[1], vx_link_rx_status[1], // 1
vx_rx_error[0], vx_link_error[0], vx_link_status[0], vx_link_rx_status[0] // 0
};

assign register_data_out[53] =
{
//4'b0000,
1'b0, ~vx_rx_deser_rdy[0], vx_rx_code_err[0], vx_rx_disp_err[0],
//4'b0000,
//4'b0000,
//4'b0000,
vx_rx_monitor[0],
vx_rx_error[11], vx_link_error[11], vx_link_status[11], vx_link_rx_status[11], // 11
vx_rx_error[10], vx_link_error[10], vx_link_status[10], vx_link_rx_status[10], // 10
vx_rx_error[9], vx_link_error[9], vx_link_status[9], vx_link_rx_status[9], // 9
vx_rx_error[8], vx_link_error[8], vx_link_status[8], vx_link_rx_status[8] // 8
};
</pre>

vx_rx_monitor (12-bit) is from deserializer_10b.sv:

<pre>
assign monitor_out[9:0] = lastByte[9:0];
assign monitor_out[10] = comma;
assign monitor_out[11] = ready;
</pre>

=== Register 54 VX TX serial data monitor ===

* contents of vx_tx_monitor from vx_ser_tx.sv:
<pre>
assign monitor_out[8:0] = data_to_encoder; // 8-bit + k
assign monitor_out[9] = valid;
assign monitor_out[15:10] = 0;
assign monitor_out[25:16] = encoded_data; // 10-bit
assign monitor_out[26] = encoded_valid;
assign monitor_out[27] = 0;
assign monitor_out[31:28] = 0;
</pre>

=== Register 55 QSFP, SFP, VX link loss counters ===

<pre>
assign register_data_out[55] = {
vx_rx_error_counter,
qsfp_link_rx_status_drop_counter,
sfp_link_rx_status_drop_counter,
vx_link_rx_status_drop_counter
};
</pre>

<pre>
24..31 - CDM VX RX error counters, count any errors in the VX RX data path (bad serial data, fifo overflow, etc)
16..23 - GDM QSFP link loss counter, increments on qsfp_tx_link_rx_status 1->0 (reg10)
8..15 - CDM SFP link loss counter, increments on sfp_link_rx_status 1->0 (reg15)
0...7 - CDM VX link loss counter, increments on vx_tx_link_rx_status 1->0 (reg10)
</pre>

=== Register 68 GPS control and status ===

<pre>
assign register_data_out[68] =
{
8'b00000000, // 23+8
rb_1pps_counter, // 16+8 bits
gps_1pps_counter, // 8+8 bits
1'b0,
1'b0,
rb_ser_in, // 5
gps_data_in, // 4
gps_aux_out, // 3
gps_aux_in, // 2
rb_1pps_in, // 1
gps_1pps_in // 0
};
</pre>

= Firmware registers branch develop_ko =

== Register map ==

<pre>
0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
1 | ALL | ALL | RW | read write scratch register
2 | ALL | CDM | ?? | MGT not used
3 | ALL | CDM | RO | MGT debug_data
4 | ALL | CDM | RW | clk_config_vec
5 | ALL | CDM | ?? | not used
6 | ALL | CDM | RO | CDM_link_data_processing:o_error_time
7 | ALL | CDM | RO | CDM_link_data_processing:o_error_count
</pre>

== register 0 0x80010000 ==

GDM:
<pre>
0 - gdm_link_interface:i_mgt_rst
2 - gdm_link_interface:i_link_down_latched_rst
8 - GDM_link_data_processing:i_rst
10..9 - GDM_link_data_processing:i_data_mode
</pre>

CDM:
<pre>
0 - cdm_link_interface:i_mgt_rst
2 - cdm_link_interface:i_link_down_latched_rst
8 - CDM_link_data_processing:i_rst
10..9 - CDM_link_data_processing:i_data_mode
</pre>

== register 1 0x80010004 ==

GDM:
<pre>
nlinks-1..0 - gdm_link_interface:i_rx_slide_trigger
</pre>

CDM:
<pre>
nlinks-1..0 - cdm_link_interface:i_rx_slide_trigger
</pre>

== register 2 0x80010008 ==

GDM:
<pre>
nlinks-1..0 - gdm_link_interface:o_link_power_good
nlinks+15..16 - gdm_link_interface:o_link_status
</pre>

CDM:
<pre>
nlinks-1..0 - cdm_link_interface:o_link_power_good
nlinks+15..16 - cdm_link_interface:o_link_status
</pre>

== register 3 0x8001000c ==

GDM: simple loopback register

CDM:
<pre>
31..0 - debug_data - cdm_link_interface:o_debug
</pre>

o_debug:
<pre>
rx_link_rst & rx_error & rx_link_up & rx_receiving_data &
std_logic_vector(rx_state_count) & tx_state_count_on_rx_clk & i_rx_ctrl3(0) &
i_rx_ctrl1(1 downto 0) & i_rx_ctrl0(1 downto 0) &
rx_data_is_k28p1_k28p5 &
i_rx_data;
</pre>

== register 4 0x80010010 ==

GDM write:
<pre>
0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS
</pre>

GDM read:
<pre>
0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS
4 - clk_config_vec(4) - CLK_LOSXTn_LS
5 - clk_config_vec(5) - CLK_LOLn_LS
6 - clk_config_vec(6) - CLK_INTn_LS
7 - constant 1
31..8 - constant 0
</pre>

== register 5 0x80010014 ==

not used

== register 6 0x80010018 ==

GDM:
<pre>
3..0 - GDM_link_data_processing:i_status_select
</pre>

CDM:
<pre>
31..0 - CDM_link_data_processing:o_error_count
</pre>

== register 7 0x8001001c ==

GDM:
<pre>
31..0 - GDM_link_data_processing:o_status_vector
</pre>

CDM:
<pre>
31..0 - CDM_link_data_processing:o_error_count
</pre>

= GDM, CDM, VX packet communications =

* timestamp math
<pre>
1 clock is 8 ns is 125 MHz
8 bits of clocks is 256 clocks is 2048 ns is ~2 usec
16 bits of clocks is ~500 usec is 0.5 msec
24 bits of clocks is ~134 msec
32 bits of clocks is ~34 sec
40 bits of clocks is ~8.7 ksec is 2.4 hours
48 bits of clocks is ~625 hours is ~26 days
56 bits of clocks is ~6.6 kdays is ~18 kyears
62 bits of clocks is ~10 Mhours is 427 kdays is ~1.1 kyears
64 bits of clocks is ~4.4 kyears
</pre>
* 0x02 - TRG packet, 8 bytes, 640 ns on lvds link
<pre>
0 - 0x02
1 - trg_counter[7:0]
2 - ts64 low byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 high byte 3
7 - trg_mask[7:0]
8 - trg_mask[15:8]
</pre>
* 0x03 - TSM packet, 26 bytes, 2080 ns on lvds link
<pre>
0 - 0x03
1 - tsm_counter[7:0]
2 - ts64 low byte 0
3 - 1
4 - 2
5 - 3
6 - 4
7 - 5
8 - 6
9 - ts64 high byte 7
10 - gps_ts64 low byte 0
11 - 1
12 - 2
13 - 3
14 - 4
15 - 5
16 - 6
17 - gps_ts64 high byte 7
18 - gps_data low byte 0
19 - 1
20 - 2
21 - 3
22 - 4
23 - 5
24 - 6
25 - gps_data high byte 7
</pre>
* 0x81 - VX hitmap packet, 10 bytes, 800 ns on lvds link, 48 ns on fiber link
<pre>
0 - 0x81
1 - VX ID
2 - hitmap low byte, nits 7:0
3 - 15:8
4 - 23:16
5 - 31:24
6 - ...:32
7 - ...
8 - ...
9 - hitmap low byte, bits 63:...
</pre>
* 0x82 - CDM hitmap packet, 108 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x82
1 - cdm_hitmap_trigger_counter[7:0]
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - cdm_hitmap_or12 byte 0 (7:0)
11 - cdm_hitmap_or12 byte 1 (11:8) plus 4 bits: 12=0, 13=0, 14=0, 15=cdm_hitmap_grand_or
12 - cdm_hitmap_data, low byte, 12*64 bits = 768 bits = 48 words = 96 bytes
...
107 - cdm_hitmap_data, high byte
</pre>

= AXI bus timing =

* AXI 100 MHz clock, 10 ns, 32-bit data
* AXI single-dword read: 36 clock repeat rate, 360 ns is 2.777 MHz, 4 bytes per transfer is 11.11 Mbytes/sec
* AXI single-qword read: 13 clock repeat rate, 130 ns is 7.7 MHz, 4 bytes per transfer is 30 Mbytes/sec, not accounting for the gap
* AXI memcpy read: 4 transfers at 13 clocks, gap, 4 transfers at 13 clocks, gap, etc. below 30 Mbytes/sec.
[[Image:Ds-dm-axi-read.png|100px]]
[[Image:Ds-dm-axi-read-64.png|100px]]
[[Image:Ds-dm-axi-read-memcpy.png|100px]]
* AXI single-dword write: 20 clocks repeat rate, 200 ns is 5 MHz, 4 bytes per transfer is 20 Mbytes/sec
* AXI single-qword write: 13+20 clocks repeat rate, 330 ns is 3 MHz, 16 bytes per burst is 48 Mbytes/sec
[[Image:Ds-dm-axi-write.png|100px]]
[[Image:Ds-dm-axi-write-64.png|100px]]

= AXI bus addresses =

* see AXI/AMBA addresses assigned inthe FPGA project: (s_axi/reg0 is the DS-DM AXI registers)
<pre>
daq00:ds-dm-gcdm$ grep assign_bd_address scripts/GDM_CDM_XU8_bd.tcl
assign_bd_address -offset 0x80010000 -range 0x00004000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs axi_register_interfa_0/s_axi/reg0] -force
assign_bd_address -offset 0x000400000000 -range 0x40000000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs ddr4/C0_DDR4_MEMORY_MAP/C0_DDR4_ADDRESS_BLOCK] -force
assign_bd_address -offset 0x80000000 -range 0x00010000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs system_management_wiz/S_AXI_LITE/Reg] -force
daq00:ds-dm-gcdm$
</pre>

* see AXI/AMBA addresses exported from FPGA project to Linux kernel: (uio for debug bridge should say "debug bridge")
<pre>
root@gdm0:~# cat /sys/class/uio/uio*/name
axi-pmon
axi-pmon
axi-pmon
axi-pmon
root@gdm0:~#
root@gdm0:~# ls -l /sys/class/uio/
total 0
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio0 -> ../../devices/platform/amba/ffa00000.perf-monitor/uio/uio0
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio1 -> ../../devices/platform/amba/fd0b0000.perf-monitor/uio/uio1
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio2 -> ../../devices/platform/amba/fd490000.perf-monitor/uio/uio2
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio3 -> ../../devices/platform/amba/ffa10000.perf-monitor/uio/uio3
root@gdm0:~# ls -l /sys/class/uio/../../devices/platform/amba/
total 0
-rw-r--r-- 1 root root 4096 Oct 18 19:37 driver_override
drwxr-xr-x 3 root root 0 Oct 18 01:36 fd070000.memory-controller
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd0b0000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd400000.zynqmp_phy
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd490000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd500000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd510000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd520000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd530000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd540000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd550000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd560000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd570000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd6e0000.cci
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff000000.serial
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff020000.i2c
drwxr-xr-x 6 root root 0 Oct 18 01:36 ff0a0000.gpio
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0b0000.ethernet
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0e0000.ethernet
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0f0000.spi
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff160000.mmc
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff170000.mmc
drwxr-xr-x 3 root root 0 Oct 18 01:36 ff960000.memory-controller
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff9d0000.usb0
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa00000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa10000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa50000.ams
drwxr-xr-x 5 root root 0 Oct 18 01:36 ffa60000.rtc
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa80000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa90000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffaa0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffab0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffac0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffad0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffae0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffaf0000.dma
-r--r--r-- 1 root root 4096 Oct 18 19:37 modalias
lrwxrwxrwx 1 root root 0 Oct 18 19:37 of_node -> ../../../firmware/devicetree/base/amba
drwxr-xr-x 2 root root 0 Oct 18 19:37 power
lrwxrwxrwx 1 root root 0 Oct 18 01:36 subsystem -> ../../../bus/platform
-rw-r--r-- 1 root root 4096 Oct 18 01:36 uevent
root@gdm0:~#
</pre>

= Build firmware =

== Build from git clone ==

THESE ARE K.O.'s NOTES FOR CREATING THE PETALINUX DIRECTORY.

THEY DO NOT WORK!

COPY PETALINUX FROM A WORKING PROJECT AND USE "make gdm" and "make cdm" AS DESCRIBED BELOW.

* git clone git@edev-group.triumf.ca:fw/exp/darkside/gcdm.git
* #Makefile change VIVADO_SETTINGS_SCRIPT := /opt/Xilinx/Vivado/2022.1/settings64.sh
* #. /opt/Xilinx/Vivado/2022.1/settings64.sh
* . /opt/Xilinx/Vivado/2020.2/settings64.sh
* make clean
* make all_from_scratch
* . /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
* make petalinux_create
* make petalinux_rebuild_new_hw_des
* bomb out: The TMPDIR: /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp can't be located on nfs.
* mkdir /tmp/build_tmp
* rm -rf /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp/
* ln -s /tmp/build_tmp /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp
* try again
* grinds, loads a whole bunch of packages...
* finishes with desire to copy things to /tftpboot
* make sdcard_cp_to wants to copy files from PetaLinux_GDM_CDM/images/linux/ to SD card

== Build firmware ==

NOTE: directory Petalinux_GDM_CDM should already exist!

<pre>
#. /opt/Xilinx/Vivado/2020.2/settings64.sh
. /opt/Xilinx/Vivado/2022.2/settings64.sh
. /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
make clean_gdm # remove gdm build tree
make gdm # build or rebuild GDM
make copy_gdm # copy to gdm0
make clean_cdm # remove cdm build tree
make cdm # build or rebuild CDM
make copy_cdm # copy to cdm0 and cdm1
</pre>

copy to SD card:
<pre>
open a root shell
format 16 GB Sd card per above
cd .../ds-dm-gcdm
make copy
</pre>

= build times =

<pre>
CDM 12-june-2023 69aabc1c25130d970bc375aca684bd68849e6685
daq13 AMD-5700G 1688.61user 399.33system 23:28.84elapsed 148%CPU
dsdaqgw AMD-7700 1090.55user 247.34system 16:03.55elapsed 138%CPU
dsdaqgw AMD-7700 CDM incremental 196.68user 67.62system 7:35.42elapsed 58%CPU
dsdaqgw AMD-7700 CDM incremental 684.72user 94.17system 7:30.17elapsed 173%CPU
dsdaqgw AMD-7700 GDM incremental 849.84user 99.79system 9:04.56elapsed 174%CPU
</pre>

= prepare bootable sd card =

== format the sd card ==

this only needs to be done once

* become root
* cd ~olchansk/git/ds-dm-gcdm
* use "lsblk" to identify the SD card (should show as 8/16/32 GB block device)/ /dev/sdd in this case
* make sdcard_format SDCARD_DEVICE=/dev/sdd
* disconnect sd card, reconnect the sd card (to detect new partition tables, etc)

== copy CDM boot files ==

<pre>
cd /home/dsdmdev/git/ds-dm-gcdm
make copy
</pre>

== copy boot files to the sd card ==

* as root: identify partition labels, run "blkid", should say "BOOT", "rootfs" and "data"
* mount
<pre>
mkdir /media/olchansk/BOOT
mkdir /media/olchansk/rootfs
mkdir /media/olchansk/data
mount -L BOOT /media/olchansk/BOOT
mount -L rootfs /media/olchansk/rootfs
mount -L data /media/olchansk/data
cp PetaLinux_GDM_CDM/images/linux/BOOT.BIN /media/olchansk/BOOT/
cp PetaLinux_GDM_CDM/images/linux/boot.scr /media/olchansk/BOOT/
cp PetaLinux_GDM_CDM/images/linux/image.ub /media/olchansk/BOOT/
umount /media/olchansk/BOOT
umount /media/olchansk/rootfs
umount /media/olchansk/data
eject /dev/sdd
</pre>

= boot messages =

<pre>
Xilinx Zynq MP First Stage Boot Loader
Release 2020.2 Sep 24 2022 - 13:29:15
NOTICE: ATF running on XCZU4CG/silicon v4/RTL5.1 at 0xfffea000
NOTICE: BL31: v2.2(release):xlnx_rebase_v2.2_2020.3
NOTICE: BL31: Built : 18:02:46, Sep 28 2022

U-Boot 2020.01 (Sep 28 2022 - 18:03:39 +0000)

Model: DarkSide 20k DM
Board: Xilinx ZynqMP
DRAM: 2 GiB
usb dr_mode not found
PMUFW: v1.1
EL Level: EL2
Chip ID: zu4
NAND: 0 MiB
MMC: mmc@ff160000: 0, mmc@ff170000: 1
In: serial@ff000000
Out: serial@ff000000
Err: serial@ff000000
Bootmode: SD_MODE1
Reset reason: SOFT
Net:
ZYNQ GEM: ff0b0000, mdio bus ff0b0000, phyaddr -1, interface rgmii-id

Warning: ethernet@ff0b0000 (eth0) using random MAC address - d6:62:5f:13:00:44
eth0: ethernet@ff0b0000
ZYNQ GEM: ff0e0000, mdio bus ff0e0000, phyaddr -1, interface rgmii-id
Could not get PHY for eth1: addr -1

Hit any key to stop autoboot: 0
ZynqMP>
CTRL-A Z for help | 115200 8N1 | NOR | Minicom 2.7.1 | VT102 | Online 122:2 | ttyACM0
</pre>

= load FPGA from u-boot =

* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/124682257/U-Boot+FPGA+Driver

<pre>
ZynqMP> fpga info
Xilinx Device
Descriptor @ 0x000000007fddb2c0
Family: ZynqMP PL
Interface type: csu_dma configuration interface (ZynqMP)
Device Size: 1 bytes
Cookie: 0x0 (0)
Device name: zu4
Device Function Table @ 0x000000007fda5fe8
PCAP status 0xa0002fde
ZynqMP>
</pre>

* cp CDM_XU8_top.bit /tftpboot/fpga.bit

<pre>
dhcp
tftpb 0x10000000 fpga.bit
fpga loadb 0 0x10000000 ${filesize}
</pre>

<pre>
ZynqMP> dhcp
BOOTP broadcast 1
DHCP client bound to address 192.168.0.100 (1 ms)
*** Warning: no boot file name; using 'C0A80064.img'
Using ethernet@ff0b0000 device
TFTP from server 192.168.0.1; our IP address is 192.168.0.100
Filename 'C0A80064.img'.
Load address: 0x8000000
Loading: *
TFTP error: 'file /tftpboot/C0A80064.img not found for 192.168.0.100' (1)
Not retrying...
ZynqMP> tftpb 0x10000000 fpga.bit
Using ethernet@ff0b0000 device
TFTP from server 192.168.0.1; our IP address is 192.168.0.100
Filename 'fpga.bit'.
Load address: 0x10000000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
############
6.2 MiB/s
done
Bytes transferred = 7797807 (76fc2f hex)
ZynqMP> fpga loadb 0 0x10000000 ${filesize}
design filename = "CDM_XU8_top;UserID=0XFFFFFFFF;Version=2022.2"
part number = "xczu4cg-fbvb900-1-e"
date = "2024/08/14"
time = "14:18:22"
bytes in bitstream = 7797692
zynqmp_align_dma_buffer: Align buffer at 0000000010000073 to 000000000fffff80(swap 0)
ZynqMP>
</pre>

= load FPGA from Linux =

* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841847/Solution+ZynqMP+PL+Programming

== this will reset the CPU ==
<pre>
cp fpga.bit /lib/firmware/
echo fpga.bit > /sys/class/fpga_manager/fpga0/firmware
</pre>

* make .bin file:
<pre>
bootgen -image CDM_XU8_top.bif -arch zynqmp -o ./fpga.bin -w
</pre>
* cat CDM_XU8_top.bif
<pre>
dsdaqgw:ds-dm-gcdm$ cat CDM_XU8_top.bif
all:
{
[destination_device = pl] ./Vivado_CDM_XU8/CDM_XU8.runs/impl_1/CDM_XU8_top.bit
}
dsdaqgw:ds-dm-gcdm$
</pre>

== this will reset the CPU ==

<pre>
cp fpga.bin /lib/firmware/
echo fpga.bin > /sys/class/fpga_manager/fpga0/firmware
</pre>

== this will reset the CPU ==

<pre>
root@dsdm:~# ./fpgautil -b fpga.bin -f Full
</pre>

== this will reset the CPU ==

<pre>
DTSO file from here: https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841847/Solution+ZynqMP+PL+Programming
echo 0 > /sys/class/fpga_manager/fpga0/flags
mount -t configfs configfs /configfs
root@dsdm:~# dtc -O dtb -o fpga.dtbo -b 0 -@ fpga.dtso
root@dsdm:~# cp fpga.dtbo /lib/firmware/
root@dsdm:~# cp fpga.bit /lib/firmware/
root@dsdm:~# rmdir /configfs/device-tree/overlays/fpga
root@dsdm:~# mkdir /configfs/device-tree/overlays/fpga
root@dsdm:~# echo -n "fpga.dtbo" > /configfs/device-tree/overlays/fpga/path
</pre>

= fpgautil =

* https://github.com/Xilinx/meta-xilinx/blob/master/meta-xilinx-core/recipes-bsp/fpga-manager-script/files/fpgautil.c

<pre>
git clone https://github.com/Xilinx/meta-xilinx.git
cd meta-xilinx/meta-xilinx-core/recipes-bsp/fpga-manager-script/files/
scp fpgautil.c root@dsdm:
ssh root@dsdm
make fpgautil
ls -l ./fpgautil
</pre>

<pre>
root@dsdm:~# ls -l ./fpgautil
-rwxr-xr-x 1 root root 72256 Aug 16 00:15 ./fpgautil
root@dsdm:~# ./fpgautil

fpgautil: FPGA Utility for Loading/reading PL Configuration

Usage: fpgautil -b <bin file path> -o <dtbo file path>

Options: -b <binfile> (Bin file path)
-o <dtbofile> (DTBO file path)
-f <flags> Optional: <Bitstream type flags>
f := <Full | Partial >
-n <Fpga region info> FPGA Regions represent FPGA's
and partial reconfiguration
regions of FPGA's in the
Device Tree
Default: <full>
-s <secure flags> Optional: <Secure flags>
s := <AuthDDR | AuthOCM | EnUsrKey | EnDevKey | AuthEnUsrKeyDDR | AuthEnUsrKeyOCM | AuthEnDevKeyDDR | AuthEnDevKeyOCM>
-k <AesKey> Optional: <AES User Key>
-r <Readback> Optional: <file name>
Default: By default Read back contents will be stored in readback.bin file
-t Optional: <Readback Type>
0 - Configuration Register readback
1 - Configuration Data Frames readback
Default: 0 (Configuration register readback)
-R Optional: Remove overlay from a live tree

Examples:
(Load Full bitstream using Overlay)
fpgautil -b top.bit.bin -o can.dtbo -f Full -n full
(Load Partial bitstream using Overlay)
fpgautil -b rm0.bit.bin -o rm0.dtbo -f Partial -n PR0
(Load Full bitstream using sysfs interface)
fpgautil -b top.bit.bin -f Full
(Load Partial bitstream using sysfs interface)
fpgautil -b rm0.bit.bin -f Partial
(Load Authenticated bitstream through the sysfs interface)
fpgautil -b top.bit.bin -f Full -s AuthDDR
(Load Parital Encrypted Userkey bitstream using Overlay)
fpgautil -b top.bit.bin -o pl.dtbo -f Partial -s EnUsrKey -k <32byte key value>
(Read PL Configuration Registers)
fpgautil -b top.bit.bin -r
(Remove Partial Overlay)
fpgautil -R -n PR0
(Remove Full Overlay)
fpgautil -R -n full
Note: fpgautil -R is responsible for only removing the dtbo file from the livetree. it will not remove the PL logic from the FPGA region.

root@dsdm:~#
</pre>

= Boot from network =

== u-boot ==

<pre>
ZynqMP> setenv bootcmd run bootcmd_dhcp
ZynqMP> saveenv
ZynqMP> reset
</pre>

== boot.scr ==

<pre>
# boot.scr
# mkimage -C none -A arm -T script -d boot.scr boot.scr.uimg
echo Loading FPGA!
#tftpb 0x10000000 fpga.bit
tftpb 0x10000000 {ipaddr}.bit
fpga loadb 0 0x10000000 ${filesize}
echo Booting Linux!
run bootcmd_pxe
echo Done!
</pre>

<pre>
mkimage -C none -A arm -T script -d boot.scr boot.scr.uimg
</pre>

== tftpboot ==

<pre>
cp /home/olchansk/git/ds-dm-gcdm/boot.scr.uimg /tftpboot
ln -s /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/images/linux /tftpboot/xilinx-dsdm
mkdir /tftpboot/pxelinux.cfg
cat > /tftpboot/pxelinux.cfg/default-arm-zynqmp <<EOF
LABEL Linux
KERNEL xilinx-dsdm/Image
FDT xilinx-dsdm/system.dtb
#INITRD rootfs.cpio.gz.u-boot
EOF
</pre>

== boot sequence ==

* xilinx magic load BOOT.BIN from SD card
* load FPGA form BOOT.BIN
* load and run u-boot from BOOT.BIN or from image.ub
* u-boot load environment from ??? probably SD card uboot-redund.env, this includes our bootcmd
* run bootcmd which run bootcmd_dhcp which does:
* from /tftpboot:
* load and run boot.scr.uimg which does:
* load FPGA image xilinx-dsdm/${ipaddr}.bit
* run bootcmd_pxe which does:
* load pxelinux.cfg/default-arm-zynqmp which does:
* load xilinx-dsdm/Image ### this is the linux kernel
* load xilinx-dsdm/system.dtb ### this is the device tree
* start linux kernel
* linux kernel does dhcp
* linux kernel does nfs mount /nfsroot/%s,vers=3,tcp ### %s is replaced by the hostname supplied by DHCP
* userland starts and runs to console and ssh login.

= Xilinx ILA =

References:
* https://github.com/Xilinx/XilinxVirtualCable/tree/master
* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/644579329/Xilinx+Virtual+Cable
* https://support.xilinx.com/s/article/974879?language=en_US
* https://docs.xilinx.com/v/u/en-US/pg245-debug-bridge
* https://docs.xilinx.com/r/2020.2-English/ug908-vivado-programming-debugging/Initializing-Vivado-IDE-hw_server
* (we do not use this one) https://github.com/paulscherrerinstitute/xvcSupport/blob/master/README.md

Build xvcserver_cdm.exe: (it is built as a static executable, can be copied and run anywhere)
<pre>
ssh dsdaq@gdm0
cd /home/dsdaq/online/ds-dm-software
git pull ### get latest version
make xvcserver_cdm.exe
ssh root@gdm0
/home/dsdaq/online/ds-dm-software/xvcserver_cdm.exe
INFO: To connect to this xvcServer instance, use url: TCP:gdm0:2542
</pre>

To activate and use the vivado logic analyzer:
* data path: vivado -> hw_server -> xvcserver -> mmap axi bus -> debug bridge -> jtag -> ILA
* define ILAs in the code
* instantiate the xilinx debug bridge at AXI bus address 0x80020000 (FIXME!!! this collides with Ian's AXI addresses)
* build and boot the new FPGA firmware. updating the linux kernel is not necessary.
* login root@gdm0, run: /home/dsdaq/online/ds-dm-software/xvcserver_cdm.exe -v ### with "-v" for the first time to see that vivado does connect to it, without "-v", normally.
* login dsdaqgw, run: hw_server -s tcp:localhost:3121 -e "set auto-open-servers xilinx-xvc:gdm0:2542" ### tells us to connect to port localhost:3121
* login dsdaqgw, run vivado, open project, open hardware manager, open target, open new target, "connect to remote server", hostname "localhost", port "3121", next (bombs, try again, 3 times), popup add virtual cable, enter hostname "gdm0" port "2542", "ok", it shows in "hardware targets", "next", "finish", error popup "[Common 17-163] Missing value for option 'objects', please type 'set_property -help' for usage info", ignore it, in "hardware", right click the "gdm0" one, open target, under "hardware" and "debug bridge" we should see all the ILAs, under "hardware device properties", the "probes file" should have the ".ltx" file generated by vivado "Vivado_GDM_XU8/GDM_XU8.runs/impl_1/debug_nets.ltx", click on an ILA, a waveform should open.

= Software =

* ssh cdm0 # or gdm0
* sudo apt install i2c-tools libi2c-dev
* git clone https://bitbucket.org/team-ds-dm/ds-dm-software
* cd ds-dm-software
* make

== test_cdm.exe ==

=== CDM SFP status ===

<pre>
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0xbb2f0ae7
CDM firmware 0xbb2f0ae7
arg 1: [--sfp]
Polling SFP status...
identifier 0x03
connector 0x07
encoding 0x01
wavelength 0x0352 (850 nm)
vendor_name [FINISAR CORP. ]
vendor_pn [FTLF8526P3BNL ]
vendor_rev [A ]
vendor_sn [N3AB9M8 ]
vendor_date [200319 ]
dm_type 0x68
temp 29.0 C
vcc 3.323 V
tx_bias 7.250 mA
tx_power 478.4 uW
rx_power 2.3 uW
SFP good 1, status: temp 30.6 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 818 uW
...
</pre>

=== GDM QSFP status ===

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --qsfp3 --qsfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x53aee418
CDM firmware 0x53aee418
arg 1: [--qsfp3]
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out hi
arg 2: [--qsfp]
Polling QSFP status...
identifier 0x0d
status 0x02
los 0x8f
temp 28.2 C
vcc 3.323 V
rx_power 0.1 0.1 0.1 0.1 uW
tx_bias 7.6 7.6 7.6 0.0 mA
tx_power 792.2 773.8 823.0 0.1 uW
vendor_name [FINISAR CORP ]
vendor_pn [FTL410QD4C ]
vendor_rev [A ]
wavelength 850
max_temp 70 C
vendor_sn [X79AC0R ]
vendor_date [220309 ]
QSFP good 1, status: temp 27.7 C, los 0x8b, tx_bias 7.5 7.6 7.6 0.0 mA, tx_power 792 772 821 0 uW, rx_power 0 0 466 0 uW
</pre>

=== GDM clock status ===

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
</pre>

* clock chip not loaded, not running:

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x0f 0x16 0x04 0x00 0x01 0x68 0x09 0x00 0x02 0xf2 0x00 0x1f 0xf0 0x22 0xf2, 0x507: 0x00, 0x52A: 0x00, 0x53F: 0x04
Clock chip state 0, status: SYSINCAL XAXB_ERR LOL CAL_PLL IN0 IN_SEL_0 FASTLOCK_STATUS

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x1f7caf52 (528265042) should be ~125 MHz
0x1034 rx_clk: 0x00000000 (0) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x1f7caf52 (528265042) should be ~125 MHz
0x103C tx_clk: 0x00000000 (0) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip good (IN0 - external 10 MHz clock)

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x44 0x00 0xd0 0x01 0x1f 0xff 0x22 0xf2, 0x507: 0x3f, 0x52A: 0x01, 0x53F: 0x00
Clock chip state 1, status: LOS_IN2 OOF_IN2 IN0 IN_SEL_REGCTRL IN_SEL_0

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735a3b (125000251) should be ~125 MHz
0x1034 rx_clk: 0x07735a3c (125000252) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a3b (125000251) should be ~125 MHz
0x103C tx_clk: 0x07735a3c (125000252) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

=== CDM clock status ===

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
</pre>

* clock chip not loaded, not running:

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x0f 0x16 0x04 0x00 0x01 0x68 0x19 0x00 0x02 0xf2 0x00 0x1f 0xf0 0x22 0xf2, 0x507: 0x00, 0x52A: 0x00, 0x53F: 0x04
Clock chip state 0, status: SYSINCAL XAXB_ERR LOL CAL_PLL IN0 IN_SEL_0 FASTLOCK_STATUS

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735851 (124999761) should be ~125 MHz
0x1034 rx_clk: 0x00000000 (0) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x0127fefa (19398394) should be ~125 MHz
0x103C tx_clk: 0x00000000 (0) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip uses internal clock (IN1 - internal oscillator), observe rx_clk frequency is not same as others

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0x7f, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN1 IN_SEL_1 HOLD_HIST_VALID

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735853 (124999763) should be ~125 MHz
0x1034 rx_clk: 0x07735b0a (125000458) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735853 (124999763) should be ~125 MHz
0x103C tx_clk: 0x07735852 (124999762) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip uses SFP recovered clock (IN2 - sfp rx recovered clock), observe mgt_rx_ref_clk_raw (CDM 125 MHz oscillator) is different from others (SFP RX recovered clock)

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0xbf, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN2 IN_SEL_1 HOLD_HIST_VALID
^C
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773581b (124999707) should be ~125 MHz
0x1034 rx_clk: 0x07735ad6 (125000406) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735ad7 (125000407) should be ~125 MHz
0x103C tx_clk: 0x07735ad6 (125000406) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
^C
</pre>

=== CDM link status, PRBS test mode ===

<pre>
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
</pre>
* fiber disconnected, no link:
<pre>
CDM firmware: 0xbb2f0ae7
0x1000 SFP c.c. status: 0x00000031
CLK_IN_SEL_LS 0x1
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x00000025
sfp_mod_absent_N 1
sfp_rx_los_N 0
link_power_good 1
rx_link_up 0
rx_receiving_data 0
rx_error 1
rx_lnk_up_and_running 0
tx_link_up 0
tx_sending_data 0
tx_link_up_and_running 0
link_up_and_running 0
0x1014 SFP link data: 0x466a8187
rx_data 0x8187
k28p1_k28p5 0
rx_ctrl0 0x1
rx_ctrl1 0x1
rx_ctrl3 0x1
tx_state 0x1
rx_state 0x6
rx_receiving_data 0
rx_link_up 0
rx_error 1
rx_link_rst 0
0x2000 link test mode: 0x00000200, seconds: 0x00079093, errors: 0xffffffff
</pre>
* fiber connected, good link:
<pre>
CDM firmware: 0xbb2f0ae7
0x1000 SFP c.c. status: 0x000000b2
CLK_IN_SEL_LS 0x2
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x000007dc
sfp_mod_absent_N 0
sfp_rx_los_N 0
link_power_good 1
rx_link_up 1
rx_receiving_data 1
rx_error 0
rx_lnk_up_and_running 1
tx_link_up 1
tx_sending_data 1
tx_link_up_and_running 1
link_up_and_running 1
0x1014 SFP link data: 0x35c02774
rx_data 0x2774
k28p1_k28p5 0
rx_ctrl0 0x0
rx_ctrl1 0x0
rx_ctrl3 0x0
tx_state 0x3
rx_state 0x5
rx_receiving_data 1
rx_link_up 1
rx_error 0
rx_link_rst 0
0x2000 link test mode: 0x00000200, seconds: 0x00078a8c, errors: 0x00000000
</pre>

=== GDM link status, PRBS test mode ===

<pre>
1 link connected, no errors:
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
GDM firmware: 0x6b2ee010
0x1014: 0x00000008, 18: 0x00000008, 1C: 0x00000000, 24: 0x00000fff
0x2000: 0x00000200, time: 0x00078aa4, errors:
0xffffffff 0xffffffff 0xffffffff
0x00000000 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
</pre>

=== CDM link status ===

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test0

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0xbf, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN2 IN_SEL_1 HOLD_HIST_VALID

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x077357a0 (124999584) should be ~125 MHz
0x1034 rx_clk: 0x07735a5c (125000284) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a5b (125000283) should be ~125 MHz
0x103C tx_clk: 0x07735a5b (125000283) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM firmware: 0x6d2ef81a
0x1000 SFP c.c. status: 0x000000b2
CLK_IN_SEL_LS 0x2
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x000007dc
sfp_mod_absent_N 0
sfp_rx_los_N 0
link_power_good 1
rx_link_up 1
rx_receiving_data 1
rx_error 0
rx_lnk_up_and_running 1
tx_link_up 1
tx_sending_data 1
tx_link_up_and_running 1
link_up_and_running 1
0x1014 SFP link data: 0x35c6bcbc
rx_data 0xbcbc
k28p1_k28p5 0
rx_ctrl0 0x3
rx_ctrl1 0x0
rx_ctrl3 0x0
tx_state 0x3
rx_state 0x5
rx_receiving_data 1
rx_link_up 1
rx_error 0
rx_link_rst 0
0x2000 link test mode: 0x00000000, seconds: 0x00001671, errors: 0xffffffff

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 15
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
reg[15] is 0x0033bcbc (3390652)
</pre>

=== GDM link status ===

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test0

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x44 0x00 0xd0 0x01 0x1f 0xff 0x22 0xf2, 0x507: 0x3f, 0x52A: 0x01, 0x53F: 0x02
Clock chip state 1, status: LOS_IN2 OOF_IN2 IN0 IN_SEL_REGCTRL IN_SEL_0 HOLD_HIST_VALID

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735a0a (125000202) should be ~125 MHz
0x1034 rx_clk: 0x07735a0a (125000202) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a0a (125000202) should be ~125 MHz
0x103C tx_clk: 0x07735a0a (125000202) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly

register 0x1018 bit 0x800

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM firmware: 0x6d2ebce6
0x1014: 0x00000800, 18: 0x00000800, 1C: 0x00000000, 24: 0x00000fff
0x2000: 0x00000000, time: 0x00003d2f, errors:
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff

link data alternates 0xbcbc and 0x1cbc

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 22
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
reg[22] is 0xbcbc93ab (-1128492117)

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 22
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
reg[22] is 0x1cbc1aaf (482089647)
root@gdm0:~#

</pre>

=== Run trg and tsm ===

<pre>
on the GDM:

ssh root@gdm00
./test_cdm.exe --gdm-clocks
./test_cdm.exe --load-cc
./test_cdm.exe --cc
./test_cdm.exe --reset-mgt
./test_cdm.exe --gdm-clocks
./test_cdm.exe --writereg 2 0xff # enable LEMO NIM inputs
./test_cdm.exe --writereg 3 0xba54 # LEDs: lemo1, lemo2, trg, tsm
./test_cdm.exe --writereg 4 0x99 $ # LEMO out is trg_in_pulse
./test_cdm.exe --writereg 9 0x32010 # trg and tsm from trg_pulser and tsm_pulse
./test_cdm.exe --writereg 24 1250000 # trg pulser 100 Hz
./test_cdm.exe --writereg 26 125000000 # tsm pulser 1 Hz
./test_cdm.exe --writereg 23 0x40000000 # route trg_in and tsm_in to qsfp tx bits 0 and 1
./test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz

on the CDM:

ssh root@cdm01
./test_cdm.exe --cdm-clocks
./test_cdm.exe --load-cc
./test_cdm.exe --reset-mgt
./test_cdm.exe --cdm-clocks
./test_cdm.exe --cdm-link # issue --reset-mgt on CDM and GDM until link is good
./test_cdm.exe --writereg 2 0xff # enable LEMO NIM inputs
./test_cdm.exe --writereg 3 0xba54 # enable LEDs: lemo1, lemo2, trg, tsm
./test_cdm.exe --writereg 4 0x99 # enable LEMO output trg_in
./test_cdm.exe --writereg 9 0x8040 # enable trg_in and tsm_in from sfp rx bits 0 and 1
./test_cdm.exe --writereg 7 0xff00 # drive VX LVDS lines to logic level 0
./test_cdm.exe --writereg 8 0x2 # VX LVDS with trg and tsm, misrouted in second VX
#./test_cdm.exe --writereg 8 0x7 # VX LVDS with tsm
./test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz
</pre>

switch GDM and CDM to packetizer trg and tsm:

<pre>
on the GDM:

/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x00000000

on the CDM:

/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x0804
/home/dsdaq/online/ds-dm-software/test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz
</pre>

=== Run packet loopback ===

GDM CPU -> fifo_to_fpga -> GDM QSFP -> CDM SFP -> fifo_from_fpga -> CDM CPU

On the GDM: (CDM is connected to first QSFP port)

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --load-cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --reset-mgt
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 32 1 ### tell GDM to use first QSFP port
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link ### confirm link status is "3"
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 29 2 ### enable GDM packet data injection
/home/dsdaq/online/ds-dm-software/test_cdm.exe --test-fifo-write-loop
</pre>

One the CDM:

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --load-cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --reset-mgt
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 29 3 ### enable CDM packet data injection
/home/dsdaq/online/ds-dm-software/test_cdm.exe --test-fifo-read
</pre>

= dsvslice integration =

== VX setup ==

* general
** Start acq from user code = y, all others = n (as of Mar 2023: will start when first trigger received)
** Use NIM IO = y
** Use external clock = y
** LVDS quartet is input = n, y, n, y
** LVDS quartet mode = User, User, User, User

* trigger from front panel NIM:
** Trigger on external signal = y, all others = n
** connect CDM EXT_OUT(2) to VX "TrigIn"

* trigger from LVDS "Sync" mode
** Trigger on LVDS Sync signal = y, all others = n
** LVDS quartet mode = User, Sync, User, User

* trigger from LVDS "User" mode
** Trigger on LVDS pair 12 signal = y, all others = n
** LVDS quartet is input = n, y, n, y
** LVDS quartet mode = User, User, User, User

== GDM setup ==

* GDM is gdm0
* set inputs to NIM mode
* set outputs to TTL mode (this GDM has wrong NIM output circuit)
* use top QSFP slot, connect split cable 1 into CDM01, cable 2 into CDM02
* connect non-inverted NIM trigger signal to top-LEMO-left EXT_IN_LV(1)
* connect non-inverted NIM TSM signal to top-LEME-right EXT_IN_LV(2)
* GDM LEDs: TRIG, TSM, trigger enabled, trigger_out
* GDM LEMO_OUT: trigger, trigger

== CDM setup ==

* set CDM LEMO inputs to NIM
* set CDM LEMO outputs to NIM
* CDM01 is cdm0
* CDM02 is cdm1
* connect GDM fiber links to SFP port
* connect 1st VX port of CDM01 to VX1
* connect 1st VX port of CDM02 to VX2
* connect LEMO EXT_OUT(2) to VX "TrigIn", CDM01 to VX1, CDM02 to VX2
* power up
* CDM LEDs: GDM TRIG, GDM TSM, trigger enabled, trigger out
* CDM LEMO_OUT: gdm_trg, trigger

== After power up ==

* start the CDM frontend from the MIDAS "Programs" page. To start manually, see the Start Command on the Programs page.
* CDM frontend should enable the VX clock, disable the trigger
* from the MIDAS status page, goto the CDM page
* in the CDM01, CDM02 data table, the 2nd number should read 0x35c08008, if it does not and the last 4 digits randomly change, reset the GDM links:
** in order GDM, CDM01, CDM02, press "reset1" of each board, then press "reset4", then press "reset0", 2nd number should read as above. if it does not, STOP HERE.
* start a run
* CDM frontend will enable the trigger
* GDM frontend will enable the trigger
* LEDs on the GDM should flash, LEDs on the CDM should flash, TrigIn and TrigOut of the VX should flash
* stop a run
* GDM frontend will disable the trigger
* CDM frontend will disable the trigger

== Phase measurement ==

* pip3 install matplotlib
* pip3 install scipy
* export PYTHONPATH=$HOME/packages/midas/python
* #git clone https://github.com/J033X071C/PhaseMeasurement
* git clone https://bitbucket.org/team-ds-dm/phasemeasurement.git
* cd phasemeasurement
* python3 ./phaseMeasurement.py --help
<pre>
daq00:PhaseMeasurement$ python3 ./phaseMeasurement.py --help
usage: phaseMeasurement.py [-h] fileName numberEvents numberVX sizeEvents stopEvent minHist maxHist numberBin writeToTXT saveAsPDF

Read data from midas file (in .lz4 format) to calculate phase between the clock of VX1 and VX2

positional arguments:
fileName Name of the file we want to read data from (Example: run00389.mid.lz4)
numberEvents Number of events recorded in the file
numberVX Number of VX used in this run (usually 2...)
sizeEvents Number of points per event
stopEvent Number of events you want to go through to calculate phase
minHist Minimal value for the x axis of the phase measurement histogram (in ns)
maxHist Maximal value for the x axis of the phase measurement histogram (in ns)
numberBin Number of bins wanted for the generated histogram
writeToTXT Write argument as yes to generate text file with results of calculation
saveAsPDF Save generated plots to PDF files

optional arguments:
-h, --help show this help message and exit
daq00:PhaseMeasurement$
</pre>
* try an old file with
* python3 ./phaseMeasurement.py run00877.mid.lz4 10000 2 10000 500 -20 20 81 yes yes
<pre>
bin size = 0.494 ns
num_events = 347
mean = -1.705 ns
rms = 3.087 ns
mean_error = 0.166 ns
centroid = -1.706 ns.
width (sigma) = 0.363 ns.
error on the centroid = 0.016558 ns.
</pre>
* ls -l *.txt *.pdf
<pre>
dsdaq@dsvslice:~/online/PhaseMeasurement$ ls -l *.txt *.pdf
-rw-rw-r-- 1 dsdaq dsdaq 64728 Dec 14 16:56 run00877.mid.lz4_Plots.pdf
-rw-rw-r-- 1 dsdaq dsdaq 274 Dec 14 16:56 run00877.mid.lz4.txt
dsdaq@dsvslice:~/online/PhaseMeasurement$
</pre>
* scope settings (from email message)
<pre>
From fcote-lortie@triumf.ca Thu Dec 15 17:11:46 2022
From: Francis Cote-Lortie <fcote-lortie@triumf.ca>
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: Re: How to use scope as a waveform generator
Date: Fri, 16 Dec 2022 01:11:44 +0000

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display) by pressing on the button. It will go from 0 to 1.

The settings that we are using right now are:
Type of waveform: Sine wave
Offset: 0 V
Amplitude: 1 Vpp
Frequency: 50 kHz
Noise: 0 V
________________________________
From: Francis Cote-Lortie <fcote-lortie@triumf.ca>
Sent: Thursday, December 15, 2022 4:58 PM
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: Re: How to use scope as a waveform generator

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display)

The settings that we are using right now are:
Type of waveform: Sine wave
Offset: 0 V
Amplitude: 1 Vpp
Frequency: 50 kHz
Noise: 0 V
________________________________
From: Francis Cote-Lortie
Sent: Thursday, December 15, 2022 4:54 PM
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: How to use scope as a waveform generator

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display)
</pre>

== Standalone link test ==

<pre>
CDM: program clock chip
busybox devmem 0x80011000 32 0x8
busybox devmem 0x80011000 32 0x0
/home/dsdaq/si5394-i2c-file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b

GDM, CDM: link reset
busybox devmem 0x80011008 32 1

GDM, CDM: release reset
busybox devmem 0x80011008 32 0

CDM: link status (NOTE: SFP LOS and mod_absent are swapped!!!)
busybox devmem 0x80011010 32
0x00000024 <- fiber plugged
0x00000025 <- fiber unplugged
0x00000027 <- SFP unplugged
0x000007DC <- successful link with GDM

CDM: link state machine and data
busybox devmem 0x80011014 32
0x35C06FF6

CDM: set link to counting mode
busybox devmem 0x80012000 32 0x101
busybox devmem 0x80012000 32 0x100

CDM: time counter and error counter
root@cdm1:~# busybox devmem 0x80012000 32
0x00000100 <--- link mode
root@cdm1:~# busybox devmem 0x80012004 32
0x0000058C <--- seconds counter
root@cdm1:~# busybox devmem 0x80012004 32
0x0000058D
root@cdm1:~# busybox devmem 0x80012008 32
0x00000000 <--- error counter

GDM: no link
root@gdm0:~# busybox devmem 0x80011014 32
0x00000000
root@gdm0:~# busybox devmem 0x80011018 32
0x00000000
root@gdm0:~# busybox devmem 0x8001101c 32
0x00000000
root@gdm0:~# busybox devmem 0x80011024 32
0x00000FFF
root@gdm0:~#

GDM: good link channel 10, counting mode
root@gdm0:~# busybox devmem 0x80012000 32 0x101
root@gdm0:~# busybox devmem 0x80012000 32 0x100
root@gdm0:~# busybox devmem 0x80012008 32
0x3A8B68C2
root@gdm0:~# busybox devmem 0x80012008 32
0x42E03BEF
root@gdm0:~# busybox devmem 0x8001200c 32
0xDA090972
root@gdm0:~# busybox devmem 0x8001200c 32
0xDE6F22E9
root@gdm0:~# busybox devmem 0x80012019 32
Bus error
root@gdm0:~# busybox devmem 0x80012010 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012014 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012018 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x8001201c 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012020 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012024 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012028 32
0x00000000
root@gdm0:~# busybox devmem 0x8001202c 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012030 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012034 32
0xFFFFFFFF
root@gdm0:~#
</pre>

mapping of link channels:

<pre>
qsfp0 lane0 - 0x0100 - link 8
qsfp0 lane1 - 0x0200 - link 9
qsfp0 lane2 - 0x0400 - link 10
qsfp0 lane3 - n/c
qsfp1 lane0 - 0x0800 - link 11
qsfp1 lane1 - 0x0010 - link 4
qsfp1 lane2 - 0x0020 - link 5
qsfp1 lane3 - n/c
qsfp2 lane0 - 0x0040 - link 6
qsfp2 lane1 - 0x0080 - link 7
qsfp2 lane2 - 0x0001 - link 0
qsfp2 lane3 - n/c
qsfp3 lane0 - 0x0002 - link 1
qsfp3 lane1 - 0x0004 - link 2
qsfp3 lane2 - 0x0008 - link 3
qsfp3 lane3 - n/c
</pre>

script to start the test with 2 CDMs:

<pre>
ssh dsdaq@dsvslice
ssh root@gdm0 busybox devmem 0x80011008 32 1
ssh root@cdm0 busybox devmem 0x80011008 32 1
ssh root@cdm1 busybox devmem 0x80011008 32 1
ssh root@gdm0 busybox devmem 0x80011008 32 0
ssh root@cdm1 busybox devmem 0x80011008 32 0
ssh root@cdm0 busybox devmem 0x80011008 32 0
ssh root@gdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
ssh root@cdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
ssh root@cdm1 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
</pre>

== LEMO trigger GDM to CDM to VX ==

on the GDM:

<pre>
/home/dsdaq/si5394-i2c-file /home/dsdaq/GDM_v1.0_IN0_EXT1_and_IN1_fixed_Si5394-RevA-Registers.txt 0 0x6b
busybox devmem 0x80011008 32 0x1
busybox devmem 0x80011008 32 0x0
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x80001230 # QSFP TX fixed pattern
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 2 0xff # enable LEMO inputs
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 3 0x7654 # enable LED, one per LEMO input
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x0F0F # enable LEMO to trg_in and tsm_in
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x40001230 # enable trg_in and tsm_in output to QSFP
</pre>

on the CDM:

<pre>
/home/dsdaq/si5394-i2c-file /home/dsdaq/CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
/home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp
busybox devmem 0x80011008 32 1
busybox devmem 0x80011008 32 0
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x8040 # trg_in from sfp[0], tsm_in form sfp[1]
/home/dsdaq/online/ds-dm-software/test_cdm.exe 11 # trg_in counter
/home/dsdaq/online/ds-dm-software/test_cdm.exe 12 # tsm_in counter
</pre>

== GPS receiver VCL-2705 ==

* Valiant VCL-2705 GPS receiver
* https://www.valiantcom.com/time-distribution/gps-receiver-irig-b.html
* USB connection is /dev/ttyACM0, 115200 bps, "GNSSAUX" prompt
* minicom -D /dev/ttyACM0 -b 115200
* user manual download instructoins - see sheet of paper with user name and password in the shipping box
* usb commands:
<pre>
gnss-help
gnss-showver -> F/W: Ver 1.7 Feb 17 2020 16:20:43

gnss-showselftest -> no antenna connected
Overall : FAIL
EPROM Test: PASS
Antenna : NOT DETECTED
GNSS : COMMUNICATION OK

gnss-showsettings
GNSS NMEA BAUDRATE :115200
GNSS ANTENNA LENGTH :30 meters
GNSS USER CONFIGURED DELAY:-65 nanoseconds
GNSS 1PPS PULSE WIDTH :200 milliseconds
GNSS MODE :GPS
GNSS STATUS :STATIONARY

GNSSAUX> gnss-showserial
SERIAL :2704H01V17MAX310

GNSSAUX> gnss-showmode
GNSS MODE : GPS

GNSSAUX> gnss-showstatus
GNSS STATUS: STATIONARY

GNSSAUX> gnss-showalarms
CURRENT ALARMS GNSS
ANTENNA : NOT DETECTED

GNSSAUX> gnss-showerrors
CURRENT ALARMS GNSS
ANTENNA : **NOT DETECTED
ERROR STATISTICS GNSS
RMC GOOD DURATION : Secs 0
RMC BAD DURATION : Secs 0
LOCK GOOD SECS DURATION : Secs 0
LOCK BAD SECS DURATION : Secs 0
SATINFO GOOD ITERATIONS :0
SATINFO REJECT ITERATIONS :0
SATINFO NOTALKER ITERATIONS :0
CURRENT MONITOR STATE GNSS
STATE :Phase-1 HUNTING ANTENNA DETECT

GNSSAUX> gnss-showsatinfo
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-showsats
Total Sats: 0

GNSSAUX> gnss-showmyloc
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-show1ppsstate
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-showjamstatus
Not Available !

GNSSAUX> gnss-showspoofstatus
Not Available !

--- antenna connected, can see the sky ---

GNSSAUX> gnss-showselftest
Overall : PASS
EPROM Test: PASS
Antenna : DETECTED
GNSS : COMMUNICATION OK

GNSSAUX> gnss-showalarms
CURRENT ALARMS GNSS
ANTENNA : DETECTED
GNSS LOCK : AVAILABLE

GNSSAUX> gnss-showerrors
CURRENT ALARMS GNSS
ANTENNA : DETECTED
GNSS LOCK : AVAILABLE
ERROR STATISTICS GNSS
RMC GOOD DURATION : Mins 1,Secs 34
RMC BAD DURATION : Secs 55
LOCK GOOD SECS DURATION : Mins 1,Secs 34
LOCK BAD SECS DURATION : Secs 55
SATINFO GOOD ITERATIONS :3
SATINFO REJECT ITERATIONS :0
SATINFO NOTALKER ITERATIONS :0
CURRENT MONITOR STATE GNSS
STATE :Phase-4 NORMAL OPERATION, Monitoring GNSSLOCK

GNSSAUX> gnss-showsatinfo
GNSS MODE :GPS
SATELLITE INFORMATION TALKER:GPS
NMEA ID : 01-32
NO OF SATELLITES IN VIEW: 08
NO OF XXGSV MSGS : 03
SatNo PRN NO (SV ID) ELEVATION (degs) AZIMUTH (degs) C/No (SNR)
1 0 0 0 0
2 0 0 0 0
3 0 0 0 0
4 0 0 0 0
5 0 0 0 0
6 0 0 0 0
7 0 0 0 0
8 0 0 0 0
<<<<<< End of Sat Info >>>>>>

GNSSAUX> gnss-showsats
Total Sats: 8

GNSSAUX> gnss-showmyloc
GNSS RECEIVER LOCATION:
Latitude : 4914.81911
Longitude: 12313.69595

GNSSAUX> gnss-show1ppsstate
GPS 1PPS STATE: LOCKED

GNSSAUX> gnss-showmyloc

GNSS RECEIVER LOCATION:
Latitude : 4914.80688
Longitude: 12313.69531

enter into google maps search box as: 49 14.80688, -123 13.69531, observe the space, the moved dot and the minus.

GNSSAUX> gnss-resetgnss

Executing....Please Wait....
$$$$$END
</pre>

* 1PPS BNC output: period 1 sec, pulse width 200 ms, 3.3V into 1MOhm, 1.38V into 50Ohm.
* IRIG-B BNC output: 5.6V into 1MOhm, 2.4V into 50Ohm.
* IRIG-B format selector: default is all up.

== Rb clock PRS10 ==

* https://www.thinksrs.com/products/prs10.html
* 10 MHz output is sine wave around 5V peak to peak
* 1PPS BNC output is 10 usec pulse, 5V into 1MHohm.
* RS232 connection: minicom -D /dev/ttyUSB0 -b 9600
* ser2net config: localhost,3001:raw:600:usb-5-2-1.0:9600 -XONXOFF -RTSCTS LOCAL
* ssh daq13, cd /home/olchansk/git/ser2net, ./ser2net -c ~/daq/ds/ser2net.conf -d
* ssh daq13, cd ~/daq/ds, python3 prs10.py
* RS232 commands:
<pre>
ID?
PRS10_3.56_SN_105719
VB1
SN?
RS1 -- reset
ST? -- status
FC? -- 10MHz OCXO drive voltage DAC settings
DS? -- "detected signals"
GA? -- gain of frequency lock loop between ovenized oscillator and Rb cell, 0=use ovenized oscillator only
MO? -- magnetic offset of the Rb cell, range 2300..3600, if out of range, unit must be set to different operating mode, see prs10m.pdf
MR? -- magnetic read
TT? -- time-tag, time in ns between 1PPS out and 1PPS in
TS? -- time slope, ???
TO? -- time offset, ???
PS? -- pulse slope, ???
PL? -- 0=phase lock off, 1=phase lock on, lock to 1PPS input
PT? -- phase lock integrator time constant, PT8 is integrator time constant 18.2 hours, natural time constant 2.25 hours
PF? -- phase lock stability factor, PF2 is "1"
PI? -- phase lock integrator

Analog to digital 12 bit ADC, values 0.000 to 4.998

AD0? -- Spare (J204)
AD1? -- +24V(heater supply) divided by 10.
AD2? -- +24V(electronics supply) divided by 10
AD3? -- Drain voltage to lamp FET divided by 10
AD4? -- Gate voltage to lamp FET divided by 10
AD5? -- Crystal heater control voltage
AD6? -- Resonance cell heater control voltage
AD7? -- Discharge lamp heater control voltage
AD8? -- Amplified ac photosignal
AD9? -- Photocell’s I/V converter voltage divided by 4
AD10? -- Case temperature (10 mV/°C)
AD11? -- Crystal thermistors
AD12? -- Cell thermistors
AD13? -- Lamp thermistors
AD14? -- Frequency calibration pot / external calibration voltage
AD15? -- Analog ground

A/D via CPU E-port:

AD16? -- Varactor voltage for 22.48 MHz VCXO (inside RF synthesizer) / 4
AD17? -- Varactor voltage for 360 MHz VCO (output of RF synthesizer) / 4
AD18? -- Gain control voltage for amplifier which drives frequency multiplier / 4
AD19? -- RF synthesizer’s lock indicator voltage (nominally 4.8 V when locked )

ST?

ST1 : Power supplies and Discharge Lamp
ST1 bit, Condition which sets bit, Corrective Action
0 -- +24 for electronic < +22 Vdc
1 -- +24 for electronics > +30 Vdc
2 -- +24 for heaters <+22 Vdc
3 -- +24 for heaters > +30 Vdc
4 -- Lamp light level too low
5 -- Lamp light level too high
6 -- Gate voltage too low
7 -- Gate voltage too high

ST2: RF Synthesizer
ST2 bit, Condition which sets bit, Corrective Action
0 -- RF synthesizer PLL unlocked
1 -- RF crystal varactor too low
2 -- RF crystal varactor too high
3 -- RF VCO control too low
4 -- RF VCO control too high
5 -- RF AGC control too low
6 -- RF AGC control too high
7 -- Bad PLL parameter

ST3: Temperature Controllers
ST3 bit, Condition which sets bit
0 -- Lamp temp below set point
1 -- Lamp temp above set point
2 -- Crystal temp below set point
3 -- Crystal temp above set point
4 -- Cell temp below set point
5 -- Cell temp above set point
6 -- Case temperature too low
7 -- Case temperature too high

ST4: Frequency Lock-Loop Control
ST4 bit, Condition which sets bit
0 -- Frequency lock control is off
1 -- Frequency lock is disabled
2 -- 10 MHz EFC is too high
3 -- 10 MHz EFC is too low
4 -- Analog cal voltage > 4.9 V
5 -- Analog cal voltage < 0.1
6 -- not used
7 -- not used

ST5: Frequency Lock to External 1pps
ST5 bit, Condition which sets bit
0 -- PLL disabled
1 -- < 256 good 1pps inputs
2 -- PLL active
3 -- > 256 bad 1pps inputs
4 -- Excessive time interval
5 -- PLL restarted
6 -- f control saturated
7 -- No 1pps input

ST6: System Level Events
ST6 bit and Condition which sets bit
0 Lamp restart
1 Watchdog time-out and reset
2 Bad interrupt vector
3 EEPROM write failure
4 EEPROM data corruption
5 Bad command syntax
6 Bad command parameter
7 Unit has been reset
</pre>
* ST? on warm start
<pre>
received: PRS_10
received: 255,255,255,243,34,255
received: 0,0,0,1,34,0
...
received: 0,0,0,1,34,0
received: 0,0,0,0,2,0
...
received: 0,0,0,0,2,0
received: 0,0,0,0,4,0
</pre>
* ST? on loss of external 1PPS
<pre>
...
received: 0,0,0,0,4,0
disconnect 1PPS input
received: 0,0,0,0,132,0
...
reconnect 1PPS input
received: 0,0,0,0,132,0
received: 0,0,0,0,4,0
...
</pre>
* ST? on coldish start
<pre>
daq13:ds$ python3 prs10.py
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,0,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 0,0,1,1,34,0 ] old [ 80,0,0,1,34,1 ] counter: 0
received [ 0,0,0,1,34,0 ] old [ 0,0,1,1,34,0 ] counter: 0
received [ 0,0,16,1,34,0 ] old [ 0,0,0,1,34,0 ] counter: 6
received [ 0,0,20,1,34,0 ] old [ 0,0,16,1,34,0 ] counter: 5
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 8
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 3
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 49
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 250
...
</pre>
* ST? on cold start, note: no bump in the 10MHz clock as reported by DS-DM clock chip
<pre>
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,21,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 64,0,21,1,34,0 ] old [ 80,0,21,1,34,1 ] counter: 0
received [ 0,0,21,1,34,0 ] old [ 64,0,21,1,34,0 ] counter: 0
received [ 0,0,20,1,34,0 ] old [ 0,0,21,1,34,0 ] counter: 158
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 23
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 47
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 40
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 249
</pre>

= det fac integration test =

Connections:
* GPS receiver "IRIG-B SEL" both switches "up" - both "on", IRIG-B format B004
* GPS receiver USB-B -> long cable -> daq13 USB-A
* GPS receiver "1PPS out" -> long BNC cable -> BNC-T -> scope (5V, no 50ohm) and Rb clock BNC "1PPS in"
* GPS receiver "IRIG-B 50ohms" -> long BNC cable -> BNC-T -> scope (5V, no 50ohm) and DS-DM LEMO input 1 (TTL mode)
* Rb clock RS232 -> RS232 straight cable -> RS232-to-USB adapter -> daq13 USB-A
* Rb clock "1PPS out" BNC -> scope (5V, no 50ohm, trig threshold rising edge 2V)
* Rb clock "10MHz output 50 Ohm" BNC -> lemo -> lemo-T -> scope (sine wave, 5V, no 50ohm) and DS-DM clock input.

Programs to run:
* on daq13: cd /home/olchansk/git/ser2net, ./ser2net -c ~/daq/ds/ser2net.conf -d
<pre>
localhost,3001:raw:600:usb-5-2-1.0:9600 -XONXOFF -RTSCTS LOCAL
</pre>
* on daq13: cd /home/olchansk/daq/ds, python3 prs10.py ### connects to ser2net
<pre>
daq13:ds$ python3 prs10.py
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,21,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 64,0,21,1,34,0 ] old [ 80,0,21,1,34,1 ] counter: 0
received [ 0,0,21,1,34,0 ] old [ 64,0,21,1,34,0 ] counter: 0
received [ 0,0,20,1,34,0 ] old [ 0,0,21,1,34,0 ] counter: 158
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 23
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 47
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 40
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 249
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 12096
received [ 0,0,0,0,20,0 ] old [ 0,0,0,0,132,0 ] counter: 2
received [ 0,0,0,0,148,0 ] old [ 0,0,0,0,20,0 ] counter: 0
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,148,0 ] counter: 575
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 11449
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 2755
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 34386
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 41035
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 113401
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 33375
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 54767
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 85059
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 33222
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 119234
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 121990
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 128184
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 56002
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 428237
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 8250
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 1388
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 30506
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 142704
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 179451
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 106182
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 68747
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 65424
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 157587
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 6932
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 1388
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 20255
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 4
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 225941
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 72183
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 26970
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 1
39287
</pre>
* on ds-dm: ./test_cdm.exe --irigb ### note sbs mismatch is because I should wrap around at 16 bits
<pre>
dataframe: S00000000S100001010S...S001010101S000001000S, sec: 00, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16724 should be 82260
dataframe: S10000000S100001010S...S101010101S000001000S, sec: 01, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16725 should be 82261
dataframe: S01000000S100001010S...S011010101S000001000S, sec: 02, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16726 should be 82262
dataframe: S11000000S100001010S...S111010101S000001000S, sec: 03, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16727 should be 82263
dataframe: S00100000S100001010S...S000110101S000001000S, sec: 04, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16728 should be 82264
dataframe: S10100000S100001010S...S100110101S000001000S, sec: 05, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16729 should be 82265
</pre>

= DS-IOGC GPS interface board =

* git repository: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0/-/tree/main?ref_type=heads
* schematics: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0/-/blob/main/Altium/Project%20Outputs%20for%20DS-IOGC-Rev0/SCH-DS-IOGC-Rev0.pdf?ref_type=heads
* Rev1 git repository: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev1/
* schematics: [[:Image:SCH-DS-IOGC-Rev0.pdf|SCH-DS-IOGC]]

== Rb clock cable ==
<pre>
Rb clock ----- DS-IOGC side, pin numbers are as labeled on the cable connectors

1 - 1pps out - 8 - 1pps out --- correct
2 - nc
3 - nc
4 - TXD - 3 - RU-DATA-OUT - USB-RX input --- correct
5 - 1pps in - 2 - 1pps in --- correct
6 - +24V - 10 - +24V --- should by pin 1 to use both +24V pins?
7 - RXD - 15 - RU-DATA-IN - USB-TX output --- correct
8 - nc
9 - +24V - 10 - +24V --- correct
10 - GND - 9 - GND --- correct
</pre>

== VX connections ==
<pre>
VXA_TX0 - FPGA-OUT-LED2 - D6 LED ("10 MHz clock")
VXA_TX1 - FPGA-OUT-LED1 - D5 LED ("GPS DATA")
VXA_TX2 - FPGA-OUT-RU1PPS-EN - enable 1pps to Rb clock
VXA_TX3 - not used (62.5 Hz clock)

VXB_TX0 - FPGA-TX - PRS-10 RS-232 out
VXB_TX1 - FPGA-OUT-AUX - 5V TTL J5 LEMO out
VXB_TX2 - FPGA-OUT-OPTO1PPS-EN - PRS-10 power enable
VXB_TX3 - not used (62.5 MHz clock)

VXA_RX0 - FPGA-IN-OPTO1PPS - 1pps from GPS
VXA_RX1 - FPGA-IN-GPSDATA - J4 GPS data
VXA_RX2 - n/c
VXA_RX3 - n/c

VXB_RX0 - n/c
VXB_RX1 - FPGA-IN-AUX - J5 LEMO TTL input
VXB_RX2 - FPGA-RX - PRS-10 RS-232 in
VXB_RX3 - FPGA-IN_RU1PPS - PRS-10 1pps output
</pre>

== acceptance tests ==

* ./test_cdm_local.exe --writereg 7 0x4000 ### power up
* ./test_cdm_local.exe --writereg 7 0x4100 ### right LED D6
* ./test_cdm_local.exe --writereg 7 0x4200 ### left LED D5
* ./test_cdm_local.exe --writereg 7 0x6000 ### J5 LEMO measure +5VDC
* install LEMO jumper between LEMO J5 and J6
* write 0x4000 then ./test_cdm_local.exe 6 ### reads 0x5C, bit 5 0x20 reads 0
* write 0x6000 then ./test_cdm_local.exe 6 ### reads 0x7C, bit 5 0x20 reads 1
* write 0x4000 to clear all bits
* 10 MHz output ok - IN0 is good: ./test_cdm_local.exe --cc
<pre>
Clock chip state 1, status: OOF_IN2 IN1 IN_SEL_1 HOLD_HIST_VALID
</pre>
* without 10 MHz reports IN0 LOS (no signal) and OOF (wrong frequency)
<pre>
Clock chip state 1, status: LOS_IN0 OOF_IN0 OOF_IN2 IN1 IN_SEL_1 HOLD_HIST_VALID
</pre>
* IRIG-B via AUX-IN is ok:
<pre>
dataframe: S00100110S111001000S...S011110000S010000000S, sec: 34, min: 17, hrs: 00, day of year: 223, year: 2024, date: 10 aug, sbs: 1054 should be 1054
</pre>
* GPS 1pps to SMB-in ok (LED flashes)
* GPS 1pps to FPGA ok
* GPS data to FPGA ok
* GPS 1pps to PRS-10 enabled from FPGA ok
* PRS-10 Rb clock 1pps out to FPGA ok
* PRS-10 Rb clock 1pps out to FPGA to LEMO AUX out ok
* can see GPS 1pps, IRIG-B, PRS-10 1pps out, 10 MHz on the scope
* PRS-10 syncs on leading edge (0->1) of GPS 1pps signal
* NOT TESTED - smb output
* NOT TESTED - optical converter fiber to SMB
* NOT TESTED - optical converter SMB to fiber
* NOT TESTED - SMB loopback
* NOT TESTED - fiber loopback

= VX busy logic =

= DS-20K DAQ =

== Overview ==

DS-DM, GDM and CDM are key parts of the DS-20K DAQ system:
* common clock distribution from external clock (atomic clock, GPS) to GDM to per-quadrant CDMs to VX digitizers
* common trigger distribution from GDM internal algorithm or external input to all VX digitizers
* run control: GDM, CDM, VX all start recording data at the same time (clock and timestamp reset)
* collection of trigger data from VX digitizers to per-quadrant CDMs to GDM

== Deliverables ==

* hardware and firmware for GDM to CDM clock distribution
* hardware and firmware for CDM to VX clock distribution
* hardware and firmware for GDM external clock input (atomic clock or GPS)
* hardware and firmware for CDM and VX serial communications (VX LVDS I/O connector)
* firmware for run control (timestamp reset and sync): GDM to CDM to VX
* firmware for common trigger distribution: GDM to CDM to VX
* firmware for trigger data flow: VX to CDM to GDM
* firmware for busy control: VX to CDM to GDM back to CDM to VX
* firmware for flow control: FEP to GDM MIDAS frontend to GDM to CDM to VX
* GDM MIDAS frontend: clock selector and monitoring, trigger and run control, busy and flow control, GDM housekeeping
* CDM MIDAS frontend: clock monitoring, CDM housekeeping

specific performance:
* GDM external clock: 10 MHz GPS clock
* GDM to CDM fiber link:
** clock XXX MHz
** link data rate: XXX Gbit/sec
** CDM recovered clock: XXX MHz
** CDM recovered clock jitter: XXX ns
** phase alignment between CDMs: XXX ns
** phase alignment between CDMs persists across reboots, power cycles, firmware updates
** phase alignment between CDMs should be easy to measure
** phase alignment between CDMs should be easy to recalibrate if hardware parts are replaced (DS-DM boards, fiber transceivers, fiber cables, etc)
** data packet bandwidth: XXX Mbytes/sec
** data packet latency: XXX clocks
** data packet skew between CDMs: XXX clocks
* CDM to VX clock:
** clock: XXX MHz
** jitter, all CDM clock outputs: XXX MHz
** phase alignment between all CDM clock outputs: XXX ns
* CDM to VX trigger:
** TBD (use the VX "sync" input or VX LVDS I/O line or VX serial link packet)
* CDM to VX serial link:
** clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
** bit rate: XXX bits/sec
** latency: XXX link clocks
** maximum skew between VXes: XXX ns
* VX to CDM serial link:
** clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
** bit rate: XXX bits/sec
** latency: XXX link clocks
** maximum skew between VXes: XXX ns
* timestamp reset:
** maximum skew between VXes: XXX ns
* busy round trip time: XXX ns (VX to CDM to GDM back to CDM to VX)
* flow control latency: XXX ns (FEP software to GDM MIDAS frontend to GDM to CDM to VX)

== Technical risk items ==

this refers to unexpected behaviour and performance of
system components, causes big difficulty in implementing the system,
prevents delivery of deliverables, and prevents or negatively affects operation
of the DS-20K DAQ or of the whole experiment.

(14-sep-2022, list is not sorted by any criterial: severity, probability, ease of investigation)

(stability of course is long term stability, across hours, days, weeks, months, years)

* stability of Enclustra FPGA modules (crashes/year, failures to boot/year, flash corruption/year)
* stability of GDM external clock PLL (lock loss/year)
* stability of CDM recovered clock (lock loss/year, unexpected phase drifts, etc)
* unexpected failures or bit error rates in GDM-CDM fiber links
* stability of CDM VX clock outputs (stability of clock cleaner chip)
* stability of VX internal clock distribution (VX PLL lock loss events)
* stability of VX CAEN base firmware (different versions of CAEN base firmware have different clock distribution behaviour)
* strange things in CAEN base firmware (unexpected clocking of LVDS I/O, unexpected phase shifts between clocks, etc)
* DS-DM and VX hardware problems (incompatible LVSD I/O, incompatible clock signals, etc)

== Milestones ==

(14-sep-2022: at this stage of the project, priority must be given to identifying and retiring (so called) technical risk factors. it is not good to build the complete system only to discover that (for example) some Enclustra FPGA modules require 5 attempts to boot and erase their flash memory contents once a month. Both example are real-life actual problems that caused big difficulties in GRIFFIN/TIGRESS and ALPHA-g experiments).

Development and testing milestones in time reversed order:

* full DAQ data challenge: all VXes, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
* one quadrant data challenge: 1 VME crate of VX, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
* vertical slice data challenge: 1 VME crate, 2 VX, 2 CDM (1 VX per CDM), GDM, etc operate as designed
* GDM-CDM link finalized (data rate frozen, data packet format frozen, data content permitted to change)
* CDM-VX serial link finalized (data rate frozen, data packet format frozen, data content permitted to change)
* run control (timestamp reset) and trigger distribution design agreed upon, frozen (list of possible triggers permitted to change)
* VX to CDM to GDM data flow design agreed upon, frozen (data contents permitted to change)
* major technical risk items retired (all hardware and firmware is working as expecred without mysteries and surprises, all problems are identified, investigated, resolved, solutions tested)
* stable operation of CDM-VX serial links in vertical slice system
* stable operation of GDM to CDM clock in vertical slice system
* stable operation of CDM to VX clock in vertical slice system
* vertical slice system assembled (1 VME crate, 2 VX, 2 CDM, 1 GDM, network, FEP, TSP, MIDAS)

= ZZZ =

ZZZ

DS-DM

2024-09-05T21:23:05Z

Mwalczak:

= DS-DM =

DarkSide-20k Global and Crate Data Manager board (GDM and CDM).

Global Data Manager (GDM):
* clock distribution to CDM boards (including GPS/atomic clock source)
* collection of trigger data from CDM boards, processing and distribution of trigger decision to CDM boards
* run control
* integration with GPS 10MHz and 1pps clocks and GPS/IRIG date and time information

Crate Data Manager (CDM):
* clock distribution from GDM to CAEN VX digitizers
* receive trigger data from CAEN VX digitizers
* send trigger data to GDM
* run control and dead time control

= Links =

* https://dsvslice.triumf.ca/vx_napoli - DS vertical slice at TRIUMF
* https://edev-group.triumf.ca/fw/exp/darkside/gcdm - git repository, DS-DM firmware
* https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0 - git repository, DS-DM board
* https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0/-/blob/master/Altium/Project%20Outputs%20for%20DS-DM-Rev0/SCH-DS-xDM-Rev0.PDF - DS-DM schematics
* https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0/-/blob/master/Testing/Clk3_XO_125_to_fpgaIN_recoveredMGTclk_to_IN2_Si5394-RevA-Project.slabtimeproj
* https://docs.xilinx.com/v/u/en-US/ds593 - Xilinx Platform Cable USB II
* https://www.enclustra.com/en/products/system-on-chip-modules/mercury-xu8/ - Enclustra
* https://bitbucket.org/team-ds-dm/ds-dm-software - git repository, DS-DM MIDAS frontend
* https://bitbucket.org/team-ds-dm/phasemeasurement - VX Phase measurement script
* https://ladd00.triumf.ca/daqinv/frontend/list/178 - inventory database
* https://daq00.triumf.ca/elog-ds/DS-DAQ - DS-DAQ elog
* https://bitbucket.org/ttriumfdaq/dsproto_vx2740/src/develop/ - VX DSFE MIDAS frontend
* https://dsdaqgw.triumf.ca/vslice/ - vertical slice DAQ
* https://daq00.triumf.ca/DaqWiki/index.php/DarkSide - DS-DAQ Wiki page
* https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0 - GPS and Rb Clock adapter board

= Onboard hardware =

* jtag chain: arm_dap_0 0x5BA00477, xczu4_1 0x04721093
* Eclustra Mercury+ XU8 module: ME-XU8-4CG-1E-D11E-R2.1
** Xilinx® Zynq Ultrascale+™ MPSoC XCZU4CG-1FBVB900E
** DDR4 ECC SDRAM (PS) 2 GB
** DDR4 SDRAM (PL) 1GB
* ethernet mac chip: AT24MAC402-SSHM-T ("602" chip is wrong)
* USB UART for Enclustra serial console, micro-USB, 115200n8
* clock chip: SI5394A-A-GM and oscillator CS-044-054.0M (54 MHz)
* U23 3.3V current meter and thermometer, LTC2990IMS#TRPBF
* LEDs:
** LED_FP A/B/C/D 0/1/2/3
** led1 - 3V3_SW_ON, SOM_POWER_GOOD - Enclustra FPGA module 3.3V power is good
** led2 - LTM4624 PGOOD
** led3 - FPGA_DONE - FPGA has booted
** led4 - TP-S-1, PCLK_P
** led5 - TP-S-2, PCLK_N
* LEMO connectors (top to bottom)
** J4 - input (NIM/TTL) (EXT_IN_LV(1), EXT_IN_LV(2))
** J5 - input (NIM/TTL) (EXT_IN_LV(3), EXT_IN_LV(4))
** J6 - external clock (GPS 10MHz and PPS)
** J7 - output (NIM/TTL) (EXT_OUT(1), EXT_OUT(2))
* SMA connectors
** J9, J10 - CLK_CCA from U6 C.C.
** J11, J12 - CLK_TP0
* RJ45 ethernet connector (100mbit: green light on, yellow light flashes when there is traffic)
* SFP connector (SFP is FTLF8526P3BNL, 6 Gbit/sec, 850 nm, 300m 50/125um OM3 MMF)
* 4 QSFP connectors (GDM)
* 6 VX connectors (CDM)

= Buttons, jumpers and switches =

Buttons:

* PB1 - HRST - reboot FPGA (power-on reset)
* PB2 - SRST - (SRSTn) - reboot ARM CPU

Switches:

* SW1 - boot mode BM0, BM1 [-->]
* SW2 - LEMO output NIM<->TTL
* SW3 - LEMO input 1 and 2 NIM/TTL
* SW4 - LEMO input 2 and 4 NIM/TTL
* SW5 - LEMO clock input NIM/TTL
* SW6 - serial console select. [PS<--PL] PS is ARM CPU, PL is FPGA.

= Front panel =

<pre>
| top
|
| LED-FP1 | LED_FP(0,1,2,3)
|
| SFP J???
|
| LEMO J4-LEFT, J4-RIGHT | EXT_IN_LV(1), EXT_IN_LV(2)
| LEMO J5-LEFT, J5-RIGHT | EXT_IN_LV(3), EXT_IN_LV(4)
| LEMO J6-LEFT, J6-RIGHT | CLK_EXT1, CLK_EXT0 (125 MHz only)
| LEMO J7-LEFT, J7-RIGHT | EXT_OUT(1), EXT_OUT(2)
|
| J-VX-1
| J-VX-2 or QSFP-1
| J-VX-3 or QSFP-2
| J-VX-4 or QSFP-3
| J-VX-5 or QSFP-4
| J-VX-6
|
| RJ45 J3 ethernet
|
| bottom
</pre>

= VX adapter board =

LVDS I/O connector
<pre>
34 pin connector: 0|:::: :::: :::: :::: :|15,16 n/c
</pre>

== split-cable connection ==

<pre>
0 .. 7 -> N/C

8 -> VX_RX(3) - not used
9 -> VX_RX(2) - busy VX to CDM
10 -> VX_RX(1) - DS20K 125 MHz serial data VX to CDM
11 -> VX_RX(0) - DS20K 62.5MHz clock VX to CDM

12 <- VX_TX(0) - TRG CDM to VX
13 <- VX_TX(1) - TSM CDM to VX, to be VETO CDM to VX
14 <- VX_TX(2) (set by jumper routed here or to VX CLKIN SYNC) - 125 MHz serial data CDM to VX
15 <- n/c (CLK routed to VX CLKIN CLK) - 62.5 MHz clock CDM to VX

16 - n/c
</pre>

== one-to-one connection ==

<pre>
0 -> VX2_RX(3)
1 -> VX2_RX(2)
2 -> VX2_RX(0)
3 -> VX2_RX(1)

4 <- CLK
5 <- VX2_TX(0)
6 <- VX2_TX(1)
7 <- VX2_TX(2)

8 -> VX1_RX(3)
9 -> VX1_RX(2)
10 -> VX1_RX(1)
11 -> VX1_RX(0)

12 <- VX1_TX(0)
13 <- VX1_TX(1)
14 <- VX1_TX(2) (set by jumper routed here or to VX CLKIN SYNC)
15 <- n/c (CLK routed to VX CLKIN CLK)

16 - n/c
</pre>

= Board schematics =

* [[File:SCH-DS-xDM-Rev0.PDF]]
* note: FPGA pin annotations ("IO", "SCLK", "PCLK", etc) on the schematics are bogus, instead, trace them to the FPGA pins.
* note: ENC A is J800, ENC B is J801, ENC C is J900 (schematic name to enclustra name)
* note: Enclustra special pins: "GC" is "clock capable", "HDGC" is "clock capable", "MGTREFCLK" is MGT reference clocks.
* board modifications:
** ethernet mac chip
** NIM output (no U15, etc)
** RJ45 wrong pinout (board mod or special ethernet cable)
** 125 MHz clock mods (TBW)
** disconnect QSFP0_SEL from SFP_RS0 and QSFP1_SEL from SFP_RS1, these signals are not used by modern SFPs
** provide SFP i2c modsel to allow SFP and QSFP at the same time on address 0x50

= FPGA MGT blocks =

<pre>
* SFP - ENC B45,B47 and B48,B50 - FPGA D5,D6 MGTHTX0_D and D1,D2 MGTHRX0_D
* QSFP0 TX0 - ENC C13,C17 - FPGA H5,H6 MGTHTX0_C
* QSFP0 TX1 - ENC C21,C25 - FPGA G7,G8 MGTHTX1_C
* QSFP0 TX2 - ENC C29,C23 - FPGA F5,F6 MGTHTX2_C
* QSFP1 TX0 - ENC C37,C41 - FPGA E7,E8 MGTHTX3_C
* QSFP1 TX1 - ENC C45,C47 - FPGA P5,P6 MGTHTX0_B
* QSFP1 TX2 - ENC C51,C53 - FPGA M5,M6 MGTHTX1_B
* QSFP2 TX0 - ENC C57,C59 - FPGA L3,L4 MGTHRX2_B
* QSFP2 TX1 - ENC C63,C65 - FPGA K5,K6 MGTHTX3_B
* QSFP2 TX2 - ENC C75,C77 - FPGA W3,W4 MGTHTX0_A
* QSFP3 TX0 - ENC C79,C81 - FPGA V5,V6 MGTHTX1_A
* QSFP3 TX1 - ENC C85,C87 - FPGA T5,T6 MGTHRX2_A
* QSFP3 TX2 - ENC C89,C91 - FPGA R3,R4 MGTHTX3_A
</pre>

= Clock distribution =

Simplified:

* 125 MHz osc -> CLK_XO_125 -> MGTREFCLK0_A -> not used
* 125 MHz osc -> CLK3_XO_125 -> MGTREFCLK1_B -> SFP RX ref clock, QSFP RX and TX ref clock (this is not final design!)
* (disconnected) 125 MHz osc -> CLK2_XO_125 -> FPGA AG8,AH8 (GC)
* 125 MHz osc -> C.C. in1

* C.C. in0 <- CLK_EXT1 (10 MHz GPS clock)
* C.C. in1 <- 125 MHz osc
* (disconnected) C.C. in2 <- CLK_CC_IN <- MGTREFCLK0_D <- SFP RX clock (cannot be used because of uncontrollable phase)
* C.C. in2 <- CLK2_XO_125 <- FPGA AG8,AH8 (GC) <- SFP RX recovered 125 MHz clock
* C.C. in3 <- CLK_FB

* C.C. 125 MHz -> CLK_CC_OUT0 -> MGTREFCLK0_B -> QSFP RX and TX ref clock (final design)
* C.C. 125 MHz -> CLK_CC_OUT1 -> MGTREFCLK1_D -> SFP TX clock
* C.C. 62.5 MHz -> VX clock fanout

proposed changes:
* add C.C. 125 MHz -> new CLK_CC_OUT2 (old CLK2_XO_125) -> FPGA AG8,AH8 (GC)
* change C.C. in2 <- new CLK_CC_IN <- FPGA AK8,AK9 (non-GC)

Complete:

<pre>
125 MHz oscillator - U5 fan out -
q0 -> CLK_XO_125 -> ENC C72,C74 -> FPGA R7,R8 MGTREFCLK0_A (not used)
q1 -> U6 C.C. in1
q2 -> CLK3_XO_125 -> ENC C7,C9 -> FPGA J7,J8 MGTREFCLK1_B -> SFP RX reference clock, QSFP RX and TX reference clocks (not final design!)
q3 -> disconnected on the board, was CLK2_XO_125 -> ENC C151,C153 -> FPGA AG8,AH8 (GC)

U6 C.C (clock cleaner) -

in0 <- CLK_EXT1 (presumably GPS 10 MHz ref clock)
in1 <- 125 MHz oscillator via U5
in2 <- (was: CLK_CC_IN <- ENC B10,B12 <- FPGA D9,D10 MGTREFCLK0_D <- SFP RX clock, 125 MHz)
in2 <- CLK2_XO_125 <- ENC C151,B153 <- FPGA AG8,AH8 (GC) <- mgt_link_data_to_processing.rx_data_clk (SFP RX data clock, 125 MHz)
in3 <- CLK_FB

out0 -> CLK_CCA -> U12 (125 MHz)
out1 -> CLK_CCB -> VX1..6 (62.5 MHz)
out2 -> CLK_CCC -> VX7..12 (62.5 MHz)
out3 -> CLK_FB into in3

CLK_CCA -> U12 (125 MHz fan out) ->

Q0 -> not used
Q1 -> CLK_CC_OUT0 -> ENC C3-5 -> FPGA L7,L8 MGTREFCLK0_B -> QSFP RX and TX reference clocks (final design)
Q2 -> CLK_CC_OUT1 -> ENC B3-5 -> FPGA B9,B10 MGTREFCLK1_D -> SFP TX clock
Q3 -> not used
Q4 -> not used
Q5 -> SMA J9/J10

CLK_TP0 - FPGA N7,N8 MGTREFCLK1_A - ENC C69,C71 - DS-DM SMA J11, J12 (NOT IN CDM PROJECT)

Notes:
* CLK_XO_125 (125 MHz osc) is not used
* 62.5 MHz VX clock does not go into the FPGA
* CLK_EXT0 going to in1 of U5 cannot be used. only permitted frequency is 125 MHz (it drives the MGT reference clocks) and is this frequency is too high for LEMO cables and connectors

Proposed modifications:
- CLK_CC_IN: from FPGA output pin (ENC C142,C144 - FPGA AK8,AK9) to clock cleaner in2 (instead of CLK2_XO_125 pins)
- CLK_CCA -> U12 -> currently unused out3 -> CLK2_XO_125 FPGA pins
- repurpose CLK_EXT0 at GPS 1pps/IRIG input to FPGA
</pre>

= I2C bus =

* I2C_SCL is J-ENC-A pin 111 I2C_SCL is FPGA I2C_SCL_PL AB13 (IO_L1N_TOL1D_64) and I2C_SCL_PS F18 (PS_MIO10)
* I2C_SDA is J-ENC-A pin 113 I2C_SDA is FPGA I2C_SDA_PL AH13 (IO_L7N_T1L1Q_AD13N_64) and I2C_SDA_PS G18 (PS_MIO11)

* XU8 secure EEPROM ATSHA204A at 0x64, this is 0110'010X -> linux _011'0010 is 0x32. (but responds to scan and read at 0x33)
* U4 ethernet mac chip, EEPROM at 1010 A2 A1 A0 X and MAC/serial_no at 1011 A2 A1 A0 X. A0=VCC, A1=VCC, A2=GND -> linux _101'0011 and _101'1011 is 0x53 and 0x5B.
* U6 clock chip, address 1101 0 A1 A0 X. A1=VCC, A0=N/C (internal pull-up) -> linux _110'1011 is 0x6b
* U23 voltmeter at 10011 ADR1 ADR0. ADR0=GND, ADR1=VCC -> linux _100'1110 is 0x4e
* SFP, address 1010000X -> linux _101'0000 is 0x50. additional SFP data at 0x51
* QSFP0, QSFP1, QSFP2, QSFP3 (QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL)

= I2C clock builder connection =

* use Silicon Labs USB "Clock builder pro field programmer", www.silabs.com/CBProgrammer
* connect rainbow jumper cable pins:
** black - 1-GND to GND on DS-DM
** white - 3-SCLK to SCL on the DS-DM
** grey - 7-SDA_SDIO to SDA on the DS-DM
* power up the DS-DM
* plug USB programmer into Windows laptop
* on Windows, run "ClockBuilder Pro"
* it should report "Field programmer detected", press "EVB GUI"
* in EVB GUI, press "Config", set I2C address 0x6B
* press "Scan", it should find Si5394A-A-GM
* select the "Status" tab, should see real-time status of clock chip

= GDM MGT configuration =

* TX configuration:
* GDM MGT transceivers are configured as "multilane" TX and RX.
* there is 12 TXes ("lanes")
* MGT reference 125 MHz clock goes into [2:0]gtreclk00_in and [11:0]gtrefclk0_in
* one MGT is designated as "master"
* PLL of master MGT converts reference clock into common TX clock and common tx_user_clk2 which becomes tx_data_clk
* common tx_user_clk2 aka tx_data_clk goes into all TXes and clocks tx_data.
* tx_user_clk2 aka tx_data_clk is 125 MHz but not same phase as MGT reference clock.

* RX interim configuration:
* there is 12 RXes ("lanes")
* each RX produces it's own recovered RX clock
* "multilane" configuration assumes all RX recovered clocks run at the same frequency (TX on the other end are driven by common TX clock, see above), but have different phase
* one RX recovered clock is designated as "master" (rx_user_clk2 aka rx_data_clk) and a phase-matching fifo/buffer is used to bring rx_data from all 12 RXes to this common rx_data_clk
* this works because each CDM SFP TX runs on the SFP RX recovered clock which is frequency-locked with the GDM QSFP TX clock.

* RX final configuration:
* MGTs permit using the common TX clock (tx_user_clk2 aka tx_data_clk) as the common rx_data_clk (they are frequency locked through the CDM).
* this permits use of tx_data_clk as the main clock domain in the GDM and removes the need to bring rx_data into the tx_data_clk domain (actually this is done in the MGT RX phase matching fifo/buffer).

= Clock path =

<pre>
10 MHz ext clock or GDM 125 MHz oscillator
-> GDM QSFP MGT reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz (GDM main clock domain) and TX bit clock 2.5 GHz
-> GDM QSFP optic transmitter
-> CDM SFP optic receiver
-> CDM SFP MGT, RX reference clock is CDM 125 MHz oscillator
-> MGT RX recovered clock 125 MHz (CDM main clock domain)
-> CC_CLK_IN -> CDM C.C. -> CC_CLK_OUT1 -> CDM SFP TX reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz and TX bit clock 2.5 GHz
-> (tx_data phase matching fifo from CDM main clock domain to tx_data_clk)
-> CDM SFP optic transmitter
-> GDM QSFP RX optic receiver (12x)
-> GDM QSFP MGT (RX reference clock is same as TX reference clock)
-> MGT RX recovered clock (12x recovered clocks)
-> in multi-lane configuration, one of them is the "master" recovered clock rx_data_clk
-> (rx_data phase matching fifo from rx_data_clk to GDM main clock domain)
</pre>

= Clock domains =

== GPS ==

* no GPS : GDM runs from internal 125 MHz oscillator
* external 10 MHz clock : GDM runs from external 10 MHz clock and optional 1pps signal (use VME-NIMIO32 NIM outputs)
* GPS receiver : GDM runs from GPS 10 MHz clock and GPS IRIG serial data
* LNGS GPS:
** provides 1pps and serial data over fiber from GPS receiver (master). LNGS xxx box is not used.
** serial data goes to GDM, decoded, 1pps signal extracted, goes to LEMO output
** 1pps from GDM is used to train the Rubidium clock which provides a 10 MHz clock
** 10 MHz output from Rubidium clock goes to GDM 10 MHz external clock input

== GDM ==

* AXI clock (100 MHz) - AXI registers
* 125 MHz oscillator - to clock cleaner
* 10 MHz external clock LEMO input - to clock cleaner
* FPGA 125 MHz clock CLK_CC_IN - to clock cleaner - not used (could be used for cascaded GDMs)
* 125 MHz oscillator - CLK3_XO_125 - mgt_b_ref_clk QSFP MGT reference clock (interim GDM design)
* clock cleaner output 125 MHz fanout:
** CLK_CC_OUT0 - QSFP MGT reference clock (final design)
** CLK_CC_OUT1 - not used (CDM SFP reference clock)
** CLK_CC_OUT2 - not used
* QSFP MGT TX data clock 125 MHz tx_data_clk (main clock domain)
** QSFP TX data
** QSFP RX data (in final design, rx_data_clk is same as tx_data_clk. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk == tx_data_clk)
** ds20k block
* (in interim design, rx_data_clk is disconnected from tx_data_clk. rx_data_clk is the "multilane master clock" which is one of the 12x RX recovered clocks. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk)

note: all these clocks are frequency locked to 125 MHz

== CDM ==

* AXI clock (100 MHz) - AXI registers
* 10 MHz external clock LEMO input - to clock cleaner (not used)
* 125 MHz oscillator to fanout
** to clock cleaner
** to SFP MGT RX reference clock: CLK3_XO_125 to mgt_rx_ref_clk to gtrefclk01_in
* SFP MGT RX recovered clock 125 MHz
** MGT PLL to MGT rx_user_clk2 aka rx_data_clk
** MGT rxrecclkout_out to mgt_rx_rec_clk to CLK_CC_IN to C.C.
* SFP MGT rx_user_clk2 aka rx_data_clk (250 MHz/8 bit, 125 MHz/16 bit, 62.5 MHz/32 bit data) (main clock domain)
** SFP RX data
** ds20k block
** VX TX clock PLLs
** VX RX clock PLLs
* C.C. fan out
** 62.5 MHz VX clocks (12x)
** CLK_CC_OUT0 (not used, GDM QSFP MGT reference clock)
** CLK_CC_OUT1 125 MHz to mgt_tx_ref_clk to gtrefclk00_in to SFP MGT TX reference clock
** CLK_CC_OUT2 (not used)
* SFP MGT tx_user_clk2 aka tx_data_clk
** SFP TX data
** TX data phase matching fifo from main clock domain to tx_data_clk
* VX TX clock PLLs
** 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
** VX TX data phase matching from main clock domain to VX TX clock (12 total)
** VX TX serializer
** VX TX LVDS transmitter
* VX RX clock PLLs
** 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
** VX RX LVDS receivers (12 total)
** VX RX deserializers (12 total)
** VX TX data phase matching from VX RX clock to main clock domain

== VX ==

* everything runs on the VX main 125 MHz clock
* correct phase of VX to CDM LVDS data is adjusted by scan of CDM VX RX PLL clock phase (VX to CDM link is now established)
* correct phase of CDM to VX LVDS data is adjusted by scan of CDM VX RX PLL clock phase (link is established after VX to CDM idle data pattern changes from "VX RX data bad" to "good".
* after good phases are found by scan, they are not expected to change unless cables are changed, CDM and VX modules are changed or CDM firmware is rebuilt. (rebuild of VX firmware should not affect LVDS data phase).
* if there is excessive link errors, phase scan must be repeated.

= Board test plan =

To test:
* Enclustra FPGA board
* SFP port - SFP_RS0/RS1 connected to QSFP0_SEL/1_SEL is wrong? our Finisar SFP says RS0, RS1 N/C, so probably okey.
* SW5 CLK_EXT1 NIM works. TTL needs to be tested. flipping SW5 CLK_EXT0 side from NIM to TTL makes CLK_EXT1 go LOS and OOF in the clock chip. R23 and R53 should be removed?
* BOOT_MODE 0 and 1

Partial:
* U23 3.3V current meter and thermometer. V1,V2 is current monitor, same as in application note. V3,V4 is thermometer, same in application note, except capacitor C118 is 0.1u instead of 470pF. Tested ok: Tint, VCC, V1, V2, V1-V2. Test failed: TR2 reads 50-something degC instead of same as Tint.

Done:
* LED_FP1A..D: tested ok. K.O. 15 sep 2022
* USB UART: tested ok. K.O. 15 sep 2022
* J4A, J4B, J5A, J5B LEMO inputs (NIM/TTL) EXT_IN_LV(1..4). TTL threshold 1.7V, NIM threshold -0.3V. 50 Ohm termination. TTL straight, NIM inverted.
* J6A, J6B LEMO clock inputs: 10 MHz TTL from chronobox works, 10 MHz NIM from IO32 works. 18-apr-2023
* J7A, J7B LEMO outputs EXT_OUT(1), EXT_OUT(2) (NIM/TTL) (tested 23nov2022, K.O.). TTL ???, NIM inverted.
** TTL out no 50 ohm termination: 0=0V, 1=5V, rise and fall time ~5 ns
** TTL out with 50 ohm termination: 0=0V, 1=2.5V, rise and fall time <2ns
** NIM out no 50 ohm termination: 0=+50mV, 1=-1.8V, rise and fall time ~3ns
** NIM out with 50 ohm termination: 0=0V, 1=-0.9V, rise and fall time <2ns
* ethernet MAC i2c chip (K.O. 20sep2022: can read: i2cdump 0 0x5b, nov2022 have u-boot driver)
* CDM VX ports 2x(CLK, 3 tx, 4 rx) tested using test_cdm.exe and LVDS loopback in VX firmware.
* SFP i2c tested KO 22jun2023
* QSFP i2c tested KO 22jun2023
* i2c testing complete 22jun2023
* QSFP rx,tx tested 26june2023. lane0,1,2 ok, lane3 tx not connected, laser is off. qsfp0,1,2,3 all lanes ok.
* SFP rx,tx tested 26june2023. rx and tx okey. LOS ok, mod_absent ok.

Failure:
* ethernet: does not connect to alliedtelesys switch. connects to my USB-eth adapter at 100 Mbit speed. uboot mii status reports connection speed oscillating between 1000, 100 and 10. K.O. 16-sep-2022
* SFP LOS and mod_absent are swapped (in the FPGA pin definitions?)
* QSFP slot numbering is wrong.

= Checklist for newly build boards =

* put new board on workbench
* check - vme connector present, vme extraction handles present
* check - standoff are removed from all thru-holes
* plug Enclustra module
* check - SW6 both switches are in the "PS" position
* connect micro-usb cable from linux PC
* connect ethernet from 1gige capable network switch
* connect power from lab power supply - GND, +5V and -12V nominal, +5.7V and -12.5V actual
* power up, +5V current 2.10-2.8A, -12V current 0.05A
* on linux PC, open a new terminal, run: minicom -D /dev/ttyACM0
* in minicom window, observe messages about Xilinx first stage boot loader, etc
* on the ethernet switch, observe network link is 1gige speed (not 10mbit, not 100mbit).
* if everything boots okey, there will be a login prompt, login as root, password root.
* busybox devmem 0x80010000 # read firmware version number, i.e. 0xEDAD0A77
* TBW - test LEMO inputs
* TBW - test LEMO outputs
* TBW - test VX connectors
* TBW - test SFP connector
* TBW - test QSFP connector

= Serial console =

* check that linux computer has correct udev rules to allow access to /dev/ttyACM devices, see https://daq00.triumf.ca/DaqWiki/index.php/SLinstall#Configure_USB_device_permissions and https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Configure_USB_device_permissions
* connect micro-USB cable to connector J-UCB, other end connect to linux computer
* observe /dev/ttyACM0 was created
* run "minicom -D /dev/ttyACM0" (default serial settings are ok, otherwise, 115200n8)
* should have gdm-cdm login
* username root, password root

= i2c =

<pre>
ZynqMP> i2c bus
Bus 0: i2c@ff020000
ZynqMP> i2c dev 0
Setting bus to 0
ZynqMP> i2c probe
Valid chip addresses: 33 4E 53 5B 6B 77
ZynqMP> i2c md 0x5b 0x98
0098: fc c2 3d 00 00 12 1a 6e 0a 90 85 04 94 10 08 50 ..=....n.......P
</pre>

<pre>
root@gdm-cdm:~# i2cdetect 0
Warning: Can't use SMBus Quick Write command, will skip some addresses
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0.
I will probe address range 0x03-0x77.
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f
00:
10:
20:
30: -- -- -- 33 -- -- -- --
40:
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60:
70:
root@gdm-cdm:~#
root@gdm-cdm:~# i2cdump 0 0x5b
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0, address 0x5b, mode byte
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00 ???????P?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e ........??=..??n
a0: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00 ???????P?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e ........??=..??n
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@gdm-cdm:~#
</pre>

<pre>
root@gdm0:~# i2cdetect -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#
</pre>

<pre>
root@cdm0:~# i2cdetect -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: 50 51 -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@cdm0:~#
</pre>

* 0x33 - XU8 secure EEPROM (should be at 0x32)
* 0x4e - U23 current and temperature monitor
* 0x50, 0x51 - SFP
* 0x50 - QSFP, 4 QSFP modules enabled by GPIO QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL
* 0x53, 0x5b - ethernet mac eeprom
* 0x6b - U6 clock chip

== U23 ==

NOTE: Text reads double of Tint. not sure why. K.O. 21-mar-2024.

* internal temperature only

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x00 b # control register: "repeat mode, internal temperature only"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x03 # "Tint ready" and "busy", "busy is always 1 in repeat mode"
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 00 03 03 81 db 2a ce 2a 8e 00 6f 00 45 20 3f ?.????*?*?.o.E ?
10: 01 00 01 01 01 db 2a ce 2a 8e 00 6f 00 45 20 3f ?.????*?*?.o.E ?
...
readback:
reg0 - 03 - Tint ready
reg1 - 00 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB 0x81, bit 0x80 is "DV, data valid", bit 0x40 is "SS, sensor short", 0x20 is "SO, sensor open"
reg5 - Tint LSB 0xDB, Tint = 0x01DB = 475 * 0.0625 degC = 29.6 degC
reg6..F - stale data
</pre>

* Tint, V1, V2, TR2, VCC

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x18 b # control register: "repeat mode, V1, V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 7f 18 7f 7f 81 b9 aa d0 aa 8f 83 5e 83 5e a0 41 ???????????^?^?A
10: 01 18 01 01 01 b9 2a d0 2a 8f 03 5e 03 5e 20 41 ??????*?*??^?^ A
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x1b9*0.0625 = 27.5 degC
reg6 - V1 MSB 0xaa, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 LSB 0xd0, V1 = 0x2ad0*305.18/1000000 = 3.3447 V (correct)
reg8 - V2 MSB 0xaa, ditto
reg9 - V2 LSB 0x8f, V2 = 0x2a8f*305.18/1000000 = 3.3249 V (correct, smaller than V1)
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x5e. TR2 = 0x35e*0.0625 = 53.875 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x41, VCC = 2.5+0x2041*305.18/1000000 = 5.019 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.99 A (about right?)
</pre>

* Tint, V1-V2, TR2, VCC

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x19 b # control register: "repeat mode, V1-V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 7f 19 7f 7f 81 69 83 f2 83 f2 83 3a 83 3a a0 44 ?????i?????:?:?D
10: 01 19 01 01 01 69 03 f2 03 f2 03 3a 03 3a 20 44 ?????i?????:?: D
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x169*0.0625 = 22.5 degC
reg6 - V1 or V1-V2 MSB 0x83, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 or V1-V2 LSB 0xf2, V1-V2 = 0x3f2*19.42/1000000 = 0.0196142 V (correct, compare with V1 and V2 measured above)
reg8 - V2 or V1-V2 MSB
reg9 - V2 or V1-V2 LSB
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x3a. TR2 = 0x33a*0.0625 = 51.625 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x44, VCC = 2.5+0x2044*305.18/1000000 = 5.021 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.9807 A (about right?)
</pre>

== SFP ==

<pre>
root@cdm0:~# i2cdump 0 0x50
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 04 07 00 00 00 00 40 40 0c 00 01 3d 00 00 00 ???....@@?.?=...
10: 0c 02 00 1e 46 49 4e 49 53 41 52 20 43 4f 52 50 ??.?FINISAR CORP
20: 2e 20 20 20 00 00 90 65 46 54 4c 46 38 35 32 36 . ..?eFTLF8526
30: 50 33 42 4e 4c 20 20 20 41 20 20 20 03 52 00 9d P3BNL A ?R.?
40: 00 1a 00 00 4e 33 41 42 34 4c 56 20 20 20 20 20 .?..N3AB4LV
50: 20 20 20 20 32 30 30 33 31 39 20 20 68 f0 03 de 200319 h???
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@cdm0:~#
root@cdm0:~# i2cdump 0 0x51
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 5a 00 d8 00 55 00 e2 00 90 88 71 48 8c a0 75 30 Z.?.U.?.??qH??u0
10: 21 34 01 f4 1b 58 03 e8 31 2d 04 eb 1f 07 06 31 !4???X??1-?????1
20: 31 2d 00 64 27 10 00 9e 00 00 00 00 00 00 00 00 1-.d'?.?........
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
40: 00 00 00 00 3f 80 00 00 00 00 00 00 01 00 00 00 ....??......?...
50: 01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 b7 ?...?...?......?
60: 19 9f 80 c5 0e 17 12 c4 1f 99 00 00 00 00 30 00 ??????????....0.
70: 00 00 00 00 00 00 00 00 ff ff ff ff ff ff ff 01 ...............?
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@cdm0:~#
</pre>

== QSFP ==

<pre>
QSFP i2c enable lines, active low:

QSFP0_SEL - JENC-A 82 - PS-MIO40 - linux gpio 378
QSFP1_SEL - JENC-A 84 - PS-MIO41 - linux gpio 379
QSFP2_SEL - JENC-A 100 - PS-MIO44 - linux gpio 382
QSFP3_SEL - JENC-A 106 - PS-MIO43 - linux gpio 381 (notice 2 and 3 are out of order)
</pre>

<pre>
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
# echo 378 >> /sys/class/gpio/export ### SEL0 338+40
# echo 379 >> /sys/class/gpio/export ### SEL1 338+41
# echo 381 >> /sys/class/gpio/export ### SEL3 338+43
# echo 382 >> /sys/class/gpio/export ### SEL2 338+44
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) in hi
gpio-379 ( |sysfs ) in hi
gpio-381 ( |sysfs ) in hi
gpio-382 ( |sysfs ) in hi
root@gdm0:~# echo out >> /sys/class/gpio/gpio381/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio382/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio378/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio379/direction
root@gdm0:~#
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out lo
gpio-379 ( |sysfs ) out lo
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out lo
root@gdm0:~#
echo 1 >> /sys/class/gpio/gpio381/value
echo 1 >> /sys/class/gpio/gpio382/value
echo 1 >> /sys/class/gpio/gpio378/value
echo 1 >> /sys/class/gpio/gpio379/value
cat /sys/kernel/debug/gpio
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out hi
gpio-382 ( |sysfs ) out hi
root@gdm0:~#
root@gdm0:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#

NOTICE NOTHING AT ADDRESS 0x50
</pre>

<pre>
ENABLE QSFP0, OBSERVE IT IS AT ADDRESS 0x50

root@gdm0:~# echo 0 >> /sys/class/gpio/gpio378/value
root@gdm0:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: 50 -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#
root@gdm0:~# i2cdump -y 0 0x50
No size specified (using byte-data access)
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 0d 00 02 8e 00 00 01 00 00 05 55 00 05 00 00 00 ?.??..?..?U.?...
10: 00 00 00 00 00 00 1e b8 00 00 81 a0 00 00 00 00 ......??..??....
20: 00 00 12 12 00 01 00 01 00 01 0e e0 0f 20 0e e0 ..??.?.?.???? ??
30: 00 00 1f a0 1e 3b 1f 72 00 01 00 00 00 00 00 00 ..???;?r.?......
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
60: 00 00 00 00 00 00 00 00 00 00 1f 00 00 00 08 00 ..........?...?.
70: 00 00 00 00 00 00 00 ff ff ff ff ff ff ff ff 00 ................
80: 0d 00 0c 04 00 00 00 40 40 02 d5 05 67 00 00 96 ?.??...@@???g..?
90: 00 00 c8 00 46 49 4e 49 53 41 52 20 43 4f 52 50 ..?.FINISAR CORP
a0: 20 20 20 20 07 00 90 65 46 54 4c 34 31 30 51 44 ?.?eFTL410QD
b0: 34 43 20 20 20 20 20 20 41 20 42 68 07 d0 00 43 4C A Bh??.C
c0: 00 07 0f de 58 37 39 41 43 30 52 20 20 20 20 20 .???X79AC0R
d0: 20 20 20 20 32 32 30 33 30 39 20 20 3c 10 00 9e 220309 <?.?
e0: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@gdm0:~#
</pre>

== ethernet mac eeprom ==

* correct chip with 84-bit ethernet mac address

<pre>
root@cdm1:~# i2cdump 0 0x53
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
root@cdm1:~#

root@cdm1:~# i2cdump 0 0x5b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00 ????4??2?K?.?...
90: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c ..........??=?Q<
a0: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00 ????4??2?K?.?...
b0: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c ..........??=?Q<
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@cdm1:~#
</pre>

* wrong "602" chip with 64-bit IPv6 address

<pre>
root@cdm0:~# i2cdump 0 0x53
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
root@cdm0:~# i2cdump 0 0x5b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00 ???????Q?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e ........??=..??.
a0: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00 ???????Q?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e ........??=..??.
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@cdm0:~#
</pre>

= read ethernet mac address from i2c =

(this code is copied from uboot command line i2c code)

in uboot sources board/xilinx/common/board.c replace original function with this:

<pre>
// special code to read ethernet MAC address from the DS-DM-Rev0 board. K.O. Sep-2022

int zynq_board_read_rom_ethaddr(unsigned char *ethaddr)
{
struct udevice *bus;
int ret;
int busnum = 0;

ret = uclass_get_device_by_seq(UCLASS_I2C, busnum, &bus);
if (ret) {
printf("%s: No bus %d\n", __func__, busnum);
return ret;
}

int chip_addr = 0x5B;

struct udevice *dev;

ret = i2c_get_chip(bus, chip_addr, 1, &dev);
if (ret) {
printf("%s: Bus %d no chip 0x%02x\n", __func__, busnum, chip_addr);
return ret;
}

int dev_addr = 0x98;

unsigned char data[8];

ret = dm_i2c_read(dev, dev_addr, data, 8);

if (ret) {
printf("%s: Bus %d chip 0x%02x read error %d\n", __func__, busnum, chip_addr, ret);
return ret;
}

printf("%s: Bus %d chip 0x%02x addr 0x%02x read: 0x %02x %02x %02x %02x %02x %02x %02x %02x\n", __func__, busnum, chip_addr, dev_addr, data[0], data[1], data[2], data[3], data[4], data[5], data[6], data[7]);

// see Atmel-8807-SEEPROM-AT24MAC402-602-Datasheet.pdf

if (data[0] == 0) {
// eiu-48 chip
ethaddr[0] = data[2];
ethaddr[1] = data[3];
ethaddr[2] = data[4];
ethaddr[3] = data[5];
ethaddr[4] = data[6];
ethaddr[5] = data[7];
} else {
// eiu-64 chip
ethaddr[0] = data[0];
ethaddr[1] = data[1];
ethaddr[2] = data[2];
ethaddr[3] = data[5];
ethaddr[4] = data[6];
ethaddr[5] = data[7];
}

printf("%s: ethaddr %02x:%02x:%02x:%02x:%02x:%02x\n", __func__, ethaddr[0], ethaddr[1], ethaddr[2], ethaddr[3], ethaddr[4], ethaddr[5]);

return ret;
}
</pre>

also this should have worked if i2c_xxx() functions were enabled in uboot:
<pre>
i2c_set_bus_num(0);
i2c_probe(0x5b);
i2c_read(0x5b, 0x9a, ethaddr, 6);
</pre>

= read ethernet mac address from i2c (SHOULD WORK) =

from: https://stackoverflow.com/questions/43637540/is-there-linux-or-u-boot-support-to-read-a-mac-address-from-a-chip-at-startup

<pre>
#ethernet related setup
setup_eth=run readmac buildmac
#read mac address from eeprom
readmac=i2c dev 0; i2c read 50 FA.1 6 $loadaddr
#build the ethaddr variable
#not very nice, but does the job
buildmac=\
e=" "; sep=" " \
for i in 0 1 2 3 4 5 ; do\
setexpr x $loadaddr + $i\
setexpr.b b *$x\
e="$e$sep$b"\
sep=":"\
done &&\
setenv ethaddr $e
</pre>

= read ethernet mac address from i2c (DOES NOT WORK) =

this method does not work: inside board/xilinx/common/board.c:int zynq_board_read_rom_ethaddr(unsigned char *ethaddr), uclass_get_device_by_ofnode(UCLASS_I2C_EEPROM, eeprom, &dev) returns ENODEV and read from i2c does not happen. K.O. Sep-2022

Read:
* https://support.xilinx.com/s/question/0D52E00006hpKlsSAE/uboot-and-ethernet-mac-address-from-eeprom?language=en_US (last answer)
* https://github.com/Xilinx/u-boot-xlnx/blob/master/board/xilinx/common/board.c (zynq_board_read_rom_ethaddr())

Note:
* 0x5B is the i2c chip address
* 0x9A is the data offset inside the chip, see datasheet or i2c read dump above.

Edit:
* emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/u-boot/files/platform-top.h
<pre>
#include <configs/xilinx_zynqmp.h>
#include <configs/platform-auto.h>
//#define CONFIG_I2C_EEPROM
//#define CONFIG_SYS_I2C_EEPROM_ADDR 0x5b
//#define CONFIG_SYS_I2C_EEPROM_ADDR_OVERFLOW 0x0
#define CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET 0x9A
#error HERE!
</pre>
* emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi
<pre>
/include/ "system-conf.dtsi"
/ {
chosen {
xlnx,eeprom = &eeprom;
};
};

&i2c0 {
eeprom: eeprom@5b { /* u88 */
compatible = "atmel,24mac402";
reg = <0x5b>;
};
};
</pre>
* components/yocto/workspace/sources/u-boot-xlnx/configs/xilinx_zynqmp_virt_defconfig
<pre>
CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET=0x9A
</pre>

= enable VX clock =

<pre>
devmem 0x80010010 32 0x8; sleep 1; devmem 0x80010010 32 0x0;
si5394-i2c-file /media/sd-mmcblk1p1/00_freerun.txt 0 0x6b
</pre>

= clock chip configuration =

file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt:

* VCO is 14 GHz
* Tvco is 71.43 ps
* N0 divider is 14, frequency is 1000 MHz
* out0 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz
* out1 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
* out2 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
* out3 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz is the feedback for zero delay

= clock chip monitoring =

from si5395-94-92-family.pdf:
* reg 0x1: page select, set to 0 or set to 5 to read 0x53F
* reg 0x2: 0x94
* reg 0x3: 0x53 -> device is a si5394
* reg 0xC: LOSXAXB
* reg 0xD: LOS and OOF for the 4 clock inputs
* reg 0xE: LOL and HOLD
* reg 0xF: CAL_PLL
* reg 0x11: sticky bits for reg 0xC
* reg 0x12: sticky bits for reg 0xD
* reg 0x13: sticky bits for reg 0xE
* reg 0x14: sticky bits for reg 0xF
* reg 0x1C: device reset
* reg 0x1E: low power, hard reset, SYNC
* reg 0x507: currently selected input clock
* reg 0x52A: input clock select
* reg 0x535: FORCE_HOLD
* reg 0x53F: HOLD_HIST_VALID and FASTLOCK_STATUS

= arm and linux benchmark =

memory benchmark:

<pre>
daq13$ arm-linux-gnueabi-gcc -o memcpy.armv7 memcpy.cc -march=armv7 -static -O2
scp memcpy.armv7 to ...
root@gdm-cdm:~# ./memcpy.armv7
memcpy 1 KiBytes: 1288 MB/sec
memcpy 2 KiBytes: 1924 MB/sec
memcpy 4 KiBytes: 2554 MB/sec
memcpy 8 KiBytes: 3054 MB/sec
memcpy 16 KiBytes: 3262 MB/sec
memcpy 32 KiBytes: 3250 MB/sec
memcpy 64 KiBytes: 3456 MB/sec
memcpy 128 KiBytes: 3556 MB/sec
memcpy 256 KiBytes: 3780 MB/sec
memcpy 512 KiBytes: 3795 MB/sec
memcpy 1024 KiBytes: 3789 MB/sec
memcpy 2048 KiBytes: 3729 MB/sec
memcpy 4096 KiBytes: 3717 MB/sec
memcpy 8192 KiBytes: 3687 MB/sec
memcpy 16384 KiBytes: 3632 MB/sec
memcpy 32768 KiBytes: 3529 MB/sec
memcpy 65536 KiBytes: 3318 MB/sec
memcpy 131072 KiBytes: 2893 MB/sec
root@gdm-cdm:~#
</pre>

ethernet receive:

<pre>
daq13:bin$ ./ttcp -t -s -n 100000 10.0.0.24
ttcp-t: buflen=8192, nbuf=100000, align=16384/0, port=5001 tcp -> 10.0.0.24
ttcp-t: socket
ttcp-t: connect
ttcp-t: 819200000 bytes in 7.25 real seconds = 110358.39 KB/sec +++
ttcp-t: 100000 I/O calls, msec/call = 0.07, calls/sec = 13794.80
ttcp-t: 0.0user 0.2sys 0:07real 3% 0i+0d 760maxrss 0+2pf 1461+31csw
daq13:bin$

root@gdm-cdm:~# ./ttcp.armv7 -r -s
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 10.0.0.25
ttcp-r: 819200000 bytes in 7.27 real seconds = 110098.22 KB/sec +++
ttcp-r: 212040 I/O calls, msec/call = 0.04, calls/sec = 29181.53
ttcp-r: 0.1user 5.7sys 0:07real 81% 0i+0d 584maxrss 0+2pf 125601+2699csw
root@gdm-cdm:~#
</pre>

ethernet transmit:

<pre>
root@gdm-cdm:~# ./ttcp.armv7 -t -s -n 100000 10.0.0.25
ttcp-t: buflen=8192, nbuf=100000, align=16384/0, port=5001 tcp -> 10.0.0.25
ttcp-t: socket
ttcp-t: connect
ttcp-t: 819200000 bytes in 6.95 real seconds = 115078.69 KB/sec +++
ttcp-t: 100000 I/O calls, msec/call = 0.07, calls/sec = 14384.84
ttcp-t: 0.0user 0.7sys 0:06real 11% 0i+0d 584maxrss 0+2pf 1162+1017csw
root@gdm-cdm:~#

daq13:bin$ ./ttcp -r -s
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 10.0.0.24
ttcp-r: 819200000 bytes in 6.97 real seconds = 114841.84 KB/sec +++
ttcp-r: 161335 I/O calls, msec/call = 0.04, calls/sec = 23160.01
ttcp-r: 0.0user 1.9sys 0:06real 28% 0i+0d 760maxrss 0+2pf 80646+51csw
daq13:bin$
</pre>

= Install Xilinx tools =

* install Vivado 2020.2
<pre>
login at https://www.xilinx.com/myprofile.html
go to "Downloads"
go to archive,
find 2020.2
download Xilinx_Unified_2020.2_1118_1232_Lin64.bin
sh ./Xilinx_Unified_2020.2_1118_1232_Lin64.bin
banner window should open with spinner "downloading installation data"
"a newer version is available" -> say "continue"
next
"select install type" window:
provide email and password,
select "download image"
select directory /home/olchansk/Xilinx/Downloads/2020.2\
select "linux" and "full image"
next
download summary: space required 38.52 Gbytes
download
installation progress
downloading spinner, 16 M/s 47 minutes...
"download image has been created successfully". Ok.
check contents of /home/olchansk/Xilinx/Downloads/2020.2
ls -l /home/olchansk/Xilinx/Downloads/2020.2
total 67
drwxr-xr-x 2 olchansk users 9 Sep 1 16:22 bin
drwxr-xr-x 3 olchansk users 15 Sep 1 16:23 data
drwxr-xr-x 4 olchansk users 4 Sep 1 16:22 lib
drwxr-xr-x 2 olchansk users 644 Sep 1 16:22 payload
drwxr-xr-x 2 olchansk users 7 Sep 1 16:22 scripts
drwxr-xr-x 4 olchansk users 4 Sep 1 16:22 tps
-rwxr-xr-x 1 olchansk users 3256 Nov 18 2020 xsetup
daq13:2020.2$
./xsetup
spinned loading installation data
xilinx design tools 2022.1 now available -> say continue
"welcome" -> next
"select product" -> vivado -> next -> vivado hl system edition -> next
select devices: only zynq ultrascale+ mpsoc -> next
select destination: /opt/Xilinx (as root, mkdir /opt/Xilinx, chmod olchansk.users /opt/Xilinx)
install ...
complete
move /home/olchansk/Xilinx/Downloads/2020.2 to /daq/daqstore/olchansk/Xilinx/Downloads/
</pre>
* install petalinux 2020.2
<pre>
./xsetup
"a newer version is available" -> say "continue"
next
"select product to install" -> select Petalinux (Linux only) -> next
"select destination directory" -> select "/opt/Xilinx" (disk space required 2.64 GB) -> next
"summary" -> install ...
error about missing /tmp/tmp-something files
"installation completed successfully" (hard to dismiss, "ok" button is partially cut-off)
done?
I think it failed, /opt/Xilinx/PetaLinux/2020.2/bin is empty except for petalinux-v2020.2-final-installer.run
try to run it by hand, same error about /tmp/tmp-something files. strange...
notice it complains about "truncate", which truncate finds ~/bin/truncate, get rid of it,
try again
now complains about missing texinfo and zlib1g:i386
apt install texinfo -> ok
apt install zlib1g:i386 -> installs bunch of gcc stuff -> ok
try again
reports "already installed" -> delete /opt/Xilinx/.xinstall/PetaLinux_2020.2/, delete entries in ~/.Xilinx/registry/installedSW.xml
try again
success
</pre>
* install vivado 2022.1 and petalinux 2022.1 - everything is pretty much the same

= Petalinux =

* cd PetaLinux_GDM_CDM
* petalinux-config
* enable i2c MAC address and DHCP
<pre>
git clone https://bitbucket.org:/team-ds-dm/ds-dm-u-boot-xlnx.git
cd ds-dm-u-boot-xlnx
git checkout ds-dm-u-boot-xlnx

linux-components ->
uboot -> ext-local-src
external u-boot local source -> ds-dm-u-boot-xlnx (path to the customized uboot git repository)
</pre>
* enable DHCP
<pre>
Subsystem AUTO Hardware Settings -> Ethernet Settings
randomize MAC address -> NO
ethernet mac address -> leave empty
obtain ip address automatically -> YES
</pre>
* set hostname and product names
<pre>
Firmware Version Configuration ->
Host name -> "ds-dm"
Product name -> "Petalinux_GDM_CDM"
</pre>
* configure linux kernel
<pre>
petalinux-config -c kernel
</pre>
* enable NFS-Root
<pre>
petalinux-config
Image Packaging Configuration > Root File System Type -> set to NFS
Location of NFS root directory set to "/nfsroot"

petalinux-config -c kernel
Networking support > IP: kernel level configuration
enable DHCP, BOOTP, RARP
File systems > Network file systems > Root file systems on NFS
</pre>
* manually fix linux kernel command line:
<pre>
grep nfsroot PetaLinux_GDM_CDM/project-spec/configs/config
edit CONFIG_SUBSYSTEM_BOOTARGS_GENERATED to read
earlycon console=ttyPS0,115200 clk_ignore_unused panic=60 root=/dev/nfs nfsroot=/nfsroot/%s ip=dhcp rw
</pre>
* check configuration in
** PetaLinux_GDM_CDM/project-spec/configs/config
** PetaLinux_GDM_CDM/project-spec/configs/rootfs_config
** PetaLinux_GDM_CDM/components/plnx_workspace/device-tree/device-tree/system-conf.dtsi

= JTAG server =

localhost:3121

= ds20k block =

<pre>
module ds20k
(
// CLOCK INPUTs
input wire clk,
input reset, // pulse for power-up reset
input wire pll_is_locked, // clock cleaner PLL is locked to selected input clock

// REGISTER_DATA

input wire [255:0] [31:0] register_data_in,
output reg [255:0] [31:0] register_data_out,
input wire register_write_strobe, // pulse when AXI write transaction puts new data in register_data_in
input wire register_read_ack, // pulse after AXI read transaction captures data from register_data_out, used to read from FIFO

// GDM QSFP FIBER LINKS
output reg [11:0] [15:0] qsfp_tx_data,
output reg [11:0] [1:0] qsfp_tx_ctrl,
input wire [11:0] [15:0] qsfp_rx_data_error, // not sure what this is.
input wire [11:0] [15:0] qsfp_rx_data,
input wire [11:0] [1:0] qsfp_rx_ctrl,
input wire [11:0] qsfp_rx_is_good, // single bit indicating that RX link is up and data is good.

// CDM SFP FIBER LINKS
output reg [15:0] sfp_tx_data,
output reg [1:0] sfp_tx_ctrl,
input wire [15:0] sfp_rx_data,
input wire [1:0] sfp_rx_ctrl,
input wire sfp_rx_is_good, // single bit indicating that RX link is up and data is good.

// VX_RXs
input wire [3:0] vx1_rx,
input wire [3:0] vx2_rx,
input wire [3:0] vx3_rx,
input wire [3:0] vx4_rx,
input wire [3:0] vx5_rx,
input wire [3:0] vx6_rx,
input wire [3:0] vx7_rx,
input wire [3:0] vx8_rx,
input wire [3:0] vx9_rx,
input wire [3:0] vx10_rx,
input wire [3:0] vx11_rx,
input wire [3:0] vx12_rx,

// VX_TXs
output reg [2:0] vx1_tx_out,
output reg [2:0] vx2_tx_out,
output reg [2:0] vx3_tx_out,
output reg [2:0] vx4_tx_out,
output reg [2:0] vx5_tx_out,
output reg [2:0] vx6_tx_out,
output reg [2:0] vx7_tx_out,
output reg [2:0] vx8_tx_out,
output reg [2:0] vx9_tx_out,
output reg [2:0] vx10_tx_out,
output reg [2:0] vx11_tx_out,
output reg [2:0] vx12_tx_out,

// remove input wire gdm_trg,
// remove input wire gdm_tsm,

// LEMO INPUTs
input wire [4:1] ext_in_lv_async, // direct connection to LEMO connectors, not clocked

// LEMO OUTPUTs
output reg [2:1] ext_out, // direct connection to LEMO connectors, not clocked

// FRONT PANEL LEDs
output reg [3:0] fp_led_out, // direct connection to LEMO connectors, not clocked

// trigger and tsm output
// remove output reg trg_out,
// remove output reg tsm_out
);
</pre>

= world view =

Note:
* red lines: clocks
* green lines: AXI/Avalon packet streams
* blue lines: serial data

[[File:GDM-CDM-VX.drawio.svg]]

== description ==

same thing, in words:

<pre>
detector
digitizer, 125 MHz
digital filter
digital discriminator
hit map, 64 bits at 125 MHz (could be 250 MHz, filter and discriminator clock)
packetizer, 64 bits -> id, timestamp, 8x 8-bit words, eop
8/10 serializer, 12.5 MHz parallel in, 125 MHz serial out
lvds line to CDM vx_rx
BBB: also VX busy to lvds line to CDM vx_rx, do not want to depend on serial comm for vx busy, do not want to inject async data into the hit map packet stream
--- CDM
lvds line capture in IBUF register, vxN_rx clock, 125 MHz clock, from PLL with adjustable phase, scan phase to find sweet spot
vx rx 10/8 deserializer, vxN_rx clock, stobes out parallel data every 10 clocks at 12.5 MHz
vx rx phase transfer from vxN_rx clock to main CDM clock, have 10 clocks for transfer to happen
from here everything is on the CDM main clock
vx rx packet adapter, 12.5 MHz 8-bit packets to 125 MHz 16-bit packets, internal FIFO to avoid data overrun
sfp tx mux, all VX packet streams into one CDM sfp tx packet stream
BBB: capture VX RX busy to CDM main clock (IBUF register), grand-or becomes vx_rx_busy
sfp tx packetizer (data,eop,vx_rx_busy -> data,k)
sfp tx 16/20 serializer, 16 bit at 125 MHz in, 2500 MHz serial out (20 bits at 125 MHz)
sfp tx fiber link to GDM, 2 Gigabits/sec
--- GDM
qsfp rx deserializer, 16 bit at 125 MHz output on main GDM clock
qsfp rx depacketizer (data,k -> packet data,eop; qsfp_rx_busy)
qsfp rx demux, hit map packets routed to GDM trigger logic block
GDM trigger logic block looks at hit map, generates yes/no trigger decision, encodes it as a trigger packet
TSM generator encodes GPS time data as a TSM packet
qsfp tx mux - trigger packets, tsm packets, etc to qsfp tx packet stream
qsfp tx packetizer (data,eop;trg,tsm,bsy -> data,k)
NB: the same tx data is sent to all 12 qsfp tx ports, to make sure we do not accidentally desync the CDMs.
qsfp tx 16/20 serializer, 16-bit at 125 MHz to 20-bit at 250 MHz to 2500 MHz serial out
qsfp tx fiber link to CDM, 2 Gigabits/sec
--- CDM
sfp rx 20/16 deserializer, 16-bit at 125 MHz
sfp rx depacketizer, (data,k -> data,eop;trg_in,tsm_in,bsy_in)
sfp rx demux (in reality, noop, all packets go to same place, vx tx)
vx tx mux (packets from GDM, packets with simulated waveforms from CDM midas frontend via AXI FIFO)
vx tx packet adapter 16-bit at 125 MHz to 8-bit at 12.5 MHz strobed every 10 clocks. NB: most important, this 10-clock strobe runs in sync between all CDMs!
vx tx phase transfer from main CDM clock to vxN_tx clock, have 10 clocks for transfer to happen
from here we run on the vxN_tx clock
vx txN serializer, 8-bit at 12.5 MHz to 125 MHz serial
vx_txN OBUF register, vxN_rx clock, 125 MHz clock, from PLL with adjustable phase, scan phase to find sweet spot
lvds line to vx
BBB: bsy_in from GDM is converted from pulse to level, goes out lvds line to vx, sync to vxN_tx clock
TTT: trg_in from GDM is a pulse, does out lvds line to vx, sync to vxN_tx clock.
NB: tsm is always a packet, bsy is always a signal (no packet), trg can be a packet or signal.
--- VX
lvds data captured by 125 MHz ADC sampling clock (CAEN base firmware logic)
lvds data connected to Yair's block
10/8 deserializer
depacketizer (data,k -> data,eop) to avoid accidental desync, we do not send any no trg, no tsm, no bsy K-codes.
demux
trigger packets go their way (4x 64 bit words of data go to event header: timestamps, hitmap data)
tsm packets go their way (4x 64 bit words of data go to tsm event header: timestamps, GPS time data)
BBB: bsy from lvds line stops waveform acquisition
TTT: trg from lvds line cause waveform acquisition, same as trg packet, but has no timestamp and other data attached to it.
</pre>

= Firmware registers =

== Block 0 ==

<pre>
0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
</pre>

== Block 1 ==

== Block 2 ==

== Block 3 ==

=== DS20k block register map ===

* busybox devmem 0x80013000 32

<pre>
reg | version | xDM | xx | description
0 | 0x20230731 | ALL | RO | ds20k version
0 | 0x20240118 | ALL | RW | ds20k version and command
1 | 0x20230731 | ALL | RW | scratch read/write register
2 | 0x20230731 | ALL | RW | configure inputs and outputs
3 | 0x20230731 | ALL | RW | FP_LED mux
4 | 0x20230731 | ALL | RW | EXT_OUT mux
5 | 0x20230731 | ALL | RO | VX_RX state
6 | 0x20230731 | ALL | RO | VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state
7 | 0x20230731 | ALL | RW | LED_OUT, EXT_OUT, VX_TX outputs
8 | 0x20230731 | ALL | RW | VX_TX mux and config
9 | 0x20230731 | ALL | RW | trigger config
10 | 0x20231013 | ALL | RO | status register
11 | 0x20230731 | ALL | RO | trigger counter
12 | 0x20230731 | ALL | RO | time slice marker counter
13 | 0x20240814 | ALL | RO | GPS 1pps period, 125 MHz
14 | 0x20240814 | ALL | RO | Ru clock 1pps period, 125 MHz
15 | 0x20230811 | CDM | RO | SFP RX status
16 | 0x20230811 | CDM | RW | SFP TX control
17 | 0x20230811 | GDM | RO | QSFP RX data 0, 1
18 | |. |. | 2, 3
19 | |. |. | 4, 5
20 | |. |. | 6, 7
21 | | |. | 8, 9
22 | |. |. | 10, 11
23 | 0x20230811 | GDM | RW | QSFP TX control
24 | 0x20231013 | ALL | RW | trigger pulser period
25 | 0x20231013 | ALL | RW | trigger pulser burst control
26 | 0x20231013 | ALL | RW | tsm pulser period
27 | 0x20231204 | ALL | RW | data fifo CPU to FPGA
28 | 0x20231204 | ALL | RW | packet loopback control
28 | 0x20240118 | ALL | RW | data fifo FPGA to CPU
29 | 0x20231208 | GDM | RW | bitmap of active qsfp ports
29 | 0x20240118 | ALL | RW | packet loopback control
29 | 0x20240510 | ALL | RW | packet routing
30 | 0x20231208 | GDM | RO | qsfp link status ports 0..7
31 | 0x20231208 | GDM | RO | qsfp link status ports 8..11
32 | 0x20240118 | GDM | RW | enabled QSFP ports
33 | 0x20240118 | CDM | RW | enabled VX ports
34,35 | 0x20240118 | ALL | RO | time stamp 64 bits
36,37 | 0x20240118 | ALL | RO | old time stamp
38,39,40 | 0x20240118 | CDM | RO | VX busy counters
41,42,43 | 0x20240118 | GDM | RO | QSFP busy counters
44 | 0x20240118 | CDM | RO | cdm_bsy_up_counter and cdm_bsy_pulse_counter
45 | 0x20240118 | CDM | RO | cdm_veto_pulse_counter and cdm_veto_up_counter
46 | 0x20240118 | GDM | RO | gdm_bsy_pulse_counter and gdm_bsy_up_counter
47 | 0x20240118 | GDM | RO | gdm_bsy_refresh_counter
48 | 0x20240118 | GDM | RO | gdm_veto_up_counter and gdm_veto_pulse_counter
49,50,51 | 0x20240424 | CDM | RO | VX RX serial data monitor, 8 bits per VX port
52,53 | 0x20240424 | CDM | RO | VX serial link status, 4 bits per VX port
54 | 0x20240424 | CDM | RO | VX TX serial data monitor
55 | 0x20240430 | ALL | RO | QSFP, SFP and VX link loss counters
56 | 0x20240510 | ALL | RO | sfp_rx_packet_counter
57 | same | ALL | RO | sfp_tx_packet_counter
58 | same | ALL | RO | qsfp_rx_packet_counter[0]
59 | same | ALL | RO | qsfp_tx_packet_counter
60 | same | ALL | RO | vx_rx_packet_counter[0]
61 | same | ALL | RO | vx_tx_packet_counter
62 | 0x20240719 | CDM | RO | cdm_hitmap_period, ports 0, 1
63 | same | CDM | RO | ports 2, 3
64 | same | CDM | RO | ports 4, 5
65 | same | CDM | RO | ports 6, 7
66 | same | CDM | RO | ports 8, 9
67 | same | CDM | RO | ports 10, 11
68 | 0x20240814 | ALL | RW | GPS control and status
</pre>

=== Register 0 0x80013000 ds20k version ===

on read: ds20k version 0xYYYYMMDD

on write:
* 0x00 - noop - as of version 0x20240118, write a zero after writing a command
* 0x01 - CMD_reset - reset logic to good state
* 0x02 - CMD_arm_ts - arm timestamp reset
* 0x03 - CMD_trg - issue a trigger
* 0x04 - CMD_tsm - issue a tsm
* 0x05 - CMD_vx_rx_reset - reset the VX receive path
* 0x06 - CMD_vx_tx_reset - reset the VX transmit path
* 0x07 - CMD_hitmap_trg - generate a hitmap trigger and data packet
* 0x08 - CMD_trg_pulser_reset - reset the trigger pulser
* 0x09 - CMD_tsm_pulser_reset - reset the tsm pulser

=== Register 1 0x80013004 scratch ===

scratch read-write register

=== Register 2 0x80013008 input and output config ===

<pre>
bit | version | fpga name | description
0 | ALL | lemo_enable | enable LEMO input 1
1 | | | 2
2 | | | 3
3 | | | 4
4 | ALL | lemo_invert | invert LEMO input 1
5 | | | 2
6 | | | 3
7 | | | 4
9 | ALL | ext_out_disable | disable LEMO output 1
10 | | | 2
11 | ALL | ext_out_invert | invert LEMO output 1
12 | | | 2
</pre>

=== Register 3 0x8001300C FP_LED control ===

<pre>
wire [15:0] led_out_mux_sel = register_data_in[3][15:0];
wire [3:0] led_out_invert = register_data_in[3][19:16];
</pre>

<pre>
led_out_mux_sel is 4 groups (one per LED) of 4 bits (choice 0..15):
mux | version | fpga name | description
0 | ALL | | power on default
1 | ALL | led_out_reg | register 7 bits
2 | 0x20231013 | pll_locked | clock chip PLL is locked
3 | 0x20231013 | sfp_link_status | SFP link is good
3 | 0x20240118 | sfp_link_status or qsfp_tx_link_rx_status | SFP/QSFP link is good
4 | ALL | lemo_in_sync[1] | LEMO input 1
5 | ALL | lemo_in_sync[2] | LEMO input 2
6 | ALL | lemo_in_sync[3] | LEMO input 3
7 | ALL | lemo_in_sync[4] | LEMO input 4
8 | ALL | ext_out[1] | LEMO output 1
9 | ALL | ext_out[2] | LEMO output 2
A | 0x20231013 | trg_in | trigger
B | 0x20231013 | tsm_in | time slice marker
C | 0x20240118 | gdm_bsy | GDM busy: OR of all CDM busy
D | 0x20240118 | cdm_bsy | CDM busy: OR of all VX busy
E | 0x20240118 | cdm_veto | GDM busy -> GDM veto -> CDM veto -> VX trigger veto
F | ALL | | fixed logic level 1
</pre>

=== Register 4 0x80013010 LEMO OUT control ===

<pre>
wire [7:0] ext_out_mux_sel = register_data_in[4][7:0];
</pre>

<pre>
ext_out_mux_sel is 2 groups (one per LEMO) of 4 bits (choice 0..15):

mux | version | fpga name | description
0 | ALL | | power on default
1 | ALL | ext_out_reg | register 7
2 | ALL | trg_pulser | pulser trigger
3 | 0x20240724 | vx1_tx_out[2] | vx1 serial data out
4 | 0x20240724 | vx1_rx[1] | vx1 serial data in
5 | 0x20240724 | vx_rx_iob[0] | vx1 serial data in captured by IOB register
6 | ALL | lemo_in_async[1]| test synchronizer
7 | ALL | lemo_in_sync[1] | test synchronizer
8 | 0x20240724 | trg_in_pulse | trigger signal
9 | 0x20240724 | tsm_in_pulse | time slice marker signal
A | 0x20240118 | cdm_bsy | CDM busy from VX
B | 0x20240118 | gdm_bsy | GDM busy from CDM
C | 0x20240118 | cdm_veto | veto from GDM to CDM to VX
D | 0x20240118 | vx1_rx[1] | serial data VX to CDM
E | not used (sink)
F | ALL | 1 | fixed logic level 1
</pre>

=== Register 5 0x80013014 VX_RX status ===

<pre>
assign register_data_out[5] = {
vx8_rx[3], vx8_rx[2], vx8_rx[1], vx8_rx[0],
vx7_rx[3], vx7_rx[2], vx7_rx[1], vx7_rx[0],
vx6_rx[3], vx6_rx[2], vx6_rx[1], vx6_rx[0],
vx5_rx[3], vx5_rx[2], vx5_rx[1], vx5_rx[0],
vx4_rx[3], vx4_rx[2], vx4_rx[1], vx4_rx[0],
vx3_rx[3], vx3_rx[2], vx3_rx[1], vx3_rx[0],
vx2_rx[3], vx2_rx[2], vx2_rx[1], vx2_rx[0],
vx1_rx[3], vx1_rx[2], vx1_rx[1], vx1_rx[0]
};
</pre>

=== Register 6 0x80013018 VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state ===

<pre>
assign register_data_out[6] = {
vx2_tx_out[2], vx2_tx_out[1], vx2_tx_out[0], vx1_tx_out[2],
vx1_tx_out[1], vx1_tx_out[0], ext_out[2], ext_out[1],
fp_led_out[3], fp_led_out[2], fp_led_out[1], fp_led_out[0],
ext_in_lv[4], ext_in_lv[3], ext_in_lv[2], ext_in_lv[1],
vx12_rx[3], vx12_rx[2], vx12_rx[1], vx12_rx[0],
vx11_rx[3], vx11_rx[2], vx11_rx[1], vx11_rx[0],
vx10_rx[3], vx10_rx[2], vx10_rx[1], vx10_rx[0],
vx9_rx[3], vx9_rx[2], vx9_rx[1], vx9_rx[0]
};
</pre>

<pre>
bit | version | fpga name | description
0 | ALL | vx9_rx | VX_RX
1 | | |
2 | | |
3 | | |
4 | ALL | vx10_rx | VX_RX
5 | | |
6 | | |
7 | | |
8 | ALL | vx11_rx | VX_RX
9 | | |
10 | | |
11 | | |
12 | ALL | vx12_rx | VX_RX
13 | | |
14 | | |
15 | | |
16 | ALL | ext_in_lv | LEMO inputs
17 | | |
18 | | |
19 | | |
20 | ALL | FP_LED | FP_LEDs
21 | | |
22 | | |
23 | | |
24 | ALL | ext_out[1] | LEMO outputs
25 | | ext_out[2] |
26 | ALL | vx1_tx | VX1_TX
27 | | |
28 | | |
29 | ALL | vx2_tx | VX2_TX
30 | | |
31 | | |
</pre>

=== Register 7 0x8001301C LED_OUT, EXT_OUT, VX_TX outputs ===

<pre>
wire [3:0] led_out_reg = register_data_in[7][3:0];
wire [2:1] ext_out_reg = register_data_in[7][5:4];
// register_data_in[7][6];
// register_data_in[7][7];
wire [7:0] vx_tx_out_reg = register_data_in[7][15:8];
</pre>

<pre>
bit | version | fpga name | description
0 | ALL | led_out_reg | FP_LED 1
1 | | | 2
2 | | | 3
3 | | | 4
4 | ALL | ext_out_reg | LEMO OUT 1
5 | | | 2
6 | - | |
7 | - | |
8 | ALL | vx_tx_out_reg | VX1_TX 0
9 | | | 1
10 | | | 2
11 | | | -
12 | ALL | | VX2_TX 0
13 | | | 1
14 | | | 2
15 | | | -
16 | - | |
17 | | |
18 | | |
19 | | |
20 | | |
21 | | |
22 | | |
23 | | |
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 8 0x80013020 VX_TX config ===

<pre>
wire [3:0] vx_tx_mux_sel = register_data_in[8][3:0];
vx_tx_loopback = register_data_in[8][31];
</pre>

<pre>
vx_tx_mux_sel is 4 bits (choice 0..15):
0 - power on default, control by vx_tx_out_reg
1 - GDM
2 - CDM
3 - pulser loopback test
4 - pulser loopback test
5 - 62.5 MHz output
6 - 125 MHz output
7 - trg, tsm, serial
8 - trg, veto, serial
9 - GPS box control (ds20k rev 0x20240814)
10
11
12
13
14
15 - production config: trg, veto, serial
</pre>

=== Register 9 0x80013024 trigger control ===

from version 0x20240724

<pre>
wire [15:0] trg_src_mask = register_data_in[9][15:0];
wire [31:16] tsm_src_mask = register_data_in[9][31:16];
</pre>

<pre>
wire [15:0] trg_src_bits =
{
1'b0,
1'b0,
1'b0,
1'b0,

1'b0, // gdm_hitmap_trigger,
cdm_hitmap_trigger,
sfp_rx_tsm,
sfp_rx_trg,

sfp_rx_data[1],
sfp_rx_data[0],
tsm_pulser,
trg_pulser,

lemo_in_sync[4],
lemo_in_sync[3],
lemo_in_sync[2],
lemo_in_sync[1]
};

wire [15:0] trg_bits = trg_src_bits & trg_src_mask;
wire [15:0] tsm_bits = trg_src_bits & tsm_src_mask;
</pre>

before that:

<pre>
wire [7:0] trg_src_mask = register_data_in[9][7:0];
wire [7:0] tsm_src_mask = register_data_in[9][15:8];
wire trg_pulser_enable = register_data_in[9][16];
wire tsm_pulser_enable = register_data_in[9][17];
wire trg_software = register_data_in[9][18];
wire tsm_software = register_data_in[9][19];
// bits 20:31 not used
</pre>

<pre>
wire [7:0] xxx_src_bits =
{
sfp_rx_data[1],
sfp_rx_data[0],
tsm_pulser & tsm_pulser_enable,
trg_pulser & trg_pulser_enable,
lemo_in_sync[4],
lemo_in_sync[3],
lemo_in_sync[2],
lemo_in_sync[1]
};
</pre>

<pre>
trg_src_mask and tsm_src_mask bits:
0 - LEMO IN 1
1 - LEMO IN 2
2 - LEMO IN 3
3 - LEMO IN 4
4 - trg_pulser
5 - tsm_pulser
6 - sfp_rx_data[0] // to become sfp_trg_in, selected from sfp_rx_data[0], cdm_rx trg_out and gdm trigger packet
7 - sfp_rx_data[1] // to become sfp_tsm_in, selected from sfp_rx_data[1], cdm_rx tsm_out and gdm tsm packet
</pre>

=== Register 10 0x80013028 status register ===

<pre>
bit | version | fpga name | description
0 | 0x20231013 | pll_locked | clock chip PLL is locked
1 | 0x20240118 | ts_reset_armed | timestamp reset is armed
2 | 0x20240118 | qsfp_tx_link_rx_status | QSFP link status is good for all enabled ports
3 | 0x20240424 | vx_tx_link_rx_status | VX link status is good for all enabled ports
4 | 0x20240118 | cdm_bsy | VX busy grand-or
5 | 0x20240118 | gdm_bsy | QSFP busy grand-or
6 | 0x20240118 | gdm_veto | gdm_veto = gdm_busy
7 | 0x20240118 | cdm_veto | CDM veto from GDM to VX
8 | 0x20240725 | bor_started | begin-of-run sequence started
9 | 0x20240725 | bor_finished | begin-of-run sequence fininished, see commands 10 and 11
10 | | |
11 | | |
12 | | |
13 | | |
14 | | |
15 | | |
16 | | |
17 | | |
18 | | |
19 | | |
20 | | |
21 | | |
22 | | |
23 | | |
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 11 0x8001302C trg_counter ===

trigger counter

=== Register 12 0x80013030 tsm_counter ===

time slice marker counter

=== Register 13 0x80013034 GPS 1pps period ===

GPS 1pps period in 8 ns clocks

=== Register 14 0x80013038 Rb clock 1pps period ===

PRS-10 Rb clock 1pps output period in 8 ns clocks

=== Register 15 0x8001303C SFP RX status ===

<pre>
bit | ds20k version | fpga signal name | description
0 | ALL | sfp_rx_data[15:0] | cdm sfp received data
16 | same | sfp_rx_data_is_k[0] |
17 | same | sfp_rx_data_is_k[1] |
18 | | 0 |
19 | 0x20231204 | sfp_rx_sel_lpb | sfp tx->rx loopback
20 | 0x20231013 | sfp_link_status | sfp link connected, exchanging data
21 | same | sfp_link_rx_status | sfp link receiving correct idle pattern from GDM TX
22 | same | sfp_link_error | sfp link receiver error (badk or overflow)
23 | same | sfp_rx_data_error | sfp transceiver state machine is in error state
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 16 0x80013040 SFP TX control ===

<pre>
wire [15:0] sfp_tx_data_reg = register_data_in[16][15:0];
wire [1:0] sfp_tx_ctrl_reg = register_data_in[16][17:16];
wire sfp_rx_sel_lpb = register_data_in[16][28]; // TX->RX serial loopback
wire sfp_tx_sel_lpb = register_data_in[16][29]; // RX->TX serial loopback
wire sfp_tx_sel_trg = register_data_in[16][30]; // 16 individual bits
wire sfp_tx_sel_reg = register_data_in[16][31]; // from register
</pre>

=== Register 17-22 0x80013044 QSFP RX data ===

QSFP RX data links 0..11

=== Register 23 0x8001305C QSFP TX control ===

<pre>
wire [15:0] qsfp_tx_data_reg = register_data_in[23][15:0];
wire [1:0] qsfp_tx_ctrl_reg = register_data_in[23][17:16];
wire qsfp_rx_sel_lpb = register_data_in[23][28]; // TX->RX loopback
wire qsfp_tx_sel_lpb = register_data_in[23][29]; // RX->TX loopback
wire qsfp_tx_sel_trg = register_data_in[23][30]; // 16 individual bits
wire qsfp_tx_sel_reg = register_data_in[23][31]; // data from register
</pre>

=== Register 24 0x80013060 trigger pulser period ===

trigger pulser period in units of 8 ns (125 MHz clock)

=== Register 25 0x80013064 trigger burst pulser ===

<pre>
wire [7:0] conf_burst_count = conf_pulser_burst_ctrl[31:24];
wire [23:0] conf_burst_period = conf_pulser_burst_ctrl[23:0];
</pre>

=== Register 26 0x80013068 tsm pulser period ===

time slice marker period in units of 8 ns (125 MHz clock)

=== Register 27 0x8001306C data write fifo ===

<pre>
wire fifo_reset = register_data_in[27][31];
wire fifo_to_fpga_wr1 = register_data_in[27][27];
wire fifo_to_fpga_wr2 = register_data_in[27][26];
wire [16:0] fifo_to_fpga_din = register_data_in[27][16:0];
</pre>

<pre>
assign register_data_out[27][31:24] = register_data_in[27][31:24]; // echo write bits
assign register_data_out[27][23] = fifo_to_fpga_full;
assign register_data_out[27][22] = fifo_to_fpga_empty;
</pre>

=== Register 28 0x80013070 data read fifo ===

<pre>
wire fifo_reset = register_data_in[28][31];
wire fifo_from_fpga_rd1 = register_data_in[28][25];
wire fifo_from_fpga_rd2 = register_data_in[28][24];
</pre>

<pre>
assign register_data_out[28][31:24] = register_data_in[28][31:24]; // echo write bits
assign register_data_out[28][21] = fifo_from_fpga_full;
assign register_data_out[28][20] = fifo_from_fpga_empty;
assign register_data_out[28][16:0] = fifo_from_fpga_dout;
</pre>

=== Register 29 0x80013074 packet routing ===

Control packet routing and loopbacks:

<pre>
wire [3:0] dn_route_ctrl = register_data_in[29][3:0];
wire [3:0] up_route_ctrl = register_data_in[29][7:4];
wire [3:0] fifo_to_fpga_route_ctrl = register_data_in[29][11:8];
//wire [3:0] spare_route_ctrl = register_data_in[29][15:12];

wire dn_mux_trg_enable = register_data_in[29][16];
wire dn_mux_tsm_enable = register_data_in[29][17];
wire dn_mux_sfp_rx_fifo_enable = register_data_in[29][18];
// 19
// 20..23

wire up_mux_vx_rx_enable = register_data_in[29][24];
// 25
// 26
// 27
wire fifo_from_fpga_hitmap_enable = register_data_in[29][28];
// 29..31
</pre>

fifo_to_fpga output routing:
* 0 - to down packet mux
* 1 - to up packet mux
* 2 - to fifo_from_fpga mux
* 3 - not used

down packet mux inputs:
* fifo_to_fpga_0
* trg_pkt16 enabled by dn_mux_trg_enable
* tsm_pkt16 enabled by dn_mux_tsm_enable
* sfp_rx_fifo_pkt16 enabled by dn_mux_sfp_rx_fifo_enable
* up_pkt16_2 loopback from up packet mux

down packet mux output routing:
* 0 - to vx_tx_pkt16 to VX TX (vx_link_tx and vx_ser_tx)
* 1 - to fifo_from_fpga mux
* 2 - to up packet mux loopback
* 3 - to qsfp_tx_pkt16 to GDM QSFP TX (cdm_link_tx)

up packet mux inputs:
* vx_rx_pkt16 enabled by up_mux_vx_rx_enable data from 12 VX RX links
* fifo_to_fpga_1
* dn_pkt16_2 loopback from down packet mux

up packet mux output routing:
* 0 - to sfp_tx_pkt16 to CDM SFP TX (VX data to GDM)
* 1 - to fifo_from_fpga mux
* 2 - to down packet mux loop loopback
* 3 - not used

fifo_from_fpga mux inputs:
* fifo_to_fpga_2
* dn_pkt16_1 from down packet mux
* up_pkt16_1 from up packet mux
* qsfp_tx_pkt16 from GDM QSFP link 0 (there is no GDM QSFP 12-to-1 mux)
* hitmap_pkt16 enabled by fifo_from_fpga_hitmap_enable in register 29

=== Register 30 0x80013078 qsfp link status ports 0..7 ===

<pre>
assign register_data_out[30] =
{
qsfp_rx_data_error[7], qsfp_link_error[7], qsfp_link_status[7], qsfp_link_rx_status[7], // 7
qsfp_rx_data_error[6], qsfp_link_error[6], qsfp_link_status[6], qsfp_link_rx_status[6], // 6
qsfp_rx_data_error[5], qsfp_link_error[5], qsfp_link_status[5], qsfp_link_rx_status[5], // 5
qsfp_rx_data_error[4], qsfp_link_error[4], qsfp_link_status[4], qsfp_link_rx_status[4], // 4
qsfp_rx_data_error[3], qsfp_link_error[3], qsfp_link_status[3], qsfp_link_rx_status[3], // 3
qsfp_rx_data_error[2], qsfp_link_error[2], qsfp_link_status[2], qsfp_link_rx_status[2], // 2
qsfp_rx_data_error[1], qsfp_link_error[1], qsfp_link_status[1], qsfp_link_rx_status[1], // 1
qsfp_rx_data_error[0], qsfp_link_error[0], qsfp_link_status[0], qsfp_link_rx_status[0] // 0
};
</pre>

=== Register 31 0x8001307C qsfp link status ports 8..11 ===

<pre>
assign register_data_out[31] =
{
4'b0000,
4'b0000,
4'b0000,
4'b0000,
qsfp_rx_data_error[11], qsfp_link_error[11], qsfp_link_status[11], qsfp_link_rx_status[11], // 11
qsfp_rx_data_error[10], qsfp_link_error[10], qsfp_link_status[10], qsfp_link_rx_status[10], // 10
qsfp_rx_data_error[9], qsfp_link_error[9], qsfp_link_status[9], qsfp_link_rx_status[9], // 9
qsfp_rx_data_error[8], qsfp_link_error[8], qsfp_link_status[8], qsfp_link_rx_status[8] // 8
};
</pre>

=== Register 32 0x80013080 bitmap of enabled qsfp ports ===

<pre>
wire [11:0] qsfp_mask = register_data_in[32][11:0];
wire qsfp_bsy_force = register_data_in[32][12];
// not used = register_data_in[32][15:13];
wire [15:0] gdm_veto_extend = register_data_in[32][31:16];
</pre>

On the GDM:

qsfp_rx_bsy[11..0] are pulses received from the CDMs

qsfp_rx_bsy_or is the grand-or of qsfp_rx_bsy masked by qsfp_mask

if qsfp_rx_bsy_or is high, gdm_bsy goes up and stays up for vx_bsy_extend*2 clocks

gdm_veto = gdm_bsy (bsy of any one VX causes trigger veto to all of them)

gdm_veto transition 0->1 causes gdm_veto_pulse. as long as gdm_veto is high, gdm_veto_pulse is generated every gdm_veto_extend clocks

gdm_veto_pulse is sent to all CDMs.

For this to work right, cdm_bsy_extend should not be bigger than gdm_veto_extend.

=== Register 33 0x80013084 bitmap of enabled VX ports ===

<pre>
wire [11:0] vx_mask = register_data_in[33][11:0];
wire vx_bsy_force = register_data_in[33][12];
// not used = register_data_in[33][15:13];
wire [15:0] vx_bsy_extend = register_data_in[33][31:16];
</pre>

On the CDM:

cdm_bsy is a grand-or of all vx_bsy masked by vx_mask (list of active VXes).

vx_bsy_extend controls how often state of cdm_bsy is sent to the GDM. when cdm_bsy goes 0->1, we send a cdm_bsy_pulse and keep resending it every vx_bsy_extend clocks as long as cdm_bsy stays high.

cdm_bsy_pulse is sent to the GDM.

sfp_rx_veto is the received from the GDM

if sfp_rx_veto goes up, cdm_veto goes up and stays up for gdm_veto_extend clocks.

for this to work right, CDM gdm_veto_extend must be bigger than GDM gdm_veto_extend.

cdm_veto goes to VXes on v1_tx_out[1] which is LVDS input 13.

=== Register 34, 35 0x80013088, 8C current timestamp ===

current 64-bit timestamp, 125 MHz

=== Register 36, 37 0x80013090, 94 old timestamp ===

old 64-bit timestamp, 125 MHz. timestamp saved at run start when it is reset to 0.

=== Register 38, 39, 40 0x80013098, 9C, A0 VX busy counters ===

8 bits per VX port, counters overflow to 255, reset at run start.

=== Register 41, 42, 43 0x800130A4, A8, AC QSFP busy counters ===

8 bits per QSFP port, counters overflow to 255, reset at run start.

=== Register 44, 45, 46, 47, 48 0x800130B0, B4, B8, BC, C0 CDM and GDM busy and veto counters ===

* cdm_busy = grand-or of all VX busy for enabled VXes
* gdm_busy = grand-or of all CDM busy for enabled CDMs
* gdm_veto = gdm_busy

<pre>
44 | lo 16 bits | cdm_bsy_up_counter | CDM busy, increments when cdm_busy goes 0->1
44 | hi 16 bits | cdm_bsy_pulse_counter | CDM busy to GDM, increments for each cdm_bsy_pulse sent to the GDM

45 | lo 16 bits | cdm_veto_pulse_counter | CDM veto from GDM, increments for each sfp_rx_veto received from the GDM
45 | hi 16 bits | cdm_veto_up_counter | CDM veto to VX, increments each time cdm_veto is set to 1.

46 | lo 16 bits | gdm_bsy_pulse_counter | GDM busy from CDM, increments for each qsfp_rx_busy received from CDMs (unless they overlap)
46 | hi 16 bits | gdm_bsy_up_counter | GDM busy, increments each time gdm_bsy goes 0->1

47 | lo 16 bits | gdm_bsy_refresh_counter | GDM busy, increments each time gdm_bsy is extended by new qsfp_rx_busy
47 | hi 16 bits | spare |

48 | lo 16 bits | gdm_veto_up_counter | GDM veto, increments each time gdm_veto goes 0->1
48 | hi 16 bits | gdm_veto_pulse_counter | GDM veto to CDM, increments for each gdm_veto_pulse sent to the CDM
</pre>

=== Register 49, 50, 51 0x800130xx VX RX serial data monitor ===

VX RX data, 8-bit per VX channel. k-bit is omitted.

<pre>
assign register_data_out[49][7:0] = vx_rx_data[0]; // vx1
assign register_data_out[49][15:8] = vx_rx_data[1]; // vx2
assign register_data_out[49][23:16] = vx_rx_data[2]; // vx3
assign register_data_out[49][31:24] = vx_rx_data[3]; // vx4

assign register_data_out[50][7:0] = vx_rx_data[4]; // vx5
assign register_data_out[50][15:8] = vx_rx_data[5]; // vx6
assign register_data_out[50][23:16] = vx_rx_data[6]; // vx7
assign register_data_out[50][31:24] = vx_rx_data[7]; // vx8

assign register_data_out[51][7:0] = vx_rx_data[8]; // vx9
assign register_data_out[51][15:8] = vx_rx_data[9]; // vx10
assign register_data_out[51][23:16] = vx_rx_data[10]; // vx11
assign register_data_out[51][31:24] = vx_rx_data[11]; // vx12
</pre>

=== Register 52, 53 VX link status ===

<pre>
assign register_data_out[52] =
{
vx_rx_error[7], vx_link_error[7], vx_link_status[7], vx_link_rx_status[7], // 7
vx_rx_error[6], vx_link_error[6], vx_link_status[6], vx_link_rx_status[6], // 6
vx_rx_error[5], vx_link_error[5], vx_link_status[5], vx_link_rx_status[5], // 5
vx_rx_error[4], vx_link_error[4], vx_link_status[4], vx_link_rx_status[4], // 4
vx_rx_error[3], vx_link_error[3], vx_link_status[3], vx_link_rx_status[3], // 3
vx_rx_error[2], vx_link_error[2], vx_link_status[2], vx_link_rx_status[2], // 2
vx_rx_error[1], vx_link_error[1], vx_link_status[1], vx_link_rx_status[1], // 1
vx_rx_error[0], vx_link_error[0], vx_link_status[0], vx_link_rx_status[0] // 0
};

assign register_data_out[53] =
{
//4'b0000,
1'b0, ~vx_rx_deser_rdy[0], vx_rx_code_err[0], vx_rx_disp_err[0],
//4'b0000,
//4'b0000,
//4'b0000,
vx_rx_monitor[0],
vx_rx_error[11], vx_link_error[11], vx_link_status[11], vx_link_rx_status[11], // 11
vx_rx_error[10], vx_link_error[10], vx_link_status[10], vx_link_rx_status[10], // 10
vx_rx_error[9], vx_link_error[9], vx_link_status[9], vx_link_rx_status[9], // 9
vx_rx_error[8], vx_link_error[8], vx_link_status[8], vx_link_rx_status[8] // 8
};
</pre>

vx_rx_monitor (12-bit) is from deserializer_10b.sv:

<pre>
assign monitor_out[9:0] = lastByte[9:0];
assign monitor_out[10] = comma;
assign monitor_out[11] = ready;
</pre>

=== Register 54 VX TX serial data monitor ===

* contents of vx_tx_monitor from vx_ser_tx.sv:
<pre>
assign monitor_out[8:0] = data_to_encoder; // 8-bit + k
assign monitor_out[9] = valid;
assign monitor_out[15:10] = 0;
assign monitor_out[25:16] = encoded_data; // 10-bit
assign monitor_out[26] = encoded_valid;
assign monitor_out[27] = 0;
assign monitor_out[31:28] = 0;
</pre>

=== Register 55 QSFP, SFP, VX link loss counters ===

<pre>
assign register_data_out[55] = {
vx_rx_error_counter,
qsfp_link_rx_status_drop_counter,
sfp_link_rx_status_drop_counter,
vx_link_rx_status_drop_counter
};
</pre>

<pre>
24..31 - CDM VX RX error counters, count any errors in the VX RX data path (bad serial data, fifo overflow, etc)
16..23 - GDM QSFP link loss counter, increments on qsfp_tx_link_rx_status 1->0 (reg10)
8..15 - CDM SFP link loss counter, increments on sfp_link_rx_status 1->0 (reg15)
0...7 - CDM VX link loss counter, increments on vx_tx_link_rx_status 1->0 (reg10)
</pre>

=== Register 68 GPS control and status ===

<pre>
assign register_data_out[68] =
{
8'b00000000, // 23+8
rb_1pps_counter, // 16+8 bits
gps_1pps_counter, // 8+8 bits
1'b0,
1'b0,
rb_ser_in, // 5
gps_data_in, // 4
gps_aux_out, // 3
gps_aux_in, // 2
rb_1pps_in, // 1
gps_1pps_in // 0
};
</pre>

= Firmware registers branch develop_ko =

== Register map ==

<pre>
0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
1 | ALL | ALL | RW | read write scratch register
2 | ALL | CDM | ?? | MGT not used
3 | ALL | CDM | RO | MGT debug_data
4 | ALL | CDM | RW | clk_config_vec
5 | ALL | CDM | ?? | not used
6 | ALL | CDM | RO | CDM_link_data_processing:o_error_time
7 | ALL | CDM | RO | CDM_link_data_processing:o_error_count
</pre>

== register 0 0x80010000 ==

GDM:
<pre>
0 - gdm_link_interface:i_mgt_rst
2 - gdm_link_interface:i_link_down_latched_rst
8 - GDM_link_data_processing:i_rst
10..9 - GDM_link_data_processing:i_data_mode
</pre>

CDM:
<pre>
0 - cdm_link_interface:i_mgt_rst
2 - cdm_link_interface:i_link_down_latched_rst
8 - CDM_link_data_processing:i_rst
10..9 - CDM_link_data_processing:i_data_mode
</pre>

== register 1 0x80010004 ==

GDM:
<pre>
nlinks-1..0 - gdm_link_interface:i_rx_slide_trigger
</pre>

CDM:
<pre>
nlinks-1..0 - cdm_link_interface:i_rx_slide_trigger
</pre>

== register 2 0x80010008 ==

GDM:
<pre>
nlinks-1..0 - gdm_link_interface:o_link_power_good
nlinks+15..16 - gdm_link_interface:o_link_status
</pre>

CDM:
<pre>
nlinks-1..0 - cdm_link_interface:o_link_power_good
nlinks+15..16 - cdm_link_interface:o_link_status
</pre>

== register 3 0x8001000c ==

GDM: simple loopback register

CDM:
<pre>
31..0 - debug_data - cdm_link_interface:o_debug
</pre>

o_debug:
<pre>
rx_link_rst & rx_error & rx_link_up & rx_receiving_data &
std_logic_vector(rx_state_count) & tx_state_count_on_rx_clk & i_rx_ctrl3(0) &
i_rx_ctrl1(1 downto 0) & i_rx_ctrl0(1 downto 0) &
rx_data_is_k28p1_k28p5 &
i_rx_data;
</pre>

== register 4 0x80010010 ==

GDM write:
<pre>
0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS
</pre>

GDM read:
<pre>
0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS
4 - clk_config_vec(4) - CLK_LOSXTn_LS
5 - clk_config_vec(5) - CLK_LOLn_LS
6 - clk_config_vec(6) - CLK_INTn_LS
7 - constant 1
31..8 - constant 0
</pre>

== register 5 0x80010014 ==

not used

== register 6 0x80010018 ==

GDM:
<pre>
3..0 - GDM_link_data_processing:i_status_select
</pre>

CDM:
<pre>
31..0 - CDM_link_data_processing:o_error_count
</pre>

== register 7 0x8001001c ==

GDM:
<pre>
31..0 - GDM_link_data_processing:o_status_vector
</pre>

CDM:
<pre>
31..0 - CDM_link_data_processing:o_error_count
</pre>

= GDM, CDM, VX packet communications =

* timestamp math
<pre>
1 clock is 8 ns is 125 MHz
8 bits of clocks is 256 clocks is 2048 ns is ~2 usec
16 bits of clocks is ~500 usec is 0.5 msec
24 bits of clocks is ~134 msec
32 bits of clocks is ~34 sec
40 bits of clocks is ~8.7 ksec is 2.4 hours
48 bits of clocks is ~625 hours is ~26 days
56 bits of clocks is ~6.6 kdays is ~18 kyears
62 bits of clocks is ~10 Mhours is 427 kdays is ~1.1 kyears
64 bits of clocks is ~4.4 kyears
</pre>
* 0x02 - TRG packet, 8 bytes, 640 ns on lvds link
<pre>
0 - 0x02
1 - trg_counter[7:0]
2 - ts64 low byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 high byte 3
7 - trg_mask[7:0]
8 - trg_mask[15:8]
</pre>
* 0x03 - TSM packet, 26 bytes, 2080 ns on lvds link
<pre>
0 - 0x03
1 - tsm_counter[7:0]
2 - ts64 low byte 0
3 - 1
4 - 2
5 - 3
6 - 4
7 - 5
8 - 6
9 - ts64 high byte 7
10 - gps_ts64 low byte 0
11 - 1
12 - 2
13 - 3
14 - 4
15 - 5
16 - 6
17 - gps_ts64 high byte 7
18 - gps_data low byte 0
19 - 1
20 - 2
21 - 3
22 - 4
23 - 5
24 - 6
25 - gps_data high byte 7
</pre>
* 0x81 - VX hitmap packet, 10 bytes, 800 ns on lvds link, 48 ns on fiber link
<pre>
0 - 0x81
1 - VX ID
2 - hitmap low byte, nits 7:0
3 - 15:8
4 - 23:16
5 - 31:24
6 - ...:32
7 - ...
8 - ...
9 - hitmap low byte, bits 63:...
</pre>
* 0x82 - CDM hitmap packet, 108 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x82
1 - cdm_hitmap_trigger_counter[7:0]
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - cdm_hitmap_or12 byte 0 (7:0)
11 - cdm_hitmap_or12 byte 1 (11:8) plus 4 bits: 12=0, 13=0, 14=0, 15=cdm_hitmap_grand_or
12 - cdm_hitmap_data, low byte, 12*64 bits = 768 bits = 48 words = 96 bytes
...
107 - cdm_hitmap_data, high byte
</pre>

= AXI bus timing =

* AXI 100 MHz clock, 10 ns, 32-bit data
* AXI single-dword read: 36 clock repeat rate, 360 ns is 2.777 MHz, 4 bytes per transfer is 11.11 Mbytes/sec
* AXI single-qword read: 13 clock repeat rate, 130 ns is 7.7 MHz, 4 bytes per transfer is 30 Mbytes/sec, not accounting for the gap
* AXI memcpy read: 4 transfers at 13 clocks, gap, 4 transfers at 13 clocks, gap, etc. below 30 Mbytes/sec.
[[Image:Ds-dm-axi-read.png|100px]]
[[Image:Ds-dm-axi-read-64.png|100px]]
[[Image:Ds-dm-axi-read-memcpy.png|100px]]
* AXI single-dword write: 20 clocks repeat rate, 200 ns is 5 MHz, 4 bytes per transfer is 20 Mbytes/sec
* AXI single-qword write: 13+20 clocks repeat rate, 330 ns is 3 MHz, 16 bytes per burst is 48 Mbytes/sec
[[Image:Ds-dm-axi-write.png|100px]]
[[Image:Ds-dm-axi-write-64.png|100px]]

= AXI bus addresses =

* see AXI/AMBA addresses assigned inthe FPGA project: (s_axi/reg0 is the DS-DM AXI registers)
<pre>
daq00:ds-dm-gcdm$ grep assign_bd_address scripts/GDM_CDM_XU8_bd.tcl
assign_bd_address -offset 0x80010000 -range 0x00004000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs axi_register_interfa_0/s_axi/reg0] -force
assign_bd_address -offset 0x000400000000 -range 0x40000000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs ddr4/C0_DDR4_MEMORY_MAP/C0_DDR4_ADDRESS_BLOCK] -force
assign_bd_address -offset 0x80000000 -range 0x00010000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs system_management_wiz/S_AXI_LITE/Reg] -force
daq00:ds-dm-gcdm$
</pre>

* see AXI/AMBA addresses exported from FPGA project to Linux kernel: (uio for debug bridge should say "debug bridge")
<pre>
root@gdm0:~# cat /sys/class/uio/uio*/name
axi-pmon
axi-pmon
axi-pmon
axi-pmon
root@gdm0:~#
root@gdm0:~# ls -l /sys/class/uio/
total 0
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio0 -> ../../devices/platform/amba/ffa00000.perf-monitor/uio/uio0
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio1 -> ../../devices/platform/amba/fd0b0000.perf-monitor/uio/uio1
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio2 -> ../../devices/platform/amba/fd490000.perf-monitor/uio/uio2
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio3 -> ../../devices/platform/amba/ffa10000.perf-monitor/uio/uio3
root@gdm0:~# ls -l /sys/class/uio/../../devices/platform/amba/
total 0
-rw-r--r-- 1 root root 4096 Oct 18 19:37 driver_override
drwxr-xr-x 3 root root 0 Oct 18 01:36 fd070000.memory-controller
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd0b0000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd400000.zynqmp_phy
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd490000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd500000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd510000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd520000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd530000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd540000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd550000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd560000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd570000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd6e0000.cci
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff000000.serial
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff020000.i2c
drwxr-xr-x 6 root root 0 Oct 18 01:36 ff0a0000.gpio
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0b0000.ethernet
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0e0000.ethernet
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0f0000.spi
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff160000.mmc
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff170000.mmc
drwxr-xr-x 3 root root 0 Oct 18 01:36 ff960000.memory-controller
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff9d0000.usb0
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa00000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa10000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa50000.ams
drwxr-xr-x 5 root root 0 Oct 18 01:36 ffa60000.rtc
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa80000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa90000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffaa0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffab0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffac0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffad0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffae0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffaf0000.dma
-r--r--r-- 1 root root 4096 Oct 18 19:37 modalias
lrwxrwxrwx 1 root root 0 Oct 18 19:37 of_node -> ../../../firmware/devicetree/base/amba
drwxr-xr-x 2 root root 0 Oct 18 19:37 power
lrwxrwxrwx 1 root root 0 Oct 18 01:36 subsystem -> ../../../bus/platform
-rw-r--r-- 1 root root 4096 Oct 18 01:36 uevent
root@gdm0:~#
</pre>

= Build firmware =

== Build from git clone ==

THESE ARE K.O.'s NOTES FOR CREATING THE PETALINUX DIRECTORY.

THEY DO NOT WORK!

COPY PETALINUX FROM A WORKING PROJECT AND USE "make gdm" and "make cdm" AS DESCRIBED BELOW.

* git clone git@edev-group.triumf.ca:fw/exp/darkside/gcdm.git
* #Makefile change VIVADO_SETTINGS_SCRIPT := /opt/Xilinx/Vivado/2022.1/settings64.sh
* #. /opt/Xilinx/Vivado/2022.1/settings64.sh
* . /opt/Xilinx/Vivado/2020.2/settings64.sh
* make clean
* make all_from_scratch
* . /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
* make petalinux_create
* make petalinux_rebuild_new_hw_des
* bomb out: The TMPDIR: /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp can't be located on nfs.
* mkdir /tmp/build_tmp
* rm -rf /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp/
* ln -s /tmp/build_tmp /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp
* try again
* grinds, loads a whole bunch of packages...
* finishes with desire to copy things to /tftpboot
* make sdcard_cp_to wants to copy files from PetaLinux_GDM_CDM/images/linux/ to SD card

== Build firmware ==

NOTE: directory Petalinux_GDM_CDM should already exist!

<pre>
#. /opt/Xilinx/Vivado/2020.2/settings64.sh
. /opt/Xilinx/Vivado/2022.2/settings64.sh
. /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
make clean_gdm # remove gdm build tree
make gdm # build or rebuild GDM
make copy_gdm # copy to gdm0
make clean_cdm # remove cdm build tree
make cdm # build or rebuild CDM
make copy_cdm # copy to cdm0 and cdm1
</pre>

copy to SD card:
<pre>
open a root shell
format 16 GB Sd card per above
cd .../ds-dm-gcdm
make copy
</pre>

= build times =

<pre>
CDM 12-june-2023 69aabc1c25130d970bc375aca684bd68849e6685
daq13 AMD-5700G 1688.61user 399.33system 23:28.84elapsed 148%CPU
dsdaqgw AMD-7700 1090.55user 247.34system 16:03.55elapsed 138%CPU
dsdaqgw AMD-7700 CDM incremental 196.68user 67.62system 7:35.42elapsed 58%CPU
dsdaqgw AMD-7700 CDM incremental 684.72user 94.17system 7:30.17elapsed 173%CPU
dsdaqgw AMD-7700 GDM incremental 849.84user 99.79system 9:04.56elapsed 174%CPU
</pre>

= prepare bootable sd card =

== format the sd card ==

this only needs to be done once

* become root
* cd ~olchansk/git/ds-dm-gcdm
* use "lsblk" to identify the SD card (should show as 8/16/32 GB block device)/ /dev/sdd in this case
* make sdcard_format SDCARD_DEVICE=/dev/sdd
* disconnect sd card, reconnect the sd card (to detect new partition tables, etc)

== copy CDM boot files ==

<pre>
cd /home/dsdmdev/git/ds-dm-gcdm
make copy
</pre>

== copy boot files to the sd card ==

* as root: identify partition labels, run "blkid", should say "BOOT", "rootfs" and "data"
* mount
<pre>
mkdir /media/olchansk/BOOT
mkdir /media/olchansk/rootfs
mkdir /media/olchansk/data
mount -L BOOT /media/olchansk/BOOT
mount -L rootfs /media/olchansk/rootfs
mount -L data /media/olchansk/data
cp PetaLinux_GDM_CDM/images/linux/BOOT.BIN /media/olchansk/BOOT/
cp PetaLinux_GDM_CDM/images/linux/boot.scr /media/olchansk/BOOT/
cp PetaLinux_GDM_CDM/images/linux/image.ub /media/olchansk/BOOT/
umount /media/olchansk/BOOT
umount /media/olchansk/rootfs
umount /media/olchansk/data
eject /dev/sdd
</pre>

= boot messages =

<pre>
Xilinx Zynq MP First Stage Boot Loader
Release 2020.2 Sep 24 2022 - 13:29:15
NOTICE: ATF running on XCZU4CG/silicon v4/RTL5.1 at 0xfffea000
NOTICE: BL31: v2.2(release):xlnx_rebase_v2.2_2020.3
NOTICE: BL31: Built : 18:02:46, Sep 28 2022

U-Boot 2020.01 (Sep 28 2022 - 18:03:39 +0000)

Model: DarkSide 20k DM
Board: Xilinx ZynqMP
DRAM: 2 GiB
usb dr_mode not found
PMUFW: v1.1
EL Level: EL2
Chip ID: zu4
NAND: 0 MiB
MMC: mmc@ff160000: 0, mmc@ff170000: 1
In: serial@ff000000
Out: serial@ff000000
Err: serial@ff000000
Bootmode: SD_MODE1
Reset reason: SOFT
Net:
ZYNQ GEM: ff0b0000, mdio bus ff0b0000, phyaddr -1, interface rgmii-id

Warning: ethernet@ff0b0000 (eth0) using random MAC address - d6:62:5f:13:00:44
eth0: ethernet@ff0b0000
ZYNQ GEM: ff0e0000, mdio bus ff0e0000, phyaddr -1, interface rgmii-id
Could not get PHY for eth1: addr -1

Hit any key to stop autoboot: 0
ZynqMP>
CTRL-A Z for help | 115200 8N1 | NOR | Minicom 2.7.1 | VT102 | Online 122:2 | ttyACM0
</pre>

= load FPGA from u-boot =

* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/124682257/U-Boot+FPGA+Driver

<pre>
ZynqMP> fpga info
Xilinx Device
Descriptor @ 0x000000007fddb2c0
Family: ZynqMP PL
Interface type: csu_dma configuration interface (ZynqMP)
Device Size: 1 bytes
Cookie: 0x0 (0)
Device name: zu4
Device Function Table @ 0x000000007fda5fe8
PCAP status 0xa0002fde
ZynqMP>
</pre>

* cp CDM_XU8_top.bit /tftpboot/fpga.bit

<pre>
dhcp
tftpb 0x10000000 fpga.bit
fpga loadb 0 0x10000000 ${filesize}
</pre>

<pre>
ZynqMP> dhcp
BOOTP broadcast 1
DHCP client bound to address 192.168.0.100 (1 ms)
*** Warning: no boot file name; using 'C0A80064.img'
Using ethernet@ff0b0000 device
TFTP from server 192.168.0.1; our IP address is 192.168.0.100
Filename 'C0A80064.img'.
Load address: 0x8000000
Loading: *
TFTP error: 'file /tftpboot/C0A80064.img not found for 192.168.0.100' (1)
Not retrying...
ZynqMP> tftpb 0x10000000 fpga.bit
Using ethernet@ff0b0000 device
TFTP from server 192.168.0.1; our IP address is 192.168.0.100
Filename 'fpga.bit'.
Load address: 0x10000000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
############
6.2 MiB/s
done
Bytes transferred = 7797807 (76fc2f hex)
ZynqMP> fpga loadb 0 0x10000000 ${filesize}
design filename = "CDM_XU8_top;UserID=0XFFFFFFFF;Version=2022.2"
part number = "xczu4cg-fbvb900-1-e"
date = "2024/08/14"
time = "14:18:22"
bytes in bitstream = 7797692
zynqmp_align_dma_buffer: Align buffer at 0000000010000073 to 000000000fffff80(swap 0)
ZynqMP>
</pre>

= load FPGA from Linux =

* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841847/Solution+ZynqMP+PL+Programming

== this will reset the CPU ==
<pre>
cp fpga.bit /lib/firmware/
echo fpga.bit > /sys/class/fpga_manager/fpga0/firmware
</pre>

* make .bin file:
<pre>
bootgen -image CDM_XU8_top.bif -arch zynqmp -o ./fpga.bin -w
</pre>
* cat CDM_XU8_top.bif
<pre>
dsdaqgw:ds-dm-gcdm$ cat CDM_XU8_top.bif
all:
{
[destination_device = pl] ./Vivado_CDM_XU8/CDM_XU8.runs/impl_1/CDM_XU8_top.bit
}
dsdaqgw:ds-dm-gcdm$
</pre>

== this will reset the CPU ==

<pre>
cp fpga.bin /lib/firmware/
echo fpga.bin > /sys/class/fpga_manager/fpga0/firmware
</pre>

== this will reset the CPU ==

<pre>
root@dsdm:~# ./fpgautil -b fpga.bin -f Full
</pre>

== this will reset the CPU ==

<pre>
DTSO file from here: https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841847/Solution+ZynqMP+PL+Programming
echo 0 > /sys/class/fpga_manager/fpga0/flags
mount -t configfs configfs /configfs
root@dsdm:~# dtc -O dtb -o fpga.dtbo -b 0 -@ fpga.dtso
root@dsdm:~# cp fpga.dtbo /lib/firmware/
root@dsdm:~# cp fpga.bit /lib/firmware/
root@dsdm:~# rmdir /configfs/device-tree/overlays/fpga
root@dsdm:~# mkdir /configfs/device-tree/overlays/fpga
root@dsdm:~# echo -n "fpga.dtbo" > /configfs/device-tree/overlays/fpga/path
</pre>

= fpgautil =

* https://github.com/Xilinx/meta-xilinx/blob/master/meta-xilinx-core/recipes-bsp/fpga-manager-script/files/fpgautil.c

<pre>
git clone https://github.com/Xilinx/meta-xilinx.git
cd meta-xilinx/meta-xilinx-core/recipes-bsp/fpga-manager-script/files/
scp fpgautil.c root@dsdm:
ssh root@dsdm
make fpgautil
ls -l ./fpgautil
</pre>

<pre>
root@dsdm:~# ls -l ./fpgautil
-rwxr-xr-x 1 root root 72256 Aug 16 00:15 ./fpgautil
root@dsdm:~# ./fpgautil

fpgautil: FPGA Utility for Loading/reading PL Configuration

Usage: fpgautil -b <bin file path> -o <dtbo file path>

Options: -b <binfile> (Bin file path)
-o <dtbofile> (DTBO file path)
-f <flags> Optional: <Bitstream type flags>
f := <Full | Partial >
-n <Fpga region info> FPGA Regions represent FPGA's
and partial reconfiguration
regions of FPGA's in the
Device Tree
Default: <full>
-s <secure flags> Optional: <Secure flags>
s := <AuthDDR | AuthOCM | EnUsrKey | EnDevKey | AuthEnUsrKeyDDR | AuthEnUsrKeyOCM | AuthEnDevKeyDDR | AuthEnDevKeyOCM>
-k <AesKey> Optional: <AES User Key>
-r <Readback> Optional: <file name>
Default: By default Read back contents will be stored in readback.bin file
-t Optional: <Readback Type>
0 - Configuration Register readback
1 - Configuration Data Frames readback
Default: 0 (Configuration register readback)
-R Optional: Remove overlay from a live tree

Examples:
(Load Full bitstream using Overlay)
fpgautil -b top.bit.bin -o can.dtbo -f Full -n full
(Load Partial bitstream using Overlay)
fpgautil -b rm0.bit.bin -o rm0.dtbo -f Partial -n PR0
(Load Full bitstream using sysfs interface)
fpgautil -b top.bit.bin -f Full
(Load Partial bitstream using sysfs interface)
fpgautil -b rm0.bit.bin -f Partial
(Load Authenticated bitstream through the sysfs interface)
fpgautil -b top.bit.bin -f Full -s AuthDDR
(Load Parital Encrypted Userkey bitstream using Overlay)
fpgautil -b top.bit.bin -o pl.dtbo -f Partial -s EnUsrKey -k <32byte key value>
(Read PL Configuration Registers)
fpgautil -b top.bit.bin -r
(Remove Partial Overlay)
fpgautil -R -n PR0
(Remove Full Overlay)
fpgautil -R -n full
Note: fpgautil -R is responsible for only removing the dtbo file from the livetree. it will not remove the PL logic from the FPGA region.

root@dsdm:~#
</pre>

= Boot from network =

== u-boot ==

<pre>
ZynqMP> setenv bootcmd run bootcmd_dhcp
ZynqMP> saveenv
ZynqMP> reset
</pre>

== boot.scr ==

<pre>
# boot.scr
# mkimage -C none -A arm -T script -d boot.scr boot.scr.uimg
echo Loading FPGA!
#tftpb 0x10000000 fpga.bit
tftpb 0x10000000 {ipaddr}.bit
fpga loadb 0 0x10000000 ${filesize}
echo Booting Linux!
run bootcmd_pxe
echo Done!
</pre>

<pre>
mkimage -C none -A arm -T script -d boot.scr boot.scr.uimg
</pre>

== tftpboot ==

<pre>
cp /home/olchansk/git/ds-dm-gcdm/boot.scr.uimg /tftpboot
ln -s /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/images/linux /tftpboot/xilinx-dsdm
mkdir /tftpboot/pxelinux.cfg
cat > /tftpboot/pxelinux.cfg/default-arm-zynqmp <<EOF
LABEL Linux
KERNEL xilinx-dsdm/Image
FDT xilinx-dsdm/system.dtb
#INITRD rootfs.cpio.gz.u-boot
EOF
</pre>

== boot sequence ==

* xilinx magic load BOOT.BIN from SD card
* load FPGA form BOOT.BIN
* load and run u-boot from BOOT.BIN or from image.ub
* u-boot load environment from ??? probably SD card uboot-redund.env, this includes our bootcmd
* run bootcmd which run bootcmd_dhcp which does:
* from /tftpboot:
* load and run boot.scr.uimg which does:
* load FPGA image xilinx-dsdm/${ipaddr}.bit
* run bootcmd_pxe which does:
* load pxelinux.cfg/default-arm-zynqmp which does:
* load xilinx-dsdm/Image ### this is the linux kernel
* load xilinx-dsdm/system.dtb ### this is the device tree
* start linux kernel
* linux kernel does dhcp
* linux kernel does nfs mount /nfsroot/%s,vers=3,tcp ### %s is replaced by the hostname supplied by DHCP
* userland starts and runs to console and ssh login.

= Xilinx ILA =

References:
* https://github.com/Xilinx/XilinxVirtualCable/tree/master
* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/644579329/Xilinx+Virtual+Cable
* https://support.xilinx.com/s/article/974879?language=en_US
* https://docs.xilinx.com/v/u/en-US/pg245-debug-bridge
* https://docs.xilinx.com/r/2020.2-English/ug908-vivado-programming-debugging/Initializing-Vivado-IDE-hw_server
* (we do not use this one) https://github.com/paulscherrerinstitute/xvcSupport/blob/master/README.md

Build xvcserver_cdm.exe: (it is built as a static executable, can be copied and run anywhere)
<pre>
ssh dsdaq@gdm0
cd /home/dsdaq/online/ds-dm-software
git pull ### get latest version
make xvcserver_cdm.exe
ssh root@gdm0
/home/dsdaq/online/ds-dm-software/xvcserver_cdm.exe
INFO: To connect to this xvcServer instance, use url: TCP:gdm0:2542
</pre>

To activate and use the vivado logic analyzer:
* data path: vivado -> hw_server -> xvcserver -> mmap axi bus -> debug bridge -> jtag -> ILA
* define ILAs in the code
* instantiate the xilinx debug bridge at AXI bus address 0x80020000 (FIXME!!! this collides with Ian's AXI addresses)
* build and boot the new FPGA firmware. updating the linux kernel is not necessary.
* login root@gdm0, run: /home/dsdaq/online/ds-dm-software/xvcserver_cdm.exe -v ### with "-v" for the first time to see that vivado does connect to it, without "-v", normally.
* login dsdaqgw, run: hw_server -s tcp:localhost:3121 -e "set auto-open-servers xilinx-xvc:gdm0:2542" ### tells us to connect to port localhost:3121
* login dsdaqgw, run vivado, open project, open hardware manager, open target, open new target, "connect to remote server", hostname "localhost", port "3121", next (bombs, try again, 3 times), popup add virtual cable, enter hostname "gdm0" port "2542", "ok", it shows in "hardware targets", "next", "finish", error popup "[Common 17-163] Missing value for option 'objects', please type 'set_property -help' for usage info", ignore it, in "hardware", right click the "gdm0" one, open target, under "hardware" and "debug bridge" we should see all the ILAs, under "hardware device properties", the "probes file" should have the ".ltx" file generated by vivado "Vivado_GDM_XU8/GDM_XU8.runs/impl_1/debug_nets.ltx", click on an ILA, a waveform should open.

= Software =

* ssh cdm0 # or gdm0
* sudo apt install i2c-tools libi2c-dev
* git clone https://bitbucket.org/team-ds-dm/ds-dm-software
* cd ds-dm-software
* make

== test_cdm.exe ==

=== CDM SFP status ===

<pre>
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0xbb2f0ae7
CDM firmware 0xbb2f0ae7
arg 1: [--sfp]
Polling SFP status...
identifier 0x03
connector 0x07
encoding 0x01
wavelength 0x0352 (850 nm)
vendor_name [FINISAR CORP. ]
vendor_pn [FTLF8526P3BNL ]
vendor_rev [A ]
vendor_sn [N3AB9M8 ]
vendor_date [200319 ]
dm_type 0x68
temp 29.0 C
vcc 3.323 V
tx_bias 7.250 mA
tx_power 478.4 uW
rx_power 2.3 uW
SFP good 1, status: temp 30.6 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 818 uW
...
</pre>

=== GDM QSFP status ===

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --qsfp3 --qsfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x53aee418
CDM firmware 0x53aee418
arg 1: [--qsfp3]
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out hi
arg 2: [--qsfp]
Polling QSFP status...
identifier 0x0d
status 0x02
los 0x8f
temp 28.2 C
vcc 3.323 V
rx_power 0.1 0.1 0.1 0.1 uW
tx_bias 7.6 7.6 7.6 0.0 mA
tx_power 792.2 773.8 823.0 0.1 uW
vendor_name [FINISAR CORP ]
vendor_pn [FTL410QD4C ]
vendor_rev [A ]
wavelength 850
max_temp 70 C
vendor_sn [X79AC0R ]
vendor_date [220309 ]
QSFP good 1, status: temp 27.7 C, los 0x8b, tx_bias 7.5 7.6 7.6 0.0 mA, tx_power 792 772 821 0 uW, rx_power 0 0 466 0 uW
</pre>

=== GDM clock status ===

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
</pre>

* clock chip not loaded, not running:

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x0f 0x16 0x04 0x00 0x01 0x68 0x09 0x00 0x02 0xf2 0x00 0x1f 0xf0 0x22 0xf2, 0x507: 0x00, 0x52A: 0x00, 0x53F: 0x04
Clock chip state 0, status: SYSINCAL XAXB_ERR LOL CAL_PLL IN0 IN_SEL_0 FASTLOCK_STATUS

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x1f7caf52 (528265042) should be ~125 MHz
0x1034 rx_clk: 0x00000000 (0) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x1f7caf52 (528265042) should be ~125 MHz
0x103C tx_clk: 0x00000000 (0) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip good (IN0 - external 10 MHz clock)

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x44 0x00 0xd0 0x01 0x1f 0xff 0x22 0xf2, 0x507: 0x3f, 0x52A: 0x01, 0x53F: 0x00
Clock chip state 1, status: LOS_IN2 OOF_IN2 IN0 IN_SEL_REGCTRL IN_SEL_0

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735a3b (125000251) should be ~125 MHz
0x1034 rx_clk: 0x07735a3c (125000252) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a3b (125000251) should be ~125 MHz
0x103C tx_clk: 0x07735a3c (125000252) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

=== CDM clock status ===

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
</pre>

* clock chip not loaded, not running:

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x0f 0x16 0x04 0x00 0x01 0x68 0x19 0x00 0x02 0xf2 0x00 0x1f 0xf0 0x22 0xf2, 0x507: 0x00, 0x52A: 0x00, 0x53F: 0x04
Clock chip state 0, status: SYSINCAL XAXB_ERR LOL CAL_PLL IN0 IN_SEL_0 FASTLOCK_STATUS

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735851 (124999761) should be ~125 MHz
0x1034 rx_clk: 0x00000000 (0) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x0127fefa (19398394) should be ~125 MHz
0x103C tx_clk: 0x00000000 (0) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip uses internal clock (IN1 - internal oscillator), observe rx_clk frequency is not same as others

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0x7f, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN1 IN_SEL_1 HOLD_HIST_VALID

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735853 (124999763) should be ~125 MHz
0x1034 rx_clk: 0x07735b0a (125000458) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735853 (124999763) should be ~125 MHz
0x103C tx_clk: 0x07735852 (124999762) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip uses SFP recovered clock (IN2 - sfp rx recovered clock), observe mgt_rx_ref_clk_raw (CDM 125 MHz oscillator) is different from others (SFP RX recovered clock)

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0xbf, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN2 IN_SEL_1 HOLD_HIST_VALID
^C
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773581b (124999707) should be ~125 MHz
0x1034 rx_clk: 0x07735ad6 (125000406) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735ad7 (125000407) should be ~125 MHz
0x103C tx_clk: 0x07735ad6 (125000406) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
^C
</pre>

=== CDM link status, PRBS test mode ===

<pre>
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
</pre>
* fiber disconnected, no link:
<pre>
CDM firmware: 0xbb2f0ae7
0x1000 SFP c.c. status: 0x00000031
CLK_IN_SEL_LS 0x1
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x00000025
sfp_mod_absent_N 1
sfp_rx_los_N 0
link_power_good 1
rx_link_up 0
rx_receiving_data 0
rx_error 1
rx_lnk_up_and_running 0
tx_link_up 0
tx_sending_data 0
tx_link_up_and_running 0
link_up_and_running 0
0x1014 SFP link data: 0x466a8187
rx_data 0x8187
k28p1_k28p5 0
rx_ctrl0 0x1
rx_ctrl1 0x1
rx_ctrl3 0x1
tx_state 0x1
rx_state 0x6
rx_receiving_data 0
rx_link_up 0
rx_error 1
rx_link_rst 0
0x2000 link test mode: 0x00000200, seconds: 0x00079093, errors: 0xffffffff
</pre>
* fiber connected, good link:
<pre>
CDM firmware: 0xbb2f0ae7
0x1000 SFP c.c. status: 0x000000b2
CLK_IN_SEL_LS 0x2
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x000007dc
sfp_mod_absent_N 0
sfp_rx_los_N 0
link_power_good 1
rx_link_up 1
rx_receiving_data 1
rx_error 0
rx_lnk_up_and_running 1
tx_link_up 1
tx_sending_data 1
tx_link_up_and_running 1
link_up_and_running 1
0x1014 SFP link data: 0x35c02774
rx_data 0x2774
k28p1_k28p5 0
rx_ctrl0 0x0
rx_ctrl1 0x0
rx_ctrl3 0x0
tx_state 0x3
rx_state 0x5
rx_receiving_data 1
rx_link_up 1
rx_error 0
rx_link_rst 0
0x2000 link test mode: 0x00000200, seconds: 0x00078a8c, errors: 0x00000000
</pre>

=== GDM link status, PRBS test mode ===

<pre>
1 link connected, no errors:
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
GDM firmware: 0x6b2ee010
0x1014: 0x00000008, 18: 0x00000008, 1C: 0x00000000, 24: 0x00000fff
0x2000: 0x00000200, time: 0x00078aa4, errors:
0xffffffff 0xffffffff 0xffffffff
0x00000000 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
</pre>

=== CDM link status ===

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test0

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0xbf, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN2 IN_SEL_1 HOLD_HIST_VALID

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x077357a0 (124999584) should be ~125 MHz
0x1034 rx_clk: 0x07735a5c (125000284) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a5b (125000283) should be ~125 MHz
0x103C tx_clk: 0x07735a5b (125000283) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM firmware: 0x6d2ef81a
0x1000 SFP c.c. status: 0x000000b2
CLK_IN_SEL_LS 0x2
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x000007dc
sfp_mod_absent_N 0
sfp_rx_los_N 0
link_power_good 1
rx_link_up 1
rx_receiving_data 1
rx_error 0
rx_lnk_up_and_running 1
tx_link_up 1
tx_sending_data 1
tx_link_up_and_running 1
link_up_and_running 1
0x1014 SFP link data: 0x35c6bcbc
rx_data 0xbcbc
k28p1_k28p5 0
rx_ctrl0 0x3
rx_ctrl1 0x0
rx_ctrl3 0x0
tx_state 0x3
rx_state 0x5
rx_receiving_data 1
rx_link_up 1
rx_error 0
rx_link_rst 0
0x2000 link test mode: 0x00000000, seconds: 0x00001671, errors: 0xffffffff

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 15
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
reg[15] is 0x0033bcbc (3390652)
</pre>

=== GDM link status ===

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test0

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x44 0x00 0xd0 0x01 0x1f 0xff 0x22 0xf2, 0x507: 0x3f, 0x52A: 0x01, 0x53F: 0x02
Clock chip state 1, status: LOS_IN2 OOF_IN2 IN0 IN_SEL_REGCTRL IN_SEL_0 HOLD_HIST_VALID

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735a0a (125000202) should be ~125 MHz
0x1034 rx_clk: 0x07735a0a (125000202) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a0a (125000202) should be ~125 MHz
0x103C tx_clk: 0x07735a0a (125000202) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly

register 0x1018 bit 0x800

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM firmware: 0x6d2ebce6
0x1014: 0x00000800, 18: 0x00000800, 1C: 0x00000000, 24: 0x00000fff
0x2000: 0x00000000, time: 0x00003d2f, errors:
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff

link data alternates 0xbcbc and 0x1cbc

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 22
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
reg[22] is 0xbcbc93ab (-1128492117)

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 22
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
reg[22] is 0x1cbc1aaf (482089647)
root@gdm0:~#

</pre>

=== Run trg and tsm ===

<pre>
on the GDM:

ssh root@gdm00
./test_cdm.exe --gdm-clocks
./test_cdm.exe --load-cc
./test_cdm.exe --cc
./test_cdm.exe --reset-mgt
./test_cdm.exe --gdm-clocks
./test_cdm.exe --writereg 2 0xff # enable LEMO NIM inputs
./test_cdm.exe --writereg 3 0xba54 # LEDs: lemo1, lemo2, trg, tsm
./test_cdm.exe --writereg 4 0x99 $ # LEMO out is trg_in_pulse
./test_cdm.exe --writereg 9 0x32010 # trg and tsm from trg_pulser and tsm_pulse
./test_cdm.exe --writereg 24 1250000 # trg pulser 100 Hz
./test_cdm.exe --writereg 26 125000000 # tsm pulser 1 Hz
./test_cdm.exe --writereg 23 0x40000000 # route trg_in and tsm_in to qsfp tx bits 0 and 1
./test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz

on the CDM:

ssh root@cdm01
./test_cdm.exe --cdm-clocks
./test_cdm.exe --load-cc
./test_cdm.exe --reset-mgt
./test_cdm.exe --cdm-clocks
./test_cdm.exe --cdm-link # issue --reset-mgt on CDM and GDM until link is good
./test_cdm.exe --writereg 2 0xff # enable LEMO NIM inputs
./test_cdm.exe --writereg 3 0xba54 # enable LEDs: lemo1, lemo2, trg, tsm
./test_cdm.exe --writereg 4 0x99 # enable LEMO output trg_in
./test_cdm.exe --writereg 9 0x8040 # enable trg_in and tsm_in from sfp rx bits 0 and 1
./test_cdm.exe --writereg 7 0xff00 # drive VX LVDS lines to logic level 0
./test_cdm.exe --writereg 8 0x2 # VX LVDS with trg and tsm, misrouted in second VX
#./test_cdm.exe --writereg 8 0x7 # VX LVDS with tsm
./test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz
</pre>

switch GDM and CDM to packetizer trg and tsm:

<pre>
on the GDM:

/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x00000000

on the CDM:

/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x0804
/home/dsdaq/online/ds-dm-software/test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz
</pre>

=== Run packet loopback ===

GDM CPU -> fifo_to_fpga -> GDM QSFP -> CDM SFP -> fifo_from_fpga -> CDM CPU

On the GDM: (CDM is connected to first QSFP port)

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --load-cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --reset-mgt
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 32 1 ### tell GDM to use first QSFP port
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link ### confirm link status is "3"
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 29 2 ### enable GDM packet data injection
/home/dsdaq/online/ds-dm-software/test_cdm.exe --test-fifo-write-loop
</pre>

One the CDM:

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --load-cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --reset-mgt
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 29 3 ### enable CDM packet data injection
/home/dsdaq/online/ds-dm-software/test_cdm.exe --test-fifo-read
</pre>

= dsvslice integration =

== VX setup ==

* general
** Start acq from user code = y, all others = n (as of Mar 2023: will start when first trigger received)
** Use NIM IO = y
** Use external clock = y
** LVDS quartet is input = n, y, n, y
** LVDS quartet mode = User, User, User, User

* trigger from front panel NIM:
** Trigger on external signal = y, all others = n
** connect CDM EXT_OUT(2) to VX "TrigIn"

* trigger from LVDS "Sync" mode
** Trigger on LVDS Sync signal = y, all others = n
** LVDS quartet mode = User, Sync, User, User

* trigger from LVDS "User" mode
** Trigger on LVDS pair 12 signal = y, all others = n
** LVDS quartet is input = n, y, n, y
** LVDS quartet mode = User, User, User, User

== GDM setup ==

* GDM is gdm0
* set inputs to NIM mode
* set outputs to TTL mode (this GDM has wrong NIM output circuit)
* use top QSFP slot, connect split cable 1 into CDM01, cable 2 into CDM02
* connect non-inverted NIM trigger signal to top-LEMO-left EXT_IN_LV(1)
* connect non-inverted NIM TSM signal to top-LEME-right EXT_IN_LV(2)
* GDM LEDs: TRIG, TSM, trigger enabled, trigger_out
* GDM LEMO_OUT: trigger, trigger

== CDM setup ==

* set CDM LEMO inputs to NIM
* set CDM LEMO outputs to NIM
* CDM01 is cdm0
* CDM02 is cdm1
* connect GDM fiber links to SFP port
* connect 1st VX port of CDM01 to VX1
* connect 1st VX port of CDM02 to VX2
* connect LEMO EXT_OUT(2) to VX "TrigIn", CDM01 to VX1, CDM02 to VX2
* power up
* CDM LEDs: GDM TRIG, GDM TSM, trigger enabled, trigger out
* CDM LEMO_OUT: gdm_trg, trigger

== After power up ==

* start the CDM frontend from the MIDAS "Programs" page. To start manually, see the Start Command on the Programs page.
* CDM frontend should enable the VX clock, disable the trigger
* from the MIDAS status page, goto the CDM page
* in the CDM01, CDM02 data table, the 2nd number should read 0x35c08008, if it does not and the last 4 digits randomly change, reset the GDM links:
** in order GDM, CDM01, CDM02, press "reset1" of each board, then press "reset4", then press "reset0", 2nd number should read as above. if it does not, STOP HERE.
* start a run
* CDM frontend will enable the trigger
* GDM frontend will enable the trigger
* LEDs on the GDM should flash, LEDs on the CDM should flash, TrigIn and TrigOut of the VX should flash
* stop a run
* GDM frontend will disable the trigger
* CDM frontend will disable the trigger

== Phase measurement ==

* pip3 install matplotlib
* pip3 install scipy
* export PYTHONPATH=$HOME/packages/midas/python
* #git clone https://github.com/J033X071C/PhaseMeasurement
* git clone https://bitbucket.org/team-ds-dm/phasemeasurement.git
* cd phasemeasurement
* python3 ./phaseMeasurement.py --help
<pre>
daq00:PhaseMeasurement$ python3 ./phaseMeasurement.py --help
usage: phaseMeasurement.py [-h] fileName numberEvents numberVX sizeEvents stopEvent minHist maxHist numberBin writeToTXT saveAsPDF

Read data from midas file (in .lz4 format) to calculate phase between the clock of VX1 and VX2

positional arguments:
fileName Name of the file we want to read data from (Example: run00389.mid.lz4)
numberEvents Number of events recorded in the file
numberVX Number of VX used in this run (usually 2...)
sizeEvents Number of points per event
stopEvent Number of events you want to go through to calculate phase
minHist Minimal value for the x axis of the phase measurement histogram (in ns)
maxHist Maximal value for the x axis of the phase measurement histogram (in ns)
numberBin Number of bins wanted for the generated histogram
writeToTXT Write argument as yes to generate text file with results of calculation
saveAsPDF Save generated plots to PDF files

optional arguments:
-h, --help show this help message and exit
daq00:PhaseMeasurement$
</pre>
* try an old file with
* python3 ./phaseMeasurement.py run00877.mid.lz4 10000 2 10000 500 -20 20 81 yes yes
<pre>
bin size = 0.494 ns
num_events = 347
mean = -1.705 ns
rms = 3.087 ns
mean_error = 0.166 ns
centroid = -1.706 ns.
width (sigma) = 0.363 ns.
error on the centroid = 0.016558 ns.
</pre>
* ls -l *.txt *.pdf
<pre>
dsdaq@dsvslice:~/online/PhaseMeasurement$ ls -l *.txt *.pdf
-rw-rw-r-- 1 dsdaq dsdaq 64728 Dec 14 16:56 run00877.mid.lz4_Plots.pdf
-rw-rw-r-- 1 dsdaq dsdaq 274 Dec 14 16:56 run00877.mid.lz4.txt
dsdaq@dsvslice:~/online/PhaseMeasurement$
</pre>
* scope settings (from email message)
<pre>
From fcote-lortie@triumf.ca Thu Dec 15 17:11:46 2022
From: Francis Cote-Lortie <fcote-lortie@triumf.ca>
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: Re: How to use scope as a waveform generator
Date: Fri, 16 Dec 2022 01:11:44 +0000

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display) by pressing on the button. It will go from 0 to 1.

The settings that we are using right now are:
Type of waveform: Sine wave
Offset: 0 V
Amplitude: 1 Vpp
Frequency: 50 kHz
Noise: 0 V
________________________________
From: Francis Cote-Lortie <fcote-lortie@triumf.ca>
Sent: Thursday, December 15, 2022 4:58 PM
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: Re: How to use scope as a waveform generator

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display)

The settings that we are using right now are:
Type of waveform: Sine wave
Offset: 0 V
Amplitude: 1 Vpp
Frequency: 50 kHz
Noise: 0 V
________________________________
From: Francis Cote-Lortie
Sent: Thursday, December 15, 2022 4:54 PM
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: How to use scope as a waveform generator

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display)
</pre>

== Standalone link test ==

<pre>
CDM: program clock chip
busybox devmem 0x80011000 32 0x8
busybox devmem 0x80011000 32 0x0
/home/dsdaq/si5394-i2c-file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b

GDM, CDM: link reset
busybox devmem 0x80011008 32 1

GDM, CDM: release reset
busybox devmem 0x80011008 32 0

CDM: link status (NOTE: SFP LOS and mod_absent are swapped!!!)
busybox devmem 0x80011010 32
0x00000024 <- fiber plugged
0x00000025 <- fiber unplugged
0x00000027 <- SFP unplugged
0x000007DC <- successful link with GDM

CDM: link state machine and data
busybox devmem 0x80011014 32
0x35C06FF6

CDM: set link to counting mode
busybox devmem 0x80012000 32 0x101
busybox devmem 0x80012000 32 0x100

CDM: time counter and error counter
root@cdm1:~# busybox devmem 0x80012000 32
0x00000100 <--- link mode
root@cdm1:~# busybox devmem 0x80012004 32
0x0000058C <--- seconds counter
root@cdm1:~# busybox devmem 0x80012004 32
0x0000058D
root@cdm1:~# busybox devmem 0x80012008 32
0x00000000 <--- error counter

GDM: no link
root@gdm0:~# busybox devmem 0x80011014 32
0x00000000
root@gdm0:~# busybox devmem 0x80011018 32
0x00000000
root@gdm0:~# busybox devmem 0x8001101c 32
0x00000000
root@gdm0:~# busybox devmem 0x80011024 32
0x00000FFF
root@gdm0:~#

GDM: good link channel 10, counting mode
root@gdm0:~# busybox devmem 0x80012000 32 0x101
root@gdm0:~# busybox devmem 0x80012000 32 0x100
root@gdm0:~# busybox devmem 0x80012008 32
0x3A8B68C2
root@gdm0:~# busybox devmem 0x80012008 32
0x42E03BEF
root@gdm0:~# busybox devmem 0x8001200c 32
0xDA090972
root@gdm0:~# busybox devmem 0x8001200c 32
0xDE6F22E9
root@gdm0:~# busybox devmem 0x80012019 32
Bus error
root@gdm0:~# busybox devmem 0x80012010 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012014 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012018 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x8001201c 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012020 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012024 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012028 32
0x00000000
root@gdm0:~# busybox devmem 0x8001202c 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012030 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012034 32
0xFFFFFFFF
root@gdm0:~#
</pre>

mapping of link channels:

<pre>
qsfp0 lane0 - 0x0100 - link 8
qsfp0 lane1 - 0x0200 - link 9
qsfp0 lane2 - 0x0400 - link 10
qsfp0 lane3 - n/c
qsfp1 lane0 - 0x0800 - link 11
qsfp1 lane1 - 0x0010 - link 4
qsfp1 lane2 - 0x0020 - link 5
qsfp1 lane3 - n/c
qsfp2 lane0 - 0x0040 - link 6
qsfp2 lane1 - 0x0080 - link 7
qsfp2 lane2 - 0x0001 - link 0
qsfp2 lane3 - n/c
qsfp3 lane0 - 0x0002 - link 1
qsfp3 lane1 - 0x0004 - link 2
qsfp3 lane2 - 0x0008 - link 3
qsfp3 lane3 - n/c
</pre>

script to start the test with 2 CDMs:

<pre>
ssh dsdaq@dsvslice
ssh root@gdm0 busybox devmem 0x80011008 32 1
ssh root@cdm0 busybox devmem 0x80011008 32 1
ssh root@cdm1 busybox devmem 0x80011008 32 1
ssh root@gdm0 busybox devmem 0x80011008 32 0
ssh root@cdm1 busybox devmem 0x80011008 32 0
ssh root@cdm0 busybox devmem 0x80011008 32 0
ssh root@gdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
ssh root@cdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
ssh root@cdm1 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
</pre>

== LEMO trigger GDM to CDM to VX ==

on the GDM:

<pre>
/home/dsdaq/si5394-i2c-file /home/dsdaq/GDM_v1.0_IN0_EXT1_and_IN1_fixed_Si5394-RevA-Registers.txt 0 0x6b
busybox devmem 0x80011008 32 0x1
busybox devmem 0x80011008 32 0x0
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x80001230 # QSFP TX fixed pattern
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 2 0xff # enable LEMO inputs
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 3 0x7654 # enable LED, one per LEMO input
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x0F0F # enable LEMO to trg_in and tsm_in
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x40001230 # enable trg_in and tsm_in output to QSFP
</pre>

on the CDM:

<pre>
/home/dsdaq/si5394-i2c-file /home/dsdaq/CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
/home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp
busybox devmem 0x80011008 32 1
busybox devmem 0x80011008 32 0
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x8040 # trg_in from sfp[0], tsm_in form sfp[1]
/home/dsdaq/online/ds-dm-software/test_cdm.exe 11 # trg_in counter
/home/dsdaq/online/ds-dm-software/test_cdm.exe 12 # tsm_in counter
</pre>

== GPS receiver VCL-2705 ==

* Valiant VCL-2705 GPS receiver
* https://www.valiantcom.com/time-distribution/gps-receiver-irig-b.html
* USB connection is /dev/ttyACM0, 115200 bps, "GNSSAUX" prompt
* minicom -D /dev/ttyACM0 -b 115200
* user manual download instructoins - see sheet of paper with user name and password in the shipping box
* usb commands:
<pre>
gnss-help
gnss-showver -> F/W: Ver 1.7 Feb 17 2020 16:20:43

gnss-showselftest -> no antenna connected
Overall : FAIL
EPROM Test: PASS
Antenna : NOT DETECTED
GNSS : COMMUNICATION OK

gnss-showsettings
GNSS NMEA BAUDRATE :115200
GNSS ANTENNA LENGTH :30 meters
GNSS USER CONFIGURED DELAY:-65 nanoseconds
GNSS 1PPS PULSE WIDTH :200 milliseconds
GNSS MODE :GPS
GNSS STATUS :STATIONARY

GNSSAUX> gnss-showserial
SERIAL :2704H01V17MAX310

GNSSAUX> gnss-showmode
GNSS MODE : GPS

GNSSAUX> gnss-showstatus
GNSS STATUS: STATIONARY

GNSSAUX> gnss-showalarms
CURRENT ALARMS GNSS
ANTENNA : NOT DETECTED

GNSSAUX> gnss-showerrors
CURRENT ALARMS GNSS
ANTENNA : **NOT DETECTED
ERROR STATISTICS GNSS
RMC GOOD DURATION : Secs 0
RMC BAD DURATION : Secs 0
LOCK GOOD SECS DURATION : Secs 0
LOCK BAD SECS DURATION : Secs 0
SATINFO GOOD ITERATIONS :0
SATINFO REJECT ITERATIONS :0
SATINFO NOTALKER ITERATIONS :0
CURRENT MONITOR STATE GNSS
STATE :Phase-1 HUNTING ANTENNA DETECT

GNSSAUX> gnss-showsatinfo
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-showsats
Total Sats: 0

GNSSAUX> gnss-showmyloc
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-show1ppsstate
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-showjamstatus
Not Available !

GNSSAUX> gnss-showspoofstatus
Not Available !

--- antenna connected, can see the sky ---

GNSSAUX> gnss-showselftest
Overall : PASS
EPROM Test: PASS
Antenna : DETECTED
GNSS : COMMUNICATION OK

GNSSAUX> gnss-showalarms
CURRENT ALARMS GNSS
ANTENNA : DETECTED
GNSS LOCK : AVAILABLE

GNSSAUX> gnss-showerrors
CURRENT ALARMS GNSS
ANTENNA : DETECTED
GNSS LOCK : AVAILABLE
ERROR STATISTICS GNSS
RMC GOOD DURATION : Mins 1,Secs 34
RMC BAD DURATION : Secs 55
LOCK GOOD SECS DURATION : Mins 1,Secs 34
LOCK BAD SECS DURATION : Secs 55
SATINFO GOOD ITERATIONS :3
SATINFO REJECT ITERATIONS :0
SATINFO NOTALKER ITERATIONS :0
CURRENT MONITOR STATE GNSS
STATE :Phase-4 NORMAL OPERATION, Monitoring GNSSLOCK

GNSSAUX> gnss-showsatinfo
GNSS MODE :GPS
SATELLITE INFORMATION TALKER:GPS
NMEA ID : 01-32
NO OF SATELLITES IN VIEW: 08
NO OF XXGSV MSGS : 03
SatNo PRN NO (SV ID) ELEVATION (degs) AZIMUTH (degs) C/No (SNR)
1 0 0 0 0
2 0 0 0 0
3 0 0 0 0
4 0 0 0 0
5 0 0 0 0
6 0 0 0 0
7 0 0 0 0
8 0 0 0 0
<<<<<< End of Sat Info >>>>>>

GNSSAUX> gnss-showsats
Total Sats: 8

GNSSAUX> gnss-showmyloc
GNSS RECEIVER LOCATION:
Latitude : 4914.81911
Longitude: 12313.69595

GNSSAUX> gnss-show1ppsstate
GPS 1PPS STATE: LOCKED

GNSSAUX> gnss-showmyloc

GNSS RECEIVER LOCATION:
Latitude : 4914.80688
Longitude: 12313.69531

enter into google maps search box as: 49 14.80688, -123 13.69531, observe the space, the moved dot and the minus.

GNSSAUX> gnss-resetgnss

Executing....Please Wait....
$$$$$END
</pre>

* 1PPS BNC output: period 1 sec, pulse width 200 ms, 3.3V into 1MOhm, 1.38V into 50Ohm.
* IRIG-B BNC output: 5.6V into 1MOhm, 2.4V into 50Ohm.
* IRIG-B format selector: default is all up.

== Rb clock PRS10 ==

* https://www.thinksrs.com/products/prs10.html
* 10 MHz output is sine wave around 5V peak to peak
* 1PPS BNC output is 10 usec pulse, 5V into 1MHohm.
* RS232 connection: minicom -D /dev/ttyUSB0 -b 9600
* ser2net config: localhost,3001:raw:600:usb-5-2-1.0:9600 -XONXOFF -RTSCTS LOCAL
* ssh daq13, cd /home/olchansk/git/ser2net, ./ser2net -c ~/daq/ds/ser2net.conf -d
* ssh daq13, cd ~/daq/ds, python3 prs10.py
* RS232 commands:
<pre>
ID?
PRS10_3.56_SN_105719
VB1
SN?
RS1 -- reset
ST? -- status
FC? -- 10MHz OCXO drive voltage DAC settings
DS? -- "detected signals"
GA? -- gain of frequency lock loop between ovenized oscillator and Rb cell, 0=use ovenized oscillator only
MO? -- magnetic offset of the Rb cell, range 2300..3600, if out of range, unit must be set to different operating mode, see prs10m.pdf
MR? -- magnetic read
TT? -- time-tag, time in ns between 1PPS out and 1PPS in
TS? -- time slope, ???
TO? -- time offset, ???
PS? -- pulse slope, ???
PL? -- 0=phase lock off, 1=phase lock on, lock to 1PPS input
PT? -- phase lock integrator time constant, PT8 is integrator time constant 18.2 hours, natural time constant 2.25 hours
PF? -- phase lock stability factor, PF2 is "1"
PI? -- phase lock integrator

Analog to digital 12 bit ADC, values 0.000 to 4.998

AD0? -- Spare (J204)
AD1? -- +24V(heater supply) divided by 10.
AD2? -- +24V(electronics supply) divided by 10
AD3? -- Drain voltage to lamp FET divided by 10
AD4? -- Gate voltage to lamp FET divided by 10
AD5? -- Crystal heater control voltage
AD6? -- Resonance cell heater control voltage
AD7? -- Discharge lamp heater control voltage
AD8? -- Amplified ac photosignal
AD9? -- Photocell’s I/V converter voltage divided by 4
AD10? -- Case temperature (10 mV/°C)
AD11? -- Crystal thermistors
AD12? -- Cell thermistors
AD13? -- Lamp thermistors
AD14? -- Frequency calibration pot / external calibration voltage
AD15? -- Analog ground

A/D via CPU E-port:

AD16? -- Varactor voltage for 22.48 MHz VCXO (inside RF synthesizer) / 4
AD17? -- Varactor voltage for 360 MHz VCO (output of RF synthesizer) / 4
AD18? -- Gain control voltage for amplifier which drives frequency multiplier / 4
AD19? -- RF synthesizer’s lock indicator voltage (nominally 4.8 V when locked )

ST?

ST1 : Power supplies and Discharge Lamp
ST1 bit, Condition which sets bit, Corrective Action
0 -- +24 for electronic < +22 Vdc
1 -- +24 for electronics > +30 Vdc
2 -- +24 for heaters <+22 Vdc
3 -- +24 for heaters > +30 Vdc
4 -- Lamp light level too low
5 -- Lamp light level too high
6 -- Gate voltage too low
7 -- Gate voltage too high

ST2: RF Synthesizer
ST2 bit, Condition which sets bit, Corrective Action
0 -- RF synthesizer PLL unlocked
1 -- RF crystal varactor too low
2 -- RF crystal varactor too high
3 -- RF VCO control too low
4 -- RF VCO control too high
5 -- RF AGC control too low
6 -- RF AGC control too high
7 -- Bad PLL parameter

ST3: Temperature Controllers
ST3 bit, Condition which sets bit
0 -- Lamp temp below set point
1 -- Lamp temp above set point
2 -- Crystal temp below set point
3 -- Crystal temp above set point
4 -- Cell temp below set point
5 -- Cell temp above set point
6 -- Case temperature too low
7 -- Case temperature too high

ST4: Frequency Lock-Loop Control
ST4 bit, Condition which sets bit
0 -- Frequency lock control is off
1 -- Frequency lock is disabled
2 -- 10 MHz EFC is too high
3 -- 10 MHz EFC is too low
4 -- Analog cal voltage > 4.9 V
5 -- Analog cal voltage < 0.1
6 -- not used
7 -- not used

ST5: Frequency Lock to External 1pps
ST5 bit, Condition which sets bit
0 -- PLL disabled
1 -- < 256 good 1pps inputs
2 -- PLL active
3 -- > 256 bad 1pps inputs
4 -- Excessive time interval
5 -- PLL restarted
6 -- f control saturated
7 -- No 1pps input

ST6: System Level Events
ST6 bit and Condition which sets bit
0 Lamp restart
1 Watchdog time-out and reset
2 Bad interrupt vector
3 EEPROM write failure
4 EEPROM data corruption
5 Bad command syntax
6 Bad command parameter
7 Unit has been reset
</pre>
* ST? on warm start
<pre>
received: PRS_10
received: 255,255,255,243,34,255
received: 0,0,0,1,34,0
...
received: 0,0,0,1,34,0
received: 0,0,0,0,2,0
...
received: 0,0,0,0,2,0
received: 0,0,0,0,4,0
</pre>
* ST? on loss of external 1PPS
<pre>
...
received: 0,0,0,0,4,0
disconnect 1PPS input
received: 0,0,0,0,132,0
...
reconnect 1PPS input
received: 0,0,0,0,132,0
received: 0,0,0,0,4,0
...
</pre>
* ST? on coldish start
<pre>
daq13:ds$ python3 prs10.py
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,0,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 0,0,1,1,34,0 ] old [ 80,0,0,1,34,1 ] counter: 0
received [ 0,0,0,1,34,0 ] old [ 0,0,1,1,34,0 ] counter: 0
received [ 0,0,16,1,34,0 ] old [ 0,0,0,1,34,0 ] counter: 6
received [ 0,0,20,1,34,0 ] old [ 0,0,16,1,34,0 ] counter: 5
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 8
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 3
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 49
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 250
...
</pre>
* ST? on cold start, note: no bump in the 10MHz clock as reported by DS-DM clock chip
<pre>
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,21,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 64,0,21,1,34,0 ] old [ 80,0,21,1,34,1 ] counter: 0
received [ 0,0,21,1,34,0 ] old [ 64,0,21,1,34,0 ] counter: 0
received [ 0,0,20,1,34,0 ] old [ 0,0,21,1,34,0 ] counter: 158
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 23
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 47
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 40
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 249
</pre>

= det fac integration test =

Connections:
* GPS receiver "IRIG-B SEL" both switches "up" - both "on", IRIG-B format B004
* GPS receiver USB-B -> long cable -> daq13 USB-A
* GPS receiver "1PPS out" -> long BNC cable -> BNC-T -> scope (5V, no 50ohm) and Rb clock BNC "1PPS in"
* GPS receiver "IRIG-B 50ohms" -> long BNC cable -> BNC-T -> scope (5V, no 50ohm) and DS-DM LEMO input 1 (TTL mode)
* Rb clock RS232 -> RS232 straight cable -> RS232-to-USB adapter -> daq13 USB-A
* Rb clock "1PPS out" BNC -> scope (5V, no 50ohm, trig threshold rising edge 2V)
* Rb clock "10MHz output 50 Ohm" BNC -> lemo -> lemo-T -> scope (sine wave, 5V, no 50ohm) and DS-DM clock input.

Programs to run:
* on daq13: cd /home/olchansk/git/ser2net, ./ser2net -c ~/daq/ds/ser2net.conf -d
<pre>
localhost,3001:raw:600:usb-5-2-1.0:9600 -XONXOFF -RTSCTS LOCAL
</pre>
* on daq13: cd /home/olchansk/daq/ds, python3 prs10.py ### connects to ser2net
<pre>
daq13:ds$ python3 prs10.py
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,21,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 64,0,21,1,34,0 ] old [ 80,0,21,1,34,1 ] counter: 0
received [ 0,0,21,1,34,0 ] old [ 64,0,21,1,34,0 ] counter: 0
received [ 0,0,20,1,34,0 ] old [ 0,0,21,1,34,0 ] counter: 158
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 23
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 47
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 40
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 249
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 12096
received [ 0,0,0,0,20,0 ] old [ 0,0,0,0,132,0 ] counter: 2
received [ 0,0,0,0,148,0 ] old [ 0,0,0,0,20,0 ] counter: 0
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,148,0 ] counter: 575
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 11449
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 2755
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 34386
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 41035
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 113401
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 33375
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 54767
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 85059
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 33222
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 119234
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 121990
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 128184
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 56002
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 428237
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 8250
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 1388
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 30506
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 142704
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 179451
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 106182
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 68747
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 65424
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 157587
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 6932
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 1388
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 20255
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 4
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 225941
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 72183
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 26970
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 1
39287
</pre>
* on ds-dm: ./test_cdm.exe --irigb ### note sbs mismatch is because I should wrap around at 16 bits
<pre>
dataframe: S00000000S100001010S...S001010101S000001000S, sec: 00, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16724 should be 82260
dataframe: S10000000S100001010S...S101010101S000001000S, sec: 01, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16725 should be 82261
dataframe: S01000000S100001010S...S011010101S000001000S, sec: 02, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16726 should be 82262
dataframe: S11000000S100001010S...S111010101S000001000S, sec: 03, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16727 should be 82263
dataframe: S00100000S100001010S...S000110101S000001000S, sec: 04, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16728 should be 82264
dataframe: S10100000S100001010S...S100110101S000001000S, sec: 05, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16729 should be 82265
</pre>

= DS-IOGC GPS interface board =

* git repository: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0/-/tree/main?ref_type=heads
* schematics: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0/-/blob/main/Altium/Project%20Outputs%20for%20DS-IOGC-Rev0/SCH-DS-IOGC-Rev0.pdf?ref_type=heads
* Rev1 git repository: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev1/
* schematics: [[Image:SCH-DS-IOGC-Rev0.pdf|SCH-DS-IOGC]]

== Rb clock cable ==
<pre>
Rb clock ----- DS-IOGC side, pin numbers are as labeled on the cable connectors

1 - 1pps out - 8 - 1pps out --- correct
2 - nc
3 - nc
4 - TXD - 3 - RU-DATA-OUT - USB-RX input --- correct
5 - 1pps in - 2 - 1pps in --- correct
6 - +24V - 10 - +24V --- should by pin 1 to use both +24V pins?
7 - RXD - 15 - RU-DATA-IN - USB-TX output --- correct
8 - nc
9 - +24V - 10 - +24V --- correct
10 - GND - 9 - GND --- correct
</pre>

== VX connections ==
<pre>
VXA_TX0 - FPGA-OUT-LED2 - D6 LED ("10 MHz clock")
VXA_TX1 - FPGA-OUT-LED1 - D5 LED ("GPS DATA")
VXA_TX2 - FPGA-OUT-RU1PPS-EN - enable 1pps to Rb clock
VXA_TX3 - not used (62.5 Hz clock)

VXB_TX0 - FPGA-TX - PRS-10 RS-232 out
VXB_TX1 - FPGA-OUT-AUX - 5V TTL J5 LEMO out
VXB_TX2 - FPGA-OUT-OPTO1PPS-EN - PRS-10 power enable
VXB_TX3 - not used (62.5 MHz clock)

VXA_RX0 - FPGA-IN-OPTO1PPS - 1pps from GPS
VXA_RX1 - FPGA-IN-GPSDATA - J4 GPS data
VXA_RX2 - n/c
VXA_RX3 - n/c

VXB_RX0 - n/c
VXB_RX1 - FPGA-IN-AUX - J5 LEMO TTL input
VXB_RX2 - FPGA-RX - PRS-10 RS-232 in
VXB_RX3 - FPGA-IN_RU1PPS - PRS-10 1pps output
</pre>

== acceptance tests ==

* ./test_cdm_local.exe --writereg 7 0x4000 ### power up
* ./test_cdm_local.exe --writereg 7 0x4100 ### right LED D6
* ./test_cdm_local.exe --writereg 7 0x4200 ### left LED D5
* ./test_cdm_local.exe --writereg 7 0x6000 ### J5 LEMO measure +5VDC
* install LEMO jumper between LEMO J5 and J6
* write 0x4000 then ./test_cdm_local.exe 6 ### reads 0x5C, bit 5 0x20 reads 0
* write 0x6000 then ./test_cdm_local.exe 6 ### reads 0x7C, bit 5 0x20 reads 1
* write 0x4000 to clear all bits
* 10 MHz output ok - IN0 is good: ./test_cdm_local.exe --cc
<pre>
Clock chip state 1, status: OOF_IN2 IN1 IN_SEL_1 HOLD_HIST_VALID
</pre>
* without 10 MHz reports IN0 LOS (no signal) and OOF (wrong frequency)
<pre>
Clock chip state 1, status: LOS_IN0 OOF_IN0 OOF_IN2 IN1 IN_SEL_1 HOLD_HIST_VALID
</pre>
* IRIG-B via AUX-IN is ok:
<pre>
dataframe: S00100110S111001000S...S011110000S010000000S, sec: 34, min: 17, hrs: 00, day of year: 223, year: 2024, date: 10 aug, sbs: 1054 should be 1054
</pre>
* GPS 1pps to SMB-in ok (LED flashes)
* GPS 1pps to FPGA ok
* GPS data to FPGA ok
* GPS 1pps to PRS-10 enabled from FPGA ok
* PRS-10 Rb clock 1pps out to FPGA ok
* PRS-10 Rb clock 1pps out to FPGA to LEMO AUX out ok
* can see GPS 1pps, IRIG-B, PRS-10 1pps out, 10 MHz on the scope
* PRS-10 syncs on leading edge (0->1) of GPS 1pps signal
* NOT TESTED - smb output
* NOT TESTED - optical converter fiber to SMB
* NOT TESTED - optical converter SMB to fiber
* NOT TESTED - SMB loopback
* NOT TESTED - fiber loopback

= VX busy logic =

= DS-20K DAQ =

== Overview ==

DS-DM, GDM and CDM are key parts of the DS-20K DAQ system:
* common clock distribution from external clock (atomic clock, GPS) to GDM to per-quadrant CDMs to VX digitizers
* common trigger distribution from GDM internal algorithm or external input to all VX digitizers
* run control: GDM, CDM, VX all start recording data at the same time (clock and timestamp reset)
* collection of trigger data from VX digitizers to per-quadrant CDMs to GDM

== Deliverables ==

* hardware and firmware for GDM to CDM clock distribution
* hardware and firmware for CDM to VX clock distribution
* hardware and firmware for GDM external clock input (atomic clock or GPS)
* hardware and firmware for CDM and VX serial communications (VX LVDS I/O connector)
* firmware for run control (timestamp reset and sync): GDM to CDM to VX
* firmware for common trigger distribution: GDM to CDM to VX
* firmware for trigger data flow: VX to CDM to GDM
* firmware for busy control: VX to CDM to GDM back to CDM to VX
* firmware for flow control: FEP to GDM MIDAS frontend to GDM to CDM to VX
* GDM MIDAS frontend: clock selector and monitoring, trigger and run control, busy and flow control, GDM housekeeping
* CDM MIDAS frontend: clock monitoring, CDM housekeeping

specific performance:
* GDM external clock: 10 MHz GPS clock
* GDM to CDM fiber link:
** clock XXX MHz
** link data rate: XXX Gbit/sec
** CDM recovered clock: XXX MHz
** CDM recovered clock jitter: XXX ns
** phase alignment between CDMs: XXX ns
** phase alignment between CDMs persists across reboots, power cycles, firmware updates
** phase alignment between CDMs should be easy to measure
** phase alignment between CDMs should be easy to recalibrate if hardware parts are replaced (DS-DM boards, fiber transceivers, fiber cables, etc)
** data packet bandwidth: XXX Mbytes/sec
** data packet latency: XXX clocks
** data packet skew between CDMs: XXX clocks
* CDM to VX clock:
** clock: XXX MHz
** jitter, all CDM clock outputs: XXX MHz
** phase alignment between all CDM clock outputs: XXX ns
* CDM to VX trigger:
** TBD (use the VX "sync" input or VX LVDS I/O line or VX serial link packet)
* CDM to VX serial link:
** clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
** bit rate: XXX bits/sec
** latency: XXX link clocks
** maximum skew between VXes: XXX ns
* VX to CDM serial link:
** clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
** bit rate: XXX bits/sec
** latency: XXX link clocks
** maximum skew between VXes: XXX ns
* timestamp reset:
** maximum skew between VXes: XXX ns
* busy round trip time: XXX ns (VX to CDM to GDM back to CDM to VX)
* flow control latency: XXX ns (FEP software to GDM MIDAS frontend to GDM to CDM to VX)

== Technical risk items ==

this refers to unexpected behaviour and performance of
system components, causes big difficulty in implementing the system,
prevents delivery of deliverables, and prevents or negatively affects operation
of the DS-20K DAQ or of the whole experiment.

(14-sep-2022, list is not sorted by any criterial: severity, probability, ease of investigation)

(stability of course is long term stability, across hours, days, weeks, months, years)

* stability of Enclustra FPGA modules (crashes/year, failures to boot/year, flash corruption/year)
* stability of GDM external clock PLL (lock loss/year)
* stability of CDM recovered clock (lock loss/year, unexpected phase drifts, etc)
* unexpected failures or bit error rates in GDM-CDM fiber links
* stability of CDM VX clock outputs (stability of clock cleaner chip)
* stability of VX internal clock distribution (VX PLL lock loss events)
* stability of VX CAEN base firmware (different versions of CAEN base firmware have different clock distribution behaviour)
* strange things in CAEN base firmware (unexpected clocking of LVDS I/O, unexpected phase shifts between clocks, etc)
* DS-DM and VX hardware problems (incompatible LVSD I/O, incompatible clock signals, etc)

== Milestones ==

(14-sep-2022: at this stage of the project, priority must be given to identifying and retiring (so called) technical risk factors. it is not good to build the complete system only to discover that (for example) some Enclustra FPGA modules require 5 attempts to boot and erase their flash memory contents once a month. Both example are real-life actual problems that caused big difficulties in GRIFFIN/TIGRESS and ALPHA-g experiments).

Development and testing milestones in time reversed order:

* full DAQ data challenge: all VXes, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
* one quadrant data challenge: 1 VME crate of VX, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
* vertical slice data challenge: 1 VME crate, 2 VX, 2 CDM (1 VX per CDM), GDM, etc operate as designed
* GDM-CDM link finalized (data rate frozen, data packet format frozen, data content permitted to change)
* CDM-VX serial link finalized (data rate frozen, data packet format frozen, data content permitted to change)
* run control (timestamp reset) and trigger distribution design agreed upon, frozen (list of possible triggers permitted to change)
* VX to CDM to GDM data flow design agreed upon, frozen (data contents permitted to change)
* major technical risk items retired (all hardware and firmware is working as expecred without mysteries and surprises, all problems are identified, investigated, resolved, solutions tested)
* stable operation of CDM-VX serial links in vertical slice system
* stable operation of GDM to CDM clock in vertical slice system
* stable operation of CDM to VX clock in vertical slice system
* vertical slice system assembled (1 VME crate, 2 VX, 2 CDM, 1 GDM, network, FEP, TSP, MIDAS)

= ZZZ =

ZZZ

File:SCH-DS-IOGC-Rev0.pdf

2024-09-05T20:55:07Z

Mwalczak: Mwalczak uploaded a new version of File:SCH-DS-IOGC-Rev0.pdf