DaqWiki - User contributions [en]

Ubuntu

2026-04-17T23:18:30Z

Olchansk: /* Remove obsolete packages */

= Prerequisites =

* before setting up new machine run memory test
* prepare flash drive with free version of memtest86: https://www.memtest86.com
* test boot from flash drive, test takes ~ few hours
* test will end with summary page, if passed continue with Ubuntu
* number that might be worth noting is memory latency

= Ubuntu version =

<pre>
lsb_release -a
uname -a
</pre>

= Ubuntu installer =

* updated for Ububtu LTS 20.04.01, 22.04.1, 24.04 (only minor differences)

* download the latest Ubuntu LTS desktop installer iso image
* dd the image to a USB key
* power down, disconnect all disks (all HDDs, all SSDs, all M.2)
* connect the SSD to be used as system disk
* if system will use mirrored SSDs (using ZFS mirror), leave second SSD disconnected, we will activate it later
* power up
* boot from USB key in legacy mode or UEFI mode (select this in the BIOS boot menu - F2 or F8 for ASUS, F11 for Supermicro)
* follow the instruction:
* "try ubuntu or install ubuntu" - choose "install"
* select language - accept default
* "updates and other software" - accept default settings ("normal install")
* "installation type" - select "advanced features" and "experimental: use ZFS"
* accept partition choice
* "where are you?" - select "Vancouver" (PST time zone)
* "who are you?" - leave all fields blank, except "username" set to "wheel", "password" set to the root password. hostname will be set later after configuring the network
* don't install third party sw
* installation runs in a few minutes, when finished, reboot
* login as user wheel
* answer annouying questions:
* "livepatch" - say "next"
* "help improve" - select "do not send", say "next"
* "privacy" - leave "location" as "off", say "next"
* "ready to go", say "done"
* right-click on the desktop, say "open in terminal", a shell will open
* say "sudo /bin/bash", enter the root password, you now have the root shell
* run nm-connection-editor to configure the network. use netmask 255.255.224.0, gateway 142.90.100.18, DNS 142.90.100.19, search path "triumf.ca"
* after network is up (can ping ladd00), continue with post-installation steps below

= Install instructions =

== prepare ==
<pre>
apt update
apt upgrade
</pre>

== install ssh ==
<pre>
apt install ssh
</pre>

== install git/scripts ==
<pre>
apt -y install git
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd scripts
git pull
</pre>

* if needed, update git/scripts repository from ladd00 to daq00:
* git remote -v ### if it says daq00, we are good
* git remote set-url origin https://daq00.triumf.ca/~olchansk/git/scripts.git
* git pull ### check that it works

== configure hostname ==
<pre>
vi /etc/hostname
</pre>

== disable swap ==

ubuntu installer creates a 2 GB swap partition, not useful
on 32-64 GB machine, disable it:
<pre>
vi /etc/fstab ### comment out the "swap" line
</pre>

* on 64 GB RAM machines swap is not useful
* on machines booted from network (NFS-ROOT), swap does not work
* on machines running from flash (RPi, etc), flash is too slow for useful swap
* swap configured by linux installers invariably has wrong size and is not useful

<pre>
systemctl disable dphys-swapfile
systemctl stop dphys-swapfile
dphys-swapfile uninstall
</pre>

== maybe reboot ==

this is a good point to reboot the machine to boot
the latest kernel and to set the correct hostname

== install etckeeper ==

keep contents of /etc in a git repository:

<pre>
apt -y install etckeeper
</pre>

== set timezone ==
<pre>
timedatectl list-timezones | grep -i vancouver
timedatectl set-timezone America/Vancouver
</pre>

== install time synchronization ==

check if chrony is installed correctly and is synched to TRIUMF time servers:

<pre>
chronyc sources
chronyc tracking
</pre>

if not, remove old chrony and

<pre>
apt -y remove chrony
apt -y purge chrony
</pre>

and install it from scratch:

<pre>
apt -y install chrony
cd ~/git/scripts
git pull
cd ~
cp ~/git/scripts/etc/triumf.sources /etc/chrony/sources.d/
systemctl disable systemd-timesyncd.service
systemctl stop systemd-timesyncd.service
systemctl disable ntp
systemctl stop ntp
systemctl enable chrony
systemctl restart chrony
chronyc sources
chronyc tracking
</pre>

== reenable systemd-timesyncd ==

ONLY IF CHRONY DOES NOT WORK

To configure systemd-timesyncd, set "NTP=" in /etc/systemd/timesyncd.conf

<pre>
apt remove chrony
cat /etc/systemd/timesyncd.conf
systemctl enable systemd-timesyncd.service
systemctl restart systemd-timesyncd.service
systemctl status systemd-timesyncd.service
timedatectl status
timedatectl timesync-status
</pre>

== enable outgoing email (debian 11) ==

this is different from ubuntu 20. it uses /etc/mailname and it hardwires the hostname into main.cf.

== enable outgoing email ==

we have an unusual email configuration. outgoing email should work to deliver error messages, notices, etc. incoming email is disabled, we do not receive email for local users.

this causes problems with TRIUMF smtp server. if our message cannot be delivered (wrong email address or receipient computer is turned off), TRIUMF smtp server will generate a delivery failure notification email and try to send it to the "from" address of the failed message. but the "from" address does not receive any email, so another delivery failure notification email is generated and an attempt to deliver it. which again fails, rinse and repeat.

as solution, kray created a special rule, email from scrap.triumf.ca does not generate delivery failure notices. failed messages sit in the queue for 5 days, then they are deleted. (K.O. - confirmed with kray 3jan2024).

to make this work we use the msmtp MTA package.

<pre>
cd ~
apt -y install mailutils msmtp msmtp-mta # say "no" to apparmor support
apt -y remove postfix
apt -y purge postfix # remove old config files
apt -y install bsd-mailx
cd ~/git/scripts/etc
git pull
/bin/cp -fv aliases /etc/aliases
/bin/cp -fv msmtprc /etc/msmtprc
/bin/rm -vf ~root/.forward
/bin/rm -vf /etc/mailname
Mail root
Subject: test
test
^D
CC: <CR>
</pre>

== enable outgoing email (postfix) ==

THIS IS OBSOLETE!!!

* TRIUMF: use smtp.triumf.ca
* CERN: use cernmx.cern.ch

<pre>
apt install postfix ### select "satellite system", enter full hostname "xxx.triumf.ca", enter "smtp.triumf.ca"
apt install mailutils
dpkg-reconfigure postfix ### (if postfix already installed)
</pre>
<pre>
echo olchansk@triumf.ca lindner@triumf.ca bsmith@triumf.ca dfujimoto@triumf.ca >> ~root/.forward
mailx root
test
^D
</pre>

== enable ping for all users (debian 11) ==

Without this tweak, Debian will report "operation not permitted" if a user tries to ping somewhere.

<pre>
echo 'net.ipv4.ping_group_range = 0 1000' > /etc/sysctl.d/99-ping.conf
</pre>

== disable apparmor ==

On NFS-Root network booted machines!

If "man man" returns "permission denied" and syslog reports apparmor "sendmsg DENIED" errors, disable apparmor. This is supposedly fixed in kernel 6.0 and later (to be confirmed), see https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/1784499

Disable apparmor, see https://ubuntu.com/server/docs/security-apparmor

This takes effect after a reboot.

<pre>
systemctl stop apparmor.service
systemctl disable apparmor.service
</pre>

If on boot apparmor appears to still be confining apps (for example, mysql), edit the file /etc/default/grub and change

<pre>
GRUB_CMDLINE_LINUX=""
</pre>

to

<pre>
GRUB_CMDLINE_LINUX="apparmor=0"
</pre>

Run sudo update-grub and reboot.

== install missing packages ==

(apt eats terminal input, even the "yes |" trick does not quite work,
repeat the following commands until they report that everything
is installed)

<pre>
yes | apt -y install ssh tcsh ethtool ncat rsync strace net-tools traceroute time minicom screen git lsof debsums tmux iptables telnet pax rpm mtools at gdisk tcpdump
yes | apt -y install sysstat smartmontools lm-sensors
yes | apt -y install lsb-release
apt -y install vim # in addition to default vim-tiny, requested by IRIS
apt -y install gedit # requested by TACTIC
apt -y install tcl
apt -y install mc # requested by sol
apt -y install pax rpm alien ### package converter tools
apt -y install flex bison
apt -y install neofetch
apt -y install snmp snmp-mibs-downloader
apt -y install git subversion g++ gfortran cmake doxygen
apt -y install curl libcurl4 libcurl4-openssl-dev
### conflits with mysql packages ### apt -y install mariadb-client libmariadb-dev ### mysql client for MIDAS
apt -y install mysql-client libmysqlclient-dev
apt -y install postgresql-common libpq-dev ### postgresql client for MIDAS
yes | apt -y install libz-dev libzstd-dev sqlite3 libsqlite3-dev unixodbc-dev
yes | apt -y install libssl-dev
yes | apt -y install emacs xemacs21 joe
yes | apt -y install gnuplot dos2unix
yes | apt -y install mutt bsd-mailx # email clients
yes | apt -y install liblz4-tool pbzip2 libbz2-dev
yes | apt -y install libc6-dev-i386 # otherwise no /usr/include/sys/types.h
yes | apt -y install libreadline-dev
yes | apt -y install ubuntu-mate-themes
yes | apt -y install libmotif-dev libxmu-dev
yes | apt -y install libusb-dev libusb-1.0-0-dev
yes | apt -y install i2c-tools libi2c-dev libi2c0
yes | apt -y install xfig gsfonts-x11 gsfonts-other # install fonts for xfig
yes | apt -y install libjson-perl
yes | apt -y install libgsl-dev # additional GNU Scientific Library
yes | apt -y install qt5-default # Qt development
yes | apt -y install python3-full python3-dev python3-dbg python3-pip ### for pyROOT
yes | apt -y install imagemagick imagemagick-common ckeditor # for elog
yes | apt -y install libjpeg-dev libjpeg-progs libjpeg-tools
yes | apt -y install linux-tools-common linux-tools-generic # cpupower frequency-info
yes | apt -y install rdesktop remmina remmina-plugin"*" # requested by POL
yes | apt -y install nlohmann-json3-dev # required to build MIDAS with ROOT 6.30 on Ubuntu-22
apt -y install dpkg-dev cmake g++ gcc binutils libx11-dev libxpm-dev libxft-dev libxext-dev python3 libssl-dev libafterimage0 # from https://root.cern/install/dependencies/
apt -y install gfortran libpcre3-dev xlibmesa-glu-dev libglew-dev libftgl-dev libmysqlclient-dev libfftw3-dev libcfitsio-dev graphviz-dev libldap2-dev python3-dev python3-numpy libxml2-dev libkrb5-dev libgsl0-dev qtwebengine5-dev nlohmann-json3-dev libtbb-dev libavahi-compat-libdnssd-dev # from https://root.cern/install/dependencies/
apt -y install libvdt-dev # for ROOT 6.32 on Ubuntu-24
apt -y install autoconf automake gperf # gnu package build tools
apt -y install u-boot-tools # for Xilinx petalinux
#apt -y install linux-headers-generic # to build linux kernel drivers
apt -y install htop
</pre>

Ubuntu LTS 20.04:
<pre>
yes | apt -y install linux-image-generic-hwe-20.04 linux-tools-virtual-hwe-20.04 # enable linux 5.11 series kernel
</pre>

Ubuntu LTS 22.04:
<pre>
apt -y install linux-generic-hwe-22.04 # enable linux 6.8.0 series kernel
</pre>

Ubuntu LTS 24.04:
<pre>
apt -y install linux-generic-hwe-24.04 # enable linux 6.14.0 series kernel
</pre>

== remove snap ==

remove snap at this point.

Do

<pre>
snap list
</pre>

to see a list of installed snaps. Use

<pre>
snap remove <item>
</pre>

to remove each. You will need to remove snapd last.

<pre>
systemctl stop snapd
systemctl disable snapd
apt purge snapd
rm -rf ~/snap
</pre>

== install non-snap firefox ==

delay this until the very end, download of firefox-esr deb package is very slow.

See https://askubuntu.com/questions/1399383/how-to-install-firefox-as-a-traditional-deb-package-without-snap-in-ubuntu-22

<pre>
add-apt-repository ppa:mozillateam/ppa
apt install firefox-esr
echo run firefox-esr
</pre>

Enable automatic updates:

* rerun [[#Enable_automatic_updates]]

== configure DNS ==

<pre>
cd ~/git/scripts
git pull
mkdir /etc/systemd/resolved.conf.d
cp etc/resolved-triumf.conf /etc/systemd/resolved.conf.d/
systemctl restart systemd-resolved
resolvectl
#systemd-analyze cat-config systemd/resolved.conf
</pre>

== install ganglia ==

<pre>
apt -y install ganglia-monitor
cd ~root/git/scripts/ganglia
git pull
make install
./ganglia-all.perl
</pre>

fix gmond start before network is ready:

<pre>
mkdir /etc/systemd/system/ganglia-monitor.service.d
echo -e "[Unit]\nAfter=network-online.target\n" > /etc/systemd/system/ganglia-monitor.service.d/local.conf
systemctl daemon-reload
systemctl cat ganglia-monitor.service
</pre>

== install ganglia server ==

'''On the main computer only! (daq00, dsdaqgw, etc)'''

* zfs create rpool/ganglia
* apt install gmetad php php-xml rrdtool
* mv /etc/ganglia/gmetad.conf /etc/ganglia/gmetad.conf-stock
* create /etc/ganglia/gmetad.conf with the following contents:
<pre>
data_source "my cluster" 15 localhost
RRAs "RRA:AVERAGE:0.5:1:244" "RRA:AVERAGE:0.5:24:244" "RRA:AVERAGE:0.5:168:244" "RRA:AVERAGE:0.5:672:244" "RRA:AVERAGE:0.5:5760:374"
setuid_username "ganglia"
rrd_rootdir "/ganglia/rrds"
case_sensitive_hostnames 0
</pre>
* mkdir /ganglia/rrds
* chown ganglia:ganglia /ganglia/rrds
* systemctl restart gmetad
* create /etc/ganglia/gmond-collect.conf
<pre>
globals {
daemonize = yes
setuid = yes
user = nobody
debug_level = 0
max_udp_msg_len = 1472
mute = no
#deaf = yes
deaf = no
allow_extra_data = yes
host_dmax = 0 /* 86400 */ /*secs. Expires (removes from web interface) hosts in 1 day */
host_tmax = 20 /*secs */
cleanup_threshold = 300 /*secs */
gexec = no
send_metadata_interval = 600 /*secs */
}

/*
* The cluster attributes specified will be used as part of the <CLUSTER>
* tag that will wrap all hosts collected by this instance.
*/
cluster {
name = "DAQ"
owner = "TRIUMF DAQ"
latlong = "unspecified"
url = "https://daq.triumf.ca"
}

/* The host section describes attributes of the host, like the location */
host {
location = "unspecified"
}

udp_send_channel {
host = daq00.triumf.ca
bind_hostname = yes
port = 8649
}

udp_recv_channel {
#mcast_join = 239.2.11.71
port = 8649
#bind = 239.2.11.71
retry_bind = true
# Size of the UDP buffer. If you are handling lots of metrics you really
# should bump it up to e.g. 10MB or even higher.
#buffer = 10485760
buffer = 200000
family = ipv4

acl {
default = "deny"
access {
# vlan1 daq00
ip = 142.90.111.168
mask = 19
action = "allow"
}
access {
# MUSR VLAN
ip = 142.90.154.73
mask = 8
action = "allow"
}
access {
# KVM network
ip = 192.168.1.1
mask = 8
action = "allow"
}
}
}

udp_recv_channel {
#mcast_join = 239.2.11.71
port = 8649
#bind = 239.2.11.71
retry_bind = true
# Size of the UDP buffer. If you are handling lots of metrics you really
# should bump it up to e.g. 10MB or even higher.
#buffer = 10485760
buffer = 200000
family = ipv6

acl {
default = "deny"
access {
# ALPHA network
ip = 2001:1458:202:fd::100:aa
mask = 8
action = "allow"
}
#access {
# # ALPHA-g network
# ip = 192.168.1.1
# mask = 8
# action = "allow"
#}
}
}

tcp_accept_channel {
port = 8649
bind = localhost
# If you want to gzip XML output
gzip_output = no
}
</pre>
* add to /etc/rc.local
<pre>
/usr/sbin/gmond -c /etc/ganglia/gmond-collect.conf &
systemctl restart gmetad &
</pre>
* cd /ganglia
* git clone https://daq00.triumf.ca/~olchansk/git/ganglia-web.git
* cd ganglia-web
* git checkout KO1
* ln -s /ganglia/ganglia-web /var/www/html/ganglia
* make dwoo directories
<pre>
mkdir /var/lib/ganglia-web
chown www-data:www-data /var/lib/ganglia-web
mkdir /var/lib/ganglia-web/dwoo
chown www-data:www-data /var/lib/ganglia-web/dwoo
mkdir /var/lib/ganglia-web/dwoo/compiled
chown www-data:www-data /var/lib/ganglia-web/dwoo/compiled
mkdir /var/lib/ganglia-web/dwoo/cache
chown www-data:www-data /var/lib/ganglia-web/dwoo/cache
</pre>
* open https://alphacpc05.cern.ch/ganglia/

== install gonodeinfo ==

* go to https://bitbucket.org/dd1/gonodeinfo follow instructions:
<pre>
apt -y install golang
mkdir ~/git
cd ~/git
#git clone https://bitbucket.org/dd1/gonodeinfo.git
git clone https://daq00.triumf.ca/~olchansk/git/gonodeinfo.git
cd gonodeinfo
git remote set-url origin https://daq00.triumf.ca/~olchansk/git/gonodeinfo.git
git pull
make
make install # install gonodeinfo agent
cd ~ # this is important
</pre>
* edit /etc/gonodeinfo.conf
* change "Description", "Location", "User" and "Administrator" as appropriate (or delete them)
* change "Servers" to read: Servers: daq00.triumf.ca:8601
* run "gonodeinfo -v"
* if error is "connection refused". go to the nodeinfo server to add this client to the access control list:
* on the gonodeinfo server: run /opt/gonodeinfo/gonodereceive.exe -a daq13
* try gonodeinfo again, there should be no error
* on the gonodeinfo server: run gonodereport, look at the web pages, the new machine should be listed now

== install emailonreboot ==

send an email if computer is rebooted

<pre>
ssh root
mkdir -p ~/git
cd ~/git
git clone https://daq00.triumf.ca/~olchansk/git/rpms.git
cd rpms/emailonreboot
git pull
make
make install
</pre>

== install monitor_nfs ==

monitor NFS mounts and complain about dead, stale and hung mounts

<pre>
ssh root
mkdir -p ~/git
cd ~/git
git clone https://daq00.triumf.ca/~olchansk/git/rpms.git
cd rpms/monitor_nfs
git pull
make
make install
</pre>

== install fonts for EPICS ==

<pre>
apt -y install xfonts-100dpi xfonts-75dpi
killall Xorg # restart Xorg (i.e. "killall Xorg", this will log you out from the console)
xlsfonts | grep -i helvetica ### should show fonts with different sizes, not just size 0 (scalable)
</pre>

== install libz.so.1 for CentOS compatibility ==

KO - confirm which versions on quartus need this.

<pre>
yes | apt-get -y install zlib1g
yes | apt-get -y install zlib1g:i386 libc6:i386 libgcc1:i386 gcc-6-base:i386
</pre>

== install ld-ldb-x86-64.so.3 for Quartus compatibility ==

Not clear from package this is supposed to come from. Copied from U-20.

Without this, Quartus lmgrd does not run.

<pre>
cd /lib64
ln -s ld-linux-x86-64.so.2 ld-lsb-x86-64.so.2
ln -s ld-linux-x86-64.so.2 ld-lsb-x86-64.so.3
</pre>

should look like this:
<pre>
root@daq13:/lib64# ls -l
total 3
lrwxrwxrwx 1 root root 44 Jan 28 09:07 ld-linux-x86-64.so.2 -> ../lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
lrwxrwxrwx 1 root root 20 Feb 18 16:45 ld-lsb-x86-64.so.2 -> ld-linux-x86-64.so.2
lrwxrwxrwx 1 root root 20 Feb 18 16:45 ld-lsb-x86-64.so.3 -> ld-linux-x86-64.so.2
root@daq13:/lib64#
</pre>

== install libpng12.so.0 for Quartus 13.0sp1 and 13.1.4.182 ==

<pre>
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0.50.0
/bin/cp -pv libpng12.so.0 libpng12.so.0.50.0 /lib/x86_64-linux-gnu/
</pre>

== install packages for Xilinx ==

ubuntu LTS 22.04 vivado 2020.1

<pre>
apt install autoconf libtool
apt install libtinfo5
apt install texinfo
apt install zlib1g:i386
</pre>

== install packages for building ROOT ==

<pre>
apt -y install libx11-dev libxpm-dev libxft-dev libxext-dev libpng-dev libjpeg-dev xlibmesa-glu-dev libxml2-dev libgsl-dev cmake
</pre>

== install wine ==

As far as I know, only needed for BNMR/BNQR

<pre>
apt install wine winetricks
</pre>

== install lightdm ==

unlike the default gdm login manager, lightdm shows the machine hostname and does not require an extra mouse click to swicth from screen saver to login mode.

<pre>
apt -y install lightdm
# select lightdm
</pre>

== install desktop environments ==

note: default display manager and default desktop are deficient, please do not skip this step.

note: if apt asks to choose the display manager, select "lightdm"

note: KO - I recommend the "MATE" desktop.

note: you will have to cut-and-paste this several times because "apt" eats commands, even with "-y" and even piped from "yes".

note: DF - on U24 this may re-install snap

<pre>
# install MATE desktop
apt -y install ubuntu-mate-core ubuntu-mate-desktop ubuntu-mate-themes
# install Cinnamon desktop
apt -y install cinnamon
# install KDE desktop
apt -y install kde-standard kubuntu-settings-desktop
# install Lxqt desktop
# apt -y install lxqt # conflict over kubuntu-desktop, kubuntu-settings-desktop and desktop-base
# install Xfce4 desktop
apt -y install xfce4
</pre>

== install ROOT ==

Please install ROOT per instructions at https://root.cern.ch.

NOTE1: The ROOT package available from Ubuntu repositories is severely out of date and cannot be used with MIDAS and ROOTANA. ### DO NOT DO THIS! apt-get install root-system

NOTE2: as of 2017-Jan-09, ROOT binary kits for Ubuntu do not work (use GCC 5 instead of GCC6), build from source instead.

== Install x2go ==

<pre>
apt-get update
apt-get install x2goserver x2goserver-xsession
</pre>

== enable root login from ladd00/daq00 ==
<pre>
ssh localhost
CTRL-C
/bin/cp ~root/git/scripts/etc/authorized_keys ~root/.ssh/
</pre>

== disable ssh access from outside of TRIUMF ==

to stop ssh login spam, disable ssh access from outside of TRIUMF. this can be done by requesting a firewall block through the helpdesk or by local firewall rule:

<pre>
echo iptables -I INPUT ! -s 142.90.0.0/255.255.0.0 -p tcp --dport 22 -j REJECT >> /etc/rc.local
/etc/rc.local
</pre>

== install smart-status ==
<pre>
ln -s ~/git/scripts/smart-status/smart-status.perl ~root/
</pre>

== enable boot menu and boot messages ==

This will enable the grub menu (with a 10 sec timeout) and
replace black screen with exciting linux boot messages.

* emacs -nw /etc/default/grub
<pre>
GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=menu
GRUB_TIMEOUT=10
GRUB_RECORDFAIL_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="vga=769 video=640x480"
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX=""
#GRUB_GFXMODE=640x480
</pre>
* update grub config:
<pre>
grub-mkconfig -o /boot/grub/grub.cfg
</pre>

== Disable welcome message on ssh ==

On ssh, there is a lengthy welcome message. To disable:

<pre>
vim /etc/ssh/sshd_config
</pre>

Ensure that the file reads

<pre>
PrintMod no
</pre>

Then,

<pre>
vim /etc/pam.d/sshd
</pre>

comment out the lines:

<pre>
session optional pam_motd.so motd=/run/motd.dynamic
session optional pam_motd.so noupdate
</pre>

= Enable automatic updates =

<pre>
apt install unattended-upgrades
cd ~/git/scripts
git pull
/bin/cp -v etc/99apt-conf-ko /etc/apt/apt.conf.d/
apt-config dump | grep Unattended
</pre>

Following is obsolete:

* emacs -nw /etc/apt/apt.conf.d/50unattended-upgrades
** uncomment in Allowed-Origins "-security" and "-updates"
** add in Allowed-Origins: "Google LLC:stable";
** uncomment/add: "Unattended-Upgrade::Mail "root";
* emacs -nw /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
</pre>
* test: unattended-upgrade --dry-run -v

NOTE: update-on-shutdown is disabled.

NOTE: there is no update-on-boot, but:

NOTE: if machine was off for a long time, the systemd update timer would have expired and it will fire soon after reboot, causing an automatic update run. this is unwanted, and there is no fix or workaround for it. K.O. June-2023.

= Fix bpool is full (obsolete) =

THIS IS CAUSED BY OBSOLETE PACKAGE zsys. PLEASE: apt remove zsys

= IPMI instructions =

IPMI is the board management hardware on Supermicro and other server motherboards. This includes hardware sensors - fan rotation speed, temperatures and power supply voltages.

<pre>
apt-get install ipmitool
systemctl enable ipmievd
systemctl restart ipmievd
</pre>

Run:
* ipmitool sel list ### event list
* ipmitool sel elist ### event list
* ipmitool sel clear ### clear event list (if it becomes full)
* ipmitool sensor ### report hardware sensors

= move /home/wheel (U-24) =

Ubuntu LTS 24 installed on ZFS has rpool/USERDATA/home_xxx mounted on /home,
has to be moved or autofs /home will not work.

<pre>
zfs list | grep USERDATA | grep home | cut -f1 -d" "
zfs set -u mountpoint=/home1 `zfs list | grep USERDATA | grep home | cut -f1 -d" "`
emacs -nw /etc/passwd # change mount for wheel account to /home1/wheel
</pre>

This will not take effect until rebooting. Please ensure that ssh from root@daq00 to this computer works before rebooting; if ssh from root@daq00 doesn't work and you mess this up you will be locked out of wheel account. Once you verify that this works, reboot and make sure that when you login as wheel that home directory is /home1/wheel.

= move /home/wheel =

note: this MUST be done if ZFS root and NIS/autofs with /home.

Default location of wheel's home directory will collide with autofs /home, it has to be moved,
for example to /wheel.

<pre>
# logout from the wheel user
# go to another computer
ssh root@daqubuntuxxx
zfs list | grep wheel ### identify zfs name wheel_xxxxxx
#zfs set mountpoint=/wheel rpool/USERDATA/wheel_hm8fzh
zfs set mountpoint=/wheel `zfs list | grep wheel | cut -f1 -d" "`
zfs list | grep wheel
emacs -nw /etc/passwd ### change wheel's home directory from /home/wheel to /wheel
su - wheel ### check that user wheel still works
</pre>

This will break wheel's ability to run snap programs, such as firefox, install chrome as listed below.

= enable NIS (ubuntu 22.04, 24.04, debian 11, 12) =

<pre>
apt -y install rpcbind nis
echo DAQ-NIS >> /etc/defaultdomain
echo ypserver daq00.triumf.ca >> /etc/yp.conf
systemctl enable ypbind.service
systemctl restart ypbind.service
systemctl status ypbind.service
ypwhich -m
</pre>

enable ypserv:

<pre>
sed -i s/NISSERVER=false/NISSERVER=slave/ /etc/default/nis
/usr/lib/yp/ypinit -s daq00
echo ypserver localhost >> /etc/yp.conf
sed -i "s/ypserver .*/ypserver localhost/" /etc/yp.conf
systemctl enable ypserv
systemctl restart ypserv
systemctl restart ypbind
</pre>

update /etc/nsswitch.conf and enable hourly update of NIS maps:

<pre>
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
cp -pv nsswitch.conf-U24 /etc/nsswitch.conf
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>

If this is a new machine, then on the master NIS node (daq00), add this new node to /etc/netgroup, and update NIS maps (cd /var/yp; make)

= enable NIS (ubuntu 20.04) =

* apt-get -y install portmap nis ### will ask for NIS domain (DAQ-NIS)
* dpkg-reconfigure nis ### reconfigure if already installed
* ypwhich -m
* edit /etc/default/nis
** set "NISSERVER=slave"
** Ubuntu LTS 20.04, check that "YPBINDARGS=" is blank, remove "-no-dbus" if it is there
* #edit /etc/yp.conf, comment-out everything, add "domain DAQ-NIS server localhost"
* edit /etc/yp.conf, comment-out everything, add "ypserver localhost"
* /usr/lib/yp/ypinit -s daq00
* systemctl enable nis
* systemctl restart nis
* ypwhich
* ypwhich -m
* ypcat -k passwd
* vi /etc/nsswitch.conf ### add the automount line, modify the passwd, group and shadow lines to read this:
<pre>
# begin get data from nis
passwd: files nis
group: files nis
shadow: files nis
automount: files nis
netgroup: files nis
# end get data from nis
</pre>
* enable hourly update of NIS maps
<pre>
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>
* ### NOT NEEDED sudo vi /etc/idmapd.conf ### add line: "Domain = triumf.ca"

= enable autofs =

<pre>
apt -y install autofs
systemctl enable autofs
systemctl restart autofs
ls -l /home/olchansk ### test autofs, check file owner is correct
</pre>

= enable NFS server =

<pre>
apt install nfs-kernel-server
#edit /etc/exports
systemctl enable nfs-server
systemctl restart nfs-server
</pre>

= NIS master =

notes for setting up the NIS master

== wheel user ==

"wheel" is the default administrative user. We do not want it's password exported to NIS (encrypted password hash is world visible) and we do not want it's home directory exported to NFS (~wheel/.ssh is world visible and potentially writable: anybody can change ~wheel/.ssh/authorized_keys).

* move wheel's home directory from /home/wheel to /wheel (see special section about this)
* change wheel's UID and GID from 1000 to a value below MINUID in /var/yp/Makefile

== coherent uids ==

we do not want system accounts defined in /etc/passwd of the NIS master
to be included in the NIS map "passwd". this causes trouble on NIS clients
where newly installed packages fail to create local system users because same
user already exists in NIS.

This is controlled by MINUID in /var/yp/Makefile.

Historical TRIUMF uids start from around 200, but several clusters do not have any historic TRIUMF uids below 500 and MINUID is set to:
* DAQ-NIS: MINUID=200
* ISAC-NIS: MINUID=500
* TITAN-NIS: MINUID=500
* MUSR-NIS: MINUID=500
* TIG-NIS: MINUID=500 (100 on SL6 mother8pi)

Ubuntu 20 has two programs to create users:
* adduser - creates new users with UID 1000 and up as specified in /etc/adduser.conf. No problems here.
* adduser --system - creates new system users with UID 100 and up as specified in /etc/adduser.conf. No problems here.
* useradd - creates new users with UID 1000 and up as specified in /etc/login.defs. No problems here.
* useradd --system - creates new system users with UID 999 and down (read "man useradd", section at the end about SYS_UID_MAX). This collides with NIS MINUID, these system users will be included in the NIS map and cause trouble.

This problem cannot be fixed, SYS_UID_MIN, SYS_UID_MAX and UID_MIN in /etc/login.defs do not seem
to have any effect on UIDs chosen by "useradd --system". (tested on Ubuntu LTS 20.04).

So far only these system accounts seem to be affected by this:
* systemd-coredump
* ganglia

To fix:
* run "sort -r -n -t: -k3 /etc/passwd" to identify the last unused system user uid (range 100..200)
* run "sort -r -n -t: -k3 /etc/group" to identify the last unused system user gid (range 100.200)
* systemd-coredump: manually change UID and GID (package systemd-coredump is usually not installed)
* ganglia: same thing, then change ownership on all ganglia files.

Also read systemd author's opinion on system vs user UIDs:
https://github.com/systemd/systemd/issues/4850#issuecomment-265698275

= Fix systemd-logind NIS breakage =

!!! THIS IS NOT NEEDED FOR UBUNTU LTS 20.04 !!!

there is a delay in ssh logins for normal users. "ssh -v" shows the delay is after "pledge...". this
fix removes the delay.

systemd developers think that we should not use NIS and made sure there are
problems if we do. To give them credit, they do offer a workaround. Read this:
https://github.com/poettering/systemd/commit/695fe4078f0df6564a1be1c4a6a9e8a640d23b67

<pre>
mkdir /etc/systemd/system/systemd-logind.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-logind.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-logind.service
</pre>

= Fix systemd-udevd NIS breakage =

see same problem as above with udev getting stuck. ubuntu lts 20.04.

<pre>
mkdir /etc/systemd/system/systemd-udevd.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-udevd.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-udevd.service
</pre>

= Configure USB device permissions =

Configure USB device permissions for user access to USB-serial devices, Altera USB Blaster, etc.

* create file /etc/udev/rules.d/99-usb-chmod.rules with this contents:

<pre>
emacs -nw /etc/udev/rules.d/99-usb-chmod.rules
ACTION=="add", SUBSYSTEM=="usbmisc", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /dev/%c"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /proc/%c"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVICE}"
ACTION=="add", ENV{PHYSDEVBUS}=="usb-serial", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVPATH}=="/class/tty/ttyS*", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyUSB*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyACM*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyS*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", DEVPATH=="*video*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
</pre>

* reload udev rules: udevadm control --reload-rules
* apply new permissions: udevadm trigger --action=add
* watch udev activity: udevadm monitor -p

= Configure lightdm display manager =

* enable it
<pre>
echo lightdm | dpkg-reconfigure -fteletype lightdm
systemctl disable gdm
systemctl disable sddm
systemctl enable lightdm
</pre>

* make the MATE desktop as default
<pre>
cd ~root/git/scripts/
git pull
/bin/cp -v etc/lightdm_default_mate.conf /etc/lightdm/lightdm.conf.d/
</pre>

* enable login by NIS users
<pre>
/bin/cp -v etc/lightdm_enable_nis_login.conf /etc/lightdm/lightdm.conf.d/
</pre>

* restart lightdm
<pre>
systemctl stop gdm
systemctl restart lightdm
</pre>

= Install libpng12.so.0 =

Quartus 16 needs libpng12:

<pre>
wget http://mirrors.kernel.org/ubuntu/pool/main/libp/libpng/libpng12-0_1.2.54-1ubuntu1_amd64.deb
dpkg --install libpng12-0_1.2.54-1ubuntu1_amd64.deb
</pre>

= Install google-chrome =

<pre>
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
dpkg -i google-chrome-stable_current_amd64.deb
</pre>

confirm autoupdate is enabled, observe dl.google.com is present in the list of repositories:
<pre>
apt update
...
Get:5 https://dl.google.com/linux/chrome/deb stable/main amd64 Packages [1,094 B]
...
</pre>

FOLLOWING IS OBSOLETE:

Instructions from here:
https://www.ubuntuupdates.org/ppa/google_chrome?dist=stable

<pre>
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-tmp.list'
apt update
apt install google-chrome-stable
/bin/rm -f /etc/apt/sources.list.d/google-tmp.list
</pre>

= Install amanda client =

ONLY ON MACHINES THAT HOST HOME DIRECTORIES

* apt install amanda-client
* edit /etc/amandahosts
<pre>
amanda.triumf.ca amanda amdump
</pre>
* check permissions on /etc/amandahosts:
<pre>
root@daq00:/var/log/amanda# ls -l /etc/amandahosts
-rw------- 1 backup backup 49 Jan 27 10:48 /etc/amandahosts
</pre>
* fix if needed: chown backup.backup /etc/amandahosts; chmod a= /etc/amandahosts; chmod u=wr /etc/amandahosts
* edit /etc/amanda-security.conf, add this line:
<pre>
runtar:gnutar_path=/usr/bin/tar
</pre>

On the amanda machine:

* in amanda disklist, use dump type "bsdtcp-comp-user-tar"
* su - amanda and run amcheck -c daily daq00
<pre>
-bash-4.1$ amcheck -c daily daq00

Amanda Backup Client Hosts Check
--------------------------------
Client check: 1 host checked in 0.092 seconds. 0 problems found.

(brought to you by Amanda 3.3.7p1.git.685ff76d)
</pre>

= Enable rc.local =

For reasons unknown, Ubuntu LTS 20.04 does not enable /etc/rc.local. Do this:

<pre>
cd ~/git/scripts
git pull
cp -n -v etc/rc.local /etc/
chmod a+rx /etc/rc.local
cp etc/rc-local.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable rc-local
systemctl start rc-local
systemctl status rc-local
</pre>

= Remove unwanted packages =

<pre>
apt purge bash-completion # broken, adds unwanted "\" if "ls -l $ROOTSYS/<tab>"
apt purge zsys # broken, do not use
apt purge sddm # login manager
apt purge avahi-daemon avahi-autoipd # not sure what it does, observed using 100% CPU
apt purge modemmanager # probes all serial ports to see if it's a modem
</pre>

= Disable unwanted services =

<pre>
systemctl disable mpd
systemctl disable snapd
systemctl disable ModemManager
systemctl --global mask tracker-extract-3.service
systemctl --global mask tracker-miner-fs-3.service
systemctl daemon-reload
</pre>

= Disable sleep and suspend =

note: we see some computers randomly shutdown or go to sleep, log files indicates the "sleep" or "suspend" button was pushed by user, but no such buttons actually exist. this is the fix for this:

<pre>
systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target systemd-suspend.service systemd-hybrid-sleep.service
</pre>

= Enable crontab @reboot for MIDAS =

startup scripts have a bug - cron @reboot entries for normal users can run before autofs is ready, so if the home directory is on autofs/NFS, it cannot be accessed and the cron job fails. If MIDAS is supposed to be started by cron @reboot, it will not start (there *will* be an error message in /var/log/cron).

<pre>
mkdir /etc/systemd/system/cron.service.d
echo -e "[Unit]\nAfter=ypbind.service autofs.service\n" > /etc/systemd/system/cron.service.d/local.conf
systemctl daemon-reload
systemctl cat cron.service
</pre>

Explore the systemd dependency tree using "systemctl list-dependencies" maybe with "--all".

Visualize the exact boot sequence from previous boot: "systemd-analyze plot > xxx.svg", look at the svg file using a web browser.

Crontab entry to start midas: (install in the midas user crontab, not root crontab)

<pre>
su - midasuser
crontab -l
#@reboot /bin/bash -l -c "/home/trinat/bin/start-daq-applications"
#@reboot /bin/tcsh -c "/home/trinat/bin/start-daq-applications"
</pre>

= Install apache httpd proxy for midas and elog =

This will configure the HTTPS/SSL certificate using "certbot" and "letsencrypt" and configure an HTTPS web server using apache2.

First, configure apache2:

* execute these commands:
<pre>
apt -y install apache2
cd /etc/apache2
</pre>
* create new file conf-available/ssl-daq14.conf # use actual hostname instead of daq14
<pre>
SSLSessionCache shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout 300
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
SSLCryptoDevice builtin
</pre>
* create new file sites-available/daq14-ssl.conf # use actual hostname instead of daq14
<pre>
<IfModule mod_ssl.c>
<VirtualHost *:443>
ServerName daq14.triumf.ca
DocumentRoot /var/www/html
ErrorLog /var/log/apache2/daq14.log
SSLEngine on
# note SSLProtocol, SSLCipherSuite and some other settings are overwritten by /etc/letsencrypt/options-ssl-apache.conf
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SEED:!IDEA:!RC4
## use port specified in elogd.cfg
#ProxyPass /elog/ http://localhost:8082/ retry=1
## use mhttpd port
#ProxyPass / http://localhost:8080/ retry=1
Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"
<Location />
SSLRequireSSL
AuthType Basic
AuthName "DAQ password protected site"
Require valid-user
# create password file: touch /etc/apache2/htpasswd
# to add new user or change password: htpasswd /etc/apache2/htpasswd username
AuthUserFile /etc/apache2/htpasswd
RequestHeader set X-Remote-User %{REMOTE_USER}s
</Location>
#SSLCertificateFile /root/server.cert
#SSLCertificateKeyFile /root/server.key
</VirtualHost>
</IfModule>
</pre>
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* enable ssl module and new configurations:
<pre>
a2enmod ssl
a2enmod headers
a2enmod proxy
a2enmod proxy_http
a2enconf ssl-daq14
a2ensite daq14-ssl
</pre>
* disable default ssl sites
<pre>
a2dissite 000-default-le-ssl
a2dissite 000-default
ls -l /etc/apache2/sites-enabled/ ### should show only daq14-ssl.conf
</pre>
* check that there are no syntax problems
<pre>
apache2ctl configtest
</pre>
* enable and start apache2:
<pre>
systemctl enable apache2
systemctl restart apache2
systemctl status apache2
</pre>
* apache2 may fail to start, look in /var/log/apache2/error.log and /var/log/apache2/daq14.log
* if it says "Failed to configure ... certificate", proceed to the step for setting certbot.
* try to access https://daq14.triumf.ca
** you should see a complaint about self-signed certificate
** you should see a request for password (do not login yet)
** if you get "connection refused", HTTPS port 443 may need to be enabled in the local firewall, look at documentation for ufw.
Second, configure certbot:

(Note: as of 2018-01-18 certbot requires use of http port 80 to get the initial https certificate,
renewal can continue to use the https port 443)

(Note: as of 2019-01-?? certbot requires use of port 80 for renewals)

(Note: unsurprisingly, this requires outside access to connect with letsencrypt, so won't work if PC is only accessible from on-site network)

* check that port 80 is not used by anything:
* netstat -an | grep LISTEN | grep ^tcp | grep 80
* lsof -P | grep -i tcp | grep LISTEN | grep 80
* if lsof reports that apache2 is listening on port 80, follow the apache2 instructions above (remove "listen 80" from apache2.conf

* install certbot (if necessary open tcp port 80 in the firewall, see documentation for ufw):
<pre>
apt install certbot python3-certbot-apache
certbot certonly --standalone --installer apache
</pre>
* then answer questions:
* "activate HTTPS for daq14.triumf.ca" - say ok
* "enter email address" - enter your own email address
* "please read terms..." - read the terms and say "agree"
* it will take a few moments...
* "congratulations..." - say ok.
<pre>
certbot install --apache --cert-name daq14.triumf.ca
</pre>
* then answer questions:
* "choose redirect..." - say "1" (no redirect)
* look inside /etc/apache2/sites-enabled/daq14-ssl.conf to see that SSLCertificateFile & co point to certbot certificates in
/etc/letsencrypt/live/daq14.triumf.ca/
* to check current renewal and to update the certbot config file in /etc/letsencrypt/renewal, run this:
<pre>
certbot renew --standalone --installer apache --force-renewal
</pre>

NOTE: this certificate will expire in 3 months, automatic renewal should work with current version of certbot

Third, activate password protection:

* as shown in the config file above, create password file and initial user: (replace "midas" with specific username)
<pre>
touch /etc/apache2/htpasswd
htpasswd /etc/apache2/htpasswd midas
</pre>

* restart apache2
<pre>
systemctl restart apache2
systemctl status apache2
</pre>

From here:
* enable proxy for MIDAS mhttpd - uncomment redirect in the config file above
* enable proxy for ELOG - ditto
<pre>
a2enmod proxy
a2enmod proxy_http
apache2ctl configtest
systemctl restart apache2
</pre>

* try accessing MIDAS https://daq14.triumf.ca/ (make sure mhttpd is running)
* if it's not working, check odb setting FIXME!
* try accessing ELog https://daq14.triumf.ca/elog/ (make sure elogd is running)
* if it's not working, check elogd.cfg file and make sure
<pre>
SSL = 0
</pre>

NOTE: if certbot fails with errors about 'module' object has no attribute 'pyopenssl',
try this: pip install requests==2.6.0

== generate self-signed certificate ==

<pre>
# cd $HOME
# openssl req -nodes -new -x509 -keyout server.key -out server.cert -days 1001
...+....+..+..........+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+..+...+.........+......+.+...+...+.....+...............+.........+...+.+......+...+...........+....+...+..+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+......+.+...+..+.......+..+...+.......+......+...+..+...+......+....+...............+..+...+....+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
......+......+.+..+......+.+......+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.....+......+.+.........+......+.....+.+..+...+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+....+......+.....+...+...+.......+..+.+........+.+...+......+..+..........+..+.+...........+...+.......+......+.....+.......+...+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:CH
State or Province Name (full name) [Some-State]:Geneve
Locality Name (eg, city) []:CERN
Organization Name (eg, company) [Internet Widgits Pty Ltd]:CERN
Organizational Unit Name (eg, section) []:ALPHA experiment
Common Name (e.g. server FQDN or YOUR name) []:alphacpc05.cern.ch
Email Address []:
root@alphacpc05:~#
root@alphacpc05:~#
root@alphacpc05:~# ls -l
-rw-r--r-- 1 root root 1375 juil. 10 21:43 server.cert
-rw------- 1 root root 1708 juil. 10 21:42 server.key
root@alphacpc05:~# systemctl restart apache2
</pre>

= Enable elog PDF preview =

NOTE: looks like U-24 already has this correctly.

see https://stackoverflow.com/questions/52998331/imagemagick-security-policy-pdf-blocking-conversion

* xemacs -nw /etc/ImageMagick-6/policy.xml
* remove this section at the end:
<pre>

<policy domain="coder" rights="none" pattern="PS" />
<policy domain="coder" rights="none" pattern="PS2" />
<policy domain="coder" rights="none" pattern="PS3" />
<policy domain="coder" rights="none" pattern="EPS" />
<policy domain="coder" rights="none" pattern="PDF" />
<policy domain="coder" rights="none" pattern="XPS" />
</pre>

= Install Jupyter notebook =

<pre>
From https://jupyter.org/install
apt install python3-pip
pip install jupyterlab
pip install notebook
~/.local/bin/jupyter notebook
watch the http://localhost:8888 URL that it printed
say "no" to offer to start firefox (it will not work!)
URL is: http://localhost:8888/tree?token=xxx
from the machine where you are running the web browser (i.e. google-chrome), run (replace trinat@trinatdaq with the username and machine name where you started jupyter)
open a new shell and run: ssh -v trinat@trinatdaq -L 8888:localhost:8888
in the web browser, open http://localhost:8888
this gives us the login page
in the password or token entry field, put the token from the "tree?token=xxx" above (printed by jupyter on startup)
push button "login"
jupyter page should open with the list of files in the trinat home directory
congratulate Brian with full success
</pre>

= Install ZFS quota report =

If there are any ZFS volumes, install script to report disk and quota usage

<pre>
cd ~/git/scripts/quotareport
git pull
mkdir /var/www/html/zfsquotareport
cp -pv ~/git/scripts/quotareport/sorttable.js /var/www/html/zfsquotareport/
ln -s $PWD/zfsquotareport.perl /etc/cron.daily/
touch /etc/crontab
</pre>

If httpd is configured to redirect "/" to MIDAS mhttpd:
* add following to /etc/apache2/sites-enabled/xxx-ssl.conf in front of "ProxyPass / ..."
* run "systemctl reload apache2"
<pre>
## do not proxy zfs quota report directory
ProxyPass /zfsquotareport/ !
</pre>

= Install PHP =

* apt install php libapache2-mod-php
* systemctl restart apache2
* create /var/www/html/info.php
<pre>
<?php

phpinfo();
</pre>
* open https://daq00.triumf.ca/info.php

= Configure TRIUMF printers =

<pre>
systemctl stop cups
systemctl stop cups-browsed.service
systemctl disable cups
systemctl disable cups-browsed.service
systemctl stop snap.cups.cupsd.service
systemctl stop snap.cups.cups-browsed.service
systemctl disable snap.cups.cupsd.service
systemctl disable snap.cups.cups-browsed.service
echo "ServerName printers.triumf.ca" > /etc/cups/client.conf
lpstat -a
</pre>

= Enable core dumps =

By default, Ubuntu LTS 20.04 installs the apport package
which disabled core dumps from user applications. (google it up!).
It is not meant to do this and documentation claims that
it is not installed and not enabled by default. Oh, well...

<pre>
apt purge apport
apt autoremove ### will remove apport-symptoms and a few other packages
</pre>

After this, core dumps are written to file "core" in the current directory.
See /proc/sys/kernel/core_pattern and /proc/sys/kernel/core_uses_pid.

Enable core dump file names to include process id, add following to /etc/rc.local

<pre>
echo "echo 1 > /proc/sys/kernel/core_uses_pid" >> /etc/rc.local
</pre>

= Enable debugger =

By default, Ubuntu LTS 20.04 does not permit debugger to attach and debug
already running programs. To enable it, add following to /etc/rc.local

<pre>
echo "echo 0 > /proc/sys/kernel/yama/ptrace_scope" >> /etc/rc.local
</pre>

= Disable Ubuntu Pro nag =

best I can tell, impossible at this time.

== do not do this ==

!!! does nothing !!!

<pre>
pro config set apt_news=false
</pre>

== do not do this ==

!!! breaks automatic updates because 20apt-esm-hook.conf is missing !!!

If "apt upgrade" requests Ubuntu Pro or esm-apps, disable the nag:
<pre>
/bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf
</pre>

== do not do this ==

!!! likely same as above, breaks automatic updates !!!

* comment out /etc/apt/apt.conf.d/20apt-esm-hook.conf

== do not do this ==

!!! removes too many packages !!!

<pre>
apt remove ubuntu-pro-client
</pre>

= Update packages =

<pre>
apt update # update package list
apt upgrade # install updated packages and update "kept back" packages
apt autoremove # remove packages that apt thinks should be removed
</pre>

= Remove obsolete packages =

DO NOT DO THIS, IT REMOVES TOO MUCH !!!
<pre>
apt list '~o'
apt purge '~o'
</pre>

= Cleanup residual configs =

<pre>
apt list '~c'
apt purge '~c'
</pre>

= Install firefox-esr =

install firefox-esr here

= Finish installation =

Congratulations. There is nothing more to do!

* reboot
<pre>
shutdown -r now
</pre>

= Update to new version of Ubuntu =

* run "do-release-upgrade -c"
* if it does not report new release Ubuntu 24, check /etc/update-manager/release-upgrades has "Prompt=lts"

== Update Ubuntu LTS 20.04 to LTS 22.04 ==

<pre>
apt remove zsys
</pre>

=== daqubuntu ===

<pre>
# reboot to clear out all updates
# vi /etc/update-manager/release-upgrades # set "Prompt=normal"
# do-release-upgrade -c
Checking for a new Ubuntu release
New release '22.04 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
# do-release-upgrade
...
say yes...
...
login.defs, say "Y" (erase local changes, use packaged version)
/etc/systemd/resolved.conf, say "Y" (same as above)
firefox snap, say yes
unable to reach snap store, say "skip"
/etc/gmond.conf, say "Y"
/var/yp/Makefile, say "install the package maintainer's version"
/etc/ypserv.conf, same thing
/etc/ypserv.securenets, same thing
/etc/default/nis, same thing
/etc/speech-dispatcher/modules/mary-generic.conf, same thing
/etc/apt/apt.conf.d/50unattended-upgrades, same thing
...
278 packages are going to be removed, say yes
...
restart required, say yes
...
no ping... yes ping...
...
ssh daqubuntu, ok
apt update, fail, DNS does not work, "host security.ubuntu.com" does not resolve.
fix resolver per https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Disable_NetworkManager
apt update, apt upgrade now works, 0 packages to update
NIS does not work.
</pre>

=== midm9a ===

<pre>
login.defs
firefox snap
gmond.conf
ypserv
/etc/default/nis
unattended-upgrades
amanda-security.conf
remove obsolete (no)
reboot
configure dns
reenable nis
</pre>

=== daq17 ===

<pre>
firefox snap
imagemagick policy.xml
gmond.conf
chrony.conf
/var/yp/Makefile
ypserv.conf
ypserv.securenets
/etc/default/nis
50unattended-upgrades
</pre>

=== daq00 ===

per https://serverpilot.io/docs/how-to-upgrade-ubuntu-20.04-to-22.04/

<pre>
do-release-upgrade -f DistUpgradeViewNonInteractive
</pre>

if it exists "too soon" without doing anything, run it without "-f xxx", most likely it does not like something about this machine. in case of daq00 it did not like how the EFI partitions were mounted. after fixing it, non-interactive upgrade was successful.

=== isdaq08 ===

* prepare
<pre>
cd ~/git/scripts
git pull
cd ~
apt -y install debsums
</pre>
* check for modified config files that make upgrade unhappy, deal with all files reported by debsums.
<pre>
root@isdaq08:~# debsums -ce
/etc/ganglia/gmond.conf
/etc/yp.conf
/etc/apt/apt.conf.d/10periodic
root@isdaq08:~#
</pre>
* restore original /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
</pre>
* apt remove ganglia-monitor
* apt remove nis
* "debsums -ce" is now empty

Run the upgrade:

* do-release-upgrade -f DistUpgradeViewNonInteractive

Post upgrade:

* configure DNS
* apt -y install linux-generic-hwe-22.04
* /bin/cp -v ~/git/scripts/etc/99apt-conf-ko /etc/apt/apt.conf.d/ # restore nightly updates
* /bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf # remove the ubuntu-pro nag
* install missing packages
* restore ganglia
* restore nis
* check zpool status, may need zpool upgrade
* reboot

== upgrade U-22 to U-24 ==

generic instructions. below are notes from upgrades of specific machines.

NOTE: at CERN saw installation getting stuck on restart of autofs (automount, rsyslogd 100% CPU, huge /var/log/syslog), stop autofs before upgrade?
<pre>
systemctl stop autofs
</pre>

<pre>
cleanup zsys
</pre>

<pre>
maybe: if snap is already removed, remove firefox to prevent snap from reinstalling.
install non-snap firefox-esr
</pre>

<pre>
remove thunderbird to prevent installation of snap thunderbird in U-24
</pre>

<pre>
mount EFI partition as /boot/efi, otherwise upgrade bombs
usually mount /dev/sda1 /boot/efi
</pre>

<pre>
debsums -ce
apt -y remove desktop-base # causes installer crash
apt -y remove thunderbird # avoid forced conversion to snap
apt update
apt -y upgrade
apt -y autoremove
do-release-upgrade -c # confirm upgrade will be to U-24
do-release-upgrade
# say "y" to all questions
# after installation starts, accept default answers to all questions
# should run for about 1 hour or so
# after upgrade finishes
apt update
apt -y upgrade
apt -y autoremove
shutdown -r now
</pre>

post upgrade:

* check zpool status, may need zpool upgrade
* cd /etc/apt/sources.list.d, reenable 3rd party repos (mozilla, google, etc)

if installation bombs:

<pre>
apt update
apt upgrade # will tell us to run "apt --fix-broken install", do it
apt --fix-broken install
apt update
apt upgrade # should resume the upgrade, will run for a long time
apt update
apt upgrade # should do nothing
apt autoremove
shutdown -r now
</pre>

=== daqubuntu, U-24 ===

* prepare
<pre>
cd ~/git/scripts
git pull
cd ~
apt -y install debsums
</pre>
* check for modified config files that make upgrade unhappy, deal with all files reported by debsums.
<pre>
root@daqubuntu:~# debsums -ce
/etc/ganglia/gmond.conf
debsums: missing file /etc/init.d/nis (from nis package)
/etc/default/nis
/etc/ypserv.conf
/etc/ypserv.securenets
/var/yp/Makefile
/etc/update-manager/release-upgrades
/etc/apt/apt.conf.d/10periodic
/etc/yp.conf
root@daqubuntu:~#
* restore original /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
</pre>
* apt remove ganglia-monitor
* apt remove nis
* apt autoremove
* restore original release-upgrades: "Prompt: lts"
* "debsums -ce" is now empty

Check for upgrade:

<pre>
root@daqubuntu:~# do-release-upgrade -c
Checking for a new Ubuntu release
There is no development version of an LTS available.
To upgrade to the latest non-LTS development release
set Prompt=normal in /etc/update-manager/release-upgrades.
root@daqubuntu:~#
</pre>

Run the upgrade:

* do-release-upgrade -f DistUpgradeViewNonInteractive

Post upgrade:

* configure DNS
* apt -y install linux-generic-hwe-22.04
* /bin/cp -v ~/git/scripts/etc/99apt-conf-ko /etc/apt/apt.conf.d/ # restore nightly updates
* /bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf # remove the ubuntu-pro nag
* install missing packages
* restore ganglia
* restore nis
* check zpool status, may need zpool upgrade
* cd /etc/apt/sources.list.d, reenable 3rd party repos (mozilla, google, etc)
* reboot

=== daq14, U-20-22-24 ===

* apt update, apt upgrade
* apt -y install linux-image-generic-hwe-20.04 linux-tools-virtual-hwe-20.04 ### install kernel 5.15
* shutdown -r now
* stuck waiting for daq14 to shutdown...
* reboot into kernel 5.15
* ???
<pre>
cd ~/git/scripts
git pull
cd ~
apt -y install debsums
</pre>
* debsums -ce
<pre>
/etc/apache2/ports.conf
/etc/dnsmasq.conf
/etc/ganglia/gmond.conf
/etc/yp.conf
/etc/sudoers
</pre>
* apache2 restore original ports.conf, uncomment "Listen 80"
* cp -pv /etc/dnsmasq.conf.dpkg-dist /etc/dnsmasq.conf
* apt remove ganglia-monitor
* edit /etc/yp.conf, remove everything after "# ypserver ypserver.network.com"
* "debsums -ce" is now empty
* do-release-upgrade -f DistUpgradeViewNonInteractive
* runs for a long time
* stuck on "/etc/default/nis", type "Y", press enter, nothing for a bit, then resumes running
* finished
* configure DNS
* reboot
* have kernel 6.8
* apt update; apt upgrade
* apt upgrade guile-2.2-libs ### would not auto-update, "kept back", has to be done by hand
* apt autoremove
* debsums -ce
<pre>
debsums: missing file /etc/init.d/nis (from nis package)
/etc/default/nis
</pre>
* diff /etc/default/nis.dpkg-dist /etc/default/nis
* cp -pv /etc/default/nis.dpkg-dist /etc/default/nis
* debsums -ce
<pre>
debsums: missing file /etc/init.d/nis (from nis package)
</pre>
* we ignore this and run the update
* do-release-upgrade -c
<pre>
Checking for a new Ubuntu release
New release '24.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
</pre>
* do-release-upgrade -f DistUpgradeViewNonInteractive
* bombs out without any error messages
* in /var/log/dist-upgrade/main.log reports "Failed to find a replacement for xapp" and other packages
* apt remove xapp usrmerge ureadahead thunderbird-gnome-support
* no go, complains about even more packages.
* apt list | grep installed | grep -v jammy ### show packages installed from non-ubuntu sources
* remove all packages marked "install,local" ### ubuntu updater does not know where they came from and so cannot update them.
* apt remove desktop-base ### not happy about this package in /var/log/dist-upgrade/apt.log
* apt autoremove
* do-release-upgrade -f DistUpgradeViewNonInteractive
* running for a long time...

=== alpha04 U-20-24 ===

* apt update, apt upgrade, apt autoremove
* reboot into latest kernel (already done)
* debsums -ce
<pre>
root@alpha04:~# debsums -ce
/etc/dnsmasq.conf
/etc/ganglia/gmond.conf
/etc/default/nis
/etc/yp.conf
root@alpha04:~#
</pre>
* move /etc/dnsmasq.conf to /etc/dnsmasq.d/alpha04.conf
* apt remove dnsmasq
* apt remove ganglia-monitor
* apt remove nis
* apt autoremove
* debsums -ce ### is now empty
* do-release-upgrade -f DistUpgradeViewNonInteractive
* it runs for a long time...
* complained about /etc/fwupd config files, not sure why...
* finished
* apt update, apt upgrade, apt autoremove
* restore dnsmasq: apt install dnsmasq, systemctl status dnsmasq
* restore ganglia, per instructions
* restore NIS: apt -y install rpcbind nis, ypwhich, ypwhich -m
* zpool upgrade rpool ### also upgrade any other zfs pools, see zpool status
* remove unwanted packages, per instructions
* run gonodeinfo
* reboot
* done

== vera00 U20-22-24 ==

* everything same as daq14 for U20-22
* kernel is still 5.15
* U22-24 is going...
* stuck for a few minutes on /etc/fwupd config files
* have kernel 6.8.0-49
* same steps as daq14
* reboot
* same steps as daq14
* done

== phaarmonster U22-24 ==

* debsums -ce
<pre>
/etc/amandahosts
/etc/apache2/ports.conf
/etc/ganglia/gmond.conf
</pre>
* do-release-upgrade -c ### reports U24.04.1
* do-release-upgrade -f DistUpgradeViewNonInteractive ### bombs
* apt remove desktop-base; apt autoremove
* do-release-upgrade ### (interactive) runs ok
* bombed !!!
* apt upgrade spews errors and tells us to run "apt --fix-broken install"
* apt --fix-broken install ### runs
* bombs with thunderbird snap errors
* again... no go
* thunderbird snap complains about mounting /home, but /home is a symlink
* rm /home, mkdir /home
* again, runs ok
* asks about /etc/fwupd/fwupd.conf - say "Y" to install updated package version
* apt install completes
* apt update; apt upgrade ### running for a long time...
* finished
* install missing packages, etc
* reboot
* long wait... came back
* DNS does now work, systemd-resolved missing, apt install systemd-resolved, "configure DNS"
* done.

== isdaq10 U22-24 ==

* do-release-upgrade -f DistUpgradeViewNonInteractive
* bombed on desktop-base, "apt remove desktop-base; apt autoremove"
* bombed with errors
* running "apt --fix-broken install"
* complained about thunderbird snap
* complained about /etc/fwupd/fwupd.conf (say Y)
* finished ok
* apt update
* apt upgrade ### reports "1382 upgraded, 140 newly installed, 0 to remove and 29 not upgraded"

== iris01 ==

* debsums -ce
<pre>
/etc/ganglia/gmond.conf
/etc/default/nis
/etc/yp.conf
</pre>
* apt remove desktop-base
* apt remove thunderbird
* apt autoremove
* do-release-upgrade -f DistUpgradeViewNonInteractive
* bombed with dpkg errors:
<pre>
hplip wants wrong python
ganglia-monitor wants wrong libapr1
sssd-ad wants bunch of wrong libraries
xemacs21-mule
libnfsidmap1 wants libldap2
adn so forth...
</pre>
* apt --fix-broken install -y
* bombs on ganglia - ganglia group absent from /etc/groups
* fix group ganglia by hand, remove ganglia group from NIS on isdaq00
* apt --fix-broken install
* apt upgrade

== iris00 ==

* stuck on "autofs: restarting", login as root, kill iris midas, kill automount, systemctl restart autofs, noble got unstuck, ctrl-c systemctl restart autofs
* noble running...
* bombed with dpkg errors
* check ganglia user, group - both ok
* apt --fix-broken install
* apt upgrade

== tigstore01 ==

* no bomb-out

== midm9b ==

* apt remove desktop-base thunderbird
* bombed
* apt --fix-broken-install
* apt upgrade

= Upgrade to new version of Debian =

https://www.debian.org/releases/bookworm/amd64/release-notes/ch-upgrading.en.html

== 32-bit VME processor Debian 11 to 12 to 13 ==

* cd git/scripts; git pull; cd ~
* apt update
* apt upgrade
* edit /etc/apt/sources.list
<pre>
deb http://deb.debian.org/debian/ trixie main
#deb http://deb.debian.org/debian/ bookworm main
#deb http://deb.debian.org/debian/ bullseye main
#deb http://deb.debian.org/debian/ buster main
#deb-src http://deb.debian.org/debian/ bullseye main
</pre>
* apt update
* apt upgrade --without-new-pkgs
* apt full-upgrade
* apt list '~c'; apt purge '~c' # purge left-over config files [residual-config]
* reboot

= Ubuntu package manager =

* apt-get install xxx # install package xxx
* apt-get update
* apt-get upgrade
* apt-get dist-upgrade
* apt-get autoremove # remove automatically installed packages required by a removed package
* apt-get remove xxx # remove package xxx
* apt-cache search . # list all available packages
* apt-cache show "." | grep ^Package # list al available packages
* apt-cache madison root-system # show all available versions of package root-system
* apt list # list all installed packages
* dpkg --listfiles libpng16-16 # list all files from this package
* apt list --installed # list all installed packages
* dpkg -S /bin/bash # what package provides this file?
* dpkg -L bash # what files provided by this package?
* debsums -ce # show modified config files
* apt-config dump # show apt configuration
* apt purge '~c' # purge all [residual-config] packages
* ls -l /var/lib/dpkg/info/ # show post-install scripts

= Ubuntu zsys =

NOTE: DO NOT USE ZSYS, see https://github.com/ubuntu/zsys/issues/218 and https://github.com/ubuntu/zsys/issues/230

* scripted removal of old snapshots (replace "echo zfs destroy" with "zfs destroy")
<pre>
zfs list -t all | cut -f1 -d " " | grep autozsys | xargs -n1 echo zfs destroy
</pre>

* manual removal of old snapshots
<pre>
zsysctl show
zsysctl state remove xy69ye -s
zsysctl state remove xy69ye
zsysctl state remove xy69ye -u wheel
</pre>
* apt remove zsys

NOTE: old zsys snapshots must be cleaned manually, "zsysctl state remove xxx --system" is broken and does not remove user data snapshots

* manages system snapshots
* documentation: https://github.com/ubuntu/zsys
* documentation: (go to next article via link "newer" at the bottom) https://didrocks.fr/2020/05/21/zfs-focus-on-ubuntu-20.04-lts-whats-new/
* ubuntu 20.04 bug, too many snapshots cause /boot to become full and updates fail. https://github.com/ubuntu/zsys/issues/155
* solution: use custom /etc/zsys.conf, limit number of snapshots to 10, see trinatdaq:/etc/zsys.conf
* zsys commands:
<pre>
update-grub # list of all snapshots, errors if some snapshots are broken
zsysctl state remove lnc0k7 --system # remove snapshot
xemacs -nw /etc/zsys.conf; zsysctl service reload; zsysctl service gc # cause gc to run with new settings in zsys.conf
zfs list -r -t snapshot -o name,used,referenced,creation bpool/BOOT # list snapshots
zsysctl show # show snapshots
</pre>

= Ubuntu cloning =

to clone a ubuntu image:

<pre>
cd /nfsroot/lxcpet
emacs -nw etc/hostname ### change hostname
emacs -nw etc/mailname ### change hostname (debian 11)
emacs -nw etc/defaultdomain ### change the NIS domainname
emacs -nw etc/yp.conf ### change the NIS server
cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
emacs -nw root/.ssh/authorized_keys ### update root ssh keys
</pre>

= Ubuntu boot loader =

== maintenance commands ==

* update-initramfs -v -u
* grub-install /dev/sda

= Convert from single to dual mirrored ZFS SSD =

Assuming Ubuntu LTS 22.04 with "instal on ZFS" option, we will
add a second SSD, configure ZFS to use both SSDs in mirrored
configuration and setup grub to boot from either SSD. This
is intended to create a full redundant system where failure
of either SSD does not break the system.

== partition ==

* identify first SSD
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* connect second SSD of identical size
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
/dev/sdb WD Blue SA510 2.5 250GB 22243Z803852 25 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* if second SSD is not autodetected, reboot
* Clone partition table automatically
If both SSDs are identical size, use this simpler method of duplicating the partition table:
<pre>
root@midm9b:~# sfdisk -d /dev/sda > part_table
root@midm9b:~# grep -v ^label-id part_table | sed -e 's/, *uuid=[0-9A-F-]*//' | sfdisk /dev/sdb
</pre>
The grep and sed in the second command are there to prevent disk ID and partition IDs from being cloned. Alternatively the part_table file can be edited manually to remove the label-id line and the uuid entries from the individual partitions.

* Clone partition table manually (e.g. for different size disks)
* list partition table of first SSD:
<pre>
root@midm9b:~# fdisk -l /dev/sda
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>
* create identical partitions on second SSD, use sector numbers from above.
<pre>
root@midm9b:~# gdisk /dev/sdb
GPT fdisk (gdisk) version 1.0.8

Partition table scan:
MBR: not present
BSD: not present
APM: not present
GPT: not present

Creating new GPT entries in memory.

Command (? for help): n
Partition number (1-128, default 1):
First sector (34-488397134, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-488397134, default = 488397134) or {+-}size{KMGTP}: 1050623
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): ef00
Changed type of partition to 'EFI system partition'

Command (? for help): n
Partition number (2-128, default 2):
First sector (34-488397134, default = 1050624) or {+-}size{KMGTP}:
Last sector (1050624-488397134, default = 488397134) or {+-}size{KMGTP}: 5244927
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 8200
Changed type of partition to 'Linux swap'

Command (? for help): n
Partition number (3-128, default 3):
First sector (34-488397134, default = 5244928) or {+-}size{KMGTP}:
Last sector (5244928-488397134, default = 488397134) or {+-}size{KMGTP}: 9439231
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): be00
Changed type of partition to 'Solaris boot'

Command (? for help): n
Partition number (4-128, default 4):
First sector (34-488397134, default = 9439232) or {+-}size{KMGTP}:
Last sector (9439232-488397134, default = 488397134) or {+-}size{KMGTP}:
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): bf00
Changed type of partition to 'Solaris root'

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sdb.
The operation has completed successfully.
root@midm9b:~# fdisk -l /dev/sda /dev/sdb
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root

Disk /dev/sdb: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: EB251739-30C6-422F-A505-5887B5A0B603

Device Start End Sectors Size Type
/dev/sdb1 2048 1050623 1048576 512M EFI System
/dev/sdb2 1050624 5244927 4194304 2G Linux swap
/dev/sdb3 5244928 9439231 4194304 2G Solaris boot
/dev/sdb4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>

== update ZFS pools ==

* identify second SSD partitions
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part2
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part2 -> ../../sdb2
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
</pre>
* convert bpool from single disk to mirrored disk:
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~# zpool attach bpool 99e03dc0-7d4d-f24b-8fa1-f042b9f135db /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3
root@midm9b:~# zpool status bpool
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors
</pre>
* convert rpool
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
root@midm9b:~# zpool attach rpool f6fd54f8-3af7-b943-ae3d-a4e480537fb9 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4
root@midm9b:~# zpool status rpool
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Jan 20 19:40:45 2023
5.83G scanned at 664M/s, 2.92M issued at 332K/s, 9.11G total
0B resilvered, 0.03% done, no estimated completion time
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~#
</pre>
* wait for resilver to complete
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
scan: resilvered 9.65G in 00:00:36 with 0 errors on Fri Jan 20 19:41:21 2023
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
</pre>

== update boot loader ==

INSTALL SYSLINUX: https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#EFI_boot_using_syslinux

DO *NOT* DO THE FOLOWING:

* create and mount EFI partitions:
<pre>
root@midm9b:~# mkfs.msdos /dev/sdb1
root@midm9b:~# mkdir /boot/efi-sda
root@midm9b:~# mkdir /boot/efi-sdb
root@midm20c:~# blkid | grep vfat ### identify UUID
/dev/sdb1: UUID="DD89-5081" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="d0cb6be4-2f67-5b42-9b26-9e6905e9f774"
/dev/sdc1: UUID="D970-86BA" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="e6d3b5b9-a512-44a2-9205-1a4db06ed2a2"
/dev/sda1: UUID="DDA1-044C" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="6dc9dff0-1c13-8045-a906-7803d3074c70"
root@midm20c:~# cat /etc/fstab | grep vfat ### add mount points with correct UUID
#UUID=D970-86BA /boot/efi vfat umask=0022,fmask=0022,dmask=0022 0 1
UUID=DDA1-044C /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022 0 1
UUID=DD89-5081 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022 0 1
root@midm9b:~# mount -a
root@midm9b:~# df -kl
Filesystem 1K-blocks Used Available Use% Mounted on
...
/dev/sda1 523244 13720 509524 3% /boot/efi
/dev/sdb1 523244 4 523240 1% /boot/efi-sdb
...
root@midm9b:~# rsync -av /boot/efi/ /boot/efi-sdb/
sending incremental file list
EFI/
...
root@midm9b:~# ls -l /boot/efi-sda
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~# ls -l /boot/efi-sdb
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~#
</pre>
* add systemd "nofail" flag to /etc/fstab, without this, systemd will stop booting if one SSD is missing
<pre>
daq00:~$ cat /etc/fstab | grep vfat
#UUID=31A7-24BE /boot/efi vfat umask=0022,fmask=0022,dmask=0022 0 1
/dev/sda1 /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
/dev/sdb1 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
</pre>
* setup script to update grub on second SSD, it must be run manually after every kernel update
<pre>
root@midm9b:~# ln -s ~/git/scripts/etc/update_efi_grub.perl ~/
root@midm9b:~# ~/update_efi_grub.perl -u
EFI dir: /boot/efi-sda
/boot/efi-sda: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sda/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sda: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sda/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
EFI dir: /boot/efi-sdb
/boot/efi-sdb: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sdb/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sdb: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sdb/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
root@midm9b:~#
</pre>

= Disable NetworkManager =

== Debian-12 ==

* use netplan, https://www.debian.org/doc/manuals/debian-reference/ch05.en.html#_the_modern_network_configuration_for_cloud
* apt install netplan.io
* systemctl enable systemd-networkd
* systemctl restart systemd-networkd
* create /etc/netplan/50-dhcp.yaml
<pre>
network:
version: 2
ethernets:
all-en:
match:
name: "en*"
dhcp4: true
dhcp6: true
ignore-carrier: true ### do not drop IP address if network link drops
</pre>
* netplan apply
* netplan try
* ifconfig -a ### to check IP address settings
* netstat -rn ### to check default route
* cat /etc/resolv.conf ### to check DNS
* ls -l /run/systemd/netif/leases ### systemd-networkd dhcp leases
* NOTE: without "ignore-carrier" it will drop the IP address if network link drops, re-do dhcp when links comes back
* NOTE: wait-network-online will wait for all interfaces to get an IP address

== Ubuntu-20 ==

NOTE: THIS IS BROKEN IN UBUNTU LTS 22.04

NetworkManager is useful for configuring dynamic
network interfaces, i.e. laptops that often move
between networks, or connect to multiple choice
of wifi networks, etc.

For machines with statically configured network interfaces,
NetworkManager is not necessary.

As it has been observed to become confused and observed
to malfunction when network links go up and down (it keeps
unnecessarily reconfiguring the ip address, etc), it can
be usefuil to disable it.

* list all network interfaces
<pre>
# /bin/ls -1 /sys/class/net/
enp0s31f6
lo
</pre>
* edit /etc/network/interfaces:
<pre>
rename enp0s31f6=eth0
auto eth0
iface eth0 inet static
address 142.90.120.94/19
gateway 142.90.100.18
</pre>
* statically configure systemd-resolved
** create /etc/systemd/resolved.conf.d/resolved.conf with this contents:
<pre>
[Resolve]
DNS=142.90.100.19
Domains=triumf.ca
</pre>
** systemctl restart systemd-resolved
** resolvectl
** systemd-analyze cat-config systemd/resolved.conf
* disable NetworkManager
<pre>
systemctl disable NetworkManager
</pre>
* reboot

== U-22, U-24 ifup-ko ==

Network configuration of modern linux is confused. There are at least 3 configuration methods, each with different shortcomings:
* the old ifup method is barely documented
* NetworkManager is well documented and tooled, but sometimes does strange things
* systemd-networkd is mysterious, and likely to do strange stuff, like all systemd stuff
* netplan is the latest method, configuration is simple but uses NetworkManager or systemd-networkd as backend.

This is a solution for a specific situation of fixed computer with one fixed wired interface and maybe one or more additional interfaces for fixed wired private networks.

Install /etc/ifup-ko, edit it with IP addresses of main and additional interfaces, let systemd run it in the right place in the boot sequence replacing NetworkManager and NetworkManager-wait-online .

As bonus, /etc/ifup-ko waits up to 10 seconds for the main interface link to come up. If this is not needed, comment it out from the script.

* prepare
<pre>
cd ~/git/scripts
git pull
cd ifup-ko
make install
</pre>
* confirm interface names
<pre>
systemctl start ifup-ko ### should finish immediately or after 10 seconds
systemctl status ifup-ko -n 1000 ### observe list of interfaces is correct, name of main interface is correct
</pre>
* edit /etc/ifup-ko
** add host IP address to the "ifconfig" line
** add gateway IP address to the "ip route add" line
* test
<pre>
systemctl start ifup-ko ### should finish immediately
systemctl status ifup-ko -n 1000 ### observe everything is configured as expected
</pre>
* cut-over
<pre>
systemctl disable networkd-dispatcher
systemctl disable NetworkManager
systemctl disable wpa_supplicant # if no Wifi or Wifi not in use
</pre>
* reboot

= Disable systemd-networkd =

On netbooted machines, systemd-networkd should be disabled - when "apt upgrade" runs
and needs to update and configure systemd, it will stop systemd-networkd, which
will stop the network, which will stop the NFS mounted root filesystem, which will stop
the machine.

<pre>
systemctl disable systemd-networkd.service
systemctl disable systemd-networkd.socket
systemctl mask systemd-networkd.service
systemctl mask systemd-networkd.socket
</pre>

= Configure ECC memory =

== Configure EDAC ==

* apt install edac-utils rasdaemon

=== Intel i3-2120 ===
<pre>
root@musr00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X9SCL/X9SCM
root@musr00:~# edac-ctl --status
edac-ctl: drivers not loaded.
</pre>

=== Intel E-2236 ===
<pre>
root@daq00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SCM-F
root@daq00:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq00:~# edac-util
edac-util: No errors to report.
root@daq00:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
</pre>
* check edac sysfs files (Intel)
<pre>
root@daq00:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_noinfo_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 max_location
-r--r--r-- 1 root root 4096 Jan 25 15:10 mc_name
drwxr-xr-x 2 root root 0 Jan 25 15:10 power
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank0
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank1
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank2
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank3
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank4
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank5
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank6
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank7
--w------- 1 root root 4096 Jan 25 15:10 reset_counters
-r--r--r-- 1 root root 4096 Jan 25 15:10 seconds_since_reset
-r--r--r-- 1 root root 4096 Jan 25 15:10 size_mb
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Jan 25 15:10 uevent
root@daq00:~#
</pre>

=== Intel E3-1270 v6 ===
<pre>
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --status
edac-ctl: drivers are loaded.
root@grsnis01:~# edac-util
edac-util: No errors to report.
root@grsnis01:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@grsnis01:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 max_location
-r--r--r-- 1 root root 4096 Feb 19 12:35 mc_name
drwxr-xr-x 2 root root 0 Feb 19 12:35 power
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank0
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank1
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank2
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank3
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank4
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank5
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank6
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank7
--w------- 1 root root 4096 Feb 19 12:35 reset_counters
-r--r--r-- 1 root root 4096 Feb 19 12:35 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 19 12:35 size_mb
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 19 12:35 uevent
root@grsnis01:~#
</pre>

=== Intel E3-1245 v6 ===

<pre>
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --status
edac-ctl: drivers are loaded.
[root@alphagdaq ~]# edac-util
edac-util: No errors to report.
[root@alphagdaq ~]# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
[root@alphagdaq ~]# ras-mc-ctl --layout
+-----------------------------------------------+
| mc0 |
| csrow0 | csrow1 | csrow2 | csrow3 |
----------+-----------------------------------------------+
channel1: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
----------+-----------------------------------------------+
[root@alphagdaq ~]# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#3channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#3channel#1 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#2channel#0 0 0
[root@alphagdaq ~]# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SSH-F
[root@alphagdaq ~]# ras-mc-ctl --summary
DBD::SQLite::db prepare failed: no such table: mc_event at /usr/sbin/ras-mc-ctl line 1129.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1130.
[root@alphagdaq ~]#
</pre>

=== AMD 3700X ===

(memory is non-ECC)

<pre>
root@daq13:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@daq13:~#
root@daq13:~#
root@daq13:~# edac-ctl --status
edac-ctl: drivers not loaded.
root@daq13:~# edac-util
edac-util: Error: No memory controller data found.
root@daq13:~# edac-util -s
edac-util: EDAC drivers loaded. No memory controllers found
root@daq13:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 2 root root 0 Jan 25 15:26 power
lrwxrwxrwx 1 root root 0 Jan 21 16:16 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Jan 21 16:16 uevent
</pre>

(memory is ECC)

<pre>
root@trinatdaq:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@trinatdaq:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@trinatdaq:~# edac-util
edac-util: No errors to report.
root@trinatdaq:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Dec 15 13:04 mc0
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
lrwxrwxrwx 1 root root 0 Dec 13 18:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dec 13 18:31 uevent
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_noinfo_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 max_location
-r--r--r-- 1 root root 4096 Dec 15 13:04 mc_name
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank4
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank5
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank6
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank7
--w------- 1 root root 4096 Dec 15 13:04 reset_counters
-rw-r--r-- 1 root root 4096 Dec 15 13:04 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Dec 15 13:04 seconds_since_reset
-r--r--r-- 1 root root 4096 Dec 15 13:04 size_mb
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Dec 15 13:04 uevent
root@trinatdaq:~#
</pre>

=== AMD 5000G ===

* no linux driver for AMD 5000-series "G" CPU
* no mention of ECC in the BIOS settings
* unclear status of ECC support in AMD documentation (sais only "pro" "G" CPUs have ECC)
* unclear status of ECC support in ASUS documentation (web page out of date)

=== AMD 5600X ===

<pre>
root@daq17:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-XE GAMING WIFI
root@daq17:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq17:~# edac-util
edac-util: No errors to report.
root@daq17:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@daq17:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Aug 19 19:27 mc0
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
lrwxrwxrwx 1 root root 0 May 10 10:11 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 May 10 10:11 uevent
root@daq17:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_noinfo_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 max_location
-r--r--r-- 1 root root 4096 Aug 19 19:27 mc_name
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank4
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank5
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank6
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank7
--w------- 1 root root 4096 Aug 19 19:27 reset_counters
-rw-r--r-- 1 root root 4096 Aug 19 19:27 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Aug 19 19:27 seconds_since_reset
-r--r--r-- 1 root root 4096 Aug 19 19:27 size_mb
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Aug 19 19:27 uevent
root@daq17:~#
</pre>

=== AMD 3955WX ===

<pre>
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --status
edac-ctl: drivers are loaded.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util
edac-util: No errors to report.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@alphasuperdaq:~/git/scripts/quotareport# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 19 root root 0 Dez 12 04:48 mc0
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
lrwxrwxrwx 1 root root 0 Dez 9 05:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dez 9 05:31 uevent
root@alphasuperdaq:~/git/scripts/quotareport#
root@alphasuperdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 max_location
-r--r--r-- 1 root root 4096 Feb 28 22:19 mc_name
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank0
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank1
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank10
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank11
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank12
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank13
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank14
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank15
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank2
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank3
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank4
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank5
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank6
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank7
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank8
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank9
--w------- 1 root root 4096 Feb 28 22:19 reset_counters
-rw-r--r-- 1 root root 4096 Feb 28 22:19 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Feb 28 22:19 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 28 22:19 size_mb
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 28 22:19 uevent
root@alphasuperdaq:~#
root@alphasuperdaq:~# ras-mc-ctl --layout
Use of uninitialized value $max_pos[3] in modulus (%) at /usr/sbin/ras-mc-ctl line 868.
Use of uninitialized value $d in numeric ge (>=) at /usr/sbin/ras-mc-ctl line 869.
Use of uninitialized value $d in sprintf at /usr/sbin/ras-mc-ctl line 872.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| mc0 |
| csrow0 | csrow1 |
| channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 | channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

0: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
root@alphasuperdaq:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#2 0 0
mc#0csrow#1channel#7 0 0
mc#0csrow#0channel#3 0 0
mc#0csrow#1channel#4 0 0
mc#0csrow#1channel#2 0 0
mc#0csrow#0channel#7 0 0
mc#0csrow#1channel#3 0 0
mc#0csrow#0channel#4 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#5 0 0
mc#0csrow#0channel#6 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#0channel#5 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#6 0 0
root@alphasuperdaq:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: ASUSTeK COMPUTER INC. model Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@alphasuperdaq:~#
</pre>

=== AMD 7700X ===

<pre>
root@dsfe05:~# apt install edac-utils
root@dsfe05:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro H13SAE-MF
root@dsfe05:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@dsfe05:~# edac-util
edac-util: No errors to report.
root@dsfe05:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@dsfe05:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 May 14 09:33 ce_count
-r--r--r-- 1 root root 4096 May 14 09:33 ce_noinfo_count
-r--r--r-- 1 root root 4096 May 14 09:33 max_location
-r--r--r-- 1 root root 4096 May 14 09:33 mc_name
drwxr-xr-x 2 root root 0 May 14 09:33 power
drwxr-xr-x 3 root root 0 May 14 09:33 rank4
drwxr-xr-x 3 root root 0 May 14 09:33 rank5
--w------- 1 root root 4096 May 14 09:33 reset_counters
-r--r--r-- 1 root root 4096 May 14 09:33 seconds_since_reset
-r--r--r-- 1 root root 4096 May 14 09:33 size_mb
-r--r--r-- 1 root root 4096 May 14 09:33 ue_count
-r--r--r-- 1 root root 4096 May 14 09:33 ue_noinfo_count
-rw-r--r-- 1 root root 4096 May 14 09:33 uevent
root@dsfe05:~#
</pre>

== Configure rasdaemon ==

<pre>
apt install rasdaemon
</pre>
<pre>
systemctl enable rasdaemon
systemctl restart rasdaemon
systemctl status rasdaemon
</pre>

<pre>
● rasdaemon.service - RAS daemon to log the RAS events
Loaded: loaded (/lib/systemd/system/rasdaemon.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-01-25 15:16:37 PST; 3min 5s ago
Main PID: 2477175 (rasdaemon)
Tasks: 1 (limit: 76958)
Memory: 17.1M
CGroup: /system.slice/rasdaemon.service
└─2477175 /usr/sbin/rasdaemon -f -r

Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Listening to events for cpus 0 to 11
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mc_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording aer_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording extlog_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mce_record events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording arm_event events
</pre>

== Get reports ==

* Intel 2x32GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --layout
+-------------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-------------------------+
channel1: | 16384 MB | 16384 MB |
channel0: | 16384 MB | 16384 MB |
----------+-------------------------+
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#0channel#1 0 0
root@daq00:~#
</pre>

* Intel 4x16GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#1 0 0
mc#0csrow#2channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#3channel#0 0 0
mc#0csrow#3channel#1 0 0
root@daq00:~#
root@daq00:~# ras-mc-ctl --layout
+-----------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-----------------------+
channel1: | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB |
----------+-----------------------+
root@daq00:~#
root@daq00:~#
root@daq00:~#
root@daq00:~# ras-mc-ctl --print-labels
ras-mc-ctl: Error: No dimm labels for Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@daq00:~#
</pre>

note: ubuntu LTS 22.04 DBD::SQLite::db error is not there.

* AMD 7700 2x32GB DDR5 ECC DIMMs

<pre>
root@dsfe05:~# systemctl status rasdaemon
● rasdaemon.service - RAS daemon to log the RAS events
Loaded: loaded (/lib/systemd/system/rasdaemon.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2024-05-14 09:36:43 PDT; 33ms ago
Process: 4088418 ExecStartPost=/usr/sbin/rasdaemon --enable (code=exited, status=0/SUCCESS)
Main PID: 4088417 (rasdaemon)
Tasks: 1 (limit: 37300)
Memory: 788.0K
CPU: 5ms
CGroup: /system.slice/rasdaemon.service
└─4088417 /usr/sbin/rasdaemon -f -r

May 14 09:36:43 dsfe05 rasdaemon[4088417]: ras:aer_event event enabled
May 14 09:36:43 dsfe05 rasdaemon[4088417]: Enabled event ras:aer_event
May 14 09:36:43 dsfe05 rasdaemon[4088417]: mce:mce_record event enabled
May 14 09:36:43 dsfe05 rasdaemon[4088417]: Enabled event mce:mce_record
May 14 09:36:43 dsfe05 rasdaemon[4088417]: ras:extlog_mem_event event enabled
May 14 09:36:43 dsfe05 rasdaemon[4088417]: Enabled event ras:extlog_mem_event
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording mc_event events
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording aer_event events
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording extlog_event events
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording mce_record events
root@dsfe05:~# ras-mc-ctl --layout
Use of uninitialized value $max_pos[3] in modulus (%) at /usr/sbin/ras-mc-ctl line 907.
Use of uninitialized value $d in numeric ge (>=) at /usr/sbin/ras-mc-ctl line 908.
Use of uninitialized value $d in sprintf at /usr/sbin/ras-mc-ctl line 911.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
+-----------------------------------------------------------------------------------------------+
| mc0 |
| csrow0 | csrow1 | csrow2 | csrow3 |
| channel0 | channel1 | channel0 | channel1 | channel0 | channel1 | channel0 | channel1 |
----+-----------------------------------------------------------------------------------------------+

0: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB |
----+-----------------------------------------------------------------------------------------------+
root@dsfe05:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#2channel#1 0 0
mc#0csrow#2channel#0 0 0
root@dsfe05:~# ras-mc-ctl --print-labels
ras-mc-ctl: Error: No dimm labels for Supermicro model H13SAE-MF
root@dsfe05:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model H13SAE-MF
root@dsfe05:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

No MCE errors.
root@dsfe05:~#
</pre>

= sensors =

== VME CPU V7865 ==

add to /etc/rc.local

<pre>
modprobe coretemp
modprobe lm75
modprobe lm90
modprobe max1668
</pre>

available sensors:

<pre>
root@lxdaq23:~# sensors
max6657-i2c-0-4c
Adapter: SMBus I801 adapter at 0400
temp1: +29.1°C (low = -55.0°C, high = +105.0°C)
(crit = +105.0°C, hyst = +95.0°C)
temp2: +31.5°C (low = -55.0°C, high = +105.0°C)
(crit = +105.0°C, hyst = +95.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Core 0: +25.0°C (crit = +100.0°C)
Core 1: +25.0°C (crit = +100.0°C)

max1805-i2c-0-18
Adapter: SMBus I801 adapter at 0400
temp1: +35.0°C (low = -55.0°C, high = +127.0°C)
temp2: +63.0°C (low = -55.0°C, high = +127.0°C)
temp3: FAULT (low = -55.0°C, high = +127.0°C) ALARM (HIGH)

lm75-i2c-0-48
Adapter: SMBus I801 adapter at 0400
temp1: +34.5°C (high = +80.0°C, hyst = +75.0°C)

root@lxdaq23:~#
</pre>

== ASUS P7P55D EVO ==

* BIOS version 2101

<pre>
root@iris01:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Core 0: +34.0°C (high = +83.0°C, crit = +99.0°C)
Core 1: +37.0°C (high = +83.0°C, crit = +99.0°C)
Core 2: +38.0°C (high = +83.0°C, crit = +99.0°C)
Core 3: +35.0°C (high = +83.0°C, crit = +99.0°C)

nouveau-pci-0100
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.05 V)
temp1: +46.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

atk0110-acpi-0
Adapter: ACPI interface
Vcore Voltage: 864.00 mV (min = +0.80 V, max = +1.60 V)
+3.3V Voltage: 3.38 V (min = +2.97 V, max = +3.63 V)
+5V Voltage: 5.04 V (min = +4.50 V, max = +5.50 V)
+12V Voltage: 12.15 V (min = +10.20 V, max = +13.80 V)
CPU Fan Speed: 968 RPM (min = 600 RPM, max = 7200 RPM)
Chassis1 Fan Speed: 1288 RPM (min = 600 RPM, max = 7200 RPM)
Chassis2 Fan Speed: 1316 RPM (min = 600 RPM, max = 7200 RPM)
Power Fan Speed: 0 RPM (min = 0 RPM, max = 7200 RPM)
CPU Temperature: +34.0°C (high = +45.0°C, crit = +45.5°C)
MB Temperature: +30.0°C (high = +45.0°C, crit = +46.0°C)

root@iris01:~#
</pre>

== ASUS Z97-WS ==

* BIOS version 2704
* load sensors drivers
<pre>
echo modprobe coretemp >> /etc/rc.local
echo modprobe nct6775 >> /etc/rc.local
</pre>
* in /boot/grub/grub.cfg, add: GRUB_CMDLINE_LINUX_DEFAULT="acpi_enforce_resources=no"
* update grub and reboot: grub-mkconfig -o /boot/grub/grub.cfg

<pre>
root@isdaq08:~# sensors
acpitz-acpi-0
Adapter: ACPI interface
temp1: +27.8°C
temp2: +29.8°C

nct6791-isa-0290
Adapter: ISA adapter
Vcore: 888.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
AVCC: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
+3.3V: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.99 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 0.00 V (min = +0.00 V, max = +0.00 V)
3VSB: 3.44 V (min = +0.00 V, max = +0.00 V) ALARM
Vbat: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 0.00 V (min = +0.00 V, max = +0.00 V)
in11: 840.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in12: 0.00 V (min = +0.00 V, max = +0.00 V)
in13: 0.00 V (min = +0.00 V, max = +0.00 V) ALARM
in14: 0.00 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1041 RPM (min = 0 RPM)
fan2: 1040 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan6: 0 RPM (min = 0 RPM)
SYSTIN: +34.0°C (high = +0.0°C, hyst = +0.0°C) ALARM sensor = thermistor
CPUTIN: +41.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: -128.0°C sensor = thermistor
AUXTIN1: -128.0°C sensor = thermistor
AUXTIN2: +35.0°C sensor = thermistor
AUXTIN3: +127.0°C sensor = thermistor
PECI Agent 0: +41.0°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
PCH_DIM0_TEMP: +0.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +42.0°C (high = +80.0°C, crit = +100.0°C)
Core 0: +42.0°C (high = +80.0°C, crit = +100.0°C)
Core 1: +40.0°C (high = +80.0°C, crit = +100.0°C)
Core 2: +39.0°C (high = +80.0°C, crit = +100.0°C)
Core 3: +39.0°C (high = +80.0°C, crit = +100.0°C)

root@isdaq08:~#
</pre>

== ASUS Z170-DELUXE ==

* BIOS version 3801
* load sensors drivers
<pre>
echo modprobe coretemp >> /etc/rc.local
echo modprobe jc42 >> /etc/rc.local
echo modprobe lm92 >> /etc/rc.local
echo modprobe nct6775 >> /etc/rc.local
</pre>
* in /etc/default/grub, add: GRUB_CMDLINE_LINUX_DEFAULT="acpi_enforce_resources=no"
* update grub and reboot: grub-mkconfig -o /boot/grub/grub.cfg

<pre>
root@iris00:~# sensors
nct6793-isa-0290
Adapter: ISA adapter
in0: 600.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 144.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in6: 0.00 V (min = +0.00 V, max = +0.00 V)
in7: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.14 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1000.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in10: 600.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 592.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 968.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1370 RPM (min = 0 RPM)
fan2: 1437 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan6: 0 RPM (min = 0 RPM)
SYSTIN: +32.0°C (high = +98.0°C, hyst = +95.0°C) sensor = thermistor
CPUTIN: +42.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: -128.0°C sensor = thermistor
AUXTIN1: +50.0°C sensor = thermistor
AUXTIN2: +22.0°C sensor = thermistor
AUXTIN3: +28.0°C sensor = thermistor
PECI Agent 0: +50.0°C (high = +98.0°C, hyst = +95.0°C)
(crit = +100.0°C)
PECI Agent 0 Calibration: +42.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
TSI2_TEMP: +3892314.0°C
TSI3_TEMP: +3892314.0°C
TSI4_TEMP: +3892314.0°C
TSI5_TEMP: +3892314.0°C
TSI6_TEMP: +3892314.0°C
TSI7_TEMP: +3892314.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

jc42-i2c-0-1a
Adapter: SMBus I801 adapter at f040
temp1: +36.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

jc42-i2c-0-18
Adapter: SMBus I801 adapter at f040
temp1: +34.8°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

jc42-i2c-0-1b
Adapter: SMBus I801 adapter at f040
temp1: +35.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

jc42-i2c-0-19
Adapter: SMBus I801 adapter at f040
temp1: +36.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +52.0°C (high = +84.0°C, crit = +100.0°C)
Core 0: +52.0°C (high = +84.0°C, crit = +100.0°C)
Core 1: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 2: +48.0°C (high = +84.0°C, crit = +100.0°C)
Core 3: +47.0°C (high = +84.0°C, crit = +100.0°C)

root@iris00:~#
</pre>

== ASUS Z390M-PRO GAMING (WI-FI) ==

* BIOS 3006
* load sensors drivers
<pre>
echo modprobe coretemp >> /etc/rc.local
echo modprobe nct6775 >> /etc/rc.local
</pre>

<pre>
root@daq18:~# sensors
nct6798-isa-0290
Adapter: ISA adapter
in0: 696.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.42 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 208.00 mV (min = +0.00 V, max = +0.00 V)
in6: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.42 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.17 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.07 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.36 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.33 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in14: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 0 RPM (min = 0 RPM)
fan2: 1131 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 1006 RPM (min = 0 RPM)
fan6: 0 RPM (min = 0 RPM)
fan7: 0 RPM (min = 0 RPM)
SYSTIN: +32.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
CPUTIN: +29.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
AUXTIN0: +25.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
AUXTIN1: +7.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
AUXTIN2: +8.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
AUXTIN3: +24.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
AUXTIN4: +83.0°C (high = +80.0°C, hyst = +75.0°C) ALARM
(crit = +100.0°C)
PECI Agent 0 Calibration: +29.0°C (high = +80.0°C, hyst = +75.0°C)
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +39.0°C (high = +82.0°C, crit = +100.0°C)
Core 0: +39.0°C (high = +82.0°C, crit = +100.0°C)
Core 1: +33.0°C (high = +82.0°C, crit = +100.0°C)
Core 2: +32.0°C (high = +82.0°C, crit = +100.0°C)
Core 3: +31.0°C (high = +82.0°C, crit = +100.0°C)
Core 4: +31.0°C (high = +82.0°C, crit = +100.0°C)
Core 5: +30.0°C (high = +82.0°C, crit = +100.0°C)

acpitz-acpi-0
Adapter: ACPI interface
temp1: +27.8°C

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: +28.0°C

root@daq18:~#
</pre>

== ASUS H110M-A/M.2 ==

* BIOS version 4202
* echo modprobe coretemp >> /etc/rc.local
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@midpol:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +33.0°C (high = +80.0°C, crit = +100.0°C)
Core 0: +33.0°C (high = +80.0°C, crit = +100.0°C)
Core 1: +30.0°C (high = +80.0°C, crit = +100.0°C)

acpitz-acpi-0
Adapter: ACPI interface
temp1: +27.8°C (crit = +119.0°C)
temp2: +29.8°C (crit = +119.0°C)

nct6793-isa-0290
Adapter: ISA adapter
in0: 368.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 152.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in6: 928.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.14 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1000.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in10: 152.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in11: 128.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in12: 136.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in13: 120.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 136.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1004 RPM (min = 0 RPM)
fan2: 1143 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan6: 0 RPM (min = 0 RPM)
SYSTIN: +118.0°C (high = +98.0°C, hyst = +95.0°C) sensor = thermistor
CPUTIN: +29.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +30.0°C sensor = thermistor
AUXTIN1: +112.0°C sensor = thermistor
AUXTIN2: +111.0°C sensor = thermistor
AUXTIN3: +110.0°C sensor = thermistor
PECI Agent 0: +31.0°C (high = +98.0°C, hyst = +95.0°C)
(crit = +100.0°C)
PECI Agent 0 Calibration: +36.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
TSI2_TEMP: +3892314.0°C
TSI3_TEMP: +3892314.0°C
TSI4_TEMP: +3892314.0°C
TSI5_TEMP: +3892314.0°C
TSI6_TEMP: +3892314.0°C
TSI7_TEMP: +3892314.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

root@midpol:~#
</pre>

== ASUS P9X79 WS ==

* https://www.asus.com/supportonly/P9X79%20WS/HelpDesk_Manual/
* BIOS version 4802
* modprobe nct6775
* modprobe coretemp

<pre>
root@daq14:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 0: +29.0°C (high = +82.0°C, crit = +100.0°C)
Core 1: +24.0°C (high = +82.0°C, crit = +100.0°C)
Core 2: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +82.0°C, crit = +100.0°C)

nouveau-pci-0200
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.00 V)
temp1: +39.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

nct6776-isa-0290
Adapter: ISA adapter
Vcore: 1.04 V (min = +0.00 V, max = +1.74 V)
in1: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
AVCC: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
+3.3V: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 2.04 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 904.00 mV (min = +0.00 V, max = +0.00 V) ALARM
3VSB: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
Vbat: 3.30 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1265 RPM (min = 0 RPM)
fan2: 1909 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +34.0°C (high = +0.0°C, hyst = +0.0°C) ALARM sensor = thermistor
CPUTIN: +58.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermal diode
AUXTIN: +31.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
PECI Agent 0: +31.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +96.0°C)
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

root@daq14:~#
</pre>

== ASUS TUF GAMING B550M-PLUS WIFI II ==

* BIOS 2803, 2806
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@midm9a:~# sensors
nct6798-isa-0290
Adapter: ISA adapter
in0: 488.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.82 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in14: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 0 RPM (min = 0 RPM)
fan2: 760 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan7: 1264 RPM (min = 0 RPM)
SYSTIN: +25.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +22.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +95.0°C sensor = thermistor
AUXTIN1: +25.0°C sensor = thermistor
AUXTIN2: +25.0°C sensor = thermistor
AUXTIN3: +25.0°C sensor = thermistor
PECI Agent 0 Calibration: +23.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +32.4°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

amdgpu-pci-0800
Adapter: PCI adapter
vddgfx: 1.45 V
vddnb: 993.00 mV
edge: +28.0°C
PPT: 20.00 W

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +33.4°C

root@midm9a:~#
</pre>

== ASUS ASUS ROG STRIX B550-XE GAMING WIFI ==

* BIOS 2423, 2604
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@daq13:~# sensors
nct6798-isa-0290
Adapter: ISA adapter
in0: 344.00 mV (min = +0.00 V, max = +1.74 V)
in1: 992.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in6: 216.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.30 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.81 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in11: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 280.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 845 RPM (min = 0 RPM)
fan2: 998 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +28.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +27.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +94.0°C sensor = thermistor
AUXTIN1: +28.0°C sensor = thermistor
AUXTIN2: +28.0°C sensor = thermistor
AUXTIN3: +97.0°C sensor = thermistor
PECI Agent 0 Calibration: +27.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +33.6°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

amdgpu-pci-0600
Adapter: PCI adapter
vddgfx: 1.45 V
vddnb: 999.00 mV
edge: +29.0°C
PPT: 14.00 W

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: +30.0°C

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +33.9°C

root@daq13:~#
</pre>

== ASUS ASUS ROG STRIX B550-E GAMING ==

* bios 2803
* echo modprobe jc42 >> /etc/rc.local
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@daq17:~# sensors
jc42-i2c-1-1b
Adapter: SMBus PIIX4 adapter port 0 at 0b00
temp1: +25.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: +28.0°C

nouveau-pci-0800
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.00 V)
temp1: +34.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

nct6798-isa-0290
Adapter: ISA adapter
in0: 288.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 224.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.31 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.79 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 280.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 843 RPM (min = 0 RPM)
fan2: 629 RPM (min = 0 RPM)
fan3: 746 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +22.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +25.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +93.0°C sensor = thermistor
AUXTIN1: +22.0°C sensor = thermistor
AUXTIN2: +22.0°C sensor = thermistor
AUXTIN3: +96.0°C sensor = thermistor
PECI Agent 0 Calibration: +25.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +27.6°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

jc42-i2c-1-1a
Adapter: SMBus PIIX4 adapter port 0 at 0b00
temp1: +23.2°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

asusec-isa-0000
Adapter: ISA adapter
CPU_Opt: 0 RPM
Chipset: +34.0°C
CPU: +25.0°C
Motherboard: +22.0°C
T_Sensor: -40.0°C
VRM: +31.0°C

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +28.0°C
Tccd1: +27.5°C

root@daq17:~#
</pre>

== ASUS PRIME B650-PLUS ==

* BIOS 1811
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@dsdaqgw:~# sensors
amdgpu-pci-0b00
Adapter: PCI adapter
vddgfx: 930.00 mV
vddnb: 1.19 V
edge: +38.0°C
PPT: 25.10 W

nct6799-isa-0290
Adapter: ISA adapter
in0: 920.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.04 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 320.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.28 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.28 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.10 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.04 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 416.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 328.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 0 RPM (min = 0 RPM)
fan2: 1253 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan7: 0 RPM (min = 0 RPM)
SYSTIN: +33.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +35.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +78.0°C sensor = thermistor
AUXTIN1: +11.0°C sensor = thermistor
AUXTIN2: +20.0°C sensor = thermistor
AUXTIN3: +82.0°C sensor = thermistor
PECI Agent 0 Calibration: +35.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +42.6°C
intrusion0: ALARM
intrusion1: OK
beep_enable: disabled

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +42.6°C
Tccd1: +36.4°C

root@dsdaqgw:~#
</pre>

= Enable CPU turbo mode =

* Intel CPU has a nominal CPU frequency (i.e. 3.4GHz) and a turbo-boost CPU frequency (i.e. 4.0GHz). Here we will enable this turbo-boost mode.
* Find out CPU capability
<pre>
root@daq01:~# lscpu | grep Hz
Model name: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
CPU MHz: 3965.803
CPU max MHz: 4000.0000
CPU min MHz: 800.0000
root@daq01:~#
</pre>
* Look up this CPU in the Intel ARK database - google for the CPU model name, i.e.
https://ark.intel.com/content/www/us/en/ark/products/88196/intel-core-i7-6700-processor-8m-cache-up-to-4-00-ghz.html
* Find current frequency settings:
<pre>
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "powersave" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 2.72 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
root@daq01:~#
</pre>
* Note the following:
** current governor is "powersave"
** "performance" governor is available
** "boost state support" is supported and active.
* Confirm CPU frequency governor:
<pre>
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
powersave
powersave
powersave
powersave
powersave
powersave
powersave
powersave
root@daq01:~#
</pre>
* Change governor to "performance":
<pre>
root@daq01:~# cpupower frequency-set --governor performance
Setting cpu: 0
Setting cpu: 1
Setting cpu: 2
Setting cpu: 3
Setting cpu: 4
Setting cpu: 5
Setting cpu: 6
Setting cpu: 7
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
performance
performance
performance
performance
performance
performance
performance
performance
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "performance" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 3.93 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
</pre>
* monitor CPU frequency:
<pre>
root@daq01:~# cpupower monitor
| Nehalem || Mperf || Idle_Stats
CPU| C3 | C6 | PC3 | PC6 || C0 | Cx | Freq || POLL | C1 | C1E | C3 | C6 | C7s | C8
0| 0.00| 0.00| 0.00| 0.00|| 88.80| 11.20| 3973|| 0.00| 0.00| 0.01| 0.02| 0.31| 0.00| 4.25
4| 0.00| 0.00| 0.00| 0.00|| 4.70| 95.30| 3945|| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 95.03
1| 0.73| 3.70| 0.00| 0.00|| 4.52| 95.48| 3864|| 0.00| 0.01| 1.19| 0.44| 2.82| 0.00| 90.23
5| 0.73| 3.70| 0.00| 0.00|| 0.37| 99.63| 3807|| 0.00| 0.00| 0.03| 0.09| 1.70| 0.00| 97.64
2| 2.28| 12.86| 0.00| 0.00|| 1.41| 98.59| 3829|| 0.00| 0.86| 3.17| 0.46| 7.70| 0.00| 85.87
6| 2.28| 12.86| 0.00| 0.00|| 2.88| 97.12| 3856|| 0.00| 0.11| 4.56| 2.15| 10.31| 0.00| 78.99
3| 1.33| 4.81| 0.00| 0.00|| 0.99| 99.01| 3804|| 0.00| 0.49| 0.79| 0.01| 1.03| 0.00| 96.12
7| 1.34| 4.81| 0.00| 0.00|| 1.26| 98.74| 3818|| 0.00| 0.01| 2.32| 0.47| 5.02| 0.00| 90.06
root@daq01:~#
</pre>
* check that the CPU is not overheating:
<pre>
root@daq01:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 1: +38.0°C (high = +84.0°C, crit = +100.0°C)
Core 2: +34.0°C (high = +84.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +84.0°C, crit = +100.0°C)
</pre>
* congratulations, we are running at 4 GHz now!

= Setup ubuntu as gateway to private network =

See also:
* https://daq.triumf.ca/DaqWiki/index.php/VME-CPU#Setup_the_boot_host_computer_.28el7.29
* http://www.triumf.info/wiki/DAQwiki/index.php/Dhcpd_on_eth1

== Steps to do ==

!!! UPDATED 16feb2024 Ubuntu-22.04.03 !!!

* assign network numbers to the private network, i.e. 192.168.1.x, 192.168.2.x, etc
* (on the gateway machine, each private network interface has to have a different network number)
* (each network interface can have multiple networks attached, via VLANs or via eth0:0, eth0:1 constructs)
* assign IP addresses on the private network, save them in /etc/hosts i.e. "hvps 192.168.1.10"
* (for simplicity, assign 192.168.1.1 to the gateway machine itself)
* (IP addresses 192.168.1.0 and 192.168.1.255 are "special", do not use them)
* setup DNS server (dnsmasq) to serve contents of /etc/hosts via DNS (otherwise, many programs will see inconsistent name to IP address mapping)
* setup DHCP server (dnsmasq) to give out the IP addresses
* setup TFTP server (dnsmasq), pxelinux and NFS for diskless booting
* setup time server (chronyd) to provide common time to all devices
* setup NAT so machines on private network can access the internet (to get OS updates, etc)
* setup NIS and NFS so machines on the private network can use common home directories
* setup rsync backup of machines on the private network

== setup hosts ==

* edit /etc/hosts
<pre>
192.168.1.101 dsfe01
... and so forth
</pre>

== setup dns and dhcp ==

!!! updated 16feb2024 for Ubuntu 22.04.3 !!!

!!! note: stock systemd-resolved remains, is configured to forward queries to dnsmasq, configured to forward queries to TRIUMF DNS !!!

!!! note: per authors of systemd, bare hostnames are not permitted, a DNS domain name must always be used. DNS domain name "dsdaq" is used in this example !!!

* apt install dnsmasq
* ensure dnsmasq starts after all interfaces are up (Ubuntu-22)
<pre>
mkdir /etc/systemd/system/dnsmasq.service.d
echo -e "[Unit]\nAfter=network-online.target\n" > /etc/systemd/system/dnsmasq.service.d/local.conf
</pre>
* edit /etc/dnsmasq.conf
<pre>
# /etc/dnsmasq.conf
# DNS settings
#port=0 # disable DNS function
port=53 # enable DNS function
bind-interfaces # do not collide with systemd-resolved, we use 127.0.0.1:53, they use 127.0.0.53:53
domain-needed
bogus-priv
no-resolv
#log-queries # log DNS quesries

# TRIUMF DNS settings

server=142.90.100.19
expand-hosts
domain=dsdaq
local=/dsdaq/
localmx # do not forward MX queries to TRIUMF

# DHCP settings
interface=enp1s0f0 # VX network 192.168.0.x
#interface=missing # FEP and TSP network 192.168.1.x
interface=enp1s0f1 # controls network 192.168.2.x
#dhcp-range=192.168.1.50,192.168.1.150,infinite
dhcp-range=192.168.0.0,static
dhcp-range=192.168.2.0,static
log-dhcp # log DHCP queries
#quiet-dhcp
dhcp-ignore=tag:!known
#dhcp-boot=pxelinux.0

dhcp-option=option:dns-server,192.168.0.248
dhcp-option=option:ntp-server,192.168.0.248

# TFTP settings

enable-tftp
tftp-root=/tftpboot
</pre>
* #mkdir /tftpboot ### per tftp-root (if no ZFS)
* zfs create -o mountpoint=/tftpboot rpool/tftpboot ### (if root is ZFS)
* create resolved-dsdaq.conf with main IP address of dnsmasq
<pre>
[Resolve]
DNS=192.168.0.248
Domains=dsdaq triumf.ca
</pre>
* mkdir -p /etc/systemd/resolved.conf.d/
* /bin/rm -f /etc/systemd/resolved.conf.d/*.conf
* cp resolved-dsdaq.conf /etc/systemd/resolved.conf.d/
* systemctl stop systemd-resolved.service
* systemctl disable systemd-resolved.service
* systemctl enable dnsmasq
* systemctl restart dnsmasq
* try to "ping" or "host" some names from /etc/hosts, it should work
* try to ping daq00, daq00.triumf.ca, all should work
* resolved-dsdaq.conf goes into /etc/systemd/resolved.conf.d/ of all machines on the private network
* if not using systemd-resolved, edit /etc/resolv.conf

== setup chronyd ==

* enable ntp server:
* disable systemd-timesyncd, configure and enable chronyd per instructions above
* create dsdaq.conf
<pre>
# chrony config for dsdaq server

#allow 192.168.0.0
#allow 192.168.1.0
#allow 192.168.2.0
allow all

# end
</pre>
* cp dsdaq.conf /etc/chrony/conf.d/
* systemctl restart chronyd
* chronyc tracking ### wait until time is synchronized (a few seconds)
* create dsdaq.sources # use hostname or IP address of chronyd server
<pre>
# Put this file in /etc/chrony/sources.d
# systemctl restart chrony
# chronyc sources
# chronyc tracking
server dsdaqgw iburst prefer
# end
</pre>
* dsdaq.sources goes to /etc/chrony/sources.d of all machines on the private network

== setup diskless network booting ==

=== setup pxelinux for legacy pxe boot ===

* add bits in dnsmasq.conf
<pre>
dhcp-host=ac:1f:6b:9e:7f:4a,dsfe01,infinite
dhcp-boot=pxelinux.0
dhcp-option=17,"192.168.0.251:/nfsroot/%s,vers=3,tcp"
</pre>
* setup pxelinux for Ubuntu-18
<pre>
cd ~
wget https://www.kernel.org/pub/linux/utils/boot/syslinux/4.xx/syslinux-4.03.tar.bz2
tar xjvf syslinux-4.03.tar.bz2
cd syslinux-4.03
cp -pv ./core/pxelinux.0 ./com32/hdt/hdt.c32 ./memdisk/memdisk ./com32/menu/menu.c32 /zssd/tftpboot/
</pre>
* cd /zssd/tftpboot
<pre>
wget http://ladd00.triumf.ca/tftpboot/memtest86+-4.20.iso.zip
wget http://ladd00.triumf.ca/tftpboot/memtest86+-5.01.iso.gz
wget http://ladd00.triumf.ca/tftpboot/modules.alias
wget http://ladd00.triumf.ca/tftpboot/modules.pcimap
wget http://ladd00.triumf.ca/tftpboot/pci.ids
</pre>
* mkdir pxelinux.cfg
* emacs -nw pxelinux.cfg/default
<pre>
default menu.c32
prompt 0

menu title Welcome to the DSVSLICE PXE boot menu

timeout 50

label hdt
kernel hdt.c32

label memtest86+-5.01
kernel memdisk iso initrd=memtest86+-5.01.iso.gz

label memtest86+-4.20
kernel memdisk iso initrd=memtest86+-4.20.iso.zip

label vmlinuz-5.3.0-26-generic
menu default
kernel vmlinuz-5.3.0-26-generic
append initrd=initrd.img-5.3.0-26-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.1.1:/zssd/nfsroot/dsfe01 toram ip=dhcp panic=60 BOOTIF=enp1s0f0

#end
</pre>

=== setup pxelinux for efi pxe boot ===

* https://c-nergy.be/blog/?p=13808
* add dnsmasq.conf bits. note: to use dhcp root-path, see the "nfsroot=auto" patch below and make sure to use the "dhcp-option-force" command (mkinitramfs dhcp client does not ask for root-path, we have to force-feed it).
<pre>
# uefi pxe

dhcp-boot=tag:uefipxe,uefi/syslinux.efi
dhcp-option-force=tag:fe01,option:root-path,192.168.0.248:/nfsroot/fe01

# VX network 192.168.0.x

dhcp-host=40:a6:b7:c1:d9:c5,fe01,infinite,set:uefipxe,set:fe01
</pre>
* apt install syslinux pxelinux syslinux-common syslinux-efi syslinux-utils
<pre>
mkdir /tftpboot/uefi
cp /usr/lib/SYSLINUX.EFI/efi64/syslinux.efi /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/ldlinux.e64 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/menu.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/hdt.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libutil.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libmenu.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libcom32.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libgpl.c32 /tftpboot/uefi/
</pre>
* try to boot, it should bomb with "cannot load pxelinux.cfg/default"
* mkdir /tftpboot/uefi/pxelinux.cfg
* create /tftpboot/uefi/pxelinux.cfg/default, note nfsroot path is hardwired, note "http:" is used to load vmlinuz and initrd files (because tftp is super slow)
<pre>
default menu.c32
prompt 0

menu title Welcome to the DSDAQGW UEFI PXE boot menu

timeout 50

label vmlinuz-6.5.0-17-generic
kernel http://192.168.0.248:8088/uefi/vmlinuz-6.5.0-17-generic
append initrd=http://192.168.0.248:8088/uefi/initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=auto rw ip=dhcp panic=60

# append initrd=http://192.168.0.248:8088/uefi/initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.0.248:/nfsroot/fe01 rw ip=dhcp panic=60

# append initrd=initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.0.248:/nfsroot/fe01 rw ip=dhcp panic=60
# append initrd=initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=auto ip=dhcp rw panic=60

#end
</pre>
* try to boot, it will bomb with "cannot load http://...."
* install mini_httpd on port 8088, see https://acme.com/software/mini_httpd/
<pre>
apt install mini-httpd
emacs -nw /etc/default/mini-httpd # set "START=1"
emacs -nw /etc/mini-httpd.conf # set "host=192.168.0.248", "port=8088", "data_dir=/tftpboot"
mkdir /etc/systemd/system/mini-httpd.service.d
echo -e "[Unit]\nAfter=network-online.target\n" > /etc/systemd/system/mini-httpd.service.d/local.conf
systemctl enable mini-httpd
systemctl restart mini-httpd
systemctl status mini-httpd
wget http://192.168.0.248:8088/uefi/syslinux.efi
tail -100 /var/log/mini_httpd.log
</pre>
* fix U-22 initramfs bug for "nfsroot=auto", otherwise, "nfsroot=" has to be different for each machine and you have to have separate pxelinux config files for each machine
** emacs -nw /usr/lib/initramfs-tools/etc/dhcp/dhclient-enter-hooks.d/config
** add "echo ROOTPATH=..." if it is missing
<pre>
echo "ROOTSERVER='${new_routers%% *}'"
echo "ROOTPATH='$new_root_path'"
echo "HOSTNAME='$new_host_name'"
</pre>
* fix U-24 initramfs bug for "nfsroot=auto", otherwise, "nfsroot=" has to be different for each machine and you have to have separate pxelinux config files for each machine
** emacs -nw /usr/share/initramfs-tools/dhcpcd-hooks/70-net-conf
** add "ROOTPATH=..." if it is missing
<pre>
DNSDOMAIN='${new_domain_name-}'
ROOTSERVER='${new_routers-}'
ROOTPATH='${new_root_path-}'
filename='${new_filename-}'
DHCPLEASETIME='${new_dhcp_lease_time-}'
</pre>
** regenerate initramfs (be careful you generate it for the right kernel!)
** see https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/2054482
<pre>
mkinitramfs 6.5.0-18-generic
mkinitramfs 6.8.0-51-generic -o /boot/initrd.img-6.8.0-51-generic
</pre>
* copy linux kernel and initrd
<pre>
cp /boot/vmlinuz-6.5.0-18-generic /tftpboot/uefi/
cp /boot/initrd.img-6.5.0-18-generic /tftpboot/uefi/
chmod a+r /tftpboot/uefi/*
</pre>
* try to boot, should bomb with messages about "trying to mount root filesystem"
* tail /var/log/syslog
<pre>
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 vendor class: PXEClient:Arch:00007:UNDI:003016
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 1:netmask, 2:time-offset, 3:router, 4, 5,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 6:dns-server, 12:hostname, 13:boot-file-size,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 15:domain-name, 17:root-path, 18:extension-path,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 22:max-datagram-reassembly, 23:default-ttl,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 28:broadcast, 40:nis-domain, 41:nis-server,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 42:ntp-server, 43:vendor-encap, 50:requested-address,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 51:lease-time, 54:server-identifier, 58:T1,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 59:T2, 60:vendor-class, 66:tftp-server, 67:bootfile-name,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 97:client-machine-id, 128, 129, 130, 131,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 132, 133, 134, 135
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 next server: 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 broadcast response
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 1 option: 53 message-type 2
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 18 option: 67 bootfile-name uefi/syslinux.efi
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 12 hostname fe01
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 vendor class: PXEClient:Arch:00007:UNDI:003016
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 1:netmask, 2:time-offset, 3:router, 4, 5,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 6:dns-server, 12:hostname, 13:boot-file-size,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 15:domain-name, 17:root-path, 18:extension-path,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 22:max-datagram-reassembly, 23:default-ttl,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 28:broadcast, 40:nis-domain, 41:nis-server,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 42:ntp-server, 43:vendor-encap, 50:requested-address,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 51:lease-time, 54:server-identifier, 58:T1,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 59:T2, 60:vendor-class, 66:tftp-server, 67:bootfile-name,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 97:client-machine-id, 128, 129, 130, 131,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 132, 133, 134, 135
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 next server: 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 broadcast response
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 1 option: 53 message-type 5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 18 option: 67 bootfile-name uefi/syslinux.efi
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 12 hostname fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: error 8 User aborted the transfer received from 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/syslinux.efi to 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/syslinux.efi to 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 requested options: 1:netmask, 3:router, 6:dns-server
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 bootfile name: uefi/syslinux.efi
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 next server: 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 broadcast response
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 1 option: 53 message-type 2
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 requested options: 1:netmask, 3:router, 6:dns-server
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 bootfile name: uefi/syslinux.efi
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 next server: 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 broadcast response
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 1 option: 53 message-type 5
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/ldlinux.e64 to 192.168.0.110
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/01-40-a6-b7-c1-d9-c5 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8006E not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8006 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A800 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A80 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/pxelinux.cfg/default to 192.168.0.110
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/menu.c32 to 192.168.0.110
Feb 16 20:43:10 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/libutil.c32 to 192.168.0.110
Feb 16 20:43:10 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/pxelinux.cfg/default to 192.168.0.110
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 client provides name: dsdaqgw.triumf.ca
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 1:netmask, 28:broadcast, 2:time-offset, 3:router,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 15:domain-name, 6:dns-server, 119:domain-search,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 12:hostname, 44:netbios-ns, 47:netbios-scope,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 26:mtu, 121:classless-static-route, 42:ntp-server
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 bootfile name: uefi/syslinux.efi
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 next server: 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 1 option: 53 message-type 2
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 51 lease-time infinite
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 client provides name: dsdaqgw.triumf.ca
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 1:netmask, 28:broadcast, 2:time-offset, 3:router,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 15:domain-name, 6:dns-server, 119:domain-search,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 12:hostname, 44:netbios-ns, 47:netbios-scope,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 26:mtu, 121:classless-static-route, 42:ntp-server
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 bootfile name: uefi/syslinux.efi
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 next server: 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 1 option: 53 message-type 5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 51 lease-time infinite
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 12 hostname fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw rpc.mountd[3350210]: authenticated mount request from 192.168.0.110:981 for /nfsroot/fe01 (/nfsroot/fe01)
Feb 16 20:45:07 dsdaqgw rpc.mountd[3350210]: authenticated unmount request from 192.168.0.110:859 for /nfsroot/fe01/tmp/autoDY4k5u (/nfsroot/fe01)
</pre>
* tail /var/log/mini_httpd.log
<pre>
192.168.0.110 - - [16/Feb/2024:20:43:15 -0800] "GET /uefi/vmlinuz-6.5.0-17-generic HTTP/1.0" 200 14227944 "" "Syslinux/6.04"
192.168.0.110 - - [16/Feb/2024:20:43:24 -0800] "GET /uefi/initrd.img-6.5.0-17-generic HTTP/1.0" 200 137824833 "" "Syslinux/6.04"
</pre>

=== setup efi http boot ===

https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-deployment-prep-uefi-httpboot.html

=== setup linux kernel ===

* copy the kernel files
<pre>
cd /boot
rsync -av config* initrd* System.map* vmlinuz* /tftpboot/
</pre>
* cd /tftpboot
* chmod a+r *

=== setup nfs ===

* apt-get install nfs-kernel-server
* enable NFS over UDP, edit /etc/nfs.conf add "udp=y":
<pre>
udp=y
</pre>
<pre>
systemctl restart nfs-server.service
</pre>
* emacs -nw /etc/exports
<pre>
/nfsroot/dsfe01 dsfe01(rw,no_root_squash,async,no_subtree_check)
</pre>
* enable services
<pre>
systemctl enable nfs-server
systemctl enable nfs-mountd
systemctl enable nfs-idmapd
systemctl restart nfs-server
systemctl restart nfs-mountd
systemctl restart nfs-idmapd
</pre>
* after editing /etc/exports, run
<pre>
exportfs -av
</pre>

=== setup userland ===

!!! ubuntu-18 version !!!

* zfs create rpool/nfsroot
* zfs set dedup=verify rpool/nfsroot ### enable deduplication to save disk space because most linux images have mostly identical files
* clone ubuntu
<pre>
mkdir /nfsroot/dsfe01
cd /
rsync -avx . /nfsroot/dsfe01
</pre>
* edit config files:
* cd /nfsroot/dsfe01
* emacs -nw etc/hostname ### change to dsfe01
* emacs -nw etc/mailname ### change to dsfe01
* emacs -nw etc/yp.conf ### change daq00.triumf.ca to musr00.triumf.ca
* emacs -nw etc/defaultdomain ### change to MUSR-NIS
* cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
* emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
* emacs -nw root/.ssh/authorized_keys ### update root ssh keys
* emacs -nw etc/fstab ### add this
<pre>
192.168.1.1:/nfsroot/dsfe01 / nfs defaults,nolock 0 0
</pre>
* emacs -nw etc/chrony/chrony.conf
** comment-out all "pool" and "server" entries
** add entry "server 192.168.1.1 iburst"

After dsfe01 is booted:

* disable services:
<pre>
systemctl disable apache2
systemctl disable dnsmasq
systemctl disable zfs-import-cache
</pre>

To setup additional machines, clone dsfe01 instead of cloning the gateway machine

=== Allow manpages to be viewed ===

If <code>/</code> is mounted over NFS, <code>man</code> will report a permission error. Fix it with:

<pre>
ln -s /etc/apparmor.d/usr.bin.man /etc/apparmor.d/disable/
apparmor_parser -R /etc/apparmor.d/usr.bin.man
</pre>

== setup shared home directory ==

=== on the gateway machine ===
* define netgroups
* emacs -nw /etc/netgroup
<pre>
dsfe (dsfe01,,) (dsfe02,,)
</pre>
* emacs -nw /etc/nsswitch.conf ### edit the netgroup line to read:
<pre>
netgroup: files
</pre>
* export the home directories:
* emacs -nw /etc/exports ### add this:
<pre>
/zssd/home1 @dsfe(rw,no_root_squash,async,no_subtree_check)
</pre>
* exportfs -rc

=== on the frontend machine ===

* mkdir /home
* emacs -nw /etc/fstab ### add this:
<pre>
192.168.1.1:/zssd/home1 /home nfs defaults 0 0
</pre>
* mount -a

== setup NAT ==

NAT allows machines on the private network to connect to the internet: https://en.wikipedia.org/wiki/Network_address_translation

In these examples:
* replace "eno1" with name of the outgoing interface (the one connected to the TRIUMF network).
* replace "enp11s0" with name of the private network interface (192.168.1.x network)

* emacs -nw /etc/rc.local ### add this:
<pre>
# /etc/rc.local

# enable NAT

/sbin/iptables -t nat -A POSTROUTING -o eno1 -j MASQUERADE
iptables -L -v

# uncomment following lines if machine has prohibitive FORWARD rules:
#/sbin/iptables -I FORWARD -i eno1 -o enp11s0 -m state --state RELATED,ESTABLISHED -j ACCEPT
#/sbin/iptables -I FORWARD -i enp11s0 -o eno1 -j ACCEPT
#iptables -L -v

iptables -L -v
sysctl -w net.ipv4.ip_forward=1
#sysctl -a | grep forward

sh /etc/firewall-rfc1918.sh

# end
</pre>
* emacs -nw /etc/firewall-rfc1918.sh
<pre>
# firewall-rfc1918.sh

# prevent RFC1918 private network IP addresses from
# going in and out from our uplink.

ETH=eno1

iptables -F in-rfc1918
iptables -N in-rfc1918
iptables -A in-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A in-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A in-rfc1918 --dst 192.168.0.0/16 -j REJECT

iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -I INPUT -j in-rfc1918 -i $ETH

iptables -F out-rfc1918
iptables -N out-rfc1918
iptables -A out-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A out-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A out-rfc1918 --dst 192.168.0.0/16 -j REJECT

iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -I OUTPUT -j out-rfc1918 -o $ETH

iptables -D FORWARD -j out-rfc1918 -o $ETH
iptables -D FORWARD -j out-rfc1918 -o $ETH
iptables -I FORWARD -j out-rfc1918 -o $ETH

# allow TRIUMF-SECURE network

iptables -I in-rfc1918 -s 10.90.0.0/255.255.0.0 -j ACCEPT
iptables -I out-rfc1918 -d 10.90.0.0/255.255.0.0 -j ACCEPT

# show configuration

iptables -L -v

#end
</pre>

= KVM =

<pre>
apt install cpu-checker

root@daq13:~# kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
root@daq13:~#

(if not, shutdown, go into BIOS settings, enable CPU virtualization)

apt install virtinst ### will install many packages
apt install libvirt-clients libvirt-daemon-system-systemd libvirt-daemon qemu qemu-kvm libvirt-daemon-system virtinst bridge-utils

root@daq13:/home1/wheel# virsh list --all
Id Name State
------------------------------
1 ubuntu-guest running

apt install virt-manager

virt-install --name ubuntu-guest --os-variant ubuntu20.04 --vcpus 2 --ram 2048 --location /daq/daqstore/olchansk/linux/Ubuntu/ubuntu-20.04.3-desktop-amd64.iso --network bridge=virbr0,model=virtio --graphics none --extra-args='console=ttyS0,115200n8 serial'

virtual machine will start, boot, etc
to get out of it, CTRL + Shift followed by ]

ssh wheel@daq13
virt-manager

run virt-install again, omit "--graphics none", open graphics console from virt-manager, it booted into ubuntu installer desktop

virt-install --name test10 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --filesystem /kvm_ladd00,/ --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial" --graphics none

virt-install --name test14 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --disk /tmp/xxx/ladd00.img,bus=sata --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial rdshell" --graphics none --check path_in_use=off
</pre>

build image

<pre>
dd if=/dev/zero of=/tmp/xxx/ladd00.img bs=1024M count=20
mkfs.ext3 /tmp/xxx/ladd00.img ### ext4 fails to mount by SL6 kernel, "unknown ext4 options"
cd /kvm_ladd00/
mount -o loop /tmp/xxx/ladd00.img /mnt/tmp
rsync -av . /mnt/tmp/ --delete
umount /mnt/tmp
</pre>

on the guest, configure network: /etc/rc.local
<pre>
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

ifconfig eth2 192.168.122.2
route add -net 0.0.0.0 gw 192.168.122.1
ifconfig -a
netstat -rn

# end
</pre>

== virsh commands ==

<pre>
virsh list --all

virsh start kvm-el7
virsh console kvm-el7
virsh destroy kvm-el7

virsh install ...
virsh undefine kvm-el7

virsh autostart kvm-ladd00
virsh dominfo kvm-ladd00
</pre>

== virtualize SL6 ladd00 ==

* on ladd00:
<pre>
yum install dracut-network
mkinitrd /boot/initramfs-2.6.32-754.35.1.el6.x86_64-netboot.img 2.6.32-754.35.1.el6.x86_64
</pre>
* on daq00
<pre>
zfs create rpool/kvm-ladd00
cd /kvm-ladd00
rsync -avx ladd00:/ . --exclude nfsroot
brctl addbr virbr0
ifconfig virbr0 192.168.1.1
echo /kvm-ladd00 192.168.1.2(rw,no_root_squash,no_all_squash,async,no_subtree_check) >> /etc/exports
exportfs -rv
</pre>
* create virtual machine
<pre>
virt-install --name kvm-ladd00 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --network bridge=virbr0,model=virtio --boot kernel=/kvm-ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm-ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64-netboot.img,kernel_args="root=/dev/nfs ip=192.168.1.2:192.168.1.1:192.168.1.1:255.255.255.0:ladd00::off nfsroot=192.168.1.1:/kvm-ladd00,vers=3,tcp console=ttyS0,115200n8 serial rdshell" --graphics none --nodisks --check path_in_use=off
</pre>
* adjust kvm-ladd00 image
<pre>
disable network manager
edit fstab
edit yp.conf
edit resolv.conf
edit root/.ssh/authorized_keys
enable rngd or /dev/random does not work, sshd does not work
</pre>
* virsh shutdown test24
* virsh --connect qemu:///system start test24
* virsh console test24 ### to exit, ctrl+[ or ctrl+]
* virsh undefine test24
* virsh autostart kvm-ladd00
* virsh dominfo kvm-ladd00
<pre>
root@daq00:~# virsh dominfo kvm-ladd00
Id: 1
Name: kvm-ladd00
UUID: 1d1f8fed-8b65-4411-a51b-e0ecf359d2f1
OS Type: hvm
State: running
CPU(s): 2
CPU time: 27.7s
Max memory: 2097152 KiB
Used memory: 2097152 KiB
Persistent: yes
Autostart: enable
Managed save: no
Security model: apparmor
Security DOI: 0
Security label: libvirt-1d1f8fed-8b65-4411-a51b-e0ecf359d2f1 (enforcing)
root@daq00:~#
</pre>
* delete unused images in /var/lib/libvirt/images
* virsh edit kvm-ladd00 # change boot command line, etc

== virtualize CentOS-7 daqstore ==

* similar to ladd00 above:
* on daqstore, install dracut-network, already there
<pre>
yum install dracut-network
yum install qemu-kvm libvirt libvirt-python libguestfs-tools virt-install
# yum install busybox ### no rpm package?!?
</pre>
<pre>
dracut -a nfs -v /boot/initramfs-3.10.0-1160.119.1.el7.x86_64-virt.img 3.10.0-1160.119.1.el7.x86_64 --force
</pre>
<pre>
scp daqstore:/boot/initramfs-3.10.0-1160.119.1.el7.x86_64-virt.img /kvm-el7/boot/
</pre>
* on daq00
<pre>
zfs create rpool/kvm-el7
cd /kvm-el7
rsync -avx daqstore:/ .
echo 192.168.1.3 kvm-el7 >> /etc/hosts
systemctl restart dnsmasq
echo /kvm-el 192.168.1.3(rw,no_root_squash,no_all_squash,async,no_subtree_check) >> /etc/exports
exportfs -rv
</pre>
* manage virtual machine
<pre>
virsh console kvm-el7
virsh destroy kvm-el7
virsh undefine kvm-el7
</pre>
* create virtual machine
<pre>
virt-install --name kvm-el7 --os-variant centos7 --vcpus 2 --ram 2048 --import --network bridge=virbr0,model=e1000e --boot kernel=/kvm-el7/boot/vmlinuz-3.10.0-1160.119.1.el7.x86_64,initrd=/kvm-el7/boot/initramfs-3.10.0-1160.119.1.el7.x86_64-virt.img,kernel_args="root=/dev/nfs ip=192.168.1.3:192.168.1.1:192.168.1.1:255.255.255.0:kvm-el7::off nfsroot=192.168.1.1:/kvm-el7,vers=3,tcp rw console=ttyS0,115200n8 serial rdshell" --graphics none --nodisks --check path_in_use=off
</pre>
* adjust kvm-el7 image
<pre>
disable network manager
edit fstab
edit hostname
disable selinux in /etc/sysconfig/selinux

UP TO HERE --- DNS does not work!!!

edit yp.conf
edit resolv.conf
edit root/.ssh/authorized_keys
enable rngd or /dev/random does not work, sshd does not work
</pre>
* virsh shutdown test24
* virsh --connect qemu:///system start test24
* virsh console test24 ### to exit, ctrl+[ or ctrl+]
* virsh undefine test24
* virsh autostart kvm-ladd00
* virsh dominfo kvm-ladd00
<pre>
root@daq00:~# virsh dominfo kvm-ladd00
Id: 1
Name: kvm-ladd00
UUID: 1d1f8fed-8b65-4411-a51b-e0ecf359d2f1
OS Type: hvm
State: running
CPU(s): 2
CPU time: 27.7s
Max memory: 2097152 KiB
Used memory: 2097152 KiB
Persistent: yes
Autostart: enable
Managed save: no
Security model: apparmor
Security DOI: 0
Security label: libvirt-1d1f8fed-8b65-4411-a51b-e0ecf359d2f1 (enforcing)
root@daq00:~#
</pre>
* delete unused images in /var/lib/libvirt/images
* virsh edit kvm-ladd00 # change boot command line, etc

= ARM64 cross-compiler =

== Ubuntu-22 ==

* arm64, aarch64 are Xilinx FPGA Cortex-A53, RPi4, RPi5 machines
* install packages:
<pre>
apt install g++-12-aarch64-linux-gnu gcc-12-aarch64-linux-gnu-base libstdc++-12-dev-arm64-cross
</pre>
* run:
<pre>
aarch64-linux-gnu-gcc-12 -o ttcp.aarch64 ttcp.c -static
aarch64-linux-gnu-g++-12 -o fecdm.exe -O2 -g -Wall -Wuninitialized -std=c++20 fecdm.o dsdm.o /home/dsdaqdev/packages_common/midas/linux-aarch64-remoteonly/lib/libmidas.a -pthread -lrt -lutil /nfsroot/gdm00/usr/lib/aarch64-linux-gnu/libi2c.a -static
</pre>

== Ubuntu-24 ==

* arm64, aarch64 are Xilinx FPGA Cortex-A53, RPi4, RPi5 machines
* install packages:
<pre>
apt install g++-aarch64-linux-gnu
</pre>
* build:
<pre>
aarch64-linux-gnu-g++ -c -o xvcserver_cdm.o -O2 -g -Wall -Wuninitialized -std=c++20 xvcserver_cdm.cxx
aarch64-linux-gnu-g++ -o xvcserver_cdm.exe -O2 -g -Wall -Wuninitialized -std=c++20 xvcserver_cdm.o -pthread -lrt -lutil -static
</pre>

= ARM cross-compiler =

NOTE: updated for U-24

* armv7 (Cyclone-V SoC, RPi3, MityARM CAMAC) machines (Debian-12 armhf target, Ubuntu 24.04 host)
* install packages:
<pre>
apt install g++-arm-linux-gnueabihf gcc-arm-linux-gnueabihf libc6-dev-armhf-cross
</pre>
* build MIDAS frontend (static linking)
<pre>
arm-linux-gnueabihf-g++ -std=c++11 -Wall -Wuninitialized -g -O2 -I/home/dldaq/packages/midas/include -I/home/dldaq/packages/midas/mvodb -c koi2c.cxx
arm-linux-gnueabihf-g++ -o fedldb.exe -std=c++11 -Wall -Wuninitialized -g -O2 -I/home/dldaq/packages/midas/include -I/home/dldaq/packages/midas/mvodb fedldb.o koi2c.o /home/dldaq/packages/midas/linux-armv7-remoteonly/lib/libmidas.a -L/usr/arm-linux-gnueabihf/lib -L/nfsroot/dltdc/usr/lib/arm-linux-gnueabihf -static -lm -lz -lutil -lnsl -lpthread -lrt -li2c
/usr/lib/gcc-cross/arm-linux-gnueabihf/13/../../../../arm-linux-gnueabihf/bin/ld: /home/dldaq/packages/midas/linux-armv7-remoteonly/lib/libmidas.a(system.o): in function `ss_socket_connect_tcp(char const*, int, int*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)':
/home/dldaq/packages/midas/src/system.cxx:4984:(.text+0x252a): warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
</pre>

= 32-bit intel cross-compiler =

== Ubuntu 22.04 ==

<pre>
apt install libstdc++-11-dev:i386
apt install zlib1g-dev:i386
</pre>

NOTES:
* "g++ -m32" does not find libstdc++, please use "g++ -m32 -L/usr/lib/gcc/i686-linux-gnu/11/"
* to cross-build 32-bit MIDAS, use "make linux32".
* executables cross-build on Ubuntu-22 do NOT run on 32-bit Debain-11 (GLIBC and GLIBCXX version mismatch)
* executables cross-build on Ubuntu-22 run on 32-bit Debian-12.

== Ubuntu 24.04 ==

<pre>
apt install gcc-i686-linux-gnu
apt install g++-i686-linux-gnu
apt install libstdc++-13-dev:i386
apt install lib32z1 lib32z1-dev
i686-linux-gnu-gcc -o ttcp.i386 ttcp.c
</pre>

NOTES:
* executables cross-build on Ubuntu-24 will NOT run on 32-bit Debian-12 (GLIBC mismatch, static executables maybe work)
* executables cross-build on Ubuntu-24 run on 32-bit Debian-13

= SSH settings for EPICS =

* TRIUMF EPICS runs obsolete version of SSH
* add this to the use .ssh/config
<pre>
Host sbp1*
HostKeyAlgorithms +ssh-rsa
PubKeyAcceptedAlgorithms +ssh-rsa
KexAlgorithms +diffie-hellman-group1-sha1
ForwardX11 yes
ForwardX11Trusted yes
</pre>

= changes for VME processors =

<pre>
apt -y remove sysstat man-db
apt -y purge dkms
apt -y purge mdadm
apt -y purge fwupd
apt -y purge packagekit
apt -y purge accountsservice
apt -y purge plocate
apt -y purge upower power-profiles-daemon
apt -y autoremove
</pre>

for D-12 32-bit CPUs:

<pre>
apt remove "*libavahi*"
</pre>

= remove snap (U-24) =

Note: snap stores data in $USER/snap/$SNAPNAME, removing a snap on one machine will remove this data from all users even if they want to use snap on some other machine.

Prepare:

NOTE: first remove chromium and firefox, see below.

NOTE: if possible, stop autofs before removing snap - otherwise it will mount all user home directories and complain that it cannot remove some snap data from them

<pre>
systemctl stop autofs
ls -ld /home1/*/snap/* ### remove the per-user snap directories
</pre>

Remove snaps:

<pre>
snap list
echo snap remove --purge chromium ### see below
echo snap remove --purge firefox ### see below
snap remove thunderbird
snap remove cups
snap remove hello-world
snap remove firmware-updater
snap remove gtk-common-themes
snap remove snapd-desktop-integration
snap remove snap-store
snap remove hunspell-dictionaries-1-7-2004
snap remove gnome-system-monitor
snap remove gnome-3-26-1604
snap remove gnome-3-28-1804
snap remove gnome-3-34-1804
snap remove gnome-3-38-2004
snap remove gnome-42-2204
snap remove gnome-46-2404
snap remove mesa-2404
snap remove core
snap remove core18
snap remove core20
snap remove core22
snap remove core24
snap remove bare
snap remove snapd
snap list
</pre>
<pre>
root@daqubuntu:~# snap list
No snaps are installed yet. Try 'snap install hello-world'.
root@daqubuntu:~#
</pre>

Identify packages that install snaps:

<pre>
apt list | grep snap | grep installed | grep -v -e snappy -e snapshot
</pre>

Typical output:

<pre>
firefox/noble,now 1:1snap1-0ubuntu5 amd64 [installed]
gir1.2-snapd-2/noble,now 1.64-0ubuntu5 amd64 [installed,automatic]
libsnapd-glib-2-1/noble,now 1.64-0ubuntu5 amd64 [installed,automatic]
libsnapd-qt-2-1/noble,now 1.64-0ubuntu5 amd64 [installed,automatic]
plasma-discover-backend-snap/noble,now 5.27.11-0ubuntu2 amd64 [installed]
snapd/noble-updates,now 2.66.1+24.04 amd64 [installed]
</pre>

Remove packages that install snaps:

<pre>
apt remove chromium-browser
apt remove chromium-codecs-ffmpeg-extra
apt remove thunderbird
apt remove firefox
apt remove plasma-discover-backend-snap
apt remove plasma-discover-snap-backend
apt remove snapd
apt purge snapd
# package gir1.2-snapd-2 is required by ubuntu-mate-desktop & co
# libsnapd-glib-2-1 is required by gstreamer, gnome-remote-desktop & co
ls -l /etc/systemd/system/ | grep snap ### remove unwanted stuff
</pre>

Remove Chromium:

* ls -ld /home/*/snap/chromium/*
* echo /bin/rm -rf `ls -1d /home/*/snap/chromium/*`
* snap remove chromium

Remove Firefox:

* ls -ld /home/*/snap/firefox/*
* echo /bin/rm -rf `ls -1d /home/*/snap/firefox/*` ### this will delete "snap firefox" profiles of all users!!!
* snap remove firefox ### this will also delete "snap firefox" profiles of all users!!!

Remove gir1.2-snapd-2:
* echo rm -vf /usr/lib/x86_64-linux-gnu/girepository-1.0/Snapd-2.typelib

If "snap remove" is stuck in "change in progress" (this will remove all snaps and break snapd, which is ok,
see https://forum.snapcraft.io/t/snap-remove-taking-forever-abort-wasnt-working/48915)
<pre>
rm /var/lib/snapd/state.json
systemctl restart snapd
</pre>

Prevent snap from reinstalling:
<pre>
cd ~/git/scripts/etc
git pull
cp etc-apt-preferencesd-disable-snap /etc/apt/preferences.d/
</pre>

= install non-snap thunderbird =

from: https://ubuntuhandbook.org/index.php/2024/03/install-thunderbird-deb-ubuntu-2404/

* remove snap thunderbird
<pre>
snap remove --purge thunderbird
apt remove --purge thunderbird
</pre>
* add mozilla repository
<pre>
already done after installing firefox-esr
</pre>
* ppa deb and ubuntu snap thunderbird package names are the same, change priority and hide snap package: create /etc/apt/preferences.d/mozillateamppa
<pre>
Package: thunderbird*
Pin: release o=LP-PPA-mozillateam
Pin-Priority: 1001

Package: thunderbird*
Pin: release o=Ubuntu
Pin-Priority: -1
</pre>
* check that it worked, it should say "build" instead of "snap"
<pre>
apt update
apt list | grep thunderbird | grep -v locale
...
thunderbird/noble 1:128.8.1+build1-0ubuntu0.24.04.1~mt1 amd64
...
</pre>
* install
<pre>
apt install thunderbird
</pre>
* run: thunderbird

= EFI boot using syslinux =

* rationale 1: GRUB is the stock boot loader with U-24. It is unnecessarily complicated. EFI BIOS can boot the linux kernel directly, without GRUB, but unfortunately a small shim bootloader is required to specify the initrd file and the root filesystem. the syslinux boot loader can do this in a very simple way.
* rationale 2: GRUB bootloader configuration is overcomplicated, and when it breaks, it is almost impossible to debug and to recover.
* rationale 3: GRUB bootloader scripts in U-24 have no support for booting from redundant SSDs.
* in the case of GRUB bootloader failure, it is simplest to boot the Ubuntu installer in recovery mode and convert the bootloader from GRUB to syslinux. Open firefox on this page and cut-and-paste the steps (only copy of vmlinux and initrd cannot copy-and-paste as of this writing).
* in the case of servers with redundant SSDs for OS and home directories (ZFS mirror), it is simplest to use the syslinux bootloader to ensure that the machine boots from either SSD (all combinations SSD failures, fail of either EFI partiion, fail of either ZFS mirror partition, machine should boot)
* check partition tables, SATA SSD
<pre>
fdisk -l /dev/sda
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: A3F34DAC-DCB4-B74C-B59E-41E754807812

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root
</pre>
* check partition tables, NVME SSD
<pre>
fdisk -l /dev/nvme0n1
Disk /dev/nvme0n1: 1.75 TiB, 1920383410176 bytes, 3750748848 sectors
Disk model: SAMSUNG MZ1L21T9HCLS-00A07
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 131072 bytes / 131072 bytes
Disklabel type: gpt
Disk identifier: 04ECCD46-DC2A-454C-B4A8-CCC18AA532F7

Device Start End Sectors Size Type
/dev/nvme0n1p1 2048 2203647 2201600 1G EFI System
/dev/nvme0n1p2 2203648 6397951 4194304 2G Linux filesystem
/dev/nvme0n1p3 6397952 23175167 16777216 8G Linux swap
/dev/nvme0n1p4 23175168 3750746111 3727570944 1.7T Linux filesystem
</pre>
* prepare boot device EFI partition, SATA SSDs
<pre>
mkfs.msdos /dev/sda1
mkfs.msdos /dev/sdb1
mkdir /boot/efi-sda
mkdir /boot/efi-sdb
mount /dev/sda1 /boot/efi-sda
mount /dev/sdb1 /boot/efi-sdb
</pre>
* prepare boot device EFI partition, NVME SSDs
<pre>
mkfs.msdos /dev/nvme0n1p1
mkfs.msdos /dev/nvme1n1p1
mkdir /boot/efi-0
mkdir /boot/efi-1
mount /dev/nvme0n1p1 /boot/efi-0
mount /dev/nvme1n1p1 /boot/efi-1
</pre>
* add them to fstab, note the "nofail" mount option
<pre>
blkid | grep vfat
/dev/sdb1: UUID="F30C-13B5" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="20e423b5-ac29-ec42-bab5-f366aefbbd2b"
/dev/sda1: UUID="F2DD-7321" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="9427646c-ce5f-fe47-9ed1-4b84cf4c348f"
grep ^UUID /etc/fstab
UUID=F2DD-7321 /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
UUID=F30C-13B5 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
</pre>
* prepare the EFI partitions (remove old previous subdirectories, only empty efi/boot should be there)
<pre>
cd /boot/efi-sda
mkdir -p efi/boot
</pre>
* get syslinux-6.03
<pre>
cd ~
wget https://daq00.triumf.ca/~olchansk/linux/syslinux-6.03.tar.xz
xz -d < syslinux-6.03.tar.xz | tar xvf -
</pre>
* from syslinux-6.03 copy files:
<pre>
cd /boot/efi-sda/efi/boot
cp ~/syslinux-6.03/efi64/efi/syslinux.efi .
cp ~/syslinux-6.03/efi64/com32/elflink/ldlinux/ldlinux.e64 .
cp syslinux.efi bootx64.efi
</pre>
* identify the ZFS rpool label
<pre>
zfs list | grep ROOT | grep "/$" | cut -f1 -d" "
rpool/ROOT/ubuntu_9yvb17
</pre>
* create syslinux.cfg, change the root=ZFS label to match this computer
<pre>
cat << EOF | sed "s;root=.*$;root=ZFS=`zfs list | grep ROOT | grep "/$" | cut -f1 -d" "`;" > syslinux.cfg
default linux
label linux
kernel vmlinuz
append ro initrd=initrd.img root=ZFS=rpool/ROOT/ubuntu_02ruwj
EOF
</pre>
* copy linux boot files:
<pre>
cp /boot/vmlinuz vmlinuz
cp /boot/initrd.img initrd.img
</pre>
* repeat with /boot/efi-sdb, etc
* install script to set syslinux to boot the latest kernel
<pre>
cd ~/git/scripts
git pull
ln -s ~/git/scripts/etc/update_efi_syslinux.perl ~
</pre>
* update syslinux to boot the latest kernel
** run "~/update_efi_syslinux.perl" to check that it finds the EFI partitions and finds the correct kernel
** run "~/update_efi_syslinux.perl -u" to do the actual update
* or maybe install Ubuntu syslinux 6.04 and use files from there:
<pre>
apt install "syslinux*"
</pre>

= legacy boot using syslinux =

* NOTE: extlinux is not compatible with ext4 "64bit" feature, it should be turned off:
<pre>
mkfs.ext4 -O ^64bit /dev/sdX1
resize2fs -s /dev/sdX1
</pre>

* install syslinux and extlinux (THIS DOES NOT WORK!!!)

<pre>
apt -y install syslinux extlinux
dd if=/usr/lib/syslinux/mbr/mbr.bin of=/dev/sdX ### NOT /dev/sdX1 NOT !!!
cd /boot
cp /usr/lib/syslinux/modules/bios/menu.c32 .
extlinux -i .
</pre>

* install syslinux and extlinux

* copy from old SL6 USB disk (this is extlinux 6.02)
<pre>
root@localhost:/boot# ls -l
-rwxr-xr-x 1 root root 218952 Jan 28 17:40 extlinux
-rw-r--r-- 1 root root 402 Jan 29 14:45 extlinux.conf
-rw-r--r-- 1 root root 496 Jan 29 14:39 extlinux.conf~
-r--r--r-- 1 root root 122044 Jan 29 14:39 ldlinux.c32
-r--r--r-- 1 root root 67072 Jan 29 14:39 ldlinux.sys
-rwxr-xr-x 1 root root 24156 Jan 28 17:40 libutil.c32
-rw-r--r-- 1 root root 304 Jan 28 17:40 mbr.bin
-rw-r--r-- 1 root root 26140 Jan 28 17:40 memdisk
-rw-r--r-- 1 root root 69043 Jan 28 17:40 memtest86+-4.20.iso.zip
-rw-r--r-- 1 root root 183012 Jan 28 17:40 memtest86+-5.01
-rw-r--r-- 1 root root 26568 Jan 28 17:40 menu.c32
</pre>

* install
<pre>
dd if=mbr.bin of=/dev/sdX ### NOT /dev/sdX1 NOT !!!
extlinux -i .
</pre>
* check that partition /dev/sdX1 is marked bootable (fdisk command "a")

* create /boot/extlinux.conf

<pre>
DEFAULT menu.c32
PROMPT 0
TIMEOUT 50

MENU TITLE TRIUMF DAQ USB BOOT32 ver K.O. 2025jan28

LABEL linux
MENU DEFAULT
kernel /vmlinuz
append initrd=/initrd.img panic=60 rootdelay=5 rootwait rw root=/dev/sda1

LABEL linux-6.1.0-28-686
kernel vmlinuz-6.1.0-28-686
append initrd=initrd.img-6.1.0-28-686 panic=60 rootdelay=5 rootwait rw root=/dev/sda1

LABEL memtest
kernel memtest86+-1.65
</pre>

= Add user to login spash screen =

For user login to local machine, if it doesn't work by default. User should exist already from NIS service.

<pre>
sudo /bin/bash # as root
cd /var/lib/AccountsService/users
ls # should at least display the "wheel" user
cp wheel username # we're going to copy the user account settings over to our new user
</pre>

Ubuntu

2026-04-16T22:17:39Z

Olchansk: /* Install apache httpd proxy for midas and elog */

= Prerequisites =

* before setting up new machine run memory test
* prepare flash drive with free version of memtest86: https://www.memtest86.com
* test boot from flash drive, test takes ~ few hours
* test will end with summary page, if passed continue with Ubuntu
* number that might be worth noting is memory latency

= Ubuntu version =

<pre>
lsb_release -a
uname -a
</pre>

= Ubuntu installer =

* updated for Ububtu LTS 20.04.01, 22.04.1, 24.04 (only minor differences)

* download the latest Ubuntu LTS desktop installer iso image
* dd the image to a USB key
* power down, disconnect all disks (all HDDs, all SSDs, all M.2)
* connect the SSD to be used as system disk
* if system will use mirrored SSDs (using ZFS mirror), leave second SSD disconnected, we will activate it later
* power up
* boot from USB key in legacy mode or UEFI mode (select this in the BIOS boot menu - F2 or F8 for ASUS, F11 for Supermicro)
* follow the instruction:
* "try ubuntu or install ubuntu" - choose "install"
* select language - accept default
* "updates and other software" - accept default settings ("normal install")
* "installation type" - select "advanced features" and "experimental: use ZFS"
* accept partition choice
* "where are you?" - select "Vancouver" (PST time zone)
* "who are you?" - leave all fields blank, except "username" set to "wheel", "password" set to the root password. hostname will be set later after configuring the network
* don't install third party sw
* installation runs in a few minutes, when finished, reboot
* login as user wheel
* answer annouying questions:
* "livepatch" - say "next"
* "help improve" - select "do not send", say "next"
* "privacy" - leave "location" as "off", say "next"
* "ready to go", say "done"
* right-click on the desktop, say "open in terminal", a shell will open
* say "sudo /bin/bash", enter the root password, you now have the root shell
* run nm-connection-editor to configure the network. use netmask 255.255.224.0, gateway 142.90.100.18, DNS 142.90.100.19, search path "triumf.ca"
* after network is up (can ping ladd00), continue with post-installation steps below

= Install instructions =

== prepare ==
<pre>
apt update
apt upgrade
</pre>

== install ssh ==
<pre>
apt install ssh
</pre>

== install git/scripts ==
<pre>
apt -y install git
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd scripts
git pull
</pre>

* if needed, update git/scripts repository from ladd00 to daq00:
* git remote -v ### if it says daq00, we are good
* git remote set-url origin https://daq00.triumf.ca/~olchansk/git/scripts.git
* git pull ### check that it works

== configure hostname ==
<pre>
vi /etc/hostname
</pre>

== disable swap ==

ubuntu installer creates a 2 GB swap partition, not useful
on 32-64 GB machine, disable it:
<pre>
vi /etc/fstab ### comment out the "swap" line
</pre>

* on 64 GB RAM machines swap is not useful
* on machines booted from network (NFS-ROOT), swap does not work
* on machines running from flash (RPi, etc), flash is too slow for useful swap
* swap configured by linux installers invariably has wrong size and is not useful

<pre>
systemctl disable dphys-swapfile
systemctl stop dphys-swapfile
dphys-swapfile uninstall
</pre>

== maybe reboot ==

this is a good point to reboot the machine to boot
the latest kernel and to set the correct hostname

== install etckeeper ==

keep contents of /etc in a git repository:

<pre>
apt -y install etckeeper
</pre>

== set timezone ==
<pre>
timedatectl list-timezones | grep -i vancouver
timedatectl set-timezone America/Vancouver
</pre>

== install time synchronization ==

check if chrony is installed correctly and is synched to TRIUMF time servers:

<pre>
chronyc sources
chronyc tracking
</pre>

if not, remove old chrony and

<pre>
apt -y remove chrony
apt -y purge chrony
</pre>

and install it from scratch:

<pre>
apt -y install chrony
cd ~/git/scripts
git pull
cd ~
cp ~/git/scripts/etc/triumf.sources /etc/chrony/sources.d/
systemctl disable systemd-timesyncd.service
systemctl stop systemd-timesyncd.service
systemctl disable ntp
systemctl stop ntp
systemctl enable chrony
systemctl restart chrony
chronyc sources
chronyc tracking
</pre>

== reenable systemd-timesyncd ==

ONLY IF CHRONY DOES NOT WORK

To configure systemd-timesyncd, set "NTP=" in /etc/systemd/timesyncd.conf

<pre>
apt remove chrony
cat /etc/systemd/timesyncd.conf
systemctl enable systemd-timesyncd.service
systemctl restart systemd-timesyncd.service
systemctl status systemd-timesyncd.service
timedatectl status
timedatectl timesync-status
</pre>

== enable outgoing email (debian 11) ==

this is different from ubuntu 20. it uses /etc/mailname and it hardwires the hostname into main.cf.

== enable outgoing email ==

we have an unusual email configuration. outgoing email should work to deliver error messages, notices, etc. incoming email is disabled, we do not receive email for local users.

this causes problems with TRIUMF smtp server. if our message cannot be delivered (wrong email address or receipient computer is turned off), TRIUMF smtp server will generate a delivery failure notification email and try to send it to the "from" address of the failed message. but the "from" address does not receive any email, so another delivery failure notification email is generated and an attempt to deliver it. which again fails, rinse and repeat.

as solution, kray created a special rule, email from scrap.triumf.ca does not generate delivery failure notices. failed messages sit in the queue for 5 days, then they are deleted. (K.O. - confirmed with kray 3jan2024).

to make this work we use the msmtp MTA package.

<pre>
cd ~
apt -y install mailutils msmtp msmtp-mta # say "no" to apparmor support
apt -y remove postfix
apt -y purge postfix # remove old config files
apt -y install bsd-mailx
cd ~/git/scripts/etc
git pull
/bin/cp -fv aliases /etc/aliases
/bin/cp -fv msmtprc /etc/msmtprc
/bin/rm -vf ~root/.forward
/bin/rm -vf /etc/mailname
Mail root
Subject: test
test
^D
CC: <CR>
</pre>

== enable outgoing email (postfix) ==

THIS IS OBSOLETE!!!

* TRIUMF: use smtp.triumf.ca
* CERN: use cernmx.cern.ch

<pre>
apt install postfix ### select "satellite system", enter full hostname "xxx.triumf.ca", enter "smtp.triumf.ca"
apt install mailutils
dpkg-reconfigure postfix ### (if postfix already installed)
</pre>
<pre>
echo olchansk@triumf.ca lindner@triumf.ca bsmith@triumf.ca dfujimoto@triumf.ca >> ~root/.forward
mailx root
test
^D
</pre>

== enable ping for all users (debian 11) ==

Without this tweak, Debian will report "operation not permitted" if a user tries to ping somewhere.

<pre>
echo 'net.ipv4.ping_group_range = 0 1000' > /etc/sysctl.d/99-ping.conf
</pre>

== disable apparmor ==

On NFS-Root network booted machines!

If "man man" returns "permission denied" and syslog reports apparmor "sendmsg DENIED" errors, disable apparmor. This is supposedly fixed in kernel 6.0 and later (to be confirmed), see https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/1784499

Disable apparmor, see https://ubuntu.com/server/docs/security-apparmor

This takes effect after a reboot.

<pre>
systemctl stop apparmor.service
systemctl disable apparmor.service
</pre>

If on boot apparmor appears to still be confining apps (for example, mysql), edit the file /etc/default/grub and change

<pre>
GRUB_CMDLINE_LINUX=""
</pre>

to

<pre>
GRUB_CMDLINE_LINUX="apparmor=0"
</pre>

Run sudo update-grub and reboot.

== install missing packages ==

(apt eats terminal input, even the "yes |" trick does not quite work,
repeat the following commands until they report that everything
is installed)

<pre>
yes | apt -y install ssh tcsh ethtool ncat rsync strace net-tools traceroute time minicom screen git lsof debsums tmux iptables telnet pax rpm mtools at gdisk tcpdump
yes | apt -y install sysstat smartmontools lm-sensors
yes | apt -y install lsb-release
apt -y install vim # in addition to default vim-tiny, requested by IRIS
apt -y install gedit # requested by TACTIC
apt -y install tcl
apt -y install mc # requested by sol
apt -y install pax rpm alien ### package converter tools
apt -y install flex bison
apt -y install neofetch
apt -y install snmp snmp-mibs-downloader
apt -y install git subversion g++ gfortran cmake doxygen
apt -y install curl libcurl4 libcurl4-openssl-dev
### conflits with mysql packages ### apt -y install mariadb-client libmariadb-dev ### mysql client for MIDAS
apt -y install mysql-client libmysqlclient-dev
apt -y install postgresql-common libpq-dev ### postgresql client for MIDAS
yes | apt -y install libz-dev libzstd-dev sqlite3 libsqlite3-dev unixodbc-dev
yes | apt -y install libssl-dev
yes | apt -y install emacs xemacs21 joe
yes | apt -y install gnuplot dos2unix
yes | apt -y install mutt bsd-mailx # email clients
yes | apt -y install liblz4-tool pbzip2 libbz2-dev
yes | apt -y install libc6-dev-i386 # otherwise no /usr/include/sys/types.h
yes | apt -y install libreadline-dev
yes | apt -y install ubuntu-mate-themes
yes | apt -y install libmotif-dev libxmu-dev
yes | apt -y install libusb-dev libusb-1.0-0-dev
yes | apt -y install i2c-tools libi2c-dev libi2c0
yes | apt -y install xfig gsfonts-x11 gsfonts-other # install fonts for xfig
yes | apt -y install libjson-perl
yes | apt -y install libgsl-dev # additional GNU Scientific Library
yes | apt -y install qt5-default # Qt development
yes | apt -y install python3-full python3-dev python3-dbg python3-pip ### for pyROOT
yes | apt -y install imagemagick imagemagick-common ckeditor # for elog
yes | apt -y install libjpeg-dev libjpeg-progs libjpeg-tools
yes | apt -y install linux-tools-common linux-tools-generic # cpupower frequency-info
yes | apt -y install rdesktop remmina remmina-plugin"*" # requested by POL
yes | apt -y install nlohmann-json3-dev # required to build MIDAS with ROOT 6.30 on Ubuntu-22
apt -y install dpkg-dev cmake g++ gcc binutils libx11-dev libxpm-dev libxft-dev libxext-dev python3 libssl-dev libafterimage0 # from https://root.cern/install/dependencies/
apt -y install gfortran libpcre3-dev xlibmesa-glu-dev libglew-dev libftgl-dev libmysqlclient-dev libfftw3-dev libcfitsio-dev graphviz-dev libldap2-dev python3-dev python3-numpy libxml2-dev libkrb5-dev libgsl0-dev qtwebengine5-dev nlohmann-json3-dev libtbb-dev libavahi-compat-libdnssd-dev # from https://root.cern/install/dependencies/
apt -y install libvdt-dev # for ROOT 6.32 on Ubuntu-24
apt -y install autoconf automake gperf # gnu package build tools
apt -y install u-boot-tools # for Xilinx petalinux
#apt -y install linux-headers-generic # to build linux kernel drivers
apt -y install htop
</pre>

Ubuntu LTS 20.04:
<pre>
yes | apt -y install linux-image-generic-hwe-20.04 linux-tools-virtual-hwe-20.04 # enable linux 5.11 series kernel
</pre>

Ubuntu LTS 22.04:
<pre>
apt -y install linux-generic-hwe-22.04 # enable linux 6.8.0 series kernel
</pre>

Ubuntu LTS 24.04:
<pre>
apt -y install linux-generic-hwe-24.04 # enable linux 6.14.0 series kernel
</pre>

== remove snap ==

remove snap at this point.

Do

<pre>
snap list
</pre>

to see a list of installed snaps. Use

<pre>
snap remove <item>
</pre>

to remove each. You will need to remove snapd last.

<pre>
systemctl stop snapd
systemctl disable snapd
apt purge snapd
rm -rf ~/snap
</pre>

== install non-snap firefox ==

delay this until the very end, download of firefox-esr deb package is very slow.

See https://askubuntu.com/questions/1399383/how-to-install-firefox-as-a-traditional-deb-package-without-snap-in-ubuntu-22

<pre>
add-apt-repository ppa:mozillateam/ppa
apt install firefox-esr
echo run firefox-esr
</pre>

Enable automatic updates:

* rerun [[#Enable_automatic_updates]]

== configure DNS ==

<pre>
cd ~/git/scripts
git pull
mkdir /etc/systemd/resolved.conf.d
cp etc/resolved-triumf.conf /etc/systemd/resolved.conf.d/
systemctl restart systemd-resolved
resolvectl
#systemd-analyze cat-config systemd/resolved.conf
</pre>

== install ganglia ==

<pre>
apt -y install ganglia-monitor
cd ~root/git/scripts/ganglia
git pull
make install
./ganglia-all.perl
</pre>

fix gmond start before network is ready:

<pre>
mkdir /etc/systemd/system/ganglia-monitor.service.d
echo -e "[Unit]\nAfter=network-online.target\n" > /etc/systemd/system/ganglia-monitor.service.d/local.conf
systemctl daemon-reload
systemctl cat ganglia-monitor.service
</pre>

== install ganglia server ==

'''On the main computer only! (daq00, dsdaqgw, etc)'''

* zfs create rpool/ganglia
* apt install gmetad php php-xml rrdtool
* mv /etc/ganglia/gmetad.conf /etc/ganglia/gmetad.conf-stock
* create /etc/ganglia/gmetad.conf with the following contents:
<pre>
data_source "my cluster" 15 localhost
RRAs "RRA:AVERAGE:0.5:1:244" "RRA:AVERAGE:0.5:24:244" "RRA:AVERAGE:0.5:168:244" "RRA:AVERAGE:0.5:672:244" "RRA:AVERAGE:0.5:5760:374"
setuid_username "ganglia"
rrd_rootdir "/ganglia/rrds"
case_sensitive_hostnames 0
</pre>
* mkdir /ganglia/rrds
* chown ganglia:ganglia /ganglia/rrds
* systemctl restart gmetad
* create /etc/ganglia/gmond-collect.conf
<pre>
globals {
daemonize = yes
setuid = yes
user = nobody
debug_level = 0
max_udp_msg_len = 1472
mute = no
#deaf = yes
deaf = no
allow_extra_data = yes
host_dmax = 0 /* 86400 */ /*secs. Expires (removes from web interface) hosts in 1 day */
host_tmax = 20 /*secs */
cleanup_threshold = 300 /*secs */
gexec = no
send_metadata_interval = 600 /*secs */
}

/*
* The cluster attributes specified will be used as part of the <CLUSTER>
* tag that will wrap all hosts collected by this instance.
*/
cluster {
name = "DAQ"
owner = "TRIUMF DAQ"
latlong = "unspecified"
url = "https://daq.triumf.ca"
}

/* The host section describes attributes of the host, like the location */
host {
location = "unspecified"
}

udp_send_channel {
host = daq00.triumf.ca
bind_hostname = yes
port = 8649
}

udp_recv_channel {
#mcast_join = 239.2.11.71
port = 8649
#bind = 239.2.11.71
retry_bind = true
# Size of the UDP buffer. If you are handling lots of metrics you really
# should bump it up to e.g. 10MB or even higher.
#buffer = 10485760
buffer = 200000
family = ipv4

acl {
default = "deny"
access {
# vlan1 daq00
ip = 142.90.111.168
mask = 19
action = "allow"
}
access {
# MUSR VLAN
ip = 142.90.154.73
mask = 8
action = "allow"
}
access {
# KVM network
ip = 192.168.1.1
mask = 8
action = "allow"
}
}
}

udp_recv_channel {
#mcast_join = 239.2.11.71
port = 8649
#bind = 239.2.11.71
retry_bind = true
# Size of the UDP buffer. If you are handling lots of metrics you really
# should bump it up to e.g. 10MB or even higher.
#buffer = 10485760
buffer = 200000
family = ipv6

acl {
default = "deny"
access {
# ALPHA network
ip = 2001:1458:202:fd::100:aa
mask = 8
action = "allow"
}
#access {
# # ALPHA-g network
# ip = 192.168.1.1
# mask = 8
# action = "allow"
#}
}
}

tcp_accept_channel {
port = 8649
bind = localhost
# If you want to gzip XML output
gzip_output = no
}
</pre>
* add to /etc/rc.local
<pre>
/usr/sbin/gmond -c /etc/ganglia/gmond-collect.conf &
systemctl restart gmetad &
</pre>
* cd /ganglia
* git clone https://daq00.triumf.ca/~olchansk/git/ganglia-web.git
* cd ganglia-web
* git checkout KO1
* ln -s /ganglia/ganglia-web /var/www/html/ganglia
* make dwoo directories
<pre>
mkdir /var/lib/ganglia-web
chown www-data:www-data /var/lib/ganglia-web
mkdir /var/lib/ganglia-web/dwoo
chown www-data:www-data /var/lib/ganglia-web/dwoo
mkdir /var/lib/ganglia-web/dwoo/compiled
chown www-data:www-data /var/lib/ganglia-web/dwoo/compiled
mkdir /var/lib/ganglia-web/dwoo/cache
chown www-data:www-data /var/lib/ganglia-web/dwoo/cache
</pre>
* open https://alphacpc05.cern.ch/ganglia/

== install gonodeinfo ==

* go to https://bitbucket.org/dd1/gonodeinfo follow instructions:
<pre>
apt -y install golang
mkdir ~/git
cd ~/git
#git clone https://bitbucket.org/dd1/gonodeinfo.git
git clone https://daq00.triumf.ca/~olchansk/git/gonodeinfo.git
cd gonodeinfo
git remote set-url origin https://daq00.triumf.ca/~olchansk/git/gonodeinfo.git
git pull
make
make install # install gonodeinfo agent
cd ~ # this is important
</pre>
* edit /etc/gonodeinfo.conf
* change "Description", "Location", "User" and "Administrator" as appropriate (or delete them)
* change "Servers" to read: Servers: daq00.triumf.ca:8601
* run "gonodeinfo -v"
* if error is "connection refused". go to the nodeinfo server to add this client to the access control list:
* on the gonodeinfo server: run /opt/gonodeinfo/gonodereceive.exe -a daq13
* try gonodeinfo again, there should be no error
* on the gonodeinfo server: run gonodereport, look at the web pages, the new machine should be listed now

== install emailonreboot ==

send an email if computer is rebooted

<pre>
ssh root
mkdir -p ~/git
cd ~/git
git clone https://daq00.triumf.ca/~olchansk/git/rpms.git
cd rpms/emailonreboot
git pull
make
make install
</pre>

== install monitor_nfs ==

monitor NFS mounts and complain about dead, stale and hung mounts

<pre>
ssh root
mkdir -p ~/git
cd ~/git
git clone https://daq00.triumf.ca/~olchansk/git/rpms.git
cd rpms/monitor_nfs
git pull
make
make install
</pre>

== install fonts for EPICS ==

<pre>
apt -y install xfonts-100dpi xfonts-75dpi
killall Xorg # restart Xorg (i.e. "killall Xorg", this will log you out from the console)
xlsfonts | grep -i helvetica ### should show fonts with different sizes, not just size 0 (scalable)
</pre>

== install libz.so.1 for CentOS compatibility ==

KO - confirm which versions on quartus need this.

<pre>
yes | apt-get -y install zlib1g
yes | apt-get -y install zlib1g:i386 libc6:i386 libgcc1:i386 gcc-6-base:i386
</pre>

== install ld-ldb-x86-64.so.3 for Quartus compatibility ==

Not clear from package this is supposed to come from. Copied from U-20.

Without this, Quartus lmgrd does not run.

<pre>
cd /lib64
ln -s ld-linux-x86-64.so.2 ld-lsb-x86-64.so.2
ln -s ld-linux-x86-64.so.2 ld-lsb-x86-64.so.3
</pre>

should look like this:
<pre>
root@daq13:/lib64# ls -l
total 3
lrwxrwxrwx 1 root root 44 Jan 28 09:07 ld-linux-x86-64.so.2 -> ../lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
lrwxrwxrwx 1 root root 20 Feb 18 16:45 ld-lsb-x86-64.so.2 -> ld-linux-x86-64.so.2
lrwxrwxrwx 1 root root 20 Feb 18 16:45 ld-lsb-x86-64.so.3 -> ld-linux-x86-64.so.2
root@daq13:/lib64#
</pre>

== install libpng12.so.0 for Quartus 13.0sp1 and 13.1.4.182 ==

<pre>
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0.50.0
/bin/cp -pv libpng12.so.0 libpng12.so.0.50.0 /lib/x86_64-linux-gnu/
</pre>

== install packages for Xilinx ==

ubuntu LTS 22.04 vivado 2020.1

<pre>
apt install autoconf libtool
apt install libtinfo5
apt install texinfo
apt install zlib1g:i386
</pre>

== install packages for building ROOT ==

<pre>
apt -y install libx11-dev libxpm-dev libxft-dev libxext-dev libpng-dev libjpeg-dev xlibmesa-glu-dev libxml2-dev libgsl-dev cmake
</pre>

== install wine ==

As far as I know, only needed for BNMR/BNQR

<pre>
apt install wine winetricks
</pre>

== install lightdm ==

unlike the default gdm login manager, lightdm shows the machine hostname and does not require an extra mouse click to swicth from screen saver to login mode.

<pre>
apt -y install lightdm
# select lightdm
</pre>

== install desktop environments ==

note: default display manager and default desktop are deficient, please do not skip this step.

note: if apt asks to choose the display manager, select "lightdm"

note: KO - I recommend the "MATE" desktop.

note: you will have to cut-and-paste this several times because "apt" eats commands, even with "-y" and even piped from "yes".

note: DF - on U24 this may re-install snap

<pre>
# install MATE desktop
apt -y install ubuntu-mate-core ubuntu-mate-desktop ubuntu-mate-themes
# install Cinnamon desktop
apt -y install cinnamon
# install KDE desktop
apt -y install kde-standard kubuntu-settings-desktop
# install Lxqt desktop
# apt -y install lxqt # conflict over kubuntu-desktop, kubuntu-settings-desktop and desktop-base
# install Xfce4 desktop
apt -y install xfce4
</pre>

== install ROOT ==

Please install ROOT per instructions at https://root.cern.ch.

NOTE1: The ROOT package available from Ubuntu repositories is severely out of date and cannot be used with MIDAS and ROOTANA. ### DO NOT DO THIS! apt-get install root-system

NOTE2: as of 2017-Jan-09, ROOT binary kits for Ubuntu do not work (use GCC 5 instead of GCC6), build from source instead.

== Install x2go ==

<pre>
apt-get update
apt-get install x2goserver x2goserver-xsession
</pre>

== enable root login from ladd00/daq00 ==
<pre>
ssh localhost
CTRL-C
/bin/cp ~root/git/scripts/etc/authorized_keys ~root/.ssh/
</pre>

== disable ssh access from outside of TRIUMF ==

to stop ssh login spam, disable ssh access from outside of TRIUMF. this can be done by requesting a firewall block through the helpdesk or by local firewall rule:

<pre>
echo iptables -I INPUT ! -s 142.90.0.0/255.255.0.0 -p tcp --dport 22 -j REJECT >> /etc/rc.local
/etc/rc.local
</pre>

== install smart-status ==
<pre>
ln -s ~/git/scripts/smart-status/smart-status.perl ~root/
</pre>

== enable boot menu and boot messages ==

This will enable the grub menu (with a 10 sec timeout) and
replace black screen with exciting linux boot messages.

* emacs -nw /etc/default/grub
<pre>
GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=menu
GRUB_TIMEOUT=10
GRUB_RECORDFAIL_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="vga=769 video=640x480"
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX=""
#GRUB_GFXMODE=640x480
</pre>
* update grub config:
<pre>
grub-mkconfig -o /boot/grub/grub.cfg
</pre>

== Disable welcome message on ssh ==

On ssh, there is a lengthy welcome message. To disable:

<pre>
vim /etc/ssh/sshd_config
</pre>

Ensure that the file reads

<pre>
PrintMod no
</pre>

Then,

<pre>
vim /etc/pam.d/sshd
</pre>

comment out the lines:

<pre>
session optional pam_motd.so motd=/run/motd.dynamic
session optional pam_motd.so noupdate
</pre>

= Enable automatic updates =

<pre>
apt install unattended-upgrades
cd ~/git/scripts
git pull
/bin/cp -v etc/99apt-conf-ko /etc/apt/apt.conf.d/
apt-config dump | grep Unattended
</pre>

Following is obsolete:

* emacs -nw /etc/apt/apt.conf.d/50unattended-upgrades
** uncomment in Allowed-Origins "-security" and "-updates"
** add in Allowed-Origins: "Google LLC:stable";
** uncomment/add: "Unattended-Upgrade::Mail "root";
* emacs -nw /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
</pre>
* test: unattended-upgrade --dry-run -v

NOTE: update-on-shutdown is disabled.

NOTE: there is no update-on-boot, but:

NOTE: if machine was off for a long time, the systemd update timer would have expired and it will fire soon after reboot, causing an automatic update run. this is unwanted, and there is no fix or workaround for it. K.O. June-2023.

= Fix bpool is full (obsolete) =

THIS IS CAUSED BY OBSOLETE PACKAGE zsys. PLEASE: apt remove zsys

= IPMI instructions =

IPMI is the board management hardware on Supermicro and other server motherboards. This includes hardware sensors - fan rotation speed, temperatures and power supply voltages.

<pre>
apt-get install ipmitool
systemctl enable ipmievd
systemctl restart ipmievd
</pre>

Run:
* ipmitool sel list ### event list
* ipmitool sel elist ### event list
* ipmitool sel clear ### clear event list (if it becomes full)
* ipmitool sensor ### report hardware sensors

= move /home/wheel (U-24) =

Ubuntu LTS 24 installed on ZFS has rpool/USERDATA/home_xxx mounted on /home,
has to be moved or autofs /home will not work.

<pre>
zfs list | grep USERDATA | grep home | cut -f1 -d" "
zfs set -u mountpoint=/home1 `zfs list | grep USERDATA | grep home | cut -f1 -d" "`
emacs -nw /etc/passwd # change mount for wheel account to /home1/wheel
</pre>

This will not take effect until rebooting. Please ensure that ssh from root@daq00 to this computer works before rebooting; if ssh from root@daq00 doesn't work and you mess this up you will be locked out of wheel account. Once you verify that this works, reboot and make sure that when you login as wheel that home directory is /home1/wheel.

= move /home/wheel =

note: this MUST be done if ZFS root and NIS/autofs with /home.

Default location of wheel's home directory will collide with autofs /home, it has to be moved,
for example to /wheel.

<pre>
# logout from the wheel user
# go to another computer
ssh root@daqubuntuxxx
zfs list | grep wheel ### identify zfs name wheel_xxxxxx
#zfs set mountpoint=/wheel rpool/USERDATA/wheel_hm8fzh
zfs set mountpoint=/wheel `zfs list | grep wheel | cut -f1 -d" "`
zfs list | grep wheel
emacs -nw /etc/passwd ### change wheel's home directory from /home/wheel to /wheel
su - wheel ### check that user wheel still works
</pre>

This will break wheel's ability to run snap programs, such as firefox, install chrome as listed below.

= enable NIS (ubuntu 22.04, 24.04, debian 11, 12) =

<pre>
apt -y install rpcbind nis
echo DAQ-NIS >> /etc/defaultdomain
echo ypserver daq00.triumf.ca >> /etc/yp.conf
systemctl enable ypbind.service
systemctl restart ypbind.service
systemctl status ypbind.service
ypwhich -m
</pre>

enable ypserv:

<pre>
sed -i s/NISSERVER=false/NISSERVER=slave/ /etc/default/nis
/usr/lib/yp/ypinit -s daq00
echo ypserver localhost >> /etc/yp.conf
sed -i "s/ypserver .*/ypserver localhost/" /etc/yp.conf
systemctl enable ypserv
systemctl restart ypserv
systemctl restart ypbind
</pre>

update /etc/nsswitch.conf and enable hourly update of NIS maps:

<pre>
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
cp -pv nsswitch.conf-U24 /etc/nsswitch.conf
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>

If this is a new machine, then on the master NIS node (daq00), add this new node to /etc/netgroup, and update NIS maps (cd /var/yp; make)

= enable NIS (ubuntu 20.04) =

* apt-get -y install portmap nis ### will ask for NIS domain (DAQ-NIS)
* dpkg-reconfigure nis ### reconfigure if already installed
* ypwhich -m
* edit /etc/default/nis
** set "NISSERVER=slave"
** Ubuntu LTS 20.04, check that "YPBINDARGS=" is blank, remove "-no-dbus" if it is there
* #edit /etc/yp.conf, comment-out everything, add "domain DAQ-NIS server localhost"
* edit /etc/yp.conf, comment-out everything, add "ypserver localhost"
* /usr/lib/yp/ypinit -s daq00
* systemctl enable nis
* systemctl restart nis
* ypwhich
* ypwhich -m
* ypcat -k passwd
* vi /etc/nsswitch.conf ### add the automount line, modify the passwd, group and shadow lines to read this:
<pre>
# begin get data from nis
passwd: files nis
group: files nis
shadow: files nis
automount: files nis
netgroup: files nis
# end get data from nis
</pre>
* enable hourly update of NIS maps
<pre>
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>
* ### NOT NEEDED sudo vi /etc/idmapd.conf ### add line: "Domain = triumf.ca"

= enable autofs =

<pre>
apt -y install autofs
systemctl enable autofs
systemctl restart autofs
ls -l /home/olchansk ### test autofs, check file owner is correct
</pre>

= enable NFS server =

<pre>
apt install nfs-kernel-server
#edit /etc/exports
systemctl enable nfs-server
systemctl restart nfs-server
</pre>

= NIS master =

notes for setting up the NIS master

== wheel user ==

"wheel" is the default administrative user. We do not want it's password exported to NIS (encrypted password hash is world visible) and we do not want it's home directory exported to NFS (~wheel/.ssh is world visible and potentially writable: anybody can change ~wheel/.ssh/authorized_keys).

* move wheel's home directory from /home/wheel to /wheel (see special section about this)
* change wheel's UID and GID from 1000 to a value below MINUID in /var/yp/Makefile

== coherent uids ==

we do not want system accounts defined in /etc/passwd of the NIS master
to be included in the NIS map "passwd". this causes trouble on NIS clients
where newly installed packages fail to create local system users because same
user already exists in NIS.

This is controlled by MINUID in /var/yp/Makefile.

Historical TRIUMF uids start from around 200, but several clusters do not have any historic TRIUMF uids below 500 and MINUID is set to:
* DAQ-NIS: MINUID=200
* ISAC-NIS: MINUID=500
* TITAN-NIS: MINUID=500
* MUSR-NIS: MINUID=500
* TIG-NIS: MINUID=500 (100 on SL6 mother8pi)

Ubuntu 20 has two programs to create users:
* adduser - creates new users with UID 1000 and up as specified in /etc/adduser.conf. No problems here.
* adduser --system - creates new system users with UID 100 and up as specified in /etc/adduser.conf. No problems here.
* useradd - creates new users with UID 1000 and up as specified in /etc/login.defs. No problems here.
* useradd --system - creates new system users with UID 999 and down (read "man useradd", section at the end about SYS_UID_MAX). This collides with NIS MINUID, these system users will be included in the NIS map and cause trouble.

This problem cannot be fixed, SYS_UID_MIN, SYS_UID_MAX and UID_MIN in /etc/login.defs do not seem
to have any effect on UIDs chosen by "useradd --system". (tested on Ubuntu LTS 20.04).

So far only these system accounts seem to be affected by this:
* systemd-coredump
* ganglia

To fix:
* run "sort -r -n -t: -k3 /etc/passwd" to identify the last unused system user uid (range 100..200)
* run "sort -r -n -t: -k3 /etc/group" to identify the last unused system user gid (range 100.200)
* systemd-coredump: manually change UID and GID (package systemd-coredump is usually not installed)
* ganglia: same thing, then change ownership on all ganglia files.

Also read systemd author's opinion on system vs user UIDs:
https://github.com/systemd/systemd/issues/4850#issuecomment-265698275

= Fix systemd-logind NIS breakage =

!!! THIS IS NOT NEEDED FOR UBUNTU LTS 20.04 !!!

there is a delay in ssh logins for normal users. "ssh -v" shows the delay is after "pledge...". this
fix removes the delay.

systemd developers think that we should not use NIS and made sure there are
problems if we do. To give them credit, they do offer a workaround. Read this:
https://github.com/poettering/systemd/commit/695fe4078f0df6564a1be1c4a6a9e8a640d23b67

<pre>
mkdir /etc/systemd/system/systemd-logind.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-logind.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-logind.service
</pre>

= Fix systemd-udevd NIS breakage =

see same problem as above with udev getting stuck. ubuntu lts 20.04.

<pre>
mkdir /etc/systemd/system/systemd-udevd.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-udevd.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-udevd.service
</pre>

= Configure USB device permissions =

Configure USB device permissions for user access to USB-serial devices, Altera USB Blaster, etc.

* create file /etc/udev/rules.d/99-usb-chmod.rules with this contents:

<pre>
emacs -nw /etc/udev/rules.d/99-usb-chmod.rules
ACTION=="add", SUBSYSTEM=="usbmisc", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /dev/%c"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /proc/%c"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVICE}"
ACTION=="add", ENV{PHYSDEVBUS}=="usb-serial", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVPATH}=="/class/tty/ttyS*", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyUSB*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyACM*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyS*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", DEVPATH=="*video*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
</pre>

* reload udev rules: udevadm control --reload-rules
* apply new permissions: udevadm trigger --action=add
* watch udev activity: udevadm monitor -p

= Configure lightdm display manager =

* enable it
<pre>
echo lightdm | dpkg-reconfigure -fteletype lightdm
systemctl disable gdm
systemctl disable sddm
systemctl enable lightdm
</pre>

* make the MATE desktop as default
<pre>
cd ~root/git/scripts/
git pull
/bin/cp -v etc/lightdm_default_mate.conf /etc/lightdm/lightdm.conf.d/
</pre>

* enable login by NIS users
<pre>
/bin/cp -v etc/lightdm_enable_nis_login.conf /etc/lightdm/lightdm.conf.d/
</pre>

* restart lightdm
<pre>
systemctl stop gdm
systemctl restart lightdm
</pre>

= Install libpng12.so.0 =

Quartus 16 needs libpng12:

<pre>
wget http://mirrors.kernel.org/ubuntu/pool/main/libp/libpng/libpng12-0_1.2.54-1ubuntu1_amd64.deb
dpkg --install libpng12-0_1.2.54-1ubuntu1_amd64.deb
</pre>

= Install google-chrome =

<pre>
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
dpkg -i google-chrome-stable_current_amd64.deb
</pre>

confirm autoupdate is enabled, observe dl.google.com is present in the list of repositories:
<pre>
apt update
...
Get:5 https://dl.google.com/linux/chrome/deb stable/main amd64 Packages [1,094 B]
...
</pre>

FOLLOWING IS OBSOLETE:

Instructions from here:
https://www.ubuntuupdates.org/ppa/google_chrome?dist=stable

<pre>
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-tmp.list'
apt update
apt install google-chrome-stable
/bin/rm -f /etc/apt/sources.list.d/google-tmp.list
</pre>

= Install amanda client =

ONLY ON MACHINES THAT HOST HOME DIRECTORIES

* apt install amanda-client
* edit /etc/amandahosts
<pre>
amanda.triumf.ca amanda amdump
</pre>
* check permissions on /etc/amandahosts:
<pre>
root@daq00:/var/log/amanda# ls -l /etc/amandahosts
-rw------- 1 backup backup 49 Jan 27 10:48 /etc/amandahosts
</pre>
* fix if needed: chown backup.backup /etc/amandahosts; chmod a= /etc/amandahosts; chmod u=wr /etc/amandahosts
* edit /etc/amanda-security.conf, add this line:
<pre>
runtar:gnutar_path=/usr/bin/tar
</pre>

On the amanda machine:

* in amanda disklist, use dump type "bsdtcp-comp-user-tar"
* su - amanda and run amcheck -c daily daq00
<pre>
-bash-4.1$ amcheck -c daily daq00

Amanda Backup Client Hosts Check
--------------------------------
Client check: 1 host checked in 0.092 seconds. 0 problems found.

(brought to you by Amanda 3.3.7p1.git.685ff76d)
</pre>

= Enable rc.local =

For reasons unknown, Ubuntu LTS 20.04 does not enable /etc/rc.local. Do this:

<pre>
cd ~/git/scripts
git pull
cp -n -v etc/rc.local /etc/
chmod a+rx /etc/rc.local
cp etc/rc-local.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable rc-local
systemctl start rc-local
systemctl status rc-local
</pre>

= Remove unwanted packages =

<pre>
apt purge bash-completion # broken, adds unwanted "\" if "ls -l $ROOTSYS/<tab>"
apt purge zsys # broken, do not use
apt purge sddm # login manager
apt purge avahi-daemon avahi-autoipd # not sure what it does, observed using 100% CPU
apt purge modemmanager # probes all serial ports to see if it's a modem
</pre>

= Disable unwanted services =

<pre>
systemctl disable mpd
systemctl disable snapd
systemctl disable ModemManager
systemctl --global mask tracker-extract-3.service
systemctl --global mask tracker-miner-fs-3.service
systemctl daemon-reload
</pre>

= Disable sleep and suspend =

note: we see some computers randomly shutdown or go to sleep, log files indicates the "sleep" or "suspend" button was pushed by user, but no such buttons actually exist. this is the fix for this:

<pre>
systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target systemd-suspend.service systemd-hybrid-sleep.service
</pre>

= Enable crontab @reboot for MIDAS =

startup scripts have a bug - cron @reboot entries for normal users can run before autofs is ready, so if the home directory is on autofs/NFS, it cannot be accessed and the cron job fails. If MIDAS is supposed to be started by cron @reboot, it will not start (there *will* be an error message in /var/log/cron).

<pre>
mkdir /etc/systemd/system/cron.service.d
echo -e "[Unit]\nAfter=ypbind.service autofs.service\n" > /etc/systemd/system/cron.service.d/local.conf
systemctl daemon-reload
systemctl cat cron.service
</pre>

Explore the systemd dependency tree using "systemctl list-dependencies" maybe with "--all".

Visualize the exact boot sequence from previous boot: "systemd-analyze plot > xxx.svg", look at the svg file using a web browser.

Crontab entry to start midas: (install in the midas user crontab, not root crontab)

<pre>
su - midasuser
crontab -l
#@reboot /bin/bash -l -c "/home/trinat/bin/start-daq-applications"
#@reboot /bin/tcsh -c "/home/trinat/bin/start-daq-applications"
</pre>

= Install apache httpd proxy for midas and elog =

This will configure the HTTPS/SSL certificate using "certbot" and "letsencrypt" and configure an HTTPS web server using apache2.

First, configure apache2:

* execute these commands:
<pre>
apt -y install apache2
cd /etc/apache2
</pre>
* create new file conf-available/ssl-daq14.conf # use actual hostname instead of daq14
<pre>
SSLSessionCache shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout 300
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
SSLCryptoDevice builtin
</pre>
* create new file sites-available/daq14-ssl.conf # use actual hostname instead of daq14
<pre>
<IfModule mod_ssl.c>
<VirtualHost *:443>
ServerName daq14.triumf.ca
DocumentRoot /var/www/html
ErrorLog /var/log/apache2/daq14.log
SSLEngine on
# note SSLProtocol, SSLCipherSuite and some other settings are overwritten by /etc/letsencrypt/options-ssl-apache.conf
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SEED:!IDEA:!RC4
## use port specified in elogd.cfg
#ProxyPass /elog/ http://localhost:8082/ retry=1
## use mhttpd port
#ProxyPass / http://localhost:8080/ retry=1
Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"
<Location />
SSLRequireSSL
AuthType Basic
AuthName "DAQ password protected site"
Require valid-user
# create password file: touch /etc/apache2/htpasswd
# to add new user or change password: htpasswd /etc/apache2/htpasswd username
AuthUserFile /etc/apache2/htpasswd
RequestHeader set X-Remote-User %{REMOTE_USER}s
</Location>
#SSLCertificateFile /root/server.cert
#SSLCertificateKeyFile /root/server.key
</VirtualHost>
</IfModule>
</pre>
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* enable ssl module and new configurations:
<pre>
a2enmod ssl
a2enmod headers
a2enmod proxy
a2enmod proxy_http
a2enconf ssl-daq14
a2ensite daq14-ssl
</pre>
* disable default ssl sites
<pre>
a2dissite 000-default-le-ssl
a2dissite 000-default
ls -l /etc/apache2/sites-enabled/ ### should show only daq14-ssl.conf
</pre>
* check that there are no syntax problems
<pre>
apache2ctl configtest
</pre>
* enable and start apache2:
<pre>
systemctl enable apache2
systemctl restart apache2
systemctl status apache2
</pre>
* apache2 may fail to start, look in /var/log/apache2/error.log and /var/log/apache2/daq14.log
* if it says "Failed to configure ... certificate", proceed to the step for setting certbot.
* try to access https://daq14.triumf.ca
** you should see a complaint about self-signed certificate
** you should see a request for password (do not login yet)
** if you get "connection refused", HTTPS port 443 may need to be enabled in the local firewall, look at documentation for ufw.
Second, configure certbot:

(Note: as of 2018-01-18 certbot requires use of http port 80 to get the initial https certificate,
renewal can continue to use the https port 443)

(Note: as of 2019-01-?? certbot requires use of port 80 for renewals)

(Note: unsurprisingly, this requires outside access to connect with letsencrypt, so won't work if PC is only accessible from on-site network)

* check that port 80 is not used by anything:
* netstat -an | grep LISTEN | grep ^tcp | grep 80
* lsof -P | grep -i tcp | grep LISTEN | grep 80
* if lsof reports that apache2 is listening on port 80, follow the apache2 instructions above (remove "listen 80" from apache2.conf

* install certbot (if necessary open tcp port 80 in the firewall, see documentation for ufw):
<pre>
apt install certbot python3-certbot-apache
certbot certonly --standalone --installer apache
</pre>
* then answer questions:
* "activate HTTPS for daq14.triumf.ca" - say ok
* "enter email address" - enter your own email address
* "please read terms..." - read the terms and say "agree"
* it will take a few moments...
* "congratulations..." - say ok.
<pre>
certbot install --apache --cert-name daq14.triumf.ca
</pre>
* then answer questions:
* "choose redirect..." - say "1" (no redirect)
* look inside /etc/apache2/sites-enabled/daq14-ssl.conf to see that SSLCertificateFile & co point to certbot certificates in
/etc/letsencrypt/live/daq14.triumf.ca/
* to check current renewal and to update the certbot config file in /etc/letsencrypt/renewal, run this:
<pre>
certbot renew --standalone --installer apache --force-renewal
</pre>

NOTE: this certificate will expire in 3 months, automatic renewal should work with current version of certbot

Third, activate password protection:

* as shown in the config file above, create password file and initial user: (replace "midas" with specific username)
<pre>
touch /etc/apache2/htpasswd
htpasswd /etc/apache2/htpasswd midas
</pre>

* restart apache2
<pre>
systemctl restart apache2
systemctl status apache2
</pre>

From here:
* enable proxy for MIDAS mhttpd - uncomment redirect in the config file above
* enable proxy for ELOG - ditto
<pre>
a2enmod proxy
a2enmod proxy_http
apache2ctl configtest
systemctl restart apache2
</pre>

* try accessing MIDAS https://daq14.triumf.ca/ (make sure mhttpd is running)
* if it's not working, check odb setting FIXME!
* try accessing ELog https://daq14.triumf.ca/elog/ (make sure elogd is running)
* if it's not working, check elogd.cfg file and make sure
<pre>
SSL = 0
</pre>

NOTE: if certbot fails with errors about 'module' object has no attribute 'pyopenssl',
try this: pip install requests==2.6.0

== generate self-signed certificate ==

<pre>
# cd $HOME
# openssl req -nodes -new -x509 -keyout server.key -out server.cert -days 1001
...+....+..+..........+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+..+...+.........+......+.+...+...+.....+...............+.........+...+.+......+...+...........+....+...+..+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+......+.+...+..+.......+..+...+.......+......+...+..+...+......+....+...............+..+...+....+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
......+......+.+..+......+.+......+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.....+......+.+.........+......+.....+.+..+...+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+....+......+.....+...+...+.......+..+.+........+.+...+......+..+..........+..+.+...........+...+.......+......+.....+.......+...+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:CH
State or Province Name (full name) [Some-State]:Geneve
Locality Name (eg, city) []:CERN
Organization Name (eg, company) [Internet Widgits Pty Ltd]:CERN
Organizational Unit Name (eg, section) []:ALPHA experiment
Common Name (e.g. server FQDN or YOUR name) []:alphacpc05.cern.ch
Email Address []:
root@alphacpc05:~#
root@alphacpc05:~#
root@alphacpc05:~# ls -l
-rw-r--r-- 1 root root 1375 juil. 10 21:43 server.cert
-rw------- 1 root root 1708 juil. 10 21:42 server.key
root@alphacpc05:~# systemctl restart apache2
</pre>

= Enable elog PDF preview =

NOTE: looks like U-24 already has this correctly.

see https://stackoverflow.com/questions/52998331/imagemagick-security-policy-pdf-blocking-conversion

* xemacs -nw /etc/ImageMagick-6/policy.xml
* remove this section at the end:
<pre>

<policy domain="coder" rights="none" pattern="PS" />
<policy domain="coder" rights="none" pattern="PS2" />
<policy domain="coder" rights="none" pattern="PS3" />
<policy domain="coder" rights="none" pattern="EPS" />
<policy domain="coder" rights="none" pattern="PDF" />
<policy domain="coder" rights="none" pattern="XPS" />
</pre>

= Install Jupyter notebook =

<pre>
From https://jupyter.org/install
apt install python3-pip
pip install jupyterlab
pip install notebook
~/.local/bin/jupyter notebook
watch the http://localhost:8888 URL that it printed
say "no" to offer to start firefox (it will not work!)
URL is: http://localhost:8888/tree?token=xxx
from the machine where you are running the web browser (i.e. google-chrome), run (replace trinat@trinatdaq with the username and machine name where you started jupyter)
open a new shell and run: ssh -v trinat@trinatdaq -L 8888:localhost:8888
in the web browser, open http://localhost:8888
this gives us the login page
in the password or token entry field, put the token from the "tree?token=xxx" above (printed by jupyter on startup)
push button "login"
jupyter page should open with the list of files in the trinat home directory
congratulate Brian with full success
</pre>

= Install ZFS quota report =

If there are any ZFS volumes, install script to report disk and quota usage

<pre>
cd ~/git/scripts/quotareport
git pull
mkdir /var/www/html/zfsquotareport
cp -pv ~/git/scripts/quotareport/sorttable.js /var/www/html/zfsquotareport/
ln -s $PWD/zfsquotareport.perl /etc/cron.daily/
touch /etc/crontab
</pre>

If httpd is configured to redirect "/" to MIDAS mhttpd:
* add following to /etc/apache2/sites-enabled/xxx-ssl.conf in front of "ProxyPass / ..."
* run "systemctl reload apache2"
<pre>
## do not proxy zfs quota report directory
ProxyPass /zfsquotareport/ !
</pre>

= Install PHP =

* apt install php libapache2-mod-php
* systemctl restart apache2
* create /var/www/html/info.php
<pre>
<?php

phpinfo();
</pre>
* open https://daq00.triumf.ca/info.php

= Configure TRIUMF printers =

<pre>
systemctl stop cups
systemctl stop cups-browsed.service
systemctl disable cups
systemctl disable cups-browsed.service
systemctl stop snap.cups.cupsd.service
systemctl stop snap.cups.cups-browsed.service
systemctl disable snap.cups.cupsd.service
systemctl disable snap.cups.cups-browsed.service
echo "ServerName printers.triumf.ca" > /etc/cups/client.conf
lpstat -a
</pre>

= Enable core dumps =

By default, Ubuntu LTS 20.04 installs the apport package
which disabled core dumps from user applications. (google it up!).
It is not meant to do this and documentation claims that
it is not installed and not enabled by default. Oh, well...

<pre>
apt purge apport
apt autoremove ### will remove apport-symptoms and a few other packages
</pre>

After this, core dumps are written to file "core" in the current directory.
See /proc/sys/kernel/core_pattern and /proc/sys/kernel/core_uses_pid.

Enable core dump file names to include process id, add following to /etc/rc.local

<pre>
echo "echo 1 > /proc/sys/kernel/core_uses_pid" >> /etc/rc.local
</pre>

= Enable debugger =

By default, Ubuntu LTS 20.04 does not permit debugger to attach and debug
already running programs. To enable it, add following to /etc/rc.local

<pre>
echo "echo 0 > /proc/sys/kernel/yama/ptrace_scope" >> /etc/rc.local
</pre>

= Disable Ubuntu Pro nag =

best I can tell, impossible at this time.

== do not do this ==

!!! does nothing !!!

<pre>
pro config set apt_news=false
</pre>

== do not do this ==

!!! breaks automatic updates because 20apt-esm-hook.conf is missing !!!

If "apt upgrade" requests Ubuntu Pro or esm-apps, disable the nag:
<pre>
/bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf
</pre>

== do not do this ==

!!! likely same as above, breaks automatic updates !!!

* comment out /etc/apt/apt.conf.d/20apt-esm-hook.conf

== do not do this ==

!!! removes too many packages !!!

<pre>
apt remove ubuntu-pro-client
</pre>

= Update packages =

<pre>
apt update # update package list
apt upgrade # install updated packages and update "kept back" packages
apt autoremove # remove packages that apt thinks should be removed
</pre>

= Remove obsolete packages =

NOTE: do this only if booted into the latest linux kernel - compare "uname -a" with "ls -l /boot"

<pre>
apt list '~o'
apt purge '~o'
</pre>

= Cleanup residual configs =

<pre>
apt list '~c'
apt purge '~c'
</pre>

= Install firefox-esr =

install firefox-esr here

= Finish installation =

Congratulations. There is nothing more to do!

* reboot
<pre>
shutdown -r now
</pre>

= Update to new version of Ubuntu =

* run "do-release-upgrade -c"
* if it does not report new release Ubuntu 24, check /etc/update-manager/release-upgrades has "Prompt=lts"

== Update Ubuntu LTS 20.04 to LTS 22.04 ==

<pre>
apt remove zsys
</pre>

=== daqubuntu ===

<pre>
# reboot to clear out all updates
# vi /etc/update-manager/release-upgrades # set "Prompt=normal"
# do-release-upgrade -c
Checking for a new Ubuntu release
New release '22.04 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
# do-release-upgrade
...
say yes...
...
login.defs, say "Y" (erase local changes, use packaged version)
/etc/systemd/resolved.conf, say "Y" (same as above)
firefox snap, say yes
unable to reach snap store, say "skip"
/etc/gmond.conf, say "Y"
/var/yp/Makefile, say "install the package maintainer's version"
/etc/ypserv.conf, same thing
/etc/ypserv.securenets, same thing
/etc/default/nis, same thing
/etc/speech-dispatcher/modules/mary-generic.conf, same thing
/etc/apt/apt.conf.d/50unattended-upgrades, same thing
...
278 packages are going to be removed, say yes
...
restart required, say yes
...
no ping... yes ping...
...
ssh daqubuntu, ok
apt update, fail, DNS does not work, "host security.ubuntu.com" does not resolve.
fix resolver per https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Disable_NetworkManager
apt update, apt upgrade now works, 0 packages to update
NIS does not work.
</pre>

=== midm9a ===

<pre>
login.defs
firefox snap
gmond.conf
ypserv
/etc/default/nis
unattended-upgrades
amanda-security.conf
remove obsolete (no)
reboot
configure dns
reenable nis
</pre>

=== daq17 ===

<pre>
firefox snap
imagemagick policy.xml
gmond.conf
chrony.conf
/var/yp/Makefile
ypserv.conf
ypserv.securenets
/etc/default/nis
50unattended-upgrades
</pre>

=== daq00 ===

per https://serverpilot.io/docs/how-to-upgrade-ubuntu-20.04-to-22.04/

<pre>
do-release-upgrade -f DistUpgradeViewNonInteractive
</pre>

if it exists "too soon" without doing anything, run it without "-f xxx", most likely it does not like something about this machine. in case of daq00 it did not like how the EFI partitions were mounted. after fixing it, non-interactive upgrade was successful.

=== isdaq08 ===

* prepare
<pre>
cd ~/git/scripts
git pull
cd ~
apt -y install debsums
</pre>
* check for modified config files that make upgrade unhappy, deal with all files reported by debsums.
<pre>
root@isdaq08:~# debsums -ce
/etc/ganglia/gmond.conf
/etc/yp.conf
/etc/apt/apt.conf.d/10periodic
root@isdaq08:~#
</pre>
* restore original /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
</pre>
* apt remove ganglia-monitor
* apt remove nis
* "debsums -ce" is now empty

Run the upgrade:

* do-release-upgrade -f DistUpgradeViewNonInteractive

Post upgrade:

* configure DNS
* apt -y install linux-generic-hwe-22.04
* /bin/cp -v ~/git/scripts/etc/99apt-conf-ko /etc/apt/apt.conf.d/ # restore nightly updates
* /bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf # remove the ubuntu-pro nag
* install missing packages
* restore ganglia
* restore nis
* check zpool status, may need zpool upgrade
* reboot

== upgrade U-22 to U-24 ==

generic instructions. below are notes from upgrades of specific machines.

NOTE: at CERN saw installation getting stuck on restart of autofs (automount, rsyslogd 100% CPU, huge /var/log/syslog), stop autofs before upgrade?
<pre>
systemctl stop autofs
</pre>

<pre>
cleanup zsys
</pre>

<pre>
maybe: if snap is already removed, remove firefox to prevent snap from reinstalling.
install non-snap firefox-esr
</pre>

<pre>
remove thunderbird to prevent installation of snap thunderbird in U-24
</pre>

<pre>
mount EFI partition as /boot/efi, otherwise upgrade bombs
usually mount /dev/sda1 /boot/efi
</pre>

<pre>
debsums -ce
apt -y remove desktop-base # causes installer crash
apt -y remove thunderbird # avoid forced conversion to snap
apt update
apt -y upgrade
apt -y autoremove
do-release-upgrade -c # confirm upgrade will be to U-24
do-release-upgrade
# say "y" to all questions
# after installation starts, accept default answers to all questions
# should run for about 1 hour or so
# after upgrade finishes
apt update
apt -y upgrade
apt -y autoremove
shutdown -r now
</pre>

post upgrade:

* check zpool status, may need zpool upgrade
* cd /etc/apt/sources.list.d, reenable 3rd party repos (mozilla, google, etc)

if installation bombs:

<pre>
apt update
apt upgrade # will tell us to run "apt --fix-broken install", do it
apt --fix-broken install
apt update
apt upgrade # should resume the upgrade, will run for a long time
apt update
apt upgrade # should do nothing
apt autoremove
shutdown -r now
</pre>

=== daqubuntu, U-24 ===

* prepare
<pre>
cd ~/git/scripts
git pull
cd ~
apt -y install debsums
</pre>
* check for modified config files that make upgrade unhappy, deal with all files reported by debsums.
<pre>
root@daqubuntu:~# debsums -ce
/etc/ganglia/gmond.conf
debsums: missing file /etc/init.d/nis (from nis package)
/etc/default/nis
/etc/ypserv.conf
/etc/ypserv.securenets
/var/yp/Makefile
/etc/update-manager/release-upgrades
/etc/apt/apt.conf.d/10periodic
/etc/yp.conf
root@daqubuntu:~#
* restore original /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
</pre>
* apt remove ganglia-monitor
* apt remove nis
* apt autoremove
* restore original release-upgrades: "Prompt: lts"
* "debsums -ce" is now empty

Check for upgrade:

<pre>
root@daqubuntu:~# do-release-upgrade -c
Checking for a new Ubuntu release
There is no development version of an LTS available.
To upgrade to the latest non-LTS development release
set Prompt=normal in /etc/update-manager/release-upgrades.
root@daqubuntu:~#
</pre>

Run the upgrade:

* do-release-upgrade -f DistUpgradeViewNonInteractive

Post upgrade:

* configure DNS
* apt -y install linux-generic-hwe-22.04
* /bin/cp -v ~/git/scripts/etc/99apt-conf-ko /etc/apt/apt.conf.d/ # restore nightly updates
* /bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf # remove the ubuntu-pro nag
* install missing packages
* restore ganglia
* restore nis
* check zpool status, may need zpool upgrade
* cd /etc/apt/sources.list.d, reenable 3rd party repos (mozilla, google, etc)
* reboot

=== daq14, U-20-22-24 ===

* apt update, apt upgrade
* apt -y install linux-image-generic-hwe-20.04 linux-tools-virtual-hwe-20.04 ### install kernel 5.15
* shutdown -r now
* stuck waiting for daq14 to shutdown...
* reboot into kernel 5.15
* ???
<pre>
cd ~/git/scripts
git pull
cd ~
apt -y install debsums
</pre>
* debsums -ce
<pre>
/etc/apache2/ports.conf
/etc/dnsmasq.conf
/etc/ganglia/gmond.conf
/etc/yp.conf
/etc/sudoers
</pre>
* apache2 restore original ports.conf, uncomment "Listen 80"
* cp -pv /etc/dnsmasq.conf.dpkg-dist /etc/dnsmasq.conf
* apt remove ganglia-monitor
* edit /etc/yp.conf, remove everything after "# ypserver ypserver.network.com"
* "debsums -ce" is now empty
* do-release-upgrade -f DistUpgradeViewNonInteractive
* runs for a long time
* stuck on "/etc/default/nis", type "Y", press enter, nothing for a bit, then resumes running
* finished
* configure DNS
* reboot
* have kernel 6.8
* apt update; apt upgrade
* apt upgrade guile-2.2-libs ### would not auto-update, "kept back", has to be done by hand
* apt autoremove
* debsums -ce
<pre>
debsums: missing file /etc/init.d/nis (from nis package)
/etc/default/nis
</pre>
* diff /etc/default/nis.dpkg-dist /etc/default/nis
* cp -pv /etc/default/nis.dpkg-dist /etc/default/nis
* debsums -ce
<pre>
debsums: missing file /etc/init.d/nis (from nis package)
</pre>
* we ignore this and run the update
* do-release-upgrade -c
<pre>
Checking for a new Ubuntu release
New release '24.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
</pre>
* do-release-upgrade -f DistUpgradeViewNonInteractive
* bombs out without any error messages
* in /var/log/dist-upgrade/main.log reports "Failed to find a replacement for xapp" and other packages
* apt remove xapp usrmerge ureadahead thunderbird-gnome-support
* no go, complains about even more packages.
* apt list | grep installed | grep -v jammy ### show packages installed from non-ubuntu sources
* remove all packages marked "install,local" ### ubuntu updater does not know where they came from and so cannot update them.
* apt remove desktop-base ### not happy about this package in /var/log/dist-upgrade/apt.log
* apt autoremove
* do-release-upgrade -f DistUpgradeViewNonInteractive
* running for a long time...

=== alpha04 U-20-24 ===

* apt update, apt upgrade, apt autoremove
* reboot into latest kernel (already done)
* debsums -ce
<pre>
root@alpha04:~# debsums -ce
/etc/dnsmasq.conf
/etc/ganglia/gmond.conf
/etc/default/nis
/etc/yp.conf
root@alpha04:~#
</pre>
* move /etc/dnsmasq.conf to /etc/dnsmasq.d/alpha04.conf
* apt remove dnsmasq
* apt remove ganglia-monitor
* apt remove nis
* apt autoremove
* debsums -ce ### is now empty
* do-release-upgrade -f DistUpgradeViewNonInteractive
* it runs for a long time...
* complained about /etc/fwupd config files, not sure why...
* finished
* apt update, apt upgrade, apt autoremove
* restore dnsmasq: apt install dnsmasq, systemctl status dnsmasq
* restore ganglia, per instructions
* restore NIS: apt -y install rpcbind nis, ypwhich, ypwhich -m
* zpool upgrade rpool ### also upgrade any other zfs pools, see zpool status
* remove unwanted packages, per instructions
* run gonodeinfo
* reboot
* done

== vera00 U20-22-24 ==

* everything same as daq14 for U20-22
* kernel is still 5.15
* U22-24 is going...
* stuck for a few minutes on /etc/fwupd config files
* have kernel 6.8.0-49
* same steps as daq14
* reboot
* same steps as daq14
* done

== phaarmonster U22-24 ==

* debsums -ce
<pre>
/etc/amandahosts
/etc/apache2/ports.conf
/etc/ganglia/gmond.conf
</pre>
* do-release-upgrade -c ### reports U24.04.1
* do-release-upgrade -f DistUpgradeViewNonInteractive ### bombs
* apt remove desktop-base; apt autoremove
* do-release-upgrade ### (interactive) runs ok
* bombed !!!
* apt upgrade spews errors and tells us to run "apt --fix-broken install"
* apt --fix-broken install ### runs
* bombs with thunderbird snap errors
* again... no go
* thunderbird snap complains about mounting /home, but /home is a symlink
* rm /home, mkdir /home
* again, runs ok
* asks about /etc/fwupd/fwupd.conf - say "Y" to install updated package version
* apt install completes
* apt update; apt upgrade ### running for a long time...
* finished
* install missing packages, etc
* reboot
* long wait... came back
* DNS does now work, systemd-resolved missing, apt install systemd-resolved, "configure DNS"
* done.

== isdaq10 U22-24 ==

* do-release-upgrade -f DistUpgradeViewNonInteractive
* bombed on desktop-base, "apt remove desktop-base; apt autoremove"
* bombed with errors
* running "apt --fix-broken install"
* complained about thunderbird snap
* complained about /etc/fwupd/fwupd.conf (say Y)
* finished ok
* apt update
* apt upgrade ### reports "1382 upgraded, 140 newly installed, 0 to remove and 29 not upgraded"

== iris01 ==

* debsums -ce
<pre>
/etc/ganglia/gmond.conf
/etc/default/nis
/etc/yp.conf
</pre>
* apt remove desktop-base
* apt remove thunderbird
* apt autoremove
* do-release-upgrade -f DistUpgradeViewNonInteractive
* bombed with dpkg errors:
<pre>
hplip wants wrong python
ganglia-monitor wants wrong libapr1
sssd-ad wants bunch of wrong libraries
xemacs21-mule
libnfsidmap1 wants libldap2
adn so forth...
</pre>
* apt --fix-broken install -y
* bombs on ganglia - ganglia group absent from /etc/groups
* fix group ganglia by hand, remove ganglia group from NIS on isdaq00
* apt --fix-broken install
* apt upgrade

== iris00 ==

* stuck on "autofs: restarting", login as root, kill iris midas, kill automount, systemctl restart autofs, noble got unstuck, ctrl-c systemctl restart autofs
* noble running...
* bombed with dpkg errors
* check ganglia user, group - both ok
* apt --fix-broken install
* apt upgrade

== tigstore01 ==

* no bomb-out

== midm9b ==

* apt remove desktop-base thunderbird
* bombed
* apt --fix-broken-install
* apt upgrade

= Upgrade to new version of Debian =

https://www.debian.org/releases/bookworm/amd64/release-notes/ch-upgrading.en.html

== 32-bit VME processor Debian 11 to 12 to 13 ==

* cd git/scripts; git pull; cd ~
* apt update
* apt upgrade
* edit /etc/apt/sources.list
<pre>
deb http://deb.debian.org/debian/ trixie main
#deb http://deb.debian.org/debian/ bookworm main
#deb http://deb.debian.org/debian/ bullseye main
#deb http://deb.debian.org/debian/ buster main
#deb-src http://deb.debian.org/debian/ bullseye main
</pre>
* apt update
* apt upgrade --without-new-pkgs
* apt full-upgrade
* apt list '~c'; apt purge '~c' # purge left-over config files [residual-config]
* reboot

= Ubuntu package manager =

* apt-get install xxx # install package xxx
* apt-get update
* apt-get upgrade
* apt-get dist-upgrade
* apt-get autoremove # remove automatically installed packages required by a removed package
* apt-get remove xxx # remove package xxx
* apt-cache search . # list all available packages
* apt-cache show "." | grep ^Package # list al available packages
* apt-cache madison root-system # show all available versions of package root-system
* apt list # list all installed packages
* dpkg --listfiles libpng16-16 # list all files from this package
* apt list --installed # list all installed packages
* dpkg -S /bin/bash # what package provides this file?
* dpkg -L bash # what files provided by this package?
* debsums -ce # show modified config files
* apt-config dump # show apt configuration
* apt purge '~c' # purge all [residual-config] packages
* ls -l /var/lib/dpkg/info/ # show post-install scripts

= Ubuntu zsys =

NOTE: DO NOT USE ZSYS, see https://github.com/ubuntu/zsys/issues/218 and https://github.com/ubuntu/zsys/issues/230

* scripted removal of old snapshots (replace "echo zfs destroy" with "zfs destroy")
<pre>
zfs list -t all | cut -f1 -d " " | grep autozsys | xargs -n1 echo zfs destroy
</pre>

* manual removal of old snapshots
<pre>
zsysctl show
zsysctl state remove xy69ye -s
zsysctl state remove xy69ye
zsysctl state remove xy69ye -u wheel
</pre>
* apt remove zsys

NOTE: old zsys snapshots must be cleaned manually, "zsysctl state remove xxx --system" is broken and does not remove user data snapshots

* manages system snapshots
* documentation: https://github.com/ubuntu/zsys
* documentation: (go to next article via link "newer" at the bottom) https://didrocks.fr/2020/05/21/zfs-focus-on-ubuntu-20.04-lts-whats-new/
* ubuntu 20.04 bug, too many snapshots cause /boot to become full and updates fail. https://github.com/ubuntu/zsys/issues/155
* solution: use custom /etc/zsys.conf, limit number of snapshots to 10, see trinatdaq:/etc/zsys.conf
* zsys commands:
<pre>
update-grub # list of all snapshots, errors if some snapshots are broken
zsysctl state remove lnc0k7 --system # remove snapshot
xemacs -nw /etc/zsys.conf; zsysctl service reload; zsysctl service gc # cause gc to run with new settings in zsys.conf
zfs list -r -t snapshot -o name,used,referenced,creation bpool/BOOT # list snapshots
zsysctl show # show snapshots
</pre>

= Ubuntu cloning =

to clone a ubuntu image:

<pre>
cd /nfsroot/lxcpet
emacs -nw etc/hostname ### change hostname
emacs -nw etc/mailname ### change hostname (debian 11)
emacs -nw etc/defaultdomain ### change the NIS domainname
emacs -nw etc/yp.conf ### change the NIS server
cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
emacs -nw root/.ssh/authorized_keys ### update root ssh keys
</pre>

= Ubuntu boot loader =

== maintenance commands ==

* update-initramfs -v -u
* grub-install /dev/sda

= Convert from single to dual mirrored ZFS SSD =

Assuming Ubuntu LTS 22.04 with "instal on ZFS" option, we will
add a second SSD, configure ZFS to use both SSDs in mirrored
configuration and setup grub to boot from either SSD. This
is intended to create a full redundant system where failure
of either SSD does not break the system.

== partition ==

* identify first SSD
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* connect second SSD of identical size
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
/dev/sdb WD Blue SA510 2.5 250GB 22243Z803852 25 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* if second SSD is not autodetected, reboot
* Clone partition table automatically
If both SSDs are identical size, use this simpler method of duplicating the partition table:
<pre>
root@midm9b:~# sfdisk -d /dev/sda > part_table
root@midm9b:~# grep -v ^label-id part_table | sed -e 's/, *uuid=[0-9A-F-]*//' | sfdisk /dev/sdb
</pre>
The grep and sed in the second command are there to prevent disk ID and partition IDs from being cloned. Alternatively the part_table file can be edited manually to remove the label-id line and the uuid entries from the individual partitions.

* Clone partition table manually (e.g. for different size disks)
* list partition table of first SSD:
<pre>
root@midm9b:~# fdisk -l /dev/sda
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>
* create identical partitions on second SSD, use sector numbers from above.
<pre>
root@midm9b:~# gdisk /dev/sdb
GPT fdisk (gdisk) version 1.0.8

Partition table scan:
MBR: not present
BSD: not present
APM: not present
GPT: not present

Creating new GPT entries in memory.

Command (? for help): n
Partition number (1-128, default 1):
First sector (34-488397134, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-488397134, default = 488397134) or {+-}size{KMGTP}: 1050623
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): ef00
Changed type of partition to 'EFI system partition'

Command (? for help): n
Partition number (2-128, default 2):
First sector (34-488397134, default = 1050624) or {+-}size{KMGTP}:
Last sector (1050624-488397134, default = 488397134) or {+-}size{KMGTP}: 5244927
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 8200
Changed type of partition to 'Linux swap'

Command (? for help): n
Partition number (3-128, default 3):
First sector (34-488397134, default = 5244928) or {+-}size{KMGTP}:
Last sector (5244928-488397134, default = 488397134) or {+-}size{KMGTP}: 9439231
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): be00
Changed type of partition to 'Solaris boot'

Command (? for help): n
Partition number (4-128, default 4):
First sector (34-488397134, default = 9439232) or {+-}size{KMGTP}:
Last sector (9439232-488397134, default = 488397134) or {+-}size{KMGTP}:
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): bf00
Changed type of partition to 'Solaris root'

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sdb.
The operation has completed successfully.
root@midm9b:~# fdisk -l /dev/sda /dev/sdb
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root

Disk /dev/sdb: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: EB251739-30C6-422F-A505-5887B5A0B603

Device Start End Sectors Size Type
/dev/sdb1 2048 1050623 1048576 512M EFI System
/dev/sdb2 1050624 5244927 4194304 2G Linux swap
/dev/sdb3 5244928 9439231 4194304 2G Solaris boot
/dev/sdb4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>

== update ZFS pools ==

* identify second SSD partitions
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part2
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part2 -> ../../sdb2
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
</pre>
* convert bpool from single disk to mirrored disk:
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~# zpool attach bpool 99e03dc0-7d4d-f24b-8fa1-f042b9f135db /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3
root@midm9b:~# zpool status bpool
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors
</pre>
* convert rpool
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
root@midm9b:~# zpool attach rpool f6fd54f8-3af7-b943-ae3d-a4e480537fb9 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4
root@midm9b:~# zpool status rpool
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Jan 20 19:40:45 2023
5.83G scanned at 664M/s, 2.92M issued at 332K/s, 9.11G total
0B resilvered, 0.03% done, no estimated completion time
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~#
</pre>
* wait for resilver to complete
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
scan: resilvered 9.65G in 00:00:36 with 0 errors on Fri Jan 20 19:41:21 2023
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
</pre>

== update boot loader ==

INSTALL SYSLINUX: https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#EFI_boot_using_syslinux

DO *NOT* DO THE FOLOWING:

* create and mount EFI partitions:
<pre>
root@midm9b:~# mkfs.msdos /dev/sdb1
root@midm9b:~# mkdir /boot/efi-sda
root@midm9b:~# mkdir /boot/efi-sdb
root@midm20c:~# blkid | grep vfat ### identify UUID
/dev/sdb1: UUID="DD89-5081" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="d0cb6be4-2f67-5b42-9b26-9e6905e9f774"
/dev/sdc1: UUID="D970-86BA" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="e6d3b5b9-a512-44a2-9205-1a4db06ed2a2"
/dev/sda1: UUID="DDA1-044C" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="6dc9dff0-1c13-8045-a906-7803d3074c70"
root@midm20c:~# cat /etc/fstab | grep vfat ### add mount points with correct UUID
#UUID=D970-86BA /boot/efi vfat umask=0022,fmask=0022,dmask=0022 0 1
UUID=DDA1-044C /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022 0 1
UUID=DD89-5081 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022 0 1
root@midm9b:~# mount -a
root@midm9b:~# df -kl
Filesystem 1K-blocks Used Available Use% Mounted on
...
/dev/sda1 523244 13720 509524 3% /boot/efi
/dev/sdb1 523244 4 523240 1% /boot/efi-sdb
...
root@midm9b:~# rsync -av /boot/efi/ /boot/efi-sdb/
sending incremental file list
EFI/
...
root@midm9b:~# ls -l /boot/efi-sda
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~# ls -l /boot/efi-sdb
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~#
</pre>
* add systemd "nofail" flag to /etc/fstab, without this, systemd will stop booting if one SSD is missing
<pre>
daq00:~$ cat /etc/fstab | grep vfat
#UUID=31A7-24BE /boot/efi vfat umask=0022,fmask=0022,dmask=0022 0 1
/dev/sda1 /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
/dev/sdb1 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
</pre>
* setup script to update grub on second SSD, it must be run manually after every kernel update
<pre>
root@midm9b:~# ln -s ~/git/scripts/etc/update_efi_grub.perl ~/
root@midm9b:~# ~/update_efi_grub.perl -u
EFI dir: /boot/efi-sda
/boot/efi-sda: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sda/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sda: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sda/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
EFI dir: /boot/efi-sdb
/boot/efi-sdb: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sdb/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sdb: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sdb/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
root@midm9b:~#
</pre>

= Disable NetworkManager =

== Debian-12 ==

* use netplan, https://www.debian.org/doc/manuals/debian-reference/ch05.en.html#_the_modern_network_configuration_for_cloud
* apt install netplan.io
* systemctl enable systemd-networkd
* systemctl restart systemd-networkd
* create /etc/netplan/50-dhcp.yaml
<pre>
network:
version: 2
ethernets:
all-en:
match:
name: "en*"
dhcp4: true
dhcp6: true
ignore-carrier: true ### do not drop IP address if network link drops
</pre>
* netplan apply
* netplan try
* ifconfig -a ### to check IP address settings
* netstat -rn ### to check default route
* cat /etc/resolv.conf ### to check DNS
* ls -l /run/systemd/netif/leases ### systemd-networkd dhcp leases
* NOTE: without "ignore-carrier" it will drop the IP address if network link drops, re-do dhcp when links comes back
* NOTE: wait-network-online will wait for all interfaces to get an IP address

== Ubuntu-20 ==

NOTE: THIS IS BROKEN IN UBUNTU LTS 22.04

NetworkManager is useful for configuring dynamic
network interfaces, i.e. laptops that often move
between networks, or connect to multiple choice
of wifi networks, etc.

For machines with statically configured network interfaces,
NetworkManager is not necessary.

As it has been observed to become confused and observed
to malfunction when network links go up and down (it keeps
unnecessarily reconfiguring the ip address, etc), it can
be usefuil to disable it.

* list all network interfaces
<pre>
# /bin/ls -1 /sys/class/net/
enp0s31f6
lo
</pre>
* edit /etc/network/interfaces:
<pre>
rename enp0s31f6=eth0
auto eth0
iface eth0 inet static
address 142.90.120.94/19
gateway 142.90.100.18
</pre>
* statically configure systemd-resolved
** create /etc/systemd/resolved.conf.d/resolved.conf with this contents:
<pre>
[Resolve]
DNS=142.90.100.19
Domains=triumf.ca
</pre>
** systemctl restart systemd-resolved
** resolvectl
** systemd-analyze cat-config systemd/resolved.conf
* disable NetworkManager
<pre>
systemctl disable NetworkManager
</pre>
* reboot

== U-22, U-24 ifup-ko ==

Network configuration of modern linux is confused. There are at least 3 configuration methods, each with different shortcomings:
* the old ifup method is barely documented
* NetworkManager is well documented and tooled, but sometimes does strange things
* systemd-networkd is mysterious, and likely to do strange stuff, like all systemd stuff
* netplan is the latest method, configuration is simple but uses NetworkManager or systemd-networkd as backend.

This is a solution for a specific situation of fixed computer with one fixed wired interface and maybe one or more additional interfaces for fixed wired private networks.

Install /etc/ifup-ko, edit it with IP addresses of main and additional interfaces, let systemd run it in the right place in the boot sequence replacing NetworkManager and NetworkManager-wait-online .

As bonus, /etc/ifup-ko waits up to 10 seconds for the main interface link to come up. If this is not needed, comment it out from the script.

* prepare
<pre>
cd ~/git/scripts
git pull
cd ifup-ko
make install
</pre>
* confirm interface names
<pre>
systemctl start ifup-ko ### should finish immediately or after 10 seconds
systemctl status ifup-ko -n 1000 ### observe list of interfaces is correct, name of main interface is correct
</pre>
* edit /etc/ifup-ko
** add host IP address to the "ifconfig" line
** add gateway IP address to the "ip route add" line
* test
<pre>
systemctl start ifup-ko ### should finish immediately
systemctl status ifup-ko -n 1000 ### observe everything is configured as expected
</pre>
* cut-over
<pre>
systemctl disable networkd-dispatcher
systemctl disable NetworkManager
systemctl disable wpa_supplicant # if no Wifi or Wifi not in use
</pre>
* reboot

= Disable systemd-networkd =

On netbooted machines, systemd-networkd should be disabled - when "apt upgrade" runs
and needs to update and configure systemd, it will stop systemd-networkd, which
will stop the network, which will stop the NFS mounted root filesystem, which will stop
the machine.

<pre>
systemctl disable systemd-networkd.service
systemctl disable systemd-networkd.socket
systemctl mask systemd-networkd.service
systemctl mask systemd-networkd.socket
</pre>

= Configure ECC memory =

== Configure EDAC ==

* apt install edac-utils rasdaemon

=== Intel i3-2120 ===
<pre>
root@musr00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X9SCL/X9SCM
root@musr00:~# edac-ctl --status
edac-ctl: drivers not loaded.
</pre>

=== Intel E-2236 ===
<pre>
root@daq00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SCM-F
root@daq00:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq00:~# edac-util
edac-util: No errors to report.
root@daq00:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
</pre>
* check edac sysfs files (Intel)
<pre>
root@daq00:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_noinfo_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 max_location
-r--r--r-- 1 root root 4096 Jan 25 15:10 mc_name
drwxr-xr-x 2 root root 0 Jan 25 15:10 power
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank0
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank1
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank2
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank3
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank4
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank5
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank6
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank7
--w------- 1 root root 4096 Jan 25 15:10 reset_counters
-r--r--r-- 1 root root 4096 Jan 25 15:10 seconds_since_reset
-r--r--r-- 1 root root 4096 Jan 25 15:10 size_mb
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Jan 25 15:10 uevent
root@daq00:~#
</pre>

=== Intel E3-1270 v6 ===
<pre>
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --status
edac-ctl: drivers are loaded.
root@grsnis01:~# edac-util
edac-util: No errors to report.
root@grsnis01:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@grsnis01:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 max_location
-r--r--r-- 1 root root 4096 Feb 19 12:35 mc_name
drwxr-xr-x 2 root root 0 Feb 19 12:35 power
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank0
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank1
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank2
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank3
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank4
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank5
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank6
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank7
--w------- 1 root root 4096 Feb 19 12:35 reset_counters
-r--r--r-- 1 root root 4096 Feb 19 12:35 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 19 12:35 size_mb
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 19 12:35 uevent
root@grsnis01:~#
</pre>

=== Intel E3-1245 v6 ===

<pre>
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --status
edac-ctl: drivers are loaded.
[root@alphagdaq ~]# edac-util
edac-util: No errors to report.
[root@alphagdaq ~]# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
[root@alphagdaq ~]# ras-mc-ctl --layout
+-----------------------------------------------+
| mc0 |
| csrow0 | csrow1 | csrow2 | csrow3 |
----------+-----------------------------------------------+
channel1: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
----------+-----------------------------------------------+
[root@alphagdaq ~]# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#3channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#3channel#1 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#2channel#0 0 0
[root@alphagdaq ~]# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SSH-F
[root@alphagdaq ~]# ras-mc-ctl --summary
DBD::SQLite::db prepare failed: no such table: mc_event at /usr/sbin/ras-mc-ctl line 1129.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1130.
[root@alphagdaq ~]#
</pre>

=== AMD 3700X ===

(memory is non-ECC)

<pre>
root@daq13:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@daq13:~#
root@daq13:~#
root@daq13:~# edac-ctl --status
edac-ctl: drivers not loaded.
root@daq13:~# edac-util
edac-util: Error: No memory controller data found.
root@daq13:~# edac-util -s
edac-util: EDAC drivers loaded. No memory controllers found
root@daq13:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 2 root root 0 Jan 25 15:26 power
lrwxrwxrwx 1 root root 0 Jan 21 16:16 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Jan 21 16:16 uevent
</pre>

(memory is ECC)

<pre>
root@trinatdaq:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@trinatdaq:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@trinatdaq:~# edac-util
edac-util: No errors to report.
root@trinatdaq:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Dec 15 13:04 mc0
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
lrwxrwxrwx 1 root root 0 Dec 13 18:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dec 13 18:31 uevent
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_noinfo_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 max_location
-r--r--r-- 1 root root 4096 Dec 15 13:04 mc_name
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank4
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank5
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank6
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank7
--w------- 1 root root 4096 Dec 15 13:04 reset_counters
-rw-r--r-- 1 root root 4096 Dec 15 13:04 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Dec 15 13:04 seconds_since_reset
-r--r--r-- 1 root root 4096 Dec 15 13:04 size_mb
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Dec 15 13:04 uevent
root@trinatdaq:~#
</pre>

=== AMD 5000G ===

* no linux driver for AMD 5000-series "G" CPU
* no mention of ECC in the BIOS settings
* unclear status of ECC support in AMD documentation (sais only "pro" "G" CPUs have ECC)
* unclear status of ECC support in ASUS documentation (web page out of date)

=== AMD 5600X ===

<pre>
root@daq17:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-XE GAMING WIFI
root@daq17:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq17:~# edac-util
edac-util: No errors to report.
root@daq17:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@daq17:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Aug 19 19:27 mc0
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
lrwxrwxrwx 1 root root 0 May 10 10:11 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 May 10 10:11 uevent
root@daq17:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_noinfo_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 max_location
-r--r--r-- 1 root root 4096 Aug 19 19:27 mc_name
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank4
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank5
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank6
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank7
--w------- 1 root root 4096 Aug 19 19:27 reset_counters
-rw-r--r-- 1 root root 4096 Aug 19 19:27 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Aug 19 19:27 seconds_since_reset
-r--r--r-- 1 root root 4096 Aug 19 19:27 size_mb
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Aug 19 19:27 uevent
root@daq17:~#
</pre>

=== AMD 3955WX ===

<pre>
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --status
edac-ctl: drivers are loaded.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util
edac-util: No errors to report.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@alphasuperdaq:~/git/scripts/quotareport# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 19 root root 0 Dez 12 04:48 mc0
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
lrwxrwxrwx 1 root root 0 Dez 9 05:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dez 9 05:31 uevent
root@alphasuperdaq:~/git/scripts/quotareport#
root@alphasuperdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 max_location
-r--r--r-- 1 root root 4096 Feb 28 22:19 mc_name
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank0
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank1
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank10
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank11
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank12
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank13
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank14
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank15
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank2
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank3
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank4
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank5
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank6
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank7
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank8
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank9
--w------- 1 root root 4096 Feb 28 22:19 reset_counters
-rw-r--r-- 1 root root 4096 Feb 28 22:19 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Feb 28 22:19 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 28 22:19 size_mb
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 28 22:19 uevent
root@alphasuperdaq:~#
root@alphasuperdaq:~# ras-mc-ctl --layout
Use of uninitialized value $max_pos[3] in modulus (%) at /usr/sbin/ras-mc-ctl line 868.
Use of uninitialized value $d in numeric ge (>=) at /usr/sbin/ras-mc-ctl line 869.
Use of uninitialized value $d in sprintf at /usr/sbin/ras-mc-ctl line 872.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| mc0 |
| csrow0 | csrow1 |
| channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 | channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

0: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
root@alphasuperdaq:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#2 0 0
mc#0csrow#1channel#7 0 0
mc#0csrow#0channel#3 0 0
mc#0csrow#1channel#4 0 0
mc#0csrow#1channel#2 0 0
mc#0csrow#0channel#7 0 0
mc#0csrow#1channel#3 0 0
mc#0csrow#0channel#4 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#5 0 0
mc#0csrow#0channel#6 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#0channel#5 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#6 0 0
root@alphasuperdaq:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: ASUSTeK COMPUTER INC. model Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@alphasuperdaq:~#
</pre>

=== AMD 7700X ===

<pre>
root@dsfe05:~# apt install edac-utils
root@dsfe05:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro H13SAE-MF
root@dsfe05:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@dsfe05:~# edac-util
edac-util: No errors to report.
root@dsfe05:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@dsfe05:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 May 14 09:33 ce_count
-r--r--r-- 1 root root 4096 May 14 09:33 ce_noinfo_count
-r--r--r-- 1 root root 4096 May 14 09:33 max_location
-r--r--r-- 1 root root 4096 May 14 09:33 mc_name
drwxr-xr-x 2 root root 0 May 14 09:33 power
drwxr-xr-x 3 root root 0 May 14 09:33 rank4
drwxr-xr-x 3 root root 0 May 14 09:33 rank5
--w------- 1 root root 4096 May 14 09:33 reset_counters
-r--r--r-- 1 root root 4096 May 14 09:33 seconds_since_reset
-r--r--r-- 1 root root 4096 May 14 09:33 size_mb
-r--r--r-- 1 root root 4096 May 14 09:33 ue_count
-r--r--r-- 1 root root 4096 May 14 09:33 ue_noinfo_count
-rw-r--r-- 1 root root 4096 May 14 09:33 uevent
root@dsfe05:~#
</pre>

== Configure rasdaemon ==

<pre>
apt install rasdaemon
</pre>
<pre>
systemctl enable rasdaemon
systemctl restart rasdaemon
systemctl status rasdaemon
</pre>

<pre>
● rasdaemon.service - RAS daemon to log the RAS events
Loaded: loaded (/lib/systemd/system/rasdaemon.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-01-25 15:16:37 PST; 3min 5s ago
Main PID: 2477175 (rasdaemon)
Tasks: 1 (limit: 76958)
Memory: 17.1M
CGroup: /system.slice/rasdaemon.service
└─2477175 /usr/sbin/rasdaemon -f -r

Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Listening to events for cpus 0 to 11
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mc_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording aer_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording extlog_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mce_record events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording arm_event events
</pre>

== Get reports ==

* Intel 2x32GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --layout
+-------------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-------------------------+
channel1: | 16384 MB | 16384 MB |
channel0: | 16384 MB | 16384 MB |
----------+-------------------------+
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#0channel#1 0 0
root@daq00:~#
</pre>

* Intel 4x16GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#1 0 0
mc#0csrow#2channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#3channel#0 0 0
mc#0csrow#3channel#1 0 0
root@daq00:~#
root@daq00:~# ras-mc-ctl --layout
+-----------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-----------------------+
channel1: | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB |
----------+-----------------------+
root@daq00:~#
root@daq00:~#
root@daq00:~#
root@daq00:~# ras-mc-ctl --print-labels
ras-mc-ctl: Error: No dimm labels for Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@daq00:~#
</pre>

note: ubuntu LTS 22.04 DBD::SQLite::db error is not there.

* AMD 7700 2x32GB DDR5 ECC DIMMs

<pre>
root@dsfe05:~# systemctl status rasdaemon
● rasdaemon.service - RAS daemon to log the RAS events
Loaded: loaded (/lib/systemd/system/rasdaemon.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2024-05-14 09:36:43 PDT; 33ms ago
Process: 4088418 ExecStartPost=/usr/sbin/rasdaemon --enable (code=exited, status=0/SUCCESS)
Main PID: 4088417 (rasdaemon)
Tasks: 1 (limit: 37300)
Memory: 788.0K
CPU: 5ms
CGroup: /system.slice/rasdaemon.service
└─4088417 /usr/sbin/rasdaemon -f -r

May 14 09:36:43 dsfe05 rasdaemon[4088417]: ras:aer_event event enabled
May 14 09:36:43 dsfe05 rasdaemon[4088417]: Enabled event ras:aer_event
May 14 09:36:43 dsfe05 rasdaemon[4088417]: mce:mce_record event enabled
May 14 09:36:43 dsfe05 rasdaemon[4088417]: Enabled event mce:mce_record
May 14 09:36:43 dsfe05 rasdaemon[4088417]: ras:extlog_mem_event event enabled
May 14 09:36:43 dsfe05 rasdaemon[4088417]: Enabled event ras:extlog_mem_event
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording mc_event events
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording aer_event events
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording extlog_event events
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording mce_record events
root@dsfe05:~# ras-mc-ctl --layout
Use of uninitialized value $max_pos[3] in modulus (%) at /usr/sbin/ras-mc-ctl line 907.
Use of uninitialized value $d in numeric ge (>=) at /usr/sbin/ras-mc-ctl line 908.
Use of uninitialized value $d in sprintf at /usr/sbin/ras-mc-ctl line 911.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
+-----------------------------------------------------------------------------------------------+
| mc0 |
| csrow0 | csrow1 | csrow2 | csrow3 |
| channel0 | channel1 | channel0 | channel1 | channel0 | channel1 | channel0 | channel1 |
----+-----------------------------------------------------------------------------------------------+

0: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB |
----+-----------------------------------------------------------------------------------------------+
root@dsfe05:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#2channel#1 0 0
mc#0csrow#2channel#0 0 0
root@dsfe05:~# ras-mc-ctl --print-labels
ras-mc-ctl: Error: No dimm labels for Supermicro model H13SAE-MF
root@dsfe05:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model H13SAE-MF
root@dsfe05:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

No MCE errors.
root@dsfe05:~#
</pre>

= sensors =

== VME CPU V7865 ==

add to /etc/rc.local

<pre>
modprobe coretemp
modprobe lm75
modprobe lm90
modprobe max1668
</pre>

available sensors:

<pre>
root@lxdaq23:~# sensors
max6657-i2c-0-4c
Adapter: SMBus I801 adapter at 0400
temp1: +29.1°C (low = -55.0°C, high = +105.0°C)
(crit = +105.0°C, hyst = +95.0°C)
temp2: +31.5°C (low = -55.0°C, high = +105.0°C)
(crit = +105.0°C, hyst = +95.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Core 0: +25.0°C (crit = +100.0°C)
Core 1: +25.0°C (crit = +100.0°C)

max1805-i2c-0-18
Adapter: SMBus I801 adapter at 0400
temp1: +35.0°C (low = -55.0°C, high = +127.0°C)
temp2: +63.0°C (low = -55.0°C, high = +127.0°C)
temp3: FAULT (low = -55.0°C, high = +127.0°C) ALARM (HIGH)

lm75-i2c-0-48
Adapter: SMBus I801 adapter at 0400
temp1: +34.5°C (high = +80.0°C, hyst = +75.0°C)

root@lxdaq23:~#
</pre>

== ASUS P7P55D EVO ==

* BIOS version 2101

<pre>
root@iris01:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Core 0: +34.0°C (high = +83.0°C, crit = +99.0°C)
Core 1: +37.0°C (high = +83.0°C, crit = +99.0°C)
Core 2: +38.0°C (high = +83.0°C, crit = +99.0°C)
Core 3: +35.0°C (high = +83.0°C, crit = +99.0°C)

nouveau-pci-0100
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.05 V)
temp1: +46.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

atk0110-acpi-0
Adapter: ACPI interface
Vcore Voltage: 864.00 mV (min = +0.80 V, max = +1.60 V)
+3.3V Voltage: 3.38 V (min = +2.97 V, max = +3.63 V)
+5V Voltage: 5.04 V (min = +4.50 V, max = +5.50 V)
+12V Voltage: 12.15 V (min = +10.20 V, max = +13.80 V)
CPU Fan Speed: 968 RPM (min = 600 RPM, max = 7200 RPM)
Chassis1 Fan Speed: 1288 RPM (min = 600 RPM, max = 7200 RPM)
Chassis2 Fan Speed: 1316 RPM (min = 600 RPM, max = 7200 RPM)
Power Fan Speed: 0 RPM (min = 0 RPM, max = 7200 RPM)
CPU Temperature: +34.0°C (high = +45.0°C, crit = +45.5°C)
MB Temperature: +30.0°C (high = +45.0°C, crit = +46.0°C)

root@iris01:~#
</pre>

== ASUS Z97-WS ==

* BIOS version 2704
* load sensors drivers
<pre>
echo modprobe coretemp >> /etc/rc.local
echo modprobe nct6775 >> /etc/rc.local
</pre>
* in /boot/grub/grub.cfg, add: GRUB_CMDLINE_LINUX_DEFAULT="acpi_enforce_resources=no"
* update grub and reboot: grub-mkconfig -o /boot/grub/grub.cfg

<pre>
root@isdaq08:~# sensors
acpitz-acpi-0
Adapter: ACPI interface
temp1: +27.8°C
temp2: +29.8°C

nct6791-isa-0290
Adapter: ISA adapter
Vcore: 888.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
AVCC: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
+3.3V: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.99 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 0.00 V (min = +0.00 V, max = +0.00 V)
3VSB: 3.44 V (min = +0.00 V, max = +0.00 V) ALARM
Vbat: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 0.00 V (min = +0.00 V, max = +0.00 V)
in11: 840.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in12: 0.00 V (min = +0.00 V, max = +0.00 V)
in13: 0.00 V (min = +0.00 V, max = +0.00 V) ALARM
in14: 0.00 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1041 RPM (min = 0 RPM)
fan2: 1040 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan6: 0 RPM (min = 0 RPM)
SYSTIN: +34.0°C (high = +0.0°C, hyst = +0.0°C) ALARM sensor = thermistor
CPUTIN: +41.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: -128.0°C sensor = thermistor
AUXTIN1: -128.0°C sensor = thermistor
AUXTIN2: +35.0°C sensor = thermistor
AUXTIN3: +127.0°C sensor = thermistor
PECI Agent 0: +41.0°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
PCH_DIM0_TEMP: +0.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +42.0°C (high = +80.0°C, crit = +100.0°C)
Core 0: +42.0°C (high = +80.0°C, crit = +100.0°C)
Core 1: +40.0°C (high = +80.0°C, crit = +100.0°C)
Core 2: +39.0°C (high = +80.0°C, crit = +100.0°C)
Core 3: +39.0°C (high = +80.0°C, crit = +100.0°C)

root@isdaq08:~#
</pre>

== ASUS Z170-DELUXE ==

* BIOS version 3801
* load sensors drivers
<pre>
echo modprobe coretemp >> /etc/rc.local
echo modprobe jc42 >> /etc/rc.local
echo modprobe lm92 >> /etc/rc.local
echo modprobe nct6775 >> /etc/rc.local
</pre>
* in /etc/default/grub, add: GRUB_CMDLINE_LINUX_DEFAULT="acpi_enforce_resources=no"
* update grub and reboot: grub-mkconfig -o /boot/grub/grub.cfg

<pre>
root@iris00:~# sensors
nct6793-isa-0290
Adapter: ISA adapter
in0: 600.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 144.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in6: 0.00 V (min = +0.00 V, max = +0.00 V)
in7: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.14 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1000.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in10: 600.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 592.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 968.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1370 RPM (min = 0 RPM)
fan2: 1437 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan6: 0 RPM (min = 0 RPM)
SYSTIN: +32.0°C (high = +98.0°C, hyst = +95.0°C) sensor = thermistor
CPUTIN: +42.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: -128.0°C sensor = thermistor
AUXTIN1: +50.0°C sensor = thermistor
AUXTIN2: +22.0°C sensor = thermistor
AUXTIN3: +28.0°C sensor = thermistor
PECI Agent 0: +50.0°C (high = +98.0°C, hyst = +95.0°C)
(crit = +100.0°C)
PECI Agent 0 Calibration: +42.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
TSI2_TEMP: +3892314.0°C
TSI3_TEMP: +3892314.0°C
TSI4_TEMP: +3892314.0°C
TSI5_TEMP: +3892314.0°C
TSI6_TEMP: +3892314.0°C
TSI7_TEMP: +3892314.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

jc42-i2c-0-1a
Adapter: SMBus I801 adapter at f040
temp1: +36.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

jc42-i2c-0-18
Adapter: SMBus I801 adapter at f040
temp1: +34.8°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

jc42-i2c-0-1b
Adapter: SMBus I801 adapter at f040
temp1: +35.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

jc42-i2c-0-19
Adapter: SMBus I801 adapter at f040
temp1: +36.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +52.0°C (high = +84.0°C, crit = +100.0°C)
Core 0: +52.0°C (high = +84.0°C, crit = +100.0°C)
Core 1: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 2: +48.0°C (high = +84.0°C, crit = +100.0°C)
Core 3: +47.0°C (high = +84.0°C, crit = +100.0°C)

root@iris00:~#
</pre>

== ASUS Z390M-PRO GAMING (WI-FI) ==

* BIOS 3006
* load sensors drivers
<pre>
echo modprobe coretemp >> /etc/rc.local
echo modprobe nct6775 >> /etc/rc.local
</pre>

<pre>
root@daq18:~# sensors
nct6798-isa-0290
Adapter: ISA adapter
in0: 696.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.42 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 208.00 mV (min = +0.00 V, max = +0.00 V)
in6: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.42 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.17 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.07 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.36 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.33 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in14: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 0 RPM (min = 0 RPM)
fan2: 1131 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 1006 RPM (min = 0 RPM)
fan6: 0 RPM (min = 0 RPM)
fan7: 0 RPM (min = 0 RPM)
SYSTIN: +32.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
CPUTIN: +29.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
AUXTIN0: +25.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
AUXTIN1: +7.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
AUXTIN2: +8.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
AUXTIN3: +24.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
AUXTIN4: +83.0°C (high = +80.0°C, hyst = +75.0°C) ALARM
(crit = +100.0°C)
PECI Agent 0 Calibration: +29.0°C (high = +80.0°C, hyst = +75.0°C)
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +39.0°C (high = +82.0°C, crit = +100.0°C)
Core 0: +39.0°C (high = +82.0°C, crit = +100.0°C)
Core 1: +33.0°C (high = +82.0°C, crit = +100.0°C)
Core 2: +32.0°C (high = +82.0°C, crit = +100.0°C)
Core 3: +31.0°C (high = +82.0°C, crit = +100.0°C)
Core 4: +31.0°C (high = +82.0°C, crit = +100.0°C)
Core 5: +30.0°C (high = +82.0°C, crit = +100.0°C)

acpitz-acpi-0
Adapter: ACPI interface
temp1: +27.8°C

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: +28.0°C

root@daq18:~#
</pre>

== ASUS H110M-A/M.2 ==

* BIOS version 4202
* echo modprobe coretemp >> /etc/rc.local
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@midpol:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +33.0°C (high = +80.0°C, crit = +100.0°C)
Core 0: +33.0°C (high = +80.0°C, crit = +100.0°C)
Core 1: +30.0°C (high = +80.0°C, crit = +100.0°C)

acpitz-acpi-0
Adapter: ACPI interface
temp1: +27.8°C (crit = +119.0°C)
temp2: +29.8°C (crit = +119.0°C)

nct6793-isa-0290
Adapter: ISA adapter
in0: 368.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 152.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in6: 928.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.14 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1000.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in10: 152.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in11: 128.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in12: 136.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in13: 120.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 136.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1004 RPM (min = 0 RPM)
fan2: 1143 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan6: 0 RPM (min = 0 RPM)
SYSTIN: +118.0°C (high = +98.0°C, hyst = +95.0°C) sensor = thermistor
CPUTIN: +29.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +30.0°C sensor = thermistor
AUXTIN1: +112.0°C sensor = thermistor
AUXTIN2: +111.0°C sensor = thermistor
AUXTIN3: +110.0°C sensor = thermistor
PECI Agent 0: +31.0°C (high = +98.0°C, hyst = +95.0°C)
(crit = +100.0°C)
PECI Agent 0 Calibration: +36.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
TSI2_TEMP: +3892314.0°C
TSI3_TEMP: +3892314.0°C
TSI4_TEMP: +3892314.0°C
TSI5_TEMP: +3892314.0°C
TSI6_TEMP: +3892314.0°C
TSI7_TEMP: +3892314.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

root@midpol:~#
</pre>

== ASUS P9X79 WS ==

* https://www.asus.com/supportonly/P9X79%20WS/HelpDesk_Manual/
* BIOS version 4802
* modprobe nct6775
* modprobe coretemp

<pre>
root@daq14:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 0: +29.0°C (high = +82.0°C, crit = +100.0°C)
Core 1: +24.0°C (high = +82.0°C, crit = +100.0°C)
Core 2: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +82.0°C, crit = +100.0°C)

nouveau-pci-0200
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.00 V)
temp1: +39.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

nct6776-isa-0290
Adapter: ISA adapter
Vcore: 1.04 V (min = +0.00 V, max = +1.74 V)
in1: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
AVCC: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
+3.3V: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 2.04 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 904.00 mV (min = +0.00 V, max = +0.00 V) ALARM
3VSB: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
Vbat: 3.30 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1265 RPM (min = 0 RPM)
fan2: 1909 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +34.0°C (high = +0.0°C, hyst = +0.0°C) ALARM sensor = thermistor
CPUTIN: +58.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermal diode
AUXTIN: +31.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
PECI Agent 0: +31.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +96.0°C)
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

root@daq14:~#
</pre>

== ASUS TUF GAMING B550M-PLUS WIFI II ==

* BIOS 2803, 2806
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@midm9a:~# sensors
nct6798-isa-0290
Adapter: ISA adapter
in0: 488.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.82 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in14: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 0 RPM (min = 0 RPM)
fan2: 760 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan7: 1264 RPM (min = 0 RPM)
SYSTIN: +25.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +22.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +95.0°C sensor = thermistor
AUXTIN1: +25.0°C sensor = thermistor
AUXTIN2: +25.0°C sensor = thermistor
AUXTIN3: +25.0°C sensor = thermistor
PECI Agent 0 Calibration: +23.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +32.4°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

amdgpu-pci-0800
Adapter: PCI adapter
vddgfx: 1.45 V
vddnb: 993.00 mV
edge: +28.0°C
PPT: 20.00 W

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +33.4°C

root@midm9a:~#
</pre>

== ASUS ASUS ROG STRIX B550-XE GAMING WIFI ==

* BIOS 2423, 2604
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@daq13:~# sensors
nct6798-isa-0290
Adapter: ISA adapter
in0: 344.00 mV (min = +0.00 V, max = +1.74 V)
in1: 992.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in6: 216.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.30 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.81 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in11: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 280.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 845 RPM (min = 0 RPM)
fan2: 998 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +28.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +27.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +94.0°C sensor = thermistor
AUXTIN1: +28.0°C sensor = thermistor
AUXTIN2: +28.0°C sensor = thermistor
AUXTIN3: +97.0°C sensor = thermistor
PECI Agent 0 Calibration: +27.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +33.6°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

amdgpu-pci-0600
Adapter: PCI adapter
vddgfx: 1.45 V
vddnb: 999.00 mV
edge: +29.0°C
PPT: 14.00 W

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: +30.0°C

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +33.9°C

root@daq13:~#
</pre>

== ASUS ASUS ROG STRIX B550-E GAMING ==

* bios 2803
* echo modprobe jc42 >> /etc/rc.local
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@daq17:~# sensors
jc42-i2c-1-1b
Adapter: SMBus PIIX4 adapter port 0 at 0b00
temp1: +25.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: +28.0°C

nouveau-pci-0800
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.00 V)
temp1: +34.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

nct6798-isa-0290
Adapter: ISA adapter
in0: 288.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 224.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.31 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.79 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 280.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 843 RPM (min = 0 RPM)
fan2: 629 RPM (min = 0 RPM)
fan3: 746 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +22.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +25.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +93.0°C sensor = thermistor
AUXTIN1: +22.0°C sensor = thermistor
AUXTIN2: +22.0°C sensor = thermistor
AUXTIN3: +96.0°C sensor = thermistor
PECI Agent 0 Calibration: +25.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +27.6°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

jc42-i2c-1-1a
Adapter: SMBus PIIX4 adapter port 0 at 0b00
temp1: +23.2°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

asusec-isa-0000
Adapter: ISA adapter
CPU_Opt: 0 RPM
Chipset: +34.0°C
CPU: +25.0°C
Motherboard: +22.0°C
T_Sensor: -40.0°C
VRM: +31.0°C

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +28.0°C
Tccd1: +27.5°C

root@daq17:~#
</pre>

== ASUS PRIME B650-PLUS ==

* BIOS 1811
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@dsdaqgw:~# sensors
amdgpu-pci-0b00
Adapter: PCI adapter
vddgfx: 930.00 mV
vddnb: 1.19 V
edge: +38.0°C
PPT: 25.10 W

nct6799-isa-0290
Adapter: ISA adapter
in0: 920.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.04 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 320.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.28 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.28 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.10 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.04 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 416.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 328.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 0 RPM (min = 0 RPM)
fan2: 1253 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan7: 0 RPM (min = 0 RPM)
SYSTIN: +33.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +35.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +78.0°C sensor = thermistor
AUXTIN1: +11.0°C sensor = thermistor
AUXTIN2: +20.0°C sensor = thermistor
AUXTIN3: +82.0°C sensor = thermistor
PECI Agent 0 Calibration: +35.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +42.6°C
intrusion0: ALARM
intrusion1: OK
beep_enable: disabled

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +42.6°C
Tccd1: +36.4°C

root@dsdaqgw:~#
</pre>

= Enable CPU turbo mode =

* Intel CPU has a nominal CPU frequency (i.e. 3.4GHz) and a turbo-boost CPU frequency (i.e. 4.0GHz). Here we will enable this turbo-boost mode.
* Find out CPU capability
<pre>
root@daq01:~# lscpu | grep Hz
Model name: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
CPU MHz: 3965.803
CPU max MHz: 4000.0000
CPU min MHz: 800.0000
root@daq01:~#
</pre>
* Look up this CPU in the Intel ARK database - google for the CPU model name, i.e.
https://ark.intel.com/content/www/us/en/ark/products/88196/intel-core-i7-6700-processor-8m-cache-up-to-4-00-ghz.html
* Find current frequency settings:
<pre>
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "powersave" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 2.72 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
root@daq01:~#
</pre>
* Note the following:
** current governor is "powersave"
** "performance" governor is available
** "boost state support" is supported and active.
* Confirm CPU frequency governor:
<pre>
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
powersave
powersave
powersave
powersave
powersave
powersave
powersave
powersave
root@daq01:~#
</pre>
* Change governor to "performance":
<pre>
root@daq01:~# cpupower frequency-set --governor performance
Setting cpu: 0
Setting cpu: 1
Setting cpu: 2
Setting cpu: 3
Setting cpu: 4
Setting cpu: 5
Setting cpu: 6
Setting cpu: 7
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
performance
performance
performance
performance
performance
performance
performance
performance
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "performance" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 3.93 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
</pre>
* monitor CPU frequency:
<pre>
root@daq01:~# cpupower monitor
| Nehalem || Mperf || Idle_Stats
CPU| C3 | C6 | PC3 | PC6 || C0 | Cx | Freq || POLL | C1 | C1E | C3 | C6 | C7s | C8
0| 0.00| 0.00| 0.00| 0.00|| 88.80| 11.20| 3973|| 0.00| 0.00| 0.01| 0.02| 0.31| 0.00| 4.25
4| 0.00| 0.00| 0.00| 0.00|| 4.70| 95.30| 3945|| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 95.03
1| 0.73| 3.70| 0.00| 0.00|| 4.52| 95.48| 3864|| 0.00| 0.01| 1.19| 0.44| 2.82| 0.00| 90.23
5| 0.73| 3.70| 0.00| 0.00|| 0.37| 99.63| 3807|| 0.00| 0.00| 0.03| 0.09| 1.70| 0.00| 97.64
2| 2.28| 12.86| 0.00| 0.00|| 1.41| 98.59| 3829|| 0.00| 0.86| 3.17| 0.46| 7.70| 0.00| 85.87
6| 2.28| 12.86| 0.00| 0.00|| 2.88| 97.12| 3856|| 0.00| 0.11| 4.56| 2.15| 10.31| 0.00| 78.99
3| 1.33| 4.81| 0.00| 0.00|| 0.99| 99.01| 3804|| 0.00| 0.49| 0.79| 0.01| 1.03| 0.00| 96.12
7| 1.34| 4.81| 0.00| 0.00|| 1.26| 98.74| 3818|| 0.00| 0.01| 2.32| 0.47| 5.02| 0.00| 90.06
root@daq01:~#
</pre>
* check that the CPU is not overheating:
<pre>
root@daq01:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 1: +38.0°C (high = +84.0°C, crit = +100.0°C)
Core 2: +34.0°C (high = +84.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +84.0°C, crit = +100.0°C)
</pre>
* congratulations, we are running at 4 GHz now!

= Setup ubuntu as gateway to private network =

See also:
* https://daq.triumf.ca/DaqWiki/index.php/VME-CPU#Setup_the_boot_host_computer_.28el7.29
* http://www.triumf.info/wiki/DAQwiki/index.php/Dhcpd_on_eth1

== Steps to do ==

!!! UPDATED 16feb2024 Ubuntu-22.04.03 !!!

* assign network numbers to the private network, i.e. 192.168.1.x, 192.168.2.x, etc
* (on the gateway machine, each private network interface has to have a different network number)
* (each network interface can have multiple networks attached, via VLANs or via eth0:0, eth0:1 constructs)
* assign IP addresses on the private network, save them in /etc/hosts i.e. "hvps 192.168.1.10"
* (for simplicity, assign 192.168.1.1 to the gateway machine itself)
* (IP addresses 192.168.1.0 and 192.168.1.255 are "special", do not use them)
* setup DNS server (dnsmasq) to serve contents of /etc/hosts via DNS (otherwise, many programs will see inconsistent name to IP address mapping)
* setup DHCP server (dnsmasq) to give out the IP addresses
* setup TFTP server (dnsmasq), pxelinux and NFS for diskless booting
* setup time server (chronyd) to provide common time to all devices
* setup NAT so machines on private network can access the internet (to get OS updates, etc)
* setup NIS and NFS so machines on the private network can use common home directories
* setup rsync backup of machines on the private network

== setup hosts ==

* edit /etc/hosts
<pre>
192.168.1.101 dsfe01
... and so forth
</pre>

== setup dns and dhcp ==

!!! updated 16feb2024 for Ubuntu 22.04.3 !!!

!!! note: stock systemd-resolved remains, is configured to forward queries to dnsmasq, configured to forward queries to TRIUMF DNS !!!

!!! note: per authors of systemd, bare hostnames are not permitted, a DNS domain name must always be used. DNS domain name "dsdaq" is used in this example !!!

* apt install dnsmasq
* ensure dnsmasq starts after all interfaces are up (Ubuntu-22)
<pre>
mkdir /etc/systemd/system/dnsmasq.service.d
echo -e "[Unit]\nAfter=network-online.target\n" > /etc/systemd/system/dnsmasq.service.d/local.conf
</pre>
* edit /etc/dnsmasq.conf
<pre>
# /etc/dnsmasq.conf
# DNS settings
#port=0 # disable DNS function
port=53 # enable DNS function
bind-interfaces # do not collide with systemd-resolved, we use 127.0.0.1:53, they use 127.0.0.53:53
domain-needed
bogus-priv
no-resolv
#log-queries # log DNS quesries

# TRIUMF DNS settings

server=142.90.100.19
expand-hosts
domain=dsdaq
local=/dsdaq/
localmx # do not forward MX queries to TRIUMF

# DHCP settings
interface=enp1s0f0 # VX network 192.168.0.x
#interface=missing # FEP and TSP network 192.168.1.x
interface=enp1s0f1 # controls network 192.168.2.x
#dhcp-range=192.168.1.50,192.168.1.150,infinite
dhcp-range=192.168.0.0,static
dhcp-range=192.168.2.0,static
log-dhcp # log DHCP queries
#quiet-dhcp
dhcp-ignore=tag:!known
#dhcp-boot=pxelinux.0

dhcp-option=option:dns-server,192.168.0.248
dhcp-option=option:ntp-server,192.168.0.248

# TFTP settings

enable-tftp
tftp-root=/tftpboot
</pre>
* #mkdir /tftpboot ### per tftp-root (if no ZFS)
* zfs create -o mountpoint=/tftpboot rpool/tftpboot ### (if root is ZFS)
* create resolved-dsdaq.conf with main IP address of dnsmasq
<pre>
[Resolve]
DNS=192.168.0.248
Domains=dsdaq triumf.ca
</pre>
* mkdir -p /etc/systemd/resolved.conf.d/
* /bin/rm -f /etc/systemd/resolved.conf.d/*.conf
* cp resolved-dsdaq.conf /etc/systemd/resolved.conf.d/
* systemctl stop systemd-resolved.service
* systemctl disable systemd-resolved.service
* systemctl enable dnsmasq
* systemctl restart dnsmasq
* try to "ping" or "host" some names from /etc/hosts, it should work
* try to ping daq00, daq00.triumf.ca, all should work
* resolved-dsdaq.conf goes into /etc/systemd/resolved.conf.d/ of all machines on the private network
* if not using systemd-resolved, edit /etc/resolv.conf

== setup chronyd ==

* enable ntp server:
* disable systemd-timesyncd, configure and enable chronyd per instructions above
* create dsdaq.conf
<pre>
# chrony config for dsdaq server

#allow 192.168.0.0
#allow 192.168.1.0
#allow 192.168.2.0
allow all

# end
</pre>
* cp dsdaq.conf /etc/chrony/conf.d/
* systemctl restart chronyd
* chronyc tracking ### wait until time is synchronized (a few seconds)
* create dsdaq.sources # use hostname or IP address of chronyd server
<pre>
# Put this file in /etc/chrony/sources.d
# systemctl restart chrony
# chronyc sources
# chronyc tracking
server dsdaqgw iburst prefer
# end
</pre>
* dsdaq.sources goes to /etc/chrony/sources.d of all machines on the private network

== setup diskless network booting ==

=== setup pxelinux for legacy pxe boot ===

* add bits in dnsmasq.conf
<pre>
dhcp-host=ac:1f:6b:9e:7f:4a,dsfe01,infinite
dhcp-boot=pxelinux.0
dhcp-option=17,"192.168.0.251:/nfsroot/%s,vers=3,tcp"
</pre>
* setup pxelinux for Ubuntu-18
<pre>
cd ~
wget https://www.kernel.org/pub/linux/utils/boot/syslinux/4.xx/syslinux-4.03.tar.bz2
tar xjvf syslinux-4.03.tar.bz2
cd syslinux-4.03
cp -pv ./core/pxelinux.0 ./com32/hdt/hdt.c32 ./memdisk/memdisk ./com32/menu/menu.c32 /zssd/tftpboot/
</pre>
* cd /zssd/tftpboot
<pre>
wget http://ladd00.triumf.ca/tftpboot/memtest86+-4.20.iso.zip
wget http://ladd00.triumf.ca/tftpboot/memtest86+-5.01.iso.gz
wget http://ladd00.triumf.ca/tftpboot/modules.alias
wget http://ladd00.triumf.ca/tftpboot/modules.pcimap
wget http://ladd00.triumf.ca/tftpboot/pci.ids
</pre>
* mkdir pxelinux.cfg
* emacs -nw pxelinux.cfg/default
<pre>
default menu.c32
prompt 0

menu title Welcome to the DSVSLICE PXE boot menu

timeout 50

label hdt
kernel hdt.c32

label memtest86+-5.01
kernel memdisk iso initrd=memtest86+-5.01.iso.gz

label memtest86+-4.20
kernel memdisk iso initrd=memtest86+-4.20.iso.zip

label vmlinuz-5.3.0-26-generic
menu default
kernel vmlinuz-5.3.0-26-generic
append initrd=initrd.img-5.3.0-26-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.1.1:/zssd/nfsroot/dsfe01 toram ip=dhcp panic=60 BOOTIF=enp1s0f0

#end
</pre>

=== setup pxelinux for efi pxe boot ===

* https://c-nergy.be/blog/?p=13808
* add dnsmasq.conf bits. note: to use dhcp root-path, see the "nfsroot=auto" patch below and make sure to use the "dhcp-option-force" command (mkinitramfs dhcp client does not ask for root-path, we have to force-feed it).
<pre>
# uefi pxe

dhcp-boot=tag:uefipxe,uefi/syslinux.efi
dhcp-option-force=tag:fe01,option:root-path,192.168.0.248:/nfsroot/fe01

# VX network 192.168.0.x

dhcp-host=40:a6:b7:c1:d9:c5,fe01,infinite,set:uefipxe,set:fe01
</pre>
* apt install syslinux pxelinux syslinux-common syslinux-efi syslinux-utils
<pre>
mkdir /tftpboot/uefi
cp /usr/lib/SYSLINUX.EFI/efi64/syslinux.efi /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/ldlinux.e64 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/menu.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/hdt.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libutil.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libmenu.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libcom32.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libgpl.c32 /tftpboot/uefi/
</pre>
* try to boot, it should bomb with "cannot load pxelinux.cfg/default"
* mkdir /tftpboot/uefi/pxelinux.cfg
* create /tftpboot/uefi/pxelinux.cfg/default, note nfsroot path is hardwired, note "http:" is used to load vmlinuz and initrd files (because tftp is super slow)
<pre>
default menu.c32
prompt 0

menu title Welcome to the DSDAQGW UEFI PXE boot menu

timeout 50

label vmlinuz-6.5.0-17-generic
kernel http://192.168.0.248:8088/uefi/vmlinuz-6.5.0-17-generic
append initrd=http://192.168.0.248:8088/uefi/initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=auto rw ip=dhcp panic=60

# append initrd=http://192.168.0.248:8088/uefi/initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.0.248:/nfsroot/fe01 rw ip=dhcp panic=60

# append initrd=initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.0.248:/nfsroot/fe01 rw ip=dhcp panic=60
# append initrd=initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=auto ip=dhcp rw panic=60

#end
</pre>
* try to boot, it will bomb with "cannot load http://...."
* install mini_httpd on port 8088, see https://acme.com/software/mini_httpd/
<pre>
apt install mini-httpd
emacs -nw /etc/default/mini-httpd # set "START=1"
emacs -nw /etc/mini-httpd.conf # set "host=192.168.0.248", "port=8088", "data_dir=/tftpboot"
mkdir /etc/systemd/system/mini-httpd.service.d
echo -e "[Unit]\nAfter=network-online.target\n" > /etc/systemd/system/mini-httpd.service.d/local.conf
systemctl enable mini-httpd
systemctl restart mini-httpd
systemctl status mini-httpd
wget http://192.168.0.248:8088/uefi/syslinux.efi
tail -100 /var/log/mini_httpd.log
</pre>
* fix U-22 initramfs bug for "nfsroot=auto", otherwise, "nfsroot=" has to be different for each machine and you have to have separate pxelinux config files for each machine
** emacs -nw /usr/lib/initramfs-tools/etc/dhcp/dhclient-enter-hooks.d/config
** add "echo ROOTPATH=..." if it is missing
<pre>
echo "ROOTSERVER='${new_routers%% *}'"
echo "ROOTPATH='$new_root_path'"
echo "HOSTNAME='$new_host_name'"
</pre>
* fix U-24 initramfs bug for "nfsroot=auto", otherwise, "nfsroot=" has to be different for each machine and you have to have separate pxelinux config files for each machine
** emacs -nw /usr/share/initramfs-tools/dhcpcd-hooks/70-net-conf
** add "ROOTPATH=..." if it is missing
<pre>
DNSDOMAIN='${new_domain_name-}'
ROOTSERVER='${new_routers-}'
ROOTPATH='${new_root_path-}'
filename='${new_filename-}'
DHCPLEASETIME='${new_dhcp_lease_time-}'
</pre>
** regenerate initramfs (be careful you generate it for the right kernel!)
** see https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/2054482
<pre>
mkinitramfs 6.5.0-18-generic
mkinitramfs 6.8.0-51-generic -o /boot/initrd.img-6.8.0-51-generic
</pre>
* copy linux kernel and initrd
<pre>
cp /boot/vmlinuz-6.5.0-18-generic /tftpboot/uefi/
cp /boot/initrd.img-6.5.0-18-generic /tftpboot/uefi/
chmod a+r /tftpboot/uefi/*
</pre>
* try to boot, should bomb with messages about "trying to mount root filesystem"
* tail /var/log/syslog
<pre>
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 vendor class: PXEClient:Arch:00007:UNDI:003016
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 1:netmask, 2:time-offset, 3:router, 4, 5,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 6:dns-server, 12:hostname, 13:boot-file-size,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 15:domain-name, 17:root-path, 18:extension-path,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 22:max-datagram-reassembly, 23:default-ttl,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 28:broadcast, 40:nis-domain, 41:nis-server,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 42:ntp-server, 43:vendor-encap, 50:requested-address,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 51:lease-time, 54:server-identifier, 58:T1,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 59:T2, 60:vendor-class, 66:tftp-server, 67:bootfile-name,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 97:client-machine-id, 128, 129, 130, 131,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 132, 133, 134, 135
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 next server: 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 broadcast response
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 1 option: 53 message-type 2
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 18 option: 67 bootfile-name uefi/syslinux.efi
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 12 hostname fe01
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 vendor class: PXEClient:Arch:00007:UNDI:003016
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 1:netmask, 2:time-offset, 3:router, 4, 5,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 6:dns-server, 12:hostname, 13:boot-file-size,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 15:domain-name, 17:root-path, 18:extension-path,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 22:max-datagram-reassembly, 23:default-ttl,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 28:broadcast, 40:nis-domain, 41:nis-server,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 42:ntp-server, 43:vendor-encap, 50:requested-address,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 51:lease-time, 54:server-identifier, 58:T1,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 59:T2, 60:vendor-class, 66:tftp-server, 67:bootfile-name,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 97:client-machine-id, 128, 129, 130, 131,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 132, 133, 134, 135
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 next server: 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 broadcast response
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 1 option: 53 message-type 5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 18 option: 67 bootfile-name uefi/syslinux.efi
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 12 hostname fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: error 8 User aborted the transfer received from 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/syslinux.efi to 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/syslinux.efi to 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 requested options: 1:netmask, 3:router, 6:dns-server
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 bootfile name: uefi/syslinux.efi
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 next server: 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 broadcast response
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 1 option: 53 message-type 2
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 requested options: 1:netmask, 3:router, 6:dns-server
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 bootfile name: uefi/syslinux.efi
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 next server: 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 broadcast response
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 1 option: 53 message-type 5
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/ldlinux.e64 to 192.168.0.110
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/01-40-a6-b7-c1-d9-c5 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8006E not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8006 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A800 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A80 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/pxelinux.cfg/default to 192.168.0.110
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/menu.c32 to 192.168.0.110
Feb 16 20:43:10 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/libutil.c32 to 192.168.0.110
Feb 16 20:43:10 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/pxelinux.cfg/default to 192.168.0.110
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 client provides name: dsdaqgw.triumf.ca
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 1:netmask, 28:broadcast, 2:time-offset, 3:router,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 15:domain-name, 6:dns-server, 119:domain-search,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 12:hostname, 44:netbios-ns, 47:netbios-scope,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 26:mtu, 121:classless-static-route, 42:ntp-server
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 bootfile name: uefi/syslinux.efi
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 next server: 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 1 option: 53 message-type 2
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 51 lease-time infinite
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 client provides name: dsdaqgw.triumf.ca
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 1:netmask, 28:broadcast, 2:time-offset, 3:router,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 15:domain-name, 6:dns-server, 119:domain-search,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 12:hostname, 44:netbios-ns, 47:netbios-scope,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 26:mtu, 121:classless-static-route, 42:ntp-server
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 bootfile name: uefi/syslinux.efi
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 next server: 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 1 option: 53 message-type 5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 51 lease-time infinite
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 12 hostname fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw rpc.mountd[3350210]: authenticated mount request from 192.168.0.110:981 for /nfsroot/fe01 (/nfsroot/fe01)
Feb 16 20:45:07 dsdaqgw rpc.mountd[3350210]: authenticated unmount request from 192.168.0.110:859 for /nfsroot/fe01/tmp/autoDY4k5u (/nfsroot/fe01)
</pre>
* tail /var/log/mini_httpd.log
<pre>
192.168.0.110 - - [16/Feb/2024:20:43:15 -0800] "GET /uefi/vmlinuz-6.5.0-17-generic HTTP/1.0" 200 14227944 "" "Syslinux/6.04"
192.168.0.110 - - [16/Feb/2024:20:43:24 -0800] "GET /uefi/initrd.img-6.5.0-17-generic HTTP/1.0" 200 137824833 "" "Syslinux/6.04"
</pre>

=== setup efi http boot ===

https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-deployment-prep-uefi-httpboot.html

=== setup linux kernel ===

* copy the kernel files
<pre>
cd /boot
rsync -av config* initrd* System.map* vmlinuz* /tftpboot/
</pre>
* cd /tftpboot
* chmod a+r *

=== setup nfs ===

* apt-get install nfs-kernel-server
* enable NFS over UDP, edit /etc/nfs.conf add "udp=y":
<pre>
udp=y
</pre>
<pre>
systemctl restart nfs-server.service
</pre>
* emacs -nw /etc/exports
<pre>
/nfsroot/dsfe01 dsfe01(rw,no_root_squash,async,no_subtree_check)
</pre>
* enable services
<pre>
systemctl enable nfs-server
systemctl enable nfs-mountd
systemctl enable nfs-idmapd
systemctl restart nfs-server
systemctl restart nfs-mountd
systemctl restart nfs-idmapd
</pre>
* after editing /etc/exports, run
<pre>
exportfs -av
</pre>

=== setup userland ===

!!! ubuntu-18 version !!!

* zfs create rpool/nfsroot
* zfs set dedup=verify rpool/nfsroot ### enable deduplication to save disk space because most linux images have mostly identical files
* clone ubuntu
<pre>
mkdir /nfsroot/dsfe01
cd /
rsync -avx . /nfsroot/dsfe01
</pre>
* edit config files:
* cd /nfsroot/dsfe01
* emacs -nw etc/hostname ### change to dsfe01
* emacs -nw etc/mailname ### change to dsfe01
* emacs -nw etc/yp.conf ### change daq00.triumf.ca to musr00.triumf.ca
* emacs -nw etc/defaultdomain ### change to MUSR-NIS
* cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
* emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
* emacs -nw root/.ssh/authorized_keys ### update root ssh keys
* emacs -nw etc/fstab ### add this
<pre>
192.168.1.1:/nfsroot/dsfe01 / nfs defaults,nolock 0 0
</pre>
* emacs -nw etc/chrony/chrony.conf
** comment-out all "pool" and "server" entries
** add entry "server 192.168.1.1 iburst"

After dsfe01 is booted:

* disable services:
<pre>
systemctl disable apache2
systemctl disable dnsmasq
systemctl disable zfs-import-cache
</pre>

To setup additional machines, clone dsfe01 instead of cloning the gateway machine

=== Allow manpages to be viewed ===

If <code>/</code> is mounted over NFS, <code>man</code> will report a permission error. Fix it with:

<pre>
ln -s /etc/apparmor.d/usr.bin.man /etc/apparmor.d/disable/
apparmor_parser -R /etc/apparmor.d/usr.bin.man
</pre>

== setup shared home directory ==

=== on the gateway machine ===
* define netgroups
* emacs -nw /etc/netgroup
<pre>
dsfe (dsfe01,,) (dsfe02,,)
</pre>
* emacs -nw /etc/nsswitch.conf ### edit the netgroup line to read:
<pre>
netgroup: files
</pre>
* export the home directories:
* emacs -nw /etc/exports ### add this:
<pre>
/zssd/home1 @dsfe(rw,no_root_squash,async,no_subtree_check)
</pre>
* exportfs -rc

=== on the frontend machine ===

* mkdir /home
* emacs -nw /etc/fstab ### add this:
<pre>
192.168.1.1:/zssd/home1 /home nfs defaults 0 0
</pre>
* mount -a

== setup NAT ==

NAT allows machines on the private network to connect to the internet: https://en.wikipedia.org/wiki/Network_address_translation

In these examples:
* replace "eno1" with name of the outgoing interface (the one connected to the TRIUMF network).
* replace "enp11s0" with name of the private network interface (192.168.1.x network)

* emacs -nw /etc/rc.local ### add this:
<pre>
# /etc/rc.local

# enable NAT

/sbin/iptables -t nat -A POSTROUTING -o eno1 -j MASQUERADE
iptables -L -v

# uncomment following lines if machine has prohibitive FORWARD rules:
#/sbin/iptables -I FORWARD -i eno1 -o enp11s0 -m state --state RELATED,ESTABLISHED -j ACCEPT
#/sbin/iptables -I FORWARD -i enp11s0 -o eno1 -j ACCEPT
#iptables -L -v

iptables -L -v
sysctl -w net.ipv4.ip_forward=1
#sysctl -a | grep forward

sh /etc/firewall-rfc1918.sh

# end
</pre>
* emacs -nw /etc/firewall-rfc1918.sh
<pre>
# firewall-rfc1918.sh

# prevent RFC1918 private network IP addresses from
# going in and out from our uplink.

ETH=eno1

iptables -F in-rfc1918
iptables -N in-rfc1918
iptables -A in-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A in-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A in-rfc1918 --dst 192.168.0.0/16 -j REJECT

iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -I INPUT -j in-rfc1918 -i $ETH

iptables -F out-rfc1918
iptables -N out-rfc1918
iptables -A out-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A out-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A out-rfc1918 --dst 192.168.0.0/16 -j REJECT

iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -I OUTPUT -j out-rfc1918 -o $ETH

iptables -D FORWARD -j out-rfc1918 -o $ETH
iptables -D FORWARD -j out-rfc1918 -o $ETH
iptables -I FORWARD -j out-rfc1918 -o $ETH

# allow TRIUMF-SECURE network

iptables -I in-rfc1918 -s 10.90.0.0/255.255.0.0 -j ACCEPT
iptables -I out-rfc1918 -d 10.90.0.0/255.255.0.0 -j ACCEPT

# show configuration

iptables -L -v

#end
</pre>

= KVM =

<pre>
apt install cpu-checker

root@daq13:~# kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
root@daq13:~#

(if not, shutdown, go into BIOS settings, enable CPU virtualization)

apt install virtinst ### will install many packages
apt install libvirt-clients libvirt-daemon-system-systemd libvirt-daemon qemu qemu-kvm libvirt-daemon-system virtinst bridge-utils

root@daq13:/home1/wheel# virsh list --all
Id Name State
------------------------------
1 ubuntu-guest running

apt install virt-manager

virt-install --name ubuntu-guest --os-variant ubuntu20.04 --vcpus 2 --ram 2048 --location /daq/daqstore/olchansk/linux/Ubuntu/ubuntu-20.04.3-desktop-amd64.iso --network bridge=virbr0,model=virtio --graphics none --extra-args='console=ttyS0,115200n8 serial'

virtual machine will start, boot, etc
to get out of it, CTRL + Shift followed by ]

ssh wheel@daq13
virt-manager

run virt-install again, omit "--graphics none", open graphics console from virt-manager, it booted into ubuntu installer desktop

virt-install --name test10 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --filesystem /kvm_ladd00,/ --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial" --graphics none

virt-install --name test14 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --disk /tmp/xxx/ladd00.img,bus=sata --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial rdshell" --graphics none --check path_in_use=off
</pre>

build image

<pre>
dd if=/dev/zero of=/tmp/xxx/ladd00.img bs=1024M count=20
mkfs.ext3 /tmp/xxx/ladd00.img ### ext4 fails to mount by SL6 kernel, "unknown ext4 options"
cd /kvm_ladd00/
mount -o loop /tmp/xxx/ladd00.img /mnt/tmp
rsync -av . /mnt/tmp/ --delete
umount /mnt/tmp
</pre>

on the guest, configure network: /etc/rc.local
<pre>
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

ifconfig eth2 192.168.122.2
route add -net 0.0.0.0 gw 192.168.122.1
ifconfig -a
netstat -rn

# end
</pre>

== virsh commands ==

<pre>
virsh list --all

virsh start kvm-el7
virsh console kvm-el7
virsh destroy kvm-el7

virsh install ...
virsh undefine kvm-el7

virsh autostart kvm-ladd00
virsh dominfo kvm-ladd00
</pre>

== virtualize SL6 ladd00 ==

* on ladd00:
<pre>
yum install dracut-network
mkinitrd /boot/initramfs-2.6.32-754.35.1.el6.x86_64-netboot.img 2.6.32-754.35.1.el6.x86_64
</pre>
* on daq00
<pre>
zfs create rpool/kvm-ladd00
cd /kvm-ladd00
rsync -avx ladd00:/ . --exclude nfsroot
brctl addbr virbr0
ifconfig virbr0 192.168.1.1
echo /kvm-ladd00 192.168.1.2(rw,no_root_squash,no_all_squash,async,no_subtree_check) >> /etc/exports
exportfs -rv
</pre>
* create virtual machine
<pre>
virt-install --name kvm-ladd00 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --network bridge=virbr0,model=virtio --boot kernel=/kvm-ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm-ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64-netboot.img,kernel_args="root=/dev/nfs ip=192.168.1.2:192.168.1.1:192.168.1.1:255.255.255.0:ladd00::off nfsroot=192.168.1.1:/kvm-ladd00,vers=3,tcp console=ttyS0,115200n8 serial rdshell" --graphics none --nodisks --check path_in_use=off
</pre>
* adjust kvm-ladd00 image
<pre>
disable network manager
edit fstab
edit yp.conf
edit resolv.conf
edit root/.ssh/authorized_keys
enable rngd or /dev/random does not work, sshd does not work
</pre>
* virsh shutdown test24
* virsh --connect qemu:///system start test24
* virsh console test24 ### to exit, ctrl+[ or ctrl+]
* virsh undefine test24
* virsh autostart kvm-ladd00
* virsh dominfo kvm-ladd00
<pre>
root@daq00:~# virsh dominfo kvm-ladd00
Id: 1
Name: kvm-ladd00
UUID: 1d1f8fed-8b65-4411-a51b-e0ecf359d2f1
OS Type: hvm
State: running
CPU(s): 2
CPU time: 27.7s
Max memory: 2097152 KiB
Used memory: 2097152 KiB
Persistent: yes
Autostart: enable
Managed save: no
Security model: apparmor
Security DOI: 0
Security label: libvirt-1d1f8fed-8b65-4411-a51b-e0ecf359d2f1 (enforcing)
root@daq00:~#
</pre>
* delete unused images in /var/lib/libvirt/images
* virsh edit kvm-ladd00 # change boot command line, etc

== virtualize CentOS-7 daqstore ==

* similar to ladd00 above:
* on daqstore, install dracut-network, already there
<pre>
yum install dracut-network
yum install qemu-kvm libvirt libvirt-python libguestfs-tools virt-install
# yum install busybox ### no rpm package?!?
</pre>
<pre>
dracut -a nfs -v /boot/initramfs-3.10.0-1160.119.1.el7.x86_64-virt.img 3.10.0-1160.119.1.el7.x86_64 --force
</pre>
<pre>
scp daqstore:/boot/initramfs-3.10.0-1160.119.1.el7.x86_64-virt.img /kvm-el7/boot/
</pre>
* on daq00
<pre>
zfs create rpool/kvm-el7
cd /kvm-el7
rsync -avx daqstore:/ .
echo 192.168.1.3 kvm-el7 >> /etc/hosts
systemctl restart dnsmasq
echo /kvm-el 192.168.1.3(rw,no_root_squash,no_all_squash,async,no_subtree_check) >> /etc/exports
exportfs -rv
</pre>
* manage virtual machine
<pre>
virsh console kvm-el7
virsh destroy kvm-el7
virsh undefine kvm-el7
</pre>
* create virtual machine
<pre>
virt-install --name kvm-el7 --os-variant centos7 --vcpus 2 --ram 2048 --import --network bridge=virbr0,model=e1000e --boot kernel=/kvm-el7/boot/vmlinuz-3.10.0-1160.119.1.el7.x86_64,initrd=/kvm-el7/boot/initramfs-3.10.0-1160.119.1.el7.x86_64-virt.img,kernel_args="root=/dev/nfs ip=192.168.1.3:192.168.1.1:192.168.1.1:255.255.255.0:kvm-el7::off nfsroot=192.168.1.1:/kvm-el7,vers=3,tcp rw console=ttyS0,115200n8 serial rdshell" --graphics none --nodisks --check path_in_use=off
</pre>
* adjust kvm-el7 image
<pre>
disable network manager
edit fstab
edit hostname
disable selinux in /etc/sysconfig/selinux

UP TO HERE --- DNS does not work!!!

edit yp.conf
edit resolv.conf
edit root/.ssh/authorized_keys
enable rngd or /dev/random does not work, sshd does not work
</pre>
* virsh shutdown test24
* virsh --connect qemu:///system start test24
* virsh console test24 ### to exit, ctrl+[ or ctrl+]
* virsh undefine test24
* virsh autostart kvm-ladd00
* virsh dominfo kvm-ladd00
<pre>
root@daq00:~# virsh dominfo kvm-ladd00
Id: 1
Name: kvm-ladd00
UUID: 1d1f8fed-8b65-4411-a51b-e0ecf359d2f1
OS Type: hvm
State: running
CPU(s): 2
CPU time: 27.7s
Max memory: 2097152 KiB
Used memory: 2097152 KiB
Persistent: yes
Autostart: enable
Managed save: no
Security model: apparmor
Security DOI: 0
Security label: libvirt-1d1f8fed-8b65-4411-a51b-e0ecf359d2f1 (enforcing)
root@daq00:~#
</pre>
* delete unused images in /var/lib/libvirt/images
* virsh edit kvm-ladd00 # change boot command line, etc

= ARM64 cross-compiler =

== Ubuntu-22 ==

* arm64, aarch64 are Xilinx FPGA Cortex-A53, RPi4, RPi5 machines
* install packages:
<pre>
apt install g++-12-aarch64-linux-gnu gcc-12-aarch64-linux-gnu-base libstdc++-12-dev-arm64-cross
</pre>
* run:
<pre>
aarch64-linux-gnu-gcc-12 -o ttcp.aarch64 ttcp.c -static
aarch64-linux-gnu-g++-12 -o fecdm.exe -O2 -g -Wall -Wuninitialized -std=c++20 fecdm.o dsdm.o /home/dsdaqdev/packages_common/midas/linux-aarch64-remoteonly/lib/libmidas.a -pthread -lrt -lutil /nfsroot/gdm00/usr/lib/aarch64-linux-gnu/libi2c.a -static
</pre>

== Ubuntu-24 ==

* arm64, aarch64 are Xilinx FPGA Cortex-A53, RPi4, RPi5 machines
* install packages:
<pre>
apt install g++-aarch64-linux-gnu
</pre>
* build:
<pre>
aarch64-linux-gnu-g++ -c -o xvcserver_cdm.o -O2 -g -Wall -Wuninitialized -std=c++20 xvcserver_cdm.cxx
aarch64-linux-gnu-g++ -o xvcserver_cdm.exe -O2 -g -Wall -Wuninitialized -std=c++20 xvcserver_cdm.o -pthread -lrt -lutil -static
</pre>

= ARM cross-compiler =

NOTE: updated for U-24

* armv7 (Cyclone-V SoC, RPi3, MityARM CAMAC) machines (Debian-12 armhf target, Ubuntu 24.04 host)
* install packages:
<pre>
apt install g++-arm-linux-gnueabihf gcc-arm-linux-gnueabihf libc6-dev-armhf-cross
</pre>
* build MIDAS frontend (static linking)
<pre>
arm-linux-gnueabihf-g++ -std=c++11 -Wall -Wuninitialized -g -O2 -I/home/dldaq/packages/midas/include -I/home/dldaq/packages/midas/mvodb -c koi2c.cxx
arm-linux-gnueabihf-g++ -o fedldb.exe -std=c++11 -Wall -Wuninitialized -g -O2 -I/home/dldaq/packages/midas/include -I/home/dldaq/packages/midas/mvodb fedldb.o koi2c.o /home/dldaq/packages/midas/linux-armv7-remoteonly/lib/libmidas.a -L/usr/arm-linux-gnueabihf/lib -L/nfsroot/dltdc/usr/lib/arm-linux-gnueabihf -static -lm -lz -lutil -lnsl -lpthread -lrt -li2c
/usr/lib/gcc-cross/arm-linux-gnueabihf/13/../../../../arm-linux-gnueabihf/bin/ld: /home/dldaq/packages/midas/linux-armv7-remoteonly/lib/libmidas.a(system.o): in function `ss_socket_connect_tcp(char const*, int, int*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)':
/home/dldaq/packages/midas/src/system.cxx:4984:(.text+0x252a): warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
</pre>

= 32-bit intel cross-compiler =

== Ubuntu 22.04 ==

<pre>
apt install libstdc++-11-dev:i386
apt install zlib1g-dev:i386
</pre>

NOTES:
* "g++ -m32" does not find libstdc++, please use "g++ -m32 -L/usr/lib/gcc/i686-linux-gnu/11/"
* to cross-build 32-bit MIDAS, use "make linux32".
* executables cross-build on Ubuntu-22 do NOT run on 32-bit Debain-11 (GLIBC and GLIBCXX version mismatch)
* executables cross-build on Ubuntu-22 run on 32-bit Debian-12.

== Ubuntu 24.04 ==

<pre>
apt install gcc-i686-linux-gnu
apt install g++-i686-linux-gnu
apt install libstdc++-13-dev:i386
apt install lib32z1 lib32z1-dev
i686-linux-gnu-gcc -o ttcp.i386 ttcp.c
</pre>

NOTES:
* executables cross-build on Ubuntu-24 will NOT run on 32-bit Debian-12 (GLIBC mismatch, static executables maybe work)
* executables cross-build on Ubuntu-24 run on 32-bit Debian-13

= SSH settings for EPICS =

* TRIUMF EPICS runs obsolete version of SSH
* add this to the use .ssh/config
<pre>
Host sbp1*
HostKeyAlgorithms +ssh-rsa
PubKeyAcceptedAlgorithms +ssh-rsa
KexAlgorithms +diffie-hellman-group1-sha1
ForwardX11 yes
ForwardX11Trusted yes
</pre>

= changes for VME processors =

<pre>
apt -y remove sysstat man-db
apt -y purge dkms
apt -y purge mdadm
apt -y purge fwupd
apt -y purge packagekit
apt -y purge accountsservice
apt -y purge plocate
apt -y purge upower power-profiles-daemon
apt -y autoremove
</pre>

for D-12 32-bit CPUs:

<pre>
apt remove "*libavahi*"
</pre>

= remove snap (U-24) =

Note: snap stores data in $USER/snap/$SNAPNAME, removing a snap on one machine will remove this data from all users even if they want to use snap on some other machine.

Prepare:

NOTE: first remove chromium and firefox, see below.

NOTE: if possible, stop autofs before removing snap - otherwise it will mount all user home directories and complain that it cannot remove some snap data from them

<pre>
systemctl stop autofs
ls -ld /home1/*/snap/* ### remove the per-user snap directories
</pre>

Remove snaps:

<pre>
snap list
echo snap remove --purge chromium ### see below
echo snap remove --purge firefox ### see below
snap remove thunderbird
snap remove cups
snap remove hello-world
snap remove firmware-updater
snap remove gtk-common-themes
snap remove snapd-desktop-integration
snap remove snap-store
snap remove hunspell-dictionaries-1-7-2004
snap remove gnome-system-monitor
snap remove gnome-3-26-1604
snap remove gnome-3-28-1804
snap remove gnome-3-34-1804
snap remove gnome-3-38-2004
snap remove gnome-42-2204
snap remove gnome-46-2404
snap remove mesa-2404
snap remove core
snap remove core18
snap remove core20
snap remove core22
snap remove core24
snap remove bare
snap remove snapd
snap list
</pre>
<pre>
root@daqubuntu:~# snap list
No snaps are installed yet. Try 'snap install hello-world'.
root@daqubuntu:~#
</pre>

Identify packages that install snaps:

<pre>
apt list | grep snap | grep installed | grep -v -e snappy -e snapshot
</pre>

Typical output:

<pre>
firefox/noble,now 1:1snap1-0ubuntu5 amd64 [installed]
gir1.2-snapd-2/noble,now 1.64-0ubuntu5 amd64 [installed,automatic]
libsnapd-glib-2-1/noble,now 1.64-0ubuntu5 amd64 [installed,automatic]
libsnapd-qt-2-1/noble,now 1.64-0ubuntu5 amd64 [installed,automatic]
plasma-discover-backend-snap/noble,now 5.27.11-0ubuntu2 amd64 [installed]
snapd/noble-updates,now 2.66.1+24.04 amd64 [installed]
</pre>

Remove packages that install snaps:

<pre>
apt remove chromium-browser
apt remove chromium-codecs-ffmpeg-extra
apt remove thunderbird
apt remove firefox
apt remove plasma-discover-backend-snap
apt remove plasma-discover-snap-backend
apt remove snapd
apt purge snapd
# package gir1.2-snapd-2 is required by ubuntu-mate-desktop & co
# libsnapd-glib-2-1 is required by gstreamer, gnome-remote-desktop & co
ls -l /etc/systemd/system/ | grep snap ### remove unwanted stuff
</pre>

Remove Chromium:

* ls -ld /home/*/snap/chromium/*
* echo /bin/rm -rf `ls -1d /home/*/snap/chromium/*`
* snap remove chromium

Remove Firefox:

* ls -ld /home/*/snap/firefox/*
* echo /bin/rm -rf `ls -1d /home/*/snap/firefox/*` ### this will delete "snap firefox" profiles of all users!!!
* snap remove firefox ### this will also delete "snap firefox" profiles of all users!!!

Remove gir1.2-snapd-2:
* echo rm -vf /usr/lib/x86_64-linux-gnu/girepository-1.0/Snapd-2.typelib

If "snap remove" is stuck in "change in progress" (this will remove all snaps and break snapd, which is ok,
see https://forum.snapcraft.io/t/snap-remove-taking-forever-abort-wasnt-working/48915)
<pre>
rm /var/lib/snapd/state.json
systemctl restart snapd
</pre>

Prevent snap from reinstalling:
<pre>
cd ~/git/scripts/etc
git pull
cp etc-apt-preferencesd-disable-snap /etc/apt/preferences.d/
</pre>

= install non-snap thunderbird =

from: https://ubuntuhandbook.org/index.php/2024/03/install-thunderbird-deb-ubuntu-2404/

* remove snap thunderbird
<pre>
snap remove --purge thunderbird
apt remove --purge thunderbird
</pre>
* add mozilla repository
<pre>
already done after installing firefox-esr
</pre>
* ppa deb and ubuntu snap thunderbird package names are the same, change priority and hide snap package: create /etc/apt/preferences.d/mozillateamppa
<pre>
Package: thunderbird*
Pin: release o=LP-PPA-mozillateam
Pin-Priority: 1001

Package: thunderbird*
Pin: release o=Ubuntu
Pin-Priority: -1
</pre>
* check that it worked, it should say "build" instead of "snap"
<pre>
apt update
apt list | grep thunderbird | grep -v locale
...
thunderbird/noble 1:128.8.1+build1-0ubuntu0.24.04.1~mt1 amd64
...
</pre>
* install
<pre>
apt install thunderbird
</pre>
* run: thunderbird

= EFI boot using syslinux =

* rationale 1: GRUB is the stock boot loader with U-24. It is unnecessarily complicated. EFI BIOS can boot the linux kernel directly, without GRUB, but unfortunately a small shim bootloader is required to specify the initrd file and the root filesystem. the syslinux boot loader can do this in a very simple way.
* rationale 2: GRUB bootloader configuration is overcomplicated, and when it breaks, it is almost impossible to debug and to recover.
* rationale 3: GRUB bootloader scripts in U-24 have no support for booting from redundant SSDs.
* in the case of GRUB bootloader failure, it is simplest to boot the Ubuntu installer in recovery mode and convert the bootloader from GRUB to syslinux. Open firefox on this page and cut-and-paste the steps (only copy of vmlinux and initrd cannot copy-and-paste as of this writing).
* in the case of servers with redundant SSDs for OS and home directories (ZFS mirror), it is simplest to use the syslinux bootloader to ensure that the machine boots from either SSD (all combinations SSD failures, fail of either EFI partiion, fail of either ZFS mirror partition, machine should boot)
* check partition tables, SATA SSD
<pre>
fdisk -l /dev/sda
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: A3F34DAC-DCB4-B74C-B59E-41E754807812

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root
</pre>
* check partition tables, NVME SSD
<pre>
fdisk -l /dev/nvme0n1
Disk /dev/nvme0n1: 1.75 TiB, 1920383410176 bytes, 3750748848 sectors
Disk model: SAMSUNG MZ1L21T9HCLS-00A07
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 131072 bytes / 131072 bytes
Disklabel type: gpt
Disk identifier: 04ECCD46-DC2A-454C-B4A8-CCC18AA532F7

Device Start End Sectors Size Type
/dev/nvme0n1p1 2048 2203647 2201600 1G EFI System
/dev/nvme0n1p2 2203648 6397951 4194304 2G Linux filesystem
/dev/nvme0n1p3 6397952 23175167 16777216 8G Linux swap
/dev/nvme0n1p4 23175168 3750746111 3727570944 1.7T Linux filesystem
</pre>
* prepare boot device EFI partition, SATA SSDs
<pre>
mkfs.msdos /dev/sda1
mkfs.msdos /dev/sdb1
mkdir /boot/efi-sda
mkdir /boot/efi-sdb
mount /dev/sda1 /boot/efi-sda
mount /dev/sdb1 /boot/efi-sdb
</pre>
* prepare boot device EFI partition, NVME SSDs
<pre>
mkfs.msdos /dev/nvme0n1p1
mkfs.msdos /dev/nvme1n1p1
mkdir /boot/efi-0
mkdir /boot/efi-1
mount /dev/nvme0n1p1 /boot/efi-0
mount /dev/nvme1n1p1 /boot/efi-1
</pre>
* add them to fstab, note the "nofail" mount option
<pre>
blkid | grep vfat
/dev/sdb1: UUID="F30C-13B5" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="20e423b5-ac29-ec42-bab5-f366aefbbd2b"
/dev/sda1: UUID="F2DD-7321" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="9427646c-ce5f-fe47-9ed1-4b84cf4c348f"
grep ^UUID /etc/fstab
UUID=F2DD-7321 /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
UUID=F30C-13B5 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
</pre>
* prepare the EFI partitions (remove old previous subdirectories, only empty efi/boot should be there)
<pre>
cd /boot/efi-sda
mkdir -p efi/boot
</pre>
* get syslinux-6.03
<pre>
cd ~
wget https://daq00.triumf.ca/~olchansk/linux/syslinux-6.03.tar.xz
xz -d < syslinux-6.03.tar.xz | tar xvf -
</pre>
* from syslinux-6.03 copy files:
<pre>
cd /boot/efi-sda/efi/boot
cp ~/syslinux-6.03/efi64/efi/syslinux.efi .
cp ~/syslinux-6.03/efi64/com32/elflink/ldlinux/ldlinux.e64 .
cp syslinux.efi bootx64.efi
</pre>
* identify the ZFS rpool label
<pre>
zfs list | grep ROOT | grep "/$" | cut -f1 -d" "
rpool/ROOT/ubuntu_9yvb17
</pre>
* create syslinux.cfg, change the root=ZFS label to match this computer
<pre>
cat << EOF | sed "s;root=.*$;root=ZFS=`zfs list | grep ROOT | grep "/$" | cut -f1 -d" "`;" > syslinux.cfg
default linux
label linux
kernel vmlinuz
append ro initrd=initrd.img root=ZFS=rpool/ROOT/ubuntu_02ruwj
EOF
</pre>
* copy linux boot files:
<pre>
cp /boot/vmlinuz vmlinuz
cp /boot/initrd.img initrd.img
</pre>
* repeat with /boot/efi-sdb, etc
* install script to set syslinux to boot the latest kernel
<pre>
cd ~/git/scripts
git pull
ln -s ~/git/scripts/etc/update_efi_syslinux.perl ~
</pre>
* update syslinux to boot the latest kernel
** run "~/update_efi_syslinux.perl" to check that it finds the EFI partitions and finds the correct kernel
** run "~/update_efi_syslinux.perl -u" to do the actual update
* or maybe install Ubuntu syslinux 6.04 and use files from there:
<pre>
apt install "syslinux*"
</pre>

= legacy boot using syslinux =

* NOTE: extlinux is not compatible with ext4 "64bit" feature, it should be turned off:
<pre>
mkfs.ext4 -O ^64bit /dev/sdX1
resize2fs -s /dev/sdX1
</pre>

* install syslinux and extlinux (THIS DOES NOT WORK!!!)

<pre>
apt -y install syslinux extlinux
dd if=/usr/lib/syslinux/mbr/mbr.bin of=/dev/sdX ### NOT /dev/sdX1 NOT !!!
cd /boot
cp /usr/lib/syslinux/modules/bios/menu.c32 .
extlinux -i .
</pre>

* install syslinux and extlinux

* copy from old SL6 USB disk (this is extlinux 6.02)
<pre>
root@localhost:/boot# ls -l
-rwxr-xr-x 1 root root 218952 Jan 28 17:40 extlinux
-rw-r--r-- 1 root root 402 Jan 29 14:45 extlinux.conf
-rw-r--r-- 1 root root 496 Jan 29 14:39 extlinux.conf~
-r--r--r-- 1 root root 122044 Jan 29 14:39 ldlinux.c32
-r--r--r-- 1 root root 67072 Jan 29 14:39 ldlinux.sys
-rwxr-xr-x 1 root root 24156 Jan 28 17:40 libutil.c32
-rw-r--r-- 1 root root 304 Jan 28 17:40 mbr.bin
-rw-r--r-- 1 root root 26140 Jan 28 17:40 memdisk
-rw-r--r-- 1 root root 69043 Jan 28 17:40 memtest86+-4.20.iso.zip
-rw-r--r-- 1 root root 183012 Jan 28 17:40 memtest86+-5.01
-rw-r--r-- 1 root root 26568 Jan 28 17:40 menu.c32
</pre>

* install
<pre>
dd if=mbr.bin of=/dev/sdX ### NOT /dev/sdX1 NOT !!!
extlinux -i .
</pre>
* check that partition /dev/sdX1 is marked bootable (fdisk command "a")

* create /boot/extlinux.conf

<pre>
DEFAULT menu.c32
PROMPT 0
TIMEOUT 50

MENU TITLE TRIUMF DAQ USB BOOT32 ver K.O. 2025jan28

LABEL linux
MENU DEFAULT
kernel /vmlinuz
append initrd=/initrd.img panic=60 rootdelay=5 rootwait rw root=/dev/sda1

LABEL linux-6.1.0-28-686
kernel vmlinuz-6.1.0-28-686
append initrd=initrd.img-6.1.0-28-686 panic=60 rootdelay=5 rootwait rw root=/dev/sda1

LABEL memtest
kernel memtest86+-1.65
</pre>

= Add user to login spash screen =

For user login to local machine, if it doesn't work by default. User should exist already from NIS service.

<pre>
sudo /bin/bash # as root
cd /var/lib/AccountsService/users
ls # should at least display the "wheel" user
cp wheel username # we're going to copy the user account settings over to our new user
</pre>

DS-DM

2026-04-01T23:50:34Z

Olchansk: /* Register 71 packet error bits */

= DS-DM =

DarkSide-20k Global and Crate Data Manager board (GDM and CDM).

Global Data Manager (GDM):
* clock distribution to CDM boards (including GPS/atomic clock source)
* collection of trigger data from CDM boards, processing and distribution of trigger decision to CDM boards
* run control
* integration with GPS 10MHz and 1pps clocks and GPS/IRIG date and time information

Crate Data Manager (CDM):
* clock distribution from GDM to CAEN VX digitizers
* receive trigger data from CAEN VX digitizers
* send trigger data to GDM
* run control and dead time control

= Links =

* https://dsdaqgw.triumf.ca/vslice/ - vertical slice DAQ at TRIUMF
* https://daq00.triumf.ca/DaqWiki/index.php/DarkSide - this DS-DAQ Wiki page
* https://dsvslice.triumf.ca/vx_napoli - DS vertical slice at TRIUMF
* https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0 - DS-DM board Rev0
* https://edev-group.triumf.ca/hw/exp/dark-side-20k/cdm-gdm-data-manager-vme-cards/rev1 - DS-DM board Rev1
* https://docs.xilinx.com/v/u/en-US/ds593 - Xilinx Platform Cable USB II
* https://www.enclustra.com/en/products/system-on-chip-modules/mercury-xu8/ - Enclustra
* https://bitbucket.org/team-ds-dm/phasemeasurement - VX Phase measurement script
* https://daq00.triumf.ca/daqinv/frontend/list/178 - inventory database
* https://daq00.triumf.ca/elog-ds/DS-DAQ - DS-DAQ elog
* https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0 - GPS and Rb Clock adapter board

= Git repos =

* dsdmdev@dsdaqgw.triumf.ca:git/gcdm.git - git repository, DS-DM firmware, branch develop_ko2, contact K.O. for ssh key access
* https://edev-group.triumf.ca/fw/exp/darkside/gcdm - git repository, DS-DM firmware (broken, do not use)
* https://bitbucket.org/team-ds-dm/ds-dm-software - git repository, DS-DM MIDAS frontend
* https://bitbucket.org/ttriumfdaq/dsproto_vx2740/src/develop/ - VX DSFE MIDAS frontend
* https://openproject.triumf.ca/svn/darkside-20k/ - VX firmware project

= Onboard hardware =

* jtag chain: arm_dap_0 0x5BA00477, xczu4_1 0x04721093
* Eclustra Mercury+ XU8 module: ME-XU8-4CG-1E-D11E-R2.1
** Xilinx® Zynq Ultrascale+™ MPSoC XCZU4CG-1FBVB900E
** DDR4 ECC SDRAM (PS) 2 GB
** DDR4 SDRAM (PL) 1GB
* ethernet mac chip: AT24MAC402-SSHM-T ("602" chip is wrong)
* USB UART for Enclustra serial console, micro-USB, 115200n8
* clock chip: SI5394A-A-GM and oscillator CS-044-054.0M (54 MHz)
* U23 3.3V current meter and thermometer, LTC2990IMS#TRPBF
* LEDs:
** LED_FP A/B/C/D 0/1/2/3
** led1 - 3V3_SW_ON, SOM_POWER_GOOD - Enclustra FPGA module 3.3V power is good
** led2 - LTM4624 PGOOD
** led3 - FPGA_DONE - FPGA has booted
** led4 - TP-S-1, PCLK_P
** led5 - TP-S-2, PCLK_N
* LEMO connectors (top to bottom)
** J4 - input (NIM/TTL) (EXT_IN_LV(1), EXT_IN_LV(2))
** J5 - input (NIM/TTL) (EXT_IN_LV(3), EXT_IN_LV(4))
** J6 - external clock (GPS 10MHz and PPS)
** J7 - output (NIM/TTL) (EXT_OUT(1), EXT_OUT(2))
* SMA connectors
** J9, J10 - CLK_CCA from U6 C.C.
** J11, J12 - CLK_TP0
* RJ45 ethernet connector (100mbit: green light on, yellow light flashes when there is traffic)
* SFP connector (SFP is FTLF8526P3BNL, 6 Gbit/sec, 850 nm, 300m 50/125um OM3 MMF)
* 4 QSFP connectors (GDM)
* 6 VX connectors (CDM)

= Buttons, jumpers and switches =

Buttons:

* PB1 - HRST - reboot FPGA (power-on reset)
* PB2 - SRST - (SRSTn) - reboot ARM CPU

Switches:

* SW1 - boot mode BM0, BM1 [-->]
* SW2 - LEMO output NIM<->TTL
* SW3 - LEMO input 1 and 2 NIM/TTL
* SW4 - LEMO input 2 and 4 NIM/TTL
* SW5 - LEMO clock input NIM/TTL
* SW6 - serial console select. [PS<--PL] PS is ARM CPU, PL is FPGA.

= Front panel =

<pre>
| top
|
| LED-FP1 | LED_FP(0,1,2,3)
|
| SFP J???
|
| LEMO J4-LEFT, J4-RIGHT | EXT_IN_LV(1), EXT_IN_LV(2)
| LEMO J5-LEFT, J5-RIGHT | EXT_IN_LV(3), EXT_IN_LV(4)
| LEMO J6-LEFT, J6-RIGHT | CLK_EXT1, CLK_EXT0 (125 MHz only)
| LEMO J7-LEFT, J7-RIGHT | EXT_OUT(1), EXT_OUT(2)
|
| J-VX-1
| J-VX-2 or QSFP-1
| J-VX-3 or QSFP-2
| J-VX-4 or QSFP-3
| J-VX-5 or QSFP-4
| J-VX-6
|
| RJ45 J3 ethernet
|
| bottom
</pre>

= VX adapter board =

LVDS I/O connector
<pre>
34 pin connector: 0|:::: :::: :::: :::: :|15,16 n/c
</pre>

== split-cable connection ==

<pre>
0 .. 7 -> N/C

8 -> VX_RX(3) - not used
9 -> VX_RX(2) - busy VX to CDM
10 -> VX_RX(1) - DS20K 125 MHz serial data VX to CDM
11 -> VX_RX(0) - DS20K 62.5MHz clock VX to CDM

12 <- VX_TX(0) - TRG CDM to VX
13 <- VX_TX(1) - TSM CDM to VX, to be VETO CDM to VX
14 <- VX_TX(2) (set by jumper routed here or to VX CLKIN SYNC) - 125 MHz serial data CDM to VX
15 <- n/c (CLK routed to VX CLKIN CLK) - 62.5 MHz clock CDM to VX

16 - n/c
</pre>

== one-to-one connection ==

<pre>
0 -> VX2_RX(3)
1 -> VX2_RX(2)
2 -> VX2_RX(0)
3 -> VX2_RX(1)

4 <- CLK
5 <- VX2_TX(0)
6 <- VX2_TX(1)
7 <- VX2_TX(2)

8 -> VX1_RX(3)
9 -> VX1_RX(2)
10 -> VX1_RX(1)
11 -> VX1_RX(0)

12 <- VX1_TX(0)
13 <- VX1_TX(1)
14 <- VX1_TX(2) (set by jumper routed here or to VX CLKIN SYNC)
15 <- n/c (CLK routed to VX CLKIN CLK)

16 - n/c
</pre>

= Board schematics Rev0 =

* [[File:SCH-DS-xDM-Rev0.PDF]]
* note: FPGA pin annotations ("IO", "SCLK", "PCLK", etc) on the schematics are bogus, instead, trace them to the FPGA pins.
* note: ENC A is J800, ENC B is J801, ENC C is J900 (schematic name to enclustra name)
* note: Enclustra special pins: "GC" is "clock capable", "HDGC" is "clock capable", "MGTREFCLK" is MGT reference clocks.
* board modifications:
** ethernet mac chip
** NIM output (no U15, etc)
** RJ45 wrong pinout (board mod or special ethernet cable)
** 125 MHz clock mods (TBW)
** disconnect QSFP0_SEL from SFP_RS0 and QSFP1_SEL from SFP_RS1, these signals are not used by modern SFPs
** provide SFP i2c modsel to allow SFP and QSFP at the same time on address 0x50

= Board schematics Rev1 =

* [[File:SCH-DS-xDM-Rev1.PDF]]
* ENC-A is J800, ENC-B is J801, ENC-C is J900
* modifications from Rev0:
* clocks:
** 125MHz osc: CLK_XO_125, CLK3_XO_125 (CLK2_XO_125 removed)
** C.C. in2 is CLK_CC_IN from ENC-C142,144
** C.C. out0 via CLK_A to CLK_CC_OUT0 (ENC-C3,5), CLK_CC_OUT1 (ENC-B3,5), CLK_CC_OUT2 (ENC-C151,153)
* RTC chip: TP_S pins gone, RTC_I2C_SCL, RTC_I2C_SDA, RTC_1PPS and RTC_32KHZ added
** RTC_1Hz ENC-C160 pull up 10k to 3.3V
** RTC_32k ENC-B129 pull up 10k to 1.8V
* QSFP, SFP control lines:
** HW_ID_xDM ENC-B131 gone, reused as SFP_ModPrsN
** SFP_ModPrsN (MOD_ABS) ENC-B131 input (no pulls)
** SFP_TX_fault (was QSFP2_SEL) QSFP_ModPrsN ENC-C157 input (no pulls)
** SFP_TX_disable from QSFP_LPMode ENC-C163 output (pull up 10k to 3.3V)
** SFP_ModDet n/c (was QSFP_ModPrsN) renamed SFP_ModPrsN input
** SFP_RX_LOS to QSFP_IntN ENC-C159 input (pull up 2k2 to 3.3V)
** SFP_RS0, SFP_RS1 n/c, assembly option: floating
** QSFP_ResetN - FPGA output, pull up 10k to 3.3V
** QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL - FPGA outputs, pull up 10k to 3.3V
** QSFP_LPmode - FPGA output, pull up 10k to 3.3V
** QSFP_ModPrsN - ??? - pull up 2k2 to 3.3V
** QSFP_IntN - ??? - pull up 2k2 to 3.3V
* VX_TX and VX_RX reassigned for better length matching
* cosmetic changes:
** front panel USB-C connector (replaces Micro-USB)
** (reset push button PB1 PORn ENC-A132 aka PS_POR_B aka PS_POR#)
** (reset push button PB2 SRSTn ENC-A124 aka PS_SRST_B aka PS_SRST#)
** front panel PB3 HRSTn same as PB1
* Enclustra errata:
** possible cross talk between pins AA2 (J801-B90, VX9_RX2) and AE3 (J801-B129, RTC_32KHZ). no RTC on the CDM.
** ethernet link up failure (see Microchip KSZ9031 errata)

= FPGA MGT blocks =

<pre>
* SFP - ENC B45,B47 and B48,B50 - FPGA D5,D6 MGTHTX0_D and D1,D2 MGTHRX0_D
* QSFP0 TX0 - ENC C13,C17 - FPGA H5,H6 MGTHTX0_C
* QSFP0 TX1 - ENC C21,C25 - FPGA G7,G8 MGTHTX1_C
* QSFP0 TX2 - ENC C29,C23 - FPGA F5,F6 MGTHTX2_C
* QSFP1 TX0 - ENC C37,C41 - FPGA E7,E8 MGTHTX3_C
* QSFP1 TX1 - ENC C45,C47 - FPGA P5,P6 MGTHTX0_B
* QSFP1 TX2 - ENC C51,C53 - FPGA M5,M6 MGTHTX1_B
* QSFP2 TX0 - ENC C57,C59 - FPGA L3,L4 MGTHRX2_B
* QSFP2 TX1 - ENC C63,C65 - FPGA K5,K6 MGTHTX3_B
* QSFP2 TX2 - ENC C75,C77 - FPGA W3,W4 MGTHTX0_A
* QSFP3 TX0 - ENC C79,C81 - FPGA V5,V6 MGTHTX1_A
* QSFP3 TX1 - ENC C85,C87 - FPGA T5,T6 MGTHRX2_A
* QSFP3 TX2 - ENC C89,C91 - FPGA R3,R4 MGTHTX3_A
</pre>

= Clock distribution Rev0 =

Simplified:

* 125 MHz osc -> CLK_XO_125 -> MGTREFCLK0_A -> not used
* 125 MHz osc -> CLK3_XO_125 -> MGTREFCLK1_B -> SFP RX ref clock, QSFP RX and TX ref clock (this is not final design!)
* (disconnected) 125 MHz osc -> CLK2_XO_125 -> FPGA AG8,AH8 (GC)
* 125 MHz osc -> C.C. in1

* C.C. in0 <- CLK_EXT1 (10 MHz GPS clock)
* C.C. in1 <- 125 MHz osc
* (disconnected) C.C. in2 <- CLK_CC_IN <- MGTREFCLK0_D <- SFP RX clock (cannot be used because of uncontrollable phase)
* C.C. in2 <- CLK2_XO_125 <- FPGA AG8,AH8 (GC) <- SFP RX recovered 125 MHz clock
* C.C. in3 <- CLK_FB

* C.C. 125 MHz -> CLK_CC_OUT0 -> MGTREFCLK0_B -> QSFP RX and TX ref clock (final design)
* C.C. 125 MHz -> CLK_CC_OUT1 -> MGTREFCLK1_D -> SFP TX clock
* C.C. 62.5 MHz -> VX clock fanout

proposed changes:
* add C.C. 125 MHz -> new CLK_CC_OUT2 (old CLK2_XO_125) -> FPGA AG8,AH8 (GC)
* change C.C. in2 <- new CLK_CC_IN <- FPGA AK8,AK9 (non-GC)

Complete:

<pre>
125 MHz oscillator - U5 fan out -
q0 -> CLK_XO_125 -> ENC C72,C74 -> FPGA R7,R8 MGTREFCLK0_A (not used)
q1 -> U6 C.C. in1
q2 -> CLK3_XO_125 -> ENC C7,C9 -> FPGA J7,J8 MGTREFCLK1_B -> SFP RX reference clock, QSFP RX and TX reference clocks (not final design!)
q3 -> disconnected on the board, was CLK2_XO_125 -> ENC C151,C153 -> FPGA AG8,AH8 (GC)

U6 C.C (clock cleaner) -

in0 <- CLK_EXT1 (presumably GPS 10 MHz ref clock)
in1 <- 125 MHz oscillator via U5
in2 <- (was: CLK_CC_IN <- ENC B10,B12 <- FPGA D9,D10 MGTREFCLK0_D <- SFP RX clock, 125 MHz)
in2 <- CLK2_XO_125 <- ENC C151,B153 <- FPGA AG8,AH8 (GC) <- mgt_link_data_to_processing.rx_data_clk (SFP RX data clock, 125 MHz)
in3 <- CLK_FB

out0 -> CLK_CCA -> U12 (125 MHz)
out1 -> CLK_CCB -> VX1..6 (62.5 MHz)
out2 -> CLK_CCC -> VX7..12 (62.5 MHz)
out3 -> CLK_FB into in3

CLK_CCA -> U12 (125 MHz fan out) ->

Q0 -> not used
Q1 -> CLK_CC_OUT0 -> ENC C3-5 -> FPGA L7,L8 MGTREFCLK0_B -> QSFP RX and TX reference clocks (final design)
Q2 -> CLK_CC_OUT1 -> ENC B3-5 -> FPGA B9,B10 MGTREFCLK1_D -> SFP TX clock
Q3 -> not used
Q4 -> not used
Q5 -> SMA J9/J10

CLK_TP0 - FPGA N7,N8 MGTREFCLK1_A - ENC C69,C71 - DS-DM SMA J11, J12 (NOT IN CDM PROJECT)

Notes:
* CLK_XO_125 (125 MHz osc) is not used
* 62.5 MHz VX clock does not go into the FPGA
* CLK_EXT0 going to in1 of U5 cannot be used. only permitted frequency is 125 MHz (it drives the MGT reference clocks) and is this frequency is too high for LEMO cables and connectors

Proposed modifications:
- CLK_CC_IN: from FPGA output pin (ENC C142,C144 - FPGA AK8,AK9) to clock cleaner in2 (instead of CLK2_XO_125 pins)
- CLK_CCA -> U12 -> currently unused out3 -> CLK2_XO_125 FPGA pins
- repurpose CLK_EXT0 at GPS 1pps/IRIG input to FPGA
</pre>

= Clock distribution Rev1 =

Simplified:

* 125 MHz osc -> CLK_XO_125 -> MGTREFCLK0_A -> clk_xo_125. not used
* 125 MHz osc -> CLK3_XO_125 -> MGTREFCLK1_B -> CDM SFP RX ref clock
* 125 MHz osc -> C.C. in1

* C.C. in0 <- CLK_EXT1 (10 MHz GPS clock)
* C.C. in1 <- 125 MHz osc
* C.C. in2 <- CLK_CC_IN <- FPGA AK9,AK8 <- CLK_CC_IN1, rx_clk_mgt, SFP RX recovered clock, 125 MHz
* C.C. in3 <- CLK_FB

* C.C. 125 MHz -> CLK_CC_OUT0 -> MGTREFCLK0_B -> GDM QSFP RX and TX ref clock, CDM rx_clk mux
* C.C. 125 MHz -> CLK_CC_OUT1 -> MGTREFCLK1_D -> SFP TX clock
* C.C. 125 MHz -> CLK_CC_OUT2 -> FPGA AG8,AH8 (GC) -> not used
* C.C. 62.5 MHz -> VX clock fanout

Complete:

<pre>
125 MHz oscillator - U5 fan out -
q0 -> CLK_XO_125 -> ENC C72,C74 -> FPGA R8,R7 MGTREFCLK0_A -> XDC CLK_XO_125_P -> VHDL not used
q1 -> U6 C.C. in1
q2 -> CLK3_XO_125 -> ENC C7,C9 -> FPGA J8,J7 MGTREFCLK1_B -> XDC GDM missing, CDM CLK3_XO_125_P -> VHDL SFP RX reference clock (mgt_rx_ref_clk)
q3 -> not used, was CLK2_XO_125

U6 C.C (clock cleaner) -

in0 <- CLK_EXT1 (GPS 10 MHz clock)
in1 <- 125 MHz oscillator via U5
in2 <- CLK_CC_IN <- ENC C142,C144 <- FPGA AK9,AK8 <- XDC GDM missing, CDM CLK_CC_IN1 <- VHDL rx_clk_mgt
in3 <- CLK_FB

out0 -> CLK_CCA -> U12 (125 MHz)
out1 -> CLK_CCB -> VX1..6 (62.5 MHz)
out2 -> CLK_CCC -> VX7..12 (62.5 MHz)
out3 -> CLK_FB into in3

CLK_CCA -> U12 (125 MHz fan out) ->

Q0 -> not used
Q1 -> CLK_CC_OUT0 -> ENC C3,C5 -> FPGA L8,L7 MGTREFCLK0_B -> XDC GDM clk_mgtrefclk0_x0y1_p, CDM i_CLK_CC_OUT0_P -> VHDL GDM mgt_b_ref_clk QSFP RX and TX reference clocks, CDM alternate rx_clk via clock mux
Q2 -> CLK_CC_OUT1 -> ENC B3,B5 -> FPGA B10,B9 MGTREFCLK1_D -> XDC i_CLK_CC_OUT1_P -> VHDL GDM not used, CDM mgt_tx_ref_clk SFP TX clock
Q3 -> CLK_CC_OUT2 -> ENC C151,C153 -> FPGA AG8,AH8 (GC) -> XDC GDM missing, CDM i_CLK_CC_OUT2_P
Q4 -> not used
Q5 -> SMA J9/J10

CLK_TP0 <-> ENC C69,C71 <-> FPGA N8,N7 MGTREFCLK1_A (not used) -> XDC CLK_TP0_P -> VHDL not used (DS-DM SMA J11, J12)

disconnected <- OLD_CLK_CC_IN <- ENC B10,B12 <- FPGA D9,D10 MGTREFCLK0_D <- XDC o_CLK_CC_IN <- VHDL CDM o_OLD_CLK_CC_IN SFP o_mgt_rx_rec_clk

Notes:
* Enclustra ENC-B is J801, ENC-C is J900
* "(GC)" is clock-capable FPGA pin
* CLK_XO_125 (125 MHz osc) is not used in FPGA
* 62.5 MHz VX clock does not go into the FPGA
* CLK_EXT0 going to in1 of U5 cannot be used. only permitted frequency is 125 MHz (it drives the MGT reference clocks) and is this frequency is too high for LEMO cables and connectors
</pre>

= I2C bus =

* I2C_SCL is J-ENC-A pin 111 I2C_SCL is FPGA I2C_SCL_PL AB13 (IO_L1N_TOL1D_64) and I2C_SCL_PS F18 (PS_MIO10)
* I2C_SDA is J-ENC-A pin 113 I2C_SDA is FPGA I2C_SDA_PL AH13 (IO_L7N_T1L1Q_AD13N_64) and I2C_SDA_PS G18 (PS_MIO11)

* XU8 secure EEPROM ATSHA204A at 0x64, this is 0110'010X -> linux _011'0010 is 0x32. (but responds to scan and read at 0x33)
* U4 ethernet mac chip, EEPROM at 1010 A2 A1 A0 X and MAC/serial_no at 1011 A2 A1 A0 X. A0=VCC, A1=VCC, A2=GND -> linux _101'0011 and _101'1011 is 0x53 and 0x5B.
* U6 clock chip, address 1101 0 A1 A0 X. A1=VCC, A0=N/C (internal pull-up) -> linux _110'1011 is 0x6b
* U23 voltmeter at 10011 ADR1 ADR0. ADR0=GND, ADR1=VCC -> linux _100'1110 is 0x4e
* SFP, address 1010000X -> linux _101'0000 is 0x50. additional SFP data at 0x51
* QSFP0, QSFP1, QSFP2, QSFP3 (QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL)

= I2C clock builder connection =

* use Silicon Labs USB "Clock builder pro field programmer", www.silabs.com/CBProgrammer
* connect rainbow jumper cable pins:
** black - 1-GND to GND on DS-DM
** white - 3-SCLK to SCL on the DS-DM
** grey - 7-SDA_SDIO to SDA on the DS-DM
* power up the DS-DM
* plug USB programmer into Windows laptop
* on Windows, run "ClockBuilder Pro"
* it should report "Field programmer detected", press "EVB GUI"
* in EVB GUI, press "Config", set I2C address 0x6B
* press "Scan", it should find Si5394A-A-GM
* select the "Status" tab, should see real-time status of clock chip

= GDM MGT configuration =

* TX configuration:
* GDM MGT transceivers are configured as "multilane" TX and RX.
* there is 12 TXes ("lanes")
* MGT reference 125 MHz clock goes into [2:0]gtreclk00_in and [11:0]gtrefclk0_in
* one MGT is designated as "master"
* PLL of master MGT converts reference clock into common TX clock and common tx_user_clk2 which becomes tx_data_clk
* common tx_user_clk2 aka tx_data_clk goes into all TXes and clocks tx_data.
* tx_user_clk2 aka tx_data_clk is 125 MHz but not same phase as MGT reference clock.

* RX interim configuration:
* there is 12 RXes ("lanes")
* each RX produces it's own recovered RX clock
* "multilane" configuration assumes all RX recovered clocks run at the same frequency (TX on the other end are driven by common TX clock, see above), but have different phase
* one RX recovered clock is designated as "master" (rx_user_clk2 aka rx_data_clk) and a phase-matching fifo/buffer is used to bring rx_data from all 12 RXes to this common rx_data_clk
* this works because each CDM SFP TX runs on the SFP RX recovered clock which is frequency-locked with the GDM QSFP TX clock.

* RX final configuration:
* MGTs permit using the common TX clock (tx_user_clk2 aka tx_data_clk) as the common rx_data_clk (they are frequency locked through the CDM).
* this permits use of tx_data_clk as the main clock domain in the GDM and removes the need to bring rx_data into the tx_data_clk domain (actually this is done in the MGT RX phase matching fifo/buffer).

= Clock path =

NOTE: MUST REVIEW!!!

<pre>
10 MHz ext clock or GDM 125 MHz oscillator
-> GDM QSFP MGT reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz (GDM main clock domain) and TX bit clock 2.5 GHz
-> GDM QSFP optic transmitter
-> CDM SFP optic receiver
-> CDM SFP MGT, RX reference clock is CDM 125 MHz oscillator
-> MGT RX recovered clock 125 MHz (CDM main clock domain)
-> CC_CLK_IN -> CDM C.C. -> CC_CLK_OUT1 -> CDM SFP TX reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz and TX bit clock 2.5 GHz
-> (tx_data phase matching fifo from CDM main clock domain to tx_data_clk)
-> CDM SFP optic transmitter
-> GDM QSFP RX optic receiver (12x)
-> GDM QSFP MGT (RX reference clock is same as TX reference clock)
-> MGT RX recovered clock (12x recovered clocks)
-> in multi-lane configuration, one of them is the "master" recovered clock rx_data_clk
-> (rx_data phase matching fifo from rx_data_clk to GDM main clock domain)
</pre>

= CDM rx_clk mux =

when CDM SFP is not connected, there is no SFP recovered clock and a mux is used to switch between clk_cc_out0 (power up default) and rx_clk_mgt (SFP recovered clock)

== Test SFP disconnected ==

note: if I say "--cc-in1", CC seems to lock on the 10 MHz GPS external clock,
to prevent this, test sequence includes reloading the CC and the reset of MGT.

<pre>
./test_cdm_local.exe --write32 0x30 0x0 ### rx_clk mux select CC clock
./test_cdm_local.exe --load-cc
./test_cdm_local.exe --reset-mgt
./test_cdm_local.exe --cdm-clocks
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773594e (125000014) should be ~125 MHz <=== all 4 clocks wobble close to 125 MHz
0x1034 rx_clk: 0x0773594f (125000015) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x0773594f (125000015) should be ~125 MHz
0x103C tx_clk: 0x0773594f (125000015) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
./test_cdm_local.exe --write32 0x30 0x1 ### rx_clk mux select SFP recovered clock
./test_cdm_local.exe --cdm-clocks
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773598b (125000075) should be ~125 MHz
0x1034 rx_clk: 0x076d58ec (124606700) should be ~125 MHz <=== off frequency because there is no valid SFP recovered clock
0x1038 mgt_tx_ref_clk_raw: 0x0773598c (125000076) should be ~125 MHz
0x103C tx_clk: 0x0773598b (125000075) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

= Clock domains =

== GPS ==

* no GPS : GDM runs from internal 125 MHz oscillator
* external 10 MHz clock : GDM runs from external 10 MHz clock and optional 1pps signal (use VME-NIMIO32 NIM outputs)
* GPS receiver : GDM runs from GPS 10 MHz clock and GPS IRIG serial data
* LNGS GPS:
** provides 1pps and serial data over fiber from GPS receiver (master). LNGS xxx box is not used.
** serial data goes to GDM, decoded, 1pps signal extracted, goes to LEMO output
** 1pps from GDM is used to train the Rubidium clock which provides a 10 MHz clock
** 10 MHz output from Rubidium clock goes to GDM 10 MHz external clock input

== GDM ==

* AXI clock (100 MHz) - AXI registers
* 125 MHz oscillator - to clock cleaner
* 10 MHz external clock LEMO input - to clock cleaner
* FPGA 125 MHz clock CLK_CC_IN - to clock cleaner - not used (could be used for cascaded GDMs)
* 125 MHz oscillator - CLK3_XO_125 - mgt_b_ref_clk QSFP MGT reference clock (interim GDM design)
* clock cleaner output 125 MHz fanout:
** CLK_CC_OUT0 - QSFP MGT reference clock (final design)
** CLK_CC_OUT1 - not used (CDM SFP reference clock)
** CLK_CC_OUT2 - not used
* QSFP MGT TX data clock 125 MHz tx_data_clk (main clock domain)
** QSFP TX data
** QSFP RX data (in final design, rx_data_clk is same as tx_data_clk. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk == tx_data_clk)
** ds20k block
* (in interim design, rx_data_clk is disconnected from tx_data_clk. rx_data_clk is the "multilane master clock" which is one of the 12x RX recovered clocks. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk)

note: all these clocks are frequency locked to 125 MHz

== CDM ==

* AXI clock (100 MHz) - AXI registers
* 10 MHz external clock LEMO input - to clock cleaner (not used)
* 125 MHz oscillator to fanout
** to clock cleaner
** to SFP MGT RX reference clock: CLK3_XO_125 to mgt_rx_ref_clk to gtrefclk01_in
* SFP MGT RX recovered clock 125 MHz
** MGT PLL to MGT rx_user_clk2 aka rx_data_clk
** MGT rxrecclkout_out to mgt_rx_rec_clk to CLK_CC_IN to C.C.
* SFP MGT rx_user_clk2 aka rx_data_clk (250 MHz/8 bit, 125 MHz/16 bit, 62.5 MHz/32 bit data) (main clock domain)
** SFP RX data
** ds20k block
** VX TX clock PLLs
** VX RX clock PLLs
* C.C. fan out
** 62.5 MHz VX clocks (12x)
** CLK_CC_OUT0 (not used, GDM QSFP MGT reference clock)
** CLK_CC_OUT1 125 MHz to mgt_tx_ref_clk to gtrefclk00_in to SFP MGT TX reference clock
** CLK_CC_OUT2 (not used)
* SFP MGT tx_user_clk2 aka tx_data_clk
** SFP TX data
** TX data phase matching fifo from main clock domain to tx_data_clk
* VX TX clock PLLs
** 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
** VX TX data phase matching from main clock domain to VX TX clock (12 total)
** VX TX serializer
** VX TX LVDS transmitter
* VX RX clock PLLs
** 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
** VX RX LVDS receivers (12 total)
** VX RX deserializers (12 total)
** VX TX data phase matching from VX RX clock to main clock domain

== VX ==

* everything runs on the VX main 125 MHz clock
* correct phase of VX to CDM LVDS data is adjusted by scan of CDM VX RX PLL clock phase (VX to CDM link is now established)
* correct phase of CDM to VX LVDS data is adjusted by scan of CDM VX RX PLL clock phase (link is established after VX to CDM idle data pattern changes from "VX RX data bad" to "good".
* after good phases are found by scan, they are not expected to change unless cables are changed, CDM and VX modules are changed or CDM firmware is rebuilt. (rebuild of VX firmware should not affect LVDS data phase).
* if there is excessive link errors, phase scan must be repeated.

= Board test plan =

To test:
* Enclustra FPGA board
* SFP port - SFP_RS0/RS1 connected to QSFP0_SEL/1_SEL is wrong? our Finisar SFP says RS0, RS1 N/C, so probably okey.
* SW5 CLK_EXT1 NIM works. TTL needs to be tested. flipping SW5 CLK_EXT0 side from NIM to TTL makes CLK_EXT1 go LOS and OOF in the clock chip. R23 and R53 should be removed?
* BOOT_MODE 0 and 1

Partial:
* U23 3.3V current meter and thermometer. V1,V2 is current monitor, same as in application note. V3,V4 is thermometer, same in application note, except capacitor C118 is 0.1u instead of 470pF. Tested ok: Tint, VCC, V1, V2, V1-V2. Test failed: TR2 reads 50-something degC instead of same as Tint.

Done:
* LED_FP1A..D: tested ok. K.O. 15 sep 2022
* USB UART: tested ok. K.O. 15 sep 2022
* J4A, J4B, J5A, J5B LEMO inputs (NIM/TTL) EXT_IN_LV(1..4). TTL threshold 1.7V, NIM threshold -0.3V. 50 Ohm termination. TTL straight, NIM inverted.
* J6A, J6B LEMO clock inputs: 10 MHz TTL from chronobox works, 10 MHz NIM from IO32 works. 18-apr-2023
* J7A, J7B LEMO outputs EXT_OUT(1), EXT_OUT(2) (NIM/TTL) (tested 23nov2022, K.O.). TTL ???, NIM inverted.
** TTL out no 50 ohm termination: 0=0V, 1=5V, rise and fall time ~5 ns
** TTL out with 50 ohm termination: 0=0V, 1=2.5V, rise and fall time <2ns
** NIM out no 50 ohm termination: 0=+50mV, 1=-1.8V, rise and fall time ~3ns
** NIM out with 50 ohm termination: 0=0V, 1=-0.9V, rise and fall time <2ns
* ethernet MAC i2c chip (K.O. 20sep2022: can read: i2cdump 0 0x5b, nov2022 have u-boot driver)
* CDM VX ports 2x(CLK, 3 tx, 4 rx) tested using test_cdm.exe and LVDS loopback in VX firmware.
* SFP i2c tested KO 22jun2023
* QSFP i2c tested KO 22jun2023
* i2c testing complete 22jun2023
* QSFP rx,tx tested 26june2023. lane0,1,2 ok, lane3 tx not connected, laser is off. qsfp0,1,2,3 all lanes ok.
* SFP rx,tx tested 26june2023. rx and tx okey. LOS ok, mod_absent ok.

Failure:
* ethernet: does not connect to alliedtelesys switch. connects to my USB-eth adapter at 100 Mbit speed. uboot mii status reports connection speed oscillating between 1000, 100 and 10. K.O. 16-sep-2022
* SFP LOS and mod_absent are swapped (in the FPGA pin definitions?)
* QSFP slot numbering is wrong.

= Checklist for newly build boards =

== setup and boot ==
* put new board on workbench
* check - vme connector present, vme extraction handles present
* check - standoff are removed from all thru-holes
* plug Enclustra module
* check - SW1 both switches are in the "6" and "3" positions
* check - SW6 both switches are in the "PS" position
* Rev0: connect micro-usb cable from linux PC
* Rev1: connect usb-C cable from linux PC (USB C-to-C or C-to-A)
* connect ethernet from 1gige capable network switch
* connect power from lab power supply - GND, +5V and -12V nominal, +5.7V and -12.5V actual
* power up, +5V current 2.10-2.8A, -12V current 0.05A
* on linux PC, open a new terminal, run: minicom -D /dev/ttyACM0 -b 115200
* in minicom window, observe messages about Xilinx first stage boot loader, etc
* on the ethernet switch, observe network link is 1gige speed (not 10mbit, not 100mbit).
* if everything boots okey, there will be a login prompt, login as root, password root.
* alternatively, from root@daq13: ssh root@dsdm
* busybox devmem 0x80010000 # read firmware version number, i.e. 0xEDAD0A77
== test i2c ==
* check i2c: (no SFP, no QSFP plugged in!)
<pre>
root@dsdm:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@dsdm:~#
</pre>
* test clock chip:
<pre>
root@dsdm:~# ./test_cdm_local.exe --load-cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Loading CC registers...
Clock chip data block size 519
root@dsdm:~# ./test_cdm_local.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Polling CC status...
Clock chip state 1, status: LOS_IN0 OOF_IN0 IN1 IN_SEL_1
Ctrl-C
</pre>
* test RTC:
<pre>
root@dsdm:~# ./test_cdm_local.exe --rtc-read
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Read the RTC chip!
reg 0x00: 0 0 1 0 0 1 1 0 0x26
reg 0x01: 0 0 1 1 1 0 0 1 0x39
reg 0x02: 0 0 0 0 0 1 1 1 0x07
reg 0x03: 0 0 0 0 0 0 1 0 0x02
reg 0x04: 0 0 0 0 0 0 1 0 0x02
reg 0x05: 0 0 0 0 0 0 0 1 0x01
reg 0x06: 0 0 0 0 0 0 0 0 0x00
reg 0x07: 0 0 0 0 0 0 0 0 0x00
reg 0x08: 0 0 0 0 0 0 0 0 0x00
reg 0x09: 0 0 0 0 0 0 0 0 0x00
reg 0x10: 0 0 0 0 0 0 0 0 0x00
reg 0x11: 0 0 0 0 0 0 0 0 0x00
reg 0x12: 0 0 0 0 0 0 0 0 0x00
reg 0x13: 0 0 0 0 0 0 0 0 0x00
reg 0x14: 0 0 0 1 1 0 0 0 0x18
reg 0x15: 0 0 0 0 1 0 0 0 0x08
reg 0x16: 0 0 0 0 0 0 0 0 0x00
reg 0x17: 0 0 0 1 0 1 1 1 0x17
reg 0x18: 0 1 0 0 0 0 0 0 0x40
seconds: 26, minutes: 39, hours: 07, day 2, date: 00-01-02, temp 23.25C
root@dsdm:~#
</pre>

== test clocks ==

<pre>
root@dsdm:~# ./test_cdm_local.exe --reset-mgt
...
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735cb9 (125000889) should be ~125 MHz
0x1034 rx_clk: 0x07735cb9 (125000889) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735cb9 (125000889) should be ~125 MHz
0x103C tx_clk: 0x07735cb9 (125000889) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
^C
root@dsdm:~# ./test_cdm_local.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735c6a (125000810) should be ~125 MHz
0x1034 rx_clk: 0x07735c69 (125000809) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735c69 (125000809) should be ~125 MHz
0x103C tx_clk: 0x07735c6a (125000810) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
^C
root@dsdm:~#
</pre>

== test LED, LEMO ==

* test LEDs, observe all 4 LEDs turn on and off every 1 second
<pre>
root@dsdm:~# ./test_cdm_local.exe --blink-led
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
LED on
LED off
</pre>
* TBW - test LEMO outputs
<pre>
have test firmware lemo outputs are RTC chip 1pps and 48kHz
see them on the scope
</pre>
* TBW - test LEMO inputs
<pre>
./test_cdm_local.exe --monreg 6
</pre>

== test VX ==

* on the CDM, connect all split blue cables to VX modules
* set all VX modules to LVDS loopback mode (in VX settings MIDAS page)
* ssh root@cdm00
* run "./test_cdm.exe --test-vx", it should report:
<pre>
CDM registers: 0x0000000000000000 0x00000000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000eeeeeeeeeeee 0x00007700: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000080808080808 0x00000100: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000040404040404 0x00000200: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000020202020202 0x00000400: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000000000000000 0x00000800: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000808080808080 0x00001000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000404040404040 0x00002000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000202020202020 0x00004000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000000000000000 0x00008000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
all on!
all off!
</pre>
* on the VX settings MIDAS page LVDS input section, observe line status is cycling on and off.
* note: VX_RX(0) is not tested by this, corresponding VX_TX is used as the VX clock

== test SFP (CDM) ==

* connect SFP module
* ssh root@cdm00
* run "./test_cdm.exe --sfp", it should report:
<pre>
root@cdm00:~# ./test_cdm.exe --sfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x6a332b58, ds20k version 0x20241209
DS-DM firmware build 0x6a332b58, ds20k version 0x20241209
Polling SFP status...
identifier 0x03
connector 0x07
encoding 0x01
wavelength 0x0352 (850 nm)
vendor_name [FINISAR CORP. ]
vendor_pn [FTLF8526P3BNL ]
vendor_rev [A ]
vendor_sn [N3AB4LV ]
vendor_date [200319 ]
dm_type 0x68
temp 30.9 C
vcc 3.285 V
tx_bias 7.362 mA
tx_power 475.7 uW
rx_power 859.1 uW
SFP good 1, status: temp 30.9 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 859 uW
SFP good 1, status: temp 30.8 C, tx_bias 7.4 mA, tx_power 475 uW, rx_power 859 uW
SFP good 1, status: temp 30.8 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 859 uW
SFP good 1, status: temp 30.7 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 859 uW
SFP good 1, status: temp 30.8 C, tx_bias 7.4 mA, tx_power 475 uW, rx_power 858 uW
^C
root@cdm00:~#
</pre>
* NOTE1: if fiber cable is disconnected or GDM is not running, rx_power would read 0 uW
* NOTE2: better SFP modules have smaller tx_bias and larger tx_power (more bang for the buck)
* NOTE3: bad fiber cable and bad fiber connection will result in smaller rx_power

== test QSFP (GDM) ==

* connect QSFP module (any slot)
* ssh root@gdm01
* run "./test_cdm.exe --qsfp3 --qsfp", try --qsfp0, --qsfp1, --qsfp2, --qsfp3 for the 4 QSFP slots
<pre>
root@gdm01:~# ./test_cdm.exe --qsfp3 --qsfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x9e30baf9, ds20k version 0x20241209
DS-DM firmware build 0x9e30baf9, ds20k version 0x20241209
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out hi
Polling QSFP status...
identifier 0x0d
status 0x02
los 0x89
temp 36.7 C
vcc 3.302 V
rx_power 0.1 413.7 460.3 0.1 uW
tx_bias 7.8 7.9 7.9 0.0 mA
tx_power 806.0 842.9 865.2 0.1 uW
vendor_name [FINISAR CORP ]
vendor_pn [FTL410QD4C ]
vendor_rev [A ]
wavelength 850
max_temp 70 C
vendor_sn [X79AC0R ]
vendor_date [220309 ]
QSFP good 1, status: temp 36.7 C, los 0x89, tx_bias 7.8 7.9 7.9 0.0 mA, tx_power 806 843 865 0 uW, rx_power 0 414 460 0 uW
^C
</pre>
* NOTE1: 4th tx_bias and tx_power are zero because 4th channel does not transmit any data, QSFP detects this and automatically shuts down the optical transmitter

= Serial console =

* check that linux computer has correct udev rules to allow access to /dev/ttyACM devices, see https://daq00.triumf.ca/DaqWiki/index.php/SLinstall#Configure_USB_device_permissions and https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Configure_USB_device_permissions
* connect micro-USB cable to connector J-UCB, other end connect to linux computer
* observe /dev/ttyACM0 was created
* run "minicom -D /dev/ttyACM0 -b 115200"
* should have gdm-cdm login
* username root, password root

= i2c =

<pre>
ZynqMP> i2c bus
Bus 0: i2c@ff020000
ZynqMP> i2c dev 0
Setting bus to 0
ZynqMP> i2c probe
Valid chip addresses: 33 4E 53 5B 6B 77
ZynqMP> i2c md 0x5b 0x98
0098: fc c2 3d 00 00 12 1a 6e 0a 90 85 04 94 10 08 50 ..=....n.......P
</pre>

<pre>
root@gdm-cdm:~# i2cdetect 0
Warning: Can't use SMBus Quick Write command, will skip some addresses
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0.
I will probe address range 0x03-0x77.
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f
00:
10:
20:
30: -- -- -- 33 -- -- -- --
40:
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60:
70:
root@gdm-cdm:~#
root@gdm-cdm:~# i2cdump 0 0x5b
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0, address 0x5b, mode byte
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00 ???????P?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e ........??=..??n
a0: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00 ???????P?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e ........??=..??n
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@gdm-cdm:~#
</pre>

<pre>
root@gdm0:~# i2cdetect -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#
</pre>

<pre>
root@cdm0:~# i2cdetect -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: 50 51 -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@cdm0:~#
</pre>

* 0x33 - XU8 secure EEPROM (should be at 0x32)
* 0x4e - U23 current and temperature monitor
* 0x50, 0x51 - SFP
* 0x50 - QSFP, 4 QSFP modules enabled by GPIO QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL
* 0x53, 0x5b - ethernet mac eeprom
* 0x6b - U6 clock chip

== U23 ==

NOTE: Text reads double of Tint. not sure why. K.O. 21-mar-2024.

* internal temperature only

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x00 b # control register: "repeat mode, internal temperature only"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x03 # "Tint ready" and "busy", "busy is always 1 in repeat mode"
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 00 03 03 81 db 2a ce 2a 8e 00 6f 00 45 20 3f ?.????*?*?.o.E ?
10: 01 00 01 01 01 db 2a ce 2a 8e 00 6f 00 45 20 3f ?.????*?*?.o.E ?
...
readback:
reg0 - 03 - Tint ready
reg1 - 00 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB 0x81, bit 0x80 is "DV, data valid", bit 0x40 is "SS, sensor short", 0x20 is "SO, sensor open"
reg5 - Tint LSB 0xDB, Tint = 0x01DB = 475 * 0.0625 degC = 29.6 degC
reg6..F - stale data
</pre>

* Tint, V1, V2, TR2, VCC

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x18 b # control register: "repeat mode, V1, V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 7f 18 7f 7f 81 b9 aa d0 aa 8f 83 5e 83 5e a0 41 ???????????^?^?A
10: 01 18 01 01 01 b9 2a d0 2a 8f 03 5e 03 5e 20 41 ??????*?*??^?^ A
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x1b9*0.0625 = 27.5 degC
reg6 - V1 MSB 0xaa, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 LSB 0xd0, V1 = 0x2ad0*305.18/1000000 = 3.3447 V (correct)
reg8 - V2 MSB 0xaa, ditto
reg9 - V2 LSB 0x8f, V2 = 0x2a8f*305.18/1000000 = 3.3249 V (correct, smaller than V1)
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x5e. TR2 = 0x35e*0.0625 = 53.875 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x41, VCC = 2.5+0x2041*305.18/1000000 = 5.019 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.99 A (about right?)
</pre>

* Tint, V1-V2, TR2, VCC

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x19 b # control register: "repeat mode, V1-V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 7f 19 7f 7f 81 69 83 f2 83 f2 83 3a 83 3a a0 44 ?????i?????:?:?D
10: 01 19 01 01 01 69 03 f2 03 f2 03 3a 03 3a 20 44 ?????i?????:?: D
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x169*0.0625 = 22.5 degC
reg6 - V1 or V1-V2 MSB 0x83, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 or V1-V2 LSB 0xf2, V1-V2 = 0x3f2*19.42/1000000 = 0.0196142 V (correct, compare with V1 and V2 measured above)
reg8 - V2 or V1-V2 MSB
reg9 - V2 or V1-V2 LSB
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x3a. TR2 = 0x33a*0.0625 = 51.625 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x44, VCC = 2.5+0x2044*305.18/1000000 = 5.021 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.9807 A (about right?)
</pre>

== SFP ==

<pre>
root@cdm0:~# i2cdump 0 0x50
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 04 07 00 00 00 00 40 40 0c 00 01 3d 00 00 00 ???....@@?.?=...
10: 0c 02 00 1e 46 49 4e 49 53 41 52 20 43 4f 52 50 ??.?FINISAR CORP
20: 2e 20 20 20 00 00 90 65 46 54 4c 46 38 35 32 36 . ..?eFTLF8526
30: 50 33 42 4e 4c 20 20 20 41 20 20 20 03 52 00 9d P3BNL A ?R.?
40: 00 1a 00 00 4e 33 41 42 34 4c 56 20 20 20 20 20 .?..N3AB4LV
50: 20 20 20 20 32 30 30 33 31 39 20 20 68 f0 03 de 200319 h???
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@cdm0:~#
root@cdm0:~# i2cdump 0 0x51
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 5a 00 d8 00 55 00 e2 00 90 88 71 48 8c a0 75 30 Z.?.U.?.??qH??u0
10: 21 34 01 f4 1b 58 03 e8 31 2d 04 eb 1f 07 06 31 !4???X??1-?????1
20: 31 2d 00 64 27 10 00 9e 00 00 00 00 00 00 00 00 1-.d'?.?........
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
40: 00 00 00 00 3f 80 00 00 00 00 00 00 01 00 00 00 ....??......?...
50: 01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 b7 ?...?...?......?
60: 19 9f 80 c5 0e 17 12 c4 1f 99 00 00 00 00 30 00 ??????????....0.
70: 00 00 00 00 00 00 00 00 ff ff ff ff ff ff ff 01 ...............?
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@cdm0:~#
</pre>

== QSFP ==

<pre>
QSFP i2c enable lines, active low:

QSFP0_SEL - JENC-A 82 - PS-MIO40 - linux gpio 378
QSFP1_SEL - JENC-A 84 - PS-MIO41 - linux gpio 379
QSFP2_SEL - JENC-A 100 - PS-MIO44 - linux gpio 382
QSFP3_SEL - JENC-A 106 - PS-MIO43 - linux gpio 381 (notice 2 and 3 are out of order)
</pre>

<pre>
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
# echo 378 >> /sys/class/gpio/export ### SEL0 338+40
# echo 379 >> /sys/class/gpio/export ### SEL1 338+41
# echo 381 >> /sys/class/gpio/export ### SEL3 338+43
# echo 382 >> /sys/class/gpio/export ### SEL2 338+44
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) in hi
gpio-379 ( |sysfs ) in hi
gpio-381 ( |sysfs ) in hi
gpio-382 ( |sysfs ) in hi
root@gdm0:~# echo out >> /sys/class/gpio/gpio381/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio382/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio378/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio379/direction
root@gdm0:~#
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out lo
gpio-379 ( |sysfs ) out lo
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out lo
root@gdm0:~#
echo 1 >> /sys/class/gpio/gpio381/value
echo 1 >> /sys/class/gpio/gpio382/value
echo 1 >> /sys/class/gpio/gpio378/value
echo 1 >> /sys/class/gpio/gpio379/value
cat /sys/kernel/debug/gpio
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out hi
gpio-382 ( |sysfs ) out hi
root@gdm0:~#
root@gdm0:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#

NOTICE NOTHING AT ADDRESS 0x50
</pre>

<pre>
ENABLE QSFP0, OBSERVE IT IS AT ADDRESS 0x50

root@gdm0:~# echo 0 >> /sys/class/gpio/gpio378/value
root@gdm0:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: 50 -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#
root@gdm0:~# i2cdump -y 0 0x50
No size specified (using byte-data access)
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 0d 00 02 8e 00 00 01 00 00 05 55 00 05 00 00 00 ?.??..?..?U.?...
10: 00 00 00 00 00 00 1e b8 00 00 81 a0 00 00 00 00 ......??..??....
20: 00 00 12 12 00 01 00 01 00 01 0e e0 0f 20 0e e0 ..??.?.?.???? ??
30: 00 00 1f a0 1e 3b 1f 72 00 01 00 00 00 00 00 00 ..???;?r.?......
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
60: 00 00 00 00 00 00 00 00 00 00 1f 00 00 00 08 00 ..........?...?.
70: 00 00 00 00 00 00 00 ff ff ff ff ff ff ff ff 00 ................
80: 0d 00 0c 04 00 00 00 40 40 02 d5 05 67 00 00 96 ?.??...@@???g..?
90: 00 00 c8 00 46 49 4e 49 53 41 52 20 43 4f 52 50 ..?.FINISAR CORP
a0: 20 20 20 20 07 00 90 65 46 54 4c 34 31 30 51 44 ?.?eFTL410QD
b0: 34 43 20 20 20 20 20 20 41 20 42 68 07 d0 00 43 4C A Bh??.C
c0: 00 07 0f de 58 37 39 41 43 30 52 20 20 20 20 20 .???X79AC0R
d0: 20 20 20 20 32 32 30 33 30 39 20 20 3c 10 00 9e 220309 <?.?
e0: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@gdm0:~#
</pre>

== ethernet mac eeprom ==

* correct chip with 84-bit ethernet mac address

<pre>
root@cdm1:~# i2cdump 0 0x53
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
root@cdm1:~#

root@cdm1:~# i2cdump 0 0x5b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00 ????4??2?K?.?...
90: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c ..........??=?Q<
a0: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00 ????4??2?K?.?...
b0: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c ..........??=?Q<
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@cdm1:~#
</pre>

* wrong "602" chip with 64-bit IPv6 address

<pre>
root@cdm0:~# i2cdump 0 0x53
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
root@cdm0:~# i2cdump 0 0x5b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00 ???????Q?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e ........??=..??.
a0: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00 ???????Q?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e ........??=..??.
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@cdm0:~#
</pre>

= read ethernet mac address from i2c =

(this code is copied from uboot command line i2c code)

in uboot sources board/xilinx/common/board.c replace original function with this:

<pre>
// special code to read ethernet MAC address from the DS-DM-Rev0 board. K.O. Sep-2022

int zynq_board_read_rom_ethaddr(unsigned char *ethaddr)
{
struct udevice *bus;
int ret;
int busnum = 0;

ret = uclass_get_device_by_seq(UCLASS_I2C, busnum, &bus);
if (ret) {
printf("%s: No bus %d\n", __func__, busnum);
return ret;
}

int chip_addr = 0x5B;

struct udevice *dev;

ret = i2c_get_chip(bus, chip_addr, 1, &dev);
if (ret) {
printf("%s: Bus %d no chip 0x%02x\n", __func__, busnum, chip_addr);
return ret;
}

int dev_addr = 0x98;

unsigned char data[8];

ret = dm_i2c_read(dev, dev_addr, data, 8);

if (ret) {
printf("%s: Bus %d chip 0x%02x read error %d\n", __func__, busnum, chip_addr, ret);
return ret;
}

printf("%s: Bus %d chip 0x%02x addr 0x%02x read: 0x %02x %02x %02x %02x %02x %02x %02x %02x\n", __func__, busnum, chip_addr, dev_addr, data[0], data[1], data[2], data[3], data[4], data[5], data[6], data[7]);

// see Atmel-8807-SEEPROM-AT24MAC402-602-Datasheet.pdf

if (data[0] == 0) {
// eiu-48 chip
ethaddr[0] = data[2];
ethaddr[1] = data[3];
ethaddr[2] = data[4];
ethaddr[3] = data[5];
ethaddr[4] = data[6];
ethaddr[5] = data[7];
} else {
// eiu-64 chip
ethaddr[0] = data[0];
ethaddr[1] = data[1];
ethaddr[2] = data[2];
ethaddr[3] = data[5];
ethaddr[4] = data[6];
ethaddr[5] = data[7];
}

printf("%s: ethaddr %02x:%02x:%02x:%02x:%02x:%02x\n", __func__, ethaddr[0], ethaddr[1], ethaddr[2], ethaddr[3], ethaddr[4], ethaddr[5]);

return ret;
}
</pre>

also this should have worked if i2c_xxx() functions were enabled in uboot:
<pre>
i2c_set_bus_num(0);
i2c_probe(0x5b);
i2c_read(0x5b, 0x9a, ethaddr, 6);
</pre>

= read ethernet mac address from i2c (SHOULD WORK) =

from: https://stackoverflow.com/questions/43637540/is-there-linux-or-u-boot-support-to-read-a-mac-address-from-a-chip-at-startup

<pre>
#ethernet related setup
setup_eth=run readmac buildmac
#read mac address from eeprom
readmac=i2c dev 0; i2c read 50 FA.1 6 $loadaddr
#build the ethaddr variable
#not very nice, but does the job
buildmac=\
e=" "; sep=" " \
for i in 0 1 2 3 4 5 ; do\
setexpr x $loadaddr + $i\
setexpr.b b *$x\
e="$e$sep$b"\
sep=":"\
done &&\
setenv ethaddr $e
</pre>

= read ethernet mac address from i2c (DOES NOT WORK) =

this method does not work: inside board/xilinx/common/board.c:int zynq_board_read_rom_ethaddr(unsigned char *ethaddr), uclass_get_device_by_ofnode(UCLASS_I2C_EEPROM, eeprom, &dev) returns ENODEV and read from i2c does not happen. K.O. Sep-2022

Read:
* https://support.xilinx.com/s/question/0D52E00006hpKlsSAE/uboot-and-ethernet-mac-address-from-eeprom?language=en_US (last answer)
* https://github.com/Xilinx/u-boot-xlnx/blob/master/board/xilinx/common/board.c (zynq_board_read_rom_ethaddr())

Note:
* 0x5B is the i2c chip address
* 0x9A is the data offset inside the chip, see datasheet or i2c read dump above.

Edit:
* emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/u-boot/files/platform-top.h
<pre>
#include <configs/xilinx_zynqmp.h>
#include <configs/platform-auto.h>
//#define CONFIG_I2C_EEPROM
//#define CONFIG_SYS_I2C_EEPROM_ADDR 0x5b
//#define CONFIG_SYS_I2C_EEPROM_ADDR_OVERFLOW 0x0
#define CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET 0x9A
#error HERE!
</pre>
* emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi
<pre>
/include/ "system-conf.dtsi"
/ {
chosen {
xlnx,eeprom = &eeprom;
};
};

&i2c0 {
eeprom: eeprom@5b { /* u88 */
compatible = "atmel,24mac402";
reg = <0x5b>;
};
};
</pre>
* components/yocto/workspace/sources/u-boot-xlnx/configs/xilinx_zynqmp_virt_defconfig
<pre>
CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET=0x9A
</pre>

= RTC chip =

* DS3231 RTC chip
* FPGA connections:
<pre>
I2C SCL <- J-ENC A85 <- FPGA E17 <- XDC TP_S <- VHDL RTC_I2C_SCL output (10k pull-up to 3.3V)
I2C SDA <-> J-ENC A87 <-> FPGA D17 <-> XDC TP_S <-> VHDL RTC_I2C_SDA bidir (10k pull-up to 3.3V)
RTC_1Hz -> J-ENC C160 -> FPGA AH12 -> XDC "slow_io" -> VHDL RTC_1PPS input (10k pull-up to 1.2V)
RTC_32k -> J-ENC B129 -> FPGA AE3 -> XDC "free pin" -> VHDL RTC_32KHZ input (10k pull-up to 1.8V)
</pre>
* test notes
<pre>
FPGA registers:

rtc_i2c_clock_ie <= not register_data_in_block_0(14)(0);
rtc_i2c_clock_out <= register_data_in_block_0(14)(1);
rtc_i2c_data_ie <= not register_data_in_block_0(14)(4);
rtc_i2c_data_out <= register_data_in_block_0(14)(5);

ext_out(1) <= rtc_1pps_in;
ext_out(2) <= rtc_32khz_in;

register_data_out_block_0(13) <= x"000000" & rtc_i2c_data_ie & rtc_i2c_data_out & rtc_i2c_clock_ie & rtc_i2c_clock_out & rtc_1pps_in & rtc_32khz_in & rtc_i2c_data_in & rtc_i2c_clock_in;

After power us on the RTC chip:
i2c clock 3.3V
i2c data 3.3V
32kHz 0.778V see clock running with amplitude 0->1.8V (FPGA pullup) period about 30 usec (~33 kHz)
1pps 1.2V (FPGA pullup)

./test1.exe --read32 0x34
dsdm_read32[0x00000034] is 0x000000ab (171) -> 1010'1011 -> data_ie, clock_ie, 1pps, data_in, clock_in
./test1.exe --read32 0x38
dsdm_read32[0x00000038] is 0x00000000 (0)

./test1.exe --write32 0x38 0x01 -> clock oe
./test1.exe --read32 0x34
Ddsdm_read32[0x00000034] is 0x0000008a (138) -> 1000'1010 -> data_ie, 1pps, data_in, clock 0V, data 3.3V

./test1.exe --write32 0x38 0x10 -> data_oe
./test1.exe --read32 0x34
dsdm_read32[0x00000034] is 0x00000029 (41) -> 0010'1001 -> clock_ie, 1pps, clock_in, clock 3.3V, data 0V

root@dsdm:~# ./test_cdm_local.exe --rtc-read
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Read the RTC chip!
reg 0x00: 0 1 0 1 0 1 1 1 0x57
reg 0x01: 0 0 1 0 0 0 1 0 0x22
reg 0x02: 0 0 0 0 0 0 0 0 0x00
reg 0x03: 0 0 0 0 0 0 0 0 0x00
reg 0x04: 0 0 0 0 0 0 0 0 0x00
reg 0x05: 0 0 0 0 0 0 0 0 0x00
reg 0x06: 0 0 0 0 0 0 0 0 0x00
reg 0x07: 0 0 0 0 0 0 0 0 0x00
reg 0x08: 0 0 0 0 0 0 0 0 0x00
reg 0x09: 0 0 0 0 0 0 0 0 0x00
reg 0x10: 0 0 0 0 0 0 0 0 0x00
reg 0x11: 0 0 0 0 0 0 0 0 0x00
reg 0x12: 0 0 0 0 0 0 0 0 0x00
reg 0x13: 0 0 0 0 0 0 0 0 0x00
reg 0x14: 0 0 0 1 1 0 0 0 0x18
reg 0x15: 0 0 0 0 1 0 0 0 0x08
reg 0x16: 0 0 0 0 0 0 0 0 0x00
reg 0x17: 0 0 0 1 0 1 1 1 0x17
reg 0x18: 1 1 0 0 0 0 0 0 0xc0
seconds: 57, minutes: 22, hours: 00, day 0, date: 00-00-00, temp 23.75C
root@dsdm:~#
</pre>

= clock chip configuration =

file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt:

* VCO is 14 GHz
* Tvco is 71.43 ps
* N0 divider is 14, frequency is 1000 MHz
* out0 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz
* out1 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
* out2 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
* out3 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz is the feedback for zero delay

= clock chip monitoring =

from si5395-94-92-family.pdf:
* reg 0x1: page select, set to 0 or set to 5 to read 0x53F
* reg 0x2: 0x94
* reg 0x3: 0x53 -> device is a si5394
* reg 0xC: LOSXAXB
* reg 0xD: LOS and OOF for the 4 clock inputs
* reg 0xE: LOL and HOLD
* reg 0xF: CAL_PLL
* reg 0x11: sticky bits for reg 0xC
* reg 0x12: sticky bits for reg 0xD
* reg 0x13: sticky bits for reg 0xE
* reg 0x14: sticky bits for reg 0xF
* reg 0x1C: device reset
* reg 0x1E: low power, hard reset, SYNC
* reg 0x507: currently selected input clock
* reg 0x52A: input clock select
* reg 0x535: FORCE_HOLD
* reg 0x53F: HOLD_HIST_VALID and FASTLOCK_STATUS

= Install Xilinx tools =

* install Vivado 2020.2
<pre>
login at https://www.xilinx.com/myprofile.html
go to "Downloads"
go to archive,
find 2020.2
download Xilinx_Unified_2020.2_1118_1232_Lin64.bin
sh ./Xilinx_Unified_2020.2_1118_1232_Lin64.bin
banner window should open with spinner "downloading installation data"
"a newer version is available" -> say "continue"
next
"select install type" window:
provide email and password,
select "download image"
select directory /home/olchansk/Xilinx/Downloads/2020.2\
select "linux" and "full image"
next
download summary: space required 38.52 Gbytes
download
installation progress
downloading spinner, 16 M/s 47 minutes...
"download image has been created successfully". Ok.
check contents of /home/olchansk/Xilinx/Downloads/2020.2
ls -l /home/olchansk/Xilinx/Downloads/2020.2
total 67
drwxr-xr-x 2 olchansk users 9 Sep 1 16:22 bin
drwxr-xr-x 3 olchansk users 15 Sep 1 16:23 data
drwxr-xr-x 4 olchansk users 4 Sep 1 16:22 lib
drwxr-xr-x 2 olchansk users 644 Sep 1 16:22 payload
drwxr-xr-x 2 olchansk users 7 Sep 1 16:22 scripts
drwxr-xr-x 4 olchansk users 4 Sep 1 16:22 tps
-rwxr-xr-x 1 olchansk users 3256 Nov 18 2020 xsetup
daq13:2020.2$
./xsetup
spinned loading installation data
xilinx design tools 2022.1 now available -> say continue
"welcome" -> next
"select product" -> vivado -> next -> vivado hl system edition -> next
select devices: only zynq ultrascale+ mpsoc -> next
select destination: /opt/Xilinx (as root, mkdir /opt/Xilinx, chmod olchansk.users /opt/Xilinx)
install ...
complete
move /home/olchansk/Xilinx/Downloads/2020.2 to /daq/daqstore/olchansk/Xilinx/Downloads/
</pre>
* install petalinux 2020.2
<pre>
./xsetup
"a newer version is available" -> say "continue"
next
"select product to install" -> select Petalinux (Linux only) -> next
"select destination directory" -> select "/opt/Xilinx" (disk space required 2.64 GB) -> next
"summary" -> install ...
error about missing /tmp/tmp-something files
"installation completed successfully" (hard to dismiss, "ok" button is partially cut-off)
done?
I think it failed, /opt/Xilinx/PetaLinux/2020.2/bin is empty except for petalinux-v2020.2-final-installer.run
try to run it by hand, same error about /tmp/tmp-something files. strange...
notice it complains about "truncate", which truncate finds ~/bin/truncate, get rid of it,
try again
now complains about missing texinfo and zlib1g:i386
apt install texinfo -> ok
apt install zlib1g:i386 -> installs bunch of gcc stuff -> ok
try again
reports "already installed" -> delete /opt/Xilinx/.xinstall/PetaLinux_2020.2/, delete entries in ~/.Xilinx/registry/installedSW.xml
try again
success
</pre>
* install vivado 2022.1 and petalinux 2022.1 - everything is pretty much the same

= Petalinux =

* cd PetaLinux_GDM_CDM
* petalinux-config
* enable i2c MAC address and DHCP
<pre>
git clone https://bitbucket.org:/team-ds-dm/ds-dm-u-boot-xlnx.git
cd ds-dm-u-boot-xlnx
git checkout ds-dm-u-boot-xlnx

linux-components ->
uboot -> ext-local-src
external u-boot local source -> ds-dm-u-boot-xlnx (path to the customized uboot git repository)
</pre>
* enable DHCP
<pre>
Subsystem AUTO Hardware Settings -> Ethernet Settings
randomize MAC address -> NO
ethernet mac address -> leave empty
obtain ip address automatically -> YES
</pre>
* set hostname and product names
<pre>
Firmware Version Configuration ->
Host name -> "ds-dm"
Product name -> "Petalinux_GDM_CDM"
</pre>
* configure linux kernel
<pre>
petalinux-config -c kernel
</pre>
* enable NFS-Root
<pre>
petalinux-config
Image Packaging Configuration > Root File System Type -> set to NFS
Location of NFS root directory set to "/nfsroot"

petalinux-config -c kernel
Networking support > IP: kernel level configuration
enable DHCP, BOOTP, RARP
File systems > Network file systems > Root file systems on NFS
</pre>
* manually fix linux kernel command line:
<pre>
grep nfsroot PetaLinux_GDM_CDM/project-spec/configs/config
edit CONFIG_SUBSYSTEM_BOOTARGS_GENERATED to read
earlycon console=ttyPS0,115200 clk_ignore_unused panic=60 root=/dev/nfs nfsroot=/nfsroot/%s ip=dhcp rw
</pre>
* check configuration in
** PetaLinux_GDM_CDM/project-spec/configs/config
** PetaLinux_GDM_CDM/project-spec/configs/rootfs_config
** PetaLinux_GDM_CDM/components/plnx_workspace/device-tree/device-tree/system-conf.dtsi

= JTAG server =

localhost:3121

= ds20k block =

<pre>
module ds20k
(
// CLOCK INPUTs
input wire clk,
input wire reset, // pulse for power-up reset
input wire si5394_loln, // clock cleaner PLL is locked to selected input clock

// REGISTER_DATA

input wire [255:0] [31:0] register_data_in,
output reg [255:0] [31:0] register_data_out,
input wire register_we,
input wire register_re,

// GDM QSFP FIBER LINKS
input wire qsfp_tx_data_ready,
output reg [11:0] [15:0] qsfp_tx_data,
output reg [11:0] [1:0] qsfp_tx_data_is_k,
input wire [11:0] [15:0] qsfp_rx_data_error, // not sure what this is.
input wire [11:0] [15:0] qsfp_rx_data,
input wire [11:0] [1:0] qsfp_rx_data_is_k,

// CDM SFP FIBER LINKS
input wire sfp_tx_data_ready,
output reg [15:0] sfp_tx_data,
output reg [1:0] sfp_tx_data_is_k,
input wire sfp_rx_data_error, // single bit indicating that RX link is up and data is good.
input wire [15:0] sfp_rx_data,
input wire [1:0] sfp_rx_data_is_k,

// VX_RXs
input wire [3:0] vx1_rx_async,
input wire [3:0] vx2_rx_async,
input wire [3:0] vx3_rx_async,
input wire [3:0] vx4_rx_async,
input wire [3:0] vx5_rx_async,
input wire [3:0] vx6_rx_async,
input wire [3:0] vx7_rx_async,
input wire [3:0] vx8_rx_async,
input wire [3:0] vx9_rx_async,
input wire [3:0] vx10_rx_async,
input wire [3:0] vx11_rx_async,
input wire [3:0] vx12_rx_async,

// VX_TXs
output reg [2:0] vx1_tx_out,
output reg [2:0] vx2_tx_out,
output reg [2:0] vx3_tx_out,
output reg [2:0] vx4_tx_out,
output reg [2:0] vx5_tx_out,
output reg [2:0] vx6_tx_out,
output reg [2:0] vx7_tx_out,
output reg [2:0] vx8_tx_out,
output reg [2:0] vx9_tx_out,
output reg [2:0] vx10_tx_out,
output reg [2:0] vx11_tx_out,
output reg [2:0] vx12_tx_out,

// VX serial comunications
input wire [11:0] vx_rx_ser_in,
output reg [11:0] vx_tx_ser_out,

// LEMO INPUTs
input wire [4:1] ext_in_async, // direct connection to LEMO connectors, not clocked

// LEMO OUTPUTs
output reg [2:1] ext_out, // direct connection to LEMO connectors, not clocked

// FRONT PANEL LEDs
output reg [3:0] fp_led_out, // direct connection to LEMO connectors, not clocked
);
</pre>

= world view =

Main components:
* clock distribution - to ensure all digitizers run synchronously and waveform timestamp are easy to assemble intophysics events.
* global trigger and run control - all digitizers are triggered at the same time to record calibration events, pulser events and the run startup synchronization sequence.
* busy distribution - when some digitizers go busy and stop accepting triggers, physics events may become incomplete, this must be recorded and managed.
* hitmap distribution - to create an online picture of the detector for monitoring and for triggering.

== global clock distribution ==

Clock distribution ensures all digitizers run synchronously (on the same clock) and waveform timestamps are easy to assemble into physics events. The clock source is synchronized with GPS to ensure a metrologically validated clock frequency (tied to the SI definition of 1 second) and date stamp (via GPS IRIG-B date stamp). Individual digitizer waveform timestamp are reset at the begin of run to establish a valid time zero (run start time, date stamped with the GPS IRIG-B date stamp).

Base clock is 125 MHz (8 ns). Individual digitizers are synchronized at initialization (power-up) time. Because of incontrollable delays in the electronics (i.e. variations between individual PLL chips in each digitizer), waveform timestamps between individual digitizers may be off by 1-2 clocks (8-16 ns). After the system is running and is stable (thermally and electrically), timestamp waveform variation and jitter are less than 1 ns.

This is validated by feeding a common calibration pulser signal into analog inputs of each digitizer (at 100 Hz) and measuring (fitting) the pulse time in the waveform. With sufficient statistics, differences between waveform timestamp are measured with picosecond precision.

CDM PLL allows fine phase adjustment of the 62.5 MHz VX2746 clock in groups of 6 clocks (2 groups per CDM) with precision 71.43 ps.

Due to FPGA technology limitations (serial link hard IPs, PLLs, etc) the pattern of 1-2 clock delays between individual digitizers may change/shift from power up to power up. Significant additional work would be required to ensure this pattern is repeatable and stable after every power up and initialization. As mentioned before, after power up and initialization, variation between waveform timestamp is sub-nanosecond as measured by the analog calibration pulser.

Clock chain:
* GPS receiver (1pps clock reference and IRIG-B date stamp)
* LNGS GPS clock distribution (fiber link to converter box at the experiment)
* GPS 1pps to PRS-10 Rb atomic clock (10 MHz clock output)
* 10 MHz clock from PRS-10 to GDM PLL (extra low jitter oscillator, 125 MHz clock output)
* 125 MHz clock to GDM fiber link transmitters (2.5 Gbps line rate)
* CDM fiber link receiver and clock recovery (125 MHz clock)
* 125 MHz recovered clock to CDM PLL (extra low jitter oscillator, 62.5 MHz clock output)
* 62.5 MHz CDM clock to VX2745 PLL via LVDS line
* VX2745 PLL (125 MHz clock output)
* 125 MHz clock to digitizer chips, clock counter and waveform timestamp.

This is the normal mode of operation, with global clock synchronized with the GPS clock (metrologically referenced to SI 1 Hz definition).

If GPS signal is not available, degraded modes of operation are available (in order of worsening performance)
* PRS-10 Rb atomic clock synchronized with a battery-powered time-of-day chip
* PRS-10 Rb atomic clock synchronized with Internet time (NTP) assuming Internet network connection is up
* PRS-10 Rb clock standalone
* GDM internal oscillator

All degraded modes provide high precision low jitter 125 MHz system clock, but long term stability (seconds per day) will vary. (TBM)

In the absence of GPS IRIG-B date stamp, the battery-powered time-of-day chip will be used (TBI) with fallback to Internet time (NTP). Date stamp long term stability is expected to be better than 1 second per day (TBI, TBM).

Note:
* TBM: to be measured
* TBI: to be implemented

Current status (April 2025):
* full clock chain from GDM PLL to VX2745 digitizers is implemented and tested
* waveform timestamp variations are measured (see report from Marek)
* GDM external clock is validated (using VME-GRIFFIN-CDM high quality clock source module)
* PRS-10 10 MHz and 1pps connection to GDM is implemented and tested
* GPS receiver 1pps connection to PRS-10 is implemented and tested
* GPS receiver IRIG-B date stamp connection to GDM tested using a software decoder. Hardware decoder and connection to time slice marker, TBI.
* LNGS date stamp decoder and connection to time slice marker, TBI
* LNGS 1pps connection to PRS-10, not available
* operate vertical slize system from GPS, not available (GPS is in the det lab, vertical slice is in the daq lab).

== global trigger distribution ==

Global trigger distribution provides a synchronous waveform trigger to all VX2745, all CDM and all VX2745 are provided wiht the same data and are all triggered at the same time (without accounting for the 1-2 clock offset described in the clock section).

Global trigger is used for run control: the first trigger of a run resets the waveform timestamp counter, the CDM timestamp counter and the GDM timestamp counter. At the begin of run, GDM generates a sequence of triggers following an easy to identify pattern. If a VX2745 misses a trigger (for any reason), online and offline software can easily identify the tru run start time using this pattern.

After a run has started, global trigger can be generated by a programmable pulser, by an external LEMO signal (i.e. external calibration signal, laser Q-switch, etc) and by the VX2745 trigger hitmap.

Trigger sources (TRG):
* TRG pulser
* LEMO inputs
* hitmap trigger

Time slice marker sources (TSM):
* TSM pulser

Trigger chain, K-code and LVDS:
* TRG/TSM source
* TRG/TSM K-code from GDM to CDM
* CDM K-code to LVDS line 12 and 13 lines (TRG and TSM)

Latency from GDM LEMO input to CDM LVDS out: TBM

Trigger chain, packet trigger:
* GDM TRG/TSM source to TRG/TSM packet generator
* or GDM hitmap group trigger packet generator
* TRG/TSM packet from GDM to CDM
* TRG/TSM packet from CDM to VX2745 via LVDS line, 62.5 MHz, 8b10b encoding.

Latency from GDM LEMO input to VX2745 internal trigger: TBM

Notes:
TBM - to be measured
TBI - to be implemented

Current status (Jan 2026):
* LVDS TRG and TSM distribution is done and tested
* packet TRG, TRIG48 and TSM distribution from GDM to CDM is done and tested
* packet TRG, TRIG48 and TSM from CDM to VX2745 is done and tested
* packet TRIG48 processing in the VX is done and tested

== global busy distribution ==

If any VX2745 goes busy (for any reason) and stops accepting triggers, physics data may become incomplete. This situation needs to be recorded and managed.

Busy chain:
* VX2745 firmware busy logic (CAEN and Yair) generates the VX busy signal on LVDS line 9 (CDM VX_RX_2)
* CDM monitors VX busy
* CDM records busy transitions
* CDM computes the "CDM busy" as grand-OR of all VX busy
* CDM periodically transmits CDM busy status the GDM using special K-codes (NB: confirm transmission period)
* GDM receives CDM busy updates and maintains "CDM busy" for each CDM
* GDM records busy transitions
* GDM computes "GDM busy" as grand-OR of all CDM busy
* effectively, GDM busy is a grand-OR of all VX busy (with max delay, TBM)

In the pilot implementation, any VX busy stops the experiment by sending a veto signal to all VX2745 which stops them from accepting triggers.

Veto chain:
* GDM computes "GDM veto" as GDM busy
* GDM periodically transmits GDM veto status to all CDMs using special K-codes (NB: confirm transmission period)
* CDM receive GDM veto updates and maintain the "CDM veto" signal
* CDM computes "VX veto" as CDM veto
* CDM transmits VX veto to all VX2745 on LVDS pair 13
* VX2745 firmware (Yair) has VX veto block all physics triggers (NB: except for the time slice marker?)

Total delay from VX busy to VX veto (on the LVDS lines), TBM.

Notes:
* TBM - to be measured
* TBI - to be implemented

Current status (Jan 2026):
* full busy chain from VX to CDM to GDM is implemented, tested
* veto chain from GDM to CDM is implemented, tested
* veto chain from CDM to LVDS pair 13 is implemented, tested
* LVDS pair 13 veto inside the VX is implemented, tested
* TRG switched from LVDS pair 12 to TRG packet, testing completed
* TSM switched from LVDS pair 13 to TSM packet, testing completed
* fine tuning of busy chain, VX buffer levels, etc, test in progress

== vx hitmap and hitmap trigger ==

To monitor the system in real time and to generate group triggers, we use the VX hitmaps.

Each VX2745 has 64 channels, each channel has a trigger logic block (digital discriminator) that generates a waveform self-trigger and sends a hitmap bit to the CDM (64-bits per VX2745) as a data packet. CDM passes hitmap data packets to the GDM and also records non-empty hitmap packets. CDM MIDAS frontend passes non-empty hitmap packets to MIDAS for online monitoring and visualization. GDM collects all the hitmaps and generates a global trigger. Alternatively, it generates a group trigger TRIG48 with a 48-bit bitmap indicating which VX2745 should trigger a waveform.

Pilot implementation computes the hitmap trigger as a grand-OR of all hit maps.

Hitmap chain:
* VX2745 analog inputs to digitizers to digital trigger logic block, 1 bit per channel (CAEN, Yair)
* 64-bits of hitmap (1 bit/channel) are collected and periodically transmitted to the CDM (Sam De Jong) (NB: confirm transmission period: see CDM page table "VX hit map" column "period", 0xD1 clocks is 1672 ns, should be reduced to 1500 ns, TBI)
* VX hitmap packets are transmitted to the CDM on the LVDS line 10 (CDM VX_RX_1) at 62.5 MHz, 8b10b encoded.
* CDM receives hitmap packets (12 input data streams, 12-to-1 packet mux)
* CDM sends all hitmap packets to the GDM at 125 MHz, 16b20b encoded.
* non-empty hitmap packets are written to the CDM-to-CPU data FIFO (and via MIDAS frontend to MIDAS SYSTEM event buffer)
* grand-OR of VX hitmap packets (12*64 bits) can generate a CDM-local VX trigger
* GDM receives hitmap packets, computes "GDM hitmap trigger"
* if enabled, GDM hitmap trigger causes GDM trigger, which is processed as described in the global trigger section

Delay, latency and jitter from analog pulse on the VX2745 analog input to VX trigger on LVDS line 12, TBM.

TRIG48 VX group trigger:
* GDM receives hitmap packets, computes the 48-bit bitmap of which VX2745 modules should trigger
* how to specify this computation is not clear. a generic 12*64 MLU is impossible, insufficient RAM on the FPGA
* 48-bit bitmap is sent to all CDMs as a TRIG48 data packet
* CDM receives TRIG48 data packets from GDM and retransmits them to the VX2745 (all CDMs and all VXes see the same packets at the same time).
* VX2745 receives the 48-bit trigger bitmap, compares it with it's own 48-bit bitmap if any common bits are set, triggers waveform readout of all channels (there will be no more than 48 VX2745 in the experiment)

Pilot implementation of TRIG48 triggers on the CDM grand-or and generates a TRIG48 packet that contains a 48-bit bitmap stored in an AXI register.

Example uses of TRIG48:
* to trigger individual VX2745 modules: set the VX 48-bit map to the module number (vx05-0x1, vx06-0x2, vx07-0x4, etc), generate TRIG48 packet with a bitmap of VX modules that should trigger: 0xFFFF'FFFF'FFFF will trigger all modules, 0x1 will trigger only vx05, 0x2 will trigger only vx06, etc. (there is no more than 48 modules).
* to trigger groups of VX2745 modules: set the bitmap of group 1 modules to 0x1, set the bitmap of group 2 modules to 0x2, generate TRIG48 packet with bitmap 0x0001 to trigger only group 1 modules, 0x0002 to trigger only group 2 modules, 0x3 to trigger both groups.

Notes:
* TBM - to be measured
* TBI - to be implemented

Current status (Jan 2026):
* full chain (VX analog input to hitmap generator to CDM to GDM) is implemented, testing in progress
* GDM grand-or trigger is implemented
* VX hitmap packet generator, testing in progress (Yair firmware)
* CDM hitmap processing, implemented, tested
* GDM hitmap processing, implemented, tested
* pilot TRIG48 trigger generator in GDM, CDM, tested
* TRIG48 trigger in VX, implemented, tested
* MIDAS frontend readout of hitmap data from FPGA FIFO: implemented, testing in progress (FIFO code: Ian, done; integration and C++ code: KO)
* online monitoring and visualization in MIDAS: TBI

== firmware map ==

Note:
* red lines: clocks
* green lines: AXI/Avalon packet streams
* blue lines: serial data

[[File:GDM-CDM-VX.drawio.svg]]

== description ==

same thing, in words:

<pre>
detector
digitizer, 125 MHz
digital filter
digital discriminator
hit map, 64 bits at 125 MHz (could be 250 MHz, filter and discriminator clock)
packetizer, 64 bits -> id, timestamp, 8x 8-bit words, eop
8/10 serializer, 12.5 MHz parallel in, 125 MHz serial out
lvds line to CDM vx_rx
BBB: also VX busy to lvds line to CDM vx_rx, do not want to depend on serial comm for vx busy, do not want to inject async data into the hit map packet stream
--- CDM
lvds line capture in IBUF register, vxN_rx clock, 125 MHz clock, from PLL with adjustable phase, scan phase to find sweet spot
vx rx 10/8 deserializer, vxN_rx clock, stobes out parallel data every 10 clocks at 12.5 MHz
vx rx phase transfer from vxN_rx clock to main CDM clock, have 10 clocks for transfer to happen
from here everything is on the CDM main clock
vx rx packet adapter, 12.5 MHz 8-bit packets to 125 MHz 16-bit packets, internal FIFO to avoid data overrun
sfp tx mux, all VX packet streams into one CDM sfp tx packet stream
BBB: capture VX RX busy to CDM main clock (IBUF register), grand-or becomes vx_rx_busy
sfp tx packetizer (data,eop,vx_rx_busy -> data,k)
sfp tx 16/20 serializer, 16 bit at 125 MHz in, 2500 MHz serial out (20 bits at 125 MHz)
sfp tx fiber link to GDM, 2 Gigabits/sec
--- GDM
qsfp rx deserializer, 16 bit at 125 MHz output on main GDM clock
qsfp rx depacketizer (data,k -> packet data,eop; qsfp_rx_busy)
qsfp rx demux, hit map packets routed to GDM trigger logic block
GDM trigger logic block looks at hit map, generates yes/no trigger decision, encodes it as a trigger packet
TSM generator encodes GPS time data as a TSM packet
qsfp tx mux - trigger packets, tsm packets, etc to qsfp tx packet stream
qsfp tx packetizer (data,eop;trg,tsm,bsy -> data,k)
NB: the same tx data is sent to all 12 qsfp tx ports, to make sure we do not accidentally desync the CDMs.
qsfp tx 16/20 serializer, 16-bit at 125 MHz to 20-bit at 250 MHz to 2500 MHz serial out
qsfp tx fiber link to CDM, 2 Gigabits/sec
--- CDM
sfp rx 20/16 deserializer, 16-bit at 125 MHz
sfp rx depacketizer, (data,k -> data,eop;trg_in,tsm_in,bsy_in)
sfp rx demux (in reality, noop, all packets go to same place, vx tx)
vx tx mux (packets from GDM, packets with simulated waveforms from CDM midas frontend via AXI FIFO)
vx tx packet adapter 16-bit at 125 MHz to 8-bit at 12.5 MHz strobed every 10 clocks. NB: most important, this 10-clock strobe runs in sync between all CDMs!
vx tx phase transfer from main CDM clock to vxN_tx clock, have 10 clocks for transfer to happen
from here we run on the vxN_tx clock
vx txN serializer, 8-bit at 12.5 MHz to 125 MHz serial
vx_txN OBUF register, vxN_rx clock, 125 MHz clock, from PLL with adjustable phase, scan phase to find sweet spot
lvds line to vx
BBB: bsy_in from GDM is converted from pulse to level, goes out lvds line to vx, sync to vxN_tx clock
TTT: trg_in from GDM is a pulse, does out lvds line to vx, sync to vxN_tx clock.
NB: tsm is always a packet, bsy is always a signal (no packet), trg can be a packet or signal.
--- VX
lvds data captured by 125 MHz ADC sampling clock (CAEN base firmware logic)
lvds data connected to Yair's block
10/8 deserializer
depacketizer (data,k -> data,eop) to avoid accidental desync, we do not send any no trg, no tsm, no bsy K-codes.
demux
trigger packets go their way (4x 64 bit words of data go to event header: timestamps, hitmap data)
tsm packets go their way (4x 64 bit words of data go to tsm event header: timestamps, GPS time data)
BBB: bsy from lvds line stops waveform acquisition
TTT: trg from lvds line cause waveform acquisition, same as trg packet, but has no timestamp and other data attached to it.
</pre>

== CDM-VX connections ==

Note:
* rounded boxes are VX registers set from MIDAS

[[File:CDM-VX.drawio.svg]]

= Firmware registers =

== Block 0 ==

<pre>
0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
</pre>

== Block 0 GDM ==

* 0x0000+4*i - GDM AXI clock registers

<pre>
register_data_out_block_0(1) <= x"204D4447"; --GDM<space>
register_data_out_block_0(2) <= x"00000000"; --blank
register_data_out_block_0(3) <= x"00000000"; --blank

--Firmware Version
register_data_out_block_0(4) <= x"abcd0123"; --Firmware version name register
register_data_out_block_0(5) <= x"00000001"; --Firmware version number register
register_data_out_block_0(6) <= x"abcd0123"; --Firmware version register
register_data_out_block_0(7) <= x"00000001"; --Firmware version register

--Git Commit/Tag/Compile Date
register_data_out_block_0(8) <= x"0123abcd"; --Firmware Git info (0-3)
register_data_out_block_0(9) <= x"00000001"; --Firmware Git info (1)
register_data_out_block_0(10) <= x"00000001"; --Firmware Git info (2)
register_data_out_block_0(11) <= x"00000001"; --Firmware Git info (3)

--Simple loopback register for testing
register_data_out_block_0(12) <= register_data_in_block_0(12);

--Hardware registers (HW_ID_xDM, etc)
--register_data_out_block_0(13) <= x"0000000" & "000" & hardware_id;

--unspecified
set_reg_block_0_gen : for i in 14 to nregisters_block_0-1 generate
register_data_out_block_0(i) <= register_data_in_block_0(i);
end generate set_reg_block_0_gen;

register_data_out_block_0(13) <= x"000000" & rtc_i2c_data_ie & rtc_i2c_data_out & rtc_i2c_clock_ie & rtc_i2c_clock_out & rtc_1pps_in & rtc_32khz_in & rtc_i2c_data_in & rtc_i2c_clock_in;

rtc_i2c_clock_ie <= not register_data_in_block_0(14)(0);
rtc_i2c_clock_out <= register_data_in_block_0(14)(1);
rtc_i2c_data_ie <= not register_data_in_block_0(14)(4);
rtc_i2c_data_out <= register_data_in_block_0(14)(5);
</pre>

== Block 0 CDM ==

* 0x0000+4*i - CDM AXI clock registers:

<pre>
USR_ACCESS_7series_inst: USR_ACCESSE2
port map(-- CFGCLK => CFGCLK,
DATA => register_data_out_block_0(0)); -- where timestamp is contained
-- DATAVALID => DATAVALID);

--static values
register_data_out_block_0(1) <= x"204D4443"; --CDM<space>
register_data_out_block_0(2) <= x"00000000"; --blank
register_data_out_block_0(3) <= x"00000000"; --blank

--Firmware Version
register_data_out_block_0(4) <= x"abcd0123"; --Firmware version name register
register_data_out_block_0(5) <= x"00000001"; --Firmware version number register
register_data_out_block_0(6) <= x"abcd0123"; --Firmware version name register
register_data_out_block_0(7) <= x"00000001"; --Firmware version number register

--Git Commit/Tag/Compile Date
register_data_out_block_0(8) <= x"0123abcd"; --Firmware Git info (0-3)
register_data_out_block_0(9) <= x"00000001"; --Firmware Git info (1)
register_data_out_block_0(10) <= x"00000001"; --Firmware Git info (2)
register_data_out_block_0(11) <= x"00000001"; --Firmware Git info (3)

--Simple loopback register for testing
register_data_out_block_0(12) <= register_data_in_block_0(12);

--Hardware registers (HW_ID_xDM, etc)
--register_data_out_block_0(13) <= x"0000000" & "000" & hardware_id;

--unspecified
set_reg_block_0_gen : for i in 14 to nregisters_block_0-1 generate
register_data_out_block_0(i) <= register_data_in_block_0(i);
end generate set_reg_block_0_gen;

rx_clk_sel <= register_data_in_block_0(12)(0); -- rx_clk select 0:clk_cc_out0, 1:rx_clk_mgt

rtc_i2c_clock_ie <= not register_data_in_block_0(14)(0);
rtc_i2c_clock_out <= register_data_in_block_0(14)(1);
rtc_i2c_data_ie <= not register_data_in_block_0(14)(4);
rtc_i2c_data_out <= register_data_in_block_0(14)(5);

register_data_out_block_0(13) <= x"000000" & rtc_i2c_data_ie & rtc_i2c_data_out & rtc_i2c_clock_ie & rtc_i2c_clock_out & rtc_1pps_in & rtc_32khz_in & rtc_i2c_data_in & rtc_i2c_clock_in;
</pre>

== Block 1 GDM ==

* 0x0800+4*i - GDM registers:

<pre>
clk_config_pro : process (read_write_clk)
begin
if rising_edge (read_write_clk) then
CLK_IN_SEL_LS <= clk_config_vec(1 downto 0);
CLK_EXT_SEL_LS <= clk_config_vec(2);
CLK_RSTn_LS <= not clk_config_vec(3);
clk_config_vec(4) <= CLK_LOSXTn_LS;
clk_config_vec(5) <= CLK_LOLn_LS;
clk_config_vec(6) <= CLK_INTn_LS;
clk_config_vec(7) <= '1';

clk_config_vec(3 downto 0) <= register_data_in_block_1(0)(3 downto 0);
register_data_out_block_1(0)( 7 downto 0) <= clk_config_vec(7 downto 0);
register_data_out_block_1(0)(31 downto 8) <= (others =>'0');
register_data_out_block_1(1) <= (others =>'0'); --free
end if;
end process clk_config_pro;

register_data_out_block_1(2) <= register_data_in_block_1(2);
register_data_out_block_1(3) <= register_data_in_block_1(3);
register_data_out_block_1(4)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(5)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(6)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(7)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(8) <= register_data_in_block_1(8);
register_data_out_block_1(9)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(10) <= (others => '0');
register_data_out_block_1(11) <= (others => '0');

frequency_measure_clks <= read_write_clk & clk_50MHz & mgt_data_clk & mgt_ref_clk_raw; -- & mgt_data_clk & mgt_ref_clk_raw;
register_data_out_block_1(12) <= register_data_out_block_1(14); --MGT rx and tx reference clocks are the same
register_data_out_block_1(13) <= register_data_out_block_1(15); --MGT rx and tx data clocks are the same
frequency_measure_inst : frequency_measure
generic map(register_clk_frequency => 100000000,
max_frequency => 300000000,
nfrequency_measurments => nfrequency_measure_clks)
port map(i_register_clk => read_write_clk,
i_clks => frequency_measure_clks,
i_clk_locks => (others =>'1'),
o_frequencies => register_data_out_block_1(14+nfrequency_measure_clks-1 downto 14)); --see register_data_out_block_1(12,13) above

zero_spare_link_registers : for i in 14+nfrequency_measure_clks to nregisters_block_1-1 generate
register_data_out_block_1(i) <= (others => '0');
end generate zero_spare_link_registers;

gdm_link_interface_inst : gdm_link_interface
generic map(enable_rxslide => '1',
nquads => nquads,
nlinks => nlinks)
port map(i_mgt_ref_clk_p => mgt_b_ref_clk_p,
i_mgt_ref_clk_n => mgt_b_ref_clk_n,
i_mgt_rx_n => mgt_b_rx_n,
i_mgt_rx_p => mgt_b_rx_p,
o_mgt_tx_n => mgt_b_tx_n,
o_mgt_tx_p => mgt_b_tx_p,
--raw, pre-pll, mgt clock for frequency measurement
o_mgt_ref_clk_raw => mgt_ref_clk_raw,
--free running clock and reset
i_mgt_free_running_clk => clk_50MHz,
i_mgt_rst => register_data_in_block_1(2)(0),
i_rx_slide_trigger => register_data_in_block_1(3)(nlinks-1 downto 0),
--data interface for the links
o_data_clk => mgt_data_clk,
o_mgt_data_link_array => mgt_link_data_to_processing,
i_mgt_data_link_array => mgt_link_data_from_processing,
--status ports
o_link_power_good => register_data_out_block_1(4)(nlinks-1 downto 0),
o_tx_link_up => register_data_out_block_1(5)(nlinks-1 downto 0),
o_rx_link_up => register_data_out_block_1(6)(nlinks-1 downto 0),
o_rx_error => register_data_out_block_1(7)(nlinks-1 downto 0),
i_link_down_latched_rst => register_data_in_block_1(8)(0),
o_link_down_latched => register_data_out_block_1(9)(nlinks-1 downto 0),
--debug
i_debug_clk => read_write_clk,
o_debug => mgt_debug_data);
</pre>

<pre>
#### Register Block 1 (CDM)
Clock Chip and MGT Link, Control and Status Registers
| Register | Axi Address | Description | note
|:-|:-|:-|:-|
| 0 | 0x80011000 | Clock Chip Status and Control | CLK_IN_SEL_LS(1-0)(read-only), CLK_EXT_SEL_LS(2), CLK_RSTn_LS(3), CLK_LOSXTn_LS(4), CLK_LOLn_LS(5), CLK_I\
NTn_LS(6)
| 1 | 0x80011004 | free for use | |
| 2 | 0x80011008 | MGT Link Reset | mgt_rst(0), Reset for RX Error Latched(1)|
| 3 | 0x8001100c | MGT rx slide control | not needed or used |
| 4 | 0x80011010 | link_up_and_running(10) tx_link_up_and_running(9), tx_sending_data(8), tx_link_up(7), rx_link_up_and_running(6), rx_error(5), rx_receivi\
ng_data(4), rx_link_up(3), link_power_good(2), sfp_rx_los(1), sfp_mod_absent(0) | Should be 0x7DC |
| 5 | 0x80011014 | reserved debug register | |
| 6-11 | ... | free for use | |
| 12 | 0x80011030 | mgt_rx_ref_clk_raw | ~0x7735940 (125 MHz) |
| 13 | 0x80011034 | rx_clk | ~0x7735940 (125 MHz) |
| 14 | 0x80011038 | mgt_tx_ref_clk_raw | ~0x7735940 (125 MHz) |
| 15 | 0x8001103c | tx_clk | ~0x7735940 (125 MHz) |
| 16 | 0x80011040 | clk_50MHz | 0x2faf080 (50 MHz) |
| 17 | 0x80011044 | Register_Block1_clk | 0x5f5e100 (100 MHz) |
| 18-20 | ... | free for use | |
| 21 | 0x80011054 | free for use | |
</pre>

=== Register 0x1000 ===

<pre>
printf("0x1000 SFP c.c. status: 0x%08x\n", reg1000);
printf(" CLK_IN_SEL_LS 0x%x\n", (reg1000>>0) & 3);
printf(" CLK_EXT_SEL_LS %d\n", (reg1000>>2) & 1);
printf(" CLK_CLK_RSTn_LS %d\n", (reg1000>>3) & 1);
printf(" CLK_LOSXTn_LS %d\n", (reg1000>>4) & 1);
printf(" CLK_LOLn_LS %d\n", (reg1000>>5) & 1);
printf(" CLK_INTn_LS %d\n", (reg1000>>6) & 1);
printf("0x1008 SFP link reset: 0x%08x\n", dev->dsdm_read32(0x1008));
</pre>

=== Register 0x1010 ===

<pre>
printf("0x1010 SFP link status: 0x%08x\n", reg1010);
printf(" sfp_mod_absent_N %d\n", (reg1010>>0) & 1);
printf(" sfp_rx_los_N %d\n", (reg1010>>1) & 1);
printf(" link_power_good %d\n", (reg1010>>2) & 1);
printf(" rx_link_up %d\n", (reg1010>>3) & 1);
printf(" rx_receiving_data %d\n", (reg1010>>4) & 1);
printf(" rx_error %d\n", (reg1010>>5) & 1);
printf(" rx_lnk_up_and_running %d\n", (reg1010>>6) & 1);
printf(" tx_link_up %d\n", (reg1010>>7) & 1);
printf(" tx_sending_data %d\n", (reg1010>>8) & 1);
printf(" tx_link_up_and_running %d\n", (reg1010>>9) & 1);
printf(" link_up_and_running %d\n", (reg1010>>10) & 1);

</pre>

=== Register 0x1014 ===

<pre>
printf("0x1014 SFP link data: 0x%08x\n", reg1014);
printf(" rx_data 0x%04x\n", (reg1014>>0) & 0xFFFF);
printf(" k28p1_k28p5 %d\n", (reg1014>>16) & 1);
printf(" rx_ctrl0 0x%01x\n", (reg1014>>17) & 3);
printf(" rx_ctrl1 0x%01x\n", (reg1014>>19) & 3);
printf(" rx_ctrl3 0x%01x\n", (reg1014>>21) & 1);
printf(" tx_state 0x%01x\n", (reg1014>>22) & 3);
printf(" rx_state 0x%01x\n", (reg1014>>24) & 0xF);
printf(" rx_receiving_data %d\n", (reg1014>>28) & 1);
printf(" rx_link_up %d\n", (reg1014>>29) & 1);
printf(" rx_error %d\n", (reg1014>>30) & 1);
printf(" rx_link_rst %d\n", (reg1014>>31) & 1);
</pre>

* see cdm_link_establishment.vhd
* tx_state:
<pre>
0 = tx_idle
1 = waiting_for_rx_link
2 = sending_k28p1_k28p5
3 = sending_data
</pre>

* rx_state:
<pre>
0 = rx_idle
1 = waiting_for_ks
2 = rx_retry
3 = rx_up_switch_tx_clk_source
4 = waiting_for_data
5 = receiving_data
6 = rx_receive_error
</pre>

== Block 2 GDM ==

* 0x1000+4*i - GDM registers:

<pre>
register_data_out_block_2(0) <= register_data_in_block_2(0);
GDM_link_data_processing_inst : GDM_link_data_processing
generic map(mgt_data_frequency => 125000000)
port map(i_register_clk => read_write_clk,
i_rst => register_data_in_block_2(0)(0),
i_data_mode => register_data_in_block_2(0)( 9 downto 8),
i_fixed_pattern_data => register_data_in_block_2(0)(31 downto 16),
o_error_time => register_data_out_block_2(1),
o_error_count => register_data_out_block_2(13 downto 2),
--link data interface
i_data_clk => mgt_data_clk,
i_mgt_data_link_array => mgt_link_data_to_processing,
o_mgt_data_link_array => mgt_link_data_from_processing,
--data interface
o_data_clk => data_clk,
o_tx_data_ready => tx_data_ready,
i_tx_data_is_k => tx_data_is_k,
i_tx_data => tx_data,
o_rx_data_error => rx_data_error,
o_rx_data_is_k => rx_data_is_k,
o_rx_data => rx_data);
</pre>

== Block 2 CDM ==

* 0x1000+4*i - CDM registers:
* reg 0
<pre>
read bits
7..0 - clk_config_vec
1,0 - CLK_IN_SEL_LS
2 - CLK_EXT_SEL_LS
3 - disconnected CLK_RSTn_LS
4 - CLK_LOSXTn_LS
5 - CLK_LOLn_LS
6 - CLK_INTn_LS
7 - tx_clock_status
31..8 - 0
</pre>
<pre>
write bits
3..2 - clk_config_vec[3..2]
</pre>
* reg 1 - 0
* reg 2 - RW
<pre>
bit
0 - cdm_link_interface i_mgt_rst
1 - cdm_link_interface i_link_down_latched_rst
</pre>
* reg 3 - RW
<pre>
bit
0 - cdm_link_interface i_rx_slide_trigger
</pre>
* reg 4 - RO
<pre>
bit
0 - sfp_mod_present
1 - sfp_rx_los
2 - link_power_good
3 - rx_link_up
4 - rx_receiving_data
5 - rx_error
6 - rx_link_up_and_running
7 - tx_link_up
8 - tx_sending_data
9 - tx_link_up_and_running
10 - link_up_and_running
</pre>
* reg 5 - RO - CDM SFP debug_data
* reg 6 - 0
* reg 7 - 0
* reg 8 - 0
* reg 9 - 0
* reg 10 - 0
* reg 11 - 0
* reg 12 - frequency AXI clock
* reg 13 - freq clk_50MHz
* reg 14 - freq clk_100MHz
* reg 15 - freq clk_200MHz
* reg 16 - freq clk_xo_125 - DS-DM 125 MHz oscillator
* reg 17 - freq clk_cc_out0 - C.C. 125 MHz output
* reg 18 - CDM SFP mgt_rx_ref_clk_raw
* reg 19 - CDM SFP mgt_tx_ref_clk_raw
* reg 20 - rx_clk_mgt, 125 MHz CDM SFP RX recovered clock
* reg 21 - rx_clk, 125 MHz, switched between clk_xo_125 and rx_clk_mgt
* reg 22 - tx_clk, 125 MHz
* reg 23 - spare

== Block 3 GDM ==

0x3000+4*i : DS20k registers

== Block 4 CDM ==

* 0x2800+4*i - CDM SFP data link registers:
* 0 - R/W
** bit 0 - 0x1 - reset
** bits 7..1 - not used
** bits 9..8 - data_mode: 00: data from ds20k block, 01: tx_data_cnt, 10: tx_data_prbs, 11: tx_fixed_pattern_data
** bits 15..10 - not used
** bits 31..16 - fixed pattern data
* 1 - RO - o_error_time
* 2 - RO - o_error_count
* 3, etc - not used

== Block 6 CDM ==

* 0x3000+4*i : DS20k registers

== Block 7 CDM ==

VX_TX and VX_RX delays, i=0..11
* 0x3800+4*i - VX_TX delays
* 0x3830+4*i - VX_RX delays

VX_TX delay data (2 bits):
* 4 values: 0,1,2,3 corresponding to the 4 phases of the 125 MHz clock rx_clk and rx_clk shifted by 90 deg

VX_RX delay data (32 bits):
* 0x8000'0000: reset IDELAYE3, follow by write of 0
* 0x0001'xxxx: load the IDELAYE3 9-bit tap counter, follow by write of 0x0000'xxxx
* 16 low bits have:
* bits 0..8 are the 9 bits of IDELAY3 tap counter (~1.1ns for all 511 delays)
* bit 9 is the 500 MHz DDR phase (1 ns)
* bits 10..15 are 16 bits of 500 MHz delay (2 ns step)
* numerically increasing 16-bit delay values are carefully arranged to produce linearly increasing delay (there is some overlap between IDELAYE3 taps above ~tap 500 (or so, IDELAYE3 is uncalibrated) and flip of 500 MHz DDR phase

== DS20k registers ==

=== DS20k block register map ===

* busybox devmem 0x80013000 32

Note: R=readable, W=writable, L=latched by CMD_LATCH, B=reset on begin of run, Z=reset by CMD_LATCH

<pre>
reg | version | xDM | xx | description
0 | 0x20230731 | ALL | RO | ds20k version
0 | 0x20240118 | ALL | RW | ds20k version and command
1 | 0x20230731 | ALL | RW | scratch read/write register
2 | 0x20230731 | ALL | RW | configure inputs and outputs
3 | 0x20230731 | ALL | RW | FP_LED mux
4 | 0x20230731 | ALL | RW | EXT_OUT mux
5 | 0x20230731 | ALL | RO | VX_RX state
6 | 0x20230731 | ALL | RO | VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state
7 | 0x20230731 | ALL | RW | LED_OUT, EXT_OUT, VX_TX outputs
8 | 0x20230731 | ALL | RW | VX_TX mux and config
9 | 0x20230731 | ALL | RW | trg and tsm source
10 | 0x20231013 | ALL | RO | status register
11 | 0x20230731 | ALL | ROLB | trg counter
12 | 0x20230731 | ALL | ROLB | tsm counter
13 | 0x20240814 | ALL | RO | GPS 1pps period, 125 MHz
14 | 0x20240814 | ALL | RO | Ru clock 1pps period, 125 MHz
15 | 0x20230811 | CDM | RO | SFP RX status
16 | 0x20230811 | CDM | RW | SFP TX control
17 | 0x20230811 | GDM | RO | QSFP RX data 0, 1
18 | |. |. | 2, 3
19 | |. |. | 4, 5
20 | |. |. | 6, 7
21 | | |. | 8, 9
22 | |. |. | 10, 11
23 | 0x20230811 | GDM | RW | QSFP TX control
24 | 0x20231013 | ALL | RW | trigger pulser period
25 | 0x20231013 | ALL | RW | trigger pulser burst control
26 | 0x20231013 | ALL | RW | tsm pulser period
27 | 0x20231204 | ALL | RW | data fifo CPU to FPGA
28 | 0x20231204 | ALL | RW | packet loopback control
28 | 0x20240118 | ALL | RW | data fifo FPGA to CPU
29 | 0x20231208 | GDM | RW | bitmap of active qsfp ports
29 | 0x20240118 | ALL | RW | packet loopback control
29 | 0x20240510 | ALL | RW | packet routing
30 | 0x20231208 | GDM | RO | qsfp link status ports 0..7
31 | 0x20231208 | GDM | RO | qsfp link status ports 8..11
32 | 0x20240118 | GDM | RW | enabled QSFP ports
33 | 0x20240118 | CDM | RW | enabled VX ports
34,35 | 0x20240118 | ALL | ROB | time stamp 64 bits
36,37 | 0x20240118 | ALL | RO | old time stamp
38,39,40 | 0x20240118 | CDM | ROLZ | vx_bsy_counter_latched | VX busy counters
41,42,43 | 0x20240118 | GDM | ROLZ | QSFP busy counters
44 | 0x20240118 | CDM | ROLZ | cdm_bsy_up_counter and cdm_bsy_pulse_counter
45 | 0x20240118 | CDM | ROLZ | cdm_veto_pulse_counter and cdm_veto_up_counter
46 | 0x20240118 | GDM | ROLZ | gdm_bsy_pulse_counter and gdm_bsy_up_counter
47 | 0x20240118 | GDM | ROLZ | gdm_bsy_refresh_counter
48 | 0x20240118 | GDM | ROLZ | gdm_veto_up_counter and gdm_veto_pulse_counter
49,50,51 | 0x20240424 | CDM | RO | VX RX serial data monitor, 8 bits per VX port
52,53 | 0x20240424 | CDM | RO | VX serial link status, 4 bits per VX port
54 | 0x20240424 | CDM | RO | VX TX serial data monitor
55 | 0x20240430 | ALL | RO | QSFP, SFP and VX link loss counters
56 | 0x20240510 | ALL | RO | sfp_rx_packet_counter
57 | same | ALL | RO | sfp_tx_packet_counter
58 | same | ALL | RO | qsfp_rx_packet_counter[0]
59 | same | ALL | RO | qsfp_tx_packet_counter
60 | same | ALL | RO | vx_rx_packet_counter[0]
61 | same | ALL | RO | vx_tx_packet_counter
62 | 0x20240719 | CDM | RO | cdm_hitmap_period, ports 0, 1
63 | same | CDM | RO | ports 2, 3
64 | same | CDM | RO | ports 4, 5
65 | same | CDM | RO | ports 6, 7
66 | same | CDM | RO | ports 8, 9
67 | same | CDM | RO | ports 10, 11
62 | 0x20260401 | CDM | ROL | vx_rx_hitmap_period latched by cmd_latch, VX ports 0, 1
63 | same | CDM | ROL | ports 2, 3
64 | same | CDM | ROL | ports 4, 5
65 | same | CDM | ROL | ports 6, 7
66 | same | CDM | ROL | ports 8, 9
67 | same | CDM | ROL | ports 10, 11
68 | 0x20240814 | ALL | RW | GPS control and status
69 | 0x20241104 | CDM | ROLB | vx_tx_trg_packet_counter | counter of TRG packets CDM->VX
70 | 0x20241104 | CDM | ROLB | vx_tx_tsm_packet_counter | counter of TSM packets CDM->VX
71 | 0x20241104 | ALL | RO | packet error bits
72,73 | 0x20241104 | CDM | ROL | vx_tx_trg_data_latched[63:0] trigger packet data
74,75 | 0x20241104 | CDM | ROL | vx_tx_tsm_data_latched[63:0] tsm packet data (truncated to 64 bits)
76 | 0x20241104 | CDM | ROLB | cdm_hitmap_trigger_counter_latched, counter of hitmap triggers generated in the CDM
77,78,79 | 0x20241104 | CDM | ROL | vx_rx_hitmap_data_latched | VX_RX 80 bits of HITMAP packet from VX
80 | 0x20241209 | CDM | RO | vx_lvds_rx_link_status | VX LVDS link status
81 | 0x20241209 | CDM | RO | vx_lvds_rx_trg_data | VX LVDS TRG packet first 32 bits
81 | 0x20251010 | CDM | - | not used | not used
82 | 0x20241209 | CDM | RO | vx_lvds_rx_tsm_data | VX LVDS TSM packet first 32 bits
82 | 0x20251010 | CDM | - | not used | not used
83 | 0x20241209 | CDM | ROL | vx_lvds_rx_decoded_packet_counter_latched | VX LVDS decoded packet counter
84 | 0x20241209 | CDM | ROL | vx_lvds_rx_packet_error_counter_latched | VX LVDS packet error counter
85 | 0x20250417 | CDM | ROLB | vx_tx_trg_counter | VX LVDS TRG counter (VX LVDS pair 12)
86 | 0x20250417 | CDM | ROLB | vx_tx_tsm_counter | VX LVDS TSM counter (VX LVDS pair 13)
87 | 0x20250417 | CDM | ROLB | vx_tx_veto_counter | VX LVDS Veto counter (VX LVDS pair 13)
88 | 0x20250422 | GDM | ROLB | trg_in_lost_counter | counter of TRG killed by gdm_busy
89 | 0x20250422 | ALL | RO | fifo_from_fpga_lost_packet_counter | counter of packets lost due to FIFO full
90 | 0x20250428 | CDM | ROL | vx_bsy_status_latched | status of VX busy, CDM busy, CDM veto
91 | 0x20250428 | CDM | ROL | vx_hitmap_status_latched | status of VX hitmap
92 | 0x20250428 | GDM | ROL | cdm_bdy_status_latched | status of CDM busy, GDM busy, GDM veto
93 | 0x20250508 | GDM | ROL | gdm_hitmap_status_latched | status of GDM hitmap
94 | 0x20250508 | GDM | ROLB | gdm_hitmap_trigger_counter_latched | counter of hitmap triggers generated in the GDM
95 | 0x20250512 | GDM | ROLB | gdm_trig48_counter_latched | counter of TRIG48 trigger packets
96 | 0x20250512 | GDM | ROLB | gdm_trig48_lost_counter_latched | counter of TRIG48 triggers lost to busy
97 | 0x20250512 | CDM | ROLB | vx_tx_trig48_packet_counter_latched | counter of TRIG48 packets from CDM to VX
98, 99 | 0x20250512 | GDM | RW | trig48_ctrl48, trig48_ctrl16 | GDM TRIG48 test registers
100,101 | 0x20251008 | CDM | ROL | vx_lvds_rx_trig48_{data,bitmap48}_latched | VX LVDS TRIG48 packet first 32 bits, bitmap48 first 32 bits
102 | 0x20251008 | ALL | RO | cmd_counter | counter of reg0 commands
103,104,105 | 0x20260401 | CDM | ROL | vx_rx_hitmap_vx_id_latched | VX ID from VX hitmap packets, VX ports 0..11, in order, latched by cmd_latch
106 | 0x20260401 | CDM | RW | conf_vx_rx_hitmap_cumulative_period | period of CDM CUMULATIVE_HITMAP packets, set to 0 to disable them.
</pre>

=== Register 0 0x80013000 ds20k version ===

on read: ds20k version 0xYYYYMMDD

on write:
* 0 - noop - as of version 0x20240118, write a zero after writing a command
* 1 - cmd_reset - reset logic to good state
* 2 - cmd_arm_ts - arm timestamp reset
* 3 - cmd_trg - issue a trigger
* 4 - cmd_tsm - issue a tsm
* 5 - cmd_vx_rx_reset - reset the VX receive path
* 6 - cmd_vx_tx_reset - reset the VX transmit path
* 7 - cmd_gdm_trig48 - generate a TRIG48 trigger and data packet (GDM)
* 8 - cmd_trg_pulser_reset - reset the trigger pulser
* 9 - cmd_tsm_pulser_reset - reset the tsm pulser
* 10 - cmd_bor_start - start begin-of-run trigger sequence
* 11 - cmd_bor_clear - after run has started, clear begin-of-run status bits
* 12 - cmd_latch - latch counters & etc into AXI registers for coherent readout
* 13 - cmd_cdm_hitmap_trigger - generate a hitmap trigger and data packet (CDM)

=== Register 1 0x80013004 scratch ===

scratch read-write register

=== Register 2 0x80013008 input and output config ===

<pre>
bit | version | fpga name | description
0 | ALL | lemo_enable | enable LEMO input 1
1 | | | 2
2 | | | 3
3 | | | 4
4 | ALL | lemo_invert | invert LEMO input 1
5 | | | 2
6 | | | 3
7 | | | 4
9 | ALL | ext_out_disable | disable LEMO output 1
10 | | | 2
11 | ALL | ext_out_invert | invert LEMO output 1
12 | | | 2
</pre>

=== Register 3 0x8001300C FP_LED control ===

<pre>
wire [15:0] led_out_mux_sel = register_data_in[3][15:0];
wire [3:0] led_out_invert = register_data_in[3][19:16];
</pre>

<pre>
led_out_mux_sel is 4 groups (one per LED) of 4 bits (choice 0..15):
mux | version | fpga name | description
0 | ALL | | power on default
1 | ALL | led_out_reg | register 7 bits
2 | 0x20231013 | pll_locked | clock chip PLL is locked
3 | 0x20231013 | sfp_link_status | SFP link is good
3 | 0x20240118 | sfp_link_status or qsfp_tx_link_rx_status | SFP/QSFP link is good
4 | ALL | lemo_in_sync[1] | LEMO input 1
5 | ALL | lemo_in_sync[2] | LEMO input 2
6 | ALL | lemo_in_sync[3] | LEMO input 3
7 | ALL | lemo_in_sync[4] | LEMO input 4
8 | ALL | ext_out[1] | LEMO output 1
9 | ALL | ext_out[2] | LEMO output 2
A | 0x20231013 | trg_in | trigger
B | 0x20231013 | tsm_in | time slice marker
C | 0x20240118 | gdm_bsy | GDM busy: OR of all CDM busy
D | 0x20240118 | cdm_bsy | CDM busy: OR of all VX busy
E | 0x20240118 | cdm_veto | GDM busy -> GDM veto -> CDM veto -> VX trigger veto
F | ALL | | fixed logic level 1
</pre>

=== Register 4 0x80013010 LEMO OUT control ===

<pre>
wire [7:0] ext_out_mux_sel = register_data_in[4][7:0];
</pre>

<pre>
ext_out_mux_sel is 2 groups (one per LEMO) of 4 bits (choice 0..15):

mux | version | fpga name | description
0 | ALL | | power on default
1 | ALL | ext_out_reg | register 7
2 | ALL | trg_pulser | pulser trigger
3 | 0x20240724 | vx1_tx_out[2] | vx1 serial data out
4 | 0x20240724 | vx1_rx[1] | vx1 serial data in
5 | 0x20240724 | vx_rx_iob[0] | vx1 serial data in captured by IOB register
6 | ALL | lemo_in_async[1]| test synchronizer
7 | ALL | lemo_in_sync[1] | test synchronizer
8 | 0x20240724 | trg_in_pulse | trigger signal
9 | 0x20240724 | tsm_in_pulse | time slice marker signal
A | 0x20240118 | cdm_bsy | CDM busy from VX
B | 0x20240118 | gdm_bsy | GDM busy from CDM
C | 0x20240118 | cdm_veto | veto from GDM to CDM to VX
D | 0x20240118 | vx1_rx[1] | serial data VX to CDM
E | not used (sink)
F | ALL | 1 | fixed logic level 1
</pre>

=== Register 5 0x80013014 VX_RX status ===

<pre>
assign register_data_out[5] = {
vx8_rx[3], vx8_rx[2], vx8_rx[1], vx8_rx[0],
vx7_rx[3], vx7_rx[2], vx7_rx[1], vx7_rx[0],
vx6_rx[3], vx6_rx[2], vx6_rx[1], vx6_rx[0],
vx5_rx[3], vx5_rx[2], vx5_rx[1], vx5_rx[0],
vx4_rx[3], vx4_rx[2], vx4_rx[1], vx4_rx[0],
vx3_rx[3], vx3_rx[2], vx3_rx[1], vx3_rx[0],
vx2_rx[3], vx2_rx[2], vx2_rx[1], vx2_rx[0],
vx1_rx[3], vx1_rx[2], vx1_rx[1], vx1_rx[0]
};
</pre>

=== Register 6 0x80013018 VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state ===

<pre>
assign register_data_out[6] = {
vx2_tx_out[2], vx2_tx_out[1], vx2_tx_out[0], vx1_tx_out[2],
vx1_tx_out[1], vx1_tx_out[0], ext_out[2], ext_out[1],
fp_led_out[3], fp_led_out[2], fp_led_out[1], fp_led_out[0],
ext_in_lv[4], ext_in_lv[3], ext_in_lv[2], ext_in_lv[1],
vx12_rx[3], vx12_rx[2], vx12_rx[1], vx12_rx[0],
vx11_rx[3], vx11_rx[2], vx11_rx[1], vx11_rx[0],
vx10_rx[3], vx10_rx[2], vx10_rx[1], vx10_rx[0],
vx9_rx[3], vx9_rx[2], vx9_rx[1], vx9_rx[0]
};
</pre>

<pre>
bit | version | fpga name | description
0 | ALL | vx9_rx | VX_RX
1 | | |
2 | | |
3 | | |
4 | ALL | vx10_rx | VX_RX
5 | | |
6 | | |
7 | | |
8 | ALL | vx11_rx | VX_RX
9 | | |
10 | | |
11 | | |
12 | ALL | vx12_rx | VX_RX
13 | | |
14 | | |
15 | | |
16 | ALL | ext_in_lv | LEMO inputs
17 | | |
18 | | |
19 | | |
20 | ALL | FP_LED | FP_LEDs
21 | | |
22 | | |
23 | | |
24 | ALL | ext_out[1] | LEMO outputs
25 | | ext_out[2] |
26 | ALL | vx1_tx | VX1_TX
27 | | |
28 | | |
29 | ALL | vx2_tx | VX2_TX
30 | | |
31 | | |
</pre>

=== Register 7 0x8001301C LED_OUT, EXT_OUT, VX_TX outputs ===

<pre>
wire [3:0] led_out_reg = register_data_in[7][3:0];
wire [2:1] ext_out_reg = register_data_in[7][5:4];
// register_data_in[7][6];
// register_data_in[7][7];
wire [7:0] vx_tx_out_reg = register_data_in[7][15:8];
</pre>

<pre>
bit | version | fpga name | description
0 | ALL | led_out_reg | FP_LED 1
1 | | | 2
2 | | | 3
3 | | | 4
4 | ALL | ext_out_reg | LEMO OUT 1
5 | | | 2
6 | - | |
7 | - | |
8 | ALL | vx_tx_out_reg | VX1_TX 0
9 | | | 1
10 | | | 2
11 | | | -
12 | ALL | | VX2_TX 0
13 | | | 1
14 | | | 2
15 | | | -
16 | - | |
17 | | |
18 | | |
19 | | |
20 | | |
21 | | |
22 | | |
23 | | |
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 8 0x80013020 VX_TX config ===

<pre>
wire [3:0] vx_tx_mux_sel = register_data_in[8][3:0];
wire vx_tx_from_sfp = register_data_in[8][30];
wire vx_tx_loopback = register_data_in[8][31];
</pre>

<pre>
vx_tx_mux_sel is 4 bits (choice 0..15):
0 - power on default, control by vx_tx_out_reg
1 - GDM
2 - CDM
3 - pulser loopback test
4 - pulser loopback test
5 - 62.5 MHz output
6 - 125 MHz output
7 - trg, tsm, serial
8 - trg, tsm, lvds serial rx to serial tx loopback
9 - GPS box control (ds20k rev 0x20240814)
10
11
12
13
14
15 - production config: trg, veto, serial
</pre>

=== Register 9 trg and tsm source ===

from version 0x20240724

<pre>
wire [15:0] trg_src_mask = register_data_in[9][15:0];
wire [31:16] tsm_src_mask = register_data_in[9][31:16];
</pre>

<pre>
wire [15:0] trg_src_bits =
{
1'b0, // 15
1'b0, // 14
vx_tx_tsm_done, // 13
vx_tx_trg_done, // 12

1'b0, // gdm_hitmap_trigger, // 11
cdm_hitmap_trigger, // 10
sfp_rx_tsm, // 9
sfp_rx_trg, // 8

sfp_rx_data[1], // 7
sfp_rx_data[0], // 6
tsm_pulser, // 5
trg_pulser, // 4

lemo_in_sync[4], // 3
lemo_in_sync[3], // 2
lemo_in_sync[2], // 1
lemo_in_sync[1] // 0
};

wire [15:0] trg_bits = trg_src_bits & trg_src_mask;
wire [15:0] tsm_bits = trg_src_bits & tsm_src_mask;
</pre>

before that:

<pre>
wire [7:0] trg_src_mask = register_data_in[9][7:0];
wire [7:0] tsm_src_mask = register_data_in[9][15:8];
wire trg_pulser_enable = register_data_in[9][16];
wire tsm_pulser_enable = register_data_in[9][17];
wire trg_software = register_data_in[9][18];
wire tsm_software = register_data_in[9][19];
// bits 20:31 not used
</pre>

<pre>
wire [7:0] xxx_src_bits =
{
sfp_rx_data[1],
sfp_rx_data[0],
tsm_pulser & tsm_pulser_enable,
trg_pulser & trg_pulser_enable,
lemo_in_sync[4],
lemo_in_sync[3],
lemo_in_sync[2],
lemo_in_sync[1]
};
</pre>

<pre>
trg_src_mask and tsm_src_mask bits:
0 - LEMO IN 1
1 - LEMO IN 2
2 - LEMO IN 3
3 - LEMO IN 4
4 - trg_pulser
5 - tsm_pulser
6 - sfp_rx_data[0] // to become sfp_trg_in, selected from sfp_rx_data[0], cdm_rx trg_out and gdm trigger packet
7 - sfp_rx_data[1] // to become sfp_tsm_in, selected from sfp_rx_data[1], cdm_rx tsm_out and gdm tsm packet
</pre>

=== Register 10 0x80013028 status register ===

<pre>
bit | version | fpga name | description
0 | 0x20231013 | pll_locked | clock chip PLL is locked
1 | 0x20240118 | ts_reset_armed | timestamp reset is armed
2 | 0x20240118 | qsfp_tx_link_rx_status | QSFP link status is good for all enabled ports
3 | 0x20240424 | vx_tx_link_rx_status | VX link status is good for all enabled ports
4 | 0x20240118 | cdm_bsy | VX busy grand-or
5 | 0x20240118 | gdm_bsy | QSFP busy grand-or
6 | 0x20240118 | gdm_veto | gdm_veto = gdm_busy
7 | 0x20240118 | cdm_veto | CDM veto from GDM to VX
8 | 0x20240725 | bor_started | begin-of-run sequence started
9 | 0x20240725 | bor_finished | begin-of-run sequence fininished, see commands 10 and 11
10 | | |
11 | | |
12 | | |
13 | | |
14 | | |
15 | | |
16 | | |
17 | | |
18 | | |
19 | | |
20 | | |
21 | | |
22 | | |
23 | | |
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 11 0x8001302C trg_counter ===

trigger counter

=== Register 12 0x80013030 tsm_counter ===

time slice marker counter

=== Register 13 0x80013034 GPS 1pps period ===

GPS 1pps period in 8 ns clocks

=== Register 14 0x80013038 Rb clock 1pps period ===

PRS-10 Rb clock 1pps output period in 8 ns clocks

=== Register 15 0x8001303C SFP RX status ===

<pre>
bit | ds20k version | fpga signal name | description
0 | ALL | sfp_rx_data[15:0] | cdm sfp received data
16 | same | sfp_rx_data_is_k[0] |
17 | same | sfp_rx_data_is_k[1] |
18 | | 0 |
19 | 0x20231204 | sfp_rx_sel_lpb | sfp tx->rx loopback
20 | 0x20231013 | sfp_link_status | sfp link connected, exchanging data
21 | same | sfp_link_rx_status | sfp link receiving correct idle pattern from GDM TX
22 | same | sfp_link_error | sfp link receiver error (badk or overflow)
23 | same | sfp_rx_data_error | sfp transceiver state machine is in error state
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 16 SFP TX control ===

<pre>
wire [15:0] sfp_tx_data_reg = register_data_in[16][15:0]; // SFP TX data K-code
wire [1:0] sfp_tx_ctrl_reg = register_data_in[16][17:16]; // SFP TX data K-code
// 18
// 19
// 23:20
// 27:24
wire sfp_rx_sel_lpb = register_data_in[16][28]; // TX->RX serial loopback
wire sfp_tx_sel_lpb = register_data_in[16][29]; // RX->TX serial loopback
wire sfp_tx_sel_trg = register_data_in[16][30]; // 16 individual bits
wire sfp_tx_sel_reg = register_data_in[16][31]; // TX data from register 16 bits 17:0
</pre>

=== Register 17-22 QSFP RX data ===

QSFP RX data links 0..11

=== Register 23 QSFP TX control ===

<pre>
wire [15:0] qsfp_tx_data_reg = register_data_in[23][15:0];
wire [1:0] qsfp_tx_ctrl_reg = register_data_in[23][17:16];

wire qsfp_tx_enable_trg = register_data_in[23][24]; // enable QSFP TX trg_in_pulse k-code
wire qsfp_tx_enable_tsm = register_data_in[23][25]; // enable QSFP TX tsm_in_pulse k-code

//wire qsfp_rx_sel_lpb = register_data_in[23][28]; // TX->RX loopback
wire qsfp_tx_sel_lpb = register_data_in[23][29]; // RX->TX loopback
wire qsfp_tx_sel_trg = register_data_in[23][30]; // 16 individual bits
wire qsfp_tx_sel_reg = register_data_in[23][31]; // data from register
</pre>

=== Register 24 0x80013060 trigger pulser period ===

trigger pulser period in units of 8 ns (125 MHz clock)

=== Register 25 0x80013064 trigger burst pulser ===

<pre>
wire [7:0] conf_burst_count = conf_pulser_burst_ctrl[31:24];
wire [23:0] conf_burst_period = conf_pulser_burst_ctrl[23:0];
</pre>

=== Register 26 0x80013068 tsm pulser period ===

time slice marker period in units of 8 ns (125 MHz clock)

=== Register 27 0x8001306C data write fifo ===

<pre>
wire fifo_reset = register_data_in[27][31];
wire fifo_to_fpga_wr1 = register_data_in[27][27];
wire fifo_to_fpga_wr2 = register_data_in[27][26];
wire [16:0] fifo_to_fpga_din = register_data_in[27][16:0];
</pre>

<pre>
assign register_data_out[27][31:24] = register_data_in[27][31:24]; // echo write bits
assign register_data_out[27][23] = fifo_to_fpga_full;
assign register_data_out[27][22] = fifo_to_fpga_empty;
</pre>

=== Register 28 0x80013070 data read fifo ===

<pre>
wire fifo_reset = register_data_in[28][31];
wire fifo_from_fpga_rd1 = register_data_in[28][25];
wire fifo_from_fpga_rd2 = register_data_in[28][24];
</pre>

<pre>
assign register_data_out[28][31:24] = register_data_in[28][31:24]; // echo write bits
assign register_data_out[28][21] = fifo_from_fpga_full;
assign register_data_out[28][20] = fifo_from_fpga_empty;
assign register_data_out[28][16:0] = fifo_from_fpga_dout;
</pre>

=== Register 29 0x80013074 packet routing ===

Control packet routing and loopbacks:

<pre>
wire [3:0] dn_route_ctrl = register_data_in[29][3:0];
wire [3:0] up_route_ctrl = register_data_in[29][7:4];
wire [3:0] fifo_to_fpga_route_ctrl = register_data_in[29][11:8];
//wire [3:0] spare_route_ctrl = register_data_in[29][15:12];

wire dn_mux_trg_enable = register_data_in[29][16];
wire dn_mux_tsm_enable = register_data_in[29][17];
wire dn_mux_sfp_rx_fifo_enable = register_data_in[29][18];
// 19
// 20..23

wire up_mux_vx_rx_enable = register_data_in[29][24];
// 25
// 26
// 27
wire fifo_from_fpga_hitmap_enable = register_data_in[29][28];
wire fifo_from_fpga_cdm_busy_enable = register_data_in[29][29];
wire fifo_from_fpga_gdm_busy_enable = register_data_in[29][30];
wire fifo_from_fpga_tsm_enable = register_data_in[29][31];
</pre>

fifo_to_fpga output routing:
* 0 - to down packet mux
* 1 - to up packet mux
* 2 - to fifo_from_fpga mux
* 3 - not used

down packet mux inputs:
* fifo_to_fpga_0
* trg_pkt16 enabled by dn_mux_trg_enable
* tsm_pkt16 enabled by dn_mux_tsm_enable
* sfp_rx_fifo_pkt16 enabled by dn_mux_sfp_rx_fifo_enable
* up_pkt16_2 loopback from up packet mux

down packet mux output routing:
* 0 - to vx_tx_pkt16 to VX TX (vx_link_tx and vx_ser_tx)
* 1 - to fifo_from_fpga mux
* 2 - to up packet mux loopback
* 3 - to qsfp_tx_pkt16 to GDM QSFP TX (cdm_link_tx)

up packet mux inputs:
* vx_rx_pkt16 enabled by up_mux_vx_rx_enable data from 12 VX RX links
* fifo_to_fpga_1
* dn_pkt16_2 loopback from down packet mux

up packet mux output routing:
* 0 - to sfp_tx_pkt16 to CDM SFP TX (VX data to GDM)
* 1 - to fifo_from_fpga mux
* 2 - to down packet mux loop loopback
* 3 - not used

internal_mux inputs:
* 0 - cdm_hitmap_pkt16 enabled by fifo_from_fpga_hitmap_enable (reg 29)
* 1 - cdm_busy_pkt16 enabled by fifo_from_fpga_cdm_busy_enable
* 2 - gdm_busy_pkt16 enabled by fifo_from_fpga_gdm_busy_enable
* 3 - gdm_hitmap_pkt16 enabled by fifo_from_fpga_hitmap_enable
* 4 - not used

debug_mux inputs:
* 0 - not used
* 1 - dn_pkt16_1
* 2 - up_pkt16_1
* 3 - qsfp0_rx_pkt16 enabled by fifo_from_fpga_qsfp0_rx_enable (from GDM QSFP link 0, there is no GDM QSFP 12-to-1 mux)
* 4 - not used

fifo_from_fpga mux inputs:
* 0 - fifo_to_fpga_2
* 1 - tsm2_pkt16 enabled by fifo_from_fpga_tsm_enable (reg 29)
* 2 - not used
* 3 - debug_pkt16 from debug_mux
* 4 - internal_pkt16 from internal_mux

=== Register 30 0x80013078 qsfp link status ports 0..7 ===

<pre>
assign register_data_out[30] =
{
qsfp_rx_data_error[7], qsfp_link_error[7], qsfp_link_status[7], qsfp_link_rx_status[7], // 7
qsfp_rx_data_error[6], qsfp_link_error[6], qsfp_link_status[6], qsfp_link_rx_status[6], // 6
qsfp_rx_data_error[5], qsfp_link_error[5], qsfp_link_status[5], qsfp_link_rx_status[5], // 5
qsfp_rx_data_error[4], qsfp_link_error[4], qsfp_link_status[4], qsfp_link_rx_status[4], // 4
qsfp_rx_data_error[3], qsfp_link_error[3], qsfp_link_status[3], qsfp_link_rx_status[3], // 3
qsfp_rx_data_error[2], qsfp_link_error[2], qsfp_link_status[2], qsfp_link_rx_status[2], // 2
qsfp_rx_data_error[1], qsfp_link_error[1], qsfp_link_status[1], qsfp_link_rx_status[1], // 1
qsfp_rx_data_error[0], qsfp_link_error[0], qsfp_link_status[0], qsfp_link_rx_status[0] // 0
};
</pre>

=== Register 31 0x8001307C qsfp link status ports 8..11 ===

<pre>
assign register_data_out[31] =
{
4'b0000,
4'b0000,
4'b0000,
4'b0000,
qsfp_rx_data_error[11], qsfp_link_error[11], qsfp_link_status[11], qsfp_link_rx_status[11], // 11
qsfp_rx_data_error[10], qsfp_link_error[10], qsfp_link_status[10], qsfp_link_rx_status[10], // 10
qsfp_rx_data_error[9], qsfp_link_error[9], qsfp_link_status[9], qsfp_link_rx_status[9], // 9
qsfp_rx_data_error[8], qsfp_link_error[8], qsfp_link_status[8], qsfp_link_rx_status[8] // 8
};
</pre>

=== Register 32 0x80013080 bitmap of enabled qsfp ports ===

<pre>
wire [11:0] qsfp_mask = register_data_in[32][11:0];
wire qsfp_bsy_force = register_data_in[32][12];
wire gdm_veto_enable = register_data_in[32][13];
// not used = register_data_in[32][15:14];
wire [15:0] gdm_veto_extend = register_data_in[32][31:16];
</pre>

On the GDM:

qsfp_rx_bsy[11..0] are pulses received from the CDMs

qsfp_rx_bsy_or is the grand-or of qsfp_rx_bsy masked by qsfp_mask

if qsfp_rx_bsy_or is high, gdm_bsy goes up and stays up for vx_bsy_extend*2 clocks

gdm_veto = gdm_bsy (bsy of any one VX causes trigger veto to all of them)

if (gdm_veto_enable) then gdm_veto will block trg_in_pulse

gdm_veto transition 0->1 causes gdm_veto_pulse. as long as gdm_veto is high, gdm_veto_pulse is generated every gdm_veto_extend clocks

gdm_veto_pulse is sent to all CDMs.

For this to work right, cdm_bsy_extend should not be bigger than gdm_veto_extend.

=== Register 33 0x80013084 bitmap of enabled VX ports ===

<pre>
wire [11:0] vx_mask = register_data_in[33][11:0];
wire vx_bsy_force = register_data_in[33][12];
// not used = register_data_in[33][15:13];
wire [15:0] vx_bsy_extend = register_data_in[33][31:16];
</pre>

On the CDM:

cdm_bsy is a grand-or of all vx_bsy masked by vx_mask (list of active VXes).

vx_bsy_extend controls how often state of cdm_bsy is sent to the GDM. when cdm_bsy goes 0->1, we send a cdm_bsy_pulse and keep resending it every vx_bsy_extend clocks as long as cdm_bsy stays high.

cdm_bsy_pulse is sent to the GDM.

sfp_rx_veto is the received from the GDM

if sfp_rx_veto goes up, cdm_veto goes up and stays up for gdm_veto_extend clocks.

for this to work right, CDM gdm_veto_extend must be bigger than GDM gdm_veto_extend.

cdm_veto goes to VXes on v1_tx_out[1] which is LVDS input 13.

=== Register 34, 35 0x80013088, 8C current timestamp ===

current 64-bit timestamp, 125 MHz

=== Register 36, 37 0x80013090, 94 old timestamp ===

old 64-bit timestamp, 125 MHz. timestamp saved at run start when it is reset to 0.

=== Register 38, 39, 40 0x80013098, 9C, A0 VX busy counters ===

8 bits per VX port, counters overflow to 255, reset at run start.

=== Register 41, 42, 43 0x800130A4, A8, AC QSFP busy counters ===

8 bits per QSFP port, counters overflow to 255, reset at run start.

=== Register 44, 45, 46, 47, 48 CDM and GDM busy and veto counters ===

* cdm_busy = grand-or of all VX busy for enabled VXes
* gdm_busy = grand-or of all CDM busy for enabled CDMs
* gdm_veto = gdm_busy

<pre>
44 | lo 16 bits | cdm_bsy_up_counter | CDM busy, increments when cdm_busy goes 0->1
44 | hi 16 bits | cdm_bsy_pulse_counter | CDM busy to GDM, increments for each cdm_bsy_pulse sent to the GDM

45 | lo 16 bits | cdm_veto_pulse_counter | CDM veto from GDM, increments for each sfp_rx_veto received from the GDM
45 | hi 16 bits | cdm_veto_up_counter | CDM veto to VX, increments each time cdm_veto is set to 1.

46 | lo 16 bits | gdm_bsy_pulse_counter | GDM busy from CDM, increments for each qsfp_rx_busy received from CDMs (unless they overlap)
46 | hi 16 bits | gdm_bsy_up_counter | GDM busy, increments each time gdm_bsy goes 0->1

47 | lo 16 bits | gdm_bsy_refresh_counter | GDM busy, increments each time gdm_bsy is extended by new qsfp_rx_busy
47 | hi 16 bits | spare |

48 | lo 16 bits | gdm_veto_up_counter | GDM veto, increments each time gdm_veto goes 0->1
48 | hi 16 bits | gdm_veto_pulse_counter | GDM veto to CDM, increments for each gdm_veto_pulse sent to the CDM
</pre>

=== Register 49, 50, 51 0x800130xx VX RX serial data monitor ===

VX RX data, 8-bit per VX channel. k-bit is omitted.

<pre>
assign register_data_out[49][7:0] = vx_rx_data[0]; // vx1
assign register_data_out[49][15:8] = vx_rx_data[1]; // vx2
assign register_data_out[49][23:16] = vx_rx_data[2]; // vx3
assign register_data_out[49][31:24] = vx_rx_data[3]; // vx4

assign register_data_out[50][7:0] = vx_rx_data[4]; // vx5
assign register_data_out[50][15:8] = vx_rx_data[5]; // vx6
assign register_data_out[50][23:16] = vx_rx_data[6]; // vx7
assign register_data_out[50][31:24] = vx_rx_data[7]; // vx8

assign register_data_out[51][7:0] = vx_rx_data[8]; // vx9
assign register_data_out[51][15:8] = vx_rx_data[9]; // vx10
assign register_data_out[51][23:16] = vx_rx_data[10]; // vx11
assign register_data_out[51][31:24] = vx_rx_data[11]; // vx12
</pre>

=== Register 52, 53 VX link status ===

<pre>
assign register_data_out[52] =
{
vx_rx_error[7], vx_link_error[7], vx_link_status[7], vx_link_rx_status[7], // 7
vx_rx_error[6], vx_link_error[6], vx_link_status[6], vx_link_rx_status[6], // 6
vx_rx_error[5], vx_link_error[5], vx_link_status[5], vx_link_rx_status[5], // 5
vx_rx_error[4], vx_link_error[4], vx_link_status[4], vx_link_rx_status[4], // 4
vx_rx_error[3], vx_link_error[3], vx_link_status[3], vx_link_rx_status[3], // 3
vx_rx_error[2], vx_link_error[2], vx_link_status[2], vx_link_rx_status[2], // 2
vx_rx_error[1], vx_link_error[1], vx_link_status[1], vx_link_rx_status[1], // 1
vx_rx_error[0], vx_link_error[0], vx_link_status[0], vx_link_rx_status[0] // 0
};

assign register_data_out[53] =
{
//4'b0000,
1'b0, ~vx_rx_deser_rdy[0], vx_rx_code_err[0], vx_rx_disp_err[0],
//4'b0000,
//4'b0000,
//4'b0000,
vx_rx_monitor[0],
vx_rx_error[11], vx_link_error[11], vx_link_status[11], vx_link_rx_status[11], // 11
vx_rx_error[10], vx_link_error[10], vx_link_status[10], vx_link_rx_status[10], // 10
vx_rx_error[9], vx_link_error[9], vx_link_status[9], vx_link_rx_status[9], // 9
vx_rx_error[8], vx_link_error[8], vx_link_status[8], vx_link_rx_status[8] // 8
};
</pre>

vx_rx_monitor (12-bit) is from deserializer_10b.sv:

<pre>
assign monitor_out[9:0] = lastByte[9:0];
assign monitor_out[10] = comma;
assign monitor_out[11] = ready;
</pre>

=== Register 54 VX TX serial data monitor ===

* contents of vx_tx_monitor from vx_ser_tx.sv:
<pre>
assign monitor_out[8:0] = data_to_encoder; // 8-bit + k
assign monitor_out[9] = valid;
assign monitor_out[15:10] = 0;
assign monitor_out[25:16] = encoded_data; // 10-bit
assign monitor_out[26] = encoded_valid;
assign monitor_out[27] = 0;
assign monitor_out[31:28] = 0;
</pre>

=== Register 55 QSFP, SFP, VX link loss counters ===

<pre>
assign register_data_out[55] = {
vx_rx_error_counter,
qsfp_link_rx_status_drop_counter,
sfp_link_rx_status_drop_counter,
vx_link_rx_status_drop_counter
};
</pre>

<pre>
24..31 - CDM VX RX error counters, count any errors in the VX RX data path (bad serial data, fifo overflow, etc)
16..23 - GDM QSFP link loss counter, increments on qsfp_tx_link_rx_status 1->0 (reg10)
8..15 - CDM SFP link loss counter, increments on sfp_link_rx_status 1->0 (reg15)
0...7 - CDM VX link loss counter, increments on vx_tx_link_rx_status 1->0 (reg10)
</pre>

=== Register 68 GPS control and status ===

<pre>
assign register_data_out[68] =
{
8'b00000000, // 23+8
rb_1pps_counter, // 16+8 bits
gps_1pps_counter, // 8+8 bits
1'b0,
1'b0,
rb_ser_in, // 5
gps_data_in, // 4
gps_aux_out, // 3
gps_aux_in, // 2
rb_1pps_in, // 1
gps_1pps_in // 0
};
</pre>

=== Register 71 packet error bits ===

<pre>
assign register_data_out[71] =
{
8'b00000000, // 24+8 bits
1'b0, // 23
1'b0, // 22
fifo_to_fpga_full_latch, // 21
fifo_from_fpga_full_latch, // 20
gdm_hitmap_fifo_full_latch, // 19
gdm_busy_fifo_full_latch, // 18
cdm_busy_fifo_full_latch, // 17
cdm_hitmap_fifo_full_latch, // 16
1'b0, // 15
vx_rx_hitmap_error_latch, // 14
hitmap_encode_error_latch, // 13
gdm_trig48_encode_error_latch, // 12
1'b0, // vx_tx_trig48_pkt8_error_latch, // 11
tsm2_encode_error_latch, // 10
tsm_encode_error_latch, // 9
trg_encode_error_latch, // 8
gdm_hitmap_error_latch, // 7
gdm_hitmap_encode_error_latch, // 6
gdm_busy_encode_error_latch, // 5
cdm_busy_encode_error_latch, // 4
cdm_hitmap_encode_error_latch, // 3
cdm_hitmap_decode_error_latch, // 2
1'b0, // vx_tx_tsm_pkt8_error_latch, // 1
1'b0, // vx_tx_trg_pkt8_error_latch // 0
};
</pre>

=== Register 90 VX busy status register ===

<pre>
vx_bsy_status_latched[11:0] <= vx_bsy[11:0];
vx_bsy_status_latched[23:12] <= vx_bsy_collect[11:0];
vx_bsy_status_latched[24] <= cdm_bsy;
vx_bsy_status_latched[25] <= cdm_bsy_collect;
vx_bsy_status_latched[26] <= cdm_veto;
vx_bsy_status_latched[27] <= cdm_veto_collect;
vx_bsy_status_latched[28] <= 0;
vx_bsy_status_latched[29] <= 0;
vx_bsy_status_latched[30] <= 0;
vx_bsy_status_latched[31] <= cdm_bsy_stuck;
</pre>

=== Register 91 VX hitmap status register ===

<pre>
hitmap_status_latched[11:0] <= hitmap_or12[11:0];
hitmap_status_latched[23:12] <= hitmap_or12_collect[11:0];
hitmap_status_latched[24] <= hitmap_grand_or;
hitmap_status_latched[25] <= hitmap_grand_or_collect;
hitmap_status_latched[26] <= 0;
hitmap_status_latched[27] <= 0;
hitmap_status_latched[28] <= 0;
hitmap_status_latched[29] <= 0;
hitmap_status_latched[30] <= 0;
hitmap_status_latched[31] <= hitmap_grand_or_stuck;
</pre>

=== Register 92 GDM busy status register ===

<pre>
gdm_bsy_status_latched[11:0] <= qsfp_rx_bsy[11:0];
gdm_bsy_status_latched[23:12] <= qsfp_rx_bsy_collect[11:0];
gdm_bsy_status_latched[24] <= gdm_bsy;
gdm_bsy_status_latched[25] <= gdm_bsy_collect;
gdm_bsy_status_latched[26] <= gdm_veto;
gdm_bsy_status_latched[27] <= gdm_veto_collect;
gdm_bsy_status_latched[28] <= 0;
gdm_bsy_status_latched[29] <= 0;
gdm_bsy_status_latched[30] <= 0;
gdm_bsy_status_latched[31] <= gdm_bsy_stuck;
</pre>

=== Register 93 GDM hitmap status register ===

<pre>
hitmap_status_latched[11:0] <= hitmap_grand_or_12[11:0];
hitmap_status_latched[23:12] <= hitmap_grand_or_12_collect[11:0];
hitmap_status_latched[24] <= hitmap_grand_or;
hitmap_status_latched[25] <= hitmap_grand_or_collect;
hitmap_status_latched[26] <= 0;
hitmap_status_latched[27] <= 0;
hitmap_status_latched[28] <= 0;
hitmap_status_latched[29] <= 0;
hitmap_status_latched[30] <= 0;
hitmap_status_latched[31] <= hitmap_grand_or_stuck;
</pre>

=== Register 98, 99 GDM TRIG48 test register and TRIG48_CTRL ===

<pre>
assign gdm_trig48_ctrl_bitmap48[31:0] = register_data_in[98][31:0];
assign gdm_trig48_ctrl_bitmap48[47:32] = register_data_in[99][15:0];

assign gdm_trig48_ctrl[15:0] = register_data_in[99][31:16];
</pre>

<pre>
wire enable_grand_or = trig48_ctrl[15];
wire enable_test_trig48 = trig48_ctrl[14];
</pre>

= Firmware registers branch develop_ko =

== Register map ==

<pre>
0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
1 | ALL | ALL | RW | read write scratch register
2 | ALL | CDM | ?? | MGT not used
3 | ALL | CDM | RO | MGT debug_data
4 | ALL | CDM | RW | clk_config_vec
5 | ALL | CDM | ?? | not used
6 | ALL | CDM | RO | CDM_link_data_processing:o_error_time
7 | ALL | CDM | RO | CDM_link_data_processing:o_error_count
</pre>

== register 0 0x80010000 ==

GDM:
<pre>
0 - gdm_link_interface:i_mgt_rst
2 - gdm_link_interface:i_link_down_latched_rst
8 - GDM_link_data_processing:i_rst
10..9 - GDM_link_data_processing:i_data_mode
</pre>

CDM:
<pre>
0 - cdm_link_interface:i_mgt_rst
2 - cdm_link_interface:i_link_down_latched_rst
8 - CDM_link_data_processing:i_rst
10..9 - CDM_link_data_processing:i_data_mode
</pre>

== register 1 0x80010004 ==

GDM:
<pre>
nlinks-1..0 - gdm_link_interface:i_rx_slide_trigger
</pre>

CDM:
<pre>
nlinks-1..0 - cdm_link_interface:i_rx_slide_trigger
</pre>

== register 2 0x80010008 ==

GDM:
<pre>
nlinks-1..0 - gdm_link_interface:o_link_power_good
nlinks+15..16 - gdm_link_interface:o_link_status
</pre>

CDM:
<pre>
nlinks-1..0 - cdm_link_interface:o_link_power_good
nlinks+15..16 - cdm_link_interface:o_link_status
</pre>

== register 3 0x8001000c ==

GDM: simple loopback register

CDM:
<pre>
31..0 - debug_data - cdm_link_interface:o_debug
</pre>

o_debug:
<pre>
rx_link_rst & rx_error & rx_link_up & rx_receiving_data &
std_logic_vector(rx_state_count) & tx_state_count_on_rx_clk & i_rx_ctrl3(0) &
i_rx_ctrl1(1 downto 0) & i_rx_ctrl0(1 downto 0) &
rx_data_is_k28p1_k28p5 &
i_rx_data;
</pre>

== register 4 0x80010010 ==

GDM write:
<pre>
0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS
</pre>

GDM read:
<pre>
0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS
4 - clk_config_vec(4) - CLK_LOSXTn_LS
5 - clk_config_vec(5) - CLK_LOLn_LS
6 - clk_config_vec(6) - CLK_INTn_LS
7 - constant 1
31..8 - constant 0
</pre>

== register 5 0x80010014 ==

not used

== register 6 0x80010018 ==

GDM:
<pre>
3..0 - GDM_link_data_processing:i_status_select
</pre>

CDM:
<pre>
31..0 - CDM_link_data_processing:o_error_count
</pre>

== register 7 0x8001001c ==

GDM:
<pre>
31..0 - GDM_link_data_processing:o_status_vector
</pre>

CDM:
<pre>
31..0 - CDM_link_data_processing:o_error_count
</pre>

= GDM, CDM, VX packet communications =

* timestamp math
<pre>
1 clock is 8 ns is 125 MHz
8 bits of clocks is 256 clocks is 2048 ns is ~2 usec
16 bits of clocks is ~500 usec is 0.5 msec
24 bits of clocks is ~134 msec
32 bits of clocks is ~34 sec
40 bits of clocks is ~8.7 ksec is 2.4 hours
48 bits of clocks is ~625 hours is ~26 days
56 bits of clocks is ~6.6 kdays is ~18 kyears
62 bits of clocks is ~10 Mhours is 427 kdays is ~1.1 kyears
64 bits of clocks is ~4.4 kyears
</pre>
* summary
<pre>
0x01 - not used
0x02 - 8 bytes - 4 words - 2 dwords - 1 qwords - TRG packet
0x03 - 12 bytes - 6 words - 3 dwords - 2 qwords - TRIG48 packet
0x10 - 26 bytes - 13 words - 7 dwords - 4 qwords - TSM packet
0x81 - 10 bytes - 5 words - 3 dwords - 2 qwords - VX_HITMAP packet
0x82 - 108 bytes - 54 words - 27 dwords - 14 qwords - CDM_HITMAP packet
0x83 - 12 bytes - 6 words - 3 dwords - 2 qwords - CDM_BUSY packet
0x84 - 12 bytes - 6 words - 3 dwords - 2 qwords - GDM_BUSY packet
0x85 - 12 bytes - 6 words - 3 dwords - 1 qwords - GDM_HITMAP packet
0x86 - 128 bytes - 64 words - 32 dwords - 16 qwords - CDM CUMULATIVE_HITMAP packet
</pre>
* 0x02 - TRG packet, 8 bytes, 80 adc clocks, 640 ns on lvds link
<pre>
0 - 0x02
1 - trg_counter[7:0]
2 - ts64 low byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 high byte 3
6 - trg_in_latch[7:0]
7 - trg_in_latch[15:8]
</pre>
* 0x03 - TRIG48 packet, 12 bytes, 120 adc clocks, 960 ns on lvds link
<pre>
0 - 0x03
1 - trg_counter[7:0]
2 - ts64 low byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 high byte 3
6 - vx_bitmap[7:0]
7 - vx_bitmap[15:8]
8 - vx_bitmap[23:16]
9 - vx_bitmap[31:24]
10 - vx_bitmap[39:32]
11 - vx_bitmap[47:40]
</pre>
* 0x10 - TSM packet, 26 bytes, 260 adc clocks, 2080 ns on lvds link
<pre>
0 - 0x10
1 - tsm_counter[7:0]
2 - gdm_ts64 low byte 0
3 - 1
4 - 2
5 - 3
6 - 4
7 - 5
8 - 6
9 - gdm_ts64 high byte 7
10 - gps_ts64 low byte 0
11 - 1
12 - 2
13 - 3
14 - 4
15 - 5
16 - 6
17 - gps_ts64 high byte 7
18 - gps_data64 low byte 0
19 - 1
20 - 2
21 - 3
22 - 4
23 - 5
24 - 6
25 - gps_data64 high byte 7
</pre>
* 0x81 - VX hitmap packet, 10 bytes, 100 adc clocks, 800 ns on lvds link, 48 ns on fiber link
<pre>
0 - 0x81
1 - VX ID
2 - hitmap low byte, nits 7:0
3 - 15:8
4 - 23:16
5 - 31:24
6 - ...:32
7 - ...
8 - ...
9 - hitmap low byte, bits 63:...
</pre>
* 0x82 - CDM hitmap packet, 108 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x82
1 - cdm_hitmap_trigger_counter[7:0]
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - cdm_hitmap_or12 byte 0 (7:0)
11 - cdm_hitmap_or12 byte 1 (11:8) plus 4 bits: 12=0, 13=0, 14=0, 15=cdm_hitmap_grand_or
12 - cdm_hitmap_data, low byte, 12*64 bits = 768 bits = 48 words = 96 bytes
...
107 - cdm_hitmap_data, high byte
</pre>
* 0x83 - CDM busy packet, 12 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x83
1 - cdm_busy_packet_counter[7:0], reset at begin of run
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - vx_bsy[7:0]
11 - {0,0,cdm_veto,cdm_bsy,vx_bsy[11:8]}
</pre>
* 0x84 - GDM busy packet, 12 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x84
1 - gdm_busy_packet_counter[7:0], reset at begin of run
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - qsfp_rx_bsy[7:0]
11 - {0,0,gdm_veto,gdm_bsy,qsfp_rx_bsy[11:8]}
</pre>
* 0x85 - GDM hitmap trigger packet, 12 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x85
1 - gdm_hitmap_packet_counter[7:0], reset at begin of run
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - hitmap_grand_or_12[7:0]
11 - {hitmap_grand_or,0,0,0,hitmap_grand_or_12[11:8]}
</pre>
* 0x86 - CDM cumulative hitmap packet, 128 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x86
1 - vx_rx_hitmap_packet_counter[7:0], reset at begin of run
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - reserved
11 - reserved
12 - repeating 12 times the 80-bit VX HITMAP packets: 0x81, VX ID, 64 bits of hitmap
</pre>

= AXI bus timing =

* AXI 100 MHz clock, 10 ns, 32-bit data
* AXI single-dword read: 36 clock repeat rate, 360 ns is 2.777 MHz, 4 bytes per transfer is 11.11 Mbytes/sec
* AXI single-qword read: 13 clock repeat rate, 130 ns is 7.7 MHz, 4 bytes per transfer is 30 Mbytes/sec, not accounting for the gap
* AXI memcpy read: 4 transfers at 13 clocks, gap, 4 transfers at 13 clocks, gap, etc. below 30 Mbytes/sec.
[[Image:Ds-dm-axi-read.png|100px]]
[[Image:Ds-dm-axi-read-64.png|100px]]
[[Image:Ds-dm-axi-read-memcpy.png|100px]]
* AXI single-dword write: 20 clocks repeat rate, 200 ns is 5 MHz, 4 bytes per transfer is 20 Mbytes/sec
* AXI single-qword write: 13+20 clocks repeat rate, 330 ns is 3 MHz, 16 bytes per burst is 48 Mbytes/sec
[[Image:Ds-dm-axi-write.png|100px]]
[[Image:Ds-dm-axi-write-64.png|100px]]

= AXI bus addresses =

* see AXI/AMBA addresses assigned inthe FPGA project: (s_axi/reg0 is the DS-DM AXI registers)
<pre>
daq00:ds-dm-gcdm$ grep assign_bd_address scripts/GDM_CDM_XU8_bd.tcl
assign_bd_address -offset 0x80010000 -range 0x00004000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs axi_register_interfa_0/s_axi/reg0] -force
assign_bd_address -offset 0x000400000000 -range 0x40000000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs ddr4/C0_DDR4_MEMORY_MAP/C0_DDR4_ADDRESS_BLOCK] -force
assign_bd_address -offset 0x80000000 -range 0x00010000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs system_management_wiz/S_AXI_LITE/Reg] -force
daq00:ds-dm-gcdm$
</pre>

* see AXI/AMBA addresses exported from FPGA project to Linux kernel: (uio for debug bridge should say "debug bridge")
<pre>
root@gdm0:~# cat /sys/class/uio/uio*/name
axi-pmon
axi-pmon
axi-pmon
axi-pmon
root@gdm0:~#
root@gdm0:~# ls -l /sys/class/uio/
total 0
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio0 -> ../../devices/platform/amba/ffa00000.perf-monitor/uio/uio0
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio1 -> ../../devices/platform/amba/fd0b0000.perf-monitor/uio/uio1
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio2 -> ../../devices/platform/amba/fd490000.perf-monitor/uio/uio2
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio3 -> ../../devices/platform/amba/ffa10000.perf-monitor/uio/uio3
root@gdm0:~# ls -l /sys/class/uio/../../devices/platform/amba/
total 0
-rw-r--r-- 1 root root 4096 Oct 18 19:37 driver_override
drwxr-xr-x 3 root root 0 Oct 18 01:36 fd070000.memory-controller
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd0b0000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd400000.zynqmp_phy
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd490000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd500000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd510000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd520000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd530000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd540000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd550000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd560000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd570000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd6e0000.cci
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff000000.serial
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff020000.i2c
drwxr-xr-x 6 root root 0 Oct 18 01:36 ff0a0000.gpio
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0b0000.ethernet
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0e0000.ethernet
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0f0000.spi
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff160000.mmc
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff170000.mmc
drwxr-xr-x 3 root root 0 Oct 18 01:36 ff960000.memory-controller
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff9d0000.usb0
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa00000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa10000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa50000.ams
drwxr-xr-x 5 root root 0 Oct 18 01:36 ffa60000.rtc
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa80000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa90000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffaa0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffab0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffac0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffad0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffae0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffaf0000.dma
-r--r--r-- 1 root root 4096 Oct 18 19:37 modalias
lrwxrwxrwx 1 root root 0 Oct 18 19:37 of_node -> ../../../firmware/devicetree/base/amba
drwxr-xr-x 2 root root 0 Oct 18 19:37 power
lrwxrwxrwx 1 root root 0 Oct 18 01:36 subsystem -> ../../../bus/platform
-rw-r--r-- 1 root root 4096 Oct 18 01:36 uevent
root@gdm0:~#
</pre>

= Build firmware =

== Build from git clone ==

THESE ARE K.O.'s NOTES FOR CREATING THE PETALINUX DIRECTORY.

THEY DO NOT WORK!

COPY PETALINUX FROM A WORKING PROJECT AND USE "make gdm" and "make cdm" AS DESCRIBED BELOW.

* git clone git@edev-group.triumf.ca:fw/exp/darkside/gcdm.git
* #Makefile change VIVADO_SETTINGS_SCRIPT := /opt/Xilinx/Vivado/2022.1/settings64.sh
* #. /opt/Xilinx/Vivado/2022.1/settings64.sh
* . /opt/Xilinx/Vivado/2020.2/settings64.sh
* make clean
* make all_from_scratch
* . /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
* make petalinux_create
* make petalinux_rebuild_new_hw_des
* bomb out: The TMPDIR: /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp can't be located on nfs.
* mkdir /tmp/build_tmp
* rm -rf /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp/
* ln -s /tmp/build_tmp /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp
* try again
* grinds, loads a whole bunch of packages...
* finishes with desire to copy things to /tftpboot
* make sdcard_cp_to wants to copy files from PetaLinux_GDM_CDM/images/linux/ to SD card

== Build firmware ==

NOTE: directory Petalinux_GDM_CDM should already exist!

<pre>
#. /opt/Xilinx/Vivado/2020.2/settings64.sh
. /opt/Xilinx/Vivado/2022.2/settings64.sh
. /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
make clean_gdm # remove gdm build tree
make gdm # build or rebuild GDM
make copy_gdm # copy to gdm0
make clean_cdm # remove cdm build tree
make cdm # build or rebuild CDM
make copy_cdm # copy to cdm0 and cdm1
</pre>

copy to SD card:
<pre>
open a root shell
format 16 GB Sd card per above
cd .../ds-dm-gcdm
make copy
</pre>

= build times =

<pre>
CDM 12-june-2023 69aabc1c25130d970bc375aca684bd68849e6685
daq13 AMD-5700G 1688.61user 399.33system 23:28.84elapsed 148%CPU
dsdaqgw AMD-7700 1090.55user 247.34system 16:03.55elapsed 138%CPU
dsdaqgw AMD-7700 CDM incremental 196.68user 67.62system 7:35.42elapsed 58%CPU
dsdaqgw AMD-7700 CDM incremental 684.72user 94.17system 7:30.17elapsed 173%CPU
dsdaqgw AMD-7700 GDM incremental 849.84user 99.79system 9:04.56elapsed 174%CPU
</pre>

= make bootable sd card =

== format the sd card ==

this only needs to be done once

* become root@dsdaqgw
* cd ~olchansk/git/ds-dm-gcdm
* use "lsblk" to identify the SD card (should show as 8/16/32 GB block device)/ /dev/sdd in this case
* make sdcard_format SDCARD_DEVICE=/dev/sdd
* disconnect sd card, reconnect the sd card (to detect new partition tables, etc)

== copy boot files to the sd card ==

* as root: identify partition labels, run "blkid", should say "BOOT", "rootfs" and "data"
* mount
<pre>
mkdir -p /mnt/BOOT
mount -L BOOT /mnt/BOOT
cp PetaLinux_GDM_CDM/images/linux/BOOT_{GDM,CDM}.BIN /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/boot.scr /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/image.ub /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/uboot.env /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/uboot-redund.env /mnt/BOOT/
umount /mnt/BOOT
eject /dev/sdX
</pre>
* FIXME: missing step: cp BOOT_GDM.BIN BOOT.BIN or cp BOOT_CDM.BIN BOOT.BIN

= boot messages =

<pre>
Xilinx Zynq MP First Stage Boot Loader
Release 2020.2 Sep 24 2022 - 13:29:15
NOTICE: ATF running on XCZU4CG/silicon v4/RTL5.1 at 0xfffea000
NOTICE: BL31: v2.2(release):xlnx_rebase_v2.2_2020.3
NOTICE: BL31: Built : 18:02:46, Sep 28 2022

U-Boot 2020.01 (Sep 28 2022 - 18:03:39 +0000)

Model: DarkSide 20k DM
Board: Xilinx ZynqMP
DRAM: 2 GiB
usb dr_mode not found
PMUFW: v1.1
EL Level: EL2
Chip ID: zu4
NAND: 0 MiB
MMC: mmc@ff160000: 0, mmc@ff170000: 1
In: serial@ff000000
Out: serial@ff000000
Err: serial@ff000000
Bootmode: SD_MODE1
Reset reason: SOFT
Net:
ZYNQ GEM: ff0b0000, mdio bus ff0b0000, phyaddr -1, interface rgmii-id

Warning: ethernet@ff0b0000 (eth0) using random MAC address - d6:62:5f:13:00:44
eth0: ethernet@ff0b0000
ZYNQ GEM: ff0e0000, mdio bus ff0e0000, phyaddr -1, interface rgmii-id
Could not get PHY for eth1: addr -1

Hit any key to stop autoboot: 0
ZynqMP>
CTRL-A Z for help | 115200 8N1 | NOR | Minicom 2.7.1 | VT102 | Online 122:2 | ttyACM0
</pre>

= load FPGA from u-boot =

* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/124682257/U-Boot+FPGA+Driver

<pre>
ZynqMP> fpga info
Xilinx Device
Descriptor @ 0x000000007fddb2c0
Family: ZynqMP PL
Interface type: csu_dma configuration interface (ZynqMP)
Device Size: 1 bytes
Cookie: 0x0 (0)
Device name: zu4
Device Function Table @ 0x000000007fda5fe8
PCAP status 0xa0002fde
ZynqMP>
</pre>

* cp CDM_XU8_top.bit /tftpboot/fpga.bit

<pre>
dhcp
tftpb 0x10000000 fpga.bit
fpga loadb 0 0x10000000 ${filesize}
</pre>

<pre>
ZynqMP> dhcp
BOOTP broadcast 1
DHCP client bound to address 192.168.0.100 (1 ms)
*** Warning: no boot file name; using 'C0A80064.img'
Using ethernet@ff0b0000 device
TFTP from server 192.168.0.1; our IP address is 192.168.0.100
Filename 'C0A80064.img'.
Load address: 0x8000000
Loading: *
TFTP error: 'file /tftpboot/C0A80064.img not found for 192.168.0.100' (1)
Not retrying...
ZynqMP> tftpb 0x10000000 fpga.bit
Using ethernet@ff0b0000 device
TFTP from server 192.168.0.1; our IP address is 192.168.0.100
Filename 'fpga.bit'.
Load address: 0x10000000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
############
6.2 MiB/s
done
Bytes transferred = 7797807 (76fc2f hex)
ZynqMP> fpga loadb 0 0x10000000 ${filesize}
design filename = "CDM_XU8_top;UserID=0XFFFFFFFF;Version=2022.2"
part number = "xczu4cg-fbvb900-1-e"
date = "2024/08/14"
time = "14:18:22"
bytes in bitstream = 7797692
zynqmp_align_dma_buffer: Align buffer at 0000000010000073 to 000000000fffff80(swap 0)
ZynqMP>
</pre>

= load FPGA from Linux =

* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841847/Solution+ZynqMP+PL+Programming

== this will reset the CPU ==
<pre>
cp fpga.bit /lib/firmware/
echo fpga.bit > /sys/class/fpga_manager/fpga0/firmware
</pre>

* make .bin file:
<pre>
bootgen -image CDM_XU8_top.bif -arch zynqmp -o ./fpga.bin -w
</pre>
* cat CDM_XU8_top.bif
<pre>
dsdaqgw:ds-dm-gcdm$ cat CDM_XU8_top.bif
all:
{
[destination_device = pl] ./Vivado_CDM_XU8/CDM_XU8.runs/impl_1/CDM_XU8_top.bit
}
dsdaqgw:ds-dm-gcdm$
</pre>

== this will reset the CPU ==

<pre>
cp fpga.bin /lib/firmware/
echo fpga.bin > /sys/class/fpga_manager/fpga0/firmware
</pre>

== this will reset the CPU ==

<pre>
root@dsdm:~# ./fpgautil -b fpga.bin -f Full
</pre>

== this will reset the CPU ==

<pre>
DTSO file from here: https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841847/Solution+ZynqMP+PL+Programming
echo 0 > /sys/class/fpga_manager/fpga0/flags
mount -t configfs configfs /configfs
root@dsdm:~# dtc -O dtb -o fpga.dtbo -b 0 -@ fpga.dtso
root@dsdm:~# cp fpga.dtbo /lib/firmware/
root@dsdm:~# cp fpga.bit /lib/firmware/
root@dsdm:~# rmdir /configfs/device-tree/overlays/fpga
root@dsdm:~# mkdir /configfs/device-tree/overlays/fpga
root@dsdm:~# echo -n "fpga.dtbo" > /configfs/device-tree/overlays/fpga/path
</pre>

= fpgautil =

* https://github.com/Xilinx/meta-xilinx/blob/master/meta-xilinx-core/recipes-bsp/fpga-manager-script/files/fpgautil.c

<pre>
git clone https://github.com/Xilinx/meta-xilinx.git
cd meta-xilinx/meta-xilinx-core/recipes-bsp/fpga-manager-script/files/
scp fpgautil.c root@dsdm:
ssh root@dsdm
make fpgautil
ls -l ./fpgautil
</pre>

<pre>
root@dsdm:~# ls -l ./fpgautil
-rwxr-xr-x 1 root root 72256 Aug 16 00:15 ./fpgautil
root@dsdm:~# ./fpgautil

fpgautil: FPGA Utility for Loading/reading PL Configuration

Usage: fpgautil -b <bin file path> -o <dtbo file path>

Options: -b <binfile> (Bin file path)
-o <dtbofile> (DTBO file path)
-f <flags> Optional: <Bitstream type flags>
f := <Full | Partial >
-n <Fpga region info> FPGA Regions represent FPGA's
and partial reconfiguration
regions of FPGA's in the
Device Tree
Default: <full>
-s <secure flags> Optional: <Secure flags>
s := <AuthDDR | AuthOCM | EnUsrKey | EnDevKey | AuthEnUsrKeyDDR | AuthEnUsrKeyOCM | AuthEnDevKeyDDR | AuthEnDevKeyOCM>
-k <AesKey> Optional: <AES User Key>
-r <Readback> Optional: <file name>
Default: By default Read back contents will be stored in readback.bin file
-t Optional: <Readback Type>
0 - Configuration Register readback
1 - Configuration Data Frames readback
Default: 0 (Configuration register readback)
-R Optional: Remove overlay from a live tree

Examples:
(Load Full bitstream using Overlay)
fpgautil -b top.bit.bin -o can.dtbo -f Full -n full
(Load Partial bitstream using Overlay)
fpgautil -b rm0.bit.bin -o rm0.dtbo -f Partial -n PR0
(Load Full bitstream using sysfs interface)
fpgautil -b top.bit.bin -f Full
(Load Partial bitstream using sysfs interface)
fpgautil -b rm0.bit.bin -f Partial
(Load Authenticated bitstream through the sysfs interface)
fpgautil -b top.bit.bin -f Full -s AuthDDR
(Load Parital Encrypted Userkey bitstream using Overlay)
fpgautil -b top.bit.bin -o pl.dtbo -f Partial -s EnUsrKey -k <32byte key value>
(Read PL Configuration Registers)
fpgautil -b top.bit.bin -r
(Remove Partial Overlay)
fpgautil -R -n PR0
(Remove Full Overlay)
fpgautil -R -n full
Note: fpgautil -R is responsible for only removing the dtbo file from the livetree. it will not remove the PL logic from the FPGA region.

root@dsdm:~#
</pre>

= fw_printenv =

to access u-boot environment from Linux:
* apt install -y libubootenv-tool
* create /etc/fw_env.config
<pre>
/media/BOOT/uboot.env 0 0x40000
/media/BOOT/uboot-redund.env 0 0x40000
</pre>
* if uboot.env files do not exist, run "saveenv" from u-boot command prompt
* fw_printenv and fw_setenv should work

= Boot from network =

== u-boot ==

<pre>
ZynqMP> setenv bootcmd run bootcmd_dhcp
ZynqMP> saveenv
ZynqMP> reset
</pre>

== boot.scr ==

<pre>
# boot.scr
# mkimage -C none -A arm -T script -d boot.scr boot.scr.uimg
echo Loading FPGA!
#tftpb 0x10000000 fpga.bit
tftpb 0x10000000 {ipaddr}.bit
fpga loadb 0 0x10000000 ${filesize}
echo Booting Linux!
run bootcmd_pxe
echo Done!
</pre>

<pre>
mkimage -C none -A arm -T script -d boot.scr boot.scr.uimg
</pre>

== tftpboot ==

<pre>
cp /home/olchansk/git/ds-dm-gcdm/boot.scr.uimg /tftpboot
ln -s /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/images/linux /tftpboot/xilinx-dsdm
mkdir /tftpboot/pxelinux.cfg
cat > /tftpboot/pxelinux.cfg/default-arm-zynqmp <<EOF
LABEL Linux
KERNEL xilinx-dsdm/Image
FDT xilinx-dsdm/system.dtb
#INITRD rootfs.cpio.gz.u-boot
EOF
</pre>

== boot sequence ==

* xilinx magic load BOOT.BIN from SD card
* load FPGA form BOOT.BIN
* load and run u-boot from BOOT.BIN or from image.ub
* u-boot load environment from ??? probably SD card uboot-redund.env, this includes our bootcmd
* run bootcmd which run bootcmd_dhcp which does:
* from /tftpboot:
* load and run boot.scr.uimg which does:
* load FPGA image xilinx-dsdm/${ipaddr}.bit
* run bootcmd_pxe which does:
* load pxelinux.cfg/default-arm-zynqmp which does:
* load xilinx-dsdm/Image ### this is the linux kernel
* load xilinx-dsdm/system.dtb ### this is the device tree
* start linux kernel
* linux kernel does dhcp
* linux kernel does nfs mount /nfsroot/%s,vers=3,tcp ### %s is replaced by the hostname supplied by DHCP
* userland starts and runs to console and ssh login.

= Xilinx ILA =

References:
* https://github.com/Xilinx/XilinxVirtualCable/tree/master
* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/644579329/Xilinx+Virtual+Cable
* https://support.xilinx.com/s/article/974879?language=en_US
* https://docs.xilinx.com/v/u/en-US/pg245-debug-bridge
* https://docs.xilinx.com/r/2020.2-English/ug908-vivado-programming-debugging/Initializing-Vivado-IDE-hw_server
* (we do not use this one) https://github.com/paulscherrerinstitute/xvcSupport/blob/master/README.md

Build xvcserver_cdm.exe: (it is built as a static executable, can be copied and run anywhere)
<pre>
ssh dsdaq@gdm0
cd /home/dsdaq/online/ds-dm-software
git pull ### get latest version
make xvcserver_cdm.exe
ssh root@gdm0
/home/dsdaq/online/ds-dm-software/xvcserver_cdm.exe
INFO: To connect to this xvcServer instance, use url: TCP:gdm0:2542
</pre>

To activate and use the vivado logic analyzer:
* data path: vivado -> hw_server -> xvcserver -> mmap axi bus -> debug bridge -> jtag -> ILA
* define ILAs in the code
* instantiate the xilinx debug bridge at AXI bus address 0x80020000 (FIXME!!! this collides with Ian's AXI addresses)
* build and boot the new FPGA firmware. updating the linux kernel is not necessary.
* login root@gdm0, run: /home/dsdaq/online/ds-dm-software/xvcserver_cdm.exe -v ### with "-v" for the first time to see that vivado does connect to it, without "-v", normally.
* login dsdaqgw, run: hw_server -s tcp:localhost:3121 -e "set auto-open-servers xilinx-xvc:gdm0:2542" ### tells us to connect to port localhost:3121
* login dsdaqgw, run vivado, open project, open hardware manager, open target, open new target, "connect to remote server", hostname "localhost", port "3121", next (bombs, try again, 3 times), popup add virtual cable, enter hostname "gdm0" port "2542", "ok", it shows in "hardware targets", "next", "finish", error popup "[Common 17-163] Missing value for option 'objects', please type 'set_property -help' for usage info", ignore it, in "hardware", right click the "gdm0" one, open target, under "hardware" and "debug bridge" we should see all the ILAs, under "hardware device properties", the "probes file" should have the ".ltx" file generated by vivado "Vivado_GDM_XU8/GDM_XU8.runs/impl_1/debug_nets.ltx", click on an ILA, a waveform should open.

= Software =

* ssh cdm0 # or gdm0
* sudo apt install i2c-tools libi2c-dev
* git clone https://bitbucket.org/team-ds-dm/ds-dm-software
* cd ds-dm-software
* make

== test_cdm.exe ==

=== CDM SFP status ===

<pre>
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0xbb2f0ae7
CDM firmware 0xbb2f0ae7
arg 1: [--sfp]
Polling SFP status...
identifier 0x03
connector 0x07
encoding 0x01
wavelength 0x0352 (850 nm)
vendor_name [FINISAR CORP. ]
vendor_pn [FTLF8526P3BNL ]
vendor_rev [A ]
vendor_sn [N3AB9M8 ]
vendor_date [200319 ]
dm_type 0x68
temp 29.0 C
vcc 3.323 V
tx_bias 7.250 mA
tx_power 478.4 uW
rx_power 2.3 uW
SFP good 1, status: temp 30.6 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 818 uW
...
</pre>

=== GDM QSFP status ===

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --qsfp3 --qsfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x53aee418
CDM firmware 0x53aee418
arg 1: [--qsfp3]
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out hi
arg 2: [--qsfp]
Polling QSFP status...
identifier 0x0d
status 0x02
los 0x8f
temp 28.2 C
vcc 3.323 V
rx_power 0.1 0.1 0.1 0.1 uW
tx_bias 7.6 7.6 7.6 0.0 mA
tx_power 792.2 773.8 823.0 0.1 uW
vendor_name [FINISAR CORP ]
vendor_pn [FTL410QD4C ]
vendor_rev [A ]
wavelength 850
max_temp 70 C
vendor_sn [X79AC0R ]
vendor_date [220309 ]
QSFP good 1, status: temp 27.7 C, los 0x8b, tx_bias 7.5 7.6 7.6 0.0 mA, tx_power 792 772 821 0 uW, rx_power 0 0 466 0 uW
</pre>

=== GDM clock status ===

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
</pre>

* clock chip not loaded, not running:

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x0f 0x16 0x04 0x00 0x01 0x68 0x09 0x00 0x02 0xf2 0x00 0x1f 0xf0 0x22 0xf2, 0x507: 0x00, 0x52A: 0x00, 0x53F: 0x04
Clock chip state 0, status: SYSINCAL XAXB_ERR LOL CAL_PLL IN0 IN_SEL_0 FASTLOCK_STATUS

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x1f7caf52 (528265042) should be ~125 MHz
0x1034 rx_clk: 0x00000000 (0) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x1f7caf52 (528265042) should be ~125 MHz
0x103C tx_clk: 0x00000000 (0) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip good (IN0 - external 10 MHz clock)

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x44 0x00 0xd0 0x01 0x1f 0xff 0x22 0xf2, 0x507: 0x3f, 0x52A: 0x01, 0x53F: 0x00
Clock chip state 1, status: LOS_IN2 OOF_IN2 IN0 IN_SEL_REGCTRL IN_SEL_0

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735a3b (125000251) should be ~125 MHz
0x1034 rx_clk: 0x07735a3c (125000252) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a3b (125000251) should be ~125 MHz
0x103C tx_clk: 0x07735a3c (125000252) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

=== CDM clock status ===

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
</pre>

* clock chip not loaded, not running:

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x0f 0x16 0x04 0x00 0x01 0x68 0x19 0x00 0x02 0xf2 0x00 0x1f 0xf0 0x22 0xf2, 0x507: 0x00, 0x52A: 0x00, 0x53F: 0x04
Clock chip state 0, status: SYSINCAL XAXB_ERR LOL CAL_PLL IN0 IN_SEL_0 FASTLOCK_STATUS

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735851 (124999761) should be ~125 MHz
0x1034 rx_clk: 0x00000000 (0) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x0127fefa (19398394) should be ~125 MHz
0x103C tx_clk: 0x00000000 (0) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip uses internal clock (IN1 - internal oscillator), observe rx_clk frequency is not same as others

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0x7f, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN1 IN_SEL_1 HOLD_HIST_VALID

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735853 (124999763) should be ~125 MHz
0x1034 rx_clk: 0x07735b0a (125000458) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735853 (124999763) should be ~125 MHz
0x103C tx_clk: 0x07735852 (124999762) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip uses SFP recovered clock (IN2 - sfp rx recovered clock), observe mgt_rx_ref_clk_raw (CDM 125 MHz oscillator) is different from others (SFP RX recovered clock)

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0xbf, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN2 IN_SEL_1 HOLD_HIST_VALID
^C
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773581b (124999707) should be ~125 MHz
0x1034 rx_clk: 0x07735ad6 (125000406) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735ad7 (125000407) should be ~125 MHz
0x103C tx_clk: 0x07735ad6 (125000406) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
^C
</pre>

=== CDM link status, PRBS test mode ===

<pre>
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
</pre>
* fiber disconnected, no link:
<pre>
CDM firmware: 0xbb2f0ae7
0x1000 SFP c.c. status: 0x00000031
CLK_IN_SEL_LS 0x1
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x00000025
sfp_mod_absent_N 1
sfp_rx_los_N 0
link_power_good 1
rx_link_up 0
rx_receiving_data 0
rx_error 1
rx_lnk_up_and_running 0
tx_link_up 0
tx_sending_data 0
tx_link_up_and_running 0
link_up_and_running 0
0x1014 SFP link data: 0x466a8187
rx_data 0x8187
k28p1_k28p5 0
rx_ctrl0 0x1
rx_ctrl1 0x1
rx_ctrl3 0x1
tx_state 0x1
rx_state 0x6
rx_receiving_data 0
rx_link_up 0
rx_error 1
rx_link_rst 0
0x2000 link test mode: 0x00000200, seconds: 0x00079093, errors: 0xffffffff
</pre>
* fiber connected, good link:
<pre>
CDM firmware: 0xbb2f0ae7
0x1000 SFP c.c. status: 0x000000b2
CLK_IN_SEL_LS 0x2
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x000007dc
sfp_mod_absent_N 0
sfp_rx_los_N 0
link_power_good 1
rx_link_up 1
rx_receiving_data 1
rx_error 0
rx_lnk_up_and_running 1
tx_link_up 1
tx_sending_data 1
tx_link_up_and_running 1
link_up_and_running 1
0x1014 SFP link data: 0x35c02774
rx_data 0x2774
k28p1_k28p5 0
rx_ctrl0 0x0
rx_ctrl1 0x0
rx_ctrl3 0x0
tx_state 0x3
rx_state 0x5
rx_receiving_data 1
rx_link_up 1
rx_error 0
rx_link_rst 0
0x2000 link test mode: 0x00000200, seconds: 0x00078a8c, errors: 0x00000000
</pre>

=== GDM link status, PRBS test mode ===

<pre>
1 link connected, no errors:
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
GDM firmware: 0x6b2ee010
0x1014: 0x00000008, 18: 0x00000008, 1C: 0x00000000, 24: 0x00000fff
0x2000: 0x00000200, time: 0x00078aa4, errors:
0xffffffff 0xffffffff 0xffffffff
0x00000000 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
</pre>

=== CDM link status ===

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test0

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0xbf, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN2 IN_SEL_1 HOLD_HIST_VALID

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x077357a0 (124999584) should be ~125 MHz
0x1034 rx_clk: 0x07735a5c (125000284) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a5b (125000283) should be ~125 MHz
0x103C tx_clk: 0x07735a5b (125000283) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM firmware: 0x6d2ef81a
0x1000 SFP c.c. status: 0x000000b2
CLK_IN_SEL_LS 0x2
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x000007dc
sfp_mod_absent_N 0
sfp_rx_los_N 0
link_power_good 1
rx_link_up 1
rx_receiving_data 1
rx_error 0
rx_lnk_up_and_running 1
tx_link_up 1
tx_sending_data 1
tx_link_up_and_running 1
link_up_and_running 1
0x1014 SFP link data: 0x35c6bcbc
rx_data 0xbcbc
k28p1_k28p5 0
rx_ctrl0 0x3
rx_ctrl1 0x0
rx_ctrl3 0x0
tx_state 0x3
rx_state 0x5
rx_receiving_data 1
rx_link_up 1
rx_error 0
rx_link_rst 0
0x2000 link test mode: 0x00000000, seconds: 0x00001671, errors: 0xffffffff

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 15
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
reg[15] is 0x0033bcbc (3390652)
</pre>

=== GDM link status ===

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test0

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x44 0x00 0xd0 0x01 0x1f 0xff 0x22 0xf2, 0x507: 0x3f, 0x52A: 0x01, 0x53F: 0x02
Clock chip state 1, status: LOS_IN2 OOF_IN2 IN0 IN_SEL_REGCTRL IN_SEL_0 HOLD_HIST_VALID

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735a0a (125000202) should be ~125 MHz
0x1034 rx_clk: 0x07735a0a (125000202) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a0a (125000202) should be ~125 MHz
0x103C tx_clk: 0x07735a0a (125000202) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly

register 0x1018 bit 0x800

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM firmware: 0x6d2ebce6
0x1014: 0x00000800, 18: 0x00000800, 1C: 0x00000000, 24: 0x00000fff
0x2000: 0x00000000, time: 0x00003d2f, errors:
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff

link data alternates 0xbcbc and 0x1cbc

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 22
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
reg[22] is 0xbcbc93ab (-1128492117)

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 22
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
reg[22] is 0x1cbc1aaf (482089647)
root@gdm0:~#

</pre>

=== Run trg and tsm ===

<pre>
on the GDM:

ssh root@gdm00
./test_cdm.exe --gdm-clocks
./test_cdm.exe --load-cc
./test_cdm.exe --cc
./test_cdm.exe --reset-mgt
./test_cdm.exe --gdm-clocks
./test_cdm.exe --writereg 2 0xff # enable LEMO NIM inputs
./test_cdm.exe --writereg 3 0xba54 # LEDs: lemo1, lemo2, trg, tsm
./test_cdm.exe --writereg 4 0x99 $ # LEMO out is trg_in_pulse
./test_cdm.exe --writereg 9 0x32010 # trg and tsm from trg_pulser and tsm_pulse
./test_cdm.exe --writereg 24 1250000 # trg pulser 100 Hz
./test_cdm.exe --writereg 26 125000000 # tsm pulser 1 Hz
./test_cdm.exe --writereg 23 0x40000000 # route trg_in and tsm_in to qsfp tx bits 0 and 1
./test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz

on the CDM:

ssh root@cdm01
./test_cdm.exe --cdm-clocks
./test_cdm.exe --load-cc
./test_cdm.exe --reset-mgt
./test_cdm.exe --cdm-clocks
./test_cdm.exe --cdm-link # issue --reset-mgt on CDM and GDM until link is good
./test_cdm.exe --writereg 2 0xff # enable LEMO NIM inputs
./test_cdm.exe --writereg 3 0xba54 # enable LEDs: lemo1, lemo2, trg, tsm
./test_cdm.exe --writereg 4 0x99 # enable LEMO output trg_in
./test_cdm.exe --writereg 9 0x8040 # enable trg_in and tsm_in from sfp rx bits 0 and 1
./test_cdm.exe --writereg 7 0xff00 # drive VX LVDS lines to logic level 0
./test_cdm.exe --writereg 8 0x2 # VX LVDS with trg and tsm, misrouted in second VX
#./test_cdm.exe --writereg 8 0x7 # VX LVDS with tsm
./test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz
</pre>

switch GDM and CDM to packetizer trg and tsm:

<pre>
on the GDM:

/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x00000000

on the CDM:

/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x0804
/home/dsdaq/online/ds-dm-software/test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz
</pre>

=== Run packet loopback ===

GDM CPU -> fifo_to_fpga -> GDM QSFP -> CDM SFP -> fifo_from_fpga -> CDM CPU

On the GDM: (CDM is connected to first QSFP port)

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --load-cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --reset-mgt
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 32 1 ### tell GDM to use first QSFP port
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link ### confirm link status is "3"
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 29 2 ### enable GDM packet data injection
/home/dsdaq/online/ds-dm-software/test_cdm.exe --test-fifo-write-loop
</pre>

One the CDM:

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --load-cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --reset-mgt
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 29 3 ### enable CDM packet data injection
/home/dsdaq/online/ds-dm-software/test_cdm.exe --test-fifo-read
</pre>

== fecdm ==

=== Power up and clock management ===

* Clock power up and setup management is handled by fecdm state machine
* GDM clock setup sequence:
<pre>
- on power up, CC is not initialized, XAXB clock is not running
- on GDM frontend start (in HandleInit()), CC is loaded and reset
- CC will lock to IN0 (10 MHz external clock) or IN1 (125 MHz internal oscillator)
- if locked to IN0, CC will enter and exit the HOLDOVER mode if 10 MHz clock is unstable
</pre>
* CDM clock setup sequence:
<pre>
- on power up, CC is not initialized, XAXB clock is not running
- fecdm detects this, loads the CC configuration registers, resets the CC
- sets the CC input IN1, DSDM internal oscillator
- if SFP is in good state, SFP MGT is running, CC is happy with IN2 input (no IN2_OOF, IN2_LOS), SFP recpvered clock (clk_rx_mgt) frequency is close to 125 MHz:
- switches CC to input IN2 (SFP recovered clock, clk_rx_mgt)
- waits for CC locked to IN2 (no LOL, no HOLD). (this provides stable SFP tx clock).
- resets the SFP MGT, requests GDM MGT reset (via ODB)
- MGT reset sequence: GDM MGT reset 1, CDMnn MGT reset 1, GDM MGT reset 0, CDM MGT reset 0
- CDM MGT rx_state cycles from 0 to 5. if it goes to 6 or never reaches 5, GDM MGT reset is requested again.
- during this sequence, clk_rx_mgt may wobble, CC may enter HOLDOVER mode, VX clock should stay steady.
- once CC is locked to IN2, SFP MGT rx_state is 5, link is ready for use.
</pre>
* Request changes for Ian
<pre>
17 Apr 2025:
- CDM SFP MGT rx_state is getting stuck in 4 (waiting for alternating idle chars), do we need this? go to state 5 without waiting?
- GDM QSFP MGT reset 1 does not seem to do anything, change it to cause all CDM MGTs to enter an error state, e.g. start transmitting invalid K-codes. transmitting all 0 or all 1 will break the recovered clock, so maybe do not do that.
- having all CDM SFP MGTs go to error state immediatley after GDM QSFT MGT reset 1 will greatly simplify the link reset sequence.
</pre>

=== Trigger configuration ===

* GDM
* CDM in SFP trigger mode:
** SFP K-code encoded sfp_rx_trg, sfp_rx_tsm are routed to LVDS vx_tx_trg and vx_tx_tsm
** GDM TRG, TSM (and all other) packets are routed to LVDS serial data to VX
** to make counters count and lights blink:
** CDM TRG and TSM trigger mask is set to sfp_rx_trg, sfp_rx_tsm and vx_tx_trg_done, vx_tx_tsm_done
** CDM LEDs display trg_in_pulse, tsm_in_pulse
** output of TRG and TSM packets is disabled (they come from the GDM)

= dsvslice integration =

== VX setup ==

* general
** Start acq from user code = y, all others = n (as of Mar 2023: will start when first trigger received)
** Use NIM IO = y
** Use external clock = y
** LVDS quartet is input = n, y, n, y
** LVDS quartet mode = User, User, User, User

* trigger from front panel NIM:
** Trigger on external signal = y, all others = n
** connect CDM EXT_OUT(2) to VX "TrigIn"

* trigger from LVDS "Sync" mode
** Trigger on LVDS Sync signal = y, all others = n
** LVDS quartet mode = User, Sync, User, User

* trigger from LVDS "User" mode
** Trigger on LVDS pair 12 signal = y, all others = n
** LVDS quartet is input = n, y, n, y
** LVDS quartet mode = User, User, User, User

== GDM setup ==

* GDM is gdm0
* set inputs to NIM mode
* set outputs to TTL mode (this GDM has wrong NIM output circuit)
* use top QSFP slot, connect split cable 1 into CDM01, cable 2 into CDM02
* connect non-inverted NIM trigger signal to top-LEMO-left EXT_IN_LV(1)
* connect non-inverted NIM TSM signal to top-LEME-right EXT_IN_LV(2)
* GDM LEDs: TRIG, TSM, trigger enabled, trigger_out
* GDM LEMO_OUT: trigger, trigger

== CDM setup ==

* set CDM LEMO inputs to NIM
* set CDM LEMO outputs to NIM
* CDM01 is cdm0
* CDM02 is cdm1
* connect GDM fiber links to SFP port
* connect 1st VX port of CDM01 to VX1
* connect 1st VX port of CDM02 to VX2
* connect LEMO EXT_OUT(2) to VX "TrigIn", CDM01 to VX1, CDM02 to VX2
* power up
* CDM LEDs: GDM TRIG, GDM TSM, trigger enabled, trigger out
* CDM LEMO_OUT: gdm_trg, trigger

== After power up ==

* start the CDM frontend from the MIDAS "Programs" page. To start manually, see the Start Command on the Programs page.
* CDM frontend should enable the VX clock, disable the trigger
* from the MIDAS status page, goto the CDM page
* outdated: in the CDMx data tables, the 2nd number should read 0x35c08008, if it does not and the last 4 digits randomly change, reset the GDM links
*if the FEs complain - do in order: for GDM, CDM01, CDM02, ..., press "reset mgt" of each board, then press "unreset mgt", if it does not help, STOP HERE
* start a run
* CDM frontend will enable the trigger
* GDM frontend will enable the trigger
* LEDs on the GDM should flash, LEDs on the CDM should flash, TrigIn and TrigOut of the VX should flash
* stop a run
* GDM frontend will disable the trigger
* CDM frontend will disable the trigger

== Phase measurement ==

* pip3 install matplotlib
* pip3 install scipy
* export PYTHONPATH=$HOME/packages/midas/python
* #git clone https://github.com/J033X071C/PhaseMeasurement
* git clone https://bitbucket.org/team-ds-dm/phasemeasurement.git
* cd phasemeasurement
* python3 ./phaseMeasurement.py --help
<pre>
daq00:PhaseMeasurement$ python3 ./phaseMeasurement.py --help
usage: phaseMeasurement.py [-h] fileName numberEvents numberVX sizeEvents stopEvent minHist maxHist numberBin writeToTXT saveAsPDF

Read data from midas file (in .lz4 format) to calculate phase between the clock of VX1 and VX2

positional arguments:
fileName Name of the file we want to read data from (Example: run00389.mid.lz4)
numberEvents Number of events recorded in the file
numberVX Number of VX used in this run (usually 2...)
sizeEvents Number of points per event
stopEvent Number of events you want to go through to calculate phase
minHist Minimal value for the x axis of the phase measurement histogram (in ns)
maxHist Maximal value for the x axis of the phase measurement histogram (in ns)
numberBin Number of bins wanted for the generated histogram
writeToTXT Write argument as yes to generate text file with results of calculation
saveAsPDF Save generated plots to PDF files

optional arguments:
-h, --help show this help message and exit
daq00:PhaseMeasurement$
</pre>
* try an old file with
* python3 ./phaseMeasurement.py run00877.mid.lz4 10000 2 10000 500 -20 20 81 yes yes
<pre>
bin size = 0.494 ns
num_events = 347
mean = -1.705 ns
rms = 3.087 ns
mean_error = 0.166 ns
centroid = -1.706 ns.
width (sigma) = 0.363 ns.
error on the centroid = 0.016558 ns.
</pre>
* ls -l *.txt *.pdf
<pre>
dsdaq@dsvslice:~/online/PhaseMeasurement$ ls -l *.txt *.pdf
-rw-rw-r-- 1 dsdaq dsdaq 64728 Dec 14 16:56 run00877.mid.lz4_Plots.pdf
-rw-rw-r-- 1 dsdaq dsdaq 274 Dec 14 16:56 run00877.mid.lz4.txt
dsdaq@dsvslice:~/online/PhaseMeasurement$
</pre>
* scope settings (from email message)
<pre>
From fcote-lortie@triumf.ca Thu Dec 15 17:11:46 2022
From: Francis Cote-Lortie <fcote-lortie@triumf.ca>
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: Re: How to use scope as a waveform generator
Date: Fri, 16 Dec 2022 01:11:44 +0000

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display) by pressing on the button. It will go from 0 to 1.

The settings that we are using right now are:
Type of waveform: Sine wave
Offset: 0 V
Amplitude: 1 Vpp
Frequency: 50 kHz
Noise: 0 V
________________________________
From: Francis Cote-Lortie <fcote-lortie@triumf.ca>
Sent: Thursday, December 15, 2022 4:58 PM
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: Re: How to use scope as a waveform generator

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display)

The settings that we are using right now are:
Type of waveform: Sine wave
Offset: 0 V
Amplitude: 1 Vpp
Frequency: 50 kHz
Noise: 0 V
________________________________
From: Francis Cote-Lortie
Sent: Thursday, December 15, 2022 4:54 PM
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: How to use scope as a waveform generator

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display)
</pre>

== Standalone link test ==

<pre>
CDM: program clock chip
busybox devmem 0x80011000 32 0x8
busybox devmem 0x80011000 32 0x0
/home/dsdaq/si5394-i2c-file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b

GDM, CDM: link reset
busybox devmem 0x80011008 32 1

GDM, CDM: release reset
busybox devmem 0x80011008 32 0

CDM: link status (NOTE: SFP LOS and mod_absent are swapped!!!)
busybox devmem 0x80011010 32
0x00000024 <- fiber plugged
0x00000025 <- fiber unplugged
0x00000027 <- SFP unplugged
0x000007DC <- successful link with GDM

CDM: link state machine and data
busybox devmem 0x80011014 32
0x35C06FF6

CDM: set link to counting mode
busybox devmem 0x80012000 32 0x101
busybox devmem 0x80012000 32 0x100

CDM: time counter and error counter
root@cdm1:~# busybox devmem 0x80012000 32
0x00000100 <--- link mode
root@cdm1:~# busybox devmem 0x80012004 32
0x0000058C <--- seconds counter
root@cdm1:~# busybox devmem 0x80012004 32
0x0000058D
root@cdm1:~# busybox devmem 0x80012008 32
0x00000000 <--- error counter

GDM: no link
root@gdm0:~# busybox devmem 0x80011014 32
0x00000000
root@gdm0:~# busybox devmem 0x80011018 32
0x00000000
root@gdm0:~# busybox devmem 0x8001101c 32
0x00000000
root@gdm0:~# busybox devmem 0x80011024 32
0x00000FFF
root@gdm0:~#

GDM: good link channel 10, counting mode
root@gdm0:~# busybox devmem 0x80012000 32 0x101
root@gdm0:~# busybox devmem 0x80012000 32 0x100
root@gdm0:~# busybox devmem 0x80012008 32
0x3A8B68C2
root@gdm0:~# busybox devmem 0x80012008 32
0x42E03BEF
root@gdm0:~# busybox devmem 0x8001200c 32
0xDA090972
root@gdm0:~# busybox devmem 0x8001200c 32
0xDE6F22E9
root@gdm0:~# busybox devmem 0x80012019 32
Bus error
root@gdm0:~# busybox devmem 0x80012010 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012014 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012018 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x8001201c 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012020 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012024 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012028 32
0x00000000
root@gdm0:~# busybox devmem 0x8001202c 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012030 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012034 32
0xFFFFFFFF
root@gdm0:~#
</pre>

mapping of link channels:

<pre>
qsfp0 lane0 - 0x0100 - link 8
qsfp0 lane1 - 0x0200 - link 9
qsfp0 lane2 - 0x0400 - link 10
qsfp0 lane3 - n/c
qsfp1 lane0 - 0x0800 - link 11
qsfp1 lane1 - 0x0010 - link 4
qsfp1 lane2 - 0x0020 - link 5
qsfp1 lane3 - n/c
qsfp2 lane0 - 0x0040 - link 6
qsfp2 lane1 - 0x0080 - link 7
qsfp2 lane2 - 0x0001 - link 0
qsfp2 lane3 - n/c
qsfp3 lane0 - 0x0002 - link 1
qsfp3 lane1 - 0x0004 - link 2
qsfp3 lane2 - 0x0008 - link 3
qsfp3 lane3 - n/c
</pre>

script to start the test with 2 CDMs:

<pre>
ssh dsdaq@dsvslice
ssh root@gdm0 busybox devmem 0x80011008 32 1
ssh root@cdm0 busybox devmem 0x80011008 32 1
ssh root@cdm1 busybox devmem 0x80011008 32 1
ssh root@gdm0 busybox devmem 0x80011008 32 0
ssh root@cdm1 busybox devmem 0x80011008 32 0
ssh root@cdm0 busybox devmem 0x80011008 32 0
ssh root@gdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
ssh root@cdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
ssh root@cdm1 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
</pre>

== LEMO trigger GDM to CDM to VX ==

on the GDM:

<pre>
/home/dsdaq/si5394-i2c-file /home/dsdaq/GDM_v1.0_IN0_EXT1_and_IN1_fixed_Si5394-RevA-Registers.txt 0 0x6b
busybox devmem 0x80011008 32 0x1
busybox devmem 0x80011008 32 0x0
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x80001230 # QSFP TX fixed pattern
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 2 0xff # enable LEMO inputs
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 3 0x7654 # enable LED, one per LEMO input
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x0F0F # enable LEMO to trg_in and tsm_in
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x40001230 # enable trg_in and tsm_in output to QSFP
</pre>

on the CDM:

<pre>
/home/dsdaq/si5394-i2c-file /home/dsdaq/CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
/home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp
busybox devmem 0x80011008 32 1
busybox devmem 0x80011008 32 0
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x8040 # trg_in from sfp[0], tsm_in form sfp[1]
/home/dsdaq/online/ds-dm-software/test_cdm.exe 11 # trg_in counter
/home/dsdaq/online/ds-dm-software/test_cdm.exe 12 # tsm_in counter
</pre>

== GPS receiver VCL-2705 ==

* Valiant VCL-2705 GPS receiver
* https://www.valiantcom.com/time-distribution/gps-receiver-irig-b.html
* USB connection is /dev/ttyACM0, 115200 bps, "GNSSAUX" prompt
* minicom -D /dev/ttyACM0 -b 115200
* user manual download instructoins - see sheet of paper with user name and password in the shipping box
* usb commands:
<pre>
gnss-help
gnss-showver -> F/W: Ver 1.7 Feb 17 2020 16:20:43

gnss-showselftest -> no antenna connected
Overall : FAIL
EPROM Test: PASS
Antenna : NOT DETECTED
GNSS : COMMUNICATION OK

gnss-showsettings
GNSS NMEA BAUDRATE :115200
GNSS ANTENNA LENGTH :30 meters
GNSS USER CONFIGURED DELAY:-65 nanoseconds
GNSS 1PPS PULSE WIDTH :200 milliseconds
GNSS MODE :GPS
GNSS STATUS :STATIONARY

GNSSAUX> gnss-showserial
SERIAL :2704H01V17MAX310

GNSSAUX> gnss-showmode
GNSS MODE : GPS

GNSSAUX> gnss-showstatus
GNSS STATUS: STATIONARY

GNSSAUX> gnss-showalarms
CURRENT ALARMS GNSS
ANTENNA : NOT DETECTED

GNSSAUX> gnss-showerrors
CURRENT ALARMS GNSS
ANTENNA : **NOT DETECTED
ERROR STATISTICS GNSS
RMC GOOD DURATION : Secs 0
RMC BAD DURATION : Secs 0
LOCK GOOD SECS DURATION : Secs 0
LOCK BAD SECS DURATION : Secs 0
SATINFO GOOD ITERATIONS :0
SATINFO REJECT ITERATIONS :0
SATINFO NOTALKER ITERATIONS :0
CURRENT MONITOR STATE GNSS
STATE :Phase-1 HUNTING ANTENNA DETECT

GNSSAUX> gnss-showsatinfo
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-showsats
Total Sats: 0

GNSSAUX> gnss-showmyloc
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-show1ppsstate
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-showjamstatus
Not Available !

GNSSAUX> gnss-showspoofstatus
Not Available !

--- antenna connected, can see the sky ---

GNSSAUX> gnss-showselftest
Overall : PASS
EPROM Test: PASS
Antenna : DETECTED
GNSS : COMMUNICATION OK

GNSSAUX> gnss-showalarms
CURRENT ALARMS GNSS
ANTENNA : DETECTED
GNSS LOCK : AVAILABLE

GNSSAUX> gnss-showerrors
CURRENT ALARMS GNSS
ANTENNA : DETECTED
GNSS LOCK : AVAILABLE
ERROR STATISTICS GNSS
RMC GOOD DURATION : Mins 1,Secs 34
RMC BAD DURATION : Secs 55
LOCK GOOD SECS DURATION : Mins 1,Secs 34
LOCK BAD SECS DURATION : Secs 55
SATINFO GOOD ITERATIONS :3
SATINFO REJECT ITERATIONS :0
SATINFO NOTALKER ITERATIONS :0
CURRENT MONITOR STATE GNSS
STATE :Phase-4 NORMAL OPERATION, Monitoring GNSSLOCK

GNSSAUX> gnss-showsatinfo
GNSS MODE :GPS
SATELLITE INFORMATION TALKER:GPS
NMEA ID : 01-32
NO OF SATELLITES IN VIEW: 08
NO OF XXGSV MSGS : 03
SatNo PRN NO (SV ID) ELEVATION (degs) AZIMUTH (degs) C/No (SNR)
1 0 0 0 0
2 0 0 0 0
3 0 0 0 0
4 0 0 0 0
5 0 0 0 0
6 0 0 0 0
7 0 0 0 0
8 0 0 0 0
<<<<<< End of Sat Info >>>>>>

GNSSAUX> gnss-showsats
Total Sats: 8

GNSSAUX> gnss-showmyloc
GNSS RECEIVER LOCATION:
Latitude : 4914.81911
Longitude: 12313.69595

GNSSAUX> gnss-show1ppsstate
GPS 1PPS STATE: LOCKED

GNSSAUX> gnss-showmyloc

GNSS RECEIVER LOCATION:
Latitude : 4914.80688
Longitude: 12313.69531

enter into google maps search box as: 49 14.80688, -123 13.69531, observe the space, the moved dot and the minus.

GNSSAUX> gnss-resetgnss

Executing....Please Wait....
$$$$$END
</pre>

* 1PPS BNC output: period 1 sec, pulse width 200 ms, 3.3V into 1MOhm, 1.38V into 50Ohm.
* IRIG-B BNC output: 5.6V into 1MOhm, 2.4V into 50Ohm.
* IRIG-B format selector: default is all up.

== Rb clock PRS10 ==

* https://www.thinksrs.com/products/prs10.html
* 10 MHz output is sine wave around 5V peak to peak
* 1PPS BNC output is 10 usec pulse, 5V into 1MHohm.
* RS232 connection: minicom -D /dev/ttyUSB0 -b 9600
* ser2net config: localhost,3001:raw:600:usb-5-2-1.0:9600 -XONXOFF -RTSCTS LOCAL
* ssh daq13, cd /home/olchansk/git/ser2net, ./ser2net -c ~/daq/ds/ser2net.conf -d
* ssh daq13, cd ~/daq/ds, python3 prs10.py
* RS232 commands:
<pre>
ID?
PRS10_3.56_SN_105719
VB1
SN?
RS1 -- reset
ST? -- status
FC? -- 10MHz OCXO drive voltage DAC settings
DS? -- "detected signals"
GA? -- gain of frequency lock loop between ovenized oscillator and Rb cell, 0=use ovenized oscillator only
MO? -- magnetic offset of the Rb cell, range 2300..3600, if out of range, unit must be set to different operating mode, see prs10m.pdf
MR? -- magnetic read
TT? -- time-tag, time in ns between 1PPS out and 1PPS in
TS? -- time slope, ???
TO? -- time offset, ???
PS? -- pulse slope, ???
PL? -- 0=phase lock off, 1=phase lock on, lock to 1PPS input
PT? -- phase lock integrator time constant, PT8 is integrator time constant 18.2 hours, natural time constant 2.25 hours
PF? -- phase lock stability factor, PF2 is "1"
PI? -- phase lock integrator

Analog to digital 12 bit ADC, values 0.000 to 4.998

AD0? -- Spare (J204)
AD1? -- +24V(heater supply) divided by 10.
AD2? -- +24V(electronics supply) divided by 10
AD3? -- Drain voltage to lamp FET divided by 10
AD4? -- Gate voltage to lamp FET divided by 10
AD5? -- Crystal heater control voltage
AD6? -- Resonance cell heater control voltage
AD7? -- Discharge lamp heater control voltage
AD8? -- Amplified ac photosignal
AD9? -- Photocell’s I/V converter voltage divided by 4
AD10? -- Case temperature (10 mV/°C)
AD11? -- Crystal thermistors
AD12? -- Cell thermistors
AD13? -- Lamp thermistors
AD14? -- Frequency calibration pot / external calibration voltage
AD15? -- Analog ground

A/D via CPU E-port:

AD16? -- Varactor voltage for 22.48 MHz VCXO (inside RF synthesizer) / 4
AD17? -- Varactor voltage for 360 MHz VCO (output of RF synthesizer) / 4
AD18? -- Gain control voltage for amplifier which drives frequency multiplier / 4
AD19? -- RF synthesizer’s lock indicator voltage (nominally 4.8 V when locked )

ST?

ST1 : Power supplies and Discharge Lamp
ST1 bit, Condition which sets bit, Corrective Action
0 -- +24 for electronic < +22 Vdc
1 -- +24 for electronics > +30 Vdc
2 -- +24 for heaters <+22 Vdc
3 -- +24 for heaters > +30 Vdc
4 -- Lamp light level too low
5 -- Lamp light level too high
6 -- Gate voltage too low
7 -- Gate voltage too high

ST2: RF Synthesizer
ST2 bit, Condition which sets bit, Corrective Action
0 -- RF synthesizer PLL unlocked
1 -- RF crystal varactor too low
2 -- RF crystal varactor too high
3 -- RF VCO control too low
4 -- RF VCO control too high
5 -- RF AGC control too low
6 -- RF AGC control too high
7 -- Bad PLL parameter

ST3: Temperature Controllers
ST3 bit, Condition which sets bit
0 -- Lamp temp below set point
1 -- Lamp temp above set point
2 -- Crystal temp below set point
3 -- Crystal temp above set point
4 -- Cell temp below set point
5 -- Cell temp above set point
6 -- Case temperature too low
7 -- Case temperature too high

ST4: Frequency Lock-Loop Control
ST4 bit, Condition which sets bit
0 -- Frequency lock control is off
1 -- Frequency lock is disabled
2 -- 10 MHz EFC is too high
3 -- 10 MHz EFC is too low
4 -- Analog cal voltage > 4.9 V
5 -- Analog cal voltage < 0.1
6 -- not used
7 -- not used

ST5: Frequency Lock to External 1pps
ST5 bit, Condition which sets bit
0 -- PLL disabled
1 -- < 256 good 1pps inputs
2 -- PLL active
3 -- > 256 bad 1pps inputs
4 -- Excessive time interval
5 -- PLL restarted
6 -- f control saturated
7 -- No 1pps input

ST6: System Level Events
ST6 bit and Condition which sets bit
0 Lamp restart
1 Watchdog time-out and reset
2 Bad interrupt vector
3 EEPROM write failure
4 EEPROM data corruption
5 Bad command syntax
6 Bad command parameter
7 Unit has been reset
</pre>
* ST? on warm start
<pre>
received: PRS_10
received: 255,255,255,243,34,255
received: 0,0,0,1,34,0
...
received: 0,0,0,1,34,0
received: 0,0,0,0,2,0
...
received: 0,0,0,0,2,0
received: 0,0,0,0,4,0
</pre>
* ST? on loss of external 1PPS
<pre>
...
received: 0,0,0,0,4,0
disconnect 1PPS input
received: 0,0,0,0,132,0
...
reconnect 1PPS input
received: 0,0,0,0,132,0
received: 0,0,0,0,4,0
...
</pre>
* ST? on coldish start
<pre>
daq13:ds$ python3 prs10.py
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,0,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 0,0,1,1,34,0 ] old [ 80,0,0,1,34,1 ] counter: 0
received [ 0,0,0,1,34,0 ] old [ 0,0,1,1,34,0 ] counter: 0
received [ 0,0,16,1,34,0 ] old [ 0,0,0,1,34,0 ] counter: 6
received [ 0,0,20,1,34,0 ] old [ 0,0,16,1,34,0 ] counter: 5
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 8
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 3
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 49
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 250
...
</pre>
* ST? on cold start, note: no bump in the 10MHz clock as reported by DS-DM clock chip
<pre>
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,21,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 64,0,21,1,34,0 ] old [ 80,0,21,1,34,1 ] counter: 0
received [ 0,0,21,1,34,0 ] old [ 64,0,21,1,34,0 ] counter: 0
received [ 0,0,20,1,34,0 ] old [ 0,0,21,1,34,0 ] counter: 158
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 23
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 47
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 40
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 249
</pre>

= det fac integration test =

Connections:
* GPS receiver "IRIG-B SEL" both switches "up" - both "on", IRIG-B format B004
* GPS receiver USB-B -> long cable -> daq13 USB-A
* GPS receiver "1PPS out" -> long BNC cable -> BNC-T -> scope (5V, no 50ohm) and Rb clock BNC "1PPS in"
* GPS receiver "IRIG-B 50ohms" -> long BNC cable -> BNC-T -> scope (5V, no 50ohm) and DS-DM LEMO input 1 (TTL mode)
* Rb clock RS232 -> RS232 straight cable -> RS232-to-USB adapter -> daq13 USB-A
* Rb clock "1PPS out" BNC -> scope (5V, no 50ohm, trig threshold rising edge 2V)
* Rb clock "10MHz output 50 Ohm" BNC -> lemo -> lemo-T -> scope (sine wave, 5V, no 50ohm) and DS-DM clock input.

Programs to run:
* on daq13: cd /home/olchansk/git/ser2net, ./ser2net -c ~/daq/ds/ser2net.conf -d
<pre>
localhost,3001:raw:600:usb-5-2-1.0:9600 -XONXOFF -RTSCTS LOCAL
</pre>
* on daq13: cd /home/olchansk/daq/ds, python3 prs10.py ### connects to ser2net
<pre>
daq13:ds$ python3 prs10.py
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,21,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 64,0,21,1,34,0 ] old [ 80,0,21,1,34,1 ] counter: 0
received [ 0,0,21,1,34,0 ] old [ 64,0,21,1,34,0 ] counter: 0
received [ 0,0,20,1,34,0 ] old [ 0,0,21,1,34,0 ] counter: 158
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 23
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 47
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 40
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 249
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 12096
received [ 0,0,0,0,20,0 ] old [ 0,0,0,0,132,0 ] counter: 2
received [ 0,0,0,0,148,0 ] old [ 0,0,0,0,20,0 ] counter: 0
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,148,0 ] counter: 575
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 11449
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 2755
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 34386
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 41035
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 113401
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 33375
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 54767
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 85059
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 33222
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 119234
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 121990
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 128184
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 56002
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 428237
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 8250
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 1388
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 30506
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 142704
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 179451
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 106182
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 68747
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 65424
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 157587
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 6932
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 1388
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 20255
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 4
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 225941
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 72183
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 26970
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 1
39287
</pre>
* on ds-dm: ./test_cdm.exe --irigb ### note sbs mismatch is because I should wrap around at 16 bits
<pre>
dataframe: S00000000S100001010S...S001010101S000001000S, sec: 00, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16724 should be 82260
dataframe: S10000000S100001010S...S101010101S000001000S, sec: 01, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16725 should be 82261
dataframe: S01000000S100001010S...S011010101S000001000S, sec: 02, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16726 should be 82262
dataframe: S11000000S100001010S...S111010101S000001000S, sec: 03, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16727 should be 82263
dataframe: S00100000S100001010S...S000110101S000001000S, sec: 04, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16728 should be 82264
dataframe: S10100000S100001010S...S100110101S000001000S, sec: 05, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16729 should be 82265
</pre>

= DS-IOGC GPS interface board =

* Rev0 git repository: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0/-/tree/main?ref_type=heads
* Rev0 schematics: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0/-/blob/main/Altium/Project%20Outputs%20for%20DS-IOGC-Rev0/SCH-DS-IOGC-Rev0.pdf?ref_type=heads
* Rev0 schematics: [[:Image:SCH-DS-IOGC-Rev0.pdf|SCH-DS-IOGC-Rev0]]
* Rev1 git repository: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev1/
* Rev1 schematics: [[:Image:SCH-DS-IOGC-Rev1.pdf|SCH-DS-IOGC-Rev1]]

== Changes Rev0 to Rev1 ==

<pre>
From: Peter Margetak <pmargetak@triumf.ca>
Subject: IOGC REV1 review
Date: Wed, 4 Sep 2024 07:31:19 +0000

Hi Konstantin,
Pls have a look at SCH for new rev. I'd like to send it to mfr next week so if you can comment by early next week. Meanwhile I work on layout and other stuff.

Changes:

New ICs - all powered +5V
U20 - inverters for RUclk RX/TX
U21 - non inverting line driver for RU-1pps-out (so you don't have to route if via GDM to see it on scope)
U22 - non inverting buffer for ext 1pps input

All Lemo connectors have the same position but they are double lemos now => new panel needed
@Marek Walczak<mailto:mwalczak@triumf.ca> you can print it ahead once pcb is done + update IOGC docs and panel description

J2A/B - Test ports for RU-1pps in and out
J5A/B - inputs for external GPS data and External source of 1pps
J6A/B - aux in/out for GDM

SW1 - no change - select RX/TX USB/GDM
SW2 - select latch sensitivity for rising/falling edge
SW3 - select source of GPS data (opto or ext) AND select source of 1pps input (latch or ext)

p.
</pre>

== PRS-10 Rb clock device ==

The Rb clock PRS-10 device provides these connections:
<pre>
RS232 RX input - serial communication, non-standard RS232
RS232 TX output - serial communication, non-standard RS232
10 MHz clock output - coax 50 Ohm high resolution 10 MHz clock
1pps output - 1 Hz clock corresponding to the 10 MHz clock
1pps input - 1pps signal from GPS receiver
</pre>

Mode of operation:
* 10 MHz clock is always running
* 1pps output is always running
* if 1pps input from GPS received is present, after 256 pulses PRS-10 will sync it's 1pps output with the 1pps input by adjusting the frequency of the 10 MHz clock
* when unlocked: 1pps output and 1pps input unrelated
* when locked to GPS: 1pps output and 1pps input always go up and down at the same time

Theory of operation:
* 10 MHz clock is produced by a high-quality crystal (stable on the scale of seconds)
* crystal oscillator is synchronized to a Rb cell (stable on the scale of hours and days)
* Rb cell resonant frequency is synchronized to the GPS 1pps signal (stable on the scale of months and years)

== Rev1 connections ==

* LEMO connectors (front panel)
<pre>
LEMO J2A output - Rb clock 1pps in monitor
LEMO J2B output - Rb clock 1pps out monitor
LEMO J5A input - GPS IRIG-B from GPS receiver to FPGA (VCL-2705)
LEMO J5B input - GPS 1pps from GPS receiver to PRS-10 (VCL-2705)
LEMO J6A output - AUX-OUT from FPGA (dual-LEMO PCB-side)
LEMO J6B input - AUX-IN to FPGA (dual-LEMO away-from -PCB)
</pre>

* SMB connectors (back)
<pre>
SMB J3 output - GPS 1pps loopback to LNGS
SMB J4 input - LNGS GPS data input
</pre>

* LEDs
<pre>
D1 - same as LEMO J2A out (Rb clock 1pps in)
D2 - same as SMB J3 out (GPS 1pps from LNGS or from a GPS receiver)
D5 - controlled by FPGA-OUT-LED1
D6 - controlled by FPGA-OUT-LED2
D7 - PRS-10 24V power ok
</pre>

* switches
<pre>
SW1A and SW1B - route PRS-10 RS232 to USB or to FPGA
SW2A - route PRS-10 1pps input from SMB J4 (LNGS) or from LEMO J5B (GPS receiver 1pps)
SW2B - route FPGA-IN-GPSDATA input from SMB J4 (LNGS) or from LEMO J5A (GPS receiver IRIG-B data)
SW3 - LNGS 1pps from rising edge or from falling edge of SMB J4 (LNGS)
</pre>

== Rb clock cable ==
<pre>
Rb clock ----- DS-IOGC side, pin numbers are as labeled on the cable connectors

1 - 1pps out - 8 - 1pps out --- correct
2 - nc
3 - nc
4 - TXD - 3 - RU-DATA-OUT - USB-RX input --- correct
5 - 1pps in - 2 - 1pps in --- correct
6 - +24V - 10 - +24V --- should by pin 1 to use both +24V pins?
7 - RXD - 15 - RU-DATA-IN - USB-TX output --- correct
8 - nc
9 - +24V - 10 - +24V --- correct
10 - GND - 9 - GND --- correct
</pre>

== VX connections ==
<pre>
VXA_TX0 - FPGA-OUT-LED2 - D6 LED ("10 MHz clock")
VXA_TX1 - FPGA-OUT-LED1 - D5 LED ("GPS DATA")
VXA_TX2 - FPGA-OUT-RU1PPS-EN - enable 1pps to Rb clock
VXA_TX3 - not used (62.5 Hz clock)

VXB_TX0 - FPGA-TX - PRS-10 RS-232 out
VXB_TX1 - FPGA-OUT-AUX - 5V TTL J5 LEMO out
VXB_TX2 - FPGA-OUT-OPTO1PPS-EN - PRS-10 power enable
VXB_TX3 - not used (62.5 MHz clock)

VXA_RX0 - FPGA-IN-OPTO1PPS - 1pps from GPS
VXA_RX1 - FPGA-IN-GPSDATA - J4 GPS data
VXA_RX2 - n/c
VXA_RX3 - n/c

VXB_RX0 - n/c
VXB_RX1 - FPGA-IN-AUX - J5 LEMO TTL input
VXB_RX2 - FPGA-RX - PRS-10 RS-232 in
VXB_RX3 - FPGA-IN_RU1PPS - PRS-10 1pps output
</pre>

== test sequence ==

* ./test_cdm_local.exe --writereg 7 0 ### clear reg 7
* ./test_cdm_local.exe --writereg 8 0 ### set vx_tx mux to vx_tx control from reg 7
* connect blue cable to GDM port 6 (next to the ethernet connector)
* ./test_cdm_local.exe --writereg 7 0x4000 ### power up
* test LEDs:
* ./test_cdm_local.exe --writereg 7 0x4100 ### right LED2 D6
* ./test_cdm_local.exe --writereg 7 0x4200 ### left LED1 D5
* test AUX-IN and AUX-OUT:
* ./test_cdm_local.exe --writereg 7 0x6000 ### J6A LEMO (PCB-side) measure +5VDC
* install LEMO jumper between LEMO J6A and J6B
* write 0x4000 then ./test_cdm_local.exe 6 ### reads 0x5C, bit 5 0x20 reads 0
* write 0x6000 then ./test_cdm_local.exe 6 ### reads 0x7C, bit 5 0x20 reads 1
* write 0x4000 to clear all bits
* test the 10 MHz clock:
* init the c.c: ./test_cdm_local.exe --load-cc
* connect 10 MHz clock output to DS-DM LEMO J6-LEFT (CLK_EXT1)
* 10 MHz output ok - IN0 is good: ./test_cdm_local.exe --cc
<pre>
Clock chip state 1, status: OOF_IN2 IN1 IN_SEL_1 HOLD_HIST_VALID
</pre>
* without 10 MHz reports IN0 LOS (no signal) and OOF (wrong frequency)
<pre>
Clock chip state 1, status: LOS_IN0 OOF_IN0 OOF_IN2 IN1 IN_SEL_1 HOLD_HIST_VALID
</pre>
* test GPS IRIG-B data:
* connect GPS IRIG-B signal to J6B (dual-LEMO away from PCB)
* IRIG-B via AUX-IN is ok: ./test_cdm_local.exe --irigb
<pre>
dataframe: S00100110S111001000S...S011110000S010000000S, sec: 34, min: 17, hrs: 00, day of year: 223, year: 2024, date: 10 aug, sbs: 1054 should be 1054
</pre>
* test IRIG-B via J5A GPS-EXT:
* connect GPS IRIG-B signal to J5A (dual-LEMO next to PCB)
* set SW2B: switch on the side of SMB connectors, slide direction of 24V power supply connector
* ./test_cdm_local.exe --irigb built for VXA_RX1 should work
* test GPS 1pps signal:
* connect GPS 1pps to J5B (dual-LEMO away from PCB)
* nothing happens, no output from U22
* remove R51
* set SW2A: switch on the side away from SMB connectors, slide direction of 24V power supply connector
* observe LEDs D1 and D2 blink at 1Hz
* SKIP THIS connect GPS 1pps to J4 using LEMO-SMB adapter
* SKIP THIS nothing happens, nLE is always 3.2V, both settings of SW2
* SKIP THIS !!!failed here!!!
* ./test_cdm_local.exe --writereg 8 9 ### set vx_tx mux to GPS control, hard enables PRS-10 power and PRS-10 1pps in
* observe 1pps counters in reg 68 count at 1 Hz: rb_1pps 0x51->0x52, gps_1pps 0xd5->oxd6
<pre>
root@dsdm:~# ./test_cdm_local.exe 68
ds20k_reg[68] is 0x0051d520 (5362976)
ds20k_reg[68] is 0x0052d624 (5428772)
...
</pre>
* observe PRS-10 can see the 1pps signal "130" changes to "2" after 243 seconds to "4"
* observe period of GPS and PRS-10 1pps is identical (plus/minus 1 clock)
<pre>
root@dsdm:~# ./test_cdm_local.exe 13 14
ds20k_reg[13] is 0x077356d4 (124999380)
ds20k_reg[14] is 0x077356d4 (124999380)
</pre>
* observe GDM/CDM clocks are exactly 125 MHz, we are running on DS-DM internal oscillator
<pre>
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735943 (125000003) should be ~125 MHz
0x1034 rx_clk: 0x07735943 (125000003) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735943 (125000003) should be ~125 MHz
0x103C tx_clk: 0x07735943 (125000003) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>
* if tx_clk is missing, reset the mgt: ./test_cdm_local.exe --reset-mgt
* switch CC to external clock:
<pre>
root@dsdm:~# ./test_cdm_local.exe --cc-in0
CC use clock input 0: 10 MHz LEMO external clock
root@dsdm:~# ./test_cdm_local.exe --cc
Polling CC status...
Clock chip state 1, status: IN0 IN_SEL_REGCTRL IN_SEL_0 HOLD_HIST_VALID
</pre>
* observe rx_clk and tx_clk are now slightly off: they run from PRS-10 10 MHz clock and are measured against the DS-DM internal oscillator.
<pre>
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x94b12519, ds20k version 0x20240814
DS-DM firmware build 0x94b12519, ds20k version 0x20240814
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x077358e1 (124999905) should be ~125 MHz
0x1034 rx_clk: 0x07735b49 (125000521) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735b49 (125000521) should be ~125 MHz
0x103C tx_clk: 0x07735b49 (125000521) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>
* observe GPS and PRS-10 1pps period is now exactly 125*10^6 of 8ns clocks (equal to 1 second)
<pre>
root@dsdm:~# ./test_cdm_local.exe 13 14
ds20k_reg[13] is 0x0773593f (124999999)
ds20k_reg[14] is 0x0773593f (124999999)
</pre>
* look at them repeatedly, observe reg 13 "GPS 1pps period" has some wobble, reg 14 "Rb clock 1pps period" is steady. this is as expected: if GPS 1pps was steady, we do not need to Rb clock. PRS-10 1pps is derived from the PRS-10 10 MHz clock and is measured against the 125 MHz clock derived from the same 10 MHz clock in the GDM FPGA.

Rev0 Test status:

* GPS 1pps to SMB-in ok (LED flashes)
* GPS 1pps to FPGA ok
* GPS data to FPGA ok
* GPS 1pps to PRS-10 enabled from FPGA ok
* PRS-10 Rb clock 1pps out to FPGA ok
* PRS-10 Rb clock 1pps out to FPGA to LEMO AUX out ok
* can see GPS 1pps, IRIG-B, PRS-10 1pps out, 10 MHz on the scope, ok
* PRS-10 syncs on leading edge (0->1) of GPS 1pps signal, ok
* reg 13 and 14 1pps periods are identical, ok
* CC locks on PRS-10 10 MHz clock, DS-DM runs on PRS-10 clock, ok
* NOT TESTED - smb output
* NOT TESTED - optical converter fiber to SMB
* NOT TESTED - optical converter SMB to fiber
* NOT TESTED - SMB loopback
* NOT TESTED - fiber loopback

Rev1 Test status: (21 Nov 2024)

* power up ok
* LEDs ok
* LEMO AUX-IN, AUX-out ok
* GPS IRIG-B to FPGA via J6B AUX-IN ok
* GPS IRIG-B to FPGA via J5A GPS-EXT ok
* PRS-10 Rb clock 1pps out to FPGA ok
* PRS-10 Rb clock 10 MHz to C.C. ok
* PRS-10 RS-232 ok
* J5B 1pps input fail. input signal is 3.3V without termination, 1.5V when plugged into IOGC (on R51), no output from U22.

= VX busy logic =

= DS-20K DAQ =

== Overview ==

DS-DM, GDM and CDM are key parts of the DS-20K DAQ system:
* common clock distribution from external clock (atomic clock, GPS) to GDM to per-quadrant CDMs to VX digitizers
* common trigger distribution from GDM internal algorithm or external input to all VX digitizers
* run control: GDM, CDM, VX all start recording data at the same time (clock and timestamp reset)
* collection of trigger data from VX digitizers to per-quadrant CDMs to GDM

== Deliverables ==

* hardware and firmware for GDM to CDM clock distribution
* hardware and firmware for CDM to VX clock distribution
* hardware and firmware for GDM external clock input (atomic clock or GPS)
* hardware and firmware for CDM and VX serial communications (VX LVDS I/O connector)
* firmware for run control (timestamp reset and sync): GDM to CDM to VX
* firmware for common trigger distribution: GDM to CDM to VX
* firmware for trigger data flow: VX to CDM to GDM
* firmware for busy control: VX to CDM to GDM back to CDM to VX
* firmware for flow control: FEP to GDM MIDAS frontend to GDM to CDM to VX
* GDM MIDAS frontend: clock selector and monitoring, trigger and run control, busy and flow control, GDM housekeeping
* CDM MIDAS frontend: clock monitoring, CDM housekeeping

specific performance:
* GDM external clock: 10 MHz GPS clock
* GDM to CDM fiber link:
** clock XXX MHz
** link data rate: XXX Gbit/sec
** CDM recovered clock: XXX MHz
** CDM recovered clock jitter: XXX ns
** phase alignment between CDMs: XXX ns
** phase alignment between CDMs persists across reboots, power cycles, firmware updates
** phase alignment between CDMs should be easy to measure
** phase alignment between CDMs should be easy to recalibrate if hardware parts are replaced (DS-DM boards, fiber transceivers, fiber cables, etc)
** data packet bandwidth: XXX Mbytes/sec
** data packet latency: XXX clocks
** data packet skew between CDMs: XXX clocks
* CDM to VX clock:
** clock: XXX MHz
** jitter, all CDM clock outputs: XXX MHz
** phase alignment between all CDM clock outputs: XXX ns
* CDM to VX trigger:
** TBD (use the VX "sync" input or VX LVDS I/O line or VX serial link packet)
* CDM to VX serial link:
** clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
** bit rate: XXX bits/sec
** latency: XXX link clocks
** maximum skew between VXes: XXX ns
* VX to CDM serial link:
** clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
** bit rate: XXX bits/sec
** latency: XXX link clocks
** maximum skew between VXes: XXX ns
* timestamp reset:
** maximum skew between VXes: XXX ns
* busy round trip time: XXX ns (VX to CDM to GDM back to CDM to VX)
* flow control latency: XXX ns (FEP software to GDM MIDAS frontend to GDM to CDM to VX)

== Technical risk items ==

this refers to unexpected behaviour and performance of
system components, causes big difficulty in implementing the system,
prevents delivery of deliverables, and prevents or negatively affects operation
of the DS-20K DAQ or of the whole experiment.

(14-sep-2022, list is not sorted by any criterial: severity, probability, ease of investigation)

(stability of course is long term stability, across hours, days, weeks, months, years)

* stability of Enclustra FPGA modules (crashes/year, failures to boot/year, flash corruption/year)
* stability of GDM external clock PLL (lock loss/year)
* stability of CDM recovered clock (lock loss/year, unexpected phase drifts, etc)
* unexpected failures or bit error rates in GDM-CDM fiber links
* stability of CDM VX clock outputs (stability of clock cleaner chip)
* stability of VX internal clock distribution (VX PLL lock loss events)
* stability of VX CAEN base firmware (different versions of CAEN base firmware have different clock distribution behaviour)
* strange things in CAEN base firmware (unexpected clocking of LVDS I/O, unexpected phase shifts between clocks, etc)
* DS-DM and VX hardware problems (incompatible LVSD I/O, incompatible clock signals, etc)

== Milestones ==

(14-sep-2022: at this stage of the project, priority must be given to identifying and retiring (so called) technical risk factors. it is not good to build the complete system only to discover that (for example) some Enclustra FPGA modules require 5 attempts to boot and erase their flash memory contents once a month. Both example are real-life actual problems that caused big difficulties in GRIFFIN/TIGRESS and ALPHA-g experiments).

Development and testing milestones in time reversed order:

* full DAQ data challenge: all VXes, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
* one quadrant data challenge: 1 VME crate of VX, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
* vertical slice data challenge: 1 VME crate, 2 VX, 2 CDM (1 VX per CDM), GDM, etc operate as designed
* GDM-CDM link finalized (data rate frozen, data packet format frozen, data content permitted to change)
* CDM-VX serial link finalized (data rate frozen, data packet format frozen, data content permitted to change)
* run control (timestamp reset) and trigger distribution design agreed upon, frozen (list of possible triggers permitted to change)
* VX to CDM to GDM data flow design agreed upon, frozen (data contents permitted to change)
* major technical risk items retired (all hardware and firmware is working as expecred without mysteries and surprises, all problems are identified, investigated, resolved, solutions tested)
* stable operation of CDM-VX serial links in vertical slice system
* stable operation of GDM to CDM clock in vertical slice system
* stable operation of CDM to VX clock in vertical slice system
* vertical slice system assembled (1 VME crate, 2 VX, 2 CDM, 1 GDM, network, FEP, TSP, MIDAS)

= ZZZ =

ZZZ

DS-DM

2026-04-01T23:46:49Z

Olchansk: /* DS20k block register map */

= DS-DM =

DarkSide-20k Global and Crate Data Manager board (GDM and CDM).

Global Data Manager (GDM):
* clock distribution to CDM boards (including GPS/atomic clock source)
* collection of trigger data from CDM boards, processing and distribution of trigger decision to CDM boards
* run control
* integration with GPS 10MHz and 1pps clocks and GPS/IRIG date and time information

Crate Data Manager (CDM):
* clock distribution from GDM to CAEN VX digitizers
* receive trigger data from CAEN VX digitizers
* send trigger data to GDM
* run control and dead time control

= Links =

* https://dsdaqgw.triumf.ca/vslice/ - vertical slice DAQ at TRIUMF
* https://daq00.triumf.ca/DaqWiki/index.php/DarkSide - this DS-DAQ Wiki page
* https://dsvslice.triumf.ca/vx_napoli - DS vertical slice at TRIUMF
* https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0 - DS-DM board Rev0
* https://edev-group.triumf.ca/hw/exp/dark-side-20k/cdm-gdm-data-manager-vme-cards/rev1 - DS-DM board Rev1
* https://docs.xilinx.com/v/u/en-US/ds593 - Xilinx Platform Cable USB II
* https://www.enclustra.com/en/products/system-on-chip-modules/mercury-xu8/ - Enclustra
* https://bitbucket.org/team-ds-dm/phasemeasurement - VX Phase measurement script
* https://daq00.triumf.ca/daqinv/frontend/list/178 - inventory database
* https://daq00.triumf.ca/elog-ds/DS-DAQ - DS-DAQ elog
* https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0 - GPS and Rb Clock adapter board

= Git repos =

* dsdmdev@dsdaqgw.triumf.ca:git/gcdm.git - git repository, DS-DM firmware, branch develop_ko2, contact K.O. for ssh key access
* https://edev-group.triumf.ca/fw/exp/darkside/gcdm - git repository, DS-DM firmware (broken, do not use)
* https://bitbucket.org/team-ds-dm/ds-dm-software - git repository, DS-DM MIDAS frontend
* https://bitbucket.org/ttriumfdaq/dsproto_vx2740/src/develop/ - VX DSFE MIDAS frontend
* https://openproject.triumf.ca/svn/darkside-20k/ - VX firmware project

= Onboard hardware =

* jtag chain: arm_dap_0 0x5BA00477, xczu4_1 0x04721093
* Eclustra Mercury+ XU8 module: ME-XU8-4CG-1E-D11E-R2.1
** Xilinx® Zynq Ultrascale+™ MPSoC XCZU4CG-1FBVB900E
** DDR4 ECC SDRAM (PS) 2 GB
** DDR4 SDRAM (PL) 1GB
* ethernet mac chip: AT24MAC402-SSHM-T ("602" chip is wrong)
* USB UART for Enclustra serial console, micro-USB, 115200n8
* clock chip: SI5394A-A-GM and oscillator CS-044-054.0M (54 MHz)
* U23 3.3V current meter and thermometer, LTC2990IMS#TRPBF
* LEDs:
** LED_FP A/B/C/D 0/1/2/3
** led1 - 3V3_SW_ON, SOM_POWER_GOOD - Enclustra FPGA module 3.3V power is good
** led2 - LTM4624 PGOOD
** led3 - FPGA_DONE - FPGA has booted
** led4 - TP-S-1, PCLK_P
** led5 - TP-S-2, PCLK_N
* LEMO connectors (top to bottom)
** J4 - input (NIM/TTL) (EXT_IN_LV(1), EXT_IN_LV(2))
** J5 - input (NIM/TTL) (EXT_IN_LV(3), EXT_IN_LV(4))
** J6 - external clock (GPS 10MHz and PPS)
** J7 - output (NIM/TTL) (EXT_OUT(1), EXT_OUT(2))
* SMA connectors
** J9, J10 - CLK_CCA from U6 C.C.
** J11, J12 - CLK_TP0
* RJ45 ethernet connector (100mbit: green light on, yellow light flashes when there is traffic)
* SFP connector (SFP is FTLF8526P3BNL, 6 Gbit/sec, 850 nm, 300m 50/125um OM3 MMF)
* 4 QSFP connectors (GDM)
* 6 VX connectors (CDM)

= Buttons, jumpers and switches =

Buttons:

* PB1 - HRST - reboot FPGA (power-on reset)
* PB2 - SRST - (SRSTn) - reboot ARM CPU

Switches:

* SW1 - boot mode BM0, BM1 [-->]
* SW2 - LEMO output NIM<->TTL
* SW3 - LEMO input 1 and 2 NIM/TTL
* SW4 - LEMO input 2 and 4 NIM/TTL
* SW5 - LEMO clock input NIM/TTL
* SW6 - serial console select. [PS<--PL] PS is ARM CPU, PL is FPGA.

= Front panel =

<pre>
| top
|
| LED-FP1 | LED_FP(0,1,2,3)
|
| SFP J???
|
| LEMO J4-LEFT, J4-RIGHT | EXT_IN_LV(1), EXT_IN_LV(2)
| LEMO J5-LEFT, J5-RIGHT | EXT_IN_LV(3), EXT_IN_LV(4)
| LEMO J6-LEFT, J6-RIGHT | CLK_EXT1, CLK_EXT0 (125 MHz only)
| LEMO J7-LEFT, J7-RIGHT | EXT_OUT(1), EXT_OUT(2)
|
| J-VX-1
| J-VX-2 or QSFP-1
| J-VX-3 or QSFP-2
| J-VX-4 or QSFP-3
| J-VX-5 or QSFP-4
| J-VX-6
|
| RJ45 J3 ethernet
|
| bottom
</pre>

= VX adapter board =

LVDS I/O connector
<pre>
34 pin connector: 0|:::: :::: :::: :::: :|15,16 n/c
</pre>

== split-cable connection ==

<pre>
0 .. 7 -> N/C

8 -> VX_RX(3) - not used
9 -> VX_RX(2) - busy VX to CDM
10 -> VX_RX(1) - DS20K 125 MHz serial data VX to CDM
11 -> VX_RX(0) - DS20K 62.5MHz clock VX to CDM

12 <- VX_TX(0) - TRG CDM to VX
13 <- VX_TX(1) - TSM CDM to VX, to be VETO CDM to VX
14 <- VX_TX(2) (set by jumper routed here or to VX CLKIN SYNC) - 125 MHz serial data CDM to VX
15 <- n/c (CLK routed to VX CLKIN CLK) - 62.5 MHz clock CDM to VX

16 - n/c
</pre>

== one-to-one connection ==

<pre>
0 -> VX2_RX(3)
1 -> VX2_RX(2)
2 -> VX2_RX(0)
3 -> VX2_RX(1)

4 <- CLK
5 <- VX2_TX(0)
6 <- VX2_TX(1)
7 <- VX2_TX(2)

8 -> VX1_RX(3)
9 -> VX1_RX(2)
10 -> VX1_RX(1)
11 -> VX1_RX(0)

12 <- VX1_TX(0)
13 <- VX1_TX(1)
14 <- VX1_TX(2) (set by jumper routed here or to VX CLKIN SYNC)
15 <- n/c (CLK routed to VX CLKIN CLK)

16 - n/c
</pre>

= Board schematics Rev0 =

* [[File:SCH-DS-xDM-Rev0.PDF]]
* note: FPGA pin annotations ("IO", "SCLK", "PCLK", etc) on the schematics are bogus, instead, trace them to the FPGA pins.
* note: ENC A is J800, ENC B is J801, ENC C is J900 (schematic name to enclustra name)
* note: Enclustra special pins: "GC" is "clock capable", "HDGC" is "clock capable", "MGTREFCLK" is MGT reference clocks.
* board modifications:
** ethernet mac chip
** NIM output (no U15, etc)
** RJ45 wrong pinout (board mod or special ethernet cable)
** 125 MHz clock mods (TBW)
** disconnect QSFP0_SEL from SFP_RS0 and QSFP1_SEL from SFP_RS1, these signals are not used by modern SFPs
** provide SFP i2c modsel to allow SFP and QSFP at the same time on address 0x50

= Board schematics Rev1 =

* [[File:SCH-DS-xDM-Rev1.PDF]]
* ENC-A is J800, ENC-B is J801, ENC-C is J900
* modifications from Rev0:
* clocks:
** 125MHz osc: CLK_XO_125, CLK3_XO_125 (CLK2_XO_125 removed)
** C.C. in2 is CLK_CC_IN from ENC-C142,144
** C.C. out0 via CLK_A to CLK_CC_OUT0 (ENC-C3,5), CLK_CC_OUT1 (ENC-B3,5), CLK_CC_OUT2 (ENC-C151,153)
* RTC chip: TP_S pins gone, RTC_I2C_SCL, RTC_I2C_SDA, RTC_1PPS and RTC_32KHZ added
** RTC_1Hz ENC-C160 pull up 10k to 3.3V
** RTC_32k ENC-B129 pull up 10k to 1.8V
* QSFP, SFP control lines:
** HW_ID_xDM ENC-B131 gone, reused as SFP_ModPrsN
** SFP_ModPrsN (MOD_ABS) ENC-B131 input (no pulls)
** SFP_TX_fault (was QSFP2_SEL) QSFP_ModPrsN ENC-C157 input (no pulls)
** SFP_TX_disable from QSFP_LPMode ENC-C163 output (pull up 10k to 3.3V)
** SFP_ModDet n/c (was QSFP_ModPrsN) renamed SFP_ModPrsN input
** SFP_RX_LOS to QSFP_IntN ENC-C159 input (pull up 2k2 to 3.3V)
** SFP_RS0, SFP_RS1 n/c, assembly option: floating
** QSFP_ResetN - FPGA output, pull up 10k to 3.3V
** QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL - FPGA outputs, pull up 10k to 3.3V
** QSFP_LPmode - FPGA output, pull up 10k to 3.3V
** QSFP_ModPrsN - ??? - pull up 2k2 to 3.3V
** QSFP_IntN - ??? - pull up 2k2 to 3.3V
* VX_TX and VX_RX reassigned for better length matching
* cosmetic changes:
** front panel USB-C connector (replaces Micro-USB)
** (reset push button PB1 PORn ENC-A132 aka PS_POR_B aka PS_POR#)
** (reset push button PB2 SRSTn ENC-A124 aka PS_SRST_B aka PS_SRST#)
** front panel PB3 HRSTn same as PB1
* Enclustra errata:
** possible cross talk between pins AA2 (J801-B90, VX9_RX2) and AE3 (J801-B129, RTC_32KHZ). no RTC on the CDM.
** ethernet link up failure (see Microchip KSZ9031 errata)

= FPGA MGT blocks =

<pre>
* SFP - ENC B45,B47 and B48,B50 - FPGA D5,D6 MGTHTX0_D and D1,D2 MGTHRX0_D
* QSFP0 TX0 - ENC C13,C17 - FPGA H5,H6 MGTHTX0_C
* QSFP0 TX1 - ENC C21,C25 - FPGA G7,G8 MGTHTX1_C
* QSFP0 TX2 - ENC C29,C23 - FPGA F5,F6 MGTHTX2_C
* QSFP1 TX0 - ENC C37,C41 - FPGA E7,E8 MGTHTX3_C
* QSFP1 TX1 - ENC C45,C47 - FPGA P5,P6 MGTHTX0_B
* QSFP1 TX2 - ENC C51,C53 - FPGA M5,M6 MGTHTX1_B
* QSFP2 TX0 - ENC C57,C59 - FPGA L3,L4 MGTHRX2_B
* QSFP2 TX1 - ENC C63,C65 - FPGA K5,K6 MGTHTX3_B
* QSFP2 TX2 - ENC C75,C77 - FPGA W3,W4 MGTHTX0_A
* QSFP3 TX0 - ENC C79,C81 - FPGA V5,V6 MGTHTX1_A
* QSFP3 TX1 - ENC C85,C87 - FPGA T5,T6 MGTHRX2_A
* QSFP3 TX2 - ENC C89,C91 - FPGA R3,R4 MGTHTX3_A
</pre>

= Clock distribution Rev0 =

Simplified:

* 125 MHz osc -> CLK_XO_125 -> MGTREFCLK0_A -> not used
* 125 MHz osc -> CLK3_XO_125 -> MGTREFCLK1_B -> SFP RX ref clock, QSFP RX and TX ref clock (this is not final design!)
* (disconnected) 125 MHz osc -> CLK2_XO_125 -> FPGA AG8,AH8 (GC)
* 125 MHz osc -> C.C. in1

* C.C. in0 <- CLK_EXT1 (10 MHz GPS clock)
* C.C. in1 <- 125 MHz osc
* (disconnected) C.C. in2 <- CLK_CC_IN <- MGTREFCLK0_D <- SFP RX clock (cannot be used because of uncontrollable phase)
* C.C. in2 <- CLK2_XO_125 <- FPGA AG8,AH8 (GC) <- SFP RX recovered 125 MHz clock
* C.C. in3 <- CLK_FB

* C.C. 125 MHz -> CLK_CC_OUT0 -> MGTREFCLK0_B -> QSFP RX and TX ref clock (final design)
* C.C. 125 MHz -> CLK_CC_OUT1 -> MGTREFCLK1_D -> SFP TX clock
* C.C. 62.5 MHz -> VX clock fanout

proposed changes:
* add C.C. 125 MHz -> new CLK_CC_OUT2 (old CLK2_XO_125) -> FPGA AG8,AH8 (GC)
* change C.C. in2 <- new CLK_CC_IN <- FPGA AK8,AK9 (non-GC)

Complete:

<pre>
125 MHz oscillator - U5 fan out -
q0 -> CLK_XO_125 -> ENC C72,C74 -> FPGA R7,R8 MGTREFCLK0_A (not used)
q1 -> U6 C.C. in1
q2 -> CLK3_XO_125 -> ENC C7,C9 -> FPGA J7,J8 MGTREFCLK1_B -> SFP RX reference clock, QSFP RX and TX reference clocks (not final design!)
q3 -> disconnected on the board, was CLK2_XO_125 -> ENC C151,C153 -> FPGA AG8,AH8 (GC)

U6 C.C (clock cleaner) -

in0 <- CLK_EXT1 (presumably GPS 10 MHz ref clock)
in1 <- 125 MHz oscillator via U5
in2 <- (was: CLK_CC_IN <- ENC B10,B12 <- FPGA D9,D10 MGTREFCLK0_D <- SFP RX clock, 125 MHz)
in2 <- CLK2_XO_125 <- ENC C151,B153 <- FPGA AG8,AH8 (GC) <- mgt_link_data_to_processing.rx_data_clk (SFP RX data clock, 125 MHz)
in3 <- CLK_FB

out0 -> CLK_CCA -> U12 (125 MHz)
out1 -> CLK_CCB -> VX1..6 (62.5 MHz)
out2 -> CLK_CCC -> VX7..12 (62.5 MHz)
out3 -> CLK_FB into in3

CLK_CCA -> U12 (125 MHz fan out) ->

Q0 -> not used
Q1 -> CLK_CC_OUT0 -> ENC C3-5 -> FPGA L7,L8 MGTREFCLK0_B -> QSFP RX and TX reference clocks (final design)
Q2 -> CLK_CC_OUT1 -> ENC B3-5 -> FPGA B9,B10 MGTREFCLK1_D -> SFP TX clock
Q3 -> not used
Q4 -> not used
Q5 -> SMA J9/J10

CLK_TP0 - FPGA N7,N8 MGTREFCLK1_A - ENC C69,C71 - DS-DM SMA J11, J12 (NOT IN CDM PROJECT)

Notes:
* CLK_XO_125 (125 MHz osc) is not used
* 62.5 MHz VX clock does not go into the FPGA
* CLK_EXT0 going to in1 of U5 cannot be used. only permitted frequency is 125 MHz (it drives the MGT reference clocks) and is this frequency is too high for LEMO cables and connectors

Proposed modifications:
- CLK_CC_IN: from FPGA output pin (ENC C142,C144 - FPGA AK8,AK9) to clock cleaner in2 (instead of CLK2_XO_125 pins)
- CLK_CCA -> U12 -> currently unused out3 -> CLK2_XO_125 FPGA pins
- repurpose CLK_EXT0 at GPS 1pps/IRIG input to FPGA
</pre>

= Clock distribution Rev1 =

Simplified:

* 125 MHz osc -> CLK_XO_125 -> MGTREFCLK0_A -> clk_xo_125. not used
* 125 MHz osc -> CLK3_XO_125 -> MGTREFCLK1_B -> CDM SFP RX ref clock
* 125 MHz osc -> C.C. in1

* C.C. in0 <- CLK_EXT1 (10 MHz GPS clock)
* C.C. in1 <- 125 MHz osc
* C.C. in2 <- CLK_CC_IN <- FPGA AK9,AK8 <- CLK_CC_IN1, rx_clk_mgt, SFP RX recovered clock, 125 MHz
* C.C. in3 <- CLK_FB

* C.C. 125 MHz -> CLK_CC_OUT0 -> MGTREFCLK0_B -> GDM QSFP RX and TX ref clock, CDM rx_clk mux
* C.C. 125 MHz -> CLK_CC_OUT1 -> MGTREFCLK1_D -> SFP TX clock
* C.C. 125 MHz -> CLK_CC_OUT2 -> FPGA AG8,AH8 (GC) -> not used
* C.C. 62.5 MHz -> VX clock fanout

Complete:

<pre>
125 MHz oscillator - U5 fan out -
q0 -> CLK_XO_125 -> ENC C72,C74 -> FPGA R8,R7 MGTREFCLK0_A -> XDC CLK_XO_125_P -> VHDL not used
q1 -> U6 C.C. in1
q2 -> CLK3_XO_125 -> ENC C7,C9 -> FPGA J8,J7 MGTREFCLK1_B -> XDC GDM missing, CDM CLK3_XO_125_P -> VHDL SFP RX reference clock (mgt_rx_ref_clk)
q3 -> not used, was CLK2_XO_125

U6 C.C (clock cleaner) -

in0 <- CLK_EXT1 (GPS 10 MHz clock)
in1 <- 125 MHz oscillator via U5
in2 <- CLK_CC_IN <- ENC C142,C144 <- FPGA AK9,AK8 <- XDC GDM missing, CDM CLK_CC_IN1 <- VHDL rx_clk_mgt
in3 <- CLK_FB

out0 -> CLK_CCA -> U12 (125 MHz)
out1 -> CLK_CCB -> VX1..6 (62.5 MHz)
out2 -> CLK_CCC -> VX7..12 (62.5 MHz)
out3 -> CLK_FB into in3

CLK_CCA -> U12 (125 MHz fan out) ->

Q0 -> not used
Q1 -> CLK_CC_OUT0 -> ENC C3,C5 -> FPGA L8,L7 MGTREFCLK0_B -> XDC GDM clk_mgtrefclk0_x0y1_p, CDM i_CLK_CC_OUT0_P -> VHDL GDM mgt_b_ref_clk QSFP RX and TX reference clocks, CDM alternate rx_clk via clock mux
Q2 -> CLK_CC_OUT1 -> ENC B3,B5 -> FPGA B10,B9 MGTREFCLK1_D -> XDC i_CLK_CC_OUT1_P -> VHDL GDM not used, CDM mgt_tx_ref_clk SFP TX clock
Q3 -> CLK_CC_OUT2 -> ENC C151,C153 -> FPGA AG8,AH8 (GC) -> XDC GDM missing, CDM i_CLK_CC_OUT2_P
Q4 -> not used
Q5 -> SMA J9/J10

CLK_TP0 <-> ENC C69,C71 <-> FPGA N8,N7 MGTREFCLK1_A (not used) -> XDC CLK_TP0_P -> VHDL not used (DS-DM SMA J11, J12)

disconnected <- OLD_CLK_CC_IN <- ENC B10,B12 <- FPGA D9,D10 MGTREFCLK0_D <- XDC o_CLK_CC_IN <- VHDL CDM o_OLD_CLK_CC_IN SFP o_mgt_rx_rec_clk

Notes:
* Enclustra ENC-B is J801, ENC-C is J900
* "(GC)" is clock-capable FPGA pin
* CLK_XO_125 (125 MHz osc) is not used in FPGA
* 62.5 MHz VX clock does not go into the FPGA
* CLK_EXT0 going to in1 of U5 cannot be used. only permitted frequency is 125 MHz (it drives the MGT reference clocks) and is this frequency is too high for LEMO cables and connectors
</pre>

= I2C bus =

* I2C_SCL is J-ENC-A pin 111 I2C_SCL is FPGA I2C_SCL_PL AB13 (IO_L1N_TOL1D_64) and I2C_SCL_PS F18 (PS_MIO10)
* I2C_SDA is J-ENC-A pin 113 I2C_SDA is FPGA I2C_SDA_PL AH13 (IO_L7N_T1L1Q_AD13N_64) and I2C_SDA_PS G18 (PS_MIO11)

* XU8 secure EEPROM ATSHA204A at 0x64, this is 0110'010X -> linux _011'0010 is 0x32. (but responds to scan and read at 0x33)
* U4 ethernet mac chip, EEPROM at 1010 A2 A1 A0 X and MAC/serial_no at 1011 A2 A1 A0 X. A0=VCC, A1=VCC, A2=GND -> linux _101'0011 and _101'1011 is 0x53 and 0x5B.
* U6 clock chip, address 1101 0 A1 A0 X. A1=VCC, A0=N/C (internal pull-up) -> linux _110'1011 is 0x6b
* U23 voltmeter at 10011 ADR1 ADR0. ADR0=GND, ADR1=VCC -> linux _100'1110 is 0x4e
* SFP, address 1010000X -> linux _101'0000 is 0x50. additional SFP data at 0x51
* QSFP0, QSFP1, QSFP2, QSFP3 (QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL)

= I2C clock builder connection =

* use Silicon Labs USB "Clock builder pro field programmer", www.silabs.com/CBProgrammer
* connect rainbow jumper cable pins:
** black - 1-GND to GND on DS-DM
** white - 3-SCLK to SCL on the DS-DM
** grey - 7-SDA_SDIO to SDA on the DS-DM
* power up the DS-DM
* plug USB programmer into Windows laptop
* on Windows, run "ClockBuilder Pro"
* it should report "Field programmer detected", press "EVB GUI"
* in EVB GUI, press "Config", set I2C address 0x6B
* press "Scan", it should find Si5394A-A-GM
* select the "Status" tab, should see real-time status of clock chip

= GDM MGT configuration =

* TX configuration:
* GDM MGT transceivers are configured as "multilane" TX and RX.
* there is 12 TXes ("lanes")
* MGT reference 125 MHz clock goes into [2:0]gtreclk00_in and [11:0]gtrefclk0_in
* one MGT is designated as "master"
* PLL of master MGT converts reference clock into common TX clock and common tx_user_clk2 which becomes tx_data_clk
* common tx_user_clk2 aka tx_data_clk goes into all TXes and clocks tx_data.
* tx_user_clk2 aka tx_data_clk is 125 MHz but not same phase as MGT reference clock.

* RX interim configuration:
* there is 12 RXes ("lanes")
* each RX produces it's own recovered RX clock
* "multilane" configuration assumes all RX recovered clocks run at the same frequency (TX on the other end are driven by common TX clock, see above), but have different phase
* one RX recovered clock is designated as "master" (rx_user_clk2 aka rx_data_clk) and a phase-matching fifo/buffer is used to bring rx_data from all 12 RXes to this common rx_data_clk
* this works because each CDM SFP TX runs on the SFP RX recovered clock which is frequency-locked with the GDM QSFP TX clock.

* RX final configuration:
* MGTs permit using the common TX clock (tx_user_clk2 aka tx_data_clk) as the common rx_data_clk (they are frequency locked through the CDM).
* this permits use of tx_data_clk as the main clock domain in the GDM and removes the need to bring rx_data into the tx_data_clk domain (actually this is done in the MGT RX phase matching fifo/buffer).

= Clock path =

NOTE: MUST REVIEW!!!

<pre>
10 MHz ext clock or GDM 125 MHz oscillator
-> GDM QSFP MGT reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz (GDM main clock domain) and TX bit clock 2.5 GHz
-> GDM QSFP optic transmitter
-> CDM SFP optic receiver
-> CDM SFP MGT, RX reference clock is CDM 125 MHz oscillator
-> MGT RX recovered clock 125 MHz (CDM main clock domain)
-> CC_CLK_IN -> CDM C.C. -> CC_CLK_OUT1 -> CDM SFP TX reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz and TX bit clock 2.5 GHz
-> (tx_data phase matching fifo from CDM main clock domain to tx_data_clk)
-> CDM SFP optic transmitter
-> GDM QSFP RX optic receiver (12x)
-> GDM QSFP MGT (RX reference clock is same as TX reference clock)
-> MGT RX recovered clock (12x recovered clocks)
-> in multi-lane configuration, one of them is the "master" recovered clock rx_data_clk
-> (rx_data phase matching fifo from rx_data_clk to GDM main clock domain)
</pre>

= CDM rx_clk mux =

when CDM SFP is not connected, there is no SFP recovered clock and a mux is used to switch between clk_cc_out0 (power up default) and rx_clk_mgt (SFP recovered clock)

== Test SFP disconnected ==

note: if I say "--cc-in1", CC seems to lock on the 10 MHz GPS external clock,
to prevent this, test sequence includes reloading the CC and the reset of MGT.

<pre>
./test_cdm_local.exe --write32 0x30 0x0 ### rx_clk mux select CC clock
./test_cdm_local.exe --load-cc
./test_cdm_local.exe --reset-mgt
./test_cdm_local.exe --cdm-clocks
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773594e (125000014) should be ~125 MHz <=== all 4 clocks wobble close to 125 MHz
0x1034 rx_clk: 0x0773594f (125000015) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x0773594f (125000015) should be ~125 MHz
0x103C tx_clk: 0x0773594f (125000015) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
./test_cdm_local.exe --write32 0x30 0x1 ### rx_clk mux select SFP recovered clock
./test_cdm_local.exe --cdm-clocks
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773598b (125000075) should be ~125 MHz
0x1034 rx_clk: 0x076d58ec (124606700) should be ~125 MHz <=== off frequency because there is no valid SFP recovered clock
0x1038 mgt_tx_ref_clk_raw: 0x0773598c (125000076) should be ~125 MHz
0x103C tx_clk: 0x0773598b (125000075) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

= Clock domains =

== GPS ==

* no GPS : GDM runs from internal 125 MHz oscillator
* external 10 MHz clock : GDM runs from external 10 MHz clock and optional 1pps signal (use VME-NIMIO32 NIM outputs)
* GPS receiver : GDM runs from GPS 10 MHz clock and GPS IRIG serial data
* LNGS GPS:
** provides 1pps and serial data over fiber from GPS receiver (master). LNGS xxx box is not used.
** serial data goes to GDM, decoded, 1pps signal extracted, goes to LEMO output
** 1pps from GDM is used to train the Rubidium clock which provides a 10 MHz clock
** 10 MHz output from Rubidium clock goes to GDM 10 MHz external clock input

== GDM ==

* AXI clock (100 MHz) - AXI registers
* 125 MHz oscillator - to clock cleaner
* 10 MHz external clock LEMO input - to clock cleaner
* FPGA 125 MHz clock CLK_CC_IN - to clock cleaner - not used (could be used for cascaded GDMs)
* 125 MHz oscillator - CLK3_XO_125 - mgt_b_ref_clk QSFP MGT reference clock (interim GDM design)
* clock cleaner output 125 MHz fanout:
** CLK_CC_OUT0 - QSFP MGT reference clock (final design)
** CLK_CC_OUT1 - not used (CDM SFP reference clock)
** CLK_CC_OUT2 - not used
* QSFP MGT TX data clock 125 MHz tx_data_clk (main clock domain)
** QSFP TX data
** QSFP RX data (in final design, rx_data_clk is same as tx_data_clk. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk == tx_data_clk)
** ds20k block
* (in interim design, rx_data_clk is disconnected from tx_data_clk. rx_data_clk is the "multilane master clock" which is one of the 12x RX recovered clocks. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk)

note: all these clocks are frequency locked to 125 MHz

== CDM ==

* AXI clock (100 MHz) - AXI registers
* 10 MHz external clock LEMO input - to clock cleaner (not used)
* 125 MHz oscillator to fanout
** to clock cleaner
** to SFP MGT RX reference clock: CLK3_XO_125 to mgt_rx_ref_clk to gtrefclk01_in
* SFP MGT RX recovered clock 125 MHz
** MGT PLL to MGT rx_user_clk2 aka rx_data_clk
** MGT rxrecclkout_out to mgt_rx_rec_clk to CLK_CC_IN to C.C.
* SFP MGT rx_user_clk2 aka rx_data_clk (250 MHz/8 bit, 125 MHz/16 bit, 62.5 MHz/32 bit data) (main clock domain)
** SFP RX data
** ds20k block
** VX TX clock PLLs
** VX RX clock PLLs
* C.C. fan out
** 62.5 MHz VX clocks (12x)
** CLK_CC_OUT0 (not used, GDM QSFP MGT reference clock)
** CLK_CC_OUT1 125 MHz to mgt_tx_ref_clk to gtrefclk00_in to SFP MGT TX reference clock
** CLK_CC_OUT2 (not used)
* SFP MGT tx_user_clk2 aka tx_data_clk
** SFP TX data
** TX data phase matching fifo from main clock domain to tx_data_clk
* VX TX clock PLLs
** 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
** VX TX data phase matching from main clock domain to VX TX clock (12 total)
** VX TX serializer
** VX TX LVDS transmitter
* VX RX clock PLLs
** 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
** VX RX LVDS receivers (12 total)
** VX RX deserializers (12 total)
** VX TX data phase matching from VX RX clock to main clock domain

== VX ==

* everything runs on the VX main 125 MHz clock
* correct phase of VX to CDM LVDS data is adjusted by scan of CDM VX RX PLL clock phase (VX to CDM link is now established)
* correct phase of CDM to VX LVDS data is adjusted by scan of CDM VX RX PLL clock phase (link is established after VX to CDM idle data pattern changes from "VX RX data bad" to "good".
* after good phases are found by scan, they are not expected to change unless cables are changed, CDM and VX modules are changed or CDM firmware is rebuilt. (rebuild of VX firmware should not affect LVDS data phase).
* if there is excessive link errors, phase scan must be repeated.

= Board test plan =

To test:
* Enclustra FPGA board
* SFP port - SFP_RS0/RS1 connected to QSFP0_SEL/1_SEL is wrong? our Finisar SFP says RS0, RS1 N/C, so probably okey.
* SW5 CLK_EXT1 NIM works. TTL needs to be tested. flipping SW5 CLK_EXT0 side from NIM to TTL makes CLK_EXT1 go LOS and OOF in the clock chip. R23 and R53 should be removed?
* BOOT_MODE 0 and 1

Partial:
* U23 3.3V current meter and thermometer. V1,V2 is current monitor, same as in application note. V3,V4 is thermometer, same in application note, except capacitor C118 is 0.1u instead of 470pF. Tested ok: Tint, VCC, V1, V2, V1-V2. Test failed: TR2 reads 50-something degC instead of same as Tint.

Done:
* LED_FP1A..D: tested ok. K.O. 15 sep 2022
* USB UART: tested ok. K.O. 15 sep 2022
* J4A, J4B, J5A, J5B LEMO inputs (NIM/TTL) EXT_IN_LV(1..4). TTL threshold 1.7V, NIM threshold -0.3V. 50 Ohm termination. TTL straight, NIM inverted.
* J6A, J6B LEMO clock inputs: 10 MHz TTL from chronobox works, 10 MHz NIM from IO32 works. 18-apr-2023
* J7A, J7B LEMO outputs EXT_OUT(1), EXT_OUT(2) (NIM/TTL) (tested 23nov2022, K.O.). TTL ???, NIM inverted.
** TTL out no 50 ohm termination: 0=0V, 1=5V, rise and fall time ~5 ns
** TTL out with 50 ohm termination: 0=0V, 1=2.5V, rise and fall time <2ns
** NIM out no 50 ohm termination: 0=+50mV, 1=-1.8V, rise and fall time ~3ns
** NIM out with 50 ohm termination: 0=0V, 1=-0.9V, rise and fall time <2ns
* ethernet MAC i2c chip (K.O. 20sep2022: can read: i2cdump 0 0x5b, nov2022 have u-boot driver)
* CDM VX ports 2x(CLK, 3 tx, 4 rx) tested using test_cdm.exe and LVDS loopback in VX firmware.
* SFP i2c tested KO 22jun2023
* QSFP i2c tested KO 22jun2023
* i2c testing complete 22jun2023
* QSFP rx,tx tested 26june2023. lane0,1,2 ok, lane3 tx not connected, laser is off. qsfp0,1,2,3 all lanes ok.
* SFP rx,tx tested 26june2023. rx and tx okey. LOS ok, mod_absent ok.

Failure:
* ethernet: does not connect to alliedtelesys switch. connects to my USB-eth adapter at 100 Mbit speed. uboot mii status reports connection speed oscillating between 1000, 100 and 10. K.O. 16-sep-2022
* SFP LOS and mod_absent are swapped (in the FPGA pin definitions?)
* QSFP slot numbering is wrong.

= Checklist for newly build boards =

== setup and boot ==
* put new board on workbench
* check - vme connector present, vme extraction handles present
* check - standoff are removed from all thru-holes
* plug Enclustra module
* check - SW1 both switches are in the "6" and "3" positions
* check - SW6 both switches are in the "PS" position
* Rev0: connect micro-usb cable from linux PC
* Rev1: connect usb-C cable from linux PC (USB C-to-C or C-to-A)
* connect ethernet from 1gige capable network switch
* connect power from lab power supply - GND, +5V and -12V nominal, +5.7V and -12.5V actual
* power up, +5V current 2.10-2.8A, -12V current 0.05A
* on linux PC, open a new terminal, run: minicom -D /dev/ttyACM0 -b 115200
* in minicom window, observe messages about Xilinx first stage boot loader, etc
* on the ethernet switch, observe network link is 1gige speed (not 10mbit, not 100mbit).
* if everything boots okey, there will be a login prompt, login as root, password root.
* alternatively, from root@daq13: ssh root@dsdm
* busybox devmem 0x80010000 # read firmware version number, i.e. 0xEDAD0A77
== test i2c ==
* check i2c: (no SFP, no QSFP plugged in!)
<pre>
root@dsdm:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@dsdm:~#
</pre>
* test clock chip:
<pre>
root@dsdm:~# ./test_cdm_local.exe --load-cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Loading CC registers...
Clock chip data block size 519
root@dsdm:~# ./test_cdm_local.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Polling CC status...
Clock chip state 1, status: LOS_IN0 OOF_IN0 IN1 IN_SEL_1
Ctrl-C
</pre>
* test RTC:
<pre>
root@dsdm:~# ./test_cdm_local.exe --rtc-read
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Read the RTC chip!
reg 0x00: 0 0 1 0 0 1 1 0 0x26
reg 0x01: 0 0 1 1 1 0 0 1 0x39
reg 0x02: 0 0 0 0 0 1 1 1 0x07
reg 0x03: 0 0 0 0 0 0 1 0 0x02
reg 0x04: 0 0 0 0 0 0 1 0 0x02
reg 0x05: 0 0 0 0 0 0 0 1 0x01
reg 0x06: 0 0 0 0 0 0 0 0 0x00
reg 0x07: 0 0 0 0 0 0 0 0 0x00
reg 0x08: 0 0 0 0 0 0 0 0 0x00
reg 0x09: 0 0 0 0 0 0 0 0 0x00
reg 0x10: 0 0 0 0 0 0 0 0 0x00
reg 0x11: 0 0 0 0 0 0 0 0 0x00
reg 0x12: 0 0 0 0 0 0 0 0 0x00
reg 0x13: 0 0 0 0 0 0 0 0 0x00
reg 0x14: 0 0 0 1 1 0 0 0 0x18
reg 0x15: 0 0 0 0 1 0 0 0 0x08
reg 0x16: 0 0 0 0 0 0 0 0 0x00
reg 0x17: 0 0 0 1 0 1 1 1 0x17
reg 0x18: 0 1 0 0 0 0 0 0 0x40
seconds: 26, minutes: 39, hours: 07, day 2, date: 00-01-02, temp 23.25C
root@dsdm:~#
</pre>

== test clocks ==

<pre>
root@dsdm:~# ./test_cdm_local.exe --reset-mgt
...
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735cb9 (125000889) should be ~125 MHz
0x1034 rx_clk: 0x07735cb9 (125000889) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735cb9 (125000889) should be ~125 MHz
0x103C tx_clk: 0x07735cb9 (125000889) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
^C
root@dsdm:~# ./test_cdm_local.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735c6a (125000810) should be ~125 MHz
0x1034 rx_clk: 0x07735c69 (125000809) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735c69 (125000809) should be ~125 MHz
0x103C tx_clk: 0x07735c6a (125000810) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
^C
root@dsdm:~#
</pre>

== test LED, LEMO ==

* test LEDs, observe all 4 LEDs turn on and off every 1 second
<pre>
root@dsdm:~# ./test_cdm_local.exe --blink-led
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
LED on
LED off
</pre>
* TBW - test LEMO outputs
<pre>
have test firmware lemo outputs are RTC chip 1pps and 48kHz
see them on the scope
</pre>
* TBW - test LEMO inputs
<pre>
./test_cdm_local.exe --monreg 6
</pre>

== test VX ==

* on the CDM, connect all split blue cables to VX modules
* set all VX modules to LVDS loopback mode (in VX settings MIDAS page)
* ssh root@cdm00
* run "./test_cdm.exe --test-vx", it should report:
<pre>
CDM registers: 0x0000000000000000 0x00000000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000eeeeeeeeeeee 0x00007700: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000080808080808 0x00000100: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000040404040404 0x00000200: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000020202020202 0x00000400: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000000000000000 0x00000800: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000808080808080 0x00001000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000404040404040 0x00002000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000202020202020 0x00004000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000000000000000 0x00008000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
all on!
all off!
</pre>
* on the VX settings MIDAS page LVDS input section, observe line status is cycling on and off.
* note: VX_RX(0) is not tested by this, corresponding VX_TX is used as the VX clock

== test SFP (CDM) ==

* connect SFP module
* ssh root@cdm00
* run "./test_cdm.exe --sfp", it should report:
<pre>
root@cdm00:~# ./test_cdm.exe --sfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x6a332b58, ds20k version 0x20241209
DS-DM firmware build 0x6a332b58, ds20k version 0x20241209
Polling SFP status...
identifier 0x03
connector 0x07
encoding 0x01
wavelength 0x0352 (850 nm)
vendor_name [FINISAR CORP. ]
vendor_pn [FTLF8526P3BNL ]
vendor_rev [A ]
vendor_sn [N3AB4LV ]
vendor_date [200319 ]
dm_type 0x68
temp 30.9 C
vcc 3.285 V
tx_bias 7.362 mA
tx_power 475.7 uW
rx_power 859.1 uW
SFP good 1, status: temp 30.9 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 859 uW
SFP good 1, status: temp 30.8 C, tx_bias 7.4 mA, tx_power 475 uW, rx_power 859 uW
SFP good 1, status: temp 30.8 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 859 uW
SFP good 1, status: temp 30.7 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 859 uW
SFP good 1, status: temp 30.8 C, tx_bias 7.4 mA, tx_power 475 uW, rx_power 858 uW
^C
root@cdm00:~#
</pre>
* NOTE1: if fiber cable is disconnected or GDM is not running, rx_power would read 0 uW
* NOTE2: better SFP modules have smaller tx_bias and larger tx_power (more bang for the buck)
* NOTE3: bad fiber cable and bad fiber connection will result in smaller rx_power

== test QSFP (GDM) ==

* connect QSFP module (any slot)
* ssh root@gdm01
* run "./test_cdm.exe --qsfp3 --qsfp", try --qsfp0, --qsfp1, --qsfp2, --qsfp3 for the 4 QSFP slots
<pre>
root@gdm01:~# ./test_cdm.exe --qsfp3 --qsfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x9e30baf9, ds20k version 0x20241209
DS-DM firmware build 0x9e30baf9, ds20k version 0x20241209
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out hi
Polling QSFP status...
identifier 0x0d
status 0x02
los 0x89
temp 36.7 C
vcc 3.302 V
rx_power 0.1 413.7 460.3 0.1 uW
tx_bias 7.8 7.9 7.9 0.0 mA
tx_power 806.0 842.9 865.2 0.1 uW
vendor_name [FINISAR CORP ]
vendor_pn [FTL410QD4C ]
vendor_rev [A ]
wavelength 850
max_temp 70 C
vendor_sn [X79AC0R ]
vendor_date [220309 ]
QSFP good 1, status: temp 36.7 C, los 0x89, tx_bias 7.8 7.9 7.9 0.0 mA, tx_power 806 843 865 0 uW, rx_power 0 414 460 0 uW
^C
</pre>
* NOTE1: 4th tx_bias and tx_power are zero because 4th channel does not transmit any data, QSFP detects this and automatically shuts down the optical transmitter

= Serial console =

* check that linux computer has correct udev rules to allow access to /dev/ttyACM devices, see https://daq00.triumf.ca/DaqWiki/index.php/SLinstall#Configure_USB_device_permissions and https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Configure_USB_device_permissions
* connect micro-USB cable to connector J-UCB, other end connect to linux computer
* observe /dev/ttyACM0 was created
* run "minicom -D /dev/ttyACM0 -b 115200"
* should have gdm-cdm login
* username root, password root

= i2c =

<pre>
ZynqMP> i2c bus
Bus 0: i2c@ff020000
ZynqMP> i2c dev 0
Setting bus to 0
ZynqMP> i2c probe
Valid chip addresses: 33 4E 53 5B 6B 77
ZynqMP> i2c md 0x5b 0x98
0098: fc c2 3d 00 00 12 1a 6e 0a 90 85 04 94 10 08 50 ..=....n.......P
</pre>

<pre>
root@gdm-cdm:~# i2cdetect 0
Warning: Can't use SMBus Quick Write command, will skip some addresses
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0.
I will probe address range 0x03-0x77.
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f
00:
10:
20:
30: -- -- -- 33 -- -- -- --
40:
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60:
70:
root@gdm-cdm:~#
root@gdm-cdm:~# i2cdump 0 0x5b
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0, address 0x5b, mode byte
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00 ???????P?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e ........??=..??n
a0: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00 ???????P?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e ........??=..??n
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@gdm-cdm:~#
</pre>

<pre>
root@gdm0:~# i2cdetect -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#
</pre>

<pre>
root@cdm0:~# i2cdetect -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: 50 51 -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@cdm0:~#
</pre>

* 0x33 - XU8 secure EEPROM (should be at 0x32)
* 0x4e - U23 current and temperature monitor
* 0x50, 0x51 - SFP
* 0x50 - QSFP, 4 QSFP modules enabled by GPIO QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL
* 0x53, 0x5b - ethernet mac eeprom
* 0x6b - U6 clock chip

== U23 ==

NOTE: Text reads double of Tint. not sure why. K.O. 21-mar-2024.

* internal temperature only

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x00 b # control register: "repeat mode, internal temperature only"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x03 # "Tint ready" and "busy", "busy is always 1 in repeat mode"
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 00 03 03 81 db 2a ce 2a 8e 00 6f 00 45 20 3f ?.????*?*?.o.E ?
10: 01 00 01 01 01 db 2a ce 2a 8e 00 6f 00 45 20 3f ?.????*?*?.o.E ?
...
readback:
reg0 - 03 - Tint ready
reg1 - 00 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB 0x81, bit 0x80 is "DV, data valid", bit 0x40 is "SS, sensor short", 0x20 is "SO, sensor open"
reg5 - Tint LSB 0xDB, Tint = 0x01DB = 475 * 0.0625 degC = 29.6 degC
reg6..F - stale data
</pre>

* Tint, V1, V2, TR2, VCC

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x18 b # control register: "repeat mode, V1, V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 7f 18 7f 7f 81 b9 aa d0 aa 8f 83 5e 83 5e a0 41 ???????????^?^?A
10: 01 18 01 01 01 b9 2a d0 2a 8f 03 5e 03 5e 20 41 ??????*?*??^?^ A
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x1b9*0.0625 = 27.5 degC
reg6 - V1 MSB 0xaa, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 LSB 0xd0, V1 = 0x2ad0*305.18/1000000 = 3.3447 V (correct)
reg8 - V2 MSB 0xaa, ditto
reg9 - V2 LSB 0x8f, V2 = 0x2a8f*305.18/1000000 = 3.3249 V (correct, smaller than V1)
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x5e. TR2 = 0x35e*0.0625 = 53.875 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x41, VCC = 2.5+0x2041*305.18/1000000 = 5.019 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.99 A (about right?)
</pre>

* Tint, V1-V2, TR2, VCC

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x19 b # control register: "repeat mode, V1-V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 7f 19 7f 7f 81 69 83 f2 83 f2 83 3a 83 3a a0 44 ?????i?????:?:?D
10: 01 19 01 01 01 69 03 f2 03 f2 03 3a 03 3a 20 44 ?????i?????:?: D
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x169*0.0625 = 22.5 degC
reg6 - V1 or V1-V2 MSB 0x83, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 or V1-V2 LSB 0xf2, V1-V2 = 0x3f2*19.42/1000000 = 0.0196142 V (correct, compare with V1 and V2 measured above)
reg8 - V2 or V1-V2 MSB
reg9 - V2 or V1-V2 LSB
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x3a. TR2 = 0x33a*0.0625 = 51.625 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x44, VCC = 2.5+0x2044*305.18/1000000 = 5.021 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.9807 A (about right?)
</pre>

== SFP ==

<pre>
root@cdm0:~# i2cdump 0 0x50
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 04 07 00 00 00 00 40 40 0c 00 01 3d 00 00 00 ???....@@?.?=...
10: 0c 02 00 1e 46 49 4e 49 53 41 52 20 43 4f 52 50 ??.?FINISAR CORP
20: 2e 20 20 20 00 00 90 65 46 54 4c 46 38 35 32 36 . ..?eFTLF8526
30: 50 33 42 4e 4c 20 20 20 41 20 20 20 03 52 00 9d P3BNL A ?R.?
40: 00 1a 00 00 4e 33 41 42 34 4c 56 20 20 20 20 20 .?..N3AB4LV
50: 20 20 20 20 32 30 30 33 31 39 20 20 68 f0 03 de 200319 h???
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@cdm0:~#
root@cdm0:~# i2cdump 0 0x51
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 5a 00 d8 00 55 00 e2 00 90 88 71 48 8c a0 75 30 Z.?.U.?.??qH??u0
10: 21 34 01 f4 1b 58 03 e8 31 2d 04 eb 1f 07 06 31 !4???X??1-?????1
20: 31 2d 00 64 27 10 00 9e 00 00 00 00 00 00 00 00 1-.d'?.?........
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
40: 00 00 00 00 3f 80 00 00 00 00 00 00 01 00 00 00 ....??......?...
50: 01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 b7 ?...?...?......?
60: 19 9f 80 c5 0e 17 12 c4 1f 99 00 00 00 00 30 00 ??????????....0.
70: 00 00 00 00 00 00 00 00 ff ff ff ff ff ff ff 01 ...............?
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@cdm0:~#
</pre>

== QSFP ==

<pre>
QSFP i2c enable lines, active low:

QSFP0_SEL - JENC-A 82 - PS-MIO40 - linux gpio 378
QSFP1_SEL - JENC-A 84 - PS-MIO41 - linux gpio 379
QSFP2_SEL - JENC-A 100 - PS-MIO44 - linux gpio 382
QSFP3_SEL - JENC-A 106 - PS-MIO43 - linux gpio 381 (notice 2 and 3 are out of order)
</pre>

<pre>
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
# echo 378 >> /sys/class/gpio/export ### SEL0 338+40
# echo 379 >> /sys/class/gpio/export ### SEL1 338+41
# echo 381 >> /sys/class/gpio/export ### SEL3 338+43
# echo 382 >> /sys/class/gpio/export ### SEL2 338+44
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) in hi
gpio-379 ( |sysfs ) in hi
gpio-381 ( |sysfs ) in hi
gpio-382 ( |sysfs ) in hi
root@gdm0:~# echo out >> /sys/class/gpio/gpio381/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio382/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio378/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio379/direction
root@gdm0:~#
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out lo
gpio-379 ( |sysfs ) out lo
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out lo
root@gdm0:~#
echo 1 >> /sys/class/gpio/gpio381/value
echo 1 >> /sys/class/gpio/gpio382/value
echo 1 >> /sys/class/gpio/gpio378/value
echo 1 >> /sys/class/gpio/gpio379/value
cat /sys/kernel/debug/gpio
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out hi
gpio-382 ( |sysfs ) out hi
root@gdm0:~#
root@gdm0:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#

NOTICE NOTHING AT ADDRESS 0x50
</pre>

<pre>
ENABLE QSFP0, OBSERVE IT IS AT ADDRESS 0x50

root@gdm0:~# echo 0 >> /sys/class/gpio/gpio378/value
root@gdm0:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: 50 -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#
root@gdm0:~# i2cdump -y 0 0x50
No size specified (using byte-data access)
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 0d 00 02 8e 00 00 01 00 00 05 55 00 05 00 00 00 ?.??..?..?U.?...
10: 00 00 00 00 00 00 1e b8 00 00 81 a0 00 00 00 00 ......??..??....
20: 00 00 12 12 00 01 00 01 00 01 0e e0 0f 20 0e e0 ..??.?.?.???? ??
30: 00 00 1f a0 1e 3b 1f 72 00 01 00 00 00 00 00 00 ..???;?r.?......
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
60: 00 00 00 00 00 00 00 00 00 00 1f 00 00 00 08 00 ..........?...?.
70: 00 00 00 00 00 00 00 ff ff ff ff ff ff ff ff 00 ................
80: 0d 00 0c 04 00 00 00 40 40 02 d5 05 67 00 00 96 ?.??...@@???g..?
90: 00 00 c8 00 46 49 4e 49 53 41 52 20 43 4f 52 50 ..?.FINISAR CORP
a0: 20 20 20 20 07 00 90 65 46 54 4c 34 31 30 51 44 ?.?eFTL410QD
b0: 34 43 20 20 20 20 20 20 41 20 42 68 07 d0 00 43 4C A Bh??.C
c0: 00 07 0f de 58 37 39 41 43 30 52 20 20 20 20 20 .???X79AC0R
d0: 20 20 20 20 32 32 30 33 30 39 20 20 3c 10 00 9e 220309 <?.?
e0: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@gdm0:~#
</pre>

== ethernet mac eeprom ==

* correct chip with 84-bit ethernet mac address

<pre>
root@cdm1:~# i2cdump 0 0x53
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
root@cdm1:~#

root@cdm1:~# i2cdump 0 0x5b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00 ????4??2?K?.?...
90: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c ..........??=?Q<
a0: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00 ????4??2?K?.?...
b0: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c ..........??=?Q<
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@cdm1:~#
</pre>

* wrong "602" chip with 64-bit IPv6 address

<pre>
root@cdm0:~# i2cdump 0 0x53
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
root@cdm0:~# i2cdump 0 0x5b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00 ???????Q?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e ........??=..??.
a0: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00 ???????Q?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e ........??=..??.
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@cdm0:~#
</pre>

= read ethernet mac address from i2c =

(this code is copied from uboot command line i2c code)

in uboot sources board/xilinx/common/board.c replace original function with this:

<pre>
// special code to read ethernet MAC address from the DS-DM-Rev0 board. K.O. Sep-2022

int zynq_board_read_rom_ethaddr(unsigned char *ethaddr)
{
struct udevice *bus;
int ret;
int busnum = 0;

ret = uclass_get_device_by_seq(UCLASS_I2C, busnum, &bus);
if (ret) {
printf("%s: No bus %d\n", __func__, busnum);
return ret;
}

int chip_addr = 0x5B;

struct udevice *dev;

ret = i2c_get_chip(bus, chip_addr, 1, &dev);
if (ret) {
printf("%s: Bus %d no chip 0x%02x\n", __func__, busnum, chip_addr);
return ret;
}

int dev_addr = 0x98;

unsigned char data[8];

ret = dm_i2c_read(dev, dev_addr, data, 8);

if (ret) {
printf("%s: Bus %d chip 0x%02x read error %d\n", __func__, busnum, chip_addr, ret);
return ret;
}

printf("%s: Bus %d chip 0x%02x addr 0x%02x read: 0x %02x %02x %02x %02x %02x %02x %02x %02x\n", __func__, busnum, chip_addr, dev_addr, data[0], data[1], data[2], data[3], data[4], data[5], data[6], data[7]);

// see Atmel-8807-SEEPROM-AT24MAC402-602-Datasheet.pdf

if (data[0] == 0) {
// eiu-48 chip
ethaddr[0] = data[2];
ethaddr[1] = data[3];
ethaddr[2] = data[4];
ethaddr[3] = data[5];
ethaddr[4] = data[6];
ethaddr[5] = data[7];
} else {
// eiu-64 chip
ethaddr[0] = data[0];
ethaddr[1] = data[1];
ethaddr[2] = data[2];
ethaddr[3] = data[5];
ethaddr[4] = data[6];
ethaddr[5] = data[7];
}

printf("%s: ethaddr %02x:%02x:%02x:%02x:%02x:%02x\n", __func__, ethaddr[0], ethaddr[1], ethaddr[2], ethaddr[3], ethaddr[4], ethaddr[5]);

return ret;
}
</pre>

also this should have worked if i2c_xxx() functions were enabled in uboot:
<pre>
i2c_set_bus_num(0);
i2c_probe(0x5b);
i2c_read(0x5b, 0x9a, ethaddr, 6);
</pre>

= read ethernet mac address from i2c (SHOULD WORK) =

from: https://stackoverflow.com/questions/43637540/is-there-linux-or-u-boot-support-to-read-a-mac-address-from-a-chip-at-startup

<pre>
#ethernet related setup
setup_eth=run readmac buildmac
#read mac address from eeprom
readmac=i2c dev 0; i2c read 50 FA.1 6 $loadaddr
#build the ethaddr variable
#not very nice, but does the job
buildmac=\
e=" "; sep=" " \
for i in 0 1 2 3 4 5 ; do\
setexpr x $loadaddr + $i\
setexpr.b b *$x\
e="$e$sep$b"\
sep=":"\
done &&\
setenv ethaddr $e
</pre>

= read ethernet mac address from i2c (DOES NOT WORK) =

this method does not work: inside board/xilinx/common/board.c:int zynq_board_read_rom_ethaddr(unsigned char *ethaddr), uclass_get_device_by_ofnode(UCLASS_I2C_EEPROM, eeprom, &dev) returns ENODEV and read from i2c does not happen. K.O. Sep-2022

Read:
* https://support.xilinx.com/s/question/0D52E00006hpKlsSAE/uboot-and-ethernet-mac-address-from-eeprom?language=en_US (last answer)
* https://github.com/Xilinx/u-boot-xlnx/blob/master/board/xilinx/common/board.c (zynq_board_read_rom_ethaddr())

Note:
* 0x5B is the i2c chip address
* 0x9A is the data offset inside the chip, see datasheet or i2c read dump above.

Edit:
* emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/u-boot/files/platform-top.h
<pre>
#include <configs/xilinx_zynqmp.h>
#include <configs/platform-auto.h>
//#define CONFIG_I2C_EEPROM
//#define CONFIG_SYS_I2C_EEPROM_ADDR 0x5b
//#define CONFIG_SYS_I2C_EEPROM_ADDR_OVERFLOW 0x0
#define CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET 0x9A
#error HERE!
</pre>
* emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi
<pre>
/include/ "system-conf.dtsi"
/ {
chosen {
xlnx,eeprom = &eeprom;
};
};

&i2c0 {
eeprom: eeprom@5b { /* u88 */
compatible = "atmel,24mac402";
reg = <0x5b>;
};
};
</pre>
* components/yocto/workspace/sources/u-boot-xlnx/configs/xilinx_zynqmp_virt_defconfig
<pre>
CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET=0x9A
</pre>

= RTC chip =

* DS3231 RTC chip
* FPGA connections:
<pre>
I2C SCL <- J-ENC A85 <- FPGA E17 <- XDC TP_S <- VHDL RTC_I2C_SCL output (10k pull-up to 3.3V)
I2C SDA <-> J-ENC A87 <-> FPGA D17 <-> XDC TP_S <-> VHDL RTC_I2C_SDA bidir (10k pull-up to 3.3V)
RTC_1Hz -> J-ENC C160 -> FPGA AH12 -> XDC "slow_io" -> VHDL RTC_1PPS input (10k pull-up to 1.2V)
RTC_32k -> J-ENC B129 -> FPGA AE3 -> XDC "free pin" -> VHDL RTC_32KHZ input (10k pull-up to 1.8V)
</pre>
* test notes
<pre>
FPGA registers:

rtc_i2c_clock_ie <= not register_data_in_block_0(14)(0);
rtc_i2c_clock_out <= register_data_in_block_0(14)(1);
rtc_i2c_data_ie <= not register_data_in_block_0(14)(4);
rtc_i2c_data_out <= register_data_in_block_0(14)(5);

ext_out(1) <= rtc_1pps_in;
ext_out(2) <= rtc_32khz_in;

register_data_out_block_0(13) <= x"000000" & rtc_i2c_data_ie & rtc_i2c_data_out & rtc_i2c_clock_ie & rtc_i2c_clock_out & rtc_1pps_in & rtc_32khz_in & rtc_i2c_data_in & rtc_i2c_clock_in;

After power us on the RTC chip:
i2c clock 3.3V
i2c data 3.3V
32kHz 0.778V see clock running with amplitude 0->1.8V (FPGA pullup) period about 30 usec (~33 kHz)
1pps 1.2V (FPGA pullup)

./test1.exe --read32 0x34
dsdm_read32[0x00000034] is 0x000000ab (171) -> 1010'1011 -> data_ie, clock_ie, 1pps, data_in, clock_in
./test1.exe --read32 0x38
dsdm_read32[0x00000038] is 0x00000000 (0)

./test1.exe --write32 0x38 0x01 -> clock oe
./test1.exe --read32 0x34
Ddsdm_read32[0x00000034] is 0x0000008a (138) -> 1000'1010 -> data_ie, 1pps, data_in, clock 0V, data 3.3V

./test1.exe --write32 0x38 0x10 -> data_oe
./test1.exe --read32 0x34
dsdm_read32[0x00000034] is 0x00000029 (41) -> 0010'1001 -> clock_ie, 1pps, clock_in, clock 3.3V, data 0V

root@dsdm:~# ./test_cdm_local.exe --rtc-read
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Read the RTC chip!
reg 0x00: 0 1 0 1 0 1 1 1 0x57
reg 0x01: 0 0 1 0 0 0 1 0 0x22
reg 0x02: 0 0 0 0 0 0 0 0 0x00
reg 0x03: 0 0 0 0 0 0 0 0 0x00
reg 0x04: 0 0 0 0 0 0 0 0 0x00
reg 0x05: 0 0 0 0 0 0 0 0 0x00
reg 0x06: 0 0 0 0 0 0 0 0 0x00
reg 0x07: 0 0 0 0 0 0 0 0 0x00
reg 0x08: 0 0 0 0 0 0 0 0 0x00
reg 0x09: 0 0 0 0 0 0 0 0 0x00
reg 0x10: 0 0 0 0 0 0 0 0 0x00
reg 0x11: 0 0 0 0 0 0 0 0 0x00
reg 0x12: 0 0 0 0 0 0 0 0 0x00
reg 0x13: 0 0 0 0 0 0 0 0 0x00
reg 0x14: 0 0 0 1 1 0 0 0 0x18
reg 0x15: 0 0 0 0 1 0 0 0 0x08
reg 0x16: 0 0 0 0 0 0 0 0 0x00
reg 0x17: 0 0 0 1 0 1 1 1 0x17
reg 0x18: 1 1 0 0 0 0 0 0 0xc0
seconds: 57, minutes: 22, hours: 00, day 0, date: 00-00-00, temp 23.75C
root@dsdm:~#
</pre>

= clock chip configuration =

file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt:

* VCO is 14 GHz
* Tvco is 71.43 ps
* N0 divider is 14, frequency is 1000 MHz
* out0 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz
* out1 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
* out2 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
* out3 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz is the feedback for zero delay

= clock chip monitoring =

from si5395-94-92-family.pdf:
* reg 0x1: page select, set to 0 or set to 5 to read 0x53F
* reg 0x2: 0x94
* reg 0x3: 0x53 -> device is a si5394
* reg 0xC: LOSXAXB
* reg 0xD: LOS and OOF for the 4 clock inputs
* reg 0xE: LOL and HOLD
* reg 0xF: CAL_PLL
* reg 0x11: sticky bits for reg 0xC
* reg 0x12: sticky bits for reg 0xD
* reg 0x13: sticky bits for reg 0xE
* reg 0x14: sticky bits for reg 0xF
* reg 0x1C: device reset
* reg 0x1E: low power, hard reset, SYNC
* reg 0x507: currently selected input clock
* reg 0x52A: input clock select
* reg 0x535: FORCE_HOLD
* reg 0x53F: HOLD_HIST_VALID and FASTLOCK_STATUS

= Install Xilinx tools =

* install Vivado 2020.2
<pre>
login at https://www.xilinx.com/myprofile.html
go to "Downloads"
go to archive,
find 2020.2
download Xilinx_Unified_2020.2_1118_1232_Lin64.bin
sh ./Xilinx_Unified_2020.2_1118_1232_Lin64.bin
banner window should open with spinner "downloading installation data"
"a newer version is available" -> say "continue"
next
"select install type" window:
provide email and password,
select "download image"
select directory /home/olchansk/Xilinx/Downloads/2020.2\
select "linux" and "full image"
next
download summary: space required 38.52 Gbytes
download
installation progress
downloading spinner, 16 M/s 47 minutes...
"download image has been created successfully". Ok.
check contents of /home/olchansk/Xilinx/Downloads/2020.2
ls -l /home/olchansk/Xilinx/Downloads/2020.2
total 67
drwxr-xr-x 2 olchansk users 9 Sep 1 16:22 bin
drwxr-xr-x 3 olchansk users 15 Sep 1 16:23 data
drwxr-xr-x 4 olchansk users 4 Sep 1 16:22 lib
drwxr-xr-x 2 olchansk users 644 Sep 1 16:22 payload
drwxr-xr-x 2 olchansk users 7 Sep 1 16:22 scripts
drwxr-xr-x 4 olchansk users 4 Sep 1 16:22 tps
-rwxr-xr-x 1 olchansk users 3256 Nov 18 2020 xsetup
daq13:2020.2$
./xsetup
spinned loading installation data
xilinx design tools 2022.1 now available -> say continue
"welcome" -> next
"select product" -> vivado -> next -> vivado hl system edition -> next
select devices: only zynq ultrascale+ mpsoc -> next
select destination: /opt/Xilinx (as root, mkdir /opt/Xilinx, chmod olchansk.users /opt/Xilinx)
install ...
complete
move /home/olchansk/Xilinx/Downloads/2020.2 to /daq/daqstore/olchansk/Xilinx/Downloads/
</pre>
* install petalinux 2020.2
<pre>
./xsetup
"a newer version is available" -> say "continue"
next
"select product to install" -> select Petalinux (Linux only) -> next
"select destination directory" -> select "/opt/Xilinx" (disk space required 2.64 GB) -> next
"summary" -> install ...
error about missing /tmp/tmp-something files
"installation completed successfully" (hard to dismiss, "ok" button is partially cut-off)
done?
I think it failed, /opt/Xilinx/PetaLinux/2020.2/bin is empty except for petalinux-v2020.2-final-installer.run
try to run it by hand, same error about /tmp/tmp-something files. strange...
notice it complains about "truncate", which truncate finds ~/bin/truncate, get rid of it,
try again
now complains about missing texinfo and zlib1g:i386
apt install texinfo -> ok
apt install zlib1g:i386 -> installs bunch of gcc stuff -> ok
try again
reports "already installed" -> delete /opt/Xilinx/.xinstall/PetaLinux_2020.2/, delete entries in ~/.Xilinx/registry/installedSW.xml
try again
success
</pre>
* install vivado 2022.1 and petalinux 2022.1 - everything is pretty much the same

= Petalinux =

* cd PetaLinux_GDM_CDM
* petalinux-config
* enable i2c MAC address and DHCP
<pre>
git clone https://bitbucket.org:/team-ds-dm/ds-dm-u-boot-xlnx.git
cd ds-dm-u-boot-xlnx
git checkout ds-dm-u-boot-xlnx

linux-components ->
uboot -> ext-local-src
external u-boot local source -> ds-dm-u-boot-xlnx (path to the customized uboot git repository)
</pre>
* enable DHCP
<pre>
Subsystem AUTO Hardware Settings -> Ethernet Settings
randomize MAC address -> NO
ethernet mac address -> leave empty
obtain ip address automatically -> YES
</pre>
* set hostname and product names
<pre>
Firmware Version Configuration ->
Host name -> "ds-dm"
Product name -> "Petalinux_GDM_CDM"
</pre>
* configure linux kernel
<pre>
petalinux-config -c kernel
</pre>
* enable NFS-Root
<pre>
petalinux-config
Image Packaging Configuration > Root File System Type -> set to NFS
Location of NFS root directory set to "/nfsroot"

petalinux-config -c kernel
Networking support > IP: kernel level configuration
enable DHCP, BOOTP, RARP
File systems > Network file systems > Root file systems on NFS
</pre>
* manually fix linux kernel command line:
<pre>
grep nfsroot PetaLinux_GDM_CDM/project-spec/configs/config
edit CONFIG_SUBSYSTEM_BOOTARGS_GENERATED to read
earlycon console=ttyPS0,115200 clk_ignore_unused panic=60 root=/dev/nfs nfsroot=/nfsroot/%s ip=dhcp rw
</pre>
* check configuration in
** PetaLinux_GDM_CDM/project-spec/configs/config
** PetaLinux_GDM_CDM/project-spec/configs/rootfs_config
** PetaLinux_GDM_CDM/components/plnx_workspace/device-tree/device-tree/system-conf.dtsi

= JTAG server =

localhost:3121

= ds20k block =

<pre>
module ds20k
(
// CLOCK INPUTs
input wire clk,
input wire reset, // pulse for power-up reset
input wire si5394_loln, // clock cleaner PLL is locked to selected input clock

// REGISTER_DATA

input wire [255:0] [31:0] register_data_in,
output reg [255:0] [31:0] register_data_out,
input wire register_we,
input wire register_re,

// GDM QSFP FIBER LINKS
input wire qsfp_tx_data_ready,
output reg [11:0] [15:0] qsfp_tx_data,
output reg [11:0] [1:0] qsfp_tx_data_is_k,
input wire [11:0] [15:0] qsfp_rx_data_error, // not sure what this is.
input wire [11:0] [15:0] qsfp_rx_data,
input wire [11:0] [1:0] qsfp_rx_data_is_k,

// CDM SFP FIBER LINKS
input wire sfp_tx_data_ready,
output reg [15:0] sfp_tx_data,
output reg [1:0] sfp_tx_data_is_k,
input wire sfp_rx_data_error, // single bit indicating that RX link is up and data is good.
input wire [15:0] sfp_rx_data,
input wire [1:0] sfp_rx_data_is_k,

// VX_RXs
input wire [3:0] vx1_rx_async,
input wire [3:0] vx2_rx_async,
input wire [3:0] vx3_rx_async,
input wire [3:0] vx4_rx_async,
input wire [3:0] vx5_rx_async,
input wire [3:0] vx6_rx_async,
input wire [3:0] vx7_rx_async,
input wire [3:0] vx8_rx_async,
input wire [3:0] vx9_rx_async,
input wire [3:0] vx10_rx_async,
input wire [3:0] vx11_rx_async,
input wire [3:0] vx12_rx_async,

// VX_TXs
output reg [2:0] vx1_tx_out,
output reg [2:0] vx2_tx_out,
output reg [2:0] vx3_tx_out,
output reg [2:0] vx4_tx_out,
output reg [2:0] vx5_tx_out,
output reg [2:0] vx6_tx_out,
output reg [2:0] vx7_tx_out,
output reg [2:0] vx8_tx_out,
output reg [2:0] vx9_tx_out,
output reg [2:0] vx10_tx_out,
output reg [2:0] vx11_tx_out,
output reg [2:0] vx12_tx_out,

// VX serial comunications
input wire [11:0] vx_rx_ser_in,
output reg [11:0] vx_tx_ser_out,

// LEMO INPUTs
input wire [4:1] ext_in_async, // direct connection to LEMO connectors, not clocked

// LEMO OUTPUTs
output reg [2:1] ext_out, // direct connection to LEMO connectors, not clocked

// FRONT PANEL LEDs
output reg [3:0] fp_led_out, // direct connection to LEMO connectors, not clocked
);
</pre>

= world view =

Main components:
* clock distribution - to ensure all digitizers run synchronously and waveform timestamp are easy to assemble intophysics events.
* global trigger and run control - all digitizers are triggered at the same time to record calibration events, pulser events and the run startup synchronization sequence.
* busy distribution - when some digitizers go busy and stop accepting triggers, physics events may become incomplete, this must be recorded and managed.
* hitmap distribution - to create an online picture of the detector for monitoring and for triggering.

== global clock distribution ==

Clock distribution ensures all digitizers run synchronously (on the same clock) and waveform timestamps are easy to assemble into physics events. The clock source is synchronized with GPS to ensure a metrologically validated clock frequency (tied to the SI definition of 1 second) and date stamp (via GPS IRIG-B date stamp). Individual digitizer waveform timestamp are reset at the begin of run to establish a valid time zero (run start time, date stamped with the GPS IRIG-B date stamp).

Base clock is 125 MHz (8 ns). Individual digitizers are synchronized at initialization (power-up) time. Because of incontrollable delays in the electronics (i.e. variations between individual PLL chips in each digitizer), waveform timestamps between individual digitizers may be off by 1-2 clocks (8-16 ns). After the system is running and is stable (thermally and electrically), timestamp waveform variation and jitter are less than 1 ns.

This is validated by feeding a common calibration pulser signal into analog inputs of each digitizer (at 100 Hz) and measuring (fitting) the pulse time in the waveform. With sufficient statistics, differences between waveform timestamp are measured with picosecond precision.

CDM PLL allows fine phase adjustment of the 62.5 MHz VX2746 clock in groups of 6 clocks (2 groups per CDM) with precision 71.43 ps.

Due to FPGA technology limitations (serial link hard IPs, PLLs, etc) the pattern of 1-2 clock delays between individual digitizers may change/shift from power up to power up. Significant additional work would be required to ensure this pattern is repeatable and stable after every power up and initialization. As mentioned before, after power up and initialization, variation between waveform timestamp is sub-nanosecond as measured by the analog calibration pulser.

Clock chain:
* GPS receiver (1pps clock reference and IRIG-B date stamp)
* LNGS GPS clock distribution (fiber link to converter box at the experiment)
* GPS 1pps to PRS-10 Rb atomic clock (10 MHz clock output)
* 10 MHz clock from PRS-10 to GDM PLL (extra low jitter oscillator, 125 MHz clock output)
* 125 MHz clock to GDM fiber link transmitters (2.5 Gbps line rate)
* CDM fiber link receiver and clock recovery (125 MHz clock)
* 125 MHz recovered clock to CDM PLL (extra low jitter oscillator, 62.5 MHz clock output)
* 62.5 MHz CDM clock to VX2745 PLL via LVDS line
* VX2745 PLL (125 MHz clock output)
* 125 MHz clock to digitizer chips, clock counter and waveform timestamp.

This is the normal mode of operation, with global clock synchronized with the GPS clock (metrologically referenced to SI 1 Hz definition).

If GPS signal is not available, degraded modes of operation are available (in order of worsening performance)
* PRS-10 Rb atomic clock synchronized with a battery-powered time-of-day chip
* PRS-10 Rb atomic clock synchronized with Internet time (NTP) assuming Internet network connection is up
* PRS-10 Rb clock standalone
* GDM internal oscillator

All degraded modes provide high precision low jitter 125 MHz system clock, but long term stability (seconds per day) will vary. (TBM)

In the absence of GPS IRIG-B date stamp, the battery-powered time-of-day chip will be used (TBI) with fallback to Internet time (NTP). Date stamp long term stability is expected to be better than 1 second per day (TBI, TBM).

Note:
* TBM: to be measured
* TBI: to be implemented

Current status (April 2025):
* full clock chain from GDM PLL to VX2745 digitizers is implemented and tested
* waveform timestamp variations are measured (see report from Marek)
* GDM external clock is validated (using VME-GRIFFIN-CDM high quality clock source module)
* PRS-10 10 MHz and 1pps connection to GDM is implemented and tested
* GPS receiver 1pps connection to PRS-10 is implemented and tested
* GPS receiver IRIG-B date stamp connection to GDM tested using a software decoder. Hardware decoder and connection to time slice marker, TBI.
* LNGS date stamp decoder and connection to time slice marker, TBI
* LNGS 1pps connection to PRS-10, not available
* operate vertical slize system from GPS, not available (GPS is in the det lab, vertical slice is in the daq lab).

== global trigger distribution ==

Global trigger distribution provides a synchronous waveform trigger to all VX2745, all CDM and all VX2745 are provided wiht the same data and are all triggered at the same time (without accounting for the 1-2 clock offset described in the clock section).

Global trigger is used for run control: the first trigger of a run resets the waveform timestamp counter, the CDM timestamp counter and the GDM timestamp counter. At the begin of run, GDM generates a sequence of triggers following an easy to identify pattern. If a VX2745 misses a trigger (for any reason), online and offline software can easily identify the tru run start time using this pattern.

After a run has started, global trigger can be generated by a programmable pulser, by an external LEMO signal (i.e. external calibration signal, laser Q-switch, etc) and by the VX2745 trigger hitmap.

Trigger sources (TRG):
* TRG pulser
* LEMO inputs
* hitmap trigger

Time slice marker sources (TSM):
* TSM pulser

Trigger chain, K-code and LVDS:
* TRG/TSM source
* TRG/TSM K-code from GDM to CDM
* CDM K-code to LVDS line 12 and 13 lines (TRG and TSM)

Latency from GDM LEMO input to CDM LVDS out: TBM

Trigger chain, packet trigger:
* GDM TRG/TSM source to TRG/TSM packet generator
* or GDM hitmap group trigger packet generator
* TRG/TSM packet from GDM to CDM
* TRG/TSM packet from CDM to VX2745 via LVDS line, 62.5 MHz, 8b10b encoding.

Latency from GDM LEMO input to VX2745 internal trigger: TBM

Notes:
TBM - to be measured
TBI - to be implemented

Current status (Jan 2026):
* LVDS TRG and TSM distribution is done and tested
* packet TRG, TRIG48 and TSM distribution from GDM to CDM is done and tested
* packet TRG, TRIG48 and TSM from CDM to VX2745 is done and tested
* packet TRIG48 processing in the VX is done and tested

== global busy distribution ==

If any VX2745 goes busy (for any reason) and stops accepting triggers, physics data may become incomplete. This situation needs to be recorded and managed.

Busy chain:
* VX2745 firmware busy logic (CAEN and Yair) generates the VX busy signal on LVDS line 9 (CDM VX_RX_2)
* CDM monitors VX busy
* CDM records busy transitions
* CDM computes the "CDM busy" as grand-OR of all VX busy
* CDM periodically transmits CDM busy status the GDM using special K-codes (NB: confirm transmission period)
* GDM receives CDM busy updates and maintains "CDM busy" for each CDM
* GDM records busy transitions
* GDM computes "GDM busy" as grand-OR of all CDM busy
* effectively, GDM busy is a grand-OR of all VX busy (with max delay, TBM)

In the pilot implementation, any VX busy stops the experiment by sending a veto signal to all VX2745 which stops them from accepting triggers.

Veto chain:
* GDM computes "GDM veto" as GDM busy
* GDM periodically transmits GDM veto status to all CDMs using special K-codes (NB: confirm transmission period)
* CDM receive GDM veto updates and maintain the "CDM veto" signal
* CDM computes "VX veto" as CDM veto
* CDM transmits VX veto to all VX2745 on LVDS pair 13
* VX2745 firmware (Yair) has VX veto block all physics triggers (NB: except for the time slice marker?)

Total delay from VX busy to VX veto (on the LVDS lines), TBM.

Notes:
* TBM - to be measured
* TBI - to be implemented

Current status (Jan 2026):
* full busy chain from VX to CDM to GDM is implemented, tested
* veto chain from GDM to CDM is implemented, tested
* veto chain from CDM to LVDS pair 13 is implemented, tested
* LVDS pair 13 veto inside the VX is implemented, tested
* TRG switched from LVDS pair 12 to TRG packet, testing completed
* TSM switched from LVDS pair 13 to TSM packet, testing completed
* fine tuning of busy chain, VX buffer levels, etc, test in progress

== vx hitmap and hitmap trigger ==

To monitor the system in real time and to generate group triggers, we use the VX hitmaps.

Each VX2745 has 64 channels, each channel has a trigger logic block (digital discriminator) that generates a waveform self-trigger and sends a hitmap bit to the CDM (64-bits per VX2745) as a data packet. CDM passes hitmap data packets to the GDM and also records non-empty hitmap packets. CDM MIDAS frontend passes non-empty hitmap packets to MIDAS for online monitoring and visualization. GDM collects all the hitmaps and generates a global trigger. Alternatively, it generates a group trigger TRIG48 with a 48-bit bitmap indicating which VX2745 should trigger a waveform.

Pilot implementation computes the hitmap trigger as a grand-OR of all hit maps.

Hitmap chain:
* VX2745 analog inputs to digitizers to digital trigger logic block, 1 bit per channel (CAEN, Yair)
* 64-bits of hitmap (1 bit/channel) are collected and periodically transmitted to the CDM (Sam De Jong) (NB: confirm transmission period: see CDM page table "VX hit map" column "period", 0xD1 clocks is 1672 ns, should be reduced to 1500 ns, TBI)
* VX hitmap packets are transmitted to the CDM on the LVDS line 10 (CDM VX_RX_1) at 62.5 MHz, 8b10b encoded.
* CDM receives hitmap packets (12 input data streams, 12-to-1 packet mux)
* CDM sends all hitmap packets to the GDM at 125 MHz, 16b20b encoded.
* non-empty hitmap packets are written to the CDM-to-CPU data FIFO (and via MIDAS frontend to MIDAS SYSTEM event buffer)
* grand-OR of VX hitmap packets (12*64 bits) can generate a CDM-local VX trigger
* GDM receives hitmap packets, computes "GDM hitmap trigger"
* if enabled, GDM hitmap trigger causes GDM trigger, which is processed as described in the global trigger section

Delay, latency and jitter from analog pulse on the VX2745 analog input to VX trigger on LVDS line 12, TBM.

TRIG48 VX group trigger:
* GDM receives hitmap packets, computes the 48-bit bitmap of which VX2745 modules should trigger
* how to specify this computation is not clear. a generic 12*64 MLU is impossible, insufficient RAM on the FPGA
* 48-bit bitmap is sent to all CDMs as a TRIG48 data packet
* CDM receives TRIG48 data packets from GDM and retransmits them to the VX2745 (all CDMs and all VXes see the same packets at the same time).
* VX2745 receives the 48-bit trigger bitmap, compares it with it's own 48-bit bitmap if any common bits are set, triggers waveform readout of all channels (there will be no more than 48 VX2745 in the experiment)

Pilot implementation of TRIG48 triggers on the CDM grand-or and generates a TRIG48 packet that contains a 48-bit bitmap stored in an AXI register.

Example uses of TRIG48:
* to trigger individual VX2745 modules: set the VX 48-bit map to the module number (vx05-0x1, vx06-0x2, vx07-0x4, etc), generate TRIG48 packet with a bitmap of VX modules that should trigger: 0xFFFF'FFFF'FFFF will trigger all modules, 0x1 will trigger only vx05, 0x2 will trigger only vx06, etc. (there is no more than 48 modules).
* to trigger groups of VX2745 modules: set the bitmap of group 1 modules to 0x1, set the bitmap of group 2 modules to 0x2, generate TRIG48 packet with bitmap 0x0001 to trigger only group 1 modules, 0x0002 to trigger only group 2 modules, 0x3 to trigger both groups.

Notes:
* TBM - to be measured
* TBI - to be implemented

Current status (Jan 2026):
* full chain (VX analog input to hitmap generator to CDM to GDM) is implemented, testing in progress
* GDM grand-or trigger is implemented
* VX hitmap packet generator, testing in progress (Yair firmware)
* CDM hitmap processing, implemented, tested
* GDM hitmap processing, implemented, tested
* pilot TRIG48 trigger generator in GDM, CDM, tested
* TRIG48 trigger in VX, implemented, tested
* MIDAS frontend readout of hitmap data from FPGA FIFO: implemented, testing in progress (FIFO code: Ian, done; integration and C++ code: KO)
* online monitoring and visualization in MIDAS: TBI

== firmware map ==

Note:
* red lines: clocks
* green lines: AXI/Avalon packet streams
* blue lines: serial data

[[File:GDM-CDM-VX.drawio.svg]]

== description ==

same thing, in words:

<pre>
detector
digitizer, 125 MHz
digital filter
digital discriminator
hit map, 64 bits at 125 MHz (could be 250 MHz, filter and discriminator clock)
packetizer, 64 bits -> id, timestamp, 8x 8-bit words, eop
8/10 serializer, 12.5 MHz parallel in, 125 MHz serial out
lvds line to CDM vx_rx
BBB: also VX busy to lvds line to CDM vx_rx, do not want to depend on serial comm for vx busy, do not want to inject async data into the hit map packet stream
--- CDM
lvds line capture in IBUF register, vxN_rx clock, 125 MHz clock, from PLL with adjustable phase, scan phase to find sweet spot
vx rx 10/8 deserializer, vxN_rx clock, stobes out parallel data every 10 clocks at 12.5 MHz
vx rx phase transfer from vxN_rx clock to main CDM clock, have 10 clocks for transfer to happen
from here everything is on the CDM main clock
vx rx packet adapter, 12.5 MHz 8-bit packets to 125 MHz 16-bit packets, internal FIFO to avoid data overrun
sfp tx mux, all VX packet streams into one CDM sfp tx packet stream
BBB: capture VX RX busy to CDM main clock (IBUF register), grand-or becomes vx_rx_busy
sfp tx packetizer (data,eop,vx_rx_busy -> data,k)
sfp tx 16/20 serializer, 16 bit at 125 MHz in, 2500 MHz serial out (20 bits at 125 MHz)
sfp tx fiber link to GDM, 2 Gigabits/sec
--- GDM
qsfp rx deserializer, 16 bit at 125 MHz output on main GDM clock
qsfp rx depacketizer (data,k -> packet data,eop; qsfp_rx_busy)
qsfp rx demux, hit map packets routed to GDM trigger logic block
GDM trigger logic block looks at hit map, generates yes/no trigger decision, encodes it as a trigger packet
TSM generator encodes GPS time data as a TSM packet
qsfp tx mux - trigger packets, tsm packets, etc to qsfp tx packet stream
qsfp tx packetizer (data,eop;trg,tsm,bsy -> data,k)
NB: the same tx data is sent to all 12 qsfp tx ports, to make sure we do not accidentally desync the CDMs.
qsfp tx 16/20 serializer, 16-bit at 125 MHz to 20-bit at 250 MHz to 2500 MHz serial out
qsfp tx fiber link to CDM, 2 Gigabits/sec
--- CDM
sfp rx 20/16 deserializer, 16-bit at 125 MHz
sfp rx depacketizer, (data,k -> data,eop;trg_in,tsm_in,bsy_in)
sfp rx demux (in reality, noop, all packets go to same place, vx tx)
vx tx mux (packets from GDM, packets with simulated waveforms from CDM midas frontend via AXI FIFO)
vx tx packet adapter 16-bit at 125 MHz to 8-bit at 12.5 MHz strobed every 10 clocks. NB: most important, this 10-clock strobe runs in sync between all CDMs!
vx tx phase transfer from main CDM clock to vxN_tx clock, have 10 clocks for transfer to happen
from here we run on the vxN_tx clock
vx txN serializer, 8-bit at 12.5 MHz to 125 MHz serial
vx_txN OBUF register, vxN_rx clock, 125 MHz clock, from PLL with adjustable phase, scan phase to find sweet spot
lvds line to vx
BBB: bsy_in from GDM is converted from pulse to level, goes out lvds line to vx, sync to vxN_tx clock
TTT: trg_in from GDM is a pulse, does out lvds line to vx, sync to vxN_tx clock.
NB: tsm is always a packet, bsy is always a signal (no packet), trg can be a packet or signal.
--- VX
lvds data captured by 125 MHz ADC sampling clock (CAEN base firmware logic)
lvds data connected to Yair's block
10/8 deserializer
depacketizer (data,k -> data,eop) to avoid accidental desync, we do not send any no trg, no tsm, no bsy K-codes.
demux
trigger packets go their way (4x 64 bit words of data go to event header: timestamps, hitmap data)
tsm packets go their way (4x 64 bit words of data go to tsm event header: timestamps, GPS time data)
BBB: bsy from lvds line stops waveform acquisition
TTT: trg from lvds line cause waveform acquisition, same as trg packet, but has no timestamp and other data attached to it.
</pre>

== CDM-VX connections ==

Note:
* rounded boxes are VX registers set from MIDAS

[[File:CDM-VX.drawio.svg]]

= Firmware registers =

== Block 0 ==

<pre>
0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
</pre>

== Block 0 GDM ==

* 0x0000+4*i - GDM AXI clock registers

<pre>
register_data_out_block_0(1) <= x"204D4447"; --GDM<space>
register_data_out_block_0(2) <= x"00000000"; --blank
register_data_out_block_0(3) <= x"00000000"; --blank

--Firmware Version
register_data_out_block_0(4) <= x"abcd0123"; --Firmware version name register
register_data_out_block_0(5) <= x"00000001"; --Firmware version number register
register_data_out_block_0(6) <= x"abcd0123"; --Firmware version register
register_data_out_block_0(7) <= x"00000001"; --Firmware version register

--Git Commit/Tag/Compile Date
register_data_out_block_0(8) <= x"0123abcd"; --Firmware Git info (0-3)
register_data_out_block_0(9) <= x"00000001"; --Firmware Git info (1)
register_data_out_block_0(10) <= x"00000001"; --Firmware Git info (2)
register_data_out_block_0(11) <= x"00000001"; --Firmware Git info (3)

--Simple loopback register for testing
register_data_out_block_0(12) <= register_data_in_block_0(12);

--Hardware registers (HW_ID_xDM, etc)
--register_data_out_block_0(13) <= x"0000000" & "000" & hardware_id;

--unspecified
set_reg_block_0_gen : for i in 14 to nregisters_block_0-1 generate
register_data_out_block_0(i) <= register_data_in_block_0(i);
end generate set_reg_block_0_gen;

register_data_out_block_0(13) <= x"000000" & rtc_i2c_data_ie & rtc_i2c_data_out & rtc_i2c_clock_ie & rtc_i2c_clock_out & rtc_1pps_in & rtc_32khz_in & rtc_i2c_data_in & rtc_i2c_clock_in;

rtc_i2c_clock_ie <= not register_data_in_block_0(14)(0);
rtc_i2c_clock_out <= register_data_in_block_0(14)(1);
rtc_i2c_data_ie <= not register_data_in_block_0(14)(4);
rtc_i2c_data_out <= register_data_in_block_0(14)(5);
</pre>

== Block 0 CDM ==

* 0x0000+4*i - CDM AXI clock registers:

<pre>
USR_ACCESS_7series_inst: USR_ACCESSE2
port map(-- CFGCLK => CFGCLK,
DATA => register_data_out_block_0(0)); -- where timestamp is contained
-- DATAVALID => DATAVALID);

--static values
register_data_out_block_0(1) <= x"204D4443"; --CDM<space>
register_data_out_block_0(2) <= x"00000000"; --blank
register_data_out_block_0(3) <= x"00000000"; --blank

--Firmware Version
register_data_out_block_0(4) <= x"abcd0123"; --Firmware version name register
register_data_out_block_0(5) <= x"00000001"; --Firmware version number register
register_data_out_block_0(6) <= x"abcd0123"; --Firmware version name register
register_data_out_block_0(7) <= x"00000001"; --Firmware version number register

--Git Commit/Tag/Compile Date
register_data_out_block_0(8) <= x"0123abcd"; --Firmware Git info (0-3)
register_data_out_block_0(9) <= x"00000001"; --Firmware Git info (1)
register_data_out_block_0(10) <= x"00000001"; --Firmware Git info (2)
register_data_out_block_0(11) <= x"00000001"; --Firmware Git info (3)

--Simple loopback register for testing
register_data_out_block_0(12) <= register_data_in_block_0(12);

--Hardware registers (HW_ID_xDM, etc)
--register_data_out_block_0(13) <= x"0000000" & "000" & hardware_id;

--unspecified
set_reg_block_0_gen : for i in 14 to nregisters_block_0-1 generate
register_data_out_block_0(i) <= register_data_in_block_0(i);
end generate set_reg_block_0_gen;

rx_clk_sel <= register_data_in_block_0(12)(0); -- rx_clk select 0:clk_cc_out0, 1:rx_clk_mgt

rtc_i2c_clock_ie <= not register_data_in_block_0(14)(0);
rtc_i2c_clock_out <= register_data_in_block_0(14)(1);
rtc_i2c_data_ie <= not register_data_in_block_0(14)(4);
rtc_i2c_data_out <= register_data_in_block_0(14)(5);

register_data_out_block_0(13) <= x"000000" & rtc_i2c_data_ie & rtc_i2c_data_out & rtc_i2c_clock_ie & rtc_i2c_clock_out & rtc_1pps_in & rtc_32khz_in & rtc_i2c_data_in & rtc_i2c_clock_in;
</pre>

== Block 1 GDM ==

* 0x0800+4*i - GDM registers:

<pre>
clk_config_pro : process (read_write_clk)
begin
if rising_edge (read_write_clk) then
CLK_IN_SEL_LS <= clk_config_vec(1 downto 0);
CLK_EXT_SEL_LS <= clk_config_vec(2);
CLK_RSTn_LS <= not clk_config_vec(3);
clk_config_vec(4) <= CLK_LOSXTn_LS;
clk_config_vec(5) <= CLK_LOLn_LS;
clk_config_vec(6) <= CLK_INTn_LS;
clk_config_vec(7) <= '1';

clk_config_vec(3 downto 0) <= register_data_in_block_1(0)(3 downto 0);
register_data_out_block_1(0)( 7 downto 0) <= clk_config_vec(7 downto 0);
register_data_out_block_1(0)(31 downto 8) <= (others =>'0');
register_data_out_block_1(1) <= (others =>'0'); --free
end if;
end process clk_config_pro;

register_data_out_block_1(2) <= register_data_in_block_1(2);
register_data_out_block_1(3) <= register_data_in_block_1(3);
register_data_out_block_1(4)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(5)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(6)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(7)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(8) <= register_data_in_block_1(8);
register_data_out_block_1(9)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(10) <= (others => '0');
register_data_out_block_1(11) <= (others => '0');

frequency_measure_clks <= read_write_clk & clk_50MHz & mgt_data_clk & mgt_ref_clk_raw; -- & mgt_data_clk & mgt_ref_clk_raw;
register_data_out_block_1(12) <= register_data_out_block_1(14); --MGT rx and tx reference clocks are the same
register_data_out_block_1(13) <= register_data_out_block_1(15); --MGT rx and tx data clocks are the same
frequency_measure_inst : frequency_measure
generic map(register_clk_frequency => 100000000,
max_frequency => 300000000,
nfrequency_measurments => nfrequency_measure_clks)
port map(i_register_clk => read_write_clk,
i_clks => frequency_measure_clks,
i_clk_locks => (others =>'1'),
o_frequencies => register_data_out_block_1(14+nfrequency_measure_clks-1 downto 14)); --see register_data_out_block_1(12,13) above

zero_spare_link_registers : for i in 14+nfrequency_measure_clks to nregisters_block_1-1 generate
register_data_out_block_1(i) <= (others => '0');
end generate zero_spare_link_registers;

gdm_link_interface_inst : gdm_link_interface
generic map(enable_rxslide => '1',
nquads => nquads,
nlinks => nlinks)
port map(i_mgt_ref_clk_p => mgt_b_ref_clk_p,
i_mgt_ref_clk_n => mgt_b_ref_clk_n,
i_mgt_rx_n => mgt_b_rx_n,
i_mgt_rx_p => mgt_b_rx_p,
o_mgt_tx_n => mgt_b_tx_n,
o_mgt_tx_p => mgt_b_tx_p,
--raw, pre-pll, mgt clock for frequency measurement
o_mgt_ref_clk_raw => mgt_ref_clk_raw,
--free running clock and reset
i_mgt_free_running_clk => clk_50MHz,
i_mgt_rst => register_data_in_block_1(2)(0),
i_rx_slide_trigger => register_data_in_block_1(3)(nlinks-1 downto 0),
--data interface for the links
o_data_clk => mgt_data_clk,
o_mgt_data_link_array => mgt_link_data_to_processing,
i_mgt_data_link_array => mgt_link_data_from_processing,
--status ports
o_link_power_good => register_data_out_block_1(4)(nlinks-1 downto 0),
o_tx_link_up => register_data_out_block_1(5)(nlinks-1 downto 0),
o_rx_link_up => register_data_out_block_1(6)(nlinks-1 downto 0),
o_rx_error => register_data_out_block_1(7)(nlinks-1 downto 0),
i_link_down_latched_rst => register_data_in_block_1(8)(0),
o_link_down_latched => register_data_out_block_1(9)(nlinks-1 downto 0),
--debug
i_debug_clk => read_write_clk,
o_debug => mgt_debug_data);
</pre>

<pre>
#### Register Block 1 (CDM)
Clock Chip and MGT Link, Control and Status Registers
| Register | Axi Address | Description | note
|:-|:-|:-|:-|
| 0 | 0x80011000 | Clock Chip Status and Control | CLK_IN_SEL_LS(1-0)(read-only), CLK_EXT_SEL_LS(2), CLK_RSTn_LS(3), CLK_LOSXTn_LS(4), CLK_LOLn_LS(5), CLK_I\
NTn_LS(6)
| 1 | 0x80011004 | free for use | |
| 2 | 0x80011008 | MGT Link Reset | mgt_rst(0), Reset for RX Error Latched(1)|
| 3 | 0x8001100c | MGT rx slide control | not needed or used |
| 4 | 0x80011010 | link_up_and_running(10) tx_link_up_and_running(9), tx_sending_data(8), tx_link_up(7), rx_link_up_and_running(6), rx_error(5), rx_receivi\
ng_data(4), rx_link_up(3), link_power_good(2), sfp_rx_los(1), sfp_mod_absent(0) | Should be 0x7DC |
| 5 | 0x80011014 | reserved debug register | |
| 6-11 | ... | free for use | |
| 12 | 0x80011030 | mgt_rx_ref_clk_raw | ~0x7735940 (125 MHz) |
| 13 | 0x80011034 | rx_clk | ~0x7735940 (125 MHz) |
| 14 | 0x80011038 | mgt_tx_ref_clk_raw | ~0x7735940 (125 MHz) |
| 15 | 0x8001103c | tx_clk | ~0x7735940 (125 MHz) |
| 16 | 0x80011040 | clk_50MHz | 0x2faf080 (50 MHz) |
| 17 | 0x80011044 | Register_Block1_clk | 0x5f5e100 (100 MHz) |
| 18-20 | ... | free for use | |
| 21 | 0x80011054 | free for use | |
</pre>

=== Register 0x1000 ===

<pre>
printf("0x1000 SFP c.c. status: 0x%08x\n", reg1000);
printf(" CLK_IN_SEL_LS 0x%x\n", (reg1000>>0) & 3);
printf(" CLK_EXT_SEL_LS %d\n", (reg1000>>2) & 1);
printf(" CLK_CLK_RSTn_LS %d\n", (reg1000>>3) & 1);
printf(" CLK_LOSXTn_LS %d\n", (reg1000>>4) & 1);
printf(" CLK_LOLn_LS %d\n", (reg1000>>5) & 1);
printf(" CLK_INTn_LS %d\n", (reg1000>>6) & 1);
printf("0x1008 SFP link reset: 0x%08x\n", dev->dsdm_read32(0x1008));
</pre>

=== Register 0x1010 ===

<pre>
printf("0x1010 SFP link status: 0x%08x\n", reg1010);
printf(" sfp_mod_absent_N %d\n", (reg1010>>0) & 1);
printf(" sfp_rx_los_N %d\n", (reg1010>>1) & 1);
printf(" link_power_good %d\n", (reg1010>>2) & 1);
printf(" rx_link_up %d\n", (reg1010>>3) & 1);
printf(" rx_receiving_data %d\n", (reg1010>>4) & 1);
printf(" rx_error %d\n", (reg1010>>5) & 1);
printf(" rx_lnk_up_and_running %d\n", (reg1010>>6) & 1);
printf(" tx_link_up %d\n", (reg1010>>7) & 1);
printf(" tx_sending_data %d\n", (reg1010>>8) & 1);
printf(" tx_link_up_and_running %d\n", (reg1010>>9) & 1);
printf(" link_up_and_running %d\n", (reg1010>>10) & 1);

</pre>

=== Register 0x1014 ===

<pre>
printf("0x1014 SFP link data: 0x%08x\n", reg1014);
printf(" rx_data 0x%04x\n", (reg1014>>0) & 0xFFFF);
printf(" k28p1_k28p5 %d\n", (reg1014>>16) & 1);
printf(" rx_ctrl0 0x%01x\n", (reg1014>>17) & 3);
printf(" rx_ctrl1 0x%01x\n", (reg1014>>19) & 3);
printf(" rx_ctrl3 0x%01x\n", (reg1014>>21) & 1);
printf(" tx_state 0x%01x\n", (reg1014>>22) & 3);
printf(" rx_state 0x%01x\n", (reg1014>>24) & 0xF);
printf(" rx_receiving_data %d\n", (reg1014>>28) & 1);
printf(" rx_link_up %d\n", (reg1014>>29) & 1);
printf(" rx_error %d\n", (reg1014>>30) & 1);
printf(" rx_link_rst %d\n", (reg1014>>31) & 1);
</pre>

* see cdm_link_establishment.vhd
* tx_state:
<pre>
0 = tx_idle
1 = waiting_for_rx_link
2 = sending_k28p1_k28p5
3 = sending_data
</pre>

* rx_state:
<pre>
0 = rx_idle
1 = waiting_for_ks
2 = rx_retry
3 = rx_up_switch_tx_clk_source
4 = waiting_for_data
5 = receiving_data
6 = rx_receive_error
</pre>

== Block 2 GDM ==

* 0x1000+4*i - GDM registers:

<pre>
register_data_out_block_2(0) <= register_data_in_block_2(0);
GDM_link_data_processing_inst : GDM_link_data_processing
generic map(mgt_data_frequency => 125000000)
port map(i_register_clk => read_write_clk,
i_rst => register_data_in_block_2(0)(0),
i_data_mode => register_data_in_block_2(0)( 9 downto 8),
i_fixed_pattern_data => register_data_in_block_2(0)(31 downto 16),
o_error_time => register_data_out_block_2(1),
o_error_count => register_data_out_block_2(13 downto 2),
--link data interface
i_data_clk => mgt_data_clk,
i_mgt_data_link_array => mgt_link_data_to_processing,
o_mgt_data_link_array => mgt_link_data_from_processing,
--data interface
o_data_clk => data_clk,
o_tx_data_ready => tx_data_ready,
i_tx_data_is_k => tx_data_is_k,
i_tx_data => tx_data,
o_rx_data_error => rx_data_error,
o_rx_data_is_k => rx_data_is_k,
o_rx_data => rx_data);
</pre>

== Block 2 CDM ==

* 0x1000+4*i - CDM registers:
* reg 0
<pre>
read bits
7..0 - clk_config_vec
1,0 - CLK_IN_SEL_LS
2 - CLK_EXT_SEL_LS
3 - disconnected CLK_RSTn_LS
4 - CLK_LOSXTn_LS
5 - CLK_LOLn_LS
6 - CLK_INTn_LS
7 - tx_clock_status
31..8 - 0
</pre>
<pre>
write bits
3..2 - clk_config_vec[3..2]
</pre>
* reg 1 - 0
* reg 2 - RW
<pre>
bit
0 - cdm_link_interface i_mgt_rst
1 - cdm_link_interface i_link_down_latched_rst
</pre>
* reg 3 - RW
<pre>
bit
0 - cdm_link_interface i_rx_slide_trigger
</pre>
* reg 4 - RO
<pre>
bit
0 - sfp_mod_present
1 - sfp_rx_los
2 - link_power_good
3 - rx_link_up
4 - rx_receiving_data
5 - rx_error
6 - rx_link_up_and_running
7 - tx_link_up
8 - tx_sending_data
9 - tx_link_up_and_running
10 - link_up_and_running
</pre>
* reg 5 - RO - CDM SFP debug_data
* reg 6 - 0
* reg 7 - 0
* reg 8 - 0
* reg 9 - 0
* reg 10 - 0
* reg 11 - 0
* reg 12 - frequency AXI clock
* reg 13 - freq clk_50MHz
* reg 14 - freq clk_100MHz
* reg 15 - freq clk_200MHz
* reg 16 - freq clk_xo_125 - DS-DM 125 MHz oscillator
* reg 17 - freq clk_cc_out0 - C.C. 125 MHz output
* reg 18 - CDM SFP mgt_rx_ref_clk_raw
* reg 19 - CDM SFP mgt_tx_ref_clk_raw
* reg 20 - rx_clk_mgt, 125 MHz CDM SFP RX recovered clock
* reg 21 - rx_clk, 125 MHz, switched between clk_xo_125 and rx_clk_mgt
* reg 22 - tx_clk, 125 MHz
* reg 23 - spare

== Block 3 GDM ==

0x3000+4*i : DS20k registers

== Block 4 CDM ==

* 0x2800+4*i - CDM SFP data link registers:
* 0 - R/W
** bit 0 - 0x1 - reset
** bits 7..1 - not used
** bits 9..8 - data_mode: 00: data from ds20k block, 01: tx_data_cnt, 10: tx_data_prbs, 11: tx_fixed_pattern_data
** bits 15..10 - not used
** bits 31..16 - fixed pattern data
* 1 - RO - o_error_time
* 2 - RO - o_error_count
* 3, etc - not used

== Block 6 CDM ==

* 0x3000+4*i : DS20k registers

== Block 7 CDM ==

VX_TX and VX_RX delays, i=0..11
* 0x3800+4*i - VX_TX delays
* 0x3830+4*i - VX_RX delays

VX_TX delay data (2 bits):
* 4 values: 0,1,2,3 corresponding to the 4 phases of the 125 MHz clock rx_clk and rx_clk shifted by 90 deg

VX_RX delay data (32 bits):
* 0x8000'0000: reset IDELAYE3, follow by write of 0
* 0x0001'xxxx: load the IDELAYE3 9-bit tap counter, follow by write of 0x0000'xxxx
* 16 low bits have:
* bits 0..8 are the 9 bits of IDELAY3 tap counter (~1.1ns for all 511 delays)
* bit 9 is the 500 MHz DDR phase (1 ns)
* bits 10..15 are 16 bits of 500 MHz delay (2 ns step)
* numerically increasing 16-bit delay values are carefully arranged to produce linearly increasing delay (there is some overlap between IDELAYE3 taps above ~tap 500 (or so, IDELAYE3 is uncalibrated) and flip of 500 MHz DDR phase

== DS20k registers ==

=== DS20k block register map ===

* busybox devmem 0x80013000 32

Note: R=readable, W=writable, L=latched by CMD_LATCH, B=reset on begin of run, Z=reset by CMD_LATCH

<pre>
reg | version | xDM | xx | description
0 | 0x20230731 | ALL | RO | ds20k version
0 | 0x20240118 | ALL | RW | ds20k version and command
1 | 0x20230731 | ALL | RW | scratch read/write register
2 | 0x20230731 | ALL | RW | configure inputs and outputs
3 | 0x20230731 | ALL | RW | FP_LED mux
4 | 0x20230731 | ALL | RW | EXT_OUT mux
5 | 0x20230731 | ALL | RO | VX_RX state
6 | 0x20230731 | ALL | RO | VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state
7 | 0x20230731 | ALL | RW | LED_OUT, EXT_OUT, VX_TX outputs
8 | 0x20230731 | ALL | RW | VX_TX mux and config
9 | 0x20230731 | ALL | RW | trg and tsm source
10 | 0x20231013 | ALL | RO | status register
11 | 0x20230731 | ALL | ROLB | trg counter
12 | 0x20230731 | ALL | ROLB | tsm counter
13 | 0x20240814 | ALL | RO | GPS 1pps period, 125 MHz
14 | 0x20240814 | ALL | RO | Ru clock 1pps period, 125 MHz
15 | 0x20230811 | CDM | RO | SFP RX status
16 | 0x20230811 | CDM | RW | SFP TX control
17 | 0x20230811 | GDM | RO | QSFP RX data 0, 1
18 | |. |. | 2, 3
19 | |. |. | 4, 5
20 | |. |. | 6, 7
21 | | |. | 8, 9
22 | |. |. | 10, 11
23 | 0x20230811 | GDM | RW | QSFP TX control
24 | 0x20231013 | ALL | RW | trigger pulser period
25 | 0x20231013 | ALL | RW | trigger pulser burst control
26 | 0x20231013 | ALL | RW | tsm pulser period
27 | 0x20231204 | ALL | RW | data fifo CPU to FPGA
28 | 0x20231204 | ALL | RW | packet loopback control
28 | 0x20240118 | ALL | RW | data fifo FPGA to CPU
29 | 0x20231208 | GDM | RW | bitmap of active qsfp ports
29 | 0x20240118 | ALL | RW | packet loopback control
29 | 0x20240510 | ALL | RW | packet routing
30 | 0x20231208 | GDM | RO | qsfp link status ports 0..7
31 | 0x20231208 | GDM | RO | qsfp link status ports 8..11
32 | 0x20240118 | GDM | RW | enabled QSFP ports
33 | 0x20240118 | CDM | RW | enabled VX ports
34,35 | 0x20240118 | ALL | ROB | time stamp 64 bits
36,37 | 0x20240118 | ALL | RO | old time stamp
38,39,40 | 0x20240118 | CDM | ROLZ | vx_bsy_counter_latched | VX busy counters
41,42,43 | 0x20240118 | GDM | ROLZ | QSFP busy counters
44 | 0x20240118 | CDM | ROLZ | cdm_bsy_up_counter and cdm_bsy_pulse_counter
45 | 0x20240118 | CDM | ROLZ | cdm_veto_pulse_counter and cdm_veto_up_counter
46 | 0x20240118 | GDM | ROLZ | gdm_bsy_pulse_counter and gdm_bsy_up_counter
47 | 0x20240118 | GDM | ROLZ | gdm_bsy_refresh_counter
48 | 0x20240118 | GDM | ROLZ | gdm_veto_up_counter and gdm_veto_pulse_counter
49,50,51 | 0x20240424 | CDM | RO | VX RX serial data monitor, 8 bits per VX port
52,53 | 0x20240424 | CDM | RO | VX serial link status, 4 bits per VX port
54 | 0x20240424 | CDM | RO | VX TX serial data monitor
55 | 0x20240430 | ALL | RO | QSFP, SFP and VX link loss counters
56 | 0x20240510 | ALL | RO | sfp_rx_packet_counter
57 | same | ALL | RO | sfp_tx_packet_counter
58 | same | ALL | RO | qsfp_rx_packet_counter[0]
59 | same | ALL | RO | qsfp_tx_packet_counter
60 | same | ALL | RO | vx_rx_packet_counter[0]
61 | same | ALL | RO | vx_tx_packet_counter
62 | 0x20240719 | CDM | RO | cdm_hitmap_period, ports 0, 1
63 | same | CDM | RO | ports 2, 3
64 | same | CDM | RO | ports 4, 5
65 | same | CDM | RO | ports 6, 7
66 | same | CDM | RO | ports 8, 9
67 | same | CDM | RO | ports 10, 11
62 | 0x20260401 | CDM | ROL | vx_rx_hitmap_period latched by cmd_latch, VX ports 0, 1
63 | same | CDM | ROL | ports 2, 3
64 | same | CDM | ROL | ports 4, 5
65 | same | CDM | ROL | ports 6, 7
66 | same | CDM | ROL | ports 8, 9
67 | same | CDM | ROL | ports 10, 11
68 | 0x20240814 | ALL | RW | GPS control and status
69 | 0x20241104 | CDM | ROLB | vx_tx_trg_packet_counter | counter of TRG packets CDM->VX
70 | 0x20241104 | CDM | ROLB | vx_tx_tsm_packet_counter | counter of TSM packets CDM->VX
71 | 0x20241104 | ALL | RO | packet error bits
72,73 | 0x20241104 | CDM | ROL | vx_tx_trg_data_latched[63:0] trigger packet data
74,75 | 0x20241104 | CDM | ROL | vx_tx_tsm_data_latched[63:0] tsm packet data (truncated to 64 bits)
76 | 0x20241104 | CDM | ROLB | cdm_hitmap_trigger_counter_latched, counter of hitmap triggers generated in the CDM
77,78,79 | 0x20241104 | CDM | ROL | vx_rx_hitmap_data_latched | VX_RX 80 bits of HITMAP packet from VX
80 | 0x20241209 | CDM | RO | vx_lvds_rx_link_status | VX LVDS link status
81 | 0x20241209 | CDM | RO | vx_lvds_rx_trg_data | VX LVDS TRG packet first 32 bits
81 | 0x20251010 | CDM | - | not used | not used
82 | 0x20241209 | CDM | RO | vx_lvds_rx_tsm_data | VX LVDS TSM packet first 32 bits
82 | 0x20251010 | CDM | - | not used | not used
83 | 0x20241209 | CDM | ROL | vx_lvds_rx_decoded_packet_counter_latched | VX LVDS decoded packet counter
84 | 0x20241209 | CDM | ROL | vx_lvds_rx_packet_error_counter_latched | VX LVDS packet error counter
85 | 0x20250417 | CDM | ROLB | vx_tx_trg_counter | VX LVDS TRG counter (VX LVDS pair 12)
86 | 0x20250417 | CDM | ROLB | vx_tx_tsm_counter | VX LVDS TSM counter (VX LVDS pair 13)
87 | 0x20250417 | CDM | ROLB | vx_tx_veto_counter | VX LVDS Veto counter (VX LVDS pair 13)
88 | 0x20250422 | GDM | ROLB | trg_in_lost_counter | counter of TRG killed by gdm_busy
89 | 0x20250422 | ALL | RO | fifo_from_fpga_lost_packet_counter | counter of packets lost due to FIFO full
90 | 0x20250428 | CDM | ROL | vx_bsy_status_latched | status of VX busy, CDM busy, CDM veto
91 | 0x20250428 | CDM | ROL | vx_hitmap_status_latched | status of VX hitmap
92 | 0x20250428 | GDM | ROL | cdm_bdy_status_latched | status of CDM busy, GDM busy, GDM veto
93 | 0x20250508 | GDM | ROL | gdm_hitmap_status_latched | status of GDM hitmap
94 | 0x20250508 | GDM | ROLB | gdm_hitmap_trigger_counter_latched | counter of hitmap triggers generated in the GDM
95 | 0x20250512 | GDM | ROLB | gdm_trig48_counter_latched | counter of TRIG48 trigger packets
96 | 0x20250512 | GDM | ROLB | gdm_trig48_lost_counter_latched | counter of TRIG48 triggers lost to busy
97 | 0x20250512 | CDM | ROLB | vx_tx_trig48_packet_counter_latched | counter of TRIG48 packets from CDM to VX
98, 99 | 0x20250512 | GDM | RW | trig48_ctrl48, trig48_ctrl16 | GDM TRIG48 test registers
100,101 | 0x20251008 | CDM | ROL | vx_lvds_rx_trig48_{data,bitmap48}_latched | VX LVDS TRIG48 packet first 32 bits, bitmap48 first 32 bits
102 | 0x20251008 | ALL | RO | cmd_counter | counter of reg0 commands
103,104,105 | 0x20260401 | CDM | ROL | vx_rx_hitmap_vx_id_latched | VX ID from VX hitmap packets, VX ports 0..11, in order, latched by cmd_latch
106 | 0x20260401 | CDM | RW | conf_vx_rx_hitmap_cumulative_period | period of CDM CUMULATIVE_HITMAP packets, set to 0 to disable them.
</pre>

=== Register 0 0x80013000 ds20k version ===

on read: ds20k version 0xYYYYMMDD

on write:
* 0 - noop - as of version 0x20240118, write a zero after writing a command
* 1 - cmd_reset - reset logic to good state
* 2 - cmd_arm_ts - arm timestamp reset
* 3 - cmd_trg - issue a trigger
* 4 - cmd_tsm - issue a tsm
* 5 - cmd_vx_rx_reset - reset the VX receive path
* 6 - cmd_vx_tx_reset - reset the VX transmit path
* 7 - cmd_gdm_trig48 - generate a TRIG48 trigger and data packet (GDM)
* 8 - cmd_trg_pulser_reset - reset the trigger pulser
* 9 - cmd_tsm_pulser_reset - reset the tsm pulser
* 10 - cmd_bor_start - start begin-of-run trigger sequence
* 11 - cmd_bor_clear - after run has started, clear begin-of-run status bits
* 12 - cmd_latch - latch counters & etc into AXI registers for coherent readout
* 13 - cmd_cdm_hitmap_trigger - generate a hitmap trigger and data packet (CDM)

=== Register 1 0x80013004 scratch ===

scratch read-write register

=== Register 2 0x80013008 input and output config ===

<pre>
bit | version | fpga name | description
0 | ALL | lemo_enable | enable LEMO input 1
1 | | | 2
2 | | | 3
3 | | | 4
4 | ALL | lemo_invert | invert LEMO input 1
5 | | | 2
6 | | | 3
7 | | | 4
9 | ALL | ext_out_disable | disable LEMO output 1
10 | | | 2
11 | ALL | ext_out_invert | invert LEMO output 1
12 | | | 2
</pre>

=== Register 3 0x8001300C FP_LED control ===

<pre>
wire [15:0] led_out_mux_sel = register_data_in[3][15:0];
wire [3:0] led_out_invert = register_data_in[3][19:16];
</pre>

<pre>
led_out_mux_sel is 4 groups (one per LED) of 4 bits (choice 0..15):
mux | version | fpga name | description
0 | ALL | | power on default
1 | ALL | led_out_reg | register 7 bits
2 | 0x20231013 | pll_locked | clock chip PLL is locked
3 | 0x20231013 | sfp_link_status | SFP link is good
3 | 0x20240118 | sfp_link_status or qsfp_tx_link_rx_status | SFP/QSFP link is good
4 | ALL | lemo_in_sync[1] | LEMO input 1
5 | ALL | lemo_in_sync[2] | LEMO input 2
6 | ALL | lemo_in_sync[3] | LEMO input 3
7 | ALL | lemo_in_sync[4] | LEMO input 4
8 | ALL | ext_out[1] | LEMO output 1
9 | ALL | ext_out[2] | LEMO output 2
A | 0x20231013 | trg_in | trigger
B | 0x20231013 | tsm_in | time slice marker
C | 0x20240118 | gdm_bsy | GDM busy: OR of all CDM busy
D | 0x20240118 | cdm_bsy | CDM busy: OR of all VX busy
E | 0x20240118 | cdm_veto | GDM busy -> GDM veto -> CDM veto -> VX trigger veto
F | ALL | | fixed logic level 1
</pre>

=== Register 4 0x80013010 LEMO OUT control ===

<pre>
wire [7:0] ext_out_mux_sel = register_data_in[4][7:0];
</pre>

<pre>
ext_out_mux_sel is 2 groups (one per LEMO) of 4 bits (choice 0..15):

mux | version | fpga name | description
0 | ALL | | power on default
1 | ALL | ext_out_reg | register 7
2 | ALL | trg_pulser | pulser trigger
3 | 0x20240724 | vx1_tx_out[2] | vx1 serial data out
4 | 0x20240724 | vx1_rx[1] | vx1 serial data in
5 | 0x20240724 | vx_rx_iob[0] | vx1 serial data in captured by IOB register
6 | ALL | lemo_in_async[1]| test synchronizer
7 | ALL | lemo_in_sync[1] | test synchronizer
8 | 0x20240724 | trg_in_pulse | trigger signal
9 | 0x20240724 | tsm_in_pulse | time slice marker signal
A | 0x20240118 | cdm_bsy | CDM busy from VX
B | 0x20240118 | gdm_bsy | GDM busy from CDM
C | 0x20240118 | cdm_veto | veto from GDM to CDM to VX
D | 0x20240118 | vx1_rx[1] | serial data VX to CDM
E | not used (sink)
F | ALL | 1 | fixed logic level 1
</pre>

=== Register 5 0x80013014 VX_RX status ===

<pre>
assign register_data_out[5] = {
vx8_rx[3], vx8_rx[2], vx8_rx[1], vx8_rx[0],
vx7_rx[3], vx7_rx[2], vx7_rx[1], vx7_rx[0],
vx6_rx[3], vx6_rx[2], vx6_rx[1], vx6_rx[0],
vx5_rx[3], vx5_rx[2], vx5_rx[1], vx5_rx[0],
vx4_rx[3], vx4_rx[2], vx4_rx[1], vx4_rx[0],
vx3_rx[3], vx3_rx[2], vx3_rx[1], vx3_rx[0],
vx2_rx[3], vx2_rx[2], vx2_rx[1], vx2_rx[0],
vx1_rx[3], vx1_rx[2], vx1_rx[1], vx1_rx[0]
};
</pre>

=== Register 6 0x80013018 VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state ===

<pre>
assign register_data_out[6] = {
vx2_tx_out[2], vx2_tx_out[1], vx2_tx_out[0], vx1_tx_out[2],
vx1_tx_out[1], vx1_tx_out[0], ext_out[2], ext_out[1],
fp_led_out[3], fp_led_out[2], fp_led_out[1], fp_led_out[0],
ext_in_lv[4], ext_in_lv[3], ext_in_lv[2], ext_in_lv[1],
vx12_rx[3], vx12_rx[2], vx12_rx[1], vx12_rx[0],
vx11_rx[3], vx11_rx[2], vx11_rx[1], vx11_rx[0],
vx10_rx[3], vx10_rx[2], vx10_rx[1], vx10_rx[0],
vx9_rx[3], vx9_rx[2], vx9_rx[1], vx9_rx[0]
};
</pre>

<pre>
bit | version | fpga name | description
0 | ALL | vx9_rx | VX_RX
1 | | |
2 | | |
3 | | |
4 | ALL | vx10_rx | VX_RX
5 | | |
6 | | |
7 | | |
8 | ALL | vx11_rx | VX_RX
9 | | |
10 | | |
11 | | |
12 | ALL | vx12_rx | VX_RX
13 | | |
14 | | |
15 | | |
16 | ALL | ext_in_lv | LEMO inputs
17 | | |
18 | | |
19 | | |
20 | ALL | FP_LED | FP_LEDs
21 | | |
22 | | |
23 | | |
24 | ALL | ext_out[1] | LEMO outputs
25 | | ext_out[2] |
26 | ALL | vx1_tx | VX1_TX
27 | | |
28 | | |
29 | ALL | vx2_tx | VX2_TX
30 | | |
31 | | |
</pre>

=== Register 7 0x8001301C LED_OUT, EXT_OUT, VX_TX outputs ===

<pre>
wire [3:0] led_out_reg = register_data_in[7][3:0];
wire [2:1] ext_out_reg = register_data_in[7][5:4];
// register_data_in[7][6];
// register_data_in[7][7];
wire [7:0] vx_tx_out_reg = register_data_in[7][15:8];
</pre>

<pre>
bit | version | fpga name | description
0 | ALL | led_out_reg | FP_LED 1
1 | | | 2
2 | | | 3
3 | | | 4
4 | ALL | ext_out_reg | LEMO OUT 1
5 | | | 2
6 | - | |
7 | - | |
8 | ALL | vx_tx_out_reg | VX1_TX 0
9 | | | 1
10 | | | 2
11 | | | -
12 | ALL | | VX2_TX 0
13 | | | 1
14 | | | 2
15 | | | -
16 | - | |
17 | | |
18 | | |
19 | | |
20 | | |
21 | | |
22 | | |
23 | | |
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 8 0x80013020 VX_TX config ===

<pre>
wire [3:0] vx_tx_mux_sel = register_data_in[8][3:0];
wire vx_tx_from_sfp = register_data_in[8][30];
wire vx_tx_loopback = register_data_in[8][31];
</pre>

<pre>
vx_tx_mux_sel is 4 bits (choice 0..15):
0 - power on default, control by vx_tx_out_reg
1 - GDM
2 - CDM
3 - pulser loopback test
4 - pulser loopback test
5 - 62.5 MHz output
6 - 125 MHz output
7 - trg, tsm, serial
8 - trg, tsm, lvds serial rx to serial tx loopback
9 - GPS box control (ds20k rev 0x20240814)
10
11
12
13
14
15 - production config: trg, veto, serial
</pre>

=== Register 9 trg and tsm source ===

from version 0x20240724

<pre>
wire [15:0] trg_src_mask = register_data_in[9][15:0];
wire [31:16] tsm_src_mask = register_data_in[9][31:16];
</pre>

<pre>
wire [15:0] trg_src_bits =
{
1'b0, // 15
1'b0, // 14
vx_tx_tsm_done, // 13
vx_tx_trg_done, // 12

1'b0, // gdm_hitmap_trigger, // 11
cdm_hitmap_trigger, // 10
sfp_rx_tsm, // 9
sfp_rx_trg, // 8

sfp_rx_data[1], // 7
sfp_rx_data[0], // 6
tsm_pulser, // 5
trg_pulser, // 4

lemo_in_sync[4], // 3
lemo_in_sync[3], // 2
lemo_in_sync[2], // 1
lemo_in_sync[1] // 0
};

wire [15:0] trg_bits = trg_src_bits & trg_src_mask;
wire [15:0] tsm_bits = trg_src_bits & tsm_src_mask;
</pre>

before that:

<pre>
wire [7:0] trg_src_mask = register_data_in[9][7:0];
wire [7:0] tsm_src_mask = register_data_in[9][15:8];
wire trg_pulser_enable = register_data_in[9][16];
wire tsm_pulser_enable = register_data_in[9][17];
wire trg_software = register_data_in[9][18];
wire tsm_software = register_data_in[9][19];
// bits 20:31 not used
</pre>

<pre>
wire [7:0] xxx_src_bits =
{
sfp_rx_data[1],
sfp_rx_data[0],
tsm_pulser & tsm_pulser_enable,
trg_pulser & trg_pulser_enable,
lemo_in_sync[4],
lemo_in_sync[3],
lemo_in_sync[2],
lemo_in_sync[1]
};
</pre>

<pre>
trg_src_mask and tsm_src_mask bits:
0 - LEMO IN 1
1 - LEMO IN 2
2 - LEMO IN 3
3 - LEMO IN 4
4 - trg_pulser
5 - tsm_pulser
6 - sfp_rx_data[0] // to become sfp_trg_in, selected from sfp_rx_data[0], cdm_rx trg_out and gdm trigger packet
7 - sfp_rx_data[1] // to become sfp_tsm_in, selected from sfp_rx_data[1], cdm_rx tsm_out and gdm tsm packet
</pre>

=== Register 10 0x80013028 status register ===

<pre>
bit | version | fpga name | description
0 | 0x20231013 | pll_locked | clock chip PLL is locked
1 | 0x20240118 | ts_reset_armed | timestamp reset is armed
2 | 0x20240118 | qsfp_tx_link_rx_status | QSFP link status is good for all enabled ports
3 | 0x20240424 | vx_tx_link_rx_status | VX link status is good for all enabled ports
4 | 0x20240118 | cdm_bsy | VX busy grand-or
5 | 0x20240118 | gdm_bsy | QSFP busy grand-or
6 | 0x20240118 | gdm_veto | gdm_veto = gdm_busy
7 | 0x20240118 | cdm_veto | CDM veto from GDM to VX
8 | 0x20240725 | bor_started | begin-of-run sequence started
9 | 0x20240725 | bor_finished | begin-of-run sequence fininished, see commands 10 and 11
10 | | |
11 | | |
12 | | |
13 | | |
14 | | |
15 | | |
16 | | |
17 | | |
18 | | |
19 | | |
20 | | |
21 | | |
22 | | |
23 | | |
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 11 0x8001302C trg_counter ===

trigger counter

=== Register 12 0x80013030 tsm_counter ===

time slice marker counter

=== Register 13 0x80013034 GPS 1pps period ===

GPS 1pps period in 8 ns clocks

=== Register 14 0x80013038 Rb clock 1pps period ===

PRS-10 Rb clock 1pps output period in 8 ns clocks

=== Register 15 0x8001303C SFP RX status ===

<pre>
bit | ds20k version | fpga signal name | description
0 | ALL | sfp_rx_data[15:0] | cdm sfp received data
16 | same | sfp_rx_data_is_k[0] |
17 | same | sfp_rx_data_is_k[1] |
18 | | 0 |
19 | 0x20231204 | sfp_rx_sel_lpb | sfp tx->rx loopback
20 | 0x20231013 | sfp_link_status | sfp link connected, exchanging data
21 | same | sfp_link_rx_status | sfp link receiving correct idle pattern from GDM TX
22 | same | sfp_link_error | sfp link receiver error (badk or overflow)
23 | same | sfp_rx_data_error | sfp transceiver state machine is in error state
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 16 SFP TX control ===

<pre>
wire [15:0] sfp_tx_data_reg = register_data_in[16][15:0]; // SFP TX data K-code
wire [1:0] sfp_tx_ctrl_reg = register_data_in[16][17:16]; // SFP TX data K-code
// 18
// 19
// 23:20
// 27:24
wire sfp_rx_sel_lpb = register_data_in[16][28]; // TX->RX serial loopback
wire sfp_tx_sel_lpb = register_data_in[16][29]; // RX->TX serial loopback
wire sfp_tx_sel_trg = register_data_in[16][30]; // 16 individual bits
wire sfp_tx_sel_reg = register_data_in[16][31]; // TX data from register 16 bits 17:0
</pre>

=== Register 17-22 QSFP RX data ===

QSFP RX data links 0..11

=== Register 23 QSFP TX control ===

<pre>
wire [15:0] qsfp_tx_data_reg = register_data_in[23][15:0];
wire [1:0] qsfp_tx_ctrl_reg = register_data_in[23][17:16];

wire qsfp_tx_enable_trg = register_data_in[23][24]; // enable QSFP TX trg_in_pulse k-code
wire qsfp_tx_enable_tsm = register_data_in[23][25]; // enable QSFP TX tsm_in_pulse k-code

//wire qsfp_rx_sel_lpb = register_data_in[23][28]; // TX->RX loopback
wire qsfp_tx_sel_lpb = register_data_in[23][29]; // RX->TX loopback
wire qsfp_tx_sel_trg = register_data_in[23][30]; // 16 individual bits
wire qsfp_tx_sel_reg = register_data_in[23][31]; // data from register
</pre>

=== Register 24 0x80013060 trigger pulser period ===

trigger pulser period in units of 8 ns (125 MHz clock)

=== Register 25 0x80013064 trigger burst pulser ===

<pre>
wire [7:0] conf_burst_count = conf_pulser_burst_ctrl[31:24];
wire [23:0] conf_burst_period = conf_pulser_burst_ctrl[23:0];
</pre>

=== Register 26 0x80013068 tsm pulser period ===

time slice marker period in units of 8 ns (125 MHz clock)

=== Register 27 0x8001306C data write fifo ===

<pre>
wire fifo_reset = register_data_in[27][31];
wire fifo_to_fpga_wr1 = register_data_in[27][27];
wire fifo_to_fpga_wr2 = register_data_in[27][26];
wire [16:0] fifo_to_fpga_din = register_data_in[27][16:0];
</pre>

<pre>
assign register_data_out[27][31:24] = register_data_in[27][31:24]; // echo write bits
assign register_data_out[27][23] = fifo_to_fpga_full;
assign register_data_out[27][22] = fifo_to_fpga_empty;
</pre>

=== Register 28 0x80013070 data read fifo ===

<pre>
wire fifo_reset = register_data_in[28][31];
wire fifo_from_fpga_rd1 = register_data_in[28][25];
wire fifo_from_fpga_rd2 = register_data_in[28][24];
</pre>

<pre>
assign register_data_out[28][31:24] = register_data_in[28][31:24]; // echo write bits
assign register_data_out[28][21] = fifo_from_fpga_full;
assign register_data_out[28][20] = fifo_from_fpga_empty;
assign register_data_out[28][16:0] = fifo_from_fpga_dout;
</pre>

=== Register 29 0x80013074 packet routing ===

Control packet routing and loopbacks:

<pre>
wire [3:0] dn_route_ctrl = register_data_in[29][3:0];
wire [3:0] up_route_ctrl = register_data_in[29][7:4];
wire [3:0] fifo_to_fpga_route_ctrl = register_data_in[29][11:8];
//wire [3:0] spare_route_ctrl = register_data_in[29][15:12];

wire dn_mux_trg_enable = register_data_in[29][16];
wire dn_mux_tsm_enable = register_data_in[29][17];
wire dn_mux_sfp_rx_fifo_enable = register_data_in[29][18];
// 19
// 20..23

wire up_mux_vx_rx_enable = register_data_in[29][24];
// 25
// 26
// 27
wire fifo_from_fpga_hitmap_enable = register_data_in[29][28];
wire fifo_from_fpga_cdm_busy_enable = register_data_in[29][29];
wire fifo_from_fpga_gdm_busy_enable = register_data_in[29][30];
wire fifo_from_fpga_tsm_enable = register_data_in[29][31];
</pre>

fifo_to_fpga output routing:
* 0 - to down packet mux
* 1 - to up packet mux
* 2 - to fifo_from_fpga mux
* 3 - not used

down packet mux inputs:
* fifo_to_fpga_0
* trg_pkt16 enabled by dn_mux_trg_enable
* tsm_pkt16 enabled by dn_mux_tsm_enable
* sfp_rx_fifo_pkt16 enabled by dn_mux_sfp_rx_fifo_enable
* up_pkt16_2 loopback from up packet mux

down packet mux output routing:
* 0 - to vx_tx_pkt16 to VX TX (vx_link_tx and vx_ser_tx)
* 1 - to fifo_from_fpga mux
* 2 - to up packet mux loopback
* 3 - to qsfp_tx_pkt16 to GDM QSFP TX (cdm_link_tx)

up packet mux inputs:
* vx_rx_pkt16 enabled by up_mux_vx_rx_enable data from 12 VX RX links
* fifo_to_fpga_1
* dn_pkt16_2 loopback from down packet mux

up packet mux output routing:
* 0 - to sfp_tx_pkt16 to CDM SFP TX (VX data to GDM)
* 1 - to fifo_from_fpga mux
* 2 - to down packet mux loop loopback
* 3 - not used

internal_mux inputs:
* 0 - cdm_hitmap_pkt16 enabled by fifo_from_fpga_hitmap_enable (reg 29)
* 1 - cdm_busy_pkt16 enabled by fifo_from_fpga_cdm_busy_enable
* 2 - gdm_busy_pkt16 enabled by fifo_from_fpga_gdm_busy_enable
* 3 - gdm_hitmap_pkt16 enabled by fifo_from_fpga_hitmap_enable
* 4 - not used

debug_mux inputs:
* 0 - not used
* 1 - dn_pkt16_1
* 2 - up_pkt16_1
* 3 - qsfp0_rx_pkt16 enabled by fifo_from_fpga_qsfp0_rx_enable (from GDM QSFP link 0, there is no GDM QSFP 12-to-1 mux)
* 4 - not used

fifo_from_fpga mux inputs:
* 0 - fifo_to_fpga_2
* 1 - tsm2_pkt16 enabled by fifo_from_fpga_tsm_enable (reg 29)
* 2 - not used
* 3 - debug_pkt16 from debug_mux
* 4 - internal_pkt16 from internal_mux

=== Register 30 0x80013078 qsfp link status ports 0..7 ===

<pre>
assign register_data_out[30] =
{
qsfp_rx_data_error[7], qsfp_link_error[7], qsfp_link_status[7], qsfp_link_rx_status[7], // 7
qsfp_rx_data_error[6], qsfp_link_error[6], qsfp_link_status[6], qsfp_link_rx_status[6], // 6
qsfp_rx_data_error[5], qsfp_link_error[5], qsfp_link_status[5], qsfp_link_rx_status[5], // 5
qsfp_rx_data_error[4], qsfp_link_error[4], qsfp_link_status[4], qsfp_link_rx_status[4], // 4
qsfp_rx_data_error[3], qsfp_link_error[3], qsfp_link_status[3], qsfp_link_rx_status[3], // 3
qsfp_rx_data_error[2], qsfp_link_error[2], qsfp_link_status[2], qsfp_link_rx_status[2], // 2
qsfp_rx_data_error[1], qsfp_link_error[1], qsfp_link_status[1], qsfp_link_rx_status[1], // 1
qsfp_rx_data_error[0], qsfp_link_error[0], qsfp_link_status[0], qsfp_link_rx_status[0] // 0
};
</pre>

=== Register 31 0x8001307C qsfp link status ports 8..11 ===

<pre>
assign register_data_out[31] =
{
4'b0000,
4'b0000,
4'b0000,
4'b0000,
qsfp_rx_data_error[11], qsfp_link_error[11], qsfp_link_status[11], qsfp_link_rx_status[11], // 11
qsfp_rx_data_error[10], qsfp_link_error[10], qsfp_link_status[10], qsfp_link_rx_status[10], // 10
qsfp_rx_data_error[9], qsfp_link_error[9], qsfp_link_status[9], qsfp_link_rx_status[9], // 9
qsfp_rx_data_error[8], qsfp_link_error[8], qsfp_link_status[8], qsfp_link_rx_status[8] // 8
};
</pre>

=== Register 32 0x80013080 bitmap of enabled qsfp ports ===

<pre>
wire [11:0] qsfp_mask = register_data_in[32][11:0];
wire qsfp_bsy_force = register_data_in[32][12];
wire gdm_veto_enable = register_data_in[32][13];
// not used = register_data_in[32][15:14];
wire [15:0] gdm_veto_extend = register_data_in[32][31:16];
</pre>

On the GDM:

qsfp_rx_bsy[11..0] are pulses received from the CDMs

qsfp_rx_bsy_or is the grand-or of qsfp_rx_bsy masked by qsfp_mask

if qsfp_rx_bsy_or is high, gdm_bsy goes up and stays up for vx_bsy_extend*2 clocks

gdm_veto = gdm_bsy (bsy of any one VX causes trigger veto to all of them)

if (gdm_veto_enable) then gdm_veto will block trg_in_pulse

gdm_veto transition 0->1 causes gdm_veto_pulse. as long as gdm_veto is high, gdm_veto_pulse is generated every gdm_veto_extend clocks

gdm_veto_pulse is sent to all CDMs.

For this to work right, cdm_bsy_extend should not be bigger than gdm_veto_extend.

=== Register 33 0x80013084 bitmap of enabled VX ports ===

<pre>
wire [11:0] vx_mask = register_data_in[33][11:0];
wire vx_bsy_force = register_data_in[33][12];
// not used = register_data_in[33][15:13];
wire [15:0] vx_bsy_extend = register_data_in[33][31:16];
</pre>

On the CDM:

cdm_bsy is a grand-or of all vx_bsy masked by vx_mask (list of active VXes).

vx_bsy_extend controls how often state of cdm_bsy is sent to the GDM. when cdm_bsy goes 0->1, we send a cdm_bsy_pulse and keep resending it every vx_bsy_extend clocks as long as cdm_bsy stays high.

cdm_bsy_pulse is sent to the GDM.

sfp_rx_veto is the received from the GDM

if sfp_rx_veto goes up, cdm_veto goes up and stays up for gdm_veto_extend clocks.

for this to work right, CDM gdm_veto_extend must be bigger than GDM gdm_veto_extend.

cdm_veto goes to VXes on v1_tx_out[1] which is LVDS input 13.

=== Register 34, 35 0x80013088, 8C current timestamp ===

current 64-bit timestamp, 125 MHz

=== Register 36, 37 0x80013090, 94 old timestamp ===

old 64-bit timestamp, 125 MHz. timestamp saved at run start when it is reset to 0.

=== Register 38, 39, 40 0x80013098, 9C, A0 VX busy counters ===

8 bits per VX port, counters overflow to 255, reset at run start.

=== Register 41, 42, 43 0x800130A4, A8, AC QSFP busy counters ===

8 bits per QSFP port, counters overflow to 255, reset at run start.

=== Register 44, 45, 46, 47, 48 CDM and GDM busy and veto counters ===

* cdm_busy = grand-or of all VX busy for enabled VXes
* gdm_busy = grand-or of all CDM busy for enabled CDMs
* gdm_veto = gdm_busy

<pre>
44 | lo 16 bits | cdm_bsy_up_counter | CDM busy, increments when cdm_busy goes 0->1
44 | hi 16 bits | cdm_bsy_pulse_counter | CDM busy to GDM, increments for each cdm_bsy_pulse sent to the GDM

45 | lo 16 bits | cdm_veto_pulse_counter | CDM veto from GDM, increments for each sfp_rx_veto received from the GDM
45 | hi 16 bits | cdm_veto_up_counter | CDM veto to VX, increments each time cdm_veto is set to 1.

46 | lo 16 bits | gdm_bsy_pulse_counter | GDM busy from CDM, increments for each qsfp_rx_busy received from CDMs (unless they overlap)
46 | hi 16 bits | gdm_bsy_up_counter | GDM busy, increments each time gdm_bsy goes 0->1

47 | lo 16 bits | gdm_bsy_refresh_counter | GDM busy, increments each time gdm_bsy is extended by new qsfp_rx_busy
47 | hi 16 bits | spare |

48 | lo 16 bits | gdm_veto_up_counter | GDM veto, increments each time gdm_veto goes 0->1
48 | hi 16 bits | gdm_veto_pulse_counter | GDM veto to CDM, increments for each gdm_veto_pulse sent to the CDM
</pre>

=== Register 49, 50, 51 0x800130xx VX RX serial data monitor ===

VX RX data, 8-bit per VX channel. k-bit is omitted.

<pre>
assign register_data_out[49][7:0] = vx_rx_data[0]; // vx1
assign register_data_out[49][15:8] = vx_rx_data[1]; // vx2
assign register_data_out[49][23:16] = vx_rx_data[2]; // vx3
assign register_data_out[49][31:24] = vx_rx_data[3]; // vx4

assign register_data_out[50][7:0] = vx_rx_data[4]; // vx5
assign register_data_out[50][15:8] = vx_rx_data[5]; // vx6
assign register_data_out[50][23:16] = vx_rx_data[6]; // vx7
assign register_data_out[50][31:24] = vx_rx_data[7]; // vx8

assign register_data_out[51][7:0] = vx_rx_data[8]; // vx9
assign register_data_out[51][15:8] = vx_rx_data[9]; // vx10
assign register_data_out[51][23:16] = vx_rx_data[10]; // vx11
assign register_data_out[51][31:24] = vx_rx_data[11]; // vx12
</pre>

=== Register 52, 53 VX link status ===

<pre>
assign register_data_out[52] =
{
vx_rx_error[7], vx_link_error[7], vx_link_status[7], vx_link_rx_status[7], // 7
vx_rx_error[6], vx_link_error[6], vx_link_status[6], vx_link_rx_status[6], // 6
vx_rx_error[5], vx_link_error[5], vx_link_status[5], vx_link_rx_status[5], // 5
vx_rx_error[4], vx_link_error[4], vx_link_status[4], vx_link_rx_status[4], // 4
vx_rx_error[3], vx_link_error[3], vx_link_status[3], vx_link_rx_status[3], // 3
vx_rx_error[2], vx_link_error[2], vx_link_status[2], vx_link_rx_status[2], // 2
vx_rx_error[1], vx_link_error[1], vx_link_status[1], vx_link_rx_status[1], // 1
vx_rx_error[0], vx_link_error[0], vx_link_status[0], vx_link_rx_status[0] // 0
};

assign register_data_out[53] =
{
//4'b0000,
1'b0, ~vx_rx_deser_rdy[0], vx_rx_code_err[0], vx_rx_disp_err[0],
//4'b0000,
//4'b0000,
//4'b0000,
vx_rx_monitor[0],
vx_rx_error[11], vx_link_error[11], vx_link_status[11], vx_link_rx_status[11], // 11
vx_rx_error[10], vx_link_error[10], vx_link_status[10], vx_link_rx_status[10], // 10
vx_rx_error[9], vx_link_error[9], vx_link_status[9], vx_link_rx_status[9], // 9
vx_rx_error[8], vx_link_error[8], vx_link_status[8], vx_link_rx_status[8] // 8
};
</pre>

vx_rx_monitor (12-bit) is from deserializer_10b.sv:

<pre>
assign monitor_out[9:0] = lastByte[9:0];
assign monitor_out[10] = comma;
assign monitor_out[11] = ready;
</pre>

=== Register 54 VX TX serial data monitor ===

* contents of vx_tx_monitor from vx_ser_tx.sv:
<pre>
assign monitor_out[8:0] = data_to_encoder; // 8-bit + k
assign monitor_out[9] = valid;
assign monitor_out[15:10] = 0;
assign monitor_out[25:16] = encoded_data; // 10-bit
assign monitor_out[26] = encoded_valid;
assign monitor_out[27] = 0;
assign monitor_out[31:28] = 0;
</pre>

=== Register 55 QSFP, SFP, VX link loss counters ===

<pre>
assign register_data_out[55] = {
vx_rx_error_counter,
qsfp_link_rx_status_drop_counter,
sfp_link_rx_status_drop_counter,
vx_link_rx_status_drop_counter
};
</pre>

<pre>
24..31 - CDM VX RX error counters, count any errors in the VX RX data path (bad serial data, fifo overflow, etc)
16..23 - GDM QSFP link loss counter, increments on qsfp_tx_link_rx_status 1->0 (reg10)
8..15 - CDM SFP link loss counter, increments on sfp_link_rx_status 1->0 (reg15)
0...7 - CDM VX link loss counter, increments on vx_tx_link_rx_status 1->0 (reg10)
</pre>

=== Register 68 GPS control and status ===

<pre>
assign register_data_out[68] =
{
8'b00000000, // 23+8
rb_1pps_counter, // 16+8 bits
gps_1pps_counter, // 8+8 bits
1'b0,
1'b0,
rb_ser_in, // 5
gps_data_in, // 4
gps_aux_out, // 3
gps_aux_in, // 2
rb_1pps_in, // 1
gps_1pps_in // 0
};
</pre>

=== Register 71 packet error bits ===

<pre>
assign register_data_out[71] =
{
8'b00000000, // 24+8 bits
1'b0, // 23
1'b0, // 22
fifo_to_fpga_full_latch, // 21
fifo_from_fpga_full_latch, // 20
gdm_hitmap_fifo_full_latch, // 19
gdm_busy_fifo_full_latch, // 18
cdm_busy_fifo_full_latch, // 17
cdm_hitmap_fifo_full_latch, // 16
1'b0, // 15
1'b0, // 14
1'b0, // 13
gdm_trig48_encode_error_latch, // 12
1'b0, // vx_tx_trig48_pkt8_error_latch, // 11
tsm2_encode_error_latch, // 10
tsm_encode_error_latch, // 9
trg_encode_error_latch, // 8
gdm_hitmap_error_latch, // 7
gdm_hitmap_encode_error_latch, // 6
gdm_busy_encode_error_latch, // 5
cdm_busy_encode_error_latch, // 4
cdm_hitmap_encode_error_latch, // 3
vx_rx_hitmap_error_latch, // 2
1'b0, // vx_tx_tsm_pkt8_error_latch, // 1
1'b0, // vx_tx_trg_pkt8_error_latch // 0
};
</pre>

=== Register 90 VX busy status register ===

<pre>
vx_bsy_status_latched[11:0] <= vx_bsy[11:0];
vx_bsy_status_latched[23:12] <= vx_bsy_collect[11:0];
vx_bsy_status_latched[24] <= cdm_bsy;
vx_bsy_status_latched[25] <= cdm_bsy_collect;
vx_bsy_status_latched[26] <= cdm_veto;
vx_bsy_status_latched[27] <= cdm_veto_collect;
vx_bsy_status_latched[28] <= 0;
vx_bsy_status_latched[29] <= 0;
vx_bsy_status_latched[30] <= 0;
vx_bsy_status_latched[31] <= cdm_bsy_stuck;
</pre>

=== Register 91 VX hitmap status register ===

<pre>
hitmap_status_latched[11:0] <= hitmap_or12[11:0];
hitmap_status_latched[23:12] <= hitmap_or12_collect[11:0];
hitmap_status_latched[24] <= hitmap_grand_or;
hitmap_status_latched[25] <= hitmap_grand_or_collect;
hitmap_status_latched[26] <= 0;
hitmap_status_latched[27] <= 0;
hitmap_status_latched[28] <= 0;
hitmap_status_latched[29] <= 0;
hitmap_status_latched[30] <= 0;
hitmap_status_latched[31] <= hitmap_grand_or_stuck;
</pre>

=== Register 92 GDM busy status register ===

<pre>
gdm_bsy_status_latched[11:0] <= qsfp_rx_bsy[11:0];
gdm_bsy_status_latched[23:12] <= qsfp_rx_bsy_collect[11:0];
gdm_bsy_status_latched[24] <= gdm_bsy;
gdm_bsy_status_latched[25] <= gdm_bsy_collect;
gdm_bsy_status_latched[26] <= gdm_veto;
gdm_bsy_status_latched[27] <= gdm_veto_collect;
gdm_bsy_status_latched[28] <= 0;
gdm_bsy_status_latched[29] <= 0;
gdm_bsy_status_latched[30] <= 0;
gdm_bsy_status_latched[31] <= gdm_bsy_stuck;
</pre>

=== Register 93 GDM hitmap status register ===

<pre>
hitmap_status_latched[11:0] <= hitmap_grand_or_12[11:0];
hitmap_status_latched[23:12] <= hitmap_grand_or_12_collect[11:0];
hitmap_status_latched[24] <= hitmap_grand_or;
hitmap_status_latched[25] <= hitmap_grand_or_collect;
hitmap_status_latched[26] <= 0;
hitmap_status_latched[27] <= 0;
hitmap_status_latched[28] <= 0;
hitmap_status_latched[29] <= 0;
hitmap_status_latched[30] <= 0;
hitmap_status_latched[31] <= hitmap_grand_or_stuck;
</pre>

=== Register 98, 99 GDM TRIG48 test register and TRIG48_CTRL ===

<pre>
assign gdm_trig48_ctrl_bitmap48[31:0] = register_data_in[98][31:0];
assign gdm_trig48_ctrl_bitmap48[47:32] = register_data_in[99][15:0];

assign gdm_trig48_ctrl[15:0] = register_data_in[99][31:16];
</pre>

<pre>
wire enable_grand_or = trig48_ctrl[15];
wire enable_test_trig48 = trig48_ctrl[14];
</pre>

= Firmware registers branch develop_ko =

== Register map ==

<pre>
0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
1 | ALL | ALL | RW | read write scratch register
2 | ALL | CDM | ?? | MGT not used
3 | ALL | CDM | RO | MGT debug_data
4 | ALL | CDM | RW | clk_config_vec
5 | ALL | CDM | ?? | not used
6 | ALL | CDM | RO | CDM_link_data_processing:o_error_time
7 | ALL | CDM | RO | CDM_link_data_processing:o_error_count
</pre>

== register 0 0x80010000 ==

GDM:
<pre>
0 - gdm_link_interface:i_mgt_rst
2 - gdm_link_interface:i_link_down_latched_rst
8 - GDM_link_data_processing:i_rst
10..9 - GDM_link_data_processing:i_data_mode
</pre>

CDM:
<pre>
0 - cdm_link_interface:i_mgt_rst
2 - cdm_link_interface:i_link_down_latched_rst
8 - CDM_link_data_processing:i_rst
10..9 - CDM_link_data_processing:i_data_mode
</pre>

== register 1 0x80010004 ==

GDM:
<pre>
nlinks-1..0 - gdm_link_interface:i_rx_slide_trigger
</pre>

CDM:
<pre>
nlinks-1..0 - cdm_link_interface:i_rx_slide_trigger
</pre>

== register 2 0x80010008 ==

GDM:
<pre>
nlinks-1..0 - gdm_link_interface:o_link_power_good
nlinks+15..16 - gdm_link_interface:o_link_status
</pre>

CDM:
<pre>
nlinks-1..0 - cdm_link_interface:o_link_power_good
nlinks+15..16 - cdm_link_interface:o_link_status
</pre>

== register 3 0x8001000c ==

GDM: simple loopback register

CDM:
<pre>
31..0 - debug_data - cdm_link_interface:o_debug
</pre>

o_debug:
<pre>
rx_link_rst & rx_error & rx_link_up & rx_receiving_data &
std_logic_vector(rx_state_count) & tx_state_count_on_rx_clk & i_rx_ctrl3(0) &
i_rx_ctrl1(1 downto 0) & i_rx_ctrl0(1 downto 0) &
rx_data_is_k28p1_k28p5 &
i_rx_data;
</pre>

== register 4 0x80010010 ==

GDM write:
<pre>
0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS
</pre>

GDM read:
<pre>
0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS
4 - clk_config_vec(4) - CLK_LOSXTn_LS
5 - clk_config_vec(5) - CLK_LOLn_LS
6 - clk_config_vec(6) - CLK_INTn_LS
7 - constant 1
31..8 - constant 0
</pre>

== register 5 0x80010014 ==

not used

== register 6 0x80010018 ==

GDM:
<pre>
3..0 - GDM_link_data_processing:i_status_select
</pre>

CDM:
<pre>
31..0 - CDM_link_data_processing:o_error_count
</pre>

== register 7 0x8001001c ==

GDM:
<pre>
31..0 - GDM_link_data_processing:o_status_vector
</pre>

CDM:
<pre>
31..0 - CDM_link_data_processing:o_error_count
</pre>

= GDM, CDM, VX packet communications =

* timestamp math
<pre>
1 clock is 8 ns is 125 MHz
8 bits of clocks is 256 clocks is 2048 ns is ~2 usec
16 bits of clocks is ~500 usec is 0.5 msec
24 bits of clocks is ~134 msec
32 bits of clocks is ~34 sec
40 bits of clocks is ~8.7 ksec is 2.4 hours
48 bits of clocks is ~625 hours is ~26 days
56 bits of clocks is ~6.6 kdays is ~18 kyears
62 bits of clocks is ~10 Mhours is 427 kdays is ~1.1 kyears
64 bits of clocks is ~4.4 kyears
</pre>
* summary
<pre>
0x01 - not used
0x02 - 8 bytes - 4 words - 2 dwords - 1 qwords - TRG packet
0x03 - 12 bytes - 6 words - 3 dwords - 2 qwords - TRIG48 packet
0x10 - 26 bytes - 13 words - 7 dwords - 4 qwords - TSM packet
0x81 - 10 bytes - 5 words - 3 dwords - 2 qwords - VX_HITMAP packet
0x82 - 108 bytes - 54 words - 27 dwords - 14 qwords - CDM_HITMAP packet
0x83 - 12 bytes - 6 words - 3 dwords - 2 qwords - CDM_BUSY packet
0x84 - 12 bytes - 6 words - 3 dwords - 2 qwords - GDM_BUSY packet
0x85 - 12 bytes - 6 words - 3 dwords - 1 qwords - GDM_HITMAP packet
0x86 - 128 bytes - 64 words - 32 dwords - 16 qwords - CDM CUMULATIVE_HITMAP packet
</pre>
* 0x02 - TRG packet, 8 bytes, 80 adc clocks, 640 ns on lvds link
<pre>
0 - 0x02
1 - trg_counter[7:0]
2 - ts64 low byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 high byte 3
6 - trg_in_latch[7:0]
7 - trg_in_latch[15:8]
</pre>
* 0x03 - TRIG48 packet, 12 bytes, 120 adc clocks, 960 ns on lvds link
<pre>
0 - 0x03
1 - trg_counter[7:0]
2 - ts64 low byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 high byte 3
6 - vx_bitmap[7:0]
7 - vx_bitmap[15:8]
8 - vx_bitmap[23:16]
9 - vx_bitmap[31:24]
10 - vx_bitmap[39:32]
11 - vx_bitmap[47:40]
</pre>
* 0x10 - TSM packet, 26 bytes, 260 adc clocks, 2080 ns on lvds link
<pre>
0 - 0x10
1 - tsm_counter[7:0]
2 - gdm_ts64 low byte 0
3 - 1
4 - 2
5 - 3
6 - 4
7 - 5
8 - 6
9 - gdm_ts64 high byte 7
10 - gps_ts64 low byte 0
11 - 1
12 - 2
13 - 3
14 - 4
15 - 5
16 - 6
17 - gps_ts64 high byte 7
18 - gps_data64 low byte 0
19 - 1
20 - 2
21 - 3
22 - 4
23 - 5
24 - 6
25 - gps_data64 high byte 7
</pre>
* 0x81 - VX hitmap packet, 10 bytes, 100 adc clocks, 800 ns on lvds link, 48 ns on fiber link
<pre>
0 - 0x81
1 - VX ID
2 - hitmap low byte, nits 7:0
3 - 15:8
4 - 23:16
5 - 31:24
6 - ...:32
7 - ...
8 - ...
9 - hitmap low byte, bits 63:...
</pre>
* 0x82 - CDM hitmap packet, 108 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x82
1 - cdm_hitmap_trigger_counter[7:0]
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - cdm_hitmap_or12 byte 0 (7:0)
11 - cdm_hitmap_or12 byte 1 (11:8) plus 4 bits: 12=0, 13=0, 14=0, 15=cdm_hitmap_grand_or
12 - cdm_hitmap_data, low byte, 12*64 bits = 768 bits = 48 words = 96 bytes
...
107 - cdm_hitmap_data, high byte
</pre>
* 0x83 - CDM busy packet, 12 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x83
1 - cdm_busy_packet_counter[7:0], reset at begin of run
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - vx_bsy[7:0]
11 - {0,0,cdm_veto,cdm_bsy,vx_bsy[11:8]}
</pre>
* 0x84 - GDM busy packet, 12 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x84
1 - gdm_busy_packet_counter[7:0], reset at begin of run
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - qsfp_rx_bsy[7:0]
11 - {0,0,gdm_veto,gdm_bsy,qsfp_rx_bsy[11:8]}
</pre>
* 0x85 - GDM hitmap trigger packet, 12 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x85
1 - gdm_hitmap_packet_counter[7:0], reset at begin of run
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - hitmap_grand_or_12[7:0]
11 - {hitmap_grand_or,0,0,0,hitmap_grand_or_12[11:8]}
</pre>
* 0x86 - CDM cumulative hitmap packet, 128 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x86
1 - vx_rx_hitmap_packet_counter[7:0], reset at begin of run
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - reserved
11 - reserved
12 - repeating 12 times the 80-bit VX HITMAP packets: 0x81, VX ID, 64 bits of hitmap
</pre>

= AXI bus timing =

* AXI 100 MHz clock, 10 ns, 32-bit data
* AXI single-dword read: 36 clock repeat rate, 360 ns is 2.777 MHz, 4 bytes per transfer is 11.11 Mbytes/sec
* AXI single-qword read: 13 clock repeat rate, 130 ns is 7.7 MHz, 4 bytes per transfer is 30 Mbytes/sec, not accounting for the gap
* AXI memcpy read: 4 transfers at 13 clocks, gap, 4 transfers at 13 clocks, gap, etc. below 30 Mbytes/sec.
[[Image:Ds-dm-axi-read.png|100px]]
[[Image:Ds-dm-axi-read-64.png|100px]]
[[Image:Ds-dm-axi-read-memcpy.png|100px]]
* AXI single-dword write: 20 clocks repeat rate, 200 ns is 5 MHz, 4 bytes per transfer is 20 Mbytes/sec
* AXI single-qword write: 13+20 clocks repeat rate, 330 ns is 3 MHz, 16 bytes per burst is 48 Mbytes/sec
[[Image:Ds-dm-axi-write.png|100px]]
[[Image:Ds-dm-axi-write-64.png|100px]]

= AXI bus addresses =

* see AXI/AMBA addresses assigned inthe FPGA project: (s_axi/reg0 is the DS-DM AXI registers)
<pre>
daq00:ds-dm-gcdm$ grep assign_bd_address scripts/GDM_CDM_XU8_bd.tcl
assign_bd_address -offset 0x80010000 -range 0x00004000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs axi_register_interfa_0/s_axi/reg0] -force
assign_bd_address -offset 0x000400000000 -range 0x40000000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs ddr4/C0_DDR4_MEMORY_MAP/C0_DDR4_ADDRESS_BLOCK] -force
assign_bd_address -offset 0x80000000 -range 0x00010000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs system_management_wiz/S_AXI_LITE/Reg] -force
daq00:ds-dm-gcdm$
</pre>

* see AXI/AMBA addresses exported from FPGA project to Linux kernel: (uio for debug bridge should say "debug bridge")
<pre>
root@gdm0:~# cat /sys/class/uio/uio*/name
axi-pmon
axi-pmon
axi-pmon
axi-pmon
root@gdm0:~#
root@gdm0:~# ls -l /sys/class/uio/
total 0
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio0 -> ../../devices/platform/amba/ffa00000.perf-monitor/uio/uio0
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio1 -> ../../devices/platform/amba/fd0b0000.perf-monitor/uio/uio1
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio2 -> ../../devices/platform/amba/fd490000.perf-monitor/uio/uio2
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio3 -> ../../devices/platform/amba/ffa10000.perf-monitor/uio/uio3
root@gdm0:~# ls -l /sys/class/uio/../../devices/platform/amba/
total 0
-rw-r--r-- 1 root root 4096 Oct 18 19:37 driver_override
drwxr-xr-x 3 root root 0 Oct 18 01:36 fd070000.memory-controller
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd0b0000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd400000.zynqmp_phy
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd490000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd500000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd510000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd520000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd530000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd540000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd550000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd560000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd570000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd6e0000.cci
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff000000.serial
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff020000.i2c
drwxr-xr-x 6 root root 0 Oct 18 01:36 ff0a0000.gpio
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0b0000.ethernet
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0e0000.ethernet
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0f0000.spi
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff160000.mmc
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff170000.mmc
drwxr-xr-x 3 root root 0 Oct 18 01:36 ff960000.memory-controller
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff9d0000.usb0
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa00000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa10000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa50000.ams
drwxr-xr-x 5 root root 0 Oct 18 01:36 ffa60000.rtc
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa80000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa90000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffaa0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffab0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffac0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffad0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffae0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffaf0000.dma
-r--r--r-- 1 root root 4096 Oct 18 19:37 modalias
lrwxrwxrwx 1 root root 0 Oct 18 19:37 of_node -> ../../../firmware/devicetree/base/amba
drwxr-xr-x 2 root root 0 Oct 18 19:37 power
lrwxrwxrwx 1 root root 0 Oct 18 01:36 subsystem -> ../../../bus/platform
-rw-r--r-- 1 root root 4096 Oct 18 01:36 uevent
root@gdm0:~#
</pre>

= Build firmware =

== Build from git clone ==

THESE ARE K.O.'s NOTES FOR CREATING THE PETALINUX DIRECTORY.

THEY DO NOT WORK!

COPY PETALINUX FROM A WORKING PROJECT AND USE "make gdm" and "make cdm" AS DESCRIBED BELOW.

* git clone git@edev-group.triumf.ca:fw/exp/darkside/gcdm.git
* #Makefile change VIVADO_SETTINGS_SCRIPT := /opt/Xilinx/Vivado/2022.1/settings64.sh
* #. /opt/Xilinx/Vivado/2022.1/settings64.sh
* . /opt/Xilinx/Vivado/2020.2/settings64.sh
* make clean
* make all_from_scratch
* . /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
* make petalinux_create
* make petalinux_rebuild_new_hw_des
* bomb out: The TMPDIR: /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp can't be located on nfs.
* mkdir /tmp/build_tmp
* rm -rf /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp/
* ln -s /tmp/build_tmp /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp
* try again
* grinds, loads a whole bunch of packages...
* finishes with desire to copy things to /tftpboot
* make sdcard_cp_to wants to copy files from PetaLinux_GDM_CDM/images/linux/ to SD card

== Build firmware ==

NOTE: directory Petalinux_GDM_CDM should already exist!

<pre>
#. /opt/Xilinx/Vivado/2020.2/settings64.sh
. /opt/Xilinx/Vivado/2022.2/settings64.sh
. /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
make clean_gdm # remove gdm build tree
make gdm # build or rebuild GDM
make copy_gdm # copy to gdm0
make clean_cdm # remove cdm build tree
make cdm # build or rebuild CDM
make copy_cdm # copy to cdm0 and cdm1
</pre>

copy to SD card:
<pre>
open a root shell
format 16 GB Sd card per above
cd .../ds-dm-gcdm
make copy
</pre>

= build times =

<pre>
CDM 12-june-2023 69aabc1c25130d970bc375aca684bd68849e6685
daq13 AMD-5700G 1688.61user 399.33system 23:28.84elapsed 148%CPU
dsdaqgw AMD-7700 1090.55user 247.34system 16:03.55elapsed 138%CPU
dsdaqgw AMD-7700 CDM incremental 196.68user 67.62system 7:35.42elapsed 58%CPU
dsdaqgw AMD-7700 CDM incremental 684.72user 94.17system 7:30.17elapsed 173%CPU
dsdaqgw AMD-7700 GDM incremental 849.84user 99.79system 9:04.56elapsed 174%CPU
</pre>

= make bootable sd card =

== format the sd card ==

this only needs to be done once

* become root@dsdaqgw
* cd ~olchansk/git/ds-dm-gcdm
* use "lsblk" to identify the SD card (should show as 8/16/32 GB block device)/ /dev/sdd in this case
* make sdcard_format SDCARD_DEVICE=/dev/sdd
* disconnect sd card, reconnect the sd card (to detect new partition tables, etc)

== copy boot files to the sd card ==

* as root: identify partition labels, run "blkid", should say "BOOT", "rootfs" and "data"
* mount
<pre>
mkdir -p /mnt/BOOT
mount -L BOOT /mnt/BOOT
cp PetaLinux_GDM_CDM/images/linux/BOOT_{GDM,CDM}.BIN /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/boot.scr /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/image.ub /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/uboot.env /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/uboot-redund.env /mnt/BOOT/
umount /mnt/BOOT
eject /dev/sdX
</pre>
* FIXME: missing step: cp BOOT_GDM.BIN BOOT.BIN or cp BOOT_CDM.BIN BOOT.BIN

= boot messages =

<pre>
Xilinx Zynq MP First Stage Boot Loader
Release 2020.2 Sep 24 2022 - 13:29:15
NOTICE: ATF running on XCZU4CG/silicon v4/RTL5.1 at 0xfffea000
NOTICE: BL31: v2.2(release):xlnx_rebase_v2.2_2020.3
NOTICE: BL31: Built : 18:02:46, Sep 28 2022

U-Boot 2020.01 (Sep 28 2022 - 18:03:39 +0000)

Model: DarkSide 20k DM
Board: Xilinx ZynqMP
DRAM: 2 GiB
usb dr_mode not found
PMUFW: v1.1
EL Level: EL2
Chip ID: zu4
NAND: 0 MiB
MMC: mmc@ff160000: 0, mmc@ff170000: 1
In: serial@ff000000
Out: serial@ff000000
Err: serial@ff000000
Bootmode: SD_MODE1
Reset reason: SOFT
Net:
ZYNQ GEM: ff0b0000, mdio bus ff0b0000, phyaddr -1, interface rgmii-id

Warning: ethernet@ff0b0000 (eth0) using random MAC address - d6:62:5f:13:00:44
eth0: ethernet@ff0b0000
ZYNQ GEM: ff0e0000, mdio bus ff0e0000, phyaddr -1, interface rgmii-id
Could not get PHY for eth1: addr -1

Hit any key to stop autoboot: 0
ZynqMP>
CTRL-A Z for help | 115200 8N1 | NOR | Minicom 2.7.1 | VT102 | Online 122:2 | ttyACM0
</pre>

= load FPGA from u-boot =

* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/124682257/U-Boot+FPGA+Driver

<pre>
ZynqMP> fpga info
Xilinx Device
Descriptor @ 0x000000007fddb2c0
Family: ZynqMP PL
Interface type: csu_dma configuration interface (ZynqMP)
Device Size: 1 bytes
Cookie: 0x0 (0)
Device name: zu4
Device Function Table @ 0x000000007fda5fe8
PCAP status 0xa0002fde
ZynqMP>
</pre>

* cp CDM_XU8_top.bit /tftpboot/fpga.bit

<pre>
dhcp
tftpb 0x10000000 fpga.bit
fpga loadb 0 0x10000000 ${filesize}
</pre>

<pre>
ZynqMP> dhcp
BOOTP broadcast 1
DHCP client bound to address 192.168.0.100 (1 ms)
*** Warning: no boot file name; using 'C0A80064.img'
Using ethernet@ff0b0000 device
TFTP from server 192.168.0.1; our IP address is 192.168.0.100
Filename 'C0A80064.img'.
Load address: 0x8000000
Loading: *
TFTP error: 'file /tftpboot/C0A80064.img not found for 192.168.0.100' (1)
Not retrying...
ZynqMP> tftpb 0x10000000 fpga.bit
Using ethernet@ff0b0000 device
TFTP from server 192.168.0.1; our IP address is 192.168.0.100
Filename 'fpga.bit'.
Load address: 0x10000000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
############
6.2 MiB/s
done
Bytes transferred = 7797807 (76fc2f hex)
ZynqMP> fpga loadb 0 0x10000000 ${filesize}
design filename = "CDM_XU8_top;UserID=0XFFFFFFFF;Version=2022.2"
part number = "xczu4cg-fbvb900-1-e"
date = "2024/08/14"
time = "14:18:22"
bytes in bitstream = 7797692
zynqmp_align_dma_buffer: Align buffer at 0000000010000073 to 000000000fffff80(swap 0)
ZynqMP>
</pre>

= load FPGA from Linux =

* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841847/Solution+ZynqMP+PL+Programming

== this will reset the CPU ==
<pre>
cp fpga.bit /lib/firmware/
echo fpga.bit > /sys/class/fpga_manager/fpga0/firmware
</pre>

* make .bin file:
<pre>
bootgen -image CDM_XU8_top.bif -arch zynqmp -o ./fpga.bin -w
</pre>
* cat CDM_XU8_top.bif
<pre>
dsdaqgw:ds-dm-gcdm$ cat CDM_XU8_top.bif
all:
{
[destination_device = pl] ./Vivado_CDM_XU8/CDM_XU8.runs/impl_1/CDM_XU8_top.bit
}
dsdaqgw:ds-dm-gcdm$
</pre>

== this will reset the CPU ==

<pre>
cp fpga.bin /lib/firmware/
echo fpga.bin > /sys/class/fpga_manager/fpga0/firmware
</pre>

== this will reset the CPU ==

<pre>
root@dsdm:~# ./fpgautil -b fpga.bin -f Full
</pre>

== this will reset the CPU ==

<pre>
DTSO file from here: https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841847/Solution+ZynqMP+PL+Programming
echo 0 > /sys/class/fpga_manager/fpga0/flags
mount -t configfs configfs /configfs
root@dsdm:~# dtc -O dtb -o fpga.dtbo -b 0 -@ fpga.dtso
root@dsdm:~# cp fpga.dtbo /lib/firmware/
root@dsdm:~# cp fpga.bit /lib/firmware/
root@dsdm:~# rmdir /configfs/device-tree/overlays/fpga
root@dsdm:~# mkdir /configfs/device-tree/overlays/fpga
root@dsdm:~# echo -n "fpga.dtbo" > /configfs/device-tree/overlays/fpga/path
</pre>

= fpgautil =

* https://github.com/Xilinx/meta-xilinx/blob/master/meta-xilinx-core/recipes-bsp/fpga-manager-script/files/fpgautil.c

<pre>
git clone https://github.com/Xilinx/meta-xilinx.git
cd meta-xilinx/meta-xilinx-core/recipes-bsp/fpga-manager-script/files/
scp fpgautil.c root@dsdm:
ssh root@dsdm
make fpgautil
ls -l ./fpgautil
</pre>

<pre>
root@dsdm:~# ls -l ./fpgautil
-rwxr-xr-x 1 root root 72256 Aug 16 00:15 ./fpgautil
root@dsdm:~# ./fpgautil

fpgautil: FPGA Utility for Loading/reading PL Configuration

Usage: fpgautil -b <bin file path> -o <dtbo file path>

Options: -b <binfile> (Bin file path)
-o <dtbofile> (DTBO file path)
-f <flags> Optional: <Bitstream type flags>
f := <Full | Partial >
-n <Fpga region info> FPGA Regions represent FPGA's
and partial reconfiguration
regions of FPGA's in the
Device Tree
Default: <full>
-s <secure flags> Optional: <Secure flags>
s := <AuthDDR | AuthOCM | EnUsrKey | EnDevKey | AuthEnUsrKeyDDR | AuthEnUsrKeyOCM | AuthEnDevKeyDDR | AuthEnDevKeyOCM>
-k <AesKey> Optional: <AES User Key>
-r <Readback> Optional: <file name>
Default: By default Read back contents will be stored in readback.bin file
-t Optional: <Readback Type>
0 - Configuration Register readback
1 - Configuration Data Frames readback
Default: 0 (Configuration register readback)
-R Optional: Remove overlay from a live tree

Examples:
(Load Full bitstream using Overlay)
fpgautil -b top.bit.bin -o can.dtbo -f Full -n full
(Load Partial bitstream using Overlay)
fpgautil -b rm0.bit.bin -o rm0.dtbo -f Partial -n PR0
(Load Full bitstream using sysfs interface)
fpgautil -b top.bit.bin -f Full
(Load Partial bitstream using sysfs interface)
fpgautil -b rm0.bit.bin -f Partial
(Load Authenticated bitstream through the sysfs interface)
fpgautil -b top.bit.bin -f Full -s AuthDDR
(Load Parital Encrypted Userkey bitstream using Overlay)
fpgautil -b top.bit.bin -o pl.dtbo -f Partial -s EnUsrKey -k <32byte key value>
(Read PL Configuration Registers)
fpgautil -b top.bit.bin -r
(Remove Partial Overlay)
fpgautil -R -n PR0
(Remove Full Overlay)
fpgautil -R -n full
Note: fpgautil -R is responsible for only removing the dtbo file from the livetree. it will not remove the PL logic from the FPGA region.

root@dsdm:~#
</pre>

= fw_printenv =

to access u-boot environment from Linux:
* apt install -y libubootenv-tool
* create /etc/fw_env.config
<pre>
/media/BOOT/uboot.env 0 0x40000
/media/BOOT/uboot-redund.env 0 0x40000
</pre>
* if uboot.env files do not exist, run "saveenv" from u-boot command prompt
* fw_printenv and fw_setenv should work

= Boot from network =

== u-boot ==

<pre>
ZynqMP> setenv bootcmd run bootcmd_dhcp
ZynqMP> saveenv
ZynqMP> reset
</pre>

== boot.scr ==

<pre>
# boot.scr
# mkimage -C none -A arm -T script -d boot.scr boot.scr.uimg
echo Loading FPGA!
#tftpb 0x10000000 fpga.bit
tftpb 0x10000000 {ipaddr}.bit
fpga loadb 0 0x10000000 ${filesize}
echo Booting Linux!
run bootcmd_pxe
echo Done!
</pre>

<pre>
mkimage -C none -A arm -T script -d boot.scr boot.scr.uimg
</pre>

== tftpboot ==

<pre>
cp /home/olchansk/git/ds-dm-gcdm/boot.scr.uimg /tftpboot
ln -s /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/images/linux /tftpboot/xilinx-dsdm
mkdir /tftpboot/pxelinux.cfg
cat > /tftpboot/pxelinux.cfg/default-arm-zynqmp <<EOF
LABEL Linux
KERNEL xilinx-dsdm/Image
FDT xilinx-dsdm/system.dtb
#INITRD rootfs.cpio.gz.u-boot
EOF
</pre>

== boot sequence ==

* xilinx magic load BOOT.BIN from SD card
* load FPGA form BOOT.BIN
* load and run u-boot from BOOT.BIN or from image.ub
* u-boot load environment from ??? probably SD card uboot-redund.env, this includes our bootcmd
* run bootcmd which run bootcmd_dhcp which does:
* from /tftpboot:
* load and run boot.scr.uimg which does:
* load FPGA image xilinx-dsdm/${ipaddr}.bit
* run bootcmd_pxe which does:
* load pxelinux.cfg/default-arm-zynqmp which does:
* load xilinx-dsdm/Image ### this is the linux kernel
* load xilinx-dsdm/system.dtb ### this is the device tree
* start linux kernel
* linux kernel does dhcp
* linux kernel does nfs mount /nfsroot/%s,vers=3,tcp ### %s is replaced by the hostname supplied by DHCP
* userland starts and runs to console and ssh login.

= Xilinx ILA =

References:
* https://github.com/Xilinx/XilinxVirtualCable/tree/master
* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/644579329/Xilinx+Virtual+Cable
* https://support.xilinx.com/s/article/974879?language=en_US
* https://docs.xilinx.com/v/u/en-US/pg245-debug-bridge
* https://docs.xilinx.com/r/2020.2-English/ug908-vivado-programming-debugging/Initializing-Vivado-IDE-hw_server
* (we do not use this one) https://github.com/paulscherrerinstitute/xvcSupport/blob/master/README.md

Build xvcserver_cdm.exe: (it is built as a static executable, can be copied and run anywhere)
<pre>
ssh dsdaq@gdm0
cd /home/dsdaq/online/ds-dm-software
git pull ### get latest version
make xvcserver_cdm.exe
ssh root@gdm0
/home/dsdaq/online/ds-dm-software/xvcserver_cdm.exe
INFO: To connect to this xvcServer instance, use url: TCP:gdm0:2542
</pre>

To activate and use the vivado logic analyzer:
* data path: vivado -> hw_server -> xvcserver -> mmap axi bus -> debug bridge -> jtag -> ILA
* define ILAs in the code
* instantiate the xilinx debug bridge at AXI bus address 0x80020000 (FIXME!!! this collides with Ian's AXI addresses)
* build and boot the new FPGA firmware. updating the linux kernel is not necessary.
* login root@gdm0, run: /home/dsdaq/online/ds-dm-software/xvcserver_cdm.exe -v ### with "-v" for the first time to see that vivado does connect to it, without "-v", normally.
* login dsdaqgw, run: hw_server -s tcp:localhost:3121 -e "set auto-open-servers xilinx-xvc:gdm0:2542" ### tells us to connect to port localhost:3121
* login dsdaqgw, run vivado, open project, open hardware manager, open target, open new target, "connect to remote server", hostname "localhost", port "3121", next (bombs, try again, 3 times), popup add virtual cable, enter hostname "gdm0" port "2542", "ok", it shows in "hardware targets", "next", "finish", error popup "[Common 17-163] Missing value for option 'objects', please type 'set_property -help' for usage info", ignore it, in "hardware", right click the "gdm0" one, open target, under "hardware" and "debug bridge" we should see all the ILAs, under "hardware device properties", the "probes file" should have the ".ltx" file generated by vivado "Vivado_GDM_XU8/GDM_XU8.runs/impl_1/debug_nets.ltx", click on an ILA, a waveform should open.

= Software =

* ssh cdm0 # or gdm0
* sudo apt install i2c-tools libi2c-dev
* git clone https://bitbucket.org/team-ds-dm/ds-dm-software
* cd ds-dm-software
* make

== test_cdm.exe ==

=== CDM SFP status ===

<pre>
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0xbb2f0ae7
CDM firmware 0xbb2f0ae7
arg 1: [--sfp]
Polling SFP status...
identifier 0x03
connector 0x07
encoding 0x01
wavelength 0x0352 (850 nm)
vendor_name [FINISAR CORP. ]
vendor_pn [FTLF8526P3BNL ]
vendor_rev [A ]
vendor_sn [N3AB9M8 ]
vendor_date [200319 ]
dm_type 0x68
temp 29.0 C
vcc 3.323 V
tx_bias 7.250 mA
tx_power 478.4 uW
rx_power 2.3 uW
SFP good 1, status: temp 30.6 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 818 uW
...
</pre>

=== GDM QSFP status ===

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --qsfp3 --qsfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x53aee418
CDM firmware 0x53aee418
arg 1: [--qsfp3]
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out hi
arg 2: [--qsfp]
Polling QSFP status...
identifier 0x0d
status 0x02
los 0x8f
temp 28.2 C
vcc 3.323 V
rx_power 0.1 0.1 0.1 0.1 uW
tx_bias 7.6 7.6 7.6 0.0 mA
tx_power 792.2 773.8 823.0 0.1 uW
vendor_name [FINISAR CORP ]
vendor_pn [FTL410QD4C ]
vendor_rev [A ]
wavelength 850
max_temp 70 C
vendor_sn [X79AC0R ]
vendor_date [220309 ]
QSFP good 1, status: temp 27.7 C, los 0x8b, tx_bias 7.5 7.6 7.6 0.0 mA, tx_power 792 772 821 0 uW, rx_power 0 0 466 0 uW
</pre>

=== GDM clock status ===

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
</pre>

* clock chip not loaded, not running:

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x0f 0x16 0x04 0x00 0x01 0x68 0x09 0x00 0x02 0xf2 0x00 0x1f 0xf0 0x22 0xf2, 0x507: 0x00, 0x52A: 0x00, 0x53F: 0x04
Clock chip state 0, status: SYSINCAL XAXB_ERR LOL CAL_PLL IN0 IN_SEL_0 FASTLOCK_STATUS

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x1f7caf52 (528265042) should be ~125 MHz
0x1034 rx_clk: 0x00000000 (0) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x1f7caf52 (528265042) should be ~125 MHz
0x103C tx_clk: 0x00000000 (0) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip good (IN0 - external 10 MHz clock)

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x44 0x00 0xd0 0x01 0x1f 0xff 0x22 0xf2, 0x507: 0x3f, 0x52A: 0x01, 0x53F: 0x00
Clock chip state 1, status: LOS_IN2 OOF_IN2 IN0 IN_SEL_REGCTRL IN_SEL_0

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735a3b (125000251) should be ~125 MHz
0x1034 rx_clk: 0x07735a3c (125000252) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a3b (125000251) should be ~125 MHz
0x103C tx_clk: 0x07735a3c (125000252) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

=== CDM clock status ===

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
</pre>

* clock chip not loaded, not running:

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x0f 0x16 0x04 0x00 0x01 0x68 0x19 0x00 0x02 0xf2 0x00 0x1f 0xf0 0x22 0xf2, 0x507: 0x00, 0x52A: 0x00, 0x53F: 0x04
Clock chip state 0, status: SYSINCAL XAXB_ERR LOL CAL_PLL IN0 IN_SEL_0 FASTLOCK_STATUS

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735851 (124999761) should be ~125 MHz
0x1034 rx_clk: 0x00000000 (0) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x0127fefa (19398394) should be ~125 MHz
0x103C tx_clk: 0x00000000 (0) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip uses internal clock (IN1 - internal oscillator), observe rx_clk frequency is not same as others

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0x7f, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN1 IN_SEL_1 HOLD_HIST_VALID

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735853 (124999763) should be ~125 MHz
0x1034 rx_clk: 0x07735b0a (125000458) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735853 (124999763) should be ~125 MHz
0x103C tx_clk: 0x07735852 (124999762) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip uses SFP recovered clock (IN2 - sfp rx recovered clock), observe mgt_rx_ref_clk_raw (CDM 125 MHz oscillator) is different from others (SFP RX recovered clock)

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0xbf, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN2 IN_SEL_1 HOLD_HIST_VALID
^C
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773581b (124999707) should be ~125 MHz
0x1034 rx_clk: 0x07735ad6 (125000406) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735ad7 (125000407) should be ~125 MHz
0x103C tx_clk: 0x07735ad6 (125000406) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
^C
</pre>

=== CDM link status, PRBS test mode ===

<pre>
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
</pre>
* fiber disconnected, no link:
<pre>
CDM firmware: 0xbb2f0ae7
0x1000 SFP c.c. status: 0x00000031
CLK_IN_SEL_LS 0x1
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x00000025
sfp_mod_absent_N 1
sfp_rx_los_N 0
link_power_good 1
rx_link_up 0
rx_receiving_data 0
rx_error 1
rx_lnk_up_and_running 0
tx_link_up 0
tx_sending_data 0
tx_link_up_and_running 0
link_up_and_running 0
0x1014 SFP link data: 0x466a8187
rx_data 0x8187
k28p1_k28p5 0
rx_ctrl0 0x1
rx_ctrl1 0x1
rx_ctrl3 0x1
tx_state 0x1
rx_state 0x6
rx_receiving_data 0
rx_link_up 0
rx_error 1
rx_link_rst 0
0x2000 link test mode: 0x00000200, seconds: 0x00079093, errors: 0xffffffff
</pre>
* fiber connected, good link:
<pre>
CDM firmware: 0xbb2f0ae7
0x1000 SFP c.c. status: 0x000000b2
CLK_IN_SEL_LS 0x2
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x000007dc
sfp_mod_absent_N 0
sfp_rx_los_N 0
link_power_good 1
rx_link_up 1
rx_receiving_data 1
rx_error 0
rx_lnk_up_and_running 1
tx_link_up 1
tx_sending_data 1
tx_link_up_and_running 1
link_up_and_running 1
0x1014 SFP link data: 0x35c02774
rx_data 0x2774
k28p1_k28p5 0
rx_ctrl0 0x0
rx_ctrl1 0x0
rx_ctrl3 0x0
tx_state 0x3
rx_state 0x5
rx_receiving_data 1
rx_link_up 1
rx_error 0
rx_link_rst 0
0x2000 link test mode: 0x00000200, seconds: 0x00078a8c, errors: 0x00000000
</pre>

=== GDM link status, PRBS test mode ===

<pre>
1 link connected, no errors:
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
GDM firmware: 0x6b2ee010
0x1014: 0x00000008, 18: 0x00000008, 1C: 0x00000000, 24: 0x00000fff
0x2000: 0x00000200, time: 0x00078aa4, errors:
0xffffffff 0xffffffff 0xffffffff
0x00000000 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
</pre>

=== CDM link status ===

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test0

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0xbf, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN2 IN_SEL_1 HOLD_HIST_VALID

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x077357a0 (124999584) should be ~125 MHz
0x1034 rx_clk: 0x07735a5c (125000284) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a5b (125000283) should be ~125 MHz
0x103C tx_clk: 0x07735a5b (125000283) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM firmware: 0x6d2ef81a
0x1000 SFP c.c. status: 0x000000b2
CLK_IN_SEL_LS 0x2
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x000007dc
sfp_mod_absent_N 0
sfp_rx_los_N 0
link_power_good 1
rx_link_up 1
rx_receiving_data 1
rx_error 0
rx_lnk_up_and_running 1
tx_link_up 1
tx_sending_data 1
tx_link_up_and_running 1
link_up_and_running 1
0x1014 SFP link data: 0x35c6bcbc
rx_data 0xbcbc
k28p1_k28p5 0
rx_ctrl0 0x3
rx_ctrl1 0x0
rx_ctrl3 0x0
tx_state 0x3
rx_state 0x5
rx_receiving_data 1
rx_link_up 1
rx_error 0
rx_link_rst 0
0x2000 link test mode: 0x00000000, seconds: 0x00001671, errors: 0xffffffff

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 15
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
reg[15] is 0x0033bcbc (3390652)
</pre>

=== GDM link status ===

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test0

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x44 0x00 0xd0 0x01 0x1f 0xff 0x22 0xf2, 0x507: 0x3f, 0x52A: 0x01, 0x53F: 0x02
Clock chip state 1, status: LOS_IN2 OOF_IN2 IN0 IN_SEL_REGCTRL IN_SEL_0 HOLD_HIST_VALID

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735a0a (125000202) should be ~125 MHz
0x1034 rx_clk: 0x07735a0a (125000202) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a0a (125000202) should be ~125 MHz
0x103C tx_clk: 0x07735a0a (125000202) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly

register 0x1018 bit 0x800

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM firmware: 0x6d2ebce6
0x1014: 0x00000800, 18: 0x00000800, 1C: 0x00000000, 24: 0x00000fff
0x2000: 0x00000000, time: 0x00003d2f, errors:
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff

link data alternates 0xbcbc and 0x1cbc

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 22
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
reg[22] is 0xbcbc93ab (-1128492117)

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 22
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
reg[22] is 0x1cbc1aaf (482089647)
root@gdm0:~#

</pre>

=== Run trg and tsm ===

<pre>
on the GDM:

ssh root@gdm00
./test_cdm.exe --gdm-clocks
./test_cdm.exe --load-cc
./test_cdm.exe --cc
./test_cdm.exe --reset-mgt
./test_cdm.exe --gdm-clocks
./test_cdm.exe --writereg 2 0xff # enable LEMO NIM inputs
./test_cdm.exe --writereg 3 0xba54 # LEDs: lemo1, lemo2, trg, tsm
./test_cdm.exe --writereg 4 0x99 $ # LEMO out is trg_in_pulse
./test_cdm.exe --writereg 9 0x32010 # trg and tsm from trg_pulser and tsm_pulse
./test_cdm.exe --writereg 24 1250000 # trg pulser 100 Hz
./test_cdm.exe --writereg 26 125000000 # tsm pulser 1 Hz
./test_cdm.exe --writereg 23 0x40000000 # route trg_in and tsm_in to qsfp tx bits 0 and 1
./test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz

on the CDM:

ssh root@cdm01
./test_cdm.exe --cdm-clocks
./test_cdm.exe --load-cc
./test_cdm.exe --reset-mgt
./test_cdm.exe --cdm-clocks
./test_cdm.exe --cdm-link # issue --reset-mgt on CDM and GDM until link is good
./test_cdm.exe --writereg 2 0xff # enable LEMO NIM inputs
./test_cdm.exe --writereg 3 0xba54 # enable LEDs: lemo1, lemo2, trg, tsm
./test_cdm.exe --writereg 4 0x99 # enable LEMO output trg_in
./test_cdm.exe --writereg 9 0x8040 # enable trg_in and tsm_in from sfp rx bits 0 and 1
./test_cdm.exe --writereg 7 0xff00 # drive VX LVDS lines to logic level 0
./test_cdm.exe --writereg 8 0x2 # VX LVDS with trg and tsm, misrouted in second VX
#./test_cdm.exe --writereg 8 0x7 # VX LVDS with tsm
./test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz
</pre>

switch GDM and CDM to packetizer trg and tsm:

<pre>
on the GDM:

/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x00000000

on the CDM:

/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x0804
/home/dsdaq/online/ds-dm-software/test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz
</pre>

=== Run packet loopback ===

GDM CPU -> fifo_to_fpga -> GDM QSFP -> CDM SFP -> fifo_from_fpga -> CDM CPU

On the GDM: (CDM is connected to first QSFP port)

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --load-cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --reset-mgt
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 32 1 ### tell GDM to use first QSFP port
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link ### confirm link status is "3"
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 29 2 ### enable GDM packet data injection
/home/dsdaq/online/ds-dm-software/test_cdm.exe --test-fifo-write-loop
</pre>

One the CDM:

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --load-cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --reset-mgt
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 29 3 ### enable CDM packet data injection
/home/dsdaq/online/ds-dm-software/test_cdm.exe --test-fifo-read
</pre>

== fecdm ==

=== Power up and clock management ===

* Clock power up and setup management is handled by fecdm state machine
* GDM clock setup sequence:
<pre>
- on power up, CC is not initialized, XAXB clock is not running
- on GDM frontend start (in HandleInit()), CC is loaded and reset
- CC will lock to IN0 (10 MHz external clock) or IN1 (125 MHz internal oscillator)
- if locked to IN0, CC will enter and exit the HOLDOVER mode if 10 MHz clock is unstable
</pre>
* CDM clock setup sequence:
<pre>
- on power up, CC is not initialized, XAXB clock is not running
- fecdm detects this, loads the CC configuration registers, resets the CC
- sets the CC input IN1, DSDM internal oscillator
- if SFP is in good state, SFP MGT is running, CC is happy with IN2 input (no IN2_OOF, IN2_LOS), SFP recpvered clock (clk_rx_mgt) frequency is close to 125 MHz:
- switches CC to input IN2 (SFP recovered clock, clk_rx_mgt)
- waits for CC locked to IN2 (no LOL, no HOLD). (this provides stable SFP tx clock).
- resets the SFP MGT, requests GDM MGT reset (via ODB)
- MGT reset sequence: GDM MGT reset 1, CDMnn MGT reset 1, GDM MGT reset 0, CDM MGT reset 0
- CDM MGT rx_state cycles from 0 to 5. if it goes to 6 or never reaches 5, GDM MGT reset is requested again.
- during this sequence, clk_rx_mgt may wobble, CC may enter HOLDOVER mode, VX clock should stay steady.
- once CC is locked to IN2, SFP MGT rx_state is 5, link is ready for use.
</pre>
* Request changes for Ian
<pre>
17 Apr 2025:
- CDM SFP MGT rx_state is getting stuck in 4 (waiting for alternating idle chars), do we need this? go to state 5 without waiting?
- GDM QSFP MGT reset 1 does not seem to do anything, change it to cause all CDM MGTs to enter an error state, e.g. start transmitting invalid K-codes. transmitting all 0 or all 1 will break the recovered clock, so maybe do not do that.
- having all CDM SFP MGTs go to error state immediatley after GDM QSFT MGT reset 1 will greatly simplify the link reset sequence.
</pre>

=== Trigger configuration ===

* GDM
* CDM in SFP trigger mode:
** SFP K-code encoded sfp_rx_trg, sfp_rx_tsm are routed to LVDS vx_tx_trg and vx_tx_tsm
** GDM TRG, TSM (and all other) packets are routed to LVDS serial data to VX
** to make counters count and lights blink:
** CDM TRG and TSM trigger mask is set to sfp_rx_trg, sfp_rx_tsm and vx_tx_trg_done, vx_tx_tsm_done
** CDM LEDs display trg_in_pulse, tsm_in_pulse
** output of TRG and TSM packets is disabled (they come from the GDM)

= dsvslice integration =

== VX setup ==

* general
** Start acq from user code = y, all others = n (as of Mar 2023: will start when first trigger received)
** Use NIM IO = y
** Use external clock = y
** LVDS quartet is input = n, y, n, y
** LVDS quartet mode = User, User, User, User

* trigger from front panel NIM:
** Trigger on external signal = y, all others = n
** connect CDM EXT_OUT(2) to VX "TrigIn"

* trigger from LVDS "Sync" mode
** Trigger on LVDS Sync signal = y, all others = n
** LVDS quartet mode = User, Sync, User, User

* trigger from LVDS "User" mode
** Trigger on LVDS pair 12 signal = y, all others = n
** LVDS quartet is input = n, y, n, y
** LVDS quartet mode = User, User, User, User

== GDM setup ==

* GDM is gdm0
* set inputs to NIM mode
* set outputs to TTL mode (this GDM has wrong NIM output circuit)
* use top QSFP slot, connect split cable 1 into CDM01, cable 2 into CDM02
* connect non-inverted NIM trigger signal to top-LEMO-left EXT_IN_LV(1)
* connect non-inverted NIM TSM signal to top-LEME-right EXT_IN_LV(2)
* GDM LEDs: TRIG, TSM, trigger enabled, trigger_out
* GDM LEMO_OUT: trigger, trigger

== CDM setup ==

* set CDM LEMO inputs to NIM
* set CDM LEMO outputs to NIM
* CDM01 is cdm0
* CDM02 is cdm1
* connect GDM fiber links to SFP port
* connect 1st VX port of CDM01 to VX1
* connect 1st VX port of CDM02 to VX2
* connect LEMO EXT_OUT(2) to VX "TrigIn", CDM01 to VX1, CDM02 to VX2
* power up
* CDM LEDs: GDM TRIG, GDM TSM, trigger enabled, trigger out
* CDM LEMO_OUT: gdm_trg, trigger

== After power up ==

* start the CDM frontend from the MIDAS "Programs" page. To start manually, see the Start Command on the Programs page.
* CDM frontend should enable the VX clock, disable the trigger
* from the MIDAS status page, goto the CDM page
* outdated: in the CDMx data tables, the 2nd number should read 0x35c08008, if it does not and the last 4 digits randomly change, reset the GDM links
*if the FEs complain - do in order: for GDM, CDM01, CDM02, ..., press "reset mgt" of each board, then press "unreset mgt", if it does not help, STOP HERE
* start a run
* CDM frontend will enable the trigger
* GDM frontend will enable the trigger
* LEDs on the GDM should flash, LEDs on the CDM should flash, TrigIn and TrigOut of the VX should flash
* stop a run
* GDM frontend will disable the trigger
* CDM frontend will disable the trigger

== Phase measurement ==

* pip3 install matplotlib
* pip3 install scipy
* export PYTHONPATH=$HOME/packages/midas/python
* #git clone https://github.com/J033X071C/PhaseMeasurement
* git clone https://bitbucket.org/team-ds-dm/phasemeasurement.git
* cd phasemeasurement
* python3 ./phaseMeasurement.py --help
<pre>
daq00:PhaseMeasurement$ python3 ./phaseMeasurement.py --help
usage: phaseMeasurement.py [-h] fileName numberEvents numberVX sizeEvents stopEvent minHist maxHist numberBin writeToTXT saveAsPDF

Read data from midas file (in .lz4 format) to calculate phase between the clock of VX1 and VX2

positional arguments:
fileName Name of the file we want to read data from (Example: run00389.mid.lz4)
numberEvents Number of events recorded in the file
numberVX Number of VX used in this run (usually 2...)
sizeEvents Number of points per event
stopEvent Number of events you want to go through to calculate phase
minHist Minimal value for the x axis of the phase measurement histogram (in ns)
maxHist Maximal value for the x axis of the phase measurement histogram (in ns)
numberBin Number of bins wanted for the generated histogram
writeToTXT Write argument as yes to generate text file with results of calculation
saveAsPDF Save generated plots to PDF files

optional arguments:
-h, --help show this help message and exit
daq00:PhaseMeasurement$
</pre>
* try an old file with
* python3 ./phaseMeasurement.py run00877.mid.lz4 10000 2 10000 500 -20 20 81 yes yes
<pre>
bin size = 0.494 ns
num_events = 347
mean = -1.705 ns
rms = 3.087 ns
mean_error = 0.166 ns
centroid = -1.706 ns.
width (sigma) = 0.363 ns.
error on the centroid = 0.016558 ns.
</pre>
* ls -l *.txt *.pdf
<pre>
dsdaq@dsvslice:~/online/PhaseMeasurement$ ls -l *.txt *.pdf
-rw-rw-r-- 1 dsdaq dsdaq 64728 Dec 14 16:56 run00877.mid.lz4_Plots.pdf
-rw-rw-r-- 1 dsdaq dsdaq 274 Dec 14 16:56 run00877.mid.lz4.txt
dsdaq@dsvslice:~/online/PhaseMeasurement$
</pre>
* scope settings (from email message)
<pre>
From fcote-lortie@triumf.ca Thu Dec 15 17:11:46 2022
From: Francis Cote-Lortie <fcote-lortie@triumf.ca>
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: Re: How to use scope as a waveform generator
Date: Fri, 16 Dec 2022 01:11:44 +0000

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display) by pressing on the button. It will go from 0 to 1.

The settings that we are using right now are:
Type of waveform: Sine wave
Offset: 0 V
Amplitude: 1 Vpp
Frequency: 50 kHz
Noise: 0 V
________________________________
From: Francis Cote-Lortie <fcote-lortie@triumf.ca>
Sent: Thursday, December 15, 2022 4:58 PM
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: Re: How to use scope as a waveform generator

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display)

The settings that we are using right now are:
Type of waveform: Sine wave
Offset: 0 V
Amplitude: 1 Vpp
Frequency: 50 kHz
Noise: 0 V
________________________________
From: Francis Cote-Lortie
Sent: Thursday, December 15, 2022 4:54 PM
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: How to use scope as a waveform generator

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display)
</pre>

== Standalone link test ==

<pre>
CDM: program clock chip
busybox devmem 0x80011000 32 0x8
busybox devmem 0x80011000 32 0x0
/home/dsdaq/si5394-i2c-file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b

GDM, CDM: link reset
busybox devmem 0x80011008 32 1

GDM, CDM: release reset
busybox devmem 0x80011008 32 0

CDM: link status (NOTE: SFP LOS and mod_absent are swapped!!!)
busybox devmem 0x80011010 32
0x00000024 <- fiber plugged
0x00000025 <- fiber unplugged
0x00000027 <- SFP unplugged
0x000007DC <- successful link with GDM

CDM: link state machine and data
busybox devmem 0x80011014 32
0x35C06FF6

CDM: set link to counting mode
busybox devmem 0x80012000 32 0x101
busybox devmem 0x80012000 32 0x100

CDM: time counter and error counter
root@cdm1:~# busybox devmem 0x80012000 32
0x00000100 <--- link mode
root@cdm1:~# busybox devmem 0x80012004 32
0x0000058C <--- seconds counter
root@cdm1:~# busybox devmem 0x80012004 32
0x0000058D
root@cdm1:~# busybox devmem 0x80012008 32
0x00000000 <--- error counter

GDM: no link
root@gdm0:~# busybox devmem 0x80011014 32
0x00000000
root@gdm0:~# busybox devmem 0x80011018 32
0x00000000
root@gdm0:~# busybox devmem 0x8001101c 32
0x00000000
root@gdm0:~# busybox devmem 0x80011024 32
0x00000FFF
root@gdm0:~#

GDM: good link channel 10, counting mode
root@gdm0:~# busybox devmem 0x80012000 32 0x101
root@gdm0:~# busybox devmem 0x80012000 32 0x100
root@gdm0:~# busybox devmem 0x80012008 32
0x3A8B68C2
root@gdm0:~# busybox devmem 0x80012008 32
0x42E03BEF
root@gdm0:~# busybox devmem 0x8001200c 32
0xDA090972
root@gdm0:~# busybox devmem 0x8001200c 32
0xDE6F22E9
root@gdm0:~# busybox devmem 0x80012019 32
Bus error
root@gdm0:~# busybox devmem 0x80012010 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012014 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012018 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x8001201c 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012020 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012024 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012028 32
0x00000000
root@gdm0:~# busybox devmem 0x8001202c 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012030 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012034 32
0xFFFFFFFF
root@gdm0:~#
</pre>

mapping of link channels:

<pre>
qsfp0 lane0 - 0x0100 - link 8
qsfp0 lane1 - 0x0200 - link 9
qsfp0 lane2 - 0x0400 - link 10
qsfp0 lane3 - n/c
qsfp1 lane0 - 0x0800 - link 11
qsfp1 lane1 - 0x0010 - link 4
qsfp1 lane2 - 0x0020 - link 5
qsfp1 lane3 - n/c
qsfp2 lane0 - 0x0040 - link 6
qsfp2 lane1 - 0x0080 - link 7
qsfp2 lane2 - 0x0001 - link 0
qsfp2 lane3 - n/c
qsfp3 lane0 - 0x0002 - link 1
qsfp3 lane1 - 0x0004 - link 2
qsfp3 lane2 - 0x0008 - link 3
qsfp3 lane3 - n/c
</pre>

script to start the test with 2 CDMs:

<pre>
ssh dsdaq@dsvslice
ssh root@gdm0 busybox devmem 0x80011008 32 1
ssh root@cdm0 busybox devmem 0x80011008 32 1
ssh root@cdm1 busybox devmem 0x80011008 32 1
ssh root@gdm0 busybox devmem 0x80011008 32 0
ssh root@cdm1 busybox devmem 0x80011008 32 0
ssh root@cdm0 busybox devmem 0x80011008 32 0
ssh root@gdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
ssh root@cdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
ssh root@cdm1 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
</pre>

== LEMO trigger GDM to CDM to VX ==

on the GDM:

<pre>
/home/dsdaq/si5394-i2c-file /home/dsdaq/GDM_v1.0_IN0_EXT1_and_IN1_fixed_Si5394-RevA-Registers.txt 0 0x6b
busybox devmem 0x80011008 32 0x1
busybox devmem 0x80011008 32 0x0
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x80001230 # QSFP TX fixed pattern
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 2 0xff # enable LEMO inputs
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 3 0x7654 # enable LED, one per LEMO input
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x0F0F # enable LEMO to trg_in and tsm_in
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x40001230 # enable trg_in and tsm_in output to QSFP
</pre>

on the CDM:

<pre>
/home/dsdaq/si5394-i2c-file /home/dsdaq/CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
/home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp
busybox devmem 0x80011008 32 1
busybox devmem 0x80011008 32 0
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x8040 # trg_in from sfp[0], tsm_in form sfp[1]
/home/dsdaq/online/ds-dm-software/test_cdm.exe 11 # trg_in counter
/home/dsdaq/online/ds-dm-software/test_cdm.exe 12 # tsm_in counter
</pre>

== GPS receiver VCL-2705 ==

* Valiant VCL-2705 GPS receiver
* https://www.valiantcom.com/time-distribution/gps-receiver-irig-b.html
* USB connection is /dev/ttyACM0, 115200 bps, "GNSSAUX" prompt
* minicom -D /dev/ttyACM0 -b 115200
* user manual download instructoins - see sheet of paper with user name and password in the shipping box
* usb commands:
<pre>
gnss-help
gnss-showver -> F/W: Ver 1.7 Feb 17 2020 16:20:43

gnss-showselftest -> no antenna connected
Overall : FAIL
EPROM Test: PASS
Antenna : NOT DETECTED
GNSS : COMMUNICATION OK

gnss-showsettings
GNSS NMEA BAUDRATE :115200
GNSS ANTENNA LENGTH :30 meters
GNSS USER CONFIGURED DELAY:-65 nanoseconds
GNSS 1PPS PULSE WIDTH :200 milliseconds
GNSS MODE :GPS
GNSS STATUS :STATIONARY

GNSSAUX> gnss-showserial
SERIAL :2704H01V17MAX310

GNSSAUX> gnss-showmode
GNSS MODE : GPS

GNSSAUX> gnss-showstatus
GNSS STATUS: STATIONARY

GNSSAUX> gnss-showalarms
CURRENT ALARMS GNSS
ANTENNA : NOT DETECTED

GNSSAUX> gnss-showerrors
CURRENT ALARMS GNSS
ANTENNA : **NOT DETECTED
ERROR STATISTICS GNSS
RMC GOOD DURATION : Secs 0
RMC BAD DURATION : Secs 0
LOCK GOOD SECS DURATION : Secs 0
LOCK BAD SECS DURATION : Secs 0
SATINFO GOOD ITERATIONS :0
SATINFO REJECT ITERATIONS :0
SATINFO NOTALKER ITERATIONS :0
CURRENT MONITOR STATE GNSS
STATE :Phase-1 HUNTING ANTENNA DETECT

GNSSAUX> gnss-showsatinfo
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-showsats
Total Sats: 0

GNSSAUX> gnss-showmyloc
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-show1ppsstate
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-showjamstatus
Not Available !

GNSSAUX> gnss-showspoofstatus
Not Available !

--- antenna connected, can see the sky ---

GNSSAUX> gnss-showselftest
Overall : PASS
EPROM Test: PASS
Antenna : DETECTED
GNSS : COMMUNICATION OK

GNSSAUX> gnss-showalarms
CURRENT ALARMS GNSS
ANTENNA : DETECTED
GNSS LOCK : AVAILABLE

GNSSAUX> gnss-showerrors
CURRENT ALARMS GNSS
ANTENNA : DETECTED
GNSS LOCK : AVAILABLE
ERROR STATISTICS GNSS
RMC GOOD DURATION : Mins 1,Secs 34
RMC BAD DURATION : Secs 55
LOCK GOOD SECS DURATION : Mins 1,Secs 34
LOCK BAD SECS DURATION : Secs 55
SATINFO GOOD ITERATIONS :3
SATINFO REJECT ITERATIONS :0
SATINFO NOTALKER ITERATIONS :0
CURRENT MONITOR STATE GNSS
STATE :Phase-4 NORMAL OPERATION, Monitoring GNSSLOCK

GNSSAUX> gnss-showsatinfo
GNSS MODE :GPS
SATELLITE INFORMATION TALKER:GPS
NMEA ID : 01-32
NO OF SATELLITES IN VIEW: 08
NO OF XXGSV MSGS : 03
SatNo PRN NO (SV ID) ELEVATION (degs) AZIMUTH (degs) C/No (SNR)
1 0 0 0 0
2 0 0 0 0
3 0 0 0 0
4 0 0 0 0
5 0 0 0 0
6 0 0 0 0
7 0 0 0 0
8 0 0 0 0
<<<<<< End of Sat Info >>>>>>

GNSSAUX> gnss-showsats
Total Sats: 8

GNSSAUX> gnss-showmyloc
GNSS RECEIVER LOCATION:
Latitude : 4914.81911
Longitude: 12313.69595

GNSSAUX> gnss-show1ppsstate
GPS 1PPS STATE: LOCKED

GNSSAUX> gnss-showmyloc

GNSS RECEIVER LOCATION:
Latitude : 4914.80688
Longitude: 12313.69531

enter into google maps search box as: 49 14.80688, -123 13.69531, observe the space, the moved dot and the minus.

GNSSAUX> gnss-resetgnss

Executing....Please Wait....
$$$$$END
</pre>

* 1PPS BNC output: period 1 sec, pulse width 200 ms, 3.3V into 1MOhm, 1.38V into 50Ohm.
* IRIG-B BNC output: 5.6V into 1MOhm, 2.4V into 50Ohm.
* IRIG-B format selector: default is all up.

== Rb clock PRS10 ==

* https://www.thinksrs.com/products/prs10.html
* 10 MHz output is sine wave around 5V peak to peak
* 1PPS BNC output is 10 usec pulse, 5V into 1MHohm.
* RS232 connection: minicom -D /dev/ttyUSB0 -b 9600
* ser2net config: localhost,3001:raw:600:usb-5-2-1.0:9600 -XONXOFF -RTSCTS LOCAL
* ssh daq13, cd /home/olchansk/git/ser2net, ./ser2net -c ~/daq/ds/ser2net.conf -d
* ssh daq13, cd ~/daq/ds, python3 prs10.py
* RS232 commands:
<pre>
ID?
PRS10_3.56_SN_105719
VB1
SN?
RS1 -- reset
ST? -- status
FC? -- 10MHz OCXO drive voltage DAC settings
DS? -- "detected signals"
GA? -- gain of frequency lock loop between ovenized oscillator and Rb cell, 0=use ovenized oscillator only
MO? -- magnetic offset of the Rb cell, range 2300..3600, if out of range, unit must be set to different operating mode, see prs10m.pdf
MR? -- magnetic read
TT? -- time-tag, time in ns between 1PPS out and 1PPS in
TS? -- time slope, ???
TO? -- time offset, ???
PS? -- pulse slope, ???
PL? -- 0=phase lock off, 1=phase lock on, lock to 1PPS input
PT? -- phase lock integrator time constant, PT8 is integrator time constant 18.2 hours, natural time constant 2.25 hours
PF? -- phase lock stability factor, PF2 is "1"
PI? -- phase lock integrator

Analog to digital 12 bit ADC, values 0.000 to 4.998

AD0? -- Spare (J204)
AD1? -- +24V(heater supply) divided by 10.
AD2? -- +24V(electronics supply) divided by 10
AD3? -- Drain voltage to lamp FET divided by 10
AD4? -- Gate voltage to lamp FET divided by 10
AD5? -- Crystal heater control voltage
AD6? -- Resonance cell heater control voltage
AD7? -- Discharge lamp heater control voltage
AD8? -- Amplified ac photosignal
AD9? -- Photocell’s I/V converter voltage divided by 4
AD10? -- Case temperature (10 mV/°C)
AD11? -- Crystal thermistors
AD12? -- Cell thermistors
AD13? -- Lamp thermistors
AD14? -- Frequency calibration pot / external calibration voltage
AD15? -- Analog ground

A/D via CPU E-port:

AD16? -- Varactor voltage for 22.48 MHz VCXO (inside RF synthesizer) / 4
AD17? -- Varactor voltage for 360 MHz VCO (output of RF synthesizer) / 4
AD18? -- Gain control voltage for amplifier which drives frequency multiplier / 4
AD19? -- RF synthesizer’s lock indicator voltage (nominally 4.8 V when locked )

ST?

ST1 : Power supplies and Discharge Lamp
ST1 bit, Condition which sets bit, Corrective Action
0 -- +24 for electronic < +22 Vdc
1 -- +24 for electronics > +30 Vdc
2 -- +24 for heaters <+22 Vdc
3 -- +24 for heaters > +30 Vdc
4 -- Lamp light level too low
5 -- Lamp light level too high
6 -- Gate voltage too low
7 -- Gate voltage too high

ST2: RF Synthesizer
ST2 bit, Condition which sets bit, Corrective Action
0 -- RF synthesizer PLL unlocked
1 -- RF crystal varactor too low
2 -- RF crystal varactor too high
3 -- RF VCO control too low
4 -- RF VCO control too high
5 -- RF AGC control too low
6 -- RF AGC control too high
7 -- Bad PLL parameter

ST3: Temperature Controllers
ST3 bit, Condition which sets bit
0 -- Lamp temp below set point
1 -- Lamp temp above set point
2 -- Crystal temp below set point
3 -- Crystal temp above set point
4 -- Cell temp below set point
5 -- Cell temp above set point
6 -- Case temperature too low
7 -- Case temperature too high

ST4: Frequency Lock-Loop Control
ST4 bit, Condition which sets bit
0 -- Frequency lock control is off
1 -- Frequency lock is disabled
2 -- 10 MHz EFC is too high
3 -- 10 MHz EFC is too low
4 -- Analog cal voltage > 4.9 V
5 -- Analog cal voltage < 0.1
6 -- not used
7 -- not used

ST5: Frequency Lock to External 1pps
ST5 bit, Condition which sets bit
0 -- PLL disabled
1 -- < 256 good 1pps inputs
2 -- PLL active
3 -- > 256 bad 1pps inputs
4 -- Excessive time interval
5 -- PLL restarted
6 -- f control saturated
7 -- No 1pps input

ST6: System Level Events
ST6 bit and Condition which sets bit
0 Lamp restart
1 Watchdog time-out and reset
2 Bad interrupt vector
3 EEPROM write failure
4 EEPROM data corruption
5 Bad command syntax
6 Bad command parameter
7 Unit has been reset
</pre>
* ST? on warm start
<pre>
received: PRS_10
received: 255,255,255,243,34,255
received: 0,0,0,1,34,0
...
received: 0,0,0,1,34,0
received: 0,0,0,0,2,0
...
received: 0,0,0,0,2,0
received: 0,0,0,0,4,0
</pre>
* ST? on loss of external 1PPS
<pre>
...
received: 0,0,0,0,4,0
disconnect 1PPS input
received: 0,0,0,0,132,0
...
reconnect 1PPS input
received: 0,0,0,0,132,0
received: 0,0,0,0,4,0
...
</pre>
* ST? on coldish start
<pre>
daq13:ds$ python3 prs10.py
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,0,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 0,0,1,1,34,0 ] old [ 80,0,0,1,34,1 ] counter: 0
received [ 0,0,0,1,34,0 ] old [ 0,0,1,1,34,0 ] counter: 0
received [ 0,0,16,1,34,0 ] old [ 0,0,0,1,34,0 ] counter: 6
received [ 0,0,20,1,34,0 ] old [ 0,0,16,1,34,0 ] counter: 5
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 8
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 3
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 49
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 250
...
</pre>
* ST? on cold start, note: no bump in the 10MHz clock as reported by DS-DM clock chip
<pre>
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,21,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 64,0,21,1,34,0 ] old [ 80,0,21,1,34,1 ] counter: 0
received [ 0,0,21,1,34,0 ] old [ 64,0,21,1,34,0 ] counter: 0
received [ 0,0,20,1,34,0 ] old [ 0,0,21,1,34,0 ] counter: 158
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 23
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 47
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 40
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 249
</pre>

= det fac integration test =

Connections:
* GPS receiver "IRIG-B SEL" both switches "up" - both "on", IRIG-B format B004
* GPS receiver USB-B -> long cable -> daq13 USB-A
* GPS receiver "1PPS out" -> long BNC cable -> BNC-T -> scope (5V, no 50ohm) and Rb clock BNC "1PPS in"
* GPS receiver "IRIG-B 50ohms" -> long BNC cable -> BNC-T -> scope (5V, no 50ohm) and DS-DM LEMO input 1 (TTL mode)
* Rb clock RS232 -> RS232 straight cable -> RS232-to-USB adapter -> daq13 USB-A
* Rb clock "1PPS out" BNC -> scope (5V, no 50ohm, trig threshold rising edge 2V)
* Rb clock "10MHz output 50 Ohm" BNC -> lemo -> lemo-T -> scope (sine wave, 5V, no 50ohm) and DS-DM clock input.

Programs to run:
* on daq13: cd /home/olchansk/git/ser2net, ./ser2net -c ~/daq/ds/ser2net.conf -d
<pre>
localhost,3001:raw:600:usb-5-2-1.0:9600 -XONXOFF -RTSCTS LOCAL
</pre>
* on daq13: cd /home/olchansk/daq/ds, python3 prs10.py ### connects to ser2net
<pre>
daq13:ds$ python3 prs10.py
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,21,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 64,0,21,1,34,0 ] old [ 80,0,21,1,34,1 ] counter: 0
received [ 0,0,21,1,34,0 ] old [ 64,0,21,1,34,0 ] counter: 0
received [ 0,0,20,1,34,0 ] old [ 0,0,21,1,34,0 ] counter: 158
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 23
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 47
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 40
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 249
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 12096
received [ 0,0,0,0,20,0 ] old [ 0,0,0,0,132,0 ] counter: 2
received [ 0,0,0,0,148,0 ] old [ 0,0,0,0,20,0 ] counter: 0
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,148,0 ] counter: 575
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 11449
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 2755
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 34386
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 41035
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 113401
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 33375
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 54767
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 85059
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 33222
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 119234
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 121990
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 128184
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 56002
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 428237
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 8250
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 1388
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 30506
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 142704
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 179451
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 106182
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 68747
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 65424
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 157587
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 6932
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 1388
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 20255
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 4
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 225941
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 72183
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 26970
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 1
39287
</pre>
* on ds-dm: ./test_cdm.exe --irigb ### note sbs mismatch is because I should wrap around at 16 bits
<pre>
dataframe: S00000000S100001010S...S001010101S000001000S, sec: 00, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16724 should be 82260
dataframe: S10000000S100001010S...S101010101S000001000S, sec: 01, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16725 should be 82261
dataframe: S01000000S100001010S...S011010101S000001000S, sec: 02, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16726 should be 82262
dataframe: S11000000S100001010S...S111010101S000001000S, sec: 03, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16727 should be 82263
dataframe: S00100000S100001010S...S000110101S000001000S, sec: 04, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16728 should be 82264
dataframe: S10100000S100001010S...S100110101S000001000S, sec: 05, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16729 should be 82265
</pre>

= DS-IOGC GPS interface board =

* Rev0 git repository: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0/-/tree/main?ref_type=heads
* Rev0 schematics: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0/-/blob/main/Altium/Project%20Outputs%20for%20DS-IOGC-Rev0/SCH-DS-IOGC-Rev0.pdf?ref_type=heads
* Rev0 schematics: [[:Image:SCH-DS-IOGC-Rev0.pdf|SCH-DS-IOGC-Rev0]]
* Rev1 git repository: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev1/
* Rev1 schematics: [[:Image:SCH-DS-IOGC-Rev1.pdf|SCH-DS-IOGC-Rev1]]

== Changes Rev0 to Rev1 ==

<pre>
From: Peter Margetak <pmargetak@triumf.ca>
Subject: IOGC REV1 review
Date: Wed, 4 Sep 2024 07:31:19 +0000

Hi Konstantin,
Pls have a look at SCH for new rev. I'd like to send it to mfr next week so if you can comment by early next week. Meanwhile I work on layout and other stuff.

Changes:

New ICs - all powered +5V
U20 - inverters for RUclk RX/TX
U21 - non inverting line driver for RU-1pps-out (so you don't have to route if via GDM to see it on scope)
U22 - non inverting buffer for ext 1pps input

All Lemo connectors have the same position but they are double lemos now => new panel needed
@Marek Walczak<mailto:mwalczak@triumf.ca> you can print it ahead once pcb is done + update IOGC docs and panel description

J2A/B - Test ports for RU-1pps in and out
J5A/B - inputs for external GPS data and External source of 1pps
J6A/B - aux in/out for GDM

SW1 - no change - select RX/TX USB/GDM
SW2 - select latch sensitivity for rising/falling edge
SW3 - select source of GPS data (opto or ext) AND select source of 1pps input (latch or ext)

p.
</pre>

== PRS-10 Rb clock device ==

The Rb clock PRS-10 device provides these connections:
<pre>
RS232 RX input - serial communication, non-standard RS232
RS232 TX output - serial communication, non-standard RS232
10 MHz clock output - coax 50 Ohm high resolution 10 MHz clock
1pps output - 1 Hz clock corresponding to the 10 MHz clock
1pps input - 1pps signal from GPS receiver
</pre>

Mode of operation:
* 10 MHz clock is always running
* 1pps output is always running
* if 1pps input from GPS received is present, after 256 pulses PRS-10 will sync it's 1pps output with the 1pps input by adjusting the frequency of the 10 MHz clock
* when unlocked: 1pps output and 1pps input unrelated
* when locked to GPS: 1pps output and 1pps input always go up and down at the same time

Theory of operation:
* 10 MHz clock is produced by a high-quality crystal (stable on the scale of seconds)
* crystal oscillator is synchronized to a Rb cell (stable on the scale of hours and days)
* Rb cell resonant frequency is synchronized to the GPS 1pps signal (stable on the scale of months and years)

== Rev1 connections ==

* LEMO connectors (front panel)
<pre>
LEMO J2A output - Rb clock 1pps in monitor
LEMO J2B output - Rb clock 1pps out monitor
LEMO J5A input - GPS IRIG-B from GPS receiver to FPGA (VCL-2705)
LEMO J5B input - GPS 1pps from GPS receiver to PRS-10 (VCL-2705)
LEMO J6A output - AUX-OUT from FPGA (dual-LEMO PCB-side)
LEMO J6B input - AUX-IN to FPGA (dual-LEMO away-from -PCB)
</pre>

* SMB connectors (back)
<pre>
SMB J3 output - GPS 1pps loopback to LNGS
SMB J4 input - LNGS GPS data input
</pre>

* LEDs
<pre>
D1 - same as LEMO J2A out (Rb clock 1pps in)
D2 - same as SMB J3 out (GPS 1pps from LNGS or from a GPS receiver)
D5 - controlled by FPGA-OUT-LED1
D6 - controlled by FPGA-OUT-LED2
D7 - PRS-10 24V power ok
</pre>

* switches
<pre>
SW1A and SW1B - route PRS-10 RS232 to USB or to FPGA
SW2A - route PRS-10 1pps input from SMB J4 (LNGS) or from LEMO J5B (GPS receiver 1pps)
SW2B - route FPGA-IN-GPSDATA input from SMB J4 (LNGS) or from LEMO J5A (GPS receiver IRIG-B data)
SW3 - LNGS 1pps from rising edge or from falling edge of SMB J4 (LNGS)
</pre>

== Rb clock cable ==
<pre>
Rb clock ----- DS-IOGC side, pin numbers are as labeled on the cable connectors

1 - 1pps out - 8 - 1pps out --- correct
2 - nc
3 - nc
4 - TXD - 3 - RU-DATA-OUT - USB-RX input --- correct
5 - 1pps in - 2 - 1pps in --- correct
6 - +24V - 10 - +24V --- should by pin 1 to use both +24V pins?
7 - RXD - 15 - RU-DATA-IN - USB-TX output --- correct
8 - nc
9 - +24V - 10 - +24V --- correct
10 - GND - 9 - GND --- correct
</pre>

== VX connections ==
<pre>
VXA_TX0 - FPGA-OUT-LED2 - D6 LED ("10 MHz clock")
VXA_TX1 - FPGA-OUT-LED1 - D5 LED ("GPS DATA")
VXA_TX2 - FPGA-OUT-RU1PPS-EN - enable 1pps to Rb clock
VXA_TX3 - not used (62.5 Hz clock)

VXB_TX0 - FPGA-TX - PRS-10 RS-232 out
VXB_TX1 - FPGA-OUT-AUX - 5V TTL J5 LEMO out
VXB_TX2 - FPGA-OUT-OPTO1PPS-EN - PRS-10 power enable
VXB_TX3 - not used (62.5 MHz clock)

VXA_RX0 - FPGA-IN-OPTO1PPS - 1pps from GPS
VXA_RX1 - FPGA-IN-GPSDATA - J4 GPS data
VXA_RX2 - n/c
VXA_RX3 - n/c

VXB_RX0 - n/c
VXB_RX1 - FPGA-IN-AUX - J5 LEMO TTL input
VXB_RX2 - FPGA-RX - PRS-10 RS-232 in
VXB_RX3 - FPGA-IN_RU1PPS - PRS-10 1pps output
</pre>

== test sequence ==

* ./test_cdm_local.exe --writereg 7 0 ### clear reg 7
* ./test_cdm_local.exe --writereg 8 0 ### set vx_tx mux to vx_tx control from reg 7
* connect blue cable to GDM port 6 (next to the ethernet connector)
* ./test_cdm_local.exe --writereg 7 0x4000 ### power up
* test LEDs:
* ./test_cdm_local.exe --writereg 7 0x4100 ### right LED2 D6
* ./test_cdm_local.exe --writereg 7 0x4200 ### left LED1 D5
* test AUX-IN and AUX-OUT:
* ./test_cdm_local.exe --writereg 7 0x6000 ### J6A LEMO (PCB-side) measure +5VDC
* install LEMO jumper between LEMO J6A and J6B
* write 0x4000 then ./test_cdm_local.exe 6 ### reads 0x5C, bit 5 0x20 reads 0
* write 0x6000 then ./test_cdm_local.exe 6 ### reads 0x7C, bit 5 0x20 reads 1
* write 0x4000 to clear all bits
* test the 10 MHz clock:
* init the c.c: ./test_cdm_local.exe --load-cc
* connect 10 MHz clock output to DS-DM LEMO J6-LEFT (CLK_EXT1)
* 10 MHz output ok - IN0 is good: ./test_cdm_local.exe --cc
<pre>
Clock chip state 1, status: OOF_IN2 IN1 IN_SEL_1 HOLD_HIST_VALID
</pre>
* without 10 MHz reports IN0 LOS (no signal) and OOF (wrong frequency)
<pre>
Clock chip state 1, status: LOS_IN0 OOF_IN0 OOF_IN2 IN1 IN_SEL_1 HOLD_HIST_VALID
</pre>
* test GPS IRIG-B data:
* connect GPS IRIG-B signal to J6B (dual-LEMO away from PCB)
* IRIG-B via AUX-IN is ok: ./test_cdm_local.exe --irigb
<pre>
dataframe: S00100110S111001000S...S011110000S010000000S, sec: 34, min: 17, hrs: 00, day of year: 223, year: 2024, date: 10 aug, sbs: 1054 should be 1054
</pre>
* test IRIG-B via J5A GPS-EXT:
* connect GPS IRIG-B signal to J5A (dual-LEMO next to PCB)
* set SW2B: switch on the side of SMB connectors, slide direction of 24V power supply connector
* ./test_cdm_local.exe --irigb built for VXA_RX1 should work
* test GPS 1pps signal:
* connect GPS 1pps to J5B (dual-LEMO away from PCB)
* nothing happens, no output from U22
* remove R51
* set SW2A: switch on the side away from SMB connectors, slide direction of 24V power supply connector
* observe LEDs D1 and D2 blink at 1Hz
* SKIP THIS connect GPS 1pps to J4 using LEMO-SMB adapter
* SKIP THIS nothing happens, nLE is always 3.2V, both settings of SW2
* SKIP THIS !!!failed here!!!
* ./test_cdm_local.exe --writereg 8 9 ### set vx_tx mux to GPS control, hard enables PRS-10 power and PRS-10 1pps in
* observe 1pps counters in reg 68 count at 1 Hz: rb_1pps 0x51->0x52, gps_1pps 0xd5->oxd6
<pre>
root@dsdm:~# ./test_cdm_local.exe 68
ds20k_reg[68] is 0x0051d520 (5362976)
ds20k_reg[68] is 0x0052d624 (5428772)
...
</pre>
* observe PRS-10 can see the 1pps signal "130" changes to "2" after 243 seconds to "4"
* observe period of GPS and PRS-10 1pps is identical (plus/minus 1 clock)
<pre>
root@dsdm:~# ./test_cdm_local.exe 13 14
ds20k_reg[13] is 0x077356d4 (124999380)
ds20k_reg[14] is 0x077356d4 (124999380)
</pre>
* observe GDM/CDM clocks are exactly 125 MHz, we are running on DS-DM internal oscillator
<pre>
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735943 (125000003) should be ~125 MHz
0x1034 rx_clk: 0x07735943 (125000003) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735943 (125000003) should be ~125 MHz
0x103C tx_clk: 0x07735943 (125000003) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>
* if tx_clk is missing, reset the mgt: ./test_cdm_local.exe --reset-mgt
* switch CC to external clock:
<pre>
root@dsdm:~# ./test_cdm_local.exe --cc-in0
CC use clock input 0: 10 MHz LEMO external clock
root@dsdm:~# ./test_cdm_local.exe --cc
Polling CC status...
Clock chip state 1, status: IN0 IN_SEL_REGCTRL IN_SEL_0 HOLD_HIST_VALID
</pre>
* observe rx_clk and tx_clk are now slightly off: they run from PRS-10 10 MHz clock and are measured against the DS-DM internal oscillator.
<pre>
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x94b12519, ds20k version 0x20240814
DS-DM firmware build 0x94b12519, ds20k version 0x20240814
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x077358e1 (124999905) should be ~125 MHz
0x1034 rx_clk: 0x07735b49 (125000521) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735b49 (125000521) should be ~125 MHz
0x103C tx_clk: 0x07735b49 (125000521) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>
* observe GPS and PRS-10 1pps period is now exactly 125*10^6 of 8ns clocks (equal to 1 second)
<pre>
root@dsdm:~# ./test_cdm_local.exe 13 14
ds20k_reg[13] is 0x0773593f (124999999)
ds20k_reg[14] is 0x0773593f (124999999)
</pre>
* look at them repeatedly, observe reg 13 "GPS 1pps period" has some wobble, reg 14 "Rb clock 1pps period" is steady. this is as expected: if GPS 1pps was steady, we do not need to Rb clock. PRS-10 1pps is derived from the PRS-10 10 MHz clock and is measured against the 125 MHz clock derived from the same 10 MHz clock in the GDM FPGA.

Rev0 Test status:

* GPS 1pps to SMB-in ok (LED flashes)
* GPS 1pps to FPGA ok
* GPS data to FPGA ok
* GPS 1pps to PRS-10 enabled from FPGA ok
* PRS-10 Rb clock 1pps out to FPGA ok
* PRS-10 Rb clock 1pps out to FPGA to LEMO AUX out ok
* can see GPS 1pps, IRIG-B, PRS-10 1pps out, 10 MHz on the scope, ok
* PRS-10 syncs on leading edge (0->1) of GPS 1pps signal, ok
* reg 13 and 14 1pps periods are identical, ok
* CC locks on PRS-10 10 MHz clock, DS-DM runs on PRS-10 clock, ok
* NOT TESTED - smb output
* NOT TESTED - optical converter fiber to SMB
* NOT TESTED - optical converter SMB to fiber
* NOT TESTED - SMB loopback
* NOT TESTED - fiber loopback

Rev1 Test status: (21 Nov 2024)

* power up ok
* LEDs ok
* LEMO AUX-IN, AUX-out ok
* GPS IRIG-B to FPGA via J6B AUX-IN ok
* GPS IRIG-B to FPGA via J5A GPS-EXT ok
* PRS-10 Rb clock 1pps out to FPGA ok
* PRS-10 Rb clock 10 MHz to C.C. ok
* PRS-10 RS-232 ok
* J5B 1pps input fail. input signal is 3.3V without termination, 1.5V when plugged into IOGC (on R51), no output from U22.

= VX busy logic =

= DS-20K DAQ =

== Overview ==

DS-DM, GDM and CDM are key parts of the DS-20K DAQ system:
* common clock distribution from external clock (atomic clock, GPS) to GDM to per-quadrant CDMs to VX digitizers
* common trigger distribution from GDM internal algorithm or external input to all VX digitizers
* run control: GDM, CDM, VX all start recording data at the same time (clock and timestamp reset)
* collection of trigger data from VX digitizers to per-quadrant CDMs to GDM

== Deliverables ==

* hardware and firmware for GDM to CDM clock distribution
* hardware and firmware for CDM to VX clock distribution
* hardware and firmware for GDM external clock input (atomic clock or GPS)
* hardware and firmware for CDM and VX serial communications (VX LVDS I/O connector)
* firmware for run control (timestamp reset and sync): GDM to CDM to VX
* firmware for common trigger distribution: GDM to CDM to VX
* firmware for trigger data flow: VX to CDM to GDM
* firmware for busy control: VX to CDM to GDM back to CDM to VX
* firmware for flow control: FEP to GDM MIDAS frontend to GDM to CDM to VX
* GDM MIDAS frontend: clock selector and monitoring, trigger and run control, busy and flow control, GDM housekeeping
* CDM MIDAS frontend: clock monitoring, CDM housekeeping

specific performance:
* GDM external clock: 10 MHz GPS clock
* GDM to CDM fiber link:
** clock XXX MHz
** link data rate: XXX Gbit/sec
** CDM recovered clock: XXX MHz
** CDM recovered clock jitter: XXX ns
** phase alignment between CDMs: XXX ns
** phase alignment between CDMs persists across reboots, power cycles, firmware updates
** phase alignment between CDMs should be easy to measure
** phase alignment between CDMs should be easy to recalibrate if hardware parts are replaced (DS-DM boards, fiber transceivers, fiber cables, etc)
** data packet bandwidth: XXX Mbytes/sec
** data packet latency: XXX clocks
** data packet skew between CDMs: XXX clocks
* CDM to VX clock:
** clock: XXX MHz
** jitter, all CDM clock outputs: XXX MHz
** phase alignment between all CDM clock outputs: XXX ns
* CDM to VX trigger:
** TBD (use the VX "sync" input or VX LVDS I/O line or VX serial link packet)
* CDM to VX serial link:
** clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
** bit rate: XXX bits/sec
** latency: XXX link clocks
** maximum skew between VXes: XXX ns
* VX to CDM serial link:
** clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
** bit rate: XXX bits/sec
** latency: XXX link clocks
** maximum skew between VXes: XXX ns
* timestamp reset:
** maximum skew between VXes: XXX ns
* busy round trip time: XXX ns (VX to CDM to GDM back to CDM to VX)
* flow control latency: XXX ns (FEP software to GDM MIDAS frontend to GDM to CDM to VX)

== Technical risk items ==

this refers to unexpected behaviour and performance of
system components, causes big difficulty in implementing the system,
prevents delivery of deliverables, and prevents or negatively affects operation
of the DS-20K DAQ or of the whole experiment.

(14-sep-2022, list is not sorted by any criterial: severity, probability, ease of investigation)

(stability of course is long term stability, across hours, days, weeks, months, years)

* stability of Enclustra FPGA modules (crashes/year, failures to boot/year, flash corruption/year)
* stability of GDM external clock PLL (lock loss/year)
* stability of CDM recovered clock (lock loss/year, unexpected phase drifts, etc)
* unexpected failures or bit error rates in GDM-CDM fiber links
* stability of CDM VX clock outputs (stability of clock cleaner chip)
* stability of VX internal clock distribution (VX PLL lock loss events)
* stability of VX CAEN base firmware (different versions of CAEN base firmware have different clock distribution behaviour)
* strange things in CAEN base firmware (unexpected clocking of LVDS I/O, unexpected phase shifts between clocks, etc)
* DS-DM and VX hardware problems (incompatible LVSD I/O, incompatible clock signals, etc)

== Milestones ==

(14-sep-2022: at this stage of the project, priority must be given to identifying and retiring (so called) technical risk factors. it is not good to build the complete system only to discover that (for example) some Enclustra FPGA modules require 5 attempts to boot and erase their flash memory contents once a month. Both example are real-life actual problems that caused big difficulties in GRIFFIN/TIGRESS and ALPHA-g experiments).

Development and testing milestones in time reversed order:

* full DAQ data challenge: all VXes, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
* one quadrant data challenge: 1 VME crate of VX, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
* vertical slice data challenge: 1 VME crate, 2 VX, 2 CDM (1 VX per CDM), GDM, etc operate as designed
* GDM-CDM link finalized (data rate frozen, data packet format frozen, data content permitted to change)
* CDM-VX serial link finalized (data rate frozen, data packet format frozen, data content permitted to change)
* run control (timestamp reset) and trigger distribution design agreed upon, frozen (list of possible triggers permitted to change)
* VX to CDM to GDM data flow design agreed upon, frozen (data contents permitted to change)
* major technical risk items retired (all hardware and firmware is working as expecred without mysteries and surprises, all problems are identified, investigated, resolved, solutions tested)
* stable operation of CDM-VX serial links in vertical slice system
* stable operation of GDM to CDM clock in vertical slice system
* stable operation of CDM to VX clock in vertical slice system
* vertical slice system assembled (1 VME crate, 2 VX, 2 CDM, 1 GDM, network, FEP, TSP, MIDAS)

= ZZZ =

ZZZ

DS-DM

2026-04-01T22:33:45Z

Olchansk: /* GDM, CDM, VX packet communications */

= DS-DM =

DarkSide-20k Global and Crate Data Manager board (GDM and CDM).

Global Data Manager (GDM):
* clock distribution to CDM boards (including GPS/atomic clock source)
* collection of trigger data from CDM boards, processing and distribution of trigger decision to CDM boards
* run control
* integration with GPS 10MHz and 1pps clocks and GPS/IRIG date and time information

Crate Data Manager (CDM):
* clock distribution from GDM to CAEN VX digitizers
* receive trigger data from CAEN VX digitizers
* send trigger data to GDM
* run control and dead time control

= Links =

* https://dsdaqgw.triumf.ca/vslice/ - vertical slice DAQ at TRIUMF
* https://daq00.triumf.ca/DaqWiki/index.php/DarkSide - this DS-DAQ Wiki page
* https://dsvslice.triumf.ca/vx_napoli - DS vertical slice at TRIUMF
* https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0 - DS-DM board Rev0
* https://edev-group.triumf.ca/hw/exp/dark-side-20k/cdm-gdm-data-manager-vme-cards/rev1 - DS-DM board Rev1
* https://docs.xilinx.com/v/u/en-US/ds593 - Xilinx Platform Cable USB II
* https://www.enclustra.com/en/products/system-on-chip-modules/mercury-xu8/ - Enclustra
* https://bitbucket.org/team-ds-dm/phasemeasurement - VX Phase measurement script
* https://daq00.triumf.ca/daqinv/frontend/list/178 - inventory database
* https://daq00.triumf.ca/elog-ds/DS-DAQ - DS-DAQ elog
* https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0 - GPS and Rb Clock adapter board

= Git repos =

* dsdmdev@dsdaqgw.triumf.ca:git/gcdm.git - git repository, DS-DM firmware, branch develop_ko2, contact K.O. for ssh key access
* https://edev-group.triumf.ca/fw/exp/darkside/gcdm - git repository, DS-DM firmware (broken, do not use)
* https://bitbucket.org/team-ds-dm/ds-dm-software - git repository, DS-DM MIDAS frontend
* https://bitbucket.org/ttriumfdaq/dsproto_vx2740/src/develop/ - VX DSFE MIDAS frontend
* https://openproject.triumf.ca/svn/darkside-20k/ - VX firmware project

= Onboard hardware =

* jtag chain: arm_dap_0 0x5BA00477, xczu4_1 0x04721093
* Eclustra Mercury+ XU8 module: ME-XU8-4CG-1E-D11E-R2.1
** Xilinx® Zynq Ultrascale+™ MPSoC XCZU4CG-1FBVB900E
** DDR4 ECC SDRAM (PS) 2 GB
** DDR4 SDRAM (PL) 1GB
* ethernet mac chip: AT24MAC402-SSHM-T ("602" chip is wrong)
* USB UART for Enclustra serial console, micro-USB, 115200n8
* clock chip: SI5394A-A-GM and oscillator CS-044-054.0M (54 MHz)
* U23 3.3V current meter and thermometer, LTC2990IMS#TRPBF
* LEDs:
** LED_FP A/B/C/D 0/1/2/3
** led1 - 3V3_SW_ON, SOM_POWER_GOOD - Enclustra FPGA module 3.3V power is good
** led2 - LTM4624 PGOOD
** led3 - FPGA_DONE - FPGA has booted
** led4 - TP-S-1, PCLK_P
** led5 - TP-S-2, PCLK_N
* LEMO connectors (top to bottom)
** J4 - input (NIM/TTL) (EXT_IN_LV(1), EXT_IN_LV(2))
** J5 - input (NIM/TTL) (EXT_IN_LV(3), EXT_IN_LV(4))
** J6 - external clock (GPS 10MHz and PPS)
** J7 - output (NIM/TTL) (EXT_OUT(1), EXT_OUT(2))
* SMA connectors
** J9, J10 - CLK_CCA from U6 C.C.
** J11, J12 - CLK_TP0
* RJ45 ethernet connector (100mbit: green light on, yellow light flashes when there is traffic)
* SFP connector (SFP is FTLF8526P3BNL, 6 Gbit/sec, 850 nm, 300m 50/125um OM3 MMF)
* 4 QSFP connectors (GDM)
* 6 VX connectors (CDM)

= Buttons, jumpers and switches =

Buttons:

* PB1 - HRST - reboot FPGA (power-on reset)
* PB2 - SRST - (SRSTn) - reboot ARM CPU

Switches:

* SW1 - boot mode BM0, BM1 [-->]
* SW2 - LEMO output NIM<->TTL
* SW3 - LEMO input 1 and 2 NIM/TTL
* SW4 - LEMO input 2 and 4 NIM/TTL
* SW5 - LEMO clock input NIM/TTL
* SW6 - serial console select. [PS<--PL] PS is ARM CPU, PL is FPGA.

= Front panel =

<pre>
| top
|
| LED-FP1 | LED_FP(0,1,2,3)
|
| SFP J???
|
| LEMO J4-LEFT, J4-RIGHT | EXT_IN_LV(1), EXT_IN_LV(2)
| LEMO J5-LEFT, J5-RIGHT | EXT_IN_LV(3), EXT_IN_LV(4)
| LEMO J6-LEFT, J6-RIGHT | CLK_EXT1, CLK_EXT0 (125 MHz only)
| LEMO J7-LEFT, J7-RIGHT | EXT_OUT(1), EXT_OUT(2)
|
| J-VX-1
| J-VX-2 or QSFP-1
| J-VX-3 or QSFP-2
| J-VX-4 or QSFP-3
| J-VX-5 or QSFP-4
| J-VX-6
|
| RJ45 J3 ethernet
|
| bottom
</pre>

= VX adapter board =

LVDS I/O connector
<pre>
34 pin connector: 0|:::: :::: :::: :::: :|15,16 n/c
</pre>

== split-cable connection ==

<pre>
0 .. 7 -> N/C

8 -> VX_RX(3) - not used
9 -> VX_RX(2) - busy VX to CDM
10 -> VX_RX(1) - DS20K 125 MHz serial data VX to CDM
11 -> VX_RX(0) - DS20K 62.5MHz clock VX to CDM

12 <- VX_TX(0) - TRG CDM to VX
13 <- VX_TX(1) - TSM CDM to VX, to be VETO CDM to VX
14 <- VX_TX(2) (set by jumper routed here or to VX CLKIN SYNC) - 125 MHz serial data CDM to VX
15 <- n/c (CLK routed to VX CLKIN CLK) - 62.5 MHz clock CDM to VX

16 - n/c
</pre>

== one-to-one connection ==

<pre>
0 -> VX2_RX(3)
1 -> VX2_RX(2)
2 -> VX2_RX(0)
3 -> VX2_RX(1)

4 <- CLK
5 <- VX2_TX(0)
6 <- VX2_TX(1)
7 <- VX2_TX(2)

8 -> VX1_RX(3)
9 -> VX1_RX(2)
10 -> VX1_RX(1)
11 -> VX1_RX(0)

12 <- VX1_TX(0)
13 <- VX1_TX(1)
14 <- VX1_TX(2) (set by jumper routed here or to VX CLKIN SYNC)
15 <- n/c (CLK routed to VX CLKIN CLK)

16 - n/c
</pre>

= Board schematics Rev0 =

* [[File:SCH-DS-xDM-Rev0.PDF]]
* note: FPGA pin annotations ("IO", "SCLK", "PCLK", etc) on the schematics are bogus, instead, trace them to the FPGA pins.
* note: ENC A is J800, ENC B is J801, ENC C is J900 (schematic name to enclustra name)
* note: Enclustra special pins: "GC" is "clock capable", "HDGC" is "clock capable", "MGTREFCLK" is MGT reference clocks.
* board modifications:
** ethernet mac chip
** NIM output (no U15, etc)
** RJ45 wrong pinout (board mod or special ethernet cable)
** 125 MHz clock mods (TBW)
** disconnect QSFP0_SEL from SFP_RS0 and QSFP1_SEL from SFP_RS1, these signals are not used by modern SFPs
** provide SFP i2c modsel to allow SFP and QSFP at the same time on address 0x50

= Board schematics Rev1 =

* [[File:SCH-DS-xDM-Rev1.PDF]]
* ENC-A is J800, ENC-B is J801, ENC-C is J900
* modifications from Rev0:
* clocks:
** 125MHz osc: CLK_XO_125, CLK3_XO_125 (CLK2_XO_125 removed)
** C.C. in2 is CLK_CC_IN from ENC-C142,144
** C.C. out0 via CLK_A to CLK_CC_OUT0 (ENC-C3,5), CLK_CC_OUT1 (ENC-B3,5), CLK_CC_OUT2 (ENC-C151,153)
* RTC chip: TP_S pins gone, RTC_I2C_SCL, RTC_I2C_SDA, RTC_1PPS and RTC_32KHZ added
** RTC_1Hz ENC-C160 pull up 10k to 3.3V
** RTC_32k ENC-B129 pull up 10k to 1.8V
* QSFP, SFP control lines:
** HW_ID_xDM ENC-B131 gone, reused as SFP_ModPrsN
** SFP_ModPrsN (MOD_ABS) ENC-B131 input (no pulls)
** SFP_TX_fault (was QSFP2_SEL) QSFP_ModPrsN ENC-C157 input (no pulls)
** SFP_TX_disable from QSFP_LPMode ENC-C163 output (pull up 10k to 3.3V)
** SFP_ModDet n/c (was QSFP_ModPrsN) renamed SFP_ModPrsN input
** SFP_RX_LOS to QSFP_IntN ENC-C159 input (pull up 2k2 to 3.3V)
** SFP_RS0, SFP_RS1 n/c, assembly option: floating
** QSFP_ResetN - FPGA output, pull up 10k to 3.3V
** QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL - FPGA outputs, pull up 10k to 3.3V
** QSFP_LPmode - FPGA output, pull up 10k to 3.3V
** QSFP_ModPrsN - ??? - pull up 2k2 to 3.3V
** QSFP_IntN - ??? - pull up 2k2 to 3.3V
* VX_TX and VX_RX reassigned for better length matching
* cosmetic changes:
** front panel USB-C connector (replaces Micro-USB)
** (reset push button PB1 PORn ENC-A132 aka PS_POR_B aka PS_POR#)
** (reset push button PB2 SRSTn ENC-A124 aka PS_SRST_B aka PS_SRST#)
** front panel PB3 HRSTn same as PB1
* Enclustra errata:
** possible cross talk between pins AA2 (J801-B90, VX9_RX2) and AE3 (J801-B129, RTC_32KHZ). no RTC on the CDM.
** ethernet link up failure (see Microchip KSZ9031 errata)

= FPGA MGT blocks =

<pre>
* SFP - ENC B45,B47 and B48,B50 - FPGA D5,D6 MGTHTX0_D and D1,D2 MGTHRX0_D
* QSFP0 TX0 - ENC C13,C17 - FPGA H5,H6 MGTHTX0_C
* QSFP0 TX1 - ENC C21,C25 - FPGA G7,G8 MGTHTX1_C
* QSFP0 TX2 - ENC C29,C23 - FPGA F5,F6 MGTHTX2_C
* QSFP1 TX0 - ENC C37,C41 - FPGA E7,E8 MGTHTX3_C
* QSFP1 TX1 - ENC C45,C47 - FPGA P5,P6 MGTHTX0_B
* QSFP1 TX2 - ENC C51,C53 - FPGA M5,M6 MGTHTX1_B
* QSFP2 TX0 - ENC C57,C59 - FPGA L3,L4 MGTHRX2_B
* QSFP2 TX1 - ENC C63,C65 - FPGA K5,K6 MGTHTX3_B
* QSFP2 TX2 - ENC C75,C77 - FPGA W3,W4 MGTHTX0_A
* QSFP3 TX0 - ENC C79,C81 - FPGA V5,V6 MGTHTX1_A
* QSFP3 TX1 - ENC C85,C87 - FPGA T5,T6 MGTHRX2_A
* QSFP3 TX2 - ENC C89,C91 - FPGA R3,R4 MGTHTX3_A
</pre>

= Clock distribution Rev0 =

Simplified:

* 125 MHz osc -> CLK_XO_125 -> MGTREFCLK0_A -> not used
* 125 MHz osc -> CLK3_XO_125 -> MGTREFCLK1_B -> SFP RX ref clock, QSFP RX and TX ref clock (this is not final design!)
* (disconnected) 125 MHz osc -> CLK2_XO_125 -> FPGA AG8,AH8 (GC)
* 125 MHz osc -> C.C. in1

* C.C. in0 <- CLK_EXT1 (10 MHz GPS clock)
* C.C. in1 <- 125 MHz osc
* (disconnected) C.C. in2 <- CLK_CC_IN <- MGTREFCLK0_D <- SFP RX clock (cannot be used because of uncontrollable phase)
* C.C. in2 <- CLK2_XO_125 <- FPGA AG8,AH8 (GC) <- SFP RX recovered 125 MHz clock
* C.C. in3 <- CLK_FB

* C.C. 125 MHz -> CLK_CC_OUT0 -> MGTREFCLK0_B -> QSFP RX and TX ref clock (final design)
* C.C. 125 MHz -> CLK_CC_OUT1 -> MGTREFCLK1_D -> SFP TX clock
* C.C. 62.5 MHz -> VX clock fanout

proposed changes:
* add C.C. 125 MHz -> new CLK_CC_OUT2 (old CLK2_XO_125) -> FPGA AG8,AH8 (GC)
* change C.C. in2 <- new CLK_CC_IN <- FPGA AK8,AK9 (non-GC)

Complete:

<pre>
125 MHz oscillator - U5 fan out -
q0 -> CLK_XO_125 -> ENC C72,C74 -> FPGA R7,R8 MGTREFCLK0_A (not used)
q1 -> U6 C.C. in1
q2 -> CLK3_XO_125 -> ENC C7,C9 -> FPGA J7,J8 MGTREFCLK1_B -> SFP RX reference clock, QSFP RX and TX reference clocks (not final design!)
q3 -> disconnected on the board, was CLK2_XO_125 -> ENC C151,C153 -> FPGA AG8,AH8 (GC)

U6 C.C (clock cleaner) -

in0 <- CLK_EXT1 (presumably GPS 10 MHz ref clock)
in1 <- 125 MHz oscillator via U5
in2 <- (was: CLK_CC_IN <- ENC B10,B12 <- FPGA D9,D10 MGTREFCLK0_D <- SFP RX clock, 125 MHz)
in2 <- CLK2_XO_125 <- ENC C151,B153 <- FPGA AG8,AH8 (GC) <- mgt_link_data_to_processing.rx_data_clk (SFP RX data clock, 125 MHz)
in3 <- CLK_FB

out0 -> CLK_CCA -> U12 (125 MHz)
out1 -> CLK_CCB -> VX1..6 (62.5 MHz)
out2 -> CLK_CCC -> VX7..12 (62.5 MHz)
out3 -> CLK_FB into in3

CLK_CCA -> U12 (125 MHz fan out) ->

Q0 -> not used
Q1 -> CLK_CC_OUT0 -> ENC C3-5 -> FPGA L7,L8 MGTREFCLK0_B -> QSFP RX and TX reference clocks (final design)
Q2 -> CLK_CC_OUT1 -> ENC B3-5 -> FPGA B9,B10 MGTREFCLK1_D -> SFP TX clock
Q3 -> not used
Q4 -> not used
Q5 -> SMA J9/J10

CLK_TP0 - FPGA N7,N8 MGTREFCLK1_A - ENC C69,C71 - DS-DM SMA J11, J12 (NOT IN CDM PROJECT)

Notes:
* CLK_XO_125 (125 MHz osc) is not used
* 62.5 MHz VX clock does not go into the FPGA
* CLK_EXT0 going to in1 of U5 cannot be used. only permitted frequency is 125 MHz (it drives the MGT reference clocks) and is this frequency is too high for LEMO cables and connectors

Proposed modifications:
- CLK_CC_IN: from FPGA output pin (ENC C142,C144 - FPGA AK8,AK9) to clock cleaner in2 (instead of CLK2_XO_125 pins)
- CLK_CCA -> U12 -> currently unused out3 -> CLK2_XO_125 FPGA pins
- repurpose CLK_EXT0 at GPS 1pps/IRIG input to FPGA
</pre>

= Clock distribution Rev1 =

Simplified:

* 125 MHz osc -> CLK_XO_125 -> MGTREFCLK0_A -> clk_xo_125. not used
* 125 MHz osc -> CLK3_XO_125 -> MGTREFCLK1_B -> CDM SFP RX ref clock
* 125 MHz osc -> C.C. in1

* C.C. in0 <- CLK_EXT1 (10 MHz GPS clock)
* C.C. in1 <- 125 MHz osc
* C.C. in2 <- CLK_CC_IN <- FPGA AK9,AK8 <- CLK_CC_IN1, rx_clk_mgt, SFP RX recovered clock, 125 MHz
* C.C. in3 <- CLK_FB

* C.C. 125 MHz -> CLK_CC_OUT0 -> MGTREFCLK0_B -> GDM QSFP RX and TX ref clock, CDM rx_clk mux
* C.C. 125 MHz -> CLK_CC_OUT1 -> MGTREFCLK1_D -> SFP TX clock
* C.C. 125 MHz -> CLK_CC_OUT2 -> FPGA AG8,AH8 (GC) -> not used
* C.C. 62.5 MHz -> VX clock fanout

Complete:

<pre>
125 MHz oscillator - U5 fan out -
q0 -> CLK_XO_125 -> ENC C72,C74 -> FPGA R8,R7 MGTREFCLK0_A -> XDC CLK_XO_125_P -> VHDL not used
q1 -> U6 C.C. in1
q2 -> CLK3_XO_125 -> ENC C7,C9 -> FPGA J8,J7 MGTREFCLK1_B -> XDC GDM missing, CDM CLK3_XO_125_P -> VHDL SFP RX reference clock (mgt_rx_ref_clk)
q3 -> not used, was CLK2_XO_125

U6 C.C (clock cleaner) -

in0 <- CLK_EXT1 (GPS 10 MHz clock)
in1 <- 125 MHz oscillator via U5
in2 <- CLK_CC_IN <- ENC C142,C144 <- FPGA AK9,AK8 <- XDC GDM missing, CDM CLK_CC_IN1 <- VHDL rx_clk_mgt
in3 <- CLK_FB

out0 -> CLK_CCA -> U12 (125 MHz)
out1 -> CLK_CCB -> VX1..6 (62.5 MHz)
out2 -> CLK_CCC -> VX7..12 (62.5 MHz)
out3 -> CLK_FB into in3

CLK_CCA -> U12 (125 MHz fan out) ->

Q0 -> not used
Q1 -> CLK_CC_OUT0 -> ENC C3,C5 -> FPGA L8,L7 MGTREFCLK0_B -> XDC GDM clk_mgtrefclk0_x0y1_p, CDM i_CLK_CC_OUT0_P -> VHDL GDM mgt_b_ref_clk QSFP RX and TX reference clocks, CDM alternate rx_clk via clock mux
Q2 -> CLK_CC_OUT1 -> ENC B3,B5 -> FPGA B10,B9 MGTREFCLK1_D -> XDC i_CLK_CC_OUT1_P -> VHDL GDM not used, CDM mgt_tx_ref_clk SFP TX clock
Q3 -> CLK_CC_OUT2 -> ENC C151,C153 -> FPGA AG8,AH8 (GC) -> XDC GDM missing, CDM i_CLK_CC_OUT2_P
Q4 -> not used
Q5 -> SMA J9/J10

CLK_TP0 <-> ENC C69,C71 <-> FPGA N8,N7 MGTREFCLK1_A (not used) -> XDC CLK_TP0_P -> VHDL not used (DS-DM SMA J11, J12)

disconnected <- OLD_CLK_CC_IN <- ENC B10,B12 <- FPGA D9,D10 MGTREFCLK0_D <- XDC o_CLK_CC_IN <- VHDL CDM o_OLD_CLK_CC_IN SFP o_mgt_rx_rec_clk

Notes:
* Enclustra ENC-B is J801, ENC-C is J900
* "(GC)" is clock-capable FPGA pin
* CLK_XO_125 (125 MHz osc) is not used in FPGA
* 62.5 MHz VX clock does not go into the FPGA
* CLK_EXT0 going to in1 of U5 cannot be used. only permitted frequency is 125 MHz (it drives the MGT reference clocks) and is this frequency is too high for LEMO cables and connectors
</pre>

= I2C bus =

* I2C_SCL is J-ENC-A pin 111 I2C_SCL is FPGA I2C_SCL_PL AB13 (IO_L1N_TOL1D_64) and I2C_SCL_PS F18 (PS_MIO10)
* I2C_SDA is J-ENC-A pin 113 I2C_SDA is FPGA I2C_SDA_PL AH13 (IO_L7N_T1L1Q_AD13N_64) and I2C_SDA_PS G18 (PS_MIO11)

* XU8 secure EEPROM ATSHA204A at 0x64, this is 0110'010X -> linux _011'0010 is 0x32. (but responds to scan and read at 0x33)
* U4 ethernet mac chip, EEPROM at 1010 A2 A1 A0 X and MAC/serial_no at 1011 A2 A1 A0 X. A0=VCC, A1=VCC, A2=GND -> linux _101'0011 and _101'1011 is 0x53 and 0x5B.
* U6 clock chip, address 1101 0 A1 A0 X. A1=VCC, A0=N/C (internal pull-up) -> linux _110'1011 is 0x6b
* U23 voltmeter at 10011 ADR1 ADR0. ADR0=GND, ADR1=VCC -> linux _100'1110 is 0x4e
* SFP, address 1010000X -> linux _101'0000 is 0x50. additional SFP data at 0x51
* QSFP0, QSFP1, QSFP2, QSFP3 (QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL)

= I2C clock builder connection =

* use Silicon Labs USB "Clock builder pro field programmer", www.silabs.com/CBProgrammer
* connect rainbow jumper cable pins:
** black - 1-GND to GND on DS-DM
** white - 3-SCLK to SCL on the DS-DM
** grey - 7-SDA_SDIO to SDA on the DS-DM
* power up the DS-DM
* plug USB programmer into Windows laptop
* on Windows, run "ClockBuilder Pro"
* it should report "Field programmer detected", press "EVB GUI"
* in EVB GUI, press "Config", set I2C address 0x6B
* press "Scan", it should find Si5394A-A-GM
* select the "Status" tab, should see real-time status of clock chip

= GDM MGT configuration =

* TX configuration:
* GDM MGT transceivers are configured as "multilane" TX and RX.
* there is 12 TXes ("lanes")
* MGT reference 125 MHz clock goes into [2:0]gtreclk00_in and [11:0]gtrefclk0_in
* one MGT is designated as "master"
* PLL of master MGT converts reference clock into common TX clock and common tx_user_clk2 which becomes tx_data_clk
* common tx_user_clk2 aka tx_data_clk goes into all TXes and clocks tx_data.
* tx_user_clk2 aka tx_data_clk is 125 MHz but not same phase as MGT reference clock.

* RX interim configuration:
* there is 12 RXes ("lanes")
* each RX produces it's own recovered RX clock
* "multilane" configuration assumes all RX recovered clocks run at the same frequency (TX on the other end are driven by common TX clock, see above), but have different phase
* one RX recovered clock is designated as "master" (rx_user_clk2 aka rx_data_clk) and a phase-matching fifo/buffer is used to bring rx_data from all 12 RXes to this common rx_data_clk
* this works because each CDM SFP TX runs on the SFP RX recovered clock which is frequency-locked with the GDM QSFP TX clock.

* RX final configuration:
* MGTs permit using the common TX clock (tx_user_clk2 aka tx_data_clk) as the common rx_data_clk (they are frequency locked through the CDM).
* this permits use of tx_data_clk as the main clock domain in the GDM and removes the need to bring rx_data into the tx_data_clk domain (actually this is done in the MGT RX phase matching fifo/buffer).

= Clock path =

NOTE: MUST REVIEW!!!

<pre>
10 MHz ext clock or GDM 125 MHz oscillator
-> GDM QSFP MGT reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz (GDM main clock domain) and TX bit clock 2.5 GHz
-> GDM QSFP optic transmitter
-> CDM SFP optic receiver
-> CDM SFP MGT, RX reference clock is CDM 125 MHz oscillator
-> MGT RX recovered clock 125 MHz (CDM main clock domain)
-> CC_CLK_IN -> CDM C.C. -> CC_CLK_OUT1 -> CDM SFP TX reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz and TX bit clock 2.5 GHz
-> (tx_data phase matching fifo from CDM main clock domain to tx_data_clk)
-> CDM SFP optic transmitter
-> GDM QSFP RX optic receiver (12x)
-> GDM QSFP MGT (RX reference clock is same as TX reference clock)
-> MGT RX recovered clock (12x recovered clocks)
-> in multi-lane configuration, one of them is the "master" recovered clock rx_data_clk
-> (rx_data phase matching fifo from rx_data_clk to GDM main clock domain)
</pre>

= CDM rx_clk mux =

when CDM SFP is not connected, there is no SFP recovered clock and a mux is used to switch between clk_cc_out0 (power up default) and rx_clk_mgt (SFP recovered clock)

== Test SFP disconnected ==

note: if I say "--cc-in1", CC seems to lock on the 10 MHz GPS external clock,
to prevent this, test sequence includes reloading the CC and the reset of MGT.

<pre>
./test_cdm_local.exe --write32 0x30 0x0 ### rx_clk mux select CC clock
./test_cdm_local.exe --load-cc
./test_cdm_local.exe --reset-mgt
./test_cdm_local.exe --cdm-clocks
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773594e (125000014) should be ~125 MHz <=== all 4 clocks wobble close to 125 MHz
0x1034 rx_clk: 0x0773594f (125000015) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x0773594f (125000015) should be ~125 MHz
0x103C tx_clk: 0x0773594f (125000015) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
./test_cdm_local.exe --write32 0x30 0x1 ### rx_clk mux select SFP recovered clock
./test_cdm_local.exe --cdm-clocks
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773598b (125000075) should be ~125 MHz
0x1034 rx_clk: 0x076d58ec (124606700) should be ~125 MHz <=== off frequency because there is no valid SFP recovered clock
0x1038 mgt_tx_ref_clk_raw: 0x0773598c (125000076) should be ~125 MHz
0x103C tx_clk: 0x0773598b (125000075) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

= Clock domains =

== GPS ==

* no GPS : GDM runs from internal 125 MHz oscillator
* external 10 MHz clock : GDM runs from external 10 MHz clock and optional 1pps signal (use VME-NIMIO32 NIM outputs)
* GPS receiver : GDM runs from GPS 10 MHz clock and GPS IRIG serial data
* LNGS GPS:
** provides 1pps and serial data over fiber from GPS receiver (master). LNGS xxx box is not used.
** serial data goes to GDM, decoded, 1pps signal extracted, goes to LEMO output
** 1pps from GDM is used to train the Rubidium clock which provides a 10 MHz clock
** 10 MHz output from Rubidium clock goes to GDM 10 MHz external clock input

== GDM ==

* AXI clock (100 MHz) - AXI registers
* 125 MHz oscillator - to clock cleaner
* 10 MHz external clock LEMO input - to clock cleaner
* FPGA 125 MHz clock CLK_CC_IN - to clock cleaner - not used (could be used for cascaded GDMs)
* 125 MHz oscillator - CLK3_XO_125 - mgt_b_ref_clk QSFP MGT reference clock (interim GDM design)
* clock cleaner output 125 MHz fanout:
** CLK_CC_OUT0 - QSFP MGT reference clock (final design)
** CLK_CC_OUT1 - not used (CDM SFP reference clock)
** CLK_CC_OUT2 - not used
* QSFP MGT TX data clock 125 MHz tx_data_clk (main clock domain)
** QSFP TX data
** QSFP RX data (in final design, rx_data_clk is same as tx_data_clk. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk == tx_data_clk)
** ds20k block
* (in interim design, rx_data_clk is disconnected from tx_data_clk. rx_data_clk is the "multilane master clock" which is one of the 12x RX recovered clocks. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk)

note: all these clocks are frequency locked to 125 MHz

== CDM ==

* AXI clock (100 MHz) - AXI registers
* 10 MHz external clock LEMO input - to clock cleaner (not used)
* 125 MHz oscillator to fanout
** to clock cleaner
** to SFP MGT RX reference clock: CLK3_XO_125 to mgt_rx_ref_clk to gtrefclk01_in
* SFP MGT RX recovered clock 125 MHz
** MGT PLL to MGT rx_user_clk2 aka rx_data_clk
** MGT rxrecclkout_out to mgt_rx_rec_clk to CLK_CC_IN to C.C.
* SFP MGT rx_user_clk2 aka rx_data_clk (250 MHz/8 bit, 125 MHz/16 bit, 62.5 MHz/32 bit data) (main clock domain)
** SFP RX data
** ds20k block
** VX TX clock PLLs
** VX RX clock PLLs
* C.C. fan out
** 62.5 MHz VX clocks (12x)
** CLK_CC_OUT0 (not used, GDM QSFP MGT reference clock)
** CLK_CC_OUT1 125 MHz to mgt_tx_ref_clk to gtrefclk00_in to SFP MGT TX reference clock
** CLK_CC_OUT2 (not used)
* SFP MGT tx_user_clk2 aka tx_data_clk
** SFP TX data
** TX data phase matching fifo from main clock domain to tx_data_clk
* VX TX clock PLLs
** 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
** VX TX data phase matching from main clock domain to VX TX clock (12 total)
** VX TX serializer
** VX TX LVDS transmitter
* VX RX clock PLLs
** 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
** VX RX LVDS receivers (12 total)
** VX RX deserializers (12 total)
** VX TX data phase matching from VX RX clock to main clock domain

== VX ==

* everything runs on the VX main 125 MHz clock
* correct phase of VX to CDM LVDS data is adjusted by scan of CDM VX RX PLL clock phase (VX to CDM link is now established)
* correct phase of CDM to VX LVDS data is adjusted by scan of CDM VX RX PLL clock phase (link is established after VX to CDM idle data pattern changes from "VX RX data bad" to "good".
* after good phases are found by scan, they are not expected to change unless cables are changed, CDM and VX modules are changed or CDM firmware is rebuilt. (rebuild of VX firmware should not affect LVDS data phase).
* if there is excessive link errors, phase scan must be repeated.

= Board test plan =

To test:
* Enclustra FPGA board
* SFP port - SFP_RS0/RS1 connected to QSFP0_SEL/1_SEL is wrong? our Finisar SFP says RS0, RS1 N/C, so probably okey.
* SW5 CLK_EXT1 NIM works. TTL needs to be tested. flipping SW5 CLK_EXT0 side from NIM to TTL makes CLK_EXT1 go LOS and OOF in the clock chip. R23 and R53 should be removed?
* BOOT_MODE 0 and 1

Partial:
* U23 3.3V current meter and thermometer. V1,V2 is current monitor, same as in application note. V3,V4 is thermometer, same in application note, except capacitor C118 is 0.1u instead of 470pF. Tested ok: Tint, VCC, V1, V2, V1-V2. Test failed: TR2 reads 50-something degC instead of same as Tint.

Done:
* LED_FP1A..D: tested ok. K.O. 15 sep 2022
* USB UART: tested ok. K.O. 15 sep 2022
* J4A, J4B, J5A, J5B LEMO inputs (NIM/TTL) EXT_IN_LV(1..4). TTL threshold 1.7V, NIM threshold -0.3V. 50 Ohm termination. TTL straight, NIM inverted.
* J6A, J6B LEMO clock inputs: 10 MHz TTL from chronobox works, 10 MHz NIM from IO32 works. 18-apr-2023
* J7A, J7B LEMO outputs EXT_OUT(1), EXT_OUT(2) (NIM/TTL) (tested 23nov2022, K.O.). TTL ???, NIM inverted.
** TTL out no 50 ohm termination: 0=0V, 1=5V, rise and fall time ~5 ns
** TTL out with 50 ohm termination: 0=0V, 1=2.5V, rise and fall time <2ns
** NIM out no 50 ohm termination: 0=+50mV, 1=-1.8V, rise and fall time ~3ns
** NIM out with 50 ohm termination: 0=0V, 1=-0.9V, rise and fall time <2ns
* ethernet MAC i2c chip (K.O. 20sep2022: can read: i2cdump 0 0x5b, nov2022 have u-boot driver)
* CDM VX ports 2x(CLK, 3 tx, 4 rx) tested using test_cdm.exe and LVDS loopback in VX firmware.
* SFP i2c tested KO 22jun2023
* QSFP i2c tested KO 22jun2023
* i2c testing complete 22jun2023
* QSFP rx,tx tested 26june2023. lane0,1,2 ok, lane3 tx not connected, laser is off. qsfp0,1,2,3 all lanes ok.
* SFP rx,tx tested 26june2023. rx and tx okey. LOS ok, mod_absent ok.

Failure:
* ethernet: does not connect to alliedtelesys switch. connects to my USB-eth adapter at 100 Mbit speed. uboot mii status reports connection speed oscillating between 1000, 100 and 10. K.O. 16-sep-2022
* SFP LOS and mod_absent are swapped (in the FPGA pin definitions?)
* QSFP slot numbering is wrong.

= Checklist for newly build boards =

== setup and boot ==
* put new board on workbench
* check - vme connector present, vme extraction handles present
* check - standoff are removed from all thru-holes
* plug Enclustra module
* check - SW1 both switches are in the "6" and "3" positions
* check - SW6 both switches are in the "PS" position
* Rev0: connect micro-usb cable from linux PC
* Rev1: connect usb-C cable from linux PC (USB C-to-C or C-to-A)
* connect ethernet from 1gige capable network switch
* connect power from lab power supply - GND, +5V and -12V nominal, +5.7V and -12.5V actual
* power up, +5V current 2.10-2.8A, -12V current 0.05A
* on linux PC, open a new terminal, run: minicom -D /dev/ttyACM0 -b 115200
* in minicom window, observe messages about Xilinx first stage boot loader, etc
* on the ethernet switch, observe network link is 1gige speed (not 10mbit, not 100mbit).
* if everything boots okey, there will be a login prompt, login as root, password root.
* alternatively, from root@daq13: ssh root@dsdm
* busybox devmem 0x80010000 # read firmware version number, i.e. 0xEDAD0A77
== test i2c ==
* check i2c: (no SFP, no QSFP plugged in!)
<pre>
root@dsdm:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@dsdm:~#
</pre>
* test clock chip:
<pre>
root@dsdm:~# ./test_cdm_local.exe --load-cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Loading CC registers...
Clock chip data block size 519
root@dsdm:~# ./test_cdm_local.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Polling CC status...
Clock chip state 1, status: LOS_IN0 OOF_IN0 IN1 IN_SEL_1
Ctrl-C
</pre>
* test RTC:
<pre>
root@dsdm:~# ./test_cdm_local.exe --rtc-read
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Read the RTC chip!
reg 0x00: 0 0 1 0 0 1 1 0 0x26
reg 0x01: 0 0 1 1 1 0 0 1 0x39
reg 0x02: 0 0 0 0 0 1 1 1 0x07
reg 0x03: 0 0 0 0 0 0 1 0 0x02
reg 0x04: 0 0 0 0 0 0 1 0 0x02
reg 0x05: 0 0 0 0 0 0 0 1 0x01
reg 0x06: 0 0 0 0 0 0 0 0 0x00
reg 0x07: 0 0 0 0 0 0 0 0 0x00
reg 0x08: 0 0 0 0 0 0 0 0 0x00
reg 0x09: 0 0 0 0 0 0 0 0 0x00
reg 0x10: 0 0 0 0 0 0 0 0 0x00
reg 0x11: 0 0 0 0 0 0 0 0 0x00
reg 0x12: 0 0 0 0 0 0 0 0 0x00
reg 0x13: 0 0 0 0 0 0 0 0 0x00
reg 0x14: 0 0 0 1 1 0 0 0 0x18
reg 0x15: 0 0 0 0 1 0 0 0 0x08
reg 0x16: 0 0 0 0 0 0 0 0 0x00
reg 0x17: 0 0 0 1 0 1 1 1 0x17
reg 0x18: 0 1 0 0 0 0 0 0 0x40
seconds: 26, minutes: 39, hours: 07, day 2, date: 00-01-02, temp 23.25C
root@dsdm:~#
</pre>

== test clocks ==

<pre>
root@dsdm:~# ./test_cdm_local.exe --reset-mgt
...
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735cb9 (125000889) should be ~125 MHz
0x1034 rx_clk: 0x07735cb9 (125000889) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735cb9 (125000889) should be ~125 MHz
0x103C tx_clk: 0x07735cb9 (125000889) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
^C
root@dsdm:~# ./test_cdm_local.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735c6a (125000810) should be ~125 MHz
0x1034 rx_clk: 0x07735c69 (125000809) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735c69 (125000809) should be ~125 MHz
0x103C tx_clk: 0x07735c6a (125000810) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
^C
root@dsdm:~#
</pre>

== test LED, LEMO ==

* test LEDs, observe all 4 LEDs turn on and off every 1 second
<pre>
root@dsdm:~# ./test_cdm_local.exe --blink-led
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
LED on
LED off
</pre>
* TBW - test LEMO outputs
<pre>
have test firmware lemo outputs are RTC chip 1pps and 48kHz
see them on the scope
</pre>
* TBW - test LEMO inputs
<pre>
./test_cdm_local.exe --monreg 6
</pre>

== test VX ==

* on the CDM, connect all split blue cables to VX modules
* set all VX modules to LVDS loopback mode (in VX settings MIDAS page)
* ssh root@cdm00
* run "./test_cdm.exe --test-vx", it should report:
<pre>
CDM registers: 0x0000000000000000 0x00000000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000eeeeeeeeeeee 0x00007700: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000080808080808 0x00000100: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000040404040404 0x00000200: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000020202020202 0x00000400: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000000000000000 0x00000800: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000808080808080 0x00001000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000404040404040 0x00002000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000202020202020 0x00004000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000000000000000 0x00008000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
all on!
all off!
</pre>
* on the VX settings MIDAS page LVDS input section, observe line status is cycling on and off.
* note: VX_RX(0) is not tested by this, corresponding VX_TX is used as the VX clock

== test SFP (CDM) ==

* connect SFP module
* ssh root@cdm00
* run "./test_cdm.exe --sfp", it should report:
<pre>
root@cdm00:~# ./test_cdm.exe --sfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x6a332b58, ds20k version 0x20241209
DS-DM firmware build 0x6a332b58, ds20k version 0x20241209
Polling SFP status...
identifier 0x03
connector 0x07
encoding 0x01
wavelength 0x0352 (850 nm)
vendor_name [FINISAR CORP. ]
vendor_pn [FTLF8526P3BNL ]
vendor_rev [A ]
vendor_sn [N3AB4LV ]
vendor_date [200319 ]
dm_type 0x68
temp 30.9 C
vcc 3.285 V
tx_bias 7.362 mA
tx_power 475.7 uW
rx_power 859.1 uW
SFP good 1, status: temp 30.9 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 859 uW
SFP good 1, status: temp 30.8 C, tx_bias 7.4 mA, tx_power 475 uW, rx_power 859 uW
SFP good 1, status: temp 30.8 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 859 uW
SFP good 1, status: temp 30.7 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 859 uW
SFP good 1, status: temp 30.8 C, tx_bias 7.4 mA, tx_power 475 uW, rx_power 858 uW
^C
root@cdm00:~#
</pre>
* NOTE1: if fiber cable is disconnected or GDM is not running, rx_power would read 0 uW
* NOTE2: better SFP modules have smaller tx_bias and larger tx_power (more bang for the buck)
* NOTE3: bad fiber cable and bad fiber connection will result in smaller rx_power

== test QSFP (GDM) ==

* connect QSFP module (any slot)
* ssh root@gdm01
* run "./test_cdm.exe --qsfp3 --qsfp", try --qsfp0, --qsfp1, --qsfp2, --qsfp3 for the 4 QSFP slots
<pre>
root@gdm01:~# ./test_cdm.exe --qsfp3 --qsfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x9e30baf9, ds20k version 0x20241209
DS-DM firmware build 0x9e30baf9, ds20k version 0x20241209
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out hi
Polling QSFP status...
identifier 0x0d
status 0x02
los 0x89
temp 36.7 C
vcc 3.302 V
rx_power 0.1 413.7 460.3 0.1 uW
tx_bias 7.8 7.9 7.9 0.0 mA
tx_power 806.0 842.9 865.2 0.1 uW
vendor_name [FINISAR CORP ]
vendor_pn [FTL410QD4C ]
vendor_rev [A ]
wavelength 850
max_temp 70 C
vendor_sn [X79AC0R ]
vendor_date [220309 ]
QSFP good 1, status: temp 36.7 C, los 0x89, tx_bias 7.8 7.9 7.9 0.0 mA, tx_power 806 843 865 0 uW, rx_power 0 414 460 0 uW
^C
</pre>
* NOTE1: 4th tx_bias and tx_power are zero because 4th channel does not transmit any data, QSFP detects this and automatically shuts down the optical transmitter

= Serial console =

* check that linux computer has correct udev rules to allow access to /dev/ttyACM devices, see https://daq00.triumf.ca/DaqWiki/index.php/SLinstall#Configure_USB_device_permissions and https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Configure_USB_device_permissions
* connect micro-USB cable to connector J-UCB, other end connect to linux computer
* observe /dev/ttyACM0 was created
* run "minicom -D /dev/ttyACM0 -b 115200"
* should have gdm-cdm login
* username root, password root

= i2c =

<pre>
ZynqMP> i2c bus
Bus 0: i2c@ff020000
ZynqMP> i2c dev 0
Setting bus to 0
ZynqMP> i2c probe
Valid chip addresses: 33 4E 53 5B 6B 77
ZynqMP> i2c md 0x5b 0x98
0098: fc c2 3d 00 00 12 1a 6e 0a 90 85 04 94 10 08 50 ..=....n.......P
</pre>

<pre>
root@gdm-cdm:~# i2cdetect 0
Warning: Can't use SMBus Quick Write command, will skip some addresses
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0.
I will probe address range 0x03-0x77.
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f
00:
10:
20:
30: -- -- -- 33 -- -- -- --
40:
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60:
70:
root@gdm-cdm:~#
root@gdm-cdm:~# i2cdump 0 0x5b
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0, address 0x5b, mode byte
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00 ???????P?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e ........??=..??n
a0: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00 ???????P?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e ........??=..??n
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@gdm-cdm:~#
</pre>

<pre>
root@gdm0:~# i2cdetect -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#
</pre>

<pre>
root@cdm0:~# i2cdetect -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: 50 51 -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@cdm0:~#
</pre>

* 0x33 - XU8 secure EEPROM (should be at 0x32)
* 0x4e - U23 current and temperature monitor
* 0x50, 0x51 - SFP
* 0x50 - QSFP, 4 QSFP modules enabled by GPIO QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL
* 0x53, 0x5b - ethernet mac eeprom
* 0x6b - U6 clock chip

== U23 ==

NOTE: Text reads double of Tint. not sure why. K.O. 21-mar-2024.

* internal temperature only

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x00 b # control register: "repeat mode, internal temperature only"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x03 # "Tint ready" and "busy", "busy is always 1 in repeat mode"
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 00 03 03 81 db 2a ce 2a 8e 00 6f 00 45 20 3f ?.????*?*?.o.E ?
10: 01 00 01 01 01 db 2a ce 2a 8e 00 6f 00 45 20 3f ?.????*?*?.o.E ?
...
readback:
reg0 - 03 - Tint ready
reg1 - 00 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB 0x81, bit 0x80 is "DV, data valid", bit 0x40 is "SS, sensor short", 0x20 is "SO, sensor open"
reg5 - Tint LSB 0xDB, Tint = 0x01DB = 475 * 0.0625 degC = 29.6 degC
reg6..F - stale data
</pre>

* Tint, V1, V2, TR2, VCC

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x18 b # control register: "repeat mode, V1, V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 7f 18 7f 7f 81 b9 aa d0 aa 8f 83 5e 83 5e a0 41 ???????????^?^?A
10: 01 18 01 01 01 b9 2a d0 2a 8f 03 5e 03 5e 20 41 ??????*?*??^?^ A
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x1b9*0.0625 = 27.5 degC
reg6 - V1 MSB 0xaa, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 LSB 0xd0, V1 = 0x2ad0*305.18/1000000 = 3.3447 V (correct)
reg8 - V2 MSB 0xaa, ditto
reg9 - V2 LSB 0x8f, V2 = 0x2a8f*305.18/1000000 = 3.3249 V (correct, smaller than V1)
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x5e. TR2 = 0x35e*0.0625 = 53.875 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x41, VCC = 2.5+0x2041*305.18/1000000 = 5.019 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.99 A (about right?)
</pre>

* Tint, V1-V2, TR2, VCC

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x19 b # control register: "repeat mode, V1-V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 7f 19 7f 7f 81 69 83 f2 83 f2 83 3a 83 3a a0 44 ?????i?????:?:?D
10: 01 19 01 01 01 69 03 f2 03 f2 03 3a 03 3a 20 44 ?????i?????:?: D
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x169*0.0625 = 22.5 degC
reg6 - V1 or V1-V2 MSB 0x83, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 or V1-V2 LSB 0xf2, V1-V2 = 0x3f2*19.42/1000000 = 0.0196142 V (correct, compare with V1 and V2 measured above)
reg8 - V2 or V1-V2 MSB
reg9 - V2 or V1-V2 LSB
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x3a. TR2 = 0x33a*0.0625 = 51.625 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x44, VCC = 2.5+0x2044*305.18/1000000 = 5.021 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.9807 A (about right?)
</pre>

== SFP ==

<pre>
root@cdm0:~# i2cdump 0 0x50
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 04 07 00 00 00 00 40 40 0c 00 01 3d 00 00 00 ???....@@?.?=...
10: 0c 02 00 1e 46 49 4e 49 53 41 52 20 43 4f 52 50 ??.?FINISAR CORP
20: 2e 20 20 20 00 00 90 65 46 54 4c 46 38 35 32 36 . ..?eFTLF8526
30: 50 33 42 4e 4c 20 20 20 41 20 20 20 03 52 00 9d P3BNL A ?R.?
40: 00 1a 00 00 4e 33 41 42 34 4c 56 20 20 20 20 20 .?..N3AB4LV
50: 20 20 20 20 32 30 30 33 31 39 20 20 68 f0 03 de 200319 h???
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@cdm0:~#
root@cdm0:~# i2cdump 0 0x51
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 5a 00 d8 00 55 00 e2 00 90 88 71 48 8c a0 75 30 Z.?.U.?.??qH??u0
10: 21 34 01 f4 1b 58 03 e8 31 2d 04 eb 1f 07 06 31 !4???X??1-?????1
20: 31 2d 00 64 27 10 00 9e 00 00 00 00 00 00 00 00 1-.d'?.?........
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
40: 00 00 00 00 3f 80 00 00 00 00 00 00 01 00 00 00 ....??......?...
50: 01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 b7 ?...?...?......?
60: 19 9f 80 c5 0e 17 12 c4 1f 99 00 00 00 00 30 00 ??????????....0.
70: 00 00 00 00 00 00 00 00 ff ff ff ff ff ff ff 01 ...............?
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@cdm0:~#
</pre>

== QSFP ==

<pre>
QSFP i2c enable lines, active low:

QSFP0_SEL - JENC-A 82 - PS-MIO40 - linux gpio 378
QSFP1_SEL - JENC-A 84 - PS-MIO41 - linux gpio 379
QSFP2_SEL - JENC-A 100 - PS-MIO44 - linux gpio 382
QSFP3_SEL - JENC-A 106 - PS-MIO43 - linux gpio 381 (notice 2 and 3 are out of order)
</pre>

<pre>
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
# echo 378 >> /sys/class/gpio/export ### SEL0 338+40
# echo 379 >> /sys/class/gpio/export ### SEL1 338+41
# echo 381 >> /sys/class/gpio/export ### SEL3 338+43
# echo 382 >> /sys/class/gpio/export ### SEL2 338+44
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) in hi
gpio-379 ( |sysfs ) in hi
gpio-381 ( |sysfs ) in hi
gpio-382 ( |sysfs ) in hi
root@gdm0:~# echo out >> /sys/class/gpio/gpio381/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio382/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio378/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio379/direction
root@gdm0:~#
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out lo
gpio-379 ( |sysfs ) out lo
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out lo
root@gdm0:~#
echo 1 >> /sys/class/gpio/gpio381/value
echo 1 >> /sys/class/gpio/gpio382/value
echo 1 >> /sys/class/gpio/gpio378/value
echo 1 >> /sys/class/gpio/gpio379/value
cat /sys/kernel/debug/gpio
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out hi
gpio-382 ( |sysfs ) out hi
root@gdm0:~#
root@gdm0:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#

NOTICE NOTHING AT ADDRESS 0x50
</pre>

<pre>
ENABLE QSFP0, OBSERVE IT IS AT ADDRESS 0x50

root@gdm0:~# echo 0 >> /sys/class/gpio/gpio378/value
root@gdm0:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: 50 -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#
root@gdm0:~# i2cdump -y 0 0x50
No size specified (using byte-data access)
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 0d 00 02 8e 00 00 01 00 00 05 55 00 05 00 00 00 ?.??..?..?U.?...
10: 00 00 00 00 00 00 1e b8 00 00 81 a0 00 00 00 00 ......??..??....
20: 00 00 12 12 00 01 00 01 00 01 0e e0 0f 20 0e e0 ..??.?.?.???? ??
30: 00 00 1f a0 1e 3b 1f 72 00 01 00 00 00 00 00 00 ..???;?r.?......
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
60: 00 00 00 00 00 00 00 00 00 00 1f 00 00 00 08 00 ..........?...?.
70: 00 00 00 00 00 00 00 ff ff ff ff ff ff ff ff 00 ................
80: 0d 00 0c 04 00 00 00 40 40 02 d5 05 67 00 00 96 ?.??...@@???g..?
90: 00 00 c8 00 46 49 4e 49 53 41 52 20 43 4f 52 50 ..?.FINISAR CORP
a0: 20 20 20 20 07 00 90 65 46 54 4c 34 31 30 51 44 ?.?eFTL410QD
b0: 34 43 20 20 20 20 20 20 41 20 42 68 07 d0 00 43 4C A Bh??.C
c0: 00 07 0f de 58 37 39 41 43 30 52 20 20 20 20 20 .???X79AC0R
d0: 20 20 20 20 32 32 30 33 30 39 20 20 3c 10 00 9e 220309 <?.?
e0: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@gdm0:~#
</pre>

== ethernet mac eeprom ==

* correct chip with 84-bit ethernet mac address

<pre>
root@cdm1:~# i2cdump 0 0x53
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
root@cdm1:~#

root@cdm1:~# i2cdump 0 0x5b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00 ????4??2?K?.?...
90: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c ..........??=?Q<
a0: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00 ????4??2?K?.?...
b0: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c ..........??=?Q<
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@cdm1:~#
</pre>

* wrong "602" chip with 64-bit IPv6 address

<pre>
root@cdm0:~# i2cdump 0 0x53
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
root@cdm0:~# i2cdump 0 0x5b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00 ???????Q?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e ........??=..??.
a0: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00 ???????Q?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e ........??=..??.
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@cdm0:~#
</pre>

= read ethernet mac address from i2c =

(this code is copied from uboot command line i2c code)

in uboot sources board/xilinx/common/board.c replace original function with this:

<pre>
// special code to read ethernet MAC address from the DS-DM-Rev0 board. K.O. Sep-2022

int zynq_board_read_rom_ethaddr(unsigned char *ethaddr)
{
struct udevice *bus;
int ret;
int busnum = 0;

ret = uclass_get_device_by_seq(UCLASS_I2C, busnum, &bus);
if (ret) {
printf("%s: No bus %d\n", __func__, busnum);
return ret;
}

int chip_addr = 0x5B;

struct udevice *dev;

ret = i2c_get_chip(bus, chip_addr, 1, &dev);
if (ret) {
printf("%s: Bus %d no chip 0x%02x\n", __func__, busnum, chip_addr);
return ret;
}

int dev_addr = 0x98;

unsigned char data[8];

ret = dm_i2c_read(dev, dev_addr, data, 8);

if (ret) {
printf("%s: Bus %d chip 0x%02x read error %d\n", __func__, busnum, chip_addr, ret);
return ret;
}

printf("%s: Bus %d chip 0x%02x addr 0x%02x read: 0x %02x %02x %02x %02x %02x %02x %02x %02x\n", __func__, busnum, chip_addr, dev_addr, data[0], data[1], data[2], data[3], data[4], data[5], data[6], data[7]);

// see Atmel-8807-SEEPROM-AT24MAC402-602-Datasheet.pdf

if (data[0] == 0) {
// eiu-48 chip
ethaddr[0] = data[2];
ethaddr[1] = data[3];
ethaddr[2] = data[4];
ethaddr[3] = data[5];
ethaddr[4] = data[6];
ethaddr[5] = data[7];
} else {
// eiu-64 chip
ethaddr[0] = data[0];
ethaddr[1] = data[1];
ethaddr[2] = data[2];
ethaddr[3] = data[5];
ethaddr[4] = data[6];
ethaddr[5] = data[7];
}

printf("%s: ethaddr %02x:%02x:%02x:%02x:%02x:%02x\n", __func__, ethaddr[0], ethaddr[1], ethaddr[2], ethaddr[3], ethaddr[4], ethaddr[5]);

return ret;
}
</pre>

also this should have worked if i2c_xxx() functions were enabled in uboot:
<pre>
i2c_set_bus_num(0);
i2c_probe(0x5b);
i2c_read(0x5b, 0x9a, ethaddr, 6);
</pre>

= read ethernet mac address from i2c (SHOULD WORK) =

from: https://stackoverflow.com/questions/43637540/is-there-linux-or-u-boot-support-to-read-a-mac-address-from-a-chip-at-startup

<pre>
#ethernet related setup
setup_eth=run readmac buildmac
#read mac address from eeprom
readmac=i2c dev 0; i2c read 50 FA.1 6 $loadaddr
#build the ethaddr variable
#not very nice, but does the job
buildmac=\
e=" "; sep=" " \
for i in 0 1 2 3 4 5 ; do\
setexpr x $loadaddr + $i\
setexpr.b b *$x\
e="$e$sep$b"\
sep=":"\
done &&\
setenv ethaddr $e
</pre>

= read ethernet mac address from i2c (DOES NOT WORK) =

this method does not work: inside board/xilinx/common/board.c:int zynq_board_read_rom_ethaddr(unsigned char *ethaddr), uclass_get_device_by_ofnode(UCLASS_I2C_EEPROM, eeprom, &dev) returns ENODEV and read from i2c does not happen. K.O. Sep-2022

Read:
* https://support.xilinx.com/s/question/0D52E00006hpKlsSAE/uboot-and-ethernet-mac-address-from-eeprom?language=en_US (last answer)
* https://github.com/Xilinx/u-boot-xlnx/blob/master/board/xilinx/common/board.c (zynq_board_read_rom_ethaddr())

Note:
* 0x5B is the i2c chip address
* 0x9A is the data offset inside the chip, see datasheet or i2c read dump above.

Edit:
* emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/u-boot/files/platform-top.h
<pre>
#include <configs/xilinx_zynqmp.h>
#include <configs/platform-auto.h>
//#define CONFIG_I2C_EEPROM
//#define CONFIG_SYS_I2C_EEPROM_ADDR 0x5b
//#define CONFIG_SYS_I2C_EEPROM_ADDR_OVERFLOW 0x0
#define CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET 0x9A
#error HERE!
</pre>
* emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi
<pre>
/include/ "system-conf.dtsi"
/ {
chosen {
xlnx,eeprom = &eeprom;
};
};

&i2c0 {
eeprom: eeprom@5b { /* u88 */
compatible = "atmel,24mac402";
reg = <0x5b>;
};
};
</pre>
* components/yocto/workspace/sources/u-boot-xlnx/configs/xilinx_zynqmp_virt_defconfig
<pre>
CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET=0x9A
</pre>

= RTC chip =

* DS3231 RTC chip
* FPGA connections:
<pre>
I2C SCL <- J-ENC A85 <- FPGA E17 <- XDC TP_S <- VHDL RTC_I2C_SCL output (10k pull-up to 3.3V)
I2C SDA <-> J-ENC A87 <-> FPGA D17 <-> XDC TP_S <-> VHDL RTC_I2C_SDA bidir (10k pull-up to 3.3V)
RTC_1Hz -> J-ENC C160 -> FPGA AH12 -> XDC "slow_io" -> VHDL RTC_1PPS input (10k pull-up to 1.2V)
RTC_32k -> J-ENC B129 -> FPGA AE3 -> XDC "free pin" -> VHDL RTC_32KHZ input (10k pull-up to 1.8V)
</pre>
* test notes
<pre>
FPGA registers:

rtc_i2c_clock_ie <= not register_data_in_block_0(14)(0);
rtc_i2c_clock_out <= register_data_in_block_0(14)(1);
rtc_i2c_data_ie <= not register_data_in_block_0(14)(4);
rtc_i2c_data_out <= register_data_in_block_0(14)(5);

ext_out(1) <= rtc_1pps_in;
ext_out(2) <= rtc_32khz_in;

register_data_out_block_0(13) <= x"000000" & rtc_i2c_data_ie & rtc_i2c_data_out & rtc_i2c_clock_ie & rtc_i2c_clock_out & rtc_1pps_in & rtc_32khz_in & rtc_i2c_data_in & rtc_i2c_clock_in;

After power us on the RTC chip:
i2c clock 3.3V
i2c data 3.3V
32kHz 0.778V see clock running with amplitude 0->1.8V (FPGA pullup) period about 30 usec (~33 kHz)
1pps 1.2V (FPGA pullup)

./test1.exe --read32 0x34
dsdm_read32[0x00000034] is 0x000000ab (171) -> 1010'1011 -> data_ie, clock_ie, 1pps, data_in, clock_in
./test1.exe --read32 0x38
dsdm_read32[0x00000038] is 0x00000000 (0)

./test1.exe --write32 0x38 0x01 -> clock oe
./test1.exe --read32 0x34
Ddsdm_read32[0x00000034] is 0x0000008a (138) -> 1000'1010 -> data_ie, 1pps, data_in, clock 0V, data 3.3V

./test1.exe --write32 0x38 0x10 -> data_oe
./test1.exe --read32 0x34
dsdm_read32[0x00000034] is 0x00000029 (41) -> 0010'1001 -> clock_ie, 1pps, clock_in, clock 3.3V, data 0V

root@dsdm:~# ./test_cdm_local.exe --rtc-read
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Read the RTC chip!
reg 0x00: 0 1 0 1 0 1 1 1 0x57
reg 0x01: 0 0 1 0 0 0 1 0 0x22
reg 0x02: 0 0 0 0 0 0 0 0 0x00
reg 0x03: 0 0 0 0 0 0 0 0 0x00
reg 0x04: 0 0 0 0 0 0 0 0 0x00
reg 0x05: 0 0 0 0 0 0 0 0 0x00
reg 0x06: 0 0 0 0 0 0 0 0 0x00
reg 0x07: 0 0 0 0 0 0 0 0 0x00
reg 0x08: 0 0 0 0 0 0 0 0 0x00
reg 0x09: 0 0 0 0 0 0 0 0 0x00
reg 0x10: 0 0 0 0 0 0 0 0 0x00
reg 0x11: 0 0 0 0 0 0 0 0 0x00
reg 0x12: 0 0 0 0 0 0 0 0 0x00
reg 0x13: 0 0 0 0 0 0 0 0 0x00
reg 0x14: 0 0 0 1 1 0 0 0 0x18
reg 0x15: 0 0 0 0 1 0 0 0 0x08
reg 0x16: 0 0 0 0 0 0 0 0 0x00
reg 0x17: 0 0 0 1 0 1 1 1 0x17
reg 0x18: 1 1 0 0 0 0 0 0 0xc0
seconds: 57, minutes: 22, hours: 00, day 0, date: 00-00-00, temp 23.75C
root@dsdm:~#
</pre>

= clock chip configuration =

file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt:

* VCO is 14 GHz
* Tvco is 71.43 ps
* N0 divider is 14, frequency is 1000 MHz
* out0 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz
* out1 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
* out2 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
* out3 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz is the feedback for zero delay

= clock chip monitoring =

from si5395-94-92-family.pdf:
* reg 0x1: page select, set to 0 or set to 5 to read 0x53F
* reg 0x2: 0x94
* reg 0x3: 0x53 -> device is a si5394
* reg 0xC: LOSXAXB
* reg 0xD: LOS and OOF for the 4 clock inputs
* reg 0xE: LOL and HOLD
* reg 0xF: CAL_PLL
* reg 0x11: sticky bits for reg 0xC
* reg 0x12: sticky bits for reg 0xD
* reg 0x13: sticky bits for reg 0xE
* reg 0x14: sticky bits for reg 0xF
* reg 0x1C: device reset
* reg 0x1E: low power, hard reset, SYNC
* reg 0x507: currently selected input clock
* reg 0x52A: input clock select
* reg 0x535: FORCE_HOLD
* reg 0x53F: HOLD_HIST_VALID and FASTLOCK_STATUS

= Install Xilinx tools =

* install Vivado 2020.2
<pre>
login at https://www.xilinx.com/myprofile.html
go to "Downloads"
go to archive,
find 2020.2
download Xilinx_Unified_2020.2_1118_1232_Lin64.bin
sh ./Xilinx_Unified_2020.2_1118_1232_Lin64.bin
banner window should open with spinner "downloading installation data"
"a newer version is available" -> say "continue"
next
"select install type" window:
provide email and password,
select "download image"
select directory /home/olchansk/Xilinx/Downloads/2020.2\
select "linux" and "full image"
next
download summary: space required 38.52 Gbytes
download
installation progress
downloading spinner, 16 M/s 47 minutes...
"download image has been created successfully". Ok.
check contents of /home/olchansk/Xilinx/Downloads/2020.2
ls -l /home/olchansk/Xilinx/Downloads/2020.2
total 67
drwxr-xr-x 2 olchansk users 9 Sep 1 16:22 bin
drwxr-xr-x 3 olchansk users 15 Sep 1 16:23 data
drwxr-xr-x 4 olchansk users 4 Sep 1 16:22 lib
drwxr-xr-x 2 olchansk users 644 Sep 1 16:22 payload
drwxr-xr-x 2 olchansk users 7 Sep 1 16:22 scripts
drwxr-xr-x 4 olchansk users 4 Sep 1 16:22 tps
-rwxr-xr-x 1 olchansk users 3256 Nov 18 2020 xsetup
daq13:2020.2$
./xsetup
spinned loading installation data
xilinx design tools 2022.1 now available -> say continue
"welcome" -> next
"select product" -> vivado -> next -> vivado hl system edition -> next
select devices: only zynq ultrascale+ mpsoc -> next
select destination: /opt/Xilinx (as root, mkdir /opt/Xilinx, chmod olchansk.users /opt/Xilinx)
install ...
complete
move /home/olchansk/Xilinx/Downloads/2020.2 to /daq/daqstore/olchansk/Xilinx/Downloads/
</pre>
* install petalinux 2020.2
<pre>
./xsetup
"a newer version is available" -> say "continue"
next
"select product to install" -> select Petalinux (Linux only) -> next
"select destination directory" -> select "/opt/Xilinx" (disk space required 2.64 GB) -> next
"summary" -> install ...
error about missing /tmp/tmp-something files
"installation completed successfully" (hard to dismiss, "ok" button is partially cut-off)
done?
I think it failed, /opt/Xilinx/PetaLinux/2020.2/bin is empty except for petalinux-v2020.2-final-installer.run
try to run it by hand, same error about /tmp/tmp-something files. strange...
notice it complains about "truncate", which truncate finds ~/bin/truncate, get rid of it,
try again
now complains about missing texinfo and zlib1g:i386
apt install texinfo -> ok
apt install zlib1g:i386 -> installs bunch of gcc stuff -> ok
try again
reports "already installed" -> delete /opt/Xilinx/.xinstall/PetaLinux_2020.2/, delete entries in ~/.Xilinx/registry/installedSW.xml
try again
success
</pre>
* install vivado 2022.1 and petalinux 2022.1 - everything is pretty much the same

= Petalinux =

* cd PetaLinux_GDM_CDM
* petalinux-config
* enable i2c MAC address and DHCP
<pre>
git clone https://bitbucket.org:/team-ds-dm/ds-dm-u-boot-xlnx.git
cd ds-dm-u-boot-xlnx
git checkout ds-dm-u-boot-xlnx

linux-components ->
uboot -> ext-local-src
external u-boot local source -> ds-dm-u-boot-xlnx (path to the customized uboot git repository)
</pre>
* enable DHCP
<pre>
Subsystem AUTO Hardware Settings -> Ethernet Settings
randomize MAC address -> NO
ethernet mac address -> leave empty
obtain ip address automatically -> YES
</pre>
* set hostname and product names
<pre>
Firmware Version Configuration ->
Host name -> "ds-dm"
Product name -> "Petalinux_GDM_CDM"
</pre>
* configure linux kernel
<pre>
petalinux-config -c kernel
</pre>
* enable NFS-Root
<pre>
petalinux-config
Image Packaging Configuration > Root File System Type -> set to NFS
Location of NFS root directory set to "/nfsroot"

petalinux-config -c kernel
Networking support > IP: kernel level configuration
enable DHCP, BOOTP, RARP
File systems > Network file systems > Root file systems on NFS
</pre>
* manually fix linux kernel command line:
<pre>
grep nfsroot PetaLinux_GDM_CDM/project-spec/configs/config
edit CONFIG_SUBSYSTEM_BOOTARGS_GENERATED to read
earlycon console=ttyPS0,115200 clk_ignore_unused panic=60 root=/dev/nfs nfsroot=/nfsroot/%s ip=dhcp rw
</pre>
* check configuration in
** PetaLinux_GDM_CDM/project-spec/configs/config
** PetaLinux_GDM_CDM/project-spec/configs/rootfs_config
** PetaLinux_GDM_CDM/components/plnx_workspace/device-tree/device-tree/system-conf.dtsi

= JTAG server =

localhost:3121

= ds20k block =

<pre>
module ds20k
(
// CLOCK INPUTs
input wire clk,
input wire reset, // pulse for power-up reset
input wire si5394_loln, // clock cleaner PLL is locked to selected input clock

// REGISTER_DATA

input wire [255:0] [31:0] register_data_in,
output reg [255:0] [31:0] register_data_out,
input wire register_we,
input wire register_re,

// GDM QSFP FIBER LINKS
input wire qsfp_tx_data_ready,
output reg [11:0] [15:0] qsfp_tx_data,
output reg [11:0] [1:0] qsfp_tx_data_is_k,
input wire [11:0] [15:0] qsfp_rx_data_error, // not sure what this is.
input wire [11:0] [15:0] qsfp_rx_data,
input wire [11:0] [1:0] qsfp_rx_data_is_k,

// CDM SFP FIBER LINKS
input wire sfp_tx_data_ready,
output reg [15:0] sfp_tx_data,
output reg [1:0] sfp_tx_data_is_k,
input wire sfp_rx_data_error, // single bit indicating that RX link is up and data is good.
input wire [15:0] sfp_rx_data,
input wire [1:0] sfp_rx_data_is_k,

// VX_RXs
input wire [3:0] vx1_rx_async,
input wire [3:0] vx2_rx_async,
input wire [3:0] vx3_rx_async,
input wire [3:0] vx4_rx_async,
input wire [3:0] vx5_rx_async,
input wire [3:0] vx6_rx_async,
input wire [3:0] vx7_rx_async,
input wire [3:0] vx8_rx_async,
input wire [3:0] vx9_rx_async,
input wire [3:0] vx10_rx_async,
input wire [3:0] vx11_rx_async,
input wire [3:0] vx12_rx_async,

// VX_TXs
output reg [2:0] vx1_tx_out,
output reg [2:0] vx2_tx_out,
output reg [2:0] vx3_tx_out,
output reg [2:0] vx4_tx_out,
output reg [2:0] vx5_tx_out,
output reg [2:0] vx6_tx_out,
output reg [2:0] vx7_tx_out,
output reg [2:0] vx8_tx_out,
output reg [2:0] vx9_tx_out,
output reg [2:0] vx10_tx_out,
output reg [2:0] vx11_tx_out,
output reg [2:0] vx12_tx_out,

// VX serial comunications
input wire [11:0] vx_rx_ser_in,
output reg [11:0] vx_tx_ser_out,

// LEMO INPUTs
input wire [4:1] ext_in_async, // direct connection to LEMO connectors, not clocked

// LEMO OUTPUTs
output reg [2:1] ext_out, // direct connection to LEMO connectors, not clocked

// FRONT PANEL LEDs
output reg [3:0] fp_led_out, // direct connection to LEMO connectors, not clocked
);
</pre>

= world view =

Main components:
* clock distribution - to ensure all digitizers run synchronously and waveform timestamp are easy to assemble intophysics events.
* global trigger and run control - all digitizers are triggered at the same time to record calibration events, pulser events and the run startup synchronization sequence.
* busy distribution - when some digitizers go busy and stop accepting triggers, physics events may become incomplete, this must be recorded and managed.
* hitmap distribution - to create an online picture of the detector for monitoring and for triggering.

== global clock distribution ==

Clock distribution ensures all digitizers run synchronously (on the same clock) and waveform timestamps are easy to assemble into physics events. The clock source is synchronized with GPS to ensure a metrologically validated clock frequency (tied to the SI definition of 1 second) and date stamp (via GPS IRIG-B date stamp). Individual digitizer waveform timestamp are reset at the begin of run to establish a valid time zero (run start time, date stamped with the GPS IRIG-B date stamp).

Base clock is 125 MHz (8 ns). Individual digitizers are synchronized at initialization (power-up) time. Because of incontrollable delays in the electronics (i.e. variations between individual PLL chips in each digitizer), waveform timestamps between individual digitizers may be off by 1-2 clocks (8-16 ns). After the system is running and is stable (thermally and electrically), timestamp waveform variation and jitter are less than 1 ns.

This is validated by feeding a common calibration pulser signal into analog inputs of each digitizer (at 100 Hz) and measuring (fitting) the pulse time in the waveform. With sufficient statistics, differences between waveform timestamp are measured with picosecond precision.

CDM PLL allows fine phase adjustment of the 62.5 MHz VX2746 clock in groups of 6 clocks (2 groups per CDM) with precision 71.43 ps.

Due to FPGA technology limitations (serial link hard IPs, PLLs, etc) the pattern of 1-2 clock delays between individual digitizers may change/shift from power up to power up. Significant additional work would be required to ensure this pattern is repeatable and stable after every power up and initialization. As mentioned before, after power up and initialization, variation between waveform timestamp is sub-nanosecond as measured by the analog calibration pulser.

Clock chain:
* GPS receiver (1pps clock reference and IRIG-B date stamp)
* LNGS GPS clock distribution (fiber link to converter box at the experiment)
* GPS 1pps to PRS-10 Rb atomic clock (10 MHz clock output)
* 10 MHz clock from PRS-10 to GDM PLL (extra low jitter oscillator, 125 MHz clock output)
* 125 MHz clock to GDM fiber link transmitters (2.5 Gbps line rate)
* CDM fiber link receiver and clock recovery (125 MHz clock)
* 125 MHz recovered clock to CDM PLL (extra low jitter oscillator, 62.5 MHz clock output)
* 62.5 MHz CDM clock to VX2745 PLL via LVDS line
* VX2745 PLL (125 MHz clock output)
* 125 MHz clock to digitizer chips, clock counter and waveform timestamp.

This is the normal mode of operation, with global clock synchronized with the GPS clock (metrologically referenced to SI 1 Hz definition).

If GPS signal is not available, degraded modes of operation are available (in order of worsening performance)
* PRS-10 Rb atomic clock synchronized with a battery-powered time-of-day chip
* PRS-10 Rb atomic clock synchronized with Internet time (NTP) assuming Internet network connection is up
* PRS-10 Rb clock standalone
* GDM internal oscillator

All degraded modes provide high precision low jitter 125 MHz system clock, but long term stability (seconds per day) will vary. (TBM)

In the absence of GPS IRIG-B date stamp, the battery-powered time-of-day chip will be used (TBI) with fallback to Internet time (NTP). Date stamp long term stability is expected to be better than 1 second per day (TBI, TBM).

Note:
* TBM: to be measured
* TBI: to be implemented

Current status (April 2025):
* full clock chain from GDM PLL to VX2745 digitizers is implemented and tested
* waveform timestamp variations are measured (see report from Marek)
* GDM external clock is validated (using VME-GRIFFIN-CDM high quality clock source module)
* PRS-10 10 MHz and 1pps connection to GDM is implemented and tested
* GPS receiver 1pps connection to PRS-10 is implemented and tested
* GPS receiver IRIG-B date stamp connection to GDM tested using a software decoder. Hardware decoder and connection to time slice marker, TBI.
* LNGS date stamp decoder and connection to time slice marker, TBI
* LNGS 1pps connection to PRS-10, not available
* operate vertical slize system from GPS, not available (GPS is in the det lab, vertical slice is in the daq lab).

== global trigger distribution ==

Global trigger distribution provides a synchronous waveform trigger to all VX2745, all CDM and all VX2745 are provided wiht the same data and are all triggered at the same time (without accounting for the 1-2 clock offset described in the clock section).

Global trigger is used for run control: the first trigger of a run resets the waveform timestamp counter, the CDM timestamp counter and the GDM timestamp counter. At the begin of run, GDM generates a sequence of triggers following an easy to identify pattern. If a VX2745 misses a trigger (for any reason), online and offline software can easily identify the tru run start time using this pattern.

After a run has started, global trigger can be generated by a programmable pulser, by an external LEMO signal (i.e. external calibration signal, laser Q-switch, etc) and by the VX2745 trigger hitmap.

Trigger sources (TRG):
* TRG pulser
* LEMO inputs
* hitmap trigger

Time slice marker sources (TSM):
* TSM pulser

Trigger chain, K-code and LVDS:
* TRG/TSM source
* TRG/TSM K-code from GDM to CDM
* CDM K-code to LVDS line 12 and 13 lines (TRG and TSM)

Latency from GDM LEMO input to CDM LVDS out: TBM

Trigger chain, packet trigger:
* GDM TRG/TSM source to TRG/TSM packet generator
* or GDM hitmap group trigger packet generator
* TRG/TSM packet from GDM to CDM
* TRG/TSM packet from CDM to VX2745 via LVDS line, 62.5 MHz, 8b10b encoding.

Latency from GDM LEMO input to VX2745 internal trigger: TBM

Notes:
TBM - to be measured
TBI - to be implemented

Current status (Jan 2026):
* LVDS TRG and TSM distribution is done and tested
* packet TRG, TRIG48 and TSM distribution from GDM to CDM is done and tested
* packet TRG, TRIG48 and TSM from CDM to VX2745 is done and tested
* packet TRIG48 processing in the VX is done and tested

== global busy distribution ==

If any VX2745 goes busy (for any reason) and stops accepting triggers, physics data may become incomplete. This situation needs to be recorded and managed.

Busy chain:
* VX2745 firmware busy logic (CAEN and Yair) generates the VX busy signal on LVDS line 9 (CDM VX_RX_2)
* CDM monitors VX busy
* CDM records busy transitions
* CDM computes the "CDM busy" as grand-OR of all VX busy
* CDM periodically transmits CDM busy status the GDM using special K-codes (NB: confirm transmission period)
* GDM receives CDM busy updates and maintains "CDM busy" for each CDM
* GDM records busy transitions
* GDM computes "GDM busy" as grand-OR of all CDM busy
* effectively, GDM busy is a grand-OR of all VX busy (with max delay, TBM)

In the pilot implementation, any VX busy stops the experiment by sending a veto signal to all VX2745 which stops them from accepting triggers.

Veto chain:
* GDM computes "GDM veto" as GDM busy
* GDM periodically transmits GDM veto status to all CDMs using special K-codes (NB: confirm transmission period)
* CDM receive GDM veto updates and maintain the "CDM veto" signal
* CDM computes "VX veto" as CDM veto
* CDM transmits VX veto to all VX2745 on LVDS pair 13
* VX2745 firmware (Yair) has VX veto block all physics triggers (NB: except for the time slice marker?)

Total delay from VX busy to VX veto (on the LVDS lines), TBM.

Notes:
* TBM - to be measured
* TBI - to be implemented

Current status (Jan 2026):
* full busy chain from VX to CDM to GDM is implemented, tested
* veto chain from GDM to CDM is implemented, tested
* veto chain from CDM to LVDS pair 13 is implemented, tested
* LVDS pair 13 veto inside the VX is implemented, tested
* TRG switched from LVDS pair 12 to TRG packet, testing completed
* TSM switched from LVDS pair 13 to TSM packet, testing completed
* fine tuning of busy chain, VX buffer levels, etc, test in progress

== vx hitmap and hitmap trigger ==

To monitor the system in real time and to generate group triggers, we use the VX hitmaps.

Each VX2745 has 64 channels, each channel has a trigger logic block (digital discriminator) that generates a waveform self-trigger and sends a hitmap bit to the CDM (64-bits per VX2745) as a data packet. CDM passes hitmap data packets to the GDM and also records non-empty hitmap packets. CDM MIDAS frontend passes non-empty hitmap packets to MIDAS for online monitoring and visualization. GDM collects all the hitmaps and generates a global trigger. Alternatively, it generates a group trigger TRIG48 with a 48-bit bitmap indicating which VX2745 should trigger a waveform.

Pilot implementation computes the hitmap trigger as a grand-OR of all hit maps.

Hitmap chain:
* VX2745 analog inputs to digitizers to digital trigger logic block, 1 bit per channel (CAEN, Yair)
* 64-bits of hitmap (1 bit/channel) are collected and periodically transmitted to the CDM (Sam De Jong) (NB: confirm transmission period: see CDM page table "VX hit map" column "period", 0xD1 clocks is 1672 ns, should be reduced to 1500 ns, TBI)
* VX hitmap packets are transmitted to the CDM on the LVDS line 10 (CDM VX_RX_1) at 62.5 MHz, 8b10b encoded.
* CDM receives hitmap packets (12 input data streams, 12-to-1 packet mux)
* CDM sends all hitmap packets to the GDM at 125 MHz, 16b20b encoded.
* non-empty hitmap packets are written to the CDM-to-CPU data FIFO (and via MIDAS frontend to MIDAS SYSTEM event buffer)
* grand-OR of VX hitmap packets (12*64 bits) can generate a CDM-local VX trigger
* GDM receives hitmap packets, computes "GDM hitmap trigger"
* if enabled, GDM hitmap trigger causes GDM trigger, which is processed as described in the global trigger section

Delay, latency and jitter from analog pulse on the VX2745 analog input to VX trigger on LVDS line 12, TBM.

TRIG48 VX group trigger:
* GDM receives hitmap packets, computes the 48-bit bitmap of which VX2745 modules should trigger
* how to specify this computation is not clear. a generic 12*64 MLU is impossible, insufficient RAM on the FPGA
* 48-bit bitmap is sent to all CDMs as a TRIG48 data packet
* CDM receives TRIG48 data packets from GDM and retransmits them to the VX2745 (all CDMs and all VXes see the same packets at the same time).
* VX2745 receives the 48-bit trigger bitmap, compares it with it's own 48-bit bitmap if any common bits are set, triggers waveform readout of all channels (there will be no more than 48 VX2745 in the experiment)

Pilot implementation of TRIG48 triggers on the CDM grand-or and generates a TRIG48 packet that contains a 48-bit bitmap stored in an AXI register.

Example uses of TRIG48:
* to trigger individual VX2745 modules: set the VX 48-bit map to the module number (vx05-0x1, vx06-0x2, vx07-0x4, etc), generate TRIG48 packet with a bitmap of VX modules that should trigger: 0xFFFF'FFFF'FFFF will trigger all modules, 0x1 will trigger only vx05, 0x2 will trigger only vx06, etc. (there is no more than 48 modules).
* to trigger groups of VX2745 modules: set the bitmap of group 1 modules to 0x1, set the bitmap of group 2 modules to 0x2, generate TRIG48 packet with bitmap 0x0001 to trigger only group 1 modules, 0x0002 to trigger only group 2 modules, 0x3 to trigger both groups.

Notes:
* TBM - to be measured
* TBI - to be implemented

Current status (Jan 2026):
* full chain (VX analog input to hitmap generator to CDM to GDM) is implemented, testing in progress
* GDM grand-or trigger is implemented
* VX hitmap packet generator, testing in progress (Yair firmware)
* CDM hitmap processing, implemented, tested
* GDM hitmap processing, implemented, tested
* pilot TRIG48 trigger generator in GDM, CDM, tested
* TRIG48 trigger in VX, implemented, tested
* MIDAS frontend readout of hitmap data from FPGA FIFO: implemented, testing in progress (FIFO code: Ian, done; integration and C++ code: KO)
* online monitoring and visualization in MIDAS: TBI

== firmware map ==

Note:
* red lines: clocks
* green lines: AXI/Avalon packet streams
* blue lines: serial data

[[File:GDM-CDM-VX.drawio.svg]]

== description ==

same thing, in words:

<pre>
detector
digitizer, 125 MHz
digital filter
digital discriminator
hit map, 64 bits at 125 MHz (could be 250 MHz, filter and discriminator clock)
packetizer, 64 bits -> id, timestamp, 8x 8-bit words, eop
8/10 serializer, 12.5 MHz parallel in, 125 MHz serial out
lvds line to CDM vx_rx
BBB: also VX busy to lvds line to CDM vx_rx, do not want to depend on serial comm for vx busy, do not want to inject async data into the hit map packet stream
--- CDM
lvds line capture in IBUF register, vxN_rx clock, 125 MHz clock, from PLL with adjustable phase, scan phase to find sweet spot
vx rx 10/8 deserializer, vxN_rx clock, stobes out parallel data every 10 clocks at 12.5 MHz
vx rx phase transfer from vxN_rx clock to main CDM clock, have 10 clocks for transfer to happen
from here everything is on the CDM main clock
vx rx packet adapter, 12.5 MHz 8-bit packets to 125 MHz 16-bit packets, internal FIFO to avoid data overrun
sfp tx mux, all VX packet streams into one CDM sfp tx packet stream
BBB: capture VX RX busy to CDM main clock (IBUF register), grand-or becomes vx_rx_busy
sfp tx packetizer (data,eop,vx_rx_busy -> data,k)
sfp tx 16/20 serializer, 16 bit at 125 MHz in, 2500 MHz serial out (20 bits at 125 MHz)
sfp tx fiber link to GDM, 2 Gigabits/sec
--- GDM
qsfp rx deserializer, 16 bit at 125 MHz output on main GDM clock
qsfp rx depacketizer (data,k -> packet data,eop; qsfp_rx_busy)
qsfp rx demux, hit map packets routed to GDM trigger logic block
GDM trigger logic block looks at hit map, generates yes/no trigger decision, encodes it as a trigger packet
TSM generator encodes GPS time data as a TSM packet
qsfp tx mux - trigger packets, tsm packets, etc to qsfp tx packet stream
qsfp tx packetizer (data,eop;trg,tsm,bsy -> data,k)
NB: the same tx data is sent to all 12 qsfp tx ports, to make sure we do not accidentally desync the CDMs.
qsfp tx 16/20 serializer, 16-bit at 125 MHz to 20-bit at 250 MHz to 2500 MHz serial out
qsfp tx fiber link to CDM, 2 Gigabits/sec
--- CDM
sfp rx 20/16 deserializer, 16-bit at 125 MHz
sfp rx depacketizer, (data,k -> data,eop;trg_in,tsm_in,bsy_in)
sfp rx demux (in reality, noop, all packets go to same place, vx tx)
vx tx mux (packets from GDM, packets with simulated waveforms from CDM midas frontend via AXI FIFO)
vx tx packet adapter 16-bit at 125 MHz to 8-bit at 12.5 MHz strobed every 10 clocks. NB: most important, this 10-clock strobe runs in sync between all CDMs!
vx tx phase transfer from main CDM clock to vxN_tx clock, have 10 clocks for transfer to happen
from here we run on the vxN_tx clock
vx txN serializer, 8-bit at 12.5 MHz to 125 MHz serial
vx_txN OBUF register, vxN_rx clock, 125 MHz clock, from PLL with adjustable phase, scan phase to find sweet spot
lvds line to vx
BBB: bsy_in from GDM is converted from pulse to level, goes out lvds line to vx, sync to vxN_tx clock
TTT: trg_in from GDM is a pulse, does out lvds line to vx, sync to vxN_tx clock.
NB: tsm is always a packet, bsy is always a signal (no packet), trg can be a packet or signal.
--- VX
lvds data captured by 125 MHz ADC sampling clock (CAEN base firmware logic)
lvds data connected to Yair's block
10/8 deserializer
depacketizer (data,k -> data,eop) to avoid accidental desync, we do not send any no trg, no tsm, no bsy K-codes.
demux
trigger packets go their way (4x 64 bit words of data go to event header: timestamps, hitmap data)
tsm packets go their way (4x 64 bit words of data go to tsm event header: timestamps, GPS time data)
BBB: bsy from lvds line stops waveform acquisition
TTT: trg from lvds line cause waveform acquisition, same as trg packet, but has no timestamp and other data attached to it.
</pre>

== CDM-VX connections ==

Note:
* rounded boxes are VX registers set from MIDAS

[[File:CDM-VX.drawio.svg]]

= Firmware registers =

== Block 0 ==

<pre>
0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
</pre>

== Block 0 GDM ==

* 0x0000+4*i - GDM AXI clock registers

<pre>
register_data_out_block_0(1) <= x"204D4447"; --GDM<space>
register_data_out_block_0(2) <= x"00000000"; --blank
register_data_out_block_0(3) <= x"00000000"; --blank

--Firmware Version
register_data_out_block_0(4) <= x"abcd0123"; --Firmware version name register
register_data_out_block_0(5) <= x"00000001"; --Firmware version number register
register_data_out_block_0(6) <= x"abcd0123"; --Firmware version register
register_data_out_block_0(7) <= x"00000001"; --Firmware version register

--Git Commit/Tag/Compile Date
register_data_out_block_0(8) <= x"0123abcd"; --Firmware Git info (0-3)
register_data_out_block_0(9) <= x"00000001"; --Firmware Git info (1)
register_data_out_block_0(10) <= x"00000001"; --Firmware Git info (2)
register_data_out_block_0(11) <= x"00000001"; --Firmware Git info (3)

--Simple loopback register for testing
register_data_out_block_0(12) <= register_data_in_block_0(12);

--Hardware registers (HW_ID_xDM, etc)
--register_data_out_block_0(13) <= x"0000000" & "000" & hardware_id;

--unspecified
set_reg_block_0_gen : for i in 14 to nregisters_block_0-1 generate
register_data_out_block_0(i) <= register_data_in_block_0(i);
end generate set_reg_block_0_gen;

register_data_out_block_0(13) <= x"000000" & rtc_i2c_data_ie & rtc_i2c_data_out & rtc_i2c_clock_ie & rtc_i2c_clock_out & rtc_1pps_in & rtc_32khz_in & rtc_i2c_data_in & rtc_i2c_clock_in;

rtc_i2c_clock_ie <= not register_data_in_block_0(14)(0);
rtc_i2c_clock_out <= register_data_in_block_0(14)(1);
rtc_i2c_data_ie <= not register_data_in_block_0(14)(4);
rtc_i2c_data_out <= register_data_in_block_0(14)(5);
</pre>

== Block 0 CDM ==

* 0x0000+4*i - CDM AXI clock registers:

<pre>
USR_ACCESS_7series_inst: USR_ACCESSE2
port map(-- CFGCLK => CFGCLK,
DATA => register_data_out_block_0(0)); -- where timestamp is contained
-- DATAVALID => DATAVALID);

--static values
register_data_out_block_0(1) <= x"204D4443"; --CDM<space>
register_data_out_block_0(2) <= x"00000000"; --blank
register_data_out_block_0(3) <= x"00000000"; --blank

--Firmware Version
register_data_out_block_0(4) <= x"abcd0123"; --Firmware version name register
register_data_out_block_0(5) <= x"00000001"; --Firmware version number register
register_data_out_block_0(6) <= x"abcd0123"; --Firmware version name register
register_data_out_block_0(7) <= x"00000001"; --Firmware version number register

--Git Commit/Tag/Compile Date
register_data_out_block_0(8) <= x"0123abcd"; --Firmware Git info (0-3)
register_data_out_block_0(9) <= x"00000001"; --Firmware Git info (1)
register_data_out_block_0(10) <= x"00000001"; --Firmware Git info (2)
register_data_out_block_0(11) <= x"00000001"; --Firmware Git info (3)

--Simple loopback register for testing
register_data_out_block_0(12) <= register_data_in_block_0(12);

--Hardware registers (HW_ID_xDM, etc)
--register_data_out_block_0(13) <= x"0000000" & "000" & hardware_id;

--unspecified
set_reg_block_0_gen : for i in 14 to nregisters_block_0-1 generate
register_data_out_block_0(i) <= register_data_in_block_0(i);
end generate set_reg_block_0_gen;

rx_clk_sel <= register_data_in_block_0(12)(0); -- rx_clk select 0:clk_cc_out0, 1:rx_clk_mgt

rtc_i2c_clock_ie <= not register_data_in_block_0(14)(0);
rtc_i2c_clock_out <= register_data_in_block_0(14)(1);
rtc_i2c_data_ie <= not register_data_in_block_0(14)(4);
rtc_i2c_data_out <= register_data_in_block_0(14)(5);

register_data_out_block_0(13) <= x"000000" & rtc_i2c_data_ie & rtc_i2c_data_out & rtc_i2c_clock_ie & rtc_i2c_clock_out & rtc_1pps_in & rtc_32khz_in & rtc_i2c_data_in & rtc_i2c_clock_in;
</pre>

== Block 1 GDM ==

* 0x0800+4*i - GDM registers:

<pre>
clk_config_pro : process (read_write_clk)
begin
if rising_edge (read_write_clk) then
CLK_IN_SEL_LS <= clk_config_vec(1 downto 0);
CLK_EXT_SEL_LS <= clk_config_vec(2);
CLK_RSTn_LS <= not clk_config_vec(3);
clk_config_vec(4) <= CLK_LOSXTn_LS;
clk_config_vec(5) <= CLK_LOLn_LS;
clk_config_vec(6) <= CLK_INTn_LS;
clk_config_vec(7) <= '1';

clk_config_vec(3 downto 0) <= register_data_in_block_1(0)(3 downto 0);
register_data_out_block_1(0)( 7 downto 0) <= clk_config_vec(7 downto 0);
register_data_out_block_1(0)(31 downto 8) <= (others =>'0');
register_data_out_block_1(1) <= (others =>'0'); --free
end if;
end process clk_config_pro;

register_data_out_block_1(2) <= register_data_in_block_1(2);
register_data_out_block_1(3) <= register_data_in_block_1(3);
register_data_out_block_1(4)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(5)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(6)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(7)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(8) <= register_data_in_block_1(8);
register_data_out_block_1(9)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(10) <= (others => '0');
register_data_out_block_1(11) <= (others => '0');

frequency_measure_clks <= read_write_clk & clk_50MHz & mgt_data_clk & mgt_ref_clk_raw; -- & mgt_data_clk & mgt_ref_clk_raw;
register_data_out_block_1(12) <= register_data_out_block_1(14); --MGT rx and tx reference clocks are the same
register_data_out_block_1(13) <= register_data_out_block_1(15); --MGT rx and tx data clocks are the same
frequency_measure_inst : frequency_measure
generic map(register_clk_frequency => 100000000,
max_frequency => 300000000,
nfrequency_measurments => nfrequency_measure_clks)
port map(i_register_clk => read_write_clk,
i_clks => frequency_measure_clks,
i_clk_locks => (others =>'1'),
o_frequencies => register_data_out_block_1(14+nfrequency_measure_clks-1 downto 14)); --see register_data_out_block_1(12,13) above

zero_spare_link_registers : for i in 14+nfrequency_measure_clks to nregisters_block_1-1 generate
register_data_out_block_1(i) <= (others => '0');
end generate zero_spare_link_registers;

gdm_link_interface_inst : gdm_link_interface
generic map(enable_rxslide => '1',
nquads => nquads,
nlinks => nlinks)
port map(i_mgt_ref_clk_p => mgt_b_ref_clk_p,
i_mgt_ref_clk_n => mgt_b_ref_clk_n,
i_mgt_rx_n => mgt_b_rx_n,
i_mgt_rx_p => mgt_b_rx_p,
o_mgt_tx_n => mgt_b_tx_n,
o_mgt_tx_p => mgt_b_tx_p,
--raw, pre-pll, mgt clock for frequency measurement
o_mgt_ref_clk_raw => mgt_ref_clk_raw,
--free running clock and reset
i_mgt_free_running_clk => clk_50MHz,
i_mgt_rst => register_data_in_block_1(2)(0),
i_rx_slide_trigger => register_data_in_block_1(3)(nlinks-1 downto 0),
--data interface for the links
o_data_clk => mgt_data_clk,
o_mgt_data_link_array => mgt_link_data_to_processing,
i_mgt_data_link_array => mgt_link_data_from_processing,
--status ports
o_link_power_good => register_data_out_block_1(4)(nlinks-1 downto 0),
o_tx_link_up => register_data_out_block_1(5)(nlinks-1 downto 0),
o_rx_link_up => register_data_out_block_1(6)(nlinks-1 downto 0),
o_rx_error => register_data_out_block_1(7)(nlinks-1 downto 0),
i_link_down_latched_rst => register_data_in_block_1(8)(0),
o_link_down_latched => register_data_out_block_1(9)(nlinks-1 downto 0),
--debug
i_debug_clk => read_write_clk,
o_debug => mgt_debug_data);
</pre>

<pre>
#### Register Block 1 (CDM)
Clock Chip and MGT Link, Control and Status Registers
| Register | Axi Address | Description | note
|:-|:-|:-|:-|
| 0 | 0x80011000 | Clock Chip Status and Control | CLK_IN_SEL_LS(1-0)(read-only), CLK_EXT_SEL_LS(2), CLK_RSTn_LS(3), CLK_LOSXTn_LS(4), CLK_LOLn_LS(5), CLK_I\
NTn_LS(6)
| 1 | 0x80011004 | free for use | |
| 2 | 0x80011008 | MGT Link Reset | mgt_rst(0), Reset for RX Error Latched(1)|
| 3 | 0x8001100c | MGT rx slide control | not needed or used |
| 4 | 0x80011010 | link_up_and_running(10) tx_link_up_and_running(9), tx_sending_data(8), tx_link_up(7), rx_link_up_and_running(6), rx_error(5), rx_receivi\
ng_data(4), rx_link_up(3), link_power_good(2), sfp_rx_los(1), sfp_mod_absent(0) | Should be 0x7DC |
| 5 | 0x80011014 | reserved debug register | |
| 6-11 | ... | free for use | |
| 12 | 0x80011030 | mgt_rx_ref_clk_raw | ~0x7735940 (125 MHz) |
| 13 | 0x80011034 | rx_clk | ~0x7735940 (125 MHz) |
| 14 | 0x80011038 | mgt_tx_ref_clk_raw | ~0x7735940 (125 MHz) |
| 15 | 0x8001103c | tx_clk | ~0x7735940 (125 MHz) |
| 16 | 0x80011040 | clk_50MHz | 0x2faf080 (50 MHz) |
| 17 | 0x80011044 | Register_Block1_clk | 0x5f5e100 (100 MHz) |
| 18-20 | ... | free for use | |
| 21 | 0x80011054 | free for use | |
</pre>

=== Register 0x1000 ===

<pre>
printf("0x1000 SFP c.c. status: 0x%08x\n", reg1000);
printf(" CLK_IN_SEL_LS 0x%x\n", (reg1000>>0) & 3);
printf(" CLK_EXT_SEL_LS %d\n", (reg1000>>2) & 1);
printf(" CLK_CLK_RSTn_LS %d\n", (reg1000>>3) & 1);
printf(" CLK_LOSXTn_LS %d\n", (reg1000>>4) & 1);
printf(" CLK_LOLn_LS %d\n", (reg1000>>5) & 1);
printf(" CLK_INTn_LS %d\n", (reg1000>>6) & 1);
printf("0x1008 SFP link reset: 0x%08x\n", dev->dsdm_read32(0x1008));
</pre>

=== Register 0x1010 ===

<pre>
printf("0x1010 SFP link status: 0x%08x\n", reg1010);
printf(" sfp_mod_absent_N %d\n", (reg1010>>0) & 1);
printf(" sfp_rx_los_N %d\n", (reg1010>>1) & 1);
printf(" link_power_good %d\n", (reg1010>>2) & 1);
printf(" rx_link_up %d\n", (reg1010>>3) & 1);
printf(" rx_receiving_data %d\n", (reg1010>>4) & 1);
printf(" rx_error %d\n", (reg1010>>5) & 1);
printf(" rx_lnk_up_and_running %d\n", (reg1010>>6) & 1);
printf(" tx_link_up %d\n", (reg1010>>7) & 1);
printf(" tx_sending_data %d\n", (reg1010>>8) & 1);
printf(" tx_link_up_and_running %d\n", (reg1010>>9) & 1);
printf(" link_up_and_running %d\n", (reg1010>>10) & 1);

</pre>

=== Register 0x1014 ===

<pre>
printf("0x1014 SFP link data: 0x%08x\n", reg1014);
printf(" rx_data 0x%04x\n", (reg1014>>0) & 0xFFFF);
printf(" k28p1_k28p5 %d\n", (reg1014>>16) & 1);
printf(" rx_ctrl0 0x%01x\n", (reg1014>>17) & 3);
printf(" rx_ctrl1 0x%01x\n", (reg1014>>19) & 3);
printf(" rx_ctrl3 0x%01x\n", (reg1014>>21) & 1);
printf(" tx_state 0x%01x\n", (reg1014>>22) & 3);
printf(" rx_state 0x%01x\n", (reg1014>>24) & 0xF);
printf(" rx_receiving_data %d\n", (reg1014>>28) & 1);
printf(" rx_link_up %d\n", (reg1014>>29) & 1);
printf(" rx_error %d\n", (reg1014>>30) & 1);
printf(" rx_link_rst %d\n", (reg1014>>31) & 1);
</pre>

* see cdm_link_establishment.vhd
* tx_state:
<pre>
0 = tx_idle
1 = waiting_for_rx_link
2 = sending_k28p1_k28p5
3 = sending_data
</pre>

* rx_state:
<pre>
0 = rx_idle
1 = waiting_for_ks
2 = rx_retry
3 = rx_up_switch_tx_clk_source
4 = waiting_for_data
5 = receiving_data
6 = rx_receive_error
</pre>

== Block 2 GDM ==

* 0x1000+4*i - GDM registers:

<pre>
register_data_out_block_2(0) <= register_data_in_block_2(0);
GDM_link_data_processing_inst : GDM_link_data_processing
generic map(mgt_data_frequency => 125000000)
port map(i_register_clk => read_write_clk,
i_rst => register_data_in_block_2(0)(0),
i_data_mode => register_data_in_block_2(0)( 9 downto 8),
i_fixed_pattern_data => register_data_in_block_2(0)(31 downto 16),
o_error_time => register_data_out_block_2(1),
o_error_count => register_data_out_block_2(13 downto 2),
--link data interface
i_data_clk => mgt_data_clk,
i_mgt_data_link_array => mgt_link_data_to_processing,
o_mgt_data_link_array => mgt_link_data_from_processing,
--data interface
o_data_clk => data_clk,
o_tx_data_ready => tx_data_ready,
i_tx_data_is_k => tx_data_is_k,
i_tx_data => tx_data,
o_rx_data_error => rx_data_error,
o_rx_data_is_k => rx_data_is_k,
o_rx_data => rx_data);
</pre>

== Block 2 CDM ==

* 0x1000+4*i - CDM registers:
* reg 0
<pre>
read bits
7..0 - clk_config_vec
1,0 - CLK_IN_SEL_LS
2 - CLK_EXT_SEL_LS
3 - disconnected CLK_RSTn_LS
4 - CLK_LOSXTn_LS
5 - CLK_LOLn_LS
6 - CLK_INTn_LS
7 - tx_clock_status
31..8 - 0
</pre>
<pre>
write bits
3..2 - clk_config_vec[3..2]
</pre>
* reg 1 - 0
* reg 2 - RW
<pre>
bit
0 - cdm_link_interface i_mgt_rst
1 - cdm_link_interface i_link_down_latched_rst
</pre>
* reg 3 - RW
<pre>
bit
0 - cdm_link_interface i_rx_slide_trigger
</pre>
* reg 4 - RO
<pre>
bit
0 - sfp_mod_present
1 - sfp_rx_los
2 - link_power_good
3 - rx_link_up
4 - rx_receiving_data
5 - rx_error
6 - rx_link_up_and_running
7 - tx_link_up
8 - tx_sending_data
9 - tx_link_up_and_running
10 - link_up_and_running
</pre>
* reg 5 - RO - CDM SFP debug_data
* reg 6 - 0
* reg 7 - 0
* reg 8 - 0
* reg 9 - 0
* reg 10 - 0
* reg 11 - 0
* reg 12 - frequency AXI clock
* reg 13 - freq clk_50MHz
* reg 14 - freq clk_100MHz
* reg 15 - freq clk_200MHz
* reg 16 - freq clk_xo_125 - DS-DM 125 MHz oscillator
* reg 17 - freq clk_cc_out0 - C.C. 125 MHz output
* reg 18 - CDM SFP mgt_rx_ref_clk_raw
* reg 19 - CDM SFP mgt_tx_ref_clk_raw
* reg 20 - rx_clk_mgt, 125 MHz CDM SFP RX recovered clock
* reg 21 - rx_clk, 125 MHz, switched between clk_xo_125 and rx_clk_mgt
* reg 22 - tx_clk, 125 MHz
* reg 23 - spare

== Block 3 GDM ==

0x3000+4*i : DS20k registers

== Block 4 CDM ==

* 0x2800+4*i - CDM SFP data link registers:
* 0 - R/W
** bit 0 - 0x1 - reset
** bits 7..1 - not used
** bits 9..8 - data_mode: 00: data from ds20k block, 01: tx_data_cnt, 10: tx_data_prbs, 11: tx_fixed_pattern_data
** bits 15..10 - not used
** bits 31..16 - fixed pattern data
* 1 - RO - o_error_time
* 2 - RO - o_error_count
* 3, etc - not used

== Block 6 CDM ==

* 0x3000+4*i : DS20k registers

== Block 7 CDM ==

VX_TX and VX_RX delays, i=0..11
* 0x3800+4*i - VX_TX delays
* 0x3830+4*i - VX_RX delays

VX_TX delay data (2 bits):
* 4 values: 0,1,2,3 corresponding to the 4 phases of the 125 MHz clock rx_clk and rx_clk shifted by 90 deg

VX_RX delay data (32 bits):
* 0x8000'0000: reset IDELAYE3, follow by write of 0
* 0x0001'xxxx: load the IDELAYE3 9-bit tap counter, follow by write of 0x0000'xxxx
* 16 low bits have:
* bits 0..8 are the 9 bits of IDELAY3 tap counter (~1.1ns for all 511 delays)
* bit 9 is the 500 MHz DDR phase (1 ns)
* bits 10..15 are 16 bits of 500 MHz delay (2 ns step)
* numerically increasing 16-bit delay values are carefully arranged to produce linearly increasing delay (there is some overlap between IDELAYE3 taps above ~tap 500 (or so, IDELAYE3 is uncalibrated) and flip of 500 MHz DDR phase

== DS20k registers ==

=== DS20k block register map ===

* busybox devmem 0x80013000 32

Note: R=readable, W=writable, L=latched by CMD_LATCH, B=reset on begin of run, Z=reset by CMD_LATCH

<pre>
reg | version | xDM | xx | description
0 | 0x20230731 | ALL | RO | ds20k version
0 | 0x20240118 | ALL | RW | ds20k version and command
1 | 0x20230731 | ALL | RW | scratch read/write register
2 | 0x20230731 | ALL | RW | configure inputs and outputs
3 | 0x20230731 | ALL | RW | FP_LED mux
4 | 0x20230731 | ALL | RW | EXT_OUT mux
5 | 0x20230731 | ALL | RO | VX_RX state
6 | 0x20230731 | ALL | RO | VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state
7 | 0x20230731 | ALL | RW | LED_OUT, EXT_OUT, VX_TX outputs
8 | 0x20230731 | ALL | RW | VX_TX mux and config
9 | 0x20230731 | ALL | RW | trg and tsm source
10 | 0x20231013 | ALL | RO | status register
11 | 0x20230731 | ALL | ROLB | trg counter
12 | 0x20230731 | ALL | ROLB | tsm counter
13 | 0x20240814 | ALL | RO | GPS 1pps period, 125 MHz
14 | 0x20240814 | ALL | RO | Ru clock 1pps period, 125 MHz
15 | 0x20230811 | CDM | RO | SFP RX status
16 | 0x20230811 | CDM | RW | SFP TX control
17 | 0x20230811 | GDM | RO | QSFP RX data 0, 1
18 | |. |. | 2, 3
19 | |. |. | 4, 5
20 | |. |. | 6, 7
21 | | |. | 8, 9
22 | |. |. | 10, 11
23 | 0x20230811 | GDM | RW | QSFP TX control
24 | 0x20231013 | ALL | RW | trigger pulser period
25 | 0x20231013 | ALL | RW | trigger pulser burst control
26 | 0x20231013 | ALL | RW | tsm pulser period
27 | 0x20231204 | ALL | RW | data fifo CPU to FPGA
28 | 0x20231204 | ALL | RW | packet loopback control
28 | 0x20240118 | ALL | RW | data fifo FPGA to CPU
29 | 0x20231208 | GDM | RW | bitmap of active qsfp ports
29 | 0x20240118 | ALL | RW | packet loopback control
29 | 0x20240510 | ALL | RW | packet routing
30 | 0x20231208 | GDM | RO | qsfp link status ports 0..7
31 | 0x20231208 | GDM | RO | qsfp link status ports 8..11
32 | 0x20240118 | GDM | RW | enabled QSFP ports
33 | 0x20240118 | CDM | RW | enabled VX ports
34,35 | 0x20240118 | ALL | ROB | time stamp 64 bits
36,37 | 0x20240118 | ALL | RO | old time stamp
38,39,40 | 0x20240118 | CDM | ROLZ | vx_bsy_counter_latched | VX busy counters
41,42,43 | 0x20240118 | GDM | ROLZ | QSFP busy counters
44 | 0x20240118 | CDM | ROLZ | cdm_bsy_up_counter and cdm_bsy_pulse_counter
45 | 0x20240118 | CDM | ROLZ | cdm_veto_pulse_counter and cdm_veto_up_counter
46 | 0x20240118 | GDM | ROLZ | gdm_bsy_pulse_counter and gdm_bsy_up_counter
47 | 0x20240118 | GDM | ROLZ | gdm_bsy_refresh_counter
48 | 0x20240118 | GDM | ROLZ | gdm_veto_up_counter and gdm_veto_pulse_counter
49,50,51 | 0x20240424 | CDM | RO | VX RX serial data monitor, 8 bits per VX port
52,53 | 0x20240424 | CDM | RO | VX serial link status, 4 bits per VX port
54 | 0x20240424 | CDM | RO | VX TX serial data monitor
55 | 0x20240430 | ALL | RO | QSFP, SFP and VX link loss counters
56 | 0x20240510 | ALL | RO | sfp_rx_packet_counter
57 | same | ALL | RO | sfp_tx_packet_counter
58 | same | ALL | RO | qsfp_rx_packet_counter[0]
59 | same | ALL | RO | qsfp_tx_packet_counter
60 | same | ALL | RO | vx_rx_packet_counter[0]
61 | same | ALL | RO | vx_tx_packet_counter
62 | 0x20240719 | CDM | RO | cdm_hitmap_period, ports 0, 1
63 | same | CDM | RO | ports 2, 3
64 | same | CDM | RO | ports 4, 5
65 | same | CDM | RO | ports 6, 7
66 | same | CDM | RO | ports 8, 9
67 | same | CDM | RO | ports 10, 11
62 | 0x20260401 | CDM | ROL | vx_rx_hitmap_period latched by cmd_latch, VX ports 0, 1
63 | same | CDM | ROL | ports 2, 3
64 | same | CDM | ROL | ports 4, 5
65 | same | CDM | ROL | ports 6, 7
66 | same | CDM | ROL | ports 8, 9
67 | same | CDM | ROL | ports 10, 11
68 | 0x20240814 | ALL | RW | GPS control and status
69 | 0x20241104 | CDM | ROLB | vx_tx_trg_packet_counter | counter of TRG packets CDM->VX
70 | 0x20241104 | CDM | ROLB | vx_tx_tsm_packet_counter | counter of TSM packets CDM->VX
71 | 0x20241104 | ALL | RO | packet error bits
72,73 | 0x20241104 | CDM | ROL | vx_tx_trg_data_latched[63:0] trigger packet data
74,75 | 0x20241104 | CDM | ROL | vx_tx_tsm_data_latched[63:0] tsm packet data (truncated to 64 bits)
76 | 0x20241104 | CDM | ROLB | cdm_hitmap_trigger_counter_latched, counter of hitmap triggers generated in the CDM
77,78,79 | 0x20241104 | CDM | ROL | vx_rx_hitmap_data_latched | VX_RX 80 bits of HITMAP packet from VX
80 | 0x20241209 | CDM | RO | vx_lvds_rx_link_status | VX LVDS link status
81 | 0x20241209 | CDM | RO | vx_lvds_rx_trg_data | VX LVDS TRG packet first 32 bits
81 | 0x20251010 | CDM | - | not used | not used
82 | 0x20241209 | CDM | RO | vx_lvds_rx_tsm_data | VX LVDS TSM packet first 32 bits
82 | 0x20251010 | CDM | - | not used | not used
83 | 0x20241209 | CDM | ROL | vx_lvds_rx_decoded_packet_counter_latched | VX LVDS decoded packet counter
84 | 0x20241209 | CDM | ROL | vx_lvds_rx_packet_error_counter_latched | VX LVDS packet error counter
85 | 0x20250417 | CDM | ROLB | vx_tx_trg_counter | VX LVDS TRG counter (VX LVDS pair 12)
86 | 0x20250417 | CDM | ROLB | vx_tx_tsm_counter | VX LVDS TSM counter (VX LVDS pair 13)
87 | 0x20250417 | CDM | ROLB | vx_tx_veto_counter | VX LVDS Veto counter (VX LVDS pair 13)
88 | 0x20250422 | GDM | ROLB | trg_in_lost_counter | counter of TRG killed by gdm_busy
89 | 0x20250422 | ALL | RO | fifo_from_fpga_lost_packet_counter | counter of packets lost due to FIFO full
90 | 0x20250428 | CDM | ROL | vx_bsy_status_latched | status of VX busy, CDM busy, CDM veto
91 | 0x20250428 | CDM | ROL | vx_hitmap_status_latched | status of VX hitmap
92 | 0x20250428 | GDM | ROL | cdm_bdy_status_latched | status of CDM busy, GDM busy, GDM veto
93 | 0x20250508 | GDM | ROL | gdm_hitmap_status_latched | status of GDM hitmap
94 | 0x20250508 | GDM | ROLB | gdm_hitmap_trigger_counter_latched | counter of hitmap triggers generated in the GDM
95 | 0x20250512 | GDM | ROLB | gdm_trig48_counter_latched | counter of TRIG48 trigger packets
96 | 0x20250512 | GDM | ROLB | gdm_trig48_lost_counter_latched | counter of TRIG48 triggers lost to busy
97 | 0x20250512 | CDM | ROLB | vx_tx_trig48_packet_counter_latched | counter of TRIG48 packets from CDM to VX
98, 99 | 0x20250512 | GDM | RW | trig48_ctrl48, trig48_ctrl16 | GDM TRIG48 test registers
100,101 | 0x20251008 | CDM | ROL | vx_lvds_rx_trig48_{data,bitmap48}_latched | VX LVDS TRIG48 packet first 32 bits, bitmap48 first 32 bits
102 | 0x20251008 | ALL | RO | cmd_counter | counter of reg0 commands
103,104,105 | 0x20260104 | CDM | ROL | vx_rx_hitmap_vx_id_latched | VX ID from VX hitmap packets, VX ports 0..11, in order, latched by cmd_latch
</pre>

=== Register 0 0x80013000 ds20k version ===

on read: ds20k version 0xYYYYMMDD

on write:
* 0 - noop - as of version 0x20240118, write a zero after writing a command
* 1 - cmd_reset - reset logic to good state
* 2 - cmd_arm_ts - arm timestamp reset
* 3 - cmd_trg - issue a trigger
* 4 - cmd_tsm - issue a tsm
* 5 - cmd_vx_rx_reset - reset the VX receive path
* 6 - cmd_vx_tx_reset - reset the VX transmit path
* 7 - cmd_gdm_trig48 - generate a TRIG48 trigger and data packet (GDM)
* 8 - cmd_trg_pulser_reset - reset the trigger pulser
* 9 - cmd_tsm_pulser_reset - reset the tsm pulser
* 10 - cmd_bor_start - start begin-of-run trigger sequence
* 11 - cmd_bor_clear - after run has started, clear begin-of-run status bits
* 12 - cmd_latch - latch counters & etc into AXI registers for coherent readout
* 13 - cmd_cdm_hitmap_trigger - generate a hitmap trigger and data packet (CDM)

=== Register 1 0x80013004 scratch ===

scratch read-write register

=== Register 2 0x80013008 input and output config ===

<pre>
bit | version | fpga name | description
0 | ALL | lemo_enable | enable LEMO input 1
1 | | | 2
2 | | | 3
3 | | | 4
4 | ALL | lemo_invert | invert LEMO input 1
5 | | | 2
6 | | | 3
7 | | | 4
9 | ALL | ext_out_disable | disable LEMO output 1
10 | | | 2
11 | ALL | ext_out_invert | invert LEMO output 1
12 | | | 2
</pre>

=== Register 3 0x8001300C FP_LED control ===

<pre>
wire [15:0] led_out_mux_sel = register_data_in[3][15:0];
wire [3:0] led_out_invert = register_data_in[3][19:16];
</pre>

<pre>
led_out_mux_sel is 4 groups (one per LED) of 4 bits (choice 0..15):
mux | version | fpga name | description
0 | ALL | | power on default
1 | ALL | led_out_reg | register 7 bits
2 | 0x20231013 | pll_locked | clock chip PLL is locked
3 | 0x20231013 | sfp_link_status | SFP link is good
3 | 0x20240118 | sfp_link_status or qsfp_tx_link_rx_status | SFP/QSFP link is good
4 | ALL | lemo_in_sync[1] | LEMO input 1
5 | ALL | lemo_in_sync[2] | LEMO input 2
6 | ALL | lemo_in_sync[3] | LEMO input 3
7 | ALL | lemo_in_sync[4] | LEMO input 4
8 | ALL | ext_out[1] | LEMO output 1
9 | ALL | ext_out[2] | LEMO output 2
A | 0x20231013 | trg_in | trigger
B | 0x20231013 | tsm_in | time slice marker
C | 0x20240118 | gdm_bsy | GDM busy: OR of all CDM busy
D | 0x20240118 | cdm_bsy | CDM busy: OR of all VX busy
E | 0x20240118 | cdm_veto | GDM busy -> GDM veto -> CDM veto -> VX trigger veto
F | ALL | | fixed logic level 1
</pre>

=== Register 4 0x80013010 LEMO OUT control ===

<pre>
wire [7:0] ext_out_mux_sel = register_data_in[4][7:0];
</pre>

<pre>
ext_out_mux_sel is 2 groups (one per LEMO) of 4 bits (choice 0..15):

mux | version | fpga name | description
0 | ALL | | power on default
1 | ALL | ext_out_reg | register 7
2 | ALL | trg_pulser | pulser trigger
3 | 0x20240724 | vx1_tx_out[2] | vx1 serial data out
4 | 0x20240724 | vx1_rx[1] | vx1 serial data in
5 | 0x20240724 | vx_rx_iob[0] | vx1 serial data in captured by IOB register
6 | ALL | lemo_in_async[1]| test synchronizer
7 | ALL | lemo_in_sync[1] | test synchronizer
8 | 0x20240724 | trg_in_pulse | trigger signal
9 | 0x20240724 | tsm_in_pulse | time slice marker signal
A | 0x20240118 | cdm_bsy | CDM busy from VX
B | 0x20240118 | gdm_bsy | GDM busy from CDM
C | 0x20240118 | cdm_veto | veto from GDM to CDM to VX
D | 0x20240118 | vx1_rx[1] | serial data VX to CDM
E | not used (sink)
F | ALL | 1 | fixed logic level 1
</pre>

=== Register 5 0x80013014 VX_RX status ===

<pre>
assign register_data_out[5] = {
vx8_rx[3], vx8_rx[2], vx8_rx[1], vx8_rx[0],
vx7_rx[3], vx7_rx[2], vx7_rx[1], vx7_rx[0],
vx6_rx[3], vx6_rx[2], vx6_rx[1], vx6_rx[0],
vx5_rx[3], vx5_rx[2], vx5_rx[1], vx5_rx[0],
vx4_rx[3], vx4_rx[2], vx4_rx[1], vx4_rx[0],
vx3_rx[3], vx3_rx[2], vx3_rx[1], vx3_rx[0],
vx2_rx[3], vx2_rx[2], vx2_rx[1], vx2_rx[0],
vx1_rx[3], vx1_rx[2], vx1_rx[1], vx1_rx[0]
};
</pre>

=== Register 6 0x80013018 VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state ===

<pre>
assign register_data_out[6] = {
vx2_tx_out[2], vx2_tx_out[1], vx2_tx_out[0], vx1_tx_out[2],
vx1_tx_out[1], vx1_tx_out[0], ext_out[2], ext_out[1],
fp_led_out[3], fp_led_out[2], fp_led_out[1], fp_led_out[0],
ext_in_lv[4], ext_in_lv[3], ext_in_lv[2], ext_in_lv[1],
vx12_rx[3], vx12_rx[2], vx12_rx[1], vx12_rx[0],
vx11_rx[3], vx11_rx[2], vx11_rx[1], vx11_rx[0],
vx10_rx[3], vx10_rx[2], vx10_rx[1], vx10_rx[0],
vx9_rx[3], vx9_rx[2], vx9_rx[1], vx9_rx[0]
};
</pre>

<pre>
bit | version | fpga name | description
0 | ALL | vx9_rx | VX_RX
1 | | |
2 | | |
3 | | |
4 | ALL | vx10_rx | VX_RX
5 | | |
6 | | |
7 | | |
8 | ALL | vx11_rx | VX_RX
9 | | |
10 | | |
11 | | |
12 | ALL | vx12_rx | VX_RX
13 | | |
14 | | |
15 | | |
16 | ALL | ext_in_lv | LEMO inputs
17 | | |
18 | | |
19 | | |
20 | ALL | FP_LED | FP_LEDs
21 | | |
22 | | |
23 | | |
24 | ALL | ext_out[1] | LEMO outputs
25 | | ext_out[2] |
26 | ALL | vx1_tx | VX1_TX
27 | | |
28 | | |
29 | ALL | vx2_tx | VX2_TX
30 | | |
31 | | |
</pre>

=== Register 7 0x8001301C LED_OUT, EXT_OUT, VX_TX outputs ===

<pre>
wire [3:0] led_out_reg = register_data_in[7][3:0];
wire [2:1] ext_out_reg = register_data_in[7][5:4];
// register_data_in[7][6];
// register_data_in[7][7];
wire [7:0] vx_tx_out_reg = register_data_in[7][15:8];
</pre>

<pre>
bit | version | fpga name | description
0 | ALL | led_out_reg | FP_LED 1
1 | | | 2
2 | | | 3
3 | | | 4
4 | ALL | ext_out_reg | LEMO OUT 1
5 | | | 2
6 | - | |
7 | - | |
8 | ALL | vx_tx_out_reg | VX1_TX 0
9 | | | 1
10 | | | 2
11 | | | -
12 | ALL | | VX2_TX 0
13 | | | 1
14 | | | 2
15 | | | -
16 | - | |
17 | | |
18 | | |
19 | | |
20 | | |
21 | | |
22 | | |
23 | | |
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 8 0x80013020 VX_TX config ===

<pre>
wire [3:0] vx_tx_mux_sel = register_data_in[8][3:0];
wire vx_tx_from_sfp = register_data_in[8][30];
wire vx_tx_loopback = register_data_in[8][31];
</pre>

<pre>
vx_tx_mux_sel is 4 bits (choice 0..15):
0 - power on default, control by vx_tx_out_reg
1 - GDM
2 - CDM
3 - pulser loopback test
4 - pulser loopback test
5 - 62.5 MHz output
6 - 125 MHz output
7 - trg, tsm, serial
8 - trg, tsm, lvds serial rx to serial tx loopback
9 - GPS box control (ds20k rev 0x20240814)
10
11
12
13
14
15 - production config: trg, veto, serial
</pre>

=== Register 9 trg and tsm source ===

from version 0x20240724

<pre>
wire [15:0] trg_src_mask = register_data_in[9][15:0];
wire [31:16] tsm_src_mask = register_data_in[9][31:16];
</pre>

<pre>
wire [15:0] trg_src_bits =
{
1'b0, // 15
1'b0, // 14
vx_tx_tsm_done, // 13
vx_tx_trg_done, // 12

1'b0, // gdm_hitmap_trigger, // 11
cdm_hitmap_trigger, // 10
sfp_rx_tsm, // 9
sfp_rx_trg, // 8

sfp_rx_data[1], // 7
sfp_rx_data[0], // 6
tsm_pulser, // 5
trg_pulser, // 4

lemo_in_sync[4], // 3
lemo_in_sync[3], // 2
lemo_in_sync[2], // 1
lemo_in_sync[1] // 0
};

wire [15:0] trg_bits = trg_src_bits & trg_src_mask;
wire [15:0] tsm_bits = trg_src_bits & tsm_src_mask;
</pre>

before that:

<pre>
wire [7:0] trg_src_mask = register_data_in[9][7:0];
wire [7:0] tsm_src_mask = register_data_in[9][15:8];
wire trg_pulser_enable = register_data_in[9][16];
wire tsm_pulser_enable = register_data_in[9][17];
wire trg_software = register_data_in[9][18];
wire tsm_software = register_data_in[9][19];
// bits 20:31 not used
</pre>

<pre>
wire [7:0] xxx_src_bits =
{
sfp_rx_data[1],
sfp_rx_data[0],
tsm_pulser & tsm_pulser_enable,
trg_pulser & trg_pulser_enable,
lemo_in_sync[4],
lemo_in_sync[3],
lemo_in_sync[2],
lemo_in_sync[1]
};
</pre>

<pre>
trg_src_mask and tsm_src_mask bits:
0 - LEMO IN 1
1 - LEMO IN 2
2 - LEMO IN 3
3 - LEMO IN 4
4 - trg_pulser
5 - tsm_pulser
6 - sfp_rx_data[0] // to become sfp_trg_in, selected from sfp_rx_data[0], cdm_rx trg_out and gdm trigger packet
7 - sfp_rx_data[1] // to become sfp_tsm_in, selected from sfp_rx_data[1], cdm_rx tsm_out and gdm tsm packet
</pre>

=== Register 10 0x80013028 status register ===

<pre>
bit | version | fpga name | description
0 | 0x20231013 | pll_locked | clock chip PLL is locked
1 | 0x20240118 | ts_reset_armed | timestamp reset is armed
2 | 0x20240118 | qsfp_tx_link_rx_status | QSFP link status is good for all enabled ports
3 | 0x20240424 | vx_tx_link_rx_status | VX link status is good for all enabled ports
4 | 0x20240118 | cdm_bsy | VX busy grand-or
5 | 0x20240118 | gdm_bsy | QSFP busy grand-or
6 | 0x20240118 | gdm_veto | gdm_veto = gdm_busy
7 | 0x20240118 | cdm_veto | CDM veto from GDM to VX
8 | 0x20240725 | bor_started | begin-of-run sequence started
9 | 0x20240725 | bor_finished | begin-of-run sequence fininished, see commands 10 and 11
10 | | |
11 | | |
12 | | |
13 | | |
14 | | |
15 | | |
16 | | |
17 | | |
18 | | |
19 | | |
20 | | |
21 | | |
22 | | |
23 | | |
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 11 0x8001302C trg_counter ===

trigger counter

=== Register 12 0x80013030 tsm_counter ===

time slice marker counter

=== Register 13 0x80013034 GPS 1pps period ===

GPS 1pps period in 8 ns clocks

=== Register 14 0x80013038 Rb clock 1pps period ===

PRS-10 Rb clock 1pps output period in 8 ns clocks

=== Register 15 0x8001303C SFP RX status ===

<pre>
bit | ds20k version | fpga signal name | description
0 | ALL | sfp_rx_data[15:0] | cdm sfp received data
16 | same | sfp_rx_data_is_k[0] |
17 | same | sfp_rx_data_is_k[1] |
18 | | 0 |
19 | 0x20231204 | sfp_rx_sel_lpb | sfp tx->rx loopback
20 | 0x20231013 | sfp_link_status | sfp link connected, exchanging data
21 | same | sfp_link_rx_status | sfp link receiving correct idle pattern from GDM TX
22 | same | sfp_link_error | sfp link receiver error (badk or overflow)
23 | same | sfp_rx_data_error | sfp transceiver state machine is in error state
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 16 SFP TX control ===

<pre>
wire [15:0] sfp_tx_data_reg = register_data_in[16][15:0]; // SFP TX data K-code
wire [1:0] sfp_tx_ctrl_reg = register_data_in[16][17:16]; // SFP TX data K-code
// 18
// 19
// 23:20
// 27:24
wire sfp_rx_sel_lpb = register_data_in[16][28]; // TX->RX serial loopback
wire sfp_tx_sel_lpb = register_data_in[16][29]; // RX->TX serial loopback
wire sfp_tx_sel_trg = register_data_in[16][30]; // 16 individual bits
wire sfp_tx_sel_reg = register_data_in[16][31]; // TX data from register 16 bits 17:0
</pre>

=== Register 17-22 QSFP RX data ===

QSFP RX data links 0..11

=== Register 23 QSFP TX control ===

<pre>
wire [15:0] qsfp_tx_data_reg = register_data_in[23][15:0];
wire [1:0] qsfp_tx_ctrl_reg = register_data_in[23][17:16];

wire qsfp_tx_enable_trg = register_data_in[23][24]; // enable QSFP TX trg_in_pulse k-code
wire qsfp_tx_enable_tsm = register_data_in[23][25]; // enable QSFP TX tsm_in_pulse k-code

//wire qsfp_rx_sel_lpb = register_data_in[23][28]; // TX->RX loopback
wire qsfp_tx_sel_lpb = register_data_in[23][29]; // RX->TX loopback
wire qsfp_tx_sel_trg = register_data_in[23][30]; // 16 individual bits
wire qsfp_tx_sel_reg = register_data_in[23][31]; // data from register
</pre>

=== Register 24 0x80013060 trigger pulser period ===

trigger pulser period in units of 8 ns (125 MHz clock)

=== Register 25 0x80013064 trigger burst pulser ===

<pre>
wire [7:0] conf_burst_count = conf_pulser_burst_ctrl[31:24];
wire [23:0] conf_burst_period = conf_pulser_burst_ctrl[23:0];
</pre>

=== Register 26 0x80013068 tsm pulser period ===

time slice marker period in units of 8 ns (125 MHz clock)

=== Register 27 0x8001306C data write fifo ===

<pre>
wire fifo_reset = register_data_in[27][31];
wire fifo_to_fpga_wr1 = register_data_in[27][27];
wire fifo_to_fpga_wr2 = register_data_in[27][26];
wire [16:0] fifo_to_fpga_din = register_data_in[27][16:0];
</pre>

<pre>
assign register_data_out[27][31:24] = register_data_in[27][31:24]; // echo write bits
assign register_data_out[27][23] = fifo_to_fpga_full;
assign register_data_out[27][22] = fifo_to_fpga_empty;
</pre>

=== Register 28 0x80013070 data read fifo ===

<pre>
wire fifo_reset = register_data_in[28][31];
wire fifo_from_fpga_rd1 = register_data_in[28][25];
wire fifo_from_fpga_rd2 = register_data_in[28][24];
</pre>

<pre>
assign register_data_out[28][31:24] = register_data_in[28][31:24]; // echo write bits
assign register_data_out[28][21] = fifo_from_fpga_full;
assign register_data_out[28][20] = fifo_from_fpga_empty;
assign register_data_out[28][16:0] = fifo_from_fpga_dout;
</pre>

=== Register 29 0x80013074 packet routing ===

Control packet routing and loopbacks:

<pre>
wire [3:0] dn_route_ctrl = register_data_in[29][3:0];
wire [3:0] up_route_ctrl = register_data_in[29][7:4];
wire [3:0] fifo_to_fpga_route_ctrl = register_data_in[29][11:8];
//wire [3:0] spare_route_ctrl = register_data_in[29][15:12];

wire dn_mux_trg_enable = register_data_in[29][16];
wire dn_mux_tsm_enable = register_data_in[29][17];
wire dn_mux_sfp_rx_fifo_enable = register_data_in[29][18];
// 19
// 20..23

wire up_mux_vx_rx_enable = register_data_in[29][24];
// 25
// 26
// 27
wire fifo_from_fpga_hitmap_enable = register_data_in[29][28];
wire fifo_from_fpga_cdm_busy_enable = register_data_in[29][29];
wire fifo_from_fpga_gdm_busy_enable = register_data_in[29][30];
wire fifo_from_fpga_tsm_enable = register_data_in[29][31];
</pre>

fifo_to_fpga output routing:
* 0 - to down packet mux
* 1 - to up packet mux
* 2 - to fifo_from_fpga mux
* 3 - not used

down packet mux inputs:
* fifo_to_fpga_0
* trg_pkt16 enabled by dn_mux_trg_enable
* tsm_pkt16 enabled by dn_mux_tsm_enable
* sfp_rx_fifo_pkt16 enabled by dn_mux_sfp_rx_fifo_enable
* up_pkt16_2 loopback from up packet mux

down packet mux output routing:
* 0 - to vx_tx_pkt16 to VX TX (vx_link_tx and vx_ser_tx)
* 1 - to fifo_from_fpga mux
* 2 - to up packet mux loopback
* 3 - to qsfp_tx_pkt16 to GDM QSFP TX (cdm_link_tx)

up packet mux inputs:
* vx_rx_pkt16 enabled by up_mux_vx_rx_enable data from 12 VX RX links
* fifo_to_fpga_1
* dn_pkt16_2 loopback from down packet mux

up packet mux output routing:
* 0 - to sfp_tx_pkt16 to CDM SFP TX (VX data to GDM)
* 1 - to fifo_from_fpga mux
* 2 - to down packet mux loop loopback
* 3 - not used

internal_mux inputs:
* 0 - cdm_hitmap_pkt16 enabled by fifo_from_fpga_hitmap_enable (reg 29)
* 1 - cdm_busy_pkt16 enabled by fifo_from_fpga_cdm_busy_enable
* 2 - gdm_busy_pkt16 enabled by fifo_from_fpga_gdm_busy_enable
* 3 - gdm_hitmap_pkt16 enabled by fifo_from_fpga_hitmap_enable
* 4 - not used

debug_mux inputs:
* 0 - not used
* 1 - dn_pkt16_1
* 2 - up_pkt16_1
* 3 - qsfp0_rx_pkt16 enabled by fifo_from_fpga_qsfp0_rx_enable (from GDM QSFP link 0, there is no GDM QSFP 12-to-1 mux)
* 4 - not used

fifo_from_fpga mux inputs:
* 0 - fifo_to_fpga_2
* 1 - tsm2_pkt16 enabled by fifo_from_fpga_tsm_enable (reg 29)
* 2 - not used
* 3 - debug_pkt16 from debug_mux
* 4 - internal_pkt16 from internal_mux

=== Register 30 0x80013078 qsfp link status ports 0..7 ===

<pre>
assign register_data_out[30] =
{
qsfp_rx_data_error[7], qsfp_link_error[7], qsfp_link_status[7], qsfp_link_rx_status[7], // 7
qsfp_rx_data_error[6], qsfp_link_error[6], qsfp_link_status[6], qsfp_link_rx_status[6], // 6
qsfp_rx_data_error[5], qsfp_link_error[5], qsfp_link_status[5], qsfp_link_rx_status[5], // 5
qsfp_rx_data_error[4], qsfp_link_error[4], qsfp_link_status[4], qsfp_link_rx_status[4], // 4
qsfp_rx_data_error[3], qsfp_link_error[3], qsfp_link_status[3], qsfp_link_rx_status[3], // 3
qsfp_rx_data_error[2], qsfp_link_error[2], qsfp_link_status[2], qsfp_link_rx_status[2], // 2
qsfp_rx_data_error[1], qsfp_link_error[1], qsfp_link_status[1], qsfp_link_rx_status[1], // 1
qsfp_rx_data_error[0], qsfp_link_error[0], qsfp_link_status[0], qsfp_link_rx_status[0] // 0
};
</pre>

=== Register 31 0x8001307C qsfp link status ports 8..11 ===

<pre>
assign register_data_out[31] =
{
4'b0000,
4'b0000,
4'b0000,
4'b0000,
qsfp_rx_data_error[11], qsfp_link_error[11], qsfp_link_status[11], qsfp_link_rx_status[11], // 11
qsfp_rx_data_error[10], qsfp_link_error[10], qsfp_link_status[10], qsfp_link_rx_status[10], // 10
qsfp_rx_data_error[9], qsfp_link_error[9], qsfp_link_status[9], qsfp_link_rx_status[9], // 9
qsfp_rx_data_error[8], qsfp_link_error[8], qsfp_link_status[8], qsfp_link_rx_status[8] // 8
};
</pre>

=== Register 32 0x80013080 bitmap of enabled qsfp ports ===

<pre>
wire [11:0] qsfp_mask = register_data_in[32][11:0];
wire qsfp_bsy_force = register_data_in[32][12];
wire gdm_veto_enable = register_data_in[32][13];
// not used = register_data_in[32][15:14];
wire [15:0] gdm_veto_extend = register_data_in[32][31:16];
</pre>

On the GDM:

qsfp_rx_bsy[11..0] are pulses received from the CDMs

qsfp_rx_bsy_or is the grand-or of qsfp_rx_bsy masked by qsfp_mask

if qsfp_rx_bsy_or is high, gdm_bsy goes up and stays up for vx_bsy_extend*2 clocks

gdm_veto = gdm_bsy (bsy of any one VX causes trigger veto to all of them)

if (gdm_veto_enable) then gdm_veto will block trg_in_pulse

gdm_veto transition 0->1 causes gdm_veto_pulse. as long as gdm_veto is high, gdm_veto_pulse is generated every gdm_veto_extend clocks

gdm_veto_pulse is sent to all CDMs.

For this to work right, cdm_bsy_extend should not be bigger than gdm_veto_extend.

=== Register 33 0x80013084 bitmap of enabled VX ports ===

<pre>
wire [11:0] vx_mask = register_data_in[33][11:0];
wire vx_bsy_force = register_data_in[33][12];
// not used = register_data_in[33][15:13];
wire [15:0] vx_bsy_extend = register_data_in[33][31:16];
</pre>

On the CDM:

cdm_bsy is a grand-or of all vx_bsy masked by vx_mask (list of active VXes).

vx_bsy_extend controls how often state of cdm_bsy is sent to the GDM. when cdm_bsy goes 0->1, we send a cdm_bsy_pulse and keep resending it every vx_bsy_extend clocks as long as cdm_bsy stays high.

cdm_bsy_pulse is sent to the GDM.

sfp_rx_veto is the received from the GDM

if sfp_rx_veto goes up, cdm_veto goes up and stays up for gdm_veto_extend clocks.

for this to work right, CDM gdm_veto_extend must be bigger than GDM gdm_veto_extend.

cdm_veto goes to VXes on v1_tx_out[1] which is LVDS input 13.

=== Register 34, 35 0x80013088, 8C current timestamp ===

current 64-bit timestamp, 125 MHz

=== Register 36, 37 0x80013090, 94 old timestamp ===

old 64-bit timestamp, 125 MHz. timestamp saved at run start when it is reset to 0.

=== Register 38, 39, 40 0x80013098, 9C, A0 VX busy counters ===

8 bits per VX port, counters overflow to 255, reset at run start.

=== Register 41, 42, 43 0x800130A4, A8, AC QSFP busy counters ===

8 bits per QSFP port, counters overflow to 255, reset at run start.

=== Register 44, 45, 46, 47, 48 CDM and GDM busy and veto counters ===

* cdm_busy = grand-or of all VX busy for enabled VXes
* gdm_busy = grand-or of all CDM busy for enabled CDMs
* gdm_veto = gdm_busy

<pre>
44 | lo 16 bits | cdm_bsy_up_counter | CDM busy, increments when cdm_busy goes 0->1
44 | hi 16 bits | cdm_bsy_pulse_counter | CDM busy to GDM, increments for each cdm_bsy_pulse sent to the GDM

45 | lo 16 bits | cdm_veto_pulse_counter | CDM veto from GDM, increments for each sfp_rx_veto received from the GDM
45 | hi 16 bits | cdm_veto_up_counter | CDM veto to VX, increments each time cdm_veto is set to 1.

46 | lo 16 bits | gdm_bsy_pulse_counter | GDM busy from CDM, increments for each qsfp_rx_busy received from CDMs (unless they overlap)
46 | hi 16 bits | gdm_bsy_up_counter | GDM busy, increments each time gdm_bsy goes 0->1

47 | lo 16 bits | gdm_bsy_refresh_counter | GDM busy, increments each time gdm_bsy is extended by new qsfp_rx_busy
47 | hi 16 bits | spare |

48 | lo 16 bits | gdm_veto_up_counter | GDM veto, increments each time gdm_veto goes 0->1
48 | hi 16 bits | gdm_veto_pulse_counter | GDM veto to CDM, increments for each gdm_veto_pulse sent to the CDM
</pre>

=== Register 49, 50, 51 0x800130xx VX RX serial data monitor ===

VX RX data, 8-bit per VX channel. k-bit is omitted.

<pre>
assign register_data_out[49][7:0] = vx_rx_data[0]; // vx1
assign register_data_out[49][15:8] = vx_rx_data[1]; // vx2
assign register_data_out[49][23:16] = vx_rx_data[2]; // vx3
assign register_data_out[49][31:24] = vx_rx_data[3]; // vx4

assign register_data_out[50][7:0] = vx_rx_data[4]; // vx5
assign register_data_out[50][15:8] = vx_rx_data[5]; // vx6
assign register_data_out[50][23:16] = vx_rx_data[6]; // vx7
assign register_data_out[50][31:24] = vx_rx_data[7]; // vx8

assign register_data_out[51][7:0] = vx_rx_data[8]; // vx9
assign register_data_out[51][15:8] = vx_rx_data[9]; // vx10
assign register_data_out[51][23:16] = vx_rx_data[10]; // vx11
assign register_data_out[51][31:24] = vx_rx_data[11]; // vx12
</pre>

=== Register 52, 53 VX link status ===

<pre>
assign register_data_out[52] =
{
vx_rx_error[7], vx_link_error[7], vx_link_status[7], vx_link_rx_status[7], // 7
vx_rx_error[6], vx_link_error[6], vx_link_status[6], vx_link_rx_status[6], // 6
vx_rx_error[5], vx_link_error[5], vx_link_status[5], vx_link_rx_status[5], // 5
vx_rx_error[4], vx_link_error[4], vx_link_status[4], vx_link_rx_status[4], // 4
vx_rx_error[3], vx_link_error[3], vx_link_status[3], vx_link_rx_status[3], // 3
vx_rx_error[2], vx_link_error[2], vx_link_status[2], vx_link_rx_status[2], // 2
vx_rx_error[1], vx_link_error[1], vx_link_status[1], vx_link_rx_status[1], // 1
vx_rx_error[0], vx_link_error[0], vx_link_status[0], vx_link_rx_status[0] // 0
};

assign register_data_out[53] =
{
//4'b0000,
1'b0, ~vx_rx_deser_rdy[0], vx_rx_code_err[0], vx_rx_disp_err[0],
//4'b0000,
//4'b0000,
//4'b0000,
vx_rx_monitor[0],
vx_rx_error[11], vx_link_error[11], vx_link_status[11], vx_link_rx_status[11], // 11
vx_rx_error[10], vx_link_error[10], vx_link_status[10], vx_link_rx_status[10], // 10
vx_rx_error[9], vx_link_error[9], vx_link_status[9], vx_link_rx_status[9], // 9
vx_rx_error[8], vx_link_error[8], vx_link_status[8], vx_link_rx_status[8] // 8
};
</pre>

vx_rx_monitor (12-bit) is from deserializer_10b.sv:

<pre>
assign monitor_out[9:0] = lastByte[9:0];
assign monitor_out[10] = comma;
assign monitor_out[11] = ready;
</pre>

=== Register 54 VX TX serial data monitor ===

* contents of vx_tx_monitor from vx_ser_tx.sv:
<pre>
assign monitor_out[8:0] = data_to_encoder; // 8-bit + k
assign monitor_out[9] = valid;
assign monitor_out[15:10] = 0;
assign monitor_out[25:16] = encoded_data; // 10-bit
assign monitor_out[26] = encoded_valid;
assign monitor_out[27] = 0;
assign monitor_out[31:28] = 0;
</pre>

=== Register 55 QSFP, SFP, VX link loss counters ===

<pre>
assign register_data_out[55] = {
vx_rx_error_counter,
qsfp_link_rx_status_drop_counter,
sfp_link_rx_status_drop_counter,
vx_link_rx_status_drop_counter
};
</pre>

<pre>
24..31 - CDM VX RX error counters, count any errors in the VX RX data path (bad serial data, fifo overflow, etc)
16..23 - GDM QSFP link loss counter, increments on qsfp_tx_link_rx_status 1->0 (reg10)
8..15 - CDM SFP link loss counter, increments on sfp_link_rx_status 1->0 (reg15)
0...7 - CDM VX link loss counter, increments on vx_tx_link_rx_status 1->0 (reg10)
</pre>

=== Register 68 GPS control and status ===

<pre>
assign register_data_out[68] =
{
8'b00000000, // 23+8
rb_1pps_counter, // 16+8 bits
gps_1pps_counter, // 8+8 bits
1'b0,
1'b0,
rb_ser_in, // 5
gps_data_in, // 4
gps_aux_out, // 3
gps_aux_in, // 2
rb_1pps_in, // 1
gps_1pps_in // 0
};
</pre>

=== Register 71 packet error bits ===

<pre>
assign register_data_out[71] =
{
8'b00000000, // 24+8 bits
1'b0, // 23
1'b0, // 22
fifo_to_fpga_full_latch, // 21
fifo_from_fpga_full_latch, // 20
gdm_hitmap_fifo_full_latch, // 19
gdm_busy_fifo_full_latch, // 18
cdm_busy_fifo_full_latch, // 17
cdm_hitmap_fifo_full_latch, // 16
1'b0, // 15
1'b0, // 14
1'b0, // 13
gdm_trig48_encode_error_latch, // 12
1'b0, // vx_tx_trig48_pkt8_error_latch, // 11
tsm2_encode_error_latch, // 10
tsm_encode_error_latch, // 9
trg_encode_error_latch, // 8
gdm_hitmap_error_latch, // 7
gdm_hitmap_encode_error_latch, // 6
gdm_busy_encode_error_latch, // 5
cdm_busy_encode_error_latch, // 4
cdm_hitmap_encode_error_latch, // 3
vx_rx_hitmap_error_latch, // 2
1'b0, // vx_tx_tsm_pkt8_error_latch, // 1
1'b0, // vx_tx_trg_pkt8_error_latch // 0
};
</pre>

=== Register 90 VX busy status register ===

<pre>
vx_bsy_status_latched[11:0] <= vx_bsy[11:0];
vx_bsy_status_latched[23:12] <= vx_bsy_collect[11:0];
vx_bsy_status_latched[24] <= cdm_bsy;
vx_bsy_status_latched[25] <= cdm_bsy_collect;
vx_bsy_status_latched[26] <= cdm_veto;
vx_bsy_status_latched[27] <= cdm_veto_collect;
vx_bsy_status_latched[28] <= 0;
vx_bsy_status_latched[29] <= 0;
vx_bsy_status_latched[30] <= 0;
vx_bsy_status_latched[31] <= cdm_bsy_stuck;
</pre>

=== Register 91 VX hitmap status register ===

<pre>
hitmap_status_latched[11:0] <= hitmap_or12[11:0];
hitmap_status_latched[23:12] <= hitmap_or12_collect[11:0];
hitmap_status_latched[24] <= hitmap_grand_or;
hitmap_status_latched[25] <= hitmap_grand_or_collect;
hitmap_status_latched[26] <= 0;
hitmap_status_latched[27] <= 0;
hitmap_status_latched[28] <= 0;
hitmap_status_latched[29] <= 0;
hitmap_status_latched[30] <= 0;
hitmap_status_latched[31] <= hitmap_grand_or_stuck;
</pre>

=== Register 92 GDM busy status register ===

<pre>
gdm_bsy_status_latched[11:0] <= qsfp_rx_bsy[11:0];
gdm_bsy_status_latched[23:12] <= qsfp_rx_bsy_collect[11:0];
gdm_bsy_status_latched[24] <= gdm_bsy;
gdm_bsy_status_latched[25] <= gdm_bsy_collect;
gdm_bsy_status_latched[26] <= gdm_veto;
gdm_bsy_status_latched[27] <= gdm_veto_collect;
gdm_bsy_status_latched[28] <= 0;
gdm_bsy_status_latched[29] <= 0;
gdm_bsy_status_latched[30] <= 0;
gdm_bsy_status_latched[31] <= gdm_bsy_stuck;
</pre>

=== Register 93 GDM hitmap status register ===

<pre>
hitmap_status_latched[11:0] <= hitmap_grand_or_12[11:0];
hitmap_status_latched[23:12] <= hitmap_grand_or_12_collect[11:0];
hitmap_status_latched[24] <= hitmap_grand_or;
hitmap_status_latched[25] <= hitmap_grand_or_collect;
hitmap_status_latched[26] <= 0;
hitmap_status_latched[27] <= 0;
hitmap_status_latched[28] <= 0;
hitmap_status_latched[29] <= 0;
hitmap_status_latched[30] <= 0;
hitmap_status_latched[31] <= hitmap_grand_or_stuck;
</pre>

=== Register 98, 99 GDM TRIG48 test register and TRIG48_CTRL ===

<pre>
assign gdm_trig48_ctrl_bitmap48[31:0] = register_data_in[98][31:0];
assign gdm_trig48_ctrl_bitmap48[47:32] = register_data_in[99][15:0];

assign gdm_trig48_ctrl[15:0] = register_data_in[99][31:16];
</pre>

<pre>
wire enable_grand_or = trig48_ctrl[15];
wire enable_test_trig48 = trig48_ctrl[14];
</pre>

= Firmware registers branch develop_ko =

== Register map ==

<pre>
0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
1 | ALL | ALL | RW | read write scratch register
2 | ALL | CDM | ?? | MGT not used
3 | ALL | CDM | RO | MGT debug_data
4 | ALL | CDM | RW | clk_config_vec
5 | ALL | CDM | ?? | not used
6 | ALL | CDM | RO | CDM_link_data_processing:o_error_time
7 | ALL | CDM | RO | CDM_link_data_processing:o_error_count
</pre>

== register 0 0x80010000 ==

GDM:
<pre>
0 - gdm_link_interface:i_mgt_rst
2 - gdm_link_interface:i_link_down_latched_rst
8 - GDM_link_data_processing:i_rst
10..9 - GDM_link_data_processing:i_data_mode
</pre>

CDM:
<pre>
0 - cdm_link_interface:i_mgt_rst
2 - cdm_link_interface:i_link_down_latched_rst
8 - CDM_link_data_processing:i_rst
10..9 - CDM_link_data_processing:i_data_mode
</pre>

== register 1 0x80010004 ==

GDM:
<pre>
nlinks-1..0 - gdm_link_interface:i_rx_slide_trigger
</pre>

CDM:
<pre>
nlinks-1..0 - cdm_link_interface:i_rx_slide_trigger
</pre>

== register 2 0x80010008 ==

GDM:
<pre>
nlinks-1..0 - gdm_link_interface:o_link_power_good
nlinks+15..16 - gdm_link_interface:o_link_status
</pre>

CDM:
<pre>
nlinks-1..0 - cdm_link_interface:o_link_power_good
nlinks+15..16 - cdm_link_interface:o_link_status
</pre>

== register 3 0x8001000c ==

GDM: simple loopback register

CDM:
<pre>
31..0 - debug_data - cdm_link_interface:o_debug
</pre>

o_debug:
<pre>
rx_link_rst & rx_error & rx_link_up & rx_receiving_data &
std_logic_vector(rx_state_count) & tx_state_count_on_rx_clk & i_rx_ctrl3(0) &
i_rx_ctrl1(1 downto 0) & i_rx_ctrl0(1 downto 0) &
rx_data_is_k28p1_k28p5 &
i_rx_data;
</pre>

== register 4 0x80010010 ==

GDM write:
<pre>
0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS
</pre>

GDM read:
<pre>
0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS
4 - clk_config_vec(4) - CLK_LOSXTn_LS
5 - clk_config_vec(5) - CLK_LOLn_LS
6 - clk_config_vec(6) - CLK_INTn_LS
7 - constant 1
31..8 - constant 0
</pre>

== register 5 0x80010014 ==

not used

== register 6 0x80010018 ==

GDM:
<pre>
3..0 - GDM_link_data_processing:i_status_select
</pre>

CDM:
<pre>
31..0 - CDM_link_data_processing:o_error_count
</pre>

== register 7 0x8001001c ==

GDM:
<pre>
31..0 - GDM_link_data_processing:o_status_vector
</pre>

CDM:
<pre>
31..0 - CDM_link_data_processing:o_error_count
</pre>

= GDM, CDM, VX packet communications =

* timestamp math
<pre>
1 clock is 8 ns is 125 MHz
8 bits of clocks is 256 clocks is 2048 ns is ~2 usec
16 bits of clocks is ~500 usec is 0.5 msec
24 bits of clocks is ~134 msec
32 bits of clocks is ~34 sec
40 bits of clocks is ~8.7 ksec is 2.4 hours
48 bits of clocks is ~625 hours is ~26 days
56 bits of clocks is ~6.6 kdays is ~18 kyears
62 bits of clocks is ~10 Mhours is 427 kdays is ~1.1 kyears
64 bits of clocks is ~4.4 kyears
</pre>
* summary
<pre>
0x01 - not used
0x02 - 8 bytes - 4 words - 2 dwords - 1 qwords - TRG packet
0x03 - 12 bytes - 6 words - 3 dwords - 2 qwords - TRIG48 packet
0x10 - 26 bytes - 13 words - 7 dwords - 4 qwords - TSM packet
0x81 - 10 bytes - 5 words - 3 dwords - 2 qwords - VX_HITMAP packet
0x82 - 108 bytes - 54 words - 27 dwords - 14 qwords - CDM_HITMAP packet
0x83 - 12 bytes - 6 words - 3 dwords - 2 qwords - CDM_BUSY packet
0x84 - 12 bytes - 6 words - 3 dwords - 2 qwords - GDM_BUSY packet
0x85 - 12 bytes - 6 words - 3 dwords - 1 qwords - GDM_HITMAP packet
0x86 - 128 bytes - 64 words - 32 dwords - 16 qwords - CDM CUMULATIVE_HITMAP packet
</pre>
* 0x02 - TRG packet, 8 bytes, 80 adc clocks, 640 ns on lvds link
<pre>
0 - 0x02
1 - trg_counter[7:0]
2 - ts64 low byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 high byte 3
6 - trg_in_latch[7:0]
7 - trg_in_latch[15:8]
</pre>
* 0x03 - TRIG48 packet, 12 bytes, 120 adc clocks, 960 ns on lvds link
<pre>
0 - 0x03
1 - trg_counter[7:0]
2 - ts64 low byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 high byte 3
6 - vx_bitmap[7:0]
7 - vx_bitmap[15:8]
8 - vx_bitmap[23:16]
9 - vx_bitmap[31:24]
10 - vx_bitmap[39:32]
11 - vx_bitmap[47:40]
</pre>
* 0x10 - TSM packet, 26 bytes, 260 adc clocks, 2080 ns on lvds link
<pre>
0 - 0x10
1 - tsm_counter[7:0]
2 - gdm_ts64 low byte 0
3 - 1
4 - 2
5 - 3
6 - 4
7 - 5
8 - 6
9 - gdm_ts64 high byte 7
10 - gps_ts64 low byte 0
11 - 1
12 - 2
13 - 3
14 - 4
15 - 5
16 - 6
17 - gps_ts64 high byte 7
18 - gps_data64 low byte 0
19 - 1
20 - 2
21 - 3
22 - 4
23 - 5
24 - 6
25 - gps_data64 high byte 7
</pre>
* 0x81 - VX hitmap packet, 10 bytes, 100 adc clocks, 800 ns on lvds link, 48 ns on fiber link
<pre>
0 - 0x81
1 - VX ID
2 - hitmap low byte, nits 7:0
3 - 15:8
4 - 23:16
5 - 31:24
6 - ...:32
7 - ...
8 - ...
9 - hitmap low byte, bits 63:...
</pre>
* 0x82 - CDM hitmap packet, 108 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x82
1 - cdm_hitmap_trigger_counter[7:0]
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - cdm_hitmap_or12 byte 0 (7:0)
11 - cdm_hitmap_or12 byte 1 (11:8) plus 4 bits: 12=0, 13=0, 14=0, 15=cdm_hitmap_grand_or
12 - cdm_hitmap_data, low byte, 12*64 bits = 768 bits = 48 words = 96 bytes
...
107 - cdm_hitmap_data, high byte
</pre>
* 0x83 - CDM busy packet, 12 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x83
1 - cdm_busy_packet_counter[7:0], reset at begin of run
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - vx_bsy[7:0]
11 - {0,0,cdm_veto,cdm_bsy,vx_bsy[11:8]}
</pre>
* 0x84 - GDM busy packet, 12 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x84
1 - gdm_busy_packet_counter[7:0], reset at begin of run
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - qsfp_rx_bsy[7:0]
11 - {0,0,gdm_veto,gdm_bsy,qsfp_rx_bsy[11:8]}
</pre>
* 0x85 - GDM hitmap trigger packet, 12 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x85
1 - gdm_hitmap_packet_counter[7:0], reset at begin of run
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - hitmap_grand_or_12[7:0]
11 - {hitmap_grand_or,0,0,0,hitmap_grand_or_12[11:8]}
</pre>
* 0x86 - CDM cumulative hitmap packet, 128 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x86
1 - vx_rx_hitmap_packet_counter[7:0], reset at begin of run
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - reserved
11 - reserved
12 - repeating 12 times the 80-bit VX HITMAP packets: 0x81, VX ID, 64 bits of hitmap
</pre>

= AXI bus timing =

* AXI 100 MHz clock, 10 ns, 32-bit data
* AXI single-dword read: 36 clock repeat rate, 360 ns is 2.777 MHz, 4 bytes per transfer is 11.11 Mbytes/sec
* AXI single-qword read: 13 clock repeat rate, 130 ns is 7.7 MHz, 4 bytes per transfer is 30 Mbytes/sec, not accounting for the gap
* AXI memcpy read: 4 transfers at 13 clocks, gap, 4 transfers at 13 clocks, gap, etc. below 30 Mbytes/sec.
[[Image:Ds-dm-axi-read.png|100px]]
[[Image:Ds-dm-axi-read-64.png|100px]]
[[Image:Ds-dm-axi-read-memcpy.png|100px]]
* AXI single-dword write: 20 clocks repeat rate, 200 ns is 5 MHz, 4 bytes per transfer is 20 Mbytes/sec
* AXI single-qword write: 13+20 clocks repeat rate, 330 ns is 3 MHz, 16 bytes per burst is 48 Mbytes/sec
[[Image:Ds-dm-axi-write.png|100px]]
[[Image:Ds-dm-axi-write-64.png|100px]]

= AXI bus addresses =

* see AXI/AMBA addresses assigned inthe FPGA project: (s_axi/reg0 is the DS-DM AXI registers)
<pre>
daq00:ds-dm-gcdm$ grep assign_bd_address scripts/GDM_CDM_XU8_bd.tcl
assign_bd_address -offset 0x80010000 -range 0x00004000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs axi_register_interfa_0/s_axi/reg0] -force
assign_bd_address -offset 0x000400000000 -range 0x40000000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs ddr4/C0_DDR4_MEMORY_MAP/C0_DDR4_ADDRESS_BLOCK] -force
assign_bd_address -offset 0x80000000 -range 0x00010000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs system_management_wiz/S_AXI_LITE/Reg] -force
daq00:ds-dm-gcdm$
</pre>

* see AXI/AMBA addresses exported from FPGA project to Linux kernel: (uio for debug bridge should say "debug bridge")
<pre>
root@gdm0:~# cat /sys/class/uio/uio*/name
axi-pmon
axi-pmon
axi-pmon
axi-pmon
root@gdm0:~#
root@gdm0:~# ls -l /sys/class/uio/
total 0
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio0 -> ../../devices/platform/amba/ffa00000.perf-monitor/uio/uio0
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio1 -> ../../devices/platform/amba/fd0b0000.perf-monitor/uio/uio1
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio2 -> ../../devices/platform/amba/fd490000.perf-monitor/uio/uio2
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio3 -> ../../devices/platform/amba/ffa10000.perf-monitor/uio/uio3
root@gdm0:~# ls -l /sys/class/uio/../../devices/platform/amba/
total 0
-rw-r--r-- 1 root root 4096 Oct 18 19:37 driver_override
drwxr-xr-x 3 root root 0 Oct 18 01:36 fd070000.memory-controller
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd0b0000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd400000.zynqmp_phy
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd490000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd500000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd510000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd520000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd530000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd540000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd550000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd560000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd570000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd6e0000.cci
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff000000.serial
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff020000.i2c
drwxr-xr-x 6 root root 0 Oct 18 01:36 ff0a0000.gpio
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0b0000.ethernet
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0e0000.ethernet
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0f0000.spi
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff160000.mmc
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff170000.mmc
drwxr-xr-x 3 root root 0 Oct 18 01:36 ff960000.memory-controller
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff9d0000.usb0
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa00000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa10000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa50000.ams
drwxr-xr-x 5 root root 0 Oct 18 01:36 ffa60000.rtc
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa80000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa90000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffaa0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffab0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffac0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffad0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffae0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffaf0000.dma
-r--r--r-- 1 root root 4096 Oct 18 19:37 modalias
lrwxrwxrwx 1 root root 0 Oct 18 19:37 of_node -> ../../../firmware/devicetree/base/amba
drwxr-xr-x 2 root root 0 Oct 18 19:37 power
lrwxrwxrwx 1 root root 0 Oct 18 01:36 subsystem -> ../../../bus/platform
-rw-r--r-- 1 root root 4096 Oct 18 01:36 uevent
root@gdm0:~#
</pre>

= Build firmware =

== Build from git clone ==

THESE ARE K.O.'s NOTES FOR CREATING THE PETALINUX DIRECTORY.

THEY DO NOT WORK!

COPY PETALINUX FROM A WORKING PROJECT AND USE "make gdm" and "make cdm" AS DESCRIBED BELOW.

* git clone git@edev-group.triumf.ca:fw/exp/darkside/gcdm.git
* #Makefile change VIVADO_SETTINGS_SCRIPT := /opt/Xilinx/Vivado/2022.1/settings64.sh
* #. /opt/Xilinx/Vivado/2022.1/settings64.sh
* . /opt/Xilinx/Vivado/2020.2/settings64.sh
* make clean
* make all_from_scratch
* . /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
* make petalinux_create
* make petalinux_rebuild_new_hw_des
* bomb out: The TMPDIR: /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp can't be located on nfs.
* mkdir /tmp/build_tmp
* rm -rf /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp/
* ln -s /tmp/build_tmp /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp
* try again
* grinds, loads a whole bunch of packages...
* finishes with desire to copy things to /tftpboot
* make sdcard_cp_to wants to copy files from PetaLinux_GDM_CDM/images/linux/ to SD card

== Build firmware ==

NOTE: directory Petalinux_GDM_CDM should already exist!

<pre>
#. /opt/Xilinx/Vivado/2020.2/settings64.sh
. /opt/Xilinx/Vivado/2022.2/settings64.sh
. /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
make clean_gdm # remove gdm build tree
make gdm # build or rebuild GDM
make copy_gdm # copy to gdm0
make clean_cdm # remove cdm build tree
make cdm # build or rebuild CDM
make copy_cdm # copy to cdm0 and cdm1
</pre>

copy to SD card:
<pre>
open a root shell
format 16 GB Sd card per above
cd .../ds-dm-gcdm
make copy
</pre>

= build times =

<pre>
CDM 12-june-2023 69aabc1c25130d970bc375aca684bd68849e6685
daq13 AMD-5700G 1688.61user 399.33system 23:28.84elapsed 148%CPU
dsdaqgw AMD-7700 1090.55user 247.34system 16:03.55elapsed 138%CPU
dsdaqgw AMD-7700 CDM incremental 196.68user 67.62system 7:35.42elapsed 58%CPU
dsdaqgw AMD-7700 CDM incremental 684.72user 94.17system 7:30.17elapsed 173%CPU
dsdaqgw AMD-7700 GDM incremental 849.84user 99.79system 9:04.56elapsed 174%CPU
</pre>

= make bootable sd card =

== format the sd card ==

this only needs to be done once

* become root@dsdaqgw
* cd ~olchansk/git/ds-dm-gcdm
* use "lsblk" to identify the SD card (should show as 8/16/32 GB block device)/ /dev/sdd in this case
* make sdcard_format SDCARD_DEVICE=/dev/sdd
* disconnect sd card, reconnect the sd card (to detect new partition tables, etc)

== copy boot files to the sd card ==

* as root: identify partition labels, run "blkid", should say "BOOT", "rootfs" and "data"
* mount
<pre>
mkdir -p /mnt/BOOT
mount -L BOOT /mnt/BOOT
cp PetaLinux_GDM_CDM/images/linux/BOOT_{GDM,CDM}.BIN /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/boot.scr /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/image.ub /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/uboot.env /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/uboot-redund.env /mnt/BOOT/
umount /mnt/BOOT
eject /dev/sdX
</pre>
* FIXME: missing step: cp BOOT_GDM.BIN BOOT.BIN or cp BOOT_CDM.BIN BOOT.BIN

= boot messages =

<pre>
Xilinx Zynq MP First Stage Boot Loader
Release 2020.2 Sep 24 2022 - 13:29:15
NOTICE: ATF running on XCZU4CG/silicon v4/RTL5.1 at 0xfffea000
NOTICE: BL31: v2.2(release):xlnx_rebase_v2.2_2020.3
NOTICE: BL31: Built : 18:02:46, Sep 28 2022

U-Boot 2020.01 (Sep 28 2022 - 18:03:39 +0000)

Model: DarkSide 20k DM
Board: Xilinx ZynqMP
DRAM: 2 GiB
usb dr_mode not found
PMUFW: v1.1
EL Level: EL2
Chip ID: zu4
NAND: 0 MiB
MMC: mmc@ff160000: 0, mmc@ff170000: 1
In: serial@ff000000
Out: serial@ff000000
Err: serial@ff000000
Bootmode: SD_MODE1
Reset reason: SOFT
Net:
ZYNQ GEM: ff0b0000, mdio bus ff0b0000, phyaddr -1, interface rgmii-id

Warning: ethernet@ff0b0000 (eth0) using random MAC address - d6:62:5f:13:00:44
eth0: ethernet@ff0b0000
ZYNQ GEM: ff0e0000, mdio bus ff0e0000, phyaddr -1, interface rgmii-id
Could not get PHY for eth1: addr -1

Hit any key to stop autoboot: 0
ZynqMP>
CTRL-A Z for help | 115200 8N1 | NOR | Minicom 2.7.1 | VT102 | Online 122:2 | ttyACM0
</pre>

= load FPGA from u-boot =

* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/124682257/U-Boot+FPGA+Driver

<pre>
ZynqMP> fpga info
Xilinx Device
Descriptor @ 0x000000007fddb2c0
Family: ZynqMP PL
Interface type: csu_dma configuration interface (ZynqMP)
Device Size: 1 bytes
Cookie: 0x0 (0)
Device name: zu4
Device Function Table @ 0x000000007fda5fe8
PCAP status 0xa0002fde
ZynqMP>
</pre>

* cp CDM_XU8_top.bit /tftpboot/fpga.bit

<pre>
dhcp
tftpb 0x10000000 fpga.bit
fpga loadb 0 0x10000000 ${filesize}
</pre>

<pre>
ZynqMP> dhcp
BOOTP broadcast 1
DHCP client bound to address 192.168.0.100 (1 ms)
*** Warning: no boot file name; using 'C0A80064.img'
Using ethernet@ff0b0000 device
TFTP from server 192.168.0.1; our IP address is 192.168.0.100
Filename 'C0A80064.img'.
Load address: 0x8000000
Loading: *
TFTP error: 'file /tftpboot/C0A80064.img not found for 192.168.0.100' (1)
Not retrying...
ZynqMP> tftpb 0x10000000 fpga.bit
Using ethernet@ff0b0000 device
TFTP from server 192.168.0.1; our IP address is 192.168.0.100
Filename 'fpga.bit'.
Load address: 0x10000000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
############
6.2 MiB/s
done
Bytes transferred = 7797807 (76fc2f hex)
ZynqMP> fpga loadb 0 0x10000000 ${filesize}
design filename = "CDM_XU8_top;UserID=0XFFFFFFFF;Version=2022.2"
part number = "xczu4cg-fbvb900-1-e"
date = "2024/08/14"
time = "14:18:22"
bytes in bitstream = 7797692
zynqmp_align_dma_buffer: Align buffer at 0000000010000073 to 000000000fffff80(swap 0)
ZynqMP>
</pre>

= load FPGA from Linux =

* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841847/Solution+ZynqMP+PL+Programming

== this will reset the CPU ==
<pre>
cp fpga.bit /lib/firmware/
echo fpga.bit > /sys/class/fpga_manager/fpga0/firmware
</pre>

* make .bin file:
<pre>
bootgen -image CDM_XU8_top.bif -arch zynqmp -o ./fpga.bin -w
</pre>
* cat CDM_XU8_top.bif
<pre>
dsdaqgw:ds-dm-gcdm$ cat CDM_XU8_top.bif
all:
{
[destination_device = pl] ./Vivado_CDM_XU8/CDM_XU8.runs/impl_1/CDM_XU8_top.bit
}
dsdaqgw:ds-dm-gcdm$
</pre>

== this will reset the CPU ==

<pre>
cp fpga.bin /lib/firmware/
echo fpga.bin > /sys/class/fpga_manager/fpga0/firmware
</pre>

== this will reset the CPU ==

<pre>
root@dsdm:~# ./fpgautil -b fpga.bin -f Full
</pre>

== this will reset the CPU ==

<pre>
DTSO file from here: https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841847/Solution+ZynqMP+PL+Programming
echo 0 > /sys/class/fpga_manager/fpga0/flags
mount -t configfs configfs /configfs
root@dsdm:~# dtc -O dtb -o fpga.dtbo -b 0 -@ fpga.dtso
root@dsdm:~# cp fpga.dtbo /lib/firmware/
root@dsdm:~# cp fpga.bit /lib/firmware/
root@dsdm:~# rmdir /configfs/device-tree/overlays/fpga
root@dsdm:~# mkdir /configfs/device-tree/overlays/fpga
root@dsdm:~# echo -n "fpga.dtbo" > /configfs/device-tree/overlays/fpga/path
</pre>

= fpgautil =

* https://github.com/Xilinx/meta-xilinx/blob/master/meta-xilinx-core/recipes-bsp/fpga-manager-script/files/fpgautil.c

<pre>
git clone https://github.com/Xilinx/meta-xilinx.git
cd meta-xilinx/meta-xilinx-core/recipes-bsp/fpga-manager-script/files/
scp fpgautil.c root@dsdm:
ssh root@dsdm
make fpgautil
ls -l ./fpgautil
</pre>

<pre>
root@dsdm:~# ls -l ./fpgautil
-rwxr-xr-x 1 root root 72256 Aug 16 00:15 ./fpgautil
root@dsdm:~# ./fpgautil

fpgautil: FPGA Utility for Loading/reading PL Configuration

Usage: fpgautil -b <bin file path> -o <dtbo file path>

Options: -b <binfile> (Bin file path)
-o <dtbofile> (DTBO file path)
-f <flags> Optional: <Bitstream type flags>
f := <Full | Partial >
-n <Fpga region info> FPGA Regions represent FPGA's
and partial reconfiguration
regions of FPGA's in the
Device Tree
Default: <full>
-s <secure flags> Optional: <Secure flags>
s := <AuthDDR | AuthOCM | EnUsrKey | EnDevKey | AuthEnUsrKeyDDR | AuthEnUsrKeyOCM | AuthEnDevKeyDDR | AuthEnDevKeyOCM>
-k <AesKey> Optional: <AES User Key>
-r <Readback> Optional: <file name>
Default: By default Read back contents will be stored in readback.bin file
-t Optional: <Readback Type>
0 - Configuration Register readback
1 - Configuration Data Frames readback
Default: 0 (Configuration register readback)
-R Optional: Remove overlay from a live tree

Examples:
(Load Full bitstream using Overlay)
fpgautil -b top.bit.bin -o can.dtbo -f Full -n full
(Load Partial bitstream using Overlay)
fpgautil -b rm0.bit.bin -o rm0.dtbo -f Partial -n PR0
(Load Full bitstream using sysfs interface)
fpgautil -b top.bit.bin -f Full
(Load Partial bitstream using sysfs interface)
fpgautil -b rm0.bit.bin -f Partial
(Load Authenticated bitstream through the sysfs interface)
fpgautil -b top.bit.bin -f Full -s AuthDDR
(Load Parital Encrypted Userkey bitstream using Overlay)
fpgautil -b top.bit.bin -o pl.dtbo -f Partial -s EnUsrKey -k <32byte key value>
(Read PL Configuration Registers)
fpgautil -b top.bit.bin -r
(Remove Partial Overlay)
fpgautil -R -n PR0
(Remove Full Overlay)
fpgautil -R -n full
Note: fpgautil -R is responsible for only removing the dtbo file from the livetree. it will not remove the PL logic from the FPGA region.

root@dsdm:~#
</pre>

= fw_printenv =

to access u-boot environment from Linux:
* apt install -y libubootenv-tool
* create /etc/fw_env.config
<pre>
/media/BOOT/uboot.env 0 0x40000
/media/BOOT/uboot-redund.env 0 0x40000
</pre>
* if uboot.env files do not exist, run "saveenv" from u-boot command prompt
* fw_printenv and fw_setenv should work

= Boot from network =

== u-boot ==

<pre>
ZynqMP> setenv bootcmd run bootcmd_dhcp
ZynqMP> saveenv
ZynqMP> reset
</pre>

== boot.scr ==

<pre>
# boot.scr
# mkimage -C none -A arm -T script -d boot.scr boot.scr.uimg
echo Loading FPGA!
#tftpb 0x10000000 fpga.bit
tftpb 0x10000000 {ipaddr}.bit
fpga loadb 0 0x10000000 ${filesize}
echo Booting Linux!
run bootcmd_pxe
echo Done!
</pre>

<pre>
mkimage -C none -A arm -T script -d boot.scr boot.scr.uimg
</pre>

== tftpboot ==

<pre>
cp /home/olchansk/git/ds-dm-gcdm/boot.scr.uimg /tftpboot
ln -s /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/images/linux /tftpboot/xilinx-dsdm
mkdir /tftpboot/pxelinux.cfg
cat > /tftpboot/pxelinux.cfg/default-arm-zynqmp <<EOF
LABEL Linux
KERNEL xilinx-dsdm/Image
FDT xilinx-dsdm/system.dtb
#INITRD rootfs.cpio.gz.u-boot
EOF
</pre>

== boot sequence ==

* xilinx magic load BOOT.BIN from SD card
* load FPGA form BOOT.BIN
* load and run u-boot from BOOT.BIN or from image.ub
* u-boot load environment from ??? probably SD card uboot-redund.env, this includes our bootcmd
* run bootcmd which run bootcmd_dhcp which does:
* from /tftpboot:
* load and run boot.scr.uimg which does:
* load FPGA image xilinx-dsdm/${ipaddr}.bit
* run bootcmd_pxe which does:
* load pxelinux.cfg/default-arm-zynqmp which does:
* load xilinx-dsdm/Image ### this is the linux kernel
* load xilinx-dsdm/system.dtb ### this is the device tree
* start linux kernel
* linux kernel does dhcp
* linux kernel does nfs mount /nfsroot/%s,vers=3,tcp ### %s is replaced by the hostname supplied by DHCP
* userland starts and runs to console and ssh login.

= Xilinx ILA =

References:
* https://github.com/Xilinx/XilinxVirtualCable/tree/master
* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/644579329/Xilinx+Virtual+Cable
* https://support.xilinx.com/s/article/974879?language=en_US
* https://docs.xilinx.com/v/u/en-US/pg245-debug-bridge
* https://docs.xilinx.com/r/2020.2-English/ug908-vivado-programming-debugging/Initializing-Vivado-IDE-hw_server
* (we do not use this one) https://github.com/paulscherrerinstitute/xvcSupport/blob/master/README.md

Build xvcserver_cdm.exe: (it is built as a static executable, can be copied and run anywhere)
<pre>
ssh dsdaq@gdm0
cd /home/dsdaq/online/ds-dm-software
git pull ### get latest version
make xvcserver_cdm.exe
ssh root@gdm0
/home/dsdaq/online/ds-dm-software/xvcserver_cdm.exe
INFO: To connect to this xvcServer instance, use url: TCP:gdm0:2542
</pre>

To activate and use the vivado logic analyzer:
* data path: vivado -> hw_server -> xvcserver -> mmap axi bus -> debug bridge -> jtag -> ILA
* define ILAs in the code
* instantiate the xilinx debug bridge at AXI bus address 0x80020000 (FIXME!!! this collides with Ian's AXI addresses)
* build and boot the new FPGA firmware. updating the linux kernel is not necessary.
* login root@gdm0, run: /home/dsdaq/online/ds-dm-software/xvcserver_cdm.exe -v ### with "-v" for the first time to see that vivado does connect to it, without "-v", normally.
* login dsdaqgw, run: hw_server -s tcp:localhost:3121 -e "set auto-open-servers xilinx-xvc:gdm0:2542" ### tells us to connect to port localhost:3121
* login dsdaqgw, run vivado, open project, open hardware manager, open target, open new target, "connect to remote server", hostname "localhost", port "3121", next (bombs, try again, 3 times), popup add virtual cable, enter hostname "gdm0" port "2542", "ok", it shows in "hardware targets", "next", "finish", error popup "[Common 17-163] Missing value for option 'objects', please type 'set_property -help' for usage info", ignore it, in "hardware", right click the "gdm0" one, open target, under "hardware" and "debug bridge" we should see all the ILAs, under "hardware device properties", the "probes file" should have the ".ltx" file generated by vivado "Vivado_GDM_XU8/GDM_XU8.runs/impl_1/debug_nets.ltx", click on an ILA, a waveform should open.

= Software =

* ssh cdm0 # or gdm0
* sudo apt install i2c-tools libi2c-dev
* git clone https://bitbucket.org/team-ds-dm/ds-dm-software
* cd ds-dm-software
* make

== test_cdm.exe ==

=== CDM SFP status ===

<pre>
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0xbb2f0ae7
CDM firmware 0xbb2f0ae7
arg 1: [--sfp]
Polling SFP status...
identifier 0x03
connector 0x07
encoding 0x01
wavelength 0x0352 (850 nm)
vendor_name [FINISAR CORP. ]
vendor_pn [FTLF8526P3BNL ]
vendor_rev [A ]
vendor_sn [N3AB9M8 ]
vendor_date [200319 ]
dm_type 0x68
temp 29.0 C
vcc 3.323 V
tx_bias 7.250 mA
tx_power 478.4 uW
rx_power 2.3 uW
SFP good 1, status: temp 30.6 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 818 uW
...
</pre>

=== GDM QSFP status ===

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --qsfp3 --qsfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x53aee418
CDM firmware 0x53aee418
arg 1: [--qsfp3]
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out hi
arg 2: [--qsfp]
Polling QSFP status...
identifier 0x0d
status 0x02
los 0x8f
temp 28.2 C
vcc 3.323 V
rx_power 0.1 0.1 0.1 0.1 uW
tx_bias 7.6 7.6 7.6 0.0 mA
tx_power 792.2 773.8 823.0 0.1 uW
vendor_name [FINISAR CORP ]
vendor_pn [FTL410QD4C ]
vendor_rev [A ]
wavelength 850
max_temp 70 C
vendor_sn [X79AC0R ]
vendor_date [220309 ]
QSFP good 1, status: temp 27.7 C, los 0x8b, tx_bias 7.5 7.6 7.6 0.0 mA, tx_power 792 772 821 0 uW, rx_power 0 0 466 0 uW
</pre>

=== GDM clock status ===

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
</pre>

* clock chip not loaded, not running:

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x0f 0x16 0x04 0x00 0x01 0x68 0x09 0x00 0x02 0xf2 0x00 0x1f 0xf0 0x22 0xf2, 0x507: 0x00, 0x52A: 0x00, 0x53F: 0x04
Clock chip state 0, status: SYSINCAL XAXB_ERR LOL CAL_PLL IN0 IN_SEL_0 FASTLOCK_STATUS

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x1f7caf52 (528265042) should be ~125 MHz
0x1034 rx_clk: 0x00000000 (0) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x1f7caf52 (528265042) should be ~125 MHz
0x103C tx_clk: 0x00000000 (0) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip good (IN0 - external 10 MHz clock)

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x44 0x00 0xd0 0x01 0x1f 0xff 0x22 0xf2, 0x507: 0x3f, 0x52A: 0x01, 0x53F: 0x00
Clock chip state 1, status: LOS_IN2 OOF_IN2 IN0 IN_SEL_REGCTRL IN_SEL_0

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735a3b (125000251) should be ~125 MHz
0x1034 rx_clk: 0x07735a3c (125000252) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a3b (125000251) should be ~125 MHz
0x103C tx_clk: 0x07735a3c (125000252) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

=== CDM clock status ===

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
</pre>

* clock chip not loaded, not running:

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x0f 0x16 0x04 0x00 0x01 0x68 0x19 0x00 0x02 0xf2 0x00 0x1f 0xf0 0x22 0xf2, 0x507: 0x00, 0x52A: 0x00, 0x53F: 0x04
Clock chip state 0, status: SYSINCAL XAXB_ERR LOL CAL_PLL IN0 IN_SEL_0 FASTLOCK_STATUS

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735851 (124999761) should be ~125 MHz
0x1034 rx_clk: 0x00000000 (0) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x0127fefa (19398394) should be ~125 MHz
0x103C tx_clk: 0x00000000 (0) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip uses internal clock (IN1 - internal oscillator), observe rx_clk frequency is not same as others

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0x7f, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN1 IN_SEL_1 HOLD_HIST_VALID

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735853 (124999763) should be ~125 MHz
0x1034 rx_clk: 0x07735b0a (125000458) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735853 (124999763) should be ~125 MHz
0x103C tx_clk: 0x07735852 (124999762) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip uses SFP recovered clock (IN2 - sfp rx recovered clock), observe mgt_rx_ref_clk_raw (CDM 125 MHz oscillator) is different from others (SFP RX recovered clock)

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0xbf, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN2 IN_SEL_1 HOLD_HIST_VALID
^C
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773581b (124999707) should be ~125 MHz
0x1034 rx_clk: 0x07735ad6 (125000406) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735ad7 (125000407) should be ~125 MHz
0x103C tx_clk: 0x07735ad6 (125000406) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
^C
</pre>

=== CDM link status, PRBS test mode ===

<pre>
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
</pre>
* fiber disconnected, no link:
<pre>
CDM firmware: 0xbb2f0ae7
0x1000 SFP c.c. status: 0x00000031
CLK_IN_SEL_LS 0x1
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x00000025
sfp_mod_absent_N 1
sfp_rx_los_N 0
link_power_good 1
rx_link_up 0
rx_receiving_data 0
rx_error 1
rx_lnk_up_and_running 0
tx_link_up 0
tx_sending_data 0
tx_link_up_and_running 0
link_up_and_running 0
0x1014 SFP link data: 0x466a8187
rx_data 0x8187
k28p1_k28p5 0
rx_ctrl0 0x1
rx_ctrl1 0x1
rx_ctrl3 0x1
tx_state 0x1
rx_state 0x6
rx_receiving_data 0
rx_link_up 0
rx_error 1
rx_link_rst 0
0x2000 link test mode: 0x00000200, seconds: 0x00079093, errors: 0xffffffff
</pre>
* fiber connected, good link:
<pre>
CDM firmware: 0xbb2f0ae7
0x1000 SFP c.c. status: 0x000000b2
CLK_IN_SEL_LS 0x2
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x000007dc
sfp_mod_absent_N 0
sfp_rx_los_N 0
link_power_good 1
rx_link_up 1
rx_receiving_data 1
rx_error 0
rx_lnk_up_and_running 1
tx_link_up 1
tx_sending_data 1
tx_link_up_and_running 1
link_up_and_running 1
0x1014 SFP link data: 0x35c02774
rx_data 0x2774
k28p1_k28p5 0
rx_ctrl0 0x0
rx_ctrl1 0x0
rx_ctrl3 0x0
tx_state 0x3
rx_state 0x5
rx_receiving_data 1
rx_link_up 1
rx_error 0
rx_link_rst 0
0x2000 link test mode: 0x00000200, seconds: 0x00078a8c, errors: 0x00000000
</pre>

=== GDM link status, PRBS test mode ===

<pre>
1 link connected, no errors:
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
GDM firmware: 0x6b2ee010
0x1014: 0x00000008, 18: 0x00000008, 1C: 0x00000000, 24: 0x00000fff
0x2000: 0x00000200, time: 0x00078aa4, errors:
0xffffffff 0xffffffff 0xffffffff
0x00000000 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
</pre>

=== CDM link status ===

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test0

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0xbf, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN2 IN_SEL_1 HOLD_HIST_VALID

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x077357a0 (124999584) should be ~125 MHz
0x1034 rx_clk: 0x07735a5c (125000284) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a5b (125000283) should be ~125 MHz
0x103C tx_clk: 0x07735a5b (125000283) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM firmware: 0x6d2ef81a
0x1000 SFP c.c. status: 0x000000b2
CLK_IN_SEL_LS 0x2
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x000007dc
sfp_mod_absent_N 0
sfp_rx_los_N 0
link_power_good 1
rx_link_up 1
rx_receiving_data 1
rx_error 0
rx_lnk_up_and_running 1
tx_link_up 1
tx_sending_data 1
tx_link_up_and_running 1
link_up_and_running 1
0x1014 SFP link data: 0x35c6bcbc
rx_data 0xbcbc
k28p1_k28p5 0
rx_ctrl0 0x3
rx_ctrl1 0x0
rx_ctrl3 0x0
tx_state 0x3
rx_state 0x5
rx_receiving_data 1
rx_link_up 1
rx_error 0
rx_link_rst 0
0x2000 link test mode: 0x00000000, seconds: 0x00001671, errors: 0xffffffff

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 15
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
reg[15] is 0x0033bcbc (3390652)
</pre>

=== GDM link status ===

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test0

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x44 0x00 0xd0 0x01 0x1f 0xff 0x22 0xf2, 0x507: 0x3f, 0x52A: 0x01, 0x53F: 0x02
Clock chip state 1, status: LOS_IN2 OOF_IN2 IN0 IN_SEL_REGCTRL IN_SEL_0 HOLD_HIST_VALID

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735a0a (125000202) should be ~125 MHz
0x1034 rx_clk: 0x07735a0a (125000202) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a0a (125000202) should be ~125 MHz
0x103C tx_clk: 0x07735a0a (125000202) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly

register 0x1018 bit 0x800

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM firmware: 0x6d2ebce6
0x1014: 0x00000800, 18: 0x00000800, 1C: 0x00000000, 24: 0x00000fff
0x2000: 0x00000000, time: 0x00003d2f, errors:
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff

link data alternates 0xbcbc and 0x1cbc

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 22
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
reg[22] is 0xbcbc93ab (-1128492117)

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 22
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
reg[22] is 0x1cbc1aaf (482089647)
root@gdm0:~#

</pre>

=== Run trg and tsm ===

<pre>
on the GDM:

ssh root@gdm00
./test_cdm.exe --gdm-clocks
./test_cdm.exe --load-cc
./test_cdm.exe --cc
./test_cdm.exe --reset-mgt
./test_cdm.exe --gdm-clocks
./test_cdm.exe --writereg 2 0xff # enable LEMO NIM inputs
./test_cdm.exe --writereg 3 0xba54 # LEDs: lemo1, lemo2, trg, tsm
./test_cdm.exe --writereg 4 0x99 $ # LEMO out is trg_in_pulse
./test_cdm.exe --writereg 9 0x32010 # trg and tsm from trg_pulser and tsm_pulse
./test_cdm.exe --writereg 24 1250000 # trg pulser 100 Hz
./test_cdm.exe --writereg 26 125000000 # tsm pulser 1 Hz
./test_cdm.exe --writereg 23 0x40000000 # route trg_in and tsm_in to qsfp tx bits 0 and 1
./test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz

on the CDM:

ssh root@cdm01
./test_cdm.exe --cdm-clocks
./test_cdm.exe --load-cc
./test_cdm.exe --reset-mgt
./test_cdm.exe --cdm-clocks
./test_cdm.exe --cdm-link # issue --reset-mgt on CDM and GDM until link is good
./test_cdm.exe --writereg 2 0xff # enable LEMO NIM inputs
./test_cdm.exe --writereg 3 0xba54 # enable LEDs: lemo1, lemo2, trg, tsm
./test_cdm.exe --writereg 4 0x99 # enable LEMO output trg_in
./test_cdm.exe --writereg 9 0x8040 # enable trg_in and tsm_in from sfp rx bits 0 and 1
./test_cdm.exe --writereg 7 0xff00 # drive VX LVDS lines to logic level 0
./test_cdm.exe --writereg 8 0x2 # VX LVDS with trg and tsm, misrouted in second VX
#./test_cdm.exe --writereg 8 0x7 # VX LVDS with tsm
./test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz
</pre>

switch GDM and CDM to packetizer trg and tsm:

<pre>
on the GDM:

/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x00000000

on the CDM:

/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x0804
/home/dsdaq/online/ds-dm-software/test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz
</pre>

=== Run packet loopback ===

GDM CPU -> fifo_to_fpga -> GDM QSFP -> CDM SFP -> fifo_from_fpga -> CDM CPU

On the GDM: (CDM is connected to first QSFP port)

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --load-cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --reset-mgt
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 32 1 ### tell GDM to use first QSFP port
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link ### confirm link status is "3"
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 29 2 ### enable GDM packet data injection
/home/dsdaq/online/ds-dm-software/test_cdm.exe --test-fifo-write-loop
</pre>

One the CDM:

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --load-cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --reset-mgt
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 29 3 ### enable CDM packet data injection
/home/dsdaq/online/ds-dm-software/test_cdm.exe --test-fifo-read
</pre>

== fecdm ==

=== Power up and clock management ===

* Clock power up and setup management is handled by fecdm state machine
* GDM clock setup sequence:
<pre>
- on power up, CC is not initialized, XAXB clock is not running
- on GDM frontend start (in HandleInit()), CC is loaded and reset
- CC will lock to IN0 (10 MHz external clock) or IN1 (125 MHz internal oscillator)
- if locked to IN0, CC will enter and exit the HOLDOVER mode if 10 MHz clock is unstable
</pre>
* CDM clock setup sequence:
<pre>
- on power up, CC is not initialized, XAXB clock is not running
- fecdm detects this, loads the CC configuration registers, resets the CC
- sets the CC input IN1, DSDM internal oscillator
- if SFP is in good state, SFP MGT is running, CC is happy with IN2 input (no IN2_OOF, IN2_LOS), SFP recpvered clock (clk_rx_mgt) frequency is close to 125 MHz:
- switches CC to input IN2 (SFP recovered clock, clk_rx_mgt)
- waits for CC locked to IN2 (no LOL, no HOLD). (this provides stable SFP tx clock).
- resets the SFP MGT, requests GDM MGT reset (via ODB)
- MGT reset sequence: GDM MGT reset 1, CDMnn MGT reset 1, GDM MGT reset 0, CDM MGT reset 0
- CDM MGT rx_state cycles from 0 to 5. if it goes to 6 or never reaches 5, GDM MGT reset is requested again.
- during this sequence, clk_rx_mgt may wobble, CC may enter HOLDOVER mode, VX clock should stay steady.
- once CC is locked to IN2, SFP MGT rx_state is 5, link is ready for use.
</pre>
* Request changes for Ian
<pre>
17 Apr 2025:
- CDM SFP MGT rx_state is getting stuck in 4 (waiting for alternating idle chars), do we need this? go to state 5 without waiting?
- GDM QSFP MGT reset 1 does not seem to do anything, change it to cause all CDM MGTs to enter an error state, e.g. start transmitting invalid K-codes. transmitting all 0 or all 1 will break the recovered clock, so maybe do not do that.
- having all CDM SFP MGTs go to error state immediatley after GDM QSFT MGT reset 1 will greatly simplify the link reset sequence.
</pre>

=== Trigger configuration ===

* GDM
* CDM in SFP trigger mode:
** SFP K-code encoded sfp_rx_trg, sfp_rx_tsm are routed to LVDS vx_tx_trg and vx_tx_tsm
** GDM TRG, TSM (and all other) packets are routed to LVDS serial data to VX
** to make counters count and lights blink:
** CDM TRG and TSM trigger mask is set to sfp_rx_trg, sfp_rx_tsm and vx_tx_trg_done, vx_tx_tsm_done
** CDM LEDs display trg_in_pulse, tsm_in_pulse
** output of TRG and TSM packets is disabled (they come from the GDM)

= dsvslice integration =

== VX setup ==

* general
** Start acq from user code = y, all others = n (as of Mar 2023: will start when first trigger received)
** Use NIM IO = y
** Use external clock = y
** LVDS quartet is input = n, y, n, y
** LVDS quartet mode = User, User, User, User

* trigger from front panel NIM:
** Trigger on external signal = y, all others = n
** connect CDM EXT_OUT(2) to VX "TrigIn"

* trigger from LVDS "Sync" mode
** Trigger on LVDS Sync signal = y, all others = n
** LVDS quartet mode = User, Sync, User, User

* trigger from LVDS "User" mode
** Trigger on LVDS pair 12 signal = y, all others = n
** LVDS quartet is input = n, y, n, y
** LVDS quartet mode = User, User, User, User

== GDM setup ==

* GDM is gdm0
* set inputs to NIM mode
* set outputs to TTL mode (this GDM has wrong NIM output circuit)
* use top QSFP slot, connect split cable 1 into CDM01, cable 2 into CDM02
* connect non-inverted NIM trigger signal to top-LEMO-left EXT_IN_LV(1)
* connect non-inverted NIM TSM signal to top-LEME-right EXT_IN_LV(2)
* GDM LEDs: TRIG, TSM, trigger enabled, trigger_out
* GDM LEMO_OUT: trigger, trigger

== CDM setup ==

* set CDM LEMO inputs to NIM
* set CDM LEMO outputs to NIM
* CDM01 is cdm0
* CDM02 is cdm1
* connect GDM fiber links to SFP port
* connect 1st VX port of CDM01 to VX1
* connect 1st VX port of CDM02 to VX2
* connect LEMO EXT_OUT(2) to VX "TrigIn", CDM01 to VX1, CDM02 to VX2
* power up
* CDM LEDs: GDM TRIG, GDM TSM, trigger enabled, trigger out
* CDM LEMO_OUT: gdm_trg, trigger

== After power up ==

* start the CDM frontend from the MIDAS "Programs" page. To start manually, see the Start Command on the Programs page.
* CDM frontend should enable the VX clock, disable the trigger
* from the MIDAS status page, goto the CDM page
* outdated: in the CDMx data tables, the 2nd number should read 0x35c08008, if it does not and the last 4 digits randomly change, reset the GDM links
*if the FEs complain - do in order: for GDM, CDM01, CDM02, ..., press "reset mgt" of each board, then press "unreset mgt", if it does not help, STOP HERE
* start a run
* CDM frontend will enable the trigger
* GDM frontend will enable the trigger
* LEDs on the GDM should flash, LEDs on the CDM should flash, TrigIn and TrigOut of the VX should flash
* stop a run
* GDM frontend will disable the trigger
* CDM frontend will disable the trigger

== Phase measurement ==

* pip3 install matplotlib
* pip3 install scipy
* export PYTHONPATH=$HOME/packages/midas/python
* #git clone https://github.com/J033X071C/PhaseMeasurement
* git clone https://bitbucket.org/team-ds-dm/phasemeasurement.git
* cd phasemeasurement
* python3 ./phaseMeasurement.py --help
<pre>
daq00:PhaseMeasurement$ python3 ./phaseMeasurement.py --help
usage: phaseMeasurement.py [-h] fileName numberEvents numberVX sizeEvents stopEvent minHist maxHist numberBin writeToTXT saveAsPDF

Read data from midas file (in .lz4 format) to calculate phase between the clock of VX1 and VX2

positional arguments:
fileName Name of the file we want to read data from (Example: run00389.mid.lz4)
numberEvents Number of events recorded in the file
numberVX Number of VX used in this run (usually 2...)
sizeEvents Number of points per event
stopEvent Number of events you want to go through to calculate phase
minHist Minimal value for the x axis of the phase measurement histogram (in ns)
maxHist Maximal value for the x axis of the phase measurement histogram (in ns)
numberBin Number of bins wanted for the generated histogram
writeToTXT Write argument as yes to generate text file with results of calculation
saveAsPDF Save generated plots to PDF files

optional arguments:
-h, --help show this help message and exit
daq00:PhaseMeasurement$
</pre>
* try an old file with
* python3 ./phaseMeasurement.py run00877.mid.lz4 10000 2 10000 500 -20 20 81 yes yes
<pre>
bin size = 0.494 ns
num_events = 347
mean = -1.705 ns
rms = 3.087 ns
mean_error = 0.166 ns
centroid = -1.706 ns.
width (sigma) = 0.363 ns.
error on the centroid = 0.016558 ns.
</pre>
* ls -l *.txt *.pdf
<pre>
dsdaq@dsvslice:~/online/PhaseMeasurement$ ls -l *.txt *.pdf
-rw-rw-r-- 1 dsdaq dsdaq 64728 Dec 14 16:56 run00877.mid.lz4_Plots.pdf
-rw-rw-r-- 1 dsdaq dsdaq 274 Dec 14 16:56 run00877.mid.lz4.txt
dsdaq@dsvslice:~/online/PhaseMeasurement$
</pre>
* scope settings (from email message)
<pre>
From fcote-lortie@triumf.ca Thu Dec 15 17:11:46 2022
From: Francis Cote-Lortie <fcote-lortie@triumf.ca>
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: Re: How to use scope as a waveform generator
Date: Fri, 16 Dec 2022 01:11:44 +0000

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display) by pressing on the button. It will go from 0 to 1.

The settings that we are using right now are:
Type of waveform: Sine wave
Offset: 0 V
Amplitude: 1 Vpp
Frequency: 50 kHz
Noise: 0 V
________________________________
From: Francis Cote-Lortie <fcote-lortie@triumf.ca>
Sent: Thursday, December 15, 2022 4:58 PM
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: Re: How to use scope as a waveform generator

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display)

The settings that we are using right now are:
Type of waveform: Sine wave
Offset: 0 V
Amplitude: 1 Vpp
Frequency: 50 kHz
Noise: 0 V
________________________________
From: Francis Cote-Lortie
Sent: Thursday, December 15, 2022 4:54 PM
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: How to use scope as a waveform generator

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display)
</pre>

== Standalone link test ==

<pre>
CDM: program clock chip
busybox devmem 0x80011000 32 0x8
busybox devmem 0x80011000 32 0x0
/home/dsdaq/si5394-i2c-file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b

GDM, CDM: link reset
busybox devmem 0x80011008 32 1

GDM, CDM: release reset
busybox devmem 0x80011008 32 0

CDM: link status (NOTE: SFP LOS and mod_absent are swapped!!!)
busybox devmem 0x80011010 32
0x00000024 <- fiber plugged
0x00000025 <- fiber unplugged
0x00000027 <- SFP unplugged
0x000007DC <- successful link with GDM

CDM: link state machine and data
busybox devmem 0x80011014 32
0x35C06FF6

CDM: set link to counting mode
busybox devmem 0x80012000 32 0x101
busybox devmem 0x80012000 32 0x100

CDM: time counter and error counter
root@cdm1:~# busybox devmem 0x80012000 32
0x00000100 <--- link mode
root@cdm1:~# busybox devmem 0x80012004 32
0x0000058C <--- seconds counter
root@cdm1:~# busybox devmem 0x80012004 32
0x0000058D
root@cdm1:~# busybox devmem 0x80012008 32
0x00000000 <--- error counter

GDM: no link
root@gdm0:~# busybox devmem 0x80011014 32
0x00000000
root@gdm0:~# busybox devmem 0x80011018 32
0x00000000
root@gdm0:~# busybox devmem 0x8001101c 32
0x00000000
root@gdm0:~# busybox devmem 0x80011024 32
0x00000FFF
root@gdm0:~#

GDM: good link channel 10, counting mode
root@gdm0:~# busybox devmem 0x80012000 32 0x101
root@gdm0:~# busybox devmem 0x80012000 32 0x100
root@gdm0:~# busybox devmem 0x80012008 32
0x3A8B68C2
root@gdm0:~# busybox devmem 0x80012008 32
0x42E03BEF
root@gdm0:~# busybox devmem 0x8001200c 32
0xDA090972
root@gdm0:~# busybox devmem 0x8001200c 32
0xDE6F22E9
root@gdm0:~# busybox devmem 0x80012019 32
Bus error
root@gdm0:~# busybox devmem 0x80012010 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012014 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012018 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x8001201c 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012020 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012024 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012028 32
0x00000000
root@gdm0:~# busybox devmem 0x8001202c 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012030 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012034 32
0xFFFFFFFF
root@gdm0:~#
</pre>

mapping of link channels:

<pre>
qsfp0 lane0 - 0x0100 - link 8
qsfp0 lane1 - 0x0200 - link 9
qsfp0 lane2 - 0x0400 - link 10
qsfp0 lane3 - n/c
qsfp1 lane0 - 0x0800 - link 11
qsfp1 lane1 - 0x0010 - link 4
qsfp1 lane2 - 0x0020 - link 5
qsfp1 lane3 - n/c
qsfp2 lane0 - 0x0040 - link 6
qsfp2 lane1 - 0x0080 - link 7
qsfp2 lane2 - 0x0001 - link 0
qsfp2 lane3 - n/c
qsfp3 lane0 - 0x0002 - link 1
qsfp3 lane1 - 0x0004 - link 2
qsfp3 lane2 - 0x0008 - link 3
qsfp3 lane3 - n/c
</pre>

script to start the test with 2 CDMs:

<pre>
ssh dsdaq@dsvslice
ssh root@gdm0 busybox devmem 0x80011008 32 1
ssh root@cdm0 busybox devmem 0x80011008 32 1
ssh root@cdm1 busybox devmem 0x80011008 32 1
ssh root@gdm0 busybox devmem 0x80011008 32 0
ssh root@cdm1 busybox devmem 0x80011008 32 0
ssh root@cdm0 busybox devmem 0x80011008 32 0
ssh root@gdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
ssh root@cdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
ssh root@cdm1 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
</pre>

== LEMO trigger GDM to CDM to VX ==

on the GDM:

<pre>
/home/dsdaq/si5394-i2c-file /home/dsdaq/GDM_v1.0_IN0_EXT1_and_IN1_fixed_Si5394-RevA-Registers.txt 0 0x6b
busybox devmem 0x80011008 32 0x1
busybox devmem 0x80011008 32 0x0
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x80001230 # QSFP TX fixed pattern
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 2 0xff # enable LEMO inputs
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 3 0x7654 # enable LED, one per LEMO input
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x0F0F # enable LEMO to trg_in and tsm_in
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x40001230 # enable trg_in and tsm_in output to QSFP
</pre>

on the CDM:

<pre>
/home/dsdaq/si5394-i2c-file /home/dsdaq/CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
/home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp
busybox devmem 0x80011008 32 1
busybox devmem 0x80011008 32 0
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x8040 # trg_in from sfp[0], tsm_in form sfp[1]
/home/dsdaq/online/ds-dm-software/test_cdm.exe 11 # trg_in counter
/home/dsdaq/online/ds-dm-software/test_cdm.exe 12 # tsm_in counter
</pre>

== GPS receiver VCL-2705 ==

* Valiant VCL-2705 GPS receiver
* https://www.valiantcom.com/time-distribution/gps-receiver-irig-b.html
* USB connection is /dev/ttyACM0, 115200 bps, "GNSSAUX" prompt
* minicom -D /dev/ttyACM0 -b 115200
* user manual download instructoins - see sheet of paper with user name and password in the shipping box
* usb commands:
<pre>
gnss-help
gnss-showver -> F/W: Ver 1.7 Feb 17 2020 16:20:43

gnss-showselftest -> no antenna connected
Overall : FAIL
EPROM Test: PASS
Antenna : NOT DETECTED
GNSS : COMMUNICATION OK

gnss-showsettings
GNSS NMEA BAUDRATE :115200
GNSS ANTENNA LENGTH :30 meters
GNSS USER CONFIGURED DELAY:-65 nanoseconds
GNSS 1PPS PULSE WIDTH :200 milliseconds
GNSS MODE :GPS
GNSS STATUS :STATIONARY

GNSSAUX> gnss-showserial
SERIAL :2704H01V17MAX310

GNSSAUX> gnss-showmode
GNSS MODE : GPS

GNSSAUX> gnss-showstatus
GNSS STATUS: STATIONARY

GNSSAUX> gnss-showalarms
CURRENT ALARMS GNSS
ANTENNA : NOT DETECTED

GNSSAUX> gnss-showerrors
CURRENT ALARMS GNSS
ANTENNA : **NOT DETECTED
ERROR STATISTICS GNSS
RMC GOOD DURATION : Secs 0
RMC BAD DURATION : Secs 0
LOCK GOOD SECS DURATION : Secs 0
LOCK BAD SECS DURATION : Secs 0
SATINFO GOOD ITERATIONS :0
SATINFO REJECT ITERATIONS :0
SATINFO NOTALKER ITERATIONS :0
CURRENT MONITOR STATE GNSS
STATE :Phase-1 HUNTING ANTENNA DETECT

GNSSAUX> gnss-showsatinfo
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-showsats
Total Sats: 0

GNSSAUX> gnss-showmyloc
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-show1ppsstate
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-showjamstatus
Not Available !

GNSSAUX> gnss-showspoofstatus
Not Available !

--- antenna connected, can see the sky ---

GNSSAUX> gnss-showselftest
Overall : PASS
EPROM Test: PASS
Antenna : DETECTED
GNSS : COMMUNICATION OK

GNSSAUX> gnss-showalarms
CURRENT ALARMS GNSS
ANTENNA : DETECTED
GNSS LOCK : AVAILABLE

GNSSAUX> gnss-showerrors
CURRENT ALARMS GNSS
ANTENNA : DETECTED
GNSS LOCK : AVAILABLE
ERROR STATISTICS GNSS
RMC GOOD DURATION : Mins 1,Secs 34
RMC BAD DURATION : Secs 55
LOCK GOOD SECS DURATION : Mins 1,Secs 34
LOCK BAD SECS DURATION : Secs 55
SATINFO GOOD ITERATIONS :3
SATINFO REJECT ITERATIONS :0
SATINFO NOTALKER ITERATIONS :0
CURRENT MONITOR STATE GNSS
STATE :Phase-4 NORMAL OPERATION, Monitoring GNSSLOCK

GNSSAUX> gnss-showsatinfo
GNSS MODE :GPS
SATELLITE INFORMATION TALKER:GPS
NMEA ID : 01-32
NO OF SATELLITES IN VIEW: 08
NO OF XXGSV MSGS : 03
SatNo PRN NO (SV ID) ELEVATION (degs) AZIMUTH (degs) C/No (SNR)
1 0 0 0 0
2 0 0 0 0
3 0 0 0 0
4 0 0 0 0
5 0 0 0 0
6 0 0 0 0
7 0 0 0 0
8 0 0 0 0
<<<<<< End of Sat Info >>>>>>

GNSSAUX> gnss-showsats
Total Sats: 8

GNSSAUX> gnss-showmyloc
GNSS RECEIVER LOCATION:
Latitude : 4914.81911
Longitude: 12313.69595

GNSSAUX> gnss-show1ppsstate
GPS 1PPS STATE: LOCKED

GNSSAUX> gnss-showmyloc

GNSS RECEIVER LOCATION:
Latitude : 4914.80688
Longitude: 12313.69531

enter into google maps search box as: 49 14.80688, -123 13.69531, observe the space, the moved dot and the minus.

GNSSAUX> gnss-resetgnss

Executing....Please Wait....
$$$$$END
</pre>

* 1PPS BNC output: period 1 sec, pulse width 200 ms, 3.3V into 1MOhm, 1.38V into 50Ohm.
* IRIG-B BNC output: 5.6V into 1MOhm, 2.4V into 50Ohm.
* IRIG-B format selector: default is all up.

== Rb clock PRS10 ==

* https://www.thinksrs.com/products/prs10.html
* 10 MHz output is sine wave around 5V peak to peak
* 1PPS BNC output is 10 usec pulse, 5V into 1MHohm.
* RS232 connection: minicom -D /dev/ttyUSB0 -b 9600
* ser2net config: localhost,3001:raw:600:usb-5-2-1.0:9600 -XONXOFF -RTSCTS LOCAL
* ssh daq13, cd /home/olchansk/git/ser2net, ./ser2net -c ~/daq/ds/ser2net.conf -d
* ssh daq13, cd ~/daq/ds, python3 prs10.py
* RS232 commands:
<pre>
ID?
PRS10_3.56_SN_105719
VB1
SN?
RS1 -- reset
ST? -- status
FC? -- 10MHz OCXO drive voltage DAC settings
DS? -- "detected signals"
GA? -- gain of frequency lock loop between ovenized oscillator and Rb cell, 0=use ovenized oscillator only
MO? -- magnetic offset of the Rb cell, range 2300..3600, if out of range, unit must be set to different operating mode, see prs10m.pdf
MR? -- magnetic read
TT? -- time-tag, time in ns between 1PPS out and 1PPS in
TS? -- time slope, ???
TO? -- time offset, ???
PS? -- pulse slope, ???
PL? -- 0=phase lock off, 1=phase lock on, lock to 1PPS input
PT? -- phase lock integrator time constant, PT8 is integrator time constant 18.2 hours, natural time constant 2.25 hours
PF? -- phase lock stability factor, PF2 is "1"
PI? -- phase lock integrator

Analog to digital 12 bit ADC, values 0.000 to 4.998

AD0? -- Spare (J204)
AD1? -- +24V(heater supply) divided by 10.
AD2? -- +24V(electronics supply) divided by 10
AD3? -- Drain voltage to lamp FET divided by 10
AD4? -- Gate voltage to lamp FET divided by 10
AD5? -- Crystal heater control voltage
AD6? -- Resonance cell heater control voltage
AD7? -- Discharge lamp heater control voltage
AD8? -- Amplified ac photosignal
AD9? -- Photocell’s I/V converter voltage divided by 4
AD10? -- Case temperature (10 mV/°C)
AD11? -- Crystal thermistors
AD12? -- Cell thermistors
AD13? -- Lamp thermistors
AD14? -- Frequency calibration pot / external calibration voltage
AD15? -- Analog ground

A/D via CPU E-port:

AD16? -- Varactor voltage for 22.48 MHz VCXO (inside RF synthesizer) / 4
AD17? -- Varactor voltage for 360 MHz VCO (output of RF synthesizer) / 4
AD18? -- Gain control voltage for amplifier which drives frequency multiplier / 4
AD19? -- RF synthesizer’s lock indicator voltage (nominally 4.8 V when locked )

ST?

ST1 : Power supplies and Discharge Lamp
ST1 bit, Condition which sets bit, Corrective Action
0 -- +24 for electronic < +22 Vdc
1 -- +24 for electronics > +30 Vdc
2 -- +24 for heaters <+22 Vdc
3 -- +24 for heaters > +30 Vdc
4 -- Lamp light level too low
5 -- Lamp light level too high
6 -- Gate voltage too low
7 -- Gate voltage too high

ST2: RF Synthesizer
ST2 bit, Condition which sets bit, Corrective Action
0 -- RF synthesizer PLL unlocked
1 -- RF crystal varactor too low
2 -- RF crystal varactor too high
3 -- RF VCO control too low
4 -- RF VCO control too high
5 -- RF AGC control too low
6 -- RF AGC control too high
7 -- Bad PLL parameter

ST3: Temperature Controllers
ST3 bit, Condition which sets bit
0 -- Lamp temp below set point
1 -- Lamp temp above set point
2 -- Crystal temp below set point
3 -- Crystal temp above set point
4 -- Cell temp below set point
5 -- Cell temp above set point
6 -- Case temperature too low
7 -- Case temperature too high

ST4: Frequency Lock-Loop Control
ST4 bit, Condition which sets bit
0 -- Frequency lock control is off
1 -- Frequency lock is disabled
2 -- 10 MHz EFC is too high
3 -- 10 MHz EFC is too low
4 -- Analog cal voltage > 4.9 V
5 -- Analog cal voltage < 0.1
6 -- not used
7 -- not used

ST5: Frequency Lock to External 1pps
ST5 bit, Condition which sets bit
0 -- PLL disabled
1 -- < 256 good 1pps inputs
2 -- PLL active
3 -- > 256 bad 1pps inputs
4 -- Excessive time interval
5 -- PLL restarted
6 -- f control saturated
7 -- No 1pps input

ST6: System Level Events
ST6 bit and Condition which sets bit
0 Lamp restart
1 Watchdog time-out and reset
2 Bad interrupt vector
3 EEPROM write failure
4 EEPROM data corruption
5 Bad command syntax
6 Bad command parameter
7 Unit has been reset
</pre>
* ST? on warm start
<pre>
received: PRS_10
received: 255,255,255,243,34,255
received: 0,0,0,1,34,0
...
received: 0,0,0,1,34,0
received: 0,0,0,0,2,0
...
received: 0,0,0,0,2,0
received: 0,0,0,0,4,0
</pre>
* ST? on loss of external 1PPS
<pre>
...
received: 0,0,0,0,4,0
disconnect 1PPS input
received: 0,0,0,0,132,0
...
reconnect 1PPS input
received: 0,0,0,0,132,0
received: 0,0,0,0,4,0
...
</pre>
* ST? on coldish start
<pre>
daq13:ds$ python3 prs10.py
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,0,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 0,0,1,1,34,0 ] old [ 80,0,0,1,34,1 ] counter: 0
received [ 0,0,0,1,34,0 ] old [ 0,0,1,1,34,0 ] counter: 0
received [ 0,0,16,1,34,0 ] old [ 0,0,0,1,34,0 ] counter: 6
received [ 0,0,20,1,34,0 ] old [ 0,0,16,1,34,0 ] counter: 5
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 8
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 3
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 49
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 250
...
</pre>
* ST? on cold start, note: no bump in the 10MHz clock as reported by DS-DM clock chip
<pre>
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,21,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 64,0,21,1,34,0 ] old [ 80,0,21,1,34,1 ] counter: 0
received [ 0,0,21,1,34,0 ] old [ 64,0,21,1,34,0 ] counter: 0
received [ 0,0,20,1,34,0 ] old [ 0,0,21,1,34,0 ] counter: 158
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 23
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 47
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 40
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 249
</pre>

= det fac integration test =

Connections:
* GPS receiver "IRIG-B SEL" both switches "up" - both "on", IRIG-B format B004
* GPS receiver USB-B -> long cable -> daq13 USB-A
* GPS receiver "1PPS out" -> long BNC cable -> BNC-T -> scope (5V, no 50ohm) and Rb clock BNC "1PPS in"
* GPS receiver "IRIG-B 50ohms" -> long BNC cable -> BNC-T -> scope (5V, no 50ohm) and DS-DM LEMO input 1 (TTL mode)
* Rb clock RS232 -> RS232 straight cable -> RS232-to-USB adapter -> daq13 USB-A
* Rb clock "1PPS out" BNC -> scope (5V, no 50ohm, trig threshold rising edge 2V)
* Rb clock "10MHz output 50 Ohm" BNC -> lemo -> lemo-T -> scope (sine wave, 5V, no 50ohm) and DS-DM clock input.

Programs to run:
* on daq13: cd /home/olchansk/git/ser2net, ./ser2net -c ~/daq/ds/ser2net.conf -d
<pre>
localhost,3001:raw:600:usb-5-2-1.0:9600 -XONXOFF -RTSCTS LOCAL
</pre>
* on daq13: cd /home/olchansk/daq/ds, python3 prs10.py ### connects to ser2net
<pre>
daq13:ds$ python3 prs10.py
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,21,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 64,0,21,1,34,0 ] old [ 80,0,21,1,34,1 ] counter: 0
received [ 0,0,21,1,34,0 ] old [ 64,0,21,1,34,0 ] counter: 0
received [ 0,0,20,1,34,0 ] old [ 0,0,21,1,34,0 ] counter: 158
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 23
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 47
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 40
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 249
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 12096
received [ 0,0,0,0,20,0 ] old [ 0,0,0,0,132,0 ] counter: 2
received [ 0,0,0,0,148,0 ] old [ 0,0,0,0,20,0 ] counter: 0
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,148,0 ] counter: 575
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 11449
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 2755
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 34386
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 41035
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 113401
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 33375
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 54767
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 85059
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 33222
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 119234
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 121990
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 128184
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 56002
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 428237
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 8250
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 1388
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 30506
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 142704
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 179451
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 106182
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 68747
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 65424
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 157587
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 6932
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 1388
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 20255
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 4
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 225941
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 72183
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 26970
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 1
39287
</pre>
* on ds-dm: ./test_cdm.exe --irigb ### note sbs mismatch is because I should wrap around at 16 bits
<pre>
dataframe: S00000000S100001010S...S001010101S000001000S, sec: 00, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16724 should be 82260
dataframe: S10000000S100001010S...S101010101S000001000S, sec: 01, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16725 should be 82261
dataframe: S01000000S100001010S...S011010101S000001000S, sec: 02, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16726 should be 82262
dataframe: S11000000S100001010S...S111010101S000001000S, sec: 03, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16727 should be 82263
dataframe: S00100000S100001010S...S000110101S000001000S, sec: 04, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16728 should be 82264
dataframe: S10100000S100001010S...S100110101S000001000S, sec: 05, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16729 should be 82265
</pre>

= DS-IOGC GPS interface board =

* Rev0 git repository: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0/-/tree/main?ref_type=heads
* Rev0 schematics: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0/-/blob/main/Altium/Project%20Outputs%20for%20DS-IOGC-Rev0/SCH-DS-IOGC-Rev0.pdf?ref_type=heads
* Rev0 schematics: [[:Image:SCH-DS-IOGC-Rev0.pdf|SCH-DS-IOGC-Rev0]]
* Rev1 git repository: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev1/
* Rev1 schematics: [[:Image:SCH-DS-IOGC-Rev1.pdf|SCH-DS-IOGC-Rev1]]

== Changes Rev0 to Rev1 ==

<pre>
From: Peter Margetak <pmargetak@triumf.ca>
Subject: IOGC REV1 review
Date: Wed, 4 Sep 2024 07:31:19 +0000

Hi Konstantin,
Pls have a look at SCH for new rev. I'd like to send it to mfr next week so if you can comment by early next week. Meanwhile I work on layout and other stuff.

Changes:

New ICs - all powered +5V
U20 - inverters for RUclk RX/TX
U21 - non inverting line driver for RU-1pps-out (so you don't have to route if via GDM to see it on scope)
U22 - non inverting buffer for ext 1pps input

All Lemo connectors have the same position but they are double lemos now => new panel needed
@Marek Walczak<mailto:mwalczak@triumf.ca> you can print it ahead once pcb is done + update IOGC docs and panel description

J2A/B - Test ports for RU-1pps in and out
J5A/B - inputs for external GPS data and External source of 1pps
J6A/B - aux in/out for GDM

SW1 - no change - select RX/TX USB/GDM
SW2 - select latch sensitivity for rising/falling edge
SW3 - select source of GPS data (opto or ext) AND select source of 1pps input (latch or ext)

p.
</pre>

== PRS-10 Rb clock device ==

The Rb clock PRS-10 device provides these connections:
<pre>
RS232 RX input - serial communication, non-standard RS232
RS232 TX output - serial communication, non-standard RS232
10 MHz clock output - coax 50 Ohm high resolution 10 MHz clock
1pps output - 1 Hz clock corresponding to the 10 MHz clock
1pps input - 1pps signal from GPS receiver
</pre>

Mode of operation:
* 10 MHz clock is always running
* 1pps output is always running
* if 1pps input from GPS received is present, after 256 pulses PRS-10 will sync it's 1pps output with the 1pps input by adjusting the frequency of the 10 MHz clock
* when unlocked: 1pps output and 1pps input unrelated
* when locked to GPS: 1pps output and 1pps input always go up and down at the same time

Theory of operation:
* 10 MHz clock is produced by a high-quality crystal (stable on the scale of seconds)
* crystal oscillator is synchronized to a Rb cell (stable on the scale of hours and days)
* Rb cell resonant frequency is synchronized to the GPS 1pps signal (stable on the scale of months and years)

== Rev1 connections ==

* LEMO connectors (front panel)
<pre>
LEMO J2A output - Rb clock 1pps in monitor
LEMO J2B output - Rb clock 1pps out monitor
LEMO J5A input - GPS IRIG-B from GPS receiver to FPGA (VCL-2705)
LEMO J5B input - GPS 1pps from GPS receiver to PRS-10 (VCL-2705)
LEMO J6A output - AUX-OUT from FPGA (dual-LEMO PCB-side)
LEMO J6B input - AUX-IN to FPGA (dual-LEMO away-from -PCB)
</pre>

* SMB connectors (back)
<pre>
SMB J3 output - GPS 1pps loopback to LNGS
SMB J4 input - LNGS GPS data input
</pre>

* LEDs
<pre>
D1 - same as LEMO J2A out (Rb clock 1pps in)
D2 - same as SMB J3 out (GPS 1pps from LNGS or from a GPS receiver)
D5 - controlled by FPGA-OUT-LED1
D6 - controlled by FPGA-OUT-LED2
D7 - PRS-10 24V power ok
</pre>

* switches
<pre>
SW1A and SW1B - route PRS-10 RS232 to USB or to FPGA
SW2A - route PRS-10 1pps input from SMB J4 (LNGS) or from LEMO J5B (GPS receiver 1pps)
SW2B - route FPGA-IN-GPSDATA input from SMB J4 (LNGS) or from LEMO J5A (GPS receiver IRIG-B data)
SW3 - LNGS 1pps from rising edge or from falling edge of SMB J4 (LNGS)
</pre>

== Rb clock cable ==
<pre>
Rb clock ----- DS-IOGC side, pin numbers are as labeled on the cable connectors

1 - 1pps out - 8 - 1pps out --- correct
2 - nc
3 - nc
4 - TXD - 3 - RU-DATA-OUT - USB-RX input --- correct
5 - 1pps in - 2 - 1pps in --- correct
6 - +24V - 10 - +24V --- should by pin 1 to use both +24V pins?
7 - RXD - 15 - RU-DATA-IN - USB-TX output --- correct
8 - nc
9 - +24V - 10 - +24V --- correct
10 - GND - 9 - GND --- correct
</pre>

== VX connections ==
<pre>
VXA_TX0 - FPGA-OUT-LED2 - D6 LED ("10 MHz clock")
VXA_TX1 - FPGA-OUT-LED1 - D5 LED ("GPS DATA")
VXA_TX2 - FPGA-OUT-RU1PPS-EN - enable 1pps to Rb clock
VXA_TX3 - not used (62.5 Hz clock)

VXB_TX0 - FPGA-TX - PRS-10 RS-232 out
VXB_TX1 - FPGA-OUT-AUX - 5V TTL J5 LEMO out
VXB_TX2 - FPGA-OUT-OPTO1PPS-EN - PRS-10 power enable
VXB_TX3 - not used (62.5 MHz clock)

VXA_RX0 - FPGA-IN-OPTO1PPS - 1pps from GPS
VXA_RX1 - FPGA-IN-GPSDATA - J4 GPS data
VXA_RX2 - n/c
VXA_RX3 - n/c

VXB_RX0 - n/c
VXB_RX1 - FPGA-IN-AUX - J5 LEMO TTL input
VXB_RX2 - FPGA-RX - PRS-10 RS-232 in
VXB_RX3 - FPGA-IN_RU1PPS - PRS-10 1pps output
</pre>

== test sequence ==

* ./test_cdm_local.exe --writereg 7 0 ### clear reg 7
* ./test_cdm_local.exe --writereg 8 0 ### set vx_tx mux to vx_tx control from reg 7
* connect blue cable to GDM port 6 (next to the ethernet connector)
* ./test_cdm_local.exe --writereg 7 0x4000 ### power up
* test LEDs:
* ./test_cdm_local.exe --writereg 7 0x4100 ### right LED2 D6
* ./test_cdm_local.exe --writereg 7 0x4200 ### left LED1 D5
* test AUX-IN and AUX-OUT:
* ./test_cdm_local.exe --writereg 7 0x6000 ### J6A LEMO (PCB-side) measure +5VDC
* install LEMO jumper between LEMO J6A and J6B
* write 0x4000 then ./test_cdm_local.exe 6 ### reads 0x5C, bit 5 0x20 reads 0
* write 0x6000 then ./test_cdm_local.exe 6 ### reads 0x7C, bit 5 0x20 reads 1
* write 0x4000 to clear all bits
* test the 10 MHz clock:
* init the c.c: ./test_cdm_local.exe --load-cc
* connect 10 MHz clock output to DS-DM LEMO J6-LEFT (CLK_EXT1)
* 10 MHz output ok - IN0 is good: ./test_cdm_local.exe --cc
<pre>
Clock chip state 1, status: OOF_IN2 IN1 IN_SEL_1 HOLD_HIST_VALID
</pre>
* without 10 MHz reports IN0 LOS (no signal) and OOF (wrong frequency)
<pre>
Clock chip state 1, status: LOS_IN0 OOF_IN0 OOF_IN2 IN1 IN_SEL_1 HOLD_HIST_VALID
</pre>
* test GPS IRIG-B data:
* connect GPS IRIG-B signal to J6B (dual-LEMO away from PCB)
* IRIG-B via AUX-IN is ok: ./test_cdm_local.exe --irigb
<pre>
dataframe: S00100110S111001000S...S011110000S010000000S, sec: 34, min: 17, hrs: 00, day of year: 223, year: 2024, date: 10 aug, sbs: 1054 should be 1054
</pre>
* test IRIG-B via J5A GPS-EXT:
* connect GPS IRIG-B signal to J5A (dual-LEMO next to PCB)
* set SW2B: switch on the side of SMB connectors, slide direction of 24V power supply connector
* ./test_cdm_local.exe --irigb built for VXA_RX1 should work
* test GPS 1pps signal:
* connect GPS 1pps to J5B (dual-LEMO away from PCB)
* nothing happens, no output from U22
* remove R51
* set SW2A: switch on the side away from SMB connectors, slide direction of 24V power supply connector
* observe LEDs D1 and D2 blink at 1Hz
* SKIP THIS connect GPS 1pps to J4 using LEMO-SMB adapter
* SKIP THIS nothing happens, nLE is always 3.2V, both settings of SW2
* SKIP THIS !!!failed here!!!
* ./test_cdm_local.exe --writereg 8 9 ### set vx_tx mux to GPS control, hard enables PRS-10 power and PRS-10 1pps in
* observe 1pps counters in reg 68 count at 1 Hz: rb_1pps 0x51->0x52, gps_1pps 0xd5->oxd6
<pre>
root@dsdm:~# ./test_cdm_local.exe 68
ds20k_reg[68] is 0x0051d520 (5362976)
ds20k_reg[68] is 0x0052d624 (5428772)
...
</pre>
* observe PRS-10 can see the 1pps signal "130" changes to "2" after 243 seconds to "4"
* observe period of GPS and PRS-10 1pps is identical (plus/minus 1 clock)
<pre>
root@dsdm:~# ./test_cdm_local.exe 13 14
ds20k_reg[13] is 0x077356d4 (124999380)
ds20k_reg[14] is 0x077356d4 (124999380)
</pre>
* observe GDM/CDM clocks are exactly 125 MHz, we are running on DS-DM internal oscillator
<pre>
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735943 (125000003) should be ~125 MHz
0x1034 rx_clk: 0x07735943 (125000003) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735943 (125000003) should be ~125 MHz
0x103C tx_clk: 0x07735943 (125000003) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>
* if tx_clk is missing, reset the mgt: ./test_cdm_local.exe --reset-mgt
* switch CC to external clock:
<pre>
root@dsdm:~# ./test_cdm_local.exe --cc-in0
CC use clock input 0: 10 MHz LEMO external clock
root@dsdm:~# ./test_cdm_local.exe --cc
Polling CC status...
Clock chip state 1, status: IN0 IN_SEL_REGCTRL IN_SEL_0 HOLD_HIST_VALID
</pre>
* observe rx_clk and tx_clk are now slightly off: they run from PRS-10 10 MHz clock and are measured against the DS-DM internal oscillator.
<pre>
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x94b12519, ds20k version 0x20240814
DS-DM firmware build 0x94b12519, ds20k version 0x20240814
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x077358e1 (124999905) should be ~125 MHz
0x1034 rx_clk: 0x07735b49 (125000521) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735b49 (125000521) should be ~125 MHz
0x103C tx_clk: 0x07735b49 (125000521) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>
* observe GPS and PRS-10 1pps period is now exactly 125*10^6 of 8ns clocks (equal to 1 second)
<pre>
root@dsdm:~# ./test_cdm_local.exe 13 14
ds20k_reg[13] is 0x0773593f (124999999)
ds20k_reg[14] is 0x0773593f (124999999)
</pre>
* look at them repeatedly, observe reg 13 "GPS 1pps period" has some wobble, reg 14 "Rb clock 1pps period" is steady. this is as expected: if GPS 1pps was steady, we do not need to Rb clock. PRS-10 1pps is derived from the PRS-10 10 MHz clock and is measured against the 125 MHz clock derived from the same 10 MHz clock in the GDM FPGA.

Rev0 Test status:

* GPS 1pps to SMB-in ok (LED flashes)
* GPS 1pps to FPGA ok
* GPS data to FPGA ok
* GPS 1pps to PRS-10 enabled from FPGA ok
* PRS-10 Rb clock 1pps out to FPGA ok
* PRS-10 Rb clock 1pps out to FPGA to LEMO AUX out ok
* can see GPS 1pps, IRIG-B, PRS-10 1pps out, 10 MHz on the scope, ok
* PRS-10 syncs on leading edge (0->1) of GPS 1pps signal, ok
* reg 13 and 14 1pps periods are identical, ok
* CC locks on PRS-10 10 MHz clock, DS-DM runs on PRS-10 clock, ok
* NOT TESTED - smb output
* NOT TESTED - optical converter fiber to SMB
* NOT TESTED - optical converter SMB to fiber
* NOT TESTED - SMB loopback
* NOT TESTED - fiber loopback

Rev1 Test status: (21 Nov 2024)

* power up ok
* LEDs ok
* LEMO AUX-IN, AUX-out ok
* GPS IRIG-B to FPGA via J6B AUX-IN ok
* GPS IRIG-B to FPGA via J5A GPS-EXT ok
* PRS-10 Rb clock 1pps out to FPGA ok
* PRS-10 Rb clock 10 MHz to C.C. ok
* PRS-10 RS-232 ok
* J5B 1pps input fail. input signal is 3.3V without termination, 1.5V when plugged into IOGC (on R51), no output from U22.

= VX busy logic =

= DS-20K DAQ =

== Overview ==

DS-DM, GDM and CDM are key parts of the DS-20K DAQ system:
* common clock distribution from external clock (atomic clock, GPS) to GDM to per-quadrant CDMs to VX digitizers
* common trigger distribution from GDM internal algorithm or external input to all VX digitizers
* run control: GDM, CDM, VX all start recording data at the same time (clock and timestamp reset)
* collection of trigger data from VX digitizers to per-quadrant CDMs to GDM

== Deliverables ==

* hardware and firmware for GDM to CDM clock distribution
* hardware and firmware for CDM to VX clock distribution
* hardware and firmware for GDM external clock input (atomic clock or GPS)
* hardware and firmware for CDM and VX serial communications (VX LVDS I/O connector)
* firmware for run control (timestamp reset and sync): GDM to CDM to VX
* firmware for common trigger distribution: GDM to CDM to VX
* firmware for trigger data flow: VX to CDM to GDM
* firmware for busy control: VX to CDM to GDM back to CDM to VX
* firmware for flow control: FEP to GDM MIDAS frontend to GDM to CDM to VX
* GDM MIDAS frontend: clock selector and monitoring, trigger and run control, busy and flow control, GDM housekeeping
* CDM MIDAS frontend: clock monitoring, CDM housekeeping

specific performance:
* GDM external clock: 10 MHz GPS clock
* GDM to CDM fiber link:
** clock XXX MHz
** link data rate: XXX Gbit/sec
** CDM recovered clock: XXX MHz
** CDM recovered clock jitter: XXX ns
** phase alignment between CDMs: XXX ns
** phase alignment between CDMs persists across reboots, power cycles, firmware updates
** phase alignment between CDMs should be easy to measure
** phase alignment between CDMs should be easy to recalibrate if hardware parts are replaced (DS-DM boards, fiber transceivers, fiber cables, etc)
** data packet bandwidth: XXX Mbytes/sec
** data packet latency: XXX clocks
** data packet skew between CDMs: XXX clocks
* CDM to VX clock:
** clock: XXX MHz
** jitter, all CDM clock outputs: XXX MHz
** phase alignment between all CDM clock outputs: XXX ns
* CDM to VX trigger:
** TBD (use the VX "sync" input or VX LVDS I/O line or VX serial link packet)
* CDM to VX serial link:
** clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
** bit rate: XXX bits/sec
** latency: XXX link clocks
** maximum skew between VXes: XXX ns
* VX to CDM serial link:
** clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
** bit rate: XXX bits/sec
** latency: XXX link clocks
** maximum skew between VXes: XXX ns
* timestamp reset:
** maximum skew between VXes: XXX ns
* busy round trip time: XXX ns (VX to CDM to GDM back to CDM to VX)
* flow control latency: XXX ns (FEP software to GDM MIDAS frontend to GDM to CDM to VX)

== Technical risk items ==

this refers to unexpected behaviour and performance of
system components, causes big difficulty in implementing the system,
prevents delivery of deliverables, and prevents or negatively affects operation
of the DS-20K DAQ or of the whole experiment.

(14-sep-2022, list is not sorted by any criterial: severity, probability, ease of investigation)

(stability of course is long term stability, across hours, days, weeks, months, years)

* stability of Enclustra FPGA modules (crashes/year, failures to boot/year, flash corruption/year)
* stability of GDM external clock PLL (lock loss/year)
* stability of CDM recovered clock (lock loss/year, unexpected phase drifts, etc)
* unexpected failures or bit error rates in GDM-CDM fiber links
* stability of CDM VX clock outputs (stability of clock cleaner chip)
* stability of VX internal clock distribution (VX PLL lock loss events)
* stability of VX CAEN base firmware (different versions of CAEN base firmware have different clock distribution behaviour)
* strange things in CAEN base firmware (unexpected clocking of LVDS I/O, unexpected phase shifts between clocks, etc)
* DS-DM and VX hardware problems (incompatible LVSD I/O, incompatible clock signals, etc)

== Milestones ==

(14-sep-2022: at this stage of the project, priority must be given to identifying and retiring (so called) technical risk factors. it is not good to build the complete system only to discover that (for example) some Enclustra FPGA modules require 5 attempts to boot and erase their flash memory contents once a month. Both example are real-life actual problems that caused big difficulties in GRIFFIN/TIGRESS and ALPHA-g experiments).

Development and testing milestones in time reversed order:

* full DAQ data challenge: all VXes, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
* one quadrant data challenge: 1 VME crate of VX, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
* vertical slice data challenge: 1 VME crate, 2 VX, 2 CDM (1 VX per CDM), GDM, etc operate as designed
* GDM-CDM link finalized (data rate frozen, data packet format frozen, data content permitted to change)
* CDM-VX serial link finalized (data rate frozen, data packet format frozen, data content permitted to change)
* run control (timestamp reset) and trigger distribution design agreed upon, frozen (list of possible triggers permitted to change)
* VX to CDM to GDM data flow design agreed upon, frozen (data contents permitted to change)
* major technical risk items retired (all hardware and firmware is working as expecred without mysteries and surprises, all problems are identified, investigated, resolved, solutions tested)
* stable operation of CDM-VX serial links in vertical slice system
* stable operation of GDM to CDM clock in vertical slice system
* stable operation of CDM to VX clock in vertical slice system
* vertical slice system assembled (1 VME crate, 2 VX, 2 CDM, 1 GDM, network, FEP, TSP, MIDAS)

= ZZZ =

ZZZ

DS-DM

2026-04-01T17:42:58Z

Olchansk: /* DS20k block register map */

= DS-DM =

DarkSide-20k Global and Crate Data Manager board (GDM and CDM).

Global Data Manager (GDM):
* clock distribution to CDM boards (including GPS/atomic clock source)
* collection of trigger data from CDM boards, processing and distribution of trigger decision to CDM boards
* run control
* integration with GPS 10MHz and 1pps clocks and GPS/IRIG date and time information

Crate Data Manager (CDM):
* clock distribution from GDM to CAEN VX digitizers
* receive trigger data from CAEN VX digitizers
* send trigger data to GDM
* run control and dead time control

= Links =

* https://dsdaqgw.triumf.ca/vslice/ - vertical slice DAQ at TRIUMF
* https://daq00.triumf.ca/DaqWiki/index.php/DarkSide - this DS-DAQ Wiki page
* https://dsvslice.triumf.ca/vx_napoli - DS vertical slice at TRIUMF
* https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0 - DS-DM board Rev0
* https://edev-group.triumf.ca/hw/exp/dark-side-20k/cdm-gdm-data-manager-vme-cards/rev1 - DS-DM board Rev1
* https://docs.xilinx.com/v/u/en-US/ds593 - Xilinx Platform Cable USB II
* https://www.enclustra.com/en/products/system-on-chip-modules/mercury-xu8/ - Enclustra
* https://bitbucket.org/team-ds-dm/phasemeasurement - VX Phase measurement script
* https://daq00.triumf.ca/daqinv/frontend/list/178 - inventory database
* https://daq00.triumf.ca/elog-ds/DS-DAQ - DS-DAQ elog
* https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0 - GPS and Rb Clock adapter board

= Git repos =

* dsdmdev@dsdaqgw.triumf.ca:git/gcdm.git - git repository, DS-DM firmware, branch develop_ko2, contact K.O. for ssh key access
* https://edev-group.triumf.ca/fw/exp/darkside/gcdm - git repository, DS-DM firmware (broken, do not use)
* https://bitbucket.org/team-ds-dm/ds-dm-software - git repository, DS-DM MIDAS frontend
* https://bitbucket.org/ttriumfdaq/dsproto_vx2740/src/develop/ - VX DSFE MIDAS frontend
* https://openproject.triumf.ca/svn/darkside-20k/ - VX firmware project

= Onboard hardware =

* jtag chain: arm_dap_0 0x5BA00477, xczu4_1 0x04721093
* Eclustra Mercury+ XU8 module: ME-XU8-4CG-1E-D11E-R2.1
** Xilinx® Zynq Ultrascale+™ MPSoC XCZU4CG-1FBVB900E
** DDR4 ECC SDRAM (PS) 2 GB
** DDR4 SDRAM (PL) 1GB
* ethernet mac chip: AT24MAC402-SSHM-T ("602" chip is wrong)
* USB UART for Enclustra serial console, micro-USB, 115200n8
* clock chip: SI5394A-A-GM and oscillator CS-044-054.0M (54 MHz)
* U23 3.3V current meter and thermometer, LTC2990IMS#TRPBF
* LEDs:
** LED_FP A/B/C/D 0/1/2/3
** led1 - 3V3_SW_ON, SOM_POWER_GOOD - Enclustra FPGA module 3.3V power is good
** led2 - LTM4624 PGOOD
** led3 - FPGA_DONE - FPGA has booted
** led4 - TP-S-1, PCLK_P
** led5 - TP-S-2, PCLK_N
* LEMO connectors (top to bottom)
** J4 - input (NIM/TTL) (EXT_IN_LV(1), EXT_IN_LV(2))
** J5 - input (NIM/TTL) (EXT_IN_LV(3), EXT_IN_LV(4))
** J6 - external clock (GPS 10MHz and PPS)
** J7 - output (NIM/TTL) (EXT_OUT(1), EXT_OUT(2))
* SMA connectors
** J9, J10 - CLK_CCA from U6 C.C.
** J11, J12 - CLK_TP0
* RJ45 ethernet connector (100mbit: green light on, yellow light flashes when there is traffic)
* SFP connector (SFP is FTLF8526P3BNL, 6 Gbit/sec, 850 nm, 300m 50/125um OM3 MMF)
* 4 QSFP connectors (GDM)
* 6 VX connectors (CDM)

= Buttons, jumpers and switches =

Buttons:

* PB1 - HRST - reboot FPGA (power-on reset)
* PB2 - SRST - (SRSTn) - reboot ARM CPU

Switches:

* SW1 - boot mode BM0, BM1 [-->]
* SW2 - LEMO output NIM<->TTL
* SW3 - LEMO input 1 and 2 NIM/TTL
* SW4 - LEMO input 2 and 4 NIM/TTL
* SW5 - LEMO clock input NIM/TTL
* SW6 - serial console select. [PS<--PL] PS is ARM CPU, PL is FPGA.

= Front panel =

<pre>
| top
|
| LED-FP1 | LED_FP(0,1,2,3)
|
| SFP J???
|
| LEMO J4-LEFT, J4-RIGHT | EXT_IN_LV(1), EXT_IN_LV(2)
| LEMO J5-LEFT, J5-RIGHT | EXT_IN_LV(3), EXT_IN_LV(4)
| LEMO J6-LEFT, J6-RIGHT | CLK_EXT1, CLK_EXT0 (125 MHz only)
| LEMO J7-LEFT, J7-RIGHT | EXT_OUT(1), EXT_OUT(2)
|
| J-VX-1
| J-VX-2 or QSFP-1
| J-VX-3 or QSFP-2
| J-VX-4 or QSFP-3
| J-VX-5 or QSFP-4
| J-VX-6
|
| RJ45 J3 ethernet
|
| bottom
</pre>

= VX adapter board =

LVDS I/O connector
<pre>
34 pin connector: 0|:::: :::: :::: :::: :|15,16 n/c
</pre>

== split-cable connection ==

<pre>
0 .. 7 -> N/C

8 -> VX_RX(3) - not used
9 -> VX_RX(2) - busy VX to CDM
10 -> VX_RX(1) - DS20K 125 MHz serial data VX to CDM
11 -> VX_RX(0) - DS20K 62.5MHz clock VX to CDM

12 <- VX_TX(0) - TRG CDM to VX
13 <- VX_TX(1) - TSM CDM to VX, to be VETO CDM to VX
14 <- VX_TX(2) (set by jumper routed here or to VX CLKIN SYNC) - 125 MHz serial data CDM to VX
15 <- n/c (CLK routed to VX CLKIN CLK) - 62.5 MHz clock CDM to VX

16 - n/c
</pre>

== one-to-one connection ==

<pre>
0 -> VX2_RX(3)
1 -> VX2_RX(2)
2 -> VX2_RX(0)
3 -> VX2_RX(1)

4 <- CLK
5 <- VX2_TX(0)
6 <- VX2_TX(1)
7 <- VX2_TX(2)

8 -> VX1_RX(3)
9 -> VX1_RX(2)
10 -> VX1_RX(1)
11 -> VX1_RX(0)

12 <- VX1_TX(0)
13 <- VX1_TX(1)
14 <- VX1_TX(2) (set by jumper routed here or to VX CLKIN SYNC)
15 <- n/c (CLK routed to VX CLKIN CLK)

16 - n/c
</pre>

= Board schematics Rev0 =

* [[File:SCH-DS-xDM-Rev0.PDF]]
* note: FPGA pin annotations ("IO", "SCLK", "PCLK", etc) on the schematics are bogus, instead, trace them to the FPGA pins.
* note: ENC A is J800, ENC B is J801, ENC C is J900 (schematic name to enclustra name)
* note: Enclustra special pins: "GC" is "clock capable", "HDGC" is "clock capable", "MGTREFCLK" is MGT reference clocks.
* board modifications:
** ethernet mac chip
** NIM output (no U15, etc)
** RJ45 wrong pinout (board mod or special ethernet cable)
** 125 MHz clock mods (TBW)
** disconnect QSFP0_SEL from SFP_RS0 and QSFP1_SEL from SFP_RS1, these signals are not used by modern SFPs
** provide SFP i2c modsel to allow SFP and QSFP at the same time on address 0x50

= Board schematics Rev1 =

* [[File:SCH-DS-xDM-Rev1.PDF]]
* ENC-A is J800, ENC-B is J801, ENC-C is J900
* modifications from Rev0:
* clocks:
** 125MHz osc: CLK_XO_125, CLK3_XO_125 (CLK2_XO_125 removed)
** C.C. in2 is CLK_CC_IN from ENC-C142,144
** C.C. out0 via CLK_A to CLK_CC_OUT0 (ENC-C3,5), CLK_CC_OUT1 (ENC-B3,5), CLK_CC_OUT2 (ENC-C151,153)
* RTC chip: TP_S pins gone, RTC_I2C_SCL, RTC_I2C_SDA, RTC_1PPS and RTC_32KHZ added
** RTC_1Hz ENC-C160 pull up 10k to 3.3V
** RTC_32k ENC-B129 pull up 10k to 1.8V
* QSFP, SFP control lines:
** HW_ID_xDM ENC-B131 gone, reused as SFP_ModPrsN
** SFP_ModPrsN (MOD_ABS) ENC-B131 input (no pulls)
** SFP_TX_fault (was QSFP2_SEL) QSFP_ModPrsN ENC-C157 input (no pulls)
** SFP_TX_disable from QSFP_LPMode ENC-C163 output (pull up 10k to 3.3V)
** SFP_ModDet n/c (was QSFP_ModPrsN) renamed SFP_ModPrsN input
** SFP_RX_LOS to QSFP_IntN ENC-C159 input (pull up 2k2 to 3.3V)
** SFP_RS0, SFP_RS1 n/c, assembly option: floating
** QSFP_ResetN - FPGA output, pull up 10k to 3.3V
** QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL - FPGA outputs, pull up 10k to 3.3V
** QSFP_LPmode - FPGA output, pull up 10k to 3.3V
** QSFP_ModPrsN - ??? - pull up 2k2 to 3.3V
** QSFP_IntN - ??? - pull up 2k2 to 3.3V
* VX_TX and VX_RX reassigned for better length matching
* cosmetic changes:
** front panel USB-C connector (replaces Micro-USB)
** (reset push button PB1 PORn ENC-A132 aka PS_POR_B aka PS_POR#)
** (reset push button PB2 SRSTn ENC-A124 aka PS_SRST_B aka PS_SRST#)
** front panel PB3 HRSTn same as PB1
* Enclustra errata:
** possible cross talk between pins AA2 (J801-B90, VX9_RX2) and AE3 (J801-B129, RTC_32KHZ). no RTC on the CDM.
** ethernet link up failure (see Microchip KSZ9031 errata)

= FPGA MGT blocks =

<pre>
* SFP - ENC B45,B47 and B48,B50 - FPGA D5,D6 MGTHTX0_D and D1,D2 MGTHRX0_D
* QSFP0 TX0 - ENC C13,C17 - FPGA H5,H6 MGTHTX0_C
* QSFP0 TX1 - ENC C21,C25 - FPGA G7,G8 MGTHTX1_C
* QSFP0 TX2 - ENC C29,C23 - FPGA F5,F6 MGTHTX2_C
* QSFP1 TX0 - ENC C37,C41 - FPGA E7,E8 MGTHTX3_C
* QSFP1 TX1 - ENC C45,C47 - FPGA P5,P6 MGTHTX0_B
* QSFP1 TX2 - ENC C51,C53 - FPGA M5,M6 MGTHTX1_B
* QSFP2 TX0 - ENC C57,C59 - FPGA L3,L4 MGTHRX2_B
* QSFP2 TX1 - ENC C63,C65 - FPGA K5,K6 MGTHTX3_B
* QSFP2 TX2 - ENC C75,C77 - FPGA W3,W4 MGTHTX0_A
* QSFP3 TX0 - ENC C79,C81 - FPGA V5,V6 MGTHTX1_A
* QSFP3 TX1 - ENC C85,C87 - FPGA T5,T6 MGTHRX2_A
* QSFP3 TX2 - ENC C89,C91 - FPGA R3,R4 MGTHTX3_A
</pre>

= Clock distribution Rev0 =

Simplified:

* 125 MHz osc -> CLK_XO_125 -> MGTREFCLK0_A -> not used
* 125 MHz osc -> CLK3_XO_125 -> MGTREFCLK1_B -> SFP RX ref clock, QSFP RX and TX ref clock (this is not final design!)
* (disconnected) 125 MHz osc -> CLK2_XO_125 -> FPGA AG8,AH8 (GC)
* 125 MHz osc -> C.C. in1

* C.C. in0 <- CLK_EXT1 (10 MHz GPS clock)
* C.C. in1 <- 125 MHz osc
* (disconnected) C.C. in2 <- CLK_CC_IN <- MGTREFCLK0_D <- SFP RX clock (cannot be used because of uncontrollable phase)
* C.C. in2 <- CLK2_XO_125 <- FPGA AG8,AH8 (GC) <- SFP RX recovered 125 MHz clock
* C.C. in3 <- CLK_FB

* C.C. 125 MHz -> CLK_CC_OUT0 -> MGTREFCLK0_B -> QSFP RX and TX ref clock (final design)
* C.C. 125 MHz -> CLK_CC_OUT1 -> MGTREFCLK1_D -> SFP TX clock
* C.C. 62.5 MHz -> VX clock fanout

proposed changes:
* add C.C. 125 MHz -> new CLK_CC_OUT2 (old CLK2_XO_125) -> FPGA AG8,AH8 (GC)
* change C.C. in2 <- new CLK_CC_IN <- FPGA AK8,AK9 (non-GC)

Complete:

<pre>
125 MHz oscillator - U5 fan out -
q0 -> CLK_XO_125 -> ENC C72,C74 -> FPGA R7,R8 MGTREFCLK0_A (not used)
q1 -> U6 C.C. in1
q2 -> CLK3_XO_125 -> ENC C7,C9 -> FPGA J7,J8 MGTREFCLK1_B -> SFP RX reference clock, QSFP RX and TX reference clocks (not final design!)
q3 -> disconnected on the board, was CLK2_XO_125 -> ENC C151,C153 -> FPGA AG8,AH8 (GC)

U6 C.C (clock cleaner) -

in0 <- CLK_EXT1 (presumably GPS 10 MHz ref clock)
in1 <- 125 MHz oscillator via U5
in2 <- (was: CLK_CC_IN <- ENC B10,B12 <- FPGA D9,D10 MGTREFCLK0_D <- SFP RX clock, 125 MHz)
in2 <- CLK2_XO_125 <- ENC C151,B153 <- FPGA AG8,AH8 (GC) <- mgt_link_data_to_processing.rx_data_clk (SFP RX data clock, 125 MHz)
in3 <- CLK_FB

out0 -> CLK_CCA -> U12 (125 MHz)
out1 -> CLK_CCB -> VX1..6 (62.5 MHz)
out2 -> CLK_CCC -> VX7..12 (62.5 MHz)
out3 -> CLK_FB into in3

CLK_CCA -> U12 (125 MHz fan out) ->

Q0 -> not used
Q1 -> CLK_CC_OUT0 -> ENC C3-5 -> FPGA L7,L8 MGTREFCLK0_B -> QSFP RX and TX reference clocks (final design)
Q2 -> CLK_CC_OUT1 -> ENC B3-5 -> FPGA B9,B10 MGTREFCLK1_D -> SFP TX clock
Q3 -> not used
Q4 -> not used
Q5 -> SMA J9/J10

CLK_TP0 - FPGA N7,N8 MGTREFCLK1_A - ENC C69,C71 - DS-DM SMA J11, J12 (NOT IN CDM PROJECT)

Notes:
* CLK_XO_125 (125 MHz osc) is not used
* 62.5 MHz VX clock does not go into the FPGA
* CLK_EXT0 going to in1 of U5 cannot be used. only permitted frequency is 125 MHz (it drives the MGT reference clocks) and is this frequency is too high for LEMO cables and connectors

Proposed modifications:
- CLK_CC_IN: from FPGA output pin (ENC C142,C144 - FPGA AK8,AK9) to clock cleaner in2 (instead of CLK2_XO_125 pins)
- CLK_CCA -> U12 -> currently unused out3 -> CLK2_XO_125 FPGA pins
- repurpose CLK_EXT0 at GPS 1pps/IRIG input to FPGA
</pre>

= Clock distribution Rev1 =

Simplified:

* 125 MHz osc -> CLK_XO_125 -> MGTREFCLK0_A -> clk_xo_125. not used
* 125 MHz osc -> CLK3_XO_125 -> MGTREFCLK1_B -> CDM SFP RX ref clock
* 125 MHz osc -> C.C. in1

* C.C. in0 <- CLK_EXT1 (10 MHz GPS clock)
* C.C. in1 <- 125 MHz osc
* C.C. in2 <- CLK_CC_IN <- FPGA AK9,AK8 <- CLK_CC_IN1, rx_clk_mgt, SFP RX recovered clock, 125 MHz
* C.C. in3 <- CLK_FB

* C.C. 125 MHz -> CLK_CC_OUT0 -> MGTREFCLK0_B -> GDM QSFP RX and TX ref clock, CDM rx_clk mux
* C.C. 125 MHz -> CLK_CC_OUT1 -> MGTREFCLK1_D -> SFP TX clock
* C.C. 125 MHz -> CLK_CC_OUT2 -> FPGA AG8,AH8 (GC) -> not used
* C.C. 62.5 MHz -> VX clock fanout

Complete:

<pre>
125 MHz oscillator - U5 fan out -
q0 -> CLK_XO_125 -> ENC C72,C74 -> FPGA R8,R7 MGTREFCLK0_A -> XDC CLK_XO_125_P -> VHDL not used
q1 -> U6 C.C. in1
q2 -> CLK3_XO_125 -> ENC C7,C9 -> FPGA J8,J7 MGTREFCLK1_B -> XDC GDM missing, CDM CLK3_XO_125_P -> VHDL SFP RX reference clock (mgt_rx_ref_clk)
q3 -> not used, was CLK2_XO_125

U6 C.C (clock cleaner) -

in0 <- CLK_EXT1 (GPS 10 MHz clock)
in1 <- 125 MHz oscillator via U5
in2 <- CLK_CC_IN <- ENC C142,C144 <- FPGA AK9,AK8 <- XDC GDM missing, CDM CLK_CC_IN1 <- VHDL rx_clk_mgt
in3 <- CLK_FB

out0 -> CLK_CCA -> U12 (125 MHz)
out1 -> CLK_CCB -> VX1..6 (62.5 MHz)
out2 -> CLK_CCC -> VX7..12 (62.5 MHz)
out3 -> CLK_FB into in3

CLK_CCA -> U12 (125 MHz fan out) ->

Q0 -> not used
Q1 -> CLK_CC_OUT0 -> ENC C3,C5 -> FPGA L8,L7 MGTREFCLK0_B -> XDC GDM clk_mgtrefclk0_x0y1_p, CDM i_CLK_CC_OUT0_P -> VHDL GDM mgt_b_ref_clk QSFP RX and TX reference clocks, CDM alternate rx_clk via clock mux
Q2 -> CLK_CC_OUT1 -> ENC B3,B5 -> FPGA B10,B9 MGTREFCLK1_D -> XDC i_CLK_CC_OUT1_P -> VHDL GDM not used, CDM mgt_tx_ref_clk SFP TX clock
Q3 -> CLK_CC_OUT2 -> ENC C151,C153 -> FPGA AG8,AH8 (GC) -> XDC GDM missing, CDM i_CLK_CC_OUT2_P
Q4 -> not used
Q5 -> SMA J9/J10

CLK_TP0 <-> ENC C69,C71 <-> FPGA N8,N7 MGTREFCLK1_A (not used) -> XDC CLK_TP0_P -> VHDL not used (DS-DM SMA J11, J12)

disconnected <- OLD_CLK_CC_IN <- ENC B10,B12 <- FPGA D9,D10 MGTREFCLK0_D <- XDC o_CLK_CC_IN <- VHDL CDM o_OLD_CLK_CC_IN SFP o_mgt_rx_rec_clk

Notes:
* Enclustra ENC-B is J801, ENC-C is J900
* "(GC)" is clock-capable FPGA pin
* CLK_XO_125 (125 MHz osc) is not used in FPGA
* 62.5 MHz VX clock does not go into the FPGA
* CLK_EXT0 going to in1 of U5 cannot be used. only permitted frequency is 125 MHz (it drives the MGT reference clocks) and is this frequency is too high for LEMO cables and connectors
</pre>

= I2C bus =

* I2C_SCL is J-ENC-A pin 111 I2C_SCL is FPGA I2C_SCL_PL AB13 (IO_L1N_TOL1D_64) and I2C_SCL_PS F18 (PS_MIO10)
* I2C_SDA is J-ENC-A pin 113 I2C_SDA is FPGA I2C_SDA_PL AH13 (IO_L7N_T1L1Q_AD13N_64) and I2C_SDA_PS G18 (PS_MIO11)

* XU8 secure EEPROM ATSHA204A at 0x64, this is 0110'010X -> linux _011'0010 is 0x32. (but responds to scan and read at 0x33)
* U4 ethernet mac chip, EEPROM at 1010 A2 A1 A0 X and MAC/serial_no at 1011 A2 A1 A0 X. A0=VCC, A1=VCC, A2=GND -> linux _101'0011 and _101'1011 is 0x53 and 0x5B.
* U6 clock chip, address 1101 0 A1 A0 X. A1=VCC, A0=N/C (internal pull-up) -> linux _110'1011 is 0x6b
* U23 voltmeter at 10011 ADR1 ADR0. ADR0=GND, ADR1=VCC -> linux _100'1110 is 0x4e
* SFP, address 1010000X -> linux _101'0000 is 0x50. additional SFP data at 0x51
* QSFP0, QSFP1, QSFP2, QSFP3 (QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL)

= I2C clock builder connection =

* use Silicon Labs USB "Clock builder pro field programmer", www.silabs.com/CBProgrammer
* connect rainbow jumper cable pins:
** black - 1-GND to GND on DS-DM
** white - 3-SCLK to SCL on the DS-DM
** grey - 7-SDA_SDIO to SDA on the DS-DM
* power up the DS-DM
* plug USB programmer into Windows laptop
* on Windows, run "ClockBuilder Pro"
* it should report "Field programmer detected", press "EVB GUI"
* in EVB GUI, press "Config", set I2C address 0x6B
* press "Scan", it should find Si5394A-A-GM
* select the "Status" tab, should see real-time status of clock chip

= GDM MGT configuration =

* TX configuration:
* GDM MGT transceivers are configured as "multilane" TX and RX.
* there is 12 TXes ("lanes")
* MGT reference 125 MHz clock goes into [2:0]gtreclk00_in and [11:0]gtrefclk0_in
* one MGT is designated as "master"
* PLL of master MGT converts reference clock into common TX clock and common tx_user_clk2 which becomes tx_data_clk
* common tx_user_clk2 aka tx_data_clk goes into all TXes and clocks tx_data.
* tx_user_clk2 aka tx_data_clk is 125 MHz but not same phase as MGT reference clock.

* RX interim configuration:
* there is 12 RXes ("lanes")
* each RX produces it's own recovered RX clock
* "multilane" configuration assumes all RX recovered clocks run at the same frequency (TX on the other end are driven by common TX clock, see above), but have different phase
* one RX recovered clock is designated as "master" (rx_user_clk2 aka rx_data_clk) and a phase-matching fifo/buffer is used to bring rx_data from all 12 RXes to this common rx_data_clk
* this works because each CDM SFP TX runs on the SFP RX recovered clock which is frequency-locked with the GDM QSFP TX clock.

* RX final configuration:
* MGTs permit using the common TX clock (tx_user_clk2 aka tx_data_clk) as the common rx_data_clk (they are frequency locked through the CDM).
* this permits use of tx_data_clk as the main clock domain in the GDM and removes the need to bring rx_data into the tx_data_clk domain (actually this is done in the MGT RX phase matching fifo/buffer).

= Clock path =

NOTE: MUST REVIEW!!!

<pre>
10 MHz ext clock or GDM 125 MHz oscillator
-> GDM QSFP MGT reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz (GDM main clock domain) and TX bit clock 2.5 GHz
-> GDM QSFP optic transmitter
-> CDM SFP optic receiver
-> CDM SFP MGT, RX reference clock is CDM 125 MHz oscillator
-> MGT RX recovered clock 125 MHz (CDM main clock domain)
-> CC_CLK_IN -> CDM C.C. -> CC_CLK_OUT1 -> CDM SFP TX reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz and TX bit clock 2.5 GHz
-> (tx_data phase matching fifo from CDM main clock domain to tx_data_clk)
-> CDM SFP optic transmitter
-> GDM QSFP RX optic receiver (12x)
-> GDM QSFP MGT (RX reference clock is same as TX reference clock)
-> MGT RX recovered clock (12x recovered clocks)
-> in multi-lane configuration, one of them is the "master" recovered clock rx_data_clk
-> (rx_data phase matching fifo from rx_data_clk to GDM main clock domain)
</pre>

= CDM rx_clk mux =

when CDM SFP is not connected, there is no SFP recovered clock and a mux is used to switch between clk_cc_out0 (power up default) and rx_clk_mgt (SFP recovered clock)

== Test SFP disconnected ==

note: if I say "--cc-in1", CC seems to lock on the 10 MHz GPS external clock,
to prevent this, test sequence includes reloading the CC and the reset of MGT.

<pre>
./test_cdm_local.exe --write32 0x30 0x0 ### rx_clk mux select CC clock
./test_cdm_local.exe --load-cc
./test_cdm_local.exe --reset-mgt
./test_cdm_local.exe --cdm-clocks
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773594e (125000014) should be ~125 MHz <=== all 4 clocks wobble close to 125 MHz
0x1034 rx_clk: 0x0773594f (125000015) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x0773594f (125000015) should be ~125 MHz
0x103C tx_clk: 0x0773594f (125000015) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
./test_cdm_local.exe --write32 0x30 0x1 ### rx_clk mux select SFP recovered clock
./test_cdm_local.exe --cdm-clocks
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773598b (125000075) should be ~125 MHz
0x1034 rx_clk: 0x076d58ec (124606700) should be ~125 MHz <=== off frequency because there is no valid SFP recovered clock
0x1038 mgt_tx_ref_clk_raw: 0x0773598c (125000076) should be ~125 MHz
0x103C tx_clk: 0x0773598b (125000075) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

= Clock domains =

== GPS ==

* no GPS : GDM runs from internal 125 MHz oscillator
* external 10 MHz clock : GDM runs from external 10 MHz clock and optional 1pps signal (use VME-NIMIO32 NIM outputs)
* GPS receiver : GDM runs from GPS 10 MHz clock and GPS IRIG serial data
* LNGS GPS:
** provides 1pps and serial data over fiber from GPS receiver (master). LNGS xxx box is not used.
** serial data goes to GDM, decoded, 1pps signal extracted, goes to LEMO output
** 1pps from GDM is used to train the Rubidium clock which provides a 10 MHz clock
** 10 MHz output from Rubidium clock goes to GDM 10 MHz external clock input

== GDM ==

* AXI clock (100 MHz) - AXI registers
* 125 MHz oscillator - to clock cleaner
* 10 MHz external clock LEMO input - to clock cleaner
* FPGA 125 MHz clock CLK_CC_IN - to clock cleaner - not used (could be used for cascaded GDMs)
* 125 MHz oscillator - CLK3_XO_125 - mgt_b_ref_clk QSFP MGT reference clock (interim GDM design)
* clock cleaner output 125 MHz fanout:
** CLK_CC_OUT0 - QSFP MGT reference clock (final design)
** CLK_CC_OUT1 - not used (CDM SFP reference clock)
** CLK_CC_OUT2 - not used
* QSFP MGT TX data clock 125 MHz tx_data_clk (main clock domain)
** QSFP TX data
** QSFP RX data (in final design, rx_data_clk is same as tx_data_clk. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk == tx_data_clk)
** ds20k block
* (in interim design, rx_data_clk is disconnected from tx_data_clk. rx_data_clk is the "multilane master clock" which is one of the 12x RX recovered clocks. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk)

note: all these clocks are frequency locked to 125 MHz

== CDM ==

* AXI clock (100 MHz) - AXI registers
* 10 MHz external clock LEMO input - to clock cleaner (not used)
* 125 MHz oscillator to fanout
** to clock cleaner
** to SFP MGT RX reference clock: CLK3_XO_125 to mgt_rx_ref_clk to gtrefclk01_in
* SFP MGT RX recovered clock 125 MHz
** MGT PLL to MGT rx_user_clk2 aka rx_data_clk
** MGT rxrecclkout_out to mgt_rx_rec_clk to CLK_CC_IN to C.C.
* SFP MGT rx_user_clk2 aka rx_data_clk (250 MHz/8 bit, 125 MHz/16 bit, 62.5 MHz/32 bit data) (main clock domain)
** SFP RX data
** ds20k block
** VX TX clock PLLs
** VX RX clock PLLs
* C.C. fan out
** 62.5 MHz VX clocks (12x)
** CLK_CC_OUT0 (not used, GDM QSFP MGT reference clock)
** CLK_CC_OUT1 125 MHz to mgt_tx_ref_clk to gtrefclk00_in to SFP MGT TX reference clock
** CLK_CC_OUT2 (not used)
* SFP MGT tx_user_clk2 aka tx_data_clk
** SFP TX data
** TX data phase matching fifo from main clock domain to tx_data_clk
* VX TX clock PLLs
** 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
** VX TX data phase matching from main clock domain to VX TX clock (12 total)
** VX TX serializer
** VX TX LVDS transmitter
* VX RX clock PLLs
** 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
** VX RX LVDS receivers (12 total)
** VX RX deserializers (12 total)
** VX TX data phase matching from VX RX clock to main clock domain

== VX ==

* everything runs on the VX main 125 MHz clock
* correct phase of VX to CDM LVDS data is adjusted by scan of CDM VX RX PLL clock phase (VX to CDM link is now established)
* correct phase of CDM to VX LVDS data is adjusted by scan of CDM VX RX PLL clock phase (link is established after VX to CDM idle data pattern changes from "VX RX data bad" to "good".
* after good phases are found by scan, they are not expected to change unless cables are changed, CDM and VX modules are changed or CDM firmware is rebuilt. (rebuild of VX firmware should not affect LVDS data phase).
* if there is excessive link errors, phase scan must be repeated.

= Board test plan =

To test:
* Enclustra FPGA board
* SFP port - SFP_RS0/RS1 connected to QSFP0_SEL/1_SEL is wrong? our Finisar SFP says RS0, RS1 N/C, so probably okey.
* SW5 CLK_EXT1 NIM works. TTL needs to be tested. flipping SW5 CLK_EXT0 side from NIM to TTL makes CLK_EXT1 go LOS and OOF in the clock chip. R23 and R53 should be removed?
* BOOT_MODE 0 and 1

Partial:
* U23 3.3V current meter and thermometer. V1,V2 is current monitor, same as in application note. V3,V4 is thermometer, same in application note, except capacitor C118 is 0.1u instead of 470pF. Tested ok: Tint, VCC, V1, V2, V1-V2. Test failed: TR2 reads 50-something degC instead of same as Tint.

Done:
* LED_FP1A..D: tested ok. K.O. 15 sep 2022
* USB UART: tested ok. K.O. 15 sep 2022
* J4A, J4B, J5A, J5B LEMO inputs (NIM/TTL) EXT_IN_LV(1..4). TTL threshold 1.7V, NIM threshold -0.3V. 50 Ohm termination. TTL straight, NIM inverted.
* J6A, J6B LEMO clock inputs: 10 MHz TTL from chronobox works, 10 MHz NIM from IO32 works. 18-apr-2023
* J7A, J7B LEMO outputs EXT_OUT(1), EXT_OUT(2) (NIM/TTL) (tested 23nov2022, K.O.). TTL ???, NIM inverted.
** TTL out no 50 ohm termination: 0=0V, 1=5V, rise and fall time ~5 ns
** TTL out with 50 ohm termination: 0=0V, 1=2.5V, rise and fall time <2ns
** NIM out no 50 ohm termination: 0=+50mV, 1=-1.8V, rise and fall time ~3ns
** NIM out with 50 ohm termination: 0=0V, 1=-0.9V, rise and fall time <2ns
* ethernet MAC i2c chip (K.O. 20sep2022: can read: i2cdump 0 0x5b, nov2022 have u-boot driver)
* CDM VX ports 2x(CLK, 3 tx, 4 rx) tested using test_cdm.exe and LVDS loopback in VX firmware.
* SFP i2c tested KO 22jun2023
* QSFP i2c tested KO 22jun2023
* i2c testing complete 22jun2023
* QSFP rx,tx tested 26june2023. lane0,1,2 ok, lane3 tx not connected, laser is off. qsfp0,1,2,3 all lanes ok.
* SFP rx,tx tested 26june2023. rx and tx okey. LOS ok, mod_absent ok.

Failure:
* ethernet: does not connect to alliedtelesys switch. connects to my USB-eth adapter at 100 Mbit speed. uboot mii status reports connection speed oscillating between 1000, 100 and 10. K.O. 16-sep-2022
* SFP LOS and mod_absent are swapped (in the FPGA pin definitions?)
* QSFP slot numbering is wrong.

= Checklist for newly build boards =

== setup and boot ==
* put new board on workbench
* check - vme connector present, vme extraction handles present
* check - standoff are removed from all thru-holes
* plug Enclustra module
* check - SW1 both switches are in the "6" and "3" positions
* check - SW6 both switches are in the "PS" position
* Rev0: connect micro-usb cable from linux PC
* Rev1: connect usb-C cable from linux PC (USB C-to-C or C-to-A)
* connect ethernet from 1gige capable network switch
* connect power from lab power supply - GND, +5V and -12V nominal, +5.7V and -12.5V actual
* power up, +5V current 2.10-2.8A, -12V current 0.05A
* on linux PC, open a new terminal, run: minicom -D /dev/ttyACM0 -b 115200
* in minicom window, observe messages about Xilinx first stage boot loader, etc
* on the ethernet switch, observe network link is 1gige speed (not 10mbit, not 100mbit).
* if everything boots okey, there will be a login prompt, login as root, password root.
* alternatively, from root@daq13: ssh root@dsdm
* busybox devmem 0x80010000 # read firmware version number, i.e. 0xEDAD0A77
== test i2c ==
* check i2c: (no SFP, no QSFP plugged in!)
<pre>
root@dsdm:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@dsdm:~#
</pre>
* test clock chip:
<pre>
root@dsdm:~# ./test_cdm_local.exe --load-cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Loading CC registers...
Clock chip data block size 519
root@dsdm:~# ./test_cdm_local.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Polling CC status...
Clock chip state 1, status: LOS_IN0 OOF_IN0 IN1 IN_SEL_1
Ctrl-C
</pre>
* test RTC:
<pre>
root@dsdm:~# ./test_cdm_local.exe --rtc-read
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Read the RTC chip!
reg 0x00: 0 0 1 0 0 1 1 0 0x26
reg 0x01: 0 0 1 1 1 0 0 1 0x39
reg 0x02: 0 0 0 0 0 1 1 1 0x07
reg 0x03: 0 0 0 0 0 0 1 0 0x02
reg 0x04: 0 0 0 0 0 0 1 0 0x02
reg 0x05: 0 0 0 0 0 0 0 1 0x01
reg 0x06: 0 0 0 0 0 0 0 0 0x00
reg 0x07: 0 0 0 0 0 0 0 0 0x00
reg 0x08: 0 0 0 0 0 0 0 0 0x00
reg 0x09: 0 0 0 0 0 0 0 0 0x00
reg 0x10: 0 0 0 0 0 0 0 0 0x00
reg 0x11: 0 0 0 0 0 0 0 0 0x00
reg 0x12: 0 0 0 0 0 0 0 0 0x00
reg 0x13: 0 0 0 0 0 0 0 0 0x00
reg 0x14: 0 0 0 1 1 0 0 0 0x18
reg 0x15: 0 0 0 0 1 0 0 0 0x08
reg 0x16: 0 0 0 0 0 0 0 0 0x00
reg 0x17: 0 0 0 1 0 1 1 1 0x17
reg 0x18: 0 1 0 0 0 0 0 0 0x40
seconds: 26, minutes: 39, hours: 07, day 2, date: 00-01-02, temp 23.25C
root@dsdm:~#
</pre>

== test clocks ==

<pre>
root@dsdm:~# ./test_cdm_local.exe --reset-mgt
...
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735cb9 (125000889) should be ~125 MHz
0x1034 rx_clk: 0x07735cb9 (125000889) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735cb9 (125000889) should be ~125 MHz
0x103C tx_clk: 0x07735cb9 (125000889) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
^C
root@dsdm:~# ./test_cdm_local.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735c6a (125000810) should be ~125 MHz
0x1034 rx_clk: 0x07735c69 (125000809) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735c69 (125000809) should be ~125 MHz
0x103C tx_clk: 0x07735c6a (125000810) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
^C
root@dsdm:~#
</pre>

== test LED, LEMO ==

* test LEDs, observe all 4 LEDs turn on and off every 1 second
<pre>
root@dsdm:~# ./test_cdm_local.exe --blink-led
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
LED on
LED off
</pre>
* TBW - test LEMO outputs
<pre>
have test firmware lemo outputs are RTC chip 1pps and 48kHz
see them on the scope
</pre>
* TBW - test LEMO inputs
<pre>
./test_cdm_local.exe --monreg 6
</pre>

== test VX ==

* on the CDM, connect all split blue cables to VX modules
* set all VX modules to LVDS loopback mode (in VX settings MIDAS page)
* ssh root@cdm00
* run "./test_cdm.exe --test-vx", it should report:
<pre>
CDM registers: 0x0000000000000000 0x00000000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000eeeeeeeeeeee 0x00007700: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000080808080808 0x00000100: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000040404040404 0x00000200: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000020202020202 0x00000400: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000000000000000 0x00000800: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000808080808080 0x00001000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000404040404040 0x00002000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000202020202020 0x00004000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000000000000000 0x00008000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
all on!
all off!
</pre>
* on the VX settings MIDAS page LVDS input section, observe line status is cycling on and off.
* note: VX_RX(0) is not tested by this, corresponding VX_TX is used as the VX clock

== test SFP (CDM) ==

* connect SFP module
* ssh root@cdm00
* run "./test_cdm.exe --sfp", it should report:
<pre>
root@cdm00:~# ./test_cdm.exe --sfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x6a332b58, ds20k version 0x20241209
DS-DM firmware build 0x6a332b58, ds20k version 0x20241209
Polling SFP status...
identifier 0x03
connector 0x07
encoding 0x01
wavelength 0x0352 (850 nm)
vendor_name [FINISAR CORP. ]
vendor_pn [FTLF8526P3BNL ]
vendor_rev [A ]
vendor_sn [N3AB4LV ]
vendor_date [200319 ]
dm_type 0x68
temp 30.9 C
vcc 3.285 V
tx_bias 7.362 mA
tx_power 475.7 uW
rx_power 859.1 uW
SFP good 1, status: temp 30.9 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 859 uW
SFP good 1, status: temp 30.8 C, tx_bias 7.4 mA, tx_power 475 uW, rx_power 859 uW
SFP good 1, status: temp 30.8 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 859 uW
SFP good 1, status: temp 30.7 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 859 uW
SFP good 1, status: temp 30.8 C, tx_bias 7.4 mA, tx_power 475 uW, rx_power 858 uW
^C
root@cdm00:~#
</pre>
* NOTE1: if fiber cable is disconnected or GDM is not running, rx_power would read 0 uW
* NOTE2: better SFP modules have smaller tx_bias and larger tx_power (more bang for the buck)
* NOTE3: bad fiber cable and bad fiber connection will result in smaller rx_power

== test QSFP (GDM) ==

* connect QSFP module (any slot)
* ssh root@gdm01
* run "./test_cdm.exe --qsfp3 --qsfp", try --qsfp0, --qsfp1, --qsfp2, --qsfp3 for the 4 QSFP slots
<pre>
root@gdm01:~# ./test_cdm.exe --qsfp3 --qsfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x9e30baf9, ds20k version 0x20241209
DS-DM firmware build 0x9e30baf9, ds20k version 0x20241209
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out hi
Polling QSFP status...
identifier 0x0d
status 0x02
los 0x89
temp 36.7 C
vcc 3.302 V
rx_power 0.1 413.7 460.3 0.1 uW
tx_bias 7.8 7.9 7.9 0.0 mA
tx_power 806.0 842.9 865.2 0.1 uW
vendor_name [FINISAR CORP ]
vendor_pn [FTL410QD4C ]
vendor_rev [A ]
wavelength 850
max_temp 70 C
vendor_sn [X79AC0R ]
vendor_date [220309 ]
QSFP good 1, status: temp 36.7 C, los 0x89, tx_bias 7.8 7.9 7.9 0.0 mA, tx_power 806 843 865 0 uW, rx_power 0 414 460 0 uW
^C
</pre>
* NOTE1: 4th tx_bias and tx_power are zero because 4th channel does not transmit any data, QSFP detects this and automatically shuts down the optical transmitter

= Serial console =

* check that linux computer has correct udev rules to allow access to /dev/ttyACM devices, see https://daq00.triumf.ca/DaqWiki/index.php/SLinstall#Configure_USB_device_permissions and https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Configure_USB_device_permissions
* connect micro-USB cable to connector J-UCB, other end connect to linux computer
* observe /dev/ttyACM0 was created
* run "minicom -D /dev/ttyACM0 -b 115200"
* should have gdm-cdm login
* username root, password root

= i2c =

<pre>
ZynqMP> i2c bus
Bus 0: i2c@ff020000
ZynqMP> i2c dev 0
Setting bus to 0
ZynqMP> i2c probe
Valid chip addresses: 33 4E 53 5B 6B 77
ZynqMP> i2c md 0x5b 0x98
0098: fc c2 3d 00 00 12 1a 6e 0a 90 85 04 94 10 08 50 ..=....n.......P
</pre>

<pre>
root@gdm-cdm:~# i2cdetect 0
Warning: Can't use SMBus Quick Write command, will skip some addresses
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0.
I will probe address range 0x03-0x77.
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f
00:
10:
20:
30: -- -- -- 33 -- -- -- --
40:
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60:
70:
root@gdm-cdm:~#
root@gdm-cdm:~# i2cdump 0 0x5b
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0, address 0x5b, mode byte
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00 ???????P?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e ........??=..??n
a0: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00 ???????P?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e ........??=..??n
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@gdm-cdm:~#
</pre>

<pre>
root@gdm0:~# i2cdetect -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#
</pre>

<pre>
root@cdm0:~# i2cdetect -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: 50 51 -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@cdm0:~#
</pre>

* 0x33 - XU8 secure EEPROM (should be at 0x32)
* 0x4e - U23 current and temperature monitor
* 0x50, 0x51 - SFP
* 0x50 - QSFP, 4 QSFP modules enabled by GPIO QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL
* 0x53, 0x5b - ethernet mac eeprom
* 0x6b - U6 clock chip

== U23 ==

NOTE: Text reads double of Tint. not sure why. K.O. 21-mar-2024.

* internal temperature only

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x00 b # control register: "repeat mode, internal temperature only"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x03 # "Tint ready" and "busy", "busy is always 1 in repeat mode"
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 00 03 03 81 db 2a ce 2a 8e 00 6f 00 45 20 3f ?.????*?*?.o.E ?
10: 01 00 01 01 01 db 2a ce 2a 8e 00 6f 00 45 20 3f ?.????*?*?.o.E ?
...
readback:
reg0 - 03 - Tint ready
reg1 - 00 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB 0x81, bit 0x80 is "DV, data valid", bit 0x40 is "SS, sensor short", 0x20 is "SO, sensor open"
reg5 - Tint LSB 0xDB, Tint = 0x01DB = 475 * 0.0625 degC = 29.6 degC
reg6..F - stale data
</pre>

* Tint, V1, V2, TR2, VCC

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x18 b # control register: "repeat mode, V1, V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 7f 18 7f 7f 81 b9 aa d0 aa 8f 83 5e 83 5e a0 41 ???????????^?^?A
10: 01 18 01 01 01 b9 2a d0 2a 8f 03 5e 03 5e 20 41 ??????*?*??^?^ A
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x1b9*0.0625 = 27.5 degC
reg6 - V1 MSB 0xaa, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 LSB 0xd0, V1 = 0x2ad0*305.18/1000000 = 3.3447 V (correct)
reg8 - V2 MSB 0xaa, ditto
reg9 - V2 LSB 0x8f, V2 = 0x2a8f*305.18/1000000 = 3.3249 V (correct, smaller than V1)
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x5e. TR2 = 0x35e*0.0625 = 53.875 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x41, VCC = 2.5+0x2041*305.18/1000000 = 5.019 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.99 A (about right?)
</pre>

* Tint, V1-V2, TR2, VCC

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x19 b # control register: "repeat mode, V1-V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 7f 19 7f 7f 81 69 83 f2 83 f2 83 3a 83 3a a0 44 ?????i?????:?:?D
10: 01 19 01 01 01 69 03 f2 03 f2 03 3a 03 3a 20 44 ?????i?????:?: D
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x169*0.0625 = 22.5 degC
reg6 - V1 or V1-V2 MSB 0x83, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 or V1-V2 LSB 0xf2, V1-V2 = 0x3f2*19.42/1000000 = 0.0196142 V (correct, compare with V1 and V2 measured above)
reg8 - V2 or V1-V2 MSB
reg9 - V2 or V1-V2 LSB
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x3a. TR2 = 0x33a*0.0625 = 51.625 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x44, VCC = 2.5+0x2044*305.18/1000000 = 5.021 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.9807 A (about right?)
</pre>

== SFP ==

<pre>
root@cdm0:~# i2cdump 0 0x50
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 04 07 00 00 00 00 40 40 0c 00 01 3d 00 00 00 ???....@@?.?=...
10: 0c 02 00 1e 46 49 4e 49 53 41 52 20 43 4f 52 50 ??.?FINISAR CORP
20: 2e 20 20 20 00 00 90 65 46 54 4c 46 38 35 32 36 . ..?eFTLF8526
30: 50 33 42 4e 4c 20 20 20 41 20 20 20 03 52 00 9d P3BNL A ?R.?
40: 00 1a 00 00 4e 33 41 42 34 4c 56 20 20 20 20 20 .?..N3AB4LV
50: 20 20 20 20 32 30 30 33 31 39 20 20 68 f0 03 de 200319 h???
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@cdm0:~#
root@cdm0:~# i2cdump 0 0x51
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 5a 00 d8 00 55 00 e2 00 90 88 71 48 8c a0 75 30 Z.?.U.?.??qH??u0
10: 21 34 01 f4 1b 58 03 e8 31 2d 04 eb 1f 07 06 31 !4???X??1-?????1
20: 31 2d 00 64 27 10 00 9e 00 00 00 00 00 00 00 00 1-.d'?.?........
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
40: 00 00 00 00 3f 80 00 00 00 00 00 00 01 00 00 00 ....??......?...
50: 01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 b7 ?...?...?......?
60: 19 9f 80 c5 0e 17 12 c4 1f 99 00 00 00 00 30 00 ??????????....0.
70: 00 00 00 00 00 00 00 00 ff ff ff ff ff ff ff 01 ...............?
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@cdm0:~#
</pre>

== QSFP ==

<pre>
QSFP i2c enable lines, active low:

QSFP0_SEL - JENC-A 82 - PS-MIO40 - linux gpio 378
QSFP1_SEL - JENC-A 84 - PS-MIO41 - linux gpio 379
QSFP2_SEL - JENC-A 100 - PS-MIO44 - linux gpio 382
QSFP3_SEL - JENC-A 106 - PS-MIO43 - linux gpio 381 (notice 2 and 3 are out of order)
</pre>

<pre>
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
# echo 378 >> /sys/class/gpio/export ### SEL0 338+40
# echo 379 >> /sys/class/gpio/export ### SEL1 338+41
# echo 381 >> /sys/class/gpio/export ### SEL3 338+43
# echo 382 >> /sys/class/gpio/export ### SEL2 338+44
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) in hi
gpio-379 ( |sysfs ) in hi
gpio-381 ( |sysfs ) in hi
gpio-382 ( |sysfs ) in hi
root@gdm0:~# echo out >> /sys/class/gpio/gpio381/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio382/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio378/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio379/direction
root@gdm0:~#
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out lo
gpio-379 ( |sysfs ) out lo
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out lo
root@gdm0:~#
echo 1 >> /sys/class/gpio/gpio381/value
echo 1 >> /sys/class/gpio/gpio382/value
echo 1 >> /sys/class/gpio/gpio378/value
echo 1 >> /sys/class/gpio/gpio379/value
cat /sys/kernel/debug/gpio
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out hi
gpio-382 ( |sysfs ) out hi
root@gdm0:~#
root@gdm0:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#

NOTICE NOTHING AT ADDRESS 0x50
</pre>

<pre>
ENABLE QSFP0, OBSERVE IT IS AT ADDRESS 0x50

root@gdm0:~# echo 0 >> /sys/class/gpio/gpio378/value
root@gdm0:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: 50 -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#
root@gdm0:~# i2cdump -y 0 0x50
No size specified (using byte-data access)
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 0d 00 02 8e 00 00 01 00 00 05 55 00 05 00 00 00 ?.??..?..?U.?...
10: 00 00 00 00 00 00 1e b8 00 00 81 a0 00 00 00 00 ......??..??....
20: 00 00 12 12 00 01 00 01 00 01 0e e0 0f 20 0e e0 ..??.?.?.???? ??
30: 00 00 1f a0 1e 3b 1f 72 00 01 00 00 00 00 00 00 ..???;?r.?......
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
60: 00 00 00 00 00 00 00 00 00 00 1f 00 00 00 08 00 ..........?...?.
70: 00 00 00 00 00 00 00 ff ff ff ff ff ff ff ff 00 ................
80: 0d 00 0c 04 00 00 00 40 40 02 d5 05 67 00 00 96 ?.??...@@???g..?
90: 00 00 c8 00 46 49 4e 49 53 41 52 20 43 4f 52 50 ..?.FINISAR CORP
a0: 20 20 20 20 07 00 90 65 46 54 4c 34 31 30 51 44 ?.?eFTL410QD
b0: 34 43 20 20 20 20 20 20 41 20 42 68 07 d0 00 43 4C A Bh??.C
c0: 00 07 0f de 58 37 39 41 43 30 52 20 20 20 20 20 .???X79AC0R
d0: 20 20 20 20 32 32 30 33 30 39 20 20 3c 10 00 9e 220309 <?.?
e0: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@gdm0:~#
</pre>

== ethernet mac eeprom ==

* correct chip with 84-bit ethernet mac address

<pre>
root@cdm1:~# i2cdump 0 0x53
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
root@cdm1:~#

root@cdm1:~# i2cdump 0 0x5b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00 ????4??2?K?.?...
90: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c ..........??=?Q<
a0: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00 ????4??2?K?.?...
b0: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c ..........??=?Q<
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@cdm1:~#
</pre>

* wrong "602" chip with 64-bit IPv6 address

<pre>
root@cdm0:~# i2cdump 0 0x53
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
root@cdm0:~# i2cdump 0 0x5b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00 ???????Q?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e ........??=..??.
a0: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00 ???????Q?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e ........??=..??.
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@cdm0:~#
</pre>

= read ethernet mac address from i2c =

(this code is copied from uboot command line i2c code)

in uboot sources board/xilinx/common/board.c replace original function with this:

<pre>
// special code to read ethernet MAC address from the DS-DM-Rev0 board. K.O. Sep-2022

int zynq_board_read_rom_ethaddr(unsigned char *ethaddr)
{
struct udevice *bus;
int ret;
int busnum = 0;

ret = uclass_get_device_by_seq(UCLASS_I2C, busnum, &bus);
if (ret) {
printf("%s: No bus %d\n", __func__, busnum);
return ret;
}

int chip_addr = 0x5B;

struct udevice *dev;

ret = i2c_get_chip(bus, chip_addr, 1, &dev);
if (ret) {
printf("%s: Bus %d no chip 0x%02x\n", __func__, busnum, chip_addr);
return ret;
}

int dev_addr = 0x98;

unsigned char data[8];

ret = dm_i2c_read(dev, dev_addr, data, 8);

if (ret) {
printf("%s: Bus %d chip 0x%02x read error %d\n", __func__, busnum, chip_addr, ret);
return ret;
}

printf("%s: Bus %d chip 0x%02x addr 0x%02x read: 0x %02x %02x %02x %02x %02x %02x %02x %02x\n", __func__, busnum, chip_addr, dev_addr, data[0], data[1], data[2], data[3], data[4], data[5], data[6], data[7]);

// see Atmel-8807-SEEPROM-AT24MAC402-602-Datasheet.pdf

if (data[0] == 0) {
// eiu-48 chip
ethaddr[0] = data[2];
ethaddr[1] = data[3];
ethaddr[2] = data[4];
ethaddr[3] = data[5];
ethaddr[4] = data[6];
ethaddr[5] = data[7];
} else {
// eiu-64 chip
ethaddr[0] = data[0];
ethaddr[1] = data[1];
ethaddr[2] = data[2];
ethaddr[3] = data[5];
ethaddr[4] = data[6];
ethaddr[5] = data[7];
}

printf("%s: ethaddr %02x:%02x:%02x:%02x:%02x:%02x\n", __func__, ethaddr[0], ethaddr[1], ethaddr[2], ethaddr[3], ethaddr[4], ethaddr[5]);

return ret;
}
</pre>

also this should have worked if i2c_xxx() functions were enabled in uboot:
<pre>
i2c_set_bus_num(0);
i2c_probe(0x5b);
i2c_read(0x5b, 0x9a, ethaddr, 6);
</pre>

= read ethernet mac address from i2c (SHOULD WORK) =

from: https://stackoverflow.com/questions/43637540/is-there-linux-or-u-boot-support-to-read-a-mac-address-from-a-chip-at-startup

<pre>
#ethernet related setup
setup_eth=run readmac buildmac
#read mac address from eeprom
readmac=i2c dev 0; i2c read 50 FA.1 6 $loadaddr
#build the ethaddr variable
#not very nice, but does the job
buildmac=\
e=" "; sep=" " \
for i in 0 1 2 3 4 5 ; do\
setexpr x $loadaddr + $i\
setexpr.b b *$x\
e="$e$sep$b"\
sep=":"\
done &&\
setenv ethaddr $e
</pre>

= read ethernet mac address from i2c (DOES NOT WORK) =

this method does not work: inside board/xilinx/common/board.c:int zynq_board_read_rom_ethaddr(unsigned char *ethaddr), uclass_get_device_by_ofnode(UCLASS_I2C_EEPROM, eeprom, &dev) returns ENODEV and read from i2c does not happen. K.O. Sep-2022

Read:
* https://support.xilinx.com/s/question/0D52E00006hpKlsSAE/uboot-and-ethernet-mac-address-from-eeprom?language=en_US (last answer)
* https://github.com/Xilinx/u-boot-xlnx/blob/master/board/xilinx/common/board.c (zynq_board_read_rom_ethaddr())

Note:
* 0x5B is the i2c chip address
* 0x9A is the data offset inside the chip, see datasheet or i2c read dump above.

Edit:
* emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/u-boot/files/platform-top.h
<pre>
#include <configs/xilinx_zynqmp.h>
#include <configs/platform-auto.h>
//#define CONFIG_I2C_EEPROM
//#define CONFIG_SYS_I2C_EEPROM_ADDR 0x5b
//#define CONFIG_SYS_I2C_EEPROM_ADDR_OVERFLOW 0x0
#define CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET 0x9A
#error HERE!
</pre>
* emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi
<pre>
/include/ "system-conf.dtsi"
/ {
chosen {
xlnx,eeprom = &eeprom;
};
};

&i2c0 {
eeprom: eeprom@5b { /* u88 */
compatible = "atmel,24mac402";
reg = <0x5b>;
};
};
</pre>
* components/yocto/workspace/sources/u-boot-xlnx/configs/xilinx_zynqmp_virt_defconfig
<pre>
CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET=0x9A
</pre>

= RTC chip =

* DS3231 RTC chip
* FPGA connections:
<pre>
I2C SCL <- J-ENC A85 <- FPGA E17 <- XDC TP_S <- VHDL RTC_I2C_SCL output (10k pull-up to 3.3V)
I2C SDA <-> J-ENC A87 <-> FPGA D17 <-> XDC TP_S <-> VHDL RTC_I2C_SDA bidir (10k pull-up to 3.3V)
RTC_1Hz -> J-ENC C160 -> FPGA AH12 -> XDC "slow_io" -> VHDL RTC_1PPS input (10k pull-up to 1.2V)
RTC_32k -> J-ENC B129 -> FPGA AE3 -> XDC "free pin" -> VHDL RTC_32KHZ input (10k pull-up to 1.8V)
</pre>
* test notes
<pre>
FPGA registers:

rtc_i2c_clock_ie <= not register_data_in_block_0(14)(0);
rtc_i2c_clock_out <= register_data_in_block_0(14)(1);
rtc_i2c_data_ie <= not register_data_in_block_0(14)(4);
rtc_i2c_data_out <= register_data_in_block_0(14)(5);

ext_out(1) <= rtc_1pps_in;
ext_out(2) <= rtc_32khz_in;

register_data_out_block_0(13) <= x"000000" & rtc_i2c_data_ie & rtc_i2c_data_out & rtc_i2c_clock_ie & rtc_i2c_clock_out & rtc_1pps_in & rtc_32khz_in & rtc_i2c_data_in & rtc_i2c_clock_in;

After power us on the RTC chip:
i2c clock 3.3V
i2c data 3.3V
32kHz 0.778V see clock running with amplitude 0->1.8V (FPGA pullup) period about 30 usec (~33 kHz)
1pps 1.2V (FPGA pullup)

./test1.exe --read32 0x34
dsdm_read32[0x00000034] is 0x000000ab (171) -> 1010'1011 -> data_ie, clock_ie, 1pps, data_in, clock_in
./test1.exe --read32 0x38
dsdm_read32[0x00000038] is 0x00000000 (0)

./test1.exe --write32 0x38 0x01 -> clock oe
./test1.exe --read32 0x34
Ddsdm_read32[0x00000034] is 0x0000008a (138) -> 1000'1010 -> data_ie, 1pps, data_in, clock 0V, data 3.3V

./test1.exe --write32 0x38 0x10 -> data_oe
./test1.exe --read32 0x34
dsdm_read32[0x00000034] is 0x00000029 (41) -> 0010'1001 -> clock_ie, 1pps, clock_in, clock 3.3V, data 0V

root@dsdm:~# ./test_cdm_local.exe --rtc-read
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Read the RTC chip!
reg 0x00: 0 1 0 1 0 1 1 1 0x57
reg 0x01: 0 0 1 0 0 0 1 0 0x22
reg 0x02: 0 0 0 0 0 0 0 0 0x00
reg 0x03: 0 0 0 0 0 0 0 0 0x00
reg 0x04: 0 0 0 0 0 0 0 0 0x00
reg 0x05: 0 0 0 0 0 0 0 0 0x00
reg 0x06: 0 0 0 0 0 0 0 0 0x00
reg 0x07: 0 0 0 0 0 0 0 0 0x00
reg 0x08: 0 0 0 0 0 0 0 0 0x00
reg 0x09: 0 0 0 0 0 0 0 0 0x00
reg 0x10: 0 0 0 0 0 0 0 0 0x00
reg 0x11: 0 0 0 0 0 0 0 0 0x00
reg 0x12: 0 0 0 0 0 0 0 0 0x00
reg 0x13: 0 0 0 0 0 0 0 0 0x00
reg 0x14: 0 0 0 1 1 0 0 0 0x18
reg 0x15: 0 0 0 0 1 0 0 0 0x08
reg 0x16: 0 0 0 0 0 0 0 0 0x00
reg 0x17: 0 0 0 1 0 1 1 1 0x17
reg 0x18: 1 1 0 0 0 0 0 0 0xc0
seconds: 57, minutes: 22, hours: 00, day 0, date: 00-00-00, temp 23.75C
root@dsdm:~#
</pre>

= clock chip configuration =

file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt:

* VCO is 14 GHz
* Tvco is 71.43 ps
* N0 divider is 14, frequency is 1000 MHz
* out0 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz
* out1 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
* out2 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
* out3 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz is the feedback for zero delay

= clock chip monitoring =

from si5395-94-92-family.pdf:
* reg 0x1: page select, set to 0 or set to 5 to read 0x53F
* reg 0x2: 0x94
* reg 0x3: 0x53 -> device is a si5394
* reg 0xC: LOSXAXB
* reg 0xD: LOS and OOF for the 4 clock inputs
* reg 0xE: LOL and HOLD
* reg 0xF: CAL_PLL
* reg 0x11: sticky bits for reg 0xC
* reg 0x12: sticky bits for reg 0xD
* reg 0x13: sticky bits for reg 0xE
* reg 0x14: sticky bits for reg 0xF
* reg 0x1C: device reset
* reg 0x1E: low power, hard reset, SYNC
* reg 0x507: currently selected input clock
* reg 0x52A: input clock select
* reg 0x535: FORCE_HOLD
* reg 0x53F: HOLD_HIST_VALID and FASTLOCK_STATUS

= Install Xilinx tools =

* install Vivado 2020.2
<pre>
login at https://www.xilinx.com/myprofile.html
go to "Downloads"
go to archive,
find 2020.2
download Xilinx_Unified_2020.2_1118_1232_Lin64.bin
sh ./Xilinx_Unified_2020.2_1118_1232_Lin64.bin
banner window should open with spinner "downloading installation data"
"a newer version is available" -> say "continue"
next
"select install type" window:
provide email and password,
select "download image"
select directory /home/olchansk/Xilinx/Downloads/2020.2\
select "linux" and "full image"
next
download summary: space required 38.52 Gbytes
download
installation progress
downloading spinner, 16 M/s 47 minutes...
"download image has been created successfully". Ok.
check contents of /home/olchansk/Xilinx/Downloads/2020.2
ls -l /home/olchansk/Xilinx/Downloads/2020.2
total 67
drwxr-xr-x 2 olchansk users 9 Sep 1 16:22 bin
drwxr-xr-x 3 olchansk users 15 Sep 1 16:23 data
drwxr-xr-x 4 olchansk users 4 Sep 1 16:22 lib
drwxr-xr-x 2 olchansk users 644 Sep 1 16:22 payload
drwxr-xr-x 2 olchansk users 7 Sep 1 16:22 scripts
drwxr-xr-x 4 olchansk users 4 Sep 1 16:22 tps
-rwxr-xr-x 1 olchansk users 3256 Nov 18 2020 xsetup
daq13:2020.2$
./xsetup
spinned loading installation data
xilinx design tools 2022.1 now available -> say continue
"welcome" -> next
"select product" -> vivado -> next -> vivado hl system edition -> next
select devices: only zynq ultrascale+ mpsoc -> next
select destination: /opt/Xilinx (as root, mkdir /opt/Xilinx, chmod olchansk.users /opt/Xilinx)
install ...
complete
move /home/olchansk/Xilinx/Downloads/2020.2 to /daq/daqstore/olchansk/Xilinx/Downloads/
</pre>
* install petalinux 2020.2
<pre>
./xsetup
"a newer version is available" -> say "continue"
next
"select product to install" -> select Petalinux (Linux only) -> next
"select destination directory" -> select "/opt/Xilinx" (disk space required 2.64 GB) -> next
"summary" -> install ...
error about missing /tmp/tmp-something files
"installation completed successfully" (hard to dismiss, "ok" button is partially cut-off)
done?
I think it failed, /opt/Xilinx/PetaLinux/2020.2/bin is empty except for petalinux-v2020.2-final-installer.run
try to run it by hand, same error about /tmp/tmp-something files. strange...
notice it complains about "truncate", which truncate finds ~/bin/truncate, get rid of it,
try again
now complains about missing texinfo and zlib1g:i386
apt install texinfo -> ok
apt install zlib1g:i386 -> installs bunch of gcc stuff -> ok
try again
reports "already installed" -> delete /opt/Xilinx/.xinstall/PetaLinux_2020.2/, delete entries in ~/.Xilinx/registry/installedSW.xml
try again
success
</pre>
* install vivado 2022.1 and petalinux 2022.1 - everything is pretty much the same

= Petalinux =

* cd PetaLinux_GDM_CDM
* petalinux-config
* enable i2c MAC address and DHCP
<pre>
git clone https://bitbucket.org:/team-ds-dm/ds-dm-u-boot-xlnx.git
cd ds-dm-u-boot-xlnx
git checkout ds-dm-u-boot-xlnx

linux-components ->
uboot -> ext-local-src
external u-boot local source -> ds-dm-u-boot-xlnx (path to the customized uboot git repository)
</pre>
* enable DHCP
<pre>
Subsystem AUTO Hardware Settings -> Ethernet Settings
randomize MAC address -> NO
ethernet mac address -> leave empty
obtain ip address automatically -> YES
</pre>
* set hostname and product names
<pre>
Firmware Version Configuration ->
Host name -> "ds-dm"
Product name -> "Petalinux_GDM_CDM"
</pre>
* configure linux kernel
<pre>
petalinux-config -c kernel
</pre>
* enable NFS-Root
<pre>
petalinux-config
Image Packaging Configuration > Root File System Type -> set to NFS
Location of NFS root directory set to "/nfsroot"

petalinux-config -c kernel
Networking support > IP: kernel level configuration
enable DHCP, BOOTP, RARP
File systems > Network file systems > Root file systems on NFS
</pre>
* manually fix linux kernel command line:
<pre>
grep nfsroot PetaLinux_GDM_CDM/project-spec/configs/config
edit CONFIG_SUBSYSTEM_BOOTARGS_GENERATED to read
earlycon console=ttyPS0,115200 clk_ignore_unused panic=60 root=/dev/nfs nfsroot=/nfsroot/%s ip=dhcp rw
</pre>
* check configuration in
** PetaLinux_GDM_CDM/project-spec/configs/config
** PetaLinux_GDM_CDM/project-spec/configs/rootfs_config
** PetaLinux_GDM_CDM/components/plnx_workspace/device-tree/device-tree/system-conf.dtsi

= JTAG server =

localhost:3121

= ds20k block =

<pre>
module ds20k
(
// CLOCK INPUTs
input wire clk,
input wire reset, // pulse for power-up reset
input wire si5394_loln, // clock cleaner PLL is locked to selected input clock

// REGISTER_DATA

input wire [255:0] [31:0] register_data_in,
output reg [255:0] [31:0] register_data_out,
input wire register_we,
input wire register_re,

// GDM QSFP FIBER LINKS
input wire qsfp_tx_data_ready,
output reg [11:0] [15:0] qsfp_tx_data,
output reg [11:0] [1:0] qsfp_tx_data_is_k,
input wire [11:0] [15:0] qsfp_rx_data_error, // not sure what this is.
input wire [11:0] [15:0] qsfp_rx_data,
input wire [11:0] [1:0] qsfp_rx_data_is_k,

// CDM SFP FIBER LINKS
input wire sfp_tx_data_ready,
output reg [15:0] sfp_tx_data,
output reg [1:0] sfp_tx_data_is_k,
input wire sfp_rx_data_error, // single bit indicating that RX link is up and data is good.
input wire [15:0] sfp_rx_data,
input wire [1:0] sfp_rx_data_is_k,

// VX_RXs
input wire [3:0] vx1_rx_async,
input wire [3:0] vx2_rx_async,
input wire [3:0] vx3_rx_async,
input wire [3:0] vx4_rx_async,
input wire [3:0] vx5_rx_async,
input wire [3:0] vx6_rx_async,
input wire [3:0] vx7_rx_async,
input wire [3:0] vx8_rx_async,
input wire [3:0] vx9_rx_async,
input wire [3:0] vx10_rx_async,
input wire [3:0] vx11_rx_async,
input wire [3:0] vx12_rx_async,

// VX_TXs
output reg [2:0] vx1_tx_out,
output reg [2:0] vx2_tx_out,
output reg [2:0] vx3_tx_out,
output reg [2:0] vx4_tx_out,
output reg [2:0] vx5_tx_out,
output reg [2:0] vx6_tx_out,
output reg [2:0] vx7_tx_out,
output reg [2:0] vx8_tx_out,
output reg [2:0] vx9_tx_out,
output reg [2:0] vx10_tx_out,
output reg [2:0] vx11_tx_out,
output reg [2:0] vx12_tx_out,

// VX serial comunications
input wire [11:0] vx_rx_ser_in,
output reg [11:0] vx_tx_ser_out,

// LEMO INPUTs
input wire [4:1] ext_in_async, // direct connection to LEMO connectors, not clocked

// LEMO OUTPUTs
output reg [2:1] ext_out, // direct connection to LEMO connectors, not clocked

// FRONT PANEL LEDs
output reg [3:0] fp_led_out, // direct connection to LEMO connectors, not clocked
);
</pre>

= world view =

Main components:
* clock distribution - to ensure all digitizers run synchronously and waveform timestamp are easy to assemble intophysics events.
* global trigger and run control - all digitizers are triggered at the same time to record calibration events, pulser events and the run startup synchronization sequence.
* busy distribution - when some digitizers go busy and stop accepting triggers, physics events may become incomplete, this must be recorded and managed.
* hitmap distribution - to create an online picture of the detector for monitoring and for triggering.

== global clock distribution ==

Clock distribution ensures all digitizers run synchronously (on the same clock) and waveform timestamps are easy to assemble into physics events. The clock source is synchronized with GPS to ensure a metrologically validated clock frequency (tied to the SI definition of 1 second) and date stamp (via GPS IRIG-B date stamp). Individual digitizer waveform timestamp are reset at the begin of run to establish a valid time zero (run start time, date stamped with the GPS IRIG-B date stamp).

Base clock is 125 MHz (8 ns). Individual digitizers are synchronized at initialization (power-up) time. Because of incontrollable delays in the electronics (i.e. variations between individual PLL chips in each digitizer), waveform timestamps between individual digitizers may be off by 1-2 clocks (8-16 ns). After the system is running and is stable (thermally and electrically), timestamp waveform variation and jitter are less than 1 ns.

This is validated by feeding a common calibration pulser signal into analog inputs of each digitizer (at 100 Hz) and measuring (fitting) the pulse time in the waveform. With sufficient statistics, differences between waveform timestamp are measured with picosecond precision.

CDM PLL allows fine phase adjustment of the 62.5 MHz VX2746 clock in groups of 6 clocks (2 groups per CDM) with precision 71.43 ps.

Due to FPGA technology limitations (serial link hard IPs, PLLs, etc) the pattern of 1-2 clock delays between individual digitizers may change/shift from power up to power up. Significant additional work would be required to ensure this pattern is repeatable and stable after every power up and initialization. As mentioned before, after power up and initialization, variation between waveform timestamp is sub-nanosecond as measured by the analog calibration pulser.

Clock chain:
* GPS receiver (1pps clock reference and IRIG-B date stamp)
* LNGS GPS clock distribution (fiber link to converter box at the experiment)
* GPS 1pps to PRS-10 Rb atomic clock (10 MHz clock output)
* 10 MHz clock from PRS-10 to GDM PLL (extra low jitter oscillator, 125 MHz clock output)
* 125 MHz clock to GDM fiber link transmitters (2.5 Gbps line rate)
* CDM fiber link receiver and clock recovery (125 MHz clock)
* 125 MHz recovered clock to CDM PLL (extra low jitter oscillator, 62.5 MHz clock output)
* 62.5 MHz CDM clock to VX2745 PLL via LVDS line
* VX2745 PLL (125 MHz clock output)
* 125 MHz clock to digitizer chips, clock counter and waveform timestamp.

This is the normal mode of operation, with global clock synchronized with the GPS clock (metrologically referenced to SI 1 Hz definition).

If GPS signal is not available, degraded modes of operation are available (in order of worsening performance)
* PRS-10 Rb atomic clock synchronized with a battery-powered time-of-day chip
* PRS-10 Rb atomic clock synchronized with Internet time (NTP) assuming Internet network connection is up
* PRS-10 Rb clock standalone
* GDM internal oscillator

All degraded modes provide high precision low jitter 125 MHz system clock, but long term stability (seconds per day) will vary. (TBM)

In the absence of GPS IRIG-B date stamp, the battery-powered time-of-day chip will be used (TBI) with fallback to Internet time (NTP). Date stamp long term stability is expected to be better than 1 second per day (TBI, TBM).

Note:
* TBM: to be measured
* TBI: to be implemented

Current status (April 2025):
* full clock chain from GDM PLL to VX2745 digitizers is implemented and tested
* waveform timestamp variations are measured (see report from Marek)
* GDM external clock is validated (using VME-GRIFFIN-CDM high quality clock source module)
* PRS-10 10 MHz and 1pps connection to GDM is implemented and tested
* GPS receiver 1pps connection to PRS-10 is implemented and tested
* GPS receiver IRIG-B date stamp connection to GDM tested using a software decoder. Hardware decoder and connection to time slice marker, TBI.
* LNGS date stamp decoder and connection to time slice marker, TBI
* LNGS 1pps connection to PRS-10, not available
* operate vertical slize system from GPS, not available (GPS is in the det lab, vertical slice is in the daq lab).

== global trigger distribution ==

Global trigger distribution provides a synchronous waveform trigger to all VX2745, all CDM and all VX2745 are provided wiht the same data and are all triggered at the same time (without accounting for the 1-2 clock offset described in the clock section).

Global trigger is used for run control: the first trigger of a run resets the waveform timestamp counter, the CDM timestamp counter and the GDM timestamp counter. At the begin of run, GDM generates a sequence of triggers following an easy to identify pattern. If a VX2745 misses a trigger (for any reason), online and offline software can easily identify the tru run start time using this pattern.

After a run has started, global trigger can be generated by a programmable pulser, by an external LEMO signal (i.e. external calibration signal, laser Q-switch, etc) and by the VX2745 trigger hitmap.

Trigger sources (TRG):
* TRG pulser
* LEMO inputs
* hitmap trigger

Time slice marker sources (TSM):
* TSM pulser

Trigger chain, K-code and LVDS:
* TRG/TSM source
* TRG/TSM K-code from GDM to CDM
* CDM K-code to LVDS line 12 and 13 lines (TRG and TSM)

Latency from GDM LEMO input to CDM LVDS out: TBM

Trigger chain, packet trigger:
* GDM TRG/TSM source to TRG/TSM packet generator
* or GDM hitmap group trigger packet generator
* TRG/TSM packet from GDM to CDM
* TRG/TSM packet from CDM to VX2745 via LVDS line, 62.5 MHz, 8b10b encoding.

Latency from GDM LEMO input to VX2745 internal trigger: TBM

Notes:
TBM - to be measured
TBI - to be implemented

Current status (Jan 2026):
* LVDS TRG and TSM distribution is done and tested
* packet TRG, TRIG48 and TSM distribution from GDM to CDM is done and tested
* packet TRG, TRIG48 and TSM from CDM to VX2745 is done and tested
* packet TRIG48 processing in the VX is done and tested

== global busy distribution ==

If any VX2745 goes busy (for any reason) and stops accepting triggers, physics data may become incomplete. This situation needs to be recorded and managed.

Busy chain:
* VX2745 firmware busy logic (CAEN and Yair) generates the VX busy signal on LVDS line 9 (CDM VX_RX_2)
* CDM monitors VX busy
* CDM records busy transitions
* CDM computes the "CDM busy" as grand-OR of all VX busy
* CDM periodically transmits CDM busy status the GDM using special K-codes (NB: confirm transmission period)
* GDM receives CDM busy updates and maintains "CDM busy" for each CDM
* GDM records busy transitions
* GDM computes "GDM busy" as grand-OR of all CDM busy
* effectively, GDM busy is a grand-OR of all VX busy (with max delay, TBM)

In the pilot implementation, any VX busy stops the experiment by sending a veto signal to all VX2745 which stops them from accepting triggers.

Veto chain:
* GDM computes "GDM veto" as GDM busy
* GDM periodically transmits GDM veto status to all CDMs using special K-codes (NB: confirm transmission period)
* CDM receive GDM veto updates and maintain the "CDM veto" signal
* CDM computes "VX veto" as CDM veto
* CDM transmits VX veto to all VX2745 on LVDS pair 13
* VX2745 firmware (Yair) has VX veto block all physics triggers (NB: except for the time slice marker?)

Total delay from VX busy to VX veto (on the LVDS lines), TBM.

Notes:
* TBM - to be measured
* TBI - to be implemented

Current status (Jan 2026):
* full busy chain from VX to CDM to GDM is implemented, tested
* veto chain from GDM to CDM is implemented, tested
* veto chain from CDM to LVDS pair 13 is implemented, tested
* LVDS pair 13 veto inside the VX is implemented, tested
* TRG switched from LVDS pair 12 to TRG packet, testing completed
* TSM switched from LVDS pair 13 to TSM packet, testing completed
* fine tuning of busy chain, VX buffer levels, etc, test in progress

== vx hitmap and hitmap trigger ==

To monitor the system in real time and to generate group triggers, we use the VX hitmaps.

Each VX2745 has 64 channels, each channel has a trigger logic block (digital discriminator) that generates a waveform self-trigger and sends a hitmap bit to the CDM (64-bits per VX2745) as a data packet. CDM passes hitmap data packets to the GDM and also records non-empty hitmap packets. CDM MIDAS frontend passes non-empty hitmap packets to MIDAS for online monitoring and visualization. GDM collects all the hitmaps and generates a global trigger. Alternatively, it generates a group trigger TRIG48 with a 48-bit bitmap indicating which VX2745 should trigger a waveform.

Pilot implementation computes the hitmap trigger as a grand-OR of all hit maps.

Hitmap chain:
* VX2745 analog inputs to digitizers to digital trigger logic block, 1 bit per channel (CAEN, Yair)
* 64-bits of hitmap (1 bit/channel) are collected and periodically transmitted to the CDM (Sam De Jong) (NB: confirm transmission period: see CDM page table "VX hit map" column "period", 0xD1 clocks is 1672 ns, should be reduced to 1500 ns, TBI)
* VX hitmap packets are transmitted to the CDM on the LVDS line 10 (CDM VX_RX_1) at 62.5 MHz, 8b10b encoded.
* CDM receives hitmap packets (12 input data streams, 12-to-1 packet mux)
* CDM sends all hitmap packets to the GDM at 125 MHz, 16b20b encoded.
* non-empty hitmap packets are written to the CDM-to-CPU data FIFO (and via MIDAS frontend to MIDAS SYSTEM event buffer)
* grand-OR of VX hitmap packets (12*64 bits) can generate a CDM-local VX trigger
* GDM receives hitmap packets, computes "GDM hitmap trigger"
* if enabled, GDM hitmap trigger causes GDM trigger, which is processed as described in the global trigger section

Delay, latency and jitter from analog pulse on the VX2745 analog input to VX trigger on LVDS line 12, TBM.

TRIG48 VX group trigger:
* GDM receives hitmap packets, computes the 48-bit bitmap of which VX2745 modules should trigger
* how to specify this computation is not clear. a generic 12*64 MLU is impossible, insufficient RAM on the FPGA
* 48-bit bitmap is sent to all CDMs as a TRIG48 data packet
* CDM receives TRIG48 data packets from GDM and retransmits them to the VX2745 (all CDMs and all VXes see the same packets at the same time).
* VX2745 receives the 48-bit trigger bitmap, compares it with it's own 48-bit bitmap if any common bits are set, triggers waveform readout of all channels (there will be no more than 48 VX2745 in the experiment)

Pilot implementation of TRIG48 triggers on the CDM grand-or and generates a TRIG48 packet that contains a 48-bit bitmap stored in an AXI register.

Example uses of TRIG48:
* to trigger individual VX2745 modules: set the VX 48-bit map to the module number (vx05-0x1, vx06-0x2, vx07-0x4, etc), generate TRIG48 packet with a bitmap of VX modules that should trigger: 0xFFFF'FFFF'FFFF will trigger all modules, 0x1 will trigger only vx05, 0x2 will trigger only vx06, etc. (there is no more than 48 modules).
* to trigger groups of VX2745 modules: set the bitmap of group 1 modules to 0x1, set the bitmap of group 2 modules to 0x2, generate TRIG48 packet with bitmap 0x0001 to trigger only group 1 modules, 0x0002 to trigger only group 2 modules, 0x3 to trigger both groups.

Notes:
* TBM - to be measured
* TBI - to be implemented

Current status (Jan 2026):
* full chain (VX analog input to hitmap generator to CDM to GDM) is implemented, testing in progress
* GDM grand-or trigger is implemented
* VX hitmap packet generator, testing in progress (Yair firmware)
* CDM hitmap processing, implemented, tested
* GDM hitmap processing, implemented, tested
* pilot TRIG48 trigger generator in GDM, CDM, tested
* TRIG48 trigger in VX, implemented, tested
* MIDAS frontend readout of hitmap data from FPGA FIFO: implemented, testing in progress (FIFO code: Ian, done; integration and C++ code: KO)
* online monitoring and visualization in MIDAS: TBI

== firmware map ==

Note:
* red lines: clocks
* green lines: AXI/Avalon packet streams
* blue lines: serial data

[[File:GDM-CDM-VX.drawio.svg]]

== description ==

same thing, in words:

<pre>
detector
digitizer, 125 MHz
digital filter
digital discriminator
hit map, 64 bits at 125 MHz (could be 250 MHz, filter and discriminator clock)
packetizer, 64 bits -> id, timestamp, 8x 8-bit words, eop
8/10 serializer, 12.5 MHz parallel in, 125 MHz serial out
lvds line to CDM vx_rx
BBB: also VX busy to lvds line to CDM vx_rx, do not want to depend on serial comm for vx busy, do not want to inject async data into the hit map packet stream
--- CDM
lvds line capture in IBUF register, vxN_rx clock, 125 MHz clock, from PLL with adjustable phase, scan phase to find sweet spot
vx rx 10/8 deserializer, vxN_rx clock, stobes out parallel data every 10 clocks at 12.5 MHz
vx rx phase transfer from vxN_rx clock to main CDM clock, have 10 clocks for transfer to happen
from here everything is on the CDM main clock
vx rx packet adapter, 12.5 MHz 8-bit packets to 125 MHz 16-bit packets, internal FIFO to avoid data overrun
sfp tx mux, all VX packet streams into one CDM sfp tx packet stream
BBB: capture VX RX busy to CDM main clock (IBUF register), grand-or becomes vx_rx_busy
sfp tx packetizer (data,eop,vx_rx_busy -> data,k)
sfp tx 16/20 serializer, 16 bit at 125 MHz in, 2500 MHz serial out (20 bits at 125 MHz)
sfp tx fiber link to GDM, 2 Gigabits/sec
--- GDM
qsfp rx deserializer, 16 bit at 125 MHz output on main GDM clock
qsfp rx depacketizer (data,k -> packet data,eop; qsfp_rx_busy)
qsfp rx demux, hit map packets routed to GDM trigger logic block
GDM trigger logic block looks at hit map, generates yes/no trigger decision, encodes it as a trigger packet
TSM generator encodes GPS time data as a TSM packet
qsfp tx mux - trigger packets, tsm packets, etc to qsfp tx packet stream
qsfp tx packetizer (data,eop;trg,tsm,bsy -> data,k)
NB: the same tx data is sent to all 12 qsfp tx ports, to make sure we do not accidentally desync the CDMs.
qsfp tx 16/20 serializer, 16-bit at 125 MHz to 20-bit at 250 MHz to 2500 MHz serial out
qsfp tx fiber link to CDM, 2 Gigabits/sec
--- CDM
sfp rx 20/16 deserializer, 16-bit at 125 MHz
sfp rx depacketizer, (data,k -> data,eop;trg_in,tsm_in,bsy_in)
sfp rx demux (in reality, noop, all packets go to same place, vx tx)
vx tx mux (packets from GDM, packets with simulated waveforms from CDM midas frontend via AXI FIFO)
vx tx packet adapter 16-bit at 125 MHz to 8-bit at 12.5 MHz strobed every 10 clocks. NB: most important, this 10-clock strobe runs in sync between all CDMs!
vx tx phase transfer from main CDM clock to vxN_tx clock, have 10 clocks for transfer to happen
from here we run on the vxN_tx clock
vx txN serializer, 8-bit at 12.5 MHz to 125 MHz serial
vx_txN OBUF register, vxN_rx clock, 125 MHz clock, from PLL with adjustable phase, scan phase to find sweet spot
lvds line to vx
BBB: bsy_in from GDM is converted from pulse to level, goes out lvds line to vx, sync to vxN_tx clock
TTT: trg_in from GDM is a pulse, does out lvds line to vx, sync to vxN_tx clock.
NB: tsm is always a packet, bsy is always a signal (no packet), trg can be a packet or signal.
--- VX
lvds data captured by 125 MHz ADC sampling clock (CAEN base firmware logic)
lvds data connected to Yair's block
10/8 deserializer
depacketizer (data,k -> data,eop) to avoid accidental desync, we do not send any no trg, no tsm, no bsy K-codes.
demux
trigger packets go their way (4x 64 bit words of data go to event header: timestamps, hitmap data)
tsm packets go their way (4x 64 bit words of data go to tsm event header: timestamps, GPS time data)
BBB: bsy from lvds line stops waveform acquisition
TTT: trg from lvds line cause waveform acquisition, same as trg packet, but has no timestamp and other data attached to it.
</pre>

== CDM-VX connections ==

Note:
* rounded boxes are VX registers set from MIDAS

[[File:CDM-VX.drawio.svg]]

= Firmware registers =

== Block 0 ==

<pre>
0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
</pre>

== Block 0 GDM ==

* 0x0000+4*i - GDM AXI clock registers

<pre>
register_data_out_block_0(1) <= x"204D4447"; --GDM<space>
register_data_out_block_0(2) <= x"00000000"; --blank
register_data_out_block_0(3) <= x"00000000"; --blank

--Firmware Version
register_data_out_block_0(4) <= x"abcd0123"; --Firmware version name register
register_data_out_block_0(5) <= x"00000001"; --Firmware version number register
register_data_out_block_0(6) <= x"abcd0123"; --Firmware version register
register_data_out_block_0(7) <= x"00000001"; --Firmware version register

--Git Commit/Tag/Compile Date
register_data_out_block_0(8) <= x"0123abcd"; --Firmware Git info (0-3)
register_data_out_block_0(9) <= x"00000001"; --Firmware Git info (1)
register_data_out_block_0(10) <= x"00000001"; --Firmware Git info (2)
register_data_out_block_0(11) <= x"00000001"; --Firmware Git info (3)

--Simple loopback register for testing
register_data_out_block_0(12) <= register_data_in_block_0(12);

--Hardware registers (HW_ID_xDM, etc)
--register_data_out_block_0(13) <= x"0000000" & "000" & hardware_id;

--unspecified
set_reg_block_0_gen : for i in 14 to nregisters_block_0-1 generate
register_data_out_block_0(i) <= register_data_in_block_0(i);
end generate set_reg_block_0_gen;

register_data_out_block_0(13) <= x"000000" & rtc_i2c_data_ie & rtc_i2c_data_out & rtc_i2c_clock_ie & rtc_i2c_clock_out & rtc_1pps_in & rtc_32khz_in & rtc_i2c_data_in & rtc_i2c_clock_in;

rtc_i2c_clock_ie <= not register_data_in_block_0(14)(0);
rtc_i2c_clock_out <= register_data_in_block_0(14)(1);
rtc_i2c_data_ie <= not register_data_in_block_0(14)(4);
rtc_i2c_data_out <= register_data_in_block_0(14)(5);
</pre>

== Block 0 CDM ==

* 0x0000+4*i - CDM AXI clock registers:

<pre>
USR_ACCESS_7series_inst: USR_ACCESSE2
port map(-- CFGCLK => CFGCLK,
DATA => register_data_out_block_0(0)); -- where timestamp is contained
-- DATAVALID => DATAVALID);

--static values
register_data_out_block_0(1) <= x"204D4443"; --CDM<space>
register_data_out_block_0(2) <= x"00000000"; --blank
register_data_out_block_0(3) <= x"00000000"; --blank

--Firmware Version
register_data_out_block_0(4) <= x"abcd0123"; --Firmware version name register
register_data_out_block_0(5) <= x"00000001"; --Firmware version number register
register_data_out_block_0(6) <= x"abcd0123"; --Firmware version name register
register_data_out_block_0(7) <= x"00000001"; --Firmware version number register

--Git Commit/Tag/Compile Date
register_data_out_block_0(8) <= x"0123abcd"; --Firmware Git info (0-3)
register_data_out_block_0(9) <= x"00000001"; --Firmware Git info (1)
register_data_out_block_0(10) <= x"00000001"; --Firmware Git info (2)
register_data_out_block_0(11) <= x"00000001"; --Firmware Git info (3)

--Simple loopback register for testing
register_data_out_block_0(12) <= register_data_in_block_0(12);

--Hardware registers (HW_ID_xDM, etc)
--register_data_out_block_0(13) <= x"0000000" & "000" & hardware_id;

--unspecified
set_reg_block_0_gen : for i in 14 to nregisters_block_0-1 generate
register_data_out_block_0(i) <= register_data_in_block_0(i);
end generate set_reg_block_0_gen;

rx_clk_sel <= register_data_in_block_0(12)(0); -- rx_clk select 0:clk_cc_out0, 1:rx_clk_mgt

rtc_i2c_clock_ie <= not register_data_in_block_0(14)(0);
rtc_i2c_clock_out <= register_data_in_block_0(14)(1);
rtc_i2c_data_ie <= not register_data_in_block_0(14)(4);
rtc_i2c_data_out <= register_data_in_block_0(14)(5);

register_data_out_block_0(13) <= x"000000" & rtc_i2c_data_ie & rtc_i2c_data_out & rtc_i2c_clock_ie & rtc_i2c_clock_out & rtc_1pps_in & rtc_32khz_in & rtc_i2c_data_in & rtc_i2c_clock_in;
</pre>

== Block 1 GDM ==

* 0x0800+4*i - GDM registers:

<pre>
clk_config_pro : process (read_write_clk)
begin
if rising_edge (read_write_clk) then
CLK_IN_SEL_LS <= clk_config_vec(1 downto 0);
CLK_EXT_SEL_LS <= clk_config_vec(2);
CLK_RSTn_LS <= not clk_config_vec(3);
clk_config_vec(4) <= CLK_LOSXTn_LS;
clk_config_vec(5) <= CLK_LOLn_LS;
clk_config_vec(6) <= CLK_INTn_LS;
clk_config_vec(7) <= '1';

clk_config_vec(3 downto 0) <= register_data_in_block_1(0)(3 downto 0);
register_data_out_block_1(0)( 7 downto 0) <= clk_config_vec(7 downto 0);
register_data_out_block_1(0)(31 downto 8) <= (others =>'0');
register_data_out_block_1(1) <= (others =>'0'); --free
end if;
end process clk_config_pro;

register_data_out_block_1(2) <= register_data_in_block_1(2);
register_data_out_block_1(3) <= register_data_in_block_1(3);
register_data_out_block_1(4)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(5)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(6)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(7)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(8) <= register_data_in_block_1(8);
register_data_out_block_1(9)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(10) <= (others => '0');
register_data_out_block_1(11) <= (others => '0');

frequency_measure_clks <= read_write_clk & clk_50MHz & mgt_data_clk & mgt_ref_clk_raw; -- & mgt_data_clk & mgt_ref_clk_raw;
register_data_out_block_1(12) <= register_data_out_block_1(14); --MGT rx and tx reference clocks are the same
register_data_out_block_1(13) <= register_data_out_block_1(15); --MGT rx and tx data clocks are the same
frequency_measure_inst : frequency_measure
generic map(register_clk_frequency => 100000000,
max_frequency => 300000000,
nfrequency_measurments => nfrequency_measure_clks)
port map(i_register_clk => read_write_clk,
i_clks => frequency_measure_clks,
i_clk_locks => (others =>'1'),
o_frequencies => register_data_out_block_1(14+nfrequency_measure_clks-1 downto 14)); --see register_data_out_block_1(12,13) above

zero_spare_link_registers : for i in 14+nfrequency_measure_clks to nregisters_block_1-1 generate
register_data_out_block_1(i) <= (others => '0');
end generate zero_spare_link_registers;

gdm_link_interface_inst : gdm_link_interface
generic map(enable_rxslide => '1',
nquads => nquads,
nlinks => nlinks)
port map(i_mgt_ref_clk_p => mgt_b_ref_clk_p,
i_mgt_ref_clk_n => mgt_b_ref_clk_n,
i_mgt_rx_n => mgt_b_rx_n,
i_mgt_rx_p => mgt_b_rx_p,
o_mgt_tx_n => mgt_b_tx_n,
o_mgt_tx_p => mgt_b_tx_p,
--raw, pre-pll, mgt clock for frequency measurement
o_mgt_ref_clk_raw => mgt_ref_clk_raw,
--free running clock and reset
i_mgt_free_running_clk => clk_50MHz,
i_mgt_rst => register_data_in_block_1(2)(0),
i_rx_slide_trigger => register_data_in_block_1(3)(nlinks-1 downto 0),
--data interface for the links
o_data_clk => mgt_data_clk,
o_mgt_data_link_array => mgt_link_data_to_processing,
i_mgt_data_link_array => mgt_link_data_from_processing,
--status ports
o_link_power_good => register_data_out_block_1(4)(nlinks-1 downto 0),
o_tx_link_up => register_data_out_block_1(5)(nlinks-1 downto 0),
o_rx_link_up => register_data_out_block_1(6)(nlinks-1 downto 0),
o_rx_error => register_data_out_block_1(7)(nlinks-1 downto 0),
i_link_down_latched_rst => register_data_in_block_1(8)(0),
o_link_down_latched => register_data_out_block_1(9)(nlinks-1 downto 0),
--debug
i_debug_clk => read_write_clk,
o_debug => mgt_debug_data);
</pre>

<pre>
#### Register Block 1 (CDM)
Clock Chip and MGT Link, Control and Status Registers
| Register | Axi Address | Description | note
|:-|:-|:-|:-|
| 0 | 0x80011000 | Clock Chip Status and Control | CLK_IN_SEL_LS(1-0)(read-only), CLK_EXT_SEL_LS(2), CLK_RSTn_LS(3), CLK_LOSXTn_LS(4), CLK_LOLn_LS(5), CLK_I\
NTn_LS(6)
| 1 | 0x80011004 | free for use | |
| 2 | 0x80011008 | MGT Link Reset | mgt_rst(0), Reset for RX Error Latched(1)|
| 3 | 0x8001100c | MGT rx slide control | not needed or used |
| 4 | 0x80011010 | link_up_and_running(10) tx_link_up_and_running(9), tx_sending_data(8), tx_link_up(7), rx_link_up_and_running(6), rx_error(5), rx_receivi\
ng_data(4), rx_link_up(3), link_power_good(2), sfp_rx_los(1), sfp_mod_absent(0) | Should be 0x7DC |
| 5 | 0x80011014 | reserved debug register | |
| 6-11 | ... | free for use | |
| 12 | 0x80011030 | mgt_rx_ref_clk_raw | ~0x7735940 (125 MHz) |
| 13 | 0x80011034 | rx_clk | ~0x7735940 (125 MHz) |
| 14 | 0x80011038 | mgt_tx_ref_clk_raw | ~0x7735940 (125 MHz) |
| 15 | 0x8001103c | tx_clk | ~0x7735940 (125 MHz) |
| 16 | 0x80011040 | clk_50MHz | 0x2faf080 (50 MHz) |
| 17 | 0x80011044 | Register_Block1_clk | 0x5f5e100 (100 MHz) |
| 18-20 | ... | free for use | |
| 21 | 0x80011054 | free for use | |
</pre>

=== Register 0x1000 ===

<pre>
printf("0x1000 SFP c.c. status: 0x%08x\n", reg1000);
printf(" CLK_IN_SEL_LS 0x%x\n", (reg1000>>0) & 3);
printf(" CLK_EXT_SEL_LS %d\n", (reg1000>>2) & 1);
printf(" CLK_CLK_RSTn_LS %d\n", (reg1000>>3) & 1);
printf(" CLK_LOSXTn_LS %d\n", (reg1000>>4) & 1);
printf(" CLK_LOLn_LS %d\n", (reg1000>>5) & 1);
printf(" CLK_INTn_LS %d\n", (reg1000>>6) & 1);
printf("0x1008 SFP link reset: 0x%08x\n", dev->dsdm_read32(0x1008));
</pre>

=== Register 0x1010 ===

<pre>
printf("0x1010 SFP link status: 0x%08x\n", reg1010);
printf(" sfp_mod_absent_N %d\n", (reg1010>>0) & 1);
printf(" sfp_rx_los_N %d\n", (reg1010>>1) & 1);
printf(" link_power_good %d\n", (reg1010>>2) & 1);
printf(" rx_link_up %d\n", (reg1010>>3) & 1);
printf(" rx_receiving_data %d\n", (reg1010>>4) & 1);
printf(" rx_error %d\n", (reg1010>>5) & 1);
printf(" rx_lnk_up_and_running %d\n", (reg1010>>6) & 1);
printf(" tx_link_up %d\n", (reg1010>>7) & 1);
printf(" tx_sending_data %d\n", (reg1010>>8) & 1);
printf(" tx_link_up_and_running %d\n", (reg1010>>9) & 1);
printf(" link_up_and_running %d\n", (reg1010>>10) & 1);

</pre>

=== Register 0x1014 ===

<pre>
printf("0x1014 SFP link data: 0x%08x\n", reg1014);
printf(" rx_data 0x%04x\n", (reg1014>>0) & 0xFFFF);
printf(" k28p1_k28p5 %d\n", (reg1014>>16) & 1);
printf(" rx_ctrl0 0x%01x\n", (reg1014>>17) & 3);
printf(" rx_ctrl1 0x%01x\n", (reg1014>>19) & 3);
printf(" rx_ctrl3 0x%01x\n", (reg1014>>21) & 1);
printf(" tx_state 0x%01x\n", (reg1014>>22) & 3);
printf(" rx_state 0x%01x\n", (reg1014>>24) & 0xF);
printf(" rx_receiving_data %d\n", (reg1014>>28) & 1);
printf(" rx_link_up %d\n", (reg1014>>29) & 1);
printf(" rx_error %d\n", (reg1014>>30) & 1);
printf(" rx_link_rst %d\n", (reg1014>>31) & 1);
</pre>

* see cdm_link_establishment.vhd
* tx_state:
<pre>
0 = tx_idle
1 = waiting_for_rx_link
2 = sending_k28p1_k28p5
3 = sending_data
</pre>

* rx_state:
<pre>
0 = rx_idle
1 = waiting_for_ks
2 = rx_retry
3 = rx_up_switch_tx_clk_source
4 = waiting_for_data
5 = receiving_data
6 = rx_receive_error
</pre>

== Block 2 GDM ==

* 0x1000+4*i - GDM registers:

<pre>
register_data_out_block_2(0) <= register_data_in_block_2(0);
GDM_link_data_processing_inst : GDM_link_data_processing
generic map(mgt_data_frequency => 125000000)
port map(i_register_clk => read_write_clk,
i_rst => register_data_in_block_2(0)(0),
i_data_mode => register_data_in_block_2(0)( 9 downto 8),
i_fixed_pattern_data => register_data_in_block_2(0)(31 downto 16),
o_error_time => register_data_out_block_2(1),
o_error_count => register_data_out_block_2(13 downto 2),
--link data interface
i_data_clk => mgt_data_clk,
i_mgt_data_link_array => mgt_link_data_to_processing,
o_mgt_data_link_array => mgt_link_data_from_processing,
--data interface
o_data_clk => data_clk,
o_tx_data_ready => tx_data_ready,
i_tx_data_is_k => tx_data_is_k,
i_tx_data => tx_data,
o_rx_data_error => rx_data_error,
o_rx_data_is_k => rx_data_is_k,
o_rx_data => rx_data);
</pre>

== Block 2 CDM ==

* 0x1000+4*i - CDM registers:
* reg 0
<pre>
read bits
7..0 - clk_config_vec
1,0 - CLK_IN_SEL_LS
2 - CLK_EXT_SEL_LS
3 - disconnected CLK_RSTn_LS
4 - CLK_LOSXTn_LS
5 - CLK_LOLn_LS
6 - CLK_INTn_LS
7 - tx_clock_status
31..8 - 0
</pre>
<pre>
write bits
3..2 - clk_config_vec[3..2]
</pre>
* reg 1 - 0
* reg 2 - RW
<pre>
bit
0 - cdm_link_interface i_mgt_rst
1 - cdm_link_interface i_link_down_latched_rst
</pre>
* reg 3 - RW
<pre>
bit
0 - cdm_link_interface i_rx_slide_trigger
</pre>
* reg 4 - RO
<pre>
bit
0 - sfp_mod_present
1 - sfp_rx_los
2 - link_power_good
3 - rx_link_up
4 - rx_receiving_data
5 - rx_error
6 - rx_link_up_and_running
7 - tx_link_up
8 - tx_sending_data
9 - tx_link_up_and_running
10 - link_up_and_running
</pre>
* reg 5 - RO - CDM SFP debug_data
* reg 6 - 0
* reg 7 - 0
* reg 8 - 0
* reg 9 - 0
* reg 10 - 0
* reg 11 - 0
* reg 12 - frequency AXI clock
* reg 13 - freq clk_50MHz
* reg 14 - freq clk_100MHz
* reg 15 - freq clk_200MHz
* reg 16 - freq clk_xo_125 - DS-DM 125 MHz oscillator
* reg 17 - freq clk_cc_out0 - C.C. 125 MHz output
* reg 18 - CDM SFP mgt_rx_ref_clk_raw
* reg 19 - CDM SFP mgt_tx_ref_clk_raw
* reg 20 - rx_clk_mgt, 125 MHz CDM SFP RX recovered clock
* reg 21 - rx_clk, 125 MHz, switched between clk_xo_125 and rx_clk_mgt
* reg 22 - tx_clk, 125 MHz
* reg 23 - spare

== Block 3 GDM ==

0x3000+4*i : DS20k registers

== Block 4 CDM ==

* 0x2800+4*i - CDM SFP data link registers:
* 0 - R/W
** bit 0 - 0x1 - reset
** bits 7..1 - not used
** bits 9..8 - data_mode: 00: data from ds20k block, 01: tx_data_cnt, 10: tx_data_prbs, 11: tx_fixed_pattern_data
** bits 15..10 - not used
** bits 31..16 - fixed pattern data
* 1 - RO - o_error_time
* 2 - RO - o_error_count
* 3, etc - not used

== Block 6 CDM ==

* 0x3000+4*i : DS20k registers

== Block 7 CDM ==

VX_TX and VX_RX delays, i=0..11
* 0x3800+4*i - VX_TX delays
* 0x3830+4*i - VX_RX delays

VX_TX delay data (2 bits):
* 4 values: 0,1,2,3 corresponding to the 4 phases of the 125 MHz clock rx_clk and rx_clk shifted by 90 deg

VX_RX delay data (32 bits):
* 0x8000'0000: reset IDELAYE3, follow by write of 0
* 0x0001'xxxx: load the IDELAYE3 9-bit tap counter, follow by write of 0x0000'xxxx
* 16 low bits have:
* bits 0..8 are the 9 bits of IDELAY3 tap counter (~1.1ns for all 511 delays)
* bit 9 is the 500 MHz DDR phase (1 ns)
* bits 10..15 are 16 bits of 500 MHz delay (2 ns step)
* numerically increasing 16-bit delay values are carefully arranged to produce linearly increasing delay (there is some overlap between IDELAYE3 taps above ~tap 500 (or so, IDELAYE3 is uncalibrated) and flip of 500 MHz DDR phase

== DS20k registers ==

=== DS20k block register map ===

* busybox devmem 0x80013000 32

Note: R=readable, W=writable, L=latched by CMD_LATCH, B=reset on begin of run, Z=reset by CMD_LATCH

<pre>
reg | version | xDM | xx | description
0 | 0x20230731 | ALL | RO | ds20k version
0 | 0x20240118 | ALL | RW | ds20k version and command
1 | 0x20230731 | ALL | RW | scratch read/write register
2 | 0x20230731 | ALL | RW | configure inputs and outputs
3 | 0x20230731 | ALL | RW | FP_LED mux
4 | 0x20230731 | ALL | RW | EXT_OUT mux
5 | 0x20230731 | ALL | RO | VX_RX state
6 | 0x20230731 | ALL | RO | VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state
7 | 0x20230731 | ALL | RW | LED_OUT, EXT_OUT, VX_TX outputs
8 | 0x20230731 | ALL | RW | VX_TX mux and config
9 | 0x20230731 | ALL | RW | trg and tsm source
10 | 0x20231013 | ALL | RO | status register
11 | 0x20230731 | ALL | ROLB | trg counter
12 | 0x20230731 | ALL | ROLB | tsm counter
13 | 0x20240814 | ALL | RO | GPS 1pps period, 125 MHz
14 | 0x20240814 | ALL | RO | Ru clock 1pps period, 125 MHz
15 | 0x20230811 | CDM | RO | SFP RX status
16 | 0x20230811 | CDM | RW | SFP TX control
17 | 0x20230811 | GDM | RO | QSFP RX data 0, 1
18 | |. |. | 2, 3
19 | |. |. | 4, 5
20 | |. |. | 6, 7
21 | | |. | 8, 9
22 | |. |. | 10, 11
23 | 0x20230811 | GDM | RW | QSFP TX control
24 | 0x20231013 | ALL | RW | trigger pulser period
25 | 0x20231013 | ALL | RW | trigger pulser burst control
26 | 0x20231013 | ALL | RW | tsm pulser period
27 | 0x20231204 | ALL | RW | data fifo CPU to FPGA
28 | 0x20231204 | ALL | RW | packet loopback control
28 | 0x20240118 | ALL | RW | data fifo FPGA to CPU
29 | 0x20231208 | GDM | RW | bitmap of active qsfp ports
29 | 0x20240118 | ALL | RW | packet loopback control
29 | 0x20240510 | ALL | RW | packet routing
30 | 0x20231208 | GDM | RO | qsfp link status ports 0..7
31 | 0x20231208 | GDM | RO | qsfp link status ports 8..11
32 | 0x20240118 | GDM | RW | enabled QSFP ports
33 | 0x20240118 | CDM | RW | enabled VX ports
34,35 | 0x20240118 | ALL | ROB | time stamp 64 bits
36,37 | 0x20240118 | ALL | RO | old time stamp
38,39,40 | 0x20240118 | CDM | ROLZ | vx_bsy_counter_latched | VX busy counters
41,42,43 | 0x20240118 | GDM | ROLZ | QSFP busy counters
44 | 0x20240118 | CDM | ROLZ | cdm_bsy_up_counter and cdm_bsy_pulse_counter
45 | 0x20240118 | CDM | ROLZ | cdm_veto_pulse_counter and cdm_veto_up_counter
46 | 0x20240118 | GDM | ROLZ | gdm_bsy_pulse_counter and gdm_bsy_up_counter
47 | 0x20240118 | GDM | ROLZ | gdm_bsy_refresh_counter
48 | 0x20240118 | GDM | ROLZ | gdm_veto_up_counter and gdm_veto_pulse_counter
49,50,51 | 0x20240424 | CDM | RO | VX RX serial data monitor, 8 bits per VX port
52,53 | 0x20240424 | CDM | RO | VX serial link status, 4 bits per VX port
54 | 0x20240424 | CDM | RO | VX TX serial data monitor
55 | 0x20240430 | ALL | RO | QSFP, SFP and VX link loss counters
56 | 0x20240510 | ALL | RO | sfp_rx_packet_counter
57 | same | ALL | RO | sfp_tx_packet_counter
58 | same | ALL | RO | qsfp_rx_packet_counter[0]
59 | same | ALL | RO | qsfp_tx_packet_counter
60 | same | ALL | RO | vx_rx_packet_counter[0]
61 | same | ALL | RO | vx_tx_packet_counter
62 | 0x20240719 | CDM | RO | cdm_hitmap_period, ports 0, 1
63 | same | CDM | RO | ports 2, 3
64 | same | CDM | RO | ports 4, 5
65 | same | CDM | RO | ports 6, 7
66 | same | CDM | RO | ports 8, 9
67 | same | CDM | RO | ports 10, 11
62 | 0x20260401 | CDM | ROL | vx_rx_hitmap_period latched by cmd_latch, VX ports 0, 1
63 | same | CDM | ROL | ports 2, 3
64 | same | CDM | ROL | ports 4, 5
65 | same | CDM | ROL | ports 6, 7
66 | same | CDM | ROL | ports 8, 9
67 | same | CDM | ROL | ports 10, 11
68 | 0x20240814 | ALL | RW | GPS control and status
69 | 0x20241104 | CDM | ROLB | vx_tx_trg_packet_counter | counter of TRG packets CDM->VX
70 | 0x20241104 | CDM | ROLB | vx_tx_tsm_packet_counter | counter of TSM packets CDM->VX
71 | 0x20241104 | ALL | RO | packet error bits
72,73 | 0x20241104 | CDM | ROL | vx_tx_trg_data_latched[63:0] trigger packet data
74,75 | 0x20241104 | CDM | ROL | vx_tx_tsm_data_latched[63:0] tsm packet data (truncated to 64 bits)
76 | 0x20241104 | CDM | ROLB | cdm_hitmap_trigger_counter_latched, counter of hitmap triggers generated in the CDM
77,78,79 | 0x20241104 | CDM | ROL | vx_rx_hitmap_data_latched | VX_RX 80 bits of HITMAP packet from VX
80 | 0x20241209 | CDM | RO | vx_lvds_rx_link_status | VX LVDS link status
81 | 0x20241209 | CDM | RO | vx_lvds_rx_trg_data | VX LVDS TRG packet first 32 bits
81 | 0x20251010 | CDM | - | not used | not used
82 | 0x20241209 | CDM | RO | vx_lvds_rx_tsm_data | VX LVDS TSM packet first 32 bits
82 | 0x20251010 | CDM | - | not used | not used
83 | 0x20241209 | CDM | ROL | vx_lvds_rx_decoded_packet_counter_latched | VX LVDS decoded packet counter
84 | 0x20241209 | CDM | ROL | vx_lvds_rx_packet_error_counter_latched | VX LVDS packet error counter
85 | 0x20250417 | CDM | ROLB | vx_tx_trg_counter | VX LVDS TRG counter (VX LVDS pair 12)
86 | 0x20250417 | CDM | ROLB | vx_tx_tsm_counter | VX LVDS TSM counter (VX LVDS pair 13)
87 | 0x20250417 | CDM | ROLB | vx_tx_veto_counter | VX LVDS Veto counter (VX LVDS pair 13)
88 | 0x20250422 | GDM | ROLB | trg_in_lost_counter | counter of TRG killed by gdm_busy
89 | 0x20250422 | ALL | RO | fifo_from_fpga_lost_packet_counter | counter of packets lost due to FIFO full
90 | 0x20250428 | CDM | ROL | vx_bsy_status_latched | status of VX busy, CDM busy, CDM veto
91 | 0x20250428 | CDM | ROL | vx_hitmap_status_latched | status of VX hitmap
92 | 0x20250428 | GDM | ROL | cdm_bdy_status_latched | status of CDM busy, GDM busy, GDM veto
93 | 0x20250508 | GDM | ROL | gdm_hitmap_status_latched | status of GDM hitmap
94 | 0x20250508 | GDM | ROLB | gdm_hitmap_trigger_counter_latched | counter of hitmap triggers generated in the GDM
95 | 0x20250512 | GDM | ROLB | gdm_trig48_counter_latched | counter of TRIG48 trigger packets
96 | 0x20250512 | GDM | ROLB | gdm_trig48_lost_counter_latched | counter of TRIG48 triggers lost to busy
97 | 0x20250512 | CDM | ROLB | vx_tx_trig48_packet_counter_latched | counter of TRIG48 packets from CDM to VX
98, 99 | 0x20250512 | GDM | RW | trig48_ctrl48, trig48_ctrl16 | GDM TRIG48 test registers
100,101 | 0x20251008 | CDM | ROL | vx_lvds_rx_trig48_{data,bitmap48}_latched | VX LVDS TRIG48 packet first 32 bits, bitmap48 first 32 bits
102 | 0x20251008 | ALL | RO | cmd_counter | counter of reg0 commands
103,104,105 | 0x20260104 | CDM | ROL | vx_rx_hitmap_vx_id_latched | VX ID from VX hitmap packets, VX ports 0..11, in order, latched by cmd_latch
</pre>

=== Register 0 0x80013000 ds20k version ===

on read: ds20k version 0xYYYYMMDD

on write:
* 0 - noop - as of version 0x20240118, write a zero after writing a command
* 1 - cmd_reset - reset logic to good state
* 2 - cmd_arm_ts - arm timestamp reset
* 3 - cmd_trg - issue a trigger
* 4 - cmd_tsm - issue a tsm
* 5 - cmd_vx_rx_reset - reset the VX receive path
* 6 - cmd_vx_tx_reset - reset the VX transmit path
* 7 - cmd_gdm_trig48 - generate a TRIG48 trigger and data packet (GDM)
* 8 - cmd_trg_pulser_reset - reset the trigger pulser
* 9 - cmd_tsm_pulser_reset - reset the tsm pulser
* 10 - cmd_bor_start - start begin-of-run trigger sequence
* 11 - cmd_bor_clear - after run has started, clear begin-of-run status bits
* 12 - cmd_latch - latch counters & etc into AXI registers for coherent readout
* 13 - cmd_cdm_hitmap_trigger - generate a hitmap trigger and data packet (CDM)

=== Register 1 0x80013004 scratch ===

scratch read-write register

=== Register 2 0x80013008 input and output config ===

<pre>
bit | version | fpga name | description
0 | ALL | lemo_enable | enable LEMO input 1
1 | | | 2
2 | | | 3
3 | | | 4
4 | ALL | lemo_invert | invert LEMO input 1
5 | | | 2
6 | | | 3
7 | | | 4
9 | ALL | ext_out_disable | disable LEMO output 1
10 | | | 2
11 | ALL | ext_out_invert | invert LEMO output 1
12 | | | 2
</pre>

=== Register 3 0x8001300C FP_LED control ===

<pre>
wire [15:0] led_out_mux_sel = register_data_in[3][15:0];
wire [3:0] led_out_invert = register_data_in[3][19:16];
</pre>

<pre>
led_out_mux_sel is 4 groups (one per LED) of 4 bits (choice 0..15):
mux | version | fpga name | description
0 | ALL | | power on default
1 | ALL | led_out_reg | register 7 bits
2 | 0x20231013 | pll_locked | clock chip PLL is locked
3 | 0x20231013 | sfp_link_status | SFP link is good
3 | 0x20240118 | sfp_link_status or qsfp_tx_link_rx_status | SFP/QSFP link is good
4 | ALL | lemo_in_sync[1] | LEMO input 1
5 | ALL | lemo_in_sync[2] | LEMO input 2
6 | ALL | lemo_in_sync[3] | LEMO input 3
7 | ALL | lemo_in_sync[4] | LEMO input 4
8 | ALL | ext_out[1] | LEMO output 1
9 | ALL | ext_out[2] | LEMO output 2
A | 0x20231013 | trg_in | trigger
B | 0x20231013 | tsm_in | time slice marker
C | 0x20240118 | gdm_bsy | GDM busy: OR of all CDM busy
D | 0x20240118 | cdm_bsy | CDM busy: OR of all VX busy
E | 0x20240118 | cdm_veto | GDM busy -> GDM veto -> CDM veto -> VX trigger veto
F | ALL | | fixed logic level 1
</pre>

=== Register 4 0x80013010 LEMO OUT control ===

<pre>
wire [7:0] ext_out_mux_sel = register_data_in[4][7:0];
</pre>

<pre>
ext_out_mux_sel is 2 groups (one per LEMO) of 4 bits (choice 0..15):

mux | version | fpga name | description
0 | ALL | | power on default
1 | ALL | ext_out_reg | register 7
2 | ALL | trg_pulser | pulser trigger
3 | 0x20240724 | vx1_tx_out[2] | vx1 serial data out
4 | 0x20240724 | vx1_rx[1] | vx1 serial data in
5 | 0x20240724 | vx_rx_iob[0] | vx1 serial data in captured by IOB register
6 | ALL | lemo_in_async[1]| test synchronizer
7 | ALL | lemo_in_sync[1] | test synchronizer
8 | 0x20240724 | trg_in_pulse | trigger signal
9 | 0x20240724 | tsm_in_pulse | time slice marker signal
A | 0x20240118 | cdm_bsy | CDM busy from VX
B | 0x20240118 | gdm_bsy | GDM busy from CDM
C | 0x20240118 | cdm_veto | veto from GDM to CDM to VX
D | 0x20240118 | vx1_rx[1] | serial data VX to CDM
E | not used (sink)
F | ALL | 1 | fixed logic level 1
</pre>

=== Register 5 0x80013014 VX_RX status ===

<pre>
assign register_data_out[5] = {
vx8_rx[3], vx8_rx[2], vx8_rx[1], vx8_rx[0],
vx7_rx[3], vx7_rx[2], vx7_rx[1], vx7_rx[0],
vx6_rx[3], vx6_rx[2], vx6_rx[1], vx6_rx[0],
vx5_rx[3], vx5_rx[2], vx5_rx[1], vx5_rx[0],
vx4_rx[3], vx4_rx[2], vx4_rx[1], vx4_rx[0],
vx3_rx[3], vx3_rx[2], vx3_rx[1], vx3_rx[0],
vx2_rx[3], vx2_rx[2], vx2_rx[1], vx2_rx[0],
vx1_rx[3], vx1_rx[2], vx1_rx[1], vx1_rx[0]
};
</pre>

=== Register 6 0x80013018 VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state ===

<pre>
assign register_data_out[6] = {
vx2_tx_out[2], vx2_tx_out[1], vx2_tx_out[0], vx1_tx_out[2],
vx1_tx_out[1], vx1_tx_out[0], ext_out[2], ext_out[1],
fp_led_out[3], fp_led_out[2], fp_led_out[1], fp_led_out[0],
ext_in_lv[4], ext_in_lv[3], ext_in_lv[2], ext_in_lv[1],
vx12_rx[3], vx12_rx[2], vx12_rx[1], vx12_rx[0],
vx11_rx[3], vx11_rx[2], vx11_rx[1], vx11_rx[0],
vx10_rx[3], vx10_rx[2], vx10_rx[1], vx10_rx[0],
vx9_rx[3], vx9_rx[2], vx9_rx[1], vx9_rx[0]
};
</pre>

<pre>
bit | version | fpga name | description
0 | ALL | vx9_rx | VX_RX
1 | | |
2 | | |
3 | | |
4 | ALL | vx10_rx | VX_RX
5 | | |
6 | | |
7 | | |
8 | ALL | vx11_rx | VX_RX
9 | | |
10 | | |
11 | | |
12 | ALL | vx12_rx | VX_RX
13 | | |
14 | | |
15 | | |
16 | ALL | ext_in_lv | LEMO inputs
17 | | |
18 | | |
19 | | |
20 | ALL | FP_LED | FP_LEDs
21 | | |
22 | | |
23 | | |
24 | ALL | ext_out[1] | LEMO outputs
25 | | ext_out[2] |
26 | ALL | vx1_tx | VX1_TX
27 | | |
28 | | |
29 | ALL | vx2_tx | VX2_TX
30 | | |
31 | | |
</pre>

=== Register 7 0x8001301C LED_OUT, EXT_OUT, VX_TX outputs ===

<pre>
wire [3:0] led_out_reg = register_data_in[7][3:0];
wire [2:1] ext_out_reg = register_data_in[7][5:4];
// register_data_in[7][6];
// register_data_in[7][7];
wire [7:0] vx_tx_out_reg = register_data_in[7][15:8];
</pre>

<pre>
bit | version | fpga name | description
0 | ALL | led_out_reg | FP_LED 1
1 | | | 2
2 | | | 3
3 | | | 4
4 | ALL | ext_out_reg | LEMO OUT 1
5 | | | 2
6 | - | |
7 | - | |
8 | ALL | vx_tx_out_reg | VX1_TX 0
9 | | | 1
10 | | | 2
11 | | | -
12 | ALL | | VX2_TX 0
13 | | | 1
14 | | | 2
15 | | | -
16 | - | |
17 | | |
18 | | |
19 | | |
20 | | |
21 | | |
22 | | |
23 | | |
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 8 0x80013020 VX_TX config ===

<pre>
wire [3:0] vx_tx_mux_sel = register_data_in[8][3:0];
wire vx_tx_from_sfp = register_data_in[8][30];
wire vx_tx_loopback = register_data_in[8][31];
</pre>

<pre>
vx_tx_mux_sel is 4 bits (choice 0..15):
0 - power on default, control by vx_tx_out_reg
1 - GDM
2 - CDM
3 - pulser loopback test
4 - pulser loopback test
5 - 62.5 MHz output
6 - 125 MHz output
7 - trg, tsm, serial
8 - trg, tsm, lvds serial rx to serial tx loopback
9 - GPS box control (ds20k rev 0x20240814)
10
11
12
13
14
15 - production config: trg, veto, serial
</pre>

=== Register 9 trg and tsm source ===

from version 0x20240724

<pre>
wire [15:0] trg_src_mask = register_data_in[9][15:0];
wire [31:16] tsm_src_mask = register_data_in[9][31:16];
</pre>

<pre>
wire [15:0] trg_src_bits =
{
1'b0, // 15
1'b0, // 14
vx_tx_tsm_done, // 13
vx_tx_trg_done, // 12

1'b0, // gdm_hitmap_trigger, // 11
cdm_hitmap_trigger, // 10
sfp_rx_tsm, // 9
sfp_rx_trg, // 8

sfp_rx_data[1], // 7
sfp_rx_data[0], // 6
tsm_pulser, // 5
trg_pulser, // 4

lemo_in_sync[4], // 3
lemo_in_sync[3], // 2
lemo_in_sync[2], // 1
lemo_in_sync[1] // 0
};

wire [15:0] trg_bits = trg_src_bits & trg_src_mask;
wire [15:0] tsm_bits = trg_src_bits & tsm_src_mask;
</pre>

before that:

<pre>
wire [7:0] trg_src_mask = register_data_in[9][7:0];
wire [7:0] tsm_src_mask = register_data_in[9][15:8];
wire trg_pulser_enable = register_data_in[9][16];
wire tsm_pulser_enable = register_data_in[9][17];
wire trg_software = register_data_in[9][18];
wire tsm_software = register_data_in[9][19];
// bits 20:31 not used
</pre>

<pre>
wire [7:0] xxx_src_bits =
{
sfp_rx_data[1],
sfp_rx_data[0],
tsm_pulser & tsm_pulser_enable,
trg_pulser & trg_pulser_enable,
lemo_in_sync[4],
lemo_in_sync[3],
lemo_in_sync[2],
lemo_in_sync[1]
};
</pre>

<pre>
trg_src_mask and tsm_src_mask bits:
0 - LEMO IN 1
1 - LEMO IN 2
2 - LEMO IN 3
3 - LEMO IN 4
4 - trg_pulser
5 - tsm_pulser
6 - sfp_rx_data[0] // to become sfp_trg_in, selected from sfp_rx_data[0], cdm_rx trg_out and gdm trigger packet
7 - sfp_rx_data[1] // to become sfp_tsm_in, selected from sfp_rx_data[1], cdm_rx tsm_out and gdm tsm packet
</pre>

=== Register 10 0x80013028 status register ===

<pre>
bit | version | fpga name | description
0 | 0x20231013 | pll_locked | clock chip PLL is locked
1 | 0x20240118 | ts_reset_armed | timestamp reset is armed
2 | 0x20240118 | qsfp_tx_link_rx_status | QSFP link status is good for all enabled ports
3 | 0x20240424 | vx_tx_link_rx_status | VX link status is good for all enabled ports
4 | 0x20240118 | cdm_bsy | VX busy grand-or
5 | 0x20240118 | gdm_bsy | QSFP busy grand-or
6 | 0x20240118 | gdm_veto | gdm_veto = gdm_busy
7 | 0x20240118 | cdm_veto | CDM veto from GDM to VX
8 | 0x20240725 | bor_started | begin-of-run sequence started
9 | 0x20240725 | bor_finished | begin-of-run sequence fininished, see commands 10 and 11
10 | | |
11 | | |
12 | | |
13 | | |
14 | | |
15 | | |
16 | | |
17 | | |
18 | | |
19 | | |
20 | | |
21 | | |
22 | | |
23 | | |
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 11 0x8001302C trg_counter ===

trigger counter

=== Register 12 0x80013030 tsm_counter ===

time slice marker counter

=== Register 13 0x80013034 GPS 1pps period ===

GPS 1pps period in 8 ns clocks

=== Register 14 0x80013038 Rb clock 1pps period ===

PRS-10 Rb clock 1pps output period in 8 ns clocks

=== Register 15 0x8001303C SFP RX status ===

<pre>
bit | ds20k version | fpga signal name | description
0 | ALL | sfp_rx_data[15:0] | cdm sfp received data
16 | same | sfp_rx_data_is_k[0] |
17 | same | sfp_rx_data_is_k[1] |
18 | | 0 |
19 | 0x20231204 | sfp_rx_sel_lpb | sfp tx->rx loopback
20 | 0x20231013 | sfp_link_status | sfp link connected, exchanging data
21 | same | sfp_link_rx_status | sfp link receiving correct idle pattern from GDM TX
22 | same | sfp_link_error | sfp link receiver error (badk or overflow)
23 | same | sfp_rx_data_error | sfp transceiver state machine is in error state
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 16 SFP TX control ===

<pre>
wire [15:0] sfp_tx_data_reg = register_data_in[16][15:0]; // SFP TX data K-code
wire [1:0] sfp_tx_ctrl_reg = register_data_in[16][17:16]; // SFP TX data K-code
// 18
// 19
// 23:20
// 27:24
wire sfp_rx_sel_lpb = register_data_in[16][28]; // TX->RX serial loopback
wire sfp_tx_sel_lpb = register_data_in[16][29]; // RX->TX serial loopback
wire sfp_tx_sel_trg = register_data_in[16][30]; // 16 individual bits
wire sfp_tx_sel_reg = register_data_in[16][31]; // TX data from register 16 bits 17:0
</pre>

=== Register 17-22 QSFP RX data ===

QSFP RX data links 0..11

=== Register 23 QSFP TX control ===

<pre>
wire [15:0] qsfp_tx_data_reg = register_data_in[23][15:0];
wire [1:0] qsfp_tx_ctrl_reg = register_data_in[23][17:16];

wire qsfp_tx_enable_trg = register_data_in[23][24]; // enable QSFP TX trg_in_pulse k-code
wire qsfp_tx_enable_tsm = register_data_in[23][25]; // enable QSFP TX tsm_in_pulse k-code

//wire qsfp_rx_sel_lpb = register_data_in[23][28]; // TX->RX loopback
wire qsfp_tx_sel_lpb = register_data_in[23][29]; // RX->TX loopback
wire qsfp_tx_sel_trg = register_data_in[23][30]; // 16 individual bits
wire qsfp_tx_sel_reg = register_data_in[23][31]; // data from register
</pre>

=== Register 24 0x80013060 trigger pulser period ===

trigger pulser period in units of 8 ns (125 MHz clock)

=== Register 25 0x80013064 trigger burst pulser ===

<pre>
wire [7:0] conf_burst_count = conf_pulser_burst_ctrl[31:24];
wire [23:0] conf_burst_period = conf_pulser_burst_ctrl[23:0];
</pre>

=== Register 26 0x80013068 tsm pulser period ===

time slice marker period in units of 8 ns (125 MHz clock)

=== Register 27 0x8001306C data write fifo ===

<pre>
wire fifo_reset = register_data_in[27][31];
wire fifo_to_fpga_wr1 = register_data_in[27][27];
wire fifo_to_fpga_wr2 = register_data_in[27][26];
wire [16:0] fifo_to_fpga_din = register_data_in[27][16:0];
</pre>

<pre>
assign register_data_out[27][31:24] = register_data_in[27][31:24]; // echo write bits
assign register_data_out[27][23] = fifo_to_fpga_full;
assign register_data_out[27][22] = fifo_to_fpga_empty;
</pre>

=== Register 28 0x80013070 data read fifo ===

<pre>
wire fifo_reset = register_data_in[28][31];
wire fifo_from_fpga_rd1 = register_data_in[28][25];
wire fifo_from_fpga_rd2 = register_data_in[28][24];
</pre>

<pre>
assign register_data_out[28][31:24] = register_data_in[28][31:24]; // echo write bits
assign register_data_out[28][21] = fifo_from_fpga_full;
assign register_data_out[28][20] = fifo_from_fpga_empty;
assign register_data_out[28][16:0] = fifo_from_fpga_dout;
</pre>

=== Register 29 0x80013074 packet routing ===

Control packet routing and loopbacks:

<pre>
wire [3:0] dn_route_ctrl = register_data_in[29][3:0];
wire [3:0] up_route_ctrl = register_data_in[29][7:4];
wire [3:0] fifo_to_fpga_route_ctrl = register_data_in[29][11:8];
//wire [3:0] spare_route_ctrl = register_data_in[29][15:12];

wire dn_mux_trg_enable = register_data_in[29][16];
wire dn_mux_tsm_enable = register_data_in[29][17];
wire dn_mux_sfp_rx_fifo_enable = register_data_in[29][18];
// 19
// 20..23

wire up_mux_vx_rx_enable = register_data_in[29][24];
// 25
// 26
// 27
wire fifo_from_fpga_hitmap_enable = register_data_in[29][28];
wire fifo_from_fpga_cdm_busy_enable = register_data_in[29][29];
wire fifo_from_fpga_gdm_busy_enable = register_data_in[29][30];
wire fifo_from_fpga_tsm_enable = register_data_in[29][31];
</pre>

fifo_to_fpga output routing:
* 0 - to down packet mux
* 1 - to up packet mux
* 2 - to fifo_from_fpga mux
* 3 - not used

down packet mux inputs:
* fifo_to_fpga_0
* trg_pkt16 enabled by dn_mux_trg_enable
* tsm_pkt16 enabled by dn_mux_tsm_enable
* sfp_rx_fifo_pkt16 enabled by dn_mux_sfp_rx_fifo_enable
* up_pkt16_2 loopback from up packet mux

down packet mux output routing:
* 0 - to vx_tx_pkt16 to VX TX (vx_link_tx and vx_ser_tx)
* 1 - to fifo_from_fpga mux
* 2 - to up packet mux loopback
* 3 - to qsfp_tx_pkt16 to GDM QSFP TX (cdm_link_tx)

up packet mux inputs:
* vx_rx_pkt16 enabled by up_mux_vx_rx_enable data from 12 VX RX links
* fifo_to_fpga_1
* dn_pkt16_2 loopback from down packet mux

up packet mux output routing:
* 0 - to sfp_tx_pkt16 to CDM SFP TX (VX data to GDM)
* 1 - to fifo_from_fpga mux
* 2 - to down packet mux loop loopback
* 3 - not used

internal_mux inputs:
* 0 - cdm_hitmap_pkt16 enabled by fifo_from_fpga_hitmap_enable (reg 29)
* 1 - cdm_busy_pkt16 enabled by fifo_from_fpga_cdm_busy_enable
* 2 - gdm_busy_pkt16 enabled by fifo_from_fpga_gdm_busy_enable
* 3 - gdm_hitmap_pkt16 enabled by fifo_from_fpga_hitmap_enable
* 4 - not used

debug_mux inputs:
* 0 - not used
* 1 - dn_pkt16_1
* 2 - up_pkt16_1
* 3 - qsfp0_rx_pkt16 enabled by fifo_from_fpga_qsfp0_rx_enable (from GDM QSFP link 0, there is no GDM QSFP 12-to-1 mux)
* 4 - not used

fifo_from_fpga mux inputs:
* 0 - fifo_to_fpga_2
* 1 - tsm2_pkt16 enabled by fifo_from_fpga_tsm_enable (reg 29)
* 2 - not used
* 3 - debug_pkt16 from debug_mux
* 4 - internal_pkt16 from internal_mux

=== Register 30 0x80013078 qsfp link status ports 0..7 ===

<pre>
assign register_data_out[30] =
{
qsfp_rx_data_error[7], qsfp_link_error[7], qsfp_link_status[7], qsfp_link_rx_status[7], // 7
qsfp_rx_data_error[6], qsfp_link_error[6], qsfp_link_status[6], qsfp_link_rx_status[6], // 6
qsfp_rx_data_error[5], qsfp_link_error[5], qsfp_link_status[5], qsfp_link_rx_status[5], // 5
qsfp_rx_data_error[4], qsfp_link_error[4], qsfp_link_status[4], qsfp_link_rx_status[4], // 4
qsfp_rx_data_error[3], qsfp_link_error[3], qsfp_link_status[3], qsfp_link_rx_status[3], // 3
qsfp_rx_data_error[2], qsfp_link_error[2], qsfp_link_status[2], qsfp_link_rx_status[2], // 2
qsfp_rx_data_error[1], qsfp_link_error[1], qsfp_link_status[1], qsfp_link_rx_status[1], // 1
qsfp_rx_data_error[0], qsfp_link_error[0], qsfp_link_status[0], qsfp_link_rx_status[0] // 0
};
</pre>

=== Register 31 0x8001307C qsfp link status ports 8..11 ===

<pre>
assign register_data_out[31] =
{
4'b0000,
4'b0000,
4'b0000,
4'b0000,
qsfp_rx_data_error[11], qsfp_link_error[11], qsfp_link_status[11], qsfp_link_rx_status[11], // 11
qsfp_rx_data_error[10], qsfp_link_error[10], qsfp_link_status[10], qsfp_link_rx_status[10], // 10
qsfp_rx_data_error[9], qsfp_link_error[9], qsfp_link_status[9], qsfp_link_rx_status[9], // 9
qsfp_rx_data_error[8], qsfp_link_error[8], qsfp_link_status[8], qsfp_link_rx_status[8] // 8
};
</pre>

=== Register 32 0x80013080 bitmap of enabled qsfp ports ===

<pre>
wire [11:0] qsfp_mask = register_data_in[32][11:0];
wire qsfp_bsy_force = register_data_in[32][12];
wire gdm_veto_enable = register_data_in[32][13];
// not used = register_data_in[32][15:14];
wire [15:0] gdm_veto_extend = register_data_in[32][31:16];
</pre>

On the GDM:

qsfp_rx_bsy[11..0] are pulses received from the CDMs

qsfp_rx_bsy_or is the grand-or of qsfp_rx_bsy masked by qsfp_mask

if qsfp_rx_bsy_or is high, gdm_bsy goes up and stays up for vx_bsy_extend*2 clocks

gdm_veto = gdm_bsy (bsy of any one VX causes trigger veto to all of them)

if (gdm_veto_enable) then gdm_veto will block trg_in_pulse

gdm_veto transition 0->1 causes gdm_veto_pulse. as long as gdm_veto is high, gdm_veto_pulse is generated every gdm_veto_extend clocks

gdm_veto_pulse is sent to all CDMs.

For this to work right, cdm_bsy_extend should not be bigger than gdm_veto_extend.

=== Register 33 0x80013084 bitmap of enabled VX ports ===

<pre>
wire [11:0] vx_mask = register_data_in[33][11:0];
wire vx_bsy_force = register_data_in[33][12];
// not used = register_data_in[33][15:13];
wire [15:0] vx_bsy_extend = register_data_in[33][31:16];
</pre>

On the CDM:

cdm_bsy is a grand-or of all vx_bsy masked by vx_mask (list of active VXes).

vx_bsy_extend controls how often state of cdm_bsy is sent to the GDM. when cdm_bsy goes 0->1, we send a cdm_bsy_pulse and keep resending it every vx_bsy_extend clocks as long as cdm_bsy stays high.

cdm_bsy_pulse is sent to the GDM.

sfp_rx_veto is the received from the GDM

if sfp_rx_veto goes up, cdm_veto goes up and stays up for gdm_veto_extend clocks.

for this to work right, CDM gdm_veto_extend must be bigger than GDM gdm_veto_extend.

cdm_veto goes to VXes on v1_tx_out[1] which is LVDS input 13.

=== Register 34, 35 0x80013088, 8C current timestamp ===

current 64-bit timestamp, 125 MHz

=== Register 36, 37 0x80013090, 94 old timestamp ===

old 64-bit timestamp, 125 MHz. timestamp saved at run start when it is reset to 0.

=== Register 38, 39, 40 0x80013098, 9C, A0 VX busy counters ===

8 bits per VX port, counters overflow to 255, reset at run start.

=== Register 41, 42, 43 0x800130A4, A8, AC QSFP busy counters ===

8 bits per QSFP port, counters overflow to 255, reset at run start.

=== Register 44, 45, 46, 47, 48 CDM and GDM busy and veto counters ===

* cdm_busy = grand-or of all VX busy for enabled VXes
* gdm_busy = grand-or of all CDM busy for enabled CDMs
* gdm_veto = gdm_busy

<pre>
44 | lo 16 bits | cdm_bsy_up_counter | CDM busy, increments when cdm_busy goes 0->1
44 | hi 16 bits | cdm_bsy_pulse_counter | CDM busy to GDM, increments for each cdm_bsy_pulse sent to the GDM

45 | lo 16 bits | cdm_veto_pulse_counter | CDM veto from GDM, increments for each sfp_rx_veto received from the GDM
45 | hi 16 bits | cdm_veto_up_counter | CDM veto to VX, increments each time cdm_veto is set to 1.

46 | lo 16 bits | gdm_bsy_pulse_counter | GDM busy from CDM, increments for each qsfp_rx_busy received from CDMs (unless they overlap)
46 | hi 16 bits | gdm_bsy_up_counter | GDM busy, increments each time gdm_bsy goes 0->1

47 | lo 16 bits | gdm_bsy_refresh_counter | GDM busy, increments each time gdm_bsy is extended by new qsfp_rx_busy
47 | hi 16 bits | spare |

48 | lo 16 bits | gdm_veto_up_counter | GDM veto, increments each time gdm_veto goes 0->1
48 | hi 16 bits | gdm_veto_pulse_counter | GDM veto to CDM, increments for each gdm_veto_pulse sent to the CDM
</pre>

=== Register 49, 50, 51 0x800130xx VX RX serial data monitor ===

VX RX data, 8-bit per VX channel. k-bit is omitted.

<pre>
assign register_data_out[49][7:0] = vx_rx_data[0]; // vx1
assign register_data_out[49][15:8] = vx_rx_data[1]; // vx2
assign register_data_out[49][23:16] = vx_rx_data[2]; // vx3
assign register_data_out[49][31:24] = vx_rx_data[3]; // vx4

assign register_data_out[50][7:0] = vx_rx_data[4]; // vx5
assign register_data_out[50][15:8] = vx_rx_data[5]; // vx6
assign register_data_out[50][23:16] = vx_rx_data[6]; // vx7
assign register_data_out[50][31:24] = vx_rx_data[7]; // vx8

assign register_data_out[51][7:0] = vx_rx_data[8]; // vx9
assign register_data_out[51][15:8] = vx_rx_data[9]; // vx10
assign register_data_out[51][23:16] = vx_rx_data[10]; // vx11
assign register_data_out[51][31:24] = vx_rx_data[11]; // vx12
</pre>

=== Register 52, 53 VX link status ===

<pre>
assign register_data_out[52] =
{
vx_rx_error[7], vx_link_error[7], vx_link_status[7], vx_link_rx_status[7], // 7
vx_rx_error[6], vx_link_error[6], vx_link_status[6], vx_link_rx_status[6], // 6
vx_rx_error[5], vx_link_error[5], vx_link_status[5], vx_link_rx_status[5], // 5
vx_rx_error[4], vx_link_error[4], vx_link_status[4], vx_link_rx_status[4], // 4
vx_rx_error[3], vx_link_error[3], vx_link_status[3], vx_link_rx_status[3], // 3
vx_rx_error[2], vx_link_error[2], vx_link_status[2], vx_link_rx_status[2], // 2
vx_rx_error[1], vx_link_error[1], vx_link_status[1], vx_link_rx_status[1], // 1
vx_rx_error[0], vx_link_error[0], vx_link_status[0], vx_link_rx_status[0] // 0
};

assign register_data_out[53] =
{
//4'b0000,
1'b0, ~vx_rx_deser_rdy[0], vx_rx_code_err[0], vx_rx_disp_err[0],
//4'b0000,
//4'b0000,
//4'b0000,
vx_rx_monitor[0],
vx_rx_error[11], vx_link_error[11], vx_link_status[11], vx_link_rx_status[11], // 11
vx_rx_error[10], vx_link_error[10], vx_link_status[10], vx_link_rx_status[10], // 10
vx_rx_error[9], vx_link_error[9], vx_link_status[9], vx_link_rx_status[9], // 9
vx_rx_error[8], vx_link_error[8], vx_link_status[8], vx_link_rx_status[8] // 8
};
</pre>

vx_rx_monitor (12-bit) is from deserializer_10b.sv:

<pre>
assign monitor_out[9:0] = lastByte[9:0];
assign monitor_out[10] = comma;
assign monitor_out[11] = ready;
</pre>

=== Register 54 VX TX serial data monitor ===

* contents of vx_tx_monitor from vx_ser_tx.sv:
<pre>
assign monitor_out[8:0] = data_to_encoder; // 8-bit + k
assign monitor_out[9] = valid;
assign monitor_out[15:10] = 0;
assign monitor_out[25:16] = encoded_data; // 10-bit
assign monitor_out[26] = encoded_valid;
assign monitor_out[27] = 0;
assign monitor_out[31:28] = 0;
</pre>

=== Register 55 QSFP, SFP, VX link loss counters ===

<pre>
assign register_data_out[55] = {
vx_rx_error_counter,
qsfp_link_rx_status_drop_counter,
sfp_link_rx_status_drop_counter,
vx_link_rx_status_drop_counter
};
</pre>

<pre>
24..31 - CDM VX RX error counters, count any errors in the VX RX data path (bad serial data, fifo overflow, etc)
16..23 - GDM QSFP link loss counter, increments on qsfp_tx_link_rx_status 1->0 (reg10)
8..15 - CDM SFP link loss counter, increments on sfp_link_rx_status 1->0 (reg15)
0...7 - CDM VX link loss counter, increments on vx_tx_link_rx_status 1->0 (reg10)
</pre>

=== Register 68 GPS control and status ===

<pre>
assign register_data_out[68] =
{
8'b00000000, // 23+8
rb_1pps_counter, // 16+8 bits
gps_1pps_counter, // 8+8 bits
1'b0,
1'b0,
rb_ser_in, // 5
gps_data_in, // 4
gps_aux_out, // 3
gps_aux_in, // 2
rb_1pps_in, // 1
gps_1pps_in // 0
};
</pre>

=== Register 71 packet error bits ===

<pre>
assign register_data_out[71] =
{
8'b00000000, // 24+8 bits
1'b0, // 23
1'b0, // 22
fifo_to_fpga_full_latch, // 21
fifo_from_fpga_full_latch, // 20
gdm_hitmap_fifo_full_latch, // 19
gdm_busy_fifo_full_latch, // 18
cdm_busy_fifo_full_latch, // 17
cdm_hitmap_fifo_full_latch, // 16
1'b0, // 15
1'b0, // 14
1'b0, // 13
gdm_trig48_encode_error_latch, // 12
1'b0, // vx_tx_trig48_pkt8_error_latch, // 11
tsm2_encode_error_latch, // 10
tsm_encode_error_latch, // 9
trg_encode_error_latch, // 8
gdm_hitmap_error_latch, // 7
gdm_hitmap_encode_error_latch, // 6
gdm_busy_encode_error_latch, // 5
cdm_busy_encode_error_latch, // 4
cdm_hitmap_encode_error_latch, // 3
vx_rx_hitmap_error_latch, // 2
1'b0, // vx_tx_tsm_pkt8_error_latch, // 1
1'b0, // vx_tx_trg_pkt8_error_latch // 0
};
</pre>

=== Register 90 VX busy status register ===

<pre>
vx_bsy_status_latched[11:0] <= vx_bsy[11:0];
vx_bsy_status_latched[23:12] <= vx_bsy_collect[11:0];
vx_bsy_status_latched[24] <= cdm_bsy;
vx_bsy_status_latched[25] <= cdm_bsy_collect;
vx_bsy_status_latched[26] <= cdm_veto;
vx_bsy_status_latched[27] <= cdm_veto_collect;
vx_bsy_status_latched[28] <= 0;
vx_bsy_status_latched[29] <= 0;
vx_bsy_status_latched[30] <= 0;
vx_bsy_status_latched[31] <= cdm_bsy_stuck;
</pre>

=== Register 91 VX hitmap status register ===

<pre>
hitmap_status_latched[11:0] <= hitmap_or12[11:0];
hitmap_status_latched[23:12] <= hitmap_or12_collect[11:0];
hitmap_status_latched[24] <= hitmap_grand_or;
hitmap_status_latched[25] <= hitmap_grand_or_collect;
hitmap_status_latched[26] <= 0;
hitmap_status_latched[27] <= 0;
hitmap_status_latched[28] <= 0;
hitmap_status_latched[29] <= 0;
hitmap_status_latched[30] <= 0;
hitmap_status_latched[31] <= hitmap_grand_or_stuck;
</pre>

=== Register 92 GDM busy status register ===

<pre>
gdm_bsy_status_latched[11:0] <= qsfp_rx_bsy[11:0];
gdm_bsy_status_latched[23:12] <= qsfp_rx_bsy_collect[11:0];
gdm_bsy_status_latched[24] <= gdm_bsy;
gdm_bsy_status_latched[25] <= gdm_bsy_collect;
gdm_bsy_status_latched[26] <= gdm_veto;
gdm_bsy_status_latched[27] <= gdm_veto_collect;
gdm_bsy_status_latched[28] <= 0;
gdm_bsy_status_latched[29] <= 0;
gdm_bsy_status_latched[30] <= 0;
gdm_bsy_status_latched[31] <= gdm_bsy_stuck;
</pre>

=== Register 93 GDM hitmap status register ===

<pre>
hitmap_status_latched[11:0] <= hitmap_grand_or_12[11:0];
hitmap_status_latched[23:12] <= hitmap_grand_or_12_collect[11:0];
hitmap_status_latched[24] <= hitmap_grand_or;
hitmap_status_latched[25] <= hitmap_grand_or_collect;
hitmap_status_latched[26] <= 0;
hitmap_status_latched[27] <= 0;
hitmap_status_latched[28] <= 0;
hitmap_status_latched[29] <= 0;
hitmap_status_latched[30] <= 0;
hitmap_status_latched[31] <= hitmap_grand_or_stuck;
</pre>

=== Register 98, 99 GDM TRIG48 test register and TRIG48_CTRL ===

<pre>
assign gdm_trig48_ctrl_bitmap48[31:0] = register_data_in[98][31:0];
assign gdm_trig48_ctrl_bitmap48[47:32] = register_data_in[99][15:0];

assign gdm_trig48_ctrl[15:0] = register_data_in[99][31:16];
</pre>

<pre>
wire enable_grand_or = trig48_ctrl[15];
wire enable_test_trig48 = trig48_ctrl[14];
</pre>

= Firmware registers branch develop_ko =

== Register map ==

<pre>
0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
1 | ALL | ALL | RW | read write scratch register
2 | ALL | CDM | ?? | MGT not used
3 | ALL | CDM | RO | MGT debug_data
4 | ALL | CDM | RW | clk_config_vec
5 | ALL | CDM | ?? | not used
6 | ALL | CDM | RO | CDM_link_data_processing:o_error_time
7 | ALL | CDM | RO | CDM_link_data_processing:o_error_count
</pre>

== register 0 0x80010000 ==

GDM:
<pre>
0 - gdm_link_interface:i_mgt_rst
2 - gdm_link_interface:i_link_down_latched_rst
8 - GDM_link_data_processing:i_rst
10..9 - GDM_link_data_processing:i_data_mode
</pre>

CDM:
<pre>
0 - cdm_link_interface:i_mgt_rst
2 - cdm_link_interface:i_link_down_latched_rst
8 - CDM_link_data_processing:i_rst
10..9 - CDM_link_data_processing:i_data_mode
</pre>

== register 1 0x80010004 ==

GDM:
<pre>
nlinks-1..0 - gdm_link_interface:i_rx_slide_trigger
</pre>

CDM:
<pre>
nlinks-1..0 - cdm_link_interface:i_rx_slide_trigger
</pre>

== register 2 0x80010008 ==

GDM:
<pre>
nlinks-1..0 - gdm_link_interface:o_link_power_good
nlinks+15..16 - gdm_link_interface:o_link_status
</pre>

CDM:
<pre>
nlinks-1..0 - cdm_link_interface:o_link_power_good
nlinks+15..16 - cdm_link_interface:o_link_status
</pre>

== register 3 0x8001000c ==

GDM: simple loopback register

CDM:
<pre>
31..0 - debug_data - cdm_link_interface:o_debug
</pre>

o_debug:
<pre>
rx_link_rst & rx_error & rx_link_up & rx_receiving_data &
std_logic_vector(rx_state_count) & tx_state_count_on_rx_clk & i_rx_ctrl3(0) &
i_rx_ctrl1(1 downto 0) & i_rx_ctrl0(1 downto 0) &
rx_data_is_k28p1_k28p5 &
i_rx_data;
</pre>

== register 4 0x80010010 ==

GDM write:
<pre>
0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS
</pre>

GDM read:
<pre>
0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS
4 - clk_config_vec(4) - CLK_LOSXTn_LS
5 - clk_config_vec(5) - CLK_LOLn_LS
6 - clk_config_vec(6) - CLK_INTn_LS
7 - constant 1
31..8 - constant 0
</pre>

== register 5 0x80010014 ==

not used

== register 6 0x80010018 ==

GDM:
<pre>
3..0 - GDM_link_data_processing:i_status_select
</pre>

CDM:
<pre>
31..0 - CDM_link_data_processing:o_error_count
</pre>

== register 7 0x8001001c ==

GDM:
<pre>
31..0 - GDM_link_data_processing:o_status_vector
</pre>

CDM:
<pre>
31..0 - CDM_link_data_processing:o_error_count
</pre>

= GDM, CDM, VX packet communications =

* timestamp math
<pre>
1 clock is 8 ns is 125 MHz
8 bits of clocks is 256 clocks is 2048 ns is ~2 usec
16 bits of clocks is ~500 usec is 0.5 msec
24 bits of clocks is ~134 msec
32 bits of clocks is ~34 sec
40 bits of clocks is ~8.7 ksec is 2.4 hours
48 bits of clocks is ~625 hours is ~26 days
56 bits of clocks is ~6.6 kdays is ~18 kyears
62 bits of clocks is ~10 Mhours is 427 kdays is ~1.1 kyears
64 bits of clocks is ~4.4 kyears
</pre>
* summary
<pre>
0x01 - not used
0x02 - 8 bytes - 4 words - 2 dwords - 1 qwords - TRG packet
0x03 - 12 bytes - 6 words - 3 dwords - 2 qwords - TRIG48 packet
0x10 - 26 bytes - 13 words - 7 dwords - 4 qwords - TSM packet
0x81 - 10 bytes - 5 words - 3 dwords - 2 qwords - VX_HITMAP packet
0x82 - 108 bytes - 54 words - 27 dwords - 14 qwords - CDM_HITMAP packet
0x83 - 12 bytes - 6 words - 3 dwords - 2 qwords - CDM_BUSY packet
0x84 - 12 bytes - 6 words - 3 dwords - 2 qwords - GDM_BUSY packet
0x85 - 12 bytes - 6 words - 3 dwords - 1 qwords - GDM_HITMAP packet
</pre>
* 0x02 - TRG packet, 8 bytes, 80 adc clocks, 640 ns on lvds link
<pre>
0 - 0x02
1 - trg_counter[7:0]
2 - ts64 low byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 high byte 3
6 - trg_in_latch[7:0]
7 - trg_in_latch[15:8]
</pre>
* 0x03 - TRIG48 packet, 12 bytes, 120 adc clocks, 960 ns on lvds link
<pre>
0 - 0x03
1 - trg_counter[7:0]
2 - ts64 low byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 high byte 3
6 - vx_bitmap[7:0]
7 - vx_bitmap[15:8]
8 - vx_bitmap[23:16]
9 - vx_bitmap[31:24]
10 - vx_bitmap[39:32]
11 - vx_bitmap[47:40]
</pre>
* 0x10 - TSM packet, 26 bytes, 260 adc clocks, 2080 ns on lvds link
<pre>
0 - 0x10
1 - tsm_counter[7:0]
2 - gdm_ts64 low byte 0
3 - 1
4 - 2
5 - 3
6 - 4
7 - 5
8 - 6
9 - gdm_ts64 high byte 7
10 - gps_ts64 low byte 0
11 - 1
12 - 2
13 - 3
14 - 4
15 - 5
16 - 6
17 - gps_ts64 high byte 7
18 - gps_data64 low byte 0
19 - 1
20 - 2
21 - 3
22 - 4
23 - 5
24 - 6
25 - gps_data64 high byte 7
</pre>
* 0x81 - VX hitmap packet, 10 bytes, 100 adc clocks, 800 ns on lvds link, 48 ns on fiber link
<pre>
0 - 0x81
1 - VX ID
2 - hitmap low byte, nits 7:0
3 - 15:8
4 - 23:16
5 - 31:24
6 - ...:32
7 - ...
8 - ...
9 - hitmap low byte, bits 63:...
</pre>
* 0x82 - CDM hitmap packet, 108 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x82
1 - cdm_hitmap_trigger_counter[7:0]
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - cdm_hitmap_or12 byte 0 (7:0)
11 - cdm_hitmap_or12 byte 1 (11:8) plus 4 bits: 12=0, 13=0, 14=0, 15=cdm_hitmap_grand_or
12 - cdm_hitmap_data, low byte, 12*64 bits = 768 bits = 48 words = 96 bytes
...
107 - cdm_hitmap_data, high byte
</pre>
* 0x83 - CDM busy packet, 12 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x83
1 - cdm_busy_packet_counter[7:0], reset at begin of run
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - vx_bsy[7:0]
11 - {0,0,cdm_veto,cdm_bsy,vx_bsy[11:8]}
</pre>
* 0x84 - GDM busy packet, 12 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x84
1 - gdm_busy_packet_counter[7:0], reset at begin of run
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - qsfp_rx_bsy[7:0]
11 - {0,0,gdm_veto,gdm_bsy,qsfp_rx_bsy[11:8]}
</pre>
* 0x85 - GDM hitmap trigger packet, 12 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x85
1 - gdm_hitmap_packet_counter[7:0], reset at begin of run
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - hitmap_grand_or_12[7:0]
11 - {hitmap_grand_or,0,0,0,hitmap_grand_or_12[11:8]}
</pre>

= AXI bus timing =

* AXI 100 MHz clock, 10 ns, 32-bit data
* AXI single-dword read: 36 clock repeat rate, 360 ns is 2.777 MHz, 4 bytes per transfer is 11.11 Mbytes/sec
* AXI single-qword read: 13 clock repeat rate, 130 ns is 7.7 MHz, 4 bytes per transfer is 30 Mbytes/sec, not accounting for the gap
* AXI memcpy read: 4 transfers at 13 clocks, gap, 4 transfers at 13 clocks, gap, etc. below 30 Mbytes/sec.
[[Image:Ds-dm-axi-read.png|100px]]
[[Image:Ds-dm-axi-read-64.png|100px]]
[[Image:Ds-dm-axi-read-memcpy.png|100px]]
* AXI single-dword write: 20 clocks repeat rate, 200 ns is 5 MHz, 4 bytes per transfer is 20 Mbytes/sec
* AXI single-qword write: 13+20 clocks repeat rate, 330 ns is 3 MHz, 16 bytes per burst is 48 Mbytes/sec
[[Image:Ds-dm-axi-write.png|100px]]
[[Image:Ds-dm-axi-write-64.png|100px]]

= AXI bus addresses =

* see AXI/AMBA addresses assigned inthe FPGA project: (s_axi/reg0 is the DS-DM AXI registers)
<pre>
daq00:ds-dm-gcdm$ grep assign_bd_address scripts/GDM_CDM_XU8_bd.tcl
assign_bd_address -offset 0x80010000 -range 0x00004000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs axi_register_interfa_0/s_axi/reg0] -force
assign_bd_address -offset 0x000400000000 -range 0x40000000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs ddr4/C0_DDR4_MEMORY_MAP/C0_DDR4_ADDRESS_BLOCK] -force
assign_bd_address -offset 0x80000000 -range 0x00010000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs system_management_wiz/S_AXI_LITE/Reg] -force
daq00:ds-dm-gcdm$
</pre>

* see AXI/AMBA addresses exported from FPGA project to Linux kernel: (uio for debug bridge should say "debug bridge")
<pre>
root@gdm0:~# cat /sys/class/uio/uio*/name
axi-pmon
axi-pmon
axi-pmon
axi-pmon
root@gdm0:~#
root@gdm0:~# ls -l /sys/class/uio/
total 0
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio0 -> ../../devices/platform/amba/ffa00000.perf-monitor/uio/uio0
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio1 -> ../../devices/platform/amba/fd0b0000.perf-monitor/uio/uio1
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio2 -> ../../devices/platform/amba/fd490000.perf-monitor/uio/uio2
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio3 -> ../../devices/platform/amba/ffa10000.perf-monitor/uio/uio3
root@gdm0:~# ls -l /sys/class/uio/../../devices/platform/amba/
total 0
-rw-r--r-- 1 root root 4096 Oct 18 19:37 driver_override
drwxr-xr-x 3 root root 0 Oct 18 01:36 fd070000.memory-controller
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd0b0000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd400000.zynqmp_phy
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd490000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd500000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd510000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd520000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd530000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd540000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd550000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd560000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd570000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd6e0000.cci
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff000000.serial
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff020000.i2c
drwxr-xr-x 6 root root 0 Oct 18 01:36 ff0a0000.gpio
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0b0000.ethernet
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0e0000.ethernet
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0f0000.spi
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff160000.mmc
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff170000.mmc
drwxr-xr-x 3 root root 0 Oct 18 01:36 ff960000.memory-controller
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff9d0000.usb0
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa00000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa10000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa50000.ams
drwxr-xr-x 5 root root 0 Oct 18 01:36 ffa60000.rtc
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa80000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa90000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffaa0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffab0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffac0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffad0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffae0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffaf0000.dma
-r--r--r-- 1 root root 4096 Oct 18 19:37 modalias
lrwxrwxrwx 1 root root 0 Oct 18 19:37 of_node -> ../../../firmware/devicetree/base/amba
drwxr-xr-x 2 root root 0 Oct 18 19:37 power
lrwxrwxrwx 1 root root 0 Oct 18 01:36 subsystem -> ../../../bus/platform
-rw-r--r-- 1 root root 4096 Oct 18 01:36 uevent
root@gdm0:~#
</pre>

= Build firmware =

== Build from git clone ==

THESE ARE K.O.'s NOTES FOR CREATING THE PETALINUX DIRECTORY.

THEY DO NOT WORK!

COPY PETALINUX FROM A WORKING PROJECT AND USE "make gdm" and "make cdm" AS DESCRIBED BELOW.

* git clone git@edev-group.triumf.ca:fw/exp/darkside/gcdm.git
* #Makefile change VIVADO_SETTINGS_SCRIPT := /opt/Xilinx/Vivado/2022.1/settings64.sh
* #. /opt/Xilinx/Vivado/2022.1/settings64.sh
* . /opt/Xilinx/Vivado/2020.2/settings64.sh
* make clean
* make all_from_scratch
* . /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
* make petalinux_create
* make petalinux_rebuild_new_hw_des
* bomb out: The TMPDIR: /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp can't be located on nfs.
* mkdir /tmp/build_tmp
* rm -rf /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp/
* ln -s /tmp/build_tmp /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp
* try again
* grinds, loads a whole bunch of packages...
* finishes with desire to copy things to /tftpboot
* make sdcard_cp_to wants to copy files from PetaLinux_GDM_CDM/images/linux/ to SD card

== Build firmware ==

NOTE: directory Petalinux_GDM_CDM should already exist!

<pre>
#. /opt/Xilinx/Vivado/2020.2/settings64.sh
. /opt/Xilinx/Vivado/2022.2/settings64.sh
. /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
make clean_gdm # remove gdm build tree
make gdm # build or rebuild GDM
make copy_gdm # copy to gdm0
make clean_cdm # remove cdm build tree
make cdm # build or rebuild CDM
make copy_cdm # copy to cdm0 and cdm1
</pre>

copy to SD card:
<pre>
open a root shell
format 16 GB Sd card per above
cd .../ds-dm-gcdm
make copy
</pre>

= build times =

<pre>
CDM 12-june-2023 69aabc1c25130d970bc375aca684bd68849e6685
daq13 AMD-5700G 1688.61user 399.33system 23:28.84elapsed 148%CPU
dsdaqgw AMD-7700 1090.55user 247.34system 16:03.55elapsed 138%CPU
dsdaqgw AMD-7700 CDM incremental 196.68user 67.62system 7:35.42elapsed 58%CPU
dsdaqgw AMD-7700 CDM incremental 684.72user 94.17system 7:30.17elapsed 173%CPU
dsdaqgw AMD-7700 GDM incremental 849.84user 99.79system 9:04.56elapsed 174%CPU
</pre>

= make bootable sd card =

== format the sd card ==

this only needs to be done once

* become root@dsdaqgw
* cd ~olchansk/git/ds-dm-gcdm
* use "lsblk" to identify the SD card (should show as 8/16/32 GB block device)/ /dev/sdd in this case
* make sdcard_format SDCARD_DEVICE=/dev/sdd
* disconnect sd card, reconnect the sd card (to detect new partition tables, etc)

== copy boot files to the sd card ==

* as root: identify partition labels, run "blkid", should say "BOOT", "rootfs" and "data"
* mount
<pre>
mkdir -p /mnt/BOOT
mount -L BOOT /mnt/BOOT
cp PetaLinux_GDM_CDM/images/linux/BOOT_{GDM,CDM}.BIN /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/boot.scr /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/image.ub /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/uboot.env /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/uboot-redund.env /mnt/BOOT/
umount /mnt/BOOT
eject /dev/sdX
</pre>
* FIXME: missing step: cp BOOT_GDM.BIN BOOT.BIN or cp BOOT_CDM.BIN BOOT.BIN

= boot messages =

<pre>
Xilinx Zynq MP First Stage Boot Loader
Release 2020.2 Sep 24 2022 - 13:29:15
NOTICE: ATF running on XCZU4CG/silicon v4/RTL5.1 at 0xfffea000
NOTICE: BL31: v2.2(release):xlnx_rebase_v2.2_2020.3
NOTICE: BL31: Built : 18:02:46, Sep 28 2022

U-Boot 2020.01 (Sep 28 2022 - 18:03:39 +0000)

Model: DarkSide 20k DM
Board: Xilinx ZynqMP
DRAM: 2 GiB
usb dr_mode not found
PMUFW: v1.1
EL Level: EL2
Chip ID: zu4
NAND: 0 MiB
MMC: mmc@ff160000: 0, mmc@ff170000: 1
In: serial@ff000000
Out: serial@ff000000
Err: serial@ff000000
Bootmode: SD_MODE1
Reset reason: SOFT
Net:
ZYNQ GEM: ff0b0000, mdio bus ff0b0000, phyaddr -1, interface rgmii-id

Warning: ethernet@ff0b0000 (eth0) using random MAC address - d6:62:5f:13:00:44
eth0: ethernet@ff0b0000
ZYNQ GEM: ff0e0000, mdio bus ff0e0000, phyaddr -1, interface rgmii-id
Could not get PHY for eth1: addr -1

Hit any key to stop autoboot: 0
ZynqMP>
CTRL-A Z for help | 115200 8N1 | NOR | Minicom 2.7.1 | VT102 | Online 122:2 | ttyACM0
</pre>

= load FPGA from u-boot =

* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/124682257/U-Boot+FPGA+Driver

<pre>
ZynqMP> fpga info
Xilinx Device
Descriptor @ 0x000000007fddb2c0
Family: ZynqMP PL
Interface type: csu_dma configuration interface (ZynqMP)
Device Size: 1 bytes
Cookie: 0x0 (0)
Device name: zu4
Device Function Table @ 0x000000007fda5fe8
PCAP status 0xa0002fde
ZynqMP>
</pre>

* cp CDM_XU8_top.bit /tftpboot/fpga.bit

<pre>
dhcp
tftpb 0x10000000 fpga.bit
fpga loadb 0 0x10000000 ${filesize}
</pre>

<pre>
ZynqMP> dhcp
BOOTP broadcast 1
DHCP client bound to address 192.168.0.100 (1 ms)
*** Warning: no boot file name; using 'C0A80064.img'
Using ethernet@ff0b0000 device
TFTP from server 192.168.0.1; our IP address is 192.168.0.100
Filename 'C0A80064.img'.
Load address: 0x8000000
Loading: *
TFTP error: 'file /tftpboot/C0A80064.img not found for 192.168.0.100' (1)
Not retrying...
ZynqMP> tftpb 0x10000000 fpga.bit
Using ethernet@ff0b0000 device
TFTP from server 192.168.0.1; our IP address is 192.168.0.100
Filename 'fpga.bit'.
Load address: 0x10000000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
############
6.2 MiB/s
done
Bytes transferred = 7797807 (76fc2f hex)
ZynqMP> fpga loadb 0 0x10000000 ${filesize}
design filename = "CDM_XU8_top;UserID=0XFFFFFFFF;Version=2022.2"
part number = "xczu4cg-fbvb900-1-e"
date = "2024/08/14"
time = "14:18:22"
bytes in bitstream = 7797692
zynqmp_align_dma_buffer: Align buffer at 0000000010000073 to 000000000fffff80(swap 0)
ZynqMP>
</pre>

= load FPGA from Linux =

* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841847/Solution+ZynqMP+PL+Programming

== this will reset the CPU ==
<pre>
cp fpga.bit /lib/firmware/
echo fpga.bit > /sys/class/fpga_manager/fpga0/firmware
</pre>

* make .bin file:
<pre>
bootgen -image CDM_XU8_top.bif -arch zynqmp -o ./fpga.bin -w
</pre>
* cat CDM_XU8_top.bif
<pre>
dsdaqgw:ds-dm-gcdm$ cat CDM_XU8_top.bif
all:
{
[destination_device = pl] ./Vivado_CDM_XU8/CDM_XU8.runs/impl_1/CDM_XU8_top.bit
}
dsdaqgw:ds-dm-gcdm$
</pre>

== this will reset the CPU ==

<pre>
cp fpga.bin /lib/firmware/
echo fpga.bin > /sys/class/fpga_manager/fpga0/firmware
</pre>

== this will reset the CPU ==

<pre>
root@dsdm:~# ./fpgautil -b fpga.bin -f Full
</pre>

== this will reset the CPU ==

<pre>
DTSO file from here: https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841847/Solution+ZynqMP+PL+Programming
echo 0 > /sys/class/fpga_manager/fpga0/flags
mount -t configfs configfs /configfs
root@dsdm:~# dtc -O dtb -o fpga.dtbo -b 0 -@ fpga.dtso
root@dsdm:~# cp fpga.dtbo /lib/firmware/
root@dsdm:~# cp fpga.bit /lib/firmware/
root@dsdm:~# rmdir /configfs/device-tree/overlays/fpga
root@dsdm:~# mkdir /configfs/device-tree/overlays/fpga
root@dsdm:~# echo -n "fpga.dtbo" > /configfs/device-tree/overlays/fpga/path
</pre>

= fpgautil =

* https://github.com/Xilinx/meta-xilinx/blob/master/meta-xilinx-core/recipes-bsp/fpga-manager-script/files/fpgautil.c

<pre>
git clone https://github.com/Xilinx/meta-xilinx.git
cd meta-xilinx/meta-xilinx-core/recipes-bsp/fpga-manager-script/files/
scp fpgautil.c root@dsdm:
ssh root@dsdm
make fpgautil
ls -l ./fpgautil
</pre>

<pre>
root@dsdm:~# ls -l ./fpgautil
-rwxr-xr-x 1 root root 72256 Aug 16 00:15 ./fpgautil
root@dsdm:~# ./fpgautil

fpgautil: FPGA Utility for Loading/reading PL Configuration

Usage: fpgautil -b <bin file path> -o <dtbo file path>

Options: -b <binfile> (Bin file path)
-o <dtbofile> (DTBO file path)
-f <flags> Optional: <Bitstream type flags>
f := <Full | Partial >
-n <Fpga region info> FPGA Regions represent FPGA's
and partial reconfiguration
regions of FPGA's in the
Device Tree
Default: <full>
-s <secure flags> Optional: <Secure flags>
s := <AuthDDR | AuthOCM | EnUsrKey | EnDevKey | AuthEnUsrKeyDDR | AuthEnUsrKeyOCM | AuthEnDevKeyDDR | AuthEnDevKeyOCM>
-k <AesKey> Optional: <AES User Key>
-r <Readback> Optional: <file name>
Default: By default Read back contents will be stored in readback.bin file
-t Optional: <Readback Type>
0 - Configuration Register readback
1 - Configuration Data Frames readback
Default: 0 (Configuration register readback)
-R Optional: Remove overlay from a live tree

Examples:
(Load Full bitstream using Overlay)
fpgautil -b top.bit.bin -o can.dtbo -f Full -n full
(Load Partial bitstream using Overlay)
fpgautil -b rm0.bit.bin -o rm0.dtbo -f Partial -n PR0
(Load Full bitstream using sysfs interface)
fpgautil -b top.bit.bin -f Full
(Load Partial bitstream using sysfs interface)
fpgautil -b rm0.bit.bin -f Partial
(Load Authenticated bitstream through the sysfs interface)
fpgautil -b top.bit.bin -f Full -s AuthDDR
(Load Parital Encrypted Userkey bitstream using Overlay)
fpgautil -b top.bit.bin -o pl.dtbo -f Partial -s EnUsrKey -k <32byte key value>
(Read PL Configuration Registers)
fpgautil -b top.bit.bin -r
(Remove Partial Overlay)
fpgautil -R -n PR0
(Remove Full Overlay)
fpgautil -R -n full
Note: fpgautil -R is responsible for only removing the dtbo file from the livetree. it will not remove the PL logic from the FPGA region.

root@dsdm:~#
</pre>

= fw_printenv =

to access u-boot environment from Linux:
* apt install -y libubootenv-tool
* create /etc/fw_env.config
<pre>
/media/BOOT/uboot.env 0 0x40000
/media/BOOT/uboot-redund.env 0 0x40000
</pre>
* if uboot.env files do not exist, run "saveenv" from u-boot command prompt
* fw_printenv and fw_setenv should work

= Boot from network =

== u-boot ==

<pre>
ZynqMP> setenv bootcmd run bootcmd_dhcp
ZynqMP> saveenv
ZynqMP> reset
</pre>

== boot.scr ==

<pre>
# boot.scr
# mkimage -C none -A arm -T script -d boot.scr boot.scr.uimg
echo Loading FPGA!
#tftpb 0x10000000 fpga.bit
tftpb 0x10000000 {ipaddr}.bit
fpga loadb 0 0x10000000 ${filesize}
echo Booting Linux!
run bootcmd_pxe
echo Done!
</pre>

<pre>
mkimage -C none -A arm -T script -d boot.scr boot.scr.uimg
</pre>

== tftpboot ==

<pre>
cp /home/olchansk/git/ds-dm-gcdm/boot.scr.uimg /tftpboot
ln -s /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/images/linux /tftpboot/xilinx-dsdm
mkdir /tftpboot/pxelinux.cfg
cat > /tftpboot/pxelinux.cfg/default-arm-zynqmp <<EOF
LABEL Linux
KERNEL xilinx-dsdm/Image
FDT xilinx-dsdm/system.dtb
#INITRD rootfs.cpio.gz.u-boot
EOF
</pre>

== boot sequence ==

* xilinx magic load BOOT.BIN from SD card
* load FPGA form BOOT.BIN
* load and run u-boot from BOOT.BIN or from image.ub
* u-boot load environment from ??? probably SD card uboot-redund.env, this includes our bootcmd
* run bootcmd which run bootcmd_dhcp which does:
* from /tftpboot:
* load and run boot.scr.uimg which does:
* load FPGA image xilinx-dsdm/${ipaddr}.bit
* run bootcmd_pxe which does:
* load pxelinux.cfg/default-arm-zynqmp which does:
* load xilinx-dsdm/Image ### this is the linux kernel
* load xilinx-dsdm/system.dtb ### this is the device tree
* start linux kernel
* linux kernel does dhcp
* linux kernel does nfs mount /nfsroot/%s,vers=3,tcp ### %s is replaced by the hostname supplied by DHCP
* userland starts and runs to console and ssh login.

= Xilinx ILA =

References:
* https://github.com/Xilinx/XilinxVirtualCable/tree/master
* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/644579329/Xilinx+Virtual+Cable
* https://support.xilinx.com/s/article/974879?language=en_US
* https://docs.xilinx.com/v/u/en-US/pg245-debug-bridge
* https://docs.xilinx.com/r/2020.2-English/ug908-vivado-programming-debugging/Initializing-Vivado-IDE-hw_server
* (we do not use this one) https://github.com/paulscherrerinstitute/xvcSupport/blob/master/README.md

Build xvcserver_cdm.exe: (it is built as a static executable, can be copied and run anywhere)
<pre>
ssh dsdaq@gdm0
cd /home/dsdaq/online/ds-dm-software
git pull ### get latest version
make xvcserver_cdm.exe
ssh root@gdm0
/home/dsdaq/online/ds-dm-software/xvcserver_cdm.exe
INFO: To connect to this xvcServer instance, use url: TCP:gdm0:2542
</pre>

To activate and use the vivado logic analyzer:
* data path: vivado -> hw_server -> xvcserver -> mmap axi bus -> debug bridge -> jtag -> ILA
* define ILAs in the code
* instantiate the xilinx debug bridge at AXI bus address 0x80020000 (FIXME!!! this collides with Ian's AXI addresses)
* build and boot the new FPGA firmware. updating the linux kernel is not necessary.
* login root@gdm0, run: /home/dsdaq/online/ds-dm-software/xvcserver_cdm.exe -v ### with "-v" for the first time to see that vivado does connect to it, without "-v", normally.
* login dsdaqgw, run: hw_server -s tcp:localhost:3121 -e "set auto-open-servers xilinx-xvc:gdm0:2542" ### tells us to connect to port localhost:3121
* login dsdaqgw, run vivado, open project, open hardware manager, open target, open new target, "connect to remote server", hostname "localhost", port "3121", next (bombs, try again, 3 times), popup add virtual cable, enter hostname "gdm0" port "2542", "ok", it shows in "hardware targets", "next", "finish", error popup "[Common 17-163] Missing value for option 'objects', please type 'set_property -help' for usage info", ignore it, in "hardware", right click the "gdm0" one, open target, under "hardware" and "debug bridge" we should see all the ILAs, under "hardware device properties", the "probes file" should have the ".ltx" file generated by vivado "Vivado_GDM_XU8/GDM_XU8.runs/impl_1/debug_nets.ltx", click on an ILA, a waveform should open.

= Software =

* ssh cdm0 # or gdm0
* sudo apt install i2c-tools libi2c-dev
* git clone https://bitbucket.org/team-ds-dm/ds-dm-software
* cd ds-dm-software
* make

== test_cdm.exe ==

=== CDM SFP status ===

<pre>
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0xbb2f0ae7
CDM firmware 0xbb2f0ae7
arg 1: [--sfp]
Polling SFP status...
identifier 0x03
connector 0x07
encoding 0x01
wavelength 0x0352 (850 nm)
vendor_name [FINISAR CORP. ]
vendor_pn [FTLF8526P3BNL ]
vendor_rev [A ]
vendor_sn [N3AB9M8 ]
vendor_date [200319 ]
dm_type 0x68
temp 29.0 C
vcc 3.323 V
tx_bias 7.250 mA
tx_power 478.4 uW
rx_power 2.3 uW
SFP good 1, status: temp 30.6 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 818 uW
...
</pre>

=== GDM QSFP status ===

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --qsfp3 --qsfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x53aee418
CDM firmware 0x53aee418
arg 1: [--qsfp3]
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out hi
arg 2: [--qsfp]
Polling QSFP status...
identifier 0x0d
status 0x02
los 0x8f
temp 28.2 C
vcc 3.323 V
rx_power 0.1 0.1 0.1 0.1 uW
tx_bias 7.6 7.6 7.6 0.0 mA
tx_power 792.2 773.8 823.0 0.1 uW
vendor_name [FINISAR CORP ]
vendor_pn [FTL410QD4C ]
vendor_rev [A ]
wavelength 850
max_temp 70 C
vendor_sn [X79AC0R ]
vendor_date [220309 ]
QSFP good 1, status: temp 27.7 C, los 0x8b, tx_bias 7.5 7.6 7.6 0.0 mA, tx_power 792 772 821 0 uW, rx_power 0 0 466 0 uW
</pre>

=== GDM clock status ===

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
</pre>

* clock chip not loaded, not running:

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x0f 0x16 0x04 0x00 0x01 0x68 0x09 0x00 0x02 0xf2 0x00 0x1f 0xf0 0x22 0xf2, 0x507: 0x00, 0x52A: 0x00, 0x53F: 0x04
Clock chip state 0, status: SYSINCAL XAXB_ERR LOL CAL_PLL IN0 IN_SEL_0 FASTLOCK_STATUS

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x1f7caf52 (528265042) should be ~125 MHz
0x1034 rx_clk: 0x00000000 (0) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x1f7caf52 (528265042) should be ~125 MHz
0x103C tx_clk: 0x00000000 (0) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip good (IN0 - external 10 MHz clock)

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x44 0x00 0xd0 0x01 0x1f 0xff 0x22 0xf2, 0x507: 0x3f, 0x52A: 0x01, 0x53F: 0x00
Clock chip state 1, status: LOS_IN2 OOF_IN2 IN0 IN_SEL_REGCTRL IN_SEL_0

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735a3b (125000251) should be ~125 MHz
0x1034 rx_clk: 0x07735a3c (125000252) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a3b (125000251) should be ~125 MHz
0x103C tx_clk: 0x07735a3c (125000252) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

=== CDM clock status ===

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
</pre>

* clock chip not loaded, not running:

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x0f 0x16 0x04 0x00 0x01 0x68 0x19 0x00 0x02 0xf2 0x00 0x1f 0xf0 0x22 0xf2, 0x507: 0x00, 0x52A: 0x00, 0x53F: 0x04
Clock chip state 0, status: SYSINCAL XAXB_ERR LOL CAL_PLL IN0 IN_SEL_0 FASTLOCK_STATUS

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735851 (124999761) should be ~125 MHz
0x1034 rx_clk: 0x00000000 (0) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x0127fefa (19398394) should be ~125 MHz
0x103C tx_clk: 0x00000000 (0) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip uses internal clock (IN1 - internal oscillator), observe rx_clk frequency is not same as others

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0x7f, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN1 IN_SEL_1 HOLD_HIST_VALID

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735853 (124999763) should be ~125 MHz
0x1034 rx_clk: 0x07735b0a (125000458) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735853 (124999763) should be ~125 MHz
0x103C tx_clk: 0x07735852 (124999762) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip uses SFP recovered clock (IN2 - sfp rx recovered clock), observe mgt_rx_ref_clk_raw (CDM 125 MHz oscillator) is different from others (SFP RX recovered clock)

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0xbf, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN2 IN_SEL_1 HOLD_HIST_VALID
^C
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773581b (124999707) should be ~125 MHz
0x1034 rx_clk: 0x07735ad6 (125000406) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735ad7 (125000407) should be ~125 MHz
0x103C tx_clk: 0x07735ad6 (125000406) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
^C
</pre>

=== CDM link status, PRBS test mode ===

<pre>
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
</pre>
* fiber disconnected, no link:
<pre>
CDM firmware: 0xbb2f0ae7
0x1000 SFP c.c. status: 0x00000031
CLK_IN_SEL_LS 0x1
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x00000025
sfp_mod_absent_N 1
sfp_rx_los_N 0
link_power_good 1
rx_link_up 0
rx_receiving_data 0
rx_error 1
rx_lnk_up_and_running 0
tx_link_up 0
tx_sending_data 0
tx_link_up_and_running 0
link_up_and_running 0
0x1014 SFP link data: 0x466a8187
rx_data 0x8187
k28p1_k28p5 0
rx_ctrl0 0x1
rx_ctrl1 0x1
rx_ctrl3 0x1
tx_state 0x1
rx_state 0x6
rx_receiving_data 0
rx_link_up 0
rx_error 1
rx_link_rst 0
0x2000 link test mode: 0x00000200, seconds: 0x00079093, errors: 0xffffffff
</pre>
* fiber connected, good link:
<pre>
CDM firmware: 0xbb2f0ae7
0x1000 SFP c.c. status: 0x000000b2
CLK_IN_SEL_LS 0x2
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x000007dc
sfp_mod_absent_N 0
sfp_rx_los_N 0
link_power_good 1
rx_link_up 1
rx_receiving_data 1
rx_error 0
rx_lnk_up_and_running 1
tx_link_up 1
tx_sending_data 1
tx_link_up_and_running 1
link_up_and_running 1
0x1014 SFP link data: 0x35c02774
rx_data 0x2774
k28p1_k28p5 0
rx_ctrl0 0x0
rx_ctrl1 0x0
rx_ctrl3 0x0
tx_state 0x3
rx_state 0x5
rx_receiving_data 1
rx_link_up 1
rx_error 0
rx_link_rst 0
0x2000 link test mode: 0x00000200, seconds: 0x00078a8c, errors: 0x00000000
</pre>

=== GDM link status, PRBS test mode ===

<pre>
1 link connected, no errors:
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
GDM firmware: 0x6b2ee010
0x1014: 0x00000008, 18: 0x00000008, 1C: 0x00000000, 24: 0x00000fff
0x2000: 0x00000200, time: 0x00078aa4, errors:
0xffffffff 0xffffffff 0xffffffff
0x00000000 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
</pre>

=== CDM link status ===

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test0

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0xbf, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN2 IN_SEL_1 HOLD_HIST_VALID

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x077357a0 (124999584) should be ~125 MHz
0x1034 rx_clk: 0x07735a5c (125000284) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a5b (125000283) should be ~125 MHz
0x103C tx_clk: 0x07735a5b (125000283) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM firmware: 0x6d2ef81a
0x1000 SFP c.c. status: 0x000000b2
CLK_IN_SEL_LS 0x2
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x000007dc
sfp_mod_absent_N 0
sfp_rx_los_N 0
link_power_good 1
rx_link_up 1
rx_receiving_data 1
rx_error 0
rx_lnk_up_and_running 1
tx_link_up 1
tx_sending_data 1
tx_link_up_and_running 1
link_up_and_running 1
0x1014 SFP link data: 0x35c6bcbc
rx_data 0xbcbc
k28p1_k28p5 0
rx_ctrl0 0x3
rx_ctrl1 0x0
rx_ctrl3 0x0
tx_state 0x3
rx_state 0x5
rx_receiving_data 1
rx_link_up 1
rx_error 0
rx_link_rst 0
0x2000 link test mode: 0x00000000, seconds: 0x00001671, errors: 0xffffffff

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 15
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
reg[15] is 0x0033bcbc (3390652)
</pre>

=== GDM link status ===

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test0

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x44 0x00 0xd0 0x01 0x1f 0xff 0x22 0xf2, 0x507: 0x3f, 0x52A: 0x01, 0x53F: 0x02
Clock chip state 1, status: LOS_IN2 OOF_IN2 IN0 IN_SEL_REGCTRL IN_SEL_0 HOLD_HIST_VALID

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735a0a (125000202) should be ~125 MHz
0x1034 rx_clk: 0x07735a0a (125000202) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a0a (125000202) should be ~125 MHz
0x103C tx_clk: 0x07735a0a (125000202) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly

register 0x1018 bit 0x800

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM firmware: 0x6d2ebce6
0x1014: 0x00000800, 18: 0x00000800, 1C: 0x00000000, 24: 0x00000fff
0x2000: 0x00000000, time: 0x00003d2f, errors:
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff

link data alternates 0xbcbc and 0x1cbc

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 22
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
reg[22] is 0xbcbc93ab (-1128492117)

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 22
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
reg[22] is 0x1cbc1aaf (482089647)
root@gdm0:~#

</pre>

=== Run trg and tsm ===

<pre>
on the GDM:

ssh root@gdm00
./test_cdm.exe --gdm-clocks
./test_cdm.exe --load-cc
./test_cdm.exe --cc
./test_cdm.exe --reset-mgt
./test_cdm.exe --gdm-clocks
./test_cdm.exe --writereg 2 0xff # enable LEMO NIM inputs
./test_cdm.exe --writereg 3 0xba54 # LEDs: lemo1, lemo2, trg, tsm
./test_cdm.exe --writereg 4 0x99 $ # LEMO out is trg_in_pulse
./test_cdm.exe --writereg 9 0x32010 # trg and tsm from trg_pulser and tsm_pulse
./test_cdm.exe --writereg 24 1250000 # trg pulser 100 Hz
./test_cdm.exe --writereg 26 125000000 # tsm pulser 1 Hz
./test_cdm.exe --writereg 23 0x40000000 # route trg_in and tsm_in to qsfp tx bits 0 and 1
./test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz

on the CDM:

ssh root@cdm01
./test_cdm.exe --cdm-clocks
./test_cdm.exe --load-cc
./test_cdm.exe --reset-mgt
./test_cdm.exe --cdm-clocks
./test_cdm.exe --cdm-link # issue --reset-mgt on CDM and GDM until link is good
./test_cdm.exe --writereg 2 0xff # enable LEMO NIM inputs
./test_cdm.exe --writereg 3 0xba54 # enable LEDs: lemo1, lemo2, trg, tsm
./test_cdm.exe --writereg 4 0x99 # enable LEMO output trg_in
./test_cdm.exe --writereg 9 0x8040 # enable trg_in and tsm_in from sfp rx bits 0 and 1
./test_cdm.exe --writereg 7 0xff00 # drive VX LVDS lines to logic level 0
./test_cdm.exe --writereg 8 0x2 # VX LVDS with trg and tsm, misrouted in second VX
#./test_cdm.exe --writereg 8 0x7 # VX LVDS with tsm
./test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz
</pre>

switch GDM and CDM to packetizer trg and tsm:

<pre>
on the GDM:

/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x00000000

on the CDM:

/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x0804
/home/dsdaq/online/ds-dm-software/test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz
</pre>

=== Run packet loopback ===

GDM CPU -> fifo_to_fpga -> GDM QSFP -> CDM SFP -> fifo_from_fpga -> CDM CPU

On the GDM: (CDM is connected to first QSFP port)

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --load-cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --reset-mgt
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 32 1 ### tell GDM to use first QSFP port
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link ### confirm link status is "3"
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 29 2 ### enable GDM packet data injection
/home/dsdaq/online/ds-dm-software/test_cdm.exe --test-fifo-write-loop
</pre>

One the CDM:

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --load-cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --reset-mgt
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 29 3 ### enable CDM packet data injection
/home/dsdaq/online/ds-dm-software/test_cdm.exe --test-fifo-read
</pre>

== fecdm ==

=== Power up and clock management ===

* Clock power up and setup management is handled by fecdm state machine
* GDM clock setup sequence:
<pre>
- on power up, CC is not initialized, XAXB clock is not running
- on GDM frontend start (in HandleInit()), CC is loaded and reset
- CC will lock to IN0 (10 MHz external clock) or IN1 (125 MHz internal oscillator)
- if locked to IN0, CC will enter and exit the HOLDOVER mode if 10 MHz clock is unstable
</pre>
* CDM clock setup sequence:
<pre>
- on power up, CC is not initialized, XAXB clock is not running
- fecdm detects this, loads the CC configuration registers, resets the CC
- sets the CC input IN1, DSDM internal oscillator
- if SFP is in good state, SFP MGT is running, CC is happy with IN2 input (no IN2_OOF, IN2_LOS), SFP recpvered clock (clk_rx_mgt) frequency is close to 125 MHz:
- switches CC to input IN2 (SFP recovered clock, clk_rx_mgt)
- waits for CC locked to IN2 (no LOL, no HOLD). (this provides stable SFP tx clock).
- resets the SFP MGT, requests GDM MGT reset (via ODB)
- MGT reset sequence: GDM MGT reset 1, CDMnn MGT reset 1, GDM MGT reset 0, CDM MGT reset 0
- CDM MGT rx_state cycles from 0 to 5. if it goes to 6 or never reaches 5, GDM MGT reset is requested again.
- during this sequence, clk_rx_mgt may wobble, CC may enter HOLDOVER mode, VX clock should stay steady.
- once CC is locked to IN2, SFP MGT rx_state is 5, link is ready for use.
</pre>
* Request changes for Ian
<pre>
17 Apr 2025:
- CDM SFP MGT rx_state is getting stuck in 4 (waiting for alternating idle chars), do we need this? go to state 5 without waiting?
- GDM QSFP MGT reset 1 does not seem to do anything, change it to cause all CDM MGTs to enter an error state, e.g. start transmitting invalid K-codes. transmitting all 0 or all 1 will break the recovered clock, so maybe do not do that.
- having all CDM SFP MGTs go to error state immediatley after GDM QSFT MGT reset 1 will greatly simplify the link reset sequence.
</pre>

=== Trigger configuration ===

* GDM
* CDM in SFP trigger mode:
** SFP K-code encoded sfp_rx_trg, sfp_rx_tsm are routed to LVDS vx_tx_trg and vx_tx_tsm
** GDM TRG, TSM (and all other) packets are routed to LVDS serial data to VX
** to make counters count and lights blink:
** CDM TRG and TSM trigger mask is set to sfp_rx_trg, sfp_rx_tsm and vx_tx_trg_done, vx_tx_tsm_done
** CDM LEDs display trg_in_pulse, tsm_in_pulse
** output of TRG and TSM packets is disabled (they come from the GDM)

= dsvslice integration =

== VX setup ==

* general
** Start acq from user code = y, all others = n (as of Mar 2023: will start when first trigger received)
** Use NIM IO = y
** Use external clock = y
** LVDS quartet is input = n, y, n, y
** LVDS quartet mode = User, User, User, User

* trigger from front panel NIM:
** Trigger on external signal = y, all others = n
** connect CDM EXT_OUT(2) to VX "TrigIn"

* trigger from LVDS "Sync" mode
** Trigger on LVDS Sync signal = y, all others = n
** LVDS quartet mode = User, Sync, User, User

* trigger from LVDS "User" mode
** Trigger on LVDS pair 12 signal = y, all others = n
** LVDS quartet is input = n, y, n, y
** LVDS quartet mode = User, User, User, User

== GDM setup ==

* GDM is gdm0
* set inputs to NIM mode
* set outputs to TTL mode (this GDM has wrong NIM output circuit)
* use top QSFP slot, connect split cable 1 into CDM01, cable 2 into CDM02
* connect non-inverted NIM trigger signal to top-LEMO-left EXT_IN_LV(1)
* connect non-inverted NIM TSM signal to top-LEME-right EXT_IN_LV(2)
* GDM LEDs: TRIG, TSM, trigger enabled, trigger_out
* GDM LEMO_OUT: trigger, trigger

== CDM setup ==

* set CDM LEMO inputs to NIM
* set CDM LEMO outputs to NIM
* CDM01 is cdm0
* CDM02 is cdm1
* connect GDM fiber links to SFP port
* connect 1st VX port of CDM01 to VX1
* connect 1st VX port of CDM02 to VX2
* connect LEMO EXT_OUT(2) to VX "TrigIn", CDM01 to VX1, CDM02 to VX2
* power up
* CDM LEDs: GDM TRIG, GDM TSM, trigger enabled, trigger out
* CDM LEMO_OUT: gdm_trg, trigger

== After power up ==

* start the CDM frontend from the MIDAS "Programs" page. To start manually, see the Start Command on the Programs page.
* CDM frontend should enable the VX clock, disable the trigger
* from the MIDAS status page, goto the CDM page
* outdated: in the CDMx data tables, the 2nd number should read 0x35c08008, if it does not and the last 4 digits randomly change, reset the GDM links
*if the FEs complain - do in order: for GDM, CDM01, CDM02, ..., press "reset mgt" of each board, then press "unreset mgt", if it does not help, STOP HERE
* start a run
* CDM frontend will enable the trigger
* GDM frontend will enable the trigger
* LEDs on the GDM should flash, LEDs on the CDM should flash, TrigIn and TrigOut of the VX should flash
* stop a run
* GDM frontend will disable the trigger
* CDM frontend will disable the trigger

== Phase measurement ==

* pip3 install matplotlib
* pip3 install scipy
* export PYTHONPATH=$HOME/packages/midas/python
* #git clone https://github.com/J033X071C/PhaseMeasurement
* git clone https://bitbucket.org/team-ds-dm/phasemeasurement.git
* cd phasemeasurement
* python3 ./phaseMeasurement.py --help
<pre>
daq00:PhaseMeasurement$ python3 ./phaseMeasurement.py --help
usage: phaseMeasurement.py [-h] fileName numberEvents numberVX sizeEvents stopEvent minHist maxHist numberBin writeToTXT saveAsPDF

Read data from midas file (in .lz4 format) to calculate phase between the clock of VX1 and VX2

positional arguments:
fileName Name of the file we want to read data from (Example: run00389.mid.lz4)
numberEvents Number of events recorded in the file
numberVX Number of VX used in this run (usually 2...)
sizeEvents Number of points per event
stopEvent Number of events you want to go through to calculate phase
minHist Minimal value for the x axis of the phase measurement histogram (in ns)
maxHist Maximal value for the x axis of the phase measurement histogram (in ns)
numberBin Number of bins wanted for the generated histogram
writeToTXT Write argument as yes to generate text file with results of calculation
saveAsPDF Save generated plots to PDF files

optional arguments:
-h, --help show this help message and exit
daq00:PhaseMeasurement$
</pre>
* try an old file with
* python3 ./phaseMeasurement.py run00877.mid.lz4 10000 2 10000 500 -20 20 81 yes yes
<pre>
bin size = 0.494 ns
num_events = 347
mean = -1.705 ns
rms = 3.087 ns
mean_error = 0.166 ns
centroid = -1.706 ns.
width (sigma) = 0.363 ns.
error on the centroid = 0.016558 ns.
</pre>
* ls -l *.txt *.pdf
<pre>
dsdaq@dsvslice:~/online/PhaseMeasurement$ ls -l *.txt *.pdf
-rw-rw-r-- 1 dsdaq dsdaq 64728 Dec 14 16:56 run00877.mid.lz4_Plots.pdf
-rw-rw-r-- 1 dsdaq dsdaq 274 Dec 14 16:56 run00877.mid.lz4.txt
dsdaq@dsvslice:~/online/PhaseMeasurement$
</pre>
* scope settings (from email message)
<pre>
From fcote-lortie@triumf.ca Thu Dec 15 17:11:46 2022
From: Francis Cote-Lortie <fcote-lortie@triumf.ca>
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: Re: How to use scope as a waveform generator
Date: Fri, 16 Dec 2022 01:11:44 +0000

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display) by pressing on the button. It will go from 0 to 1.

The settings that we are using right now are:
Type of waveform: Sine wave
Offset: 0 V
Amplitude: 1 Vpp
Frequency: 50 kHz
Noise: 0 V
________________________________
From: Francis Cote-Lortie <fcote-lortie@triumf.ca>
Sent: Thursday, December 15, 2022 4:58 PM
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: Re: How to use scope as a waveform generator

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display)

The settings that we are using right now are:
Type of waveform: Sine wave
Offset: 0 V
Amplitude: 1 Vpp
Frequency: 50 kHz
Noise: 0 V
________________________________
From: Francis Cote-Lortie
Sent: Thursday, December 15, 2022 4:54 PM
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: How to use scope as a waveform generator

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display)
</pre>

== Standalone link test ==

<pre>
CDM: program clock chip
busybox devmem 0x80011000 32 0x8
busybox devmem 0x80011000 32 0x0
/home/dsdaq/si5394-i2c-file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b

GDM, CDM: link reset
busybox devmem 0x80011008 32 1

GDM, CDM: release reset
busybox devmem 0x80011008 32 0

CDM: link status (NOTE: SFP LOS and mod_absent are swapped!!!)
busybox devmem 0x80011010 32
0x00000024 <- fiber plugged
0x00000025 <- fiber unplugged
0x00000027 <- SFP unplugged
0x000007DC <- successful link with GDM

CDM: link state machine and data
busybox devmem 0x80011014 32
0x35C06FF6

CDM: set link to counting mode
busybox devmem 0x80012000 32 0x101
busybox devmem 0x80012000 32 0x100

CDM: time counter and error counter
root@cdm1:~# busybox devmem 0x80012000 32
0x00000100 <--- link mode
root@cdm1:~# busybox devmem 0x80012004 32
0x0000058C <--- seconds counter
root@cdm1:~# busybox devmem 0x80012004 32
0x0000058D
root@cdm1:~# busybox devmem 0x80012008 32
0x00000000 <--- error counter

GDM: no link
root@gdm0:~# busybox devmem 0x80011014 32
0x00000000
root@gdm0:~# busybox devmem 0x80011018 32
0x00000000
root@gdm0:~# busybox devmem 0x8001101c 32
0x00000000
root@gdm0:~# busybox devmem 0x80011024 32
0x00000FFF
root@gdm0:~#

GDM: good link channel 10, counting mode
root@gdm0:~# busybox devmem 0x80012000 32 0x101
root@gdm0:~# busybox devmem 0x80012000 32 0x100
root@gdm0:~# busybox devmem 0x80012008 32
0x3A8B68C2
root@gdm0:~# busybox devmem 0x80012008 32
0x42E03BEF
root@gdm0:~# busybox devmem 0x8001200c 32
0xDA090972
root@gdm0:~# busybox devmem 0x8001200c 32
0xDE6F22E9
root@gdm0:~# busybox devmem 0x80012019 32
Bus error
root@gdm0:~# busybox devmem 0x80012010 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012014 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012018 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x8001201c 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012020 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012024 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012028 32
0x00000000
root@gdm0:~# busybox devmem 0x8001202c 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012030 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012034 32
0xFFFFFFFF
root@gdm0:~#
</pre>

mapping of link channels:

<pre>
qsfp0 lane0 - 0x0100 - link 8
qsfp0 lane1 - 0x0200 - link 9
qsfp0 lane2 - 0x0400 - link 10
qsfp0 lane3 - n/c
qsfp1 lane0 - 0x0800 - link 11
qsfp1 lane1 - 0x0010 - link 4
qsfp1 lane2 - 0x0020 - link 5
qsfp1 lane3 - n/c
qsfp2 lane0 - 0x0040 - link 6
qsfp2 lane1 - 0x0080 - link 7
qsfp2 lane2 - 0x0001 - link 0
qsfp2 lane3 - n/c
qsfp3 lane0 - 0x0002 - link 1
qsfp3 lane1 - 0x0004 - link 2
qsfp3 lane2 - 0x0008 - link 3
qsfp3 lane3 - n/c
</pre>

script to start the test with 2 CDMs:

<pre>
ssh dsdaq@dsvslice
ssh root@gdm0 busybox devmem 0x80011008 32 1
ssh root@cdm0 busybox devmem 0x80011008 32 1
ssh root@cdm1 busybox devmem 0x80011008 32 1
ssh root@gdm0 busybox devmem 0x80011008 32 0
ssh root@cdm1 busybox devmem 0x80011008 32 0
ssh root@cdm0 busybox devmem 0x80011008 32 0
ssh root@gdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
ssh root@cdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
ssh root@cdm1 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
</pre>

== LEMO trigger GDM to CDM to VX ==

on the GDM:

<pre>
/home/dsdaq/si5394-i2c-file /home/dsdaq/GDM_v1.0_IN0_EXT1_and_IN1_fixed_Si5394-RevA-Registers.txt 0 0x6b
busybox devmem 0x80011008 32 0x1
busybox devmem 0x80011008 32 0x0
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x80001230 # QSFP TX fixed pattern
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 2 0xff # enable LEMO inputs
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 3 0x7654 # enable LED, one per LEMO input
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x0F0F # enable LEMO to trg_in and tsm_in
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x40001230 # enable trg_in and tsm_in output to QSFP
</pre>

on the CDM:

<pre>
/home/dsdaq/si5394-i2c-file /home/dsdaq/CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
/home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp
busybox devmem 0x80011008 32 1
busybox devmem 0x80011008 32 0
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x8040 # trg_in from sfp[0], tsm_in form sfp[1]
/home/dsdaq/online/ds-dm-software/test_cdm.exe 11 # trg_in counter
/home/dsdaq/online/ds-dm-software/test_cdm.exe 12 # tsm_in counter
</pre>

== GPS receiver VCL-2705 ==

* Valiant VCL-2705 GPS receiver
* https://www.valiantcom.com/time-distribution/gps-receiver-irig-b.html
* USB connection is /dev/ttyACM0, 115200 bps, "GNSSAUX" prompt
* minicom -D /dev/ttyACM0 -b 115200
* user manual download instructoins - see sheet of paper with user name and password in the shipping box
* usb commands:
<pre>
gnss-help
gnss-showver -> F/W: Ver 1.7 Feb 17 2020 16:20:43

gnss-showselftest -> no antenna connected
Overall : FAIL
EPROM Test: PASS
Antenna : NOT DETECTED
GNSS : COMMUNICATION OK

gnss-showsettings
GNSS NMEA BAUDRATE :115200
GNSS ANTENNA LENGTH :30 meters
GNSS USER CONFIGURED DELAY:-65 nanoseconds
GNSS 1PPS PULSE WIDTH :200 milliseconds
GNSS MODE :GPS
GNSS STATUS :STATIONARY

GNSSAUX> gnss-showserial
SERIAL :2704H01V17MAX310

GNSSAUX> gnss-showmode
GNSS MODE : GPS

GNSSAUX> gnss-showstatus
GNSS STATUS: STATIONARY

GNSSAUX> gnss-showalarms
CURRENT ALARMS GNSS
ANTENNA : NOT DETECTED

GNSSAUX> gnss-showerrors
CURRENT ALARMS GNSS
ANTENNA : **NOT DETECTED
ERROR STATISTICS GNSS
RMC GOOD DURATION : Secs 0
RMC BAD DURATION : Secs 0
LOCK GOOD SECS DURATION : Secs 0
LOCK BAD SECS DURATION : Secs 0
SATINFO GOOD ITERATIONS :0
SATINFO REJECT ITERATIONS :0
SATINFO NOTALKER ITERATIONS :0
CURRENT MONITOR STATE GNSS
STATE :Phase-1 HUNTING ANTENNA DETECT

GNSSAUX> gnss-showsatinfo
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-showsats
Total Sats: 0

GNSSAUX> gnss-showmyloc
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-show1ppsstate
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-showjamstatus
Not Available !

GNSSAUX> gnss-showspoofstatus
Not Available !

--- antenna connected, can see the sky ---

GNSSAUX> gnss-showselftest
Overall : PASS
EPROM Test: PASS
Antenna : DETECTED
GNSS : COMMUNICATION OK

GNSSAUX> gnss-showalarms
CURRENT ALARMS GNSS
ANTENNA : DETECTED
GNSS LOCK : AVAILABLE

GNSSAUX> gnss-showerrors
CURRENT ALARMS GNSS
ANTENNA : DETECTED
GNSS LOCK : AVAILABLE
ERROR STATISTICS GNSS
RMC GOOD DURATION : Mins 1,Secs 34
RMC BAD DURATION : Secs 55
LOCK GOOD SECS DURATION : Mins 1,Secs 34
LOCK BAD SECS DURATION : Secs 55
SATINFO GOOD ITERATIONS :3
SATINFO REJECT ITERATIONS :0
SATINFO NOTALKER ITERATIONS :0
CURRENT MONITOR STATE GNSS
STATE :Phase-4 NORMAL OPERATION, Monitoring GNSSLOCK

GNSSAUX> gnss-showsatinfo
GNSS MODE :GPS
SATELLITE INFORMATION TALKER:GPS
NMEA ID : 01-32
NO OF SATELLITES IN VIEW: 08
NO OF XXGSV MSGS : 03
SatNo PRN NO (SV ID) ELEVATION (degs) AZIMUTH (degs) C/No (SNR)
1 0 0 0 0
2 0 0 0 0
3 0 0 0 0
4 0 0 0 0
5 0 0 0 0
6 0 0 0 0
7 0 0 0 0
8 0 0 0 0
<<<<<< End of Sat Info >>>>>>

GNSSAUX> gnss-showsats
Total Sats: 8

GNSSAUX> gnss-showmyloc
GNSS RECEIVER LOCATION:
Latitude : 4914.81911
Longitude: 12313.69595

GNSSAUX> gnss-show1ppsstate
GPS 1PPS STATE: LOCKED

GNSSAUX> gnss-showmyloc

GNSS RECEIVER LOCATION:
Latitude : 4914.80688
Longitude: 12313.69531

enter into google maps search box as: 49 14.80688, -123 13.69531, observe the space, the moved dot and the minus.

GNSSAUX> gnss-resetgnss

Executing....Please Wait....
$$$$$END
</pre>

* 1PPS BNC output: period 1 sec, pulse width 200 ms, 3.3V into 1MOhm, 1.38V into 50Ohm.
* IRIG-B BNC output: 5.6V into 1MOhm, 2.4V into 50Ohm.
* IRIG-B format selector: default is all up.

== Rb clock PRS10 ==

* https://www.thinksrs.com/products/prs10.html
* 10 MHz output is sine wave around 5V peak to peak
* 1PPS BNC output is 10 usec pulse, 5V into 1MHohm.
* RS232 connection: minicom -D /dev/ttyUSB0 -b 9600
* ser2net config: localhost,3001:raw:600:usb-5-2-1.0:9600 -XONXOFF -RTSCTS LOCAL
* ssh daq13, cd /home/olchansk/git/ser2net, ./ser2net -c ~/daq/ds/ser2net.conf -d
* ssh daq13, cd ~/daq/ds, python3 prs10.py
* RS232 commands:
<pre>
ID?
PRS10_3.56_SN_105719
VB1
SN?
RS1 -- reset
ST? -- status
FC? -- 10MHz OCXO drive voltage DAC settings
DS? -- "detected signals"
GA? -- gain of frequency lock loop between ovenized oscillator and Rb cell, 0=use ovenized oscillator only
MO? -- magnetic offset of the Rb cell, range 2300..3600, if out of range, unit must be set to different operating mode, see prs10m.pdf
MR? -- magnetic read
TT? -- time-tag, time in ns between 1PPS out and 1PPS in
TS? -- time slope, ???
TO? -- time offset, ???
PS? -- pulse slope, ???
PL? -- 0=phase lock off, 1=phase lock on, lock to 1PPS input
PT? -- phase lock integrator time constant, PT8 is integrator time constant 18.2 hours, natural time constant 2.25 hours
PF? -- phase lock stability factor, PF2 is "1"
PI? -- phase lock integrator

Analog to digital 12 bit ADC, values 0.000 to 4.998

AD0? -- Spare (J204)
AD1? -- +24V(heater supply) divided by 10.
AD2? -- +24V(electronics supply) divided by 10
AD3? -- Drain voltage to lamp FET divided by 10
AD4? -- Gate voltage to lamp FET divided by 10
AD5? -- Crystal heater control voltage
AD6? -- Resonance cell heater control voltage
AD7? -- Discharge lamp heater control voltage
AD8? -- Amplified ac photosignal
AD9? -- Photocell’s I/V converter voltage divided by 4
AD10? -- Case temperature (10 mV/°C)
AD11? -- Crystal thermistors
AD12? -- Cell thermistors
AD13? -- Lamp thermistors
AD14? -- Frequency calibration pot / external calibration voltage
AD15? -- Analog ground

A/D via CPU E-port:

AD16? -- Varactor voltage for 22.48 MHz VCXO (inside RF synthesizer) / 4
AD17? -- Varactor voltage for 360 MHz VCO (output of RF synthesizer) / 4
AD18? -- Gain control voltage for amplifier which drives frequency multiplier / 4
AD19? -- RF synthesizer’s lock indicator voltage (nominally 4.8 V when locked )

ST?

ST1 : Power supplies and Discharge Lamp
ST1 bit, Condition which sets bit, Corrective Action
0 -- +24 for electronic < +22 Vdc
1 -- +24 for electronics > +30 Vdc
2 -- +24 for heaters <+22 Vdc
3 -- +24 for heaters > +30 Vdc
4 -- Lamp light level too low
5 -- Lamp light level too high
6 -- Gate voltage too low
7 -- Gate voltage too high

ST2: RF Synthesizer
ST2 bit, Condition which sets bit, Corrective Action
0 -- RF synthesizer PLL unlocked
1 -- RF crystal varactor too low
2 -- RF crystal varactor too high
3 -- RF VCO control too low
4 -- RF VCO control too high
5 -- RF AGC control too low
6 -- RF AGC control too high
7 -- Bad PLL parameter

ST3: Temperature Controllers
ST3 bit, Condition which sets bit
0 -- Lamp temp below set point
1 -- Lamp temp above set point
2 -- Crystal temp below set point
3 -- Crystal temp above set point
4 -- Cell temp below set point
5 -- Cell temp above set point
6 -- Case temperature too low
7 -- Case temperature too high

ST4: Frequency Lock-Loop Control
ST4 bit, Condition which sets bit
0 -- Frequency lock control is off
1 -- Frequency lock is disabled
2 -- 10 MHz EFC is too high
3 -- 10 MHz EFC is too low
4 -- Analog cal voltage > 4.9 V
5 -- Analog cal voltage < 0.1
6 -- not used
7 -- not used

ST5: Frequency Lock to External 1pps
ST5 bit, Condition which sets bit
0 -- PLL disabled
1 -- < 256 good 1pps inputs
2 -- PLL active
3 -- > 256 bad 1pps inputs
4 -- Excessive time interval
5 -- PLL restarted
6 -- f control saturated
7 -- No 1pps input

ST6: System Level Events
ST6 bit and Condition which sets bit
0 Lamp restart
1 Watchdog time-out and reset
2 Bad interrupt vector
3 EEPROM write failure
4 EEPROM data corruption
5 Bad command syntax
6 Bad command parameter
7 Unit has been reset
</pre>
* ST? on warm start
<pre>
received: PRS_10
received: 255,255,255,243,34,255
received: 0,0,0,1,34,0
...
received: 0,0,0,1,34,0
received: 0,0,0,0,2,0
...
received: 0,0,0,0,2,0
received: 0,0,0,0,4,0
</pre>
* ST? on loss of external 1PPS
<pre>
...
received: 0,0,0,0,4,0
disconnect 1PPS input
received: 0,0,0,0,132,0
...
reconnect 1PPS input
received: 0,0,0,0,132,0
received: 0,0,0,0,4,0
...
</pre>
* ST? on coldish start
<pre>
daq13:ds$ python3 prs10.py
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,0,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 0,0,1,1,34,0 ] old [ 80,0,0,1,34,1 ] counter: 0
received [ 0,0,0,1,34,0 ] old [ 0,0,1,1,34,0 ] counter: 0
received [ 0,0,16,1,34,0 ] old [ 0,0,0,1,34,0 ] counter: 6
received [ 0,0,20,1,34,0 ] old [ 0,0,16,1,34,0 ] counter: 5
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 8
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 3
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 49
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 250
...
</pre>
* ST? on cold start, note: no bump in the 10MHz clock as reported by DS-DM clock chip
<pre>
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,21,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 64,0,21,1,34,0 ] old [ 80,0,21,1,34,1 ] counter: 0
received [ 0,0,21,1,34,0 ] old [ 64,0,21,1,34,0 ] counter: 0
received [ 0,0,20,1,34,0 ] old [ 0,0,21,1,34,0 ] counter: 158
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 23
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 47
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 40
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 249
</pre>

= det fac integration test =

Connections:
* GPS receiver "IRIG-B SEL" both switches "up" - both "on", IRIG-B format B004
* GPS receiver USB-B -> long cable -> daq13 USB-A
* GPS receiver "1PPS out" -> long BNC cable -> BNC-T -> scope (5V, no 50ohm) and Rb clock BNC "1PPS in"
* GPS receiver "IRIG-B 50ohms" -> long BNC cable -> BNC-T -> scope (5V, no 50ohm) and DS-DM LEMO input 1 (TTL mode)
* Rb clock RS232 -> RS232 straight cable -> RS232-to-USB adapter -> daq13 USB-A
* Rb clock "1PPS out" BNC -> scope (5V, no 50ohm, trig threshold rising edge 2V)
* Rb clock "10MHz output 50 Ohm" BNC -> lemo -> lemo-T -> scope (sine wave, 5V, no 50ohm) and DS-DM clock input.

Programs to run:
* on daq13: cd /home/olchansk/git/ser2net, ./ser2net -c ~/daq/ds/ser2net.conf -d
<pre>
localhost,3001:raw:600:usb-5-2-1.0:9600 -XONXOFF -RTSCTS LOCAL
</pre>
* on daq13: cd /home/olchansk/daq/ds, python3 prs10.py ### connects to ser2net
<pre>
daq13:ds$ python3 prs10.py
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,21,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 64,0,21,1,34,0 ] old [ 80,0,21,1,34,1 ] counter: 0
received [ 0,0,21,1,34,0 ] old [ 64,0,21,1,34,0 ] counter: 0
received [ 0,0,20,1,34,0 ] old [ 0,0,21,1,34,0 ] counter: 158
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 23
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 47
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 40
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 249
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 12096
received [ 0,0,0,0,20,0 ] old [ 0,0,0,0,132,0 ] counter: 2
received [ 0,0,0,0,148,0 ] old [ 0,0,0,0,20,0 ] counter: 0
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,148,0 ] counter: 575
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 11449
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 2755
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 34386
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 41035
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 113401
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 33375
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 54767
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 85059
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 33222
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 119234
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 121990
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 128184
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 56002
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 428237
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 8250
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 1388
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 30506
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 142704
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 179451
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 106182
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 68747
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 65424
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 157587
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 6932
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 1388
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 20255
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 4
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 225941
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 72183
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 26970
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 1
39287
</pre>
* on ds-dm: ./test_cdm.exe --irigb ### note sbs mismatch is because I should wrap around at 16 bits
<pre>
dataframe: S00000000S100001010S...S001010101S000001000S, sec: 00, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16724 should be 82260
dataframe: S10000000S100001010S...S101010101S000001000S, sec: 01, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16725 should be 82261
dataframe: S01000000S100001010S...S011010101S000001000S, sec: 02, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16726 should be 82262
dataframe: S11000000S100001010S...S111010101S000001000S, sec: 03, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16727 should be 82263
dataframe: S00100000S100001010S...S000110101S000001000S, sec: 04, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16728 should be 82264
dataframe: S10100000S100001010S...S100110101S000001000S, sec: 05, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16729 should be 82265
</pre>

= DS-IOGC GPS interface board =

* Rev0 git repository: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0/-/tree/main?ref_type=heads
* Rev0 schematics: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0/-/blob/main/Altium/Project%20Outputs%20for%20DS-IOGC-Rev0/SCH-DS-IOGC-Rev0.pdf?ref_type=heads
* Rev0 schematics: [[:Image:SCH-DS-IOGC-Rev0.pdf|SCH-DS-IOGC-Rev0]]
* Rev1 git repository: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev1/
* Rev1 schematics: [[:Image:SCH-DS-IOGC-Rev1.pdf|SCH-DS-IOGC-Rev1]]

== Changes Rev0 to Rev1 ==

<pre>
From: Peter Margetak <pmargetak@triumf.ca>
Subject: IOGC REV1 review
Date: Wed, 4 Sep 2024 07:31:19 +0000

Hi Konstantin,
Pls have a look at SCH for new rev. I'd like to send it to mfr next week so if you can comment by early next week. Meanwhile I work on layout and other stuff.

Changes:

New ICs - all powered +5V
U20 - inverters for RUclk RX/TX
U21 - non inverting line driver for RU-1pps-out (so you don't have to route if via GDM to see it on scope)
U22 - non inverting buffer for ext 1pps input

All Lemo connectors have the same position but they are double lemos now => new panel needed
@Marek Walczak<mailto:mwalczak@triumf.ca> you can print it ahead once pcb is done + update IOGC docs and panel description

J2A/B - Test ports for RU-1pps in and out
J5A/B - inputs for external GPS data and External source of 1pps
J6A/B - aux in/out for GDM

SW1 - no change - select RX/TX USB/GDM
SW2 - select latch sensitivity for rising/falling edge
SW3 - select source of GPS data (opto or ext) AND select source of 1pps input (latch or ext)

p.
</pre>

== PRS-10 Rb clock device ==

The Rb clock PRS-10 device provides these connections:
<pre>
RS232 RX input - serial communication, non-standard RS232
RS232 TX output - serial communication, non-standard RS232
10 MHz clock output - coax 50 Ohm high resolution 10 MHz clock
1pps output - 1 Hz clock corresponding to the 10 MHz clock
1pps input - 1pps signal from GPS receiver
</pre>

Mode of operation:
* 10 MHz clock is always running
* 1pps output is always running
* if 1pps input from GPS received is present, after 256 pulses PRS-10 will sync it's 1pps output with the 1pps input by adjusting the frequency of the 10 MHz clock
* when unlocked: 1pps output and 1pps input unrelated
* when locked to GPS: 1pps output and 1pps input always go up and down at the same time

Theory of operation:
* 10 MHz clock is produced by a high-quality crystal (stable on the scale of seconds)
* crystal oscillator is synchronized to a Rb cell (stable on the scale of hours and days)
* Rb cell resonant frequency is synchronized to the GPS 1pps signal (stable on the scale of months and years)

== Rev1 connections ==

* LEMO connectors (front panel)
<pre>
LEMO J2A output - Rb clock 1pps in monitor
LEMO J2B output - Rb clock 1pps out monitor
LEMO J5A input - GPS IRIG-B from GPS receiver to FPGA (VCL-2705)
LEMO J5B input - GPS 1pps from GPS receiver to PRS-10 (VCL-2705)
LEMO J6A output - AUX-OUT from FPGA (dual-LEMO PCB-side)
LEMO J6B input - AUX-IN to FPGA (dual-LEMO away-from -PCB)
</pre>

* SMB connectors (back)
<pre>
SMB J3 output - GPS 1pps loopback to LNGS
SMB J4 input - LNGS GPS data input
</pre>

* LEDs
<pre>
D1 - same as LEMO J2A out (Rb clock 1pps in)
D2 - same as SMB J3 out (GPS 1pps from LNGS or from a GPS receiver)
D5 - controlled by FPGA-OUT-LED1
D6 - controlled by FPGA-OUT-LED2
D7 - PRS-10 24V power ok
</pre>

* switches
<pre>
SW1A and SW1B - route PRS-10 RS232 to USB or to FPGA
SW2A - route PRS-10 1pps input from SMB J4 (LNGS) or from LEMO J5B (GPS receiver 1pps)
SW2B - route FPGA-IN-GPSDATA input from SMB J4 (LNGS) or from LEMO J5A (GPS receiver IRIG-B data)
SW3 - LNGS 1pps from rising edge or from falling edge of SMB J4 (LNGS)
</pre>

== Rb clock cable ==
<pre>
Rb clock ----- DS-IOGC side, pin numbers are as labeled on the cable connectors

1 - 1pps out - 8 - 1pps out --- correct
2 - nc
3 - nc
4 - TXD - 3 - RU-DATA-OUT - USB-RX input --- correct
5 - 1pps in - 2 - 1pps in --- correct
6 - +24V - 10 - +24V --- should by pin 1 to use both +24V pins?
7 - RXD - 15 - RU-DATA-IN - USB-TX output --- correct
8 - nc
9 - +24V - 10 - +24V --- correct
10 - GND - 9 - GND --- correct
</pre>

== VX connections ==
<pre>
VXA_TX0 - FPGA-OUT-LED2 - D6 LED ("10 MHz clock")
VXA_TX1 - FPGA-OUT-LED1 - D5 LED ("GPS DATA")
VXA_TX2 - FPGA-OUT-RU1PPS-EN - enable 1pps to Rb clock
VXA_TX3 - not used (62.5 Hz clock)

VXB_TX0 - FPGA-TX - PRS-10 RS-232 out
VXB_TX1 - FPGA-OUT-AUX - 5V TTL J5 LEMO out
VXB_TX2 - FPGA-OUT-OPTO1PPS-EN - PRS-10 power enable
VXB_TX3 - not used (62.5 MHz clock)

VXA_RX0 - FPGA-IN-OPTO1PPS - 1pps from GPS
VXA_RX1 - FPGA-IN-GPSDATA - J4 GPS data
VXA_RX2 - n/c
VXA_RX3 - n/c

VXB_RX0 - n/c
VXB_RX1 - FPGA-IN-AUX - J5 LEMO TTL input
VXB_RX2 - FPGA-RX - PRS-10 RS-232 in
VXB_RX3 - FPGA-IN_RU1PPS - PRS-10 1pps output
</pre>

== test sequence ==

* ./test_cdm_local.exe --writereg 7 0 ### clear reg 7
* ./test_cdm_local.exe --writereg 8 0 ### set vx_tx mux to vx_tx control from reg 7
* connect blue cable to GDM port 6 (next to the ethernet connector)
* ./test_cdm_local.exe --writereg 7 0x4000 ### power up
* test LEDs:
* ./test_cdm_local.exe --writereg 7 0x4100 ### right LED2 D6
* ./test_cdm_local.exe --writereg 7 0x4200 ### left LED1 D5
* test AUX-IN and AUX-OUT:
* ./test_cdm_local.exe --writereg 7 0x6000 ### J6A LEMO (PCB-side) measure +5VDC
* install LEMO jumper between LEMO J6A and J6B
* write 0x4000 then ./test_cdm_local.exe 6 ### reads 0x5C, bit 5 0x20 reads 0
* write 0x6000 then ./test_cdm_local.exe 6 ### reads 0x7C, bit 5 0x20 reads 1
* write 0x4000 to clear all bits
* test the 10 MHz clock:
* init the c.c: ./test_cdm_local.exe --load-cc
* connect 10 MHz clock output to DS-DM LEMO J6-LEFT (CLK_EXT1)
* 10 MHz output ok - IN0 is good: ./test_cdm_local.exe --cc
<pre>
Clock chip state 1, status: OOF_IN2 IN1 IN_SEL_1 HOLD_HIST_VALID
</pre>
* without 10 MHz reports IN0 LOS (no signal) and OOF (wrong frequency)
<pre>
Clock chip state 1, status: LOS_IN0 OOF_IN0 OOF_IN2 IN1 IN_SEL_1 HOLD_HIST_VALID
</pre>
* test GPS IRIG-B data:
* connect GPS IRIG-B signal to J6B (dual-LEMO away from PCB)
* IRIG-B via AUX-IN is ok: ./test_cdm_local.exe --irigb
<pre>
dataframe: S00100110S111001000S...S011110000S010000000S, sec: 34, min: 17, hrs: 00, day of year: 223, year: 2024, date: 10 aug, sbs: 1054 should be 1054
</pre>
* test IRIG-B via J5A GPS-EXT:
* connect GPS IRIG-B signal to J5A (dual-LEMO next to PCB)
* set SW2B: switch on the side of SMB connectors, slide direction of 24V power supply connector
* ./test_cdm_local.exe --irigb built for VXA_RX1 should work
* test GPS 1pps signal:
* connect GPS 1pps to J5B (dual-LEMO away from PCB)
* nothing happens, no output from U22
* remove R51
* set SW2A: switch on the side away from SMB connectors, slide direction of 24V power supply connector
* observe LEDs D1 and D2 blink at 1Hz
* SKIP THIS connect GPS 1pps to J4 using LEMO-SMB adapter
* SKIP THIS nothing happens, nLE is always 3.2V, both settings of SW2
* SKIP THIS !!!failed here!!!
* ./test_cdm_local.exe --writereg 8 9 ### set vx_tx mux to GPS control, hard enables PRS-10 power and PRS-10 1pps in
* observe 1pps counters in reg 68 count at 1 Hz: rb_1pps 0x51->0x52, gps_1pps 0xd5->oxd6
<pre>
root@dsdm:~# ./test_cdm_local.exe 68
ds20k_reg[68] is 0x0051d520 (5362976)
ds20k_reg[68] is 0x0052d624 (5428772)
...
</pre>
* observe PRS-10 can see the 1pps signal "130" changes to "2" after 243 seconds to "4"
* observe period of GPS and PRS-10 1pps is identical (plus/minus 1 clock)
<pre>
root@dsdm:~# ./test_cdm_local.exe 13 14
ds20k_reg[13] is 0x077356d4 (124999380)
ds20k_reg[14] is 0x077356d4 (124999380)
</pre>
* observe GDM/CDM clocks are exactly 125 MHz, we are running on DS-DM internal oscillator
<pre>
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735943 (125000003) should be ~125 MHz
0x1034 rx_clk: 0x07735943 (125000003) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735943 (125000003) should be ~125 MHz
0x103C tx_clk: 0x07735943 (125000003) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>
* if tx_clk is missing, reset the mgt: ./test_cdm_local.exe --reset-mgt
* switch CC to external clock:
<pre>
root@dsdm:~# ./test_cdm_local.exe --cc-in0
CC use clock input 0: 10 MHz LEMO external clock
root@dsdm:~# ./test_cdm_local.exe --cc
Polling CC status...
Clock chip state 1, status: IN0 IN_SEL_REGCTRL IN_SEL_0 HOLD_HIST_VALID
</pre>
* observe rx_clk and tx_clk are now slightly off: they run from PRS-10 10 MHz clock and are measured against the DS-DM internal oscillator.
<pre>
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x94b12519, ds20k version 0x20240814
DS-DM firmware build 0x94b12519, ds20k version 0x20240814
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x077358e1 (124999905) should be ~125 MHz
0x1034 rx_clk: 0x07735b49 (125000521) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735b49 (125000521) should be ~125 MHz
0x103C tx_clk: 0x07735b49 (125000521) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>
* observe GPS and PRS-10 1pps period is now exactly 125*10^6 of 8ns clocks (equal to 1 second)
<pre>
root@dsdm:~# ./test_cdm_local.exe 13 14
ds20k_reg[13] is 0x0773593f (124999999)
ds20k_reg[14] is 0x0773593f (124999999)
</pre>
* look at them repeatedly, observe reg 13 "GPS 1pps period" has some wobble, reg 14 "Rb clock 1pps period" is steady. this is as expected: if GPS 1pps was steady, we do not need to Rb clock. PRS-10 1pps is derived from the PRS-10 10 MHz clock and is measured against the 125 MHz clock derived from the same 10 MHz clock in the GDM FPGA.

Rev0 Test status:

* GPS 1pps to SMB-in ok (LED flashes)
* GPS 1pps to FPGA ok
* GPS data to FPGA ok
* GPS 1pps to PRS-10 enabled from FPGA ok
* PRS-10 Rb clock 1pps out to FPGA ok
* PRS-10 Rb clock 1pps out to FPGA to LEMO AUX out ok
* can see GPS 1pps, IRIG-B, PRS-10 1pps out, 10 MHz on the scope, ok
* PRS-10 syncs on leading edge (0->1) of GPS 1pps signal, ok
* reg 13 and 14 1pps periods are identical, ok
* CC locks on PRS-10 10 MHz clock, DS-DM runs on PRS-10 clock, ok
* NOT TESTED - smb output
* NOT TESTED - optical converter fiber to SMB
* NOT TESTED - optical converter SMB to fiber
* NOT TESTED - SMB loopback
* NOT TESTED - fiber loopback

Rev1 Test status: (21 Nov 2024)

* power up ok
* LEDs ok
* LEMO AUX-IN, AUX-out ok
* GPS IRIG-B to FPGA via J6B AUX-IN ok
* GPS IRIG-B to FPGA via J5A GPS-EXT ok
* PRS-10 Rb clock 1pps out to FPGA ok
* PRS-10 Rb clock 10 MHz to C.C. ok
* PRS-10 RS-232 ok
* J5B 1pps input fail. input signal is 3.3V without termination, 1.5V when plugged into IOGC (on R51), no output from U22.

= VX busy logic =

= DS-20K DAQ =

== Overview ==

DS-DM, GDM and CDM are key parts of the DS-20K DAQ system:
* common clock distribution from external clock (atomic clock, GPS) to GDM to per-quadrant CDMs to VX digitizers
* common trigger distribution from GDM internal algorithm or external input to all VX digitizers
* run control: GDM, CDM, VX all start recording data at the same time (clock and timestamp reset)
* collection of trigger data from VX digitizers to per-quadrant CDMs to GDM

== Deliverables ==

* hardware and firmware for GDM to CDM clock distribution
* hardware and firmware for CDM to VX clock distribution
* hardware and firmware for GDM external clock input (atomic clock or GPS)
* hardware and firmware for CDM and VX serial communications (VX LVDS I/O connector)
* firmware for run control (timestamp reset and sync): GDM to CDM to VX
* firmware for common trigger distribution: GDM to CDM to VX
* firmware for trigger data flow: VX to CDM to GDM
* firmware for busy control: VX to CDM to GDM back to CDM to VX
* firmware for flow control: FEP to GDM MIDAS frontend to GDM to CDM to VX
* GDM MIDAS frontend: clock selector and monitoring, trigger and run control, busy and flow control, GDM housekeeping
* CDM MIDAS frontend: clock monitoring, CDM housekeeping

specific performance:
* GDM external clock: 10 MHz GPS clock
* GDM to CDM fiber link:
** clock XXX MHz
** link data rate: XXX Gbit/sec
** CDM recovered clock: XXX MHz
** CDM recovered clock jitter: XXX ns
** phase alignment between CDMs: XXX ns
** phase alignment between CDMs persists across reboots, power cycles, firmware updates
** phase alignment between CDMs should be easy to measure
** phase alignment between CDMs should be easy to recalibrate if hardware parts are replaced (DS-DM boards, fiber transceivers, fiber cables, etc)
** data packet bandwidth: XXX Mbytes/sec
** data packet latency: XXX clocks
** data packet skew between CDMs: XXX clocks
* CDM to VX clock:
** clock: XXX MHz
** jitter, all CDM clock outputs: XXX MHz
** phase alignment between all CDM clock outputs: XXX ns
* CDM to VX trigger:
** TBD (use the VX "sync" input or VX LVDS I/O line or VX serial link packet)
* CDM to VX serial link:
** clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
** bit rate: XXX bits/sec
** latency: XXX link clocks
** maximum skew between VXes: XXX ns
* VX to CDM serial link:
** clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
** bit rate: XXX bits/sec
** latency: XXX link clocks
** maximum skew between VXes: XXX ns
* timestamp reset:
** maximum skew between VXes: XXX ns
* busy round trip time: XXX ns (VX to CDM to GDM back to CDM to VX)
* flow control latency: XXX ns (FEP software to GDM MIDAS frontend to GDM to CDM to VX)

== Technical risk items ==

this refers to unexpected behaviour and performance of
system components, causes big difficulty in implementing the system,
prevents delivery of deliverables, and prevents or negatively affects operation
of the DS-20K DAQ or of the whole experiment.

(14-sep-2022, list is not sorted by any criterial: severity, probability, ease of investigation)

(stability of course is long term stability, across hours, days, weeks, months, years)

* stability of Enclustra FPGA modules (crashes/year, failures to boot/year, flash corruption/year)
* stability of GDM external clock PLL (lock loss/year)
* stability of CDM recovered clock (lock loss/year, unexpected phase drifts, etc)
* unexpected failures or bit error rates in GDM-CDM fiber links
* stability of CDM VX clock outputs (stability of clock cleaner chip)
* stability of VX internal clock distribution (VX PLL lock loss events)
* stability of VX CAEN base firmware (different versions of CAEN base firmware have different clock distribution behaviour)
* strange things in CAEN base firmware (unexpected clocking of LVDS I/O, unexpected phase shifts between clocks, etc)
* DS-DM and VX hardware problems (incompatible LVSD I/O, incompatible clock signals, etc)

== Milestones ==

(14-sep-2022: at this stage of the project, priority must be given to identifying and retiring (so called) technical risk factors. it is not good to build the complete system only to discover that (for example) some Enclustra FPGA modules require 5 attempts to boot and erase their flash memory contents once a month. Both example are real-life actual problems that caused big difficulties in GRIFFIN/TIGRESS and ALPHA-g experiments).

Development and testing milestones in time reversed order:

* full DAQ data challenge: all VXes, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
* one quadrant data challenge: 1 VME crate of VX, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
* vertical slice data challenge: 1 VME crate, 2 VX, 2 CDM (1 VX per CDM), GDM, etc operate as designed
* GDM-CDM link finalized (data rate frozen, data packet format frozen, data content permitted to change)
* CDM-VX serial link finalized (data rate frozen, data packet format frozen, data content permitted to change)
* run control (timestamp reset) and trigger distribution design agreed upon, frozen (list of possible triggers permitted to change)
* VX to CDM to GDM data flow design agreed upon, frozen (data contents permitted to change)
* major technical risk items retired (all hardware and firmware is working as expecred without mysteries and surprises, all problems are identified, investigated, resolved, solutions tested)
* stable operation of CDM-VX serial links in vertical slice system
* stable operation of GDM to CDM clock in vertical slice system
* stable operation of CDM to VX clock in vertical slice system
* vertical slice system assembled (1 VME crate, 2 VX, 2 CDM, 1 GDM, network, FEP, TSP, MIDAS)

= ZZZ =

ZZZ

DS-DM

2026-03-28T01:45:28Z

Olchansk: /* Block 3 GDM */

= DS-DM =

DarkSide-20k Global and Crate Data Manager board (GDM and CDM).

Global Data Manager (GDM):
* clock distribution to CDM boards (including GPS/atomic clock source)
* collection of trigger data from CDM boards, processing and distribution of trigger decision to CDM boards
* run control
* integration with GPS 10MHz and 1pps clocks and GPS/IRIG date and time information

Crate Data Manager (CDM):
* clock distribution from GDM to CAEN VX digitizers
* receive trigger data from CAEN VX digitizers
* send trigger data to GDM
* run control and dead time control

= Links =

* https://dsdaqgw.triumf.ca/vslice/ - vertical slice DAQ at TRIUMF
* https://daq00.triumf.ca/DaqWiki/index.php/DarkSide - this DS-DAQ Wiki page
* https://dsvslice.triumf.ca/vx_napoli - DS vertical slice at TRIUMF
* https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0 - DS-DM board Rev0
* https://edev-group.triumf.ca/hw/exp/dark-side-20k/cdm-gdm-data-manager-vme-cards/rev1 - DS-DM board Rev1
* https://docs.xilinx.com/v/u/en-US/ds593 - Xilinx Platform Cable USB II
* https://www.enclustra.com/en/products/system-on-chip-modules/mercury-xu8/ - Enclustra
* https://bitbucket.org/team-ds-dm/phasemeasurement - VX Phase measurement script
* https://daq00.triumf.ca/daqinv/frontend/list/178 - inventory database
* https://daq00.triumf.ca/elog-ds/DS-DAQ - DS-DAQ elog
* https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0 - GPS and Rb Clock adapter board

= Git repos =

* dsdmdev@dsdaqgw.triumf.ca:git/gcdm.git - git repository, DS-DM firmware, branch develop_ko2, contact K.O. for ssh key access
* https://edev-group.triumf.ca/fw/exp/darkside/gcdm - git repository, DS-DM firmware (broken, do not use)
* https://bitbucket.org/team-ds-dm/ds-dm-software - git repository, DS-DM MIDAS frontend
* https://bitbucket.org/ttriumfdaq/dsproto_vx2740/src/develop/ - VX DSFE MIDAS frontend
* https://openproject.triumf.ca/svn/darkside-20k/ - VX firmware project

= Onboard hardware =

* jtag chain: arm_dap_0 0x5BA00477, xczu4_1 0x04721093
* Eclustra Mercury+ XU8 module: ME-XU8-4CG-1E-D11E-R2.1
** Xilinx® Zynq Ultrascale+™ MPSoC XCZU4CG-1FBVB900E
** DDR4 ECC SDRAM (PS) 2 GB
** DDR4 SDRAM (PL) 1GB
* ethernet mac chip: AT24MAC402-SSHM-T ("602" chip is wrong)
* USB UART for Enclustra serial console, micro-USB, 115200n8
* clock chip: SI5394A-A-GM and oscillator CS-044-054.0M (54 MHz)
* U23 3.3V current meter and thermometer, LTC2990IMS#TRPBF
* LEDs:
** LED_FP A/B/C/D 0/1/2/3
** led1 - 3V3_SW_ON, SOM_POWER_GOOD - Enclustra FPGA module 3.3V power is good
** led2 - LTM4624 PGOOD
** led3 - FPGA_DONE - FPGA has booted
** led4 - TP-S-1, PCLK_P
** led5 - TP-S-2, PCLK_N
* LEMO connectors (top to bottom)
** J4 - input (NIM/TTL) (EXT_IN_LV(1), EXT_IN_LV(2))
** J5 - input (NIM/TTL) (EXT_IN_LV(3), EXT_IN_LV(4))
** J6 - external clock (GPS 10MHz and PPS)
** J7 - output (NIM/TTL) (EXT_OUT(1), EXT_OUT(2))
* SMA connectors
** J9, J10 - CLK_CCA from U6 C.C.
** J11, J12 - CLK_TP0
* RJ45 ethernet connector (100mbit: green light on, yellow light flashes when there is traffic)
* SFP connector (SFP is FTLF8526P3BNL, 6 Gbit/sec, 850 nm, 300m 50/125um OM3 MMF)
* 4 QSFP connectors (GDM)
* 6 VX connectors (CDM)

= Buttons, jumpers and switches =

Buttons:

* PB1 - HRST - reboot FPGA (power-on reset)
* PB2 - SRST - (SRSTn) - reboot ARM CPU

Switches:

* SW1 - boot mode BM0, BM1 [-->]
* SW2 - LEMO output NIM<->TTL
* SW3 - LEMO input 1 and 2 NIM/TTL
* SW4 - LEMO input 2 and 4 NIM/TTL
* SW5 - LEMO clock input NIM/TTL
* SW6 - serial console select. [PS<--PL] PS is ARM CPU, PL is FPGA.

= Front panel =

<pre>
| top
|
| LED-FP1 | LED_FP(0,1,2,3)
|
| SFP J???
|
| LEMO J4-LEFT, J4-RIGHT | EXT_IN_LV(1), EXT_IN_LV(2)
| LEMO J5-LEFT, J5-RIGHT | EXT_IN_LV(3), EXT_IN_LV(4)
| LEMO J6-LEFT, J6-RIGHT | CLK_EXT1, CLK_EXT0 (125 MHz only)
| LEMO J7-LEFT, J7-RIGHT | EXT_OUT(1), EXT_OUT(2)
|
| J-VX-1
| J-VX-2 or QSFP-1
| J-VX-3 or QSFP-2
| J-VX-4 or QSFP-3
| J-VX-5 or QSFP-4
| J-VX-6
|
| RJ45 J3 ethernet
|
| bottom
</pre>

= VX adapter board =

LVDS I/O connector
<pre>
34 pin connector: 0|:::: :::: :::: :::: :|15,16 n/c
</pre>

== split-cable connection ==

<pre>
0 .. 7 -> N/C

8 -> VX_RX(3) - not used
9 -> VX_RX(2) - busy VX to CDM
10 -> VX_RX(1) - DS20K 125 MHz serial data VX to CDM
11 -> VX_RX(0) - DS20K 62.5MHz clock VX to CDM

12 <- VX_TX(0) - TRG CDM to VX
13 <- VX_TX(1) - TSM CDM to VX, to be VETO CDM to VX
14 <- VX_TX(2) (set by jumper routed here or to VX CLKIN SYNC) - 125 MHz serial data CDM to VX
15 <- n/c (CLK routed to VX CLKIN CLK) - 62.5 MHz clock CDM to VX

16 - n/c
</pre>

== one-to-one connection ==

<pre>
0 -> VX2_RX(3)
1 -> VX2_RX(2)
2 -> VX2_RX(0)
3 -> VX2_RX(1)

4 <- CLK
5 <- VX2_TX(0)
6 <- VX2_TX(1)
7 <- VX2_TX(2)

8 -> VX1_RX(3)
9 -> VX1_RX(2)
10 -> VX1_RX(1)
11 -> VX1_RX(0)

12 <- VX1_TX(0)
13 <- VX1_TX(1)
14 <- VX1_TX(2) (set by jumper routed here or to VX CLKIN SYNC)
15 <- n/c (CLK routed to VX CLKIN CLK)

16 - n/c
</pre>

= Board schematics Rev0 =

* [[File:SCH-DS-xDM-Rev0.PDF]]
* note: FPGA pin annotations ("IO", "SCLK", "PCLK", etc) on the schematics are bogus, instead, trace them to the FPGA pins.
* note: ENC A is J800, ENC B is J801, ENC C is J900 (schematic name to enclustra name)
* note: Enclustra special pins: "GC" is "clock capable", "HDGC" is "clock capable", "MGTREFCLK" is MGT reference clocks.
* board modifications:
** ethernet mac chip
** NIM output (no U15, etc)
** RJ45 wrong pinout (board mod or special ethernet cable)
** 125 MHz clock mods (TBW)
** disconnect QSFP0_SEL from SFP_RS0 and QSFP1_SEL from SFP_RS1, these signals are not used by modern SFPs
** provide SFP i2c modsel to allow SFP and QSFP at the same time on address 0x50

= Board schematics Rev1 =

* [[File:SCH-DS-xDM-Rev1.PDF]]
* ENC-A is J800, ENC-B is J801, ENC-C is J900
* modifications from Rev0:
* clocks:
** 125MHz osc: CLK_XO_125, CLK3_XO_125 (CLK2_XO_125 removed)
** C.C. in2 is CLK_CC_IN from ENC-C142,144
** C.C. out0 via CLK_A to CLK_CC_OUT0 (ENC-C3,5), CLK_CC_OUT1 (ENC-B3,5), CLK_CC_OUT2 (ENC-C151,153)
* RTC chip: TP_S pins gone, RTC_I2C_SCL, RTC_I2C_SDA, RTC_1PPS and RTC_32KHZ added
** RTC_1Hz ENC-C160 pull up 10k to 3.3V
** RTC_32k ENC-B129 pull up 10k to 1.8V
* QSFP, SFP control lines:
** HW_ID_xDM ENC-B131 gone, reused as SFP_ModPrsN
** SFP_ModPrsN (MOD_ABS) ENC-B131 input (no pulls)
** SFP_TX_fault (was QSFP2_SEL) QSFP_ModPrsN ENC-C157 input (no pulls)
** SFP_TX_disable from QSFP_LPMode ENC-C163 output (pull up 10k to 3.3V)
** SFP_ModDet n/c (was QSFP_ModPrsN) renamed SFP_ModPrsN input
** SFP_RX_LOS to QSFP_IntN ENC-C159 input (pull up 2k2 to 3.3V)
** SFP_RS0, SFP_RS1 n/c, assembly option: floating
** QSFP_ResetN - FPGA output, pull up 10k to 3.3V
** QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL - FPGA outputs, pull up 10k to 3.3V
** QSFP_LPmode - FPGA output, pull up 10k to 3.3V
** QSFP_ModPrsN - ??? - pull up 2k2 to 3.3V
** QSFP_IntN - ??? - pull up 2k2 to 3.3V
* VX_TX and VX_RX reassigned for better length matching
* cosmetic changes:
** front panel USB-C connector (replaces Micro-USB)
** (reset push button PB1 PORn ENC-A132 aka PS_POR_B aka PS_POR#)
** (reset push button PB2 SRSTn ENC-A124 aka PS_SRST_B aka PS_SRST#)
** front panel PB3 HRSTn same as PB1
* Enclustra errata:
** possible cross talk between pins AA2 (J801-B90, VX9_RX2) and AE3 (J801-B129, RTC_32KHZ). no RTC on the CDM.
** ethernet link up failure (see Microchip KSZ9031 errata)

= FPGA MGT blocks =

<pre>
* SFP - ENC B45,B47 and B48,B50 - FPGA D5,D6 MGTHTX0_D and D1,D2 MGTHRX0_D
* QSFP0 TX0 - ENC C13,C17 - FPGA H5,H6 MGTHTX0_C
* QSFP0 TX1 - ENC C21,C25 - FPGA G7,G8 MGTHTX1_C
* QSFP0 TX2 - ENC C29,C23 - FPGA F5,F6 MGTHTX2_C
* QSFP1 TX0 - ENC C37,C41 - FPGA E7,E8 MGTHTX3_C
* QSFP1 TX1 - ENC C45,C47 - FPGA P5,P6 MGTHTX0_B
* QSFP1 TX2 - ENC C51,C53 - FPGA M5,M6 MGTHTX1_B
* QSFP2 TX0 - ENC C57,C59 - FPGA L3,L4 MGTHRX2_B
* QSFP2 TX1 - ENC C63,C65 - FPGA K5,K6 MGTHTX3_B
* QSFP2 TX2 - ENC C75,C77 - FPGA W3,W4 MGTHTX0_A
* QSFP3 TX0 - ENC C79,C81 - FPGA V5,V6 MGTHTX1_A
* QSFP3 TX1 - ENC C85,C87 - FPGA T5,T6 MGTHRX2_A
* QSFP3 TX2 - ENC C89,C91 - FPGA R3,R4 MGTHTX3_A
</pre>

= Clock distribution Rev0 =

Simplified:

* 125 MHz osc -> CLK_XO_125 -> MGTREFCLK0_A -> not used
* 125 MHz osc -> CLK3_XO_125 -> MGTREFCLK1_B -> SFP RX ref clock, QSFP RX and TX ref clock (this is not final design!)
* (disconnected) 125 MHz osc -> CLK2_XO_125 -> FPGA AG8,AH8 (GC)
* 125 MHz osc -> C.C. in1

* C.C. in0 <- CLK_EXT1 (10 MHz GPS clock)
* C.C. in1 <- 125 MHz osc
* (disconnected) C.C. in2 <- CLK_CC_IN <- MGTREFCLK0_D <- SFP RX clock (cannot be used because of uncontrollable phase)
* C.C. in2 <- CLK2_XO_125 <- FPGA AG8,AH8 (GC) <- SFP RX recovered 125 MHz clock
* C.C. in3 <- CLK_FB

* C.C. 125 MHz -> CLK_CC_OUT0 -> MGTREFCLK0_B -> QSFP RX and TX ref clock (final design)
* C.C. 125 MHz -> CLK_CC_OUT1 -> MGTREFCLK1_D -> SFP TX clock
* C.C. 62.5 MHz -> VX clock fanout

proposed changes:
* add C.C. 125 MHz -> new CLK_CC_OUT2 (old CLK2_XO_125) -> FPGA AG8,AH8 (GC)
* change C.C. in2 <- new CLK_CC_IN <- FPGA AK8,AK9 (non-GC)

Complete:

<pre>
125 MHz oscillator - U5 fan out -
q0 -> CLK_XO_125 -> ENC C72,C74 -> FPGA R7,R8 MGTREFCLK0_A (not used)
q1 -> U6 C.C. in1
q2 -> CLK3_XO_125 -> ENC C7,C9 -> FPGA J7,J8 MGTREFCLK1_B -> SFP RX reference clock, QSFP RX and TX reference clocks (not final design!)
q3 -> disconnected on the board, was CLK2_XO_125 -> ENC C151,C153 -> FPGA AG8,AH8 (GC)

U6 C.C (clock cleaner) -

in0 <- CLK_EXT1 (presumably GPS 10 MHz ref clock)
in1 <- 125 MHz oscillator via U5
in2 <- (was: CLK_CC_IN <- ENC B10,B12 <- FPGA D9,D10 MGTREFCLK0_D <- SFP RX clock, 125 MHz)
in2 <- CLK2_XO_125 <- ENC C151,B153 <- FPGA AG8,AH8 (GC) <- mgt_link_data_to_processing.rx_data_clk (SFP RX data clock, 125 MHz)
in3 <- CLK_FB

out0 -> CLK_CCA -> U12 (125 MHz)
out1 -> CLK_CCB -> VX1..6 (62.5 MHz)
out2 -> CLK_CCC -> VX7..12 (62.5 MHz)
out3 -> CLK_FB into in3

CLK_CCA -> U12 (125 MHz fan out) ->

Q0 -> not used
Q1 -> CLK_CC_OUT0 -> ENC C3-5 -> FPGA L7,L8 MGTREFCLK0_B -> QSFP RX and TX reference clocks (final design)
Q2 -> CLK_CC_OUT1 -> ENC B3-5 -> FPGA B9,B10 MGTREFCLK1_D -> SFP TX clock
Q3 -> not used
Q4 -> not used
Q5 -> SMA J9/J10

CLK_TP0 - FPGA N7,N8 MGTREFCLK1_A - ENC C69,C71 - DS-DM SMA J11, J12 (NOT IN CDM PROJECT)

Notes:
* CLK_XO_125 (125 MHz osc) is not used
* 62.5 MHz VX clock does not go into the FPGA
* CLK_EXT0 going to in1 of U5 cannot be used. only permitted frequency is 125 MHz (it drives the MGT reference clocks) and is this frequency is too high for LEMO cables and connectors

Proposed modifications:
- CLK_CC_IN: from FPGA output pin (ENC C142,C144 - FPGA AK8,AK9) to clock cleaner in2 (instead of CLK2_XO_125 pins)
- CLK_CCA -> U12 -> currently unused out3 -> CLK2_XO_125 FPGA pins
- repurpose CLK_EXT0 at GPS 1pps/IRIG input to FPGA
</pre>

= Clock distribution Rev1 =

Simplified:

* 125 MHz osc -> CLK_XO_125 -> MGTREFCLK0_A -> clk_xo_125. not used
* 125 MHz osc -> CLK3_XO_125 -> MGTREFCLK1_B -> CDM SFP RX ref clock
* 125 MHz osc -> C.C. in1

* C.C. in0 <- CLK_EXT1 (10 MHz GPS clock)
* C.C. in1 <- 125 MHz osc
* C.C. in2 <- CLK_CC_IN <- FPGA AK9,AK8 <- CLK_CC_IN1, rx_clk_mgt, SFP RX recovered clock, 125 MHz
* C.C. in3 <- CLK_FB

* C.C. 125 MHz -> CLK_CC_OUT0 -> MGTREFCLK0_B -> GDM QSFP RX and TX ref clock, CDM rx_clk mux
* C.C. 125 MHz -> CLK_CC_OUT1 -> MGTREFCLK1_D -> SFP TX clock
* C.C. 125 MHz -> CLK_CC_OUT2 -> FPGA AG8,AH8 (GC) -> not used
* C.C. 62.5 MHz -> VX clock fanout

Complete:

<pre>
125 MHz oscillator - U5 fan out -
q0 -> CLK_XO_125 -> ENC C72,C74 -> FPGA R8,R7 MGTREFCLK0_A -> XDC CLK_XO_125_P -> VHDL not used
q1 -> U6 C.C. in1
q2 -> CLK3_XO_125 -> ENC C7,C9 -> FPGA J8,J7 MGTREFCLK1_B -> XDC GDM missing, CDM CLK3_XO_125_P -> VHDL SFP RX reference clock (mgt_rx_ref_clk)
q3 -> not used, was CLK2_XO_125

U6 C.C (clock cleaner) -

in0 <- CLK_EXT1 (GPS 10 MHz clock)
in1 <- 125 MHz oscillator via U5
in2 <- CLK_CC_IN <- ENC C142,C144 <- FPGA AK9,AK8 <- XDC GDM missing, CDM CLK_CC_IN1 <- VHDL rx_clk_mgt
in3 <- CLK_FB

out0 -> CLK_CCA -> U12 (125 MHz)
out1 -> CLK_CCB -> VX1..6 (62.5 MHz)
out2 -> CLK_CCC -> VX7..12 (62.5 MHz)
out3 -> CLK_FB into in3

CLK_CCA -> U12 (125 MHz fan out) ->

Q0 -> not used
Q1 -> CLK_CC_OUT0 -> ENC C3,C5 -> FPGA L8,L7 MGTREFCLK0_B -> XDC GDM clk_mgtrefclk0_x0y1_p, CDM i_CLK_CC_OUT0_P -> VHDL GDM mgt_b_ref_clk QSFP RX and TX reference clocks, CDM alternate rx_clk via clock mux
Q2 -> CLK_CC_OUT1 -> ENC B3,B5 -> FPGA B10,B9 MGTREFCLK1_D -> XDC i_CLK_CC_OUT1_P -> VHDL GDM not used, CDM mgt_tx_ref_clk SFP TX clock
Q3 -> CLK_CC_OUT2 -> ENC C151,C153 -> FPGA AG8,AH8 (GC) -> XDC GDM missing, CDM i_CLK_CC_OUT2_P
Q4 -> not used
Q5 -> SMA J9/J10

CLK_TP0 <-> ENC C69,C71 <-> FPGA N8,N7 MGTREFCLK1_A (not used) -> XDC CLK_TP0_P -> VHDL not used (DS-DM SMA J11, J12)

disconnected <- OLD_CLK_CC_IN <- ENC B10,B12 <- FPGA D9,D10 MGTREFCLK0_D <- XDC o_CLK_CC_IN <- VHDL CDM o_OLD_CLK_CC_IN SFP o_mgt_rx_rec_clk

Notes:
* Enclustra ENC-B is J801, ENC-C is J900
* "(GC)" is clock-capable FPGA pin
* CLK_XO_125 (125 MHz osc) is not used in FPGA
* 62.5 MHz VX clock does not go into the FPGA
* CLK_EXT0 going to in1 of U5 cannot be used. only permitted frequency is 125 MHz (it drives the MGT reference clocks) and is this frequency is too high for LEMO cables and connectors
</pre>

= I2C bus =

* I2C_SCL is J-ENC-A pin 111 I2C_SCL is FPGA I2C_SCL_PL AB13 (IO_L1N_TOL1D_64) and I2C_SCL_PS F18 (PS_MIO10)
* I2C_SDA is J-ENC-A pin 113 I2C_SDA is FPGA I2C_SDA_PL AH13 (IO_L7N_T1L1Q_AD13N_64) and I2C_SDA_PS G18 (PS_MIO11)

* XU8 secure EEPROM ATSHA204A at 0x64, this is 0110'010X -> linux _011'0010 is 0x32. (but responds to scan and read at 0x33)
* U4 ethernet mac chip, EEPROM at 1010 A2 A1 A0 X and MAC/serial_no at 1011 A2 A1 A0 X. A0=VCC, A1=VCC, A2=GND -> linux _101'0011 and _101'1011 is 0x53 and 0x5B.
* U6 clock chip, address 1101 0 A1 A0 X. A1=VCC, A0=N/C (internal pull-up) -> linux _110'1011 is 0x6b
* U23 voltmeter at 10011 ADR1 ADR0. ADR0=GND, ADR1=VCC -> linux _100'1110 is 0x4e
* SFP, address 1010000X -> linux _101'0000 is 0x50. additional SFP data at 0x51
* QSFP0, QSFP1, QSFP2, QSFP3 (QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL)

= I2C clock builder connection =

* use Silicon Labs USB "Clock builder pro field programmer", www.silabs.com/CBProgrammer
* connect rainbow jumper cable pins:
** black - 1-GND to GND on DS-DM
** white - 3-SCLK to SCL on the DS-DM
** grey - 7-SDA_SDIO to SDA on the DS-DM
* power up the DS-DM
* plug USB programmer into Windows laptop
* on Windows, run "ClockBuilder Pro"
* it should report "Field programmer detected", press "EVB GUI"
* in EVB GUI, press "Config", set I2C address 0x6B
* press "Scan", it should find Si5394A-A-GM
* select the "Status" tab, should see real-time status of clock chip

= GDM MGT configuration =

* TX configuration:
* GDM MGT transceivers are configured as "multilane" TX and RX.
* there is 12 TXes ("lanes")
* MGT reference 125 MHz clock goes into [2:0]gtreclk00_in and [11:0]gtrefclk0_in
* one MGT is designated as "master"
* PLL of master MGT converts reference clock into common TX clock and common tx_user_clk2 which becomes tx_data_clk
* common tx_user_clk2 aka tx_data_clk goes into all TXes and clocks tx_data.
* tx_user_clk2 aka tx_data_clk is 125 MHz but not same phase as MGT reference clock.

* RX interim configuration:
* there is 12 RXes ("lanes")
* each RX produces it's own recovered RX clock
* "multilane" configuration assumes all RX recovered clocks run at the same frequency (TX on the other end are driven by common TX clock, see above), but have different phase
* one RX recovered clock is designated as "master" (rx_user_clk2 aka rx_data_clk) and a phase-matching fifo/buffer is used to bring rx_data from all 12 RXes to this common rx_data_clk
* this works because each CDM SFP TX runs on the SFP RX recovered clock which is frequency-locked with the GDM QSFP TX clock.

* RX final configuration:
* MGTs permit using the common TX clock (tx_user_clk2 aka tx_data_clk) as the common rx_data_clk (they are frequency locked through the CDM).
* this permits use of tx_data_clk as the main clock domain in the GDM and removes the need to bring rx_data into the tx_data_clk domain (actually this is done in the MGT RX phase matching fifo/buffer).

= Clock path =

NOTE: MUST REVIEW!!!

<pre>
10 MHz ext clock or GDM 125 MHz oscillator
-> GDM QSFP MGT reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz (GDM main clock domain) and TX bit clock 2.5 GHz
-> GDM QSFP optic transmitter
-> CDM SFP optic receiver
-> CDM SFP MGT, RX reference clock is CDM 125 MHz oscillator
-> MGT RX recovered clock 125 MHz (CDM main clock domain)
-> CC_CLK_IN -> CDM C.C. -> CC_CLK_OUT1 -> CDM SFP TX reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz and TX bit clock 2.5 GHz
-> (tx_data phase matching fifo from CDM main clock domain to tx_data_clk)
-> CDM SFP optic transmitter
-> GDM QSFP RX optic receiver (12x)
-> GDM QSFP MGT (RX reference clock is same as TX reference clock)
-> MGT RX recovered clock (12x recovered clocks)
-> in multi-lane configuration, one of them is the "master" recovered clock rx_data_clk
-> (rx_data phase matching fifo from rx_data_clk to GDM main clock domain)
</pre>

= CDM rx_clk mux =

when CDM SFP is not connected, there is no SFP recovered clock and a mux is used to switch between clk_cc_out0 (power up default) and rx_clk_mgt (SFP recovered clock)

== Test SFP disconnected ==

note: if I say "--cc-in1", CC seems to lock on the 10 MHz GPS external clock,
to prevent this, test sequence includes reloading the CC and the reset of MGT.

<pre>
./test_cdm_local.exe --write32 0x30 0x0 ### rx_clk mux select CC clock
./test_cdm_local.exe --load-cc
./test_cdm_local.exe --reset-mgt
./test_cdm_local.exe --cdm-clocks
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773594e (125000014) should be ~125 MHz <=== all 4 clocks wobble close to 125 MHz
0x1034 rx_clk: 0x0773594f (125000015) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x0773594f (125000015) should be ~125 MHz
0x103C tx_clk: 0x0773594f (125000015) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
./test_cdm_local.exe --write32 0x30 0x1 ### rx_clk mux select SFP recovered clock
./test_cdm_local.exe --cdm-clocks
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773598b (125000075) should be ~125 MHz
0x1034 rx_clk: 0x076d58ec (124606700) should be ~125 MHz <=== off frequency because there is no valid SFP recovered clock
0x1038 mgt_tx_ref_clk_raw: 0x0773598c (125000076) should be ~125 MHz
0x103C tx_clk: 0x0773598b (125000075) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

= Clock domains =

== GPS ==

* no GPS : GDM runs from internal 125 MHz oscillator
* external 10 MHz clock : GDM runs from external 10 MHz clock and optional 1pps signal (use VME-NIMIO32 NIM outputs)
* GPS receiver : GDM runs from GPS 10 MHz clock and GPS IRIG serial data
* LNGS GPS:
** provides 1pps and serial data over fiber from GPS receiver (master). LNGS xxx box is not used.
** serial data goes to GDM, decoded, 1pps signal extracted, goes to LEMO output
** 1pps from GDM is used to train the Rubidium clock which provides a 10 MHz clock
** 10 MHz output from Rubidium clock goes to GDM 10 MHz external clock input

== GDM ==

* AXI clock (100 MHz) - AXI registers
* 125 MHz oscillator - to clock cleaner
* 10 MHz external clock LEMO input - to clock cleaner
* FPGA 125 MHz clock CLK_CC_IN - to clock cleaner - not used (could be used for cascaded GDMs)
* 125 MHz oscillator - CLK3_XO_125 - mgt_b_ref_clk QSFP MGT reference clock (interim GDM design)
* clock cleaner output 125 MHz fanout:
** CLK_CC_OUT0 - QSFP MGT reference clock (final design)
** CLK_CC_OUT1 - not used (CDM SFP reference clock)
** CLK_CC_OUT2 - not used
* QSFP MGT TX data clock 125 MHz tx_data_clk (main clock domain)
** QSFP TX data
** QSFP RX data (in final design, rx_data_clk is same as tx_data_clk. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk == tx_data_clk)
** ds20k block
* (in interim design, rx_data_clk is disconnected from tx_data_clk. rx_data_clk is the "multilane master clock" which is one of the 12x RX recovered clocks. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk)

note: all these clocks are frequency locked to 125 MHz

== CDM ==

* AXI clock (100 MHz) - AXI registers
* 10 MHz external clock LEMO input - to clock cleaner (not used)
* 125 MHz oscillator to fanout
** to clock cleaner
** to SFP MGT RX reference clock: CLK3_XO_125 to mgt_rx_ref_clk to gtrefclk01_in
* SFP MGT RX recovered clock 125 MHz
** MGT PLL to MGT rx_user_clk2 aka rx_data_clk
** MGT rxrecclkout_out to mgt_rx_rec_clk to CLK_CC_IN to C.C.
* SFP MGT rx_user_clk2 aka rx_data_clk (250 MHz/8 bit, 125 MHz/16 bit, 62.5 MHz/32 bit data) (main clock domain)
** SFP RX data
** ds20k block
** VX TX clock PLLs
** VX RX clock PLLs
* C.C. fan out
** 62.5 MHz VX clocks (12x)
** CLK_CC_OUT0 (not used, GDM QSFP MGT reference clock)
** CLK_CC_OUT1 125 MHz to mgt_tx_ref_clk to gtrefclk00_in to SFP MGT TX reference clock
** CLK_CC_OUT2 (not used)
* SFP MGT tx_user_clk2 aka tx_data_clk
** SFP TX data
** TX data phase matching fifo from main clock domain to tx_data_clk
* VX TX clock PLLs
** 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
** VX TX data phase matching from main clock domain to VX TX clock (12 total)
** VX TX serializer
** VX TX LVDS transmitter
* VX RX clock PLLs
** 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
** VX RX LVDS receivers (12 total)
** VX RX deserializers (12 total)
** VX TX data phase matching from VX RX clock to main clock domain

== VX ==

* everything runs on the VX main 125 MHz clock
* correct phase of VX to CDM LVDS data is adjusted by scan of CDM VX RX PLL clock phase (VX to CDM link is now established)
* correct phase of CDM to VX LVDS data is adjusted by scan of CDM VX RX PLL clock phase (link is established after VX to CDM idle data pattern changes from "VX RX data bad" to "good".
* after good phases are found by scan, they are not expected to change unless cables are changed, CDM and VX modules are changed or CDM firmware is rebuilt. (rebuild of VX firmware should not affect LVDS data phase).
* if there is excessive link errors, phase scan must be repeated.

= Board test plan =

To test:
* Enclustra FPGA board
* SFP port - SFP_RS0/RS1 connected to QSFP0_SEL/1_SEL is wrong? our Finisar SFP says RS0, RS1 N/C, so probably okey.
* SW5 CLK_EXT1 NIM works. TTL needs to be tested. flipping SW5 CLK_EXT0 side from NIM to TTL makes CLK_EXT1 go LOS and OOF in the clock chip. R23 and R53 should be removed?
* BOOT_MODE 0 and 1

Partial:
* U23 3.3V current meter and thermometer. V1,V2 is current monitor, same as in application note. V3,V4 is thermometer, same in application note, except capacitor C118 is 0.1u instead of 470pF. Tested ok: Tint, VCC, V1, V2, V1-V2. Test failed: TR2 reads 50-something degC instead of same as Tint.

Done:
* LED_FP1A..D: tested ok. K.O. 15 sep 2022
* USB UART: tested ok. K.O. 15 sep 2022
* J4A, J4B, J5A, J5B LEMO inputs (NIM/TTL) EXT_IN_LV(1..4). TTL threshold 1.7V, NIM threshold -0.3V. 50 Ohm termination. TTL straight, NIM inverted.
* J6A, J6B LEMO clock inputs: 10 MHz TTL from chronobox works, 10 MHz NIM from IO32 works. 18-apr-2023
* J7A, J7B LEMO outputs EXT_OUT(1), EXT_OUT(2) (NIM/TTL) (tested 23nov2022, K.O.). TTL ???, NIM inverted.
** TTL out no 50 ohm termination: 0=0V, 1=5V, rise and fall time ~5 ns
** TTL out with 50 ohm termination: 0=0V, 1=2.5V, rise and fall time <2ns
** NIM out no 50 ohm termination: 0=+50mV, 1=-1.8V, rise and fall time ~3ns
** NIM out with 50 ohm termination: 0=0V, 1=-0.9V, rise and fall time <2ns
* ethernet MAC i2c chip (K.O. 20sep2022: can read: i2cdump 0 0x5b, nov2022 have u-boot driver)
* CDM VX ports 2x(CLK, 3 tx, 4 rx) tested using test_cdm.exe and LVDS loopback in VX firmware.
* SFP i2c tested KO 22jun2023
* QSFP i2c tested KO 22jun2023
* i2c testing complete 22jun2023
* QSFP rx,tx tested 26june2023. lane0,1,2 ok, lane3 tx not connected, laser is off. qsfp0,1,2,3 all lanes ok.
* SFP rx,tx tested 26june2023. rx and tx okey. LOS ok, mod_absent ok.

Failure:
* ethernet: does not connect to alliedtelesys switch. connects to my USB-eth adapter at 100 Mbit speed. uboot mii status reports connection speed oscillating between 1000, 100 and 10. K.O. 16-sep-2022
* SFP LOS and mod_absent are swapped (in the FPGA pin definitions?)
* QSFP slot numbering is wrong.

= Checklist for newly build boards =

== setup and boot ==
* put new board on workbench
* check - vme connector present, vme extraction handles present
* check - standoff are removed from all thru-holes
* plug Enclustra module
* check - SW1 both switches are in the "6" and "3" positions
* check - SW6 both switches are in the "PS" position
* Rev0: connect micro-usb cable from linux PC
* Rev1: connect usb-C cable from linux PC (USB C-to-C or C-to-A)
* connect ethernet from 1gige capable network switch
* connect power from lab power supply - GND, +5V and -12V nominal, +5.7V and -12.5V actual
* power up, +5V current 2.10-2.8A, -12V current 0.05A
* on linux PC, open a new terminal, run: minicom -D /dev/ttyACM0 -b 115200
* in minicom window, observe messages about Xilinx first stage boot loader, etc
* on the ethernet switch, observe network link is 1gige speed (not 10mbit, not 100mbit).
* if everything boots okey, there will be a login prompt, login as root, password root.
* alternatively, from root@daq13: ssh root@dsdm
* busybox devmem 0x80010000 # read firmware version number, i.e. 0xEDAD0A77
== test i2c ==
* check i2c: (no SFP, no QSFP plugged in!)
<pre>
root@dsdm:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@dsdm:~#
</pre>
* test clock chip:
<pre>
root@dsdm:~# ./test_cdm_local.exe --load-cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Loading CC registers...
Clock chip data block size 519
root@dsdm:~# ./test_cdm_local.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Polling CC status...
Clock chip state 1, status: LOS_IN0 OOF_IN0 IN1 IN_SEL_1
Ctrl-C
</pre>
* test RTC:
<pre>
root@dsdm:~# ./test_cdm_local.exe --rtc-read
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Read the RTC chip!
reg 0x00: 0 0 1 0 0 1 1 0 0x26
reg 0x01: 0 0 1 1 1 0 0 1 0x39
reg 0x02: 0 0 0 0 0 1 1 1 0x07
reg 0x03: 0 0 0 0 0 0 1 0 0x02
reg 0x04: 0 0 0 0 0 0 1 0 0x02
reg 0x05: 0 0 0 0 0 0 0 1 0x01
reg 0x06: 0 0 0 0 0 0 0 0 0x00
reg 0x07: 0 0 0 0 0 0 0 0 0x00
reg 0x08: 0 0 0 0 0 0 0 0 0x00
reg 0x09: 0 0 0 0 0 0 0 0 0x00
reg 0x10: 0 0 0 0 0 0 0 0 0x00
reg 0x11: 0 0 0 0 0 0 0 0 0x00
reg 0x12: 0 0 0 0 0 0 0 0 0x00
reg 0x13: 0 0 0 0 0 0 0 0 0x00
reg 0x14: 0 0 0 1 1 0 0 0 0x18
reg 0x15: 0 0 0 0 1 0 0 0 0x08
reg 0x16: 0 0 0 0 0 0 0 0 0x00
reg 0x17: 0 0 0 1 0 1 1 1 0x17
reg 0x18: 0 1 0 0 0 0 0 0 0x40
seconds: 26, minutes: 39, hours: 07, day 2, date: 00-01-02, temp 23.25C
root@dsdm:~#
</pre>

== test clocks ==

<pre>
root@dsdm:~# ./test_cdm_local.exe --reset-mgt
...
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735cb9 (125000889) should be ~125 MHz
0x1034 rx_clk: 0x07735cb9 (125000889) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735cb9 (125000889) should be ~125 MHz
0x103C tx_clk: 0x07735cb9 (125000889) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
^C
root@dsdm:~# ./test_cdm_local.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735c6a (125000810) should be ~125 MHz
0x1034 rx_clk: 0x07735c69 (125000809) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735c69 (125000809) should be ~125 MHz
0x103C tx_clk: 0x07735c6a (125000810) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
^C
root@dsdm:~#
</pre>

== test LED, LEMO ==

* test LEDs, observe all 4 LEDs turn on and off every 1 second
<pre>
root@dsdm:~# ./test_cdm_local.exe --blink-led
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
LED on
LED off
</pre>
* TBW - test LEMO outputs
<pre>
have test firmware lemo outputs are RTC chip 1pps and 48kHz
see them on the scope
</pre>
* TBW - test LEMO inputs
<pre>
./test_cdm_local.exe --monreg 6
</pre>

== test VX ==

* on the CDM, connect all split blue cables to VX modules
* set all VX modules to LVDS loopback mode (in VX settings MIDAS page)
* ssh root@cdm00
* run "./test_cdm.exe --test-vx", it should report:
<pre>
CDM registers: 0x0000000000000000 0x00000000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000eeeeeeeeeeee 0x00007700: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000080808080808 0x00000100: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000040404040404 0x00000200: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000020202020202 0x00000400: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000000000000000 0x00000800: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000808080808080 0x00001000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000404040404040 0x00002000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000202020202020 0x00004000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000000000000000 0x00008000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
all on!
all off!
</pre>
* on the VX settings MIDAS page LVDS input section, observe line status is cycling on and off.
* note: VX_RX(0) is not tested by this, corresponding VX_TX is used as the VX clock

== test SFP (CDM) ==

* connect SFP module
* ssh root@cdm00
* run "./test_cdm.exe --sfp", it should report:
<pre>
root@cdm00:~# ./test_cdm.exe --sfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x6a332b58, ds20k version 0x20241209
DS-DM firmware build 0x6a332b58, ds20k version 0x20241209
Polling SFP status...
identifier 0x03
connector 0x07
encoding 0x01
wavelength 0x0352 (850 nm)
vendor_name [FINISAR CORP. ]
vendor_pn [FTLF8526P3BNL ]
vendor_rev [A ]
vendor_sn [N3AB4LV ]
vendor_date [200319 ]
dm_type 0x68
temp 30.9 C
vcc 3.285 V
tx_bias 7.362 mA
tx_power 475.7 uW
rx_power 859.1 uW
SFP good 1, status: temp 30.9 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 859 uW
SFP good 1, status: temp 30.8 C, tx_bias 7.4 mA, tx_power 475 uW, rx_power 859 uW
SFP good 1, status: temp 30.8 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 859 uW
SFP good 1, status: temp 30.7 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 859 uW
SFP good 1, status: temp 30.8 C, tx_bias 7.4 mA, tx_power 475 uW, rx_power 858 uW
^C
root@cdm00:~#
</pre>
* NOTE1: if fiber cable is disconnected or GDM is not running, rx_power would read 0 uW
* NOTE2: better SFP modules have smaller tx_bias and larger tx_power (more bang for the buck)
* NOTE3: bad fiber cable and bad fiber connection will result in smaller rx_power

== test QSFP (GDM) ==

* connect QSFP module (any slot)
* ssh root@gdm01
* run "./test_cdm.exe --qsfp3 --qsfp", try --qsfp0, --qsfp1, --qsfp2, --qsfp3 for the 4 QSFP slots
<pre>
root@gdm01:~# ./test_cdm.exe --qsfp3 --qsfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x9e30baf9, ds20k version 0x20241209
DS-DM firmware build 0x9e30baf9, ds20k version 0x20241209
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out hi
Polling QSFP status...
identifier 0x0d
status 0x02
los 0x89
temp 36.7 C
vcc 3.302 V
rx_power 0.1 413.7 460.3 0.1 uW
tx_bias 7.8 7.9 7.9 0.0 mA
tx_power 806.0 842.9 865.2 0.1 uW
vendor_name [FINISAR CORP ]
vendor_pn [FTL410QD4C ]
vendor_rev [A ]
wavelength 850
max_temp 70 C
vendor_sn [X79AC0R ]
vendor_date [220309 ]
QSFP good 1, status: temp 36.7 C, los 0x89, tx_bias 7.8 7.9 7.9 0.0 mA, tx_power 806 843 865 0 uW, rx_power 0 414 460 0 uW
^C
</pre>
* NOTE1: 4th tx_bias and tx_power are zero because 4th channel does not transmit any data, QSFP detects this and automatically shuts down the optical transmitter

= Serial console =

* check that linux computer has correct udev rules to allow access to /dev/ttyACM devices, see https://daq00.triumf.ca/DaqWiki/index.php/SLinstall#Configure_USB_device_permissions and https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Configure_USB_device_permissions
* connect micro-USB cable to connector J-UCB, other end connect to linux computer
* observe /dev/ttyACM0 was created
* run "minicom -D /dev/ttyACM0 -b 115200"
* should have gdm-cdm login
* username root, password root

= i2c =

<pre>
ZynqMP> i2c bus
Bus 0: i2c@ff020000
ZynqMP> i2c dev 0
Setting bus to 0
ZynqMP> i2c probe
Valid chip addresses: 33 4E 53 5B 6B 77
ZynqMP> i2c md 0x5b 0x98
0098: fc c2 3d 00 00 12 1a 6e 0a 90 85 04 94 10 08 50 ..=....n.......P
</pre>

<pre>
root@gdm-cdm:~# i2cdetect 0
Warning: Can't use SMBus Quick Write command, will skip some addresses
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0.
I will probe address range 0x03-0x77.
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f
00:
10:
20:
30: -- -- -- 33 -- -- -- --
40:
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60:
70:
root@gdm-cdm:~#
root@gdm-cdm:~# i2cdump 0 0x5b
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0, address 0x5b, mode byte
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00 ???????P?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e ........??=..??n
a0: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00 ???????P?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e ........??=..??n
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@gdm-cdm:~#
</pre>

<pre>
root@gdm0:~# i2cdetect -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#
</pre>

<pre>
root@cdm0:~# i2cdetect -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: 50 51 -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@cdm0:~#
</pre>

* 0x33 - XU8 secure EEPROM (should be at 0x32)
* 0x4e - U23 current and temperature monitor
* 0x50, 0x51 - SFP
* 0x50 - QSFP, 4 QSFP modules enabled by GPIO QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL
* 0x53, 0x5b - ethernet mac eeprom
* 0x6b - U6 clock chip

== U23 ==

NOTE: Text reads double of Tint. not sure why. K.O. 21-mar-2024.

* internal temperature only

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x00 b # control register: "repeat mode, internal temperature only"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x03 # "Tint ready" and "busy", "busy is always 1 in repeat mode"
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 00 03 03 81 db 2a ce 2a 8e 00 6f 00 45 20 3f ?.????*?*?.o.E ?
10: 01 00 01 01 01 db 2a ce 2a 8e 00 6f 00 45 20 3f ?.????*?*?.o.E ?
...
readback:
reg0 - 03 - Tint ready
reg1 - 00 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB 0x81, bit 0x80 is "DV, data valid", bit 0x40 is "SS, sensor short", 0x20 is "SO, sensor open"
reg5 - Tint LSB 0xDB, Tint = 0x01DB = 475 * 0.0625 degC = 29.6 degC
reg6..F - stale data
</pre>

* Tint, V1, V2, TR2, VCC

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x18 b # control register: "repeat mode, V1, V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 7f 18 7f 7f 81 b9 aa d0 aa 8f 83 5e 83 5e a0 41 ???????????^?^?A
10: 01 18 01 01 01 b9 2a d0 2a 8f 03 5e 03 5e 20 41 ??????*?*??^?^ A
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x1b9*0.0625 = 27.5 degC
reg6 - V1 MSB 0xaa, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 LSB 0xd0, V1 = 0x2ad0*305.18/1000000 = 3.3447 V (correct)
reg8 - V2 MSB 0xaa, ditto
reg9 - V2 LSB 0x8f, V2 = 0x2a8f*305.18/1000000 = 3.3249 V (correct, smaller than V1)
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x5e. TR2 = 0x35e*0.0625 = 53.875 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x41, VCC = 2.5+0x2041*305.18/1000000 = 5.019 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.99 A (about right?)
</pre>

* Tint, V1-V2, TR2, VCC

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x19 b # control register: "repeat mode, V1-V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 7f 19 7f 7f 81 69 83 f2 83 f2 83 3a 83 3a a0 44 ?????i?????:?:?D
10: 01 19 01 01 01 69 03 f2 03 f2 03 3a 03 3a 20 44 ?????i?????:?: D
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x169*0.0625 = 22.5 degC
reg6 - V1 or V1-V2 MSB 0x83, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 or V1-V2 LSB 0xf2, V1-V2 = 0x3f2*19.42/1000000 = 0.0196142 V (correct, compare with V1 and V2 measured above)
reg8 - V2 or V1-V2 MSB
reg9 - V2 or V1-V2 LSB
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x3a. TR2 = 0x33a*0.0625 = 51.625 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x44, VCC = 2.5+0x2044*305.18/1000000 = 5.021 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.9807 A (about right?)
</pre>

== SFP ==

<pre>
root@cdm0:~# i2cdump 0 0x50
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 04 07 00 00 00 00 40 40 0c 00 01 3d 00 00 00 ???....@@?.?=...
10: 0c 02 00 1e 46 49 4e 49 53 41 52 20 43 4f 52 50 ??.?FINISAR CORP
20: 2e 20 20 20 00 00 90 65 46 54 4c 46 38 35 32 36 . ..?eFTLF8526
30: 50 33 42 4e 4c 20 20 20 41 20 20 20 03 52 00 9d P3BNL A ?R.?
40: 00 1a 00 00 4e 33 41 42 34 4c 56 20 20 20 20 20 .?..N3AB4LV
50: 20 20 20 20 32 30 30 33 31 39 20 20 68 f0 03 de 200319 h???
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@cdm0:~#
root@cdm0:~# i2cdump 0 0x51
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 5a 00 d8 00 55 00 e2 00 90 88 71 48 8c a0 75 30 Z.?.U.?.??qH??u0
10: 21 34 01 f4 1b 58 03 e8 31 2d 04 eb 1f 07 06 31 !4???X??1-?????1
20: 31 2d 00 64 27 10 00 9e 00 00 00 00 00 00 00 00 1-.d'?.?........
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
40: 00 00 00 00 3f 80 00 00 00 00 00 00 01 00 00 00 ....??......?...
50: 01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 b7 ?...?...?......?
60: 19 9f 80 c5 0e 17 12 c4 1f 99 00 00 00 00 30 00 ??????????....0.
70: 00 00 00 00 00 00 00 00 ff ff ff ff ff ff ff 01 ...............?
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@cdm0:~#
</pre>

== QSFP ==

<pre>
QSFP i2c enable lines, active low:

QSFP0_SEL - JENC-A 82 - PS-MIO40 - linux gpio 378
QSFP1_SEL - JENC-A 84 - PS-MIO41 - linux gpio 379
QSFP2_SEL - JENC-A 100 - PS-MIO44 - linux gpio 382
QSFP3_SEL - JENC-A 106 - PS-MIO43 - linux gpio 381 (notice 2 and 3 are out of order)
</pre>

<pre>
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
# echo 378 >> /sys/class/gpio/export ### SEL0 338+40
# echo 379 >> /sys/class/gpio/export ### SEL1 338+41
# echo 381 >> /sys/class/gpio/export ### SEL3 338+43
# echo 382 >> /sys/class/gpio/export ### SEL2 338+44
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) in hi
gpio-379 ( |sysfs ) in hi
gpio-381 ( |sysfs ) in hi
gpio-382 ( |sysfs ) in hi
root@gdm0:~# echo out >> /sys/class/gpio/gpio381/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio382/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio378/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio379/direction
root@gdm0:~#
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out lo
gpio-379 ( |sysfs ) out lo
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out lo
root@gdm0:~#
echo 1 >> /sys/class/gpio/gpio381/value
echo 1 >> /sys/class/gpio/gpio382/value
echo 1 >> /sys/class/gpio/gpio378/value
echo 1 >> /sys/class/gpio/gpio379/value
cat /sys/kernel/debug/gpio
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out hi
gpio-382 ( |sysfs ) out hi
root@gdm0:~#
root@gdm0:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#

NOTICE NOTHING AT ADDRESS 0x50
</pre>

<pre>
ENABLE QSFP0, OBSERVE IT IS AT ADDRESS 0x50

root@gdm0:~# echo 0 >> /sys/class/gpio/gpio378/value
root@gdm0:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: 50 -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#
root@gdm0:~# i2cdump -y 0 0x50
No size specified (using byte-data access)
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 0d 00 02 8e 00 00 01 00 00 05 55 00 05 00 00 00 ?.??..?..?U.?...
10: 00 00 00 00 00 00 1e b8 00 00 81 a0 00 00 00 00 ......??..??....
20: 00 00 12 12 00 01 00 01 00 01 0e e0 0f 20 0e e0 ..??.?.?.???? ??
30: 00 00 1f a0 1e 3b 1f 72 00 01 00 00 00 00 00 00 ..???;?r.?......
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
60: 00 00 00 00 00 00 00 00 00 00 1f 00 00 00 08 00 ..........?...?.
70: 00 00 00 00 00 00 00 ff ff ff ff ff ff ff ff 00 ................
80: 0d 00 0c 04 00 00 00 40 40 02 d5 05 67 00 00 96 ?.??...@@???g..?
90: 00 00 c8 00 46 49 4e 49 53 41 52 20 43 4f 52 50 ..?.FINISAR CORP
a0: 20 20 20 20 07 00 90 65 46 54 4c 34 31 30 51 44 ?.?eFTL410QD
b0: 34 43 20 20 20 20 20 20 41 20 42 68 07 d0 00 43 4C A Bh??.C
c0: 00 07 0f de 58 37 39 41 43 30 52 20 20 20 20 20 .???X79AC0R
d0: 20 20 20 20 32 32 30 33 30 39 20 20 3c 10 00 9e 220309 <?.?
e0: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@gdm0:~#
</pre>

== ethernet mac eeprom ==

* correct chip with 84-bit ethernet mac address

<pre>
root@cdm1:~# i2cdump 0 0x53
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
root@cdm1:~#

root@cdm1:~# i2cdump 0 0x5b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00 ????4??2?K?.?...
90: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c ..........??=?Q<
a0: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00 ????4??2?K?.?...
b0: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c ..........??=?Q<
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@cdm1:~#
</pre>

* wrong "602" chip with 64-bit IPv6 address

<pre>
root@cdm0:~# i2cdump 0 0x53
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
root@cdm0:~# i2cdump 0 0x5b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00 ???????Q?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e ........??=..??.
a0: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00 ???????Q?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e ........??=..??.
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@cdm0:~#
</pre>

= read ethernet mac address from i2c =

(this code is copied from uboot command line i2c code)

in uboot sources board/xilinx/common/board.c replace original function with this:

<pre>
// special code to read ethernet MAC address from the DS-DM-Rev0 board. K.O. Sep-2022

int zynq_board_read_rom_ethaddr(unsigned char *ethaddr)
{
struct udevice *bus;
int ret;
int busnum = 0;

ret = uclass_get_device_by_seq(UCLASS_I2C, busnum, &bus);
if (ret) {
printf("%s: No bus %d\n", __func__, busnum);
return ret;
}

int chip_addr = 0x5B;

struct udevice *dev;

ret = i2c_get_chip(bus, chip_addr, 1, &dev);
if (ret) {
printf("%s: Bus %d no chip 0x%02x\n", __func__, busnum, chip_addr);
return ret;
}

int dev_addr = 0x98;

unsigned char data[8];

ret = dm_i2c_read(dev, dev_addr, data, 8);

if (ret) {
printf("%s: Bus %d chip 0x%02x read error %d\n", __func__, busnum, chip_addr, ret);
return ret;
}

printf("%s: Bus %d chip 0x%02x addr 0x%02x read: 0x %02x %02x %02x %02x %02x %02x %02x %02x\n", __func__, busnum, chip_addr, dev_addr, data[0], data[1], data[2], data[3], data[4], data[5], data[6], data[7]);

// see Atmel-8807-SEEPROM-AT24MAC402-602-Datasheet.pdf

if (data[0] == 0) {
// eiu-48 chip
ethaddr[0] = data[2];
ethaddr[1] = data[3];
ethaddr[2] = data[4];
ethaddr[3] = data[5];
ethaddr[4] = data[6];
ethaddr[5] = data[7];
} else {
// eiu-64 chip
ethaddr[0] = data[0];
ethaddr[1] = data[1];
ethaddr[2] = data[2];
ethaddr[3] = data[5];
ethaddr[4] = data[6];
ethaddr[5] = data[7];
}

printf("%s: ethaddr %02x:%02x:%02x:%02x:%02x:%02x\n", __func__, ethaddr[0], ethaddr[1], ethaddr[2], ethaddr[3], ethaddr[4], ethaddr[5]);

return ret;
}
</pre>

also this should have worked if i2c_xxx() functions were enabled in uboot:
<pre>
i2c_set_bus_num(0);
i2c_probe(0x5b);
i2c_read(0x5b, 0x9a, ethaddr, 6);
</pre>

= read ethernet mac address from i2c (SHOULD WORK) =

from: https://stackoverflow.com/questions/43637540/is-there-linux-or-u-boot-support-to-read-a-mac-address-from-a-chip-at-startup

<pre>
#ethernet related setup
setup_eth=run readmac buildmac
#read mac address from eeprom
readmac=i2c dev 0; i2c read 50 FA.1 6 $loadaddr
#build the ethaddr variable
#not very nice, but does the job
buildmac=\
e=" "; sep=" " \
for i in 0 1 2 3 4 5 ; do\
setexpr x $loadaddr + $i\
setexpr.b b *$x\
e="$e$sep$b"\
sep=":"\
done &&\
setenv ethaddr $e
</pre>

= read ethernet mac address from i2c (DOES NOT WORK) =

this method does not work: inside board/xilinx/common/board.c:int zynq_board_read_rom_ethaddr(unsigned char *ethaddr), uclass_get_device_by_ofnode(UCLASS_I2C_EEPROM, eeprom, &dev) returns ENODEV and read from i2c does not happen. K.O. Sep-2022

Read:
* https://support.xilinx.com/s/question/0D52E00006hpKlsSAE/uboot-and-ethernet-mac-address-from-eeprom?language=en_US (last answer)
* https://github.com/Xilinx/u-boot-xlnx/blob/master/board/xilinx/common/board.c (zynq_board_read_rom_ethaddr())

Note:
* 0x5B is the i2c chip address
* 0x9A is the data offset inside the chip, see datasheet or i2c read dump above.

Edit:
* emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/u-boot/files/platform-top.h
<pre>
#include <configs/xilinx_zynqmp.h>
#include <configs/platform-auto.h>
//#define CONFIG_I2C_EEPROM
//#define CONFIG_SYS_I2C_EEPROM_ADDR 0x5b
//#define CONFIG_SYS_I2C_EEPROM_ADDR_OVERFLOW 0x0
#define CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET 0x9A
#error HERE!
</pre>
* emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi
<pre>
/include/ "system-conf.dtsi"
/ {
chosen {
xlnx,eeprom = &eeprom;
};
};

&i2c0 {
eeprom: eeprom@5b { /* u88 */
compatible = "atmel,24mac402";
reg = <0x5b>;
};
};
</pre>
* components/yocto/workspace/sources/u-boot-xlnx/configs/xilinx_zynqmp_virt_defconfig
<pre>
CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET=0x9A
</pre>

= RTC chip =

* DS3231 RTC chip
* FPGA connections:
<pre>
I2C SCL <- J-ENC A85 <- FPGA E17 <- XDC TP_S <- VHDL RTC_I2C_SCL output (10k pull-up to 3.3V)
I2C SDA <-> J-ENC A87 <-> FPGA D17 <-> XDC TP_S <-> VHDL RTC_I2C_SDA bidir (10k pull-up to 3.3V)
RTC_1Hz -> J-ENC C160 -> FPGA AH12 -> XDC "slow_io" -> VHDL RTC_1PPS input (10k pull-up to 1.2V)
RTC_32k -> J-ENC B129 -> FPGA AE3 -> XDC "free pin" -> VHDL RTC_32KHZ input (10k pull-up to 1.8V)
</pre>
* test notes
<pre>
FPGA registers:

rtc_i2c_clock_ie <= not register_data_in_block_0(14)(0);
rtc_i2c_clock_out <= register_data_in_block_0(14)(1);
rtc_i2c_data_ie <= not register_data_in_block_0(14)(4);
rtc_i2c_data_out <= register_data_in_block_0(14)(5);

ext_out(1) <= rtc_1pps_in;
ext_out(2) <= rtc_32khz_in;

register_data_out_block_0(13) <= x"000000" & rtc_i2c_data_ie & rtc_i2c_data_out & rtc_i2c_clock_ie & rtc_i2c_clock_out & rtc_1pps_in & rtc_32khz_in & rtc_i2c_data_in & rtc_i2c_clock_in;

After power us on the RTC chip:
i2c clock 3.3V
i2c data 3.3V
32kHz 0.778V see clock running with amplitude 0->1.8V (FPGA pullup) period about 30 usec (~33 kHz)
1pps 1.2V (FPGA pullup)

./test1.exe --read32 0x34
dsdm_read32[0x00000034] is 0x000000ab (171) -> 1010'1011 -> data_ie, clock_ie, 1pps, data_in, clock_in
./test1.exe --read32 0x38
dsdm_read32[0x00000038] is 0x00000000 (0)

./test1.exe --write32 0x38 0x01 -> clock oe
./test1.exe --read32 0x34
Ddsdm_read32[0x00000034] is 0x0000008a (138) -> 1000'1010 -> data_ie, 1pps, data_in, clock 0V, data 3.3V

./test1.exe --write32 0x38 0x10 -> data_oe
./test1.exe --read32 0x34
dsdm_read32[0x00000034] is 0x00000029 (41) -> 0010'1001 -> clock_ie, 1pps, clock_in, clock 3.3V, data 0V

root@dsdm:~# ./test_cdm_local.exe --rtc-read
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Read the RTC chip!
reg 0x00: 0 1 0 1 0 1 1 1 0x57
reg 0x01: 0 0 1 0 0 0 1 0 0x22
reg 0x02: 0 0 0 0 0 0 0 0 0x00
reg 0x03: 0 0 0 0 0 0 0 0 0x00
reg 0x04: 0 0 0 0 0 0 0 0 0x00
reg 0x05: 0 0 0 0 0 0 0 0 0x00
reg 0x06: 0 0 0 0 0 0 0 0 0x00
reg 0x07: 0 0 0 0 0 0 0 0 0x00
reg 0x08: 0 0 0 0 0 0 0 0 0x00
reg 0x09: 0 0 0 0 0 0 0 0 0x00
reg 0x10: 0 0 0 0 0 0 0 0 0x00
reg 0x11: 0 0 0 0 0 0 0 0 0x00
reg 0x12: 0 0 0 0 0 0 0 0 0x00
reg 0x13: 0 0 0 0 0 0 0 0 0x00
reg 0x14: 0 0 0 1 1 0 0 0 0x18
reg 0x15: 0 0 0 0 1 0 0 0 0x08
reg 0x16: 0 0 0 0 0 0 0 0 0x00
reg 0x17: 0 0 0 1 0 1 1 1 0x17
reg 0x18: 1 1 0 0 0 0 0 0 0xc0
seconds: 57, minutes: 22, hours: 00, day 0, date: 00-00-00, temp 23.75C
root@dsdm:~#
</pre>

= clock chip configuration =

file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt:

* VCO is 14 GHz
* Tvco is 71.43 ps
* N0 divider is 14, frequency is 1000 MHz
* out0 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz
* out1 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
* out2 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
* out3 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz is the feedback for zero delay

= clock chip monitoring =

from si5395-94-92-family.pdf:
* reg 0x1: page select, set to 0 or set to 5 to read 0x53F
* reg 0x2: 0x94
* reg 0x3: 0x53 -> device is a si5394
* reg 0xC: LOSXAXB
* reg 0xD: LOS and OOF for the 4 clock inputs
* reg 0xE: LOL and HOLD
* reg 0xF: CAL_PLL
* reg 0x11: sticky bits for reg 0xC
* reg 0x12: sticky bits for reg 0xD
* reg 0x13: sticky bits for reg 0xE
* reg 0x14: sticky bits for reg 0xF
* reg 0x1C: device reset
* reg 0x1E: low power, hard reset, SYNC
* reg 0x507: currently selected input clock
* reg 0x52A: input clock select
* reg 0x535: FORCE_HOLD
* reg 0x53F: HOLD_HIST_VALID and FASTLOCK_STATUS

= Install Xilinx tools =

* install Vivado 2020.2
<pre>
login at https://www.xilinx.com/myprofile.html
go to "Downloads"
go to archive,
find 2020.2
download Xilinx_Unified_2020.2_1118_1232_Lin64.bin
sh ./Xilinx_Unified_2020.2_1118_1232_Lin64.bin
banner window should open with spinner "downloading installation data"
"a newer version is available" -> say "continue"
next
"select install type" window:
provide email and password,
select "download image"
select directory /home/olchansk/Xilinx/Downloads/2020.2\
select "linux" and "full image"
next
download summary: space required 38.52 Gbytes
download
installation progress
downloading spinner, 16 M/s 47 minutes...
"download image has been created successfully". Ok.
check contents of /home/olchansk/Xilinx/Downloads/2020.2
ls -l /home/olchansk/Xilinx/Downloads/2020.2
total 67
drwxr-xr-x 2 olchansk users 9 Sep 1 16:22 bin
drwxr-xr-x 3 olchansk users 15 Sep 1 16:23 data
drwxr-xr-x 4 olchansk users 4 Sep 1 16:22 lib
drwxr-xr-x 2 olchansk users 644 Sep 1 16:22 payload
drwxr-xr-x 2 olchansk users 7 Sep 1 16:22 scripts
drwxr-xr-x 4 olchansk users 4 Sep 1 16:22 tps
-rwxr-xr-x 1 olchansk users 3256 Nov 18 2020 xsetup
daq13:2020.2$
./xsetup
spinned loading installation data
xilinx design tools 2022.1 now available -> say continue
"welcome" -> next
"select product" -> vivado -> next -> vivado hl system edition -> next
select devices: only zynq ultrascale+ mpsoc -> next
select destination: /opt/Xilinx (as root, mkdir /opt/Xilinx, chmod olchansk.users /opt/Xilinx)
install ...
complete
move /home/olchansk/Xilinx/Downloads/2020.2 to /daq/daqstore/olchansk/Xilinx/Downloads/
</pre>
* install petalinux 2020.2
<pre>
./xsetup
"a newer version is available" -> say "continue"
next
"select product to install" -> select Petalinux (Linux only) -> next
"select destination directory" -> select "/opt/Xilinx" (disk space required 2.64 GB) -> next
"summary" -> install ...
error about missing /tmp/tmp-something files
"installation completed successfully" (hard to dismiss, "ok" button is partially cut-off)
done?
I think it failed, /opt/Xilinx/PetaLinux/2020.2/bin is empty except for petalinux-v2020.2-final-installer.run
try to run it by hand, same error about /tmp/tmp-something files. strange...
notice it complains about "truncate", which truncate finds ~/bin/truncate, get rid of it,
try again
now complains about missing texinfo and zlib1g:i386
apt install texinfo -> ok
apt install zlib1g:i386 -> installs bunch of gcc stuff -> ok
try again
reports "already installed" -> delete /opt/Xilinx/.xinstall/PetaLinux_2020.2/, delete entries in ~/.Xilinx/registry/installedSW.xml
try again
success
</pre>
* install vivado 2022.1 and petalinux 2022.1 - everything is pretty much the same

= Petalinux =

* cd PetaLinux_GDM_CDM
* petalinux-config
* enable i2c MAC address and DHCP
<pre>
git clone https://bitbucket.org:/team-ds-dm/ds-dm-u-boot-xlnx.git
cd ds-dm-u-boot-xlnx
git checkout ds-dm-u-boot-xlnx

linux-components ->
uboot -> ext-local-src
external u-boot local source -> ds-dm-u-boot-xlnx (path to the customized uboot git repository)
</pre>
* enable DHCP
<pre>
Subsystem AUTO Hardware Settings -> Ethernet Settings
randomize MAC address -> NO
ethernet mac address -> leave empty
obtain ip address automatically -> YES
</pre>
* set hostname and product names
<pre>
Firmware Version Configuration ->
Host name -> "ds-dm"
Product name -> "Petalinux_GDM_CDM"
</pre>
* configure linux kernel
<pre>
petalinux-config -c kernel
</pre>
* enable NFS-Root
<pre>
petalinux-config
Image Packaging Configuration > Root File System Type -> set to NFS
Location of NFS root directory set to "/nfsroot"

petalinux-config -c kernel
Networking support > IP: kernel level configuration
enable DHCP, BOOTP, RARP
File systems > Network file systems > Root file systems on NFS
</pre>
* manually fix linux kernel command line:
<pre>
grep nfsroot PetaLinux_GDM_CDM/project-spec/configs/config
edit CONFIG_SUBSYSTEM_BOOTARGS_GENERATED to read
earlycon console=ttyPS0,115200 clk_ignore_unused panic=60 root=/dev/nfs nfsroot=/nfsroot/%s ip=dhcp rw
</pre>
* check configuration in
** PetaLinux_GDM_CDM/project-spec/configs/config
** PetaLinux_GDM_CDM/project-spec/configs/rootfs_config
** PetaLinux_GDM_CDM/components/plnx_workspace/device-tree/device-tree/system-conf.dtsi

= JTAG server =

localhost:3121

= ds20k block =

<pre>
module ds20k
(
// CLOCK INPUTs
input wire clk,
input wire reset, // pulse for power-up reset
input wire si5394_loln, // clock cleaner PLL is locked to selected input clock

// REGISTER_DATA

input wire [255:0] [31:0] register_data_in,
output reg [255:0] [31:0] register_data_out,
input wire register_we,
input wire register_re,

// GDM QSFP FIBER LINKS
input wire qsfp_tx_data_ready,
output reg [11:0] [15:0] qsfp_tx_data,
output reg [11:0] [1:0] qsfp_tx_data_is_k,
input wire [11:0] [15:0] qsfp_rx_data_error, // not sure what this is.
input wire [11:0] [15:0] qsfp_rx_data,
input wire [11:0] [1:0] qsfp_rx_data_is_k,

// CDM SFP FIBER LINKS
input wire sfp_tx_data_ready,
output reg [15:0] sfp_tx_data,
output reg [1:0] sfp_tx_data_is_k,
input wire sfp_rx_data_error, // single bit indicating that RX link is up and data is good.
input wire [15:0] sfp_rx_data,
input wire [1:0] sfp_rx_data_is_k,

// VX_RXs
input wire [3:0] vx1_rx_async,
input wire [3:0] vx2_rx_async,
input wire [3:0] vx3_rx_async,
input wire [3:0] vx4_rx_async,
input wire [3:0] vx5_rx_async,
input wire [3:0] vx6_rx_async,
input wire [3:0] vx7_rx_async,
input wire [3:0] vx8_rx_async,
input wire [3:0] vx9_rx_async,
input wire [3:0] vx10_rx_async,
input wire [3:0] vx11_rx_async,
input wire [3:0] vx12_rx_async,

// VX_TXs
output reg [2:0] vx1_tx_out,
output reg [2:0] vx2_tx_out,
output reg [2:0] vx3_tx_out,
output reg [2:0] vx4_tx_out,
output reg [2:0] vx5_tx_out,
output reg [2:0] vx6_tx_out,
output reg [2:0] vx7_tx_out,
output reg [2:0] vx8_tx_out,
output reg [2:0] vx9_tx_out,
output reg [2:0] vx10_tx_out,
output reg [2:0] vx11_tx_out,
output reg [2:0] vx12_tx_out,

// VX serial comunications
input wire [11:0] vx_rx_ser_in,
output reg [11:0] vx_tx_ser_out,

// LEMO INPUTs
input wire [4:1] ext_in_async, // direct connection to LEMO connectors, not clocked

// LEMO OUTPUTs
output reg [2:1] ext_out, // direct connection to LEMO connectors, not clocked

// FRONT PANEL LEDs
output reg [3:0] fp_led_out, // direct connection to LEMO connectors, not clocked
);
</pre>

= world view =

Main components:
* clock distribution - to ensure all digitizers run synchronously and waveform timestamp are easy to assemble intophysics events.
* global trigger and run control - all digitizers are triggered at the same time to record calibration events, pulser events and the run startup synchronization sequence.
* busy distribution - when some digitizers go busy and stop accepting triggers, physics events may become incomplete, this must be recorded and managed.
* hitmap distribution - to create an online picture of the detector for monitoring and for triggering.

== global clock distribution ==

Clock distribution ensures all digitizers run synchronously (on the same clock) and waveform timestamps are easy to assemble into physics events. The clock source is synchronized with GPS to ensure a metrologically validated clock frequency (tied to the SI definition of 1 second) and date stamp (via GPS IRIG-B date stamp). Individual digitizer waveform timestamp are reset at the begin of run to establish a valid time zero (run start time, date stamped with the GPS IRIG-B date stamp).

Base clock is 125 MHz (8 ns). Individual digitizers are synchronized at initialization (power-up) time. Because of incontrollable delays in the electronics (i.e. variations between individual PLL chips in each digitizer), waveform timestamps between individual digitizers may be off by 1-2 clocks (8-16 ns). After the system is running and is stable (thermally and electrically), timestamp waveform variation and jitter are less than 1 ns.

This is validated by feeding a common calibration pulser signal into analog inputs of each digitizer (at 100 Hz) and measuring (fitting) the pulse time in the waveform. With sufficient statistics, differences between waveform timestamp are measured with picosecond precision.

CDM PLL allows fine phase adjustment of the 62.5 MHz VX2746 clock in groups of 6 clocks (2 groups per CDM) with precision 71.43 ps.

Due to FPGA technology limitations (serial link hard IPs, PLLs, etc) the pattern of 1-2 clock delays between individual digitizers may change/shift from power up to power up. Significant additional work would be required to ensure this pattern is repeatable and stable after every power up and initialization. As mentioned before, after power up and initialization, variation between waveform timestamp is sub-nanosecond as measured by the analog calibration pulser.

Clock chain:
* GPS receiver (1pps clock reference and IRIG-B date stamp)
* LNGS GPS clock distribution (fiber link to converter box at the experiment)
* GPS 1pps to PRS-10 Rb atomic clock (10 MHz clock output)
* 10 MHz clock from PRS-10 to GDM PLL (extra low jitter oscillator, 125 MHz clock output)
* 125 MHz clock to GDM fiber link transmitters (2.5 Gbps line rate)
* CDM fiber link receiver and clock recovery (125 MHz clock)
* 125 MHz recovered clock to CDM PLL (extra low jitter oscillator, 62.5 MHz clock output)
* 62.5 MHz CDM clock to VX2745 PLL via LVDS line
* VX2745 PLL (125 MHz clock output)
* 125 MHz clock to digitizer chips, clock counter and waveform timestamp.

This is the normal mode of operation, with global clock synchronized with the GPS clock (metrologically referenced to SI 1 Hz definition).

If GPS signal is not available, degraded modes of operation are available (in order of worsening performance)
* PRS-10 Rb atomic clock synchronized with a battery-powered time-of-day chip
* PRS-10 Rb atomic clock synchronized with Internet time (NTP) assuming Internet network connection is up
* PRS-10 Rb clock standalone
* GDM internal oscillator

All degraded modes provide high precision low jitter 125 MHz system clock, but long term stability (seconds per day) will vary. (TBM)

In the absence of GPS IRIG-B date stamp, the battery-powered time-of-day chip will be used (TBI) with fallback to Internet time (NTP). Date stamp long term stability is expected to be better than 1 second per day (TBI, TBM).

Note:
* TBM: to be measured
* TBI: to be implemented

Current status (April 2025):
* full clock chain from GDM PLL to VX2745 digitizers is implemented and tested
* waveform timestamp variations are measured (see report from Marek)
* GDM external clock is validated (using VME-GRIFFIN-CDM high quality clock source module)
* PRS-10 10 MHz and 1pps connection to GDM is implemented and tested
* GPS receiver 1pps connection to PRS-10 is implemented and tested
* GPS receiver IRIG-B date stamp connection to GDM tested using a software decoder. Hardware decoder and connection to time slice marker, TBI.
* LNGS date stamp decoder and connection to time slice marker, TBI
* LNGS 1pps connection to PRS-10, not available
* operate vertical slize system from GPS, not available (GPS is in the det lab, vertical slice is in the daq lab).

== global trigger distribution ==

Global trigger distribution provides a synchronous waveform trigger to all VX2745, all CDM and all VX2745 are provided wiht the same data and are all triggered at the same time (without accounting for the 1-2 clock offset described in the clock section).

Global trigger is used for run control: the first trigger of a run resets the waveform timestamp counter, the CDM timestamp counter and the GDM timestamp counter. At the begin of run, GDM generates a sequence of triggers following an easy to identify pattern. If a VX2745 misses a trigger (for any reason), online and offline software can easily identify the tru run start time using this pattern.

After a run has started, global trigger can be generated by a programmable pulser, by an external LEMO signal (i.e. external calibration signal, laser Q-switch, etc) and by the VX2745 trigger hitmap.

Trigger sources (TRG):
* TRG pulser
* LEMO inputs
* hitmap trigger

Time slice marker sources (TSM):
* TSM pulser

Trigger chain, K-code and LVDS:
* TRG/TSM source
* TRG/TSM K-code from GDM to CDM
* CDM K-code to LVDS line 12 and 13 lines (TRG and TSM)

Latency from GDM LEMO input to CDM LVDS out: TBM

Trigger chain, packet trigger:
* GDM TRG/TSM source to TRG/TSM packet generator
* or GDM hitmap group trigger packet generator
* TRG/TSM packet from GDM to CDM
* TRG/TSM packet from CDM to VX2745 via LVDS line, 62.5 MHz, 8b10b encoding.

Latency from GDM LEMO input to VX2745 internal trigger: TBM

Notes:
TBM - to be measured
TBI - to be implemented

Current status (Jan 2026):
* LVDS TRG and TSM distribution is done and tested
* packet TRG, TRIG48 and TSM distribution from GDM to CDM is done and tested
* packet TRG, TRIG48 and TSM from CDM to VX2745 is done and tested
* packet TRIG48 processing in the VX is done and tested

== global busy distribution ==

If any VX2745 goes busy (for any reason) and stops accepting triggers, physics data may become incomplete. This situation needs to be recorded and managed.

Busy chain:
* VX2745 firmware busy logic (CAEN and Yair) generates the VX busy signal on LVDS line 9 (CDM VX_RX_2)
* CDM monitors VX busy
* CDM records busy transitions
* CDM computes the "CDM busy" as grand-OR of all VX busy
* CDM periodically transmits CDM busy status the GDM using special K-codes (NB: confirm transmission period)
* GDM receives CDM busy updates and maintains "CDM busy" for each CDM
* GDM records busy transitions
* GDM computes "GDM busy" as grand-OR of all CDM busy
* effectively, GDM busy is a grand-OR of all VX busy (with max delay, TBM)

In the pilot implementation, any VX busy stops the experiment by sending a veto signal to all VX2745 which stops them from accepting triggers.

Veto chain:
* GDM computes "GDM veto" as GDM busy
* GDM periodically transmits GDM veto status to all CDMs using special K-codes (NB: confirm transmission period)
* CDM receive GDM veto updates and maintain the "CDM veto" signal
* CDM computes "VX veto" as CDM veto
* CDM transmits VX veto to all VX2745 on LVDS pair 13
* VX2745 firmware (Yair) has VX veto block all physics triggers (NB: except for the time slice marker?)

Total delay from VX busy to VX veto (on the LVDS lines), TBM.

Notes:
* TBM - to be measured
* TBI - to be implemented

Current status (Jan 2026):
* full busy chain from VX to CDM to GDM is implemented, tested
* veto chain from GDM to CDM is implemented, tested
* veto chain from CDM to LVDS pair 13 is implemented, tested
* LVDS pair 13 veto inside the VX is implemented, tested
* TRG switched from LVDS pair 12 to TRG packet, testing completed
* TSM switched from LVDS pair 13 to TSM packet, testing completed
* fine tuning of busy chain, VX buffer levels, etc, test in progress

== vx hitmap and hitmap trigger ==

To monitor the system in real time and to generate group triggers, we use the VX hitmaps.

Each VX2745 has 64 channels, each channel has a trigger logic block (digital discriminator) that generates a waveform self-trigger and sends a hitmap bit to the CDM (64-bits per VX2745) as a data packet. CDM passes hitmap data packets to the GDM and also records non-empty hitmap packets. CDM MIDAS frontend passes non-empty hitmap packets to MIDAS for online monitoring and visualization. GDM collects all the hitmaps and generates a global trigger. Alternatively, it generates a group trigger TRIG48 with a 48-bit bitmap indicating which VX2745 should trigger a waveform.

Pilot implementation computes the hitmap trigger as a grand-OR of all hit maps.

Hitmap chain:
* VX2745 analog inputs to digitizers to digital trigger logic block, 1 bit per channel (CAEN, Yair)
* 64-bits of hitmap (1 bit/channel) are collected and periodically transmitted to the CDM (Sam De Jong) (NB: confirm transmission period: see CDM page table "VX hit map" column "period", 0xD1 clocks is 1672 ns, should be reduced to 1500 ns, TBI)
* VX hitmap packets are transmitted to the CDM on the LVDS line 10 (CDM VX_RX_1) at 62.5 MHz, 8b10b encoded.
* CDM receives hitmap packets (12 input data streams, 12-to-1 packet mux)
* CDM sends all hitmap packets to the GDM at 125 MHz, 16b20b encoded.
* non-empty hitmap packets are written to the CDM-to-CPU data FIFO (and via MIDAS frontend to MIDAS SYSTEM event buffer)
* grand-OR of VX hitmap packets (12*64 bits) can generate a CDM-local VX trigger
* GDM receives hitmap packets, computes "GDM hitmap trigger"
* if enabled, GDM hitmap trigger causes GDM trigger, which is processed as described in the global trigger section

Delay, latency and jitter from analog pulse on the VX2745 analog input to VX trigger on LVDS line 12, TBM.

TRIG48 VX group trigger:
* GDM receives hitmap packets, computes the 48-bit bitmap of which VX2745 modules should trigger
* how to specify this computation is not clear. a generic 12*64 MLU is impossible, insufficient RAM on the FPGA
* 48-bit bitmap is sent to all CDMs as a TRIG48 data packet
* CDM receives TRIG48 data packets from GDM and retransmits them to the VX2745 (all CDMs and all VXes see the same packets at the same time).
* VX2745 receives the 48-bit trigger bitmap, compares it with it's own 48-bit bitmap if any common bits are set, triggers waveform readout of all channels (there will be no more than 48 VX2745 in the experiment)

Pilot implementation of TRIG48 triggers on the CDM grand-or and generates a TRIG48 packet that contains a 48-bit bitmap stored in an AXI register.

Example uses of TRIG48:
* to trigger individual VX2745 modules: set the VX 48-bit map to the module number (vx05-0x1, vx06-0x2, vx07-0x4, etc), generate TRIG48 packet with a bitmap of VX modules that should trigger: 0xFFFF'FFFF'FFFF will trigger all modules, 0x1 will trigger only vx05, 0x2 will trigger only vx06, etc. (there is no more than 48 modules).
* to trigger groups of VX2745 modules: set the bitmap of group 1 modules to 0x1, set the bitmap of group 2 modules to 0x2, generate TRIG48 packet with bitmap 0x0001 to trigger only group 1 modules, 0x0002 to trigger only group 2 modules, 0x3 to trigger both groups.

Notes:
* TBM - to be measured
* TBI - to be implemented

Current status (Jan 2026):
* full chain (VX analog input to hitmap generator to CDM to GDM) is implemented, testing in progress
* GDM grand-or trigger is implemented
* VX hitmap packet generator, testing in progress (Yair firmware)
* CDM hitmap processing, implemented, tested
* GDM hitmap processing, implemented, tested
* pilot TRIG48 trigger generator in GDM, CDM, tested
* TRIG48 trigger in VX, implemented, tested
* MIDAS frontend readout of hitmap data from FPGA FIFO: implemented, testing in progress (FIFO code: Ian, done; integration and C++ code: KO)
* online monitoring and visualization in MIDAS: TBI

== firmware map ==

Note:
* red lines: clocks
* green lines: AXI/Avalon packet streams
* blue lines: serial data

[[File:GDM-CDM-VX.drawio.svg]]

== description ==

same thing, in words:

<pre>
detector
digitizer, 125 MHz
digital filter
digital discriminator
hit map, 64 bits at 125 MHz (could be 250 MHz, filter and discriminator clock)
packetizer, 64 bits -> id, timestamp, 8x 8-bit words, eop
8/10 serializer, 12.5 MHz parallel in, 125 MHz serial out
lvds line to CDM vx_rx
BBB: also VX busy to lvds line to CDM vx_rx, do not want to depend on serial comm for vx busy, do not want to inject async data into the hit map packet stream
--- CDM
lvds line capture in IBUF register, vxN_rx clock, 125 MHz clock, from PLL with adjustable phase, scan phase to find sweet spot
vx rx 10/8 deserializer, vxN_rx clock, stobes out parallel data every 10 clocks at 12.5 MHz
vx rx phase transfer from vxN_rx clock to main CDM clock, have 10 clocks for transfer to happen
from here everything is on the CDM main clock
vx rx packet adapter, 12.5 MHz 8-bit packets to 125 MHz 16-bit packets, internal FIFO to avoid data overrun
sfp tx mux, all VX packet streams into one CDM sfp tx packet stream
BBB: capture VX RX busy to CDM main clock (IBUF register), grand-or becomes vx_rx_busy
sfp tx packetizer (data,eop,vx_rx_busy -> data,k)
sfp tx 16/20 serializer, 16 bit at 125 MHz in, 2500 MHz serial out (20 bits at 125 MHz)
sfp tx fiber link to GDM, 2 Gigabits/sec
--- GDM
qsfp rx deserializer, 16 bit at 125 MHz output on main GDM clock
qsfp rx depacketizer (data,k -> packet data,eop; qsfp_rx_busy)
qsfp rx demux, hit map packets routed to GDM trigger logic block
GDM trigger logic block looks at hit map, generates yes/no trigger decision, encodes it as a trigger packet
TSM generator encodes GPS time data as a TSM packet
qsfp tx mux - trigger packets, tsm packets, etc to qsfp tx packet stream
qsfp tx packetizer (data,eop;trg,tsm,bsy -> data,k)
NB: the same tx data is sent to all 12 qsfp tx ports, to make sure we do not accidentally desync the CDMs.
qsfp tx 16/20 serializer, 16-bit at 125 MHz to 20-bit at 250 MHz to 2500 MHz serial out
qsfp tx fiber link to CDM, 2 Gigabits/sec
--- CDM
sfp rx 20/16 deserializer, 16-bit at 125 MHz
sfp rx depacketizer, (data,k -> data,eop;trg_in,tsm_in,bsy_in)
sfp rx demux (in reality, noop, all packets go to same place, vx tx)
vx tx mux (packets from GDM, packets with simulated waveforms from CDM midas frontend via AXI FIFO)
vx tx packet adapter 16-bit at 125 MHz to 8-bit at 12.5 MHz strobed every 10 clocks. NB: most important, this 10-clock strobe runs in sync between all CDMs!
vx tx phase transfer from main CDM clock to vxN_tx clock, have 10 clocks for transfer to happen
from here we run on the vxN_tx clock
vx txN serializer, 8-bit at 12.5 MHz to 125 MHz serial
vx_txN OBUF register, vxN_rx clock, 125 MHz clock, from PLL with adjustable phase, scan phase to find sweet spot
lvds line to vx
BBB: bsy_in from GDM is converted from pulse to level, goes out lvds line to vx, sync to vxN_tx clock
TTT: trg_in from GDM is a pulse, does out lvds line to vx, sync to vxN_tx clock.
NB: tsm is always a packet, bsy is always a signal (no packet), trg can be a packet or signal.
--- VX
lvds data captured by 125 MHz ADC sampling clock (CAEN base firmware logic)
lvds data connected to Yair's block
10/8 deserializer
depacketizer (data,k -> data,eop) to avoid accidental desync, we do not send any no trg, no tsm, no bsy K-codes.
demux
trigger packets go their way (4x 64 bit words of data go to event header: timestamps, hitmap data)
tsm packets go their way (4x 64 bit words of data go to tsm event header: timestamps, GPS time data)
BBB: bsy from lvds line stops waveform acquisition
TTT: trg from lvds line cause waveform acquisition, same as trg packet, but has no timestamp and other data attached to it.
</pre>

== CDM-VX connections ==

Note:
* rounded boxes are VX registers set from MIDAS

[[File:CDM-VX.drawio.svg]]

= Firmware registers =

== Block 0 ==

<pre>
0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
</pre>

== Block 0 GDM ==

* 0x0000+4*i - GDM AXI clock registers

<pre>
register_data_out_block_0(1) <= x"204D4447"; --GDM<space>
register_data_out_block_0(2) <= x"00000000"; --blank
register_data_out_block_0(3) <= x"00000000"; --blank

--Firmware Version
register_data_out_block_0(4) <= x"abcd0123"; --Firmware version name register
register_data_out_block_0(5) <= x"00000001"; --Firmware version number register
register_data_out_block_0(6) <= x"abcd0123"; --Firmware version register
register_data_out_block_0(7) <= x"00000001"; --Firmware version register

--Git Commit/Tag/Compile Date
register_data_out_block_0(8) <= x"0123abcd"; --Firmware Git info (0-3)
register_data_out_block_0(9) <= x"00000001"; --Firmware Git info (1)
register_data_out_block_0(10) <= x"00000001"; --Firmware Git info (2)
register_data_out_block_0(11) <= x"00000001"; --Firmware Git info (3)

--Simple loopback register for testing
register_data_out_block_0(12) <= register_data_in_block_0(12);

--Hardware registers (HW_ID_xDM, etc)
--register_data_out_block_0(13) <= x"0000000" & "000" & hardware_id;

--unspecified
set_reg_block_0_gen : for i in 14 to nregisters_block_0-1 generate
register_data_out_block_0(i) <= register_data_in_block_0(i);
end generate set_reg_block_0_gen;

register_data_out_block_0(13) <= x"000000" & rtc_i2c_data_ie & rtc_i2c_data_out & rtc_i2c_clock_ie & rtc_i2c_clock_out & rtc_1pps_in & rtc_32khz_in & rtc_i2c_data_in & rtc_i2c_clock_in;

rtc_i2c_clock_ie <= not register_data_in_block_0(14)(0);
rtc_i2c_clock_out <= register_data_in_block_0(14)(1);
rtc_i2c_data_ie <= not register_data_in_block_0(14)(4);
rtc_i2c_data_out <= register_data_in_block_0(14)(5);
</pre>

== Block 0 CDM ==

* 0x0000+4*i - CDM AXI clock registers:

<pre>
USR_ACCESS_7series_inst: USR_ACCESSE2
port map(-- CFGCLK => CFGCLK,
DATA => register_data_out_block_0(0)); -- where timestamp is contained
-- DATAVALID => DATAVALID);

--static values
register_data_out_block_0(1) <= x"204D4443"; --CDM<space>
register_data_out_block_0(2) <= x"00000000"; --blank
register_data_out_block_0(3) <= x"00000000"; --blank

--Firmware Version
register_data_out_block_0(4) <= x"abcd0123"; --Firmware version name register
register_data_out_block_0(5) <= x"00000001"; --Firmware version number register
register_data_out_block_0(6) <= x"abcd0123"; --Firmware version name register
register_data_out_block_0(7) <= x"00000001"; --Firmware version number register

--Git Commit/Tag/Compile Date
register_data_out_block_0(8) <= x"0123abcd"; --Firmware Git info (0-3)
register_data_out_block_0(9) <= x"00000001"; --Firmware Git info (1)
register_data_out_block_0(10) <= x"00000001"; --Firmware Git info (2)
register_data_out_block_0(11) <= x"00000001"; --Firmware Git info (3)

--Simple loopback register for testing
register_data_out_block_0(12) <= register_data_in_block_0(12);

--Hardware registers (HW_ID_xDM, etc)
--register_data_out_block_0(13) <= x"0000000" & "000" & hardware_id;

--unspecified
set_reg_block_0_gen : for i in 14 to nregisters_block_0-1 generate
register_data_out_block_0(i) <= register_data_in_block_0(i);
end generate set_reg_block_0_gen;

rx_clk_sel <= register_data_in_block_0(12)(0); -- rx_clk select 0:clk_cc_out0, 1:rx_clk_mgt

rtc_i2c_clock_ie <= not register_data_in_block_0(14)(0);
rtc_i2c_clock_out <= register_data_in_block_0(14)(1);
rtc_i2c_data_ie <= not register_data_in_block_0(14)(4);
rtc_i2c_data_out <= register_data_in_block_0(14)(5);

register_data_out_block_0(13) <= x"000000" & rtc_i2c_data_ie & rtc_i2c_data_out & rtc_i2c_clock_ie & rtc_i2c_clock_out & rtc_1pps_in & rtc_32khz_in & rtc_i2c_data_in & rtc_i2c_clock_in;
</pre>

== Block 1 GDM ==

* 0x0800+4*i - GDM registers:

<pre>
clk_config_pro : process (read_write_clk)
begin
if rising_edge (read_write_clk) then
CLK_IN_SEL_LS <= clk_config_vec(1 downto 0);
CLK_EXT_SEL_LS <= clk_config_vec(2);
CLK_RSTn_LS <= not clk_config_vec(3);
clk_config_vec(4) <= CLK_LOSXTn_LS;
clk_config_vec(5) <= CLK_LOLn_LS;
clk_config_vec(6) <= CLK_INTn_LS;
clk_config_vec(7) <= '1';

clk_config_vec(3 downto 0) <= register_data_in_block_1(0)(3 downto 0);
register_data_out_block_1(0)( 7 downto 0) <= clk_config_vec(7 downto 0);
register_data_out_block_1(0)(31 downto 8) <= (others =>'0');
register_data_out_block_1(1) <= (others =>'0'); --free
end if;
end process clk_config_pro;

register_data_out_block_1(2) <= register_data_in_block_1(2);
register_data_out_block_1(3) <= register_data_in_block_1(3);
register_data_out_block_1(4)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(5)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(6)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(7)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(8) <= register_data_in_block_1(8);
register_data_out_block_1(9)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(10) <= (others => '0');
register_data_out_block_1(11) <= (others => '0');

frequency_measure_clks <= read_write_clk & clk_50MHz & mgt_data_clk & mgt_ref_clk_raw; -- & mgt_data_clk & mgt_ref_clk_raw;
register_data_out_block_1(12) <= register_data_out_block_1(14); --MGT rx and tx reference clocks are the same
register_data_out_block_1(13) <= register_data_out_block_1(15); --MGT rx and tx data clocks are the same
frequency_measure_inst : frequency_measure
generic map(register_clk_frequency => 100000000,
max_frequency => 300000000,
nfrequency_measurments => nfrequency_measure_clks)
port map(i_register_clk => read_write_clk,
i_clks => frequency_measure_clks,
i_clk_locks => (others =>'1'),
o_frequencies => register_data_out_block_1(14+nfrequency_measure_clks-1 downto 14)); --see register_data_out_block_1(12,13) above

zero_spare_link_registers : for i in 14+nfrequency_measure_clks to nregisters_block_1-1 generate
register_data_out_block_1(i) <= (others => '0');
end generate zero_spare_link_registers;

gdm_link_interface_inst : gdm_link_interface
generic map(enable_rxslide => '1',
nquads => nquads,
nlinks => nlinks)
port map(i_mgt_ref_clk_p => mgt_b_ref_clk_p,
i_mgt_ref_clk_n => mgt_b_ref_clk_n,
i_mgt_rx_n => mgt_b_rx_n,
i_mgt_rx_p => mgt_b_rx_p,
o_mgt_tx_n => mgt_b_tx_n,
o_mgt_tx_p => mgt_b_tx_p,
--raw, pre-pll, mgt clock for frequency measurement
o_mgt_ref_clk_raw => mgt_ref_clk_raw,
--free running clock and reset
i_mgt_free_running_clk => clk_50MHz,
i_mgt_rst => register_data_in_block_1(2)(0),
i_rx_slide_trigger => register_data_in_block_1(3)(nlinks-1 downto 0),
--data interface for the links
o_data_clk => mgt_data_clk,
o_mgt_data_link_array => mgt_link_data_to_processing,
i_mgt_data_link_array => mgt_link_data_from_processing,
--status ports
o_link_power_good => register_data_out_block_1(4)(nlinks-1 downto 0),
o_tx_link_up => register_data_out_block_1(5)(nlinks-1 downto 0),
o_rx_link_up => register_data_out_block_1(6)(nlinks-1 downto 0),
o_rx_error => register_data_out_block_1(7)(nlinks-1 downto 0),
i_link_down_latched_rst => register_data_in_block_1(8)(0),
o_link_down_latched => register_data_out_block_1(9)(nlinks-1 downto 0),
--debug
i_debug_clk => read_write_clk,
o_debug => mgt_debug_data);
</pre>

<pre>
#### Register Block 1 (CDM)
Clock Chip and MGT Link, Control and Status Registers
| Register | Axi Address | Description | note
|:-|:-|:-|:-|
| 0 | 0x80011000 | Clock Chip Status and Control | CLK_IN_SEL_LS(1-0)(read-only), CLK_EXT_SEL_LS(2), CLK_RSTn_LS(3), CLK_LOSXTn_LS(4), CLK_LOLn_LS(5), CLK_I\
NTn_LS(6)
| 1 | 0x80011004 | free for use | |
| 2 | 0x80011008 | MGT Link Reset | mgt_rst(0), Reset for RX Error Latched(1)|
| 3 | 0x8001100c | MGT rx slide control | not needed or used |
| 4 | 0x80011010 | link_up_and_running(10) tx_link_up_and_running(9), tx_sending_data(8), tx_link_up(7), rx_link_up_and_running(6), rx_error(5), rx_receivi\
ng_data(4), rx_link_up(3), link_power_good(2), sfp_rx_los(1), sfp_mod_absent(0) | Should be 0x7DC |
| 5 | 0x80011014 | reserved debug register | |
| 6-11 | ... | free for use | |
| 12 | 0x80011030 | mgt_rx_ref_clk_raw | ~0x7735940 (125 MHz) |
| 13 | 0x80011034 | rx_clk | ~0x7735940 (125 MHz) |
| 14 | 0x80011038 | mgt_tx_ref_clk_raw | ~0x7735940 (125 MHz) |
| 15 | 0x8001103c | tx_clk | ~0x7735940 (125 MHz) |
| 16 | 0x80011040 | clk_50MHz | 0x2faf080 (50 MHz) |
| 17 | 0x80011044 | Register_Block1_clk | 0x5f5e100 (100 MHz) |
| 18-20 | ... | free for use | |
| 21 | 0x80011054 | free for use | |
</pre>

=== Register 0x1000 ===

<pre>
printf("0x1000 SFP c.c. status: 0x%08x\n", reg1000);
printf(" CLK_IN_SEL_LS 0x%x\n", (reg1000>>0) & 3);
printf(" CLK_EXT_SEL_LS %d\n", (reg1000>>2) & 1);
printf(" CLK_CLK_RSTn_LS %d\n", (reg1000>>3) & 1);
printf(" CLK_LOSXTn_LS %d\n", (reg1000>>4) & 1);
printf(" CLK_LOLn_LS %d\n", (reg1000>>5) & 1);
printf(" CLK_INTn_LS %d\n", (reg1000>>6) & 1);
printf("0x1008 SFP link reset: 0x%08x\n", dev->dsdm_read32(0x1008));
</pre>

=== Register 0x1010 ===

<pre>
printf("0x1010 SFP link status: 0x%08x\n", reg1010);
printf(" sfp_mod_absent_N %d\n", (reg1010>>0) & 1);
printf(" sfp_rx_los_N %d\n", (reg1010>>1) & 1);
printf(" link_power_good %d\n", (reg1010>>2) & 1);
printf(" rx_link_up %d\n", (reg1010>>3) & 1);
printf(" rx_receiving_data %d\n", (reg1010>>4) & 1);
printf(" rx_error %d\n", (reg1010>>5) & 1);
printf(" rx_lnk_up_and_running %d\n", (reg1010>>6) & 1);
printf(" tx_link_up %d\n", (reg1010>>7) & 1);
printf(" tx_sending_data %d\n", (reg1010>>8) & 1);
printf(" tx_link_up_and_running %d\n", (reg1010>>9) & 1);
printf(" link_up_and_running %d\n", (reg1010>>10) & 1);

</pre>

=== Register 0x1014 ===

<pre>
printf("0x1014 SFP link data: 0x%08x\n", reg1014);
printf(" rx_data 0x%04x\n", (reg1014>>0) & 0xFFFF);
printf(" k28p1_k28p5 %d\n", (reg1014>>16) & 1);
printf(" rx_ctrl0 0x%01x\n", (reg1014>>17) & 3);
printf(" rx_ctrl1 0x%01x\n", (reg1014>>19) & 3);
printf(" rx_ctrl3 0x%01x\n", (reg1014>>21) & 1);
printf(" tx_state 0x%01x\n", (reg1014>>22) & 3);
printf(" rx_state 0x%01x\n", (reg1014>>24) & 0xF);
printf(" rx_receiving_data %d\n", (reg1014>>28) & 1);
printf(" rx_link_up %d\n", (reg1014>>29) & 1);
printf(" rx_error %d\n", (reg1014>>30) & 1);
printf(" rx_link_rst %d\n", (reg1014>>31) & 1);
</pre>

* see cdm_link_establishment.vhd
* tx_state:
<pre>
0 = tx_idle
1 = waiting_for_rx_link
2 = sending_k28p1_k28p5
3 = sending_data
</pre>

* rx_state:
<pre>
0 = rx_idle
1 = waiting_for_ks
2 = rx_retry
3 = rx_up_switch_tx_clk_source
4 = waiting_for_data
5 = receiving_data
6 = rx_receive_error
</pre>

== Block 2 GDM ==

* 0x1000+4*i - GDM registers:

<pre>
register_data_out_block_2(0) <= register_data_in_block_2(0);
GDM_link_data_processing_inst : GDM_link_data_processing
generic map(mgt_data_frequency => 125000000)
port map(i_register_clk => read_write_clk,
i_rst => register_data_in_block_2(0)(0),
i_data_mode => register_data_in_block_2(0)( 9 downto 8),
i_fixed_pattern_data => register_data_in_block_2(0)(31 downto 16),
o_error_time => register_data_out_block_2(1),
o_error_count => register_data_out_block_2(13 downto 2),
--link data interface
i_data_clk => mgt_data_clk,
i_mgt_data_link_array => mgt_link_data_to_processing,
o_mgt_data_link_array => mgt_link_data_from_processing,
--data interface
o_data_clk => data_clk,
o_tx_data_ready => tx_data_ready,
i_tx_data_is_k => tx_data_is_k,
i_tx_data => tx_data,
o_rx_data_error => rx_data_error,
o_rx_data_is_k => rx_data_is_k,
o_rx_data => rx_data);
</pre>

== Block 2 CDM ==

* 0x1000+4*i - CDM registers:
* reg 0
<pre>
read bits
7..0 - clk_config_vec
1,0 - CLK_IN_SEL_LS
2 - CLK_EXT_SEL_LS
3 - disconnected CLK_RSTn_LS
4 - CLK_LOSXTn_LS
5 - CLK_LOLn_LS
6 - CLK_INTn_LS
7 - tx_clock_status
31..8 - 0
</pre>
<pre>
write bits
3..2 - clk_config_vec[3..2]
</pre>
* reg 1 - 0
* reg 2 - RW
<pre>
bit
0 - cdm_link_interface i_mgt_rst
1 - cdm_link_interface i_link_down_latched_rst
</pre>
* reg 3 - RW
<pre>
bit
0 - cdm_link_interface i_rx_slide_trigger
</pre>
* reg 4 - RO
<pre>
bit
0 - sfp_mod_present
1 - sfp_rx_los
2 - link_power_good
3 - rx_link_up
4 - rx_receiving_data
5 - rx_error
6 - rx_link_up_and_running
7 - tx_link_up
8 - tx_sending_data
9 - tx_link_up_and_running
10 - link_up_and_running
</pre>
* reg 5 - RO - CDM SFP debug_data
* reg 6 - 0
* reg 7 - 0
* reg 8 - 0
* reg 9 - 0
* reg 10 - 0
* reg 11 - 0
* reg 12 - frequency AXI clock
* reg 13 - freq clk_50MHz
* reg 14 - freq clk_100MHz
* reg 15 - freq clk_200MHz
* reg 16 - freq clk_xo_125 - DS-DM 125 MHz oscillator
* reg 17 - freq clk_cc_out0 - C.C. 125 MHz output
* reg 18 - CDM SFP mgt_rx_ref_clk_raw
* reg 19 - CDM SFP mgt_tx_ref_clk_raw
* reg 20 - rx_clk_mgt, 125 MHz CDM SFP RX recovered clock
* reg 21 - rx_clk, 125 MHz, switched between clk_xo_125 and rx_clk_mgt
* reg 22 - tx_clk, 125 MHz
* reg 23 - spare

== Block 3 GDM ==

0x3000+4*i : DS20k registers

== Block 4 CDM ==

* 0x2800+4*i - CDM SFP data link registers:
* 0 - R/W
** bit 0 - 0x1 - reset
** bits 7..1 - not used
** bits 9..8 - data_mode: 00: data from ds20k block, 01: tx_data_cnt, 10: tx_data_prbs, 11: tx_fixed_pattern_data
** bits 15..10 - not used
** bits 31..16 - fixed pattern data
* 1 - RO - o_error_time
* 2 - RO - o_error_count
* 3, etc - not used

== Block 6 CDM ==

* 0x3000+4*i : DS20k registers

== Block 7 CDM ==

VX_TX and VX_RX delays, i=0..11
* 0x3800+4*i - VX_TX delays
* 0x3830+4*i - VX_RX delays

VX_TX delay data (2 bits):
* 4 values: 0,1,2,3 corresponding to the 4 phases of the 125 MHz clock rx_clk and rx_clk shifted by 90 deg

VX_RX delay data (32 bits):
* 0x8000'0000: reset IDELAYE3, follow by write of 0
* 0x0001'xxxx: load the IDELAYE3 9-bit tap counter, follow by write of 0x0000'xxxx
* 16 low bits have:
* bits 0..8 are the 9 bits of IDELAY3 tap counter (~1.1ns for all 511 delays)
* bit 9 is the 500 MHz DDR phase (1 ns)
* bits 10..15 are 16 bits of 500 MHz delay (2 ns step)
* numerically increasing 16-bit delay values are carefully arranged to produce linearly increasing delay (there is some overlap between IDELAYE3 taps above ~tap 500 (or so, IDELAYE3 is uncalibrated) and flip of 500 MHz DDR phase

== DS20k registers ==

=== DS20k block register map ===

* busybox devmem 0x80013000 32

Note: R=readable, W=writable, L=latched by CMD_LATCH, B=reset on begin of run, Z=reset by CMD_LATCH

<pre>
reg | version | xDM | xx | description
0 | 0x20230731 | ALL | RO | ds20k version
0 | 0x20240118 | ALL | RW | ds20k version and command
1 | 0x20230731 | ALL | RW | scratch read/write register
2 | 0x20230731 | ALL | RW | configure inputs and outputs
3 | 0x20230731 | ALL | RW | FP_LED mux
4 | 0x20230731 | ALL | RW | EXT_OUT mux
5 | 0x20230731 | ALL | RO | VX_RX state
6 | 0x20230731 | ALL | RO | VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state
7 | 0x20230731 | ALL | RW | LED_OUT, EXT_OUT, VX_TX outputs
8 | 0x20230731 | ALL | RW | VX_TX mux and config
9 | 0x20230731 | ALL | RW | trg and tsm source
10 | 0x20231013 | ALL | RO | status register
11 | 0x20230731 | ALL | ROLB | trg counter
12 | 0x20230731 | ALL | ROLB | tsm counter
13 | 0x20240814 | ALL | RO | GPS 1pps period, 125 MHz
14 | 0x20240814 | ALL | RO | Ru clock 1pps period, 125 MHz
15 | 0x20230811 | CDM | RO | SFP RX status
16 | 0x20230811 | CDM | RW | SFP TX control
17 | 0x20230811 | GDM | RO | QSFP RX data 0, 1
18 | |. |. | 2, 3
19 | |. |. | 4, 5
20 | |. |. | 6, 7
21 | | |. | 8, 9
22 | |. |. | 10, 11
23 | 0x20230811 | GDM | RW | QSFP TX control
24 | 0x20231013 | ALL | RW | trigger pulser period
25 | 0x20231013 | ALL | RW | trigger pulser burst control
26 | 0x20231013 | ALL | RW | tsm pulser period
27 | 0x20231204 | ALL | RW | data fifo CPU to FPGA
28 | 0x20231204 | ALL | RW | packet loopback control
28 | 0x20240118 | ALL | RW | data fifo FPGA to CPU
29 | 0x20231208 | GDM | RW | bitmap of active qsfp ports
29 | 0x20240118 | ALL | RW | packet loopback control
29 | 0x20240510 | ALL | RW | packet routing
30 | 0x20231208 | GDM | RO | qsfp link status ports 0..7
31 | 0x20231208 | GDM | RO | qsfp link status ports 8..11
32 | 0x20240118 | GDM | RW | enabled QSFP ports
33 | 0x20240118 | CDM | RW | enabled VX ports
34,35 | 0x20240118 | ALL | ROB | time stamp 64 bits
36,37 | 0x20240118 | ALL | RO | old time stamp
38,39,40 | 0x20240118 | CDM | ROLZ | vx_bsy_counter_latched | VX busy counters
41,42,43 | 0x20240118 | GDM | ROLZ | QSFP busy counters
44 | 0x20240118 | CDM | ROLZ | cdm_bsy_up_counter and cdm_bsy_pulse_counter
45 | 0x20240118 | CDM | ROLZ | cdm_veto_pulse_counter and cdm_veto_up_counter
46 | 0x20240118 | GDM | ROLZ | gdm_bsy_pulse_counter and gdm_bsy_up_counter
47 | 0x20240118 | GDM | ROLZ | gdm_bsy_refresh_counter
48 | 0x20240118 | GDM | ROLZ | gdm_veto_up_counter and gdm_veto_pulse_counter
49,50,51 | 0x20240424 | CDM | RO | VX RX serial data monitor, 8 bits per VX port
52,53 | 0x20240424 | CDM | RO | VX serial link status, 4 bits per VX port
54 | 0x20240424 | CDM | RO | VX TX serial data monitor
55 | 0x20240430 | ALL | RO | QSFP, SFP and VX link loss counters
56 | 0x20240510 | ALL | RO | sfp_rx_packet_counter
57 | same | ALL | RO | sfp_tx_packet_counter
58 | same | ALL | RO | qsfp_rx_packet_counter[0]
59 | same | ALL | RO | qsfp_tx_packet_counter
60 | same | ALL | RO | vx_rx_packet_counter[0]
61 | same | ALL | RO | vx_tx_packet_counter
62 | 0x20240719 | CDM | RO | cdm_hitmap_period, ports 0, 1
63 | same | CDM | RO | ports 2, 3
64 | same | CDM | RO | ports 4, 5
65 | same | CDM | RO | ports 6, 7
66 | same | CDM | RO | ports 8, 9
67 | same | CDM | RO | ports 10, 11
68 | 0x20240814 | ALL | RW | GPS control and status
69 | 0x20241104 | CDM | ROLB | vx_tx_trg_packet_counter | counter of TRG packets CDM->VX
70 | 0x20241104 | CDM | ROLB | vx_tx_tsm_packet_counter | counter of TSM packets CDM->VX
71 | 0x20241104 | ALL | RO | packet error bits
72,73 | 0x20241104 | CDM | ROL | vx_tx_trg_data_latched[63:0] trigger packet data
74,75 | 0x20241104 | CDM | ROL | vx_tx_tsm_data_latched[63:0] tsm packet data (truncated to 64 bits)
76 | 0x20241104 | CDM | ROLB | cdm_hitmap_trigger_counter_latched, counter of hitmap triggers generated in the CDM
77,78,79 | 0x20241104 | CDM | ROL | vx_rx_hitmap_data_latched | VX_RX 80 bits of HITMAP packet from VX
80 | 0x20241209 | CDM | RO | vx_lvds_rx_link_status | VX LVDS link status
81 | 0x20241209 | CDM | RO | vx_lvds_rx_trg_data | VX LVDS TRG packet first 32 bits
81 | 0x20251010 | CDM | - | not used | not used
82 | 0x20241209 | CDM | RO | vx_lvds_rx_tsm_data | VX LVDS TSM packet first 32 bits
82 | 0x20251010 | CDM | - | not used | not used
83 | 0x20241209 | CDM | ROL | vx_lvds_rx_decoded_packet_counter_latched | VX LVDS decoded packet counter
84 | 0x20241209 | CDM | ROL | vx_lvds_rx_packet_error_counter_latched | VX LVDS packet error counter
85 | 0x20250417 | CDM | ROLB | vx_tx_trg_counter | VX LVDS TRG counter (VX LVDS pair 12)
86 | 0x20250417 | CDM | ROLB | vx_tx_tsm_counter | VX LVDS TSM counter (VX LVDS pair 13)
87 | 0x20250417 | CDM | ROLB | vx_tx_veto_counter | VX LVDS Veto counter (VX LVDS pair 13)
88 | 0x20250422 | GDM | ROLB | trg_in_lost_counter | counter of TRG killed by gdm_busy
89 | 0x20250422 | ALL | RO | fifo_from_fpga_lost_packet_counter | counter of packets lost due to FIFO full
90 | 0x20250428 | CDM | ROL | vx_bsy_status_latched | status of VX busy, CDM busy, CDM veto
91 | 0x20250428 | CDM | ROL | vx_hitmap_status_latched | status of VX hitmap
92 | 0x20250428 | GDM | ROL | cdm_bdy_status_latched | status of CDM busy, GDM busy, GDM veto
93 | 0x20250508 | GDM | ROL | gdm_hitmap_status_latched | status of GDM hitmap
94 | 0x20250508 | GDM | ROLB | gdm_hitmap_trigger_counter_latched | counter of hitmap triggers generated in the GDM
95 | 0x20250512 | GDM | ROLB | gdm_trig48_counter_latched | counter of TRIG48 trigger packets
96 | 0x20250512 | GDM | ROLB | gdm_trig48_lost_counter_latched | counter of TRIG48 triggers lost to busy
97 | 0x20250512 | CDM | ROLB | vx_tx_trig48_packet_counter_latched | counter of TRIG48 packets from CDM to VX
98, 99 | 0x20250512 | GDM | RW | trig48_ctrl48, trig48_ctrl16 | GDM TRIG48 test registers
100,101 | 0x20251008 | CDM | ROL | vx_lvds_rx_trig48_{data,bitmap48}_latched | VX LVDS TRIG48 packet first 32 bits, bitmap48 first 32 bits
102 | 0x20251008 | ALL | RO | cmd_counter | counter of reg0 commands
</pre>

=== Register 0 0x80013000 ds20k version ===

on read: ds20k version 0xYYYYMMDD

on write:
* 0 - noop - as of version 0x20240118, write a zero after writing a command
* 1 - cmd_reset - reset logic to good state
* 2 - cmd_arm_ts - arm timestamp reset
* 3 - cmd_trg - issue a trigger
* 4 - cmd_tsm - issue a tsm
* 5 - cmd_vx_rx_reset - reset the VX receive path
* 6 - cmd_vx_tx_reset - reset the VX transmit path
* 7 - cmd_gdm_trig48 - generate a TRIG48 trigger and data packet (GDM)
* 8 - cmd_trg_pulser_reset - reset the trigger pulser
* 9 - cmd_tsm_pulser_reset - reset the tsm pulser
* 10 - cmd_bor_start - start begin-of-run trigger sequence
* 11 - cmd_bor_clear - after run has started, clear begin-of-run status bits
* 12 - cmd_latch - latch counters & etc into AXI registers for coherent readout
* 13 - cmd_cdm_hitmap_trigger - generate a hitmap trigger and data packet (CDM)

=== Register 1 0x80013004 scratch ===

scratch read-write register

=== Register 2 0x80013008 input and output config ===

<pre>
bit | version | fpga name | description
0 | ALL | lemo_enable | enable LEMO input 1
1 | | | 2
2 | | | 3
3 | | | 4
4 | ALL | lemo_invert | invert LEMO input 1
5 | | | 2
6 | | | 3
7 | | | 4
9 | ALL | ext_out_disable | disable LEMO output 1
10 | | | 2
11 | ALL | ext_out_invert | invert LEMO output 1
12 | | | 2
</pre>

=== Register 3 0x8001300C FP_LED control ===

<pre>
wire [15:0] led_out_mux_sel = register_data_in[3][15:0];
wire [3:0] led_out_invert = register_data_in[3][19:16];
</pre>

<pre>
led_out_mux_sel is 4 groups (one per LED) of 4 bits (choice 0..15):
mux | version | fpga name | description
0 | ALL | | power on default
1 | ALL | led_out_reg | register 7 bits
2 | 0x20231013 | pll_locked | clock chip PLL is locked
3 | 0x20231013 | sfp_link_status | SFP link is good
3 | 0x20240118 | sfp_link_status or qsfp_tx_link_rx_status | SFP/QSFP link is good
4 | ALL | lemo_in_sync[1] | LEMO input 1
5 | ALL | lemo_in_sync[2] | LEMO input 2
6 | ALL | lemo_in_sync[3] | LEMO input 3
7 | ALL | lemo_in_sync[4] | LEMO input 4
8 | ALL | ext_out[1] | LEMO output 1
9 | ALL | ext_out[2] | LEMO output 2
A | 0x20231013 | trg_in | trigger
B | 0x20231013 | tsm_in | time slice marker
C | 0x20240118 | gdm_bsy | GDM busy: OR of all CDM busy
D | 0x20240118 | cdm_bsy | CDM busy: OR of all VX busy
E | 0x20240118 | cdm_veto | GDM busy -> GDM veto -> CDM veto -> VX trigger veto
F | ALL | | fixed logic level 1
</pre>

=== Register 4 0x80013010 LEMO OUT control ===

<pre>
wire [7:0] ext_out_mux_sel = register_data_in[4][7:0];
</pre>

<pre>
ext_out_mux_sel is 2 groups (one per LEMO) of 4 bits (choice 0..15):

mux | version | fpga name | description
0 | ALL | | power on default
1 | ALL | ext_out_reg | register 7
2 | ALL | trg_pulser | pulser trigger
3 | 0x20240724 | vx1_tx_out[2] | vx1 serial data out
4 | 0x20240724 | vx1_rx[1] | vx1 serial data in
5 | 0x20240724 | vx_rx_iob[0] | vx1 serial data in captured by IOB register
6 | ALL | lemo_in_async[1]| test synchronizer
7 | ALL | lemo_in_sync[1] | test synchronizer
8 | 0x20240724 | trg_in_pulse | trigger signal
9 | 0x20240724 | tsm_in_pulse | time slice marker signal
A | 0x20240118 | cdm_bsy | CDM busy from VX
B | 0x20240118 | gdm_bsy | GDM busy from CDM
C | 0x20240118 | cdm_veto | veto from GDM to CDM to VX
D | 0x20240118 | vx1_rx[1] | serial data VX to CDM
E | not used (sink)
F | ALL | 1 | fixed logic level 1
</pre>

=== Register 5 0x80013014 VX_RX status ===

<pre>
assign register_data_out[5] = {
vx8_rx[3], vx8_rx[2], vx8_rx[1], vx8_rx[0],
vx7_rx[3], vx7_rx[2], vx7_rx[1], vx7_rx[0],
vx6_rx[3], vx6_rx[2], vx6_rx[1], vx6_rx[0],
vx5_rx[3], vx5_rx[2], vx5_rx[1], vx5_rx[0],
vx4_rx[3], vx4_rx[2], vx4_rx[1], vx4_rx[0],
vx3_rx[3], vx3_rx[2], vx3_rx[1], vx3_rx[0],
vx2_rx[3], vx2_rx[2], vx2_rx[1], vx2_rx[0],
vx1_rx[3], vx1_rx[2], vx1_rx[1], vx1_rx[0]
};
</pre>

=== Register 6 0x80013018 VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state ===

<pre>
assign register_data_out[6] = {
vx2_tx_out[2], vx2_tx_out[1], vx2_tx_out[0], vx1_tx_out[2],
vx1_tx_out[1], vx1_tx_out[0], ext_out[2], ext_out[1],
fp_led_out[3], fp_led_out[2], fp_led_out[1], fp_led_out[0],
ext_in_lv[4], ext_in_lv[3], ext_in_lv[2], ext_in_lv[1],
vx12_rx[3], vx12_rx[2], vx12_rx[1], vx12_rx[0],
vx11_rx[3], vx11_rx[2], vx11_rx[1], vx11_rx[0],
vx10_rx[3], vx10_rx[2], vx10_rx[1], vx10_rx[0],
vx9_rx[3], vx9_rx[2], vx9_rx[1], vx9_rx[0]
};
</pre>

<pre>
bit | version | fpga name | description
0 | ALL | vx9_rx | VX_RX
1 | | |
2 | | |
3 | | |
4 | ALL | vx10_rx | VX_RX
5 | | |
6 | | |
7 | | |
8 | ALL | vx11_rx | VX_RX
9 | | |
10 | | |
11 | | |
12 | ALL | vx12_rx | VX_RX
13 | | |
14 | | |
15 | | |
16 | ALL | ext_in_lv | LEMO inputs
17 | | |
18 | | |
19 | | |
20 | ALL | FP_LED | FP_LEDs
21 | | |
22 | | |
23 | | |
24 | ALL | ext_out[1] | LEMO outputs
25 | | ext_out[2] |
26 | ALL | vx1_tx | VX1_TX
27 | | |
28 | | |
29 | ALL | vx2_tx | VX2_TX
30 | | |
31 | | |
</pre>

=== Register 7 0x8001301C LED_OUT, EXT_OUT, VX_TX outputs ===

<pre>
wire [3:0] led_out_reg = register_data_in[7][3:0];
wire [2:1] ext_out_reg = register_data_in[7][5:4];
// register_data_in[7][6];
// register_data_in[7][7];
wire [7:0] vx_tx_out_reg = register_data_in[7][15:8];
</pre>

<pre>
bit | version | fpga name | description
0 | ALL | led_out_reg | FP_LED 1
1 | | | 2
2 | | | 3
3 | | | 4
4 | ALL | ext_out_reg | LEMO OUT 1
5 | | | 2
6 | - | |
7 | - | |
8 | ALL | vx_tx_out_reg | VX1_TX 0
9 | | | 1
10 | | | 2
11 | | | -
12 | ALL | | VX2_TX 0
13 | | | 1
14 | | | 2
15 | | | -
16 | - | |
17 | | |
18 | | |
19 | | |
20 | | |
21 | | |
22 | | |
23 | | |
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 8 0x80013020 VX_TX config ===

<pre>
wire [3:0] vx_tx_mux_sel = register_data_in[8][3:0];
wire vx_tx_from_sfp = register_data_in[8][30];
wire vx_tx_loopback = register_data_in[8][31];
</pre>

<pre>
vx_tx_mux_sel is 4 bits (choice 0..15):
0 - power on default, control by vx_tx_out_reg
1 - GDM
2 - CDM
3 - pulser loopback test
4 - pulser loopback test
5 - 62.5 MHz output
6 - 125 MHz output
7 - trg, tsm, serial
8 - trg, tsm, lvds serial rx to serial tx loopback
9 - GPS box control (ds20k rev 0x20240814)
10
11
12
13
14
15 - production config: trg, veto, serial
</pre>

=== Register 9 trg and tsm source ===

from version 0x20240724

<pre>
wire [15:0] trg_src_mask = register_data_in[9][15:0];
wire [31:16] tsm_src_mask = register_data_in[9][31:16];
</pre>

<pre>
wire [15:0] trg_src_bits =
{
1'b0, // 15
1'b0, // 14
vx_tx_tsm_done, // 13
vx_tx_trg_done, // 12

1'b0, // gdm_hitmap_trigger, // 11
cdm_hitmap_trigger, // 10
sfp_rx_tsm, // 9
sfp_rx_trg, // 8

sfp_rx_data[1], // 7
sfp_rx_data[0], // 6
tsm_pulser, // 5
trg_pulser, // 4

lemo_in_sync[4], // 3
lemo_in_sync[3], // 2
lemo_in_sync[2], // 1
lemo_in_sync[1] // 0
};

wire [15:0] trg_bits = trg_src_bits & trg_src_mask;
wire [15:0] tsm_bits = trg_src_bits & tsm_src_mask;
</pre>

before that:

<pre>
wire [7:0] trg_src_mask = register_data_in[9][7:0];
wire [7:0] tsm_src_mask = register_data_in[9][15:8];
wire trg_pulser_enable = register_data_in[9][16];
wire tsm_pulser_enable = register_data_in[9][17];
wire trg_software = register_data_in[9][18];
wire tsm_software = register_data_in[9][19];
// bits 20:31 not used
</pre>

<pre>
wire [7:0] xxx_src_bits =
{
sfp_rx_data[1],
sfp_rx_data[0],
tsm_pulser & tsm_pulser_enable,
trg_pulser & trg_pulser_enable,
lemo_in_sync[4],
lemo_in_sync[3],
lemo_in_sync[2],
lemo_in_sync[1]
};
</pre>

<pre>
trg_src_mask and tsm_src_mask bits:
0 - LEMO IN 1
1 - LEMO IN 2
2 - LEMO IN 3
3 - LEMO IN 4
4 - trg_pulser
5 - tsm_pulser
6 - sfp_rx_data[0] // to become sfp_trg_in, selected from sfp_rx_data[0], cdm_rx trg_out and gdm trigger packet
7 - sfp_rx_data[1] // to become sfp_tsm_in, selected from sfp_rx_data[1], cdm_rx tsm_out and gdm tsm packet
</pre>

=== Register 10 0x80013028 status register ===

<pre>
bit | version | fpga name | description
0 | 0x20231013 | pll_locked | clock chip PLL is locked
1 | 0x20240118 | ts_reset_armed | timestamp reset is armed
2 | 0x20240118 | qsfp_tx_link_rx_status | QSFP link status is good for all enabled ports
3 | 0x20240424 | vx_tx_link_rx_status | VX link status is good for all enabled ports
4 | 0x20240118 | cdm_bsy | VX busy grand-or
5 | 0x20240118 | gdm_bsy | QSFP busy grand-or
6 | 0x20240118 | gdm_veto | gdm_veto = gdm_busy
7 | 0x20240118 | cdm_veto | CDM veto from GDM to VX
8 | 0x20240725 | bor_started | begin-of-run sequence started
9 | 0x20240725 | bor_finished | begin-of-run sequence fininished, see commands 10 and 11
10 | | |
11 | | |
12 | | |
13 | | |
14 | | |
15 | | |
16 | | |
17 | | |
18 | | |
19 | | |
20 | | |
21 | | |
22 | | |
23 | | |
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 11 0x8001302C trg_counter ===

trigger counter

=== Register 12 0x80013030 tsm_counter ===

time slice marker counter

=== Register 13 0x80013034 GPS 1pps period ===

GPS 1pps period in 8 ns clocks

=== Register 14 0x80013038 Rb clock 1pps period ===

PRS-10 Rb clock 1pps output period in 8 ns clocks

=== Register 15 0x8001303C SFP RX status ===

<pre>
bit | ds20k version | fpga signal name | description
0 | ALL | sfp_rx_data[15:0] | cdm sfp received data
16 | same | sfp_rx_data_is_k[0] |
17 | same | sfp_rx_data_is_k[1] |
18 | | 0 |
19 | 0x20231204 | sfp_rx_sel_lpb | sfp tx->rx loopback
20 | 0x20231013 | sfp_link_status | sfp link connected, exchanging data
21 | same | sfp_link_rx_status | sfp link receiving correct idle pattern from GDM TX
22 | same | sfp_link_error | sfp link receiver error (badk or overflow)
23 | same | sfp_rx_data_error | sfp transceiver state machine is in error state
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 16 SFP TX control ===

<pre>
wire [15:0] sfp_tx_data_reg = register_data_in[16][15:0]; // SFP TX data K-code
wire [1:0] sfp_tx_ctrl_reg = register_data_in[16][17:16]; // SFP TX data K-code
// 18
// 19
// 23:20
// 27:24
wire sfp_rx_sel_lpb = register_data_in[16][28]; // TX->RX serial loopback
wire sfp_tx_sel_lpb = register_data_in[16][29]; // RX->TX serial loopback
wire sfp_tx_sel_trg = register_data_in[16][30]; // 16 individual bits
wire sfp_tx_sel_reg = register_data_in[16][31]; // TX data from register 16 bits 17:0
</pre>

=== Register 17-22 QSFP RX data ===

QSFP RX data links 0..11

=== Register 23 QSFP TX control ===

<pre>
wire [15:0] qsfp_tx_data_reg = register_data_in[23][15:0];
wire [1:0] qsfp_tx_ctrl_reg = register_data_in[23][17:16];

wire qsfp_tx_enable_trg = register_data_in[23][24]; // enable QSFP TX trg_in_pulse k-code
wire qsfp_tx_enable_tsm = register_data_in[23][25]; // enable QSFP TX tsm_in_pulse k-code

//wire qsfp_rx_sel_lpb = register_data_in[23][28]; // TX->RX loopback
wire qsfp_tx_sel_lpb = register_data_in[23][29]; // RX->TX loopback
wire qsfp_tx_sel_trg = register_data_in[23][30]; // 16 individual bits
wire qsfp_tx_sel_reg = register_data_in[23][31]; // data from register
</pre>

=== Register 24 0x80013060 trigger pulser period ===

trigger pulser period in units of 8 ns (125 MHz clock)

=== Register 25 0x80013064 trigger burst pulser ===

<pre>
wire [7:0] conf_burst_count = conf_pulser_burst_ctrl[31:24];
wire [23:0] conf_burst_period = conf_pulser_burst_ctrl[23:0];
</pre>

=== Register 26 0x80013068 tsm pulser period ===

time slice marker period in units of 8 ns (125 MHz clock)

=== Register 27 0x8001306C data write fifo ===

<pre>
wire fifo_reset = register_data_in[27][31];
wire fifo_to_fpga_wr1 = register_data_in[27][27];
wire fifo_to_fpga_wr2 = register_data_in[27][26];
wire [16:0] fifo_to_fpga_din = register_data_in[27][16:0];
</pre>

<pre>
assign register_data_out[27][31:24] = register_data_in[27][31:24]; // echo write bits
assign register_data_out[27][23] = fifo_to_fpga_full;
assign register_data_out[27][22] = fifo_to_fpga_empty;
</pre>

=== Register 28 0x80013070 data read fifo ===

<pre>
wire fifo_reset = register_data_in[28][31];
wire fifo_from_fpga_rd1 = register_data_in[28][25];
wire fifo_from_fpga_rd2 = register_data_in[28][24];
</pre>

<pre>
assign register_data_out[28][31:24] = register_data_in[28][31:24]; // echo write bits
assign register_data_out[28][21] = fifo_from_fpga_full;
assign register_data_out[28][20] = fifo_from_fpga_empty;
assign register_data_out[28][16:0] = fifo_from_fpga_dout;
</pre>

=== Register 29 0x80013074 packet routing ===

Control packet routing and loopbacks:

<pre>
wire [3:0] dn_route_ctrl = register_data_in[29][3:0];
wire [3:0] up_route_ctrl = register_data_in[29][7:4];
wire [3:0] fifo_to_fpga_route_ctrl = register_data_in[29][11:8];
//wire [3:0] spare_route_ctrl = register_data_in[29][15:12];

wire dn_mux_trg_enable = register_data_in[29][16];
wire dn_mux_tsm_enable = register_data_in[29][17];
wire dn_mux_sfp_rx_fifo_enable = register_data_in[29][18];
// 19
// 20..23

wire up_mux_vx_rx_enable = register_data_in[29][24];
// 25
// 26
// 27
wire fifo_from_fpga_hitmap_enable = register_data_in[29][28];
wire fifo_from_fpga_cdm_busy_enable = register_data_in[29][29];
wire fifo_from_fpga_gdm_busy_enable = register_data_in[29][30];
wire fifo_from_fpga_tsm_enable = register_data_in[29][31];
</pre>

fifo_to_fpga output routing:
* 0 - to down packet mux
* 1 - to up packet mux
* 2 - to fifo_from_fpga mux
* 3 - not used

down packet mux inputs:
* fifo_to_fpga_0
* trg_pkt16 enabled by dn_mux_trg_enable
* tsm_pkt16 enabled by dn_mux_tsm_enable
* sfp_rx_fifo_pkt16 enabled by dn_mux_sfp_rx_fifo_enable
* up_pkt16_2 loopback from up packet mux

down packet mux output routing:
* 0 - to vx_tx_pkt16 to VX TX (vx_link_tx and vx_ser_tx)
* 1 - to fifo_from_fpga mux
* 2 - to up packet mux loopback
* 3 - to qsfp_tx_pkt16 to GDM QSFP TX (cdm_link_tx)

up packet mux inputs:
* vx_rx_pkt16 enabled by up_mux_vx_rx_enable data from 12 VX RX links
* fifo_to_fpga_1
* dn_pkt16_2 loopback from down packet mux

up packet mux output routing:
* 0 - to sfp_tx_pkt16 to CDM SFP TX (VX data to GDM)
* 1 - to fifo_from_fpga mux
* 2 - to down packet mux loop loopback
* 3 - not used

internal_mux inputs:
* 0 - cdm_hitmap_pkt16 enabled by fifo_from_fpga_hitmap_enable (reg 29)
* 1 - cdm_busy_pkt16 enabled by fifo_from_fpga_cdm_busy_enable
* 2 - gdm_busy_pkt16 enabled by fifo_from_fpga_gdm_busy_enable
* 3 - gdm_hitmap_pkt16 enabled by fifo_from_fpga_hitmap_enable
* 4 - not used

debug_mux inputs:
* 0 - not used
* 1 - dn_pkt16_1
* 2 - up_pkt16_1
* 3 - qsfp0_rx_pkt16 enabled by fifo_from_fpga_qsfp0_rx_enable (from GDM QSFP link 0, there is no GDM QSFP 12-to-1 mux)
* 4 - not used

fifo_from_fpga mux inputs:
* 0 - fifo_to_fpga_2
* 1 - tsm2_pkt16 enabled by fifo_from_fpga_tsm_enable (reg 29)
* 2 - not used
* 3 - debug_pkt16 from debug_mux
* 4 - internal_pkt16 from internal_mux

=== Register 30 0x80013078 qsfp link status ports 0..7 ===

<pre>
assign register_data_out[30] =
{
qsfp_rx_data_error[7], qsfp_link_error[7], qsfp_link_status[7], qsfp_link_rx_status[7], // 7
qsfp_rx_data_error[6], qsfp_link_error[6], qsfp_link_status[6], qsfp_link_rx_status[6], // 6
qsfp_rx_data_error[5], qsfp_link_error[5], qsfp_link_status[5], qsfp_link_rx_status[5], // 5
qsfp_rx_data_error[4], qsfp_link_error[4], qsfp_link_status[4], qsfp_link_rx_status[4], // 4
qsfp_rx_data_error[3], qsfp_link_error[3], qsfp_link_status[3], qsfp_link_rx_status[3], // 3
qsfp_rx_data_error[2], qsfp_link_error[2], qsfp_link_status[2], qsfp_link_rx_status[2], // 2
qsfp_rx_data_error[1], qsfp_link_error[1], qsfp_link_status[1], qsfp_link_rx_status[1], // 1
qsfp_rx_data_error[0], qsfp_link_error[0], qsfp_link_status[0], qsfp_link_rx_status[0] // 0
};
</pre>

=== Register 31 0x8001307C qsfp link status ports 8..11 ===

<pre>
assign register_data_out[31] =
{
4'b0000,
4'b0000,
4'b0000,
4'b0000,
qsfp_rx_data_error[11], qsfp_link_error[11], qsfp_link_status[11], qsfp_link_rx_status[11], // 11
qsfp_rx_data_error[10], qsfp_link_error[10], qsfp_link_status[10], qsfp_link_rx_status[10], // 10
qsfp_rx_data_error[9], qsfp_link_error[9], qsfp_link_status[9], qsfp_link_rx_status[9], // 9
qsfp_rx_data_error[8], qsfp_link_error[8], qsfp_link_status[8], qsfp_link_rx_status[8] // 8
};
</pre>

=== Register 32 0x80013080 bitmap of enabled qsfp ports ===

<pre>
wire [11:0] qsfp_mask = register_data_in[32][11:0];
wire qsfp_bsy_force = register_data_in[32][12];
wire gdm_veto_enable = register_data_in[32][13];
// not used = register_data_in[32][15:14];
wire [15:0] gdm_veto_extend = register_data_in[32][31:16];
</pre>

On the GDM:

qsfp_rx_bsy[11..0] are pulses received from the CDMs

qsfp_rx_bsy_or is the grand-or of qsfp_rx_bsy masked by qsfp_mask

if qsfp_rx_bsy_or is high, gdm_bsy goes up and stays up for vx_bsy_extend*2 clocks

gdm_veto = gdm_bsy (bsy of any one VX causes trigger veto to all of them)

if (gdm_veto_enable) then gdm_veto will block trg_in_pulse

gdm_veto transition 0->1 causes gdm_veto_pulse. as long as gdm_veto is high, gdm_veto_pulse is generated every gdm_veto_extend clocks

gdm_veto_pulse is sent to all CDMs.

For this to work right, cdm_bsy_extend should not be bigger than gdm_veto_extend.

=== Register 33 0x80013084 bitmap of enabled VX ports ===

<pre>
wire [11:0] vx_mask = register_data_in[33][11:0];
wire vx_bsy_force = register_data_in[33][12];
// not used = register_data_in[33][15:13];
wire [15:0] vx_bsy_extend = register_data_in[33][31:16];
</pre>

On the CDM:

cdm_bsy is a grand-or of all vx_bsy masked by vx_mask (list of active VXes).

vx_bsy_extend controls how often state of cdm_bsy is sent to the GDM. when cdm_bsy goes 0->1, we send a cdm_bsy_pulse and keep resending it every vx_bsy_extend clocks as long as cdm_bsy stays high.

cdm_bsy_pulse is sent to the GDM.

sfp_rx_veto is the received from the GDM

if sfp_rx_veto goes up, cdm_veto goes up and stays up for gdm_veto_extend clocks.

for this to work right, CDM gdm_veto_extend must be bigger than GDM gdm_veto_extend.

cdm_veto goes to VXes on v1_tx_out[1] which is LVDS input 13.

=== Register 34, 35 0x80013088, 8C current timestamp ===

current 64-bit timestamp, 125 MHz

=== Register 36, 37 0x80013090, 94 old timestamp ===

old 64-bit timestamp, 125 MHz. timestamp saved at run start when it is reset to 0.

=== Register 38, 39, 40 0x80013098, 9C, A0 VX busy counters ===

8 bits per VX port, counters overflow to 255, reset at run start.

=== Register 41, 42, 43 0x800130A4, A8, AC QSFP busy counters ===

8 bits per QSFP port, counters overflow to 255, reset at run start.

=== Register 44, 45, 46, 47, 48 CDM and GDM busy and veto counters ===

* cdm_busy = grand-or of all VX busy for enabled VXes
* gdm_busy = grand-or of all CDM busy for enabled CDMs
* gdm_veto = gdm_busy

<pre>
44 | lo 16 bits | cdm_bsy_up_counter | CDM busy, increments when cdm_busy goes 0->1
44 | hi 16 bits | cdm_bsy_pulse_counter | CDM busy to GDM, increments for each cdm_bsy_pulse sent to the GDM

45 | lo 16 bits | cdm_veto_pulse_counter | CDM veto from GDM, increments for each sfp_rx_veto received from the GDM
45 | hi 16 bits | cdm_veto_up_counter | CDM veto to VX, increments each time cdm_veto is set to 1.

46 | lo 16 bits | gdm_bsy_pulse_counter | GDM busy from CDM, increments for each qsfp_rx_busy received from CDMs (unless they overlap)
46 | hi 16 bits | gdm_bsy_up_counter | GDM busy, increments each time gdm_bsy goes 0->1

47 | lo 16 bits | gdm_bsy_refresh_counter | GDM busy, increments each time gdm_bsy is extended by new qsfp_rx_busy
47 | hi 16 bits | spare |

48 | lo 16 bits | gdm_veto_up_counter | GDM veto, increments each time gdm_veto goes 0->1
48 | hi 16 bits | gdm_veto_pulse_counter | GDM veto to CDM, increments for each gdm_veto_pulse sent to the CDM
</pre>

=== Register 49, 50, 51 0x800130xx VX RX serial data monitor ===

VX RX data, 8-bit per VX channel. k-bit is omitted.

<pre>
assign register_data_out[49][7:0] = vx_rx_data[0]; // vx1
assign register_data_out[49][15:8] = vx_rx_data[1]; // vx2
assign register_data_out[49][23:16] = vx_rx_data[2]; // vx3
assign register_data_out[49][31:24] = vx_rx_data[3]; // vx4

assign register_data_out[50][7:0] = vx_rx_data[4]; // vx5
assign register_data_out[50][15:8] = vx_rx_data[5]; // vx6
assign register_data_out[50][23:16] = vx_rx_data[6]; // vx7
assign register_data_out[50][31:24] = vx_rx_data[7]; // vx8

assign register_data_out[51][7:0] = vx_rx_data[8]; // vx9
assign register_data_out[51][15:8] = vx_rx_data[9]; // vx10
assign register_data_out[51][23:16] = vx_rx_data[10]; // vx11
assign register_data_out[51][31:24] = vx_rx_data[11]; // vx12
</pre>

=== Register 52, 53 VX link status ===

<pre>
assign register_data_out[52] =
{
vx_rx_error[7], vx_link_error[7], vx_link_status[7], vx_link_rx_status[7], // 7
vx_rx_error[6], vx_link_error[6], vx_link_status[6], vx_link_rx_status[6], // 6
vx_rx_error[5], vx_link_error[5], vx_link_status[5], vx_link_rx_status[5], // 5
vx_rx_error[4], vx_link_error[4], vx_link_status[4], vx_link_rx_status[4], // 4
vx_rx_error[3], vx_link_error[3], vx_link_status[3], vx_link_rx_status[3], // 3
vx_rx_error[2], vx_link_error[2], vx_link_status[2], vx_link_rx_status[2], // 2
vx_rx_error[1], vx_link_error[1], vx_link_status[1], vx_link_rx_status[1], // 1
vx_rx_error[0], vx_link_error[0], vx_link_status[0], vx_link_rx_status[0] // 0
};

assign register_data_out[53] =
{
//4'b0000,
1'b0, ~vx_rx_deser_rdy[0], vx_rx_code_err[0], vx_rx_disp_err[0],
//4'b0000,
//4'b0000,
//4'b0000,
vx_rx_monitor[0],
vx_rx_error[11], vx_link_error[11], vx_link_status[11], vx_link_rx_status[11], // 11
vx_rx_error[10], vx_link_error[10], vx_link_status[10], vx_link_rx_status[10], // 10
vx_rx_error[9], vx_link_error[9], vx_link_status[9], vx_link_rx_status[9], // 9
vx_rx_error[8], vx_link_error[8], vx_link_status[8], vx_link_rx_status[8] // 8
};
</pre>

vx_rx_monitor (12-bit) is from deserializer_10b.sv:

<pre>
assign monitor_out[9:0] = lastByte[9:0];
assign monitor_out[10] = comma;
assign monitor_out[11] = ready;
</pre>

=== Register 54 VX TX serial data monitor ===

* contents of vx_tx_monitor from vx_ser_tx.sv:
<pre>
assign monitor_out[8:0] = data_to_encoder; // 8-bit + k
assign monitor_out[9] = valid;
assign monitor_out[15:10] = 0;
assign monitor_out[25:16] = encoded_data; // 10-bit
assign monitor_out[26] = encoded_valid;
assign monitor_out[27] = 0;
assign monitor_out[31:28] = 0;
</pre>

=== Register 55 QSFP, SFP, VX link loss counters ===

<pre>
assign register_data_out[55] = {
vx_rx_error_counter,
qsfp_link_rx_status_drop_counter,
sfp_link_rx_status_drop_counter,
vx_link_rx_status_drop_counter
};
</pre>

<pre>
24..31 - CDM VX RX error counters, count any errors in the VX RX data path (bad serial data, fifo overflow, etc)
16..23 - GDM QSFP link loss counter, increments on qsfp_tx_link_rx_status 1->0 (reg10)
8..15 - CDM SFP link loss counter, increments on sfp_link_rx_status 1->0 (reg15)
0...7 - CDM VX link loss counter, increments on vx_tx_link_rx_status 1->0 (reg10)
</pre>

=== Register 68 GPS control and status ===

<pre>
assign register_data_out[68] =
{
8'b00000000, // 23+8
rb_1pps_counter, // 16+8 bits
gps_1pps_counter, // 8+8 bits
1'b0,
1'b0,
rb_ser_in, // 5
gps_data_in, // 4
gps_aux_out, // 3
gps_aux_in, // 2
rb_1pps_in, // 1
gps_1pps_in // 0
};
</pre>

=== Register 71 packet error bits ===

<pre>
assign register_data_out[71] =
{
8'b00000000, // 24+8 bits
1'b0, // 23
1'b0, // 22
fifo_to_fpga_full_latch, // 21
fifo_from_fpga_full_latch, // 20
gdm_hitmap_fifo_full_latch, // 19
gdm_busy_fifo_full_latch, // 18
cdm_busy_fifo_full_latch, // 17
cdm_hitmap_fifo_full_latch, // 16
1'b0, // 15
1'b0, // 14
1'b0, // 13
gdm_trig48_encode_error_latch, // 12
1'b0, // vx_tx_trig48_pkt8_error_latch, // 11
tsm2_encode_error_latch, // 10
tsm_encode_error_latch, // 9
trg_encode_error_latch, // 8
gdm_hitmap_error_latch, // 7
gdm_hitmap_encode_error_latch, // 6
gdm_busy_encode_error_latch, // 5
cdm_busy_encode_error_latch, // 4
cdm_hitmap_encode_error_latch, // 3
vx_rx_hitmap_error_latch, // 2
1'b0, // vx_tx_tsm_pkt8_error_latch, // 1
1'b0, // vx_tx_trg_pkt8_error_latch // 0
};
</pre>

=== Register 90 VX busy status register ===

<pre>
vx_bsy_status_latched[11:0] <= vx_bsy[11:0];
vx_bsy_status_latched[23:12] <= vx_bsy_collect[11:0];
vx_bsy_status_latched[24] <= cdm_bsy;
vx_bsy_status_latched[25] <= cdm_bsy_collect;
vx_bsy_status_latched[26] <= cdm_veto;
vx_bsy_status_latched[27] <= cdm_veto_collect;
vx_bsy_status_latched[28] <= 0;
vx_bsy_status_latched[29] <= 0;
vx_bsy_status_latched[30] <= 0;
vx_bsy_status_latched[31] <= cdm_bsy_stuck;
</pre>

=== Register 91 VX hitmap status register ===

<pre>
hitmap_status_latched[11:0] <= hitmap_or12[11:0];
hitmap_status_latched[23:12] <= hitmap_or12_collect[11:0];
hitmap_status_latched[24] <= hitmap_grand_or;
hitmap_status_latched[25] <= hitmap_grand_or_collect;
hitmap_status_latched[26] <= 0;
hitmap_status_latched[27] <= 0;
hitmap_status_latched[28] <= 0;
hitmap_status_latched[29] <= 0;
hitmap_status_latched[30] <= 0;
hitmap_status_latched[31] <= hitmap_grand_or_stuck;
</pre>

=== Register 92 GDM busy status register ===

<pre>
gdm_bsy_status_latched[11:0] <= qsfp_rx_bsy[11:0];
gdm_bsy_status_latched[23:12] <= qsfp_rx_bsy_collect[11:0];
gdm_bsy_status_latched[24] <= gdm_bsy;
gdm_bsy_status_latched[25] <= gdm_bsy_collect;
gdm_bsy_status_latched[26] <= gdm_veto;
gdm_bsy_status_latched[27] <= gdm_veto_collect;
gdm_bsy_status_latched[28] <= 0;
gdm_bsy_status_latched[29] <= 0;
gdm_bsy_status_latched[30] <= 0;
gdm_bsy_status_latched[31] <= gdm_bsy_stuck;
</pre>

=== Register 93 GDM hitmap status register ===

<pre>
hitmap_status_latched[11:0] <= hitmap_grand_or_12[11:0];
hitmap_status_latched[23:12] <= hitmap_grand_or_12_collect[11:0];
hitmap_status_latched[24] <= hitmap_grand_or;
hitmap_status_latched[25] <= hitmap_grand_or_collect;
hitmap_status_latched[26] <= 0;
hitmap_status_latched[27] <= 0;
hitmap_status_latched[28] <= 0;
hitmap_status_latched[29] <= 0;
hitmap_status_latched[30] <= 0;
hitmap_status_latched[31] <= hitmap_grand_or_stuck;
</pre>

=== Register 98, 99 GDM TRIG48 test register and TRIG48_CTRL ===

<pre>
assign gdm_trig48_ctrl_bitmap48[31:0] = register_data_in[98][31:0];
assign gdm_trig48_ctrl_bitmap48[47:32] = register_data_in[99][15:0];

assign gdm_trig48_ctrl[15:0] = register_data_in[99][31:16];
</pre>

<pre>
wire enable_grand_or = trig48_ctrl[15];
wire enable_test_trig48 = trig48_ctrl[14];
</pre>

= Firmware registers branch develop_ko =

== Register map ==

<pre>
0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
1 | ALL | ALL | RW | read write scratch register
2 | ALL | CDM | ?? | MGT not used
3 | ALL | CDM | RO | MGT debug_data
4 | ALL | CDM | RW | clk_config_vec
5 | ALL | CDM | ?? | not used
6 | ALL | CDM | RO | CDM_link_data_processing:o_error_time
7 | ALL | CDM | RO | CDM_link_data_processing:o_error_count
</pre>

== register 0 0x80010000 ==

GDM:
<pre>
0 - gdm_link_interface:i_mgt_rst
2 - gdm_link_interface:i_link_down_latched_rst
8 - GDM_link_data_processing:i_rst
10..9 - GDM_link_data_processing:i_data_mode
</pre>

CDM:
<pre>
0 - cdm_link_interface:i_mgt_rst
2 - cdm_link_interface:i_link_down_latched_rst
8 - CDM_link_data_processing:i_rst
10..9 - CDM_link_data_processing:i_data_mode
</pre>

== register 1 0x80010004 ==

GDM:
<pre>
nlinks-1..0 - gdm_link_interface:i_rx_slide_trigger
</pre>

CDM:
<pre>
nlinks-1..0 - cdm_link_interface:i_rx_slide_trigger
</pre>

== register 2 0x80010008 ==

GDM:
<pre>
nlinks-1..0 - gdm_link_interface:o_link_power_good
nlinks+15..16 - gdm_link_interface:o_link_status
</pre>

CDM:
<pre>
nlinks-1..0 - cdm_link_interface:o_link_power_good
nlinks+15..16 - cdm_link_interface:o_link_status
</pre>

== register 3 0x8001000c ==

GDM: simple loopback register

CDM:
<pre>
31..0 - debug_data - cdm_link_interface:o_debug
</pre>

o_debug:
<pre>
rx_link_rst & rx_error & rx_link_up & rx_receiving_data &
std_logic_vector(rx_state_count) & tx_state_count_on_rx_clk & i_rx_ctrl3(0) &
i_rx_ctrl1(1 downto 0) & i_rx_ctrl0(1 downto 0) &
rx_data_is_k28p1_k28p5 &
i_rx_data;
</pre>

== register 4 0x80010010 ==

GDM write:
<pre>
0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS
</pre>

GDM read:
<pre>
0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS
4 - clk_config_vec(4) - CLK_LOSXTn_LS
5 - clk_config_vec(5) - CLK_LOLn_LS
6 - clk_config_vec(6) - CLK_INTn_LS
7 - constant 1
31..8 - constant 0
</pre>

== register 5 0x80010014 ==

not used

== register 6 0x80010018 ==

GDM:
<pre>
3..0 - GDM_link_data_processing:i_status_select
</pre>

CDM:
<pre>
31..0 - CDM_link_data_processing:o_error_count
</pre>

== register 7 0x8001001c ==

GDM:
<pre>
31..0 - GDM_link_data_processing:o_status_vector
</pre>

CDM:
<pre>
31..0 - CDM_link_data_processing:o_error_count
</pre>

= GDM, CDM, VX packet communications =

* timestamp math
<pre>
1 clock is 8 ns is 125 MHz
8 bits of clocks is 256 clocks is 2048 ns is ~2 usec
16 bits of clocks is ~500 usec is 0.5 msec
24 bits of clocks is ~134 msec
32 bits of clocks is ~34 sec
40 bits of clocks is ~8.7 ksec is 2.4 hours
48 bits of clocks is ~625 hours is ~26 days
56 bits of clocks is ~6.6 kdays is ~18 kyears
62 bits of clocks is ~10 Mhours is 427 kdays is ~1.1 kyears
64 bits of clocks is ~4.4 kyears
</pre>
* summary
<pre>
0x01 - not used
0x02 - 8 bytes - 4 words - 2 dwords - 1 qwords - TRG packet
0x03 - 12 bytes - 6 words - 3 dwords - 2 qwords - TRIG48 packet
0x10 - 26 bytes - 13 words - 7 dwords - 4 qwords - TSM packet
0x81 - 10 bytes - 5 words - 3 dwords - 2 qwords - VX_HITMAP packet
0x82 - 108 bytes - 54 words - 27 dwords - 14 qwords - CDM_HITMAP packet
0x83 - 12 bytes - 6 words - 3 dwords - 2 qwords - CDM_BUSY packet
0x84 - 12 bytes - 6 words - 3 dwords - 2 qwords - GDM_BUSY packet
0x85 - 12 bytes - 6 words - 3 dwords - 1 qwords - GDM_HITMAP packet
</pre>
* 0x02 - TRG packet, 8 bytes, 80 adc clocks, 640 ns on lvds link
<pre>
0 - 0x02
1 - trg_counter[7:0]
2 - ts64 low byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 high byte 3
6 - trg_in_latch[7:0]
7 - trg_in_latch[15:8]
</pre>
* 0x03 - TRIG48 packet, 12 bytes, 120 adc clocks, 960 ns on lvds link
<pre>
0 - 0x03
1 - trg_counter[7:0]
2 - ts64 low byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 high byte 3
6 - vx_bitmap[7:0]
7 - vx_bitmap[15:8]
8 - vx_bitmap[23:16]
9 - vx_bitmap[31:24]
10 - vx_bitmap[39:32]
11 - vx_bitmap[47:40]
</pre>
* 0x10 - TSM packet, 26 bytes, 260 adc clocks, 2080 ns on lvds link
<pre>
0 - 0x10
1 - tsm_counter[7:0]
2 - gdm_ts64 low byte 0
3 - 1
4 - 2
5 - 3
6 - 4
7 - 5
8 - 6
9 - gdm_ts64 high byte 7
10 - gps_ts64 low byte 0
11 - 1
12 - 2
13 - 3
14 - 4
15 - 5
16 - 6
17 - gps_ts64 high byte 7
18 - gps_data64 low byte 0
19 - 1
20 - 2
21 - 3
22 - 4
23 - 5
24 - 6
25 - gps_data64 high byte 7
</pre>
* 0x81 - VX hitmap packet, 10 bytes, 100 adc clocks, 800 ns on lvds link, 48 ns on fiber link
<pre>
0 - 0x81
1 - VX ID
2 - hitmap low byte, nits 7:0
3 - 15:8
4 - 23:16
5 - 31:24
6 - ...:32
7 - ...
8 - ...
9 - hitmap low byte, bits 63:...
</pre>
* 0x82 - CDM hitmap packet, 108 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x82
1 - cdm_hitmap_trigger_counter[7:0]
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - cdm_hitmap_or12 byte 0 (7:0)
11 - cdm_hitmap_or12 byte 1 (11:8) plus 4 bits: 12=0, 13=0, 14=0, 15=cdm_hitmap_grand_or
12 - cdm_hitmap_data, low byte, 12*64 bits = 768 bits = 48 words = 96 bytes
...
107 - cdm_hitmap_data, high byte
</pre>
* 0x83 - CDM busy packet, 12 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x83
1 - cdm_busy_packet_counter[7:0], reset at begin of run
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - vx_bsy[7:0]
11 - {0,0,cdm_veto,cdm_bsy,vx_bsy[11:8]}
</pre>
* 0x84 - GDM busy packet, 12 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x84
1 - gdm_busy_packet_counter[7:0], reset at begin of run
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - qsfp_rx_bsy[7:0]
11 - {0,0,gdm_veto,gdm_bsy,qsfp_rx_bsy[11:8]}
</pre>
* 0x85 - GDM hitmap trigger packet, 12 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x85
1 - gdm_hitmap_packet_counter[7:0], reset at begin of run
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - hitmap_grand_or_12[7:0]
11 - {hitmap_grand_or,0,0,0,hitmap_grand_or_12[11:8]}
</pre>

= AXI bus timing =

* AXI 100 MHz clock, 10 ns, 32-bit data
* AXI single-dword read: 36 clock repeat rate, 360 ns is 2.777 MHz, 4 bytes per transfer is 11.11 Mbytes/sec
* AXI single-qword read: 13 clock repeat rate, 130 ns is 7.7 MHz, 4 bytes per transfer is 30 Mbytes/sec, not accounting for the gap
* AXI memcpy read: 4 transfers at 13 clocks, gap, 4 transfers at 13 clocks, gap, etc. below 30 Mbytes/sec.
[[Image:Ds-dm-axi-read.png|100px]]
[[Image:Ds-dm-axi-read-64.png|100px]]
[[Image:Ds-dm-axi-read-memcpy.png|100px]]
* AXI single-dword write: 20 clocks repeat rate, 200 ns is 5 MHz, 4 bytes per transfer is 20 Mbytes/sec
* AXI single-qword write: 13+20 clocks repeat rate, 330 ns is 3 MHz, 16 bytes per burst is 48 Mbytes/sec
[[Image:Ds-dm-axi-write.png|100px]]
[[Image:Ds-dm-axi-write-64.png|100px]]

= AXI bus addresses =

* see AXI/AMBA addresses assigned inthe FPGA project: (s_axi/reg0 is the DS-DM AXI registers)
<pre>
daq00:ds-dm-gcdm$ grep assign_bd_address scripts/GDM_CDM_XU8_bd.tcl
assign_bd_address -offset 0x80010000 -range 0x00004000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs axi_register_interfa_0/s_axi/reg0] -force
assign_bd_address -offset 0x000400000000 -range 0x40000000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs ddr4/C0_DDR4_MEMORY_MAP/C0_DDR4_ADDRESS_BLOCK] -force
assign_bd_address -offset 0x80000000 -range 0x00010000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs system_management_wiz/S_AXI_LITE/Reg] -force
daq00:ds-dm-gcdm$
</pre>

* see AXI/AMBA addresses exported from FPGA project to Linux kernel: (uio for debug bridge should say "debug bridge")
<pre>
root@gdm0:~# cat /sys/class/uio/uio*/name
axi-pmon
axi-pmon
axi-pmon
axi-pmon
root@gdm0:~#
root@gdm0:~# ls -l /sys/class/uio/
total 0
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio0 -> ../../devices/platform/amba/ffa00000.perf-monitor/uio/uio0
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio1 -> ../../devices/platform/amba/fd0b0000.perf-monitor/uio/uio1
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio2 -> ../../devices/platform/amba/fd490000.perf-monitor/uio/uio2
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio3 -> ../../devices/platform/amba/ffa10000.perf-monitor/uio/uio3
root@gdm0:~# ls -l /sys/class/uio/../../devices/platform/amba/
total 0
-rw-r--r-- 1 root root 4096 Oct 18 19:37 driver_override
drwxr-xr-x 3 root root 0 Oct 18 01:36 fd070000.memory-controller
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd0b0000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd400000.zynqmp_phy
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd490000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd500000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd510000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd520000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd530000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd540000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd550000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd560000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd570000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd6e0000.cci
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff000000.serial
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff020000.i2c
drwxr-xr-x 6 root root 0 Oct 18 01:36 ff0a0000.gpio
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0b0000.ethernet
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0e0000.ethernet
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0f0000.spi
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff160000.mmc
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff170000.mmc
drwxr-xr-x 3 root root 0 Oct 18 01:36 ff960000.memory-controller
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff9d0000.usb0
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa00000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa10000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa50000.ams
drwxr-xr-x 5 root root 0 Oct 18 01:36 ffa60000.rtc
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa80000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa90000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffaa0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffab0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffac0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffad0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffae0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffaf0000.dma
-r--r--r-- 1 root root 4096 Oct 18 19:37 modalias
lrwxrwxrwx 1 root root 0 Oct 18 19:37 of_node -> ../../../firmware/devicetree/base/amba
drwxr-xr-x 2 root root 0 Oct 18 19:37 power
lrwxrwxrwx 1 root root 0 Oct 18 01:36 subsystem -> ../../../bus/platform
-rw-r--r-- 1 root root 4096 Oct 18 01:36 uevent
root@gdm0:~#
</pre>

= Build firmware =

== Build from git clone ==

THESE ARE K.O.'s NOTES FOR CREATING THE PETALINUX DIRECTORY.

THEY DO NOT WORK!

COPY PETALINUX FROM A WORKING PROJECT AND USE "make gdm" and "make cdm" AS DESCRIBED BELOW.

* git clone git@edev-group.triumf.ca:fw/exp/darkside/gcdm.git
* #Makefile change VIVADO_SETTINGS_SCRIPT := /opt/Xilinx/Vivado/2022.1/settings64.sh
* #. /opt/Xilinx/Vivado/2022.1/settings64.sh
* . /opt/Xilinx/Vivado/2020.2/settings64.sh
* make clean
* make all_from_scratch
* . /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
* make petalinux_create
* make petalinux_rebuild_new_hw_des
* bomb out: The TMPDIR: /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp can't be located on nfs.
* mkdir /tmp/build_tmp
* rm -rf /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp/
* ln -s /tmp/build_tmp /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp
* try again
* grinds, loads a whole bunch of packages...
* finishes with desire to copy things to /tftpboot
* make sdcard_cp_to wants to copy files from PetaLinux_GDM_CDM/images/linux/ to SD card

== Build firmware ==

NOTE: directory Petalinux_GDM_CDM should already exist!

<pre>
#. /opt/Xilinx/Vivado/2020.2/settings64.sh
. /opt/Xilinx/Vivado/2022.2/settings64.sh
. /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
make clean_gdm # remove gdm build tree
make gdm # build or rebuild GDM
make copy_gdm # copy to gdm0
make clean_cdm # remove cdm build tree
make cdm # build or rebuild CDM
make copy_cdm # copy to cdm0 and cdm1
</pre>

copy to SD card:
<pre>
open a root shell
format 16 GB Sd card per above
cd .../ds-dm-gcdm
make copy
</pre>

= build times =

<pre>
CDM 12-june-2023 69aabc1c25130d970bc375aca684bd68849e6685
daq13 AMD-5700G 1688.61user 399.33system 23:28.84elapsed 148%CPU
dsdaqgw AMD-7700 1090.55user 247.34system 16:03.55elapsed 138%CPU
dsdaqgw AMD-7700 CDM incremental 196.68user 67.62system 7:35.42elapsed 58%CPU
dsdaqgw AMD-7700 CDM incremental 684.72user 94.17system 7:30.17elapsed 173%CPU
dsdaqgw AMD-7700 GDM incremental 849.84user 99.79system 9:04.56elapsed 174%CPU
</pre>

= make bootable sd card =

== format the sd card ==

this only needs to be done once

* become root@dsdaqgw
* cd ~olchansk/git/ds-dm-gcdm
* use "lsblk" to identify the SD card (should show as 8/16/32 GB block device)/ /dev/sdd in this case
* make sdcard_format SDCARD_DEVICE=/dev/sdd
* disconnect sd card, reconnect the sd card (to detect new partition tables, etc)

== copy boot files to the sd card ==

* as root: identify partition labels, run "blkid", should say "BOOT", "rootfs" and "data"
* mount
<pre>
mkdir -p /mnt/BOOT
mount -L BOOT /mnt/BOOT
cp PetaLinux_GDM_CDM/images/linux/BOOT_{GDM,CDM}.BIN /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/boot.scr /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/image.ub /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/uboot.env /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/uboot-redund.env /mnt/BOOT/
umount /mnt/BOOT
eject /dev/sdX
</pre>
* FIXME: missing step: cp BOOT_GDM.BIN BOOT.BIN or cp BOOT_CDM.BIN BOOT.BIN

= boot messages =

<pre>
Xilinx Zynq MP First Stage Boot Loader
Release 2020.2 Sep 24 2022 - 13:29:15
NOTICE: ATF running on XCZU4CG/silicon v4/RTL5.1 at 0xfffea000
NOTICE: BL31: v2.2(release):xlnx_rebase_v2.2_2020.3
NOTICE: BL31: Built : 18:02:46, Sep 28 2022

U-Boot 2020.01 (Sep 28 2022 - 18:03:39 +0000)

Model: DarkSide 20k DM
Board: Xilinx ZynqMP
DRAM: 2 GiB
usb dr_mode not found
PMUFW: v1.1
EL Level: EL2
Chip ID: zu4
NAND: 0 MiB
MMC: mmc@ff160000: 0, mmc@ff170000: 1
In: serial@ff000000
Out: serial@ff000000
Err: serial@ff000000
Bootmode: SD_MODE1
Reset reason: SOFT
Net:
ZYNQ GEM: ff0b0000, mdio bus ff0b0000, phyaddr -1, interface rgmii-id

Warning: ethernet@ff0b0000 (eth0) using random MAC address - d6:62:5f:13:00:44
eth0: ethernet@ff0b0000
ZYNQ GEM: ff0e0000, mdio bus ff0e0000, phyaddr -1, interface rgmii-id
Could not get PHY for eth1: addr -1

Hit any key to stop autoboot: 0
ZynqMP>
CTRL-A Z for help | 115200 8N1 | NOR | Minicom 2.7.1 | VT102 | Online 122:2 | ttyACM0
</pre>

= load FPGA from u-boot =

* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/124682257/U-Boot+FPGA+Driver

<pre>
ZynqMP> fpga info
Xilinx Device
Descriptor @ 0x000000007fddb2c0
Family: ZynqMP PL
Interface type: csu_dma configuration interface (ZynqMP)
Device Size: 1 bytes
Cookie: 0x0 (0)
Device name: zu4
Device Function Table @ 0x000000007fda5fe8
PCAP status 0xa0002fde
ZynqMP>
</pre>

* cp CDM_XU8_top.bit /tftpboot/fpga.bit

<pre>
dhcp
tftpb 0x10000000 fpga.bit
fpga loadb 0 0x10000000 ${filesize}
</pre>

<pre>
ZynqMP> dhcp
BOOTP broadcast 1
DHCP client bound to address 192.168.0.100 (1 ms)
*** Warning: no boot file name; using 'C0A80064.img'
Using ethernet@ff0b0000 device
TFTP from server 192.168.0.1; our IP address is 192.168.0.100
Filename 'C0A80064.img'.
Load address: 0x8000000
Loading: *
TFTP error: 'file /tftpboot/C0A80064.img not found for 192.168.0.100' (1)
Not retrying...
ZynqMP> tftpb 0x10000000 fpga.bit
Using ethernet@ff0b0000 device
TFTP from server 192.168.0.1; our IP address is 192.168.0.100
Filename 'fpga.bit'.
Load address: 0x10000000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
############
6.2 MiB/s
done
Bytes transferred = 7797807 (76fc2f hex)
ZynqMP> fpga loadb 0 0x10000000 ${filesize}
design filename = "CDM_XU8_top;UserID=0XFFFFFFFF;Version=2022.2"
part number = "xczu4cg-fbvb900-1-e"
date = "2024/08/14"
time = "14:18:22"
bytes in bitstream = 7797692
zynqmp_align_dma_buffer: Align buffer at 0000000010000073 to 000000000fffff80(swap 0)
ZynqMP>
</pre>

= load FPGA from Linux =

* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841847/Solution+ZynqMP+PL+Programming

== this will reset the CPU ==
<pre>
cp fpga.bit /lib/firmware/
echo fpga.bit > /sys/class/fpga_manager/fpga0/firmware
</pre>

* make .bin file:
<pre>
bootgen -image CDM_XU8_top.bif -arch zynqmp -o ./fpga.bin -w
</pre>
* cat CDM_XU8_top.bif
<pre>
dsdaqgw:ds-dm-gcdm$ cat CDM_XU8_top.bif
all:
{
[destination_device = pl] ./Vivado_CDM_XU8/CDM_XU8.runs/impl_1/CDM_XU8_top.bit
}
dsdaqgw:ds-dm-gcdm$
</pre>

== this will reset the CPU ==

<pre>
cp fpga.bin /lib/firmware/
echo fpga.bin > /sys/class/fpga_manager/fpga0/firmware
</pre>

== this will reset the CPU ==

<pre>
root@dsdm:~# ./fpgautil -b fpga.bin -f Full
</pre>

== this will reset the CPU ==

<pre>
DTSO file from here: https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841847/Solution+ZynqMP+PL+Programming
echo 0 > /sys/class/fpga_manager/fpga0/flags
mount -t configfs configfs /configfs
root@dsdm:~# dtc -O dtb -o fpga.dtbo -b 0 -@ fpga.dtso
root@dsdm:~# cp fpga.dtbo /lib/firmware/
root@dsdm:~# cp fpga.bit /lib/firmware/
root@dsdm:~# rmdir /configfs/device-tree/overlays/fpga
root@dsdm:~# mkdir /configfs/device-tree/overlays/fpga
root@dsdm:~# echo -n "fpga.dtbo" > /configfs/device-tree/overlays/fpga/path
</pre>

= fpgautil =

* https://github.com/Xilinx/meta-xilinx/blob/master/meta-xilinx-core/recipes-bsp/fpga-manager-script/files/fpgautil.c

<pre>
git clone https://github.com/Xilinx/meta-xilinx.git
cd meta-xilinx/meta-xilinx-core/recipes-bsp/fpga-manager-script/files/
scp fpgautil.c root@dsdm:
ssh root@dsdm
make fpgautil
ls -l ./fpgautil
</pre>

<pre>
root@dsdm:~# ls -l ./fpgautil
-rwxr-xr-x 1 root root 72256 Aug 16 00:15 ./fpgautil
root@dsdm:~# ./fpgautil

fpgautil: FPGA Utility for Loading/reading PL Configuration

Usage: fpgautil -b <bin file path> -o <dtbo file path>

Options: -b <binfile> (Bin file path)
-o <dtbofile> (DTBO file path)
-f <flags> Optional: <Bitstream type flags>
f := <Full | Partial >
-n <Fpga region info> FPGA Regions represent FPGA's
and partial reconfiguration
regions of FPGA's in the
Device Tree
Default: <full>
-s <secure flags> Optional: <Secure flags>
s := <AuthDDR | AuthOCM | EnUsrKey | EnDevKey | AuthEnUsrKeyDDR | AuthEnUsrKeyOCM | AuthEnDevKeyDDR | AuthEnDevKeyOCM>
-k <AesKey> Optional: <AES User Key>
-r <Readback> Optional: <file name>
Default: By default Read back contents will be stored in readback.bin file
-t Optional: <Readback Type>
0 - Configuration Register readback
1 - Configuration Data Frames readback
Default: 0 (Configuration register readback)
-R Optional: Remove overlay from a live tree

Examples:
(Load Full bitstream using Overlay)
fpgautil -b top.bit.bin -o can.dtbo -f Full -n full
(Load Partial bitstream using Overlay)
fpgautil -b rm0.bit.bin -o rm0.dtbo -f Partial -n PR0
(Load Full bitstream using sysfs interface)
fpgautil -b top.bit.bin -f Full
(Load Partial bitstream using sysfs interface)
fpgautil -b rm0.bit.bin -f Partial
(Load Authenticated bitstream through the sysfs interface)
fpgautil -b top.bit.bin -f Full -s AuthDDR
(Load Parital Encrypted Userkey bitstream using Overlay)
fpgautil -b top.bit.bin -o pl.dtbo -f Partial -s EnUsrKey -k <32byte key value>
(Read PL Configuration Registers)
fpgautil -b top.bit.bin -r
(Remove Partial Overlay)
fpgautil -R -n PR0
(Remove Full Overlay)
fpgautil -R -n full
Note: fpgautil -R is responsible for only removing the dtbo file from the livetree. it will not remove the PL logic from the FPGA region.

root@dsdm:~#
</pre>

= fw_printenv =

to access u-boot environment from Linux:
* apt install -y libubootenv-tool
* create /etc/fw_env.config
<pre>
/media/BOOT/uboot.env 0 0x40000
/media/BOOT/uboot-redund.env 0 0x40000
</pre>
* if uboot.env files do not exist, run "saveenv" from u-boot command prompt
* fw_printenv and fw_setenv should work

= Boot from network =

== u-boot ==

<pre>
ZynqMP> setenv bootcmd run bootcmd_dhcp
ZynqMP> saveenv
ZynqMP> reset
</pre>

== boot.scr ==

<pre>
# boot.scr
# mkimage -C none -A arm -T script -d boot.scr boot.scr.uimg
echo Loading FPGA!
#tftpb 0x10000000 fpga.bit
tftpb 0x10000000 {ipaddr}.bit
fpga loadb 0 0x10000000 ${filesize}
echo Booting Linux!
run bootcmd_pxe
echo Done!
</pre>

<pre>
mkimage -C none -A arm -T script -d boot.scr boot.scr.uimg
</pre>

== tftpboot ==

<pre>
cp /home/olchansk/git/ds-dm-gcdm/boot.scr.uimg /tftpboot
ln -s /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/images/linux /tftpboot/xilinx-dsdm
mkdir /tftpboot/pxelinux.cfg
cat > /tftpboot/pxelinux.cfg/default-arm-zynqmp <<EOF
LABEL Linux
KERNEL xilinx-dsdm/Image
FDT xilinx-dsdm/system.dtb
#INITRD rootfs.cpio.gz.u-boot
EOF
</pre>

== boot sequence ==

* xilinx magic load BOOT.BIN from SD card
* load FPGA form BOOT.BIN
* load and run u-boot from BOOT.BIN or from image.ub
* u-boot load environment from ??? probably SD card uboot-redund.env, this includes our bootcmd
* run bootcmd which run bootcmd_dhcp which does:
* from /tftpboot:
* load and run boot.scr.uimg which does:
* load FPGA image xilinx-dsdm/${ipaddr}.bit
* run bootcmd_pxe which does:
* load pxelinux.cfg/default-arm-zynqmp which does:
* load xilinx-dsdm/Image ### this is the linux kernel
* load xilinx-dsdm/system.dtb ### this is the device tree
* start linux kernel
* linux kernel does dhcp
* linux kernel does nfs mount /nfsroot/%s,vers=3,tcp ### %s is replaced by the hostname supplied by DHCP
* userland starts and runs to console and ssh login.

= Xilinx ILA =

References:
* https://github.com/Xilinx/XilinxVirtualCable/tree/master
* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/644579329/Xilinx+Virtual+Cable
* https://support.xilinx.com/s/article/974879?language=en_US
* https://docs.xilinx.com/v/u/en-US/pg245-debug-bridge
* https://docs.xilinx.com/r/2020.2-English/ug908-vivado-programming-debugging/Initializing-Vivado-IDE-hw_server
* (we do not use this one) https://github.com/paulscherrerinstitute/xvcSupport/blob/master/README.md

Build xvcserver_cdm.exe: (it is built as a static executable, can be copied and run anywhere)
<pre>
ssh dsdaq@gdm0
cd /home/dsdaq/online/ds-dm-software
git pull ### get latest version
make xvcserver_cdm.exe
ssh root@gdm0
/home/dsdaq/online/ds-dm-software/xvcserver_cdm.exe
INFO: To connect to this xvcServer instance, use url: TCP:gdm0:2542
</pre>

To activate and use the vivado logic analyzer:
* data path: vivado -> hw_server -> xvcserver -> mmap axi bus -> debug bridge -> jtag -> ILA
* define ILAs in the code
* instantiate the xilinx debug bridge at AXI bus address 0x80020000 (FIXME!!! this collides with Ian's AXI addresses)
* build and boot the new FPGA firmware. updating the linux kernel is not necessary.
* login root@gdm0, run: /home/dsdaq/online/ds-dm-software/xvcserver_cdm.exe -v ### with "-v" for the first time to see that vivado does connect to it, without "-v", normally.
* login dsdaqgw, run: hw_server -s tcp:localhost:3121 -e "set auto-open-servers xilinx-xvc:gdm0:2542" ### tells us to connect to port localhost:3121
* login dsdaqgw, run vivado, open project, open hardware manager, open target, open new target, "connect to remote server", hostname "localhost", port "3121", next (bombs, try again, 3 times), popup add virtual cable, enter hostname "gdm0" port "2542", "ok", it shows in "hardware targets", "next", "finish", error popup "[Common 17-163] Missing value for option 'objects', please type 'set_property -help' for usage info", ignore it, in "hardware", right click the "gdm0" one, open target, under "hardware" and "debug bridge" we should see all the ILAs, under "hardware device properties", the "probes file" should have the ".ltx" file generated by vivado "Vivado_GDM_XU8/GDM_XU8.runs/impl_1/debug_nets.ltx", click on an ILA, a waveform should open.

= Software =

* ssh cdm0 # or gdm0
* sudo apt install i2c-tools libi2c-dev
* git clone https://bitbucket.org/team-ds-dm/ds-dm-software
* cd ds-dm-software
* make

== test_cdm.exe ==

=== CDM SFP status ===

<pre>
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0xbb2f0ae7
CDM firmware 0xbb2f0ae7
arg 1: [--sfp]
Polling SFP status...
identifier 0x03
connector 0x07
encoding 0x01
wavelength 0x0352 (850 nm)
vendor_name [FINISAR CORP. ]
vendor_pn [FTLF8526P3BNL ]
vendor_rev [A ]
vendor_sn [N3AB9M8 ]
vendor_date [200319 ]
dm_type 0x68
temp 29.0 C
vcc 3.323 V
tx_bias 7.250 mA
tx_power 478.4 uW
rx_power 2.3 uW
SFP good 1, status: temp 30.6 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 818 uW
...
</pre>

=== GDM QSFP status ===

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --qsfp3 --qsfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x53aee418
CDM firmware 0x53aee418
arg 1: [--qsfp3]
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out hi
arg 2: [--qsfp]
Polling QSFP status...
identifier 0x0d
status 0x02
los 0x8f
temp 28.2 C
vcc 3.323 V
rx_power 0.1 0.1 0.1 0.1 uW
tx_bias 7.6 7.6 7.6 0.0 mA
tx_power 792.2 773.8 823.0 0.1 uW
vendor_name [FINISAR CORP ]
vendor_pn [FTL410QD4C ]
vendor_rev [A ]
wavelength 850
max_temp 70 C
vendor_sn [X79AC0R ]
vendor_date [220309 ]
QSFP good 1, status: temp 27.7 C, los 0x8b, tx_bias 7.5 7.6 7.6 0.0 mA, tx_power 792 772 821 0 uW, rx_power 0 0 466 0 uW
</pre>

=== GDM clock status ===

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
</pre>

* clock chip not loaded, not running:

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x0f 0x16 0x04 0x00 0x01 0x68 0x09 0x00 0x02 0xf2 0x00 0x1f 0xf0 0x22 0xf2, 0x507: 0x00, 0x52A: 0x00, 0x53F: 0x04
Clock chip state 0, status: SYSINCAL XAXB_ERR LOL CAL_PLL IN0 IN_SEL_0 FASTLOCK_STATUS

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x1f7caf52 (528265042) should be ~125 MHz
0x1034 rx_clk: 0x00000000 (0) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x1f7caf52 (528265042) should be ~125 MHz
0x103C tx_clk: 0x00000000 (0) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip good (IN0 - external 10 MHz clock)

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x44 0x00 0xd0 0x01 0x1f 0xff 0x22 0xf2, 0x507: 0x3f, 0x52A: 0x01, 0x53F: 0x00
Clock chip state 1, status: LOS_IN2 OOF_IN2 IN0 IN_SEL_REGCTRL IN_SEL_0

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735a3b (125000251) should be ~125 MHz
0x1034 rx_clk: 0x07735a3c (125000252) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a3b (125000251) should be ~125 MHz
0x103C tx_clk: 0x07735a3c (125000252) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

=== CDM clock status ===

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
</pre>

* clock chip not loaded, not running:

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x0f 0x16 0x04 0x00 0x01 0x68 0x19 0x00 0x02 0xf2 0x00 0x1f 0xf0 0x22 0xf2, 0x507: 0x00, 0x52A: 0x00, 0x53F: 0x04
Clock chip state 0, status: SYSINCAL XAXB_ERR LOL CAL_PLL IN0 IN_SEL_0 FASTLOCK_STATUS

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735851 (124999761) should be ~125 MHz
0x1034 rx_clk: 0x00000000 (0) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x0127fefa (19398394) should be ~125 MHz
0x103C tx_clk: 0x00000000 (0) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip uses internal clock (IN1 - internal oscillator), observe rx_clk frequency is not same as others

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0x7f, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN1 IN_SEL_1 HOLD_HIST_VALID

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735853 (124999763) should be ~125 MHz
0x1034 rx_clk: 0x07735b0a (125000458) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735853 (124999763) should be ~125 MHz
0x103C tx_clk: 0x07735852 (124999762) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip uses SFP recovered clock (IN2 - sfp rx recovered clock), observe mgt_rx_ref_clk_raw (CDM 125 MHz oscillator) is different from others (SFP RX recovered clock)

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0xbf, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN2 IN_SEL_1 HOLD_HIST_VALID
^C
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773581b (124999707) should be ~125 MHz
0x1034 rx_clk: 0x07735ad6 (125000406) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735ad7 (125000407) should be ~125 MHz
0x103C tx_clk: 0x07735ad6 (125000406) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
^C
</pre>

=== CDM link status, PRBS test mode ===

<pre>
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
</pre>
* fiber disconnected, no link:
<pre>
CDM firmware: 0xbb2f0ae7
0x1000 SFP c.c. status: 0x00000031
CLK_IN_SEL_LS 0x1
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x00000025
sfp_mod_absent_N 1
sfp_rx_los_N 0
link_power_good 1
rx_link_up 0
rx_receiving_data 0
rx_error 1
rx_lnk_up_and_running 0
tx_link_up 0
tx_sending_data 0
tx_link_up_and_running 0
link_up_and_running 0
0x1014 SFP link data: 0x466a8187
rx_data 0x8187
k28p1_k28p5 0
rx_ctrl0 0x1
rx_ctrl1 0x1
rx_ctrl3 0x1
tx_state 0x1
rx_state 0x6
rx_receiving_data 0
rx_link_up 0
rx_error 1
rx_link_rst 0
0x2000 link test mode: 0x00000200, seconds: 0x00079093, errors: 0xffffffff
</pre>
* fiber connected, good link:
<pre>
CDM firmware: 0xbb2f0ae7
0x1000 SFP c.c. status: 0x000000b2
CLK_IN_SEL_LS 0x2
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x000007dc
sfp_mod_absent_N 0
sfp_rx_los_N 0
link_power_good 1
rx_link_up 1
rx_receiving_data 1
rx_error 0
rx_lnk_up_and_running 1
tx_link_up 1
tx_sending_data 1
tx_link_up_and_running 1
link_up_and_running 1
0x1014 SFP link data: 0x35c02774
rx_data 0x2774
k28p1_k28p5 0
rx_ctrl0 0x0
rx_ctrl1 0x0
rx_ctrl3 0x0
tx_state 0x3
rx_state 0x5
rx_receiving_data 1
rx_link_up 1
rx_error 0
rx_link_rst 0
0x2000 link test mode: 0x00000200, seconds: 0x00078a8c, errors: 0x00000000
</pre>

=== GDM link status, PRBS test mode ===

<pre>
1 link connected, no errors:
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
GDM firmware: 0x6b2ee010
0x1014: 0x00000008, 18: 0x00000008, 1C: 0x00000000, 24: 0x00000fff
0x2000: 0x00000200, time: 0x00078aa4, errors:
0xffffffff 0xffffffff 0xffffffff
0x00000000 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
</pre>

=== CDM link status ===

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test0

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0xbf, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN2 IN_SEL_1 HOLD_HIST_VALID

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x077357a0 (124999584) should be ~125 MHz
0x1034 rx_clk: 0x07735a5c (125000284) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a5b (125000283) should be ~125 MHz
0x103C tx_clk: 0x07735a5b (125000283) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM firmware: 0x6d2ef81a
0x1000 SFP c.c. status: 0x000000b2
CLK_IN_SEL_LS 0x2
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x000007dc
sfp_mod_absent_N 0
sfp_rx_los_N 0
link_power_good 1
rx_link_up 1
rx_receiving_data 1
rx_error 0
rx_lnk_up_and_running 1
tx_link_up 1
tx_sending_data 1
tx_link_up_and_running 1
link_up_and_running 1
0x1014 SFP link data: 0x35c6bcbc
rx_data 0xbcbc
k28p1_k28p5 0
rx_ctrl0 0x3
rx_ctrl1 0x0
rx_ctrl3 0x0
tx_state 0x3
rx_state 0x5
rx_receiving_data 1
rx_link_up 1
rx_error 0
rx_link_rst 0
0x2000 link test mode: 0x00000000, seconds: 0x00001671, errors: 0xffffffff

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 15
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
reg[15] is 0x0033bcbc (3390652)
</pre>

=== GDM link status ===

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test0

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x44 0x00 0xd0 0x01 0x1f 0xff 0x22 0xf2, 0x507: 0x3f, 0x52A: 0x01, 0x53F: 0x02
Clock chip state 1, status: LOS_IN2 OOF_IN2 IN0 IN_SEL_REGCTRL IN_SEL_0 HOLD_HIST_VALID

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735a0a (125000202) should be ~125 MHz
0x1034 rx_clk: 0x07735a0a (125000202) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a0a (125000202) should be ~125 MHz
0x103C tx_clk: 0x07735a0a (125000202) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly

register 0x1018 bit 0x800

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM firmware: 0x6d2ebce6
0x1014: 0x00000800, 18: 0x00000800, 1C: 0x00000000, 24: 0x00000fff
0x2000: 0x00000000, time: 0x00003d2f, errors:
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff

link data alternates 0xbcbc and 0x1cbc

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 22
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
reg[22] is 0xbcbc93ab (-1128492117)

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 22
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
reg[22] is 0x1cbc1aaf (482089647)
root@gdm0:~#

</pre>

=== Run trg and tsm ===

<pre>
on the GDM:

ssh root@gdm00
./test_cdm.exe --gdm-clocks
./test_cdm.exe --load-cc
./test_cdm.exe --cc
./test_cdm.exe --reset-mgt
./test_cdm.exe --gdm-clocks
./test_cdm.exe --writereg 2 0xff # enable LEMO NIM inputs
./test_cdm.exe --writereg 3 0xba54 # LEDs: lemo1, lemo2, trg, tsm
./test_cdm.exe --writereg 4 0x99 $ # LEMO out is trg_in_pulse
./test_cdm.exe --writereg 9 0x32010 # trg and tsm from trg_pulser and tsm_pulse
./test_cdm.exe --writereg 24 1250000 # trg pulser 100 Hz
./test_cdm.exe --writereg 26 125000000 # tsm pulser 1 Hz
./test_cdm.exe --writereg 23 0x40000000 # route trg_in and tsm_in to qsfp tx bits 0 and 1
./test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz

on the CDM:

ssh root@cdm01
./test_cdm.exe --cdm-clocks
./test_cdm.exe --load-cc
./test_cdm.exe --reset-mgt
./test_cdm.exe --cdm-clocks
./test_cdm.exe --cdm-link # issue --reset-mgt on CDM and GDM until link is good
./test_cdm.exe --writereg 2 0xff # enable LEMO NIM inputs
./test_cdm.exe --writereg 3 0xba54 # enable LEDs: lemo1, lemo2, trg, tsm
./test_cdm.exe --writereg 4 0x99 # enable LEMO output trg_in
./test_cdm.exe --writereg 9 0x8040 # enable trg_in and tsm_in from sfp rx bits 0 and 1
./test_cdm.exe --writereg 7 0xff00 # drive VX LVDS lines to logic level 0
./test_cdm.exe --writereg 8 0x2 # VX LVDS with trg and tsm, misrouted in second VX
#./test_cdm.exe --writereg 8 0x7 # VX LVDS with tsm
./test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz
</pre>

switch GDM and CDM to packetizer trg and tsm:

<pre>
on the GDM:

/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x00000000

on the CDM:

/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x0804
/home/dsdaq/online/ds-dm-software/test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz
</pre>

=== Run packet loopback ===

GDM CPU -> fifo_to_fpga -> GDM QSFP -> CDM SFP -> fifo_from_fpga -> CDM CPU

On the GDM: (CDM is connected to first QSFP port)

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --load-cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --reset-mgt
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 32 1 ### tell GDM to use first QSFP port
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link ### confirm link status is "3"
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 29 2 ### enable GDM packet data injection
/home/dsdaq/online/ds-dm-software/test_cdm.exe --test-fifo-write-loop
</pre>

One the CDM:

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --load-cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --reset-mgt
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 29 3 ### enable CDM packet data injection
/home/dsdaq/online/ds-dm-software/test_cdm.exe --test-fifo-read
</pre>

== fecdm ==

=== Power up and clock management ===

* Clock power up and setup management is handled by fecdm state machine
* GDM clock setup sequence:
<pre>
- on power up, CC is not initialized, XAXB clock is not running
- on GDM frontend start (in HandleInit()), CC is loaded and reset
- CC will lock to IN0 (10 MHz external clock) or IN1 (125 MHz internal oscillator)
- if locked to IN0, CC will enter and exit the HOLDOVER mode if 10 MHz clock is unstable
</pre>
* CDM clock setup sequence:
<pre>
- on power up, CC is not initialized, XAXB clock is not running
- fecdm detects this, loads the CC configuration registers, resets the CC
- sets the CC input IN1, DSDM internal oscillator
- if SFP is in good state, SFP MGT is running, CC is happy with IN2 input (no IN2_OOF, IN2_LOS), SFP recpvered clock (clk_rx_mgt) frequency is close to 125 MHz:
- switches CC to input IN2 (SFP recovered clock, clk_rx_mgt)
- waits for CC locked to IN2 (no LOL, no HOLD). (this provides stable SFP tx clock).
- resets the SFP MGT, requests GDM MGT reset (via ODB)
- MGT reset sequence: GDM MGT reset 1, CDMnn MGT reset 1, GDM MGT reset 0, CDM MGT reset 0
- CDM MGT rx_state cycles from 0 to 5. if it goes to 6 or never reaches 5, GDM MGT reset is requested again.
- during this sequence, clk_rx_mgt may wobble, CC may enter HOLDOVER mode, VX clock should stay steady.
- once CC is locked to IN2, SFP MGT rx_state is 5, link is ready for use.
</pre>
* Request changes for Ian
<pre>
17 Apr 2025:
- CDM SFP MGT rx_state is getting stuck in 4 (waiting for alternating idle chars), do we need this? go to state 5 without waiting?
- GDM QSFP MGT reset 1 does not seem to do anything, change it to cause all CDM MGTs to enter an error state, e.g. start transmitting invalid K-codes. transmitting all 0 or all 1 will break the recovered clock, so maybe do not do that.
- having all CDM SFP MGTs go to error state immediatley after GDM QSFT MGT reset 1 will greatly simplify the link reset sequence.
</pre>

=== Trigger configuration ===

* GDM
* CDM in SFP trigger mode:
** SFP K-code encoded sfp_rx_trg, sfp_rx_tsm are routed to LVDS vx_tx_trg and vx_tx_tsm
** GDM TRG, TSM (and all other) packets are routed to LVDS serial data to VX
** to make counters count and lights blink:
** CDM TRG and TSM trigger mask is set to sfp_rx_trg, sfp_rx_tsm and vx_tx_trg_done, vx_tx_tsm_done
** CDM LEDs display trg_in_pulse, tsm_in_pulse
** output of TRG and TSM packets is disabled (they come from the GDM)

= dsvslice integration =

== VX setup ==

* general
** Start acq from user code = y, all others = n (as of Mar 2023: will start when first trigger received)
** Use NIM IO = y
** Use external clock = y
** LVDS quartet is input = n, y, n, y
** LVDS quartet mode = User, User, User, User

* trigger from front panel NIM:
** Trigger on external signal = y, all others = n
** connect CDM EXT_OUT(2) to VX "TrigIn"

* trigger from LVDS "Sync" mode
** Trigger on LVDS Sync signal = y, all others = n
** LVDS quartet mode = User, Sync, User, User

* trigger from LVDS "User" mode
** Trigger on LVDS pair 12 signal = y, all others = n
** LVDS quartet is input = n, y, n, y
** LVDS quartet mode = User, User, User, User

== GDM setup ==

* GDM is gdm0
* set inputs to NIM mode
* set outputs to TTL mode (this GDM has wrong NIM output circuit)
* use top QSFP slot, connect split cable 1 into CDM01, cable 2 into CDM02
* connect non-inverted NIM trigger signal to top-LEMO-left EXT_IN_LV(1)
* connect non-inverted NIM TSM signal to top-LEME-right EXT_IN_LV(2)
* GDM LEDs: TRIG, TSM, trigger enabled, trigger_out
* GDM LEMO_OUT: trigger, trigger

== CDM setup ==

* set CDM LEMO inputs to NIM
* set CDM LEMO outputs to NIM
* CDM01 is cdm0
* CDM02 is cdm1
* connect GDM fiber links to SFP port
* connect 1st VX port of CDM01 to VX1
* connect 1st VX port of CDM02 to VX2
* connect LEMO EXT_OUT(2) to VX "TrigIn", CDM01 to VX1, CDM02 to VX2
* power up
* CDM LEDs: GDM TRIG, GDM TSM, trigger enabled, trigger out
* CDM LEMO_OUT: gdm_trg, trigger

== After power up ==

* start the CDM frontend from the MIDAS "Programs" page. To start manually, see the Start Command on the Programs page.
* CDM frontend should enable the VX clock, disable the trigger
* from the MIDAS status page, goto the CDM page
* outdated: in the CDMx data tables, the 2nd number should read 0x35c08008, if it does not and the last 4 digits randomly change, reset the GDM links
*if the FEs complain - do in order: for GDM, CDM01, CDM02, ..., press "reset mgt" of each board, then press "unreset mgt", if it does not help, STOP HERE
* start a run
* CDM frontend will enable the trigger
* GDM frontend will enable the trigger
* LEDs on the GDM should flash, LEDs on the CDM should flash, TrigIn and TrigOut of the VX should flash
* stop a run
* GDM frontend will disable the trigger
* CDM frontend will disable the trigger

== Phase measurement ==

* pip3 install matplotlib
* pip3 install scipy
* export PYTHONPATH=$HOME/packages/midas/python
* #git clone https://github.com/J033X071C/PhaseMeasurement
* git clone https://bitbucket.org/team-ds-dm/phasemeasurement.git
* cd phasemeasurement
* python3 ./phaseMeasurement.py --help
<pre>
daq00:PhaseMeasurement$ python3 ./phaseMeasurement.py --help
usage: phaseMeasurement.py [-h] fileName numberEvents numberVX sizeEvents stopEvent minHist maxHist numberBin writeToTXT saveAsPDF

Read data from midas file (in .lz4 format) to calculate phase between the clock of VX1 and VX2

positional arguments:
fileName Name of the file we want to read data from (Example: run00389.mid.lz4)
numberEvents Number of events recorded in the file
numberVX Number of VX used in this run (usually 2...)
sizeEvents Number of points per event
stopEvent Number of events you want to go through to calculate phase
minHist Minimal value for the x axis of the phase measurement histogram (in ns)
maxHist Maximal value for the x axis of the phase measurement histogram (in ns)
numberBin Number of bins wanted for the generated histogram
writeToTXT Write argument as yes to generate text file with results of calculation
saveAsPDF Save generated plots to PDF files

optional arguments:
-h, --help show this help message and exit
daq00:PhaseMeasurement$
</pre>
* try an old file with
* python3 ./phaseMeasurement.py run00877.mid.lz4 10000 2 10000 500 -20 20 81 yes yes
<pre>
bin size = 0.494 ns
num_events = 347
mean = -1.705 ns
rms = 3.087 ns
mean_error = 0.166 ns
centroid = -1.706 ns.
width (sigma) = 0.363 ns.
error on the centroid = 0.016558 ns.
</pre>
* ls -l *.txt *.pdf
<pre>
dsdaq@dsvslice:~/online/PhaseMeasurement$ ls -l *.txt *.pdf
-rw-rw-r-- 1 dsdaq dsdaq 64728 Dec 14 16:56 run00877.mid.lz4_Plots.pdf
-rw-rw-r-- 1 dsdaq dsdaq 274 Dec 14 16:56 run00877.mid.lz4.txt
dsdaq@dsvslice:~/online/PhaseMeasurement$
</pre>
* scope settings (from email message)
<pre>
From fcote-lortie@triumf.ca Thu Dec 15 17:11:46 2022
From: Francis Cote-Lortie <fcote-lortie@triumf.ca>
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: Re: How to use scope as a waveform generator
Date: Fri, 16 Dec 2022 01:11:44 +0000

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display) by pressing on the button. It will go from 0 to 1.

The settings that we are using right now are:
Type of waveform: Sine wave
Offset: 0 V
Amplitude: 1 Vpp
Frequency: 50 kHz
Noise: 0 V
________________________________
From: Francis Cote-Lortie <fcote-lortie@triumf.ca>
Sent: Thursday, December 15, 2022 4:58 PM
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: Re: How to use scope as a waveform generator

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display)

The settings that we are using right now are:
Type of waveform: Sine wave
Offset: 0 V
Amplitude: 1 Vpp
Frequency: 50 kHz
Noise: 0 V
________________________________
From: Francis Cote-Lortie
Sent: Thursday, December 15, 2022 4:54 PM
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: How to use scope as a waveform generator

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display)
</pre>

== Standalone link test ==

<pre>
CDM: program clock chip
busybox devmem 0x80011000 32 0x8
busybox devmem 0x80011000 32 0x0
/home/dsdaq/si5394-i2c-file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b

GDM, CDM: link reset
busybox devmem 0x80011008 32 1

GDM, CDM: release reset
busybox devmem 0x80011008 32 0

CDM: link status (NOTE: SFP LOS and mod_absent are swapped!!!)
busybox devmem 0x80011010 32
0x00000024 <- fiber plugged
0x00000025 <- fiber unplugged
0x00000027 <- SFP unplugged
0x000007DC <- successful link with GDM

CDM: link state machine and data
busybox devmem 0x80011014 32
0x35C06FF6

CDM: set link to counting mode
busybox devmem 0x80012000 32 0x101
busybox devmem 0x80012000 32 0x100

CDM: time counter and error counter
root@cdm1:~# busybox devmem 0x80012000 32
0x00000100 <--- link mode
root@cdm1:~# busybox devmem 0x80012004 32
0x0000058C <--- seconds counter
root@cdm1:~# busybox devmem 0x80012004 32
0x0000058D
root@cdm1:~# busybox devmem 0x80012008 32
0x00000000 <--- error counter

GDM: no link
root@gdm0:~# busybox devmem 0x80011014 32
0x00000000
root@gdm0:~# busybox devmem 0x80011018 32
0x00000000
root@gdm0:~# busybox devmem 0x8001101c 32
0x00000000
root@gdm0:~# busybox devmem 0x80011024 32
0x00000FFF
root@gdm0:~#

GDM: good link channel 10, counting mode
root@gdm0:~# busybox devmem 0x80012000 32 0x101
root@gdm0:~# busybox devmem 0x80012000 32 0x100
root@gdm0:~# busybox devmem 0x80012008 32
0x3A8B68C2
root@gdm0:~# busybox devmem 0x80012008 32
0x42E03BEF
root@gdm0:~# busybox devmem 0x8001200c 32
0xDA090972
root@gdm0:~# busybox devmem 0x8001200c 32
0xDE6F22E9
root@gdm0:~# busybox devmem 0x80012019 32
Bus error
root@gdm0:~# busybox devmem 0x80012010 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012014 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012018 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x8001201c 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012020 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012024 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012028 32
0x00000000
root@gdm0:~# busybox devmem 0x8001202c 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012030 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012034 32
0xFFFFFFFF
root@gdm0:~#
</pre>

mapping of link channels:

<pre>
qsfp0 lane0 - 0x0100 - link 8
qsfp0 lane1 - 0x0200 - link 9
qsfp0 lane2 - 0x0400 - link 10
qsfp0 lane3 - n/c
qsfp1 lane0 - 0x0800 - link 11
qsfp1 lane1 - 0x0010 - link 4
qsfp1 lane2 - 0x0020 - link 5
qsfp1 lane3 - n/c
qsfp2 lane0 - 0x0040 - link 6
qsfp2 lane1 - 0x0080 - link 7
qsfp2 lane2 - 0x0001 - link 0
qsfp2 lane3 - n/c
qsfp3 lane0 - 0x0002 - link 1
qsfp3 lane1 - 0x0004 - link 2
qsfp3 lane2 - 0x0008 - link 3
qsfp3 lane3 - n/c
</pre>

script to start the test with 2 CDMs:

<pre>
ssh dsdaq@dsvslice
ssh root@gdm0 busybox devmem 0x80011008 32 1
ssh root@cdm0 busybox devmem 0x80011008 32 1
ssh root@cdm1 busybox devmem 0x80011008 32 1
ssh root@gdm0 busybox devmem 0x80011008 32 0
ssh root@cdm1 busybox devmem 0x80011008 32 0
ssh root@cdm0 busybox devmem 0x80011008 32 0
ssh root@gdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
ssh root@cdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
ssh root@cdm1 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
</pre>

== LEMO trigger GDM to CDM to VX ==

on the GDM:

<pre>
/home/dsdaq/si5394-i2c-file /home/dsdaq/GDM_v1.0_IN0_EXT1_and_IN1_fixed_Si5394-RevA-Registers.txt 0 0x6b
busybox devmem 0x80011008 32 0x1
busybox devmem 0x80011008 32 0x0
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x80001230 # QSFP TX fixed pattern
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 2 0xff # enable LEMO inputs
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 3 0x7654 # enable LED, one per LEMO input
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x0F0F # enable LEMO to trg_in and tsm_in
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x40001230 # enable trg_in and tsm_in output to QSFP
</pre>

on the CDM:

<pre>
/home/dsdaq/si5394-i2c-file /home/dsdaq/CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
/home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp
busybox devmem 0x80011008 32 1
busybox devmem 0x80011008 32 0
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x8040 # trg_in from sfp[0], tsm_in form sfp[1]
/home/dsdaq/online/ds-dm-software/test_cdm.exe 11 # trg_in counter
/home/dsdaq/online/ds-dm-software/test_cdm.exe 12 # tsm_in counter
</pre>

== GPS receiver VCL-2705 ==

* Valiant VCL-2705 GPS receiver
* https://www.valiantcom.com/time-distribution/gps-receiver-irig-b.html
* USB connection is /dev/ttyACM0, 115200 bps, "GNSSAUX" prompt
* minicom -D /dev/ttyACM0 -b 115200
* user manual download instructoins - see sheet of paper with user name and password in the shipping box
* usb commands:
<pre>
gnss-help
gnss-showver -> F/W: Ver 1.7 Feb 17 2020 16:20:43

gnss-showselftest -> no antenna connected
Overall : FAIL
EPROM Test: PASS
Antenna : NOT DETECTED
GNSS : COMMUNICATION OK

gnss-showsettings
GNSS NMEA BAUDRATE :115200
GNSS ANTENNA LENGTH :30 meters
GNSS USER CONFIGURED DELAY:-65 nanoseconds
GNSS 1PPS PULSE WIDTH :200 milliseconds
GNSS MODE :GPS
GNSS STATUS :STATIONARY

GNSSAUX> gnss-showserial
SERIAL :2704H01V17MAX310

GNSSAUX> gnss-showmode
GNSS MODE : GPS

GNSSAUX> gnss-showstatus
GNSS STATUS: STATIONARY

GNSSAUX> gnss-showalarms
CURRENT ALARMS GNSS
ANTENNA : NOT DETECTED

GNSSAUX> gnss-showerrors
CURRENT ALARMS GNSS
ANTENNA : **NOT DETECTED
ERROR STATISTICS GNSS
RMC GOOD DURATION : Secs 0
RMC BAD DURATION : Secs 0
LOCK GOOD SECS DURATION : Secs 0
LOCK BAD SECS DURATION : Secs 0
SATINFO GOOD ITERATIONS :0
SATINFO REJECT ITERATIONS :0
SATINFO NOTALKER ITERATIONS :0
CURRENT MONITOR STATE GNSS
STATE :Phase-1 HUNTING ANTENNA DETECT

GNSSAUX> gnss-showsatinfo
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-showsats
Total Sats: 0

GNSSAUX> gnss-showmyloc
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-show1ppsstate
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-showjamstatus
Not Available !

GNSSAUX> gnss-showspoofstatus
Not Available !

--- antenna connected, can see the sky ---

GNSSAUX> gnss-showselftest
Overall : PASS
EPROM Test: PASS
Antenna : DETECTED
GNSS : COMMUNICATION OK

GNSSAUX> gnss-showalarms
CURRENT ALARMS GNSS
ANTENNA : DETECTED
GNSS LOCK : AVAILABLE

GNSSAUX> gnss-showerrors
CURRENT ALARMS GNSS
ANTENNA : DETECTED
GNSS LOCK : AVAILABLE
ERROR STATISTICS GNSS
RMC GOOD DURATION : Mins 1,Secs 34
RMC BAD DURATION : Secs 55
LOCK GOOD SECS DURATION : Mins 1,Secs 34
LOCK BAD SECS DURATION : Secs 55
SATINFO GOOD ITERATIONS :3
SATINFO REJECT ITERATIONS :0
SATINFO NOTALKER ITERATIONS :0
CURRENT MONITOR STATE GNSS
STATE :Phase-4 NORMAL OPERATION, Monitoring GNSSLOCK

GNSSAUX> gnss-showsatinfo
GNSS MODE :GPS
SATELLITE INFORMATION TALKER:GPS
NMEA ID : 01-32
NO OF SATELLITES IN VIEW: 08
NO OF XXGSV MSGS : 03
SatNo PRN NO (SV ID) ELEVATION (degs) AZIMUTH (degs) C/No (SNR)
1 0 0 0 0
2 0 0 0 0
3 0 0 0 0
4 0 0 0 0
5 0 0 0 0
6 0 0 0 0
7 0 0 0 0
8 0 0 0 0
<<<<<< End of Sat Info >>>>>>

GNSSAUX> gnss-showsats
Total Sats: 8

GNSSAUX> gnss-showmyloc
GNSS RECEIVER LOCATION:
Latitude : 4914.81911
Longitude: 12313.69595

GNSSAUX> gnss-show1ppsstate
GPS 1PPS STATE: LOCKED

GNSSAUX> gnss-showmyloc

GNSS RECEIVER LOCATION:
Latitude : 4914.80688
Longitude: 12313.69531

enter into google maps search box as: 49 14.80688, -123 13.69531, observe the space, the moved dot and the minus.

GNSSAUX> gnss-resetgnss

Executing....Please Wait....
$$$$$END
</pre>

* 1PPS BNC output: period 1 sec, pulse width 200 ms, 3.3V into 1MOhm, 1.38V into 50Ohm.
* IRIG-B BNC output: 5.6V into 1MOhm, 2.4V into 50Ohm.
* IRIG-B format selector: default is all up.

== Rb clock PRS10 ==

* https://www.thinksrs.com/products/prs10.html
* 10 MHz output is sine wave around 5V peak to peak
* 1PPS BNC output is 10 usec pulse, 5V into 1MHohm.
* RS232 connection: minicom -D /dev/ttyUSB0 -b 9600
* ser2net config: localhost,3001:raw:600:usb-5-2-1.0:9600 -XONXOFF -RTSCTS LOCAL
* ssh daq13, cd /home/olchansk/git/ser2net, ./ser2net -c ~/daq/ds/ser2net.conf -d
* ssh daq13, cd ~/daq/ds, python3 prs10.py
* RS232 commands:
<pre>
ID?
PRS10_3.56_SN_105719
VB1
SN?
RS1 -- reset
ST? -- status
FC? -- 10MHz OCXO drive voltage DAC settings
DS? -- "detected signals"
GA? -- gain of frequency lock loop between ovenized oscillator and Rb cell, 0=use ovenized oscillator only
MO? -- magnetic offset of the Rb cell, range 2300..3600, if out of range, unit must be set to different operating mode, see prs10m.pdf
MR? -- magnetic read
TT? -- time-tag, time in ns between 1PPS out and 1PPS in
TS? -- time slope, ???
TO? -- time offset, ???
PS? -- pulse slope, ???
PL? -- 0=phase lock off, 1=phase lock on, lock to 1PPS input
PT? -- phase lock integrator time constant, PT8 is integrator time constant 18.2 hours, natural time constant 2.25 hours
PF? -- phase lock stability factor, PF2 is "1"
PI? -- phase lock integrator

Analog to digital 12 bit ADC, values 0.000 to 4.998

AD0? -- Spare (J204)
AD1? -- +24V(heater supply) divided by 10.
AD2? -- +24V(electronics supply) divided by 10
AD3? -- Drain voltage to lamp FET divided by 10
AD4? -- Gate voltage to lamp FET divided by 10
AD5? -- Crystal heater control voltage
AD6? -- Resonance cell heater control voltage
AD7? -- Discharge lamp heater control voltage
AD8? -- Amplified ac photosignal
AD9? -- Photocell’s I/V converter voltage divided by 4
AD10? -- Case temperature (10 mV/°C)
AD11? -- Crystal thermistors
AD12? -- Cell thermistors
AD13? -- Lamp thermistors
AD14? -- Frequency calibration pot / external calibration voltage
AD15? -- Analog ground

A/D via CPU E-port:

AD16? -- Varactor voltage for 22.48 MHz VCXO (inside RF synthesizer) / 4
AD17? -- Varactor voltage for 360 MHz VCO (output of RF synthesizer) / 4
AD18? -- Gain control voltage for amplifier which drives frequency multiplier / 4
AD19? -- RF synthesizer’s lock indicator voltage (nominally 4.8 V when locked )

ST?

ST1 : Power supplies and Discharge Lamp
ST1 bit, Condition which sets bit, Corrective Action
0 -- +24 for electronic < +22 Vdc
1 -- +24 for electronics > +30 Vdc
2 -- +24 for heaters <+22 Vdc
3 -- +24 for heaters > +30 Vdc
4 -- Lamp light level too low
5 -- Lamp light level too high
6 -- Gate voltage too low
7 -- Gate voltage too high

ST2: RF Synthesizer
ST2 bit, Condition which sets bit, Corrective Action
0 -- RF synthesizer PLL unlocked
1 -- RF crystal varactor too low
2 -- RF crystal varactor too high
3 -- RF VCO control too low
4 -- RF VCO control too high
5 -- RF AGC control too low
6 -- RF AGC control too high
7 -- Bad PLL parameter

ST3: Temperature Controllers
ST3 bit, Condition which sets bit
0 -- Lamp temp below set point
1 -- Lamp temp above set point
2 -- Crystal temp below set point
3 -- Crystal temp above set point
4 -- Cell temp below set point
5 -- Cell temp above set point
6 -- Case temperature too low
7 -- Case temperature too high

ST4: Frequency Lock-Loop Control
ST4 bit, Condition which sets bit
0 -- Frequency lock control is off
1 -- Frequency lock is disabled
2 -- 10 MHz EFC is too high
3 -- 10 MHz EFC is too low
4 -- Analog cal voltage > 4.9 V
5 -- Analog cal voltage < 0.1
6 -- not used
7 -- not used

ST5: Frequency Lock to External 1pps
ST5 bit, Condition which sets bit
0 -- PLL disabled
1 -- < 256 good 1pps inputs
2 -- PLL active
3 -- > 256 bad 1pps inputs
4 -- Excessive time interval
5 -- PLL restarted
6 -- f control saturated
7 -- No 1pps input

ST6: System Level Events
ST6 bit and Condition which sets bit
0 Lamp restart
1 Watchdog time-out and reset
2 Bad interrupt vector
3 EEPROM write failure
4 EEPROM data corruption
5 Bad command syntax
6 Bad command parameter
7 Unit has been reset
</pre>
* ST? on warm start
<pre>
received: PRS_10
received: 255,255,255,243,34,255
received: 0,0,0,1,34,0
...
received: 0,0,0,1,34,0
received: 0,0,0,0,2,0
...
received: 0,0,0,0,2,0
received: 0,0,0,0,4,0
</pre>
* ST? on loss of external 1PPS
<pre>
...
received: 0,0,0,0,4,0
disconnect 1PPS input
received: 0,0,0,0,132,0
...
reconnect 1PPS input
received: 0,0,0,0,132,0
received: 0,0,0,0,4,0
...
</pre>
* ST? on coldish start
<pre>
daq13:ds$ python3 prs10.py
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,0,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 0,0,1,1,34,0 ] old [ 80,0,0,1,34,1 ] counter: 0
received [ 0,0,0,1,34,0 ] old [ 0,0,1,1,34,0 ] counter: 0
received [ 0,0,16,1,34,0 ] old [ 0,0,0,1,34,0 ] counter: 6
received [ 0,0,20,1,34,0 ] old [ 0,0,16,1,34,0 ] counter: 5
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 8
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 3
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 49
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 250
...
</pre>
* ST? on cold start, note: no bump in the 10MHz clock as reported by DS-DM clock chip
<pre>
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,21,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 64,0,21,1,34,0 ] old [ 80,0,21,1,34,1 ] counter: 0
received [ 0,0,21,1,34,0 ] old [ 64,0,21,1,34,0 ] counter: 0
received [ 0,0,20,1,34,0 ] old [ 0,0,21,1,34,0 ] counter: 158
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 23
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 47
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 40
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 249
</pre>

= det fac integration test =

Connections:
* GPS receiver "IRIG-B SEL" both switches "up" - both "on", IRIG-B format B004
* GPS receiver USB-B -> long cable -> daq13 USB-A
* GPS receiver "1PPS out" -> long BNC cable -> BNC-T -> scope (5V, no 50ohm) and Rb clock BNC "1PPS in"
* GPS receiver "IRIG-B 50ohms" -> long BNC cable -> BNC-T -> scope (5V, no 50ohm) and DS-DM LEMO input 1 (TTL mode)
* Rb clock RS232 -> RS232 straight cable -> RS232-to-USB adapter -> daq13 USB-A
* Rb clock "1PPS out" BNC -> scope (5V, no 50ohm, trig threshold rising edge 2V)
* Rb clock "10MHz output 50 Ohm" BNC -> lemo -> lemo-T -> scope (sine wave, 5V, no 50ohm) and DS-DM clock input.

Programs to run:
* on daq13: cd /home/olchansk/git/ser2net, ./ser2net -c ~/daq/ds/ser2net.conf -d
<pre>
localhost,3001:raw:600:usb-5-2-1.0:9600 -XONXOFF -RTSCTS LOCAL
</pre>
* on daq13: cd /home/olchansk/daq/ds, python3 prs10.py ### connects to ser2net
<pre>
daq13:ds$ python3 prs10.py
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,21,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 64,0,21,1,34,0 ] old [ 80,0,21,1,34,1 ] counter: 0
received [ 0,0,21,1,34,0 ] old [ 64,0,21,1,34,0 ] counter: 0
received [ 0,0,20,1,34,0 ] old [ 0,0,21,1,34,0 ] counter: 158
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 23
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 47
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 40
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 249
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 12096
received [ 0,0,0,0,20,0 ] old [ 0,0,0,0,132,0 ] counter: 2
received [ 0,0,0,0,148,0 ] old [ 0,0,0,0,20,0 ] counter: 0
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,148,0 ] counter: 575
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 11449
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 2755
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 34386
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 41035
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 113401
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 33375
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 54767
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 85059
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 33222
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 119234
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 121990
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 128184
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 56002
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 428237
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 8250
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 1388
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 30506
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 142704
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 179451
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 106182
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 68747
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 65424
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 157587
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 6932
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 1388
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 20255
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 4
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 225941
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 72183
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 26970
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 1
39287
</pre>
* on ds-dm: ./test_cdm.exe --irigb ### note sbs mismatch is because I should wrap around at 16 bits
<pre>
dataframe: S00000000S100001010S...S001010101S000001000S, sec: 00, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16724 should be 82260
dataframe: S10000000S100001010S...S101010101S000001000S, sec: 01, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16725 should be 82261
dataframe: S01000000S100001010S...S011010101S000001000S, sec: 02, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16726 should be 82262
dataframe: S11000000S100001010S...S111010101S000001000S, sec: 03, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16727 should be 82263
dataframe: S00100000S100001010S...S000110101S000001000S, sec: 04, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16728 should be 82264
dataframe: S10100000S100001010S...S100110101S000001000S, sec: 05, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16729 should be 82265
</pre>

= DS-IOGC GPS interface board =

* Rev0 git repository: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0/-/tree/main?ref_type=heads
* Rev0 schematics: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0/-/blob/main/Altium/Project%20Outputs%20for%20DS-IOGC-Rev0/SCH-DS-IOGC-Rev0.pdf?ref_type=heads
* Rev0 schematics: [[:Image:SCH-DS-IOGC-Rev0.pdf|SCH-DS-IOGC-Rev0]]
* Rev1 git repository: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev1/
* Rev1 schematics: [[:Image:SCH-DS-IOGC-Rev1.pdf|SCH-DS-IOGC-Rev1]]

== Changes Rev0 to Rev1 ==

<pre>
From: Peter Margetak <pmargetak@triumf.ca>
Subject: IOGC REV1 review
Date: Wed, 4 Sep 2024 07:31:19 +0000

Hi Konstantin,
Pls have a look at SCH for new rev. I'd like to send it to mfr next week so if you can comment by early next week. Meanwhile I work on layout and other stuff.

Changes:

New ICs - all powered +5V
U20 - inverters for RUclk RX/TX
U21 - non inverting line driver for RU-1pps-out (so you don't have to route if via GDM to see it on scope)
U22 - non inverting buffer for ext 1pps input

All Lemo connectors have the same position but they are double lemos now => new panel needed
@Marek Walczak<mailto:mwalczak@triumf.ca> you can print it ahead once pcb is done + update IOGC docs and panel description

J2A/B - Test ports for RU-1pps in and out
J5A/B - inputs for external GPS data and External source of 1pps
J6A/B - aux in/out for GDM

SW1 - no change - select RX/TX USB/GDM
SW2 - select latch sensitivity for rising/falling edge
SW3 - select source of GPS data (opto or ext) AND select source of 1pps input (latch or ext)

p.
</pre>

== PRS-10 Rb clock device ==

The Rb clock PRS-10 device provides these connections:
<pre>
RS232 RX input - serial communication, non-standard RS232
RS232 TX output - serial communication, non-standard RS232
10 MHz clock output - coax 50 Ohm high resolution 10 MHz clock
1pps output - 1 Hz clock corresponding to the 10 MHz clock
1pps input - 1pps signal from GPS receiver
</pre>

Mode of operation:
* 10 MHz clock is always running
* 1pps output is always running
* if 1pps input from GPS received is present, after 256 pulses PRS-10 will sync it's 1pps output with the 1pps input by adjusting the frequency of the 10 MHz clock
* when unlocked: 1pps output and 1pps input unrelated
* when locked to GPS: 1pps output and 1pps input always go up and down at the same time

Theory of operation:
* 10 MHz clock is produced by a high-quality crystal (stable on the scale of seconds)
* crystal oscillator is synchronized to a Rb cell (stable on the scale of hours and days)
* Rb cell resonant frequency is synchronized to the GPS 1pps signal (stable on the scale of months and years)

== Rev1 connections ==

* LEMO connectors (front panel)
<pre>
LEMO J2A output - Rb clock 1pps in monitor
LEMO J2B output - Rb clock 1pps out monitor
LEMO J5A input - GPS IRIG-B from GPS receiver to FPGA (VCL-2705)
LEMO J5B input - GPS 1pps from GPS receiver to PRS-10 (VCL-2705)
LEMO J6A output - AUX-OUT from FPGA (dual-LEMO PCB-side)
LEMO J6B input - AUX-IN to FPGA (dual-LEMO away-from -PCB)
</pre>

* SMB connectors (back)
<pre>
SMB J3 output - GPS 1pps loopback to LNGS
SMB J4 input - LNGS GPS data input
</pre>

* LEDs
<pre>
D1 - same as LEMO J2A out (Rb clock 1pps in)
D2 - same as SMB J3 out (GPS 1pps from LNGS or from a GPS receiver)
D5 - controlled by FPGA-OUT-LED1
D6 - controlled by FPGA-OUT-LED2
D7 - PRS-10 24V power ok
</pre>

* switches
<pre>
SW1A and SW1B - route PRS-10 RS232 to USB or to FPGA
SW2A - route PRS-10 1pps input from SMB J4 (LNGS) or from LEMO J5B (GPS receiver 1pps)
SW2B - route FPGA-IN-GPSDATA input from SMB J4 (LNGS) or from LEMO J5A (GPS receiver IRIG-B data)
SW3 - LNGS 1pps from rising edge or from falling edge of SMB J4 (LNGS)
</pre>

== Rb clock cable ==
<pre>
Rb clock ----- DS-IOGC side, pin numbers are as labeled on the cable connectors

1 - 1pps out - 8 - 1pps out --- correct
2 - nc
3 - nc
4 - TXD - 3 - RU-DATA-OUT - USB-RX input --- correct
5 - 1pps in - 2 - 1pps in --- correct
6 - +24V - 10 - +24V --- should by pin 1 to use both +24V pins?
7 - RXD - 15 - RU-DATA-IN - USB-TX output --- correct
8 - nc
9 - +24V - 10 - +24V --- correct
10 - GND - 9 - GND --- correct
</pre>

== VX connections ==
<pre>
VXA_TX0 - FPGA-OUT-LED2 - D6 LED ("10 MHz clock")
VXA_TX1 - FPGA-OUT-LED1 - D5 LED ("GPS DATA")
VXA_TX2 - FPGA-OUT-RU1PPS-EN - enable 1pps to Rb clock
VXA_TX3 - not used (62.5 Hz clock)

VXB_TX0 - FPGA-TX - PRS-10 RS-232 out
VXB_TX1 - FPGA-OUT-AUX - 5V TTL J5 LEMO out
VXB_TX2 - FPGA-OUT-OPTO1PPS-EN - PRS-10 power enable
VXB_TX3 - not used (62.5 MHz clock)

VXA_RX0 - FPGA-IN-OPTO1PPS - 1pps from GPS
VXA_RX1 - FPGA-IN-GPSDATA - J4 GPS data
VXA_RX2 - n/c
VXA_RX3 - n/c

VXB_RX0 - n/c
VXB_RX1 - FPGA-IN-AUX - J5 LEMO TTL input
VXB_RX2 - FPGA-RX - PRS-10 RS-232 in
VXB_RX3 - FPGA-IN_RU1PPS - PRS-10 1pps output
</pre>

== test sequence ==

* ./test_cdm_local.exe --writereg 7 0 ### clear reg 7
* ./test_cdm_local.exe --writereg 8 0 ### set vx_tx mux to vx_tx control from reg 7
* connect blue cable to GDM port 6 (next to the ethernet connector)
* ./test_cdm_local.exe --writereg 7 0x4000 ### power up
* test LEDs:
* ./test_cdm_local.exe --writereg 7 0x4100 ### right LED2 D6
* ./test_cdm_local.exe --writereg 7 0x4200 ### left LED1 D5
* test AUX-IN and AUX-OUT:
* ./test_cdm_local.exe --writereg 7 0x6000 ### J6A LEMO (PCB-side) measure +5VDC
* install LEMO jumper between LEMO J6A and J6B
* write 0x4000 then ./test_cdm_local.exe 6 ### reads 0x5C, bit 5 0x20 reads 0
* write 0x6000 then ./test_cdm_local.exe 6 ### reads 0x7C, bit 5 0x20 reads 1
* write 0x4000 to clear all bits
* test the 10 MHz clock:
* init the c.c: ./test_cdm_local.exe --load-cc
* connect 10 MHz clock output to DS-DM LEMO J6-LEFT (CLK_EXT1)
* 10 MHz output ok - IN0 is good: ./test_cdm_local.exe --cc
<pre>
Clock chip state 1, status: OOF_IN2 IN1 IN_SEL_1 HOLD_HIST_VALID
</pre>
* without 10 MHz reports IN0 LOS (no signal) and OOF (wrong frequency)
<pre>
Clock chip state 1, status: LOS_IN0 OOF_IN0 OOF_IN2 IN1 IN_SEL_1 HOLD_HIST_VALID
</pre>
* test GPS IRIG-B data:
* connect GPS IRIG-B signal to J6B (dual-LEMO away from PCB)
* IRIG-B via AUX-IN is ok: ./test_cdm_local.exe --irigb
<pre>
dataframe: S00100110S111001000S...S011110000S010000000S, sec: 34, min: 17, hrs: 00, day of year: 223, year: 2024, date: 10 aug, sbs: 1054 should be 1054
</pre>
* test IRIG-B via J5A GPS-EXT:
* connect GPS IRIG-B signal to J5A (dual-LEMO next to PCB)
* set SW2B: switch on the side of SMB connectors, slide direction of 24V power supply connector
* ./test_cdm_local.exe --irigb built for VXA_RX1 should work
* test GPS 1pps signal:
* connect GPS 1pps to J5B (dual-LEMO away from PCB)
* nothing happens, no output from U22
* remove R51
* set SW2A: switch on the side away from SMB connectors, slide direction of 24V power supply connector
* observe LEDs D1 and D2 blink at 1Hz
* SKIP THIS connect GPS 1pps to J4 using LEMO-SMB adapter
* SKIP THIS nothing happens, nLE is always 3.2V, both settings of SW2
* SKIP THIS !!!failed here!!!
* ./test_cdm_local.exe --writereg 8 9 ### set vx_tx mux to GPS control, hard enables PRS-10 power and PRS-10 1pps in
* observe 1pps counters in reg 68 count at 1 Hz: rb_1pps 0x51->0x52, gps_1pps 0xd5->oxd6
<pre>
root@dsdm:~# ./test_cdm_local.exe 68
ds20k_reg[68] is 0x0051d520 (5362976)
ds20k_reg[68] is 0x0052d624 (5428772)
...
</pre>
* observe PRS-10 can see the 1pps signal "130" changes to "2" after 243 seconds to "4"
* observe period of GPS and PRS-10 1pps is identical (plus/minus 1 clock)
<pre>
root@dsdm:~# ./test_cdm_local.exe 13 14
ds20k_reg[13] is 0x077356d4 (124999380)
ds20k_reg[14] is 0x077356d4 (124999380)
</pre>
* observe GDM/CDM clocks are exactly 125 MHz, we are running on DS-DM internal oscillator
<pre>
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735943 (125000003) should be ~125 MHz
0x1034 rx_clk: 0x07735943 (125000003) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735943 (125000003) should be ~125 MHz
0x103C tx_clk: 0x07735943 (125000003) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>
* if tx_clk is missing, reset the mgt: ./test_cdm_local.exe --reset-mgt
* switch CC to external clock:
<pre>
root@dsdm:~# ./test_cdm_local.exe --cc-in0
CC use clock input 0: 10 MHz LEMO external clock
root@dsdm:~# ./test_cdm_local.exe --cc
Polling CC status...
Clock chip state 1, status: IN0 IN_SEL_REGCTRL IN_SEL_0 HOLD_HIST_VALID
</pre>
* observe rx_clk and tx_clk are now slightly off: they run from PRS-10 10 MHz clock and are measured against the DS-DM internal oscillator.
<pre>
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x94b12519, ds20k version 0x20240814
DS-DM firmware build 0x94b12519, ds20k version 0x20240814
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x077358e1 (124999905) should be ~125 MHz
0x1034 rx_clk: 0x07735b49 (125000521) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735b49 (125000521) should be ~125 MHz
0x103C tx_clk: 0x07735b49 (125000521) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>
* observe GPS and PRS-10 1pps period is now exactly 125*10^6 of 8ns clocks (equal to 1 second)
<pre>
root@dsdm:~# ./test_cdm_local.exe 13 14
ds20k_reg[13] is 0x0773593f (124999999)
ds20k_reg[14] is 0x0773593f (124999999)
</pre>
* look at them repeatedly, observe reg 13 "GPS 1pps period" has some wobble, reg 14 "Rb clock 1pps period" is steady. this is as expected: if GPS 1pps was steady, we do not need to Rb clock. PRS-10 1pps is derived from the PRS-10 10 MHz clock and is measured against the 125 MHz clock derived from the same 10 MHz clock in the GDM FPGA.

Rev0 Test status:

* GPS 1pps to SMB-in ok (LED flashes)
* GPS 1pps to FPGA ok
* GPS data to FPGA ok
* GPS 1pps to PRS-10 enabled from FPGA ok
* PRS-10 Rb clock 1pps out to FPGA ok
* PRS-10 Rb clock 1pps out to FPGA to LEMO AUX out ok
* can see GPS 1pps, IRIG-B, PRS-10 1pps out, 10 MHz on the scope, ok
* PRS-10 syncs on leading edge (0->1) of GPS 1pps signal, ok
* reg 13 and 14 1pps periods are identical, ok
* CC locks on PRS-10 10 MHz clock, DS-DM runs on PRS-10 clock, ok
* NOT TESTED - smb output
* NOT TESTED - optical converter fiber to SMB
* NOT TESTED - optical converter SMB to fiber
* NOT TESTED - SMB loopback
* NOT TESTED - fiber loopback

Rev1 Test status: (21 Nov 2024)

* power up ok
* LEDs ok
* LEMO AUX-IN, AUX-out ok
* GPS IRIG-B to FPGA via J6B AUX-IN ok
* GPS IRIG-B to FPGA via J5A GPS-EXT ok
* PRS-10 Rb clock 1pps out to FPGA ok
* PRS-10 Rb clock 10 MHz to C.C. ok
* PRS-10 RS-232 ok
* J5B 1pps input fail. input signal is 3.3V without termination, 1.5V when plugged into IOGC (on R51), no output from U22.

= VX busy logic =

= DS-20K DAQ =

== Overview ==

DS-DM, GDM and CDM are key parts of the DS-20K DAQ system:
* common clock distribution from external clock (atomic clock, GPS) to GDM to per-quadrant CDMs to VX digitizers
* common trigger distribution from GDM internal algorithm or external input to all VX digitizers
* run control: GDM, CDM, VX all start recording data at the same time (clock and timestamp reset)
* collection of trigger data from VX digitizers to per-quadrant CDMs to GDM

== Deliverables ==

* hardware and firmware for GDM to CDM clock distribution
* hardware and firmware for CDM to VX clock distribution
* hardware and firmware for GDM external clock input (atomic clock or GPS)
* hardware and firmware for CDM and VX serial communications (VX LVDS I/O connector)
* firmware for run control (timestamp reset and sync): GDM to CDM to VX
* firmware for common trigger distribution: GDM to CDM to VX
* firmware for trigger data flow: VX to CDM to GDM
* firmware for busy control: VX to CDM to GDM back to CDM to VX
* firmware for flow control: FEP to GDM MIDAS frontend to GDM to CDM to VX
* GDM MIDAS frontend: clock selector and monitoring, trigger and run control, busy and flow control, GDM housekeeping
* CDM MIDAS frontend: clock monitoring, CDM housekeeping

specific performance:
* GDM external clock: 10 MHz GPS clock
* GDM to CDM fiber link:
** clock XXX MHz
** link data rate: XXX Gbit/sec
** CDM recovered clock: XXX MHz
** CDM recovered clock jitter: XXX ns
** phase alignment between CDMs: XXX ns
** phase alignment between CDMs persists across reboots, power cycles, firmware updates
** phase alignment between CDMs should be easy to measure
** phase alignment between CDMs should be easy to recalibrate if hardware parts are replaced (DS-DM boards, fiber transceivers, fiber cables, etc)
** data packet bandwidth: XXX Mbytes/sec
** data packet latency: XXX clocks
** data packet skew between CDMs: XXX clocks
* CDM to VX clock:
** clock: XXX MHz
** jitter, all CDM clock outputs: XXX MHz
** phase alignment between all CDM clock outputs: XXX ns
* CDM to VX trigger:
** TBD (use the VX "sync" input or VX LVDS I/O line or VX serial link packet)
* CDM to VX serial link:
** clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
** bit rate: XXX bits/sec
** latency: XXX link clocks
** maximum skew between VXes: XXX ns
* VX to CDM serial link:
** clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
** bit rate: XXX bits/sec
** latency: XXX link clocks
** maximum skew between VXes: XXX ns
* timestamp reset:
** maximum skew between VXes: XXX ns
* busy round trip time: XXX ns (VX to CDM to GDM back to CDM to VX)
* flow control latency: XXX ns (FEP software to GDM MIDAS frontend to GDM to CDM to VX)

== Technical risk items ==

this refers to unexpected behaviour and performance of
system components, causes big difficulty in implementing the system,
prevents delivery of deliverables, and prevents or negatively affects operation
of the DS-20K DAQ or of the whole experiment.

(14-sep-2022, list is not sorted by any criterial: severity, probability, ease of investigation)

(stability of course is long term stability, across hours, days, weeks, months, years)

* stability of Enclustra FPGA modules (crashes/year, failures to boot/year, flash corruption/year)
* stability of GDM external clock PLL (lock loss/year)
* stability of CDM recovered clock (lock loss/year, unexpected phase drifts, etc)
* unexpected failures or bit error rates in GDM-CDM fiber links
* stability of CDM VX clock outputs (stability of clock cleaner chip)
* stability of VX internal clock distribution (VX PLL lock loss events)
* stability of VX CAEN base firmware (different versions of CAEN base firmware have different clock distribution behaviour)
* strange things in CAEN base firmware (unexpected clocking of LVDS I/O, unexpected phase shifts between clocks, etc)
* DS-DM and VX hardware problems (incompatible LVSD I/O, incompatible clock signals, etc)

== Milestones ==

(14-sep-2022: at this stage of the project, priority must be given to identifying and retiring (so called) technical risk factors. it is not good to build the complete system only to discover that (for example) some Enclustra FPGA modules require 5 attempts to boot and erase their flash memory contents once a month. Both example are real-life actual problems that caused big difficulties in GRIFFIN/TIGRESS and ALPHA-g experiments).

Development and testing milestones in time reversed order:

* full DAQ data challenge: all VXes, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
* one quadrant data challenge: 1 VME crate of VX, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
* vertical slice data challenge: 1 VME crate, 2 VX, 2 CDM (1 VX per CDM), GDM, etc operate as designed
* GDM-CDM link finalized (data rate frozen, data packet format frozen, data content permitted to change)
* CDM-VX serial link finalized (data rate frozen, data packet format frozen, data content permitted to change)
* run control (timestamp reset) and trigger distribution design agreed upon, frozen (list of possible triggers permitted to change)
* VX to CDM to GDM data flow design agreed upon, frozen (data contents permitted to change)
* major technical risk items retired (all hardware and firmware is working as expecred without mysteries and surprises, all problems are identified, investigated, resolved, solutions tested)
* stable operation of CDM-VX serial links in vertical slice system
* stable operation of GDM to CDM clock in vertical slice system
* stable operation of CDM to VX clock in vertical slice system
* vertical slice system assembled (1 VME crate, 2 VX, 2 CDM, 1 GDM, network, FEP, TSP, MIDAS)

= ZZZ =

ZZZ

DS-DM

2026-03-28T00:56:07Z

Olchansk: /* Clock distribution Rev1 */

= DS-DM =

DarkSide-20k Global and Crate Data Manager board (GDM and CDM).

Global Data Manager (GDM):
* clock distribution to CDM boards (including GPS/atomic clock source)
* collection of trigger data from CDM boards, processing and distribution of trigger decision to CDM boards
* run control
* integration with GPS 10MHz and 1pps clocks and GPS/IRIG date and time information

Crate Data Manager (CDM):
* clock distribution from GDM to CAEN VX digitizers
* receive trigger data from CAEN VX digitizers
* send trigger data to GDM
* run control and dead time control

= Links =

* https://dsdaqgw.triumf.ca/vslice/ - vertical slice DAQ at TRIUMF
* https://daq00.triumf.ca/DaqWiki/index.php/DarkSide - this DS-DAQ Wiki page
* https://dsvslice.triumf.ca/vx_napoli - DS vertical slice at TRIUMF
* https://edev-group.triumf.ca/hw/vme/dark-side-20k-data-manager-card/rev0 - DS-DM board Rev0
* https://edev-group.triumf.ca/hw/exp/dark-side-20k/cdm-gdm-data-manager-vme-cards/rev1 - DS-DM board Rev1
* https://docs.xilinx.com/v/u/en-US/ds593 - Xilinx Platform Cable USB II
* https://www.enclustra.com/en/products/system-on-chip-modules/mercury-xu8/ - Enclustra
* https://bitbucket.org/team-ds-dm/phasemeasurement - VX Phase measurement script
* https://daq00.triumf.ca/daqinv/frontend/list/178 - inventory database
* https://daq00.triumf.ca/elog-ds/DS-DAQ - DS-DAQ elog
* https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0 - GPS and Rb Clock adapter board

= Git repos =

* dsdmdev@dsdaqgw.triumf.ca:git/gcdm.git - git repository, DS-DM firmware, branch develop_ko2, contact K.O. for ssh key access
* https://edev-group.triumf.ca/fw/exp/darkside/gcdm - git repository, DS-DM firmware (broken, do not use)
* https://bitbucket.org/team-ds-dm/ds-dm-software - git repository, DS-DM MIDAS frontend
* https://bitbucket.org/ttriumfdaq/dsproto_vx2740/src/develop/ - VX DSFE MIDAS frontend
* https://openproject.triumf.ca/svn/darkside-20k/ - VX firmware project

= Onboard hardware =

* jtag chain: arm_dap_0 0x5BA00477, xczu4_1 0x04721093
* Eclustra Mercury+ XU8 module: ME-XU8-4CG-1E-D11E-R2.1
** Xilinx® Zynq Ultrascale+™ MPSoC XCZU4CG-1FBVB900E
** DDR4 ECC SDRAM (PS) 2 GB
** DDR4 SDRAM (PL) 1GB
* ethernet mac chip: AT24MAC402-SSHM-T ("602" chip is wrong)
* USB UART for Enclustra serial console, micro-USB, 115200n8
* clock chip: SI5394A-A-GM and oscillator CS-044-054.0M (54 MHz)
* U23 3.3V current meter and thermometer, LTC2990IMS#TRPBF
* LEDs:
** LED_FP A/B/C/D 0/1/2/3
** led1 - 3V3_SW_ON, SOM_POWER_GOOD - Enclustra FPGA module 3.3V power is good
** led2 - LTM4624 PGOOD
** led3 - FPGA_DONE - FPGA has booted
** led4 - TP-S-1, PCLK_P
** led5 - TP-S-2, PCLK_N
* LEMO connectors (top to bottom)
** J4 - input (NIM/TTL) (EXT_IN_LV(1), EXT_IN_LV(2))
** J5 - input (NIM/TTL) (EXT_IN_LV(3), EXT_IN_LV(4))
** J6 - external clock (GPS 10MHz and PPS)
** J7 - output (NIM/TTL) (EXT_OUT(1), EXT_OUT(2))
* SMA connectors
** J9, J10 - CLK_CCA from U6 C.C.
** J11, J12 - CLK_TP0
* RJ45 ethernet connector (100mbit: green light on, yellow light flashes when there is traffic)
* SFP connector (SFP is FTLF8526P3BNL, 6 Gbit/sec, 850 nm, 300m 50/125um OM3 MMF)
* 4 QSFP connectors (GDM)
* 6 VX connectors (CDM)

= Buttons, jumpers and switches =

Buttons:

* PB1 - HRST - reboot FPGA (power-on reset)
* PB2 - SRST - (SRSTn) - reboot ARM CPU

Switches:

* SW1 - boot mode BM0, BM1 [-->]
* SW2 - LEMO output NIM<->TTL
* SW3 - LEMO input 1 and 2 NIM/TTL
* SW4 - LEMO input 2 and 4 NIM/TTL
* SW5 - LEMO clock input NIM/TTL
* SW6 - serial console select. [PS<--PL] PS is ARM CPU, PL is FPGA.

= Front panel =

<pre>
| top
|
| LED-FP1 | LED_FP(0,1,2,3)
|
| SFP J???
|
| LEMO J4-LEFT, J4-RIGHT | EXT_IN_LV(1), EXT_IN_LV(2)
| LEMO J5-LEFT, J5-RIGHT | EXT_IN_LV(3), EXT_IN_LV(4)
| LEMO J6-LEFT, J6-RIGHT | CLK_EXT1, CLK_EXT0 (125 MHz only)
| LEMO J7-LEFT, J7-RIGHT | EXT_OUT(1), EXT_OUT(2)
|
| J-VX-1
| J-VX-2 or QSFP-1
| J-VX-3 or QSFP-2
| J-VX-4 or QSFP-3
| J-VX-5 or QSFP-4
| J-VX-6
|
| RJ45 J3 ethernet
|
| bottom
</pre>

= VX adapter board =

LVDS I/O connector
<pre>
34 pin connector: 0|:::: :::: :::: :::: :|15,16 n/c
</pre>

== split-cable connection ==

<pre>
0 .. 7 -> N/C

8 -> VX_RX(3) - not used
9 -> VX_RX(2) - busy VX to CDM
10 -> VX_RX(1) - DS20K 125 MHz serial data VX to CDM
11 -> VX_RX(0) - DS20K 62.5MHz clock VX to CDM

12 <- VX_TX(0) - TRG CDM to VX
13 <- VX_TX(1) - TSM CDM to VX, to be VETO CDM to VX
14 <- VX_TX(2) (set by jumper routed here or to VX CLKIN SYNC) - 125 MHz serial data CDM to VX
15 <- n/c (CLK routed to VX CLKIN CLK) - 62.5 MHz clock CDM to VX

16 - n/c
</pre>

== one-to-one connection ==

<pre>
0 -> VX2_RX(3)
1 -> VX2_RX(2)
2 -> VX2_RX(0)
3 -> VX2_RX(1)

4 <- CLK
5 <- VX2_TX(0)
6 <- VX2_TX(1)
7 <- VX2_TX(2)

8 -> VX1_RX(3)
9 -> VX1_RX(2)
10 -> VX1_RX(1)
11 -> VX1_RX(0)

12 <- VX1_TX(0)
13 <- VX1_TX(1)
14 <- VX1_TX(2) (set by jumper routed here or to VX CLKIN SYNC)
15 <- n/c (CLK routed to VX CLKIN CLK)

16 - n/c
</pre>

= Board schematics Rev0 =

* [[File:SCH-DS-xDM-Rev0.PDF]]
* note: FPGA pin annotations ("IO", "SCLK", "PCLK", etc) on the schematics are bogus, instead, trace them to the FPGA pins.
* note: ENC A is J800, ENC B is J801, ENC C is J900 (schematic name to enclustra name)
* note: Enclustra special pins: "GC" is "clock capable", "HDGC" is "clock capable", "MGTREFCLK" is MGT reference clocks.
* board modifications:
** ethernet mac chip
** NIM output (no U15, etc)
** RJ45 wrong pinout (board mod or special ethernet cable)
** 125 MHz clock mods (TBW)
** disconnect QSFP0_SEL from SFP_RS0 and QSFP1_SEL from SFP_RS1, these signals are not used by modern SFPs
** provide SFP i2c modsel to allow SFP and QSFP at the same time on address 0x50

= Board schematics Rev1 =

* [[File:SCH-DS-xDM-Rev1.PDF]]
* ENC-A is J800, ENC-B is J801, ENC-C is J900
* modifications from Rev0:
* clocks:
** 125MHz osc: CLK_XO_125, CLK3_XO_125 (CLK2_XO_125 removed)
** C.C. in2 is CLK_CC_IN from ENC-C142,144
** C.C. out0 via CLK_A to CLK_CC_OUT0 (ENC-C3,5), CLK_CC_OUT1 (ENC-B3,5), CLK_CC_OUT2 (ENC-C151,153)
* RTC chip: TP_S pins gone, RTC_I2C_SCL, RTC_I2C_SDA, RTC_1PPS and RTC_32KHZ added
** RTC_1Hz ENC-C160 pull up 10k to 3.3V
** RTC_32k ENC-B129 pull up 10k to 1.8V
* QSFP, SFP control lines:
** HW_ID_xDM ENC-B131 gone, reused as SFP_ModPrsN
** SFP_ModPrsN (MOD_ABS) ENC-B131 input (no pulls)
** SFP_TX_fault (was QSFP2_SEL) QSFP_ModPrsN ENC-C157 input (no pulls)
** SFP_TX_disable from QSFP_LPMode ENC-C163 output (pull up 10k to 3.3V)
** SFP_ModDet n/c (was QSFP_ModPrsN) renamed SFP_ModPrsN input
** SFP_RX_LOS to QSFP_IntN ENC-C159 input (pull up 2k2 to 3.3V)
** SFP_RS0, SFP_RS1 n/c, assembly option: floating
** QSFP_ResetN - FPGA output, pull up 10k to 3.3V
** QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL - FPGA outputs, pull up 10k to 3.3V
** QSFP_LPmode - FPGA output, pull up 10k to 3.3V
** QSFP_ModPrsN - ??? - pull up 2k2 to 3.3V
** QSFP_IntN - ??? - pull up 2k2 to 3.3V
* VX_TX and VX_RX reassigned for better length matching
* cosmetic changes:
** front panel USB-C connector (replaces Micro-USB)
** (reset push button PB1 PORn ENC-A132 aka PS_POR_B aka PS_POR#)
** (reset push button PB2 SRSTn ENC-A124 aka PS_SRST_B aka PS_SRST#)
** front panel PB3 HRSTn same as PB1
* Enclustra errata:
** possible cross talk between pins AA2 (J801-B90, VX9_RX2) and AE3 (J801-B129, RTC_32KHZ). no RTC on the CDM.
** ethernet link up failure (see Microchip KSZ9031 errata)

= FPGA MGT blocks =

<pre>
* SFP - ENC B45,B47 and B48,B50 - FPGA D5,D6 MGTHTX0_D and D1,D2 MGTHRX0_D
* QSFP0 TX0 - ENC C13,C17 - FPGA H5,H6 MGTHTX0_C
* QSFP0 TX1 - ENC C21,C25 - FPGA G7,G8 MGTHTX1_C
* QSFP0 TX2 - ENC C29,C23 - FPGA F5,F6 MGTHTX2_C
* QSFP1 TX0 - ENC C37,C41 - FPGA E7,E8 MGTHTX3_C
* QSFP1 TX1 - ENC C45,C47 - FPGA P5,P6 MGTHTX0_B
* QSFP1 TX2 - ENC C51,C53 - FPGA M5,M6 MGTHTX1_B
* QSFP2 TX0 - ENC C57,C59 - FPGA L3,L4 MGTHRX2_B
* QSFP2 TX1 - ENC C63,C65 - FPGA K5,K6 MGTHTX3_B
* QSFP2 TX2 - ENC C75,C77 - FPGA W3,W4 MGTHTX0_A
* QSFP3 TX0 - ENC C79,C81 - FPGA V5,V6 MGTHTX1_A
* QSFP3 TX1 - ENC C85,C87 - FPGA T5,T6 MGTHRX2_A
* QSFP3 TX2 - ENC C89,C91 - FPGA R3,R4 MGTHTX3_A
</pre>

= Clock distribution Rev0 =

Simplified:

* 125 MHz osc -> CLK_XO_125 -> MGTREFCLK0_A -> not used
* 125 MHz osc -> CLK3_XO_125 -> MGTREFCLK1_B -> SFP RX ref clock, QSFP RX and TX ref clock (this is not final design!)
* (disconnected) 125 MHz osc -> CLK2_XO_125 -> FPGA AG8,AH8 (GC)
* 125 MHz osc -> C.C. in1

* C.C. in0 <- CLK_EXT1 (10 MHz GPS clock)
* C.C. in1 <- 125 MHz osc
* (disconnected) C.C. in2 <- CLK_CC_IN <- MGTREFCLK0_D <- SFP RX clock (cannot be used because of uncontrollable phase)
* C.C. in2 <- CLK2_XO_125 <- FPGA AG8,AH8 (GC) <- SFP RX recovered 125 MHz clock
* C.C. in3 <- CLK_FB

* C.C. 125 MHz -> CLK_CC_OUT0 -> MGTREFCLK0_B -> QSFP RX and TX ref clock (final design)
* C.C. 125 MHz -> CLK_CC_OUT1 -> MGTREFCLK1_D -> SFP TX clock
* C.C. 62.5 MHz -> VX clock fanout

proposed changes:
* add C.C. 125 MHz -> new CLK_CC_OUT2 (old CLK2_XO_125) -> FPGA AG8,AH8 (GC)
* change C.C. in2 <- new CLK_CC_IN <- FPGA AK8,AK9 (non-GC)

Complete:

<pre>
125 MHz oscillator - U5 fan out -
q0 -> CLK_XO_125 -> ENC C72,C74 -> FPGA R7,R8 MGTREFCLK0_A (not used)
q1 -> U6 C.C. in1
q2 -> CLK3_XO_125 -> ENC C7,C9 -> FPGA J7,J8 MGTREFCLK1_B -> SFP RX reference clock, QSFP RX and TX reference clocks (not final design!)
q3 -> disconnected on the board, was CLK2_XO_125 -> ENC C151,C153 -> FPGA AG8,AH8 (GC)

U6 C.C (clock cleaner) -

in0 <- CLK_EXT1 (presumably GPS 10 MHz ref clock)
in1 <- 125 MHz oscillator via U5
in2 <- (was: CLK_CC_IN <- ENC B10,B12 <- FPGA D9,D10 MGTREFCLK0_D <- SFP RX clock, 125 MHz)
in2 <- CLK2_XO_125 <- ENC C151,B153 <- FPGA AG8,AH8 (GC) <- mgt_link_data_to_processing.rx_data_clk (SFP RX data clock, 125 MHz)
in3 <- CLK_FB

out0 -> CLK_CCA -> U12 (125 MHz)
out1 -> CLK_CCB -> VX1..6 (62.5 MHz)
out2 -> CLK_CCC -> VX7..12 (62.5 MHz)
out3 -> CLK_FB into in3

CLK_CCA -> U12 (125 MHz fan out) ->

Q0 -> not used
Q1 -> CLK_CC_OUT0 -> ENC C3-5 -> FPGA L7,L8 MGTREFCLK0_B -> QSFP RX and TX reference clocks (final design)
Q2 -> CLK_CC_OUT1 -> ENC B3-5 -> FPGA B9,B10 MGTREFCLK1_D -> SFP TX clock
Q3 -> not used
Q4 -> not used
Q5 -> SMA J9/J10

CLK_TP0 - FPGA N7,N8 MGTREFCLK1_A - ENC C69,C71 - DS-DM SMA J11, J12 (NOT IN CDM PROJECT)

Notes:
* CLK_XO_125 (125 MHz osc) is not used
* 62.5 MHz VX clock does not go into the FPGA
* CLK_EXT0 going to in1 of U5 cannot be used. only permitted frequency is 125 MHz (it drives the MGT reference clocks) and is this frequency is too high for LEMO cables and connectors

Proposed modifications:
- CLK_CC_IN: from FPGA output pin (ENC C142,C144 - FPGA AK8,AK9) to clock cleaner in2 (instead of CLK2_XO_125 pins)
- CLK_CCA -> U12 -> currently unused out3 -> CLK2_XO_125 FPGA pins
- repurpose CLK_EXT0 at GPS 1pps/IRIG input to FPGA
</pre>

= Clock distribution Rev1 =

Simplified:

* 125 MHz osc -> CLK_XO_125 -> MGTREFCLK0_A -> clk_xo_125. not used
* 125 MHz osc -> CLK3_XO_125 -> MGTREFCLK1_B -> CDM SFP RX ref clock
* 125 MHz osc -> C.C. in1

* C.C. in0 <- CLK_EXT1 (10 MHz GPS clock)
* C.C. in1 <- 125 MHz osc
* C.C. in2 <- CLK_CC_IN <- FPGA AK9,AK8 <- CLK_CC_IN1, rx_clk_mgt, SFP RX recovered clock, 125 MHz
* C.C. in3 <- CLK_FB

* C.C. 125 MHz -> CLK_CC_OUT0 -> MGTREFCLK0_B -> GDM QSFP RX and TX ref clock, CDM rx_clk mux
* C.C. 125 MHz -> CLK_CC_OUT1 -> MGTREFCLK1_D -> SFP TX clock
* C.C. 125 MHz -> CLK_CC_OUT2 -> FPGA AG8,AH8 (GC) -> not used
* C.C. 62.5 MHz -> VX clock fanout

Complete:

<pre>
125 MHz oscillator - U5 fan out -
q0 -> CLK_XO_125 -> ENC C72,C74 -> FPGA R8,R7 MGTREFCLK0_A -> XDC CLK_XO_125_P -> VHDL not used
q1 -> U6 C.C. in1
q2 -> CLK3_XO_125 -> ENC C7,C9 -> FPGA J8,J7 MGTREFCLK1_B -> XDC GDM missing, CDM CLK3_XO_125_P -> VHDL SFP RX reference clock (mgt_rx_ref_clk)
q3 -> not used, was CLK2_XO_125

U6 C.C (clock cleaner) -

in0 <- CLK_EXT1 (GPS 10 MHz clock)
in1 <- 125 MHz oscillator via U5
in2 <- CLK_CC_IN <- ENC C142,C144 <- FPGA AK9,AK8 <- XDC GDM missing, CDM CLK_CC_IN1 <- VHDL rx_clk_mgt
in3 <- CLK_FB

out0 -> CLK_CCA -> U12 (125 MHz)
out1 -> CLK_CCB -> VX1..6 (62.5 MHz)
out2 -> CLK_CCC -> VX7..12 (62.5 MHz)
out3 -> CLK_FB into in3

CLK_CCA -> U12 (125 MHz fan out) ->

Q0 -> not used
Q1 -> CLK_CC_OUT0 -> ENC C3,C5 -> FPGA L8,L7 MGTREFCLK0_B -> XDC GDM clk_mgtrefclk0_x0y1_p, CDM i_CLK_CC_OUT0_P -> VHDL GDM mgt_b_ref_clk QSFP RX and TX reference clocks, CDM alternate rx_clk via clock mux
Q2 -> CLK_CC_OUT1 -> ENC B3,B5 -> FPGA B10,B9 MGTREFCLK1_D -> XDC i_CLK_CC_OUT1_P -> VHDL GDM not used, CDM mgt_tx_ref_clk SFP TX clock
Q3 -> CLK_CC_OUT2 -> ENC C151,C153 -> FPGA AG8,AH8 (GC) -> XDC GDM missing, CDM i_CLK_CC_OUT2_P
Q4 -> not used
Q5 -> SMA J9/J10

CLK_TP0 <-> ENC C69,C71 <-> FPGA N8,N7 MGTREFCLK1_A (not used) -> XDC CLK_TP0_P -> VHDL not used (DS-DM SMA J11, J12)

disconnected <- OLD_CLK_CC_IN <- ENC B10,B12 <- FPGA D9,D10 MGTREFCLK0_D <- XDC o_CLK_CC_IN <- VHDL CDM o_OLD_CLK_CC_IN SFP o_mgt_rx_rec_clk

Notes:
* Enclustra ENC-B is J801, ENC-C is J900
* "(GC)" is clock-capable FPGA pin
* CLK_XO_125 (125 MHz osc) is not used in FPGA
* 62.5 MHz VX clock does not go into the FPGA
* CLK_EXT0 going to in1 of U5 cannot be used. only permitted frequency is 125 MHz (it drives the MGT reference clocks) and is this frequency is too high for LEMO cables and connectors
</pre>

= I2C bus =

* I2C_SCL is J-ENC-A pin 111 I2C_SCL is FPGA I2C_SCL_PL AB13 (IO_L1N_TOL1D_64) and I2C_SCL_PS F18 (PS_MIO10)
* I2C_SDA is J-ENC-A pin 113 I2C_SDA is FPGA I2C_SDA_PL AH13 (IO_L7N_T1L1Q_AD13N_64) and I2C_SDA_PS G18 (PS_MIO11)

* XU8 secure EEPROM ATSHA204A at 0x64, this is 0110'010X -> linux _011'0010 is 0x32. (but responds to scan and read at 0x33)
* U4 ethernet mac chip, EEPROM at 1010 A2 A1 A0 X and MAC/serial_no at 1011 A2 A1 A0 X. A0=VCC, A1=VCC, A2=GND -> linux _101'0011 and _101'1011 is 0x53 and 0x5B.
* U6 clock chip, address 1101 0 A1 A0 X. A1=VCC, A0=N/C (internal pull-up) -> linux _110'1011 is 0x6b
* U23 voltmeter at 10011 ADR1 ADR0. ADR0=GND, ADR1=VCC -> linux _100'1110 is 0x4e
* SFP, address 1010000X -> linux _101'0000 is 0x50. additional SFP data at 0x51
* QSFP0, QSFP1, QSFP2, QSFP3 (QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL)

= I2C clock builder connection =

* use Silicon Labs USB "Clock builder pro field programmer", www.silabs.com/CBProgrammer
* connect rainbow jumper cable pins:
** black - 1-GND to GND on DS-DM
** white - 3-SCLK to SCL on the DS-DM
** grey - 7-SDA_SDIO to SDA on the DS-DM
* power up the DS-DM
* plug USB programmer into Windows laptop
* on Windows, run "ClockBuilder Pro"
* it should report "Field programmer detected", press "EVB GUI"
* in EVB GUI, press "Config", set I2C address 0x6B
* press "Scan", it should find Si5394A-A-GM
* select the "Status" tab, should see real-time status of clock chip

= GDM MGT configuration =

* TX configuration:
* GDM MGT transceivers are configured as "multilane" TX and RX.
* there is 12 TXes ("lanes")
* MGT reference 125 MHz clock goes into [2:0]gtreclk00_in and [11:0]gtrefclk0_in
* one MGT is designated as "master"
* PLL of master MGT converts reference clock into common TX clock and common tx_user_clk2 which becomes tx_data_clk
* common tx_user_clk2 aka tx_data_clk goes into all TXes and clocks tx_data.
* tx_user_clk2 aka tx_data_clk is 125 MHz but not same phase as MGT reference clock.

* RX interim configuration:
* there is 12 RXes ("lanes")
* each RX produces it's own recovered RX clock
* "multilane" configuration assumes all RX recovered clocks run at the same frequency (TX on the other end are driven by common TX clock, see above), but have different phase
* one RX recovered clock is designated as "master" (rx_user_clk2 aka rx_data_clk) and a phase-matching fifo/buffer is used to bring rx_data from all 12 RXes to this common rx_data_clk
* this works because each CDM SFP TX runs on the SFP RX recovered clock which is frequency-locked with the GDM QSFP TX clock.

* RX final configuration:
* MGTs permit using the common TX clock (tx_user_clk2 aka tx_data_clk) as the common rx_data_clk (they are frequency locked through the CDM).
* this permits use of tx_data_clk as the main clock domain in the GDM and removes the need to bring rx_data into the tx_data_clk domain (actually this is done in the MGT RX phase matching fifo/buffer).

= Clock path =

NOTE: MUST REVIEW!!!

<pre>
10 MHz ext clock or GDM 125 MHz oscillator
-> GDM QSFP MGT reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz (GDM main clock domain) and TX bit clock 2.5 GHz
-> GDM QSFP optic transmitter
-> CDM SFP optic receiver
-> CDM SFP MGT, RX reference clock is CDM 125 MHz oscillator
-> MGT RX recovered clock 125 MHz (CDM main clock domain)
-> CC_CLK_IN -> CDM C.C. -> CC_CLK_OUT1 -> CDM SFP TX reference clock 125 MHz
-> MGT PLL -> tx_data_clk 125 MHz and TX bit clock 2.5 GHz
-> (tx_data phase matching fifo from CDM main clock domain to tx_data_clk)
-> CDM SFP optic transmitter
-> GDM QSFP RX optic receiver (12x)
-> GDM QSFP MGT (RX reference clock is same as TX reference clock)
-> MGT RX recovered clock (12x recovered clocks)
-> in multi-lane configuration, one of them is the "master" recovered clock rx_data_clk
-> (rx_data phase matching fifo from rx_data_clk to GDM main clock domain)
</pre>

= CDM rx_clk mux =

when CDM SFP is not connected, there is no SFP recovered clock and a mux is used to switch between clk_cc_out0 (power up default) and rx_clk_mgt (SFP recovered clock)

== Test SFP disconnected ==

note: if I say "--cc-in1", CC seems to lock on the 10 MHz GPS external clock,
to prevent this, test sequence includes reloading the CC and the reset of MGT.

<pre>
./test_cdm_local.exe --write32 0x30 0x0 ### rx_clk mux select CC clock
./test_cdm_local.exe --load-cc
./test_cdm_local.exe --reset-mgt
./test_cdm_local.exe --cdm-clocks
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773594e (125000014) should be ~125 MHz <=== all 4 clocks wobble close to 125 MHz
0x1034 rx_clk: 0x0773594f (125000015) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x0773594f (125000015) should be ~125 MHz
0x103C tx_clk: 0x0773594f (125000015) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
./test_cdm_local.exe --write32 0x30 0x1 ### rx_clk mux select SFP recovered clock
./test_cdm_local.exe --cdm-clocks
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773598b (125000075) should be ~125 MHz
0x1034 rx_clk: 0x076d58ec (124606700) should be ~125 MHz <=== off frequency because there is no valid SFP recovered clock
0x1038 mgt_tx_ref_clk_raw: 0x0773598c (125000076) should be ~125 MHz
0x103C tx_clk: 0x0773598b (125000075) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

= Clock domains =

== GPS ==

* no GPS : GDM runs from internal 125 MHz oscillator
* external 10 MHz clock : GDM runs from external 10 MHz clock and optional 1pps signal (use VME-NIMIO32 NIM outputs)
* GPS receiver : GDM runs from GPS 10 MHz clock and GPS IRIG serial data
* LNGS GPS:
** provides 1pps and serial data over fiber from GPS receiver (master). LNGS xxx box is not used.
** serial data goes to GDM, decoded, 1pps signal extracted, goes to LEMO output
** 1pps from GDM is used to train the Rubidium clock which provides a 10 MHz clock
** 10 MHz output from Rubidium clock goes to GDM 10 MHz external clock input

== GDM ==

* AXI clock (100 MHz) - AXI registers
* 125 MHz oscillator - to clock cleaner
* 10 MHz external clock LEMO input - to clock cleaner
* FPGA 125 MHz clock CLK_CC_IN - to clock cleaner - not used (could be used for cascaded GDMs)
* 125 MHz oscillator - CLK3_XO_125 - mgt_b_ref_clk QSFP MGT reference clock (interim GDM design)
* clock cleaner output 125 MHz fanout:
** CLK_CC_OUT0 - QSFP MGT reference clock (final design)
** CLK_CC_OUT1 - not used (CDM SFP reference clock)
** CLK_CC_OUT2 - not used
* QSFP MGT TX data clock 125 MHz tx_data_clk (main clock domain)
** QSFP TX data
** QSFP RX data (in final design, rx_data_clk is same as tx_data_clk. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk == tx_data_clk)
** ds20k block
* (in interim design, rx_data_clk is disconnected from tx_data_clk. rx_data_clk is the "multilane master clock" which is one of the 12x RX recovered clocks. inside the MGT, 12x phase matching fifos for RX data from 12x RX recovered clocks to rx_data_clk)

note: all these clocks are frequency locked to 125 MHz

== CDM ==

* AXI clock (100 MHz) - AXI registers
* 10 MHz external clock LEMO input - to clock cleaner (not used)
* 125 MHz oscillator to fanout
** to clock cleaner
** to SFP MGT RX reference clock: CLK3_XO_125 to mgt_rx_ref_clk to gtrefclk01_in
* SFP MGT RX recovered clock 125 MHz
** MGT PLL to MGT rx_user_clk2 aka rx_data_clk
** MGT rxrecclkout_out to mgt_rx_rec_clk to CLK_CC_IN to C.C.
* SFP MGT rx_user_clk2 aka rx_data_clk (250 MHz/8 bit, 125 MHz/16 bit, 62.5 MHz/32 bit data) (main clock domain)
** SFP RX data
** ds20k block
** VX TX clock PLLs
** VX RX clock PLLs
* C.C. fan out
** 62.5 MHz VX clocks (12x)
** CLK_CC_OUT0 (not used, GDM QSFP MGT reference clock)
** CLK_CC_OUT1 125 MHz to mgt_tx_ref_clk to gtrefclk00_in to SFP MGT TX reference clock
** CLK_CC_OUT2 (not used)
* SFP MGT tx_user_clk2 aka tx_data_clk
** SFP TX data
** TX data phase matching fifo from main clock domain to tx_data_clk
* VX TX clock PLLs
** 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
** VX TX data phase matching from main clock domain to VX TX clock (12 total)
** VX TX serializer
** VX TX LVDS transmitter
* VX RX clock PLLs
** 2 PLLs, 6 clocks each (12 total). phase of each clock independently adjusted via AXI registers
** VX RX LVDS receivers (12 total)
** VX RX deserializers (12 total)
** VX TX data phase matching from VX RX clock to main clock domain

== VX ==

* everything runs on the VX main 125 MHz clock
* correct phase of VX to CDM LVDS data is adjusted by scan of CDM VX RX PLL clock phase (VX to CDM link is now established)
* correct phase of CDM to VX LVDS data is adjusted by scan of CDM VX RX PLL clock phase (link is established after VX to CDM idle data pattern changes from "VX RX data bad" to "good".
* after good phases are found by scan, they are not expected to change unless cables are changed, CDM and VX modules are changed or CDM firmware is rebuilt. (rebuild of VX firmware should not affect LVDS data phase).
* if there is excessive link errors, phase scan must be repeated.

= Board test plan =

To test:
* Enclustra FPGA board
* SFP port - SFP_RS0/RS1 connected to QSFP0_SEL/1_SEL is wrong? our Finisar SFP says RS0, RS1 N/C, so probably okey.
* SW5 CLK_EXT1 NIM works. TTL needs to be tested. flipping SW5 CLK_EXT0 side from NIM to TTL makes CLK_EXT1 go LOS and OOF in the clock chip. R23 and R53 should be removed?
* BOOT_MODE 0 and 1

Partial:
* U23 3.3V current meter and thermometer. V1,V2 is current monitor, same as in application note. V3,V4 is thermometer, same in application note, except capacitor C118 is 0.1u instead of 470pF. Tested ok: Tint, VCC, V1, V2, V1-V2. Test failed: TR2 reads 50-something degC instead of same as Tint.

Done:
* LED_FP1A..D: tested ok. K.O. 15 sep 2022
* USB UART: tested ok. K.O. 15 sep 2022
* J4A, J4B, J5A, J5B LEMO inputs (NIM/TTL) EXT_IN_LV(1..4). TTL threshold 1.7V, NIM threshold -0.3V. 50 Ohm termination. TTL straight, NIM inverted.
* J6A, J6B LEMO clock inputs: 10 MHz TTL from chronobox works, 10 MHz NIM from IO32 works. 18-apr-2023
* J7A, J7B LEMO outputs EXT_OUT(1), EXT_OUT(2) (NIM/TTL) (tested 23nov2022, K.O.). TTL ???, NIM inverted.
** TTL out no 50 ohm termination: 0=0V, 1=5V, rise and fall time ~5 ns
** TTL out with 50 ohm termination: 0=0V, 1=2.5V, rise and fall time <2ns
** NIM out no 50 ohm termination: 0=+50mV, 1=-1.8V, rise and fall time ~3ns
** NIM out with 50 ohm termination: 0=0V, 1=-0.9V, rise and fall time <2ns
* ethernet MAC i2c chip (K.O. 20sep2022: can read: i2cdump 0 0x5b, nov2022 have u-boot driver)
* CDM VX ports 2x(CLK, 3 tx, 4 rx) tested using test_cdm.exe and LVDS loopback in VX firmware.
* SFP i2c tested KO 22jun2023
* QSFP i2c tested KO 22jun2023
* i2c testing complete 22jun2023
* QSFP rx,tx tested 26june2023. lane0,1,2 ok, lane3 tx not connected, laser is off. qsfp0,1,2,3 all lanes ok.
* SFP rx,tx tested 26june2023. rx and tx okey. LOS ok, mod_absent ok.

Failure:
* ethernet: does not connect to alliedtelesys switch. connects to my USB-eth adapter at 100 Mbit speed. uboot mii status reports connection speed oscillating between 1000, 100 and 10. K.O. 16-sep-2022
* SFP LOS and mod_absent are swapped (in the FPGA pin definitions?)
* QSFP slot numbering is wrong.

= Checklist for newly build boards =

== setup and boot ==
* put new board on workbench
* check - vme connector present, vme extraction handles present
* check - standoff are removed from all thru-holes
* plug Enclustra module
* check - SW1 both switches are in the "6" and "3" positions
* check - SW6 both switches are in the "PS" position
* Rev0: connect micro-usb cable from linux PC
* Rev1: connect usb-C cable from linux PC (USB C-to-C or C-to-A)
* connect ethernet from 1gige capable network switch
* connect power from lab power supply - GND, +5V and -12V nominal, +5.7V and -12.5V actual
* power up, +5V current 2.10-2.8A, -12V current 0.05A
* on linux PC, open a new terminal, run: minicom -D /dev/ttyACM0 -b 115200
* in minicom window, observe messages about Xilinx first stage boot loader, etc
* on the ethernet switch, observe network link is 1gige speed (not 10mbit, not 100mbit).
* if everything boots okey, there will be a login prompt, login as root, password root.
* alternatively, from root@daq13: ssh root@dsdm
* busybox devmem 0x80010000 # read firmware version number, i.e. 0xEDAD0A77
== test i2c ==
* check i2c: (no SFP, no QSFP plugged in!)
<pre>
root@dsdm:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@dsdm:~#
</pre>
* test clock chip:
<pre>
root@dsdm:~# ./test_cdm_local.exe --load-cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Loading CC registers...
Clock chip data block size 519
root@dsdm:~# ./test_cdm_local.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Polling CC status...
Clock chip state 1, status: LOS_IN0 OOF_IN0 IN1 IN_SEL_1
Ctrl-C
</pre>
* test RTC:
<pre>
root@dsdm:~# ./test_cdm_local.exe --rtc-read
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Read the RTC chip!
reg 0x00: 0 0 1 0 0 1 1 0 0x26
reg 0x01: 0 0 1 1 1 0 0 1 0x39
reg 0x02: 0 0 0 0 0 1 1 1 0x07
reg 0x03: 0 0 0 0 0 0 1 0 0x02
reg 0x04: 0 0 0 0 0 0 1 0 0x02
reg 0x05: 0 0 0 0 0 0 0 1 0x01
reg 0x06: 0 0 0 0 0 0 0 0 0x00
reg 0x07: 0 0 0 0 0 0 0 0 0x00
reg 0x08: 0 0 0 0 0 0 0 0 0x00
reg 0x09: 0 0 0 0 0 0 0 0 0x00
reg 0x10: 0 0 0 0 0 0 0 0 0x00
reg 0x11: 0 0 0 0 0 0 0 0 0x00
reg 0x12: 0 0 0 0 0 0 0 0 0x00
reg 0x13: 0 0 0 0 0 0 0 0 0x00
reg 0x14: 0 0 0 1 1 0 0 0 0x18
reg 0x15: 0 0 0 0 1 0 0 0 0x08
reg 0x16: 0 0 0 0 0 0 0 0 0x00
reg 0x17: 0 0 0 1 0 1 1 1 0x17
reg 0x18: 0 1 0 0 0 0 0 0 0x40
seconds: 26, minutes: 39, hours: 07, day 2, date: 00-01-02, temp 23.25C
root@dsdm:~#
</pre>

== test clocks ==

<pre>
root@dsdm:~# ./test_cdm_local.exe --reset-mgt
...
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735cb9 (125000889) should be ~125 MHz
0x1034 rx_clk: 0x07735cb9 (125000889) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735cb9 (125000889) should be ~125 MHz
0x103C tx_clk: 0x07735cb9 (125000889) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
^C
root@dsdm:~# ./test_cdm_local.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735c6a (125000810) should be ~125 MHz
0x1034 rx_clk: 0x07735c69 (125000809) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735c69 (125000809) should be ~125 MHz
0x103C tx_clk: 0x07735c6a (125000810) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
^C
root@dsdm:~#
</pre>

== test LED, LEMO ==

* test LEDs, observe all 4 LEDs turn on and off every 1 second
<pre>
root@dsdm:~# ./test_cdm_local.exe --blink-led
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
LED on
LED off
</pre>
* TBW - test LEMO outputs
<pre>
have test firmware lemo outputs are RTC chip 1pps and 48kHz
see them on the scope
</pre>
* TBW - test LEMO inputs
<pre>
./test_cdm_local.exe --monreg 6
</pre>

== test VX ==

* on the CDM, connect all split blue cables to VX modules
* set all VX modules to LVDS loopback mode (in VX settings MIDAS page)
* ssh root@cdm00
* run "./test_cdm.exe --test-vx", it should report:
<pre>
CDM registers: 0x0000000000000000 0x00000000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000eeeeeeeeeeee 0x00007700: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000080808080808 0x00000100: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000040404040404 0x00000200: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000020202020202 0x00000400: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000000000000000 0x00000800: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000808080808080 0x00001000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000404040404040 0x00002000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000202020202020 0x00004000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
CDM registers: 0x0000000000000000 0x00008000: port0 ok port1 ok port2 ok port3 ok port4 ok port5 ok
all on!
all off!
</pre>
* on the VX settings MIDAS page LVDS input section, observe line status is cycling on and off.
* note: VX_RX(0) is not tested by this, corresponding VX_TX is used as the VX clock

== test SFP (CDM) ==

* connect SFP module
* ssh root@cdm00
* run "./test_cdm.exe --sfp", it should report:
<pre>
root@cdm00:~# ./test_cdm.exe --sfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x6a332b58, ds20k version 0x20241209
DS-DM firmware build 0x6a332b58, ds20k version 0x20241209
Polling SFP status...
identifier 0x03
connector 0x07
encoding 0x01
wavelength 0x0352 (850 nm)
vendor_name [FINISAR CORP. ]
vendor_pn [FTLF8526P3BNL ]
vendor_rev [A ]
vendor_sn [N3AB4LV ]
vendor_date [200319 ]
dm_type 0x68
temp 30.9 C
vcc 3.285 V
tx_bias 7.362 mA
tx_power 475.7 uW
rx_power 859.1 uW
SFP good 1, status: temp 30.9 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 859 uW
SFP good 1, status: temp 30.8 C, tx_bias 7.4 mA, tx_power 475 uW, rx_power 859 uW
SFP good 1, status: temp 30.8 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 859 uW
SFP good 1, status: temp 30.7 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 859 uW
SFP good 1, status: temp 30.8 C, tx_bias 7.4 mA, tx_power 475 uW, rx_power 858 uW
^C
root@cdm00:~#
</pre>
* NOTE1: if fiber cable is disconnected or GDM is not running, rx_power would read 0 uW
* NOTE2: better SFP modules have smaller tx_bias and larger tx_power (more bang for the buck)
* NOTE3: bad fiber cable and bad fiber connection will result in smaller rx_power

== test QSFP (GDM) ==

* connect QSFP module (any slot)
* ssh root@gdm01
* run "./test_cdm.exe --qsfp3 --qsfp", try --qsfp0, --qsfp1, --qsfp2, --qsfp3 for the 4 QSFP slots
<pre>
root@gdm01:~# ./test_cdm.exe --qsfp3 --qsfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x9e30baf9, ds20k version 0x20241209
DS-DM firmware build 0x9e30baf9, ds20k version 0x20241209
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out hi
Polling QSFP status...
identifier 0x0d
status 0x02
los 0x89
temp 36.7 C
vcc 3.302 V
rx_power 0.1 413.7 460.3 0.1 uW
tx_bias 7.8 7.9 7.9 0.0 mA
tx_power 806.0 842.9 865.2 0.1 uW
vendor_name [FINISAR CORP ]
vendor_pn [FTL410QD4C ]
vendor_rev [A ]
wavelength 850
max_temp 70 C
vendor_sn [X79AC0R ]
vendor_date [220309 ]
QSFP good 1, status: temp 36.7 C, los 0x89, tx_bias 7.8 7.9 7.9 0.0 mA, tx_power 806 843 865 0 uW, rx_power 0 414 460 0 uW
^C
</pre>
* NOTE1: 4th tx_bias and tx_power are zero because 4th channel does not transmit any data, QSFP detects this and automatically shuts down the optical transmitter

= Serial console =

* check that linux computer has correct udev rules to allow access to /dev/ttyACM devices, see https://daq00.triumf.ca/DaqWiki/index.php/SLinstall#Configure_USB_device_permissions and https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Configure_USB_device_permissions
* connect micro-USB cable to connector J-UCB, other end connect to linux computer
* observe /dev/ttyACM0 was created
* run "minicom -D /dev/ttyACM0 -b 115200"
* should have gdm-cdm login
* username root, password root

= i2c =

<pre>
ZynqMP> i2c bus
Bus 0: i2c@ff020000
ZynqMP> i2c dev 0
Setting bus to 0
ZynqMP> i2c probe
Valid chip addresses: 33 4E 53 5B 6B 77
ZynqMP> i2c md 0x5b 0x98
0098: fc c2 3d 00 00 12 1a 6e 0a 90 85 04 94 10 08 50 ..=....n.......P
</pre>

<pre>
root@gdm-cdm:~# i2cdetect 0
Warning: Can't use SMBus Quick Write command, will skip some addresses
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0.
I will probe address range 0x03-0x77.
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f
00:
10:
20:
30: -- -- -- 33 -- -- -- --
40:
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60:
70:
root@gdm-cdm:~#
root@gdm-cdm:~# i2cdump 0 0x5b
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-0, address 0x5b, mode byte
Continue? [Y/n]
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00 ???????P?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e ........??=..??n
a0: 0a 90 85 04 94 10 08 50 90 5b a0 00 a0 00 00 00 ???????P?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 6e ........??=..??n
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@gdm-cdm:~#
</pre>

<pre>
root@gdm0:~# i2cdetect -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#
</pre>

<pre>
root@cdm0:~# i2cdetect -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: 50 51 -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@cdm0:~#
</pre>

* 0x33 - XU8 secure EEPROM (should be at 0x32)
* 0x4e - U23 current and temperature monitor
* 0x50, 0x51 - SFP
* 0x50 - QSFP, 4 QSFP modules enabled by GPIO QSFP0_SEL, QSFP1_SEL, QSFP2_SEL, QSFP3_SEL
* 0x53, 0x5b - ethernet mac eeprom
* 0x6b - U6 clock chip

== U23 ==

NOTE: Text reads double of Tint. not sure why. K.O. 21-mar-2024.

* internal temperature only

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x00 b # control register: "repeat mode, internal temperature only"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x03 # "Tint ready" and "busy", "busy is always 1 in repeat mode"
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 00 03 03 81 db 2a ce 2a 8e 00 6f 00 45 20 3f ?.????*?*?.o.E ?
10: 01 00 01 01 01 db 2a ce 2a 8e 00 6f 00 45 20 3f ?.????*?*?.o.E ?
...
readback:
reg0 - 03 - Tint ready
reg1 - 00 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB 0x81, bit 0x80 is "DV, data valid", bit 0x40 is "SS, sensor short", 0x20 is "SO, sensor open"
reg5 - Tint LSB 0xDB, Tint = 0x01DB = 475 * 0.0625 degC = 29.6 degC
reg6..F - stale data
</pre>

* Tint, V1, V2, TR2, VCC

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x18 b # control register: "repeat mode, V1, V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 7f 18 7f 7f 81 b9 aa d0 aa 8f 83 5e 83 5e a0 41 ???????????^?^?A
10: 01 18 01 01 01 b9 2a d0 2a 8f 03 5e 03 5e 20 41 ??????*?*??^?^ A
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x1b9*0.0625 = 27.5 degC
reg6 - V1 MSB 0xaa, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 LSB 0xd0, V1 = 0x2ad0*305.18/1000000 = 3.3447 V (correct)
reg8 - V2 MSB 0xaa, ditto
reg9 - V2 LSB 0x8f, V2 = 0x2a8f*305.18/1000000 = 3.3249 V (correct, smaller than V1)
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x5e. TR2 = 0x35e*0.0625 = 53.875 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x41, VCC = 2.5+0x2041*305.18/1000000 = 5.019 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.99 A (about right?)
</pre>

* Tint, V1-V2, TR2, VCC

<pre>
root@gdm0:~# i2cset -y 0 0x4e 0x01 0x19 b # control register: "repeat mode, V1-V2, TR2"
root@gdm0:~# i2cset -y 0 0x4e 0x02 0xff b # trigger
root@gdm0:~# i2cget -y 0 0x4e 0x00 b # status register
0x7f # all data is ready
root@gdm0:~# i2cdump -y 0 0x4e b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 7f 19 7f 7f 81 69 83 f2 83 f2 83 3a 83 3a a0 44 ?????i?????:?:?D
10: 01 19 01 01 01 69 03 f2 03 f2 03 3a 03 3a 20 44 ?????i?????:?: D
reg0 - 7F - all data ready
reg1 - 18 - what we put there
reg2 - trigger
reg3 - not used
reg4 - Tint MSB and DV, SS, SO.
reg5 - Tint LSB 0x169*0.0625 = 22.5 degC
reg6 - V1 or V1-V2 MSB 0x83, bit 0x80 is DV, bit 0x40 is sign
reg7 - V1 or V1-V2 LSB 0xf2, V1-V2 = 0x3f2*19.42/1000000 = 0.0196142 V (correct, compare with V1 and V2 measured above)
reg8 - V2 or V1-V2 MSB
reg9 - V2 or V1-V2 LSB
regA - V3 MSB or TR2 MSB 0x83, 0x80=DV, 0x40=SS, 0x20=SO
regB - V3 LSB or TR2 LSB 0x3a. TR2 = 0x33a*0.0625 = 51.625 degC (wrong, thermistor Q5 is next to U23, should read same as Tint)
regC - V4 MSB or TR2 MSB
regD - V4 LSB or TR2 LSB
regE - VCC MSB 0xa0, bit 0x80 is DV, 0x40 is sign
regF - VCC LSB 0x44, VCC = 2.5+0x2044*305.18/1000000 = 5.021 V (correct, VCC is +5V)

3V3_SW current is (V2-V1)/0.020 = 0.9807 A (about right?)
</pre>

== SFP ==

<pre>
root@cdm0:~# i2cdump 0 0x50
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 03 04 07 00 00 00 00 40 40 0c 00 01 3d 00 00 00 ???....@@?.?=...
10: 0c 02 00 1e 46 49 4e 49 53 41 52 20 43 4f 52 50 ??.?FINISAR CORP
20: 2e 20 20 20 00 00 90 65 46 54 4c 46 38 35 32 36 . ..?eFTLF8526
30: 50 33 42 4e 4c 20 20 20 41 20 20 20 03 52 00 9d P3BNL A ?R.?
40: 00 1a 00 00 4e 33 41 42 34 4c 56 20 20 20 20 20 .?..N3AB4LV
50: 20 20 20 20 32 30 30 33 31 39 20 20 68 f0 03 de 200319 h???
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@cdm0:~#
root@cdm0:~# i2cdump 0 0x51
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 5a 00 d8 00 55 00 e2 00 90 88 71 48 8c a0 75 30 Z.?.U.?.??qH??u0
10: 21 34 01 f4 1b 58 03 e8 31 2d 04 eb 1f 07 06 31 !4???X??1-?????1
20: 31 2d 00 64 27 10 00 9e 00 00 00 00 00 00 00 00 1-.d'?.?........
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
40: 00 00 00 00 3f 80 00 00 00 00 00 00 01 00 00 00 ....??......?...
50: 01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 b7 ?...?...?......?
60: 19 9f 80 c5 0e 17 12 c4 1f 99 00 00 00 00 30 00 ??????????....0.
70: 00 00 00 00 00 00 00 00 ff ff ff ff ff ff ff 01 ...............?
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@cdm0:~#
</pre>

== QSFP ==

<pre>
QSFP i2c enable lines, active low:

QSFP0_SEL - JENC-A 82 - PS-MIO40 - linux gpio 378
QSFP1_SEL - JENC-A 84 - PS-MIO41 - linux gpio 379
QSFP2_SEL - JENC-A 100 - PS-MIO44 - linux gpio 382
QSFP3_SEL - JENC-A 106 - PS-MIO43 - linux gpio 381 (notice 2 and 3 are out of order)
</pre>

<pre>
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
# echo 378 >> /sys/class/gpio/export ### SEL0 338+40
# echo 379 >> /sys/class/gpio/export ### SEL1 338+41
# echo 381 >> /sys/class/gpio/export ### SEL3 338+43
# echo 382 >> /sys/class/gpio/export ### SEL2 338+44
# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) in hi
gpio-379 ( |sysfs ) in hi
gpio-381 ( |sysfs ) in hi
gpio-382 ( |sysfs ) in hi
root@gdm0:~# echo out >> /sys/class/gpio/gpio381/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio382/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio378/direction
root@gdm0:~# echo out >> /sys/class/gpio/gpio379/direction
root@gdm0:~#
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out lo
gpio-379 ( |sysfs ) out lo
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out lo
root@gdm0:~#
echo 1 >> /sys/class/gpio/gpio381/value
echo 1 >> /sys/class/gpio/gpio382/value
echo 1 >> /sys/class/gpio/gpio378/value
echo 1 >> /sys/class/gpio/gpio379/value
cat /sys/kernel/debug/gpio
root@gdm0:~# cat /sys/kernel/debug/gpio
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out hi
gpio-382 ( |sysfs ) out hi
root@gdm0:~#
root@gdm0:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: -- -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#

NOTICE NOTHING AT ADDRESS 0x50
</pre>

<pre>
ENABLE QSFP0, OBSERVE IT IS AT ADDRESS 0x50

root@gdm0:~# echo 0 >> /sys/class/gpio/gpio378/value
root@gdm0:~# i2cdetect -y -r 0
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- 33 -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- 4e --
50: 50 -- -- 53 -- -- -- -- -- -- -- 5b -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- 6b -- -- -- --
70: -- -- -- -- -- -- -- --
root@gdm0:~#
root@gdm0:~# i2cdump -y 0 0x50
No size specified (using byte-data access)
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: 0d 00 02 8e 00 00 01 00 00 05 55 00 05 00 00 00 ?.??..?..?U.?...
10: 00 00 00 00 00 00 1e b8 00 00 81 a0 00 00 00 00 ......??..??....
20: 00 00 12 12 00 01 00 01 00 01 0e e0 0f 20 0e e0 ..??.?.?.???? ??
30: 00 00 1f a0 1e 3b 1f 72 00 01 00 00 00 00 00 00 ..???;?r.?......
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
60: 00 00 00 00 00 00 00 00 00 00 1f 00 00 00 08 00 ..........?...?.
70: 00 00 00 00 00 00 00 ff ff ff ff ff ff ff ff 00 ................
80: 0d 00 0c 04 00 00 00 40 40 02 d5 05 67 00 00 96 ?.??...@@???g..?
90: 00 00 c8 00 46 49 4e 49 53 41 52 20 43 4f 52 50 ..?.FINISAR CORP
a0: 20 20 20 20 07 00 90 65 46 54 4c 34 31 30 51 44 ?.?eFTL410QD
b0: 34 43 20 20 20 20 20 20 41 20 42 68 07 d0 00 43 4C A Bh??.C
c0: 00 07 0f de 58 37 39 41 43 30 52 20 20 20 20 20 .???X79AC0R
d0: 20 20 20 20 32 32 30 33 30 39 20 20 3c 10 00 9e 220309 <?.?
e0: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
root@gdm0:~#
</pre>

== ethernet mac eeprom ==

* correct chip with 84-bit ethernet mac address

<pre>
root@cdm1:~# i2cdump 0 0x53
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
root@cdm1:~#

root@cdm1:~# i2cdump 0 0x5b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00 ????4??2?K?.?...
90: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c ..........??=?Q<
a0: 0a 80 c2 04 34 10 08 32 a8 4b a0 00 a0 00 00 00 ????4??2?K?.?...
b0: 00 00 00 00 00 00 00 00 00 00 fc c2 3d 1a 51 3c ..........??=?Q<
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@cdm1:~#
</pre>

* wrong "602" chip with 64-bit IPv6 address

<pre>
root@cdm0:~# i2cdump 0 0x53
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
root@cdm0:~# i2cdump 0 0x5b
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
80: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00 ???????Q?[?.?...
90: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e ........??=..??.
a0: 0a 90 85 04 94 10 08 51 10 5b a0 00 a0 00 00 00 ???????Q?[?.?...
b0: 00 00 00 00 00 00 00 00 fc c2 3d 00 00 12 1a 2e ........??=..??.
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
root@cdm0:~#
</pre>

= read ethernet mac address from i2c =

(this code is copied from uboot command line i2c code)

in uboot sources board/xilinx/common/board.c replace original function with this:

<pre>
// special code to read ethernet MAC address from the DS-DM-Rev0 board. K.O. Sep-2022

int zynq_board_read_rom_ethaddr(unsigned char *ethaddr)
{
struct udevice *bus;
int ret;
int busnum = 0;

ret = uclass_get_device_by_seq(UCLASS_I2C, busnum, &bus);
if (ret) {
printf("%s: No bus %d\n", __func__, busnum);
return ret;
}

int chip_addr = 0x5B;

struct udevice *dev;

ret = i2c_get_chip(bus, chip_addr, 1, &dev);
if (ret) {
printf("%s: Bus %d no chip 0x%02x\n", __func__, busnum, chip_addr);
return ret;
}

int dev_addr = 0x98;

unsigned char data[8];

ret = dm_i2c_read(dev, dev_addr, data, 8);

if (ret) {
printf("%s: Bus %d chip 0x%02x read error %d\n", __func__, busnum, chip_addr, ret);
return ret;
}

printf("%s: Bus %d chip 0x%02x addr 0x%02x read: 0x %02x %02x %02x %02x %02x %02x %02x %02x\n", __func__, busnum, chip_addr, dev_addr, data[0], data[1], data[2], data[3], data[4], data[5], data[6], data[7]);

// see Atmel-8807-SEEPROM-AT24MAC402-602-Datasheet.pdf

if (data[0] == 0) {
// eiu-48 chip
ethaddr[0] = data[2];
ethaddr[1] = data[3];
ethaddr[2] = data[4];
ethaddr[3] = data[5];
ethaddr[4] = data[6];
ethaddr[5] = data[7];
} else {
// eiu-64 chip
ethaddr[0] = data[0];
ethaddr[1] = data[1];
ethaddr[2] = data[2];
ethaddr[3] = data[5];
ethaddr[4] = data[6];
ethaddr[5] = data[7];
}

printf("%s: ethaddr %02x:%02x:%02x:%02x:%02x:%02x\n", __func__, ethaddr[0], ethaddr[1], ethaddr[2], ethaddr[3], ethaddr[4], ethaddr[5]);

return ret;
}
</pre>

also this should have worked if i2c_xxx() functions were enabled in uboot:
<pre>
i2c_set_bus_num(0);
i2c_probe(0x5b);
i2c_read(0x5b, 0x9a, ethaddr, 6);
</pre>

= read ethernet mac address from i2c (SHOULD WORK) =

from: https://stackoverflow.com/questions/43637540/is-there-linux-or-u-boot-support-to-read-a-mac-address-from-a-chip-at-startup

<pre>
#ethernet related setup
setup_eth=run readmac buildmac
#read mac address from eeprom
readmac=i2c dev 0; i2c read 50 FA.1 6 $loadaddr
#build the ethaddr variable
#not very nice, but does the job
buildmac=\
e=" "; sep=" " \
for i in 0 1 2 3 4 5 ; do\
setexpr x $loadaddr + $i\
setexpr.b b *$x\
e="$e$sep$b"\
sep=":"\
done &&\
setenv ethaddr $e
</pre>

= read ethernet mac address from i2c (DOES NOT WORK) =

this method does not work: inside board/xilinx/common/board.c:int zynq_board_read_rom_ethaddr(unsigned char *ethaddr), uclass_get_device_by_ofnode(UCLASS_I2C_EEPROM, eeprom, &dev) returns ENODEV and read from i2c does not happen. K.O. Sep-2022

Read:
* https://support.xilinx.com/s/question/0D52E00006hpKlsSAE/uboot-and-ethernet-mac-address-from-eeprom?language=en_US (last answer)
* https://github.com/Xilinx/u-boot-xlnx/blob/master/board/xilinx/common/board.c (zynq_board_read_rom_ethaddr())

Note:
* 0x5B is the i2c chip address
* 0x9A is the data offset inside the chip, see datasheet or i2c read dump above.

Edit:
* emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/u-boot/files/platform-top.h
<pre>
#include <configs/xilinx_zynqmp.h>
#include <configs/platform-auto.h>
//#define CONFIG_I2C_EEPROM
//#define CONFIG_SYS_I2C_EEPROM_ADDR 0x5b
//#define CONFIG_SYS_I2C_EEPROM_ADDR_OVERFLOW 0x0
#define CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET 0x9A
#error HERE!
</pre>
* emacs -nw PetaLinux_GDM_CDM/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi
<pre>
/include/ "system-conf.dtsi"
/ {
chosen {
xlnx,eeprom = &eeprom;
};
};

&i2c0 {
eeprom: eeprom@5b { /* u88 */
compatible = "atmel,24mac402";
reg = <0x5b>;
};
};
</pre>
* components/yocto/workspace/sources/u-boot-xlnx/configs/xilinx_zynqmp_virt_defconfig
<pre>
CONFIG_ZYNQ_GEM_I2C_MAC_OFFSET=0x9A
</pre>

= RTC chip =

* DS3231 RTC chip
* FPGA connections:
<pre>
I2C SCL <- J-ENC A85 <- FPGA E17 <- XDC TP_S <- VHDL RTC_I2C_SCL output (10k pull-up to 3.3V)
I2C SDA <-> J-ENC A87 <-> FPGA D17 <-> XDC TP_S <-> VHDL RTC_I2C_SDA bidir (10k pull-up to 3.3V)
RTC_1Hz -> J-ENC C160 -> FPGA AH12 -> XDC "slow_io" -> VHDL RTC_1PPS input (10k pull-up to 1.2V)
RTC_32k -> J-ENC B129 -> FPGA AE3 -> XDC "free pin" -> VHDL RTC_32KHZ input (10k pull-up to 1.8V)
</pre>
* test notes
<pre>
FPGA registers:

rtc_i2c_clock_ie <= not register_data_in_block_0(14)(0);
rtc_i2c_clock_out <= register_data_in_block_0(14)(1);
rtc_i2c_data_ie <= not register_data_in_block_0(14)(4);
rtc_i2c_data_out <= register_data_in_block_0(14)(5);

ext_out(1) <= rtc_1pps_in;
ext_out(2) <= rtc_32khz_in;

register_data_out_block_0(13) <= x"000000" & rtc_i2c_data_ie & rtc_i2c_data_out & rtc_i2c_clock_ie & rtc_i2c_clock_out & rtc_1pps_in & rtc_32khz_in & rtc_i2c_data_in & rtc_i2c_clock_in;

After power us on the RTC chip:
i2c clock 3.3V
i2c data 3.3V
32kHz 0.778V see clock running with amplitude 0->1.8V (FPGA pullup) period about 30 usec (~33 kHz)
1pps 1.2V (FPGA pullup)

./test1.exe --read32 0x34
dsdm_read32[0x00000034] is 0x000000ab (171) -> 1010'1011 -> data_ie, clock_ie, 1pps, data_in, clock_in
./test1.exe --read32 0x38
dsdm_read32[0x00000038] is 0x00000000 (0)

./test1.exe --write32 0x38 0x01 -> clock oe
./test1.exe --read32 0x34
Ddsdm_read32[0x00000034] is 0x0000008a (138) -> 1000'1010 -> data_ie, 1pps, data_in, clock 0V, data 3.3V

./test1.exe --write32 0x38 0x10 -> data_oe
./test1.exe --read32 0x34
dsdm_read32[0x00000034] is 0x00000029 (41) -> 0010'1001 -> clock_ie, 1pps, clock_in, clock 3.3V, data 0V

root@dsdm:~# ./test_cdm_local.exe --rtc-read
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x0e30d3a8, ds20k version 0x20241104
DS-DM firmware build 0x0e30d3a8, ds20k version 0x20241104
Read the RTC chip!
reg 0x00: 0 1 0 1 0 1 1 1 0x57
reg 0x01: 0 0 1 0 0 0 1 0 0x22
reg 0x02: 0 0 0 0 0 0 0 0 0x00
reg 0x03: 0 0 0 0 0 0 0 0 0x00
reg 0x04: 0 0 0 0 0 0 0 0 0x00
reg 0x05: 0 0 0 0 0 0 0 0 0x00
reg 0x06: 0 0 0 0 0 0 0 0 0x00
reg 0x07: 0 0 0 0 0 0 0 0 0x00
reg 0x08: 0 0 0 0 0 0 0 0 0x00
reg 0x09: 0 0 0 0 0 0 0 0 0x00
reg 0x10: 0 0 0 0 0 0 0 0 0x00
reg 0x11: 0 0 0 0 0 0 0 0 0x00
reg 0x12: 0 0 0 0 0 0 0 0 0x00
reg 0x13: 0 0 0 0 0 0 0 0 0x00
reg 0x14: 0 0 0 1 1 0 0 0 0x18
reg 0x15: 0 0 0 0 1 0 0 0 0x08
reg 0x16: 0 0 0 0 0 0 0 0 0x00
reg 0x17: 0 0 0 1 0 1 1 1 0x17
reg 0x18: 1 1 0 0 0 0 0 0 0xc0
seconds: 57, minutes: 22, hours: 00, day 0, date: 00-00-00, temp 23.75C
root@dsdm:~#
</pre>

= clock chip configuration =

file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt:

* VCO is 14 GHz
* Tvco is 71.43 ps
* N0 divider is 14, frequency is 1000 MHz
* out0 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz
* out1 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
* out2 from N0 divider 0x7 is (7+1)*2 = 16, freq 62.5 MHz
* out3 from N0 divider 0x3 is (3+1)*2 = 8, freq 125 MHz is the feedback for zero delay

= clock chip monitoring =

from si5395-94-92-family.pdf:
* reg 0x1: page select, set to 0 or set to 5 to read 0x53F
* reg 0x2: 0x94
* reg 0x3: 0x53 -> device is a si5394
* reg 0xC: LOSXAXB
* reg 0xD: LOS and OOF for the 4 clock inputs
* reg 0xE: LOL and HOLD
* reg 0xF: CAL_PLL
* reg 0x11: sticky bits for reg 0xC
* reg 0x12: sticky bits for reg 0xD
* reg 0x13: sticky bits for reg 0xE
* reg 0x14: sticky bits for reg 0xF
* reg 0x1C: device reset
* reg 0x1E: low power, hard reset, SYNC
* reg 0x507: currently selected input clock
* reg 0x52A: input clock select
* reg 0x535: FORCE_HOLD
* reg 0x53F: HOLD_HIST_VALID and FASTLOCK_STATUS

= Install Xilinx tools =

* install Vivado 2020.2
<pre>
login at https://www.xilinx.com/myprofile.html
go to "Downloads"
go to archive,
find 2020.2
download Xilinx_Unified_2020.2_1118_1232_Lin64.bin
sh ./Xilinx_Unified_2020.2_1118_1232_Lin64.bin
banner window should open with spinner "downloading installation data"
"a newer version is available" -> say "continue"
next
"select install type" window:
provide email and password,
select "download image"
select directory /home/olchansk/Xilinx/Downloads/2020.2\
select "linux" and "full image"
next
download summary: space required 38.52 Gbytes
download
installation progress
downloading spinner, 16 M/s 47 minutes...
"download image has been created successfully". Ok.
check contents of /home/olchansk/Xilinx/Downloads/2020.2
ls -l /home/olchansk/Xilinx/Downloads/2020.2
total 67
drwxr-xr-x 2 olchansk users 9 Sep 1 16:22 bin
drwxr-xr-x 3 olchansk users 15 Sep 1 16:23 data
drwxr-xr-x 4 olchansk users 4 Sep 1 16:22 lib
drwxr-xr-x 2 olchansk users 644 Sep 1 16:22 payload
drwxr-xr-x 2 olchansk users 7 Sep 1 16:22 scripts
drwxr-xr-x 4 olchansk users 4 Sep 1 16:22 tps
-rwxr-xr-x 1 olchansk users 3256 Nov 18 2020 xsetup
daq13:2020.2$
./xsetup
spinned loading installation data
xilinx design tools 2022.1 now available -> say continue
"welcome" -> next
"select product" -> vivado -> next -> vivado hl system edition -> next
select devices: only zynq ultrascale+ mpsoc -> next
select destination: /opt/Xilinx (as root, mkdir /opt/Xilinx, chmod olchansk.users /opt/Xilinx)
install ...
complete
move /home/olchansk/Xilinx/Downloads/2020.2 to /daq/daqstore/olchansk/Xilinx/Downloads/
</pre>
* install petalinux 2020.2
<pre>
./xsetup
"a newer version is available" -> say "continue"
next
"select product to install" -> select Petalinux (Linux only) -> next
"select destination directory" -> select "/opt/Xilinx" (disk space required 2.64 GB) -> next
"summary" -> install ...
error about missing /tmp/tmp-something files
"installation completed successfully" (hard to dismiss, "ok" button is partially cut-off)
done?
I think it failed, /opt/Xilinx/PetaLinux/2020.2/bin is empty except for petalinux-v2020.2-final-installer.run
try to run it by hand, same error about /tmp/tmp-something files. strange...
notice it complains about "truncate", which truncate finds ~/bin/truncate, get rid of it,
try again
now complains about missing texinfo and zlib1g:i386
apt install texinfo -> ok
apt install zlib1g:i386 -> installs bunch of gcc stuff -> ok
try again
reports "already installed" -> delete /opt/Xilinx/.xinstall/PetaLinux_2020.2/, delete entries in ~/.Xilinx/registry/installedSW.xml
try again
success
</pre>
* install vivado 2022.1 and petalinux 2022.1 - everything is pretty much the same

= Petalinux =

* cd PetaLinux_GDM_CDM
* petalinux-config
* enable i2c MAC address and DHCP
<pre>
git clone https://bitbucket.org:/team-ds-dm/ds-dm-u-boot-xlnx.git
cd ds-dm-u-boot-xlnx
git checkout ds-dm-u-boot-xlnx

linux-components ->
uboot -> ext-local-src
external u-boot local source -> ds-dm-u-boot-xlnx (path to the customized uboot git repository)
</pre>
* enable DHCP
<pre>
Subsystem AUTO Hardware Settings -> Ethernet Settings
randomize MAC address -> NO
ethernet mac address -> leave empty
obtain ip address automatically -> YES
</pre>
* set hostname and product names
<pre>
Firmware Version Configuration ->
Host name -> "ds-dm"
Product name -> "Petalinux_GDM_CDM"
</pre>
* configure linux kernel
<pre>
petalinux-config -c kernel
</pre>
* enable NFS-Root
<pre>
petalinux-config
Image Packaging Configuration > Root File System Type -> set to NFS
Location of NFS root directory set to "/nfsroot"

petalinux-config -c kernel
Networking support > IP: kernel level configuration
enable DHCP, BOOTP, RARP
File systems > Network file systems > Root file systems on NFS
</pre>
* manually fix linux kernel command line:
<pre>
grep nfsroot PetaLinux_GDM_CDM/project-spec/configs/config
edit CONFIG_SUBSYSTEM_BOOTARGS_GENERATED to read
earlycon console=ttyPS0,115200 clk_ignore_unused panic=60 root=/dev/nfs nfsroot=/nfsroot/%s ip=dhcp rw
</pre>
* check configuration in
** PetaLinux_GDM_CDM/project-spec/configs/config
** PetaLinux_GDM_CDM/project-spec/configs/rootfs_config
** PetaLinux_GDM_CDM/components/plnx_workspace/device-tree/device-tree/system-conf.dtsi

= JTAG server =

localhost:3121

= ds20k block =

<pre>
module ds20k
(
// CLOCK INPUTs
input wire clk,
input wire reset, // pulse for power-up reset
input wire si5394_loln, // clock cleaner PLL is locked to selected input clock

// REGISTER_DATA

input wire [255:0] [31:0] register_data_in,
output reg [255:0] [31:0] register_data_out,
input wire register_we,
input wire register_re,

// GDM QSFP FIBER LINKS
input wire qsfp_tx_data_ready,
output reg [11:0] [15:0] qsfp_tx_data,
output reg [11:0] [1:0] qsfp_tx_data_is_k,
input wire [11:0] [15:0] qsfp_rx_data_error, // not sure what this is.
input wire [11:0] [15:0] qsfp_rx_data,
input wire [11:0] [1:0] qsfp_rx_data_is_k,

// CDM SFP FIBER LINKS
input wire sfp_tx_data_ready,
output reg [15:0] sfp_tx_data,
output reg [1:0] sfp_tx_data_is_k,
input wire sfp_rx_data_error, // single bit indicating that RX link is up and data is good.
input wire [15:0] sfp_rx_data,
input wire [1:0] sfp_rx_data_is_k,

// VX_RXs
input wire [3:0] vx1_rx_async,
input wire [3:0] vx2_rx_async,
input wire [3:0] vx3_rx_async,
input wire [3:0] vx4_rx_async,
input wire [3:0] vx5_rx_async,
input wire [3:0] vx6_rx_async,
input wire [3:0] vx7_rx_async,
input wire [3:0] vx8_rx_async,
input wire [3:0] vx9_rx_async,
input wire [3:0] vx10_rx_async,
input wire [3:0] vx11_rx_async,
input wire [3:0] vx12_rx_async,

// VX_TXs
output reg [2:0] vx1_tx_out,
output reg [2:0] vx2_tx_out,
output reg [2:0] vx3_tx_out,
output reg [2:0] vx4_tx_out,
output reg [2:0] vx5_tx_out,
output reg [2:0] vx6_tx_out,
output reg [2:0] vx7_tx_out,
output reg [2:0] vx8_tx_out,
output reg [2:0] vx9_tx_out,
output reg [2:0] vx10_tx_out,
output reg [2:0] vx11_tx_out,
output reg [2:0] vx12_tx_out,

// VX serial comunications
input wire [11:0] vx_rx_ser_in,
output reg [11:0] vx_tx_ser_out,

// LEMO INPUTs
input wire [4:1] ext_in_async, // direct connection to LEMO connectors, not clocked

// LEMO OUTPUTs
output reg [2:1] ext_out, // direct connection to LEMO connectors, not clocked

// FRONT PANEL LEDs
output reg [3:0] fp_led_out, // direct connection to LEMO connectors, not clocked
);
</pre>

= world view =

Main components:
* clock distribution - to ensure all digitizers run synchronously and waveform timestamp are easy to assemble intophysics events.
* global trigger and run control - all digitizers are triggered at the same time to record calibration events, pulser events and the run startup synchronization sequence.
* busy distribution - when some digitizers go busy and stop accepting triggers, physics events may become incomplete, this must be recorded and managed.
* hitmap distribution - to create an online picture of the detector for monitoring and for triggering.

== global clock distribution ==

Clock distribution ensures all digitizers run synchronously (on the same clock) and waveform timestamps are easy to assemble into physics events. The clock source is synchronized with GPS to ensure a metrologically validated clock frequency (tied to the SI definition of 1 second) and date stamp (via GPS IRIG-B date stamp). Individual digitizer waveform timestamp are reset at the begin of run to establish a valid time zero (run start time, date stamped with the GPS IRIG-B date stamp).

Base clock is 125 MHz (8 ns). Individual digitizers are synchronized at initialization (power-up) time. Because of incontrollable delays in the electronics (i.e. variations between individual PLL chips in each digitizer), waveform timestamps between individual digitizers may be off by 1-2 clocks (8-16 ns). After the system is running and is stable (thermally and electrically), timestamp waveform variation and jitter are less than 1 ns.

This is validated by feeding a common calibration pulser signal into analog inputs of each digitizer (at 100 Hz) and measuring (fitting) the pulse time in the waveform. With sufficient statistics, differences between waveform timestamp are measured with picosecond precision.

CDM PLL allows fine phase adjustment of the 62.5 MHz VX2746 clock in groups of 6 clocks (2 groups per CDM) with precision 71.43 ps.

Due to FPGA technology limitations (serial link hard IPs, PLLs, etc) the pattern of 1-2 clock delays between individual digitizers may change/shift from power up to power up. Significant additional work would be required to ensure this pattern is repeatable and stable after every power up and initialization. As mentioned before, after power up and initialization, variation between waveform timestamp is sub-nanosecond as measured by the analog calibration pulser.

Clock chain:
* GPS receiver (1pps clock reference and IRIG-B date stamp)
* LNGS GPS clock distribution (fiber link to converter box at the experiment)
* GPS 1pps to PRS-10 Rb atomic clock (10 MHz clock output)
* 10 MHz clock from PRS-10 to GDM PLL (extra low jitter oscillator, 125 MHz clock output)
* 125 MHz clock to GDM fiber link transmitters (2.5 Gbps line rate)
* CDM fiber link receiver and clock recovery (125 MHz clock)
* 125 MHz recovered clock to CDM PLL (extra low jitter oscillator, 62.5 MHz clock output)
* 62.5 MHz CDM clock to VX2745 PLL via LVDS line
* VX2745 PLL (125 MHz clock output)
* 125 MHz clock to digitizer chips, clock counter and waveform timestamp.

This is the normal mode of operation, with global clock synchronized with the GPS clock (metrologically referenced to SI 1 Hz definition).

If GPS signal is not available, degraded modes of operation are available (in order of worsening performance)
* PRS-10 Rb atomic clock synchronized with a battery-powered time-of-day chip
* PRS-10 Rb atomic clock synchronized with Internet time (NTP) assuming Internet network connection is up
* PRS-10 Rb clock standalone
* GDM internal oscillator

All degraded modes provide high precision low jitter 125 MHz system clock, but long term stability (seconds per day) will vary. (TBM)

In the absence of GPS IRIG-B date stamp, the battery-powered time-of-day chip will be used (TBI) with fallback to Internet time (NTP). Date stamp long term stability is expected to be better than 1 second per day (TBI, TBM).

Note:
* TBM: to be measured
* TBI: to be implemented

Current status (April 2025):
* full clock chain from GDM PLL to VX2745 digitizers is implemented and tested
* waveform timestamp variations are measured (see report from Marek)
* GDM external clock is validated (using VME-GRIFFIN-CDM high quality clock source module)
* PRS-10 10 MHz and 1pps connection to GDM is implemented and tested
* GPS receiver 1pps connection to PRS-10 is implemented and tested
* GPS receiver IRIG-B date stamp connection to GDM tested using a software decoder. Hardware decoder and connection to time slice marker, TBI.
* LNGS date stamp decoder and connection to time slice marker, TBI
* LNGS 1pps connection to PRS-10, not available
* operate vertical slize system from GPS, not available (GPS is in the det lab, vertical slice is in the daq lab).

== global trigger distribution ==

Global trigger distribution provides a synchronous waveform trigger to all VX2745, all CDM and all VX2745 are provided wiht the same data and are all triggered at the same time (without accounting for the 1-2 clock offset described in the clock section).

Global trigger is used for run control: the first trigger of a run resets the waveform timestamp counter, the CDM timestamp counter and the GDM timestamp counter. At the begin of run, GDM generates a sequence of triggers following an easy to identify pattern. If a VX2745 misses a trigger (for any reason), online and offline software can easily identify the tru run start time using this pattern.

After a run has started, global trigger can be generated by a programmable pulser, by an external LEMO signal (i.e. external calibration signal, laser Q-switch, etc) and by the VX2745 trigger hitmap.

Trigger sources (TRG):
* TRG pulser
* LEMO inputs
* hitmap trigger

Time slice marker sources (TSM):
* TSM pulser

Trigger chain, K-code and LVDS:
* TRG/TSM source
* TRG/TSM K-code from GDM to CDM
* CDM K-code to LVDS line 12 and 13 lines (TRG and TSM)

Latency from GDM LEMO input to CDM LVDS out: TBM

Trigger chain, packet trigger:
* GDM TRG/TSM source to TRG/TSM packet generator
* or GDM hitmap group trigger packet generator
* TRG/TSM packet from GDM to CDM
* TRG/TSM packet from CDM to VX2745 via LVDS line, 62.5 MHz, 8b10b encoding.

Latency from GDM LEMO input to VX2745 internal trigger: TBM

Notes:
TBM - to be measured
TBI - to be implemented

Current status (Jan 2026):
* LVDS TRG and TSM distribution is done and tested
* packet TRG, TRIG48 and TSM distribution from GDM to CDM is done and tested
* packet TRG, TRIG48 and TSM from CDM to VX2745 is done and tested
* packet TRIG48 processing in the VX is done and tested

== global busy distribution ==

If any VX2745 goes busy (for any reason) and stops accepting triggers, physics data may become incomplete. This situation needs to be recorded and managed.

Busy chain:
* VX2745 firmware busy logic (CAEN and Yair) generates the VX busy signal on LVDS line 9 (CDM VX_RX_2)
* CDM monitors VX busy
* CDM records busy transitions
* CDM computes the "CDM busy" as grand-OR of all VX busy
* CDM periodically transmits CDM busy status the GDM using special K-codes (NB: confirm transmission period)
* GDM receives CDM busy updates and maintains "CDM busy" for each CDM
* GDM records busy transitions
* GDM computes "GDM busy" as grand-OR of all CDM busy
* effectively, GDM busy is a grand-OR of all VX busy (with max delay, TBM)

In the pilot implementation, any VX busy stops the experiment by sending a veto signal to all VX2745 which stops them from accepting triggers.

Veto chain:
* GDM computes "GDM veto" as GDM busy
* GDM periodically transmits GDM veto status to all CDMs using special K-codes (NB: confirm transmission period)
* CDM receive GDM veto updates and maintain the "CDM veto" signal
* CDM computes "VX veto" as CDM veto
* CDM transmits VX veto to all VX2745 on LVDS pair 13
* VX2745 firmware (Yair) has VX veto block all physics triggers (NB: except for the time slice marker?)

Total delay from VX busy to VX veto (on the LVDS lines), TBM.

Notes:
* TBM - to be measured
* TBI - to be implemented

Current status (Jan 2026):
* full busy chain from VX to CDM to GDM is implemented, tested
* veto chain from GDM to CDM is implemented, tested
* veto chain from CDM to LVDS pair 13 is implemented, tested
* LVDS pair 13 veto inside the VX is implemented, tested
* TRG switched from LVDS pair 12 to TRG packet, testing completed
* TSM switched from LVDS pair 13 to TSM packet, testing completed
* fine tuning of busy chain, VX buffer levels, etc, test in progress

== vx hitmap and hitmap trigger ==

To monitor the system in real time and to generate group triggers, we use the VX hitmaps.

Each VX2745 has 64 channels, each channel has a trigger logic block (digital discriminator) that generates a waveform self-trigger and sends a hitmap bit to the CDM (64-bits per VX2745) as a data packet. CDM passes hitmap data packets to the GDM and also records non-empty hitmap packets. CDM MIDAS frontend passes non-empty hitmap packets to MIDAS for online monitoring and visualization. GDM collects all the hitmaps and generates a global trigger. Alternatively, it generates a group trigger TRIG48 with a 48-bit bitmap indicating which VX2745 should trigger a waveform.

Pilot implementation computes the hitmap trigger as a grand-OR of all hit maps.

Hitmap chain:
* VX2745 analog inputs to digitizers to digital trigger logic block, 1 bit per channel (CAEN, Yair)
* 64-bits of hitmap (1 bit/channel) are collected and periodically transmitted to the CDM (Sam De Jong) (NB: confirm transmission period: see CDM page table "VX hit map" column "period", 0xD1 clocks is 1672 ns, should be reduced to 1500 ns, TBI)
* VX hitmap packets are transmitted to the CDM on the LVDS line 10 (CDM VX_RX_1) at 62.5 MHz, 8b10b encoded.
* CDM receives hitmap packets (12 input data streams, 12-to-1 packet mux)
* CDM sends all hitmap packets to the GDM at 125 MHz, 16b20b encoded.
* non-empty hitmap packets are written to the CDM-to-CPU data FIFO (and via MIDAS frontend to MIDAS SYSTEM event buffer)
* grand-OR of VX hitmap packets (12*64 bits) can generate a CDM-local VX trigger
* GDM receives hitmap packets, computes "GDM hitmap trigger"
* if enabled, GDM hitmap trigger causes GDM trigger, which is processed as described in the global trigger section

Delay, latency and jitter from analog pulse on the VX2745 analog input to VX trigger on LVDS line 12, TBM.

TRIG48 VX group trigger:
* GDM receives hitmap packets, computes the 48-bit bitmap of which VX2745 modules should trigger
* how to specify this computation is not clear. a generic 12*64 MLU is impossible, insufficient RAM on the FPGA
* 48-bit bitmap is sent to all CDMs as a TRIG48 data packet
* CDM receives TRIG48 data packets from GDM and retransmits them to the VX2745 (all CDMs and all VXes see the same packets at the same time).
* VX2745 receives the 48-bit trigger bitmap, compares it with it's own 48-bit bitmap if any common bits are set, triggers waveform readout of all channels (there will be no more than 48 VX2745 in the experiment)

Pilot implementation of TRIG48 triggers on the CDM grand-or and generates a TRIG48 packet that contains a 48-bit bitmap stored in an AXI register.

Example uses of TRIG48:
* to trigger individual VX2745 modules: set the VX 48-bit map to the module number (vx05-0x1, vx06-0x2, vx07-0x4, etc), generate TRIG48 packet with a bitmap of VX modules that should trigger: 0xFFFF'FFFF'FFFF will trigger all modules, 0x1 will trigger only vx05, 0x2 will trigger only vx06, etc. (there is no more than 48 modules).
* to trigger groups of VX2745 modules: set the bitmap of group 1 modules to 0x1, set the bitmap of group 2 modules to 0x2, generate TRIG48 packet with bitmap 0x0001 to trigger only group 1 modules, 0x0002 to trigger only group 2 modules, 0x3 to trigger both groups.

Notes:
* TBM - to be measured
* TBI - to be implemented

Current status (Jan 2026):
* full chain (VX analog input to hitmap generator to CDM to GDM) is implemented, testing in progress
* GDM grand-or trigger is implemented
* VX hitmap packet generator, testing in progress (Yair firmware)
* CDM hitmap processing, implemented, tested
* GDM hitmap processing, implemented, tested
* pilot TRIG48 trigger generator in GDM, CDM, tested
* TRIG48 trigger in VX, implemented, tested
* MIDAS frontend readout of hitmap data from FPGA FIFO: implemented, testing in progress (FIFO code: Ian, done; integration and C++ code: KO)
* online monitoring and visualization in MIDAS: TBI

== firmware map ==

Note:
* red lines: clocks
* green lines: AXI/Avalon packet streams
* blue lines: serial data

[[File:GDM-CDM-VX.drawio.svg]]

== description ==

same thing, in words:

<pre>
detector
digitizer, 125 MHz
digital filter
digital discriminator
hit map, 64 bits at 125 MHz (could be 250 MHz, filter and discriminator clock)
packetizer, 64 bits -> id, timestamp, 8x 8-bit words, eop
8/10 serializer, 12.5 MHz parallel in, 125 MHz serial out
lvds line to CDM vx_rx
BBB: also VX busy to lvds line to CDM vx_rx, do not want to depend on serial comm for vx busy, do not want to inject async data into the hit map packet stream
--- CDM
lvds line capture in IBUF register, vxN_rx clock, 125 MHz clock, from PLL with adjustable phase, scan phase to find sweet spot
vx rx 10/8 deserializer, vxN_rx clock, stobes out parallel data every 10 clocks at 12.5 MHz
vx rx phase transfer from vxN_rx clock to main CDM clock, have 10 clocks for transfer to happen
from here everything is on the CDM main clock
vx rx packet adapter, 12.5 MHz 8-bit packets to 125 MHz 16-bit packets, internal FIFO to avoid data overrun
sfp tx mux, all VX packet streams into one CDM sfp tx packet stream
BBB: capture VX RX busy to CDM main clock (IBUF register), grand-or becomes vx_rx_busy
sfp tx packetizer (data,eop,vx_rx_busy -> data,k)
sfp tx 16/20 serializer, 16 bit at 125 MHz in, 2500 MHz serial out (20 bits at 125 MHz)
sfp tx fiber link to GDM, 2 Gigabits/sec
--- GDM
qsfp rx deserializer, 16 bit at 125 MHz output on main GDM clock
qsfp rx depacketizer (data,k -> packet data,eop; qsfp_rx_busy)
qsfp rx demux, hit map packets routed to GDM trigger logic block
GDM trigger logic block looks at hit map, generates yes/no trigger decision, encodes it as a trigger packet
TSM generator encodes GPS time data as a TSM packet
qsfp tx mux - trigger packets, tsm packets, etc to qsfp tx packet stream
qsfp tx packetizer (data,eop;trg,tsm,bsy -> data,k)
NB: the same tx data is sent to all 12 qsfp tx ports, to make sure we do not accidentally desync the CDMs.
qsfp tx 16/20 serializer, 16-bit at 125 MHz to 20-bit at 250 MHz to 2500 MHz serial out
qsfp tx fiber link to CDM, 2 Gigabits/sec
--- CDM
sfp rx 20/16 deserializer, 16-bit at 125 MHz
sfp rx depacketizer, (data,k -> data,eop;trg_in,tsm_in,bsy_in)
sfp rx demux (in reality, noop, all packets go to same place, vx tx)
vx tx mux (packets from GDM, packets with simulated waveforms from CDM midas frontend via AXI FIFO)
vx tx packet adapter 16-bit at 125 MHz to 8-bit at 12.5 MHz strobed every 10 clocks. NB: most important, this 10-clock strobe runs in sync between all CDMs!
vx tx phase transfer from main CDM clock to vxN_tx clock, have 10 clocks for transfer to happen
from here we run on the vxN_tx clock
vx txN serializer, 8-bit at 12.5 MHz to 125 MHz serial
vx_txN OBUF register, vxN_rx clock, 125 MHz clock, from PLL with adjustable phase, scan phase to find sweet spot
lvds line to vx
BBB: bsy_in from GDM is converted from pulse to level, goes out lvds line to vx, sync to vxN_tx clock
TTT: trg_in from GDM is a pulse, does out lvds line to vx, sync to vxN_tx clock.
NB: tsm is always a packet, bsy is always a signal (no packet), trg can be a packet or signal.
--- VX
lvds data captured by 125 MHz ADC sampling clock (CAEN base firmware logic)
lvds data connected to Yair's block
10/8 deserializer
depacketizer (data,k -> data,eop) to avoid accidental desync, we do not send any no trg, no tsm, no bsy K-codes.
demux
trigger packets go their way (4x 64 bit words of data go to event header: timestamps, hitmap data)
tsm packets go their way (4x 64 bit words of data go to tsm event header: timestamps, GPS time data)
BBB: bsy from lvds line stops waveform acquisition
TTT: trg from lvds line cause waveform acquisition, same as trg packet, but has no timestamp and other data attached to it.
</pre>

== CDM-VX connections ==

Note:
* rounded boxes are VX registers set from MIDAS

[[File:CDM-VX.drawio.svg]]

= Firmware registers =

== Block 0 ==

<pre>
0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
</pre>

== Block 0 GDM ==

* 0x0000+4*i - GDM AXI clock registers

<pre>
register_data_out_block_0(1) <= x"204D4447"; --GDM<space>
register_data_out_block_0(2) <= x"00000000"; --blank
register_data_out_block_0(3) <= x"00000000"; --blank

--Firmware Version
register_data_out_block_0(4) <= x"abcd0123"; --Firmware version name register
register_data_out_block_0(5) <= x"00000001"; --Firmware version number register
register_data_out_block_0(6) <= x"abcd0123"; --Firmware version register
register_data_out_block_0(7) <= x"00000001"; --Firmware version register

--Git Commit/Tag/Compile Date
register_data_out_block_0(8) <= x"0123abcd"; --Firmware Git info (0-3)
register_data_out_block_0(9) <= x"00000001"; --Firmware Git info (1)
register_data_out_block_0(10) <= x"00000001"; --Firmware Git info (2)
register_data_out_block_0(11) <= x"00000001"; --Firmware Git info (3)

--Simple loopback register for testing
register_data_out_block_0(12) <= register_data_in_block_0(12);

--Hardware registers (HW_ID_xDM, etc)
--register_data_out_block_0(13) <= x"0000000" & "000" & hardware_id;

--unspecified
set_reg_block_0_gen : for i in 14 to nregisters_block_0-1 generate
register_data_out_block_0(i) <= register_data_in_block_0(i);
end generate set_reg_block_0_gen;

register_data_out_block_0(13) <= x"000000" & rtc_i2c_data_ie & rtc_i2c_data_out & rtc_i2c_clock_ie & rtc_i2c_clock_out & rtc_1pps_in & rtc_32khz_in & rtc_i2c_data_in & rtc_i2c_clock_in;

rtc_i2c_clock_ie <= not register_data_in_block_0(14)(0);
rtc_i2c_clock_out <= register_data_in_block_0(14)(1);
rtc_i2c_data_ie <= not register_data_in_block_0(14)(4);
rtc_i2c_data_out <= register_data_in_block_0(14)(5);
</pre>

== Block 0 CDM ==

* 0x0000+4*i - CDM AXI clock registers:

<pre>
USR_ACCESS_7series_inst: USR_ACCESSE2
port map(-- CFGCLK => CFGCLK,
DATA => register_data_out_block_0(0)); -- where timestamp is contained
-- DATAVALID => DATAVALID);

--static values
register_data_out_block_0(1) <= x"204D4443"; --CDM<space>
register_data_out_block_0(2) <= x"00000000"; --blank
register_data_out_block_0(3) <= x"00000000"; --blank

--Firmware Version
register_data_out_block_0(4) <= x"abcd0123"; --Firmware version name register
register_data_out_block_0(5) <= x"00000001"; --Firmware version number register
register_data_out_block_0(6) <= x"abcd0123"; --Firmware version name register
register_data_out_block_0(7) <= x"00000001"; --Firmware version number register

--Git Commit/Tag/Compile Date
register_data_out_block_0(8) <= x"0123abcd"; --Firmware Git info (0-3)
register_data_out_block_0(9) <= x"00000001"; --Firmware Git info (1)
register_data_out_block_0(10) <= x"00000001"; --Firmware Git info (2)
register_data_out_block_0(11) <= x"00000001"; --Firmware Git info (3)

--Simple loopback register for testing
register_data_out_block_0(12) <= register_data_in_block_0(12);

--Hardware registers (HW_ID_xDM, etc)
--register_data_out_block_0(13) <= x"0000000" & "000" & hardware_id;

--unspecified
set_reg_block_0_gen : for i in 14 to nregisters_block_0-1 generate
register_data_out_block_0(i) <= register_data_in_block_0(i);
end generate set_reg_block_0_gen;

rx_clk_sel <= register_data_in_block_0(12)(0); -- rx_clk select 0:clk_cc_out0, 1:rx_clk_mgt

rtc_i2c_clock_ie <= not register_data_in_block_0(14)(0);
rtc_i2c_clock_out <= register_data_in_block_0(14)(1);
rtc_i2c_data_ie <= not register_data_in_block_0(14)(4);
rtc_i2c_data_out <= register_data_in_block_0(14)(5);

register_data_out_block_0(13) <= x"000000" & rtc_i2c_data_ie & rtc_i2c_data_out & rtc_i2c_clock_ie & rtc_i2c_clock_out & rtc_1pps_in & rtc_32khz_in & rtc_i2c_data_in & rtc_i2c_clock_in;
</pre>

== Block 1 GDM ==

* 0x0800+4*i - GDM registers:

<pre>
clk_config_pro : process (read_write_clk)
begin
if rising_edge (read_write_clk) then
CLK_IN_SEL_LS <= clk_config_vec(1 downto 0);
CLK_EXT_SEL_LS <= clk_config_vec(2);
CLK_RSTn_LS <= not clk_config_vec(3);
clk_config_vec(4) <= CLK_LOSXTn_LS;
clk_config_vec(5) <= CLK_LOLn_LS;
clk_config_vec(6) <= CLK_INTn_LS;
clk_config_vec(7) <= '1';

clk_config_vec(3 downto 0) <= register_data_in_block_1(0)(3 downto 0);
register_data_out_block_1(0)( 7 downto 0) <= clk_config_vec(7 downto 0);
register_data_out_block_1(0)(31 downto 8) <= (others =>'0');
register_data_out_block_1(1) <= (others =>'0'); --free
end if;
end process clk_config_pro;

register_data_out_block_1(2) <= register_data_in_block_1(2);
register_data_out_block_1(3) <= register_data_in_block_1(3);
register_data_out_block_1(4)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(5)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(6)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(7)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(8) <= register_data_in_block_1(8);
register_data_out_block_1(9)(31 downto nlinks) <= (others => '0');
register_data_out_block_1(10) <= (others => '0');
register_data_out_block_1(11) <= (others => '0');

frequency_measure_clks <= read_write_clk & clk_50MHz & mgt_data_clk & mgt_ref_clk_raw; -- & mgt_data_clk & mgt_ref_clk_raw;
register_data_out_block_1(12) <= register_data_out_block_1(14); --MGT rx and tx reference clocks are the same
register_data_out_block_1(13) <= register_data_out_block_1(15); --MGT rx and tx data clocks are the same
frequency_measure_inst : frequency_measure
generic map(register_clk_frequency => 100000000,
max_frequency => 300000000,
nfrequency_measurments => nfrequency_measure_clks)
port map(i_register_clk => read_write_clk,
i_clks => frequency_measure_clks,
i_clk_locks => (others =>'1'),
o_frequencies => register_data_out_block_1(14+nfrequency_measure_clks-1 downto 14)); --see register_data_out_block_1(12,13) above

zero_spare_link_registers : for i in 14+nfrequency_measure_clks to nregisters_block_1-1 generate
register_data_out_block_1(i) <= (others => '0');
end generate zero_spare_link_registers;

gdm_link_interface_inst : gdm_link_interface
generic map(enable_rxslide => '1',
nquads => nquads,
nlinks => nlinks)
port map(i_mgt_ref_clk_p => mgt_b_ref_clk_p,
i_mgt_ref_clk_n => mgt_b_ref_clk_n,
i_mgt_rx_n => mgt_b_rx_n,
i_mgt_rx_p => mgt_b_rx_p,
o_mgt_tx_n => mgt_b_tx_n,
o_mgt_tx_p => mgt_b_tx_p,
--raw, pre-pll, mgt clock for frequency measurement
o_mgt_ref_clk_raw => mgt_ref_clk_raw,
--free running clock and reset
i_mgt_free_running_clk => clk_50MHz,
i_mgt_rst => register_data_in_block_1(2)(0),
i_rx_slide_trigger => register_data_in_block_1(3)(nlinks-1 downto 0),
--data interface for the links
o_data_clk => mgt_data_clk,
o_mgt_data_link_array => mgt_link_data_to_processing,
i_mgt_data_link_array => mgt_link_data_from_processing,
--status ports
o_link_power_good => register_data_out_block_1(4)(nlinks-1 downto 0),
o_tx_link_up => register_data_out_block_1(5)(nlinks-1 downto 0),
o_rx_link_up => register_data_out_block_1(6)(nlinks-1 downto 0),
o_rx_error => register_data_out_block_1(7)(nlinks-1 downto 0),
i_link_down_latched_rst => register_data_in_block_1(8)(0),
o_link_down_latched => register_data_out_block_1(9)(nlinks-1 downto 0),
--debug
i_debug_clk => read_write_clk,
o_debug => mgt_debug_data);
</pre>

<pre>
#### Register Block 1 (CDM)
Clock Chip and MGT Link, Control and Status Registers
| Register | Axi Address | Description | note
|:-|:-|:-|:-|
| 0 | 0x80011000 | Clock Chip Status and Control | CLK_IN_SEL_LS(1-0)(read-only), CLK_EXT_SEL_LS(2), CLK_RSTn_LS(3), CLK_LOSXTn_LS(4), CLK_LOLn_LS(5), CLK_I\
NTn_LS(6)
| 1 | 0x80011004 | free for use | |
| 2 | 0x80011008 | MGT Link Reset | mgt_rst(0), Reset for RX Error Latched(1)|
| 3 | 0x8001100c | MGT rx slide control | not needed or used |
| 4 | 0x80011010 | link_up_and_running(10) tx_link_up_and_running(9), tx_sending_data(8), tx_link_up(7), rx_link_up_and_running(6), rx_error(5), rx_receivi\
ng_data(4), rx_link_up(3), link_power_good(2), sfp_rx_los(1), sfp_mod_absent(0) | Should be 0x7DC |
| 5 | 0x80011014 | reserved debug register | |
| 6-11 | ... | free for use | |
| 12 | 0x80011030 | mgt_rx_ref_clk_raw | ~0x7735940 (125 MHz) |
| 13 | 0x80011034 | rx_clk | ~0x7735940 (125 MHz) |
| 14 | 0x80011038 | mgt_tx_ref_clk_raw | ~0x7735940 (125 MHz) |
| 15 | 0x8001103c | tx_clk | ~0x7735940 (125 MHz) |
| 16 | 0x80011040 | clk_50MHz | 0x2faf080 (50 MHz) |
| 17 | 0x80011044 | Register_Block1_clk | 0x5f5e100 (100 MHz) |
| 18-20 | ... | free for use | |
| 21 | 0x80011054 | free for use | |
</pre>

=== Register 0x1000 ===

<pre>
printf("0x1000 SFP c.c. status: 0x%08x\n", reg1000);
printf(" CLK_IN_SEL_LS 0x%x\n", (reg1000>>0) & 3);
printf(" CLK_EXT_SEL_LS %d\n", (reg1000>>2) & 1);
printf(" CLK_CLK_RSTn_LS %d\n", (reg1000>>3) & 1);
printf(" CLK_LOSXTn_LS %d\n", (reg1000>>4) & 1);
printf(" CLK_LOLn_LS %d\n", (reg1000>>5) & 1);
printf(" CLK_INTn_LS %d\n", (reg1000>>6) & 1);
printf("0x1008 SFP link reset: 0x%08x\n", dev->dsdm_read32(0x1008));
</pre>

=== Register 0x1010 ===

<pre>
printf("0x1010 SFP link status: 0x%08x\n", reg1010);
printf(" sfp_mod_absent_N %d\n", (reg1010>>0) & 1);
printf(" sfp_rx_los_N %d\n", (reg1010>>1) & 1);
printf(" link_power_good %d\n", (reg1010>>2) & 1);
printf(" rx_link_up %d\n", (reg1010>>3) & 1);
printf(" rx_receiving_data %d\n", (reg1010>>4) & 1);
printf(" rx_error %d\n", (reg1010>>5) & 1);
printf(" rx_lnk_up_and_running %d\n", (reg1010>>6) & 1);
printf(" tx_link_up %d\n", (reg1010>>7) & 1);
printf(" tx_sending_data %d\n", (reg1010>>8) & 1);
printf(" tx_link_up_and_running %d\n", (reg1010>>9) & 1);
printf(" link_up_and_running %d\n", (reg1010>>10) & 1);

</pre>

=== Register 0x1014 ===

<pre>
printf("0x1014 SFP link data: 0x%08x\n", reg1014);
printf(" rx_data 0x%04x\n", (reg1014>>0) & 0xFFFF);
printf(" k28p1_k28p5 %d\n", (reg1014>>16) & 1);
printf(" rx_ctrl0 0x%01x\n", (reg1014>>17) & 3);
printf(" rx_ctrl1 0x%01x\n", (reg1014>>19) & 3);
printf(" rx_ctrl3 0x%01x\n", (reg1014>>21) & 1);
printf(" tx_state 0x%01x\n", (reg1014>>22) & 3);
printf(" rx_state 0x%01x\n", (reg1014>>24) & 0xF);
printf(" rx_receiving_data %d\n", (reg1014>>28) & 1);
printf(" rx_link_up %d\n", (reg1014>>29) & 1);
printf(" rx_error %d\n", (reg1014>>30) & 1);
printf(" rx_link_rst %d\n", (reg1014>>31) & 1);
</pre>

* see cdm_link_establishment.vhd
* tx_state:
<pre>
0 = tx_idle
1 = waiting_for_rx_link
2 = sending_k28p1_k28p5
3 = sending_data
</pre>

* rx_state:
<pre>
0 = rx_idle
1 = waiting_for_ks
2 = rx_retry
3 = rx_up_switch_tx_clk_source
4 = waiting_for_data
5 = receiving_data
6 = rx_receive_error
</pre>

== Block 2 GDM ==

* 0x1000+4*i - GDM registers:

<pre>
register_data_out_block_2(0) <= register_data_in_block_2(0);
GDM_link_data_processing_inst : GDM_link_data_processing
generic map(mgt_data_frequency => 125000000)
port map(i_register_clk => read_write_clk,
i_rst => register_data_in_block_2(0)(0),
i_data_mode => register_data_in_block_2(0)( 9 downto 8),
i_fixed_pattern_data => register_data_in_block_2(0)(31 downto 16),
o_error_time => register_data_out_block_2(1),
o_error_count => register_data_out_block_2(13 downto 2),
--link data interface
i_data_clk => mgt_data_clk,
i_mgt_data_link_array => mgt_link_data_to_processing,
o_mgt_data_link_array => mgt_link_data_from_processing,
--data interface
o_data_clk => data_clk,
o_tx_data_ready => tx_data_ready,
i_tx_data_is_k => tx_data_is_k,
i_tx_data => tx_data,
o_rx_data_error => rx_data_error,
o_rx_data_is_k => rx_data_is_k,
o_rx_data => rx_data);
</pre>

== Block 2 CDM ==

* 0x1000+4*i - CDM registers:
* reg 0
<pre>
read bits
7..0 - clk_config_vec
1,0 - CLK_IN_SEL_LS
2 - CLK_EXT_SEL_LS
3 - disconnected CLK_RSTn_LS
4 - CLK_LOSXTn_LS
5 - CLK_LOLn_LS
6 - CLK_INTn_LS
7 - tx_clock_status
31..8 - 0
</pre>
<pre>
write bits
3..2 - clk_config_vec[3..2]
</pre>
* reg 1 - 0
* reg 2 - RW
<pre>
bit
0 - cdm_link_interface i_mgt_rst
1 - cdm_link_interface i_link_down_latched_rst
</pre>
* reg 3 - RW
<pre>
bit
0 - cdm_link_interface i_rx_slide_trigger
</pre>
* reg 4 - RO
<pre>
bit
0 - sfp_mod_present
1 - sfp_rx_los
2 - link_power_good
3 - rx_link_up
4 - rx_receiving_data
5 - rx_error
6 - rx_link_up_and_running
7 - tx_link_up
8 - tx_sending_data
9 - tx_link_up_and_running
10 - link_up_and_running
</pre>
* reg 5 - RO - CDM SFP debug_data
* reg 6 - 0
* reg 7 - 0
* reg 8 - 0
* reg 9 - 0
* reg 10 - 0
* reg 11 - 0
* reg 12 - frequency AXI clock
* reg 13 - freq clk_50MHz
* reg 14 - freq clk_100MHz
* reg 15 - freq clk_200MHz
* reg 16 - freq clk_xo_125 - DS-DM 125 MHz oscillator
* reg 17 - freq clk_cc_out0 - C.C. 125 MHz output
* reg 18 - CDM SFP mgt_rx_ref_clk_raw
* reg 19 - CDM SFP mgt_tx_ref_clk_raw
* reg 20 - rx_clk_mgt, 125 MHz CDM SFP RX recovered clock
* reg 21 - rx_clk, 125 MHz, switched between clk_xo_125 and rx_clk_mgt
* reg 22 - tx_clk, 125 MHz
* reg 23 - spare

== Block 3 GDM ==

DS20k registers

== Block 4 CDM ==

* 0x2800+4*i - CDM SFP data link registers:
* 0 - R/W
** bit 0 - 0x1 - reset
** bits 7..1 - not used
** bits 9..8 - data_mode: 00: data from ds20k block, 01: tx_data_cnt, 10: tx_data_prbs, 11: tx_fixed_pattern_data
** bits 15..10 - not used
** bits 31..16 - fixed pattern data
* 1 - RO - o_error_time
* 2 - RO - o_error_count
* 3, etc - not used

== Block 6 CDM ==

* 0x3000+4*i : DS20k registers

== Block 7 CDM ==

VX_TX and VX_RX delays, i=0..11
* 0x3800+4*i - VX_TX delays
* 0x3830+4*i - VX_RX delays

VX_TX delay data (2 bits):
* 4 values: 0,1,2,3 corresponding to the 4 phases of the 125 MHz clock rx_clk and rx_clk shifted by 90 deg

VX_RX delay data (32 bits):
* 0x8000'0000: reset IDELAYE3, follow by write of 0
* 0x0001'xxxx: load the IDELAYE3 9-bit tap counter, follow by write of 0x0000'xxxx
* 16 low bits have:
* bits 0..8 are the 9 bits of IDELAY3 tap counter (~1.1ns for all 511 delays)
* bit 9 is the 500 MHz DDR phase (1 ns)
* bits 10..15 are 16 bits of 500 MHz delay (2 ns step)
* numerically increasing 16-bit delay values are carefully arranged to produce linearly increasing delay (there is some overlap between IDELAYE3 taps above ~tap 500 (or so, IDELAYE3 is uncalibrated) and flip of 500 MHz DDR phase

== DS20k registers ==

=== DS20k block register map ===

* busybox devmem 0x80013000 32

Note: R=readable, W=writable, L=latched by CMD_LATCH, B=reset on begin of run, Z=reset by CMD_LATCH

<pre>
reg | version | xDM | xx | description
0 | 0x20230731 | ALL | RO | ds20k version
0 | 0x20240118 | ALL | RW | ds20k version and command
1 | 0x20230731 | ALL | RW | scratch read/write register
2 | 0x20230731 | ALL | RW | configure inputs and outputs
3 | 0x20230731 | ALL | RW | FP_LED mux
4 | 0x20230731 | ALL | RW | EXT_OUT mux
5 | 0x20230731 | ALL | RO | VX_RX state
6 | 0x20230731 | ALL | RO | VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state
7 | 0x20230731 | ALL | RW | LED_OUT, EXT_OUT, VX_TX outputs
8 | 0x20230731 | ALL | RW | VX_TX mux and config
9 | 0x20230731 | ALL | RW | trg and tsm source
10 | 0x20231013 | ALL | RO | status register
11 | 0x20230731 | ALL | ROLB | trg counter
12 | 0x20230731 | ALL | ROLB | tsm counter
13 | 0x20240814 | ALL | RO | GPS 1pps period, 125 MHz
14 | 0x20240814 | ALL | RO | Ru clock 1pps period, 125 MHz
15 | 0x20230811 | CDM | RO | SFP RX status
16 | 0x20230811 | CDM | RW | SFP TX control
17 | 0x20230811 | GDM | RO | QSFP RX data 0, 1
18 | |. |. | 2, 3
19 | |. |. | 4, 5
20 | |. |. | 6, 7
21 | | |. | 8, 9
22 | |. |. | 10, 11
23 | 0x20230811 | GDM | RW | QSFP TX control
24 | 0x20231013 | ALL | RW | trigger pulser period
25 | 0x20231013 | ALL | RW | trigger pulser burst control
26 | 0x20231013 | ALL | RW | tsm pulser period
27 | 0x20231204 | ALL | RW | data fifo CPU to FPGA
28 | 0x20231204 | ALL | RW | packet loopback control
28 | 0x20240118 | ALL | RW | data fifo FPGA to CPU
29 | 0x20231208 | GDM | RW | bitmap of active qsfp ports
29 | 0x20240118 | ALL | RW | packet loopback control
29 | 0x20240510 | ALL | RW | packet routing
30 | 0x20231208 | GDM | RO | qsfp link status ports 0..7
31 | 0x20231208 | GDM | RO | qsfp link status ports 8..11
32 | 0x20240118 | GDM | RW | enabled QSFP ports
33 | 0x20240118 | CDM | RW | enabled VX ports
34,35 | 0x20240118 | ALL | ROB | time stamp 64 bits
36,37 | 0x20240118 | ALL | RO | old time stamp
38,39,40 | 0x20240118 | CDM | ROLZ | vx_bsy_counter_latched | VX busy counters
41,42,43 | 0x20240118 | GDM | ROLZ | QSFP busy counters
44 | 0x20240118 | CDM | ROLZ | cdm_bsy_up_counter and cdm_bsy_pulse_counter
45 | 0x20240118 | CDM | ROLZ | cdm_veto_pulse_counter and cdm_veto_up_counter
46 | 0x20240118 | GDM | ROLZ | gdm_bsy_pulse_counter and gdm_bsy_up_counter
47 | 0x20240118 | GDM | ROLZ | gdm_bsy_refresh_counter
48 | 0x20240118 | GDM | ROLZ | gdm_veto_up_counter and gdm_veto_pulse_counter
49,50,51 | 0x20240424 | CDM | RO | VX RX serial data monitor, 8 bits per VX port
52,53 | 0x20240424 | CDM | RO | VX serial link status, 4 bits per VX port
54 | 0x20240424 | CDM | RO | VX TX serial data monitor
55 | 0x20240430 | ALL | RO | QSFP, SFP and VX link loss counters
56 | 0x20240510 | ALL | RO | sfp_rx_packet_counter
57 | same | ALL | RO | sfp_tx_packet_counter
58 | same | ALL | RO | qsfp_rx_packet_counter[0]
59 | same | ALL | RO | qsfp_tx_packet_counter
60 | same | ALL | RO | vx_rx_packet_counter[0]
61 | same | ALL | RO | vx_tx_packet_counter
62 | 0x20240719 | CDM | RO | cdm_hitmap_period, ports 0, 1
63 | same | CDM | RO | ports 2, 3
64 | same | CDM | RO | ports 4, 5
65 | same | CDM | RO | ports 6, 7
66 | same | CDM | RO | ports 8, 9
67 | same | CDM | RO | ports 10, 11
68 | 0x20240814 | ALL | RW | GPS control and status
69 | 0x20241104 | CDM | ROLB | vx_tx_trg_packet_counter | counter of TRG packets CDM->VX
70 | 0x20241104 | CDM | ROLB | vx_tx_tsm_packet_counter | counter of TSM packets CDM->VX
71 | 0x20241104 | ALL | RO | packet error bits
72,73 | 0x20241104 | CDM | ROL | vx_tx_trg_data_latched[63:0] trigger packet data
74,75 | 0x20241104 | CDM | ROL | vx_tx_tsm_data_latched[63:0] tsm packet data (truncated to 64 bits)
76 | 0x20241104 | CDM | ROLB | cdm_hitmap_trigger_counter_latched, counter of hitmap triggers generated in the CDM
77,78,79 | 0x20241104 | CDM | ROL | vx_rx_hitmap_data_latched | VX_RX 80 bits of HITMAP packet from VX
80 | 0x20241209 | CDM | RO | vx_lvds_rx_link_status | VX LVDS link status
81 | 0x20241209 | CDM | RO | vx_lvds_rx_trg_data | VX LVDS TRG packet first 32 bits
81 | 0x20251010 | CDM | - | not used | not used
82 | 0x20241209 | CDM | RO | vx_lvds_rx_tsm_data | VX LVDS TSM packet first 32 bits
82 | 0x20251010 | CDM | - | not used | not used
83 | 0x20241209 | CDM | ROL | vx_lvds_rx_decoded_packet_counter_latched | VX LVDS decoded packet counter
84 | 0x20241209 | CDM | ROL | vx_lvds_rx_packet_error_counter_latched | VX LVDS packet error counter
85 | 0x20250417 | CDM | ROLB | vx_tx_trg_counter | VX LVDS TRG counter (VX LVDS pair 12)
86 | 0x20250417 | CDM | ROLB | vx_tx_tsm_counter | VX LVDS TSM counter (VX LVDS pair 13)
87 | 0x20250417 | CDM | ROLB | vx_tx_veto_counter | VX LVDS Veto counter (VX LVDS pair 13)
88 | 0x20250422 | GDM | ROLB | trg_in_lost_counter | counter of TRG killed by gdm_busy
89 | 0x20250422 | ALL | RO | fifo_from_fpga_lost_packet_counter | counter of packets lost due to FIFO full
90 | 0x20250428 | CDM | ROL | vx_bsy_status_latched | status of VX busy, CDM busy, CDM veto
91 | 0x20250428 | CDM | ROL | vx_hitmap_status_latched | status of VX hitmap
92 | 0x20250428 | GDM | ROL | cdm_bdy_status_latched | status of CDM busy, GDM busy, GDM veto
93 | 0x20250508 | GDM | ROL | gdm_hitmap_status_latched | status of GDM hitmap
94 | 0x20250508 | GDM | ROLB | gdm_hitmap_trigger_counter_latched | counter of hitmap triggers generated in the GDM
95 | 0x20250512 | GDM | ROLB | gdm_trig48_counter_latched | counter of TRIG48 trigger packets
96 | 0x20250512 | GDM | ROLB | gdm_trig48_lost_counter_latched | counter of TRIG48 triggers lost to busy
97 | 0x20250512 | CDM | ROLB | vx_tx_trig48_packet_counter_latched | counter of TRIG48 packets from CDM to VX
98, 99 | 0x20250512 | GDM | RW | trig48_ctrl48, trig48_ctrl16 | GDM TRIG48 test registers
100,101 | 0x20251008 | CDM | ROL | vx_lvds_rx_trig48_{data,bitmap48}_latched | VX LVDS TRIG48 packet first 32 bits, bitmap48 first 32 bits
102 | 0x20251008 | ALL | RO | cmd_counter | counter of reg0 commands
</pre>

=== Register 0 0x80013000 ds20k version ===

on read: ds20k version 0xYYYYMMDD

on write:
* 0 - noop - as of version 0x20240118, write a zero after writing a command
* 1 - cmd_reset - reset logic to good state
* 2 - cmd_arm_ts - arm timestamp reset
* 3 - cmd_trg - issue a trigger
* 4 - cmd_tsm - issue a tsm
* 5 - cmd_vx_rx_reset - reset the VX receive path
* 6 - cmd_vx_tx_reset - reset the VX transmit path
* 7 - cmd_gdm_trig48 - generate a TRIG48 trigger and data packet (GDM)
* 8 - cmd_trg_pulser_reset - reset the trigger pulser
* 9 - cmd_tsm_pulser_reset - reset the tsm pulser
* 10 - cmd_bor_start - start begin-of-run trigger sequence
* 11 - cmd_bor_clear - after run has started, clear begin-of-run status bits
* 12 - cmd_latch - latch counters & etc into AXI registers for coherent readout
* 13 - cmd_cdm_hitmap_trigger - generate a hitmap trigger and data packet (CDM)

=== Register 1 0x80013004 scratch ===

scratch read-write register

=== Register 2 0x80013008 input and output config ===

<pre>
bit | version | fpga name | description
0 | ALL | lemo_enable | enable LEMO input 1
1 | | | 2
2 | | | 3
3 | | | 4
4 | ALL | lemo_invert | invert LEMO input 1
5 | | | 2
6 | | | 3
7 | | | 4
9 | ALL | ext_out_disable | disable LEMO output 1
10 | | | 2
11 | ALL | ext_out_invert | invert LEMO output 1
12 | | | 2
</pre>

=== Register 3 0x8001300C FP_LED control ===

<pre>
wire [15:0] led_out_mux_sel = register_data_in[3][15:0];
wire [3:0] led_out_invert = register_data_in[3][19:16];
</pre>

<pre>
led_out_mux_sel is 4 groups (one per LED) of 4 bits (choice 0..15):
mux | version | fpga name | description
0 | ALL | | power on default
1 | ALL | led_out_reg | register 7 bits
2 | 0x20231013 | pll_locked | clock chip PLL is locked
3 | 0x20231013 | sfp_link_status | SFP link is good
3 | 0x20240118 | sfp_link_status or qsfp_tx_link_rx_status | SFP/QSFP link is good
4 | ALL | lemo_in_sync[1] | LEMO input 1
5 | ALL | lemo_in_sync[2] | LEMO input 2
6 | ALL | lemo_in_sync[3] | LEMO input 3
7 | ALL | lemo_in_sync[4] | LEMO input 4
8 | ALL | ext_out[1] | LEMO output 1
9 | ALL | ext_out[2] | LEMO output 2
A | 0x20231013 | trg_in | trigger
B | 0x20231013 | tsm_in | time slice marker
C | 0x20240118 | gdm_bsy | GDM busy: OR of all CDM busy
D | 0x20240118 | cdm_bsy | CDM busy: OR of all VX busy
E | 0x20240118 | cdm_veto | GDM busy -> GDM veto -> CDM veto -> VX trigger veto
F | ALL | | fixed logic level 1
</pre>

=== Register 4 0x80013010 LEMO OUT control ===

<pre>
wire [7:0] ext_out_mux_sel = register_data_in[4][7:0];
</pre>

<pre>
ext_out_mux_sel is 2 groups (one per LEMO) of 4 bits (choice 0..15):

mux | version | fpga name | description
0 | ALL | | power on default
1 | ALL | ext_out_reg | register 7
2 | ALL | trg_pulser | pulser trigger
3 | 0x20240724 | vx1_tx_out[2] | vx1 serial data out
4 | 0x20240724 | vx1_rx[1] | vx1 serial data in
5 | 0x20240724 | vx_rx_iob[0] | vx1 serial data in captured by IOB register
6 | ALL | lemo_in_async[1]| test synchronizer
7 | ALL | lemo_in_sync[1] | test synchronizer
8 | 0x20240724 | trg_in_pulse | trigger signal
9 | 0x20240724 | tsm_in_pulse | time slice marker signal
A | 0x20240118 | cdm_bsy | CDM busy from VX
B | 0x20240118 | gdm_bsy | GDM busy from CDM
C | 0x20240118 | cdm_veto | veto from GDM to CDM to VX
D | 0x20240118 | vx1_rx[1] | serial data VX to CDM
E | not used (sink)
F | ALL | 1 | fixed logic level 1
</pre>

=== Register 5 0x80013014 VX_RX status ===

<pre>
assign register_data_out[5] = {
vx8_rx[3], vx8_rx[2], vx8_rx[1], vx8_rx[0],
vx7_rx[3], vx7_rx[2], vx7_rx[1], vx7_rx[0],
vx6_rx[3], vx6_rx[2], vx6_rx[1], vx6_rx[0],
vx5_rx[3], vx5_rx[2], vx5_rx[1], vx5_rx[0],
vx4_rx[3], vx4_rx[2], vx4_rx[1], vx4_rx[0],
vx3_rx[3], vx3_rx[2], vx3_rx[1], vx3_rx[0],
vx2_rx[3], vx2_rx[2], vx2_rx[1], vx2_rx[0],
vx1_rx[3], vx1_rx[2], vx1_rx[1], vx1_rx[0]
};
</pre>

=== Register 6 0x80013018 VX_RX, LEMO_IN, VX_TX, FP_LED, EXT_OUT state ===

<pre>
assign register_data_out[6] = {
vx2_tx_out[2], vx2_tx_out[1], vx2_tx_out[0], vx1_tx_out[2],
vx1_tx_out[1], vx1_tx_out[0], ext_out[2], ext_out[1],
fp_led_out[3], fp_led_out[2], fp_led_out[1], fp_led_out[0],
ext_in_lv[4], ext_in_lv[3], ext_in_lv[2], ext_in_lv[1],
vx12_rx[3], vx12_rx[2], vx12_rx[1], vx12_rx[0],
vx11_rx[3], vx11_rx[2], vx11_rx[1], vx11_rx[0],
vx10_rx[3], vx10_rx[2], vx10_rx[1], vx10_rx[0],
vx9_rx[3], vx9_rx[2], vx9_rx[1], vx9_rx[0]
};
</pre>

<pre>
bit | version | fpga name | description
0 | ALL | vx9_rx | VX_RX
1 | | |
2 | | |
3 | | |
4 | ALL | vx10_rx | VX_RX
5 | | |
6 | | |
7 | | |
8 | ALL | vx11_rx | VX_RX
9 | | |
10 | | |
11 | | |
12 | ALL | vx12_rx | VX_RX
13 | | |
14 | | |
15 | | |
16 | ALL | ext_in_lv | LEMO inputs
17 | | |
18 | | |
19 | | |
20 | ALL | FP_LED | FP_LEDs
21 | | |
22 | | |
23 | | |
24 | ALL | ext_out[1] | LEMO outputs
25 | | ext_out[2] |
26 | ALL | vx1_tx | VX1_TX
27 | | |
28 | | |
29 | ALL | vx2_tx | VX2_TX
30 | | |
31 | | |
</pre>

=== Register 7 0x8001301C LED_OUT, EXT_OUT, VX_TX outputs ===

<pre>
wire [3:0] led_out_reg = register_data_in[7][3:0];
wire [2:1] ext_out_reg = register_data_in[7][5:4];
// register_data_in[7][6];
// register_data_in[7][7];
wire [7:0] vx_tx_out_reg = register_data_in[7][15:8];
</pre>

<pre>
bit | version | fpga name | description
0 | ALL | led_out_reg | FP_LED 1
1 | | | 2
2 | | | 3
3 | | | 4
4 | ALL | ext_out_reg | LEMO OUT 1
5 | | | 2
6 | - | |
7 | - | |
8 | ALL | vx_tx_out_reg | VX1_TX 0
9 | | | 1
10 | | | 2
11 | | | -
12 | ALL | | VX2_TX 0
13 | | | 1
14 | | | 2
15 | | | -
16 | - | |
17 | | |
18 | | |
19 | | |
20 | | |
21 | | |
22 | | |
23 | | |
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 8 0x80013020 VX_TX config ===

<pre>
wire [3:0] vx_tx_mux_sel = register_data_in[8][3:0];
wire vx_tx_from_sfp = register_data_in[8][30];
wire vx_tx_loopback = register_data_in[8][31];
</pre>

<pre>
vx_tx_mux_sel is 4 bits (choice 0..15):
0 - power on default, control by vx_tx_out_reg
1 - GDM
2 - CDM
3 - pulser loopback test
4 - pulser loopback test
5 - 62.5 MHz output
6 - 125 MHz output
7 - trg, tsm, serial
8 - trg, tsm, lvds serial rx to serial tx loopback
9 - GPS box control (ds20k rev 0x20240814)
10
11
12
13
14
15 - production config: trg, veto, serial
</pre>

=== Register 9 trg and tsm source ===

from version 0x20240724

<pre>
wire [15:0] trg_src_mask = register_data_in[9][15:0];
wire [31:16] tsm_src_mask = register_data_in[9][31:16];
</pre>

<pre>
wire [15:0] trg_src_bits =
{
1'b0, // 15
1'b0, // 14
vx_tx_tsm_done, // 13
vx_tx_trg_done, // 12

1'b0, // gdm_hitmap_trigger, // 11
cdm_hitmap_trigger, // 10
sfp_rx_tsm, // 9
sfp_rx_trg, // 8

sfp_rx_data[1], // 7
sfp_rx_data[0], // 6
tsm_pulser, // 5
trg_pulser, // 4

lemo_in_sync[4], // 3
lemo_in_sync[3], // 2
lemo_in_sync[2], // 1
lemo_in_sync[1] // 0
};

wire [15:0] trg_bits = trg_src_bits & trg_src_mask;
wire [15:0] tsm_bits = trg_src_bits & tsm_src_mask;
</pre>

before that:

<pre>
wire [7:0] trg_src_mask = register_data_in[9][7:0];
wire [7:0] tsm_src_mask = register_data_in[9][15:8];
wire trg_pulser_enable = register_data_in[9][16];
wire tsm_pulser_enable = register_data_in[9][17];
wire trg_software = register_data_in[9][18];
wire tsm_software = register_data_in[9][19];
// bits 20:31 not used
</pre>

<pre>
wire [7:0] xxx_src_bits =
{
sfp_rx_data[1],
sfp_rx_data[0],
tsm_pulser & tsm_pulser_enable,
trg_pulser & trg_pulser_enable,
lemo_in_sync[4],
lemo_in_sync[3],
lemo_in_sync[2],
lemo_in_sync[1]
};
</pre>

<pre>
trg_src_mask and tsm_src_mask bits:
0 - LEMO IN 1
1 - LEMO IN 2
2 - LEMO IN 3
3 - LEMO IN 4
4 - trg_pulser
5 - tsm_pulser
6 - sfp_rx_data[0] // to become sfp_trg_in, selected from sfp_rx_data[0], cdm_rx trg_out and gdm trigger packet
7 - sfp_rx_data[1] // to become sfp_tsm_in, selected from sfp_rx_data[1], cdm_rx tsm_out and gdm tsm packet
</pre>

=== Register 10 0x80013028 status register ===

<pre>
bit | version | fpga name | description
0 | 0x20231013 | pll_locked | clock chip PLL is locked
1 | 0x20240118 | ts_reset_armed | timestamp reset is armed
2 | 0x20240118 | qsfp_tx_link_rx_status | QSFP link status is good for all enabled ports
3 | 0x20240424 | vx_tx_link_rx_status | VX link status is good for all enabled ports
4 | 0x20240118 | cdm_bsy | VX busy grand-or
5 | 0x20240118 | gdm_bsy | QSFP busy grand-or
6 | 0x20240118 | gdm_veto | gdm_veto = gdm_busy
7 | 0x20240118 | cdm_veto | CDM veto from GDM to VX
8 | 0x20240725 | bor_started | begin-of-run sequence started
9 | 0x20240725 | bor_finished | begin-of-run sequence fininished, see commands 10 and 11
10 | | |
11 | | |
12 | | |
13 | | |
14 | | |
15 | | |
16 | | |
17 | | |
18 | | |
19 | | |
20 | | |
21 | | |
22 | | |
23 | | |
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 11 0x8001302C trg_counter ===

trigger counter

=== Register 12 0x80013030 tsm_counter ===

time slice marker counter

=== Register 13 0x80013034 GPS 1pps period ===

GPS 1pps period in 8 ns clocks

=== Register 14 0x80013038 Rb clock 1pps period ===

PRS-10 Rb clock 1pps output period in 8 ns clocks

=== Register 15 0x8001303C SFP RX status ===

<pre>
bit | ds20k version | fpga signal name | description
0 | ALL | sfp_rx_data[15:0] | cdm sfp received data
16 | same | sfp_rx_data_is_k[0] |
17 | same | sfp_rx_data_is_k[1] |
18 | | 0 |
19 | 0x20231204 | sfp_rx_sel_lpb | sfp tx->rx loopback
20 | 0x20231013 | sfp_link_status | sfp link connected, exchanging data
21 | same | sfp_link_rx_status | sfp link receiving correct idle pattern from GDM TX
22 | same | sfp_link_error | sfp link receiver error (badk or overflow)
23 | same | sfp_rx_data_error | sfp transceiver state machine is in error state
24 | | |
25 | | |
26 | | |
27 | | |
28 | | |
29 | | |
30 | | |
31 | | |
</pre>

=== Register 16 SFP TX control ===

<pre>
wire [15:0] sfp_tx_data_reg = register_data_in[16][15:0]; // SFP TX data K-code
wire [1:0] sfp_tx_ctrl_reg = register_data_in[16][17:16]; // SFP TX data K-code
// 18
// 19
// 23:20
// 27:24
wire sfp_rx_sel_lpb = register_data_in[16][28]; // TX->RX serial loopback
wire sfp_tx_sel_lpb = register_data_in[16][29]; // RX->TX serial loopback
wire sfp_tx_sel_trg = register_data_in[16][30]; // 16 individual bits
wire sfp_tx_sel_reg = register_data_in[16][31]; // TX data from register 16 bits 17:0
</pre>

=== Register 17-22 QSFP RX data ===

QSFP RX data links 0..11

=== Register 23 QSFP TX control ===

<pre>
wire [15:0] qsfp_tx_data_reg = register_data_in[23][15:0];
wire [1:0] qsfp_tx_ctrl_reg = register_data_in[23][17:16];

wire qsfp_tx_enable_trg = register_data_in[23][24]; // enable QSFP TX trg_in_pulse k-code
wire qsfp_tx_enable_tsm = register_data_in[23][25]; // enable QSFP TX tsm_in_pulse k-code

//wire qsfp_rx_sel_lpb = register_data_in[23][28]; // TX->RX loopback
wire qsfp_tx_sel_lpb = register_data_in[23][29]; // RX->TX loopback
wire qsfp_tx_sel_trg = register_data_in[23][30]; // 16 individual bits
wire qsfp_tx_sel_reg = register_data_in[23][31]; // data from register
</pre>

=== Register 24 0x80013060 trigger pulser period ===

trigger pulser period in units of 8 ns (125 MHz clock)

=== Register 25 0x80013064 trigger burst pulser ===

<pre>
wire [7:0] conf_burst_count = conf_pulser_burst_ctrl[31:24];
wire [23:0] conf_burst_period = conf_pulser_burst_ctrl[23:0];
</pre>

=== Register 26 0x80013068 tsm pulser period ===

time slice marker period in units of 8 ns (125 MHz clock)

=== Register 27 0x8001306C data write fifo ===

<pre>
wire fifo_reset = register_data_in[27][31];
wire fifo_to_fpga_wr1 = register_data_in[27][27];
wire fifo_to_fpga_wr2 = register_data_in[27][26];
wire [16:0] fifo_to_fpga_din = register_data_in[27][16:0];
</pre>

<pre>
assign register_data_out[27][31:24] = register_data_in[27][31:24]; // echo write bits
assign register_data_out[27][23] = fifo_to_fpga_full;
assign register_data_out[27][22] = fifo_to_fpga_empty;
</pre>

=== Register 28 0x80013070 data read fifo ===

<pre>
wire fifo_reset = register_data_in[28][31];
wire fifo_from_fpga_rd1 = register_data_in[28][25];
wire fifo_from_fpga_rd2 = register_data_in[28][24];
</pre>

<pre>
assign register_data_out[28][31:24] = register_data_in[28][31:24]; // echo write bits
assign register_data_out[28][21] = fifo_from_fpga_full;
assign register_data_out[28][20] = fifo_from_fpga_empty;
assign register_data_out[28][16:0] = fifo_from_fpga_dout;
</pre>

=== Register 29 0x80013074 packet routing ===

Control packet routing and loopbacks:

<pre>
wire [3:0] dn_route_ctrl = register_data_in[29][3:0];
wire [3:0] up_route_ctrl = register_data_in[29][7:4];
wire [3:0] fifo_to_fpga_route_ctrl = register_data_in[29][11:8];
//wire [3:0] spare_route_ctrl = register_data_in[29][15:12];

wire dn_mux_trg_enable = register_data_in[29][16];
wire dn_mux_tsm_enable = register_data_in[29][17];
wire dn_mux_sfp_rx_fifo_enable = register_data_in[29][18];
// 19
// 20..23

wire up_mux_vx_rx_enable = register_data_in[29][24];
// 25
// 26
// 27
wire fifo_from_fpga_hitmap_enable = register_data_in[29][28];
wire fifo_from_fpga_cdm_busy_enable = register_data_in[29][29];
wire fifo_from_fpga_gdm_busy_enable = register_data_in[29][30];
wire fifo_from_fpga_tsm_enable = register_data_in[29][31];
</pre>

fifo_to_fpga output routing:
* 0 - to down packet mux
* 1 - to up packet mux
* 2 - to fifo_from_fpga mux
* 3 - not used

down packet mux inputs:
* fifo_to_fpga_0
* trg_pkt16 enabled by dn_mux_trg_enable
* tsm_pkt16 enabled by dn_mux_tsm_enable
* sfp_rx_fifo_pkt16 enabled by dn_mux_sfp_rx_fifo_enable
* up_pkt16_2 loopback from up packet mux

down packet mux output routing:
* 0 - to vx_tx_pkt16 to VX TX (vx_link_tx and vx_ser_tx)
* 1 - to fifo_from_fpga mux
* 2 - to up packet mux loopback
* 3 - to qsfp_tx_pkt16 to GDM QSFP TX (cdm_link_tx)

up packet mux inputs:
* vx_rx_pkt16 enabled by up_mux_vx_rx_enable data from 12 VX RX links
* fifo_to_fpga_1
* dn_pkt16_2 loopback from down packet mux

up packet mux output routing:
* 0 - to sfp_tx_pkt16 to CDM SFP TX (VX data to GDM)
* 1 - to fifo_from_fpga mux
* 2 - to down packet mux loop loopback
* 3 - not used

internal_mux inputs:
* 0 - cdm_hitmap_pkt16 enabled by fifo_from_fpga_hitmap_enable (reg 29)
* 1 - cdm_busy_pkt16 enabled by fifo_from_fpga_cdm_busy_enable
* 2 - gdm_busy_pkt16 enabled by fifo_from_fpga_gdm_busy_enable
* 3 - gdm_hitmap_pkt16 enabled by fifo_from_fpga_hitmap_enable
* 4 - not used

debug_mux inputs:
* 0 - not used
* 1 - dn_pkt16_1
* 2 - up_pkt16_1
* 3 - qsfp0_rx_pkt16 enabled by fifo_from_fpga_qsfp0_rx_enable (from GDM QSFP link 0, there is no GDM QSFP 12-to-1 mux)
* 4 - not used

fifo_from_fpga mux inputs:
* 0 - fifo_to_fpga_2
* 1 - tsm2_pkt16 enabled by fifo_from_fpga_tsm_enable (reg 29)
* 2 - not used
* 3 - debug_pkt16 from debug_mux
* 4 - internal_pkt16 from internal_mux

=== Register 30 0x80013078 qsfp link status ports 0..7 ===

<pre>
assign register_data_out[30] =
{
qsfp_rx_data_error[7], qsfp_link_error[7], qsfp_link_status[7], qsfp_link_rx_status[7], // 7
qsfp_rx_data_error[6], qsfp_link_error[6], qsfp_link_status[6], qsfp_link_rx_status[6], // 6
qsfp_rx_data_error[5], qsfp_link_error[5], qsfp_link_status[5], qsfp_link_rx_status[5], // 5
qsfp_rx_data_error[4], qsfp_link_error[4], qsfp_link_status[4], qsfp_link_rx_status[4], // 4
qsfp_rx_data_error[3], qsfp_link_error[3], qsfp_link_status[3], qsfp_link_rx_status[3], // 3
qsfp_rx_data_error[2], qsfp_link_error[2], qsfp_link_status[2], qsfp_link_rx_status[2], // 2
qsfp_rx_data_error[1], qsfp_link_error[1], qsfp_link_status[1], qsfp_link_rx_status[1], // 1
qsfp_rx_data_error[0], qsfp_link_error[0], qsfp_link_status[0], qsfp_link_rx_status[0] // 0
};
</pre>

=== Register 31 0x8001307C qsfp link status ports 8..11 ===

<pre>
assign register_data_out[31] =
{
4'b0000,
4'b0000,
4'b0000,
4'b0000,
qsfp_rx_data_error[11], qsfp_link_error[11], qsfp_link_status[11], qsfp_link_rx_status[11], // 11
qsfp_rx_data_error[10], qsfp_link_error[10], qsfp_link_status[10], qsfp_link_rx_status[10], // 10
qsfp_rx_data_error[9], qsfp_link_error[9], qsfp_link_status[9], qsfp_link_rx_status[9], // 9
qsfp_rx_data_error[8], qsfp_link_error[8], qsfp_link_status[8], qsfp_link_rx_status[8] // 8
};
</pre>

=== Register 32 0x80013080 bitmap of enabled qsfp ports ===

<pre>
wire [11:0] qsfp_mask = register_data_in[32][11:0];
wire qsfp_bsy_force = register_data_in[32][12];
wire gdm_veto_enable = register_data_in[32][13];
// not used = register_data_in[32][15:14];
wire [15:0] gdm_veto_extend = register_data_in[32][31:16];
</pre>

On the GDM:

qsfp_rx_bsy[11..0] are pulses received from the CDMs

qsfp_rx_bsy_or is the grand-or of qsfp_rx_bsy masked by qsfp_mask

if qsfp_rx_bsy_or is high, gdm_bsy goes up and stays up for vx_bsy_extend*2 clocks

gdm_veto = gdm_bsy (bsy of any one VX causes trigger veto to all of them)

if (gdm_veto_enable) then gdm_veto will block trg_in_pulse

gdm_veto transition 0->1 causes gdm_veto_pulse. as long as gdm_veto is high, gdm_veto_pulse is generated every gdm_veto_extend clocks

gdm_veto_pulse is sent to all CDMs.

For this to work right, cdm_bsy_extend should not be bigger than gdm_veto_extend.

=== Register 33 0x80013084 bitmap of enabled VX ports ===

<pre>
wire [11:0] vx_mask = register_data_in[33][11:0];
wire vx_bsy_force = register_data_in[33][12];
// not used = register_data_in[33][15:13];
wire [15:0] vx_bsy_extend = register_data_in[33][31:16];
</pre>

On the CDM:

cdm_bsy is a grand-or of all vx_bsy masked by vx_mask (list of active VXes).

vx_bsy_extend controls how often state of cdm_bsy is sent to the GDM. when cdm_bsy goes 0->1, we send a cdm_bsy_pulse and keep resending it every vx_bsy_extend clocks as long as cdm_bsy stays high.

cdm_bsy_pulse is sent to the GDM.

sfp_rx_veto is the received from the GDM

if sfp_rx_veto goes up, cdm_veto goes up and stays up for gdm_veto_extend clocks.

for this to work right, CDM gdm_veto_extend must be bigger than GDM gdm_veto_extend.

cdm_veto goes to VXes on v1_tx_out[1] which is LVDS input 13.

=== Register 34, 35 0x80013088, 8C current timestamp ===

current 64-bit timestamp, 125 MHz

=== Register 36, 37 0x80013090, 94 old timestamp ===

old 64-bit timestamp, 125 MHz. timestamp saved at run start when it is reset to 0.

=== Register 38, 39, 40 0x80013098, 9C, A0 VX busy counters ===

8 bits per VX port, counters overflow to 255, reset at run start.

=== Register 41, 42, 43 0x800130A4, A8, AC QSFP busy counters ===

8 bits per QSFP port, counters overflow to 255, reset at run start.

=== Register 44, 45, 46, 47, 48 CDM and GDM busy and veto counters ===

* cdm_busy = grand-or of all VX busy for enabled VXes
* gdm_busy = grand-or of all CDM busy for enabled CDMs
* gdm_veto = gdm_busy

<pre>
44 | lo 16 bits | cdm_bsy_up_counter | CDM busy, increments when cdm_busy goes 0->1
44 | hi 16 bits | cdm_bsy_pulse_counter | CDM busy to GDM, increments for each cdm_bsy_pulse sent to the GDM

45 | lo 16 bits | cdm_veto_pulse_counter | CDM veto from GDM, increments for each sfp_rx_veto received from the GDM
45 | hi 16 bits | cdm_veto_up_counter | CDM veto to VX, increments each time cdm_veto is set to 1.

46 | lo 16 bits | gdm_bsy_pulse_counter | GDM busy from CDM, increments for each qsfp_rx_busy received from CDMs (unless they overlap)
46 | hi 16 bits | gdm_bsy_up_counter | GDM busy, increments each time gdm_bsy goes 0->1

47 | lo 16 bits | gdm_bsy_refresh_counter | GDM busy, increments each time gdm_bsy is extended by new qsfp_rx_busy
47 | hi 16 bits | spare |

48 | lo 16 bits | gdm_veto_up_counter | GDM veto, increments each time gdm_veto goes 0->1
48 | hi 16 bits | gdm_veto_pulse_counter | GDM veto to CDM, increments for each gdm_veto_pulse sent to the CDM
</pre>

=== Register 49, 50, 51 0x800130xx VX RX serial data monitor ===

VX RX data, 8-bit per VX channel. k-bit is omitted.

<pre>
assign register_data_out[49][7:0] = vx_rx_data[0]; // vx1
assign register_data_out[49][15:8] = vx_rx_data[1]; // vx2
assign register_data_out[49][23:16] = vx_rx_data[2]; // vx3
assign register_data_out[49][31:24] = vx_rx_data[3]; // vx4

assign register_data_out[50][7:0] = vx_rx_data[4]; // vx5
assign register_data_out[50][15:8] = vx_rx_data[5]; // vx6
assign register_data_out[50][23:16] = vx_rx_data[6]; // vx7
assign register_data_out[50][31:24] = vx_rx_data[7]; // vx8

assign register_data_out[51][7:0] = vx_rx_data[8]; // vx9
assign register_data_out[51][15:8] = vx_rx_data[9]; // vx10
assign register_data_out[51][23:16] = vx_rx_data[10]; // vx11
assign register_data_out[51][31:24] = vx_rx_data[11]; // vx12
</pre>

=== Register 52, 53 VX link status ===

<pre>
assign register_data_out[52] =
{
vx_rx_error[7], vx_link_error[7], vx_link_status[7], vx_link_rx_status[7], // 7
vx_rx_error[6], vx_link_error[6], vx_link_status[6], vx_link_rx_status[6], // 6
vx_rx_error[5], vx_link_error[5], vx_link_status[5], vx_link_rx_status[5], // 5
vx_rx_error[4], vx_link_error[4], vx_link_status[4], vx_link_rx_status[4], // 4
vx_rx_error[3], vx_link_error[3], vx_link_status[3], vx_link_rx_status[3], // 3
vx_rx_error[2], vx_link_error[2], vx_link_status[2], vx_link_rx_status[2], // 2
vx_rx_error[1], vx_link_error[1], vx_link_status[1], vx_link_rx_status[1], // 1
vx_rx_error[0], vx_link_error[0], vx_link_status[0], vx_link_rx_status[0] // 0
};

assign register_data_out[53] =
{
//4'b0000,
1'b0, ~vx_rx_deser_rdy[0], vx_rx_code_err[0], vx_rx_disp_err[0],
//4'b0000,
//4'b0000,
//4'b0000,
vx_rx_monitor[0],
vx_rx_error[11], vx_link_error[11], vx_link_status[11], vx_link_rx_status[11], // 11
vx_rx_error[10], vx_link_error[10], vx_link_status[10], vx_link_rx_status[10], // 10
vx_rx_error[9], vx_link_error[9], vx_link_status[9], vx_link_rx_status[9], // 9
vx_rx_error[8], vx_link_error[8], vx_link_status[8], vx_link_rx_status[8] // 8
};
</pre>

vx_rx_monitor (12-bit) is from deserializer_10b.sv:

<pre>
assign monitor_out[9:0] = lastByte[9:0];
assign monitor_out[10] = comma;
assign monitor_out[11] = ready;
</pre>

=== Register 54 VX TX serial data monitor ===

* contents of vx_tx_monitor from vx_ser_tx.sv:
<pre>
assign monitor_out[8:0] = data_to_encoder; // 8-bit + k
assign monitor_out[9] = valid;
assign monitor_out[15:10] = 0;
assign monitor_out[25:16] = encoded_data; // 10-bit
assign monitor_out[26] = encoded_valid;
assign monitor_out[27] = 0;
assign monitor_out[31:28] = 0;
</pre>

=== Register 55 QSFP, SFP, VX link loss counters ===

<pre>
assign register_data_out[55] = {
vx_rx_error_counter,
qsfp_link_rx_status_drop_counter,
sfp_link_rx_status_drop_counter,
vx_link_rx_status_drop_counter
};
</pre>

<pre>
24..31 - CDM VX RX error counters, count any errors in the VX RX data path (bad serial data, fifo overflow, etc)
16..23 - GDM QSFP link loss counter, increments on qsfp_tx_link_rx_status 1->0 (reg10)
8..15 - CDM SFP link loss counter, increments on sfp_link_rx_status 1->0 (reg15)
0...7 - CDM VX link loss counter, increments on vx_tx_link_rx_status 1->0 (reg10)
</pre>

=== Register 68 GPS control and status ===

<pre>
assign register_data_out[68] =
{
8'b00000000, // 23+8
rb_1pps_counter, // 16+8 bits
gps_1pps_counter, // 8+8 bits
1'b0,
1'b0,
rb_ser_in, // 5
gps_data_in, // 4
gps_aux_out, // 3
gps_aux_in, // 2
rb_1pps_in, // 1
gps_1pps_in // 0
};
</pre>

=== Register 71 packet error bits ===

<pre>
assign register_data_out[71] =
{
8'b00000000, // 24+8 bits
1'b0, // 23
1'b0, // 22
fifo_to_fpga_full_latch, // 21
fifo_from_fpga_full_latch, // 20
gdm_hitmap_fifo_full_latch, // 19
gdm_busy_fifo_full_latch, // 18
cdm_busy_fifo_full_latch, // 17
cdm_hitmap_fifo_full_latch, // 16
1'b0, // 15
1'b0, // 14
1'b0, // 13
gdm_trig48_encode_error_latch, // 12
1'b0, // vx_tx_trig48_pkt8_error_latch, // 11
tsm2_encode_error_latch, // 10
tsm_encode_error_latch, // 9
trg_encode_error_latch, // 8
gdm_hitmap_error_latch, // 7
gdm_hitmap_encode_error_latch, // 6
gdm_busy_encode_error_latch, // 5
cdm_busy_encode_error_latch, // 4
cdm_hitmap_encode_error_latch, // 3
vx_rx_hitmap_error_latch, // 2
1'b0, // vx_tx_tsm_pkt8_error_latch, // 1
1'b0, // vx_tx_trg_pkt8_error_latch // 0
};
</pre>

=== Register 90 VX busy status register ===

<pre>
vx_bsy_status_latched[11:0] <= vx_bsy[11:0];
vx_bsy_status_latched[23:12] <= vx_bsy_collect[11:0];
vx_bsy_status_latched[24] <= cdm_bsy;
vx_bsy_status_latched[25] <= cdm_bsy_collect;
vx_bsy_status_latched[26] <= cdm_veto;
vx_bsy_status_latched[27] <= cdm_veto_collect;
vx_bsy_status_latched[28] <= 0;
vx_bsy_status_latched[29] <= 0;
vx_bsy_status_latched[30] <= 0;
vx_bsy_status_latched[31] <= cdm_bsy_stuck;
</pre>

=== Register 91 VX hitmap status register ===

<pre>
hitmap_status_latched[11:0] <= hitmap_or12[11:0];
hitmap_status_latched[23:12] <= hitmap_or12_collect[11:0];
hitmap_status_latched[24] <= hitmap_grand_or;
hitmap_status_latched[25] <= hitmap_grand_or_collect;
hitmap_status_latched[26] <= 0;
hitmap_status_latched[27] <= 0;
hitmap_status_latched[28] <= 0;
hitmap_status_latched[29] <= 0;
hitmap_status_latched[30] <= 0;
hitmap_status_latched[31] <= hitmap_grand_or_stuck;
</pre>

=== Register 92 GDM busy status register ===

<pre>
gdm_bsy_status_latched[11:0] <= qsfp_rx_bsy[11:0];
gdm_bsy_status_latched[23:12] <= qsfp_rx_bsy_collect[11:0];
gdm_bsy_status_latched[24] <= gdm_bsy;
gdm_bsy_status_latched[25] <= gdm_bsy_collect;
gdm_bsy_status_latched[26] <= gdm_veto;
gdm_bsy_status_latched[27] <= gdm_veto_collect;
gdm_bsy_status_latched[28] <= 0;
gdm_bsy_status_latched[29] <= 0;
gdm_bsy_status_latched[30] <= 0;
gdm_bsy_status_latched[31] <= gdm_bsy_stuck;
</pre>

=== Register 93 GDM hitmap status register ===

<pre>
hitmap_status_latched[11:0] <= hitmap_grand_or_12[11:0];
hitmap_status_latched[23:12] <= hitmap_grand_or_12_collect[11:0];
hitmap_status_latched[24] <= hitmap_grand_or;
hitmap_status_latched[25] <= hitmap_grand_or_collect;
hitmap_status_latched[26] <= 0;
hitmap_status_latched[27] <= 0;
hitmap_status_latched[28] <= 0;
hitmap_status_latched[29] <= 0;
hitmap_status_latched[30] <= 0;
hitmap_status_latched[31] <= hitmap_grand_or_stuck;
</pre>

=== Register 98, 99 GDM TRIG48 test register and TRIG48_CTRL ===

<pre>
assign gdm_trig48_ctrl_bitmap48[31:0] = register_data_in[98][31:0];
assign gdm_trig48_ctrl_bitmap48[47:32] = register_data_in[99][15:0];

assign gdm_trig48_ctrl[15:0] = register_data_in[99][31:16];
</pre>

<pre>
wire enable_grand_or = trig48_ctrl[15];
wire enable_test_trig48 = trig48_ctrl[14];
</pre>

= Firmware registers branch develop_ko =

== Register map ==

<pre>
0 | ALL | ALL | RO | USR_ACCESSE2 see https://docs.xilinx.com/r/en-US/ug974-vivado-ultrascale-libraries/USR_ACCESSE2
1 | ALL | ALL | RW | read write scratch register
2 | ALL | CDM | ?? | MGT not used
3 | ALL | CDM | RO | MGT debug_data
4 | ALL | CDM | RW | clk_config_vec
5 | ALL | CDM | ?? | not used
6 | ALL | CDM | RO | CDM_link_data_processing:o_error_time
7 | ALL | CDM | RO | CDM_link_data_processing:o_error_count
</pre>

== register 0 0x80010000 ==

GDM:
<pre>
0 - gdm_link_interface:i_mgt_rst
2 - gdm_link_interface:i_link_down_latched_rst
8 - GDM_link_data_processing:i_rst
10..9 - GDM_link_data_processing:i_data_mode
</pre>

CDM:
<pre>
0 - cdm_link_interface:i_mgt_rst
2 - cdm_link_interface:i_link_down_latched_rst
8 - CDM_link_data_processing:i_rst
10..9 - CDM_link_data_processing:i_data_mode
</pre>

== register 1 0x80010004 ==

GDM:
<pre>
nlinks-1..0 - gdm_link_interface:i_rx_slide_trigger
</pre>

CDM:
<pre>
nlinks-1..0 - cdm_link_interface:i_rx_slide_trigger
</pre>

== register 2 0x80010008 ==

GDM:
<pre>
nlinks-1..0 - gdm_link_interface:o_link_power_good
nlinks+15..16 - gdm_link_interface:o_link_status
</pre>

CDM:
<pre>
nlinks-1..0 - cdm_link_interface:o_link_power_good
nlinks+15..16 - cdm_link_interface:o_link_status
</pre>

== register 3 0x8001000c ==

GDM: simple loopback register

CDM:
<pre>
31..0 - debug_data - cdm_link_interface:o_debug
</pre>

o_debug:
<pre>
rx_link_rst & rx_error & rx_link_up & rx_receiving_data &
std_logic_vector(rx_state_count) & tx_state_count_on_rx_clk & i_rx_ctrl3(0) &
i_rx_ctrl1(1 downto 0) & i_rx_ctrl0(1 downto 0) &
rx_data_is_k28p1_k28p5 &
i_rx_data;
</pre>

== register 4 0x80010010 ==

GDM write:
<pre>
0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS
</pre>

GDM read:
<pre>
0 - clk_config_vec(0) - CLK_IN_SEL_LS(0)
1 - clk_config_vec(1) - CLK_IN_SEL_LS(1)
2 - clk_config_vec(2) - CLK_EXT_SEL_LS
3 - clk_config_vec(3) - CLK_RSTn_LS
4 - clk_config_vec(4) - CLK_LOSXTn_LS
5 - clk_config_vec(5) - CLK_LOLn_LS
6 - clk_config_vec(6) - CLK_INTn_LS
7 - constant 1
31..8 - constant 0
</pre>

== register 5 0x80010014 ==

not used

== register 6 0x80010018 ==

GDM:
<pre>
3..0 - GDM_link_data_processing:i_status_select
</pre>

CDM:
<pre>
31..0 - CDM_link_data_processing:o_error_count
</pre>

== register 7 0x8001001c ==

GDM:
<pre>
31..0 - GDM_link_data_processing:o_status_vector
</pre>

CDM:
<pre>
31..0 - CDM_link_data_processing:o_error_count
</pre>

= GDM, CDM, VX packet communications =

* timestamp math
<pre>
1 clock is 8 ns is 125 MHz
8 bits of clocks is 256 clocks is 2048 ns is ~2 usec
16 bits of clocks is ~500 usec is 0.5 msec
24 bits of clocks is ~134 msec
32 bits of clocks is ~34 sec
40 bits of clocks is ~8.7 ksec is 2.4 hours
48 bits of clocks is ~625 hours is ~26 days
56 bits of clocks is ~6.6 kdays is ~18 kyears
62 bits of clocks is ~10 Mhours is 427 kdays is ~1.1 kyears
64 bits of clocks is ~4.4 kyears
</pre>
* summary
<pre>
0x01 - not used
0x02 - 8 bytes - 4 words - 2 dwords - 1 qwords - TRG packet
0x03 - 12 bytes - 6 words - 3 dwords - 2 qwords - TRIG48 packet
0x10 - 26 bytes - 13 words - 7 dwords - 4 qwords - TSM packet
0x81 - 10 bytes - 5 words - 3 dwords - 2 qwords - VX_HITMAP packet
0x82 - 108 bytes - 54 words - 27 dwords - 14 qwords - CDM_HITMAP packet
0x83 - 12 bytes - 6 words - 3 dwords - 2 qwords - CDM_BUSY packet
0x84 - 12 bytes - 6 words - 3 dwords - 2 qwords - GDM_BUSY packet
0x85 - 12 bytes - 6 words - 3 dwords - 1 qwords - GDM_HITMAP packet
</pre>
* 0x02 - TRG packet, 8 bytes, 80 adc clocks, 640 ns on lvds link
<pre>
0 - 0x02
1 - trg_counter[7:0]
2 - ts64 low byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 high byte 3
6 - trg_in_latch[7:0]
7 - trg_in_latch[15:8]
</pre>
* 0x03 - TRIG48 packet, 12 bytes, 120 adc clocks, 960 ns on lvds link
<pre>
0 - 0x03
1 - trg_counter[7:0]
2 - ts64 low byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 high byte 3
6 - vx_bitmap[7:0]
7 - vx_bitmap[15:8]
8 - vx_bitmap[23:16]
9 - vx_bitmap[31:24]
10 - vx_bitmap[39:32]
11 - vx_bitmap[47:40]
</pre>
* 0x10 - TSM packet, 26 bytes, 260 adc clocks, 2080 ns on lvds link
<pre>
0 - 0x10
1 - tsm_counter[7:0]
2 - gdm_ts64 low byte 0
3 - 1
4 - 2
5 - 3
6 - 4
7 - 5
8 - 6
9 - gdm_ts64 high byte 7
10 - gps_ts64 low byte 0
11 - 1
12 - 2
13 - 3
14 - 4
15 - 5
16 - 6
17 - gps_ts64 high byte 7
18 - gps_data64 low byte 0
19 - 1
20 - 2
21 - 3
22 - 4
23 - 5
24 - 6
25 - gps_data64 high byte 7
</pre>
* 0x81 - VX hitmap packet, 10 bytes, 100 adc clocks, 800 ns on lvds link, 48 ns on fiber link
<pre>
0 - 0x81
1 - VX ID
2 - hitmap low byte, nits 7:0
3 - 15:8
4 - 23:16
5 - 31:24
6 - ...:32
7 - ...
8 - ...
9 - hitmap low byte, bits 63:...
</pre>
* 0x82 - CDM hitmap packet, 108 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x82
1 - cdm_hitmap_trigger_counter[7:0]
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - cdm_hitmap_or12 byte 0 (7:0)
11 - cdm_hitmap_or12 byte 1 (11:8) plus 4 bits: 12=0, 13=0, 14=0, 15=cdm_hitmap_grand_or
12 - cdm_hitmap_data, low byte, 12*64 bits = 768 bits = 48 words = 96 bytes
...
107 - cdm_hitmap_data, high byte
</pre>
* 0x83 - CDM busy packet, 12 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x83
1 - cdm_busy_packet_counter[7:0], reset at begin of run
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - vx_bsy[7:0]
11 - {0,0,cdm_veto,cdm_bsy,vx_bsy[11:8]}
</pre>
* 0x84 - GDM busy packet, 12 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x84
1 - gdm_busy_packet_counter[7:0], reset at begin of run
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - qsfp_rx_bsy[7:0]
11 - {0,0,gdm_veto,gdm_bsy,qsfp_rx_bsy[11:8]}
</pre>
* 0x85 - GDM hitmap trigger packet, 12 bytes, not sent on lvds link, not sent on fiber link
<pre>
0 - 0x85
1 - gdm_hitmap_packet_counter[7:0], reset at begin of run
2 - ts64 byte 0
3 - ts64 byte 1
4 - ts64 byte 2
5 - ts64 byte 3
6 - ts64 byte 4
7 - ts64 byte 5
8 - ts64 byte 6
9 - ts64 byte 7
10 - hitmap_grand_or_12[7:0]
11 - {hitmap_grand_or,0,0,0,hitmap_grand_or_12[11:8]}
</pre>

= AXI bus timing =

* AXI 100 MHz clock, 10 ns, 32-bit data
* AXI single-dword read: 36 clock repeat rate, 360 ns is 2.777 MHz, 4 bytes per transfer is 11.11 Mbytes/sec
* AXI single-qword read: 13 clock repeat rate, 130 ns is 7.7 MHz, 4 bytes per transfer is 30 Mbytes/sec, not accounting for the gap
* AXI memcpy read: 4 transfers at 13 clocks, gap, 4 transfers at 13 clocks, gap, etc. below 30 Mbytes/sec.
[[Image:Ds-dm-axi-read.png|100px]]
[[Image:Ds-dm-axi-read-64.png|100px]]
[[Image:Ds-dm-axi-read-memcpy.png|100px]]
* AXI single-dword write: 20 clocks repeat rate, 200 ns is 5 MHz, 4 bytes per transfer is 20 Mbytes/sec
* AXI single-qword write: 13+20 clocks repeat rate, 330 ns is 3 MHz, 16 bytes per burst is 48 Mbytes/sec
[[Image:Ds-dm-axi-write.png|100px]]
[[Image:Ds-dm-axi-write-64.png|100px]]

= AXI bus addresses =

* see AXI/AMBA addresses assigned inthe FPGA project: (s_axi/reg0 is the DS-DM AXI registers)
<pre>
daq00:ds-dm-gcdm$ grep assign_bd_address scripts/GDM_CDM_XU8_bd.tcl
assign_bd_address -offset 0x80010000 -range 0x00004000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs axi_register_interfa_0/s_axi/reg0] -force
assign_bd_address -offset 0x000400000000 -range 0x40000000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs ddr4/C0_DDR4_MEMORY_MAP/C0_DDR4_ADDRESS_BLOCK] -force
assign_bd_address -offset 0x80000000 -range 0x00010000 -target_address_space [get_bd_addr_spaces zynq_ultra_ps_e/Data] [get_bd_addr_segs system_management_wiz/S_AXI_LITE/Reg] -force
daq00:ds-dm-gcdm$
</pre>

* see AXI/AMBA addresses exported from FPGA project to Linux kernel: (uio for debug bridge should say "debug bridge")
<pre>
root@gdm0:~# cat /sys/class/uio/uio*/name
axi-pmon
axi-pmon
axi-pmon
axi-pmon
root@gdm0:~#
root@gdm0:~# ls -l /sys/class/uio/
total 0
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio0 -> ../../devices/platform/amba/ffa00000.perf-monitor/uio/uio0
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio1 -> ../../devices/platform/amba/fd0b0000.perf-monitor/uio/uio1
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio2 -> ../../devices/platform/amba/fd490000.perf-monitor/uio/uio2
lrwxrwxrwx 1 root root 0 Oct 18 01:36 uio3 -> ../../devices/platform/amba/ffa10000.perf-monitor/uio/uio3
root@gdm0:~# ls -l /sys/class/uio/../../devices/platform/amba/
total 0
-rw-r--r-- 1 root root 4096 Oct 18 19:37 driver_override
drwxr-xr-x 3 root root 0 Oct 18 01:36 fd070000.memory-controller
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd0b0000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd400000.zynqmp_phy
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd490000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd500000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd510000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd520000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd530000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd540000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd550000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd560000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd570000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 fd6e0000.cci
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff000000.serial
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff020000.i2c
drwxr-xr-x 6 root root 0 Oct 18 01:36 ff0a0000.gpio
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0b0000.ethernet
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0e0000.ethernet
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff0f0000.spi
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff160000.mmc
drwxr-xr-x 5 root root 0 Oct 18 01:36 ff170000.mmc
drwxr-xr-x 3 root root 0 Oct 18 01:36 ff960000.memory-controller
drwxr-xr-x 4 root root 0 Oct 18 01:36 ff9d0000.usb0
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa00000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa10000.perf-monitor
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa50000.ams
drwxr-xr-x 5 root root 0 Oct 18 01:36 ffa60000.rtc
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa80000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffa90000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffaa0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffab0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffac0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffad0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffae0000.dma
drwxr-xr-x 4 root root 0 Oct 18 01:36 ffaf0000.dma
-r--r--r-- 1 root root 4096 Oct 18 19:37 modalias
lrwxrwxrwx 1 root root 0 Oct 18 19:37 of_node -> ../../../firmware/devicetree/base/amba
drwxr-xr-x 2 root root 0 Oct 18 19:37 power
lrwxrwxrwx 1 root root 0 Oct 18 01:36 subsystem -> ../../../bus/platform
-rw-r--r-- 1 root root 4096 Oct 18 01:36 uevent
root@gdm0:~#
</pre>

= Build firmware =

== Build from git clone ==

THESE ARE K.O.'s NOTES FOR CREATING THE PETALINUX DIRECTORY.

THEY DO NOT WORK!

COPY PETALINUX FROM A WORKING PROJECT AND USE "make gdm" and "make cdm" AS DESCRIBED BELOW.

* git clone git@edev-group.triumf.ca:fw/exp/darkside/gcdm.git
* #Makefile change VIVADO_SETTINGS_SCRIPT := /opt/Xilinx/Vivado/2022.1/settings64.sh
* #. /opt/Xilinx/Vivado/2022.1/settings64.sh
* . /opt/Xilinx/Vivado/2020.2/settings64.sh
* make clean
* make all_from_scratch
* . /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
* make petalinux_create
* make petalinux_rebuild_new_hw_des
* bomb out: The TMPDIR: /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp can't be located on nfs.
* mkdir /tmp/build_tmp
* rm -rf /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp/
* ln -s /tmp/build_tmp /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/build/tmp
* try again
* grinds, loads a whole bunch of packages...
* finishes with desire to copy things to /tftpboot
* make sdcard_cp_to wants to copy files from PetaLinux_GDM_CDM/images/linux/ to SD card

== Build firmware ==

NOTE: directory Petalinux_GDM_CDM should already exist!

<pre>
#. /opt/Xilinx/Vivado/2020.2/settings64.sh
. /opt/Xilinx/Vivado/2022.2/settings64.sh
. /opt/Xilinx/PetaLinux/2020.2/tool/settings.sh
make clean_gdm # remove gdm build tree
make gdm # build or rebuild GDM
make copy_gdm # copy to gdm0
make clean_cdm # remove cdm build tree
make cdm # build or rebuild CDM
make copy_cdm # copy to cdm0 and cdm1
</pre>

copy to SD card:
<pre>
open a root shell
format 16 GB Sd card per above
cd .../ds-dm-gcdm
make copy
</pre>

= build times =

<pre>
CDM 12-june-2023 69aabc1c25130d970bc375aca684bd68849e6685
daq13 AMD-5700G 1688.61user 399.33system 23:28.84elapsed 148%CPU
dsdaqgw AMD-7700 1090.55user 247.34system 16:03.55elapsed 138%CPU
dsdaqgw AMD-7700 CDM incremental 196.68user 67.62system 7:35.42elapsed 58%CPU
dsdaqgw AMD-7700 CDM incremental 684.72user 94.17system 7:30.17elapsed 173%CPU
dsdaqgw AMD-7700 GDM incremental 849.84user 99.79system 9:04.56elapsed 174%CPU
</pre>

= make bootable sd card =

== format the sd card ==

this only needs to be done once

* become root@dsdaqgw
* cd ~olchansk/git/ds-dm-gcdm
* use "lsblk" to identify the SD card (should show as 8/16/32 GB block device)/ /dev/sdd in this case
* make sdcard_format SDCARD_DEVICE=/dev/sdd
* disconnect sd card, reconnect the sd card (to detect new partition tables, etc)

== copy boot files to the sd card ==

* as root: identify partition labels, run "blkid", should say "BOOT", "rootfs" and "data"
* mount
<pre>
mkdir -p /mnt/BOOT
mount -L BOOT /mnt/BOOT
cp PetaLinux_GDM_CDM/images/linux/BOOT_{GDM,CDM}.BIN /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/boot.scr /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/image.ub /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/uboot.env /mnt/BOOT/
cp PetaLinux_GDM_CDM/images/linux/uboot-redund.env /mnt/BOOT/
umount /mnt/BOOT
eject /dev/sdX
</pre>
* FIXME: missing step: cp BOOT_GDM.BIN BOOT.BIN or cp BOOT_CDM.BIN BOOT.BIN

= boot messages =

<pre>
Xilinx Zynq MP First Stage Boot Loader
Release 2020.2 Sep 24 2022 - 13:29:15
NOTICE: ATF running on XCZU4CG/silicon v4/RTL5.1 at 0xfffea000
NOTICE: BL31: v2.2(release):xlnx_rebase_v2.2_2020.3
NOTICE: BL31: Built : 18:02:46, Sep 28 2022

U-Boot 2020.01 (Sep 28 2022 - 18:03:39 +0000)

Model: DarkSide 20k DM
Board: Xilinx ZynqMP
DRAM: 2 GiB
usb dr_mode not found
PMUFW: v1.1
EL Level: EL2
Chip ID: zu4
NAND: 0 MiB
MMC: mmc@ff160000: 0, mmc@ff170000: 1
In: serial@ff000000
Out: serial@ff000000
Err: serial@ff000000
Bootmode: SD_MODE1
Reset reason: SOFT
Net:
ZYNQ GEM: ff0b0000, mdio bus ff0b0000, phyaddr -1, interface rgmii-id

Warning: ethernet@ff0b0000 (eth0) using random MAC address - d6:62:5f:13:00:44
eth0: ethernet@ff0b0000
ZYNQ GEM: ff0e0000, mdio bus ff0e0000, phyaddr -1, interface rgmii-id
Could not get PHY for eth1: addr -1

Hit any key to stop autoboot: 0
ZynqMP>
CTRL-A Z for help | 115200 8N1 | NOR | Minicom 2.7.1 | VT102 | Online 122:2 | ttyACM0
</pre>

= load FPGA from u-boot =

* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/124682257/U-Boot+FPGA+Driver

<pre>
ZynqMP> fpga info
Xilinx Device
Descriptor @ 0x000000007fddb2c0
Family: ZynqMP PL
Interface type: csu_dma configuration interface (ZynqMP)
Device Size: 1 bytes
Cookie: 0x0 (0)
Device name: zu4
Device Function Table @ 0x000000007fda5fe8
PCAP status 0xa0002fde
ZynqMP>
</pre>

* cp CDM_XU8_top.bit /tftpboot/fpga.bit

<pre>
dhcp
tftpb 0x10000000 fpga.bit
fpga loadb 0 0x10000000 ${filesize}
</pre>

<pre>
ZynqMP> dhcp
BOOTP broadcast 1
DHCP client bound to address 192.168.0.100 (1 ms)
*** Warning: no boot file name; using 'C0A80064.img'
Using ethernet@ff0b0000 device
TFTP from server 192.168.0.1; our IP address is 192.168.0.100
Filename 'C0A80064.img'.
Load address: 0x8000000
Loading: *
TFTP error: 'file /tftpboot/C0A80064.img not found for 192.168.0.100' (1)
Not retrying...
ZynqMP> tftpb 0x10000000 fpga.bit
Using ethernet@ff0b0000 device
TFTP from server 192.168.0.1; our IP address is 192.168.0.100
Filename 'fpga.bit'.
Load address: 0x10000000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
############
6.2 MiB/s
done
Bytes transferred = 7797807 (76fc2f hex)
ZynqMP> fpga loadb 0 0x10000000 ${filesize}
design filename = "CDM_XU8_top;UserID=0XFFFFFFFF;Version=2022.2"
part number = "xczu4cg-fbvb900-1-e"
date = "2024/08/14"
time = "14:18:22"
bytes in bitstream = 7797692
zynqmp_align_dma_buffer: Align buffer at 0000000010000073 to 000000000fffff80(swap 0)
ZynqMP>
</pre>

= load FPGA from Linux =

* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841847/Solution+ZynqMP+PL+Programming

== this will reset the CPU ==
<pre>
cp fpga.bit /lib/firmware/
echo fpga.bit > /sys/class/fpga_manager/fpga0/firmware
</pre>

* make .bin file:
<pre>
bootgen -image CDM_XU8_top.bif -arch zynqmp -o ./fpga.bin -w
</pre>
* cat CDM_XU8_top.bif
<pre>
dsdaqgw:ds-dm-gcdm$ cat CDM_XU8_top.bif
all:
{
[destination_device = pl] ./Vivado_CDM_XU8/CDM_XU8.runs/impl_1/CDM_XU8_top.bit
}
dsdaqgw:ds-dm-gcdm$
</pre>

== this will reset the CPU ==

<pre>
cp fpga.bin /lib/firmware/
echo fpga.bin > /sys/class/fpga_manager/fpga0/firmware
</pre>

== this will reset the CPU ==

<pre>
root@dsdm:~# ./fpgautil -b fpga.bin -f Full
</pre>

== this will reset the CPU ==

<pre>
DTSO file from here: https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841847/Solution+ZynqMP+PL+Programming
echo 0 > /sys/class/fpga_manager/fpga0/flags
mount -t configfs configfs /configfs
root@dsdm:~# dtc -O dtb -o fpga.dtbo -b 0 -@ fpga.dtso
root@dsdm:~# cp fpga.dtbo /lib/firmware/
root@dsdm:~# cp fpga.bit /lib/firmware/
root@dsdm:~# rmdir /configfs/device-tree/overlays/fpga
root@dsdm:~# mkdir /configfs/device-tree/overlays/fpga
root@dsdm:~# echo -n "fpga.dtbo" > /configfs/device-tree/overlays/fpga/path
</pre>

= fpgautil =

* https://github.com/Xilinx/meta-xilinx/blob/master/meta-xilinx-core/recipes-bsp/fpga-manager-script/files/fpgautil.c

<pre>
git clone https://github.com/Xilinx/meta-xilinx.git
cd meta-xilinx/meta-xilinx-core/recipes-bsp/fpga-manager-script/files/
scp fpgautil.c root@dsdm:
ssh root@dsdm
make fpgautil
ls -l ./fpgautil
</pre>

<pre>
root@dsdm:~# ls -l ./fpgautil
-rwxr-xr-x 1 root root 72256 Aug 16 00:15 ./fpgautil
root@dsdm:~# ./fpgautil

fpgautil: FPGA Utility for Loading/reading PL Configuration

Usage: fpgautil -b <bin file path> -o <dtbo file path>

Options: -b <binfile> (Bin file path)
-o <dtbofile> (DTBO file path)
-f <flags> Optional: <Bitstream type flags>
f := <Full | Partial >
-n <Fpga region info> FPGA Regions represent FPGA's
and partial reconfiguration
regions of FPGA's in the
Device Tree
Default: <full>
-s <secure flags> Optional: <Secure flags>
s := <AuthDDR | AuthOCM | EnUsrKey | EnDevKey | AuthEnUsrKeyDDR | AuthEnUsrKeyOCM | AuthEnDevKeyDDR | AuthEnDevKeyOCM>
-k <AesKey> Optional: <AES User Key>
-r <Readback> Optional: <file name>
Default: By default Read back contents will be stored in readback.bin file
-t Optional: <Readback Type>
0 - Configuration Register readback
1 - Configuration Data Frames readback
Default: 0 (Configuration register readback)
-R Optional: Remove overlay from a live tree

Examples:
(Load Full bitstream using Overlay)
fpgautil -b top.bit.bin -o can.dtbo -f Full -n full
(Load Partial bitstream using Overlay)
fpgautil -b rm0.bit.bin -o rm0.dtbo -f Partial -n PR0
(Load Full bitstream using sysfs interface)
fpgautil -b top.bit.bin -f Full
(Load Partial bitstream using sysfs interface)
fpgautil -b rm0.bit.bin -f Partial
(Load Authenticated bitstream through the sysfs interface)
fpgautil -b top.bit.bin -f Full -s AuthDDR
(Load Parital Encrypted Userkey bitstream using Overlay)
fpgautil -b top.bit.bin -o pl.dtbo -f Partial -s EnUsrKey -k <32byte key value>
(Read PL Configuration Registers)
fpgautil -b top.bit.bin -r
(Remove Partial Overlay)
fpgautil -R -n PR0
(Remove Full Overlay)
fpgautil -R -n full
Note: fpgautil -R is responsible for only removing the dtbo file from the livetree. it will not remove the PL logic from the FPGA region.

root@dsdm:~#
</pre>

= fw_printenv =

to access u-boot environment from Linux:
* apt install -y libubootenv-tool
* create /etc/fw_env.config
<pre>
/media/BOOT/uboot.env 0 0x40000
/media/BOOT/uboot-redund.env 0 0x40000
</pre>
* if uboot.env files do not exist, run "saveenv" from u-boot command prompt
* fw_printenv and fw_setenv should work

= Boot from network =

== u-boot ==

<pre>
ZynqMP> setenv bootcmd run bootcmd_dhcp
ZynqMP> saveenv
ZynqMP> reset
</pre>

== boot.scr ==

<pre>
# boot.scr
# mkimage -C none -A arm -T script -d boot.scr boot.scr.uimg
echo Loading FPGA!
#tftpb 0x10000000 fpga.bit
tftpb 0x10000000 {ipaddr}.bit
fpga loadb 0 0x10000000 ${filesize}
echo Booting Linux!
run bootcmd_pxe
echo Done!
</pre>

<pre>
mkimage -C none -A arm -T script -d boot.scr boot.scr.uimg
</pre>

== tftpboot ==

<pre>
cp /home/olchansk/git/ds-dm-gcdm/boot.scr.uimg /tftpboot
ln -s /home/olchansk/git/ds-dm-gcdm/PetaLinux_GDM_CDM/images/linux /tftpboot/xilinx-dsdm
mkdir /tftpboot/pxelinux.cfg
cat > /tftpboot/pxelinux.cfg/default-arm-zynqmp <<EOF
LABEL Linux
KERNEL xilinx-dsdm/Image
FDT xilinx-dsdm/system.dtb
#INITRD rootfs.cpio.gz.u-boot
EOF
</pre>

== boot sequence ==

* xilinx magic load BOOT.BIN from SD card
* load FPGA form BOOT.BIN
* load and run u-boot from BOOT.BIN or from image.ub
* u-boot load environment from ??? probably SD card uboot-redund.env, this includes our bootcmd
* run bootcmd which run bootcmd_dhcp which does:
* from /tftpboot:
* load and run boot.scr.uimg which does:
* load FPGA image xilinx-dsdm/${ipaddr}.bit
* run bootcmd_pxe which does:
* load pxelinux.cfg/default-arm-zynqmp which does:
* load xilinx-dsdm/Image ### this is the linux kernel
* load xilinx-dsdm/system.dtb ### this is the device tree
* start linux kernel
* linux kernel does dhcp
* linux kernel does nfs mount /nfsroot/%s,vers=3,tcp ### %s is replaced by the hostname supplied by DHCP
* userland starts and runs to console and ssh login.

= Xilinx ILA =

References:
* https://github.com/Xilinx/XilinxVirtualCable/tree/master
* https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/644579329/Xilinx+Virtual+Cable
* https://support.xilinx.com/s/article/974879?language=en_US
* https://docs.xilinx.com/v/u/en-US/pg245-debug-bridge
* https://docs.xilinx.com/r/2020.2-English/ug908-vivado-programming-debugging/Initializing-Vivado-IDE-hw_server
* (we do not use this one) https://github.com/paulscherrerinstitute/xvcSupport/blob/master/README.md

Build xvcserver_cdm.exe: (it is built as a static executable, can be copied and run anywhere)
<pre>
ssh dsdaq@gdm0
cd /home/dsdaq/online/ds-dm-software
git pull ### get latest version
make xvcserver_cdm.exe
ssh root@gdm0
/home/dsdaq/online/ds-dm-software/xvcserver_cdm.exe
INFO: To connect to this xvcServer instance, use url: TCP:gdm0:2542
</pre>

To activate and use the vivado logic analyzer:
* data path: vivado -> hw_server -> xvcserver -> mmap axi bus -> debug bridge -> jtag -> ILA
* define ILAs in the code
* instantiate the xilinx debug bridge at AXI bus address 0x80020000 (FIXME!!! this collides with Ian's AXI addresses)
* build and boot the new FPGA firmware. updating the linux kernel is not necessary.
* login root@gdm0, run: /home/dsdaq/online/ds-dm-software/xvcserver_cdm.exe -v ### with "-v" for the first time to see that vivado does connect to it, without "-v", normally.
* login dsdaqgw, run: hw_server -s tcp:localhost:3121 -e "set auto-open-servers xilinx-xvc:gdm0:2542" ### tells us to connect to port localhost:3121
* login dsdaqgw, run vivado, open project, open hardware manager, open target, open new target, "connect to remote server", hostname "localhost", port "3121", next (bombs, try again, 3 times), popup add virtual cable, enter hostname "gdm0" port "2542", "ok", it shows in "hardware targets", "next", "finish", error popup "[Common 17-163] Missing value for option 'objects', please type 'set_property -help' for usage info", ignore it, in "hardware", right click the "gdm0" one, open target, under "hardware" and "debug bridge" we should see all the ILAs, under "hardware device properties", the "probes file" should have the ".ltx" file generated by vivado "Vivado_GDM_XU8/GDM_XU8.runs/impl_1/debug_nets.ltx", click on an ILA, a waveform should open.

= Software =

* ssh cdm0 # or gdm0
* sudo apt install i2c-tools libi2c-dev
* git clone https://bitbucket.org/team-ds-dm/ds-dm-software
* cd ds-dm-software
* make

== test_cdm.exe ==

=== CDM SFP status ===

<pre>
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0xbb2f0ae7
CDM firmware 0xbb2f0ae7
arg 1: [--sfp]
Polling SFP status...
identifier 0x03
connector 0x07
encoding 0x01
wavelength 0x0352 (850 nm)
vendor_name [FINISAR CORP. ]
vendor_pn [FTLF8526P3BNL ]
vendor_rev [A ]
vendor_sn [N3AB9M8 ]
vendor_date [200319 ]
dm_type 0x68
temp 29.0 C
vcc 3.323 V
tx_bias 7.250 mA
tx_power 478.4 uW
rx_power 2.3 uW
SFP good 1, status: temp 30.6 C, tx_bias 7.4 mA, tx_power 476 uW, rx_power 818 uW
...
</pre>

=== GDM QSFP status ===

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --qsfp3 --qsfp
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x53aee418
CDM firmware 0x53aee418
arg 1: [--qsfp3]
gpiochip0: GPIOs 338-511, parent: platform/ff0a0000.gpio, zynqmp_gpio:
gpio-378 ( |sysfs ) out hi
gpio-379 ( |sysfs ) out hi
gpio-381 ( |sysfs ) out lo
gpio-382 ( |sysfs ) out hi
arg 2: [--qsfp]
Polling QSFP status...
identifier 0x0d
status 0x02
los 0x8f
temp 28.2 C
vcc 3.323 V
rx_power 0.1 0.1 0.1 0.1 uW
tx_bias 7.6 7.6 7.6 0.0 mA
tx_power 792.2 773.8 823.0 0.1 uW
vendor_name [FINISAR CORP ]
vendor_pn [FTL410QD4C ]
vendor_rev [A ]
wavelength 850
max_temp 70 C
vendor_sn [X79AC0R ]
vendor_date [220309 ]
QSFP good 1, status: temp 27.7 C, los 0x8b, tx_bias 7.5 7.6 7.6 0.0 mA, tx_power 792 772 821 0 uW, rx_power 0 0 466 0 uW
</pre>

=== GDM clock status ===

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
</pre>

* clock chip not loaded, not running:

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x0f 0x16 0x04 0x00 0x01 0x68 0x09 0x00 0x02 0xf2 0x00 0x1f 0xf0 0x22 0xf2, 0x507: 0x00, 0x52A: 0x00, 0x53F: 0x04
Clock chip state 0, status: SYSINCAL XAXB_ERR LOL CAL_PLL IN0 IN_SEL_0 FASTLOCK_STATUS

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x1f7caf52 (528265042) should be ~125 MHz
0x1034 rx_clk: 0x00000000 (0) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x1f7caf52 (528265042) should be ~125 MHz
0x103C tx_clk: 0x00000000 (0) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip good (IN0 - external 10 MHz clock)

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x44 0x00 0xd0 0x01 0x1f 0xff 0x22 0xf2, 0x507: 0x3f, 0x52A: 0x01, 0x53F: 0x00
Clock chip state 1, status: LOS_IN2 OOF_IN2 IN0 IN_SEL_REGCTRL IN_SEL_0

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735a3b (125000251) should be ~125 MHz
0x1034 rx_clk: 0x07735a3c (125000252) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a3b (125000251) should be ~125 MHz
0x103C tx_clk: 0x07735a3c (125000252) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

=== CDM clock status ===

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
</pre>

* clock chip not loaded, not running:

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x0f 0x16 0x04 0x00 0x01 0x68 0x19 0x00 0x02 0xf2 0x00 0x1f 0xf0 0x22 0xf2, 0x507: 0x00, 0x52A: 0x00, 0x53F: 0x04
Clock chip state 0, status: SYSINCAL XAXB_ERR LOL CAL_PLL IN0 IN_SEL_0 FASTLOCK_STATUS

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735851 (124999761) should be ~125 MHz
0x1034 rx_clk: 0x00000000 (0) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x0127fefa (19398394) should be ~125 MHz
0x103C tx_clk: 0x00000000 (0) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip uses internal clock (IN1 - internal oscillator), observe rx_clk frequency is not same as others

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0x7f, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN1 IN_SEL_1 HOLD_HIST_VALID

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebec9
CDM firmware 0x6d2ebec9
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735853 (124999763) should be ~125 MHz
0x1034 rx_clk: 0x07735b0a (125000458) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735853 (124999763) should be ~125 MHz
0x103C tx_clk: 0x07735852 (124999762) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>

* clock chip uses SFP recovered clock (IN2 - sfp rx recovered clock), observe mgt_rx_ref_clk_raw (CDM 125 MHz oscillator) is different from others (SFP RX recovered clock)

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0xbf, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN2 IN_SEL_1 HOLD_HIST_VALID
^C
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x0773581b (124999707) should be ~125 MHz
0x1034 rx_clk: 0x07735ad6 (125000406) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735ad7 (125000407) should be ~125 MHz
0x103C tx_clk: 0x07735ad6 (125000406) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
^C
</pre>

=== CDM link status, PRBS test mode ===

<pre>
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
</pre>
* fiber disconnected, no link:
<pre>
CDM firmware: 0xbb2f0ae7
0x1000 SFP c.c. status: 0x00000031
CLK_IN_SEL_LS 0x1
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x00000025
sfp_mod_absent_N 1
sfp_rx_los_N 0
link_power_good 1
rx_link_up 0
rx_receiving_data 0
rx_error 1
rx_lnk_up_and_running 0
tx_link_up 0
tx_sending_data 0
tx_link_up_and_running 0
link_up_and_running 0
0x1014 SFP link data: 0x466a8187
rx_data 0x8187
k28p1_k28p5 0
rx_ctrl0 0x1
rx_ctrl1 0x1
rx_ctrl3 0x1
tx_state 0x1
rx_state 0x6
rx_receiving_data 0
rx_link_up 0
rx_error 1
rx_link_rst 0
0x2000 link test mode: 0x00000200, seconds: 0x00079093, errors: 0xffffffff
</pre>
* fiber connected, good link:
<pre>
CDM firmware: 0xbb2f0ae7
0x1000 SFP c.c. status: 0x000000b2
CLK_IN_SEL_LS 0x2
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x000007dc
sfp_mod_absent_N 0
sfp_rx_los_N 0
link_power_good 1
rx_link_up 1
rx_receiving_data 1
rx_error 0
rx_lnk_up_and_running 1
tx_link_up 1
tx_sending_data 1
tx_link_up_and_running 1
link_up_and_running 1
0x1014 SFP link data: 0x35c02774
rx_data 0x2774
k28p1_k28p5 0
rx_ctrl0 0x0
rx_ctrl1 0x0
rx_ctrl3 0x0
tx_state 0x3
rx_state 0x5
rx_receiving_data 1
rx_link_up 1
rx_error 0
rx_link_rst 0
0x2000 link test mode: 0x00000200, seconds: 0x00078a8c, errors: 0x00000000
</pre>

=== GDM link status, PRBS test mode ===

<pre>
1 link connected, no errors:
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
GDM firmware: 0x6b2ee010
0x1014: 0x00000008, 18: 0x00000008, 1C: 0x00000000, 24: 0x00000fff
0x2000: 0x00000200, time: 0x00078aa4, errors:
0xffffffff 0xffffffff 0xffffffff
0x00000000 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
</pre>

=== CDM link status ===

<pre>
root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test0

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x00 0x00 0xd0 0x01 0x1f 0xfe 0x22 0xf2, 0x507: 0xbf, 0x52A: 0x02, 0x53F: 0x02
Clock chip state 1, status: IN2 IN_SEL_1 HOLD_HIST_VALID

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x077357a0 (124999584) should be ~125 MHz
0x1034 rx_clk: 0x07735a5c (125000284) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a5b (125000283) should be ~125 MHz
0x103C tx_clk: 0x07735a5b (125000283) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
CDM firmware: 0x6d2ef81a
0x1000 SFP c.c. status: 0x000000b2
CLK_IN_SEL_LS 0x2
CLK_EXT_SEL_LS 0
CLK_CLK_RSTn_LS 0
CLK_LOSXTn_LS 1
CLK_LOLn_LS 1
CLK_INTn_LS 0
0x1008 SFP link reset: 0x00000000
0x1010 SFP link status: 0x000007dc
sfp_mod_absent_N 0
sfp_rx_los_N 0
link_power_good 1
rx_link_up 1
rx_receiving_data 1
rx_error 0
rx_lnk_up_and_running 1
tx_link_up 1
tx_sending_data 1
tx_link_up_and_running 1
link_up_and_running 1
0x1014 SFP link data: 0x35c6bcbc
rx_data 0xbcbc
k28p1_k28p5 0
rx_ctrl0 0x3
rx_ctrl1 0x0
rx_ctrl3 0x0
tx_state 0x3
rx_state 0x5
rx_receiving_data 1
rx_link_up 1
rx_error 0
rx_link_rst 0
0x2000 link test mode: 0x00000000, seconds: 0x00001671, errors: 0xffffffff

root@cdm1:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 15
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ef81a
CDM firmware 0x6d2ef81a
reg[15] is 0x0033bcbc (3390652)
</pre>

=== GDM link status ===

<pre>
root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test0

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --cc
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
Polling CC status...
Clock chip registers: 0x06 0x00 0x94 0x53 0x00 0x00 0x00 0x00 0x00 0x00 0x01 0x68 0x00 0x44 0x00 0xd0 0x01 0x1f 0xff 0x22 0xf2, 0x507: 0x3f, 0x52A: 0x01, 0x53F: 0x02
Clock chip state 1, status: LOS_IN2 OOF_IN2 IN0 IN_SEL_REGCTRL IN_SEL_0 HOLD_HIST_VALID

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735a0a (125000202) should be ~125 MHz
0x1034 rx_clk: 0x07735a0a (125000202) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735a0a (125000202) should be ~125 MHz
0x103C tx_clk: 0x07735a0a (125000202) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly

register 0x1018 bit 0x800

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
GDM firmware: 0x6d2ebce6
0x1014: 0x00000800, 18: 0x00000800, 1C: 0x00000000, 24: 0x00000fff
0x2000: 0x00000000, time: 0x00003d2f, errors:
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff
0xffffffff 0xffffffff 0xffffffff

link data alternates 0xbcbc and 0x1cbc

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 22
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
reg[22] is 0xbcbc93ab (-1128492117)

root@gdm0:~# /home/dsdaq/online/ds-dm-software/test_cdm.exe 22
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware revision 0x6d2ebce6
CDM firmware 0x6d2ebce6
reg[22] is 0x1cbc1aaf (482089647)
root@gdm0:~#

</pre>

=== Run trg and tsm ===

<pre>
on the GDM:

ssh root@gdm00
./test_cdm.exe --gdm-clocks
./test_cdm.exe --load-cc
./test_cdm.exe --cc
./test_cdm.exe --reset-mgt
./test_cdm.exe --gdm-clocks
./test_cdm.exe --writereg 2 0xff # enable LEMO NIM inputs
./test_cdm.exe --writereg 3 0xba54 # LEDs: lemo1, lemo2, trg, tsm
./test_cdm.exe --writereg 4 0x99 $ # LEMO out is trg_in_pulse
./test_cdm.exe --writereg 9 0x32010 # trg and tsm from trg_pulser and tsm_pulse
./test_cdm.exe --writereg 24 1250000 # trg pulser 100 Hz
./test_cdm.exe --writereg 26 125000000 # tsm pulser 1 Hz
./test_cdm.exe --writereg 23 0x40000000 # route trg_in and tsm_in to qsfp tx bits 0 and 1
./test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz

on the CDM:

ssh root@cdm01
./test_cdm.exe --cdm-clocks
./test_cdm.exe --load-cc
./test_cdm.exe --reset-mgt
./test_cdm.exe --cdm-clocks
./test_cdm.exe --cdm-link # issue --reset-mgt on CDM and GDM until link is good
./test_cdm.exe --writereg 2 0xff # enable LEMO NIM inputs
./test_cdm.exe --writereg 3 0xba54 # enable LEDs: lemo1, lemo2, trg, tsm
./test_cdm.exe --writereg 4 0x99 # enable LEMO output trg_in
./test_cdm.exe --writereg 9 0x8040 # enable trg_in and tsm_in from sfp rx bits 0 and 1
./test_cdm.exe --writereg 7 0xff00 # drive VX LVDS lines to logic level 0
./test_cdm.exe --writereg 8 0x2 # VX LVDS with trg and tsm, misrouted in second VX
#./test_cdm.exe --writereg 8 0x7 # VX LVDS with tsm
./test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz
</pre>

switch GDM and CDM to packetizer trg and tsm:

<pre>
on the GDM:

/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x00000000

on the CDM:

/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x0804
/home/dsdaq/online/ds-dm-software/test_cdm.exe --counters # observe counters are counting at 100 Hz and 1 Hz
</pre>

=== Run packet loopback ===

GDM CPU -> fifo_to_fpga -> GDM QSFP -> CDM SFP -> fifo_from_fpga -> CDM CPU

On the GDM: (CDM is connected to first QSFP port)

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --load-cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --reset-mgt
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 32 1 ### tell GDM to use first QSFP port
/home/dsdaq/online/ds-dm-software/test_cdm.exe --gdm-link ### confirm link status is "3"
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 29 2 ### enable GDM packet data injection
/home/dsdaq/online/ds-dm-software/test_cdm.exe --test-fifo-write-loop
</pre>

One the CDM:

<pre>
/home/dsdaq/online/ds-dm-software/test_cdm.exe --load-cc
/home/dsdaq/online/ds-dm-software/test_cdm.exe --reset-mgt
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 29 3 ### enable CDM packet data injection
/home/dsdaq/online/ds-dm-software/test_cdm.exe --test-fifo-read
</pre>

== fecdm ==

=== Power up and clock management ===

* Clock power up and setup management is handled by fecdm state machine
* GDM clock setup sequence:
<pre>
- on power up, CC is not initialized, XAXB clock is not running
- on GDM frontend start (in HandleInit()), CC is loaded and reset
- CC will lock to IN0 (10 MHz external clock) or IN1 (125 MHz internal oscillator)
- if locked to IN0, CC will enter and exit the HOLDOVER mode if 10 MHz clock is unstable
</pre>
* CDM clock setup sequence:
<pre>
- on power up, CC is not initialized, XAXB clock is not running
- fecdm detects this, loads the CC configuration registers, resets the CC
- sets the CC input IN1, DSDM internal oscillator
- if SFP is in good state, SFP MGT is running, CC is happy with IN2 input (no IN2_OOF, IN2_LOS), SFP recpvered clock (clk_rx_mgt) frequency is close to 125 MHz:
- switches CC to input IN2 (SFP recovered clock, clk_rx_mgt)
- waits for CC locked to IN2 (no LOL, no HOLD). (this provides stable SFP tx clock).
- resets the SFP MGT, requests GDM MGT reset (via ODB)
- MGT reset sequence: GDM MGT reset 1, CDMnn MGT reset 1, GDM MGT reset 0, CDM MGT reset 0
- CDM MGT rx_state cycles from 0 to 5. if it goes to 6 or never reaches 5, GDM MGT reset is requested again.
- during this sequence, clk_rx_mgt may wobble, CC may enter HOLDOVER mode, VX clock should stay steady.
- once CC is locked to IN2, SFP MGT rx_state is 5, link is ready for use.
</pre>
* Request changes for Ian
<pre>
17 Apr 2025:
- CDM SFP MGT rx_state is getting stuck in 4 (waiting for alternating idle chars), do we need this? go to state 5 without waiting?
- GDM QSFP MGT reset 1 does not seem to do anything, change it to cause all CDM MGTs to enter an error state, e.g. start transmitting invalid K-codes. transmitting all 0 or all 1 will break the recovered clock, so maybe do not do that.
- having all CDM SFP MGTs go to error state immediatley after GDM QSFT MGT reset 1 will greatly simplify the link reset sequence.
</pre>

=== Trigger configuration ===

* GDM
* CDM in SFP trigger mode:
** SFP K-code encoded sfp_rx_trg, sfp_rx_tsm are routed to LVDS vx_tx_trg and vx_tx_tsm
** GDM TRG, TSM (and all other) packets are routed to LVDS serial data to VX
** to make counters count and lights blink:
** CDM TRG and TSM trigger mask is set to sfp_rx_trg, sfp_rx_tsm and vx_tx_trg_done, vx_tx_tsm_done
** CDM LEDs display trg_in_pulse, tsm_in_pulse
** output of TRG and TSM packets is disabled (they come from the GDM)

= dsvslice integration =

== VX setup ==

* general
** Start acq from user code = y, all others = n (as of Mar 2023: will start when first trigger received)
** Use NIM IO = y
** Use external clock = y
** LVDS quartet is input = n, y, n, y
** LVDS quartet mode = User, User, User, User

* trigger from front panel NIM:
** Trigger on external signal = y, all others = n
** connect CDM EXT_OUT(2) to VX "TrigIn"

* trigger from LVDS "Sync" mode
** Trigger on LVDS Sync signal = y, all others = n
** LVDS quartet mode = User, Sync, User, User

* trigger from LVDS "User" mode
** Trigger on LVDS pair 12 signal = y, all others = n
** LVDS quartet is input = n, y, n, y
** LVDS quartet mode = User, User, User, User

== GDM setup ==

* GDM is gdm0
* set inputs to NIM mode
* set outputs to TTL mode (this GDM has wrong NIM output circuit)
* use top QSFP slot, connect split cable 1 into CDM01, cable 2 into CDM02
* connect non-inverted NIM trigger signal to top-LEMO-left EXT_IN_LV(1)
* connect non-inverted NIM TSM signal to top-LEME-right EXT_IN_LV(2)
* GDM LEDs: TRIG, TSM, trigger enabled, trigger_out
* GDM LEMO_OUT: trigger, trigger

== CDM setup ==

* set CDM LEMO inputs to NIM
* set CDM LEMO outputs to NIM
* CDM01 is cdm0
* CDM02 is cdm1
* connect GDM fiber links to SFP port
* connect 1st VX port of CDM01 to VX1
* connect 1st VX port of CDM02 to VX2
* connect LEMO EXT_OUT(2) to VX "TrigIn", CDM01 to VX1, CDM02 to VX2
* power up
* CDM LEDs: GDM TRIG, GDM TSM, trigger enabled, trigger out
* CDM LEMO_OUT: gdm_trg, trigger

== After power up ==

* start the CDM frontend from the MIDAS "Programs" page. To start manually, see the Start Command on the Programs page.
* CDM frontend should enable the VX clock, disable the trigger
* from the MIDAS status page, goto the CDM page
* outdated: in the CDMx data tables, the 2nd number should read 0x35c08008, if it does not and the last 4 digits randomly change, reset the GDM links
*if the FEs complain - do in order: for GDM, CDM01, CDM02, ..., press "reset mgt" of each board, then press "unreset mgt", if it does not help, STOP HERE
* start a run
* CDM frontend will enable the trigger
* GDM frontend will enable the trigger
* LEDs on the GDM should flash, LEDs on the CDM should flash, TrigIn and TrigOut of the VX should flash
* stop a run
* GDM frontend will disable the trigger
* CDM frontend will disable the trigger

== Phase measurement ==

* pip3 install matplotlib
* pip3 install scipy
* export PYTHONPATH=$HOME/packages/midas/python
* #git clone https://github.com/J033X071C/PhaseMeasurement
* git clone https://bitbucket.org/team-ds-dm/phasemeasurement.git
* cd phasemeasurement
* python3 ./phaseMeasurement.py --help
<pre>
daq00:PhaseMeasurement$ python3 ./phaseMeasurement.py --help
usage: phaseMeasurement.py [-h] fileName numberEvents numberVX sizeEvents stopEvent minHist maxHist numberBin writeToTXT saveAsPDF

Read data from midas file (in .lz4 format) to calculate phase between the clock of VX1 and VX2

positional arguments:
fileName Name of the file we want to read data from (Example: run00389.mid.lz4)
numberEvents Number of events recorded in the file
numberVX Number of VX used in this run (usually 2...)
sizeEvents Number of points per event
stopEvent Number of events you want to go through to calculate phase
minHist Minimal value for the x axis of the phase measurement histogram (in ns)
maxHist Maximal value for the x axis of the phase measurement histogram (in ns)
numberBin Number of bins wanted for the generated histogram
writeToTXT Write argument as yes to generate text file with results of calculation
saveAsPDF Save generated plots to PDF files

optional arguments:
-h, --help show this help message and exit
daq00:PhaseMeasurement$
</pre>
* try an old file with
* python3 ./phaseMeasurement.py run00877.mid.lz4 10000 2 10000 500 -20 20 81 yes yes
<pre>
bin size = 0.494 ns
num_events = 347
mean = -1.705 ns
rms = 3.087 ns
mean_error = 0.166 ns
centroid = -1.706 ns.
width (sigma) = 0.363 ns.
error on the centroid = 0.016558 ns.
</pre>
* ls -l *.txt *.pdf
<pre>
dsdaq@dsvslice:~/online/PhaseMeasurement$ ls -l *.txt *.pdf
-rw-rw-r-- 1 dsdaq dsdaq 64728 Dec 14 16:56 run00877.mid.lz4_Plots.pdf
-rw-rw-r-- 1 dsdaq dsdaq 274 Dec 14 16:56 run00877.mid.lz4.txt
dsdaq@dsvslice:~/online/PhaseMeasurement$
</pre>
* scope settings (from email message)
<pre>
From fcote-lortie@triumf.ca Thu Dec 15 17:11:46 2022
From: Francis Cote-Lortie <fcote-lortie@triumf.ca>
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: Re: How to use scope as a waveform generator
Date: Fri, 16 Dec 2022 01:11:44 +0000

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display) by pressing on the button. It will go from 0 to 1.

The settings that we are using right now are:
Type of waveform: Sine wave
Offset: 0 V
Amplitude: 1 Vpp
Frequency: 50 kHz
Noise: 0 V
________________________________
From: Francis Cote-Lortie <fcote-lortie@triumf.ca>
Sent: Thursday, December 15, 2022 4:58 PM
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: Re: How to use scope as a waveform generator

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display)

The settings that we are using right now are:
Type of waveform: Sine wave
Offset: 0 V
Amplitude: 1 Vpp
Frequency: 50 kHz
Noise: 0 V
________________________________
From: Francis Cote-Lortie
Sent: Thursday, December 15, 2022 4:54 PM
To: Konstantin Olchanski <olchansk@triumf.ca>
Subject: How to use scope as a waveform generator

1. Turn the power on (bottom left of the scope)
2. Access the waveform generator display by pressing the Gen button (bottom right of the scope)
3. The waveform generator display allows you to choose wave type (sine, square, etc.), frequency, amplitude, offset, etc. The display is a touch screen. Make the
waveform you want by using the different options.
4. Turn the output on (first option on the display)
</pre>

== Standalone link test ==

<pre>
CDM: program clock chip
busybox devmem 0x80011000 32 0x8
busybox devmem 0x80011000 32 0x0
/home/dsdaq/si5394-i2c-file CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b

GDM, CDM: link reset
busybox devmem 0x80011008 32 1

GDM, CDM: release reset
busybox devmem 0x80011008 32 0

CDM: link status (NOTE: SFP LOS and mod_absent are swapped!!!)
busybox devmem 0x80011010 32
0x00000024 <- fiber plugged
0x00000025 <- fiber unplugged
0x00000027 <- SFP unplugged
0x000007DC <- successful link with GDM

CDM: link state machine and data
busybox devmem 0x80011014 32
0x35C06FF6

CDM: set link to counting mode
busybox devmem 0x80012000 32 0x101
busybox devmem 0x80012000 32 0x100

CDM: time counter and error counter
root@cdm1:~# busybox devmem 0x80012000 32
0x00000100 <--- link mode
root@cdm1:~# busybox devmem 0x80012004 32
0x0000058C <--- seconds counter
root@cdm1:~# busybox devmem 0x80012004 32
0x0000058D
root@cdm1:~# busybox devmem 0x80012008 32
0x00000000 <--- error counter

GDM: no link
root@gdm0:~# busybox devmem 0x80011014 32
0x00000000
root@gdm0:~# busybox devmem 0x80011018 32
0x00000000
root@gdm0:~# busybox devmem 0x8001101c 32
0x00000000
root@gdm0:~# busybox devmem 0x80011024 32
0x00000FFF
root@gdm0:~#

GDM: good link channel 10, counting mode
root@gdm0:~# busybox devmem 0x80012000 32 0x101
root@gdm0:~# busybox devmem 0x80012000 32 0x100
root@gdm0:~# busybox devmem 0x80012008 32
0x3A8B68C2
root@gdm0:~# busybox devmem 0x80012008 32
0x42E03BEF
root@gdm0:~# busybox devmem 0x8001200c 32
0xDA090972
root@gdm0:~# busybox devmem 0x8001200c 32
0xDE6F22E9
root@gdm0:~# busybox devmem 0x80012019 32
Bus error
root@gdm0:~# busybox devmem 0x80012010 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012014 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012018 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x8001201c 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012020 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012024 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012028 32
0x00000000
root@gdm0:~# busybox devmem 0x8001202c 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012030 32
0xFFFFFFFF
root@gdm0:~# busybox devmem 0x80012034 32
0xFFFFFFFF
root@gdm0:~#
</pre>

mapping of link channels:

<pre>
qsfp0 lane0 - 0x0100 - link 8
qsfp0 lane1 - 0x0200 - link 9
qsfp0 lane2 - 0x0400 - link 10
qsfp0 lane3 - n/c
qsfp1 lane0 - 0x0800 - link 11
qsfp1 lane1 - 0x0010 - link 4
qsfp1 lane2 - 0x0020 - link 5
qsfp1 lane3 - n/c
qsfp2 lane0 - 0x0040 - link 6
qsfp2 lane1 - 0x0080 - link 7
qsfp2 lane2 - 0x0001 - link 0
qsfp2 lane3 - n/c
qsfp3 lane0 - 0x0002 - link 1
qsfp3 lane1 - 0x0004 - link 2
qsfp3 lane2 - 0x0008 - link 3
qsfp3 lane3 - n/c
</pre>

script to start the test with 2 CDMs:

<pre>
ssh dsdaq@dsvslice
ssh root@gdm0 busybox devmem 0x80011008 32 1
ssh root@cdm0 busybox devmem 0x80011008 32 1
ssh root@cdm1 busybox devmem 0x80011008 32 1
ssh root@gdm0 busybox devmem 0x80011008 32 0
ssh root@cdm1 busybox devmem 0x80011008 32 0
ssh root@cdm0 busybox devmem 0x80011008 32 0
ssh root@gdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
ssh root@cdm0 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
ssh root@cdm1 /home/dsdaq/online/ds-dm-software/test_cdm.exe --link-test2
</pre>

== LEMO trigger GDM to CDM to VX ==

on the GDM:

<pre>
/home/dsdaq/si5394-i2c-file /home/dsdaq/GDM_v1.0_IN0_EXT1_and_IN1_fixed_Si5394-RevA-Registers.txt 0 0x6b
busybox devmem 0x80011008 32 0x1
busybox devmem 0x80011008 32 0x0
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x80001230 # QSFP TX fixed pattern
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 2 0xff # enable LEMO inputs
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 3 0x7654 # enable LED, one per LEMO input
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x0F0F # enable LEMO to trg_in and tsm_in
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 23 0x40001230 # enable trg_in and tsm_in output to QSFP
</pre>

on the CDM:

<pre>
/home/dsdaq/si5394-i2c-file /home/dsdaq/CDM_v3.0_IN1_fixed_and_IN2_RX_Recovered_VX_62.5MHz_Si5394-RevA-Registers.txt 0 0x6b
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-clocks
/home/dsdaq/online/ds-dm-software/test_cdm.exe --sfp
busybox devmem 0x80011008 32 1
busybox devmem 0x80011008 32 0
/home/dsdaq/online/ds-dm-software/test_cdm.exe --cdm-link
/home/dsdaq/online/ds-dm-software/test_cdm.exe --writereg 9 0x8040 # trg_in from sfp[0], tsm_in form sfp[1]
/home/dsdaq/online/ds-dm-software/test_cdm.exe 11 # trg_in counter
/home/dsdaq/online/ds-dm-software/test_cdm.exe 12 # tsm_in counter
</pre>

== GPS receiver VCL-2705 ==

* Valiant VCL-2705 GPS receiver
* https://www.valiantcom.com/time-distribution/gps-receiver-irig-b.html
* USB connection is /dev/ttyACM0, 115200 bps, "GNSSAUX" prompt
* minicom -D /dev/ttyACM0 -b 115200
* user manual download instructoins - see sheet of paper with user name and password in the shipping box
* usb commands:
<pre>
gnss-help
gnss-showver -> F/W: Ver 1.7 Feb 17 2020 16:20:43

gnss-showselftest -> no antenna connected
Overall : FAIL
EPROM Test: PASS
Antenna : NOT DETECTED
GNSS : COMMUNICATION OK

gnss-showsettings
GNSS NMEA BAUDRATE :115200
GNSS ANTENNA LENGTH :30 meters
GNSS USER CONFIGURED DELAY:-65 nanoseconds
GNSS 1PPS PULSE WIDTH :200 milliseconds
GNSS MODE :GPS
GNSS STATUS :STATIONARY

GNSSAUX> gnss-showserial
SERIAL :2704H01V17MAX310

GNSSAUX> gnss-showmode
GNSS MODE : GPS

GNSSAUX> gnss-showstatus
GNSS STATUS: STATIONARY

GNSSAUX> gnss-showalarms
CURRENT ALARMS GNSS
ANTENNA : NOT DETECTED

GNSSAUX> gnss-showerrors
CURRENT ALARMS GNSS
ANTENNA : **NOT DETECTED
ERROR STATISTICS GNSS
RMC GOOD DURATION : Secs 0
RMC BAD DURATION : Secs 0
LOCK GOOD SECS DURATION : Secs 0
LOCK BAD SECS DURATION : Secs 0
SATINFO GOOD ITERATIONS :0
SATINFO REJECT ITERATIONS :0
SATINFO NOTALKER ITERATIONS :0
CURRENT MONITOR STATE GNSS
STATE :Phase-1 HUNTING ANTENNA DETECT

GNSSAUX> gnss-showsatinfo
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-showsats
Total Sats: 0

GNSSAUX> gnss-showmyloc
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-show1ppsstate
GNSS RECEIVER ANTENNA Not Detected !

GNSSAUX> gnss-showjamstatus
Not Available !

GNSSAUX> gnss-showspoofstatus
Not Available !

--- antenna connected, can see the sky ---

GNSSAUX> gnss-showselftest
Overall : PASS
EPROM Test: PASS
Antenna : DETECTED
GNSS : COMMUNICATION OK

GNSSAUX> gnss-showalarms
CURRENT ALARMS GNSS
ANTENNA : DETECTED
GNSS LOCK : AVAILABLE

GNSSAUX> gnss-showerrors
CURRENT ALARMS GNSS
ANTENNA : DETECTED
GNSS LOCK : AVAILABLE
ERROR STATISTICS GNSS
RMC GOOD DURATION : Mins 1,Secs 34
RMC BAD DURATION : Secs 55
LOCK GOOD SECS DURATION : Mins 1,Secs 34
LOCK BAD SECS DURATION : Secs 55
SATINFO GOOD ITERATIONS :3
SATINFO REJECT ITERATIONS :0
SATINFO NOTALKER ITERATIONS :0
CURRENT MONITOR STATE GNSS
STATE :Phase-4 NORMAL OPERATION, Monitoring GNSSLOCK

GNSSAUX> gnss-showsatinfo
GNSS MODE :GPS
SATELLITE INFORMATION TALKER:GPS
NMEA ID : 01-32
NO OF SATELLITES IN VIEW: 08
NO OF XXGSV MSGS : 03
SatNo PRN NO (SV ID) ELEVATION (degs) AZIMUTH (degs) C/No (SNR)
1 0 0 0 0
2 0 0 0 0
3 0 0 0 0
4 0 0 0 0
5 0 0 0 0
6 0 0 0 0
7 0 0 0 0
8 0 0 0 0
<<<<<< End of Sat Info >>>>>>

GNSSAUX> gnss-showsats
Total Sats: 8

GNSSAUX> gnss-showmyloc
GNSS RECEIVER LOCATION:
Latitude : 4914.81911
Longitude: 12313.69595

GNSSAUX> gnss-show1ppsstate
GPS 1PPS STATE: LOCKED

GNSSAUX> gnss-showmyloc

GNSS RECEIVER LOCATION:
Latitude : 4914.80688
Longitude: 12313.69531

enter into google maps search box as: 49 14.80688, -123 13.69531, observe the space, the moved dot and the minus.

GNSSAUX> gnss-resetgnss

Executing....Please Wait....
$$$$$END
</pre>

* 1PPS BNC output: period 1 sec, pulse width 200 ms, 3.3V into 1MOhm, 1.38V into 50Ohm.
* IRIG-B BNC output: 5.6V into 1MOhm, 2.4V into 50Ohm.
* IRIG-B format selector: default is all up.

== Rb clock PRS10 ==

* https://www.thinksrs.com/products/prs10.html
* 10 MHz output is sine wave around 5V peak to peak
* 1PPS BNC output is 10 usec pulse, 5V into 1MHohm.
* RS232 connection: minicom -D /dev/ttyUSB0 -b 9600
* ser2net config: localhost,3001:raw:600:usb-5-2-1.0:9600 -XONXOFF -RTSCTS LOCAL
* ssh daq13, cd /home/olchansk/git/ser2net, ./ser2net -c ~/daq/ds/ser2net.conf -d
* ssh daq13, cd ~/daq/ds, python3 prs10.py
* RS232 commands:
<pre>
ID?
PRS10_3.56_SN_105719
VB1
SN?
RS1 -- reset
ST? -- status
FC? -- 10MHz OCXO drive voltage DAC settings
DS? -- "detected signals"
GA? -- gain of frequency lock loop between ovenized oscillator and Rb cell, 0=use ovenized oscillator only
MO? -- magnetic offset of the Rb cell, range 2300..3600, if out of range, unit must be set to different operating mode, see prs10m.pdf
MR? -- magnetic read
TT? -- time-tag, time in ns between 1PPS out and 1PPS in
TS? -- time slope, ???
TO? -- time offset, ???
PS? -- pulse slope, ???
PL? -- 0=phase lock off, 1=phase lock on, lock to 1PPS input
PT? -- phase lock integrator time constant, PT8 is integrator time constant 18.2 hours, natural time constant 2.25 hours
PF? -- phase lock stability factor, PF2 is "1"
PI? -- phase lock integrator

Analog to digital 12 bit ADC, values 0.000 to 4.998

AD0? -- Spare (J204)
AD1? -- +24V(heater supply) divided by 10.
AD2? -- +24V(electronics supply) divided by 10
AD3? -- Drain voltage to lamp FET divided by 10
AD4? -- Gate voltage to lamp FET divided by 10
AD5? -- Crystal heater control voltage
AD6? -- Resonance cell heater control voltage
AD7? -- Discharge lamp heater control voltage
AD8? -- Amplified ac photosignal
AD9? -- Photocell’s I/V converter voltage divided by 4
AD10? -- Case temperature (10 mV/°C)
AD11? -- Crystal thermistors
AD12? -- Cell thermistors
AD13? -- Lamp thermistors
AD14? -- Frequency calibration pot / external calibration voltage
AD15? -- Analog ground

A/D via CPU E-port:

AD16? -- Varactor voltage for 22.48 MHz VCXO (inside RF synthesizer) / 4
AD17? -- Varactor voltage for 360 MHz VCO (output of RF synthesizer) / 4
AD18? -- Gain control voltage for amplifier which drives frequency multiplier / 4
AD19? -- RF synthesizer’s lock indicator voltage (nominally 4.8 V when locked )

ST?

ST1 : Power supplies and Discharge Lamp
ST1 bit, Condition which sets bit, Corrective Action
0 -- +24 for electronic < +22 Vdc
1 -- +24 for electronics > +30 Vdc
2 -- +24 for heaters <+22 Vdc
3 -- +24 for heaters > +30 Vdc
4 -- Lamp light level too low
5 -- Lamp light level too high
6 -- Gate voltage too low
7 -- Gate voltage too high

ST2: RF Synthesizer
ST2 bit, Condition which sets bit, Corrective Action
0 -- RF synthesizer PLL unlocked
1 -- RF crystal varactor too low
2 -- RF crystal varactor too high
3 -- RF VCO control too low
4 -- RF VCO control too high
5 -- RF AGC control too low
6 -- RF AGC control too high
7 -- Bad PLL parameter

ST3: Temperature Controllers
ST3 bit, Condition which sets bit
0 -- Lamp temp below set point
1 -- Lamp temp above set point
2 -- Crystal temp below set point
3 -- Crystal temp above set point
4 -- Cell temp below set point
5 -- Cell temp above set point
6 -- Case temperature too low
7 -- Case temperature too high

ST4: Frequency Lock-Loop Control
ST4 bit, Condition which sets bit
0 -- Frequency lock control is off
1 -- Frequency lock is disabled
2 -- 10 MHz EFC is too high
3 -- 10 MHz EFC is too low
4 -- Analog cal voltage > 4.9 V
5 -- Analog cal voltage < 0.1
6 -- not used
7 -- not used

ST5: Frequency Lock to External 1pps
ST5 bit, Condition which sets bit
0 -- PLL disabled
1 -- < 256 good 1pps inputs
2 -- PLL active
3 -- > 256 bad 1pps inputs
4 -- Excessive time interval
5 -- PLL restarted
6 -- f control saturated
7 -- No 1pps input

ST6: System Level Events
ST6 bit and Condition which sets bit
0 Lamp restart
1 Watchdog time-out and reset
2 Bad interrupt vector
3 EEPROM write failure
4 EEPROM data corruption
5 Bad command syntax
6 Bad command parameter
7 Unit has been reset
</pre>
* ST? on warm start
<pre>
received: PRS_10
received: 255,255,255,243,34,255
received: 0,0,0,1,34,0
...
received: 0,0,0,1,34,0
received: 0,0,0,0,2,0
...
received: 0,0,0,0,2,0
received: 0,0,0,0,4,0
</pre>
* ST? on loss of external 1PPS
<pre>
...
received: 0,0,0,0,4,0
disconnect 1PPS input
received: 0,0,0,0,132,0
...
reconnect 1PPS input
received: 0,0,0,0,132,0
received: 0,0,0,0,4,0
...
</pre>
* ST? on coldish start
<pre>
daq13:ds$ python3 prs10.py
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,0,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 0,0,1,1,34,0 ] old [ 80,0,0,1,34,1 ] counter: 0
received [ 0,0,0,1,34,0 ] old [ 0,0,1,1,34,0 ] counter: 0
received [ 0,0,16,1,34,0 ] old [ 0,0,0,1,34,0 ] counter: 6
received [ 0,0,20,1,34,0 ] old [ 0,0,16,1,34,0 ] counter: 5
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 8
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 3
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 49
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 250
...
</pre>
* ST? on cold start, note: no bump in the 10MHz clock as reported by DS-DM clock chip
<pre>
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,21,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 64,0,21,1,34,0 ] old [ 80,0,21,1,34,1 ] counter: 0
received [ 0,0,21,1,34,0 ] old [ 64,0,21,1,34,0 ] counter: 0
received [ 0,0,20,1,34,0 ] old [ 0,0,21,1,34,0 ] counter: 158
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 23
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 47
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 40
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 249
</pre>

= det fac integration test =

Connections:
* GPS receiver "IRIG-B SEL" both switches "up" - both "on", IRIG-B format B004
* GPS receiver USB-B -> long cable -> daq13 USB-A
* GPS receiver "1PPS out" -> long BNC cable -> BNC-T -> scope (5V, no 50ohm) and Rb clock BNC "1PPS in"
* GPS receiver "IRIG-B 50ohms" -> long BNC cable -> BNC-T -> scope (5V, no 50ohm) and DS-DM LEMO input 1 (TTL mode)
* Rb clock RS232 -> RS232 straight cable -> RS232-to-USB adapter -> daq13 USB-A
* Rb clock "1PPS out" BNC -> scope (5V, no 50ohm, trig threshold rising edge 2V)
* Rb clock "10MHz output 50 Ohm" BNC -> lemo -> lemo-T -> scope (sine wave, 5V, no 50ohm) and DS-DM clock input.

Programs to run:
* on daq13: cd /home/olchansk/git/ser2net, ./ser2net -c ~/daq/ds/ser2net.conf -d
<pre>
localhost,3001:raw:600:usb-5-2-1.0:9600 -XONXOFF -RTSCTS LOCAL
</pre>
* on daq13: cd /home/olchansk/daq/ds, python3 prs10.py ### connects to ser2net
<pre>
daq13:ds$ python3 prs10.py
Connected
received [ ] old [ b'' ] counter: 0
received [ PRS_10 ] old [ ] counter: 0
received [ 255,255,255,243,34,255 ] old [ PRS_10 ] counter: 0
received [ 80,0,21,1,34,1 ] old [ 255,255,255,243,34,255 ] counter: 0
received [ 64,0,21,1,34,0 ] old [ 80,0,21,1,34,1 ] counter: 0
received [ 0,0,21,1,34,0 ] old [ 64,0,21,1,34,0 ] counter: 0
received [ 0,0,20,1,34,0 ] old [ 0,0,21,1,34,0 ] counter: 158
received [ 0,0,4,1,34,0 ] old [ 0,0,20,1,34,0 ] counter: 23
received [ 0,0,0,1,34,0 ] old [ 0,0,4,1,34,0 ] counter: 47
received [ 0,0,0,0,2,0 ] old [ 0,0,0,1,34,0 ] counter: 40
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,2,0 ] counter: 249
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 12096
received [ 0,0,0,0,20,0 ] old [ 0,0,0,0,132,0 ] counter: 2
received [ 0,0,0,0,148,0 ] old [ 0,0,0,0,20,0 ] counter: 0
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,148,0 ] counter: 575
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 11449
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 2755
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 34386
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 41035
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 113401
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 33375
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 54767
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 85059
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 33222
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 119234
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 121990
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 128184
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 56002
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 428237
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 8250
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 1388
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 30506
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 142704
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 179451
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 106182
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 68747
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 65424
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 157587
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 6932
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 1388
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 20255
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 4
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 225941
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 72183
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 0
received [ 0,0,0,0,132,0 ] old [ 0,0,0,0,4,0 ] counter: 26970
received [ 0,0,0,0,4,0 ] old [ 0,0,0,0,132,0 ] counter: 1
39287
</pre>
* on ds-dm: ./test_cdm.exe --irigb ### note sbs mismatch is because I should wrap around at 16 bits
<pre>
dataframe: S00000000S100001010S...S001010101S000001000S, sec: 00, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16724 should be 82260
dataframe: S10000000S100001010S...S101010101S000001000S, sec: 01, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16725 should be 82261
dataframe: S01000000S100001010S...S011010101S000001000S, sec: 02, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16726 should be 82262
dataframe: S11000000S100001010S...S111010101S000001000S, sec: 03, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16727 should be 82263
dataframe: S00100000S100001010S...S000110101S000001000S, sec: 04, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16728 should be 82264
dataframe: S10100000S100001010S...S100110101S000001000S, sec: 05, min: 51, hrs: 22, day of year: 300, year: 2023, date: 27 oct, sbs: 16729 should be 82265
</pre>

= DS-IOGC GPS interface board =

* Rev0 git repository: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0/-/tree/main?ref_type=heads
* Rev0 schematics: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev0/-/blob/main/Altium/Project%20Outputs%20for%20DS-IOGC-Rev0/SCH-DS-IOGC-Rev0.pdf?ref_type=heads
* Rev0 schematics: [[:Image:SCH-DS-IOGC-Rev0.pdf|SCH-DS-IOGC-Rev0]]
* Rev1 git repository: https://edev-group.triumf.ca/hw/exp/dark-side-20k/dark-side-iogc/rev1/
* Rev1 schematics: [[:Image:SCH-DS-IOGC-Rev1.pdf|SCH-DS-IOGC-Rev1]]

== Changes Rev0 to Rev1 ==

<pre>
From: Peter Margetak <pmargetak@triumf.ca>
Subject: IOGC REV1 review
Date: Wed, 4 Sep 2024 07:31:19 +0000

Hi Konstantin,
Pls have a look at SCH for new rev. I'd like to send it to mfr next week so if you can comment by early next week. Meanwhile I work on layout and other stuff.

Changes:

New ICs - all powered +5V
U20 - inverters for RUclk RX/TX
U21 - non inverting line driver for RU-1pps-out (so you don't have to route if via GDM to see it on scope)
U22 - non inverting buffer for ext 1pps input

All Lemo connectors have the same position but they are double lemos now => new panel needed
@Marek Walczak<mailto:mwalczak@triumf.ca> you can print it ahead once pcb is done + update IOGC docs and panel description

J2A/B - Test ports for RU-1pps in and out
J5A/B - inputs for external GPS data and External source of 1pps
J6A/B - aux in/out for GDM

SW1 - no change - select RX/TX USB/GDM
SW2 - select latch sensitivity for rising/falling edge
SW3 - select source of GPS data (opto or ext) AND select source of 1pps input (latch or ext)

p.
</pre>

== PRS-10 Rb clock device ==

The Rb clock PRS-10 device provides these connections:
<pre>
RS232 RX input - serial communication, non-standard RS232
RS232 TX output - serial communication, non-standard RS232
10 MHz clock output - coax 50 Ohm high resolution 10 MHz clock
1pps output - 1 Hz clock corresponding to the 10 MHz clock
1pps input - 1pps signal from GPS receiver
</pre>

Mode of operation:
* 10 MHz clock is always running
* 1pps output is always running
* if 1pps input from GPS received is present, after 256 pulses PRS-10 will sync it's 1pps output with the 1pps input by adjusting the frequency of the 10 MHz clock
* when unlocked: 1pps output and 1pps input unrelated
* when locked to GPS: 1pps output and 1pps input always go up and down at the same time

Theory of operation:
* 10 MHz clock is produced by a high-quality crystal (stable on the scale of seconds)
* crystal oscillator is synchronized to a Rb cell (stable on the scale of hours and days)
* Rb cell resonant frequency is synchronized to the GPS 1pps signal (stable on the scale of months and years)

== Rev1 connections ==

* LEMO connectors (front panel)
<pre>
LEMO J2A output - Rb clock 1pps in monitor
LEMO J2B output - Rb clock 1pps out monitor
LEMO J5A input - GPS IRIG-B from GPS receiver to FPGA (VCL-2705)
LEMO J5B input - GPS 1pps from GPS receiver to PRS-10 (VCL-2705)
LEMO J6A output - AUX-OUT from FPGA (dual-LEMO PCB-side)
LEMO J6B input - AUX-IN to FPGA (dual-LEMO away-from -PCB)
</pre>

* SMB connectors (back)
<pre>
SMB J3 output - GPS 1pps loopback to LNGS
SMB J4 input - LNGS GPS data input
</pre>

* LEDs
<pre>
D1 - same as LEMO J2A out (Rb clock 1pps in)
D2 - same as SMB J3 out (GPS 1pps from LNGS or from a GPS receiver)
D5 - controlled by FPGA-OUT-LED1
D6 - controlled by FPGA-OUT-LED2
D7 - PRS-10 24V power ok
</pre>

* switches
<pre>
SW1A and SW1B - route PRS-10 RS232 to USB or to FPGA
SW2A - route PRS-10 1pps input from SMB J4 (LNGS) or from LEMO J5B (GPS receiver 1pps)
SW2B - route FPGA-IN-GPSDATA input from SMB J4 (LNGS) or from LEMO J5A (GPS receiver IRIG-B data)
SW3 - LNGS 1pps from rising edge or from falling edge of SMB J4 (LNGS)
</pre>

== Rb clock cable ==
<pre>
Rb clock ----- DS-IOGC side, pin numbers are as labeled on the cable connectors

1 - 1pps out - 8 - 1pps out --- correct
2 - nc
3 - nc
4 - TXD - 3 - RU-DATA-OUT - USB-RX input --- correct
5 - 1pps in - 2 - 1pps in --- correct
6 - +24V - 10 - +24V --- should by pin 1 to use both +24V pins?
7 - RXD - 15 - RU-DATA-IN - USB-TX output --- correct
8 - nc
9 - +24V - 10 - +24V --- correct
10 - GND - 9 - GND --- correct
</pre>

== VX connections ==
<pre>
VXA_TX0 - FPGA-OUT-LED2 - D6 LED ("10 MHz clock")
VXA_TX1 - FPGA-OUT-LED1 - D5 LED ("GPS DATA")
VXA_TX2 - FPGA-OUT-RU1PPS-EN - enable 1pps to Rb clock
VXA_TX3 - not used (62.5 Hz clock)

VXB_TX0 - FPGA-TX - PRS-10 RS-232 out
VXB_TX1 - FPGA-OUT-AUX - 5V TTL J5 LEMO out
VXB_TX2 - FPGA-OUT-OPTO1PPS-EN - PRS-10 power enable
VXB_TX3 - not used (62.5 MHz clock)

VXA_RX0 - FPGA-IN-OPTO1PPS - 1pps from GPS
VXA_RX1 - FPGA-IN-GPSDATA - J4 GPS data
VXA_RX2 - n/c
VXA_RX3 - n/c

VXB_RX0 - n/c
VXB_RX1 - FPGA-IN-AUX - J5 LEMO TTL input
VXB_RX2 - FPGA-RX - PRS-10 RS-232 in
VXB_RX3 - FPGA-IN_RU1PPS - PRS-10 1pps output
</pre>

== test sequence ==

* ./test_cdm_local.exe --writereg 7 0 ### clear reg 7
* ./test_cdm_local.exe --writereg 8 0 ### set vx_tx mux to vx_tx control from reg 7
* connect blue cable to GDM port 6 (next to the ethernet connector)
* ./test_cdm_local.exe --writereg 7 0x4000 ### power up
* test LEDs:
* ./test_cdm_local.exe --writereg 7 0x4100 ### right LED2 D6
* ./test_cdm_local.exe --writereg 7 0x4200 ### left LED1 D5
* test AUX-IN and AUX-OUT:
* ./test_cdm_local.exe --writereg 7 0x6000 ### J6A LEMO (PCB-side) measure +5VDC
* install LEMO jumper between LEMO J6A and J6B
* write 0x4000 then ./test_cdm_local.exe 6 ### reads 0x5C, bit 5 0x20 reads 0
* write 0x6000 then ./test_cdm_local.exe 6 ### reads 0x7C, bit 5 0x20 reads 1
* write 0x4000 to clear all bits
* test the 10 MHz clock:
* init the c.c: ./test_cdm_local.exe --load-cc
* connect 10 MHz clock output to DS-DM LEMO J6-LEFT (CLK_EXT1)
* 10 MHz output ok - IN0 is good: ./test_cdm_local.exe --cc
<pre>
Clock chip state 1, status: OOF_IN2 IN1 IN_SEL_1 HOLD_HIST_VALID
</pre>
* without 10 MHz reports IN0 LOS (no signal) and OOF (wrong frequency)
<pre>
Clock chip state 1, status: LOS_IN0 OOF_IN0 OOF_IN2 IN1 IN_SEL_1 HOLD_HIST_VALID
</pre>
* test GPS IRIG-B data:
* connect GPS IRIG-B signal to J6B (dual-LEMO away from PCB)
* IRIG-B via AUX-IN is ok: ./test_cdm_local.exe --irigb
<pre>
dataframe: S00100110S111001000S...S011110000S010000000S, sec: 34, min: 17, hrs: 00, day of year: 223, year: 2024, date: 10 aug, sbs: 1054 should be 1054
</pre>
* test IRIG-B via J5A GPS-EXT:
* connect GPS IRIG-B signal to J5A (dual-LEMO next to PCB)
* set SW2B: switch on the side of SMB connectors, slide direction of 24V power supply connector
* ./test_cdm_local.exe --irigb built for VXA_RX1 should work
* test GPS 1pps signal:
* connect GPS 1pps to J5B (dual-LEMO away from PCB)
* nothing happens, no output from U22
* remove R51
* set SW2A: switch on the side away from SMB connectors, slide direction of 24V power supply connector
* observe LEDs D1 and D2 blink at 1Hz
* SKIP THIS connect GPS 1pps to J4 using LEMO-SMB adapter
* SKIP THIS nothing happens, nLE is always 3.2V, both settings of SW2
* SKIP THIS !!!failed here!!!
* ./test_cdm_local.exe --writereg 8 9 ### set vx_tx mux to GPS control, hard enables PRS-10 power and PRS-10 1pps in
* observe 1pps counters in reg 68 count at 1 Hz: rb_1pps 0x51->0x52, gps_1pps 0xd5->oxd6
<pre>
root@dsdm:~# ./test_cdm_local.exe 68
ds20k_reg[68] is 0x0051d520 (5362976)
ds20k_reg[68] is 0x0052d624 (5428772)
...
</pre>
* observe PRS-10 can see the 1pps signal "130" changes to "2" after 243 seconds to "4"
* observe period of GPS and PRS-10 1pps is identical (plus/minus 1 clock)
<pre>
root@dsdm:~# ./test_cdm_local.exe 13 14
ds20k_reg[13] is 0x077356d4 (124999380)
ds20k_reg[14] is 0x077356d4 (124999380)
</pre>
* observe GDM/CDM clocks are exactly 125 MHz, we are running on DS-DM internal oscillator
<pre>
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x07735943 (125000003) should be ~125 MHz
0x1034 rx_clk: 0x07735943 (125000003) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735943 (125000003) should be ~125 MHz
0x103C tx_clk: 0x07735943 (125000003) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>
* if tx_clk is missing, reset the mgt: ./test_cdm_local.exe --reset-mgt
* switch CC to external clock:
<pre>
root@dsdm:~# ./test_cdm_local.exe --cc-in0
CC use clock input 0: 10 MHz LEMO external clock
root@dsdm:~# ./test_cdm_local.exe --cc
Polling CC status...
Clock chip state 1, status: IN0 IN_SEL_REGCTRL IN_SEL_0 HOLD_HIST_VALID
</pre>
* observe rx_clk and tx_clk are now slightly off: they run from PRS-10 10 MHz clock and are measured against the DS-DM internal oscillator.
<pre>
root@dsdm:~# ./test_cdm_local.exe --gdm-clocks
DS-DM mapping /dev/mem at 0x80010000
DS-DM FPGA firmware build 0x94b12519, ds20k version 0x20240814
DS-DM firmware build 0x94b12519, ds20k version 0x20240814
GDM clock frequency counters:
0x1030 mgt_rx_ref_clk_raw: 0x077358e1 (124999905) should be ~125 MHz
0x1034 rx_clk: 0x07735b49 (125000521) should be ~125 MHz
0x1038 mgt_tx_ref_clk_raw: 0x07735b49 (125000521) should be ~125 MHz
0x103C tx_clk: 0x07735b49 (125000521) should be ~125 MHz
0x1040 clk_50MHz: 0x02faf080 (50000000) should be 50 MHz exactly
0x1044 Block1_clk: 0x05f5e100 (100000000) should be 100 MHz exactly
</pre>
* observe GPS and PRS-10 1pps period is now exactly 125*10^6 of 8ns clocks (equal to 1 second)
<pre>
root@dsdm:~# ./test_cdm_local.exe 13 14
ds20k_reg[13] is 0x0773593f (124999999)
ds20k_reg[14] is 0x0773593f (124999999)
</pre>
* look at them repeatedly, observe reg 13 "GPS 1pps period" has some wobble, reg 14 "Rb clock 1pps period" is steady. this is as expected: if GPS 1pps was steady, we do not need to Rb clock. PRS-10 1pps is derived from the PRS-10 10 MHz clock and is measured against the 125 MHz clock derived from the same 10 MHz clock in the GDM FPGA.

Rev0 Test status:

* GPS 1pps to SMB-in ok (LED flashes)
* GPS 1pps to FPGA ok
* GPS data to FPGA ok
* GPS 1pps to PRS-10 enabled from FPGA ok
* PRS-10 Rb clock 1pps out to FPGA ok
* PRS-10 Rb clock 1pps out to FPGA to LEMO AUX out ok
* can see GPS 1pps, IRIG-B, PRS-10 1pps out, 10 MHz on the scope, ok
* PRS-10 syncs on leading edge (0->1) of GPS 1pps signal, ok
* reg 13 and 14 1pps periods are identical, ok
* CC locks on PRS-10 10 MHz clock, DS-DM runs on PRS-10 clock, ok
* NOT TESTED - smb output
* NOT TESTED - optical converter fiber to SMB
* NOT TESTED - optical converter SMB to fiber
* NOT TESTED - SMB loopback
* NOT TESTED - fiber loopback

Rev1 Test status: (21 Nov 2024)

* power up ok
* LEDs ok
* LEMO AUX-IN, AUX-out ok
* GPS IRIG-B to FPGA via J6B AUX-IN ok
* GPS IRIG-B to FPGA via J5A GPS-EXT ok
* PRS-10 Rb clock 1pps out to FPGA ok
* PRS-10 Rb clock 10 MHz to C.C. ok
* PRS-10 RS-232 ok
* J5B 1pps input fail. input signal is 3.3V without termination, 1.5V when plugged into IOGC (on R51), no output from U22.

= VX busy logic =

= DS-20K DAQ =

== Overview ==

DS-DM, GDM and CDM are key parts of the DS-20K DAQ system:
* common clock distribution from external clock (atomic clock, GPS) to GDM to per-quadrant CDMs to VX digitizers
* common trigger distribution from GDM internal algorithm or external input to all VX digitizers
* run control: GDM, CDM, VX all start recording data at the same time (clock and timestamp reset)
* collection of trigger data from VX digitizers to per-quadrant CDMs to GDM

== Deliverables ==

* hardware and firmware for GDM to CDM clock distribution
* hardware and firmware for CDM to VX clock distribution
* hardware and firmware for GDM external clock input (atomic clock or GPS)
* hardware and firmware for CDM and VX serial communications (VX LVDS I/O connector)
* firmware for run control (timestamp reset and sync): GDM to CDM to VX
* firmware for common trigger distribution: GDM to CDM to VX
* firmware for trigger data flow: VX to CDM to GDM
* firmware for busy control: VX to CDM to GDM back to CDM to VX
* firmware for flow control: FEP to GDM MIDAS frontend to GDM to CDM to VX
* GDM MIDAS frontend: clock selector and monitoring, trigger and run control, busy and flow control, GDM housekeeping
* CDM MIDAS frontend: clock monitoring, CDM housekeeping

specific performance:
* GDM external clock: 10 MHz GPS clock
* GDM to CDM fiber link:
** clock XXX MHz
** link data rate: XXX Gbit/sec
** CDM recovered clock: XXX MHz
** CDM recovered clock jitter: XXX ns
** phase alignment between CDMs: XXX ns
** phase alignment between CDMs persists across reboots, power cycles, firmware updates
** phase alignment between CDMs should be easy to measure
** phase alignment between CDMs should be easy to recalibrate if hardware parts are replaced (DS-DM boards, fiber transceivers, fiber cables, etc)
** data packet bandwidth: XXX Mbytes/sec
** data packet latency: XXX clocks
** data packet skew between CDMs: XXX clocks
* CDM to VX clock:
** clock: XXX MHz
** jitter, all CDM clock outputs: XXX MHz
** phase alignment between all CDM clock outputs: XXX ns
* CDM to VX trigger:
** TBD (use the VX "sync" input or VX LVDS I/O line or VX serial link packet)
* CDM to VX serial link:
** clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
** bit rate: XXX bits/sec
** latency: XXX link clocks
** maximum skew between VXes: XXX ns
* VX to CDM serial link:
** clock: XXX MHz (TBD: VX external clock, or LVDS I/O line or link recovered clock)
** bit rate: XXX bits/sec
** latency: XXX link clocks
** maximum skew between VXes: XXX ns
* timestamp reset:
** maximum skew between VXes: XXX ns
* busy round trip time: XXX ns (VX to CDM to GDM back to CDM to VX)
* flow control latency: XXX ns (FEP software to GDM MIDAS frontend to GDM to CDM to VX)

== Technical risk items ==

this refers to unexpected behaviour and performance of
system components, causes big difficulty in implementing the system,
prevents delivery of deliverables, and prevents or negatively affects operation
of the DS-20K DAQ or of the whole experiment.

(14-sep-2022, list is not sorted by any criterial: severity, probability, ease of investigation)

(stability of course is long term stability, across hours, days, weeks, months, years)

* stability of Enclustra FPGA modules (crashes/year, failures to boot/year, flash corruption/year)
* stability of GDM external clock PLL (lock loss/year)
* stability of CDM recovered clock (lock loss/year, unexpected phase drifts, etc)
* unexpected failures or bit error rates in GDM-CDM fiber links
* stability of CDM VX clock outputs (stability of clock cleaner chip)
* stability of VX internal clock distribution (VX PLL lock loss events)
* stability of VX CAEN base firmware (different versions of CAEN base firmware have different clock distribution behaviour)
* strange things in CAEN base firmware (unexpected clocking of LVDS I/O, unexpected phase shifts between clocks, etc)
* DS-DM and VX hardware problems (incompatible LVSD I/O, incompatible clock signals, etc)

== Milestones ==

(14-sep-2022: at this stage of the project, priority must be given to identifying and retiring (so called) technical risk factors. it is not good to build the complete system only to discover that (for example) some Enclustra FPGA modules require 5 attempts to boot and erase their flash memory contents once a month. Both example are real-life actual problems that caused big difficulties in GRIFFIN/TIGRESS and ALPHA-g experiments).

Development and testing milestones in time reversed order:

* full DAQ data challenge: all VXes, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
* one quadrant data challenge: 1 VME crate of VX, CDM, GDM, network, FEP, TSP, MIDAS operate as designed
* vertical slice data challenge: 1 VME crate, 2 VX, 2 CDM (1 VX per CDM), GDM, etc operate as designed
* GDM-CDM link finalized (data rate frozen, data packet format frozen, data content permitted to change)
* CDM-VX serial link finalized (data rate frozen, data packet format frozen, data content permitted to change)
* run control (timestamp reset) and trigger distribution design agreed upon, frozen (list of possible triggers permitted to change)
* VX to CDM to GDM data flow design agreed upon, frozen (data contents permitted to change)
* major technical risk items retired (all hardware and firmware is working as expecred without mysteries and surprises, all problems are identified, investigated, resolved, solutions tested)
* stable operation of CDM-VX serial links in vertical slice system
* stable operation of GDM to CDM clock in vertical slice system
* stable operation of CDM to VX clock in vertical slice system
* vertical slice system assembled (1 VME crate, 2 VX, 2 CDM, 1 GDM, network, FEP, TSP, MIDAS)

= ZZZ =

ZZZ