DaqWiki - User contributions [en]

Ubuntu

2024-04-18T23:26:43Z

Lmartin: /* setup nfs */

= About Ubuntu =

AAA

= Ubuntu version =

<pre>
lsb_release -a
uname -a
</pre>

= Ubuntu installer =

* updated for Ububtu LTS 20.04.01, 22.04.1

* download the latest Ubuntu LTS desktop installer iso image
* dd the image to a USB key
* power down, disconnect all disks (all HDDs, all SSDs, all M.2)
* connect the SSD to be used as system disk
* if system will use mirrored SSDs (using ZFS mirror), leave second SSD disconnected, we will activate it later
* power up
* boot from USB key in legacy mode or UEFI mode (select this in the BIOS boot menu - F8 for ASUS, F11 for Supermicro)
* follow the instruction:
* "try ubuntu or install ubuntu" - choose "install"
* select language - accept default
* "updates and other software" - accept default settings ("normal install")
* "installation type" - select "advanced features" and "experimental: use ZFS"
* accept partition choice
* "where are you?" - select "Vancouver" (PST time zone)
* "who are you?" - leave all fields blank, except "username" set to "wheel", "password" set to the root password. hostname will be set later after configuring the network
* installation runs in a few minutes, when finished, reboot
* login as user wheel
* answer annouying questions:
* "livepatch" - say "next"
* "help improve" - select "do not send", say "next"
* "privacy" - leave "location" as "off", say "next"
* "ready to go", say "done"
* right-click on the desktop, say "open in terminal", a shell will open
* say "sudo /bin/bash", enter the root password, you now have the root shell
* run nm-connection-editor to configure the network. use netmask 255.255.224.0, gateway 142.90.100.18, DNS 142.90.100.19, search path "triumf.ca"
* after network is up (can ping ladd00), continue with post-installation steps below

= Install instructions =

== prepare ==
<pre>
apt update
apt upgrade
</pre>

== install ssh ==
<pre>
apt install ssh
</pre>

== install git/scripts ==
<pre>
apt -y install git
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd scripts
git pull
</pre>

== configure hostname ==
<pre>
vi /etc/hostname
</pre>

== disable swap ==

ubuntu installer creates a 2 GB swap partition, not useful
on 32-64 GB machine, disable it:
<pre>
vi /etc/fstab ### comment out the "swap" line
</pre>

== maybe reboot ==

this is a good point to reboot the machine to boot
the latest kernel and to set the correct hostname

== install etckeeper ==

keep contents of /etc in a git repository:

<pre>
apt -y install etckeeper
</pre>

== set timezone ==
<pre>
timedatectl list-timezones | grep -i vancouver
timedatectl set-timezone America/Vancouver
</pre>

== install time synchronization ==
<pre>
apt -y install chrony
#echo server time1.triumf.ca iburst >> /etc/chrony/chrony.conf
#echo server time2.triumf.ca iburst >> /etc/chrony/chrony.conf
#echo server time3.triumf.ca iburst >> /etc/chrony/chrony.conf
cd ~/git/scripts
git pull
cd ~
cp ~/git/scripts/etc/triumf.sources /etc/chrony/sources.d/
systemctl disable systemd-timesyncd.service
systemctl stop systemd-timesyncd.service
systemctl disable ntp
systemctl stop ntp
systemctl enable chrony
systemctl restart chrony
chronyc sources
chronyc tracking
</pre>

NOTE1: if time1, time2, time3 are already listed in /etc/crony/chrony.conf, please remove them and restart chrony.

NOTE2: if time1, time2, time3 are not listed in "chronyc tracking" or if they are not selected by "chronyc tracking", check that /etc/crony/chrony.conf contains "sourcedir /etc/chrony/sources.d". old versions of this file may not have it.

NOTE3: read https://chrony-project.org/faq.html#_should_i_prefer_chrony_over_timesyncd_if_i_do_not_need_to_run_a_server

== reenable systemd-timesyncd ==

ONLY IF CHRONY DOES NOT WORK

To configure systemd-timesyncd, set "NTP=" in /etc/systemd/timesyncd.conf

<pre>
apt remove chrony
cat /etc/systemd/timesyncd.conf
systemctl enable systemd-timesyncd.service
systemctl restart systemd-timesyncd.service
systemctl status systemd-timesyncd.service
timedatectl status
timedatectl timesync-status
</pre>

== enable outgoing email (debian 11) ==

this is different from ubuntu 20. it uses /etc/mailname and it hardwires the hostname into main.cf.

== enable outgoing email ==

we have an unusual email configuration. outgoing email should work to deliver error messages, notices, etc. incoming email is disabled, we do not receive email for local users.

this causes problems with TRIUMF smtp server. if our message cannot be delivered (wrong email address or receipient computer is turned off), TRIUMF smtp server will generate a delivery failure notification email and try to send it to the "from" address of the failed message. but the "from" address does not receive any email, so another delivery failure notification email is generated and an attempt to deliver it. which again fails, rinse and repeat.

as solution, kray created a special rule, email from scrap.triumf.ca does not generate delivery failure notices. failed messages sit in the queue for 5 days, then they are deleted. (K.O. - confirmed with kray 3jan2024).

to make this work we use the msmtp MTA package.

<pre>
cd ~
apt -y remove postfix
apt -y purge postfix # remove old config files
apt -y install mailutils msmtp msmtp-mta # say "no" to apparmor support
apt -y install bsd-mailx
cd ~/git/scripts/etc
git pull
/bin/cp -fv aliases /etc/aliases
/bin/cp -fv msmtprc /etc/msmtprc
/bin/rm -vf ~root/.forward
/bin/rm -vf /etc/mailname
Mail root
Subject: test
test
^D
CC: <CR>
</pre>

== enable outgoing email (postfix) ==

THIS IS OBSOLETE!!!

* TRIUMF: use smtp.triumf.ca
* CERN: use cernmx.cern.ch

<pre>
apt install postfix ### select "satellite system", enter full hostname "xxx.triumf.ca", enter "smtp.triumf.ca"
apt install mailutils
dpkg-reconfigure postfix ### (if postfix already installed)
</pre>
<pre>
echo olchansk@triumf.ca lindner@triumf.ca bsmith@triumf.ca >> ~root/.forward
mailx root
test
^D
</pre>

== enable ping for all users (debian 11) ==

Without this tweak, Debian will report "operation not permitted" if a user tries to ping somewhere.

<pre>
echo 'net.ipv4.ping_group_range = 0 1000' > /etc/sysctl.d/99-ping.conf
</pre>

== disable apparmor ==

On NFS-Root network booted machines!

If "man man" returns "permission denied" and syslog reports apparmor "sendmsg DENIED" errors, disable apparmor. This is supposedly fixed in kernel 6.0 and later (to be confirmed), see https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/1784499

Disable apparmor, see https://ubuntu.com/server/docs/security-apparmor

This takes effect after a reboot.

<pre>
systemctl stop apparmor.service
systemctl disable apparmor.service
</pre>

== install missing packages ==

(apt eats terminal input, even the "yes |" trick does not quite work,
repeat the following commands until they report that everything
is installed)

<pre>
yes | apt -y install ssh tcsh ethtool ncat rsync strace net-tools sysstat smartmontools lm-sensors traceroute time minicom screen git lsof debsums tmux iptables
yes | apt -y install lsb-release
yes | apt -y install flex bison
yes | apt -y install neofetch
yes | apt -y install snmp snmp-mibs-downloader
yes | apt -y install git subversion g++ gfortran cmake doxygen
yes | apt -y install curl libcurl4 libcurl4-openssl-dev
yes | apt -y install mariadb-client ### mysql client
yes | apt -y install libz-dev libzstd-dev sqlite3 libsqlite3-dev unixodbc-dev
yes | apt -y install libssl-dev
yes | apt -y install emacs xemacs21 joe
yes | apt -y install gnuplot dos2unix
yes | apt -y install mutt bsd-mailx # email clients
yes | apt -y install liblz4-tool pbzip2
yes | apt -y install libc6-dev-i386 # otherwise no /usr/include/sys/types.h
yes | apt -y install libreadline-dev
yes | apt -y install ubuntu-mate-themes
yes | apt -y install libmotif-dev libxmu-dev
yes | apt -y install libusb-dev libusb-1.0-0-dev
yes | apt -y install i2c-tools libi2c-dev libi2c0
yes | apt -y install xfig gsfonts-x11 gsfonts-other # install fonts for xfig
yes | apt -y install libjson-perl
yes | apt -y install libgsl-dev # additional GNU Scientific Library
yes | apt -y install qt5-default # Qt development
yes | apt -y install python3-full python3-dev python3-dbg python3-pip ### for pyROOT
yes | apt -y install imagemagick imagemagick-common ckeditor # for elog
yes | apt -y install libjpeg-dev libjpeg-progs libjpeg-tools
yes | apt -y install linux-tools-common linux-tools-generic # cpupower frequency-info
yes | apt -y install rdesktop remmina remmina-plugin"*" # requested by POL
yes | apt -y install nlohmann-json3-dev # required to build MIDAS with ROOT 6.30 on Ubuntu-22
yes | apt -y install dpkg-dev cmake g++ gcc binutils libx11-dev libxpm-dev libxft-dev libxext-dev python3 libssl-dev libafterimage0 # from https://root.cern/install/dependencies/
yes | apt -y install apt install gfortran libpcre3-dev xlibmesa-glu-dev libglew-dev libftgl-dev libmysqlclient-dev libfftw3-dev libcfitsio-dev graphviz-dev libavahi-compat-libdnssd-dev libldap2-dev python3-dev python3-numpy libxml2-dev libkrb5-dev libgsl0-dev qtwebengine5-dev nlohmann-json3-dev # from https://root.cern/install/dependencies/
</pre>

Ubuntu LTS 20.04:
<pre>
yes | apt -y install linux-image-generic-hwe-20.04 linux-tools-virtual-hwe-20.04 # enable linux 5.11 series kernel
</pre>

Ubuntu LTS 22.04:
<pre>
apt -y install linux-generic-hwe-22.04 # enable linux 6.2.0 series kernel
</pre>

== disable swap (debian 11) ==

* on 64 GB RAM machines swap is not useful
* on machines booted from network (NFS-ROOT), swap does not work
* on machines running from flash (RPi, etc), flash is too slow for useful swap
* swap configured by linux installers invariably has wrong size and is not useful

<pre>
systemctl disable dphys-swapfile
systemctl stop dphys-swapfile
dphys-swapfile uninstall
</pre>

== configure DNS ==

<pre>
cd ~/git/scripts
git pull
mkdir /etc/systemd/resolved.conf.d
cp etc/resolved-triumf.conf /etc/systemd/resolved.conf.d/
systemctl restart systemd-resolved
resolvectl
#systemd-analyze cat-config systemd/resolved.conf
</pre>

== install ganglia ==

<pre>
apt -y install ganglia-monitor
cd ~root/git/scripts/ganglia
git pull
make install
./ganglia-all.perl
</pre>

== install gonodeinfo ==

* go to https://bitbucket.org/dd1/gonodeinfo follow instructions:
<pre>
yes | apt-get -y install golang
mkdir ~/git
cd ~/git
#git clone https://bitbucket.org/dd1/gonodeinfo.git
git clone https://daq00.triumf.ca/~olchansk/git/gonodeinfo.git
cd gonodeinfo
git pull
make
make install # install gonodeinfo agent
cd ~ # this is important
</pre>
* edit /etc/gonodeinfo.conf
* change "Description", "Location", "User" and "Administrator" as appropriate (or delete them)
* change "Servers" to read: Servers: daq00.triumf.ca:8601
* run "gonodeinfo -v"
* if error is "connection refused". go to the nodeinfo server to add this client to the access control list:
* on the gonodeinfo server: run /opt/gonodeinfo/gonodereceive.exe -a daq13
* try gonodeinfo again, there should be no error
* on the gonodeinfo server: run gonodereport, look at the web pages, the new machine should be listed now

== install fonts for EPICS ==

* apt install xfonts-100dpi xfonts-75dpi
* restart Xorg (i.e. "killall Xorg", this will log you out from the console)
* xlsfonts | grep -i helvetica ### should show fonts with different sizes, not just size 0 (scalable)

== install libz.so.1 for CentOS compatibility ==

KO - confirm which versions on quartus need this.

<pre>
yes | apt-get -y install zlib1g
yes | apt-get -y install zlib1g:i386 libc6:i386 libgcc1:i386 gcc-6-base:i386
</pre>

== install libpng12.so.0 for Quartus compatibility ==

(does not work anymore!!!)

<pre>
wget http://ftp.ca.debian.org/debian/pool/main/libp/libpng/libpng12-0_1.2.50-2+deb8u2_amd64.deb
dpkg --install libpng12-0_1.2.50-2+deb8u2_amd64.deb
</pre>

== install libpng12.so.0 for Quartus 13.0sp1 ==

<pre>
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0.50.0
/bin/cp -pv libpng12.so.0 libpng12.so.0.50.0 /lib/x86_64-linux-gnu/
</pre>

== install packages for Xilinx ==

ubuntu LTS 22.04 vivado 2020.1

<pre>
apt install autoconf libtool
apt install libtinfo5
apt install texinfo
apt install zlib1g:i386
</pre>

== install packages for building ROOT ==

<pre>
apt -y install libx11-dev libxpm-dev libxft-dev libxext-dev libpng-dev libjpeg-dev xlibmesa-glu-dev libxml2-dev libgsl-dev cmake
</pre>

== install 32-bit libraries for PHYSICA ==

these instructions are for running 32-bit physica executable built for SL6 on ubuntu LTS 20.04

install physica sources (cannot build, do not have g77)

<pre>
cd ~/packages
git clone https://bitbucket.org/ttriumfdaq/physica.git
</pre>

install 32-bit libraries using ubuntu package manager:

<pre>
apt install lib32z1 # libz.so
</pre>

copy 32-bit SL6 shared libraries to /lib32

<pre>
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libX11.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libgd.so.2 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libpng12.so.0 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libreadline.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libncurses.so.5 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libg2c.so.0 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libxcb.so.1 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libXpm.so.4 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libjpeg.so.62 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libfontconfig.so.1 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libfreetype.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libtinfo.so.5 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libXau.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libexpat.so.1 /lib32/
</pre>

ldd should report:

<pre>
trinatdaq:trinat> ldd /usr/local/physica/physica.exe
linux-gate.so.1 (0xf7fa2000)
libX11.so.6 => /lib32/libX11.so.6 (0xf7e43000)
libgd.so.2 => /lib32/libgd.so.2 (0xf7dfe000)
libpng12.so.0 => /lib32/libpng12.so.0 (0xf7dd6000)
libz.so.1 => /lib32/libz.so.1 (0xf7db8000)
libreadline.so.6 => /lib32/libreadline.so.6 (0xf7d7e000)
libncurses.so.5 => /lib32/libncurses.so.5 (0xf7d5b000)
libg2c.so.0 => /lib32/libg2c.so.0 (0xf7d3d000)
libm.so.6 => /lib32/libm.so.6 (0xf7c39000)
libgcc_s.so.1 => /lib32/libgcc_s.so.1 (0xf7c1a000)
libc.so.6 => /lib32/libc.so.6 (0xf7a2f000)
libxcb.so.1 => /lib32/libxcb.so.1 (0xf7a05000)
libdl.so.2 => /lib32/libdl.so.2 (0xf79ff000)
libXpm.so.4 => /lib32/libXpm.so.4 (0xf79ee000)
libjpeg.so.62 => /lib32/libjpeg.so.62 (0xf7997000)
libfontconfig.so.1 => /lib32/libfontconfig.so.1 (0xf7962000)
libfreetype.so.6 => /lib32/libfreetype.so.6 (0xf78c9000)
libtinfo.so.5 => /lib32/libtinfo.so.5 (0xf78b0000)
/lib/ld-linux.so.2 (0xf7fa4000)
libXau.so.6 => /lib32/libXau.so.6 (0xf78ad000)
libexpat.so.1 => /lib32/libexpat.so.1 (0xf7885000)
trinatdaq:trinat>
</pre>

set login environment:

<pre>
setenv TRIUMF_FONTS $HOME/packages/physica/fonts
setenv PHYSICA_DIR $HOME/packages/physica
alias physica $PHYSICA_DIR/physica-SL6-32
</pre>

test:

<pre>
cd ~/packages/physica
physica
@rangauss.pcm
</pre>

== install lightdm ==

unlike the default gdm login manager, lightdm shows the machine hostname and does not require an extra mouse click to swicth from screen saver to login mode.

<pre>
apt -y install lightdm
# select lightdm
</pre>

== install desktop environments ==

note: default display manager and default desktop are deficient, please do not skip this step.

note: if apt asks to choose the display manager, select "lightdm"

note: KO - I recommend the "MATE" desktop.

note: you will have to cut-and-paste this several times because "apt" eats commands, even with "-y" and even piped from "yes".

<pre>
# install MATE desktop
DEBIAN_FRONTEND=noninteractive apt -y install ubuntu-mate-core ubuntu-mate-desktop ubuntu-mate-themes
# install Cinnamon desktop
DEBIAN_FRONTEND=noninteractive apt -y install cinnamon
# install KDE desktop
DEBIAN_FRONTEND=noninteractive apt -y install kubuntu-desktop
# install Lxqt desktop
DEBIAN_FRONTEND=noninteractive apt -y install lxqt
# install Xfce4 desktop
DEBIAN_FRONTEND=noninteractive apt -y install xfce4
</pre>

== install ROOT ==

Please install ROOT per instructions at https://root.cern.ch.

NOTE1: The ROOT package available from Ubuntu repositories is severely out of date and cannot be used with MIDAS and ROOTANA. ### DO NOT DO THIS! apt-get install root-system

NOTE2: as of 2017-Jan-09, ROOT binary kits for Ubuntu do not work (use GCC 5 instead of GCC6), build from source instead.

== Install x2go ==

KO - is this still needed? does it cause any security problems?

x2go instructions, thanks to Art O.

<pre>
add-apt-repository ppa:x2go/stable
apt-get update
apt-get install x2goserver x2goserver-xsession
</pre>

== enable root login from ladd00/daq00 ==
<pre>
ssh localhost
CTRL-C
/bin/cp ~root/git/scripts/etc/authorized_keys ~root/.ssh/
</pre>

== disable ssh access from outside of TRIUMF ==

to stop ssh login spam, disable ssh access from outside of TRIUMF. this can be done by requesting a firewall block through the helpdesk or by local firewall rule:

<pre>
echo iptables -I INPUT ! -s 142.90.0.0/255.255.0.0 -p tcp --dport 22 -j REJECT >> /etc/rc.local
/etc/rc.local
</pre>

== install smart-status ==
<pre>
ln -s ~/git/scripts/smart-status/smart-status.perl ~root/
</pre>

== enable boot menu and boot messages ==

This will enable the grub menu (with a 10 sec timeout) and
replace black screen with exciting linux boot messages.

* emacs -nw /etc/default/grub
<pre>
GRUB_DEFAULT=0
#GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="vga=769 video=640x480"
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX=""
#GRUB_GFXMODE=640x480
</pre>
* update grub config:
<pre>
grub-mkconfig -o /boot/grub/grub.cfg
</pre>

== reboot ==

this completes installation of the base system.

following sections modify basic ubuntu to fix known problems and to enable special stuff.

= Enable automatic updates =

<pre>
apt install unattended-upgrades
cd ~/git/scripts
git pull
/bin/cp -v etc/99apt-conf-ko /etc/apt/apt.conf.d/
apt-config dump | grep Unattended
</pre>

Following is obsolete:

* emacs -nw /etc/apt/apt.conf.d/50unattended-upgrades
** uncomment in Allowed-Origins "-security" and "-updates"
** add in Allowed-Origins: "Google LLC:stable";
** uncomment/add: "Unattended-Upgrade::Mail "root";
* emacs -nw /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
</pre>
* test: unattended-upgrade --dry-run -v

NOTE: update-on-shutdown is disabled.

NOTE: there is no update-on-boot, but:

NOTE: if machine was off for a long time, the systemd update timer would have expired and it will fire soon after reboot, causing an automatic update run. this is unwanted, and there is no fix or workaround for it. K.O. June-2023.

= Fix bpool is full (obsolete) =

THIS IS CAUSED BY OBSOLETE PACKAGE zsys. PLEASE: apt remove zsys

!!! only if ROOT on ZFS !!!

There is an error in the zsys package that causes bpool to run out of space,
see [[#Ubuntu zsys]] for more details.

To fix:
<pre>
cd ~/git/scripts
git pull
cp etc/zsys.conf /etc/
zsysctl service reload
zsysctl service gc
zpool list bpool
zfs list bpool
df /boot
</pre>

= IPMI instructions =

IPMI is the board management hardware on Supermicro and other server motherboards. This includes hardware sensors - fan rotation speed, temperatures and power supply voltages.

<pre>
apt-get install ipmitool
systemctl enable ipmievd
systemctl restart ipmievd
</pre>

Run:
* ipmitool sel list ### event list
* ipmitool sel elist ### event list
* ipmitool sel clear ### clear event list (if it becomes full)
* ipmitool sensor ### report hardware sensors

= move /home/wheel =

note: this MUST be done if ZFS root and NIS/autofs with /home.

Default location of wheel's home directory will collide with autofs /home, it has to be moved,
for example to /wheel.

<pre>
# logout from the wheel user
# go to another computer
ssh root@daqubuntuxxx
zfs list | grep wheel ### identify zfs name wheel_xxxxxx
zfs set mountpoint=/wheel rpool/USERDATA/wheel_hm8fzh
emacs -nw /etc/passwd ### change wheel's home directory from /home/wheel to /wheel
su - wheel ### check that user wheel still works
</pre>

This will break wheel's ability to run snap programs, such as firefox, install chrome as listed below.

= enable NIS (ubuntu 22.04, debian 11) =

<pre>
apt -y install rpcbind nis
echo DAQ-NIS >> /etc/defaultdomain
echo ypserver daq00.triumf.ca >> /etc/yp.conf
systemctl enable ypbind.service
systemctl restart ypbind.service
systemctl status ypbind.service
ypwhich -m
</pre>

enable ypserv:

<pre>
sed -i s/NISSERVER=false/NISSERVER=slave/ /etc/default/nis
/usr/lib/yp/ypinit -s daq00
echo ypserver localhost >> /etc/yp.conf
sed -i "s/ypserver .*/ypserver localhost/" /etc/yp.conf
systemctl enable ypserv
systemctl restart ypserv
systemctl restart ypbind
</pre>

edit /etc/nsswitch.conf to read:

<pre>
# begin get data from nis
passwd: files nis
group: files nis
shadow: files nis
automount: files nis
netgroup: files nis
# end get data from nis
</pre>

enable hourly update of nis maps:

<pre>
mkdir ~root/git
cd ~root/git
git clone http://daq00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>

If this is a new machine, then on the master NIS node (daq00), add this new node to /etc/netgroup, and update NIS maps (cd /var/yp; make)

= enable NIS (ubuntu 20.04) =

* apt-get -y install portmap nis ### will ask for NIS domain (DAQ-NIS)
* dpkg-reconfigure nis ### reconfigure if already installed
* ypwhich -m
* edit /etc/default/nis
** set "NISSERVER=slave"
** Ubuntu LTS 20.04, check that "YPBINDARGS=" is blank, remove "-no-dbus" if it is there
* #edit /etc/yp.conf, comment-out everything, add "domain DAQ-NIS server localhost"
* edit /etc/yp.conf, comment-out everything, add "ypserver localhost"
* /usr/lib/yp/ypinit -s daq00
* systemctl enable nis
* systemctl restart nis
* ypwhich
* ypwhich -m
* ypcat -k passwd
* vi /etc/nsswitch.conf ### add the automount line, modify the passwd, group and shadow lines to read this:
<pre>
# begin get data from nis
passwd: files nis
group: files nis
shadow: files nis
automount: files nis
netgroup: files nis
# end get data from nis
</pre>
* enable hourly update of NIS maps
<pre>
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>
* ### NOT NEEDED sudo vi /etc/idmapd.conf ### add line: "Domain = triumf.ca"

= enable autofs =

<pre>
apt -y install autofs
systemctl enable autofs
systemctl restart autofs
ls -l /home/olchansk ### test autofs, check file owner is correct
</pre>

= enable NFS server =

<pre>
apt install nfs-kernel-server
#edit /etc/exports
systemctl enable nfs-server
systemctl restart nfs-server
</pre>

= NIS master =

notes for setting up the NIS master

== wheel user ==

"wheel" is the default administrative user. We do not want it's password exported to NIS (encrypted password hash is world visible) and we do not want it's home directory exported to NFS (~wheel/.ssh is world visible and potentially writable: anybody can change ~wheel/.ssh/authorized_keys).

* move wheel's home directory from /home/wheel to /wheel (see special section about this)
* change wheel's UID and GID from 1000 to a value below MINUID in /var/yp/Makefile

== coherent uids ==

we do not want system accounts defined in /etc/passwd of the NIS master
to be included in the NIS map "passwd". this causes trouble on NIS clients
where newly installed packages fail to create local system users because same
user already exists in NIS.

This is controlled by MINUID in /var/yp/Makefile.

Historical TRIUMF uids start from around 200, but several clusters do not have any historic TRIUMF uids below 500 and MINUID is set to:
* DAQ-NIS: MINUID=200
* ISAC-NIS: MINUID=500
* TITAN-NIS: MINUID=500
* MUSR-NIS: MINUID=500
* TIG-NIS: MINUID=500 (100 on SL6 mother8pi)

Ubuntu 20 has two programs to create users:
* adduser - creates new users with UID 1000 and up as specified in /etc/adduser.conf. No problems here.
* adduser --system - creates new system users with UID 100 and up as specified in /etc/adduser.conf. No problems here.
* useradd - creates new users with UID 1000 and up as specified in /etc/login.defs. No problems here.
* useradd --system - creates new system users with UID 999 and down (read "man useradd", section at the end about SYS_UID_MAX). This collides with NIS MINUID, these system users will be included in the NIS map and cause trouble.

This problem cannot be fixed, SYS_UID_MIN, SYS_UID_MAX and UID_MIN in /etc/login.defs do not seem
to have any effect on UIDs chosen by "useradd --system". (tested on Ubuntu LTS 20.04).

So far only these system accounts seem to be affected by this:
* systemd-coredump
* ganglia

To fix:
* run "sort -r -n -t: -k3 /etc/passwd" to identify the last unused system user uid (range 100..200)
* run "sort -r -n -t: -k3 /etc/group" to identify the last unused system user gid (range 100.200)
* systemd-coredump: manually change UID and GID (package systemd-coredump is usually not installed)
* ganglia: same thing, then change ownership on all ganglia files.

Also read systemd author's opinion on system vs user UIDs:
https://github.com/systemd/systemd/issues/4850#issuecomment-265698275

= Fix systemd-logind NIS breakage =

!!! THIS IS NOT NEEDED FOR UBUNTU LTS 20.04 !!!

there is a delay in ssh logins for normal users. "ssh -v" shows the delay is after "pledge...". this
fix removes the delay.

systemd developers think that we should not use NIS and made sure there are
problems if we do. To give them credit, they do offer a workaround. Read this:
https://github.com/poettering/systemd/commit/695fe4078f0df6564a1be1c4a6a9e8a640d23b67

<pre>
mkdir /etc/systemd/system/systemd-logind.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-logind.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-logind.service
</pre>

= Fix systemd-udevd NIS breakage =

see same problem as above with udev getting stuck. ubuntu lts 20.04.

<pre>
mkdir /etc/systemd/system/systemd-udevd.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-udevd.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-udevd.service
</pre>

= Configure USB device permissions =

Configure USB device permissions for user access to USB-serial devices, Altera USB Blaster, etc.

* create file /etc/udev/rules.d/99-usb-chmod.rules with this contents:

<pre>
emacs -nw /etc/udev/rules.d/99-usb-chmod.rules
ACTION=="add", SUBSYSTEM=="usbmisc", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /dev/%c"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /proc/%c"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVICE}"
ACTION=="add", ENV{PHYSDEVBUS}=="usb-serial", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVPATH}=="/class/tty/ttyS*", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyUSB*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyACM*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyS*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", DEVPATH=="*video*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
</pre>

* reload udev rules: udevadm control --reload-rules
* apply new permissions: udevadm trigger --action=add
* watch udev activity: udevadm monitor -p

= Configure lightdm display manager =

* enable it
<pre>
echo lightdm | dpkg-reconfigure -fteletype lightdm
systemctl disable gdm
systemctl disable sddm
systemctl enable lightdm
</pre>

* make the MATE desktop as default
<pre>
cd ~root/git/scripts/
git pull
/bin/cp -v etc/lightdm_default_mate.conf /etc/lightdm/lightdm.conf.d/
</pre>

* enable login by NIS users
<pre>
/bin/cp -v etc/lightdm_enable_nis_login.conf /etc/lightdm/lightdm.conf.d/
</pre>

* restart lightdm
<pre>
systemctl stop gdm
systemctl restart lightdm
</pre>

= Install libpng12.so.0 =

Quartus 16 needs libpng12:

<pre>
wget http://mirrors.kernel.org/ubuntu/pool/main/libp/libpng/libpng12-0_1.2.54-1ubuntu1_amd64.deb
dpkg --install libpng12-0_1.2.54-1ubuntu1_amd64.deb
</pre>

= Install google-chrome =

<pre>
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
dpkg -i google-chrome-stable_current_amd64.deb
</pre>

confirm autoupdate is enabled, observe dl.google.com is present in the list of repositories:
<pre>
apt update
...
Get:5 https://dl.google.com/linux/chrome/deb stable/main amd64 Packages [1,094 B]
...
</pre>

FOLLOWING IS OBSOLETE:

Instructions from here:
https://www.ubuntuupdates.org/ppa/google_chrome?dist=stable

<pre>
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-tmp.list'
apt update
apt install google-chrome-stable
/bin/rm -f /etc/apt/sources.list.d/google-tmp.list
</pre>

= Install amanda client =

ONLY ONE MACHINES THAT HOST HOME DIRECTORIES

* apt install amanda-client
* edit /etc/amandahosts
<pre>
amanda.triumf.ca amanda amdump
</pre>
* check permissions on /etc/amandahosts:
<pre>
root@daq00:/var/log/amanda# ls -l /etc/amandahosts
-rw------- 1 backup backup 49 Jan 27 10:48 /etc/amandahosts
</pre>
* fix if needed: chown backup.backup /etc/amandahosts; chmod a= /etc/amandahosts; chmod u=wr /etc/amandahosts
* edit /etc/amanda-security.conf, add this line:
<pre>
runtar:gnutar_path=/usr/bin/tar
</pre>

On the amanda machine:

* in amanda disklist, use dump type "bsdtcp-comp-user-tar"
* su - amanda and run amcheck -c daily daq00
<pre>
-bash-4.1$ amcheck -c daily daq00

Amanda Backup Client Hosts Check
--------------------------------
Client check: 1 host checked in 0.092 seconds. 0 problems found.

(brought to you by Amanda 3.3.7p1.git.685ff76d)
</pre>

= Enable rc.local =

For reasons unknown, Ubuntu LTS 20.04 does not enable /etc/rc.local. Do this:

<pre>
cd ~/git/scripts
git pull
cp -n -v etc/rc.local /etc/
chmod a+rx /etc/rc.local
cp etc/rc-local.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable rc-local
systemctl start rc-local
systemctl status rc-local
</pre>

= Remove unwanted packages =

<pre>
apt remove zsys # broken, do not use
apt remove sddm # login manager
apt remove avahi-daemon avahi-autoipd # not sure what it does, observed using 100% CPU
apt remove modemmanager # probes all serial ports to see if it's a modem
</pre>

= Disable unwanted services =

<pre>
systemctl disable mpd
systemctl disable snapd
systemctl disable ModemManager
systemctl --global mask tracker-extract-3.service
systemctl --global mask tracker-miner-fs-3.service
systemctl daemon-reload
</pre>

= Disable sleep and suspend =

note: we see some computers randomly shutdown or go to sleep, log files indicates the "sleep" or "suspend" button was pushed by user, but no such buttons actually exist. this is the fix for this:

<pre>
systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target systemd-suspend.service systemd-hybrid-sleep.service
</pre>

= Enable crontab @reboot for MIDAS =

startup scripts have a bug - cron @reboot entries for normal users can run before autofs is ready, so if the home directory is on autofs/NFS, it cannot be accessed and the cron job fails. If MIDAS is supposed to be started by cron @reboot, it will not start (there *will* be an error message in /var/log/cron).

<pre>
mkdir /etc/systemd/system/cron.service.d
echo -e "[Unit]\nAfter=ypbind.service autofs.service\n" > /etc/systemd/system/cron.service.d/local.conf
systemctl daemon-reload
systemctl cat cron.service
</pre>

Explore the systemd dependency tree using "systemctl list-dependencies" maybe with "--all".

Visualize the exact boot sequence from previous boot: "systemd-analyze plot > xxx.svg", look at the svg file using a web browser.

Crontab entry to start midas: (install in the midas user crontab, not root crontab)

<pre>
su - midasuser
crontab -l
#@reboot /bin/bash -l -c "/home/trinat/bin/start-daq-applications"
#@reboot /bin/tcsh -c "/home/trinat/bin/start-daq-applications"
</pre>

= Install apache httpd proxy for midas and elog =

This will configure the HTTPS/SSL certificate using "certbot" and "letsencrypt" and configure an HTTPS web server using apache2.

First, configure apache2:

* execute these commands:
<pre>
apt -y install apache2
cd /etc/apache2
</pre>
* create new file conf-available/ssl-daq14.conf # use actual hostname instead of daq14
<pre>
SSLSessionCache shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout 300
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
SSLCryptoDevice builtin
</pre>
* create new file sites-available/daq14-ssl.conf # use actual hostname instead of daq14
<pre>
<IfModule mod_ssl.c>
<VirtualHost *:443>
ServerName daq14.triumf.ca
DocumentRoot /var/www/html
ErrorLog /var/log/apache2/daq14.log
SSLEngine on
# note SSLProtocol, SSLCipherSuite and some other settings are overwritten by /etc/letsencrypt/options-ssl-apache.conf
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SEED:!IDEA:!RC4
## use port specified in elogd.cfg
#ProxyPass /elog/ http://localhost:8082/ retry=1
## use mhttpd port
#ProxyPass / http://localhost:8080/ retry=1
Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"
<Location />
SSLRequireSSL
AuthType Basic
AuthName "DAQ password protected site"
Require valid-user
# create password file: touch /etc/apache2/htpasswd
# to add new user or change password: htpasswd /etc/apache2/htpasswd username
AuthUserFile /etc/apache2/htpasswd
</Location>
</VirtualHost>
</IfModule>
</pre>
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* enable ssl module
* enable new configurations
<pre>
a2enmod ssl
a2enmod headers
a2enmod proxy
a2enmod proxy_http
a2enconf ssl-daq14
a2ensite daq14-ssl
</pre>
* disable default ssl sites
<pre>
a2dissite 000-default-le-ssl
a2dissite 000-default
ls -l /etc/apache2/sites-enabled/ ### should show only daq14-ssl.conf
</pre>
* check that there are no syntax problems
<pre>
apache2ctl configtest
</pre>
* enable and start apache2:
<pre>
systemctl enable apache2
systemctl restart apache2
systemctl status apache2
</pre>
* apache2 may fail to start, look in /var/log/apache2/error.log and /var/log/apache2/daq14.log
* if it says "Failed to configure ... certificate", proceed to the step for setting certbot.
* try to access https://daq14.triumf.ca
** you should see a complaint about self-signed certificate
** you should see a request for password (do not login yet)
** if you get "connection refused", HTTPS port 443 may need to be enabled in the local firewall, look at documentation for ufw.
Second, configure certbot:

(Note: as of 2018-01-18 certbot requires use of http port 80 to get the initial https certificate,
renewal can continue to use the https port 443)

(Note: as of 2019-01-?? certbot requires use of port 80 for renewals)

(Note: unsurprisingly, this requires outside access to connect with letsencrypt, so won't work if PC is only accessible from on-site network)

* check that port 80 is not used by anything:
* netstat -an | grep LISTEN | grep ^tcp | grep 80
* lsof -P | grep -i tcp | grep LISTEN | grep 80
* if lsof reports that apache2 is listening on port 80, follow the apache2 instructions above (remove "listen 80" from apache2.conf

* install certbot (if necessary open tcp port 80 in the firewall, see documentation for ufw):
<pre>
apt install certbot python3-certbot-apache
certbot certonly --standalone --installer apache
</pre>
* then answer questions:
* "activate HTTPS for daq14.triumf.ca" - say ok
* "enter email address" - enter your own email address
* "please read terms..." - read the terms and say "agree"
* it will take a few moments...
* "congratulations..." - say ok.
<pre>
certbot install --apache --cert-name daq14.triumf.ca
</pre>
* then answer questions:
* "choose redirect..." - say "1" (no redirect)
* look inside /etc/apache2/sites-enabled/daq14-ssl.conf to see that SSLCertificateFile & co point to certbot certificates in
/etc/letsencrypt/live/daq14.triumf.ca/
* to check current renewal and to update the certbot config file in /etc/letsencrypt/renewal, run this:
<pre>
certbot renew --standalone --installer apache --force-renewal
</pre>

NOTE: this certificate will expire in 3 months, automatic renewal should work with current version of certbot

Third, activate password protection:

* as shown in the config file above, create password file and initial user: (replace "midas" with specific username)
<pre>
touch /etc/apache2/htpasswd
htpasswd /etc/apache2/htpasswd midas
</pre>

* restart apache2
<pre>
systemctl restart apache2
systemctl status apache2
</pre>

From here:
* enable proxy for MIDAS mhttpd - uncomment redirect in the config file above
* enable proxy for ELOG - ditto
<pre>
a2enmod proxy
a2enmod proxy_http
apache2ctl configtest
systemctl restart apache2
</pre>

* try accessing MIDAS https://daq14.triumf.ca/ (make sure mhttpd is running)
* if it's not working, check odb setting FIXME!
* try accessing ELog https://daq14.triumf.ca/elog/ (make sure elogd is running)
* if it's not working, check elogd.cfg file and make sure
<pre>
SSL = 0
</pre>

NOTE: if certbot fails with errors about 'module' object has no attribute 'pyopenssl',
try this: pip install requests==2.6.0

= Enable elog PDF preview =

see https://stackoverflow.com/questions/52998331/imagemagick-security-policy-pdf-blocking-conversion

* xemacs -nw /etc/ImageMagick-6/policy.xml
* remove this section at the end:
<pre>

<policy domain="coder" rights="none" pattern="PS" />
<policy domain="coder" rights="none" pattern="PS2" />
<policy domain="coder" rights="none" pattern="PS3" />
<policy domain="coder" rights="none" pattern="EPS" />
<policy domain="coder" rights="none" pattern="PDF" />
<policy domain="coder" rights="none" pattern="XPS" />
</pre>

= Install Jupyter notebook =

<pre>
From https://jupyter.org/install
apt install python3-pip
pip install jupyterlab
pip install notebook
~/.local/bin/jupyter notebook
watch the http://localhost:8888 URL that it printed
say "no" to offer to start firefox (it will not work!)
URL is: http://localhost:8888/tree?token=xxx
from the machine where you are running the web browser (i.e. google-chrome), run (replace trinat@trinatdaq with the username and machine name where you started jupyter)
open a new shell and run: ssh -v trinat@trinatdaq -L 8888:localhost:8888
in the web browser, open http://localhost:8888
this gives us the login page
in the password or token entry field, put the token from the "tree?token=xxx" above (printed by jupyter on startup)
push button "login"
jupyter page should open with the list of files in the trinat home directory
congratulate Brian with full success
</pre>

= Install ZFS quota report =

If there are any ZFS volumes, install script to report disk and quota usage

<pre>
cd ~/git/scripts/quotareport
git pull
mkdir /var/www/html/zfsquotareport
cp -pv ~/git/scripts/quotareport/sorttable.js /var/www/html/zfsquotareport/
ln -s $PWD/zfsquotareport.perl /etc/cron.daily/
touch /etc/crontab
</pre>

If httpd is configured to redirect "/" to MIDAS mhttpd:
* add following to /etc/apache2/sites-enabled/xxx-ssl.conf in front of "ProxyPass / ..."
* run "systemctl reload apache2"
<pre>
## do not proxy zfs quota report directory
ProxyPass /zfsquotareport/ !
</pre>

= Install PHP =

* apt install php libapache2-mod-php
* systemctl restart apache2
* create /var/www/html/info.php
<pre>
<?php

phpinfo();
</pre>
* open https://daq00.triumf.ca/info.php

= Configure TRIUMF printers =

<pre>
systemctl stop cups
systemctl disable cups
echo "ServerName printers.triumf.ca" > /etc/cups/client.conf
lpstat -a
</pre>

= Enable core dumps =

By default, Ubuntu LTS 20.04 installs the apport package
which disabled core dumps from user applications. (google it up!).
It is not meant to do this and documentation claims that
it is not installed and not enabled by default. Oh, well...

<pre>
apt remove apport
apt autoremove ### will remove apport-symptoms and a few other packages
</pre>

After this, core dumps are written to file "core" in the current directory.
See /proc/sys/kernel/core_pattern and /proc/sys/kernel/core_uses_pid.

Enable core dump file names to include process id, add following to /etc/rc.local

<pre>
echo 1 > /proc/sys/kernel/core_uses_pid
</pre>

= Enable debugger =

By default, Ubuntu LTS 20.04 does not permit debugger to attach and debug
already running programs. To enable it, add following to /etc/rc.local

<pre>
echo 0 > /proc/sys/kernel/yama/ptrace_scope
</pre>

= Disable Ubuntu Pro nag =

If "apt upgrade" requests Ubuntu Pro or esm-apps, disable the nag:
<pre>
/bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf
</pre>

= Update packages =

* apt-get update # update package list
* apt-get dist-upgrade # install updated packages and update "kept back" packages
* apt-get autoremove # remove packages that apt thinks should be removed

= Finish installation =

Congratulations. There is nothing more to do!

* reboot
<pre>
shutdown -r now
</pre>

= Install ZFS =

!!! after installing all the packages, after updating the system, after updating the linux kernel, after rebooting into latest kernel !!!

<pre>
apt-get install zfsutils-linux
</pre>

Follow generic ZFS instructions: [[ZFS]]

= Update to new version of Ubuntu =

<pre>
vi /etc/update-manager/release-upgrades # set "Prompt=normal"
do-release-upgrade
</pre>

Update Ubuntu LTS 20.04 to LTS 22.04:

<pre>
apt remove zsys
</pre>

== daqubuntu ==

<pre>
# reboot to clear out all updates
# vi /etc/update-manager/release-upgrades # set "Prompt=normal"
# do-release-upgrade -c
Checking for a new Ubuntu release
New release '22.04 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
# do-release-upgrade
...
say yes...
...
login.defs, say "Y" (erase local changes, use packaged version)
/etc/systemd/resolved.conf, say "Y" (same as above)
firefox snap, say yes
unable to reach snap store, say "skip"
/etc/gmond.conf, say "Y"
/var/yp/Makefile, say "install the package maintainer's version"
/etc/ypserv.conf, same thing
/etc/ypserv.securenets, same thing
/etc/default/nis, same thing
/etc/speech-dispatcher/modules/mary-generic.conf, same thing
/etc/apt/apt.conf.d/50unattended-upgrades, same thing
...
278 packages are going to be removed, say yes
...
restart required, say yes
...
no ping... yes ping...
...
ssh daqubuntu, ok
apt update, fail, DNS does not work, "host security.ubuntu.com" does not resolve.
fix resolver per https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Disable_NetworkManager
apt update, apt upgrade now works, 0 packages to update
NIS does not work.
</pre>

== midm9a ==

<pre>
login.defs
firefox snap
gmond.conf
ypserv
/etc/default/nis
unattended-upgrades
amanda-security.conf
remove obsolete (no)
reboot
configure dns
reenable nis
</pre>

== daq17 ==

<pre>
firefox snap
imagemagick policy.xml
gmond.conf
chrony.conf
/var/yp/Makefile
ypserv.conf
ypserv.securenets
/etc/default/nis
50unattended-upgrades
</pre>

== daq00 ==

per https://serverpilot.io/docs/how-to-upgrade-ubuntu-20.04-to-22.04/

<pre>
do-release-upgrade -f DistUpgradeViewNonInteractive
</pre>

if it exists "too soon" without doing anything, run it without "-f xxx", most likely it does not like something about this machine. in case of daq00 it did not like how the EFI partitions were mounted. after fixing it, non-interactive upgrade was successful.

== isdaq08 ==

* prepare
<pre>
cd ~/git/scripts
git pull
cd ~
apt -y install debsums
</pre>
* check for modified config files that make upgrade unhappy, deal with all files reported by debsums.
<pre>
root@isdaq08:~# debsums -ce
/etc/ganglia/gmond.conf
/etc/yp.conf
/etc/apt/apt.conf.d/10periodic
root@isdaq08:~#
</pre>
* restore original /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
</pre>
* apt remove ganglia-monitor
* apt remove nis
* "debsums -ce" is now empty

Run the upgrade:

* do-release-upgrade -f DistUpgradeViewNonInteractive

Post upgrade:

* configure DNS
* apt -y install linux-generic-hwe-22.04
* /bin/cp -v ~/git/scripts/etc/99apt-conf-ko /etc/apt/apt.conf.d/ # restore nightly updates
* /bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf # remove the ubuntu-pro nag
* install missing packages
* restore ganglia
* restore nis
* check zpool status, may need zpool upgrade
* reboot

= Upgrade to new version of Debian =

https://www.debian.org/releases/bookworm/amd64/release-notes/ch-upgrading.en.html

== 32-bit VME processor Debian 11 to 12 ==

* cd git/scripts; git pull; cd ~
* apt update
* apt upgrade
* edit /etc/apt/sources.list
<pre>
deb http://deb.debian.org/debian/ bookworm main
#deb http://deb.debian.org/debian/ bullseye main
#deb-src http://deb.debian.org/debian/ bullseye main
</pre>
* apt update
* apt upgrade --without-new-pkgs
* apt full-upgrade
* apt list '~c'; apt purge '~c' # purge left-over config files [residual-config]
* reboot

= Ubuntu package manager =

* apt-get install xxx # install package xxx
* apt-get update
* apt-get upgrade
* apt-get dist-upgrade
* apt-get autoremove # remove automatically installed packages required by a removed package
* apt-get remove xxx # remove package xxx
* apt-cache search . # list all available packages
* apt-cache show "." | grep ^Package # list al available packages
* apt-cache madison root-system # show all available versions of package root-system
* apt list # list all installed packages
* dpkg --listfiles libpng16-16 # list all files from this package
* apt list --installed # list all installed packages
* dpkg -S /bin/bash # what package provides this file?
* dpkg -L bash # what files provided by this package?
* debsums -ce # show modified config files
* apt-config dump # show apt configuration

= Ubuntu zsys =

NOTE: DO NOT USE ZSYS, see https://github.com/ubuntu/zsys/issues/218 and https://github.com/ubuntu/zsys/issues/230

* manual removal of old snapshots
<pre>
zsysctl show
zsysctl state remove xy69ye -s
zsysctl state remove xy69ye
zsysctl state remove xy69ye -u wheel
</pre>
* apt remove zsys

NOTE: old zsys snapshots must be cleaned manually, "zsysctl state remove xxx --system" is broken and does not remove user data snapshots

* manages system snapshots
* documentation: https://github.com/ubuntu/zsys
* documentation: (go to next article via link "newer" at the bottom) https://didrocks.fr/2020/05/21/zfs-focus-on-ubuntu-20.04-lts-whats-new/
* ubuntu 20.04 bug, too many snapshots cause /boot to become full and updates fail. https://github.com/ubuntu/zsys/issues/155
* solution: use custom /etc/zsys.conf, limit number of snapshots to 10, see trinatdaq:/etc/zsys.conf
* zsys commands:
<pre>
update-grub # list of all snapshots, errors if some snapshots are broken
zsysctl state remove lnc0k7 --system # remove snapshot
xemacs -nw /etc/zsys.conf; zsysctl service reload; zsysctl service gc # cause gc to run with new settings in zsys.conf
zfs list -r -t snapshot -o name,used,referenced,creation bpool/BOOT # list snapshots
zsysctl show # show snapshots
</pre>

= Ubuntu cloning =

to clone a ubuntu image:

<pre>
cd /nfsroot/lxcpet
emacs -nw etc/hostname ### change hostname
emacs -nw etc/mailname ### change hostname (debian 11)
emacs -nw etc/defaultdomain ### change the NIS domainname
emacs -nw etc/yp.conf ### change the NIS server
cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
emacs -nw root/.ssh/authorized_keys ### update root ssh keys
</pre>

= Ubuntu boot loader =

== boot from ZFS ==

* use UEFI boot with syslinux, see here: https://daq.triumf.ca/DaqWiki/index.php/SLinstall#Configure_UEFI_boot
* apt install zfs-initramfs
* update-initramfs -v -u
* ZFS structure:
<pre>
root@daq00:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 147G 1.62T 96K /
rpool/ROOT 17.8G 1.62T 96K none
rpool/ROOT/ubuntu_00aaaa 17.8G 1.62T 6.22G /
</pre>
* copy OS image to rpool/ROOT/ubuntu_00aaaa
* zfs set mountpoint=/ rpool
* zfs set mountpoint=none rpool/ROOT
* zfs set mountpoint=/ rpool/ROOT/ubuntu_00aaaa
* zfs get all | grep mountpoint
<pre>
rpool mountpoint / local
rpool/ROOT mountpoint none local
rpool/ROOT/ubuntu_00aaaa mountpoint / local
</pre>
* in linux kernel command line (syslinux.cfg), set "root=" to "root=ZFS=rpool/ROOT/ubuntu_00aaaa"

== boot from ZFS mirror ==

=== setup the EFI partitions ===

* assuming /dev/sdb is already setup for EFI boot, setup /dev/sda the same way:
* partition the second boot disk same as first boot disk:
<pre>
root@grsnis01:~# gdisk -l /dev/sdb
Found valid GPT with protective MBR; using GPT.
Number Start (sector) End (sector) Size Code Name
1 2048 1050623 512.0 MiB EF00 EFI system partition
2 1050624 3907029134 1.8 TiB 8300 Linux filesystem
root@grsnis01:~#
</pre>
* mkfs.msdos /dev/sdX1
* create mount points
<pre>
mkdir /boot/efi-sda
mkdir /boot/efi-sdb
</pre>
* add to /etc/fstab
<pre>
/dev/sda1 /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
/dev/sdb1 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
</pre>
* mount -a
* df | grep boot
<pre>
root@grsnis01:~# df | grep boot
/dev/sdb1 523248 98100 425148 19% /boot/efi-sdb
/dev/sda1 523248 4 523244 1% /boot/efi-sda
</pre>
* copy boot files to new boot disk
* cd /boot/efi-sdX; rsync -av . /boot/efi-sdY
* set BIOS to boot from "UEFI Hard drive", disable legacy boot (except for booting from USB key in legacy mode)
* if using UEFI boot syslinux per these instructions, linux kernel update has to be done manually:
* run ~/git/scripts/etc/update_efi_mirror.perl, follow instructions that it prints.

=== setup zfs partitions ===

use partitions compatible with Ubuntu "install on ZFS"

* gdisk "o" to create new GPT partition table
* gdisk "n" +512M ef00 to create EFI partition
* gdisk "n" +2G 8200 to create linux swap partition (not used)
* gdisk "n" +2G BE00 to create ZFS bpool partition
* gdisk "n" xxx BF00 create ZFS rpool partition

<pre>
# gdisk -l /dev/sda
Number Start (sector) End (sector) Size Code Name
1 2048 1050623 512.0 MiB EF00 EFI System Partition
2 1050624 5244927 2.0 GiB 8200
3 5244928 9439231 2.0 GiB BE00
4 9439232 234441614 107.3 GiB BF00
root@midm9a:~#
</pre>

=== setup zfs mirror ===

* see here: https://daq.triumf.ca/DaqWiki/index.php/ZFS#Convert_pool_from_single_to_mirror
<pre>
root@grsnis01:~# ls -l /dev/disk/by-id/ata*part2
lrwxrwxrwx 1 root root 10 Feb 19 16:47 /dev/disk/by-id/ata-WDC_WDS200T2B0A-00SM50_205007801081-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Feb 19 16:47 /dev/disk/by-id/ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 -> ../../sdb2

root@grsnis01:~# zpool status
pool: rpool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 ONLINE 0 0 0

errors: No known data errors

root@grsnis01:~# zpool attach rpool ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 /dev/disk/by-id/ata-WDC_WDS200T2B0A-00SM50_205007801081-part2

root@grsnis01:~# zpool status
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Feb 19 16:54:39 2021
12.6G scanned at 3.16G/s, 1.02G issued at 262M/s, 12.6G total
1.02G resilvered, 8.09% done, 0 days 00:00:45 to go
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801081-part2 ONLINE 0 0 0 (resilvering)

errors: No known data errors
</pre>
* wait
<pre>
root@grsnis01:~# zpool status
pool: rpool
state: ONLINE
scan: resilvered 12.7G in 0 days 00:00:40 with 0 errors on Fri Feb 19 16:55:19 2021
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801081-part2 ONLINE 0 0 0

errors: No known data errors
</pre>

== maintenance commands ==

* update-initramfs -v -u
* grub-install /dev/sda

= Convert from single to dual mirrored ZFS SSD =

Assuming Ubuntu LTS 22.04 with "instal on ZFS" option, we will
add a second SSD, configure ZFS to use both SSDs in mirrored
configuration and setup grub to boot from either SSD. This
is intended to create a full redundant system where failure
of either SSD does not break the system.

* identify first SSD
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* connect second SSD of identical size
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
/dev/sdb WD Blue SA510 2.5 250GB 22243Z803852 25 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* if second SSD is not autodetected, reboot
* Clone partition table automatically
If both SSDs are identical size, use this simpler method of duplicating the partition table:
<pre>
root@midm9b:~# sfdisk -d /dev/sda > part_table
root@midm9b:~# grep -v ^label-id part_table | sed -e 's/, *uuid=[0-9A-F-]*//' | sfdisk /dev/sdb
</pre>
The grep and sed in the second command are there to prevent disk ID and partition IDs from being cloned. Alternatively the part_table file can be edited manually to remove the label-id line and the uuid entries from the individual partitions.

* Clone partition table manually (e.g. for different size disks)
* list partition table of first SSD:
<pre>
root@midm9b:~# fdisk -l /dev/sda
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>
* create identical partitions on second SSD, use sector numbers from above.
<pre>
root@midm9b:~# gdisk /dev/sdb
GPT fdisk (gdisk) version 1.0.8

Partition table scan:
MBR: not present
BSD: not present
APM: not present
GPT: not present

Creating new GPT entries in memory.

Command (? for help): n
Partition number (1-128, default 1):
First sector (34-488397134, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-488397134, default = 488397134) or {+-}size{KMGTP}: 1050623
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): ef00
Changed type of partition to 'EFI system partition'

Command (? for help): n
Partition number (2-128, default 2):
First sector (34-488397134, default = 1050624) or {+-}size{KMGTP}:
Last sector (1050624-488397134, default = 488397134) or {+-}size{KMGTP}: 5244927
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 8200
Changed type of partition to 'Linux swap'

Command (? for help): n
Partition number (3-128, default 3):
First sector (34-488397134, default = 5244928) or {+-}size{KMGTP}:
Last sector (5244928-488397134, default = 488397134) or {+-}size{KMGTP}: 9439231
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): be00
Changed type of partition to 'Solaris boot'

Command (? for help): n
Partition number (4-128, default 4):
First sector (34-488397134, default = 9439232) or {+-}size{KMGTP}:
Last sector (9439232-488397134, default = 488397134) or {+-}size{KMGTP}:
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): bf00
Changed type of partition to 'Solaris root'

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sdb.
The operation has completed successfully.
root@midm9b:~# fdisk -l /dev/sda /dev/sdb
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root

Disk /dev/sdb: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: EB251739-30C6-422F-A505-5887B5A0B603

Device Start End Sectors Size Type
/dev/sdb1 2048 1050623 1048576 512M EFI System
/dev/sdb2 1050624 5244927 4194304 2G Linux swap
/dev/sdb3 5244928 9439231 4194304 2G Solaris boot
/dev/sdb4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>
* identify second SSD partitions
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part3
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part3 -> ../../sda3
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 -> ../../sdb3
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
</pre>
* convert bpool from single disk to mirrored disk:
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~# zpool attach bpool 99e03dc0-7d4d-f24b-8fa1-f042b9f135db /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3
root@midm9b:~# zpool status bpool
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors
</pre>
* convert rpool
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
root@midm9b:~# zpool attach rpool f6fd54f8-3af7-b943-ae3d-a4e480537fb9 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4
root@midm9b:~# zpool status rpool
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Jan 20 19:40:45 2023
5.83G scanned at 664M/s, 2.92M issued at 332K/s, 9.11G total
0B resilvered, 0.03% done, no estimated completion time
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~#
</pre>
* wait for resilver to complete
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
scan: resilvered 9.65G in 00:00:36 with 0 errors on Fri Jan 20 19:41:21 2023
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
</pre>
* enable booting from second SSD: (instead of /dev/sda1, /dev/sdb1, use UUID=xxx)
<pre>
root@midm9b:~# mkfs.msdos /dev/sdb1
root@midm9b:~# mkdir /boot/efi-sda
root@midm9b:~# mkdir /boot/efi-sdb
root@midm9b:~# echo "/dev/sda1 /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1" >> /etc/fstab
root@midm9b:~# echo "/dev/sdb1 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1" >> /etc/fstab
root@midm9b:~# mount -a
root@midm9b:~# df -kl
Filesystem 1K-blocks Used Available Use% Mounted on
...
/dev/sda1 523244 13720 509524 3% /boot/efi
/dev/sdb1 523244 4 523240 1% /boot/efi-sdb
...
root@midm9b:~# rsync -av /boot/efi/ /boot/efi-sdb/
sending incremental file list
EFI/
...
root@midm9b:~# ls -l /boot/efi-sda
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~# ls -l /boot/efi-sdb
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~#
</pre>
* setup script to update grub on second SSD, it must be run manually after every kernel update
<pre>
root@midm9b:~# ln -s ~/git/scripts/etc/update_efi_grub.perl ~/
root@midm9b:~# ~/update_efi_grub.perl -u
EFI dir: /boot/efi-sda
/boot/efi-sda: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sda/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sda: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sda/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
EFI dir: /boot/efi-sdb
/boot/efi-sdb: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sdb/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sdb: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sdb/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
root@midm9b:~#
</pre>

= Disable NetworkManager =

NOTE: THIS IS BROKEN IN UBUNTU LTS 22.04

NetworkManager is useful for configuring dynamic
network interfaces, i.e. laptops that often move
between networks, or connect to multiple choice
of wifi networks, etc.

For machines with statically configured network interfaces,
NetworkManager is not necessary.

As it has been observed to become confused and observed
to malfunction when network links go up and down (it keeps
unnecessarily reconfiguring the ip address, etc), it can
be usefuil to disable it.

* list all network interfaces
<pre>
# /bin/ls -1 /sys/class/net/
enp0s31f6
lo
</pre>
* edit /etc/network/interfaces:
<pre>
rename enp0s31f6=eth0
auto eth0
iface eth0 inet static
address 142.90.120.94/19
gateway 142.90.100.18
</pre>
* statically configure systemd-resolved
** create /etc/systemd/resolved.conf.d/resolved.conf with this contents:
<pre>
[Resolve]
DNS=142.90.100.19
Domains=triumf.ca
</pre>
** systemctl restart systemd-resolved
** resolvectl
** systemd-analyze cat-config systemd/resolved.conf
* disable NetworkManager
<pre>
systemctl disable NetworkManager
</pre>
* reboot

= Configure ECC memory =

== Configure EDAC ==

* apt install edac-utils

=== Intel i3-2120 ===
<pre>
root@musr00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X9SCL/X9SCM
root@musr00:~# edac-ctl --status
edac-ctl: drivers not loaded.
</pre>

=== Intel E-2236 ===
<pre>
root@daq00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SCM-F
root@daq00:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq00:~# edac-util
edac-util: No errors to report.
root@daq00:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
</pre>
* check edac sysfs files (Intel)
<pre>
root@daq00:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_noinfo_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 max_location
-r--r--r-- 1 root root 4096 Jan 25 15:10 mc_name
drwxr-xr-x 2 root root 0 Jan 25 15:10 power
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank0
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank1
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank2
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank3
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank4
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank5
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank6
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank7
--w------- 1 root root 4096 Jan 25 15:10 reset_counters
-r--r--r-- 1 root root 4096 Jan 25 15:10 seconds_since_reset
-r--r--r-- 1 root root 4096 Jan 25 15:10 size_mb
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Jan 25 15:10 uevent
root@daq00:~#
</pre>

=== Intel E3-1270 v6 ===
<pre>
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --status
edac-ctl: drivers are loaded.
root@grsnis01:~# edac-util
edac-util: No errors to report.
root@grsnis01:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@grsnis01:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 max_location
-r--r--r-- 1 root root 4096 Feb 19 12:35 mc_name
drwxr-xr-x 2 root root 0 Feb 19 12:35 power
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank0
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank1
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank2
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank3
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank4
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank5
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank6
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank7
--w------- 1 root root 4096 Feb 19 12:35 reset_counters
-r--r--r-- 1 root root 4096 Feb 19 12:35 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 19 12:35 size_mb
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 19 12:35 uevent
root@grsnis01:~#
</pre>

=== Intel E3-1245 v6 ===

<pre>
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --status
edac-ctl: drivers are loaded.
[root@alphagdaq ~]# edac-util
edac-util: No errors to report.
[root@alphagdaq ~]# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
[root@alphagdaq ~]# ras-mc-ctl --layout
+-----------------------------------------------+
| mc0 |
| csrow0 | csrow1 | csrow2 | csrow3 |
----------+-----------------------------------------------+
channel1: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
----------+-----------------------------------------------+
[root@alphagdaq ~]# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#3channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#3channel#1 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#2channel#0 0 0
[root@alphagdaq ~]# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SSH-F
[root@alphagdaq ~]# ras-mc-ctl --summary
DBD::SQLite::db prepare failed: no such table: mc_event at /usr/sbin/ras-mc-ctl line 1129.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1130.
[root@alphagdaq ~]#
</pre>

=== AMD 3700X ===

(memory is non-ECC)

<pre>
root@daq13:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@daq13:~#
root@daq13:~#
root@daq13:~# edac-ctl --status
edac-ctl: drivers not loaded.
root@daq13:~# edac-util
edac-util: Error: No memory controller data found.
root@daq13:~# edac-util -s
edac-util: EDAC drivers loaded. No memory controllers found
root@daq13:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 2 root root 0 Jan 25 15:26 power
lrwxrwxrwx 1 root root 0 Jan 21 16:16 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Jan 21 16:16 uevent
</pre>

(memory is ECC)

<pre>
root@trinatdaq:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@trinatdaq:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@trinatdaq:~# edac-util
edac-util: No errors to report.
root@trinatdaq:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Dec 15 13:04 mc0
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
lrwxrwxrwx 1 root root 0 Dec 13 18:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dec 13 18:31 uevent
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_noinfo_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 max_location
-r--r--r-- 1 root root 4096 Dec 15 13:04 mc_name
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank4
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank5
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank6
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank7
--w------- 1 root root 4096 Dec 15 13:04 reset_counters
-rw-r--r-- 1 root root 4096 Dec 15 13:04 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Dec 15 13:04 seconds_since_reset
-r--r--r-- 1 root root 4096 Dec 15 13:04 size_mb
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Dec 15 13:04 uevent
root@trinatdaq:~#
</pre>

=== AMD 5000G ===

* no linux driver for AMD 5000-series "G" CPU
* no mention of ECC in the BIOS settings
* unclear status of ECC support in AMD documentation (sais only "pro" "G" CPUs have ECC)
* unclear status of ECC support in ASUS documentation (web page out of date)

=== AMD 5600X ===

<pre>
root@daq17:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-XE GAMING WIFI
root@daq17:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq17:~# edac-util
edac-util: No errors to report.
root@daq17:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@daq17:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Aug 19 19:27 mc0
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
lrwxrwxrwx 1 root root 0 May 10 10:11 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 May 10 10:11 uevent
root@daq17:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_noinfo_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 max_location
-r--r--r-- 1 root root 4096 Aug 19 19:27 mc_name
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank4
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank5
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank6
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank7
--w------- 1 root root 4096 Aug 19 19:27 reset_counters
-rw-r--r-- 1 root root 4096 Aug 19 19:27 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Aug 19 19:27 seconds_since_reset
-r--r--r-- 1 root root 4096 Aug 19 19:27 size_mb
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Aug 19 19:27 uevent
root@daq17:~#
</pre>

=== AMD 3955WX ===

<pre>
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --status
edac-ctl: drivers are loaded.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util
edac-util: No errors to report.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@alphasuperdaq:~/git/scripts/quotareport# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 19 root root 0 Dez 12 04:48 mc0
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
lrwxrwxrwx 1 root root 0 Dez 9 05:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dez 9 05:31 uevent
root@alphasuperdaq:~/git/scripts/quotareport#
root@alphasuperdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 max_location
-r--r--r-- 1 root root 4096 Feb 28 22:19 mc_name
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank0
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank1
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank10
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank11
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank12
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank13
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank14
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank15
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank2
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank3
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank4
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank5
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank6
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank7
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank8
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank9
--w------- 1 root root 4096 Feb 28 22:19 reset_counters
-rw-r--r-- 1 root root 4096 Feb 28 22:19 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Feb 28 22:19 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 28 22:19 size_mb
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 28 22:19 uevent
root@alphasuperdaq:~#
root@alphasuperdaq:~# ras-mc-ctl --layout
Use of uninitialized value $max_pos[3] in modulus (%) at /usr/sbin/ras-mc-ctl line 868.
Use of uninitialized value $d in numeric ge (>=) at /usr/sbin/ras-mc-ctl line 869.
Use of uninitialized value $d in sprintf at /usr/sbin/ras-mc-ctl line 872.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| mc0 |
| csrow0 | csrow1 |
| channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 | channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

0: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
root@alphasuperdaq:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#2 0 0
mc#0csrow#1channel#7 0 0
mc#0csrow#0channel#3 0 0
mc#0csrow#1channel#4 0 0
mc#0csrow#1channel#2 0 0
mc#0csrow#0channel#7 0 0
mc#0csrow#1channel#3 0 0
mc#0csrow#0channel#4 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#5 0 0
mc#0csrow#0channel#6 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#0channel#5 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#6 0 0
root@alphasuperdaq:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: ASUSTeK COMPUTER INC. model Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@alphasuperdaq:~#
</pre>

== Configure rasdaemon ==

<pre>
apt install rasdaemon
</pre>
<pre>
systemctl enable rasdaemon
systemctl restart rasdaemon
systemctl status rasdaemon
</pre>

<pre>
● rasdaemon.service - RAS daemon to log the RAS events
Loaded: loaded (/lib/systemd/system/rasdaemon.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-01-25 15:16:37 PST; 3min 5s ago
Main PID: 2477175 (rasdaemon)
Tasks: 1 (limit: 76958)
Memory: 17.1M
CGroup: /system.slice/rasdaemon.service
└─2477175 /usr/sbin/rasdaemon -f -r

Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Listening to events for cpus 0 to 11
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mc_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording aer_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording extlog_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mce_record events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording arm_event events
</pre>

== Get reports ==

* Intel 2x32GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --layout
+-------------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-------------------------+
channel1: | 16384 MB | 16384 MB |
channel0: | 16384 MB | 16384 MB |
----------+-------------------------+
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#0channel#1 0 0
root@daq00:~#
</pre>

* Intel 4x16GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#1 0 0
mc#0csrow#2channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#3channel#0 0 0
mc#0csrow#3channel#1 0 0
root@daq00:~#
root@daq00:~# ras-mc-ctl --layout
+-----------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-----------------------+
channel1: | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB |
----------+-----------------------+
root@daq00:~#
root@daq00:~#
root@daq00:~#
root@daq00:~# ras-mc-ctl --print-labels
ras-mc-ctl: Error: No dimm labels for Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@daq00:~#
</pre>

note: ubuntu LTS 22.04 DBD::SQLite::db error is not there.

= sensors =

== ASUS P9X79 WS ==

* https://www.asus.com/supportonly/P9X79%20WS/HelpDesk_Manual/
* BIOS version 4802
* modprobe nct6775
* modprobe coretemp

<pre>
root@daq14:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 0: +29.0°C (high = +82.0°C, crit = +100.0°C)
Core 1: +24.0°C (high = +82.0°C, crit = +100.0°C)
Core 2: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +82.0°C, crit = +100.0°C)

nouveau-pci-0200
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.00 V)
temp1: +39.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

nct6776-isa-0290
Adapter: ISA adapter
Vcore: 1.04 V (min = +0.00 V, max = +1.74 V)
in1: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
AVCC: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
+3.3V: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 2.04 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 904.00 mV (min = +0.00 V, max = +0.00 V) ALARM
3VSB: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
Vbat: 3.30 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1265 RPM (min = 0 RPM)
fan2: 1909 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +34.0°C (high = +0.0°C, hyst = +0.0°C) ALARM sensor = thermistor
CPUTIN: +58.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermal diode
AUXTIN: +31.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
PECI Agent 0: +31.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +96.0°C)
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

root@daq14:~#
</pre>

== ASUS TUF GAMING B550M-PLUS WIFI II ==

* BIOS 2803, 2806
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@midm9a:~# sensors
nct6798-isa-0290
Adapter: ISA adapter
in0: 488.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.82 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in14: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 0 RPM (min = 0 RPM)
fan2: 760 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan7: 1264 RPM (min = 0 RPM)
SYSTIN: +25.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +22.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +95.0°C sensor = thermistor
AUXTIN1: +25.0°C sensor = thermistor
AUXTIN2: +25.0°C sensor = thermistor
AUXTIN3: +25.0°C sensor = thermistor
PECI Agent 0 Calibration: +23.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +32.4°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

amdgpu-pci-0800
Adapter: PCI adapter
vddgfx: 1.45 V
vddnb: 993.00 mV
edge: +28.0°C
PPT: 20.00 W

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +33.4°C

root@midm9a:~#
</pre>

== ASUS ASUS ROG STRIX B550-XE GAMING WIFI ==

* BIOS 2423, 2604
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@daq13:~# sensors
nct6798-isa-0290
Adapter: ISA adapter
in0: 344.00 mV (min = +0.00 V, max = +1.74 V)
in1: 992.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in6: 216.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.30 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.81 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in11: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 280.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 845 RPM (min = 0 RPM)
fan2: 998 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +28.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +27.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +94.0°C sensor = thermistor
AUXTIN1: +28.0°C sensor = thermistor
AUXTIN2: +28.0°C sensor = thermistor
AUXTIN3: +97.0°C sensor = thermistor
PECI Agent 0 Calibration: +27.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +33.6°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

amdgpu-pci-0600
Adapter: PCI adapter
vddgfx: 1.45 V
vddnb: 999.00 mV
edge: +29.0°C
PPT: 14.00 W

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: +30.0°C

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +33.9°C

root@daq13:~#
</pre>

== ASUS ASUS ROG STRIX B550-E GAMING ==

* bios 2803
* echo modprobe jc42 >> /etc/rc.local
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@daq17:~# sensors
jc42-i2c-1-1b
Adapter: SMBus PIIX4 adapter port 0 at 0b00
temp1: +25.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: +28.0°C

nouveau-pci-0800
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.00 V)
temp1: +34.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

nct6798-isa-0290
Adapter: ISA adapter
in0: 288.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 224.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.31 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.79 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 280.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 843 RPM (min = 0 RPM)
fan2: 629 RPM (min = 0 RPM)
fan3: 746 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +22.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +25.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +93.0°C sensor = thermistor
AUXTIN1: +22.0°C sensor = thermistor
AUXTIN2: +22.0°C sensor = thermistor
AUXTIN3: +96.0°C sensor = thermistor
PECI Agent 0 Calibration: +25.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +27.6°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

jc42-i2c-1-1a
Adapter: SMBus PIIX4 adapter port 0 at 0b00
temp1: +23.2°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

asusec-isa-0000
Adapter: ISA adapter
CPU_Opt: 0 RPM
Chipset: +34.0°C
CPU: +25.0°C
Motherboard: +22.0°C
T_Sensor: -40.0°C
VRM: +31.0°C

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +28.0°C
Tccd1: +27.5°C

root@daq17:~#
</pre>

== ASUS PRIME B650-PLUS ==

* BIOS 1811
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@dsdaqgw:~# sensors
amdgpu-pci-0b00
Adapter: PCI adapter
vddgfx: 930.00 mV
vddnb: 1.19 V
edge: +38.0°C
PPT: 25.10 W

nct6799-isa-0290
Adapter: ISA adapter
in0: 920.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.04 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 320.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.28 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.28 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.10 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.04 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 416.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 328.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 0 RPM (min = 0 RPM)
fan2: 1253 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan7: 0 RPM (min = 0 RPM)
SYSTIN: +33.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +35.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +78.0°C sensor = thermistor
AUXTIN1: +11.0°C sensor = thermistor
AUXTIN2: +20.0°C sensor = thermistor
AUXTIN3: +82.0°C sensor = thermistor
PECI Agent 0 Calibration: +35.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +42.6°C
intrusion0: ALARM
intrusion1: OK
beep_enable: disabled

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +42.6°C
Tccd1: +36.4°C

root@dsdaqgw:~#
</pre>

= Enable CPU turbo mode =

* Intel CPU has a nominal CPU frequency (i.e. 3.4GHz) and a turbo-boost CPU frequency (i.e. 4.0GHz). Here we will enable this turbo-boost mode.
* Find out CPU capability
<pre>
root@daq01:~# lscpu | grep Hz
Model name: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
CPU MHz: 3965.803
CPU max MHz: 4000.0000
CPU min MHz: 800.0000
root@daq01:~#
</pre>
* Look up this CPU in the Intel ARK database - google for the CPU model name, i.e.
https://ark.intel.com/content/www/us/en/ark/products/88196/intel-core-i7-6700-processor-8m-cache-up-to-4-00-ghz.html
* Find current frequency settings:
<pre>
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "powersave" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 2.72 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
root@daq01:~#
</pre>
* Note the following:
** current governor is "powersave"
** "performance" governor is available
** "boost state support" is supported and active.
* Confirm CPU frequency governor:
<pre>
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
powersave
powersave
powersave
powersave
powersave
powersave
powersave
powersave
root@daq01:~#
</pre>
* Change governor to "performance":
<pre>
root@daq01:~# cpupower frequency-set --governor performance
Setting cpu: 0
Setting cpu: 1
Setting cpu: 2
Setting cpu: 3
Setting cpu: 4
Setting cpu: 5
Setting cpu: 6
Setting cpu: 7
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
performance
performance
performance
performance
performance
performance
performance
performance
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "performance" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 3.93 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
</pre>
* monitor CPU frequency:
<pre>
root@daq01:~# cpupower monitor
| Nehalem || Mperf || Idle_Stats
CPU| C3 | C6 | PC3 | PC6 || C0 | Cx | Freq || POLL | C1 | C1E | C3 | C6 | C7s | C8
0| 0.00| 0.00| 0.00| 0.00|| 88.80| 11.20| 3973|| 0.00| 0.00| 0.01| 0.02| 0.31| 0.00| 4.25
4| 0.00| 0.00| 0.00| 0.00|| 4.70| 95.30| 3945|| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 95.03
1| 0.73| 3.70| 0.00| 0.00|| 4.52| 95.48| 3864|| 0.00| 0.01| 1.19| 0.44| 2.82| 0.00| 90.23
5| 0.73| 3.70| 0.00| 0.00|| 0.37| 99.63| 3807|| 0.00| 0.00| 0.03| 0.09| 1.70| 0.00| 97.64
2| 2.28| 12.86| 0.00| 0.00|| 1.41| 98.59| 3829|| 0.00| 0.86| 3.17| 0.46| 7.70| 0.00| 85.87
6| 2.28| 12.86| 0.00| 0.00|| 2.88| 97.12| 3856|| 0.00| 0.11| 4.56| 2.15| 10.31| 0.00| 78.99
3| 1.33| 4.81| 0.00| 0.00|| 0.99| 99.01| 3804|| 0.00| 0.49| 0.79| 0.01| 1.03| 0.00| 96.12
7| 1.34| 4.81| 0.00| 0.00|| 1.26| 98.74| 3818|| 0.00| 0.01| 2.32| 0.47| 5.02| 0.00| 90.06
root@daq01:~#
</pre>
* check that the CPU is not overheating:
<pre>
root@daq01:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 1: +38.0°C (high = +84.0°C, crit = +100.0°C)
Core 2: +34.0°C (high = +84.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +84.0°C, crit = +100.0°C)
</pre>
* congratulations, we are running at 4 GHz now!

= Setup ubuntu as gateway to private network =

See also:
* https://daq.triumf.ca/DaqWiki/index.php/VME-CPU#Setup_the_boot_host_computer_.28el7.29
* http://www.triumf.info/wiki/DAQwiki/index.php/Dhcpd_on_eth1

== Steps to do ==

!!! UPDATED 16feb2024 Ubuntu-22.04.03 !!!

* assign network numbers to the private network, i.e. 192.168.1.x, 192.168.2.x, etc
* (on the gateway machine, each private network interface has to have a different network number)
* (each network interface can have multiple networks attached, via VLANs or via eth0:0, eth0:1 constructs)
* assign IP addresses on the private network, save them in /etc/hosts i.e. "hvps 192.168.1.10"
* (for simplicity, assign 192.168.1.1 to the gateway machine itself)
* (IP addresses 192.168.1.0 and 192.168.1.255 are "special", do not use them)
* setup DNS server (dnsmasq) to serve contents of /etc/hosts via DNS (otherwise, many programs will see inconsistent name to IP address mapping)
* setup DHCP server (dnsmasq) to give out the IP addresses
* setup TFTP server (dnsmasq), pxelinux and NFS for diskless booting
* setup time server (chronyd) to provide common time to all devices
* setup NAT so machines on private network can access the internet (to get OS updates, etc)
* setup NIS and NFS so machines on the private network can use common home directories
* setup rsync backup of machines on the private network

== setup hosts ==

* edit /etc/hosts
<pre>
192.168.1.101 dsfe01
... and so forth
</pre>

== setup dns and dhcp ==

!!! updated 16feb2024 for Ubuntu 22.04.3 !!!

!!! note: stock systemd-resolved remains, is configured to forward queries to dnsmasq, configured to forward queries to TRIUMF DNS !!!

!!! note: per authors of systemd, bare hostnames are not permitted, a DNS domain name must always be used. DNS domain name "dsdaq" is used in this example !!!

* apt install dnsmasq
* ensure dnsmasq starts after all interfaces are up (Ubuntu-22)
<pre>
mkdir /etc/systemd/system/dnsmasq.service.d
echo -e "[Unit]\nAfter=network-online.target\n" > /etc/systemd/system/dnsmasq.service.d/local.conf
</pre>
* edit /etc/dnsmasq.conf
<pre>
# /etc/dnsmasq.conf
# DNS settings
#port=0 # disable DNS function
port=53 # enable DNS function
bind-interfaces # do not collide with systemd-resolved, we use 127.0.0.1:53, they use 127.0.0.53:53
domain-needed
bogus-priv
no-resolv
#log-queries # log DNS quesries

# TRIUMF DNS settings

server=142.90.100.19
expand-hosts
domain=dsdaq
local=/dsdaq/
localmx # do not forward MX queries to TRIUMF

# DHCP settings
interface=enp1s0f0 # VX network 192.168.0.x
#interface=missing # FEP and TSP network 192.168.1.x
interface=enp1s0f1 # controls network 192.168.2.x
#dhcp-range=192.168.1.50,192.168.1.150,infinite
dhcp-range=192.168.0.0,static
dhcp-range=192.168.2.0,static
log-dhcp # log DHCP queries
#quiet-dhcp
dhcp-ignore=tag:!known
#dhcp-boot=pxelinux.0

dhcp-option=option:dns-server,192.168.0.248
dhcp-option=option:ntp-server,192.168.0.248

# TFTP settings

enable-tftp
tftp-root=/tftpboot
</pre>
* #mkdir /tftpboot ### per tftp-root (if no ZFS)
* zfs create -o mountpoint=/tftpboot rpool/tftpboot ### (if root is ZFS)
* create resolved-dsdaq.conf with main IP address of dnsmasq
<pre>
[Resolve]
DNS=192.168.0.248
Domains=dsdaq triumf.ca
</pre>
* mkdir -p /etc/systemd/resolved.conf.d/
* /bin/rm -f /etc/systemd/resolved.conf.d/*.conf
* cp resolved-dsdaq.conf /etc/systemd/resolved.conf.d/
* systemctl stop systemd-resolved.service
* systemctl disable systemd-resolved.service
* systemctl enable dnsmasq
* systemctl restart dnsmasq
* try to "ping" or "host" some names from /etc/hosts, it should work
* try to ping daq00, daq00.triumf.ca, all should work
* resolved-dsdaq.conf goes into /etc/systemd/resolved.conf.d/ of all machines on the private network
* if not using systemd-resolved, edit /etc/resolv.conf

== setup chronyd ==

* enable ntp server:
* disable systemd-timesyncd, configure and enable chronyd per instructions above
* create dsdaq.conf
<pre>
# chrony config for dsdaq server

#allow 192.168.0.0
#allow 192.168.1.0
#allow 192.168.2.0
allow all

# end
</pre>
* cp dsdaq.conf /etc/chrony/conf.d/
* systemctl restart chronyd
* chronyc tracking ### wait until time is synchronized (a few seconds)
* create dsdaq.sources # use hostname or IP address of chronyd server
<pre>
# Put this file in /etc/chrony/sources.d
# systemctl restart chrony
# chronyc sources
# chronyc tracking
server dsdaqgw iburst prefer
# end
</pre>
* dsdaq.sources goes to /etc/chrony/sources.d of all machines on the private network

== setup diskless network booting ==

=== setup pxelinux for legacy pxe boot ===

* add bits in dnsmasq.conf
<pre>
dhcp-host=ac:1f:6b:9e:7f:4a,dsfe01,infinite
dhcp-boot=pxelinux.0
dhcp-option=17,"192.168.0.251:/nfsroot/%s,vers=3"
</pre>
* setup pxelinux for Ubuntu-18
<pre>
cd ~
wget https://www.kernel.org/pub/linux/utils/boot/syslinux/4.xx/syslinux-4.03.tar.bz2
tar xjvf syslinux-4.03.tar.bz2
cd syslinux-4.03
cp -pv ./core/pxelinux.0 ./com32/hdt/hdt.c32 ./memdisk/memdisk ./com32/menu/menu.c32 /zssd/tftpboot/
</pre>
* cd /zssd/tftpboot
<pre>
wget http://ladd00.triumf.ca/tftpboot/memtest86+-4.20.iso.zip
wget http://ladd00.triumf.ca/tftpboot/memtest86+-5.01.iso.gz
wget http://ladd00.triumf.ca/tftpboot/modules.alias
wget http://ladd00.triumf.ca/tftpboot/modules.pcimap
wget http://ladd00.triumf.ca/tftpboot/pci.ids
</pre>
* mkdir pxelinux.cfg
* emacs -nw pxelinux.cfg/default
<pre>
default menu.c32
prompt 0

menu title Welcome to the DSVSLICE PXE boot menu

timeout 50

label hdt
kernel hdt.c32

label memtest86+-5.01
kernel memdisk iso initrd=memtest86+-5.01.iso.gz

label memtest86+-4.20
kernel memdisk iso initrd=memtest86+-4.20.iso.zip

label vmlinuz-5.3.0-26-generic
menu default
kernel vmlinuz-5.3.0-26-generic
append initrd=initrd.img-5.3.0-26-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.1.1:/zssd/nfsroot/dsfe01 toram ip=dhcp panic=60 BOOTIF=enp1s0f0

#end
</pre>

=== setup pxelinux for efi pxe boot ===

* https://c-nergy.be/blog/?p=13808
* add dnsmasq.conf bits. note: root-path does not actually work, it is hardwired pxelinux.cfg/default file.
<pre>
# uefi pxe

dhcp-boot=tag:uefipxe,uefi/syslinux.efi
dhcp-option-force=tag:fe01,option:root-path,192.168.0.248:/nfsroot/fe01

# VX network 192.168.0.x

dhcp-host=40:a6:b7:c1:d9:c5,fe01,infinite,set:uefipxe,set:fe01
</pre>
* apt install syslinux pxelinux syslinux-common syslinux-efi syslinux-utils
<pre>
mkdir /tftpboot/uefi
cp /usr/lib/SYSLINUX.EFI/efi64/syslinux.efi /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/ldlinux.e64 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/menu.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/hdt.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libutil.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libmenu.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libcom32.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libgpl.c32 /tftpboot/uefi/
</pre>
* try to boot, it should bomb with "cannot load pxelinux.cfg/default"
* mkdir /tftpboot/uefi/pxelinux.cfg
* create /tftpboot/uefi/pxelinux.cfg/default, note nfsroot path is hardwired, note "http:" is used to load vmlinuz and initrd files (because tftp is super slow)
<pre>
default menu.c32
prompt 0

menu title Welcome to the DSDAQGW UEFI PXE boot menu

timeout 50

label vmlinuz-6.5.0-17-generic
kernel http://192.168.0.248:8088/uefi/vmlinuz-6.5.0-17-generic
append initrd=http://192.168.0.248:8088/uefi/initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=auto rw ip=dhcp panic=60

# append initrd=http://192.168.0.248:8088/uefi/initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.0.248:/nfsroot/fe01 rw ip=dhcp panic=60

# append initrd=initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.0.248:/nfsroot/fe01 rw ip=dhcp panic=60
# append initrd=initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=auto ip=dhcp rw panic=60

#end
</pre>
* try to boot, it will bomb with "cannot load http://...."
* install mini_httpd on port 8088, see https://acme.com/software/mini_httpd/
<pre>
apt install mini-httpd
emacs -nw /etc/default/mini-httpd # set "START=1"
emacs -nw /etc/mini-httpd.conf # set "host=192.168.0.248", "port=8088", "data_dir=/tftpboot"
mkdir /etc/systemd/system/mini-httpd.service.d
echo -e "[Unit]\nAfter=network-online.target\n" > /etc/systemd/system/mini-httpd.service.d/local.conf
systemctl enable mini-httpd
systemctl restart mini-httpd
systemctl status mini-httpd
wget http://192.168.0.248:8088/uefi/syslinux.efi
tail -100 /var/log/mini_httpd.log
</pre>
* fix initramfs bug for "nfsroot=auto", otherwise, "nfsroot=" has to be different for each machine and you have to have separate pxelinux config files for each machine
** emacs -nw /usr/lib/initramfs-tools/etc/dhcp/dhclient-enter-hooks.d/config
** add "echo ROOTPATH=..." if it is missing (Ubuntu LTS 22.04)
<pre>
echo "ROOTSERVER='${new_routers%% *}'"
echo "ROOTPATH='$new_root_path'"
echo "HOSTNAME='$new_host_name'"
</pre>
** regenerate initramfs (be careful you generate it for the right kernel!)
** see https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/2054482
<pre>
mkinitramfs 6.5.0-18-generic
</pre>
* copy linux kernel and initrd
<pre>
cp /boot/vmlinuz-6.5.0-18-generic /tftpboot/uefi/
cp /boot/initrd.img-6.5.0-18-generic /tftpboot/uefi/
chmod a+r /tftpboot/uefi/*
</pre>
* try to boot, should bomb with messages about "trying to mount root filesystem"
* tail /var/log/syslog
<pre>
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 vendor class: PXEClient:Arch:00007:UNDI:003016
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 1:netmask, 2:time-offset, 3:router, 4, 5,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 6:dns-server, 12:hostname, 13:boot-file-size,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 15:domain-name, 17:root-path, 18:extension-path,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 22:max-datagram-reassembly, 23:default-ttl,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 28:broadcast, 40:nis-domain, 41:nis-server,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 42:ntp-server, 43:vendor-encap, 50:requested-address,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 51:lease-time, 54:server-identifier, 58:T1,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 59:T2, 60:vendor-class, 66:tftp-server, 67:bootfile-name,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 97:client-machine-id, 128, 129, 130, 131,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 132, 133, 134, 135
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 next server: 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 broadcast response
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 1 option: 53 message-type 2
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 18 option: 67 bootfile-name uefi/syslinux.efi
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 12 hostname fe01
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 vendor class: PXEClient:Arch:00007:UNDI:003016
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 1:netmask, 2:time-offset, 3:router, 4, 5,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 6:dns-server, 12:hostname, 13:boot-file-size,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 15:domain-name, 17:root-path, 18:extension-path,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 22:max-datagram-reassembly, 23:default-ttl,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 28:broadcast, 40:nis-domain, 41:nis-server,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 42:ntp-server, 43:vendor-encap, 50:requested-address,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 51:lease-time, 54:server-identifier, 58:T1,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 59:T2, 60:vendor-class, 66:tftp-server, 67:bootfile-name,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 97:client-machine-id, 128, 129, 130, 131,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 132, 133, 134, 135
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 next server: 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 broadcast response
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 1 option: 53 message-type 5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 18 option: 67 bootfile-name uefi/syslinux.efi
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 12 hostname fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: error 8 User aborted the transfer received from 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/syslinux.efi to 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/syslinux.efi to 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 requested options: 1:netmask, 3:router, 6:dns-server
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 bootfile name: uefi/syslinux.efi
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 next server: 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 broadcast response
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 1 option: 53 message-type 2
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 requested options: 1:netmask, 3:router, 6:dns-server
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 bootfile name: uefi/syslinux.efi
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 next server: 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 broadcast response
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 1 option: 53 message-type 5
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/ldlinux.e64 to 192.168.0.110
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/01-40-a6-b7-c1-d9-c5 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8006E not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8006 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A800 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A80 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/pxelinux.cfg/default to 192.168.0.110
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/menu.c32 to 192.168.0.110
Feb 16 20:43:10 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/libutil.c32 to 192.168.0.110
Feb 16 20:43:10 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/pxelinux.cfg/default to 192.168.0.110
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 client provides name: dsdaqgw.triumf.ca
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 1:netmask, 28:broadcast, 2:time-offset, 3:router,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 15:domain-name, 6:dns-server, 119:domain-search,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 12:hostname, 44:netbios-ns, 47:netbios-scope,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 26:mtu, 121:classless-static-route, 42:ntp-server
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 bootfile name: uefi/syslinux.efi
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 next server: 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 1 option: 53 message-type 2
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 51 lease-time infinite
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 client provides name: dsdaqgw.triumf.ca
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 1:netmask, 28:broadcast, 2:time-offset, 3:router,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 15:domain-name, 6:dns-server, 119:domain-search,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 12:hostname, 44:netbios-ns, 47:netbios-scope,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 26:mtu, 121:classless-static-route, 42:ntp-server
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 bootfile name: uefi/syslinux.efi
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 next server: 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 1 option: 53 message-type 5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 51 lease-time infinite
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 12 hostname fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw rpc.mountd[3350210]: authenticated mount request from 192.168.0.110:981 for /nfsroot/fe01 (/nfsroot/fe01)
Feb 16 20:45:07 dsdaqgw rpc.mountd[3350210]: authenticated unmount request from 192.168.0.110:859 for /nfsroot/fe01/tmp/autoDY4k5u (/nfsroot/fe01)
</pre>
* tail /var/log/mini_httpd.log
<pre>
192.168.0.110 - - [16/Feb/2024:20:43:15 -0800] "GET /uefi/vmlinuz-6.5.0-17-generic HTTP/1.0" 200 14227944 "" "Syslinux/6.04"
192.168.0.110 - - [16/Feb/2024:20:43:24 -0800] "GET /uefi/initrd.img-6.5.0-17-generic HTTP/1.0" 200 137824833 "" "Syslinux/6.04"
</pre>

=== setup efi http boot ===

https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-deployment-prep-uefi-httpboot.html

=== setup linux kernel ===

* copy the kernel files
<pre>
cd /boot
rsync -av config* initrd* System.map* vmlinuz* /tftpboot/
</pre>
* cd /tftpboot
* chmod a+r *

=== setup nfs ===

* apt-get install nfs-kernel-server
* enable NFS over UDP, edit /etc/nfs.conf add "udp=y":
<pre>
udp=y
</pre>
<pre>
systemctl restart nfs-server.service
</pre>
* emacs -nw /etc/exports
<pre>
/nfsroot/dsfe01 dsfe01(rw,no_root_squash,async,no_subtree_check)
</pre>
* enable services
<pre>
systemctl enable nfs-server
systemctl enable nfs-mountd
systemctl enable nfs-idmapd
systemctl restart nfs-server
systemctl restart nfs-mountd
systemctl restart nfs-idmapd
</pre>
* after editing /etc/exports, run
<pre>
exportfs -av
</pre>

=== setup userland ===

!!! ubuntu-18 version !!!

* zfs create rpool/nfsroot
* zfs set dedup=verify rpool/nfsroot ### enable deduplication to save disk space because most linux images have mostly identical files
* clone ubuntu
<pre>
mkdir /nfsroot/dsfe01
cd /
rsync -avx . /nfsroot/dsfe01
</pre>
* edit config files:
* cd /nfsroot/dsfe01
* emacs -nw etc/hostname ### change to dsfe01
* emacs -nw etc/mailname ### change to dsfe01
* emacs -nw etc/yp.conf ### change daq00.triumf.ca to musr00.triumf.ca
* emacs -nw etc/defaultdomain ### change to MUSR-NIS
* cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
* emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
* emacs -nw root/.ssh/authorized_keys ### update root ssh keys
* emacs -nw etc/fstab ### add this
<pre>
192.168.1.1:/nfsroot/dsfe01 / nfs defaults,nolock 0 0
</pre>
* emacs -nw etc/chrony/chrony.conf
** comment-out all "pool" and "server" entries
** add entry "server 192.168.1.1 iburst"

After dsfe01 is booted:

* disable services:
<pre>
systemctl disable apache2
systemctl disable dnsmasq
systemctl disable zfs-import-cache
</pre>

To setup additional machines, clone dsfe01 instead of cloning the gateway machine

=== Allow manpages to be viewed ===

If <code>/</code> is mounted over NFS, <code>man</code> will report a permission error. Fix it with:

<pre>
ln -s /etc/apparmor.d/usr.bin.man /etc/apparmor.d/disable/
apparmor_parser -R /etc/apparmor.d/usr.bin.man
</pre>

== setup shared home directory ==

=== on the gateway machine ===
* define netgroups
* emacs -nw /etc/netgroup
<pre>
dsfe (dsfe01,,) (dsfe02,,)
</pre>
* emacs -nw /etc/nsswitch.conf ### edit the netgroup line to read:
<pre>
netgroup: files
</pre>
* export the home directories:
* emacs -nw /etc/exports ### add this:
<pre>
/zssd/home1 @dsfe(rw,no_root_squash,async,no_subtree_check)
</pre>
* exportfs -rc

=== on the frontend machine ===

* mkdir /home
* emacs -nw /etc/fstab ### add this:
<pre>
192.168.1.1:/zssd/home1 /home nfs defaults 0 0
</pre>
* mount -a

== setup NAT ==

NAT allows machines on the private network to connect to the internet: https://en.wikipedia.org/wiki/Network_address_translation

In these examples:
* replace "eno1" with name of the outgoing interface (the one connected to the TRIUMF network).
* replace "enp11s0" with name of the private network interface (192.168.1.x network)

* emacs -nw /etc/rc.local ### add this:
<pre>
# /etc/rc.local

# enable NAT

/sbin/iptables -t nat -A POSTROUTING -o eno1 -j MASQUERADE
iptables -L -v

# uncomment following lines if machine has prohibitive FORWARD rules:
#/sbin/iptables -I FORWARD -i eno1 -o enp11s0 -m state --state RELATED,ESTABLISHED -j ACCEPT
#/sbin/iptables -I FORWARD -i enp11s0 -o eno1 -j ACCEPT
#iptables -L -v

iptables -L -v
sysctl -w net.ipv4.ip_forward=1
#sysctl -a | grep forward

sh /etc/firewall-rfc1918.sh

# end
</pre>
* emacs -nw /etc/firewall-rfc1918.sh
<pre>
# firewall-rfc1918.sh

# prevent RFC1918 private network IP addresses from
# going in and out from our uplink.

ETH=eno1

iptables -F in-rfc1918
iptables -N in-rfc1918
iptables -A in-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A in-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A in-rfc1918 --dst 192.168.0.0/16 -j REJECT

iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -I INPUT -j in-rfc1918 -i $ETH

iptables -F out-rfc1918
iptables -N out-rfc1918
iptables -A out-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A out-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A out-rfc1918 --dst 192.168.0.0/16 -j REJECT

iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -I OUTPUT -j out-rfc1918 -o $ETH

iptables -D FORWARD -j out-rfc1918 -o $ETH
iptables -D FORWARD -j out-rfc1918 -o $ETH
iptables -I FORWARD -j out-rfc1918 -o $ETH

# allow TRIUMF-SECURE network

iptables -I in-rfc1918 -s 10.90.0.0/255.255.0.0 -j ACCEPT
iptables -I out-rfc1918 -d 10.90.0.0/255.255.0.0 -j ACCEPT

# show configuration

iptables -L -v

#end
</pre>

= KVM =

<pre>
apt install cpu-checker

root@daq13:~# kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
root@daq13:~#

(if not, shutdown, go into BIOS settings, enable CPU virtualization)

apt install virtinst ### will install many packages
apt install libvirt-clients libvirt-daemon-system-systemd libvirt-daemon qemu qemu-kvm libvirt-daemon-system virtinst bridge-utils

root@daq13:/home1/wheel# virsh list --all
Id Name State
------------------------------
1 ubuntu-guest running

apt install virt-manager

virt-install --name ubuntu-guest --os-variant ubuntu20.04 --vcpus 2 --ram 2048 --location /daq/daqstore/olchansk/linux/Ubuntu/ubuntu-20.04.3-desktop-amd64.iso --network bridge=virbr0,model=virtio --graphics none --extra-args='console=ttyS0,115200n8 serial'

virtual machine will start, boot, etc
to get out of it, CTRL + Shift followed by ]

ssh wheel@daq13
virt-manager

run virt-install again, omit "--graphics none", open graphics console from virt-manager, it booted into ubuntu installer desktop

virt-install --name test10 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --filesystem /kvm_ladd00,/ --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial" --graphics none

virt-install --name test14 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --disk /tmp/xxx/ladd00.img,bus=sata --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial rdshell" --graphics none --check path_in_use=off
</pre>

build image

<pre>
dd if=/dev/zero of=/tmp/xxx/ladd00.img bs=1024M count=20
mkfs.ext3 /tmp/xxx/ladd00.img ### ext4 fails to mount by SL6 kernel, "unknown ext4 options"
cd /kvm_ladd00/
mount -o loop /tmp/xxx/ladd00.img /mnt/tmp
rsync -av . /mnt/tmp/ --delete
umount /mnt/tmp
</pre>

on the guest, configure network: /etc/rc.local
<pre>
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

ifconfig eth2 192.168.122.2
route add -net 0.0.0.0 gw 192.168.122.1
ifconfig -a
netstat -rn

# end
</pre>

= ARM cross-compiler =

NOTE!!!

THIS IS NOT AN AARCH64 (arm64) CROSSCOMPILER!

NOTE!!!

* install packages:
<pre>
apt install libgcc-9-dev-arm64-cross
apt install gcc-arm-linux-gnueabi
apt install gcc-arm-linux-gnueabihf
apt install g++-arm-linux-gnueabihf
apt install g++-arm-linux-gnueabi
</pre>
* find out the correct -march setting, on the target machine, run:
<pre>
root@gdm00:~# g++ -Q --help=target | grep march
-march= armv8-a
</pre>

<pre>
arm-linux-gnueabi-gcc -o ttcp1 ttcp.c -march=armv7 -static
arm-linux-gnueabi-gcc -o memcpy.armv7 memcpy.cc -march=armv7 -static -O2
</pre>

= 32-bit intel cross-compiler =

Ubuntu 22.04:

<pre>
apt install libstdc++-11-dev:i386
apt install zlib1g-dev:i386
</pre>

NOTES:
* "g++ -m32" does not find libstdc++, please use "g++ -m32 -L/usr/lib/gcc/i686-linux-gnu/11/"
* to cross-build 32-bit MIDAS, use "make linux32".
* executables cross-build on Ubuntu-22 do NOT run on 32-bit Debain-11 (GLIBC and GLIBCXX version mismatch)
* executables cross-build on Ubuntu-22 run on 32-bit Debian-12.

Ubuntu

2024-02-26T20:19:55Z

Lmartin: /* Install apache httpd proxy for midas and elog */ deleted duplicate paragraph

= About Ubuntu =

AAA

= Ubuntu version =

<pre>
lsb_release -a
uname -a
</pre>

= Ubuntu installer =

* updated for Ububtu LTS 20.04.01, 22.04.1

* download the latest Ubuntu LTS desktop installer iso image
* dd the image to a USB key
* power down, disconnect all disks (all HDDs, all SSDs, all M.2)
* connect the SSD to be used as system disk
* if system will use mirrored SSDs (using ZFS mirror), leave second SSD disconnected, we will activate it later
* power up
* boot from USB key in legacy mode or UEFI mode (select this in the BIOS boot menu - F8 for ASUS, F11 for Supermicro)
* follow the instruction:
* "try ubuntu or install ubuntu" - choose "install"
* select language - accept default
* "updates and other software" - accept default settings ("normal install")
* "installation type" - select "advanced features" and "experimental: use ZFS"
* accept partition choice
* "where are you?" - select "Vancouver" (PST time zone)
* "who are you?" - leave all fields blank, except "username" set to "wheel", "password" set to the root password. hostname will be set later after configuring the network
* installation runs in a few minutes, when finished, reboot
* login as user wheel
* answer annouying questions:
* "livepatch" - say "next"
* "help improve" - select "do not send", say "next"
* "privacy" - leave "location" as "off", say "next"
* "ready to go", say "done"
* right-click on the desktop, say "open in terminal", a shell will open
* say "sudo /bin/bash", enter the root password, you now have the root shell
* run nm-connection-editor to configure the network. use netmask 255.255.224.0, gateway 142.90.100.18, DNS 142.90.100.19, search path "triumf.ca"
* after network is up (can ping ladd00), continue with post-installation steps below

= Install instructions =

== prepare ==
<pre>
apt update
apt upgrade
</pre>

== install ssh ==
<pre>
apt install ssh
</pre>

== install git/scripts ==
<pre>
apt -y install git
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd scripts
git pull
</pre>

== configure hostname ==
<pre>
vi /etc/hostname
</pre>

== disable swap ==

ubuntu installer creates a 2 GB swap partition, not useful
on 32-64 GB machine, disable it:
<pre>
vi /etc/fstab ### comment out the "swap" line
</pre>

== maybe reboot ==

this is a good point to reboot the machine to boot
the latest kernel and to set the correct hostname

== install etckeeper ==

keep contents of /etc in a git repository:

<pre>
apt -y install etckeeper
</pre>

== set timezone ==
<pre>
timedatectl list-timezones | grep -i vancouver
timedatectl set-timezone America/Vancouver
</pre>

== install time synchronization ==
<pre>
apt -y install chrony
#echo server time1.triumf.ca iburst >> /etc/chrony/chrony.conf
#echo server time2.triumf.ca iburst >> /etc/chrony/chrony.conf
#echo server time3.triumf.ca iburst >> /etc/chrony/chrony.conf
cd ~/git/scripts
git pull
cd ~
cp ~/git/scripts/etc/triumf.sources /etc/chrony/sources.d/
systemctl disable systemd-timesyncd.service
systemctl stop systemd-timesyncd.service
systemctl disable ntp
systemctl stop ntp
systemctl enable chrony
systemctl restart chrony
chronyc sources
chronyc tracking
</pre>

NOTE1: if time1, time2, time3 are already listed in /etc/crony/chrony.conf, please remove them and restart chrony.

NOTE2: if time1, time2, time3 are not listed in "chronyc tracking" or if they are not selected by "chronyc tracking", check that /etc/crony/chrony.conf contains "sourcedir /etc/chrony/sources.d". old versions of this file may not have it.

NOTE3: read https://chrony-project.org/faq.html#_should_i_prefer_chrony_over_timesyncd_if_i_do_not_need_to_run_a_server

== reenable systemd-timesyncd ==

ONLY IF CHRONY DOES NOT WORK

To configure systemd-timesyncd, set "NTP=" in /etc/systemd/timesyncd.conf

<pre>
apt remove chrony
cat /etc/systemd/timesyncd.conf
systemctl enable systemd-timesyncd.service
systemctl restart systemd-timesyncd.service
systemctl status systemd-timesyncd.service
timedatectl status
timedatectl timesync-status
</pre>

== enable outgoing email (debian 11) ==

this is different from ubuntu 20. it uses /etc/mailname and it hardwires the hostname into main.cf.

== enable outgoing email ==

we have an unusual email configuration. outgoing email should work to deliver error messages, notices, etc. incoming email is disabled, we do not receive email for local users.

this causes problems with TRIUMF smtp server. if our message cannot be delivered (wrong email address or receipient computer is turned off), TRIUMF smtp server will generate a delivery failure notification email and try to send it to the "from" address of the failed message. but the "from" address does not receive any email, so another delivery failure notification email is generated and an attempt to deliver it. which again fails, rinse and repeat.

as solution, kray created a special rule, email from scrap.triumf.ca does not generate delivery failure notices. failed messages sit in the queue for 5 days, then they are deleted. (K.O. - confirmed with kray 3jan2024).

to make this work we use the msmtp MTA package.

<pre>
cd ~
apt -y remove postfix
apt -y purge postfix # remove old config files
apt -y install mailutils msmtp msmtp-mta # say "no" to apparmor support
apt -y install bsd-mailx
cd ~/git/scripts/etc
git pull
/bin/cp -fv aliases /etc/aliases
/bin/cp -fv msmtprc /etc/msmtprc
/bin/rm -vf ~root/.forward
/bin/rm -vf /etc/mailname
Mail root
Subject: test
test
^D
CC: <CR>
</pre>

== enable outgoing email (postfix) ==

THIS IS OBSOLETE!!!

* TRIUMF: use smtp.triumf.ca
* CERN: use cernmx.cern.ch

<pre>
apt install postfix ### select "satellite system", enter full hostname "xxx.triumf.ca", enter "smtp.triumf.ca"
apt install mailutils
dpkg-reconfigure postfix ### (if postfix already installed)
</pre>
<pre>
echo olchansk@triumf.ca lindner@triumf.ca bsmith@triumf.ca >> ~root/.forward
mailx root
test
^D
</pre>

== enable ping for all users (debian 11) ==

Without this tweak, Debian will report "operation not permitted" if a user tries to ping somewhere.

<pre>
echo 'net.ipv4.ping_group_range = 0 1000' > /etc/sysctl.d/99-ping.conf
</pre>

== disable apparmor ==

On NFS-Root network booted machines!

If "man man" returns "permission denied" and syslog reports apparmor "sendmsg DENIED" errors, disable apparmor. This is supposedly fixed in kernel 6.0 and later (to be confirmed), see https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/1784499

Disable apparmor, see https://ubuntu.com/server/docs/security-apparmor

This takes effect after a reboot.

<pre>
systemctl stop apparmor.service
systemctl disable apparmor.service
</pre>

== install missing packages ==

(apt eats terminal input, even the "yes |" trick does not quite work,
repeat the following commands until they report that everything
is installed)

<pre>
yes | apt -y install ssh tcsh ethtool ncat rsync strace net-tools sysstat smartmontools lm-sensors traceroute time minicom screen git lsof debsums tmux iptables
yes | apt -y install lsb-release
yes | apt -y install flex bison
yes | apt -y install neofetch
yes | apt -y install snmp snmp-mibs-downloader
yes | apt -y install git subversion g++ gfortran cmake doxygen
yes | apt -y install curl libcurl4 libcurl4-openssl-dev
yes | apt -y install mariadb-client ### mysql client
yes | apt -y install libz-dev libzstd-dev sqlite3 libsqlite3-dev unixodbc-dev
yes | apt -y install libssl-dev
yes | apt -y install emacs xemacs21 joe
yes | apt -y install gnuplot dos2unix
yes | apt -y install mutt bsd-mailx # email clients
yes | apt -y install liblz4-tool pbzip2
yes | apt -y install libc6-dev-i386 # otherwise no /usr/include/sys/types.h
yes | apt -y install libreadline-dev
yes | apt -y install ubuntu-mate-themes
yes | apt -y install libmotif-dev libxmu-dev
yes | apt -y install libusb-dev libusb-1.0-0-dev
yes | apt -y install xfig gsfonts-x11 gsfonts-other # install fonts for xfig
yes | apt -y install libjson-perl
yes | apt -y install libgsl-dev # additional GNU Scientific Library
yes | apt -y install qt5-default # Qt development
yes | apt -y install python3-full python3-dev python3-dbg python3-pip ### for pyROOT
yes | apt -y install imagemagick imagemagick-common ckeditor # for elog
yes | apt -y install libjpeg-dev libjpeg-progs libjpeg-tools
yes | apt -y install linux-tools-common linux-tools-generic # cpupower frequency-info
yes | apt -y install rdesktop remmina remmina-plugin"*" # requested by POL
yes | apt -y install nlohmann-json3-dev # required to build MIDAS with ROOT 6.30 on Ubuntu-22
yes | apt -y install dpkg-dev cmake g++ gcc binutils libx11-dev libxpm-dev libxft-dev libxext-dev python3 libssl-dev libafterimage0 # from https://root.cern/install/dependencies/
yes | apt -y install apt install gfortran libpcre3-dev xlibmesa-glu-dev libglew-dev libftgl-dev libmysqlclient-dev libfftw3-dev libcfitsio-dev graphviz-dev libavahi-compat-libdnssd-dev libldap2-dev python3-dev python3-numpy libxml2-dev libkrb5-dev libgsl0-dev qtwebengine5-dev nlohmann-json3-dev # from https://root.cern/install/dependencies/
</pre>

Ubuntu LTS 20.04:
<pre>
yes | apt -y install linux-image-generic-hwe-20.04 linux-tools-virtual-hwe-20.04 # enable linux 5.11 series kernel
</pre>

Ubuntu LTS 22.04:
<pre>
apt -y install linux-generic-hwe-22.04 # enable linux 6.2.0 series kernel
</pre>

== disable swap (debian 11) ==

* on 64 GB RAM machines swap is not useful
* on machines booted from network (NFS-ROOT), swap does not work
* on machines running from flash (RPi, etc), flash is too slow for useful swap
* swap configured by linux installers invariably has wrong size and is not useful

<pre>
systemctl disable dphys-swapfile
systemctl stop dphys-swapfile
dphys-swapfile uninstall
</pre>

== configure DNS ==

<pre>
cd ~/git/scripts
git pull
mkdir /etc/systemd/resolved.conf.d
cp etc/resolved-triumf.conf /etc/systemd/resolved.conf.d/
systemctl restart systemd-resolved
resolvectl
#systemd-analyze cat-config systemd/resolved.conf
</pre>

== install ganglia ==

<pre>
apt -y install ganglia-monitor
cd ~root/git/scripts/ganglia
git pull
make install
./ganglia-all.perl
</pre>

== install gonodeinfo ==

* go to https://bitbucket.org/dd1/gonodeinfo follow instructions:
<pre>
yes | apt-get -y install golang
mkdir ~/git
cd ~/git
#git clone https://bitbucket.org/dd1/gonodeinfo.git
git clone https://daq00.triumf.ca/~olchansk/git/gonodeinfo.git
cd gonodeinfo
git pull
make
make install # install gonodeinfo agent
cd ~ # this is important
</pre>
* edit /etc/gonodeinfo.conf
* change "Description", "Location", "User" and "Administrator" as appropriate (or delete them)
* change "Servers" to read: Servers: daq00.triumf.ca:8601
* run "gonodeinfo -v"
* if error is "connection refused". go to the nodeinfo server to add this client to the access control list:
* on the gonodeinfo server: run /opt/gonodeinfo/gonodereceive.exe -a daq13
* try gonodeinfo again, there should be no error
* on the gonodeinfo server: run gonodereport, look at the web pages, the new machine should be listed now

== install fonts for EPICS ==

* apt install xfonts-100dpi xfonts-75dpi
* restart Xorg (i.e. "killall Xorg", this will log you out from the console)
* xlsfonts | grep -i helvetica ### should show fonts with different sizes, not just size 0 (scalable)

== install libz.so.1 for CentOS compatibility ==

KO - confirm which versions on quartus need this.

<pre>
yes | apt-get -y install zlib1g
yes | apt-get -y install zlib1g:i386 libc6:i386 libgcc1:i386 gcc-6-base:i386
</pre>

== install libpng12.so.0 for Quartus compatibility ==

(does not work anymore!!!)

<pre>
wget http://ftp.ca.debian.org/debian/pool/main/libp/libpng/libpng12-0_1.2.50-2+deb8u2_amd64.deb
dpkg --install libpng12-0_1.2.50-2+deb8u2_amd64.deb
</pre>

== install libpng12.so.0 for Quartus 13.0sp1 ==

<pre>
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0.50.0
/bin/cp -pv libpng12.so.0 libpng12.so.0.50.0 /lib/x86_64-linux-gnu/
</pre>

== install packages for Xilinx ==

ubuntu LTS 22.04 vivado 2020.1

<pre>
apt install autoconf libtool
apt install libtinfo5
apt install texinfo
apt install zlib1g:i386
</pre>

== install packages for building ROOT ==

<pre>
apt -y install libx11-dev libxpm-dev libxft-dev libxext-dev libpng-dev libjpeg-dev xlibmesa-glu-dev libxml2-dev libgsl-dev cmake
</pre>

== install 32-bit libraries for PHYSICA ==

these instructions are for running 32-bit physica executable built for SL6 on ubuntu LTS 20.04

install physica sources (cannot build, do not have g77)

<pre>
cd ~/packages
git clone https://bitbucket.org/ttriumfdaq/physica.git
</pre>

install 32-bit libraries using ubuntu package manager:

<pre>
apt install lib32z1 # libz.so
</pre>

copy 32-bit SL6 shared libraries to /lib32

<pre>
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libX11.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libgd.so.2 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libpng12.so.0 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libreadline.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libncurses.so.5 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libg2c.so.0 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libxcb.so.1 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libXpm.so.4 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libjpeg.so.62 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libfontconfig.so.1 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libfreetype.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libtinfo.so.5 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libXau.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libexpat.so.1 /lib32/
</pre>

ldd should report:

<pre>
trinatdaq:trinat> ldd /usr/local/physica/physica.exe
linux-gate.so.1 (0xf7fa2000)
libX11.so.6 => /lib32/libX11.so.6 (0xf7e43000)
libgd.so.2 => /lib32/libgd.so.2 (0xf7dfe000)
libpng12.so.0 => /lib32/libpng12.so.0 (0xf7dd6000)
libz.so.1 => /lib32/libz.so.1 (0xf7db8000)
libreadline.so.6 => /lib32/libreadline.so.6 (0xf7d7e000)
libncurses.so.5 => /lib32/libncurses.so.5 (0xf7d5b000)
libg2c.so.0 => /lib32/libg2c.so.0 (0xf7d3d000)
libm.so.6 => /lib32/libm.so.6 (0xf7c39000)
libgcc_s.so.1 => /lib32/libgcc_s.so.1 (0xf7c1a000)
libc.so.6 => /lib32/libc.so.6 (0xf7a2f000)
libxcb.so.1 => /lib32/libxcb.so.1 (0xf7a05000)
libdl.so.2 => /lib32/libdl.so.2 (0xf79ff000)
libXpm.so.4 => /lib32/libXpm.so.4 (0xf79ee000)
libjpeg.so.62 => /lib32/libjpeg.so.62 (0xf7997000)
libfontconfig.so.1 => /lib32/libfontconfig.so.1 (0xf7962000)
libfreetype.so.6 => /lib32/libfreetype.so.6 (0xf78c9000)
libtinfo.so.5 => /lib32/libtinfo.so.5 (0xf78b0000)
/lib/ld-linux.so.2 (0xf7fa4000)
libXau.so.6 => /lib32/libXau.so.6 (0xf78ad000)
libexpat.so.1 => /lib32/libexpat.so.1 (0xf7885000)
trinatdaq:trinat>
</pre>

set login environment:

<pre>
setenv TRIUMF_FONTS $HOME/packages/physica/fonts
setenv PHYSICA_DIR $HOME/packages/physica
alias physica $PHYSICA_DIR/physica-SL6-32
</pre>

test:

<pre>
cd ~/packages/physica
physica
@rangauss.pcm
</pre>

== install lightdm ==

unlike the default gdm login manager, lightdm shows the machine hostname and does not require an extra mouse click to swicth from screen saver to login mode.

<pre>
apt -y install lightdm
# select lightdm
</pre>

== install desktop environments ==

note: default display manager and default desktop are deficient, please do not skip this step.

note: if apt asks to choose the display manager, select "lightdm"

note: KO - I recommend the "MATE" desktop.

note: you will have to cut-and-paste this several times because "apt" eats commands, even with "-y" and even piped from "yes".

<pre>
# install MATE desktop
DEBIAN_FRONTEND=noninteractive apt -y install ubuntu-mate-core ubuntu-mate-desktop ubuntu-mate-themes
# install Cinnamon desktop
DEBIAN_FRONTEND=noninteractive apt -y install cinnamon
# install KDE desktop
DEBIAN_FRONTEND=noninteractive apt -y install kubuntu-desktop
# install Lxqt desktop
DEBIAN_FRONTEND=noninteractive apt -y install lxqt
# install Xfce4 desktop
DEBIAN_FRONTEND=noninteractive apt -y install xfce4
</pre>

== install ROOT ==

Please install ROOT per instructions at https://root.cern.ch.

NOTE1: The ROOT package available from Ubuntu repositories is severely out of date and cannot be used with MIDAS and ROOTANA. ### DO NOT DO THIS! apt-get install root-system

NOTE2: as of 2017-Jan-09, ROOT binary kits for Ubuntu do not work (use GCC 5 instead of GCC6), build from source instead.

== Install x2go ==

KO - is this still needed? does it cause any security problems?

x2go instructions, thanks to Art O.

<pre>
add-apt-repository ppa:x2go/stable
apt-get update
apt-get install x2goserver x2goserver-xsession
</pre>

== enable root login from ladd00/daq00 ==
<pre>
ssh localhost
CTRL-C
/bin/cp ~root/git/scripts/etc/authorized_keys ~root/.ssh/
</pre>

== disable ssh access from outside of TRIUMF ==

to stop ssh login spam, disable ssh access from outside of TRIUMF. this can be done by requesting a firewall block through the helpdesk or by local firewall rule:

<pre>
echo iptables -I INPUT ! -s 142.90.0.0/255.255.0.0 -p tcp --dport 22 -j REJECT >> /etc/rc.local
/etc/rc.local
</pre>

== install smart-status ==
<pre>
ln -s ~/git/scripts/smart-status/smart-status.perl ~root/
</pre>

== enable boot menu and boot messages ==

This will enable the grub menu (with a 10 sec timeout) and
replace black screen with exciting linux boot messages.

* emacs -nw /etc/default/grub
<pre>
GRUB_DEFAULT=0
#GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="vga=769 video=640x480"
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX=""
#GRUB_GFXMODE=640x480
</pre>
* update grub config:
<pre>
grub-mkconfig -o /boot/grub/grub.cfg
</pre>

== reboot ==

this completes installation of the base system.

following sections modify basic ubuntu to fix known problems and to enable special stuff.

= Enable automatic updates =

<pre>
apt install unattended-upgrades
cd ~/git/scripts
git pull
/bin/cp -v etc/99apt-conf-ko /etc/apt/apt.conf.d/
apt-config dump | grep Unattended
</pre>

Following is obsolete:

* emacs -nw /etc/apt/apt.conf.d/50unattended-upgrades
** uncomment in Allowed-Origins "-security" and "-updates"
** add in Allowed-Origins: "Google LLC:stable";
** uncomment/add: "Unattended-Upgrade::Mail "root";
* emacs -nw /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
</pre>
* test: unattended-upgrade --dry-run -v

NOTE: update-on-shutdown is disabled.

NOTE: there is no update-on-boot, but:

NOTE: if machine was off for a long time, the systemd update timer would have expired and it will fire soon after reboot, causing an automatic update run. this is unwanted, and there is no fix or workaround for it. K.O. June-2023.

= Fix bpool is full (obsolete) =

THIS IS CAUSED BY OBSOLETE PACKAGE zsys. PLEASE: apt remove zsys

!!! only if ROOT on ZFS !!!

There is an error in the zsys package that causes bpool to run out of space,
see [[#Ubuntu zsys]] for more details.

To fix:
<pre>
cd ~/git/scripts
git pull
cp etc/zsys.conf /etc/
zsysctl service reload
zsysctl service gc
zpool list bpool
zfs list bpool
df /boot
</pre>

= IPMI instructions =

IPMI is the board management hardware on Supermicro and other server motherboards. This includes hardware sensors - fan rotation speed, temperatures and power supply voltages.

<pre>
apt-get install ipmitool
systemctl enable ipmievd
systemctl restart ipmievd
</pre>

Run:
* ipmitool sel list ### event list
* ipmitool sel elist ### event list
* ipmitool sel clear ### clear event list (if it becomes full)
* ipmitool sensor ### report hardware sensors

= move /home/wheel =

note: this MUST be done if ZFS root and NIS/autofs with /home.

Default location of wheel's home directory will collide with autofs /home, it has to be moved,
for example to /wheel.

<pre>
# logout from the wheel user
# go to another computer
ssh root@daqubuntuxxx
zfs list | grep wheel ### identify zfs name wheel_xxxxxx
zfs set mountpoint=/wheel rpool/USERDATA/wheel_hm8fzh
emacs -nw /etc/passwd ### change wheel's home directory from /home/wheel to /wheel
su - wheel ### check that user wheel still works
</pre>

This will break wheel's ability to run snap programs, such as firefox, install chrome as listed below.

= enable NIS (ubuntu 22.04, debian 11) =

<pre>
apt -y install rpcbind nis
echo DAQ-NIS >> /etc/defaultdomain
echo ypserver daq00.triumf.ca >> /etc/yp.conf
systemctl enable ypbind.service
systemctl restart ypbind.service
systemctl status ypbind.service
ypwhich -m
</pre>

enable ypserv:

<pre>
sed -i s/NISSERVER=false/NISSERVER=slave/ /etc/default/nis
/usr/lib/yp/ypinit -s daq00
echo ypserver localhost >> /etc/yp.conf
sed -i "s/ypserver .*/ypserver localhost/" /etc/yp.conf
systemctl enable ypserv
systemctl restart ypserv
systemctl restart ypbind
</pre>

edit /etc/nsswitch.conf to read:

<pre>
# begin get data from nis
passwd: files nis
group: files nis
shadow: files nis
automount: files nis
netgroup: files nis
# end get data from nis
</pre>

enable hourly update of nis maps:

<pre>
mkdir ~root/git
cd ~root/git
git clone http://daq00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>

If this is a new machine, then on the master NIS node (daq00), add this new node to /etc/netgroup, and update NIS maps (cd /var/yp; make)

= enable NIS (ubuntu 20.04) =

* apt-get -y install portmap nis ### will ask for NIS domain (DAQ-NIS)
* dpkg-reconfigure nis ### reconfigure if already installed
* ypwhich -m
* edit /etc/default/nis
** set "NISSERVER=slave"
** Ubuntu LTS 20.04, check that "YPBINDARGS=" is blank, remove "-no-dbus" if it is there
* #edit /etc/yp.conf, comment-out everything, add "domain DAQ-NIS server localhost"
* edit /etc/yp.conf, comment-out everything, add "ypserver localhost"
* /usr/lib/yp/ypinit -s daq00
* systemctl enable nis
* systemctl restart nis
* ypwhich
* ypwhich -m
* ypcat -k passwd
* vi /etc/nsswitch.conf ### add the automount line, modify the passwd, group and shadow lines to read this:
<pre>
# begin get data from nis
passwd: files nis
group: files nis
shadow: files nis
automount: files nis
netgroup: files nis
# end get data from nis
</pre>
* enable hourly update of NIS maps
<pre>
mkdir ~root/git
cd ~root/git
git clone http://ladd00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>
* ### NOT NEEDED sudo vi /etc/idmapd.conf ### add line: "Domain = triumf.ca"

= enable autofs =

<pre>
apt -y install autofs
systemctl enable autofs
systemctl restart autofs
ls -l /home/olchansk ### test autofs, check file owner is correct
</pre>

= enable NFS server =

<pre>
apt install nfs-kernel-server
#edit /etc/exports
systemctl enable nfs-server
systemctl restart nfs-server
</pre>

= NIS master =

notes for setting up the NIS master

== wheel user ==

"wheel" is the default administrative user. We do not want it's password exported to NIS (encrypted password hash is world visible) and we do not want it's home directory exported to NFS (~wheel/.ssh is world visible and potentially writable: anybody can change ~wheel/.ssh/authorized_keys).

* move wheel's home directory from /home/wheel to /wheel (see special section about this)
* change wheel's UID and GID from 1000 to a value below MINUID in /var/yp/Makefile

== coherent uids ==

we do not want system accounts defined in /etc/passwd of the NIS master
to be included in the NIS map "passwd". this causes trouble on NIS clients
where newly installed packages fail to create local system users because same
user already exists in NIS.

This is controlled by MINUID in /var/yp/Makefile.

Historical TRIUMF uids start from around 200, but several clusters do not have any historic TRIUMF uids below 500 and MINUID is set to:
* DAQ-NIS: MINUID=200
* ISAC-NIS: MINUID=500
* TITAN-NIS: MINUID=500
* MUSR-NIS: MINUID=500
* TIG-NIS: MINUID=500 (100 on SL6 mother8pi)

Ubuntu 20 has two programs to create users:
* adduser - creates new users with UID 1000 and up as specified in /etc/adduser.conf. No problems here.
* adduser --system - creates new system users with UID 100 and up as specified in /etc/adduser.conf. No problems here.
* useradd - creates new users with UID 1000 and up as specified in /etc/login.defs. No problems here.
* useradd --system - creates new system users with UID 999 and down (read "man useradd", section at the end about SYS_UID_MAX). This collides with NIS MINUID, these system users will be included in the NIS map and cause trouble.

This problem cannot be fixed, SYS_UID_MIN, SYS_UID_MAX and UID_MIN in /etc/login.defs do not seem
to have any effect on UIDs chosen by "useradd --system". (tested on Ubuntu LTS 20.04).

So far only these system accounts seem to be affected by this:
* systemd-coredump
* ganglia

To fix:
* run "sort -r -n -t: -k3 /etc/passwd" to identify the last unused system user uid (range 100..200)
* run "sort -r -n -t: -k3 /etc/group" to identify the last unused system user gid (range 100.200)
* systemd-coredump: manually change UID and GID (package systemd-coredump is usually not installed)
* ganglia: same thing, then change ownership on all ganglia files.

Also read systemd author's opinion on system vs user UIDs:
https://github.com/systemd/systemd/issues/4850#issuecomment-265698275

= Fix systemd-logind NIS breakage =

!!! THIS IS NOT NEEDED FOR UBUNTU LTS 20.04 !!!

there is a delay in ssh logins for normal users. "ssh -v" shows the delay is after "pledge...". this
fix removes the delay.

systemd developers think that we should not use NIS and made sure there are
problems if we do. To give them credit, they do offer a workaround. Read this:
https://github.com/poettering/systemd/commit/695fe4078f0df6564a1be1c4a6a9e8a640d23b67

<pre>
mkdir /etc/systemd/system/systemd-logind.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-logind.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-logind.service
</pre>

= Fix systemd-udevd NIS breakage =

see same problem as above with udev getting stuck. ubuntu lts 20.04.

<pre>
mkdir /etc/systemd/system/systemd-udevd.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-udevd.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-udevd.service
</pre>

= Configure USB device permissions =

Configure USB device permissions for user access to USB-serial devices, Altera USB Blaster, etc.

* create file /etc/udev/rules.d/99-usb-chmod.rules with this contents:

<pre>
emacs -nw /etc/udev/rules.d/99-usb-chmod.rules
ACTION=="add", SUBSYSTEM=="usbmisc", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /dev/%c"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /proc/%c"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVICE}"
ACTION=="add", ENV{PHYSDEVBUS}=="usb-serial", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVPATH}=="/class/tty/ttyS*", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyUSB*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyACM*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyS*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", DEVPATH=="*video*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
</pre>

* reload udev rules: udevadm control --reload-rules
* apply new permissions: udevadm trigger --action=add
* watch udev activity: udevadm monitor -p

= Configure lightdm display manager =

* enable it
<pre>
echo lightdm | dpkg-reconfigure -fteletype lightdm
systemctl disable gdm
systemctl disable sddm
systemctl enable lightdm
</pre>

* make the MATE desktop as default
<pre>
cd ~root/git/scripts/
git pull
/bin/cp -v etc/lightdm_default_mate.conf /etc/lightdm/lightdm.conf.d/
</pre>

* enable login by NIS users
<pre>
/bin/cp -v etc/lightdm_enable_nis_login.conf /etc/lightdm/lightdm.conf.d/
</pre>

* restart lightdm
<pre>
systemctl stop gdm
systemctl restart lightdm
</pre>

= Install libpng12.so.0 =

Quartus 16 needs libpng12:

<pre>
wget http://mirrors.kernel.org/ubuntu/pool/main/libp/libpng/libpng12-0_1.2.54-1ubuntu1_amd64.deb
dpkg --install libpng12-0_1.2.54-1ubuntu1_amd64.deb
</pre>

= Install google-chrome =

<pre>
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
dpkg -i google-chrome-stable_current_amd64.deb
</pre>

confirm autoupdate is enabled, observe dl.google.com is present in the list of repositories:
<pre>
apt update
...
Get:5 https://dl.google.com/linux/chrome/deb stable/main amd64 Packages [1,094 B]
...
</pre>

FOLLOWING IS OBSOLETE:

Instructions from here:
https://www.ubuntuupdates.org/ppa/google_chrome?dist=stable

<pre>
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-tmp.list'
apt update
apt install google-chrome-stable
/bin/rm -f /etc/apt/sources.list.d/google-tmp.list
</pre>

= Install amanda client =

ONLY ONE MACHINES THAT HOST HOME DIRECTORIES

* apt install amanda-client
* edit /etc/amandahosts
<pre>
amanda.triumf.ca amanda amdump
</pre>
* check permissions on /etc/amandahosts:
<pre>
root@daq00:/var/log/amanda# ls -l /etc/amandahosts
-rw------- 1 backup backup 49 Jan 27 10:48 /etc/amandahosts
</pre>
* fix if needed: chown backup.backup /etc/amandahosts; chmod a= /etc/amandahosts; chmod u=wr /etc/amandahosts
* edit /etc/amanda-security.conf, add this line:
<pre>
runtar:gnutar_path=/usr/bin/tar
</pre>

On the amanda machine:

* in amanda disklist, use dump type "bsdtcp-comp-user-tar"
* su - amanda and run amcheck -c daily daq00
<pre>
-bash-4.1$ amcheck -c daily daq00

Amanda Backup Client Hosts Check
--------------------------------
Client check: 1 host checked in 0.092 seconds. 0 problems found.

(brought to you by Amanda 3.3.7p1.git.685ff76d)
</pre>

= Enable rc.local =

For reasons unknown, Ubuntu LTS 20.04 does not enable /etc/rc.local. Do this:

<pre>
cd ~/git/scripts
git pull
cp -n -v etc/rc.local /etc/
chmod a+rx /etc/rc.local
cp etc/rc-local.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable rc-local
systemctl start rc-local
systemctl status rc-local
</pre>

= Remove unwanted packages =

<pre>
apt remove zsys # broken, do not use
apt remove sddm # login manager
apt remove avahi-daemon avahi-autoipd # not sure what it does, observed using 100% CPU
apt remove modemmanager # probes all serial ports to see if it's a modem
</pre>

= Disable unwanted services =

<pre>
systemctl disable mpd
systemctl disable snapd
systemctl disable ModemManager
systemctl --global mask tracker-extract-3.service
systemctl --global mask tracker-miner-fs-3.service
systemctl daemon-reload
</pre>

= Disable sleep and suspend =

note: we see some computers randomly shutdown or go to sleep, log files indicates the "sleep" or "suspend" button was pushed by user, but no such buttons actually exist. this is the fix for this:

<pre>
systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target systemd-suspend.service systemd-hybrid-sleep.service
</pre>

= Enable crontab @reboot for MIDAS =

startup scripts have a bug - cron @reboot entries for normal users can run before autofs is ready, so if the home directory is on autofs/NFS, it cannot be accessed and the cron job fails. If MIDAS is supposed to be started by cron @reboot, it will not start (there *will* be an error message in /var/log/cron).

<pre>
mkdir /etc/systemd/system/cron.service.d
echo -e "[Unit]\nAfter=ypbind.service autofs.service\n" > /etc/systemd/system/cron.service.d/local.conf
systemctl daemon-reload
systemctl cat cron.service
</pre>

Explore the systemd dependency tree using "systemctl list-dependencies" maybe with "--all".

Visualize the exact boot sequence from previous boot: "systemd-analyze plot > xxx.svg", look at the svg file using a web browser.

Crontab entry to start midas: (install in the midas user crontab, not root crontab)

<pre>
su - midasuser
crontab -l
#@reboot /bin/bash -l -c "/home/trinat/bin/start-daq-applications"
#@reboot /bin/tcsh -c "/home/trinat/bin/start-daq-applications"
</pre>

= Install apache httpd proxy for midas and elog =

This will configure the HTTPS/SSL certificate using "certbot" and "letsencrypt" and configure an HTTPS web server using apache2.

First, configure apache2:

* execute these commands:
<pre>
apt -y install apache2
cd /etc/apache2
</pre>
* create new file conf-available/ssl-daq14.conf # use actual hostname instead of daq14
<pre>
SSLSessionCache shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout 300
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
SSLCryptoDevice builtin
</pre>
* create new file sites-available/daq14-ssl.conf # use actual hostname instead of daq14
<pre>
<IfModule mod_ssl.c>
<VirtualHost *:443>
ServerName daq14.triumf.ca
DocumentRoot /var/www/html
ErrorLog /var/log/apache2/daq14.log
SSLEngine on
# note SSLProtocol, SSLCipherSuite and some other settings are overwritten by /etc/letsencrypt/options-ssl-apache.conf
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SEED:!IDEA:!RC4
## use port specified in elogd.cfg
#ProxyPass /elog/ http://localhost:8082/ retry=1
## use mhttpd port
#ProxyPass / http://localhost:8080/ retry=1
Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"
<Location />
SSLRequireSSL
AuthType Basic
AuthName "DAQ password protected site"
Require valid-user
# create password file: touch /etc/apache2/htpasswd
# to add new user or change password: htpasswd /etc/apache2/htpasswd username
AuthUserFile /etc/apache2/htpasswd
</Location>
</VirtualHost>
</IfModule>
</pre>
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* enable ssl module
* enable new configurations
<pre>
a2enmod ssl
a2enmod headers
a2enmod proxy
a2enmod proxy_http
a2enconf ssl-daq14
a2ensite daq14-ssl
</pre>
* disable default ssl sites
<pre>
a2dissite 000-default-le-ssl
a2dissite 000-default
ls -l /etc/apache2/sites-enabled/ ### should show only daq14-ssl.conf
</pre>
* check that there are no syntax problems
<pre>
apache2ctl configtest
</pre>
* enable and start apache2:
<pre>
systemctl enable apache2
systemctl restart apache2
systemctl status apache2
</pre>
* apache2 may fail to start, look in /var/log/apache2/error.log and /var/log/apache2/daq14.log
* if it says "Failed to configure ... certificate", proceed to the step for setting certbot.
* try to access https://daq14.triumf.ca
** you should see a complaint about self-signed certificate
** you should see a request for password (do not login yet)
** if you get "connection refused", HTTPS port 443 may need to be enabled in the local firewall, look at documentation for ufw.
Second, configure certbot:

(Note: as of 2018-01-18 certbot requires use of http port 80 to get the initial https certificate,
renewal can continue to use the https port 443)

(Note: as of 2019-01-?? certbot requires use of port 80 for renewals)

(Note: unsurprisingly, this requires outside access to connect with letsencrypt, so won't work if PC is only accessible from on-site network)

* check that port 80 is not used by anything:
* netstat -an | grep LISTEN | grep ^tcp | grep 80
* lsof -P | grep -i tcp | grep LISTEN | grep 80
* if lsof reports that apache2 is listening on port 80, follow the apache2 instructions above (remove "listen 80" from apache2.conf

* install certbot (if necessary open tcp port 80 in the firewall, see documentation for ufw):
<pre>
apt install certbot python3-certbot-apache
certbot certonly --standalone --installer apache
</pre>
* then answer questions:
* "activate HTTPS for daq14.triumf.ca" - say ok
* "enter email address" - enter your own email address
* "please read terms..." - read the terms and say "agree"
* it will take a few moments...
* "congratulations..." - say ok.
<pre>
certbot install --apache --cert-name daq14.triumf.ca
</pre>
* then answer questions:
* "choose redirect..." - say "1" (no redirect)
* look inside /etc/apache2/sites-enabled/daq14-ssl.conf to see that SSLCertificateFile & co point to certbot certificates in
/etc/letsencrypt/live/daq14.triumf.ca/
* to check current renewal and to update the certbot config file in /etc/letsencrypt/renewal, run this:
<pre>
certbot renew --standalone --installer apache --force-renewal
</pre>

NOTE: this certificate will expire in 3 months, automatic renewal should work with current version of certbot

Third, activate password protection:

* as shown in the config file above, create password file and initial user: (replace "midas" with specific username)
<pre>
touch /etc/apache2/htpasswd
htpasswd /etc/apache2/htpasswd midas
</pre>

* restart apache2
<pre>
systemctl restart apache2
systemctl status apache2
</pre>

From here:
* enable proxy for MIDAS mhttpd - uncomment redirect in the config file above
* enable proxy for ELOG - ditto
<pre>
a2enmod proxy
a2enmod proxy_http
apache2ctl configtest
systemctl restart apache2
</pre>

* try accessing MIDAS https://daq14.triumf.ca/ (make sure mhttpd is running)
* if it's not working, check odb setting FIXME!
* try accessing ELog https://daq14.triumf.ca/elog/ (make sure elogd is running)
* if it's not working, check elogd.cfg file and make sure
<pre>
SSL = 0
</pre>

NOTE: if certbot fails with errors about 'module' object has no attribute 'pyopenssl',
try this: pip install requests==2.6.0

= Enable elog PDF preview =

see https://stackoverflow.com/questions/52998331/imagemagick-security-policy-pdf-blocking-conversion

* xemacs -nw /etc/ImageMagick-6/policy.xml
* remove this section at the end:
<pre>

<policy domain="coder" rights="none" pattern="PS" />
<policy domain="coder" rights="none" pattern="PS2" />
<policy domain="coder" rights="none" pattern="PS3" />
<policy domain="coder" rights="none" pattern="EPS" />
<policy domain="coder" rights="none" pattern="PDF" />
<policy domain="coder" rights="none" pattern="XPS" />
</pre>

= Install Jupyter notebook =

<pre>
From https://jupyter.org/install
apt install python3-pip
pip install jupyterlab
pip install notebook
~/.local/bin/jupyter notebook
watch the http://localhost:8888 URL that it printed
say "no" to offer to start firefox (it will not work!)
URL is: http://localhost:8888/tree?token=xxx
from the machine where you are running the web browser (i.e. google-chrome), run (replace trinat@trinatdaq with the username and machine name where you started jupyter)
open a new shell and run: ssh -v trinat@trinatdaq -L 8888:localhost:8888
in the web browser, open http://localhost:8888
this gives us the login page
in the password or token entry field, put the token from the "tree?token=xxx" above (printed by jupyter on startup)
push button "login"
jupyter page should open with the list of files in the trinat home directory
congratulate Brian with full success
</pre>

= Install ZFS quota report =

If there are any ZFS volumes, install script to report disk and quota usage

<pre>
cd ~/git/scripts/quotareport
git pull
mkdir /var/www/html/zfsquotareport
cp -pv ~/git/scripts/quotareport/sorttable.js /var/www/html/zfsquotareport/
ln -s $PWD/zfsquotareport.perl /etc/cron.daily/
touch /etc/crontab
</pre>

If httpd is configured to redirect "/" to MIDAS mhttpd:
* add following to /etc/apache2/sites-enabled/xxx-ssl.conf in front of "ProxyPass / ..."
* run "systemctl reload apache2"
<pre>
## do not proxy zfs quota report directory
ProxyPass /zfsquotareport/ !
</pre>

= Install PHP =

* apt install php libapache2-mod-php
* systemctl restart apache2
* create /var/www/html/info.php
<pre>
<?php

phpinfo();
</pre>
* open https://daq00.triumf.ca/info.php

= Configure TRIUMF printers =

<pre>
systemctl stop cups
systemctl disable cups
echo "ServerName printers.triumf.ca" > /etc/cups/client.conf
lpstat -a
</pre>

= Enable core dumps =

By default, Ubuntu LTS 20.04 installs the apport package
which disabled core dumps from user applications. (google it up!).
It is not meant to do this and documentation claims that
it is not installed and not enabled by default. Oh, well...

<pre>
apt remove apport
apt autoremove ### will remove apport-symptoms and a few other packages
</pre>

After this, core dumps are written to file "core" in the current directory.
See /proc/sys/kernel/core_pattern and /proc/sys/kernel/core_uses_pid.

Enable core dump file names to include process id, add following to /etc/rc.local

<pre>
echo 1 > /proc/sys/kernel/core_uses_pid
</pre>

= Enable debugger =

By default, Ubuntu LTS 20.04 does not permit debugger to attach and debug
already running programs. To enable it, add following to /etc/rc.local

<pre>
echo 0 > /proc/sys/kernel/yama/ptrace_scope
</pre>

= Disable Ubuntu Pro nag =

If "apt upgrade" requests Ubuntu Pro or esm-apps, disable the nag:
<pre>
/bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf
</pre>

= Update packages =

* apt-get update # update package list
* apt-get dist-upgrade # install updated packages and update "kept back" packages
* apt-get autoremove # remove packages that apt thinks should be removed

= Finish installation =

Congratulations. There is nothing more to do!

* reboot
<pre>
shutdown -r now
</pre>

= Install ZFS =

!!! after installing all the packages, after updating the system, after updating the linux kernel, after rebooting into latest kernel !!!

<pre>
apt-get install zfsutils-linux
</pre>

Follow generic ZFS instructions: [[ZFS]]

= Update to new version of Ubuntu =

<pre>
vi /etc/update-manager/release-upgrades # set "Prompt=normal"
do-release-upgrade
</pre>

Update Ubuntu LTS 20.04 to LTS 22.04:

<pre>
apt remove zsys
</pre>

== daqubuntu ==

<pre>
# reboot to clear out all updates
# vi /etc/update-manager/release-upgrades # set "Prompt=normal"
# do-release-upgrade -c
Checking for a new Ubuntu release
New release '22.04 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
# do-release-upgrade
...
say yes...
...
login.defs, say "Y" (erase local changes, use packaged version)
/etc/systemd/resolved.conf, say "Y" (same as above)
firefox snap, say yes
unable to reach snap store, say "skip"
/etc/gmond.conf, say "Y"
/var/yp/Makefile, say "install the package maintainer's version"
/etc/ypserv.conf, same thing
/etc/ypserv.securenets, same thing
/etc/default/nis, same thing
/etc/speech-dispatcher/modules/mary-generic.conf, same thing
/etc/apt/apt.conf.d/50unattended-upgrades, same thing
...
278 packages are going to be removed, say yes
...
restart required, say yes
...
no ping... yes ping...
...
ssh daqubuntu, ok
apt update, fail, DNS does not work, "host security.ubuntu.com" does not resolve.
fix resolver per https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Disable_NetworkManager
apt update, apt upgrade now works, 0 packages to update
NIS does not work.
</pre>

== midm9a ==

<pre>
login.defs
firefox snap
gmond.conf
ypserv
/etc/default/nis
unattended-upgrades
amanda-security.conf
remove obsolete (no)
reboot
configure dns
reenable nis
</pre>

== daq17 ==

<pre>
firefox snap
imagemagick policy.xml
gmond.conf
chrony.conf
/var/yp/Makefile
ypserv.conf
ypserv.securenets
/etc/default/nis
50unattended-upgrades
</pre>

== daq00 ==

per https://serverpilot.io/docs/how-to-upgrade-ubuntu-20.04-to-22.04/

<pre>
do-release-upgrade -f DistUpgradeViewNonInteractive
</pre>

if it exists "too soon" without doing anything, run it without "-f xxx", most likely it does not like something about this machine. in case of daq00 it did not like how the EFI partitions were mounted. after fixing it, non-interactive upgrade was successful.

== isdaq08 ==

* prepare
<pre>
cd ~/git/scripts
git pull
cd ~
apt -y install debsums
</pre>
* check for modified config files that make upgrade unhappy, deal with all files reported by debsums.
<pre>
root@isdaq08:~# debsums -ce
/etc/ganglia/gmond.conf
/etc/yp.conf
/etc/apt/apt.conf.d/10periodic
root@isdaq08:~#
</pre>
* restore original /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
</pre>
* apt remove ganglia-monitor
* apt remove nis
* "debsums -ce" is now empty

Run the upgrade:

* do-release-upgrade -f DistUpgradeViewNonInteractive

Post upgrade:

* configure DNS
* apt -y install linux-generic-hwe-22.04
* /bin/cp -v ~/git/scripts/etc/99apt-conf-ko /etc/apt/apt.conf.d/ # restore nightly updates
* /bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf # remove the ubuntu-pro nag
* install missing packages
* restore ganglia
* restore nis
* check zpool status, may need zpool upgrade
* reboot

= Upgrade to new version of Debian =

https://www.debian.org/releases/bookworm/amd64/release-notes/ch-upgrading.en.html

== 32-bit VME processor Debian 11 to 12 ==

* cd git/scripts; git pull; cd ~
* apt update
* apt upgrade
* edit /etc/apt/sources.list
<pre>
deb http://deb.debian.org/debian/ bookworm main
#deb http://deb.debian.org/debian/ bullseye main
#deb-src http://deb.debian.org/debian/ bullseye main
</pre>
* apt update
* apt upgrade --without-new-pkgs
* apt full-upgrade
* apt list '~c'; apt purge '~c' # purge left-over config files [residual-config]
* reboot

= Ubuntu package manager =

* apt-get install xxx # install package xxx
* apt-get update
* apt-get upgrade
* apt-get dist-upgrade
* apt-get autoremove # remove automatically installed packages required by a removed package
* apt-get remove xxx # remove package xxx
* apt-cache search . # list all available packages
* apt-cache show "." | grep ^Package # list al available packages
* apt-cache madison root-system # show all available versions of package root-system
* apt list # list all installed packages
* dpkg --listfiles libpng16-16 # list all files from this package
* apt list --installed # list all installed packages
* dpkg -S /bin/bash # what package provides this file?
* dpkg -L bash # what files provided by this package?
* debsums -ce # show modified config files
* apt-config dump # show apt configuration

= Ubuntu zsys =

NOTE: DO NOT USE ZSYS, see https://github.com/ubuntu/zsys/issues/218 and https://github.com/ubuntu/zsys/issues/230

* manual removal of old snapshots
<pre>
zsysctl show
zsysctl state remove xy69ye -s
zsysctl state remove xy69ye
zsysctl state remove xy69ye -u wheel
</pre>
* apt remove zsys

NOTE: old zsys snapshots must be cleaned manually, "zsysctl state remove xxx --system" is broken and does not remove user data snapshots

* manages system snapshots
* documentation: https://github.com/ubuntu/zsys
* documentation: (go to next article via link "newer" at the bottom) https://didrocks.fr/2020/05/21/zfs-focus-on-ubuntu-20.04-lts-whats-new/
* ubuntu 20.04 bug, too many snapshots cause /boot to become full and updates fail. https://github.com/ubuntu/zsys/issues/155
* solution: use custom /etc/zsys.conf, limit number of snapshots to 10, see trinatdaq:/etc/zsys.conf
* zsys commands:
<pre>
update-grub # list of all snapshots, errors if some snapshots are broken
zsysctl state remove lnc0k7 --system # remove snapshot
xemacs -nw /etc/zsys.conf; zsysctl service reload; zsysctl service gc # cause gc to run with new settings in zsys.conf
zfs list -r -t snapshot -o name,used,referenced,creation bpool/BOOT # list snapshots
zsysctl show # show snapshots
</pre>

= Ubuntu cloning =

to clone a ubuntu image:

<pre>
cd /nfsroot/lxcpet
emacs -nw etc/hostname ### change hostname
emacs -nw etc/mailname ### change hostname (debian 11)
emacs -nw etc/defaultdomain ### change the NIS domainname
emacs -nw etc/yp.conf ### change the NIS server
cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
emacs -nw root/.ssh/authorized_keys ### update root ssh keys
</pre>

= Ubuntu boot loader =

== boot from ZFS ==

* use UEFI boot with syslinux, see here: https://daq.triumf.ca/DaqWiki/index.php/SLinstall#Configure_UEFI_boot
* apt install zfs-initramfs
* update-initramfs -v -u
* ZFS structure:
<pre>
root@daq00:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 147G 1.62T 96K /
rpool/ROOT 17.8G 1.62T 96K none
rpool/ROOT/ubuntu_00aaaa 17.8G 1.62T 6.22G /
</pre>
* copy OS image to rpool/ROOT/ubuntu_00aaaa
* zfs set mountpoint=/ rpool
* zfs set mountpoint=none rpool/ROOT
* zfs set mountpoint=/ rpool/ROOT/ubuntu_00aaaa
* zfs get all | grep mountpoint
<pre>
rpool mountpoint / local
rpool/ROOT mountpoint none local
rpool/ROOT/ubuntu_00aaaa mountpoint / local
</pre>
* in linux kernel command line (syslinux.cfg), set "root=" to "root=ZFS=rpool/ROOT/ubuntu_00aaaa"

== boot from ZFS mirror ==

=== setup the EFI partitions ===

* assuming /dev/sdb is already setup for EFI boot, setup /dev/sda the same way:
* partition the second boot disk same as first boot disk:
<pre>
root@grsnis01:~# gdisk -l /dev/sdb
Found valid GPT with protective MBR; using GPT.
Number Start (sector) End (sector) Size Code Name
1 2048 1050623 512.0 MiB EF00 EFI system partition
2 1050624 3907029134 1.8 TiB 8300 Linux filesystem
root@grsnis01:~#
</pre>
* mkfs.msdos /dev/sdX1
* create mount points
<pre>
mkdir /boot/efi-sda
mkdir /boot/efi-sdb
</pre>
* add to /etc/fstab
<pre>
/dev/sda1 /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
/dev/sdb1 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
</pre>
* mount -a
* df | grep boot
<pre>
root@grsnis01:~# df | grep boot
/dev/sdb1 523248 98100 425148 19% /boot/efi-sdb
/dev/sda1 523248 4 523244 1% /boot/efi-sda
</pre>
* copy boot files to new boot disk
* cd /boot/efi-sdX; rsync -av . /boot/efi-sdY
* set BIOS to boot from "UEFI Hard drive", disable legacy boot (except for booting from USB key in legacy mode)
* if using UEFI boot syslinux per these instructions, linux kernel update has to be done manually:
* run ~/git/scripts/etc/update_efi_mirror.perl, follow instructions that it prints.

=== setup zfs partitions ===

use partitions compatible with Ubuntu "install on ZFS"

* gdisk "o" to create new GPT partition table
* gdisk "n" +512M ef00 to create EFI partition
* gdisk "n" +2G 8200 to create linux swap partition (not used)
* gdisk "n" +2G BE00 to create ZFS bpool partition
* gdisk "n" xxx BF00 create ZFS rpool partition

<pre>
# gdisk -l /dev/sda
Number Start (sector) End (sector) Size Code Name
1 2048 1050623 512.0 MiB EF00 EFI System Partition
2 1050624 5244927 2.0 GiB 8200
3 5244928 9439231 2.0 GiB BE00
4 9439232 234441614 107.3 GiB BF00
root@midm9a:~#
</pre>

=== setup zfs mirror ===

* see here: https://daq.triumf.ca/DaqWiki/index.php/ZFS#Convert_pool_from_single_to_mirror
<pre>
root@grsnis01:~# ls -l /dev/disk/by-id/ata*part2
lrwxrwxrwx 1 root root 10 Feb 19 16:47 /dev/disk/by-id/ata-WDC_WDS200T2B0A-00SM50_205007801081-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Feb 19 16:47 /dev/disk/by-id/ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 -> ../../sdb2

root@grsnis01:~# zpool status
pool: rpool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 ONLINE 0 0 0

errors: No known data errors

root@grsnis01:~# zpool attach rpool ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 /dev/disk/by-id/ata-WDC_WDS200T2B0A-00SM50_205007801081-part2

root@grsnis01:~# zpool status
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Feb 19 16:54:39 2021
12.6G scanned at 3.16G/s, 1.02G issued at 262M/s, 12.6G total
1.02G resilvered, 8.09% done, 0 days 00:00:45 to go
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801081-part2 ONLINE 0 0 0 (resilvering)

errors: No known data errors
</pre>
* wait
<pre>
root@grsnis01:~# zpool status
pool: rpool
state: ONLINE
scan: resilvered 12.7G in 0 days 00:00:40 with 0 errors on Fri Feb 19 16:55:19 2021
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801081-part2 ONLINE 0 0 0

errors: No known data errors
</pre>

== maintenance commands ==

* update-initramfs -v -u
* grub-install /dev/sda

= Convert from single to dual mirrored ZFS SSD =

Assuming Ubuntu LTS 22.04 with "instal on ZFS" option, we will
add a second SSD, configure ZFS to use both SSDs in mirrored
configuration and setup grub to boot from either SSD. This
is intended to create a full redundant system where failure
of either SSD does not break the system.

* identify first SSD
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* connect second SSD of identical size
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
/dev/sdb WD Blue SA510 2.5 250GB 22243Z803852 25 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* if second SSD is not autodetected, reboot
* Clone partition table automatically
If both SSDs are identical size, use this simpler method of duplicating the partition table:
<pre>
root@midm9b:~# sfdisk -d /dev/sda > part_table
root@midm9b:~# grep -v ^label-id part_table | sed -e 's/, *uuid=[0-9A-F-]*//' | sfdisk /dev/sdb
</pre>
The grep and sed in the second command are there to prevent disk ID and partition IDs from being cloned. Alternatively the part_table file can be edited manually to remove the label-id line and the uuid entries from the individual partitions.

* Clone partition table manually (e.g. for different size disks)
* list partition table of first SSD:
<pre>
root@midm9b:~# fdisk -l /dev/sda
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>
* create identical partitions on second SSD, use sector numbers from above.
<pre>
root@midm9b:~# gdisk /dev/sdb
GPT fdisk (gdisk) version 1.0.8

Partition table scan:
MBR: not present
BSD: not present
APM: not present
GPT: not present

Creating new GPT entries in memory.

Command (? for help): n
Partition number (1-128, default 1):
First sector (34-488397134, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-488397134, default = 488397134) or {+-}size{KMGTP}: 1050623
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): ef00
Changed type of partition to 'EFI system partition'

Command (? for help): n
Partition number (2-128, default 2):
First sector (34-488397134, default = 1050624) or {+-}size{KMGTP}:
Last sector (1050624-488397134, default = 488397134) or {+-}size{KMGTP}: 5244927
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 8200
Changed type of partition to 'Linux swap'

Command (? for help): n
Partition number (3-128, default 3):
First sector (34-488397134, default = 5244928) or {+-}size{KMGTP}:
Last sector (5244928-488397134, default = 488397134) or {+-}size{KMGTP}: 9439231
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): be00
Changed type of partition to 'Solaris boot'

Command (? for help): n
Partition number (4-128, default 4):
First sector (34-488397134, default = 9439232) or {+-}size{KMGTP}:
Last sector (9439232-488397134, default = 488397134) or {+-}size{KMGTP}:
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): bf00
Changed type of partition to 'Solaris root'

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sdb.
The operation has completed successfully.
root@midm9b:~# fdisk -l /dev/sda /dev/sdb
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root

Disk /dev/sdb: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: EB251739-30C6-422F-A505-5887B5A0B603

Device Start End Sectors Size Type
/dev/sdb1 2048 1050623 1048576 512M EFI System
/dev/sdb2 1050624 5244927 4194304 2G Linux swap
/dev/sdb3 5244928 9439231 4194304 2G Solaris boot
/dev/sdb4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>
* identify second SSD partitions
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part3
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part3 -> ../../sda3
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 -> ../../sdb3
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
</pre>
* convert bpool from single disk to mirrored disk:
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~# zpool attach bpool 99e03dc0-7d4d-f24b-8fa1-f042b9f135db /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3
root@midm9b:~# zpool status bpool
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors
</pre>
* convert rpool
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
root@midm9b:~# zpool attach rpool f6fd54f8-3af7-b943-ae3d-a4e480537fb9 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4
root@midm9b:~# zpool status rpool
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Jan 20 19:40:45 2023
5.83G scanned at 664M/s, 2.92M issued at 332K/s, 9.11G total
0B resilvered, 0.03% done, no estimated completion time
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~#
</pre>
* wait for resilver to complete
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
scan: resilvered 9.65G in 00:00:36 with 0 errors on Fri Jan 20 19:41:21 2023
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
</pre>
* enable booting from second SSD: (instead of /dev/sda1, /dev/sdb1, use UUID=xxx)
<pre>
root@midm9b:~# mkfs.msdos /dev/sdb1
root@midm9b:~# mkdir /boot/efi-sda
root@midm9b:~# mkdir /boot/efi-sdb
root@midm9b:~# echo "/dev/sda1 /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1" >> /etc/fstab
root@midm9b:~# echo "/dev/sdb1 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1" >> /etc/fstab
root@midm9b:~# mount -a
root@midm9b:~# df -kl
Filesystem 1K-blocks Used Available Use% Mounted on
...
/dev/sda1 523244 13720 509524 3% /boot/efi
/dev/sdb1 523244 4 523240 1% /boot/efi-sdb
...
root@midm9b:~# rsync -av /boot/efi/ /boot/efi-sdb/
sending incremental file list
EFI/
...
root@midm9b:~# ls -l /boot/efi-sda
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~# ls -l /boot/efi-sdb
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~#
</pre>
* setup script to update grub on second SSD, it must be run manually after every kernel update
<pre>
root@midm9b:~# ln -s ~/git/scripts/etc/update_efi_grub.perl ~/
root@midm9b:~# ~/update_efi_grub.perl -u
EFI dir: /boot/efi-sda
/boot/efi-sda: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sda/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sda: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sda/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
EFI dir: /boot/efi-sdb
/boot/efi-sdb: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sdb/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sdb: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sdb/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
root@midm9b:~#
</pre>

= Disable NetworkManager =

NOTE: THIS IS BROKEN IN UBUNTU LTS 22.04

NetworkManager is useful for configuring dynamic
network interfaces, i.e. laptops that often move
between networks, or connect to multiple choice
of wifi networks, etc.

For machines with statically configured network interfaces,
NetworkManager is not necessary.

As it has been observed to become confused and observed
to malfunction when network links go up and down (it keeps
unnecessarily reconfiguring the ip address, etc), it can
be usefuil to disable it.

* list all network interfaces
<pre>
# /bin/ls -1 /sys/class/net/
enp0s31f6
lo
</pre>
* edit /etc/network/interfaces:
<pre>
rename enp0s31f6=eth0
auto eth0
iface eth0 inet static
address 142.90.120.94/19
gateway 142.90.100.18
</pre>
* statically configure systemd-resolved
** create /etc/systemd/resolved.conf.d/resolved.conf with this contents:
<pre>
[Resolve]
DNS=142.90.100.19
Domains=triumf.ca
</pre>
** systemctl restart systemd-resolved
** resolvectl
** systemd-analyze cat-config systemd/resolved.conf
* disable NetworkManager
<pre>
systemctl disable NetworkManager
</pre>
* reboot

= Configure ECC memory =

== Configure EDAC ==

* apt install edac-utils

=== Intel i3-2120 ===
<pre>
root@musr00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X9SCL/X9SCM
root@musr00:~# edac-ctl --status
edac-ctl: drivers not loaded.
</pre>

=== Intel E-2236 ===
<pre>
root@daq00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SCM-F
root@daq00:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq00:~# edac-util
edac-util: No errors to report.
root@daq00:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
</pre>
* check edac sysfs files (Intel)
<pre>
root@daq00:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_noinfo_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 max_location
-r--r--r-- 1 root root 4096 Jan 25 15:10 mc_name
drwxr-xr-x 2 root root 0 Jan 25 15:10 power
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank0
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank1
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank2
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank3
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank4
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank5
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank6
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank7
--w------- 1 root root 4096 Jan 25 15:10 reset_counters
-r--r--r-- 1 root root 4096 Jan 25 15:10 seconds_since_reset
-r--r--r-- 1 root root 4096 Jan 25 15:10 size_mb
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Jan 25 15:10 uevent
root@daq00:~#
</pre>

=== Intel E3-1270 v6 ===
<pre>
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --status
edac-ctl: drivers are loaded.
root@grsnis01:~# edac-util
edac-util: No errors to report.
root@grsnis01:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@grsnis01:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 max_location
-r--r--r-- 1 root root 4096 Feb 19 12:35 mc_name
drwxr-xr-x 2 root root 0 Feb 19 12:35 power
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank0
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank1
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank2
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank3
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank4
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank5
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank6
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank7
--w------- 1 root root 4096 Feb 19 12:35 reset_counters
-r--r--r-- 1 root root 4096 Feb 19 12:35 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 19 12:35 size_mb
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 19 12:35 uevent
root@grsnis01:~#
</pre>

=== Intel E3-1245 v6 ===

<pre>
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --status
edac-ctl: drivers are loaded.
[root@alphagdaq ~]# edac-util
edac-util: No errors to report.
[root@alphagdaq ~]# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
[root@alphagdaq ~]# ras-mc-ctl --layout
+-----------------------------------------------+
| mc0 |
| csrow0 | csrow1 | csrow2 | csrow3 |
----------+-----------------------------------------------+
channel1: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
----------+-----------------------------------------------+
[root@alphagdaq ~]# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#3channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#3channel#1 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#2channel#0 0 0
[root@alphagdaq ~]# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SSH-F
[root@alphagdaq ~]# ras-mc-ctl --summary
DBD::SQLite::db prepare failed: no such table: mc_event at /usr/sbin/ras-mc-ctl line 1129.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1130.
[root@alphagdaq ~]#
</pre>

=== AMD 3700X ===

(memory is non-ECC)

<pre>
root@daq13:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@daq13:~#
root@daq13:~#
root@daq13:~# edac-ctl --status
edac-ctl: drivers not loaded.
root@daq13:~# edac-util
edac-util: Error: No memory controller data found.
root@daq13:~# edac-util -s
edac-util: EDAC drivers loaded. No memory controllers found
root@daq13:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 2 root root 0 Jan 25 15:26 power
lrwxrwxrwx 1 root root 0 Jan 21 16:16 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Jan 21 16:16 uevent
</pre>

(memory is ECC)

<pre>
root@trinatdaq:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@trinatdaq:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@trinatdaq:~# edac-util
edac-util: No errors to report.
root@trinatdaq:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Dec 15 13:04 mc0
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
lrwxrwxrwx 1 root root 0 Dec 13 18:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dec 13 18:31 uevent
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_noinfo_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 max_location
-r--r--r-- 1 root root 4096 Dec 15 13:04 mc_name
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank4
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank5
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank6
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank7
--w------- 1 root root 4096 Dec 15 13:04 reset_counters
-rw-r--r-- 1 root root 4096 Dec 15 13:04 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Dec 15 13:04 seconds_since_reset
-r--r--r-- 1 root root 4096 Dec 15 13:04 size_mb
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Dec 15 13:04 uevent
root@trinatdaq:~#
</pre>

=== AMD 5000G ===

* no linux driver for AMD 5000-series "G" CPU
* no mention of ECC in the BIOS settings
* unclear status of ECC support in AMD documentation (sais only "pro" "G" CPUs have ECC)
* unclear status of ECC support in ASUS documentation (web page out of date)

=== AMD 5600X ===

<pre>
root@daq17:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-XE GAMING WIFI
root@daq17:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq17:~# edac-util
edac-util: No errors to report.
root@daq17:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@daq17:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Aug 19 19:27 mc0
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
lrwxrwxrwx 1 root root 0 May 10 10:11 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 May 10 10:11 uevent
root@daq17:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_noinfo_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 max_location
-r--r--r-- 1 root root 4096 Aug 19 19:27 mc_name
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank4
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank5
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank6
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank7
--w------- 1 root root 4096 Aug 19 19:27 reset_counters
-rw-r--r-- 1 root root 4096 Aug 19 19:27 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Aug 19 19:27 seconds_since_reset
-r--r--r-- 1 root root 4096 Aug 19 19:27 size_mb
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Aug 19 19:27 uevent
root@daq17:~#
</pre>

=== AMD 3955WX ===

<pre>
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --status
edac-ctl: drivers are loaded.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util
edac-util: No errors to report.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@alphasuperdaq:~/git/scripts/quotareport# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 19 root root 0 Dez 12 04:48 mc0
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
lrwxrwxrwx 1 root root 0 Dez 9 05:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dez 9 05:31 uevent
root@alphasuperdaq:~/git/scripts/quotareport#
root@alphasuperdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 max_location
-r--r--r-- 1 root root 4096 Feb 28 22:19 mc_name
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank0
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank1
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank10
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank11
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank12
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank13
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank14
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank15
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank2
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank3
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank4
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank5
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank6
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank7
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank8
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank9
--w------- 1 root root 4096 Feb 28 22:19 reset_counters
-rw-r--r-- 1 root root 4096 Feb 28 22:19 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Feb 28 22:19 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 28 22:19 size_mb
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 28 22:19 uevent
root@alphasuperdaq:~#
root@alphasuperdaq:~# ras-mc-ctl --layout
Use of uninitialized value $max_pos[3] in modulus (%) at /usr/sbin/ras-mc-ctl line 868.
Use of uninitialized value $d in numeric ge (>=) at /usr/sbin/ras-mc-ctl line 869.
Use of uninitialized value $d in sprintf at /usr/sbin/ras-mc-ctl line 872.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| mc0 |
| csrow0 | csrow1 |
| channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 | channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

0: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
root@alphasuperdaq:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#2 0 0
mc#0csrow#1channel#7 0 0
mc#0csrow#0channel#3 0 0
mc#0csrow#1channel#4 0 0
mc#0csrow#1channel#2 0 0
mc#0csrow#0channel#7 0 0
mc#0csrow#1channel#3 0 0
mc#0csrow#0channel#4 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#5 0 0
mc#0csrow#0channel#6 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#0channel#5 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#6 0 0
root@alphasuperdaq:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: ASUSTeK COMPUTER INC. model Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@alphasuperdaq:~#
</pre>

== Configure rasdaemon ==

<pre>
apt install rasdaemon
</pre>
<pre>
systemctl enable rasdaemon
systemctl restart rasdaemon
systemctl status rasdaemon
</pre>

<pre>
● rasdaemon.service - RAS daemon to log the RAS events
Loaded: loaded (/lib/systemd/system/rasdaemon.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-01-25 15:16:37 PST; 3min 5s ago
Main PID: 2477175 (rasdaemon)
Tasks: 1 (limit: 76958)
Memory: 17.1M
CGroup: /system.slice/rasdaemon.service
└─2477175 /usr/sbin/rasdaemon -f -r

Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Listening to events for cpus 0 to 11
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mc_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording aer_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording extlog_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mce_record events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording arm_event events
</pre>

== Get reports ==

* Intel 2x32GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --layout
+-------------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-------------------------+
channel1: | 16384 MB | 16384 MB |
channel0: | 16384 MB | 16384 MB |
----------+-------------------------+
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#0channel#1 0 0
root@daq00:~#
</pre>

* Intel 4x16GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#1 0 0
mc#0csrow#2channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#3channel#0 0 0
mc#0csrow#3channel#1 0 0
root@daq00:~#
root@daq00:~# ras-mc-ctl --layout
+-----------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-----------------------+
channel1: | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB |
----------+-----------------------+
root@daq00:~#
root@daq00:~#
root@daq00:~#
root@daq00:~# ras-mc-ctl --print-labels
ras-mc-ctl: Error: No dimm labels for Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@daq00:~#
</pre>

note: ubuntu LTS 22.04 DBD::SQLite::db error is not there.

= sensors =

== ASUS P9X79 WS ==

* https://www.asus.com/supportonly/P9X79%20WS/HelpDesk_Manual/
* BIOS version 4802
* modprobe nct6775
* modprobe coretemp

<pre>
root@daq14:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 0: +29.0°C (high = +82.0°C, crit = +100.0°C)
Core 1: +24.0°C (high = +82.0°C, crit = +100.0°C)
Core 2: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +82.0°C, crit = +100.0°C)

nouveau-pci-0200
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.00 V)
temp1: +39.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

nct6776-isa-0290
Adapter: ISA adapter
Vcore: 1.04 V (min = +0.00 V, max = +1.74 V)
in1: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
AVCC: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
+3.3V: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 2.04 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 904.00 mV (min = +0.00 V, max = +0.00 V) ALARM
3VSB: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
Vbat: 3.30 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1265 RPM (min = 0 RPM)
fan2: 1909 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +34.0°C (high = +0.0°C, hyst = +0.0°C) ALARM sensor = thermistor
CPUTIN: +58.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermal diode
AUXTIN: +31.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
PECI Agent 0: +31.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +96.0°C)
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

root@daq14:~#
</pre>

== ASUS TUF GAMING B550M-PLUS WIFI II ==

* BIOS 2803, 2806
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@midm9a:~# sensors
nct6798-isa-0290
Adapter: ISA adapter
in0: 488.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.82 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in14: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 0 RPM (min = 0 RPM)
fan2: 760 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan7: 1264 RPM (min = 0 RPM)
SYSTIN: +25.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +22.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +95.0°C sensor = thermistor
AUXTIN1: +25.0°C sensor = thermistor
AUXTIN2: +25.0°C sensor = thermistor
AUXTIN3: +25.0°C sensor = thermistor
PECI Agent 0 Calibration: +23.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +32.4°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

amdgpu-pci-0800
Adapter: PCI adapter
vddgfx: 1.45 V
vddnb: 993.00 mV
edge: +28.0°C
PPT: 20.00 W

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +33.4°C

root@midm9a:~#
</pre>

== ASUS ASUS ROG STRIX B550-XE GAMING WIFI ==

* BIOS 2423, 2604
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@daq13:~# sensors
nct6798-isa-0290
Adapter: ISA adapter
in0: 344.00 mV (min = +0.00 V, max = +1.74 V)
in1: 992.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in6: 216.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.30 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.81 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in11: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 280.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 845 RPM (min = 0 RPM)
fan2: 998 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +28.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +27.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +94.0°C sensor = thermistor
AUXTIN1: +28.0°C sensor = thermistor
AUXTIN2: +28.0°C sensor = thermistor
AUXTIN3: +97.0°C sensor = thermistor
PECI Agent 0 Calibration: +27.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +33.6°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

amdgpu-pci-0600
Adapter: PCI adapter
vddgfx: 1.45 V
vddnb: 999.00 mV
edge: +29.0°C
PPT: 14.00 W

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: +30.0°C

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +33.9°C

root@daq13:~#
</pre>

== ASUS ASUS ROG STRIX B550-E GAMING ==

* bios 2803
* echo modprobe jc42 >> /etc/rc.local
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@daq17:~# sensors
jc42-i2c-1-1b
Adapter: SMBus PIIX4 adapter port 0 at 0b00
temp1: +25.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: +28.0°C

nouveau-pci-0800
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.00 V)
temp1: +34.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

nct6798-isa-0290
Adapter: ISA adapter
in0: 288.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 224.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.31 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.79 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 280.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 843 RPM (min = 0 RPM)
fan2: 629 RPM (min = 0 RPM)
fan3: 746 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +22.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +25.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +93.0°C sensor = thermistor
AUXTIN1: +22.0°C sensor = thermistor
AUXTIN2: +22.0°C sensor = thermistor
AUXTIN3: +96.0°C sensor = thermistor
PECI Agent 0 Calibration: +25.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +27.6°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

jc42-i2c-1-1a
Adapter: SMBus PIIX4 adapter port 0 at 0b00
temp1: +23.2°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

asusec-isa-0000
Adapter: ISA adapter
CPU_Opt: 0 RPM
Chipset: +34.0°C
CPU: +25.0°C
Motherboard: +22.0°C
T_Sensor: -40.0°C
VRM: +31.0°C

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +28.0°C
Tccd1: +27.5°C

root@daq17:~#
</pre>

== ASUS PRIME B650-PLUS ==

* BIOS 1811
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@dsdaqgw:~# sensors
amdgpu-pci-0b00
Adapter: PCI adapter
vddgfx: 930.00 mV
vddnb: 1.19 V
edge: +38.0°C
PPT: 25.10 W

nct6799-isa-0290
Adapter: ISA adapter
in0: 920.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.04 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 320.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.28 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.28 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.10 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.04 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 416.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 328.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 0 RPM (min = 0 RPM)
fan2: 1253 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan7: 0 RPM (min = 0 RPM)
SYSTIN: +33.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +35.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +78.0°C sensor = thermistor
AUXTIN1: +11.0°C sensor = thermistor
AUXTIN2: +20.0°C sensor = thermistor
AUXTIN3: +82.0°C sensor = thermistor
PECI Agent 0 Calibration: +35.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +42.6°C
intrusion0: ALARM
intrusion1: OK
beep_enable: disabled

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +42.6°C
Tccd1: +36.4°C

root@dsdaqgw:~#
</pre>

= Enable CPU turbo mode =

* Intel CPU has a nominal CPU frequency (i.e. 3.4GHz) and a turbo-boost CPU frequency (i.e. 4.0GHz). Here we will enable this turbo-boost mode.
* Find out CPU capability
<pre>
root@daq01:~# lscpu | grep Hz
Model name: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
CPU MHz: 3965.803
CPU max MHz: 4000.0000
CPU min MHz: 800.0000
root@daq01:~#
</pre>
* Look up this CPU in the Intel ARK database - google for the CPU model name, i.e.
https://ark.intel.com/content/www/us/en/ark/products/88196/intel-core-i7-6700-processor-8m-cache-up-to-4-00-ghz.html
* Find current frequency settings:
<pre>
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "powersave" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 2.72 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
root@daq01:~#
</pre>
* Note the following:
** current governor is "powersave"
** "performance" governor is available
** "boost state support" is supported and active.
* Confirm CPU frequency governor:
<pre>
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
powersave
powersave
powersave
powersave
powersave
powersave
powersave
powersave
root@daq01:~#
</pre>
* Change governor to "performance":
<pre>
root@daq01:~# cpupower frequency-set --governor performance
Setting cpu: 0
Setting cpu: 1
Setting cpu: 2
Setting cpu: 3
Setting cpu: 4
Setting cpu: 5
Setting cpu: 6
Setting cpu: 7
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
performance
performance
performance
performance
performance
performance
performance
performance
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "performance" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 3.93 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
</pre>
* monitor CPU frequency:
<pre>
root@daq01:~# cpupower monitor
| Nehalem || Mperf || Idle_Stats
CPU| C3 | C6 | PC3 | PC6 || C0 | Cx | Freq || POLL | C1 | C1E | C3 | C6 | C7s | C8
0| 0.00| 0.00| 0.00| 0.00|| 88.80| 11.20| 3973|| 0.00| 0.00| 0.01| 0.02| 0.31| 0.00| 4.25
4| 0.00| 0.00| 0.00| 0.00|| 4.70| 95.30| 3945|| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 95.03
1| 0.73| 3.70| 0.00| 0.00|| 4.52| 95.48| 3864|| 0.00| 0.01| 1.19| 0.44| 2.82| 0.00| 90.23
5| 0.73| 3.70| 0.00| 0.00|| 0.37| 99.63| 3807|| 0.00| 0.00| 0.03| 0.09| 1.70| 0.00| 97.64
2| 2.28| 12.86| 0.00| 0.00|| 1.41| 98.59| 3829|| 0.00| 0.86| 3.17| 0.46| 7.70| 0.00| 85.87
6| 2.28| 12.86| 0.00| 0.00|| 2.88| 97.12| 3856|| 0.00| 0.11| 4.56| 2.15| 10.31| 0.00| 78.99
3| 1.33| 4.81| 0.00| 0.00|| 0.99| 99.01| 3804|| 0.00| 0.49| 0.79| 0.01| 1.03| 0.00| 96.12
7| 1.34| 4.81| 0.00| 0.00|| 1.26| 98.74| 3818|| 0.00| 0.01| 2.32| 0.47| 5.02| 0.00| 90.06
root@daq01:~#
</pre>
* check that the CPU is not overheating:
<pre>
root@daq01:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 1: +38.0°C (high = +84.0°C, crit = +100.0°C)
Core 2: +34.0°C (high = +84.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +84.0°C, crit = +100.0°C)
</pre>
* congratulations, we are running at 4 GHz now!

= Setup ubuntu as gateway to private network =

See also:
* https://daq.triumf.ca/DaqWiki/index.php/VME-CPU#Setup_the_boot_host_computer_.28el7.29
* http://www.triumf.info/wiki/DAQwiki/index.php/Dhcpd_on_eth1

== Steps to do ==

!!! UPDATED 16feb2024 Ubuntu-22.04.03 !!!

* assign network numbers to the private network, i.e. 192.168.1.x, 192.168.2.x, etc
* (on the gateway machine, each private network interface has to have a different network number)
* (each network interface can have multiple networks attached, via VLANs or via eth0:0, eth0:1 constructs)
* assign IP addresses on the private network, save them in /etc/hosts i.e. "hvps 192.168.1.10"
* (for simplicity, assign 192.168.1.1 to the gateway machine itself)
* (IP addresses 192.168.1.0 and 192.168.1.255 are "special", do not use them)
* setup DNS server (dnsmasq) to serve contents of /etc/hosts via DNS (otherwise, many programs will see inconsistent name to IP address mapping)
* setup DHCP server (dnsmasq) to give out the IP addresses
* setup TFTP server (dnsmasq), pxelinux and NFS for diskless booting
* setup time server (chronyd) to provide common time to all devices
* setup NAT so machines on private network can access the internet (to get OS updates, etc)
* setup NIS and NFS so machines on the private network can use common home directories
* setup rsync backup of machines on the private network

== setup hosts ==

* edit /etc/hosts
<pre>
192.168.1.101 dsfe01
... and so forth
</pre>

== setup dns and dhcp ==

!!! updated 16feb2024 for Ubuntu 22.04.3 !!!

!!! note: stock systemd-resolved remains, is configured to forward queries to dnsmasq, configured to forward queries to TRIUMF DNS !!!

!!! note: per authors of systemd, bare hostnames are not permitted, a DNS domain name must always be used. DNS domain name "dsdaq" is used in this example !!!

* apt install dnsmasq
* edit /etc/dnsmasq.conf
<pre>
# /etc/dnsmasq.conf
# DNS settings
#port=0 # disable DNS function
port=53 # enable DNS function
bind-interfaces # do not collide with systemd-resolved, we use 127.0.0.1:53, they use 127.0.0.53:53
domain-needed
bogus-priv
no-resolv
#log-queries # log DNS quesries

# TRIUMF DNS settings

server=142.90.100.19
expand-hosts
domain=dsdaq
local=/dsdaq/
localmx # do not forward MX queries to TRIUMF

# DHCP settings
interface=enp1s0f0 # VX network 192.168.0.x
#interface=missing # FEP and TSP network 192.168.1.x
interface=enp1s0f1 # controls network 192.168.2.x
#dhcp-range=192.168.1.50,192.168.1.150,infinite
dhcp-range=192.168.0.0,static
dhcp-range=192.168.2.0,static
log-dhcp # log DHCP queries
#quiet-dhcp
dhcp-ignore=tag:!known
#dhcp-boot=pxelinux.0

dhcp-option=option:dns-server,192.168.0.248
dhcp-option=option:ntp-server,192.168.0.248

# TFTP settings

enable-tftp
tftp-root=/tftpboot
</pre>
* #mkdir /tftpboot ### per tftp-root (if no ZFS)
* zfs create -o mountpoint=/tftpboot rpool/tftpboot ### (if root is ZFS)
* create resolved-dsdaq.conf with main IP address of dnsmasq
<pre>
[Resolve]
DNS=192.168.0.248
Domains=dsdaq triumf.ca
</pre>
* mkdir -p /etc/systemd/resolved.conf.d/
* /bin/rm -f /etc/systemd/resolved.conf.d/*.conf
* cp resolved-dsdaq.conf /etc/systemd/resolved.conf.d/
* systemctl stop systemd-resolved.service
* systemctl disable systemd-resolved.service
* systemctl enable dnsmasq
* systemctl restart dnsmasq
* try to "ping" or "host" some names from /etc/hosts, it should work
* try to ping daq00, daq00.triumf.ca, all should work
* resolved-dsdaq.conf goes into /etc/systemd/resolved.conf.d/ of all machines on the private network
* if not using systemd-resolved, edit /etc/resolv.conf

== setup chronyd ==

* enable ntp server:
* disable systemd-timesyncd, configure and enable chronyd per instructions above
* create dsdaq.conf
<pre>
# chrony config for dsdaq server

#allow 192.168.0.0
#allow 192.168.1.0
#allow 192.168.2.0
allow all

# end
</pre>
* cp dsdaq.conf /etc/chrony/conf.d/
* systemctl restart chronyd
* chronyc tracking ### wait until time is synchronized (a few seconds)
* create dsdaq.sources # use hostname or IP address of chronyd server
<pre>
# Put this file in /etc/chrony/sources.d
# systemctl restart chrony
# chronyc sources
# chronyc tracking
server dsdaqgw iburst prefer
# end
</pre>
* dsdaq.sources goes to /etc/chrony/sources.d of all machines on the private network

== setup diskless network booting ==

=== setup pxelinux for legacy pxe boot ===

* add bits in dnsmasq.conf
<pre>
dhcp-host=ac:1f:6b:9e:7f:4a,dsfe01,infinite
dhcp-boot=pxelinux.0
dhcp-option=17,"192.168.0.251:/nfsroot/%s,vers=3"
</pre>
* setup pxelinux for Ubuntu-18
<pre>
cd ~
wget https://www.kernel.org/pub/linux/utils/boot/syslinux/4.xx/syslinux-4.03.tar.bz2
tar xjvf syslinux-4.03.tar.bz2
cd syslinux-4.03
cp -pv ./core/pxelinux.0 ./com32/hdt/hdt.c32 ./memdisk/memdisk ./com32/menu/menu.c32 /zssd/tftpboot/
</pre>
* cd /zssd/tftpboot
<pre>
wget http://ladd00.triumf.ca/tftpboot/memtest86+-4.20.iso.zip
wget http://ladd00.triumf.ca/tftpboot/memtest86+-5.01.iso.gz
wget http://ladd00.triumf.ca/tftpboot/modules.alias
wget http://ladd00.triumf.ca/tftpboot/modules.pcimap
wget http://ladd00.triumf.ca/tftpboot/pci.ids
</pre>
* mkdir pxelinux.cfg
* emacs -nw pxelinux.cfg/default
<pre>
default menu.c32
prompt 0

menu title Welcome to the DSVSLICE PXE boot menu

timeout 50

label hdt
kernel hdt.c32

label memtest86+-5.01
kernel memdisk iso initrd=memtest86+-5.01.iso.gz

label memtest86+-4.20
kernel memdisk iso initrd=memtest86+-4.20.iso.zip

label vmlinuz-5.3.0-26-generic
menu default
kernel vmlinuz-5.3.0-26-generic
append initrd=initrd.img-5.3.0-26-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.1.1:/zssd/nfsroot/dsfe01 toram ip=dhcp panic=60 BOOTIF=enp1s0f0

#end
</pre>

=== setup pxelinux for efi pxe boot ===

* https://c-nergy.be/blog/?p=13808
* add dnsmasq.conf bits. note: root-path does not actually work, it is hardwired pxelinux.cfg/default file.
<pre>
# uefi pxe

dhcp-boot=tag:uefipxe,uefi/syslinux.efi
dhcp-option-force=tag:fe01,option:root-path,192.168.0.248:/nfsroot/fe01

# VX network 192.168.0.x

dhcp-host=40:a6:b7:c1:d9:c5,fe01,infinite,set:uefipxe,set:fe01
</pre>
* apt install syslinux pxelinux syslinux-common syslinux-efi syslinux-utils
<pre>
mkdir /tftpboot/uefi
cp /usr/lib/SYSLINUX.EFI/efi64/syslinux.efi /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/ldlinux.e64 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/menu.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/hdt.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libutil.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libmenu.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libcom32.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libgpl.c32 /tftpboot/uefi/
</pre>
* try to boot, it should bomb with "cannot load pxelinux.cfg/default"
* mkdir /tftpboot/uefi/pxelinux.cfg
* create /tftpboot/uefi/pxelinux.cfg/default, note nfsroot path is hardwired, note "http:" is used to load vmlinuz and initrd files (because tftp is super slow)
<pre>
default menu.c32
prompt 0

menu title Welcome to the DSDAQGW UEFI PXE boot menu

timeout 50

label vmlinuz-6.5.0-17-generic
kernel http://192.168.0.248:8088/uefi/vmlinuz-6.5.0-17-generic
append initrd=http://192.168.0.248:8088/uefi/initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=auto rw ip=dhcp panic=60

# append initrd=http://192.168.0.248:8088/uefi/initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.0.248:/nfsroot/fe01 rw ip=dhcp panic=60

# append initrd=initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.0.248:/nfsroot/fe01 rw ip=dhcp panic=60
# append initrd=initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=auto ip=dhcp rw panic=60

#end
</pre>
* try to boot, it will bomb with "cannot load http://...."
* install mini_httpd on port 8088, see https://acme.com/software/mini_httpd/
<pre>
apt install mini-httpd
emacs -nw /etc/default/mini-httpd # set "START=1"
emacs -nw /etc/mini-httpd.conf # set "host=192.168.0.248", "port=8088", "data_dir=/tftpboot"
systemctl enable mini-httpd
systemctl restart mini-httpd
systemctl status mini-httpd
wget http://192.168.0.248:8088/uefi/syslinux.efi
tail -100 /var/log/mini_httpd.log
</pre>
* fix initramfs bug for "nfsroot=auto", otherwise, "nfsroot=" has to be different for each machine and you have to have separate pxelinux config files for each machine
** emacs -nw /usr/lib/initramfs-tools/etc/dhcp/dhclient-enter-hooks.d/config
** add "echo ROOTPATH=..." if it is missing (Ubuntu LTS 22.04)
<pre>
echo "ROOTSERVER='${new_routers%% *}'"
echo "ROOTPATH='$new_root_path'"
echo "HOSTNAME='$new_host_name'"
</pre>
** regenerate initramfs (be careful you generate it for the right kernel!)
** see https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/2054482
<pre>
mkinitramfs 6.5.0-18-generic
</pre>
* copy linux kernel and initrd
<pre>
cp /boot/vmlinuz-6.5.0-18-generic /tftpboot/uefi/
cp /boot/initrd.img-6.5.0-18-generic /tftpboot/uefi/
chmod a+r /tftpboot/uefi/*
</pre>
* try to boot, should bomb with messages about "trying to mount root filesystem"
* tail /var/log/syslog
<pre>
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 vendor class: PXEClient:Arch:00007:UNDI:003016
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 1:netmask, 2:time-offset, 3:router, 4, 5,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 6:dns-server, 12:hostname, 13:boot-file-size,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 15:domain-name, 17:root-path, 18:extension-path,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 22:max-datagram-reassembly, 23:default-ttl,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 28:broadcast, 40:nis-domain, 41:nis-server,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 42:ntp-server, 43:vendor-encap, 50:requested-address,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 51:lease-time, 54:server-identifier, 58:T1,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 59:T2, 60:vendor-class, 66:tftp-server, 67:bootfile-name,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 97:client-machine-id, 128, 129, 130, 131,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 132, 133, 134, 135
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 next server: 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 broadcast response
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 1 option: 53 message-type 2
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 18 option: 67 bootfile-name uefi/syslinux.efi
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 12 hostname fe01
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 vendor class: PXEClient:Arch:00007:UNDI:003016
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 1:netmask, 2:time-offset, 3:router, 4, 5,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 6:dns-server, 12:hostname, 13:boot-file-size,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 15:domain-name, 17:root-path, 18:extension-path,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 22:max-datagram-reassembly, 23:default-ttl,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 28:broadcast, 40:nis-domain, 41:nis-server,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 42:ntp-server, 43:vendor-encap, 50:requested-address,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 51:lease-time, 54:server-identifier, 58:T1,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 59:T2, 60:vendor-class, 66:tftp-server, 67:bootfile-name,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 97:client-machine-id, 128, 129, 130, 131,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 132, 133, 134, 135
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 next server: 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 broadcast response
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 1 option: 53 message-type 5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 18 option: 67 bootfile-name uefi/syslinux.efi
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 12 hostname fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: error 8 User aborted the transfer received from 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/syslinux.efi to 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/syslinux.efi to 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 requested options: 1:netmask, 3:router, 6:dns-server
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 bootfile name: uefi/syslinux.efi
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 next server: 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 broadcast response
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 1 option: 53 message-type 2
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 requested options: 1:netmask, 3:router, 6:dns-server
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 bootfile name: uefi/syslinux.efi
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 next server: 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 broadcast response
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 1 option: 53 message-type 5
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/ldlinux.e64 to 192.168.0.110
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/01-40-a6-b7-c1-d9-c5 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8006E not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8006 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A800 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A80 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/pxelinux.cfg/default to 192.168.0.110
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/menu.c32 to 192.168.0.110
Feb 16 20:43:10 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/libutil.c32 to 192.168.0.110
Feb 16 20:43:10 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/pxelinux.cfg/default to 192.168.0.110
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 client provides name: dsdaqgw.triumf.ca
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 1:netmask, 28:broadcast, 2:time-offset, 3:router,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 15:domain-name, 6:dns-server, 119:domain-search,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 12:hostname, 44:netbios-ns, 47:netbios-scope,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 26:mtu, 121:classless-static-route, 42:ntp-server
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 bootfile name: uefi/syslinux.efi
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 next server: 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 1 option: 53 message-type 2
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 51 lease-time infinite
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 client provides name: dsdaqgw.triumf.ca
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 1:netmask, 28:broadcast, 2:time-offset, 3:router,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 15:domain-name, 6:dns-server, 119:domain-search,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 12:hostname, 44:netbios-ns, 47:netbios-scope,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 26:mtu, 121:classless-static-route, 42:ntp-server
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 bootfile name: uefi/syslinux.efi
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 next server: 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 1 option: 53 message-type 5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 51 lease-time infinite
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 12 hostname fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw rpc.mountd[3350210]: authenticated mount request from 192.168.0.110:981 for /nfsroot/fe01 (/nfsroot/fe01)
Feb 16 20:45:07 dsdaqgw rpc.mountd[3350210]: authenticated unmount request from 192.168.0.110:859 for /nfsroot/fe01/tmp/autoDY4k5u (/nfsroot/fe01)
</pre>
* tail /var/log/mini_httpd.log
<pre>
192.168.0.110 - - [16/Feb/2024:20:43:15 -0800] "GET /uefi/vmlinuz-6.5.0-17-generic HTTP/1.0" 200 14227944 "" "Syslinux/6.04"
192.168.0.110 - - [16/Feb/2024:20:43:24 -0800] "GET /uefi/initrd.img-6.5.0-17-generic HTTP/1.0" 200 137824833 "" "Syslinux/6.04"
</pre>

=== setup efi http boot ===

https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-deployment-prep-uefi-httpboot.html

=== setup linux kernel ===

* copy the kernel files
<pre>
cd /boot
rsync -av config* initrd* System.map* vmlinuz* /tftpboot/
</pre>
* cd /tftpboot
* chmod a+r *

=== setup nfs ===

* apt-get install nfs-kernel-server
* emacs -nw /etc/exports
<pre>
/nfsroot/dsfe01 dsfe01(rw,no_root_squash,async,no_subtree_check)
</pre>
* enable services
<pre>
systemctl enable nfs-server
systemctl enable nfs-mountd
systemctl enable nfs-idmapd
systemctl restart nfs-server
systemctl restart nfs-mountd
systemctl restart nfs-idmapd
</pre>
* after editing /etc/exports, run
<pre>
exportfs -av
</pre>

=== setup userland ===

!!! ubuntu-18 version !!!

* zfs create rpool/nfsroot
* zfs set dedup=verify rpool/nfsroot ### enable deduplication to save disk space because most linux images have mostly identical files
* clone ubuntu
<pre>
mkdir /nfsroot/dsfe01
cd /
rsync -avx . /nfsroot/dsfe01
</pre>
* edit config files:
* cd /nfsroot/dsfe01
* emacs -nw etc/hostname ### change to dsfe01
* emacs -nw etc/mailname ### change to dsfe01
* emacs -nw etc/yp.conf ### change daq00.triumf.ca to musr00.triumf.ca
* emacs -nw etc/defaultdomain ### change to MUSR-NIS
* cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
* emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
* emacs -nw root/.ssh/authorized_keys ### update root ssh keys
* emacs -nw etc/fstab ### add this
<pre>
192.168.1.1:/nfsroot/dsfe01 / nfs defaults,nolock 0 0
</pre>
* emacs -nw etc/chrony/chrony.conf
** comment-out all "pool" and "server" entries
** add entry "server 192.168.1.1 iburst"

After dsfe01 is booted:

* disable services:
<pre>
systemctl disable apache2
systemctl disable dnsmasq
systemctl disable zfs-import-cache
</pre>

To setup additional machines, clone dsfe01 instead of cloning the gateway machine

=== Allow manpages to be viewed ===

If <code>/</code> is mounted over NFS, <code>man</code> will report a permission error. Fix it with:

<pre>
ln -s /etc/apparmor.d/usr.bin.man /etc/apparmor.d/disable/
apparmor_parser -R /etc/apparmor.d/usr.bin.man
</pre>

== setup shared home directory ==

=== on the gateway machine ===
* define netgroups
* emacs -nw /etc/netgroup
<pre>
dsfe (dsfe01,,) (dsfe02,,)
</pre>
* emacs -nw /etc/nsswitch.conf ### edit the netgroup line to read:
<pre>
netgroup: files
</pre>
* export the home directories:
* emacs -nw /etc/exports ### add this:
<pre>
/zssd/home1 @dsfe(rw,no_root_squash,async,no_subtree_check)
</pre>
* exportfs -rc

=== on the frontend machine ===

* mkdir /home
* emacs -nw /etc/fstab ### add this:
<pre>
192.168.1.1:/zssd/home1 /home nfs defaults 0 0
</pre>
* mount -a

== setup NAT ==

NAT allows machines on the private network to connect to the internet: https://en.wikipedia.org/wiki/Network_address_translation

In these examples:
* replace "eno1" with name of the outgoing interface (the one connected to the TRIUMF network).
* replace "enp11s0" with name of the private network interface (192.168.1.x network)

* emacs -nw /etc/rc.local ### add this:
<pre>
# /etc/rc.local

# enable NAT

/sbin/iptables -t nat -A POSTROUTING -o eno1 -j MASQUERADE
iptables -L -v

# uncomment following lines if machine has prohibitive FORWARD rules:
#/sbin/iptables -I FORWARD -i eno1 -o enp11s0 -m state --state RELATED,ESTABLISHED -j ACCEPT
#/sbin/iptables -I FORWARD -i enp11s0 -o eno1 -j ACCEPT
#iptables -L -v

iptables -L -v
sysctl -w net.ipv4.ip_forward=1
#sysctl -a | grep forward

sh /etc/firewall-rfc1918.sh

# end
</pre>
* emacs -nw /etc/firewall-rfc1918.sh
<pre>
# firewall-rfc1918.sh

# prevent RFC1918 private network IP addresses from
# going in and out from our uplink.

ETH=eno1

iptables -F in-rfc1918
iptables -N in-rfc1918
iptables -A in-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A in-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A in-rfc1918 --dst 192.168.0.0/16 -j REJECT

iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -I INPUT -j in-rfc1918 -i $ETH

iptables -F out-rfc1918
iptables -N out-rfc1918
iptables -A out-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A out-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A out-rfc1918 --dst 192.168.0.0/16 -j REJECT

iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -I OUTPUT -j out-rfc1918 -o $ETH

iptables -D FORWARD -j out-rfc1918 -o $ETH
iptables -D FORWARD -j out-rfc1918 -o $ETH
iptables -I FORWARD -j out-rfc1918 -o $ETH

# allow TRIUMF-SECURE network

iptables -I in-rfc1918 -s 10.90.0.0/255.255.0.0 -j ACCEPT
iptables -I out-rfc1918 -d 10.90.0.0/255.255.0.0 -j ACCEPT

# show configuration

iptables -L -v

#end
</pre>

= KVM =

<pre>
apt install cpu-checker

root@daq13:~# kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
root@daq13:~#

(if not, shutdown, go into BIOS settings, enable CPU virtualization)

apt install virtinst ### will install many packages
apt install libvirt-clients libvirt-daemon-system-systemd libvirt-daemon qemu qemu-kvm libvirt-daemon-system virtinst bridge-utils

root@daq13:/home1/wheel# virsh list --all
Id Name State
------------------------------
1 ubuntu-guest running

apt install virt-manager

virt-install --name ubuntu-guest --os-variant ubuntu20.04 --vcpus 2 --ram 2048 --location /daq/daqstore/olchansk/linux/Ubuntu/ubuntu-20.04.3-desktop-amd64.iso --network bridge=virbr0,model=virtio --graphics none --extra-args='console=ttyS0,115200n8 serial'

virtual machine will start, boot, etc
to get out of it, CTRL + Shift followed by ]

ssh wheel@daq13
virt-manager

run virt-install again, omit "--graphics none", open graphics console from virt-manager, it booted into ubuntu installer desktop

virt-install --name test10 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --filesystem /kvm_ladd00,/ --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial" --graphics none

virt-install --name test14 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --disk /tmp/xxx/ladd00.img,bus=sata --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial rdshell" --graphics none --check path_in_use=off
</pre>

build image

<pre>
dd if=/dev/zero of=/tmp/xxx/ladd00.img bs=1024M count=20
mkfs.ext3 /tmp/xxx/ladd00.img ### ext4 fails to mount by SL6 kernel, "unknown ext4 options"
cd /kvm_ladd00/
mount -o loop /tmp/xxx/ladd00.img /mnt/tmp
rsync -av . /mnt/tmp/ --delete
umount /mnt/tmp
</pre>

on the guest, configure network: /etc/rc.local
<pre>
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

ifconfig eth2 192.168.122.2
route add -net 0.0.0.0 gw 192.168.122.1
ifconfig -a
netstat -rn

# end
</pre>

= ARM cross-compiler =

<pre>
apt install libgcc-9-dev-arm64-cross
apt install gcc-arm-linux-gnueabi
apt install gcc-arm-linux-gnueabihf
apt install g++-arm-linux-gnueabihf
apt install g++-arm-linux-gnueabi
</pre>

<pre>
arm-linux-gnueabi-gcc -o ttcp1 ttcp.c -march=armv7 -static
arm-linux-gnueabi-gcc -o memcpy.armv7 memcpy.cc -march=armv7 -static -O2
</pre>

= 32-bit intel cross-compiler =

Ubuntu 22.04:

<pre>
apt install libstdc++-11-dev:i386
apt install zlib1g-dev:i386
</pre>

NOTES:
* "g++ -m32" does not find libstdc++, please use "g++ -m32 -L/usr/lib/gcc/i686-linux-gnu/11/"
* to cross-build 32-bit MIDAS, use "make linux32".
* executables cross-build on Ubuntu-22 do NOT run on 32-bit Debain-11 (GLIBC and GLIBCXX version mismatch)
* executables cross-build on Ubuntu-22 run on 32-bit Debian-12.

Ubuntu

2024-02-21T00:22:39Z

Lmartin: /* Install apache httpd proxy for midas and elog */

= About Ubuntu =

AAA

= Ubuntu version =

<pre>
lsb_release -a
uname -a
</pre>

= Ubuntu installer =

* updated for Ububtu LTS 20.04.01, 22.04.1

* download the latest Ubuntu LTS desktop installer iso image
* dd the image to a USB key
* power down, disconnect all disks (all HDDs, all SSDs, all M.2)
* connect the SSD to be used as system disk
* if system will use mirrored SSDs (using ZFS mirror), leave second SSD disconnected, we will activate it later
* power up
* boot from USB key in legacy mode or UEFI mode (select this in the BIOS boot menu - F8 for ASUS, F11 for Supermicro)
* follow the instruction:
* "try ubuntu or install ubuntu" - choose "install"
* select language - accept default
* "updates and other software" - accept default settings ("normal install")
* "installation type" - select "advanced features" and "experimental: use ZFS"
* accept partition choice
* "where are you?" - select "Vancouver" (PST time zone)
* "who are you?" - leave all fields blank, except "username" set to "wheel", "password" set to the root password. hostname will be set later after configuring the network
* installation runs in a few minutes, when finished, reboot
* login as user wheel
* answer annouying questions:
* "livepatch" - say "next"
* "help improve" - select "do not send", say "next"
* "privacy" - leave "location" as "off", say "next"
* "ready to go", say "done"
* right-click on the desktop, say "open in terminal", a shell will open
* say "sudo /bin/bash", enter the root password, you now have the root shell
* run nm-connection-editor to configure the network. use netmask 255.255.224.0, gateway 142.90.100.18, DNS 142.90.100.19, search path "triumf.ca"
* after network is up (can ping ladd00), continue with post-installation steps below

= Install instructions =

== prepare ==
<pre>
apt update
apt upgrade
</pre>

== install ssh ==
<pre>
apt install ssh
</pre>

== install git/scripts ==
<pre>
apt -y install git
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd scripts
git pull
</pre>

== configure hostname ==
<pre>
vi /etc/hostname
</pre>

== disable swap ==

ubuntu installer creates a 2 GB swap partition, not useful
on 32-64 GB machine, disable it:
<pre>
vi /etc/fstab ### comment out the "swap" line
</pre>

== maybe reboot ==

this is a good point to reboot the machine to boot
the latest kernel and to set the correct hostname

== install etckeeper ==

keep contents of /etc in a git repository:

<pre>
apt -y install etckeeper
</pre>

== set timezone ==
<pre>
timedatectl list-timezones | grep -i vancouver
timedatectl set-timezone America/Vancouver
</pre>

== install time synchronization ==
<pre>
apt -y install chrony
#echo server time1.triumf.ca iburst >> /etc/chrony/chrony.conf
#echo server time2.triumf.ca iburst >> /etc/chrony/chrony.conf
#echo server time3.triumf.ca iburst >> /etc/chrony/chrony.conf
cd ~/git/scripts
git pull
cd ~
cp ~/git/scripts/etc/triumf.sources /etc/chrony/sources.d/
systemctl disable systemd-timesyncd.service
systemctl stop systemd-timesyncd.service
systemctl disable ntp
systemctl stop ntp
systemctl enable chrony
systemctl restart chrony
chronyc sources
chronyc tracking
</pre>

NOTE1: if time1, time2, time3 are already listed in /etc/crony/chrony.conf, please remove them and restart chrony.

NOTE2: if time1, time2, time3 are not listed in "chronyc tracking" or if they are not selected by "chronyc tracking", check that /etc/crony/chrony.conf contains "sourcedir /etc/chrony/sources.d". old versions of this file may not have it.

NOTE3: read https://chrony-project.org/faq.html#_should_i_prefer_chrony_over_timesyncd_if_i_do_not_need_to_run_a_server

== reenable systemd-timesyncd ==

ONLY IF CHRONY DOES NOT WORK

To configure systemd-timesyncd, set "NTP=" in /etc/systemd/timesyncd.conf

<pre>
apt remove chrony
cat /etc/systemd/timesyncd.conf
systemctl enable systemd-timesyncd.service
systemctl restart systemd-timesyncd.service
systemctl status systemd-timesyncd.service
timedatectl status
timedatectl timesync-status
</pre>

== enable outgoing email (debian 11) ==

this is different from ubuntu 20. it uses /etc/mailname and it hardwires the hostname into main.cf.

== enable outgoing email ==

we have an unusual email configuration. outgoing email should work to deliver error messages, notices, etc. incoming email is disabled, we do not receive email for local users.

this causes problems with TRIUMF smtp server. if our message cannot be delivered (wrong email address or receipient computer is turned off), TRIUMF smtp server will generate a delivery failure notification email and try to send it to the "from" address of the failed message. but the "from" address does not receive any email, so another delivery failure notification email is generated and an attempt to deliver it. which again fails, rinse and repeat.

as solution, kray created a special rule, email from scrap.triumf.ca does not generate delivery failure notices. failed messages sit in the queue for 5 days, then they are deleted. (K.O. - confirmed with kray 3jan2024).

to make this work we use the msmtp MTA package.

<pre>
cd ~
apt -y remove postfix
apt -y purge postfix # remove old config files
apt -y install mailutils msmtp msmtp-mta # say "no" to apparmor support
apt -y install bsd-mailx
cd ~/git/scripts/etc
git pull
/bin/cp -fv aliases /etc/aliases
/bin/cp -fv msmtprc /etc/msmtprc
/bin/rm -vf ~root/.forward
/bin/rm -vf /etc/mailname
Mail root
Subject: test
test
^D
CC: <CR>
</pre>

== enable outgoing email (postfix) ==

THIS IS OBSOLETE!!!

* TRIUMF: use smtp.triumf.ca
* CERN: use cernmx.cern.ch

<pre>
apt install postfix ### select "satellite system", enter full hostname "xxx.triumf.ca", enter "smtp.triumf.ca"
apt install mailutils
dpkg-reconfigure postfix ### (if postfix already installed)
</pre>
<pre>
echo olchansk@triumf.ca lindner@triumf.ca bsmith@triumf.ca >> ~root/.forward
mailx root
test
^D
</pre>

== enable ping for all users (debian 11) ==

Without this tweak, Debian will report "operation not permitted" if a user tries to ping somewhere.

<pre>
echo 'net.ipv4.ping_group_range = 0 1000' > /etc/sysctl.d/99-ping.conf
</pre>

== disable apparmor ==

On NFS-Root network booted machines!

If "man man" returns "permission denied" and syslog reports apparmor "sendmsg DENIED" errors, disable apparmor. This is supposedly fixed in kernel 6.0 and later (to be confirmed), see https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/1784499

Disable apparmor, see https://ubuntu.com/server/docs/security-apparmor

This takes effect after a reboot.

<pre>
systemctl stop apparmor.service
systemctl disable apparmor.service
</pre>

== install missing packages ==

(apt eats terminal input, even the "yes |" trick does not quite work,
repeat the following commands until they report that everything
is installed)

<pre>
yes | apt -y install ssh tcsh ethtool ncat rsync strace net-tools sysstat smartmontools lm-sensors traceroute time minicom screen git lsof debsums tmux iptables
yes | apt -y install lsb-release
yes | apt -y install flex bison
yes | apt -y install neofetch
yes | apt -y install snmp snmp-mibs-downloader
yes | apt -y install git subversion g++ gfortran cmake doxygen
yes | apt -y install curl libcurl4 libcurl4-openssl-dev
yes | apt -y install mariadb-client ### mysql client
yes | apt -y install libz-dev libzstd-dev sqlite3 libsqlite3-dev unixodbc-dev
yes | apt -y install libssl-dev
yes | apt -y install emacs xemacs21 joe
yes | apt -y install gnuplot dos2unix
yes | apt -y install mutt bsd-mailx # email clients
yes | apt -y install liblz4-tool pbzip2
yes | apt -y install libc6-dev-i386 # otherwise no /usr/include/sys/types.h
yes | apt -y install libreadline-dev
yes | apt -y install ubuntu-mate-themes
yes | apt -y install libmotif-dev libxmu-dev
yes | apt -y install libusb-dev libusb-1.0-0-dev
yes | apt -y install xfig gsfonts-x11 gsfonts-other # install fonts for xfig
yes | apt -y install libjson-perl
yes | apt -y install libgsl-dev # additional GNU Scientific Library
yes | apt -y install qt5-default # Qt development
yes | apt -y install python3-full python3-dev python3-dbg python3-pip ### for pyROOT
yes | apt -y install imagemagick imagemagick-common ckeditor # for elog
yes | apt -y install libjpeg-dev libjpeg-progs libjpeg-tools
yes | apt -y install linux-tools-common linux-tools-generic # cpupower frequency-info
yes | apt -y install rdesktop remmina remmina-plugin"*" # requested by POL
yes | apt -y install nlohmann-json3-dev # required to build MIDAS with ROOT 6.30 on Ubuntu-22
yes | apt -y install dpkg-dev cmake g++ gcc binutils libx11-dev libxpm-dev libxft-dev libxext-dev python3 libssl-dev libafterimage0 # from https://root.cern/install/dependencies/
yes | apt -y install apt install gfortran libpcre3-dev xlibmesa-glu-dev libglew-dev libftgl-dev libmysqlclient-dev libfftw3-dev libcfitsio-dev graphviz-dev libavahi-compat-libdnssd-dev libldap2-dev python3-dev python3-numpy libxml2-dev libkrb5-dev libgsl0-dev qtwebengine5-dev nlohmann-json3-dev # from https://root.cern/install/dependencies/
</pre>

Ubuntu LTS 20.04:
<pre>
yes | apt -y install linux-image-generic-hwe-20.04 linux-tools-virtual-hwe-20.04 # enable linux 5.11 series kernel
</pre>

Ubuntu LTS 22.04:
<pre>
apt -y install linux-generic-hwe-22.04 # enable linux 6.2.0 series kernel
</pre>

== disable swap (debian 11) ==

* on 64 GB RAM machines swap is not useful
* on machines booted from network (NFS-ROOT), swap does not work
* on machines running from flash (RPi, etc), flash is too slow for useful swap
* swap configured by linux installers invariably has wrong size and is not useful

<pre>
systemctl disable dphys-swapfile
systemctl stop dphys-swapfile
dphys-swapfile uninstall
</pre>

== configure DNS ==

<pre>
cd ~/git/scripts
git pull
mkdir /etc/systemd/resolved.conf.d
cp etc/resolved-triumf.conf /etc/systemd/resolved.conf.d/
systemctl restart systemd-resolved
resolvectl
#systemd-analyze cat-config systemd/resolved.conf
</pre>

== install ganglia ==

<pre>
apt -y install ganglia-monitor
cd ~root/git/scripts/ganglia
git pull
make install
./ganglia-all.perl
</pre>

== install gonodeinfo ==

* go to https://bitbucket.org/dd1/gonodeinfo follow instructions:
<pre>
yes | apt-get -y install golang
mkdir ~/git
cd ~/git
#git clone https://bitbucket.org/dd1/gonodeinfo.git
git clone https://daq00.triumf.ca/~olchansk/git/gonodeinfo.git
cd gonodeinfo
git pull
make
make install # install gonodeinfo agent
cd ~ # this is important
</pre>
* edit /etc/gonodeinfo.conf
* change "Description", "Location", "User" and "Administrator" as appropriate (or delete them)
* change "Servers" to read: Servers: daq00.triumf.ca:8601
* run "gonodeinfo -v"
* if error is "connection refused". go to the nodeinfo server to add this client to the access control list:
* on the gonodeinfo server: run /opt/gonodeinfo/gonodereceive.exe -a daq13
* try gonodeinfo again, there should be no error
* on the gonodeinfo server: run gonodereport, look at the web pages, the new machine should be listed now

== install fonts for EPICS ==

* apt install xfonts-100dpi xfonts-75dpi
* restart Xorg (i.e. "killall Xorg", this will log you out from the console)
* xlsfonts | grep -i helvetica ### should show fonts with different sizes, not just size 0 (scalable)

== install libz.so.1 for CentOS compatibility ==

KO - confirm which versions on quartus need this.

<pre>
yes | apt-get -y install zlib1g
yes | apt-get -y install zlib1g:i386 libc6:i386 libgcc1:i386 gcc-6-base:i386
</pre>

== install libpng12.so.0 for Quartus compatibility ==

(does not work anymore!!!)

<pre>
wget http://ftp.ca.debian.org/debian/pool/main/libp/libpng/libpng12-0_1.2.50-2+deb8u2_amd64.deb
dpkg --install libpng12-0_1.2.50-2+deb8u2_amd64.deb
</pre>

== install libpng12.so.0 for Quartus 13.0sp1 ==

<pre>
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0.50.0
/bin/cp -pv libpng12.so.0 libpng12.so.0.50.0 /lib/x86_64-linux-gnu/
</pre>

== install packages for Xilinx ==

ubuntu LTS 22.04 vivado 2020.1

<pre>
apt install autoconf libtool
apt install libtinfo5
apt install texinfo
apt install zlib1g:i386
</pre>

== install packages for building ROOT ==

<pre>
apt -y install libx11-dev libxpm-dev libxft-dev libxext-dev libpng-dev libjpeg-dev xlibmesa-glu-dev libxml2-dev libgsl-dev cmake
</pre>

== install 32-bit libraries for PHYSICA ==

these instructions are for running 32-bit physica executable built for SL6 on ubuntu LTS 20.04

install physica sources (cannot build, do not have g77)

<pre>
cd ~/packages
git clone https://bitbucket.org/ttriumfdaq/physica.git
</pre>

install 32-bit libraries using ubuntu package manager:

<pre>
apt install lib32z1 # libz.so
</pre>

copy 32-bit SL6 shared libraries to /lib32

<pre>
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libX11.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libgd.so.2 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libpng12.so.0 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libreadline.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libncurses.so.5 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libg2c.so.0 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libxcb.so.1 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libXpm.so.4 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libjpeg.so.62 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libfontconfig.so.1 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libfreetype.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libtinfo.so.5 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libXau.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libexpat.so.1 /lib32/
</pre>

ldd should report:

<pre>
trinatdaq:trinat> ldd /usr/local/physica/physica.exe
linux-gate.so.1 (0xf7fa2000)
libX11.so.6 => /lib32/libX11.so.6 (0xf7e43000)
libgd.so.2 => /lib32/libgd.so.2 (0xf7dfe000)
libpng12.so.0 => /lib32/libpng12.so.0 (0xf7dd6000)
libz.so.1 => /lib32/libz.so.1 (0xf7db8000)
libreadline.so.6 => /lib32/libreadline.so.6 (0xf7d7e000)
libncurses.so.5 => /lib32/libncurses.so.5 (0xf7d5b000)
libg2c.so.0 => /lib32/libg2c.so.0 (0xf7d3d000)
libm.so.6 => /lib32/libm.so.6 (0xf7c39000)
libgcc_s.so.1 => /lib32/libgcc_s.so.1 (0xf7c1a000)
libc.so.6 => /lib32/libc.so.6 (0xf7a2f000)
libxcb.so.1 => /lib32/libxcb.so.1 (0xf7a05000)
libdl.so.2 => /lib32/libdl.so.2 (0xf79ff000)
libXpm.so.4 => /lib32/libXpm.so.4 (0xf79ee000)
libjpeg.so.62 => /lib32/libjpeg.so.62 (0xf7997000)
libfontconfig.so.1 => /lib32/libfontconfig.so.1 (0xf7962000)
libfreetype.so.6 => /lib32/libfreetype.so.6 (0xf78c9000)
libtinfo.so.5 => /lib32/libtinfo.so.5 (0xf78b0000)
/lib/ld-linux.so.2 (0xf7fa4000)
libXau.so.6 => /lib32/libXau.so.6 (0xf78ad000)
libexpat.so.1 => /lib32/libexpat.so.1 (0xf7885000)
trinatdaq:trinat>
</pre>

set login environment:

<pre>
setenv TRIUMF_FONTS $HOME/packages/physica/fonts
setenv PHYSICA_DIR $HOME/packages/physica
alias physica $PHYSICA_DIR/physica-SL6-32
</pre>

test:

<pre>
cd ~/packages/physica
physica
@rangauss.pcm
</pre>

== install lightdm ==

unlike the default gdm login manager, lightdm shows the machine hostname and does not require an extra mouse click to swicth from screen saver to login mode.

<pre>
apt -y install lightdm
# select lightdm
</pre>

== install desktop environments ==

note: default display manager and default desktop are deficient, please do not skip this step.

note: if apt asks to choose the display manager, select "lightdm"

note: KO - I recommend the "MATE" desktop.

note: you will have to cut-and-paste this several times because "apt" eats commands, even with "-y" and even piped from "yes".

<pre>
# install MATE desktop
DEBIAN_FRONTEND=noninteractive apt -y install ubuntu-mate-core ubuntu-mate-desktop ubuntu-mate-themes
# install Cinnamon desktop
DEBIAN_FRONTEND=noninteractive apt -y install cinnamon
# install KDE desktop
DEBIAN_FRONTEND=noninteractive apt -y install kubuntu-desktop
# install Lxqt desktop
DEBIAN_FRONTEND=noninteractive apt -y install lxqt
# install Xfce4 desktop
DEBIAN_FRONTEND=noninteractive apt -y install xfce4
</pre>

== install ROOT ==

Please install ROOT per instructions at https://root.cern.ch.

NOTE1: The ROOT package available from Ubuntu repositories is severely out of date and cannot be used with MIDAS and ROOTANA. ### DO NOT DO THIS! apt-get install root-system

NOTE2: as of 2017-Jan-09, ROOT binary kits for Ubuntu do not work (use GCC 5 instead of GCC6), build from source instead.

== Install x2go ==

KO - is this still needed? does it cause any security problems?

x2go instructions, thanks to Art O.

<pre>
add-apt-repository ppa:x2go/stable
apt-get update
apt-get install x2goserver x2goserver-xsession
</pre>

== enable root login from ladd00/daq00 ==
<pre>
ssh localhost
CTRL-C
/bin/cp ~root/git/scripts/etc/authorized_keys ~root/.ssh/
</pre>

== disable ssh access from outside of TRIUMF ==

to stop ssh login spam, disable ssh access from outside of TRIUMF. this can be done by requesting a firewall block through the helpdesk or by local firewall rule:

<pre>
echo iptables -I INPUT ! -s 142.90.0.0/255.255.0.0 -p tcp --dport 22 -j REJECT >> /etc/rc.local
/etc/rc.local
</pre>

== install smart-status ==
<pre>
ln -s ~/git/scripts/smart-status/smart-status.perl ~root/
</pre>

== enable boot menu and boot messages ==

This will enable the grub menu (with a 10 sec timeout) and
replace black screen with exciting linux boot messages.

* emacs -nw /etc/default/grub
<pre>
GRUB_DEFAULT=0
#GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="vga=769 video=640x480"
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX=""
#GRUB_GFXMODE=640x480
</pre>
* update grub config:
<pre>
grub-mkconfig -o /boot/grub/grub.cfg
</pre>

== reboot ==

this completes installation of the base system.

following sections modify basic ubuntu to fix known problems and to enable special stuff.

= Enable automatic updates =

<pre>
apt install unattended-upgrades
cd ~/git/scripts
git pull
/bin/cp -v etc/99apt-conf-ko /etc/apt/apt.conf.d/
apt-config dump | grep Unattended
</pre>

Following is obsolete:

* emacs -nw /etc/apt/apt.conf.d/50unattended-upgrades
** uncomment in Allowed-Origins "-security" and "-updates"
** add in Allowed-Origins: "Google LLC:stable";
** uncomment/add: "Unattended-Upgrade::Mail "root";
* emacs -nw /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
</pre>
* test: unattended-upgrade --dry-run -v

NOTE: update-on-shutdown is disabled.

NOTE: there is no update-on-boot, but:

NOTE: if machine was off for a long time, the systemd update timer would have expired and it will fire soon after reboot, causing an automatic update run. this is unwanted, and there is no fix or workaround for it. K.O. June-2023.

= Fix bpool is full (obsolete) =

THIS IS CAUSED BY OBSOLETE PACKAGE zsys. PLEASE: apt remove zsys

!!! only if ROOT on ZFS !!!

There is an error in the zsys package that causes bpool to run out of space,
see [[#Ubuntu zsys]] for more details.

To fix:
<pre>
cd ~/git/scripts
git pull
cp etc/zsys.conf /etc/
zsysctl service reload
zsysctl service gc
zpool list bpool
zfs list bpool
df /boot
</pre>

= IPMI instructions =

IPMI is the board management hardware on Supermicro and other server motherboards. This includes hardware sensors - fan rotation speed, temperatures and power supply voltages.

<pre>
apt-get install ipmitool
systemctl enable ipmievd
systemctl restart ipmievd
</pre>

Run:
* ipmitool sel list ### event list
* ipmitool sel elist ### event list
* ipmitool sel clear ### clear event list (if it becomes full)
* ipmitool sensor ### report hardware sensors

= move /home/wheel =

note: this MUST be done if ZFS root and NIS/autofs with /home.

Default location of wheel's home directory will collide with autofs /home, it has to be moved,
for example to /wheel.

<pre>
# logout from the wheel user
# go to another computer
ssh root@daqubuntuxxx
zfs list | grep wheel ### identify zfs name wheel_xxxxxx
zfs set mountpoint=/wheel rpool/USERDATA/wheel_hm8fzh
emacs -nw /etc/passwd ### change wheel's home directory from /home/wheel to /wheel
su - wheel ### check that user wheel still works
</pre>

This will break wheel's ability to run snap programs, such as firefox, install chrome as listed below.

= enable NIS (ubuntu 22.04, debian 11) =

<pre>
apt -y install rpcbind nis
echo DAQ-NIS >> /etc/defaultdomain
echo ypserver daq00.triumf.ca >> /etc/yp.conf
systemctl enable ypbind.service
systemctl restart ypbind.service
systemctl status ypbind.service
ypwhich -m
</pre>

enable ypserv:

<pre>
sed -i s/NISSERVER=false/NISSERVER=slave/ /etc/default/nis
/usr/lib/yp/ypinit -s daq00
echo ypserver localhost >> /etc/yp.conf
sed -i "s/ypserver .*/ypserver localhost/" /etc/yp.conf
systemctl enable ypserv
systemctl restart ypserv
systemctl restart ypbind
</pre>

edit /etc/nsswitch.conf to read:

<pre>
# begin get data from nis
passwd: files nis
group: files nis
shadow: files nis
automount: files nis
netgroup: files nis
# end get data from nis
</pre>

enable hourly update of nis maps:

<pre>
mkdir ~root/git
cd ~root/git
git clone http://daq00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>

If this is a new machine, then on the master NIS node (daq00), add this new node to /etc/netgroup, and update NIS maps (cd /var/yp; make)

= enable NIS (ubuntu 20.04) =

* apt-get -y install portmap nis ### will ask for NIS domain (DAQ-NIS)
* dpkg-reconfigure nis ### reconfigure if already installed
* ypwhich -m
* edit /etc/default/nis
** set "NISSERVER=slave"
** Ubuntu LTS 20.04, check that "YPBINDARGS=" is blank, remove "-no-dbus" if it is there
* #edit /etc/yp.conf, comment-out everything, add "domain DAQ-NIS server localhost"
* edit /etc/yp.conf, comment-out everything, add "ypserver localhost"
* /usr/lib/yp/ypinit -s daq00
* systemctl enable nis
* systemctl restart nis
* ypwhich
* ypwhich -m
* ypcat -k passwd
* vi /etc/nsswitch.conf ### add the automount line, modify the passwd, group and shadow lines to read this:
<pre>
# begin get data from nis
passwd: files nis
group: files nis
shadow: files nis
automount: files nis
netgroup: files nis
# end get data from nis
</pre>
* enable hourly update of NIS maps
<pre>
mkdir ~root/git
cd ~root/git
git clone http://ladd00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>
* ### NOT NEEDED sudo vi /etc/idmapd.conf ### add line: "Domain = triumf.ca"

= enable autofs =

<pre>
apt -y install autofs
systemctl enable autofs
systemctl restart autofs
ls -l /home/olchansk ### test autofs, check file owner is correct
</pre>

= enable NFS server =

<pre>
apt install nfs-kernel-server
#edit /etc/exports
systemctl enable nfs-server
systemctl restart nfs-server
</pre>

= NIS master =

notes for setting up the NIS master

== wheel user ==

"wheel" is the default administrative user. We do not want it's password exported to NIS (encrypted password hash is world visible) and we do not want it's home directory exported to NFS (~wheel/.ssh is world visible and potentially writable: anybody can change ~wheel/.ssh/authorized_keys).

* move wheel's home directory from /home/wheel to /wheel (see special section about this)
* change wheel's UID and GID from 1000 to a value below MINUID in /var/yp/Makefile

== coherent uids ==

we do not want system accounts defined in /etc/passwd of the NIS master
to be included in the NIS map "passwd". this causes trouble on NIS clients
where newly installed packages fail to create local system users because same
user already exists in NIS.

This is controlled by MINUID in /var/yp/Makefile.

Historical TRIUMF uids start from around 200, but several clusters do not have any historic TRIUMF uids below 500 and MINUID is set to:
* DAQ-NIS: MINUID=200
* ISAC-NIS: MINUID=500
* TITAN-NIS: MINUID=500
* MUSR-NIS: MINUID=500
* TIG-NIS: MINUID=500 (100 on SL6 mother8pi)

Ubuntu 20 has two programs to create users:
* adduser - creates new users with UID 1000 and up as specified in /etc/adduser.conf. No problems here.
* adduser --system - creates new system users with UID 100 and up as specified in /etc/adduser.conf. No problems here.
* useradd - creates new users with UID 1000 and up as specified in /etc/login.defs. No problems here.
* useradd --system - creates new system users with UID 999 and down (read "man useradd", section at the end about SYS_UID_MAX). This collides with NIS MINUID, these system users will be included in the NIS map and cause trouble.

This problem cannot be fixed, SYS_UID_MIN, SYS_UID_MAX and UID_MIN in /etc/login.defs do not seem
to have any effect on UIDs chosen by "useradd --system". (tested on Ubuntu LTS 20.04).

So far only these system accounts seem to be affected by this:
* systemd-coredump
* ganglia

To fix:
* run "sort -r -n -t: -k3 /etc/passwd" to identify the last unused system user uid (range 100..200)
* run "sort -r -n -t: -k3 /etc/group" to identify the last unused system user gid (range 100.200)
* systemd-coredump: manually change UID and GID (package systemd-coredump is usually not installed)
* ganglia: same thing, then change ownership on all ganglia files.

Also read systemd author's opinion on system vs user UIDs:
https://github.com/systemd/systemd/issues/4850#issuecomment-265698275

= Fix systemd-logind NIS breakage =

!!! THIS IS NOT NEEDED FOR UBUNTU LTS 20.04 !!!

there is a delay in ssh logins for normal users. "ssh -v" shows the delay is after "pledge...". this
fix removes the delay.

systemd developers think that we should not use NIS and made sure there are
problems if we do. To give them credit, they do offer a workaround. Read this:
https://github.com/poettering/systemd/commit/695fe4078f0df6564a1be1c4a6a9e8a640d23b67

<pre>
mkdir /etc/systemd/system/systemd-logind.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-logind.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-logind.service
</pre>

= Fix systemd-udevd NIS breakage =

see same problem as above with udev getting stuck. ubuntu lts 20.04.

<pre>
mkdir /etc/systemd/system/systemd-udevd.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-udevd.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-udevd.service
</pre>

= Configure USB device permissions =

Configure USB device permissions for user access to USB-serial devices, Altera USB Blaster, etc.

* create file /etc/udev/rules.d/99-usb-chmod.rules with this contents:

<pre>
emacs -nw /etc/udev/rules.d/99-usb-chmod.rules
ACTION=="add", SUBSYSTEM=="usbmisc", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /dev/%c"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /proc/%c"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVICE}"
ACTION=="add", ENV{PHYSDEVBUS}=="usb-serial", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVPATH}=="/class/tty/ttyS*", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyUSB*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyACM*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyS*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", DEVPATH=="*video*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
</pre>

* reload udev rules: udevadm control --reload-rules
* apply new permissions: udevadm trigger --action=add
* watch udev activity: udevadm monitor -p

= Configure lightdm display manager =

* enable it
<pre>
echo lightdm | dpkg-reconfigure -fteletype lightdm
systemctl disable gdm
systemctl disable sddm
systemctl enable lightdm
</pre>

* make the MATE desktop as default
<pre>
cd ~root/git/scripts/
git pull
/bin/cp -v etc/lightdm_default_mate.conf /etc/lightdm/lightdm.conf.d/
</pre>

* enable login by NIS users
<pre>
/bin/cp -v etc/lightdm_enable_nis_login.conf /etc/lightdm/lightdm.conf.d/
</pre>

* restart lightdm
<pre>
systemctl stop gdm
systemctl restart lightdm
</pre>

= Install libpng12.so.0 =

Quartus 16 needs libpng12:

<pre>
wget http://mirrors.kernel.org/ubuntu/pool/main/libp/libpng/libpng12-0_1.2.54-1ubuntu1_amd64.deb
dpkg --install libpng12-0_1.2.54-1ubuntu1_amd64.deb
</pre>

= Install google-chrome =

<pre>
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
dpkg -i google-chrome-stable_current_amd64.deb
</pre>

confirm autoupdate is enabled, observe dl.google.com is present in the list of repositories:
<pre>
apt update
...
Get:5 https://dl.google.com/linux/chrome/deb stable/main amd64 Packages [1,094 B]
...
</pre>

FOLLOWING IS OBSOLETE:

Instructions from here:
https://www.ubuntuupdates.org/ppa/google_chrome?dist=stable

<pre>
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-tmp.list'
apt update
apt install google-chrome-stable
/bin/rm -f /etc/apt/sources.list.d/google-tmp.list
</pre>

= Install amanda client =

ONLY ONE MACHINES THAT HOST HOME DIRECTORIES

* apt install amanda-client
* edit /etc/amandahosts
<pre>
amanda.triumf.ca amanda amdump
</pre>
* check permissions on /etc/amandahosts:
<pre>
root@daq00:/var/log/amanda# ls -l /etc/amandahosts
-rw------- 1 backup backup 49 Jan 27 10:48 /etc/amandahosts
</pre>
* fix if needed: chown backup.backup /etc/amandahosts; chmod a= /etc/amandahosts; chmod u=wr /etc/amandahosts
* edit /etc/amanda-security.conf, add this line:
<pre>
runtar:gnutar_path=/usr/bin/tar
</pre>

On the amanda machine:

* in amanda disklist, use dump type "bsdtcp-comp-user-tar"
* su - amanda and run amcheck -c daily daq00
<pre>
-bash-4.1$ amcheck -c daily daq00

Amanda Backup Client Hosts Check
--------------------------------
Client check: 1 host checked in 0.092 seconds. 0 problems found.

(brought to you by Amanda 3.3.7p1.git.685ff76d)
</pre>

= Enable rc.local =

For reasons unknown, Ubuntu LTS 20.04 does not enable /etc/rc.local. Do this:

<pre>
cd ~/git/scripts
git pull
cp -n -v etc/rc.local /etc/
chmod a+rx /etc/rc.local
cp etc/rc-local.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable rc-local
systemctl start rc-local
systemctl status rc-local
</pre>

= Remove unwanted packages =

<pre>
apt remove zsys # broken, do not use
apt remove sddm # login manager
apt remove avahi-daemon avahi-autoipd # not sure what it does, observed using 100% CPU
apt remove modemmanager # probes all serial ports to see if it's a modem
</pre>

= Disable unwanted services =

<pre>
systemctl disable mpd
systemctl disable snapd
systemctl disable ModemManager
systemctl --global mask tracker-extract-3.service
systemctl --global mask tracker-miner-fs-3.service
systemctl daemon-reload
</pre>

= Disable sleep and suspend =

note: we see some computers randomly shutdown or go to sleep, log files indicates the "sleep" or "suspend" button was pushed by user, but no such buttons actually exist. this is the fix for this:

<pre>
systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target systemd-suspend.service systemd-hybrid-sleep.service
</pre>

= Enable crontab @reboot for MIDAS =

startup scripts have a bug - cron @reboot entries for normal users can run before autofs is ready, so if the home directory is on autofs/NFS, it cannot be accessed and the cron job fails. If MIDAS is supposed to be started by cron @reboot, it will not start (there *will* be an error message in /var/log/cron).

<pre>
mkdir /etc/systemd/system/cron.service.d
echo -e "[Unit]\nAfter=ypbind.service autofs.service\n" > /etc/systemd/system/cron.service.d/local.conf
systemctl daemon-reload
systemctl cat cron.service
</pre>

Explore the systemd dependency tree using "systemctl list-dependencies" maybe with "--all".

Visualize the exact boot sequence from previous boot: "systemd-analyze plot > xxx.svg", look at the svg file using a web browser.

Crontab entry to start midas: (install in the midas user crontab, not root crontab)

<pre>
su - midasuser
crontab -l
#@reboot /bin/bash -l -c "/home/trinat/bin/start-daq-applications"
#@reboot /bin/tcsh -c "/home/trinat/bin/start-daq-applications"
</pre>

= Install apache httpd proxy for midas and elog =

This will configure the HTTPS/SSL certificate using "certbot" and "letsencrypt" and configure an HTTPS web server using apache2.

First, configure apache2:

* execute these commands:
<pre>
apt -y install apache2
cd /etc/apache2
</pre>
* create new file conf-available/ssl-daq14.conf # use actual hostname instead of daq14
<pre>
SSLSessionCache shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout 300
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
SSLCryptoDevice builtin
</pre>
* create new file sites-available/daq14-ssl.conf # use actual hostname instead of daq14
<pre>
<IfModule mod_ssl.c>
<VirtualHost *:443>
ServerName daq14.triumf.ca
DocumentRoot /var/www/html
ErrorLog /var/log/apache2/daq14.log
SSLEngine on
# note SSLProtocol, SSLCipherSuite and some other settings are overwritten by /etc/letsencrypt/options-ssl-apache.conf
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SEED:!IDEA:!RC4
## use port specified in elogd.cfg
#ProxyPass /elog/ http://localhost:8082/ retry=1
## use mhttpd port
#ProxyPass / http://localhost:8080/ retry=1
Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"
<Location />
SSLRequireSSL
AuthType Basic
AuthName "DAQ password protected site"
Require valid-user
# create password file: touch /etc/apache2/htpasswd
# to add new user or change password: htpasswd /etc/apache2/htpasswd username
AuthUserFile /etc/apache2/htpasswd
</Location>
</VirtualHost>
</IfModule>
</pre>
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* enable ssl module
* enable new configurations
<pre>
a2enmod ssl
a2enmod headers
a2enmod proxy
a2enmod proxy_http
a2enconf ssl-daq14
a2ensite daq14-ssl
</pre>
* disable default ssl sites
<pre>
a2dissite 000-default-le-ssl
a2dissite 000-default
ls -l /etc/apache2/sites-enabled/ ### should show only daq14-ssl.conf
</pre>
* check that there are no syntax problems
<pre>
apache2ctl configtest
</pre>
* enable and start apache2:
<pre>
systemctl enable apache2
systemctl restart apache2
systemctl status apache2
</pre>
* apache2 may fail to start, look in /var/log/apache2/error.log and /var/log/apache2/daq14.log
* if it says "Failed to configure ... certificate", proceed to the step for setting certbot.
* try to access https://daq14.triumf.ca
** you should see a complaint about self-signed certificate
** you should see a request for password (do not login yet)
** if you get "connection refused", HTTPS port 443 may need to be enabled in the local firewall, look at documentation for ufw.
Second, configure certbot:

(Note: as of 2018-01-18 certbot requires use of http port 80 to get the initial https certificate,
renewal can continue to use the https port 443)

(Note: as of 2019-01-?? certbot requires use of port 80 for renewals)

(Note: unsurprisingly, this requires outside access to connect with letsencrypt, so won't work if PC is only accessible from on-site network)

* check that port 80 is not used by anything:
* netstat -an | grep LISTEN | grep ^tcp | grep 80
* lsof -P | grep -i tcp | grep LISTEN | grep 80
* if lsof reports that apache2 is listening on port 80, follow the apache2 instructions above (remove "listen 80" from apache2.conf

* install certbot (if necessary open tcp port 80 in the firewall, see documentation for ufw):
<pre>
apt install certbot python3-certbot-apache
certbot certonly --standalone --installer apache
</pre>
* then answer questions:
* "activate HTTPS for daq14.triumf.ca" - say ok
* "enter email address" - enter your own email address
* "please read terms..." - read the terms and say "agree"
* it will take a few moments...
* "congratulations..." - say ok.
<pre>
certbot install --apache --cert-name daq14.triumf.ca
</pre>
* then answer questions:
* "choose redirect..." - say "1" (no redirect)
* look inside /etc/apache2/sites-enabled/daq14-ssl.conf to see that SSLCertificateFile & co point to certbot certificates in
/etc/letsencrypt/live/daq14.triumf.ca/
* to check current renewal and to update the certbot config file in /etc/letsencrypt/renewal, run this:
<pre>
certbot renew --standalone --installer apache --force-renewal
</pre>

NOTE: this certificate will expire in 3 months, automatic renewal should work with current version of certbot

Third, activate password protection:

* as shown in the config file above, create password file and initial user: (replace "midas" with specific username)
<pre>
touch /etc/apache2/htpasswd
htpasswd /etc/apache2/htpasswd midas
</pre>

* restart apache2
<pre>
systemctl restart apache2
systemctl status apache2
</pre>

From here:
* enable proxy for MIDAS mhttpd - uncomment redirect in the config file above
* enable proxy for ELOG - ditto
<pre>
a2enmod proxy
a2enmod proxy_http
apache2ctl configtest
systemctl restart apache2
</pre>

From here:
* enable proxy for MIDAS mhttpd - uncomment redirect in the config file above
* enable proxy for ELOG - ditto
<pre>
a2enmod proxy
a2enmod proxy_http
apache2ctl configtest
systemctl restart apache2
</pre>
* try accessing MIDAS https://daq14.triumf.ca/ (make sure mhttpd is running)
* if it's not working, check odb setting FIXME!
* try accessing ELog https://daq14.triumf.ca/elog/ (make sure elogd is running)
* if it's not working, check elogd.cfg file and make sure
<pre>
SSL = 0
</pre>

NOTE: if certbot fails with errors about 'module' object has no attribute 'pyopenssl',
try this: pip install requests==2.6.0

= Enable elog PDF preview =

see https://stackoverflow.com/questions/52998331/imagemagick-security-policy-pdf-blocking-conversion

* xemacs -nw /etc/ImageMagick-6/policy.xml
* remove this section at the end:
<pre>

<policy domain="coder" rights="none" pattern="PS" />
<policy domain="coder" rights="none" pattern="PS2" />
<policy domain="coder" rights="none" pattern="PS3" />
<policy domain="coder" rights="none" pattern="EPS" />
<policy domain="coder" rights="none" pattern="PDF" />
<policy domain="coder" rights="none" pattern="XPS" />
</pre>

= Install Jupyter notebook =

<pre>
From https://jupyter.org/install
apt install python3-pip
pip install jupyterlab
pip install notebook
~/.local/bin/jupyter notebook
watch the http://localhost:8888 URL that it printed
say "no" to offer to start firefox (it will not work!)
URL is: http://localhost:8888/tree?token=xxx
from the machine where you are running the web browser (i.e. google-chrome), run (replace trinat@trinatdaq with the username and machine name where you started jupyter)
open a new shell and run: ssh -v trinat@trinatdaq -L 8888:localhost:8888
in the web browser, open http://localhost:8888
this gives us the login page
in the password or token entry field, put the token from the "tree?token=xxx" above (printed by jupyter on startup)
push button "login"
jupyter page should open with the list of files in the trinat home directory
congratulate Brian with full success
</pre>

= Install ZFS quota report =

If there are any ZFS volumes, install script to report disk and quota usage

<pre>
cd ~/git/scripts/quotareport
git pull
mkdir /var/www/html/zfsquotareport
cp -pv ~/git/scripts/quotareport/sorttable.js /var/www/html/zfsquotareport/
ln -s $PWD/zfsquotareport.perl /etc/cron.daily/
touch /etc/crontab
</pre>

If httpd is configured to redirect "/" to MIDAS mhttpd:
* add following to /etc/apache2/sites-enabled/xxx-ssl.conf in front of "ProxyPass / ..."
* run "systemctl reload apache2"
<pre>
## do not proxy zfs quota report directory
ProxyPass /zfsquotareport/ !
</pre>

= Install PHP =

* apt install php libapache2-mod-php
* systemctl restart apache2
* create /var/www/html/info.php
<pre>
<?php

phpinfo();
</pre>
* open https://daq00.triumf.ca/info.php

= Configure TRIUMF printers =

<pre>
systemctl stop cups
systemctl disable cups
echo "ServerName printers.triumf.ca" > /etc/cups/client.conf
lpstat -a
</pre>

= Enable core dumps =

By default, Ubuntu LTS 20.04 installs the apport package
which disabled core dumps from user applications. (google it up!).
It is not meant to do this and documentation claims that
it is not installed and not enabled by default. Oh, well...

<pre>
apt remove apport
apt autoremove ### will remove apport-symptoms and a few other packages
</pre>

After this, core dumps are written to file "core" in the current directory.
See /proc/sys/kernel/core_pattern and /proc/sys/kernel/core_uses_pid.

Enable core dump file names to include process id, add following to /etc/rc.local

<pre>
echo 1 > /proc/sys/kernel/core_uses_pid
</pre>

= Enable debugger =

By default, Ubuntu LTS 20.04 does not permit debugger to attach and debug
already running programs. To enable it, add following to /etc/rc.local

<pre>
echo 0 > /proc/sys/kernel/yama/ptrace_scope
</pre>

= Disable Ubuntu Pro nag =

If "apt upgrade" requests Ubuntu Pro or esm-apps, disable the nag:
<pre>
/bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf
</pre>

= Update packages =

* apt-get update # update package list
* apt-get dist-upgrade # install updated packages and update "kept back" packages
* apt-get autoremove # remove packages that apt thinks should be removed

= Finish installation =

Congratulations. There is nothing more to do!

* reboot
<pre>
shutdown -r now
</pre>

= Install ZFS =

!!! after installing all the packages, after updating the system, after updating the linux kernel, after rebooting into latest kernel !!!

<pre>
apt-get install zfsutils-linux
</pre>

Follow generic ZFS instructions: [[ZFS]]

= Update to new version of Ubuntu =

<pre>
vi /etc/update-manager/release-upgrades # set "Prompt=normal"
do-release-upgrade
</pre>

Update Ubuntu LTS 20.04 to LTS 22.04:

<pre>
apt remove zsys
</pre>

== daqubuntu ==

<pre>
# reboot to clear out all updates
# vi /etc/update-manager/release-upgrades # set "Prompt=normal"
# do-release-upgrade -c
Checking for a new Ubuntu release
New release '22.04 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
# do-release-upgrade
...
say yes...
...
login.defs, say "Y" (erase local changes, use packaged version)
/etc/systemd/resolved.conf, say "Y" (same as above)
firefox snap, say yes
unable to reach snap store, say "skip"
/etc/gmond.conf, say "Y"
/var/yp/Makefile, say "install the package maintainer's version"
/etc/ypserv.conf, same thing
/etc/ypserv.securenets, same thing
/etc/default/nis, same thing
/etc/speech-dispatcher/modules/mary-generic.conf, same thing
/etc/apt/apt.conf.d/50unattended-upgrades, same thing
...
278 packages are going to be removed, say yes
...
restart required, say yes
...
no ping... yes ping...
...
ssh daqubuntu, ok
apt update, fail, DNS does not work, "host security.ubuntu.com" does not resolve.
fix resolver per https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Disable_NetworkManager
apt update, apt upgrade now works, 0 packages to update
NIS does not work.
</pre>

== midm9a ==

<pre>
login.defs
firefox snap
gmond.conf
ypserv
/etc/default/nis
unattended-upgrades
amanda-security.conf
remove obsolete (no)
reboot
configure dns
reenable nis
</pre>

== daq17 ==

<pre>
firefox snap
imagemagick policy.xml
gmond.conf
chrony.conf
/var/yp/Makefile
ypserv.conf
ypserv.securenets
/etc/default/nis
50unattended-upgrades
</pre>

== daq00 ==

per https://serverpilot.io/docs/how-to-upgrade-ubuntu-20.04-to-22.04/

<pre>
do-release-upgrade -f DistUpgradeViewNonInteractive
</pre>

if it exists "too soon" without doing anything, run it without "-f xxx", most likely it does not like something about this machine. in case of daq00 it did not like how the EFI partitions were mounted. after fixing it, non-interactive upgrade was successful.

== isdaq08 ==

* prepare
<pre>
cd ~/git/scripts
git pull
cd ~
apt -y install debsums
</pre>
* check for modified config files that make upgrade unhappy, deal with all files reported by debsums.
<pre>
root@isdaq08:~# debsums -ce
/etc/ganglia/gmond.conf
/etc/yp.conf
/etc/apt/apt.conf.d/10periodic
root@isdaq08:~#
</pre>
* restore original /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
</pre>
* apt remove ganglia-monitor
* apt remove nis
* "debsums -ce" is now empty

Run the upgrade:

* do-release-upgrade -f DistUpgradeViewNonInteractive

Post upgrade:

* configure DNS
* apt -y install linux-generic-hwe-22.04
* /bin/cp -v ~/git/scripts/etc/99apt-conf-ko /etc/apt/apt.conf.d/ # restore nightly updates
* /bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf # remove the ubuntu-pro nag
* install missing packages
* restore ganglia
* restore nis
* check zpool status, may need zpool upgrade
* reboot

= Upgrade to new version of Debian =

https://www.debian.org/releases/bookworm/amd64/release-notes/ch-upgrading.en.html

== 32-bit VME processor Debian 11 to 12 ==

* cd git/scripts; git pull; cd ~
* apt update
* apt upgrade
* edit /etc/apt/sources.list
<pre>
deb http://deb.debian.org/debian/ bookworm main
#deb http://deb.debian.org/debian/ bullseye main
#deb-src http://deb.debian.org/debian/ bullseye main
</pre>
* apt update
* apt upgrade --without-new-pkgs
* apt full-upgrade
* apt list '~c'; apt purge '~c' # purge left-over config files [residual-config]
* reboot

= Ubuntu package manager =

* apt-get install xxx # install package xxx
* apt-get update
* apt-get upgrade
* apt-get dist-upgrade
* apt-get autoremove # remove automatically installed packages required by a removed package
* apt-get remove xxx # remove package xxx
* apt-cache search . # list all available packages
* apt-cache show "." | grep ^Package # list al available packages
* apt-cache madison root-system # show all available versions of package root-system
* apt list # list all installed packages
* dpkg --listfiles libpng16-16 # list all files from this package
* apt list --installed # list all installed packages
* dpkg -S /bin/bash # what package provides this file?
* dpkg -L bash # what files provided by this package?
* debsums -ce # show modified config files
* apt-config dump # show apt configuration

= Ubuntu zsys =

NOTE: DO NOT USE ZSYS, see https://github.com/ubuntu/zsys/issues/218 and https://github.com/ubuntu/zsys/issues/230

* manual removal of old snapshots
<pre>
zsysctl show
zsysctl state remove xy69ye -s
zsysctl state remove xy69ye
zsysctl state remove xy69ye -u wheel
</pre>
* apt remove zsys

NOTE: old zsys snapshots must be cleaned manually, "zsysctl state remove xxx --system" is broken and does not remove user data snapshots

* manages system snapshots
* documentation: https://github.com/ubuntu/zsys
* documentation: (go to next article via link "newer" at the bottom) https://didrocks.fr/2020/05/21/zfs-focus-on-ubuntu-20.04-lts-whats-new/
* ubuntu 20.04 bug, too many snapshots cause /boot to become full and updates fail. https://github.com/ubuntu/zsys/issues/155
* solution: use custom /etc/zsys.conf, limit number of snapshots to 10, see trinatdaq:/etc/zsys.conf
* zsys commands:
<pre>
update-grub # list of all snapshots, errors if some snapshots are broken
zsysctl state remove lnc0k7 --system # remove snapshot
xemacs -nw /etc/zsys.conf; zsysctl service reload; zsysctl service gc # cause gc to run with new settings in zsys.conf
zfs list -r -t snapshot -o name,used,referenced,creation bpool/BOOT # list snapshots
zsysctl show # show snapshots
</pre>

= Ubuntu cloning =

to clone a ubuntu image:

<pre>
cd /nfsroot/lxcpet
emacs -nw etc/hostname ### change hostname
emacs -nw etc/mailname ### change hostname (debian 11)
emacs -nw etc/defaultdomain ### change the NIS domainname
emacs -nw etc/yp.conf ### change the NIS server
cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
emacs -nw root/.ssh/authorized_keys ### update root ssh keys
</pre>

= Ubuntu boot loader =

== boot from ZFS ==

* use UEFI boot with syslinux, see here: https://daq.triumf.ca/DaqWiki/index.php/SLinstall#Configure_UEFI_boot
* apt install zfs-initramfs
* update-initramfs -v -u
* ZFS structure:
<pre>
root@daq00:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 147G 1.62T 96K /
rpool/ROOT 17.8G 1.62T 96K none
rpool/ROOT/ubuntu_00aaaa 17.8G 1.62T 6.22G /
</pre>
* copy OS image to rpool/ROOT/ubuntu_00aaaa
* zfs set mountpoint=/ rpool
* zfs set mountpoint=none rpool/ROOT
* zfs set mountpoint=/ rpool/ROOT/ubuntu_00aaaa
* zfs get all | grep mountpoint
<pre>
rpool mountpoint / local
rpool/ROOT mountpoint none local
rpool/ROOT/ubuntu_00aaaa mountpoint / local
</pre>
* in linux kernel command line (syslinux.cfg), set "root=" to "root=ZFS=rpool/ROOT/ubuntu_00aaaa"

== boot from ZFS mirror ==

=== setup the EFI partitions ===

* assuming /dev/sdb is already setup for EFI boot, setup /dev/sda the same way:
* partition the second boot disk same as first boot disk:
<pre>
root@grsnis01:~# gdisk -l /dev/sdb
Found valid GPT with protective MBR; using GPT.
Number Start (sector) End (sector) Size Code Name
1 2048 1050623 512.0 MiB EF00 EFI system partition
2 1050624 3907029134 1.8 TiB 8300 Linux filesystem
root@grsnis01:~#
</pre>
* mkfs.msdos /dev/sdX1
* create mount points
<pre>
mkdir /boot/efi-sda
mkdir /boot/efi-sdb
</pre>
* add to /etc/fstab
<pre>
/dev/sda1 /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
/dev/sdb1 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
</pre>
* mount -a
* df | grep boot
<pre>
root@grsnis01:~# df | grep boot
/dev/sdb1 523248 98100 425148 19% /boot/efi-sdb
/dev/sda1 523248 4 523244 1% /boot/efi-sda
</pre>
* copy boot files to new boot disk
* cd /boot/efi-sdX; rsync -av . /boot/efi-sdY
* set BIOS to boot from "UEFI Hard drive", disable legacy boot (except for booting from USB key in legacy mode)
* if using UEFI boot syslinux per these instructions, linux kernel update has to be done manually:
* run ~/git/scripts/etc/update_efi_mirror.perl, follow instructions that it prints.

=== setup zfs partitions ===

use partitions compatible with Ubuntu "install on ZFS"

* gdisk "o" to create new GPT partition table
* gdisk "n" +512M ef00 to create EFI partition
* gdisk "n" +2G 8200 to create linux swap partition (not used)
* gdisk "n" +2G BE00 to create ZFS bpool partition
* gdisk "n" xxx BF00 create ZFS rpool partition

<pre>
# gdisk -l /dev/sda
Number Start (sector) End (sector) Size Code Name
1 2048 1050623 512.0 MiB EF00 EFI System Partition
2 1050624 5244927 2.0 GiB 8200
3 5244928 9439231 2.0 GiB BE00
4 9439232 234441614 107.3 GiB BF00
root@midm9a:~#
</pre>

=== setup zfs mirror ===

* see here: https://daq.triumf.ca/DaqWiki/index.php/ZFS#Convert_pool_from_single_to_mirror
<pre>
root@grsnis01:~# ls -l /dev/disk/by-id/ata*part2
lrwxrwxrwx 1 root root 10 Feb 19 16:47 /dev/disk/by-id/ata-WDC_WDS200T2B0A-00SM50_205007801081-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Feb 19 16:47 /dev/disk/by-id/ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 -> ../../sdb2

root@grsnis01:~# zpool status
pool: rpool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 ONLINE 0 0 0

errors: No known data errors

root@grsnis01:~# zpool attach rpool ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 /dev/disk/by-id/ata-WDC_WDS200T2B0A-00SM50_205007801081-part2

root@grsnis01:~# zpool status
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Feb 19 16:54:39 2021
12.6G scanned at 3.16G/s, 1.02G issued at 262M/s, 12.6G total
1.02G resilvered, 8.09% done, 0 days 00:00:45 to go
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801081-part2 ONLINE 0 0 0 (resilvering)

errors: No known data errors
</pre>
* wait
<pre>
root@grsnis01:~# zpool status
pool: rpool
state: ONLINE
scan: resilvered 12.7G in 0 days 00:00:40 with 0 errors on Fri Feb 19 16:55:19 2021
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801081-part2 ONLINE 0 0 0

errors: No known data errors
</pre>

== maintenance commands ==

* update-initramfs -v -u
* grub-install /dev/sda

= Convert from single to dual mirrored ZFS SSD =

Assuming Ubuntu LTS 22.04 with "instal on ZFS" option, we will
add a second SSD, configure ZFS to use both SSDs in mirrored
configuration and setup grub to boot from either SSD. This
is intended to create a full redundant system where failure
of either SSD does not break the system.

* identify first SSD
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* connect second SSD of identical size
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
/dev/sdb WD Blue SA510 2.5 250GB 22243Z803852 25 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* if second SSD is not autodetected, reboot
* Clone partition table automatically
If both SSDs are identical size, use this simpler method of duplicating the partition table:
<pre>
root@midm9b:~# sfdisk -d /dev/sda > part_table
root@midm9b:~# grep -v ^label-id part_table | sed -e 's/, *uuid=[0-9A-F-]*//' | sfdisk /dev/sdb
</pre>
The grep and sed in the second command are there to prevent disk ID and partition IDs from being cloned. Alternatively the part_table file can be edited manually to remove the label-id line and the uuid entries from the individual partitions.

* Clone partition table manually (e.g. for different size disks)
* list partition table of first SSD:
<pre>
root@midm9b:~# fdisk -l /dev/sda
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>
* create identical partitions on second SSD, use sector numbers from above.
<pre>
root@midm9b:~# gdisk /dev/sdb
GPT fdisk (gdisk) version 1.0.8

Partition table scan:
MBR: not present
BSD: not present
APM: not present
GPT: not present

Creating new GPT entries in memory.

Command (? for help): n
Partition number (1-128, default 1):
First sector (34-488397134, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-488397134, default = 488397134) or {+-}size{KMGTP}: 1050623
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): ef00
Changed type of partition to 'EFI system partition'

Command (? for help): n
Partition number (2-128, default 2):
First sector (34-488397134, default = 1050624) or {+-}size{KMGTP}:
Last sector (1050624-488397134, default = 488397134) or {+-}size{KMGTP}: 5244927
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 8200
Changed type of partition to 'Linux swap'

Command (? for help): n
Partition number (3-128, default 3):
First sector (34-488397134, default = 5244928) or {+-}size{KMGTP}:
Last sector (5244928-488397134, default = 488397134) or {+-}size{KMGTP}: 9439231
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): be00
Changed type of partition to 'Solaris boot'

Command (? for help): n
Partition number (4-128, default 4):
First sector (34-488397134, default = 9439232) or {+-}size{KMGTP}:
Last sector (9439232-488397134, default = 488397134) or {+-}size{KMGTP}:
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): bf00
Changed type of partition to 'Solaris root'

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sdb.
The operation has completed successfully.
root@midm9b:~# fdisk -l /dev/sda /dev/sdb
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root

Disk /dev/sdb: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: EB251739-30C6-422F-A505-5887B5A0B603

Device Start End Sectors Size Type
/dev/sdb1 2048 1050623 1048576 512M EFI System
/dev/sdb2 1050624 5244927 4194304 2G Linux swap
/dev/sdb3 5244928 9439231 4194304 2G Solaris boot
/dev/sdb4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>
* identify second SSD partitions
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part3
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part3 -> ../../sda3
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 -> ../../sdb3
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
</pre>
* convert bpool from single disk to mirrored disk:
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~# zpool attach bpool 99e03dc0-7d4d-f24b-8fa1-f042b9f135db /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3
root@midm9b:~# zpool status bpool
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors
</pre>
* convert rpool
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
root@midm9b:~# zpool attach rpool f6fd54f8-3af7-b943-ae3d-a4e480537fb9 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4
root@midm9b:~# zpool status rpool
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Jan 20 19:40:45 2023
5.83G scanned at 664M/s, 2.92M issued at 332K/s, 9.11G total
0B resilvered, 0.03% done, no estimated completion time
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~#
</pre>
* wait for resilver to complete
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
scan: resilvered 9.65G in 00:00:36 with 0 errors on Fri Jan 20 19:41:21 2023
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
</pre>
* enable booting from second SSD: (instead of /dev/sda1, /dev/sdb1, use UUID=xxx)
<pre>
root@midm9b:~# mkfs.msdos /dev/sdb1
root@midm9b:~# mkdir /boot/efi-sda
root@midm9b:~# mkdir /boot/efi-sdb
root@midm9b:~# echo "/dev/sda1 /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1" >> /etc/fstab
root@midm9b:~# echo "/dev/sdb1 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1" >> /etc/fstab
root@midm9b:~# mount -a
root@midm9b:~# df -kl
Filesystem 1K-blocks Used Available Use% Mounted on
...
/dev/sda1 523244 13720 509524 3% /boot/efi
/dev/sdb1 523244 4 523240 1% /boot/efi-sdb
...
root@midm9b:~# rsync -av /boot/efi/ /boot/efi-sdb/
sending incremental file list
EFI/
...
root@midm9b:~# ls -l /boot/efi-sda
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~# ls -l /boot/efi-sdb
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~#
</pre>
* setup script to update grub on second SSD, it must be run manually after every kernel update
<pre>
root@midm9b:~# ln -s ~/git/scripts/etc/update_efi_grub.perl ~/
root@midm9b:~# ~/update_efi_grub.perl -u
EFI dir: /boot/efi-sda
/boot/efi-sda: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sda/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sda: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sda/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
EFI dir: /boot/efi-sdb
/boot/efi-sdb: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sdb/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sdb: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sdb/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
root@midm9b:~#
</pre>

= Disable NetworkManager =

NOTE: THIS IS BROKEN IN UBUNTU LTS 22.04

NetworkManager is useful for configuring dynamic
network interfaces, i.e. laptops that often move
between networks, or connect to multiple choice
of wifi networks, etc.

For machines with statically configured network interfaces,
NetworkManager is not necessary.

As it has been observed to become confused and observed
to malfunction when network links go up and down (it keeps
unnecessarily reconfiguring the ip address, etc), it can
be usefuil to disable it.

* list all network interfaces
<pre>
# /bin/ls -1 /sys/class/net/
enp0s31f6
lo
</pre>
* edit /etc/network/interfaces:
<pre>
rename enp0s31f6=eth0
auto eth0
iface eth0 inet static
address 142.90.120.94/19
gateway 142.90.100.18
</pre>
* statically configure systemd-resolved
** create /etc/systemd/resolved.conf.d/resolved.conf with this contents:
<pre>
[Resolve]
DNS=142.90.100.19
Domains=triumf.ca
</pre>
** systemctl restart systemd-resolved
** resolvectl
** systemd-analyze cat-config systemd/resolved.conf
* disable NetworkManager
<pre>
systemctl disable NetworkManager
</pre>
* reboot

= Configure ECC memory =

== Configure EDAC ==

* apt install edac-utils

=== Intel i3-2120 ===
<pre>
root@musr00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X9SCL/X9SCM
root@musr00:~# edac-ctl --status
edac-ctl: drivers not loaded.
</pre>

=== Intel E-2236 ===
<pre>
root@daq00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SCM-F
root@daq00:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq00:~# edac-util
edac-util: No errors to report.
root@daq00:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
</pre>
* check edac sysfs files (Intel)
<pre>
root@daq00:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_noinfo_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 max_location
-r--r--r-- 1 root root 4096 Jan 25 15:10 mc_name
drwxr-xr-x 2 root root 0 Jan 25 15:10 power
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank0
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank1
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank2
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank3
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank4
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank5
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank6
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank7
--w------- 1 root root 4096 Jan 25 15:10 reset_counters
-r--r--r-- 1 root root 4096 Jan 25 15:10 seconds_since_reset
-r--r--r-- 1 root root 4096 Jan 25 15:10 size_mb
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Jan 25 15:10 uevent
root@daq00:~#
</pre>

=== Intel E3-1270 v6 ===
<pre>
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --status
edac-ctl: drivers are loaded.
root@grsnis01:~# edac-util
edac-util: No errors to report.
root@grsnis01:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@grsnis01:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 max_location
-r--r--r-- 1 root root 4096 Feb 19 12:35 mc_name
drwxr-xr-x 2 root root 0 Feb 19 12:35 power
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank0
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank1
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank2
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank3
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank4
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank5
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank6
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank7
--w------- 1 root root 4096 Feb 19 12:35 reset_counters
-r--r--r-- 1 root root 4096 Feb 19 12:35 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 19 12:35 size_mb
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 19 12:35 uevent
root@grsnis01:~#
</pre>

=== Intel E3-1245 v6 ===

<pre>
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --status
edac-ctl: drivers are loaded.
[root@alphagdaq ~]# edac-util
edac-util: No errors to report.
[root@alphagdaq ~]# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
[root@alphagdaq ~]# ras-mc-ctl --layout
+-----------------------------------------------+
| mc0 |
| csrow0 | csrow1 | csrow2 | csrow3 |
----------+-----------------------------------------------+
channel1: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
----------+-----------------------------------------------+
[root@alphagdaq ~]# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#3channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#3channel#1 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#2channel#0 0 0
[root@alphagdaq ~]# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SSH-F
[root@alphagdaq ~]# ras-mc-ctl --summary
DBD::SQLite::db prepare failed: no such table: mc_event at /usr/sbin/ras-mc-ctl line 1129.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1130.
[root@alphagdaq ~]#
</pre>

=== AMD 3700X ===

(memory is non-ECC)

<pre>
root@daq13:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@daq13:~#
root@daq13:~#
root@daq13:~# edac-ctl --status
edac-ctl: drivers not loaded.
root@daq13:~# edac-util
edac-util: Error: No memory controller data found.
root@daq13:~# edac-util -s
edac-util: EDAC drivers loaded. No memory controllers found
root@daq13:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 2 root root 0 Jan 25 15:26 power
lrwxrwxrwx 1 root root 0 Jan 21 16:16 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Jan 21 16:16 uevent
</pre>

(memory is ECC)

<pre>
root@trinatdaq:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@trinatdaq:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@trinatdaq:~# edac-util
edac-util: No errors to report.
root@trinatdaq:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Dec 15 13:04 mc0
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
lrwxrwxrwx 1 root root 0 Dec 13 18:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dec 13 18:31 uevent
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_noinfo_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 max_location
-r--r--r-- 1 root root 4096 Dec 15 13:04 mc_name
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank4
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank5
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank6
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank7
--w------- 1 root root 4096 Dec 15 13:04 reset_counters
-rw-r--r-- 1 root root 4096 Dec 15 13:04 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Dec 15 13:04 seconds_since_reset
-r--r--r-- 1 root root 4096 Dec 15 13:04 size_mb
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Dec 15 13:04 uevent
root@trinatdaq:~#
</pre>

=== AMD 5000G ===

* no linux driver for AMD 5000-series "G" CPU
* no mention of ECC in the BIOS settings
* unclear status of ECC support in AMD documentation (sais only "pro" "G" CPUs have ECC)
* unclear status of ECC support in ASUS documentation (web page out of date)

=== AMD 5600X ===

<pre>
root@daq17:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-XE GAMING WIFI
root@daq17:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq17:~# edac-util
edac-util: No errors to report.
root@daq17:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@daq17:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Aug 19 19:27 mc0
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
lrwxrwxrwx 1 root root 0 May 10 10:11 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 May 10 10:11 uevent
root@daq17:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_noinfo_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 max_location
-r--r--r-- 1 root root 4096 Aug 19 19:27 mc_name
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank4
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank5
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank6
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank7
--w------- 1 root root 4096 Aug 19 19:27 reset_counters
-rw-r--r-- 1 root root 4096 Aug 19 19:27 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Aug 19 19:27 seconds_since_reset
-r--r--r-- 1 root root 4096 Aug 19 19:27 size_mb
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Aug 19 19:27 uevent
root@daq17:~#
</pre>

=== AMD 3955WX ===

<pre>
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --status
edac-ctl: drivers are loaded.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util
edac-util: No errors to report.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@alphasuperdaq:~/git/scripts/quotareport# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 19 root root 0 Dez 12 04:48 mc0
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
lrwxrwxrwx 1 root root 0 Dez 9 05:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dez 9 05:31 uevent
root@alphasuperdaq:~/git/scripts/quotareport#
root@alphasuperdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 max_location
-r--r--r-- 1 root root 4096 Feb 28 22:19 mc_name
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank0
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank1
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank10
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank11
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank12
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank13
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank14
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank15
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank2
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank3
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank4
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank5
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank6
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank7
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank8
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank9
--w------- 1 root root 4096 Feb 28 22:19 reset_counters
-rw-r--r-- 1 root root 4096 Feb 28 22:19 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Feb 28 22:19 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 28 22:19 size_mb
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 28 22:19 uevent
root@alphasuperdaq:~#
root@alphasuperdaq:~# ras-mc-ctl --layout
Use of uninitialized value $max_pos[3] in modulus (%) at /usr/sbin/ras-mc-ctl line 868.
Use of uninitialized value $d in numeric ge (>=) at /usr/sbin/ras-mc-ctl line 869.
Use of uninitialized value $d in sprintf at /usr/sbin/ras-mc-ctl line 872.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| mc0 |
| csrow0 | csrow1 |
| channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 | channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

0: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
root@alphasuperdaq:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#2 0 0
mc#0csrow#1channel#7 0 0
mc#0csrow#0channel#3 0 0
mc#0csrow#1channel#4 0 0
mc#0csrow#1channel#2 0 0
mc#0csrow#0channel#7 0 0
mc#0csrow#1channel#3 0 0
mc#0csrow#0channel#4 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#5 0 0
mc#0csrow#0channel#6 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#0channel#5 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#6 0 0
root@alphasuperdaq:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: ASUSTeK COMPUTER INC. model Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@alphasuperdaq:~#
</pre>

== Configure rasdaemon ==

<pre>
apt install rasdaemon
</pre>
<pre>
systemctl enable rasdaemon
systemctl restart rasdaemon
systemctl status rasdaemon
</pre>

<pre>
● rasdaemon.service - RAS daemon to log the RAS events
Loaded: loaded (/lib/systemd/system/rasdaemon.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-01-25 15:16:37 PST; 3min 5s ago
Main PID: 2477175 (rasdaemon)
Tasks: 1 (limit: 76958)
Memory: 17.1M
CGroup: /system.slice/rasdaemon.service
└─2477175 /usr/sbin/rasdaemon -f -r

Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Listening to events for cpus 0 to 11
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mc_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording aer_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording extlog_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mce_record events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording arm_event events
</pre>

== Get reports ==

* Intel 2x32GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --layout
+-------------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-------------------------+
channel1: | 16384 MB | 16384 MB |
channel0: | 16384 MB | 16384 MB |
----------+-------------------------+
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#0channel#1 0 0
root@daq00:~#
</pre>

* Intel 4x16GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#1 0 0
mc#0csrow#2channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#3channel#0 0 0
mc#0csrow#3channel#1 0 0
root@daq00:~#
root@daq00:~# ras-mc-ctl --layout
+-----------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-----------------------+
channel1: | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB |
----------+-----------------------+
root@daq00:~#
root@daq00:~#
root@daq00:~#
root@daq00:~# ras-mc-ctl --print-labels
ras-mc-ctl: Error: No dimm labels for Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@daq00:~#
</pre>

note: ubuntu LTS 22.04 DBD::SQLite::db error is not there.

= sensors =

== ASUS P9X79 WS ==

* https://www.asus.com/supportonly/P9X79%20WS/HelpDesk_Manual/
* BIOS version 4802
* modprobe nct6775
* modprobe coretemp

<pre>
root@daq14:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 0: +29.0°C (high = +82.0°C, crit = +100.0°C)
Core 1: +24.0°C (high = +82.0°C, crit = +100.0°C)
Core 2: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +82.0°C, crit = +100.0°C)

nouveau-pci-0200
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.00 V)
temp1: +39.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

nct6776-isa-0290
Adapter: ISA adapter
Vcore: 1.04 V (min = +0.00 V, max = +1.74 V)
in1: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
AVCC: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
+3.3V: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 2.04 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 904.00 mV (min = +0.00 V, max = +0.00 V) ALARM
3VSB: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
Vbat: 3.30 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1265 RPM (min = 0 RPM)
fan2: 1909 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +34.0°C (high = +0.0°C, hyst = +0.0°C) ALARM sensor = thermistor
CPUTIN: +58.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermal diode
AUXTIN: +31.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
PECI Agent 0: +31.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +96.0°C)
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

root@daq14:~#
</pre>

== ASUS TUF GAMING B550M-PLUS WIFI II ==

* BIOS 2803, 2806
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@midm9a:~# sensors
nct6798-isa-0290
Adapter: ISA adapter
in0: 488.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.82 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in14: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 0 RPM (min = 0 RPM)
fan2: 760 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan7: 1264 RPM (min = 0 RPM)
SYSTIN: +25.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +22.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +95.0°C sensor = thermistor
AUXTIN1: +25.0°C sensor = thermistor
AUXTIN2: +25.0°C sensor = thermistor
AUXTIN3: +25.0°C sensor = thermistor
PECI Agent 0 Calibration: +23.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +32.4°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

amdgpu-pci-0800
Adapter: PCI adapter
vddgfx: 1.45 V
vddnb: 993.00 mV
edge: +28.0°C
PPT: 20.00 W

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +33.4°C

root@midm9a:~#
</pre>

== ASUS ASUS ROG STRIX B550-XE GAMING WIFI ==

* BIOS 2423, 2604
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@daq13:~# sensors
nct6798-isa-0290
Adapter: ISA adapter
in0: 344.00 mV (min = +0.00 V, max = +1.74 V)
in1: 992.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in6: 216.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.30 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.81 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in11: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 280.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 845 RPM (min = 0 RPM)
fan2: 998 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +28.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +27.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +94.0°C sensor = thermistor
AUXTIN1: +28.0°C sensor = thermistor
AUXTIN2: +28.0°C sensor = thermistor
AUXTIN3: +97.0°C sensor = thermistor
PECI Agent 0 Calibration: +27.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +33.6°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

amdgpu-pci-0600
Adapter: PCI adapter
vddgfx: 1.45 V
vddnb: 999.00 mV
edge: +29.0°C
PPT: 14.00 W

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: +30.0°C

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +33.9°C

root@daq13:~#
</pre>

== ASUS ASUS ROG STRIX B550-E GAMING ==

* bios 2803
* echo modprobe jc42 >> /etc/rc.local
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@daq17:~# sensors
jc42-i2c-1-1b
Adapter: SMBus PIIX4 adapter port 0 at 0b00
temp1: +25.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: +28.0°C

nouveau-pci-0800
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.00 V)
temp1: +34.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

nct6798-isa-0290
Adapter: ISA adapter
in0: 288.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 224.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.31 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.79 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 280.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 843 RPM (min = 0 RPM)
fan2: 629 RPM (min = 0 RPM)
fan3: 746 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +22.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +25.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +93.0°C sensor = thermistor
AUXTIN1: +22.0°C sensor = thermistor
AUXTIN2: +22.0°C sensor = thermistor
AUXTIN3: +96.0°C sensor = thermistor
PECI Agent 0 Calibration: +25.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +27.6°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

jc42-i2c-1-1a
Adapter: SMBus PIIX4 adapter port 0 at 0b00
temp1: +23.2°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

asusec-isa-0000
Adapter: ISA adapter
CPU_Opt: 0 RPM
Chipset: +34.0°C
CPU: +25.0°C
Motherboard: +22.0°C
T_Sensor: -40.0°C
VRM: +31.0°C

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +28.0°C
Tccd1: +27.5°C

root@daq17:~#
</pre>

== ASUS PRIME B650-PLUS ==

* BIOS 1811
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@dsdaqgw:~# sensors
amdgpu-pci-0b00
Adapter: PCI adapter
vddgfx: 930.00 mV
vddnb: 1.19 V
edge: +38.0°C
PPT: 25.10 W

nct6799-isa-0290
Adapter: ISA adapter
in0: 920.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.04 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 320.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.28 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.28 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.10 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.04 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 416.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 328.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 0 RPM (min = 0 RPM)
fan2: 1253 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan7: 0 RPM (min = 0 RPM)
SYSTIN: +33.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +35.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +78.0°C sensor = thermistor
AUXTIN1: +11.0°C sensor = thermistor
AUXTIN2: +20.0°C sensor = thermistor
AUXTIN3: +82.0°C sensor = thermistor
PECI Agent 0 Calibration: +35.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +42.6°C
intrusion0: ALARM
intrusion1: OK
beep_enable: disabled

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +42.6°C
Tccd1: +36.4°C

root@dsdaqgw:~#
</pre>

= Enable CPU turbo mode =

* Intel CPU has a nominal CPU frequency (i.e. 3.4GHz) and a turbo-boost CPU frequency (i.e. 4.0GHz). Here we will enable this turbo-boost mode.
* Find out CPU capability
<pre>
root@daq01:~# lscpu | grep Hz
Model name: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
CPU MHz: 3965.803
CPU max MHz: 4000.0000
CPU min MHz: 800.0000
root@daq01:~#
</pre>
* Look up this CPU in the Intel ARK database - google for the CPU model name, i.e.
https://ark.intel.com/content/www/us/en/ark/products/88196/intel-core-i7-6700-processor-8m-cache-up-to-4-00-ghz.html
* Find current frequency settings:
<pre>
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "powersave" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 2.72 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
root@daq01:~#
</pre>
* Note the following:
** current governor is "powersave"
** "performance" governor is available
** "boost state support" is supported and active.
* Confirm CPU frequency governor:
<pre>
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
powersave
powersave
powersave
powersave
powersave
powersave
powersave
powersave
root@daq01:~#
</pre>
* Change governor to "performance":
<pre>
root@daq01:~# cpupower frequency-set --governor performance
Setting cpu: 0
Setting cpu: 1
Setting cpu: 2
Setting cpu: 3
Setting cpu: 4
Setting cpu: 5
Setting cpu: 6
Setting cpu: 7
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
performance
performance
performance
performance
performance
performance
performance
performance
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "performance" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 3.93 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
</pre>
* monitor CPU frequency:
<pre>
root@daq01:~# cpupower monitor
| Nehalem || Mperf || Idle_Stats
CPU| C3 | C6 | PC3 | PC6 || C0 | Cx | Freq || POLL | C1 | C1E | C3 | C6 | C7s | C8
0| 0.00| 0.00| 0.00| 0.00|| 88.80| 11.20| 3973|| 0.00| 0.00| 0.01| 0.02| 0.31| 0.00| 4.25
4| 0.00| 0.00| 0.00| 0.00|| 4.70| 95.30| 3945|| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 95.03
1| 0.73| 3.70| 0.00| 0.00|| 4.52| 95.48| 3864|| 0.00| 0.01| 1.19| 0.44| 2.82| 0.00| 90.23
5| 0.73| 3.70| 0.00| 0.00|| 0.37| 99.63| 3807|| 0.00| 0.00| 0.03| 0.09| 1.70| 0.00| 97.64
2| 2.28| 12.86| 0.00| 0.00|| 1.41| 98.59| 3829|| 0.00| 0.86| 3.17| 0.46| 7.70| 0.00| 85.87
6| 2.28| 12.86| 0.00| 0.00|| 2.88| 97.12| 3856|| 0.00| 0.11| 4.56| 2.15| 10.31| 0.00| 78.99
3| 1.33| 4.81| 0.00| 0.00|| 0.99| 99.01| 3804|| 0.00| 0.49| 0.79| 0.01| 1.03| 0.00| 96.12
7| 1.34| 4.81| 0.00| 0.00|| 1.26| 98.74| 3818|| 0.00| 0.01| 2.32| 0.47| 5.02| 0.00| 90.06
root@daq01:~#
</pre>
* check that the CPU is not overheating:
<pre>
root@daq01:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 1: +38.0°C (high = +84.0°C, crit = +100.0°C)
Core 2: +34.0°C (high = +84.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +84.0°C, crit = +100.0°C)
</pre>
* congratulations, we are running at 4 GHz now!

= Setup ubuntu as gateway to private network =

See also:
* https://daq.triumf.ca/DaqWiki/index.php/VME-CPU#Setup_the_boot_host_computer_.28el7.29
* http://www.triumf.info/wiki/DAQwiki/index.php/Dhcpd_on_eth1

== Steps to do ==

!!! UPDATED 16feb2024 Ubuntu-22.04.03 !!!

* assign network numbers to the private network, i.e. 192.168.1.x, 192.168.2.x, etc
* (on the gateway machine, each private network interface has to have a different network number)
* (each network interface can have multiple networks attached, via VLANs or via eth0:0, eth0:1 constructs)
* assign IP addresses on the private network, save them in /etc/hosts i.e. "hvps 192.168.1.10"
* (for simplicity, assign 192.168.1.1 to the gateway machine itself)
* (IP addresses 192.168.1.0 and 192.168.1.255 are "special", do not use them)
* setup DNS server (dnsmasq) to serve contents of /etc/hosts via DNS (otherwise, many programs will see inconsistent name to IP address mapping)
* setup DHCP server (dnsmasq) to give out the IP addresses
* setup TFTP server (dnsmasq), pxelinux and NFS for diskless booting
* setup time server (chronyd) to provide common time to all devices
* setup NAT so machines on private network can access the internet (to get OS updates, etc)
* setup NIS and NFS so machines on the private network can use common home directories
* setup rsync backup of machines on the private network

== setup hosts ==

* edit /etc/hosts
<pre>
192.168.1.101 dsfe01
... and so forth
</pre>

== setup dns and dhcp ==

!!! updated 16feb2024 for Ubuntu 22.04.3 !!!

!!! note: stock systemd-resolved remains, is configured to forward queries to dnsmasq, configured to forward queries to TRIUMF DNS !!!

!!! note: per authors of systemd, bare hostnames are not permitted, a DNS domain name must always be used. DNS domain name "dsdaq" is used in this example !!!

* apt install dnsmasq
* edit /etc/dnsmasq.conf
<pre>
# /etc/dnsmasq.conf
# DNS settings
#port=0 # disable DNS function
port=53 # enable DNS function
bind-interfaces # do not collide with systemd-resolved, we use 127.0.0.1:53, they use 127.0.0.53:53
domain-needed
bogus-priv
no-resolv
#log-queries # log DNS quesries

# TRIUMF DNS settings

server=142.90.100.19
expand-hosts
domain=dsdaq
local=/dsdaq/
localmx # do not forward MX queries to TRIUMF

# DHCP settings
interface=enp1s0f0 # VX network 192.168.0.x
#interface=missing # FEP and TSP network 192.168.1.x
interface=enp1s0f1 # controls network 192.168.2.x
#dhcp-range=192.168.1.50,192.168.1.150,infinite
dhcp-range=192.168.0.0,static
dhcp-range=192.168.2.0,static
log-dhcp # log DHCP queries
#quiet-dhcp
dhcp-ignore=tag:!known
#dhcp-boot=pxelinux.0

dhcp-option=option:dns-server,192.168.0.248
dhcp-option=option:ntp-server,192.168.0.248

# TFTP settings

enable-tftp
tftp-root=/tftpboot
</pre>
* #mkdir /tftpboot ### per tftp-root (if no ZFS)
* zfs create -o mountpoint=/tftpboot rpool/tftpboot ### (if root is ZFS)
* create resolved-dsdaq.conf with main IP address of dnsmasq
<pre>
[Resolve]
DNS=192.168.0.248
Domains=dsdaq triumf.ca
</pre>
* mkdir -p /etc/systemd/resolved.conf.d/
* /bin/rm -f /etc/systemd/resolved.conf.d/*.conf
* cp resolved-dsdaq.conf /etc/systemd/resolved.conf.d/
* systemctl stop systemd-resolved.service
* systemctl disable systemd-resolved.service
* systemctl enable dnsmasq
* systemctl restart dnsmasq
* try to "ping" or "host" some names from /etc/hosts, it should work
* try to ping daq00, daq00.triumf.ca, all should work
* resolved-dsdaq.conf goes into /etc/systemd/resolved.conf.d/ of all machines on the private network
* if not using systemd-resolved, edit /etc/resolv.conf

== setup chronyd ==

* enable ntp server:
* disable systemd-timesyncd, configure and enable chronyd per instructions above
* create dsdaq.conf
<pre>
# chrony config for dsdaq server

#allow 192.168.0.0
#allow 192.168.1.0
#allow 192.168.2.0
allow all

# end
</pre>
* cp dsdaq.conf /etc/chrony/conf.d/
* systemctl restart chronyd
* chronyc tracking ### wait until time is synchronized (a few seconds)
* create dsdaq.sources # use hostname or IP address of chronyd server
<pre>
# Put this file in /etc/chrony/sources.d
# systemctl restart chrony
# chronyc sources
# chronyc tracking
server dsdaqgw iburst prefer
# end
</pre>
* dsdaq.sources goes to /etc/chrony/sources.d of all machines on the private network

== setup diskless network booting ==

=== setup pxelinux for legacy pxe boot ===

* add bits in dnsmasq.conf
<pre>
dhcp-host=ac:1f:6b:9e:7f:4a,dsfe01,infinite
dhcp-boot=pxelinux.0
dhcp-option=17,"192.168.0.251:/nfsroot/%s,vers=3"
</pre>
* setup pxelinux for Ubuntu-18
<pre>
cd ~
wget https://www.kernel.org/pub/linux/utils/boot/syslinux/4.xx/syslinux-4.03.tar.bz2
tar xjvf syslinux-4.03.tar.bz2
cd syslinux-4.03
cp -pv ./core/pxelinux.0 ./com32/hdt/hdt.c32 ./memdisk/memdisk ./com32/menu/menu.c32 /zssd/tftpboot/
</pre>
* cd /zssd/tftpboot
<pre>
wget http://ladd00.triumf.ca/tftpboot/memtest86+-4.20.iso.zip
wget http://ladd00.triumf.ca/tftpboot/memtest86+-5.01.iso.gz
wget http://ladd00.triumf.ca/tftpboot/modules.alias
wget http://ladd00.triumf.ca/tftpboot/modules.pcimap
wget http://ladd00.triumf.ca/tftpboot/pci.ids
</pre>
* mkdir pxelinux.cfg
* emacs -nw pxelinux.cfg/default
<pre>
default menu.c32
prompt 0

menu title Welcome to the DSVSLICE PXE boot menu

timeout 50

label hdt
kernel hdt.c32

label memtest86+-5.01
kernel memdisk iso initrd=memtest86+-5.01.iso.gz

label memtest86+-4.20
kernel memdisk iso initrd=memtest86+-4.20.iso.zip

label vmlinuz-5.3.0-26-generic
menu default
kernel vmlinuz-5.3.0-26-generic
append initrd=initrd.img-5.3.0-26-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.1.1:/zssd/nfsroot/dsfe01 toram ip=dhcp panic=60 BOOTIF=enp1s0f0

#end
</pre>

=== setup pxelinux for efi pxe boot ===

* https://c-nergy.be/blog/?p=13808
* add dnsmasq.conf bits. note: root-path does not actually work, it is hardwired pxelinux.cfg/default file.
<pre>
# uefi pxe

dhcp-boot=tag:uefipxe,uefi/syslinux.efi
dhcp-option-force=tag:fe01,option:root-path,192.168.0.248:/nfsroot/fe01

# VX network 192.168.0.x

dhcp-host=40:a6:b7:c1:d9:c5,fe01,infinite,set:uefipxe,set:fe01
</pre>
* apt install syslinux pxelinux syslinux-common syslinux-efi syslinux-utils
<pre>
mkdir /tftpboot/uefi
cp /usr/lib/SYSLINUX.EFI/efi64/syslinux.efi /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/ldlinux.e64 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/menu.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/hdt.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libutil.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libmenu.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libcom32.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libgpl.c32 /tftpboot/uefi/
</pre>
* try to boot, it should bomb with "cannot load pxelinux.cfg/default"
* mkdir /tftpboot/uefi/pxelinux.cfg
* create /tftpboot/uefi/pxelinux.cfg/default, note nfsroot path is hardwired, note "http:" is used to load vmlinuz and initrd files (because tftp is super slow)
<pre>
default menu.c32
prompt 0

menu title Welcome to the DSDAQGW UEFI PXE boot menu

timeout 50

label vmlinuz-6.5.0-17-generic
kernel http://192.168.0.248:8088/uefi/vmlinuz-6.5.0-17-generic
append initrd=http://192.168.0.248:8088/uefi/initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=auto rw ip=dhcp panic=60

# append initrd=http://192.168.0.248:8088/uefi/initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.0.248:/nfsroot/fe01 rw ip=dhcp panic=60

# append initrd=initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.0.248:/nfsroot/fe01 rw ip=dhcp panic=60
# append initrd=initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=auto ip=dhcp rw panic=60

#end
</pre>
* try to boot, it will bomb with "cannot load http://...."
* install mini_httpd on port 8088, see https://acme.com/software/mini_httpd/
<pre>
apt install mini-httpd
emacs -nw /etc/default/mini-httpd # set "START=1"
emacs -nw /etc/mini-httpd.conf # set "host=192.168.0.248", "port=8088", "data_dir=/tftpboot"
systemctl enable mini-httpd
systemctl restart mini-httpd
systemctl status mini-httpd
wget http://192.168.0.248:8088/uefi/syslinux.efi
tail -100 /var/log/mini_httpd.log
</pre>
* fix initramfs bug for "nfsroot=auto", otherwise, "nfsroot=" has to be different for each machine and you have to have separate pxelinux config files for each machine
** emacs -nw /usr/lib/initramfs-tools/etc/dhcp/dhclient-enter-hooks.d/config
** add "echo ROOTPATH=..." if it is missing (Ubuntu LTS 22.04)
<pre>
echo "ROOTSERVER='${new_routers%% *}'"
echo "ROOTPATH='$new_root_path'"
echo "HOSTNAME='$new_host_name'"
</pre>
** regenerate initramfs (be careful you generate it for the right kernel!)
** see https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/2054482
<pre>
mkinitramfs 6.5.0-18-generic
</pre>
* copy linux kernel and initrd
<pre>
cp /boot/vmlinuz-6.5.0-18-generic /tftpboot/uefi/
cp /boot/initrd.img-6.5.0-18-generic /tftpboot/uefi/
chmod a+r /tftpboot/uefi/*
</pre>
* try to boot, should bomb with messages about "trying to mount root filesystem"
* tail /var/log/syslog
<pre>
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 vendor class: PXEClient:Arch:00007:UNDI:003016
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 1:netmask, 2:time-offset, 3:router, 4, 5,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 6:dns-server, 12:hostname, 13:boot-file-size,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 15:domain-name, 17:root-path, 18:extension-path,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 22:max-datagram-reassembly, 23:default-ttl,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 28:broadcast, 40:nis-domain, 41:nis-server,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 42:ntp-server, 43:vendor-encap, 50:requested-address,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 51:lease-time, 54:server-identifier, 58:T1,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 59:T2, 60:vendor-class, 66:tftp-server, 67:bootfile-name,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 97:client-machine-id, 128, 129, 130, 131,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 132, 133, 134, 135
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 next server: 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 broadcast response
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 1 option: 53 message-type 2
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 18 option: 67 bootfile-name uefi/syslinux.efi
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 12 hostname fe01
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 vendor class: PXEClient:Arch:00007:UNDI:003016
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 1:netmask, 2:time-offset, 3:router, 4, 5,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 6:dns-server, 12:hostname, 13:boot-file-size,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 15:domain-name, 17:root-path, 18:extension-path,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 22:max-datagram-reassembly, 23:default-ttl,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 28:broadcast, 40:nis-domain, 41:nis-server,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 42:ntp-server, 43:vendor-encap, 50:requested-address,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 51:lease-time, 54:server-identifier, 58:T1,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 59:T2, 60:vendor-class, 66:tftp-server, 67:bootfile-name,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 97:client-machine-id, 128, 129, 130, 131,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 132, 133, 134, 135
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 next server: 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 broadcast response
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 1 option: 53 message-type 5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 18 option: 67 bootfile-name uefi/syslinux.efi
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 12 hostname fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: error 8 User aborted the transfer received from 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/syslinux.efi to 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/syslinux.efi to 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 requested options: 1:netmask, 3:router, 6:dns-server
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 bootfile name: uefi/syslinux.efi
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 next server: 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 broadcast response
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 1 option: 53 message-type 2
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 requested options: 1:netmask, 3:router, 6:dns-server
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 bootfile name: uefi/syslinux.efi
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 next server: 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 broadcast response
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 1 option: 53 message-type 5
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/ldlinux.e64 to 192.168.0.110
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/01-40-a6-b7-c1-d9-c5 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8006E not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8006 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A800 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A80 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/pxelinux.cfg/default to 192.168.0.110
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/menu.c32 to 192.168.0.110
Feb 16 20:43:10 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/libutil.c32 to 192.168.0.110
Feb 16 20:43:10 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/pxelinux.cfg/default to 192.168.0.110
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 client provides name: dsdaqgw.triumf.ca
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 1:netmask, 28:broadcast, 2:time-offset, 3:router,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 15:domain-name, 6:dns-server, 119:domain-search,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 12:hostname, 44:netbios-ns, 47:netbios-scope,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 26:mtu, 121:classless-static-route, 42:ntp-server
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 bootfile name: uefi/syslinux.efi
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 next server: 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 1 option: 53 message-type 2
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 51 lease-time infinite
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 client provides name: dsdaqgw.triumf.ca
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 1:netmask, 28:broadcast, 2:time-offset, 3:router,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 15:domain-name, 6:dns-server, 119:domain-search,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 12:hostname, 44:netbios-ns, 47:netbios-scope,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 26:mtu, 121:classless-static-route, 42:ntp-server
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 bootfile name: uefi/syslinux.efi
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 next server: 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 1 option: 53 message-type 5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 51 lease-time infinite
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 12 hostname fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw rpc.mountd[3350210]: authenticated mount request from 192.168.0.110:981 for /nfsroot/fe01 (/nfsroot/fe01)
Feb 16 20:45:07 dsdaqgw rpc.mountd[3350210]: authenticated unmount request from 192.168.0.110:859 for /nfsroot/fe01/tmp/autoDY4k5u (/nfsroot/fe01)
</pre>
* tail /var/log/mini_httpd.log
<pre>
192.168.0.110 - - [16/Feb/2024:20:43:15 -0800] "GET /uefi/vmlinuz-6.5.0-17-generic HTTP/1.0" 200 14227944 "" "Syslinux/6.04"
192.168.0.110 - - [16/Feb/2024:20:43:24 -0800] "GET /uefi/initrd.img-6.5.0-17-generic HTTP/1.0" 200 137824833 "" "Syslinux/6.04"
</pre>

=== setup efi http boot ===

https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-deployment-prep-uefi-httpboot.html

=== setup linux kernel ===

* copy the kernel files
<pre>
cd /boot
rsync -av config* initrd* System.map* vmlinuz* /tftpboot/
</pre>
* cd /tftpboot
* chmod a+r *

=== setup nfs ===

* apt-get install nfs-kernel-server
* emacs -nw /etc/exports
<pre>
/nfsroot/dsfe01 dsfe01(rw,no_root_squash,async,no_subtree_check)
</pre>
* enable services
<pre>
systemctl enable nfs-server
systemctl enable nfs-mountd
systemctl enable nfs-idmapd
systemctl restart nfs-server
systemctl restart nfs-mountd
systemctl restart nfs-idmapd
</pre>
* after editing /etc/exports, run
<pre>
exportfs -av
</pre>

=== setup userland ===

!!! ubuntu-18 version !!!

* zfs create rpool/nfsroot
* zfs set dedup=verify rpool/nfsroot ### enable deduplication to save disk space because most linux images have mostly identical files
* clone ubuntu
<pre>
mkdir /nfsroot/dsfe01
cd /
rsync -avx . /nfsroot/dsfe01
</pre>
* edit config files:
* cd /nfsroot/dsfe01
* emacs -nw etc/hostname ### change to dsfe01
* emacs -nw etc/mailname ### change to dsfe01
* emacs -nw etc/yp.conf ### change daq00.triumf.ca to musr00.triumf.ca
* emacs -nw etc/defaultdomain ### change to MUSR-NIS
* cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
* emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
* emacs -nw root/.ssh/authorized_keys ### update root ssh keys
* emacs -nw etc/fstab ### add this
<pre>
192.168.1.1:/nfsroot/dsfe01 / nfs defaults,nolock 0 0
</pre>
* emacs -nw etc/chrony/chrony.conf
** comment-out all "pool" and "server" entries
** add entry "server 192.168.1.1 iburst"

After dsfe01 is booted:

* disable services:
<pre>
systemctl disable apache2
systemctl disable dnsmasq
systemctl disable zfs-import-cache
</pre>

To setup additional machines, clone dsfe01 instead of cloning the gateway machine

=== Allow manpages to be viewed ===

If <code>/</code> is mounted over NFS, <code>man</code> will report a permission error. Fix it with:

<pre>
ln -s /etc/apparmor.d/usr.bin.man /etc/apparmor.d/disable/
apparmor_parser -R /etc/apparmor.d/usr.bin.man
</pre>

== setup shared home directory ==

=== on the gateway machine ===
* define netgroups
* emacs -nw /etc/netgroup
<pre>
dsfe (dsfe01,,) (dsfe02,,)
</pre>
* emacs -nw /etc/nsswitch.conf ### edit the netgroup line to read:
<pre>
netgroup: files
</pre>
* export the home directories:
* emacs -nw /etc/exports ### add this:
<pre>
/zssd/home1 @dsfe(rw,no_root_squash,async,no_subtree_check)
</pre>
* exportfs -rc

=== on the frontend machine ===

* mkdir /home
* emacs -nw /etc/fstab ### add this:
<pre>
192.168.1.1:/zssd/home1 /home nfs defaults 0 0
</pre>
* mount -a

== setup NAT ==

NAT allows machines on the private network to connect to the internet: https://en.wikipedia.org/wiki/Network_address_translation

In these examples:
* replace "eno1" with name of the outgoing interface (the one connected to the TRIUMF network).
* replace "enp11s0" with name of the private network interface (192.168.1.x network)

* emacs -nw /etc/rc.local ### add this:
<pre>
# /etc/rc.local

# enable NAT

/sbin/iptables -t nat -A POSTROUTING -o eno1 -j MASQUERADE
iptables -L -v

# uncomment following lines if machine has prohibitive FORWARD rules:
#/sbin/iptables -I FORWARD -i eno1 -o enp11s0 -m state --state RELATED,ESTABLISHED -j ACCEPT
#/sbin/iptables -I FORWARD -i enp11s0 -o eno1 -j ACCEPT
#iptables -L -v

iptables -L -v
sysctl -w net.ipv4.ip_forward=1
#sysctl -a | grep forward

sh /etc/firewall-rfc1918.sh

# end
</pre>
* emacs -nw /etc/firewall-rfc1918.sh
<pre>
# firewall-rfc1918.sh

# prevent RFC1918 private network IP addresses from
# going in and out from our uplink.

ETH=eno1

iptables -F in-rfc1918
iptables -N in-rfc1918
iptables -A in-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A in-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A in-rfc1918 --dst 192.168.0.0/16 -j REJECT

iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -I INPUT -j in-rfc1918 -i $ETH

iptables -F out-rfc1918
iptables -N out-rfc1918
iptables -A out-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A out-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A out-rfc1918 --dst 192.168.0.0/16 -j REJECT

iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -I OUTPUT -j out-rfc1918 -o $ETH

iptables -D FORWARD -j out-rfc1918 -o $ETH
iptables -D FORWARD -j out-rfc1918 -o $ETH
iptables -I FORWARD -j out-rfc1918 -o $ETH

# allow TRIUMF-SECURE network

iptables -I in-rfc1918 -s 10.90.0.0/255.255.0.0 -j ACCEPT
iptables -I out-rfc1918 -d 10.90.0.0/255.255.0.0 -j ACCEPT

# show configuration

iptables -L -v

#end
</pre>

= KVM =

<pre>
apt install cpu-checker

root@daq13:~# kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
root@daq13:~#

(if not, shutdown, go into BIOS settings, enable CPU virtualization)

apt install virtinst ### will install many packages
apt install libvirt-clients libvirt-daemon-system-systemd libvirt-daemon qemu qemu-kvm libvirt-daemon-system virtinst bridge-utils

root@daq13:/home1/wheel# virsh list --all
Id Name State
------------------------------
1 ubuntu-guest running

apt install virt-manager

virt-install --name ubuntu-guest --os-variant ubuntu20.04 --vcpus 2 --ram 2048 --location /daq/daqstore/olchansk/linux/Ubuntu/ubuntu-20.04.3-desktop-amd64.iso --network bridge=virbr0,model=virtio --graphics none --extra-args='console=ttyS0,115200n8 serial'

virtual machine will start, boot, etc
to get out of it, CTRL + Shift followed by ]

ssh wheel@daq13
virt-manager

run virt-install again, omit "--graphics none", open graphics console from virt-manager, it booted into ubuntu installer desktop

virt-install --name test10 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --filesystem /kvm_ladd00,/ --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial" --graphics none

virt-install --name test14 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --disk /tmp/xxx/ladd00.img,bus=sata --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial rdshell" --graphics none --check path_in_use=off
</pre>

build image

<pre>
dd if=/dev/zero of=/tmp/xxx/ladd00.img bs=1024M count=20
mkfs.ext3 /tmp/xxx/ladd00.img ### ext4 fails to mount by SL6 kernel, "unknown ext4 options"
cd /kvm_ladd00/
mount -o loop /tmp/xxx/ladd00.img /mnt/tmp
rsync -av . /mnt/tmp/ --delete
umount /mnt/tmp
</pre>

on the guest, configure network: /etc/rc.local
<pre>
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

ifconfig eth2 192.168.122.2
route add -net 0.0.0.0 gw 192.168.122.1
ifconfig -a
netstat -rn

# end
</pre>

= ARM cross-compiler =

<pre>
apt install libgcc-9-dev-arm64-cross
apt install gcc-arm-linux-gnueabi
apt install gcc-arm-linux-gnueabihf
apt install g++-arm-linux-gnueabihf
apt install g++-arm-linux-gnueabi
</pre>

<pre>
arm-linux-gnueabi-gcc -o ttcp1 ttcp.c -march=armv7 -static
arm-linux-gnueabi-gcc -o memcpy.armv7 memcpy.cc -march=armv7 -static -O2
</pre>

= 32-bit intel cross-compiler =

Ubuntu 22.04:

<pre>
apt install libstdc++-11-dev:i386
apt install zlib1g-dev:i386
</pre>

NOTES:
* "g++ -m32" does not find libstdc++, please use "g++ -m32 -L/usr/lib/gcc/i686-linux-gnu/11/"
* to cross-build 32-bit MIDAS, use "make linux32".
* executables cross-build on Ubuntu-22 do NOT run on 32-bit Debain-11 (GLIBC and GLIBCXX version mismatch)
* executables cross-build on Ubuntu-22 run on 32-bit Debian-12.

RPi-RT

2024-01-10T18:31:59Z

Lmartin:

== Introduction ==
These are notes from cross-compiling a real-time (i.e. "fully preemptive") kernel for raspberry pi 4B. I attempted it on the 3B+, but ran into problems with the SD card module.
Adapted from https://www.instructables.com/64bit-RT-Kernel-Compilation-for-Raspberry-Pi-4B-/ and https://www.instructables.com/64bit-RT-Kernel-Compilation-for-Raspberry-Pi-4B-/

== Prerequisites ==
Install the toolchain for cross-compilation:
<pre>
sudo apt install git bc bison flex libssl-dev make libc6-dev libncurses5-dev
sudo apt install crossbuild-essential-arm64
</pre>

== Sources ==
Go to https://github.com/raspberrypi/linux/branches and select a branch. If you choose the default branch, you do not need to explicitly select it in the following command:
<pre>
git clone --depth=1 --branch <branch> https://github.com/raspberrypi/linux
</pre>

Find the matching real-time patch file in https://mirrors.edge.kernel.org/pub/linux/kernel/projects/rt/ and download it. E.g.:
<pre>
wget https://mirrors.edge.kernel.org/pub/linux/kernel/projects/rt/6.1/patch-6.1.70-rt21.patch.gz
</pre>

Apply the patch to the kernel sources:
<pre>
cd linux
gzip -cd ../patch-6.1.70-rt21.patch.gz | patch -p1 --verbose
mkdir ../kernel-out
</pre>

== Configuring and building the kernel ==
=== Default configuration ===
<pre>
make O=../kernel-out/ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcm2711_defconfig
</pre>
=== Turn on Real-Time support ===
<pre>
make O=../kernel-out/ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- menuconfig
</pre>
Navigate to General->Preemption Model and select the Real-Time option.

=== Building the kernel ===
Compilation of the kernel will take several minutes, depending on PC power and number of parallel threads.
<pre>
make -j16 O=../kernel-out/ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- Image modules dtbs
</pre>

The -j16 option is for the number of parallel processes and should probably be chosen no larger than the number of CPU cores.

== Installation of the kernel on SD card ==
Take the SD card with your working Raspberry Pi OS or similar linux installation and insert it in an SD reader connected to the PC. If it mounts automatically, unmount as root to prepare for the following (not necessary, but makes the instructions easier). DO NOT unmount using the graphical ''EJECT'' button, as that will not just unmount but also disconnect the device.

=== Prepare SD card ===
Assuming the SD card is device /dev/sdd:
<pre>
cd ../kernel-out
mkdir -p mnt/{fat32,ext4}
sudo mount /dev/sdd1 mnt/fat32
sudo mount /dev/sdd2 mnt/ext4
</pre>

=== Copy files ===
<pre>
cd ../linux
sudo make O=../kernel-out/ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- INSTALL_MOD_PATH=mnt/ext4 modules_install
cd ../kernel-out
sudo cp arch/arm64/boot/Image mnt/fat32/kernel8_rt.img # custom kernel name to preserve existing kernel
sudo cp arch/arm64/boot/dts/broadcom/*.dtb mnt/fat32/
sudo cp arch/arm64/boot/dts/overlays/*.dtb* mnt/fat32/overlays/
</pre>

=== Choose custom kernel for booting ===
<pre>
sudo emacs mnt/fat32/config.txt
</pre>

Add the line
<pre>
kernel=kernel8_rt.img
</pre>
to select your custom kernel.

== Profit! ==
Insert the SD card in the pi, start it up and watch the boot messages. Hopefully you get to the login prompt. Once logged in, check the correct kernel was loaded with
<pre>
uname -a
</pre>

RPi-RT

2024-01-06T01:16:49Z

Lmartin: /* Profit! */

== Introduction ==
These are notes from cross-compiling a real-time (i.e. "fully preemptive") kernel for raspberry pi 3B+ and 4B.
Adapted from https://www.instructables.com/64bit-RT-Kernel-Compilation-for-Raspberry-Pi-4B-/ and https://www.instructables.com/64bit-RT-Kernel-Compilation-for-Raspberry-Pi-4B-/

== Prerequisites ==
Install the toolchain for cross-compilation:
<pre>
sudo apt install git bc bison flex libssl-dev make libc6-dev libncurses5-dev
sudo apt install crossbuild-essential-arm64
</pre>

== Sources ==
Go to https://github.com/raspberrypi/linux/branches and select a branch. If you choose the default branch, you do not need to explicitly select it in the following command:
<pre>
git clone --depth=1 --branch <branch> https://github.com/raspberrypi/linux
</pre>

Find the matching real-time patch file in https://mirrors.edge.kernel.org/pub/linux/kernel/projects/rt/ and download it. E.g.:
<pre>
wget https://mirrors.edge.kernel.org/pub/linux/kernel/projects/rt/6.1/patch-6.1.70-rt21.patch.gz
</pre>

Apply the patch to the kernel sources:
<pre>
cd linux
gzip -cd ../patch-6.1.70-rt21.patch.gz | patch -p1 --verbose
mkdir ../kernel-out
</pre>

== Configuring and building the kernel ==
=== Default configuration ===
==== rPi 3B+ ====
<pre>
make O=../kernel-out/ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcmrpi3_defconfig
</pre>
==== rPi 4B ====
<pre>
make O=../kernel-out/ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcm2711_defconfig
</pre>
=== Turn on Real-Time support ===
<pre>
make O=../kernel-out/ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- menuconfig
</pre>
Navigate to General->Preemption Model and select the Real-Time option.

=== Building the kernel ===
Compilation of the kernel will take several minutes, depending on PC power and number of parallel threads.
<pre>
make -j16 O=../kernel-out/ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- Image modules dtbs
</pre>

The -j16 option is for the number of parallel processes and should probably be chosen no larger than the number of CPU cores.

== Installation of the kernel on SD card ==
Take the SD card with your working Raspberry Pi OS or similar linux installation and insert it in an SD reader connected to the PC. If it mounts automatically, unmount as root to prepare for the following (not necessary, but makes the instructions easier). DO NOT unmount using the graphical ''EJECT'' button, as that will not just unmount but also disconnect the device.

=== Prepare SD card ===
Assuming the SD card is device /dev/sdd:
<pre>
cd ../kernel-out
mkdir -p mnt/{fat32,ext4}
sudo mount /dev/sdd1 mnt/fat32
sudo mount /dev/sdd2 mnt/ext4
</pre>

=== Copy files ===
<pre>
cd ../linux
sudo make O=../kernel-out/ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- INSTALL_MOD_PATH=mnt/ext4 modules_install
cd ../kernel-out
sudo cp arch/arm64/boot/Image mnt/fat32/kernel8_rt.img # custom kernel name to preserve existing kernel
sudo cp arch/arm64/boot/dts/broadcom/*.dtb mnt/fat32/
sudo cp arch/arm64/boot/dts/overlays/*.dtb* mnt/fat32/overlays/
</pre>

=== Choose custom kernel for booting ===
<pre>
sudo emacs mnt/fat32/config.txt
</pre>

Add the line
<pre>
kernel=kernel8_rt.img
</pre>
to select your custom kernel.

== Profit! ==
Insert the SD card in the pi, start it up and watch the boot messages. Hopefully you get to the login prompt. Once logged in, check the correct kernel was loaded with
<pre>
uname -a
</pre>

Currently this does not work on a 3B+, the new kernel does not read the SD card correctly.

RPi-RT

2024-01-06T00:59:15Z

Lmartin:

RPi-RT

2024-01-06T00:40:05Z

Lmartin: Created page with "== Introduction == These are notes from cross-compiling a real-time (i.e. "fully preemptive") kernel for raspberry pi 3B+ and 4B. Adapted from https://www.instructables.com/64bit-RT-Kernel-Compilation-for-Raspberry-Pi-4B-/ and https://www.instructables.com/64bit-RT-Kernel-Compilation-for-Raspberry-Pi-4B-/ == Prerequisites == Install the toolchain for cross-compilation: <pre> sudo apt install git bc bison flex libssl-dev make libc6-dev libncurses5-dev sudo apt install cr..."

Sysman

2024-01-05T23:43:07Z

Lmartin: /* Linux documentation */

== System management ==

* [http://daq-plone.triumf.ca/SM/docs/local Misc documentation on the daq-plone site]
* [[Daqinv]] DAQ inventory database
* [[Amanda]] AMANDA backups

== Computer clusters ==

* [[DAQ_Cluster]] [[ISAC_Cluster]] [[MUSR_Cluster]] [http://www.triumf.info/wiki/DAQwiki/index.php/TITAN TITAN]
* gonodeinfo: [https://daq00.triumf.ca/gonodeinfo/gonodereport.html all] [https://daq00.triumf.ca/gonodeinfo/group-titan.html titan] [https://daq00.triumf.ca/gonodeinfo/group-isac.html isac] [https://daq00.triumf.ca/gonodeinfo/group-cmms.html cmms] (username/password: midas/midas)
* gonodeinfo: [https://dsvslice.triumf.ca/gonodeinfo/gonodereport.html dsvslice] (username/password: midas/midas)
* [https://daqstore.triumf.ca/ganglia ganglia] (username/password: midas/midas) (runs on daqstore, see daqstore:/etc/rc.local)
* zfsquotareport: [https://daq00.triumf.ca/zfsquotareport/zfsquota.html daq00] [https://isdaq00.triumf.ca/zfsquotareport/zfsquota.html isdaq00] [https://alpha00.triumf.ca/zfsquotareport/zfsquota.html alpha00] [musr00] [midm9a] [https://iris00.triumf.ca/zfsquotareport/zfsquota.html iris00] [https://midemma01.triumf.ca/zfsquotareport/zfsquota.html midemma01] [https://trinatdaq.triumf.ca/zfsquotareport/zfsquota.html trinatdaq] [titan00] [https://daqstore.triumf.ca/zfsquotareport/zfsquota.html daqstore]
* [https://ladd00.triumf.ca/diskusers/disks.html Diskusers]
* [https://daq00.triumf.ca/~olchansk/pingmon/triumf.html network ping monitor]
* [https://daq00.triumf.ca/~daqweb/daqbackup/ daqbackup]

* [http://trdata00.triumf.ca:2288 TRDATA dcache status]
* [[DiskScrub]] Documentation for diskscrub
* [[disk_benchmarks]]
* [[graphics_benchmarks]]
* [[VMs]] DAQ VMs

== Linux documentation ==

* [[SLinstall]] SL5/SL6/CentOS7/CentOS8 install instructions
* [[Ubuntu]] Ubuntu install instructions
* [[Raspbian]] Raspbian OS for ARM processors (Raspberry Pi, Cyclone5 SoC, etc)
* [[ZFS]] ZFS instructions
* [[rPi-RT]] Real-Time kernel for Raspberry Pi

== Misc documentation ==

* [[DaqVlanConfig]] VLANs for DAQ and experiments
* [[PcHardware]] DAQ PC Configurations
* [[PcRackmount]] DAQ Rackmounted Configurations
* [[DaqSvn]] DAQ SVN repositories and instructions (obsolete)
* [[Quartus]] Altera Quartus
* [[HADOOP]] (obsolete)
* [[EPICS]]
* [[VNC]]
* [[Cloning_raid1_boot_disks]]
* [[Setup_MIDAS_experiment]] (moved to MidasWiki)
* [[IEEE_MAC_TRIUMF]] IEEE MAC addresses assigned to TRIUMF
* [[DaqWikiManagement]] DAQWiki management
* [[PHYSICA]]
* [[ROOT]]
* [[MIDAS]]
* [[ROOTANA]] : ROOT Analyzer for MIDAS (moved to MidasWiki)
* [[ROODY]]
* [[ser2net]] USB-Serial-RS232 interface
* [[GIT]]
* [[TrdataDcache]] tradata dcache system for experimental data storage
* [[dhcpd_on_eth1]] Enabling dhcpd on secondary ethernet port
* [[nextcloud]]
* [[DAQ_VMs]] DAQ and experimental group VMs
* obsolete data acquisition software: [https://daq.triumf.ca/~daqweb/doc/susiq/ SUSIQ] [https://daq.triumf.ca/~daqweb/doc/nova/ NOVA]

Ubuntu

2023-11-15T18:47:15Z

Lmartin: Chromium is only snap now. No point installing if we're turning off snap further down.

= About Ubuntu =

AAA

= Ubuntu version =

<pre>
lsb_release -a
uname -a
</pre>

= Ubuntu installer =

* updated for Ububtu LTS 20.04.01, 22.04.1

* download the latest Ubuntu LTS desktop installer iso image
* dd the image to a USB key
* power down, disconnect all disks (all HDDs, all SSDs, all M.2)
* connect the SSD to be used as system disk
* if system will use mirrored SSDs (using ZFS mirror), leave second SSD disconnected, we will activate it later
* power up
* boot from USB key in legacy mode or UEFI mode (select this in the BIOS boot menu - F8 for ASUS, F11 for Supermicro)
* follow the instruction:
* "try ubuntu or install ubuntu" - choose "install"
* select language - accept default
* "updates and other software" - accept default settings ("normal install")
* "installation type" - select "advanced features" and "experimental: use ZFS"
* accept partition choice
* "where are you?" - select "Vancouver" (PST time zone)
* "who are you?" - leave all fields blank, except "username" set to "wheel", "password" set to the root password. hostname will be set later after configuring the network
* installation runs in a few minutes, when finished, reboot
* login as user wheel
* answer annouying questions:
* "livepatch" - say "next"
* "help improve" - select "do not send", say "next"
* "privacy" - leave "location" as "off", say "next"
* "ready to go", say "done"
* right-click on the desktop, say "open in terminal", a shell will open
* say "sudo /bin/bash", enter the root password, you now have the root shell
* run nm-connection-editor to configure the network. use netmask 255.255.224.0, gateway 142.90.100.18, DNS 142.90.100.19, search path "triumf.ca"
* after network is up (can ping ladd00), continue with post-installation steps below

= Install instructions =

== prepare ==
<pre>
apt update
apt upgrade
</pre>

== install ssh ==
<pre>
apt install ssh
</pre>

== configure hostname ==
<pre>
vi /etc/hostname
</pre>

== disable swap ==

ubuntu installer creates a 2 GB swap partition, not useful
on 32-64 GB machine, disable it:
<pre>
vi /etc/fstab ### comment out the "swap" line
</pre>

== maybe reboot ==

this is a good point to reboot the machine to boot
the latest kernel and to set the correct hostname

== install etckeeper ==

keep contents of /etc in a git repository:

<pre>
apt -y install etckeeper
</pre>

== set timezone ==
<pre>
timedatectl list-timezones | grep -i vancouver
timedatectl set-timezone America/Vancouver
</pre>

== install time synchronization ==
<pre>
apt -y install chrony
echo server time1.triumf.ca iburst >> /etc/chrony/chrony.conf
echo server time2.triumf.ca iburst >> /etc/chrony/chrony.conf
echo server time3.triumf.ca iburst >> /etc/chrony/chrony.conf
systemctl disable systemd-timesyncd.service
systemctl stop systemd-timesyncd.service
systemctl disable ntp
systemctl stop ntp
systemctl enable chrony
systemctl restart chrony
chronyc sources
chronyc tracking
</pre>

== reenable systemd-timesyncd ==

ONLY IF CHRONY DOES NOT WORK

<pre>
apt remove chrony
systemctl enable systemd-timesyncd.service
systemctl restart systemd-timesyncd.service
systemctl status systemd-timesyncd.service
timedatectl status
timedatectl timesync-status
</pre>

== enable outgoing email (debian 11) ==

this is different from ubuntu 20. it uses /etc/mailname and it hardwires the hostname into main.cf.

== enable outgoing email ==

* TRIUMF: use smtp.triumf.ca
* CERN: use cernmx.cern.ch

<pre>
apt install postfix ### select "satellite system", enter full hostname "xxx.triumf.ca", enter "smtp.triumf.ca"
apt install mailutils
dpkg-reconfigure postfix ### (if postfix already installed)
</pre>
<pre>
echo olchansk@triumf.ca lindner@triumf.ca bsmith@triumf.ca >> ~root/.forward
mailx root
test
^D
</pre>

== enable ping for all users (debian 11) ==

Without this tweak, Debian will report "operation not permitted" if a user tries to ping somewhere.

<pre>
echo 'net.ipv4.ping_group_range = 0 1000' > /etc/sysctl.d/99-ping.conf
</pre>

== install missing packages ==

(apt eats terminal input, even the "yes |" trick does not quite work,
repeat the following commands until they report that everything
is installed)

<pre>
yes | apt -y install ssh tcsh ethtool ncat rsync strace net-tools sysstat smartmontools lm-sensors traceroute time minicom screen git lsof debsums tmux
yes | apt -y install lsb-release
yes | apt -y install flex bison
yes | apt -y install neofetch
yes | apt -y install snmp snmp-mibs-downloader
yes | apt -y install git subversion g++ gfortran cmake doxygen
yes | apt -y install curl libcurl4 libcurl4-openssl-dev
yes | apt -y install mariadb-client ### mysql client
yes | apt -y install libz-dev sqlite3 libsqlite3-dev unixodbc-dev
yes | apt -y install libssl-dev
yes | apt -y install emacs xemacs21 joe
yes | apt -y install gnuplot dos2unix
yes | apt -y install mutt bsd-mailx # email clients
yes | apt -y install liblz4-tool pbzip2
yes | apt -y install libc6-dev-i386 # otherwise no /usr/include/sys/types.h
yes | apt -y install libreadline-dev
yes | apt -y install ubuntu-mate-themes
yes | apt -y install libmotif-dev libxmu-dev
yes | apt -y install libusb-dev libusb-1.0-0-dev
yes | apt -y install xfig gsfonts-x11 gsfonts-other # install fonts for xfig
yes | apt -y install libjson-perl
yes | apt -y install libgsl-dev # additional GNU Scientific Library
yes | apt -y install qt5-default # Qt development
yes | apt -y install python3-full python3-dev python3-dbg python3-pip ### for pyROOT
yes | apt -y install imagemagick imagemagick-common ckeditor # for elog
yes | apt -y install libjpeg-dev libjpeg-progs libjpeg-tools
yes | apt -y install linux-tools-common linux-tools-generic # cpupower frequency-info
yes | apt -y install rdesktop remmina remmina-plugin"*" # requested by POL
</pre>

Ubuntu LTS 20.04:
<pre>
yes | apt -y install linux-image-generic-hwe-20.04 linux-tools-virtual-hwe-20.04 # enable linux 5.11 series kernel
</pre>

== install git/scripts ==
<pre>
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd scripts
git pull
</pre>

== disable swap (debian 11) ==

* on 64 GB RAM machines swap is not useful
* on machines booted from network (NFS-ROOT), swap does not work
* on machines running from flash (RPi, etc), flash is too slow for useful swap
* swap configured by linux installers invariably has wrong size and is not useful

<pre>
systemctl disable dphys-swapfile
systemctl stop dphys-swapfile
dphys-swapfile uninstall
</pre>

== configure DNS ==

<pre>
cd ~/git/scripts
git pull
mkdir /etc/systemd/resolved.conf.d
cp etc/resolved-triumf.conf /etc/systemd/resolved.conf.d/
systemctl restart systemd-resolved
resolvectl
#systemd-analyze cat-config systemd/resolved.conf
</pre>

== install ganglia ==

<pre>
yes | apt-get -y install ganglia-monitor
systemctl enable ganglia-monitor
</pre>
<pre>
cd ~root/git/scripts
git pull
cp etc/gmond-ubuntu.conf /etc/ganglia/gmond.conf
ln -s ganglia/gmond.conf /etc
# ln -s arm-linux-gnueabihf/ganglia /usr/lib/ganglia ### fix path for ARM CPU
# ln -s i386-linux-gnu/ganglia /usr/lib/ganglia ### fix path for 32-bit Intel CPU
systemctl restart ganglia-monitor
systemctl status ganglia-monitor
ps -efw | grep gmond
</pre>
<pre>
cd ~root/git/scripts/ganglia
make install
./ganglia-all.perl
</pre>

== install gonodeinfo ==

* go to https://bitbucket.org/dd1/gonodeinfo follow instructions:
<pre>
yes | apt-get -y install golang
mkdir ~/git
cd ~/git
git clone https://bitbucket.org/dd1/gonodeinfo.git
cd gonodeinfo
git pull
make
make install # install gonodeinfo agent
cd ~ # this is important
</pre>
* edit /etc/gonodeinfo.conf
* change "Description", "Location", "User" and "Administrator" as appropriate (or delete them)
* change "Servers" to read: Servers: daq00.triumf.ca:8601
* run "gonodeinfo -v"
* if error is "connection refused". go to the nodeinfo server to add this client to the access control list:
* on the gonodeinfo server: run /opt/gonodeinfo/gonodereceive.exe -a daq13
* try gonodeinfo again, there should be no error
* on the gonodeinfo server: run gonodereport, look at the web pages, the new machine should be listed now

== install fonts for EPICS ==

* apt install xfonts-100dpi xfonts-75dpi
* restart Xorg (i.e. "killall Xorg", this will log you out from the console)
* xlsfonts | grep -i helvetica ### should show fonts with different sizes, not just size 0 (scalable)

== install libz.so.1 for CentOS compatibility ==

KO - confirm which versions on quartus need this.

<pre>
yes | apt-get -y install zlib1g
yes | apt-get -y install zlib1g:i386 libc6:i386 libgcc1:i386 gcc-6-base:i386
</pre>

== install libpng12.so.0 for Quartus compatibility ==

(does not work anymore!!!)

<pre>
wget http://ftp.ca.debian.org/debian/pool/main/libp/libpng/libpng12-0_1.2.50-2+deb8u2_amd64.deb
dpkg --install libpng12-0_1.2.50-2+deb8u2_amd64.deb
</pre>

== install libpng12.so.0 for Quartus 13.0sp1 ==

<pre>
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0.50.0
/bin/cp -pv libpng12.so.0 libpng12.so.0.50.0 /lib/x86_64-linux-gnu/
</pre>

== install packages for Xilinx ==

ubuntu LTS 22.04 vivado 2020.1

<pre>
apt install autoconf libtool
apt install libtinfo5
apt install texinfo
apt install zlib1g:i386
</pre>

== install packages for building ROOT ==

<pre>
apt -y install libx11-dev libxpm-dev libxft-dev libxext-dev libpng-dev libjpeg-dev xlibmesa-glu-dev libxml2-dev libgsl-dev cmake
</pre>

== install 32-bit libraries for PHYSICA ==

these instructions are for running 32-bit physica executable built for SL6 on ubuntu LTS 20.04

install physica sources (cannot build, do not have g77)

<pre>
cd ~/packages
git clone https://bitbucket.org/ttriumfdaq/physica.git
</pre>

install 32-bit libraries using ubuntu package manager:

<pre>
apt install lib32z1 # libz.so
</pre>

copy 32-bit SL6 shared libraries to /lib32

<pre>
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libX11.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libgd.so.2 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libpng12.so.0 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libreadline.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libncurses.so.5 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libg2c.so.0 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libxcb.so.1 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libXpm.so.4 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libjpeg.so.62 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libfontconfig.so.1 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libfreetype.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libtinfo.so.5 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libXau.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libexpat.so.1 /lib32/
</pre>

ldd should report:

<pre>
trinatdaq:trinat> ldd /usr/local/physica/physica.exe
linux-gate.so.1 (0xf7fa2000)
libX11.so.6 => /lib32/libX11.so.6 (0xf7e43000)
libgd.so.2 => /lib32/libgd.so.2 (0xf7dfe000)
libpng12.so.0 => /lib32/libpng12.so.0 (0xf7dd6000)
libz.so.1 => /lib32/libz.so.1 (0xf7db8000)
libreadline.so.6 => /lib32/libreadline.so.6 (0xf7d7e000)
libncurses.so.5 => /lib32/libncurses.so.5 (0xf7d5b000)
libg2c.so.0 => /lib32/libg2c.so.0 (0xf7d3d000)
libm.so.6 => /lib32/libm.so.6 (0xf7c39000)
libgcc_s.so.1 => /lib32/libgcc_s.so.1 (0xf7c1a000)
libc.so.6 => /lib32/libc.so.6 (0xf7a2f000)
libxcb.so.1 => /lib32/libxcb.so.1 (0xf7a05000)
libdl.so.2 => /lib32/libdl.so.2 (0xf79ff000)
libXpm.so.4 => /lib32/libXpm.so.4 (0xf79ee000)
libjpeg.so.62 => /lib32/libjpeg.so.62 (0xf7997000)
libfontconfig.so.1 => /lib32/libfontconfig.so.1 (0xf7962000)
libfreetype.so.6 => /lib32/libfreetype.so.6 (0xf78c9000)
libtinfo.so.5 => /lib32/libtinfo.so.5 (0xf78b0000)
/lib/ld-linux.so.2 (0xf7fa4000)
libXau.so.6 => /lib32/libXau.so.6 (0xf78ad000)
libexpat.so.1 => /lib32/libexpat.so.1 (0xf7885000)
trinatdaq:trinat>
</pre>

set login environment:

<pre>
setenv TRIUMF_FONTS $HOME/packages/physica/fonts
setenv PHYSICA_DIR $HOME/packages/physica
alias physica $PHYSICA_DIR/physica-SL6-32
</pre>

test:

<pre>
cd ~/packages/physica
physica
@rangauss.pcm
</pre>

== install lightdm ==

unlike the default gdm login manager, lightdm shows the machine hostname and does not require an extra mouse click to swicth from screen saver to login mode.

<pre>
apt -y install lightdm
# select lightdm
</pre>

== install desktop environments ==

note: default display manager and default desktop are deficient, please do not skip this step.

note: if apt asks to choose the display manager, select "lightdm"

note: KO - I recommend the "MATE" desktop.

note: you will have to cut-and-paste this several times because "apt" eats commands, even with "-y" and even piped from "yes".

<pre>
# install MATE desktop
DEBIAN_FRONTEND=noninteractive apt -y install ubuntu-mate-core ubuntu-mate-desktop ubuntu-mate-themes
# install Cinnamon desktop
DEBIAN_FRONTEND=noninteractive apt -y install cinnamon
# install KDE desktop
DEBIAN_FRONTEND=noninteractive apt -y install kubuntu-desktop
# install Lxqt desktop
DEBIAN_FRONTEND=noninteractive apt -y install lxqt
# install Xfce4 desktop
DEBIAN_FRONTEND=noninteractive apt -y install xfce4
</pre>

== install ROOT ==

Please install ROOT per instructions at http://root.cern.ch.

NOTE1: The ROOT package available from Ubuntu repositories is severely out of date and cannot be used with MIDAS and ROOTANA. ### DO NOT DO THIS! apt-get install root-system

NOTE2: as of 2017-Jan-09, ROOT binary kits for Ubuntu do not work (use GCC 5 instead of GCC6), build from source instead.

== Install x2go ==

KO - is this still needed? does it cause any security problems?

x2go instructions, thanks to Art O.

<pre>
add-apt-repository ppa:x2go/stable
apt-get update
apt-get install x2goserver x2goserver-xsession
</pre>

== enable root login from ladd00/daq00 ==
<pre>
ssh localhost
CTRL-C
/bin/cp ~root/git/scripts/etc/authorized_keys ~root/.ssh/
</pre>

== install smart-status ==
<pre>
ln -s ~/git/scripts/smart-status/smart-status.perl ~root/
</pre>

== enable boot menu and boot messages ==

This will enable the grub menu (with a 10 sec timeout) and
replace black screen with exciting linux boot messages.

* emacs -nw /etc/default/grub
<pre>
GRUB_DEFAULT=0
#GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="vga=769 video=640x480"
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX=""
#GRUB_GFXMODE=640x480
</pre>
* update grub config:
<pre>
grub-mkconfig -o /boot/grub/grub.cfg
</pre>

== reboot ==

this completes installation of the base system.

following sections modify basic ubuntu to fix known problems and to enable special stuff.

= Enable automatic updates =

<pre>
apt install unattended-upgrades
cd ~/git/scripts
git pull
/bin/cp -v etc/99apt-conf-ko /etc/apt/apt.conf.d/
apt-config dump | grep Unattended
</pre>

Following is obsolete:

* emacs -nw /etc/apt/apt.conf.d/50unattended-upgrades
** uncomment in Allowed-Origins "-security" and "-updates"
** add in Allowed-Origins: "Google LLC:stable";
** uncomment/add: "Unattended-Upgrade::Mail "root";
* emacs -nw /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
</pre>
* test: unattended-upgrade --dry-run -v

NOTE: update-on-shutdown is disabled.

NOTE: there is no update-on-boot, but:

NOTE: if machine was off for a long time, the systemd update timer would have expired and it will fire soon after reboot, causing an automatic update run. this is unwanted, and there is no fix or workaround for it. K.O. June-2023.

= Fix bpool is full =

!!! only if ROOT on ZFS !!!

There is an error in the zsys package that causes bpool to run out of space,
see [[#Ubuntu zsys]] for more details.

To fix:
<pre>
cd ~/git/scripts
git pull
cp etc/zsys.conf /etc/
zsysctl service reload
zsysctl service gc
zpool list bpool
zfs list bpool
df /boot
</pre>

= IPMI instructions =

IPMI is the board management hardware on Supermicro and other server motherboards. This includes hardware sensors - fan rotation speed, temperatures and power supply voltages.

<pre>
apt-get install ipmitool
systemctl enable ipmievd
systemctl restart ipmievd
</pre>

Run:
* ipmitool sel list ### event list
* ipmitool sel elist ### event list
* ipmitool sel clear ### clear event list (if it becomes full)
* ipmitool sensor ### report hardware sensors

= move /home/wheel =

note: this MUST be done if ZFS root and NIS/autofs with /home.

Default location of wheel's home directory will collide with autofs /home, it has to be moved,
for example to /wheel.

<pre>
# logout from the wheel user
# go to another computer
ssh root@daqubuntuxxx
zfs list | grep wheel ### identify zfs name wheel_xxxxxx
zfs set mountpoint=/wheel rpool/USERDATA/wheel_hm8fzh
emacs -nw /etc/passwd ### change wheel's home directory from /home/wheel to /wheel
su - wheel ### check that user wheel still works
</pre>

This will break wheel's ability to run snap programs, such as firefox, install chrome as listed below.

= enable NIS (ubuntu 22.04, debian 11) =

<pre>
apt -y install rpcbind nis
echo DAQ-NIS >> /etc/defaultdomain
echo ypserver daq00.triumf.ca >> /etc/yp.conf
systemctl enable ypbind.service
systemctl restart ypbind.service
systemctl status ypbind.service
ypwhich -m
</pre>

enable ypserv:

<pre>
sed -i s/NISSERVER=false/NISSERVER=slave/ /etc/default/nis
/usr/lib/yp/ypinit -s daq00
echo ypserver localhost >> /etc/yp.conf
sed -i "s/ypserver .*/ypserver localhost/" /etc/yp.conf
systemctl enable ypserv
systemctl restart ypserv
systemctl restart ypbind
</pre>

edit /etc/nsswitch.conf to read:

<pre>
# begin get data from nis
passwd: files nis
group: files nis
shadow: files nis
automount: files nis
netgroup: files nis
# end get data from nis
</pre>

enable hourly update of nis maps:

<pre>
mkdir ~root/git
cd ~root/git
git clone http://daq00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>

If this is a new machine, then on the master NIS node (daq00), add this new node to /etc/netgroup, and update NIS maps (cd /var/yp; make)

= enable NIS (ubuntu 20.04) =

* apt-get -y install portmap nis ### will ask for NIS domain (DAQ-NIS)
* dpkg-reconfigure nis ### reconfigure if already installed
* ypwhich -m
* edit /etc/default/nis
** set "NISSERVER=slave"
** Ubuntu LTS 20.04, check that "YPBINDARGS=" is blank, remove "-no-dbus" if it is there
* #edit /etc/yp.conf, comment-out everything, add "domain DAQ-NIS server localhost"
* edit /etc/yp.conf, comment-out everything, add "ypserver localhost"
* /usr/lib/yp/ypinit -s daq00
* systemctl enable nis
* systemctl restart nis
* ypwhich
* ypwhich -m
* ypcat -k passwd
* vi /etc/nsswitch.conf ### add the automount line, modify the passwd, group and shadow lines to read this:
<pre>
# begin get data from nis
passwd: files nis
group: files nis
shadow: files nis
automount: files nis
netgroup: files nis
# end get data from nis
</pre>
* enable hourly update of NIS maps
<pre>
mkdir ~root/git
cd ~root/git
git clone http://ladd00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>
* ### NOT NEEDED sudo vi /etc/idmapd.conf ### add line: "Domain = triumf.ca"

= enable autofs =

<pre>
apt -y install autofs
systemctl enable autofs
systemctl restart autofs
ls -l /home/olchansk ### test autofs, check file owner is correct
</pre>

= enable NFS server =

<pre>
apt install nfs-kernel-server
#edit /etc/exports
systemctl enable nfs-server
systemctl restart nfs-server
</pre>

= NIS master =

notes for setting up the NIS master

== wheel user ==

"wheel" is the default administrative user. We do not want it's password exported to NIS (encrypted password hash is world visible) and we do not want it's home directory exported to NFS (~wheel/.ssh is world visible and potentially writable: anybody can change ~wheel/.ssh/authorized_keys).

* move wheel's home directory from /home/wheel to /wheel (see special section about this)
* change wheel's UID and GID from 1000 to a value below MINUID in /var/yp/Makefile

== coherent uids ==

we do not want system accounts defined in /etc/passwd of the NIS master
to be included in the NIS map "passwd". this causes trouble on NIS clients
where newly installed packages fail to create local system users because same
user already exists in NIS.

This is controlled by MINUID in /var/yp/Makefile.

Historical TRIUMF uids start from around 200, but several clusters do not have any historic TRIUMF uids below 500 and MINUID is set to:
* DAQ-NIS: MINUID=200
* ISAC-NIS: MINUID=500
* TITAN-NIS: MINUID=500
* MUSR-NIS: MINUID=500
* TIG-NIS: MINUID=500 (100 on SL6 mother8pi)

Ubuntu 20 has two programs to create users:
* adduser - creates new users with UID 1000 and up as specified in /etc/adduser.conf. No problems here.
* adduser --system - creates new system users with UID 100 and up as specified in /etc/adduser.conf. No problems here.
* useradd - creates new users with UID 1000 and up as specified in /etc/login.defs. No problems here.
* useradd --system - creates new system users with UID 999 and down (read "man useradd", section at the end about SYS_UID_MAX). This collides with NIS MINUID, these system users will be included in the NIS map and cause trouble.

This problem cannot be fixed, SYS_UID_MIN, SYS_UID_MAX and UID_MIN in /etc/login.defs do not seem
to have any effect on UIDs chosen by "useradd --system". (tested on Ubuntu LTS 20.04).

So far only these system accounts seem to be affected by this:
* systemd-coredump
* ganglia

To fix:
* run "sort -r -n -t: -k3 /etc/passwd" to identify the last unused system user uid (range 100..200)
* run "sort -r -n -t: -k3 /etc/group" to identify the last unused system user gid (range 100.200)
* systemd-coredump: manually change UID and GID (package systemd-coredump is usually not installed)
* ganglia: same thing, then change ownership on all ganglia files.

Also read systemd author's opinion on system vs user UIDs:
https://github.com/systemd/systemd/issues/4850#issuecomment-265698275

= Fix systemd-logind NIS breakage =

!!! THIS IS NOT NEEDED FOR UBUNTU LTS 20.04 !!!

there is a delay in ssh logins for normal users. "ssh -v" shows the delay is after "pledge...". this
fix removes the delay.

systemd developers think that we should not use NIS and made sure there are
problems if we do. To give them credit, they do offer a workaround. Read this:
https://github.com/poettering/systemd/commit/695fe4078f0df6564a1be1c4a6a9e8a640d23b67

<pre>
mkdir /etc/systemd/system/systemd-logind.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-logind.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-logind.service
</pre>

= Fix systemd-udevd NIS breakage =

see same problem as above with udev getting stuck. ubuntu lts 20.04.

<pre>
mkdir /etc/systemd/system/systemd-udevd.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-udevd.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-udevd.service
</pre>

= Configure USB device permissions =

Configure USB device permissions for user access to USB-serial devices, Altera USB Blaster, etc.

* create file /etc/udev/rules.d/99-usb-chmod.rules with this contents:

<pre>
emacs -nw /etc/udev/rules.d/99-usb-chmod.rules
ACTION=="add", SUBSYSTEM=="usbmisc", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /dev/%c"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /proc/%c"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVICE}"
ACTION=="add", ENV{PHYSDEVBUS}=="usb-serial", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVPATH}=="/class/tty/ttyS*", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyUSB*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyACM*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyS*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", DEVPATH=="*video*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
</pre>

* reload udev rules: udevadm control --reload-rules
* apply new permissions: udevadm trigger --action=add
* watch udev activity: udevadm monitor -p

= Configure lightdm display manager =

* enable it
<pre>
echo lightdm | dpkg-reconfigure -fteletype lightdm
systemctl disable gdm
systemctl disable sddm
systemctl enable lightdm
</pre>

* make the MATE desktop as default
<pre>
cd ~root/git/scripts/
git pull
/bin/cp -v etc/lightdm_default_mate.conf /etc/lightdm/lightdm.conf.d/
</pre>

* enable login by NIS users
<pre>
/bin/cp -v etc/lightdm_enable_nis_login.conf /etc/lightdm/lightdm.conf.d/
</pre>

* restart lightdm
<pre>
systemctl stop gdm
systemctl restart lightdm
</pre>

= Install libpng12.so.0 =

Quartus 16 needs libpng12:

<pre>
wget http://mirrors.kernel.org/ubuntu/pool/main/libp/libpng/libpng12-0_1.2.54-1ubuntu1_amd64.deb
dpkg --install libpng12-0_1.2.54-1ubuntu1_amd64.deb
</pre>

= Install google-chrome =

<pre>
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
dpkg -i google-chrome-stable_current_amd64.deb
</pre>

confirm autoupdate is enabled, observe dl.google.com is present in the list of repositories:
<pre>
apt update
...
Get:5 https://dl.google.com/linux/chrome/deb stable/main amd64 Packages [1,094 B]
...
</pre>

FOLLOWING IS OBSOLETE:

Instructions from here:
https://www.ubuntuupdates.org/ppa/google_chrome?dist=stable

<pre>
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-tmp.list'
apt update
apt install google-chrome-stable
/bin/rm -f /etc/apt/sources.list.d/google-tmp.list
</pre>

= Install amanda client =

ONLY ONE MACHINES THAT HOST HOME DIRECTORIES

* apt install amanda-client
* edit /etc/amandahosts
<pre>
amanda.triumf.ca amanda amdump
</pre>
* check permissions on /etc/amandahosts:
<pre>
root@daq00:/var/log/amanda# ls -l /etc/amandahosts
-rw------- 1 backup backup 49 Jan 27 10:48 /etc/amandahosts
</pre>
* fix if needed: chown backup.backup /etc/amandahosts; chmod a= /etc/amandahosts; chmod u=wr /etc/amandahosts
* edit /etc/amanda-security.conf, add this line:
<pre>
runtar:gnutar_path=/usr/bin/tar
</pre>

On the amanda machine:

* in amanda disklist, use dump type "bsdtcp-comp-user-tar"
* su - amanda and run amcheck -c daily daq00
<pre>
-bash-4.1$ amcheck -c daily daq00

Amanda Backup Client Hosts Check
--------------------------------
Client check: 1 host checked in 0.092 seconds. 0 problems found.

(brought to you by Amanda 3.3.7p1.git.685ff76d)
</pre>

= Enable rc.local =

For reasons unknown, Ubuntu LTS 20.04 does not enable /etc/rc.local. Do this:

<pre>
cd ~/git/scripts
git pull
cp -n -v etc/rc.local /etc/
chmod a+rx /etc/rc.local
cp etc/rc-local.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable rc-local
systemctl start rc-local
systemctl status rc-local
</pre>

= Remove unwanted packages =

<pre>
apt remove zsys
apt remove sddm
</pre>

= Disable unwanted services =

<pre>
systemctl disable mpd
systemctl disable snapd
systemctl disable ModemManager
systemctl --global mask tracker-extract-3.service
systemctl --global mask tracker-miner-fs-3.service
systemctl daemon-reload
</pre>

= Disable sleep and suspend =

note: we see some computers randomly shutdown or go to sleep, log files indicates the "sleep" or "suspend" button was pushed by user, but no such buttons actually exist. this is the fix for this:

<pre>
systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target systemd-suspend.service systemd-hybrid-sleep.service
</pre>

= Enable crontab @reboot for MIDAS =

startup scripts have a bug - cron @reboot entries for normal users can run before autofs is ready, so if the home directory is on autofs/NFS, it cannot be accessed and the cron job fails. If MIDAS is supposed to be started by cron @reboot, it will not start (there *will* be an error message in /var/log/cron).

<pre>
mkdir /etc/systemd/system/cron.service.d
echo -e "[Unit]\nAfter=ypbind.service autofs.service\n" > /etc/systemd/system/cron.service.d/local.conf
systemctl daemon-reload
systemctl cat cron.service
</pre>

Explore the systemd dependency tree using "systemctl list-dependencies" maybe with "--all".

Visualize the exact boot sequence from previous boot: "systemd-analyze plot > xxx.svg", look at the svg file using a web browser.

Crontab entry to start midas: (install in the midas user crontab, not root crontab)

<pre>
su - midasuser
crontab -l
#@reboot /bin/bash -l -c "/home/trinat/bin/start-daq-applications"
#@reboot /bin/tcsh -c "/home/trinat/bin/start-daq-applications"
</pre>

= Install apache httpd proxy for midas and elog =

This will configure the HTTPS/SSL certificate using "certbot" and "letsencrypt" and configure an HTTPS web server using apache2.

First, configure apache2:

* execute these commands:
<pre>
apt -y install apache2
cd /etc/apache2
</pre>
* create new file conf-available/ssl-daq14.conf # use actual hostname instead of daq14
<pre>
SSLSessionCache shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout 300
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
SSLCryptoDevice builtin
</pre>
* create new file sites-available/daq14-ssl.conf # use actual hostname instead of daq14
<pre>
<IfModule mod_ssl.c>
<VirtualHost *:443>
ServerName daq14.triumf.ca
DocumentRoot /var/www/html
ErrorLog /var/log/apache2/daq14.log
SSLEngine on
# note SSLProtocol, SSLCipherSuite and some other settings are overwritten by /etc/letsencrypt/options-ssl-apache.conf
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SEED:!IDEA:!RC4
## use port specified in elogd.cfg
#ProxyPass /elog/ http://localhost:8082/ retry=1
## use mhttpd port
#ProxyPass / http://localhost:8080/ retry=1
Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"
<Location />
SSLRequireSSL
AuthType Basic
AuthName "DAQ password protected site"
Require valid-user
# create password file: touch /etc/apache2/htpasswd
# to add new user or change password: htpasswd /etc/apache2/htpasswd username
AuthUserFile /etc/apache2/htpasswd
</Location>
</VirtualHost>
</IfModule>
</pre>
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* enable ssl module
* enable new configurations
<pre>
a2enmod ssl
a2enmod headers
a2enmod proxy
a2enmod proxy_http
a2enconf ssl-daq14
a2ensite daq14-ssl
</pre>
* disable default ssl sites
<pre>
a2dissite 000-default-le-ssl
a2dissite 000-default
ls -l /etc/apache2/sites-enabled/ ### should show only daq14-ssl.conf
</pre>
* check that there are no syntax problems
<pre>
apache2ctl configtest
</pre>
* enable and start apache2:
<pre>
systemctl enable apache2
systemctl restart apache2
systemctl status apache2
</pre>
* apache2 may fail to start, look in /var/log/apache2/error.log and /var/log/apache2/daq14.log
* if it says "Failed to configure ... certificate", proceed to the step for setting certbot.
* try to access https://daq14.triumf.ca
** you should see a complaint about self-signed certificate
** you should see a request for password (do not login yet)
** if you get "connection refused", HTTPS port 443 may need to be enabled in the local firewall, look at documentation for ufw.
Second, configure certbot:

(Note: as of 2018-01-18 certbot requires use of http port 80 to get the initial https certificate,
renewal can continue to use the https port 443)

(Note: as of 2019-01-?? certbot requires use of port 80 for renewals)

* check that port 80 is not used by anything:
* netstat -an | grep LISTEN | grep ^tcp | grep 80
* lsof -P | grep -i tcp | grep LISTEN | grep 80
* if lsof reports that apache2 is listening on port 80, follow the apache2 instructions above (remove "listen 80" from apache2.conf

* install certbot (if necessary open tcp port 80 in the firewall, see documentation for ufw):
<pre>
apt install certbot python3-certbot-apache
certbot certonly --standalone --installer apache
</pre>
* then answer questions:
* "activate HTTPS for daq14.triumf.ca" - say ok
* "enter email address" - enter your own email address
* "please read terms..." - read the terms and say "agree"
* it will take a few moments...
* "congratulations..." - say ok.
<pre>
certbot install --apache --cert-name daq14.triumf.ca
</pre>
* then answer questions:
* "choose redirect..." - say "1" (no redirect)
* look inside /etc/apache2/sites-enabled/daq14-ssl.conf to see that SSLCertificateFile & co point to certbot certificates in
/etc/letsencrypt/live/daq14.triumf.ca/
* to check current renewal and to update the certbot config file in /etc/letsencrypt/renewal, run this:
<pre>
certbot renew --standalone --installer apache --force-renewal
</pre>

NOTE: this certificate will expire in 3 months, automatic renewal should work with current version of certbot

Third, activate password protection:

* as shown in the config file above, create password file and initial user: (replace "midas" with specific username)
<pre>
touch /etc/apache2/htpasswd
htpasswd /etc/apache2/htpasswd midas
</pre>

* restart apache2
<pre>
systemctl restart apache2
systemctl status apache2
</pre>

From here:
* enable proxy for MIDAS mhttpd - uncomment redirect in the config file above
* enable proxy for ELOG - ditto
<pre>
a2enmod proxy
a2enmod proxy_http
apache2ctl configtest
systemctl restart apache2
</pre>

From here:
* enable proxy for MIDAS mhttpd - uncomment redirect in the config file above
* enable proxy for ELOG - ditto
<pre>
a2enmod proxy
a2enmod proxy_http
apache2ctl configtest
systemctl restart apache2
</pre>
* try accessing MIDAS https://daq14.triumf.ca/ (make sure mhttpd is running)
* if it's not working, check odb setting FIXME!
* try accessing ELog https://daq14.triumf.ca/elog/ (make sure elogd is running)
* if it's not working, check elogd.cfg file and make sure
<pre>
SSL = 0
</pre>

NOTE: if certbot fails with errors about 'module' object has no attribute 'pyopenssl',
try this: pip install requests==2.6.0

= Enable elog PDF preview =

see https://stackoverflow.com/questions/52998331/imagemagick-security-policy-pdf-blocking-conversion

* xemacs -nw /etc/ImageMagick-6/policy.xml
* remove this section at the end:
<pre>

<policy domain="coder" rights="none" pattern="PS" />
<policy domain="coder" rights="none" pattern="PS2" />
<policy domain="coder" rights="none" pattern="PS3" />
<policy domain="coder" rights="none" pattern="EPS" />
<policy domain="coder" rights="none" pattern="PDF" />
<policy domain="coder" rights="none" pattern="XPS" />
</pre>

= Install Jupyter notebook =

<pre>
From https://jupyter.org/install
apt install python3-pip
pip install jupyterlab
pip install notebook
~/.local/bin/jupyter notebook
watch the http://localhost:8888 URL that it printed
say "no" to offer to start firefox (it will not work!)
URL is: http://localhost:8888/tree?token=xxx
from the machine where you are running the web browser (i.e. google-chrome), run (replace trinat@trinatdaq with the username and machine name where you started jupyter)
open a new shell and run: ssh -v trinat@trinatdaq -L 8888:localhost:8888
in the web browser, open http://localhost:8888
this gives us the login page
in the password or token entry field, put the token from the "tree?token=xxx" above (printed by jupyter on startup)
push button "login"
jupyter page should open with the list of files in the trinat home directory
congratulate Brian with full success
</pre>

= Install ZFS quota report =

If there are any ZFS volumes, install script to report disk and quota usage

<pre>
cd ~/git/scripts/quotareport
git pull
mkdir /var/www/html/zfsquotareport
cp -pv ~/git/scripts/quotareport/sorttable.js /var/www/html/zfsquotareport/
ln -s $PWD/zfsquotareport.perl /etc/cron.daily/
touch /etc/crontab
</pre>

If httpd is configured to redirect "/" to MIDAS mhttpd:
* add following to /etc/apache2/sites-enabled/xxx-ssl.conf in front of "ProxyPass / ..."
* run "systemctl reload apache2"
<pre>
## do not proxy zfs quota report directory
ProxyPass /zfsquotareport/ !
</pre>

= Install PHP =

* apt install php libapache2-mod-php
* systemctl restart apache2
* create /var/www/html/info.php
<pre>
<?php

phpinfo();
</pre>
* open https://daq00.triumf.ca/info.php

= Configure TRIUMF printers =

<pre>
systemctl stop cups
systemctl disable cups
echo "ServerName printers.triumf.ca" > /etc/cups/client.conf
lpstat -a
</pre>

= Enable core dumps =

By default, Ubuntu LTS 20.04 installs the apport package
which disabled core dumps from user applications. (google it up!).
It is not meant to do this and documentation claims that
it is not installed and not enabled by default. Oh, well...

<pre>
apt remove apport
apt autoremove ### will remove apport-symptoms and a few other packages
</pre>

After this, core dumps are written to file "core" in the current directory.
See /proc/sys/kernel/core_pattern and /proc/sys/kernel/core_uses_pid.

= Enable debugger =

By default, Ubuntu LTS 20.04 does not permit debugger to attach and debug
already running programs. To enable it, add following to /etc/rc.local

<pre>
echo 0 > /proc/sys/kernel/yama/ptrace_scope
</pre>

= Disable Ubuntu Pro nag =

If "apt upgrade" requests Ubuntu Pro or esm-apps, disable the nag:
<pre>
/bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf
</pre>

= Update packages =

* apt-get update # update package list
* apt-get dist-upgrade # install updated packages and update "kept back" packages
* apt-get autoremove # remove packages that apt thinks should be removed

= Finish installation =

Congratulations. There is nothing more to do!

* reboot
<pre>
shutdown -r now
</pre>

= Install ZFS =

!!! after installing all the packages, after updating the system, after updating the linux kernel, after rebooting into latest kernel !!!

<pre>
apt-get install zfsutils-linux
</pre>

Follow generic ZFS instructions: [[ZFS]]

= Update to new version of Ubuntu =

<pre>
vi /etc/update-manager/release-upgrades # set "Prompt=normal"
do-release-upgrade
</pre>

Update Ubuntu LTS 20.04 to LTS 22.04:

== daqubuntu ==

<pre>
# reboot to clear out all updates
# vi /etc/update-manager/release-upgrades # set "Prompt=normal"
# do-release-upgrade -c
Checking for a new Ubuntu release
New release '22.04 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
# do-release-upgrade
...
say yes...
...
login.defs, say "Y" (erase local changes, use packaged version)
/etc/systemd/resolved.conf, say "Y" (same as above)
firefox snap, say yes
unable to reach snap store, say "skip"
/etc/gmond.conf, say "Y"
/var/yp/Makefile, say "install the package maintainer's version"
/etc/ypserv.conf, same thing
/etc/ypserv.securenets, same thing
/etc/default/nis, same thing
/etc/speech-dispatcher/modules/mary-generic.conf, same thing
/etc/apt/apt.conf.d/50unattended-upgrades, same thing
...
278 packages are going to be removed, say yes
...
restart required, say yes
...
no ping... yes ping...
...
ssh daqubuntu, ok
apt update, fail, DNS does not work, "host security.ubuntu.com" does not resolve.
fix resolver per https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Disable_NetworkManager
apt update, apt upgrade now works, 0 packages to update
NIS does not work.
</pre>

== midm9a ==

<pre>
login.defs
firefox snap
gmond.conf
ypserv
/etc/default/nis
unattended-upgrades
amanda-security.conf
remove obsolete (no)
reboot
configure dns
reenable nis
</pre>

== daq17 ==

<pre>
firefox snap
imagemagick policy.xml
gmond.conf
chrony.conf
/var/yp/Makefile
ypserv.conf
ypserv.securenets
/etc/default/nis
50unattended-upgrades
</pre>

== daq00 ==

per https://serverpilot.io/docs/how-to-upgrade-ubuntu-20.04-to-22.04/

<pre>
do-release-upgrade -f DistUpgradeViewNonInteractive
</pre>

if it exists "too soon" without doing anything, run it without "-f xxx", most likely it does not like something about this machine. in case of daq00 it did not like how the EFI partitions were mounted. after fixing it, non-interactive upgrade was successful.

= Ubuntu package manager =

* apt-get install xxx # install package xxx
* apt-get update
* apt-get upgrade
* apt-get dist-upgrade
* apt-get autoremove # remove automatically installed packages required by a removed package
* apt-get remove xxx # remove package xxx
* apt-cache search . # list all available packages
* apt-cache show "." | grep ^Package # list al available packages
* apt-cache madison root-system # show all available versions of package root-system
* apt list # list all installed packages
* dpkg --listfiles libpng16-16 # list all files from this package
* apt list --installed # list all installed packages
* dpkg -S /bin/bash # what package provides this file?
* dpkg -L bash # what files provided by this package?
* debsums -ce # show modified config files
* apt-config dump # show apt configuration

= Ubuntu zsys =

NOTE: DO NOT USE ZSYS, see https://github.com/ubuntu/zsys/issues/218 and https://github.com/ubuntu/zsys/issues/230

* manual removal of old snapshots
<pre>
zsysctl show
zsysctl state remove xy69ye -s
zsysctl state remove xy69ye
zsysctl state remove xy69ye -u wheel
</pre>
* apt remove zsys

NOTE: old zsys snapshots must be cleaned manually, "zsysctl state remove xxx --system" is broken and does not remove user data snapshots

* manages system snapshots
* documentation: https://github.com/ubuntu/zsys
* documentation: (go to next article via link "newer" at the bottom) https://didrocks.fr/2020/05/21/zfs-focus-on-ubuntu-20.04-lts-whats-new/
* ubuntu 20.04 bug, too many snapshots cause /boot to become full and updates fail. https://github.com/ubuntu/zsys/issues/155
* solution: use custom /etc/zsys.conf, limit number of snapshots to 10, see trinatdaq:/etc/zsys.conf
* zsys commands:
<pre>
update-grub # list of all snapshots, errors if some snapshots are broken
zsysctl state remove lnc0k7 --system # remove snapshot
xemacs -nw /etc/zsys.conf; zsysctl service reload; zsysctl service gc # cause gc to run with new settings in zsys.conf
zfs list -r -t snapshot -o name,used,referenced,creation bpool/BOOT # list snapshots
zsysctl show # show snapshots
</pre>

= Ubuntu cloning =

to clone a ubuntu image:

<pre>
cd /nfsroot/lxcpet
emacs -nw etc/hostname ### change hostname
emacs -nw etc/mailname ### change hostname (debian 11)
emacs -nw etc/defaultdomain ### change the NIS domainname
emacs -nw etc/yp.conf ### change the NIS server
cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
emacs -nw root/.ssh/authorized_keys ### update root ssh keys
</pre>

= Ubuntu boot loader =

== boot from ZFS ==

* use UEFI boot with syslinux, see here: https://daq.triumf.ca/DaqWiki/index.php/SLinstall#Configure_UEFI_boot
* apt install zfs-initramfs
* update-initramfs -v -u
* ZFS structure:
<pre>
root@daq00:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 147G 1.62T 96K /
rpool/ROOT 17.8G 1.62T 96K none
rpool/ROOT/ubuntu_00aaaa 17.8G 1.62T 6.22G /
</pre>
* copy OS image to rpool/ROOT/ubuntu_00aaaa
* zfs set mountpoint=/ rpool
* zfs set mountpoint=none rpool/ROOT
* zfs set mountpoint=/ rpool/ROOT/ubuntu_00aaaa
* zfs get all | grep mountpoint
<pre>
rpool mountpoint / local
rpool/ROOT mountpoint none local
rpool/ROOT/ubuntu_00aaaa mountpoint / local
</pre>
* in linux kernel command line (syslinux.cfg), set "root=" to "root=ZFS=rpool/ROOT/ubuntu_00aaaa"

== boot from ZFS mirror ==

=== setup the EFI partitions ===

* assuming /dev/sdb is already setup for EFI boot, setup /dev/sda the same way:
* partition the second boot disk same as first boot disk:
<pre>
root@grsnis01:~# gdisk -l /dev/sdb
Found valid GPT with protective MBR; using GPT.
Number Start (sector) End (sector) Size Code Name
1 2048 1050623 512.0 MiB EF00 EFI system partition
2 1050624 3907029134 1.8 TiB 8300 Linux filesystem
root@grsnis01:~#
</pre>
* mkfs.msdos /dev/sdX1
* create mount points
<pre>
mkdir /boot/efi-sda
mkdir /boot/efi-sdb
</pre>
* add to /etc/fstab
<pre>
/dev/sda1 /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
/dev/sdb1 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
</pre>
* mount -a
* df | grep boot
<pre>
root@grsnis01:~# df | grep boot
/dev/sdb1 523248 98100 425148 19% /boot/efi-sdb
/dev/sda1 523248 4 523244 1% /boot/efi-sda
</pre>
* copy boot files to new boot disk
* cd /boot/efi-sdX; rsync -av . /boot/efi-sdY
* set BIOS to boot from "UEFI Hard drive", disable legacy boot (except for booting from USB key in legacy mode)
* if using UEFI boot syslinux per these instructions, linux kernel update has to be done manually:
* run ~/git/scripts/etc/update_efi_mirror.perl, follow instructions that it prints.

=== setup zfs partitions ===

use partitions compatible with Ubuntu "install on ZFS"

* gdisk "o" to create new GPT partition table
* gdisk "n" +512M ef00 to create EFI partition
* gdisk "n" +2G 8200 to create linux swap partition (not used)
* gdisk "n" +2G BE00 to create ZFS bpool partition
* gdisk "n" xxx BF00 create ZFS rpool partition

<pre>
# gdisk -l /dev/sda
Number Start (sector) End (sector) Size Code Name
1 2048 1050623 512.0 MiB EF00 EFI System Partition
2 1050624 5244927 2.0 GiB 8200
3 5244928 9439231 2.0 GiB BE00
4 9439232 234441614 107.3 GiB BF00
root@midm9a:~#
</pre>

=== setup zfs mirror ===

* see here: https://daq.triumf.ca/DaqWiki/index.php/ZFS#Convert_pool_from_single_to_mirror
<pre>
root@grsnis01:~# ls -l /dev/disk/by-id/ata*part2
lrwxrwxrwx 1 root root 10 Feb 19 16:47 /dev/disk/by-id/ata-WDC_WDS200T2B0A-00SM50_205007801081-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Feb 19 16:47 /dev/disk/by-id/ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 -> ../../sdb2

root@grsnis01:~# zpool status
pool: rpool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 ONLINE 0 0 0

errors: No known data errors

root@grsnis01:~# zpool attach rpool ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 /dev/disk/by-id/ata-WDC_WDS200T2B0A-00SM50_205007801081-part2

root@grsnis01:~# zpool status
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Feb 19 16:54:39 2021
12.6G scanned at 3.16G/s, 1.02G issued at 262M/s, 12.6G total
1.02G resilvered, 8.09% done, 0 days 00:00:45 to go
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801081-part2 ONLINE 0 0 0 (resilvering)

errors: No known data errors
</pre>
* wait
<pre>
root@grsnis01:~# zpool status
pool: rpool
state: ONLINE
scan: resilvered 12.7G in 0 days 00:00:40 with 0 errors on Fri Feb 19 16:55:19 2021
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801081-part2 ONLINE 0 0 0

errors: No known data errors
</pre>

== maintenance commands ==

* update-initramfs -v -u
* grub-install /dev/sda

= Convert from single to dual mirrored ZFS SSD =

Assuming Ubuntu LTS 22.04 with "instal on ZFS" option, we will
add a second SSD, configure ZFS to use both SSDs in mirrored
configuration and setup grub to boot from either SSD. This
is intended to create a full redundant system where failure
of either SSD does not break the system.

* identify first SSD
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* connect second SSD of identical size
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
/dev/sdb WD Blue SA510 2.5 250GB 22243Z803852 25 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* if second SSD is not autodetected, reboot
* Clone partition table automatically
If both SSDs are identical size, use this simpler method of duplicating the partition table:
<pre>
root@midm9b:~# sfdisk -d /dev/sda > part_table
root@midm9b:~# grep -v ^label-id part_table | sed -e 's/, *uuid=[0-9A-F-]*//' | sfdisk /dev/sdb
</pre>
The grep and sed in the second command are there to prevent disk ID and partition IDs from being cloned. Alternatively the part_table file can be edited manually to remove the label-id line and the uuid entries from the individual partitions.

* Clone partition table manually (e.g. for different size disks)
* list partition table of first SSD:
<pre>
root@midm9b:~# fdisk -l /dev/sda
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>
* create identical partitions on second SSD, use sector numbers from above.
<pre>
root@midm9b:~# gdisk /dev/sdb
GPT fdisk (gdisk) version 1.0.8

Partition table scan:
MBR: not present
BSD: not present
APM: not present
GPT: not present

Creating new GPT entries in memory.

Command (? for help): n
Partition number (1-128, default 1):
First sector (34-488397134, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-488397134, default = 488397134) or {+-}size{KMGTP}: 1050623
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): ef00
Changed type of partition to 'EFI system partition'

Command (? for help): n
Partition number (2-128, default 2):
First sector (34-488397134, default = 1050624) or {+-}size{KMGTP}:
Last sector (1050624-488397134, default = 488397134) or {+-}size{KMGTP}: 5244927
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 8200
Changed type of partition to 'Linux swap'

Command (? for help): n
Partition number (3-128, default 3):
First sector (34-488397134, default = 5244928) or {+-}size{KMGTP}:
Last sector (5244928-488397134, default = 488397134) or {+-}size{KMGTP}: 9439231
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): be00
Changed type of partition to 'Solaris boot'

Command (? for help): n
Partition number (4-128, default 4):
First sector (34-488397134, default = 9439232) or {+-}size{KMGTP}:
Last sector (9439232-488397134, default = 488397134) or {+-}size{KMGTP}:
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): bf00
Changed type of partition to 'Solaris root'

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sdb.
The operation has completed successfully.
root@midm9b:~# fdisk -l /dev/sda /dev/sdb
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root

Disk /dev/sdb: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: EB251739-30C6-422F-A505-5887B5A0B603

Device Start End Sectors Size Type
/dev/sdb1 2048 1050623 1048576 512M EFI System
/dev/sdb2 1050624 5244927 4194304 2G Linux swap
/dev/sdb3 5244928 9439231 4194304 2G Solaris boot
/dev/sdb4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>
* identify second SSD partitions
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part3
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part3 -> ../../sda3
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 -> ../../sdb3
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
</pre>
* convert bpool from single disk to mirrored disk:
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~# zpool attach bpool 99e03dc0-7d4d-f24b-8fa1-f042b9f135db /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3
root@midm9b:~# zpool status bpool
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors
</pre>
* convert rpool
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
root@midm9b:~# zpool attach rpool f6fd54f8-3af7-b943-ae3d-a4e480537fb9 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4
root@midm9b:~# zpool status rpool
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Jan 20 19:40:45 2023
5.83G scanned at 664M/s, 2.92M issued at 332K/s, 9.11G total
0B resilvered, 0.03% done, no estimated completion time
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~#
</pre>
* wait for resilver to complete
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
scan: resilvered 9.65G in 00:00:36 with 0 errors on Fri Jan 20 19:41:21 2023
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
</pre>
* enable booting from second SSD: (instead of /dev/sda1, /dev/sdb1, use UUID=xxx)
<pre>
root@midm9b:~# mkfs.msdos /dev/sdb1
root@midm9b:~# mkdir /boot/efi-sda
root@midm9b:~# mkdir /boot/efi-sdb
root@midm9b:~# echo "/dev/sda1 /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1" >> /etc/fstab
root@midm9b:~# echo "/dev/sdb1 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1" >> /etc/fstab
root@midm9b:~# mount -a
root@midm9b:~# df -kl
Filesystem 1K-blocks Used Available Use% Mounted on
...
/dev/sda1 523244 13720 509524 3% /boot/efi
/dev/sdb1 523244 4 523240 1% /boot/efi-sdb
...
root@midm9b:~# rsync -av /boot/efi/ /boot/efi-sdb/
sending incremental file list
EFI/
...
root@midm9b:~# ls -l /boot/efi-sda
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~# ls -l /boot/efi-sdb
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~#
</pre>
* setup script to update grub on second SSD, it must be run manually after every kernel update
<pre>
root@midm9b:~# ln -s ~/git/scripts/etc/update_efi_grub.perl ~/
root@midm9b:~# ~/update_efi_grub.perl -u
EFI dir: /boot/efi-sda
/boot/efi-sda: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sda/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sda: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sda/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
EFI dir: /boot/efi-sdb
/boot/efi-sdb: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sdb/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sdb: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sdb/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
root@midm9b:~#
</pre>

= Disable NetworkManager =

NOTE: THIS IS BROKEN IN UBUNTU LTS 22.04

NetworkManager is useful for configuring dynamic
network interfaces, i.e. laptops that often move
between networks, or connect to multiple choice
of wifi networks, etc.

For machines with statically configured network interfaces,
NetworkManager is not necessary.

As it has been observed to become confused and observed
to malfunction when network links go up and down (it keeps
unnecessarily reconfiguring the ip address, etc), it can
be usefuil to disable it.

* list all network interfaces
<pre>
# /bin/ls -1 /sys/class/net/
enp0s31f6
lo
</pre>
* edit /etc/network/interfaces:
<pre>
rename enp0s31f6=eth0
auto eth0
iface eth0 inet static
address 142.90.120.94/19
gateway 142.90.100.18
</pre>
* statically configure systemd-resolved
** create /etc/systemd/resolved.conf.d/resolved.conf with this contents:
<pre>
[Resolve]
DNS=142.90.100.19
Domains=triumf.ca
</pre>
** systemctl restart systemd-resolved
** resolvectl
** systemd-analyze cat-config systemd/resolved.conf
* disable NetworkManager
<pre>
systemctl disable NetworkManager
</pre>
* reboot

= Configure ECC memory =

== Configure EDAC ==

* apt install edac-utils

=== Intel i3-2120 ===
<pre>
root@musr00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X9SCL/X9SCM
root@musr00:~# edac-ctl --status
edac-ctl: drivers not loaded.
</pre>

=== Intel E-2236 ===
<pre>
root@daq00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SCM-F
root@daq00:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq00:~# edac-util
edac-util: No errors to report.
root@daq00:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
</pre>
* check edac sysfs files (Intel)
<pre>
root@daq00:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_noinfo_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 max_location
-r--r--r-- 1 root root 4096 Jan 25 15:10 mc_name
drwxr-xr-x 2 root root 0 Jan 25 15:10 power
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank0
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank1
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank2
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank3
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank4
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank5
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank6
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank7
--w------- 1 root root 4096 Jan 25 15:10 reset_counters
-r--r--r-- 1 root root 4096 Jan 25 15:10 seconds_since_reset
-r--r--r-- 1 root root 4096 Jan 25 15:10 size_mb
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Jan 25 15:10 uevent
root@daq00:~#
</pre>

=== Intel E3-1270 v6 ===
<pre>
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --status
edac-ctl: drivers are loaded.
root@grsnis01:~# edac-util
edac-util: No errors to report.
root@grsnis01:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@grsnis01:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 max_location
-r--r--r-- 1 root root 4096 Feb 19 12:35 mc_name
drwxr-xr-x 2 root root 0 Feb 19 12:35 power
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank0
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank1
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank2
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank3
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank4
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank5
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank6
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank7
--w------- 1 root root 4096 Feb 19 12:35 reset_counters
-r--r--r-- 1 root root 4096 Feb 19 12:35 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 19 12:35 size_mb
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 19 12:35 uevent
root@grsnis01:~#
</pre>

=== Intel E3-1245 v6 ===

<pre>
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --status
edac-ctl: drivers are loaded.
[root@alphagdaq ~]# edac-util
edac-util: No errors to report.
[root@alphagdaq ~]# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
[root@alphagdaq ~]# ras-mc-ctl --layout
+-----------------------------------------------+
| mc0 |
| csrow0 | csrow1 | csrow2 | csrow3 |
----------+-----------------------------------------------+
channel1: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
----------+-----------------------------------------------+
[root@alphagdaq ~]# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#3channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#3channel#1 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#2channel#0 0 0
[root@alphagdaq ~]# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SSH-F
[root@alphagdaq ~]# ras-mc-ctl --summary
DBD::SQLite::db prepare failed: no such table: mc_event at /usr/sbin/ras-mc-ctl line 1129.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1130.
[root@alphagdaq ~]#
</pre>

=== AMD 3700X ===

(memory is non-ECC)

<pre>
root@daq13:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@daq13:~#
root@daq13:~#
root@daq13:~# edac-ctl --status
edac-ctl: drivers not loaded.
root@daq13:~# edac-util
edac-util: Error: No memory controller data found.
root@daq13:~# edac-util -s
edac-util: EDAC drivers loaded. No memory controllers found
root@daq13:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 2 root root 0 Jan 25 15:26 power
lrwxrwxrwx 1 root root 0 Jan 21 16:16 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Jan 21 16:16 uevent
</pre>

(memory is ECC)

<pre>
root@trinatdaq:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@trinatdaq:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@trinatdaq:~# edac-util
edac-util: No errors to report.
root@trinatdaq:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Dec 15 13:04 mc0
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
lrwxrwxrwx 1 root root 0 Dec 13 18:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dec 13 18:31 uevent
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_noinfo_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 max_location
-r--r--r-- 1 root root 4096 Dec 15 13:04 mc_name
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank4
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank5
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank6
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank7
--w------- 1 root root 4096 Dec 15 13:04 reset_counters
-rw-r--r-- 1 root root 4096 Dec 15 13:04 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Dec 15 13:04 seconds_since_reset
-r--r--r-- 1 root root 4096 Dec 15 13:04 size_mb
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Dec 15 13:04 uevent
root@trinatdaq:~#
</pre>

=== AMD 5000G ===

* no linux driver for AMD 5000-series "G" CPU
* no mention of ECC in the BIOS settings
* unclear status of ECC support in AMD documentation (sais only "pro" "G" CPUs have ECC)
* unclear status of ECC support in ASUS documentation (web page out of date)

=== AMD 5600X ===

<pre>
root@daq17:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-XE GAMING WIFI
root@daq17:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq17:~# edac-util
edac-util: No errors to report.
root@daq17:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@daq17:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Aug 19 19:27 mc0
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
lrwxrwxrwx 1 root root 0 May 10 10:11 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 May 10 10:11 uevent
root@daq17:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_noinfo_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 max_location
-r--r--r-- 1 root root 4096 Aug 19 19:27 mc_name
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank4
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank5
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank6
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank7
--w------- 1 root root 4096 Aug 19 19:27 reset_counters
-rw-r--r-- 1 root root 4096 Aug 19 19:27 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Aug 19 19:27 seconds_since_reset
-r--r--r-- 1 root root 4096 Aug 19 19:27 size_mb
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Aug 19 19:27 uevent
root@daq17:~#
</pre>

=== AMD 3955WX ===

<pre>
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --status
edac-ctl: drivers are loaded.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util
edac-util: No errors to report.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@alphasuperdaq:~/git/scripts/quotareport# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 19 root root 0 Dez 12 04:48 mc0
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
lrwxrwxrwx 1 root root 0 Dez 9 05:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dez 9 05:31 uevent
root@alphasuperdaq:~/git/scripts/quotareport#
root@alphasuperdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 max_location
-r--r--r-- 1 root root 4096 Feb 28 22:19 mc_name
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank0
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank1
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank10
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank11
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank12
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank13
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank14
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank15
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank2
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank3
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank4
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank5
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank6
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank7
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank8
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank9
--w------- 1 root root 4096 Feb 28 22:19 reset_counters
-rw-r--r-- 1 root root 4096 Feb 28 22:19 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Feb 28 22:19 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 28 22:19 size_mb
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 28 22:19 uevent
root@alphasuperdaq:~#
root@alphasuperdaq:~# ras-mc-ctl --layout
Use of uninitialized value $max_pos[3] in modulus (%) at /usr/sbin/ras-mc-ctl line 868.
Use of uninitialized value $d in numeric ge (>=) at /usr/sbin/ras-mc-ctl line 869.
Use of uninitialized value $d in sprintf at /usr/sbin/ras-mc-ctl line 872.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| mc0 |
| csrow0 | csrow1 |
| channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 | channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

0: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
root@alphasuperdaq:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#2 0 0
mc#0csrow#1channel#7 0 0
mc#0csrow#0channel#3 0 0
mc#0csrow#1channel#4 0 0
mc#0csrow#1channel#2 0 0
mc#0csrow#0channel#7 0 0
mc#0csrow#1channel#3 0 0
mc#0csrow#0channel#4 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#5 0 0
mc#0csrow#0channel#6 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#0channel#5 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#6 0 0
root@alphasuperdaq:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: ASUSTeK COMPUTER INC. model Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@alphasuperdaq:~#
</pre>

== Configure rasdaemon ==

<pre>
apt install rasdaemon
</pre>
<pre>
systemctl enable rasdaemon
systemctl restart rasdaemon
systemctl status rasdaemon
</pre>

<pre>
● rasdaemon.service - RAS daemon to log the RAS events
Loaded: loaded (/lib/systemd/system/rasdaemon.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-01-25 15:16:37 PST; 3min 5s ago
Main PID: 2477175 (rasdaemon)
Tasks: 1 (limit: 76958)
Memory: 17.1M
CGroup: /system.slice/rasdaemon.service
└─2477175 /usr/sbin/rasdaemon -f -r

Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Listening to events for cpus 0 to 11
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mc_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording aer_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording extlog_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mce_record events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording arm_event events
</pre>

== Get reports ==

* Intel 2x32GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --layout
+-------------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-------------------------+
channel1: | 16384 MB | 16384 MB |
channel0: | 16384 MB | 16384 MB |
----------+-------------------------+
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#0channel#1 0 0
root@daq00:~#
</pre>

* Intel 4x16GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#1 0 0
mc#0csrow#2channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#3channel#0 0 0
mc#0csrow#3channel#1 0 0
root@daq00:~#
root@daq00:~# ras-mc-ctl --layout
+-----------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-----------------------+
channel1: | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB |
----------+-----------------------+
root@daq00:~#
root@daq00:~#
root@daq00:~#
root@daq00:~# ras-mc-ctl --print-labels
ras-mc-ctl: Error: No dimm labels for Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@daq00:~#
</pre>

note: ubuntu LTS 22.04 DBD::SQLite::db error is not there.

= sensors =

== ASUS P9X79 WS ==

* https://www.asus.com/supportonly/P9X79%20WS/HelpDesk_Manual/
* BIOS version 4802
* modprobe nct6775
* modprobe coretemp

<pre>
root@daq14:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 0: +29.0°C (high = +82.0°C, crit = +100.0°C)
Core 1: +24.0°C (high = +82.0°C, crit = +100.0°C)
Core 2: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +82.0°C, crit = +100.0°C)

nouveau-pci-0200
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.00 V)
temp1: +39.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

nct6776-isa-0290
Adapter: ISA adapter
Vcore: 1.04 V (min = +0.00 V, max = +1.74 V)
in1: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
AVCC: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
+3.3V: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 2.04 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 904.00 mV (min = +0.00 V, max = +0.00 V) ALARM
3VSB: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
Vbat: 3.30 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1265 RPM (min = 0 RPM)
fan2: 1909 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +34.0°C (high = +0.0°C, hyst = +0.0°C) ALARM sensor = thermistor
CPUTIN: +58.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermal diode
AUXTIN: +31.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
PECI Agent 0: +31.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +96.0°C)
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

root@daq14:~#
</pre>

= Enable CPU turbo mode =

* Intel CPU has a nominal CPU frequency (i.e. 3.4GHz) and a turbo-boost CPU frequency (i.e. 4.0GHz). Here we will enable this turbo-boost mode.
* Find out CPU capability
<pre>
root@daq01:~# lscpu | grep Hz
Model name: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
CPU MHz: 3965.803
CPU max MHz: 4000.0000
CPU min MHz: 800.0000
root@daq01:~#
</pre>
* Look up this CPU in the Intel ARK database - google for the CPU model name, i.e.
https://ark.intel.com/content/www/us/en/ark/products/88196/intel-core-i7-6700-processor-8m-cache-up-to-4-00-ghz.html
* Find current frequency settings:
<pre>
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "powersave" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 2.72 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
root@daq01:~#
</pre>
* Note the following:
** current governor is "powersave"
** "performance" governor is available
** "boost state support" is supported and active.
* Confirm CPU frequency governor:
<pre>
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
powersave
powersave
powersave
powersave
powersave
powersave
powersave
powersave
root@daq01:~#
</pre>
* Change governor to "performance":
<pre>
root@daq01:~# cpupower frequency-set --governor performance
Setting cpu: 0
Setting cpu: 1
Setting cpu: 2
Setting cpu: 3
Setting cpu: 4
Setting cpu: 5
Setting cpu: 6
Setting cpu: 7
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
performance
performance
performance
performance
performance
performance
performance
performance
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "performance" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 3.93 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
</pre>
* monitor CPU frequency:
<pre>
root@daq01:~# cpupower monitor
| Nehalem || Mperf || Idle_Stats
CPU| C3 | C6 | PC3 | PC6 || C0 | Cx | Freq || POLL | C1 | C1E | C3 | C6 | C7s | C8
0| 0.00| 0.00| 0.00| 0.00|| 88.80| 11.20| 3973|| 0.00| 0.00| 0.01| 0.02| 0.31| 0.00| 4.25
4| 0.00| 0.00| 0.00| 0.00|| 4.70| 95.30| 3945|| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 95.03
1| 0.73| 3.70| 0.00| 0.00|| 4.52| 95.48| 3864|| 0.00| 0.01| 1.19| 0.44| 2.82| 0.00| 90.23
5| 0.73| 3.70| 0.00| 0.00|| 0.37| 99.63| 3807|| 0.00| 0.00| 0.03| 0.09| 1.70| 0.00| 97.64
2| 2.28| 12.86| 0.00| 0.00|| 1.41| 98.59| 3829|| 0.00| 0.86| 3.17| 0.46| 7.70| 0.00| 85.87
6| 2.28| 12.86| 0.00| 0.00|| 2.88| 97.12| 3856|| 0.00| 0.11| 4.56| 2.15| 10.31| 0.00| 78.99
3| 1.33| 4.81| 0.00| 0.00|| 0.99| 99.01| 3804|| 0.00| 0.49| 0.79| 0.01| 1.03| 0.00| 96.12
7| 1.34| 4.81| 0.00| 0.00|| 1.26| 98.74| 3818|| 0.00| 0.01| 2.32| 0.47| 5.02| 0.00| 90.06
root@daq01:~#
</pre>
* check that the CPU is not overheating:
<pre>
root@daq01:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 1: +38.0°C (high = +84.0°C, crit = +100.0°C)
Core 2: +34.0°C (high = +84.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +84.0°C, crit = +100.0°C)
</pre>
* congratulations, we are running at 4 GHz now!

= Setup ubuntu as gateway to private network =

See also:
* https://daq.triumf.ca/DaqWiki/index.php/VME-CPU#Setup_the_boot_host_computer_.28el7.29
* http://www.triumf.info/wiki/DAQwiki/index.php/Dhcpd_on_eth1

== Steps to do ==

* assign network numbers to the private network, i.e. 192.168.1.x, 192.168.2.x, etc
* (on the gateway machine, each private network interface has to have a different network number)
* (each network interface can have multiple networks attached, via VLANs or via eth0:0, eth0:1 constructs)
* assign IP addresses on the private network, save them in /etc/hosts i.e. "hvps 192.168.1.10"
* (for simplicity, assign 192.168.1.1 to the gateway machine itself)
* (IP addresses 192.168.1.0 and 192.168.1.255 are "special", do not use them)
* setup DNS server (dnsmasq) to serve contents of /etc/hosts via DNS (otherwise, many programs will see inconsistent name to IP address mapping)
* setup DHCP server (ISC dhcpd or dnsmasq) to give out the IP addresses
* setup tftp, pxelinux and NFS for diskless booting
* setup time server (chronyd) to provide common time to all devices
* setup NAT so machines on private network can access the internet (to get OS updates, etc)
* setup NIS and NFS so machines on the private network can use common home directories
* setup rsync backup of machines on the private network

== setup hosts ==

* edit /etc/hosts
<pre>
192.168.1.101 dsfe01
... and so forth
</pre>

== setup dns and dhcp ==

* apt install dnsmasq
* edit /etc/dnsmasq.conf
<pre>
# /etc/dnsmasq.conf
# DNS settings
#port=0 # disable DNS function
port=53 # enable DNS function
domain-needed
bogus-priv
no-resolv
server=142.90.100.19
# DHCP settings
interface=enp1s0f0 # DHCP interface
#dhcp-range=192.168.1.50,192.168.1.150,infinite
dhcp-range=192.168.1.0,static
#log-dhcp
quiet-dhcp
#dhcp-ignore=tag:!known
dhcp-boot=pxelinux.0
#dhcp-host=ac:1f:6b:9e:7f:4a,192.168.1.100,10m
dhcp-host=ac:1f:6b:9e:7f:4a,dsfe01,infinite
# TFTP settings
enable-tftp
tftp-root=/tftpboot
</pre>
* #mkdir /zssd/tftpboot ### per tftp-root (if no ZFS)
* zfs create -o mountpoint=/tftpboot rpool/tftpboot ### (if root is ZFS)
* systemctl stop systemd-resolved.service
* systemctl disable systemd-resolved.service
* rm /etc/resolv.conf
* create new /etc/resolv.conf with this contents:
<pre>
nameserver 127.0.0.1
search snolab.ca
</pre>
* systemctl enable dnsmasq
* systemctl restart dnsmasq

== setup chronyd ==

* enable ntp server:
* configure and enable chronyd per instructions above
* echo "allow 192.168.1.0/24" > /etc/chrony/conf.d/allow-localhost.conf
* systemctl restart chronyd
* chronyc tracking ### wait until time is synchronized (a few seconds)

== setup diskless network booting ==

=== setup pxelinux ===
<pre>
cd ~
wget https://www.kernel.org/pub/linux/utils/boot/syslinux/4.xx/syslinux-4.03.tar.bz2
tar xjvf syslinux-4.03.tar.bz2
cd syslinux-4.03
cp -pv ./core/pxelinux.0 ./com32/hdt/hdt.c32 ./memdisk/memdisk ./com32/menu/menu.c32 /zssd/tftpboot/
</pre>
* cd /zssd/tftpboot
<pre>
wget http://ladd00.triumf.ca/tftpboot/memtest86+-4.20.iso.zip
wget http://ladd00.triumf.ca/tftpboot/memtest86+-5.01.iso.gz
wget http://ladd00.triumf.ca/tftpboot/modules.alias
wget http://ladd00.triumf.ca/tftpboot/modules.pcimap
wget http://ladd00.triumf.ca/tftpboot/pci.ids
</pre>
* mkdir pxelinux.cfg
* emacs -nw pxelinux.cfg/default
<pre>
default menu.c32
prompt 0

menu title Welcome to the DSVSLICE PXE boot menu

timeout 50

label hdt
kernel hdt.c32

label memtest86+-5.01
kernel memdisk iso initrd=memtest86+-5.01.iso.gz

label memtest86+-4.20
kernel memdisk iso initrd=memtest86+-4.20.iso.zip

label vmlinuz-5.3.0-26-generic
menu default
kernel vmlinuz-5.3.0-26-generic
append initrd=initrd.img-5.3.0-26-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.1.1:/zssd/nfsroot/dsfe01 toram ip=dhcp panic=60 BOOTIF=enp1s0f0

#end
</pre>

=== setup linux kernel ===

* copy the kernel files
<pre>
cd /boot
rsync -av config* initrd* System.map* vmlinuz* /zssd/tftpboot/
</pre>
* cd /zssd/tftpboot
* chmod a+r *

=== setup nfs ===

* apt-get install nfs-kernel-server
* emacs -nw /etc/exports
<pre>
/zssd/nfsroot/dsfe01 dsfe01(rw,no_root_squash,async,no_subtree_check)
</pre>
* enable services
<pre>
systemctl enable nfs-server
systemctl enable nfs-mountd
systemctl enable nfs-idmapd
systemctl restart nfs-server
systemctl restart nfs-mountd
systemctl restart nfs-idmapd
</pre>
* after editing /etc/exports, run
<pre>
exportfs -av
</pre>

=== setup userland ===

* zfs create zssd/nfsroot
* zfs set dedup=verify zssd/nfsroot ### enable deduplication to save disk space because most linux images have mostly identical files
* clone ubuntu
<pre>
mkdir /zssd/nfsroot/dsfe01
cd /
rsync -avx . /zssd/nfsroot/dsfe01
</pre>
* edit config files:
* cd /zssd/nfsroot/dsfe01
* emacs -nw etc/hostname ### change to dsfe01
* emacs -nw etc/mailname ### change to dsfe01
* emacs -nw etc/yp.conf ### change daq00.triumf.ca to musr00.triumf.ca
* emacs -nw etc/defaultdomain ### change to MUSR-NIS
* cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
* emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
* emacs -nw root/.ssh/authorized_keys ### update root ssh keys
* emacs -nw etc/fstab ### add this
<pre>
192.168.1.1:/zssd/nfsroot/dsfe01 / nfs defaults,nolock 0 0
</pre>
* emacs -nw etc/chrony/chrony.conf
** comment-out all "pool" and "server" entries
** add entry "server 192.168.1.1 iburst"

After dsfe01 is booted:

* disable services:
<pre>
systemctl disable apache2
systemctl disable dnsmasq
systemctl disable zfs-import-cache
</pre>

To setup additional machines, clone dsfe01 instead of cloning the gateway machine

=== Allow manpages to be viewed ===

If <code>/</code> is mounted over NFS, <code>man</code> will report a permission error. Fix it with:

<pre>
ln -s /etc/apparmor.d/usr.bin.man /etc/apparmor.d/disable/
apparmor_parser -R /etc/apparmor.d/usr.bin.man
</pre>

== setup shared home directory ==

=== on the gateway machine ===
* define netgroups
* emacs -nw /etc/netgroup
<pre>
dsfe (dsfe01,,) (dsfe02,,)
</pre>
* emacs -nw /etc/nsswitch.conf ### edit the netgroup line to read:
<pre>
netgroup: files
</pre>
* export the home directories:
* emacs -nw /etc/exports ### add this:
<pre>
/zssd/home1 @dsfe(rw,no_root_squash,async,no_subtree_check)
</pre>
* exportfs -rc

=== on the frontend machine ===

* mkdir /home
* emacs -nw /etc/fstab ### add this:
<pre>
192.168.1.1:/zssd/home1 /home nfs defaults 0 0
</pre>
* mount -a

== setup NAT ==

NAT allows machines on the private network to connect to the internet: https://en.wikipedia.org/wiki/Network_address_translation

In these examples:
* replace "eno1" with name of the outgoing interface (the one connected to the TRIUMF network).
* replace "enp11s0" with name of the private network interface (192.168.1.x network)

* emacs -nw /etc/rc.local ### add this:
<pre>
# /etc/rc.local

/sbin/iptables -t nat -A POSTROUTING -o eno1 -j MASQUERADE
iptables -L -v

# uncomment following lines if machine has prohibitive FORWARD rules:
#/sbin/iptables -I FORWARD -i eno1 -o enp11s0 -m state --state RELATED,ESTABLISHED -j ACCEPT
#/sbin/iptables -I FORWARD -i enp11s0 -o eno1 -j ACCEPT
#iptables -L -v

iptables -L -v
sysctl -w net.ipv4.ip_forward=1
#sysctl -a | grep forward

sh /etc/firewall-rfc1918.sh

# end
</pre>
* emacs -nw /etc/firewall-rfc1918.sh
<pre>
# firewall-rfc1918.sh

# prevent RFC1918 private network IP addresses from
# going in and out from our uplink.

ETH=eno1

iptables -F in-rfc1918
iptables -N in-rfc1918
iptables -A in-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A in-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A in-rfc1918 --dst 192.168.0.0/16 -j REJECT
iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -I INPUT -j in-rfc1918 -i $ETH

iptables -F out-rfc1918
iptables -N out-rfc1918
iptables -A out-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A out-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A out-rfc1918 --dst 192.168.0.0/16 -j REJECT
iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -I OUTPUT -j out-rfc1918 -o $ETH

iptables -L -v

#end
</pre>

= KVM =

<pre>
apt install cpu-checker

root@daq13:~# kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
root@daq13:~#

(if not, shutdown, go into BIOS settings, enable CPU virtualization)

apt install virtinst ### will install many packages
apt install libvirt-clients libvirt-daemon-system-systemd libvirt-daemon qemu qemu-kvm libvirt-daemon-system virtinst bridge-utils

root@daq13:/home1/wheel# virsh list --all
Id Name State
------------------------------
1 ubuntu-guest running

apt install virt-manager

virt-install --name ubuntu-guest --os-variant ubuntu20.04 --vcpus 2 --ram 2048 --location /daq/daqstore/olchansk/linux/Ubuntu/ubuntu-20.04.3-desktop-amd64.iso --network bridge=virbr0,model=virtio --graphics none --extra-args='console=ttyS0,115200n8 serial'

virtual machine will start, boot, etc
to get out of it, CTRL + Shift followed by ]

ssh wheel@daq13
virt-manager

run virt-install again, omit "--graphics none", open graphics console from virt-manager, it booted into ubuntu installer desktop

virt-install --name test10 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --filesystem /kvm_ladd00,/ --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial" --graphics none

virt-install --name test14 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --disk /tmp/xxx/ladd00.img,bus=sata --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial rdshell" --graphics none --check path_in_use=off
</pre>

build image

<pre>
dd if=/dev/zero of=/tmp/xxx/ladd00.img bs=1024M count=20
mkfs.ext3 /tmp/xxx/ladd00.img ### ext4 fails to mount by SL6 kernel, "unknown ext4 options"
cd /kvm_ladd00/
mount -o loop /tmp/xxx/ladd00.img /mnt/tmp
rsync -av . /mnt/tmp/ --delete
umount /mnt/tmp
</pre>

on the guest, configure network: /etc/rc.local
<pre>
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

ifconfig eth2 192.168.122.2
route add -net 0.0.0.0 gw 192.168.122.1
ifconfig -a
netstat -rn

# end
</pre>

= ARM cross-compiler =

<pre>
apt install libgcc-9-dev-arm64-cross
apt install gcc-arm-linux-gnueabi
apt install gcc-arm-linux-gnueabihf
apt install g++-arm-linux-gnueabihf
apt install g++-arm-linux-gnueabi
</pre>

<pre>
arm-linux-gnueabi-gcc -o ttcp1 ttcp.c -march=armv7 -static
arm-linux-gnueabi-gcc -o memcpy.armv7 memcpy.cc -march=armv7 -static -O2
</pre>

= 32-bit intel cross-compiler =

Ubuntu 22.04:

<pre>
apt install libstdc++-11-dev:i386
apt install zlib1g-dev:i386
</pre>

for MIDAS, use "make linux32"

Ubuntu

2023-11-15T18:43:57Z

Lmartin: 22.04 doesn't have python3.8 or python

= About Ubuntu =

AAA

= Ubuntu version =

<pre>
lsb_release -a
uname -a
</pre>

= Ubuntu installer =

* updated for Ububtu LTS 20.04.01, 22.04.1

* download the latest Ubuntu LTS desktop installer iso image
* dd the image to a USB key
* power down, disconnect all disks (all HDDs, all SSDs, all M.2)
* connect the SSD to be used as system disk
* if system will use mirrored SSDs (using ZFS mirror), leave second SSD disconnected, we will activate it later
* power up
* boot from USB key in legacy mode or UEFI mode (select this in the BIOS boot menu - F8 for ASUS, F11 for Supermicro)
* follow the instruction:
* "try ubuntu or install ubuntu" - choose "install"
* select language - accept default
* "updates and other software" - accept default settings ("normal install")
* "installation type" - select "advanced features" and "experimental: use ZFS"
* accept partition choice
* "where are you?" - select "Vancouver" (PST time zone)
* "who are you?" - leave all fields blank, except "username" set to "wheel", "password" set to the root password. hostname will be set later after configuring the network
* installation runs in a few minutes, when finished, reboot
* login as user wheel
* answer annouying questions:
* "livepatch" - say "next"
* "help improve" - select "do not send", say "next"
* "privacy" - leave "location" as "off", say "next"
* "ready to go", say "done"
* right-click on the desktop, say "open in terminal", a shell will open
* say "sudo /bin/bash", enter the root password, you now have the root shell
* run nm-connection-editor to configure the network. use netmask 255.255.224.0, gateway 142.90.100.18, DNS 142.90.100.19, search path "triumf.ca"
* after network is up (can ping ladd00), continue with post-installation steps below

= Install instructions =

== prepare ==
<pre>
apt update
apt upgrade
</pre>

== install ssh ==
<pre>
apt install ssh
</pre>

== configure hostname ==
<pre>
vi /etc/hostname
</pre>

== disable swap ==

ubuntu installer creates a 2 GB swap partition, not useful
on 32-64 GB machine, disable it:
<pre>
vi /etc/fstab ### comment out the "swap" line
</pre>

== maybe reboot ==

this is a good point to reboot the machine to boot
the latest kernel and to set the correct hostname

== install etckeeper ==

keep contents of /etc in a git repository:

<pre>
apt -y install etckeeper
</pre>

== set timezone ==
<pre>
timedatectl list-timezones | grep -i vancouver
timedatectl set-timezone America/Vancouver
</pre>

== install time synchronization ==
<pre>
apt -y install chrony
echo server time1.triumf.ca iburst >> /etc/chrony/chrony.conf
echo server time2.triumf.ca iburst >> /etc/chrony/chrony.conf
echo server time3.triumf.ca iburst >> /etc/chrony/chrony.conf
systemctl disable systemd-timesyncd.service
systemctl stop systemd-timesyncd.service
systemctl disable ntp
systemctl stop ntp
systemctl enable chrony
systemctl restart chrony
chronyc sources
chronyc tracking
</pre>

== reenable systemd-timesyncd ==

ONLY IF CHRONY DOES NOT WORK

<pre>
apt remove chrony
systemctl enable systemd-timesyncd.service
systemctl restart systemd-timesyncd.service
systemctl status systemd-timesyncd.service
timedatectl status
timedatectl timesync-status
</pre>

== enable outgoing email (debian 11) ==

this is different from ubuntu 20. it uses /etc/mailname and it hardwires the hostname into main.cf.

== enable outgoing email ==

* TRIUMF: use smtp.triumf.ca
* CERN: use cernmx.cern.ch

<pre>
apt install postfix ### select "satellite system", enter full hostname "xxx.triumf.ca", enter "smtp.triumf.ca"
apt install mailutils
dpkg-reconfigure postfix ### (if postfix already installed)
</pre>
<pre>
echo olchansk@triumf.ca lindner@triumf.ca bsmith@triumf.ca >> ~root/.forward
mailx root
test
^D
</pre>

== enable ping for all users (debian 11) ==

Without this tweak, Debian will report "operation not permitted" if a user tries to ping somewhere.

<pre>
echo 'net.ipv4.ping_group_range = 0 1000' > /etc/sysctl.d/99-ping.conf
</pre>

== install missing packages ==

(apt eats terminal input, even the "yes |" trick does not quite work,
repeat the following commands until they report that everything
is installed)

<pre>
yes | apt -y install ssh tcsh ethtool ncat rsync strace net-tools sysstat smartmontools lm-sensors traceroute time minicom screen git lsof debsums tmux
yes | apt -y install lsb-release
yes | apt -y install flex bison
yes | apt -y install neofetch
yes | apt -y install snmp snmp-mibs-downloader
yes | apt -y install git subversion g++ gfortran cmake doxygen
yes | apt -y install curl libcurl4 libcurl4-openssl-dev
yes | apt -y install mariadb-client ### mysql client
yes | apt -y install libz-dev sqlite3 libsqlite3-dev unixodbc-dev
yes | apt -y install libssl-dev
yes | apt -y install emacs xemacs21 joe
yes | apt -y install gnuplot dos2unix
yes | apt -y install mutt bsd-mailx # email clients
yes | apt -y install liblz4-tool pbzip2
yes | apt -y install libc6-dev-i386 # otherwise no /usr/include/sys/types.h
yes | apt -y install libreadline-dev
yes | apt -y install chromium-browser chromium-codecs-ffmpeg-extra
yes | apt -y install ubuntu-mate-themes
yes | apt -y install libmotif-dev libxmu-dev
yes | apt -y install libusb-dev libusb-1.0-0-dev
yes | apt -y install xfig gsfonts-x11 gsfonts-other # install fonts for xfig
yes | apt -y install libjson-perl
yes | apt -y install libgsl-dev # additional GNU Scientific Library
yes | apt -y install qt5-default # Qt development
yes | apt -y install python3-full python3-dev python3-dbg python3-pip ### for pyROOT
yes | apt -y install imagemagick imagemagick-common ckeditor # for elog
yes | apt -y install libjpeg-dev libjpeg-progs libjpeg-tools
yes | apt -y install linux-tools-common linux-tools-generic # cpupower frequency-info
yes | apt -y install rdesktop remmina remmina-plugin"*" # requested by POL
</pre>

Ubuntu LTS 20.04:
<pre>
yes | apt -y install linux-image-generic-hwe-20.04 linux-tools-virtual-hwe-20.04 # enable linux 5.11 series kernel
</pre>

== install git/scripts ==
<pre>
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd scripts
git pull
</pre>

== disable swap (debian 11) ==

* on 64 GB RAM machines swap is not useful
* on machines booted from network (NFS-ROOT), swap does not work
* on machines running from flash (RPi, etc), flash is too slow for useful swap
* swap configured by linux installers invariably has wrong size and is not useful

<pre>
systemctl disable dphys-swapfile
systemctl stop dphys-swapfile
dphys-swapfile uninstall
</pre>

== configure DNS ==

<pre>
cd ~/git/scripts
git pull
mkdir /etc/systemd/resolved.conf.d
cp etc/resolved-triumf.conf /etc/systemd/resolved.conf.d/
systemctl restart systemd-resolved
resolvectl
#systemd-analyze cat-config systemd/resolved.conf
</pre>

== install ganglia ==

<pre>
yes | apt-get -y install ganglia-monitor
systemctl enable ganglia-monitor
</pre>
<pre>
cd ~root/git/scripts
git pull
cp etc/gmond-ubuntu.conf /etc/ganglia/gmond.conf
ln -s ganglia/gmond.conf /etc
# ln -s arm-linux-gnueabihf/ganglia /usr/lib/ganglia ### fix path for ARM CPU
# ln -s i386-linux-gnu/ganglia /usr/lib/ganglia ### fix path for 32-bit Intel CPU
systemctl restart ganglia-monitor
systemctl status ganglia-monitor
ps -efw | grep gmond
</pre>
<pre>
cd ~root/git/scripts/ganglia
make install
./ganglia-all.perl
</pre>

== install gonodeinfo ==

* go to https://bitbucket.org/dd1/gonodeinfo follow instructions:
<pre>
yes | apt-get -y install golang
mkdir ~/git
cd ~/git
git clone https://bitbucket.org/dd1/gonodeinfo.git
cd gonodeinfo
git pull
make
make install # install gonodeinfo agent
cd ~ # this is important
</pre>
* edit /etc/gonodeinfo.conf
* change "Description", "Location", "User" and "Administrator" as appropriate (or delete them)
* change "Servers" to read: Servers: daq00.triumf.ca:8601
* run "gonodeinfo -v"
* if error is "connection refused". go to the nodeinfo server to add this client to the access control list:
* on the gonodeinfo server: run /opt/gonodeinfo/gonodereceive.exe -a daq13
* try gonodeinfo again, there should be no error
* on the gonodeinfo server: run gonodereport, look at the web pages, the new machine should be listed now

== install fonts for EPICS ==

* apt install xfonts-100dpi xfonts-75dpi
* restart Xorg (i.e. "killall Xorg", this will log you out from the console)
* xlsfonts | grep -i helvetica ### should show fonts with different sizes, not just size 0 (scalable)

== install libz.so.1 for CentOS compatibility ==

KO - confirm which versions on quartus need this.

<pre>
yes | apt-get -y install zlib1g
yes | apt-get -y install zlib1g:i386 libc6:i386 libgcc1:i386 gcc-6-base:i386
</pre>

== install libpng12.so.0 for Quartus compatibility ==

(does not work anymore!!!)

<pre>
wget http://ftp.ca.debian.org/debian/pool/main/libp/libpng/libpng12-0_1.2.50-2+deb8u2_amd64.deb
dpkg --install libpng12-0_1.2.50-2+deb8u2_amd64.deb
</pre>

== install libpng12.so.0 for Quartus 13.0sp1 ==

<pre>
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0.50.0
/bin/cp -pv libpng12.so.0 libpng12.so.0.50.0 /lib/x86_64-linux-gnu/
</pre>

== install packages for Xilinx ==

ubuntu LTS 22.04 vivado 2020.1

<pre>
apt install autoconf libtool
apt install libtinfo5
apt install texinfo
apt install zlib1g:i386
</pre>

== install packages for building ROOT ==

<pre>
apt -y install libx11-dev libxpm-dev libxft-dev libxext-dev libpng-dev libjpeg-dev xlibmesa-glu-dev libxml2-dev libgsl-dev cmake
</pre>

== install 32-bit libraries for PHYSICA ==

these instructions are for running 32-bit physica executable built for SL6 on ubuntu LTS 20.04

install physica sources (cannot build, do not have g77)

<pre>
cd ~/packages
git clone https://bitbucket.org/ttriumfdaq/physica.git
</pre>

install 32-bit libraries using ubuntu package manager:

<pre>
apt install lib32z1 # libz.so
</pre>

copy 32-bit SL6 shared libraries to /lib32

<pre>
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libX11.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libgd.so.2 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libpng12.so.0 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libreadline.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libncurses.so.5 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libg2c.so.0 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libxcb.so.1 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libXpm.so.4 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libjpeg.so.62 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libfontconfig.so.1 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libfreetype.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libtinfo.so.5 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libXau.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libexpat.so.1 /lib32/
</pre>

ldd should report:

<pre>
trinatdaq:trinat> ldd /usr/local/physica/physica.exe
linux-gate.so.1 (0xf7fa2000)
libX11.so.6 => /lib32/libX11.so.6 (0xf7e43000)
libgd.so.2 => /lib32/libgd.so.2 (0xf7dfe000)
libpng12.so.0 => /lib32/libpng12.so.0 (0xf7dd6000)
libz.so.1 => /lib32/libz.so.1 (0xf7db8000)
libreadline.so.6 => /lib32/libreadline.so.6 (0xf7d7e000)
libncurses.so.5 => /lib32/libncurses.so.5 (0xf7d5b000)
libg2c.so.0 => /lib32/libg2c.so.0 (0xf7d3d000)
libm.so.6 => /lib32/libm.so.6 (0xf7c39000)
libgcc_s.so.1 => /lib32/libgcc_s.so.1 (0xf7c1a000)
libc.so.6 => /lib32/libc.so.6 (0xf7a2f000)
libxcb.so.1 => /lib32/libxcb.so.1 (0xf7a05000)
libdl.so.2 => /lib32/libdl.so.2 (0xf79ff000)
libXpm.so.4 => /lib32/libXpm.so.4 (0xf79ee000)
libjpeg.so.62 => /lib32/libjpeg.so.62 (0xf7997000)
libfontconfig.so.1 => /lib32/libfontconfig.so.1 (0xf7962000)
libfreetype.so.6 => /lib32/libfreetype.so.6 (0xf78c9000)
libtinfo.so.5 => /lib32/libtinfo.so.5 (0xf78b0000)
/lib/ld-linux.so.2 (0xf7fa4000)
libXau.so.6 => /lib32/libXau.so.6 (0xf78ad000)
libexpat.so.1 => /lib32/libexpat.so.1 (0xf7885000)
trinatdaq:trinat>
</pre>

set login environment:

<pre>
setenv TRIUMF_FONTS $HOME/packages/physica/fonts
setenv PHYSICA_DIR $HOME/packages/physica
alias physica $PHYSICA_DIR/physica-SL6-32
</pre>

test:

<pre>
cd ~/packages/physica
physica
@rangauss.pcm
</pre>

== install lightdm ==

unlike the default gdm login manager, lightdm shows the machine hostname and does not require an extra mouse click to swicth from screen saver to login mode.

<pre>
apt -y install lightdm
# select lightdm
</pre>

== install desktop environments ==

note: default display manager and default desktop are deficient, please do not skip this step.

note: if apt asks to choose the display manager, select "lightdm"

note: KO - I recommend the "MATE" desktop.

note: you will have to cut-and-paste this several times because "apt" eats commands, even with "-y" and even piped from "yes".

<pre>
# install MATE desktop
DEBIAN_FRONTEND=noninteractive apt -y install ubuntu-mate-core ubuntu-mate-desktop ubuntu-mate-themes
# install Cinnamon desktop
DEBIAN_FRONTEND=noninteractive apt -y install cinnamon
# install KDE desktop
DEBIAN_FRONTEND=noninteractive apt -y install kubuntu-desktop
# install Lxqt desktop
DEBIAN_FRONTEND=noninteractive apt -y install lxqt
# install Xfce4 desktop
DEBIAN_FRONTEND=noninteractive apt -y install xfce4
</pre>

== install ROOT ==

Please install ROOT per instructions at http://root.cern.ch.

NOTE1: The ROOT package available from Ubuntu repositories is severely out of date and cannot be used with MIDAS and ROOTANA. ### DO NOT DO THIS! apt-get install root-system

NOTE2: as of 2017-Jan-09, ROOT binary kits for Ubuntu do not work (use GCC 5 instead of GCC6), build from source instead.

== Install x2go ==

KO - is this still needed? does it cause any security problems?

x2go instructions, thanks to Art O.

<pre>
add-apt-repository ppa:x2go/stable
apt-get update
apt-get install x2goserver x2goserver-xsession
</pre>

== enable root login from ladd00/daq00 ==
<pre>
ssh localhost
CTRL-C
/bin/cp ~root/git/scripts/etc/authorized_keys ~root/.ssh/
</pre>

== install smart-status ==
<pre>
ln -s ~/git/scripts/smart-status/smart-status.perl ~root/
</pre>

== enable boot menu and boot messages ==

This will enable the grub menu (with a 10 sec timeout) and
replace black screen with exciting linux boot messages.

* emacs -nw /etc/default/grub
<pre>
GRUB_DEFAULT=0
#GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="vga=769 video=640x480"
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX=""
#GRUB_GFXMODE=640x480
</pre>
* update grub config:
<pre>
grub-mkconfig -o /boot/grub/grub.cfg
</pre>

== reboot ==

this completes installation of the base system.

following sections modify basic ubuntu to fix known problems and to enable special stuff.

= Enable automatic updates =

<pre>
apt install unattended-upgrades
cd ~/git/scripts
git pull
/bin/cp -v etc/99apt-conf-ko /etc/apt/apt.conf.d/
apt-config dump | grep Unattended
</pre>

Following is obsolete:

* emacs -nw /etc/apt/apt.conf.d/50unattended-upgrades
** uncomment in Allowed-Origins "-security" and "-updates"
** add in Allowed-Origins: "Google LLC:stable";
** uncomment/add: "Unattended-Upgrade::Mail "root";
* emacs -nw /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
</pre>
* test: unattended-upgrade --dry-run -v

NOTE: update-on-shutdown is disabled.

NOTE: there is no update-on-boot, but:

NOTE: if machine was off for a long time, the systemd update timer would have expired and it will fire soon after reboot, causing an automatic update run. this is unwanted, and there is no fix or workaround for it. K.O. June-2023.

= Fix bpool is full =

!!! only if ROOT on ZFS !!!

There is an error in the zsys package that causes bpool to run out of space,
see [[#Ubuntu zsys]] for more details.

To fix:
<pre>
cd ~/git/scripts
git pull
cp etc/zsys.conf /etc/
zsysctl service reload
zsysctl service gc
zpool list bpool
zfs list bpool
df /boot
</pre>

= IPMI instructions =

IPMI is the board management hardware on Supermicro and other server motherboards. This includes hardware sensors - fan rotation speed, temperatures and power supply voltages.

<pre>
apt-get install ipmitool
systemctl enable ipmievd
systemctl restart ipmievd
</pre>

Run:
* ipmitool sel list ### event list
* ipmitool sel elist ### event list
* ipmitool sel clear ### clear event list (if it becomes full)
* ipmitool sensor ### report hardware sensors

= move /home/wheel =

note: this MUST be done if ZFS root and NIS/autofs with /home.

Default location of wheel's home directory will collide with autofs /home, it has to be moved,
for example to /wheel.

<pre>
# logout from the wheel user
# go to another computer
ssh root@daqubuntuxxx
zfs list | grep wheel ### identify zfs name wheel_xxxxxx
zfs set mountpoint=/wheel rpool/USERDATA/wheel_hm8fzh
emacs -nw /etc/passwd ### change wheel's home directory from /home/wheel to /wheel
su - wheel ### check that user wheel still works
</pre>

This will break wheel's ability to run snap programs, such as firefox, install chrome as listed below.

= enable NIS (ubuntu 22.04, debian 11) =

<pre>
apt -y install rpcbind nis
echo DAQ-NIS >> /etc/defaultdomain
echo ypserver daq00.triumf.ca >> /etc/yp.conf
systemctl enable ypbind.service
systemctl restart ypbind.service
systemctl status ypbind.service
ypwhich -m
</pre>

enable ypserv:

<pre>
sed -i s/NISSERVER=false/NISSERVER=slave/ /etc/default/nis
/usr/lib/yp/ypinit -s daq00
echo ypserver localhost >> /etc/yp.conf
sed -i "s/ypserver .*/ypserver localhost/" /etc/yp.conf
systemctl enable ypserv
systemctl restart ypserv
systemctl restart ypbind
</pre>

edit /etc/nsswitch.conf to read:

<pre>
# begin get data from nis
passwd: files nis
group: files nis
shadow: files nis
automount: files nis
netgroup: files nis
# end get data from nis
</pre>

enable hourly update of nis maps:

<pre>
mkdir ~root/git
cd ~root/git
git clone http://daq00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>

If this is a new machine, then on the master NIS node (daq00), add this new node to /etc/netgroup, and update NIS maps (cd /var/yp; make)

= enable NIS (ubuntu 20.04) =

* apt-get -y install portmap nis ### will ask for NIS domain (DAQ-NIS)
* dpkg-reconfigure nis ### reconfigure if already installed
* ypwhich -m
* edit /etc/default/nis
** set "NISSERVER=slave"
** Ubuntu LTS 20.04, check that "YPBINDARGS=" is blank, remove "-no-dbus" if it is there
* #edit /etc/yp.conf, comment-out everything, add "domain DAQ-NIS server localhost"
* edit /etc/yp.conf, comment-out everything, add "ypserver localhost"
* /usr/lib/yp/ypinit -s daq00
* systemctl enable nis
* systemctl restart nis
* ypwhich
* ypwhich -m
* ypcat -k passwd
* vi /etc/nsswitch.conf ### add the automount line, modify the passwd, group and shadow lines to read this:
<pre>
# begin get data from nis
passwd: files nis
group: files nis
shadow: files nis
automount: files nis
netgroup: files nis
# end get data from nis
</pre>
* enable hourly update of NIS maps
<pre>
mkdir ~root/git
cd ~root/git
git clone http://ladd00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>
* ### NOT NEEDED sudo vi /etc/idmapd.conf ### add line: "Domain = triumf.ca"

= enable autofs =

<pre>
apt -y install autofs
systemctl enable autofs
systemctl restart autofs
ls -l /home/olchansk ### test autofs, check file owner is correct
</pre>

= enable NFS server =

<pre>
apt install nfs-kernel-server
#edit /etc/exports
systemctl enable nfs-server
systemctl restart nfs-server
</pre>

= NIS master =

notes for setting up the NIS master

== wheel user ==

"wheel" is the default administrative user. We do not want it's password exported to NIS (encrypted password hash is world visible) and we do not want it's home directory exported to NFS (~wheel/.ssh is world visible and potentially writable: anybody can change ~wheel/.ssh/authorized_keys).

* move wheel's home directory from /home/wheel to /wheel (see special section about this)
* change wheel's UID and GID from 1000 to a value below MINUID in /var/yp/Makefile

== coherent uids ==

we do not want system accounts defined in /etc/passwd of the NIS master
to be included in the NIS map "passwd". this causes trouble on NIS clients
where newly installed packages fail to create local system users because same
user already exists in NIS.

This is controlled by MINUID in /var/yp/Makefile.

Historical TRIUMF uids start from around 200, but several clusters do not have any historic TRIUMF uids below 500 and MINUID is set to:
* DAQ-NIS: MINUID=200
* ISAC-NIS: MINUID=500
* TITAN-NIS: MINUID=500
* MUSR-NIS: MINUID=500
* TIG-NIS: MINUID=500 (100 on SL6 mother8pi)

Ubuntu 20 has two programs to create users:
* adduser - creates new users with UID 1000 and up as specified in /etc/adduser.conf. No problems here.
* adduser --system - creates new system users with UID 100 and up as specified in /etc/adduser.conf. No problems here.
* useradd - creates new users with UID 1000 and up as specified in /etc/login.defs. No problems here.
* useradd --system - creates new system users with UID 999 and down (read "man useradd", section at the end about SYS_UID_MAX). This collides with NIS MINUID, these system users will be included in the NIS map and cause trouble.

This problem cannot be fixed, SYS_UID_MIN, SYS_UID_MAX and UID_MIN in /etc/login.defs do not seem
to have any effect on UIDs chosen by "useradd --system". (tested on Ubuntu LTS 20.04).

So far only these system accounts seem to be affected by this:
* systemd-coredump
* ganglia

To fix:
* run "sort -r -n -t: -k3 /etc/passwd" to identify the last unused system user uid (range 100..200)
* run "sort -r -n -t: -k3 /etc/group" to identify the last unused system user gid (range 100.200)
* systemd-coredump: manually change UID and GID (package systemd-coredump is usually not installed)
* ganglia: same thing, then change ownership on all ganglia files.

Also read systemd author's opinion on system vs user UIDs:
https://github.com/systemd/systemd/issues/4850#issuecomment-265698275

= Fix systemd-logind NIS breakage =

!!! THIS IS NOT NEEDED FOR UBUNTU LTS 20.04 !!!

there is a delay in ssh logins for normal users. "ssh -v" shows the delay is after "pledge...". this
fix removes the delay.

systemd developers think that we should not use NIS and made sure there are
problems if we do. To give them credit, they do offer a workaround. Read this:
https://github.com/poettering/systemd/commit/695fe4078f0df6564a1be1c4a6a9e8a640d23b67

<pre>
mkdir /etc/systemd/system/systemd-logind.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-logind.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-logind.service
</pre>

= Fix systemd-udevd NIS breakage =

see same problem as above with udev getting stuck. ubuntu lts 20.04.

<pre>
mkdir /etc/systemd/system/systemd-udevd.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-udevd.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-udevd.service
</pre>

= Configure USB device permissions =

Configure USB device permissions for user access to USB-serial devices, Altera USB Blaster, etc.

* create file /etc/udev/rules.d/99-usb-chmod.rules with this contents:

<pre>
emacs -nw /etc/udev/rules.d/99-usb-chmod.rules
ACTION=="add", SUBSYSTEM=="usbmisc", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /dev/%c"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /proc/%c"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVICE}"
ACTION=="add", ENV{PHYSDEVBUS}=="usb-serial", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVPATH}=="/class/tty/ttyS*", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyUSB*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyACM*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyS*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", DEVPATH=="*video*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
</pre>

* reload udev rules: udevadm control --reload-rules
* apply new permissions: udevadm trigger --action=add
* watch udev activity: udevadm monitor -p

= Configure lightdm display manager =

* enable it
<pre>
echo lightdm | dpkg-reconfigure -fteletype lightdm
systemctl disable gdm
systemctl disable sddm
systemctl enable lightdm
</pre>

* make the MATE desktop as default
<pre>
cd ~root/git/scripts/
git pull
/bin/cp -v etc/lightdm_default_mate.conf /etc/lightdm/lightdm.conf.d/
</pre>

* enable login by NIS users
<pre>
/bin/cp -v etc/lightdm_enable_nis_login.conf /etc/lightdm/lightdm.conf.d/
</pre>

* restart lightdm
<pre>
systemctl stop gdm
systemctl restart lightdm
</pre>

= Install libpng12.so.0 =

Quartus 16 needs libpng12:

<pre>
wget http://mirrors.kernel.org/ubuntu/pool/main/libp/libpng/libpng12-0_1.2.54-1ubuntu1_amd64.deb
dpkg --install libpng12-0_1.2.54-1ubuntu1_amd64.deb
</pre>

= Install google-chrome =

<pre>
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
dpkg -i google-chrome-stable_current_amd64.deb
</pre>

confirm autoupdate is enabled, observe dl.google.com is present in the list of repositories:
<pre>
apt update
...
Get:5 https://dl.google.com/linux/chrome/deb stable/main amd64 Packages [1,094 B]
...
</pre>

FOLLOWING IS OBSOLETE:

Instructions from here:
https://www.ubuntuupdates.org/ppa/google_chrome?dist=stable

<pre>
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-tmp.list'
apt update
apt install google-chrome-stable
/bin/rm -f /etc/apt/sources.list.d/google-tmp.list
</pre>

= Install amanda client =

ONLY ONE MACHINES THAT HOST HOME DIRECTORIES

* apt install amanda-client
* edit /etc/amandahosts
<pre>
amanda.triumf.ca amanda amdump
</pre>
* check permissions on /etc/amandahosts:
<pre>
root@daq00:/var/log/amanda# ls -l /etc/amandahosts
-rw------- 1 backup backup 49 Jan 27 10:48 /etc/amandahosts
</pre>
* fix if needed: chown backup.backup /etc/amandahosts; chmod a= /etc/amandahosts; chmod u=wr /etc/amandahosts
* edit /etc/amanda-security.conf, add this line:
<pre>
runtar:gnutar_path=/usr/bin/tar
</pre>

On the amanda machine:

* in amanda disklist, use dump type "bsdtcp-comp-user-tar"
* su - amanda and run amcheck -c daily daq00
<pre>
-bash-4.1$ amcheck -c daily daq00

Amanda Backup Client Hosts Check
--------------------------------
Client check: 1 host checked in 0.092 seconds. 0 problems found.

(brought to you by Amanda 3.3.7p1.git.685ff76d)
</pre>

= Enable rc.local =

For reasons unknown, Ubuntu LTS 20.04 does not enable /etc/rc.local. Do this:

<pre>
cd ~/git/scripts
git pull
cp -n -v etc/rc.local /etc/
chmod a+rx /etc/rc.local
cp etc/rc-local.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable rc-local
systemctl start rc-local
systemctl status rc-local
</pre>

= Remove unwanted packages =

<pre>
apt remove zsys
apt remove sddm
</pre>

= Disable unwanted services =

<pre>
systemctl disable mpd
systemctl disable snapd
systemctl disable ModemManager
systemctl --global mask tracker-extract-3.service
systemctl --global mask tracker-miner-fs-3.service
systemctl daemon-reload
</pre>

= Disable sleep and suspend =

note: we see some computers randomly shutdown or go to sleep, log files indicates the "sleep" or "suspend" button was pushed by user, but no such buttons actually exist. this is the fix for this:

<pre>
systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target systemd-suspend.service systemd-hybrid-sleep.service
</pre>

= Enable crontab @reboot for MIDAS =

startup scripts have a bug - cron @reboot entries for normal users can run before autofs is ready, so if the home directory is on autofs/NFS, it cannot be accessed and the cron job fails. If MIDAS is supposed to be started by cron @reboot, it will not start (there *will* be an error message in /var/log/cron).

<pre>
mkdir /etc/systemd/system/cron.service.d
echo -e "[Unit]\nAfter=ypbind.service autofs.service\n" > /etc/systemd/system/cron.service.d/local.conf
systemctl daemon-reload
systemctl cat cron.service
</pre>

Explore the systemd dependency tree using "systemctl list-dependencies" maybe with "--all".

Visualize the exact boot sequence from previous boot: "systemd-analyze plot > xxx.svg", look at the svg file using a web browser.

Crontab entry to start midas: (install in the midas user crontab, not root crontab)

<pre>
su - midasuser
crontab -l
#@reboot /bin/bash -l -c "/home/trinat/bin/start-daq-applications"
#@reboot /bin/tcsh -c "/home/trinat/bin/start-daq-applications"
</pre>

= Install apache httpd proxy for midas and elog =

This will configure the HTTPS/SSL certificate using "certbot" and "letsencrypt" and configure an HTTPS web server using apache2.

First, configure apache2:

* execute these commands:
<pre>
apt -y install apache2
cd /etc/apache2
</pre>
* create new file conf-available/ssl-daq14.conf # use actual hostname instead of daq14
<pre>
SSLSessionCache shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout 300
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
SSLCryptoDevice builtin
</pre>
* create new file sites-available/daq14-ssl.conf # use actual hostname instead of daq14
<pre>
<IfModule mod_ssl.c>
<VirtualHost *:443>
ServerName daq14.triumf.ca
DocumentRoot /var/www/html
ErrorLog /var/log/apache2/daq14.log
SSLEngine on
# note SSLProtocol, SSLCipherSuite and some other settings are overwritten by /etc/letsencrypt/options-ssl-apache.conf
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SEED:!IDEA:!RC4
## use port specified in elogd.cfg
#ProxyPass /elog/ http://localhost:8082/ retry=1
## use mhttpd port
#ProxyPass / http://localhost:8080/ retry=1
Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"
<Location />
SSLRequireSSL
AuthType Basic
AuthName "DAQ password protected site"
Require valid-user
# create password file: touch /etc/apache2/htpasswd
# to add new user or change password: htpasswd /etc/apache2/htpasswd username
AuthUserFile /etc/apache2/htpasswd
</Location>
</VirtualHost>
</IfModule>
</pre>
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* enable ssl module
* enable new configurations
<pre>
a2enmod ssl
a2enmod headers
a2enmod proxy
a2enmod proxy_http
a2enconf ssl-daq14
a2ensite daq14-ssl
</pre>
* disable default ssl sites
<pre>
a2dissite 000-default-le-ssl
a2dissite 000-default
ls -l /etc/apache2/sites-enabled/ ### should show only daq14-ssl.conf
</pre>
* check that there are no syntax problems
<pre>
apache2ctl configtest
</pre>
* enable and start apache2:
<pre>
systemctl enable apache2
systemctl restart apache2
systemctl status apache2
</pre>
* apache2 may fail to start, look in /var/log/apache2/error.log and /var/log/apache2/daq14.log
* if it says "Failed to configure ... certificate", proceed to the step for setting certbot.
* try to access https://daq14.triumf.ca
** you should see a complaint about self-signed certificate
** you should see a request for password (do not login yet)
** if you get "connection refused", HTTPS port 443 may need to be enabled in the local firewall, look at documentation for ufw.
Second, configure certbot:

(Note: as of 2018-01-18 certbot requires use of http port 80 to get the initial https certificate,
renewal can continue to use the https port 443)

(Note: as of 2019-01-?? certbot requires use of port 80 for renewals)

* check that port 80 is not used by anything:
* netstat -an | grep LISTEN | grep ^tcp | grep 80
* lsof -P | grep -i tcp | grep LISTEN | grep 80
* if lsof reports that apache2 is listening on port 80, follow the apache2 instructions above (remove "listen 80" from apache2.conf

* install certbot (if necessary open tcp port 80 in the firewall, see documentation for ufw):
<pre>
apt install certbot python3-certbot-apache
certbot certonly --standalone --installer apache
</pre>
* then answer questions:
* "activate HTTPS for daq14.triumf.ca" - say ok
* "enter email address" - enter your own email address
* "please read terms..." - read the terms and say "agree"
* it will take a few moments...
* "congratulations..." - say ok.
<pre>
certbot install --apache --cert-name daq14.triumf.ca
</pre>
* then answer questions:
* "choose redirect..." - say "1" (no redirect)
* look inside /etc/apache2/sites-enabled/daq14-ssl.conf to see that SSLCertificateFile & co point to certbot certificates in
/etc/letsencrypt/live/daq14.triumf.ca/
* to check current renewal and to update the certbot config file in /etc/letsencrypt/renewal, run this:
<pre>
certbot renew --standalone --installer apache --force-renewal
</pre>

NOTE: this certificate will expire in 3 months, automatic renewal should work with current version of certbot

Third, activate password protection:

* as shown in the config file above, create password file and initial user: (replace "midas" with specific username)
<pre>
touch /etc/apache2/htpasswd
htpasswd /etc/apache2/htpasswd midas
</pre>

* restart apache2
<pre>
systemctl restart apache2
systemctl status apache2
</pre>

From here:
* enable proxy for MIDAS mhttpd - uncomment redirect in the config file above
* enable proxy for ELOG - ditto
<pre>
a2enmod proxy
a2enmod proxy_http
apache2ctl configtest
systemctl restart apache2
</pre>

From here:
* enable proxy for MIDAS mhttpd - uncomment redirect in the config file above
* enable proxy for ELOG - ditto
<pre>
a2enmod proxy
a2enmod proxy_http
apache2ctl configtest
systemctl restart apache2
</pre>
* try accessing MIDAS https://daq14.triumf.ca/ (make sure mhttpd is running)
* if it's not working, check odb setting FIXME!
* try accessing ELog https://daq14.triumf.ca/elog/ (make sure elogd is running)
* if it's not working, check elogd.cfg file and make sure
<pre>
SSL = 0
</pre>

NOTE: if certbot fails with errors about 'module' object has no attribute 'pyopenssl',
try this: pip install requests==2.6.0

= Enable elog PDF preview =

see https://stackoverflow.com/questions/52998331/imagemagick-security-policy-pdf-blocking-conversion

* xemacs -nw /etc/ImageMagick-6/policy.xml
* remove this section at the end:
<pre>

<policy domain="coder" rights="none" pattern="PS" />
<policy domain="coder" rights="none" pattern="PS2" />
<policy domain="coder" rights="none" pattern="PS3" />
<policy domain="coder" rights="none" pattern="EPS" />
<policy domain="coder" rights="none" pattern="PDF" />
<policy domain="coder" rights="none" pattern="XPS" />
</pre>

= Install Jupyter notebook =

<pre>
From https://jupyter.org/install
apt install python3-pip
pip install jupyterlab
pip install notebook
~/.local/bin/jupyter notebook
watch the http://localhost:8888 URL that it printed
say "no" to offer to start firefox (it will not work!)
URL is: http://localhost:8888/tree?token=xxx
from the machine where you are running the web browser (i.e. google-chrome), run (replace trinat@trinatdaq with the username and machine name where you started jupyter)
open a new shell and run: ssh -v trinat@trinatdaq -L 8888:localhost:8888
in the web browser, open http://localhost:8888
this gives us the login page
in the password or token entry field, put the token from the "tree?token=xxx" above (printed by jupyter on startup)
push button "login"
jupyter page should open with the list of files in the trinat home directory
congratulate Brian with full success
</pre>

= Install ZFS quota report =

If there are any ZFS volumes, install script to report disk and quota usage

<pre>
cd ~/git/scripts/quotareport
git pull
mkdir /var/www/html/zfsquotareport
cp -pv ~/git/scripts/quotareport/sorttable.js /var/www/html/zfsquotareport/
ln -s $PWD/zfsquotareport.perl /etc/cron.daily/
touch /etc/crontab
</pre>

If httpd is configured to redirect "/" to MIDAS mhttpd:
* add following to /etc/apache2/sites-enabled/xxx-ssl.conf in front of "ProxyPass / ..."
* run "systemctl reload apache2"
<pre>
## do not proxy zfs quota report directory
ProxyPass /zfsquotareport/ !
</pre>

= Install PHP =

* apt install php libapache2-mod-php
* systemctl restart apache2
* create /var/www/html/info.php
<pre>
<?php

phpinfo();
</pre>
* open https://daq00.triumf.ca/info.php

= Configure TRIUMF printers =

<pre>
systemctl stop cups
systemctl disable cups
echo "ServerName printers.triumf.ca" > /etc/cups/client.conf
lpstat -a
</pre>

= Enable core dumps =

By default, Ubuntu LTS 20.04 installs the apport package
which disabled core dumps from user applications. (google it up!).
It is not meant to do this and documentation claims that
it is not installed and not enabled by default. Oh, well...

<pre>
apt remove apport
apt autoremove ### will remove apport-symptoms and a few other packages
</pre>

After this, core dumps are written to file "core" in the current directory.
See /proc/sys/kernel/core_pattern and /proc/sys/kernel/core_uses_pid.

= Enable debugger =

By default, Ubuntu LTS 20.04 does not permit debugger to attach and debug
already running programs. To enable it, add following to /etc/rc.local

<pre>
echo 0 > /proc/sys/kernel/yama/ptrace_scope
</pre>

= Disable Ubuntu Pro nag =

If "apt upgrade" requests Ubuntu Pro or esm-apps, disable the nag:
<pre>
/bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf
</pre>

= Update packages =

* apt-get update # update package list
* apt-get dist-upgrade # install updated packages and update "kept back" packages
* apt-get autoremove # remove packages that apt thinks should be removed

= Finish installation =

Congratulations. There is nothing more to do!

* reboot
<pre>
shutdown -r now
</pre>

= Install ZFS =

!!! after installing all the packages, after updating the system, after updating the linux kernel, after rebooting into latest kernel !!!

<pre>
apt-get install zfsutils-linux
</pre>

Follow generic ZFS instructions: [[ZFS]]

= Update to new version of Ubuntu =

<pre>
vi /etc/update-manager/release-upgrades # set "Prompt=normal"
do-release-upgrade
</pre>

Update Ubuntu LTS 20.04 to LTS 22.04:

== daqubuntu ==

<pre>
# reboot to clear out all updates
# vi /etc/update-manager/release-upgrades # set "Prompt=normal"
# do-release-upgrade -c
Checking for a new Ubuntu release
New release '22.04 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
# do-release-upgrade
...
say yes...
...
login.defs, say "Y" (erase local changes, use packaged version)
/etc/systemd/resolved.conf, say "Y" (same as above)
firefox snap, say yes
unable to reach snap store, say "skip"
/etc/gmond.conf, say "Y"
/var/yp/Makefile, say "install the package maintainer's version"
/etc/ypserv.conf, same thing
/etc/ypserv.securenets, same thing
/etc/default/nis, same thing
/etc/speech-dispatcher/modules/mary-generic.conf, same thing
/etc/apt/apt.conf.d/50unattended-upgrades, same thing
...
278 packages are going to be removed, say yes
...
restart required, say yes
...
no ping... yes ping...
...
ssh daqubuntu, ok
apt update, fail, DNS does not work, "host security.ubuntu.com" does not resolve.
fix resolver per https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Disable_NetworkManager
apt update, apt upgrade now works, 0 packages to update
NIS does not work.
</pre>

== midm9a ==

<pre>
login.defs
firefox snap
gmond.conf
ypserv
/etc/default/nis
unattended-upgrades
amanda-security.conf
remove obsolete (no)
reboot
configure dns
reenable nis
</pre>

== daq17 ==

<pre>
firefox snap
imagemagick policy.xml
gmond.conf
chrony.conf
/var/yp/Makefile
ypserv.conf
ypserv.securenets
/etc/default/nis
50unattended-upgrades
</pre>

== daq00 ==

per https://serverpilot.io/docs/how-to-upgrade-ubuntu-20.04-to-22.04/

<pre>
do-release-upgrade -f DistUpgradeViewNonInteractive
</pre>

if it exists "too soon" without doing anything, run it without "-f xxx", most likely it does not like something about this machine. in case of daq00 it did not like how the EFI partitions were mounted. after fixing it, non-interactive upgrade was successful.

= Ubuntu package manager =

* apt-get install xxx # install package xxx
* apt-get update
* apt-get upgrade
* apt-get dist-upgrade
* apt-get autoremove # remove automatically installed packages required by a removed package
* apt-get remove xxx # remove package xxx
* apt-cache search . # list all available packages
* apt-cache show "." | grep ^Package # list al available packages
* apt-cache madison root-system # show all available versions of package root-system
* apt list # list all installed packages
* dpkg --listfiles libpng16-16 # list all files from this package
* apt list --installed # list all installed packages
* dpkg -S /bin/bash # what package provides this file?
* dpkg -L bash # what files provided by this package?
* debsums -ce # show modified config files
* apt-config dump # show apt configuration

= Ubuntu zsys =

NOTE: DO NOT USE ZSYS, see https://github.com/ubuntu/zsys/issues/218 and https://github.com/ubuntu/zsys/issues/230

* manual removal of old snapshots
<pre>
zsysctl show
zsysctl state remove xy69ye -s
zsysctl state remove xy69ye
zsysctl state remove xy69ye -u wheel
</pre>
* apt remove zsys

NOTE: old zsys snapshots must be cleaned manually, "zsysctl state remove xxx --system" is broken and does not remove user data snapshots

* manages system snapshots
* documentation: https://github.com/ubuntu/zsys
* documentation: (go to next article via link "newer" at the bottom) https://didrocks.fr/2020/05/21/zfs-focus-on-ubuntu-20.04-lts-whats-new/
* ubuntu 20.04 bug, too many snapshots cause /boot to become full and updates fail. https://github.com/ubuntu/zsys/issues/155
* solution: use custom /etc/zsys.conf, limit number of snapshots to 10, see trinatdaq:/etc/zsys.conf
* zsys commands:
<pre>
update-grub # list of all snapshots, errors if some snapshots are broken
zsysctl state remove lnc0k7 --system # remove snapshot
xemacs -nw /etc/zsys.conf; zsysctl service reload; zsysctl service gc # cause gc to run with new settings in zsys.conf
zfs list -r -t snapshot -o name,used,referenced,creation bpool/BOOT # list snapshots
zsysctl show # show snapshots
</pre>

= Ubuntu cloning =

to clone a ubuntu image:

<pre>
cd /nfsroot/lxcpet
emacs -nw etc/hostname ### change hostname
emacs -nw etc/mailname ### change hostname (debian 11)
emacs -nw etc/defaultdomain ### change the NIS domainname
emacs -nw etc/yp.conf ### change the NIS server
cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
emacs -nw root/.ssh/authorized_keys ### update root ssh keys
</pre>

= Ubuntu boot loader =

== boot from ZFS ==

* use UEFI boot with syslinux, see here: https://daq.triumf.ca/DaqWiki/index.php/SLinstall#Configure_UEFI_boot
* apt install zfs-initramfs
* update-initramfs -v -u
* ZFS structure:
<pre>
root@daq00:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 147G 1.62T 96K /
rpool/ROOT 17.8G 1.62T 96K none
rpool/ROOT/ubuntu_00aaaa 17.8G 1.62T 6.22G /
</pre>
* copy OS image to rpool/ROOT/ubuntu_00aaaa
* zfs set mountpoint=/ rpool
* zfs set mountpoint=none rpool/ROOT
* zfs set mountpoint=/ rpool/ROOT/ubuntu_00aaaa
* zfs get all | grep mountpoint
<pre>
rpool mountpoint / local
rpool/ROOT mountpoint none local
rpool/ROOT/ubuntu_00aaaa mountpoint / local
</pre>
* in linux kernel command line (syslinux.cfg), set "root=" to "root=ZFS=rpool/ROOT/ubuntu_00aaaa"

== boot from ZFS mirror ==

=== setup the EFI partitions ===

* assuming /dev/sdb is already setup for EFI boot, setup /dev/sda the same way:
* partition the second boot disk same as first boot disk:
<pre>
root@grsnis01:~# gdisk -l /dev/sdb
Found valid GPT with protective MBR; using GPT.
Number Start (sector) End (sector) Size Code Name
1 2048 1050623 512.0 MiB EF00 EFI system partition
2 1050624 3907029134 1.8 TiB 8300 Linux filesystem
root@grsnis01:~#
</pre>
* mkfs.msdos /dev/sdX1
* create mount points
<pre>
mkdir /boot/efi-sda
mkdir /boot/efi-sdb
</pre>
* add to /etc/fstab
<pre>
/dev/sda1 /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
/dev/sdb1 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
</pre>
* mount -a
* df | grep boot
<pre>
root@grsnis01:~# df | grep boot
/dev/sdb1 523248 98100 425148 19% /boot/efi-sdb
/dev/sda1 523248 4 523244 1% /boot/efi-sda
</pre>
* copy boot files to new boot disk
* cd /boot/efi-sdX; rsync -av . /boot/efi-sdY
* set BIOS to boot from "UEFI Hard drive", disable legacy boot (except for booting from USB key in legacy mode)
* if using UEFI boot syslinux per these instructions, linux kernel update has to be done manually:
* run ~/git/scripts/etc/update_efi_mirror.perl, follow instructions that it prints.

=== setup zfs partitions ===

use partitions compatible with Ubuntu "install on ZFS"

* gdisk "o" to create new GPT partition table
* gdisk "n" +512M ef00 to create EFI partition
* gdisk "n" +2G 8200 to create linux swap partition (not used)
* gdisk "n" +2G BE00 to create ZFS bpool partition
* gdisk "n" xxx BF00 create ZFS rpool partition

<pre>
# gdisk -l /dev/sda
Number Start (sector) End (sector) Size Code Name
1 2048 1050623 512.0 MiB EF00 EFI System Partition
2 1050624 5244927 2.0 GiB 8200
3 5244928 9439231 2.0 GiB BE00
4 9439232 234441614 107.3 GiB BF00
root@midm9a:~#
</pre>

=== setup zfs mirror ===

* see here: https://daq.triumf.ca/DaqWiki/index.php/ZFS#Convert_pool_from_single_to_mirror
<pre>
root@grsnis01:~# ls -l /dev/disk/by-id/ata*part2
lrwxrwxrwx 1 root root 10 Feb 19 16:47 /dev/disk/by-id/ata-WDC_WDS200T2B0A-00SM50_205007801081-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Feb 19 16:47 /dev/disk/by-id/ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 -> ../../sdb2

root@grsnis01:~# zpool status
pool: rpool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 ONLINE 0 0 0

errors: No known data errors

root@grsnis01:~# zpool attach rpool ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 /dev/disk/by-id/ata-WDC_WDS200T2B0A-00SM50_205007801081-part2

root@grsnis01:~# zpool status
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Feb 19 16:54:39 2021
12.6G scanned at 3.16G/s, 1.02G issued at 262M/s, 12.6G total
1.02G resilvered, 8.09% done, 0 days 00:00:45 to go
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801081-part2 ONLINE 0 0 0 (resilvering)

errors: No known data errors
</pre>
* wait
<pre>
root@grsnis01:~# zpool status
pool: rpool
state: ONLINE
scan: resilvered 12.7G in 0 days 00:00:40 with 0 errors on Fri Feb 19 16:55:19 2021
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801081-part2 ONLINE 0 0 0

errors: No known data errors
</pre>

== maintenance commands ==

* update-initramfs -v -u
* grub-install /dev/sda

= Convert from single to dual mirrored ZFS SSD =

Assuming Ubuntu LTS 22.04 with "instal on ZFS" option, we will
add a second SSD, configure ZFS to use both SSDs in mirrored
configuration and setup grub to boot from either SSD. This
is intended to create a full redundant system where failure
of either SSD does not break the system.

* identify first SSD
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* connect second SSD of identical size
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
/dev/sdb WD Blue SA510 2.5 250GB 22243Z803852 25 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* if second SSD is not autodetected, reboot
* Clone partition table automatically
If both SSDs are identical size, use this simpler method of duplicating the partition table:
<pre>
root@midm9b:~# sfdisk -d /dev/sda > part_table
root@midm9b:~# grep -v ^label-id part_table | sed -e 's/, *uuid=[0-9A-F-]*//' | sfdisk /dev/sdb
</pre>
The grep and sed in the second command are there to prevent disk ID and partition IDs from being cloned. Alternatively the part_table file can be edited manually to remove the label-id line and the uuid entries from the individual partitions.

* Clone partition table manually (e.g. for different size disks)
* list partition table of first SSD:
<pre>
root@midm9b:~# fdisk -l /dev/sda
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>
* create identical partitions on second SSD, use sector numbers from above.
<pre>
root@midm9b:~# gdisk /dev/sdb
GPT fdisk (gdisk) version 1.0.8

Partition table scan:
MBR: not present
BSD: not present
APM: not present
GPT: not present

Creating new GPT entries in memory.

Command (? for help): n
Partition number (1-128, default 1):
First sector (34-488397134, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-488397134, default = 488397134) or {+-}size{KMGTP}: 1050623
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): ef00
Changed type of partition to 'EFI system partition'

Command (? for help): n
Partition number (2-128, default 2):
First sector (34-488397134, default = 1050624) or {+-}size{KMGTP}:
Last sector (1050624-488397134, default = 488397134) or {+-}size{KMGTP}: 5244927
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 8200
Changed type of partition to 'Linux swap'

Command (? for help): n
Partition number (3-128, default 3):
First sector (34-488397134, default = 5244928) or {+-}size{KMGTP}:
Last sector (5244928-488397134, default = 488397134) or {+-}size{KMGTP}: 9439231
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): be00
Changed type of partition to 'Solaris boot'

Command (? for help): n
Partition number (4-128, default 4):
First sector (34-488397134, default = 9439232) or {+-}size{KMGTP}:
Last sector (9439232-488397134, default = 488397134) or {+-}size{KMGTP}:
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): bf00
Changed type of partition to 'Solaris root'

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sdb.
The operation has completed successfully.
root@midm9b:~# fdisk -l /dev/sda /dev/sdb
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root

Disk /dev/sdb: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: EB251739-30C6-422F-A505-5887B5A0B603

Device Start End Sectors Size Type
/dev/sdb1 2048 1050623 1048576 512M EFI System
/dev/sdb2 1050624 5244927 4194304 2G Linux swap
/dev/sdb3 5244928 9439231 4194304 2G Solaris boot
/dev/sdb4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>
* identify second SSD partitions
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part3
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part3 -> ../../sda3
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 -> ../../sdb3
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
</pre>
* convert bpool from single disk to mirrored disk:
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~# zpool attach bpool 99e03dc0-7d4d-f24b-8fa1-f042b9f135db /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3
root@midm9b:~# zpool status bpool
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors
</pre>
* convert rpool
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
root@midm9b:~# zpool attach rpool f6fd54f8-3af7-b943-ae3d-a4e480537fb9 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4
root@midm9b:~# zpool status rpool
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Jan 20 19:40:45 2023
5.83G scanned at 664M/s, 2.92M issued at 332K/s, 9.11G total
0B resilvered, 0.03% done, no estimated completion time
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~#
</pre>
* wait for resilver to complete
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
scan: resilvered 9.65G in 00:00:36 with 0 errors on Fri Jan 20 19:41:21 2023
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
</pre>
* enable booting from second SSD: (instead of /dev/sda1, /dev/sdb1, use UUID=xxx)
<pre>
root@midm9b:~# mkfs.msdos /dev/sdb1
root@midm9b:~# mkdir /boot/efi-sda
root@midm9b:~# mkdir /boot/efi-sdb
root@midm9b:~# echo "/dev/sda1 /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1" >> /etc/fstab
root@midm9b:~# echo "/dev/sdb1 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1" >> /etc/fstab
root@midm9b:~# mount -a
root@midm9b:~# df -kl
Filesystem 1K-blocks Used Available Use% Mounted on
...
/dev/sda1 523244 13720 509524 3% /boot/efi
/dev/sdb1 523244 4 523240 1% /boot/efi-sdb
...
root@midm9b:~# rsync -av /boot/efi/ /boot/efi-sdb/
sending incremental file list
EFI/
...
root@midm9b:~# ls -l /boot/efi-sda
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~# ls -l /boot/efi-sdb
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~#
</pre>
* setup script to update grub on second SSD, it must be run manually after every kernel update
<pre>
root@midm9b:~# ln -s ~/git/scripts/etc/update_efi_grub.perl ~/
root@midm9b:~# ~/update_efi_grub.perl -u
EFI dir: /boot/efi-sda
/boot/efi-sda: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sda/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sda: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sda/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
EFI dir: /boot/efi-sdb
/boot/efi-sdb: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sdb/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sdb: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sdb/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
root@midm9b:~#
</pre>

= Disable NetworkManager =

NOTE: THIS IS BROKEN IN UBUNTU LTS 22.04

NetworkManager is useful for configuring dynamic
network interfaces, i.e. laptops that often move
between networks, or connect to multiple choice
of wifi networks, etc.

For machines with statically configured network interfaces,
NetworkManager is not necessary.

As it has been observed to become confused and observed
to malfunction when network links go up and down (it keeps
unnecessarily reconfiguring the ip address, etc), it can
be usefuil to disable it.

* list all network interfaces
<pre>
# /bin/ls -1 /sys/class/net/
enp0s31f6
lo
</pre>
* edit /etc/network/interfaces:
<pre>
rename enp0s31f6=eth0
auto eth0
iface eth0 inet static
address 142.90.120.94/19
gateway 142.90.100.18
</pre>
* statically configure systemd-resolved
** create /etc/systemd/resolved.conf.d/resolved.conf with this contents:
<pre>
[Resolve]
DNS=142.90.100.19
Domains=triumf.ca
</pre>
** systemctl restart systemd-resolved
** resolvectl
** systemd-analyze cat-config systemd/resolved.conf
* disable NetworkManager
<pre>
systemctl disable NetworkManager
</pre>
* reboot

= Configure ECC memory =

== Configure EDAC ==

* apt install edac-utils

=== Intel i3-2120 ===
<pre>
root@musr00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X9SCL/X9SCM
root@musr00:~# edac-ctl --status
edac-ctl: drivers not loaded.
</pre>

=== Intel E-2236 ===
<pre>
root@daq00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SCM-F
root@daq00:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq00:~# edac-util
edac-util: No errors to report.
root@daq00:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
</pre>
* check edac sysfs files (Intel)
<pre>
root@daq00:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_noinfo_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 max_location
-r--r--r-- 1 root root 4096 Jan 25 15:10 mc_name
drwxr-xr-x 2 root root 0 Jan 25 15:10 power
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank0
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank1
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank2
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank3
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank4
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank5
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank6
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank7
--w------- 1 root root 4096 Jan 25 15:10 reset_counters
-r--r--r-- 1 root root 4096 Jan 25 15:10 seconds_since_reset
-r--r--r-- 1 root root 4096 Jan 25 15:10 size_mb
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Jan 25 15:10 uevent
root@daq00:~#
</pre>

=== Intel E3-1270 v6 ===
<pre>
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --status
edac-ctl: drivers are loaded.
root@grsnis01:~# edac-util
edac-util: No errors to report.
root@grsnis01:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@grsnis01:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 max_location
-r--r--r-- 1 root root 4096 Feb 19 12:35 mc_name
drwxr-xr-x 2 root root 0 Feb 19 12:35 power
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank0
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank1
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank2
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank3
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank4
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank5
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank6
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank7
--w------- 1 root root 4096 Feb 19 12:35 reset_counters
-r--r--r-- 1 root root 4096 Feb 19 12:35 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 19 12:35 size_mb
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 19 12:35 uevent
root@grsnis01:~#
</pre>

=== Intel E3-1245 v6 ===

<pre>
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --status
edac-ctl: drivers are loaded.
[root@alphagdaq ~]# edac-util
edac-util: No errors to report.
[root@alphagdaq ~]# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
[root@alphagdaq ~]# ras-mc-ctl --layout
+-----------------------------------------------+
| mc0 |
| csrow0 | csrow1 | csrow2 | csrow3 |
----------+-----------------------------------------------+
channel1: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
----------+-----------------------------------------------+
[root@alphagdaq ~]# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#3channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#3channel#1 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#2channel#0 0 0
[root@alphagdaq ~]# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SSH-F
[root@alphagdaq ~]# ras-mc-ctl --summary
DBD::SQLite::db prepare failed: no such table: mc_event at /usr/sbin/ras-mc-ctl line 1129.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1130.
[root@alphagdaq ~]#
</pre>

=== AMD 3700X ===

(memory is non-ECC)

<pre>
root@daq13:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@daq13:~#
root@daq13:~#
root@daq13:~# edac-ctl --status
edac-ctl: drivers not loaded.
root@daq13:~# edac-util
edac-util: Error: No memory controller data found.
root@daq13:~# edac-util -s
edac-util: EDAC drivers loaded. No memory controllers found
root@daq13:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 2 root root 0 Jan 25 15:26 power
lrwxrwxrwx 1 root root 0 Jan 21 16:16 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Jan 21 16:16 uevent
</pre>

(memory is ECC)

<pre>
root@trinatdaq:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@trinatdaq:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@trinatdaq:~# edac-util
edac-util: No errors to report.
root@trinatdaq:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Dec 15 13:04 mc0
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
lrwxrwxrwx 1 root root 0 Dec 13 18:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dec 13 18:31 uevent
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_noinfo_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 max_location
-r--r--r-- 1 root root 4096 Dec 15 13:04 mc_name
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank4
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank5
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank6
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank7
--w------- 1 root root 4096 Dec 15 13:04 reset_counters
-rw-r--r-- 1 root root 4096 Dec 15 13:04 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Dec 15 13:04 seconds_since_reset
-r--r--r-- 1 root root 4096 Dec 15 13:04 size_mb
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Dec 15 13:04 uevent
root@trinatdaq:~#
</pre>

=== AMD 5000G ===

* no linux driver for AMD 5000-series "G" CPU
* no mention of ECC in the BIOS settings
* unclear status of ECC support in AMD documentation (sais only "pro" "G" CPUs have ECC)
* unclear status of ECC support in ASUS documentation (web page out of date)

=== AMD 5600X ===

<pre>
root@daq17:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-XE GAMING WIFI
root@daq17:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq17:~# edac-util
edac-util: No errors to report.
root@daq17:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@daq17:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Aug 19 19:27 mc0
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
lrwxrwxrwx 1 root root 0 May 10 10:11 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 May 10 10:11 uevent
root@daq17:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_noinfo_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 max_location
-r--r--r-- 1 root root 4096 Aug 19 19:27 mc_name
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank4
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank5
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank6
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank7
--w------- 1 root root 4096 Aug 19 19:27 reset_counters
-rw-r--r-- 1 root root 4096 Aug 19 19:27 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Aug 19 19:27 seconds_since_reset
-r--r--r-- 1 root root 4096 Aug 19 19:27 size_mb
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Aug 19 19:27 uevent
root@daq17:~#
</pre>

=== AMD 3955WX ===

<pre>
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --status
edac-ctl: drivers are loaded.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util
edac-util: No errors to report.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@alphasuperdaq:~/git/scripts/quotareport# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 19 root root 0 Dez 12 04:48 mc0
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
lrwxrwxrwx 1 root root 0 Dez 9 05:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dez 9 05:31 uevent
root@alphasuperdaq:~/git/scripts/quotareport#
root@alphasuperdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 max_location
-r--r--r-- 1 root root 4096 Feb 28 22:19 mc_name
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank0
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank1
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank10
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank11
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank12
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank13
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank14
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank15
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank2
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank3
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank4
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank5
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank6
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank7
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank8
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank9
--w------- 1 root root 4096 Feb 28 22:19 reset_counters
-rw-r--r-- 1 root root 4096 Feb 28 22:19 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Feb 28 22:19 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 28 22:19 size_mb
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 28 22:19 uevent
root@alphasuperdaq:~#
root@alphasuperdaq:~# ras-mc-ctl --layout
Use of uninitialized value $max_pos[3] in modulus (%) at /usr/sbin/ras-mc-ctl line 868.
Use of uninitialized value $d in numeric ge (>=) at /usr/sbin/ras-mc-ctl line 869.
Use of uninitialized value $d in sprintf at /usr/sbin/ras-mc-ctl line 872.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| mc0 |
| csrow0 | csrow1 |
| channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 | channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

0: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
root@alphasuperdaq:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#2 0 0
mc#0csrow#1channel#7 0 0
mc#0csrow#0channel#3 0 0
mc#0csrow#1channel#4 0 0
mc#0csrow#1channel#2 0 0
mc#0csrow#0channel#7 0 0
mc#0csrow#1channel#3 0 0
mc#0csrow#0channel#4 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#5 0 0
mc#0csrow#0channel#6 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#0channel#5 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#6 0 0
root@alphasuperdaq:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: ASUSTeK COMPUTER INC. model Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@alphasuperdaq:~#
</pre>

== Configure rasdaemon ==

<pre>
apt install rasdaemon
</pre>
<pre>
systemctl enable rasdaemon
systemctl restart rasdaemon
systemctl status rasdaemon
</pre>

<pre>
● rasdaemon.service - RAS daemon to log the RAS events
Loaded: loaded (/lib/systemd/system/rasdaemon.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-01-25 15:16:37 PST; 3min 5s ago
Main PID: 2477175 (rasdaemon)
Tasks: 1 (limit: 76958)
Memory: 17.1M
CGroup: /system.slice/rasdaemon.service
└─2477175 /usr/sbin/rasdaemon -f -r

Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Listening to events for cpus 0 to 11
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mc_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording aer_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording extlog_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mce_record events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording arm_event events
</pre>

== Get reports ==

* Intel 2x32GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --layout
+-------------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-------------------------+
channel1: | 16384 MB | 16384 MB |
channel0: | 16384 MB | 16384 MB |
----------+-------------------------+
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#0channel#1 0 0
root@daq00:~#
</pre>

* Intel 4x16GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#1 0 0
mc#0csrow#2channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#3channel#0 0 0
mc#0csrow#3channel#1 0 0
root@daq00:~#
root@daq00:~# ras-mc-ctl --layout
+-----------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-----------------------+
channel1: | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB |
----------+-----------------------+
root@daq00:~#
root@daq00:~#
root@daq00:~#
root@daq00:~# ras-mc-ctl --print-labels
ras-mc-ctl: Error: No dimm labels for Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@daq00:~#
</pre>

note: ubuntu LTS 22.04 DBD::SQLite::db error is not there.

= sensors =

== ASUS P9X79 WS ==

* https://www.asus.com/supportonly/P9X79%20WS/HelpDesk_Manual/
* BIOS version 4802
* modprobe nct6775
* modprobe coretemp

<pre>
root@daq14:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 0: +29.0°C (high = +82.0°C, crit = +100.0°C)
Core 1: +24.0°C (high = +82.0°C, crit = +100.0°C)
Core 2: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +82.0°C, crit = +100.0°C)

nouveau-pci-0200
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.00 V)
temp1: +39.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

nct6776-isa-0290
Adapter: ISA adapter
Vcore: 1.04 V (min = +0.00 V, max = +1.74 V)
in1: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
AVCC: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
+3.3V: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 2.04 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 904.00 mV (min = +0.00 V, max = +0.00 V) ALARM
3VSB: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
Vbat: 3.30 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1265 RPM (min = 0 RPM)
fan2: 1909 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +34.0°C (high = +0.0°C, hyst = +0.0°C) ALARM sensor = thermistor
CPUTIN: +58.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermal diode
AUXTIN: +31.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
PECI Agent 0: +31.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +96.0°C)
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

root@daq14:~#
</pre>

= Enable CPU turbo mode =

* Intel CPU has a nominal CPU frequency (i.e. 3.4GHz) and a turbo-boost CPU frequency (i.e. 4.0GHz). Here we will enable this turbo-boost mode.
* Find out CPU capability
<pre>
root@daq01:~# lscpu | grep Hz
Model name: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
CPU MHz: 3965.803
CPU max MHz: 4000.0000
CPU min MHz: 800.0000
root@daq01:~#
</pre>
* Look up this CPU in the Intel ARK database - google for the CPU model name, i.e.
https://ark.intel.com/content/www/us/en/ark/products/88196/intel-core-i7-6700-processor-8m-cache-up-to-4-00-ghz.html
* Find current frequency settings:
<pre>
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "powersave" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 2.72 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
root@daq01:~#
</pre>
* Note the following:
** current governor is "powersave"
** "performance" governor is available
** "boost state support" is supported and active.
* Confirm CPU frequency governor:
<pre>
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
powersave
powersave
powersave
powersave
powersave
powersave
powersave
powersave
root@daq01:~#
</pre>
* Change governor to "performance":
<pre>
root@daq01:~# cpupower frequency-set --governor performance
Setting cpu: 0
Setting cpu: 1
Setting cpu: 2
Setting cpu: 3
Setting cpu: 4
Setting cpu: 5
Setting cpu: 6
Setting cpu: 7
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
performance
performance
performance
performance
performance
performance
performance
performance
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "performance" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 3.93 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
</pre>
* monitor CPU frequency:
<pre>
root@daq01:~# cpupower monitor
| Nehalem || Mperf || Idle_Stats
CPU| C3 | C6 | PC3 | PC6 || C0 | Cx | Freq || POLL | C1 | C1E | C3 | C6 | C7s | C8
0| 0.00| 0.00| 0.00| 0.00|| 88.80| 11.20| 3973|| 0.00| 0.00| 0.01| 0.02| 0.31| 0.00| 4.25
4| 0.00| 0.00| 0.00| 0.00|| 4.70| 95.30| 3945|| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 95.03
1| 0.73| 3.70| 0.00| 0.00|| 4.52| 95.48| 3864|| 0.00| 0.01| 1.19| 0.44| 2.82| 0.00| 90.23
5| 0.73| 3.70| 0.00| 0.00|| 0.37| 99.63| 3807|| 0.00| 0.00| 0.03| 0.09| 1.70| 0.00| 97.64
2| 2.28| 12.86| 0.00| 0.00|| 1.41| 98.59| 3829|| 0.00| 0.86| 3.17| 0.46| 7.70| 0.00| 85.87
6| 2.28| 12.86| 0.00| 0.00|| 2.88| 97.12| 3856|| 0.00| 0.11| 4.56| 2.15| 10.31| 0.00| 78.99
3| 1.33| 4.81| 0.00| 0.00|| 0.99| 99.01| 3804|| 0.00| 0.49| 0.79| 0.01| 1.03| 0.00| 96.12
7| 1.34| 4.81| 0.00| 0.00|| 1.26| 98.74| 3818|| 0.00| 0.01| 2.32| 0.47| 5.02| 0.00| 90.06
root@daq01:~#
</pre>
* check that the CPU is not overheating:
<pre>
root@daq01:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 1: +38.0°C (high = +84.0°C, crit = +100.0°C)
Core 2: +34.0°C (high = +84.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +84.0°C, crit = +100.0°C)
</pre>
* congratulations, we are running at 4 GHz now!

= Setup ubuntu as gateway to private network =

See also:
* https://daq.triumf.ca/DaqWiki/index.php/VME-CPU#Setup_the_boot_host_computer_.28el7.29
* http://www.triumf.info/wiki/DAQwiki/index.php/Dhcpd_on_eth1

== Steps to do ==

* assign network numbers to the private network, i.e. 192.168.1.x, 192.168.2.x, etc
* (on the gateway machine, each private network interface has to have a different network number)
* (each network interface can have multiple networks attached, via VLANs or via eth0:0, eth0:1 constructs)
* assign IP addresses on the private network, save them in /etc/hosts i.e. "hvps 192.168.1.10"
* (for simplicity, assign 192.168.1.1 to the gateway machine itself)
* (IP addresses 192.168.1.0 and 192.168.1.255 are "special", do not use them)
* setup DNS server (dnsmasq) to serve contents of /etc/hosts via DNS (otherwise, many programs will see inconsistent name to IP address mapping)
* setup DHCP server (ISC dhcpd or dnsmasq) to give out the IP addresses
* setup tftp, pxelinux and NFS for diskless booting
* setup time server (chronyd) to provide common time to all devices
* setup NAT so machines on private network can access the internet (to get OS updates, etc)
* setup NIS and NFS so machines on the private network can use common home directories
* setup rsync backup of machines on the private network

== setup hosts ==

* edit /etc/hosts
<pre>
192.168.1.101 dsfe01
... and so forth
</pre>

== setup dns and dhcp ==

* apt install dnsmasq
* edit /etc/dnsmasq.conf
<pre>
# /etc/dnsmasq.conf
# DNS settings
#port=0 # disable DNS function
port=53 # enable DNS function
domain-needed
bogus-priv
no-resolv
server=142.90.100.19
# DHCP settings
interface=enp1s0f0 # DHCP interface
#dhcp-range=192.168.1.50,192.168.1.150,infinite
dhcp-range=192.168.1.0,static
#log-dhcp
quiet-dhcp
#dhcp-ignore=tag:!known
dhcp-boot=pxelinux.0
#dhcp-host=ac:1f:6b:9e:7f:4a,192.168.1.100,10m
dhcp-host=ac:1f:6b:9e:7f:4a,dsfe01,infinite
# TFTP settings
enable-tftp
tftp-root=/tftpboot
</pre>
* #mkdir /zssd/tftpboot ### per tftp-root (if no ZFS)
* zfs create -o mountpoint=/tftpboot rpool/tftpboot ### (if root is ZFS)
* systemctl stop systemd-resolved.service
* systemctl disable systemd-resolved.service
* rm /etc/resolv.conf
* create new /etc/resolv.conf with this contents:
<pre>
nameserver 127.0.0.1
search snolab.ca
</pre>
* systemctl enable dnsmasq
* systemctl restart dnsmasq

== setup chronyd ==

* enable ntp server:
* configure and enable chronyd per instructions above
* echo "allow 192.168.1.0/24" > /etc/chrony/conf.d/allow-localhost.conf
* systemctl restart chronyd
* chronyc tracking ### wait until time is synchronized (a few seconds)

== setup diskless network booting ==

=== setup pxelinux ===
<pre>
cd ~
wget https://www.kernel.org/pub/linux/utils/boot/syslinux/4.xx/syslinux-4.03.tar.bz2
tar xjvf syslinux-4.03.tar.bz2
cd syslinux-4.03
cp -pv ./core/pxelinux.0 ./com32/hdt/hdt.c32 ./memdisk/memdisk ./com32/menu/menu.c32 /zssd/tftpboot/
</pre>
* cd /zssd/tftpboot
<pre>
wget http://ladd00.triumf.ca/tftpboot/memtest86+-4.20.iso.zip
wget http://ladd00.triumf.ca/tftpboot/memtest86+-5.01.iso.gz
wget http://ladd00.triumf.ca/tftpboot/modules.alias
wget http://ladd00.triumf.ca/tftpboot/modules.pcimap
wget http://ladd00.triumf.ca/tftpboot/pci.ids
</pre>
* mkdir pxelinux.cfg
* emacs -nw pxelinux.cfg/default
<pre>
default menu.c32
prompt 0

menu title Welcome to the DSVSLICE PXE boot menu

timeout 50

label hdt
kernel hdt.c32

label memtest86+-5.01
kernel memdisk iso initrd=memtest86+-5.01.iso.gz

label memtest86+-4.20
kernel memdisk iso initrd=memtest86+-4.20.iso.zip

label vmlinuz-5.3.0-26-generic
menu default
kernel vmlinuz-5.3.0-26-generic
append initrd=initrd.img-5.3.0-26-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.1.1:/zssd/nfsroot/dsfe01 toram ip=dhcp panic=60 BOOTIF=enp1s0f0

#end
</pre>

=== setup linux kernel ===

* copy the kernel files
<pre>
cd /boot
rsync -av config* initrd* System.map* vmlinuz* /zssd/tftpboot/
</pre>
* cd /zssd/tftpboot
* chmod a+r *

=== setup nfs ===

* apt-get install nfs-kernel-server
* emacs -nw /etc/exports
<pre>
/zssd/nfsroot/dsfe01 dsfe01(rw,no_root_squash,async,no_subtree_check)
</pre>
* enable services
<pre>
systemctl enable nfs-server
systemctl enable nfs-mountd
systemctl enable nfs-idmapd
systemctl restart nfs-server
systemctl restart nfs-mountd
systemctl restart nfs-idmapd
</pre>
* after editing /etc/exports, run
<pre>
exportfs -av
</pre>

=== setup userland ===

* zfs create zssd/nfsroot
* zfs set dedup=verify zssd/nfsroot ### enable deduplication to save disk space because most linux images have mostly identical files
* clone ubuntu
<pre>
mkdir /zssd/nfsroot/dsfe01
cd /
rsync -avx . /zssd/nfsroot/dsfe01
</pre>
* edit config files:
* cd /zssd/nfsroot/dsfe01
* emacs -nw etc/hostname ### change to dsfe01
* emacs -nw etc/mailname ### change to dsfe01
* emacs -nw etc/yp.conf ### change daq00.triumf.ca to musr00.triumf.ca
* emacs -nw etc/defaultdomain ### change to MUSR-NIS
* cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
* emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
* emacs -nw root/.ssh/authorized_keys ### update root ssh keys
* emacs -nw etc/fstab ### add this
<pre>
192.168.1.1:/zssd/nfsroot/dsfe01 / nfs defaults,nolock 0 0
</pre>
* emacs -nw etc/chrony/chrony.conf
** comment-out all "pool" and "server" entries
** add entry "server 192.168.1.1 iburst"

After dsfe01 is booted:

* disable services:
<pre>
systemctl disable apache2
systemctl disable dnsmasq
systemctl disable zfs-import-cache
</pre>

To setup additional machines, clone dsfe01 instead of cloning the gateway machine

=== Allow manpages to be viewed ===

If <code>/</code> is mounted over NFS, <code>man</code> will report a permission error. Fix it with:

<pre>
ln -s /etc/apparmor.d/usr.bin.man /etc/apparmor.d/disable/
apparmor_parser -R /etc/apparmor.d/usr.bin.man
</pre>

== setup shared home directory ==

=== on the gateway machine ===
* define netgroups
* emacs -nw /etc/netgroup
<pre>
dsfe (dsfe01,,) (dsfe02,,)
</pre>
* emacs -nw /etc/nsswitch.conf ### edit the netgroup line to read:
<pre>
netgroup: files
</pre>
* export the home directories:
* emacs -nw /etc/exports ### add this:
<pre>
/zssd/home1 @dsfe(rw,no_root_squash,async,no_subtree_check)
</pre>
* exportfs -rc

=== on the frontend machine ===

* mkdir /home
* emacs -nw /etc/fstab ### add this:
<pre>
192.168.1.1:/zssd/home1 /home nfs defaults 0 0
</pre>
* mount -a

== setup NAT ==

NAT allows machines on the private network to connect to the internet: https://en.wikipedia.org/wiki/Network_address_translation

In these examples:
* replace "eno1" with name of the outgoing interface (the one connected to the TRIUMF network).
* replace "enp11s0" with name of the private network interface (192.168.1.x network)

* emacs -nw /etc/rc.local ### add this:
<pre>
# /etc/rc.local

/sbin/iptables -t nat -A POSTROUTING -o eno1 -j MASQUERADE
iptables -L -v

# uncomment following lines if machine has prohibitive FORWARD rules:
#/sbin/iptables -I FORWARD -i eno1 -o enp11s0 -m state --state RELATED,ESTABLISHED -j ACCEPT
#/sbin/iptables -I FORWARD -i enp11s0 -o eno1 -j ACCEPT
#iptables -L -v

iptables -L -v
sysctl -w net.ipv4.ip_forward=1
#sysctl -a | grep forward

sh /etc/firewall-rfc1918.sh

# end
</pre>
* emacs -nw /etc/firewall-rfc1918.sh
<pre>
# firewall-rfc1918.sh

# prevent RFC1918 private network IP addresses from
# going in and out from our uplink.

ETH=eno1

iptables -F in-rfc1918
iptables -N in-rfc1918
iptables -A in-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A in-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A in-rfc1918 --dst 192.168.0.0/16 -j REJECT
iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -I INPUT -j in-rfc1918 -i $ETH

iptables -F out-rfc1918
iptables -N out-rfc1918
iptables -A out-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A out-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A out-rfc1918 --dst 192.168.0.0/16 -j REJECT
iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -I OUTPUT -j out-rfc1918 -o $ETH

iptables -L -v

#end
</pre>

= KVM =

<pre>
apt install cpu-checker

root@daq13:~# kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
root@daq13:~#

(if not, shutdown, go into BIOS settings, enable CPU virtualization)

apt install virtinst ### will install many packages
apt install libvirt-clients libvirt-daemon-system-systemd libvirt-daemon qemu qemu-kvm libvirt-daemon-system virtinst bridge-utils

root@daq13:/home1/wheel# virsh list --all
Id Name State
------------------------------
1 ubuntu-guest running

apt install virt-manager

virt-install --name ubuntu-guest --os-variant ubuntu20.04 --vcpus 2 --ram 2048 --location /daq/daqstore/olchansk/linux/Ubuntu/ubuntu-20.04.3-desktop-amd64.iso --network bridge=virbr0,model=virtio --graphics none --extra-args='console=ttyS0,115200n8 serial'

virtual machine will start, boot, etc
to get out of it, CTRL + Shift followed by ]

ssh wheel@daq13
virt-manager

run virt-install again, omit "--graphics none", open graphics console from virt-manager, it booted into ubuntu installer desktop

virt-install --name test10 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --filesystem /kvm_ladd00,/ --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial" --graphics none

virt-install --name test14 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --disk /tmp/xxx/ladd00.img,bus=sata --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial rdshell" --graphics none --check path_in_use=off
</pre>

build image

<pre>
dd if=/dev/zero of=/tmp/xxx/ladd00.img bs=1024M count=20
mkfs.ext3 /tmp/xxx/ladd00.img ### ext4 fails to mount by SL6 kernel, "unknown ext4 options"
cd /kvm_ladd00/
mount -o loop /tmp/xxx/ladd00.img /mnt/tmp
rsync -av . /mnt/tmp/ --delete
umount /mnt/tmp
</pre>

on the guest, configure network: /etc/rc.local
<pre>
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

ifconfig eth2 192.168.122.2
route add -net 0.0.0.0 gw 192.168.122.1
ifconfig -a
netstat -rn

# end
</pre>

= ARM cross-compiler =

<pre>
apt install libgcc-9-dev-arm64-cross
apt install gcc-arm-linux-gnueabi
apt install gcc-arm-linux-gnueabihf
apt install g++-arm-linux-gnueabihf
apt install g++-arm-linux-gnueabi
</pre>

<pre>
arm-linux-gnueabi-gcc -o ttcp1 ttcp.c -march=armv7 -static
arm-linux-gnueabi-gcc -o memcpy.armv7 memcpy.cc -march=armv7 -static -O2
</pre>

= 32-bit intel cross-compiler =

Ubuntu 22.04:

<pre>
apt install libstdc++-11-dev:i386
apt install zlib1g-dev:i386
</pre>

for MIDAS, use "make linux32"

Ubuntu

2023-11-11T00:01:29Z

Lmartin: /* move /home/wheel */

= About Ubuntu =

AAA

= Ubuntu version =

<pre>
lsb_release -a
uname -a
</pre>

= Ubuntu installer =

* updated for Ububtu LTS 20.04.01, 22.04.1

* download the latest Ubuntu LTS desktop installer iso image
* dd the image to a USB key
* power down, disconnect all disks (all HDDs, all SSDs, all M.2)
* connect the SSD to be used as system disk
* if system will use mirrored SSDs (using ZFS mirror), leave second SSD disconnected, we will activate it later
* power up
* boot from USB key in legacy mode or UEFI mode (select this in the BIOS boot menu - F8 for ASUS, F11 for Supermicro)
* follow the instruction:
* "try ubuntu or install ubuntu" - choose "install"
* select language - accept default
* "updates and other software" - accept default settings ("normal install")
* "installation type" - select "advanced features" and "experimental: use ZFS"
* accept partition choice
* "where are you?" - select "Vancouver" (PST time zone)
* "who are you?" - leave all fields blank, except "username" set to "wheel", "password" set to the root password. hostname will be set later after configuring the network
* installation runs in a few minutes, when finished, reboot
* login as user wheel
* answer annouying questions:
* "livepatch" - say "next"
* "help improve" - select "do not send", say "next"
* "privacy" - leave "location" as "off", say "next"
* "ready to go", say "done"
* right-click on the desktop, say "open in terminal", a shell will open
* say "sudo /bin/bash", enter the root password, you now have the root shell
* run nm-connection-editor to configure the network. use netmask 255.255.224.0, gateway 142.90.100.18, DNS 142.90.100.19, search path "triumf.ca"
* after network is up (can ping ladd00), continue with post-installation steps below

= Install instructions =

== prepare ==
<pre>
apt update
apt upgrade
</pre>

== install ssh ==
<pre>
apt install ssh
</pre>

== configure hostname ==
<pre>
vi /etc/hostname
</pre>

== disable swap ==

ubuntu installer creates a 2 GB swap partition, not useful
on 32-64 GB machine, disable it:
<pre>
vi /etc/fstab ### comment out the "swap" line
</pre>

== maybe reboot ==

this is a good point to reboot the machine to boot
the latest kernel and to set the correct hostname

== install etckeeper ==

keep contents of /etc in a git repository:

<pre>
apt -y install etckeeper
</pre>

== set timezone ==
<pre>
timedatectl list-timezones | grep -i vancouver
timedatectl set-timezone America/Vancouver
</pre>

== install time synchronization ==
<pre>
apt -y install chrony
echo server time1.triumf.ca iburst >> /etc/chrony/chrony.conf
echo server time2.triumf.ca iburst >> /etc/chrony/chrony.conf
echo server time3.triumf.ca iburst >> /etc/chrony/chrony.conf
systemctl disable systemd-timesyncd.service
systemctl stop systemd-timesyncd.service
systemctl disable ntp
systemctl stop ntp
systemctl enable chrony
systemctl restart chrony
chronyc sources
chronyc tracking
</pre>

== reenable systemd-timesyncd ==

ONLY IF CHRONY DOES NOT WORK

<pre>
apt remove chrony
systemctl enable systemd-timesyncd.service
systemctl restart systemd-timesyncd.service
systemctl status systemd-timesyncd.service
timedatectl status
timedatectl timesync-status
</pre>

== enable outgoing email (debian 11) ==

this is different from ubuntu 20. it uses /etc/mailname and it hardwires the hostname into main.cf.

== enable outgoing email ==

* TRIUMF: use smtp.triumf.ca
* CERN: use cernmx.cern.ch

<pre>
apt install postfix ### select "satellite system", enter full hostname "xxx.triumf.ca", enter "smtp.triumf.ca"
apt install mailutils
dpkg-reconfigure postfix ### (if postfix already installed)
</pre>
<pre>
echo olchansk@triumf.ca lindner@triumf.ca bsmith@triumf.ca >> ~root/.forward
mailx root
test
^D
</pre>

== enable ping for all users (debian 11) ==

Without this tweak, Debian will report "operation not permitted" if a user tries to ping somewhere.

<pre>
echo 'net.ipv4.ping_group_range = 0 1000' > /etc/sysctl.d/99-ping.conf
</pre>

== install missing packages ==

(apt eats terminal input, even the "yes |" trick does not quite work,
repeat the following commands until they report that everything
is installed)

<pre>
yes | apt -y install ssh tcsh ethtool ncat rsync strace net-tools sysstat smartmontools lm-sensors traceroute time minicom screen git lsof debsums tmux
yes | apt -y install lsb-release
yes | apt -y install flex bison
yes | apt -y install neofetch
yes | apt -y install snmp snmp-mibs-downloader
yes | apt -y install git subversion g++ gfortran cmake doxygen
yes | apt -y install python
yes | apt -y install curl libcurl4 libcurl4-openssl-dev
yes | apt -y install mariadb-client ### mysql client
yes | apt -y install libz-dev sqlite3 libsqlite3-dev unixodbc-dev
yes | apt -y install libssl-dev
yes | apt -y install emacs xemacs21 joe
yes | apt -y install gnuplot dos2unix
yes | apt -y install mutt bsd-mailx # email clients
yes | apt -y install liblz4-tool pbzip2
yes | apt -y install libc6-dev-i386 # otherwise no /usr/include/sys/types.h
yes | apt -y install libreadline-dev
yes | apt -y install chromium-browser chromium-codecs-ffmpeg-extra
yes | apt -y install ubuntu-mate-themes
yes | apt -y install libmotif-dev libxmu-dev
yes | apt -y install libusb-dev libusb-1.0-0-dev
yes | apt -y install xfig gsfonts-x11 gsfonts-other # install fonts for xfig
yes | apt -y install libjson-perl
yes | apt -y install libgsl-dev # additional GNU Scientific Library
yes | apt -y install qt5-default # Qt development
yes | apt -y install python3.8-full python3.8-dev python3.8-dbg python3-pip ### for pyROOT
yes | apt -y install imagemagick imagemagick-common ckeditor # for elog
yes | apt -y install libjpeg-dev libjpeg-progs libjpeg-tools
yes | apt -y install linux-tools-common linux-tools-generic # cpupower frequency-info
yes | apt -y install rdesktop remmina remmina-plugin"*" # requested by POL
</pre>

Ubuntu LTS 20.04:
<pre>
yes | apt -y install linux-image-generic-hwe-20.04 linux-tools-virtual-hwe-20.04 # enable linux 5.11 series kernel
</pre>

== install git/scripts ==
<pre>
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd scripts
git pull
</pre>

== disable swap (debian 11) ==

* on 64 GB RAM machines swap is not useful
* on machines booted from network (NFS-ROOT), swap does not work
* on machines running from flash (RPi, etc), flash is too slow for useful swap
* swap configured by linux installers invariably has wrong size and is not useful

<pre>
systemctl disable dphys-swapfile
systemctl stop dphys-swapfile
dphys-swapfile uninstall
</pre>

== configure DNS ==

<pre>
cd ~/git/scripts
git pull
mkdir /etc/systemd/resolved.conf.d
cp etc/resolved-triumf.conf /etc/systemd/resolved.conf.d/
systemctl restart systemd-resolved
resolvectl
#systemd-analyze cat-config systemd/resolved.conf
</pre>

== install ganglia ==

<pre>
yes | apt-get -y install ganglia-monitor
systemctl enable ganglia-monitor
</pre>
<pre>
cd ~root/git/scripts
git pull
cp etc/gmond-ubuntu.conf /etc/ganglia/gmond.conf
ln -s ganglia/gmond.conf /etc
# ln -s arm-linux-gnueabihf/ganglia /usr/lib/ganglia ### fix path for ARM CPU
# ln -s i386-linux-gnu/ganglia /usr/lib/ganglia ### fix path for 32-bit Intel CPU
systemctl restart ganglia-monitor
systemctl status ganglia-monitor
ps -efw | grep gmond
</pre>
<pre>
cd ~root/git/scripts/ganglia
make install
./ganglia-all.perl
</pre>

== install gonodeinfo ==

* go to https://bitbucket.org/dd1/gonodeinfo follow instructions:
<pre>
yes | apt-get -y install golang
mkdir ~/git
cd ~/git
git clone https://bitbucket.org/dd1/gonodeinfo.git
cd gonodeinfo
git pull
make
make install # install gonodeinfo agent
cd ~ # this is important
</pre>
* edit /etc/gonodeinfo.conf
* change "Description", "Location", "User" and "Administrator" as appropriate (or delete them)
* change "Servers" to read: Servers: daq00.triumf.ca:8601
* run "gonodeinfo -v"
* if error is "connection refused". go to the nodeinfo server to add this client to the access control list:
* on the gonodeinfo server: run /opt/gonodeinfo/gonodereceive.exe -a daq13
* try gonodeinfo again, there should be no error
* on the gonodeinfo server: run gonodereport, look at the web pages, the new machine should be listed now

== install fonts for EPICS ==

* apt install xfonts-100dpi xfonts-75dpi
* restart Xorg (i.e. "killall Xorg", this will log you out from the console)
* xlsfonts | grep -i helvetica ### should show fonts with different sizes, not just size 0 (scalable)

== install libz.so.1 for CentOS compatibility ==

KO - confirm which versions on quartus need this.

<pre>
yes | apt-get -y install zlib1g
yes | apt-get -y install zlib1g:i386 libc6:i386 libgcc1:i386 gcc-6-base:i386
</pre>

== install libpng12.so.0 for Quartus compatibility ==

(does not work anymore!!!)

<pre>
wget http://ftp.ca.debian.org/debian/pool/main/libp/libpng/libpng12-0_1.2.50-2+deb8u2_amd64.deb
dpkg --install libpng12-0_1.2.50-2+deb8u2_amd64.deb
</pre>

== install libpng12.so.0 for Quartus 13.0sp1 ==

<pre>
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0.50.0
/bin/cp -pv libpng12.so.0 libpng12.so.0.50.0 /lib/x86_64-linux-gnu/
</pre>

== install packages for Xilinx ==

ubuntu LTS 22.04 vivado 2020.1

<pre>
apt install autoconf libtool
apt install libtinfo5
apt install texinfo
apt install zlib1g:i386
</pre>

== install packages for building ROOT ==

<pre>
apt -y install libx11-dev libxpm-dev libxft-dev libxext-dev libpng-dev libjpeg-dev xlibmesa-glu-dev libxml2-dev libgsl-dev cmake
</pre>

== install 32-bit libraries for PHYSICA ==

these instructions are for running 32-bit physica executable built for SL6 on ubuntu LTS 20.04

install physica sources (cannot build, do not have g77)

<pre>
cd ~/packages
git clone https://bitbucket.org/ttriumfdaq/physica.git
</pre>

install 32-bit libraries using ubuntu package manager:

<pre>
apt install lib32z1 # libz.so
</pre>

copy 32-bit SL6 shared libraries to /lib32

<pre>
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libX11.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libgd.so.2 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libpng12.so.0 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libreadline.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libncurses.so.5 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libg2c.so.0 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libxcb.so.1 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libXpm.so.4 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libjpeg.so.62 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libfontconfig.so.1 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libfreetype.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libtinfo.so.5 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libXau.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libexpat.so.1 /lib32/
</pre>

ldd should report:

<pre>
trinatdaq:trinat> ldd /usr/local/physica/physica.exe
linux-gate.so.1 (0xf7fa2000)
libX11.so.6 => /lib32/libX11.so.6 (0xf7e43000)
libgd.so.2 => /lib32/libgd.so.2 (0xf7dfe000)
libpng12.so.0 => /lib32/libpng12.so.0 (0xf7dd6000)
libz.so.1 => /lib32/libz.so.1 (0xf7db8000)
libreadline.so.6 => /lib32/libreadline.so.6 (0xf7d7e000)
libncurses.so.5 => /lib32/libncurses.so.5 (0xf7d5b000)
libg2c.so.0 => /lib32/libg2c.so.0 (0xf7d3d000)
libm.so.6 => /lib32/libm.so.6 (0xf7c39000)
libgcc_s.so.1 => /lib32/libgcc_s.so.1 (0xf7c1a000)
libc.so.6 => /lib32/libc.so.6 (0xf7a2f000)
libxcb.so.1 => /lib32/libxcb.so.1 (0xf7a05000)
libdl.so.2 => /lib32/libdl.so.2 (0xf79ff000)
libXpm.so.4 => /lib32/libXpm.so.4 (0xf79ee000)
libjpeg.so.62 => /lib32/libjpeg.so.62 (0xf7997000)
libfontconfig.so.1 => /lib32/libfontconfig.so.1 (0xf7962000)
libfreetype.so.6 => /lib32/libfreetype.so.6 (0xf78c9000)
libtinfo.so.5 => /lib32/libtinfo.so.5 (0xf78b0000)
/lib/ld-linux.so.2 (0xf7fa4000)
libXau.so.6 => /lib32/libXau.so.6 (0xf78ad000)
libexpat.so.1 => /lib32/libexpat.so.1 (0xf7885000)
trinatdaq:trinat>
</pre>

set login environment:

<pre>
setenv TRIUMF_FONTS $HOME/packages/physica/fonts
setenv PHYSICA_DIR $HOME/packages/physica
alias physica $PHYSICA_DIR/physica-SL6-32
</pre>

test:

<pre>
cd ~/packages/physica
physica
@rangauss.pcm
</pre>

== install lightdm ==

unlike the default gdm login manager, lightdm shows the machine hostname and does not require an extra mouse click to swicth from screen saver to login mode.

<pre>
apt -y install lightdm
# select lightdm
</pre>

== install desktop environments ==

note: default display manager and default desktop are deficient, please do not skip this step.

note: if apt asks to choose the display manager, select "lightdm"

note: KO - I recommend the "MATE" desktop.

note: you will have to cut-and-paste this several times because "apt" eats commands, even with "-y" and even piped from "yes".

<pre>
# install MATE desktop
DEBIAN_FRONTEND=noninteractive apt -y install ubuntu-mate-core ubuntu-mate-desktop ubuntu-mate-themes
# install Cinnamon desktop
DEBIAN_FRONTEND=noninteractive apt -y install cinnamon
# install KDE desktop
DEBIAN_FRONTEND=noninteractive apt -y install kubuntu-desktop
# install Lxqt desktop
DEBIAN_FRONTEND=noninteractive apt -y install lxqt
# install Xfce4 desktop
DEBIAN_FRONTEND=noninteractive apt -y install xfce4
</pre>

== install ROOT ==

Please install ROOT per instructions at http://root.cern.ch.

NOTE1: The ROOT package available from Ubuntu repositories is severely out of date and cannot be used with MIDAS and ROOTANA. ### DO NOT DO THIS! apt-get install root-system

NOTE2: as of 2017-Jan-09, ROOT binary kits for Ubuntu do not work (use GCC 5 instead of GCC6), build from source instead.

== Install x2go ==

KO - is this still needed? does it cause any security problems?

x2go instructions, thanks to Art O.

<pre>
add-apt-repository ppa:x2go/stable
apt-get update
apt-get install x2goserver x2goserver-xsession
</pre>

== enable root login from ladd00/daq00 ==
<pre>
ssh localhost
CTRL-C
/bin/cp ~root/git/scripts/etc/authorized_keys ~root/.ssh/
</pre>

== install smart-status ==
<pre>
ln -s ~/git/scripts/smart-status/smart-status.perl ~root/
</pre>

== enable boot menu and boot messages ==

This will enable the grub menu (with a 10 sec timeout) and
replace black screen with exciting linux boot messages.

* emacs -nw /etc/default/grub
<pre>
GRUB_DEFAULT=0
#GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="vga=769 video=640x480"
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX=""
#GRUB_GFXMODE=640x480
</pre>
* update grub config:
<pre>
grub-mkconfig -o /boot/grub/grub.cfg
</pre>

== reboot ==

this completes installation of the base system.

following sections modify basic ubuntu to fix known problems and to enable special stuff.

= Enable automatic updates =

<pre>
apt install unattended-upgrades
cd ~/git/scripts
git pull
/bin/cp -v etc/99apt-conf-ko /etc/apt/apt.conf.d/
apt-config dump | grep Unattended
</pre>

Following is obsolete:

* emacs -nw /etc/apt/apt.conf.d/50unattended-upgrades
** uncomment in Allowed-Origins "-security" and "-updates"
** add in Allowed-Origins: "Google LLC:stable";
** uncomment/add: "Unattended-Upgrade::Mail "root";
* emacs -nw /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
</pre>
* test: unattended-upgrade --dry-run -v

NOTE: update-on-shutdown is disabled.

NOTE: there is no update-on-boot, but:

NOTE: if machine was off for a long time, the systemd update timer would have expired and it will fire soon after reboot, causing an automatic update run. this is unwanted, and there is no fix or workaround for it. K.O. June-2023.

= Fix bpool is full =

!!! only if ROOT on ZFS !!!

There is an error in the zsys package that causes bpool to run out of space,
see [[#Ubuntu zsys]] for more details.

To fix:
<pre>
cd ~/git/scripts
git pull
cp etc/zsys.conf /etc/
zsysctl service reload
zsysctl service gc
zpool list bpool
zfs list bpool
df /boot
</pre>

= IPMI instructions =

IPMI is the board management hardware on Supermicro and other server motherboards. This includes hardware sensors - fan rotation speed, temperatures and power supply voltages.

<pre>
apt-get install ipmitool
systemctl enable ipmievd
systemctl restart ipmievd
</pre>

Run:
* ipmitool sel list ### event list
* ipmitool sel elist ### event list
* ipmitool sel clear ### clear event list (if it becomes full)
* ipmitool sensor ### report hardware sensors

= move /home/wheel =

note: this MUST be done if ZFS root and NIS/autofs with /home.

Default location of wheel's home directory will collide with autofs /home, it has to be moved,
for example to /wheel.

<pre>
# logout from the wheel user
# go to another computer
ssh root@daqubuntuxxx
zfs list | grep wheel ### identify zfs name wheel_xxxxxx
zfs set mountpoint=/wheel rpool/USERDATA/wheel_hm8fzh
emacs -nw /etc/passwd ### change wheel's home directory from /home/wheel to /wheel
su - wheel ### check that user wheel still works
</pre>

This will break wheel's ability to run snap programs, such as firefox, install chrome as listed below.

= enable NIS (ubuntu 22.04, debian 11) =

<pre>
apt -y install rpcbind nis
echo DAQ-NIS >> /etc/defaultdomain
echo ypserver daq00.triumf.ca >> /etc/yp.conf
systemctl enable ypbind.service
systemctl restart ypbind.service
systemctl status ypbind.service
ypwhich -m
</pre>

enable ypserv:

<pre>
sed -i s/NISSERVER=false/NISSERVER=slave/ /etc/default/nis
/usr/lib/yp/ypinit -s daq00
echo ypserver localhost >> /etc/yp.conf
sed -i "s/ypserver .*/ypserver localhost/" /etc/yp.conf
systemctl enable ypserv
systemctl restart ypserv
systemctl restart ypbind
</pre>

edit /etc/nsswitch.conf to read:

<pre>
# begin get data from nis
passwd: files nis
group: files nis
shadow: files nis
automount: files nis
netgroup: files nis
# end get data from nis
</pre>

enable hourly update of nis maps:

<pre>
mkdir ~root/git
cd ~root/git
git clone http://daq00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>

If this is a new machine, then on the master NIS node (daq00), add this new node to /etc/netgroup, and update NIS maps (cd /var/yp; make)

= enable NIS (ubuntu 20.04) =

* apt-get -y install portmap nis ### will ask for NIS domain (DAQ-NIS)
* dpkg-reconfigure nis ### reconfigure if already installed
* ypwhich -m
* edit /etc/default/nis
** set "NISSERVER=slave"
** Ubuntu LTS 20.04, check that "YPBINDARGS=" is blank, remove "-no-dbus" if it is there
* #edit /etc/yp.conf, comment-out everything, add "domain DAQ-NIS server localhost"
* edit /etc/yp.conf, comment-out everything, add "ypserver localhost"
* /usr/lib/yp/ypinit -s daq00
* systemctl enable nis
* systemctl restart nis
* ypwhich
* ypwhich -m
* ypcat -k passwd
* vi /etc/nsswitch.conf ### add the automount line, modify the passwd, group and shadow lines to read this:
<pre>
# begin get data from nis
passwd: files nis
group: files nis
shadow: files nis
automount: files nis
netgroup: files nis
# end get data from nis
</pre>
* enable hourly update of NIS maps
<pre>
mkdir ~root/git
cd ~root/git
git clone http://ladd00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>
* ### NOT NEEDED sudo vi /etc/idmapd.conf ### add line: "Domain = triumf.ca"

= enable autofs =

<pre>
apt -y install autofs
systemctl enable autofs
systemctl restart autofs
ls -l /home/olchansk ### test autofs, check file owner is correct
</pre>

= enable NFS server =

<pre>
apt install nfs-kernel-server
#edit /etc/exports
systemctl enable nfs-server
systemctl restart nfs-server
</pre>

= NIS master =

notes for setting up the NIS master

== wheel user ==

"wheel" is the default administrative user. We do not want it's password exported to NIS (encrypted password hash is world visible) and we do not want it's home directory exported to NFS (~wheel/.ssh is world visible and potentially writable: anybody can change ~wheel/.ssh/authorized_keys).

* move wheel's home directory from /home/wheel to /wheel (see special section about this)
* change wheel's UID and GID from 1000 to a value below MINUID in /var/yp/Makefile

== coherent uids ==

we do not want system accounts defined in /etc/passwd of the NIS master
to be included in the NIS map "passwd". this causes trouble on NIS clients
where newly installed packages fail to create local system users because same
user already exists in NIS.

This is controlled by MINUID in /var/yp/Makefile.

Historical TRIUMF uids start from around 200, but several clusters do not have any historic TRIUMF uids below 500 and MINUID is set to:
* DAQ-NIS: MINUID=200
* ISAC-NIS: MINUID=500
* TITAN-NIS: MINUID=500
* MUSR-NIS: MINUID=500
* TIG-NIS: MINUID=500 (100 on SL6 mother8pi)

Ubuntu 20 has two programs to create users:
* adduser - creates new users with UID 1000 and up as specified in /etc/adduser.conf. No problems here.
* adduser --system - creates new system users with UID 100 and up as specified in /etc/adduser.conf. No problems here.
* useradd - creates new users with UID 1000 and up as specified in /etc/login.defs. No problems here.
* useradd --system - creates new system users with UID 999 and down (read "man useradd", section at the end about SYS_UID_MAX). This collides with NIS MINUID, these system users will be included in the NIS map and cause trouble.

This problem cannot be fixed, SYS_UID_MIN, SYS_UID_MAX and UID_MIN in /etc/login.defs do not seem
to have any effect on UIDs chosen by "useradd --system". (tested on Ubuntu LTS 20.04).

So far only these system accounts seem to be affected by this:
* systemd-coredump
* ganglia

To fix:
* run "sort -r -n -t: -k3 /etc/passwd" to identify the last unused system user uid (range 100..200)
* run "sort -r -n -t: -k3 /etc/group" to identify the last unused system user gid (range 100.200)
* systemd-coredump: manually change UID and GID (package systemd-coredump is usually not installed)
* ganglia: same thing, then change ownership on all ganglia files.

Also read systemd author's opinion on system vs user UIDs:
https://github.com/systemd/systemd/issues/4850#issuecomment-265698275

= Fix systemd-logind NIS breakage =

!!! THIS IS NOT NEEDED FOR UBUNTU LTS 20.04 !!!

there is a delay in ssh logins for normal users. "ssh -v" shows the delay is after "pledge...". this
fix removes the delay.

systemd developers think that we should not use NIS and made sure there are
problems if we do. To give them credit, they do offer a workaround. Read this:
https://github.com/poettering/systemd/commit/695fe4078f0df6564a1be1c4a6a9e8a640d23b67

<pre>
mkdir /etc/systemd/system/systemd-logind.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-logind.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-logind.service
</pre>

= Fix systemd-udevd NIS breakage =

see same problem as above with udev getting stuck. ubuntu lts 20.04.

<pre>
mkdir /etc/systemd/system/systemd-udevd.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-udevd.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-udevd.service
</pre>

= Configure USB device permissions =

Configure USB device permissions for user access to USB-serial devices, Altera USB Blaster, etc.

* create file /etc/udev/rules.d/99-usb-chmod.rules with this contents:

<pre>
emacs -nw /etc/udev/rules.d/99-usb-chmod.rules
ACTION=="add", SUBSYSTEM=="usbmisc", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /dev/%c"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /proc/%c"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVICE}"
ACTION=="add", ENV{PHYSDEVBUS}=="usb-serial", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVPATH}=="/class/tty/ttyS*", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyUSB*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyACM*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyS*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", DEVPATH=="*video*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
</pre>

* reload udev rules: udevadm control --reload-rules
* apply new permissions: udevadm trigger --action=add
* watch udev activity: udevadm monitor -p

= Configure lightdm display manager =

* enable it
<pre>
echo lightdm | dpkg-reconfigure -fteletype lightdm
systemctl disable gdm
systemctl disable sddm
systemctl enable lightdm
</pre>

* make the MATE desktop as default
<pre>
cd ~root/git/scripts/
git pull
/bin/cp -v etc/lightdm_default_mate.conf /etc/lightdm/lightdm.conf.d/
</pre>

* enable login by NIS users
<pre>
/bin/cp -v etc/lightdm_enable_nis_login.conf /etc/lightdm/lightdm.conf.d/
</pre>

* restart lightdm
<pre>
systemctl stop gdm
systemctl restart lightdm
</pre>

= Install libpng12.so.0 =

Quartus 16 needs libpng12:

<pre>
wget http://mirrors.kernel.org/ubuntu/pool/main/libp/libpng/libpng12-0_1.2.54-1ubuntu1_amd64.deb
dpkg --install libpng12-0_1.2.54-1ubuntu1_amd64.deb
</pre>

= Install google-chrome =

<pre>
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
dpkg -i google-chrome-stable_current_amd64.deb
</pre>

confirm autoupdate is enabled, observe dl.google.com is present in the list of repositories:
<pre>
apt update
...
Get:5 https://dl.google.com/linux/chrome/deb stable/main amd64 Packages [1,094 B]
...
</pre>

FOLLOWING IS OBSOLETE:

Instructions from here:
https://www.ubuntuupdates.org/ppa/google_chrome?dist=stable

<pre>
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-tmp.list'
apt update
apt install google-chrome-stable
/bin/rm -f /etc/apt/sources.list.d/google-tmp.list
</pre>

= Install amanda client =

ONLY ONE MACHINES THAT HOST HOME DIRECTORIES

* apt install amanda-client
* edit /etc/amandahosts
<pre>
amanda.triumf.ca amanda amdump
</pre>
* check permissions on /etc/amandahosts:
<pre>
root@daq00:/var/log/amanda# ls -l /etc/amandahosts
-rw------- 1 backup backup 49 Jan 27 10:48 /etc/amandahosts
</pre>
* fix if needed: chown backup.backup /etc/amandahosts; chmod a= /etc/amandahosts; chmod u=wr /etc/amandahosts
* edit /etc/amanda-security.conf, add this line:
<pre>
runtar:gnutar_path=/usr/bin/tar
</pre>

On the amanda machine:

* in amanda disklist, use dump type "bsdtcp-comp-user-tar"
* su - amanda and run amcheck -c daily daq00
<pre>
-bash-4.1$ amcheck -c daily daq00

Amanda Backup Client Hosts Check
--------------------------------
Client check: 1 host checked in 0.092 seconds. 0 problems found.

(brought to you by Amanda 3.3.7p1.git.685ff76d)
</pre>

= Enable rc.local =

For reasons unknown, Ubuntu LTS 20.04 does not enable /etc/rc.local. Do this:

<pre>
cd ~/git/scripts
git pull
cp -n -v etc/rc.local /etc/
chmod a+rx /etc/rc.local
cp etc/rc-local.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable rc-local
systemctl start rc-local
systemctl status rc-local
</pre>

= Remove unwanted packages =

<pre>
apt remove zsys
apt remove sddm
</pre>

= Disable unwanted services =

<pre>
systemctl disable mpd
systemctl disable snapd
systemctl disable ModemManager
systemctl --global mask tracker-extract-3.service
systemctl --global mask tracker-miner-fs-3.service
systemctl daemon-reload
</pre>

= Disable sleep and suspend =

note: we see some computers randomly shutdown or go to sleep, log files indicates the "sleep" or "suspend" button was pushed by user, but no such buttons actually exist. this is the fix for this:

<pre>
systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target systemd-suspend.service systemd-hybrid-sleep.service
</pre>

= Enable crontab @reboot for MIDAS =

startup scripts have a bug - cron @reboot entries for normal users can run before autofs is ready, so if the home directory is on autofs/NFS, it cannot be accessed and the cron job fails. If MIDAS is supposed to be started by cron @reboot, it will not start (there *will* be an error message in /var/log/cron).

<pre>
mkdir /etc/systemd/system/cron.service.d
echo -e "[Unit]\nAfter=ypbind.service autofs.service\n" > /etc/systemd/system/cron.service.d/local.conf
systemctl daemon-reload
systemctl cat cron.service
</pre>

Explore the systemd dependency tree using "systemctl list-dependencies" maybe with "--all".

Visualize the exact boot sequence from previous boot: "systemd-analyze plot > xxx.svg", look at the svg file using a web browser.

Crontab entry to start midas: (install in the midas user crontab, not root crontab)

<pre>
su - midasuser
crontab -l
#@reboot /bin/bash -l -c "/home/trinat/bin/start-daq-applications"
#@reboot /bin/tcsh -c "/home/trinat/bin/start-daq-applications"
</pre>

= Install apache httpd proxy for midas and elog =

This will configure the HTTPS/SSL certificate using "certbot" and "letsencrypt" and configure an HTTPS web server using apache2.

First, configure apache2:

* execute these commands:
<pre>
apt -y install apache2
cd /etc/apache2
</pre>
* create new file conf-available/ssl-daq14.conf # use actual hostname instead of daq14
<pre>
SSLSessionCache shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout 300
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
SSLCryptoDevice builtin
</pre>
* create new file sites-available/daq14-ssl.conf # use actual hostname instead of daq14
<pre>
<IfModule mod_ssl.c>
<VirtualHost *:443>
ServerName daq14.triumf.ca
DocumentRoot /var/www/html
ErrorLog /var/log/apache2/daq14.log
SSLEngine on
# note SSLProtocol, SSLCipherSuite and some other settings are overwritten by /etc/letsencrypt/options-ssl-apache.conf
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SEED:!IDEA:!RC4
## use port specified in elogd.cfg
#ProxyPass /elog/ http://localhost:8082/ retry=1
## use mhttpd port
#ProxyPass / http://localhost:8080/ retry=1
Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"
<Location />
SSLRequireSSL
AuthType Basic
AuthName "DAQ password protected site"
Require valid-user
# create password file: touch /etc/apache2/htpasswd
# to add new user or change password: htpasswd /etc/apache2/htpasswd username
AuthUserFile /etc/apache2/htpasswd
</Location>
</VirtualHost>
</IfModule>
</pre>
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* enable ssl module
* enable new configurations
<pre>
a2enmod ssl
a2enmod headers
a2enmod proxy
a2enmod proxy_http
a2enconf ssl-daq14
a2ensite daq14-ssl
</pre>
* disable default ssl sites
<pre>
a2dissite 000-default-le-ssl
a2dissite 000-default
ls -l /etc/apache2/sites-enabled/ ### should show only daq14-ssl.conf
</pre>
* check that there are no syntax problems
<pre>
apache2ctl configtest
</pre>
* enable and start apache2:
<pre>
systemctl enable apache2
systemctl restart apache2
systemctl status apache2
</pre>
* apache2 may fail to start, look in /var/log/apache2/error.log and /var/log/apache2/daq14.log
* if it says "Failed to configure ... certificate", proceed to the step for setting certbot.
* try to access https://daq14.triumf.ca
** you should see a complaint about self-signed certificate
** you should see a request for password (do not login yet)
** if you get "connection refused", HTTPS port 443 may need to be enabled in the local firewall, look at documentation for ufw.
Second, configure certbot:

(Note: as of 2018-01-18 certbot requires use of http port 80 to get the initial https certificate,
renewal can continue to use the https port 443)

(Note: as of 2019-01-?? certbot requires use of port 80 for renewals)

* check that port 80 is not used by anything:
* netstat -an | grep LISTEN | grep ^tcp | grep 80
* lsof -P | grep -i tcp | grep LISTEN | grep 80
* if lsof reports that apache2 is listening on port 80, follow the apache2 instructions above (remove "listen 80" from apache2.conf

* install certbot (if necessary open tcp port 80 in the firewall, see documentation for ufw):
<pre>
apt install certbot python3-certbot-apache
certbot certonly --standalone --installer apache
</pre>
* then answer questions:
* "activate HTTPS for daq14.triumf.ca" - say ok
* "enter email address" - enter your own email address
* "please read terms..." - read the terms and say "agree"
* it will take a few moments...
* "congratulations..." - say ok.
<pre>
certbot install --apache --cert-name daq14.triumf.ca
</pre>
* then answer questions:
* "choose redirect..." - say "1" (no redirect)
* look inside /etc/apache2/sites-enabled/daq14-ssl.conf to see that SSLCertificateFile & co point to certbot certificates in
/etc/letsencrypt/live/daq14.triumf.ca/
* to check current renewal and to update the certbot config file in /etc/letsencrypt/renewal, run this:
<pre>
certbot renew --standalone --installer apache --force-renewal
</pre>

NOTE: this certificate will expire in 3 months, automatic renewal should work with current version of certbot

Third, activate password protection:

* as shown in the config file above, create password file and initial user: (replace "midas" with specific username)
<pre>
touch /etc/apache2/htpasswd
htpasswd /etc/apache2/htpasswd midas
</pre>

* restart apache2
<pre>
systemctl restart apache2
systemctl status apache2
</pre>

From here:
* enable proxy for MIDAS mhttpd - uncomment redirect in the config file above
* enable proxy for ELOG - ditto
<pre>
a2enmod proxy
a2enmod proxy_http
apache2ctl configtest
systemctl restart apache2
</pre>

From here:
* enable proxy for MIDAS mhttpd - uncomment redirect in the config file above
* enable proxy for ELOG - ditto
<pre>
a2enmod proxy
a2enmod proxy_http
apache2ctl configtest
systemctl restart apache2
</pre>
* try accessing MIDAS https://daq14.triumf.ca/ (make sure mhttpd is running)
* if it's not working, check odb setting FIXME!
* try accessing ELog https://daq14.triumf.ca/elog/ (make sure elogd is running)
* if it's not working, check elogd.cfg file and make sure
<pre>
SSL = 0
</pre>

NOTE: if certbot fails with errors about 'module' object has no attribute 'pyopenssl',
try this: pip install requests==2.6.0

= Enable elog PDF preview =

see https://stackoverflow.com/questions/52998331/imagemagick-security-policy-pdf-blocking-conversion

* xemacs -nw /etc/ImageMagick-6/policy.xml
* remove this section at the end:
<pre>

<policy domain="coder" rights="none" pattern="PS" />
<policy domain="coder" rights="none" pattern="PS2" />
<policy domain="coder" rights="none" pattern="PS3" />
<policy domain="coder" rights="none" pattern="EPS" />
<policy domain="coder" rights="none" pattern="PDF" />
<policy domain="coder" rights="none" pattern="XPS" />
</pre>

= Install Jupyter notebook =

<pre>
From https://jupyter.org/install
apt install python3-pip
pip install jupyterlab
pip install notebook
~/.local/bin/jupyter notebook
watch the http://localhost:8888 URL that it printed
say "no" to offer to start firefox (it will not work!)
URL is: http://localhost:8888/tree?token=xxx
from the machine where you are running the web browser (i.e. google-chrome), run (replace trinat@trinatdaq with the username and machine name where you started jupyter)
open a new shell and run: ssh -v trinat@trinatdaq -L 8888:localhost:8888
in the web browser, open http://localhost:8888
this gives us the login page
in the password or token entry field, put the token from the "tree?token=xxx" above (printed by jupyter on startup)
push button "login"
jupyter page should open with the list of files in the trinat home directory
congratulate Brian with full success
</pre>

= Install ZFS quota report =

If there are any ZFS volumes, install script to report disk and quota usage

<pre>
cd ~/git/scripts/quotareport
git pull
mkdir /var/www/html/zfsquotareport
cp -pv ~/git/scripts/quotareport/sorttable.js /var/www/html/zfsquotareport/
ln -s $PWD/zfsquotareport.perl /etc/cron.daily/
touch /etc/crontab
</pre>

If httpd is configured to redirect "/" to MIDAS mhttpd:
* add following to /etc/apache2/sites-enabled/xxx-ssl.conf in front of "ProxyPass / ..."
* run "systemctl reload apache2"
<pre>
## do not proxy zfs quota report directory
ProxyPass /zfsquotareport/ !
</pre>

= Install PHP =

* apt install php libapache2-mod-php
* systemctl restart apache2
* create /var/www/html/info.php
<pre>
<?php

phpinfo();
</pre>
* open https://daq00.triumf.ca/info.php

= Configure TRIUMF printers =

<pre>
systemctl stop cups
systemctl disable cups
echo "ServerName printers.triumf.ca" > /etc/cups/client.conf
lpstat -a
</pre>

= Enable core dumps =

By default, Ubuntu LTS 20.04 installs the apport package
which disabled core dumps from user applications. (google it up!).
It is not meant to do this and documentation claims that
it is not installed and not enabled by default. Oh, well...

<pre>
apt remove apport
apt autoremove ### will remove apport-symptoms and a few other packages
</pre>

After this, core dumps are written to file "core" in the current directory.
See /proc/sys/kernel/core_pattern and /proc/sys/kernel/core_uses_pid.

= Enable debugger =

By default, Ubuntu LTS 20.04 does not permit debugger to attach and debug
already running programs. To enable it, add following to /etc/rc.local

<pre>
echo 0 > /proc/sys/kernel/yama/ptrace_scope
</pre>

= Disable Ubuntu Pro nag =

If "apt upgrade" requests Ubuntu Pro or esm-apps, disable the nag:
<pre>
/bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf
</pre>

= Update packages =

* apt-get update # update package list
* apt-get dist-upgrade # install updated packages and update "kept back" packages
* apt-get autoremove # remove packages that apt thinks should be removed

= Finish installation =

Congratulations. There is nothing more to do!

* reboot
<pre>
shutdown -r now
</pre>

= Install ZFS =

!!! after installing all the packages, after updating the system, after updating the linux kernel, after rebooting into latest kernel !!!

<pre>
apt-get install zfsutils-linux
</pre>

Follow generic ZFS instructions: [[ZFS]]

= Update to new version of Ubuntu =

<pre>
vi /etc/update-manager/release-upgrades # set "Prompt=normal"
do-release-upgrade
</pre>

Update Ubuntu LTS 20.04 to LTS 22.04:

== daqubuntu ==

<pre>
# reboot to clear out all updates
# vi /etc/update-manager/release-upgrades # set "Prompt=normal"
# do-release-upgrade -c
Checking for a new Ubuntu release
New release '22.04 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
# do-release-upgrade
...
say yes...
...
login.defs, say "Y" (erase local changes, use packaged version)
/etc/systemd/resolved.conf, say "Y" (same as above)
firefox snap, say yes
unable to reach snap store, say "skip"
/etc/gmond.conf, say "Y"
/var/yp/Makefile, say "install the package maintainer's version"
/etc/ypserv.conf, same thing
/etc/ypserv.securenets, same thing
/etc/default/nis, same thing
/etc/speech-dispatcher/modules/mary-generic.conf, same thing
/etc/apt/apt.conf.d/50unattended-upgrades, same thing
...
278 packages are going to be removed, say yes
...
restart required, say yes
...
no ping... yes ping...
...
ssh daqubuntu, ok
apt update, fail, DNS does not work, "host security.ubuntu.com" does not resolve.
fix resolver per https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Disable_NetworkManager
apt update, apt upgrade now works, 0 packages to update
NIS does not work.
</pre>

== midm9a ==

<pre>
login.defs
firefox snap
gmond.conf
ypserv
/etc/default/nis
unattended-upgrades
amanda-security.conf
remove obsolete (no)
reboot
configure dns
reenable nis
</pre>

== daq17 ==

<pre>
firefox snap
imagemagick policy.xml
gmond.conf
chrony.conf
/var/yp/Makefile
ypserv.conf
ypserv.securenets
/etc/default/nis
50unattended-upgrades
</pre>

== daq00 ==

per https://serverpilot.io/docs/how-to-upgrade-ubuntu-20.04-to-22.04/

<pre>
do-release-upgrade -f DistUpgradeViewNonInteractive
</pre>

if it exists "too soon" without doing anything, run it without "-f xxx", most likely it does not like something about this machine. in case of daq00 it did not like how the EFI partitions were mounted. after fixing it, non-interactive upgrade was successful.

= Ubuntu package manager =

* apt-get install xxx # install package xxx
* apt-get update
* apt-get upgrade
* apt-get dist-upgrade
* apt-get autoremove # remove automatically installed packages required by a removed package
* apt-get remove xxx # remove package xxx
* apt-cache search . # list all available packages
* apt-cache show "." | grep ^Package # list al available packages
* apt-cache madison root-system # show all available versions of package root-system
* apt list # list all installed packages
* dpkg --listfiles libpng16-16 # list all files from this package
* apt list --installed # list all installed packages
* dpkg -S /bin/bash # what package provides this file?
* dpkg -L bash # what files provided by this package?
* debsums -ce # show modified config files
* apt-config dump # show apt configuration

= Ubuntu zsys =

NOTE: DO NOT USE ZSYS, see https://github.com/ubuntu/zsys/issues/218 and https://github.com/ubuntu/zsys/issues/230

* manual removal of old snapshots
<pre>
zsysctl show
zsysctl state remove xy69ye -s
zsysctl state remove xy69ye
zsysctl state remove xy69ye -u wheel
</pre>
* apt remove zsys

NOTE: old zsys snapshots must be cleaned manually, "zsysctl state remove xxx --system" is broken and does not remove user data snapshots

* manages system snapshots
* documentation: https://github.com/ubuntu/zsys
* documentation: (go to next article via link "newer" at the bottom) https://didrocks.fr/2020/05/21/zfs-focus-on-ubuntu-20.04-lts-whats-new/
* ubuntu 20.04 bug, too many snapshots cause /boot to become full and updates fail. https://github.com/ubuntu/zsys/issues/155
* solution: use custom /etc/zsys.conf, limit number of snapshots to 10, see trinatdaq:/etc/zsys.conf
* zsys commands:
<pre>
update-grub # list of all snapshots, errors if some snapshots are broken
zsysctl state remove lnc0k7 --system # remove snapshot
xemacs -nw /etc/zsys.conf; zsysctl service reload; zsysctl service gc # cause gc to run with new settings in zsys.conf
zfs list -r -t snapshot -o name,used,referenced,creation bpool/BOOT # list snapshots
zsysctl show # show snapshots
</pre>

= Ubuntu cloning =

to clone a ubuntu image:

<pre>
cd /nfsroot/lxcpet
emacs -nw etc/hostname ### change hostname
emacs -nw etc/mailname ### change hostname (debian 11)
emacs -nw etc/defaultdomain ### change the NIS domainname
emacs -nw etc/yp.conf ### change the NIS server
cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
emacs -nw root/.ssh/authorized_keys ### update root ssh keys
</pre>

= Ubuntu boot loader =

== boot from ZFS ==

* use UEFI boot with syslinux, see here: https://daq.triumf.ca/DaqWiki/index.php/SLinstall#Configure_UEFI_boot
* apt install zfs-initramfs
* update-initramfs -v -u
* ZFS structure:
<pre>
root@daq00:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 147G 1.62T 96K /
rpool/ROOT 17.8G 1.62T 96K none
rpool/ROOT/ubuntu_00aaaa 17.8G 1.62T 6.22G /
</pre>
* copy OS image to rpool/ROOT/ubuntu_00aaaa
* zfs set mountpoint=/ rpool
* zfs set mountpoint=none rpool/ROOT
* zfs set mountpoint=/ rpool/ROOT/ubuntu_00aaaa
* zfs get all | grep mountpoint
<pre>
rpool mountpoint / local
rpool/ROOT mountpoint none local
rpool/ROOT/ubuntu_00aaaa mountpoint / local
</pre>
* in linux kernel command line (syslinux.cfg), set "root=" to "root=ZFS=rpool/ROOT/ubuntu_00aaaa"

== boot from ZFS mirror ==

=== setup the EFI partitions ===

* assuming /dev/sdb is already setup for EFI boot, setup /dev/sda the same way:
* partition the second boot disk same as first boot disk:
<pre>
root@grsnis01:~# gdisk -l /dev/sdb
Found valid GPT with protective MBR; using GPT.
Number Start (sector) End (sector) Size Code Name
1 2048 1050623 512.0 MiB EF00 EFI system partition
2 1050624 3907029134 1.8 TiB 8300 Linux filesystem
root@grsnis01:~#
</pre>
* mkfs.msdos /dev/sdX1
* create mount points
<pre>
mkdir /boot/efi-sda
mkdir /boot/efi-sdb
</pre>
* add to /etc/fstab
<pre>
/dev/sda1 /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
/dev/sdb1 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
</pre>
* mount -a
* df | grep boot
<pre>
root@grsnis01:~# df | grep boot
/dev/sdb1 523248 98100 425148 19% /boot/efi-sdb
/dev/sda1 523248 4 523244 1% /boot/efi-sda
</pre>
* copy boot files to new boot disk
* cd /boot/efi-sdX; rsync -av . /boot/efi-sdY
* set BIOS to boot from "UEFI Hard drive", disable legacy boot (except for booting from USB key in legacy mode)
* if using UEFI boot syslinux per these instructions, linux kernel update has to be done manually:
* run ~/git/scripts/etc/update_efi_mirror.perl, follow instructions that it prints.

=== setup zfs partitions ===

use partitions compatible with Ubuntu "install on ZFS"

* gdisk "o" to create new GPT partition table
* gdisk "n" +512M ef00 to create EFI partition
* gdisk "n" +2G 8200 to create linux swap partition (not used)
* gdisk "n" +2G BE00 to create ZFS bpool partition
* gdisk "n" xxx BF00 create ZFS rpool partition

<pre>
# gdisk -l /dev/sda
Number Start (sector) End (sector) Size Code Name
1 2048 1050623 512.0 MiB EF00 EFI System Partition
2 1050624 5244927 2.0 GiB 8200
3 5244928 9439231 2.0 GiB BE00
4 9439232 234441614 107.3 GiB BF00
root@midm9a:~#
</pre>

=== setup zfs mirror ===

* see here: https://daq.triumf.ca/DaqWiki/index.php/ZFS#Convert_pool_from_single_to_mirror
<pre>
root@grsnis01:~# ls -l /dev/disk/by-id/ata*part2
lrwxrwxrwx 1 root root 10 Feb 19 16:47 /dev/disk/by-id/ata-WDC_WDS200T2B0A-00SM50_205007801081-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Feb 19 16:47 /dev/disk/by-id/ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 -> ../../sdb2

root@grsnis01:~# zpool status
pool: rpool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 ONLINE 0 0 0

errors: No known data errors

root@grsnis01:~# zpool attach rpool ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 /dev/disk/by-id/ata-WDC_WDS200T2B0A-00SM50_205007801081-part2

root@grsnis01:~# zpool status
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Feb 19 16:54:39 2021
12.6G scanned at 3.16G/s, 1.02G issued at 262M/s, 12.6G total
1.02G resilvered, 8.09% done, 0 days 00:00:45 to go
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801081-part2 ONLINE 0 0 0 (resilvering)

errors: No known data errors
</pre>
* wait
<pre>
root@grsnis01:~# zpool status
pool: rpool
state: ONLINE
scan: resilvered 12.7G in 0 days 00:00:40 with 0 errors on Fri Feb 19 16:55:19 2021
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801081-part2 ONLINE 0 0 0

errors: No known data errors
</pre>

== maintenance commands ==

* update-initramfs -v -u
* grub-install /dev/sda

= Convert from single to dual mirrored ZFS SSD =

Assuming Ubuntu LTS 22.04 with "instal on ZFS" option, we will
add a second SSD, configure ZFS to use both SSDs in mirrored
configuration and setup grub to boot from either SSD. This
is intended to create a full redundant system where failure
of either SSD does not break the system.

* identify first SSD
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* connect second SSD of identical size
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
/dev/sdb WD Blue SA510 2.5 250GB 22243Z803852 25 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* if second SSD is not autodetected, reboot
* Clone partition table automatically
If both SSDs are identical size, use this simpler method of duplicating the partition table:
<pre>
root@midm9b:~# sfdisk -d /dev/sda > part_table
root@midm9b:~# grep -v ^label-id part_table | sed -e 's/, *uuid=[0-9A-F-]*//' | sfdisk /dev/sdb
</pre>
The grep and sed in the second command are there to prevent disk ID and partition IDs from being cloned. Alternatively the part_table file can be edited manually to remove the label-id line and the uuid entries from the individual partitions.

* Clone partition table manually (e.g. for different size disks)
* list partition table of first SSD:
<pre>
root@midm9b:~# fdisk -l /dev/sda
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>
* create identical partitions on second SSD, use sector numbers from above.
<pre>
root@midm9b:~# gdisk /dev/sdb
GPT fdisk (gdisk) version 1.0.8

Partition table scan:
MBR: not present
BSD: not present
APM: not present
GPT: not present

Creating new GPT entries in memory.

Command (? for help): n
Partition number (1-128, default 1):
First sector (34-488397134, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-488397134, default = 488397134) or {+-}size{KMGTP}: 1050623
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): ef00
Changed type of partition to 'EFI system partition'

Command (? for help): n
Partition number (2-128, default 2):
First sector (34-488397134, default = 1050624) or {+-}size{KMGTP}:
Last sector (1050624-488397134, default = 488397134) or {+-}size{KMGTP}: 5244927
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 8200
Changed type of partition to 'Linux swap'

Command (? for help): n
Partition number (3-128, default 3):
First sector (34-488397134, default = 5244928) or {+-}size{KMGTP}:
Last sector (5244928-488397134, default = 488397134) or {+-}size{KMGTP}: 9439231
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): be00
Changed type of partition to 'Solaris boot'

Command (? for help): n
Partition number (4-128, default 4):
First sector (34-488397134, default = 9439232) or {+-}size{KMGTP}:
Last sector (9439232-488397134, default = 488397134) or {+-}size{KMGTP}:
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): bf00
Changed type of partition to 'Solaris root'

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sdb.
The operation has completed successfully.
root@midm9b:~# fdisk -l /dev/sda /dev/sdb
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root

Disk /dev/sdb: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: EB251739-30C6-422F-A505-5887B5A0B603

Device Start End Sectors Size Type
/dev/sdb1 2048 1050623 1048576 512M EFI System
/dev/sdb2 1050624 5244927 4194304 2G Linux swap
/dev/sdb3 5244928 9439231 4194304 2G Solaris boot
/dev/sdb4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>
* identify second SSD partitions
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part3
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part3 -> ../../sda3
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 -> ../../sdb3
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
</pre>
* convert bpool from single disk to mirrored disk:
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~# zpool attach bpool 99e03dc0-7d4d-f24b-8fa1-f042b9f135db /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3
root@midm9b:~# zpool status bpool
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors
</pre>
* convert rpool
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
root@midm9b:~# zpool attach rpool f6fd54f8-3af7-b943-ae3d-a4e480537fb9 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4
root@midm9b:~# zpool status rpool
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Jan 20 19:40:45 2023
5.83G scanned at 664M/s, 2.92M issued at 332K/s, 9.11G total
0B resilvered, 0.03% done, no estimated completion time
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~#
</pre>
* wait for resilver to complete
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
scan: resilvered 9.65G in 00:00:36 with 0 errors on Fri Jan 20 19:41:21 2023
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
</pre>
* enable booting from second SSD: (instead of /dev/sda1, /dev/sdb1, use UUID=xxx)
<pre>
root@midm9b:~# mkfs.msdos /dev/sdb1
root@midm9b:~# mkdir /boot/efi-sda
root@midm9b:~# mkdir /boot/efi-sdb
root@midm9b:~# echo "/dev/sda1 /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1" >> /etc/fstab
root@midm9b:~# echo "/dev/sdb1 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1" >> /etc/fstab
root@midm9b:~# mount -a
root@midm9b:~# df -kl
Filesystem 1K-blocks Used Available Use% Mounted on
...
/dev/sda1 523244 13720 509524 3% /boot/efi
/dev/sdb1 523244 4 523240 1% /boot/efi-sdb
...
root@midm9b:~# rsync -av /boot/efi/ /boot/efi-sdb/
sending incremental file list
EFI/
...
root@midm9b:~# ls -l /boot/efi-sda
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~# ls -l /boot/efi-sdb
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~#
</pre>
* setup script to update grub on second SSD, it must be run manually after every kernel update
<pre>
root@midm9b:~# ln -s ~/git/scripts/etc/update_efi_grub.perl ~/
root@midm9b:~# ~/update_efi_grub.perl -u
EFI dir: /boot/efi-sda
/boot/efi-sda: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sda/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sda: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sda/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
EFI dir: /boot/efi-sdb
/boot/efi-sdb: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sdb/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sdb: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sdb/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
root@midm9b:~#
</pre>

= Disable NetworkManager =

NOTE: THIS IS BROKEN IN UBUNTU LTS 22.04

NetworkManager is useful for configuring dynamic
network interfaces, i.e. laptops that often move
between networks, or connect to multiple choice
of wifi networks, etc.

For machines with statically configured network interfaces,
NetworkManager is not necessary.

As it has been observed to become confused and observed
to malfunction when network links go up and down (it keeps
unnecessarily reconfiguring the ip address, etc), it can
be usefuil to disable it.

* list all network interfaces
<pre>
# /bin/ls -1 /sys/class/net/
enp0s31f6
lo
</pre>
* edit /etc/network/interfaces:
<pre>
rename enp0s31f6=eth0
auto eth0
iface eth0 inet static
address 142.90.120.94/19
gateway 142.90.100.18
</pre>
* statically configure systemd-resolved
** create /etc/systemd/resolved.conf.d/resolved.conf with this contents:
<pre>
[Resolve]
DNS=142.90.100.19
Domains=triumf.ca
</pre>
** systemctl restart systemd-resolved
** resolvectl
** systemd-analyze cat-config systemd/resolved.conf
* disable NetworkManager
<pre>
systemctl disable NetworkManager
</pre>
* reboot

= Configure ECC memory =

== Configure EDAC ==

* apt install edac-utils

=== Intel i3-2120 ===
<pre>
root@musr00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X9SCL/X9SCM
root@musr00:~# edac-ctl --status
edac-ctl: drivers not loaded.
</pre>

=== Intel E-2236 ===
<pre>
root@daq00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SCM-F
root@daq00:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq00:~# edac-util
edac-util: No errors to report.
root@daq00:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
</pre>
* check edac sysfs files (Intel)
<pre>
root@daq00:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_noinfo_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 max_location
-r--r--r-- 1 root root 4096 Jan 25 15:10 mc_name
drwxr-xr-x 2 root root 0 Jan 25 15:10 power
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank0
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank1
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank2
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank3
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank4
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank5
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank6
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank7
--w------- 1 root root 4096 Jan 25 15:10 reset_counters
-r--r--r-- 1 root root 4096 Jan 25 15:10 seconds_since_reset
-r--r--r-- 1 root root 4096 Jan 25 15:10 size_mb
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Jan 25 15:10 uevent
root@daq00:~#
</pre>

=== Intel E3-1270 v6 ===
<pre>
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --status
edac-ctl: drivers are loaded.
root@grsnis01:~# edac-util
edac-util: No errors to report.
root@grsnis01:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@grsnis01:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 max_location
-r--r--r-- 1 root root 4096 Feb 19 12:35 mc_name
drwxr-xr-x 2 root root 0 Feb 19 12:35 power
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank0
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank1
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank2
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank3
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank4
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank5
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank6
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank7
--w------- 1 root root 4096 Feb 19 12:35 reset_counters
-r--r--r-- 1 root root 4096 Feb 19 12:35 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 19 12:35 size_mb
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 19 12:35 uevent
root@grsnis01:~#
</pre>

=== Intel E3-1245 v6 ===

<pre>
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --status
edac-ctl: drivers are loaded.
[root@alphagdaq ~]# edac-util
edac-util: No errors to report.
[root@alphagdaq ~]# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
[root@alphagdaq ~]# ras-mc-ctl --layout
+-----------------------------------------------+
| mc0 |
| csrow0 | csrow1 | csrow2 | csrow3 |
----------+-----------------------------------------------+
channel1: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
----------+-----------------------------------------------+
[root@alphagdaq ~]# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#3channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#3channel#1 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#2channel#0 0 0
[root@alphagdaq ~]# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SSH-F
[root@alphagdaq ~]# ras-mc-ctl --summary
DBD::SQLite::db prepare failed: no such table: mc_event at /usr/sbin/ras-mc-ctl line 1129.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1130.
[root@alphagdaq ~]#
</pre>

=== AMD 3700X ===

(memory is non-ECC)

<pre>
root@daq13:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@daq13:~#
root@daq13:~#
root@daq13:~# edac-ctl --status
edac-ctl: drivers not loaded.
root@daq13:~# edac-util
edac-util: Error: No memory controller data found.
root@daq13:~# edac-util -s
edac-util: EDAC drivers loaded. No memory controllers found
root@daq13:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 2 root root 0 Jan 25 15:26 power
lrwxrwxrwx 1 root root 0 Jan 21 16:16 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Jan 21 16:16 uevent
</pre>

(memory is ECC)

<pre>
root@trinatdaq:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@trinatdaq:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@trinatdaq:~# edac-util
edac-util: No errors to report.
root@trinatdaq:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Dec 15 13:04 mc0
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
lrwxrwxrwx 1 root root 0 Dec 13 18:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dec 13 18:31 uevent
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_noinfo_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 max_location
-r--r--r-- 1 root root 4096 Dec 15 13:04 mc_name
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank4
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank5
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank6
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank7
--w------- 1 root root 4096 Dec 15 13:04 reset_counters
-rw-r--r-- 1 root root 4096 Dec 15 13:04 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Dec 15 13:04 seconds_since_reset
-r--r--r-- 1 root root 4096 Dec 15 13:04 size_mb
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Dec 15 13:04 uevent
root@trinatdaq:~#
</pre>

=== AMD 5000G ===

* no linux driver for AMD 5000-series "G" CPU
* no mention of ECC in the BIOS settings
* unclear status of ECC support in AMD documentation (sais only "pro" "G" CPUs have ECC)
* unclear status of ECC support in ASUS documentation (web page out of date)

=== AMD 5600X ===

<pre>
root@daq17:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-XE GAMING WIFI
root@daq17:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq17:~# edac-util
edac-util: No errors to report.
root@daq17:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@daq17:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Aug 19 19:27 mc0
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
lrwxrwxrwx 1 root root 0 May 10 10:11 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 May 10 10:11 uevent
root@daq17:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_noinfo_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 max_location
-r--r--r-- 1 root root 4096 Aug 19 19:27 mc_name
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank4
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank5
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank6
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank7
--w------- 1 root root 4096 Aug 19 19:27 reset_counters
-rw-r--r-- 1 root root 4096 Aug 19 19:27 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Aug 19 19:27 seconds_since_reset
-r--r--r-- 1 root root 4096 Aug 19 19:27 size_mb
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Aug 19 19:27 uevent
root@daq17:~#
</pre>

=== AMD 3955WX ===

<pre>
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --status
edac-ctl: drivers are loaded.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util
edac-util: No errors to report.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@alphasuperdaq:~/git/scripts/quotareport# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 19 root root 0 Dez 12 04:48 mc0
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
lrwxrwxrwx 1 root root 0 Dez 9 05:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dez 9 05:31 uevent
root@alphasuperdaq:~/git/scripts/quotareport#
root@alphasuperdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 max_location
-r--r--r-- 1 root root 4096 Feb 28 22:19 mc_name
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank0
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank1
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank10
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank11
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank12
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank13
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank14
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank15
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank2
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank3
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank4
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank5
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank6
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank7
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank8
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank9
--w------- 1 root root 4096 Feb 28 22:19 reset_counters
-rw-r--r-- 1 root root 4096 Feb 28 22:19 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Feb 28 22:19 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 28 22:19 size_mb
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 28 22:19 uevent
root@alphasuperdaq:~#
root@alphasuperdaq:~# ras-mc-ctl --layout
Use of uninitialized value $max_pos[3] in modulus (%) at /usr/sbin/ras-mc-ctl line 868.
Use of uninitialized value $d in numeric ge (>=) at /usr/sbin/ras-mc-ctl line 869.
Use of uninitialized value $d in sprintf at /usr/sbin/ras-mc-ctl line 872.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| mc0 |
| csrow0 | csrow1 |
| channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 | channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

0: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
root@alphasuperdaq:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#2 0 0
mc#0csrow#1channel#7 0 0
mc#0csrow#0channel#3 0 0
mc#0csrow#1channel#4 0 0
mc#0csrow#1channel#2 0 0
mc#0csrow#0channel#7 0 0
mc#0csrow#1channel#3 0 0
mc#0csrow#0channel#4 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#5 0 0
mc#0csrow#0channel#6 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#0channel#5 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#6 0 0
root@alphasuperdaq:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: ASUSTeK COMPUTER INC. model Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@alphasuperdaq:~#
</pre>

== Configure rasdaemon ==

<pre>
apt install rasdaemon
</pre>
<pre>
systemctl enable rasdaemon
systemctl restart rasdaemon
systemctl status rasdaemon
</pre>

<pre>
● rasdaemon.service - RAS daemon to log the RAS events
Loaded: loaded (/lib/systemd/system/rasdaemon.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-01-25 15:16:37 PST; 3min 5s ago
Main PID: 2477175 (rasdaemon)
Tasks: 1 (limit: 76958)
Memory: 17.1M
CGroup: /system.slice/rasdaemon.service
└─2477175 /usr/sbin/rasdaemon -f -r

Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Listening to events for cpus 0 to 11
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mc_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording aer_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording extlog_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mce_record events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording arm_event events
</pre>

== Get reports ==

* Intel 2x32GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --layout
+-------------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-------------------------+
channel1: | 16384 MB | 16384 MB |
channel0: | 16384 MB | 16384 MB |
----------+-------------------------+
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#0channel#1 0 0
root@daq00:~#
</pre>

* Intel 4x16GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#1 0 0
mc#0csrow#2channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#3channel#0 0 0
mc#0csrow#3channel#1 0 0
root@daq00:~#
root@daq00:~# ras-mc-ctl --layout
+-----------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-----------------------+
channel1: | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB |
----------+-----------------------+
root@daq00:~#
root@daq00:~#
root@daq00:~#
root@daq00:~# ras-mc-ctl --print-labels
ras-mc-ctl: Error: No dimm labels for Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@daq00:~#
</pre>

note: ubuntu LTS 22.04 DBD::SQLite::db error is not there.

= sensors =

== ASUS P9X79 WS ==

* https://www.asus.com/supportonly/P9X79%20WS/HelpDesk_Manual/
* BIOS version 4802
* modprobe nct6775
* modprobe coretemp

<pre>
root@daq14:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 0: +29.0°C (high = +82.0°C, crit = +100.0°C)
Core 1: +24.0°C (high = +82.0°C, crit = +100.0°C)
Core 2: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +82.0°C, crit = +100.0°C)

nouveau-pci-0200
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.00 V)
temp1: +39.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

nct6776-isa-0290
Adapter: ISA adapter
Vcore: 1.04 V (min = +0.00 V, max = +1.74 V)
in1: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
AVCC: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
+3.3V: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 2.04 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 904.00 mV (min = +0.00 V, max = +0.00 V) ALARM
3VSB: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
Vbat: 3.30 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1265 RPM (min = 0 RPM)
fan2: 1909 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +34.0°C (high = +0.0°C, hyst = +0.0°C) ALARM sensor = thermistor
CPUTIN: +58.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermal diode
AUXTIN: +31.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
PECI Agent 0: +31.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +96.0°C)
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

root@daq14:~#
</pre>

= Enable CPU turbo mode =

* Intel CPU has a nominal CPU frequency (i.e. 3.4GHz) and a turbo-boost CPU frequency (i.e. 4.0GHz). Here we will enable this turbo-boost mode.
* Find out CPU capability
<pre>
root@daq01:~# lscpu | grep Hz
Model name: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
CPU MHz: 3965.803
CPU max MHz: 4000.0000
CPU min MHz: 800.0000
root@daq01:~#
</pre>
* Look up this CPU in the Intel ARK database - google for the CPU model name, i.e.
https://ark.intel.com/content/www/us/en/ark/products/88196/intel-core-i7-6700-processor-8m-cache-up-to-4-00-ghz.html
* Find current frequency settings:
<pre>
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "powersave" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 2.72 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
root@daq01:~#
</pre>
* Note the following:
** current governor is "powersave"
** "performance" governor is available
** "boost state support" is supported and active.
* Confirm CPU frequency governor:
<pre>
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
powersave
powersave
powersave
powersave
powersave
powersave
powersave
powersave
root@daq01:~#
</pre>
* Change governor to "performance":
<pre>
root@daq01:~# cpupower frequency-set --governor performance
Setting cpu: 0
Setting cpu: 1
Setting cpu: 2
Setting cpu: 3
Setting cpu: 4
Setting cpu: 5
Setting cpu: 6
Setting cpu: 7
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
performance
performance
performance
performance
performance
performance
performance
performance
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "performance" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 3.93 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
</pre>
* monitor CPU frequency:
<pre>
root@daq01:~# cpupower monitor
| Nehalem || Mperf || Idle_Stats
CPU| C3 | C6 | PC3 | PC6 || C0 | Cx | Freq || POLL | C1 | C1E | C3 | C6 | C7s | C8
0| 0.00| 0.00| 0.00| 0.00|| 88.80| 11.20| 3973|| 0.00| 0.00| 0.01| 0.02| 0.31| 0.00| 4.25
4| 0.00| 0.00| 0.00| 0.00|| 4.70| 95.30| 3945|| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 95.03
1| 0.73| 3.70| 0.00| 0.00|| 4.52| 95.48| 3864|| 0.00| 0.01| 1.19| 0.44| 2.82| 0.00| 90.23
5| 0.73| 3.70| 0.00| 0.00|| 0.37| 99.63| 3807|| 0.00| 0.00| 0.03| 0.09| 1.70| 0.00| 97.64
2| 2.28| 12.86| 0.00| 0.00|| 1.41| 98.59| 3829|| 0.00| 0.86| 3.17| 0.46| 7.70| 0.00| 85.87
6| 2.28| 12.86| 0.00| 0.00|| 2.88| 97.12| 3856|| 0.00| 0.11| 4.56| 2.15| 10.31| 0.00| 78.99
3| 1.33| 4.81| 0.00| 0.00|| 0.99| 99.01| 3804|| 0.00| 0.49| 0.79| 0.01| 1.03| 0.00| 96.12
7| 1.34| 4.81| 0.00| 0.00|| 1.26| 98.74| 3818|| 0.00| 0.01| 2.32| 0.47| 5.02| 0.00| 90.06
root@daq01:~#
</pre>
* check that the CPU is not overheating:
<pre>
root@daq01:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 1: +38.0°C (high = +84.0°C, crit = +100.0°C)
Core 2: +34.0°C (high = +84.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +84.0°C, crit = +100.0°C)
</pre>
* congratulations, we are running at 4 GHz now!

= Setup ubuntu as gateway to private network =

See also:
* https://daq.triumf.ca/DaqWiki/index.php/VME-CPU#Setup_the_boot_host_computer_.28el7.29
* http://www.triumf.info/wiki/DAQwiki/index.php/Dhcpd_on_eth1

== Steps to do ==

* assign network numbers to the private network, i.e. 192.168.1.x, 192.168.2.x, etc
* (on the gateway machine, each private network interface has to have a different network number)
* (each network interface can have multiple networks attached, via VLANs or via eth0:0, eth0:1 constructs)
* assign IP addresses on the private network, save them in /etc/hosts i.e. "hvps 192.168.1.10"
* (for simplicity, assign 192.168.1.1 to the gateway machine itself)
* (IP addresses 192.168.1.0 and 192.168.1.255 are "special", do not use them)
* setup DNS server (dnsmasq) to serve contents of /etc/hosts via DNS (otherwise, many programs will see inconsistent name to IP address mapping)
* setup DHCP server (ISC dhcpd or dnsmasq) to give out the IP addresses
* setup tftp, pxelinux and NFS for diskless booting
* setup time server (chronyd) to provide common time to all devices
* setup NAT so machines on private network can access the internet (to get OS updates, etc)
* setup NIS and NFS so machines on the private network can use common home directories
* setup rsync backup of machines on the private network

== setup hosts ==

* edit /etc/hosts
<pre>
192.168.1.101 dsfe01
... and so forth
</pre>

== setup dns and dhcp ==

* apt install dnsmasq
* edit /etc/dnsmasq.conf
<pre>
# /etc/dnsmasq.conf
# DNS settings
#port=0 # disable DNS function
port=53 # enable DNS function
domain-needed
bogus-priv
no-resolv
server=142.90.100.19
# DHCP settings
interface=enp1s0f0 # DHCP interface
#dhcp-range=192.168.1.50,192.168.1.150,infinite
dhcp-range=192.168.1.0,static
#log-dhcp
quiet-dhcp
#dhcp-ignore=tag:!known
dhcp-boot=pxelinux.0
#dhcp-host=ac:1f:6b:9e:7f:4a,192.168.1.100,10m
dhcp-host=ac:1f:6b:9e:7f:4a,dsfe01,infinite
# TFTP settings
enable-tftp
tftp-root=/tftpboot
</pre>
* #mkdir /zssd/tftpboot ### per tftp-root (if no ZFS)
* zfs create -o mountpoint=/tftpboot rpool/tftpboot ### (if root is ZFS)
* systemctl stop systemd-resolved.service
* systemctl disable systemd-resolved.service
* rm /etc/resolv.conf
* create new /etc/resolv.conf with this contents:
<pre>
nameserver 127.0.0.1
search snolab.ca
</pre>
* systemctl enable dnsmasq
* systemctl restart dnsmasq

== setup chronyd ==

* enable ntp server:
* configure and enable chronyd per instructions above
* echo "allow 192.168.1.0/24" > /etc/chrony/conf.d/allow-localhost.conf
* systemctl restart chronyd
* chronyc tracking ### wait until time is synchronized (a few seconds)

== setup diskless network booting ==

=== setup pxelinux ===
<pre>
cd ~
wget https://www.kernel.org/pub/linux/utils/boot/syslinux/4.xx/syslinux-4.03.tar.bz2
tar xjvf syslinux-4.03.tar.bz2
cd syslinux-4.03
cp -pv ./core/pxelinux.0 ./com32/hdt/hdt.c32 ./memdisk/memdisk ./com32/menu/menu.c32 /zssd/tftpboot/
</pre>
* cd /zssd/tftpboot
<pre>
wget http://ladd00.triumf.ca/tftpboot/memtest86+-4.20.iso.zip
wget http://ladd00.triumf.ca/tftpboot/memtest86+-5.01.iso.gz
wget http://ladd00.triumf.ca/tftpboot/modules.alias
wget http://ladd00.triumf.ca/tftpboot/modules.pcimap
wget http://ladd00.triumf.ca/tftpboot/pci.ids
</pre>
* mkdir pxelinux.cfg
* emacs -nw pxelinux.cfg/default
<pre>
default menu.c32
prompt 0

menu title Welcome to the DSVSLICE PXE boot menu

timeout 50

label hdt
kernel hdt.c32

label memtest86+-5.01
kernel memdisk iso initrd=memtest86+-5.01.iso.gz

label memtest86+-4.20
kernel memdisk iso initrd=memtest86+-4.20.iso.zip

label vmlinuz-5.3.0-26-generic
menu default
kernel vmlinuz-5.3.0-26-generic
append initrd=initrd.img-5.3.0-26-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.1.1:/zssd/nfsroot/dsfe01 toram ip=dhcp panic=60 BOOTIF=enp1s0f0

#end
</pre>

=== setup linux kernel ===

* copy the kernel files
<pre>
cd /boot
rsync -av config* initrd* System.map* vmlinuz* /zssd/tftpboot/
</pre>
* cd /zssd/tftpboot
* chmod a+r *

=== setup nfs ===

* apt-get install nfs-kernel-server
* emacs -nw /etc/exports
<pre>
/zssd/nfsroot/dsfe01 dsfe01(rw,no_root_squash,async,no_subtree_check)
</pre>
* enable services
<pre>
systemctl enable nfs-server
systemctl enable nfs-mountd
systemctl enable nfs-idmapd
systemctl restart nfs-server
systemctl restart nfs-mountd
systemctl restart nfs-idmapd
</pre>
* after editing /etc/exports, run
<pre>
exportfs -av
</pre>

=== setup userland ===

* zfs create zssd/nfsroot
* zfs set dedup=verify zssd/nfsroot ### enable deduplication to save disk space because most linux images have mostly identical files
* clone ubuntu
<pre>
mkdir /zssd/nfsroot/dsfe01
cd /
rsync -avx . /zssd/nfsroot/dsfe01
</pre>
* edit config files:
* cd /zssd/nfsroot/dsfe01
* emacs -nw etc/hostname ### change to dsfe01
* emacs -nw etc/mailname ### change to dsfe01
* emacs -nw etc/yp.conf ### change daq00.triumf.ca to musr00.triumf.ca
* emacs -nw etc/defaultdomain ### change to MUSR-NIS
* cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
* emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
* emacs -nw root/.ssh/authorized_keys ### update root ssh keys
* emacs -nw etc/fstab ### add this
<pre>
192.168.1.1:/zssd/nfsroot/dsfe01 / nfs defaults,nolock 0 0
</pre>
* emacs -nw etc/chrony/chrony.conf
** comment-out all "pool" and "server" entries
** add entry "server 192.168.1.1 iburst"

After dsfe01 is booted:

* disable services:
<pre>
systemctl disable apache2
systemctl disable dnsmasq
systemctl disable zfs-import-cache
</pre>

To setup additional machines, clone dsfe01 instead of cloning the gateway machine

=== Allow manpages to be viewed ===

If <code>/</code> is mounted over NFS, <code>man</code> will report a permission error. Fix it with:

<pre>
ln -s /etc/apparmor.d/usr.bin.man /etc/apparmor.d/disable/
apparmor_parser -R /etc/apparmor.d/usr.bin.man
</pre>

== setup shared home directory ==

=== on the gateway machine ===
* define netgroups
* emacs -nw /etc/netgroup
<pre>
dsfe (dsfe01,,) (dsfe02,,)
</pre>
* emacs -nw /etc/nsswitch.conf ### edit the netgroup line to read:
<pre>
netgroup: files
</pre>
* export the home directories:
* emacs -nw /etc/exports ### add this:
<pre>
/zssd/home1 @dsfe(rw,no_root_squash,async,no_subtree_check)
</pre>
* exportfs -rc

=== on the frontend machine ===

* mkdir /home
* emacs -nw /etc/fstab ### add this:
<pre>
192.168.1.1:/zssd/home1 /home nfs defaults 0 0
</pre>
* mount -a

== setup NAT ==

NAT allows machines on the private network to connect to the internet: https://en.wikipedia.org/wiki/Network_address_translation

In these examples:
* replace "eno1" with name of the outgoing interface (the one connected to the TRIUMF network).
* replace "enp11s0" with name of the private network interface (192.168.1.x network)

* emacs -nw /etc/rc.local ### add this:
<pre>
# /etc/rc.local

/sbin/iptables -t nat -A POSTROUTING -o eno1 -j MASQUERADE
iptables -L -v

# uncomment following lines if machine has prohibitive FORWARD rules:
#/sbin/iptables -I FORWARD -i eno1 -o enp11s0 -m state --state RELATED,ESTABLISHED -j ACCEPT
#/sbin/iptables -I FORWARD -i enp11s0 -o eno1 -j ACCEPT
#iptables -L -v

iptables -L -v
sysctl -w net.ipv4.ip_forward=1
#sysctl -a | grep forward

sh /etc/firewall-rfc1918.sh

# end
</pre>
* emacs -nw /etc/firewall-rfc1918.sh
<pre>
# firewall-rfc1918.sh

# prevent RFC1918 private network IP addresses from
# going in and out from our uplink.

ETH=eno1

iptables -F in-rfc1918
iptables -N in-rfc1918
iptables -A in-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A in-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A in-rfc1918 --dst 192.168.0.0/16 -j REJECT
iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -I INPUT -j in-rfc1918 -i $ETH

iptables -F out-rfc1918
iptables -N out-rfc1918
iptables -A out-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A out-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A out-rfc1918 --dst 192.168.0.0/16 -j REJECT
iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -I OUTPUT -j out-rfc1918 -o $ETH

iptables -L -v

#end
</pre>

= KVM =

<pre>
apt install cpu-checker

root@daq13:~# kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
root@daq13:~#

(if not, shutdown, go into BIOS settings, enable CPU virtualization)

apt install virtinst ### will install many packages
apt install libvirt-clients libvirt-daemon-system-systemd libvirt-daemon qemu qemu-kvm libvirt-daemon-system virtinst bridge-utils

root@daq13:/home1/wheel# virsh list --all
Id Name State
------------------------------
1 ubuntu-guest running

apt install virt-manager

virt-install --name ubuntu-guest --os-variant ubuntu20.04 --vcpus 2 --ram 2048 --location /daq/daqstore/olchansk/linux/Ubuntu/ubuntu-20.04.3-desktop-amd64.iso --network bridge=virbr0,model=virtio --graphics none --extra-args='console=ttyS0,115200n8 serial'

virtual machine will start, boot, etc
to get out of it, CTRL + Shift followed by ]

ssh wheel@daq13
virt-manager

run virt-install again, omit "--graphics none", open graphics console from virt-manager, it booted into ubuntu installer desktop

virt-install --name test10 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --filesystem /kvm_ladd00,/ --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial" --graphics none

virt-install --name test14 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --disk /tmp/xxx/ladd00.img,bus=sata --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial rdshell" --graphics none --check path_in_use=off
</pre>

build image

<pre>
dd if=/dev/zero of=/tmp/xxx/ladd00.img bs=1024M count=20
mkfs.ext3 /tmp/xxx/ladd00.img ### ext4 fails to mount by SL6 kernel, "unknown ext4 options"
cd /kvm_ladd00/
mount -o loop /tmp/xxx/ladd00.img /mnt/tmp
rsync -av . /mnt/tmp/ --delete
umount /mnt/tmp
</pre>

on the guest, configure network: /etc/rc.local
<pre>
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

ifconfig eth2 192.168.122.2
route add -net 0.0.0.0 gw 192.168.122.1
ifconfig -a
netstat -rn

# end
</pre>

= ARM cross-compiler =

<pre>
apt install libgcc-9-dev-arm64-cross
apt install gcc-arm-linux-gnueabi
apt install gcc-arm-linux-gnueabihf
apt install g++-arm-linux-gnueabihf
apt install g++-arm-linux-gnueabi
</pre>

<pre>
arm-linux-gnueabi-gcc -o ttcp1 ttcp.c -march=armv7 -static
arm-linux-gnueabi-gcc -o memcpy.armv7 memcpy.cc -march=armv7 -static -O2
</pre>

= 32-bit intel cross-compiler =

Ubuntu 22.04:

<pre>
apt install libstdc++-11-dev:i386
apt install zlib1g-dev:i386
</pre>

for MIDAS, use "make linux32"

Ubuntu

2023-11-10T22:31:49Z

Lmartin: /* Convert from single to dual mirrored ZFS SSD */

= About Ubuntu =

AAA

= Ubuntu version =

<pre>
lsb_release -a
uname -a
</pre>

= Ubuntu installer =

* updated for Ububtu LTS 20.04.01, 22.04.1

* download the latest Ubuntu LTS desktop installer iso image
* dd the image to a USB key
* power down, disconnect all disks (all HDDs, all SSDs, all M.2)
* connect the SSD to be used as system disk
* if system will use mirrored SSDs (using ZFS mirror), leave second SSD disconnected, we will activate it later
* power up
* boot from USB key in legacy mode or UEFI mode (select this in the BIOS boot menu - F8 for ASUS, F11 for Supermicro)
* follow the instruction:
* "try ubuntu or install ubuntu" - choose "install"
* select language - accept default
* "updates and other software" - accept default settings ("normal install")
* "installation type" - select "advanced features" and "experimental: use ZFS"
* accept partition choice
* "where are you?" - select "Vancouver" (PST time zone)
* "who are you?" - leave all fields blank, except "username" set to "wheel", "password" set to the root password. hostname will be set later after configuring the network
* installation runs in a few minutes, when finished, reboot
* login as user wheel
* answer annouying questions:
* "livepatch" - say "next"
* "help improve" - select "do not send", say "next"
* "privacy" - leave "location" as "off", say "next"
* "ready to go", say "done"
* right-click on the desktop, say "open in terminal", a shell will open
* say "sudo /bin/bash", enter the root password, you now have the root shell
* run nm-connection-editor to configure the network. use netmask 255.255.224.0, gateway 142.90.100.18, DNS 142.90.100.19, search path "triumf.ca"
* after network is up (can ping ladd00), continue with post-installation steps below

= Install instructions =

== prepare ==
<pre>
apt update
apt upgrade
</pre>

== install ssh ==
<pre>
apt install ssh
</pre>

== configure hostname ==
<pre>
vi /etc/hostname
</pre>

== disable swap ==

ubuntu installer creates a 2 GB swap partition, not useful
on 32-64 GB machine, disable it:
<pre>
vi /etc/fstab ### comment out the "swap" line
</pre>

== maybe reboot ==

this is a good point to reboot the machine to boot
the latest kernel and to set the correct hostname

== install etckeeper ==

keep contents of /etc in a git repository:

<pre>
apt -y install etckeeper
</pre>

== set timezone ==
<pre>
timedatectl list-timezones | grep -i vancouver
timedatectl set-timezone America/Vancouver
</pre>

== install time synchronization ==
<pre>
apt -y install chrony
echo server time1.triumf.ca iburst >> /etc/chrony/chrony.conf
echo server time2.triumf.ca iburst >> /etc/chrony/chrony.conf
echo server time3.triumf.ca iburst >> /etc/chrony/chrony.conf
systemctl disable systemd-timesyncd.service
systemctl stop systemd-timesyncd.service
systemctl disable ntp
systemctl stop ntp
systemctl enable chrony
systemctl restart chrony
chronyc sources
chronyc tracking
</pre>

== reenable systemd-timesyncd ==

ONLY IF CHRONY DOES NOT WORK

<pre>
apt remove chrony
systemctl enable systemd-timesyncd.service
systemctl restart systemd-timesyncd.service
systemctl status systemd-timesyncd.service
timedatectl status
timedatectl timesync-status
</pre>

== enable outgoing email (debian 11) ==

this is different from ubuntu 20. it uses /etc/mailname and it hardwires the hostname into main.cf.

== enable outgoing email ==

* TRIUMF: use smtp.triumf.ca
* CERN: use cernmx.cern.ch

<pre>
apt install postfix ### select "satellite system", enter full hostname "xxx.triumf.ca", enter "smtp.triumf.ca"
apt install mailutils
dpkg-reconfigure postfix ### (if postfix already installed)
</pre>
<pre>
echo olchansk@triumf.ca lindner@triumf.ca bsmith@triumf.ca >> ~root/.forward
mailx root
test
^D
</pre>

== enable ping for all users (debian 11) ==

Without this tweak, Debian will report "operation not permitted" if a user tries to ping somewhere.

<pre>
echo 'net.ipv4.ping_group_range = 0 1000' > /etc/sysctl.d/99-ping.conf
</pre>

== install missing packages ==

(apt eats terminal input, even the "yes |" trick does not quite work,
repeat the following commands until they report that everything
is installed)

<pre>
yes | apt -y install ssh tcsh ethtool ncat rsync strace net-tools sysstat smartmontools lm-sensors traceroute time minicom screen git lsof debsums tmux
yes | apt -y install lsb-release
yes | apt -y install flex bison
yes | apt -y install neofetch
yes | apt -y install snmp snmp-mibs-downloader
yes | apt -y install git subversion g++ gfortran cmake doxygen
yes | apt -y install python
yes | apt -y install curl libcurl4 libcurl4-openssl-dev
yes | apt -y install mariadb-client ### mysql client
yes | apt -y install libz-dev sqlite3 libsqlite3-dev unixodbc-dev
yes | apt -y install libssl-dev
yes | apt -y install emacs xemacs21 joe
yes | apt -y install gnuplot dos2unix
yes | apt -y install mutt bsd-mailx # email clients
yes | apt -y install liblz4-tool pbzip2
yes | apt -y install libc6-dev-i386 # otherwise no /usr/include/sys/types.h
yes | apt -y install libreadline-dev
yes | apt -y install chromium-browser chromium-codecs-ffmpeg-extra
yes | apt -y install ubuntu-mate-themes
yes | apt -y install libmotif-dev libxmu-dev
yes | apt -y install libusb-dev libusb-1.0-0-dev
yes | apt -y install xfig gsfonts-x11 gsfonts-other # install fonts for xfig
yes | apt -y install libjson-perl
yes | apt -y install libgsl-dev # additional GNU Scientific Library
yes | apt -y install qt5-default # Qt development
yes | apt -y install python3.8-full python3.8-dev python3.8-dbg python3-pip ### for pyROOT
yes | apt -y install imagemagick imagemagick-common ckeditor # for elog
yes | apt -y install libjpeg-dev libjpeg-progs libjpeg-tools
yes | apt -y install linux-tools-common linux-tools-generic # cpupower frequency-info
yes | apt -y install rdesktop remmina remmina-plugin"*" # requested by POL
</pre>

Ubuntu LTS 20.04:
<pre>
yes | apt -y install linux-image-generic-hwe-20.04 linux-tools-virtual-hwe-20.04 # enable linux 5.11 series kernel
</pre>

== install git/scripts ==
<pre>
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd scripts
git pull
</pre>

== disable swap (debian 11) ==

* on 64 GB RAM machines swap is not useful
* on machines booted from network (NFS-ROOT), swap does not work
* on machines running from flash (RPi, etc), flash is too slow for useful swap
* swap configured by linux installers invariably has wrong size and is not useful

<pre>
systemctl disable dphys-swapfile
systemctl stop dphys-swapfile
dphys-swapfile uninstall
</pre>

== configure DNS ==

<pre>
cd ~/git/scripts
git pull
mkdir /etc/systemd/resolved.conf.d
cp etc/resolved-triumf.conf /etc/systemd/resolved.conf.d/
systemctl restart systemd-resolved
resolvectl
#systemd-analyze cat-config systemd/resolved.conf
</pre>

== install ganglia ==

<pre>
yes | apt-get -y install ganglia-monitor
systemctl enable ganglia-monitor
</pre>
<pre>
cd ~root/git/scripts
git pull
cp etc/gmond-ubuntu.conf /etc/ganglia/gmond.conf
ln -s ganglia/gmond.conf /etc
# ln -s arm-linux-gnueabihf/ganglia /usr/lib/ganglia ### fix path for ARM CPU
# ln -s i386-linux-gnu/ganglia /usr/lib/ganglia ### fix path for 32-bit Intel CPU
systemctl restart ganglia-monitor
systemctl status ganglia-monitor
ps -efw | grep gmond
</pre>
<pre>
cd ~root/git/scripts/ganglia
make install
./ganglia-all.perl
</pre>

== install gonodeinfo ==

* go to https://bitbucket.org/dd1/gonodeinfo follow instructions:
<pre>
yes | apt-get -y install golang
mkdir ~/git
cd ~/git
git clone https://bitbucket.org/dd1/gonodeinfo.git
cd gonodeinfo
git pull
make
make install # install gonodeinfo agent
cd ~ # this is important
</pre>
* edit /etc/gonodeinfo.conf
* change "Description", "Location", "User" and "Administrator" as appropriate (or delete them)
* change "Servers" to read: Servers: daq00.triumf.ca:8601
* run "gonodeinfo -v"
* if error is "connection refused". go to the nodeinfo server to add this client to the access control list:
* on the gonodeinfo server: run /opt/gonodeinfo/gonodereceive.exe -a daq13
* try gonodeinfo again, there should be no error
* on the gonodeinfo server: run gonodereport, look at the web pages, the new machine should be listed now

== install fonts for EPICS ==

* apt install xfonts-100dpi xfonts-75dpi
* restart Xorg (i.e. "killall Xorg", this will log you out from the console)
* xlsfonts | grep -i helvetica ### should show fonts with different sizes, not just size 0 (scalable)

== install libz.so.1 for CentOS compatibility ==

KO - confirm which versions on quartus need this.

<pre>
yes | apt-get -y install zlib1g
yes | apt-get -y install zlib1g:i386 libc6:i386 libgcc1:i386 gcc-6-base:i386
</pre>

== install libpng12.so.0 for Quartus compatibility ==

(does not work anymore!!!)

<pre>
wget http://ftp.ca.debian.org/debian/pool/main/libp/libpng/libpng12-0_1.2.50-2+deb8u2_amd64.deb
dpkg --install libpng12-0_1.2.50-2+deb8u2_amd64.deb
</pre>

== install libpng12.so.0 for Quartus 13.0sp1 ==

<pre>
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0.50.0
/bin/cp -pv libpng12.so.0 libpng12.so.0.50.0 /lib/x86_64-linux-gnu/
</pre>

== install packages for Xilinx ==

ubuntu LTS 22.04 vivado 2020.1

<pre>
apt install autoconf libtool
apt install libtinfo5
apt install texinfo
apt install zlib1g:i386
</pre>

== install packages for building ROOT ==

<pre>
apt -y install libx11-dev libxpm-dev libxft-dev libxext-dev libpng-dev libjpeg-dev xlibmesa-glu-dev libxml2-dev libgsl-dev cmake
</pre>

== install 32-bit libraries for PHYSICA ==

these instructions are for running 32-bit physica executable built for SL6 on ubuntu LTS 20.04

install physica sources (cannot build, do not have g77)

<pre>
cd ~/packages
git clone https://bitbucket.org/ttriumfdaq/physica.git
</pre>

install 32-bit libraries using ubuntu package manager:

<pre>
apt install lib32z1 # libz.so
</pre>

copy 32-bit SL6 shared libraries to /lib32

<pre>
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libX11.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libgd.so.2 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libpng12.so.0 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libreadline.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libncurses.so.5 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libg2c.so.0 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libxcb.so.1 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libXpm.so.4 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libjpeg.so.62 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libfontconfig.so.1 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libfreetype.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libtinfo.so.5 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libXau.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libexpat.so.1 /lib32/
</pre>

ldd should report:

<pre>
trinatdaq:trinat> ldd /usr/local/physica/physica.exe
linux-gate.so.1 (0xf7fa2000)
libX11.so.6 => /lib32/libX11.so.6 (0xf7e43000)
libgd.so.2 => /lib32/libgd.so.2 (0xf7dfe000)
libpng12.so.0 => /lib32/libpng12.so.0 (0xf7dd6000)
libz.so.1 => /lib32/libz.so.1 (0xf7db8000)
libreadline.so.6 => /lib32/libreadline.so.6 (0xf7d7e000)
libncurses.so.5 => /lib32/libncurses.so.5 (0xf7d5b000)
libg2c.so.0 => /lib32/libg2c.so.0 (0xf7d3d000)
libm.so.6 => /lib32/libm.so.6 (0xf7c39000)
libgcc_s.so.1 => /lib32/libgcc_s.so.1 (0xf7c1a000)
libc.so.6 => /lib32/libc.so.6 (0xf7a2f000)
libxcb.so.1 => /lib32/libxcb.so.1 (0xf7a05000)
libdl.so.2 => /lib32/libdl.so.2 (0xf79ff000)
libXpm.so.4 => /lib32/libXpm.so.4 (0xf79ee000)
libjpeg.so.62 => /lib32/libjpeg.so.62 (0xf7997000)
libfontconfig.so.1 => /lib32/libfontconfig.so.1 (0xf7962000)
libfreetype.so.6 => /lib32/libfreetype.so.6 (0xf78c9000)
libtinfo.so.5 => /lib32/libtinfo.so.5 (0xf78b0000)
/lib/ld-linux.so.2 (0xf7fa4000)
libXau.so.6 => /lib32/libXau.so.6 (0xf78ad000)
libexpat.so.1 => /lib32/libexpat.so.1 (0xf7885000)
trinatdaq:trinat>
</pre>

set login environment:

<pre>
setenv TRIUMF_FONTS $HOME/packages/physica/fonts
setenv PHYSICA_DIR $HOME/packages/physica
alias physica $PHYSICA_DIR/physica-SL6-32
</pre>

test:

<pre>
cd ~/packages/physica
physica
@rangauss.pcm
</pre>

== install lightdm ==

unlike the default gdm login manager, lightdm shows the machine hostname and does not require an extra mouse click to swicth from screen saver to login mode.

<pre>
apt -y install lightdm
# select lightdm
</pre>

== install desktop environments ==

note: default display manager and default desktop are deficient, please do not skip this step.

note: if apt asks to choose the display manager, select "lightdm"

note: KO - I recommend the "MATE" desktop.

note: you will have to cut-and-paste this several times because "apt" eats commands, even with "-y" and even piped from "yes".

<pre>
# install MATE desktop
DEBIAN_FRONTEND=noninteractive apt -y install ubuntu-mate-core ubuntu-mate-desktop ubuntu-mate-themes
# install Cinnamon desktop
DEBIAN_FRONTEND=noninteractive apt -y install cinnamon
# install KDE desktop
DEBIAN_FRONTEND=noninteractive apt -y install kubuntu-desktop
# install Lxqt desktop
DEBIAN_FRONTEND=noninteractive apt -y install lxqt
# install Xfce4 desktop
DEBIAN_FRONTEND=noninteractive apt -y install xfce4
</pre>

== install ROOT ==

Please install ROOT per instructions at http://root.cern.ch.

NOTE1: The ROOT package available from Ubuntu repositories is severely out of date and cannot be used with MIDAS and ROOTANA. ### DO NOT DO THIS! apt-get install root-system

NOTE2: as of 2017-Jan-09, ROOT binary kits for Ubuntu do not work (use GCC 5 instead of GCC6), build from source instead.

== Install x2go ==

KO - is this still needed? does it cause any security problems?

x2go instructions, thanks to Art O.

<pre>
add-apt-repository ppa:x2go/stable
apt-get update
apt-get install x2goserver x2goserver-xsession
</pre>

== enable root login from ladd00/daq00 ==
<pre>
ssh localhost
CTRL-C
/bin/cp ~root/git/scripts/etc/authorized_keys ~root/.ssh/
</pre>

== install smart-status ==
<pre>
ln -s ~/git/scripts/smart-status/smart-status.perl ~root/
</pre>

== enable boot menu and boot messages ==

This will enable the grub menu (with a 10 sec timeout) and
replace black screen with exciting linux boot messages.

* emacs -nw /etc/default/grub
<pre>
GRUB_DEFAULT=0
#GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="vga=769 video=640x480"
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX=""
#GRUB_GFXMODE=640x480
</pre>
* update grub config:
<pre>
grub-mkconfig -o /boot/grub/grub.cfg
</pre>

== reboot ==

this completes installation of the base system.

following sections modify basic ubuntu to fix known problems and to enable special stuff.

= Enable automatic updates =

<pre>
apt install unattended-upgrades
cd ~/git/scripts
git pull
/bin/cp -v etc/99apt-conf-ko /etc/apt/apt.conf.d/
apt-config dump | grep Unattended
</pre>

Following is obsolete:

* emacs -nw /etc/apt/apt.conf.d/50unattended-upgrades
** uncomment in Allowed-Origins "-security" and "-updates"
** add in Allowed-Origins: "Google LLC:stable";
** uncomment/add: "Unattended-Upgrade::Mail "root";
* emacs -nw /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
</pre>
* test: unattended-upgrade --dry-run -v

NOTE: update-on-shutdown is disabled.

NOTE: there is no update-on-boot, but:

NOTE: if machine was off for a long time, the systemd update timer would have expired and it will fire soon after reboot, causing an automatic update run. this is unwanted, and there is no fix or workaround for it. K.O. June-2023.

= Fix bpool is full =

!!! only if ROOT on ZFS !!!

There is an error in the zsys package that causes bpool to run out of space,
see [[#Ubuntu zsys]] for more details.

To fix:
<pre>
cd ~/git/scripts
git pull
cp etc/zsys.conf /etc/
zsysctl service reload
zsysctl service gc
zpool list bpool
zfs list bpool
df /boot
</pre>

= IPMI instructions =

IPMI is the board management hardware on Supermicro and other server motherboards. This includes hardware sensors - fan rotation speed, temperatures and power supply voltages.

<pre>
apt-get install ipmitool
systemctl enable ipmievd
systemctl restart ipmievd
</pre>

Run:
* ipmitool sel list ### event list
* ipmitool sel elist ### event list
* ipmitool sel clear ### clear event list (if it becomes full)
* ipmitool sensor ### report hardware sensors

= move /home/wheel =

note: this MUST be done if ZFS root and NIS/autofs with /home.

Default location of wheel's home directory will collide with autofs /home, it has to be moved,
for example to /wheel.

<pre>
# logout from the wheel user
# go to another computer
ssh root@daqubuntuxxx
zfs list | grep wheel ### identify zfs name wheel_xxxxxx
zfs set mountpoint=/wheel rpool/USERDATA/wheel_hm8fzh
emacs -nw /etc/passwd ### change wheel's home directory from /home/wheel to /wheel
su - wheel ### check that user wheel still works
</pre>

TL edit (April 2023): I think also need to do chown wheel.wheel /wheel

= enable NIS (ubuntu 22.04, debian 11) =

<pre>
apt -y install rpcbind nis
echo DAQ-NIS >> /etc/defaultdomain
echo ypserver daq00.triumf.ca >> /etc/yp.conf
systemctl enable ypbind.service
systemctl restart ypbind.service
systemctl status ypbind.service
ypwhich -m
</pre>

enable ypserv:

<pre>
sed -i s/NISSERVER=false/NISSERVER=slave/ /etc/default/nis
/usr/lib/yp/ypinit -s daq00
echo ypserver localhost >> /etc/yp.conf
sed -i "s/ypserver .*/ypserver localhost/" /etc/yp.conf
systemctl enable ypserv
systemctl restart ypserv
systemctl restart ypbind
</pre>

edit /etc/nsswitch.conf to read:

<pre>
# begin get data from nis
passwd: files nis
group: files nis
shadow: files nis
automount: files nis
netgroup: files nis
# end get data from nis
</pre>

enable hourly update of nis maps:

<pre>
mkdir ~root/git
cd ~root/git
git clone http://daq00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>

If this is a new machine, then on the master NIS node (daq00), add this new node to /etc/netgroup, and update NIS maps (cd /var/yp; make)

= enable NIS (ubuntu 20.04) =

* apt-get -y install portmap nis ### will ask for NIS domain (DAQ-NIS)
* dpkg-reconfigure nis ### reconfigure if already installed
* ypwhich -m
* edit /etc/default/nis
** set "NISSERVER=slave"
** Ubuntu LTS 20.04, check that "YPBINDARGS=" is blank, remove "-no-dbus" if it is there
* #edit /etc/yp.conf, comment-out everything, add "domain DAQ-NIS server localhost"
* edit /etc/yp.conf, comment-out everything, add "ypserver localhost"
* /usr/lib/yp/ypinit -s daq00
* systemctl enable nis
* systemctl restart nis
* ypwhich
* ypwhich -m
* ypcat -k passwd
* vi /etc/nsswitch.conf ### add the automount line, modify the passwd, group and shadow lines to read this:
<pre>
# begin get data from nis
passwd: files nis
group: files nis
shadow: files nis
automount: files nis
netgroup: files nis
# end get data from nis
</pre>
* enable hourly update of NIS maps
<pre>
mkdir ~root/git
cd ~root/git
git clone http://ladd00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>
* ### NOT NEEDED sudo vi /etc/idmapd.conf ### add line: "Domain = triumf.ca"

= enable autofs =

<pre>
apt -y install autofs
systemctl enable autofs
systemctl restart autofs
ls -l /home/olchansk ### test autofs, check file owner is correct
</pre>

= enable NFS server =

<pre>
apt install nfs-kernel-server
#edit /etc/exports
systemctl enable nfs-server
systemctl restart nfs-server
</pre>

= NIS master =

notes for setting up the NIS master

== wheel user ==

"wheel" is the default administrative user. We do not want it's password exported to NIS (encrypted password hash is world visible) and we do not want it's home directory exported to NFS (~wheel/.ssh is world visible and potentially writable: anybody can change ~wheel/.ssh/authorized_keys).

* move wheel's home directory from /home/wheel to /wheel (see special section about this)
* change wheel's UID and GID from 1000 to a value below MINUID in /var/yp/Makefile

== coherent uids ==

we do not want system accounts defined in /etc/passwd of the NIS master
to be included in the NIS map "passwd". this causes trouble on NIS clients
where newly installed packages fail to create local system users because same
user already exists in NIS.

This is controlled by MINUID in /var/yp/Makefile.

Historical TRIUMF uids start from around 200, but several clusters do not have any historic TRIUMF uids below 500 and MINUID is set to:
* DAQ-NIS: MINUID=200
* ISAC-NIS: MINUID=500
* TITAN-NIS: MINUID=500
* MUSR-NIS: MINUID=500
* TIG-NIS: MINUID=500 (100 on SL6 mother8pi)

Ubuntu 20 has two programs to create users:
* adduser - creates new users with UID 1000 and up as specified in /etc/adduser.conf. No problems here.
* adduser --system - creates new system users with UID 100 and up as specified in /etc/adduser.conf. No problems here.
* useradd - creates new users with UID 1000 and up as specified in /etc/login.defs. No problems here.
* useradd --system - creates new system users with UID 999 and down (read "man useradd", section at the end about SYS_UID_MAX). This collides with NIS MINUID, these system users will be included in the NIS map and cause trouble.

This problem cannot be fixed, SYS_UID_MIN, SYS_UID_MAX and UID_MIN in /etc/login.defs do not seem
to have any effect on UIDs chosen by "useradd --system". (tested on Ubuntu LTS 20.04).

So far only these system accounts seem to be affected by this:
* systemd-coredump
* ganglia

To fix:
* run "sort -r -n -t: -k3 /etc/passwd" to identify the last unused system user uid (range 100..200)
* run "sort -r -n -t: -k3 /etc/group" to identify the last unused system user gid (range 100.200)
* systemd-coredump: manually change UID and GID (package systemd-coredump is usually not installed)
* ganglia: same thing, then change ownership on all ganglia files.

Also read systemd author's opinion on system vs user UIDs:
https://github.com/systemd/systemd/issues/4850#issuecomment-265698275

= Fix systemd-logind NIS breakage =

!!! THIS IS NOT NEEDED FOR UBUNTU LTS 20.04 !!!

there is a delay in ssh logins for normal users. "ssh -v" shows the delay is after "pledge...". this
fix removes the delay.

systemd developers think that we should not use NIS and made sure there are
problems if we do. To give them credit, they do offer a workaround. Read this:
https://github.com/poettering/systemd/commit/695fe4078f0df6564a1be1c4a6a9e8a640d23b67

<pre>
mkdir /etc/systemd/system/systemd-logind.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-logind.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-logind.service
</pre>

= Fix systemd-udevd NIS breakage =

see same problem as above with udev getting stuck. ubuntu lts 20.04.

<pre>
mkdir /etc/systemd/system/systemd-udevd.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-udevd.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-udevd.service
</pre>

= Configure USB device permissions =

Configure USB device permissions for user access to USB-serial devices, Altera USB Blaster, etc.

* create file /etc/udev/rules.d/99-usb-chmod.rules with this contents:

<pre>
emacs -nw /etc/udev/rules.d/99-usb-chmod.rules
ACTION=="add", SUBSYSTEM=="usbmisc", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /dev/%c"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /proc/%c"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVICE}"
ACTION=="add", ENV{PHYSDEVBUS}=="usb-serial", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVPATH}=="/class/tty/ttyS*", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyUSB*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyACM*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyS*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", DEVPATH=="*video*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
</pre>

* reload udev rules: udevadm control --reload-rules
* apply new permissions: udevadm trigger --action=add
* watch udev activity: udevadm monitor -p

= Configure lightdm display manager =

* enable it
<pre>
echo lightdm | dpkg-reconfigure -fteletype lightdm
systemctl disable gdm
systemctl disable sddm
systemctl enable lightdm
</pre>

* make the MATE desktop as default
<pre>
cd ~root/git/scripts/
git pull
/bin/cp -v etc/lightdm_default_mate.conf /etc/lightdm/lightdm.conf.d/
</pre>

* enable login by NIS users
<pre>
/bin/cp -v etc/lightdm_enable_nis_login.conf /etc/lightdm/lightdm.conf.d/
</pre>

* restart lightdm
<pre>
systemctl stop gdm
systemctl restart lightdm
</pre>

= Install libpng12.so.0 =

Quartus 16 needs libpng12:

<pre>
wget http://mirrors.kernel.org/ubuntu/pool/main/libp/libpng/libpng12-0_1.2.54-1ubuntu1_amd64.deb
dpkg --install libpng12-0_1.2.54-1ubuntu1_amd64.deb
</pre>

= Install google-chrome =

<pre>
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
dpkg -i google-chrome-stable_current_amd64.deb
</pre>

confirm autoupdate is enabled, observe dl.google.com is present in the list of repositories:
<pre>
apt update
...
Get:5 https://dl.google.com/linux/chrome/deb stable/main amd64 Packages [1,094 B]
...
</pre>

FOLLOWING IS OBSOLETE:

Instructions from here:
https://www.ubuntuupdates.org/ppa/google_chrome?dist=stable

<pre>
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-tmp.list'
apt update
apt install google-chrome-stable
/bin/rm -f /etc/apt/sources.list.d/google-tmp.list
</pre>

= Install amanda client =

ONLY ONE MACHINES THAT HOST HOME DIRECTORIES

* apt install amanda-client
* edit /etc/amandahosts
<pre>
amanda.triumf.ca amanda amdump
</pre>
* check permissions on /etc/amandahosts:
<pre>
root@daq00:/var/log/amanda# ls -l /etc/amandahosts
-rw------- 1 backup backup 49 Jan 27 10:48 /etc/amandahosts
</pre>
* fix if needed: chown backup.backup /etc/amandahosts; chmod a= /etc/amandahosts; chmod u=wr /etc/amandahosts
* edit /etc/amanda-security.conf, add this line:
<pre>
runtar:gnutar_path=/usr/bin/tar
</pre>

On the amanda machine:

* in amanda disklist, use dump type "bsdtcp-comp-user-tar"
* su - amanda and run amcheck -c daily daq00
<pre>
-bash-4.1$ amcheck -c daily daq00

Amanda Backup Client Hosts Check
--------------------------------
Client check: 1 host checked in 0.092 seconds. 0 problems found.

(brought to you by Amanda 3.3.7p1.git.685ff76d)
</pre>

= Enable rc.local =

For reasons unknown, Ubuntu LTS 20.04 does not enable /etc/rc.local. Do this:

<pre>
cd ~/git/scripts
git pull
cp -n -v etc/rc.local /etc/
chmod a+rx /etc/rc.local
cp etc/rc-local.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable rc-local
systemctl start rc-local
systemctl status rc-local
</pre>

= Remove unwanted packages =

<pre>
apt remove zsys
apt remove sddm
</pre>

= Disable unwanted services =

<pre>
systemctl disable mpd
systemctl disable snapd
systemctl disable ModemManager
systemctl --global mask tracker-extract-3.service
systemctl --global mask tracker-miner-fs-3.service
systemctl daemon-reload
</pre>

= Disable sleep and suspend =

note: we see some computers randomly shutdown or go to sleep, log files indicates the "sleep" or "suspend" button was pushed by user, but no such buttons actually exist. this is the fix for this:

<pre>
systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target systemd-suspend.service systemd-hybrid-sleep.service
</pre>

= Enable crontab @reboot for MIDAS =

startup scripts have a bug - cron @reboot entries for normal users can run before autofs is ready, so if the home directory is on autofs/NFS, it cannot be accessed and the cron job fails. If MIDAS is supposed to be started by cron @reboot, it will not start (there *will* be an error message in /var/log/cron).

<pre>
mkdir /etc/systemd/system/cron.service.d
echo -e "[Unit]\nAfter=ypbind.service autofs.service\n" > /etc/systemd/system/cron.service.d/local.conf
systemctl daemon-reload
systemctl cat cron.service
</pre>

Explore the systemd dependency tree using "systemctl list-dependencies" maybe with "--all".

Visualize the exact boot sequence from previous boot: "systemd-analyze plot > xxx.svg", look at the svg file using a web browser.

Crontab entry to start midas: (install in the midas user crontab, not root crontab)

<pre>
su - midasuser
crontab -l
#@reboot /bin/bash -l -c "/home/trinat/bin/start-daq-applications"
#@reboot /bin/tcsh -c "/home/trinat/bin/start-daq-applications"
</pre>

= Install apache httpd proxy for midas and elog =

This will configure the HTTPS/SSL certificate using "certbot" and "letsencrypt" and configure an HTTPS web server using apache2.

First, configure apache2:

* execute these commands:
<pre>
apt -y install apache2
cd /etc/apache2
</pre>
* create new file conf-available/ssl-daq14.conf # use actual hostname instead of daq14
<pre>
SSLSessionCache shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout 300
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
SSLCryptoDevice builtin
</pre>
* create new file sites-available/daq14-ssl.conf # use actual hostname instead of daq14
<pre>
<IfModule mod_ssl.c>
<VirtualHost *:443>
ServerName daq14.triumf.ca
DocumentRoot /var/www/html
ErrorLog /var/log/apache2/daq14.log
SSLEngine on
# note SSLProtocol, SSLCipherSuite and some other settings are overwritten by /etc/letsencrypt/options-ssl-apache.conf
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SEED:!IDEA:!RC4
## use port specified in elogd.cfg
#ProxyPass /elog/ http://localhost:8082/ retry=1
## use mhttpd port
#ProxyPass / http://localhost:8080/ retry=1
Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"
<Location />
SSLRequireSSL
AuthType Basic
AuthName "DAQ password protected site"
Require valid-user
# create password file: touch /etc/apache2/htpasswd
# to add new user or change password: htpasswd /etc/apache2/htpasswd username
AuthUserFile /etc/apache2/htpasswd
</Location>
</VirtualHost>
</IfModule>
</pre>
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* enable ssl module
* enable new configurations
<pre>
a2enmod ssl
a2enmod headers
a2enmod proxy
a2enmod proxy_http
a2enconf ssl-daq14
a2ensite daq14-ssl
</pre>
* disable default ssl sites
<pre>
a2dissite 000-default-le-ssl
a2dissite 000-default
ls -l /etc/apache2/sites-enabled/ ### should show only daq14-ssl.conf
</pre>
* check that there are no syntax problems
<pre>
apache2ctl configtest
</pre>
* enable and start apache2:
<pre>
systemctl enable apache2
systemctl restart apache2
systemctl status apache2
</pre>
* apache2 may fail to start, look in /var/log/apache2/error.log and /var/log/apache2/daq14.log
* if it says "Failed to configure ... certificate", proceed to the step for setting certbot.
* try to access https://daq14.triumf.ca
** you should see a complaint about self-signed certificate
** you should see a request for password (do not login yet)
** if you get "connection refused", HTTPS port 443 may need to be enabled in the local firewall, look at documentation for ufw.
Second, configure certbot:

(Note: as of 2018-01-18 certbot requires use of http port 80 to get the initial https certificate,
renewal can continue to use the https port 443)

(Note: as of 2019-01-?? certbot requires use of port 80 for renewals)

* check that port 80 is not used by anything:
* netstat -an | grep LISTEN | grep ^tcp | grep 80
* lsof -P | grep -i tcp | grep LISTEN | grep 80
* if lsof reports that apache2 is listening on port 80, follow the apache2 instructions above (remove "listen 80" from apache2.conf

* install certbot (if necessary open tcp port 80 in the firewall, see documentation for ufw):
<pre>
apt install certbot python3-certbot-apache
certbot certonly --standalone --installer apache
</pre>
* then answer questions:
* "activate HTTPS for daq14.triumf.ca" - say ok
* "enter email address" - enter your own email address
* "please read terms..." - read the terms and say "agree"
* it will take a few moments...
* "congratulations..." - say ok.
<pre>
certbot install --apache --cert-name daq14.triumf.ca
</pre>
* then answer questions:
* "choose redirect..." - say "1" (no redirect)
* look inside /etc/apache2/sites-enabled/daq14-ssl.conf to see that SSLCertificateFile & co point to certbot certificates in
/etc/letsencrypt/live/daq14.triumf.ca/
* to check current renewal and to update the certbot config file in /etc/letsencrypt/renewal, run this:
<pre>
certbot renew --standalone --installer apache --force-renewal
</pre>

NOTE: this certificate will expire in 3 months, automatic renewal should work with current version of certbot

Third, activate password protection:

* as shown in the config file above, create password file and initial user: (replace "midas" with specific username)
<pre>
touch /etc/apache2/htpasswd
htpasswd /etc/apache2/htpasswd midas
</pre>

* restart apache2
<pre>
systemctl restart apache2
systemctl status apache2
</pre>

From here:
* enable proxy for MIDAS mhttpd - uncomment redirect in the config file above
* enable proxy for ELOG - ditto
<pre>
a2enmod proxy
a2enmod proxy_http
apache2ctl configtest
systemctl restart apache2
</pre>

From here:
* enable proxy for MIDAS mhttpd - uncomment redirect in the config file above
* enable proxy for ELOG - ditto
<pre>
a2enmod proxy
a2enmod proxy_http
apache2ctl configtest
systemctl restart apache2
</pre>
* try accessing MIDAS https://daq14.triumf.ca/ (make sure mhttpd is running)
* if it's not working, check odb setting FIXME!
* try accessing ELog https://daq14.triumf.ca/elog/ (make sure elogd is running)
* if it's not working, check elogd.cfg file and make sure
<pre>
SSL = 0
</pre>

NOTE: if certbot fails with errors about 'module' object has no attribute 'pyopenssl',
try this: pip install requests==2.6.0

= Enable elog PDF preview =

see https://stackoverflow.com/questions/52998331/imagemagick-security-policy-pdf-blocking-conversion

* xemacs -nw /etc/ImageMagick-6/policy.xml
* remove this section at the end:
<pre>

<policy domain="coder" rights="none" pattern="PS" />
<policy domain="coder" rights="none" pattern="PS2" />
<policy domain="coder" rights="none" pattern="PS3" />
<policy domain="coder" rights="none" pattern="EPS" />
<policy domain="coder" rights="none" pattern="PDF" />
<policy domain="coder" rights="none" pattern="XPS" />
</pre>

= Install Jupyter notebook =

<pre>
From https://jupyter.org/install
apt install python3-pip
pip install jupyterlab
pip install notebook
~/.local/bin/jupyter notebook
watch the http://localhost:8888 URL that it printed
say "no" to offer to start firefox (it will not work!)
URL is: http://localhost:8888/tree?token=xxx
from the machine where you are running the web browser (i.e. google-chrome), run (replace trinat@trinatdaq with the username and machine name where you started jupyter)
open a new shell and run: ssh -v trinat@trinatdaq -L 8888:localhost:8888
in the web browser, open http://localhost:8888
this gives us the login page
in the password or token entry field, put the token from the "tree?token=xxx" above (printed by jupyter on startup)
push button "login"
jupyter page should open with the list of files in the trinat home directory
congratulate Brian with full success
</pre>

= Install ZFS quota report =

If there are any ZFS volumes, install script to report disk and quota usage

<pre>
cd ~/git/scripts/quotareport
git pull
mkdir /var/www/html/zfsquotareport
cp -pv ~/git/scripts/quotareport/sorttable.js /var/www/html/zfsquotareport/
ln -s $PWD/zfsquotareport.perl /etc/cron.daily/
touch /etc/crontab
</pre>

If httpd is configured to redirect "/" to MIDAS mhttpd:
* add following to /etc/apache2/sites-enabled/xxx-ssl.conf in front of "ProxyPass / ..."
* run "systemctl reload apache2"
<pre>
## do not proxy zfs quota report directory
ProxyPass /zfsquotareport/ !
</pre>

= Install PHP =

* apt install php libapache2-mod-php
* systemctl restart apache2
* create /var/www/html/info.php
<pre>
<?php

phpinfo();
</pre>
* open https://daq00.triumf.ca/info.php

= Configure TRIUMF printers =

<pre>
systemctl stop cups
systemctl disable cups
echo "ServerName printers.triumf.ca" > /etc/cups/client.conf
lpstat -a
</pre>

= Enable core dumps =

By default, Ubuntu LTS 20.04 installs the apport package
which disabled core dumps from user applications. (google it up!).
It is not meant to do this and documentation claims that
it is not installed and not enabled by default. Oh, well...

<pre>
apt remove apport
apt autoremove ### will remove apport-symptoms and a few other packages
</pre>

After this, core dumps are written to file "core" in the current directory.
See /proc/sys/kernel/core_pattern and /proc/sys/kernel/core_uses_pid.

= Enable debugger =

By default, Ubuntu LTS 20.04 does not permit debugger to attach and debug
already running programs. To enable it, add following to /etc/rc.local

<pre>
echo 0 > /proc/sys/kernel/yama/ptrace_scope
</pre>

= Disable Ubuntu Pro nag =

If "apt upgrade" requests Ubuntu Pro or esm-apps, disable the nag:
<pre>
/bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf
</pre>

= Update packages =

* apt-get update # update package list
* apt-get dist-upgrade # install updated packages and update "kept back" packages
* apt-get autoremove # remove packages that apt thinks should be removed

= Finish installation =

Congratulations. There is nothing more to do!

* reboot
<pre>
shutdown -r now
</pre>

= Install ZFS =

!!! after installing all the packages, after updating the system, after updating the linux kernel, after rebooting into latest kernel !!!

<pre>
apt-get install zfsutils-linux
</pre>

Follow generic ZFS instructions: [[ZFS]]

= Update to new version of Ubuntu =

<pre>
vi /etc/update-manager/release-upgrades # set "Prompt=normal"
do-release-upgrade
</pre>

Update Ubuntu LTS 20.04 to LTS 22.04:

== daqubuntu ==

<pre>
# reboot to clear out all updates
# vi /etc/update-manager/release-upgrades # set "Prompt=normal"
# do-release-upgrade -c
Checking for a new Ubuntu release
New release '22.04 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
# do-release-upgrade
...
say yes...
...
login.defs, say "Y" (erase local changes, use packaged version)
/etc/systemd/resolved.conf, say "Y" (same as above)
firefox snap, say yes
unable to reach snap store, say "skip"
/etc/gmond.conf, say "Y"
/var/yp/Makefile, say "install the package maintainer's version"
/etc/ypserv.conf, same thing
/etc/ypserv.securenets, same thing
/etc/default/nis, same thing
/etc/speech-dispatcher/modules/mary-generic.conf, same thing
/etc/apt/apt.conf.d/50unattended-upgrades, same thing
...
278 packages are going to be removed, say yes
...
restart required, say yes
...
no ping... yes ping...
...
ssh daqubuntu, ok
apt update, fail, DNS does not work, "host security.ubuntu.com" does not resolve.
fix resolver per https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Disable_NetworkManager
apt update, apt upgrade now works, 0 packages to update
NIS does not work.
</pre>

== midm9a ==

<pre>
login.defs
firefox snap
gmond.conf
ypserv
/etc/default/nis
unattended-upgrades
amanda-security.conf
remove obsolete (no)
reboot
configure dns
reenable nis
</pre>

== daq17 ==

<pre>
firefox snap
imagemagick policy.xml
gmond.conf
chrony.conf
/var/yp/Makefile
ypserv.conf
ypserv.securenets
/etc/default/nis
50unattended-upgrades
</pre>

== daq00 ==

per https://serverpilot.io/docs/how-to-upgrade-ubuntu-20.04-to-22.04/

<pre>
do-release-upgrade -f DistUpgradeViewNonInteractive
</pre>

if it exists "too soon" without doing anything, run it without "-f xxx", most likely it does not like something about this machine. in case of daq00 it did not like how the EFI partitions were mounted. after fixing it, non-interactive upgrade was successful.

= Ubuntu package manager =

* apt-get install xxx # install package xxx
* apt-get update
* apt-get upgrade
* apt-get dist-upgrade
* apt-get autoremove # remove automatically installed packages required by a removed package
* apt-get remove xxx # remove package xxx
* apt-cache search . # list all available packages
* apt-cache show "." | grep ^Package # list al available packages
* apt-cache madison root-system # show all available versions of package root-system
* apt list # list all installed packages
* dpkg --listfiles libpng16-16 # list all files from this package
* apt list --installed # list all installed packages
* dpkg -S /bin/bash # what package provides this file?
* dpkg -L bash # what files provided by this package?
* debsums -ce # show modified config files
* apt-config dump # show apt configuration

= Ubuntu zsys =

NOTE: DO NOT USE ZSYS, see https://github.com/ubuntu/zsys/issues/218 and https://github.com/ubuntu/zsys/issues/230

* manual removal of old snapshots
<pre>
zsysctl show
zsysctl state remove xy69ye -s
zsysctl state remove xy69ye
zsysctl state remove xy69ye -u wheel
</pre>
* apt remove zsys

NOTE: old zsys snapshots must be cleaned manually, "zsysctl state remove xxx --system" is broken and does not remove user data snapshots

* manages system snapshots
* documentation: https://github.com/ubuntu/zsys
* documentation: (go to next article via link "newer" at the bottom) https://didrocks.fr/2020/05/21/zfs-focus-on-ubuntu-20.04-lts-whats-new/
* ubuntu 20.04 bug, too many snapshots cause /boot to become full and updates fail. https://github.com/ubuntu/zsys/issues/155
* solution: use custom /etc/zsys.conf, limit number of snapshots to 10, see trinatdaq:/etc/zsys.conf
* zsys commands:
<pre>
update-grub # list of all snapshots, errors if some snapshots are broken
zsysctl state remove lnc0k7 --system # remove snapshot
xemacs -nw /etc/zsys.conf; zsysctl service reload; zsysctl service gc # cause gc to run with new settings in zsys.conf
zfs list -r -t snapshot -o name,used,referenced,creation bpool/BOOT # list snapshots
zsysctl show # show snapshots
</pre>

= Ubuntu cloning =

to clone a ubuntu image:

<pre>
cd /nfsroot/lxcpet
emacs -nw etc/hostname ### change hostname
emacs -nw etc/mailname ### change hostname (debian 11)
emacs -nw etc/defaultdomain ### change the NIS domainname
emacs -nw etc/yp.conf ### change the NIS server
cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
emacs -nw root/.ssh/authorized_keys ### update root ssh keys
</pre>

= Ubuntu boot loader =

== boot from ZFS ==

* use UEFI boot with syslinux, see here: https://daq.triumf.ca/DaqWiki/index.php/SLinstall#Configure_UEFI_boot
* apt install zfs-initramfs
* update-initramfs -v -u
* ZFS structure:
<pre>
root@daq00:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 147G 1.62T 96K /
rpool/ROOT 17.8G 1.62T 96K none
rpool/ROOT/ubuntu_00aaaa 17.8G 1.62T 6.22G /
</pre>
* copy OS image to rpool/ROOT/ubuntu_00aaaa
* zfs set mountpoint=/ rpool
* zfs set mountpoint=none rpool/ROOT
* zfs set mountpoint=/ rpool/ROOT/ubuntu_00aaaa
* zfs get all | grep mountpoint
<pre>
rpool mountpoint / local
rpool/ROOT mountpoint none local
rpool/ROOT/ubuntu_00aaaa mountpoint / local
</pre>
* in linux kernel command line (syslinux.cfg), set "root=" to "root=ZFS=rpool/ROOT/ubuntu_00aaaa"

== boot from ZFS mirror ==

=== setup the EFI partitions ===

* assuming /dev/sdb is already setup for EFI boot, setup /dev/sda the same way:
* partition the second boot disk same as first boot disk:
<pre>
root@grsnis01:~# gdisk -l /dev/sdb
Found valid GPT with protective MBR; using GPT.
Number Start (sector) End (sector) Size Code Name
1 2048 1050623 512.0 MiB EF00 EFI system partition
2 1050624 3907029134 1.8 TiB 8300 Linux filesystem
root@grsnis01:~#
</pre>
* mkfs.msdos /dev/sdX1
* create mount points
<pre>
mkdir /boot/efi-sda
mkdir /boot/efi-sdb
</pre>
* add to /etc/fstab
<pre>
/dev/sda1 /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
/dev/sdb1 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
</pre>
* mount -a
* df | grep boot
<pre>
root@grsnis01:~# df | grep boot
/dev/sdb1 523248 98100 425148 19% /boot/efi-sdb
/dev/sda1 523248 4 523244 1% /boot/efi-sda
</pre>
* copy boot files to new boot disk
* cd /boot/efi-sdX; rsync -av . /boot/efi-sdY
* set BIOS to boot from "UEFI Hard drive", disable legacy boot (except for booting from USB key in legacy mode)
* if using UEFI boot syslinux per these instructions, linux kernel update has to be done manually:
* run ~/git/scripts/etc/update_efi_mirror.perl, follow instructions that it prints.

=== setup zfs partitions ===

use partitions compatible with Ubuntu "install on ZFS"

* gdisk "o" to create new GPT partition table
* gdisk "n" +512M ef00 to create EFI partition
* gdisk "n" +2G 8200 to create linux swap partition (not used)
* gdisk "n" +2G BE00 to create ZFS bpool partition
* gdisk "n" xxx BF00 create ZFS rpool partition

<pre>
# gdisk -l /dev/sda
Number Start (sector) End (sector) Size Code Name
1 2048 1050623 512.0 MiB EF00 EFI System Partition
2 1050624 5244927 2.0 GiB 8200
3 5244928 9439231 2.0 GiB BE00
4 9439232 234441614 107.3 GiB BF00
root@midm9a:~#
</pre>

=== setup zfs mirror ===

* see here: https://daq.triumf.ca/DaqWiki/index.php/ZFS#Convert_pool_from_single_to_mirror
<pre>
root@grsnis01:~# ls -l /dev/disk/by-id/ata*part2
lrwxrwxrwx 1 root root 10 Feb 19 16:47 /dev/disk/by-id/ata-WDC_WDS200T2B0A-00SM50_205007801081-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Feb 19 16:47 /dev/disk/by-id/ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 -> ../../sdb2

root@grsnis01:~# zpool status
pool: rpool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 ONLINE 0 0 0

errors: No known data errors

root@grsnis01:~# zpool attach rpool ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 /dev/disk/by-id/ata-WDC_WDS200T2B0A-00SM50_205007801081-part2

root@grsnis01:~# zpool status
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Feb 19 16:54:39 2021
12.6G scanned at 3.16G/s, 1.02G issued at 262M/s, 12.6G total
1.02G resilvered, 8.09% done, 0 days 00:00:45 to go
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801081-part2 ONLINE 0 0 0 (resilvering)

errors: No known data errors
</pre>
* wait
<pre>
root@grsnis01:~# zpool status
pool: rpool
state: ONLINE
scan: resilvered 12.7G in 0 days 00:00:40 with 0 errors on Fri Feb 19 16:55:19 2021
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801081-part2 ONLINE 0 0 0

errors: No known data errors
</pre>

== maintenance commands ==

* update-initramfs -v -u
* grub-install /dev/sda

= Convert from single to dual mirrored ZFS SSD =

Assuming Ubuntu LTS 22.04 with "instal on ZFS" option, we will
add a second SSD, configure ZFS to use both SSDs in mirrored
configuration and setup grub to boot from either SSD. This
is intended to create a full redundant system where failure
of either SSD does not break the system.

* identify first SSD
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* connect second SSD of identical size
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
/dev/sdb WD Blue SA510 2.5 250GB 22243Z803852 25 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* if second SSD is not autodetected, reboot
* Clone partition table automatically
If both SSDs are identical size, use this simpler method of duplicating the partition table:
<pre>
root@midm9b:~# sfdisk -d /dev/sda > part_table
root@midm9b:~# grep -v ^label-id part_table | sed -e 's/, *uuid=[0-9A-F-]*//' | sfdisk /dev/sdb
</pre>
The grep and sed in the second command are there to prevent disk ID and partition IDs from being cloned. Alternatively the part_table file can be edited manually to remove the label-id line and the uuid entries from the individual partitions.

* Clone partition table manually (e.g. for different size disks)
* list partition table of first SSD:
<pre>
root@midm9b:~# fdisk -l /dev/sda
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>
* create identical partitions on second SSD, use sector numbers from above.
<pre>
root@midm9b:~# gdisk /dev/sdb
GPT fdisk (gdisk) version 1.0.8

Partition table scan:
MBR: not present
BSD: not present
APM: not present
GPT: not present

Creating new GPT entries in memory.

Command (? for help): n
Partition number (1-128, default 1):
First sector (34-488397134, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-488397134, default = 488397134) or {+-}size{KMGTP}: 1050623
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): ef00
Changed type of partition to 'EFI system partition'

Command (? for help): n
Partition number (2-128, default 2):
First sector (34-488397134, default = 1050624) or {+-}size{KMGTP}:
Last sector (1050624-488397134, default = 488397134) or {+-}size{KMGTP}: 5244927
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 8200
Changed type of partition to 'Linux swap'

Command (? for help): n
Partition number (3-128, default 3):
First sector (34-488397134, default = 5244928) or {+-}size{KMGTP}:
Last sector (5244928-488397134, default = 488397134) or {+-}size{KMGTP}: 9439231
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): be00
Changed type of partition to 'Solaris boot'

Command (? for help): n
Partition number (4-128, default 4):
First sector (34-488397134, default = 9439232) or {+-}size{KMGTP}:
Last sector (9439232-488397134, default = 488397134) or {+-}size{KMGTP}:
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): bf00
Changed type of partition to 'Solaris root'

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sdb.
The operation has completed successfully.
root@midm9b:~# fdisk -l /dev/sda /dev/sdb
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root

Disk /dev/sdb: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: EB251739-30C6-422F-A505-5887B5A0B603

Device Start End Sectors Size Type
/dev/sdb1 2048 1050623 1048576 512M EFI System
/dev/sdb2 1050624 5244927 4194304 2G Linux swap
/dev/sdb3 5244928 9439231 4194304 2G Solaris boot
/dev/sdb4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>
* identify second SSD partitions
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part3
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part3 -> ../../sda3
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 -> ../../sdb3
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
</pre>
* convert bpool from single disk to mirrored disk:
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~# zpool attach bpool 99e03dc0-7d4d-f24b-8fa1-f042b9f135db /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3
root@midm9b:~# zpool status bpool
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors
</pre>
* convert rpool
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
root@midm9b:~# zpool attach rpool f6fd54f8-3af7-b943-ae3d-a4e480537fb9 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4
root@midm9b:~# zpool status rpool
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Jan 20 19:40:45 2023
5.83G scanned at 664M/s, 2.92M issued at 332K/s, 9.11G total
0B resilvered, 0.03% done, no estimated completion time
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~#
</pre>
* wait for resilver to complete
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
scan: resilvered 9.65G in 00:00:36 with 0 errors on Fri Jan 20 19:41:21 2023
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
</pre>
* enable booting from second SSD: (instead of /dev/sda1, /dev/sdb1, use UUID=xxx)
<pre>
root@midm9b:~# mkfs.msdos /dev/sdb1
root@midm9b:~# mkdir /boot/efi-sda
root@midm9b:~# mkdir /boot/efi-sdb
root@midm9b:~# echo "/dev/sda1 /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1" >> /etc/fstab
root@midm9b:~# echo "/dev/sdb1 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1" >> /etc/fstab
root@midm9b:~# mount -a
root@midm9b:~# df -kl
Filesystem 1K-blocks Used Available Use% Mounted on
...
/dev/sda1 523244 13720 509524 3% /boot/efi
/dev/sdb1 523244 4 523240 1% /boot/efi-sdb
...
root@midm9b:~# rsync -av /boot/efi/ /boot/efi-sdb/
sending incremental file list
EFI/
...
root@midm9b:~# ls -l /boot/efi-sda
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~# ls -l /boot/efi-sdb
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~#
</pre>
* setup script to update grub on second SSD, it must be run manually after every kernel update
<pre>
root@midm9b:~# ln -s ~/git/scripts/etc/update_efi_grub.perl ~/
root@midm9b:~# ~/update_efi_grub.perl -u
EFI dir: /boot/efi-sda
/boot/efi-sda: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sda/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sda: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sda/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
EFI dir: /boot/efi-sdb
/boot/efi-sdb: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sdb/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sdb: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sdb/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
root@midm9b:~#
</pre>

= Disable NetworkManager =

NOTE: THIS IS BROKEN IN UBUNTU LTS 22.04

NetworkManager is useful for configuring dynamic
network interfaces, i.e. laptops that often move
between networks, or connect to multiple choice
of wifi networks, etc.

For machines with statically configured network interfaces,
NetworkManager is not necessary.

As it has been observed to become confused and observed
to malfunction when network links go up and down (it keeps
unnecessarily reconfiguring the ip address, etc), it can
be usefuil to disable it.

* list all network interfaces
<pre>
# /bin/ls -1 /sys/class/net/
enp0s31f6
lo
</pre>
* edit /etc/network/interfaces:
<pre>
rename enp0s31f6=eth0
auto eth0
iface eth0 inet static
address 142.90.120.94/19
gateway 142.90.100.18
</pre>
* statically configure systemd-resolved
** create /etc/systemd/resolved.conf.d/resolved.conf with this contents:
<pre>
[Resolve]
DNS=142.90.100.19
Domains=triumf.ca
</pre>
** systemctl restart systemd-resolved
** resolvectl
** systemd-analyze cat-config systemd/resolved.conf
* disable NetworkManager
<pre>
systemctl disable NetworkManager
</pre>
* reboot

= Configure ECC memory =

== Configure EDAC ==

* apt install edac-utils

=== Intel i3-2120 ===
<pre>
root@musr00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X9SCL/X9SCM
root@musr00:~# edac-ctl --status
edac-ctl: drivers not loaded.
</pre>

=== Intel E-2236 ===
<pre>
root@daq00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SCM-F
root@daq00:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq00:~# edac-util
edac-util: No errors to report.
root@daq00:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
</pre>
* check edac sysfs files (Intel)
<pre>
root@daq00:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_noinfo_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 max_location
-r--r--r-- 1 root root 4096 Jan 25 15:10 mc_name
drwxr-xr-x 2 root root 0 Jan 25 15:10 power
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank0
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank1
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank2
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank3
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank4
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank5
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank6
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank7
--w------- 1 root root 4096 Jan 25 15:10 reset_counters
-r--r--r-- 1 root root 4096 Jan 25 15:10 seconds_since_reset
-r--r--r-- 1 root root 4096 Jan 25 15:10 size_mb
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Jan 25 15:10 uevent
root@daq00:~#
</pre>

=== Intel E3-1270 v6 ===
<pre>
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --status
edac-ctl: drivers are loaded.
root@grsnis01:~# edac-util
edac-util: No errors to report.
root@grsnis01:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@grsnis01:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 max_location
-r--r--r-- 1 root root 4096 Feb 19 12:35 mc_name
drwxr-xr-x 2 root root 0 Feb 19 12:35 power
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank0
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank1
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank2
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank3
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank4
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank5
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank6
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank7
--w------- 1 root root 4096 Feb 19 12:35 reset_counters
-r--r--r-- 1 root root 4096 Feb 19 12:35 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 19 12:35 size_mb
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 19 12:35 uevent
root@grsnis01:~#
</pre>

=== Intel E3-1245 v6 ===

<pre>
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --status
edac-ctl: drivers are loaded.
[root@alphagdaq ~]# edac-util
edac-util: No errors to report.
[root@alphagdaq ~]# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
[root@alphagdaq ~]# ras-mc-ctl --layout
+-----------------------------------------------+
| mc0 |
| csrow0 | csrow1 | csrow2 | csrow3 |
----------+-----------------------------------------------+
channel1: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
----------+-----------------------------------------------+
[root@alphagdaq ~]# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#3channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#3channel#1 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#2channel#0 0 0
[root@alphagdaq ~]# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SSH-F
[root@alphagdaq ~]# ras-mc-ctl --summary
DBD::SQLite::db prepare failed: no such table: mc_event at /usr/sbin/ras-mc-ctl line 1129.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1130.
[root@alphagdaq ~]#
</pre>

=== AMD 3700X ===

(memory is non-ECC)

<pre>
root@daq13:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@daq13:~#
root@daq13:~#
root@daq13:~# edac-ctl --status
edac-ctl: drivers not loaded.
root@daq13:~# edac-util
edac-util: Error: No memory controller data found.
root@daq13:~# edac-util -s
edac-util: EDAC drivers loaded. No memory controllers found
root@daq13:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 2 root root 0 Jan 25 15:26 power
lrwxrwxrwx 1 root root 0 Jan 21 16:16 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Jan 21 16:16 uevent
</pre>

(memory is ECC)

<pre>
root@trinatdaq:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@trinatdaq:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@trinatdaq:~# edac-util
edac-util: No errors to report.
root@trinatdaq:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Dec 15 13:04 mc0
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
lrwxrwxrwx 1 root root 0 Dec 13 18:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dec 13 18:31 uevent
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_noinfo_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 max_location
-r--r--r-- 1 root root 4096 Dec 15 13:04 mc_name
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank4
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank5
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank6
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank7
--w------- 1 root root 4096 Dec 15 13:04 reset_counters
-rw-r--r-- 1 root root 4096 Dec 15 13:04 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Dec 15 13:04 seconds_since_reset
-r--r--r-- 1 root root 4096 Dec 15 13:04 size_mb
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Dec 15 13:04 uevent
root@trinatdaq:~#
</pre>

=== AMD 5000G ===

* no linux driver for AMD 5000-series "G" CPU
* no mention of ECC in the BIOS settings
* unclear status of ECC support in AMD documentation (sais only "pro" "G" CPUs have ECC)
* unclear status of ECC support in ASUS documentation (web page out of date)

=== AMD 5600X ===

<pre>
root@daq17:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-XE GAMING WIFI
root@daq17:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq17:~# edac-util
edac-util: No errors to report.
root@daq17:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@daq17:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Aug 19 19:27 mc0
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
lrwxrwxrwx 1 root root 0 May 10 10:11 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 May 10 10:11 uevent
root@daq17:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_noinfo_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 max_location
-r--r--r-- 1 root root 4096 Aug 19 19:27 mc_name
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank4
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank5
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank6
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank7
--w------- 1 root root 4096 Aug 19 19:27 reset_counters
-rw-r--r-- 1 root root 4096 Aug 19 19:27 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Aug 19 19:27 seconds_since_reset
-r--r--r-- 1 root root 4096 Aug 19 19:27 size_mb
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Aug 19 19:27 uevent
root@daq17:~#
</pre>

=== AMD 3955WX ===

<pre>
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --status
edac-ctl: drivers are loaded.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util
edac-util: No errors to report.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@alphasuperdaq:~/git/scripts/quotareport# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 19 root root 0 Dez 12 04:48 mc0
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
lrwxrwxrwx 1 root root 0 Dez 9 05:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dez 9 05:31 uevent
root@alphasuperdaq:~/git/scripts/quotareport#
root@alphasuperdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 max_location
-r--r--r-- 1 root root 4096 Feb 28 22:19 mc_name
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank0
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank1
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank10
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank11
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank12
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank13
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank14
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank15
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank2
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank3
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank4
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank5
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank6
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank7
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank8
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank9
--w------- 1 root root 4096 Feb 28 22:19 reset_counters
-rw-r--r-- 1 root root 4096 Feb 28 22:19 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Feb 28 22:19 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 28 22:19 size_mb
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 28 22:19 uevent
root@alphasuperdaq:~#
root@alphasuperdaq:~# ras-mc-ctl --layout
Use of uninitialized value $max_pos[3] in modulus (%) at /usr/sbin/ras-mc-ctl line 868.
Use of uninitialized value $d in numeric ge (>=) at /usr/sbin/ras-mc-ctl line 869.
Use of uninitialized value $d in sprintf at /usr/sbin/ras-mc-ctl line 872.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| mc0 |
| csrow0 | csrow1 |
| channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 | channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

0: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
root@alphasuperdaq:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#2 0 0
mc#0csrow#1channel#7 0 0
mc#0csrow#0channel#3 0 0
mc#0csrow#1channel#4 0 0
mc#0csrow#1channel#2 0 0
mc#0csrow#0channel#7 0 0
mc#0csrow#1channel#3 0 0
mc#0csrow#0channel#4 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#5 0 0
mc#0csrow#0channel#6 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#0channel#5 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#6 0 0
root@alphasuperdaq:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: ASUSTeK COMPUTER INC. model Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@alphasuperdaq:~#
</pre>

== Configure rasdaemon ==

<pre>
apt install rasdaemon
</pre>
<pre>
systemctl enable rasdaemon
systemctl restart rasdaemon
systemctl status rasdaemon
</pre>

<pre>
● rasdaemon.service - RAS daemon to log the RAS events
Loaded: loaded (/lib/systemd/system/rasdaemon.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-01-25 15:16:37 PST; 3min 5s ago
Main PID: 2477175 (rasdaemon)
Tasks: 1 (limit: 76958)
Memory: 17.1M
CGroup: /system.slice/rasdaemon.service
└─2477175 /usr/sbin/rasdaemon -f -r

Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Listening to events for cpus 0 to 11
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mc_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording aer_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording extlog_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mce_record events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording arm_event events
</pre>

== Get reports ==

* Intel 2x32GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --layout
+-------------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-------------------------+
channel1: | 16384 MB | 16384 MB |
channel0: | 16384 MB | 16384 MB |
----------+-------------------------+
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#0channel#1 0 0
root@daq00:~#
</pre>

* Intel 4x16GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#1 0 0
mc#0csrow#2channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#3channel#0 0 0
mc#0csrow#3channel#1 0 0
root@daq00:~#
root@daq00:~# ras-mc-ctl --layout
+-----------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-----------------------+
channel1: | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB |
----------+-----------------------+
root@daq00:~#
root@daq00:~#
root@daq00:~#
root@daq00:~# ras-mc-ctl --print-labels
ras-mc-ctl: Error: No dimm labels for Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@daq00:~#
</pre>

note: ubuntu LTS 22.04 DBD::SQLite::db error is not there.

= sensors =

== ASUS P9X79 WS ==

* https://www.asus.com/supportonly/P9X79%20WS/HelpDesk_Manual/
* BIOS version 4802
* modprobe nct6775
* modprobe coretemp

<pre>
root@daq14:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 0: +29.0°C (high = +82.0°C, crit = +100.0°C)
Core 1: +24.0°C (high = +82.0°C, crit = +100.0°C)
Core 2: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +82.0°C, crit = +100.0°C)

nouveau-pci-0200
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.00 V)
temp1: +39.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

nct6776-isa-0290
Adapter: ISA adapter
Vcore: 1.04 V (min = +0.00 V, max = +1.74 V)
in1: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
AVCC: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
+3.3V: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 2.04 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 904.00 mV (min = +0.00 V, max = +0.00 V) ALARM
3VSB: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
Vbat: 3.30 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1265 RPM (min = 0 RPM)
fan2: 1909 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +34.0°C (high = +0.0°C, hyst = +0.0°C) ALARM sensor = thermistor
CPUTIN: +58.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermal diode
AUXTIN: +31.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
PECI Agent 0: +31.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +96.0°C)
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

root@daq14:~#
</pre>

= Enable CPU turbo mode =

* Intel CPU has a nominal CPU frequency (i.e. 3.4GHz) and a turbo-boost CPU frequency (i.e. 4.0GHz). Here we will enable this turbo-boost mode.
* Find out CPU capability
<pre>
root@daq01:~# lscpu | grep Hz
Model name: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
CPU MHz: 3965.803
CPU max MHz: 4000.0000
CPU min MHz: 800.0000
root@daq01:~#
</pre>
* Look up this CPU in the Intel ARK database - google for the CPU model name, i.e.
https://ark.intel.com/content/www/us/en/ark/products/88196/intel-core-i7-6700-processor-8m-cache-up-to-4-00-ghz.html
* Find current frequency settings:
<pre>
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "powersave" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 2.72 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
root@daq01:~#
</pre>
* Note the following:
** current governor is "powersave"
** "performance" governor is available
** "boost state support" is supported and active.
* Confirm CPU frequency governor:
<pre>
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
powersave
powersave
powersave
powersave
powersave
powersave
powersave
powersave
root@daq01:~#
</pre>
* Change governor to "performance":
<pre>
root@daq01:~# cpupower frequency-set --governor performance
Setting cpu: 0
Setting cpu: 1
Setting cpu: 2
Setting cpu: 3
Setting cpu: 4
Setting cpu: 5
Setting cpu: 6
Setting cpu: 7
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
performance
performance
performance
performance
performance
performance
performance
performance
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "performance" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 3.93 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
</pre>
* monitor CPU frequency:
<pre>
root@daq01:~# cpupower monitor
| Nehalem || Mperf || Idle_Stats
CPU| C3 | C6 | PC3 | PC6 || C0 | Cx | Freq || POLL | C1 | C1E | C3 | C6 | C7s | C8
0| 0.00| 0.00| 0.00| 0.00|| 88.80| 11.20| 3973|| 0.00| 0.00| 0.01| 0.02| 0.31| 0.00| 4.25
4| 0.00| 0.00| 0.00| 0.00|| 4.70| 95.30| 3945|| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 95.03
1| 0.73| 3.70| 0.00| 0.00|| 4.52| 95.48| 3864|| 0.00| 0.01| 1.19| 0.44| 2.82| 0.00| 90.23
5| 0.73| 3.70| 0.00| 0.00|| 0.37| 99.63| 3807|| 0.00| 0.00| 0.03| 0.09| 1.70| 0.00| 97.64
2| 2.28| 12.86| 0.00| 0.00|| 1.41| 98.59| 3829|| 0.00| 0.86| 3.17| 0.46| 7.70| 0.00| 85.87
6| 2.28| 12.86| 0.00| 0.00|| 2.88| 97.12| 3856|| 0.00| 0.11| 4.56| 2.15| 10.31| 0.00| 78.99
3| 1.33| 4.81| 0.00| 0.00|| 0.99| 99.01| 3804|| 0.00| 0.49| 0.79| 0.01| 1.03| 0.00| 96.12
7| 1.34| 4.81| 0.00| 0.00|| 1.26| 98.74| 3818|| 0.00| 0.01| 2.32| 0.47| 5.02| 0.00| 90.06
root@daq01:~#
</pre>
* check that the CPU is not overheating:
<pre>
root@daq01:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 1: +38.0°C (high = +84.0°C, crit = +100.0°C)
Core 2: +34.0°C (high = +84.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +84.0°C, crit = +100.0°C)
</pre>
* congratulations, we are running at 4 GHz now!

= Setup ubuntu as gateway to private network =

See also:
* https://daq.triumf.ca/DaqWiki/index.php/VME-CPU#Setup_the_boot_host_computer_.28el7.29
* http://www.triumf.info/wiki/DAQwiki/index.php/Dhcpd_on_eth1

== Steps to do ==

* assign network numbers to the private network, i.e. 192.168.1.x, 192.168.2.x, etc
* (on the gateway machine, each private network interface has to have a different network number)
* (each network interface can have multiple networks attached, via VLANs or via eth0:0, eth0:1 constructs)
* assign IP addresses on the private network, save them in /etc/hosts i.e. "hvps 192.168.1.10"
* (for simplicity, assign 192.168.1.1 to the gateway machine itself)
* (IP addresses 192.168.1.0 and 192.168.1.255 are "special", do not use them)
* setup DNS server (dnsmasq) to serve contents of /etc/hosts via DNS (otherwise, many programs will see inconsistent name to IP address mapping)
* setup DHCP server (ISC dhcpd or dnsmasq) to give out the IP addresses
* setup tftp, pxelinux and NFS for diskless booting
* setup time server (chronyd) to provide common time to all devices
* setup NAT so machines on private network can access the internet (to get OS updates, etc)
* setup NIS and NFS so machines on the private network can use common home directories
* setup rsync backup of machines on the private network

== setup hosts ==

* edit /etc/hosts
<pre>
192.168.1.101 dsfe01
... and so forth
</pre>

== setup dns and dhcp ==

* apt install dnsmasq
* edit /etc/dnsmasq.conf
<pre>
# /etc/dnsmasq.conf
# DNS settings
#port=0 # disable DNS function
port=53 # enable DNS function
domain-needed
bogus-priv
no-resolv
server=142.90.100.19
# DHCP settings
interface=enp1s0f0 # DHCP interface
#dhcp-range=192.168.1.50,192.168.1.150,infinite
dhcp-range=192.168.1.0,static
#log-dhcp
quiet-dhcp
#dhcp-ignore=tag:!known
dhcp-boot=pxelinux.0
#dhcp-host=ac:1f:6b:9e:7f:4a,192.168.1.100,10m
dhcp-host=ac:1f:6b:9e:7f:4a,dsfe01,infinite
# TFTP settings
enable-tftp
tftp-root=/tftpboot
</pre>
* #mkdir /zssd/tftpboot ### per tftp-root (if no ZFS)
* zfs create -o mountpoint=/tftpboot rpool/tftpboot ### (if root is ZFS)
* systemctl stop systemd-resolved.service
* systemctl disable systemd-resolved.service
* rm /etc/resolv.conf
* create new /etc/resolv.conf with this contents:
<pre>
nameserver 127.0.0.1
search snolab.ca
</pre>
* systemctl enable dnsmasq
* systemctl restart dnsmasq

== setup chronyd ==

* enable ntp server:
* configure and enable chronyd per instructions above
* echo "allow 192.168.1.0/24" > /etc/chrony/conf.d/allow-localhost.conf
* systemctl restart chronyd
* chronyc tracking ### wait until time is synchronized (a few seconds)

== setup diskless network booting ==

=== setup pxelinux ===
<pre>
cd ~
wget https://www.kernel.org/pub/linux/utils/boot/syslinux/4.xx/syslinux-4.03.tar.bz2
tar xjvf syslinux-4.03.tar.bz2
cd syslinux-4.03
cp -pv ./core/pxelinux.0 ./com32/hdt/hdt.c32 ./memdisk/memdisk ./com32/menu/menu.c32 /zssd/tftpboot/
</pre>
* cd /zssd/tftpboot
<pre>
wget http://ladd00.triumf.ca/tftpboot/memtest86+-4.20.iso.zip
wget http://ladd00.triumf.ca/tftpboot/memtest86+-5.01.iso.gz
wget http://ladd00.triumf.ca/tftpboot/modules.alias
wget http://ladd00.triumf.ca/tftpboot/modules.pcimap
wget http://ladd00.triumf.ca/tftpboot/pci.ids
</pre>
* mkdir pxelinux.cfg
* emacs -nw pxelinux.cfg/default
<pre>
default menu.c32
prompt 0

menu title Welcome to the DSVSLICE PXE boot menu

timeout 50

label hdt
kernel hdt.c32

label memtest86+-5.01
kernel memdisk iso initrd=memtest86+-5.01.iso.gz

label memtest86+-4.20
kernel memdisk iso initrd=memtest86+-4.20.iso.zip

label vmlinuz-5.3.0-26-generic
menu default
kernel vmlinuz-5.3.0-26-generic
append initrd=initrd.img-5.3.0-26-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.1.1:/zssd/nfsroot/dsfe01 toram ip=dhcp panic=60 BOOTIF=enp1s0f0

#end
</pre>

=== setup linux kernel ===

* copy the kernel files
<pre>
cd /boot
rsync -av config* initrd* System.map* vmlinuz* /zssd/tftpboot/
</pre>
* cd /zssd/tftpboot
* chmod a+r *

=== setup nfs ===

* apt-get install nfs-kernel-server
* emacs -nw /etc/exports
<pre>
/zssd/nfsroot/dsfe01 dsfe01(rw,no_root_squash,async,no_subtree_check)
</pre>
* enable services
<pre>
systemctl enable nfs-server
systemctl enable nfs-mountd
systemctl enable nfs-idmapd
systemctl restart nfs-server
systemctl restart nfs-mountd
systemctl restart nfs-idmapd
</pre>
* after editing /etc/exports, run
<pre>
exportfs -av
</pre>

=== setup userland ===

* zfs create zssd/nfsroot
* zfs set dedup=verify zssd/nfsroot ### enable deduplication to save disk space because most linux images have mostly identical files
* clone ubuntu
<pre>
mkdir /zssd/nfsroot/dsfe01
cd /
rsync -avx . /zssd/nfsroot/dsfe01
</pre>
* edit config files:
* cd /zssd/nfsroot/dsfe01
* emacs -nw etc/hostname ### change to dsfe01
* emacs -nw etc/mailname ### change to dsfe01
* emacs -nw etc/yp.conf ### change daq00.triumf.ca to musr00.triumf.ca
* emacs -nw etc/defaultdomain ### change to MUSR-NIS
* cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
* emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
* emacs -nw root/.ssh/authorized_keys ### update root ssh keys
* emacs -nw etc/fstab ### add this
<pre>
192.168.1.1:/zssd/nfsroot/dsfe01 / nfs defaults,nolock 0 0
</pre>
* emacs -nw etc/chrony/chrony.conf
** comment-out all "pool" and "server" entries
** add entry "server 192.168.1.1 iburst"

After dsfe01 is booted:

* disable services:
<pre>
systemctl disable apache2
systemctl disable dnsmasq
systemctl disable zfs-import-cache
</pre>

To setup additional machines, clone dsfe01 instead of cloning the gateway machine

=== Allow manpages to be viewed ===

If <code>/</code> is mounted over NFS, <code>man</code> will report a permission error. Fix it with:

<pre>
ln -s /etc/apparmor.d/usr.bin.man /etc/apparmor.d/disable/
apparmor_parser -R /etc/apparmor.d/usr.bin.man
</pre>

== setup shared home directory ==

=== on the gateway machine ===
* define netgroups
* emacs -nw /etc/netgroup
<pre>
dsfe (dsfe01,,) (dsfe02,,)
</pre>
* emacs -nw /etc/nsswitch.conf ### edit the netgroup line to read:
<pre>
netgroup: files
</pre>
* export the home directories:
* emacs -nw /etc/exports ### add this:
<pre>
/zssd/home1 @dsfe(rw,no_root_squash,async,no_subtree_check)
</pre>
* exportfs -rc

=== on the frontend machine ===

* mkdir /home
* emacs -nw /etc/fstab ### add this:
<pre>
192.168.1.1:/zssd/home1 /home nfs defaults 0 0
</pre>
* mount -a

== setup NAT ==

NAT allows machines on the private network to connect to the internet: https://en.wikipedia.org/wiki/Network_address_translation

In these examples:
* replace "eno1" with name of the outgoing interface (the one connected to the TRIUMF network).
* replace "enp11s0" with name of the private network interface (192.168.1.x network)

* emacs -nw /etc/rc.local ### add this:
<pre>
# /etc/rc.local

/sbin/iptables -t nat -A POSTROUTING -o eno1 -j MASQUERADE
iptables -L -v

# uncomment following lines if machine has prohibitive FORWARD rules:
#/sbin/iptables -I FORWARD -i eno1 -o enp11s0 -m state --state RELATED,ESTABLISHED -j ACCEPT
#/sbin/iptables -I FORWARD -i enp11s0 -o eno1 -j ACCEPT
#iptables -L -v

iptables -L -v
sysctl -w net.ipv4.ip_forward=1
#sysctl -a | grep forward

sh /etc/firewall-rfc1918.sh

# end
</pre>
* emacs -nw /etc/firewall-rfc1918.sh
<pre>
# firewall-rfc1918.sh

# prevent RFC1918 private network IP addresses from
# going in and out from our uplink.

ETH=eno1

iptables -F in-rfc1918
iptables -N in-rfc1918
iptables -A in-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A in-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A in-rfc1918 --dst 192.168.0.0/16 -j REJECT
iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -I INPUT -j in-rfc1918 -i $ETH

iptables -F out-rfc1918
iptables -N out-rfc1918
iptables -A out-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A out-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A out-rfc1918 --dst 192.168.0.0/16 -j REJECT
iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -I OUTPUT -j out-rfc1918 -o $ETH

iptables -L -v

#end
</pre>

= KVM =

<pre>
apt install cpu-checker

root@daq13:~# kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
root@daq13:~#

(if not, shutdown, go into BIOS settings, enable CPU virtualization)

apt install virtinst ### will install many packages
apt install libvirt-clients libvirt-daemon-system-systemd libvirt-daemon qemu qemu-kvm libvirt-daemon-system virtinst bridge-utils

root@daq13:/home1/wheel# virsh list --all
Id Name State
------------------------------
1 ubuntu-guest running

apt install virt-manager

virt-install --name ubuntu-guest --os-variant ubuntu20.04 --vcpus 2 --ram 2048 --location /daq/daqstore/olchansk/linux/Ubuntu/ubuntu-20.04.3-desktop-amd64.iso --network bridge=virbr0,model=virtio --graphics none --extra-args='console=ttyS0,115200n8 serial'

virtual machine will start, boot, etc
to get out of it, CTRL + Shift followed by ]

ssh wheel@daq13
virt-manager

run virt-install again, omit "--graphics none", open graphics console from virt-manager, it booted into ubuntu installer desktop

virt-install --name test10 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --filesystem /kvm_ladd00,/ --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial" --graphics none

virt-install --name test14 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --disk /tmp/xxx/ladd00.img,bus=sata --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial rdshell" --graphics none --check path_in_use=off
</pre>

build image

<pre>
dd if=/dev/zero of=/tmp/xxx/ladd00.img bs=1024M count=20
mkfs.ext3 /tmp/xxx/ladd00.img ### ext4 fails to mount by SL6 kernel, "unknown ext4 options"
cd /kvm_ladd00/
mount -o loop /tmp/xxx/ladd00.img /mnt/tmp
rsync -av . /mnt/tmp/ --delete
umount /mnt/tmp
</pre>

on the guest, configure network: /etc/rc.local
<pre>
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

ifconfig eth2 192.168.122.2
route add -net 0.0.0.0 gw 192.168.122.1
ifconfig -a
netstat -rn

# end
</pre>

= ARM cross-compiler =

<pre>
apt install libgcc-9-dev-arm64-cross
apt install gcc-arm-linux-gnueabi
apt install gcc-arm-linux-gnueabihf
apt install g++-arm-linux-gnueabihf
apt install g++-arm-linux-gnueabi
</pre>

<pre>
arm-linux-gnueabi-gcc -o ttcp1 ttcp.c -march=armv7 -static
arm-linux-gnueabi-gcc -o memcpy.armv7 memcpy.cc -march=armv7 -static -O2
</pre>

= 32-bit intel cross-compiler =

Ubuntu 22.04:

<pre>
apt install libstdc++-11-dev:i386
apt install zlib1g-dev:i386
</pre>

for MIDAS, use "make linux32"

Ubuntu

2023-11-10T22:29:40Z

Lmartin: /* Convert from single to dual mirrored ZFS SSD */ Added simple partition table cloning option.

= About Ubuntu =

AAA

= Ubuntu version =

<pre>
lsb_release -a
uname -a
</pre>

= Ubuntu installer =

* updated for Ububtu LTS 20.04.01, 22.04.1

* download the latest Ubuntu LTS desktop installer iso image
* dd the image to a USB key
* power down, disconnect all disks (all HDDs, all SSDs, all M.2)
* connect the SSD to be used as system disk
* if system will use mirrored SSDs (using ZFS mirror), leave second SSD disconnected, we will activate it later
* power up
* boot from USB key in legacy mode or UEFI mode (select this in the BIOS boot menu - F8 for ASUS, F11 for Supermicro)
* follow the instruction:
* "try ubuntu or install ubuntu" - choose "install"
* select language - accept default
* "updates and other software" - accept default settings ("normal install")
* "installation type" - select "advanced features" and "experimental: use ZFS"
* accept partition choice
* "where are you?" - select "Vancouver" (PST time zone)
* "who are you?" - leave all fields blank, except "username" set to "wheel", "password" set to the root password. hostname will be set later after configuring the network
* installation runs in a few minutes, when finished, reboot
* login as user wheel
* answer annouying questions:
* "livepatch" - say "next"
* "help improve" - select "do not send", say "next"
* "privacy" - leave "location" as "off", say "next"
* "ready to go", say "done"
* right-click on the desktop, say "open in terminal", a shell will open
* say "sudo /bin/bash", enter the root password, you now have the root shell
* run nm-connection-editor to configure the network. use netmask 255.255.224.0, gateway 142.90.100.18, DNS 142.90.100.19, search path "triumf.ca"
* after network is up (can ping ladd00), continue with post-installation steps below

= Install instructions =

== prepare ==
<pre>
apt update
apt upgrade
</pre>

== install ssh ==
<pre>
apt install ssh
</pre>

== configure hostname ==
<pre>
vi /etc/hostname
</pre>

== disable swap ==

ubuntu installer creates a 2 GB swap partition, not useful
on 32-64 GB machine, disable it:
<pre>
vi /etc/fstab ### comment out the "swap" line
</pre>

== maybe reboot ==

this is a good point to reboot the machine to boot
the latest kernel and to set the correct hostname

== install etckeeper ==

keep contents of /etc in a git repository:

<pre>
apt -y install etckeeper
</pre>

== set timezone ==
<pre>
timedatectl list-timezones | grep -i vancouver
timedatectl set-timezone America/Vancouver
</pre>

== install time synchronization ==
<pre>
apt -y install chrony
echo server time1.triumf.ca iburst >> /etc/chrony/chrony.conf
echo server time2.triumf.ca iburst >> /etc/chrony/chrony.conf
echo server time3.triumf.ca iburst >> /etc/chrony/chrony.conf
systemctl disable systemd-timesyncd.service
systemctl stop systemd-timesyncd.service
systemctl disable ntp
systemctl stop ntp
systemctl enable chrony
systemctl restart chrony
chronyc sources
chronyc tracking
</pre>

== reenable systemd-timesyncd ==

ONLY IF CHRONY DOES NOT WORK

<pre>
apt remove chrony
systemctl enable systemd-timesyncd.service
systemctl restart systemd-timesyncd.service
systemctl status systemd-timesyncd.service
timedatectl status
timedatectl timesync-status
</pre>

== enable outgoing email (debian 11) ==

this is different from ubuntu 20. it uses /etc/mailname and it hardwires the hostname into main.cf.

== enable outgoing email ==

* TRIUMF: use smtp.triumf.ca
* CERN: use cernmx.cern.ch

<pre>
apt install postfix ### select "satellite system", enter full hostname "xxx.triumf.ca", enter "smtp.triumf.ca"
apt install mailutils
dpkg-reconfigure postfix ### (if postfix already installed)
</pre>
<pre>
echo olchansk@triumf.ca lindner@triumf.ca bsmith@triumf.ca >> ~root/.forward
mailx root
test
^D
</pre>

== enable ping for all users (debian 11) ==

Without this tweak, Debian will report "operation not permitted" if a user tries to ping somewhere.

<pre>
echo 'net.ipv4.ping_group_range = 0 1000' > /etc/sysctl.d/99-ping.conf
</pre>

== install missing packages ==

(apt eats terminal input, even the "yes |" trick does not quite work,
repeat the following commands until they report that everything
is installed)

<pre>
yes | apt -y install ssh tcsh ethtool ncat rsync strace net-tools sysstat smartmontools lm-sensors traceroute time minicom screen git lsof debsums tmux
yes | apt -y install lsb-release
yes | apt -y install flex bison
yes | apt -y install neofetch
yes | apt -y install snmp snmp-mibs-downloader
yes | apt -y install git subversion g++ gfortran cmake doxygen
yes | apt -y install python
yes | apt -y install curl libcurl4 libcurl4-openssl-dev
yes | apt -y install mariadb-client ### mysql client
yes | apt -y install libz-dev sqlite3 libsqlite3-dev unixodbc-dev
yes | apt -y install libssl-dev
yes | apt -y install emacs xemacs21 joe
yes | apt -y install gnuplot dos2unix
yes | apt -y install mutt bsd-mailx # email clients
yes | apt -y install liblz4-tool pbzip2
yes | apt -y install libc6-dev-i386 # otherwise no /usr/include/sys/types.h
yes | apt -y install libreadline-dev
yes | apt -y install chromium-browser chromium-codecs-ffmpeg-extra
yes | apt -y install ubuntu-mate-themes
yes | apt -y install libmotif-dev libxmu-dev
yes | apt -y install libusb-dev libusb-1.0-0-dev
yes | apt -y install xfig gsfonts-x11 gsfonts-other # install fonts for xfig
yes | apt -y install libjson-perl
yes | apt -y install libgsl-dev # additional GNU Scientific Library
yes | apt -y install qt5-default # Qt development
yes | apt -y install python3.8-full python3.8-dev python3.8-dbg python3-pip ### for pyROOT
yes | apt -y install imagemagick imagemagick-common ckeditor # for elog
yes | apt -y install libjpeg-dev libjpeg-progs libjpeg-tools
yes | apt -y install linux-tools-common linux-tools-generic # cpupower frequency-info
yes | apt -y install rdesktop remmina remmina-plugin"*" # requested by POL
</pre>

Ubuntu LTS 20.04:
<pre>
yes | apt -y install linux-image-generic-hwe-20.04 linux-tools-virtual-hwe-20.04 # enable linux 5.11 series kernel
</pre>

== install git/scripts ==
<pre>
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd scripts
git pull
</pre>

== disable swap (debian 11) ==

* on 64 GB RAM machines swap is not useful
* on machines booted from network (NFS-ROOT), swap does not work
* on machines running from flash (RPi, etc), flash is too slow for useful swap
* swap configured by linux installers invariably has wrong size and is not useful

<pre>
systemctl disable dphys-swapfile
systemctl stop dphys-swapfile
dphys-swapfile uninstall
</pre>

== configure DNS ==

<pre>
cd ~/git/scripts
git pull
mkdir /etc/systemd/resolved.conf.d
cp etc/resolved-triumf.conf /etc/systemd/resolved.conf.d/
systemctl restart systemd-resolved
resolvectl
#systemd-analyze cat-config systemd/resolved.conf
</pre>

== install ganglia ==

<pre>
yes | apt-get -y install ganglia-monitor
systemctl enable ganglia-monitor
</pre>
<pre>
cd ~root/git/scripts
git pull
cp etc/gmond-ubuntu.conf /etc/ganglia/gmond.conf
ln -s ganglia/gmond.conf /etc
# ln -s arm-linux-gnueabihf/ganglia /usr/lib/ganglia ### fix path for ARM CPU
# ln -s i386-linux-gnu/ganglia /usr/lib/ganglia ### fix path for 32-bit Intel CPU
systemctl restart ganglia-monitor
systemctl status ganglia-monitor
ps -efw | grep gmond
</pre>
<pre>
cd ~root/git/scripts/ganglia
make install
./ganglia-all.perl
</pre>

== install gonodeinfo ==

* go to https://bitbucket.org/dd1/gonodeinfo follow instructions:
<pre>
yes | apt-get -y install golang
mkdir ~/git
cd ~/git
git clone https://bitbucket.org/dd1/gonodeinfo.git
cd gonodeinfo
git pull
make
make install # install gonodeinfo agent
cd ~ # this is important
</pre>
* edit /etc/gonodeinfo.conf
* change "Description", "Location", "User" and "Administrator" as appropriate (or delete them)
* change "Servers" to read: Servers: daq00.triumf.ca:8601
* run "gonodeinfo -v"
* if error is "connection refused". go to the nodeinfo server to add this client to the access control list:
* on the gonodeinfo server: run /opt/gonodeinfo/gonodereceive.exe -a daq13
* try gonodeinfo again, there should be no error
* on the gonodeinfo server: run gonodereport, look at the web pages, the new machine should be listed now

== install fonts for EPICS ==

* apt install xfonts-100dpi xfonts-75dpi
* restart Xorg (i.e. "killall Xorg", this will log you out from the console)
* xlsfonts | grep -i helvetica ### should show fonts with different sizes, not just size 0 (scalable)

== install libz.so.1 for CentOS compatibility ==

KO - confirm which versions on quartus need this.

<pre>
yes | apt-get -y install zlib1g
yes | apt-get -y install zlib1g:i386 libc6:i386 libgcc1:i386 gcc-6-base:i386
</pre>

== install libpng12.so.0 for Quartus compatibility ==

(does not work anymore!!!)

<pre>
wget http://ftp.ca.debian.org/debian/pool/main/libp/libpng/libpng12-0_1.2.50-2+deb8u2_amd64.deb
dpkg --install libpng12-0_1.2.50-2+deb8u2_amd64.deb
</pre>

== install libpng12.so.0 for Quartus 13.0sp1 ==

<pre>
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0.50.0
/bin/cp -pv libpng12.so.0 libpng12.so.0.50.0 /lib/x86_64-linux-gnu/
</pre>

== install packages for Xilinx ==

ubuntu LTS 22.04 vivado 2020.1

<pre>
apt install autoconf libtool
apt install libtinfo5
apt install texinfo
apt install zlib1g:i386
</pre>

== install packages for building ROOT ==

<pre>
apt -y install libx11-dev libxpm-dev libxft-dev libxext-dev libpng-dev libjpeg-dev xlibmesa-glu-dev libxml2-dev libgsl-dev cmake
</pre>

== install 32-bit libraries for PHYSICA ==

these instructions are for running 32-bit physica executable built for SL6 on ubuntu LTS 20.04

install physica sources (cannot build, do not have g77)

<pre>
cd ~/packages
git clone https://bitbucket.org/ttriumfdaq/physica.git
</pre>

install 32-bit libraries using ubuntu package manager:

<pre>
apt install lib32z1 # libz.so
</pre>

copy 32-bit SL6 shared libraries to /lib32

<pre>
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libX11.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libgd.so.2 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libpng12.so.0 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libreadline.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libncurses.so.5 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libg2c.so.0 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libxcb.so.1 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libXpm.so.4 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libjpeg.so.62 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libfontconfig.so.1 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libfreetype.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libtinfo.so.5 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libXau.so.6 /lib32/
root@trinatdaq:~# cp /daq/daqstore/olchansk/daq/physica-SL6/libexpat.so.1 /lib32/
</pre>

ldd should report:

<pre>
trinatdaq:trinat> ldd /usr/local/physica/physica.exe
linux-gate.so.1 (0xf7fa2000)
libX11.so.6 => /lib32/libX11.so.6 (0xf7e43000)
libgd.so.2 => /lib32/libgd.so.2 (0xf7dfe000)
libpng12.so.0 => /lib32/libpng12.so.0 (0xf7dd6000)
libz.so.1 => /lib32/libz.so.1 (0xf7db8000)
libreadline.so.6 => /lib32/libreadline.so.6 (0xf7d7e000)
libncurses.so.5 => /lib32/libncurses.so.5 (0xf7d5b000)
libg2c.so.0 => /lib32/libg2c.so.0 (0xf7d3d000)
libm.so.6 => /lib32/libm.so.6 (0xf7c39000)
libgcc_s.so.1 => /lib32/libgcc_s.so.1 (0xf7c1a000)
libc.so.6 => /lib32/libc.so.6 (0xf7a2f000)
libxcb.so.1 => /lib32/libxcb.so.1 (0xf7a05000)
libdl.so.2 => /lib32/libdl.so.2 (0xf79ff000)
libXpm.so.4 => /lib32/libXpm.so.4 (0xf79ee000)
libjpeg.so.62 => /lib32/libjpeg.so.62 (0xf7997000)
libfontconfig.so.1 => /lib32/libfontconfig.so.1 (0xf7962000)
libfreetype.so.6 => /lib32/libfreetype.so.6 (0xf78c9000)
libtinfo.so.5 => /lib32/libtinfo.so.5 (0xf78b0000)
/lib/ld-linux.so.2 (0xf7fa4000)
libXau.so.6 => /lib32/libXau.so.6 (0xf78ad000)
libexpat.so.1 => /lib32/libexpat.so.1 (0xf7885000)
trinatdaq:trinat>
</pre>

set login environment:

<pre>
setenv TRIUMF_FONTS $HOME/packages/physica/fonts
setenv PHYSICA_DIR $HOME/packages/physica
alias physica $PHYSICA_DIR/physica-SL6-32
</pre>

test:

<pre>
cd ~/packages/physica
physica
@rangauss.pcm
</pre>

== install lightdm ==

unlike the default gdm login manager, lightdm shows the machine hostname and does not require an extra mouse click to swicth from screen saver to login mode.

<pre>
apt -y install lightdm
# select lightdm
</pre>

== install desktop environments ==

note: default display manager and default desktop are deficient, please do not skip this step.

note: if apt asks to choose the display manager, select "lightdm"

note: KO - I recommend the "MATE" desktop.

note: you will have to cut-and-paste this several times because "apt" eats commands, even with "-y" and even piped from "yes".

<pre>
# install MATE desktop
DEBIAN_FRONTEND=noninteractive apt -y install ubuntu-mate-core ubuntu-mate-desktop ubuntu-mate-themes
# install Cinnamon desktop
DEBIAN_FRONTEND=noninteractive apt -y install cinnamon
# install KDE desktop
DEBIAN_FRONTEND=noninteractive apt -y install kubuntu-desktop
# install Lxqt desktop
DEBIAN_FRONTEND=noninteractive apt -y install lxqt
# install Xfce4 desktop
DEBIAN_FRONTEND=noninteractive apt -y install xfce4
</pre>

== install ROOT ==

Please install ROOT per instructions at http://root.cern.ch.

NOTE1: The ROOT package available from Ubuntu repositories is severely out of date and cannot be used with MIDAS and ROOTANA. ### DO NOT DO THIS! apt-get install root-system

NOTE2: as of 2017-Jan-09, ROOT binary kits for Ubuntu do not work (use GCC 5 instead of GCC6), build from source instead.

== Install x2go ==

KO - is this still needed? does it cause any security problems?

x2go instructions, thanks to Art O.

<pre>
add-apt-repository ppa:x2go/stable
apt-get update
apt-get install x2goserver x2goserver-xsession
</pre>

== enable root login from ladd00/daq00 ==
<pre>
ssh localhost
CTRL-C
/bin/cp ~root/git/scripts/etc/authorized_keys ~root/.ssh/
</pre>

== install smart-status ==
<pre>
ln -s ~/git/scripts/smart-status/smart-status.perl ~root/
</pre>

== enable boot menu and boot messages ==

This will enable the grub menu (with a 10 sec timeout) and
replace black screen with exciting linux boot messages.

* emacs -nw /etc/default/grub
<pre>
GRUB_DEFAULT=0
#GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="vga=769 video=640x480"
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX=""
#GRUB_GFXMODE=640x480
</pre>
* update grub config:
<pre>
grub-mkconfig -o /boot/grub/grub.cfg
</pre>

== reboot ==

this completes installation of the base system.

following sections modify basic ubuntu to fix known problems and to enable special stuff.

= Enable automatic updates =

<pre>
apt install unattended-upgrades
cd ~/git/scripts
git pull
/bin/cp -v etc/99apt-conf-ko /etc/apt/apt.conf.d/
apt-config dump | grep Unattended
</pre>

Following is obsolete:

* emacs -nw /etc/apt/apt.conf.d/50unattended-upgrades
** uncomment in Allowed-Origins "-security" and "-updates"
** add in Allowed-Origins: "Google LLC:stable";
** uncomment/add: "Unattended-Upgrade::Mail "root";
* emacs -nw /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
</pre>
* test: unattended-upgrade --dry-run -v

NOTE: update-on-shutdown is disabled.

NOTE: there is no update-on-boot, but:

NOTE: if machine was off for a long time, the systemd update timer would have expired and it will fire soon after reboot, causing an automatic update run. this is unwanted, and there is no fix or workaround for it. K.O. June-2023.

= Fix bpool is full =

!!! only if ROOT on ZFS !!!

There is an error in the zsys package that causes bpool to run out of space,
see [[#Ubuntu zsys]] for more details.

To fix:
<pre>
cd ~/git/scripts
git pull
cp etc/zsys.conf /etc/
zsysctl service reload
zsysctl service gc
zpool list bpool
zfs list bpool
df /boot
</pre>

= IPMI instructions =

IPMI is the board management hardware on Supermicro and other server motherboards. This includes hardware sensors - fan rotation speed, temperatures and power supply voltages.

<pre>
apt-get install ipmitool
systemctl enable ipmievd
systemctl restart ipmievd
</pre>

Run:
* ipmitool sel list ### event list
* ipmitool sel elist ### event list
* ipmitool sel clear ### clear event list (if it becomes full)
* ipmitool sensor ### report hardware sensors

= move /home/wheel =

note: this MUST be done if ZFS root and NIS/autofs with /home.

Default location of wheel's home directory will collide with autofs /home, it has to be moved,
for example to /wheel.

<pre>
# logout from the wheel user
# go to another computer
ssh root@daqubuntuxxx
zfs list | grep wheel ### identify zfs name wheel_xxxxxx
zfs set mountpoint=/wheel rpool/USERDATA/wheel_hm8fzh
emacs -nw /etc/passwd ### change wheel's home directory from /home/wheel to /wheel
su - wheel ### check that user wheel still works
</pre>

TL edit (April 2023): I think also need to do chown wheel.wheel /wheel

= enable NIS (ubuntu 22.04, debian 11) =

<pre>
apt -y install rpcbind nis
echo DAQ-NIS >> /etc/defaultdomain
echo ypserver daq00.triumf.ca >> /etc/yp.conf
systemctl enable ypbind.service
systemctl restart ypbind.service
systemctl status ypbind.service
ypwhich -m
</pre>

enable ypserv:

<pre>
sed -i s/NISSERVER=false/NISSERVER=slave/ /etc/default/nis
/usr/lib/yp/ypinit -s daq00
echo ypserver localhost >> /etc/yp.conf
sed -i "s/ypserver .*/ypserver localhost/" /etc/yp.conf
systemctl enable ypserv
systemctl restart ypserv
systemctl restart ypbind
</pre>

edit /etc/nsswitch.conf to read:

<pre>
# begin get data from nis
passwd: files nis
group: files nis
shadow: files nis
automount: files nis
netgroup: files nis
# end get data from nis
</pre>

enable hourly update of nis maps:

<pre>
mkdir ~root/git
cd ~root/git
git clone http://daq00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>

If this is a new machine, then on the master NIS node (daq00), add this new node to /etc/netgroup, and update NIS maps (cd /var/yp; make)

= enable NIS (ubuntu 20.04) =

* apt-get -y install portmap nis ### will ask for NIS domain (DAQ-NIS)
* dpkg-reconfigure nis ### reconfigure if already installed
* ypwhich -m
* edit /etc/default/nis
** set "NISSERVER=slave"
** Ubuntu LTS 20.04, check that "YPBINDARGS=" is blank, remove "-no-dbus" if it is there
* #edit /etc/yp.conf, comment-out everything, add "domain DAQ-NIS server localhost"
* edit /etc/yp.conf, comment-out everything, add "ypserver localhost"
* /usr/lib/yp/ypinit -s daq00
* systemctl enable nis
* systemctl restart nis
* ypwhich
* ypwhich -m
* ypcat -k passwd
* vi /etc/nsswitch.conf ### add the automount line, modify the passwd, group and shadow lines to read this:
<pre>
# begin get data from nis
passwd: files nis
group: files nis
shadow: files nis
automount: files nis
netgroup: files nis
# end get data from nis
</pre>
* enable hourly update of NIS maps
<pre>
mkdir ~root/git
cd ~root/git
git clone http://ladd00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>
* ### NOT NEEDED sudo vi /etc/idmapd.conf ### add line: "Domain = triumf.ca"

= enable autofs =

<pre>
apt -y install autofs
systemctl enable autofs
systemctl restart autofs
ls -l /home/olchansk ### test autofs, check file owner is correct
</pre>

= enable NFS server =

<pre>
apt install nfs-kernel-server
#edit /etc/exports
systemctl enable nfs-server
systemctl restart nfs-server
</pre>

= NIS master =

notes for setting up the NIS master

== wheel user ==

"wheel" is the default administrative user. We do not want it's password exported to NIS (encrypted password hash is world visible) and we do not want it's home directory exported to NFS (~wheel/.ssh is world visible and potentially writable: anybody can change ~wheel/.ssh/authorized_keys).

* move wheel's home directory from /home/wheel to /wheel (see special section about this)
* change wheel's UID and GID from 1000 to a value below MINUID in /var/yp/Makefile

== coherent uids ==

we do not want system accounts defined in /etc/passwd of the NIS master
to be included in the NIS map "passwd". this causes trouble on NIS clients
where newly installed packages fail to create local system users because same
user already exists in NIS.

This is controlled by MINUID in /var/yp/Makefile.

Historical TRIUMF uids start from around 200, but several clusters do not have any historic TRIUMF uids below 500 and MINUID is set to:
* DAQ-NIS: MINUID=200
* ISAC-NIS: MINUID=500
* TITAN-NIS: MINUID=500
* MUSR-NIS: MINUID=500
* TIG-NIS: MINUID=500 (100 on SL6 mother8pi)

Ubuntu 20 has two programs to create users:
* adduser - creates new users with UID 1000 and up as specified in /etc/adduser.conf. No problems here.
* adduser --system - creates new system users with UID 100 and up as specified in /etc/adduser.conf. No problems here.
* useradd - creates new users with UID 1000 and up as specified in /etc/login.defs. No problems here.
* useradd --system - creates new system users with UID 999 and down (read "man useradd", section at the end about SYS_UID_MAX). This collides with NIS MINUID, these system users will be included in the NIS map and cause trouble.

This problem cannot be fixed, SYS_UID_MIN, SYS_UID_MAX and UID_MIN in /etc/login.defs do not seem
to have any effect on UIDs chosen by "useradd --system". (tested on Ubuntu LTS 20.04).

So far only these system accounts seem to be affected by this:
* systemd-coredump
* ganglia

To fix:
* run "sort -r -n -t: -k3 /etc/passwd" to identify the last unused system user uid (range 100..200)
* run "sort -r -n -t: -k3 /etc/group" to identify the last unused system user gid (range 100.200)
* systemd-coredump: manually change UID and GID (package systemd-coredump is usually not installed)
* ganglia: same thing, then change ownership on all ganglia files.

Also read systemd author's opinion on system vs user UIDs:
https://github.com/systemd/systemd/issues/4850#issuecomment-265698275

= Fix systemd-logind NIS breakage =

!!! THIS IS NOT NEEDED FOR UBUNTU LTS 20.04 !!!

there is a delay in ssh logins for normal users. "ssh -v" shows the delay is after "pledge...". this
fix removes the delay.

systemd developers think that we should not use NIS and made sure there are
problems if we do. To give them credit, they do offer a workaround. Read this:
https://github.com/poettering/systemd/commit/695fe4078f0df6564a1be1c4a6a9e8a640d23b67

<pre>
mkdir /etc/systemd/system/systemd-logind.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-logind.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-logind.service
</pre>

= Fix systemd-udevd NIS breakage =

see same problem as above with udev getting stuck. ubuntu lts 20.04.

<pre>
mkdir /etc/systemd/system/systemd-udevd.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-udevd.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-udevd.service
</pre>

= Configure USB device permissions =

Configure USB device permissions for user access to USB-serial devices, Altera USB Blaster, etc.

* create file /etc/udev/rules.d/99-usb-chmod.rules with this contents:

<pre>
emacs -nw /etc/udev/rules.d/99-usb-chmod.rules
ACTION=="add", SUBSYSTEM=="usbmisc", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /dev/%c"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /proc/%c"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVICE}"
ACTION=="add", ENV{PHYSDEVBUS}=="usb-serial", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVPATH}=="/class/tty/ttyS*", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyUSB*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyACM*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyS*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", DEVPATH=="*video*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
</pre>

* reload udev rules: udevadm control --reload-rules
* apply new permissions: udevadm trigger --action=add
* watch udev activity: udevadm monitor -p

= Configure lightdm display manager =

* enable it
<pre>
echo lightdm | dpkg-reconfigure -fteletype lightdm
systemctl disable gdm
systemctl disable sddm
systemctl enable lightdm
</pre>

* make the MATE desktop as default
<pre>
cd ~root/git/scripts/
git pull
/bin/cp -v etc/lightdm_default_mate.conf /etc/lightdm/lightdm.conf.d/
</pre>

* enable login by NIS users
<pre>
/bin/cp -v etc/lightdm_enable_nis_login.conf /etc/lightdm/lightdm.conf.d/
</pre>

* restart lightdm
<pre>
systemctl stop gdm
systemctl restart lightdm
</pre>

= Install libpng12.so.0 =

Quartus 16 needs libpng12:

<pre>
wget http://mirrors.kernel.org/ubuntu/pool/main/libp/libpng/libpng12-0_1.2.54-1ubuntu1_amd64.deb
dpkg --install libpng12-0_1.2.54-1ubuntu1_amd64.deb
</pre>

= Install google-chrome =

<pre>
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
dpkg -i google-chrome-stable_current_amd64.deb
</pre>

confirm autoupdate is enabled, observe dl.google.com is present in the list of repositories:
<pre>
apt update
...
Get:5 https://dl.google.com/linux/chrome/deb stable/main amd64 Packages [1,094 B]
...
</pre>

FOLLOWING IS OBSOLETE:

Instructions from here:
https://www.ubuntuupdates.org/ppa/google_chrome?dist=stable

<pre>
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-tmp.list'
apt update
apt install google-chrome-stable
/bin/rm -f /etc/apt/sources.list.d/google-tmp.list
</pre>

= Install amanda client =

ONLY ONE MACHINES THAT HOST HOME DIRECTORIES

* apt install amanda-client
* edit /etc/amandahosts
<pre>
amanda.triumf.ca amanda amdump
</pre>
* check permissions on /etc/amandahosts:
<pre>
root@daq00:/var/log/amanda# ls -l /etc/amandahosts
-rw------- 1 backup backup 49 Jan 27 10:48 /etc/amandahosts
</pre>
* fix if needed: chown backup.backup /etc/amandahosts; chmod a= /etc/amandahosts; chmod u=wr /etc/amandahosts
* edit /etc/amanda-security.conf, add this line:
<pre>
runtar:gnutar_path=/usr/bin/tar
</pre>

On the amanda machine:

* in amanda disklist, use dump type "bsdtcp-comp-user-tar"
* su - amanda and run amcheck -c daily daq00
<pre>
-bash-4.1$ amcheck -c daily daq00

Amanda Backup Client Hosts Check
--------------------------------
Client check: 1 host checked in 0.092 seconds. 0 problems found.

(brought to you by Amanda 3.3.7p1.git.685ff76d)
</pre>

= Enable rc.local =

For reasons unknown, Ubuntu LTS 20.04 does not enable /etc/rc.local. Do this:

<pre>
cd ~/git/scripts
git pull
cp -n -v etc/rc.local /etc/
chmod a+rx /etc/rc.local
cp etc/rc-local.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable rc-local
systemctl start rc-local
systemctl status rc-local
</pre>

= Remove unwanted packages =

<pre>
apt remove zsys
apt remove sddm
</pre>

= Disable unwanted services =

<pre>
systemctl disable mpd
systemctl disable snapd
systemctl disable ModemManager
systemctl --global mask tracker-extract-3.service
systemctl --global mask tracker-miner-fs-3.service
systemctl daemon-reload
</pre>

= Disable sleep and suspend =

note: we see some computers randomly shutdown or go to sleep, log files indicates the "sleep" or "suspend" button was pushed by user, but no such buttons actually exist. this is the fix for this:

<pre>
systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target systemd-suspend.service systemd-hybrid-sleep.service
</pre>

= Enable crontab @reboot for MIDAS =

startup scripts have a bug - cron @reboot entries for normal users can run before autofs is ready, so if the home directory is on autofs/NFS, it cannot be accessed and the cron job fails. If MIDAS is supposed to be started by cron @reboot, it will not start (there *will* be an error message in /var/log/cron).

<pre>
mkdir /etc/systemd/system/cron.service.d
echo -e "[Unit]\nAfter=ypbind.service autofs.service\n" > /etc/systemd/system/cron.service.d/local.conf
systemctl daemon-reload
systemctl cat cron.service
</pre>

Explore the systemd dependency tree using "systemctl list-dependencies" maybe with "--all".

Visualize the exact boot sequence from previous boot: "systemd-analyze plot > xxx.svg", look at the svg file using a web browser.

Crontab entry to start midas: (install in the midas user crontab, not root crontab)

<pre>
su - midasuser
crontab -l
#@reboot /bin/bash -l -c "/home/trinat/bin/start-daq-applications"
#@reboot /bin/tcsh -c "/home/trinat/bin/start-daq-applications"
</pre>

= Install apache httpd proxy for midas and elog =

This will configure the HTTPS/SSL certificate using "certbot" and "letsencrypt" and configure an HTTPS web server using apache2.

First, configure apache2:

* execute these commands:
<pre>
apt -y install apache2
cd /etc/apache2
</pre>
* create new file conf-available/ssl-daq14.conf # use actual hostname instead of daq14
<pre>
SSLSessionCache shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout 300
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
SSLCryptoDevice builtin
</pre>
* create new file sites-available/daq14-ssl.conf # use actual hostname instead of daq14
<pre>
<IfModule mod_ssl.c>
<VirtualHost *:443>
ServerName daq14.triumf.ca
DocumentRoot /var/www/html
ErrorLog /var/log/apache2/daq14.log
SSLEngine on
# note SSLProtocol, SSLCipherSuite and some other settings are overwritten by /etc/letsencrypt/options-ssl-apache.conf
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SEED:!IDEA:!RC4
## use port specified in elogd.cfg
#ProxyPass /elog/ http://localhost:8082/ retry=1
## use mhttpd port
#ProxyPass / http://localhost:8080/ retry=1
Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"
<Location />
SSLRequireSSL
AuthType Basic
AuthName "DAQ password protected site"
Require valid-user
# create password file: touch /etc/apache2/htpasswd
# to add new user or change password: htpasswd /etc/apache2/htpasswd username
AuthUserFile /etc/apache2/htpasswd
</Location>
</VirtualHost>
</IfModule>
</pre>
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* enable ssl module
* enable new configurations
<pre>
a2enmod ssl
a2enmod headers
a2enmod proxy
a2enmod proxy_http
a2enconf ssl-daq14
a2ensite daq14-ssl
</pre>
* disable default ssl sites
<pre>
a2dissite 000-default-le-ssl
a2dissite 000-default
ls -l /etc/apache2/sites-enabled/ ### should show only daq14-ssl.conf
</pre>
* check that there are no syntax problems
<pre>
apache2ctl configtest
</pre>
* enable and start apache2:
<pre>
systemctl enable apache2
systemctl restart apache2
systemctl status apache2
</pre>
* apache2 may fail to start, look in /var/log/apache2/error.log and /var/log/apache2/daq14.log
* if it says "Failed to configure ... certificate", proceed to the step for setting certbot.
* try to access https://daq14.triumf.ca
** you should see a complaint about self-signed certificate
** you should see a request for password (do not login yet)
** if you get "connection refused", HTTPS port 443 may need to be enabled in the local firewall, look at documentation for ufw.
Second, configure certbot:

(Note: as of 2018-01-18 certbot requires use of http port 80 to get the initial https certificate,
renewal can continue to use the https port 443)

(Note: as of 2019-01-?? certbot requires use of port 80 for renewals)

* check that port 80 is not used by anything:
* netstat -an | grep LISTEN | grep ^tcp | grep 80
* lsof -P | grep -i tcp | grep LISTEN | grep 80
* if lsof reports that apache2 is listening on port 80, follow the apache2 instructions above (remove "listen 80" from apache2.conf

* install certbot (if necessary open tcp port 80 in the firewall, see documentation for ufw):
<pre>
apt install certbot python3-certbot-apache
certbot certonly --standalone --installer apache
</pre>
* then answer questions:
* "activate HTTPS for daq14.triumf.ca" - say ok
* "enter email address" - enter your own email address
* "please read terms..." - read the terms and say "agree"
* it will take a few moments...
* "congratulations..." - say ok.
<pre>
certbot install --apache --cert-name daq14.triumf.ca
</pre>
* then answer questions:
* "choose redirect..." - say "1" (no redirect)
* look inside /etc/apache2/sites-enabled/daq14-ssl.conf to see that SSLCertificateFile & co point to certbot certificates in
/etc/letsencrypt/live/daq14.triumf.ca/
* to check current renewal and to update the certbot config file in /etc/letsencrypt/renewal, run this:
<pre>
certbot renew --standalone --installer apache --force-renewal
</pre>

NOTE: this certificate will expire in 3 months, automatic renewal should work with current version of certbot

Third, activate password protection:

* as shown in the config file above, create password file and initial user: (replace "midas" with specific username)
<pre>
touch /etc/apache2/htpasswd
htpasswd /etc/apache2/htpasswd midas
</pre>

* restart apache2
<pre>
systemctl restart apache2
systemctl status apache2
</pre>

From here:
* enable proxy for MIDAS mhttpd - uncomment redirect in the config file above
* enable proxy for ELOG - ditto
<pre>
a2enmod proxy
a2enmod proxy_http
apache2ctl configtest
systemctl restart apache2
</pre>

From here:
* enable proxy for MIDAS mhttpd - uncomment redirect in the config file above
* enable proxy for ELOG - ditto
<pre>
a2enmod proxy
a2enmod proxy_http
apache2ctl configtest
systemctl restart apache2
</pre>
* try accessing MIDAS https://daq14.triumf.ca/ (make sure mhttpd is running)
* if it's not working, check odb setting FIXME!
* try accessing ELog https://daq14.triumf.ca/elog/ (make sure elogd is running)
* if it's not working, check elogd.cfg file and make sure
<pre>
SSL = 0
</pre>

NOTE: if certbot fails with errors about 'module' object has no attribute 'pyopenssl',
try this: pip install requests==2.6.0

= Enable elog PDF preview =

see https://stackoverflow.com/questions/52998331/imagemagick-security-policy-pdf-blocking-conversion

* xemacs -nw /etc/ImageMagick-6/policy.xml
* remove this section at the end:
<pre>

<policy domain="coder" rights="none" pattern="PS" />
<policy domain="coder" rights="none" pattern="PS2" />
<policy domain="coder" rights="none" pattern="PS3" />
<policy domain="coder" rights="none" pattern="EPS" />
<policy domain="coder" rights="none" pattern="PDF" />
<policy domain="coder" rights="none" pattern="XPS" />
</pre>

= Install Jupyter notebook =

<pre>
From https://jupyter.org/install
apt install python3-pip
pip install jupyterlab
pip install notebook
~/.local/bin/jupyter notebook
watch the http://localhost:8888 URL that it printed
say "no" to offer to start firefox (it will not work!)
URL is: http://localhost:8888/tree?token=xxx
from the machine where you are running the web browser (i.e. google-chrome), run (replace trinat@trinatdaq with the username and machine name where you started jupyter)
open a new shell and run: ssh -v trinat@trinatdaq -L 8888:localhost:8888
in the web browser, open http://localhost:8888
this gives us the login page
in the password or token entry field, put the token from the "tree?token=xxx" above (printed by jupyter on startup)
push button "login"
jupyter page should open with the list of files in the trinat home directory
congratulate Brian with full success
</pre>

= Install ZFS quota report =

If there are any ZFS volumes, install script to report disk and quota usage

<pre>
cd ~/git/scripts/quotareport
git pull
mkdir /var/www/html/zfsquotareport
cp -pv ~/git/scripts/quotareport/sorttable.js /var/www/html/zfsquotareport/
ln -s $PWD/zfsquotareport.perl /etc/cron.daily/
touch /etc/crontab
</pre>

If httpd is configured to redirect "/" to MIDAS mhttpd:
* add following to /etc/apache2/sites-enabled/xxx-ssl.conf in front of "ProxyPass / ..."
* run "systemctl reload apache2"
<pre>
## do not proxy zfs quota report directory
ProxyPass /zfsquotareport/ !
</pre>

= Install PHP =

* apt install php libapache2-mod-php
* systemctl restart apache2
* create /var/www/html/info.php
<pre>
<?php

phpinfo();
</pre>
* open https://daq00.triumf.ca/info.php

= Configure TRIUMF printers =

<pre>
systemctl stop cups
systemctl disable cups
echo "ServerName printers.triumf.ca" > /etc/cups/client.conf
lpstat -a
</pre>

= Enable core dumps =

By default, Ubuntu LTS 20.04 installs the apport package
which disabled core dumps from user applications. (google it up!).
It is not meant to do this and documentation claims that
it is not installed and not enabled by default. Oh, well...

<pre>
apt remove apport
apt autoremove ### will remove apport-symptoms and a few other packages
</pre>

After this, core dumps are written to file "core" in the current directory.
See /proc/sys/kernel/core_pattern and /proc/sys/kernel/core_uses_pid.

= Enable debugger =

By default, Ubuntu LTS 20.04 does not permit debugger to attach and debug
already running programs. To enable it, add following to /etc/rc.local

<pre>
echo 0 > /proc/sys/kernel/yama/ptrace_scope
</pre>

= Disable Ubuntu Pro nag =

If "apt upgrade" requests Ubuntu Pro or esm-apps, disable the nag:
<pre>
/bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf
</pre>

= Update packages =

* apt-get update # update package list
* apt-get dist-upgrade # install updated packages and update "kept back" packages
* apt-get autoremove # remove packages that apt thinks should be removed

= Finish installation =

Congratulations. There is nothing more to do!

* reboot
<pre>
shutdown -r now
</pre>

= Install ZFS =

!!! after installing all the packages, after updating the system, after updating the linux kernel, after rebooting into latest kernel !!!

<pre>
apt-get install zfsutils-linux
</pre>

Follow generic ZFS instructions: [[ZFS]]

= Update to new version of Ubuntu =

<pre>
vi /etc/update-manager/release-upgrades # set "Prompt=normal"
do-release-upgrade
</pre>

Update Ubuntu LTS 20.04 to LTS 22.04:

== daqubuntu ==

<pre>
# reboot to clear out all updates
# vi /etc/update-manager/release-upgrades # set "Prompt=normal"
# do-release-upgrade -c
Checking for a new Ubuntu release
New release '22.04 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
# do-release-upgrade
...
say yes...
...
login.defs, say "Y" (erase local changes, use packaged version)
/etc/systemd/resolved.conf, say "Y" (same as above)
firefox snap, say yes
unable to reach snap store, say "skip"
/etc/gmond.conf, say "Y"
/var/yp/Makefile, say "install the package maintainer's version"
/etc/ypserv.conf, same thing
/etc/ypserv.securenets, same thing
/etc/default/nis, same thing
/etc/speech-dispatcher/modules/mary-generic.conf, same thing
/etc/apt/apt.conf.d/50unattended-upgrades, same thing
...
278 packages are going to be removed, say yes
...
restart required, say yes
...
no ping... yes ping...
...
ssh daqubuntu, ok
apt update, fail, DNS does not work, "host security.ubuntu.com" does not resolve.
fix resolver per https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Disable_NetworkManager
apt update, apt upgrade now works, 0 packages to update
NIS does not work.
</pre>

== midm9a ==

<pre>
login.defs
firefox snap
gmond.conf
ypserv
/etc/default/nis
unattended-upgrades
amanda-security.conf
remove obsolete (no)
reboot
configure dns
reenable nis
</pre>

== daq17 ==

<pre>
firefox snap
imagemagick policy.xml
gmond.conf
chrony.conf
/var/yp/Makefile
ypserv.conf
ypserv.securenets
/etc/default/nis
50unattended-upgrades
</pre>

== daq00 ==

per https://serverpilot.io/docs/how-to-upgrade-ubuntu-20.04-to-22.04/

<pre>
do-release-upgrade -f DistUpgradeViewNonInteractive
</pre>

if it exists "too soon" without doing anything, run it without "-f xxx", most likely it does not like something about this machine. in case of daq00 it did not like how the EFI partitions were mounted. after fixing it, non-interactive upgrade was successful.

= Ubuntu package manager =

* apt-get install xxx # install package xxx
* apt-get update
* apt-get upgrade
* apt-get dist-upgrade
* apt-get autoremove # remove automatically installed packages required by a removed package
* apt-get remove xxx # remove package xxx
* apt-cache search . # list all available packages
* apt-cache show "." | grep ^Package # list al available packages
* apt-cache madison root-system # show all available versions of package root-system
* apt list # list all installed packages
* dpkg --listfiles libpng16-16 # list all files from this package
* apt list --installed # list all installed packages
* dpkg -S /bin/bash # what package provides this file?
* dpkg -L bash # what files provided by this package?
* debsums -ce # show modified config files
* apt-config dump # show apt configuration

= Ubuntu zsys =

NOTE: DO NOT USE ZSYS, see https://github.com/ubuntu/zsys/issues/218 and https://github.com/ubuntu/zsys/issues/230

* manual removal of old snapshots
<pre>
zsysctl show
zsysctl state remove xy69ye -s
zsysctl state remove xy69ye
zsysctl state remove xy69ye -u wheel
</pre>
* apt remove zsys

NOTE: old zsys snapshots must be cleaned manually, "zsysctl state remove xxx --system" is broken and does not remove user data snapshots

* manages system snapshots
* documentation: https://github.com/ubuntu/zsys
* documentation: (go to next article via link "newer" at the bottom) https://didrocks.fr/2020/05/21/zfs-focus-on-ubuntu-20.04-lts-whats-new/
* ubuntu 20.04 bug, too many snapshots cause /boot to become full and updates fail. https://github.com/ubuntu/zsys/issues/155
* solution: use custom /etc/zsys.conf, limit number of snapshots to 10, see trinatdaq:/etc/zsys.conf
* zsys commands:
<pre>
update-grub # list of all snapshots, errors if some snapshots are broken
zsysctl state remove lnc0k7 --system # remove snapshot
xemacs -nw /etc/zsys.conf; zsysctl service reload; zsysctl service gc # cause gc to run with new settings in zsys.conf
zfs list -r -t snapshot -o name,used,referenced,creation bpool/BOOT # list snapshots
zsysctl show # show snapshots
</pre>

= Ubuntu cloning =

to clone a ubuntu image:

<pre>
cd /nfsroot/lxcpet
emacs -nw etc/hostname ### change hostname
emacs -nw etc/mailname ### change hostname (debian 11)
emacs -nw etc/defaultdomain ### change the NIS domainname
emacs -nw etc/yp.conf ### change the NIS server
cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
emacs -nw root/.ssh/authorized_keys ### update root ssh keys
</pre>

= Ubuntu boot loader =

== boot from ZFS ==

* use UEFI boot with syslinux, see here: https://daq.triumf.ca/DaqWiki/index.php/SLinstall#Configure_UEFI_boot
* apt install zfs-initramfs
* update-initramfs -v -u
* ZFS structure:
<pre>
root@daq00:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 147G 1.62T 96K /
rpool/ROOT 17.8G 1.62T 96K none
rpool/ROOT/ubuntu_00aaaa 17.8G 1.62T 6.22G /
</pre>
* copy OS image to rpool/ROOT/ubuntu_00aaaa
* zfs set mountpoint=/ rpool
* zfs set mountpoint=none rpool/ROOT
* zfs set mountpoint=/ rpool/ROOT/ubuntu_00aaaa
* zfs get all | grep mountpoint
<pre>
rpool mountpoint / local
rpool/ROOT mountpoint none local
rpool/ROOT/ubuntu_00aaaa mountpoint / local
</pre>
* in linux kernel command line (syslinux.cfg), set "root=" to "root=ZFS=rpool/ROOT/ubuntu_00aaaa"

== boot from ZFS mirror ==

=== setup the EFI partitions ===

* assuming /dev/sdb is already setup for EFI boot, setup /dev/sda the same way:
* partition the second boot disk same as first boot disk:
<pre>
root@grsnis01:~# gdisk -l /dev/sdb
Found valid GPT with protective MBR; using GPT.
Number Start (sector) End (sector) Size Code Name
1 2048 1050623 512.0 MiB EF00 EFI system partition
2 1050624 3907029134 1.8 TiB 8300 Linux filesystem
root@grsnis01:~#
</pre>
* mkfs.msdos /dev/sdX1
* create mount points
<pre>
mkdir /boot/efi-sda
mkdir /boot/efi-sdb
</pre>
* add to /etc/fstab
<pre>
/dev/sda1 /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
/dev/sdb1 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
</pre>
* mount -a
* df | grep boot
<pre>
root@grsnis01:~# df | grep boot
/dev/sdb1 523248 98100 425148 19% /boot/efi-sdb
/dev/sda1 523248 4 523244 1% /boot/efi-sda
</pre>
* copy boot files to new boot disk
* cd /boot/efi-sdX; rsync -av . /boot/efi-sdY
* set BIOS to boot from "UEFI Hard drive", disable legacy boot (except for booting from USB key in legacy mode)
* if using UEFI boot syslinux per these instructions, linux kernel update has to be done manually:
* run ~/git/scripts/etc/update_efi_mirror.perl, follow instructions that it prints.

=== setup zfs partitions ===

use partitions compatible with Ubuntu "install on ZFS"

* gdisk "o" to create new GPT partition table
* gdisk "n" +512M ef00 to create EFI partition
* gdisk "n" +2G 8200 to create linux swap partition (not used)
* gdisk "n" +2G BE00 to create ZFS bpool partition
* gdisk "n" xxx BF00 create ZFS rpool partition

<pre>
# gdisk -l /dev/sda
Number Start (sector) End (sector) Size Code Name
1 2048 1050623 512.0 MiB EF00 EFI System Partition
2 1050624 5244927 2.0 GiB 8200
3 5244928 9439231 2.0 GiB BE00
4 9439232 234441614 107.3 GiB BF00
root@midm9a:~#
</pre>

=== setup zfs mirror ===

* see here: https://daq.triumf.ca/DaqWiki/index.php/ZFS#Convert_pool_from_single_to_mirror
<pre>
root@grsnis01:~# ls -l /dev/disk/by-id/ata*part2
lrwxrwxrwx 1 root root 10 Feb 19 16:47 /dev/disk/by-id/ata-WDC_WDS200T2B0A-00SM50_205007801081-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Feb 19 16:47 /dev/disk/by-id/ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 -> ../../sdb2

root@grsnis01:~# zpool status
pool: rpool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 ONLINE 0 0 0

errors: No known data errors

root@grsnis01:~# zpool attach rpool ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 /dev/disk/by-id/ata-WDC_WDS200T2B0A-00SM50_205007801081-part2

root@grsnis01:~# zpool status
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Feb 19 16:54:39 2021
12.6G scanned at 3.16G/s, 1.02G issued at 262M/s, 12.6G total
1.02G resilvered, 8.09% done, 0 days 00:00:45 to go
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801081-part2 ONLINE 0 0 0 (resilvering)

errors: No known data errors
</pre>
* wait
<pre>
root@grsnis01:~# zpool status
pool: rpool
state: ONLINE
scan: resilvered 12.7G in 0 days 00:00:40 with 0 errors on Fri Feb 19 16:55:19 2021
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801101-part2 ONLINE 0 0 0
ata-WDC_WDS200T2B0A-00SM50_205007801081-part2 ONLINE 0 0 0

errors: No known data errors
</pre>

== maintenance commands ==

* update-initramfs -v -u
* grub-install /dev/sda

= Convert from single to dual mirrored ZFS SSD =

Assuming Ubuntu LTS 22.04 with "instal on ZFS" option, we will
add a second SSD, configure ZFS to use both SSDs in mirrored
configuration and setup grub to boot from either SSD. This
is intended to create a full redundant system where failure
of either SSD does not break the system.

* identify first SSD
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* connect second SSD of identical size
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
/dev/sdb WD Blue SA510 2.5 250GB 22243Z803852 25 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* if second SSD is not autodetected, reboot
* if both SSDs are identical size, use this simpler method of duplicating the partition table:
<pre>
root@midm9b:~# sfdisk -d /dev/sda > part_table
root@midm9b:~# grep -v ^label-id part_table | sed -e 's/, *uuid=[0-9A-F-]*//' | sfdisk /dev/sdb
</pre>
The grep and sed in the second command are there to prevent disk ID and partition IDs from being cloned. Alternatively the part_table file can be edited manually to remove the label-id line and the uuid entries from the individual partitions.

* list partition table of first SSD:
<pre>
root@midm9b:~# fdisk -l /dev/sda
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>
* create identical partitions on second SSD, use sector numbers from above.
<pre>
root@midm9b:~# gdisk /dev/sdb
GPT fdisk (gdisk) version 1.0.8

Partition table scan:
MBR: not present
BSD: not present
APM: not present
GPT: not present

Creating new GPT entries in memory.

Command (? for help): n
Partition number (1-128, default 1):
First sector (34-488397134, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-488397134, default = 488397134) or {+-}size{KMGTP}: 1050623
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): ef00
Changed type of partition to 'EFI system partition'

Command (? for help): n
Partition number (2-128, default 2):
First sector (34-488397134, default = 1050624) or {+-}size{KMGTP}:
Last sector (1050624-488397134, default = 488397134) or {+-}size{KMGTP}: 5244927
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 8200
Changed type of partition to 'Linux swap'

Command (? for help): n
Partition number (3-128, default 3):
First sector (34-488397134, default = 5244928) or {+-}size{KMGTP}:
Last sector (5244928-488397134, default = 488397134) or {+-}size{KMGTP}: 9439231
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): be00
Changed type of partition to 'Solaris boot'

Command (? for help): n
Partition number (4-128, default 4):
First sector (34-488397134, default = 9439232) or {+-}size{KMGTP}:
Last sector (9439232-488397134, default = 488397134) or {+-}size{KMGTP}:
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): bf00
Changed type of partition to 'Solaris root'

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sdb.
The operation has completed successfully.
root@midm9b:~# fdisk -l /dev/sda /dev/sdb
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root

Disk /dev/sdb: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: EB251739-30C6-422F-A505-5887B5A0B603

Device Start End Sectors Size Type
/dev/sdb1 2048 1050623 1048576 512M EFI System
/dev/sdb2 1050624 5244927 4194304 2G Linux swap
/dev/sdb3 5244928 9439231 4194304 2G Solaris boot
/dev/sdb4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>
* identify second SSD partitions
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part3
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part3 -> ../../sda3
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 -> ../../sdb3
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
</pre>
* convert bpool from single disk to mirrored disk:
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~# zpool attach bpool 99e03dc0-7d4d-f24b-8fa1-f042b9f135db /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3
root@midm9b:~# zpool status bpool
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors
</pre>
* convert rpool
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
root@midm9b:~# zpool attach rpool f6fd54f8-3af7-b943-ae3d-a4e480537fb9 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4
root@midm9b:~# zpool status rpool
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Jan 20 19:40:45 2023
5.83G scanned at 664M/s, 2.92M issued at 332K/s, 9.11G total
0B resilvered, 0.03% done, no estimated completion time
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~#
</pre>
* wait for resilver to complete
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
scan: resilvered 9.65G in 00:00:36 with 0 errors on Fri Jan 20 19:41:21 2023
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
</pre>
* enable booting from second SSD: (instead of /dev/sda1, /dev/sdb1, use UUID=xxx)
<pre>
root@midm9b:~# mkfs.msdos /dev/sdb1
root@midm9b:~# mkdir /boot/efi-sda
root@midm9b:~# mkdir /boot/efi-sdb
root@midm9b:~# echo "/dev/sda1 /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1" >> /etc/fstab
root@midm9b:~# echo "/dev/sdb1 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1" >> /etc/fstab
root@midm9b:~# mount -a
root@midm9b:~# df -kl
Filesystem 1K-blocks Used Available Use% Mounted on
...
/dev/sda1 523244 13720 509524 3% /boot/efi
/dev/sdb1 523244 4 523240 1% /boot/efi-sdb
...
root@midm9b:~# rsync -av /boot/efi/ /boot/efi-sdb/
sending incremental file list
EFI/
...
root@midm9b:~# ls -l /boot/efi-sda
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~# ls -l /boot/efi-sdb
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~#
</pre>
* setup script to update grub on second SSD, it must be run manually after every kernel update
<pre>
root@midm9b:~# ln -s ~/git/scripts/etc/update_efi_grub.perl ~/
root@midm9b:~# ~/update_efi_grub.perl -u
EFI dir: /boot/efi-sda
/boot/efi-sda: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sda/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sda: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sda/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
EFI dir: /boot/efi-sdb
/boot/efi-sdb: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sdb/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sdb: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sdb/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
root@midm9b:~#
</pre>

= Disable NetworkManager =

NOTE: THIS IS BROKEN IN UBUNTU LTS 22.04

NetworkManager is useful for configuring dynamic
network interfaces, i.e. laptops that often move
between networks, or connect to multiple choice
of wifi networks, etc.

For machines with statically configured network interfaces,
NetworkManager is not necessary.

As it has been observed to become confused and observed
to malfunction when network links go up and down (it keeps
unnecessarily reconfiguring the ip address, etc), it can
be usefuil to disable it.

* list all network interfaces
<pre>
# /bin/ls -1 /sys/class/net/
enp0s31f6
lo
</pre>
* edit /etc/network/interfaces:
<pre>
rename enp0s31f6=eth0
auto eth0
iface eth0 inet static
address 142.90.120.94/19
gateway 142.90.100.18
</pre>
* statically configure systemd-resolved
** create /etc/systemd/resolved.conf.d/resolved.conf with this contents:
<pre>
[Resolve]
DNS=142.90.100.19
Domains=triumf.ca
</pre>
** systemctl restart systemd-resolved
** resolvectl
** systemd-analyze cat-config systemd/resolved.conf
* disable NetworkManager
<pre>
systemctl disable NetworkManager
</pre>
* reboot

= Configure ECC memory =

== Configure EDAC ==

* apt install edac-utils

=== Intel i3-2120 ===
<pre>
root@musr00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X9SCL/X9SCM
root@musr00:~# edac-ctl --status
edac-ctl: drivers not loaded.
</pre>

=== Intel E-2236 ===
<pre>
root@daq00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SCM-F
root@daq00:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq00:~# edac-util
edac-util: No errors to report.
root@daq00:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
</pre>
* check edac sysfs files (Intel)
<pre>
root@daq00:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_noinfo_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 max_location
-r--r--r-- 1 root root 4096 Jan 25 15:10 mc_name
drwxr-xr-x 2 root root 0 Jan 25 15:10 power
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank0
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank1
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank2
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank3
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank4
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank5
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank6
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank7
--w------- 1 root root 4096 Jan 25 15:10 reset_counters
-r--r--r-- 1 root root 4096 Jan 25 15:10 seconds_since_reset
-r--r--r-- 1 root root 4096 Jan 25 15:10 size_mb
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Jan 25 15:10 uevent
root@daq00:~#
</pre>

=== Intel E3-1270 v6 ===
<pre>
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --status
edac-ctl: drivers are loaded.
root@grsnis01:~# edac-util
edac-util: No errors to report.
root@grsnis01:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@grsnis01:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 max_location
-r--r--r-- 1 root root 4096 Feb 19 12:35 mc_name
drwxr-xr-x 2 root root 0 Feb 19 12:35 power
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank0
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank1
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank2
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank3
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank4
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank5
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank6
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank7
--w------- 1 root root 4096 Feb 19 12:35 reset_counters
-r--r--r-- 1 root root 4096 Feb 19 12:35 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 19 12:35 size_mb
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 19 12:35 uevent
root@grsnis01:~#
</pre>

=== Intel E3-1245 v6 ===

<pre>
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --status
edac-ctl: drivers are loaded.
[root@alphagdaq ~]# edac-util
edac-util: No errors to report.
[root@alphagdaq ~]# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
[root@alphagdaq ~]# ras-mc-ctl --layout
+-----------------------------------------------+
| mc0 |
| csrow0 | csrow1 | csrow2 | csrow3 |
----------+-----------------------------------------------+
channel1: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
----------+-----------------------------------------------+
[root@alphagdaq ~]# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#3channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#3channel#1 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#2channel#0 0 0
[root@alphagdaq ~]# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SSH-F
[root@alphagdaq ~]# ras-mc-ctl --summary
DBD::SQLite::db prepare failed: no such table: mc_event at /usr/sbin/ras-mc-ctl line 1129.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1130.
[root@alphagdaq ~]#
</pre>

=== AMD 3700X ===

(memory is non-ECC)

<pre>
root@daq13:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@daq13:~#
root@daq13:~#
root@daq13:~# edac-ctl --status
edac-ctl: drivers not loaded.
root@daq13:~# edac-util
edac-util: Error: No memory controller data found.
root@daq13:~# edac-util -s
edac-util: EDAC drivers loaded. No memory controllers found
root@daq13:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 2 root root 0 Jan 25 15:26 power
lrwxrwxrwx 1 root root 0 Jan 21 16:16 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Jan 21 16:16 uevent
</pre>

(memory is ECC)

<pre>
root@trinatdaq:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@trinatdaq:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@trinatdaq:~# edac-util
edac-util: No errors to report.
root@trinatdaq:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Dec 15 13:04 mc0
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
lrwxrwxrwx 1 root root 0 Dec 13 18:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dec 13 18:31 uevent
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_noinfo_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 max_location
-r--r--r-- 1 root root 4096 Dec 15 13:04 mc_name
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank4
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank5
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank6
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank7
--w------- 1 root root 4096 Dec 15 13:04 reset_counters
-rw-r--r-- 1 root root 4096 Dec 15 13:04 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Dec 15 13:04 seconds_since_reset
-r--r--r-- 1 root root 4096 Dec 15 13:04 size_mb
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Dec 15 13:04 uevent
root@trinatdaq:~#
</pre>

=== AMD 5000G ===

* no linux driver for AMD 5000-series "G" CPU
* no mention of ECC in the BIOS settings
* unclear status of ECC support in AMD documentation (sais only "pro" "G" CPUs have ECC)
* unclear status of ECC support in ASUS documentation (web page out of date)

=== AMD 5600X ===

<pre>
root@daq17:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-XE GAMING WIFI
root@daq17:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq17:~# edac-util
edac-util: No errors to report.
root@daq17:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@daq17:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Aug 19 19:27 mc0
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
lrwxrwxrwx 1 root root 0 May 10 10:11 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 May 10 10:11 uevent
root@daq17:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_noinfo_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 max_location
-r--r--r-- 1 root root 4096 Aug 19 19:27 mc_name
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank4
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank5
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank6
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank7
--w------- 1 root root 4096 Aug 19 19:27 reset_counters
-rw-r--r-- 1 root root 4096 Aug 19 19:27 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Aug 19 19:27 seconds_since_reset
-r--r--r-- 1 root root 4096 Aug 19 19:27 size_mb
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Aug 19 19:27 uevent
root@daq17:~#
</pre>

=== AMD 3955WX ===

<pre>
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --status
edac-ctl: drivers are loaded.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util
edac-util: No errors to report.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@alphasuperdaq:~/git/scripts/quotareport# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 19 root root 0 Dez 12 04:48 mc0
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
lrwxrwxrwx 1 root root 0 Dez 9 05:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dez 9 05:31 uevent
root@alphasuperdaq:~/git/scripts/quotareport#
root@alphasuperdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 max_location
-r--r--r-- 1 root root 4096 Feb 28 22:19 mc_name
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank0
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank1
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank10
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank11
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank12
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank13
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank14
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank15
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank2
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank3
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank4
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank5
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank6
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank7
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank8
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank9
--w------- 1 root root 4096 Feb 28 22:19 reset_counters
-rw-r--r-- 1 root root 4096 Feb 28 22:19 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Feb 28 22:19 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 28 22:19 size_mb
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 28 22:19 uevent
root@alphasuperdaq:~#
root@alphasuperdaq:~# ras-mc-ctl --layout
Use of uninitialized value $max_pos[3] in modulus (%) at /usr/sbin/ras-mc-ctl line 868.
Use of uninitialized value $d in numeric ge (>=) at /usr/sbin/ras-mc-ctl line 869.
Use of uninitialized value $d in sprintf at /usr/sbin/ras-mc-ctl line 872.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| mc0 |
| csrow0 | csrow1 |
| channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 | channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

0: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
root@alphasuperdaq:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#2 0 0
mc#0csrow#1channel#7 0 0
mc#0csrow#0channel#3 0 0
mc#0csrow#1channel#4 0 0
mc#0csrow#1channel#2 0 0
mc#0csrow#0channel#7 0 0
mc#0csrow#1channel#3 0 0
mc#0csrow#0channel#4 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#5 0 0
mc#0csrow#0channel#6 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#0channel#5 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#6 0 0
root@alphasuperdaq:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: ASUSTeK COMPUTER INC. model Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@alphasuperdaq:~#
</pre>

== Configure rasdaemon ==

<pre>
apt install rasdaemon
</pre>
<pre>
systemctl enable rasdaemon
systemctl restart rasdaemon
systemctl status rasdaemon
</pre>

<pre>
● rasdaemon.service - RAS daemon to log the RAS events
Loaded: loaded (/lib/systemd/system/rasdaemon.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-01-25 15:16:37 PST; 3min 5s ago
Main PID: 2477175 (rasdaemon)
Tasks: 1 (limit: 76958)
Memory: 17.1M
CGroup: /system.slice/rasdaemon.service
└─2477175 /usr/sbin/rasdaemon -f -r

Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Listening to events for cpus 0 to 11
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mc_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording aer_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording extlog_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mce_record events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording arm_event events
</pre>

== Get reports ==

* Intel 2x32GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --layout
+-------------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-------------------------+
channel1: | 16384 MB | 16384 MB |
channel0: | 16384 MB | 16384 MB |
----------+-------------------------+
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#0channel#1 0 0
root@daq00:~#
</pre>

* Intel 4x16GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#1 0 0
mc#0csrow#2channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#3channel#0 0 0
mc#0csrow#3channel#1 0 0
root@daq00:~#
root@daq00:~# ras-mc-ctl --layout
+-----------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-----------------------+
channel1: | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB |
----------+-----------------------+
root@daq00:~#
root@daq00:~#
root@daq00:~#
root@daq00:~# ras-mc-ctl --print-labels
ras-mc-ctl: Error: No dimm labels for Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@daq00:~#
</pre>

note: ubuntu LTS 22.04 DBD::SQLite::db error is not there.

= sensors =

== ASUS P9X79 WS ==

* https://www.asus.com/supportonly/P9X79%20WS/HelpDesk_Manual/
* BIOS version 4802
* modprobe nct6775
* modprobe coretemp

<pre>
root@daq14:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 0: +29.0°C (high = +82.0°C, crit = +100.0°C)
Core 1: +24.0°C (high = +82.0°C, crit = +100.0°C)
Core 2: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +82.0°C, crit = +100.0°C)

nouveau-pci-0200
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.00 V)
temp1: +39.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

nct6776-isa-0290
Adapter: ISA adapter
Vcore: 1.04 V (min = +0.00 V, max = +1.74 V)
in1: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
AVCC: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
+3.3V: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 2.04 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 904.00 mV (min = +0.00 V, max = +0.00 V) ALARM
3VSB: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
Vbat: 3.30 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1265 RPM (min = 0 RPM)
fan2: 1909 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +34.0°C (high = +0.0°C, hyst = +0.0°C) ALARM sensor = thermistor
CPUTIN: +58.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermal diode
AUXTIN: +31.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
PECI Agent 0: +31.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +96.0°C)
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

root@daq14:~#
</pre>

= Enable CPU turbo mode =

* Intel CPU has a nominal CPU frequency (i.e. 3.4GHz) and a turbo-boost CPU frequency (i.e. 4.0GHz). Here we will enable this turbo-boost mode.
* Find out CPU capability
<pre>
root@daq01:~# lscpu | grep Hz
Model name: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
CPU MHz: 3965.803
CPU max MHz: 4000.0000
CPU min MHz: 800.0000
root@daq01:~#
</pre>
* Look up this CPU in the Intel ARK database - google for the CPU model name, i.e.
https://ark.intel.com/content/www/us/en/ark/products/88196/intel-core-i7-6700-processor-8m-cache-up-to-4-00-ghz.html
* Find current frequency settings:
<pre>
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "powersave" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 2.72 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
root@daq01:~#
</pre>
* Note the following:
** current governor is "powersave"
** "performance" governor is available
** "boost state support" is supported and active.
* Confirm CPU frequency governor:
<pre>
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
powersave
powersave
powersave
powersave
powersave
powersave
powersave
powersave
root@daq01:~#
</pre>
* Change governor to "performance":
<pre>
root@daq01:~# cpupower frequency-set --governor performance
Setting cpu: 0
Setting cpu: 1
Setting cpu: 2
Setting cpu: 3
Setting cpu: 4
Setting cpu: 5
Setting cpu: 6
Setting cpu: 7
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
performance
performance
performance
performance
performance
performance
performance
performance
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "performance" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 3.93 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
</pre>
* monitor CPU frequency:
<pre>
root@daq01:~# cpupower monitor
| Nehalem || Mperf || Idle_Stats
CPU| C3 | C6 | PC3 | PC6 || C0 | Cx | Freq || POLL | C1 | C1E | C3 | C6 | C7s | C8
0| 0.00| 0.00| 0.00| 0.00|| 88.80| 11.20| 3973|| 0.00| 0.00| 0.01| 0.02| 0.31| 0.00| 4.25
4| 0.00| 0.00| 0.00| 0.00|| 4.70| 95.30| 3945|| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 95.03
1| 0.73| 3.70| 0.00| 0.00|| 4.52| 95.48| 3864|| 0.00| 0.01| 1.19| 0.44| 2.82| 0.00| 90.23
5| 0.73| 3.70| 0.00| 0.00|| 0.37| 99.63| 3807|| 0.00| 0.00| 0.03| 0.09| 1.70| 0.00| 97.64
2| 2.28| 12.86| 0.00| 0.00|| 1.41| 98.59| 3829|| 0.00| 0.86| 3.17| 0.46| 7.70| 0.00| 85.87
6| 2.28| 12.86| 0.00| 0.00|| 2.88| 97.12| 3856|| 0.00| 0.11| 4.56| 2.15| 10.31| 0.00| 78.99
3| 1.33| 4.81| 0.00| 0.00|| 0.99| 99.01| 3804|| 0.00| 0.49| 0.79| 0.01| 1.03| 0.00| 96.12
7| 1.34| 4.81| 0.00| 0.00|| 1.26| 98.74| 3818|| 0.00| 0.01| 2.32| 0.47| 5.02| 0.00| 90.06
root@daq01:~#
</pre>
* check that the CPU is not overheating:
<pre>
root@daq01:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 1: +38.0°C (high = +84.0°C, crit = +100.0°C)
Core 2: +34.0°C (high = +84.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +84.0°C, crit = +100.0°C)
</pre>
* congratulations, we are running at 4 GHz now!

= Setup ubuntu as gateway to private network =

See also:
* https://daq.triumf.ca/DaqWiki/index.php/VME-CPU#Setup_the_boot_host_computer_.28el7.29
* http://www.triumf.info/wiki/DAQwiki/index.php/Dhcpd_on_eth1

== Steps to do ==

* assign network numbers to the private network, i.e. 192.168.1.x, 192.168.2.x, etc
* (on the gateway machine, each private network interface has to have a different network number)
* (each network interface can have multiple networks attached, via VLANs or via eth0:0, eth0:1 constructs)
* assign IP addresses on the private network, save them in /etc/hosts i.e. "hvps 192.168.1.10"
* (for simplicity, assign 192.168.1.1 to the gateway machine itself)
* (IP addresses 192.168.1.0 and 192.168.1.255 are "special", do not use them)
* setup DNS server (dnsmasq) to serve contents of /etc/hosts via DNS (otherwise, many programs will see inconsistent name to IP address mapping)
* setup DHCP server (ISC dhcpd or dnsmasq) to give out the IP addresses
* setup tftp, pxelinux and NFS for diskless booting
* setup time server (chronyd) to provide common time to all devices
* setup NAT so machines on private network can access the internet (to get OS updates, etc)
* setup NIS and NFS so machines on the private network can use common home directories
* setup rsync backup of machines on the private network

== setup hosts ==

* edit /etc/hosts
<pre>
192.168.1.101 dsfe01
... and so forth
</pre>

== setup dns and dhcp ==

* apt install dnsmasq
* edit /etc/dnsmasq.conf
<pre>
# /etc/dnsmasq.conf
# DNS settings
#port=0 # disable DNS function
port=53 # enable DNS function
domain-needed
bogus-priv
no-resolv
server=142.90.100.19
# DHCP settings
interface=enp1s0f0 # DHCP interface
#dhcp-range=192.168.1.50,192.168.1.150,infinite
dhcp-range=192.168.1.0,static
#log-dhcp
quiet-dhcp
#dhcp-ignore=tag:!known
dhcp-boot=pxelinux.0
#dhcp-host=ac:1f:6b:9e:7f:4a,192.168.1.100,10m
dhcp-host=ac:1f:6b:9e:7f:4a,dsfe01,infinite
# TFTP settings
enable-tftp
tftp-root=/tftpboot
</pre>
* #mkdir /zssd/tftpboot ### per tftp-root (if no ZFS)
* zfs create -o mountpoint=/tftpboot rpool/tftpboot ### (if root is ZFS)
* systemctl stop systemd-resolved.service
* systemctl disable systemd-resolved.service
* rm /etc/resolv.conf
* create new /etc/resolv.conf with this contents:
<pre>
nameserver 127.0.0.1
search snolab.ca
</pre>
* systemctl enable dnsmasq
* systemctl restart dnsmasq

== setup chronyd ==

* enable ntp server:
* configure and enable chronyd per instructions above
* echo "allow 192.168.1.0/24" > /etc/chrony/conf.d/allow-localhost.conf
* systemctl restart chronyd
* chronyc tracking ### wait until time is synchronized (a few seconds)

== setup diskless network booting ==

=== setup pxelinux ===
<pre>
cd ~
wget https://www.kernel.org/pub/linux/utils/boot/syslinux/4.xx/syslinux-4.03.tar.bz2
tar xjvf syslinux-4.03.tar.bz2
cd syslinux-4.03
cp -pv ./core/pxelinux.0 ./com32/hdt/hdt.c32 ./memdisk/memdisk ./com32/menu/menu.c32 /zssd/tftpboot/
</pre>
* cd /zssd/tftpboot
<pre>
wget http://ladd00.triumf.ca/tftpboot/memtest86+-4.20.iso.zip
wget http://ladd00.triumf.ca/tftpboot/memtest86+-5.01.iso.gz
wget http://ladd00.triumf.ca/tftpboot/modules.alias
wget http://ladd00.triumf.ca/tftpboot/modules.pcimap
wget http://ladd00.triumf.ca/tftpboot/pci.ids
</pre>
* mkdir pxelinux.cfg
* emacs -nw pxelinux.cfg/default
<pre>
default menu.c32
prompt 0

menu title Welcome to the DSVSLICE PXE boot menu

timeout 50

label hdt
kernel hdt.c32

label memtest86+-5.01
kernel memdisk iso initrd=memtest86+-5.01.iso.gz

label memtest86+-4.20
kernel memdisk iso initrd=memtest86+-4.20.iso.zip

label vmlinuz-5.3.0-26-generic
menu default
kernel vmlinuz-5.3.0-26-generic
append initrd=initrd.img-5.3.0-26-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.1.1:/zssd/nfsroot/dsfe01 toram ip=dhcp panic=60 BOOTIF=enp1s0f0

#end
</pre>

=== setup linux kernel ===

* copy the kernel files
<pre>
cd /boot
rsync -av config* initrd* System.map* vmlinuz* /zssd/tftpboot/
</pre>
* cd /zssd/tftpboot
* chmod a+r *

=== setup nfs ===

* apt-get install nfs-kernel-server
* emacs -nw /etc/exports
<pre>
/zssd/nfsroot/dsfe01 dsfe01(rw,no_root_squash,async,no_subtree_check)
</pre>
* enable services
<pre>
systemctl enable nfs-server
systemctl enable nfs-mountd
systemctl enable nfs-idmapd
systemctl restart nfs-server
systemctl restart nfs-mountd
systemctl restart nfs-idmapd
</pre>
* after editing /etc/exports, run
<pre>
exportfs -av
</pre>

=== setup userland ===

* zfs create zssd/nfsroot
* zfs set dedup=verify zssd/nfsroot ### enable deduplication to save disk space because most linux images have mostly identical files
* clone ubuntu
<pre>
mkdir /zssd/nfsroot/dsfe01
cd /
rsync -avx . /zssd/nfsroot/dsfe01
</pre>
* edit config files:
* cd /zssd/nfsroot/dsfe01
* emacs -nw etc/hostname ### change to dsfe01
* emacs -nw etc/mailname ### change to dsfe01
* emacs -nw etc/yp.conf ### change daq00.triumf.ca to musr00.triumf.ca
* emacs -nw etc/defaultdomain ### change to MUSR-NIS
* cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
* emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
* emacs -nw root/.ssh/authorized_keys ### update root ssh keys
* emacs -nw etc/fstab ### add this
<pre>
192.168.1.1:/zssd/nfsroot/dsfe01 / nfs defaults,nolock 0 0
</pre>
* emacs -nw etc/chrony/chrony.conf
** comment-out all "pool" and "server" entries
** add entry "server 192.168.1.1 iburst"

After dsfe01 is booted:

* disable services:
<pre>
systemctl disable apache2
systemctl disable dnsmasq
systemctl disable zfs-import-cache
</pre>

To setup additional machines, clone dsfe01 instead of cloning the gateway machine

=== Allow manpages to be viewed ===

If <code>/</code> is mounted over NFS, <code>man</code> will report a permission error. Fix it with:

<pre>
ln -s /etc/apparmor.d/usr.bin.man /etc/apparmor.d/disable/
apparmor_parser -R /etc/apparmor.d/usr.bin.man
</pre>

== setup shared home directory ==

=== on the gateway machine ===
* define netgroups
* emacs -nw /etc/netgroup
<pre>
dsfe (dsfe01,,) (dsfe02,,)
</pre>
* emacs -nw /etc/nsswitch.conf ### edit the netgroup line to read:
<pre>
netgroup: files
</pre>
* export the home directories:
* emacs -nw /etc/exports ### add this:
<pre>
/zssd/home1 @dsfe(rw,no_root_squash,async,no_subtree_check)
</pre>
* exportfs -rc

=== on the frontend machine ===

* mkdir /home
* emacs -nw /etc/fstab ### add this:
<pre>
192.168.1.1:/zssd/home1 /home nfs defaults 0 0
</pre>
* mount -a

== setup NAT ==

NAT allows machines on the private network to connect to the internet: https://en.wikipedia.org/wiki/Network_address_translation

In these examples:
* replace "eno1" with name of the outgoing interface (the one connected to the TRIUMF network).
* replace "enp11s0" with name of the private network interface (192.168.1.x network)

* emacs -nw /etc/rc.local ### add this:
<pre>
# /etc/rc.local

/sbin/iptables -t nat -A POSTROUTING -o eno1 -j MASQUERADE
iptables -L -v

# uncomment following lines if machine has prohibitive FORWARD rules:
#/sbin/iptables -I FORWARD -i eno1 -o enp11s0 -m state --state RELATED,ESTABLISHED -j ACCEPT
#/sbin/iptables -I FORWARD -i enp11s0 -o eno1 -j ACCEPT
#iptables -L -v

iptables -L -v
sysctl -w net.ipv4.ip_forward=1
#sysctl -a | grep forward

sh /etc/firewall-rfc1918.sh

# end
</pre>
* emacs -nw /etc/firewall-rfc1918.sh
<pre>
# firewall-rfc1918.sh

# prevent RFC1918 private network IP addresses from
# going in and out from our uplink.

ETH=eno1

iptables -F in-rfc1918
iptables -N in-rfc1918
iptables -A in-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A in-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A in-rfc1918 --dst 192.168.0.0/16 -j REJECT
iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -I INPUT -j in-rfc1918 -i $ETH

iptables -F out-rfc1918
iptables -N out-rfc1918
iptables -A out-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A out-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A out-rfc1918 --dst 192.168.0.0/16 -j REJECT
iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -I OUTPUT -j out-rfc1918 -o $ETH

iptables -L -v

#end
</pre>

= KVM =

<pre>
apt install cpu-checker

root@daq13:~# kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
root@daq13:~#

(if not, shutdown, go into BIOS settings, enable CPU virtualization)

apt install virtinst ### will install many packages
apt install libvirt-clients libvirt-daemon-system-systemd libvirt-daemon qemu qemu-kvm libvirt-daemon-system virtinst bridge-utils

root@daq13:/home1/wheel# virsh list --all
Id Name State
------------------------------
1 ubuntu-guest running

apt install virt-manager

virt-install --name ubuntu-guest --os-variant ubuntu20.04 --vcpus 2 --ram 2048 --location /daq/daqstore/olchansk/linux/Ubuntu/ubuntu-20.04.3-desktop-amd64.iso --network bridge=virbr0,model=virtio --graphics none --extra-args='console=ttyS0,115200n8 serial'

virtual machine will start, boot, etc
to get out of it, CTRL + Shift followed by ]

ssh wheel@daq13
virt-manager

run virt-install again, omit "--graphics none", open graphics console from virt-manager, it booted into ubuntu installer desktop

virt-install --name test10 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --filesystem /kvm_ladd00,/ --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial" --graphics none

virt-install --name test14 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --disk /tmp/xxx/ladd00.img,bus=sata --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial rdshell" --graphics none --check path_in_use=off
</pre>

build image

<pre>
dd if=/dev/zero of=/tmp/xxx/ladd00.img bs=1024M count=20
mkfs.ext3 /tmp/xxx/ladd00.img ### ext4 fails to mount by SL6 kernel, "unknown ext4 options"
cd /kvm_ladd00/
mount -o loop /tmp/xxx/ladd00.img /mnt/tmp
rsync -av . /mnt/tmp/ --delete
umount /mnt/tmp
</pre>

on the guest, configure network: /etc/rc.local
<pre>
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

ifconfig eth2 192.168.122.2
route add -net 0.0.0.0 gw 192.168.122.1
ifconfig -a
netstat -rn

# end
</pre>

= ARM cross-compiler =

<pre>
apt install libgcc-9-dev-arm64-cross
apt install gcc-arm-linux-gnueabi
apt install gcc-arm-linux-gnueabihf
apt install g++-arm-linux-gnueabihf
apt install g++-arm-linux-gnueabi
</pre>

<pre>
arm-linux-gnueabi-gcc -o ttcp1 ttcp.c -march=armv7 -static
arm-linux-gnueabi-gcc -o memcpy.armv7 memcpy.cc -march=armv7 -static -O2
</pre>

= 32-bit intel cross-compiler =

Ubuntu 22.04:

<pre>
apt install libstdc++-11-dev:i386
apt install zlib1g-dev:i386
</pre>

for MIDAS, use "make linux32"

Laser calibration

2020-07-21T20:58:30Z

Lmartin: /* Access SPI as non-root user */

== Introduction ==
Both the prototype and final TPC have aluminium strips on the central cathode that are intended for laser calibration. To that end a plane of UV laser light is sent into the TPC intersecting with the strips. Each laser pulse should then create a bunch of electrons with well defined position and time at each of the strips.

== Photoelectron efficiency tests ==
One worry with the intended geometry is that the laser hits the far strips at a very glancing angle, and it was assumed the photoelectron efficiency (i.e. number of photoelectrons per photon) would go down in favour of reflection. To prevent this two methods of surface treatment were considered, and both the validity of the aforementioned assumption as well as the effectiveness of the treatments were investigated in a test setup at UBC. Details and results are given in this [[Media:LaserTests.pdf|report]].

== Laser operating procedure ==
The following describes the standard operating procedure for running the laser setup on the prototype.
[[File:Control.jpg|thumb|Handheld control unit]]

=== Safety ===
<big>
''NEVER have the key inserted in the "ignition" when the optical fiber is disconnected, the black box is open, or the TPC is open (e.g. gas lines disconnected).''
</big>
=== Start-up ===
[[File:Shutter.jpg|thumb|Manual shutter: left open, right closed]]
# Check integrity of enclosure, i.e. optics box closed and bolted down, fiber attached at both ends.
# Turn on laser system with key on main unit.
# Push button "Flash lamp -> Ready" on handheld unit.
# Push button "Flash lamp -> Start" on handheld unit.
# Set collimator via mvat program on computer.
# Open manual shutter on laser head.
# Push button "Q-switch -> Start" on handheld unit.

=== Pause ===
==== Short pause ====
Simply push stop/start for Q-switch.
==== Longer pause ====
# Stop Q-switch.
# Stop flash lamp.
# Close manual shutter.
# After pause restart according to after-key part of start-up procedure.

=== Shutdown ===
# Stop Q-switch.
# Stop flash lamp.
# Close manual shutter.
# Wait a few minutes for cool-down.
# Turn off and remove key.

=== General best practice ===
* Limit laser shots to a minimum, to reduce wear on laser, flash lamp, and fiber.
* To that end, do non-critical steps like attenuator setting and DAQ start before starting Q-switch.
* Ideally allow system some time to warm up between turning on flash lamp and Q-switch.

== Optics ==

== Laser requirements ==

== MEMS Multiplexer ==

=== Raspberry Pi setup ===
==== Operating System ====
We're running the most recent version of Raspberry Pi OS as downloaded from [https://www.raspberrypi.org/downloads/ raspberrypi.org]

==== Required packages ====
Install the following with apt get:
* emacs
* libcap-dev (required to access SPI as non-root user)

==== SPI library ====
The recommended way to interface with SPI from a raspberry pi is now no longer using the wiringPi library, instead [https://www.raspberrypi.org/documentation/hardware/raspberrypi/spi/README.md raspberrypi.org] recommends [http://www.airspayce.com/mikem/bcm2835/ bcm2835]. This is a direct hardware C library for pin communication that includes examples for GPIO, SPI, and I2C.

===== Access SPI as non-root user =====
Following the instructions from [http://www.airspayce.com/mikem/bcm2835/] non-root access to the SPI pins requires the following:

* install libcap-dev (see [[Laser_calibration#Required_packages|Required packages]])
<pre>
sudo apt install libcap-dev
</pre>

* add current user to kmem group
<pre>
sudo adduser $USER kmem
</pre>

* give group kmem access to /dev/mem
<pre>
echo 'SUBSYSTEM=="mem", KERNEL=="mem", GROUP="kmem", MODE="0660"' | sudo tee /etc/udev/rules.d/98-mem.rules
sudo reboot
</pre>

* uncomment the BCM2835_HAVE_LIBCAP line in src/bcm2835.h
<pre>
sed -i 's/\/\/#define BCM2835_HAVE_LIBCAP/#define BCM2835_HAVE_LIBCAP/' src/bcm2835.h
</pre>

* compile libbcm2835 as normal
<pre>
./configure
make
sudo make install # (make check fails)
</pre>

* in addition to -lbcm2835, compilation of example or own code requires -lcap, e.g.
<pre>
gcc -o spi spi.c -l bcm2835 -lcap
</pre>

* after compilation, give program permission to access cap
<pre>
sudo setcap cap_sys_rawio+ep <progname>
</pre>

===== C++ =====
C++ classes to simplify the use are provided in the mems code.

Laser calibration

2020-07-21T20:55:48Z

Lmartin: /* Access SPI as non-root user */

== Introduction ==
Both the prototype and final TPC have aluminium strips on the central cathode that are intended for laser calibration. To that end a plane of UV laser light is sent into the TPC intersecting with the strips. Each laser pulse should then create a bunch of electrons with well defined position and time at each of the strips.

== Photoelectron efficiency tests ==
One worry with the intended geometry is that the laser hits the far strips at a very glancing angle, and it was assumed the photoelectron efficiency (i.e. number of photoelectrons per photon) would go down in favour of reflection. To prevent this two methods of surface treatment were considered, and both the validity of the aforementioned assumption as well as the effectiveness of the treatments were investigated in a test setup at UBC. Details and results are given in this [[Media:LaserTests.pdf|report]].

== Laser operating procedure ==
The following describes the standard operating procedure for running the laser setup on the prototype.
[[File:Control.jpg|thumb|Handheld control unit]]

=== Safety ===
<big>
''NEVER have the key inserted in the "ignition" when the optical fiber is disconnected, the black box is open, or the TPC is open (e.g. gas lines disconnected).''
</big>
=== Start-up ===
[[File:Shutter.jpg|thumb|Manual shutter: left open, right closed]]
# Check integrity of enclosure, i.e. optics box closed and bolted down, fiber attached at both ends.
# Turn on laser system with key on main unit.
# Push button "Flash lamp -> Ready" on handheld unit.
# Push button "Flash lamp -> Start" on handheld unit.
# Set collimator via mvat program on computer.
# Open manual shutter on laser head.
# Push button "Q-switch -> Start" on handheld unit.

=== Pause ===
==== Short pause ====
Simply push stop/start for Q-switch.
==== Longer pause ====
# Stop Q-switch.
# Stop flash lamp.
# Close manual shutter.
# After pause restart according to after-key part of start-up procedure.

=== Shutdown ===
# Stop Q-switch.
# Stop flash lamp.
# Close manual shutter.
# Wait a few minutes for cool-down.
# Turn off and remove key.

=== General best practice ===
* Limit laser shots to a minimum, to reduce wear on laser, flash lamp, and fiber.
* To that end, do non-critical steps like attenuator setting and DAQ start before starting Q-switch.
* Ideally allow system some time to warm up between turning on flash lamp and Q-switch.

== Optics ==

== Laser requirements ==

== MEMS Multiplexer ==

=== Raspberry Pi setup ===
==== Operating System ====
We're running the most recent version of Raspberry Pi OS as downloaded from [https://www.raspberrypi.org/downloads/ raspberrypi.org]

==== Required packages ====
Install the following with apt get:
* emacs
* libcap-dev (required to access SPI as non-root user)

==== SPI library ====
The recommended way to interface with SPI from a raspberry pi is now no longer using the wiringPi library, instead [https://www.raspberrypi.org/documentation/hardware/raspberrypi/spi/README.md raspberrypi.org] recommends [http://www.airspayce.com/mikem/bcm2835/ bcm2835]. This is a direct hardware C library for pin communication that includes examples for GPIO, SPI, and I2C.

===== Access SPI as non-root user =====
Following the instructions from [http://www.airspayce.com/mikem/bcm2835/] non-root access to the SPI pins requires the following:

====== install libcap-dev (see [[Laser_calibration#Required_packages|Required packages]]) ======
<pre>
sudo apt install libcap-dev
</pre>

====== add current user to kmem group ======
<pre>
sudo adduser $USER kmem
</pre>

====== give group kmem access to /dev/mem
<pre>
echo 'SUBSYSTEM=="mem", KERNEL=="mem", GROUP="kmem", MODE="0660"' | sudo tee /etc/udev/rules.d/98-mem.rules
sudo reboot
</pre>

====== uncomment the BCM2835_HAVE_LIBCAP line in src/bcm2835.h ======
<pre>
sed -i 's/\/\/#define BCM2835_HAVE_LIBCAP/#define BCM2835_HAVE_LIBCAP/' src/bcm2835.h
</pre>

====== compile libbcm2835 as normal ======
<pre>
./configure
make
sudo make install # (make check fails)
</pre>

====== in addition to -lbcm2835, compilation of example or own code requires -lcap, e.g. ======
<pre>
gcc -o spi spi.c -l bcm2835 -lcap
</pre>

====== after compilation, give program permission to access cap ======
<pre>
sudo setcap cap_sys_rawio+ep <progname>
</pre>

===== C++ =====
C++ classes to simplify the use are provided in the mems code.

Laser calibration

2020-07-21T20:50:15Z

Lmartin:

== Introduction ==
Both the prototype and final TPC have aluminium strips on the central cathode that are intended for laser calibration. To that end a plane of UV laser light is sent into the TPC intersecting with the strips. Each laser pulse should then create a bunch of electrons with well defined position and time at each of the strips.

== Photoelectron efficiency tests ==
One worry with the intended geometry is that the laser hits the far strips at a very glancing angle, and it was assumed the photoelectron efficiency (i.e. number of photoelectrons per photon) would go down in favour of reflection. To prevent this two methods of surface treatment were considered, and both the validity of the aforementioned assumption as well as the effectiveness of the treatments were investigated in a test setup at UBC. Details and results are given in this [[Media:LaserTests.pdf|report]].

== Laser operating procedure ==
The following describes the standard operating procedure for running the laser setup on the prototype.
[[File:Control.jpg|thumb|Handheld control unit]]

=== Safety ===
<big>
''NEVER have the key inserted in the "ignition" when the optical fiber is disconnected, the black box is open, or the TPC is open (e.g. gas lines disconnected).''
</big>
=== Start-up ===
[[File:Shutter.jpg|thumb|Manual shutter: left open, right closed]]
# Check integrity of enclosure, i.e. optics box closed and bolted down, fiber attached at both ends.
# Turn on laser system with key on main unit.
# Push button "Flash lamp -> Ready" on handheld unit.
# Push button "Flash lamp -> Start" on handheld unit.
# Set collimator via mvat program on computer.
# Open manual shutter on laser head.
# Push button "Q-switch -> Start" on handheld unit.

=== Pause ===
==== Short pause ====
Simply push stop/start for Q-switch.
==== Longer pause ====
# Stop Q-switch.
# Stop flash lamp.
# Close manual shutter.
# After pause restart according to after-key part of start-up procedure.

=== Shutdown ===
# Stop Q-switch.
# Stop flash lamp.
# Close manual shutter.
# Wait a few minutes for cool-down.
# Turn off and remove key.

=== General best practice ===
* Limit laser shots to a minimum, to reduce wear on laser, flash lamp, and fiber.
* To that end, do non-critical steps like attenuator setting and DAQ start before starting Q-switch.
* Ideally allow system some time to warm up between turning on flash lamp and Q-switch.

== Optics ==

== Laser requirements ==

== MEMS Multiplexer ==

=== Raspberry Pi setup ===
==== Operating System ====
We're running the most recent version of Raspberry Pi OS as downloaded from [https://www.raspberrypi.org/downloads/ raspberrypi.org]

==== Required packages ====
Install the following with apt get:
* emacs
* libcap-dev (required to access SPI as non-root user)

==== SPI library ====
The recommended way to interface with SPI from a raspberry pi is now no longer using the wiringPi library, instead [https://www.raspberrypi.org/documentation/hardware/raspberrypi/spi/README.md raspberrypi.org] recommends [http://www.airspayce.com/mikem/bcm2835/ bcm2835]. This is a direct hardware C library for pin communication that includes examples for GPIO, SPI, and I2C.

===== Access SPI as non-root user =====
Following the instructions from [http://www.airspayce.com/mikem/bcm2835/] non-root access to the SPI pins requires the following:
====== install libcap-dev (see [[Laser_calibration#Required_packages|Required packages]]) ======
<pre>
sudo apt install libcap-dev
</pre>
====== uncomment the BCM2835_HAVE_LIBCAP line in src/bcm2835.h ======
<pre>
sed -i 's/\/\/#define BCM2835_HAVE_LIBCAP/#define BCM2835_HAVE_LIBCAP/' src/bcm2835.h
</pre>
====== compile libbcm2835 as normal ======
<pre>
./configure
make
sudo make install # (make check fails)
</pre>
====== in addition to -lbcm2835, compilation of example or own code requires -lcap, e.g. ======
<pre>
gcc -o spi spi.c -l bcm2835 -lcap
</pre>

===== C++ =====
C++ classes to simplify the use are provided in the mems code.

Laser calibration

2020-07-21T20:45:45Z

Lmartin:

== Introduction ==
Both the prototype and final TPC have aluminium strips on the central cathode that are intended for laser calibration. To that end a plane of UV laser light is sent into the TPC intersecting with the strips. Each laser pulse should then create a bunch of electrons with well defined position and time at each of the strips.

== Photoelectron efficiency tests ==
One worry with the intended geometry is that the laser hits the far strips at a very glancing angle, and it was assumed the photoelectron efficiency (i.e. number of photoelectrons per photon) would go down in favour of reflection. To prevent this two methods of surface treatment were considered, and both the validity of the aforementioned assumption as well as the effectiveness of the treatments were investigated in a test setup at UBC. Details and results are given in this [[Media:LaserTests.pdf|report]].

== Laser operating procedure ==
The following describes the standard operating procedure for running the laser setup on the prototype.
[[File:Control.jpg|thumb|Handheld control unit]]

=== Safety ===
<big>
''NEVER have the key inserted in the "ignition" when the optical fiber is disconnected, the black box is open, or the TPC is open (e.g. gas lines disconnected).''
</big>
=== Start-up ===
[[File:Shutter.jpg|thumb|Manual shutter: left open, right closed]]
# Check integrity of enclosure, i.e. optics box closed and bolted down, fiber attached at both ends.
# Turn on laser system with key on main unit.
# Push button "Flash lamp -> Ready" on handheld unit.
# Push button "Flash lamp -> Start" on handheld unit.
# Set collimator via mvat program on computer.
# Open manual shutter on laser head.
# Push button "Q-switch -> Start" on handheld unit.

=== Pause ===
==== Short pause ====
Simply push stop/start for Q-switch.
==== Longer pause ====
# Stop Q-switch.
# Stop flash lamp.
# Close manual shutter.
# After pause restart according to after-key part of start-up procedure.

=== Shutdown ===
# Stop Q-switch.
# Stop flash lamp.
# Close manual shutter.
# Wait a few minutes for cool-down.
# Turn off and remove key.

=== General best practice ===
* Limit laser shots to a minimum, to reduce wear on laser, flash lamp, and fiber.
* To that end, do non-critical steps like attenuator setting and DAQ start before starting Q-switch.
* Ideally allow system some time to warm up between turning on flash lamp and Q-switch.

== Optics ==

== Laser requirements ==

== MEMS Multiplexer ==

=== Raspberry Pi setup ===
==== Operating System ====
We're running the most recent version of Raspberry Pi OS as downloaded from [https://www.raspberrypi.org/downloads/ raspberrypi.org]

==== Required packages ====
Install the following with apt get:
* emacs
* libcap-dev (required to access SPI as non-root user)

==== SPI library ====
The recommended way to interface with SPI from a raspberry pi is now no longer using the wiringPi library, instead [https://www.raspberrypi.org/documentation/hardware/raspberrypi/spi/README.md raspberrypi.org] recommends [http://www.airspayce.com/mikem/bcm2835/ bcm2835]. This is a direct hardware C library for pin communication that includes examples for GPIO, SPI, and I2C.

===== Access SPI as non-root user =====
Following the instructions from [http://www.airspayce.com/mikem/bcm2835/] non-root access to the SPI pins requires the following:
# install libcap-dev (see [Required packages])
# Uncomment the BCM2835_HAVE_LIBCAP line in src/bcm2835.h
<pre>
sed -i 's/\/\/#define BCM2835_HAVE_LIBCAP/#define BCM2835_HAVE_LIBCAP/' src/bcm2835.h
</pre>
# compile libbcm2835 as normal, make, sudo make install (make check fails)
# in addition to -lbcm2835, compilation of example or own code requires -lcap

===== C++ =====
C++ classes to simplify the use are provided in the mems code.

Laser calibration

2020-07-21T20:20:52Z

Lmartin:

== Introduction ==
Both the prototype and final TPC have aluminium strips on the central cathode that are intended for laser calibration. To that end a plane of UV laser light is sent into the TPC intersecting with the strips. Each laser pulse should then create a bunch of electrons with well defined position and time at each of the strips.

== Photoelectron efficiency tests ==
One worry with the intended geometry is that the laser hits the far strips at a very glancing angle, and it was assumed the photoelectron efficiency (i.e. number of photoelectrons per photon) would go down in favour of reflection. To prevent this two methods of surface treatment were considered, and both the validity of the aforementioned assumption as well as the effectiveness of the treatments were investigated in a test setup at UBC. Details and results are given in this [[Media:LaserTests.pdf|report]].

== Laser operating procedure ==
The following describes the standard operating procedure for running the laser setup on the prototype.
[[File:Control.jpg|thumb|Handheld control unit]]

=== Safety ===
<big>
''NEVER have the key inserted in the "ignition" when the optical fiber is disconnected, the black box is open, or the TPC is open (e.g. gas lines disconnected).''
</big>
=== Start-up ===
[[File:Shutter.jpg|thumb|Manual shutter: left open, right closed]]
# Check integrity of enclosure, i.e. optics box closed and bolted down, fiber attached at both ends.
# Turn on laser system with key on main unit.
# Push button "Flash lamp -> Ready" on handheld unit.
# Push button "Flash lamp -> Start" on handheld unit.
# Set collimator via mvat program on computer.
# Open manual shutter on laser head.
# Push button "Q-switch -> Start" on handheld unit.

=== Pause ===
==== Short pause ====
Simply push stop/start for Q-switch.
==== Longer pause ====
# Stop Q-switch.
# Stop flash lamp.
# Close manual shutter.
# After pause restart according to after-key part of start-up procedure.

=== Shutdown ===
# Stop Q-switch.
# Stop flash lamp.
# Close manual shutter.
# Wait a few minutes for cool-down.
# Turn off and remove key.

=== General best practice ===
* Limit laser shots to a minimum, to reduce wear on laser, flash lamp, and fiber.
* To that end, do non-critical steps like attenuator setting and DAQ start before starting Q-switch.
* Ideally allow system some time to warm up between turning on flash lamp and Q-switch.

== Optics ==

== Laser requirements ==

== MEMS Multiplexer ==

=== Raspberry Pi setup ===
==== Operating System ====
We're running the most recent version of Raspberry Pi OS as downloaded from [https://www.raspberrypi.org/downloads/ raspberrypi.org]

==== Required packages ====
Install the following with apt get:
* emacs
* libcap-dev (required to access SPI as non-root user)

==== SPI library ====
The recommended way to interface with SPI from a raspberry pi is now no longer using the wiringPi library, instead [https://www.raspberrypi.org/documentation/hardware/raspberrypi/spi/README.md raspberrypi.org] recommends [http://www.airspayce.com/mikem/bcm2835/ bcm2835]. This is a direct hardware C library for pin communication that includes examples for GPIO, SPI, and I2C.

C++ classes to simplify the use are provided in the mems code.

Laser calibration

2020-07-21T19:37:50Z

Lmartin: Starting to document MEMS stuff

== Introduction ==
Both the prototype and final TPC have aluminium strips on the central cathode that are intended for laser calibration. To that end a plane of UV laser light is sent into the TPC intersecting with the strips. Each laser pulse should then create a bunch of electrons with well defined position and time at each of the strips.

== Photoelectron efficiency tests ==
One worry with the intended geometry is that the laser hits the far strips at a very glancing angle, and it was assumed the photoelectron efficiency (i.e. number of photoelectrons per photon) would go down in favour of reflection. To prevent this two methods of surface treatment were considered, and both the validity of the aforementioned assumption as well as the effectiveness of the treatments were investigated in a test setup at UBC. Details and results are given in this [[Media:LaserTests.pdf|report]].

== Laser operating procedure ==
The following describes the standard operating procedure for running the laser setup on the prototype.
[[File:Control.jpg|thumb|Handheld control unit]]

=== Safety ===
<big>
''NEVER have the key inserted in the "ignition" when the optical fiber is disconnected, the black box is open, or the TPC is open (e.g. gas lines disconnected).''
</big>
=== Start-up ===
[[File:Shutter.jpg|thumb|Manual shutter: left open, right closed]]
# Check integrity of enclosure, i.e. optics box closed and bolted down, fiber attached at both ends.
# Turn on laser system with key on main unit.
# Push button "Flash lamp -> Ready" on handheld unit.
# Push button "Flash lamp -> Start" on handheld unit.
# Set collimator via mvat program on computer.
# Open manual shutter on laser head.
# Push button "Q-switch -> Start" on handheld unit.

=== Pause ===
==== Short pause ====
Simply push stop/start for Q-switch.
==== Longer pause ====
# Stop Q-switch.
# Stop flash lamp.
# Close manual shutter.
# After pause restart according to after-key part of start-up procedure.

=== Shutdown ===
# Stop Q-switch.
# Stop flash lamp.
# Close manual shutter.
# Wait a few minutes for cool-down.
# Turn off and remove key.

=== General best practice ===
* Limit laser shots to a minimum, to reduce wear on laser, flash lamp, and fiber.
* To that end, do non-critical steps like attenuator setting and DAQ start before starting Q-switch.
* Ideally allow system some time to warm up between turning on flash lamp and Q-switch.

== Optics ==

== Laser requirements ==

== MEMS Multiplexer ==

=== Raspberry Pi setup ===
==== Operating System ====
We're running the most recent version of Raspberry Pi OS as downloaded from [https://www.raspberrypi.org/downloads/ raspberrypi.org]

==== Required packages ====
Install the following with apt get:
* emacs

==== SPI library ====
The recommended way to interface with SPI from a raspberry pi is now no longer using the wiringPi library, instead [https://www.raspberrypi.org/documentation/hardware/raspberrypi/spi/README.md raspberrypi.org] recommends [http://www.airspayce.com/mikem/bcm2835/ bcm2835]. This is a direct hardware C library for pin communication that includes examples for GPIO, SPI, and I2C.

C++ classes to simplify the use are provided in the mems code.

Test page for MediaWiki

2017-09-22T18:05:30Z

Lmartin:

'''MediaWiki has been successfully installed.'''

Consult the [//meta.wikimedia.org/wiki/Help:Contents User's Guide] for information on using the wiki software.

== Getting started ==
* [//www.mediawiki.org/wiki/Manual:Configuration_settings Configuration settings list]
* [//www.mediawiki.org/wiki/Manual:FAQ MediaWiki FAQ]
* [https://lists.wikimedia.org/mailman/listinfo/mediawiki-announce MediaWiki release mailing list]

== Test ==

test123
test333
test456
test789

[[Image:Web_security2.svg]]

last last last

[[Image:20150422_514701.jpg]]

{|
! [[File:media/image1.jpeg|299x82px]]
!

! '''Document-131949'''
|-
| '''Design Note TRI-DN-16-22 ALPHA-g Data Acquisition Notes'''
|-
|

|

|-
| '''Document Type:'''
| '''Design''' '''Note'''
|-
| '''Release:'''
| '''2'''
| '''Release Date:'''
|-
| '''Author(s):'''
| '''Pierre Amaudruz'''
|-
|

|

|

|-
|

| '''Name:'''
| '''Signature:'''
|-
| '''Author:'''
| '''Konstantin Olchanski '''
|

|-
|

| '''Pierre Amaudruz'''
|

|-
| '''Reviewed By:'''
| '''Thomas Lindner'''
|

|-
| '''Approved By:'''
| '''Makoto Fujiwara'''
|

|}

History of Changes

{|
! Release Number
! Date
! Description of Changes
! Author(s)
|-
| 1
| 2016-02-28
| Initial Version
| Pierre Amaudruz
|-
| 2
| 2017-04-10
|

| Pierre Amaudruz
|-
| 3
| 2017-08-08
| Revise version without Grif-C
| PAA, KO
|-
|

|

|

|

|}

'''Keywords:'''

'''Distribution List:'''

= Table of Figures =

[[#_Toc493687873|''Figure 1 - Overall dataflow for scheme from the detector to the data storage area.'' 10]]

[[#_Toc493687874|''Figure 2 - Frontend local fragment assembly, similar data for the event builder. Each grey box refers to a Thread collecting and filtering locally its fragment before making it available to the fragment builder'' 11]]

= Introduction =

The ALPHA-g experiment is the next step in the line of successful anti-hydrogen experiments at CERN - ATHENA, ALPHA, ALPHA-2. These experiments have been designed, built and operated by mostly the same group of people, making it important to maximally maintain similarity and compatibility between the existing ALPHA-2 data acquisition system and the new ALPHA-g system. The ALPHA-2 and ALPHA-g machines are expected to operate together, at the same time, by the same set of people, for several years.

Looking back, the ATHENA experiment introduced the concept of using an advanced Si-strip particle detector to image annihilations of anti-hydrogen. The ALPHA experiment brought in the highly reliable MIDAS data acquisition software and the modular VME-based electronics (built at/for TRIUMF). The ALPHA-2 experiment saw the data acquisition system expanded by 50% without having to make any architectural changes.

ALPHA-g continues to build on the foundation, but because a 2 meter long Si strip detector is not practical, a TPC gas detector will be used instead.

= Scope and Assumptions =

Similar to ALPHA-2, ALPHA-g will consist of several computer controls systems in addition to data acquisition for the annihilation imaging detector.

There will be Labview and CompactPCI-based control systems for trap vacuum, for magnet controls, for positron source controls, for trap controls, for microwave and laser system controls.

All these systems are outside the scope of this document, except where, same as in ALPHA-2, they will log data into the MIDAS history system via network socket connections.

The data acquisition system for the particle detector includes a number of hardware components - digitizers for analog signals, firmware-based signal processing, trigger decision making, etc.

All these components are outside of the scope of this document, except where they require network communications for receiving experiment data, for configuration management and for monitoring. For details on the electronics see TRI-DN-16-15.

Thus the scope of this document is limited to the software (and to hardware required to run it) that receives experiment data across the network, communicates with other hardware and software systems over the network.

= Definitions and Abbreviations =

General acronyms or terms used for the ALPHA-g experiment

{|
| rTPC
| Radial Time projection Chamber
|-
| AD
| Antiproton Decelerator (CERN building where ALPHA-g is installed)
|-
| Trap
| Device where the Antihydrogen annihilation take place within the cryostat.
|-
| AW
| TPC Anode Wires
|-
| BSC
| Barrel Scintillator Detector
|-
| GRIFFIN
| Major Experiment facility at TRIUMF
|-
| ALPHA-16, ALPHA-T, FMC-32
| Similar electronics designed and built at Triumf for Griffin
|-
| MIDAS
| Data Acquisition software package from PSI & TRIUMF
|-
| MIDAS ODB
| Shared memory tree-structured database
|-
| SiPM
| Solid-state light sensors used in the BSC detector
|-
|

|

|-
| TPC AFTER ASIC
| Custom chip from Saclay for recording TPC analog signals
|-
| TPC pad board (FEAM)
| TPC Electronics signed and build at TRIUMF for ALPHA-g
|}

''Table 1 – ALPHA-g Abbreviations''

= References and related Document =

[http://midas.triumf.ca Midas documentation]

[https://daq.triumf.ca/elog-alphag/alphag/ Experiment Elog (restricted)]

[https://daq.triumf.ca/DaqWiki/index.php/Components Alpha-g Component Wiki]

[https://edev-group.triumf.ca/fw/exp/alphag Edev-gitalb repository (restricted)]

TRI-DN-16-15: ALPHA-G Detector Electronics

= Specifications =

The Data Acquisition system is overlooking all the detector parameters settings and manage the collection, analysis, recording and monitoring of the detector incoming data. These tasks can be described as follow.

== Handling of experiment data ==

Experiment data will arrive over multiple 1GigE network links into the main DAQ computer, will be processed and transmitted to permanent storage at the main CERN data center, also across a 1GigE link.

The existing network link from CERN shared by ALPHA-2 and ALPHA-g runs at 1GigE speed and the collaboration does not plan to request that CERN upgrade this link to 10GigE or faster speed.

This limits the overall data throughput to 100 Mbytes/sec shared with ALPHA-2 (which in turn is limited to 20-30 Mbytes/sec).

A Local Network switch (96x1GigE) gather all the different data sources such as:

* 16 AW and BSC digitizers (FMC-32, ALPHA-16)
** The 512 Anode wires channels (256 wires readout both ends) are digitized with 16 FMC-32 mezzanines of the ALPHA-16). Each ALPHA-16 has an Ethernet data link to the main LAN switch.
** The 128 channels of the BSB are collected through the same ALPHA-16 boards.
* 64 FEAMs cathode pad readout
** The 18432 cathode pads are digitized with 64 FEAMs which each has a optical Ethernet link to the main LAN switch.
**
* The ALPHA-T trigger board has a link to the main LAN switch as well.
* Barrel Scintillator TDC (see Electronics document)

A total of 82 x 1GigE ports out of the 96 from the switch are allocated to the main detector operation.

== ==

== Handling of Labview and Trap sequencer data ==

Same as in ALPHA-2, there will be a special MIDAS frontend to receive data from labview-based control systems (vacuum, magnets, trap sequencer, etc.). Data communication protocol and data format will be the same in ALPHA-2.

== Configuration data ==

All configuration data will be stored in the MIDAS ODB database.

Major configuration items:

<ul>
<li>Trigger configuration (multiplicity settings, channel mapping, dead channel blanking, etc.)</li>
<li>AW and BSC digitizer settings</li>
<li>TPC pad board settings and AFTER ASIC settings (FEAMs)</li>
<li>UPS uninterruptible power supply with battery backup and "double conversion" power conditioning
<ol style="list-style-type: decimal;">
<li>== Slow Controls data ==
</li></ol>
</li></ul>

All slow controls settings will be stored in the MIDAS ODB database. Slow controls data will be logged through the MIDAS history system. Integration with the ALPHA-II DAQ system will require additional transfer of ALPHA-g control data to it. This being done in MySQL format.

Major slow controls items:

* Low voltage power supply controls
* High voltage power supply controls
* Environment monitoring (temperature probes, etc.)
* SiPM bias voltage controls
* TPC gas system control and monitoring
* TPC calibration system controls
* cooling system control and monitoring
* UPS monitoring

Some slow controls components will have software interlocks for non-safety critical protection of equipment, such as automatic shutdown of electronics on overheating, automatic shutdown of detectors on UPS alarms (i.e. loss of external AC power).

(It is assumed that safety-critical interlocks will be implemented in hardware, this is outside the scope of this particular document).

= Trigger consideration =

The Alpha-g physics experiment follows different particle trapping stages such as: Positron accumulation, Antiproton accumulation, mixing stage for formation of anti-hydrogen in the trap and annihilation stage for the actual measurement. The whole cycle takes several minutes and needs to be constantly monitored with the detector to ensure that external background events (cosmics, noise) are properly identified by the detector and that annihilation events from the trap are all recorded for further analysis. This is mainly done using the barrel scintillator which will record external incoming events and internal trapped events (annihilation). Timing analysis will resolve the distinction of the two main sources of events.

Therefore the barrel scintillator is the main contributor to the trigger decision. Additional information from the anode wires is beneficial even though its timing is extremely loose due to the implicit drift time range of the detector.

The Trigger decision is based initially on simple multiplicity of the barrel scintillator detector which tags the event to record. This multiplicity is computed in each of the 16 Alpha-16 boards and transmitted to the Mater Trigger Module (Alpha-T). There the individual fragments are matched in time and final trigger decision is taken, which in turn will be redistributed to all the acquisition modules.

In case the anode wire information is needed as well, the same fragment collection scheme is done for the wire and sent through the same physical link as the Alpha-16 modules are populated with the 32 channels at 62.5Msps mezzanine (FMC-32).

The Data Acquisition system (MIDAS) will not be taking part to the trigger decision, therefore will have no interaction with the hardware for the trigger decision. The data stream will constantly be pulled from Ethernet of the acquisition modules and an “event builder” application using the timestamp information attached to each received packet for fragment reassembly. The hardware trigger module (ALPHA-T) will keep track of the trigger decision and produce trigger data which merged to the main data acquisition stream.

= Midas Software configuration =

For maximum familiarity, similarity and compatibility with the existing ALPHA-2 system, ALPHA-g will also use the MIDAS data acquisition software package.

The main features of MIDAS is a modern web-based user interface, good robustness, high reliability and high scalability. The system is easy to operate by non-experts.

Outside of ALPHA-2 at CERN, other major users of MIDAS are TIGRESS and GRIFFIN at TRIUMF, MEG at PSI, DEAP at SNOLAB, T2K/ND280 at JPARC (Japan).

Similar to ALPHA-2 the MIDAS software will be running on one single high-end computer with an Intel socket 1151 CPU and 32/64 GB RAM. This is adequate for achieving the 100 Mbytes/sec data rates expected for ALPHA-g. During tests of 10GigE networking for the GRIFFIN experiment at TRIUMF this type of hardware was capable of recording 1000 Mbytes/sec of data from network to disk. The required disk I/O capability will be absent in ALPHA-g. The intention is to stream the data directly to CASTOR data storage provided by CERN.

This computer will have 2 1GigE network interfaces, one for connecting to the CERN network, the second for connecting to the ALPHA-g private network.

Data storage will be dual mirrored (RAID1) SSDs for the operating system and home directories and dual mirrored (RAID1) 6TB HDDs for local data storage.

The MIDAS software running on this computer will have following components:

* Mhttpd - the MIDAS web server providing the main user interface for running the system
* Mlogger - the MIDAS data logger for writing experiment data to local storage and for recording MIDAS slow controls history data
* Lazylogger - for moving experiment data from local storage to permanent storage at the CERN data center (EOS and Castor).
* MIDAS frontend programs custom written to receive experiment data
* The event builder - for assembling experiment data from different frontends into events to simplify further analysis. The event builder will have code for initial data quality tests (correct data formatting, correct event timestamps, etc.).
* The online analyzer - for initial data analysis, visualization and additional data quality checks.

The MIDAS frontend programs are written using a common template, with each frontend program specialized to receive data in the format of the attached equipment:

* Frontend for AW and BSC digitizers - receive data from ALPHA-16 digitizers using TCP and UDP network socket connections (will use a modification of the frontend from GRIFFIN), provide configuration and monitoring (slow controls) of the digitizers.
* Frontend for the TPC pad boards - receive data from the TPC using TCP and UDP network socket connections. This frontend will also provide configuration and monitoring (slow controls) of the TPC pad boards and of the TPC AFTER ASICs.
* Frontend for the clock board - configuration and control of the clock board using UDP and TCP network connections.
* Frontends for slow controls, according to equipment (see list in the "specification" section). Most frontend programs will be reused from previous experiments (i.e. UPS monitoring frontends from DEAP, ALPHA-II).

Data from all the hardware components arrives through the MIDAS frontend programs asynchronously. This makes further analysis and data quality tests rather difficult. To rearrange the data into sensible pieces MIDAS provides an event builder component.

The event builder program is usually written specifically for each experiment (also using a MIDAS template). The ALPHA-g event builder will check incoming data for basic correctness, assemble event fragments from different frontends into coherent events, and check timestamps and event numbers, possibly apply additional data reduction and data compression algorithms. We expect to reuse the multi-threaded event builder developed for DEAP as the basis of the ALPHA-g event builder.

The online analyzer will look at a sample of events produced by the event builder and produce histograms and other plots of most important quantities (pulse heights, timing, etc) to confirm correct function of the detector components (i.e. each TPC pad produces hits) and of the detector as whole (i.e. hit multiplicity in the TPC pads matches hit multiplicity in the TPC AW and in the BSC). We do not expect to do full tracking and vertex reconstruction in the online analyzer - this will be done in the "near offline" analysis using the CERN data center resources (similar to ALPHA-2 procedures).

The mlogger component writes MIDAS data to local disk (or remote disk using NFS or UNIX pipe methods) and provides standard data compression (gzip-1 by default, bzip2 for maximum compression and LZ4 for maximum speed). As an option CRC32c and SHA-256/SHA-512 checksums can be computed to detect data corruption.

''Figure 1, 2'' and the diagram below illustrate the data flow described in this section:

[[File:media/image2.png|576x386px]]

''Figure 1 - Overall dataflow for scheme from the detector to the data storage area.''

[[File:media/image3.png|536x343px]]

''Figure 2 - Frontend local fragment assembly, similar data for the event builder. Each grey box refers to a Thread collecting and filtering locally its fragment before making it available to the fragment builder ''

= Computer Network configuration =

Operation of the ALPHA-g experiment will involve 2 computer networks: the ALPHA-g private network and the CERN network.

The private network is required to isolate experiment data traffic from the CERN network data traffic (this was not required for ALPHA-2, where there is no private network) and to restrict access to network-attached slow controls devices (power supplies, gas system controllers, etc.).

This private network (192.168.x.y) will be connected to 2x 1GigE network switches (96, 24ports). The first one will connect all the FEAM (64) and the ALPHA-16 (16). The other networked equipment such as the slow controls/monitoring devices etc, will be connected to the second switch (24 ports).

The 1GigE network port of the main data acquisition computer will connect to the CERN network for transmitting experiment data to the CERN data center, for receiving experiment data from the Labview based control systems (which are outside of the scope of the particle detector and of this document).

The CERN network currently consists of a single fiber 1GigE network link feeding a small (24-port) 1GigE network switch in the experiment counting room. It is not anticipated that CERN will upgrade this link to higher speeds (10GigE, etc.) within the lifetime of ALPHA-g and it is not expected that ALPHA-g will ever require these higher speeds because excessive amount of data recorded to CERN storage will create significant problems for data analysis. If the detector produces too much data, it will be handled by using stronger data reduction and data compression algorithms.

The ALPHA-g data acquisition computer will provide one-way network gateway connection from the private network to the CERN network using Linux based NAT. This is required to provide devices attached to the private network with access to DNS, NTP and other network services, i.e. for installing software and firmware updates.

This configuration is very similar to what is in use by the DEAP experiment at SNOLAB. The main difference is ALPHA-g will run the main data acquisition and NAT/gateway services on the same computers as opposed to two separate computers at DEAP.

= Computer Security considerations =

In the ALPHA-2 and ALPHA-g experiments, p-bar beam time is very precious and any disturbances during beam data taking must be minimized.

One the computer network side, the ALPHA-g data acquisition system needs to be protected from:

* Unauthorized access to proprietary information
* Unauthorized access to network-controlled devices (power supplies, etc.)
* Sundry network disturbances (unwanted network port scans, unwanted web scans, accidental or malicious denial-of-service attacks, etc.).

It is important to note that the CERN network is isolated from the main Internet by the CERN firewall. Generally, the CERN firewall allows all outgoing traffic rom CERN to the Internet (can be selectively blocked by special request to CERN IT), but blocks all incoming traffic, except where specifically permitted. A typical machine connected to the CERN network can talk to anywhere in the Internet, but from the Internet, only "ping" is permitted. "ssh" access is disabled (one is supposed to ssh gateways at lxplus.cern.ch) and "http" access is allowed by special permission from CERN IT.

Thus we do not need to worry about protecting the ALPHA-g data acquisition system against "global Internet" threats (the CERN firewall does this job). Protection is required only against access from other machines attached to the CERN network, which can be generally considered as non-malicious. Still, over the years of operating ALPHA and ALPHA-2, we have seen unwanted network scans (that required special hardening of MIDAS) and other strange activity.

The security model of ALPHA-g combines the security features of ALPHA-2 and the security features of DEAP (at SNOLAB).

Access to the MIDAS Web server will be exclusively through the apache httpd proxy with password-protected HTTPS connections (same as in ALPHA, ALPHA-2 and DEAP).

Access to machines in the private network will be generally blocked (by Linux firewall rules and NAT). Exceptions like access to UPS controls, webcams, etc. will be via the http proxy, with password-protected HTTPS connections, same as in DEAP.

Access to MIDAS ports in the data acquisition computer will be blocked by Linux firewall rules. Exceptions will be made for specific connections such as Labview controls sending data to MIDAS.

In ALPHA, ALPHA-2 and DEAP the apache httpd proxy server runs on a separate machine. In ALPHA-g we may run it on the same machine as the main data acquisition, subject to approval by CERN (they have to open HTTPS access in the CERN firewall - from the Internet to the HTTPS port on the ALPHA-g DAQ computer).

Over the many years of operating ALPHA, ALPHA-2, DEAP and other similarly configured experiments, this security model has worked well for us, with no complaints from experiment users and no security incidents.

= Safety and Hazard considerations =

There are no safety and hazard concerns about any software or equipment inside the scope of this document.

All hardware consists of commodity computer components (computers, network switches, etc.) that carries appropriate government safety certifications (i.e. CSA in Canada).

All software and software controlled equipment will have hardware-enforced limits and interlocks to ensure safe operation even in case of software failure. (This is outside the scope of this document, for more information, please refer to documents or each relevant hardware component).

Laser calibration

2017-05-12T00:31:42Z

Lmartin:

Laser calibration

2017-05-12T00:29:47Z

Lmartin: Safety notice

Laser calibration

2017-05-12T00:07:50Z

Lmartin: Laser operating procedure

File:Shutter.jpg

2017-05-12T00:06:34Z

Lmartin:

Manual shutter: left open, right closed

File:Control.jpg

2017-05-12T00:04:47Z

Lmartin:

Handheld control unit

Cathode Pad Electronics

2017-05-05T00:33:42Z

Lmartin:

The cathode pads are read-out by AFTER ASIC. One chip has 72 active channels and 4 reference channels for pedestal subtraction. Each channel consists of charge-sensitive preamplifier, pole-zero cancellation of preamplifier decay time, RC2 shaper and 512-cell SCA to store waveform. Amplifier gain, pole-zero time constant and shaping time are programmable.
Detailed information is in the manual, but the most relevant slow control settings are summarized in the following tables.

The full range of the AFTER chip and the ADC is ±1V, the gain setting determines what charge that corresponds to, leading to a gain value.
{| class="wikitable"
|-
! Gain setting !! Charge range (fC) !! Gain (mV/fC)
|-
| 0 || 120 || 8.333
|-
| 1 || 240 || 4.167
|-
| 2 || 360 || 2.778
|-
| 3 || 600 || 1.667
|}

The peaking time is defined as the time the pulse takes to go from 5% to 100% pulse height.
This peaking time translates to other relevant times as shown in the next table.
[[File:Shape.png|thumb|Illustration of various time characteristics of the filtered AFTER signal]]
{| class="wikitable"
|-
! Peaking time (5%-100%) !! Fall time (100%-5%) !! FWHM !! Rise time (50%-100%) !! Fall time (100%-50%)
|-
| 116.7 || 202.5 || 158.4 || 67.59 || 90.82
|-
| 200 || 438.2 || 311.6 || 121.3 || 190.3
|-
| 505 || 1194 || 843 || 329.2 || 513.7
|-
| 993 || 2351 || 1689 || 656.1 || 1033
|-
| 1912 || 4283 || 3213 || 1268 || 1945
|}

Phenomenologically the pulse shape can be described by the function

<math>f(t)=\left(\frac{t}{k\tau}\right)^k \exp\left(k-\frac{t}{\tau}\right)</math>

Extracting the above mentioned rise and fall times from this formula for different values for the parameter k yields a relatively good match for <math>k=4</math>. The resulting numerical relationship between peaking time <math>t_r</math> and <math>\tau</math> from this is
<math>t_r = 3.1365\tau</math>

The following tables contains the 5%-100% rise times for the different AFTER settings together with the corresponding value for <math>\tau</math>.

{| class="wikitable"
|-
! Time setting !! Peaking time (ns) !! <math>\tau</math> (ns)
|-
| 0 || 116 || 37.0
|-
| 1 ||200 || 63.8
|-
| 2 || 412 || 131.4
|-
| 3 || 505 || 161.0
|-
| 4 || 610 || 194.5
|-
| 5 || 695 || 221.6
|-
| 6 || 912 || 290.8
|-
| 7 || 993 || 316.6
|-
| 8 || 1054 || 336.0
|-
| 9 || 1134 || 361.5
|-
| 10 || 1343 || 428.2
|-
| 11 ||1421 || 453.1
|-
| 12 || 1546 || 492.9
|-
| 13 || 1626 || 518.4
|-
| 14 || 1834 || 584.7
|-
| 15 || 1912 || 609.6
|}

=== Reference ===
[[File: AFTER_MANUAL_v2.pdf]]

Cathode Pad Electronics

2017-05-04T23:15:45Z

Lmartin:

Cathode Pad Electronics

2017-05-04T19:44:03Z

Lmartin:

File:Shape.png

2017-05-04T19:29:15Z

Lmartin:

Illustration of various time characteristics of the signal

Laser calibration

2017-02-18T03:56:47Z

Lmartin: Initial commit, including UBC test pdf

File:LaserTests.pdf

2017-02-18T03:52:07Z

Lmartin: Report of the UBC laser tests on the angle-dependence of photoelectron efficiency

Report of the UBC laser tests on the angle-dependence of photoelectron efficiency

Experiments

2017-02-18T03:41:58Z

Lmartin:

= Alpha-G =

* [[Components]]
* Sub-systems
** [[rTPC Detector]]
** [[Data Acquisition Infrastructure]]
** [[Anode Wire Electronics]]
** [[Cathode Pad Electronics]]
** [[Backend Electronics]]
** [[Laser calibration]]
** [[Water cooling system]]
** [[Gas System]]
** [[Dry Air flushing system]]
** [[Barrel Scintillator detector]]
** [[Barrel Scintillator Electronics]]