Infoblox is presented as a webpage which allows you to set DNS and DHCP settings for both network and host management.

= Basic Usage =

The infoblox website is located here: https://infoblox.triumf.ca. You log in with your TRIDENT credentials (without the "@triumf.ca" on your username). You will be presented with a rather empty status screen. The actions needed are taken from the "Data Management" tab, highlighted in red below:

[[File:Infoblox status.png|thumb|1000px|left|Infoblox status page. Most basic actions are accessed through the Data Management tab, highlighted in red.]]

[[File:Infoblox datamanagement2.png|thumb|1000px|left|Infoblox Data Management IPAM tab.
The Data Management tab has several sub tabs. They are mostly different ways of doing the same thing. The most useful one for our purposes is the IPAM (IP Address Management) tab. This one loads by default. Once there select your network. ]]

[[File:Infoblox subnet.png|1000px|thumb|left|Infoblox IPAM select subnet page. In the case for this example, we will use the UCN subnet.]]

[[File:Infoblox select ip.png|1000px|thumb|left|Infoblox IP selection and device list. From here you will see all the devices and their IP addresses. You can edit the columns shown by hovering over any of the column headers and selecting the down arrow. We can add a new host by selecting an IP address with no assigned hostname. To edit a host's details click on that host's IP address. ]]

[[File:Infoblox add host.png|1000px|thumb|left|To add a host click on the "+"]]

[[File:Infoblox add host wizard step1.png|1000px|thumb|left| This then opens up the Add Host Wizard. On the first step we type in the hostname of our new host, the MAC address and click the DHCP checkbox if needed. If you are setting a static IP address, you can leave the MAC address blank.]]

[[File:Infoblox add host wizard step5.png|1000px|thumb|left|Skip to step 5, fill in as many of the details as possible. These fields are not used by the system, but are useful for the administrators. When finished, Save and Close.]]

[[File:Infoblox restart DHCP server.png|1000px|thumb|left|Infoblox may require you to restart the DHCP server in the case that you checked the DHCP checkbox. Click on "restart" in the yellow bar (keep default settings in subsequent prompts).]]

Note that we cannot yet see the DHCP logs, talk to kray or dthomson for logs and debugging if needed.

Adding Hosts to a VLAN with Infoblox

2026-04-10T21:10:09Z

Dfujimoto: /* Basic Usage */

File:Infoblox restart DHCP server.png

2026-04-10T21:09:29Z

Dfujimoto:

Infoblox may require you to restart the DHCP server in the case that you checked the DHCP checkbox.

Adding Hosts to a VLAN with Infoblox

2026-04-10T19:12:35Z

Dfujimoto:

2026-03-12T18:00:11Z

Dfujimoto: /* install desktop environments */

= Prerequisites =

* before setting up new machine run memory test
* prepare flash drive with free version of memtest86: https://www.memtest86.com
* test boot from flash drive, test takes ~ few hours
* test will end with summary page, if passed continue with Ubuntu
* number that might be worth noting is memory latency

= Ubuntu version =

<pre>
lsb_release -a
uname -a
</pre>

= Ubuntu installer =

* updated for Ububtu LTS 20.04.01, 22.04.1, 24.04 (only minor differences)

* download the latest Ubuntu LTS desktop installer iso image
* dd the image to a USB key
* power down, disconnect all disks (all HDDs, all SSDs, all M.2)
* connect the SSD to be used as system disk
* if system will use mirrored SSDs (using ZFS mirror), leave second SSD disconnected, we will activate it later
* power up
* boot from USB key in legacy mode or UEFI mode (select this in the BIOS boot menu - F2 or F8 for ASUS, F11 for Supermicro)
* follow the instruction:
* "try ubuntu or install ubuntu" - choose "install"
* select language - accept default
* "updates and other software" - accept default settings ("normal install")
* "installation type" - select "advanced features" and "experimental: use ZFS"
* accept partition choice
* "where are you?" - select "Vancouver" (PST time zone)
* "who are you?" - leave all fields blank, except "username" set to "wheel", "password" set to the root password. hostname will be set later after configuring the network
* don't install third party sw
* installation runs in a few minutes, when finished, reboot
* login as user wheel
* answer annouying questions:
* "livepatch" - say "next"
* "help improve" - select "do not send", say "next"
* "privacy" - leave "location" as "off", say "next"
* "ready to go", say "done"
* right-click on the desktop, say "open in terminal", a shell will open
* say "sudo /bin/bash", enter the root password, you now have the root shell
* run nm-connection-editor to configure the network. use netmask 255.255.224.0, gateway 142.90.100.18, DNS 142.90.100.19, search path "triumf.ca"
* after network is up (can ping ladd00), continue with post-installation steps below

= Install instructions =

== prepare ==
<pre>
apt update
apt upgrade
</pre>

== install ssh ==
<pre>
apt install ssh
</pre>

== install git/scripts ==
<pre>
apt -y install git
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd scripts
git pull
</pre>

* if needed, update git/scripts repository from ladd00 to daq00:
* git remote -v ### if it says daq00, we are good
* git remote set-url origin https://daq00.triumf.ca/~olchansk/git/scripts.git
* git pull ### check that it works

== configure hostname ==
<pre>
vi /etc/hostname
</pre>

== disable swap ==

ubuntu installer creates a 2 GB swap partition, not useful
on 32-64 GB machine, disable it:
<pre>
vi /etc/fstab ### comment out the "swap" line
</pre>

* on 64 GB RAM machines swap is not useful
* on machines booted from network (NFS-ROOT), swap does not work
* on machines running from flash (RPi, etc), flash is too slow for useful swap
* swap configured by linux installers invariably has wrong size and is not useful

<pre>
systemctl disable dphys-swapfile
systemctl stop dphys-swapfile
dphys-swapfile uninstall
</pre>

== maybe reboot ==

this is a good point to reboot the machine to boot
the latest kernel and to set the correct hostname

== install etckeeper ==

keep contents of /etc in a git repository:

<pre>
apt -y install etckeeper
</pre>

== set timezone ==
<pre>
timedatectl list-timezones | grep -i vancouver
timedatectl set-timezone America/Vancouver
</pre>

== install time synchronization ==

check if chrony is installed correctly and is synched to TRIUMF time servers:

<pre>
chronyc sources
chronyc tracking
</pre>

if not, remove old chrony and

<pre>
apt -y remove chrony
apt -y purge chrony
</pre>

and install it from scratch:

<pre>
apt -y install chrony
cd ~/git/scripts
git pull
cd ~
cp ~/git/scripts/etc/triumf.sources /etc/chrony/sources.d/
systemctl disable systemd-timesyncd.service
systemctl stop systemd-timesyncd.service
systemctl disable ntp
systemctl stop ntp
systemctl enable chrony
systemctl restart chrony
chronyc sources
chronyc tracking
</pre>

== reenable systemd-timesyncd ==

ONLY IF CHRONY DOES NOT WORK

To configure systemd-timesyncd, set "NTP=" in /etc/systemd/timesyncd.conf

<pre>
apt remove chrony
cat /etc/systemd/timesyncd.conf
systemctl enable systemd-timesyncd.service
systemctl restart systemd-timesyncd.service
systemctl status systemd-timesyncd.service
timedatectl status
timedatectl timesync-status
</pre>

== enable outgoing email (debian 11) ==

this is different from ubuntu 20. it uses /etc/mailname and it hardwires the hostname into main.cf.

== enable outgoing email ==

we have an unusual email configuration. outgoing email should work to deliver error messages, notices, etc. incoming email is disabled, we do not receive email for local users.

this causes problems with TRIUMF smtp server. if our message cannot be delivered (wrong email address or receipient computer is turned off), TRIUMF smtp server will generate a delivery failure notification email and try to send it to the "from" address of the failed message. but the "from" address does not receive any email, so another delivery failure notification email is generated and an attempt to deliver it. which again fails, rinse and repeat.

as solution, kray created a special rule, email from scrap.triumf.ca does not generate delivery failure notices. failed messages sit in the queue for 5 days, then they are deleted. (K.O. - confirmed with kray 3jan2024).

to make this work we use the msmtp MTA package.

<pre>
cd ~
apt -y install mailutils msmtp msmtp-mta # say "no" to apparmor support
apt -y remove postfix
apt -y purge postfix # remove old config files
apt -y install bsd-mailx
cd ~/git/scripts/etc
git pull
/bin/cp -fv aliases /etc/aliases
/bin/cp -fv msmtprc /etc/msmtprc
/bin/rm -vf ~root/.forward
/bin/rm -vf /etc/mailname
Mail root
Subject: test
test
^D
CC: <CR>
</pre>

== enable outgoing email (postfix) ==

THIS IS OBSOLETE!!!

* TRIUMF: use smtp.triumf.ca
* CERN: use cernmx.cern.ch

<pre>
apt install postfix ### select "satellite system", enter full hostname "xxx.triumf.ca", enter "smtp.triumf.ca"
apt install mailutils
dpkg-reconfigure postfix ### (if postfix already installed)
</pre>
<pre>
echo olchansk@triumf.ca lindner@triumf.ca bsmith@triumf.ca dfujimoto@triumf.ca >> ~root/.forward
mailx root
test
^D
</pre>

== enable ping for all users (debian 11) ==

Without this tweak, Debian will report "operation not permitted" if a user tries to ping somewhere.

<pre>
echo 'net.ipv4.ping_group_range = 0 1000' > /etc/sysctl.d/99-ping.conf
</pre>

== disable apparmor ==

On NFS-Root network booted machines!

If "man man" returns "permission denied" and syslog reports apparmor "sendmsg DENIED" errors, disable apparmor. This is supposedly fixed in kernel 6.0 and later (to be confirmed), see https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/1784499

Disable apparmor, see https://ubuntu.com/server/docs/security-apparmor

This takes effect after a reboot.

<pre>
systemctl stop apparmor.service
systemctl disable apparmor.service
</pre>

If on boot apparmor appears to still be confining apps (for example, mysql), edit the file /etc/default/grub and change

<pre>
GRUB_CMDLINE_LINUX=""
</pre>

to

<pre>
GRUB_CMDLINE_LINUX="apparmor=0"
</pre>

Run sudo update-grub and reboot.

== install missing packages ==

(apt eats terminal input, even the "yes |" trick does not quite work,
repeat the following commands until they report that everything
is installed)

<pre>
yes | apt -y install ssh tcsh ethtool ncat rsync strace net-tools traceroute time minicom screen git lsof debsums tmux iptables telnet pax rpm mtools at gdisk tcpdump
yes | apt -y install sysstat smartmontools lm-sensors
yes | apt -y install lsb-release
apt -y install vim # in addition to default vim-tiny, requested by IRIS
apt -y install gedit # requested by TACTIC
apt -y install tcl
apt -y install mc # requested by sol
apt -y install pax rpm alien ### package converter tools
apt -y install flex bison
apt -y install neofetch
apt -y install snmp snmp-mibs-downloader
apt -y install git subversion g++ gfortran cmake doxygen
apt -y install curl libcurl4 libcurl4-openssl-dev
### conflits with mysql packages ### apt -y install mariadb-client libmariadb-dev ### mysql client for MIDAS
apt -y install mysql-client libmysqlclient-dev
apt -y install postgresql-common libpq-dev ### postgresql client for MIDAS
yes | apt -y install libz-dev libzstd-dev sqlite3 libsqlite3-dev unixodbc-dev
yes | apt -y install libssl-dev
yes | apt -y install emacs xemacs21 joe
yes | apt -y install gnuplot dos2unix
yes | apt -y install mutt bsd-mailx # email clients
yes | apt -y install liblz4-tool pbzip2 libbz2-dev
yes | apt -y install libc6-dev-i386 # otherwise no /usr/include/sys/types.h
yes | apt -y install libreadline-dev
yes | apt -y install ubuntu-mate-themes
yes | apt -y install libmotif-dev libxmu-dev
yes | apt -y install libusb-dev libusb-1.0-0-dev
yes | apt -y install i2c-tools libi2c-dev libi2c0
yes | apt -y install xfig gsfonts-x11 gsfonts-other # install fonts for xfig
yes | apt -y install libjson-perl
yes | apt -y install libgsl-dev # additional GNU Scientific Library
yes | apt -y install qt5-default # Qt development
yes | apt -y install python3-full python3-dev python3-dbg python3-pip ### for pyROOT
yes | apt -y install imagemagick imagemagick-common ckeditor # for elog
yes | apt -y install libjpeg-dev libjpeg-progs libjpeg-tools
yes | apt -y install linux-tools-common linux-tools-generic # cpupower frequency-info
yes | apt -y install rdesktop remmina remmina-plugin"*" # requested by POL
yes | apt -y install nlohmann-json3-dev # required to build MIDAS with ROOT 6.30 on Ubuntu-22
apt -y install dpkg-dev cmake g++ gcc binutils libx11-dev libxpm-dev libxft-dev libxext-dev python3 libssl-dev libafterimage0 # from https://root.cern/install/dependencies/
apt -y install gfortran libpcre3-dev xlibmesa-glu-dev libglew-dev libftgl-dev libmysqlclient-dev libfftw3-dev libcfitsio-dev graphviz-dev libldap2-dev python3-dev python3-numpy libxml2-dev libkrb5-dev libgsl0-dev qtwebengine5-dev nlohmann-json3-dev libtbb-dev libavahi-compat-libdnssd-dev # from https://root.cern/install/dependencies/
apt -y install libvdt-dev # for ROOT 6.32 on Ubuntu-24
apt -y install autoconf automake gperf # gnu package build tools
apt -y install u-boot-tools # for Xilinx petalinux
#apt -y install linux-headers-generic # to build linux kernel drivers
apt -y install htop
</pre>

Ubuntu LTS 20.04:
<pre>
yes | apt -y install linux-image-generic-hwe-20.04 linux-tools-virtual-hwe-20.04 # enable linux 5.11 series kernel
</pre>

Ubuntu LTS 22.04:
<pre>
apt -y install linux-generic-hwe-22.04 # enable linux 6.8.0 series kernel
</pre>

Ubuntu LTS 24.04:
<pre>
apt -y install linux-generic-hwe-24.04 # enable linux 6.14.0 series kernel
</pre>

== remove snap ==

remove snap at this point.

Do

<pre>
snap list
</pre>

to see a list of installed snaps. Use

<pre>
snap remove <item>
</pre>

to remove each. You will need to remove snapd last.

<pre>
systemctl stop snapd
systemctl disable snapd
apt purge snapd
rm -rf ~/snap
</pre>

== install non-snap firefox ==

delay this until the very end, download of firefox-esr deb package is very slow.

See https://askubuntu.com/questions/1399383/how-to-install-firefox-as-a-traditional-deb-package-without-snap-in-ubuntu-22

<pre>
add-apt-repository ppa:mozillateam/ppa
apt install firefox-esr
echo run firefox-esr
</pre>

Enable automatic updates:

* rerun [[#Enable_automatic_updates]]

== configure DNS ==

<pre>
cd ~/git/scripts
git pull
mkdir /etc/systemd/resolved.conf.d
cp etc/resolved-triumf.conf /etc/systemd/resolved.conf.d/
systemctl restart systemd-resolved
resolvectl
#systemd-analyze cat-config systemd/resolved.conf
</pre>

== install ganglia ==

<pre>
apt -y install ganglia-monitor
cd ~root/git/scripts/ganglia
git pull
make install
./ganglia-all.perl
</pre>

fix gmond start before network is ready:

<pre>
mkdir /etc/systemd/system/ganglia-monitor.service.d
echo -e "[Unit]\nAfter=network-online.target\n" > /etc/systemd/system/ganglia-monitor.service.d/local.conf
systemctl daemon-reload
systemctl cat ganglia-monitor.service
</pre>

== install ganglia server ==

'''On the main computer only! (daq00, dsdaqgw, etc)'''

* zfs create rpool/ganglia
* apt install gmetad php php-xml rrdtool
* mv /etc/ganglia/gmetad.conf /etc/ganglia/gmetad.conf-stock
* create /etc/ganglia/gmetad.conf with the following contents:
<pre>
data_source "my cluster" 15 localhost
RRAs "RRA:AVERAGE:0.5:1:244" "RRA:AVERAGE:0.5:24:244" "RRA:AVERAGE:0.5:168:244" "RRA:AVERAGE:0.5:672:244" "RRA:AVERAGE:0.5:5760:374"
setuid_username "ganglia"
rrd_rootdir "/ganglia/rrds"
case_sensitive_hostnames 0
</pre>
* mkdir /ganglia/rrds
* chown ganglia:ganglia /ganglia/rrds
* systemctl restart gmetad
* create /etc/ganglia/gmond-collect.conf
<pre>
globals {
daemonize = yes
setuid = yes
user = nobody
debug_level = 0
max_udp_msg_len = 1472
mute = no
#deaf = yes
deaf = no
allow_extra_data = yes
host_dmax = 0 /* 86400 */ /*secs. Expires (removes from web interface) hosts in 1 day */
host_tmax = 20 /*secs */
cleanup_threshold = 300 /*secs */
gexec = no
send_metadata_interval = 600 /*secs */
}

/*
* The cluster attributes specified will be used as part of the <CLUSTER>
* tag that will wrap all hosts collected by this instance.
*/
cluster {
name = "DAQ"
owner = "TRIUMF DAQ"
latlong = "unspecified"
url = "https://daq.triumf.ca"
}

/* The host section describes attributes of the host, like the location */
host {
location = "unspecified"
}

udp_send_channel {
host = daq00.triumf.ca
bind_hostname = yes
port = 8649
}

udp_recv_channel {
#mcast_join = 239.2.11.71
port = 8649
#bind = 239.2.11.71
retry_bind = true
# Size of the UDP buffer. If you are handling lots of metrics you really
# should bump it up to e.g. 10MB or even higher.
#buffer = 10485760
buffer = 200000
family = ipv4

acl {
default = "deny"
access {
# vlan1 daq00
ip = 142.90.111.168
mask = 19
action = "allow"
}
access {
# MUSR VLAN
ip = 142.90.154.73
mask = 8
action = "allow"
}
access {
# KVM network
ip = 192.168.1.1
mask = 8
action = "allow"
}
}
}

udp_recv_channel {
#mcast_join = 239.2.11.71
port = 8649
#bind = 239.2.11.71
retry_bind = true
# Size of the UDP buffer. If you are handling lots of metrics you really
# should bump it up to e.g. 10MB or even higher.
#buffer = 10485760
buffer = 200000
family = ipv6

acl {
default = "deny"
access {
# ALPHA network
ip = 2001:1458:202:fd::100:aa
mask = 8
action = "allow"
}
#access {
# # ALPHA-g network
# ip = 192.168.1.1
# mask = 8
# action = "allow"
#}
}
}

tcp_accept_channel {
port = 8649
bind = localhost
# If you want to gzip XML output
gzip_output = no
}
</pre>
* add to /etc/rc.local
<pre>
/usr/sbin/gmond -c /etc/ganglia/gmond-collect.conf &
systemctl restart gmetad &
</pre>
* cd /ganglia
* git clone https://daq00.triumf.ca/~olchansk/git/ganglia-web.git
* cd ganglia-web
* git checkout KO1
* ln -s /ganglia/ganglia-web /var/www/html/ganglia
* make dwoo directories
<pre>
mkdir /var/lib/ganglia-web
chown www-data:www-data /var/lib/ganglia-web
mkdir /var/lib/ganglia-web/dwoo
chown www-data:www-data /var/lib/ganglia-web/dwoo
mkdir /var/lib/ganglia-web/dwoo/compiled
chown www-data:www-data /var/lib/ganglia-web/dwoo/compiled
mkdir /var/lib/ganglia-web/dwoo/cache
chown www-data:www-data /var/lib/ganglia-web/dwoo/cache
</pre>
* open https://alphacpc05.cern.ch/ganglia/

== install gonodeinfo ==

* go to https://bitbucket.org/dd1/gonodeinfo follow instructions:
<pre>
apt -y install golang
mkdir ~/git
cd ~/git
#git clone https://bitbucket.org/dd1/gonodeinfo.git
git clone https://daq00.triumf.ca/~olchansk/git/gonodeinfo.git
cd gonodeinfo
git remote set-url origin https://daq00.triumf.ca/~olchansk/git/gonodeinfo.git
git pull
make
make install # install gonodeinfo agent
cd ~ # this is important
</pre>
* edit /etc/gonodeinfo.conf
* change "Description", "Location", "User" and "Administrator" as appropriate (or delete them)
* change "Servers" to read: Servers: daq00.triumf.ca:8601
* run "gonodeinfo -v"
* if error is "connection refused". go to the nodeinfo server to add this client to the access control list:
* on the gonodeinfo server: run /opt/gonodeinfo/gonodereceive.exe -a daq13
* try gonodeinfo again, there should be no error
* on the gonodeinfo server: run gonodereport, look at the web pages, the new machine should be listed now

== install emailonreboot ==

send an email if computer is rebooted

<pre>
ssh root
mkdir -p ~/git
cd ~/git
git clone https://daq00.triumf.ca/~olchansk/git/rpms.git
cd rpms/emailonreboot
git pull
make
make install
</pre>

== install monitor_nfs ==

monitor NFS mounts and complain about dead, stale and hung mounts

<pre>
ssh root
mkdir -p ~/git
cd ~/git
git clone https://daq00.triumf.ca/~olchansk/git/rpms.git
cd rpms/monitor_nfs
git pull
make
make install
</pre>

== install fonts for EPICS ==

* apt -y install xfonts-100dpi xfonts-75dpi
* restart Xorg (i.e. "killall Xorg", this will log you out from the console)
* xlsfonts | grep -i helvetica ### should show fonts with different sizes, not just size 0 (scalable)

== install libz.so.1 for CentOS compatibility ==

KO - confirm which versions on quartus need this.

<pre>
yes | apt-get -y install zlib1g
yes | apt-get -y install zlib1g:i386 libc6:i386 libgcc1:i386 gcc-6-base:i386
</pre>

== install ld-ldb-x86-64.so.3 for Quartus compatibility ==

Not clear from package this is supposed to come from. Copied from U-20.

Without this, Quartus lmgrd does not run.

<pre>
cd /lib64
ln -s ld-linux-x86-64.so.2 ld-lsb-x86-64.so.2
ln -s ld-linux-x86-64.so.2 ld-lsb-x86-64.so.3
</pre>

should look like this:
<pre>
root@daq13:/lib64# ls -l
total 3
lrwxrwxrwx 1 root root 44 Jan 28 09:07 ld-linux-x86-64.so.2 -> ../lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
lrwxrwxrwx 1 root root 20 Feb 18 16:45 ld-lsb-x86-64.so.2 -> ld-linux-x86-64.so.2
lrwxrwxrwx 1 root root 20 Feb 18 16:45 ld-lsb-x86-64.so.3 -> ld-linux-x86-64.so.2
root@daq13:/lib64#
</pre>

== install libpng12.so.0 for Quartus 13.0sp1 and 13.1.4.182 ==

<pre>
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0.50.0
/bin/cp -pv libpng12.so.0 libpng12.so.0.50.0 /lib/x86_64-linux-gnu/
</pre>

== install packages for Xilinx ==

ubuntu LTS 22.04 vivado 2020.1

<pre>
apt install autoconf libtool
apt install libtinfo5
apt install texinfo
apt install zlib1g:i386
</pre>

== install packages for building ROOT ==

<pre>
apt -y install libx11-dev libxpm-dev libxft-dev libxext-dev libpng-dev libjpeg-dev xlibmesa-glu-dev libxml2-dev libgsl-dev cmake
</pre>

== install wine ==

As far as I know, only needed for BNMR/BNQR

<pre>
apt install wine winetricks
</pre>

== install lightdm ==

unlike the default gdm login manager, lightdm shows the machine hostname and does not require an extra mouse click to swicth from screen saver to login mode.

<pre>
apt -y install lightdm
# select lightdm
</pre>

== install desktop environments ==

note: default display manager and default desktop are deficient, please do not skip this step.

note: if apt asks to choose the display manager, select "lightdm"

note: KO - I recommend the "MATE" desktop.

note: you will have to cut-and-paste this several times because "apt" eats commands, even with "-y" and even piped from "yes".

note: DF - on U24 this may re-install snap

<pre>
# install MATE desktop
apt -y install ubuntu-mate-core ubuntu-mate-desktop ubuntu-mate-themes
# install Cinnamon desktop
apt -y install cinnamon
# install KDE desktop
apt -y install kde-standard kubuntu-settings-desktop
# install Lxqt desktop
# apt -y install lxqt # conflict over kubuntu-desktop, kubuntu-settings-desktop and desktop-base
# install Xfce4 desktop
apt -y install xfce4
</pre>

== install ROOT ==

Please install ROOT per instructions at https://root.cern.ch.

NOTE1: The ROOT package available from Ubuntu repositories is severely out of date and cannot be used with MIDAS and ROOTANA. ### DO NOT DO THIS! apt-get install root-system

NOTE2: as of 2017-Jan-09, ROOT binary kits for Ubuntu do not work (use GCC 5 instead of GCC6), build from source instead.

== Install x2go ==

<pre>
apt-get update
apt-get install x2goserver x2goserver-xsession
</pre>

== enable root login from ladd00/daq00 ==
<pre>
ssh localhost
CTRL-C
/bin/cp ~root/git/scripts/etc/authorized_keys ~root/.ssh/
</pre>

== disable ssh access from outside of TRIUMF ==

to stop ssh login spam, disable ssh access from outside of TRIUMF. this can be done by requesting a firewall block through the helpdesk or by local firewall rule:

<pre>
echo iptables -I INPUT ! -s 142.90.0.0/255.255.0.0 -p tcp --dport 22 -j REJECT >> /etc/rc.local
/etc/rc.local
</pre>

== install smart-status ==
<pre>
ln -s ~/git/scripts/smart-status/smart-status.perl ~root/
</pre>

== enable boot menu and boot messages ==

This will enable the grub menu (with a 10 sec timeout) and
replace black screen with exciting linux boot messages.

* emacs -nw /etc/default/grub
<pre>
GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=menu
GRUB_TIMEOUT=10
GRUB_RECORDFAIL_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="vga=769 video=640x480"
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX=""
#GRUB_GFXMODE=640x480
</pre>
* update grub config:
<pre>
grub-mkconfig -o /boot/grub/grub.cfg
</pre>

= Enable automatic updates =

<pre>
apt install unattended-upgrades
cd ~/git/scripts
git pull
/bin/cp -v etc/99apt-conf-ko /etc/apt/apt.conf.d/
apt-config dump | grep Unattended
</pre>

Following is obsolete:

* emacs -nw /etc/apt/apt.conf.d/50unattended-upgrades
** uncomment in Allowed-Origins "-security" and "-updates"
** add in Allowed-Origins: "Google LLC:stable";
** uncomment/add: "Unattended-Upgrade::Mail "root";
* emacs -nw /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
</pre>
* test: unattended-upgrade --dry-run -v

NOTE: update-on-shutdown is disabled.

NOTE: there is no update-on-boot, but:

NOTE: if machine was off for a long time, the systemd update timer would have expired and it will fire soon after reboot, causing an automatic update run. this is unwanted, and there is no fix or workaround for it. K.O. June-2023.

= Fix bpool is full (obsolete) =

THIS IS CAUSED BY OBSOLETE PACKAGE zsys. PLEASE: apt remove zsys

= IPMI instructions =

IPMI is the board management hardware on Supermicro and other server motherboards. This includes hardware sensors - fan rotation speed, temperatures and power supply voltages.

<pre>
apt-get install ipmitool
systemctl enable ipmievd
systemctl restart ipmievd
</pre>

Run:
* ipmitool sel list ### event list
* ipmitool sel elist ### event list
* ipmitool sel clear ### clear event list (if it becomes full)
* ipmitool sensor ### report hardware sensors

= move /home/wheel (U-24) =

Ubuntu LTS 24 installed on ZFS has rpool/USERDATA/home_xxx mounted on /home,
has to be moved or autofs /home will not work.

<pre>
zfs list | grep USERDATA | grep home | cut -f1 -d" "
zfs set -u mountpoint=/home1 `zfs list | grep USERDATA | grep home | cut -f1 -d" "`
emacs -nw /etc/passwd # change mount for wheel account to /home1/wheel
</pre>

This will not take effect until rebooting. Please ensure that ssh from root@daq00 to this computer works before rebooting; if ssh from root@daq00 doesn't work and you mess this up you will be locked out of wheel account. Once you verify that this works, reboot and make sure that when you login as wheel that home directory is /home1/wheel.

= move /home/wheel =

note: this MUST be done if ZFS root and NIS/autofs with /home.

Default location of wheel's home directory will collide with autofs /home, it has to be moved,
for example to /wheel.

<pre>
# logout from the wheel user
# go to another computer
ssh root@daqubuntuxxx
zfs list | grep wheel ### identify zfs name wheel_xxxxxx
#zfs set mountpoint=/wheel rpool/USERDATA/wheel_hm8fzh
zfs set mountpoint=/wheel `zfs list | grep wheel | cut -f1 -d" "`
zfs list | grep wheel
emacs -nw /etc/passwd ### change wheel's home directory from /home/wheel to /wheel
su - wheel ### check that user wheel still works
</pre>

This will break wheel's ability to run snap programs, such as firefox, install chrome as listed below.

= enable NIS (ubuntu 22.04, 24.04, debian 11, 12) =

<pre>
apt -y install rpcbind nis
echo DAQ-NIS >> /etc/defaultdomain
echo ypserver daq00.triumf.ca >> /etc/yp.conf
systemctl enable ypbind.service
systemctl restart ypbind.service
systemctl status ypbind.service
ypwhich -m
</pre>

enable ypserv:

<pre>
sed -i s/NISSERVER=false/NISSERVER=slave/ /etc/default/nis
/usr/lib/yp/ypinit -s daq00
echo ypserver localhost >> /etc/yp.conf
sed -i "s/ypserver .*/ypserver localhost/" /etc/yp.conf
systemctl enable ypserv
systemctl restart ypserv
systemctl restart ypbind
</pre>

update /etc/nsswitch.conf and enable hourly update of NIS maps:

<pre>
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
cp -pv nsswitch.conf-U24 /etc/nsswitch.conf
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>

If this is a new machine, then on the master NIS node (daq00), add this new node to /etc/netgroup, and update NIS maps (cd /var/yp; make)

= enable NIS (ubuntu 20.04) =

* apt-get -y install portmap nis ### will ask for NIS domain (DAQ-NIS)
* dpkg-reconfigure nis ### reconfigure if already installed
* ypwhich -m
* edit /etc/default/nis
** set "NISSERVER=slave"
** Ubuntu LTS 20.04, check that "YPBINDARGS=" is blank, remove "-no-dbus" if it is there
* #edit /etc/yp.conf, comment-out everything, add "domain DAQ-NIS server localhost"
* edit /etc/yp.conf, comment-out everything, add "ypserver localhost"
* /usr/lib/yp/ypinit -s daq00
* systemctl enable nis
* systemctl restart nis
* ypwhich
* ypwhich -m
* ypcat -k passwd
* vi /etc/nsswitch.conf ### add the automount line, modify the passwd, group and shadow lines to read this:
<pre>
# begin get data from nis
passwd: files nis
group: files nis
shadow: files nis
automount: files nis
netgroup: files nis
# end get data from nis
</pre>
* enable hourly update of NIS maps
<pre>
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>
* ### NOT NEEDED sudo vi /etc/idmapd.conf ### add line: "Domain = triumf.ca"

= enable autofs =

<pre>
apt -y install autofs
systemctl enable autofs
systemctl restart autofs
ls -l /home/olchansk ### test autofs, check file owner is correct
</pre>

= enable NFS server =

<pre>
apt install nfs-kernel-server
#edit /etc/exports
systemctl enable nfs-server
systemctl restart nfs-server
</pre>

= NIS master =

notes for setting up the NIS master

== wheel user ==

"wheel" is the default administrative user. We do not want it's password exported to NIS (encrypted password hash is world visible) and we do not want it's home directory exported to NFS (~wheel/.ssh is world visible and potentially writable: anybody can change ~wheel/.ssh/authorized_keys).

* move wheel's home directory from /home/wheel to /wheel (see special section about this)
* change wheel's UID and GID from 1000 to a value below MINUID in /var/yp/Makefile

== coherent uids ==

we do not want system accounts defined in /etc/passwd of the NIS master
to be included in the NIS map "passwd". this causes trouble on NIS clients
where newly installed packages fail to create local system users because same
user already exists in NIS.

This is controlled by MINUID in /var/yp/Makefile.

Historical TRIUMF uids start from around 200, but several clusters do not have any historic TRIUMF uids below 500 and MINUID is set to:
* DAQ-NIS: MINUID=200
* ISAC-NIS: MINUID=500
* TITAN-NIS: MINUID=500
* MUSR-NIS: MINUID=500
* TIG-NIS: MINUID=500 (100 on SL6 mother8pi)

Ubuntu 20 has two programs to create users:
* adduser - creates new users with UID 1000 and up as specified in /etc/adduser.conf. No problems here.
* adduser --system - creates new system users with UID 100 and up as specified in /etc/adduser.conf. No problems here.
* useradd - creates new users with UID 1000 and up as specified in /etc/login.defs. No problems here.
* useradd --system - creates new system users with UID 999 and down (read "man useradd", section at the end about SYS_UID_MAX). This collides with NIS MINUID, these system users will be included in the NIS map and cause trouble.

This problem cannot be fixed, SYS_UID_MIN, SYS_UID_MAX and UID_MIN in /etc/login.defs do not seem
to have any effect on UIDs chosen by "useradd --system". (tested on Ubuntu LTS 20.04).

So far only these system accounts seem to be affected by this:
* systemd-coredump
* ganglia

To fix:
* run "sort -r -n -t: -k3 /etc/passwd" to identify the last unused system user uid (range 100..200)
* run "sort -r -n -t: -k3 /etc/group" to identify the last unused system user gid (range 100.200)
* systemd-coredump: manually change UID and GID (package systemd-coredump is usually not installed)
* ganglia: same thing, then change ownership on all ganglia files.

Also read systemd author's opinion on system vs user UIDs:
https://github.com/systemd/systemd/issues/4850#issuecomment-265698275

= Fix systemd-logind NIS breakage =

!!! THIS IS NOT NEEDED FOR UBUNTU LTS 20.04 !!!

there is a delay in ssh logins for normal users. "ssh -v" shows the delay is after "pledge...". this
fix removes the delay.

systemd developers think that we should not use NIS and made sure there are
problems if we do. To give them credit, they do offer a workaround. Read this:
https://github.com/poettering/systemd/commit/695fe4078f0df6564a1be1c4a6a9e8a640d23b67

<pre>
mkdir /etc/systemd/system/systemd-logind.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-logind.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-logind.service
</pre>

= Fix systemd-udevd NIS breakage =

see same problem as above with udev getting stuck. ubuntu lts 20.04.

<pre>
mkdir /etc/systemd/system/systemd-udevd.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-udevd.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-udevd.service
</pre>

= Configure USB device permissions =

Configure USB device permissions for user access to USB-serial devices, Altera USB Blaster, etc.

* create file /etc/udev/rules.d/99-usb-chmod.rules with this contents:

<pre>
emacs -nw /etc/udev/rules.d/99-usb-chmod.rules
ACTION=="add", SUBSYSTEM=="usbmisc", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /dev/%c"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /proc/%c"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVICE}"
ACTION=="add", ENV{PHYSDEVBUS}=="usb-serial", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVPATH}=="/class/tty/ttyS*", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyUSB*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyACM*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyS*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", DEVPATH=="*video*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
</pre>

* reload udev rules: udevadm control --reload-rules
* apply new permissions: udevadm trigger --action=add
* watch udev activity: udevadm monitor -p

= Configure lightdm display manager =

* enable it
<pre>
echo lightdm | dpkg-reconfigure -fteletype lightdm
systemctl disable gdm
systemctl disable sddm
systemctl enable lightdm
</pre>

* make the MATE desktop as default
<pre>
cd ~root/git/scripts/
git pull
/bin/cp -v etc/lightdm_default_mate.conf /etc/lightdm/lightdm.conf.d/
</pre>

* enable login by NIS users
<pre>
/bin/cp -v etc/lightdm_enable_nis_login.conf /etc/lightdm/lightdm.conf.d/
</pre>

* restart lightdm
<pre>
systemctl stop gdm
systemctl restart lightdm
</pre>

= Install libpng12.so.0 =

Quartus 16 needs libpng12:

<pre>
wget http://mirrors.kernel.org/ubuntu/pool/main/libp/libpng/libpng12-0_1.2.54-1ubuntu1_amd64.deb
dpkg --install libpng12-0_1.2.54-1ubuntu1_amd64.deb
</pre>

= Install google-chrome =

<pre>
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
dpkg -i google-chrome-stable_current_amd64.deb
</pre>

confirm autoupdate is enabled, observe dl.google.com is present in the list of repositories:
<pre>
apt update
...
Get:5 https://dl.google.com/linux/chrome/deb stable/main amd64 Packages [1,094 B]
...
</pre>

FOLLOWING IS OBSOLETE:

Instructions from here:
https://www.ubuntuupdates.org/ppa/google_chrome?dist=stable

<pre>
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-tmp.list'
apt update
apt install google-chrome-stable
/bin/rm -f /etc/apt/sources.list.d/google-tmp.list
</pre>

= Install amanda client =

ONLY ON MACHINES THAT HOST HOME DIRECTORIES

* apt install amanda-client
* edit /etc/amandahosts
<pre>
amanda.triumf.ca amanda amdump
</pre>
* check permissions on /etc/amandahosts:
<pre>
root@daq00:/var/log/amanda# ls -l /etc/amandahosts
-rw------- 1 backup backup 49 Jan 27 10:48 /etc/amandahosts
</pre>
* fix if needed: chown backup.backup /etc/amandahosts; chmod a= /etc/amandahosts; chmod u=wr /etc/amandahosts
* edit /etc/amanda-security.conf, add this line:
<pre>
runtar:gnutar_path=/usr/bin/tar
</pre>

On the amanda machine:

* in amanda disklist, use dump type "bsdtcp-comp-user-tar"
* su - amanda and run amcheck -c daily daq00
<pre>
-bash-4.1$ amcheck -c daily daq00

Amanda Backup Client Hosts Check
--------------------------------
Client check: 1 host checked in 0.092 seconds. 0 problems found.

(brought to you by Amanda 3.3.7p1.git.685ff76d)
</pre>

= Enable rc.local =

For reasons unknown, Ubuntu LTS 20.04 does not enable /etc/rc.local. Do this:

<pre>
cd ~/git/scripts
git pull
cp -n -v etc/rc.local /etc/
chmod a+rx /etc/rc.local
cp etc/rc-local.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable rc-local
systemctl start rc-local
systemctl status rc-local
</pre>

= Remove unwanted packages =

<pre>
apt purge bash-completion # broken, adds unwanted "\" if "ls -l $ROOTSYS/<tab>"
apt purge zsys # broken, do not use
apt purge sddm # login manager
apt purge avahi-daemon avahi-autoipd # not sure what it does, observed using 100% CPU
apt purge modemmanager # probes all serial ports to see if it's a modem
</pre>

= Disable unwanted services =

<pre>
systemctl disable mpd
systemctl disable snapd
systemctl disable ModemManager
systemctl --global mask tracker-extract-3.service
systemctl --global mask tracker-miner-fs-3.service
systemctl daemon-reload
</pre>

= Disable sleep and suspend =

note: we see some computers randomly shutdown or go to sleep, log files indicates the "sleep" or "suspend" button was pushed by user, but no such buttons actually exist. this is the fix for this:

<pre>
systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target systemd-suspend.service systemd-hybrid-sleep.service
</pre>

= Enable crontab @reboot for MIDAS =

startup scripts have a bug - cron @reboot entries for normal users can run before autofs is ready, so if the home directory is on autofs/NFS, it cannot be accessed and the cron job fails. If MIDAS is supposed to be started by cron @reboot, it will not start (there *will* be an error message in /var/log/cron).

<pre>
mkdir /etc/systemd/system/cron.service.d
echo -e "[Unit]\nAfter=ypbind.service autofs.service\n" > /etc/systemd/system/cron.service.d/local.conf
systemctl daemon-reload
systemctl cat cron.service
</pre>

Explore the systemd dependency tree using "systemctl list-dependencies" maybe with "--all".

Visualize the exact boot sequence from previous boot: "systemd-analyze plot > xxx.svg", look at the svg file using a web browser.

Crontab entry to start midas: (install in the midas user crontab, not root crontab)

<pre>
su - midasuser
crontab -l
#@reboot /bin/bash -l -c "/home/trinat/bin/start-daq-applications"
#@reboot /bin/tcsh -c "/home/trinat/bin/start-daq-applications"
</pre>

= Install apache httpd proxy for midas and elog =

This will configure the HTTPS/SSL certificate using "certbot" and "letsencrypt" and configure an HTTPS web server using apache2.

First, configure apache2:

* execute these commands:
<pre>
apt -y install apache2
cd /etc/apache2
</pre>
* create new file conf-available/ssl-daq14.conf # use actual hostname instead of daq14
<pre>
SSLSessionCache shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout 300
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
SSLCryptoDevice builtin
</pre>
* create new file sites-available/daq14-ssl.conf # use actual hostname instead of daq14
<pre>
<IfModule mod_ssl.c>
<VirtualHost *:443>
ServerName daq14.triumf.ca
DocumentRoot /var/www/html
ErrorLog /var/log/apache2/daq14.log
SSLEngine on
# note SSLProtocol, SSLCipherSuite and some other settings are overwritten by /etc/letsencrypt/options-ssl-apache.conf
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SEED:!IDEA:!RC4
## use port specified in elogd.cfg
#ProxyPass /elog/ http://localhost:8082/ retry=1
## use mhttpd port
#ProxyPass / http://localhost:8080/ retry=1
Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"
<Location />
SSLRequireSSL
AuthType Basic
AuthName "DAQ password protected site"
Require valid-user
# create password file: touch /etc/apache2/htpasswd
# to add new user or change password: htpasswd /etc/apache2/htpasswd username
AuthUserFile /etc/apache2/htpasswd
</Location>
#SSLCertificateFile /root/server.cert
#SSLCertificateKeyFile /root/server.key
</VirtualHost>
</IfModule>
</pre>
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* enable ssl module
* enable new configurations
<pre>
a2enmod ssl
a2enmod headers
a2enmod proxy
a2enmod proxy_http
a2enconf ssl-daq14
a2ensite daq14-ssl
</pre>
* disable default ssl sites
<pre>
a2dissite 000-default-le-ssl
a2dissite 000-default
ls -l /etc/apache2/sites-enabled/ ### should show only daq14-ssl.conf
</pre>
* check that there are no syntax problems
<pre>
apache2ctl configtest
</pre>
* enable and start apache2:
<pre>
systemctl enable apache2
systemctl restart apache2
systemctl status apache2
</pre>
* apache2 may fail to start, look in /var/log/apache2/error.log and /var/log/apache2/daq14.log
* if it says "Failed to configure ... certificate", proceed to the step for setting certbot.
* try to access https://daq14.triumf.ca
** you should see a complaint about self-signed certificate
** you should see a request for password (do not login yet)
** if you get "connection refused", HTTPS port 443 may need to be enabled in the local firewall, look at documentation for ufw.
Second, configure certbot:

(Note: as of 2018-01-18 certbot requires use of http port 80 to get the initial https certificate,
renewal can continue to use the https port 443)

(Note: as of 2019-01-?? certbot requires use of port 80 for renewals)

(Note: unsurprisingly, this requires outside access to connect with letsencrypt, so won't work if PC is only accessible from on-site network)

* check that port 80 is not used by anything:
* netstat -an | grep LISTEN | grep ^tcp | grep 80
* lsof -P | grep -i tcp | grep LISTEN | grep 80
* if lsof reports that apache2 is listening on port 80, follow the apache2 instructions above (remove "listen 80" from apache2.conf

* install certbot (if necessary open tcp port 80 in the firewall, see documentation for ufw):
<pre>
apt install certbot python3-certbot-apache
certbot certonly --standalone --installer apache
</pre>
* then answer questions:
* "activate HTTPS for daq14.triumf.ca" - say ok
* "enter email address" - enter your own email address
* "please read terms..." - read the terms and say "agree"
* it will take a few moments...
* "congratulations..." - say ok.
<pre>
certbot install --apache --cert-name daq14.triumf.ca
</pre>
* then answer questions:
* "choose redirect..." - say "1" (no redirect)
* look inside /etc/apache2/sites-enabled/daq14-ssl.conf to see that SSLCertificateFile & co point to certbot certificates in
/etc/letsencrypt/live/daq14.triumf.ca/
* to check current renewal and to update the certbot config file in /etc/letsencrypt/renewal, run this:
<pre>
certbot renew --standalone --installer apache --force-renewal
</pre>

NOTE: this certificate will expire in 3 months, automatic renewal should work with current version of certbot

Third, activate password protection:

* as shown in the config file above, create password file and initial user: (replace "midas" with specific username)
<pre>
touch /etc/apache2/htpasswd
htpasswd /etc/apache2/htpasswd midas
</pre>

* restart apache2
<pre>
systemctl restart apache2
systemctl status apache2
</pre>

From here:
* enable proxy for MIDAS mhttpd - uncomment redirect in the config file above
* enable proxy for ELOG - ditto
<pre>
a2enmod proxy
a2enmod proxy_http
apache2ctl configtest
systemctl restart apache2
</pre>

* try accessing MIDAS https://daq14.triumf.ca/ (make sure mhttpd is running)
* if it's not working, check odb setting FIXME!
* try accessing ELog https://daq14.triumf.ca/elog/ (make sure elogd is running)
* if it's not working, check elogd.cfg file and make sure
<pre>
SSL = 0
</pre>

NOTE: if certbot fails with errors about 'module' object has no attribute 'pyopenssl',
try this: pip install requests==2.6.0

== generate self-signed certificate ==

<pre>
# cd $HOME
# openssl req -nodes -new -x509 -keyout server.key -out server.cert -days 1001
...+....+..+..........+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+..+...+.........+......+.+...+...+.....+...............+.........+...+.+......+...+...........+....+...+..+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+......+.+...+..+.......+..+...+.......+......+...+..+...+......+....+...............+..+...+....+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
......+......+.+..+......+.+......+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.....+......+.+.........+......+.....+.+..+...+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+....+......+.....+...+...+.......+..+.+........+.+...+......+..+..........+..+.+...........+...+.......+......+.....+.......+...+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:CH
State or Province Name (full name) [Some-State]:Geneve
Locality Name (eg, city) []:CERN
Organization Name (eg, company) [Internet Widgits Pty Ltd]:CERN
Organizational Unit Name (eg, section) []:ALPHA experiment
Common Name (e.g. server FQDN or YOUR name) []:alphacpc05.cern.ch
Email Address []:
root@alphacpc05:~#
root@alphacpc05:~#
root@alphacpc05:~# ls -l
-rw-r--r-- 1 root root 1375 juil. 10 21:43 server.cert
-rw------- 1 root root 1708 juil. 10 21:42 server.key
root@alphacpc05:~# systemctl restart apache2
</pre>

= Enable elog PDF preview =

NOTE: looks like U-24 already has this correctly.

see https://stackoverflow.com/questions/52998331/imagemagick-security-policy-pdf-blocking-conversion

* xemacs -nw /etc/ImageMagick-6/policy.xml
* remove this section at the end:
<pre>

<policy domain="coder" rights="none" pattern="PS" />
<policy domain="coder" rights="none" pattern="PS2" />
<policy domain="coder" rights="none" pattern="PS3" />
<policy domain="coder" rights="none" pattern="EPS" />
<policy domain="coder" rights="none" pattern="PDF" />
<policy domain="coder" rights="none" pattern="XPS" />
</pre>

= Install Jupyter notebook =

<pre>
From https://jupyter.org/install
apt install python3-pip
pip install jupyterlab
pip install notebook
~/.local/bin/jupyter notebook
watch the http://localhost:8888 URL that it printed
say "no" to offer to start firefox (it will not work!)
URL is: http://localhost:8888/tree?token=xxx
from the machine where you are running the web browser (i.e. google-chrome), run (replace trinat@trinatdaq with the username and machine name where you started jupyter)
open a new shell and run: ssh -v trinat@trinatdaq -L 8888:localhost:8888
in the web browser, open http://localhost:8888
this gives us the login page
in the password or token entry field, put the token from the "tree?token=xxx" above (printed by jupyter on startup)
push button "login"
jupyter page should open with the list of files in the trinat home directory
congratulate Brian with full success
</pre>

= Install ZFS quota report =

If there are any ZFS volumes, install script to report disk and quota usage

<pre>
cd ~/git/scripts/quotareport
git pull
mkdir /var/www/html/zfsquotareport
cp -pv ~/git/scripts/quotareport/sorttable.js /var/www/html/zfsquotareport/
ln -s $PWD/zfsquotareport.perl /etc/cron.daily/
touch /etc/crontab
</pre>

If httpd is configured to redirect "/" to MIDAS mhttpd:
* add following to /etc/apache2/sites-enabled/xxx-ssl.conf in front of "ProxyPass / ..."
* run "systemctl reload apache2"
<pre>
## do not proxy zfs quota report directory
ProxyPass /zfsquotareport/ !
</pre>

= Install PHP =

* apt install php libapache2-mod-php
* systemctl restart apache2
* create /var/www/html/info.php
<pre>
<?php

phpinfo();
</pre>
* open https://daq00.triumf.ca/info.php

= Configure TRIUMF printers =

<pre>
systemctl stop cups
systemctl stop cups-browsed.service
systemctl disable cups
systemctl disable cups-browsed.service
systemctl stop snap.cups.cupsd.service
systemctl stop snap.cups.cups-browsed.service
systemctl disable snap.cups.cupsd.service
systemctl disable snap.cups.cups-browsed.service
echo "ServerName printers.triumf.ca" > /etc/cups/client.conf
lpstat -a
</pre>

= Enable core dumps =

By default, Ubuntu LTS 20.04 installs the apport package
which disabled core dumps from user applications. (google it up!).
It is not meant to do this and documentation claims that
it is not installed and not enabled by default. Oh, well...

<pre>
apt purge apport
apt autoremove ### will remove apport-symptoms and a few other packages
</pre>

After this, core dumps are written to file "core" in the current directory.
See /proc/sys/kernel/core_pattern and /proc/sys/kernel/core_uses_pid.

Enable core dump file names to include process id, add following to /etc/rc.local

<pre>
echo "echo 1 > /proc/sys/kernel/core_uses_pid" >> /etc/rc.local
</pre>

= Enable debugger =

By default, Ubuntu LTS 20.04 does not permit debugger to attach and debug
already running programs. To enable it, add following to /etc/rc.local

<pre>
echo "echo 0 > /proc/sys/kernel/yama/ptrace_scope" >> /etc/rc.local
</pre>

= Disable Ubuntu Pro nag =

best I can tell, impossible at this time.

== do not do this ==

!!! does nothing !!!

<pre>
pro config set apt_news=false
</pre>

== do not do this ==

!!! breaks automatic updates because 20apt-esm-hook.conf is missing !!!

If "apt upgrade" requests Ubuntu Pro or esm-apps, disable the nag:
<pre>
/bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf
</pre>

== do not do this ==

!!! likely same as above, breaks automatic updates !!!

* comment out /etc/apt/apt.conf.d/20apt-esm-hook.conf

== do not do this ==

!!! removes too many packages !!!

<pre>
apt remove ubuntu-pro-client
</pre>

= Update packages =

* apt update # update package list
* apt upgrade # install updated packages and update "kept back" packages
* apt autoremove # remove packages that apt thinks should be removed

= Cleanup residual configs =

* apt list '~c'
* apt purge '~c'

= Install firefox-esr =

install firefox-esr here

= Finish installation =

Congratulations. There is nothing more to do!

* reboot
<pre>
shutdown -r now
</pre>

= Update to new version of Ubuntu =

* run "do-release-upgrade -c"
* if it does not report new release Ubuntu 24, check /etc/update-manager/release-upgrades has "Prompt=lts"

== Update Ubuntu LTS 20.04 to LTS 22.04 ==

<pre>
apt remove zsys
</pre>

=== daqubuntu ===

<pre>
# reboot to clear out all updates
# vi /etc/update-manager/release-upgrades # set "Prompt=normal"
# do-release-upgrade -c
Checking for a new Ubuntu release
New release '22.04 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
# do-release-upgrade
...
say yes...
...
login.defs, say "Y" (erase local changes, use packaged version)
/etc/systemd/resolved.conf, say "Y" (same as above)
firefox snap, say yes
unable to reach snap store, say "skip"
/etc/gmond.conf, say "Y"
/var/yp/Makefile, say "install the package maintainer's version"
/etc/ypserv.conf, same thing
/etc/ypserv.securenets, same thing
/etc/default/nis, same thing
/etc/speech-dispatcher/modules/mary-generic.conf, same thing
/etc/apt/apt.conf.d/50unattended-upgrades, same thing
...
278 packages are going to be removed, say yes
...
restart required, say yes
...
no ping... yes ping...
...
ssh daqubuntu, ok
apt update, fail, DNS does not work, "host security.ubuntu.com" does not resolve.
fix resolver per https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Disable_NetworkManager
apt update, apt upgrade now works, 0 packages to update
NIS does not work.
</pre>

=== midm9a ===

<pre>
login.defs
firefox snap
gmond.conf
ypserv
/etc/default/nis
unattended-upgrades
amanda-security.conf
remove obsolete (no)
reboot
configure dns
reenable nis
</pre>

=== daq17 ===

<pre>
firefox snap
imagemagick policy.xml
gmond.conf
chrony.conf
/var/yp/Makefile
ypserv.conf
ypserv.securenets
/etc/default/nis
50unattended-upgrades
</pre>

=== daq00 ===

per https://serverpilot.io/docs/how-to-upgrade-ubuntu-20.04-to-22.04/

<pre>
do-release-upgrade -f DistUpgradeViewNonInteractive
</pre>

if it exists "too soon" without doing anything, run it without "-f xxx", most likely it does not like something about this machine. in case of daq00 it did not like how the EFI partitions were mounted. after fixing it, non-interactive upgrade was successful.

=== isdaq08 ===

* prepare
<pre>
cd ~/git/scripts
git pull
cd ~
apt -y install debsums
</pre>
* check for modified config files that make upgrade unhappy, deal with all files reported by debsums.
<pre>
root@isdaq08:~# debsums -ce
/etc/ganglia/gmond.conf
/etc/yp.conf
/etc/apt/apt.conf.d/10periodic
root@isdaq08:~#
</pre>
* restore original /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
</pre>
* apt remove ganglia-monitor
* apt remove nis
* "debsums -ce" is now empty

Run the upgrade:

* do-release-upgrade -f DistUpgradeViewNonInteractive

Post upgrade:

* configure DNS
* apt -y install linux-generic-hwe-22.04
* /bin/cp -v ~/git/scripts/etc/99apt-conf-ko /etc/apt/apt.conf.d/ # restore nightly updates
* /bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf # remove the ubuntu-pro nag
* install missing packages
* restore ganglia
* restore nis
* check zpool status, may need zpool upgrade
* reboot

== upgrade U-22 to U-24 ==

generic instructions. below are notes from upgrades of specific machines.

NOTE: at CERN saw installation getting stuck on restart of autofs (automount, rsyslogd 100% CPU, huge /var/log/syslog), stop autofs before upgrade?
<pre>
systemctl stop autofs
</pre>

<pre>
cleanup zsys
</pre>

<pre>
maybe: if snap is already removed, remove firefox to prevent snap from reinstalling.
install non-snap firefox-esr
</pre>

<pre>
remove thunderbird to prevent installation of snap thunderbird in U-24
</pre>

<pre>
mount EFI partition as /boot/efi, otherwise upgrade bombs
usually mount /dev/sda1 /boot/efi
</pre>

<pre>
debsums -ce
apt -y remove desktop-base # causes installer crash
apt -y remove thunderbird # avoid forced conversion to snap
apt update
apt -y upgrade
apt -y autoremove
do-release-upgrade -c # confirm upgrade will be to U-24
do-release-upgrade
# say "y" to all questions
# after installation starts, accept default answers to all questions
# should run for about 1 hour or so
# after upgrade finishes
apt update
apt -y upgrade
apt -y autoremove
shutdown -r now
</pre>

post upgrade:

* check zpool status, may need zpool upgrade
* cd /etc/apt/sources.list.d, reenable 3rd party repos (mozilla, google, etc)

if installation bombs:

<pre>
apt update
apt upgrade # will tell us to run "apt --fix-broken install", do it
apt --fix-broken install
apt update
apt upgrade # should resume the upgrade, will run for a long time
apt update
apt upgrade # should do nothing
apt autoremove
shutdown -r now
</pre>

=== daqubuntu, U-24 ===

* prepare
<pre>
cd ~/git/scripts
git pull
cd ~
apt -y install debsums
</pre>
* check for modified config files that make upgrade unhappy, deal with all files reported by debsums.
<pre>
root@daqubuntu:~# debsums -ce
/etc/ganglia/gmond.conf
debsums: missing file /etc/init.d/nis (from nis package)
/etc/default/nis
/etc/ypserv.conf
/etc/ypserv.securenets
/var/yp/Makefile
/etc/update-manager/release-upgrades
/etc/apt/apt.conf.d/10periodic
/etc/yp.conf
root@daqubuntu:~#
* restore original /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
</pre>
* apt remove ganglia-monitor
* apt remove nis
* apt autoremove
* restore original release-upgrades: "Prompt: lts"
* "debsums -ce" is now empty

Check for upgrade:

<pre>
root@daqubuntu:~# do-release-upgrade -c
Checking for a new Ubuntu release
There is no development version of an LTS available.
To upgrade to the latest non-LTS development release
set Prompt=normal in /etc/update-manager/release-upgrades.
root@daqubuntu:~#
</pre>

Run the upgrade:

* do-release-upgrade -f DistUpgradeViewNonInteractive

Post upgrade:

* configure DNS
* apt -y install linux-generic-hwe-22.04
* /bin/cp -v ~/git/scripts/etc/99apt-conf-ko /etc/apt/apt.conf.d/ # restore nightly updates
* /bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf # remove the ubuntu-pro nag
* install missing packages
* restore ganglia
* restore nis
* check zpool status, may need zpool upgrade
* cd /etc/apt/sources.list.d, reenable 3rd party repos (mozilla, google, etc)
* reboot

=== daq14, U-20-22-24 ===

* apt update, apt upgrade
* apt -y install linux-image-generic-hwe-20.04 linux-tools-virtual-hwe-20.04 ### install kernel 5.15
* shutdown -r now
* stuck waiting for daq14 to shutdown...
* reboot into kernel 5.15
* ???
<pre>
cd ~/git/scripts
git pull
cd ~
apt -y install debsums
</pre>
* debsums -ce
<pre>
/etc/apache2/ports.conf
/etc/dnsmasq.conf
/etc/ganglia/gmond.conf
/etc/yp.conf
/etc/sudoers
</pre>
* apache2 restore original ports.conf, uncomment "Listen 80"
* cp -pv /etc/dnsmasq.conf.dpkg-dist /etc/dnsmasq.conf
* apt remove ganglia-monitor
* edit /etc/yp.conf, remove everything after "# ypserver ypserver.network.com"
* "debsums -ce" is now empty
* do-release-upgrade -f DistUpgradeViewNonInteractive
* runs for a long time
* stuck on "/etc/default/nis", type "Y", press enter, nothing for a bit, then resumes running
* finished
* configure DNS
* reboot
* have kernel 6.8
* apt update; apt upgrade
* apt upgrade guile-2.2-libs ### would not auto-update, "kept back", has to be done by hand
* apt autoremove
* debsums -ce
<pre>
debsums: missing file /etc/init.d/nis (from nis package)
/etc/default/nis
</pre>
* diff /etc/default/nis.dpkg-dist /etc/default/nis
* cp -pv /etc/default/nis.dpkg-dist /etc/default/nis
* debsums -ce
<pre>
debsums: missing file /etc/init.d/nis (from nis package)
</pre>
* we ignore this and run the update
* do-release-upgrade -c
<pre>
Checking for a new Ubuntu release
New release '24.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
</pre>
* do-release-upgrade -f DistUpgradeViewNonInteractive
* bombs out without any error messages
* in /var/log/dist-upgrade/main.log reports "Failed to find a replacement for xapp" and other packages
* apt remove xapp usrmerge ureadahead thunderbird-gnome-support
* no go, complains about even more packages.
* apt list | grep installed | grep -v jammy ### show packages installed from non-ubuntu sources
* remove all packages marked "install,local" ### ubuntu updater does not know where they came from and so cannot update them.
* apt remove desktop-base ### not happy about this package in /var/log/dist-upgrade/apt.log
* apt autoremove
* do-release-upgrade -f DistUpgradeViewNonInteractive
* running for a long time...

=== alpha04 U-20-24 ===

* apt update, apt upgrade, apt autoremove
* reboot into latest kernel (already done)
* debsums -ce
<pre>
root@alpha04:~# debsums -ce
/etc/dnsmasq.conf
/etc/ganglia/gmond.conf
/etc/default/nis
/etc/yp.conf
root@alpha04:~#
</pre>
* move /etc/dnsmasq.conf to /etc/dnsmasq.d/alpha04.conf
* apt remove dnsmasq
* apt remove ganglia-monitor
* apt remove nis
* apt autoremove
* debsums -ce ### is now empty
* do-release-upgrade -f DistUpgradeViewNonInteractive
* it runs for a long time...
* complained about /etc/fwupd config files, not sure why...
* finished
* apt update, apt upgrade, apt autoremove
* restore dnsmasq: apt install dnsmasq, systemctl status dnsmasq
* restore ganglia, per instructions
* restore NIS: apt -y install rpcbind nis, ypwhich, ypwhich -m
* zpool upgrade rpool ### also upgrade any other zfs pools, see zpool status
* remove unwanted packages, per instructions
* run gonodeinfo
* reboot
* done

== vera00 U20-22-24 ==

* everything same as daq14 for U20-22
* kernel is still 5.15
* U22-24 is going...
* stuck for a few minutes on /etc/fwupd config files
* have kernel 6.8.0-49
* same steps as daq14
* reboot
* same steps as daq14
* done

== phaarmonster U22-24 ==

* debsums -ce
<pre>
/etc/amandahosts
/etc/apache2/ports.conf
/etc/ganglia/gmond.conf
</pre>
* do-release-upgrade -c ### reports U24.04.1
* do-release-upgrade -f DistUpgradeViewNonInteractive ### bombs
* apt remove desktop-base; apt autoremove
* do-release-upgrade ### (interactive) runs ok
* bombed !!!
* apt upgrade spews errors and tells us to run "apt --fix-broken install"
* apt --fix-broken install ### runs
* bombs with thunderbird snap errors
* again... no go
* thunderbird snap complains about mounting /home, but /home is a symlink
* rm /home, mkdir /home
* again, runs ok
* asks about /etc/fwupd/fwupd.conf - say "Y" to install updated package version
* apt install completes
* apt update; apt upgrade ### running for a long time...
* finished
* install missing packages, etc
* reboot
* long wait... came back
* DNS does now work, systemd-resolved missing, apt install systemd-resolved, "configure DNS"
* done.

== isdaq10 U22-24 ==

* do-release-upgrade -f DistUpgradeViewNonInteractive
* bombed on desktop-base, "apt remove desktop-base; apt autoremove"
* bombed with errors
* running "apt --fix-broken install"
* complained about thunderbird snap
* complained about /etc/fwupd/fwupd.conf (say Y)
* finished ok
* apt update
* apt upgrade ### reports "1382 upgraded, 140 newly installed, 0 to remove and 29 not upgraded"

== iris01 ==

* debsums -ce
<pre>
/etc/ganglia/gmond.conf
/etc/default/nis
/etc/yp.conf
</pre>
* apt remove desktop-base
* apt remove thunderbird
* apt autoremove
* do-release-upgrade -f DistUpgradeViewNonInteractive
* bombed with dpkg errors:
<pre>
hplip wants wrong python
ganglia-monitor wants wrong libapr1
sssd-ad wants bunch of wrong libraries
xemacs21-mule
libnfsidmap1 wants libldap2
adn so forth...
</pre>
* apt --fix-broken install -y
* bombs on ganglia - ganglia group absent from /etc/groups
* fix group ganglia by hand, remove ganglia group from NIS on isdaq00
* apt --fix-broken install
* apt upgrade

== iris00 ==

* stuck on "autofs: restarting", login as root, kill iris midas, kill automount, systemctl restart autofs, noble got unstuck, ctrl-c systemctl restart autofs
* noble running...
* bombed with dpkg errors
* check ganglia user, group - both ok
* apt --fix-broken install
* apt upgrade

== tigstore01 ==

* no bomb-out

== midm9b ==

* apt remove desktop-base thunderbird
* bombed
* apt --fix-broken-install
* apt upgrade

= Upgrade to new version of Debian =

https://www.debian.org/releases/bookworm/amd64/release-notes/ch-upgrading.en.html

== 32-bit VME processor Debian 11 to 12 ==

* cd git/scripts; git pull; cd ~
* apt update
* apt upgrade
* edit /etc/apt/sources.list
<pre>
deb http://deb.debian.org/debian/ bookworm main
#deb http://deb.debian.org/debian/ bullseye main
#deb http://deb.debian.org/debian/ buster main
#deb-src http://deb.debian.org/debian/ bullseye main
</pre>
* apt update
* apt upgrade --without-new-pkgs
* apt full-upgrade
* apt list '~c'; apt purge '~c' # purge left-over config files [residual-config]
* reboot

= Ubuntu package manager =

* apt-get install xxx # install package xxx
* apt-get update
* apt-get upgrade
* apt-get dist-upgrade
* apt-get autoremove # remove automatically installed packages required by a removed package
* apt-get remove xxx # remove package xxx
* apt-cache search . # list all available packages
* apt-cache show "." | grep ^Package # list al available packages
* apt-cache madison root-system # show all available versions of package root-system
* apt list # list all installed packages
* dpkg --listfiles libpng16-16 # list all files from this package
* apt list --installed # list all installed packages
* dpkg -S /bin/bash # what package provides this file?
* dpkg -L bash # what files provided by this package?
* debsums -ce # show modified config files
* apt-config dump # show apt configuration
* apt purge '~c' # purge all [residual-config] packages
* ls -l /var/lib/dpkg/info/ # show post-install scripts

= Ubuntu zsys =

NOTE: DO NOT USE ZSYS, see https://github.com/ubuntu/zsys/issues/218 and https://github.com/ubuntu/zsys/issues/230

* scripted removal of old snapshots (replace "echo zfs destroy" with "zfs destroy")
<pre>
zfs list -t all | cut -f1 -d " " | grep autozsys | xargs -n1 echo zfs destroy
</pre>

* manual removal of old snapshots
<pre>
zsysctl show
zsysctl state remove xy69ye -s
zsysctl state remove xy69ye
zsysctl state remove xy69ye -u wheel
</pre>
* apt remove zsys

NOTE: old zsys snapshots must be cleaned manually, "zsysctl state remove xxx --system" is broken and does not remove user data snapshots

* manages system snapshots
* documentation: https://github.com/ubuntu/zsys
* documentation: (go to next article via link "newer" at the bottom) https://didrocks.fr/2020/05/21/zfs-focus-on-ubuntu-20.04-lts-whats-new/
* ubuntu 20.04 bug, too many snapshots cause /boot to become full and updates fail. https://github.com/ubuntu/zsys/issues/155
* solution: use custom /etc/zsys.conf, limit number of snapshots to 10, see trinatdaq:/etc/zsys.conf
* zsys commands:
<pre>
update-grub # list of all snapshots, errors if some snapshots are broken
zsysctl state remove lnc0k7 --system # remove snapshot
xemacs -nw /etc/zsys.conf; zsysctl service reload; zsysctl service gc # cause gc to run with new settings in zsys.conf
zfs list -r -t snapshot -o name,used,referenced,creation bpool/BOOT # list snapshots
zsysctl show # show snapshots
</pre>

= Ubuntu cloning =

to clone a ubuntu image:

<pre>
cd /nfsroot/lxcpet
emacs -nw etc/hostname ### change hostname
emacs -nw etc/mailname ### change hostname (debian 11)
emacs -nw etc/defaultdomain ### change the NIS domainname
emacs -nw etc/yp.conf ### change the NIS server
cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
emacs -nw root/.ssh/authorized_keys ### update root ssh keys
</pre>

= Ubuntu boot loader =

== maintenance commands ==

* update-initramfs -v -u
* grub-install /dev/sda

= Convert from single to dual mirrored ZFS SSD =

Assuming Ubuntu LTS 22.04 with "instal on ZFS" option, we will
add a second SSD, configure ZFS to use both SSDs in mirrored
configuration and setup grub to boot from either SSD. This
is intended to create a full redundant system where failure
of either SSD does not break the system.

== partition ==

* identify first SSD
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* connect second SSD of identical size
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
/dev/sdb WD Blue SA510 2.5 250GB 22243Z803852 25 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* if second SSD is not autodetected, reboot
* Clone partition table automatically
If both SSDs are identical size, use this simpler method of duplicating the partition table:
<pre>
root@midm9b:~# sfdisk -d /dev/sda > part_table
root@midm9b:~# grep -v ^label-id part_table | sed -e 's/, *uuid=[0-9A-F-]*//' | sfdisk /dev/sdb
</pre>
The grep and sed in the second command are there to prevent disk ID and partition IDs from being cloned. Alternatively the part_table file can be edited manually to remove the label-id line and the uuid entries from the individual partitions.

* Clone partition table manually (e.g. for different size disks)
* list partition table of first SSD:
<pre>
root@midm9b:~# fdisk -l /dev/sda
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>
* create identical partitions on second SSD, use sector numbers from above.
<pre>
root@midm9b:~# gdisk /dev/sdb
GPT fdisk (gdisk) version 1.0.8

Partition table scan:
MBR: not present
BSD: not present
APM: not present
GPT: not present

Creating new GPT entries in memory.

Command (? for help): n
Partition number (1-128, default 1):
First sector (34-488397134, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-488397134, default = 488397134) or {+-}size{KMGTP}: 1050623
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): ef00
Changed type of partition to 'EFI system partition'

Command (? for help): n
Partition number (2-128, default 2):
First sector (34-488397134, default = 1050624) or {+-}size{KMGTP}:
Last sector (1050624-488397134, default = 488397134) or {+-}size{KMGTP}: 5244927
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 8200
Changed type of partition to 'Linux swap'

Command (? for help): n
Partition number (3-128, default 3):
First sector (34-488397134, default = 5244928) or {+-}size{KMGTP}:
Last sector (5244928-488397134, default = 488397134) or {+-}size{KMGTP}: 9439231
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): be00
Changed type of partition to 'Solaris boot'

Command (? for help): n
Partition number (4-128, default 4):
First sector (34-488397134, default = 9439232) or {+-}size{KMGTP}:
Last sector (9439232-488397134, default = 488397134) or {+-}size{KMGTP}:
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): bf00
Changed type of partition to 'Solaris root'

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sdb.
The operation has completed successfully.
root@midm9b:~# fdisk -l /dev/sda /dev/sdb
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root

Disk /dev/sdb: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: EB251739-30C6-422F-A505-5887B5A0B603

Device Start End Sectors Size Type
/dev/sdb1 2048 1050623 1048576 512M EFI System
/dev/sdb2 1050624 5244927 4194304 2G Linux swap
/dev/sdb3 5244928 9439231 4194304 2G Solaris boot
/dev/sdb4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>

== update ZFS pools ==

* identify second SSD partitions
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part2
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part2 -> ../../sdb2
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
</pre>
* convert bpool from single disk to mirrored disk:
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~# zpool attach bpool 99e03dc0-7d4d-f24b-8fa1-f042b9f135db /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3
root@midm9b:~# zpool status bpool
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors
</pre>
* convert rpool
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
root@midm9b:~# zpool attach rpool f6fd54f8-3af7-b943-ae3d-a4e480537fb9 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4
root@midm9b:~# zpool status rpool
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Jan 20 19:40:45 2023
5.83G scanned at 664M/s, 2.92M issued at 332K/s, 9.11G total
0B resilvered, 0.03% done, no estimated completion time
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~#
</pre>
* wait for resilver to complete
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
scan: resilvered 9.65G in 00:00:36 with 0 errors on Fri Jan 20 19:41:21 2023
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
</pre>

== update boot loader ==

INSTALL SYSLINUX: https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#EFI_boot_using_syslinux

DO *NOT* DO THE FOLOWING:

* create and mount EFI partitions:
<pre>
root@midm9b:~# mkfs.msdos /dev/sdb1
root@midm9b:~# mkdir /boot/efi-sda
root@midm9b:~# mkdir /boot/efi-sdb
root@midm20c:~# blkid | grep vfat ### identify UUID
/dev/sdb1: UUID="DD89-5081" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="d0cb6be4-2f67-5b42-9b26-9e6905e9f774"
/dev/sdc1: UUID="D970-86BA" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="e6d3b5b9-a512-44a2-9205-1a4db06ed2a2"
/dev/sda1: UUID="DDA1-044C" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="6dc9dff0-1c13-8045-a906-7803d3074c70"
root@midm20c:~# cat /etc/fstab | grep vfat ### add mount points with correct UUID
#UUID=D970-86BA /boot/efi vfat umask=0022,fmask=0022,dmask=0022 0 1
UUID=DDA1-044C /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022 0 1
UUID=DD89-5081 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022 0 1
root@midm9b:~# mount -a
root@midm9b:~# df -kl
Filesystem 1K-blocks Used Available Use% Mounted on
...
/dev/sda1 523244 13720 509524 3% /boot/efi
/dev/sdb1 523244 4 523240 1% /boot/efi-sdb
...
root@midm9b:~# rsync -av /boot/efi/ /boot/efi-sdb/
sending incremental file list
EFI/
...
root@midm9b:~# ls -l /boot/efi-sda
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~# ls -l /boot/efi-sdb
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~#
</pre>
* add systemd "nofail" flag to /etc/fstab, without this, systemd will stop booting if one SSD is missing
<pre>
daq00:~$ cat /etc/fstab | grep vfat
#UUID=31A7-24BE /boot/efi vfat umask=0022,fmask=0022,dmask=0022 0 1
/dev/sda1 /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
/dev/sdb1 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
</pre>
* setup script to update grub on second SSD, it must be run manually after every kernel update
<pre>
root@midm9b:~# ln -s ~/git/scripts/etc/update_efi_grub.perl ~/
root@midm9b:~# ~/update_efi_grub.perl -u
EFI dir: /boot/efi-sda
/boot/efi-sda: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sda/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sda: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sda/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
EFI dir: /boot/efi-sdb
/boot/efi-sdb: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sdb/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sdb: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sdb/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
root@midm9b:~#
</pre>

= Disable NetworkManager =

== Debian-12 ==

* use netplan, https://www.debian.org/doc/manuals/debian-reference/ch05.en.html#_the_modern_network_configuration_for_cloud
* apt install netplan.io
* systemctl enable systemd-networkd
* systemctl restart systemd-networkd
* create /etc/netplan/50-dhcp.yaml
<pre>
network:
version: 2
ethernets:
all-en:
match:
name: "en*"
dhcp4: true
dhcp6: true
ignore-carrier: true ### do not drop IP address if network link drops
</pre>
* netplan apply
* netplan try
* ifconfig -a ### to check IP address settings
* netstat -rn ### to check default route
* cat /etc/resolv.conf ### to check DNS
* ls -l /run/systemd/netif/leases ### systemd-networkd dhcp leases
* NOTE: without "ignore-carrier" it will drop the IP address if network link drops, re-do dhcp when links comes back
* NOTE: wait-network-online will wait for all interfaces to get an IP address

== Ubuntu-20 ==

NOTE: THIS IS BROKEN IN UBUNTU LTS 22.04

NetworkManager is useful for configuring dynamic
network interfaces, i.e. laptops that often move
between networks, or connect to multiple choice
of wifi networks, etc.

For machines with statically configured network interfaces,
NetworkManager is not necessary.

As it has been observed to become confused and observed
to malfunction when network links go up and down (it keeps
unnecessarily reconfiguring the ip address, etc), it can
be usefuil to disable it.

* list all network interfaces
<pre>
# /bin/ls -1 /sys/class/net/
enp0s31f6
lo
</pre>
* edit /etc/network/interfaces:
<pre>
rename enp0s31f6=eth0
auto eth0
iface eth0 inet static
address 142.90.120.94/19
gateway 142.90.100.18
</pre>
* statically configure systemd-resolved
** create /etc/systemd/resolved.conf.d/resolved.conf with this contents:
<pre>
[Resolve]
DNS=142.90.100.19
Domains=triumf.ca
</pre>
** systemctl restart systemd-resolved
** resolvectl
** systemd-analyze cat-config systemd/resolved.conf
* disable NetworkManager
<pre>
systemctl disable NetworkManager
</pre>
* reboot

== U-22, U-24 ifup-ko ==

Network configuration of modern linux is confused. There are at least 3 configuration methods, each with different shortcomings:
* the old ifup method is barely documented
* NetworkManager is well documented and tooled, but sometimes does strange things
* systemd-networkd is mysterious, and likely to do strange stuff, like all systemd stuff
* netplan is the latest method, configuration is simple but uses NetworkManager or systemd-networkd as backend.

This is a solution for a specific situation of fixed computer with one fixed wired interface and maybe one or more additional interfaces for fixed wired private networks.

Install /etc/ifup-ko, edit it with IP addresses of main and additional interfaces, let systemd run it in the right place in the boot sequence replacing NetworkManager and NetworkManager-wait-online .

As bonus, /etc/ifup-ko waits up to 10 seconds for the main interface link to come up. If this is not needed, comment it out from the script.

* prepare
<pre>
cd ~/git/scripts
git pull
cd ifup-ko
make install
</pre>
* confirm interface names
<pre>
systemctl start ifup-ko ### should finish immediately or after 10 seconds
systemctl status ifup-ko -n 1000 ### observe list of interfaces is correct, name of main interface is correct
</pre>
* edit /etc/ifup-ko
** add host IP address to the "ifconfig" line
** add gateway IP address to the "ip route add" line
* test
<pre>
systemctl start ifup-ko ### should finish immediately
systemctl status ifup-ko -n 1000 ### observe everything is configured as expected
</pre>
* cut-over
<pre>
systemctl disable networkd-dispatcher
systemctl disable NetworkManager
systemctl disable wpa_supplicant # if no Wifi or Wifi not in use
</pre>
* reboot

= Disable systemd-networkd =

On netbooted machines, systemd-networkd should be disabled - when "apt upgrade" runs
and needs to update and configure systemd, it will stop systemd-networkd, which
will stop the network, which will stop the NFS mounted root filesystem, which will stop
the machine.

<pre>
systemctl disable systemd-networkd.service
systemctl disable systemd-networkd.socket
systemctl mask systemd-networkd.service
systemctl mask systemd-networkd.socket
</pre>

= Configure ECC memory =

== Configure EDAC ==

* apt install edac-utils rasdaemon

=== Intel i3-2120 ===
<pre>
root@musr00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X9SCL/X9SCM
root@musr00:~# edac-ctl --status
edac-ctl: drivers not loaded.
</pre>

=== Intel E-2236 ===
<pre>
root@daq00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SCM-F
root@daq00:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq00:~# edac-util
edac-util: No errors to report.
root@daq00:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
</pre>
* check edac sysfs files (Intel)
<pre>
root@daq00:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_noinfo_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 max_location
-r--r--r-- 1 root root 4096 Jan 25 15:10 mc_name
drwxr-xr-x 2 root root 0 Jan 25 15:10 power
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank0
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank1
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank2
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank3
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank4
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank5
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank6
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank7
--w------- 1 root root 4096 Jan 25 15:10 reset_counters
-r--r--r-- 1 root root 4096 Jan 25 15:10 seconds_since_reset
-r--r--r-- 1 root root 4096 Jan 25 15:10 size_mb
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Jan 25 15:10 uevent
root@daq00:~#
</pre>

=== Intel E3-1270 v6 ===
<pre>
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --status
edac-ctl: drivers are loaded.
root@grsnis01:~# edac-util
edac-util: No errors to report.
root@grsnis01:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@grsnis01:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 max_location
-r--r--r-- 1 root root 4096 Feb 19 12:35 mc_name
drwxr-xr-x 2 root root 0 Feb 19 12:35 power
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank0
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank1
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank2
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank3
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank4
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank5
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank6
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank7
--w------- 1 root root 4096 Feb 19 12:35 reset_counters
-r--r--r-- 1 root root 4096 Feb 19 12:35 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 19 12:35 size_mb
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 19 12:35 uevent
root@grsnis01:~#
</pre>

=== Intel E3-1245 v6 ===

<pre>
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --status
edac-ctl: drivers are loaded.
[root@alphagdaq ~]# edac-util
edac-util: No errors to report.
[root@alphagdaq ~]# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
[root@alphagdaq ~]# ras-mc-ctl --layout
+-----------------------------------------------+
| mc0 |
| csrow0 | csrow1 | csrow2 | csrow3 |
----------+-----------------------------------------------+
channel1: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
----------+-----------------------------------------------+
[root@alphagdaq ~]# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#3channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#3channel#1 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#2channel#0 0 0
[root@alphagdaq ~]# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SSH-F
[root@alphagdaq ~]# ras-mc-ctl --summary
DBD::SQLite::db prepare failed: no such table: mc_event at /usr/sbin/ras-mc-ctl line 1129.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1130.
[root@alphagdaq ~]#
</pre>

=== AMD 3700X ===

(memory is non-ECC)

<pre>
root@daq13:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@daq13:~#
root@daq13:~#
root@daq13:~# edac-ctl --status
edac-ctl: drivers not loaded.
root@daq13:~# edac-util
edac-util: Error: No memory controller data found.
root@daq13:~# edac-util -s
edac-util: EDAC drivers loaded. No memory controllers found
root@daq13:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 2 root root 0 Jan 25 15:26 power
lrwxrwxrwx 1 root root 0 Jan 21 16:16 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Jan 21 16:16 uevent
</pre>

(memory is ECC)

<pre>
root@trinatdaq:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@trinatdaq:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@trinatdaq:~# edac-util
edac-util: No errors to report.
root@trinatdaq:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Dec 15 13:04 mc0
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
lrwxrwxrwx 1 root root 0 Dec 13 18:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dec 13 18:31 uevent
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_noinfo_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 max_location
-r--r--r-- 1 root root 4096 Dec 15 13:04 mc_name
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank4
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank5
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank6
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank7
--w------- 1 root root 4096 Dec 15 13:04 reset_counters
-rw-r--r-- 1 root root 4096 Dec 15 13:04 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Dec 15 13:04 seconds_since_reset
-r--r--r-- 1 root root 4096 Dec 15 13:04 size_mb
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Dec 15 13:04 uevent
root@trinatdaq:~#
</pre>

=== AMD 5000G ===

* no linux driver for AMD 5000-series "G" CPU
* no mention of ECC in the BIOS settings
* unclear status of ECC support in AMD documentation (sais only "pro" "G" CPUs have ECC)
* unclear status of ECC support in ASUS documentation (web page out of date)

=== AMD 5600X ===

<pre>
root@daq17:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-XE GAMING WIFI
root@daq17:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq17:~# edac-util
edac-util: No errors to report.
root@daq17:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@daq17:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Aug 19 19:27 mc0
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
lrwxrwxrwx 1 root root 0 May 10 10:11 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 May 10 10:11 uevent
root@daq17:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_noinfo_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 max_location
-r--r--r-- 1 root root 4096 Aug 19 19:27 mc_name
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank4
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank5
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank6
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank7
--w------- 1 root root 4096 Aug 19 19:27 reset_counters
-rw-r--r-- 1 root root 4096 Aug 19 19:27 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Aug 19 19:27 seconds_since_reset
-r--r--r-- 1 root root 4096 Aug 19 19:27 size_mb
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Aug 19 19:27 uevent
root@daq17:~#
</pre>

=== AMD 3955WX ===

<pre>
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --status
edac-ctl: drivers are loaded.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util
edac-util: No errors to report.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@alphasuperdaq:~/git/scripts/quotareport# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 19 root root 0 Dez 12 04:48 mc0
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
lrwxrwxrwx 1 root root 0 Dez 9 05:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dez 9 05:31 uevent
root@alphasuperdaq:~/git/scripts/quotareport#
root@alphasuperdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 max_location
-r--r--r-- 1 root root 4096 Feb 28 22:19 mc_name
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank0
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank1
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank10
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank11
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank12
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank13
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank14
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank15
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank2
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank3
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank4
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank5
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank6
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank7
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank8
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank9
--w------- 1 root root 4096 Feb 28 22:19 reset_counters
-rw-r--r-- 1 root root 4096 Feb 28 22:19 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Feb 28 22:19 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 28 22:19 size_mb
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 28 22:19 uevent
root@alphasuperdaq:~#
root@alphasuperdaq:~# ras-mc-ctl --layout
Use of uninitialized value $max_pos[3] in modulus (%) at /usr/sbin/ras-mc-ctl line 868.
Use of uninitialized value $d in numeric ge (>=) at /usr/sbin/ras-mc-ctl line 869.
Use of uninitialized value $d in sprintf at /usr/sbin/ras-mc-ctl line 872.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| mc0 |
| csrow0 | csrow1 |
| channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 | channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

0: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
root@alphasuperdaq:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#2 0 0
mc#0csrow#1channel#7 0 0
mc#0csrow#0channel#3 0 0
mc#0csrow#1channel#4 0 0
mc#0csrow#1channel#2 0 0
mc#0csrow#0channel#7 0 0
mc#0csrow#1channel#3 0 0
mc#0csrow#0channel#4 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#5 0 0
mc#0csrow#0channel#6 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#0channel#5 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#6 0 0
root@alphasuperdaq:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: ASUSTeK COMPUTER INC. model Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@alphasuperdaq:~#
</pre>

=== AMD 7700X ===

<pre>
root@dsfe05:~# apt install edac-utils
root@dsfe05:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro H13SAE-MF
root@dsfe05:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@dsfe05:~# edac-util
edac-util: No errors to report.
root@dsfe05:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@dsfe05:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 May 14 09:33 ce_count
-r--r--r-- 1 root root 4096 May 14 09:33 ce_noinfo_count
-r--r--r-- 1 root root 4096 May 14 09:33 max_location
-r--r--r-- 1 root root 4096 May 14 09:33 mc_name
drwxr-xr-x 2 root root 0 May 14 09:33 power
drwxr-xr-x 3 root root 0 May 14 09:33 rank4
drwxr-xr-x 3 root root 0 May 14 09:33 rank5
--w------- 1 root root 4096 May 14 09:33 reset_counters
-r--r--r-- 1 root root 4096 May 14 09:33 seconds_since_reset
-r--r--r-- 1 root root 4096 May 14 09:33 size_mb
-r--r--r-- 1 root root 4096 May 14 09:33 ue_count
-r--r--r-- 1 root root 4096 May 14 09:33 ue_noinfo_count
-rw-r--r-- 1 root root 4096 May 14 09:33 uevent
root@dsfe05:~#
</pre>

== Configure rasdaemon ==

<pre>
apt install rasdaemon
</pre>
<pre>
systemctl enable rasdaemon
systemctl restart rasdaemon
systemctl status rasdaemon
</pre>

<pre>
● rasdaemon.service - RAS daemon to log the RAS events
Loaded: loaded (/lib/systemd/system/rasdaemon.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-01-25 15:16:37 PST; 3min 5s ago
Main PID: 2477175 (rasdaemon)
Tasks: 1 (limit: 76958)
Memory: 17.1M
CGroup: /system.slice/rasdaemon.service
└─2477175 /usr/sbin/rasdaemon -f -r

Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Listening to events for cpus 0 to 11
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mc_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording aer_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording extlog_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mce_record events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording arm_event events
</pre>

== Get reports ==

* Intel 2x32GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --layout
+-------------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-------------------------+
channel1: | 16384 MB | 16384 MB |
channel0: | 16384 MB | 16384 MB |
----------+-------------------------+
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#0channel#1 0 0
root@daq00:~#
</pre>

* Intel 4x16GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#1 0 0
mc#0csrow#2channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#3channel#0 0 0
mc#0csrow#3channel#1 0 0
root@daq00:~#
root@daq00:~# ras-mc-ctl --layout
+-----------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-----------------------+
channel1: | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB |
----------+-----------------------+
root@daq00:~#
root@daq00:~#
root@daq00:~#
root@daq00:~# ras-mc-ctl --print-labels
ras-mc-ctl: Error: No dimm labels for Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@daq00:~#
</pre>

note: ubuntu LTS 22.04 DBD::SQLite::db error is not there.

* AMD 7700 2x32GB DDR5 ECC DIMMs

<pre>
root@dsfe05:~# systemctl status rasdaemon
● rasdaemon.service - RAS daemon to log the RAS events
Loaded: loaded (/lib/systemd/system/rasdaemon.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2024-05-14 09:36:43 PDT; 33ms ago
Process: 4088418 ExecStartPost=/usr/sbin/rasdaemon --enable (code=exited, status=0/SUCCESS)
Main PID: 4088417 (rasdaemon)
Tasks: 1 (limit: 37300)
Memory: 788.0K
CPU: 5ms
CGroup: /system.slice/rasdaemon.service
└─4088417 /usr/sbin/rasdaemon -f -r

May 14 09:36:43 dsfe05 rasdaemon[4088417]: ras:aer_event event enabled
May 14 09:36:43 dsfe05 rasdaemon[4088417]: Enabled event ras:aer_event
May 14 09:36:43 dsfe05 rasdaemon[4088417]: mce:mce_record event enabled
May 14 09:36:43 dsfe05 rasdaemon[4088417]: Enabled event mce:mce_record
May 14 09:36:43 dsfe05 rasdaemon[4088417]: ras:extlog_mem_event event enabled
May 14 09:36:43 dsfe05 rasdaemon[4088417]: Enabled event ras:extlog_mem_event
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording mc_event events
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording aer_event events
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording extlog_event events
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording mce_record events
root@dsfe05:~# ras-mc-ctl --layout
Use of uninitialized value $max_pos[3] in modulus (%) at /usr/sbin/ras-mc-ctl line 907.
Use of uninitialized value $d in numeric ge (>=) at /usr/sbin/ras-mc-ctl line 908.
Use of uninitialized value $d in sprintf at /usr/sbin/ras-mc-ctl line 911.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
+-----------------------------------------------------------------------------------------------+
| mc0 |
| csrow0 | csrow1 | csrow2 | csrow3 |
| channel0 | channel1 | channel0 | channel1 | channel0 | channel1 | channel0 | channel1 |
----+-----------------------------------------------------------------------------------------------+

0: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB |
----+-----------------------------------------------------------------------------------------------+
root@dsfe05:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#2channel#1 0 0
mc#0csrow#2channel#0 0 0
root@dsfe05:~# ras-mc-ctl --print-labels
ras-mc-ctl: Error: No dimm labels for Supermicro model H13SAE-MF
root@dsfe05:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model H13SAE-MF
root@dsfe05:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

No MCE errors.
root@dsfe05:~#
</pre>

= sensors =

== VME CPU V7865 ==

add to /etc/rc.local

<pre>
modprobe coretemp
modprobe lm75
modprobe lm90
modprobe max1668
</pre>

available sensors:

<pre>
root@lxdaq23:~# sensors
max6657-i2c-0-4c
Adapter: SMBus I801 adapter at 0400
temp1: +29.1°C (low = -55.0°C, high = +105.0°C)
(crit = +105.0°C, hyst = +95.0°C)
temp2: +31.5°C (low = -55.0°C, high = +105.0°C)
(crit = +105.0°C, hyst = +95.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Core 0: +25.0°C (crit = +100.0°C)
Core 1: +25.0°C (crit = +100.0°C)

max1805-i2c-0-18
Adapter: SMBus I801 adapter at 0400
temp1: +35.0°C (low = -55.0°C, high = +127.0°C)
temp2: +63.0°C (low = -55.0°C, high = +127.0°C)
temp3: FAULT (low = -55.0°C, high = +127.0°C) ALARM (HIGH)

lm75-i2c-0-48
Adapter: SMBus I801 adapter at 0400
temp1: +34.5°C (high = +80.0°C, hyst = +75.0°C)

root@lxdaq23:~#
</pre>

== ASUS P7P55D EVO ==

* BIOS version 2101

<pre>
root@iris01:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Core 0: +34.0°C (high = +83.0°C, crit = +99.0°C)
Core 1: +37.0°C (high = +83.0°C, crit = +99.0°C)
Core 2: +38.0°C (high = +83.0°C, crit = +99.0°C)
Core 3: +35.0°C (high = +83.0°C, crit = +99.0°C)

nouveau-pci-0100
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.05 V)
temp1: +46.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

atk0110-acpi-0
Adapter: ACPI interface
Vcore Voltage: 864.00 mV (min = +0.80 V, max = +1.60 V)
+3.3V Voltage: 3.38 V (min = +2.97 V, max = +3.63 V)
+5V Voltage: 5.04 V (min = +4.50 V, max = +5.50 V)
+12V Voltage: 12.15 V (min = +10.20 V, max = +13.80 V)
CPU Fan Speed: 968 RPM (min = 600 RPM, max = 7200 RPM)
Chassis1 Fan Speed: 1288 RPM (min = 600 RPM, max = 7200 RPM)
Chassis2 Fan Speed: 1316 RPM (min = 600 RPM, max = 7200 RPM)
Power Fan Speed: 0 RPM (min = 0 RPM, max = 7200 RPM)
CPU Temperature: +34.0°C (high = +45.0°C, crit = +45.5°C)
MB Temperature: +30.0°C (high = +45.0°C, crit = +46.0°C)

root@iris01:~#
</pre>

== ASUS Z97-WS ==

* BIOS version 2704
* load sensors drivers
<pre>
echo modprobe coretemp >> /etc/rc.local
echo modprobe nct6775 >> /etc/rc.local
</pre>
* in /boot/grub/grub.cfg, add: GRUB_CMDLINE_LINUX_DEFAULT="acpi_enforce_resources=no"
* update grub and reboot: grub-mkconfig -o /boot/grub/grub.cfg

<pre>
root@isdaq08:~# sensors
acpitz-acpi-0
Adapter: ACPI interface
temp1: +27.8°C
temp2: +29.8°C

nct6791-isa-0290
Adapter: ISA adapter
Vcore: 888.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
AVCC: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
+3.3V: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.99 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 0.00 V (min = +0.00 V, max = +0.00 V)
3VSB: 3.44 V (min = +0.00 V, max = +0.00 V) ALARM
Vbat: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 0.00 V (min = +0.00 V, max = +0.00 V)
in11: 840.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in12: 0.00 V (min = +0.00 V, max = +0.00 V)
in13: 0.00 V (min = +0.00 V, max = +0.00 V) ALARM
in14: 0.00 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1041 RPM (min = 0 RPM)
fan2: 1040 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan6: 0 RPM (min = 0 RPM)
SYSTIN: +34.0°C (high = +0.0°C, hyst = +0.0°C) ALARM sensor = thermistor
CPUTIN: +41.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: -128.0°C sensor = thermistor
AUXTIN1: -128.0°C sensor = thermistor
AUXTIN2: +35.0°C sensor = thermistor
AUXTIN3: +127.0°C sensor = thermistor
PECI Agent 0: +41.0°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
PCH_DIM0_TEMP: +0.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +42.0°C (high = +80.0°C, crit = +100.0°C)
Core 0: +42.0°C (high = +80.0°C, crit = +100.0°C)
Core 1: +40.0°C (high = +80.0°C, crit = +100.0°C)
Core 2: +39.0°C (high = +80.0°C, crit = +100.0°C)
Core 3: +39.0°C (high = +80.0°C, crit = +100.0°C)

root@isdaq08:~#
</pre>

== ASUS Z170-DELUXE ==

* BIOS version 3801
* load sensors drivers
<pre>
echo modprobe coretemp >> /etc/rc.local
echo modprobe jc42 >> /etc/rc.local
echo modprobe lm92 >> /etc/rc.local
echo modprobe nct6775 >> /etc/rc.local
</pre>
* in /etc/default/grub, add: GRUB_CMDLINE_LINUX_DEFAULT="acpi_enforce_resources=no"
* update grub and reboot: grub-mkconfig -o /boot/grub/grub.cfg

<pre>
root@iris00:~# sensors
nct6793-isa-0290
Adapter: ISA adapter
in0: 600.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 144.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in6: 0.00 V (min = +0.00 V, max = +0.00 V)
in7: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.14 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1000.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in10: 600.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 592.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 968.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1370 RPM (min = 0 RPM)
fan2: 1437 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan6: 0 RPM (min = 0 RPM)
SYSTIN: +32.0°C (high = +98.0°C, hyst = +95.0°C) sensor = thermistor
CPUTIN: +42.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: -128.0°C sensor = thermistor
AUXTIN1: +50.0°C sensor = thermistor
AUXTIN2: +22.0°C sensor = thermistor
AUXTIN3: +28.0°C sensor = thermistor
PECI Agent 0: +50.0°C (high = +98.0°C, hyst = +95.0°C)
(crit = +100.0°C)
PECI Agent 0 Calibration: +42.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
TSI2_TEMP: +3892314.0°C
TSI3_TEMP: +3892314.0°C
TSI4_TEMP: +3892314.0°C
TSI5_TEMP: +3892314.0°C
TSI6_TEMP: +3892314.0°C
TSI7_TEMP: +3892314.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

jc42-i2c-0-1a
Adapter: SMBus I801 adapter at f040
temp1: +36.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

jc42-i2c-0-18
Adapter: SMBus I801 adapter at f040
temp1: +34.8°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

jc42-i2c-0-1b
Adapter: SMBus I801 adapter at f040
temp1: +35.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

jc42-i2c-0-19
Adapter: SMBus I801 adapter at f040
temp1: +36.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +52.0°C (high = +84.0°C, crit = +100.0°C)
Core 0: +52.0°C (high = +84.0°C, crit = +100.0°C)
Core 1: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 2: +48.0°C (high = +84.0°C, crit = +100.0°C)
Core 3: +47.0°C (high = +84.0°C, crit = +100.0°C)

root@iris00:~#
</pre>

== ASUS Z390M-PRO GAMING (WI-FI) ==

* BIOS 3006
* load sensors drivers
<pre>
echo modprobe coretemp >> /etc/rc.local
echo modprobe nct6775 >> /etc/rc.local
</pre>

<pre>
root@daq18:~# sensors
nct6798-isa-0290
Adapter: ISA adapter
in0: 696.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.42 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 208.00 mV (min = +0.00 V, max = +0.00 V)
in6: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.42 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.17 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.07 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.36 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.33 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in14: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 0 RPM (min = 0 RPM)
fan2: 1131 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 1006 RPM (min = 0 RPM)
fan6: 0 RPM (min = 0 RPM)
fan7: 0 RPM (min = 0 RPM)
SYSTIN: +32.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
CPUTIN: +29.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
AUXTIN0: +25.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
AUXTIN1: +7.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
AUXTIN2: +8.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
AUXTIN3: +24.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
AUXTIN4: +83.0°C (high = +80.0°C, hyst = +75.0°C) ALARM
(crit = +100.0°C)
PECI Agent 0 Calibration: +29.0°C (high = +80.0°C, hyst = +75.0°C)
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +39.0°C (high = +82.0°C, crit = +100.0°C)
Core 0: +39.0°C (high = +82.0°C, crit = +100.0°C)
Core 1: +33.0°C (high = +82.0°C, crit = +100.0°C)
Core 2: +32.0°C (high = +82.0°C, crit = +100.0°C)
Core 3: +31.0°C (high = +82.0°C, crit = +100.0°C)
Core 4: +31.0°C (high = +82.0°C, crit = +100.0°C)
Core 5: +30.0°C (high = +82.0°C, crit = +100.0°C)

acpitz-acpi-0
Adapter: ACPI interface
temp1: +27.8°C

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: +28.0°C

root@daq18:~#
</pre>

== ASUS H110M-A/M.2 ==

* BIOS version 4202
* echo modprobe coretemp >> /etc/rc.local
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@midpol:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +33.0°C (high = +80.0°C, crit = +100.0°C)
Core 0: +33.0°C (high = +80.0°C, crit = +100.0°C)
Core 1: +30.0°C (high = +80.0°C, crit = +100.0°C)

acpitz-acpi-0
Adapter: ACPI interface
temp1: +27.8°C (crit = +119.0°C)
temp2: +29.8°C (crit = +119.0°C)

nct6793-isa-0290
Adapter: ISA adapter
in0: 368.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 152.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in6: 928.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.14 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1000.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in10: 152.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in11: 128.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in12: 136.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in13: 120.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 136.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1004 RPM (min = 0 RPM)
fan2: 1143 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan6: 0 RPM (min = 0 RPM)
SYSTIN: +118.0°C (high = +98.0°C, hyst = +95.0°C) sensor = thermistor
CPUTIN: +29.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +30.0°C sensor = thermistor
AUXTIN1: +112.0°C sensor = thermistor
AUXTIN2: +111.0°C sensor = thermistor
AUXTIN3: +110.0°C sensor = thermistor
PECI Agent 0: +31.0°C (high = +98.0°C, hyst = +95.0°C)
(crit = +100.0°C)
PECI Agent 0 Calibration: +36.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
TSI2_TEMP: +3892314.0°C
TSI3_TEMP: +3892314.0°C
TSI4_TEMP: +3892314.0°C
TSI5_TEMP: +3892314.0°C
TSI6_TEMP: +3892314.0°C
TSI7_TEMP: +3892314.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

root@midpol:~#
</pre>

== ASUS P9X79 WS ==

* https://www.asus.com/supportonly/P9X79%20WS/HelpDesk_Manual/
* BIOS version 4802
* modprobe nct6775
* modprobe coretemp

<pre>
root@daq14:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 0: +29.0°C (high = +82.0°C, crit = +100.0°C)
Core 1: +24.0°C (high = +82.0°C, crit = +100.0°C)
Core 2: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +82.0°C, crit = +100.0°C)

nouveau-pci-0200
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.00 V)
temp1: +39.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

nct6776-isa-0290
Adapter: ISA adapter
Vcore: 1.04 V (min = +0.00 V, max = +1.74 V)
in1: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
AVCC: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
+3.3V: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 2.04 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 904.00 mV (min = +0.00 V, max = +0.00 V) ALARM
3VSB: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
Vbat: 3.30 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1265 RPM (min = 0 RPM)
fan2: 1909 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +34.0°C (high = +0.0°C, hyst = +0.0°C) ALARM sensor = thermistor
CPUTIN: +58.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermal diode
AUXTIN: +31.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
PECI Agent 0: +31.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +96.0°C)
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

root@daq14:~#
</pre>

== ASUS TUF GAMING B550M-PLUS WIFI II ==

* BIOS 2803, 2806
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@midm9a:~# sensors
nct6798-isa-0290
Adapter: ISA adapter
in0: 488.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.82 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in14: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 0 RPM (min = 0 RPM)
fan2: 760 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan7: 1264 RPM (min = 0 RPM)
SYSTIN: +25.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +22.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +95.0°C sensor = thermistor
AUXTIN1: +25.0°C sensor = thermistor
AUXTIN2: +25.0°C sensor = thermistor
AUXTIN3: +25.0°C sensor = thermistor
PECI Agent 0 Calibration: +23.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +32.4°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

amdgpu-pci-0800
Adapter: PCI adapter
vddgfx: 1.45 V
vddnb: 993.00 mV
edge: +28.0°C
PPT: 20.00 W

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +33.4°C

root@midm9a:~#
</pre>

== ASUS ASUS ROG STRIX B550-XE GAMING WIFI ==

* BIOS 2423, 2604
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@daq13:~# sensors
nct6798-isa-0290
Adapter: ISA adapter
in0: 344.00 mV (min = +0.00 V, max = +1.74 V)
in1: 992.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in6: 216.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.30 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.81 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in11: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 280.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 845 RPM (min = 0 RPM)
fan2: 998 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +28.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +27.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +94.0°C sensor = thermistor
AUXTIN1: +28.0°C sensor = thermistor
AUXTIN2: +28.0°C sensor = thermistor
AUXTIN3: +97.0°C sensor = thermistor
PECI Agent 0 Calibration: +27.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +33.6°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

amdgpu-pci-0600
Adapter: PCI adapter
vddgfx: 1.45 V
vddnb: 999.00 mV
edge: +29.0°C
PPT: 14.00 W

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: +30.0°C

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +33.9°C

root@daq13:~#
</pre>

== ASUS ASUS ROG STRIX B550-E GAMING ==

* bios 2803
* echo modprobe jc42 >> /etc/rc.local
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@daq17:~# sensors
jc42-i2c-1-1b
Adapter: SMBus PIIX4 adapter port 0 at 0b00
temp1: +25.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: +28.0°C

nouveau-pci-0800
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.00 V)
temp1: +34.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

nct6798-isa-0290
Adapter: ISA adapter
in0: 288.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 224.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.31 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.79 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 280.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 843 RPM (min = 0 RPM)
fan2: 629 RPM (min = 0 RPM)
fan3: 746 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +22.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +25.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +93.0°C sensor = thermistor
AUXTIN1: +22.0°C sensor = thermistor
AUXTIN2: +22.0°C sensor = thermistor
AUXTIN3: +96.0°C sensor = thermistor
PECI Agent 0 Calibration: +25.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +27.6°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

jc42-i2c-1-1a
Adapter: SMBus PIIX4 adapter port 0 at 0b00
temp1: +23.2°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

asusec-isa-0000
Adapter: ISA adapter
CPU_Opt: 0 RPM
Chipset: +34.0°C
CPU: +25.0°C
Motherboard: +22.0°C
T_Sensor: -40.0°C
VRM: +31.0°C

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +28.0°C
Tccd1: +27.5°C

root@daq17:~#
</pre>

== ASUS PRIME B650-PLUS ==

* BIOS 1811
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@dsdaqgw:~# sensors
amdgpu-pci-0b00
Adapter: PCI adapter
vddgfx: 930.00 mV
vddnb: 1.19 V
edge: +38.0°C
PPT: 25.10 W

nct6799-isa-0290
Adapter: ISA adapter
in0: 920.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.04 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 320.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.28 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.28 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.10 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.04 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 416.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 328.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 0 RPM (min = 0 RPM)
fan2: 1253 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan7: 0 RPM (min = 0 RPM)
SYSTIN: +33.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +35.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +78.0°C sensor = thermistor
AUXTIN1: +11.0°C sensor = thermistor
AUXTIN2: +20.0°C sensor = thermistor
AUXTIN3: +82.0°C sensor = thermistor
PECI Agent 0 Calibration: +35.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +42.6°C
intrusion0: ALARM
intrusion1: OK
beep_enable: disabled

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +42.6°C
Tccd1: +36.4°C

root@dsdaqgw:~#
</pre>

= Enable CPU turbo mode =

* Intel CPU has a nominal CPU frequency (i.e. 3.4GHz) and a turbo-boost CPU frequency (i.e. 4.0GHz). Here we will enable this turbo-boost mode.
* Find out CPU capability
<pre>
root@daq01:~# lscpu | grep Hz
Model name: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
CPU MHz: 3965.803
CPU max MHz: 4000.0000
CPU min MHz: 800.0000
root@daq01:~#
</pre>
* Look up this CPU in the Intel ARK database - google for the CPU model name, i.e.
https://ark.intel.com/content/www/us/en/ark/products/88196/intel-core-i7-6700-processor-8m-cache-up-to-4-00-ghz.html
* Find current frequency settings:
<pre>
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "powersave" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 2.72 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
root@daq01:~#
</pre>
* Note the following:
** current governor is "powersave"
** "performance" governor is available
** "boost state support" is supported and active.
* Confirm CPU frequency governor:
<pre>
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
powersave
powersave
powersave
powersave
powersave
powersave
powersave
powersave
root@daq01:~#
</pre>
* Change governor to "performance":
<pre>
root@daq01:~# cpupower frequency-set --governor performance
Setting cpu: 0
Setting cpu: 1
Setting cpu: 2
Setting cpu: 3
Setting cpu: 4
Setting cpu: 5
Setting cpu: 6
Setting cpu: 7
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
performance
performance
performance
performance
performance
performance
performance
performance
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "performance" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 3.93 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
</pre>
* monitor CPU frequency:
<pre>
root@daq01:~# cpupower monitor
| Nehalem || Mperf || Idle_Stats
CPU| C3 | C6 | PC3 | PC6 || C0 | Cx | Freq || POLL | C1 | C1E | C3 | C6 | C7s | C8
0| 0.00| 0.00| 0.00| 0.00|| 88.80| 11.20| 3973|| 0.00| 0.00| 0.01| 0.02| 0.31| 0.00| 4.25
4| 0.00| 0.00| 0.00| 0.00|| 4.70| 95.30| 3945|| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 95.03
1| 0.73| 3.70| 0.00| 0.00|| 4.52| 95.48| 3864|| 0.00| 0.01| 1.19| 0.44| 2.82| 0.00| 90.23
5| 0.73| 3.70| 0.00| 0.00|| 0.37| 99.63| 3807|| 0.00| 0.00| 0.03| 0.09| 1.70| 0.00| 97.64
2| 2.28| 12.86| 0.00| 0.00|| 1.41| 98.59| 3829|| 0.00| 0.86| 3.17| 0.46| 7.70| 0.00| 85.87
6| 2.28| 12.86| 0.00| 0.00|| 2.88| 97.12| 3856|| 0.00| 0.11| 4.56| 2.15| 10.31| 0.00| 78.99
3| 1.33| 4.81| 0.00| 0.00|| 0.99| 99.01| 3804|| 0.00| 0.49| 0.79| 0.01| 1.03| 0.00| 96.12
7| 1.34| 4.81| 0.00| 0.00|| 1.26| 98.74| 3818|| 0.00| 0.01| 2.32| 0.47| 5.02| 0.00| 90.06
root@daq01:~#
</pre>
* check that the CPU is not overheating:
<pre>
root@daq01:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 1: +38.0°C (high = +84.0°C, crit = +100.0°C)
Core 2: +34.0°C (high = +84.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +84.0°C, crit = +100.0°C)
</pre>
* congratulations, we are running at 4 GHz now!

= Setup ubuntu as gateway to private network =

See also:
* https://daq.triumf.ca/DaqWiki/index.php/VME-CPU#Setup_the_boot_host_computer_.28el7.29
* http://www.triumf.info/wiki/DAQwiki/index.php/Dhcpd_on_eth1

== Steps to do ==

!!! UPDATED 16feb2024 Ubuntu-22.04.03 !!!

* assign network numbers to the private network, i.e. 192.168.1.x, 192.168.2.x, etc
* (on the gateway machine, each private network interface has to have a different network number)
* (each network interface can have multiple networks attached, via VLANs or via eth0:0, eth0:1 constructs)
* assign IP addresses on the private network, save them in /etc/hosts i.e. "hvps 192.168.1.10"
* (for simplicity, assign 192.168.1.1 to the gateway machine itself)
* (IP addresses 192.168.1.0 and 192.168.1.255 are "special", do not use them)
* setup DNS server (dnsmasq) to serve contents of /etc/hosts via DNS (otherwise, many programs will see inconsistent name to IP address mapping)
* setup DHCP server (dnsmasq) to give out the IP addresses
* setup TFTP server (dnsmasq), pxelinux and NFS for diskless booting
* setup time server (chronyd) to provide common time to all devices
* setup NAT so machines on private network can access the internet (to get OS updates, etc)
* setup NIS and NFS so machines on the private network can use common home directories
* setup rsync backup of machines on the private network

== setup hosts ==

* edit /etc/hosts
<pre>
192.168.1.101 dsfe01
... and so forth
</pre>

== setup dns and dhcp ==

!!! updated 16feb2024 for Ubuntu 22.04.3 !!!

!!! note: stock systemd-resolved remains, is configured to forward queries to dnsmasq, configured to forward queries to TRIUMF DNS !!!

!!! note: per authors of systemd, bare hostnames are not permitted, a DNS domain name must always be used. DNS domain name "dsdaq" is used in this example !!!

* apt install dnsmasq
* ensure dnsmasq starts after all interfaces are up (Ubuntu-22)
<pre>
mkdir /etc/systemd/system/dnsmasq.service.d
echo -e "[Unit]\nAfter=network-online.target\n" > /etc/systemd/system/dnsmasq.service.d/local.conf
</pre>
* edit /etc/dnsmasq.conf
<pre>
# /etc/dnsmasq.conf
# DNS settings
#port=0 # disable DNS function
port=53 # enable DNS function
bind-interfaces # do not collide with systemd-resolved, we use 127.0.0.1:53, they use 127.0.0.53:53
domain-needed
bogus-priv
no-resolv
#log-queries # log DNS quesries

# TRIUMF DNS settings

server=142.90.100.19
expand-hosts
domain=dsdaq
local=/dsdaq/
localmx # do not forward MX queries to TRIUMF

# DHCP settings
interface=enp1s0f0 # VX network 192.168.0.x
#interface=missing # FEP and TSP network 192.168.1.x
interface=enp1s0f1 # controls network 192.168.2.x
#dhcp-range=192.168.1.50,192.168.1.150,infinite
dhcp-range=192.168.0.0,static
dhcp-range=192.168.2.0,static
log-dhcp # log DHCP queries
#quiet-dhcp
dhcp-ignore=tag:!known
#dhcp-boot=pxelinux.0

dhcp-option=option:dns-server,192.168.0.248
dhcp-option=option:ntp-server,192.168.0.248

# TFTP settings

enable-tftp
tftp-root=/tftpboot
</pre>
* #mkdir /tftpboot ### per tftp-root (if no ZFS)
* zfs create -o mountpoint=/tftpboot rpool/tftpboot ### (if root is ZFS)
* create resolved-dsdaq.conf with main IP address of dnsmasq
<pre>
[Resolve]
DNS=192.168.0.248
Domains=dsdaq triumf.ca
</pre>
* mkdir -p /etc/systemd/resolved.conf.d/
* /bin/rm -f /etc/systemd/resolved.conf.d/*.conf
* cp resolved-dsdaq.conf /etc/systemd/resolved.conf.d/
* systemctl stop systemd-resolved.service
* systemctl disable systemd-resolved.service
* systemctl enable dnsmasq
* systemctl restart dnsmasq
* try to "ping" or "host" some names from /etc/hosts, it should work
* try to ping daq00, daq00.triumf.ca, all should work
* resolved-dsdaq.conf goes into /etc/systemd/resolved.conf.d/ of all machines on the private network
* if not using systemd-resolved, edit /etc/resolv.conf

== setup chronyd ==

* enable ntp server:
* disable systemd-timesyncd, configure and enable chronyd per instructions above
* create dsdaq.conf
<pre>
# chrony config for dsdaq server

#allow 192.168.0.0
#allow 192.168.1.0
#allow 192.168.2.0
allow all

# end
</pre>
* cp dsdaq.conf /etc/chrony/conf.d/
* systemctl restart chronyd
* chronyc tracking ### wait until time is synchronized (a few seconds)
* create dsdaq.sources # use hostname or IP address of chronyd server
<pre>
# Put this file in /etc/chrony/sources.d
# systemctl restart chrony
# chronyc sources
# chronyc tracking
server dsdaqgw iburst prefer
# end
</pre>
* dsdaq.sources goes to /etc/chrony/sources.d of all machines on the private network

== setup diskless network booting ==

=== setup pxelinux for legacy pxe boot ===

* add bits in dnsmasq.conf
<pre>
dhcp-host=ac:1f:6b:9e:7f:4a,dsfe01,infinite
dhcp-boot=pxelinux.0
dhcp-option=17,"192.168.0.251:/nfsroot/%s,vers=3,tcp"
</pre>
* setup pxelinux for Ubuntu-18
<pre>
cd ~
wget https://www.kernel.org/pub/linux/utils/boot/syslinux/4.xx/syslinux-4.03.tar.bz2
tar xjvf syslinux-4.03.tar.bz2
cd syslinux-4.03
cp -pv ./core/pxelinux.0 ./com32/hdt/hdt.c32 ./memdisk/memdisk ./com32/menu/menu.c32 /zssd/tftpboot/
</pre>
* cd /zssd/tftpboot
<pre>
wget http://ladd00.triumf.ca/tftpboot/memtest86+-4.20.iso.zip
wget http://ladd00.triumf.ca/tftpboot/memtest86+-5.01.iso.gz
wget http://ladd00.triumf.ca/tftpboot/modules.alias
wget http://ladd00.triumf.ca/tftpboot/modules.pcimap
wget http://ladd00.triumf.ca/tftpboot/pci.ids
</pre>
* mkdir pxelinux.cfg
* emacs -nw pxelinux.cfg/default
<pre>
default menu.c32
prompt 0

menu title Welcome to the DSVSLICE PXE boot menu

timeout 50

label hdt
kernel hdt.c32

label memtest86+-5.01
kernel memdisk iso initrd=memtest86+-5.01.iso.gz

label memtest86+-4.20
kernel memdisk iso initrd=memtest86+-4.20.iso.zip

label vmlinuz-5.3.0-26-generic
menu default
kernel vmlinuz-5.3.0-26-generic
append initrd=initrd.img-5.3.0-26-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.1.1:/zssd/nfsroot/dsfe01 toram ip=dhcp panic=60 BOOTIF=enp1s0f0

#end
</pre>

=== setup pxelinux for efi pxe boot ===

* https://c-nergy.be/blog/?p=13808
* add dnsmasq.conf bits. note: to use dhcp root-path, see the "nfsroot=auto" patch below and make sure to use the "dhcp-option-force" command (mkinitramfs dhcp client does not ask for root-path, we have to force-feed it).
<pre>
# uefi pxe

dhcp-boot=tag:uefipxe,uefi/syslinux.efi
dhcp-option-force=tag:fe01,option:root-path,192.168.0.248:/nfsroot/fe01

# VX network 192.168.0.x

dhcp-host=40:a6:b7:c1:d9:c5,fe01,infinite,set:uefipxe,set:fe01
</pre>
* apt install syslinux pxelinux syslinux-common syslinux-efi syslinux-utils
<pre>
mkdir /tftpboot/uefi
cp /usr/lib/SYSLINUX.EFI/efi64/syslinux.efi /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/ldlinux.e64 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/menu.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/hdt.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libutil.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libmenu.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libcom32.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libgpl.c32 /tftpboot/uefi/
</pre>
* try to boot, it should bomb with "cannot load pxelinux.cfg/default"
* mkdir /tftpboot/uefi/pxelinux.cfg
* create /tftpboot/uefi/pxelinux.cfg/default, note nfsroot path is hardwired, note "http:" is used to load vmlinuz and initrd files (because tftp is super slow)
<pre>
default menu.c32
prompt 0

menu title Welcome to the DSDAQGW UEFI PXE boot menu

timeout 50

label vmlinuz-6.5.0-17-generic
kernel http://192.168.0.248:8088/uefi/vmlinuz-6.5.0-17-generic
append initrd=http://192.168.0.248:8088/uefi/initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=auto rw ip=dhcp panic=60

# append initrd=http://192.168.0.248:8088/uefi/initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.0.248:/nfsroot/fe01 rw ip=dhcp panic=60

# append initrd=initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.0.248:/nfsroot/fe01 rw ip=dhcp panic=60
# append initrd=initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=auto ip=dhcp rw panic=60

#end
</pre>
* try to boot, it will bomb with "cannot load http://...."
* install mini_httpd on port 8088, see https://acme.com/software/mini_httpd/
<pre>
apt install mini-httpd
emacs -nw /etc/default/mini-httpd # set "START=1"
emacs -nw /etc/mini-httpd.conf # set "host=192.168.0.248", "port=8088", "data_dir=/tftpboot"
mkdir /etc/systemd/system/mini-httpd.service.d
echo -e "[Unit]\nAfter=network-online.target\n" > /etc/systemd/system/mini-httpd.service.d/local.conf
systemctl enable mini-httpd
systemctl restart mini-httpd
systemctl status mini-httpd
wget http://192.168.0.248:8088/uefi/syslinux.efi
tail -100 /var/log/mini_httpd.log
</pre>
* fix U-22 initramfs bug for "nfsroot=auto", otherwise, "nfsroot=" has to be different for each machine and you have to have separate pxelinux config files for each machine
** emacs -nw /usr/lib/initramfs-tools/etc/dhcp/dhclient-enter-hooks.d/config
** add "echo ROOTPATH=..." if it is missing
<pre>
echo "ROOTSERVER='${new_routers%% *}'"
echo "ROOTPATH='$new_root_path'"
echo "HOSTNAME='$new_host_name'"
</pre>
* fix U-24 initramfs bug for "nfsroot=auto", otherwise, "nfsroot=" has to be different for each machine and you have to have separate pxelinux config files for each machine
** emacs -nw /usr/share/initramfs-tools/dhcpcd-hooks/70-net-conf
** add "ROOTPATH=..." if it is missing
<pre>
DNSDOMAIN='${new_domain_name-}'
ROOTSERVER='${new_routers-}'
ROOTPATH='${new_root_path-}'
filename='${new_filename-}'
DHCPLEASETIME='${new_dhcp_lease_time-}'
</pre>
** regenerate initramfs (be careful you generate it for the right kernel!)
** see https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/2054482
<pre>
mkinitramfs 6.5.0-18-generic
mkinitramfs 6.8.0-51-generic -o /boot/initrd.img-6.8.0-51-generic
</pre>
* copy linux kernel and initrd
<pre>
cp /boot/vmlinuz-6.5.0-18-generic /tftpboot/uefi/
cp /boot/initrd.img-6.5.0-18-generic /tftpboot/uefi/
chmod a+r /tftpboot/uefi/*
</pre>
* try to boot, should bomb with messages about "trying to mount root filesystem"
* tail /var/log/syslog
<pre>
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 vendor class: PXEClient:Arch:00007:UNDI:003016
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 1:netmask, 2:time-offset, 3:router, 4, 5,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 6:dns-server, 12:hostname, 13:boot-file-size,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 15:domain-name, 17:root-path, 18:extension-path,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 22:max-datagram-reassembly, 23:default-ttl,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 28:broadcast, 40:nis-domain, 41:nis-server,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 42:ntp-server, 43:vendor-encap, 50:requested-address,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 51:lease-time, 54:server-identifier, 58:T1,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 59:T2, 60:vendor-class, 66:tftp-server, 67:bootfile-name,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 97:client-machine-id, 128, 129, 130, 131,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 132, 133, 134, 135
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 next server: 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 broadcast response
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 1 option: 53 message-type 2
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 18 option: 67 bootfile-name uefi/syslinux.efi
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 12 hostname fe01
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 vendor class: PXEClient:Arch:00007:UNDI:003016
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 1:netmask, 2:time-offset, 3:router, 4, 5,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 6:dns-server, 12:hostname, 13:boot-file-size,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 15:domain-name, 17:root-path, 18:extension-path,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 22:max-datagram-reassembly, 23:default-ttl,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 28:broadcast, 40:nis-domain, 41:nis-server,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 42:ntp-server, 43:vendor-encap, 50:requested-address,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 51:lease-time, 54:server-identifier, 58:T1,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 59:T2, 60:vendor-class, 66:tftp-server, 67:bootfile-name,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 97:client-machine-id, 128, 129, 130, 131,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 132, 133, 134, 135
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 next server: 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 broadcast response
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 1 option: 53 message-type 5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 18 option: 67 bootfile-name uefi/syslinux.efi
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 12 hostname fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: error 8 User aborted the transfer received from 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/syslinux.efi to 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/syslinux.efi to 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 requested options: 1:netmask, 3:router, 6:dns-server
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 bootfile name: uefi/syslinux.efi
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 next server: 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 broadcast response
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 1 option: 53 message-type 2
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 requested options: 1:netmask, 3:router, 6:dns-server
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 bootfile name: uefi/syslinux.efi
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 next server: 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 broadcast response
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 1 option: 53 message-type 5
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/ldlinux.e64 to 192.168.0.110
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/01-40-a6-b7-c1-d9-c5 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8006E not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8006 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A800 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A80 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/pxelinux.cfg/default to 192.168.0.110
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/menu.c32 to 192.168.0.110
Feb 16 20:43:10 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/libutil.c32 to 192.168.0.110
Feb 16 20:43:10 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/pxelinux.cfg/default to 192.168.0.110
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 client provides name: dsdaqgw.triumf.ca
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 1:netmask, 28:broadcast, 2:time-offset, 3:router,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 15:domain-name, 6:dns-server, 119:domain-search,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 12:hostname, 44:netbios-ns, 47:netbios-scope,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 26:mtu, 121:classless-static-route, 42:ntp-server
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 bootfile name: uefi/syslinux.efi
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 next server: 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 1 option: 53 message-type 2
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 51 lease-time infinite
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 client provides name: dsdaqgw.triumf.ca
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 1:netmask, 28:broadcast, 2:time-offset, 3:router,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 15:domain-name, 6:dns-server, 119:domain-search,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 12:hostname, 44:netbios-ns, 47:netbios-scope,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 26:mtu, 121:classless-static-route, 42:ntp-server
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 bootfile name: uefi/syslinux.efi
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 next server: 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 1 option: 53 message-type 5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 51 lease-time infinite
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 12 hostname fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw rpc.mountd[3350210]: authenticated mount request from 192.168.0.110:981 for /nfsroot/fe01 (/nfsroot/fe01)
Feb 16 20:45:07 dsdaqgw rpc.mountd[3350210]: authenticated unmount request from 192.168.0.110:859 for /nfsroot/fe01/tmp/autoDY4k5u (/nfsroot/fe01)
</pre>
* tail /var/log/mini_httpd.log
<pre>
192.168.0.110 - - [16/Feb/2024:20:43:15 -0800] "GET /uefi/vmlinuz-6.5.0-17-generic HTTP/1.0" 200 14227944 "" "Syslinux/6.04"
192.168.0.110 - - [16/Feb/2024:20:43:24 -0800] "GET /uefi/initrd.img-6.5.0-17-generic HTTP/1.0" 200 137824833 "" "Syslinux/6.04"
</pre>

=== setup efi http boot ===

https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-deployment-prep-uefi-httpboot.html

=== setup linux kernel ===

* copy the kernel files
<pre>
cd /boot
rsync -av config* initrd* System.map* vmlinuz* /tftpboot/
</pre>
* cd /tftpboot
* chmod a+r *

=== setup nfs ===

* apt-get install nfs-kernel-server
* enable NFS over UDP, edit /etc/nfs.conf add "udp=y":
<pre>
udp=y
</pre>
<pre>
systemctl restart nfs-server.service
</pre>
* emacs -nw /etc/exports
<pre>
/nfsroot/dsfe01 dsfe01(rw,no_root_squash,async,no_subtree_check)
</pre>
* enable services
<pre>
systemctl enable nfs-server
systemctl enable nfs-mountd
systemctl enable nfs-idmapd
systemctl restart nfs-server
systemctl restart nfs-mountd
systemctl restart nfs-idmapd
</pre>
* after editing /etc/exports, run
<pre>
exportfs -av
</pre>

=== setup userland ===

!!! ubuntu-18 version !!!

* zfs create rpool/nfsroot
* zfs set dedup=verify rpool/nfsroot ### enable deduplication to save disk space because most linux images have mostly identical files
* clone ubuntu
<pre>
mkdir /nfsroot/dsfe01
cd /
rsync -avx . /nfsroot/dsfe01
</pre>
* edit config files:
* cd /nfsroot/dsfe01
* emacs -nw etc/hostname ### change to dsfe01
* emacs -nw etc/mailname ### change to dsfe01
* emacs -nw etc/yp.conf ### change daq00.triumf.ca to musr00.triumf.ca
* emacs -nw etc/defaultdomain ### change to MUSR-NIS
* cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
* emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
* emacs -nw root/.ssh/authorized_keys ### update root ssh keys
* emacs -nw etc/fstab ### add this
<pre>
192.168.1.1:/nfsroot/dsfe01 / nfs defaults,nolock 0 0
</pre>
* emacs -nw etc/chrony/chrony.conf
** comment-out all "pool" and "server" entries
** add entry "server 192.168.1.1 iburst"

After dsfe01 is booted:

* disable services:
<pre>
systemctl disable apache2
systemctl disable dnsmasq
systemctl disable zfs-import-cache
</pre>

To setup additional machines, clone dsfe01 instead of cloning the gateway machine

=== Allow manpages to be viewed ===

If <code>/</code> is mounted over NFS, <code>man</code> will report a permission error. Fix it with:

<pre>
ln -s /etc/apparmor.d/usr.bin.man /etc/apparmor.d/disable/
apparmor_parser -R /etc/apparmor.d/usr.bin.man
</pre>

== setup shared home directory ==

=== on the gateway machine ===
* define netgroups
* emacs -nw /etc/netgroup
<pre>
dsfe (dsfe01,,) (dsfe02,,)
</pre>
* emacs -nw /etc/nsswitch.conf ### edit the netgroup line to read:
<pre>
netgroup: files
</pre>
* export the home directories:
* emacs -nw /etc/exports ### add this:
<pre>
/zssd/home1 @dsfe(rw,no_root_squash,async,no_subtree_check)
</pre>
* exportfs -rc

=== on the frontend machine ===

* mkdir /home
* emacs -nw /etc/fstab ### add this:
<pre>
192.168.1.1:/zssd/home1 /home nfs defaults 0 0
</pre>
* mount -a

== setup NAT ==

NAT allows machines on the private network to connect to the internet: https://en.wikipedia.org/wiki/Network_address_translation

In these examples:
* replace "eno1" with name of the outgoing interface (the one connected to the TRIUMF network).
* replace "enp11s0" with name of the private network interface (192.168.1.x network)

* emacs -nw /etc/rc.local ### add this:
<pre>
# /etc/rc.local

# enable NAT

/sbin/iptables -t nat -A POSTROUTING -o eno1 -j MASQUERADE
iptables -L -v

# uncomment following lines if machine has prohibitive FORWARD rules:
#/sbin/iptables -I FORWARD -i eno1 -o enp11s0 -m state --state RELATED,ESTABLISHED -j ACCEPT
#/sbin/iptables -I FORWARD -i enp11s0 -o eno1 -j ACCEPT
#iptables -L -v

iptables -L -v
sysctl -w net.ipv4.ip_forward=1
#sysctl -a | grep forward

sh /etc/firewall-rfc1918.sh

# end
</pre>
* emacs -nw /etc/firewall-rfc1918.sh
<pre>
# firewall-rfc1918.sh

# prevent RFC1918 private network IP addresses from
# going in and out from our uplink.

ETH=eno1

iptables -F in-rfc1918
iptables -N in-rfc1918
iptables -A in-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A in-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A in-rfc1918 --dst 192.168.0.0/16 -j REJECT

iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -I INPUT -j in-rfc1918 -i $ETH

iptables -F out-rfc1918
iptables -N out-rfc1918
iptables -A out-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A out-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A out-rfc1918 --dst 192.168.0.0/16 -j REJECT

iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -I OUTPUT -j out-rfc1918 -o $ETH

iptables -D FORWARD -j out-rfc1918 -o $ETH
iptables -D FORWARD -j out-rfc1918 -o $ETH
iptables -I FORWARD -j out-rfc1918 -o $ETH

# allow TRIUMF-SECURE network

iptables -I in-rfc1918 -s 10.90.0.0/255.255.0.0 -j ACCEPT
iptables -I out-rfc1918 -d 10.90.0.0/255.255.0.0 -j ACCEPT

# show configuration

iptables -L -v

#end
</pre>

= KVM =

<pre>
apt install cpu-checker

root@daq13:~# kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
root@daq13:~#

(if not, shutdown, go into BIOS settings, enable CPU virtualization)

apt install virtinst ### will install many packages
apt install libvirt-clients libvirt-daemon-system-systemd libvirt-daemon qemu qemu-kvm libvirt-daemon-system virtinst bridge-utils

root@daq13:/home1/wheel# virsh list --all
Id Name State
------------------------------
1 ubuntu-guest running

apt install virt-manager

virt-install --name ubuntu-guest --os-variant ubuntu20.04 --vcpus 2 --ram 2048 --location /daq/daqstore/olchansk/linux/Ubuntu/ubuntu-20.04.3-desktop-amd64.iso --network bridge=virbr0,model=virtio --graphics none --extra-args='console=ttyS0,115200n8 serial'

virtual machine will start, boot, etc
to get out of it, CTRL + Shift followed by ]

ssh wheel@daq13
virt-manager

run virt-install again, omit "--graphics none", open graphics console from virt-manager, it booted into ubuntu installer desktop

virt-install --name test10 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --filesystem /kvm_ladd00,/ --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial" --graphics none

virt-install --name test14 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --disk /tmp/xxx/ladd00.img,bus=sata --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial rdshell" --graphics none --check path_in_use=off
</pre>

build image

<pre>
dd if=/dev/zero of=/tmp/xxx/ladd00.img bs=1024M count=20
mkfs.ext3 /tmp/xxx/ladd00.img ### ext4 fails to mount by SL6 kernel, "unknown ext4 options"
cd /kvm_ladd00/
mount -o loop /tmp/xxx/ladd00.img /mnt/tmp
rsync -av . /mnt/tmp/ --delete
umount /mnt/tmp
</pre>

on the guest, configure network: /etc/rc.local
<pre>
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

ifconfig eth2 192.168.122.2
route add -net 0.0.0.0 gw 192.168.122.1
ifconfig -a
netstat -rn

# end
</pre>

== virsh commands ==

<pre>
virsh list --all

virsh start kvm-el7
virsh console kvm-el7
virsh destroy kvm-el7

virsh install ...
virsh undefine kvm-el7

virsh autostart kvm-ladd00
virsh dominfo kvm-ladd00
</pre>

== virtualize SL6 ladd00 ==

* on ladd00:
<pre>
yum install dracut-network
mkinitrd /boot/initramfs-2.6.32-754.35.1.el6.x86_64-netboot.img 2.6.32-754.35.1.el6.x86_64
</pre>
* on daq00
<pre>
zfs create rpool/kvm-ladd00
cd /kvm-ladd00
rsync -avx ladd00:/ . --exclude nfsroot
brctl addbr virbr0
ifconfig virbr0 192.168.1.1
echo /kvm-ladd00 192.168.1.2(rw,no_root_squash,no_all_squash,async,no_subtree_check) >> /etc/exports
exportfs -rv
</pre>
* create virtual machine
<pre>
virt-install --name kvm-ladd00 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --network bridge=virbr0,model=virtio --boot kernel=/kvm-ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm-ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64-netboot.img,kernel_args="root=/dev/nfs ip=192.168.1.2:192.168.1.1:192.168.1.1:255.255.255.0:ladd00::off nfsroot=192.168.1.1:/kvm-ladd00,vers=3,tcp console=ttyS0,115200n8 serial rdshell" --graphics none --nodisks --check path_in_use=off
</pre>
* adjust kvm-ladd00 image
<pre>
disable network manager
edit fstab
edit yp.conf
edit resolv.conf
edit root/.ssh/authorized_keys
enable rngd or /dev/random does not work, sshd does not work
</pre>
* virsh shutdown test24
* virsh --connect qemu:///system start test24
* virsh console test24 ### to exit, ctrl+[ or ctrl+]
* virsh undefine test24
* virsh autostart kvm-ladd00
* virsh dominfo kvm-ladd00
<pre>
root@daq00:~# virsh dominfo kvm-ladd00
Id: 1
Name: kvm-ladd00
UUID: 1d1f8fed-8b65-4411-a51b-e0ecf359d2f1
OS Type: hvm
State: running
CPU(s): 2
CPU time: 27.7s
Max memory: 2097152 KiB
Used memory: 2097152 KiB
Persistent: yes
Autostart: enable
Managed save: no
Security model: apparmor
Security DOI: 0
Security label: libvirt-1d1f8fed-8b65-4411-a51b-e0ecf359d2f1 (enforcing)
root@daq00:~#
</pre>
* delete unused images in /var/lib/libvirt/images
* virsh edit kvm-ladd00 # change boot command line, etc

== virtualize CentOS-7 daqstore ==

* similar to ladd00 above:
* on daqstore, install dracut-network, already there
<pre>
yum install dracut-network
yum install qemu-kvm libvirt libvirt-python libguestfs-tools virt-install
# yum install busybox ### no rpm package?!?
</pre>
<pre>
dracut -a nfs -v /boot/initramfs-3.10.0-1160.119.1.el7.x86_64-virt.img 3.10.0-1160.119.1.el7.x86_64 --force
</pre>
<pre>
scp daqstore:/boot/initramfs-3.10.0-1160.119.1.el7.x86_64-virt.img /kvm-el7/boot/
</pre>
* on daq00
<pre>
zfs create rpool/kvm-el7
cd /kvm-el7
rsync -avx daqstore:/ .
echo 192.168.1.3 kvm-el7 >> /etc/hosts
systemctl restart dnsmasq
echo /kvm-el 192.168.1.3(rw,no_root_squash,no_all_squash,async,no_subtree_check) >> /etc/exports
exportfs -rv
</pre>
* manage virtual machine
<pre>
virsh console kvm-el7
virsh destroy kvm-el7
virsh undefine kvm-el7
</pre>
* create virtual machine
<pre>
virt-install --name kvm-el7 --os-variant centos7 --vcpus 2 --ram 2048 --import --network bridge=virbr0,model=e1000e --boot kernel=/kvm-el7/boot/vmlinuz-3.10.0-1160.119.1.el7.x86_64,initrd=/kvm-el7/boot/initramfs-3.10.0-1160.119.1.el7.x86_64-virt.img,kernel_args="root=/dev/nfs ip=192.168.1.3:192.168.1.1:192.168.1.1:255.255.255.0:kvm-el7::off nfsroot=192.168.1.1:/kvm-el7,vers=3,tcp rw console=ttyS0,115200n8 serial rdshell" --graphics none --nodisks --check path_in_use=off
</pre>
* adjust kvm-el7 image
<pre>
disable network manager
edit fstab
edit hostname
disable selinux in /etc/sysconfig/selinux

UP TO HERE --- DNS does not work!!!

edit yp.conf
edit resolv.conf
edit root/.ssh/authorized_keys
enable rngd or /dev/random does not work, sshd does not work
</pre>
* virsh shutdown test24
* virsh --connect qemu:///system start test24
* virsh console test24 ### to exit, ctrl+[ or ctrl+]
* virsh undefine test24
* virsh autostart kvm-ladd00
* virsh dominfo kvm-ladd00
<pre>
root@daq00:~# virsh dominfo kvm-ladd00
Id: 1
Name: kvm-ladd00
UUID: 1d1f8fed-8b65-4411-a51b-e0ecf359d2f1
OS Type: hvm
State: running
CPU(s): 2
CPU time: 27.7s
Max memory: 2097152 KiB
Used memory: 2097152 KiB
Persistent: yes
Autostart: enable
Managed save: no
Security model: apparmor
Security DOI: 0
Security label: libvirt-1d1f8fed-8b65-4411-a51b-e0ecf359d2f1 (enforcing)
root@daq00:~#
</pre>
* delete unused images in /var/lib/libvirt/images
* virsh edit kvm-ladd00 # change boot command line, etc

= ARM64 cross-compiler =

* arm64, aarch64 are Xilinx FPGA Cortex-A53, RPi4, RPi5 machines
* install packages:
<pre>
apt install g++-12-aarch64-linux-gnu gcc-12-aarch64-linux-gnu-base libstdc++-12-dev-arm64-cross
</pre>
* run:
<pre>
aarch64-linux-gnu-gcc-12 -o ttcp.aarch64 ttcp.c -static
aarch64-linux-gnu-g++-12 -o fecdm.exe -O2 -g -Wall -Wuninitialized -std=c++20 fecdm.o dsdm.o /home/dsdaqdev/packages_common/midas/linux-aarch64-remoteonly/lib/libmidas.a -pthread -lrt -lutil /nfsroot/gdm00/usr/lib/aarch64-linux-gnu/libi2c.a -static
</pre>

= ARM cross-compiler =

NOTE: updated for U-24

* armv7 (Cyclone-V SoC, RPi3, MityARM CAMAC) machines (Debian-12 armhf target, Ubuntu 24.04 host)
* install packages:
<pre>
apt install g++-arm-linux-gnueabihf gcc-arm-linux-gnueabihf libc6-dev-armhf-cross
</pre>
* build MIDAS frontend (static linking)
<pre>
arm-linux-gnueabihf-g++ -std=c++11 -Wall -Wuninitialized -g -O2 -I/home/dldaq/packages/midas/include -I/home/dldaq/packages/midas/mvodb -c koi2c.cxx
arm-linux-gnueabihf-g++ -o fedldb.exe -std=c++11 -Wall -Wuninitialized -g -O2 -I/home/dldaq/packages/midas/include -I/home/dldaq/packages/midas/mvodb fedldb.o koi2c.o /home/dldaq/packages/midas/linux-armv7-remoteonly/lib/libmidas.a -L/usr/arm-linux-gnueabihf/lib -L/nfsroot/dltdc/usr/lib/arm-linux-gnueabihf -static -lm -lz -lutil -lnsl -lpthread -lrt -li2c
/usr/lib/gcc-cross/arm-linux-gnueabihf/13/../../../../arm-linux-gnueabihf/bin/ld: /home/dldaq/packages/midas/linux-armv7-remoteonly/lib/libmidas.a(system.o): in function `ss_socket_connect_tcp(char const*, int, int*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)':
/home/dldaq/packages/midas/src/system.cxx:4984:(.text+0x252a): warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
</pre>

= 32-bit intel cross-compiler =

Ubuntu 22.04:

<pre>
apt install libstdc++-11-dev:i386
apt install zlib1g-dev:i386
</pre>

NOTES:
* "g++ -m32" does not find libstdc++, please use "g++ -m32 -L/usr/lib/gcc/i686-linux-gnu/11/"
* to cross-build 32-bit MIDAS, use "make linux32".
* executables cross-build on Ubuntu-22 do NOT run on 32-bit Debain-11 (GLIBC and GLIBCXX version mismatch)
* executables cross-build on Ubuntu-22 run on 32-bit Debian-12.

= SSH settings for EPICS =

* TRIUMF EPICS runs obsolete version of SSH
* add this to the use .ssh/config
<pre>
Host sbp1*
HostKeyAlgorithms +ssh-rsa
PubKeyAcceptedAlgorithms +ssh-rsa
KexAlgorithms +diffie-hellman-group1-sha1
ForwardX11 yes
ForwardX11Trusted yes
</pre>

= changes for VME processors =

<pre>
apt -y remove sysstat man-db
apt -y purge dkms
apt -y purge mdadm
apt -y purge fwupd
apt -y purge packagekit
apt -y purge accountsservice
apt -y purge plocate
apt -y purge upower power-profiles-daemon
apt -y autoremove
</pre>

for D-12 32-bit CPUs:

<pre>
apt remove "*libavahi*"
</pre>

= remove snap (U-24) =

Note: snap stores data in $USER/snap/$SNAPNAME, removing a snap on one machine will remove this data from all users even if they want to use snap on some other machine.

Prepare:

NOTE: first remove chromium and firefox, see below.

NOTE: if possible, stop autofs before removing snap - otherwise it will mount all user home directories and complain that it cannot remove some snap data from them

<pre>
systemctl stop autofs
ls -ld /home1/*/snap/* ### remove the per-user snap directories
</pre>

Remove snaps:

<pre>
snap list
echo snap remove --purge chromium ### see below
echo snap remove --purge firefox ### see below
snap remove thunderbird
snap remove cups
snap remove hello-world
snap remove firmware-updater
snap remove gtk-common-themes
snap remove snapd-desktop-integration
snap remove snap-store
snap remove hunspell-dictionaries-1-7-2004
snap remove gnome-system-monitor
snap remove gnome-3-26-1604
snap remove gnome-3-28-1804
snap remove gnome-3-34-1804
snap remove gnome-3-38-2004
snap remove gnome-42-2204
snap remove gnome-46-2404
snap remove mesa-2404
snap remove core
snap remove core18
snap remove core20
snap remove core22
snap remove core24
snap remove bare
snap remove snapd
snap list
</pre>
<pre>
root@daqubuntu:~# snap list
No snaps are installed yet. Try 'snap install hello-world'.
root@daqubuntu:~#
</pre>

Identify packages that install snaps:

<pre>
apt list | grep snap | grep installed | grep -v -e snappy -e snapshot
</pre>

Typical output:

<pre>
firefox/noble,now 1:1snap1-0ubuntu5 amd64 [installed]
gir1.2-snapd-2/noble,now 1.64-0ubuntu5 amd64 [installed,automatic]
libsnapd-glib-2-1/noble,now 1.64-0ubuntu5 amd64 [installed,automatic]
libsnapd-qt-2-1/noble,now 1.64-0ubuntu5 amd64 [installed,automatic]
plasma-discover-backend-snap/noble,now 5.27.11-0ubuntu2 amd64 [installed]
snapd/noble-updates,now 2.66.1+24.04 amd64 [installed]
</pre>

Remove packages that install snaps:

<pre>
apt remove chromium-browser
apt remove chromium-codecs-ffmpeg-extra
apt remove thunderbird
apt remove firefox
apt remove plasma-discover-backend-snap
apt remove plasma-discover-snap-backend
apt remove snapd
apt purge snapd
# package gir1.2-snapd-2 is required by ubuntu-mate-desktop & co
# libsnapd-glib-2-1 is required by gstreamer, gnome-remote-desktop & co
ls -l /etc/systemd/system/ | grep snap ### remove unwanted stuff
</pre>

Remove Chromium:

* ls -ld /home/*/snap/chromium/*
* echo /bin/rm -rf `ls -1d /home/*/snap/chromium/*`
* snap remove chromium

Remove Firefox:

* ls -ld /home/*/snap/firefox/*
* echo /bin/rm -rf `ls -1d /home/*/snap/firefox/*` ### this will delete "snap firefox" profiles of all users!!!
* snap remove firefox ### this will also delete "snap firefox" profiles of all users!!!

Remove gir1.2-snapd-2:
* echo rm -vf /usr/lib/x86_64-linux-gnu/girepository-1.0/Snapd-2.typelib

If "snap remove" is stuck in "change in progress" (this will remove all snaps and break snapd, which is ok,
see https://forum.snapcraft.io/t/snap-remove-taking-forever-abort-wasnt-working/48915)
<pre>
rm /var/lib/snapd/state.json
systemctl restart snapd
</pre>

Prevent snap from reinstalling:
<pre>
cd ~/git/scripts/etc
git pull
cp etc-apt-preferencesd-disable-snap /etc/apt/preferences.d/
</pre>

= install non-snap thunderbird =

from: https://ubuntuhandbook.org/index.php/2024/03/install-thunderbird-deb-ubuntu-2404/

* remove snap thunderbird
<pre>
snap remove --purge thunderbird
apt remove --purge thunderbird
</pre>
* add mozilla repository
<pre>
already done after installing firefox-esr
</pre>
* ppa deb and ubuntu snap thunderbird package names are the same, change priority and hide snap package: create /etc/apt/preferences.d/mozillateamppa
<pre>
Package: thunderbird*
Pin: release o=LP-PPA-mozillateam
Pin-Priority: 1001

Package: thunderbird*
Pin: release o=Ubuntu
Pin-Priority: -1
</pre>
* check that it worked, it should say "build" instead of "snap"
<pre>
apt update
apt list | grep thunderbird | grep -v locale
...
thunderbird/noble 1:128.8.1+build1-0ubuntu0.24.04.1~mt1 amd64
...
</pre>
* install
<pre>
apt install thunderbird
</pre>
* run: thunderbird

= EFI boot using syslinux =

* rationale 1: GRUB is the stock boot loader with U-24. It is unnecessarily complicated. EFI BIOS can boot the linux kernel directly, without GRUB, but unfortunately a small shim bootloader is required to specify the initrd file and the root filesystem. the syslinux boot loader can do this in a very simple way.
* rationale 2: GRUB bootloader configuration is overcomplicated, and when it breaks, it is almost impossible to debug and to recover.
* rationale 3: GRUB bootloader scripts in U-24 have no support for booting from redundant SSDs.
* in the case of GRUB bootloader failure, it is simplest to boot the Ubuntu installer in recovery mode and convert the bootloader from GRUB to syslinux. Open firefox on this page and cut-and-paste the steps (only copy of vmlinux and initrd cannot copy-and-paste as of this writing).
* in the case of servers with redundant SSDs for OS and home directories (ZFS mirror), it is simplest to use the syslinux bootloader to ensure that the machine boots from either SSD (all combinations SSD failures, fail of either EFI partiion, fail of either ZFS mirror partition, machine should boot)
* check partition tables, SATA SSD
<pre>
fdisk -l /dev/sda
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: A3F34DAC-DCB4-B74C-B59E-41E754807812

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root
</pre>
* check partition tables, NVME SSD
<pre>
fdisk -l /dev/nvme0n1
Disk /dev/nvme0n1: 1.75 TiB, 1920383410176 bytes, 3750748848 sectors
Disk model: SAMSUNG MZ1L21T9HCLS-00A07
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 131072 bytes / 131072 bytes
Disklabel type: gpt
Disk identifier: 04ECCD46-DC2A-454C-B4A8-CCC18AA532F7

Device Start End Sectors Size Type
/dev/nvme0n1p1 2048 2203647 2201600 1G EFI System
/dev/nvme0n1p2 2203648 6397951 4194304 2G Linux filesystem
/dev/nvme0n1p3 6397952 23175167 16777216 8G Linux swap
/dev/nvme0n1p4 23175168 3750746111 3727570944 1.7T Linux filesystem
</pre>
* prepare boot device EFI partition, SATA SSDs
<pre>
mkfs.msdos /dev/sda1
mkfs.msdos /dev/sdb1
mkdir /boot/efi-sda
mkdir /boot/efi-sdb
mount /dev/sda1 /boot/efi-sda
mount /dev/sdb1 /boot/efi-sdb
</pre>
* prepare boot device EFI partition, NVME SSDs
<pre>
mkfs.msdos /dev/nvme0n1p1
mkfs.msdos /dev/nvme1n1p1
mkdir /boot/efi-0
mkdir /boot/efi-1
mount /dev/nvme0n1p1 /boot/efi-0
mount /dev/nvme1n1p1 /boot/efi-1
</pre>
* add them to fstab, note the "nofail" mount option
<pre>
blkid | grep vfat
/dev/sdb1: UUID="F30C-13B5" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="20e423b5-ac29-ec42-bab5-f366aefbbd2b"
/dev/sda1: UUID="F2DD-7321" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="9427646c-ce5f-fe47-9ed1-4b84cf4c348f"
grep ^UUID /etc/fstab
UUID=F2DD-7321 /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
UUID=F30C-13B5 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022.nofail 0 1
</pre>
* prepare the EFI partitions (remove old previous subdirectories, only empty efi/boot should be there)
<pre>
cd /boot/efi-sda
mkdir -p efi/boot
</pre>
* get syslinux-6.03
<pre>
cd ~
wget https://daq00.triumf.ca/~olchansk/linux/syslinux-6.03.tar.xz
xz -d < syslinux-6.03.tar.xz | tar xvf -
</pre>
* from syslinux-6.03 copy files:
<pre>
cd /boot/efi-sda/efi/boot
cp ~/syslinux-6.03/efi64/efi/syslinux.efi .
cp ~/syslinux-6.03/efi64/com32/elflink/ldlinux/ldlinux.e64 .
cp syslinux.efi bootx64.efi
</pre>
* identify the ZFS rpool label
<pre>
zfs list | grep ROOT | grep "/$" | cut -f1 -d" "
rpool/ROOT/ubuntu_9yvb17
</pre>
* create syslinux.cfg, change the root=ZFS label to match this computer
<pre>
cat << EOF | sed "s;root=.*$;root=ZFS=`zfs list | grep ROOT | grep "/$" | cut -f1 -d" "`;" > syslinux.cfg
default linux
label linux
kernel vmlinuz
append ro initrd=initrd.img root=ZFS=rpool/ROOT/ubuntu_02ruwj
EOF
</pre>
* copy linux boot files:
<pre>
cp /boot/vmlinuz vmlinuz
cp /boot/initrd.img initrd.img
</pre>
* repeat with /boot/efi-sdb, etc
* install script to set syslinux to boot the latest kernel
<pre>
cd ~/git/scripts
git pull
ln -s ~/git/scripts/etc/update_efi_syslinux.perl ~
</pre>
* update syslinux to boot the latest kernel
** run "~/update_efi_syslinux.perl" to check that it finds the EFI partitions and finds the correct kernel
** run "~/update_efi_syslinux.perl -u" to do the actual update
* or maybe install Ubuntu syslinux 6.04 and use files from there:
<pre>
apt install "syslinux*"
</pre>

= legacy boot using syslinux =

* NOTE: extlinux is not compatible with ext4 "64bit" feature, it should be turned off:
<pre>
mkfs.ext4 -O ^64bit /dev/sdX1
resize2fs -s /dev/sdX1
</pre>

* install syslinux and extlinux (THIS DOES NOT WORK!!!)

<pre>
apt -y install syslinux extlinux
dd if=/usr/lib/syslinux/mbr/mbr.bin of=/dev/sdX ### NOT /dev/sdX1 NOT !!!
cd /boot
cp /usr/lib/syslinux/modules/bios/menu.c32 .
extlinux -i .
</pre>

* install syslinux and extlinux

* copy from old SL6 USB disk (this is extlinux 6.02)
<pre>
root@localhost:/boot# ls -l
-rwxr-xr-x 1 root root 218952 Jan 28 17:40 extlinux
-rw-r--r-- 1 root root 402 Jan 29 14:45 extlinux.conf
-rw-r--r-- 1 root root 496 Jan 29 14:39 extlinux.conf~
-r--r--r-- 1 root root 122044 Jan 29 14:39 ldlinux.c32
-r--r--r-- 1 root root 67072 Jan 29 14:39 ldlinux.sys
-rwxr-xr-x 1 root root 24156 Jan 28 17:40 libutil.c32
-rw-r--r-- 1 root root 304 Jan 28 17:40 mbr.bin
-rw-r--r-- 1 root root 26140 Jan 28 17:40 memdisk
-rw-r--r-- 1 root root 69043 Jan 28 17:40 memtest86+-4.20.iso.zip
-rw-r--r-- 1 root root 183012 Jan 28 17:40 memtest86+-5.01
-rw-r--r-- 1 root root 26568 Jan 28 17:40 menu.c32
</pre>

* install
<pre>
dd if=mbr.bin of=/dev/sdX ### NOT /dev/sdX1 NOT !!!
extlinux -i .
</pre>
* check that partition /dev/sdX1 is marked bootable (fdisk command "a")

* create /boot/extlinux.conf

<pre>
DEFAULT menu.c32
PROMPT 0
TIMEOUT 50

MENU TITLE TRIUMF DAQ USB BOOT32 ver K.O. 2025jan28

LABEL linux
MENU DEFAULT
kernel /vmlinuz
append initrd=/initrd.img panic=60 rootdelay=5 rootwait rw root=/dev/sda1

LABEL linux-6.1.0-28-686
kernel vmlinuz-6.1.0-28-686
append initrd=initrd.img-6.1.0-28-686 panic=60 rootdelay=5 rootwait rw root=/dev/sda1

LABEL memtest
kernel memtest86+-1.65
</pre>

= Add user to login spash screen =

For user login to local machine, if it doesn't work by default. User should exist already from NIS service.

<pre>
sudo /bin/bash # as root
cd /var/lib/AccountsService/users
ls # should at least display the "wheel" user
cp wheel username # we're going to copy the user account settings over to our new user
</pre>

Ubuntu

2026-03-12T17:47:40Z

Dfujimoto: /* install ganglia server */

= Prerequisites =

* before setting up new machine run memory test
* prepare flash drive with free version of memtest86: https://www.memtest86.com
* test boot from flash drive, test takes ~ few hours
* test will end with summary page, if passed continue with Ubuntu
* number that might be worth noting is memory latency

= Ubuntu version =

<pre>
lsb_release -a
uname -a
</pre>

= Ubuntu installer =

* updated for Ububtu LTS 20.04.01, 22.04.1, 24.04 (only minor differences)

* download the latest Ubuntu LTS desktop installer iso image
* dd the image to a USB key
* power down, disconnect all disks (all HDDs, all SSDs, all M.2)
* connect the SSD to be used as system disk
* if system will use mirrored SSDs (using ZFS mirror), leave second SSD disconnected, we will activate it later
* power up
* boot from USB key in legacy mode or UEFI mode (select this in the BIOS boot menu - F2 or F8 for ASUS, F11 for Supermicro)
* follow the instruction:
* "try ubuntu or install ubuntu" - choose "install"
* select language - accept default
* "updates and other software" - accept default settings ("normal install")
* "installation type" - select "advanced features" and "experimental: use ZFS"
* accept partition choice
* "where are you?" - select "Vancouver" (PST time zone)
* "who are you?" - leave all fields blank, except "username" set to "wheel", "password" set to the root password. hostname will be set later after configuring the network
* don't install third party sw
* installation runs in a few minutes, when finished, reboot
* login as user wheel
* answer annouying questions:
* "livepatch" - say "next"
* "help improve" - select "do not send", say "next"
* "privacy" - leave "location" as "off", say "next"
* "ready to go", say "done"
* right-click on the desktop, say "open in terminal", a shell will open
* say "sudo /bin/bash", enter the root password, you now have the root shell
* run nm-connection-editor to configure the network. use netmask 255.255.224.0, gateway 142.90.100.18, DNS 142.90.100.19, search path "triumf.ca"
* after network is up (can ping ladd00), continue with post-installation steps below

= Install instructions =

== prepare ==
<pre>
apt update
apt upgrade
</pre>

== install ssh ==
<pre>
apt install ssh
</pre>

== install git/scripts ==
<pre>
apt -y install git
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd scripts
git pull
</pre>

* if needed, update git/scripts repository from ladd00 to daq00:
* git remote -v ### if it says daq00, we are good
* git remote set-url origin https://daq00.triumf.ca/~olchansk/git/scripts.git
* git pull ### check that it works

== configure hostname ==
<pre>
vi /etc/hostname
</pre>

== disable swap ==

ubuntu installer creates a 2 GB swap partition, not useful
on 32-64 GB machine, disable it:
<pre>
vi /etc/fstab ### comment out the "swap" line
</pre>

* on 64 GB RAM machines swap is not useful
* on machines booted from network (NFS-ROOT), swap does not work
* on machines running from flash (RPi, etc), flash is too slow for useful swap
* swap configured by linux installers invariably has wrong size and is not useful

<pre>
systemctl disable dphys-swapfile
systemctl stop dphys-swapfile
dphys-swapfile uninstall
</pre>

== maybe reboot ==

this is a good point to reboot the machine to boot
the latest kernel and to set the correct hostname

== install etckeeper ==

keep contents of /etc in a git repository:

<pre>
apt -y install etckeeper
</pre>

== set timezone ==
<pre>
timedatectl list-timezones | grep -i vancouver
timedatectl set-timezone America/Vancouver
</pre>

== install time synchronization ==

check if chrony is installed correctly and is synched to TRIUMF time servers:

<pre>
chronyc sources
chronyc tracking
</pre>

if not, remove old chrony and

<pre>
apt -y remove chrony
apt -y purge chrony
</pre>

and install it from scratch:

<pre>
apt -y install chrony
cd ~/git/scripts
git pull
cd ~
cp ~/git/scripts/etc/triumf.sources /etc/chrony/sources.d/
systemctl disable systemd-timesyncd.service
systemctl stop systemd-timesyncd.service
systemctl disable ntp
systemctl stop ntp
systemctl enable chrony
systemctl restart chrony
chronyc sources
chronyc tracking
</pre>

== reenable systemd-timesyncd ==

ONLY IF CHRONY DOES NOT WORK

To configure systemd-timesyncd, set "NTP=" in /etc/systemd/timesyncd.conf

<pre>
apt remove chrony
cat /etc/systemd/timesyncd.conf
systemctl enable systemd-timesyncd.service
systemctl restart systemd-timesyncd.service
systemctl status systemd-timesyncd.service
timedatectl status
timedatectl timesync-status
</pre>

== enable outgoing email (debian 11) ==

this is different from ubuntu 20. it uses /etc/mailname and it hardwires the hostname into main.cf.

== enable outgoing email ==

we have an unusual email configuration. outgoing email should work to deliver error messages, notices, etc. incoming email is disabled, we do not receive email for local users.

this causes problems with TRIUMF smtp server. if our message cannot be delivered (wrong email address or receipient computer is turned off), TRIUMF smtp server will generate a delivery failure notification email and try to send it to the "from" address of the failed message. but the "from" address does not receive any email, so another delivery failure notification email is generated and an attempt to deliver it. which again fails, rinse and repeat.

as solution, kray created a special rule, email from scrap.triumf.ca does not generate delivery failure notices. failed messages sit in the queue for 5 days, then they are deleted. (K.O. - confirmed with kray 3jan2024).

to make this work we use the msmtp MTA package.

<pre>
cd ~
apt -y install mailutils msmtp msmtp-mta # say "no" to apparmor support
apt -y remove postfix
apt -y purge postfix # remove old config files
apt -y install bsd-mailx
cd ~/git/scripts/etc
git pull
/bin/cp -fv aliases /etc/aliases
/bin/cp -fv msmtprc /etc/msmtprc
/bin/rm -vf ~root/.forward
/bin/rm -vf /etc/mailname
Mail root
Subject: test
test
^D
CC: <CR>
</pre>

== enable outgoing email (postfix) ==

THIS IS OBSOLETE!!!

* TRIUMF: use smtp.triumf.ca
* CERN: use cernmx.cern.ch

<pre>
apt install postfix ### select "satellite system", enter full hostname "xxx.triumf.ca", enter "smtp.triumf.ca"
apt install mailutils
dpkg-reconfigure postfix ### (if postfix already installed)
</pre>
<pre>
echo olchansk@triumf.ca lindner@triumf.ca bsmith@triumf.ca dfujimoto@triumf.ca >> ~root/.forward
mailx root
test
^D
</pre>

== enable ping for all users (debian 11) ==

Without this tweak, Debian will report "operation not permitted" if a user tries to ping somewhere.

<pre>
echo 'net.ipv4.ping_group_range = 0 1000' > /etc/sysctl.d/99-ping.conf
</pre>

== disable apparmor ==

On NFS-Root network booted machines!

If "man man" returns "permission denied" and syslog reports apparmor "sendmsg DENIED" errors, disable apparmor. This is supposedly fixed in kernel 6.0 and later (to be confirmed), see https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/1784499

Disable apparmor, see https://ubuntu.com/server/docs/security-apparmor

This takes effect after a reboot.

<pre>
systemctl stop apparmor.service
systemctl disable apparmor.service
</pre>

If on boot apparmor appears to still be confining apps (for example, mysql), edit the file /etc/default/grub and change

<pre>
GRUB_CMDLINE_LINUX=""
</pre>

to

<pre>
GRUB_CMDLINE_LINUX="apparmor=0"
</pre>

Run sudo update-grub and reboot.

== install missing packages ==

(apt eats terminal input, even the "yes |" trick does not quite work,
repeat the following commands until they report that everything
is installed)

<pre>
yes | apt -y install ssh tcsh ethtool ncat rsync strace net-tools traceroute time minicom screen git lsof debsums tmux iptables telnet pax rpm mtools at gdisk tcpdump
yes | apt -y install sysstat smartmontools lm-sensors
yes | apt -y install lsb-release
apt -y install vim # in addition to default vim-tiny, requested by IRIS
apt -y install gedit # requested by TACTIC
apt -y install tcl
apt -y install mc # requested by sol
apt -y install pax rpm alien ### package converter tools
apt -y install flex bison
apt -y install neofetch
apt -y install snmp snmp-mibs-downloader
apt -y install git subversion g++ gfortran cmake doxygen
apt -y install curl libcurl4 libcurl4-openssl-dev
### conflits with mysql packages ### apt -y install mariadb-client libmariadb-dev ### mysql client for MIDAS
apt -y install mysql-client libmysqlclient-dev
apt -y install postgresql-common libpq-dev ### postgresql client for MIDAS
yes | apt -y install libz-dev libzstd-dev sqlite3 libsqlite3-dev unixodbc-dev
yes | apt -y install libssl-dev
yes | apt -y install emacs xemacs21 joe
yes | apt -y install gnuplot dos2unix
yes | apt -y install mutt bsd-mailx # email clients
yes | apt -y install liblz4-tool pbzip2 libbz2-dev
yes | apt -y install libc6-dev-i386 # otherwise no /usr/include/sys/types.h
yes | apt -y install libreadline-dev
yes | apt -y install ubuntu-mate-themes
yes | apt -y install libmotif-dev libxmu-dev
yes | apt -y install libusb-dev libusb-1.0-0-dev
yes | apt -y install i2c-tools libi2c-dev libi2c0
yes | apt -y install xfig gsfonts-x11 gsfonts-other # install fonts for xfig
yes | apt -y install libjson-perl
yes | apt -y install libgsl-dev # additional GNU Scientific Library
yes | apt -y install qt5-default # Qt development
yes | apt -y install python3-full python3-dev python3-dbg python3-pip ### for pyROOT
yes | apt -y install imagemagick imagemagick-common ckeditor # for elog
yes | apt -y install libjpeg-dev libjpeg-progs libjpeg-tools
yes | apt -y install linux-tools-common linux-tools-generic # cpupower frequency-info
yes | apt -y install rdesktop remmina remmina-plugin"*" # requested by POL
yes | apt -y install nlohmann-json3-dev # required to build MIDAS with ROOT 6.30 on Ubuntu-22
apt -y install dpkg-dev cmake g++ gcc binutils libx11-dev libxpm-dev libxft-dev libxext-dev python3 libssl-dev libafterimage0 # from https://root.cern/install/dependencies/
apt -y install gfortran libpcre3-dev xlibmesa-glu-dev libglew-dev libftgl-dev libmysqlclient-dev libfftw3-dev libcfitsio-dev graphviz-dev libldap2-dev python3-dev python3-numpy libxml2-dev libkrb5-dev libgsl0-dev qtwebengine5-dev nlohmann-json3-dev libtbb-dev libavahi-compat-libdnssd-dev # from https://root.cern/install/dependencies/
apt -y install libvdt-dev # for ROOT 6.32 on Ubuntu-24
apt -y install autoconf automake gperf # gnu package build tools
apt -y install u-boot-tools # for Xilinx petalinux
#apt -y install linux-headers-generic # to build linux kernel drivers
apt -y install htop
</pre>

Ubuntu LTS 20.04:
<pre>
yes | apt -y install linux-image-generic-hwe-20.04 linux-tools-virtual-hwe-20.04 # enable linux 5.11 series kernel
</pre>

Ubuntu LTS 22.04:
<pre>
apt -y install linux-generic-hwe-22.04 # enable linux 6.8.0 series kernel
</pre>

Ubuntu LTS 24.04:
<pre>
apt -y install linux-generic-hwe-24.04 # enable linux 6.14.0 series kernel
</pre>

== remove snap ==

remove snap at this point.

Do

<pre>
snap list
</pre>

to see a list of installed snaps. Use

<pre>
snap remove <item>
</pre>

to remove each. You will need to remove snapd last.

<pre>
systemctl stop snapd
systemctl disable snapd
apt purge snapd
rm -rf ~/snap
</pre>

== install non-snap firefox ==

delay this until the very end, download of firefox-esr deb package is very slow.

See https://askubuntu.com/questions/1399383/how-to-install-firefox-as-a-traditional-deb-package-without-snap-in-ubuntu-22

<pre>
add-apt-repository ppa:mozillateam/ppa
apt install firefox-esr
echo run firefox-esr
</pre>

Enable automatic updates:

* rerun [[#Enable_automatic_updates]]

== configure DNS ==

<pre>
cd ~/git/scripts
git pull
mkdir /etc/systemd/resolved.conf.d
cp etc/resolved-triumf.conf /etc/systemd/resolved.conf.d/
systemctl restart systemd-resolved
resolvectl
#systemd-analyze cat-config systemd/resolved.conf
</pre>

== install ganglia ==

<pre>
apt -y install ganglia-monitor
cd ~root/git/scripts/ganglia
git pull
make install
./ganglia-all.perl
</pre>

fix gmond start before network is ready:

<pre>
mkdir /etc/systemd/system/ganglia-monitor.service.d
echo -e "[Unit]\nAfter=network-online.target\n" > /etc/systemd/system/ganglia-monitor.service.d/local.conf
systemctl daemon-reload
systemctl cat ganglia-monitor.service
</pre>

== install ganglia server ==

'''On the main computer only! (daq00, dsdaqgw, etc)'''

* zfs create rpool/ganglia
* apt install gmetad php php-xml rrdtool
* mv /etc/ganglia/gmetad.conf /etc/ganglia/gmetad.conf-stock
* create /etc/ganglia/gmetad.conf with the following contents:
<pre>
data_source "my cluster" 15 localhost
RRAs "RRA:AVERAGE:0.5:1:244" "RRA:AVERAGE:0.5:24:244" "RRA:AVERAGE:0.5:168:244" "RRA:AVERAGE:0.5:672:244" "RRA:AVERAGE:0.5:5760:374"
setuid_username "ganglia"
rrd_rootdir "/ganglia/rrds"
case_sensitive_hostnames 0
</pre>
* mkdir /ganglia/rrds
* chown ganglia:ganglia /ganglia/rrds
* systemctl restart gmetad
* create /etc/ganglia/gmond-collect.conf
<pre>
globals {
daemonize = yes
setuid = yes
user = nobody
debug_level = 0
max_udp_msg_len = 1472
mute = no
#deaf = yes
deaf = no
allow_extra_data = yes
host_dmax = 0 /* 86400 */ /*secs. Expires (removes from web interface) hosts in 1 day */
host_tmax = 20 /*secs */
cleanup_threshold = 300 /*secs */
gexec = no
send_metadata_interval = 600 /*secs */
}

/*
* The cluster attributes specified will be used as part of the <CLUSTER>
* tag that will wrap all hosts collected by this instance.
*/
cluster {
name = "DAQ"
owner = "TRIUMF DAQ"
latlong = "unspecified"
url = "https://daq.triumf.ca"
}

/* The host section describes attributes of the host, like the location */
host {
location = "unspecified"
}

udp_send_channel {
host = daq00.triumf.ca
bind_hostname = yes
port = 8649
}

udp_recv_channel {
#mcast_join = 239.2.11.71
port = 8649
#bind = 239.2.11.71
retry_bind = true
# Size of the UDP buffer. If you are handling lots of metrics you really
# should bump it up to e.g. 10MB or even higher.
#buffer = 10485760
buffer = 200000
family = ipv4

acl {
default = "deny"
access {
# vlan1 daq00
ip = 142.90.111.168
mask = 19
action = "allow"
}
access {
# MUSR VLAN
ip = 142.90.154.73
mask = 8
action = "allow"
}
access {
# KVM network
ip = 192.168.1.1
mask = 8
action = "allow"
}
}
}

udp_recv_channel {
#mcast_join = 239.2.11.71
port = 8649
#bind = 239.2.11.71
retry_bind = true
# Size of the UDP buffer. If you are handling lots of metrics you really
# should bump it up to e.g. 10MB or even higher.
#buffer = 10485760
buffer = 200000
family = ipv6

acl {
default = "deny"
access {
# ALPHA network
ip = 2001:1458:202:fd::100:aa
mask = 8
action = "allow"
}
#access {
# # ALPHA-g network
# ip = 192.168.1.1
# mask = 8
# action = "allow"
#}
}
}

tcp_accept_channel {
port = 8649
bind = localhost
# If you want to gzip XML output
gzip_output = no
}
</pre>
* add to /etc/rc.local
<pre>
/usr/sbin/gmond -c /etc/ganglia/gmond-collect.conf &
systemctl restart gmetad &
</pre>
* cd /ganglia
* git clone https://daq00.triumf.ca/~olchansk/git/ganglia-web.git
* cd ganglia-web
* git checkout KO1
* ln -s /ganglia/ganglia-web /var/www/html/ganglia
* make dwoo directories
<pre>
mkdir /var/lib/ganglia-web
chown www-data:www-data /var/lib/ganglia-web
mkdir /var/lib/ganglia-web/dwoo
chown www-data:www-data /var/lib/ganglia-web/dwoo
mkdir /var/lib/ganglia-web/dwoo/compiled
chown www-data:www-data /var/lib/ganglia-web/dwoo/compiled
mkdir /var/lib/ganglia-web/dwoo/cache
chown www-data:www-data /var/lib/ganglia-web/dwoo/cache
</pre>
* open https://alphacpc05.cern.ch/ganglia/

== install gonodeinfo ==

* go to https://bitbucket.org/dd1/gonodeinfo follow instructions:
<pre>
apt -y install golang
mkdir ~/git
cd ~/git
#git clone https://bitbucket.org/dd1/gonodeinfo.git
git clone https://daq00.triumf.ca/~olchansk/git/gonodeinfo.git
cd gonodeinfo
git remote set-url origin https://daq00.triumf.ca/~olchansk/git/gonodeinfo.git
git pull
make
make install # install gonodeinfo agent
cd ~ # this is important
</pre>
* edit /etc/gonodeinfo.conf
* change "Description", "Location", "User" and "Administrator" as appropriate (or delete them)
* change "Servers" to read: Servers: daq00.triumf.ca:8601
* run "gonodeinfo -v"
* if error is "connection refused". go to the nodeinfo server to add this client to the access control list:
* on the gonodeinfo server: run /opt/gonodeinfo/gonodereceive.exe -a daq13
* try gonodeinfo again, there should be no error
* on the gonodeinfo server: run gonodereport, look at the web pages, the new machine should be listed now

== install emailonreboot ==

send an email if computer is rebooted

<pre>
ssh root
mkdir -p ~/git
cd ~/git
git clone https://daq00.triumf.ca/~olchansk/git/rpms.git
cd rpms/emailonreboot
git pull
make
make install
</pre>

== install monitor_nfs ==

monitor NFS mounts and complain about dead, stale and hung mounts

<pre>
ssh root
mkdir -p ~/git
cd ~/git
git clone https://daq00.triumf.ca/~olchansk/git/rpms.git
cd rpms/monitor_nfs
git pull
make
make install
</pre>

== install fonts for EPICS ==

* apt -y install xfonts-100dpi xfonts-75dpi
* restart Xorg (i.e. "killall Xorg", this will log you out from the console)
* xlsfonts | grep -i helvetica ### should show fonts with different sizes, not just size 0 (scalable)

== install libz.so.1 for CentOS compatibility ==

KO - confirm which versions on quartus need this.

<pre>
yes | apt-get -y install zlib1g
yes | apt-get -y install zlib1g:i386 libc6:i386 libgcc1:i386 gcc-6-base:i386
</pre>

== install ld-ldb-x86-64.so.3 for Quartus compatibility ==

Not clear from package this is supposed to come from. Copied from U-20.

Without this, Quartus lmgrd does not run.

<pre>
cd /lib64
ln -s ld-linux-x86-64.so.2 ld-lsb-x86-64.so.2
ln -s ld-linux-x86-64.so.2 ld-lsb-x86-64.so.3
</pre>

should look like this:
<pre>
root@daq13:/lib64# ls -l
total 3
lrwxrwxrwx 1 root root 44 Jan 28 09:07 ld-linux-x86-64.so.2 -> ../lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
lrwxrwxrwx 1 root root 20 Feb 18 16:45 ld-lsb-x86-64.so.2 -> ld-linux-x86-64.so.2
lrwxrwxrwx 1 root root 20 Feb 18 16:45 ld-lsb-x86-64.so.3 -> ld-linux-x86-64.so.2
root@daq13:/lib64#
</pre>

== install libpng12.so.0 for Quartus 13.0sp1 and 13.1.4.182 ==

<pre>
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0.50.0
/bin/cp -pv libpng12.so.0 libpng12.so.0.50.0 /lib/x86_64-linux-gnu/
</pre>

== install packages for Xilinx ==

ubuntu LTS 22.04 vivado 2020.1

<pre>
apt install autoconf libtool
apt install libtinfo5
apt install texinfo
apt install zlib1g:i386
</pre>

== install packages for building ROOT ==

<pre>
apt -y install libx11-dev libxpm-dev libxft-dev libxext-dev libpng-dev libjpeg-dev xlibmesa-glu-dev libxml2-dev libgsl-dev cmake
</pre>

== install wine ==

As far as I know, only needed for BNMR/BNQR

<pre>
apt install wine winetricks
</pre>

== install lightdm ==

unlike the default gdm login manager, lightdm shows the machine hostname and does not require an extra mouse click to swicth from screen saver to login mode.

<pre>
apt -y install lightdm
# select lightdm
</pre>

== install desktop environments ==

note: default display manager and default desktop are deficient, please do not skip this step.

note: if apt asks to choose the display manager, select "lightdm"

note: KO - I recommend the "MATE" desktop.

note: you will have to cut-and-paste this several times because "apt" eats commands, even with "-y" and even piped from "yes".

<pre>
# install MATE desktop
apt -y install ubuntu-mate-core ubuntu-mate-desktop ubuntu-mate-themes
# install Cinnamon desktop
apt -y install cinnamon
# install KDE desktop
apt -y install kde-standard kubuntu-settings-desktop
# install Lxqt desktop
# apt -y install lxqt # conflict over kubuntu-desktop, kubuntu-settings-desktop and desktop-base
# install Xfce4 desktop
apt -y install xfce4
</pre>

== install ROOT ==

Please install ROOT per instructions at https://root.cern.ch.

NOTE1: The ROOT package available from Ubuntu repositories is severely out of date and cannot be used with MIDAS and ROOTANA. ### DO NOT DO THIS! apt-get install root-system

NOTE2: as of 2017-Jan-09, ROOT binary kits for Ubuntu do not work (use GCC 5 instead of GCC6), build from source instead.

== Install x2go ==

<pre>
apt-get update
apt-get install x2goserver x2goserver-xsession
</pre>

== enable root login from ladd00/daq00 ==
<pre>
ssh localhost
CTRL-C
/bin/cp ~root/git/scripts/etc/authorized_keys ~root/.ssh/
</pre>

== disable ssh access from outside of TRIUMF ==

to stop ssh login spam, disable ssh access from outside of TRIUMF. this can be done by requesting a firewall block through the helpdesk or by local firewall rule:

<pre>
echo iptables -I INPUT ! -s 142.90.0.0/255.255.0.0 -p tcp --dport 22 -j REJECT >> /etc/rc.local
/etc/rc.local
</pre>

== install smart-status ==
<pre>
ln -s ~/git/scripts/smart-status/smart-status.perl ~root/
</pre>

== enable boot menu and boot messages ==

This will enable the grub menu (with a 10 sec timeout) and
replace black screen with exciting linux boot messages.

* emacs -nw /etc/default/grub
<pre>
GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=menu
GRUB_TIMEOUT=10
GRUB_RECORDFAIL_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="vga=769 video=640x480"
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX=""
#GRUB_GFXMODE=640x480
</pre>
* update grub config:
<pre>
grub-mkconfig -o /boot/grub/grub.cfg
</pre>

= Enable automatic updates =

<pre>
apt install unattended-upgrades
cd ~/git/scripts
git pull
/bin/cp -v etc/99apt-conf-ko /etc/apt/apt.conf.d/
apt-config dump | grep Unattended
</pre>

Following is obsolete:

* emacs -nw /etc/apt/apt.conf.d/50unattended-upgrades
** uncomment in Allowed-Origins "-security" and "-updates"
** add in Allowed-Origins: "Google LLC:stable";
** uncomment/add: "Unattended-Upgrade::Mail "root";
* emacs -nw /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
</pre>
* test: unattended-upgrade --dry-run -v

NOTE: update-on-shutdown is disabled.

NOTE: there is no update-on-boot, but:

NOTE: if machine was off for a long time, the systemd update timer would have expired and it will fire soon after reboot, causing an automatic update run. this is unwanted, and there is no fix or workaround for it. K.O. June-2023.

= Fix bpool is full (obsolete) =

THIS IS CAUSED BY OBSOLETE PACKAGE zsys. PLEASE: apt remove zsys

= IPMI instructions =

IPMI is the board management hardware on Supermicro and other server motherboards. This includes hardware sensors - fan rotation speed, temperatures and power supply voltages.

<pre>
apt-get install ipmitool
systemctl enable ipmievd
systemctl restart ipmievd
</pre>

Run:
* ipmitool sel list ### event list
* ipmitool sel elist ### event list
* ipmitool sel clear ### clear event list (if it becomes full)
* ipmitool sensor ### report hardware sensors

= move /home/wheel (U-24) =

Ubuntu LTS 24 installed on ZFS has rpool/USERDATA/home_xxx mounted on /home,
has to be moved or autofs /home will not work.

<pre>
zfs list | grep USERDATA | grep home | cut -f1 -d" "
zfs set -u mountpoint=/home1 `zfs list | grep USERDATA | grep home | cut -f1 -d" "`
emacs -nw /etc/passwd # change mount for wheel account to /home1/wheel
</pre>

This will not take effect until rebooting. Please ensure that ssh from root@daq00 to this computer works before rebooting; if ssh from root@daq00 doesn't work and you mess this up you will be locked out of wheel account. Once you verify that this works, reboot and make sure that when you login as wheel that home directory is /home1/wheel.

= move /home/wheel =

note: this MUST be done if ZFS root and NIS/autofs with /home.

Default location of wheel's home directory will collide with autofs /home, it has to be moved,
for example to /wheel.

<pre>
# logout from the wheel user
# go to another computer
ssh root@daqubuntuxxx
zfs list | grep wheel ### identify zfs name wheel_xxxxxx
#zfs set mountpoint=/wheel rpool/USERDATA/wheel_hm8fzh
zfs set mountpoint=/wheel `zfs list | grep wheel | cut -f1 -d" "`
zfs list | grep wheel
emacs -nw /etc/passwd ### change wheel's home directory from /home/wheel to /wheel
su - wheel ### check that user wheel still works
</pre>

This will break wheel's ability to run snap programs, such as firefox, install chrome as listed below.

= enable NIS (ubuntu 22.04, 24.04, debian 11, 12) =

<pre>
apt -y install rpcbind nis
echo DAQ-NIS >> /etc/defaultdomain
echo ypserver daq00.triumf.ca >> /etc/yp.conf
systemctl enable ypbind.service
systemctl restart ypbind.service
systemctl status ypbind.service
ypwhich -m
</pre>

enable ypserv:

<pre>
sed -i s/NISSERVER=false/NISSERVER=slave/ /etc/default/nis
/usr/lib/yp/ypinit -s daq00
echo ypserver localhost >> /etc/yp.conf
sed -i "s/ypserver .*/ypserver localhost/" /etc/yp.conf
systemctl enable ypserv
systemctl restart ypserv
systemctl restart ypbind
</pre>

update /etc/nsswitch.conf and enable hourly update of NIS maps:

<pre>
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
cp -pv nsswitch.conf-U24 /etc/nsswitch.conf
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>

If this is a new machine, then on the master NIS node (daq00), add this new node to /etc/netgroup, and update NIS maps (cd /var/yp; make)

= enable NIS (ubuntu 20.04) =

* apt-get -y install portmap nis ### will ask for NIS domain (DAQ-NIS)
* dpkg-reconfigure nis ### reconfigure if already installed
* ypwhich -m
* edit /etc/default/nis
** set "NISSERVER=slave"
** Ubuntu LTS 20.04, check that "YPBINDARGS=" is blank, remove "-no-dbus" if it is there
* #edit /etc/yp.conf, comment-out everything, add "domain DAQ-NIS server localhost"
* edit /etc/yp.conf, comment-out everything, add "ypserver localhost"
* /usr/lib/yp/ypinit -s daq00
* systemctl enable nis
* systemctl restart nis
* ypwhich
* ypwhich -m
* ypcat -k passwd
* vi /etc/nsswitch.conf ### add the automount line, modify the passwd, group and shadow lines to read this:
<pre>
# begin get data from nis
passwd: files nis
group: files nis
shadow: files nis
automount: files nis
netgroup: files nis
# end get data from nis
</pre>
* enable hourly update of NIS maps
<pre>
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>
* ### NOT NEEDED sudo vi /etc/idmapd.conf ### add line: "Domain = triumf.ca"

= enable autofs =

<pre>
apt -y install autofs
systemctl enable autofs
systemctl restart autofs
ls -l /home/olchansk ### test autofs, check file owner is correct
</pre>

= enable NFS server =

<pre>
apt install nfs-kernel-server
#edit /etc/exports
systemctl enable nfs-server
systemctl restart nfs-server
</pre>

= NIS master =

notes for setting up the NIS master

== wheel user ==

"wheel" is the default administrative user. We do not want it's password exported to NIS (encrypted password hash is world visible) and we do not want it's home directory exported to NFS (~wheel/.ssh is world visible and potentially writable: anybody can change ~wheel/.ssh/authorized_keys).

* move wheel's home directory from /home/wheel to /wheel (see special section about this)
* change wheel's UID and GID from 1000 to a value below MINUID in /var/yp/Makefile

== coherent uids ==

we do not want system accounts defined in /etc/passwd of the NIS master
to be included in the NIS map "passwd". this causes trouble on NIS clients
where newly installed packages fail to create local system users because same
user already exists in NIS.

This is controlled by MINUID in /var/yp/Makefile.

Historical TRIUMF uids start from around 200, but several clusters do not have any historic TRIUMF uids below 500 and MINUID is set to:
* DAQ-NIS: MINUID=200
* ISAC-NIS: MINUID=500
* TITAN-NIS: MINUID=500
* MUSR-NIS: MINUID=500
* TIG-NIS: MINUID=500 (100 on SL6 mother8pi)

Ubuntu 20 has two programs to create users:
* adduser - creates new users with UID 1000 and up as specified in /etc/adduser.conf. No problems here.
* adduser --system - creates new system users with UID 100 and up as specified in /etc/adduser.conf. No problems here.
* useradd - creates new users with UID 1000 and up as specified in /etc/login.defs. No problems here.
* useradd --system - creates new system users with UID 999 and down (read "man useradd", section at the end about SYS_UID_MAX). This collides with NIS MINUID, these system users will be included in the NIS map and cause trouble.

This problem cannot be fixed, SYS_UID_MIN, SYS_UID_MAX and UID_MIN in /etc/login.defs do not seem
to have any effect on UIDs chosen by "useradd --system". (tested on Ubuntu LTS 20.04).

So far only these system accounts seem to be affected by this:
* systemd-coredump
* ganglia

To fix:
* run "sort -r -n -t: -k3 /etc/passwd" to identify the last unused system user uid (range 100..200)
* run "sort -r -n -t: -k3 /etc/group" to identify the last unused system user gid (range 100.200)
* systemd-coredump: manually change UID and GID (package systemd-coredump is usually not installed)
* ganglia: same thing, then change ownership on all ganglia files.

Also read systemd author's opinion on system vs user UIDs:
https://github.com/systemd/systemd/issues/4850#issuecomment-265698275

= Fix systemd-logind NIS breakage =

!!! THIS IS NOT NEEDED FOR UBUNTU LTS 20.04 !!!

there is a delay in ssh logins for normal users. "ssh -v" shows the delay is after "pledge...". this
fix removes the delay.

systemd developers think that we should not use NIS and made sure there are
problems if we do. To give them credit, they do offer a workaround. Read this:
https://github.com/poettering/systemd/commit/695fe4078f0df6564a1be1c4a6a9e8a640d23b67

<pre>
mkdir /etc/systemd/system/systemd-logind.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-logind.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-logind.service
</pre>

= Fix systemd-udevd NIS breakage =

see same problem as above with udev getting stuck. ubuntu lts 20.04.

<pre>
mkdir /etc/systemd/system/systemd-udevd.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-udevd.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-udevd.service
</pre>

= Configure USB device permissions =

Configure USB device permissions for user access to USB-serial devices, Altera USB Blaster, etc.

* create file /etc/udev/rules.d/99-usb-chmod.rules with this contents:

<pre>
emacs -nw /etc/udev/rules.d/99-usb-chmod.rules
ACTION=="add", SUBSYSTEM=="usbmisc", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /dev/%c"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /proc/%c"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVICE}"
ACTION=="add", ENV{PHYSDEVBUS}=="usb-serial", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVPATH}=="/class/tty/ttyS*", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyUSB*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyACM*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyS*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", DEVPATH=="*video*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
</pre>

* reload udev rules: udevadm control --reload-rules
* apply new permissions: udevadm trigger --action=add
* watch udev activity: udevadm monitor -p

= Configure lightdm display manager =

* enable it
<pre>
echo lightdm | dpkg-reconfigure -fteletype lightdm
systemctl disable gdm
systemctl disable sddm
systemctl enable lightdm
</pre>

* make the MATE desktop as default
<pre>
cd ~root/git/scripts/
git pull
/bin/cp -v etc/lightdm_default_mate.conf /etc/lightdm/lightdm.conf.d/
</pre>

* enable login by NIS users
<pre>
/bin/cp -v etc/lightdm_enable_nis_login.conf /etc/lightdm/lightdm.conf.d/
</pre>

* restart lightdm
<pre>
systemctl stop gdm
systemctl restart lightdm
</pre>

= Install libpng12.so.0 =

Quartus 16 needs libpng12:

<pre>
wget http://mirrors.kernel.org/ubuntu/pool/main/libp/libpng/libpng12-0_1.2.54-1ubuntu1_amd64.deb
dpkg --install libpng12-0_1.2.54-1ubuntu1_amd64.deb
</pre>

= Install google-chrome =

<pre>
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
dpkg -i google-chrome-stable_current_amd64.deb
</pre>

confirm autoupdate is enabled, observe dl.google.com is present in the list of repositories:
<pre>
apt update
...
Get:5 https://dl.google.com/linux/chrome/deb stable/main amd64 Packages [1,094 B]
...
</pre>

FOLLOWING IS OBSOLETE:

Instructions from here:
https://www.ubuntuupdates.org/ppa/google_chrome?dist=stable

<pre>
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-tmp.list'
apt update
apt install google-chrome-stable
/bin/rm -f /etc/apt/sources.list.d/google-tmp.list
</pre>

= Install amanda client =

ONLY ON MACHINES THAT HOST HOME DIRECTORIES

* apt install amanda-client
* edit /etc/amandahosts
<pre>
amanda.triumf.ca amanda amdump
</pre>
* check permissions on /etc/amandahosts:
<pre>
root@daq00:/var/log/amanda# ls -l /etc/amandahosts
-rw------- 1 backup backup 49 Jan 27 10:48 /etc/amandahosts
</pre>
* fix if needed: chown backup.backup /etc/amandahosts; chmod a= /etc/amandahosts; chmod u=wr /etc/amandahosts
* edit /etc/amanda-security.conf, add this line:
<pre>
runtar:gnutar_path=/usr/bin/tar
</pre>

On the amanda machine:

* in amanda disklist, use dump type "bsdtcp-comp-user-tar"
* su - amanda and run amcheck -c daily daq00
<pre>
-bash-4.1$ amcheck -c daily daq00

Amanda Backup Client Hosts Check
--------------------------------
Client check: 1 host checked in 0.092 seconds. 0 problems found.

(brought to you by Amanda 3.3.7p1.git.685ff76d)
</pre>

= Enable rc.local =

For reasons unknown, Ubuntu LTS 20.04 does not enable /etc/rc.local. Do this:

<pre>
cd ~/git/scripts
git pull
cp -n -v etc/rc.local /etc/
chmod a+rx /etc/rc.local
cp etc/rc-local.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable rc-local
systemctl start rc-local
systemctl status rc-local
</pre>

= Remove unwanted packages =

<pre>
apt purge bash-completion # broken, adds unwanted "\" if "ls -l $ROOTSYS/<tab>"
apt purge zsys # broken, do not use
apt purge sddm # login manager
apt purge avahi-daemon avahi-autoipd # not sure what it does, observed using 100% CPU
apt purge modemmanager # probes all serial ports to see if it's a modem
</pre>

= Disable unwanted services =

<pre>
systemctl disable mpd
systemctl disable snapd
systemctl disable ModemManager
systemctl --global mask tracker-extract-3.service
systemctl --global mask tracker-miner-fs-3.service
systemctl daemon-reload
</pre>

= Disable sleep and suspend =

note: we see some computers randomly shutdown or go to sleep, log files indicates the "sleep" or "suspend" button was pushed by user, but no such buttons actually exist. this is the fix for this:

<pre>
systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target systemd-suspend.service systemd-hybrid-sleep.service
</pre>

= Enable crontab @reboot for MIDAS =

startup scripts have a bug - cron @reboot entries for normal users can run before autofs is ready, so if the home directory is on autofs/NFS, it cannot be accessed and the cron job fails. If MIDAS is supposed to be started by cron @reboot, it will not start (there *will* be an error message in /var/log/cron).

<pre>
mkdir /etc/systemd/system/cron.service.d
echo -e "[Unit]\nAfter=ypbind.service autofs.service\n" > /etc/systemd/system/cron.service.d/local.conf
systemctl daemon-reload
systemctl cat cron.service
</pre>

Explore the systemd dependency tree using "systemctl list-dependencies" maybe with "--all".

Visualize the exact boot sequence from previous boot: "systemd-analyze plot > xxx.svg", look at the svg file using a web browser.

Crontab entry to start midas: (install in the midas user crontab, not root crontab)

<pre>
su - midasuser
crontab -l
#@reboot /bin/bash -l -c "/home/trinat/bin/start-daq-applications"
#@reboot /bin/tcsh -c "/home/trinat/bin/start-daq-applications"
</pre>

= Install apache httpd proxy for midas and elog =

This will configure the HTTPS/SSL certificate using "certbot" and "letsencrypt" and configure an HTTPS web server using apache2.

First, configure apache2:

* execute these commands:
<pre>
apt -y install apache2
cd /etc/apache2
</pre>
* create new file conf-available/ssl-daq14.conf # use actual hostname instead of daq14
<pre>
SSLSessionCache shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout 300
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
SSLCryptoDevice builtin
</pre>
* create new file sites-available/daq14-ssl.conf # use actual hostname instead of daq14
<pre>
<IfModule mod_ssl.c>
<VirtualHost *:443>
ServerName daq14.triumf.ca
DocumentRoot /var/www/html
ErrorLog /var/log/apache2/daq14.log
SSLEngine on
# note SSLProtocol, SSLCipherSuite and some other settings are overwritten by /etc/letsencrypt/options-ssl-apache.conf
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SEED:!IDEA:!RC4
## use port specified in elogd.cfg
#ProxyPass /elog/ http://localhost:8082/ retry=1
## use mhttpd port
#ProxyPass / http://localhost:8080/ retry=1
Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"
<Location />
SSLRequireSSL
AuthType Basic
AuthName "DAQ password protected site"
Require valid-user
# create password file: touch /etc/apache2/htpasswd
# to add new user or change password: htpasswd /etc/apache2/htpasswd username
AuthUserFile /etc/apache2/htpasswd
</Location>
#SSLCertificateFile /root/server.cert
#SSLCertificateKeyFile /root/server.key
</VirtualHost>
</IfModule>
</pre>
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* enable ssl module
* enable new configurations
<pre>
a2enmod ssl
a2enmod headers
a2enmod proxy
a2enmod proxy_http
a2enconf ssl-daq14
a2ensite daq14-ssl
</pre>
* disable default ssl sites
<pre>
a2dissite 000-default-le-ssl
a2dissite 000-default
ls -l /etc/apache2/sites-enabled/ ### should show only daq14-ssl.conf
</pre>
* check that there are no syntax problems
<pre>
apache2ctl configtest
</pre>
* enable and start apache2:
<pre>
systemctl enable apache2
systemctl restart apache2
systemctl status apache2
</pre>
* apache2 may fail to start, look in /var/log/apache2/error.log and /var/log/apache2/daq14.log
* if it says "Failed to configure ... certificate", proceed to the step for setting certbot.
* try to access https://daq14.triumf.ca
** you should see a complaint about self-signed certificate
** you should see a request for password (do not login yet)
** if you get "connection refused", HTTPS port 443 may need to be enabled in the local firewall, look at documentation for ufw.
Second, configure certbot:

(Note: as of 2018-01-18 certbot requires use of http port 80 to get the initial https certificate,
renewal can continue to use the https port 443)

(Note: as of 2019-01-?? certbot requires use of port 80 for renewals)

(Note: unsurprisingly, this requires outside access to connect with letsencrypt, so won't work if PC is only accessible from on-site network)

* check that port 80 is not used by anything:
* netstat -an | grep LISTEN | grep ^tcp | grep 80
* lsof -P | grep -i tcp | grep LISTEN | grep 80
* if lsof reports that apache2 is listening on port 80, follow the apache2 instructions above (remove "listen 80" from apache2.conf

* install certbot (if necessary open tcp port 80 in the firewall, see documentation for ufw):
<pre>
apt install certbot python3-certbot-apache
certbot certonly --standalone --installer apache
</pre>
* then answer questions:
* "activate HTTPS for daq14.triumf.ca" - say ok
* "enter email address" - enter your own email address
* "please read terms..." - read the terms and say "agree"
* it will take a few moments...
* "congratulations..." - say ok.
<pre>
certbot install --apache --cert-name daq14.triumf.ca
</pre>
* then answer questions:
* "choose redirect..." - say "1" (no redirect)
* look inside /etc/apache2/sites-enabled/daq14-ssl.conf to see that SSLCertificateFile & co point to certbot certificates in
/etc/letsencrypt/live/daq14.triumf.ca/
* to check current renewal and to update the certbot config file in /etc/letsencrypt/renewal, run this:
<pre>
certbot renew --standalone --installer apache --force-renewal
</pre>

NOTE: this certificate will expire in 3 months, automatic renewal should work with current version of certbot

Third, activate password protection:

* as shown in the config file above, create password file and initial user: (replace "midas" with specific username)
<pre>
touch /etc/apache2/htpasswd
htpasswd /etc/apache2/htpasswd midas
</pre>

* restart apache2
<pre>
systemctl restart apache2
systemctl status apache2
</pre>

From here:
* enable proxy for MIDAS mhttpd - uncomment redirect in the config file above
* enable proxy for ELOG - ditto
<pre>
a2enmod proxy
a2enmod proxy_http
apache2ctl configtest
systemctl restart apache2
</pre>

* try accessing MIDAS https://daq14.triumf.ca/ (make sure mhttpd is running)
* if it's not working, check odb setting FIXME!
* try accessing ELog https://daq14.triumf.ca/elog/ (make sure elogd is running)
* if it's not working, check elogd.cfg file and make sure
<pre>
SSL = 0
</pre>

NOTE: if certbot fails with errors about 'module' object has no attribute 'pyopenssl',
try this: pip install requests==2.6.0

== generate self-signed certificate ==

<pre>
# cd $HOME
# openssl req -nodes -new -x509 -keyout server.key -out server.cert -days 1001
...+....+..+..........+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+..+...+.........+......+.+...+...+.....+...............+.........+...+.+......+...+...........+....+...+..+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+......+.+...+..+.......+..+...+.......+......+...+..+...+......+....+...............+..+...+....+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
......+......+.+..+......+.+......+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.....+......+.+.........+......+.....+.+..+...+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+....+......+.....+...+...+.......+..+.+........+.+...+......+..+..........+..+.+...........+...+.......+......+.....+.......+...+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:CH
State or Province Name (full name) [Some-State]:Geneve
Locality Name (eg, city) []:CERN
Organization Name (eg, company) [Internet Widgits Pty Ltd]:CERN
Organizational Unit Name (eg, section) []:ALPHA experiment
Common Name (e.g. server FQDN or YOUR name) []:alphacpc05.cern.ch
Email Address []:
root@alphacpc05:~#
root@alphacpc05:~#
root@alphacpc05:~# ls -l
-rw-r--r-- 1 root root 1375 juil. 10 21:43 server.cert
-rw------- 1 root root 1708 juil. 10 21:42 server.key
root@alphacpc05:~# systemctl restart apache2
</pre>

= Enable elog PDF preview =

NOTE: looks like U-24 already has this correctly.

see https://stackoverflow.com/questions/52998331/imagemagick-security-policy-pdf-blocking-conversion

* xemacs -nw /etc/ImageMagick-6/policy.xml
* remove this section at the end:
<pre>

<policy domain="coder" rights="none" pattern="PS" />
<policy domain="coder" rights="none" pattern="PS2" />
<policy domain="coder" rights="none" pattern="PS3" />
<policy domain="coder" rights="none" pattern="EPS" />
<policy domain="coder" rights="none" pattern="PDF" />
<policy domain="coder" rights="none" pattern="XPS" />
</pre>

= Install Jupyter notebook =

<pre>
From https://jupyter.org/install
apt install python3-pip
pip install jupyterlab
pip install notebook
~/.local/bin/jupyter notebook
watch the http://localhost:8888 URL that it printed
say "no" to offer to start firefox (it will not work!)
URL is: http://localhost:8888/tree?token=xxx
from the machine where you are running the web browser (i.e. google-chrome), run (replace trinat@trinatdaq with the username and machine name where you started jupyter)
open a new shell and run: ssh -v trinat@trinatdaq -L 8888:localhost:8888
in the web browser, open http://localhost:8888
this gives us the login page
in the password or token entry field, put the token from the "tree?token=xxx" above (printed by jupyter on startup)
push button "login"
jupyter page should open with the list of files in the trinat home directory
congratulate Brian with full success
</pre>

= Install ZFS quota report =

If there are any ZFS volumes, install script to report disk and quota usage

<pre>
cd ~/git/scripts/quotareport
git pull
mkdir /var/www/html/zfsquotareport
cp -pv ~/git/scripts/quotareport/sorttable.js /var/www/html/zfsquotareport/
ln -s $PWD/zfsquotareport.perl /etc/cron.daily/
touch /etc/crontab
</pre>

If httpd is configured to redirect "/" to MIDAS mhttpd:
* add following to /etc/apache2/sites-enabled/xxx-ssl.conf in front of "ProxyPass / ..."
* run "systemctl reload apache2"
<pre>
## do not proxy zfs quota report directory
ProxyPass /zfsquotareport/ !
</pre>

= Install PHP =

* apt install php libapache2-mod-php
* systemctl restart apache2
* create /var/www/html/info.php
<pre>
<?php

phpinfo();
</pre>
* open https://daq00.triumf.ca/info.php

= Configure TRIUMF printers =

<pre>
systemctl stop cups
systemctl stop cups-browsed.service
systemctl disable cups
systemctl disable cups-browsed.service
systemctl stop snap.cups.cupsd.service
systemctl stop snap.cups.cups-browsed.service
systemctl disable snap.cups.cupsd.service
systemctl disable snap.cups.cups-browsed.service
echo "ServerName printers.triumf.ca" > /etc/cups/client.conf
lpstat -a
</pre>

= Enable core dumps =

By default, Ubuntu LTS 20.04 installs the apport package
which disabled core dumps from user applications. (google it up!).
It is not meant to do this and documentation claims that
it is not installed and not enabled by default. Oh, well...

<pre>
apt purge apport
apt autoremove ### will remove apport-symptoms and a few other packages
</pre>

After this, core dumps are written to file "core" in the current directory.
See /proc/sys/kernel/core_pattern and /proc/sys/kernel/core_uses_pid.

Enable core dump file names to include process id, add following to /etc/rc.local

<pre>
echo "echo 1 > /proc/sys/kernel/core_uses_pid" >> /etc/rc.local
</pre>

= Enable debugger =

By default, Ubuntu LTS 20.04 does not permit debugger to attach and debug
already running programs. To enable it, add following to /etc/rc.local

<pre>
echo "echo 0 > /proc/sys/kernel/yama/ptrace_scope" >> /etc/rc.local
</pre>

= Disable Ubuntu Pro nag =

best I can tell, impossible at this time.

== do not do this ==

!!! does nothing !!!

<pre>
pro config set apt_news=false
</pre>

== do not do this ==

!!! breaks automatic updates because 20apt-esm-hook.conf is missing !!!

If "apt upgrade" requests Ubuntu Pro or esm-apps, disable the nag:
<pre>
/bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf
</pre>

== do not do this ==

!!! likely same as above, breaks automatic updates !!!

* comment out /etc/apt/apt.conf.d/20apt-esm-hook.conf

== do not do this ==

!!! removes too many packages !!!

<pre>
apt remove ubuntu-pro-client
</pre>

= Update packages =

* apt update # update package list
* apt upgrade # install updated packages and update "kept back" packages
* apt autoremove # remove packages that apt thinks should be removed

= Cleanup residual configs =

* apt list '~c'
* apt purge '~c'

= Install firefox-esr =

install firefox-esr here

= Finish installation =

Congratulations. There is nothing more to do!

* reboot
<pre>
shutdown -r now
</pre>

= Update to new version of Ubuntu =

* run "do-release-upgrade -c"
* if it does not report new release Ubuntu 24, check /etc/update-manager/release-upgrades has "Prompt=lts"

== Update Ubuntu LTS 20.04 to LTS 22.04 ==

<pre>
apt remove zsys
</pre>

=== daqubuntu ===

<pre>
# reboot to clear out all updates
# vi /etc/update-manager/release-upgrades # set "Prompt=normal"
# do-release-upgrade -c
Checking for a new Ubuntu release
New release '22.04 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
# do-release-upgrade
...
say yes...
...
login.defs, say "Y" (erase local changes, use packaged version)
/etc/systemd/resolved.conf, say "Y" (same as above)
firefox snap, say yes
unable to reach snap store, say "skip"
/etc/gmond.conf, say "Y"
/var/yp/Makefile, say "install the package maintainer's version"
/etc/ypserv.conf, same thing
/etc/ypserv.securenets, same thing
/etc/default/nis, same thing
/etc/speech-dispatcher/modules/mary-generic.conf, same thing
/etc/apt/apt.conf.d/50unattended-upgrades, same thing
...
278 packages are going to be removed, say yes
...
restart required, say yes
...
no ping... yes ping...
...
ssh daqubuntu, ok
apt update, fail, DNS does not work, "host security.ubuntu.com" does not resolve.
fix resolver per https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Disable_NetworkManager
apt update, apt upgrade now works, 0 packages to update
NIS does not work.
</pre>

=== midm9a ===

<pre>
login.defs
firefox snap
gmond.conf
ypserv
/etc/default/nis
unattended-upgrades
amanda-security.conf
remove obsolete (no)
reboot
configure dns
reenable nis
</pre>

=== daq17 ===

<pre>
firefox snap
imagemagick policy.xml
gmond.conf
chrony.conf
/var/yp/Makefile
ypserv.conf
ypserv.securenets
/etc/default/nis
50unattended-upgrades
</pre>

=== daq00 ===

per https://serverpilot.io/docs/how-to-upgrade-ubuntu-20.04-to-22.04/

<pre>
do-release-upgrade -f DistUpgradeViewNonInteractive
</pre>

if it exists "too soon" without doing anything, run it without "-f xxx", most likely it does not like something about this machine. in case of daq00 it did not like how the EFI partitions were mounted. after fixing it, non-interactive upgrade was successful.

=== isdaq08 ===

* prepare
<pre>
cd ~/git/scripts
git pull
cd ~
apt -y install debsums
</pre>
* check for modified config files that make upgrade unhappy, deal with all files reported by debsums.
<pre>
root@isdaq08:~# debsums -ce
/etc/ganglia/gmond.conf
/etc/yp.conf
/etc/apt/apt.conf.d/10periodic
root@isdaq08:~#
</pre>
* restore original /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
</pre>
* apt remove ganglia-monitor
* apt remove nis
* "debsums -ce" is now empty

Run the upgrade:

* do-release-upgrade -f DistUpgradeViewNonInteractive

Post upgrade:

* configure DNS
* apt -y install linux-generic-hwe-22.04
* /bin/cp -v ~/git/scripts/etc/99apt-conf-ko /etc/apt/apt.conf.d/ # restore nightly updates
* /bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf # remove the ubuntu-pro nag
* install missing packages
* restore ganglia
* restore nis
* check zpool status, may need zpool upgrade
* reboot

== upgrade U-22 to U-24 ==

generic instructions. below are notes from upgrades of specific machines.

NOTE: at CERN saw installation getting stuck on restart of autofs (automount, rsyslogd 100% CPU, huge /var/log/syslog), stop autofs before upgrade?
<pre>
systemctl stop autofs
</pre>

<pre>
cleanup zsys
</pre>

<pre>
maybe: if snap is already removed, remove firefox to prevent snap from reinstalling.
install non-snap firefox-esr
</pre>

<pre>
remove thunderbird to prevent installation of snap thunderbird in U-24
</pre>

<pre>
mount EFI partition as /boot/efi, otherwise upgrade bombs
usually mount /dev/sda1 /boot/efi
</pre>

<pre>
debsums -ce
apt -y remove desktop-base # causes installer crash
apt -y remove thunderbird # avoid forced conversion to snap
apt update
apt -y upgrade
apt -y autoremove
do-release-upgrade -c # confirm upgrade will be to U-24
do-release-upgrade
# say "y" to all questions
# after installation starts, accept default answers to all questions
# should run for about 1 hour or so
# after upgrade finishes
apt update
apt -y upgrade
apt -y autoremove
shutdown -r now
</pre>

post upgrade:

* check zpool status, may need zpool upgrade
* cd /etc/apt/sources.list.d, reenable 3rd party repos (mozilla, google, etc)

if installation bombs:

<pre>
apt update
apt upgrade # will tell us to run "apt --fix-broken install", do it
apt --fix-broken install
apt update
apt upgrade # should resume the upgrade, will run for a long time
apt update
apt upgrade # should do nothing
apt autoremove
shutdown -r now
</pre>

=== daqubuntu, U-24 ===

* prepare
<pre>
cd ~/git/scripts
git pull
cd ~
apt -y install debsums
</pre>
* check for modified config files that make upgrade unhappy, deal with all files reported by debsums.
<pre>
root@daqubuntu:~# debsums -ce
/etc/ganglia/gmond.conf
debsums: missing file /etc/init.d/nis (from nis package)
/etc/default/nis
/etc/ypserv.conf
/etc/ypserv.securenets
/var/yp/Makefile
/etc/update-manager/release-upgrades
/etc/apt/apt.conf.d/10periodic
/etc/yp.conf
root@daqubuntu:~#
* restore original /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
</pre>
* apt remove ganglia-monitor
* apt remove nis
* apt autoremove
* restore original release-upgrades: "Prompt: lts"
* "debsums -ce" is now empty

Check for upgrade:

<pre>
root@daqubuntu:~# do-release-upgrade -c
Checking for a new Ubuntu release
There is no development version of an LTS available.
To upgrade to the latest non-LTS development release
set Prompt=normal in /etc/update-manager/release-upgrades.
root@daqubuntu:~#
</pre>

Run the upgrade:

* do-release-upgrade -f DistUpgradeViewNonInteractive

Post upgrade:

* configure DNS
* apt -y install linux-generic-hwe-22.04
* /bin/cp -v ~/git/scripts/etc/99apt-conf-ko /etc/apt/apt.conf.d/ # restore nightly updates
* /bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf # remove the ubuntu-pro nag
* install missing packages
* restore ganglia
* restore nis
* check zpool status, may need zpool upgrade
* cd /etc/apt/sources.list.d, reenable 3rd party repos (mozilla, google, etc)
* reboot

=== daq14, U-20-22-24 ===

* apt update, apt upgrade
* apt -y install linux-image-generic-hwe-20.04 linux-tools-virtual-hwe-20.04 ### install kernel 5.15
* shutdown -r now
* stuck waiting for daq14 to shutdown...
* reboot into kernel 5.15
* ???
<pre>
cd ~/git/scripts
git pull
cd ~
apt -y install debsums
</pre>
* debsums -ce
<pre>
/etc/apache2/ports.conf
/etc/dnsmasq.conf
/etc/ganglia/gmond.conf
/etc/yp.conf
/etc/sudoers
</pre>
* apache2 restore original ports.conf, uncomment "Listen 80"
* cp -pv /etc/dnsmasq.conf.dpkg-dist /etc/dnsmasq.conf
* apt remove ganglia-monitor
* edit /etc/yp.conf, remove everything after "# ypserver ypserver.network.com"
* "debsums -ce" is now empty
* do-release-upgrade -f DistUpgradeViewNonInteractive
* runs for a long time
* stuck on "/etc/default/nis", type "Y", press enter, nothing for a bit, then resumes running
* finished
* configure DNS
* reboot
* have kernel 6.8
* apt update; apt upgrade
* apt upgrade guile-2.2-libs ### would not auto-update, "kept back", has to be done by hand
* apt autoremove
* debsums -ce
<pre>
debsums: missing file /etc/init.d/nis (from nis package)
/etc/default/nis
</pre>
* diff /etc/default/nis.dpkg-dist /etc/default/nis
* cp -pv /etc/default/nis.dpkg-dist /etc/default/nis
* debsums -ce
<pre>
debsums: missing file /etc/init.d/nis (from nis package)
</pre>
* we ignore this and run the update
* do-release-upgrade -c
<pre>
Checking for a new Ubuntu release
New release '24.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
</pre>
* do-release-upgrade -f DistUpgradeViewNonInteractive
* bombs out without any error messages
* in /var/log/dist-upgrade/main.log reports "Failed to find a replacement for xapp" and other packages
* apt remove xapp usrmerge ureadahead thunderbird-gnome-support
* no go, complains about even more packages.
* apt list | grep installed | grep -v jammy ### show packages installed from non-ubuntu sources
* remove all packages marked "install,local" ### ubuntu updater does not know where they came from and so cannot update them.
* apt remove desktop-base ### not happy about this package in /var/log/dist-upgrade/apt.log
* apt autoremove
* do-release-upgrade -f DistUpgradeViewNonInteractive
* running for a long time...

=== alpha04 U-20-24 ===

* apt update, apt upgrade, apt autoremove
* reboot into latest kernel (already done)
* debsums -ce
<pre>
root@alpha04:~# debsums -ce
/etc/dnsmasq.conf
/etc/ganglia/gmond.conf
/etc/default/nis
/etc/yp.conf
root@alpha04:~#
</pre>
* move /etc/dnsmasq.conf to /etc/dnsmasq.d/alpha04.conf
* apt remove dnsmasq
* apt remove ganglia-monitor
* apt remove nis
* apt autoremove
* debsums -ce ### is now empty
* do-release-upgrade -f DistUpgradeViewNonInteractive
* it runs for a long time...
* complained about /etc/fwupd config files, not sure why...
* finished
* apt update, apt upgrade, apt autoremove
* restore dnsmasq: apt install dnsmasq, systemctl status dnsmasq
* restore ganglia, per instructions
* restore NIS: apt -y install rpcbind nis, ypwhich, ypwhich -m
* zpool upgrade rpool ### also upgrade any other zfs pools, see zpool status
* remove unwanted packages, per instructions
* run gonodeinfo
* reboot
* done

== vera00 U20-22-24 ==

* everything same as daq14 for U20-22
* kernel is still 5.15
* U22-24 is going...
* stuck for a few minutes on /etc/fwupd config files
* have kernel 6.8.0-49
* same steps as daq14
* reboot
* same steps as daq14
* done

== phaarmonster U22-24 ==

* debsums -ce
<pre>
/etc/amandahosts
/etc/apache2/ports.conf
/etc/ganglia/gmond.conf
</pre>
* do-release-upgrade -c ### reports U24.04.1
* do-release-upgrade -f DistUpgradeViewNonInteractive ### bombs
* apt remove desktop-base; apt autoremove
* do-release-upgrade ### (interactive) runs ok
* bombed !!!
* apt upgrade spews errors and tells us to run "apt --fix-broken install"
* apt --fix-broken install ### runs
* bombs with thunderbird snap errors
* again... no go
* thunderbird snap complains about mounting /home, but /home is a symlink
* rm /home, mkdir /home
* again, runs ok
* asks about /etc/fwupd/fwupd.conf - say "Y" to install updated package version
* apt install completes
* apt update; apt upgrade ### running for a long time...
* finished
* install missing packages, etc
* reboot
* long wait... came back
* DNS does now work, systemd-resolved missing, apt install systemd-resolved, "configure DNS"
* done.

== isdaq10 U22-24 ==

* do-release-upgrade -f DistUpgradeViewNonInteractive
* bombed on desktop-base, "apt remove desktop-base; apt autoremove"
* bombed with errors
* running "apt --fix-broken install"
* complained about thunderbird snap
* complained about /etc/fwupd/fwupd.conf (say Y)
* finished ok
* apt update
* apt upgrade ### reports "1382 upgraded, 140 newly installed, 0 to remove and 29 not upgraded"

== iris01 ==

* debsums -ce
<pre>
/etc/ganglia/gmond.conf
/etc/default/nis
/etc/yp.conf
</pre>
* apt remove desktop-base
* apt remove thunderbird
* apt autoremove
* do-release-upgrade -f DistUpgradeViewNonInteractive
* bombed with dpkg errors:
<pre>
hplip wants wrong python
ganglia-monitor wants wrong libapr1
sssd-ad wants bunch of wrong libraries
xemacs21-mule
libnfsidmap1 wants libldap2
adn so forth...
</pre>
* apt --fix-broken install -y
* bombs on ganglia - ganglia group absent from /etc/groups
* fix group ganglia by hand, remove ganglia group from NIS on isdaq00
* apt --fix-broken install
* apt upgrade

== iris00 ==

* stuck on "autofs: restarting", login as root, kill iris midas, kill automount, systemctl restart autofs, noble got unstuck, ctrl-c systemctl restart autofs
* noble running...
* bombed with dpkg errors
* check ganglia user, group - both ok
* apt --fix-broken install
* apt upgrade

== tigstore01 ==

* no bomb-out

== midm9b ==

* apt remove desktop-base thunderbird
* bombed
* apt --fix-broken-install
* apt upgrade

= Upgrade to new version of Debian =

https://www.debian.org/releases/bookworm/amd64/release-notes/ch-upgrading.en.html

== 32-bit VME processor Debian 11 to 12 ==

* cd git/scripts; git pull; cd ~
* apt update
* apt upgrade
* edit /etc/apt/sources.list
<pre>
deb http://deb.debian.org/debian/ bookworm main
#deb http://deb.debian.org/debian/ bullseye main
#deb http://deb.debian.org/debian/ buster main
#deb-src http://deb.debian.org/debian/ bullseye main
</pre>
* apt update
* apt upgrade --without-new-pkgs
* apt full-upgrade
* apt list '~c'; apt purge '~c' # purge left-over config files [residual-config]
* reboot

= Ubuntu package manager =

* apt-get install xxx # install package xxx
* apt-get update
* apt-get upgrade
* apt-get dist-upgrade
* apt-get autoremove # remove automatically installed packages required by a removed package
* apt-get remove xxx # remove package xxx
* apt-cache search . # list all available packages
* apt-cache show "." | grep ^Package # list al available packages
* apt-cache madison root-system # show all available versions of package root-system
* apt list # list all installed packages
* dpkg --listfiles libpng16-16 # list all files from this package
* apt list --installed # list all installed packages
* dpkg -S /bin/bash # what package provides this file?
* dpkg -L bash # what files provided by this package?
* debsums -ce # show modified config files
* apt-config dump # show apt configuration
* apt purge '~c' # purge all [residual-config] packages
* ls -l /var/lib/dpkg/info/ # show post-install scripts

= Ubuntu zsys =

NOTE: DO NOT USE ZSYS, see https://github.com/ubuntu/zsys/issues/218 and https://github.com/ubuntu/zsys/issues/230

* scripted removal of old snapshots (replace "echo zfs destroy" with "zfs destroy")
<pre>
zfs list -t all | cut -f1 -d " " | grep autozsys | xargs -n1 echo zfs destroy
</pre>

* manual removal of old snapshots
<pre>
zsysctl show
zsysctl state remove xy69ye -s
zsysctl state remove xy69ye
zsysctl state remove xy69ye -u wheel
</pre>
* apt remove zsys

NOTE: old zsys snapshots must be cleaned manually, "zsysctl state remove xxx --system" is broken and does not remove user data snapshots

* manages system snapshots
* documentation: https://github.com/ubuntu/zsys
* documentation: (go to next article via link "newer" at the bottom) https://didrocks.fr/2020/05/21/zfs-focus-on-ubuntu-20.04-lts-whats-new/
* ubuntu 20.04 bug, too many snapshots cause /boot to become full and updates fail. https://github.com/ubuntu/zsys/issues/155
* solution: use custom /etc/zsys.conf, limit number of snapshots to 10, see trinatdaq:/etc/zsys.conf
* zsys commands:
<pre>
update-grub # list of all snapshots, errors if some snapshots are broken
zsysctl state remove lnc0k7 --system # remove snapshot
xemacs -nw /etc/zsys.conf; zsysctl service reload; zsysctl service gc # cause gc to run with new settings in zsys.conf
zfs list -r -t snapshot -o name,used,referenced,creation bpool/BOOT # list snapshots
zsysctl show # show snapshots
</pre>

= Ubuntu cloning =

to clone a ubuntu image:

<pre>
cd /nfsroot/lxcpet
emacs -nw etc/hostname ### change hostname
emacs -nw etc/mailname ### change hostname (debian 11)
emacs -nw etc/defaultdomain ### change the NIS domainname
emacs -nw etc/yp.conf ### change the NIS server
cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
emacs -nw root/.ssh/authorized_keys ### update root ssh keys
</pre>

= Ubuntu boot loader =

== maintenance commands ==

* update-initramfs -v -u
* grub-install /dev/sda

= Convert from single to dual mirrored ZFS SSD =

Assuming Ubuntu LTS 22.04 with "instal on ZFS" option, we will
add a second SSD, configure ZFS to use both SSDs in mirrored
configuration and setup grub to boot from either SSD. This
is intended to create a full redundant system where failure
of either SSD does not break the system.

== partition ==

* identify first SSD
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* connect second SSD of identical size
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
/dev/sdb WD Blue SA510 2.5 250GB 22243Z803852 25 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* if second SSD is not autodetected, reboot
* Clone partition table automatically
If both SSDs are identical size, use this simpler method of duplicating the partition table:
<pre>
root@midm9b:~# sfdisk -d /dev/sda > part_table
root@midm9b:~# grep -v ^label-id part_table | sed -e 's/, *uuid=[0-9A-F-]*//' | sfdisk /dev/sdb
</pre>
The grep and sed in the second command are there to prevent disk ID and partition IDs from being cloned. Alternatively the part_table file can be edited manually to remove the label-id line and the uuid entries from the individual partitions.

* Clone partition table manually (e.g. for different size disks)
* list partition table of first SSD:
<pre>
root@midm9b:~# fdisk -l /dev/sda
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>
* create identical partitions on second SSD, use sector numbers from above.
<pre>
root@midm9b:~# gdisk /dev/sdb
GPT fdisk (gdisk) version 1.0.8

Partition table scan:
MBR: not present
BSD: not present
APM: not present
GPT: not present

Creating new GPT entries in memory.

Command (? for help): n
Partition number (1-128, default 1):
First sector (34-488397134, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-488397134, default = 488397134) or {+-}size{KMGTP}: 1050623
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): ef00
Changed type of partition to 'EFI system partition'

Command (? for help): n
Partition number (2-128, default 2):
First sector (34-488397134, default = 1050624) or {+-}size{KMGTP}:
Last sector (1050624-488397134, default = 488397134) or {+-}size{KMGTP}: 5244927
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 8200
Changed type of partition to 'Linux swap'

Command (? for help): n
Partition number (3-128, default 3):
First sector (34-488397134, default = 5244928) or {+-}size{KMGTP}:
Last sector (5244928-488397134, default = 488397134) or {+-}size{KMGTP}: 9439231
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): be00
Changed type of partition to 'Solaris boot'

Command (? for help): n
Partition number (4-128, default 4):
First sector (34-488397134, default = 9439232) or {+-}size{KMGTP}:
Last sector (9439232-488397134, default = 488397134) or {+-}size{KMGTP}:
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): bf00
Changed type of partition to 'Solaris root'

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sdb.
The operation has completed successfully.
root@midm9b:~# fdisk -l /dev/sda /dev/sdb
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root

Disk /dev/sdb: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: EB251739-30C6-422F-A505-5887B5A0B603

Device Start End Sectors Size Type
/dev/sdb1 2048 1050623 1048576 512M EFI System
/dev/sdb2 1050624 5244927 4194304 2G Linux swap
/dev/sdb3 5244928 9439231 4194304 2G Solaris boot
/dev/sdb4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>

== update ZFS pools ==

* identify second SSD partitions
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part2
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part2 -> ../../sdb2
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
</pre>
* convert bpool from single disk to mirrored disk:
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~# zpool attach bpool 99e03dc0-7d4d-f24b-8fa1-f042b9f135db /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3
root@midm9b:~# zpool status bpool
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors
</pre>
* convert rpool
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
root@midm9b:~# zpool attach rpool f6fd54f8-3af7-b943-ae3d-a4e480537fb9 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4
root@midm9b:~# zpool status rpool
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Jan 20 19:40:45 2023
5.83G scanned at 664M/s, 2.92M issued at 332K/s, 9.11G total
0B resilvered, 0.03% done, no estimated completion time
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~#
</pre>
* wait for resilver to complete
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
scan: resilvered 9.65G in 00:00:36 with 0 errors on Fri Jan 20 19:41:21 2023
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
</pre>

== update boot loader ==

INSTALL SYSLINUX: https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#EFI_boot_using_syslinux

DO *NOT* DO THE FOLOWING:

* create and mount EFI partitions:
<pre>
root@midm9b:~# mkfs.msdos /dev/sdb1
root@midm9b:~# mkdir /boot/efi-sda
root@midm9b:~# mkdir /boot/efi-sdb
root@midm20c:~# blkid | grep vfat ### identify UUID
/dev/sdb1: UUID="DD89-5081" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="d0cb6be4-2f67-5b42-9b26-9e6905e9f774"
/dev/sdc1: UUID="D970-86BA" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="e6d3b5b9-a512-44a2-9205-1a4db06ed2a2"
/dev/sda1: UUID="DDA1-044C" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="6dc9dff0-1c13-8045-a906-7803d3074c70"
root@midm20c:~# cat /etc/fstab | grep vfat ### add mount points with correct UUID
#UUID=D970-86BA /boot/efi vfat umask=0022,fmask=0022,dmask=0022 0 1
UUID=DDA1-044C /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022 0 1
UUID=DD89-5081 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022 0 1
root@midm9b:~# mount -a
root@midm9b:~# df -kl
Filesystem 1K-blocks Used Available Use% Mounted on
...
/dev/sda1 523244 13720 509524 3% /boot/efi
/dev/sdb1 523244 4 523240 1% /boot/efi-sdb
...
root@midm9b:~# rsync -av /boot/efi/ /boot/efi-sdb/
sending incremental file list
EFI/
...
root@midm9b:~# ls -l /boot/efi-sda
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~# ls -l /boot/efi-sdb
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~#
</pre>
* add systemd "nofail" flag to /etc/fstab, without this, systemd will stop booting if one SSD is missing
<pre>
daq00:~$ cat /etc/fstab | grep vfat
#UUID=31A7-24BE /boot/efi vfat umask=0022,fmask=0022,dmask=0022 0 1
/dev/sda1 /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
/dev/sdb1 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
</pre>
* setup script to update grub on second SSD, it must be run manually after every kernel update
<pre>
root@midm9b:~# ln -s ~/git/scripts/etc/update_efi_grub.perl ~/
root@midm9b:~# ~/update_efi_grub.perl -u
EFI dir: /boot/efi-sda
/boot/efi-sda: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sda/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sda: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sda/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
EFI dir: /boot/efi-sdb
/boot/efi-sdb: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sdb/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sdb: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sdb/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
root@midm9b:~#
</pre>

= Disable NetworkManager =

== Debian-12 ==

* use netplan, https://www.debian.org/doc/manuals/debian-reference/ch05.en.html#_the_modern_network_configuration_for_cloud
* apt install netplan.io
* systemctl enable systemd-networkd
* systemctl restart systemd-networkd
* create /etc/netplan/50-dhcp.yaml
<pre>
network:
version: 2
ethernets:
all-en:
match:
name: "en*"
dhcp4: true
dhcp6: true
ignore-carrier: true ### do not drop IP address if network link drops
</pre>
* netplan apply
* netplan try
* ifconfig -a ### to check IP address settings
* netstat -rn ### to check default route
* cat /etc/resolv.conf ### to check DNS
* ls -l /run/systemd/netif/leases ### systemd-networkd dhcp leases
* NOTE: without "ignore-carrier" it will drop the IP address if network link drops, re-do dhcp when links comes back
* NOTE: wait-network-online will wait for all interfaces to get an IP address

== Ubuntu-20 ==

NOTE: THIS IS BROKEN IN UBUNTU LTS 22.04

NetworkManager is useful for configuring dynamic
network interfaces, i.e. laptops that often move
between networks, or connect to multiple choice
of wifi networks, etc.

For machines with statically configured network interfaces,
NetworkManager is not necessary.

As it has been observed to become confused and observed
to malfunction when network links go up and down (it keeps
unnecessarily reconfiguring the ip address, etc), it can
be usefuil to disable it.

* list all network interfaces
<pre>
# /bin/ls -1 /sys/class/net/
enp0s31f6
lo
</pre>
* edit /etc/network/interfaces:
<pre>
rename enp0s31f6=eth0
auto eth0
iface eth0 inet static
address 142.90.120.94/19
gateway 142.90.100.18
</pre>
* statically configure systemd-resolved
** create /etc/systemd/resolved.conf.d/resolved.conf with this contents:
<pre>
[Resolve]
DNS=142.90.100.19
Domains=triumf.ca
</pre>
** systemctl restart systemd-resolved
** resolvectl
** systemd-analyze cat-config systemd/resolved.conf
* disable NetworkManager
<pre>
systemctl disable NetworkManager
</pre>
* reboot

== U-22, U-24 ifup-ko ==

Network configuration of modern linux is confused. There are at least 3 configuration methods, each with different shortcomings:
* the old ifup method is barely documented
* NetworkManager is well documented and tooled, but sometimes does strange things
* systemd-networkd is mysterious, and likely to do strange stuff, like all systemd stuff
* netplan is the latest method, configuration is simple but uses NetworkManager or systemd-networkd as backend.

This is a solution for a specific situation of fixed computer with one fixed wired interface and maybe one or more additional interfaces for fixed wired private networks.

Install /etc/ifup-ko, edit it with IP addresses of main and additional interfaces, let systemd run it in the right place in the boot sequence replacing NetworkManager and NetworkManager-wait-online .

As bonus, /etc/ifup-ko waits up to 10 seconds for the main interface link to come up. If this is not needed, comment it out from the script.

* prepare
<pre>
cd ~/git/scripts
git pull
cd ifup-ko
make install
</pre>
* confirm interface names
<pre>
systemctl start ifup-ko ### should finish immediately or after 10 seconds
systemctl status ifup-ko -n 1000 ### observe list of interfaces is correct, name of main interface is correct
</pre>
* edit /etc/ifup-ko
** add host IP address to the "ifconfig" line
** add gateway IP address to the "ip route add" line
* test
<pre>
systemctl start ifup-ko ### should finish immediately
systemctl status ifup-ko -n 1000 ### observe everything is configured as expected
</pre>
* cut-over
<pre>
systemctl disable networkd-dispatcher
systemctl disable NetworkManager
systemctl disable wpa_supplicant # if no Wifi or Wifi not in use
</pre>
* reboot

= Disable systemd-networkd =

On netbooted machines, systemd-networkd should be disabled - when "apt upgrade" runs
and needs to update and configure systemd, it will stop systemd-networkd, which
will stop the network, which will stop the NFS mounted root filesystem, which will stop
the machine.

<pre>
systemctl disable systemd-networkd.service
systemctl disable systemd-networkd.socket
systemctl mask systemd-networkd.service
systemctl mask systemd-networkd.socket
</pre>

= Configure ECC memory =

== Configure EDAC ==

* apt install edac-utils rasdaemon

=== Intel i3-2120 ===
<pre>
root@musr00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X9SCL/X9SCM
root@musr00:~# edac-ctl --status
edac-ctl: drivers not loaded.
</pre>

=== Intel E-2236 ===
<pre>
root@daq00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SCM-F
root@daq00:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq00:~# edac-util
edac-util: No errors to report.
root@daq00:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
</pre>
* check edac sysfs files (Intel)
<pre>
root@daq00:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_noinfo_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 max_location
-r--r--r-- 1 root root 4096 Jan 25 15:10 mc_name
drwxr-xr-x 2 root root 0 Jan 25 15:10 power
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank0
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank1
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank2
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank3
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank4
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank5
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank6
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank7
--w------- 1 root root 4096 Jan 25 15:10 reset_counters
-r--r--r-- 1 root root 4096 Jan 25 15:10 seconds_since_reset
-r--r--r-- 1 root root 4096 Jan 25 15:10 size_mb
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Jan 25 15:10 uevent
root@daq00:~#
</pre>

=== Intel E3-1270 v6 ===
<pre>
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --status
edac-ctl: drivers are loaded.
root@grsnis01:~# edac-util
edac-util: No errors to report.
root@grsnis01:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@grsnis01:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 max_location
-r--r--r-- 1 root root 4096 Feb 19 12:35 mc_name
drwxr-xr-x 2 root root 0 Feb 19 12:35 power
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank0
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank1
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank2
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank3
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank4
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank5
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank6
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank7
--w------- 1 root root 4096 Feb 19 12:35 reset_counters
-r--r--r-- 1 root root 4096 Feb 19 12:35 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 19 12:35 size_mb
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 19 12:35 uevent
root@grsnis01:~#
</pre>

=== Intel E3-1245 v6 ===

<pre>
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --status
edac-ctl: drivers are loaded.
[root@alphagdaq ~]# edac-util
edac-util: No errors to report.
[root@alphagdaq ~]# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
[root@alphagdaq ~]# ras-mc-ctl --layout
+-----------------------------------------------+
| mc0 |
| csrow0 | csrow1 | csrow2 | csrow3 |
----------+-----------------------------------------------+
channel1: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
----------+-----------------------------------------------+
[root@alphagdaq ~]# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#3channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#3channel#1 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#2channel#0 0 0
[root@alphagdaq ~]# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SSH-F
[root@alphagdaq ~]# ras-mc-ctl --summary
DBD::SQLite::db prepare failed: no such table: mc_event at /usr/sbin/ras-mc-ctl line 1129.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1130.
[root@alphagdaq ~]#
</pre>

=== AMD 3700X ===

(memory is non-ECC)

<pre>
root@daq13:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@daq13:~#
root@daq13:~#
root@daq13:~# edac-ctl --status
edac-ctl: drivers not loaded.
root@daq13:~# edac-util
edac-util: Error: No memory controller data found.
root@daq13:~# edac-util -s
edac-util: EDAC drivers loaded. No memory controllers found
root@daq13:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 2 root root 0 Jan 25 15:26 power
lrwxrwxrwx 1 root root 0 Jan 21 16:16 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Jan 21 16:16 uevent
</pre>

(memory is ECC)

<pre>
root@trinatdaq:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@trinatdaq:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@trinatdaq:~# edac-util
edac-util: No errors to report.
root@trinatdaq:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Dec 15 13:04 mc0
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
lrwxrwxrwx 1 root root 0 Dec 13 18:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dec 13 18:31 uevent
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_noinfo_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 max_location
-r--r--r-- 1 root root 4096 Dec 15 13:04 mc_name
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank4
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank5
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank6
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank7
--w------- 1 root root 4096 Dec 15 13:04 reset_counters
-rw-r--r-- 1 root root 4096 Dec 15 13:04 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Dec 15 13:04 seconds_since_reset
-r--r--r-- 1 root root 4096 Dec 15 13:04 size_mb
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Dec 15 13:04 uevent
root@trinatdaq:~#
</pre>

=== AMD 5000G ===

* no linux driver for AMD 5000-series "G" CPU
* no mention of ECC in the BIOS settings
* unclear status of ECC support in AMD documentation (sais only "pro" "G" CPUs have ECC)
* unclear status of ECC support in ASUS documentation (web page out of date)

=== AMD 5600X ===

<pre>
root@daq17:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-XE GAMING WIFI
root@daq17:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq17:~# edac-util
edac-util: No errors to report.
root@daq17:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@daq17:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Aug 19 19:27 mc0
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
lrwxrwxrwx 1 root root 0 May 10 10:11 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 May 10 10:11 uevent
root@daq17:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_noinfo_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 max_location
-r--r--r-- 1 root root 4096 Aug 19 19:27 mc_name
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank4
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank5
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank6
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank7
--w------- 1 root root 4096 Aug 19 19:27 reset_counters
-rw-r--r-- 1 root root 4096 Aug 19 19:27 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Aug 19 19:27 seconds_since_reset
-r--r--r-- 1 root root 4096 Aug 19 19:27 size_mb
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Aug 19 19:27 uevent
root@daq17:~#
</pre>

=== AMD 3955WX ===

<pre>
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --status
edac-ctl: drivers are loaded.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util
edac-util: No errors to report.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@alphasuperdaq:~/git/scripts/quotareport# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 19 root root 0 Dez 12 04:48 mc0
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
lrwxrwxrwx 1 root root 0 Dez 9 05:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dez 9 05:31 uevent
root@alphasuperdaq:~/git/scripts/quotareport#
root@alphasuperdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 max_location
-r--r--r-- 1 root root 4096 Feb 28 22:19 mc_name
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank0
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank1
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank10
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank11
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank12
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank13
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank14
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank15
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank2
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank3
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank4
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank5
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank6
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank7
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank8
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank9
--w------- 1 root root 4096 Feb 28 22:19 reset_counters
-rw-r--r-- 1 root root 4096 Feb 28 22:19 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Feb 28 22:19 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 28 22:19 size_mb
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 28 22:19 uevent
root@alphasuperdaq:~#
root@alphasuperdaq:~# ras-mc-ctl --layout
Use of uninitialized value $max_pos[3] in modulus (%) at /usr/sbin/ras-mc-ctl line 868.
Use of uninitialized value $d in numeric ge (>=) at /usr/sbin/ras-mc-ctl line 869.
Use of uninitialized value $d in sprintf at /usr/sbin/ras-mc-ctl line 872.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| mc0 |
| csrow0 | csrow1 |
| channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 | channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

0: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
root@alphasuperdaq:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#2 0 0
mc#0csrow#1channel#7 0 0
mc#0csrow#0channel#3 0 0
mc#0csrow#1channel#4 0 0
mc#0csrow#1channel#2 0 0
mc#0csrow#0channel#7 0 0
mc#0csrow#1channel#3 0 0
mc#0csrow#0channel#4 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#5 0 0
mc#0csrow#0channel#6 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#0channel#5 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#6 0 0
root@alphasuperdaq:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: ASUSTeK COMPUTER INC. model Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@alphasuperdaq:~#
</pre>

=== AMD 7700X ===

<pre>
root@dsfe05:~# apt install edac-utils
root@dsfe05:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro H13SAE-MF
root@dsfe05:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@dsfe05:~# edac-util
edac-util: No errors to report.
root@dsfe05:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@dsfe05:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 May 14 09:33 ce_count
-r--r--r-- 1 root root 4096 May 14 09:33 ce_noinfo_count
-r--r--r-- 1 root root 4096 May 14 09:33 max_location
-r--r--r-- 1 root root 4096 May 14 09:33 mc_name
drwxr-xr-x 2 root root 0 May 14 09:33 power
drwxr-xr-x 3 root root 0 May 14 09:33 rank4
drwxr-xr-x 3 root root 0 May 14 09:33 rank5
--w------- 1 root root 4096 May 14 09:33 reset_counters
-r--r--r-- 1 root root 4096 May 14 09:33 seconds_since_reset
-r--r--r-- 1 root root 4096 May 14 09:33 size_mb
-r--r--r-- 1 root root 4096 May 14 09:33 ue_count
-r--r--r-- 1 root root 4096 May 14 09:33 ue_noinfo_count
-rw-r--r-- 1 root root 4096 May 14 09:33 uevent
root@dsfe05:~#
</pre>

== Configure rasdaemon ==

<pre>
apt install rasdaemon
</pre>
<pre>
systemctl enable rasdaemon
systemctl restart rasdaemon
systemctl status rasdaemon
</pre>

<pre>
● rasdaemon.service - RAS daemon to log the RAS events
Loaded: loaded (/lib/systemd/system/rasdaemon.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-01-25 15:16:37 PST; 3min 5s ago
Main PID: 2477175 (rasdaemon)
Tasks: 1 (limit: 76958)
Memory: 17.1M
CGroup: /system.slice/rasdaemon.service
└─2477175 /usr/sbin/rasdaemon -f -r

Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Listening to events for cpus 0 to 11
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mc_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording aer_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording extlog_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mce_record events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording arm_event events
</pre>

== Get reports ==

* Intel 2x32GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --layout
+-------------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-------------------------+
channel1: | 16384 MB | 16384 MB |
channel0: | 16384 MB | 16384 MB |
----------+-------------------------+
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#0channel#1 0 0
root@daq00:~#
</pre>

* Intel 4x16GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#1 0 0
mc#0csrow#2channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#3channel#0 0 0
mc#0csrow#3channel#1 0 0
root@daq00:~#
root@daq00:~# ras-mc-ctl --layout
+-----------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-----------------------+
channel1: | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB |
----------+-----------------------+
root@daq00:~#
root@daq00:~#
root@daq00:~#
root@daq00:~# ras-mc-ctl --print-labels
ras-mc-ctl: Error: No dimm labels for Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@daq00:~#
</pre>

note: ubuntu LTS 22.04 DBD::SQLite::db error is not there.

* AMD 7700 2x32GB DDR5 ECC DIMMs

<pre>
root@dsfe05:~# systemctl status rasdaemon
● rasdaemon.service - RAS daemon to log the RAS events
Loaded: loaded (/lib/systemd/system/rasdaemon.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2024-05-14 09:36:43 PDT; 33ms ago
Process: 4088418 ExecStartPost=/usr/sbin/rasdaemon --enable (code=exited, status=0/SUCCESS)
Main PID: 4088417 (rasdaemon)
Tasks: 1 (limit: 37300)
Memory: 788.0K
CPU: 5ms
CGroup: /system.slice/rasdaemon.service
└─4088417 /usr/sbin/rasdaemon -f -r

May 14 09:36:43 dsfe05 rasdaemon[4088417]: ras:aer_event event enabled
May 14 09:36:43 dsfe05 rasdaemon[4088417]: Enabled event ras:aer_event
May 14 09:36:43 dsfe05 rasdaemon[4088417]: mce:mce_record event enabled
May 14 09:36:43 dsfe05 rasdaemon[4088417]: Enabled event mce:mce_record
May 14 09:36:43 dsfe05 rasdaemon[4088417]: ras:extlog_mem_event event enabled
May 14 09:36:43 dsfe05 rasdaemon[4088417]: Enabled event ras:extlog_mem_event
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording mc_event events
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording aer_event events
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording extlog_event events
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording mce_record events
root@dsfe05:~# ras-mc-ctl --layout
Use of uninitialized value $max_pos[3] in modulus (%) at /usr/sbin/ras-mc-ctl line 907.
Use of uninitialized value $d in numeric ge (>=) at /usr/sbin/ras-mc-ctl line 908.
Use of uninitialized value $d in sprintf at /usr/sbin/ras-mc-ctl line 911.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
+-----------------------------------------------------------------------------------------------+
| mc0 |
| csrow0 | csrow1 | csrow2 | csrow3 |
| channel0 | channel1 | channel0 | channel1 | channel0 | channel1 | channel0 | channel1 |
----+-----------------------------------------------------------------------------------------------+

0: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB |
----+-----------------------------------------------------------------------------------------------+
root@dsfe05:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#2channel#1 0 0
mc#0csrow#2channel#0 0 0
root@dsfe05:~# ras-mc-ctl --print-labels
ras-mc-ctl: Error: No dimm labels for Supermicro model H13SAE-MF
root@dsfe05:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model H13SAE-MF
root@dsfe05:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

No MCE errors.
root@dsfe05:~#
</pre>

= sensors =

== VME CPU V7865 ==

add to /etc/rc.local

<pre>
modprobe coretemp
modprobe lm75
modprobe lm90
modprobe max1668
</pre>

available sensors:

<pre>
root@lxdaq23:~# sensors
max6657-i2c-0-4c
Adapter: SMBus I801 adapter at 0400
temp1: +29.1°C (low = -55.0°C, high = +105.0°C)
(crit = +105.0°C, hyst = +95.0°C)
temp2: +31.5°C (low = -55.0°C, high = +105.0°C)
(crit = +105.0°C, hyst = +95.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Core 0: +25.0°C (crit = +100.0°C)
Core 1: +25.0°C (crit = +100.0°C)

max1805-i2c-0-18
Adapter: SMBus I801 adapter at 0400
temp1: +35.0°C (low = -55.0°C, high = +127.0°C)
temp2: +63.0°C (low = -55.0°C, high = +127.0°C)
temp3: FAULT (low = -55.0°C, high = +127.0°C) ALARM (HIGH)

lm75-i2c-0-48
Adapter: SMBus I801 adapter at 0400
temp1: +34.5°C (high = +80.0°C, hyst = +75.0°C)

root@lxdaq23:~#
</pre>

== ASUS P7P55D EVO ==

* BIOS version 2101

<pre>
root@iris01:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Core 0: +34.0°C (high = +83.0°C, crit = +99.0°C)
Core 1: +37.0°C (high = +83.0°C, crit = +99.0°C)
Core 2: +38.0°C (high = +83.0°C, crit = +99.0°C)
Core 3: +35.0°C (high = +83.0°C, crit = +99.0°C)

nouveau-pci-0100
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.05 V)
temp1: +46.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

atk0110-acpi-0
Adapter: ACPI interface
Vcore Voltage: 864.00 mV (min = +0.80 V, max = +1.60 V)
+3.3V Voltage: 3.38 V (min = +2.97 V, max = +3.63 V)
+5V Voltage: 5.04 V (min = +4.50 V, max = +5.50 V)
+12V Voltage: 12.15 V (min = +10.20 V, max = +13.80 V)
CPU Fan Speed: 968 RPM (min = 600 RPM, max = 7200 RPM)
Chassis1 Fan Speed: 1288 RPM (min = 600 RPM, max = 7200 RPM)
Chassis2 Fan Speed: 1316 RPM (min = 600 RPM, max = 7200 RPM)
Power Fan Speed: 0 RPM (min = 0 RPM, max = 7200 RPM)
CPU Temperature: +34.0°C (high = +45.0°C, crit = +45.5°C)
MB Temperature: +30.0°C (high = +45.0°C, crit = +46.0°C)

root@iris01:~#
</pre>

== ASUS Z97-WS ==

* BIOS version 2704
* load sensors drivers
<pre>
echo modprobe coretemp >> /etc/rc.local
echo modprobe nct6775 >> /etc/rc.local
</pre>
* in /boot/grub/grub.cfg, add: GRUB_CMDLINE_LINUX_DEFAULT="acpi_enforce_resources=no"
* update grub and reboot: grub-mkconfig -o /boot/grub/grub.cfg

<pre>
root@isdaq08:~# sensors
acpitz-acpi-0
Adapter: ACPI interface
temp1: +27.8°C
temp2: +29.8°C

nct6791-isa-0290
Adapter: ISA adapter
Vcore: 888.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
AVCC: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
+3.3V: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.99 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 0.00 V (min = +0.00 V, max = +0.00 V)
3VSB: 3.44 V (min = +0.00 V, max = +0.00 V) ALARM
Vbat: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 0.00 V (min = +0.00 V, max = +0.00 V)
in11: 840.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in12: 0.00 V (min = +0.00 V, max = +0.00 V)
in13: 0.00 V (min = +0.00 V, max = +0.00 V) ALARM
in14: 0.00 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1041 RPM (min = 0 RPM)
fan2: 1040 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan6: 0 RPM (min = 0 RPM)
SYSTIN: +34.0°C (high = +0.0°C, hyst = +0.0°C) ALARM sensor = thermistor
CPUTIN: +41.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: -128.0°C sensor = thermistor
AUXTIN1: -128.0°C sensor = thermistor
AUXTIN2: +35.0°C sensor = thermistor
AUXTIN3: +127.0°C sensor = thermistor
PECI Agent 0: +41.0°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
PCH_DIM0_TEMP: +0.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +42.0°C (high = +80.0°C, crit = +100.0°C)
Core 0: +42.0°C (high = +80.0°C, crit = +100.0°C)
Core 1: +40.0°C (high = +80.0°C, crit = +100.0°C)
Core 2: +39.0°C (high = +80.0°C, crit = +100.0°C)
Core 3: +39.0°C (high = +80.0°C, crit = +100.0°C)

root@isdaq08:~#
</pre>

== ASUS Z170-DELUXE ==

* BIOS version 3801
* load sensors drivers
<pre>
echo modprobe coretemp >> /etc/rc.local
echo modprobe jc42 >> /etc/rc.local
echo modprobe lm92 >> /etc/rc.local
echo modprobe nct6775 >> /etc/rc.local
</pre>
* in /etc/default/grub, add: GRUB_CMDLINE_LINUX_DEFAULT="acpi_enforce_resources=no"
* update grub and reboot: grub-mkconfig -o /boot/grub/grub.cfg

<pre>
root@iris00:~# sensors
nct6793-isa-0290
Adapter: ISA adapter
in0: 600.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 144.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in6: 0.00 V (min = +0.00 V, max = +0.00 V)
in7: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.14 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1000.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in10: 600.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 592.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 968.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1370 RPM (min = 0 RPM)
fan2: 1437 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan6: 0 RPM (min = 0 RPM)
SYSTIN: +32.0°C (high = +98.0°C, hyst = +95.0°C) sensor = thermistor
CPUTIN: +42.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: -128.0°C sensor = thermistor
AUXTIN1: +50.0°C sensor = thermistor
AUXTIN2: +22.0°C sensor = thermistor
AUXTIN3: +28.0°C sensor = thermistor
PECI Agent 0: +50.0°C (high = +98.0°C, hyst = +95.0°C)
(crit = +100.0°C)
PECI Agent 0 Calibration: +42.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
TSI2_TEMP: +3892314.0°C
TSI3_TEMP: +3892314.0°C
TSI4_TEMP: +3892314.0°C
TSI5_TEMP: +3892314.0°C
TSI6_TEMP: +3892314.0°C
TSI7_TEMP: +3892314.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

jc42-i2c-0-1a
Adapter: SMBus I801 adapter at f040
temp1: +36.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

jc42-i2c-0-18
Adapter: SMBus I801 adapter at f040
temp1: +34.8°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

jc42-i2c-0-1b
Adapter: SMBus I801 adapter at f040
temp1: +35.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

jc42-i2c-0-19
Adapter: SMBus I801 adapter at f040
temp1: +36.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +52.0°C (high = +84.0°C, crit = +100.0°C)
Core 0: +52.0°C (high = +84.0°C, crit = +100.0°C)
Core 1: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 2: +48.0°C (high = +84.0°C, crit = +100.0°C)
Core 3: +47.0°C (high = +84.0°C, crit = +100.0°C)

root@iris00:~#
</pre>

== ASUS Z390M-PRO GAMING (WI-FI) ==

* BIOS 3006
* load sensors drivers
<pre>
echo modprobe coretemp >> /etc/rc.local
echo modprobe nct6775 >> /etc/rc.local
</pre>

<pre>
root@daq18:~# sensors
nct6798-isa-0290
Adapter: ISA adapter
in0: 696.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.42 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 208.00 mV (min = +0.00 V, max = +0.00 V)
in6: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.42 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.17 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.07 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.36 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.33 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in14: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 0 RPM (min = 0 RPM)
fan2: 1131 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 1006 RPM (min = 0 RPM)
fan6: 0 RPM (min = 0 RPM)
fan7: 0 RPM (min = 0 RPM)
SYSTIN: +32.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
CPUTIN: +29.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
AUXTIN0: +25.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
AUXTIN1: +7.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
AUXTIN2: +8.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
AUXTIN3: +24.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
AUXTIN4: +83.0°C (high = +80.0°C, hyst = +75.0°C) ALARM
(crit = +100.0°C)
PECI Agent 0 Calibration: +29.0°C (high = +80.0°C, hyst = +75.0°C)
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +39.0°C (high = +82.0°C, crit = +100.0°C)
Core 0: +39.0°C (high = +82.0°C, crit = +100.0°C)
Core 1: +33.0°C (high = +82.0°C, crit = +100.0°C)
Core 2: +32.0°C (high = +82.0°C, crit = +100.0°C)
Core 3: +31.0°C (high = +82.0°C, crit = +100.0°C)
Core 4: +31.0°C (high = +82.0°C, crit = +100.0°C)
Core 5: +30.0°C (high = +82.0°C, crit = +100.0°C)

acpitz-acpi-0
Adapter: ACPI interface
temp1: +27.8°C

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: +28.0°C

root@daq18:~#
</pre>

== ASUS H110M-A/M.2 ==

* BIOS version 4202
* echo modprobe coretemp >> /etc/rc.local
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@midpol:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +33.0°C (high = +80.0°C, crit = +100.0°C)
Core 0: +33.0°C (high = +80.0°C, crit = +100.0°C)
Core 1: +30.0°C (high = +80.0°C, crit = +100.0°C)

acpitz-acpi-0
Adapter: ACPI interface
temp1: +27.8°C (crit = +119.0°C)
temp2: +29.8°C (crit = +119.0°C)

nct6793-isa-0290
Adapter: ISA adapter
in0: 368.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 152.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in6: 928.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.14 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1000.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in10: 152.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in11: 128.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in12: 136.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in13: 120.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 136.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1004 RPM (min = 0 RPM)
fan2: 1143 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan6: 0 RPM (min = 0 RPM)
SYSTIN: +118.0°C (high = +98.0°C, hyst = +95.0°C) sensor = thermistor
CPUTIN: +29.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +30.0°C sensor = thermistor
AUXTIN1: +112.0°C sensor = thermistor
AUXTIN2: +111.0°C sensor = thermistor
AUXTIN3: +110.0°C sensor = thermistor
PECI Agent 0: +31.0°C (high = +98.0°C, hyst = +95.0°C)
(crit = +100.0°C)
PECI Agent 0 Calibration: +36.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
TSI2_TEMP: +3892314.0°C
TSI3_TEMP: +3892314.0°C
TSI4_TEMP: +3892314.0°C
TSI5_TEMP: +3892314.0°C
TSI6_TEMP: +3892314.0°C
TSI7_TEMP: +3892314.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

root@midpol:~#
</pre>

== ASUS P9X79 WS ==

* https://www.asus.com/supportonly/P9X79%20WS/HelpDesk_Manual/
* BIOS version 4802
* modprobe nct6775
* modprobe coretemp

<pre>
root@daq14:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 0: +29.0°C (high = +82.0°C, crit = +100.0°C)
Core 1: +24.0°C (high = +82.0°C, crit = +100.0°C)
Core 2: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +82.0°C, crit = +100.0°C)

nouveau-pci-0200
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.00 V)
temp1: +39.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

nct6776-isa-0290
Adapter: ISA adapter
Vcore: 1.04 V (min = +0.00 V, max = +1.74 V)
in1: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
AVCC: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
+3.3V: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 2.04 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 904.00 mV (min = +0.00 V, max = +0.00 V) ALARM
3VSB: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
Vbat: 3.30 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1265 RPM (min = 0 RPM)
fan2: 1909 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +34.0°C (high = +0.0°C, hyst = +0.0°C) ALARM sensor = thermistor
CPUTIN: +58.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermal diode
AUXTIN: +31.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
PECI Agent 0: +31.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +96.0°C)
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

root@daq14:~#
</pre>

== ASUS TUF GAMING B550M-PLUS WIFI II ==

* BIOS 2803, 2806
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@midm9a:~# sensors
nct6798-isa-0290
Adapter: ISA adapter
in0: 488.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.82 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in14: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 0 RPM (min = 0 RPM)
fan2: 760 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan7: 1264 RPM (min = 0 RPM)
SYSTIN: +25.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +22.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +95.0°C sensor = thermistor
AUXTIN1: +25.0°C sensor = thermistor
AUXTIN2: +25.0°C sensor = thermistor
AUXTIN3: +25.0°C sensor = thermistor
PECI Agent 0 Calibration: +23.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +32.4°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

amdgpu-pci-0800
Adapter: PCI adapter
vddgfx: 1.45 V
vddnb: 993.00 mV
edge: +28.0°C
PPT: 20.00 W

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +33.4°C

root@midm9a:~#
</pre>

== ASUS ASUS ROG STRIX B550-XE GAMING WIFI ==

* BIOS 2423, 2604
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@daq13:~# sensors
nct6798-isa-0290
Adapter: ISA adapter
in0: 344.00 mV (min = +0.00 V, max = +1.74 V)
in1: 992.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in6: 216.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.30 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.81 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in11: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 280.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 845 RPM (min = 0 RPM)
fan2: 998 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +28.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +27.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +94.0°C sensor = thermistor
AUXTIN1: +28.0°C sensor = thermistor
AUXTIN2: +28.0°C sensor = thermistor
AUXTIN3: +97.0°C sensor = thermistor
PECI Agent 0 Calibration: +27.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +33.6°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

amdgpu-pci-0600
Adapter: PCI adapter
vddgfx: 1.45 V
vddnb: 999.00 mV
edge: +29.0°C
PPT: 14.00 W

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: +30.0°C

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +33.9°C

root@daq13:~#
</pre>

== ASUS ASUS ROG STRIX B550-E GAMING ==

* bios 2803
* echo modprobe jc42 >> /etc/rc.local
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@daq17:~# sensors
jc42-i2c-1-1b
Adapter: SMBus PIIX4 adapter port 0 at 0b00
temp1: +25.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: +28.0°C

nouveau-pci-0800
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.00 V)
temp1: +34.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

nct6798-isa-0290
Adapter: ISA adapter
in0: 288.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 224.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.31 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.79 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 280.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 843 RPM (min = 0 RPM)
fan2: 629 RPM (min = 0 RPM)
fan3: 746 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +22.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +25.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +93.0°C sensor = thermistor
AUXTIN1: +22.0°C sensor = thermistor
AUXTIN2: +22.0°C sensor = thermistor
AUXTIN3: +96.0°C sensor = thermistor
PECI Agent 0 Calibration: +25.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +27.6°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

jc42-i2c-1-1a
Adapter: SMBus PIIX4 adapter port 0 at 0b00
temp1: +23.2°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

asusec-isa-0000
Adapter: ISA adapter
CPU_Opt: 0 RPM
Chipset: +34.0°C
CPU: +25.0°C
Motherboard: +22.0°C
T_Sensor: -40.0°C
VRM: +31.0°C

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +28.0°C
Tccd1: +27.5°C

root@daq17:~#
</pre>

== ASUS PRIME B650-PLUS ==

* BIOS 1811
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@dsdaqgw:~# sensors
amdgpu-pci-0b00
Adapter: PCI adapter
vddgfx: 930.00 mV
vddnb: 1.19 V
edge: +38.0°C
PPT: 25.10 W

nct6799-isa-0290
Adapter: ISA adapter
in0: 920.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.04 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 320.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.28 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.28 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.10 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.04 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 416.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 328.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 0 RPM (min = 0 RPM)
fan2: 1253 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan7: 0 RPM (min = 0 RPM)
SYSTIN: +33.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +35.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +78.0°C sensor = thermistor
AUXTIN1: +11.0°C sensor = thermistor
AUXTIN2: +20.0°C sensor = thermistor
AUXTIN3: +82.0°C sensor = thermistor
PECI Agent 0 Calibration: +35.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +42.6°C
intrusion0: ALARM
intrusion1: OK
beep_enable: disabled

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +42.6°C
Tccd1: +36.4°C

root@dsdaqgw:~#
</pre>

= Enable CPU turbo mode =

* Intel CPU has a nominal CPU frequency (i.e. 3.4GHz) and a turbo-boost CPU frequency (i.e. 4.0GHz). Here we will enable this turbo-boost mode.
* Find out CPU capability
<pre>
root@daq01:~# lscpu | grep Hz
Model name: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
CPU MHz: 3965.803
CPU max MHz: 4000.0000
CPU min MHz: 800.0000
root@daq01:~#
</pre>
* Look up this CPU in the Intel ARK database - google for the CPU model name, i.e.
https://ark.intel.com/content/www/us/en/ark/products/88196/intel-core-i7-6700-processor-8m-cache-up-to-4-00-ghz.html
* Find current frequency settings:
<pre>
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "powersave" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 2.72 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
root@daq01:~#
</pre>
* Note the following:
** current governor is "powersave"
** "performance" governor is available
** "boost state support" is supported and active.
* Confirm CPU frequency governor:
<pre>
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
powersave
powersave
powersave
powersave
powersave
powersave
powersave
powersave
root@daq01:~#
</pre>
* Change governor to "performance":
<pre>
root@daq01:~# cpupower frequency-set --governor performance
Setting cpu: 0
Setting cpu: 1
Setting cpu: 2
Setting cpu: 3
Setting cpu: 4
Setting cpu: 5
Setting cpu: 6
Setting cpu: 7
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
performance
performance
performance
performance
performance
performance
performance
performance
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "performance" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 3.93 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
</pre>
* monitor CPU frequency:
<pre>
root@daq01:~# cpupower monitor
| Nehalem || Mperf || Idle_Stats
CPU| C3 | C6 | PC3 | PC6 || C0 | Cx | Freq || POLL | C1 | C1E | C3 | C6 | C7s | C8
0| 0.00| 0.00| 0.00| 0.00|| 88.80| 11.20| 3973|| 0.00| 0.00| 0.01| 0.02| 0.31| 0.00| 4.25
4| 0.00| 0.00| 0.00| 0.00|| 4.70| 95.30| 3945|| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 95.03
1| 0.73| 3.70| 0.00| 0.00|| 4.52| 95.48| 3864|| 0.00| 0.01| 1.19| 0.44| 2.82| 0.00| 90.23
5| 0.73| 3.70| 0.00| 0.00|| 0.37| 99.63| 3807|| 0.00| 0.00| 0.03| 0.09| 1.70| 0.00| 97.64
2| 2.28| 12.86| 0.00| 0.00|| 1.41| 98.59| 3829|| 0.00| 0.86| 3.17| 0.46| 7.70| 0.00| 85.87
6| 2.28| 12.86| 0.00| 0.00|| 2.88| 97.12| 3856|| 0.00| 0.11| 4.56| 2.15| 10.31| 0.00| 78.99
3| 1.33| 4.81| 0.00| 0.00|| 0.99| 99.01| 3804|| 0.00| 0.49| 0.79| 0.01| 1.03| 0.00| 96.12
7| 1.34| 4.81| 0.00| 0.00|| 1.26| 98.74| 3818|| 0.00| 0.01| 2.32| 0.47| 5.02| 0.00| 90.06
root@daq01:~#
</pre>
* check that the CPU is not overheating:
<pre>
root@daq01:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 1: +38.0°C (high = +84.0°C, crit = +100.0°C)
Core 2: +34.0°C (high = +84.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +84.0°C, crit = +100.0°C)
</pre>
* congratulations, we are running at 4 GHz now!

= Setup ubuntu as gateway to private network =

See also:
* https://daq.triumf.ca/DaqWiki/index.php/VME-CPU#Setup_the_boot_host_computer_.28el7.29
* http://www.triumf.info/wiki/DAQwiki/index.php/Dhcpd_on_eth1

== Steps to do ==

!!! UPDATED 16feb2024 Ubuntu-22.04.03 !!!

* assign network numbers to the private network, i.e. 192.168.1.x, 192.168.2.x, etc
* (on the gateway machine, each private network interface has to have a different network number)
* (each network interface can have multiple networks attached, via VLANs or via eth0:0, eth0:1 constructs)
* assign IP addresses on the private network, save them in /etc/hosts i.e. "hvps 192.168.1.10"
* (for simplicity, assign 192.168.1.1 to the gateway machine itself)
* (IP addresses 192.168.1.0 and 192.168.1.255 are "special", do not use them)
* setup DNS server (dnsmasq) to serve contents of /etc/hosts via DNS (otherwise, many programs will see inconsistent name to IP address mapping)
* setup DHCP server (dnsmasq) to give out the IP addresses
* setup TFTP server (dnsmasq), pxelinux and NFS for diskless booting
* setup time server (chronyd) to provide common time to all devices
* setup NAT so machines on private network can access the internet (to get OS updates, etc)
* setup NIS and NFS so machines on the private network can use common home directories
* setup rsync backup of machines on the private network

== setup hosts ==

* edit /etc/hosts
<pre>
192.168.1.101 dsfe01
... and so forth
</pre>

== setup dns and dhcp ==

!!! updated 16feb2024 for Ubuntu 22.04.3 !!!

!!! note: stock systemd-resolved remains, is configured to forward queries to dnsmasq, configured to forward queries to TRIUMF DNS !!!

!!! note: per authors of systemd, bare hostnames are not permitted, a DNS domain name must always be used. DNS domain name "dsdaq" is used in this example !!!

* apt install dnsmasq
* ensure dnsmasq starts after all interfaces are up (Ubuntu-22)
<pre>
mkdir /etc/systemd/system/dnsmasq.service.d
echo -e "[Unit]\nAfter=network-online.target\n" > /etc/systemd/system/dnsmasq.service.d/local.conf
</pre>
* edit /etc/dnsmasq.conf
<pre>
# /etc/dnsmasq.conf
# DNS settings
#port=0 # disable DNS function
port=53 # enable DNS function
bind-interfaces # do not collide with systemd-resolved, we use 127.0.0.1:53, they use 127.0.0.53:53
domain-needed
bogus-priv
no-resolv
#log-queries # log DNS quesries

# TRIUMF DNS settings

server=142.90.100.19
expand-hosts
domain=dsdaq
local=/dsdaq/
localmx # do not forward MX queries to TRIUMF

# DHCP settings
interface=enp1s0f0 # VX network 192.168.0.x
#interface=missing # FEP and TSP network 192.168.1.x
interface=enp1s0f1 # controls network 192.168.2.x
#dhcp-range=192.168.1.50,192.168.1.150,infinite
dhcp-range=192.168.0.0,static
dhcp-range=192.168.2.0,static
log-dhcp # log DHCP queries
#quiet-dhcp
dhcp-ignore=tag:!known
#dhcp-boot=pxelinux.0

dhcp-option=option:dns-server,192.168.0.248
dhcp-option=option:ntp-server,192.168.0.248

# TFTP settings

enable-tftp
tftp-root=/tftpboot
</pre>
* #mkdir /tftpboot ### per tftp-root (if no ZFS)
* zfs create -o mountpoint=/tftpboot rpool/tftpboot ### (if root is ZFS)
* create resolved-dsdaq.conf with main IP address of dnsmasq
<pre>
[Resolve]
DNS=192.168.0.248
Domains=dsdaq triumf.ca
</pre>
* mkdir -p /etc/systemd/resolved.conf.d/
* /bin/rm -f /etc/systemd/resolved.conf.d/*.conf
* cp resolved-dsdaq.conf /etc/systemd/resolved.conf.d/
* systemctl stop systemd-resolved.service
* systemctl disable systemd-resolved.service
* systemctl enable dnsmasq
* systemctl restart dnsmasq
* try to "ping" or "host" some names from /etc/hosts, it should work
* try to ping daq00, daq00.triumf.ca, all should work
* resolved-dsdaq.conf goes into /etc/systemd/resolved.conf.d/ of all machines on the private network
* if not using systemd-resolved, edit /etc/resolv.conf

== setup chronyd ==

* enable ntp server:
* disable systemd-timesyncd, configure and enable chronyd per instructions above
* create dsdaq.conf
<pre>
# chrony config for dsdaq server

#allow 192.168.0.0
#allow 192.168.1.0
#allow 192.168.2.0
allow all

# end
</pre>
* cp dsdaq.conf /etc/chrony/conf.d/
* systemctl restart chronyd
* chronyc tracking ### wait until time is synchronized (a few seconds)
* create dsdaq.sources # use hostname or IP address of chronyd server
<pre>
# Put this file in /etc/chrony/sources.d
# systemctl restart chrony
# chronyc sources
# chronyc tracking
server dsdaqgw iburst prefer
# end
</pre>
* dsdaq.sources goes to /etc/chrony/sources.d of all machines on the private network

== setup diskless network booting ==

=== setup pxelinux for legacy pxe boot ===

* add bits in dnsmasq.conf
<pre>
dhcp-host=ac:1f:6b:9e:7f:4a,dsfe01,infinite
dhcp-boot=pxelinux.0
dhcp-option=17,"192.168.0.251:/nfsroot/%s,vers=3,tcp"
</pre>
* setup pxelinux for Ubuntu-18
<pre>
cd ~
wget https://www.kernel.org/pub/linux/utils/boot/syslinux/4.xx/syslinux-4.03.tar.bz2
tar xjvf syslinux-4.03.tar.bz2
cd syslinux-4.03
cp -pv ./core/pxelinux.0 ./com32/hdt/hdt.c32 ./memdisk/memdisk ./com32/menu/menu.c32 /zssd/tftpboot/
</pre>
* cd /zssd/tftpboot
<pre>
wget http://ladd00.triumf.ca/tftpboot/memtest86+-4.20.iso.zip
wget http://ladd00.triumf.ca/tftpboot/memtest86+-5.01.iso.gz
wget http://ladd00.triumf.ca/tftpboot/modules.alias
wget http://ladd00.triumf.ca/tftpboot/modules.pcimap
wget http://ladd00.triumf.ca/tftpboot/pci.ids
</pre>
* mkdir pxelinux.cfg
* emacs -nw pxelinux.cfg/default
<pre>
default menu.c32
prompt 0

menu title Welcome to the DSVSLICE PXE boot menu

timeout 50

label hdt
kernel hdt.c32

label memtest86+-5.01
kernel memdisk iso initrd=memtest86+-5.01.iso.gz

label memtest86+-4.20
kernel memdisk iso initrd=memtest86+-4.20.iso.zip

label vmlinuz-5.3.0-26-generic
menu default
kernel vmlinuz-5.3.0-26-generic
append initrd=initrd.img-5.3.0-26-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.1.1:/zssd/nfsroot/dsfe01 toram ip=dhcp panic=60 BOOTIF=enp1s0f0

#end
</pre>

=== setup pxelinux for efi pxe boot ===

* https://c-nergy.be/blog/?p=13808
* add dnsmasq.conf bits. note: to use dhcp root-path, see the "nfsroot=auto" patch below and make sure to use the "dhcp-option-force" command (mkinitramfs dhcp client does not ask for root-path, we have to force-feed it).
<pre>
# uefi pxe

dhcp-boot=tag:uefipxe,uefi/syslinux.efi
dhcp-option-force=tag:fe01,option:root-path,192.168.0.248:/nfsroot/fe01

# VX network 192.168.0.x

dhcp-host=40:a6:b7:c1:d9:c5,fe01,infinite,set:uefipxe,set:fe01
</pre>
* apt install syslinux pxelinux syslinux-common syslinux-efi syslinux-utils
<pre>
mkdir /tftpboot/uefi
cp /usr/lib/SYSLINUX.EFI/efi64/syslinux.efi /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/ldlinux.e64 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/menu.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/hdt.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libutil.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libmenu.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libcom32.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libgpl.c32 /tftpboot/uefi/
</pre>
* try to boot, it should bomb with "cannot load pxelinux.cfg/default"
* mkdir /tftpboot/uefi/pxelinux.cfg
* create /tftpboot/uefi/pxelinux.cfg/default, note nfsroot path is hardwired, note "http:" is used to load vmlinuz and initrd files (because tftp is super slow)
<pre>
default menu.c32
prompt 0

menu title Welcome to the DSDAQGW UEFI PXE boot menu

timeout 50

label vmlinuz-6.5.0-17-generic
kernel http://192.168.0.248:8088/uefi/vmlinuz-6.5.0-17-generic
append initrd=http://192.168.0.248:8088/uefi/initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=auto rw ip=dhcp panic=60

# append initrd=http://192.168.0.248:8088/uefi/initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.0.248:/nfsroot/fe01 rw ip=dhcp panic=60

# append initrd=initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.0.248:/nfsroot/fe01 rw ip=dhcp panic=60
# append initrd=initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=auto ip=dhcp rw panic=60

#end
</pre>
* try to boot, it will bomb with "cannot load http://...."
* install mini_httpd on port 8088, see https://acme.com/software/mini_httpd/
<pre>
apt install mini-httpd
emacs -nw /etc/default/mini-httpd # set "START=1"
emacs -nw /etc/mini-httpd.conf # set "host=192.168.0.248", "port=8088", "data_dir=/tftpboot"
mkdir /etc/systemd/system/mini-httpd.service.d
echo -e "[Unit]\nAfter=network-online.target\n" > /etc/systemd/system/mini-httpd.service.d/local.conf
systemctl enable mini-httpd
systemctl restart mini-httpd
systemctl status mini-httpd
wget http://192.168.0.248:8088/uefi/syslinux.efi
tail -100 /var/log/mini_httpd.log
</pre>
* fix U-22 initramfs bug for "nfsroot=auto", otherwise, "nfsroot=" has to be different for each machine and you have to have separate pxelinux config files for each machine
** emacs -nw /usr/lib/initramfs-tools/etc/dhcp/dhclient-enter-hooks.d/config
** add "echo ROOTPATH=..." if it is missing
<pre>
echo "ROOTSERVER='${new_routers%% *}'"
echo "ROOTPATH='$new_root_path'"
echo "HOSTNAME='$new_host_name'"
</pre>
* fix U-24 initramfs bug for "nfsroot=auto", otherwise, "nfsroot=" has to be different for each machine and you have to have separate pxelinux config files for each machine
** emacs -nw /usr/share/initramfs-tools/dhcpcd-hooks/70-net-conf
** add "ROOTPATH=..." if it is missing
<pre>
DNSDOMAIN='${new_domain_name-}'
ROOTSERVER='${new_routers-}'
ROOTPATH='${new_root_path-}'
filename='${new_filename-}'
DHCPLEASETIME='${new_dhcp_lease_time-}'
</pre>
** regenerate initramfs (be careful you generate it for the right kernel!)
** see https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/2054482
<pre>
mkinitramfs 6.5.0-18-generic
mkinitramfs 6.8.0-51-generic -o /boot/initrd.img-6.8.0-51-generic
</pre>
* copy linux kernel and initrd
<pre>
cp /boot/vmlinuz-6.5.0-18-generic /tftpboot/uefi/
cp /boot/initrd.img-6.5.0-18-generic /tftpboot/uefi/
chmod a+r /tftpboot/uefi/*
</pre>
* try to boot, should bomb with messages about "trying to mount root filesystem"
* tail /var/log/syslog
<pre>
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 vendor class: PXEClient:Arch:00007:UNDI:003016
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 1:netmask, 2:time-offset, 3:router, 4, 5,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 6:dns-server, 12:hostname, 13:boot-file-size,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 15:domain-name, 17:root-path, 18:extension-path,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 22:max-datagram-reassembly, 23:default-ttl,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 28:broadcast, 40:nis-domain, 41:nis-server,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 42:ntp-server, 43:vendor-encap, 50:requested-address,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 51:lease-time, 54:server-identifier, 58:T1,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 59:T2, 60:vendor-class, 66:tftp-server, 67:bootfile-name,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 97:client-machine-id, 128, 129, 130, 131,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 132, 133, 134, 135
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 next server: 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 broadcast response
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 1 option: 53 message-type 2
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 18 option: 67 bootfile-name uefi/syslinux.efi
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 12 hostname fe01
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 vendor class: PXEClient:Arch:00007:UNDI:003016
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 1:netmask, 2:time-offset, 3:router, 4, 5,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 6:dns-server, 12:hostname, 13:boot-file-size,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 15:domain-name, 17:root-path, 18:extension-path,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 22:max-datagram-reassembly, 23:default-ttl,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 28:broadcast, 40:nis-domain, 41:nis-server,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 42:ntp-server, 43:vendor-encap, 50:requested-address,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 51:lease-time, 54:server-identifier, 58:T1,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 59:T2, 60:vendor-class, 66:tftp-server, 67:bootfile-name,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 97:client-machine-id, 128, 129, 130, 131,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 132, 133, 134, 135
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 next server: 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 broadcast response
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 1 option: 53 message-type 5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 18 option: 67 bootfile-name uefi/syslinux.efi
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 12 hostname fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: error 8 User aborted the transfer received from 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/syslinux.efi to 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/syslinux.efi to 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 requested options: 1:netmask, 3:router, 6:dns-server
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 bootfile name: uefi/syslinux.efi
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 next server: 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 broadcast response
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 1 option: 53 message-type 2
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 requested options: 1:netmask, 3:router, 6:dns-server
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 bootfile name: uefi/syslinux.efi
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 next server: 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 broadcast response
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 1 option: 53 message-type 5
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/ldlinux.e64 to 192.168.0.110
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/01-40-a6-b7-c1-d9-c5 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8006E not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8006 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A800 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A80 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/pxelinux.cfg/default to 192.168.0.110
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/menu.c32 to 192.168.0.110
Feb 16 20:43:10 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/libutil.c32 to 192.168.0.110
Feb 16 20:43:10 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/pxelinux.cfg/default to 192.168.0.110
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 client provides name: dsdaqgw.triumf.ca
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 1:netmask, 28:broadcast, 2:time-offset, 3:router,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 15:domain-name, 6:dns-server, 119:domain-search,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 12:hostname, 44:netbios-ns, 47:netbios-scope,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 26:mtu, 121:classless-static-route, 42:ntp-server
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 bootfile name: uefi/syslinux.efi
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 next server: 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 1 option: 53 message-type 2
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 51 lease-time infinite
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 client provides name: dsdaqgw.triumf.ca
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 1:netmask, 28:broadcast, 2:time-offset, 3:router,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 15:domain-name, 6:dns-server, 119:domain-search,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 12:hostname, 44:netbios-ns, 47:netbios-scope,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 26:mtu, 121:classless-static-route, 42:ntp-server
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 bootfile name: uefi/syslinux.efi
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 next server: 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 1 option: 53 message-type 5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 51 lease-time infinite
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 12 hostname fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw rpc.mountd[3350210]: authenticated mount request from 192.168.0.110:981 for /nfsroot/fe01 (/nfsroot/fe01)
Feb 16 20:45:07 dsdaqgw rpc.mountd[3350210]: authenticated unmount request from 192.168.0.110:859 for /nfsroot/fe01/tmp/autoDY4k5u (/nfsroot/fe01)
</pre>
* tail /var/log/mini_httpd.log
<pre>
192.168.0.110 - - [16/Feb/2024:20:43:15 -0800] "GET /uefi/vmlinuz-6.5.0-17-generic HTTP/1.0" 200 14227944 "" "Syslinux/6.04"
192.168.0.110 - - [16/Feb/2024:20:43:24 -0800] "GET /uefi/initrd.img-6.5.0-17-generic HTTP/1.0" 200 137824833 "" "Syslinux/6.04"
</pre>

=== setup efi http boot ===

https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-deployment-prep-uefi-httpboot.html

=== setup linux kernel ===

* copy the kernel files
<pre>
cd /boot
rsync -av config* initrd* System.map* vmlinuz* /tftpboot/
</pre>
* cd /tftpboot
* chmod a+r *

=== setup nfs ===

* apt-get install nfs-kernel-server
* enable NFS over UDP, edit /etc/nfs.conf add "udp=y":
<pre>
udp=y
</pre>
<pre>
systemctl restart nfs-server.service
</pre>
* emacs -nw /etc/exports
<pre>
/nfsroot/dsfe01 dsfe01(rw,no_root_squash,async,no_subtree_check)
</pre>
* enable services
<pre>
systemctl enable nfs-server
systemctl enable nfs-mountd
systemctl enable nfs-idmapd
systemctl restart nfs-server
systemctl restart nfs-mountd
systemctl restart nfs-idmapd
</pre>
* after editing /etc/exports, run
<pre>
exportfs -av
</pre>

=== setup userland ===

!!! ubuntu-18 version !!!

* zfs create rpool/nfsroot
* zfs set dedup=verify rpool/nfsroot ### enable deduplication to save disk space because most linux images have mostly identical files
* clone ubuntu
<pre>
mkdir /nfsroot/dsfe01
cd /
rsync -avx . /nfsroot/dsfe01
</pre>
* edit config files:
* cd /nfsroot/dsfe01
* emacs -nw etc/hostname ### change to dsfe01
* emacs -nw etc/mailname ### change to dsfe01
* emacs -nw etc/yp.conf ### change daq00.triumf.ca to musr00.triumf.ca
* emacs -nw etc/defaultdomain ### change to MUSR-NIS
* cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
* emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
* emacs -nw root/.ssh/authorized_keys ### update root ssh keys
* emacs -nw etc/fstab ### add this
<pre>
192.168.1.1:/nfsroot/dsfe01 / nfs defaults,nolock 0 0
</pre>
* emacs -nw etc/chrony/chrony.conf
** comment-out all "pool" and "server" entries
** add entry "server 192.168.1.1 iburst"

After dsfe01 is booted:

* disable services:
<pre>
systemctl disable apache2
systemctl disable dnsmasq
systemctl disable zfs-import-cache
</pre>

To setup additional machines, clone dsfe01 instead of cloning the gateway machine

=== Allow manpages to be viewed ===

If <code>/</code> is mounted over NFS, <code>man</code> will report a permission error. Fix it with:

<pre>
ln -s /etc/apparmor.d/usr.bin.man /etc/apparmor.d/disable/
apparmor_parser -R /etc/apparmor.d/usr.bin.man
</pre>

== setup shared home directory ==

=== on the gateway machine ===
* define netgroups
* emacs -nw /etc/netgroup
<pre>
dsfe (dsfe01,,) (dsfe02,,)
</pre>
* emacs -nw /etc/nsswitch.conf ### edit the netgroup line to read:
<pre>
netgroup: files
</pre>
* export the home directories:
* emacs -nw /etc/exports ### add this:
<pre>
/zssd/home1 @dsfe(rw,no_root_squash,async,no_subtree_check)
</pre>
* exportfs -rc

=== on the frontend machine ===

* mkdir /home
* emacs -nw /etc/fstab ### add this:
<pre>
192.168.1.1:/zssd/home1 /home nfs defaults 0 0
</pre>
* mount -a

== setup NAT ==

NAT allows machines on the private network to connect to the internet: https://en.wikipedia.org/wiki/Network_address_translation

In these examples:
* replace "eno1" with name of the outgoing interface (the one connected to the TRIUMF network).
* replace "enp11s0" with name of the private network interface (192.168.1.x network)

* emacs -nw /etc/rc.local ### add this:
<pre>
# /etc/rc.local

# enable NAT

/sbin/iptables -t nat -A POSTROUTING -o eno1 -j MASQUERADE
iptables -L -v

# uncomment following lines if machine has prohibitive FORWARD rules:
#/sbin/iptables -I FORWARD -i eno1 -o enp11s0 -m state --state RELATED,ESTABLISHED -j ACCEPT
#/sbin/iptables -I FORWARD -i enp11s0 -o eno1 -j ACCEPT
#iptables -L -v

iptables -L -v
sysctl -w net.ipv4.ip_forward=1
#sysctl -a | grep forward

sh /etc/firewall-rfc1918.sh

# end
</pre>
* emacs -nw /etc/firewall-rfc1918.sh
<pre>
# firewall-rfc1918.sh

# prevent RFC1918 private network IP addresses from
# going in and out from our uplink.

ETH=eno1

iptables -F in-rfc1918
iptables -N in-rfc1918
iptables -A in-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A in-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A in-rfc1918 --dst 192.168.0.0/16 -j REJECT

iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -I INPUT -j in-rfc1918 -i $ETH

iptables -F out-rfc1918
iptables -N out-rfc1918
iptables -A out-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A out-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A out-rfc1918 --dst 192.168.0.0/16 -j REJECT

iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -I OUTPUT -j out-rfc1918 -o $ETH

iptables -D FORWARD -j out-rfc1918 -o $ETH
iptables -D FORWARD -j out-rfc1918 -o $ETH
iptables -I FORWARD -j out-rfc1918 -o $ETH

# allow TRIUMF-SECURE network

iptables -I in-rfc1918 -s 10.90.0.0/255.255.0.0 -j ACCEPT
iptables -I out-rfc1918 -d 10.90.0.0/255.255.0.0 -j ACCEPT

# show configuration

iptables -L -v

#end
</pre>

= KVM =

<pre>
apt install cpu-checker

root@daq13:~# kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
root@daq13:~#

(if not, shutdown, go into BIOS settings, enable CPU virtualization)

apt install virtinst ### will install many packages
apt install libvirt-clients libvirt-daemon-system-systemd libvirt-daemon qemu qemu-kvm libvirt-daemon-system virtinst bridge-utils

root@daq13:/home1/wheel# virsh list --all
Id Name State
------------------------------
1 ubuntu-guest running

apt install virt-manager

virt-install --name ubuntu-guest --os-variant ubuntu20.04 --vcpus 2 --ram 2048 --location /daq/daqstore/olchansk/linux/Ubuntu/ubuntu-20.04.3-desktop-amd64.iso --network bridge=virbr0,model=virtio --graphics none --extra-args='console=ttyS0,115200n8 serial'

virtual machine will start, boot, etc
to get out of it, CTRL + Shift followed by ]

ssh wheel@daq13
virt-manager

run virt-install again, omit "--graphics none", open graphics console from virt-manager, it booted into ubuntu installer desktop

virt-install --name test10 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --filesystem /kvm_ladd00,/ --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial" --graphics none

virt-install --name test14 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --disk /tmp/xxx/ladd00.img,bus=sata --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial rdshell" --graphics none --check path_in_use=off
</pre>

build image

<pre>
dd if=/dev/zero of=/tmp/xxx/ladd00.img bs=1024M count=20
mkfs.ext3 /tmp/xxx/ladd00.img ### ext4 fails to mount by SL6 kernel, "unknown ext4 options"
cd /kvm_ladd00/
mount -o loop /tmp/xxx/ladd00.img /mnt/tmp
rsync -av . /mnt/tmp/ --delete
umount /mnt/tmp
</pre>

on the guest, configure network: /etc/rc.local
<pre>
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

ifconfig eth2 192.168.122.2
route add -net 0.0.0.0 gw 192.168.122.1
ifconfig -a
netstat -rn

# end
</pre>

== virsh commands ==

<pre>
virsh list --all

virsh start kvm-el7
virsh console kvm-el7
virsh destroy kvm-el7

virsh install ...
virsh undefine kvm-el7

virsh autostart kvm-ladd00
virsh dominfo kvm-ladd00
</pre>

== virtualize SL6 ladd00 ==

* on ladd00:
<pre>
yum install dracut-network
mkinitrd /boot/initramfs-2.6.32-754.35.1.el6.x86_64-netboot.img 2.6.32-754.35.1.el6.x86_64
</pre>
* on daq00
<pre>
zfs create rpool/kvm-ladd00
cd /kvm-ladd00
rsync -avx ladd00:/ . --exclude nfsroot
brctl addbr virbr0
ifconfig virbr0 192.168.1.1
echo /kvm-ladd00 192.168.1.2(rw,no_root_squash,no_all_squash,async,no_subtree_check) >> /etc/exports
exportfs -rv
</pre>
* create virtual machine
<pre>
virt-install --name kvm-ladd00 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --network bridge=virbr0,model=virtio --boot kernel=/kvm-ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm-ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64-netboot.img,kernel_args="root=/dev/nfs ip=192.168.1.2:192.168.1.1:192.168.1.1:255.255.255.0:ladd00::off nfsroot=192.168.1.1:/kvm-ladd00,vers=3,tcp console=ttyS0,115200n8 serial rdshell" --graphics none --nodisks --check path_in_use=off
</pre>
* adjust kvm-ladd00 image
<pre>
disable network manager
edit fstab
edit yp.conf
edit resolv.conf
edit root/.ssh/authorized_keys
enable rngd or /dev/random does not work, sshd does not work
</pre>
* virsh shutdown test24
* virsh --connect qemu:///system start test24
* virsh console test24 ### to exit, ctrl+[ or ctrl+]
* virsh undefine test24
* virsh autostart kvm-ladd00
* virsh dominfo kvm-ladd00
<pre>
root@daq00:~# virsh dominfo kvm-ladd00
Id: 1
Name: kvm-ladd00
UUID: 1d1f8fed-8b65-4411-a51b-e0ecf359d2f1
OS Type: hvm
State: running
CPU(s): 2
CPU time: 27.7s
Max memory: 2097152 KiB
Used memory: 2097152 KiB
Persistent: yes
Autostart: enable
Managed save: no
Security model: apparmor
Security DOI: 0
Security label: libvirt-1d1f8fed-8b65-4411-a51b-e0ecf359d2f1 (enforcing)
root@daq00:~#
</pre>
* delete unused images in /var/lib/libvirt/images
* virsh edit kvm-ladd00 # change boot command line, etc

== virtualize CentOS-7 daqstore ==

* similar to ladd00 above:
* on daqstore, install dracut-network, already there
<pre>
yum install dracut-network
yum install qemu-kvm libvirt libvirt-python libguestfs-tools virt-install
# yum install busybox ### no rpm package?!?
</pre>
<pre>
dracut -a nfs -v /boot/initramfs-3.10.0-1160.119.1.el7.x86_64-virt.img 3.10.0-1160.119.1.el7.x86_64 --force
</pre>
<pre>
scp daqstore:/boot/initramfs-3.10.0-1160.119.1.el7.x86_64-virt.img /kvm-el7/boot/
</pre>
* on daq00
<pre>
zfs create rpool/kvm-el7
cd /kvm-el7
rsync -avx daqstore:/ .
echo 192.168.1.3 kvm-el7 >> /etc/hosts
systemctl restart dnsmasq
echo /kvm-el 192.168.1.3(rw,no_root_squash,no_all_squash,async,no_subtree_check) >> /etc/exports
exportfs -rv
</pre>
* manage virtual machine
<pre>
virsh console kvm-el7
virsh destroy kvm-el7
virsh undefine kvm-el7
</pre>
* create virtual machine
<pre>
virt-install --name kvm-el7 --os-variant centos7 --vcpus 2 --ram 2048 --import --network bridge=virbr0,model=e1000e --boot kernel=/kvm-el7/boot/vmlinuz-3.10.0-1160.119.1.el7.x86_64,initrd=/kvm-el7/boot/initramfs-3.10.0-1160.119.1.el7.x86_64-virt.img,kernel_args="root=/dev/nfs ip=192.168.1.3:192.168.1.1:192.168.1.1:255.255.255.0:kvm-el7::off nfsroot=192.168.1.1:/kvm-el7,vers=3,tcp rw console=ttyS0,115200n8 serial rdshell" --graphics none --nodisks --check path_in_use=off
</pre>
* adjust kvm-el7 image
<pre>
disable network manager
edit fstab
edit hostname
disable selinux in /etc/sysconfig/selinux

UP TO HERE --- DNS does not work!!!

edit yp.conf
edit resolv.conf
edit root/.ssh/authorized_keys
enable rngd or /dev/random does not work, sshd does not work
</pre>
* virsh shutdown test24
* virsh --connect qemu:///system start test24
* virsh console test24 ### to exit, ctrl+[ or ctrl+]
* virsh undefine test24
* virsh autostart kvm-ladd00
* virsh dominfo kvm-ladd00
<pre>
root@daq00:~# virsh dominfo kvm-ladd00
Id: 1
Name: kvm-ladd00
UUID: 1d1f8fed-8b65-4411-a51b-e0ecf359d2f1
OS Type: hvm
State: running
CPU(s): 2
CPU time: 27.7s
Max memory: 2097152 KiB
Used memory: 2097152 KiB
Persistent: yes
Autostart: enable
Managed save: no
Security model: apparmor
Security DOI: 0
Security label: libvirt-1d1f8fed-8b65-4411-a51b-e0ecf359d2f1 (enforcing)
root@daq00:~#
</pre>
* delete unused images in /var/lib/libvirt/images
* virsh edit kvm-ladd00 # change boot command line, etc

= ARM64 cross-compiler =

* arm64, aarch64 are Xilinx FPGA Cortex-A53, RPi4, RPi5 machines
* install packages:
<pre>
apt install g++-12-aarch64-linux-gnu gcc-12-aarch64-linux-gnu-base libstdc++-12-dev-arm64-cross
</pre>
* run:
<pre>
aarch64-linux-gnu-gcc-12 -o ttcp.aarch64 ttcp.c -static
aarch64-linux-gnu-g++-12 -o fecdm.exe -O2 -g -Wall -Wuninitialized -std=c++20 fecdm.o dsdm.o /home/dsdaqdev/packages_common/midas/linux-aarch64-remoteonly/lib/libmidas.a -pthread -lrt -lutil /nfsroot/gdm00/usr/lib/aarch64-linux-gnu/libi2c.a -static
</pre>

= ARM cross-compiler =

NOTE: updated for U-24

* armv7 (Cyclone-V SoC, RPi3, MityARM CAMAC) machines (Debian-12 armhf target, Ubuntu 24.04 host)
* install packages:
<pre>
apt install g++-arm-linux-gnueabihf gcc-arm-linux-gnueabihf libc6-dev-armhf-cross
</pre>
* build MIDAS frontend (static linking)
<pre>
arm-linux-gnueabihf-g++ -std=c++11 -Wall -Wuninitialized -g -O2 -I/home/dldaq/packages/midas/include -I/home/dldaq/packages/midas/mvodb -c koi2c.cxx
arm-linux-gnueabihf-g++ -o fedldb.exe -std=c++11 -Wall -Wuninitialized -g -O2 -I/home/dldaq/packages/midas/include -I/home/dldaq/packages/midas/mvodb fedldb.o koi2c.o /home/dldaq/packages/midas/linux-armv7-remoteonly/lib/libmidas.a -L/usr/arm-linux-gnueabihf/lib -L/nfsroot/dltdc/usr/lib/arm-linux-gnueabihf -static -lm -lz -lutil -lnsl -lpthread -lrt -li2c
/usr/lib/gcc-cross/arm-linux-gnueabihf/13/../../../../arm-linux-gnueabihf/bin/ld: /home/dldaq/packages/midas/linux-armv7-remoteonly/lib/libmidas.a(system.o): in function `ss_socket_connect_tcp(char const*, int, int*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)':
/home/dldaq/packages/midas/src/system.cxx:4984:(.text+0x252a): warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
</pre>

= 32-bit intel cross-compiler =

Ubuntu 22.04:

<pre>
apt install libstdc++-11-dev:i386
apt install zlib1g-dev:i386
</pre>

NOTES:
* "g++ -m32" does not find libstdc++, please use "g++ -m32 -L/usr/lib/gcc/i686-linux-gnu/11/"
* to cross-build 32-bit MIDAS, use "make linux32".
* executables cross-build on Ubuntu-22 do NOT run on 32-bit Debain-11 (GLIBC and GLIBCXX version mismatch)
* executables cross-build on Ubuntu-22 run on 32-bit Debian-12.

= SSH settings for EPICS =

* TRIUMF EPICS runs obsolete version of SSH
* add this to the use .ssh/config
<pre>
Host sbp1*
HostKeyAlgorithms +ssh-rsa
PubKeyAcceptedAlgorithms +ssh-rsa
KexAlgorithms +diffie-hellman-group1-sha1
ForwardX11 yes
ForwardX11Trusted yes
</pre>

= changes for VME processors =

<pre>
apt -y remove sysstat man-db
apt -y purge dkms
apt -y purge mdadm
apt -y purge fwupd
apt -y purge packagekit
apt -y purge accountsservice
apt -y purge plocate
apt -y purge upower power-profiles-daemon
apt -y autoremove
</pre>

for D-12 32-bit CPUs:

<pre>
apt remove "*libavahi*"
</pre>

= remove snap (U-24) =

Note: snap stores data in $USER/snap/$SNAPNAME, removing a snap on one machine will remove this data from all users even if they want to use snap on some other machine.

Prepare:

NOTE: first remove chromium and firefox, see below.

NOTE: if possible, stop autofs before removing snap - otherwise it will mount all user home directories and complain that it cannot remove some snap data from them

<pre>
systemctl stop autofs
ls -ld /home1/*/snap/* ### remove the per-user snap directories
</pre>

Remove snaps:

<pre>
snap list
echo snap remove --purge chromium ### see below
echo snap remove --purge firefox ### see below
snap remove thunderbird
snap remove cups
snap remove hello-world
snap remove firmware-updater
snap remove gtk-common-themes
snap remove snapd-desktop-integration
snap remove snap-store
snap remove hunspell-dictionaries-1-7-2004
snap remove gnome-system-monitor
snap remove gnome-3-26-1604
snap remove gnome-3-28-1804
snap remove gnome-3-34-1804
snap remove gnome-3-38-2004
snap remove gnome-42-2204
snap remove gnome-46-2404
snap remove mesa-2404
snap remove core
snap remove core18
snap remove core20
snap remove core22
snap remove core24
snap remove bare
snap remove snapd
snap list
</pre>
<pre>
root@daqubuntu:~# snap list
No snaps are installed yet. Try 'snap install hello-world'.
root@daqubuntu:~#
</pre>

Identify packages that install snaps:

<pre>
apt list | grep snap | grep installed | grep -v -e snappy -e snapshot
</pre>

Typical output:

<pre>
firefox/noble,now 1:1snap1-0ubuntu5 amd64 [installed]
gir1.2-snapd-2/noble,now 1.64-0ubuntu5 amd64 [installed,automatic]
libsnapd-glib-2-1/noble,now 1.64-0ubuntu5 amd64 [installed,automatic]
libsnapd-qt-2-1/noble,now 1.64-0ubuntu5 amd64 [installed,automatic]
plasma-discover-backend-snap/noble,now 5.27.11-0ubuntu2 amd64 [installed]
snapd/noble-updates,now 2.66.1+24.04 amd64 [installed]
</pre>

Remove packages that install snaps:

<pre>
apt remove chromium-browser
apt remove chromium-codecs-ffmpeg-extra
apt remove thunderbird
apt remove firefox
apt remove plasma-discover-backend-snap
apt remove plasma-discover-snap-backend
apt remove snapd
apt purge snapd
# package gir1.2-snapd-2 is required by ubuntu-mate-desktop & co
# libsnapd-glib-2-1 is required by gstreamer, gnome-remote-desktop & co
ls -l /etc/systemd/system/ | grep snap ### remove unwanted stuff
</pre>

Remove Chromium:

* ls -ld /home/*/snap/chromium/*
* echo /bin/rm -rf `ls -1d /home/*/snap/chromium/*`
* snap remove chromium

Remove Firefox:

* ls -ld /home/*/snap/firefox/*
* echo /bin/rm -rf `ls -1d /home/*/snap/firefox/*` ### this will delete "snap firefox" profiles of all users!!!
* snap remove firefox ### this will also delete "snap firefox" profiles of all users!!!

Remove gir1.2-snapd-2:
* echo rm -vf /usr/lib/x86_64-linux-gnu/girepository-1.0/Snapd-2.typelib

If "snap remove" is stuck in "change in progress" (this will remove all snaps and break snapd, which is ok,
see https://forum.snapcraft.io/t/snap-remove-taking-forever-abort-wasnt-working/48915)
<pre>
rm /var/lib/snapd/state.json
systemctl restart snapd
</pre>

Prevent snap from reinstalling:
<pre>
cd ~/git/scripts/etc
git pull
cp etc-apt-preferencesd-disable-snap /etc/apt/preferences.d/
</pre>

= install non-snap thunderbird =

from: https://ubuntuhandbook.org/index.php/2024/03/install-thunderbird-deb-ubuntu-2404/

* remove snap thunderbird
<pre>
snap remove --purge thunderbird
apt remove --purge thunderbird
</pre>
* add mozilla repository
<pre>
already done after installing firefox-esr
</pre>
* ppa deb and ubuntu snap thunderbird package names are the same, change priority and hide snap package: create /etc/apt/preferences.d/mozillateamppa
<pre>
Package: thunderbird*
Pin: release o=LP-PPA-mozillateam
Pin-Priority: 1001

Package: thunderbird*
Pin: release o=Ubuntu
Pin-Priority: -1
</pre>
* check that it worked, it should say "build" instead of "snap"
<pre>
apt update
apt list | grep thunderbird | grep -v locale
...
thunderbird/noble 1:128.8.1+build1-0ubuntu0.24.04.1~mt1 amd64
...
</pre>
* install
<pre>
apt install thunderbird
</pre>
* run: thunderbird

= EFI boot using syslinux =

* rationale 1: GRUB is the stock boot loader with U-24. It is unnecessarily complicated. EFI BIOS can boot the linux kernel directly, without GRUB, but unfortunately a small shim bootloader is required to specify the initrd file and the root filesystem. the syslinux boot loader can do this in a very simple way.
* rationale 2: GRUB bootloader configuration is overcomplicated, and when it breaks, it is almost impossible to debug and to recover.
* rationale 3: GRUB bootloader scripts in U-24 have no support for booting from redundant SSDs.
* in the case of GRUB bootloader failure, it is simplest to boot the Ubuntu installer in recovery mode and convert the bootloader from GRUB to syslinux. Open firefox on this page and cut-and-paste the steps (only copy of vmlinux and initrd cannot copy-and-paste as of this writing).
* in the case of servers with redundant SSDs for OS and home directories (ZFS mirror), it is simplest to use the syslinux bootloader to ensure that the machine boots from either SSD (all combinations SSD failures, fail of either EFI partiion, fail of either ZFS mirror partition, machine should boot)
* check partition tables, SATA SSD
<pre>
fdisk -l /dev/sda
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: A3F34DAC-DCB4-B74C-B59E-41E754807812

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root
</pre>
* check partition tables, NVME SSD
<pre>
fdisk -l /dev/nvme0n1
Disk /dev/nvme0n1: 1.75 TiB, 1920383410176 bytes, 3750748848 sectors
Disk model: SAMSUNG MZ1L21T9HCLS-00A07
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 131072 bytes / 131072 bytes
Disklabel type: gpt
Disk identifier: 04ECCD46-DC2A-454C-B4A8-CCC18AA532F7

Device Start End Sectors Size Type
/dev/nvme0n1p1 2048 2203647 2201600 1G EFI System
/dev/nvme0n1p2 2203648 6397951 4194304 2G Linux filesystem
/dev/nvme0n1p3 6397952 23175167 16777216 8G Linux swap
/dev/nvme0n1p4 23175168 3750746111 3727570944 1.7T Linux filesystem
</pre>
* prepare boot device EFI partition, SATA SSDs
<pre>
mkfs.msdos /dev/sda1
mkfs.msdos /dev/sdb1
mkdir /boot/efi-sda
mkdir /boot/efi-sdb
mount /dev/sda1 /boot/efi-sda
mount /dev/sdb1 /boot/efi-sdb
</pre>
* prepare boot device EFI partition, NVME SSDs
<pre>
mkfs.msdos /dev/nvme0n1p1
mkfs.msdos /dev/nvme1n1p1
mkdir /boot/efi-0
mkdir /boot/efi-1
mount /dev/nvme0n1p1 /boot/efi-0
mount /dev/nvme1n1p1 /boot/efi-1
</pre>
* add them to fstab, note the "nofail" mount option
<pre>
blkid | grep vfat
/dev/sdb1: UUID="F30C-13B5" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="20e423b5-ac29-ec42-bab5-f366aefbbd2b"
/dev/sda1: UUID="F2DD-7321" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="9427646c-ce5f-fe47-9ed1-4b84cf4c348f"
grep ^UUID /etc/fstab
UUID=F2DD-7321 /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
UUID=F30C-13B5 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022.nofail 0 1
</pre>
* prepare the EFI partitions (remove old previous subdirectories, only empty efi/boot should be there)
<pre>
cd /boot/efi-sda
mkdir -p efi/boot
</pre>
* get syslinux-6.03
<pre>
cd ~
wget https://daq00.triumf.ca/~olchansk/linux/syslinux-6.03.tar.xz
xz -d < syslinux-6.03.tar.xz | tar xvf -
</pre>
* from syslinux-6.03 copy files:
<pre>
cd /boot/efi-sda/efi/boot
cp ~/syslinux-6.03/efi64/efi/syslinux.efi .
cp ~/syslinux-6.03/efi64/com32/elflink/ldlinux/ldlinux.e64 .
cp syslinux.efi bootx64.efi
</pre>
* identify the ZFS rpool label
<pre>
zfs list | grep ROOT | grep "/$" | cut -f1 -d" "
rpool/ROOT/ubuntu_9yvb17
</pre>
* create syslinux.cfg, change the root=ZFS label to match this computer
<pre>
cat << EOF | sed "s;root=.*$;root=ZFS=`zfs list | grep ROOT | grep "/$" | cut -f1 -d" "`;" > syslinux.cfg
default linux
label linux
kernel vmlinuz
append ro initrd=initrd.img root=ZFS=rpool/ROOT/ubuntu_02ruwj
EOF
</pre>
* copy linux boot files:
<pre>
cp /boot/vmlinuz vmlinuz
cp /boot/initrd.img initrd.img
</pre>
* repeat with /boot/efi-sdb, etc
* install script to set syslinux to boot the latest kernel
<pre>
cd ~/git/scripts
git pull
ln -s ~/git/scripts/etc/update_efi_syslinux.perl ~
</pre>
* update syslinux to boot the latest kernel
** run "~/update_efi_syslinux.perl" to check that it finds the EFI partitions and finds the correct kernel
** run "~/update_efi_syslinux.perl -u" to do the actual update
* or maybe install Ubuntu syslinux 6.04 and use files from there:
<pre>
apt install "syslinux*"
</pre>

= legacy boot using syslinux =

* NOTE: extlinux is not compatible with ext4 "64bit" feature, it should be turned off:
<pre>
mkfs.ext4 -O ^64bit /dev/sdX1
resize2fs -s /dev/sdX1
</pre>

* install syslinux and extlinux (THIS DOES NOT WORK!!!)

<pre>
apt -y install syslinux extlinux
dd if=/usr/lib/syslinux/mbr/mbr.bin of=/dev/sdX ### NOT /dev/sdX1 NOT !!!
cd /boot
cp /usr/lib/syslinux/modules/bios/menu.c32 .
extlinux -i .
</pre>

* install syslinux and extlinux

* copy from old SL6 USB disk (this is extlinux 6.02)
<pre>
root@localhost:/boot# ls -l
-rwxr-xr-x 1 root root 218952 Jan 28 17:40 extlinux
-rw-r--r-- 1 root root 402 Jan 29 14:45 extlinux.conf
-rw-r--r-- 1 root root 496 Jan 29 14:39 extlinux.conf~
-r--r--r-- 1 root root 122044 Jan 29 14:39 ldlinux.c32
-r--r--r-- 1 root root 67072 Jan 29 14:39 ldlinux.sys
-rwxr-xr-x 1 root root 24156 Jan 28 17:40 libutil.c32
-rw-r--r-- 1 root root 304 Jan 28 17:40 mbr.bin
-rw-r--r-- 1 root root 26140 Jan 28 17:40 memdisk
-rw-r--r-- 1 root root 69043 Jan 28 17:40 memtest86+-4.20.iso.zip
-rw-r--r-- 1 root root 183012 Jan 28 17:40 memtest86+-5.01
-rw-r--r-- 1 root root 26568 Jan 28 17:40 menu.c32
</pre>

* install
<pre>
dd if=mbr.bin of=/dev/sdX ### NOT /dev/sdX1 NOT !!!
extlinux -i .
</pre>
* check that partition /dev/sdX1 is marked bootable (fdisk command "a")

* create /boot/extlinux.conf

<pre>
DEFAULT menu.c32
PROMPT 0
TIMEOUT 50

MENU TITLE TRIUMF DAQ USB BOOT32 ver K.O. 2025jan28

LABEL linux
MENU DEFAULT
kernel /vmlinuz
append initrd=/initrd.img panic=60 rootdelay=5 rootwait rw root=/dev/sda1

LABEL linux-6.1.0-28-686
kernel vmlinuz-6.1.0-28-686
append initrd=initrd.img-6.1.0-28-686 panic=60 rootdelay=5 rootwait rw root=/dev/sda1

LABEL memtest
kernel memtest86+-1.65
</pre>

= Add user to login spash screen =

For user login to local machine, if it doesn't work by default. User should exist already from NIS service.

<pre>
sudo /bin/bash # as root
cd /var/lib/AccountsService/users
ls # should at least display the "wheel" user
cp wheel username # we're going to copy the user account settings over to our new user
</pre>

Ubuntu

2026-03-12T17:46:50Z

Dfujimoto: /* remove snap */

= Prerequisites =

* before setting up new machine run memory test
* prepare flash drive with free version of memtest86: https://www.memtest86.com
* test boot from flash drive, test takes ~ few hours
* test will end with summary page, if passed continue with Ubuntu
* number that might be worth noting is memory latency

= Ubuntu version =

<pre>
lsb_release -a
uname -a
</pre>

= Ubuntu installer =

* updated for Ububtu LTS 20.04.01, 22.04.1, 24.04 (only minor differences)

* download the latest Ubuntu LTS desktop installer iso image
* dd the image to a USB key
* power down, disconnect all disks (all HDDs, all SSDs, all M.2)
* connect the SSD to be used as system disk
* if system will use mirrored SSDs (using ZFS mirror), leave second SSD disconnected, we will activate it later
* power up
* boot from USB key in legacy mode or UEFI mode (select this in the BIOS boot menu - F2 or F8 for ASUS, F11 for Supermicro)
* follow the instruction:
* "try ubuntu or install ubuntu" - choose "install"
* select language - accept default
* "updates and other software" - accept default settings ("normal install")
* "installation type" - select "advanced features" and "experimental: use ZFS"
* accept partition choice
* "where are you?" - select "Vancouver" (PST time zone)
* "who are you?" - leave all fields blank, except "username" set to "wheel", "password" set to the root password. hostname will be set later after configuring the network
* don't install third party sw
* installation runs in a few minutes, when finished, reboot
* login as user wheel
* answer annouying questions:
* "livepatch" - say "next"
* "help improve" - select "do not send", say "next"
* "privacy" - leave "location" as "off", say "next"
* "ready to go", say "done"
* right-click on the desktop, say "open in terminal", a shell will open
* say "sudo /bin/bash", enter the root password, you now have the root shell
* run nm-connection-editor to configure the network. use netmask 255.255.224.0, gateway 142.90.100.18, DNS 142.90.100.19, search path "triumf.ca"
* after network is up (can ping ladd00), continue with post-installation steps below

= Install instructions =

== prepare ==
<pre>
apt update
apt upgrade
</pre>

== install ssh ==
<pre>
apt install ssh
</pre>

== install git/scripts ==
<pre>
apt -y install git
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd scripts
git pull
</pre>

* if needed, update git/scripts repository from ladd00 to daq00:
* git remote -v ### if it says daq00, we are good
* git remote set-url origin https://daq00.triumf.ca/~olchansk/git/scripts.git
* git pull ### check that it works

== configure hostname ==
<pre>
vi /etc/hostname
</pre>

== disable swap ==

ubuntu installer creates a 2 GB swap partition, not useful
on 32-64 GB machine, disable it:
<pre>
vi /etc/fstab ### comment out the "swap" line
</pre>

* on 64 GB RAM machines swap is not useful
* on machines booted from network (NFS-ROOT), swap does not work
* on machines running from flash (RPi, etc), flash is too slow for useful swap
* swap configured by linux installers invariably has wrong size and is not useful

<pre>
systemctl disable dphys-swapfile
systemctl stop dphys-swapfile
dphys-swapfile uninstall
</pre>

== maybe reboot ==

this is a good point to reboot the machine to boot
the latest kernel and to set the correct hostname

== install etckeeper ==

keep contents of /etc in a git repository:

<pre>
apt -y install etckeeper
</pre>

== set timezone ==
<pre>
timedatectl list-timezones | grep -i vancouver
timedatectl set-timezone America/Vancouver
</pre>

== install time synchronization ==

check if chrony is installed correctly and is synched to TRIUMF time servers:

<pre>
chronyc sources
chronyc tracking
</pre>

if not, remove old chrony and

<pre>
apt -y remove chrony
apt -y purge chrony
</pre>

and install it from scratch:

<pre>
apt -y install chrony
cd ~/git/scripts
git pull
cd ~
cp ~/git/scripts/etc/triumf.sources /etc/chrony/sources.d/
systemctl disable systemd-timesyncd.service
systemctl stop systemd-timesyncd.service
systemctl disable ntp
systemctl stop ntp
systemctl enable chrony
systemctl restart chrony
chronyc sources
chronyc tracking
</pre>

== reenable systemd-timesyncd ==

ONLY IF CHRONY DOES NOT WORK

To configure systemd-timesyncd, set "NTP=" in /etc/systemd/timesyncd.conf

<pre>
apt remove chrony
cat /etc/systemd/timesyncd.conf
systemctl enable systemd-timesyncd.service
systemctl restart systemd-timesyncd.service
systemctl status systemd-timesyncd.service
timedatectl status
timedatectl timesync-status
</pre>

== enable outgoing email (debian 11) ==

this is different from ubuntu 20. it uses /etc/mailname and it hardwires the hostname into main.cf.

== enable outgoing email ==

we have an unusual email configuration. outgoing email should work to deliver error messages, notices, etc. incoming email is disabled, we do not receive email for local users.

this causes problems with TRIUMF smtp server. if our message cannot be delivered (wrong email address or receipient computer is turned off), TRIUMF smtp server will generate a delivery failure notification email and try to send it to the "from" address of the failed message. but the "from" address does not receive any email, so another delivery failure notification email is generated and an attempt to deliver it. which again fails, rinse and repeat.

as solution, kray created a special rule, email from scrap.triumf.ca does not generate delivery failure notices. failed messages sit in the queue for 5 days, then they are deleted. (K.O. - confirmed with kray 3jan2024).

to make this work we use the msmtp MTA package.

<pre>
cd ~
apt -y install mailutils msmtp msmtp-mta # say "no" to apparmor support
apt -y remove postfix
apt -y purge postfix # remove old config files
apt -y install bsd-mailx
cd ~/git/scripts/etc
git pull
/bin/cp -fv aliases /etc/aliases
/bin/cp -fv msmtprc /etc/msmtprc
/bin/rm -vf ~root/.forward
/bin/rm -vf /etc/mailname
Mail root
Subject: test
test
^D
CC: <CR>
</pre>

== enable outgoing email (postfix) ==

THIS IS OBSOLETE!!!

* TRIUMF: use smtp.triumf.ca
* CERN: use cernmx.cern.ch

<pre>
apt install postfix ### select "satellite system", enter full hostname "xxx.triumf.ca", enter "smtp.triumf.ca"
apt install mailutils
dpkg-reconfigure postfix ### (if postfix already installed)
</pre>
<pre>
echo olchansk@triumf.ca lindner@triumf.ca bsmith@triumf.ca dfujimoto@triumf.ca >> ~root/.forward
mailx root
test
^D
</pre>

== enable ping for all users (debian 11) ==

Without this tweak, Debian will report "operation not permitted" if a user tries to ping somewhere.

<pre>
echo 'net.ipv4.ping_group_range = 0 1000' > /etc/sysctl.d/99-ping.conf
</pre>

== disable apparmor ==

On NFS-Root network booted machines!

If "man man" returns "permission denied" and syslog reports apparmor "sendmsg DENIED" errors, disable apparmor. This is supposedly fixed in kernel 6.0 and later (to be confirmed), see https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/1784499

Disable apparmor, see https://ubuntu.com/server/docs/security-apparmor

This takes effect after a reboot.

<pre>
systemctl stop apparmor.service
systemctl disable apparmor.service
</pre>

If on boot apparmor appears to still be confining apps (for example, mysql), edit the file /etc/default/grub and change

<pre>
GRUB_CMDLINE_LINUX=""
</pre>

to

<pre>
GRUB_CMDLINE_LINUX="apparmor=0"
</pre>

Run sudo update-grub and reboot.

== install missing packages ==

(apt eats terminal input, even the "yes |" trick does not quite work,
repeat the following commands until they report that everything
is installed)

<pre>
yes | apt -y install ssh tcsh ethtool ncat rsync strace net-tools traceroute time minicom screen git lsof debsums tmux iptables telnet pax rpm mtools at gdisk tcpdump
yes | apt -y install sysstat smartmontools lm-sensors
yes | apt -y install lsb-release
apt -y install vim # in addition to default vim-tiny, requested by IRIS
apt -y install gedit # requested by TACTIC
apt -y install tcl
apt -y install mc # requested by sol
apt -y install pax rpm alien ### package converter tools
apt -y install flex bison
apt -y install neofetch
apt -y install snmp snmp-mibs-downloader
apt -y install git subversion g++ gfortran cmake doxygen
apt -y install curl libcurl4 libcurl4-openssl-dev
### conflits with mysql packages ### apt -y install mariadb-client libmariadb-dev ### mysql client for MIDAS
apt -y install mysql-client libmysqlclient-dev
apt -y install postgresql-common libpq-dev ### postgresql client for MIDAS
yes | apt -y install libz-dev libzstd-dev sqlite3 libsqlite3-dev unixodbc-dev
yes | apt -y install libssl-dev
yes | apt -y install emacs xemacs21 joe
yes | apt -y install gnuplot dos2unix
yes | apt -y install mutt bsd-mailx # email clients
yes | apt -y install liblz4-tool pbzip2 libbz2-dev
yes | apt -y install libc6-dev-i386 # otherwise no /usr/include/sys/types.h
yes | apt -y install libreadline-dev
yes | apt -y install ubuntu-mate-themes
yes | apt -y install libmotif-dev libxmu-dev
yes | apt -y install libusb-dev libusb-1.0-0-dev
yes | apt -y install i2c-tools libi2c-dev libi2c0
yes | apt -y install xfig gsfonts-x11 gsfonts-other # install fonts for xfig
yes | apt -y install libjson-perl
yes | apt -y install libgsl-dev # additional GNU Scientific Library
yes | apt -y install qt5-default # Qt development
yes | apt -y install python3-full python3-dev python3-dbg python3-pip ### for pyROOT
yes | apt -y install imagemagick imagemagick-common ckeditor # for elog
yes | apt -y install libjpeg-dev libjpeg-progs libjpeg-tools
yes | apt -y install linux-tools-common linux-tools-generic # cpupower frequency-info
yes | apt -y install rdesktop remmina remmina-plugin"*" # requested by POL
yes | apt -y install nlohmann-json3-dev # required to build MIDAS with ROOT 6.30 on Ubuntu-22
apt -y install dpkg-dev cmake g++ gcc binutils libx11-dev libxpm-dev libxft-dev libxext-dev python3 libssl-dev libafterimage0 # from https://root.cern/install/dependencies/
apt -y install gfortran libpcre3-dev xlibmesa-glu-dev libglew-dev libftgl-dev libmysqlclient-dev libfftw3-dev libcfitsio-dev graphviz-dev libldap2-dev python3-dev python3-numpy libxml2-dev libkrb5-dev libgsl0-dev qtwebengine5-dev nlohmann-json3-dev libtbb-dev libavahi-compat-libdnssd-dev # from https://root.cern/install/dependencies/
apt -y install libvdt-dev # for ROOT 6.32 on Ubuntu-24
apt -y install autoconf automake gperf # gnu package build tools
apt -y install u-boot-tools # for Xilinx petalinux
#apt -y install linux-headers-generic # to build linux kernel drivers
apt -y install htop
</pre>

Ubuntu LTS 20.04:
<pre>
yes | apt -y install linux-image-generic-hwe-20.04 linux-tools-virtual-hwe-20.04 # enable linux 5.11 series kernel
</pre>

Ubuntu LTS 22.04:
<pre>
apt -y install linux-generic-hwe-22.04 # enable linux 6.8.0 series kernel
</pre>

Ubuntu LTS 24.04:
<pre>
apt -y install linux-generic-hwe-24.04 # enable linux 6.14.0 series kernel
</pre>

== remove snap ==

remove snap at this point.

Do

<pre>
snap list
</pre>

to see a list of installed snaps. Use

<pre>
snap remove <item>
</pre>

to remove each. You will need to remove snapd last.

<pre>
systemctl stop snapd
systemctl disable snapd
apt purge snapd
rm -rf ~/snap
</pre>

== install non-snap firefox ==

delay this until the very end, download of firefox-esr deb package is very slow.

See https://askubuntu.com/questions/1399383/how-to-install-firefox-as-a-traditional-deb-package-without-snap-in-ubuntu-22

<pre>
add-apt-repository ppa:mozillateam/ppa
apt install firefox-esr
echo run firefox-esr
</pre>

Enable automatic updates:

* rerun [[#Enable_automatic_updates]]

== configure DNS ==

<pre>
cd ~/git/scripts
git pull
mkdir /etc/systemd/resolved.conf.d
cp etc/resolved-triumf.conf /etc/systemd/resolved.conf.d/
systemctl restart systemd-resolved
resolvectl
#systemd-analyze cat-config systemd/resolved.conf
</pre>

== install ganglia ==

<pre>
apt -y install ganglia-monitor
cd ~root/git/scripts/ganglia
git pull
make install
./ganglia-all.perl
</pre>

fix gmond start before network is ready:

<pre>
mkdir /etc/systemd/system/ganglia-monitor.service.d
echo -e "[Unit]\nAfter=network-online.target\n" > /etc/systemd/system/ganglia-monitor.service.d/local.conf
systemctl daemon-reload
systemctl cat ganglia-monitor.service
</pre>

== install ganglia server ==

on the main computer (daq00, dsdaqgw, etc)

* zfs create rpool/ganglia
* apt install gmetad php php-xml rrdtool
* mv /etc/ganglia/gmetad.conf /etc/ganglia/gmetad.conf-stock
* create /etc/ganglia/gmetad.conf with the following contents:
<pre>
data_source "my cluster" 15 localhost
RRAs "RRA:AVERAGE:0.5:1:244" "RRA:AVERAGE:0.5:24:244" "RRA:AVERAGE:0.5:168:244" "RRA:AVERAGE:0.5:672:244" "RRA:AVERAGE:0.5:5760:374"
setuid_username "ganglia"
rrd_rootdir "/ganglia/rrds"
case_sensitive_hostnames 0
</pre>
* mkdir /ganglia/rrds
* chown ganglia:ganglia /ganglia/rrds
* systemctl restart gmetad
* create /etc/ganglia/gmond-collect.conf
<pre>
globals {
daemonize = yes
setuid = yes
user = nobody
debug_level = 0
max_udp_msg_len = 1472
mute = no
#deaf = yes
deaf = no
allow_extra_data = yes
host_dmax = 0 /* 86400 */ /*secs. Expires (removes from web interface) hosts in 1 day */
host_tmax = 20 /*secs */
cleanup_threshold = 300 /*secs */
gexec = no
send_metadata_interval = 600 /*secs */
}

/*
* The cluster attributes specified will be used as part of the <CLUSTER>
* tag that will wrap all hosts collected by this instance.
*/
cluster {
name = "DAQ"
owner = "TRIUMF DAQ"
latlong = "unspecified"
url = "https://daq.triumf.ca"
}

/* The host section describes attributes of the host, like the location */
host {
location = "unspecified"
}

udp_send_channel {
host = daq00.triumf.ca
bind_hostname = yes
port = 8649
}

udp_recv_channel {
#mcast_join = 239.2.11.71
port = 8649
#bind = 239.2.11.71
retry_bind = true
# Size of the UDP buffer. If you are handling lots of metrics you really
# should bump it up to e.g. 10MB or even higher.
#buffer = 10485760
buffer = 200000
family = ipv4

acl {
default = "deny"
access {
# vlan1 daq00
ip = 142.90.111.168
mask = 19
action = "allow"
}
access {
# MUSR VLAN
ip = 142.90.154.73
mask = 8
action = "allow"
}
access {
# KVM network
ip = 192.168.1.1
mask = 8
action = "allow"
}
}
}

udp_recv_channel {
#mcast_join = 239.2.11.71
port = 8649
#bind = 239.2.11.71
retry_bind = true
# Size of the UDP buffer. If you are handling lots of metrics you really
# should bump it up to e.g. 10MB or even higher.
#buffer = 10485760
buffer = 200000
family = ipv6

acl {
default = "deny"
access {
# ALPHA network
ip = 2001:1458:202:fd::100:aa
mask = 8
action = "allow"
}
#access {
# # ALPHA-g network
# ip = 192.168.1.1
# mask = 8
# action = "allow"
#}
}
}

tcp_accept_channel {
port = 8649
bind = localhost
# If you want to gzip XML output
gzip_output = no
}
</pre>
* add to /etc/rc.local
<pre>
/usr/sbin/gmond -c /etc/ganglia/gmond-collect.conf &
systemctl restart gmetad &
</pre>
* cd /ganglia
* git clone https://daq00.triumf.ca/~olchansk/git/ganglia-web.git
* cd ganglia-web
* git checkout KO1
* ln -s /ganglia/ganglia-web /var/www/html/ganglia
* make dwoo directories
<pre>
mkdir /var/lib/ganglia-web
chown www-data:www-data /var/lib/ganglia-web
mkdir /var/lib/ganglia-web/dwoo
chown www-data:www-data /var/lib/ganglia-web/dwoo
mkdir /var/lib/ganglia-web/dwoo/compiled
chown www-data:www-data /var/lib/ganglia-web/dwoo/compiled
mkdir /var/lib/ganglia-web/dwoo/cache
chown www-data:www-data /var/lib/ganglia-web/dwoo/cache
</pre>
* open https://alphacpc05.cern.ch/ganglia/

== install gonodeinfo ==

* go to https://bitbucket.org/dd1/gonodeinfo follow instructions:
<pre>
apt -y install golang
mkdir ~/git
cd ~/git
#git clone https://bitbucket.org/dd1/gonodeinfo.git
git clone https://daq00.triumf.ca/~olchansk/git/gonodeinfo.git
cd gonodeinfo
git remote set-url origin https://daq00.triumf.ca/~olchansk/git/gonodeinfo.git
git pull
make
make install # install gonodeinfo agent
cd ~ # this is important
</pre>
* edit /etc/gonodeinfo.conf
* change "Description", "Location", "User" and "Administrator" as appropriate (or delete them)
* change "Servers" to read: Servers: daq00.triumf.ca:8601
* run "gonodeinfo -v"
* if error is "connection refused". go to the nodeinfo server to add this client to the access control list:
* on the gonodeinfo server: run /opt/gonodeinfo/gonodereceive.exe -a daq13
* try gonodeinfo again, there should be no error
* on the gonodeinfo server: run gonodereport, look at the web pages, the new machine should be listed now

== install emailonreboot ==

send an email if computer is rebooted

<pre>
ssh root
mkdir -p ~/git
cd ~/git
git clone https://daq00.triumf.ca/~olchansk/git/rpms.git
cd rpms/emailonreboot
git pull
make
make install
</pre>

== install monitor_nfs ==

monitor NFS mounts and complain about dead, stale and hung mounts

<pre>
ssh root
mkdir -p ~/git
cd ~/git
git clone https://daq00.triumf.ca/~olchansk/git/rpms.git
cd rpms/monitor_nfs
git pull
make
make install
</pre>

== install fonts for EPICS ==

* apt -y install xfonts-100dpi xfonts-75dpi
* restart Xorg (i.e. "killall Xorg", this will log you out from the console)
* xlsfonts | grep -i helvetica ### should show fonts with different sizes, not just size 0 (scalable)

== install libz.so.1 for CentOS compatibility ==

KO - confirm which versions on quartus need this.

<pre>
yes | apt-get -y install zlib1g
yes | apt-get -y install zlib1g:i386 libc6:i386 libgcc1:i386 gcc-6-base:i386
</pre>

== install ld-ldb-x86-64.so.3 for Quartus compatibility ==

Not clear from package this is supposed to come from. Copied from U-20.

Without this, Quartus lmgrd does not run.

<pre>
cd /lib64
ln -s ld-linux-x86-64.so.2 ld-lsb-x86-64.so.2
ln -s ld-linux-x86-64.so.2 ld-lsb-x86-64.so.3
</pre>

should look like this:
<pre>
root@daq13:/lib64# ls -l
total 3
lrwxrwxrwx 1 root root 44 Jan 28 09:07 ld-linux-x86-64.so.2 -> ../lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
lrwxrwxrwx 1 root root 20 Feb 18 16:45 ld-lsb-x86-64.so.2 -> ld-linux-x86-64.so.2
lrwxrwxrwx 1 root root 20 Feb 18 16:45 ld-lsb-x86-64.so.3 -> ld-linux-x86-64.so.2
root@daq13:/lib64#
</pre>

== install libpng12.so.0 for Quartus 13.0sp1 and 13.1.4.182 ==

<pre>
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0
wget https://daq00.triumf.ca/~olchansk/linux/libpng12.so.0.50.0
/bin/cp -pv libpng12.so.0 libpng12.so.0.50.0 /lib/x86_64-linux-gnu/
</pre>

== install packages for Xilinx ==

ubuntu LTS 22.04 vivado 2020.1

<pre>
apt install autoconf libtool
apt install libtinfo5
apt install texinfo
apt install zlib1g:i386
</pre>

== install packages for building ROOT ==

<pre>
apt -y install libx11-dev libxpm-dev libxft-dev libxext-dev libpng-dev libjpeg-dev xlibmesa-glu-dev libxml2-dev libgsl-dev cmake
</pre>

== install wine ==

As far as I know, only needed for BNMR/BNQR

<pre>
apt install wine winetricks
</pre>

== install lightdm ==

unlike the default gdm login manager, lightdm shows the machine hostname and does not require an extra mouse click to swicth from screen saver to login mode.

<pre>
apt -y install lightdm
# select lightdm
</pre>

== install desktop environments ==

note: default display manager and default desktop are deficient, please do not skip this step.

note: if apt asks to choose the display manager, select "lightdm"

note: KO - I recommend the "MATE" desktop.

note: you will have to cut-and-paste this several times because "apt" eats commands, even with "-y" and even piped from "yes".

<pre>
# install MATE desktop
apt -y install ubuntu-mate-core ubuntu-mate-desktop ubuntu-mate-themes
# install Cinnamon desktop
apt -y install cinnamon
# install KDE desktop
apt -y install kde-standard kubuntu-settings-desktop
# install Lxqt desktop
# apt -y install lxqt # conflict over kubuntu-desktop, kubuntu-settings-desktop and desktop-base
# install Xfce4 desktop
apt -y install xfce4
</pre>

== install ROOT ==

Please install ROOT per instructions at https://root.cern.ch.

NOTE1: The ROOT package available from Ubuntu repositories is severely out of date and cannot be used with MIDAS and ROOTANA. ### DO NOT DO THIS! apt-get install root-system

NOTE2: as of 2017-Jan-09, ROOT binary kits for Ubuntu do not work (use GCC 5 instead of GCC6), build from source instead.

== Install x2go ==

<pre>
apt-get update
apt-get install x2goserver x2goserver-xsession
</pre>

== enable root login from ladd00/daq00 ==
<pre>
ssh localhost
CTRL-C
/bin/cp ~root/git/scripts/etc/authorized_keys ~root/.ssh/
</pre>

== disable ssh access from outside of TRIUMF ==

to stop ssh login spam, disable ssh access from outside of TRIUMF. this can be done by requesting a firewall block through the helpdesk or by local firewall rule:

<pre>
echo iptables -I INPUT ! -s 142.90.0.0/255.255.0.0 -p tcp --dport 22 -j REJECT >> /etc/rc.local
/etc/rc.local
</pre>

== install smart-status ==
<pre>
ln -s ~/git/scripts/smart-status/smart-status.perl ~root/
</pre>

== enable boot menu and boot messages ==

This will enable the grub menu (with a 10 sec timeout) and
replace black screen with exciting linux boot messages.

* emacs -nw /etc/default/grub
<pre>
GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=menu
GRUB_TIMEOUT=10
GRUB_RECORDFAIL_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="vga=769 video=640x480"
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX=""
#GRUB_GFXMODE=640x480
</pre>
* update grub config:
<pre>
grub-mkconfig -o /boot/grub/grub.cfg
</pre>

= Enable automatic updates =

<pre>
apt install unattended-upgrades
cd ~/git/scripts
git pull
/bin/cp -v etc/99apt-conf-ko /etc/apt/apt.conf.d/
apt-config dump | grep Unattended
</pre>

Following is obsolete:

* emacs -nw /etc/apt/apt.conf.d/50unattended-upgrades
** uncomment in Allowed-Origins "-security" and "-updates"
** add in Allowed-Origins: "Google LLC:stable";
** uncomment/add: "Unattended-Upgrade::Mail "root";
* emacs -nw /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
</pre>
* test: unattended-upgrade --dry-run -v

NOTE: update-on-shutdown is disabled.

NOTE: there is no update-on-boot, but:

NOTE: if machine was off for a long time, the systemd update timer would have expired and it will fire soon after reboot, causing an automatic update run. this is unwanted, and there is no fix or workaround for it. K.O. June-2023.

= Fix bpool is full (obsolete) =

THIS IS CAUSED BY OBSOLETE PACKAGE zsys. PLEASE: apt remove zsys

= IPMI instructions =

IPMI is the board management hardware on Supermicro and other server motherboards. This includes hardware sensors - fan rotation speed, temperatures and power supply voltages.

<pre>
apt-get install ipmitool
systemctl enable ipmievd
systemctl restart ipmievd
</pre>

Run:
* ipmitool sel list ### event list
* ipmitool sel elist ### event list
* ipmitool sel clear ### clear event list (if it becomes full)
* ipmitool sensor ### report hardware sensors

= move /home/wheel (U-24) =

Ubuntu LTS 24 installed on ZFS has rpool/USERDATA/home_xxx mounted on /home,
has to be moved or autofs /home will not work.

<pre>
zfs list | grep USERDATA | grep home | cut -f1 -d" "
zfs set -u mountpoint=/home1 `zfs list | grep USERDATA | grep home | cut -f1 -d" "`
emacs -nw /etc/passwd # change mount for wheel account to /home1/wheel
</pre>

This will not take effect until rebooting. Please ensure that ssh from root@daq00 to this computer works before rebooting; if ssh from root@daq00 doesn't work and you mess this up you will be locked out of wheel account. Once you verify that this works, reboot and make sure that when you login as wheel that home directory is /home1/wheel.

= move /home/wheel =

note: this MUST be done if ZFS root and NIS/autofs with /home.

Default location of wheel's home directory will collide with autofs /home, it has to be moved,
for example to /wheel.

<pre>
# logout from the wheel user
# go to another computer
ssh root@daqubuntuxxx
zfs list | grep wheel ### identify zfs name wheel_xxxxxx
#zfs set mountpoint=/wheel rpool/USERDATA/wheel_hm8fzh
zfs set mountpoint=/wheel `zfs list | grep wheel | cut -f1 -d" "`
zfs list | grep wheel
emacs -nw /etc/passwd ### change wheel's home directory from /home/wheel to /wheel
su - wheel ### check that user wheel still works
</pre>

This will break wheel's ability to run snap programs, such as firefox, install chrome as listed below.

= enable NIS (ubuntu 22.04, 24.04, debian 11, 12) =

<pre>
apt -y install rpcbind nis
echo DAQ-NIS >> /etc/defaultdomain
echo ypserver daq00.triumf.ca >> /etc/yp.conf
systemctl enable ypbind.service
systemctl restart ypbind.service
systemctl status ypbind.service
ypwhich -m
</pre>

enable ypserv:

<pre>
sed -i s/NISSERVER=false/NISSERVER=slave/ /etc/default/nis
/usr/lib/yp/ypinit -s daq00
echo ypserver localhost >> /etc/yp.conf
sed -i "s/ypserver .*/ypserver localhost/" /etc/yp.conf
systemctl enable ypserv
systemctl restart ypserv
systemctl restart ypbind
</pre>

update /etc/nsswitch.conf and enable hourly update of NIS maps:

<pre>
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
cp -pv nsswitch.conf-U24 /etc/nsswitch.conf
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>

If this is a new machine, then on the master NIS node (daq00), add this new node to /etc/netgroup, and update NIS maps (cd /var/yp; make)

= enable NIS (ubuntu 20.04) =

* apt-get -y install portmap nis ### will ask for NIS domain (DAQ-NIS)
* dpkg-reconfigure nis ### reconfigure if already installed
* ypwhich -m
* edit /etc/default/nis
** set "NISSERVER=slave"
** Ubuntu LTS 20.04, check that "YPBINDARGS=" is blank, remove "-no-dbus" if it is there
* #edit /etc/yp.conf, comment-out everything, add "domain DAQ-NIS server localhost"
* edit /etc/yp.conf, comment-out everything, add "ypserver localhost"
* /usr/lib/yp/ypinit -s daq00
* systemctl enable nis
* systemctl restart nis
* ypwhich
* ypwhich -m
* ypcat -k passwd
* vi /etc/nsswitch.conf ### add the automount line, modify the passwd, group and shadow lines to read this:
<pre>
# begin get data from nis
passwd: files nis
group: files nis
shadow: files nis
automount: files nis
netgroup: files nis
# end get data from nis
</pre>
* enable hourly update of NIS maps
<pre>
mkdir ~root/git
cd ~root/git
git clone https://daq00.triumf.ca/~olchansk/git/scripts.git
cd ~/git/scripts/etc
git pull
ln -s $PWD/ypxfr-cron-hourly /etc/cron.hourly
</pre>
* ### NOT NEEDED sudo vi /etc/idmapd.conf ### add line: "Domain = triumf.ca"

= enable autofs =

<pre>
apt -y install autofs
systemctl enable autofs
systemctl restart autofs
ls -l /home/olchansk ### test autofs, check file owner is correct
</pre>

= enable NFS server =

<pre>
apt install nfs-kernel-server
#edit /etc/exports
systemctl enable nfs-server
systemctl restart nfs-server
</pre>

= NIS master =

notes for setting up the NIS master

== wheel user ==

"wheel" is the default administrative user. We do not want it's password exported to NIS (encrypted password hash is world visible) and we do not want it's home directory exported to NFS (~wheel/.ssh is world visible and potentially writable: anybody can change ~wheel/.ssh/authorized_keys).

* move wheel's home directory from /home/wheel to /wheel (see special section about this)
* change wheel's UID and GID from 1000 to a value below MINUID in /var/yp/Makefile

== coherent uids ==

we do not want system accounts defined in /etc/passwd of the NIS master
to be included in the NIS map "passwd". this causes trouble on NIS clients
where newly installed packages fail to create local system users because same
user already exists in NIS.

This is controlled by MINUID in /var/yp/Makefile.

Historical TRIUMF uids start from around 200, but several clusters do not have any historic TRIUMF uids below 500 and MINUID is set to:
* DAQ-NIS: MINUID=200
* ISAC-NIS: MINUID=500
* TITAN-NIS: MINUID=500
* MUSR-NIS: MINUID=500
* TIG-NIS: MINUID=500 (100 on SL6 mother8pi)

Ubuntu 20 has two programs to create users:
* adduser - creates new users with UID 1000 and up as specified in /etc/adduser.conf. No problems here.
* adduser --system - creates new system users with UID 100 and up as specified in /etc/adduser.conf. No problems here.
* useradd - creates new users with UID 1000 and up as specified in /etc/login.defs. No problems here.
* useradd --system - creates new system users with UID 999 and down (read "man useradd", section at the end about SYS_UID_MAX). This collides with NIS MINUID, these system users will be included in the NIS map and cause trouble.

This problem cannot be fixed, SYS_UID_MIN, SYS_UID_MAX and UID_MIN in /etc/login.defs do not seem
to have any effect on UIDs chosen by "useradd --system". (tested on Ubuntu LTS 20.04).

So far only these system accounts seem to be affected by this:
* systemd-coredump
* ganglia

To fix:
* run "sort -r -n -t: -k3 /etc/passwd" to identify the last unused system user uid (range 100..200)
* run "sort -r -n -t: -k3 /etc/group" to identify the last unused system user gid (range 100.200)
* systemd-coredump: manually change UID and GID (package systemd-coredump is usually not installed)
* ganglia: same thing, then change ownership on all ganglia files.

Also read systemd author's opinion on system vs user UIDs:
https://github.com/systemd/systemd/issues/4850#issuecomment-265698275

= Fix systemd-logind NIS breakage =

!!! THIS IS NOT NEEDED FOR UBUNTU LTS 20.04 !!!

there is a delay in ssh logins for normal users. "ssh -v" shows the delay is after "pledge...". this
fix removes the delay.

systemd developers think that we should not use NIS and made sure there are
problems if we do. To give them credit, they do offer a workaround. Read this:
https://github.com/poettering/systemd/commit/695fe4078f0df6564a1be1c4a6a9e8a640d23b67

<pre>
mkdir /etc/systemd/system/systemd-logind.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-logind.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-logind.service
</pre>

= Fix systemd-udevd NIS breakage =

see same problem as above with udev getting stuck. ubuntu lts 20.04.

<pre>
mkdir /etc/systemd/system/systemd-udevd.service.d
echo -e "[Service]\nIPAddressDeny=\n" > /etc/systemd/system/systemd-udevd.service.d/local.conf
systemctl daemon-reload
systemctl cat systemd-udevd.service
</pre>

= Configure USB device permissions =

Configure USB device permissions for user access to USB-serial devices, Altera USB Blaster, etc.

* create file /etc/udev/rules.d/99-usb-chmod.rules with this contents:

<pre>
emacs -nw /etc/udev/rules.d/99-usb-chmod.rules
ACTION=="add", SUBSYSTEM=="usbmisc", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /dev/%c"
ACTION=="add", SUBSYSTEM=="usb_device", RUN+="/bin/chmod a+wr /proc/%c"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVTYPE}=="usb_device", RUN+="/bin/chmod a+wr $env{DEVICE}"
ACTION=="add", ENV{PHYSDEVBUS}=="usb-serial", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", ENV{DEVPATH}=="/class/tty/ttyS*", RUN+="/bin/chmod a+wr $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyUSB*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyACM*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", SUBSYSTEM=="tty", DEVPATH=="*ttyS*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
ACTION=="add", DEVPATH=="*video*", RUN+="/bin/chmod a+rw $env{DEVNAME}"
</pre>

* reload udev rules: udevadm control --reload-rules
* apply new permissions: udevadm trigger --action=add
* watch udev activity: udevadm monitor -p

= Configure lightdm display manager =

* enable it
<pre>
echo lightdm | dpkg-reconfigure -fteletype lightdm
systemctl disable gdm
systemctl disable sddm
systemctl enable lightdm
</pre>

* make the MATE desktop as default
<pre>
cd ~root/git/scripts/
git pull
/bin/cp -v etc/lightdm_default_mate.conf /etc/lightdm/lightdm.conf.d/
</pre>

* enable login by NIS users
<pre>
/bin/cp -v etc/lightdm_enable_nis_login.conf /etc/lightdm/lightdm.conf.d/
</pre>

* restart lightdm
<pre>
systemctl stop gdm
systemctl restart lightdm
</pre>

= Install libpng12.so.0 =

Quartus 16 needs libpng12:

<pre>
wget http://mirrors.kernel.org/ubuntu/pool/main/libp/libpng/libpng12-0_1.2.54-1ubuntu1_amd64.deb
dpkg --install libpng12-0_1.2.54-1ubuntu1_amd64.deb
</pre>

= Install google-chrome =

<pre>
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
dpkg -i google-chrome-stable_current_amd64.deb
</pre>

confirm autoupdate is enabled, observe dl.google.com is present in the list of repositories:
<pre>
apt update
...
Get:5 https://dl.google.com/linux/chrome/deb stable/main amd64 Packages [1,094 B]
...
</pre>

FOLLOWING IS OBSOLETE:

Instructions from here:
https://www.ubuntuupdates.org/ppa/google_chrome?dist=stable

<pre>
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-tmp.list'
apt update
apt install google-chrome-stable
/bin/rm -f /etc/apt/sources.list.d/google-tmp.list
</pre>

= Install amanda client =

ONLY ON MACHINES THAT HOST HOME DIRECTORIES

* apt install amanda-client
* edit /etc/amandahosts
<pre>
amanda.triumf.ca amanda amdump
</pre>
* check permissions on /etc/amandahosts:
<pre>
root@daq00:/var/log/amanda# ls -l /etc/amandahosts
-rw------- 1 backup backup 49 Jan 27 10:48 /etc/amandahosts
</pre>
* fix if needed: chown backup.backup /etc/amandahosts; chmod a= /etc/amandahosts; chmod u=wr /etc/amandahosts
* edit /etc/amanda-security.conf, add this line:
<pre>
runtar:gnutar_path=/usr/bin/tar
</pre>

On the amanda machine:

* in amanda disklist, use dump type "bsdtcp-comp-user-tar"
* su - amanda and run amcheck -c daily daq00
<pre>
-bash-4.1$ amcheck -c daily daq00

Amanda Backup Client Hosts Check
--------------------------------
Client check: 1 host checked in 0.092 seconds. 0 problems found.

(brought to you by Amanda 3.3.7p1.git.685ff76d)
</pre>

= Enable rc.local =

For reasons unknown, Ubuntu LTS 20.04 does not enable /etc/rc.local. Do this:

<pre>
cd ~/git/scripts
git pull
cp -n -v etc/rc.local /etc/
chmod a+rx /etc/rc.local
cp etc/rc-local.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable rc-local
systemctl start rc-local
systemctl status rc-local
</pre>

= Remove unwanted packages =

<pre>
apt purge bash-completion # broken, adds unwanted "\" if "ls -l $ROOTSYS/<tab>"
apt purge zsys # broken, do not use
apt purge sddm # login manager
apt purge avahi-daemon avahi-autoipd # not sure what it does, observed using 100% CPU
apt purge modemmanager # probes all serial ports to see if it's a modem
</pre>

= Disable unwanted services =

<pre>
systemctl disable mpd
systemctl disable snapd
systemctl disable ModemManager
systemctl --global mask tracker-extract-3.service
systemctl --global mask tracker-miner-fs-3.service
systemctl daemon-reload
</pre>

= Disable sleep and suspend =

note: we see some computers randomly shutdown or go to sleep, log files indicates the "sleep" or "suspend" button was pushed by user, but no such buttons actually exist. this is the fix for this:

<pre>
systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target systemd-suspend.service systemd-hybrid-sleep.service
</pre>

= Enable crontab @reboot for MIDAS =

startup scripts have a bug - cron @reboot entries for normal users can run before autofs is ready, so if the home directory is on autofs/NFS, it cannot be accessed and the cron job fails. If MIDAS is supposed to be started by cron @reboot, it will not start (there *will* be an error message in /var/log/cron).

<pre>
mkdir /etc/systemd/system/cron.service.d
echo -e "[Unit]\nAfter=ypbind.service autofs.service\n" > /etc/systemd/system/cron.service.d/local.conf
systemctl daemon-reload
systemctl cat cron.service
</pre>

Explore the systemd dependency tree using "systemctl list-dependencies" maybe with "--all".

Visualize the exact boot sequence from previous boot: "systemd-analyze plot > xxx.svg", look at the svg file using a web browser.

Crontab entry to start midas: (install in the midas user crontab, not root crontab)

<pre>
su - midasuser
crontab -l
#@reboot /bin/bash -l -c "/home/trinat/bin/start-daq-applications"
#@reboot /bin/tcsh -c "/home/trinat/bin/start-daq-applications"
</pre>

= Install apache httpd proxy for midas and elog =

This will configure the HTTPS/SSL certificate using "certbot" and "letsencrypt" and configure an HTTPS web server using apache2.

First, configure apache2:

* execute these commands:
<pre>
apt -y install apache2
cd /etc/apache2
</pre>
* create new file conf-available/ssl-daq14.conf # use actual hostname instead of daq14
<pre>
SSLSessionCache shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout 300
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
SSLCryptoDevice builtin
</pre>
* create new file sites-available/daq14-ssl.conf # use actual hostname instead of daq14
<pre>
<IfModule mod_ssl.c>
<VirtualHost *:443>
ServerName daq14.triumf.ca
DocumentRoot /var/www/html
ErrorLog /var/log/apache2/daq14.log
SSLEngine on
# note SSLProtocol, SSLCipherSuite and some other settings are overwritten by /etc/letsencrypt/options-ssl-apache.conf
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SEED:!IDEA:!RC4
## use port specified in elogd.cfg
#ProxyPass /elog/ http://localhost:8082/ retry=1
## use mhttpd port
#ProxyPass / http://localhost:8080/ retry=1
Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"
<Location />
SSLRequireSSL
AuthType Basic
AuthName "DAQ password protected site"
Require valid-user
# create password file: touch /etc/apache2/htpasswd
# to add new user or change password: htpasswd /etc/apache2/htpasswd username
AuthUserFile /etc/apache2/htpasswd
</Location>
#SSLCertificateFile /root/server.cert
#SSLCertificateKeyFile /root/server.key
</VirtualHost>
</IfModule>
</pre>
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* stop apache2 from listening on port 80: edit /etc/apache2/ports.conf, comment-out the line "Listen 80"
* enable ssl module
* enable new configurations
<pre>
a2enmod ssl
a2enmod headers
a2enmod proxy
a2enmod proxy_http
a2enconf ssl-daq14
a2ensite daq14-ssl
</pre>
* disable default ssl sites
<pre>
a2dissite 000-default-le-ssl
a2dissite 000-default
ls -l /etc/apache2/sites-enabled/ ### should show only daq14-ssl.conf
</pre>
* check that there are no syntax problems
<pre>
apache2ctl configtest
</pre>
* enable and start apache2:
<pre>
systemctl enable apache2
systemctl restart apache2
systemctl status apache2
</pre>
* apache2 may fail to start, look in /var/log/apache2/error.log and /var/log/apache2/daq14.log
* if it says "Failed to configure ... certificate", proceed to the step for setting certbot.
* try to access https://daq14.triumf.ca
** you should see a complaint about self-signed certificate
** you should see a request for password (do not login yet)
** if you get "connection refused", HTTPS port 443 may need to be enabled in the local firewall, look at documentation for ufw.
Second, configure certbot:

(Note: as of 2018-01-18 certbot requires use of http port 80 to get the initial https certificate,
renewal can continue to use the https port 443)

(Note: as of 2019-01-?? certbot requires use of port 80 for renewals)

(Note: unsurprisingly, this requires outside access to connect with letsencrypt, so won't work if PC is only accessible from on-site network)

* check that port 80 is not used by anything:
* netstat -an | grep LISTEN | grep ^tcp | grep 80
* lsof -P | grep -i tcp | grep LISTEN | grep 80
* if lsof reports that apache2 is listening on port 80, follow the apache2 instructions above (remove "listen 80" from apache2.conf

* install certbot (if necessary open tcp port 80 in the firewall, see documentation for ufw):
<pre>
apt install certbot python3-certbot-apache
certbot certonly --standalone --installer apache
</pre>
* then answer questions:
* "activate HTTPS for daq14.triumf.ca" - say ok
* "enter email address" - enter your own email address
* "please read terms..." - read the terms and say "agree"
* it will take a few moments...
* "congratulations..." - say ok.
<pre>
certbot install --apache --cert-name daq14.triumf.ca
</pre>
* then answer questions:
* "choose redirect..." - say "1" (no redirect)
* look inside /etc/apache2/sites-enabled/daq14-ssl.conf to see that SSLCertificateFile & co point to certbot certificates in
/etc/letsencrypt/live/daq14.triumf.ca/
* to check current renewal and to update the certbot config file in /etc/letsencrypt/renewal, run this:
<pre>
certbot renew --standalone --installer apache --force-renewal
</pre>

NOTE: this certificate will expire in 3 months, automatic renewal should work with current version of certbot

Third, activate password protection:

* as shown in the config file above, create password file and initial user: (replace "midas" with specific username)
<pre>
touch /etc/apache2/htpasswd
htpasswd /etc/apache2/htpasswd midas
</pre>

* restart apache2
<pre>
systemctl restart apache2
systemctl status apache2
</pre>

From here:
* enable proxy for MIDAS mhttpd - uncomment redirect in the config file above
* enable proxy for ELOG - ditto
<pre>
a2enmod proxy
a2enmod proxy_http
apache2ctl configtest
systemctl restart apache2
</pre>

* try accessing MIDAS https://daq14.triumf.ca/ (make sure mhttpd is running)
* if it's not working, check odb setting FIXME!
* try accessing ELog https://daq14.triumf.ca/elog/ (make sure elogd is running)
* if it's not working, check elogd.cfg file and make sure
<pre>
SSL = 0
</pre>

NOTE: if certbot fails with errors about 'module' object has no attribute 'pyopenssl',
try this: pip install requests==2.6.0

== generate self-signed certificate ==

<pre>
# cd $HOME
# openssl req -nodes -new -x509 -keyout server.key -out server.cert -days 1001
...+....+..+..........+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+..+...+.........+......+.+...+...+.....+...............+.........+...+.+......+...+...........+....+...+..+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+......+.+...+..+.......+..+...+.......+......+...+..+...+......+....+...............+..+...+....+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
......+......+.+..+......+.+......+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.....+......+.+.........+......+.....+.+..+...+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+....+......+.....+...+...+.......+..+.+........+.+...+......+..+..........+..+.+...........+...+.......+......+.....+.......+...+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:CH
State or Province Name (full name) [Some-State]:Geneve
Locality Name (eg, city) []:CERN
Organization Name (eg, company) [Internet Widgits Pty Ltd]:CERN
Organizational Unit Name (eg, section) []:ALPHA experiment
Common Name (e.g. server FQDN or YOUR name) []:alphacpc05.cern.ch
Email Address []:
root@alphacpc05:~#
root@alphacpc05:~#
root@alphacpc05:~# ls -l
-rw-r--r-- 1 root root 1375 juil. 10 21:43 server.cert
-rw------- 1 root root 1708 juil. 10 21:42 server.key
root@alphacpc05:~# systemctl restart apache2
</pre>

= Enable elog PDF preview =

NOTE: looks like U-24 already has this correctly.

see https://stackoverflow.com/questions/52998331/imagemagick-security-policy-pdf-blocking-conversion

* xemacs -nw /etc/ImageMagick-6/policy.xml
* remove this section at the end:
<pre>

<policy domain="coder" rights="none" pattern="PS" />
<policy domain="coder" rights="none" pattern="PS2" />
<policy domain="coder" rights="none" pattern="PS3" />
<policy domain="coder" rights="none" pattern="EPS" />
<policy domain="coder" rights="none" pattern="PDF" />
<policy domain="coder" rights="none" pattern="XPS" />
</pre>

= Install Jupyter notebook =

<pre>
From https://jupyter.org/install
apt install python3-pip
pip install jupyterlab
pip install notebook
~/.local/bin/jupyter notebook
watch the http://localhost:8888 URL that it printed
say "no" to offer to start firefox (it will not work!)
URL is: http://localhost:8888/tree?token=xxx
from the machine where you are running the web browser (i.e. google-chrome), run (replace trinat@trinatdaq with the username and machine name where you started jupyter)
open a new shell and run: ssh -v trinat@trinatdaq -L 8888:localhost:8888
in the web browser, open http://localhost:8888
this gives us the login page
in the password or token entry field, put the token from the "tree?token=xxx" above (printed by jupyter on startup)
push button "login"
jupyter page should open with the list of files in the trinat home directory
congratulate Brian with full success
</pre>

= Install ZFS quota report =

If there are any ZFS volumes, install script to report disk and quota usage

<pre>
cd ~/git/scripts/quotareport
git pull
mkdir /var/www/html/zfsquotareport
cp -pv ~/git/scripts/quotareport/sorttable.js /var/www/html/zfsquotareport/
ln -s $PWD/zfsquotareport.perl /etc/cron.daily/
touch /etc/crontab
</pre>

If httpd is configured to redirect "/" to MIDAS mhttpd:
* add following to /etc/apache2/sites-enabled/xxx-ssl.conf in front of "ProxyPass / ..."
* run "systemctl reload apache2"
<pre>
## do not proxy zfs quota report directory
ProxyPass /zfsquotareport/ !
</pre>

= Install PHP =

* apt install php libapache2-mod-php
* systemctl restart apache2
* create /var/www/html/info.php
<pre>
<?php

phpinfo();
</pre>
* open https://daq00.triumf.ca/info.php

= Configure TRIUMF printers =

<pre>
systemctl stop cups
systemctl stop cups-browsed.service
systemctl disable cups
systemctl disable cups-browsed.service
systemctl stop snap.cups.cupsd.service
systemctl stop snap.cups.cups-browsed.service
systemctl disable snap.cups.cupsd.service
systemctl disable snap.cups.cups-browsed.service
echo "ServerName printers.triumf.ca" > /etc/cups/client.conf
lpstat -a
</pre>

= Enable core dumps =

By default, Ubuntu LTS 20.04 installs the apport package
which disabled core dumps from user applications. (google it up!).
It is not meant to do this and documentation claims that
it is not installed and not enabled by default. Oh, well...

<pre>
apt purge apport
apt autoremove ### will remove apport-symptoms and a few other packages
</pre>

After this, core dumps are written to file "core" in the current directory.
See /proc/sys/kernel/core_pattern and /proc/sys/kernel/core_uses_pid.

Enable core dump file names to include process id, add following to /etc/rc.local

<pre>
echo "echo 1 > /proc/sys/kernel/core_uses_pid" >> /etc/rc.local
</pre>

= Enable debugger =

By default, Ubuntu LTS 20.04 does not permit debugger to attach and debug
already running programs. To enable it, add following to /etc/rc.local

<pre>
echo "echo 0 > /proc/sys/kernel/yama/ptrace_scope" >> /etc/rc.local
</pre>

= Disable Ubuntu Pro nag =

best I can tell, impossible at this time.

== do not do this ==

!!! does nothing !!!

<pre>
pro config set apt_news=false
</pre>

== do not do this ==

!!! breaks automatic updates because 20apt-esm-hook.conf is missing !!!

If "apt upgrade" requests Ubuntu Pro or esm-apps, disable the nag:
<pre>
/bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf
</pre>

== do not do this ==

!!! likely same as above, breaks automatic updates !!!

* comment out /etc/apt/apt.conf.d/20apt-esm-hook.conf

== do not do this ==

!!! removes too many packages !!!

<pre>
apt remove ubuntu-pro-client
</pre>

= Update packages =

* apt update # update package list
* apt upgrade # install updated packages and update "kept back" packages
* apt autoremove # remove packages that apt thinks should be removed

= Cleanup residual configs =

* apt list '~c'
* apt purge '~c'

= Install firefox-esr =

install firefox-esr here

= Finish installation =

Congratulations. There is nothing more to do!

* reboot
<pre>
shutdown -r now
</pre>

= Update to new version of Ubuntu =

* run "do-release-upgrade -c"
* if it does not report new release Ubuntu 24, check /etc/update-manager/release-upgrades has "Prompt=lts"

== Update Ubuntu LTS 20.04 to LTS 22.04 ==

<pre>
apt remove zsys
</pre>

=== daqubuntu ===

<pre>
# reboot to clear out all updates
# vi /etc/update-manager/release-upgrades # set "Prompt=normal"
# do-release-upgrade -c
Checking for a new Ubuntu release
New release '22.04 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
# do-release-upgrade
...
say yes...
...
login.defs, say "Y" (erase local changes, use packaged version)
/etc/systemd/resolved.conf, say "Y" (same as above)
firefox snap, say yes
unable to reach snap store, say "skip"
/etc/gmond.conf, say "Y"
/var/yp/Makefile, say "install the package maintainer's version"
/etc/ypserv.conf, same thing
/etc/ypserv.securenets, same thing
/etc/default/nis, same thing
/etc/speech-dispatcher/modules/mary-generic.conf, same thing
/etc/apt/apt.conf.d/50unattended-upgrades, same thing
...
278 packages are going to be removed, say yes
...
restart required, say yes
...
no ping... yes ping...
...
ssh daqubuntu, ok
apt update, fail, DNS does not work, "host security.ubuntu.com" does not resolve.
fix resolver per https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Disable_NetworkManager
apt update, apt upgrade now works, 0 packages to update
NIS does not work.
</pre>

=== midm9a ===

<pre>
login.defs
firefox snap
gmond.conf
ypserv
/etc/default/nis
unattended-upgrades
amanda-security.conf
remove obsolete (no)
reboot
configure dns
reenable nis
</pre>

=== daq17 ===

<pre>
firefox snap
imagemagick policy.xml
gmond.conf
chrony.conf
/var/yp/Makefile
ypserv.conf
ypserv.securenets
/etc/default/nis
50unattended-upgrades
</pre>

=== daq00 ===

per https://serverpilot.io/docs/how-to-upgrade-ubuntu-20.04-to-22.04/

<pre>
do-release-upgrade -f DistUpgradeViewNonInteractive
</pre>

if it exists "too soon" without doing anything, run it without "-f xxx", most likely it does not like something about this machine. in case of daq00 it did not like how the EFI partitions were mounted. after fixing it, non-interactive upgrade was successful.

=== isdaq08 ===

* prepare
<pre>
cd ~/git/scripts
git pull
cd ~
apt -y install debsums
</pre>
* check for modified config files that make upgrade unhappy, deal with all files reported by debsums.
<pre>
root@isdaq08:~# debsums -ce
/etc/ganglia/gmond.conf
/etc/yp.conf
/etc/apt/apt.conf.d/10periodic
root@isdaq08:~#
</pre>
* restore original /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
</pre>
* apt remove ganglia-monitor
* apt remove nis
* "debsums -ce" is now empty

Run the upgrade:

* do-release-upgrade -f DistUpgradeViewNonInteractive

Post upgrade:

* configure DNS
* apt -y install linux-generic-hwe-22.04
* /bin/cp -v ~/git/scripts/etc/99apt-conf-ko /etc/apt/apt.conf.d/ # restore nightly updates
* /bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf # remove the ubuntu-pro nag
* install missing packages
* restore ganglia
* restore nis
* check zpool status, may need zpool upgrade
* reboot

== upgrade U-22 to U-24 ==

generic instructions. below are notes from upgrades of specific machines.

NOTE: at CERN saw installation getting stuck on restart of autofs (automount, rsyslogd 100% CPU, huge /var/log/syslog), stop autofs before upgrade?
<pre>
systemctl stop autofs
</pre>

<pre>
cleanup zsys
</pre>

<pre>
maybe: if snap is already removed, remove firefox to prevent snap from reinstalling.
install non-snap firefox-esr
</pre>

<pre>
remove thunderbird to prevent installation of snap thunderbird in U-24
</pre>

<pre>
mount EFI partition as /boot/efi, otherwise upgrade bombs
usually mount /dev/sda1 /boot/efi
</pre>

<pre>
debsums -ce
apt -y remove desktop-base # causes installer crash
apt -y remove thunderbird # avoid forced conversion to snap
apt update
apt -y upgrade
apt -y autoremove
do-release-upgrade -c # confirm upgrade will be to U-24
do-release-upgrade
# say "y" to all questions
# after installation starts, accept default answers to all questions
# should run for about 1 hour or so
# after upgrade finishes
apt update
apt -y upgrade
apt -y autoremove
shutdown -r now
</pre>

post upgrade:

* check zpool status, may need zpool upgrade
* cd /etc/apt/sources.list.d, reenable 3rd party repos (mozilla, google, etc)

if installation bombs:

<pre>
apt update
apt upgrade # will tell us to run "apt --fix-broken install", do it
apt --fix-broken install
apt update
apt upgrade # should resume the upgrade, will run for a long time
apt update
apt upgrade # should do nothing
apt autoremove
shutdown -r now
</pre>

=== daqubuntu, U-24 ===

* prepare
<pre>
cd ~/git/scripts
git pull
cd ~
apt -y install debsums
</pre>
* check for modified config files that make upgrade unhappy, deal with all files reported by debsums.
<pre>
root@daqubuntu:~# debsums -ce
/etc/ganglia/gmond.conf
debsums: missing file /etc/init.d/nis (from nis package)
/etc/default/nis
/etc/ypserv.conf
/etc/ypserv.securenets
/var/yp/Makefile
/etc/update-manager/release-upgrades
/etc/apt/apt.conf.d/10periodic
/etc/yp.conf
root@daqubuntu:~#
* restore original /etc/apt/apt.conf.d/10periodic
<pre>
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
</pre>
* apt remove ganglia-monitor
* apt remove nis
* apt autoremove
* restore original release-upgrades: "Prompt: lts"
* "debsums -ce" is now empty

Check for upgrade:

<pre>
root@daqubuntu:~# do-release-upgrade -c
Checking for a new Ubuntu release
There is no development version of an LTS available.
To upgrade to the latest non-LTS development release
set Prompt=normal in /etc/update-manager/release-upgrades.
root@daqubuntu:~#
</pre>

Run the upgrade:

* do-release-upgrade -f DistUpgradeViewNonInteractive

Post upgrade:

* configure DNS
* apt -y install linux-generic-hwe-22.04
* /bin/cp -v ~/git/scripts/etc/99apt-conf-ko /etc/apt/apt.conf.d/ # restore nightly updates
* /bin/rm /etc/apt/apt.conf.d/20apt-esm-hook.conf # remove the ubuntu-pro nag
* install missing packages
* restore ganglia
* restore nis
* check zpool status, may need zpool upgrade
* cd /etc/apt/sources.list.d, reenable 3rd party repos (mozilla, google, etc)
* reboot

=== daq14, U-20-22-24 ===

* apt update, apt upgrade
* apt -y install linux-image-generic-hwe-20.04 linux-tools-virtual-hwe-20.04 ### install kernel 5.15
* shutdown -r now
* stuck waiting for daq14 to shutdown...
* reboot into kernel 5.15
* ???
<pre>
cd ~/git/scripts
git pull
cd ~
apt -y install debsums
</pre>
* debsums -ce
<pre>
/etc/apache2/ports.conf
/etc/dnsmasq.conf
/etc/ganglia/gmond.conf
/etc/yp.conf
/etc/sudoers
</pre>
* apache2 restore original ports.conf, uncomment "Listen 80"
* cp -pv /etc/dnsmasq.conf.dpkg-dist /etc/dnsmasq.conf
* apt remove ganglia-monitor
* edit /etc/yp.conf, remove everything after "# ypserver ypserver.network.com"
* "debsums -ce" is now empty
* do-release-upgrade -f DistUpgradeViewNonInteractive
* runs for a long time
* stuck on "/etc/default/nis", type "Y", press enter, nothing for a bit, then resumes running
* finished
* configure DNS
* reboot
* have kernel 6.8
* apt update; apt upgrade
* apt upgrade guile-2.2-libs ### would not auto-update, "kept back", has to be done by hand
* apt autoremove
* debsums -ce
<pre>
debsums: missing file /etc/init.d/nis (from nis package)
/etc/default/nis
</pre>
* diff /etc/default/nis.dpkg-dist /etc/default/nis
* cp -pv /etc/default/nis.dpkg-dist /etc/default/nis
* debsums -ce
<pre>
debsums: missing file /etc/init.d/nis (from nis package)
</pre>
* we ignore this and run the update
* do-release-upgrade -c
<pre>
Checking for a new Ubuntu release
New release '24.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
</pre>
* do-release-upgrade -f DistUpgradeViewNonInteractive
* bombs out without any error messages
* in /var/log/dist-upgrade/main.log reports "Failed to find a replacement for xapp" and other packages
* apt remove xapp usrmerge ureadahead thunderbird-gnome-support
* no go, complains about even more packages.
* apt list | grep installed | grep -v jammy ### show packages installed from non-ubuntu sources
* remove all packages marked "install,local" ### ubuntu updater does not know where they came from and so cannot update them.
* apt remove desktop-base ### not happy about this package in /var/log/dist-upgrade/apt.log
* apt autoremove
* do-release-upgrade -f DistUpgradeViewNonInteractive
* running for a long time...

=== alpha04 U-20-24 ===

* apt update, apt upgrade, apt autoremove
* reboot into latest kernel (already done)
* debsums -ce
<pre>
root@alpha04:~# debsums -ce
/etc/dnsmasq.conf
/etc/ganglia/gmond.conf
/etc/default/nis
/etc/yp.conf
root@alpha04:~#
</pre>
* move /etc/dnsmasq.conf to /etc/dnsmasq.d/alpha04.conf
* apt remove dnsmasq
* apt remove ganglia-monitor
* apt remove nis
* apt autoremove
* debsums -ce ### is now empty
* do-release-upgrade -f DistUpgradeViewNonInteractive
* it runs for a long time...
* complained about /etc/fwupd config files, not sure why...
* finished
* apt update, apt upgrade, apt autoremove
* restore dnsmasq: apt install dnsmasq, systemctl status dnsmasq
* restore ganglia, per instructions
* restore NIS: apt -y install rpcbind nis, ypwhich, ypwhich -m
* zpool upgrade rpool ### also upgrade any other zfs pools, see zpool status
* remove unwanted packages, per instructions
* run gonodeinfo
* reboot
* done

== vera00 U20-22-24 ==

* everything same as daq14 for U20-22
* kernel is still 5.15
* U22-24 is going...
* stuck for a few minutes on /etc/fwupd config files
* have kernel 6.8.0-49
* same steps as daq14
* reboot
* same steps as daq14
* done

== phaarmonster U22-24 ==

* debsums -ce
<pre>
/etc/amandahosts
/etc/apache2/ports.conf
/etc/ganglia/gmond.conf
</pre>
* do-release-upgrade -c ### reports U24.04.1
* do-release-upgrade -f DistUpgradeViewNonInteractive ### bombs
* apt remove desktop-base; apt autoremove
* do-release-upgrade ### (interactive) runs ok
* bombed !!!
* apt upgrade spews errors and tells us to run "apt --fix-broken install"
* apt --fix-broken install ### runs
* bombs with thunderbird snap errors
* again... no go
* thunderbird snap complains about mounting /home, but /home is a symlink
* rm /home, mkdir /home
* again, runs ok
* asks about /etc/fwupd/fwupd.conf - say "Y" to install updated package version
* apt install completes
* apt update; apt upgrade ### running for a long time...
* finished
* install missing packages, etc
* reboot
* long wait... came back
* DNS does now work, systemd-resolved missing, apt install systemd-resolved, "configure DNS"
* done.

== isdaq10 U22-24 ==

* do-release-upgrade -f DistUpgradeViewNonInteractive
* bombed on desktop-base, "apt remove desktop-base; apt autoremove"
* bombed with errors
* running "apt --fix-broken install"
* complained about thunderbird snap
* complained about /etc/fwupd/fwupd.conf (say Y)
* finished ok
* apt update
* apt upgrade ### reports "1382 upgraded, 140 newly installed, 0 to remove and 29 not upgraded"

== iris01 ==

* debsums -ce
<pre>
/etc/ganglia/gmond.conf
/etc/default/nis
/etc/yp.conf
</pre>
* apt remove desktop-base
* apt remove thunderbird
* apt autoremove
* do-release-upgrade -f DistUpgradeViewNonInteractive
* bombed with dpkg errors:
<pre>
hplip wants wrong python
ganglia-monitor wants wrong libapr1
sssd-ad wants bunch of wrong libraries
xemacs21-mule
libnfsidmap1 wants libldap2
adn so forth...
</pre>
* apt --fix-broken install -y
* bombs on ganglia - ganglia group absent from /etc/groups
* fix group ganglia by hand, remove ganglia group from NIS on isdaq00
* apt --fix-broken install
* apt upgrade

== iris00 ==

* stuck on "autofs: restarting", login as root, kill iris midas, kill automount, systemctl restart autofs, noble got unstuck, ctrl-c systemctl restart autofs
* noble running...
* bombed with dpkg errors
* check ganglia user, group - both ok
* apt --fix-broken install
* apt upgrade

== tigstore01 ==

* no bomb-out

== midm9b ==

* apt remove desktop-base thunderbird
* bombed
* apt --fix-broken-install
* apt upgrade

= Upgrade to new version of Debian =

https://www.debian.org/releases/bookworm/amd64/release-notes/ch-upgrading.en.html

== 32-bit VME processor Debian 11 to 12 ==

* cd git/scripts; git pull; cd ~
* apt update
* apt upgrade
* edit /etc/apt/sources.list
<pre>
deb http://deb.debian.org/debian/ bookworm main
#deb http://deb.debian.org/debian/ bullseye main
#deb http://deb.debian.org/debian/ buster main
#deb-src http://deb.debian.org/debian/ bullseye main
</pre>
* apt update
* apt upgrade --without-new-pkgs
* apt full-upgrade
* apt list '~c'; apt purge '~c' # purge left-over config files [residual-config]
* reboot

= Ubuntu package manager =

* apt-get install xxx # install package xxx
* apt-get update
* apt-get upgrade
* apt-get dist-upgrade
* apt-get autoremove # remove automatically installed packages required by a removed package
* apt-get remove xxx # remove package xxx
* apt-cache search . # list all available packages
* apt-cache show "." | grep ^Package # list al available packages
* apt-cache madison root-system # show all available versions of package root-system
* apt list # list all installed packages
* dpkg --listfiles libpng16-16 # list all files from this package
* apt list --installed # list all installed packages
* dpkg -S /bin/bash # what package provides this file?
* dpkg -L bash # what files provided by this package?
* debsums -ce # show modified config files
* apt-config dump # show apt configuration
* apt purge '~c' # purge all [residual-config] packages
* ls -l /var/lib/dpkg/info/ # show post-install scripts

= Ubuntu zsys =

NOTE: DO NOT USE ZSYS, see https://github.com/ubuntu/zsys/issues/218 and https://github.com/ubuntu/zsys/issues/230

* scripted removal of old snapshots (replace "echo zfs destroy" with "zfs destroy")
<pre>
zfs list -t all | cut -f1 -d " " | grep autozsys | xargs -n1 echo zfs destroy
</pre>

* manual removal of old snapshots
<pre>
zsysctl show
zsysctl state remove xy69ye -s
zsysctl state remove xy69ye
zsysctl state remove xy69ye -u wheel
</pre>
* apt remove zsys

NOTE: old zsys snapshots must be cleaned manually, "zsysctl state remove xxx --system" is broken and does not remove user data snapshots

* manages system snapshots
* documentation: https://github.com/ubuntu/zsys
* documentation: (go to next article via link "newer" at the bottom) https://didrocks.fr/2020/05/21/zfs-focus-on-ubuntu-20.04-lts-whats-new/
* ubuntu 20.04 bug, too many snapshots cause /boot to become full and updates fail. https://github.com/ubuntu/zsys/issues/155
* solution: use custom /etc/zsys.conf, limit number of snapshots to 10, see trinatdaq:/etc/zsys.conf
* zsys commands:
<pre>
update-grub # list of all snapshots, errors if some snapshots are broken
zsysctl state remove lnc0k7 --system # remove snapshot
xemacs -nw /etc/zsys.conf; zsysctl service reload; zsysctl service gc # cause gc to run with new settings in zsys.conf
zfs list -r -t snapshot -o name,used,referenced,creation bpool/BOOT # list snapshots
zsysctl show # show snapshots
</pre>

= Ubuntu cloning =

to clone a ubuntu image:

<pre>
cd /nfsroot/lxcpet
emacs -nw etc/hostname ### change hostname
emacs -nw etc/mailname ### change hostname (debian 11)
emacs -nw etc/defaultdomain ### change the NIS domainname
emacs -nw etc/yp.conf ### change the NIS server
cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
emacs -nw root/.ssh/authorized_keys ### update root ssh keys
</pre>

= Ubuntu boot loader =

== maintenance commands ==

* update-initramfs -v -u
* grub-install /dev/sda

= Convert from single to dual mirrored ZFS SSD =

Assuming Ubuntu LTS 22.04 with "instal on ZFS" option, we will
add a second SSD, configure ZFS to use both SSDs in mirrored
configuration and setup grub to boot from either SSD. This
is intended to create a full redundant system where failure
of either SSD does not break the system.

== partition ==

* identify first SSD
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* connect second SSD of identical size
<pre>
root@midm9b:~# ./smart-status.perl
Disk model serial temperature realloc pending uncorr CRC err RRER Errors Link
/dev/sda WD Blue SA510 2.5 250GB 22243Z803769 24 . ? ? . ? . 6.0
/dev/sdb WD Blue SA510 2.5 250GB 22243Z803852 25 . ? ? . ? . 6.0
root@midm9b:~#
</pre>
* if second SSD is not autodetected, reboot
* Clone partition table automatically
If both SSDs are identical size, use this simpler method of duplicating the partition table:
<pre>
root@midm9b:~# sfdisk -d /dev/sda > part_table
root@midm9b:~# grep -v ^label-id part_table | sed -e 's/, *uuid=[0-9A-F-]*//' | sfdisk /dev/sdb
</pre>
The grep and sed in the second command are there to prevent disk ID and partition IDs from being cloned. Alternatively the part_table file can be edited manually to remove the label-id line and the uuid entries from the individual partitions.

* Clone partition table manually (e.g. for different size disks)
* list partition table of first SSD:
<pre>
root@midm9b:~# fdisk -l /dev/sda
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>
* create identical partitions on second SSD, use sector numbers from above.
<pre>
root@midm9b:~# gdisk /dev/sdb
GPT fdisk (gdisk) version 1.0.8

Partition table scan:
MBR: not present
BSD: not present
APM: not present
GPT: not present

Creating new GPT entries in memory.

Command (? for help): n
Partition number (1-128, default 1):
First sector (34-488397134, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-488397134, default = 488397134) or {+-}size{KMGTP}: 1050623
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): ef00
Changed type of partition to 'EFI system partition'

Command (? for help): n
Partition number (2-128, default 2):
First sector (34-488397134, default = 1050624) or {+-}size{KMGTP}:
Last sector (1050624-488397134, default = 488397134) or {+-}size{KMGTP}: 5244927
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 8200
Changed type of partition to 'Linux swap'

Command (? for help): n
Partition number (3-128, default 3):
First sector (34-488397134, default = 5244928) or {+-}size{KMGTP}:
Last sector (5244928-488397134, default = 488397134) or {+-}size{KMGTP}: 9439231
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): be00
Changed type of partition to 'Solaris boot'

Command (? for help): n
Partition number (4-128, default 4):
First sector (34-488397134, default = 9439232) or {+-}size{KMGTP}:
Last sector (9439232-488397134, default = 488397134) or {+-}size{KMGTP}:
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): bf00
Changed type of partition to 'Solaris root'

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sdb.
The operation has completed successfully.
root@midm9b:~# fdisk -l /dev/sda /dev/sdb
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 951A4174-B4C6-400D-99F5-BE9B5627FA8E

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root

Disk /dev/sdb: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: EB251739-30C6-422F-A505-5887B5A0B603

Device Start End Sectors Size Type
/dev/sdb1 2048 1050623 1048576 512M EFI System
/dev/sdb2 1050624 5244927 4194304 2G Linux swap
/dev/sdb3 5244928 9439231 4194304 2G Solaris boot
/dev/sdb4 9439232 488397134 478957903 228.4G Solaris root
root@midm9b:~#
</pre>

== update ZFS pools ==

* identify second SSD partitions
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part2
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part2 -> ../../sdb2
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
</pre>
* convert bpool from single disk to mirrored disk:
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~# zpool attach bpool 99e03dc0-7d4d-f24b-8fa1-f042b9f135db /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3
root@midm9b:~# zpool status bpool
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors
</pre>
* convert rpool
<pre>
root@midm9b:~# ls -l /dev/disk/by-id/ata*part4
lrwxrwxrwx 1 root root 10 Jan 20 18:37 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803769-part4 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jan 20 19:34 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 -> ../../sdb4
root@midm9b:~# zpool attach rpool f6fd54f8-3af7-b943-ae3d-a4e480537fb9 /dev/disk/by-id/ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4
root@midm9b:~# zpool status rpool
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Jan 20 19:40:45 2023
5.83G scanned at 664M/s, 2.92M issued at 332K/s, 9.11G total
0B resilvered, 0.03% done, no estimated completion time
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
root@midm9b:~#
</pre>
* wait for resilver to complete
<pre>
root@midm9b:~# zpool status
pool: bpool
state: ONLINE
scan: resilvered 247M in 00:00:00 with 0 errors on Fri Jan 20 19:39:40 2023
config:

NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
99e03dc0-7d4d-f24b-8fa1-f042b9f135db ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part3 ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
scan: resilvered 9.65G in 00:00:36 with 0 errors on Fri Jan 20 19:41:21 2023
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
f6fd54f8-3af7-b943-ae3d-a4e480537fb9 ONLINE 0 0 0
ata-WD_Blue_SA510_2.5_250GB_22243Z803852-part4 ONLINE 0 0 0

errors: No known data errors
</pre>

== update boot loader ==

INSTALL SYSLINUX: https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#EFI_boot_using_syslinux

DO *NOT* DO THE FOLOWING:

* create and mount EFI partitions:
<pre>
root@midm9b:~# mkfs.msdos /dev/sdb1
root@midm9b:~# mkdir /boot/efi-sda
root@midm9b:~# mkdir /boot/efi-sdb
root@midm20c:~# blkid | grep vfat ### identify UUID
/dev/sdb1: UUID="DD89-5081" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="d0cb6be4-2f67-5b42-9b26-9e6905e9f774"
/dev/sdc1: UUID="D970-86BA" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="e6d3b5b9-a512-44a2-9205-1a4db06ed2a2"
/dev/sda1: UUID="DDA1-044C" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="6dc9dff0-1c13-8045-a906-7803d3074c70"
root@midm20c:~# cat /etc/fstab | grep vfat ### add mount points with correct UUID
#UUID=D970-86BA /boot/efi vfat umask=0022,fmask=0022,dmask=0022 0 1
UUID=DDA1-044C /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022 0 1
UUID=DD89-5081 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022 0 1
root@midm9b:~# mount -a
root@midm9b:~# df -kl
Filesystem 1K-blocks Used Available Use% Mounted on
...
/dev/sda1 523244 13720 509524 3% /boot/efi
/dev/sdb1 523244 4 523240 1% /boot/efi-sdb
...
root@midm9b:~# rsync -av /boot/efi/ /boot/efi-sdb/
sending incremental file list
EFI/
...
root@midm9b:~# ls -l /boot/efi-sda
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~# ls -l /boot/efi-sdb
total 8
drwxr-xr-x 4 root root 4096 Jan 19 23:26 EFI
drwxr-xr-x 5 root root 4096 Jan 19 23:26 grub
root@midm9b:~#
</pre>
* add systemd "nofail" flag to /etc/fstab, without this, systemd will stop booting if one SSD is missing
<pre>
daq00:~$ cat /etc/fstab | grep vfat
#UUID=31A7-24BE /boot/efi vfat umask=0022,fmask=0022,dmask=0022 0 1
/dev/sda1 /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
/dev/sdb1 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
</pre>
* setup script to update grub on second SSD, it must be run manually after every kernel update
<pre>
root@midm9b:~# ln -s ~/git/scripts/etc/update_efi_grub.perl ~/
root@midm9b:~# ~/update_efi_grub.perl -u
EFI dir: /boot/efi-sda
/boot/efi-sda: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sda/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sda: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sda/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
EFI dir: /boot/efi-sdb
/boot/efi-sdb: update grub: rsync -av --delete-after --modify-window=2 /boot/efi/grub/ /boot/efi-sdb/grub
building file list ... done

sent 5,313 bytes received 11 bytes 10,648.00 bytes/sec
total size is 7,944,644 speedup is 1,492.23
/boot/efi-sdb: update efi: rsync -av --delete-after --modify-window=2 /boot/efi/EFI/ /boot/efi-sdb/EFI
building file list ... done

sent 216 bytes received 11 bytes 454.00 bytes/sec
total size is 5,452,378 speedup is 24,019.29
root@midm9b:~#
</pre>

= Disable NetworkManager =

== Debian-12 ==

* use netplan, https://www.debian.org/doc/manuals/debian-reference/ch05.en.html#_the_modern_network_configuration_for_cloud
* apt install netplan.io
* systemctl enable systemd-networkd
* systemctl restart systemd-networkd
* create /etc/netplan/50-dhcp.yaml
<pre>
network:
version: 2
ethernets:
all-en:
match:
name: "en*"
dhcp4: true
dhcp6: true
ignore-carrier: true ### do not drop IP address if network link drops
</pre>
* netplan apply
* netplan try
* ifconfig -a ### to check IP address settings
* netstat -rn ### to check default route
* cat /etc/resolv.conf ### to check DNS
* ls -l /run/systemd/netif/leases ### systemd-networkd dhcp leases
* NOTE: without "ignore-carrier" it will drop the IP address if network link drops, re-do dhcp when links comes back
* NOTE: wait-network-online will wait for all interfaces to get an IP address

== Ubuntu-20 ==

NOTE: THIS IS BROKEN IN UBUNTU LTS 22.04

NetworkManager is useful for configuring dynamic
network interfaces, i.e. laptops that often move
between networks, or connect to multiple choice
of wifi networks, etc.

For machines with statically configured network interfaces,
NetworkManager is not necessary.

As it has been observed to become confused and observed
to malfunction when network links go up and down (it keeps
unnecessarily reconfiguring the ip address, etc), it can
be usefuil to disable it.

* list all network interfaces
<pre>
# /bin/ls -1 /sys/class/net/
enp0s31f6
lo
</pre>
* edit /etc/network/interfaces:
<pre>
rename enp0s31f6=eth0
auto eth0
iface eth0 inet static
address 142.90.120.94/19
gateway 142.90.100.18
</pre>
* statically configure systemd-resolved
** create /etc/systemd/resolved.conf.d/resolved.conf with this contents:
<pre>
[Resolve]
DNS=142.90.100.19
Domains=triumf.ca
</pre>
** systemctl restart systemd-resolved
** resolvectl
** systemd-analyze cat-config systemd/resolved.conf
* disable NetworkManager
<pre>
systemctl disable NetworkManager
</pre>
* reboot

== U-22, U-24 ifup-ko ==

Network configuration of modern linux is confused. There are at least 3 configuration methods, each with different shortcomings:
* the old ifup method is barely documented
* NetworkManager is well documented and tooled, but sometimes does strange things
* systemd-networkd is mysterious, and likely to do strange stuff, like all systemd stuff
* netplan is the latest method, configuration is simple but uses NetworkManager or systemd-networkd as backend.

This is a solution for a specific situation of fixed computer with one fixed wired interface and maybe one or more additional interfaces for fixed wired private networks.

Install /etc/ifup-ko, edit it with IP addresses of main and additional interfaces, let systemd run it in the right place in the boot sequence replacing NetworkManager and NetworkManager-wait-online .

As bonus, /etc/ifup-ko waits up to 10 seconds for the main interface link to come up. If this is not needed, comment it out from the script.

* prepare
<pre>
cd ~/git/scripts
git pull
cd ifup-ko
make install
</pre>
* confirm interface names
<pre>
systemctl start ifup-ko ### should finish immediately or after 10 seconds
systemctl status ifup-ko -n 1000 ### observe list of interfaces is correct, name of main interface is correct
</pre>
* edit /etc/ifup-ko
** add host IP address to the "ifconfig" line
** add gateway IP address to the "ip route add" line
* test
<pre>
systemctl start ifup-ko ### should finish immediately
systemctl status ifup-ko -n 1000 ### observe everything is configured as expected
</pre>
* cut-over
<pre>
systemctl disable networkd-dispatcher
systemctl disable NetworkManager
systemctl disable wpa_supplicant # if no Wifi or Wifi not in use
</pre>
* reboot

= Disable systemd-networkd =

On netbooted machines, systemd-networkd should be disabled - when "apt upgrade" runs
and needs to update and configure systemd, it will stop systemd-networkd, which
will stop the network, which will stop the NFS mounted root filesystem, which will stop
the machine.

<pre>
systemctl disable systemd-networkd.service
systemctl disable systemd-networkd.socket
systemctl mask systemd-networkd.service
systemctl mask systemd-networkd.socket
</pre>

= Configure ECC memory =

== Configure EDAC ==

* apt install edac-utils rasdaemon

=== Intel i3-2120 ===
<pre>
root@musr00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X9SCL/X9SCM
root@musr00:~# edac-ctl --status
edac-ctl: drivers not loaded.
</pre>

=== Intel E-2236 ===
<pre>
root@daq00:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SCM-F
root@daq00:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq00:~# edac-util
edac-util: No errors to report.
root@daq00:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
</pre>
* check edac sysfs files (Intel)
<pre>
root@daq00:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ce_noinfo_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 max_location
-r--r--r-- 1 root root 4096 Jan 25 15:10 mc_name
drwxr-xr-x 2 root root 0 Jan 25 15:10 power
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank0
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank1
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank2
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank3
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank4
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank5
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank6
drwxr-xr-x 3 root root 0 Jan 25 15:10 rank7
--w------- 1 root root 4096 Jan 25 15:10 reset_counters
-r--r--r-- 1 root root 4096 Jan 25 15:10 seconds_since_reset
-r--r--r-- 1 root root 4096 Jan 25 15:10 size_mb
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_count
-r--r--r-- 1 root root 4096 Jan 25 15:10 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Jan 25 15:10 uevent
root@daq00:~#
</pre>

=== Intel E3-1270 v6 ===
<pre>
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
root@wheel-SYS-5019S-M:~/git/scripts# edac-ctl --status
edac-ctl: drivers are loaded.
root@grsnis01:~# edac-util
edac-util: No errors to report.
root@grsnis01:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@grsnis01:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 max_location
-r--r--r-- 1 root root 4096 Feb 19 12:35 mc_name
drwxr-xr-x 2 root root 0 Feb 19 12:35 power
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank0
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank1
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank2
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank3
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank4
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank5
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank6
drwxr-xr-x 3 root root 0 Feb 19 12:35 rank7
--w------- 1 root root 4096 Feb 19 12:35 reset_counters
-r--r--r-- 1 root root 4096 Feb 19 12:35 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 19 12:35 size_mb
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_count
-r--r--r-- 1 root root 4096 Feb 19 12:35 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 19 12:35 uevent
root@grsnis01:~#
</pre>

=== Intel E3-1245 v6 ===

<pre>
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro X11SSH-F
[root@alphagdaq ~]# edac-ctl --status
edac-ctl: drivers are loaded.
[root@alphagdaq ~]# edac-util
edac-util: No errors to report.
[root@alphagdaq ~]# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
[root@alphagdaq ~]# ras-mc-ctl --layout
+-----------------------------------------------+
| mc0 |
| csrow0 | csrow1 | csrow2 | csrow3 |
----------+-----------------------------------------------+
channel1: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB | 8192 MB | 8192 MB |
----------+-----------------------------------------------+
[root@alphagdaq ~]# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#3channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#3channel#1 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#2channel#0 0 0
[root@alphagdaq ~]# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SSH-F
[root@alphagdaq ~]# ras-mc-ctl --summary
DBD::SQLite::db prepare failed: no such table: mc_event at /usr/sbin/ras-mc-ctl line 1129.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1130.
[root@alphagdaq ~]#
</pre>

=== AMD 3700X ===

(memory is non-ECC)

<pre>
root@daq13:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@daq13:~#
root@daq13:~#
root@daq13:~# edac-ctl --status
edac-ctl: drivers not loaded.
root@daq13:~# edac-util
edac-util: Error: No memory controller data found.
root@daq13:~# edac-util -s
edac-util: EDAC drivers loaded. No memory controllers found
root@daq13:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 2 root root 0 Jan 25 15:26 power
lrwxrwxrwx 1 root root 0 Jan 21 16:16 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Jan 21 16:16 uevent
</pre>

(memory is ECC)

<pre>
root@trinatdaq:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-E GAMING
root@trinatdaq:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@trinatdaq:~# edac-util
edac-util: No errors to report.
root@trinatdaq:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Dec 15 13:04 mc0
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
lrwxrwxrwx 1 root root 0 Dec 13 18:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dec 13 18:31 uevent
root@trinatdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ce_noinfo_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 max_location
-r--r--r-- 1 root root 4096 Dec 15 13:04 mc_name
drwxr-xr-x 2 root root 0 Dec 15 13:04 power
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank4
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank5
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank6
drwxr-xr-x 3 root root 0 Dec 15 13:04 rank7
--w------- 1 root root 4096 Dec 15 13:04 reset_counters
-rw-r--r-- 1 root root 4096 Dec 15 13:04 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Dec 15 13:04 seconds_since_reset
-r--r--r-- 1 root root 4096 Dec 15 13:04 size_mb
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_count
-r--r--r-- 1 root root 4096 Dec 15 13:04 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Dec 15 13:04 uevent
root@trinatdaq:~#
</pre>

=== AMD 5000G ===

* no linux driver for AMD 5000-series "G" CPU
* no mention of ECC in the BIOS settings
* unclear status of ECC support in AMD documentation (sais only "pro" "G" CPUs have ECC)
* unclear status of ECC support in ASUS documentation (web page out of date)

=== AMD 5600X ===

<pre>
root@daq17:~# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. ROG STRIX B550-XE GAMING WIFI
root@daq17:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@daq17:~# edac-util
edac-util: No errors to report.
root@daq17:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@daq17:~# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 7 root root 0 Aug 19 19:27 mc0
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
lrwxrwxrwx 1 root root 0 May 10 10:11 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 May 10 10:11 uevent
root@daq17:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ce_noinfo_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 max_location
-r--r--r-- 1 root root 4096 Aug 19 19:27 mc_name
drwxr-xr-x 2 root root 0 Aug 19 19:27 power
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank4
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank5
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank6
drwxr-xr-x 3 root root 0 Aug 19 19:27 rank7
--w------- 1 root root 4096 Aug 19 19:27 reset_counters
-rw-r--r-- 1 root root 4096 Aug 19 19:27 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Aug 19 19:27 seconds_since_reset
-r--r--r-- 1 root root 4096 Aug 19 19:27 size_mb
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_count
-r--r--r-- 1 root root 4096 Aug 19 19:27 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Aug 19 19:27 uevent
root@daq17:~#
</pre>

=== AMD 3955WX ===

<pre>
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --mainboard
edac-ctl: mainboard: ASUSTeK COMPUTER INC. Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~/git/scripts/quotareport# edac-ctl --status
edac-ctl: drivers are loaded.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util
edac-util: No errors to report.
root@alphasuperdaq:~/git/scripts/quotareport# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@alphasuperdaq:~/git/scripts/quotareport# ls -l /sys/devices/system/edac/mc
total 0
drwxr-xr-x 19 root root 0 Dez 12 04:48 mc0
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
lrwxrwxrwx 1 root root 0 Dez 9 05:31 subsystem -> ../../../../bus/edac
-rw-r--r-- 1 root root 4096 Dez 9 05:31 uevent
root@alphasuperdaq:~/git/scripts/quotareport#
root@alphasuperdaq:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ce_noinfo_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 max_location
-r--r--r-- 1 root root 4096 Feb 28 22:19 mc_name
drwxr-xr-x 2 root root 0 Dez 12 04:48 power
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank0
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank1
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank10
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank11
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank12
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank13
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank14
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank15
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank2
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank3
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank4
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank5
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank6
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank7
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank8
drwxr-xr-x 3 root root 0 Dez 12 04:48 rank9
--w------- 1 root root 4096 Feb 28 22:19 reset_counters
-rw-r--r-- 1 root root 4096 Feb 28 22:19 sdram_scrub_rate
-r--r--r-- 1 root root 4096 Feb 28 22:19 seconds_since_reset
-r--r--r-- 1 root root 4096 Feb 28 22:19 size_mb
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_count
-r--r--r-- 1 root root 4096 Feb 28 22:19 ue_noinfo_count
-rw-r--r-- 1 root root 4096 Feb 28 22:19 uevent
root@alphasuperdaq:~#
root@alphasuperdaq:~# ras-mc-ctl --layout
Use of uninitialized value $max_pos[3] in modulus (%) at /usr/sbin/ras-mc-ctl line 868.
Use of uninitialized value $d in numeric ge (>=) at /usr/sbin/ras-mc-ctl line 869.
Use of uninitialized value $d in sprintf at /usr/sbin/ras-mc-ctl line 872.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 791.
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| mc0 |
| csrow0 | csrow1 |
| channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 | channel0 | channel1 | channel2 | channel3 | channel4 | channel5 | channel6 | channel7 |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

0: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB |
----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
root@alphasuperdaq:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#2 0 0
mc#0csrow#1channel#7 0 0
mc#0csrow#0channel#3 0 0
mc#0csrow#1channel#4 0 0
mc#0csrow#1channel#2 0 0
mc#0csrow#0channel#7 0 0
mc#0csrow#1channel#3 0 0
mc#0csrow#0channel#4 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#5 0 0
mc#0csrow#0channel#6 0 0
mc#0csrow#0channel#1 0 0
mc#0csrow#0channel#5 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#1channel#6 0 0
root@alphasuperdaq:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: ASUSTeK COMPUTER INC. model Pro WS WRX80E-SAGE SE WIFI
root@alphasuperdaq:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@alphasuperdaq:~#
</pre>

=== AMD 7700X ===

<pre>
root@dsfe05:~# apt install edac-utils
root@dsfe05:~# edac-ctl --mainboard
edac-ctl: mainboard: Supermicro H13SAE-MF
root@dsfe05:~# edac-ctl --status
edac-ctl: drivers are loaded.
root@dsfe05:~# edac-util
edac-util: No errors to report.
root@dsfe05:~# edac-util -s
edac-util: EDAC drivers are loaded. 1 MC detected
root@dsfe05:~# ls -l /sys/devices/system/edac/mc/mc0
total 0
-r--r--r-- 1 root root 4096 May 14 09:33 ce_count
-r--r--r-- 1 root root 4096 May 14 09:33 ce_noinfo_count
-r--r--r-- 1 root root 4096 May 14 09:33 max_location
-r--r--r-- 1 root root 4096 May 14 09:33 mc_name
drwxr-xr-x 2 root root 0 May 14 09:33 power
drwxr-xr-x 3 root root 0 May 14 09:33 rank4
drwxr-xr-x 3 root root 0 May 14 09:33 rank5
--w------- 1 root root 4096 May 14 09:33 reset_counters
-r--r--r-- 1 root root 4096 May 14 09:33 seconds_since_reset
-r--r--r-- 1 root root 4096 May 14 09:33 size_mb
-r--r--r-- 1 root root 4096 May 14 09:33 ue_count
-r--r--r-- 1 root root 4096 May 14 09:33 ue_noinfo_count
-rw-r--r-- 1 root root 4096 May 14 09:33 uevent
root@dsfe05:~#
</pre>

== Configure rasdaemon ==

<pre>
apt install rasdaemon
</pre>
<pre>
systemctl enable rasdaemon
systemctl restart rasdaemon
systemctl status rasdaemon
</pre>

<pre>
● rasdaemon.service - RAS daemon to log the RAS events
Loaded: loaded (/lib/systemd/system/rasdaemon.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-01-25 15:16:37 PST; 3min 5s ago
Main PID: 2477175 (rasdaemon)
Tasks: 1 (limit: 76958)
Memory: 17.1M
CGroup: /system.slice/rasdaemon.service
└─2477175 /usr/sbin/rasdaemon -f -r

Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: ras:extlog_mem_event event enabled
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Listening to events for cpus 0 to 11
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: Enabled event ras:extlog_mem_event
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mc_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording aer_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording extlog_event events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording mce_record events
Jan 25 15:16:37 daq00.triumf.ca rasdaemon[2477175]: rasdaemon: Recording arm_event events
</pre>

== Get reports ==

* Intel 2x32GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --layout
+-------------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-------------------------+
channel1: | 16384 MB | 16384 MB |
channel0: | 16384 MB | 16384 MB |
----------+-------------------------+
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#1channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#0channel#1 0 0
root@daq00:~#
</pre>

* Intel 4x16GB ECC DIMMs
<pre>
root@daq00:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#0channel#1 0 0
mc#0csrow#2channel#0 0 0
mc#0csrow#0channel#0 0 0
mc#0csrow#2channel#1 0 0
mc#0csrow#1channel#0 0 0
mc#0csrow#1channel#1 0 0
mc#0csrow#3channel#0 0 0
mc#0csrow#3channel#1 0 0
root@daq00:~#
root@daq00:~# ras-mc-ctl --layout
+-----------------------+
| mc0 |
| csrow0 | csrow1 |
----------+-----------------------+
channel1: | 8192 MB | 8192 MB |
channel0: | 8192 MB | 8192 MB |
----------+-----------------------+
root@daq00:~#
root@daq00:~#
root@daq00:~#
root@daq00:~# ras-mc-ctl --print-labels
ras-mc-ctl: Error: No dimm labels for Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model X11SCM-F
root@daq00:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

DBD::SQLite::db prepare failed: no such table: devlink_event at /usr/sbin/ras-mc-ctl line 1181.
Can't call method "execute" on an undefined value at /usr/sbin/ras-mc-ctl line 1182.
root@daq00:~#
</pre>

note: ubuntu LTS 22.04 DBD::SQLite::db error is not there.

* AMD 7700 2x32GB DDR5 ECC DIMMs

<pre>
root@dsfe05:~# systemctl status rasdaemon
● rasdaemon.service - RAS daemon to log the RAS events
Loaded: loaded (/lib/systemd/system/rasdaemon.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2024-05-14 09:36:43 PDT; 33ms ago
Process: 4088418 ExecStartPost=/usr/sbin/rasdaemon --enable (code=exited, status=0/SUCCESS)
Main PID: 4088417 (rasdaemon)
Tasks: 1 (limit: 37300)
Memory: 788.0K
CPU: 5ms
CGroup: /system.slice/rasdaemon.service
└─4088417 /usr/sbin/rasdaemon -f -r

May 14 09:36:43 dsfe05 rasdaemon[4088417]: ras:aer_event event enabled
May 14 09:36:43 dsfe05 rasdaemon[4088417]: Enabled event ras:aer_event
May 14 09:36:43 dsfe05 rasdaemon[4088417]: mce:mce_record event enabled
May 14 09:36:43 dsfe05 rasdaemon[4088417]: Enabled event mce:mce_record
May 14 09:36:43 dsfe05 rasdaemon[4088417]: ras:extlog_mem_event event enabled
May 14 09:36:43 dsfe05 rasdaemon[4088417]: Enabled event ras:extlog_mem_event
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording mc_event events
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording aer_event events
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording extlog_event events
May 14 09:36:43 dsfe05 rasdaemon[4088417]: rasdaemon: Recording mce_record events
root@dsfe05:~# ras-mc-ctl --layout
Use of uninitialized value $max_pos[3] in modulus (%) at /usr/sbin/ras-mc-ctl line 907.
Use of uninitialized value $d in numeric ge (>=) at /usr/sbin/ras-mc-ctl line 908.
Use of uninitialized value $d in sprintf at /usr/sbin/ras-mc-ctl line 911.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
Use of uninitialized value $pos[3] in join or string at /usr/sbin/ras-mc-ctl line 830.
+-----------------------------------------------------------------------------------------------+
| mc0 |
| csrow0 | csrow1 | csrow2 | csrow3 |
| channel0 | channel1 | channel0 | channel1 | channel0 | channel1 | channel0 | channel1 |
----+-----------------------------------------------------------------------------------------------+

0: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB |
----+-----------------------------------------------------------------------------------------------+
root@dsfe05:~# ras-mc-ctl --error-count
Label CE UE
mc#0csrow#2channel#1 0 0
mc#0csrow#2channel#0 0 0
root@dsfe05:~# ras-mc-ctl --print-labels
ras-mc-ctl: Error: No dimm labels for Supermicro model H13SAE-MF
root@dsfe05:~# ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: Supermicro model H13SAE-MF
root@dsfe05:~# ras-mc-ctl --summary
No Memory errors.

No PCIe AER errors.

No Extlog errors.

No MCE errors.
root@dsfe05:~#
</pre>

= sensors =

== VME CPU V7865 ==

add to /etc/rc.local

<pre>
modprobe coretemp
modprobe lm75
modprobe lm90
modprobe max1668
</pre>

available sensors:

<pre>
root@lxdaq23:~# sensors
max6657-i2c-0-4c
Adapter: SMBus I801 adapter at 0400
temp1: +29.1°C (low = -55.0°C, high = +105.0°C)
(crit = +105.0°C, hyst = +95.0°C)
temp2: +31.5°C (low = -55.0°C, high = +105.0°C)
(crit = +105.0°C, hyst = +95.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Core 0: +25.0°C (crit = +100.0°C)
Core 1: +25.0°C (crit = +100.0°C)

max1805-i2c-0-18
Adapter: SMBus I801 adapter at 0400
temp1: +35.0°C (low = -55.0°C, high = +127.0°C)
temp2: +63.0°C (low = -55.0°C, high = +127.0°C)
temp3: FAULT (low = -55.0°C, high = +127.0°C) ALARM (HIGH)

lm75-i2c-0-48
Adapter: SMBus I801 adapter at 0400
temp1: +34.5°C (high = +80.0°C, hyst = +75.0°C)

root@lxdaq23:~#
</pre>

== ASUS P7P55D EVO ==

* BIOS version 2101

<pre>
root@iris01:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Core 0: +34.0°C (high = +83.0°C, crit = +99.0°C)
Core 1: +37.0°C (high = +83.0°C, crit = +99.0°C)
Core 2: +38.0°C (high = +83.0°C, crit = +99.0°C)
Core 3: +35.0°C (high = +83.0°C, crit = +99.0°C)

nouveau-pci-0100
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.05 V)
temp1: +46.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

atk0110-acpi-0
Adapter: ACPI interface
Vcore Voltage: 864.00 mV (min = +0.80 V, max = +1.60 V)
+3.3V Voltage: 3.38 V (min = +2.97 V, max = +3.63 V)
+5V Voltage: 5.04 V (min = +4.50 V, max = +5.50 V)
+12V Voltage: 12.15 V (min = +10.20 V, max = +13.80 V)
CPU Fan Speed: 968 RPM (min = 600 RPM, max = 7200 RPM)
Chassis1 Fan Speed: 1288 RPM (min = 600 RPM, max = 7200 RPM)
Chassis2 Fan Speed: 1316 RPM (min = 600 RPM, max = 7200 RPM)
Power Fan Speed: 0 RPM (min = 0 RPM, max = 7200 RPM)
CPU Temperature: +34.0°C (high = +45.0°C, crit = +45.5°C)
MB Temperature: +30.0°C (high = +45.0°C, crit = +46.0°C)

root@iris01:~#
</pre>

== ASUS Z97-WS ==

* BIOS version 2704
* load sensors drivers
<pre>
echo modprobe coretemp >> /etc/rc.local
echo modprobe nct6775 >> /etc/rc.local
</pre>
* in /boot/grub/grub.cfg, add: GRUB_CMDLINE_LINUX_DEFAULT="acpi_enforce_resources=no"
* update grub and reboot: grub-mkconfig -o /boot/grub/grub.cfg

<pre>
root@isdaq08:~# sensors
acpitz-acpi-0
Adapter: ACPI interface
temp1: +27.8°C
temp2: +29.8°C

nct6791-isa-0290
Adapter: ISA adapter
Vcore: 888.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
AVCC: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
+3.3V: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.99 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 0.00 V (min = +0.00 V, max = +0.00 V)
3VSB: 3.44 V (min = +0.00 V, max = +0.00 V) ALARM
Vbat: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 0.00 V (min = +0.00 V, max = +0.00 V)
in11: 840.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in12: 0.00 V (min = +0.00 V, max = +0.00 V)
in13: 0.00 V (min = +0.00 V, max = +0.00 V) ALARM
in14: 0.00 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1041 RPM (min = 0 RPM)
fan2: 1040 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan6: 0 RPM (min = 0 RPM)
SYSTIN: +34.0°C (high = +0.0°C, hyst = +0.0°C) ALARM sensor = thermistor
CPUTIN: +41.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: -128.0°C sensor = thermistor
AUXTIN1: -128.0°C sensor = thermistor
AUXTIN2: +35.0°C sensor = thermistor
AUXTIN3: +127.0°C sensor = thermistor
PECI Agent 0: +41.0°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
PCH_DIM0_TEMP: +0.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +42.0°C (high = +80.0°C, crit = +100.0°C)
Core 0: +42.0°C (high = +80.0°C, crit = +100.0°C)
Core 1: +40.0°C (high = +80.0°C, crit = +100.0°C)
Core 2: +39.0°C (high = +80.0°C, crit = +100.0°C)
Core 3: +39.0°C (high = +80.0°C, crit = +100.0°C)

root@isdaq08:~#
</pre>

== ASUS Z170-DELUXE ==

* BIOS version 3801
* load sensors drivers
<pre>
echo modprobe coretemp >> /etc/rc.local
echo modprobe jc42 >> /etc/rc.local
echo modprobe lm92 >> /etc/rc.local
echo modprobe nct6775 >> /etc/rc.local
</pre>
* in /etc/default/grub, add: GRUB_CMDLINE_LINUX_DEFAULT="acpi_enforce_resources=no"
* update grub and reboot: grub-mkconfig -o /boot/grub/grub.cfg

<pre>
root@iris00:~# sensors
nct6793-isa-0290
Adapter: ISA adapter
in0: 600.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 144.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in6: 0.00 V (min = +0.00 V, max = +0.00 V)
in7: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.14 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1000.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in10: 600.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 592.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 968.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1370 RPM (min = 0 RPM)
fan2: 1437 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan6: 0 RPM (min = 0 RPM)
SYSTIN: +32.0°C (high = +98.0°C, hyst = +95.0°C) sensor = thermistor
CPUTIN: +42.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: -128.0°C sensor = thermistor
AUXTIN1: +50.0°C sensor = thermistor
AUXTIN2: +22.0°C sensor = thermistor
AUXTIN3: +28.0°C sensor = thermistor
PECI Agent 0: +50.0°C (high = +98.0°C, hyst = +95.0°C)
(crit = +100.0°C)
PECI Agent 0 Calibration: +42.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
TSI2_TEMP: +3892314.0°C
TSI3_TEMP: +3892314.0°C
TSI4_TEMP: +3892314.0°C
TSI5_TEMP: +3892314.0°C
TSI6_TEMP: +3892314.0°C
TSI7_TEMP: +3892314.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

jc42-i2c-0-1a
Adapter: SMBus I801 adapter at f040
temp1: +36.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

jc42-i2c-0-18
Adapter: SMBus I801 adapter at f040
temp1: +34.8°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

jc42-i2c-0-1b
Adapter: SMBus I801 adapter at f040
temp1: +35.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

jc42-i2c-0-19
Adapter: SMBus I801 adapter at f040
temp1: +36.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +52.0°C (high = +84.0°C, crit = +100.0°C)
Core 0: +52.0°C (high = +84.0°C, crit = +100.0°C)
Core 1: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 2: +48.0°C (high = +84.0°C, crit = +100.0°C)
Core 3: +47.0°C (high = +84.0°C, crit = +100.0°C)

root@iris00:~#
</pre>

== ASUS Z390M-PRO GAMING (WI-FI) ==

* BIOS 3006
* load sensors drivers
<pre>
echo modprobe coretemp >> /etc/rc.local
echo modprobe nct6775 >> /etc/rc.local
</pre>

<pre>
root@daq18:~# sensors
nct6798-isa-0290
Adapter: ISA adapter
in0: 696.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.42 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 208.00 mV (min = +0.00 V, max = +0.00 V)
in6: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.42 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.17 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.07 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.36 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.33 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in14: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 0 RPM (min = 0 RPM)
fan2: 1131 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 1006 RPM (min = 0 RPM)
fan6: 0 RPM (min = 0 RPM)
fan7: 0 RPM (min = 0 RPM)
SYSTIN: +32.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
CPUTIN: +29.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
AUXTIN0: +25.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
AUXTIN1: +7.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
AUXTIN2: +8.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
AUXTIN3: +24.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +100.0°C) sensor = thermistor
AUXTIN4: +83.0°C (high = +80.0°C, hyst = +75.0°C) ALARM
(crit = +100.0°C)
PECI Agent 0 Calibration: +29.0°C (high = +80.0°C, hyst = +75.0°C)
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +39.0°C (high = +82.0°C, crit = +100.0°C)
Core 0: +39.0°C (high = +82.0°C, crit = +100.0°C)
Core 1: +33.0°C (high = +82.0°C, crit = +100.0°C)
Core 2: +32.0°C (high = +82.0°C, crit = +100.0°C)
Core 3: +31.0°C (high = +82.0°C, crit = +100.0°C)
Core 4: +31.0°C (high = +82.0°C, crit = +100.0°C)
Core 5: +30.0°C (high = +82.0°C, crit = +100.0°C)

acpitz-acpi-0
Adapter: ACPI interface
temp1: +27.8°C

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: +28.0°C

root@daq18:~#
</pre>

== ASUS H110M-A/M.2 ==

* BIOS version 4202
* echo modprobe coretemp >> /etc/rc.local
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@midpol:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +33.0°C (high = +80.0°C, crit = +100.0°C)
Core 0: +33.0°C (high = +80.0°C, crit = +100.0°C)
Core 1: +30.0°C (high = +80.0°C, crit = +100.0°C)

acpitz-acpi-0
Adapter: ACPI interface
temp1: +27.8°C (crit = +119.0°C)
temp2: +29.8°C (crit = +119.0°C)

nct6793-isa-0290
Adapter: ISA adapter
in0: 368.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 152.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in6: 928.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.14 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1000.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in10: 152.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in11: 128.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in12: 136.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in13: 120.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 136.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1004 RPM (min = 0 RPM)
fan2: 1143 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan6: 0 RPM (min = 0 RPM)
SYSTIN: +118.0°C (high = +98.0°C, hyst = +95.0°C) sensor = thermistor
CPUTIN: +29.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +30.0°C sensor = thermistor
AUXTIN1: +112.0°C sensor = thermistor
AUXTIN2: +111.0°C sensor = thermistor
AUXTIN3: +110.0°C sensor = thermistor
PECI Agent 0: +31.0°C (high = +98.0°C, hyst = +95.0°C)
(crit = +100.0°C)
PECI Agent 0 Calibration: +36.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
TSI2_TEMP: +3892314.0°C
TSI3_TEMP: +3892314.0°C
TSI4_TEMP: +3892314.0°C
TSI5_TEMP: +3892314.0°C
TSI6_TEMP: +3892314.0°C
TSI7_TEMP: +3892314.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

root@midpol:~#
</pre>

== ASUS P9X79 WS ==

* https://www.asus.com/supportonly/P9X79%20WS/HelpDesk_Manual/
* BIOS version 4802
* modprobe nct6775
* modprobe coretemp

<pre>
root@daq14:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 0: +29.0°C (high = +82.0°C, crit = +100.0°C)
Core 1: +24.0°C (high = +82.0°C, crit = +100.0°C)
Core 2: +35.0°C (high = +82.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +82.0°C, crit = +100.0°C)

nouveau-pci-0200
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.00 V)
temp1: +39.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

nct6776-isa-0290
Adapter: ISA adapter
Vcore: 1.04 V (min = +0.00 V, max = +1.74 V)
in1: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
AVCC: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
+3.3V: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 2.04 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 904.00 mV (min = +0.00 V, max = +0.00 V) ALARM
3VSB: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
Vbat: 3.30 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 1265 RPM (min = 0 RPM)
fan2: 1909 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +34.0°C (high = +0.0°C, hyst = +0.0°C) ALARM sensor = thermistor
CPUTIN: +58.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermal diode
AUXTIN: +31.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
PECI Agent 0: +31.0°C (high = +80.0°C, hyst = +75.0°C)
(crit = +96.0°C)
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
PCH_MCH_TEMP: +0.0°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

root@daq14:~#
</pre>

== ASUS TUF GAMING B550M-PLUS WIFI II ==

* BIOS 2803, 2806
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@midm9a:~# sensors
nct6798-isa-0290
Adapter: ISA adapter
in0: 488.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.33 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.82 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 1.01 V (min = +0.00 V, max = +0.00 V) ALARM
in14: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
fan1: 0 RPM (min = 0 RPM)
fan2: 760 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan7: 1264 RPM (min = 0 RPM)
SYSTIN: +25.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +22.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +95.0°C sensor = thermistor
AUXTIN1: +25.0°C sensor = thermistor
AUXTIN2: +25.0°C sensor = thermistor
AUXTIN3: +25.0°C sensor = thermistor
PECI Agent 0 Calibration: +23.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +32.4°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

amdgpu-pci-0800
Adapter: PCI adapter
vddgfx: 1.45 V
vddnb: 993.00 mV
edge: +28.0°C
PPT: 20.00 W

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +33.4°C

root@midm9a:~#
</pre>

== ASUS ASUS ROG STRIX B550-XE GAMING WIFI ==

* BIOS 2423, 2604
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@daq13:~# sensors
nct6798-isa-0290
Adapter: ISA adapter
in0: 344.00 mV (min = +0.00 V, max = +1.74 V)
in1: 992.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in6: 216.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.41 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.30 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.81 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in11: 960.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.03 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 280.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 845 RPM (min = 0 RPM)
fan2: 998 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +28.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +27.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +94.0°C sensor = thermistor
AUXTIN1: +28.0°C sensor = thermistor
AUXTIN2: +28.0°C sensor = thermistor
AUXTIN3: +97.0°C sensor = thermistor
PECI Agent 0 Calibration: +27.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +33.6°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

amdgpu-pci-0600
Adapter: PCI adapter
vddgfx: 1.45 V
vddnb: 999.00 mV
edge: +29.0°C
PPT: 14.00 W

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: +30.0°C

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +33.9°C

root@daq13:~#
</pre>

== ASUS ASUS ROG STRIX B550-E GAMING ==

* bios 2803
* echo modprobe jc42 >> /etc/rc.local
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@daq17:~# sensors
jc42-i2c-1-1b
Adapter: SMBus PIIX4 adapter port 0 at 0b00
temp1: +25.0°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: +28.0°C

nouveau-pci-0800
Adapter: PCI adapter
GPU core: 900.00 mV (min = +0.85 V, max = +1.00 V)
temp1: +34.0°C (high = +95.0°C, hyst = +3.0°C)
(crit = +105.0°C, hyst = +5.0°C)
(emerg = +135.0°C, hyst = +5.0°C)

nct6798-isa-0290
Adapter: ISA adapter
in0: 288.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 224.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.36 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.31 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 1.79 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.06 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 280.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 208.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 843 RPM (min = 0 RPM)
fan2: 629 RPM (min = 0 RPM)
fan3: 746 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
SYSTIN: +22.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +25.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +93.0°C sensor = thermistor
AUXTIN1: +22.0°C sensor = thermistor
AUXTIN2: +22.0°C sensor = thermistor
AUXTIN3: +96.0°C sensor = thermistor
PECI Agent 0 Calibration: +25.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +27.6°C
intrusion0: ALARM
intrusion1: ALARM
beep_enable: disabled

jc42-i2c-1-1a
Adapter: SMBus PIIX4 adapter port 0 at 0b00
temp1: +23.2°C (low = +0.0°C) ALARM (HIGH, CRIT)
(high = +0.0°C, hyst = +0.0°C)
(crit = +0.0°C, hyst = +0.0°C)

asusec-isa-0000
Adapter: ISA adapter
CPU_Opt: 0 RPM
Chipset: +34.0°C
CPU: +25.0°C
Motherboard: +22.0°C
T_Sensor: -40.0°C
VRM: +31.0°C

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +28.0°C
Tccd1: +27.5°C

root@daq17:~#
</pre>

== ASUS PRIME B650-PLUS ==

* BIOS 1811
* echo modprobe nct6775 >> /etc/rc.local

<pre>
root@dsdaqgw:~# sensors
amdgpu-pci-0b00
Adapter: PCI adapter
vddgfx: 930.00 mV
vddnb: 1.19 V
edge: +38.0°C
PPT: 25.10 W

nct6799-isa-0290
Adapter: ISA adapter
in0: 920.00 mV (min = +0.00 V, max = +1.74 V)
in1: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in2: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in3: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in4: 1.02 V (min = +0.00 V, max = +0.00 V) ALARM
in5: 1.04 V (min = +0.00 V, max = +0.00 V) ALARM
in6: 320.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in7: 3.39 V (min = +0.00 V, max = +0.00 V) ALARM
in8: 3.28 V (min = +0.00 V, max = +0.00 V) ALARM
in9: 3.38 V (min = +0.00 V, max = +0.00 V) ALARM
in10: 1.28 V (min = +0.00 V, max = +0.00 V) ALARM
in11: 1.10 V (min = +0.00 V, max = +0.00 V) ALARM
in12: 1.04 V (min = +0.00 V, max = +0.00 V) ALARM
in13: 416.00 mV (min = +0.00 V, max = +0.00 V) ALARM
in14: 328.00 mV (min = +0.00 V, max = +0.00 V) ALARM
fan1: 0 RPM (min = 0 RPM)
fan2: 1253 RPM (min = 0 RPM)
fan3: 0 RPM (min = 0 RPM)
fan4: 0 RPM (min = 0 RPM)
fan5: 0 RPM (min = 0 RPM)
fan7: 0 RPM (min = 0 RPM)
SYSTIN: +33.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
CPUTIN: +35.5°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor
AUXTIN0: +78.0°C sensor = thermistor
AUXTIN1: +11.0°C sensor = thermistor
AUXTIN2: +20.0°C sensor = thermistor
AUXTIN3: +82.0°C sensor = thermistor
PECI Agent 0 Calibration: +35.5°C
PCH_CHIP_CPU_MAX_TEMP: +0.0°C
PCH_CHIP_TEMP: +0.0°C
PCH_CPU_TEMP: +0.0°C
TSI0_TEMP: +42.6°C
intrusion0: ALARM
intrusion1: OK
beep_enable: disabled

k10temp-pci-00c3
Adapter: PCI adapter
Tctl: +42.6°C
Tccd1: +36.4°C

root@dsdaqgw:~#
</pre>

= Enable CPU turbo mode =

* Intel CPU has a nominal CPU frequency (i.e. 3.4GHz) and a turbo-boost CPU frequency (i.e. 4.0GHz). Here we will enable this turbo-boost mode.
* Find out CPU capability
<pre>
root@daq01:~# lscpu | grep Hz
Model name: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
CPU MHz: 3965.803
CPU max MHz: 4000.0000
CPU min MHz: 800.0000
root@daq01:~#
</pre>
* Look up this CPU in the Intel ARK database - google for the CPU model name, i.e.
https://ark.intel.com/content/www/us/en/ark/products/88196/intel-core-i7-6700-processor-8m-cache-up-to-4-00-ghz.html
* Find current frequency settings:
<pre>
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "powersave" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 2.72 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
root@daq01:~#
</pre>
* Note the following:
** current governor is "powersave"
** "performance" governor is available
** "boost state support" is supported and active.
* Confirm CPU frequency governor:
<pre>
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
powersave
powersave
powersave
powersave
powersave
powersave
powersave
powersave
root@daq01:~#
</pre>
* Change governor to "performance":
<pre>
root@daq01:~# cpupower frequency-set --governor performance
Setting cpu: 0
Setting cpu: 1
Setting cpu: 2
Setting cpu: 3
Setting cpu: 4
Setting cpu: 5
Setting cpu: 6
Setting cpu: 7
root@daq01:~# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
performance
performance
performance
performance
performance
performance
performance
performance
root@daq01:~# cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: Cannot determine or is not supported.
hardware limits: 800 MHz - 4.00 GHz
available cpufreq governors: performance powersave
current policy: frequency should be within 800 MHz and 4.00 GHz.
The governor "performance" may decide which speed to use
within this range.
current CPU frequency: Unable to call hardware
current CPU frequency: 3.93 GHz (asserted by call to kernel)
boost state support:
Supported: yes
Active: yes
</pre>
* monitor CPU frequency:
<pre>
root@daq01:~# cpupower monitor
| Nehalem || Mperf || Idle_Stats
CPU| C3 | C6 | PC3 | PC6 || C0 | Cx | Freq || POLL | C1 | C1E | C3 | C6 | C7s | C8
0| 0.00| 0.00| 0.00| 0.00|| 88.80| 11.20| 3973|| 0.00| 0.00| 0.01| 0.02| 0.31| 0.00| 4.25
4| 0.00| 0.00| 0.00| 0.00|| 4.70| 95.30| 3945|| 0.00| 0.00| 0.00| 0.00| 0.00| 0.00| 95.03
1| 0.73| 3.70| 0.00| 0.00|| 4.52| 95.48| 3864|| 0.00| 0.01| 1.19| 0.44| 2.82| 0.00| 90.23
5| 0.73| 3.70| 0.00| 0.00|| 0.37| 99.63| 3807|| 0.00| 0.00| 0.03| 0.09| 1.70| 0.00| 97.64
2| 2.28| 12.86| 0.00| 0.00|| 1.41| 98.59| 3829|| 0.00| 0.86| 3.17| 0.46| 7.70| 0.00| 85.87
6| 2.28| 12.86| 0.00| 0.00|| 2.88| 97.12| 3856|| 0.00| 0.11| 4.56| 2.15| 10.31| 0.00| 78.99
3| 1.33| 4.81| 0.00| 0.00|| 0.99| 99.01| 3804|| 0.00| 0.49| 0.79| 0.01| 1.03| 0.00| 96.12
7| 1.34| 4.81| 0.00| 0.00|| 1.26| 98.74| 3818|| 0.00| 0.01| 2.32| 0.47| 5.02| 0.00| 90.06
root@daq01:~#
</pre>
* check that the CPU is not overheating:
<pre>
root@daq01:~# sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 0: +51.0°C (high = +84.0°C, crit = +100.0°C)
Core 1: +38.0°C (high = +84.0°C, crit = +100.0°C)
Core 2: +34.0°C (high = +84.0°C, crit = +100.0°C)
Core 3: +32.0°C (high = +84.0°C, crit = +100.0°C)
</pre>
* congratulations, we are running at 4 GHz now!

= Setup ubuntu as gateway to private network =

See also:
* https://daq.triumf.ca/DaqWiki/index.php/VME-CPU#Setup_the_boot_host_computer_.28el7.29
* http://www.triumf.info/wiki/DAQwiki/index.php/Dhcpd_on_eth1

== Steps to do ==

!!! UPDATED 16feb2024 Ubuntu-22.04.03 !!!

* assign network numbers to the private network, i.e. 192.168.1.x, 192.168.2.x, etc
* (on the gateway machine, each private network interface has to have a different network number)
* (each network interface can have multiple networks attached, via VLANs or via eth0:0, eth0:1 constructs)
* assign IP addresses on the private network, save them in /etc/hosts i.e. "hvps 192.168.1.10"
* (for simplicity, assign 192.168.1.1 to the gateway machine itself)
* (IP addresses 192.168.1.0 and 192.168.1.255 are "special", do not use them)
* setup DNS server (dnsmasq) to serve contents of /etc/hosts via DNS (otherwise, many programs will see inconsistent name to IP address mapping)
* setup DHCP server (dnsmasq) to give out the IP addresses
* setup TFTP server (dnsmasq), pxelinux and NFS for diskless booting
* setup time server (chronyd) to provide common time to all devices
* setup NAT so machines on private network can access the internet (to get OS updates, etc)
* setup NIS and NFS so machines on the private network can use common home directories
* setup rsync backup of machines on the private network

== setup hosts ==

* edit /etc/hosts
<pre>
192.168.1.101 dsfe01
... and so forth
</pre>

== setup dns and dhcp ==

!!! updated 16feb2024 for Ubuntu 22.04.3 !!!

!!! note: stock systemd-resolved remains, is configured to forward queries to dnsmasq, configured to forward queries to TRIUMF DNS !!!

!!! note: per authors of systemd, bare hostnames are not permitted, a DNS domain name must always be used. DNS domain name "dsdaq" is used in this example !!!

* apt install dnsmasq
* ensure dnsmasq starts after all interfaces are up (Ubuntu-22)
<pre>
mkdir /etc/systemd/system/dnsmasq.service.d
echo -e "[Unit]\nAfter=network-online.target\n" > /etc/systemd/system/dnsmasq.service.d/local.conf
</pre>
* edit /etc/dnsmasq.conf
<pre>
# /etc/dnsmasq.conf
# DNS settings
#port=0 # disable DNS function
port=53 # enable DNS function
bind-interfaces # do not collide with systemd-resolved, we use 127.0.0.1:53, they use 127.0.0.53:53
domain-needed
bogus-priv
no-resolv
#log-queries # log DNS quesries

# TRIUMF DNS settings

server=142.90.100.19
expand-hosts
domain=dsdaq
local=/dsdaq/
localmx # do not forward MX queries to TRIUMF

# DHCP settings
interface=enp1s0f0 # VX network 192.168.0.x
#interface=missing # FEP and TSP network 192.168.1.x
interface=enp1s0f1 # controls network 192.168.2.x
#dhcp-range=192.168.1.50,192.168.1.150,infinite
dhcp-range=192.168.0.0,static
dhcp-range=192.168.2.0,static
log-dhcp # log DHCP queries
#quiet-dhcp
dhcp-ignore=tag:!known
#dhcp-boot=pxelinux.0

dhcp-option=option:dns-server,192.168.0.248
dhcp-option=option:ntp-server,192.168.0.248

# TFTP settings

enable-tftp
tftp-root=/tftpboot
</pre>
* #mkdir /tftpboot ### per tftp-root (if no ZFS)
* zfs create -o mountpoint=/tftpboot rpool/tftpboot ### (if root is ZFS)
* create resolved-dsdaq.conf with main IP address of dnsmasq
<pre>
[Resolve]
DNS=192.168.0.248
Domains=dsdaq triumf.ca
</pre>
* mkdir -p /etc/systemd/resolved.conf.d/
* /bin/rm -f /etc/systemd/resolved.conf.d/*.conf
* cp resolved-dsdaq.conf /etc/systemd/resolved.conf.d/
* systemctl stop systemd-resolved.service
* systemctl disable systemd-resolved.service
* systemctl enable dnsmasq
* systemctl restart dnsmasq
* try to "ping" or "host" some names from /etc/hosts, it should work
* try to ping daq00, daq00.triumf.ca, all should work
* resolved-dsdaq.conf goes into /etc/systemd/resolved.conf.d/ of all machines on the private network
* if not using systemd-resolved, edit /etc/resolv.conf

== setup chronyd ==

* enable ntp server:
* disable systemd-timesyncd, configure and enable chronyd per instructions above
* create dsdaq.conf
<pre>
# chrony config for dsdaq server

#allow 192.168.0.0
#allow 192.168.1.0
#allow 192.168.2.0
allow all

# end
</pre>
* cp dsdaq.conf /etc/chrony/conf.d/
* systemctl restart chronyd
* chronyc tracking ### wait until time is synchronized (a few seconds)
* create dsdaq.sources # use hostname or IP address of chronyd server
<pre>
# Put this file in /etc/chrony/sources.d
# systemctl restart chrony
# chronyc sources
# chronyc tracking
server dsdaqgw iburst prefer
# end
</pre>
* dsdaq.sources goes to /etc/chrony/sources.d of all machines on the private network

== setup diskless network booting ==

=== setup pxelinux for legacy pxe boot ===

* add bits in dnsmasq.conf
<pre>
dhcp-host=ac:1f:6b:9e:7f:4a,dsfe01,infinite
dhcp-boot=pxelinux.0
dhcp-option=17,"192.168.0.251:/nfsroot/%s,vers=3,tcp"
</pre>
* setup pxelinux for Ubuntu-18
<pre>
cd ~
wget https://www.kernel.org/pub/linux/utils/boot/syslinux/4.xx/syslinux-4.03.tar.bz2
tar xjvf syslinux-4.03.tar.bz2
cd syslinux-4.03
cp -pv ./core/pxelinux.0 ./com32/hdt/hdt.c32 ./memdisk/memdisk ./com32/menu/menu.c32 /zssd/tftpboot/
</pre>
* cd /zssd/tftpboot
<pre>
wget http://ladd00.triumf.ca/tftpboot/memtest86+-4.20.iso.zip
wget http://ladd00.triumf.ca/tftpboot/memtest86+-5.01.iso.gz
wget http://ladd00.triumf.ca/tftpboot/modules.alias
wget http://ladd00.triumf.ca/tftpboot/modules.pcimap
wget http://ladd00.triumf.ca/tftpboot/pci.ids
</pre>
* mkdir pxelinux.cfg
* emacs -nw pxelinux.cfg/default
<pre>
default menu.c32
prompt 0

menu title Welcome to the DSVSLICE PXE boot menu

timeout 50

label hdt
kernel hdt.c32

label memtest86+-5.01
kernel memdisk iso initrd=memtest86+-5.01.iso.gz

label memtest86+-4.20
kernel memdisk iso initrd=memtest86+-4.20.iso.zip

label vmlinuz-5.3.0-26-generic
menu default
kernel vmlinuz-5.3.0-26-generic
append initrd=initrd.img-5.3.0-26-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.1.1:/zssd/nfsroot/dsfe01 toram ip=dhcp panic=60 BOOTIF=enp1s0f0

#end
</pre>

=== setup pxelinux for efi pxe boot ===

* https://c-nergy.be/blog/?p=13808
* add dnsmasq.conf bits. note: to use dhcp root-path, see the "nfsroot=auto" patch below and make sure to use the "dhcp-option-force" command (mkinitramfs dhcp client does not ask for root-path, we have to force-feed it).
<pre>
# uefi pxe

dhcp-boot=tag:uefipxe,uefi/syslinux.efi
dhcp-option-force=tag:fe01,option:root-path,192.168.0.248:/nfsroot/fe01

# VX network 192.168.0.x

dhcp-host=40:a6:b7:c1:d9:c5,fe01,infinite,set:uefipxe,set:fe01
</pre>
* apt install syslinux pxelinux syslinux-common syslinux-efi syslinux-utils
<pre>
mkdir /tftpboot/uefi
cp /usr/lib/SYSLINUX.EFI/efi64/syslinux.efi /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/ldlinux.e64 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/menu.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/hdt.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libutil.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libmenu.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libcom32.c32 /tftpboot/uefi/
cp /usr/lib/syslinux/modules/efi64/libgpl.c32 /tftpboot/uefi/
</pre>
* try to boot, it should bomb with "cannot load pxelinux.cfg/default"
* mkdir /tftpboot/uefi/pxelinux.cfg
* create /tftpboot/uefi/pxelinux.cfg/default, note nfsroot path is hardwired, note "http:" is used to load vmlinuz and initrd files (because tftp is super slow)
<pre>
default menu.c32
prompt 0

menu title Welcome to the DSDAQGW UEFI PXE boot menu

timeout 50

label vmlinuz-6.5.0-17-generic
kernel http://192.168.0.248:8088/uefi/vmlinuz-6.5.0-17-generic
append initrd=http://192.168.0.248:8088/uefi/initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=auto rw ip=dhcp panic=60

# append initrd=http://192.168.0.248:8088/uefi/initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.0.248:/nfsroot/fe01 rw ip=dhcp panic=60

# append initrd=initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=192.168.0.248:/nfsroot/fe01 rw ip=dhcp panic=60
# append initrd=initrd.img-6.5.0-17-generic boot=nfs root=/dev/nfs netboot=nfs nfsroot=auto ip=dhcp rw panic=60

#end
</pre>
* try to boot, it will bomb with "cannot load http://...."
* install mini_httpd on port 8088, see https://acme.com/software/mini_httpd/
<pre>
apt install mini-httpd
emacs -nw /etc/default/mini-httpd # set "START=1"
emacs -nw /etc/mini-httpd.conf # set "host=192.168.0.248", "port=8088", "data_dir=/tftpboot"
mkdir /etc/systemd/system/mini-httpd.service.d
echo -e "[Unit]\nAfter=network-online.target\n" > /etc/systemd/system/mini-httpd.service.d/local.conf
systemctl enable mini-httpd
systemctl restart mini-httpd
systemctl status mini-httpd
wget http://192.168.0.248:8088/uefi/syslinux.efi
tail -100 /var/log/mini_httpd.log
</pre>
* fix U-22 initramfs bug for "nfsroot=auto", otherwise, "nfsroot=" has to be different for each machine and you have to have separate pxelinux config files for each machine
** emacs -nw /usr/lib/initramfs-tools/etc/dhcp/dhclient-enter-hooks.d/config
** add "echo ROOTPATH=..." if it is missing
<pre>
echo "ROOTSERVER='${new_routers%% *}'"
echo "ROOTPATH='$new_root_path'"
echo "HOSTNAME='$new_host_name'"
</pre>
* fix U-24 initramfs bug for "nfsroot=auto", otherwise, "nfsroot=" has to be different for each machine and you have to have separate pxelinux config files for each machine
** emacs -nw /usr/share/initramfs-tools/dhcpcd-hooks/70-net-conf
** add "ROOTPATH=..." if it is missing
<pre>
DNSDOMAIN='${new_domain_name-}'
ROOTSERVER='${new_routers-}'
ROOTPATH='${new_root_path-}'
filename='${new_filename-}'
DHCPLEASETIME='${new_dhcp_lease_time-}'
</pre>
** regenerate initramfs (be careful you generate it for the right kernel!)
** see https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/2054482
<pre>
mkinitramfs 6.5.0-18-generic
mkinitramfs 6.8.0-51-generic -o /boot/initrd.img-6.8.0-51-generic
</pre>
* copy linux kernel and initrd
<pre>
cp /boot/vmlinuz-6.5.0-18-generic /tftpboot/uefi/
cp /boot/initrd.img-6.5.0-18-generic /tftpboot/uefi/
chmod a+r /tftpboot/uefi/*
</pre>
* try to boot, should bomb with messages about "trying to mount root filesystem"
* tail /var/log/syslog
<pre>
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 vendor class: PXEClient:Arch:00007:UNDI:003016
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 1:netmask, 2:time-offset, 3:router, 4, 5,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 6:dns-server, 12:hostname, 13:boot-file-size,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 15:domain-name, 17:root-path, 18:extension-path,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 22:max-datagram-reassembly, 23:default-ttl,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 28:broadcast, 40:nis-domain, 41:nis-server,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 42:ntp-server, 43:vendor-encap, 50:requested-address,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 51:lease-time, 54:server-identifier, 58:T1,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 59:T2, 60:vendor-class, 66:tftp-server, 67:bootfile-name,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 97:client-machine-id, 128, 129, 130, 131,
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 132, 133, 134, 135
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 next server: 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 broadcast response
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 1 option: 53 message-type 2
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 18 option: 67 bootfile-name uefi/syslinux.efi
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 12 hostname fe01
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:43:02 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 vendor class: PXEClient:Arch:00007:UNDI:003016
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 1:netmask, 2:time-offset, 3:router, 4, 5,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 6:dns-server, 12:hostname, 13:boot-file-size,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 15:domain-name, 17:root-path, 18:extension-path,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 22:max-datagram-reassembly, 23:default-ttl,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 28:broadcast, 40:nis-domain, 41:nis-server,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 42:ntp-server, 43:vendor-encap, 50:requested-address,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 51:lease-time, 54:server-identifier, 58:T1,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 59:T2, 60:vendor-class, 66:tftp-server, 67:bootfile-name,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 97:client-machine-id, 128, 129, 130, 131,
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 requested options: 132, 133, 134, 135
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 next server: 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 broadcast response
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 1 option: 53 message-type 5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 18 option: 67 bootfile-name uefi/syslinux.efi
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 12 hostname fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065885 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: error 8 User aborted the transfer received from 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/syslinux.efi to 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/syslinux.efi to 192.168.0.110
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 requested options: 1:netmask, 3:router, 6:dns-server
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 bootfile name: uefi/syslinux.efi
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 next server: 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 broadcast response
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 1 option: 53 message-type 2
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:05 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 requested options: 1:netmask, 3:router, 6:dns-server
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 bootfile name: uefi/syslinux.efi
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 next server: 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 broadcast response
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 1 option: 53 message-type 5
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 51 lease-time infinite
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:43:09 dsdaqgw dnsmasq-dhcp[3629416]: 2348065887 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/ldlinux.e64 to 192.168.0.110
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/01-40-a6-b7-c1-d9-c5 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8006E not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8006 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A800 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A80 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A8 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0A not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C0 not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: file /tftpboot/uefi/pxelinux.cfg/C not found
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/pxelinux.cfg/default to 192.168.0.110
Feb 16 20:43:09 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/menu.c32 to 192.168.0.110
Feb 16 20:43:10 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/libutil.c32 to 192.168.0.110
Feb 16 20:43:10 dsdaqgw dnsmasq-tftp[3629416]: sent /tftpboot/uefi/pxelinux.cfg/default to 192.168.0.110
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 client provides name: dsdaqgw.triumf.ca
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPDISCOVER(enp1s0f0) 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPOFFER(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 1:netmask, 28:broadcast, 2:time-offset, 3:router,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 15:domain-name, 6:dns-server, 119:domain-search,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 12:hostname, 44:netbios-ns, 47:netbios-scope,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 26:mtu, 121:classless-static-route, 42:ntp-server
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 bootfile name: uefi/syslinux.efi
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 next server: 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 1 option: 53 message-type 2
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 51 lease-time infinite
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 available DHCP subnet: 192.168.0.0/255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 client provides name: dsdaqgw.triumf.ca
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPREQUEST(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 tags: uefipxe, fe01, known, enp1s0f0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 DHCPACK(enp1s0f0) 192.168.0.110 40:a6:b7:c1:d9:c5 fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 1:netmask, 28:broadcast, 2:time-offset, 3:router,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 15:domain-name, 6:dns-server, 119:domain-search,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 12:hostname, 44:netbios-ns, 47:netbios-scope,
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 requested options: 26:mtu, 121:classless-static-route, 42:ntp-server
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 bootfile name: uefi/syslinux.efi
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 next server: 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 1 option: 53 message-type 5
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 54 server-identifier 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 51 lease-time infinite
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 1 netmask 255.255.255.0
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 28 broadcast 192.168.0.255
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 3 router 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 5 option: 15 domain-name dsdaq
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 12 hostname fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 27 option: 17 root-path 192.168.0.248:/nfsroot/fe01
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 42 ntp-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw dnsmasq-dhcp[3629416]: 3693523458 sent size: 4 option: 6 dns-server 192.168.0.248
Feb 16 20:44:54 dsdaqgw rpc.mountd[3350210]: authenticated mount request from 192.168.0.110:981 for /nfsroot/fe01 (/nfsroot/fe01)
Feb 16 20:45:07 dsdaqgw rpc.mountd[3350210]: authenticated unmount request from 192.168.0.110:859 for /nfsroot/fe01/tmp/autoDY4k5u (/nfsroot/fe01)
</pre>
* tail /var/log/mini_httpd.log
<pre>
192.168.0.110 - - [16/Feb/2024:20:43:15 -0800] "GET /uefi/vmlinuz-6.5.0-17-generic HTTP/1.0" 200 14227944 "" "Syslinux/6.04"
192.168.0.110 - - [16/Feb/2024:20:43:24 -0800] "GET /uefi/initrd.img-6.5.0-17-generic HTTP/1.0" 200 137824833 "" "Syslinux/6.04"
</pre>

=== setup efi http boot ===

https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-deployment-prep-uefi-httpboot.html

=== setup linux kernel ===

* copy the kernel files
<pre>
cd /boot
rsync -av config* initrd* System.map* vmlinuz* /tftpboot/
</pre>
* cd /tftpboot
* chmod a+r *

=== setup nfs ===

* apt-get install nfs-kernel-server
* enable NFS over UDP, edit /etc/nfs.conf add "udp=y":
<pre>
udp=y
</pre>
<pre>
systemctl restart nfs-server.service
</pre>
* emacs -nw /etc/exports
<pre>
/nfsroot/dsfe01 dsfe01(rw,no_root_squash,async,no_subtree_check)
</pre>
* enable services
<pre>
systemctl enable nfs-server
systemctl enable nfs-mountd
systemctl enable nfs-idmapd
systemctl restart nfs-server
systemctl restart nfs-mountd
systemctl restart nfs-idmapd
</pre>
* after editing /etc/exports, run
<pre>
exportfs -av
</pre>

=== setup userland ===

!!! ubuntu-18 version !!!

* zfs create rpool/nfsroot
* zfs set dedup=verify rpool/nfsroot ### enable deduplication to save disk space because most linux images have mostly identical files
* clone ubuntu
<pre>
mkdir /nfsroot/dsfe01
cd /
rsync -avx . /nfsroot/dsfe01
</pre>
* edit config files:
* cd /nfsroot/dsfe01
* emacs -nw etc/hostname ### change to dsfe01
* emacs -nw etc/mailname ### change to dsfe01
* emacs -nw etc/yp.conf ### change daq00.triumf.ca to musr00.triumf.ca
* emacs -nw etc/defaultdomain ### change to MUSR-NIS
* cp -pvf ../lxcpet-SL610/etc/ssh/*key* etc/ssh/ ### preserve the ssh keys
* emacs -nw opt/gonodeinfo/gonodeinfo.conf ### update information
* emacs -nw root/.ssh/authorized_keys ### update root ssh keys
* emacs -nw etc/fstab ### add this
<pre>
192.168.1.1:/nfsroot/dsfe01 / nfs defaults,nolock 0 0
</pre>
* emacs -nw etc/chrony/chrony.conf
** comment-out all "pool" and "server" entries
** add entry "server 192.168.1.1 iburst"

After dsfe01 is booted:

* disable services:
<pre>
systemctl disable apache2
systemctl disable dnsmasq
systemctl disable zfs-import-cache
</pre>

To setup additional machines, clone dsfe01 instead of cloning the gateway machine

=== Allow manpages to be viewed ===

If <code>/</code> is mounted over NFS, <code>man</code> will report a permission error. Fix it with:

<pre>
ln -s /etc/apparmor.d/usr.bin.man /etc/apparmor.d/disable/
apparmor_parser -R /etc/apparmor.d/usr.bin.man
</pre>

== setup shared home directory ==

=== on the gateway machine ===
* define netgroups
* emacs -nw /etc/netgroup
<pre>
dsfe (dsfe01,,) (dsfe02,,)
</pre>
* emacs -nw /etc/nsswitch.conf ### edit the netgroup line to read:
<pre>
netgroup: files
</pre>
* export the home directories:
* emacs -nw /etc/exports ### add this:
<pre>
/zssd/home1 @dsfe(rw,no_root_squash,async,no_subtree_check)
</pre>
* exportfs -rc

=== on the frontend machine ===

* mkdir /home
* emacs -nw /etc/fstab ### add this:
<pre>
192.168.1.1:/zssd/home1 /home nfs defaults 0 0
</pre>
* mount -a

== setup NAT ==

NAT allows machines on the private network to connect to the internet: https://en.wikipedia.org/wiki/Network_address_translation

In these examples:
* replace "eno1" with name of the outgoing interface (the one connected to the TRIUMF network).
* replace "enp11s0" with name of the private network interface (192.168.1.x network)

* emacs -nw /etc/rc.local ### add this:
<pre>
# /etc/rc.local

# enable NAT

/sbin/iptables -t nat -A POSTROUTING -o eno1 -j MASQUERADE
iptables -L -v

# uncomment following lines if machine has prohibitive FORWARD rules:
#/sbin/iptables -I FORWARD -i eno1 -o enp11s0 -m state --state RELATED,ESTABLISHED -j ACCEPT
#/sbin/iptables -I FORWARD -i enp11s0 -o eno1 -j ACCEPT
#iptables -L -v

iptables -L -v
sysctl -w net.ipv4.ip_forward=1
#sysctl -a | grep forward

sh /etc/firewall-rfc1918.sh

# end
</pre>
* emacs -nw /etc/firewall-rfc1918.sh
<pre>
# firewall-rfc1918.sh

# prevent RFC1918 private network IP addresses from
# going in and out from our uplink.

ETH=eno1

iptables -F in-rfc1918
iptables -N in-rfc1918
iptables -A in-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A in-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A in-rfc1918 --dst 192.168.0.0/16 -j REJECT

iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -D INPUT -j in-rfc1918 -i $ETH
iptables -I INPUT -j in-rfc1918 -i $ETH

iptables -F out-rfc1918
iptables -N out-rfc1918
iptables -A out-rfc1918 --dst 10.0.0.0/8 -j REJECT
iptables -A out-rfc1918 --dst 172.16.0.0/12 -j REJECT
iptables -A out-rfc1918 --dst 192.168.0.0/16 -j REJECT

iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -D OUTPUT -j out-rfc1918 -o $ETH
iptables -I OUTPUT -j out-rfc1918 -o $ETH

iptables -D FORWARD -j out-rfc1918 -o $ETH
iptables -D FORWARD -j out-rfc1918 -o $ETH
iptables -I FORWARD -j out-rfc1918 -o $ETH

# allow TRIUMF-SECURE network

iptables -I in-rfc1918 -s 10.90.0.0/255.255.0.0 -j ACCEPT
iptables -I out-rfc1918 -d 10.90.0.0/255.255.0.0 -j ACCEPT

# show configuration

iptables -L -v

#end
</pre>

= KVM =

<pre>
apt install cpu-checker

root@daq13:~# kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
root@daq13:~#

(if not, shutdown, go into BIOS settings, enable CPU virtualization)

apt install virtinst ### will install many packages
apt install libvirt-clients libvirt-daemon-system-systemd libvirt-daemon qemu qemu-kvm libvirt-daemon-system virtinst bridge-utils

root@daq13:/home1/wheel# virsh list --all
Id Name State
------------------------------
1 ubuntu-guest running

apt install virt-manager

virt-install --name ubuntu-guest --os-variant ubuntu20.04 --vcpus 2 --ram 2048 --location /daq/daqstore/olchansk/linux/Ubuntu/ubuntu-20.04.3-desktop-amd64.iso --network bridge=virbr0,model=virtio --graphics none --extra-args='console=ttyS0,115200n8 serial'

virtual machine will start, boot, etc
to get out of it, CTRL + Shift followed by ]

ssh wheel@daq13
virt-manager

run virt-install again, omit "--graphics none", open graphics console from virt-manager, it booted into ubuntu installer desktop

virt-install --name test10 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --filesystem /kvm_ladd00,/ --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial" --graphics none

virt-install --name test14 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --disk /tmp/xxx/ladd00.img,bus=sata --network bridge=virbr0,model=virtio --boot kernel=/kvm_ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm_ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64.img,kernel_args="root=/dev/sda console=ttyS0,115200n8 serial rdshell" --graphics none --check path_in_use=off
</pre>

build image

<pre>
dd if=/dev/zero of=/tmp/xxx/ladd00.img bs=1024M count=20
mkfs.ext3 /tmp/xxx/ladd00.img ### ext4 fails to mount by SL6 kernel, "unknown ext4 options"
cd /kvm_ladd00/
mount -o loop /tmp/xxx/ladd00.img /mnt/tmp
rsync -av . /mnt/tmp/ --delete
umount /mnt/tmp
</pre>

on the guest, configure network: /etc/rc.local
<pre>
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

ifconfig eth2 192.168.122.2
route add -net 0.0.0.0 gw 192.168.122.1
ifconfig -a
netstat -rn

# end
</pre>

== virsh commands ==

<pre>
virsh list --all

virsh start kvm-el7
virsh console kvm-el7
virsh destroy kvm-el7

virsh install ...
virsh undefine kvm-el7

virsh autostart kvm-ladd00
virsh dominfo kvm-ladd00
</pre>

== virtualize SL6 ladd00 ==

* on ladd00:
<pre>
yum install dracut-network
mkinitrd /boot/initramfs-2.6.32-754.35.1.el6.x86_64-netboot.img 2.6.32-754.35.1.el6.x86_64
</pre>
* on daq00
<pre>
zfs create rpool/kvm-ladd00
cd /kvm-ladd00
rsync -avx ladd00:/ . --exclude nfsroot
brctl addbr virbr0
ifconfig virbr0 192.168.1.1
echo /kvm-ladd00 192.168.1.2(rw,no_root_squash,no_all_squash,async,no_subtree_check) >> /etc/exports
exportfs -rv
</pre>
* create virtual machine
<pre>
virt-install --name kvm-ladd00 --os-variant centos6.10 --vcpus 2 --ram 2048 --import --network bridge=virbr0,model=virtio --boot kernel=/kvm-ladd00/boot/vmlinuz-2.6.32-754.35.1.el6.x86_64,initrd=/kvm-ladd00/boot/initramfs-2.6.32-754.35.1.el6.x86_64-netboot.img,kernel_args="root=/dev/nfs ip=192.168.1.2:192.168.1.1:192.168.1.1:255.255.255.0:ladd00::off nfsroot=192.168.1.1:/kvm-ladd00,vers=3,tcp console=ttyS0,115200n8 serial rdshell" --graphics none --nodisks --check path_in_use=off
</pre>
* adjust kvm-ladd00 image
<pre>
disable network manager
edit fstab
edit yp.conf
edit resolv.conf
edit root/.ssh/authorized_keys
enable rngd or /dev/random does not work, sshd does not work
</pre>
* virsh shutdown test24
* virsh --connect qemu:///system start test24
* virsh console test24 ### to exit, ctrl+[ or ctrl+]
* virsh undefine test24
* virsh autostart kvm-ladd00
* virsh dominfo kvm-ladd00
<pre>
root@daq00:~# virsh dominfo kvm-ladd00
Id: 1
Name: kvm-ladd00
UUID: 1d1f8fed-8b65-4411-a51b-e0ecf359d2f1
OS Type: hvm
State: running
CPU(s): 2
CPU time: 27.7s
Max memory: 2097152 KiB
Used memory: 2097152 KiB
Persistent: yes
Autostart: enable
Managed save: no
Security model: apparmor
Security DOI: 0
Security label: libvirt-1d1f8fed-8b65-4411-a51b-e0ecf359d2f1 (enforcing)
root@daq00:~#
</pre>
* delete unused images in /var/lib/libvirt/images
* virsh edit kvm-ladd00 # change boot command line, etc

== virtualize CentOS-7 daqstore ==

* similar to ladd00 above:
* on daqstore, install dracut-network, already there
<pre>
yum install dracut-network
yum install qemu-kvm libvirt libvirt-python libguestfs-tools virt-install
# yum install busybox ### no rpm package?!?
</pre>
<pre>
dracut -a nfs -v /boot/initramfs-3.10.0-1160.119.1.el7.x86_64-virt.img 3.10.0-1160.119.1.el7.x86_64 --force
</pre>
<pre>
scp daqstore:/boot/initramfs-3.10.0-1160.119.1.el7.x86_64-virt.img /kvm-el7/boot/
</pre>
* on daq00
<pre>
zfs create rpool/kvm-el7
cd /kvm-el7
rsync -avx daqstore:/ .
echo 192.168.1.3 kvm-el7 >> /etc/hosts
systemctl restart dnsmasq
echo /kvm-el 192.168.1.3(rw,no_root_squash,no_all_squash,async,no_subtree_check) >> /etc/exports
exportfs -rv
</pre>
* manage virtual machine
<pre>
virsh console kvm-el7
virsh destroy kvm-el7
virsh undefine kvm-el7
</pre>
* create virtual machine
<pre>
virt-install --name kvm-el7 --os-variant centos7 --vcpus 2 --ram 2048 --import --network bridge=virbr0,model=e1000e --boot kernel=/kvm-el7/boot/vmlinuz-3.10.0-1160.119.1.el7.x86_64,initrd=/kvm-el7/boot/initramfs-3.10.0-1160.119.1.el7.x86_64-virt.img,kernel_args="root=/dev/nfs ip=192.168.1.3:192.168.1.1:192.168.1.1:255.255.255.0:kvm-el7::off nfsroot=192.168.1.1:/kvm-el7,vers=3,tcp rw console=ttyS0,115200n8 serial rdshell" --graphics none --nodisks --check path_in_use=off
</pre>
* adjust kvm-el7 image
<pre>
disable network manager
edit fstab
edit hostname
disable selinux in /etc/sysconfig/selinux

UP TO HERE --- DNS does not work!!!

edit yp.conf
edit resolv.conf
edit root/.ssh/authorized_keys
enable rngd or /dev/random does not work, sshd does not work
</pre>
* virsh shutdown test24
* virsh --connect qemu:///system start test24
* virsh console test24 ### to exit, ctrl+[ or ctrl+]
* virsh undefine test24
* virsh autostart kvm-ladd00
* virsh dominfo kvm-ladd00
<pre>
root@daq00:~# virsh dominfo kvm-ladd00
Id: 1
Name: kvm-ladd00
UUID: 1d1f8fed-8b65-4411-a51b-e0ecf359d2f1
OS Type: hvm
State: running
CPU(s): 2
CPU time: 27.7s
Max memory: 2097152 KiB
Used memory: 2097152 KiB
Persistent: yes
Autostart: enable
Managed save: no
Security model: apparmor
Security DOI: 0
Security label: libvirt-1d1f8fed-8b65-4411-a51b-e0ecf359d2f1 (enforcing)
root@daq00:~#
</pre>
* delete unused images in /var/lib/libvirt/images
* virsh edit kvm-ladd00 # change boot command line, etc

= ARM64 cross-compiler =

* arm64, aarch64 are Xilinx FPGA Cortex-A53, RPi4, RPi5 machines
* install packages:
<pre>
apt install g++-12-aarch64-linux-gnu gcc-12-aarch64-linux-gnu-base libstdc++-12-dev-arm64-cross
</pre>
* run:
<pre>
aarch64-linux-gnu-gcc-12 -o ttcp.aarch64 ttcp.c -static
aarch64-linux-gnu-g++-12 -o fecdm.exe -O2 -g -Wall -Wuninitialized -std=c++20 fecdm.o dsdm.o /home/dsdaqdev/packages_common/midas/linux-aarch64-remoteonly/lib/libmidas.a -pthread -lrt -lutil /nfsroot/gdm00/usr/lib/aarch64-linux-gnu/libi2c.a -static
</pre>

= ARM cross-compiler =

NOTE: updated for U-24

* armv7 (Cyclone-V SoC, RPi3, MityARM CAMAC) machines (Debian-12 armhf target, Ubuntu 24.04 host)
* install packages:
<pre>
apt install g++-arm-linux-gnueabihf gcc-arm-linux-gnueabihf libc6-dev-armhf-cross
</pre>
* build MIDAS frontend (static linking)
<pre>
arm-linux-gnueabihf-g++ -std=c++11 -Wall -Wuninitialized -g -O2 -I/home/dldaq/packages/midas/include -I/home/dldaq/packages/midas/mvodb -c koi2c.cxx
arm-linux-gnueabihf-g++ -o fedldb.exe -std=c++11 -Wall -Wuninitialized -g -O2 -I/home/dldaq/packages/midas/include -I/home/dldaq/packages/midas/mvodb fedldb.o koi2c.o /home/dldaq/packages/midas/linux-armv7-remoteonly/lib/libmidas.a -L/usr/arm-linux-gnueabihf/lib -L/nfsroot/dltdc/usr/lib/arm-linux-gnueabihf -static -lm -lz -lutil -lnsl -lpthread -lrt -li2c
/usr/lib/gcc-cross/arm-linux-gnueabihf/13/../../../../arm-linux-gnueabihf/bin/ld: /home/dldaq/packages/midas/linux-armv7-remoteonly/lib/libmidas.a(system.o): in function `ss_socket_connect_tcp(char const*, int, int*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)':
/home/dldaq/packages/midas/src/system.cxx:4984:(.text+0x252a): warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
</pre>

= 32-bit intel cross-compiler =

Ubuntu 22.04:

<pre>
apt install libstdc++-11-dev:i386
apt install zlib1g-dev:i386
</pre>

NOTES:
* "g++ -m32" does not find libstdc++, please use "g++ -m32 -L/usr/lib/gcc/i686-linux-gnu/11/"
* to cross-build 32-bit MIDAS, use "make linux32".
* executables cross-build on Ubuntu-22 do NOT run on 32-bit Debain-11 (GLIBC and GLIBCXX version mismatch)
* executables cross-build on Ubuntu-22 run on 32-bit Debian-12.

= SSH settings for EPICS =

* TRIUMF EPICS runs obsolete version of SSH
* add this to the use .ssh/config
<pre>
Host sbp1*
HostKeyAlgorithms +ssh-rsa
PubKeyAcceptedAlgorithms +ssh-rsa
KexAlgorithms +diffie-hellman-group1-sha1
ForwardX11 yes
ForwardX11Trusted yes
</pre>

= changes for VME processors =

<pre>
apt -y remove sysstat man-db
apt -y purge dkms
apt -y purge mdadm
apt -y purge fwupd
apt -y purge packagekit
apt -y purge accountsservice
apt -y purge plocate
apt -y purge upower power-profiles-daemon
apt -y autoremove
</pre>

for D-12 32-bit CPUs:

<pre>
apt remove "*libavahi*"
</pre>

= remove snap (U-24) =

Note: snap stores data in $USER/snap/$SNAPNAME, removing a snap on one machine will remove this data from all users even if they want to use snap on some other machine.

Prepare:

NOTE: first remove chromium and firefox, see below.

NOTE: if possible, stop autofs before removing snap - otherwise it will mount all user home directories and complain that it cannot remove some snap data from them

<pre>
systemctl stop autofs
ls -ld /home1/*/snap/* ### remove the per-user snap directories
</pre>

Remove snaps:

<pre>
snap list
echo snap remove --purge chromium ### see below
echo snap remove --purge firefox ### see below
snap remove thunderbird
snap remove cups
snap remove hello-world
snap remove firmware-updater
snap remove gtk-common-themes
snap remove snapd-desktop-integration
snap remove snap-store
snap remove hunspell-dictionaries-1-7-2004
snap remove gnome-system-monitor
snap remove gnome-3-26-1604
snap remove gnome-3-28-1804
snap remove gnome-3-34-1804
snap remove gnome-3-38-2004
snap remove gnome-42-2204
snap remove gnome-46-2404
snap remove mesa-2404
snap remove core
snap remove core18
snap remove core20
snap remove core22
snap remove core24
snap remove bare
snap remove snapd
snap list
</pre>
<pre>
root@daqubuntu:~# snap list
No snaps are installed yet. Try 'snap install hello-world'.
root@daqubuntu:~#
</pre>

Identify packages that install snaps:

<pre>
apt list | grep snap | grep installed | grep -v -e snappy -e snapshot
</pre>

Typical output:

<pre>
firefox/noble,now 1:1snap1-0ubuntu5 amd64 [installed]
gir1.2-snapd-2/noble,now 1.64-0ubuntu5 amd64 [installed,automatic]
libsnapd-glib-2-1/noble,now 1.64-0ubuntu5 amd64 [installed,automatic]
libsnapd-qt-2-1/noble,now 1.64-0ubuntu5 amd64 [installed,automatic]
plasma-discover-backend-snap/noble,now 5.27.11-0ubuntu2 amd64 [installed]
snapd/noble-updates,now 2.66.1+24.04 amd64 [installed]
</pre>

Remove packages that install snaps:

<pre>
apt remove chromium-browser
apt remove chromium-codecs-ffmpeg-extra
apt remove thunderbird
apt remove firefox
apt remove plasma-discover-backend-snap
apt remove plasma-discover-snap-backend
apt remove snapd
apt purge snapd
# package gir1.2-snapd-2 is required by ubuntu-mate-desktop & co
# libsnapd-glib-2-1 is required by gstreamer, gnome-remote-desktop & co
ls -l /etc/systemd/system/ | grep snap ### remove unwanted stuff
</pre>

Remove Chromium:

* ls -ld /home/*/snap/chromium/*
* echo /bin/rm -rf `ls -1d /home/*/snap/chromium/*`
* snap remove chromium

Remove Firefox:

* ls -ld /home/*/snap/firefox/*
* echo /bin/rm -rf `ls -1d /home/*/snap/firefox/*` ### this will delete "snap firefox" profiles of all users!!!
* snap remove firefox ### this will also delete "snap firefox" profiles of all users!!!

Remove gir1.2-snapd-2:
* echo rm -vf /usr/lib/x86_64-linux-gnu/girepository-1.0/Snapd-2.typelib

If "snap remove" is stuck in "change in progress" (this will remove all snaps and break snapd, which is ok,
see https://forum.snapcraft.io/t/snap-remove-taking-forever-abort-wasnt-working/48915)
<pre>
rm /var/lib/snapd/state.json
systemctl restart snapd
</pre>

Prevent snap from reinstalling:
<pre>
cd ~/git/scripts/etc
git pull
cp etc-apt-preferencesd-disable-snap /etc/apt/preferences.d/
</pre>

= install non-snap thunderbird =

from: https://ubuntuhandbook.org/index.php/2024/03/install-thunderbird-deb-ubuntu-2404/

* remove snap thunderbird
<pre>
snap remove --purge thunderbird
apt remove --purge thunderbird
</pre>
* add mozilla repository
<pre>
already done after installing firefox-esr
</pre>
* ppa deb and ubuntu snap thunderbird package names are the same, change priority and hide snap package: create /etc/apt/preferences.d/mozillateamppa
<pre>
Package: thunderbird*
Pin: release o=LP-PPA-mozillateam
Pin-Priority: 1001

Package: thunderbird*
Pin: release o=Ubuntu
Pin-Priority: -1
</pre>
* check that it worked, it should say "build" instead of "snap"
<pre>
apt update
apt list | grep thunderbird | grep -v locale
...
thunderbird/noble 1:128.8.1+build1-0ubuntu0.24.04.1~mt1 amd64
...
</pre>
* install
<pre>
apt install thunderbird
</pre>
* run: thunderbird

= EFI boot using syslinux =

* rationale 1: GRUB is the stock boot loader with U-24. It is unnecessarily complicated. EFI BIOS can boot the linux kernel directly, without GRUB, but unfortunately a small shim bootloader is required to specify the initrd file and the root filesystem. the syslinux boot loader can do this in a very simple way.
* rationale 2: GRUB bootloader configuration is overcomplicated, and when it breaks, it is almost impossible to debug and to recover.
* rationale 3: GRUB bootloader scripts in U-24 have no support for booting from redundant SSDs.
* in the case of GRUB bootloader failure, it is simplest to boot the Ubuntu installer in recovery mode and convert the bootloader from GRUB to syslinux. Open firefox on this page and cut-and-paste the steps (only copy of vmlinux and initrd cannot copy-and-paste as of this writing).
* in the case of servers with redundant SSDs for OS and home directories (ZFS mirror), it is simplest to use the syslinux bootloader to ensure that the machine boots from either SSD (all combinations SSD failures, fail of either EFI partiion, fail of either ZFS mirror partition, machine should boot)
* check partition tables, SATA SSD
<pre>
fdisk -l /dev/sda
Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: WD Blue SA510 2.
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: A3F34DAC-DCB4-B74C-B59E-41E754807812

Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 5244927 4194304 2G Linux swap
/dev/sda3 5244928 9439231 4194304 2G Solaris boot
/dev/sda4 9439232 488397134 478957903 228.4G Solaris root
</pre>
* check partition tables, NVME SSD
<pre>
fdisk -l /dev/nvme0n1
Disk /dev/nvme0n1: 1.75 TiB, 1920383410176 bytes, 3750748848 sectors
Disk model: SAMSUNG MZ1L21T9HCLS-00A07
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 131072 bytes / 131072 bytes
Disklabel type: gpt
Disk identifier: 04ECCD46-DC2A-454C-B4A8-CCC18AA532F7

Device Start End Sectors Size Type
/dev/nvme0n1p1 2048 2203647 2201600 1G EFI System
/dev/nvme0n1p2 2203648 6397951 4194304 2G Linux filesystem
/dev/nvme0n1p3 6397952 23175167 16777216 8G Linux swap
/dev/nvme0n1p4 23175168 3750746111 3727570944 1.7T Linux filesystem
</pre>
* prepare boot device EFI partition, SATA SSDs
<pre>
mkfs.msdos /dev/sda1
mkfs.msdos /dev/sdb1
mkdir /boot/efi-sda
mkdir /boot/efi-sdb
mount /dev/sda1 /boot/efi-sda
mount /dev/sdb1 /boot/efi-sdb
</pre>
* prepare boot device EFI partition, NVME SSDs
<pre>
mkfs.msdos /dev/nvme0n1p1
mkfs.msdos /dev/nvme1n1p1
mkdir /boot/efi-0
mkdir /boot/efi-1
mount /dev/nvme0n1p1 /boot/efi-0
mount /dev/nvme1n1p1 /boot/efi-1
</pre>
* add them to fstab, note the "nofail" mount option
<pre>
blkid | grep vfat
/dev/sdb1: UUID="F30C-13B5" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="20e423b5-ac29-ec42-bab5-f366aefbbd2b"
/dev/sda1: UUID="F2DD-7321" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="9427646c-ce5f-fe47-9ed1-4b84cf4c348f"
grep ^UUID /etc/fstab
UUID=F2DD-7321 /boot/efi-sda vfat umask=0022,fmask=0022,dmask=0022,nofail 0 1
UUID=F30C-13B5 /boot/efi-sdb vfat umask=0022,fmask=0022,dmask=0022.nofail 0 1
</pre>
* prepare the EFI partitions (remove old previous subdirectories, only empty efi/boot should be there)
<pre>
cd /boot/efi-sda
mkdir -p efi/boot
</pre>
* get syslinux-6.03
<pre>
cd ~
wget https://daq00.triumf.ca/~olchansk/linux/syslinux-6.03.tar.xz
xz -d < syslinux-6.03.tar.xz | tar xvf -
</pre>
* from syslinux-6.03 copy files:
<pre>
cd /boot/efi-sda/efi/boot
cp ~/syslinux-6.03/efi64/efi/syslinux.efi .
cp ~/syslinux-6.03/efi64/com32/elflink/ldlinux/ldlinux.e64 .
cp syslinux.efi bootx64.efi
</pre>
* identify the ZFS rpool label
<pre>
zfs list | grep ROOT | grep "/$" | cut -f1 -d" "
rpool/ROOT/ubuntu_9yvb17
</pre>
* create syslinux.cfg, change the root=ZFS label to match this computer
<pre>
cat << EOF | sed "s;root=.*$;root=ZFS=`zfs list | grep ROOT | grep "/$" | cut -f1 -d" "`;" > syslinux.cfg
default linux
label linux
kernel vmlinuz
append ro initrd=initrd.img root=ZFS=rpool/ROOT/ubuntu_02ruwj
EOF
</pre>
* copy linux boot files:
<pre>
cp /boot/vmlinuz vmlinuz
cp /boot/initrd.img initrd.img
</pre>
* repeat with /boot/efi-sdb, etc
* install script to set syslinux to boot the latest kernel
<pre>
cd ~/git/scripts
git pull
ln -s ~/git/scripts/etc/update_efi_syslinux.perl ~
</pre>
* update syslinux to boot the latest kernel
** run "~/update_efi_syslinux.perl" to check that it finds the EFI partitions and finds the correct kernel
** run "~/update_efi_syslinux.perl -u" to do the actual update
* or maybe install Ubuntu syslinux 6.04 and use files from there:
<pre>
apt install "syslinux*"
</pre>

= legacy boot using syslinux =

* NOTE: extlinux is not compatible with ext4 "64bit" feature, it should be turned off:
<pre>
mkfs.ext4 -O ^64bit /dev/sdX1
resize2fs -s /dev/sdX1
</pre>

* install syslinux and extlinux (THIS DOES NOT WORK!!!)

<pre>
apt -y install syslinux extlinux
dd if=/usr/lib/syslinux/mbr/mbr.bin of=/dev/sdX ### NOT /dev/sdX1 NOT !!!
cd /boot
cp /usr/lib/syslinux/modules/bios/menu.c32 .
extlinux -i .
</pre>

* install syslinux and extlinux

* copy from old SL6 USB disk (this is extlinux 6.02)
<pre>
root@localhost:/boot# ls -l
-rwxr-xr-x 1 root root 218952 Jan 28 17:40 extlinux
-rw-r--r-- 1 root root 402 Jan 29 14:45 extlinux.conf
-rw-r--r-- 1 root root 496 Jan 29 14:39 extlinux.conf~
-r--r--r-- 1 root root 122044 Jan 29 14:39 ldlinux.c32
-r--r--r-- 1 root root 67072 Jan 29 14:39 ldlinux.sys
-rwxr-xr-x 1 root root 24156 Jan 28 17:40 libutil.c32
-rw-r--r-- 1 root root 304 Jan 28 17:40 mbr.bin
-rw-r--r-- 1 root root 26140 Jan 28 17:40 memdisk
-rw-r--r-- 1 root root 69043 Jan 28 17:40 memtest86+-4.20.iso.zip
-rw-r--r-- 1 root root 183012 Jan 28 17:40 memtest86+-5.01
-rw-r--r-- 1 root root 26568 Jan 28 17:40 menu.c32
</pre>

* install
<pre>
dd if=mbr.bin of=/dev/sdX ### NOT /dev/sdX1 NOT !!!
extlinux -i .
</pre>
* check that partition /dev/sdX1 is marked bootable (fdisk command "a")

* create /boot/extlinux.conf

<pre>
DEFAULT menu.c32
PROMPT 0
TIMEOUT 50

MENU TITLE TRIUMF DAQ USB BOOT32 ver K.O. 2025jan28

LABEL linux
MENU DEFAULT
kernel /vmlinuz
append initrd=/initrd.img panic=60 rootdelay=5 rootwait rw root=/dev/sda1

LABEL linux-6.1.0-28-686
kernel vmlinuz-6.1.0-28-686
append initrd=initrd.img-6.1.0-28-686 panic=60 rootdelay=5 rootwait rw root=/dev/sda1

LABEL memtest
kernel memtest86+-1.65
</pre>

= Add user to login spash screen =

For user login to local machine, if it doesn't work by default. User should exist already from NIS service.

<pre>
sudo /bin/bash # as root
cd /var/lib/AccountsService/users
ls # should at least display the "wheel" user
cp wheel username # we're going to copy the user account settings over to our new user
</pre>