VME-CPU: Difference between revisions

From DaqWiki
Jump to navigation Jump to search
 
(101 intermediate revisions by the same user not shown)
Line 50: Line 50:
/dev/sda1  *          1        1247    7821156  83  Linux
/dev/sda1  *          1        1247    7821156  83  Linux
</pre>
</pre>
* /usr/bin/time mke2fs -j /dev/sdX1 (should take about 30 seconds)
* /usr/bin/time mkfs.ext4 -O ^64bit /dev/sdX1 (should take about 30 seconds)
* tune2fs -i0 -c0 /dev/sdX1
* tune2fs -i0 -c0 /dev/sdX1
* mkdir -p /mnt/tmp
* mkdir -p /mnt/tmp
Line 84: Line 84:
vi etc/fstab ### change the "/" NFS mount-point if needed
vi etc/fstab ### change the "/" NFS mount-point if needed
vi etc/cron.d/triumf_nodeinfo ### change nodeinfo server name
vi etc/cron.d/triumf_nodeinfo ### change nodeinfo server name
</pre>
=== Cloning Ubuntu NFS-Root ===
We will clone "lxsrc" into "lxdst":
<pre>
cd lxsrc
mkdir ../lxdst
rsync -av . ../lxdst
vi etc/hostname ### change HOSTNAME
vi etc/fstab ### change the "/" NFS mount point
vi opt/gonodeinfo/gonodeinfo.conf ### change description
</pre>
</pre>


Line 140: Line 153:
** remove "MENU DEFAULT" from all entries except the new one
** remove "MENU DEFAULT" from all entries except the new one
* reboot into the new kernel
* reboot into the new kernel
=== Setup GRUB2 ===
<pre>
# not fdisk -cu -H 224 -S 56 /dev/sdc
# or fdisk -cu -H 64 -S 32 /dev/sdc
# or fdisk -H 224 -S 56 /dev/sdc # start the first partition from cylinder 10
# or fdisk -H 255 -S 63 /dev/sdc # start the first partition from cylinder 10
fdisk -H 224 -S 56 /dev/sdc # start the first partition from cylinder 10
mkfs.ext4 -O ^64bit /dev/sdc1
tune2fs -c0 -i0 /dev/sdc1
mkdir -p /mnt/tmp
mount /dev/sdc1 /mnt/tmp
cd /mnt/tmp
# NOT THIS: rsync -av /ladd/data0/root/lxdaq02-usb/ .
rsync -av /ladd/data0/backup.os/lxdaq02/ .
#cd /home/olchansk/git/grub
#./grub-install -v --directory ./grub-core /dev/sdc --boot-directory=/mnt/tmp/boot --root-directory=/mnt/tmp
cd /mnt/tmp
./opt/grub/sbin/grub-install -v --directory=/mnt/tmp/opt/grub/lib/grub/i386-pc --boot-directory=/mnt/tmp/boot --root-directory=/mnt/tmp /dev/sdc
./root/git/scripts/clone/uuidfix.perl -v /mnt/tmp
xemacs -nw etc/sysconfig/network # change hostname
xemacs -nw etc/sysconfig/network-scripts/ifcfg-eth0 # change IP address
xemacs -nw etc/nodeinfo # change description
cd /
umount /mnt/tmp
eject /dev/sdc
</pre>
Partition table:
<pre>
[root@lxdaq01 ~]# fdisk -l
Disk /dev/sda: 16.0 GB, 16013852672 bytes
255 heads, 63 sectors/track, 1946 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x62f63f6c
  Device Boot      Start        End      Blocks  Id  System
/dev/sda1  *          10        1946    15558952+  83  Linux
[root@lxdaq01 ~]#
</pre>
NOT THIS:
<pre>
[root@ladd21 grub]# fdisk -lcu /dev/sdc
Disk /dev/sdc: 16.0 GB, 16013852672 bytes
87 heads, 48 sectors/track, 7489 cylinders, total 31277056 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x855bfadf
  Device Boot      Start        End      Blocks  Id  System
/dev/sdc1            2048    31277055    15637504  83  Linux
[root@ladd21 grub]#
</pre>
grub.cgf:
<pre>
[root@ladd21 grub]# cat /media/edd35353-82d7-4e18-aeeb-2073ce19c543/boot/grub/grub.cfg
# Begin /boot/grub/grub.cfg
set default=0
set timeout=5
insmod ext3
set root=(hd0,1)
menuentry "memtest86+-4.10" {
        linux16 /boot/memtest86+-4.10
}
menuentry "SL6-V7805-KO2" {
        linux  /boot/vmlinuz-2.6.34.1-V7805-KO2 panic=60 rootwait ro root=/dev/sda1
}
menuentry "GNU/Linux, Linux 3.15.1-lfs-SVN-20140702" {
        linux  /boot/vmlinuz-3.15.1-lfs-SVN-20140702 root=/dev/sda2 ro
}
#
</pre>
== V7805 special settings ==
Because of hardware incompatibility, the V7805 cannot run vanilla RedHat/SL linux kernels.
Because of hardware incompatibility, the V7805 cannot boot from 16GB USB flash. Please use 8GB USB flash media.
To run RH/SL linux, a special linux kernel has to be built with special PCI configuration:
NOTE: kernel 2.6.34.1 works, newer kernels do not seem to work.
<pre>
ladd00:linux-2.6.34.1$ grep PCI .config
CONFIG_PCI_QUIRKS=y
# Bus options (PCI etc.)
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
CONFIG_PCI_GODIRECT=y
# CONFIG_PCI_GOOLPC is not set
# CONFIG_PCI_GOANY is not set
CONFIG_PCI_DIRECT=y
CONFIG_PCI_DOMAINS=y
# CONFIG_PCIEPORTBUS is not set
# CONFIG_PCI_MSI is not set
# CONFIG_PCI_DEBUG is not set
CONFIG_PCI_STUB=y
CONFIG_PCI_IOV=y
# CONFIG_HOTPLUG_PCI is not set
# CONFIG_PATA_CMD640_PCI is not set
CONFIG_NET_PCI=y
# CONFIG_KSZ884X_PCI is not set
# CONFIG_NE2K_PCI is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIAL_8250_PCI=y
# CONFIG_UIO_PCI_GENERIC is not set
ladd00:linux-2.6.34.1$
</pre>
== V7648 and V7750 BIOS Settings ==
Sometimes the V7648/V7750 boards lose BIOS settings and do not boot. Connect VGA monitor to see the errors.
To reset BIOS settings, short jumper E3 (next to the ethernet connectors, top of the board).
* legacy diskette A: -> set to "disabled"
* console redirection -> set to "disabled"
* local bus ide adapter -> set to "disabled"
* POST errors -> set to "disabled"
== V7805 BIOS Settings ==
Sometimes the V7805 boards lose BIOS settings and do not boot. Connect DVI (VGA with DVI->VGA adapter) monitor to see the errors.


== V7865 BIOS Settings ==
== V7865 BIOS Settings ==


* BIOS reset switch is "E11" at the top of the PCB next to the CompactFlash connector. It is the middle one of the 3 two-pin headers.
* enter BIOS by pressing "DEL" during power up
* enter BIOS by pressing "DEL" during power up
* Boot->Boot setting "Wait for F1" set to "Disabled"
* Boot->Boot setting "Wait for F1" set to "Disabled"
* Chipset->South Bridge "USB 2.0 Controller" set to "Enabled"
* Chipset->South Bridge "USB 2.0 Controller" set to "Enabled"
* Chipset->Onboard devices->FP ethernet set to "Enabled"
* Advanced->IDE configuration set to "Disabled" (unless using CompactFlash boot disk)
* Advanced->IDE configuration set to "Disabled" (unless using CompactFlash boot disk)
* Advanced->Remote access set to "Disabled"
* Advanced->Remote access set to "Disabled"
* Advanced->USB->"USB 2.0 Controller Mode" set to "HiSpeed"
* Advanced->USB->"USB 2.0 Controller Mode" set to "HiSpeed"
* Advanced->Onboard device->"Reserve VME space" set to "512 Mbytes".
* Exit-> "Save changes and Exit"
* Exit-> "Save changes and Exit"
== XVB-602 BIOS Settings ==
* BIOS reset switch is S12 on the back of the board
* enter BIOS by pressing "F2" or "DEL" during power up
* Advanced -> Onboard device -> "Reserve VME space" set to "512 Mbytes".
* -> "Onboard gigabit LAN 1" set to "enable"
* -> "Onboard gigabit LAN 2" set to "enable"
* "Save & Exit" -> "Save changes and exit"
* the processor will restart and try to boot from network using PXE.
== V7750 typical sensors output ==
<pre>
[root@lxdaq14 ~]# sensors
max1617-i2c-0-18
Adapter: SMBus I801 adapter at 1810
temp1:      +32.0 C  (low  = -55.0 C, high = +127.0 C) 
temp2:      +45.0 C  (low  = -55.0 C, high = +127.0 C) 
pc87366-isa-efe0
Adapter: ISA adapter
in0:        +1.83 V  (min =  +0.00 V, max =  +2.95 V)
in1:        +2.65 V  (min =  +0.00 V, max =  +2.95 V)
in2:        +0.00 V  (min =  +0.00 V, max =  +2.95 V)
in3:        +2.46 V  (min =  +0.00 V, max =  +2.95 V)
in4:        +1.74 V  (min =  +0.00 V, max =  +2.95 V)
in5:        +1.09 V  (min =  +0.00 V, max =  +2.95 V)
in6:        +2.51 V  (min =  +0.00 V, max =  +2.95 V)
3VSB:        +3.36 V  (min =  +2.99 V, max =  +3.59 V)
VDD:        +3.38 V  (min =  +2.99 V, max =  +3.59 V)
Vbat:        +2.95 V  (min =  +0.00 V, max =  +2.95 V)
AVDD:        +3.36 V  (min =  +2.99 V, max =  +3.59 V)
temp4:        +0.0 C  (low  =  +0.0 C, high =  +0.0 C) 
                      (crit =  +0.0 C)                 
temp5:        +0.0 C  (low  =  +0.0 C, high =  +0.0 C) 
                      (crit =  +0.0 C)                 
temp6:        +1.7 C  (low  =  +0.0 C, high =  +3.0 C) 
                      (crit =  +3.0 C)                 
cpu0_vid:  +1.325 V
[root@lxdaq14 ~]#
</pre>


== Network boot ==
== Network boot ==
Line 166: Line 366:
After booting is complete only the NFS server is required for running Linux. After Linux is fully booted, DHCP, TFTP and PXE are not used (until reboot).
After booting is complete only the NFS server is required for running Linux. After Linux is fully booted, DHCP, TFTP and PXE are not used (until reboot).


=== Setup host computer ===
=== Setup the boot host computer (before el7) ===
 
==== install packages ====
<pre>
yum install dhcp tftp-server
</pre>


AAA
==== configure dhcpd ====
 
* create emacs -nw /etc/logrotate.d/dhcp
<pre>
/var/log/dhcp.log {
    weekly
    notifempty
    missingok
    postrotate
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
/bin/kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}
</pre>
 
* put this in emacs -nw /etc/rsyslog.conf right after the line "#### RULES ####"
<pre>
#### RULES ####
 
# Annoying dhcp spam ... (need to add to rotation ...)
local3.* -/var/log/dhcp.log
local3.* ~
</pre>
 
* chmod a+rx /etc/dhcp
* emacs -nw /etc/dhcp/dhcpd.conf - add contents from the next section
* emacs -nw /etc/dhcp/dhcpd.conf - customize for this boot host:
** change "next-server" to the hostname of this host
** change "host" entries
* chkconfig dhcpd on
* service rsyslog restart # to enable dhcp.log
* service dhcpd restart # watch /var/log/messages for errors, watch /var/log/dhcp.log for dhcp activity
* try to boot the VME CPU, watch messages in dhcp.log should be similar to those shown in the next section, the VME CPU should get an IP address and stop on "TFTP load" stage
 
==== setup tftp ====
 
* emacs -nw /etc/xinetd.d/tftp line "server_args" to read "server_args            = -vvvv -s /var/lib/tftpboot"
* chkconfig tftp on
* chkconfig xinetd on
* service xinetd restart
* try to boot the VME CPU, watch messages in the syslog, should see message about "pxelinux.0 not found", the VME CPU should get an IP address and stop with an error "TFTP - File not found"
 
==== setup pxelinux ====
 
* get isolinux/pxelinux version 4.03 from http://www.syslinux.org/wiki/index.php/The_Syslinux_Project
<pre>
cd ~
wget https://www.kernel.org/pub/linux/utils/boot/syslinux/4.xx/syslinux-4.03.tar.bz2
tar xjvf syslinux-4.03.tar.bz2
cd syslinux-4.03
cp -pv ./core/pxelinux.0 ./com32/hdt/hdt.c32 ./memdisk/memdisk ./com32/menu/menu.c32 /var/lib/tftpboot/
</pre>
* mkdir /var/lib/tftpboot/pxelinux.cfg
* create /var/lib/tftpboot/pxelinux.cfg/default with contents from the next section
* get additional boot files
<pre>
cd /var/lib/tftpboot
wget http://ladd00.triumf.ca/tftpboot/memtest86+-4.20.iso.zip
wget http://ladd00.triumf.ca/tftpboot/memtest86+-5.01.iso.gz
wget http://ladd00.triumf.ca/tftpboot/modules.alias
wget http://ladd00.triumf.ca/tftpboot/modules.pcimap
wget http://ladd00.triumf.ca/tftpboot/pci.ids
</pre>
 
==== setup linux kernel files ====
 
* get vmlinuz and initramfs files from the VME processor boot image
* note that the initramfs file needs to be built with the dracut-network function, it should be about 2.5Mbytes in size.
* note the file names must match the file names in the pxelinux config file
 
<pre>
wget http://ladd00.triumf.ca/tftpboot/vmlinuz-2.6.32-431.11.2.el6.i686
wget http://ladd00.triumf.ca/tftpboot/initramfs-2.6.32-431.11.2.el6.i686.img
</pre>
 
* try to boot the VME CPU, watch messages in the syslog, the VME CPU should get an IP address, load the PXELINUX menu, load the linux kernel and stop with an error mounting NFS filesystems.
 
==== setup NFS export of VME processor boot image ====
 
* copy the VME processor boot image to /nfsroot/alphavme03 (or /home1/alphavme03)
* customise the boot image per instructions in previous section (edit sysconfig/network, etc)
* edit /etc/exports:
<pre>
/nfsroot/alphavme03 daqtmp4(rw,no_root_squash,async)
</pre>
* chkconfig nfs on; service nfs restart
* if needed adjust the boot image path name in dhcpd.conf to match the export name in /etc/exports, restart dhcpd.conf
 
* try to boot the VME CPU, watch messages in the syslog, the VME CPU should get an IP address, load the PXELINUX menu, load the linux kernel, linux kernel will start, mount the VME processor boot image ("NFS Root"), boot the userland. At the end you should be able to ping and ssh into the VME processor.
 
==== fixup file permissions ====
 
<pre>
cd /var/lib/tftpboot
chown -R root.root .
chmod a+r *
chmod a-w *
</pre>
 
=== Setup the boot host computer (el7) ===
 
==== install packages (el7) ====
<pre>
yum install dhcp tftp-server
</pre>
 
==== configure dhcpd (el7) ====
 
* create emacs -nw /etc/logrotate.d/dhcp
<pre>
/var/log/dhcp.log {
    weekly
    notifempty
    missingok
    postrotate
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
/bin/kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}
</pre>
 
* put this in emacs -nw /etc/rsyslog.conf right after the line "#### RULES ####"
<pre>
#### RULES ####
 
# Annoying dhcp spam ... (need to add to rotation ...)
local3.* -/var/log/dhcp.log
local3.* ~
</pre>
 
* systemctl restart rsyslog
* chmod a+rx /etc/dhcp
* emacs -nw /etc/dhcp/dhcpd.conf - add contents from the next section
* emacs -nw /etc/dhcp/dhcpd.conf - customize for this boot host:
** change "next-server" to the hostname of this host
** change "host" entries
* systemctl enable dhcpd
* systemctl start dhcpd
* firewall-cmd --add-service=dhcp --permanent
* firewall-cmd --reload
* systemctl restart dhcpd # watch /var/log/messages for errors, watch /var/log/dhcp.log for dhcp activity
* chmod a+r /var/log/dhcp.log
* try to boot the VME CPU, watch messages in dhcp.log should be similar to those shown in the next section, the VME CPU should get an IP address and stop on "TFTP load" stage
 
==== setup tftp (el7) ====
 
<pre>
emacs -nw /usr/lib/systemd/system/tftp.service # line "ExecStart" to have ".../in.tftpd -vvvv -s /var/lib/tftpboot"
systemctl start tftp.socket
systemctl enable tftp.socket
firewall-cmd --add-service=tftp --permanent
firewall-cmd --reload
</pre>
 
* try to boot the VME CPU, watch messages in the syslog, should see message about "pxelinux.0 not found", the VME CPU should get an IP address and stop with an error "TFTP - File not found"
 
==== setup pxelinux (preliminary) ====
 
* get isolinux/pxelinux version 4.03 from http://www.syslinux.org/wiki/index.php/The_Syslinux_Project
<pre>
cd ~
wget https://www.kernel.org/pub/linux/utils/boot/syslinux/4.xx/syslinux-4.03.tar.bz2
tar xjvf syslinux-4.03.tar.bz2
cd syslinux-4.03
cp -pv ./core/pxelinux.0 ./com32/hdt/hdt.c32 ./memdisk/memdisk ./com32/menu/menu.c32 /var/lib/tftpboot/
</pre>
* mkdir /var/lib/tftpboot/pxelinux.cfg
* create /var/lib/tftpboot/pxelinux.cfg/default with contents from the next section
* get additional boot files
<pre>
cd /var/lib/tftpboot
wget http://ladd00.triumf.ca/tftpboot/memtest86+-4.20.iso.zip
wget http://ladd00.triumf.ca/tftpboot/memtest86+-5.01.iso.gz
wget http://ladd00.triumf.ca/tftpboot/modules.alias
wget http://ladd00.triumf.ca/tftpboot/modules.pcimap
wget http://ladd00.triumf.ca/tftpboot/pci.ids
</pre>
 
==== setup linux kernel files ====
 
* get vmlinuz and initramfs files from the VME processor boot image
* note that the initramfs file needs to be built with the dracut-network function, it should be about 2.5Mbytes in size.
* note the file names must match the file names in the pxelinux config file
 
<pre>
wget http://ladd00.triumf.ca/tftpboot/vmlinuz-2.6.32-431.11.2.el6.i686
wget http://ladd00.triumf.ca/tftpboot/initramfs-2.6.32-431.11.2.el6.i686.img
</pre>
 
* try to boot the VME CPU, watch messages in the syslog, the VME CPU should get an IP address, load the PXELINUX menu, load the linux kernel and stop with an error mounting NFS filesystems.
 
==== setup NFS export of VME processor boot image ====
 
* copy the VME processor boot image to /nfsroot/alphavme03 (or /home1/alphavme03)
* customise the boot image per instructions in previous section (edit sysconfig/network, etc)
* edit /etc/exports:
<pre>
/nfsroot/alphavme03 daqtmp4(rw,no_root_squash,async)
</pre>
* chkconfig nfs on; service nfs restart
* if needed adjust the boot image path name in dhcpd.conf to match the export name in /etc/exports, restart dhcpd.conf
 
* try to boot the VME CPU, watch messages in the syslog, the VME CPU should get an IP address, load the PXELINUX menu, load the linux kernel, linux kernel will start, mount the VME processor boot image ("NFS Root"), boot the userland. At the end you should be able to ping and ssh into the VME processor.
 
==== fixup file permissions ====
 
<pre>
cd /var/lib/tftpboot
chown -R root.root .
chmod a+r *
chmod a-w *
</pre>


=== DHCP configuration ===
=== DHCP configuration ===


<pre>
<pre>
#
# /etc/dhcpd.conf
#
#
# general setup
#
log-facility local3;
# make network booting the SystemImager autoinstallclient possible
allow booting;
allow bootp;
ignore unknown-clients;
ddns-update-style ad-hoc;
# set lease time to 3 days
default-lease-time 259200;
max-lease-time 259200;
# define network addresses
subnet 142.90.96.0 netmask 255.255.224.0 {
  not authoritative;
  ignore unknown-clients;
  option domain-name "triumf.ca";
  option domain-name-servers 142.90.100.19;
  option routers 142.90.100.18;
}
# special PXELINUX options
option space pxelinux;
option pxelinux.magic      code 208 = string;
option pxelinux.configfile code 209 = text;
option pxelinux.pathprefix code 210 = text;
option pxelinux.reboottime code 211 = unsigned integer 32;
#
#
# setup for TIGRESS VME processors (boot from midtig06)
# setup for TIGRESS VME processors (boot from midtig06)
Line 179: Line 632:
group {  
group {  
         filename "pxelinux.0";  
         filename "pxelinux.0";  
         next-server ladd00;  
         #next-server ladd00;  
         option routers 142.90.100.18;  
         option routers 142.90.100.18;  
         option subnet-mask 255.255.224.0;  
         option subnet-mask 255.255.224.0;  
Line 197: Line 650:
         option pxelinux.configfile "pxelinux.cfg/V7750-SL6a";
         option pxelinux.configfile "pxelinux.cfg/V7750-SL6a";
   
   
         host lxdaq17-eth0 { option pxelinux.configfile "pxelinux.cfg/V7750-SL6d"; option host-name "lxdaq17.triumf.ca"; option root-path "nfs:ladd00:/data0/root/lxdaq17:rw"; fixed-address lxdaq17; hardware ethernet 00:20:38:00:DA:1D; }  
         #host lxdaq17-eth0 { option pxelinux.configfile "pxelinux.cfg/V7750-SL6a"; option host-name "lxdaq17.triumf.ca"; option root-path "nfs:ladd00:/data0/root/lxdaq17:rw"; fixed-address lxdaq17; hardware ethernet 00:20:38:00:DA:1D; }  
         host lxdaq17-eth1 { option pxelinux.configfile "pxelinux.cfg/V7750-SL6d"; option host-name "lxdaq17.triumf.ca"; option root-path "nfs:ladd00:/data0/root/lxdaq17:rw"; fixed-address lxdaq17; hardware ethernet 00:20:38:00:DA:1F; }  
         #host lxdaq17-eth1 { option pxelinux.configfile "pxelinux.cfg/V7750-SL6a"; option host-name "lxdaq17.triumf.ca"; option root-path "nfs:ladd00:/data0/root/lxdaq17:rw"; fixed-address lxdaq17; hardware ethernet 00:20:38:00:DA:1F; }  


}
}
Line 216: Line 669:
label hdt
label hdt
   kernel hdt.c32
   kernel hdt.c32
label memtest86+-5.01
  kernel memdisk iso initrd=memtest86+-5.01.iso.gz


label memtest86+-4.20
label memtest86+-4.20
Line 234: Line 690:


#end
#end
</pre>
=== Boot host dhcp messages ===
<pre>
grep -i b4 /var/log/dhcp.log
[root@daqtmp5 Desktop]# grep -i 0e:b4 /var/log/dhcp.log
Jun 10 12:50:26 daqtmp5 dhcpd: DHCPDISCOVER from 00:20:38:03:0e:b4 via eth0
Jun 10 12:50:26 daqtmp5 dhcpd: DHCPOFFER on 142.90.96.104 to 00:20:38:03:0e:b4 via eth0
Jun 10 12:50:28 daqtmp5 dhcpd: DHCPREQUEST for 142.90.96.104 (142.90.96.105) from 00:20:38:03:0e:b4 via eth0
Jun 10 12:50:28 daqtmp5 dhcpd: DHCPACK on 142.90.96.104 to 00:20:38:03:0e:b4 via eth0
</pre>
</pre>


Line 249: Line 715:
May  5 15:54:14 ladd00 in.tftpd[3431]: RRQ from 142.90.111.107 filename ./initramfs-2.6.32-431.11.2.el6.i686.img
May  5 15:54:14 ladd00 in.tftpd[3431]: RRQ from 142.90.111.107 filename ./initramfs-2.6.32-431.11.2.el6.i686.img
May  5 15:54:29 ladd00 rpc.mountd[1796]: authenticated mount request from lxdaq17.triumf.ca:677 for /data0/root/lxdaq17 (/data0)
May  5 15:54:29 ladd00 rpc.mountd[1796]: authenticated mount request from lxdaq17.triumf.ca:677 for /data0/root/lxdaq17 (/data0)
</pre>
== V7648, V7750, V7805, V7851 : Setup vme_universe VME drivers ==
Install Linux kernel driver:
<pre>
ssh root@lxdragon01
yum install -y kernel-devel ### el6, el7
apt install -y linux-headers-686 ### Ubuntu, Debian
#apt install -y linux-headers-6.1.0-22-686 ### non-pae kernel headers
git clone https://daq00.triumf.ca/~olchansk/git/kernel-module-universe.git
cd kernel-module-universe/vme_universe
make
/bin/rm /usr/lib/libvme.so*
/bin/rm /root/vme_universe.ko
make install
cat <<EOF >> /etc/rc.local
mkdir -p /dev/bus/vme
mknod /dev/bus/vme/ctl c 221 8
chmod a+wr /dev/bus/vme/ctl
modprobe vme_universe
#insmod /root/vme_universe.ko
EOF
rmmod vme_universe
/etc/rc.local
dmesg # observe VME driver successful load messages (see below)
*** reboot *** shutdown -r now # reboot to confirm the vme driver loads correctly
</pre>
Install MIDAS driver and test software:
<pre>
cd ~
mkdir packages
cd packages
git clone http://ladd00.triumf.ca/~daqweb/git/midas.git
git clone http://ladd00.triumf.ca/~daqweb/git/mxml.git
svn checkout https://ladd00.triumf.ca/svn/daqsvn/trunk/vme # username svn, password svn
cd vme
make -k # some programs will fail to build, this is expected.
</pre>
Test VME access:
<pre>
[root@lxdragon01 vme]# ./vmescan.exe
vmic_mmap: Mapped VME AM 0x0d addr 0x00000000 size 0x00ffffff at address 0x20100000
mvme_open:
Bus handle              = 0x3
DMA handle              = 0x93f93f0
DMA area size          = 1048576 bytes
DMA    physical address = 0xb6691000
vmic_mmap: Mapped VME AM 0x2d addr 0x00000000 size 0x0000ffff at address 0x26ff0000
vmic_mmap: Mapped VME AM 0x3d addr 0x00000000 size 0x00ffffff at address 0x21100000
at 0x00100000: data16: 0x0215, data32: 0x01130215
at 0x00110000: data16: 0xffff, data32: 0x06000000
at 0x00200000: data16: 0x0215, data32: 0x01130215
at 0x008000fc: data16: 0x0851, data32: 0xffffffff (CAEN V513)
at 0x008100fc: data16: 0x0851, data32: 0xffffffff (CAEN V513)
at 0x00e00000: data16: 0xffff, data32: 0xc0000000
mvme_close:
Bus handle              = 0x3
DMA handle              = 0x93f93f0
DMA    physical address = 0xb6691000
[root@lxdragon01 vme]#
</pre>
Successful vme_universe kernel driver load messages:
<pre>
dmesg
...
VME: Board is system controller
VME: Driver compiled for SMP system
VME: Installed VME Universe module version: 3.6.KO5
</pre>
== V7865 and XVB-602 : Setup gefvme/tsi148 VME drivers ==
(ubuntu&etc with recent linux kernels):
Check that the "wrong" driver is disabled and is not loaded.
<pre>
lsmod | grep vme_tsi148
rmmod vme_tsi148
echo blacklist vme_tsi148 > /etc/modprobe.d/blacklist-vme_tsi148.conf
and reboot
</pre>
(ubuntu&etc):
Check that /etc/rc.local is enabled, see https://daq.triumf.ca/DaqWiki/index.php/Ubuntu#Enable_rc.local
and https://daq.triumf.ca/DaqWiki/index.php/SLinstall#Post_installation_CentOS7
Install Linux kernel driver:
<pre>
ssh root@lxdaq27
cd ~
mkdir git
cd git
git clone https://daq00.triumf.ca/~olchansk/git/kernel-module-gefvme.git
cd kernel-module-gefvme
make
make install
sh /etc/rc.local
dmesg # observe VME driver successful load messages (see below)
</pre>
Install MIDAS driver and test software:
<pre>
cd ~
mkdir git
cd git
git clone https://bitbucket.org/tmidas/midas --recursive
git clone https://bitbucket.org/ttriumfdaq/vme
cd vme
make -k MIDASSYS=../midas # some programs will fail to build, this is expected.
</pre>
Test VME access:
<pre>
[root@lxdaq27 vme]# ./vmescan_gef.exe
gefvme_openWindow: Slot 0, VME addr 0x00000000, am 0x00, size 0x00000004 4
gefvme_openWindow: Configured window 0 VME AM 0x2d A16 addr 0x00000000 size 0x00010000 fd 3
gefvme_openWindow: Slot 1, VME addr 0x00ff0000, am 0x00, size 0x00000004 4
gefvme_openWindow: Configured window 1 VME AM 0x3d A24 addr 0x00000000 size 0x01000000 fd 4
at 0x00100000: data16: 0x0925, data32: 0x01120925
at 0x00af0000: data16: 0x0008, data32: 0x00000008
[root@lxdaq27 vme]#
</pre>
Successful gefvme kernel driver load messages:
<pre>
dmesg
...
[ 1790.606318] gefvme: loading out-of-tree module taints kernel.
[ 1790.606451] gefvme: module verification failed: signature and/or required key missing - tainting kernel
[ 1790.607544] gefvme: Tundra tsi148 PCI-VME Bridge Driver R01.00-KO19
[ 1790.608354]  Tempe found.
[ 1790.608361]  PCI_CLASS = 06800001
[ 1790.608362]  PCI_INTLINE = 00000013
[ 1790.608364] gefvme: Attempting to allocate resources from bus 6
[ 1790.608369] gefvme: Checking bus 6 resource 0: "PCI Bus 0000:06", start 00000000, end: 00000000, flags 0x00000000, want flags 0x00000200.
[ 1790.608373] gefvme: Checking bus 6 resource 1: "PCI Bus 0000:06", start F0000000, end: F7FFFFFF, flags 0x00000200, want flags 0x00000200.
[ 1790.608377] gefvme: Using bus 6 resource 1, start F0000000, end: F7FFFFFF, flags 0x00000200, size 0x07FFFFFF
[ 1790.608382] gefvme: V7865 PCI VME BAR0  f0000000
[ 1790.608384] PCI MEM start f0010000
[ 1790.608386] PCI MEM end  f7ffffff
[ 1790.608388] PCI MEM size  07feffff (127 MiB)
[ 1790.608420] here! read vme slot 0x5900
[ 1790.608422] Bad VME slot #00
[ 1790.608424] vme_slotnum 0
[ 1790.608428] Waitqueues and sems initialized
[ 1790.611496] interboard data initialized
[ 1790.611505]  Vendor = 10E3  Device = 0148  Revision = 01 Status = 02300006
[ 1790.611507]  Class = 00068000
[ 1790.611512]  Misc0 = 00004008
[ 1790.611513]  Irq = 0013
[ 1790.611515] VME slot number: 0
[ 1790.611753] tsi148::tempe_irqhandler: linux irq 19, tsi148 interrupts enabled: 0x03ff20fe, active: 0x00010000
[ 1790.611936] tsi148: VME master control register VMCTRL: 0x00000003
[ 1790.611941] tsi148: VME control register        VCTRL: 0x00000008
[ 1790.611946] tsi148: VME status register          VSTAT: 0x00000900
[ 1790.611950] tsi148: DMA channel 0 status: 0x02000000
[ 1790.611955] tsi148: DMA channel 1 status: 0x02000000
[ 1790.611959] tsi148: Interrupt enable register    INTEN: 0x03ff20fe
[ 1790.611964] tsi148: Interrupt enable out register INTEO: 0x03ff20fe
[ 1790.611968] tsi148: Control and status register  GCTRL: 0x30000001
[ 1790.612060] gefvme: VME system controller is enabled
[ 2178.687742] vmedrv: allocate_resource 000000001e0d58c3 from 00000000715cdf8e window 0, size 65536
[ 2178.697611] vmedrv: allocate_resource 000000004801253a from 00000000715cdf8e window 1, size 16777216
</pre>
</pre>

Latest revision as of 13:42, 19 August 2024

VME CPU information

Cloning USB and CF Flash boot cards

The V7805 VME CPU can run Linux from USB flash memory, V7865 VME CPU can run Linux from CompactFlash or USB flash memory. Disk size 8 GB or bigger is recommended for running SL5.5 Linux. Highest available speed grade devices should be used: 266X or better for CompactFlash, 200X or "30MB read/15MB write" for USB Flash. Also be aware that some CompactFlash and USB Flash devices have been observed to corrupt Linux filesystems within a few days of use. The specific flash memory brands and models we presently use do not see to have this problem.

When working CompactFlash memory, attach it to a USB-CF adapter and treat it as USB flash memory in the following instructions.

Clone disk using the script clone.perl

  • attach the USB flash disk to some computer connected to the LADD cluster
  • become root
  • check that correct device appears in the device list: fdisk -l
  • assume new device is /dev/sdc
  • select an Linux image that we will clone:
    • 64-bit SL6 image for V7865 VME processors: use /ladd/data1/root/lxiris01
    • 32-bit SL6 image for dual-Athlon machines: use /ladd/data1/root/ladd13
  • cd /home/olchansk/sysadm/clone
  • ./clone.perl /ladd/data1/root/ladd13 /dev/sdc
  • df -kl ### check that /dev/sdc is not mounted
  • disconnect the USB flash disk, try to boot from it.

The clone script has been tested in these configurations:

  • clone 64-bit SL6 VME CPU image to 8GB and 16GB USB flash, GRUB bootloader
  • clone 32-bit SL6 image to 500GB IDE-USB disk, EXTLINUX bootloader.

Note that the clone script has to be run from the correct directory per instructions above as it has to find and run the uuidfix script to make the destination disk bootable.

Clone disk manually

  • attach USB flash to the computer to be cloned (or any computer - we will use rsync to copy the data)
  • become root
  • check that correct device appears in the device list: fdisk -l
  • assume new device is /dev/sdX, original boot disk is /dev/sda.
  • repartition the device:
    • fdisk -H 224 -S 56 /dev/sdX
    • create one partition covering the whole device
    • set partition type 83 (Linux)
    • set bootable flag (command "a")
    • result should look like this:
[root@lxdaq09 ~]# fdisk -l

Disk /dev/sda: 8011 MB, 8011120640 bytes
224 heads, 56 sectors/track, 1247 cylinders
Units = cylinders of 12544 * 512 = 6422528 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        1247     7821156   83  Linux
  • /usr/bin/time mkfs.ext4 -O ^64bit /dev/sdX1 (should take about 30 seconds)
  • tune2fs -i0 -c0 /dev/sdX1
  • mkdir -p /mnt/tmp
  • mount /dev/sdX1 /mnt/tmp
  • CLONE CURRENT BOOT DISK: cd /; rsync -ax . /mnt/tmp; cd /dev; rsync -a . /mnt/tmp/dev
  • CLONE lxdaq08 32-bit boot image (V7805, V7865): cd /ladd/data1/root/lxdaq08; rsync -ax --delete-after . /mnt/tmp
  • CLONE ANOTHER FLASH DISK: cd /another/flash/disk; /usr/bin/time rsync -ax . /mnt/tmp
  • check result: run "df", new filesystem should be about as full as the original one
  • sync; cd /; umount /dev/sdX1; mount /dev/sdX1 /mnt/tmp
  • install SYSLINUX/EXTLINUX boot loader (SL5)
    • install master boot loader: cd /mnt/tmp/boot; dd if=mbr.bin of=/dev/sdX (NOTE: ***NOT** /dev/sdX1)
    • install extlinux boot loader: cd /mnt/tmp/boot; ./extlinux -i . (NOTE: notice the "./" - make sure to run the extlinux executable from .../boot, NOT the one installed in the system)
  • install GRUB boot loader (SL6) --- (NOTE: in the line below, remember to replace "/dev/sdX" with the disk name)
    • echo -e "device (hd0) /dev/sdX\nroot (hd0,0)\nsetup (hd0)\n" | grub --batch --no-floppy
  • update boot disk UUID (SL6) --- (NOTE: dumpe2fs prints the UUID of the new disk, cut-and-paste this UUID into the sed commands below)
    • dumpe2fs /dev/sdX1 | grep UUID
    • edit grub.conf: sed 's/UUID=\S*/UUID=ddc00d49-1c17-4803-ac0b-d6eb89d9e729/' -i /mnt/tmp/boot/grub/grub.conf
    • edit fstab: sed 's/UUID=\S*/UUID=ddc00d49-1c17-4803-ac0b-d6eb89d9e729/' -i /mnt/tmp/etc/fstab
  • cd /; umount /dev/sdX1
  • disconnect the new boot disk, try to boot from it.

Cloning NFS-Root

We will clone "lxsrc" into "lxdst": 

cd lxsrc
mkdir ../lxdst
rsync -av . ../lxdst
vi etc/sysconfig/network ### change HOSTNAME and NIS domain
vi etc/nodeinfo ### change description
vi etc/yp.conf ### add "domain xxx-NIS broadcast" for the new NIS domain
vi etc/fstab ### change the "/" NFS mount-point if needed
vi etc/cron.d/triumf_nodeinfo ### change nodeinfo server name

Cloning Ubuntu NFS-Root

We will clone "lxsrc" into "lxdst": 

cd lxsrc
mkdir ../lxdst
rsync -av . ../lxdst
vi etc/hostname ### change HOSTNAME
vi etc/fstab ### change the "/" NFS mount point
vi opt/gonodeinfo/gonodeinfo.conf ### change description

Extlinux boot file

DEFAULT menu.c32
PROMPT 0
TIMEOUT 50

MENU TITLE TRIUMF DAQ USB BOOT32 ver K.O. 2011feb03

LABEL automatic
  MENU DEFAULT
  com32 ifcpu.c32
  append debug multicore -- linux-V7865-32 -- linux-V7805

LABEL linux-V7865-32
  kernel vmlinuz-2.6.18-194.32.1.el5
  append initrd=initrd-2.6.18-194.32.1.el5.img panic=60 ro rootdelay=5 rootwait ro root=/dev/sda1

LABEL linux-V7865-32-old
  kernel vmlinuz-2.6.18-194.8.1.el5
  append initrd=initrd-2.6.18-194.8.1.el5-32-usbboot.img panic=60 ro rootdelay=5 rootwait ro root=/dev/sda1

LABEL linux-V7865-64
  kernel vmlinuz-2.6.18-194.11.1.el5
  append initrd=initrd-2.6.18-194.11.1.el5.V7865.img panic=60 ro rootdelay=5 rootwait ro root=/dev/sda1

LABEL linux-V7805
  kernel vmlinuz-2.6.34.1-32-v7805
  append panic=60 rootdelay=5 rootwait ro root=/dev/sda1

LABEL memtest
  kernel memtest86+-1.65

#label linux
#  kernel vmlinuz-2.6.34.1-32-v7805
#  append panic=60 rootdelay=5 rootwait ro root=/dev/sda1
#label linux
#  kernel vmlinuz-2.6.34.1-32-v7805
#  append panic=60 root=/dev/nfs nfsroot=142.90.111.60:/home1/laddvme05.triumf.ca,nfsvers=3,tcp,rsize=32768,wsize=32768 ip=::::::dhcp console=ttyS0,115200n8

Updating Linux kernel

Updating Linux kernel on USB/CF flash boot disks

  • install latest kernel: yum update
  • identify latest kernel files:
    • ls -ltr /boot | grep vmlinuz | tail -1
    • ls -ltr /boot | grep initrd | tail -1
  • edit /boot/extlinux.conf
    • duplicate the entry marked "MENU DEFAULT"
    • change file names for the first entry according to the newly installed kernel
    • remove "MENU DEFAULT" from all entries except the new one
  • reboot into the new kernel

Setup GRUB2

# not fdisk -cu -H 224 -S 56 /dev/sdc
# or fdisk -cu -H 64 -S 32 /dev/sdc
# or fdisk -H 224 -S 56 /dev/sdc # start the first partition from cylinder 10
# or fdisk -H 255 -S 63 /dev/sdc # start the first partition from cylinder 10

fdisk -H 224 -S 56 /dev/sdc # start the first partition from cylinder 10
mkfs.ext4 -O ^64bit /dev/sdc1
tune2fs -c0 -i0 /dev/sdc1
mkdir -p /mnt/tmp
mount /dev/sdc1 /mnt/tmp
cd /mnt/tmp
# NOT THIS: rsync -av /ladd/data0/root/lxdaq02-usb/ .
rsync -av /ladd/data0/backup.os/lxdaq02/ .
#cd /home/olchansk/git/grub
#./grub-install -v --directory ./grub-core /dev/sdc --boot-directory=/mnt/tmp/boot --root-directory=/mnt/tmp
cd /mnt/tmp
./opt/grub/sbin/grub-install -v --directory=/mnt/tmp/opt/grub/lib/grub/i386-pc --boot-directory=/mnt/tmp/boot --root-directory=/mnt/tmp /dev/sdc
./root/git/scripts/clone/uuidfix.perl -v /mnt/tmp
xemacs -nw etc/sysconfig/network # change hostname
xemacs -nw etc/sysconfig/network-scripts/ifcfg-eth0 # change IP address
xemacs -nw etc/nodeinfo # change description
cd /
umount /mnt/tmp
eject /dev/sdc

Partition table: 

[root@lxdaq01 ~]# fdisk -l

Disk /dev/sda: 16.0 GB, 16013852672 bytes
255 heads, 63 sectors/track, 1946 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x62f63f6c

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *          10        1946    15558952+  83  Linux
[root@lxdaq01 ~]#

NOT THIS: 

[root@ladd21 grub]# fdisk -lcu /dev/sdc

Disk /dev/sdc: 16.0 GB, 16013852672 bytes
87 heads, 48 sectors/track, 7489 cylinders, total 31277056 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x855bfadf

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1            2048    31277055    15637504   83  Linux
[root@ladd21 grub]#

grub.cgf: 

[root@ladd21 grub]# cat /media/edd35353-82d7-4e18-aeeb-2073ce19c543/boot/grub/grub.cfg
# Begin /boot/grub/grub.cfg
set default=0
set timeout=5

insmod ext3
set root=(hd0,1)

menuentry "memtest86+-4.10" {
        linux16 /boot/memtest86+-4.10
}

menuentry "SL6-V7805-KO2" {
        linux   /boot/vmlinuz-2.6.34.1-V7805-KO2 panic=60 rootwait ro root=/dev/sda1
}

menuentry "GNU/Linux, Linux 3.15.1-lfs-SVN-20140702" {
        linux   /boot/vmlinuz-3.15.1-lfs-SVN-20140702 root=/dev/sda2 ro
}

#

V7805 special settings

Because of hardware incompatibility, the V7805 cannot run vanilla RedHat/SL linux kernels.

Because of hardware incompatibility, the V7805 cannot boot from 16GB USB flash. Please use 8GB USB flash media.

To run RH/SL linux, a special linux kernel has to be built with special PCI configuration:

NOTE: kernel 2.6.34.1 works, newer kernels do not seem to work. 

ladd00:linux-2.6.34.1$ grep PCI .config
CONFIG_PCI_QUIRKS=y
# Bus options (PCI etc.)
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
CONFIG_PCI_GODIRECT=y
# CONFIG_PCI_GOOLPC is not set
# CONFIG_PCI_GOANY is not set
CONFIG_PCI_DIRECT=y
CONFIG_PCI_DOMAINS=y
# CONFIG_PCIEPORTBUS is not set
# CONFIG_PCI_MSI is not set
# CONFIG_PCI_DEBUG is not set
CONFIG_PCI_STUB=y
CONFIG_PCI_IOV=y
# CONFIG_HOTPLUG_PCI is not set
# CONFIG_PATA_CMD640_PCI is not set
CONFIG_NET_PCI=y
# CONFIG_KSZ884X_PCI is not set
# CONFIG_NE2K_PCI is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIAL_8250_PCI=y
# CONFIG_UIO_PCI_GENERIC is not set
ladd00:linux-2.6.34.1$

V7648 and V7750 BIOS Settings

Sometimes the V7648/V7750 boards lose BIOS settings and do not boot. Connect VGA monitor to see the errors.

To reset BIOS settings, short jumper E3 (next to the ethernet connectors, top of the board).

  • legacy diskette A: -> set to "disabled"
  • console redirection -> set to "disabled"
  • local bus ide adapter -> set to "disabled"
  • POST errors -> set to "disabled"

V7805 BIOS Settings

Sometimes the V7805 boards lose BIOS settings and do not boot. Connect DVI (VGA with DVI->VGA adapter) monitor to see the errors.

V7865 BIOS Settings

  • BIOS reset switch is "E11" at the top of the PCB next to the CompactFlash connector. It is the middle one of the 3 two-pin headers.
  • enter BIOS by pressing "DEL" during power up
  • Boot->Boot setting "Wait for F1" set to "Disabled"
  • Chipset->South Bridge "USB 2.0 Controller" set to "Enabled"
  • Chipset->Onboard devices->FP ethernet set to "Enabled"
  • Advanced->IDE configuration set to "Disabled" (unless using CompactFlash boot disk)
  • Advanced->Remote access set to "Disabled"
  • Advanced->USB->"USB 2.0 Controller Mode" set to "HiSpeed"
  • Advanced->Onboard device->"Reserve VME space" set to "512 Mbytes".
  • Exit-> "Save changes and Exit"

XVB-602 BIOS Settings

  • BIOS reset switch is S12 on the back of the board
  • enter BIOS by pressing "F2" or "DEL" during power up
  • Advanced -> Onboard device -> "Reserve VME space" set to "512 Mbytes".
  • -> "Onboard gigabit LAN 1" set to "enable"
  • -> "Onboard gigabit LAN 2" set to "enable"
  • "Save & Exit" -> "Save changes and exit"
  • the processor will restart and try to boot from network using PXE.

V7750 typical sensors output

[root@lxdaq14 ~]# sensors
max1617-i2c-0-18
Adapter: SMBus I801 adapter at 1810
temp1:       +32.0 C  (low  = -55.0 C, high = +127.0 C)  
temp2:       +45.0 C  (low  = -55.0 C, high = +127.0 C)  

pc87366-isa-efe0
Adapter: ISA adapter
in0:         +1.83 V  (min =  +0.00 V, max =  +2.95 V)
in1:         +2.65 V  (min =  +0.00 V, max =  +2.95 V)
in2:         +0.00 V  (min =  +0.00 V, max =  +2.95 V)
in3:         +2.46 V  (min =  +0.00 V, max =  +2.95 V)
in4:         +1.74 V  (min =  +0.00 V, max =  +2.95 V)
in5:         +1.09 V  (min =  +0.00 V, max =  +2.95 V)
in6:         +2.51 V  (min =  +0.00 V, max =  +2.95 V)
3VSB:        +3.36 V  (min =  +2.99 V, max =  +3.59 V)
VDD:         +3.38 V  (min =  +2.99 V, max =  +3.59 V)
Vbat:        +2.95 V  (min =  +0.00 V, max =  +2.95 V)
AVDD:        +3.36 V  (min =  +2.99 V, max =  +3.59 V)
temp4:        +0.0 C  (low  =  +0.0 C, high =  +0.0 C)  
                      (crit =  +0.0 C)                  
temp5:        +0.0 C  (low  =  +0.0 C, high =  +0.0 C)  
                      (crit =  +0.0 C)                  
temp6:        +1.7 C  (low  =  +0.0 C, high =  +3.0 C)  
                      (crit =  +3.0 C)                  
cpu0_vid:   +1.325 V

[root@lxdaq14 ~]#

Network boot

Explanation

Network booting of linux computers is done by, in order:

  • PXE (in-BIOS, in-EFI-BOOT or GPXE booted from disk) issues DHCP request to learn the IP address and further boot instructions
  • DHCP server (/etc/dhcp/dhcpd.conf) responds with the IP address and instructions to boot pxelinux.0
  • TFTP server (with the xinetd server) provides access to files in /var/lib/tftpboot on the boot host
  • pxe and pxelinux use tftp to load pxelinux.0, the pxelinux config file (pxelinux.cfg/default or as specified in dhcpd.conf), the linux kernel and the linux initramfs files
  • the linux kernel uses DHCP (again) to configure the network and mount the root file system (as specified in dhcpd.conf)
  • NFS server on the boot host provides access to the root filesystem.

After booting is complete only the NFS server is required for running Linux. After Linux is fully booted, DHCP, TFTP and PXE are not used (until reboot).

Setup the boot host computer (before el7)

install packages

yum install dhcp tftp-server

configure dhcpd

  • create emacs -nw /etc/logrotate.d/dhcp
/var/log/dhcp.log {
    weekly
    notifempty
    missingok
    postrotate
	/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
	/bin/kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}
  • put this in emacs -nw /etc/rsyslog.conf right after the line "#### RULES ####"
#### RULES ####

# Annoying dhcp spam ... (need to add to rotation ...)
local3.*						-/var/log/dhcp.log
local3.*						~
  • chmod a+rx /etc/dhcp
  • emacs -nw /etc/dhcp/dhcpd.conf - add contents from the next section
  • emacs -nw /etc/dhcp/dhcpd.conf - customize for this boot host:
    • change "next-server" to the hostname of this host
    • change "host" entries
  • chkconfig dhcpd on
  • service rsyslog restart # to enable dhcp.log
  • service dhcpd restart # watch /var/log/messages for errors, watch /var/log/dhcp.log for dhcp activity
  • try to boot the VME CPU, watch messages in dhcp.log should be similar to those shown in the next section, the VME CPU should get an IP address and stop on "TFTP load" stage

setup tftp

  • emacs -nw /etc/xinetd.d/tftp line "server_args" to read "server_args = -vvvv -s /var/lib/tftpboot"
  • chkconfig tftp on
  • chkconfig xinetd on
  • service xinetd restart
  • try to boot the VME CPU, watch messages in the syslog, should see message about "pxelinux.0 not found", the VME CPU should get an IP address and stop with an error "TFTP - File not found"

setup pxelinux

cd ~
wget https://www.kernel.org/pub/linux/utils/boot/syslinux/4.xx/syslinux-4.03.tar.bz2
tar xjvf syslinux-4.03.tar.bz2
cd syslinux-4.03
cp -pv ./core/pxelinux.0 ./com32/hdt/hdt.c32 ./memdisk/memdisk ./com32/menu/menu.c32 /var/lib/tftpboot/
  • mkdir /var/lib/tftpboot/pxelinux.cfg
  • create /var/lib/tftpboot/pxelinux.cfg/default with contents from the next section
  • get additional boot files
cd /var/lib/tftpboot
wget http://ladd00.triumf.ca/tftpboot/memtest86+-4.20.iso.zip
wget http://ladd00.triumf.ca/tftpboot/memtest86+-5.01.iso.gz
wget http://ladd00.triumf.ca/tftpboot/modules.alias
wget http://ladd00.triumf.ca/tftpboot/modules.pcimap
wget http://ladd00.triumf.ca/tftpboot/pci.ids

setup linux kernel files

  • get vmlinuz and initramfs files from the VME processor boot image
  • note that the initramfs file needs to be built with the dracut-network function, it should be about 2.5Mbytes in size.
  • note the file names must match the file names in the pxelinux config file
wget http://ladd00.triumf.ca/tftpboot/vmlinuz-2.6.32-431.11.2.el6.i686
wget http://ladd00.triumf.ca/tftpboot/initramfs-2.6.32-431.11.2.el6.i686.img
  • try to boot the VME CPU, watch messages in the syslog, the VME CPU should get an IP address, load the PXELINUX menu, load the linux kernel and stop with an error mounting NFS filesystems.

setup NFS export of VME processor boot image

  • copy the VME processor boot image to /nfsroot/alphavme03 (or /home1/alphavme03)
  • customise the boot image per instructions in previous section (edit sysconfig/network, etc)
  • edit /etc/exports:
/nfsroot/alphavme03 daqtmp4(rw,no_root_squash,async)
  • chkconfig nfs on; service nfs restart
  • if needed adjust the boot image path name in dhcpd.conf to match the export name in /etc/exports, restart dhcpd.conf
  • try to boot the VME CPU, watch messages in the syslog, the VME CPU should get an IP address, load the PXELINUX menu, load the linux kernel, linux kernel will start, mount the VME processor boot image ("NFS Root"), boot the userland. At the end you should be able to ping and ssh into the VME processor.

fixup file permissions

cd /var/lib/tftpboot
chown -R root.root .
chmod a+r *
chmod a-w *

Setup the boot host computer (el7)

install packages (el7)

yum install dhcp tftp-server

configure dhcpd (el7)

  • create emacs -nw /etc/logrotate.d/dhcp
/var/log/dhcp.log {
    weekly
    notifempty
    missingok
    postrotate
	/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
	/bin/kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}
  • put this in emacs -nw /etc/rsyslog.conf right after the line "#### RULES ####"
#### RULES ####

# Annoying dhcp spam ... (need to add to rotation ...)
local3.*						-/var/log/dhcp.log
local3.*						~
  • systemctl restart rsyslog
  • chmod a+rx /etc/dhcp
  • emacs -nw /etc/dhcp/dhcpd.conf - add contents from the next section
  • emacs -nw /etc/dhcp/dhcpd.conf - customize for this boot host:
    • change "next-server" to the hostname of this host
    • change "host" entries
  • systemctl enable dhcpd
  • systemctl start dhcpd
  • firewall-cmd --add-service=dhcp --permanent
  • firewall-cmd --reload
  • systemctl restart dhcpd # watch /var/log/messages for errors, watch /var/log/dhcp.log for dhcp activity
  • chmod a+r /var/log/dhcp.log
  • try to boot the VME CPU, watch messages in dhcp.log should be similar to those shown in the next section, the VME CPU should get an IP address and stop on "TFTP load" stage

setup tftp (el7)

emacs -nw /usr/lib/systemd/system/tftp.service # line "ExecStart" to have ".../in.tftpd -vvvv -s /var/lib/tftpboot"
systemctl start tftp.socket
systemctl enable tftp.socket
firewall-cmd --add-service=tftp --permanent
firewall-cmd --reload
  • try to boot the VME CPU, watch messages in the syslog, should see message about "pxelinux.0 not found", the VME CPU should get an IP address and stop with an error "TFTP - File not found"

setup pxelinux (preliminary)

cd ~
wget https://www.kernel.org/pub/linux/utils/boot/syslinux/4.xx/syslinux-4.03.tar.bz2
tar xjvf syslinux-4.03.tar.bz2
cd syslinux-4.03
cp -pv ./core/pxelinux.0 ./com32/hdt/hdt.c32 ./memdisk/memdisk ./com32/menu/menu.c32 /var/lib/tftpboot/
  • mkdir /var/lib/tftpboot/pxelinux.cfg
  • create /var/lib/tftpboot/pxelinux.cfg/default with contents from the next section
  • get additional boot files
cd /var/lib/tftpboot
wget http://ladd00.triumf.ca/tftpboot/memtest86+-4.20.iso.zip
wget http://ladd00.triumf.ca/tftpboot/memtest86+-5.01.iso.gz
wget http://ladd00.triumf.ca/tftpboot/modules.alias
wget http://ladd00.triumf.ca/tftpboot/modules.pcimap
wget http://ladd00.triumf.ca/tftpboot/pci.ids

setup linux kernel files

  • get vmlinuz and initramfs files from the VME processor boot image
  • note that the initramfs file needs to be built with the dracut-network function, it should be about 2.5Mbytes in size.
  • note the file names must match the file names in the pxelinux config file
wget http://ladd00.triumf.ca/tftpboot/vmlinuz-2.6.32-431.11.2.el6.i686
wget http://ladd00.triumf.ca/tftpboot/initramfs-2.6.32-431.11.2.el6.i686.img
  • try to boot the VME CPU, watch messages in the syslog, the VME CPU should get an IP address, load the PXELINUX menu, load the linux kernel and stop with an error mounting NFS filesystems.

setup NFS export of VME processor boot image

  • copy the VME processor boot image to /nfsroot/alphavme03 (or /home1/alphavme03)
  • customise the boot image per instructions in previous section (edit sysconfig/network, etc)
  • edit /etc/exports:
/nfsroot/alphavme03 daqtmp4(rw,no_root_squash,async)
  • chkconfig nfs on; service nfs restart
  • if needed adjust the boot image path name in dhcpd.conf to match the export name in /etc/exports, restart dhcpd.conf
  • try to boot the VME CPU, watch messages in the syslog, the VME CPU should get an IP address, load the PXELINUX menu, load the linux kernel, linux kernel will start, mount the VME processor boot image ("NFS Root"), boot the userland. At the end you should be able to ping and ssh into the VME processor.

fixup file permissions

cd /var/lib/tftpboot
chown -R root.root .
chmod a+r *
chmod a-w *

DHCP configuration

#
# /etc/dhcpd.conf
#
#
# general setup
#

log-facility local3;

# make network booting the SystemImager autoinstallclient possible
allow booting;
allow bootp;
ignore unknown-clients;
ddns-update-style ad-hoc;

# set lease time to 3 days
default-lease-time 259200;
max-lease-time 259200;

# define network addresses

subnet 142.90.96.0 netmask 255.255.224.0 {
  not authoritative;
  ignore unknown-clients;
  option domain-name "triumf.ca"; 
  option domain-name-servers 142.90.100.19;
  option routers 142.90.100.18; 
} 

# special PXELINUX options
 
option space pxelinux; 
option pxelinux.magic      code 208 = string; 
option pxelinux.configfile code 209 = text; 
option pxelinux.pathprefix code 210 = text; 
option pxelinux.reboottime code 211 = unsigned integer 32; 

#
# setup for TIGRESS VME processors (boot from midtig06)
#
 
group { 
        filename "pxelinux.0"; 
        #next-server ladd00; 
        option routers 142.90.100.18; 
        option subnet-mask 255.255.224.0; 
        option domain-name "triumf.ca"; 
        option domain-name-servers 142.90.100.19, 142.90.100.68; 
        #use-host-decl-names on; 

        site-option-space "pxelinux"; 
        if exists dhcp-parameter-request-list { 
                # Always send the PXELINUX options (specified in hexadecimal) 
                option dhcp-parameter-request-list = concat(option dhcp-parameter-request-list,d0,d1,d2,d3); 
        } 
 
        option pxelinux.reboottime 10; 
        option pxelinux.pathprefix "./"; 
        #option pxelinux.configfile "pxelinux.cfg/default";
        option pxelinux.configfile "pxelinux.cfg/V7750-SL6a";
 
        #host lxdaq17-eth0 { option pxelinux.configfile "pxelinux.cfg/V7750-SL6a"; option host-name "lxdaq17.triumf.ca"; option root-path "nfs:ladd00:/data0/root/lxdaq17:rw"; fixed-address lxdaq17; hardware ethernet 00:20:38:00:DA:1D; } 
        #host lxdaq17-eth1 { option pxelinux.configfile "pxelinux.cfg/V7750-SL6a"; option host-name "lxdaq17.triumf.ca"; option root-path "nfs:ladd00:/data0/root/lxdaq17:rw"; fixed-address lxdaq17; hardware ethernet 00:20:38:00:DA:1F; } 

}

PXELINUX configuration

[root@ladd00 pxelinux.cfg]# more /var/lib/tftpboot/pxelinux.cfg/V7750-SL6d
default menu.c32
prompt 0

menu title Welcome to the LADD00 VME V7750 PXE boot menu

timeout 50

label hdt
  kernel hdt.c32

label memtest86+-5.01 
  kernel memdisk iso initrd=memtest86+-5.01.iso.gz 

label memtest86+-4.20
  kernel memdisk iso initrd=memtest86+-4.20.iso.zip

label SL6-431.11.2
  menu default
  kernel vmlinuz-2.6.32-431.11.2.el6.i686
  append default_hugepagesz=0 hugepages=0 hugepagesz=0 highmem=0 userpte=nohigh apm=off acpi=off initrd=initramfs-2.6.32-431.11.2.el6.i686.img root=dhcp panic=60

label SL6-358.18
  kernel vmlinuz-2.6.32-358.18.1.el6.i686
  append default_hugepagesz=0 hugepages=0 hugepagesz=0 highmem=0 userpte=nohigh apm=off acpi=off initrd=initramfs-2.6.32-358.18.1.el6.i686.img root=dhcp panic=60

label SL6-220.4
  kernel vmlinuz-2.6.32-220.4.1.el6.i686
  append default_hugepagesz=0 hugepages=0 hugepagesz=0 highmem=0 userpte=nohigh apm=off acpi=off initrd=initramfs-2.6.32-220.4.1.el6.i686.img root=dhcp panic=60

#end

Boot host dhcp messages

grep -i b4 /var/log/dhcp.log
[root@daqtmp5 Desktop]# grep -i 0e:b4 /var/log/dhcp.log 
Jun 10 12:50:26 daqtmp5 dhcpd: DHCPDISCOVER from 00:20:38:03:0e:b4 via eth0
Jun 10 12:50:26 daqtmp5 dhcpd: DHCPOFFER on 142.90.96.104 to 00:20:38:03:0e:b4 via eth0
Jun 10 12:50:28 daqtmp5 dhcpd: DHCPREQUEST for 142.90.96.104 (142.90.96.105) from 00:20:38:03:0e:b4 via eth0
Jun 10 12:50:28 daqtmp5 dhcpd: DHCPACK on 142.90.96.104 to 00:20:38:03:0e:b4 via eth0

Boot host syslog messages

tail -100f /var/log/messages
May  5 15:54:08 ladd00 xinetd[1860]: START: tftp pid=3396 from=142.90.111.107
May  5 15:54:08 ladd00 in.tftpd[3398]: RRQ from 142.90.111.107 filename pxelinux.0
May  5 15:54:08 ladd00 in.tftpd[3398]: tftp: client does not accept options
May  5 15:54:08 ladd00 in.tftpd[3400]: RRQ from 142.90.111.107 filename pxelinux.0
May  5 15:54:08 ladd00 in.tftpd[3402]: RRQ from 142.90.111.107 filename ./pxelinux.cfg/V7750-SL6d
May  5 15:54:08 ladd00 in.tftpd[3404]: RRQ from 142.90.111.107 filename ./menu.c32
May  5 15:54:08 ladd00 in.tftpd[3406]: RRQ from 142.90.111.107 filename ./pxelinux.cfg/V7750-SL6d
May  5 15:54:13 ladd00 in.tftpd[3418]: RRQ from 142.90.111.107 filename ./vmlinuz-2.6.32-431.11.2.el6.i686
May  5 15:54:14 ladd00 in.tftpd[3431]: RRQ from 142.90.111.107 filename ./initramfs-2.6.32-431.11.2.el6.i686.img
May  5 15:54:29 ladd00 rpc.mountd[1796]: authenticated mount request from lxdaq17.triumf.ca:677 for /data0/root/lxdaq17 (/data0)

V7648, V7750, V7805, V7851 : Setup vme_universe VME drivers

Install Linux kernel driver: 

ssh root@lxdragon01
yum install -y kernel-devel ### el6, el7
apt install -y linux-headers-686 ### Ubuntu, Debian
#apt install -y linux-headers-6.1.0-22-686 ### non-pae kernel headers
git clone https://daq00.triumf.ca/~olchansk/git/kernel-module-universe.git
cd kernel-module-universe/vme_universe
make
/bin/rm /usr/lib/libvme.so*
/bin/rm /root/vme_universe.ko
make install
cat <<EOF >> /etc/rc.local
mkdir -p /dev/bus/vme
mknod /dev/bus/vme/ctl c 221 8
chmod a+wr /dev/bus/vme/ctl
modprobe vme_universe
#insmod /root/vme_universe.ko
EOF
rmmod vme_universe
/etc/rc.local
dmesg # observe VME driver successful load messages (see below)
*** reboot *** shutdown -r now # reboot to confirm the vme driver loads correctly

Install MIDAS driver and test software: 

cd ~
mkdir packages
cd packages
git clone http://ladd00.triumf.ca/~daqweb/git/midas.git
git clone http://ladd00.triumf.ca/~daqweb/git/mxml.git
svn checkout https://ladd00.triumf.ca/svn/daqsvn/trunk/vme # username svn, password svn
cd vme
make -k # some programs will fail to build, this is expected.

Test VME access: 

[root@lxdragon01 vme]# ./vmescan.exe 
vmic_mmap: Mapped VME AM 0x0d addr 0x00000000 size 0x00ffffff at address 0x20100000
mvme_open:
Bus handle              = 0x3
DMA handle              = 0x93f93f0
DMA area size           = 1048576 bytes
DMA    physical address = 0xb6691000
vmic_mmap: Mapped VME AM 0x2d addr 0x00000000 size 0x0000ffff at address 0x26ff0000
vmic_mmap: Mapped VME AM 0x3d addr 0x00000000 size 0x00ffffff at address 0x21100000
at 0x00100000: data16: 0x0215, data32: 0x01130215
at 0x00110000: data16: 0xffff, data32: 0x06000000
at 0x00200000: data16: 0x0215, data32: 0x01130215
at 0x008000fc: data16: 0x0851, data32: 0xffffffff (CAEN V513)
at 0x008100fc: data16: 0x0851, data32: 0xffffffff (CAEN V513)
at 0x00e00000: data16: 0xffff, data32: 0xc0000000
mvme_close:
Bus handle              = 0x3
DMA handle              = 0x93f93f0
DMA    physical address = 0xb6691000
[root@lxdragon01 vme]#

Successful vme_universe kernel driver load messages: 

dmesg
...
VME: Board is system controller
VME: Driver compiled for SMP system
VME: Installed VME Universe module version: 3.6.KO5

V7865 and XVB-602 : Setup gefvme/tsi148 VME drivers

(ubuntu&etc with recent linux kernels): Check that the "wrong" driver is disabled and is not loaded. 

lsmod | grep vme_tsi148
rmmod vme_tsi148
echo blacklist vme_tsi148 > /etc/modprobe.d/blacklist-vme_tsi148.conf
and reboot

(ubuntu&etc): Check that /etc/rc.local is enabled, see https://daq.triumf.ca/DaqWiki/index.php/Ubuntu#Enable_rc.local and https://daq.triumf.ca/DaqWiki/index.php/SLinstall#Post_installation_CentOS7

Install Linux kernel driver: 

ssh root@lxdaq27
cd ~
mkdir git
cd git
git clone https://daq00.triumf.ca/~olchansk/git/kernel-module-gefvme.git
cd kernel-module-gefvme
make
make install
sh /etc/rc.local
dmesg # observe VME driver successful load messages (see below)

Install MIDAS driver and test software: 

cd ~
mkdir git
cd git
git clone https://bitbucket.org/tmidas/midas --recursive
git clone https://bitbucket.org/ttriumfdaq/vme
cd vme
make -k MIDASSYS=../midas # some programs will fail to build, this is expected.

Test VME access: 

[root@lxdaq27 vme]# ./vmescan_gef.exe
gefvme_openWindow: Slot 0, VME addr 0x00000000, am 0x00, size 0x00000004 4
gefvme_openWindow: Configured window 0 VME AM 0x2d A16 addr 0x00000000 size 0x00010000 fd 3
gefvme_openWindow: Slot 1, VME addr 0x00ff0000, am 0x00, size 0x00000004 4
gefvme_openWindow: Configured window 1 VME AM 0x3d A24 addr 0x00000000 size 0x01000000 fd 4
at 0x00100000: data16: 0x0925, data32: 0x01120925
at 0x00af0000: data16: 0x0008, data32: 0x00000008
[root@lxdaq27 vme]#

Successful gefvme kernel driver load messages: 

dmesg
...
[ 1790.606318] gefvme: loading out-of-tree module taints kernel.
[ 1790.606451] gefvme: module verification failed: signature and/or required key missing - tainting kernel
[ 1790.607544] gefvme: Tundra tsi148 PCI-VME Bridge Driver R01.00-KO19
[ 1790.608354]   Tempe found.
[ 1790.608361]   PCI_CLASS = 06800001
[ 1790.608362]   PCI_INTLINE = 00000013
[ 1790.608364] gefvme: Attempting to allocate resources from bus 6
[ 1790.608369] gefvme: Checking bus 6 resource 0: "PCI Bus 0000:06", start 00000000, end: 00000000, flags 0x00000000, want flags 0x00000200.
[ 1790.608373] gefvme: Checking bus 6 resource 1: "PCI Bus 0000:06", start F0000000, end: F7FFFFFF, flags 0x00000200, want flags 0x00000200.
[ 1790.608377] gefvme: Using bus 6 resource 1, start F0000000, end: F7FFFFFF, flags 0x00000200, size 0x07FFFFFF
[ 1790.608382] gefvme: V7865 PCI VME BAR0  f0000000
[ 1790.608384] PCI MEM start f0010000
[ 1790.608386] PCI MEM end   f7ffffff
[ 1790.608388] PCI MEM size  07feffff (127 MiB)
[ 1790.608420] here! read vme slot 0x5900
[ 1790.608422] Bad VME slot #00
[ 1790.608424] vme_slotnum 0
[ 1790.608428] Waitqueues and sems initialized
[ 1790.611496] interboard data initialized
[ 1790.611505]   Vendor = 10E3  Device = 0148  Revision = 01 Status = 02300006
[ 1790.611507]   Class = 00068000
[ 1790.611512]   Misc0 = 00004008
[ 1790.611513]   Irq = 0013
[ 1790.611515] VME slot number: 0
[ 1790.611753] tsi148::tempe_irqhandler: linux irq 19, tsi148 interrupts enabled: 0x03ff20fe, active: 0x00010000
[ 1790.611936] tsi148: VME master control register VMCTRL: 0x00000003
[ 1790.611941] tsi148: VME control register         VCTRL: 0x00000008
[ 1790.611946] tsi148: VME status register          VSTAT: 0x00000900
[ 1790.611950] tsi148: DMA channel 0 status: 0x02000000
[ 1790.611955] tsi148: DMA channel 1 status: 0x02000000
[ 1790.611959] tsi148: Interrupt enable register     INTEN: 0x03ff20fe
[ 1790.611964] tsi148: Interrupt enable out register INTEO: 0x03ff20fe
[ 1790.611968] tsi148: Control and status register   GCTRL: 0x30000001
[ 1790.612060] gefvme: VME system controller is enabled
[ 2178.687742] vmedrv: allocate_resource 000000001e0d58c3 from 00000000715cdf8e window 0, size 65536
[ 2178.697611] vmedrv: allocate_resource 000000004801253a from 00000000715cdf8e window 1, size 16777216
Retrieved from "https://daq00.triumf.ca/DaqWiki/index.php?title=VME-CPU&oldid=7889"