Chronobox

Links

• https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano
• https://bitbucket.org/expalpha/chronobox_firmware
• https://bitbucket.org/expalpha/chronobox_software
• https://edev-group.triumf.ca/hw/alphag/chrono-box/rev0
• https://bitbucket.org/expalpha/chronobox_firmware/src/master/docs/Chrono.pdf

Chronobox connectors

|
| LEMO - CLK_IN - jumpers: SINE<->CLK<->NIM/TTL and NIM<->CLK<->TTL
|
| ECL P1
| 32
| |
| 1
|
| ECL P2
| 32
| |
| 1
|
| PLED - power-on LED
|
| jumper: bank A direction IN<->X<->OUT
| LEMO 0-1
| LED  0-1
| LEMO 2-3
| LED  2-3
|
| jumper: bank B direction IN<->X<->OUT
| LEMO 4-5
| LED  4-5
| LEMO 6-7
| LED  6-7
|

Electrical connections

• ECL P1 and P2: LeCroy 4616 compatible
• LEMO TTL inputs: 0..5V, logic level 0: <0.8V, logic level 1: >2V
• LEMO TTL outputs: logic level 0: 0V, logic level 1: 5V to 3.3V
• clock NIM input: NIM compatible
• clock TTL input: ???
• clock sine wave input: ???

Input channel mapping

• 0+16 : first ECL connector
• 16+16 : second ECL connector
• 32+8 : LEMO inputs (TTL)
• 40+18 : GPIO inputs (FPGA pins)
• 58 : external clock (10 MHz nominal)
• 59 : internal clock (100 MHz)

Functional units

Scalers

TBW

Firmware 0x61832695

• minimum pulse width (leading edge to trailing edge): longer than 10 ns (15 ns is safe) (sampled by 100 MHz/ 10 ns clock)
• minimum time between pulses (trailing edge to next leading edge): longer than 10 ns (15 ns is safe) (ditto)

Firmware 0x624e22ad

• minimum pulse width: leading edge triggered (2 ns is safe)
• minimum time between pulses (trailing edge to next leading edge): 25 ns or more is safe

Timestamps (TSC)

TBW

Firmware 0x61832695

• minimum pulse width (leading edge to trailing edge): longer than 100 ns (105 ns is safe) (sampled by 10 MHz/100 ns clock)
• minimum time between pulses (trailing edge to next leading edge): longer than 100 ns (105 ns is safe) (ditto)

Firmware 0x624e22ad

• minimum pulse width: leading edge triggered (5 ns is safe)
• minimum time between pulses (trailing edge to next leading edge): longer than 200 ns (205 ns is safe) (sampled by 10 MHz/100 ns clock)

Each TSC input has a 256 entry FIFO. (LE is 1 entry, TE is 1 entry, so 128 hits if both LE and TE enabled).

Output data fifo

TBW

Firmware before 0x61832695

Output fifo is 4096 entries deep.

Firmware 0x624e22ad

Output fifo is 64k entries deep.

Initial hardware setup

• refer to DE10-NANO information here: https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano
• prepare hardware:
  • remove chronobox from enclosure (cannot access JTAG connector when installed)
  • remove DE10-Nano from chronobox baseboard
  • check that SW10 jumpers are: U-D-U-U-D-U. (U=up, D=down, when "ALTERA" text on the FPGA is right side up)
  • reinstall DE10-Nano on chronobox baseboard
  • do NOT reinstall chronobox in enclosure
• prepare software:
  • check that you have quartus 17.1 (at /opt/intelFPGA/17.1/ in this example)
  • check that you installed chronobox software in /home/agdaq/online/chronobox_software
  • check that you installed chronobox firmware project in /home/agdaq/online/firmware/git/chronobox_firmware
• connect serial console, for more info, go here: https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano#Serial_console
  • connect a mini-USB (*not* Micto-USB!) cable from mini-USB port on the side of the ethernet connection to a PC
  • minicom -D /dev/ttyUSB0 -b 115200
• connect fpga jtag, for more info, go here: https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano#JTAG
  • connect a mini-USB (*not* Micro-USB!) cable on the side of HDMI and power connectors to a PC
  • /opt/intelFPGA/17.1/quartus/bin/jtagconfig
• prepare SD flash card for booting from network: generic instructions are here: https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano#boot_Linux_from_SD_flash
  • connect new or used 8GB SD flash card to USB flash adapter to a linux computer
  • login as root
  • identify the flash device as /dev/sdX. Use "lsblk" or "blkid" or "dmesg". be careful to not write to the computer system disk (usually /dev/sda)
  • echo -e 'o\nn\np\n1\n2048\n4095\nn\np\n2\n4096\n1681816\nn\np\n3\n\n\nt\n1\na2\nt\n2\nb\nw' | fdisk /dev/sdX
  • eject /dev/sdX
  • remove card, reintall card, this is to ensure we use the new partition table
  • fdisk -l /dev/sdX

root@daq01:~# fdisk -l /dev/sdg
Disk /dev/sdg: 7.38 GiB, 7910457344 bytes, 15450112 sectors
Disk model: STORAGE DEVICE  
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xa033868b

Device     Boot   Start      End  Sectors   Size Id Type
/dev/sdg1          2048     4095     2048     1M a2 unknown
/dev/sdg2          4096  1681816  1677721 819.2M  b W95 FAT32
/dev/sdg3       1683456 15450111 13766656   6.6G 83 Linux
root@daq01:~# 

• • dd if=/daq/daqstore/olchansk/daq/DE10-Nano/image-1-fpga.img of=/dev/sdX1 bs=1024k
  • dd if=/daq/daqstore/olchansk/daq/DE10-Nano/image-2-uboot.img of=/dev/sdX2 bs=1024k
  • eject /dev/sdX
• install prepared flash card into the chronobox DE10-Nano
• cycle power
• in the minicom window, IMMEDIATELY press the space bar to stop automatic booting (if it doesn't react, ensure flow control is disabled in minicom)
• you will have the uboot "=>" prompt
• confirm uboot version: U-Boot 2013.01.01
• setup uboot to boot linux from network, more details here: https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano#boot_Linux_from_network
  • setenv ethaddr 02:aa:bb:cc:cb:04 # for cb04, see below for explanation
  • setenv bootcmd 'run bridge_enable_handoff; run netboot'
  • setenv ramboot bootz \${loadaddr} - \${fdtaddr}
  • setenv bootargs console=ttyS0,115200 ip=dhcp root=/dev/nfs rw nfsroot=192.168.1.1:/zssd1tb/nfsroot/%s,vers=3 panic=15
  • saveenv
  • reset
• DE10-Nano should boot into linux:
  • uboot will restart
  • DHCP will run
  • tftp load of linux kernel will run
  • linux kernel will start
  • linux kernel will get an IP address
  • linux kernel will NFS-mount the root filesystem (.../nfsroot/cb04)
  • systemd will start all services
  • there will be a login prompt on minicom console
  • ssh will work (maybe 1 minute delay before it starts)
• try a few things:
  • from agdaq or agmini account, ssh root@cb04
  • "df" should show /home/agdaq is mounted (and ssh agdaq@cb04 should work)
  • "/home/agdaq/online/chronobox_software/test_cb.exe 0" should fail ("bus error") because FPGA is not loaded yet
• load sof file, for more info, go here: https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano#JTAG
  • /opt/intelFPGA/17.1/quartus/bin/quartus_pgm -m JTAG -o "p;/home/agdaq/online/firmware/git/chronobox_firmware/output_files/DE10_NANO_SoC_GHRD.sof@2"
• "test_cb.exe 0" should work now
• use srunner_cb.exe to load the pof file into FPGA boot flash
• use reboot_cb.exe to reboot the FPGA from flash
• "test_cb.exe 0" should report the expected FPGA firmware version number.
• success.

assign ethernet MAC address

Usually ethernet MAC address is assigned by manufacturer and is stored in a tiny little flash chip. The DE10-Nano board is too cheap and does not have it. So a fake ethernet address has to be assigned manually from the "locally administered range" of MAC addresses that start from "02:...", see https://en.wikipedia.org/wiki/MAC_address#Ranges_of_group_and_locally_administered_addresses

All MAC addresses must be unique on an ethernet network, but with manual assignement confusion is easy to create.

For chronobox devices, use MAC addresses that start with "l02:aa:bb:cc:xx:yy" and have last digits "cb:01", "cb:02", "cb:03", "cb04" for the first four chronoboxes.

new board test checklist

to check newly built chronobox baseboard, follow this checklist.

test ECL inputs

• stop midas frontend
• ssh agdaq@alphagdaq ### or agmini@daq16
• ssh root@cb03
• /home/agdaq/online/chronobox_software/test_cb.exe 0 ### read firmware revision, confirm communication with chronobox firmware
• /home/agdaq/online/chronobox_software/test_cb.exe inputs ### output should look like this:

inputs: ecl_p1: 0xaa55, ecl_p2: 0xaa55, lemo: 0xe0, gpio: 0x3ffff

• connect ECL ribbon cable from LeCroy 4616 NIM-to-ECL converter to chronobox ECL_P1 input (left one)
• ecl_p1 input should read 0x0000
• connect lemo jumper from NIM logic level 1 otput (i.e. any module OUT-bar output) to first channel of LeCroy 4616, ecl_p1 should read 0x0001
• move to 2nd channel, get 0x0002
• and so forth until last channel 0x8000.
• if any channel does not give expected reading, attach label (i.e. "ECL_P1 0x0002 bad") and send for repair.
• repeat with ECL_P2 input (right one), watch printed value of ecl_p2.

test TTL inputs

• NOTE: front panel LEMO connector labels are wrong. correct numbering of LEMO connectors is as shown in #Chronobox connectors
• disconnect all LEMO inputs
• set "bank A direction" and "bank B direction" for input
• /home/agdaq/online/chronobox_software/test_cb.exe lemo_in_a # set bank A to input
• /home/agdaq/online/chronobox_software/test_cb.exe lemo_in_b # set bank B to input
• all 8 LEMO LEDs should be on
• /home/agdaq/online/chronobox_software/test_cb.exe inputs ### lemo inputs should read 0xFF
• connect LEMO 0 (left bottom) to TTL output of Lecroy 222 or equivalent, set "scale" to "latch", operate push buttons to turn output on and off (LED should come in and out).
• /home/agdaq/online/chronobox_software/test_cb.exe inputs

inputs: ecl_p1: 0xaa55, ecl_p2: 0xaa55, lemo: 0xfe (~0x01), gpio: 0x3ffff

• change TTL signal from logic level 0 to 1 and back, observe the corresponding LED to turn on and off, observe the correct bit in the "lemo" column (watch the inverted input ~0x01) change.
• repeat with LEMO input 1 (left top) through 7 (right top).

test TTL outputs

• NOTE: front panel LEMO connector labels are wrong. correct numbering of LEMO connectors is as shown in #Chronobox connectors
• set "bank A direction" and "bank B direction" for output
• /home/agdaq/online/chronobox_software/test_cb.exe lemo_out_a 0
• /home/agdaq/online/chronobox_software/test_cb.exe lemo_out_b 0
• all LEMO LEDs should be off
• with voltmeter measure LEMO output voltage should be 0.135 V
• /home/agdaq/online/chronobox_software/test_cb.exe lemo_out_a 0xF
• /home/agdaq/online/chronobox_software/test_cb.exe lemo_out_b 0xF
• all LEMO LEDs should be on
• with voltmeter measure LEMO output voltage should be 3.4 to 3.8 V
• if you measure 1.69 V, the bank direction switch is still in the "input" position
• to test individual output, instead of 0xF use 0x1, 0x2, 0x4 and 0x8.

test clock input

• we test the TTL input mode first
• disconnect CLK_IN input
• /home/agdaq/online/chronobox_software/test_cb.exe intclk ### switch to internal timestamp clock
• /home/agdaq/online/chronobox_software/test_cb.exe clocks

clock status: ext_clk: counter 0x00000000, freq 0.0 Hz, ts_clk: counter 0x00cccccc, freq 9999999.4 Hz, PLL status 0xa0000000

notice: ts_clk frequency is around 10 MHz (internal oscillator) and ext_clk frequency is zero (nothing connected)

• if ts_clk is not around 10 MHz, PLL is not locked correctly, reboot the FPGA and try again: /home/agdaq/online/chronobox_software/test_cb.exe reboot
• set CLK_IN jumpers to "TTL/NIM" and "TTL" position
• connect LEMO cable from LEMO output 0 (bottom left) to CLK_IN
• /home/agdaq/online/chronobox_software/test_cb.exe lemo_out_clk 0 ### output timestamp clock on LEMO output 0
• /home/agdaq/online/chronobox_software/test_cb.exe clocks

clock status: ext_clk: counter 0x00cccccc, freq 9999999.4 Hz, ts_clk: counter 0x00cccccc, freq 9999999.4 Hz, PLL status 0x80000000

• observe ext_clk frequency is the same as ts_clk frequency
• to test NIM input mode, we need a NIM clock signal, i.e. from a VME-NIMIO32, etc. Or use a TTL-to-NIM converter to use clock on LEMO output 0.
• set CLK_IN jumpers to "TTL/NIM" and "NIM" position
• connect VME-NIMIO32 NIM output 3 (40 MHz V1190 clock) to CLK_IN
• /home/agdaq/online/chronobox_software/test_cb.exe clocks

clock status: ext_clk: counter 0x03333c0c, freq 40001687.4 Hz, ts_clk: counter 0x00cccccc, freq 9999999.4 Hz, PLL status 0x80000000

• observe ext_clk frequency is 40 MHz.
• to test "sine wave" input mode, TBW, do not have a sine wave signal source.

K.O.

test GPIO pins

TBW

Install chronobox software

cd /home/agdaq/online
git clone https://bitbucket.org/expalpha/chronobox_software.git
cd chronobox_software
make clean
make
ls -l *.exe
-rwxr-xr-x 1 olchansk users 18808 Aug 16 15:26 reboot_cb.exe
-rwxr-xr-x 1 olchansk users 47256 Aug 16 15:26 srunner_cb.exe
-rwxr-xr-x 1 olchansk users 20732 Aug 16 15:26 test_cb.exe

Install chronobox quartus firmware project

cd /home/agdaq/online/firmware/git
git clone https://bitbucket.org/expalpha/chronobox_firmware.git
cd chronobox_firmware
git fetch ### update repository
git tag ### list available tags
git checkout 20210420_ko ### checkout a specific version
git status
cat timestamp.v ### confirm firmware version
ls -l output_files/*.{jic,sof,rpd}

Firmware revisions

• 0x5aceaed2 - April 2018 - all inputs connected to counters (except ext clock), 58 inputs, 50 MHz clock
• 0x5b6de806 - August 2018 - added fpga boot flash programmer and fpga reboot
• 0x5b7c827d - August 2018 - bshaw added debounce on GPIO inputs (for flow meters), 100 MHz clock
• 0x5b873169 - August 2018 - rebuilt, no changes
• 0x5b89e4b4 - August 2018 - added external clock signal, 59 inputs, 100 MHz clock
• 0x5b8de2b0 - September 2018 - added data fifo and timestamp counters (TSCs) for the first 4 inputs
• 0x5b906c56 - September 2018 - improved overflow markers, added scalers readout into the data fifo
• 0x5da8e4c2 - October 2019 - timestamps for all inputs, inversion of inputs
• 0x5db764a7 - October 2019 - FPGA reboot works now.
• 0x607f6709 - April 2021 - working timestamp synchronization, memory mapped registers, improved sdc file, correct clock transfers
• 0x60860e61 - April 2021 - faster data FIFO readout (from ~8 Mbytes/sec to ~80 Mbytes/sec)
• 0x61832695 - Nov 2021 - increase timestamp FIFO from 4k to 64k words
• 20220406_ko_0x624e22ad - April 2022 - make inputs edge-triggered instead of sampled.
• 20220420_ko_0x62608957 - April 2022 - fix first fifo data word is zero

Firmware update

• use srunner_cb.exe to load firmware RPD file into EPCQ flash memory.
• if FPGA is not running compatible firmware, srunner_cb will fail
• connect JTAG and load the firmware sof file from the quartus project (use "make load_sof")
• if JTAG is connected, one can use "make load_jic" to load firmware jic file into EPCQ flash memory.

After correct firmware is loaded, access to chronobox registers may fail (srunner_cb will not work) because FPGA bridges are not enabled. To check, run following commands. If some bridge reports "disabled", reboot linux (do not cycle the power!).

root@cb02:~# cat /sys/class/fpga_bridge/br*/state
enabled
enabled
enabled
enabled
root@cb02:~# 

Some DE10-Nano boards have an EPCQ64 flash memory chip, some have the EPCQ128 flash memory chip.

• identify EPCQ64 and use "-64" with all srunner commands:

cd /home/agdaq/online/chronobox_software
./srunner_cb.exe -id -64 /dev/null # identify EPCS64 flash
...
Info: Silicon ID - 0x16 
Info: Serial Configuration Device - EPCS64

• identify EPCQ128 and use "-128" with all srunner commands:

./srunner_cb.exe -id -128 /dev/null # identify EPCQ128 flash
...
Info: Silicon ID - 0x18 
Info: Serial Configuration Device - EPCS128

Identify -64 or -128 flash memory, write RPD file to flash memory, verify and reboot the fpga:

cd /home/agdaq/online/chronobox_software
./srunner_cb.exe -id -64 /dev/null # identify EPCS64 flash
./srunner_cb.exe -id -128 /dev/null # identify EPCQ128 flash
#./srunner_cb.exe -read -128 test.rpd # read flash contents into a file
./srunner_cb.exe -program -128 /home/agdaq/online/firmware/git/chronobox_firmware/output_files/DE10_NANO_SoC_GHRD_auto.rpd
./srunner_cb.exe -verify -128 /home/agdaq/online/firmware/git/chronobox_firmware/output_files/DE10_NANO_SoC_GHRD_auto.rpd
./reboot_cb.exe # reboot the fpga into the new firmware

Firmware build

for building firmware use: /opt/intelFPGA/20.1/nios2eds/nios2_command_shell.sh

ssh daq01
cd online
git clone git@bitbucket.org:expalpha/chronobox_firmware.git
cd chronobox_firmware
/opt/intelFPGA/20.1/nios2eds/nios2_command_shell.sh
make quartus
ls -l output_files/*.{sof,jic,rpd}
-rw-r--r-- 1 olchansk users 7007185 Apr 20 16:57 output_files/DE10_NANO_SoC_GHRD_auto.rpd
-rw-r--r-- 1 olchansk users 8388833 Apr 20 16:57 output_files/DE10_NANO_SoC_GHRD.jic
-rw-r--r-- 1 olchansk users 7510701 Apr 20 16:57 output_files/DE10_NANO_SoC_GHRD.sof

other make targets:

• make quartus - build fpga programmer files: sof, jic and rpd.
• make clean - clean the quartus project
• make qsys - regenerate the qsys block
• make jic - regenerate the jic file
• make load_sof - load sof file into fpga via jtag
• make load_jic - load jic file into fpga boot flash (epcq) via jtag
• make verify_jic - verify jic checksum in fpga boot flash (epcq)

for jtag access, use /daq/quartus/13.1.3.178/quartus/bin/jtagconfig

daq16:chronobox_firmware$ /daq/quartus/13.1.3.178/quartus/bin/jtagconfig
1) DE-SoC [2-1.4.3]
  4BA00477   SOCVHPS
  02D020DD   5CSEBA6(.|ES)/5CSEMA6/..

2) Remote server daq16: Unable to connect

daq16:chronobox_firmware$ 

Errata

• after regenerating qsys, manually edit ./soc_system/synthesis/submodules/hps_sdram_pll.sv to comment-out line "assign pll_dr_clk = clk_out[2];"

Firmware details

• DDR3 clock is 400 MHz (DDR3-800)
• memcpy speed ~1500 Mbytes/sec
• FIFO access single word read: 80, 44, 44 clocks at 10 ns/clock is 2.3 MHz at 4 bytes/word is 9 Mbytes/sec
• FIFO access memcpy: burst 2, delay 32, b4, delay 36, b5, d1, b5, d1, b5, d1, b1, delay 53, repeat. this reads 16 words in 72 clocks at 10 ns/clock is 1.388 MHz at 4*16=64 bytes/burst is 88.9 Mbytes/sec.

Chronobox firmware registers

reg | rw/ro | quartus name | firmware | description
0 | ro | sof_revision_in | all | firmware revision timestamp code
0 | wo | latch_scalers_out, zero_scalers_out | all | see [[#reg_0x00_write_bits]]
1 | rw | reg1_led_out | all | DE10-Nano LED output
2 | ro | switches_in | all | read DE10-Nano switches
3 | ro | buttons_in | all | read DE10-Nano buttons
4 | rw | reg4_test | all | 32-bit read-write test register
5 | rw | flash_programmer_in, reg5_flash_programmer_out | 0x5b6de806 | 0xABCD srunner flash programmer
6 | ro | ecl_in | all | read state of ECL inputs
7 | ro | reg7_test_in | all | ???
8 | rw | scaler_addr_out, reg8_scaler_data_in | all | top 16 bits of address becomes scaler bus address, 32 bit read is the corresponding scaler data
9 | ro | lemo_in | all | read state of LEMO inputs
10/0xA | ro | gpio_in | all | read state of GPIO inputs
11/0xB | rw | regB_lemo_out | all | LEMO output data
12/0xC | rw | regC_gpio_out | all | GPIO output data
13/0xD | rw | regD_out_enable_out | all | enable output tristates: [31:24] - LEMO_OUT, [17:0] - GPIO_OUT
14/0xE | rw | regE, reconfig_out | 0x5b6de806 | FPGA reboot: write inverted firmware revision (reg0) to reboot the FPGA
15/0xF | ro | regF_input_num_in | 0x5b89e4b4 | number of chronobox inputs (to read scalers, add 1 for the clock counter)
16/0x10 | ro | reg10_fifo_status | 0x5b8de2b0 | data fifo status, see below
17/0x11 | ro | reg11_fifo_data | 0x5b8de2b0 | data fifo data, see below
18/0x12 | rw | cb_invert_a | 0x5bf7557e | invert inputs 31..0
19/0x13 | rw | cb_invert_b | 0x5bf7557e | invert inputs 63..32
20/0x14 | rw | cb_enable_le_a | 0x5bf7557e | enable TSC leading edge inputs 31..0
21/0x15 | rw | cb_enable_le_b | 0x5bf7557e | enable TSC leading edge inputs 63..32
22/0x16 | rw | cb_enable_te_a | 0x5bf7557e | enable TSC trailing edge inputs 31..0
23/0x17 | rw | cb_enable_te_b | 0x5bf7557e | enable TSC trailing edge inputs 63..32
24/0x18 | ro | fc_ext_clk_100_counter | 0x5bf7557e | external clock frequency counter 100MHz reference
25/0x19 | ro | fc_ext_clk_ext_counter | 0x5bf7557e | external clock frequency counter
26/0x1A | ro | fc_ts_clk_100_counter | 0x5bf7557e | timestamp clock frequency counter 100MHz reference
27/0x1B | ro | fc_ts_clk_ts_counter | 0x5bf7557e | timestamp clock frequency counter
28/0x1C | ro | ts_clk_pll_status | 0x5bf7557e | timestamp clock PLL status
29/0x1D | rw | cb_lemo_out_mux_ctrl | 0x5bfdc798 | 8*4 bits to control 8 LEMO output multiplexers (4 bits/16 options each mux, see [[#LEMO outputs]])
30/0x1E | rw | cb_sync_mask[31:0] | 0x607f6709 | source of chronobox sync signal, low bits
31/0x1F | rw | cb_sync_mask[63:32] | 0x607f6709 | source of chronobox sync signal, high bits
32 | rw | cb_sync_reg[31:0], cb_sync_status[31:0] | 0x607f6709 | control of timestamp sync
33 | ro | tdc_fifo_status | DLTDC | TDC fifo status
34 | ro | tdc_fifo_data_lo | DLTDC | TDC fifo low 32-bits
35 | ro | tdc_fifo_data_hi | DLTDC | TDC fifo high 32-bits and fifo rdack
36 | rw | dl_input_mask | DLTDC | DL trigger input mask
37 | rw | dl_ctrl       | DLTDC | DL trigger control
38 | rw | tdc_input_mask | DLTDC | TDC input mask
39 | ro | axi_fifo_status | DLTDC | AXI fifo status

reg 0x00 write bits

bit | fw revision | quartus signal | description
0 | all | | latch scalers
1 | all | | zero scalers
2 | ... | fifo_rdreq_out | fifo_rdreq_out (not used in version 0x607f6709 and later)
3 | ... | ts_clk_pll_extswitch_out | clear ts_clk_pll_extswitch_out
4 | ... | ts_clk_pll_extswitch_out | set ts_clk_pll_extswitch_out

reg 16 fifo status

Data fifo status bits:

31: fifo_full - data fifo is current full
30: fifo_empty - data fifo is currently empty
29: cb_fifo_ch_full - TSC per-channel fifos have overflown at some point
28: fifo_full_latch - data fifo has overflown at some point
24+4: 0
0+24: fifo_usedw

reg 28

Timestamp clock PLL status bits:

31 : PLL locked
30 : PLL active clock (0=internal, 1=external
29 : external clock bad
28 : internal clock bad
27 : ts_clk_pll_extswitch
0..26 : not used

reg 32 timestamp sync

Timestamp synchronization register bits:

31 : write 1 to arm timestamp sync circuit, scalers and TSCs are placed in reset state
30 : write 1 to disarm timestamp sync circuit, reset is released
...
16 : write 1 to send a timestamp sync signal
15 : sync circuit armed
14 : sync done
13 : sync signal received (cleared by writing bits 31 or 30)
12 : 0
11 : sync circuit armed, clk_ts section
10 : sync done, clk_ts section
9  : sync signal received, clk_ts section
8  : 0
7  : unused
...
0  : unused

nominal sequence for synchronizing timestamps:

• make sure nothing is sending the external sync signal (i.e. run is stopped, etc)
• arm the sync circuit: cb_write32(32, 0x80000000); cb_write32(32, 0);
• check that the arming was successful: cb_read32(32), only bits 0x8800 should be set
• scalers are in reset state, not incrementing
• TSCs are in reset state, no data in the TSC FIFO
• some time later, send the external sync signal
• or send the internal sync signal: cb_write32(32, 0x10000); cb_write32(32, 0);
• check that the timestamp reset happened: cb_read32(32), bits 0x6600 should be set, bits 0x8800 should be cleared
• check that the scalers are counting, check that there is TSC data in the TSC FIFO

reg 33, 39 TDC and AXI FIFO status

 bit |   mask     | description
  31 | 0x80000000 | fifo empty
  30 | 0x40000000 | fifo full
15:0 | 0x0000FFFF | fifo usedw (64-bit words)

Timestamp clock

Timestamp clock is 10 MHz selectable from internal oscillator or external reference on the CLK_IN input.

CLK_IN input can be selected using 2 two-position jumpers:

• NIM input: CLK<->NIM/TTL and CLK<->NIM

To select the clock from command line, use:

• test_cb.exe intclk # select internal clock
• test_cb.exe extclk # select external clock

From software:

Chronobox* cb = ...;
cb->cb_int_clock(); # select internal clock
cb->cb_ext_clock(); # select external clock

To see current status, run "test_cb.exe clocks":

# ./test_cb.exe clocks
...
Chronobox firmware revision: 0x5bf7557e
...
clock status: ext_clk: counter 0x00cccf2a, freq 10000450.9 Hz, ts_clk: counter 0x00cccf2a, freq 10000450.9 Hz, PLL status 0xc0000000

Reported is external clock frequency, currently selected timestamp clock frequency and PLL status (register 0x1C).

Normal values for the PLL status:

• internal clock: 0x80000000
• external clock: 0xC0000000
• external clock selected, but invalid: 0x60000000
• internal clock selected, external clock invalid: 0xa0000000

Disconnected/absent/broken external clock will report ext_clk frequency zero, bit 0x20000000 in the PLL status register 0x1C.

If external clock is CDM, set chronobox clock jumpers to "CLK NIM/TTL" and "NIM", use CDM 10 MHz clock output LEMO3B.

If external clock is another chronobox (usually LEMO output 0), set clock jumpers to "CLK NIM/TTL" and "TTL".

LEMO outputs

LEMO connectors can be used as TTL level outputs:

• set the "bank a" and/or "bank b" jumpers for "output"
• set the lemo output multiplexor bits in register 0x1D
• set the "lemo output enable" bit in register 0x0D, use bits 24..31 for LEMO outputs 0..7.
• observe the corresponding LED is on or off according to the LEMO output TTL logic level

The function of each LEMO output is controlled by the lemo output multiplexor. Up to 16 different signals can be routed into each output. This is controlled by register 0x1D.

Register 0x1D multiplexor control is 32 bits organized into 8 groups of 4 bits, per each of the 8 LEMO outputs: 0x76543210, i.e. value 0x00000001 routes signal function 1 into output 0, and signal function 0 into outputs 1..7.

For each output, there are 16 possible signal functions (4 bits):

bit | firmware signal | firmware revision | signal description
0 | cb_lemo_out_reg[n] | 0x5bfdc798 | output is controlled by register 0xB bits 0..7 for outputs 0..7
1 | clk_ts | 0x5bfdc798 | output is the 10MHz timestamp clock
2 | cb_sync_in_clk_ts | 0x607f6709 | daisy-chained timestamp sync signal (from control register or from sync input, see registers 30 and 31. see bit 0x10000 in register 32)
3..30 | gnd | 0x5bfdc798 | not used
31 | vcc | 0x5bfdc798 | logic level 1

front panel LEDs

The front panel LEDs are numbered 0..7 per #Chronobox_connectors

Each LED can be individually lit by setting a bit in register 0x1 bits 8..15.

If LEMO outputs are active, corresponding LEDs will show LEMO output status (logic level 1 -> LED is on, logic level 0 -> LED is off).

If LEMO inputs are active ...

Synchronization of multiple chronoboxes

To operate several chronoboxes as one unit, they must have two common signals, a clock and a sync signal.

Each chronobox must be configured as:

• master: this will issue the sync signal
• slave: will receive the clock and sync signals from master
• slave daisy-chain: and pass them to the next slave down the chain
• (one can use the CDM or the TRG as sync masters)

For the purpose of this example (recommended configuration), set the LEMO direction jumpers:

• bank A: set to "out": master and slave daisy-chain clock and sync output
• bank B: set to "in": slave sync input

Clock connections:

• sync master can be configured with external or internal clock (CLK_IN input)
• sync slave must be configured with external clock (CLK_IN input)

sync master LEMO #0 -> slave CLK_IN, LEMO #0 -> next slave CLK_IN, LEMO #0 -> ...
10 MHz clock from CDM & etc -> sync master CLK_IN, LEMO #0 -> slave CLK_IN, ...

Sync connections:

• sync signal can be generated externally by the CDM or TRG
• sync signal can be generated internally by writing 0x10000 into reg 32 of sync master
• (sync must be armed on the master and all slaves before sending the first sync signal, write 0x80000000 into reg 32)

sync master LEMO #1 -> slave LEMO #4, LEMO #1 -> next slave LEMO #4 -> ...
external sync -> master LEMO #4, LEMO #1 -> slave LEMO #4 -> ...

Sync configuration: (see register 32 for explanation!)

• master and slave: disarm sync: write 0x40000000 to reg 32, write 0 to reg 32, bits 0x8800 and 0x6600 should clear
• master and slave daisy chain: setup clock and sync outputs:
  • register 29 lemo output mux to 0xXXXX'XX21 (2=output sync signal, 1=output clock)
  • register 13 lemo output enable, set bits 0x0F00'0000 (set bank A jumper for "out")
• slave and slave daisy chain: enable external sync from LEMO #4:
  • register 30, write 0
  • register 31, write 0x10
• master: switch to external or internal clock, as needed
• slave: switch to external clock
• master and slave: arm sync: write 0x80000000 to reg 32, write 0 to reg 32, bits 0x8800 should turn on.
• timestamps, scalers and data fifo go into the reset state, all old data is cleared
• when sync is received, timestamp is reset to zero, reset state is cleared and there should be data into FIFO. reg 32 will read 0x6600.
• if using external sync (no master), wait for reg 32 bits to become 0x6600 (sync completed)
• master: issue sync signal, write 0x10000 to reg 32, write 0 to reg 32. master reg 32 should read 0x6600, slave reg 32 should read 0x6600.

FIFO data format

• 0x8ntttttt: TSC data, 24 bits "tttttt" of timestamp, 7 bits "nn" of channel number, top bit set to 1. Low bit of "t" indicates 0=leading edge, 1=trailing edge.
• 0xffTTmmmm: (before 20220420_ko_0x62608957) timestamp wrap around marker: "TT" is the top 8 bits of the timestamp, "mmmm" is a 16 bit counter
• 0xff8mmmmm: timestamp wrap around marker: bit 24 is the top bit of the timestamp (0/8), "mmmmm" is a 23 bit counter
• 0xfe00nnnn: scaler data, following "nnnn" words are the latched scalers
• 0xfd000000: start of data after chronobox sync (cbtrg only, for now)

Time range is (before 20220420_ko_0x62608957):

• 24 bits of "tttttt"
• 15 bits of "mmmm"
• computation:

(gdb) p (100.0*1.0*0x00007FFF*1.0*0x00FFFFFF)/(1e9)/3600.0
$22 = 15.270527886249999 ### in hours

Time range is:

• 24 bits of "tttttt"
• 22 bits of "mmmmm"
• computation:

(gdb) p (100.0*1.0*0x003FFFFF*1.0*0x00FFFFFF)/(1e9)/3600.0
$2 = 1954.6867557262499 ### in hours
$3 = 81.445281488593736 ### in days

timestamp wrap around marker

The timestamp data is only 24 bits, to allow timestamping with longer time range, wrap around markers are added to the data stream.

For input signals that arrive close to the time of timestamp wrap around, there is ambiguity in the ordering of the data fifo: does the wrap around marker or the signal timestamp show up first? For example for rare signals, one can see this:

wrap 1
wrap 2
timestamp 0x00000003
wrap 3
wrap 4

does the hit belong with wrap marker 2 (written to the fifo just after wrap marker 2) or with marker 3 (written to the fifo just before wrap marker 3)?

To remove this ambiguity, additional markers are written to the data stream half way between the wrap arounds, making it obvious that the signal arrived right after wrap marker 3 (but was written to the FIFO before the marker):

wrap 1 0x00
wrap 1 0x80
wrap 2 0x00
wrap 2 0x80
timestamp 0x00000003
wrap 3 0x00
wrap 3 0x80

test_cb.exe

test_cb.exe is the general test program for the chronobox.

• test_cb.exe 0 # read chronobox register 0
• test_cb.exe 4 0x1234 # write to chronobox register 4
• test_cb.exe fpga 0 # read FPGA LW bridge address 0
• test_cb.exe fpga 0 0x12345678 # write 0x12345678 to FPGA LW bridge address 0
• test_cb.exe reboot # reboot the FPGA (the ARM CPU keeps running)
• test_cb.exe scalers # read all scalers in a loop
• test_cb.exe fifo # read the data fifo in a loop
• test_cb.exe tsfifo # read the timestamp fifo in a loop (do not latch scalers)
• test_cb.exe intclk # select internal timestamp clock
• test_cb.exe extclk # select external timestamp clock (10MHz)
• test_cb.exe clocks # report current status of timestamp clock

Errata

• 0x607f6709 - after FIFO reset (timestamp sync, etc), the first word in the FIFO is always 0x00000000, not a real timestamps.
• 20220420_ko_0x62608957 - after FIFO reset (timestamp sync), the first word in the FIFO is 0xFF000000, a timestamp overflow marker, as expected. after FPGA reboot (before timestamp sync), the first word in the FIFO is also 0xFF000000 for reasons unknown.
• linux - after power up, chronobox registers are not accessible, check that the FPGA bridges are enabled, if any bridge reports "disabled", reboot linux. do NOT cycle power.

root@cb04:~# cat /sys/class/fpga_bridge/br*/state
enabled
enabled
enabled
enabled
root@cb04:~# 

TODO

• FIXED 23apr2021 - use updated "bootcmd". figure out why FPGA bridges are disabled on first power up
• figure out how to load FPGA sof from Linux, probably need new kernel
• FIXED apr2022, use fw_printenv build from source - fix fw_printenv, probably requires using newer uboot
• add mmap driver for FPGA LW bridge, probably need new kernel
• figure out how to build linux kernel from git (now using kernel from GHRD demo kit)
• enable watchdog timer to auto-reboot on hang
• figure out why ssh takes 30 second to start. read this: https://daniel-lange.com/archives/152-Openssh-taking-minutes-to-become-available,-booting-takes-half-an-hour-...-because-your-server-waits-for-a-few-bytes-of-randomness.html https://lwn.net/Articles/800509/
• FIXED 20220420_ko_0x62608957 - fix problem on startup first word in the FIFO is 0x00000000.
• FIXED 20220420_ko_0x62608957 - increase number of bits in the timestamp wraparound counter: 0xFF8mmmmm -> 23 bits is possible, keep high bit 0x00800000 as is.