Chronobox

Links

Chronobox connectors

|
| LEMO - CLK_IN - jumpers: SINE<->CLK<->NIM/TTL and NIM<->CLK<->TTL
|
| ECL P1
| 32
| |
| 1
|
| ECL P2
| 32
| |
| 1
|
| PLED - power-on LED
|
| jumper: bank A direction IN<->X<->OUT
| LEMO 0-1
| LED  0-1
| LEMO 2-3
| LED  2-3
|
| jumper: bank B direction IN<->X<->OUT
| LEMO 4-5
| LED  4-5
| LEMO 6-7
| LED  6-7
|

Electrical connections

ECL P1 and P2: LeCroy 4616 compatible
LEMO TTL inputs: 0..5V, logic level 0: <0.8V, logic level 1: >2V
LEMO TTL outputs: logic level 0: 0V, logic level 1: 5V to 3.3V
clock NIM input: NIM compatible
clock TTL input: ???
clock sine wave input: ???

Input channel mapping

0+16 : first ECL connector
16+16 : second ECL connector
32+8 : LEMO inputs (TTL)
40+18 : GPIO inputs (FPGA pins)
58 : external clock (10 MHz nominal)
59 : internal clock (100 MHz)

Functional units

Scalers

TBW

Firmware 0x61832695

minimum pulse width (leading edge to trailing edge): longer than 10 ns (15 ns is safe) (sampled by 100 MHz/ 10 ns clock)
minimum time between pulses (trailing edge to next leading edge): longer than 10 ns (15 ns is safe) (ditto)

Firmware 0x624e22ad

minimum pulse width: leading edge triggered (2 ns is safe)
minimum time between pulses (trailing edge to next leading edge): 25 ns or more is safe

Timestamps (TSC)

TBW

Firmware 0x61832695

minimum pulse width (leading edge to trailing edge): longer than 100 ns (105 ns is safe) (sampled by 10 MHz/100 ns clock)
minimum time between pulses (trailing edge to next leading edge): longer than 100 ns (105 ns is safe) (ditto)

Firmware 0x624e22ad

minimum pulse width: leading edge triggered (5 ns is safe)
minimum time between pulses (trailing edge to next leading edge): longer than 200 ns (205 ns is safe) (sampled by 10 MHz/100 ns clock)

Each TSC input has a 256 entry FIFO. (LE is 1 entry, TE is 1 entry, so 128 hits if both LE and TE enabled).

Output data fifo

TBW

Firmware before 0x61832695

Output fifo is 4096 entries deep.

Firmware 0x624e22ad

Output fifo is 64k entries deep.

Initial hardware setup

refer to DE10-NANO information here: https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano
prepare hardware:
- remove chronobox from enclosure (cannot access JTAG connector when installed)
- remove DE10-Nano from chronobox baseboard
- check that SW10 jumpers are: U-D-U-U-D-U. (U=up, D=down, when "ALTERA" text on the FPGA is right side up)
- reinstall DE10-Nano on chronobox baseboard
- do NOT reinstall chronobox in enclosure
prepare software:
- check that you have quartus 17.1 (at /opt/intelFPGA/17.1/ in this example)
- check that you installed chronobox software in /home/agdaq/online/chronobox_software
- check that you installed chronobox firmware project in /home/agdaq/online/firmware/git/chronobox_firmware
connect serial console, for more info, go here: https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano#Serial_console
- connect a mini-USB (*not* Micto-USB!) cable from mini-USB port on the side of the ethernet connection to a PC
- minicom -D /dev/ttyUSB0 -b 115200
connect fpga jtag, for more info, go here: https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano#JTAG
- connect a mini-USB (*not* Micro-USB!) cable on the side of HDMI and power connectors to a PC
- /opt/intelFPGA/17.1/quartus/bin/jtagconfig
prepare SD flash card for booting from network: generic instructions are here: https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano#boot_Linux_from_SD_flash
- connect new or used 8GB SD flash card to USB flash adapter to a linux computer
- login as root
- identify the flash device as /dev/sdX. Use "lsblk" or "blkid" or "dmesg". be careful to not write to the computer system disk (usually /dev/sda)
- echo -e 'o\nn\np\n1\n2048\n4095\nn\np\n2\n4096\n1681816\nn\np\n3\n\n\nt\n1\na2\nt\n2\nb\nw' | fdisk /dev/sdX
- eject /dev/sdX
- remove card, reintall card, this is to ensure we use the new partition table
- fdisk -l /dev/sdX

root@daq01:~# fdisk -l /dev/sdg
Disk /dev/sdg: 7.38 GiB, 7910457344 bytes, 15450112 sectors
Disk model: STORAGE DEVICE  
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xa033868b

Device     Boot   Start      End  Sectors   Size Id Type
/dev/sdg1          2048     4095     2048     1M a2 unknown
/dev/sdg2          4096  1681816  1677721 819.2M  b W95 FAT32
/dev/sdg3       1683456 15450111 13766656   6.6G 83 Linux
root@daq01:~#

- dd if=/daq/daqstore/olchansk/daq/DE10-Nano/image-1-fpga.img of=/dev/sdX1 bs=1024k
- dd if=/daq/daqstore/olchansk/daq/DE10-Nano/image-2-uboot.img of=/dev/sdX2 bs=1024k
- eject /dev/sdX
install prepared flash card into the chronobox DE10-Nano
cycle power
in the minicom window, IMMEDIATELY press the space bar to stop automatic booting (if it doesn't react, ensure flow control is disabled in minicom)
you will have the uboot "=>" prompt
confirm uboot version: U-Boot 2013.01.01
setup uboot to boot linux from network, more details here: https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano#boot_Linux_from_network
- setenv ethaddr 02:aa:bb:cc:cb:04 # for cb04, see below for explanation
- setenv bootcmd 'run bridge_enable_handoff; run netboot'
- setenv ramboot bootz \${loadaddr} - \${fdtaddr}
- setenv bootargs console=ttyS0,115200 ip=dhcp root=/dev/nfs rw nfsroot=192.168.1.1:/zssd1tb/nfsroot/%s,vers=3 panic=15
- saveenv
- reset
DE10-Nano should boot into linux:
- uboot will restart
- DHCP will run
- tftp load of linux kernel will run
- linux kernel will start
- linux kernel will get an IP address
- linux kernel will NFS-mount the root filesystem (.../nfsroot/cb04)
- systemd will start all services
- there will be a login prompt on minicom console
- ssh will work (maybe 1 minute delay before it starts)
try a few things:
- from agdaq or agmini account, ssh root@cb04
- "df" should show /home/agdaq is mounted (and ssh agdaq@cb04 should work)
- "/home/agdaq/online/chronobox_software/test_cb.exe 0" should fail ("bus error") because FPGA is not loaded yet
load sof file, for more info, go here: https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano#JTAG
- /opt/intelFPGA/17.1/quartus/bin/quartus_pgm -m JTAG -o "p;/home/agdaq/online/firmware/git/chronobox_firmware/output_files/DE10_NANO_SoC_GHRD.sof@2"
"test_cb.exe 0" should work now
use srunner_cb.exe to load the pof file into FPGA boot flash
use reboot_cb.exe to reboot the FPGA from flash
"test_cb.exe 0" should report the expected FPGA firmware version number.
success.

assign ethernet MAC address

Usually ethernet MAC address is assigned by manufacturer and is stored in a tiny little flash chip. The DE10-Nano board is too cheap and does not have it. So a fake ethernet address has to be assigned manually from the "locally administered range" of MAC addresses that start from "02:...", see https://en.wikipedia.org/wiki/MAC_address#Ranges_of_group_and_locally_administered_addresses

All MAC addresses must be unique on an ethernet network, but with manual assignement confusion is easy to create.

For chronobox devices, use MAC addresses that start with "l02:aa:bb:cc:xx:yy" and have last digits "cb:01", "cb:02", "cb:03", "cb04" for the first four chronoboxes.

new board test checklist

to check newly built chronobox baseboard, follow this checklist.

test ECL inputs

stop midas frontend
ssh agdaq@alphagdaq ### or agmini@daq16
ssh root@cb03
/home/agdaq/online/chronobox_software/test_cb.exe 0 ### read firmware revision, confirm communication with chronobox firmware
/home/agdaq/online/chronobox_software/test_cb.exe inputs ### output should look like this:

inputs: ecl_p1: 0xaa55, ecl_p2: 0xaa55, lemo: 0xe0, gpio: 0x3ffff

connect ECL ribbon cable from LeCroy 4616 NIM-to-ECL converter to chronobox ECL_P1 input (left one)
ecl_p1 input should read 0x0000
connect lemo jumper from NIM logic level 1 otput (i.e. any module OUT-bar output) to first channel of LeCroy 4616, ecl_p1 should read 0x0001
move to 2nd channel, get 0x0002
and so forth until last channel 0x8000.
if any channel does not give expected reading, attach label (i.e. "ECL_P1 0x0002 bad") and send for repair.
repeat with ECL_P2 input (right one), watch printed value of ecl_p2.

test TTL inputs

NOTE: front panel LEMO connector labels are wrong. correct numbering of LEMO connectors is as shown in #Chronobox connectors
disconnect all LEMO inputs
set "bank A direction" and "bank B direction" for input
/home/agdaq/online/chronobox_software/test_cb.exe lemo_in_a # set bank A to input
/home/agdaq/online/chronobox_software/test_cb.exe lemo_in_b # set bank B to input
all 8 LEMO LEDs should be on
/home/agdaq/online/chronobox_software/test_cb.exe inputs ### lemo inputs should read 0xFF
connect LEMO 0 (left bottom) to TTL output of Lecroy 222 or equivalent, set "scale" to "latch", operate push buttons to turn output on and off (LED should come in and out).
/home/agdaq/online/chronobox_software/test_cb.exe inputs

inputs: ecl_p1: 0xaa55, ecl_p2: 0xaa55, lemo: 0xfe (~0x01), gpio: 0x3ffff

change TTL signal from logic level 0 to 1 and back, observe the corresponding LED to turn on and off, observe the correct bit in the "lemo" column (watch the inverted input ~0x01) change.
repeat with LEMO input 1 (left top) through 7 (right top).

test TTL outputs

NOTE: front panel LEMO connector labels are wrong. correct numbering of LEMO connectors is as shown in #Chronobox connectors
set "bank A direction" and "bank B direction" for output
/home/agdaq/online/chronobox_software/test_cb.exe lemo_out_a 0
/home/agdaq/online/chronobox_software/test_cb.exe lemo_out_b 0
all LEMO LEDs should be off
with voltmeter measure LEMO output voltage should be 0.135 V
/home/agdaq/online/chronobox_software/test_cb.exe lemo_out_a 0xF
/home/agdaq/online/chronobox_software/test_cb.exe lemo_out_b 0xF
all LEMO LEDs should be on
with voltmeter measure LEMO output voltage should be 3.4 to 3.8 V
if you measure 1.69 V, the bank direction switch is still in the "input" position
to test individual output, instead of 0xF use 0x1, 0x2, 0x4 and 0x8.

test clock input

we test the TTL input mode first
disconnect CLK_IN input
/home/agdaq/online/chronobox_software/test_cb.exe intclk ### switch to internal timestamp clock
/home/agdaq/online/chronobox_software/test_cb.exe clocks

clock status: ext_clk: counter 0x00000000, freq 0.0 Hz, ts_clk: counter 0x00cccccc, freq 9999999.4 Hz, PLL status 0xa0000000

notice: ts_clk frequency is around 10 MHz (internal oscillator) and ext_clk frequency is zero (nothing connected)

if ts_clk is not around 10 MHz, PLL is not locked correctly, reboot the FPGA and try again: /home/agdaq/online/chronobox_software/test_cb.exe reboot
set CLK_IN jumpers to "TTL/NIM" and "TTL" position
connect LEMO cable from LEMO output 0 (bottom left) to CLK_IN
/home/agdaq/online/chronobox_software/test_cb.exe lemo_out_clk 0 ### output timestamp clock on LEMO output 0
/home/agdaq/online/chronobox_software/test_cb.exe clocks

clock status: ext_clk: counter 0x00cccccc, freq 9999999.4 Hz, ts_clk: counter 0x00cccccc, freq 9999999.4 Hz, PLL status 0x80000000

observe ext_clk frequency is the same as ts_clk frequency
to test NIM input mode, we need a NIM clock signal, i.e. from a VME-NIMIO32, etc. Or use a TTL-to-NIM converter to use clock on LEMO output 0.
set CLK_IN jumpers to "TTL/NIM" and "NIM" position
connect VME-NIMIO32 NIM output 3 (40 MHz V1190 clock) to CLK_IN
/home/agdaq/online/chronobox_software/test_cb.exe clocks

clock status: ext_clk: counter 0x03333c0c, freq 40001687.4 Hz, ts_clk: counter 0x00cccccc, freq 9999999.4 Hz, PLL status 0x80000000

observe ext_clk frequency is 40 MHz.
to test "sine wave" input mode, TBW, do not have a sine wave signal source.

K.O.

test GPIO pins

TBW

Install chronobox software

cd /home/agdaq/online
git clone https://bitbucket.org/expalpha/chronobox_software.git
cd chronobox_software
make clean
make
ls -l *.exe
-rwxr-xr-x 1 olchansk users 18808 Aug 16 15:26 reboot_cb.exe
-rwxr-xr-x 1 olchansk users 47256 Aug 16 15:26 srunner_cb.exe
-rwxr-xr-x 1 olchansk users 20732 Aug 16 15:26 test_cb.exe

Install chronobox quartus firmware project

cd /home/agdaq/online/firmware/git
git clone https://bitbucket.org/expalpha/chronobox_firmware.git
cd chronobox_firmware
git fetch ### update repository
git tag ### list available tags
git checkout 20210420_ko ### checkout a specific version
git status
cat timestamp.v ### confirm firmware version
ls -l output_files/*.{jic,sof,rpd}

Firmware revisions

0x5aceaed2 - April 2018 - all inputs connected to counters (except ext clock), 58 inputs, 50 MHz clock
0x5b6de806 - August 2018 - added fpga boot flash programmer and fpga reboot
0x5b7c827d - August 2018 - bshaw added debounce on GPIO inputs (for flow meters), 100 MHz clock
0x5b873169 - August 2018 - rebuilt, no changes
0x5b89e4b4 - August 2018 - added external clock signal, 59 inputs, 100 MHz clock
0x5b8de2b0 - September 2018 - added data fifo and timestamp counters (TSCs) for the first 4 inputs
0x5b906c56 - September 2018 - improved overflow markers, added scalers readout into the data fifo
0x5da8e4c2 - October 2019 - timestamps for all inputs, inversion of inputs
0x5db764a7 - October 2019 - FPGA reboot works now.
0x607f6709 - April 2021 - working timestamp synchronization, memory mapped registers, improved sdc file, correct clock transfers
0x60860e61 - April 2021 - faster data FIFO readout (from ~8 Mbytes/sec to ~80 Mbytes/sec)
0x61832695 - Nov 2021 - increase timestamp FIFO from 4k to 64k words
20220406_ko_0x624e22ad - April 2022 - make inputs edge-triggered instead of sampled.
20220420_ko_0x62608957 - April 2022 - fix first fifo data word is zero

Firmware update

use srunner_cb.exe to load firmware RPD file into EPCQ flash memory.
if FPGA is not running compatible firmware, srunner_cb will fail
connect JTAG and load the firmware sof file from the quartus project (use "make load_sof")
if JTAG is connected, one can use "make load_jic" to load firmware jic file into EPCQ flash memory.

After correct firmware is loaded, access to chronobox registers may fail (srunner_cb will not work) because FPGA bridges are not enabled. To check, run following commands. If some bridge reports "disabled", reboot linux (do not cycle the power!).

root@cb02:~# cat /sys/class/fpga_bridge/br*/state
enabled
enabled
enabled
enabled
root@cb02:~#

Some DE10-Nano boards have an EPCQ64 flash memory chip, some have the EPCQ128 flash memory chip.

identify EPCQ64 and use "-64" with all srunner commands:

cd /home/agdaq/online/chronobox_software
./srunner_cb.exe -id -64 /dev/null # identify EPCS64 flash
...
Info: Silicon ID - 0x16 
Info: Serial Configuration Device - EPCS64

identify EPCQ128 and use "-128" with all srunner commands:

./srunner_cb.exe -id -128 /dev/null # identify EPCQ128 flash
...
Info: Silicon ID - 0x18 
Info: Serial Configuration Device - EPCS128

Identify -64 or -128 flash memory, write RPD file to flash memory, verify and reboot the fpga:

cd /home/agdaq/online/chronobox_software
./srunner_cb.exe -id -64 /dev/null # identify EPCS64 flash
./srunner_cb.exe -id -128 /dev/null # identify EPCQ128 flash
#./srunner_cb.exe -read -128 test.rpd # read flash contents into a file
./srunner_cb.exe -program -128 /home/agdaq/online/firmware/git/chronobox_firmware/output_files/DE10_NANO_SoC_GHRD_auto.rpd
./srunner_cb.exe -verify -128 /home/agdaq/online/firmware/git/chronobox_firmware/output_files/DE10_NANO_SoC_GHRD_auto.rpd
./reboot_cb.exe # reboot the fpga into the new firmware

Firmware build

for building firmware use: /opt/intelFPGA/20.1/nios2eds/nios2_command_shell.sh

ssh daq01
cd online
git clone git@bitbucket.org:expalpha/chronobox_firmware.git
cd chronobox_firmware
/opt/intelFPGA/20.1/nios2eds/nios2_command_shell.sh
make quartus
ls -l output_files/*.{sof,jic,rpd}
-rw-r--r-- 1 olchansk users 7007185 Apr 20 16:57 output_files/DE10_NANO_SoC_GHRD_auto.rpd
-rw-r--r-- 1 olchansk users 8388833 Apr 20 16:57 output_files/DE10_NANO_SoC_GHRD.jic
-rw-r--r-- 1 olchansk users 7510701 Apr 20 16:57 output_files/DE10_NANO_SoC_GHRD.sof

other make targets:

make quartus - build fpga programmer files: sof, jic and rpd.
make clean - clean the quartus project
make qsys - regenerate the qsys block
make jic - regenerate the jic file
make load_sof - load sof file into fpga via jtag
make load_jic - load jic file into fpga boot flash (epcq) via jtag
make verify_jic - verify jic checksum in fpga boot flash (epcq)

for jtag access, use /daq/quartus/13.1.3.178/quartus/bin/jtagconfig

daq16:chronobox_firmware$ /daq/quartus/13.1.3.178/quartus/bin/jtagconfig
1) DE-SoC [2-1.4.3]
  4BA00477   SOCVHPS
  02D020DD   5CSEBA6(.|ES)/5CSEMA6/..

2) Remote server daq16: Unable to connect

daq16:chronobox_firmware$

Errata

after regenerating qsys, manually edit ./soc_system/synthesis/submodules/hps_sdram_pll.sv to comment-out line "assign pll_dr_clk = clk_out[2];"

Firmware details

DDR3 clock is 400 MHz (DDR3-800)
memcpy speed ~1500 Mbytes/sec
FIFO access single word read: 80, 44, 44 clocks at 10 ns/clock is 2.3 MHz at 4 bytes/word is 9 Mbytes/sec
FIFO access memcpy: burst 2, delay 32, b4, delay 36, b5, d1, b5, d1, b5, d1, b1, delay 53, repeat. this reads 16 words in 72 clocks at 10 ns/clock is 1.388 MHz at 4*16=64 bytes/burst is 88.9 Mbytes/sec.

Chronobox firmware registers

reg | rw/ro | quartus name | firmware | description
0 | ro | sof_revision_in | all | firmware revision timestamp code
0 | wo | latch_scalers_out, zero_scalers_out | all | see [[#reg_0x00_write_bits]]
1 | rw | reg1_led_out | all | DE10-Nano LED output
2 | ro | switches_in | all | read DE10-Nano switches
3 | ro | buttons_in | all | read DE10-Nano buttons
4 | rw | reg4_test | all | 32-bit read-write test register
5 | rw | flash_programmer_in, reg5_flash_programmer_out | 0x5b6de806 | 0xABCD srunner flash programmer
6 | ro | ecl_in | all | read state of ECL inputs
7 | ro | reg7_test_in | all | ???
8 | rw | scaler_addr_out, reg8_scaler_data_in | all | top 16 bits of address becomes scaler bus address, 32 bit read is the corresponding scaler data
9 | ro | lemo_in | all | read state of LEMO inputs
10/0xA | ro | gpio_in | all | read state of GPIO inputs
11/0xB | rw | regB_lemo_out | all | LEMO output data
12/0xC | rw | regC_gpio_out | all | GPIO output data
13/0xD | rw | regD_out_enable_out | all | enable output tristates: [31:24] - LEMO_OUT, [17:0] - GPIO_OUT
14/0xE | rw | regE, reconfig_out | 0x5b6de806 | FPGA reboot: write inverted firmware revision (reg0) to reboot the FPGA
15/0xF | ro | regF_input_num_in | 0x5b89e4b4 | number of chronobox inputs (to read scalers, add 1 for the clock counter)
16/0x10 | ro | reg10_fifo_status | 0x5b8de2b0 | data fifo status, see below
17/0x11 | ro | reg11_fifo_data | 0x5b8de2b0 | data fifo data, see below
18/0x12 | rw | cb_invert_a | 0x5bf7557e | invert inputs 31..0
19/0x13 | rw | cb_invert_b | 0x5bf7557e | invert inputs 63..32
20/0x14 | rw | cb_enable_le_a | 0x5bf7557e | enable TSC leading edge inputs 31..0
21/0x15 | rw | cb_enable_le_b | 0x5bf7557e | enable TSC leading edge inputs 63..32
22/0x16 | rw | cb_enable_te_a | 0x5bf7557e | enable TSC trailing edge inputs 31..0
23/0x17 | rw | cb_enable_te_b | 0x5bf7557e | enable TSC trailing edge inputs 63..32
24/0x18 | ro | fc_ext_clk_100_counter | 0x5bf7557e | external clock frequency counter 100MHz reference
25/0x19 | ro | fc_ext_clk_ext_counter | 0x5bf7557e | external clock frequency counter
26/0x1A | ro | fc_ts_clk_100_counter | 0x5bf7557e | timestamp clock frequency counter 100MHz reference
27/0x1B | ro | fc_ts_clk_ts_counter | 0x5bf7557e | timestamp clock frequency counter
28/0x1C | ro | ts_clk_pll_status | 0x5bf7557e | timestamp clock PLL status
29/0x1D | rw | cb_lemo_out_mux_ctrl | 0x5bfdc798 | 8*4 bits to control 8 LEMO output multiplexers (4 bits/16 options each mux, see [[#LEMO outputs]])
30/0x1E | rw | cb_sync_mask[31:0] | 0x607f6709 | source of chronobox sync signal, low bits
31/0x1F | rw | cb_sync_mask[63:32] | 0x607f6709 | source of chronobox sync signal, high bits
32 | rw | cb_sync_reg[31:0], cb_sync_status[31:0] | 0x607f6709 | control of timestamp sync
33 | ro | tdc_fifo_status | DLTDC | TDC fifo status
34 | ro | tdc_fifo_data_lo | DLTDC | TDC fifo low 32-bits
35 | ro | tdc_fifo_data_hi | DLTDC | TDC fifo high 32-bits and fifo rdack
36 | rw | dl_input_mask | DLTDC | DL trigger input mask
37 | rw | dl_ctrl       | DLTDC | DL trigger control
38 | rw | tdc_input_mask | DLTDC | TDC input mask
39 | ro | axi_fifo_status | DLTDC | AXI fifo status

reg 0x00 write bits

bit | fw revision | quartus signal | description
0 | all | | latch scalers
1 | all | | zero scalers
2 | ... | fifo_rdreq_out | fifo_rdreq_out (not used in version 0x607f6709 and later)
3 | ... | ts_clk_pll_extswitch_out | clear ts_clk_pll_extswitch_out
4 | ... | ts_clk_pll_extswitch_out | set ts_clk_pll_extswitch_out

reg 16 fifo status

Data fifo status bits:

31: fifo_full - data fifo is current full
30: fifo_empty - data fifo is currently empty
29: cb_fifo_ch_full - TSC per-channel fifos have overflown at some point
28: fifo_full_latch - data fifo has overflown at some point
24+4: 0
0+24: fifo_usedw

reg 28

Timestamp clock PLL status bits:

31 : PLL locked
30 : PLL active clock (0=internal, 1=external
29 : external clock bad
28 : internal clock bad
27 : ts_clk_pll_extswitch
0..26 : not used

reg 32 timestamp sync

Timestamp synchronization register bits:

31 : write 1 to arm timestamp sync circuit, scalers and TSCs are placed in reset state
30 : write 1 to disarm timestamp sync circuit, reset is released
...
16 : write 1 to send a timestamp sync signal
15 : sync circuit armed
14 : sync done
13 : sync signal received (cleared by writing bits 31 or 30)
12 : 0
11 : sync circuit armed, clk_ts section
10 : sync done, clk_ts section
9  : sync signal received, clk_ts section
8  : 0
7  : unused
...
0  : unused

nominal sequence for synchronizing timestamps:

make sure nothing is sending the external sync signal (i.e. run is stopped, etc)
arm the sync circuit: cb_write32(32, 0x80000000); cb_write32(32, 0);
check that the arming was successful: cb_read32(32), only bits 0x8800 should be set
scalers are in reset state, not incrementing
TSCs are in reset state, no data in the TSC FIFO
some time later, send the external sync signal
or send the internal sync signal: cb_write32(32, 0x10000); cb_write32(32, 0);
check that the timestamp reset happened: cb_read32(32), bits 0x6600 should be set, bits 0x8800 should be cleared
check that the scalers are counting, check that there is TSC data in the TSC FIFO

reg 33, 39 TDC and AXI FIFO status

 bit |   mask     | description
  31 | 0x80000000 | fifo empty
  30 | 0x40000000 | fifo full
15:0 | 0x0000FFFF | fifo usedw (64-bit words)

Timestamp clock

Timestamp clock is 10 MHz selectable from internal oscillator or external reference on the CLK_IN input.

CLK_IN input can be selected using 2 two-position jumpers:

NIM input: CLK<->NIM/TTL and CLK<->NIM

To select the clock from command line, use:

test_cb.exe intclk # select internal clock
test_cb.exe extclk # select external clock

From software:

Chronobox* cb = ...;
cb->cb_int_clock(); # select internal clock
cb->cb_ext_clock(); # select external clock

To see current status, run "test_cb.exe clocks":

# ./test_cb.exe clocks
...
Chronobox firmware revision: 0x5bf7557e
...
clock status: ext_clk: counter 0x00cccf2a, freq 10000450.9 Hz, ts_clk: counter 0x00cccf2a, freq 10000450.9 Hz, PLL status 0xc0000000

Reported is external clock frequency, currently selected timestamp clock frequency and PLL status (register 0x1C).

Normal values for the PLL status:

internal clock: 0x80000000
external clock: 0xC0000000
external clock selected, but invalid: 0x60000000
internal clock selected, external clock invalid: 0xa0000000

Disconnected/absent/broken external clock will report ext_clk frequency zero, bit 0x20000000 in the PLL status register 0x1C.

If external clock is CDM, set chronobox clock jumpers to "CLK NIM/TTL" and "NIM", use CDM 10 MHz clock output LEMO3B.

If external clock is another chronobox (usually LEMO output 0), set clock jumpers to "CLK NIM/TTL" and "TTL".

LEMO outputs

LEMO connectors can be used as TTL level outputs:

set the "bank a" and/or "bank b" jumpers for "output"
set the lemo output multiplexor bits in register 0x1D
set the "lemo output enable" bit in register 0x0D, use bits 24..31 for LEMO outputs 0..7.
observe the corresponding LED is on or off according to the LEMO output TTL logic level

The function of each LEMO output is controlled by the lemo output multiplexor. Up to 16 different signals can be routed into each output. This is controlled by register 0x1D.

Register 0x1D multiplexor control is 32 bits organized into 8 groups of 4 bits, per each of the 8 LEMO outputs: 0x76543210, i.e. value 0x00000001 routes signal function 1 into output 0, and signal function 0 into outputs 1..7.

For each output, there are 16 possible signal functions (4 bits):

bit | firmware signal | firmware revision | signal description
0 | cb_lemo_out_reg[n] | 0x5bfdc798 | output is controlled by register 0xB bits 0..7 for outputs 0..7
1 | clk_ts | 0x5bfdc798 | output is the 10MHz timestamp clock
2 | cb_sync_in_clk_ts | 0x607f6709 | daisy-chained timestamp sync signal (from control register or from sync input, see registers 30 and 31. see bit 0x10000 in register 32)
3..30 | gnd | 0x5bfdc798 | not used
31 | vcc | 0x5bfdc798 | logic level 1

front panel LEDs

The front panel LEDs are numbered 0..7 per #Chronobox_connectors

Each LED can be individually lit by setting a bit in register 0x1 bits 8..15.

If LEMO outputs are active, corresponding LEDs will show LEMO output status (logic level 1 -> LED is on, logic level 0 -> LED is off).

If LEMO inputs are active ...

Synchronization of multiple chronoboxes

To operate several chronoboxes as one unit, they must have two common signals, a clock and a sync signal.

Each chronobox must be configured as:

master: this will issue the sync signal
slave: will receive the clock and sync signals from master
slave daisy-chain: and pass them to the next slave down the chain
(one can use the CDM or the TRG as sync masters)

For the purpose of this example (recommended configuration), set the LEMO direction jumpers:

bank A: set to "out": master and slave daisy-chain clock and sync output
bank B: set to "in": slave sync input

Clock connections:

sync master can be configured with external or internal clock (CLK_IN input)
sync slave must be configured with external clock (CLK_IN input)

sync master LEMO #0 -> slave CLK_IN, LEMO #0 -> next slave CLK_IN, LEMO #0 -> ...
10 MHz clock from CDM & etc -> sync master CLK_IN, LEMO #0 -> slave CLK_IN, ...

Sync connections:

sync signal can be generated externally by the CDM or TRG
sync signal can be generated internally by writing 0x10000 into reg 32 of sync master
(sync must be armed on the master and all slaves before sending the first sync signal, write 0x80000000 into reg 32)

sync master LEMO #1 -> slave LEMO #4, LEMO #1 -> next slave LEMO #4 -> ...
external sync -> master LEMO #4, LEMO #1 -> slave LEMO #4 -> ...

Sync configuration: (see register 32 for explanation!)

master and slave: disarm sync: write 0x40000000 to reg 32, write 0 to reg 32, bits 0x8800 and 0x6600 should clear
master and slave daisy chain: setup clock and sync outputs:
- register 29 lemo output mux to 0xXXXX'XX21 (2=output sync signal, 1=output clock)
- register 13 lemo output enable, set bits 0x0F00'0000 (set bank A jumper for "out")
slave and slave daisy chain: enable external sync from LEMO #4:
- register 30, write 0
- register 31, write 0x10
master: switch to external or internal clock, as needed
slave: switch to external clock
master and slave: arm sync: write 0x80000000 to reg 32, write 0 to reg 32, bits 0x8800 should turn on.
timestamps, scalers and data fifo go into the reset state, all old data is cleared
when sync is received, timestamp is reset to zero, reset state is cleared and there should be data into FIFO. reg 32 will read 0x6600.
if using external sync (no master), wait for reg 32 bits to become 0x6600 (sync completed)
master: issue sync signal, write 0x10000 to reg 32, write 0 to reg 32. master reg 32 should read 0x6600, slave reg 32 should read 0x6600.

FIFO data format

0x8ntttttt: TSC data, 24 bits "tttttt" of timestamp, 7 bits "nn" of channel number, top bit set to 1. Low bit of "t" indicates 0=leading edge, 1=trailing edge.
0xffTTmmmm: (before 20220420_ko_0x62608957) timestamp wrap around marker: "TT" is the top 8 bits of the timestamp, "mmmm" is a 16 bit counter
0xff8mmmmm: timestamp wrap around marker: bit 24 is the top bit of the timestamp (0/8), "mmmmm" is a 23 bit counter
0xfe00nnnn: scaler data, following "nnnn" words are the latched scalers
0xfd000000: start of data after chronobox sync (cbtrg only, for now)

Time range is (before 20220420_ko_0x62608957):

24 bits of "tttttt"
15 bits of "mmmm"
computation:

(gdb) p (100.0*1.0*0x00007FFF*1.0*0x00FFFFFF)/(1e9)/3600.0
$22 = 15.270527886249999 ### in hours

Time range is:

24 bits of "tttttt"
22 bits of "mmmmm"
computation:

(gdb) p (100.0*1.0*0x003FFFFF*1.0*0x00FFFFFF)/(1e9)/3600.0
$2 = 1954.6867557262499 ### in hours
$3 = 81.445281488593736 ### in days

timestamp wrap around marker

The timestamp data is only 24 bits, to allow timestamping with longer time range, wrap around markers are added to the data stream.

For input signals that arrive close to the time of timestamp wrap around, there is ambiguity in the ordering of the data fifo: does the wrap around marker or the signal timestamp show up first? For example for rare signals, one can see this:

wrap 1
wrap 2
timestamp 0x00000003
wrap 3
wrap 4

does the hit belong with wrap marker 2 (written to the fifo just after wrap marker 2) or with marker 3 (written to the fifo just before wrap marker 3)?

To remove this ambiguity, additional markers are written to the data stream half way between the wrap arounds, making it obvious that the signal arrived right after wrap marker 3 (but was written to the FIFO before the marker):

wrap 1 0x00
wrap 1 0x80
wrap 2 0x00
wrap 2 0x80
timestamp 0x00000003
wrap 3 0x00
wrap 3 0x80

test_cb.exe

test_cb.exe is the general test program for the chronobox.

test_cb.exe 0 # read chronobox register 0
test_cb.exe 4 0x1234 # write to chronobox register 4
test_cb.exe fpga 0 # read FPGA LW bridge address 0
test_cb.exe fpga 0 0x12345678 # write 0x12345678 to FPGA LW bridge address 0
test_cb.exe reboot # reboot the FPGA (the ARM CPU keeps running)
test_cb.exe scalers # read all scalers in a loop
test_cb.exe fifo # read the data fifo in a loop
test_cb.exe tsfifo # read the timestamp fifo in a loop (do not latch scalers)
test_cb.exe intclk # select internal timestamp clock
test_cb.exe extclk # select external timestamp clock (10MHz)
test_cb.exe clocks # report current status of timestamp clock

Errata

0x607f6709 - after FIFO reset (timestamp sync, etc), the first word in the FIFO is always 0x00000000, not a real timestamps.
20220420_ko_0x62608957 - after FIFO reset (timestamp sync), the first word in the FIFO is 0xFF000000, a timestamp overflow marker, as expected. after FPGA reboot (before timestamp sync), the first word in the FIFO is also 0xFF000000 for reasons unknown.
linux - after power up, chronobox registers are not accessible, check that the FPGA bridges are enabled, if any bridge reports "disabled", reboot linux. do NOT cycle power.

root@cb04:~# cat /sys/class/fpga_bridge/br*/state
enabled
enabled
enabled
enabled
root@cb04:~#

TODO

FIXED 23apr2021 - use updated "bootcmd". figure out why FPGA bridges are disabled on first power up
figure out how to load FPGA sof from Linux, probably need new kernel
FIXED apr2022, use fw_printenv build from source - fix fw_printenv, probably requires using newer uboot
add mmap driver for FPGA LW bridge, probably need new kernel
figure out how to build linux kernel from git (now using kernel from GHRD demo kit)
enable watchdog timer to auto-reboot on hang
figure out why ssh takes 30 second to start. read this: https://daniel-lange.com/archives/152-Openssh-taking-minutes-to-become-available,-booting-takes-half-an-hour-...-because-your-server-waits-for-a-few-bytes-of-randomness.html https://lwn.net/Articles/800509/
FIXED 20220420_ko_0x62608957 - fix problem on startup first word in the FIFO is 0x00000000.
FIXED 20220420_ko_0x62608957 - increase number of bits in the timestamp wraparound counter: 0xFF8mmmmm -> 23 bits is possible, keep high bit 0x00800000 as is.

Chronobox

Chronobox

Links

Chronobox connectors

Electrical connections

Input channel mapping

Functional units

Scalers

Firmware 0x61832695

Firmware 0x624e22ad

Timestamps (TSC)

Firmware 0x61832695

Firmware 0x624e22ad

Output data fifo

Firmware before 0x61832695

Firmware 0x624e22ad

Initial hardware setup

assign ethernet MAC address

new board test checklist

test ECL inputs

test TTL inputs

test TTL outputs

test clock input

test GPIO pins

Install chronobox software

Install chronobox quartus firmware project

Firmware revisions

Firmware update

Firmware build

Errata

Firmware details

Chronobox firmware registers

reg 0x00 write bits

reg 16 fifo status

reg 28

reg 32 timestamp sync

reg 33, 39 TDC and AXI FIFO status

Timestamp clock

LEMO outputs

front panel LEDs

Synchronization of multiple chronoboxes

FIFO data format

timestamp wrap around marker

test_cb.exe

Errata

TODO

ZZZ

Navigation menu

Search