Chronobox

Links

• https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano
• https://bitbucket.org/expalpha/chronobox_firmware
• https://bitbucket.org/expalpha/chronobox_software

Chronobox connectors

|
| LEMO - CLK_IN - jumpers: SINE<->CLK<->NIM/TTL and NIM<->CLK<->TTL
|
| ECL P1
| 32
| |
| 1
|
| ECL P2
| 32
| |
| 1
|
| PLED - power-on LED
|
| jumper: bank A direction IN<->X<->OUT
| LEMO 0-1
| LED 0-1
| LEMO 2-3
| LED 2-3
|
| jumper: bank B direction IN<->X<->OUT
| LEMO 4-5
| LED 4-5
| LEMO 6-7
| LED 6-7
|

Input channel mapping

• 0+16 : first ECL connector
• 16+16 : second ECL connector
• 32+8 : LEMO inputs (TTL)
• 40+18 : GPIO inputs (FPGA pins)
• 58 : external clock (10 MHz nominal)
• 59 : internal clock (100 MHz)

Functional units

Scalers

TBW

Timestamps (TSC)

TBW

Each TSC input has a 256 entry FIFO. (LE is 1 entry, TE is 1 entry, so 128 hits if both LE and TE enabled).

Output data fifo

TBW

Output fifo is 4096 entries deep.

Initial hardware setup

• refer to DE10-NANO information here: https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano
• prepare hardware:
  • remove chronobox from enclosure (cannot access JTAG connector when installed)
  • remove DE10-Nano from chronobox baseboard
  • check that SW10 jumpers are: U-D-U-U-D-U. (U=up, D=down, when "ALTERA" text on the FPGA is right side up)
  • reinstall DE10-Nano on chronobox baseboard
  • do NOT reinstall chronobox in enclosure
• prepare software:
  • check that you have quartus 17.1 (at /opt/intelFPGA/17.1/ in this example)
  • check that you installed chronobox software in /home/agdaq/online/chronobox_software
  • check that you installed chronobox firmware project in /home/agdaq/online/firmware/git/chronobox_firmware
• connect serial console, for more info, go here: https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano#Serial_console
  • connect a mini-USB (*not* Micto-USB!) cable from mini-USB port on the side of the ethernet connection to a PC
  • minicom -D /dev/ttyUSB0 -b 115200
• connect fpga jtag, for more info, go here: https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano#JTAG
  • connect a mini-USB (*not* Micro-USB!) cable on the side of HDMI and power connectors to a PC
  • /opt/intelFPGA/17.1/quartus/bin/jtagconfig
• prepare SD flash card for booting from network: generic instructions are here: https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano#boot_Linux_from_SD_flash
  • connect new or used 8GB SD flash card to USB flash adapter to a linux computer
  • login as root
  • identify the flash device as /dev/sdX. Use "lsblk" or "blkid" or "dmesg". be careful to not write to the computer system disk (usually /dev/sda)
  • echo -e 'o\nn\np\n1\n2048\n4095\nn\np\n2\n4096\n1681816\nn\np\n3\n\n\nt\n1\na2\nt\n2\nb\nw' | fdisk /dev/sdX
  • eject /dev/sdX
  • remove card, reintall card, this is to ensure we use the new partition table
  • dd if=/daq/daqstore/olchansk/daq/DE10-Nano/image-1-fpga.img of=/dev/sdX1 bs=1024k
  • dd if=/daq/daqstore/olchansk/daq/DE10-Nano/image-2-uboot.img of=/dev/sdX2 bs=1024k
  • eject /dev/sdX
• install prepared flash card
• cycle power
• in the minicom window, IMMEDIATELY press the space bar to stop automatic booting
• you will have the uboot "=>" prompt
• confirm uboot version: U-Boot 2013.01.01
• setup uboot to boot linux from network, more details here: https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano#boot_Linux_from_network
  • setenv ethaddr 02:aa:bb:cc:cb:04 # for cb04, see below for explanation
  • setenv bootcmd 'run netboot'
  • setenv ramboot bootz \${loadaddr} - \${fdtaddr}
  • setenv bootargs console=ttyS0,115200 ip=dhcp root=/dev/nfs rw nfsroot=192.168.1.1:/zssd1tb/nfsroot/%s,vers=3 panic=15
  • saveenv
  • reset
• DE10-Nano should boot into linux:
  • uboot will restart
  • DHCP will run
  • tftp load of linux kernel will run
  • linux kernel will start
  • linux kernel will get an IP address
  • linux kernel will NFS-mount the root filesystem (.../nfsroot/cb04)
  • systemd will start all services
  • there will be a login prompt on minicom console
  • ssh will work (maybe 1 minute delay before it starts)
• try a few things:
  • from agdaq or agmini account, ssh root@cb04
  • "df" should show /home/agdaq is mounted (and ssh agdaq@cb04 should work)
  • "/home/agdaq/online/chronobox_software/test_cb.exe 0" should fail ("bus error") because FPGA is not loaded yet
• load sof file, for more info, go here: https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano#JTAG
  • /opt/intelFPGA/17.1/quartus/bin/quartus_pgm -m JTAG -o "p;/home/agdaq/online/firmware/git/chronobox_firmware/output_files/DE10_NANO_SoC_GHRD.sof@2"
• "test_cb.exe 0" should work now
• use srunner_cb.exe to load the pof file into FPGA boot flash
• use reboot_cb.exe to reboot the FPGA from flash
• "test_cb.exe 0" should report the expected FPGA firmware version number.
• success.

assign ethernet MAC address

Usually ethernet MAC address is assigned by manufacturer and is stored in a tiny little flash chip. The DE10-Nano board is too cheap and does not have it. So a fake ethernet address has to be assigned manually from the "locally administered range" of MAC addresses that start from "02:...", see https://en.wikipedia.org/wiki/MAC_address#Ranges_of_group_and_locally_administered_addresses

All MAC addresses must be unique on an ethernet network, but with manual assignement confusion is easy to create.

For chronobox devices, use MAC addresses that start with "l02:aa:bb:cc:xx:yy" and have last digits "cb:01", "cb:02", "cb:03", "cb04" for the first four chronoboxes.

Install chronobox software

cd /home/agdaq/online
git clone https://bitbucket.org/expalpha/chronobox_software.git
cd chronobox_software
make clean
make
ls -l *.exe
-rwxr-xr-x 1 olchansk users 18808 Aug 16 15:26 reboot_cb.exe
-rwxr-xr-x 1 olchansk users 47256 Aug 16 15:26 srunner_cb.exe
-rwxr-xr-x 1 olchansk users 20732 Aug 16 15:26 test_cb.exe

Install chronobox quartus firmware project

cd /home/agdaq/online/firmware/git
git clone https://bitbucket.org/expalpha/chronobox_firmware.git
cd chronobox_firmware
cat timestamp.v
ls -l output_files/*.jic

Firmware revisions

• 0x5aceaed2 - April 2018 - all inputs connected to counters (except ext clock), 58 inputs, 50 MHz clock
• 0x5b6de806 - August 2018 - added fpga boot flash programmer and fpga reboot
• 0x5b7c827d - August 2018 - bshaw added debounce on GPIO inputs (for flow meters), 100 MHz clock
• 0x5b873169 - August 2018 - rebuilt, no changes
• 0x5b89e4b4 - August 2018 - added external clock signal, 59 inputs, 100 MHz clock
• 0x5b8de2b0 - September 2018 - added data fifo and timestamp counters (TSCs) for the first 4 inputs
• 0x5b906c56 - September 2018 - improved overflow markers, added scalers readout into the data fifo
• 0x5da8e4c2 - October 2019 - timestamps for all inputs, inversion of inputs
• 0x5db764a7 - October 2019 - FPGA reboot works now.
• 0x607f6709 - April 2021 - working timestamp synchronization, memory mapped registers, improved sdc file, correct clock transfers

Firmware update

If FPGA is not running compatible firmware srunner_cb will not work. To proceed, load the correct SOF file via JTAG (https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano#JTAG_load_sof_file), then srunner_cb should work and will be able to load the jic or rpd file into the FPGA boot flash memory.

Different revisions of the DE10-Nano board have different FPGA boot flash chips, some have EPCQ128 (use the "-128" option), some have the EPCQ64 chip (use the "-64" option). Use "srunner_cb -id" per example below to identify which flash chip is present on each specific chronobox. Note that the DE10-Nano documentation and the firmware quartus project generally refer to the EPCS64/EPCQ64 chip. The only practical difference is the use of "-128" or "-64" srunner_cb options.

cd /home/agdaq/online/chronobox_software
./srunner_cb.exe -id -64 /dev/null # identify EPCS64 flash
./srunner_cb.exe -id -128 /dev/null # identify EPCQ128 flash
./srunner_cb.exe -read -128 test.rpd # read flash contents into a file
#./srunner_cb.exe -program -128 /home/olchansk/git/chronobox_firmware/output_files/DE10_NANO_SoC_GHRD_auto.rpd # write firmware rpd file into flash
./srunner_cb.exe -program -128 /home/agdaq/online/firmware/git/chronobox_firmware/output_files/DE10_NANO_SoC_GHRD_auto.rpd
./reboot_cb.exe # reboot the fpga into the new firmware

Firmware build

for building firmware use: /opt/intelFPGA/17.1/nios2eds/nios2_command_shell.sh

ssh daq16
cd online
git clone git@bitbucket.org:expalpha/chronobox_firmware.git
cd chronobox_firmware
/opt/intelFPGA/17.1/nios2eds/nios2_command_shell.sh
make quartus
ls -l output_files/*.{sof,jic,rpd}
-rw-r--r-- 1 olchansk users 7007185 Apr 20 16:57 output_files/DE10_NANO_SoC_GHRD_auto.rpd
-rw-r--r-- 1 olchansk users 8388833 Apr 20 16:57 output_files/DE10_NANO_SoC_GHRD.jic
-rw-r--r-- 1 olchansk users 7510701 Apr 20 16:57 output_files/DE10_NANO_SoC_GHRD.sof

other make targets:

• make quartus - build fpga programmer files: sof, jic and rpd.
• make clean - clean the quartus project
• make qsys - regenerate the qsys block
• make jic - regenerate the jic file
• make load_sof - load sof file into fpga via jtag
• make load_jic - load jic file into fpga boot flash (epcq) via jtag
• make verify_jic - verify jic checksum in fpga boot flash (epcq)

for jtag access, use /daq/quartus/13.1.3.178/quartus/bin/jtagconfig

daq16:chronobox_firmware$ /daq/quartus/13.1.3.178/quartus/bin/jtagconfig
1) DE-SoC [2-1.4.3]
  4BA00477   SOCVHPS
  02D020DD   5CSEBA6(.|ES)/5CSEMA6/..

2) Remote server daq16: Unable to connect

daq16:chronobox_firmware$ 

Chronobox firmware registers

reg | rw/ro | quartus name | firmware | description
0 | ro | sof_revision_in | all | firmware revision timestamp code
0 | wo | latch_scalers_out, zero_scalers_out | all | see [[#reg_0x00_write_bits]]
1 | rw | reg1_led_out | all | DE10-Nano LED output
2 | ro | switches_in | all | read DE10-Nano switches
3 | ro | buttons_in | all | read DE10-Nano buttons
4 | rw | reg4_test | all | 32-bit read-write test register
5 | rw | flash_programmer_in, reg5_flash_programmer_out | 0x5b6de806 | 0xABCD srunner flash programmer
6 | ro | ecl_in | all | read state of ECL inputs
7 | ro | reg7_test_in | all | ???
8 | rw | scaler_addr_out, reg8_scaler_data_in | all | top 16 bits of address becomes scaler bus address, 32 bit read is the corresponding scaler data
9 | ro | lemo_in | all | read state of LEMO inputs
10/0xA | ro | gpio_in | all | read state of GPIO inputs
11/0xB | rw | regB_lemo_out | all | LEMO output data
12/0xC | rw | regC_gpio_out | all | GPIO output data
13/0xD | rw | regD_out_enable_out | all | enable output tristates: [31:24] - LEMO_OUT, [17:0] - GPIO_OUT
14/0xE | rw | regE, reconfig_out | 0x5b6de806 | FPGA reboot: write inverted firmware revision (reg0) to reboot the FPGA
15/0xF | ro | regF_input_num_in | 0x5b89e4b4 | number of chronobox inputs (to read scalers, add 1 for the clock counter)
16/0x10 | ro | reg10_fifo_status | 0x5b8de2b0 | data fifo status, see below
17/0x11 | ro | reg11_fifo_data | 0x5b8de2b0 | data fifo data, see below
18/0x12 | rw | cb_invert_a | 0x5bf7557e | invert inputs 31..0
19/0x13 | rw | cb_invert_b | 0x5bf7557e | invert inputs 63..32
20/0x14 | rw | cb_enable_le_a | 0x5bf7557e | enable TSC leading edge inputs 31..0
21/0x15 | rw | cb_enable_le_b | 0x5bf7557e | enable TSC leading edge inputs 63..32
22/0x16 | rw | cb_enable_te_a | 0x5bf7557e | enable TSC trailing edge inputs 31..0
23/0x17 | rw | cb_enable_te_b | 0x5bf7557e | enable TSC trailing edge inputs 63..32
24/0x18 | ro | fc_ext_clk_100_counter | 0x5bf7557e | external clock frequency counter 100MHz reference
25/0x19 | ro | fc_ext_clk_ext_counter | 0x5bf7557e | external clock frequency counter
26/0x1A | ro | fc_ts_clk_100_counter | 0x5bf7557e | timestamp clock frequency counter 100MHz reference
27/0x1B | ro | fc_ts_clk_ts_counter | 0x5bf7557e | timestamp clock frequency counter
28/0x1C | ro | ts_clk_pll_status | 0x5bf7557e | timestamp clock PLL status
29/0x1D | rw | cb_lemo_out_mux_ctrl | 0x5bfdc798 | 8*4 bits to control 8 LEMO output multiplexers (4 bits/16 options each mux)
30/0x1E | rw | cb_sync_mask[31:0] | 0x607f6709 | source of chronobox sync signal, low bits
31/0x1F | rw | cb_sync_mask[63:32] | 0x607f6709 | source of chronobox sync signal, high bits
32 | rw | cb_sync_reg[31:0], cb_sync_status[31:0] | 0x607f6709 | control of timestamp sync

reg 0x00 write bits

bit | fw revision | quartus signal | description
0 | all | | latch scalers
1 | all | | zero scalers
2 | ... | fifo_rdreq_out | fifo_rdreq_out (not used in version 0x607f6709 and later)
3 | ... | ts_clk_pll_extswitch_out | clear ts_clk_pll_extswitch_out
4 | ... | ts_clk_pll_extswitch_out | set ts_clk_pll_extswitch_out

reg 16 fifo status

Data fifo status bits:

31: fifo_full - data fifo is current full
30: fifo_empty - data fifo is currently empty
29: cb_fifo_ch_full - TSC per-channel fifos have overflown at some point
28: fifo_full_latch - data fifo has overflown at some point
24+4: 0
0+24: fifo_usedw

reg 28

Timestamp clock PLL status bits:

31 : PLL locked
30 : PLL active clock (0=internal, 1=external
29 : external clock bad
28 : internal clock bad
27 : ts_clk_pll_extswitch
0..26 : not used

reg 32 timestamp sync

Timestamp synchronization register bits:

31 : write 1 to arm timestamp sync circuit, scalers and TSCs are placed in reset state
30 : write 1 to disarm timestamp sync circuit, reset is released
...
16 : write 1 to send a timestamp sync signal
15 : sync circuit armed
14 : sync done
13 : sync signal received (cleared by writing bits 31 or 30)
12 : 0
11 : sync circuit armed, clk_ts section
10 : sync done, clk_ts section
9  : sync signal received, clk_ts section
8  : 0
7  : unused
...
0  : unused

nominal sequence for synchronizing timestamps:

• make sure nothing is sending the external sync signal (i.e. run is stopped, etc)
• arm the sync circuit: cb_write32(32, 0x80000000); cb_write32(32, 0);
• check that the arming was successful: cb_read32(32), only bits 0x8800 should be set
• scalers are in reset state, not incrementing
• TSCs are in reset state, no data in the TSC FIFO
• some time later, send the external sync signal
• or send the internal sync signal: cb_write32(32, 0x10000); cb_write32(32, 0);
• check that the timestamp reset happened: cb_read32(32), bits 0x6600 should be set, bits 0x8800 should be cleared
• check that the scalers are counting, check that there is TSC data in the TSC FIFO

Timestamp clock

Timestamp clock is 10 MHz selectable from internal oscillator or external reference on the CLK_IN input.

CLK_IN input can be selected using 2 two-position jumpers:

• NIM input: CLK<->NIM/TTL and CLK<->NIM

To select the clock from command line, use:

• test_cb.exe intclk # select internal clock
• test_cb.exe extclk # select external clock

From software:

Chronobox* cb = ...;
cb->cb_int_clock(); # select internal clock
cb->cb_ext_clock(); # select external clock

To see current status, run "test_cb.exe clocks":

# ./test_cb.exe clocks
...
Chronobox firmware revision: 0x5bf7557e
...
clock status: ext_clk: counter 0x00cccf2a, freq 10000450.9 Hz, ts_clk: counter 0x00cccf2a, freq 10000450.9 Hz, PLL status 0xc0000000

Reported is external clock frequency, currently selected timestamp clock frequency and PLL status (register 0x1C).

Normal values for the PLL status:

• internal clock: 0x80000000
• external clock: 0xC0000000
• external clock selected, but invalid: 0x60000000
• internal clock selected, external clock invalid: 0xa0000000

Disconnected/absent/broken external clock will report ext_clk frequency zero, bit 0x20000000 in the PLL status register 0x1C.

If external clock is CDM, set chronobox clock jumpers to "CLK NIM/TTL" and "NIM", use CDM 10 MHz clock output LEMO3B.

If external clock is another chronobox (usually LEMO output 0), set clock jumpers to "CLK NIM/TTL" and "TTL".

LEMO outputs

LEMO connectors can be used as TTL level outputs:

• set the "bank a" and/or "bank b" jumpers for "output"
• set the lemo output multiplexor bits in register 0x1D
• set the "lemo output enable" bit in register 0x0D, use bits 24..31 for LEMO outputs 0..7.
• observe the corresponding LED is on or off according to the LEMO output TTL logic level

The function of each LEMO output is controlled by the lemo output multiplexor. Up to 16 different signals can be routed into each output. This is controlled by register 0x1D.

Register 0x1D multiplexor control is 32 bits organized into 8 groups of 4 bits, per each of the 8 LEMO outputs: 0x76543210, i.e. value 0x00000001 routes signal function 1 into output 0, and signal function 0 into outputs 1..7.

For each output, there are 16 possible signal functions (4 bits):

bit | firmware signal | firmware revision | signal description
0 | cb_lemo_out_reg[n] | 0x5bfdc798 | output is controlled by register 0xB bits 0..7 for outputs 0..7
1 | clk_ts | 0x5bfdc798 | output is the 10MHz timestamp clock
2 | cb_sync_in_clk_ts | 0x607f6709 | daisy-chained timestamp sync signal (from control register or from sync input, see registers 30 and 31. see bit 0x10000 in register 32)
3..30 | gnd | 0x5bfdc798 | not used
31 | vcc | 0x5bfdc798 | logic level 1

front panel LEDs

The front panel LEDs are numbered 0..7 per #Chronobox_connectors

Each LED can be individually lit by setting a bit in register 0x1 bits 8..15.

If LEMO outputs are active, corresponding LEDs will show LEMO output status (logic level 1 -> LED is on, logic level 0 -> LED is off).

If LEMO inputs are active ...

Synchronization of multiple chronoboxes

To operate several chronoboxes as one unit, they must have two common signals, a clock and a sync signal.

Each chronobox must be configured as:

• master: this will issue the sync signal
• slave: will receive the clock and sync signals from master
• slave daisy-chain: and pass them to the next slave down the chain
• (one can use the CDM or the TRG as sync masters)

For the purpose of this example (recommended configuration), set the LEMO direction jumpers:

• bank A: set to "out": master and slave daisy-chain clock and sync output
• bank B: set to "in": slave sync input

Clock connections:

• sync master can be configured with external or internal clock (CLK_IN input)
• sync slave must be configured with external clock (CLK_IN input)

sync master LEMO #0 -> slave CLK_IN, LEMO #0 -> next slave CLK_IN, LEMO #0 -> ...
10 MHz clock from CDM & etc -> sync master CLK_IN, LEMO #0 -> slave CLK_IN, ...

Sync connections:

• sync signal can be generated externally by the CDM or TRG
• sync signal can be generated internally by writing 0x10000 into reg 32 of sync master
• (sync must be armed on the master and all slaves before sending the first sync signal, write 0x80000000 into reg 32)

sync master LEMO #1 -> slave LEMO #4, LEMO #1 -> next slave LEMO #4 -> ...
external sync -> master LEMO #4, LEMO #1 -> slave LEMO #4 -> ...

Sync configuration:

• master: "sync_arm" activated by software, "sync" activated by software (cb_sync_mask set to zero)
  • register 0x1D output mux to 0xXXXX'XX21 (2=output sync signal, 1=output clock)
  • register 0x0D output enable, set bits 0x0F00'0000 (output enable for bank A lemo #0 clock and #1 sync)
  • register 0x1E and 0x1F, set to 0 (cb_sync external source)
• slave: "sync_arm" activated by software, "sync" activated by lemo in #4 (cb_sync_mask set bit 32+4).
  • register 0x1E set to zero
  • register 0x1F set to 0x0000'0010
  • for daisy-chain operation, set registers 0x1D and 0x0D same as master

Sync operation:

• issue a "sync_arm" command. Scalers and timestamps will be stopped. Time counter will be reset to zero, scalers will be reset to zero.
• issue a "sync" command. Scalers and timestamps will be started at the same time in all units.

FIFO data format

• 0x8ntttttt: TSC data, 24 bits "tttttt" of timestamp, 7 bits "nn" of channel number, top bit set to 1. Low bit of "t" indicates 0=leading edge, 1=trailing edge.
• 0xffTTmmmm: timestamp wrap around marker: "TT" is the top 8 bits of the timestamp, "mmmm" increments for each marker
• 0xfe00nnnn: scaler data, following "nnnn" words are the latched scalers

timestamp wrap around marker

The timestamp data is only 24 bits, to allow timestamping with longer time range, wrap around markers are added to the data stream.

For input signals that arrive close to the time of timestamp wrap around, there is ambiguity in the ordering of the data fifo: does the wrap around marker or the signal timestamp show up first? For example for rare signals, one can see this:

wrap 1
wrap 2
timestamp 0x00000003
wrap 3
wrap 4

does the hit belong with wrap marker 2 (written to the fifo just after wrap marker 2) or with marker 3 (written to the fifo just before wrap marker 3)?

To remove this ambiguity, additional markers are written to the data stream half way between the wrap arounds, making it obvious that the signal arrived right after wrap marker 3 (but was written to the FIFO before the marker):

wrap 1 0x00
wrap 1 0x80
wrap 2 0x00
wrap 2 0x80
timestamp 0x00000003
wrap 3 0x00
wrap 3 0x80

test_cb.exe

test_cb.exe is the general test program for the chronobox.

• test_cb.exe 0 # read chronobox register 0
• test_cb.exe 4 0x1234 # write to chronobox register 4
• test_cb.exe reboot # reboot the FPGA (the ARM CPU keeps running)
• test_cb.exe scalers # read all scalers in a loop
• test_cb.exe fifo # read the data fifo in a loop
• test_cb.exe intclk # select internal timestamp clock
• test_cb.exe extclk # select external timestamp clock (10MHz)
• test_cb.exe clocks # report current status of timstamp clock