Chronobox

Links

• https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano
• https://bitbucket.org/expalpha/chronobox_firmware
• https://bitbucket.org/expalpha/chronobox_software

Chronobox connectors

|
| LEMO - CLK_IN - jumpers: SINE<->CLK<->NIM/TTL and NIM<->CLK<->TTL
|
| ECL P1
| 32
| |
| 1
|
| ECL P2
| 32
| |
| 1
|
| PLED - power-on LED
|
| jumper: bank A direction IN<->X<->OUT
| LEMO 0-1
| LED 0-1
| LEMO 2-3
| LED 2-3
|
| jumper: bank B direction IN<->X<->OUT
| LEMO 4-5
| LED 4-5
| LEMO 6-7
| LED 6-7
|

Input channel mapping

• 0+16 : first ECL connector
• 16+16 : second ECL connector
• 32+8 : LEMO inputs (TTL)
• 40+18 : GPIO inputs (FPGA pins)
• 58 : external clock (10 MHz nominal)
• 59 : internal clock (100 MHz)

Initial hardware setup

• refer to DE10-NANO information here: https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano
• prepare hardware:
  • remove chronobox from enclosure (cannot access JTAG connector when installed)
  • remove DE10-Nano from chronobox baseboard
  • check that SW10 jumpers are: U-D-U-U-D-U. (U=up, D=down, when "ALTERA" text on the FPGA is right side up)
  • reinstall DE10-Nano on chronobox baseboard
  • do NOT reinstall chronobox in enclosure
• prepare software:
  • check that you have quartus 17.1 (at /opt/intelFPGA/17.1/ in this example)
  • check that you installed chronobox software in /home/agdaq/online/chronobox_software
  • check that you installed chronobox firmware project in /home/agdaq/online/firmware/git/chronobox_firmware
• connect serial console, for more info, go here: https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano#Serial_console
  • connect a mini-USB (*not* Micto-USB!) cable from mini-USB port on the side of the ethernet connection to a PC
  • minicom -D /dev/ttyUSB0 -b 115200
• connect fpga jtag, for more info, go here: https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano#JTAG
  • connect a mini-USB (*not* Micro-USB!) cable on the side of HDMI and power connectors to a PC
  • /opt/intelFPGA/17.1/quartus/bin/jtagconfig
• prepare SD flash card for booting from network: generic instructions are here: https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano#boot_Linux_from_SD_flash
  • connect new or used 8GB SD flash card to USB flash adapter to a linux computer
  • login as root
  • identify the flash device as /dev/sdX. Use "lsblk" or "blkid" or "dmesg". be careful to not write to the computer system disk (usually /dev/sda)
  • echo -e 'o\nn\np\n1\n2048\n4095\nn\np\n2\n4096\n1681816\nn\np\n3\n\n\nt\n1\na2\nt\n2\nb\nw' | fdisk /dev/sdX
  • dd if=/daq/daqstore/olchansk/daq/DE10-Nano/image-1-fpga.img of=/dev/sdX1 bs=1024k
  • dd if=/daq/daqstore/olchansk/daq/DE10-Nano/image-2-uboot.img of=/dev/sdX2 bs=1024k
  • eject /dev/sdX
• install prepared flash card
• cycle power
• in the minicom window, IMMEDIATELY press the space bar to stop automatic booting
• you will have the uboot "=>" prompt
• confirm uboot version: U-Boot 2013.01.01
• setup uboot to boot linux from network, more details here: https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano#boot_Linux_from_network
  • setenv ethaddr 02:aa:bb:cc:cb:04 # for cb04, see below for explanation
  • setenv bootcmd 'run netboot'
  • setenv ramboot bootz \${loadaddr} - \${fdtaddr}
  • setenv bootargs console=ttyS0,115200 ip=dhcp root=/dev/nfs rw nfsroot=192.168.1.1:/zssd1tb/nfsroot/%s,vers=3 panic=15
  • saveenv
  • reset
• DE10-Nano should boot into linux:
  • uboot will restart
  • DHCP will run
  • tftp load of linux kernel will run
  • linux kernel will start
  • linux kernel will get an IP address
  • linux kernel will NFS-mount the root filesystem (.../nfsroot/cb04)
  • systemd will start all services
  • there will be a login prompt on minicom console
  • ssh will work (maybe 1 minute delay before it starts)
• try a few things:
  • from agdaq or agmini account, ssh root@cb04
  • "df" should show /home/agdaq is mounted (and ssh agdaq@cb04 should work)
  • "/home/agdaq/online/chronobox_software/test_cb.exe 0" should fail ("bus error") because FPGA is not loaded yet
• load sof file, for more info, go here: https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano#JTAG
  • /opt/intelFPGA/17.1/quartus/bin/quartus_pgm -m JTAG -o "p;/home/agdaq/online/firmware/git/chronobox_firmware/output_files/DE10_NANO_SoC_GHRD.sof@2"
• "test_cb.exe 0" should work now
• use srunner_cb.exe to load the pof file into FPGA boot flash
• use reboot_cb.exe to reboot the FPGA from flash
• "test_cb.exe 0" should report the expected FPGA firmware version number.
• success.

assign ethernet MAC address

Usually ethernet MAC address is assigned by manufacturer and is stored in a tiny little flash chip. The DE10-Nano board is too cheap and does not have it. So a fake ethernet address has to be assigned manually from the "locally administered range" of MAC addresses that start from "02:...", see https://en.wikipedia.org/wiki/MAC_address#Ranges_of_group_and_locally_administered_addresses

All MAC addresses must be unique on an ethernet network, but with manual assignement confusion is easy to create.

For chronobox devices, use MAC addresses that start with "l02:aa:bb:cc:xx:yy" and have last digits "cb:01", "cb:02", "cb:03", "cb04" for the first four chronoboxes.

Install chronobox software

cd /home/agdaq/online
git clone https://bitbucket.org/expalpha/chronobox_software.git
cd chronobox_software
make clean
make
ls -l *.exe
-rwxr-xr-x 1 olchansk users 18808 Aug 16 15:26 reboot_cb.exe
-rwxr-xr-x 1 olchansk users 47256 Aug 16 15:26 srunner_cb.exe
-rwxr-xr-x 1 olchansk users 20732 Aug 16 15:26 test_cb.exe

Install chronobox quartus firmware project

cd /home/agdaq/online/firmware/git
git clone https://bitbucket.org/expalpha/chronobox_firmware.git
cd chronobox_firmware
cat timestamp.v
ls -l output_files/*.jic

Firmware revisions

• 0x5aceaed2 - April 2018 - all inputs connected to counters (except ext clock), 58 inputs, 50 MHz clock
• 0x5b6de806 - August 2018 - added fpga boot flash programmer and fpga reboot
• 0x5b7c827d - August 2018 - bshaw added debounce on GPIO inputs (for flow meters), 100 MHz clock
• 0x5b873169 - August 2018 - rebuilt, no changes
• 0x5b89e4b4 - August 2018 - added external clock signal, 59 inputs, 100 MHz clock
• 0x5b8de2b0 - September 2018 - added data fifo and timestamp counters (TSCs) for the first 4 inputs
• 0x5b906c56 - September 2018 - improved overflow markers, added scalers readout into the data fifo
• 0x5da8e4c2 - October 2019 - timestamps for all inputs, inversion of inputs
• 0x5db764a7 - October 2019 - FPGA reboot works now.

Firmware update

If FPGA is not running compatible firmware srunner_cb will not work. To proceed, load the correct SOF file via JTAG (https://daq.triumf.ca/DaqWiki/index.php/DE10-Nano#JTAG_load_sof_file), then srunner_cb should work and will be able to load the jic or rpd file into the FPGA boot flash memory.

Different revisions of the DE10-Nano board have different FPGA boot flash chips, some have EPCQ128 (use the "-128" option), some have the EPCQ64 chip (use the "-64" option). Use "srunner_cb -id" per example below to identify which flash chip is present on each specific chronobox. Note that the DE10-Nano documentation and the firmware quartus project generally refer to the EPCS64/EPCQ64 chip. The only practical difference is the use of "-128" or "-64" srunner_cb options.

cd /home/agdaq/online/chronobox_software
./srunner_cb.exe -id -64 /dev/null # identify EPCS64 flash
./srunner_cb.exe -id -128 /dev/null # identify EPCQ128 flash
./srunner_cb.exe -read -128 test.rpd # read flash contents into a file
#./srunner_cb.exe -program -128 /home/olchansk/git/chronobox_firmware/output_files/DE10_NANO_SoC_GHRD_auto.rpd # write firmware rpd file into flash
./srunner_cb.exe -program -128 /home/agdaq/online/firmware/git/chronobox_firmware/output_files/DE10_NANO_SoC_GHRD_auto.rpd
./reboot_cb.exe # reboot the fpga into the new firmware

Firmware build

• use quartus 17.1

Chronobox firmware registers

reg | rw/ro | quartus name | firmware | description
0 | ro | sof_revision_in | all | firmware revision timestamp code
0 | wo | latch_scalers_out, zero_scalers_out | all | see [[#reg_0x00_write_bits]]
1 | rw | reg1_led_out | all | DE10-Nano LED output
2 | ro | switches_in | all | read DE10-Nano switches
3 | ro | buttons_in | all | read DE10-Nano buttons
4 | rw | reg4_test | all | 32-bit read-write test register
5 | rw | flash_programmer_in, reg5_flash_programmer_out | 0x5b6de806 | 0xABCD srunner flash programmer
6 | ro | ecl_in | all | read state of ECL inputs
7 | ro | reg7_test_in | all | ???
8 | rw | scaler_addr_out, reg8_scaler_data_in | all | top 16 bits of address becomes scaler bus address, 32 bit read is the corresponding scaler data
9 | ro | lemo_in | all | read state of LEMO inputs
A | ro | gpio_in | all | read state of GPIO inputs
B | rw | regB_lemo_out | all | LEMO output data
C | rw | regC_gpio_out | all | GPIO output data
D | rw | regD_out_enable_out | all | enable output tristates: [31:24] - LEMO_OUT, [17:0] - GPIO_OUT
E | rw | regE, reconfig_out | 0x5b6de806 | FPGA reboot: write inverted firmware revision (reg0) to reboot the FPGA
F | ro | regF_input_num_in | 0x5b89e4b4 | number of chronobox inputs (to read scalers, add 1 for the clock counter)
10 | ro | reg10_fifo_status | 0x5b8de2b0 | data fifo status, see below
11 | ro | reg11_fifo_data | 0x5b8de2b0 | data fifo data, see below
12 | rw | cb_invert_a | 0x5bf7557e | invert inputs 31..0
13 | rw | cb_invert_b | 0x5bf7557e | invert inputs 63..32
14 | rw | cb_enable_le_a | 0x5bf7557e | enable TSC leading edge
15 | rw | cb_enable_le_b | 0x5bf7557e | enable TSC leading edge
16 | rw | cb_enable_te_a | 0x5bf7557e | enable TSC trailing edge
17 | rw | cb_enable_te_b | 0x5bf7557e | enable TSC trailing edge
18 | ro | fc_ext_clk_100_counter | 0x5bf7557e | external clock frequency counter 100MHz reference
19 | ro | fc_ext_clk_ext_counter | 0x5bf7557e | external clock frequency counter
1A | ro | fc_ts_clk_100_counter | 0x5bf7557e | timestamp clock frequency counter 100MHz reference
1B | ro | fc_ts_clk_ts_counter | 0x5bf7557e | timestamp clock frequency counter
1C | ro | ts_clk_pll_status | 0x5bf7557e | timestamp clock PLL status
1D | rw | cb_lemo_out_mux_ctrl | 0x5bfdc798 | 8*4 bits to control 8 LEMO output multiplexers (4 bits/16 options each mux)
1E | rw | cb_sync_mask[31:0] | NEXT | source of chronobox sync signal, low bits
1F | rw | cb_sync_mask[63:32] | NEXT | source of chronobox sync signal, high bits

reg 0x00 write bits

bit | fw revision | quartus signal | description
0 | all | | latch scalers
1 | all | | zero scalers
2 | ... | fifo_rdreq_out | fifo_rdreq_out
3 | ... | ts_clk_pll_extswitch_out | clear ts_clk_pll_extswitch_out
4 | ... | ts_clk_pll_extswitch_out | set ts_clk_pll_extswitch_out
5 | NEXT | sync_arm | arm the synchronization sequence
6 | NEXT | cb_sync | activate the synchronization

reg 0x10

Data FIFO status bits:

31: fifo_full
30: fifo_empty
29: 0
28: 0
24+4: 0
0+24: fifo_usedw

reg 0x1C

Timestamp clock PLL status bits:

31 : PLL locked
30 : PLL active clock (0=internal, 1=external
29 : external clock bad
28 : internal clock bad
27 : ts_clk_pll_extswitch
0..26 : not used

Timestamp clock

Timestamp clock is 10 MHz selectable from internal oscillator or external reference on the CLK_IN input.

CLK_IN input can be selected using 2 two-position jumpers:

• NIM input: CLK<->NIM/TTL and CLK<->NIM

To select the clock from command line, use:

• test_cb.exe intclk # select internal clock
• test_cb.exe extclk # select external clock

From software:

Chronobox* cb = ...;
cb->cb_int_clock(); # select internal clock
cb->cb_ext_clock(); # select external clock

To see current status, run "test_cb.exe clocks":

# ./test_cb.exe clocks
...
Chronobox firmware revision: 0x5bf7557e
...
clock status: ext_clk: counter 0x00cccf2a, freq 10000450.9 Hz, ts_clk: counter 0x00cccf2a, freq 10000450.9 Hz, PLL status 0xc0000000

Reported is external clock frequency, currently selected timestamp clock frequency and PLL status (register 0x1C).

Normal values for the PLL status:

• internal clock: 0x80000000
• external clock: 0xC0000000
• external clock selected, but invalid: 0x60000000
• internal clock selected, external clock invalid: 0xa0000000

Disconnected/absent/broken external clock will report ext_clk frequency zero, bit 0x20000000 in the PLL status register 0x1C.

LEMO outputs

LEMO connectors can be used as TTL level outputs:

• set the "bank a" and/or "bank b" jumpers for "output"
• set the lemo output multiplexor bits in register 0x1D
• set the "lemo output enable" bit in register 0x0D, use bits 24..31 for LEMO outputs 0..7.
• observe the corresponding LED is on or off according to the LEMO output TTL logic level

The function of each LEMO output is controlled by the lemo output multiplexor. Up to 16 different signals can be routed into each output. This is controlled by register 0x1D.

Register 0x1D multiplexor control is 32 bits organized into 8 groups of 4 bits, per each of the 8 LEMO outputs: 0x76543210, i.e. value 0x00000001 routes signal function 1 into output 0, and signal function 0 into outputs 1..7.

For each output, there are 16 possible signal functions (4 bits):

bit | firmware signal | firmware revision | signal description
0 | cb_lemo_out_reg[n] | 0x5bfdc798 | output is controlled by register 0xB bits 0..7 for outputs 0..7
1 | clk_ts | 0x5bfdc798 | output is the 10MHz timestamp clock
2 | cb_sync_out | NEXT | output the cb_sync signal
3..30 | gnd | 0x5bfdc798 | not used
31 | vcc | 0x5bfdc798 | logic level 1

front panel LEDs

The front panel LEDs are numbered 0..7 per #Chronobox_connectors

Each LED can be individually lit by setting a bit in register 0x1 bits 8..15.

If LEMO outputs are active, corresponding LEDs will show LEMO output status (logic level 1 -> LED is on, logic level 0 -> LED is off).

If LEMO inputs are active ...

Multiple chronobox connection

To operate several chronoboxes as one unit, they have to have two common signals, a clock and a sync signal.

Described is a master-slave configuration of 2 chronoboxes. Up to 2 slaves can be connected, number of LEMO outputs permitting. Alternately, slave chronoboxes can be daisy-chained for an unlimited number of slaves.

In a master-slave configuration, the master unit provides the timestamp clock and drives the sync signal. (For daisy-chained configuration, the slave can repeat the clock and sync signals into the next slave).

Electrical connection, lemo direction jumpers:

master: bank A set to "out"
slave: bank B set to "in"
slave daisy-chain: bank A set to "out"

Electrical connection, 1st slave:

master lemo out #0 -> slave clk_in (TTL mode)
master lemo out #1 -> slave lemo in #4

Electrical connection, 2nd slave:

master lemo out #2 -> slave clk_in (TTL mode)
master lemo out #3 -> slave lemo in #4

Electrical connection, daisy-chained:

slave lemo out #0 -> next slave clk_in (TTL mode)
slave lemo out #1 -> next slave lemo in #4

Clock configuration:

• master: set to internal clock or external clock
• slave: set to external clock

Sync configuration:

• master: "sync_arm" activated by software, "sync" activated by software (cb_sync_mask set to zero)
  • register 0x1D output mux to 0xXXXX'XX21 (2=output sync signal, 1=output clock)
  • register 0x0D output enable, set bits 0x3000'0000 (output enable lemo #0 clock and #1 sync)
  • register 0x1E and 0x1F, set to 0 (cb_sync external source)
• slave: "sync_arm" activated by software, "sync" activated by lemo in #4 (cb_sync_mask set bit 32+4).
  • register 0x1E set to zero
  • register 0x1F set to 0x0000'0004
  • for daisy-chain operation, set registers 0x1D and 0x0D same as master

Sync operation:

• issue a "sync_arm" command. Scalers and timestamps will be stopped. Time counter will be reset to zero, scalers will be reset to zero.
• issue a "sync" command. Scalers and timestamps will be started at the same time in all units.

FIFO data format

• 0x8ntttttt: TSC data, 24 bits "tttttt" of timestamp, 7 bits "nn" of channel number, top bit set to 1. Low bit of "t" indicates 0=leading edge, 1=trailing edge.
• 0xffTTmmmm: timestamp wrap around marker: "TT" is the top 8 bits of the timestamp, "mmmm" increments for each marker
• 0xfe00nnnn: scaler data, following "nnnn" words are the latched scalers

timestamp wrap around marker

The timestamp data is only 24 bits, to allow timestamping with longer time range, wrap around markers are added to the data stream.

For input signals that arrive close to the time of timestamp wrap around, there is ambiguity in the ordering of the data fifo: does the wrap around marker or the signal timestamp show up first? For example for rare signals, one can see this:

wrap 1
wrap 2
timestamp 0x00000003
wrap 3
wrap 4

does the hit belong with wrap marker 2 (written to the fifo just after wrap marker 2) or with marker 3 (written to the fifo just before wrap marker 3)?

To remove this ambiguity, additional markers are written to the data stream half way between the wrap arounds, making it obvious that the signal arrived right after wrap marker 3 (but was written to the FIFO before the marker):

wrap 1 0x00
wrap 1 0x80
wrap 2 0x00
wrap 2 0x80
timestamp 0x00000003
wrap 3 0x00
wrap 3 0x80

test_cb.exe

test_cb.exe is the general test program for the chronobox.

• test_cb.exe 0 # read chronobox register 0
• test_cb.exe 4 0x1234 # write to chronobox register 4
• test_cb.exe reboot # reboot the FPGA (the ARM CPU keeps running)
• test_cb.exe scalers # read all scalers in a loop
• test_cb.exe fifo # read the data fifo in a loop
• test_cb.exe intclk # select internal timestamp clock
• test_cb.exe extclk # select external timestamp clock (10MHz)
• test_cb.exe clocks # report current status of timstamp clock