PWB

Links

• https://bitbucket.org/teamalphag/pwb_rev1_firmware
• https://edev-group.triumf.ca/fw/exp/alphag/feam/rev1
• https://edev-group.triumf.ca/hw/alphag/feam/rev1
• https://ladd00.triumf.ca/daqinv/frontend/list/170 - PWB inventory database

Schematics

• File:Pwb rev0.pdf
• File:Pwb-rev1-after.pdf
• File:AlphaG Rev1.pdf

Manuals

• File:AFTER MANUAL V1.pdf AFTER SCA manual v1
• File:AFTER DataSheet1 1.pdf AFTER SCA manual v1.1
• File:AFTER SCA table2.pdf AFTER SCA channel map

Firmware

Firmware update

PWB firmware update is done using "esper-tool".

esper-tool -v upload -f file.rpd http://pwbNN update factory_rpd
esper-tool -v upload -f file.rpd http://pwbNN update file_rpd

As of PWB firmware pwb_rev1_20200706_ko:

Permission to write into the flash memory is controlled by esper variables:

• allow_write set to "false": "esper-tool upload" does a "verify": successful upload means epcq flash content is same as rpd file, error means content is not the same.
• allow_write set to "true": "esper-tool upload" updates the epcq flash from the rpd file: for each data block, read the epcq flash, compare to rpd data, if different, erase the flash block, write the flash, read the flash, if there is a mismatch (bad write), return an error.
• allow_factory_write set to "true" to enable writing to the "factory_rpd". DO NOT SET IT TO "true" UNLESS YOU MEAN TO UPDATE THE PWB BOOTLOADER. IF YOU FLASH BUM FIRMWARE or IF THERE IS A POWER OUTAGE WHILE UPDATING, THE PWB WILL BE BRICKED and has to be recovered by connecting a USB blaster.

To update firmware of multiple boards to correct version, recommended is the script "update_pwb.perl":

cd ~/online/src
./update_pwb.perl pwb12 pwb06 pwb78

Factory page and user page firmware boot

Each PWB board has 2 firmware images: boot loader (factory page) and data acquisition (user page). On power up, the FPGA loads and runs the boot loader firmware from the factory page, later the control software reboots the FPGA into the data acquisition firmware from the user page.

Because loading defective firmware can brick https://en.wikipedia.org/wiki/Brick_(electronics) the PWB, new firmware images are loaded into the user page, if anything is wrong, the PWB can still boot from the factory page and one can use the firmware update function to load a good firmware image into the user page.

The boot loader firmware in the factory page is not intended for data acquisition, it is only meant to provide three functions:

• boot enough hardware and software to communicate via the ethernet and the sata link
• firmware update (both factory page and user page)
• reboot into the user page

The boot loader firmware in the factory page is usually never updated unless absolutely needed to fix a problem with these three functions (i.e. compatibility with sata link communications).

If factory image is corrupted or the contains defective firmware and the PWB does not boot (no dhcp, no ping, no esper), the only way to unbrick it is by loading good firmware using a jtag flash programmer (usb-blaster). Connecting it requires physical access to the jtag connector on the PWB board. It is impossible when the detector is fully assembled in the experiment.

The PWB has a 32 Mbyte EPCQ flash memory chip, it is divided into 2 pages 16 Mbytes of factory page and 16 Mbytes of user page.

A complete firmware image generally contains 3 pieces:

• FPGA firmware and NIOS "feam_bootloader" software RAM image (sof file)
• NIOS "feam" software RAM image (feam.elf.flash.hex) - all the C code for esper communications and PWB data acquisition
• NIOS filesystem image (feam.webpkg.hex) - esper web pages

This is the boot sequence:

• on power-up the FPGA automatically loads the sof file from address 0 of the EPCQ flash (this is the factory image sof file)
• the FPGA is "started"
• the NIOS CPU starts executing the "feam_bootloader" C code embedded in the sof file (NIOS project hdl/software/feam_bootloader)
• feam_bootloader initializes the hardware (clocks, DDR memory, etc)
• feam_bootloader write-protects the EPCQ flash memory
• feam_bootloader copies the "feam" software from EPCQ flash to the DDR memory (after checking for correct checkum)
• feam_bootloader restarts the NIOS CPU
• the NIOS CPU starts running from DDR memory, executes the "feam" software (NIOS project hdl/software/feam)
• call main() in hdl/software/feam/src/task_init.c
• call task_init() in the same file
• infinite loop waiting for IP address via DHCP
• after have IP address, call task_esper() from hdl/software/feam/src/task_esper.c
• task_esper() creates all the esper modules, esper variables, etc,
• mod_http.c starts the mongoose web server
• PWB is open for business

This is the protection against booting corrupted firmware:

• FPGA hardware loads the sof file from epcq flash and checks for correct checksum before "starting" it
• NIOS CPU is part of the sof file, protected by sof file checksum
• NIOS feam_bootloader C code is part of the sof file, protected by the sof checksum
• feam_bootloader checks for correct signatures and checksums of the DDR memory image
• "feam" C code is protected by the checksum of the DDR memory image

NIOS terminal

$ ssh agmini@daq16
$ /opt/intelFPGA/16.1/quartus/bin/jtagconfig
1) USB-Blaster [2-1.2]
  02B030DD   5CGTFD7(B5|C5|D5)/5CGXBC7B6/..
$ /opt/intelFPGA/17.0/quartus/bin/nios2-terminal
nios2-terminal: connected to hardware target using JTAG UART on cable
nios2-terminal: "USB-Blaster [2-1.2]", device 1, instance 0
nios2-terminal: (Use the IDE stop button or Ctrl-C to terminate)
PWB Revision 1 Boot Loader
Ver 2.0  Build 357 - Wed Jun  6 15:05:35 PDT 2018
...

Flash boot loader firmware via jtag

$ ssh agmini@daq16
$ /opt/intelFPGA/16.1/quartus/bin/jtagconfig
1) USB-Blaster [2-1.2]
  02B030DD   5CGTFD7(B5|C5|D5)/5CGXBC7B6/..
$ cd ~/online/firmware/pwb_rev1
$ ls -l
$ /opt/intelFPGA/17.1/quartus/bin/quartus_pgmw
... auto detect
... load the jic file
... in menu tools->programmer, enable "unprotect device"
... start program/configure operation

Flash user page firmware via esper-tool

$ ssh agmini@daq16
$ cd online/src
$ more update_pwb.perl ### check that $fw is set to the desired firmware file
$ ./update_pwb.perl pwb06 ### or give more PWB names or give "all"

Build firmware

NOTE: quartus 16.1 should be used for jtag (jtagconfig and jtagd)

NOTE: quartus 17.0 should be used to build the PWB firmware (17.1 is not compatible)

$ ssh agmini@daq16
$ /opt/intelFPGA/17.0/nios2eds/nios2_command_shell.sh
------------------------------------------------
Altera Nios2 Command Shell [GCC 4]

Version 17.0, Build 602
------------------------------------------------
$ /opt/intelFPGA/17.0/quartus/linux64/lmgrd -c ~agmini/online/license-daq16.dat
$ cd online/firmware/git/pwb_rev1_firmware
$ git pull
$ git checkout alphag
$ git pull
$ ./scripts/compile_project.sh
$ ls -l bin/*.sof bin/*.jic bin/*.rpd
-rw-r--r-- 1 agmini alpha 12727389 Jan 24  2018 bin/feam_auto.rpd
-rw-r--r-- 1 agmini alpha 33554661 Jan 24  2018 bin/feam.jic
-rw-r--r-- 1 agmini alpha  6974754 Jan 24  2018 bin/feam.sof
$ ### feam.jic is loaded via jtag
$ ### feam_auto.rpd is loaded via esper
$ ### feam.sof is used to attach the signal tap

ESPER Variables

• Board
  • invert external trigger - invert trigger signal from CDM before it drives any logic (to undo incorrect signal polarity)

• Signalproc
  • test_mode - ADC data is replaced with a test pattern, see test mode bits in sca_x_ch_ctrl.
  • sca_a_ch_ctrl, sca_b_ch_ctrl, sca_c_ch_ctrl, sca_d_ch_ctrl - channel control bits:

11..0 - threshold - channel suppression threshold
14..12 - ctrl_test_mode - test mode:
         0=fixed patter 0xa5a,
         1=time bin counter,
         2=time bin counter with channel number,
         3=sequential adc sample counter,
         4={ch_crossed_out,trig_pos,trig_neg,adc[8:0]},
         5={trig,adc[10:0]},
         6={ch_crossed_min,adc[10:0]}
15 - ctrl_supp_mode - channel suppression mode: 0=adc<=(baseline-threshold), 1=adc<=threshold

• Link
  • link_ctrl - sata link control. The bits are:

0 - sata_link_udp_stream_in_enable - permit data flow from sata link to OFFLOAD_SATA
1 - udp_stream_out_enable - permit sca data flow to sata link
2 - sata_to_eth_enable - permit ethernet data flow from sata link to TSE_MAC
3 - eth_to_sata_enable - permit ethernet data flow from TSE MAC to sata link
4 - enable_stop_our_tx - enable flow control: allow stop_tx
#5 - stop_our_tx - manually activate the flow control signal into link_tx
6 - enable_stop_remote_tx - enable flow control: allow send "stop_tx"
#7 - stop_remote_tx - manually activate the flow control signal into link_tx
8 - tx_test_pattern_udp - udp data is replaced by test pattern 0x11111111, 0x22222222, etc.
9 - tx_test_pattern_eth - nios-to-sata data is replaced by test pattern 0x11111111, 0x22222222, etc.
10 - udp_delay_enable - delay between udp packets, see udp_delay_value below
11 - 
12 - sata_to_nios_disable
13 - nios_to_sata_disable
14 - 
15 - 
16 -
24..31 - udp_delay_value - delay between udp packets in step of 4096 ns (top 8 bits of a 16-bit counter)

Firmware data path

Main data path:

feam_top
|
sca_sigproc
|
sca_event_control
4 * sca_control (sca_channel.sv)
4 * channel_fifo

sca_control
|
state machine to control SCA read and write enables
|
sca_write_control (what does it do?!?)
sca_read_control

sca_read_control
|
state machine to store ADC samples into FIFO
selector to replace ADC samples with a test pattern
79*sca_trig_one -> ch_crossed_out - channel hit detector

sca_event_control
|
event_fifo (hdl/mf/event_descriptor_fifo)
state machine to take ADC data from 4 per-channel FIFOs and store it into DDR memory (write addr increment is 510*8 (number of samples)*8 = 4080).
state machine to read transposed ADC data from DDR memory (read addr increment is 8+8=16 - 2 samples * 2 bytes * 4 sca = 16) and create 4 per-sca data streams

data stream:
4 per-sca data streams from sca_event_control -> sca_sigproc(event_dat)
|
packet_chunker (hdl/lib/packet_chunker.sv) (4*event_dat -> event_segment_dat) - multiplex 4 data streams into 1 stream of UDP-sized packets
|
packet_length_prepender (hdl/lib/packet_length_prepender.sv)  (event_segment_dat -> udp_event_val_out)
|
udp_event_val -> info qsys (udp_stream_sca) and into sata link (mux channel 2).

TODO

• add frequency counter for the external clock (62.5 MHz)
• add random delay before memtest
• add 4 more bits to udp_delay
• fix crash on boot if run is active and triggers arrive (check that trigger is off on boot) (do not set trigger/enable_all, ext_trig_ena, inp_trig_ena, signal_proc/force_run to 1 after boot)
• increase size of jtag uart output buffer, make it unlimited if possible.
• on dhcp timeout, reboot the fpga - reboot of nios does not clear the jtag uart, which becomes full and nios stops.

ZZZ

ZZZ