PWB

Links

• https://bitbucket.org/teamalphag/pwb_rev1_firmware
• https://edev-group.triumf.ca/fw/exp/alphag/feam/rev1
• https://edev-group.triumf.ca/hw/alphag/feam/rev1
• https://ladd00.triumf.ca/daqinv/frontend/list/170 - PWB inventory database

Schematics

• File:Pwb rev0.pdf
• File:Pwb-rev1-after.pdf
• File:AlphaG Rev1.pdf

Manuals

• File:AFTER MANUAL V1.pdf AFTER SCA manual v1
• File:AFTER DataSheet1 1.pdf AFTER SCA manual v1.1
• File:AFTER SCA table2.pdf AFTER SCA channel map

Hardware

Power distribution

External power supply is:

• +5V
• +2V

Central board:

• Ext_5V - F1 - RS1 (0.1R) - Fuse_5V
• Ext_2V - F2/F3 - RS2 (0.1R) - Idig2-
• LDO3 - Fuse_5V - DIG_3.3V - selfenable - ADM7154
• LDO5 - Fuse_5V - DIG_2.5V - selfenable - ADM7154
• LDO_CLN - Fuse5V - CLN_3.3V - selfenable -
• VREG - DIG_3.3V - -2.5V SCA BIAS - selfenable - LTC1983ES6-3#TRMPBF
• LDO7 - DIG_2.5V - DDR3_VRef - enable DDR3_1P5V, DDR3_PG, DDR3_VttEN
• LDO2 - Idig2V- - DIG_1.1V - enable DIG_3.3V - LT3071
• LDO6 - Idig2V- - DDR3_1P5V - enable DIG_3.3V - LT3083

SCA Wing:

• Ext_5V - F1 - RS1 (0.3R) - Isca- - SCA_Fuse5V
• LDO1 - SCA_Fuse5V - SCA3.3V - enable ADC_PWR_EN - ADM7154
• LDO2 - SCA3.3V - ADC1.8V - selfenable - ADM7154
• LDO3 - SCA3.3V - TRN2.5V - selfenable - LT1761
• REF - SCA3.3V - 1.45V Vicm - selfenable - LM4121

Firmware

Firmware update

PWB firmware update is done using "esper-tool".

esper-tool -v upload -f file.rpd http://pwbNN update factory_rpd
esper-tool -v upload -f file.rpd http://pwbNN update file_rpd

As of PWB firmware pwb_rev1_20200706_ko:

Permission to write into the flash memory is controlled by esper variables:

• allow_write set to "false": "esper-tool upload" does a "verify": successful upload means epcq flash content is same as rpd file, error means content is not the same.
• allow_write set to "true": "esper-tool upload" updates the epcq flash from the rpd file: for each data block, read the epcq flash, compare to rpd data, if different, erase the flash block, write the flash, read the flash, if there is a mismatch (bad write), return an error.
• allow_factory_write set to "true" to enable writing to the "factory_rpd". DO NOT SET IT TO "true" UNLESS YOU MEAN TO UPDATE THE PWB BOOTLOADER. IF YOU FLASH BUM FIRMWARE or IF THERE IS A POWER OUTAGE WHILE UPDATING, THE PWB WILL BE BRICKED and has to be recovered by connecting a USB blaster.

To update firmware of multiple boards to correct version, recommended is the script "update_pwb.perl":

cd ~/online/src
./update_pwb.perl pwb12 pwb06 pwb78

Factory page and user page firmware boot

Each PWB board has 2 firmware images: boot loader (factory page) and data acquisition (user page). On power up, the FPGA loads and runs the boot loader firmware from the factory page, later the control software reboots the FPGA into the data acquisition firmware from the user page.

Because loading defective firmware can brick https://en.wikipedia.org/wiki/Brick_(electronics) the PWB, new firmware images are loaded into the user page, if anything is wrong, the PWB can still boot from the factory page and one can use the firmware update function to load a good firmware image into the user page.

The boot loader firmware in the factory page is not intended for data acquisition, it is only meant to provide three functions:

• boot enough hardware and software to communicate via the ethernet and the sata link
• firmware update (both factory page and user page)
• reboot into the user page

The boot loader firmware in the factory page is usually never updated unless absolutely needed to fix a problem with these three functions (i.e. compatibility with sata link communications).

If factory image is corrupted or the contains defective firmware and the PWB does not boot (no dhcp, no ping, no esper), the only way to unbrick it is by loading good firmware using a jtag flash programmer (usb-blaster). Connecting it requires physical access to the jtag connector on the PWB board. It is impossible when the detector is fully assembled in the experiment.

The PWB has a 32 Mbyte EPCQ flash memory chip, it is divided into 2 pages 16 Mbytes of factory page and 16 Mbytes of user page.

A complete firmware image generally contains 3 pieces:

• FPGA firmware and NIOS "feam_bootloader" software RAM image (sof file)
• NIOS "feam" software RAM image (feam.elf.flash.hex) - all the C code for esper communications and PWB data acquisition
• NIOS filesystem image (feam.webpkg.hex) - esper web pages

This is the boot sequence:

• on power-up the FPGA automatically loads the sof file from address 0 of the EPCQ flash (this is the factory image sof file)
• the FPGA is "started"
• the NIOS CPU starts executing the "feam_bootloader" C code embedded in the sof file (NIOS project hdl/software/feam_bootloader)
• feam_bootloader initializes the hardware (clocks, DDR memory, etc)
• feam_bootloader write-protects the EPCQ flash memory
• feam_bootloader copies the "feam" software from EPCQ flash to the DDR memory (after checking for correct checkum)
• feam_bootloader restarts the NIOS CPU
• the NIOS CPU starts running from DDR memory, executes the "feam" software (NIOS project hdl/software/feam)
• call main() in hdl/software/feam/src/task_init.c
• call task_init() in the same file
• infinite loop waiting for IP address via DHCP
• after have IP address, call task_esper() from hdl/software/feam/src/task_esper.c
• task_esper() creates all the esper modules, esper variables, etc,
• mod_http.c starts the mongoose web server
• PWB is open for business

This is the protection against booting corrupted firmware:

• FPGA hardware loads the sof file from epcq flash and checks for correct checksum before "starting" it
• NIOS CPU is part of the sof file, protected by sof file checksum
• NIOS feam_bootloader C code is part of the sof file, protected by the sof checksum
• feam_bootloader checks for correct signatures and checksums of the DDR memory image
• "feam" C code is protected by the checksum of the DDR memory image

NIOS terminal

$ ssh agmini@daq16
$ /opt/intelFPGA/16.1/quartus/bin/jtagconfig
1) USB-Blaster [2-1.2]
  02B030DD   5CGTFD7(B5|C5|D5)/5CGXBC7B6/..
$ /opt/intelFPGA/17.0/quartus/bin/nios2-terminal
nios2-terminal: connected to hardware target using JTAG UART on cable
nios2-terminal: "USB-Blaster [2-1.2]", device 1, instance 0
nios2-terminal: (Use the IDE stop button or Ctrl-C to terminate)
PWB Revision 1 Boot Loader
Ver 2.0  Build 357 - Wed Jun  6 15:05:35 PDT 2018
...

Flash boot loader firmware via jtag

$ ssh agmini@daq16
$ /opt/intelFPGA/16.1/quartus/bin/jtagconfig
1) USB-Blaster [2-1.2]
  02B030DD   5CGTFD7(B5|C5|D5)/5CGXBC7B6/..
$ cd ~/online/firmware/pwb_rev1
$ ls -l
$ /opt/intelFPGA/17.1/quartus/bin/quartus_pgmw
... auto detect
... load the jic file
... in menu tools->programmer, enable "unprotect device"
... start program/configure operation

Flash user page firmware via esper-tool

$ ssh agmini@daq16
$ cd online/src
$ more update_pwb.perl ### check that $fw is set to the desired firmware file
$ ./update_pwb.perl pwb06 ### or give more PWB names or give "all"

Build firmware

NOTE: quartus 16.1 should be used for jtag (jtagconfig and jtagd)

NOTE: quartus 17.0 should be used to build the PWB firmware (17.1 is not compatible)

$ ssh agmini@daq16
$ /opt/intelFPGA/17.0/nios2eds/nios2_command_shell.sh
------------------------------------------------
Altera Nios2 Command Shell [GCC 4]

Version 17.0, Build 602
------------------------------------------------
$ /opt/intelFPGA/17.0/quartus/linux64/lmgrd -c ~agmini/online/license-daq16.dat
$ cd online/firmware/git/pwb_rev1_firmware
$ git pull
$ git checkout alphag
$ git pull
$ ./scripts/compile_project.sh
$ ls -l bin/*.sof bin/*.jic bin/*.rpd
-rw-r--r-- 1 agmini alpha 12727389 Jan 24  2018 bin/feam_auto.rpd
-rw-r--r-- 1 agmini alpha 33554661 Jan 24  2018 bin/feam.jic
-rw-r--r-- 1 agmini alpha  6974754 Jan 24  2018 bin/feam.sof
$ ### feam.jic is loaded via jtag
$ ### feam_auto.rpd is loaded via esper
$ ### feam.sof is used to attach the signal tap

Scripts for building firmware

Firmware components are built in this order:

• erase previous build
• regenerate qsys
• regenerate NIOS BSP "feam_bootloader_bsp" and "feam_bsp"
• build NIOS projects "feam_bootloader" and "feam"
• build quartus sof file (with the "feam_bootloader" nios project inside)
• build jic and rpd files (with the quartus sof file and the "feam" nios project inside)
• load sof file into the FPGA using jtag (be careful about compatible "feam" nios project already in the epcq flash)
• load jic file into the epcq factory page using jtag (cycle power for them to take effect)
• load rpd file into the epcq factory or user page using esper

Use these scripts:

• scripts/compile_project.sh - build everything from scratch:
• scripts/compile_qsys.sh - regenerate the main qsys component
• scripts/compile_nios.sh - regenerate the BSPs, then same as compile_elf.sh
• scripts/compile_elf.sh - build the BSPs, "feam_bootloader" and "feam" NIOS projects
• scripts/update_elf.sh - regenerate the rpd and jic files to include the new "feam" NIOS build (keeps the old "feam_bootloader" in the sof file)
• scripts/update_bootloader.sh - also regenerate the sof file to include the new "feam_bootloader" build
• scripts/clean_quartus.sh - prepare quartus fpga sof file to build from scratch
• scripts/compile_quartus.sh - build the quartus fpga sof file incrementally (run clean_quartus.sh to build from scratch).
• scripts/load_jtag_sof.pl bin/rev1.sof - load sof file into the FPGA. this will boot the new fpga firmware, the new feam_bootloader (if update_bootloader.sh was run), but with the old "feam" already in epcq flash. (beware of incompatible sof, "feam_bootloader" and "feam" builds!)
• scripts/load_jtag_jic.pl bin/feam_rev1.jic - load jic file into the EPCQ flash, same as using the graphical flash programmer tool. (this will leave the FPGA running the quartus jtag flash loader firmware, not the PWB firmware. Cycle power for new jic file to take effect).

ESPER Variables

• Board
  • invert_ext_trig - invert trigger signal from CDM before it drives any logic (to undo incorrect signal polarity)
  • reset_nios - goggle up, them down to reset NIOS subsystem

• Signalproc
  • test_mode - ADC data is replaced with a test pattern, see test mode bits in sca_x_ch_ctrl.
  • sca_a_ch_ctrl, sca_b_ch_ctrl, sca_c_ch_ctrl, sca_d_ch_ctrl - channel control bits:

11..0 - threshold - channel suppression threshold
14..12 - ctrl_test_mode - test mode:
         0=fixed patter 0xa5a,
         1=time bin counter,
         2=time bin counter with channel number,
         3=sequential adc sample counter,
         4={ch_crossed_out,trig_pos,trig_neg,adc[8:0]},
         5={trig,adc[10:0]},
         6={ch_crossed_min,adc[10:0]}
15 - ctrl_supp_mode - channel suppression mode: 0=adc<=(baseline-threshold), 1=adc<=threshold

• Link
  • link_ctrl - sata link control. The bits are:

0 - sata_link_udp_stream_in_enable - permit data flow from sata link to OFFLOAD_SATA
1 - udp_stream_out_enable - permit sca data flow to sata link
2 - sata_to_eth_enable - permit ethernet data flow from sata link to TSE_MAC
3 - eth_to_sata_enable - permit ethernet data flow from TSE MAC to sata link
4 - enable_stop_our_tx - enable flow control: allow stop_tx
#5 - stop_our_tx - manually activate the flow control signal into link_tx
6 - enable_stop_remote_tx - enable flow control: allow send "stop_tx"
#7 - stop_remote_tx - manually activate the flow control signal into link_tx
8 - tx_test_pattern_udp - udp data is replaced by test pattern 0x11111111, 0x22222222, etc.
9 - tx_test_pattern_eth - nios-to-sata data is replaced by test pattern 0x11111111, 0x22222222, etc.
10 - udp_delay_enable - delay between udp packets, see udp_delay_value below
11 - reboot_tx - send K_REBOOT command to the sata link mate, where it has same effect as board/reset_nios.
12 - sata_to_nios_disable
13 - nios_to_sata_disable
14 - 
15 - 
16 -
24..31 - udp_delay_value - delay between udp packets (top 8 bits of a 20-bit counter)

Firmware data path

Main data path:

feam_top
|
sca_sigproc
|
sca_event_control
4 * sca_control (sca_channel.sv)
4 * channel_fifo (1024 samples)

sca_control (sca_channel.sv)
|
state machine to control SCA read and write enables
|
sca_write_control (what does it do?!?)
sca_read_control

sca_read_control
|
state machine to store ADC samples into per-channel FIFOs (channel_fifo)
selector to replace ADC samples with a test pattern
79*sca_trig_one -> ch_crossed_out - channel hit detector

sca_event_control
|
event_fifo (hdl/mf/event_descriptor_fifo)
state machine to take ADC data from 4 per-channel FIFOs (channel_fifo) and store it into DDR memory (write addr increment is 510*8 (number of samples)*8 = 4080).
state machine to read transposed ADC data from DDR memory (read addr increment is 8+8=16 - 2 samples * 2 bytes * 4 sca = 16) and create 4 per-sca data streams

data stream:
4 per-sca data streams from sca_event_control -> sca_sigproc(event_dat)
|
packet_chunker (hdl/lib/packet_chunker.sv) (4*event_dat -> event_segment_dat) - multiplex 4 data streams into 1 stream of UDP-sized packets
|
packet_length_prepender (hdl/lib/packet_length_prepender.sv)  (event_segment_dat -> udp_event_val_out)
|
udp_event_val -> info qsys (udp_stream_sca) and into sata link (mux channel 2).

Internal test pulser

Documentation from Bryerton:

---------- Forwarded message ---------
From: Bryerton Shaw <bryerton@triumf.ca>
Date: Tue, 24 Apr 2018 at 16:18
Subject: RE: PWB/FEAM internal test pulse
To: Lars Martin <lmartin@triumf.ca>
Cc: Andrea Capra <acapra@triumf.ca>


Hi Lars (and Andrea),

In the new(er) firmware the test pulse code has been modified slightly to
fit in better with the upgraded trigger system. I can add an option to
allow it to be triggered by the external trigger (currently it has an
interval option, or you can hit the manual trigger to fire it, if it’s
enabled)

The *options for the test pulser* are under the *signalproc module*:

*test_pulse_ena*

An array of four booleans, that control turning on the test pulse for each
SCA. If this is False, the other options below will do nothings for that SCA



*test_pulse_interval_ena*

An array of four booleans, that control turning on the interval trigger for
each SCA



*test_pulse_interval*

An unsigned 32-bit value that marks how many 16ns clock cycles occur
between interval triggers. Only one value, as there is only one interval
timer.



*test_pulse_wdt*

An array of four 16-bit unsigned integers that control the width of the
pulse for each SCA when the trigger occurs.



Now, there is a second set of options to control the delay of the *test
pulse interval trigger*. These options are located under the *trigger
module*




*intp_trig_ena*

Boolean value enabling/disabling the interval test pulse trigger.



*intp_trig_delay*

Unsigned 32-bit integer controlling the trigger delay, which is the time
between when the trigger is requested, and when the trigger actually fires.



You can setup the interval trigger and use that to create event packets
with the test pulse on set intervals. If the interval trigger is
insufficient, I can create another option, which is to fire the test pulse
for enabled SCAs when the external trigger goes. If so, I’ll add this
option in the *signalproc *module for you:



*test_pulse_on_ext*

An array of four Booleans, that control if the test pulse for that SCA
fires on external trigger.




Remember, in order to use the SCA test pulse at all, you must also set the
SCA to test mode. You can do this by going into the *sca0-3 modules* and
setting *test* to *functionality *(the value is *3* if you want to set it
via esper-tool)



Btw currently the SCAs are hard-coded to have all 72 channels receive the
test pulse on firing, and the option to de-select/select which channels to
fire is not offered. Would you like this ability? And if so do you want it
to set all four SCAs the same, or have an option for all 288 channels?



The waveform viewer on the webpage was removed for technical reasons, and
so all viewing is done via the MIDAS DAQ. There are some preliminary tools
I’ve created to verify the event packets and their waveforms, but I
wouldn’t call them easy to setup or use at the moment. (Or very  useful for
fine comparisons). If KO doesn’t have a way of saving the events to disk, I
could deliver something that captured and saved waveforms quite quickly.

Regards,
Bryerton


*From:* Lars Martin
*Sent:* April 23, 2018 5:10 PM
*To:* Bryerton Shaw <bryerton@triumf.ca>
*Cc:* Andrea Capra <acapra@triumf.ca>
*Subject:* PWB/FEAM internal test pulse



Hi Bryerton,

Andrea and I would like to check the response function of the PWBs. For
that Daryl suggested to first look at the internal test pulse.

Can you point us to what settings are required to make that go? And to use
the external trigger if we want to acquire the data?

Ideally we would acquire Midas data from that, but it's unclear if KO's
frontend allows for that right now. Can we access the data with Esper
instead?

Finally, we haven't had the waveform display on the web page for a while,
but Daryl said you guys use it. Are we using a different page somehow?



Hope all is well with the baby, cheers,

Lars

TODO

• IMPOSSIBLE (ext clock only connected to clock cleaner) add frequency counter for the external clock (62.5 MHz)
• DONE add random delay before memtest
• DONE add 4 more bits to udp_delay
• DONE fix crash on boot if run is active and triggers arrive (check that trigger is off on boot) (do not set trigger/enable_all, ext_trig_ena, inp_trig_ena, signal_proc/force_run to 1 after boot)
• DONE (kludge jtag uart write()) increase size of jtag uart output buffer, make it unlimited if possible.
• DONE (kludge jtag uart write()) on dhcp timeout, reboot the fpga - reboot of nios does not clear the jtag uart, which becomes full and nios stops.
• DONE add NIOS watchdog timeout for fectrl
• DONE do not use external clock until instructed by fectrl (clock cleaner default is pin select mode, default pin mode is input with pull down, so clock 0 is selected, this is the ext clock. but we can drive clock select pins from the fpga).
• DONE change clock cleaner power up: use the 2 lines status_clkin0 and status_clkin1 to tell clock cleaner to use the internal clock (clkin2) - from the FPGA drive first line low, second line high. Also change all NIOS code that programs the clock cleaner to keep these two lines in the "input" mode. Current code switches them to "output" mode.
• DONE clock cleaner power up state is wrong for both choices of clock: 62.5 MHz ext clock and 125 MHz internal clock, see explanation in feam_top.sv. It looks like I need to drive status_clkin0 and status_clkin1 to "holdover" mode, both logic level 1. Currently status_clkin0 is used as SPI read line, this has to be moved to a different line first.
• add watchdog timeout that works in factory mode ("remote update" watchdog only works in user mode)
• enable HTTP pipelining. it looks like it should work, but it does not. very old version of mongoose web server.
• reset everything if it looks like NIOS TCP/IP code has crashed, but enough code is still running to prevent tripping of the fpga-remote-update watchdog timer. such crash tends to happens when the sata link mate is rebooted and incomplete data arrives from the disrupted ethernet data path.

ZZZ

ZZZ