View source for VME-GRIF-ADC16-Rev1

= VME-GRIF-ADC16-Rev1 =

= Links =

* https://bitbucket.org/expalpha/adc_firmware/overview - firmware on bitbucket (current version)
* https://bitbucket.org/expalpha/adc_maxv_firmware - MaxV firmware on bitbucket (current version)
* https://edev-group.triumf.ca/hw/vme/grif16/rev1 - GRIF-16 rev 1 on gitlab
* https://edev-group.triumf.ca/hw/fmc/adc32/rev0 - FMC-ADC32-Rev0 on gitlab
* https://edev-group.triumf.ca/hw/fmc/adc32/rev1 - FMC-ADC32-Rev1 on gitlab
* https://edev-group.triumf.ca/fw/exp/alphag/alpha16/rev1 - firmware on gitlab (out of date)
* https://edev.triumf.ca/project/edev/vme/edevel00212 - project page on redmine (dead link)
* https://edev.triumf.ca/projects/edevel00212/repository/entry/branch/rev1/ALTIUM/GRIF-ADC16%20Rev1.pdf - rev1 schematics on redmine (dead link)

= Front panel connectors =

From top to bottom:

* SFP connector - 1GigE fiber/copper/DAC
* 4 LEDs
* eSATA - clock+trigger input
* FMC daughter card slot
* 2 LEMO
* 16 MCX analog inputs

= Onboard switches and connectors =

* FP_SW1
* FP_SW2
* SEL1 rotary switch 0..F - MOD-SEL20..23
* SEL2 rotary switch 0..F - MOD-SEL16..19
* JTAG FPGA
* JTAG MAXV
* "reset" button
* SD card slot
* "Display" connector
* 16x 3 position switches for gain and input (front/back) selection

= LEMO connectors =

<pre>
|
|
|LEMO1A|LEMO1B|
|
|
</pre>

LEMO connectors are controlled by two position switches FP_SW1 (LEMO1A) and FP_SW2 (LEMO1B):

<pre>
|
|      ---XX-
| left |..XX| right
|      ---XX-
|
</pre>

* right switch position (next to mark): DAC output
* left switch position (opposite from mark): NIM input

= Onboard LEDs =

<pre>
|
|
|0|1|2|3|
|
|
</pre>

* FP_LED0 - "clock_synched" (from qsys)
* FP_LED1 - "led_trigger_input" (same as "trigger_input_adc")
* FP_LED2 - "run active" (same as "force_run")
* FP_LED3 - "led_link" (from qsys)

= Onboard thermometers =

* Temperature readout chip: LTC2983CLX#PBF
* Thermometers: 9x RTD1..RTD9 type PT100. The first 8 are on the back of the PCB next to the analog amps, the last one is on the front next to the FPGA.
<pre>
RTD1 sensor_temp[0] opamp 1-3
RTD2 [1] amp 2-4
RTD3 [2] 5-7
RTD4 [3] 6-8
RTD5 [4] 9-11
RTD6 [5] 10-12
RTD7 [6] 13-15
RTD8 [7] 14-16
RTD9 [8] between the FPGA and U11-U15
</pre>

= Analog gain and input selector switches =

The switches labeled SW1..SW16 (right side) are the input selector switches: default position (up; next to the dot) is "rear vme connector", reverse position (down) is "front panel MCX connector".

The switches without onboard label (left side; label on the schematics is "SW1") are the gain selection switches: default position (down; next to the dot) is "gain x4", reverse position (up) is "gain x1".

= Onboard hardware =

* 2-output DAC: Maxim MAX5877EGK+D (14 bit, 250 Msps), output driver TI OPA 2690IDG4, range +/- 1V open, +/- 0.5V into 50 Ohm
* I2C MAC chip: Microchip tech 24AA02E48T-I/OT, I2C addr 0xA0, MAC_SCL, MAC_SDA
* FPGA Boot flash: Micron Serial NOR flash memory: N25Q00AA13GSF40G
* 4x 100 Ms/s digitizers: AD9253BCPZ-105, Quad, 14-Bit, 80 MSPS/105 MSPS/125 MSPS Serial LVDS 1.8 V Analog-to-Digital Converter, https://bitbucket.org/expalpha/adc_firmware/src/alphag/docs/ad9253.pdf

= Board schematics =

[[:File:GRIF-ADC16_Rev1.pdf]]

= ALPHA-g board configuration =

* rotary switches SEL1, SEL2 - set to zero
* FP_SW1 set left (NIM input)
* FP_SW2 set right (DAC output)
* analog input switches: default position (next to mark): from left to right: down, up, down, up, etc
* FMC connector: FMC-ADC32 Rev1.1 module (rev0 can be used as a sata connector, rev1 has some adc channels miswired internally) 
* FMC connector: FMC-MiniSAS module cable plugs into the bottom connector

= Firmware =

* https://bitbucket.org/expalpha/adc_firmware/overview
* use quartus 17.0.2

= ESPER variables =

{| class="wikitable"
! name !! RR !! Type !! Rev !! Description
|-
| ag.adc16_threshold || RW || INT16 || x || trigger discriminator threshold for the 100MHz ADCs
|-
| ag.adc32_threshold || RW || INT16 || x || trigger discriminator threshold for the 62.5MHz ADCs
|-
| ag.adc16_bits || RO || UINT16 || x || Output of the 16x 100MHz ADC trigger discriminators, goes into sas_bits (16 bits)
|-
| ag.adc32_bits || RO || UINT32 || x || Output of the 32x 62.5MHz ADC trigger discriminators, goes into sas_bits (32 bits)
|-
| ag.adc16_counter || RO || UINT32 || 2020 || Counter for the 100MHz ADC trigger discriminators
|-
| ag.adc32_counter || RO || UINT32 || 2020 || Counter for the 62.5MHz ADC trigger discriminators
|-
| ag.dac_data || RW || UINT32 || x || DAC data, read more [[#ag.dac_data]]
|-
| ag.dac_ctrl || RW || UINT32 || x || DAC control, read more [[#ag.dac_ctrl]]
|-
| ag.dac_ctrl_a || RW || UINT32 || 2020 || DAC control, read more [[#ag.dac_ctrl_a]]
|-
| ag.dac_ctrl_b || RW || UINT32 || 2020 || DAC control, read more [[#ag.dac_ctrl_b]]
|-
| ag.dac_ctrl_c || RW || UINT32 || 2020 || DAC control, read more [[#ag.dac_ctrl_c]]
|-
| ag.dac_ctrl_d || RW || UINT32 || 2020 || DAC control, read more [[#ag.dac_ctrl_d]]
|-
| ag.adc16_sthreshold || RW || INT16 || 2020 || data suppression threshold for the 100MHz ADCs
|-
| ag.adc32_sthreshold || RW || INT16 || 2020 || data suppression threshold for the 62.5MHz ADCs
|-
| ag.ctrl_a || RW || UINT32 || 2020 || 100MHz ADC sp control, read more [[#ag.ctrl_ab]]
|-
| ag.ctrl_b || RW || UINT32 || 2020 || 62.5MHz ADC sp control, read more [[#ag.ctrl_ab]]
|-
| ag.ctrl_c || RW || UINT32 || 2020 || alphag block control, read more [[#ag.ctrl_c]]
|-
| ag.ctrl_d || RW || UINT32 || 2020 || control of adc16 and adc32 serializer and phase matching fifo, read more [[#ag.ctrl_d]]
|-
| ag.ctrl_e || RW || UINT32 || 2020 || control, read more [[#ag.ctrl_e]]
|-
| ag.ctrl_f || RW || UINT32 || 2020 || control, read more [[#ag.ctrl_f]]
|-
| ag.stat_a || RO || UINT32 || 2020 || 100MHz ADC sp status, read more [[#ag.stat_ab]]
|-
| ag.stat_b || RO || UINT32 || 2020 || 62.5MHz ADC sp status, read more [[#ag.stat_ab]]
|-
| ag.stat_c || RO || UINT32 || 2020 || status of adc16 data test pattern and alphag block status, read more [[#ag.stat_c]]
|-
| ag.stat_d || RO || UINT32 || 2020 || status of adc32 data test pattern, read more [[#ag.stat_d]]
|}

== ag.dac_data ==

{| class="wikitable"
! bits !! Rev !! Description
|-
| 15..0 || x || DAC output amplitude, +/-8000. DAC is 14 bit, 1 bit for sign, remaining 13 bits is 0x1FFF = 8192.
|-
| 31..16 || x || DAC pulse baseline
|}

== ag.dac_ctrl ==

{| class="wikitable"
! bits !! Rev !! Quartus name !! Description
|-
| 0 || x || ~DAC_PD || inverted "DAC power down", 1=enable the DAC, 0=power down the DAC
|-
| 1 || x || DAC_SELIQ || select one of the DAC outputs, set to 0 to use the right hand LEMO output
|-
| 2 || x || DAC_XOR || invert the dac data, set to 0
|-
| 3 || x || DAC_TORB || select 1-complement data format, set to 1
|-
| 4 || x || pulser_enable || 0: dac_data drives the DAC directly, 1: dac_data is gated by the eSATA SYNC signal (DAQ trigger)
|-
| 5 || x || ramp_enable || DAC is ramped using linear ramp
|-
| 7..6 || x || unused || should be set to zero
|-
| 15..8 || x || top_len || time between ramp up and ramp down in DAC samples. time = 16ns*(top_len+2)
|-
| 23..16 || x || d_down || ramp down rate * 64, DAC counts per 1 DAC sample (16ns)
|-
| 31..24 || x || d_up || ramp up rate * 64, DAC counts per 1 DAC sample (16ns)
|}

== ag.ctrl_ab ==

ag.ctrl_a and ag.ctrl_b have the same function, except one controls
the 100MHz ADCs, the other one controls the 62.5MHz ADCs.

{| class="wikitable"
! bits !! Rev !! Quartus name !! Description
|-
| 0 || 2020 || ch_ctrl_supp_enable || enable data suppression
|-
| 1 || 2020 || ch_ctrl_force_keep || set data suppression keep_bit
|-
| 15..2 || x || unused || should be set to zero
|-
| 27..16 || 2020 || ch_ctrl_supp_keep_more || additional ADC samples to keep at the end of the waveform
|-
| 30..28 || x || unused || should be set to zero
|-
| 31 || 2020 || ch_ctrl_reset || reset the sp16 and sp32 blocks, if both reset bits are set, reset data path mux and fifos in the top-level block
|}

== ag.stat_ab ==

ag.stat_a and ag.stat_b have the same function, except one controls
the 100MHz ADCs, the other one controls the 62.5MHz ADCs.

{| class="wikitable"
! bits !! Rev !! Quartus name !! Description
|-
| 31..0 || x || unused || read zero
|}

== ag.ctrl_c ==

control of the alphag block

{| class="wikitable"
! bits !! Rev !! Quartus name !! Description
|-
| 0 || 2020 || esata_clk invert || invert esata_clk going into sas_bits to the TRG
|-
| 1 || 2020 || esata_trig invert || invert esata_trig going into sas_bits to the TRG
|-
| 2 || 2020 || nim_clk invert || invert nim_clk going into sas_bits to the TRG
|-
| 3 || 2020 || nim_trig invert || invert nim_trig going into sas_bits to the TRG
|-
| 31..4 || x || unused || should be set to zero
|}

== ag.ctrl_d ==

control of the adc16 and adc32 serializer, phase matching fifo and pattern alignement

{| class="wikitable"
! bits !! Rev !! Quartus name !! Description
|-
| 3..0 || 2020 || adc32_aligner_phff_sel || 1 clock delay for reading phase matching fifo, 1 bit per adc, for aligning together test patterns from all 4 adcs
|-
| 11..4 || x || unused || should be set to zero
|-
| 12 || 2020 || adc32_aligner_clk_sel || select inverted (0) or normal (1) clock for reading the phase matching fifo
|-
| 13 || 2020 || adc32_aligner_sync || reset the phase matching fifo (does nothing useful)
|-
| 14 || 2020 || reset_serdes_adc32 || reset the serializer
|-
| 15 || 2020 || unused || should be set to zero, reserved for a reset signal
|-
| 31..16 || 2020 || adc16_xxx || same signals repeat for the adc16 section
|}

== ag.stat_c ==

status of the adc16 serializer and phase matching fifo and status of alphag block

{| class="wikitable"
! bits !! Rev !! Quartus name !! Description
|-
| 15..0 || 2020 || adc16_wf_pattern4_ok_bits || see description of pattern bits in ag_ctrl_d below
|-
| 27..16 || x || unused || read zero, reserved for status of alphag block
|-
| 28 || 2020 || adc16_wf_pattern_ok || adc16 alignment ok (for use with pattern 5)
|-
| 29 || 2020 || adc32_wf_pattern_ok || adc32 alignment ok (for use with pattern 5)
|-
| 30 || 2020 || adc16_wf_pattern4_ok || adc16 pattern 4 alignment is good (0xFFFF in ag_stat_c)
|-
| 31 || 2020 || adc32_wf_pattern4_ok || adc32 pattern 4 alignment is good (0xFFFFFFFF in ag_stat_d)
|}

== ag.stat_d ==

status of the adc32 data test pattern

{| class="wikitable"
! bits !! Rev !! Quartus name !! Description
|-
| 31..0 || 2020 || adc32_wf_pattern4_ok_bits || status of ADC test pattern 4 (alternating 0xAAAA and 0x5555). if test pattern is off, should read 0. otherwise, if all channels are correctly aligned, should read 0xFFFFFFFF. If one or more ADCs are misaligned (off by one clock) corresponding 8 bits will read "00", i.e. 0xFF00FFFF if the 3rd ADC is misaligned. this misalignement is corrected by resetting the ADC and the serdes (via SPI and bit 14 in ag_ctrl_d), by changing the phase matching fifo clock polarity (bit 12 in ag_ctrl_d) or by enabling the 1-clock delay in bits 3..0 of ag_ctrl_d. beware that using only pattern 4 one can become off by 2 clocks, use pattern 5 to guard against this). If one individual channel is out of alignment, it's bit will be zero, i.e. 0xFFFFFF7F if last channel of first ADC is misaligned. this is corrected by reset of ADC, reset of serdes or change of phase matching fifo clock polarity.
|}

= Data format =

== Version 1 ==

* note: the first two bytes of data (packet_cnt[15:0]) are removed from the UDP packet by the udp_payload_inserter block.
* note: CRC16 is not implemented
* PACKET_TYPE = 1
* PACKET_VERSION = 1
* stat_trig_accepted is the trigger counter
* hw_id is the ethernet MAC address
* fw_id is the sof file build timestamp
* r_timestamp is 64 bits wide trigger timestamp
* r_timestamp runs at 125 MHz
* trigger_offset: trig_delay-wf_trig_point. trig_delay is from ESPER ch_trig_delay. trig_point is from ESPER ch_trig_point
* module_id: from ESPER board/module_id
* ch_type: from ESPER adc16&fmc32/ch_type
* CH_ID: channel number 0..15 for adc16 and 0..31 for fmc32
* sample_cnt: from ESPER ch_stop_point
* crc_value (not implemented)

<pre>
S1_HEADER0: { packet_cnt[15:0], PACKET_TYPE[7:0], PACKET_VERSION[7:0] }; // determine total size (used by UDP offloader, does not appear in final packet!
S2_HEADER1: { stat_trig_accepted[15:0], hw_id[47:32] }; // MSB MAC Address
S3_HEADER2: { hw_id[31:0] };	// LSB MAC Address
S4_HEADER3: { fw_id[31:0] }; // build timestamp (acts as FW version)
S5_HEADER4: { {(32-(SZ_TIMESTAMP-32)){1'b0}},r_timestamp[(SZ_TIMESTAMP-1) : 32] };
S6_HEADER5: { r_timestamp[31:0] };
S7_HEADER6: { trigger_offset[31:0] };
S8_HEADER7: { module_id[7:0], ch_type, CH_ID[6:0], 4'h0, sample_cnt[11:0] };			
S11_DATA:   { wf_data[15:0], wf_data[31:16] }; // data words
//S14_CRC16:{ crc_value, 16'h0 }; // not implemented
</pre>

== Version 2 ==

* PACKET_TYPE = 1
* PACKET_VERSION = 2
* all other header words are the same as version 1 data.

Differences from version 1 data:
* after adding the udp length prepender block from the PWB project, all data words generated by the state machine are sent out. in version 1 data, the first 2 bytes (udp packet length) were "eaten" by the udp data offloader block.
* the last word of data contains data suppression information. in version 1 data it contains normal adc samples.

* ch_ctrl_supp_enable: 0=data suppression disabled, 1=enabled
* keep_bit: at least one adc data sample is above the data suppression threshold
* keep_last: last data word where adc data was above data suppression threshold. number of adc sample words sent is keep_last + keep_more. keep_more is set in registers ag_ctrl_a and ag_ctrl_b.
* supp_baseline: waveform baseline computed from the first 64 adc samples.

<pre>
S1_HEADER0: { PACKET_TYPE[7:0], PACKET_VERSION[7:0], PACKET_TYPE[7:0], PACKET_VERSION[7:0] };
S2_HEADER1: // same as version 1 data
S3_HEADER2: // ..
S4_HEADER3: // ..
S5_HEADER4: // ..
S6_HEADER5: // ..
S7_HEADER6: // ..
S8_HEADER7: // ..
S11_DATA:    { wf_data[15:0], wf_data[31:16] };
S12_FOOTER0: { wf_data[15:0], wf_data[31:16] };
S13_FOOTER1: { 1'b1, 1'b1, ch_ctrl_supp_enable, keep_bit, keep_last[11:0], supp_baseline[15:0] };
</pre>

== Version 3 ==

* PACKET_TYPE = 1
* PACKET_VERSION = 3
* if keep_bit is 0, only the first 3 header words and the footer are sent out.
* the meaning of all data fields is the same as version 1 and version 2 data.

<pre>
S1_HEADER0: { PACKET_TYPE[7:0], PACKET_VERSION[7:0], stat_trig_accepted[15:0]};
S2_HEADER1: { module_id[7:0], ch_type, CH_ID[6:0], 4'h0, sample_cnt[11:0] };
S3_HEADER2: { r_timestamp[31:0] };
S4_HEADER3: { 16'h0000, hw_id[47:32] }; // MSB MAC Address
S5_HEADER4: { hw_id[31:0] };	// LSB MAC Address
S6_HEADER5: { {(32-(SZ_TIMESTAMP-32)){1'b0}},r_timestamp[(SZ_TIMESTAMP-1) : 32] };
S7_HEADER6: { trigger_offset[31:0] };
S8_HEADER7: { fw_id[31:0] }; // build timestamp (acts as FW version)
S11_DATA:   { wf_data[15:0], wf_data[31:16] }; // data words
S13_FOOTER1: { 1'b1, 1'b1, ch_ctrl_supp_enable, keep_bit, keep_last[11:0], supp_baseline[15:0] };
</pre>

= FMC modules =

== FMC-ADC32-Rev0 ==

* !!! DO !!! NOT !!! USE !!!
* project page: https://edev-group.triumf.ca/hw/fmc/adc32/rev0
* schematics: [[:File:FMC_32_Channel_ADC_Rev0.pdf]]

== FMC-ADC32-Rev1 ==

* !!! DO !!! NOT !!! USE !!!
* project page: https://edev-group.triumf.ca/hw/fmc/adc32/rev1
* schematics: [[:File:FMC_ADC32_Rev1.pdf]]

== FMC-ADC32-Rev1.1 ==

* project page: https://edev-group.triumf.ca/hw/fmc/adc32/rev1_1
* schematics: [[:File:FMC ADC32 Rev1_1.pdf]]
* AD9249 digitizer: https://bitbucket.org/expalpha/adc_firmware/src/alphag/docs/AD9249.pdf

== FMC-DualMiniSas-Rev1 ==

* project page: https://edev.triumf.ca/project/edev/fmc-modules/edevel00109
* schematics: [[:File:FMC_-_MiniSAS_x_2_Rev1.pdf]]

== FMC-SfpMiniSasEsata-Rev2 ==

(note: not used on the GRIF16, this information is to be moved to the ALPHA-T page)

* project page: https://edev.triumf.ca/project/edev/fmc-modules/edevel00162
* schematics: [[:File:FMC_-_SFP_and_Mini-SAS_Interface_-_Rev2.PDF]]
* ethernet mac chip: 24AA02E48T-I/OT

= TODO =

* (DONE) add esper block from pwb to show ethernet pause frames
* (DONE) implement waveform suppression
* (DONE) add esper counters for adc16 and adc32 discriminator grand-or
* (DONE) enable HTTP "connection-keep-alive"
* (DONE) add more control and status registers in the "AG" module
* (DONE) fix problem with adc16 channels not sending any data, fpga reboot does not fix this (needs power cycle to fix it?)
* (DONE) investigate counter for: Dropped Triggers Due to Full [cnt_trig_dfull] [0]
* (DONE) see this: all adc16 trigger counters stop after 63 events, except for one that keeps counting, I suspect the 16->1 packet mux is getting stuck?
* (DONE) investigate: when approaching 100 Mbytes/sec 1gige ethernet limit, channels start getting stuck in the "dropped due to busy" state until remaining channels can still send data, "dropped due to full" never truely increment as one would expect in this situation.
* verify ADC serdes receive clock is correctly defined, verify signals after serdes is not tangled with non-receive-clock clocks.
* simplify code between ADC serdes and phase fifo, replace 7-to-14 bit shift register with a mux.
* update DAC control
* connect grand-or of trigger discriminators to DAC output (with a 48-bit channel-enable mask)
* verify that trigger discriminator fires on both positive and negative pulses (positive and negative thresholds)
* add provision for coded trigger signal
* update TRG link
* verify that ADC SYNC works
* add more status registers in the "AG" module
* add independant pulser for the DAC output. program the firing rate, then start immediately or after first esata "sync" signal

= ZZZ =

ZZZ