VME-GRIF-ADC16-Rev1

Links

• https://bitbucket.org/expalpha/adc_firmware/overview - firmware on bitbucket (current version)
• https://bitbucket.org/expalpha/adc_maxv_firmware - MaxV firmware on bitbucket (current version)
• https://edev-group.triumf.ca/hw/vme/grif16/rev1 - GRIF-16 rev 1 on gitlab
• https://edev-group.triumf.ca/hw/fmc/adc32/rev0 - FMC-ADC32-Rev0 on gitlab
• https://edev-group.triumf.ca/hw/fmc/adc32/rev1 - FMC-ADC32-Rev1 on gitlab
• https://edev-group.triumf.ca/fw/exp/alphag/alpha16/rev1 - firmware on gitlab (out of date)
• https://edev.triumf.ca/project/edev/vme/edevel00212 - project page on redmine (dead link)
• https://edev.triumf.ca/projects/edevel00212/repository/entry/branch/rev1/ALTIUM/GRIF-ADC16%20Rev1.pdf - rev1 schematics on redmine (dead link)

Front panel connectors

From top to bottom:

• SFP connector - 1GigE fiber/copper/DAC
• 4 LEDs
• eSATA - clock+trigger input
• FMC daughter card slot
• 2 LEMO
• 16 MCX analog inputs

Onboard switches and connectors

• FP_SW1
• FP_SW2
• SEL1 rotary switch 0..F - MOD-SEL20..23
• SEL2 rotary switch 0..F - MOD-SEL16..19
• JTAG FPGA
• JTAG MAXV
• "reset" button
• SD card slot
• "Display" connector
• 16x 3 position switches for gain and input (front/back) selection

LEMO connectors

|
|
|LEMO1A|LEMO1B|
|
|

LEMO connectors are controlled by two position switches FP_SW1 (LEMO1A) and FP_SW2 (LEMO1B):

|
|      ---XX-
| left |..XX| right
|      ---XX-
|

• right switch position (next to mark): DAC output
• left switch position (opposite from mark): NIM input

Onboard LEDs

|
|
|0|1|2|3|
|
|

• FP_LED0 - "clock_synched" (from qsys)
• FP_LED1 - "led_trigger_input" (same as "trigger_input_adc")
• FP_LED2 - "run active" (same as "force_run")
• FP_LED3 - "led_link" (from qsys)

Onboard thermometers

• Temperature readout chip: LTC2983CLX#PBF
• Thermometers: 9x RTD1..RTD9 type PT100. The first 8 are on the back of the PCB next to the analog amps, the last one is on the front next to the FPGA.

RTD1 sensor_temp[0] opamp 1-3
RTD2 [1] amp 2-4
RTD3 [2] 5-7
RTD4 [3] 6-8
RTD5 [4] 9-11
RTD6 [5] 10-12
RTD7 [6] 13-15
RTD8 [7] 14-16
RTD9 [8] between the FPGA and U11-U15

Analog gain and input selector switches

The switches labeled SW1..SW16 (right side) are the input selector switches: default position (up; next to the dot) is "rear vme connector", reverse position (down) is "front panel MCX connector".

The switches without onboard label (left side; label on the schematics is "SW1") are the gain selection switches: default position (down; next to the dot) is "gain x4", reverse position (up) is "gain x1".

Onboard hardware

• 2-output DAC: Maxim MAX5877EGK+D (14 bit, 250 Msps), output driver TI OPA 2690IDG4, range +/- 1V open, +/- 0.5V into 50 Ohm
• I2C MAC chip: Microchip tech 24AA02E48T-I/OT, I2C addr 0xA0, MAC_SCL, MAC_SDA
• FPGA Boot flash: Micron Serial NOR flash memory: N25Q00AA13GSF40G
• 4x 100 Ms/s digitizers: AD9253BCPZ-105, Quad, 14-Bit, 80 MSPS/105 MSPS/125 MSPS Serial LVDS 1.8 V Analog-to-Digital Converter, https://bitbucket.org/expalpha/adc_firmware/src/alphag/docs/ad9253.pdf

Board schematics

File:GRIF-ADC16_Rev1.pdf

ALPHA-g board configuration

• rotary switches SEL1, SEL2 - set to zero
• FP_SW1 set left (NIM input)
• FP_SW2 set right (DAC output)
• analog input switches: default position (next to mark): from left to right: down, up, down, up, etc
• FMC connector: FMC-ADC32 Rev1.1 module (rev0 can be used as a sata connector, rev1 has some adc channels miswired internally)
• FMC connector: FMC-MiniSAS module cable plugs into the bottom connector

Firmware

• https://bitbucket.org/expalpha/adc_firmware/overview
• use quartus 17.0.2

ESPER variables

name	RR	Type	Rev	Description
ag.adc16_threshold	RW	INT16	x	trigger discriminator threshold for the 100MHz ADCs
ag.adc32_threshold	RW	INT16	x	trigger discriminator threshold for the 62.5MHz ADCs
ag.adc16_bits	RO	UINT16	x	Output of the 16x 100MHz ADC trigger discriminators, goes into sas_bits (16 bits)
ag.adc32_bits	RO	UINT32	x	Output of the 32x 62.5MHz ADC trigger discriminators, goes into sas_bits (32 bits)
ag.adc16_counter	RO	UINT32	2020	Counter for the 100MHz ADC trigger discriminators
ag.adc32_counter	RO	UINT32	2020	Counter for the 62.5MHz ADC trigger discriminators
ag.dac_data	RW	UINT32	x	DAC data, read more #ag.dac_data
ag.dac_ctrl	RW	UINT32	x	DAC control, read more #ag.dac_ctrl
ag.dac_ctrl_a	RW	UINT32	2020	DAC control, read more #ag.dac_ctrl_a
ag.dac_ctrl_b	RW	UINT32	2020	DAC control, read more #ag.dac_ctrl_b
ag.dac_ctrl_c	RW	UINT32	2020	DAC control, read more #ag.dac_ctrl_c
ag.dac_ctrl_d	RW	UINT32	2020	DAC control, read more #ag.dac_ctrl_d
ag.adc16_sthreshold	RW	INT16	2020	data suppression threshold for the 100MHz ADCs
ag.adc32_sthreshold	RW	INT16	2020	data suppression threshold for the 62.5MHz ADCs
ag.ctrl_a	RW	UINT32	2020	100MHz ADC sp control, read more #ag.ctrl_ab
ag.ctrl_b	RW	UINT32	2020	62.5MHz ADC sp control, read more #ag.ctrl_ab
ag.ctrl_c	RW	UINT32	2020	alphag block control, read more #ag.ctrl_c
ag.ctrl_d	RW	UINT32	2020	control of adc16 and adc32 serializer and phase matching fifo, read more #ag.ctrl_d
ag.ctrl_e	RW	UINT32	2020	control, read more #ag.ctrl_e
ag.ctrl_f	RW	UINT32	2020	control, read more #ag.ctrl_f
ag.stat_a	RO	UINT32	2020	100MHz ADC sp status, read more #ag.stat_ab
ag.stat_b	RO	UINT32	2020	62.5MHz ADC sp status, read more #ag.stat_ab
ag.stat_c	RO	UINT32	2020	status of adc16 data test pattern and alphag block status, read more #ag.stat_c
ag.stat_d	RO	UINT32	2020	status of adc32 data test pattern, read more #ag.stat_d

ag.dac_data

bits	Rev	Description
15..0	x	DAC output amplitude, +/-8000. DAC is 14 bit, 1 bit for sign, remaining 13 bits is 0x1FFF = 8192.
31..16	x	DAC pulse baseline

ag.dac_ctrl

bits	Rev	Quartus name	Description
0	x	~DAC_PD	inverted "DAC power down", 1=enable the DAC, 0=power down the DAC
1	x	DAC_SELIQ	select one of the DAC outputs, set to 0 to use the right hand LEMO output
2	x	DAC_XOR	invert the dac data, set to 0
3	x	DAC_TORB	select 1-complement data format, set to 1
4	x	pulser_enable	0: dac_data drives the DAC directly, 1: dac_data is gated by the eSATA SYNC signal (DAQ trigger)
5	x	ramp_enable	DAC is ramped using linear ramp
7..6	x	unused	should be set to zero
15..8	x	top_len	time between ramp up and ramp down in DAC samples. time = 16ns*(top_len+2)
23..16	x	d_down	ramp down rate * 64, DAC counts per 1 DAC sample (16ns)
31..24	x	d_up	ramp up rate * 64, DAC counts per 1 DAC sample (16ns)

ag.ctrl_ab

ag.ctrl_a and ag.ctrl_b have the same function, except one controls the 100MHz ADCs, the other one controls the 62.5MHz ADCs.

bits	Rev	Quartus name	Description
0	2020	ch_ctrl_supp_enable	enable data suppression
1	2020	ch_ctrl_force_keep	set data suppression keep_bit
15..2	x	unused	should be set to zero
27..16	2020	ch_ctrl_supp_keep_more	additional ADC samples to keep at the end of the waveform
30..28	x	unused	should be set to zero
31	2020	ch_ctrl_reset	reset the sp16 and sp32 blocks, if both reset bits are set, reset data path mux and fifos in the top-level block

ag.stat_ab

ag.stat_a and ag.stat_b have the same function, except one controls the 100MHz ADCs, the other one controls the 62.5MHz ADCs.

bits	Rev	Quartus name	Description
31..0	x	unused	read zero

ag.ctrl_c

control of the alphag block

bits	Rev	Quartus name	Description
0	2020	esata_clk invert	invert esata_clk going into sas_bits to the TRG
1	2020	esata_trig invert	invert esata_trig going into sas_bits to the TRG
2	2020	nim_clk invert	invert nim_clk going into sas_bits to the TRG
3	2020	nim_trig invert	invert nim_trig going into sas_bits to the TRG
31..4	x	unused	should be set to zero

ag.ctrl_d

control of the adc16 and adc32 serializer, phase matching fifo and pattern alignement

bits	Rev	Quartus name	Description
3..0	2020	adc32_aligner_phff_sel	1 clock delay for reading phase matching fifo, 1 bit per adc, for aligning together test patterns from all 4 adcs
11..4	x	unused	should be set to zero
12	2020	adc32_aligner_clk_sel	select inverted (0) or normal (1) clock for reading the phase matching fifo
13	2020	adc32_aligner_sync	reset the phase matching fifo (does nothing useful)
14	2020	reset_serdes_adc32	reset the serializer
15	2020	unused	should be set to zero, reserved for a reset signal
31..16	2020	adc16_xxx	same signals repeat for the adc16 section

ag.stat_c

status of the adc16 serializer and phase matching fifo and status of alphag block

bits	Rev	Quartus name	Description
15..0	2020	adc16_wf_pattern4_ok_bits	see description of pattern bits in ag_ctrl_d below
27..16	x	unused	read zero, reserved for status of alphag block
28	2020	adc16_wf_pattern_ok	adc16 alignment ok (for use with pattern 5)
29	2020	adc32_wf_pattern_ok	adc32 alignment ok (for use with pattern 5)
30	2020	adc16_wf_pattern4_ok	adc16 pattern 4 alignment is good (0xFFFF in ag_stat_c)
31	2020	adc32_wf_pattern4_ok	adc32 pattern 4 alignment is good (0xFFFFFFFF in ag_stat_d)

ag.stat_d

status of the adc32 data test pattern

bits	Rev	Quartus name	Description
31..0	2020	adc32_wf_pattern4_ok_bits	status of ADC test pattern 4 (alternating 0xAAAA and 0x5555). if test pattern is off, should read 0. otherwise, if all channels are correctly aligned, should read 0xFFFFFFFF. If one or more ADCs are misaligned (off by one clock) corresponding 8 bits will read "00", i.e. 0xFF00FFFF if the 3rd ADC is misaligned. this misalignement is corrected by resetting the ADC and the serdes (via SPI and bit 14 in ag_ctrl_d), by changing the phase matching fifo clock polarity (bit 12 in ag_ctrl_d) or by enabling the 1-clock delay in bits 3..0 of ag_ctrl_d. beware that using only pattern 4 one can become off by 2 clocks, use pattern 5 to guard against this). If one individual channel is out of alignment, it's bit will be zero, i.e. 0xFFFFFF7F if last channel of first ADC is misaligned. this is corrected by reset of ADC, reset of serdes or change of phase matching fifo clock polarity.

Data format

Version 1

• note: the first two bytes of data (packet_cnt[15:0]) are removed from the UDP packet by the udp_payload_inserter block.
• note: CRC16 is not implemented
• PACKET_TYPE = 1
• PACKET_VERSION = 1
• stat_trig_accepted is the trigger counter
• hw_id is the ethernet MAC address
• fw_id is the sof file build timestamp
• r_timestamp is 64 bits wide trigger timestamp
• r_timestamp runs at 125 MHz
• trigger_offset: trig_delay-wf_trig_point. trig_delay is from ESPER ch_trig_delay. trig_point is from ESPER ch_trig_point
• module_id: from ESPER board/module_id
• ch_type: from ESPER adc16&fmc32/ch_type
• CH_ID: channel number 0..15 for adc16 and 0..31 for fmc32
• sample_cnt: from ESPER ch_stop_point
• crc_value (not implemented)

S1_HEADER0: { packet_cnt[15:0], PACKET_TYPE[7:0], PACKET_VERSION[7:0] }; // determine total size (used by UDP offloader, does not appear in final packet!
S2_HEADER1: { stat_trig_accepted[15:0], hw_id[47:32] }; // MSB MAC Address
S3_HEADER2: { hw_id[31:0] };	// LSB MAC Address
S4_HEADER3: { fw_id[31:0] }; // build timestamp (acts as FW version)
S5_HEADER4: { {(32-(SZ_TIMESTAMP-32)){1'b0}},r_timestamp[(SZ_TIMESTAMP-1) : 32] };
S6_HEADER5: { r_timestamp[31:0] };
S7_HEADER6: { trigger_offset[31:0] };
S8_HEADER7: { module_id[7:0], ch_type, CH_ID[6:0], 4'h0, sample_cnt[11:0] };			
S11_DATA:   { wf_data[15:0], wf_data[31:16] }; // data words
//S14_CRC16:{ crc_value, 16'h0 }; // not implemented

Version 2

• PACKET_TYPE = 1
• PACKET_VERSION = 2
• all other header words are the same as version 1 data.

Differences from version 1 data:

• after adding the udp length prepender block from the PWB project, all data words generated by the state machine are sent out. in version 1 data, the first 2 bytes (udp packet length) were "eaten" by the udp data offloader block.
• the last word of data contains data suppression information. in version 1 data it contains normal adc samples.

• ch_ctrl_supp_enable: 0=data suppression disabled, 1=enabled
• keep_bit: at least one adc data sample is above the data suppression threshold
• keep_last: last data word where adc data was above data suppression threshold. number of adc sample words sent is keep_last + keep_more. keep_more is set in registers ag_ctrl_a and ag_ctrl_b.
• supp_baseline: waveform baseline computed from the first 64 adc samples.

S1_HEADER0: { PACKET_TYPE[7:0], PACKET_VERSION[7:0], PACKET_TYPE[7:0], PACKET_VERSION[7:0] };
S2_HEADER1: // same as version 1 data
S3_HEADER2: // ..
S4_HEADER3: // ..
S5_HEADER4: // ..
S6_HEADER5: // ..
S7_HEADER6: // ..
S8_HEADER7: // ..
S11_DATA:    { wf_data[15:0], wf_data[31:16] };
S12_FOOTER0: { wf_data[15:0], wf_data[31:16] };
S13_FOOTER1: { 1'b1, 1'b1, ch_ctrl_supp_enable, keep_bit, keep_last[11:0], supp_baseline[15:0] };

Version 3

• PACKET_TYPE = 1
• PACKET_VERSION = 3
• if keep_bit is 0, only the first 3 header words and the footer are sent out.
• the meaning of all data fields is the same as version 1 and version 2 data.

S1_HEADER0: { PACKET_TYPE[7:0], PACKET_VERSION[7:0], stat_trig_accepted[15:0]};
S2_HEADER1: { module_id[7:0], ch_type, CH_ID[6:0], 4'h0, sample_cnt[11:0] };
S3_HEADER2: { r_timestamp[31:0] };
S4_HEADER3: { 16'h0000, hw_id[47:32] }; // MSB MAC Address
S5_HEADER4: { hw_id[31:0] };	// LSB MAC Address
S6_HEADER5: { {(32-(SZ_TIMESTAMP-32)){1'b0}},r_timestamp[(SZ_TIMESTAMP-1) : 32] };
S7_HEADER6: { trigger_offset[31:0] };
S8_HEADER7: { fw_id[31:0] }; // build timestamp (acts as FW version)
S11_DATA:   { wf_data[15:0], wf_data[31:16] }; // data words
S13_FOOTER1: { 1'b1, 1'b1, ch_ctrl_supp_enable, keep_bit, keep_last[11:0], supp_baseline[15:0] };

FMC modules

FMC-ADC32-Rev0

• !!! DO !!! NOT !!! USE !!!
• project page: https://edev-group.triumf.ca/hw/fmc/adc32/rev0
• schematics: File:FMC_32_Channel_ADC_Rev0.pdf

FMC-ADC32-Rev1

• !!! DO !!! NOT !!! USE !!!
• project page: https://edev-group.triumf.ca/hw/fmc/adc32/rev1
• schematics: File:FMC_ADC32_Rev1.pdf

FMC-ADC32-Rev1.1

• project page: https://edev-group.triumf.ca/hw/fmc/adc32/rev1_1
• schematics: File:FMC ADC32 Rev1_1.pdf
• AD9249 digitizer: https://bitbucket.org/expalpha/adc_firmware/src/alphag/docs/AD9249.pdf

FMC-DualMiniSas-Rev1

• project page: https://edev.triumf.ca/project/edev/fmc-modules/edevel00109
• schematics: File:FMC_-_MiniSAS_x_2_Rev1.pdf

FMC-SfpMiniSasEsata-Rev2

(note: not used on the GRIF16, this information is to be moved to the ALPHA-T page)

• project page: https://edev.triumf.ca/project/edev/fmc-modules/edevel00162
• schematics: File:FMC_-_SFP_and_Mini-SAS_Interface_-_Rev2.PDF
• ethernet mac chip: 24AA02E48T-I/OT

TODO

• (DONE) add esper block from pwb to show ethernet pause frames
• (DONE) implement waveform suppression
• (DONE) add esper counters for adc16 and adc32 discriminator grand-or
• (DONE) enable HTTP "connection-keep-alive"
• (DONE) add more control and status registers in the "AG" module
• (DONE) fix problem with adc16 channels not sending any data, fpga reboot does not fix this (needs power cycle to fix it?)
• (DONE) investigate counter for: Dropped Triggers Due to Full [cnt_trig_dfull] [0]
• (DONE) see this: all adc16 trigger counters stop after 63 events, except for one that keeps counting, I suspect the 16->1 packet mux is getting stuck?
• (DONE) investigate: when approaching 100 Mbytes/sec 1gige ethernet limit, channels start getting stuck in the "dropped due to busy" state until remaining channels can still send data, "dropped due to full" never truely increment as one would expect in this situation.
• verify ADC serdes receive clock is correctly defined, verify signals after serdes is not tangled with non-receive-clock clocks.
• simplify code between ADC serdes and phase fifo, replace 7-to-14 bit shift register with a mux.
• update DAC control
• connect grand-or of trigger discriminators to DAC output (with a 48-bit channel-enable mask)
• verify that trigger discriminator fires on both positive and negative pulses (positive and negative thresholds)
• add provision for coded trigger signal
• update TRG link
• verify that ADC SYNC works
• add more status registers in the "AG" module
• add independant pulser for the DAC output. program the firing rate, then start immediately or after first esata "sync" signal

ZZZ

ZZZ