GSI TRB3: Difference between revisions

From DaqWiki
Jump to navigation Jump to search
 
(48 intermediate revisions by 2 users not shown)
Line 3: Line 3:
== Documentation ==  
== Documentation ==  


[http://trb.gsi.de/ Main TRB documentation]
* [http://trb.gsi.de/ Main TRB3 webpage]
* [http://jspc29.x-matter.uni-frankfurt.de/docu/trb3docu.pdf Main TRB3 Manual]
* [[Media:Trb3docu-2018aug10.pdf]] local copy of the manual
* [[Media:Trb3docu.pdf]] local copy of the manual
* [[Media:trb3.pdf]] local copy of the schematics
* [[Media:4conn.pdf]] schematics 4 LVDS connector add-on board (GSI-AD ADDON_4CONN2)
* [[Media:trbdist1-SCM.pdf]] [[Media:trbdist2.pdf]] [[Media:trbdist3-SCM.pdf]] schematics for external trigger and clock interface boards


== Computer Setup Instructions ==  
== Hardware Information ==


Currently have gotten this working on an Opensuse machine. 
=== Power ===


We setup TRB3 on a private network.  So need machine with second ethernet port.
The TRB3 runs off 50V DC power.  The TRB3 needs a fan in order to operate.  Between the fan and the TRB3 they draw ~0.5A on 50V.


Currently the work is being done with user trb3_user on computer daq14.  This is a local user on that computer; not part of DAQ cluster.
=== Clock and Trigger ===


Full instructions to be written
Michael Traxler provided the following picture of clock and trigger circuitry:


==  Start-up Instructions ==
[[Image:trb_clockdistribution.png|700x350px]]


These are the basic instructions to initialize and take data from TRB3, using the GSI DAQ tools.
Michael also provides following details


1) Start the trb3 program and initialize the TRB3
Clock:
<pre>
* system clock frequency of 200MHz can be taken either from the internal oscillator, or via the second LVDS pair (pin 3 and 6) of the connector "RJ-45-Clock". They are distributed to all 5 FPGAs to the primary-clock-input.
cd /home/tbr3_user/trbsoft/daqtools/users/triumf_trb171
* In all "modern" FPGA designs (since several years) you don't have to do anything in registers to use the external clock. The scheme is simple: When an external clock is available and stable and the PLL locks to it at power up, the external clock is used. If not, the internal clock is used.
source startup.sh
</pre>


2) Start webbrowser: In new terminal, do
Trigger:
* If you have a CTS (or a single TRB3 setup) the trigger is generated in the CTS, from external tigger sources. This you can see in Table 27 in the TRB3-manual (currently page 75).  For instance, one of the external trigger signals is on pair 2 of the trigger RJ45 connector
* So, for the CTS, there are 4 different inputs available on the two RJ45 connectors, which all can be used at the same time. If you need more, you can add a CTS-AddOn on the backside of the TRB3 and then you have many more trigger inputs. Each input will generate a certain "trigger type" data word in the data stream to distinguish between the trigger inputs.


<pre>
== TDC calibration ==
cd /home/tbr3_user/trbsoft/daqtools/web/
./cts_gui --noopenxterm --port=1234 --endpoint=0xc001
</pre>


open webbrowser and go to
The concept of the FPGA-TDC calibration is described here:


http://daq14.triumf.ca:1234/
http://dabc.gsi.de/doc/dabc2/hadaq_tdc_calibr.html


3) Start event builder: In new terminal, do
One important summary from that document: we require ~1e5 random hits for every channel in order to build up an individual fine counter calibration for each channel.


<pre>
== Computer and Software Setup Instructions ==
cd trbsoft/daqtools/users/triumf_trb171/
dabc_exe TdcEventBuilder.xml
</pre>


4) Enable the readout of the channels we want.
Currently have gotten the TRB3 working on three different configurations.  On this page we provide information that is generic to each setup. The links provide instructions for the parts of the operation that are unique to each setup:


i) Go to webpage
1) A GSI-provided readout package (and analysis package) which runs on OpenSuse.  See [[TRB3 GSI Opensuse instructions]].


http://daq14.triumf.ca:1234/tdc/tdc.htm
2) A MIDAS-based readout on an Opensuse machine. No instructions.


ii) Under 'board' set it to 0100.
3) A MIDAS-based readout on a Centos-7 machine. See [[TRB3 Centos-7 instructions]].


iii) Then click 'enable' radio button
On this page we provide information that is generic to each setup. The links provide instructions for the parts of the operation that are unique to each setup:


iv) Then, under 0100 set all the channel groups to on.  Should then start
We setup TRB3 on a private network.  So need machine with second ethernet port.
seeing triggers for different channels (if we have signals going into
channels).


v) Finally, disable the trigger window on the same page.  On the section c801
== Useful TRB3 commands ==
click off and make sure it says 'disabled'.  If trigger window is disabled
then all triggers are accepted...


a) Get IDs for TRB3 FPGAs (1 central FPGA and 4 FPGAs for TDCs):


== Doing TDC analysis ==
<pre>
trb3_user@linux-klyr:~/lindner> trbcmd i 0xffff
0xc001  0xe1000006e937ac28  0x05
0x0100  0x9c000006e937a028  0x00
0x0101  0x91000006e95e8328  0x01
0x0102  0x75000006e9383128  0x02
0x0103  0xe6000006e96e5228  0x03
</pre>


This is how we do the TDC analysis using the GSI tools:
The four FPGAs that actually make TDCs are numbered 0x100, 0x101, 0x102, 0x103.


A) Setup the analyzer
b) Print out raw TDCs for the events that are coming in through event builder


<pre>
<pre>
source ~/trbsoft/trb3/trb3login
hldprint localhost:6789 -onlytdc 0x0100 -num 0
</pre>
</pre>


(This needs to be run in any terminal where you want to run the go4analysis
c) Print out raw TDCs for events in file


<pre>
hldprint //data/trb3_data/pulser17342114715.hld -onlytdc 0x0100 -num 0
</pre>


B) Create the set of calibration constants for the TDC
d) Print out raw data stream from EB


<pre>
<pre>
cd /home/tbr3_user/trbsoft/daqtools/users/triumf_trb171
hldprint localhost:6789 -raw
rm cal*.cal  (remove old calibration files)
export CALTRIG=1; export CALMODE=-1;
go4analysis -number 100000 -user /data/trb3_data/saved/pulser17342114715.hld
</pre>
</pre>


(need to change this to use the hld file that you want to use)
e) Check the status of the clock


The resulting calibration files are called something like cal_010X.cal.  CALTRIG=1 means using trigger type 1 to do calibration. You can set this to different trigger type number, if required.
Check bit 8 of register 0xd300.   
 
This indicates that an external clock is being used
<pre>
[agtdc@daq16 triumf_trb171]$ trbcmd r 0xc001 0xd300
0xc001  0x00000101
</pre>
 
This indicates that an internal clock is being used
<pre>
[agtdc@daq16 triumf_trb171]$ trbcmd r 0xc001 0xd300
0xc001 0x00000001
</pre>




C) Now analyze more events, using those calibration files
f) "Ping of death" somehow ping board to reboot it:


<pre>
<pre>
rm *root
ping  -pc001 trb171
export CALTRIG=1; export CALMODE=0;
go4analysis -number 1000000 -user /data/trb3_data/saved/pulser17342114715.hld
</pre>
</pre>


now open the resulting root file
== Updating TRB3 firmware ==
 
Based on instructions from Michael
 
General points:
 
* Take care that you have a stable setup with the power-supply stable and cooling ok, as if you turn off the board while reprogramming you need a programming cable to recover the FPGA.


<pre>
* List of available firmware files are here:
root -l Go4AutoSave.root
https://jspc29.x-matter.uni-frankfurt.de/trbweb/?action=page&url=design-files
 
 
Steps to flash
 
1) Get firmware.  Ie:
<pre> wget http://jspc29.x-matter.uni-frankfurt.de/bitfiles/trb3_periph_padiwa_tdc_32_stretch_triggerlogic28-4_20181024.bit </pre>
 
2) reflashing is done via:
 
<pre> trbflash program 0x<TRB-address-of-FPGA> ./trb3_periph_padiwa_tdc_32_stretch_triggerlogic28-4_20181024.bit
</pre>
</pre>


and use TBrowser() to look at the difference histogram.  In particular, right now we look at the histogram TDC_0100/Ch3/TDC_0100_Ch3_RisingRef which shows the difference between the two synchronized input channels.  This distribution should show a timing resolution between the hits of ~18ps if all the TDC calibration is working fine...


== TRB3 MIDAS implementation ==
<pre>
trbflash program 0x0102 ./trb3_periph_padiwa_tdc_32_stretch_triggerlogic28-4_20181024.bit
trbflash program 0x0103 ./trb3_periph_padiwa_tdc_32_stretch_triggerlogic28-4_20181024.bit </pre>
 
3) after a successful flash and verify (automatic):
 
<pre> trbcmd reload 0x<TRB-address-of-FPGA> </pre>
 
4) But do it one FPGA after the other, if you do it the first time.
You can also reprogramm all of them at once with a broadcast address.


We have a working MIDAS readout of the TRB3 TDC data.  Currently this readout uses a lot of the original GSI tools for setting up the TRB3 before the readout can start.  At this time you need to do all the steps described in "Start up instructions"  in order to setup the TRB3 before starting the MIDAS frontend. Eventually we will try to remove as many as possible of the original GSI tools.
== Konstantin's notes ==


MIDAS frontend code is available here:
=== TRB3 grounding ===


|https://bitbucket.org/ttriumfdaq/trb3_frontend|
to avoid damage to the DC-coupled trigger input, TRB3 PCB should be grounded to the same
ground as the device driving this input (VME-CDM in this case). On the TRB3 side, we attach
the ground wire to one of the PCB mounting holes (confirmed ground). On the other end, it should
be attached to the VME crate ground.


The MIDAS frontend is setup to read events from the DABC event builder, which makes the events available on port 6789. You can see a running MIDAS TRB3 frontend here:
the clock input is AC coupled and probably no affected by grounding problems.


|https://daq14.triumf.ca:8443|
the esata splitter should also be grounded, see below.


I added simple decoder and histograming classes to rootana.  Currently I have only implemented the crude TDC calibration into the TRB3 bank decoders.  In the long run we may want to integrate the fine TDC calibration directly into the frontend.
=== esata spliter in use by ALPHA-g ===


== Useful TRB3 commands ==
CDM --- minisas --- 4-esata-flail --- esata --- ALPHA-T splitter --- 2x cat6 --- 2xRJ45


a) Get IDs for TRB3 FPGAs (1 central FPGA and 4 FPGAs for TDCs):
ALPHA-T esata splitter:


<pre>
* use 7ft cat6 patch cable, 568B wiring (568A will work, but color coding is not the same)
trb3_user@linux-klyr:~/lindner> trbcmd i 0xffff
* cat6 RJ45 green pair solder to splitter clock side
0xc001  0xe1000006e937ac28  0x05
* cat6 RJ45 green pair solder to splitter trigger side
0x0100  0x9c000006e937a028  0x00
* ground wire of same length solder to esata ground, other end attach to TRB3 PCB mounting hole or other good ground.
0x0101  0x91000006e95e8328  0x01
0x0102  0x75000006e9383128  0x02
0x0103  0xe6000006e96e5228  0x03
</pre>


b) Print out raw TDCs for the events that are coming in through event builder
=== second trigger input ===


<pre>
According to schematics and documentation, RJ45 "trigger" connector has 2 trigger inputs,
hldprint localhost:6789 -onlytdc 0x0100 -num 0
normal input via the "green pair" and second input via the "orange pair".
</pre>


c) Print out raw TDCs for events in file
Switching between these inputs is done by "TRIGGER_SELECT" FPGA output acting on the "IN_SEL" pin of the CDCLVD1216 LVDS mux ([[Media:Cdclvd1216.pdf]] datasheet).


<pre>
This "TRIGGER_SELECT" signal is not listed in the documentation, but I confirm with voltmeter at J28 test point that it's output is same as bit 0 of register 0xd300. writing "1" sets voltage to 2.5V, writing "0" sets it to 0V.
hldprint //data/trb3_data/pulser17342114715.hld -onlytdc 0x0100 -num 0  
</pre>


d) Print out raw data stream from EB
However, in either state, it has no effect on the trigger input. One would expect that the normal trigger will stop if the lvds mux is switched to the second input. Actually regardless of "TRIGGER_SELECT" state, the normal trigger input is always active (oscillates if connected to unterminated/ungrounded RJ45 cable), the second trigger input never produces any trigger counts.


<pre>
this is a mystery.
hldprint localhost:6789 -raw
</pre>


e) "Ping of death" somehow ping board to reboot it:
== ZZZ ==


<pre>
ZZZ
ping  -pc001 trb171
</pre>

Latest revision as of 16:25, 31 May 2021

TRB3 is an FPGA-TDC board made and sold by GSI. It can achieve ~20ps timing resolution

Documentation

Hardware Information

Power

The TRB3 runs off 50V DC power. The TRB3 needs a fan in order to operate. Between the fan and the TRB3 they draw ~0.5A on 50V.

Clock and Trigger

Michael Traxler provided the following picture of clock and trigger circuitry:

Trb clockdistribution.png

Michael also provides following details

Clock:

  • system clock frequency of 200MHz can be taken either from the internal oscillator, or via the second LVDS pair (pin 3 and 6) of the connector "RJ-45-Clock". They are distributed to all 5 FPGAs to the primary-clock-input.
  • In all "modern" FPGA designs (since several years) you don't have to do anything in registers to use the external clock. The scheme is simple: When an external clock is available and stable and the PLL locks to it at power up, the external clock is used. If not, the internal clock is used.

Trigger:

  • If you have a CTS (or a single TRB3 setup) the trigger is generated in the CTS, from external tigger sources. This you can see in Table 27 in the TRB3-manual (currently page 75). For instance, one of the external trigger signals is on pair 2 of the trigger RJ45 connector
  • So, for the CTS, there are 4 different inputs available on the two RJ45 connectors, which all can be used at the same time. If you need more, you can add a CTS-AddOn on the backside of the TRB3 and then you have many more trigger inputs. Each input will generate a certain "trigger type" data word in the data stream to distinguish between the trigger inputs.

TDC calibration

The concept of the FPGA-TDC calibration is described here:

http://dabc.gsi.de/doc/dabc2/hadaq_tdc_calibr.html

One important summary from that document: we require ~1e5 random hits for every channel in order to build up an individual fine counter calibration for each channel.

Computer and Software Setup Instructions

Currently have gotten the TRB3 working on three different configurations. On this page we provide information that is generic to each setup. The links provide instructions for the parts of the operation that are unique to each setup:

1) A GSI-provided readout package (and analysis package) which runs on OpenSuse. See TRB3 GSI Opensuse instructions.

2) A MIDAS-based readout on an Opensuse machine. No instructions.

3) A MIDAS-based readout on a Centos-7 machine. See TRB3 Centos-7 instructions.

On this page we provide information that is generic to each setup. The links provide instructions for the parts of the operation that are unique to each setup:

We setup TRB3 on a private network. So need machine with second ethernet port.

Useful TRB3 commands

a) Get IDs for TRB3 FPGAs (1 central FPGA and 4 FPGAs for TDCs): 

trb3_user@linux-klyr:~/lindner> trbcmd i 0xffff
0xc001  0xe1000006e937ac28  0x05
0x0100  0x9c000006e937a028  0x00
0x0101  0x91000006e95e8328  0x01
0x0102  0x75000006e9383128  0x02
0x0103  0xe6000006e96e5228  0x03

The four FPGAs that actually make TDCs are numbered 0x100, 0x101, 0x102, 0x103.

b) Print out raw TDCs for the events that are coming in through event builder 

hldprint localhost:6789 -onlytdc 0x0100 -num 0

c) Print out raw TDCs for events in file 

hldprint //data/trb3_data/pulser17342114715.hld -onlytdc 0x0100 -num 0

d) Print out raw data stream from EB 

hldprint localhost:6789 -raw

e) Check the status of the clock

Check bit 8 of register 0xd300.

This indicates that an external clock is being used 

[agtdc@daq16 triumf_trb171]$ trbcmd r 0xc001 0xd300
0xc001  0x00000101

This indicates that an internal clock is being used 

[agtdc@daq16 triumf_trb171]$ trbcmd r 0xc001 0xd300
0xc001  0x00000001


f) "Ping of death" somehow ping board to reboot it: 

ping  -pc001 trb171

Updating TRB3 firmware

Based on instructions from Michael

General points:

  • Take care that you have a stable setup with the power-supply stable and cooling ok, as if you turn off the board while reprogramming you need a programming cable to recover the FPGA.
  • List of available firmware files are here:

https://jspc29.x-matter.uni-frankfurt.de/trbweb/?action=page&url=design-files


Steps to flash

1) Get firmware. Ie: 

 wget http://jspc29.x-matter.uni-frankfurt.de/bitfiles/trb3_periph_padiwa_tdc_32_stretch_triggerlogic28-4_20181024.bit

2) reflashing is done via: 

 trbflash program 0x<TRB-address-of-FPGA> ./trb3_periph_padiwa_tdc_32_stretch_triggerlogic28-4_20181024.bit



 
trbflash program 0x0102 ./trb3_periph_padiwa_tdc_32_stretch_triggerlogic28-4_20181024.bit
trbflash program 0x0103 ./trb3_periph_padiwa_tdc_32_stretch_triggerlogic28-4_20181024.bit

3) after a successful flash and verify (automatic): 

 trbcmd reload 0x<TRB-address-of-FPGA>

4) But do it one FPGA after the other, if you do it the first time. You can also reprogramm all of them at once with a broadcast address.

Konstantin's notes

TRB3 grounding

to avoid damage to the DC-coupled trigger input, TRB3 PCB should be grounded to the same ground as the device driving this input (VME-CDM in this case). On the TRB3 side, we attach the ground wire to one of the PCB mounting holes (confirmed ground). On the other end, it should be attached to the VME crate ground.

the clock input is AC coupled and probably no affected by grounding problems.

the esata splitter should also be grounded, see below.

esata spliter in use by ALPHA-g

CDM --- minisas --- 4-esata-flail --- esata --- ALPHA-T splitter --- 2x cat6 --- 2xRJ45

ALPHA-T esata splitter:

  • use 7ft cat6 patch cable, 568B wiring (568A will work, but color coding is not the same)
  • cat6 RJ45 green pair solder to splitter clock side
  • cat6 RJ45 green pair solder to splitter trigger side
  • ground wire of same length solder to esata ground, other end attach to TRB3 PCB mounting hole or other good ground.

second trigger input

According to schematics and documentation, RJ45 "trigger" connector has 2 trigger inputs, normal input via the "green pair" and second input via the "orange pair".

Switching between these inputs is done by "TRIGGER_SELECT" FPGA output acting on the "IN_SEL" pin of the CDCLVD1216 LVDS mux (Media:Cdclvd1216.pdf datasheet).

This "TRIGGER_SELECT" signal is not listed in the documentation, but I confirm with voltmeter at J28 test point that it's output is same as bit 0 of register 0xd300. writing "1" sets voltage to 2.5V, writing "0" sets it to 0V.

However, in either state, it has no effect on the trigger input. One would expect that the normal trigger will stop if the lvds mux is switched to the second input. Actually regardless of "TRIGGER_SELECT" state, the normal trigger input is always active (oscillates if connected to unterminated/ungrounded RJ45 cable), the second trigger input never produces any trigger counts.

this is a mystery.

ZZZ

ZZZ

Retrieved from "https://daq00.triumf.ca/DaqWiki/index.php?title=GSI_TRB3&oldid=4636"