Back Midas Rome Roody Rootana
  Midas DAQ System  Not logged in ELOG logo
Entry  06 Jul 2016, Zhe Wang, Suggestion, Frontend crush on high event rate 
    Reply  09 Jul 2016, Zhe Wang, Suggestion, Frontend crush on high event rate 
       Reply  10 Jul 2016, Zhe Wang, Suggestion, Frontend crush on high event rate frontend.c
          Reply  13 Jul 2016, Zhe Wang, Suggestion, Frontend crush on high event rate frontend.c
          Reply  13 Jul 2016, Zhe Wang, Suggestion, Frontend crush on high event rate 
             Reply  13 Jul 2016, Zhe Wang, Suggestion, Frontend crush on high event rate 
                Reply  30 Sep 2016, Konstantin Olchanski, Suggestion, Frontend crush on high event rate 
Message ID: 1186     Entry time: 13 Jul 2016     In reply to: 1185
Author: Zhe Wang 
Topic: Suggestion 
Subject: Frontend crush on high event rate 
Somehow I don't understand why people's reply is only in my mail box.
So I pasted them here. I hope they don't mind and these information may be useful for others.

The following is some discussion.
==========================================================================================
> In read_trigger_event(), you creating a secondary bank with time in
> second. For your information, this time in second is already written in
> the event header. You can retrieve the time using macros from the
> midas.h   time = TIME_STAMP(pevent)

Removed.

>
> In frontend_init() you loop over NFADC (1) and call for each loop
> frontend_config() after opening the device on that card. In
> frontend_config() you redo a loop over NFADC, meaning that in case of
> more than one card you will find the second one not open on the first
> frontend_config (ok for one card though).
>

Corrected.

> In frontend_config() what is the return sCAEN from MallocReadoutBuffer()?
> What is the size of the requested allocated buffer?

The return size of allocated buffer is 134936.

>
> What is the value of the sCAEN from the ReadData() function in
> read_trigger_event()?

It is always 0 for success until it crashes.
However, even for the event it crashes, it also appears as 0.

>
> I didn't check all the config parameters!
>
> What is the value of count in the poll_event(). It is true if the test
> in poll_event() is too short, it cause timing corruption during
> calibration. 

Do you mean Midas timing calibration for poll_event() before all finally start up?
We havn't observed corruption at this stage.

> This never happen during CAMAC time... to be fixed!
> The alternative is to include a ss_sleep(1) instead of the prescale.
> a 1ms delay between every poll is short enough to ensure your 1KHz trigger.

We tried ss_sleep(1) in poll_event(), and it doesn't help.
We also tried add a ss_sleep(10) in the read_trigger_event().
This may work. But we can only reach 100 Hz and 1 MB/s rate. Still low.

>
> How long do you spend in the read_trigger_event()? To be measured.

We add some timers in this part of the program.
The time spent on CAEN_DGTZ_ReadData is about 100 us.
To sleep 1 ms in read_trigger_event may delay the crush, but just one minute.
To sleep 10 ms works.

>
> I still don't understand your setup as you mention using optic fiber to
> access the VME controller? do you have a A3818 or similar to the
> controller? If so why don't you connect directly the optic to the VX1751
> and prevent the use of the VME backplane?

Our connect is:
A2818 (PCI) - fiber - V2718 (Bridge) - VME - V1751
We probably need to configure other vme boards through VME at the same time,
however, these boards don't have a fiber connection.

We also tested direct fiber connect for V1751 today.
But it crashes with the same symptom.
==========================================================================================
Attachment 1: frontend.c  13 kB  Uploaded 13 Jul 2016  | Hide | Hide all
/*****************************************************************\

Name:         frontend.c
Created by: 	Zhe Wang 
Date:         03/16/2015 

Modified by: Mohan Li
Date: 07/04/2016

Contents:     Experiment specific readout code (user part) of Midas frontend.
Supported VME modules:
CAEN V2718 VME-CONET Bridge
CAEN V1751 10-Bits 1-GHz Flash ADC

Experiment: Dark noise

Currently: Use CAEN_Digitizer lib. Use Ramdom number to avoid disconnection. 

$Id: $

\********************************************************************/

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/time.h>
#include <time.h>
#include "midas.h"
#include "mcstd.h"
#include "mvmestd.h"
#include "experim.h"
#include "v1751.h"
#include "CAENDigitizer.h"

/* make frontend functions callable from the C framework */
#ifdef __cplusplus
extern "C" {
#endif

	/*-- Globals -------------------------------------------------------*/

	/* The frontend name (client name) as seen by other MIDAS clients   */
	char *frontend_name = "Frontend";
	/* The frontend file name, don't change it */
	char *frontend_file_name = __FILE__;

	/* frontend_loop is called periodically if this variable is TRUE    */
	BOOL frontend_call_loop = FALSE;

	/* a frontend status page is displayed with this frequency in ms */
	INT display_period = 500;

	/* maximum event size produced by this frontend */
	//INT max_event_size = 10000;
	INT max_event_size = 100000; //modified according to feov1721.cxx

	/* maximum event size for fragmented events (EQ_FRAGMENTED) */
	INT max_event_size_frag = 5 * 1024 * 1024;

	/* buffer size to hold events */
	INT event_buffer_size = 200 * 100000;

#define NFADC 1
#define NMax 4
	int hFADC[NFADC];

	/* VMEBaseAddress */
	uint32_t FADCBA[NMax] = {0x000C0000,0,0,0};  // FADC base address 0x80000000

	uint32_t EvtCounterFadc[NMax];

	/* Time in second*/
	uint32_t TimeInSec;

	/* initiate variables */

	FILE* logfile;

	//CAENComm_ErrorCode sCAENc;

	int l=0, d=0, h=0, Nh;
	uint32_t i, lcount, temp, lam, reg, data[50000];
	int Nmodulo=10; //print transmission information every Nmodulo events
	int tcount=0, eloop=0;
	DWORD  eStored, eSize;
	DWORD eventReady;
	DWORD BLTNB;
	DWORD recordlength;
	uint32_t recordsize = 0x1000;
	int loop, Nloop=10;
	int bshowData=0; // 1 to enable data print
	int debug = 0;
	uint32_t pct=0, ct;
	struct timeval t1;
	int   dt1, savelcount=0;
	float trg_rate =0;
	int data_test = 0; // 1 for stored data check
	int simulation = 0;// 1 for simulation mode
	

	/*-------------CAEN Digitier vairables----------*/
	int card=0;
	CAEN_DGTZ_ErrorCode sCAEN;
	CAEN_DGTZ_BoardInfo_t BoardInfo;
	char *buffer = NULL; //pointer to the read out buffer
	int c = 0;
	uint32_t size; //buffer allocated for reading data
	uint32_t bsize;
#define INTERRUPT_TIMEOUT 20000 //20000ms = 20s
#define VME_INTERRUPT_LEVEL 1
#define VME_INTERRUPT_STATUS_ID 0xAAAA
#define IRQ_EVENT_NUMBER 1

	/*-- Function declarations -----------------------------------------*/

	INT frontend_init(); 
	INT frontend_exit();
	INT begin_of_run(INT run_number, char *error);
	INT end_of_run(INT run_number, char *error);
	INT pause_run(INT run_number, char *error);
	INT resume_run(INT run_number, char *error);
	INT frontend_loop();

	INT read_trigger_event(char *pevent, INT off);
	INT frontend_config();

	/*-- Equipment list ------------------------------------------------*/

#undef USE_INT
//#define USE_INT

	EQUIPMENT equipment[] = {

		{"Trigger",               /* equipment name */
			{1, 0,                   /* event ID, trigger mask */
				"SYSTEM",               /* event buffer */
#ifdef USE_INT
				EQ_INTERRUPT,           /* equipment type */
#else
				EQ_POLLED,              /* equipment type */
#endif
				//  LAM_SOURCE(CRATE, LAM_STATION(SLOT_ADC)), /* event source */
				LAM_SOURCE(0, 0xFFFFFF),   /* event source crate 0, all stations, by Li*/
				"MIDAS",                /* format */
				TRUE,                   /* enabled */
				RO_RUNNING |            /* read only when running */
					RO_ODB,                 /* and update ODB */
				500,                    /* poll for 500ms */
				0,                      /* stop run after this event limit */
				0,                      /* number of sub events */
				0,                      /* don't log history */
				"", "", "",},
			read_trigger_event,      /* readout routine */
		},

		{""}
	};

#ifdef __cplusplus
}
#endif

/********************************************************************\
  Callback routines for system transitions

  These routines are called whenever a system transition like start/
  stop of a run occurs. The routines are called on the following
occations:

frontend_init:  When the frontend program is started. This routine
should initialize the hardware.

frontend_exit:  When the frontend program is shut down. Can be used
to releas any locked resources like memory, commu-
nications ports etc.

begin_of_run:   When a new run is started. Clear scalers, open
rungates, etc.

end_of_run:     Called on a request to stop a run. Can send
end-of-run event and close run gates.

pause_run:      When a run is paused. Should disable trigger events.

resume_run:     When a run is resumed. Should enable trigger events.

\********************************************************************/

/*-- Frontend Init -------------------------------------------------*/
INT frontend_init()
{
	// Open FADC digitizer
	for( card=0; card<NFADC; card++ )  {
                // through V2718
	        //sCAEN = CAEN_DGTZ_OpenDigitizer(CAEN_DGTZ_PCI_OpticalLink, 0, 0, FADCBA[card], &hFADC[card]);
	        // through fiber
	        sCAEN = CAEN_DGTZ_OpenDigitizer(CAEN_DGTZ_OpticalLink, 0, 0, 0, &hFADC[card]);
		if(sCAEN != CAEN_DGTZ_Success) {
			printf("Can't open digitizer\n");
			//sCAEN = CAEN_DGTZ_CloseDigitizer(hFADC[card]);
		}
	}

	frontend_config();
	
	return SUCCESS;
}

INT frontend_config()
{
	/* ------FADC configuration------ */
	for( card=0; card<NFADC; card++ )  {  

		//Print Board Info
		sCAEN = CAEN_DGTZ_GetInfo(hFADC[card], &BoardInfo);
		printf("\nConnected to CAEN Digitizer Model %s, recognized as board %d\n", BoardInfo.ModelName, card);
		printf("\tROC FPGA Release is %s\n", BoardInfo.ROC_FirmwareRel);
		printf("\tAMC FPGA Release is %s\n", BoardInfo.AMC_FirmwareRel);
		//Reset Digitizer
		sCAEN = CAEN_DGTZ_Reset(hFADC[card]);
		//Calibrate temperatire
		sCAEN = CAEN_DGTZ_Calibrate(hFADC[card]);
		//Set the lenght of each waveform (in samples)
		sCAEN = CAEN_DGTZ_SetRecordLength(hFADC[card], 1792);
		//Generate a global trigger by AND opend channels. Set trigger on channel 0 to be ACQ_ONLY
		sCAEN = CAEN_DGTZ_SetChannelSelfTrigger(hFADC[card], CAEN_DGTZ_TRGMODE_ACQ_AND_EXTOUT, 0x01);
		//Enable channel 0
		sCAEN = CAEN_DGTZ_SetChannelEnableMask(hFADC[card], 0x01); 
		//Set selfTrigger threshold 0x3a7=-4mV
		sCAEN = CAEN_DGTZ_SetChannelTriggerThreshold(hFADC[card], 0, 0x3a9);
		//Trigger under threshold
		sCAEN = CAEN_DGTZ_SetTriggerPolarity(hFADC[card], 0, CAEN_DGTZ_TriggerOnFallingEdge);
		//Post trigger
		sCAEN = CAEN_DGTZ_SetPostTriggerSize(hFADC[card], 20);
		//DC offset
		sCAEN = CAEN_DGTZ_SetChannelDCOffset(hFADC[card], 0, 0x3333);
		//Set the acquisition mode
		sCAEN = CAEN_DGTZ_SetAcquisitionMode(hFADC[card], CAEN_DGTZ_SW_CONTROLLED);
		//IO Level
		sCAEN = CAEN_DGTZ_SetIOLevel(hFADC[card], CAEN_DGTZ_IOLevel_NIM);
		//Set the max number of events to transfer in a sigle readout
		sCAEN = CAEN_DGTZ_SetMaxNumEventsBLT(hFADC[card], 1);
		//Set the behaviour when a Software tirgger arrives
		//sCAEN = CAEN_DGTZ_SetSWTriggerMode(hFADC[card], CAEN_DGTZ_TRGMODE_ACQ_ONLY);

		//---------------------------------------------------------//
		//----- Last step: Allociate memory for readout buffer-----//
		//---------------------------------------------------------//
		sCAEN = CAEN_DGTZ_MallocReadoutBuffer(hFADC[card], &buffer, &size);
		printf("MallocReadoutBuffer returned with status %d and size %d.\n", sCAEN, size);

		if(sCAEN != CAEN_DGTZ_Success) {
			printf("Errors during Digitizer Configuration.\n");
			sCAEN = CAEN_DGTZ_FreeReadoutBuffer(&buffer);
			sCAEN = CAEN_DGTZ_CloseDigitizer(hFADC[card]);
		}else{
			printf("Digitizer Configuration Successfully.\n");
		}
	}//end of FADC Configuration
	return SUCCESS;
}

/*-- Frontend Exit -------------------------------------------------*/

INT frontend_exit()
{
	//Stop DAQ
	for (card=0;card<NFADC;card++) {
		sCAEN = CAEN_DGTZ_SWStopAcquisition(hFADC[card]);
	}
	//Free memory
	sCAEN = CAEN_DGTZ_FreeReadoutBuffer(&buffer);
	//Close digitizer
	for (card=0;card<NFADC;card++) {
		sCAEN = CAEN_DGTZ_CloseDigitizer(hFADC[card]);
	}
	if(sCAEN == CAEN_DGTZ_Success){
		printf("FADC Modules stopped.\n");
	}else{
		printf("FADC Modules can not be stopped.\n");
	}
	return SUCCESS;
}

/*-- Begin of Run --------------------------------------------------*/

INT begin_of_run(INT run_number, char *error)
{
	//Create log file
	logfile = fopen("log.txt","w");
	//Start FADC
	for (card=0;card<NFADC;card++) {
		sCAEN = CAEN_DGTZ_ClearData(hFADC[card]);
		sCAEN = CAEN_DGTZ_SWStartAcquisition(hFADC[card]);
	}  
	printf("begin of run.\n");
	return SUCCESS;
}

/*-- End of Run ----------------------------------------------------*/
... 194 more lines ...
ELOG V3.1.4-2e1708b5