Back Midas Rome Roody Rootana
  Midas DAQ System  Not logged in ELOG logo
Entry  12 Dec 2021, Marius Koeppel, Bug Report, Writting MIDAS Events via FPGAs  dummy_fe.cpp
    Reply  26 Jan 2022, Konstantin Olchanski, Bug Report, Writting MIDAS Events via FPGAs  
       Reply  26 Jan 2022, Marius Koeppel, Bug Report, Writting MIDAS Events via FPGAs  
          Reply  26 Jan 2022, Konstantin Olchanski, Bug Report, Writting MIDAS Events via FPGAs  
             Reply  26 Jan 2022, Marius Koeppel, Bug Report, Writting MIDAS Events via FPGAs  
    Reply  28 Jan 2022, Stefan Ritt, Bug Report, Writting MIDAS Events via FPGAs  dummy_fe.cpp
       Reply  16 Feb 2022, Marius Koeppel, Bug Report, Writting MIDAS Events via FPGAs  
          Reply  03 Mar 2022, Stefan Ritt, Bug Report, Writting MIDAS Events via FPGAs  
             Reply  07 Mar 2022, Marius Koeppel, Bug Report, Writting MIDAS Events via FPGAs  
                Reply  25 Mar 2022, Marius Koeppel, Bug Report, Writting MIDAS Events via FPGAs  
Message ID: 2326     Entry time: 28 Jan 2022     In reply to: 2308     Reply to this: 2347
Author: Stefan Ritt 
Topic: Bug Report 
Subject: Writting MIDAS Events via FPGAs  
I finally got the dummy program working. There were several issues:

- event_buffer_size was defined as 10000 * 32 MB = 320 GB, exceeding the RAM of the computer

- SERIAL number starting with 1. Actually in midas, event serial numbers always started with zero, but this was wrong in the documentation at 
https://midas.triumf.ca/MidasWiki/index.php/Event_Structure, so I also fixed the documentation

- the event header time stamp must be seconds since 1.1.1970, and thus the function ss_time() should be used to set it

- calling set_equipment_status() for each event slows down the event collection considerably, since this function access the ODB each time

- dma_buf_dummy is defined inside the event loop, so it gets allocated and de-allocated on the stack for each event. Of course this might vanish 
when the real FPGA buffer will be used.

- The line pdata+=sizeof(dma_buf_dummy); is wrong. pdata is pointer to uint32_t, but the sizeof() operation returns the size of the 
dma_buf_dummy in bytes. Therefore, pdata gets incremented by four times the size of dma_buf_dummy

- Instead the call to std::this_thread::sleep_for(std::chrono::milliseconds(2000)); one can call the standard midas call ss_sleep(2000); which 
is a bit shorter

- Finally, sending many events to the ring buffer triggered a bug in the midas ring buffer functions which were lingering there since 2007. I'm 
glad that this happened and now could be fixed. Not sure if other experiments where affected in the last decade by that. This could have 
manifested itself in lost events or crashing front-ends. Anyhow, now it's fixed. You need to update midas to get the fix.

I attached a working version of the dummy program for your reference. Banks a different but the principle should become clear.

Stefan
Attachment 1: dummy_fe.cpp  10 kB  | Hide | Hide all
/********************************************************************\

  Name:         dummy_fe.cxx
  Created by:   Frederik Wauters
  Changed by:   Marius Koeppel

  Contents:     Dummy frontend producing stream data

\********************************************************************/

#include <algorithm>
#include <math.h>
#include <random>
#include <stdio.h>
#include <stdlib.h>

#include <bitset>
#include <iostream>
#include <unistd.h>

#include "midas.h"
#include "msystem.h"

#include <chrono>
#include <thread>

#include "mfe.h"

using namespace std;

/*-- Globals -------------------------------------------------------*/

/* The frontend name (client name) as seen by other MIDAS clients   */
const char *frontend_name = "Dummy FE SWB";
/* The frontend file name, don't change it */
const char *frontend_file_name = __FILE__;

/* frontend_loop is called periodically if this variable is TRUE    */
BOOL frontend_call_loop = FALSE;

/* a frontend status page is displayed with this frequency in ms    */
INT display_period = 0;

/* maximum event size produced by this frontend */
INT max_event_size = 1 * (1024 * 1024);// 32MB

/* maximum event size for fragmented events (EQ_FRAGMENTED) */
INT max_event_size_frag = 5 * 1024 * 1024;

/* buffer size to hold events */
INT event_buffer_size = 2 * max_event_size;

/*-- Function declarations -----------------------------------------*/

INT read_stream_thread(void *param);
uint32_t generate_random_pixel_hit_swb(uint32_t time_stamp);
uint32_t generate_random_beam_ref_hit(uint32_t time_stamp, uint32_t chipID);

BOOL equipment_common_overwrite = TRUE;//true is overwriting the common odb

/* DMA Buffer and related */
volatile uint32_t *dma_buf;
#define MUDAQ_DMABUF_DATA_ORDER 25                          // 29, 25 for 32 MB
#define MUDAQ_DMABUF_DATA_LEN (1 << MUDAQ_DMABUF_DATA_ORDER)// in bytes
size_t dma_buf_size = MUDAQ_DMABUF_DATA_LEN;
uint32_t dma_buf_nwords = dma_buf_size / sizeof(uint32_t);

/*-- Equipment list ------------------------------------------------*/

EQUIPMENT equipment[] = {

   {
      "Stream SWB", /* equipment name */
      {1, 0,        /* event ID, trigger mask */
       "SYSTEM",    /* event buffer */
       EQ_USER,     /* equipment type */
       0,           /* event source */
       "MIDAS",     /* format */
       TRUE,        /* enabled */
       RO_RUNNING,  /* read always and update ODB */
       100,         /* poll for 100ms */
       0,           /* stop run after this event limit */
       0,           /* number of sub events */
       0,           /* log history every event */
       "", "", ""},
      NULL, /* readout routine */
   },

   {""}};

/*-- Dummy routines ------------------------------------------------*/

INT poll_event(INT source, INT count, BOOL test) {
   return 1;
};

INT interrupt_configure(INT cmd, INT source, POINTER_T adr) {
   return 1;
};

/*-- Frontend Init -------------------------------------------------*/

INT frontend_init() {
   // create ring buffer for readout thread
   create_event_rb(0);

   // create readout thread
   ss_thread_create(read_stream_thread, NULL);

   return CM_SUCCESS;
}

/*-- Frontend Exit -------------------------------------------------*/

INT frontend_exit() {
   return CM_SUCCESS;
}

/*-- Frontend Loop -------------------------------------------------*/

INT frontend_loop() {
   return CM_SUCCESS;
}

/*-- Begin of Run --------------------------------------------------*/

INT begin_of_run(INT run_number, char *error) {
   return CM_SUCCESS;
}

/*-- End of Run ----------------------------------------------------*/

INT end_of_run(INT run_number, char *error) {
   return CM_SUCCESS;
}

/*-- Pause Run -----------------------------------------------------*/

INT pause_run(INT run_number, char *error) {
   return CM_SUCCESS;
}

/*-- Resume Run ----------------------------------------------------*/

INT resume_run(INT run_number, char *error) {
   return CM_SUCCESS;
}

/*------------------------------------------------------------------*/

uint32_t generate_random_pixel_hit_swb(uint32_t time_stamp) {
   uint32_t tot = rand() % 32;  // 0 to 31
   uint32_t chipID = rand() % 3;// 0 to 2
   uint32_t col = rand() % 256; // 0 to 256
   uint32_t row = rand() % 250; // 0 to 250

   uint32_t hit = (time_stamp << 28) | (chipID << 22) | (row << 14) | (col << 6) | (tot << 1);

   //   if ( print ) {
   //     printf("ts:%8.8x,chipID:%8.8x,row:%8.8x,col:%8.8x,tot:%8.8x\n", time_stamp,chipID,row,col,tot);
   //     printf("hit:%8.8x\n", hit);
   //     std::cout << std::bitset<32>(hit) << std::endl;
   //   }

   if (((hit >> 22) & 0x3f) > 2)
      printf("Hit %8.8x", hit);

   if (chipID > 2)
      printf("ChipID %8.8x", chipID);

   return hit;
}

uint32_t generate_random_beam_ref_hit(uint32_t time_stamp, uint32_t chipID) {
   uint32_t fastTS = rand() % 4194303 / 2;// 0 to 4194303

   uint32_t hit = (time_stamp << 28) | (chipID << 22) | (fastTS << 1);

   if (((hit >> 22) & 0x3f) > 4) {
      printf("Hit Ref %8.8x\n", hit);
      printf("Ref fast %8.8x\n", fastTS);
      printf("ChipID Ref %8.8x\n", chipID);
      printf("Time Ref %8.8x\n", time_stamp);
      printf("Chip %8.8x\n", ((hit >> 22) & 0x3f));
   }

   return hit;
}

INT read_stream_thread(void *param) {
   uint32_t *pdata;

   // init bank structure - 64bit alignment
   uint32_t SERIAL = 0;

   // tell framework that we are alive
   signal_readout_thread_active(0, TRUE);

   // obtain ring buffer for inter-thread data exchange
   int rbh = get_event_rbh(0);
   int status;

   int nEvents = 5000;
   size_t eventSize = 32; // in 4-byte words
   size_t dmaBufSize = nEvents * eventSize * sizeof(uint32_t); // buffer size in bytes
   uint32_t * dma_buf_dummy = (uint32_t *) malloc(dmaBufSize);

   while (is_readout_thread_enabled()) {

      // don't readout events if we are not running
      if (!readout_enabled()) {
         // do not produce events when run is stopped
         ss_sleep(10);// don't eat all CPU
         continue;
      }

      // obtain buffer space with 10 ms timeout
      status = rb_get_wp(rbh, (void **) &pdata, 10);

      // just try again if buffer has no space
      if (status == DB_TIMEOUT)
         continue;

      if (status != DB_SUCCESS) {
         cout << "!DB_SUCCESS" << endl;
         break;
      }

      for (int i = 0; i < nEvents; i++) {
         // event header
         dma_buf_dummy[ 0 + i * eventSize] = 0x00000001;           // Trigger Mask & Event ID
         dma_buf_dummy[ 1 + i * eventSize] = SERIAL++;             // Serial number
         dma_buf_dummy[ 2 + i * eventSize] = ss_time();            // time
         dma_buf_dummy[ 3 + i * eventSize] = eventSize * 4 - 4 * 4;// event size

         dma_buf_dummy[ 4 + i * eventSize] = eventSize * 4 - 6 * 4;// all bank size
         dma_buf_dummy[ 5 + i * eventSize] = 0x31;                 // flags

         // bank PCD0 first FEB
         dma_buf_dummy[ 6 + i * eventSize] = 'P' << 0 | 'C' << 8 | 'D' << 16 | '0' << 24;// bank name
         dma_buf_dummy[ 7 + i * eventSize] = 0x06;                                       // bank type TID_DWORD
         dma_buf_dummy[ 8 + i * eventSize] = 10 * 4;                                     // data size
         dma_buf_dummy[ 9 + i * eventSize] = 0x0;                                        // reserved

         dma_buf_dummy[10 + i * eventSize] = 0xE80000BC;                                // preamble
         dma_buf_dummy[11 + i * eventSize] = 0x00000000;                                // TS0
         dma_buf_dummy[12 + i * eventSize] = ss_time();                                 // TS1
         dma_buf_dummy[13 + i * eventSize] = 0xFC000000;                                // sub header
         dma_buf_dummy[14 + i * eventSize] = generate_random_pixel_hit_swb(ss_time());  // hit0
         dma_buf_dummy[15 + i * eventSize] = generate_random_pixel_hit_swb(ss_time());  // hit1
         dma_buf_dummy[16 + i * eventSize] = generate_random_beam_ref_hit(ss_time(), 3);// chip 3 beam ref bits 22:1 -> fast TS
         dma_buf_dummy[17 + i * eventSize] = generate_random_beam_ref_hit(ss_time(), 4);// chip 4 sintilator bits 22:1 -> fast TS
         dma_buf_dummy[18 + i * eventSize] = 0xFC00009C;                                // TRAILER
         dma_buf_dummy[19 + i * eventSize] = 0xAFFEAFFE;                                // PADDING

         // bank PCD1 second FEB
         dma_buf_dummy[20 + i * eventSize] = 'P' << 0 | 'C' << 8 | 'D' << 16 | '1' << 24;// bank name
         dma_buf_dummy[21 + i * eventSize] = 0x6;                                       // bank type TID_DWORD
         dma_buf_dummy[22 + i * eventSize] = 8 * 4;                                     // data size
         dma_buf_dummy[23 + i * eventSize] = 0x0;                                       // reserved

         dma_buf_dummy[24 + i * eventSize] = 0xE80001BC;                              // preamble
         dma_buf_dummy[25 + i * eventSize] = 0x00000000;                              // TS0
         dma_buf_dummy[26 + i * eventSize] = ss_time();                               // TS1
         dma_buf_dummy[27 + i * eventSize] = 0xFC000000;                              // sub header
         dma_buf_dummy[28 + i * eventSize] = generate_random_pixel_hit_swb(ss_time());// hit0
         dma_buf_dummy[29 + i * eventSize] = generate_random_pixel_hit_swb(ss_time());// hit1
         dma_buf_dummy[30 + i * eventSize] = 0xFC00009C;                              // TRAILER
         dma_buf_dummy[31 + i * eventSize] = 0xAFFEAFFE;                              // PADDING
      }

      memcpy(pdata, dma_buf_dummy, dmaBufSize);

      // print data
      if (true) {
         auto *eh = (EVENT_HEADER *) (&pdata[0]);
         auto *bh = (BANK_HEADER *) (&pdata[4]);
         auto *ba = (BANK32A *) (&pdata[6]);
         char bank_name[5];
         bank_name[4] = 0;
         memcpy(bank_name, (char *) (ba->name), 4);
         printf("EID=%4.4x TM=%4.4x SERNO=%8.8x TS=%8.8x EDsiz=%8.8x\n", eh->event_id, eh->trigger_mask, eh->serial_number, eh->time_stamp, eh->data_size);
         printf("DAsiz=%8.8x FLAG=%8.8x\n", bh->data_size, bh->flags);
         printf("BAname=%s TYP=%8.8x BAsiz=%8.8x BAres=%8.8x\n", bank_name, ba->type, ba->data_size, ba->reserved);
      }

      rb_increment_wp(rbh, dmaBufSize);// in byte length

      ss_sleep(300);// limit data rate
   }

   free(dma_buf_dummy);

   return 0;
}
ELOG V3.1.4-2e1708b5