In MEG II we also kind of achieved this rate. Marco F. will post an entry soon to describe the details. There is only one thing
I want to mention, which is our network switch. Instead of an expensive high-grade switch, we chose a cheap "Chinese" high-grade
switch. We have "rack switches", which are collector switch for each rack receiving up to 10 x 1GBit inputs, and outputting 1 x
10 GBit to an "aggregation switch", which collects all 10 GBit lines form rack switches and forwards it with (currently a single
) 10 GBit line. For the rack switch we use a
MikroTik CRS354-48G-4S+2Q+RM 54 port
and for the aggregation switch
MikroTik CRS326-24S-2Q+RM 26 Port
both cost in the order of 500 US$. We were astonished that they don't loose UDP packets when all inputs send a packet at the
same time, and they have to pipe them to the single output one after the other, but apparently the switch have enough buffers
(which is usually NOT written in the data sheets).
To avoid UDP packet loss for several events, we do traffic shaping by arming the trigger only when the previous event is
completely received by the frontend. This eliminates all flow control and other complicated methods. Marco can tell you the
details.
Another interesting aspect: While we get the data into the frontend, we have problems in getting it through midas. Your
bm_send_event_sg() is maybe a good approach which we should try. To benchmark the out-of-the-box midas, I run the dummy frontend
attached on my MacBook Pro 2.4 GHz, 4 cores, 16 GB RAM, 1 TB SSD disk. I got
Event size: 7 MB
No logging: 900 events/s = 6.7 GBytes/s
Logging with LZ4 compression: 155 events/s = 1.2 GBytes/s
Logging without compression: 170 events/s = 1.3 GBytes/s
So with this simple approach I got already more than 1 GByte of "dummy data" through midas, indicating that the buffer
management is not so bad. I did use the plain mfe.c frontend framework, no bm_send_event_sg() (but mfe.c uses rpc_send_event() which is an
optimized version of bm_send_event()).
Best,
Stefan |
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <assert.h> // assert()
#include "midas.h"
#include "experim.h"
#include "mfe.h"
/*-- Globals -------------------------------------------------------*/
/* The frontend name (client name) as seen by other MIDAS clients */
const char *frontend_name = "Sample Frontend";
/* The frontend file name, don't change it */
const char *frontend_file_name = __FILE__;
/* frontend_loop is called periodically if this variable is TRUE */
BOOL frontend_call_loop = FALSE;
/* a frontend status page is displayed with this frequency in ms */
INT display_period = 3000;
/* maximum event size produced by this frontend */
INT max_event_size = 8 * 1024 * 1024;
/* maximum event size for fragmented events (EQ_FRAGMENTED) */
INT max_event_size_frag = 5 * 1024 * 1024;
/* buffer size to hold events */
INT event_buffer_size = 20 * 1024 * 1024;
/*-- Function declarations -----------------------------------------*/
INT frontend_init(void);
INT frontend_exit(void);
INT begin_of_run(INT run_number, char *error);
INT end_of_run(INT run_number, char *error);
INT pause_run(INT run_number, char *error);
INT resume_run(INT run_number, char *error);
INT frontend_loop(void);
INT read_trigger_event(char *pevent, INT off);
INT read_periodic_event(char *pevent, INT off);
INT poll_event(INT source, INT count, BOOL test);
INT interrupt_configure(INT cmd, INT source, POINTER_T adr);
/*-- Equipment list ------------------------------------------------*/
BOOL equipment_common_overwrite = TRUE;
EQUIPMENT equipment[] = {
{"Trigger", /* equipment name */
{1, 0, /* event ID, trigger mask */
"SYSTEM", /* event buffer */
EQ_POLLED, /* equipment type */
0, /* event source */
"MIDAS", /* format */
TRUE, /* enabled */
RO_RUNNING, /* read only when running */
100, /* poll for 100ms */
0, /* stop run after this event limit */
0, /* number of sub events */
0, /* don't log history */
"", "", "",},
read_trigger_event, /* readout routine */
},
{""}
};
INT frontend_init() { return SUCCESS; }
INT frontend_exit() { return SUCCESS; }
INT begin_of_run(INT run_number, char *error) { return SUCCESS; }
INT end_of_run(INT run_number, char *error) { return SUCCESS; }
INT pause_run(INT run_number, char *error) { return SUCCESS; }
INT resume_run(INT run_number, char *error) { return SUCCESS; }
INT frontend_loop() { return SUCCESS; }
INT interrupt_configure(INT cmd, INT source, POINTER_T adr) { return SUCCESS; }
/*------------------------------------------------------------------*/
INT poll_event(INT source, INT count, BOOL test)
{
int i;
DWORD flag;
for (i = 0; i < count; i++) {
/* poll hardware and set flag to TRUE if new event is available */
flag = TRUE;
if (flag)
if (!test)
return TRUE;
}
return 0;
}
/*-- Event readout -------------------------------------------------*/
INT read_trigger_event(char *pevent, INT off)
{
UINT8 *pdata;
bk_init32(pevent);
bk_create(pevent, "ADC0", TID_UINT32, (void **)&pdata);
// generate 7 MB of dummy data
pdata += (7 * 1024 * 1024);
bk_close(pevent, pdata);
return bk_size(pevent);
}
|