Back Midas Rome Roody Rootana
  Midas DAQ System  Not logged in ELOG logo
Entry  11 Feb 2014, Randolf Pohl, Forum, Huge events (>10MB) every second or so 
    Reply  11 Feb 2014, Stefan Ritt, Forum, Huge events (>10MB) every second or so 
    Reply  18 Feb 2014, Konstantin Olchanski, Forum, Huge events (>10MB) every second or so 
    Reply  01 Mar 2014, Randolf Pohl, Forum, Huge events (>10MB) every second or so big_event.tgz
       Reply  01 Mar 2014, Stefan Ritt, Forum, Huge events (>10MB) every second or so 
Message ID: 962     Entry time: 18 Feb 2014     In reply to: 956
Author: Konstantin Olchanski 
Topic: Forum 
Subject: Huge events (>10MB) every second or so 
> I'm looking into using MIDAS for an experiment that creates one large event
> (20MB or more) every second.

Hi, there - 20 Mbyte event at 1/sec is not so large these days. (Well, depending on your hardware).

Using typical 1-2 year old PC hardware, 20 M/sec to local disk should work right away. Sending data from a 
remote front end (through the mserver), or writing to a remote disk (NFS, etc) - will of course requre a GigE 
network connection.

By default, MIDAS is configured for using about 1-2 Mbyte events, so for your case, you will need to:

- increase the event size limits in your frontend,
- increase /Experiment/MAX_EVENT_SIZE in ODB
- increase the size of the SYSTEM event buffer (/Experiment/Buffer sizes/SYSTEM in ODB)

I generally recommend sizing the SYSTEM event buffer to hold a few seconds worth of data (ot 
accommodate any delays in writing to local disk - competing  reads, internal delays of the disks, etc).

So for 20 M/s, the SYSTEM buffer size should be about 40-60 Mbytes.

For your case, you also want to buffer 3-5-10 events, so the SYSTEM buffer size would be between 100 and 
200 MBytes.

Assuming you have between 8-16-32 GBytes of RAM, this should not be a problem.

One the other hand, if you are running on a low-power ("green") ARM system with 1 Gbyte of RAM and a 
1GHz CPU, you should be able to handle the data rate of 20 Mbytes/sec, as long as your network and 
storage can handle it - I see GigE ethernet running at about 30-40 Mbytes/sec, so you should be okey,
but local storage to SD flash is only about 10 Mbytes/sec - too slow. You can try USB-attached HDD or SSD, 
this should run at up to 30-40 Mbytes/sec. I would expect no problems with this rate from MIDAS, as long 
as you can fit into your 1 GByte of RAM - obviously your SYSTEM buffer will have to be a little smaller than 
on a full-featured PC.

More information on MIDAS event size limits is here (as already reported by Stefan)
https://midas.triumf.ca/MidasWiki/index.php/Event_Buffer

Let us know how it works out for you.

K.O.
ELOG V3.1.4-2e1708b5