> I'm looking into using MIDAS for an experiment that creates one large event
> (20MB or more) every second.
Hi, there - 20 Mbyte event at 1/sec is not so large these days. (Well, depending on your hardware).
Using typical 1-2 year old PC hardware, 20 M/sec to local disk should work right away. Sending data from a
remote front end (through the mserver), or writing to a remote disk (NFS, etc) - will of course requre a GigE
network connection.
By default, MIDAS is configured for using about 1-2 Mbyte events, so for your case, you will need to:
- increase the event size limits in your frontend,
- increase /Experiment/MAX_EVENT_SIZE in ODB
- increase the size of the SYSTEM event buffer (/Experiment/Buffer sizes/SYSTEM in ODB)
I generally recommend sizing the SYSTEM event buffer to hold a few seconds worth of data (ot
accommodate any delays in writing to local disk - competing reads, internal delays of the disks, etc).
So for 20 M/s, the SYSTEM buffer size should be about 40-60 Mbytes.
For your case, you also want to buffer 3-5-10 events, so the SYSTEM buffer size would be between 100 and
200 MBytes.
Assuming you have between 8-16-32 GBytes of RAM, this should not be a problem.
One the other hand, if you are running on a low-power ("green") ARM system with 1 Gbyte of RAM and a
1GHz CPU, you should be able to handle the data rate of 20 Mbytes/sec, as long as your network and
storage can handle it - I see GigE ethernet running at about 30-40 Mbytes/sec, so you should be okey,
but local storage to SD flash is only about 10 Mbytes/sec - too slow. You can try USB-attached HDD or SSD,
this should run at up to 30-40 Mbytes/sec. I would expect no problems with this rate from MIDAS, as long
as you can fit into your 1 GByte of RAM - obviously your SYSTEM buffer will have to be a little smaller than
on a full-featured PC.
More information on MIDAS event size limits is here (as already reported by Stefan)
https://midas.triumf.ca/MidasWiki/index.php/Event_Buffer
Let us know how it works out for you.
K.O. |