Back Midas Rome Roody Rootana
  Midas DAQ System  Not logged in ELOG logo
Entry  31 Mar 2022, Stefan Ritt, Suggestion, Maximum ODB size 
    Reply  04 Apr 2022, Konstantin Olchanski, Suggestion, Maximum ODB size 
       Reply  27 Apr 2023, Marius Koeppel, Suggestion, Maximum ODB size 
          Reply  27 Apr 2023, Konstantin Olchanski, Suggestion, Maximum ODB size 
             Reply  27 Apr 2023, Marius Koeppel, Suggestion, Maximum ODB size 
                Reply  27 Apr 2023, Konstantin Olchanski, Suggestion, Maximum ODB size 
                Reply  27 Apr 2023, Konstantin Olchanski, Suggestion, Maximum ODB size 
                   Reply  28 Apr 2023, Marius Koeppel, Suggestion, Maximum ODB size create_time.pdffails.pdftest_odb.py
                      Reply  28 Apr 2023, Stefan Ritt, Suggestion, Maximum ODB size 
                         Reply  28 Apr 2023, Marius Koeppel, Suggestion, Maximum ODB size 
                         Reply  28 Apr 2023, Konstantin Olchanski, Suggestion, Maximum ODB size 
             Reply  27 Apr 2023, Stefan Ritt, Suggestion, Maximum ODB size 
                Reply  28 Apr 2023, Konstantin Olchanski, Suggestion, Maximum ODB size 
                   Reply  09 Jun 2023, Konstantin Olchanski, Suggestion, Maximum ODB size 
                      Reply  12 Jun 2023, Stefan Ritt, Suggestion, Maximum ODB size 
                         Reply  12 Jun 2023, Konstantin Olchanski, Suggestion, Maximum ODB size 
                            Reply  13 Jun 2023, Stefan Ritt, Suggestion, Maximum ODB size 
                               Reply  13 Jun 2023, Marius Koeppel, Suggestion, Maximum ODB size 
                                  Reply  13 Jun 2023, Konstantin Olchanski, Suggestion, Maximum ODB size 
                               Reply  13 Jun 2023, Konstantin Olchanski, Suggestion, Maximum ODB size 
                                  Reply  13 Jun 2023, Stefan Ritt, Suggestion, Maximum ODB size 
                                     Reply  13 Jun 2023, Konstantin Olchanski, Suggestion, Maximum ODB size 
                                        Reply  13 Jun 2023, Stefan Ritt, Suggestion, Maximum ODB size 
                                           Reply  15 Jun 2023, Konstantin Olchanski, Suggestion, Maximum ODB size 
                                              Reply  28 Jul 2023, Stefan Ritt, Suggestion, Maximum ODB size 
                                                 Reply  09 Aug 2023, Konstantin Olchanski, Suggestion, Maximum ODB size 
Message ID: 2488     Entry time: 28 Apr 2023     In reply to: 2483     Reply to this: 2525
Author: Konstantin Olchanski 
Topic: Suggestion 
Subject: Maximum ODB size 
> > Congratulations. created != "it works".
> 
> Two other tings to consider:
> 
> 1) The ODB shared memory is dumped into a binary file (".ODB.SHM") after the last client finished and read if the first client starts, to get it persistent. 
> So this could slow down starting and stopping, but only the first client, so I guess it's not an issue.
>

typical disk writing speed is 100-1000 Mbytes/sec, so writing 1 GB .ODB.SHM will take 1-10 seconds. NFS over 1gige network is 100 Mbytes/sec, so 10 seconds to 
write .ODB.SHM. embedded ARM write speed to SD flash can be as low as 10 Mbytes/sec, so up to 100 seconds.

> 
> 2) Traditionally, the ODB gets dumped to the .mid file at the beginning and end of every run, so that one know the exact configuration of the experiment
> for offline analysis. This can be turned off of course, but most experiments use it. If the ODB is dumped in any ASCII format, this can take quite long.
> Assume it takes 10 seconds at the beginning of each run, and we take a run every five minutes. Then we loose 48 mins of precious beam time every day.
> 

new default is to save as JSON, (as of my last measurement) JSON encoder is faster than the XML (and ODB?) encoder, by default result is compressed by GZIP-1 (66 
Mbytes/sec is my old benchmark, should remeasure on new DDR5 machines), compressed JSON is written .mid.gz file at disk speed (as above). Alternatively, use LZ4 
compression, runs roughly at memcpy() speed, less compression, written to .mid.lz4 at disk speed.

if data storage is ZFS, ZFS built-in LZ4 compression is now enabled by default, so result writing uncompressed .mid file (no compression of ODB dump), should be 
roughly same as when using MIDAS LZ4 compression and writing .mid.lz4.

bottom line, I need to remeasure gzip and lz4 compression speeds on new computers (DDR4 AMD 5000 series and DDR5 AMD 7000 series).

K.O.
ELOG V3.1.4-2e1708b5