Back Midas Rome Roody Rootana
  Midas DAQ System  Not logged in ELOG logo
Entry  31 Mar 2022, Stefan Ritt, Suggestion, Maximum ODB size 
    Reply  04 Apr 2022, Konstantin Olchanski, Suggestion, Maximum ODB size 
       Reply  27 Apr 2023, Marius Koeppel, Suggestion, Maximum ODB size 
          Reply  27 Apr 2023, Konstantin Olchanski, Suggestion, Maximum ODB size 
             Reply  27 Apr 2023, Marius Koeppel, Suggestion, Maximum ODB size 
                Reply  27 Apr 2023, Konstantin Olchanski, Suggestion, Maximum ODB size 
                Reply  27 Apr 2023, Konstantin Olchanski, Suggestion, Maximum ODB size 
                   Reply  28 Apr 2023, Marius Koeppel, Suggestion, Maximum ODB size create_time.pdffails.pdftest_odb.py
                      Reply  28 Apr 2023, Stefan Ritt, Suggestion, Maximum ODB size 
                         Reply  28 Apr 2023, Marius Koeppel, Suggestion, Maximum ODB size 
                         Reply  28 Apr 2023, Konstantin Olchanski, Suggestion, Maximum ODB size 
             Reply  27 Apr 2023, Stefan Ritt, Suggestion, Maximum ODB size 
                Reply  28 Apr 2023, Konstantin Olchanski, Suggestion, Maximum ODB size 
                   Reply  09 Jun 2023, Konstantin Olchanski, Suggestion, Maximum ODB size 
                      Reply  12 Jun 2023, Stefan Ritt, Suggestion, Maximum ODB size 
                         Reply  12 Jun 2023, Konstantin Olchanski, Suggestion, Maximum ODB size 
                            Reply  13 Jun 2023, Stefan Ritt, Suggestion, Maximum ODB size 
                               Reply  13 Jun 2023, Marius Koeppel, Suggestion, Maximum ODB size 
                                  Reply  13 Jun 2023, Konstantin Olchanski, Suggestion, Maximum ODB size 
                               Reply  13 Jun 2023, Konstantin Olchanski, Suggestion, Maximum ODB size 
                                  Reply  13 Jun 2023, Stefan Ritt, Suggestion, Maximum ODB size 
                                     Reply  13 Jun 2023, Konstantin Olchanski, Suggestion, Maximum ODB size 
                                        Reply  13 Jun 2023, Stefan Ritt, Suggestion, Maximum ODB size 
                                           Reply  15 Jun 2023, Konstantin Olchanski, Suggestion, Maximum ODB size 
                                              Reply  28 Jul 2023, Stefan Ritt, Suggestion, Maximum ODB size 
                                                 Reply  09 Aug 2023, Konstantin Olchanski, Suggestion, Maximum ODB size 
Message ID: 2535     Entry time: 13 Jun 2023     In reply to: 2529     Reply to this: 2537
Author: Konstantin Olchanski 
Topic: Suggestion 
Subject: Maximum ODB size 
> > I remember the same, but I tracked it down in git to the very first commit, and there is no if() there,
> > odb is saved to .ODB.SHM on every client shutdown, not just the last client. I guess we both misremebered.

small problem. build an experiment, start taking data, observe how ODB is never saved to disk because the "last client" never stops. as bonus, crash 
the computer, observe how all changes to ODB are now lost. if mlogger is configured to save odb.json at the end of run, and to write ODB dumps at 
begin and end of every data file, you can recover some of the lost data.

for better effect, ODB should be dumped to disk at periodic intervals. but. current implementation writes odb to disk while holding the ODB 
semaphore, which means all ODB access stops for the duration, specifically, there will be gaps in the history because mlogger cannot read history 
data from ODB.

a better implementation could take the ODB lock, make a copy of ODB shared memory, release the ODB lock, complete writing to disk without holding the 
lock. protection is needed against 100 midas programs trying to do this all at the same time. computers with 0.5 GB RAM (many ARM FPGA SoCs) will be 
limited to ~100 Mbyte ODB). plus deal with memory allocation failures when taking a copy of a 2GB ODB.

in theory, the mmap() shared memory (already implemented in midas) does this automatically, but we lose control
over disk writes, we see some OSes write odb to disk "too often" and at wrong times, i.e. while we are in the middle
of creating or deleting something. current sequence of open(), atomic write() and close() ensures ODB.SHM always
contains a valid odb. (minus loss of OS and disk caches to crash or power loss).

K.O.
ELOG V3.1.4-2e1708b5