 27 Apr 2023, Konstantin Olchanski, Suggestion, Maximum ODB size
|
> You said the writing into .odb is buggy. Do you mean it’s buggy in general or only in this specific case?
> We save the ODB most of the time in the .odb format.
|
|
27 Apr 2023, Konstantin Olchanski, Suggestion, Maximum ODB size
|
my vote is to bump the ODB size limit to 1999*1000*1000 (not quite 2GB). but this needs to be tested. especially save and restore from ODB, XML and JSON
files, including how long it takes to save and load a 1.9GB ODB. K.O. |
|
27 Apr 2023, Stefan Ritt, Suggestion, Maximum ODB size
|
> Congratulations. created != "it works".
Two other tings to consider:
|
28 Apr 2023, Marius Koeppel, Suggestion, Maximum ODB size 
|
> my vote is to bump the ODB size limit to 1999*1000*1000 (not quite 2GB). but this needs to be tested. especially save and restore from ODB, XML and JSON
files, including how long it takes to save and load a 1.9GB ODB. K.O.
|
|
28 Apr 2023, Stefan Ritt, Suggestion, Maximum ODB size
|
> Is this maybe related to what Stefan said about the run start - so that odbedit needs some time to load the bigger ODB?
At the run start mlogger writes the ODB to the .mid file. This needs conversion (binary ODB -> XML ASCII) which can take time.
|
|
28 Apr 2023, Marius Koeppel, Suggestion, Maximum ODB size
|
> At the run start mlogger writes the ODB to the .mid file. This needs conversion (binary ODB -> XML ASCII) which can take time.
> This does NOT depend on the ODB size, but on the ODB *content*. Every key in the ODB takes time to convert. So if your ODB as 1.5 GB
> but only a few keys, this is still fast. Only if you have 200 million keys int he ODB, then mlogger takes lots of time to convert
|
|
28 Apr 2023, Konstantin Olchanski, Suggestion, Maximum ODB size
|
> > Congratulations. created != "it works".
>
> Two other tings to consider:
|
|
28 Apr 2023, Konstantin Olchanski, Suggestion, Maximum ODB size
|
> > Is this maybe related to what Stefan said about the run start - so that odbedit needs some time to load the bigger ODB?
>
> At the run start mlogger writes the ODB to the .mid file. This needs conversion (binary ODB -> XML ASCII) which can take time.
|
|
09 Jun 2023, Konstantin Olchanski, Suggestion, Maximum ODB size
|
> > 1) The ODB shared memory is dumped into a binary file (".ODB.SHM") after the last client finished ...
correction: ODB shared memory is saved to .ODB.SHM each time a client stops, this is db_close_database().
|
|
12 Jun 2023, Stefan Ritt, Suggestion, Maximum ODB size
|
> correction: ODB shared memory is saved to .ODB.SHM each time a client stops, this is db_close_database().
The original design of the midas shared memory (back in the 1990's) was that the ODB shared memory file gets
|
|
12 Jun 2023, Konstantin Olchanski, Suggestion, Maximum ODB size
|
> > correction: ODB shared memory is saved to .ODB.SHM each time a client stops, this is db_close_database().
>
> The original design of the midas shared memory (back in the 1990's) was that the ODB shared memory file gets
|
|
13 Jun 2023, Stefan Ritt, Suggestion, Maximum ODB size
|
> I remember the same, but I tracked it down in git to the very first commit, and there is no if() there,
> odb is saved to .ODB.SHM on every client shutdown, not just the last client. I guess we both misremebered.
|
|
13 Jun 2023, Marius Koeppel, Suggestion, Maximum ODB size
|
> BTW, how do I resize the ODB. I remember we discussed this some time ago, and concluded that odbedit needs a resize flag. Has this even been
> done? If not, what is the "official" way to resize the ODB. We had some documentation about that some time ago, but I can't find it anymore.
|
|
13 Jun 2023, Konstantin Olchanski, Suggestion, Maximum ODB size
|
> > I remember the same, but I tracked it down in git to the very first commit, and there is no if() there,
> > odb is saved to .ODB.SHM on every client shutdown, not just the last client. I guess we both misremebered.
|
|
13 Jun 2023, Konstantin Olchanski, Suggestion, Maximum ODB size
|
> > BTW, how do I resize the ODB.
ODB cannot be resized "online". Everything has to stop, save content to odb.json, get rid of old ODB.SHM, ensure ODB shared memory is destroyed (SysV or |
|
13 Jun 2023, Stefan Ritt, Suggestion, Maximum ODB size
|
> small problem. build an experiment, start taking data, observe how ODB is never saved to disk because the "last client" never stops. as bonus, crash
|
|
13 Jun 2023, Konstantin Olchanski, Suggestion, Maximum ODB size
|
>
> > small problem. build an experiment, start taking data, observe how ODB is never saved to disk because the "last client" never stops. as bonus, crash
|
|
13 Jun 2023, Stefan Ritt, Suggestion, Maximum ODB size
|
> are you sure? when/how often does "last midas program finishes" happen? it does not happen on a system crash, not on power loss, not on "shutdown -r
now"
|
|
15 Jun 2023, Konstantin Olchanski, Suggestion, Maximum ODB size
|
>
> > are you sure? when/how often does "last midas program finishes" happen? it does not happen on a system crash, not on power loss, not on "shutdown -r
now"
|
|
28 Jul 2023, Stefan Ritt, Suggestion, Maximum ODB size
|
> RFE filed:
> https://bitbucket.org/tmidas/midas/issues/367/odb-should-be-saved-to-disk-periodically
|