I finished debugging the code I wrote to check and repair the ODB key area and data area free lists, this
should fix the famous "ODB is full" error.
I have one computer that runs MIDAS that seems to have a hardware problem and crashes about every 1-2
weeks. After each crash MIDAS ODB is always corrupted and I started writing the (long desired) free list
checker as a way to try to understand the nature of the corruption. (more on that at the end)
Once I had a reliable checker and I remembered how the ODB free lists work,
writing the repair code was easy.
I tested it on my collection of corrupted .ODB.SHM files, and in each case,
after repair, I can start odbedit, look at ODB data and save it to JSON file.
Of course every corruption loses some data, for full recovery, ODB must be reloaded from a backup. But at
least one does not have to kill ODB and recreate it from scratch anymore (in theory).
One type of corruption I cannot fix: if two ODB entries seem to overlap, in the key area (I cannot repair),
or in the data area (I delete one of the overlapping entries).
Because repair of the ODB free list can be dangerous, I disabled it by default.
To run odbedit in "repair" mode, do this (we could also make it an odbedit option)
MIDAS_ODB_REPAIR_FREE_LIST=1 ./bin/odbedit
While looking at corrupted ODB files, I fixed two more buglets:
1) if ODB entry has num_values set to 0 (unusual, but not an error), save odb.json produces invalid JSON
2) if /Experiment/Buffer sizes/SYSMSG has a bogus value (I tend to see "0"), odbedit would not start and I
cannot edit ODB to fix the problem.
I have this code on a private branch and I will merge and push it in the next few days.
P.S.
Cause of corruption is still not clear, best guess is that there is a hardware memory failure, but this
explanation requires a difficult sequence: memory goes bad, ODB is corrupted, linux and ZFS still running,
MIDAS writes corrupted ODB to file .ODB.SHM, linux and ZFS still running, more memory corruption, linux
dies, reboots/resets.
Alternatively, memory corruption affects the ZFS cache, ODB in RAM is okey, but .ODB.SHM in ZFS cache is
corrupted, but still gets written to disk, then more memory corruption kills linux. The ZFS volume itself
does not show any corruption, "zfs scrub" comes back clean.
Garden variety "buggy midas frontend corrupted ODB" is not good enough to explain computer reset/reboot.
K.O. |