I think I can offer some insight into your problems:
1) your mhist crash is due to the ODB timeout, it is probably set to 30 seconds in ODB /programs/mhist. you will
have to make it biigger.
2) 1.5 years of files. yes. I have 10 years of files for ALPHA at CERN. and the number of files is a problem.
But it should be better than the old system with 3 files per day (1000 files per year).
One solution you can try is symlinks. Assuming you have 10 years of history files in 10 per-year directory, you
symlink as many of them as you need into the "current" directory, then remove the symlinks.
Why remove the symlinks? I use "ls" to read the list of history files and Unix/Linux does not have a syscall to
"give me the 100 files with the newest mtime". I have to read the whole directory and that takes forever (if ZFS
on HDD), it is quick with ZFS on SSD if ZFS cache is hot (you can have a cron job do "ls" every 5 minutes to
keep the ZFS cache hot).
Now that I wrote the above, I think I see a way to make it "automatic", let me ponder this. (plus I always
wanted to implement compressed history files (using "free" lz4)).
K.O.
I am having some trouble with mhist. I suppose that the problems are at least partially due to our specific
needs which might exceed what has been tested. For context, in MEG II we have some 10^4 history variables in ~30
different events.
1. mhist -l crashes. After displaying around 7000 lines, I get the following error message:
[CODE]
[mhist,ERROR] [midas.cxx:5949:bm_validate_client_index,ERROR] My client index 10 in buffer 'SYSMSG'
is invalid: client name '', pid 0 should be my pid 3773321
[mhist,ERROR] [midas.cxx:5952:bm_validate_client_index,ERROR] Maybe this client was removed by a
timeout. See midas.log. Cannot continue, aborting...
Aborted (core dumped)
[/CODE]
Timing the execution shows around 33 seconds before the process is aborted.
I'm not sure if this would actually fix the problem, but while trying to circumvent the issue, I tried the
following: [CODE]mhist -e "Xenon" -l[/CODE] This doesn't seem to be implemented. Listing only the variables of a
single event would be nice
regardless of our specific issue.
2. mhist and history files.
We have a directory directory with about 2500 history files (mhf_...dat) for the past 1.5 years. Older
history files are archived in other directories with similar numbers of files. When trying to access them, I
encountered two issues:
It seems like it is not possible to pass a "history directory" as an argument. To dump the history for a full
year in the archive directory, I would need to run mhist many times with -f and then combine all the dumps.
If it really does not work, please consider this a feature request.
Also, even using single files does not work at the moment:
[CODE]
$ mhist -e "Xenon" -v "Det XeTmp 0-0" -t 100000 -s 200101 -p 250101 -f
/data2/history/2022/mhf_1644698398_20220212_xenon.dat
ID 980316009, Aug 13 19:10:56, size 1851749486
[/CODE]
This command was supposed to show me the rough time frame covered in this particular history file. I was
informed that the history files are in the new "FILE" format and mhist might not work with them properly.
tl;dr
[LIST]
[*] Bug: mhist -l crashes
[*] Bug: mhist -f does not work with "FILE" history format
[*] Feature request: mhist -e "Name" -l to only show variables of event "Name"
[*] Feature request: Set temporary history dir with a flag
[/LIST]
Lukas[/quote] |