A set of improvements to mlogger is in:
a) event buffer (SYSTEM) size up to 2GB
b) test version of LZ4 high speed compression, support for bzip2 and pbzip2
Details:
a) previously contents of shared memory buffers (SYSTEM, SYSMSG, etc) were periodically saved to disk
files SYSTEM.SHM, SYSMSG.SHM, etc. This was not workable for large event buffers - reading/writing 2GB
of data takes quite some time. We have decided that saving buffer contents to disk is no longer necessary
and ss_shm_close() no longer writes SYSTEM.SHM, SYSMSG.SHM, etc. From now on you will still see these
files created, but size will be 0. The file ODB.SHM is not affected by this - ODB contents is saved to
ODB.SHM via ss_shm_flush().
b) as a rework of mlogger file output drivers (using chainable c++ classes), test versions of new
compression algorithms have been added. In the present test version, they are controlled by the value of
"compression".
The plan is to ultimately have following outputs from the mlogger:
- ROOT output - save as before, but you have to use rmlogger executable
- FTP output - for high speed write over the network
- .mid output for uncompressed data
- .mid.gz - gzip1 compressed data - best compromise between compression ratio and speed - will be the
new default
- .mid.bz2 via pbzip2 (parallel bzip2) - maximum compression ratio
- .mid.lz4 - lz4 compression for high speed data taking - maximum compression speed
The current test version implements the following selections of "compression":
80 - ROOT output through the new driver (use rmlogger executable)
98 - null output (no file written)
99 - uncompressed disk output
100 - lz4 comression
200 - piped bzip2 compression
201 - piped pbzip2 compression
300 - gzip compression
301 - gzip1 compression
309 - gzip9 compression
in addition the old selections are still available:
0 - uncompressed output
1 - gzip1 compression
9 - gzip9 compression
The final implementation will include a better way to configure the mlogger output channels.
K.O. |