ID |
Date |
Author |
Topic |
Subject |
1307
|
25 Jul 2017 |
Stefan Ritt | Info | Current git repository "develop" branch broken | Dear all,
we are currently undergoing major modifications in the way mhttpd is working. I realized that
we are now at a state where mhttpd is currently broken, and it will take a few weeks in order to
get everything converted to the new scheme we plan to use. Therefore I moved the git branch
"master" to the last known stable version of midas. So for any practical purpose, please do
NOT update your "develop" branch until further notice. To get the last stable version, you can
do a
$ git checkout master
which moves you right before we started to make major modifications. Once we are finished,
we will announce this here in the forum.
Best regards,
Stefan |
9
|
10 Mar 2004 |
Jan Wouters | | Creation of secondary Midas output file. | Dear Midas Team,
I have run into a problem with Midas and was wondering if you could explain what I
am doing wrong. I have included a simple demo to illustrate what I am doing and
can send a small input data file if needed.
WHAT I AM TRYING TO DO:
Every midas event for the DANCE experiment consists of many physics events. I am
trying to create a secondary mid file where the event boundaries are now the
physics events rather than the midas events. This secondary mid file will be
analyzed using a second stage midas analyzer.
For the demo, I use the data from EV02 (one of our 15 frontends), which consists of a
variable number of fixed length structures where each structure contains the data for
one crystal from the DANCE detector.
I treat each crystal as a separate physics event and write it out in the TREK bank,
which is a demo calculated output bank, as a separate event.
(The only difference between this demo and our real system is that we would include
all the crystals from the other frontends that have approximately the same time stamp
in the output bank. Thus the output bank would consist of a varing number of
crystals in one event rather than the fixed one crystal per event used in this demo.)
THE CHANGES TO analyzer.c AND adccalib.c
I loop through the EV02 bank examining each crystal structure in turn. I calculate
"calibrated" parameters and put them into an output bank called TREK. The unusual
part of this example is that the TREK bank is no longer part of the main list of input
banks, ana_trigger_bank_list[]. Instead it is now part of a new bank list called
ana_physics_bank_list[]. See the analyzer.c file for this definition.
In adccalib.c I create the space for this new bank as follows.
EVENT_HEADER gPhysicsEventHeaders[ MAX_EVENT_SIZE / sizeof(
EVENT_HEADER ) ];
WORD* gPhysicsEventData = ( WORD * )( gPhysicsEventHeaders + 1 );
In the adc_calib routine I create the bank header as follows. Note that the serial
numbers will restart at 0 at the beginning of each midas event. Should I let the serial
number increment monotonically until the end of the run?:
gPhysicsEventHeaders->serial_number = (DWORD) - 1;
gPhysicsEventHeaders->event_id = 2;
gPhysicsEventHeaders->trigger_mask = 0;
gPhysicsEventHeaders->time_stamp = pheader->time_stamp;
In a loop that loops through all the crystals contained in EV02, I extract each crystal,
calibrate it, and store it in a TREK structure. In creating the TREK bank I assume that
each one will be a separate physics event thus I update the event serial number and
use bk_init32 to initialize the memory.
for ( short i = 0; i < nItems; i++ )
{ ++(gPhysicsEventHeaders->serial_number); // Update serial number.
bk_init32( gPhysicsEventData ); // Initialize storage.
bk_create( gPhysicsEventData, "TREK", TID_STRUCT, &trek );
trek->one = (double) pev->areahg * 1.0;
trek->two = (float) pev->timelo * 1.0;
bk_close( gPhysicsEventData, trek+1 );
pev++; // Loop to next crystal's data.
}
The output bank should consist of multiple events for each individual EV02 midas
input event.
As far as I can tell the code compiles and runs fine, but I get no data in the .mid
output file except for the ODB. I have a print statement at the beginning of each
midas event stating how many crystals were found in the EV02 bank. I also print out
the calibrated value for each crystal as it is being placed in its own TREK output
bank. The data appears correct.
I cannot place TREK in the input bank the way it normally is done in the examples
because there is not a one-to-one correspondence between a midas event and a
true physics event. Instead one midas event has many physics events. Thus the
output bank needs to be in a new memory area so that I can create a custom header
and increment the serial number properly for each event. Our follow-on analysis
using a second Midas analyzer only needs to analyze one physics event at a time
rather than one Midas event at a time, which is why we are going to all the trouble to
get this paradigm working.
I include all the code for this very simple example.
RUNNING THE CODE:
To run the example just use the run01220.mid file I will send:
./analyzer -i run01220.mid.gz -o run01220out.mid -c settings.odb_cfg -n 50
The only thing done by the settings.odb_cfg file is to turn on the TREK output bank. I
have verified that the bank is on.
SUMMARY:
I believe that I must not be creating the new TREK output bank correctly so that
midas understands that the event-by-event calculated physics data should be written
out event-by-event. I have pointed out several places in the above discussion where
I might be making a mistake.
I would like to get both this example running and a similar which create Root trees,
though the Root trees are of secondary importance. With this example I can finish
writing the second stage analyzer and get the DANCE collaboration moving forward
with their analysis. Currently, we cannot use this paradigm because I cannot create
a secondary mid file in our stage one analysis. I would be very grateful if you could
take a look at this example and tell me what I am doing incorrectly.
Jan |
10
|
10 Mar 2004 |
Stefan Ritt | | Creation of secondary Midas output file. | Dear Jan,
I had a look at your code. You create a gPhysicsEventHeader array, fill it, and expect the
framework to write it to disk. But how can the framework "guess" that you want your private
global array being written? Unfortunately it cannot do magic!
Do do what you want, you have to write a "secondary" midas file yourself. I modified your
code to do that. First, I define the event storage like
BYTE gSecEvent[ MAX_EVENT_SIZE ];
EVENT_HEADER *gPhysicsEventHeader = (EVENT_HEADER *) gSecEvent;
WORD* gPhysicsEventData = ( WORD * )( gPhysicsEventHeader + 1 );
I use gSecEvent as a BYTE array, since it only contains one avent at a time, so this is more
appropriate. Then, in the BOR routine, I open a file:
sprintf(str, "sec%05d.mid", run_number);
sec_fh = open(str, O_CREAT | O_RDWR | O_BINARY, 0644);
and close it in the EOR routine
close(sec_fh);
The event routine now manually fills events into the secondary file:
/* write event to secondary .mid file */
gPhysicsEventHeader->data_size = bk_size(gPhysicsEventData);
write(sec_fh, gPhysicsEventHeader, sizeof(EVENT_HEADER)+bk_size(gPhysicsEventData));
Note that this code is placed *inside* the for() loop over nItems, so for each detector you
create and event and write it.
That's all you need, the full file adccalib.c is attached. I tried to produce a sec01220.mid
file and was able to read it back with the mdump utility.
Best regards,
Stefan |
11
|
11 Mar 2004 |
Renee Poutissou | | Creation of secondary Midas output file. | Jan ,
Do you need to log this stage 1 output? If not, you would use the
eventbuilder mechanism to create your stage 2 events.
I use the eventbuilder mechanism with success for my TWIST experiment.
Renee |
819
|
04 Jul 2012 |
Konstantin Olchanski | Bug Report | Crash after recursive use of rpc_execute() | I am looking at a MIDAS kaboom when running out of space on the data disk - everything was freezing
up, even the VME frontend crashed sometimes.
The freeze was traced to ROOT use in mlogger - it turns out that ROOT intercepts many signal handlers,
including SIGSEGV - but instead of crashing the program as God intended, ROOT SEGV handler just hangs,
and the rest of MIDAS hangs with it. One solution is to always build mlogger without ROOT support -
does anybody use this feature anymore? Or reset the signal handlers back to the default setting somehow.
Freeze fixed, now I see a crash (seg fault) inside mlogger, in the newly introduced memmove() function
inside the MIDAS RPC code rpc_execute(). memmove() replaced memcpy() in the same place and I am
surprised we did not see this crash with memcpy().
The crash is caused by crazy arguments passed to memmove() - looks like corrupted RPC arguments
data.
Then I realized that I see a recursive call to rpc_execute(): rpc_execute() calls tr_stop() calls cm_yield() calls
ss_suspend() calls rpc_execute(). The second rpc_execute successfully completes, but leave corrupted
data for the original rpc_execute(), which happily crashes. At the moment of the crash, recursive call to
rpc_execute() is no longer visible.
Note that rpc_execute() cannot be called recursively - it is not re-entrant as it uses a global buffer for RPC
argument processing. (global tls_buffer structure).
Here is the mlogger stack trace:
#0 0x00000032a8032885 in raise () from /lib64/libc.so.6
#1 0x00000032a8034065 in abort () from /lib64/libc.so.6
#2 0x00000032a802b9fe in __assert_fail_base () from /lib64/libc.so.6
#3 0x00000032a802bac0 in __assert_fail () from /lib64/libc.so.6
#4 0x000000000041d3e6 in rpc_execute (sock=14, buffer=0x7ffff73fc010 "\340.", convert_flags=0) at
src/midas.c:11478
#5 0x0000000000429e41 in rpc_server_receive (idx=1, sock=<value optimized out>, check=<value
optimized out>) at src/midas.c:12955
#6 0x0000000000433fcd in ss_suspend (millisec=0, msg=0) at src/system.c:3927
#7 0x0000000000429b12 in cm_yield (millisec=100) at src/midas.c:4268
#8 0x00000000004137c0 in close_channels (run_number=118, p_tape_flag=0x7fffffffcd34) at
src/mlogger.cxx:3705
#9 0x000000000041390e in tr_stop (run_number=118, error=<value optimized out>) at
src/mlogger.cxx:4148
#10 0x000000000041cd42 in rpc_execute (sock=12, buffer=0x7ffff73fc010 "\340.", convert_flags=0) at
src/midas.c:11626
#11 0x0000000000429e41 in rpc_server_receive (idx=0, sock=<value optimized out>, check=<value
optimized out>) at src/midas.c:12955
#12 0x0000000000433fcd in ss_suspend (millisec=0, msg=0) at src/system.c:3927
#13 0x0000000000429b12 in cm_yield (millisec=1000) at src/midas.c:4268
#14 0x0000000000416c50 in main (argc=<value optimized out>, argv=<value optimized out>) at
src/mlogger.cxx:4431
K.O. |
820
|
04 Jul 2012 |
Konstantin Olchanski | Bug Report | Crash after recursive use of rpc_execute() | > ... I see a recursive call to rpc_execute(): rpc_execute() calls tr_stop() calls cm_yield() calls
> ss_suspend() calls rpc_execute()
> ... rpc_execute() cannot be called recursively - it is not re-entrant as it uses a global buffer
It turns out that rpc_server_receive() also need protection against recursive calls - it also uses
a global buffer to receive network data.
My solution is to protect rpc_server_receive() against recursive calls by detecting recursion and returning SS_SUCCESS (to ss_suspend()).
I was worried that this would cause a tight loop inside ss_suspend() but in practice, it looks like ss_suspend() tries to call
us about once per second. I am happy with this solution. Here is the diff:
@@ -12813,7 +12815,7 @@
/********************************************************************/
-INT rpc_server_receive(INT idx, int sock, BOOL check)
+INT rpc_server_receive1(INT idx, int sock, BOOL check)
/********************************************************************\
Routine: rpc_server_receive
@@ -13047,7 +13049,28 @@
return status;
}
+/********************************************************************/
+INT rpc_server_receive(INT idx, int sock, BOOL check)
+{
+ static int level = 0;
+ int status;
+ // Provide protection against recursive calls to rpc_server_receive() and rpc_execute()
+ // via rpc_execute() calls tr_stop() calls cm_yield() calls ss_suspend() calls rpc_execute()
+
+ if (level != 0) {
+ //printf("*** enter rpc_server_receive level %d, idx %d sock %d %d -- protection against recursive use!\n", level, idx, sock, check);
+ return SS_SUCCESS;
+ }
+
+ level++;
+ //printf(">>> enter rpc_server_receive level %d, idx %d sock %d %d\n", level, idx, sock, check);
+ status = rpc_server_receive1(idx, sock, check);
+ //printf("<<< exit rpc_server_receive level %d, idx %d sock %d %d, status %d\n", level, idx, sock, check, status);
+ level--;
+ return status;
+}
+
/********************************************************************/
INT rpc_server_shutdown(void)
/********************************************************************\
ladd02:trinat~/packages/midas>svn info src/midas.c
Path: src/midas.c
Name: midas.c
URL: svn+ssh://svn@savannah.psi.ch/repos/meg/midas/trunk/src/midas.c
Repository Root: svn+ssh://svn@savannah.psi.ch/repos/meg/midas
Repository UUID: 050218f5-8902-0410-8d0e-8a15d521e4f2
Revision: 5297
Node Kind: file
Schedule: normal
Last Changed Author: olchanski
Last Changed Rev: 5294
Last Changed Date: 2012-06-15 10:45:35 -0700 (Fri, 15 Jun 2012)
Text Last Updated: 2012-06-29 17:05:14 -0700 (Fri, 29 Jun 2012)
Checksum: 8d7907bd60723e401a3fceba7cd2ba29
K.O. |
821
|
13 Jul 2012 |
Stefan Ritt | Bug Report | Crash after recursive use of rpc_execute() | > Then I realized that I see a recursive call to rpc_execute(): rpc_execute() calls tr_stop() calls cm_yield() calls
> ss_suspend() calls rpc_execute(). The second rpc_execute successfully completes, but leave corrupted
> data for the original rpc_execute(), which happily crashes. At the moment of the crash, recursive call to
> rpc_execute() is no longer visible.
This is really strange. I did not protect rpc_execute against recursive calls since this should not happen. rpc_server_receive() is linked to rpc_call() on the client side. So there cannot be
several rpc_call() since there I do the recursive checking (also multi-thread checking) via a mutex. See line 10142 in midas.c. So there CANNOT be recursive calls to rpc_execute() because
there cannot be recursive calls to rpc_server_receive(). But apparently there are, according to your stack trace.
So even if your patch works fine, I would like to know where the recursive calls to rpc_server_receive() come from. Since we have one subproces of mserver for each client, there should only
be one client connected to each mserver process, and the client is protected via the mutex in rpc_call(). Can you please debug this? I would like to understand what is going on there. Maybe
there is a deeper underlying problem, which we better solve, otherwise it might fall back on use in the future.
For debugging, you have to see what commands rpc_call() send and what rpc_server_receive() gets, maybe by writing this into a common file together with a time stamp.
SR |
618
|
18 Aug 2009 |
Denis Calvet | Suggestion | Could not create strings other than 32 characters with odbedit -c "..." command | Hi,
I am writing shell scripts to create some tree structure in an ODB. When
creating an array of strings, the default length of each string element is 32
characters. If odbedit is used interactively to create the array of strings,
the user is prompted to enter a different length if desired. But if the
command odbedit is called from a shell script, I did not succeed in passing
the argument to get a different length.
I tried:
odbedit -c "create STRING Test[8][40]"
Or:
odbedit -c "create STRING Test[8] 40"
Or:
odbedit -c "create STRING Test[8] \n 40"
etc. all produce an array of 8 strings with 32 characters each.
I haven't tried all possible syntaxes, but I suspect the length argument is
dropped. If it has not been fixed in a later release than the one I am using,
could this problem be looked at?
Thanks,
Denis.
|
627
|
03 Sep 2009 |
Stefan Ritt | Suggestion | Could not create strings other than 32 characters with odbedit -c "..." command | > Hi,
> I am writing shell scripts to create some tree structure in an ODB. When
> creating an array of strings, the default length of each string element is 32
> characters. If odbedit is used interactively to create the array of strings,
> the user is prompted to enter a different length if desired. But if the
> command odbedit is called from a shell script, I did not succeed in passing
> the argument to get a different length.
> I tried:
> odbedit -c "create STRING Test[8][40]"
> Or:
> odbedit -c "create STRING Test[8] 40"
> Or:
> odbedit -c "create STRING Test[8] \n 40"
> etc. all produce an array of 8 strings with 32 characters each.
> I haven't tried all possible syntaxes, but I suspect the length argument is
> dropped. If it has not been fixed in a later release than the one I am using,
> could this problem be looked at?
Ok, I added a command
odbedit -c "create STRING Test[8][40]"
which works now. Please update to SVN revision 4555 of odbedit.c
- Stefan |
634
|
06 Sep 2009 |
Exaos Lee | Suggestion | Could not create strings other than 32 characters with odbedit -c "..." command | > Ok, I added a command
>
> odbedit -c "create STRING Test[8][40]"
>
> which works now. Please update to SVN revision 4555 of odbedit.c
>
> - Stefan
If I want to create only one string, should I write like this:
odbedit -c "create STRING Test[] [256]"
OK. I need it. I will try the new odbedit. |
640
|
06 Sep 2009 |
Exaos Lee | Suggestion | Could not create strings other than 32 characters with odbedit -c "..." command | > > Ok, I added a command
> >
> > odbedit -c "create STRING Test[8][40]"
> >
> > which works now. Please update to SVN revision 4555 of odbedit.c
> >
> > - Stefan
>
> If I want to create only one string, should I write like this:
>
> odbedit -c "create STRING Test[] [256]"
>
> OK. I need it. I will try the new odbedit.
"create STRING test[1][256]" works. |
208
|
21 Apr 2005 |
Konstantin Olchanski | Suggestion | Correct MIDASSYS setting? | Current MIDAS versions nag me about setting the env.variable MIDASSYS to the
"midas installation directory", but I do not have one, so what should I set
MIDASSYS to? I checkout MIDAS from cvs into /home/olchansk/daq/midas, build it
there, run it from there. I never do "make install" (I am not "root" on every
machine; I am not the only MIDAS user on every machine). What should I set
MIDASSYS to? K.O. |
209
|
22 Apr 2005 |
Stefan Ritt | Suggestion | Correct MIDASSYS setting? | > Current MIDAS versions nag me about setting the env.variable MIDASSYS to the
> "midas installation directory", but I do not have one, so what should I set
> MIDASSYS to? I checkout MIDAS from cvs into /home/olchansk/daq/midas, build it
> there, run it from there. I never do "make install" (I am not "root" on every
> machine; I am not the only MIDAS user on every machine). What should I set
> MIDASSYS to? K.O.
Then set it to /home/olchansk/daq/midas. The reason for MIDASSYS is the same as
for ROOTSYS. Having it allows other packages like ROME to access the Midas source
code, include files and libraries. |
1905
|
07 May 2020 |
Estelle | Bug Report | Conflic between Rootana and midas about the redefinition of TID_xxx data types | Dear Midas and Rootana people,
We have tried to update our midas DAQ with the new TID definitions describe in https://midas.triumf.ca/elog/Midas/1871
And we have noticed an incompatibility of this new definitions with Rootana when reading an XmlOdb in our offline analyzer.
The problem comes from the function FindArrayPath in XmlOdb.cxx and the comparison of bank type as strings.
Ex: comparing the strings "DWORD" and "UNINT32"
An naive solution would be to print the number associated to the type (ex: '6' for DWORD/UNINT32), but that would mean changing Rootana and Midas source code. Moreover, it does decrease the readability of the XmlOdb file.
Thanks for your time.
Estelle |
1911
|
20 May 2020 |
Konstantin Olchanski | Bug Report | Conflic between Rootana and midas about the redefinition of TID_xxx data types | > Dear Midas and Rootana people,
>
> We have tried to update our midas DAQ with the new TID definitions describe in https://midas.triumf.ca/elog/Midas/1871
>
> And we have noticed an incompatibility of this new definitions with Rootana when reading an XmlOdb in our offline analyzer.
>
> The problem comes from the function FindArrayPath in XmlOdb.cxx and the comparison of bank type as strings.
> Ex: comparing the strings "DWORD" and "UNINT32"
>
> An naive solution would be to print the number associated to the type (ex: '6' for DWORD/UNINT32), but that would mean changing Rootana and Midas source code. Moreover, it does decrease the readability of the XmlOdb file.
>
Hi, it is unfortunate that a change was made in MIDAS that is incompatible with existing analysis software. I shall update the ROOTANA package to deal with this ASAP.
K.O. |
854
|
24 Jan 2013 |
Konstantin Olchanski | Info | Compression benchmarks | In the DEAP experiment, the normal MIDAS mlogger gzip compression is not fast enough for some data
taking modes, so I am doing tests of other compression programs. Here is the results.
Executive summary:
fastest compression is no compression (cat at 1800 Mbytes/sec - memcpy speed), next best are:
"lzf" at 300 Mbytes/sec and "lzop" at 250 Mbytes/sec with 50% compression
"gzip -1" at around 70 Mbytes/sec with around 70% compression
"bzip2" at around 12 Mbytes/sec with around 80% compression
"pbzip2", as advertised, scales bzip2 compression linearly with the number of CPUs to 46 Mbytes/sec (4
real CPUs), then slower to a maximum 60 Mbytes/sec (8 hyper-threaded CPUs).
This confirms that our original choice of "gzip -1" method for compression using zlib inside mlogger is
still a good choice. bzip2 can gain an additional 10% compression at the cost of 6 times more CPU
utilization. lzo/lzf can do 50% compression at GigE network speed and at "normal" disk speed.
I think these numbers make a good case for adding lzo/lzf compression to mlogger.
Comments about the data:
- time measured is the "elapsed" time of the compression program. it excludes the time spent flushing
the compressed output file to disk.
- the relevant number is the first rate number (input data rate)
- test machine has 32GB of RAM, so all I/O is cached, disk speed does not affect these results
- "cat" gives a measure of overall machine "speed" (but test file is too small to give precise measurement)
- "gzip -1" is the recommended MIDAS mlogger compression setting
- "pbzip2 -p8" uses 8 "hyper-threaded" CPUs, but machine only has 4 "real" CPU cores
<pre>
cat : time 0.2s, size 431379371 431379371, comp 0%, rate 1797M/s 1797M/s
cat : time 0.6s, size 1013573981 1013573981, comp 0%, rate 1809M/s 1809M/s
cat : time 1.1s, size 2027241617 2027241617, comp 0%, rate 1826M/s 1826M/s
gzip -1 : time 6.4s, size 431379371 141008293, comp 67%, rate 67M/s 22M/s
gzip : time 30.3s, size 431379371 131017324, comp 70%, rate 14M/s 4M/s
gzip -9 : time 94.2s, size 431379371 133071189, comp 69%, rate 4M/s 1M/s
gzip -1 : time 15.2s, size 1013573981 347820209, comp 66%, rate 66M/s 22M/s
gzip -1 : time 29.4s, size 2027241617 638495283, comp 69%, rate 68M/s 21M/s
bzip2 -1 : time 34.4s, size 431379371 91905771, comp 79%, rate 12M/s 2M/s
bzip2 : time 33.9s, size 431379371 86144682, comp 80%, rate 12M/s 2M/s
bzip2 -9 : time 34.2s, size 431379371 86144682, comp 80%, rate 12M/s 2M/s
pbzip2 -p1 : time 34.9s, size 431379371 86152857, comp 80%, rate 12M/s 2M/s (1 CPU)
pbzip2 -p1 -1 : time 34.6s, size 431379371 91935441, comp 79%, rate 12M/s 2M/s
pbzip2 -p1 -9 : time 34.8s, size 431379371 86152857, comp 80%, rate 12M/s 2M/s
pbzip2 -p2 : time 17.6s, size 431379371 86152857, comp 80%, rate 24M/s 4M/s (2 CPU)
pbzip2 -p3 : time 11.9s, size 431379371 86152857, comp 80%, rate 36M/s 7M/s (3 CPU)
pbzip2 -p4 : time 9.3s, size 431379371 86152857, comp 80%, rate 46M/s 9M/s (4 CPU)
pbzip2 -p4 : time 45.3s, size 2027241617 384406870, comp 81%, rate 44M/s 8M/s
pbzip2 -p8 : time 33.3s, size 2027241617 384406870, comp 81%, rate 60M/s 11M/s
lzop -1 : time 1.6s, size 431379371 213416336, comp 51%, rate 261M/s 129M/s
lzop : time 1.7s, size 431379371 213328371, comp 51%, rate 249M/s 123M/s
lzop : time 4.3s, size 1013573981 515317099, comp 49%, rate 234M/s 119M/s
lzop : time 7.3s, size 2027241617 978374154, comp 52%, rate 277M/s 133M/s
lzop -9 : time 176.6s, size 431379371 157985635, comp 63%, rate 2M/s 0M/s
lzf : time 1.4s, size 431379371 210789363, comp 51%, rate 299M/s 146M/s
lzf : time 3.6s, size 1013573981 523007102, comp 48%, rate 282M/s 145M/s
lzf : time 6.7s, size 2027241617 972953255, comp 52%, rate 303M/s 145M/s
lzma -0 : time 27s, size 431379371 112406964, comp 74%, rate 15M/s 4M/s
lzma -1 : time 35s, size 431379371 111235594, comp 74%, rate 12M/s 3M/s
lzma: > 5 min, killed
xz -0 : time 28s, size 431379371 112424452, comp 74%, rate 15M/s 4M/s
xz -1 : time 35s, size 431379371 111252916, comp 74%, rate 12M/s 3M/s
xz: > 5 min, killed
</pre>
Columns are:
compression program
time: elapsed time of the compression program (excludes the time to flush output file to disk)
size: size of input file, size of output file
comp: compression ration (0%=no compression, 100%=file compresses into nothing)
rate: input data rate (size of input file divided by elapsed time), output data rate (size of output file
divided by elapsed time)
Machine used for testing (from /proc/cpuinfo):
Intel(R) Core(TM) i7-3820 CPU @ 3.60GHz
quad core cpu with hyper-threading (8 CPU total)
32 GB quad-channel DDR3-1600.
Script used for testing:
#!/usr/bin/perl -w
my $x = join(" ", @ARGV);
my $in = "test.mid";
my $out = "test.mid.out";
my $tout = "test.time";
my $cmd = "/usr/bin/time -o $tout -f \"%e\" /usr/bin/time $x < test.mid > test.mid.out";
print $cmd,"\n";
my $t0 = time();
system $cmd;
my $t1 = time();
my $c = `cat $tout`;
print "Elapsed time: $c";
my $t = $c;
#system "/bin/ls -l $in $out";
my $sin = -s $in;
my $sout = -s $out;
my $xt = $t1-$t0;
$xt = 1 if $xt<1;
print "Total time: $xt\n";
print sprintf("%-20s: time %5.1fs, size %12d %12d, comp %3.0f%%, rate %3dM/s %3dM/s", $x, $t, $sin,
$sout, 100*($sin-$sout)/$sin, ($sin/$t)/1e6, ($sout/$t)/1e6), "\n";
exit 0;
# end
Typical output:
[deap@deap00 pet]$ ./r.perl lzf
/usr/bin/time -o test.time -f "%e" /usr/bin/time lzf < test.mid > test.mid.out
1.27user 0.15system 0:01.44elapsed 99%CPU (0avgtext+0avgdata 2800maxresident)k
0inputs+411704outputs (0major+268minor)pagefaults 0swaps
Elapsed time: 1.44
Total time: 3
lzf : time 1.4s, size 431379371 210789363, comp 51%, rate 299M/s 146M/s
K.O. |
858
|
06 Feb 2013 |
Stefan Ritt | Info | Compression benchmarks | I redid the tests from Konstantin for our MEG experiment at PSI. The event structure is different, so it
is interesting how the two different experiments compare. We have an event size of 2.4 MB and a trigger
rate of ~10 Hz, so we produce a raw data rate of 24 MB/sec. A typical run contains 2000 events, so has a
size of 5 GB. Here are the results:
cat : time 7.8s, size 4960156030 4960156030, comp 0%, rate 639M/s 639M/s
gzip -1 : time 147.2s, size 4960156030 2468073901, comp 50%, rate 33M/s 16M/s
pbzip2 -p1 : time 679.6s, size 4960156030 1738127829, comp 65%, rate 7M/s 2M/s (1 CPU)
pbzip2 -p8 : time 96.1s, size 4960156030 1738127829, comp 65%, rate 51M/s 18M/s (8 CPU)
As one can see, our compression ratio is poorer (due to the quasi random noise in our waveforms), but the
difference between gzip -1 and pbzip2 is larger (15% instead 10% for DEAP). The single CPU version of
pbzip cannot sustain our DAQ rate of 24 MB, but the parallel version can. Actually we have a somehow old
dual-core dual-CPU board 2.5 GHz Xenon box, and make 8 hyper-threading CPUs out of the total 4 cores.
Interestingly the compression rate scales with 7.3 for 8 virtual cores, so hyper-threading does its job.
So we take all our data with the pbzip2 compression. The additional 15% as compared with gzip does
not sound much, but we produce raw 250 TB/year. So gzip gives us 132 TB/year and pbzip2 gives
us 98 TB/year, and we save quite some disks.
Note that you can run bzip2 (as all the other methods) already now with the current logger, if you specify
an external compression program in the ODB using the pipe functionality:
local:MEG:S]/>cd Logger/Channels/0/Settings/
[local:MEG:S]Settings>ls
Active y
Type Disk
Filename |pbzip2>/megdata/run%06d.mid.bz2
Format MIDAS
Compression 0
ODB dump y
Log messages 0
Buffer SYSTEM
Event ID -1
Trigger mask -1
Event limit 0
Byte limit 0
Subrun Byte limit 0
Tape capacity 0
Subdir format
Current filename /megdata/run197090.mid.bz2
</pre> |
334
|
02 Feb 2007 |
Exaos Lee | Bug Report | Compiling failed with SVN3562 under Ubuntu 6.10 | The error log is as the following:
cc -c -g -O2 -Wall -Wuninitialized -Iinclude -Idrivers -I../mxml -Llinux/lib -DINCLUDE_FTPLIB -D_LARGEFILE64_SOURCE -DHAVE_MYSQL -DHAVE_ROOT -pthread -I/opt/root/current/include -DOS_LINUX -fPIC -Wno-unused-function -o linux/lib/system.o src/system.c
src/system.c:958: error: expected declaration specifiers or ‘...’ before ‘gettid’
src/system.c:958: warning: data definition has no type or storage class
src/system.c:958: warning: type defaults to ‘int’ in declaration of ‘_syscall0’
src/system.c: In function ‘ss_gettid’:
src/system.c:1005: warning: implicit declaration of function ‘gettid’
src/system.c: In function ‘ss_suspend_init_ipc’:
src/system.c:2948: warning: pointer targets in passing argument 3 of ‘getsockname’ differ in signedness
src/system.c: In function ‘ss_suspend’:
src/system.c:3414: warning: pointer targets in passing argument 6 of ‘recvfrom’ differ in signedness
src/system.c:3441: warning: pointer targets in passing argument 6 of ‘recvfrom’ differ in signedness
make: *** [linux/lib/system.o] 错误 1
The error might be here:
void ss_force_single_thread()
{
_single_thread = TRUE;
}
#if defined(OS_DARWIN)
// blank
#elif defined(OS_LINUX)
_syscall0(pid_t,gettid);
#endif
INT ss_gettid(void)
I have no idea about the usage of _syscall0(...). |
335
|
02 Feb 2007 |
Exaos Lee | Bug Report | Compiling failed with SVN3562 under Ubuntu 6.10 | I tried to solve the problem by adding a ";". It was wrong. In fact, the macro "_syscall0(..)" doesn't need the ";".
I searched and found that somebody said "the overall _syscall$magicnumber will disappear". I don't mind whether the "_syscall" disappear or not. I just want to compile the code and do my job. I deleted the additional ";" and recompiled. The error output is as the attachment [elog:335/1]. |
635
|
06 Sep 2009 |
Exaos Lee | Bug Report | Compiling error of "src/history_odbc.cxx" | Version svn-r4556, I got a compiling error as below:
/opt/DAQ/bot/midas/src/history_odbc.cxx: In member function 'virtual int
SqlODBC::GetNumRows()':
/opt/DAQ/bot/midas/src/history_odbc.cxx:589: error: cannot convert 'SQLINTEGER*'
to 'long int*' for argument '2' to 'SQLRETURN SQLRowCount(void*, long int*)'
/opt/DAQ/bot/midas/src/history_odbc.cxx: In member function 'virtual const char*
SqlODBC::GetColumn(int)':
/opt/DAQ/bot/midas/src/history_odbc.cxx:638: error: cannot convert 'SQLINTEGER*'
to 'long int*' for argument '6' to 'SQLRETURN SQLGetData(void*, SQLUSMALLINT,
SQLSMALLINT, void*, long int, long int*)'
make[2]: *** [CMakeFiles/midas-static.dir/src/history_odbc.cxx.o] Error 1
make[2]: Leaving directory `/opt/DAQ/bot/midas/build'
make[1]: *** [CMakeFiles/midas-static.dir/all] Error 2
make[1]: Leaving directory `/opt/DAQ/bot/midas/build'
The detail error log is attached. I used my CMake script without any optimization flags. I will try the default Makefile again. |
|