We have seen on several daq systems this problem: we start a run and observe that the number of
events written by mlogger to the output file is double the number of events actually collected. Upon
inspection of the output file, we see that every event is written twice. Restarting the run usually fixes
this problem.
We now traced this to an error in mlogger.c. If we start a run and the run transition fails in some
frontend, mlogger does not disconnect from the SYSTEM buffer (it does not know the transition failed
and the run did not really start). The SYSTEM buffer connection and the associated event request
remain active. Then we start the next run and mlogger connects to the SYSTEM buffer again, creates a
second (third, etc) event request. Eventually mlogger reaches the maximum permitted number of event
requests and no more runs can be started unless mlogger is restarted.
If at some point a run actually starts successfully, there are multiple event requests present from
mlogger and theoretically, each event should be written to the output file many times. This was a
puzzle until we got a good laugh from looking at mlogger.c::receive_event() callback - in retrospect it
is obvious why events are only written in duplicate.
Then, after the run is ended, mlogger disconnects from the SYSTEM buffer, all multiple event requests
are automatically deleted and the problem is not present during the next run.
I am not yet sure how to best fix this, but I see that other midas programs (i.e. mevb) suffer form the
same problem - multiple connections to the event buffer - in presence of failed run starts. I think we
have seen "event duplication" from mevb, as well.
K.O. |