Fixed problems with mlogger starting and stopping runs.
Basic difficulty was with the mlogger using ASYNC transitions, which did not implement proper
transition sequencing according to transition sequence numbers. Basically all clients were called at the
same time, regardless of how long they took to process the transitions.
Switching from ASYNC to SYNC transitions introduces a deadlock between mlogger (not reading data
from SYSTEM buffer while inside cm_transition) and any program trying to write into the SYSTEM buffer
(buffer is full, does not listen for transition requests while waiting for mlogger which tries to call it's
Then we invented the mtransition helper program. In the original implemtation for t2k it was spawned
directly from the mlogger to stop the run (avoiding the deadlock). Then cm_transition(DETACHED) was
introduced, but the mlogger start/stop/restart run logic became broken. One problem was with when
auto restart delay is zero, mtransition tries to restart the run before previous run is stopped (instead,
mlogger should restart the run from it's tr_stop() handler). Another problem was with the auto restart
delay counting from the time when we start stopping the run - because stopping the run can take an
unpredictable time, depending on when various frontends have to do - it is impossible to have a
predictable delay between runs (again this is fixed by restarting the run from mlogger.c::tr_stop()).
All this has been straightened out by svn revision 4484. Basically the old run stop/restart logic was
restored in mlogger.c, using cm_transition(DETACH) to avoid the deadlocks.
To remind all, these are the present controls for transitions initiated by mlogger:
/experiment/transition debug flag - set to "2" to capture transition sequences into midas.log
/experiment/transition timeout and transition connect timeout - one can change default timeouts as
needed to accommodate non cooperative frontends.
/logger/async transitions - do not use mtransition - do ASYNC transitions, as before.
/logger/auto restart delay - delay between stopping the run (mlogger.c::tr_stop) and starting the next
svn rev 4484