> > (main clock) process to make sure the logger has finished logging all events before continuing to stop readout
>
> of other frontends. There is also a timeout, in case sometimes an event goes missing, to proceed with the run
> >
stop without getting stuck waiting for the logger.
> 
> yes, that's right, if trigger control and run control are in
the same frontend, you run into this trouble:
> 
> user pushes run stop button
> you get the run stop callback
> you
stop the trigger
> but have to wait for all the data to flush down the pipes all the way to mlogger
> but cannot
wait, must return from the callback otherwise run transition is stuck
> 
> if trigger control and run control are in
different frontends, things are simpler:
> 
> user pushes run stop botton
> trigger control frontend disables the
trigger, returns without waiting for anything
> data frontend (i.e. FGD, TPC), flush all the hardware FIFOs, etc to
MIDAS SYSTEM buffer (trigger is already 
> disabled, there is no new data)
> mlogger flushes SYSTEM buffer to disk
>
run control frontend reports successful run stop to the run database (and whatever else).
> 

What do you mean by
the run control frontend in this case? Usually the run stops are issued from the mhttpd interface, which I thought
meant that mhttpd is in charge of run transitions.

From what I understand, deferred transition means that whichever
frontend registered the deferred transition will take over the run transition. So, if there's no deferred transition
in the first place then this should not be a problem?

Looks like the code formatting got messed up by the elog... here it is in the attachment instead.

Thanks for sharing the videos, Konstantin.

> Unpredictable code can be dangerous:
> 
> void boil_kettle()
>
{
>    i2c_set_bit(10, 1); // bit 10 is stove heater control, turn it on
>    foo(); // wait for kettle to
start boiling
>    i2c_set_bit(10, 0); // turn heater off
> }
> 
> If foo() starts throwing exceptions, I
will likely have a fire in my kitchen!
> 

Indeed, that would certainly be problematic. However, there is a
way to get around this while not having to wrap everything in try ... catch ... blocks with more C++
machinery using the catch-all handler and RAII.

To quickly expand upon your example:


=== main.cpp ===


#include <iostream>
#include <unistd.h>

void i2c_set_bit(int bit, int val) {
	std::cout << "Bit " << bit <<
" set to " << val << "\n";
}

class i2c_temp_bit {
	int bit; /* which bit */
	int newval; /* value to set bit
to */
	int oldval; /* value to reset bit to */
public:
	i2c_temp_bit(int b, int nv, int ov)
		: bit{b},
newval{nv}, oldval{ov}
	{
		std::cout << "Bit " << bit << " set to " << newval << "\n";
	}
	~i2c_temp_bit() 

{
		std::cout << "Bit " << bit << " set to " << oldval << "\n";
	}
	i2c_temp_bit(const i2c_temp_bit&) =
delete;
	i2c_temp_bit(i2c_temp_bit&&) = delete;
	i2c_temp_bit& operator=(const i2c_temp_bit&) = delete;
	
i2c_temp_bit& operator=(i2c_temp_bit&&) = delete;
};

int check_kettle_temp() {
	static int callnum = 0;
	if
(++callnum > 5) {
		return 100;
	}
#ifdef BADNUM
	else if (callnum == 4) { /* Don't like this number */
		
throw callnum;
	}
#endif
	else {
		return 25;
	}
}

void foo() {
	std::cout << "Waiting for kettle to boil"
<< std::endl;
	do { /* Check every 100 ms */
		usleep(100000);
	} while (check_kettle_temp() < 100);
	
std::cout << "Water is boiled!" << std::endl;
}

void boil_kettle()
{
#ifdef RAII
	i2c_temp_bit temp(10, 1,
0);
	foo();
#else
	i2c_set_bit(10, 1); // bit 10 is stove heater control, turn it on
	foo(); // wait for
kettle to start boiling
	i2c_set_bit(10, 0); // turn heater off
#endif
}

int main()
{
	try {
		
boil_kettle();
	} catch (...) {
		std::cerr << "Abnormal termination\n";
	}
}

=== main.cpp ===


When we run
this:

$ g++ -o main main.cpp && ./main
Bit 10 set to 1
Waiting for kettle to boil
Water is boiled!
Bit 10
set to 0

This is as expected, of course.
But indeed as you say, when foo() can throw an exception, we get a
problem:

$ g++ -DBADNUM -o main main.cpp && ./main
Bit 10 set to 1
Waiting for kettle to boil
Abnormal
termination

The exception is caught by the catch-all handler, but because boil_kettle() never finishes
execution, bit 10 never gets reset to 0 and that's a big problem.

With RAII however, we get this instead:

$
g++ -DBADNUM -DRAII -o main main.cpp && ./main
Bit 10 set to 1
Waiting for kettle to boil
Bit 10 set to 0

Abnormal termination

Because the i2c_temp_bit object is destroyed upon unwinding the boil_kettle() function,
it calls the destructor which resets the bit.

For closure, it does behave identically to the non-RAII
implementation when there are no exceptions:

$ g++ -DRAII -o main main.cpp && ./main
Bit 10 set to 1
Waiting
for kettle to boil
Water is boiled!
Bit 10 set to 0


Personally, I don't like this as it adds way too much
complexity to the code in order to avoid C++ foot-guns (see how the copy and move constructors etc. need to
be deleted, or else requires a bit of thinking to implement properly). 
But, it is an alternative that exists
to safely handle these situations.

> If you did not want to use exceptions before watching this talk, you will after!

Not me. One issue, exceptions are unpredictable, for example, does this code always print "hello, world"?

#include <stdio.h>
#include "foo.h"
int main(...)
{
  foo();
  printf("hello, world\n");
  return 0;
}

I can wrap foo() (and each and every function call) in try/catch.

I can check if today's version of foo.h declares foo() as "nothrow" (but what if tomorrow's foo() start 
throwing?)

Unpredictable code can be dangerous:

void boil_kettle()
{
   i2c_set_bit(10, 1); // bit 10 is stove heater control, turn it on
   foo(); // wait for kettle to start boiling
   i2c_set_bit(10, 0); // turn heater off
}

If foo() starts throwing exceptions, I will likely have a fire in my kitchen!

K.O.

Bjarne Stroustrup gave a nice talk at CERN, good enough to bring Rene Brun out of retirement to ask a few 
questions at the 1:16:50 mark.

https://indico.cern.ch/event/1696048/
https://videos.cern.ch/record/3026440

Good listen, but closer to home:

Stroustrup slides have a wee reference to a CppCon-2024 video "C++ exceptions for smaller firmware", I gave 
it a listen. The guy talks fast and waves hands vigorously, but his material is solid.

In the first 30 minutes he busts the myth that embedded programming can not/should not use exceptions (with 
numbers and example code).

In the second 30 minutes he demonstrates the insane code complexity required to implement exceptions, and 
that also makes per-call error checking very cheap (compares all three "return bool", "return 
std::expected", "throw exception", with disassembly of example code)

The the third 30 minutes he talks about project level implications of using exceptions vs other methods of 
error handling (one example is audacity).

If you did not want to use exceptions before watching this talk, you will after!

https://www.youtube.com/watch?v=bY2FlayomlE

K.O.

> Deferred transition is registered by the FPN00

FPN00, yes, this rings a bell.

> (main clock) process to make sure the logger has finished logging all events before continuing to stop readout
> of other frontends. There is also a timeout, in case sometimes an event goes missing, to proceed with the run
> stop without getting stuck waiting for the logger.

yes, that's right, if trigger control and run control are in the same frontend, you run into this trouble:

user pushes run stop button
you get the run stop callback
you stop the trigger
but have to wait for all the data to flush down the pipes all the way to mlogger
but cannot wait, must return from the callback otherwise run transition is stuck

if trigger control and run control are in different frontends, things are simpler:

user pushes run stop botton
trigger control frontend disables the trigger, returns without waiting for anything
data frontend (i.e. FGD, TPC), flush all the hardware FIFOs, etc to MIDAS SYSTEM buffer (trigger is already 
disabled, there is no new data)
mlogger flushes SYSTEM buffer to disk
run control frontend reports successful run stop to the run database (and whatever else).

> Interestingly, I just checked and this feature is
> disabled in the FGD ODB, but it is enabled in the TPC ODB.

I do not have access to the current code, but I can check what I have,
and I am pretty sure it did not have any deferred transitions. Maybe
it was added after my time.

K.O.

> I was wondering if one can use multiple pysequencers at the same time and if not
> if this feature is planned in the future? Our experiment (mu3e) has one frontend
> per detector and each sequencer would only access a subset of ODB entries.

I believe sequencer design supports this, same as mlogger supports multiple output 
files and mhttpd supports multiple listener ports.

But how to do this with pysequencer, I am not sure. Simplest is to wait a few more 
weeks for Ben to return from vacation.

K.O.

you can also copy the reconnect code from history_schema.cxx.

if I remember right, the test case for correct reconnect is to start the mlogger, 
have it write something to mysql, then restart mysqld (mlogger connection is now 
broken), then have mlogger write some more to mysql. it should get an error 
"connection broken", reconnect and retry the write.

(to avoid it getting stuck, there should be a limit of 10 attempts to reconnect 
plus retry, in case each retry crashes mysqld or throws an unrelated error).

the code in question I think is mlogger mysql supoprt for begin and end of run, 
so you start a run, restart mysqld, stop the run, start a new run and should see 
a successful reconnect.

K.O.

I've been working on updating MIDAS' testing suite. It is yet incomplete, but the basic structure is there. Once the code coverage reaches a more acceptable level and it is decided that the new tests are good replacements to the existing tests, it will be merged into develop. 

Tests are located in $MIDASSYS/tests directory in the automated_testing branch. 

ENABLE

For now changes are relegated to the automated_testing branch:

git pull
git checkout automated_testing
make cmake

STRUCTURE

Because MIDAS is implemented in three main languages (cxx, python, javascript), tests must be written in three languages. So each language has its own set of tests and frameworks in which they run. This introduces some new dependencies needed to run the tests:

These dependencies are ONLY needed to run the test suites and do not affect the main build of MIDAS. Compiling the tests is controlled via the flag MIDAS_BUILD_TESTS, which is off by default, unless make test is called. Tests are located in the $MIDASSYS/tests directory, which should share roughly the same directory structure as the MDIAS source code.

 USAGE

1. Make commands:

• "make test": run all tests
• "make coverage": run all tests and generate coverage reports. Reports are located as indicated in the below table
• "make coverage-recapture": regenerate coverage report without re-running tests.

Coverage reports locations:

2. Run executables directly:

$MIDASSYS/tests/README.md has the full instructions on how to do this, as well as detailed descriptions on the tests and frameworks used. In general each cxx subdirectory has a binary executable to run the tests for that subdirectory group, whereas python and javascript have their own executables related to the frameworks running the tests.

Dear all,

I was wondering if one can use multiple pysequencers at the same time and if not
if this feature is planned in the future? Our experiment (mu3e) has one frontend
per detector and each sequencer would only access a subset of ODB entries.

Best,
Marius

We recently had an instance where mlogger lost its connection to the database resulting in 317 write errors on mysql_insert and (Good) did report connection errors to the Midas log but (Bad) didn't attempt to restore the connection. With some help from Gemini, I've drafted a potential solution to reheal. Our copy uses mysql_query_debug line 1903 in our copy that inserts rows, creates tables, etc:

   /* execut sql query */
   status = mysql_query(db, query);

   if (status)
      cm_msg(MERROR, "mysql_query_debug", "SQL error: %s", mysql_error(db));

   return status;

to add a reconnection attempt like the following (untested) block:

    /* execute sql query */
    status = mysql_query(db, query);
    
    if (status != 0) {
        unsigned int err_num = mysql_errno(db);
        cm_msg(MERROR, "mysql_query_debug", "SQL query execution failed (Error %u: %s).", err_num, mysql_error(db));
        cm_msg(MTRCE, "mysql_query_debug", "Network drop suspected. Backing off 5 seconds to heal handle socket...");
        
        #ifdef OS_WIN64
        Sleep(5000);
        #else
        sleep(5);
        #endif

        cm_msg(MTRCE, "mysql_query_debug", "Initiating handle re-authentication to server...");

        // Setting MYSQL_OPT_PROTOCOL to MYSQL_PROTOCOL_TCP ensures that even when passing NULL for host configuration
        // fallbacks, the client library strictly forces a network TCP connection instead of a local Unix socket.
        unsigned int protocol = MYSQL_PROTOCOL_TCP;
        mysql_options(db, MYSQL_OPT_PROTOCOL, &protocol);

        if (!mysql_real_connect(db, NULL, NULL, NULL, NULL, 0, NULL, 0)) {
            cm_msg(MERROR, "mysql_query_debug", "Automatic handle reconciliation failed: %s. Logging halted.", mysql_error(db));
        } else {
            cm_msg(MINFO, "mysql_query_debug", "Database connection handle successfully restored.");
            
            // Retry the blocked execution string
            status = mysql_query(db, query);
            if (status != 0) {
                cm_msg(MERROR, "mysql_query_debug", "Query retry failed after reconnection (Error %u: %s).", mysql_errno(db), mysql_error(db));
            } else {
                cm_msg(MINFO, "mysql_query_debug", "SQL query successfully recovered and executed on retry.");
            }
        }
    }

   return status;

Hello, 

The UCN group at TRIUMF has used the history system extensively with a MySQL backend. They see long wait times (up to 20 mins) to add new logged variables to MIDAS equipment. I did some testing and it seems to be a MySQL issue that is fixed in recent versions (UCN uses an older version of MySQL). 

Likely no update is needed to MIDAS source. 


Derek 

> I recommend against using them. Maybe you can explain what you do and I can suggest a way
> to avoid using the
deferred transition.

I've only recently picked up the code and I'm not sure how it was envisioned when
initially designed, so here's my guess at how it's being used.

Deferred transition is registered by the FPN00
(main clock) process to make sure the logger has finished logging all events before continuing to stop readout
of other frontends. There is also a timeout, in case sometimes an event goes missing, to proceed with the run
stop without getting stuck waiting for the logger.

If there is a better way to implement this then I'm happy to
give it a go.

> P.S. I do not remember any use of deferred transition in the T2K/ND280 FGD, TPC and GSC
frontends,
> maybe it is in some other subsystem or was introduced after my time.

This is all from the global
DAQ code, so indeed none of the other subsystems' frontends use the feature directly. But if the clock module
frontend is started then run stops should be deferred.

Interestingly, I just checked and this feature is
disabled in the FGD ODB, but it is enabled in the TPC ODB.

Regards,
Yiwen.

> Multithreaded transitions were introduced by KO in 2019. Please ask him to make deferred transitions work 
> again or simply use non-multithreaded transitions. 

Deferred transition is the bane of MIDAS, I personally can never understand how they work
and what they do, and I studied them and understood them many times now.

I recommend against using them. Maybe you can explain what you do and I can suggest a way
to avoid using the deferred transition.

Some other people use deferred transitions, and it works for them,
in conjunction with normal transitions, which have been multithreaded
for years.

So unlikely this is a new bug.

P.S. I do not remember any use of deferred transition in the T2K/ND280 FGD, TPC and GSC frontends,
maybe it is in some other subsystem or was introduced after my time.

K.O.

> I fixed that by not requiring the handle explicitly ...
>
> [it was...] an uncaught exception. If you do not want the abort, catch the exception. 

Hi, Stefan! Thank you for fixing this!

The issue was not the exception, but the failure to load the XML ODB dump from an (immutable) data file.

This should now be fixed (TBC), so all good now.

K.O.

I updated the midas forum elog to the latest version from git: 083448f7

Also investigated elogd failure to start on reboot,
it turned out to be a crasher bug, see
https://elog.psi.ch/elogs/Forum/69919

K.O.

Multithreaded transitions were introduced by KO in 2019. Please ask him to make deferred transitions work 
again or simply use non-multithreaded transitions. 

Stefan

I fixed that by not requiring the handle explicitly:

           if (mxml_get_attribute(node, "handle") != nullptr)
              o->set_hkey(std::stoi(std::string(mxml_get_attribute(node, "handle"))));

The reason for the handle is the following: If you attach an midas::odb object to a very large subtree of the ODB, this can take very long since each
ODB element requires a few RPC roundtrips. In Mu3e, this took up to one minute. 

To overcome the problem, we can initialize an midas::odb object remotely via an XML tree. The server creates a huge XML object, sends it over
a single RPC command, and the client re-creates the midas::odb tree from the XML object. Since each online midas::odb need the ODB handle for
watch functions etc. the handle got added to the XML file.

For a XML file, this makes no sense of course, so now it's optional with the code change above.

P.S.: The odbxx code does not core dump, it just produces an error which looks like core dump. This is an uncaught exception. Normal exceptions
just abort the program without much information. The odbxx exceptions add a stack dump if available on that OS. That makes it easier to debug.
If you do not want the abort, catch the exception. 

Stefan

Hi,

On the DAQ system for T2K's ND280 near detector, we use deferred
transitions to make sure all triggered events were logged before issuing run
stops to frontends.

I've recently managed to update the frontends to use a
relatively modern version of MIDAS. I then noticed that run transitions are now
by default multithreaded, when issued from e.g. mhttpd, but deferred transitions
called by cm_check_deferred_transition are still performed synchronously.

It
would be nice to make run stops use multithreaded transitions as well. A naive
patch of adding the TR_MTHREAD flag does not work, since the client handling the
deferred transition attempts to communicate with itself instead of calling
cm_transition_call_direct.

After looking into the code a bit further, I noticed
that there is an intentional check against multithreaded transitions in the
logic for determining whether the client is the one calling the transition:

https://bitbucket.org/tmidas/midas/src/fd71f63c023b7e2d4a5c91e3121651b14bd9d27b/
src/midas.cxx#lines-5009

Was there a particular concern that lead to this
particular check?


Regards,
Yiwen.

> > > > 
> > > > How can the MSL code figure out if the wait succeeded or timed out?
> > > > 
> > > > Stefan
> > > 
> > > You get a message, something like:
> > > 17:52:12.293 2026/05/29 [Sequencer,INFO] WAIT ODBValue timeout after 10.0 seconds: /Equipment/Test/Variables/V < 1 not satisfied
> > > 
> > > Do we need something else?
> > > 
> > > Zaher
> > 
> > I mean how can the following code determine the timeout?
> 
> My intention with this was dealing with something like setting a cryostat temperature or any non-critical parameter. If it is not reached within a given timeout we give up and move on with the plan rather than sitting and wasting a whole night of beam. If your ODBvalue is "mission critical" then the wait command should not be used with a timeout. If you do use the timeout option then you will have to check in the following lines what is the state of your ODBvalue (very easy). To me this is the simplest and most useful way for our use case.

I was more thinking like a return value 0/1 if the wait function. If you change the condition, you only have to change it in one location. More like normal C functions work.

Stefan