> 1) history is independent from "runs", we see a change, we apply it (even if it takes 10 sec or 2 minutes).
> 
> 2) "nothing should change during a run", we must process all changes before we start a run (starting a run takes forever),
>    and we must ignore changes during a run (i.e. updated frontend starts to write new data to history). (this is why
>    the trick to "start a new run twice" used to work).

"nothing should change during a run" violates the action when a user adds a new variable during a run. So if the user does that, they don't
care if things change during a run. Then we can also modify the history DB during the run. Note that some MIDAS installations are purely
slow control (kind of a replacement of LabView, have no runs at all). In those installations runs do not make sense at all, so keeping the
history independent of runs makes sense to me.

> It is "free" to rescan ODB every 10 second or so. Then we can output a midas message "please restart the logger",
> and set an ODB flag, then when user opens the history panel editor, it will see this flag
> and tell the user "please restart the logger to see the latest changes in history". It can even list
> the specific changes, if we want ot be verbose about it.

Sounds good to me.

> I say, let's take the low road for now and see if it's good enough:
> 
> a) have the history system report any changes in midas.log - "history event added", "new history variable added" (or "renamed"),
>    this will let user see that their changes to the equipment frontend "took" and flag any accidental/unwanted changes.
> 
> b) have mlogger periodically scan ODB and set a "please restart me" flag. observe this flag in the history editor
>    and tell the user "please restart the logger to see latest changes in the history".

Actually you don't have to actively "scan" the ODB. You have hotlinks to the logger anyway from the equipment variables. All we need 
in addition is a hotline to the settings array in the ODB. The logger receives the hotline update, checks if the names changed or got
extended, then flags this as a change.

Stefan

> 2) "advanced" c++ code:
> 
> void foo(const std::string& xxx) { ... };
> int main() { foo("bar"); }
> 
> copy-created 2nd string is avoided, but string object to hold "bar" is still must be 
> made, 1 malloc(), 1 memcpy().

Are you sure about this? I always thought that foo only receives a pointer to xxx which it puts on the stack, so
no additional malloc/free is involved.

Have a look here: https://en.cppreference.com/w/cpp/language/reference.html

It says "References are not objects; they do not necessarily occupy storage".

Stefan

> > 2) "advanced" c++ code:
> > 
> > void foo(const std::string& xxx) { ... };
> > int main() { foo("bar"); }
> > 
> > copy-created 2nd string is avoided, but string object to hold "bar" is still must be 
> > made, 1 malloc(), 1 memcpy().
> 
> Are you sure about this? I always thought that foo only receives a pointer to xxx which it puts on the stack, so
> no additional malloc/free is involved.

Yes, "bar" is not an std::string, cannot be used to call foo(), and the c++ compiler has to automagically rewrite 
the function call

from: int main() { foo("bar"); }
to:   int main() { foo(std::string("bar"); }

the temporary std::string object may be on the stack, but storage for text "bar" is on the heap (unless std::string 
is optimized to store short strings internally).

one can put a printf() inside foo() to print the address of xxx (should be on the stack) and xxx.c_str() (should be 
on the heap). one could also try to print the address of "bar" (should be in the read-only-constant-strings memory 
area). (I am not sure if compiler-linker combines all instances of "bar" into one, this is also easy to check).

K.O.

Just in time, enter std::string_view.
https://stackoverflow.com/questions/40127965/how-exactly-is-stdstring-view-faster-than-const-stdstring

I was looking at https://root.cern/doc/v638/classROOT_1_1Experimental_1_1RFile.html and they use it everywhere instead of 
std::string and const char*.

(so now we have 4 string types to deal with, counting ROOT's TString).

P.S. For extra safety, this code compiles, then explodes:

std::string_view get_temporary_string() {
  std::string s = "temporary";
  return s; // DANGER! 's' is destroyed, view dangles.
}

K.O.

> 3b) mlogger used to rescan ODB each time a new run is started, this code was removed

One more kink turned out.

One of the computers ran out of disk space, mlogger dutifully recorded the "disk full" errors to midas.log and 
disabling writing to history (all history variables).

This was only noticed about one week later (it is not a busy computer).

In the past, when mlogger reopened the history at each begin of run, the "disk full" errors would have shown 
up in midas.log and somebody would have noticed. Or the problem would have gone away if disk space was cleared 
up.

Now, mlogger just silently continues not writing to history. There is no ongoing error message, there is no 
ongoing alarm, only sign of trouble is empty history plots and history files not growing.

Perhaps we should add an mlogger action to ask the history, "are you ok?" and report in midas.log or alarm if 
history is not happy.

Or have mlogger at the begin of run automatically reenable all disabled history variables. If these variables 
are still unhappy (error writing to disk or to mysql), there will be an error message in midas.log (and 
automatic self-disable).

All these solutions should be okey as long as they do not touch disk storage and so do not cause any long 
delay in run start.

K.O.

> Now, mlogger just silently continues not writing to history. There is no ongoing error message, there is no 
> ongoing alarm, only sign of trouble is empty history plots and history files not growing.

I would recommend to use an "internal alarm". This is not an "ODB alarm" where values are compared limits, but it is directly triggered by C code. To do 
so, call 

  if (disk_full)
     al_trigger-alarm("Disk full", "Disk full, no history will be written", "Alarm", "Disk full", AT_INTERNAL);

This will cause an alarm to show up prominently on the status page and beep every few minutes.

Stefan

>  #include "manalyzer.h"
>  #include "midasio.h"
> +#include "odbxx.h"

This commit broke the standalone ("no MIDAS") build of manalyzer. Either odbxx has to be an independant package 
(like mvodb) or it has to be conditioned on HAVE_MIDAS.

(this was flagged by failed bitbucket build of rootana)

K.O.

> >  #include "manalyzer.h"
> >  #include "midasio.h"
> > +#include "odbxx.h"
> 
> This commit broke the standalone ("no MIDAS") build of manalyzer. Either odbxx has to be an independant package 
> (like mvodb) or it has to be conditioned on HAVE_MIDAS.
> 
> (this was flagged by failed bitbucket build of rootana)

Corrected. You can only use odbxx is manalyzer is built with HAVE_MIDAS. (mvodb is an independant package and is 
always available, no need to pull and build the full MIDAS).

Also notice how I now initialize odbxx from fBorOdbDump and fEorOdbDump. Also tested against multithreaded access, it 
works (as Stefan promised).

K.O.

I updated the root helper constructor to give the user more control over ROOT output file names.

You can now change it to anything you want in the module run constructor, see manalyzer_example_esoteric.cxx

Is this good enough?

struct ExampleE1: public TARunObject
{
   ExampleE1(TARunInfo* runinfo)
      : TARunObject(runinfo)
   {
#ifdef HAVE_ROOT
      if (runinfo->fRoot)
         runinfo->fRoot->fOutputFileName = "my_custom_file_name.root";
#endif
   }
}

K.O.

We had our periodic discussion on MIDAS web page user permissions. (I cannot 
find the link to the previous discussions, ouch!)

Currently any logged in user can do anything - start stop runs, start/stop 
programs, edit odb, etc.

Regularly, we have experiments that ask about "read-only" access to MIDAS and 
about more granular user permissions.

In the past, I suggested a permissions scheme that is easy to implement
with the current code base. Permission level for each user can
be stored in ODB and allow:

level 0 - root user, as now
level 1 - experiment user, any restrictions are implemented in javascript, i.e. 
all custom pages work as they do now, but (i.e.) the odb editor is read-only
level 2 - experiment operator, restrictions are implemented in the mjsonrpc 
code, i.e. can start/stop runs, start/stop programs, but cannot make any 
changes, i.e. cannot write to ODB
level 3 - read-only user - only mjsonrpc calls that do not change anything are 
permitted.

(to implement level 2, obviously, the "start run" mjsonrpc call has to be 
changed to accept the run comments, current code writes them to odb directly and 
that would fail).

First step towards implementing this was made today. Ben and Derek figured out 
the apache incantation to pass the logged user name to MIDAS and I added 
decoding of this user name in mhttpd. I do not do anything with it, yet.

In apache config, one change is needed:

> For Apache, add this line in your VirtualHost section (tested as working):
> RequestHeader set X-Remote-User %{REMOTE_USER}s

https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Install_apache_httpd_proxy_for_midas_and_elog

K.O.

There seems to be a version dependency with the numpy.bool , e.g. used here
https://bitbucket.org/tmidas/midas/src/c6ef4aff5e7e652df79160141e570bed5f4d6a3b/python/midas/sequencer.py?at=develop#sequencer.py-1714 .

This type alias does not exist for versions in-between 1.24.0 and 2.0.0 .
https://numpy.org/doc/stable/release/1.24.0-notes.html#np-str0-and-similar-are-now-deprecated

Would it be an option to specify midas-compatible numpy versions in the setup.py with extras_require ?

Hi,

On the DAQ system for T2K's ND280 near detector, we use deferred
transitions to make sure all triggered events were logged before issuing run
stops to frontends.

I've recently managed to update the frontends to use a
relatively modern version of MIDAS. I then noticed that run transitions are now
by default multithreaded, when issued from e.g. mhttpd, but deferred transitions
called by cm_check_deferred_transition are still performed synchronously.

It
would be nice to make run stops use multithreaded transitions as well. A naive
patch of adding the TR_MTHREAD flag does not work, since the client handling the
deferred transition attempts to communicate with itself instead of calling
cm_transition_call_direct.

After looking into the code a bit further, I noticed
that there is an intentional check against multithreaded transitions in the
logic for determining whether the client is the one calling the transition:

https://bitbucket.org/tmidas/midas/src/fd71f63c023b7e2d4a5c91e3121651b14bd9d27b/
src/midas.cxx#lines-5009

Was there a particular concern that lead to this
particular check?


Regards,
Yiwen.

Multithreaded transitions were introduced by KO in 2019. Please ask him to make deferred transitions work 
again or simply use non-multithreaded transitions. 

Stefan

> Multithreaded transitions were introduced by KO in 2019. Please ask him to make deferred transitions work 
> again or simply use non-multithreaded transitions. 

Deferred transition is the bane of MIDAS, I personally can never understand how they work
and what they do, and I studied them and understood them many times now.

I recommend against using them. Maybe you can explain what you do and I can suggest a way
to avoid using the deferred transition.

Some other people use deferred transitions, and it works for them,
in conjunction with normal transitions, which have been multithreaded
for years.

So unlikely this is a new bug.

P.S. I do not remember any use of deferred transition in the T2K/ND280 FGD, TPC and GSC frontends,
maybe it is in some other subsystem or was introduced after my time.

K.O.

> I recommend against using them. Maybe you can explain what you do and I can suggest a way
> to avoid using the
deferred transition.

I've only recently picked up the code and I'm not sure how it was envisioned when
initially designed, so here's my guess at how it's being used.

Deferred transition is registered by the FPN00
(main clock) process to make sure the logger has finished logging all events before continuing to stop readout
of other frontends. There is also a timeout, in case sometimes an event goes missing, to proceed with the run
stop without getting stuck waiting for the logger.

If there is a better way to implement this then I'm happy to
give it a go.

> P.S. I do not remember any use of deferred transition in the T2K/ND280 FGD, TPC and GSC
frontends,
> maybe it is in some other subsystem or was introduced after my time.

This is all from the global
DAQ code, so indeed none of the other subsystems' frontends use the feature directly. But if the clock module
frontend is started then run stops should be deferred.

Interestingly, I just checked and this feature is
disabled in the FGD ODB, but it is enabled in the TPC ODB.

Regards,
Yiwen.

We recently had an instance where mlogger lost its connection to the database resulting in 317 write errors on mysql_insert and (Good) did report connection errors to the Midas log but (Bad) didn't attempt to restore the connection. With some help from Gemini, I've drafted a potential solution to reheal. Our copy uses mysql_query_debug line 1903 in our copy that inserts rows, creates tables, etc:

   /* execut sql query */
   status = mysql_query(db, query);

   if (status)
      cm_msg(MERROR, "mysql_query_debug", "SQL error: %s", mysql_error(db));

   return status;

to add a reconnection attempt like the following (untested) block:

    /* execute sql query */
    status = mysql_query(db, query);
    
    if (status != 0) {
        unsigned int err_num = mysql_errno(db);
        cm_msg(MERROR, "mysql_query_debug", "SQL query execution failed (Error %u: %s).", err_num, mysql_error(db));
        cm_msg(MTRCE, "mysql_query_debug", "Network drop suspected. Backing off 5 seconds to heal handle socket...");
        
        #ifdef OS_WIN64
        Sleep(5000);
        #else
        sleep(5);
        #endif

        cm_msg(MTRCE, "mysql_query_debug", "Initiating handle re-authentication to server...");

        // Setting MYSQL_OPT_PROTOCOL to MYSQL_PROTOCOL_TCP ensures that even when passing NULL for host configuration
        // fallbacks, the client library strictly forces a network TCP connection instead of a local Unix socket.
        unsigned int protocol = MYSQL_PROTOCOL_TCP;
        mysql_options(db, MYSQL_OPT_PROTOCOL, &protocol);

        if (!mysql_real_connect(db, NULL, NULL, NULL, NULL, 0, NULL, 0)) {
            cm_msg(MERROR, "mysql_query_debug", "Automatic handle reconciliation failed: %s. Logging halted.", mysql_error(db));
        } else {
            cm_msg(MINFO, "mysql_query_debug", "Database connection handle successfully restored.");
            
            // Retry the blocked execution string
            status = mysql_query(db, query);
            if (status != 0) {
                cm_msg(MERROR, "mysql_query_debug", "Query retry failed after reconnection (Error %u: %s).", mysql_errno(db), mysql_error(db));
            } else {
                cm_msg(MINFO, "mysql_query_debug", "SQL query successfully recovered and executed on retry.");
            }
        }
    }

   return status;

Dear all,

I was wondering if one can use multiple pysequencers at the same time and if not
if this feature is planned in the future? Our experiment (mu3e) has one frontend
per detector and each sequencer would only access a subset of ODB entries.

Best,
Marius

you can also copy the reconnect code from history_schema.cxx.

if I remember right, the test case for correct reconnect is to start the mlogger, 
have it write something to mysql, then restart mysqld (mlogger connection is now 
broken), then have mlogger write some more to mysql. it should get an error 
"connection broken", reconnect and retry the write.

(to avoid it getting stuck, there should be a limit of 10 attempts to reconnect 
plus retry, in case each retry crashes mysqld or throws an unrelated error).

the code in question I think is mlogger mysql supoprt for begin and end of run, 
so you start a run, restart mysqld, stop the run, start a new run and should see 
a successful reconnect.

K.O.

> I was wondering if one can use multiple pysequencers at the same time and if not
> if this feature is planned in the future? Our experiment (mu3e) has one frontend
> per detector and each sequencer would only access a subset of ODB entries.

I believe sequencer design supports this, same as mlogger supports multiple output 
files and mhttpd supports multiple listener ports.

But how to do this with pysequencer, I am not sure. Simplest is to wait a few more 
weeks for Ben to return from vacation.

K.O.

> Deferred transition is registered by the FPN00

FPN00, yes, this rings a bell.

> (main clock) process to make sure the logger has finished logging all events before continuing to stop readout
> of other frontends. There is also a timeout, in case sometimes an event goes missing, to proceed with the run
> stop without getting stuck waiting for the logger.

yes, that's right, if trigger control and run control are in the same frontend, you run into this trouble:

user pushes run stop button
you get the run stop callback
you stop the trigger
but have to wait for all the data to flush down the pipes all the way to mlogger
but cannot wait, must return from the callback otherwise run transition is stuck

if trigger control and run control are in different frontends, things are simpler:

user pushes run stop botton
trigger control frontend disables the trigger, returns without waiting for anything
data frontend (i.e. FGD, TPC), flush all the hardware FIFOs, etc to MIDAS SYSTEM buffer (trigger is already 
disabled, there is no new data)
mlogger flushes SYSTEM buffer to disk
run control frontend reports successful run stop to the run database (and whatever else).

> Interestingly, I just checked and this feature is
> disabled in the FGD ODB, but it is enabled in the TPC ODB.

I do not have access to the current code, but I can check what I have,
and I am pretty sure it did not have any deferred transitions. Maybe
it was added after my time.

K.O.