I updated the root helper constructor to give the user more control over ROOT output file names.

You can now change it to anything you want in the module run constructor, see manalyzer_example_esoteric.cxx

Is this good enough?

struct ExampleE1: public TARunObject
{
   ExampleE1(TARunInfo* runinfo)
      : TARunObject(runinfo)
   {
#ifdef HAVE_ROOT
      if (runinfo->fRoot)
         runinfo->fRoot->fOutputFileName = "my_custom_file_name.root";
#endif
   }
}

K.O.

> /code/midas/manalyzer/manalyzer.cxx:799:27: error: ‘pow’ was not declared in this scope
>   799 |       bins[i] = TimeRange*pow(1.1,i)/pow(1.1,Nbins);

math.h added, pushed. nice catch.

implicit include of math.h came through TFile.h (ROOT v6.34.02), perhaps you have a newer ROOT
and they jiggled the include files somehow.

TFile.h -> TDirectoryFile.h -> TDirectory.h -> TNamed.h -> TString.h -> TMathBase.h -> cmath -> math.h

K.O.

> Thanks for reporting this. It was caused by a
> 
>   MJsonNode* node = MJsonNode::Parse(str.c_str());
> 
> not followed by a 
> 
>   delete node;
> 
> I added that now in odb::odb_from_json_string(). Can you try again?
> 
> Stefan

Close, but no cigar, node you delete is not the node you got from Parse(), see "node = subnode;".

If I delete the node returned by Parse(), I confirm the memory leak is gone.

BTW, it looks like we are parsing the whole JSON ODB dump (200k+) on every odbxx access. Can you parse it just once?

K.O.

> > BTW, it looks like we are parsing the whole JSON ODB dump (200k+) on every odbxx access. Can you parse it just once?
> You are right. I changed the code so that the dump is only parsed once. Please give it a try.

Confirmed fixed, thanks! There are 2 small changes I made in odbxx.h, please pull.

K.O.

> TFile.h -> TDirectoryFile.h -> TDirectory.h -> TNamed.h -> TString.h -> TMathBase.h -> cmath -> math.h

reading ROOT release notes, 6.38 removed TMathBase.h from TString.h, with a warning "This change may cause errors during compilation of 
ROOT-based code". Upright citizens, nice guys!

> Thanks. Are you happy for me to update the submodule commit in Midas to use this fix? I should have sufficient permission if you agree.

I am doing some last minute tests, will pull it into midas and rootana later today.

K.O.

> > Confirmed fixed, thanks! There are 2 small changes I made in odbxx.h, please pull.
> There was one missing enable_jsroot in manalyzer, please pull yourself.

pulled, pushed to rootana, thanks for fixing it!

K.O.

We finished running the DarkLight experiment and I am reporting accumulated bugs that we have run into.

1) history plots on 12 hrs, 24 hrs tend to hang with "page not responsive". most plots have 16-20 variables, 
which are recorded at 1/sec interval. (yes, we must see all the variables at the same time and yes, we want to 
record them with fine granularity).

2) starting runs gets into a funny mode if a GEM frontend aborts (hardware problems), transition page reports 
"wwrrr, timeout 0", and stays stuck forever, "cancel transition" does nothing. observe it goes from "w" 
(waiting) to "r" (RPC running) without a "c" (connecting...) and timeout should never be zero (120 sec in 
ODB).

3) ODB editor clicking on hex number versus decimal number no longer allows editing in hex, Stefan implemented 
this useful feature and it worked for a while, but now seems broken.

4) ODB editor "right click" to "delete" or "rename" key does not work, the right-click menu disappears 
immediately before I can use it (dl-server-2), click on item (it is now blue), right-click menu disappears 
before I can use it (daq17). it looks like a timing or race condition.

5) ODB editor "create link" link target name is limited to 32 bytes, links cannot be created (dl-server-2), ok 
on daq17 with current MIDAS.

6) MIDAS on dl-server-2 is "installed" in such a way that there is no connection to the git repository, no way 
to tell what git checkout it corresponds to. Help page just says "branch master", git-revision.h is empty. We 
should discourage such use of MIDAS and promote our "normal way" where for all MIDAS binary programs we know 
what source code and what git commit was used to build them.

6a) MIDAS on dl-server-2 had a pretty much non-functional history display, I reported it here, Stefan provided 
a fix, I manually retrofitted it into dl-server-2 MIDAS and we were able to run the experiment. (good)

6b) bug (5) suggests that there is more bugs being introduced and fixed without any notice to other midas 
users (via this forum or via the bitbucket bug tracker).

K.O.

> I am migrating the full CRIPT muon tomography detector from MIDAS (SVN Rev. 
> 5238, circa 2012) to a more modern release.
> The current system runs on Scientific Linux 6 and very old hardware.

Right, good vintage midas and linux. But in the current security environment,
we must run currently supported OS (and MIDAS), and we must never fall off
the yearly/bi-yearly OS upgrade threadmill.

How old is your computer hardware and do you plan to update it, as well? If you OS
is installed on an HDD, definitely move to an SSD would be good. If you are hard
on money, a RaspberryPi5 with 16GB RAM may be good enough for what you have.

Anyhow new OS choice would be Ubuntu 24 or Debian 13. I do not recommend Red Hat based OS (vanilla 
RHEL, Fedora, Alma, Rocky), they have become niche OSes with minimal vendor and community support.

> Due to substantial changes in the MIDAS codebase over the years ...

The big change in MIDAS land is move to c++, then c++11, then to c++17, and move from vanilla make to 
cmake.

MIDAS API has been reasonably stable since then, but very old MIDAS frontends would fail to build with 
latest compilers because of changes in c++ language and changes in the c and c++ libraries.

> I have encountered multiple compatibility issues during the migration. I have also attempted to 
build and run the legacy MIDAS version and the front-end code using GCC 4.8 on a modern  Linux system 
(Ubuntu 24.04), but without success.

this is non-viable, latest C/C++ compilers reject perfectly good SL6-era C/C++ code, old MIDAS would 
not compile, old frontends would not compile.

> Could you please advise on the recommended approach ...

What you are doing, we have done several times with TRIUMF experiments,
updating SL6 and CentOS-7 MIDAS instances to current MIDAS, C++ and OS:

1) new computer with Ubuntu 24 (or Debian or Raspbian). (U-26 will come out roughly in August, fo rth 
epurposes of this discussion).
2) new MIDAS. we generally recommend the head of the develop branch, but older tagged version are 
okey, too.
3) apache https proxy, etc, for secure browser connections, see
https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Install_apache_httpd_proxy_for_midas_and_elog
4) reload your old ODB into the new MIDAS
5) your old history, etc should work
6a) build your old frontends, this will be a chore, but if you look at the compile errors, you will 
see that most changes are very mechanical (i.e. const char*, etc). biggest hassle is ot make your old 
C/C++ code to build with current C++17 compiler, smaller hassle is to update for minor changes in the 
mfe.h API.
6b) bite the bullet and rewrite your frontends using the C++ tmfe API, start with 
tmfe_example_everything.cxx, remove unnecessary, add required, pretty straightforward, I can guide you 
through this (contact me directly by email).
7) minor tweeks to mlogger, mhttpd and history settings
8) rewrite all customs pages to the current mjsonrpc API

Best of luck, if you have more questions, please ask here or by direct email.

K.O.

We had our periodic discussion on MIDAS web page user permissions. (I cannot 
find the link to the previous discussions, ouch!)

Currently any logged in user can do anything - start stop runs, start/stop 
programs, edit odb, etc.

Regularly, we have experiments that ask about "read-only" access to MIDAS and 
about more granular user permissions.

In the past, I suggested a permissions scheme that is easy to implement
with the current code base. Permission level for each user can
be stored in ODB and allow:

level 0 - root user, as now
level 1 - experiment user, any restrictions are implemented in javascript, i.e. 
all custom pages work as they do now, but (i.e.) the odb editor is read-only
level 2 - experiment operator, restrictions are implemented in the mjsonrpc 
code, i.e. can start/stop runs, start/stop programs, but cannot make any 
changes, i.e. cannot write to ODB
level 3 - read-only user - only mjsonrpc calls that do not change anything are 
permitted.

(to implement level 2, obviously, the "start run" mjsonrpc call has to be 
changed to accept the run comments, current code writes them to odb directly and 
that would fail).

First step towards implementing this was made today. Ben and Derek figured out 
the apache incantation to pass the logged user name to MIDAS and I added 
decoding of this user name in mhttpd. I do not do anything with it, yet.

In apache config, one change is needed:

> For Apache, add this line in your VirtualHost section (tested as working):
> RequestHeader set X-Remote-User %{REMOTE_USER}s

https://daq00.triumf.ca/DaqWiki/index.php/Ubuntu#Install_apache_httpd_proxy_for_midas_and_elog

K.O.

> when I attempted to increase the max number of hotlinks in ODB , defined as 
> #define MAX_OPEN_RECORDS       256           /**< number of open DB records   */
> assert(sizeof(DATABASE_CLIENT) == 2112);

Yes, it is intended to work like this. If you change MAX_OPEN_RECORDS (and some settings),
you break binary compatibility with standard MIDAS and the asserts inform you about it.

It is not a light step to take - you have to recompile all MIDAS clients, and if you miss
one and run it against your non-standard MIDAS, kaboom everything will go,
there is no safety net against this.

In the ALPHA experiment at CERN, for years we have been running with MAX_OPEN_RECORDS set to 2560,
and it works, you have to change both MAX_OPEN_RECORDS in midas.h and the expected values
in the assert() statements.

The new correct values you do not need to guess or compute yourself, the code to print
them is right there and it is easy to enable.

Replacing the numeric constants with computed values of course would completely defeat
the purpose of the tests - to catch the situation where by mistake or by ignorance
(or by miscompilation) sizes of critical data structures become different from those
normally expected.

K.O.

> Indeed, updating MIDAS clients on each and every RPI etc in a running experiment may be a real challenge.

actually, only local clients must be rebuilt, remote clients connecting to the mserver do not care about ODB 
internal structure.

> Thinking forward - would it help if the ODB clients, upon initial connection but before doing anything else 
> were reading the ODB parameters from the ODB itself, so the clients were "learning" about the ODB structure 
> dynamically, at run time? Or that knowledge has to be static ?

unfortunately, the "open records" structure is allocated at compile-time inside the ODB header,
making any change to this would break binary compatibility.

I think it is possible to allocate "space for additional open records" in the ODB data area
and have the ODB open records code use it in addition to the compile-time allocated
space in the database header. This would also work for extending MAX_CLIENTS.

Of course in this approach, old midas clients would see only the clients and open records
in the database header, new midas clients would see the additional data.

It is not super hard to add this code...

K.O.

> I wonder why one needs more than 256 hotlinks at all.

I confirm that ALPHA is running with MAX_OPEN_RECORDS changed from 256 to 2048,
this is the only experiment I know of that had to increase any MIDAS ODB defaults.

The reason for this is mlogger, it opens an open record for each variable in each equipment.

This should be changed to 1 db_watch per equipment. We talked about it, but I guess we never did it.

I think this task just went almost to the top of my MIDAS to-do list.

K.O.

Due my oversight, the code for extracting ODB dumps from MIDAS data files from rootana/old_analyzer/event_dump.cxx was missing in 
manalyzer.cxx.

This is now corrected, the new manalyzer command line flag is "--save-odb", to use it:

daq00:manalyzer$ ./manalyzer_test.exe --save-odb ~/git/midas/testexpt/run00002.mid.lz4
...
Saving begin of run ODB dump for run 2 from "/home/olchansk/git/midas/testexpt/run00002.mid.lz4" to "run2bor.json"
...
Saving end of run ODB dump for run 2 from "/home/olchansk/git/midas/testexpt/run00002.mid.lz4" to "run2eor.json"
...

manalyzer commit f4cbcb7426083edc9f74298965c90a3a91f461ab

K.O.

While testing manalyzer, I found that it dies from an exception on odbxx, error message is "/home/olchansk/packages/midas/include/odbxx.h:1231: No "handle" 
attribute found in XML data".

Indeed, my data file is very old and it's XML ODB dump does not have the "handle" attribute:

daq00:midas$ more ~/git/midas/manalyzer/run9402bor.xml 

<?xml version="1.0" encoding="ISO-8859-1"?>
<!-- created by MXML on Tue Aug 11 14:47:16 2020 -->
<odb root="/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://midas.psi.ch/odb.xsd">
  <dir name="Experiment">
    <key name="ODB timeout" type="INT32">10000</key>

While current MIDAS XML ODB dumps have it:

<?xml version="1.0" encoding="ISO-8859-1"?>
<!-- created by MXML on Thu May 21 20:37:06 2026 -->
<odb root="/" filename="odb.xml" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xsi:noNamespaceSchemaLocation="/home/olchansk/packages/midas/odb.xsd">
  <dir name="System" handle="135320">
    <dir name="Flush" handle="135408">
      <key name="Flush period" type="UINT32" handle="135496">60</key>


And odbxx requires this attribute unconditionally:

            if (mxml_get_attribute(node, "handle") == nullptr)
               mthrow("No \"handle\" attribute found in XML data");
            o->set_hkey(std::stoi(std::string(mxml_get_attribute(node, "handle"))));

The "handle" attribute was added to XML ODB dumps in September 2024 (not sure to what purpose, JSON ODB dumps do not have a "handle" attribute):

git blame src/odb.cxx
...
dd23558fbd src/odb.cxx (Stefan Ritt 2024-09-20 15:30:00 +0200  9387) mxml_write_attribute(writer, "handle", std::to_string(hKey).c_str());

This change makes MIDAS data files written before this date un-analyzable (unless odbxx is turned off).

I can prevent manalyzer from crashing by catching the exception, but I think it is better if odbxx code is updated to accept the pre-Sep-2024 ODB XML data 
format (which were valid XML ODB dumps when they were made and users are stuck with them inside compress binary MIDAS data files).

P.S. also please check the odbxx code for other crashes on malformed XML ODB dumps, it should complain, fail to load the dump, but not core dump or abort. 
Malformed ODB dumps is not a theoretical situation, I am currently looking at MIDAS data files (mid.lz4) that have invalid JSON ODB dumps created from 
corrupted ODB. Luckily the JSON parser handles this gracefully, does not crash manalyzer and I can look at the data. I did have to go 10 runs into the past 
to find an uncorrupted ODB dump to reload a good ODB. Fixes to the JSON encoder and fixes for corrupt ODB are in progress.

K.O.

I updated the midas forum elog to the latest version from git: 083448f7

Also investigated elogd failure to start on reboot,
it turned out to be a crasher bug, see
https://elog.psi.ch/elogs/Forum/69919

K.O.

> I fixed that by not requiring the handle explicitly ...
>
> [it was...] an uncaught exception. If you do not want the abort, catch the exception. 

Hi, Stefan! Thank you for fixing this!

The issue was not the exception, but the failure to load the XML ODB dump from an (immutable) data file.

This should now be fixed (TBC), so all good now.

K.O.

> Multithreaded transitions were introduced by KO in 2019. Please ask him to make deferred transitions work 
> again or simply use non-multithreaded transitions. 

Deferred transition is the bane of MIDAS, I personally can never understand how they work
and what they do, and I studied them and understood them many times now.

I recommend against using them. Maybe you can explain what you do and I can suggest a way
to avoid using the deferred transition.

Some other people use deferred transitions, and it works for them,
in conjunction with normal transitions, which have been multithreaded
for years.

So unlikely this is a new bug.

P.S. I do not remember any use of deferred transition in the T2K/ND280 FGD, TPC and GSC frontends,
maybe it is in some other subsystem or was introduced after my time.

K.O.

you can also copy the reconnect code from history_schema.cxx.

if I remember right, the test case for correct reconnect is to start the mlogger, 
have it write something to mysql, then restart mysqld (mlogger connection is now 
broken), then have mlogger write some more to mysql. it should get an error 
"connection broken", reconnect and retry the write.

(to avoid it getting stuck, there should be a limit of 10 attempts to reconnect 
plus retry, in case each retry crashes mysqld or throws an unrelated error).

the code in question I think is mlogger mysql supoprt for begin and end of run, 
so you start a run, restart mysqld, stop the run, start a new run and should see 
a successful reconnect.

K.O.

> I was wondering if one can use multiple pysequencers at the same time and if not
> if this feature is planned in the future? Our experiment (mu3e) has one frontend
> per detector and each sequencer would only access a subset of ODB entries.

I believe sequencer design supports this, same as mlogger supports multiple output 
files and mhttpd supports multiple listener ports.

But how to do this with pysequencer, I am not sure. Simplest is to wait a few more 
weeks for Ben to return from vacation.

K.O.

> Deferred transition is registered by the FPN00

FPN00, yes, this rings a bell.

> (main clock) process to make sure the logger has finished logging all events before continuing to stop readout
> of other frontends. There is also a timeout, in case sometimes an event goes missing, to proceed with the run
> stop without getting stuck waiting for the logger.

yes, that's right, if trigger control and run control are in the same frontend, you run into this trouble:

user pushes run stop button
you get the run stop callback
you stop the trigger
but have to wait for all the data to flush down the pipes all the way to mlogger
but cannot wait, must return from the callback otherwise run transition is stuck

if trigger control and run control are in different frontends, things are simpler:

user pushes run stop botton
trigger control frontend disables the trigger, returns without waiting for anything
data frontend (i.e. FGD, TPC), flush all the hardware FIFOs, etc to MIDAS SYSTEM buffer (trigger is already 
disabled, there is no new data)
mlogger flushes SYSTEM buffer to disk
run control frontend reports successful run stop to the run database (and whatever else).

> Interestingly, I just checked and this feature is
> disabled in the FGD ODB, but it is enabled in the TPC ODB.

I do not have access to the current code, but I can check what I have,
and I am pretty sure it did not have any deferred transitions. Maybe
it was added after my time.

K.O.