Hi,
We're getting errors in our build system like:

/code/midas/manalyzer/manalyzer.cxx: In member function ‘void Profiler::Begin(TARunInfo*, 
std::vector<TARunObject*>)’:
/code/midas/manalyzer/manalyzer.cxx:799:27: error: ‘pow’ was not declared in this scope
  799 |       bins[i] = TimeRange*pow(1.1,i)/pow(1.1,Nbins);

The solution is to add "#include <cmath>" at the top of manalyzer.cxx; I guess on a lot of systems the 
include is implicit from some other include so doesn't cause errors. I don't have the permissions to push 
branches, could this be added please?

Thanks,

Mark.

Thanks for reporting this. It was caused by a

  MJsonNode* node = MJsonNode::Parse(str.c_str());

not followed by a 

  delete node;

I added that now in odb::odb_from_json_string(). Can you try again?

Stefan

I updated the root helper constructor to give the user more control over ROOT output file names.

You can now change it to anything you want in the module run constructor, see manalyzer_example_esoteric.cxx

Is this good enough?

struct ExampleE1: public TARunObject
{
   ExampleE1(TARunInfo* runinfo)
      : TARunObject(runinfo)
   {
#ifdef HAVE_ROOT
      if (runinfo->fRoot)
         runinfo->fRoot->fOutputFileName = "my_custom_file_name.root";
#endif
   }
}

K.O.

> >  #include "manalyzer.h"
> >  #include "midasio.h"
> > +#include "odbxx.h"
> 
> This commit broke the standalone ("no MIDAS") build of manalyzer. Either odbxx has to be an independant package 
> (like mvodb) or it has to be conditioned on HAVE_MIDAS.
> 
> (this was flagged by failed bitbucket build of rootana)

Corrected. You can only use odbxx is manalyzer is built with HAVE_MIDAS. (mvodb is an independant package and is 
always available, no need to pull and build the full MIDAS).

Also notice how I now initialize odbxx from fBorOdbDump and fEorOdbDump. Also tested against multithreaded access, it 
works (as Stefan promised).

K.O.

I was testing odbxx with manalyzer, decided to print an odb value in every event, 
and it worked fine in online mode, but bombed out when running from a data file 
(JSON ODB dump). The following code has a memory leak. No idea if XML ODB dump 
has the same problem.

int memory_leak()
{
   midas::odb::set_odb_source(midas::odb::STRING, std::string(run.fRunInfo-
>fBorOdbDump.data(), run.fRunInfo->fBorOdbDump.size()));

   while (1) {
      int time = midas::odb("/Runinfo/Start time binary");
      printf("time %d\n", time);
   }
}

K.O.

The sequencer pages were adjusted to the work with this bug fix.

> This commit breaks the sequencer pages...
> 
> > Indeed a bug. Fixed in commit
> > 
> > https://bitbucket.org/tmidas/midas/commits/5c1133df073f493d74d1fc4c03fbcfe80a3edae4
> > 
> > Stefan

>  #include "manalyzer.h"
>  #include "midasio.h"
> +#include "odbxx.h"

This commit broke the standalone ("no MIDAS") build of manalyzer. Either odbxx has to be an independant package 
(like mvodb) or it has to be conditioned on HAVE_MIDAS.

(this was flagged by failed bitbucket build of rootana)

K.O.

I added cmake support for the thread sanitizer (address sanitizer was already 
there). Use:

make cmake -j YES_THREAD_SANITIZER=1 # (or YES_ADDRESS_SANITIZER=1)

However, thread sanitizer is broken on U-24, programs refuse to start ("FATAL: 
ThreadSanitizer: unexpected memory mapping") and report what looks like bogus 
complaints about mutexes ("unlock of an unlocked mutex (or by a wrong thread)").

On macos, thread sanitizer does not report any errors or warnings or ...

P.S.

The Undefined Behaviour Sanitizer (UBSAN) complained about a few places where 
functions could have been called with a NULL pointer arguments, I added some 
assert()s to make it happy.

K.O.

With the merge of RPC_CXX code, MIDAS RPC can now return data of arbitrary large size and I am 
proceeding to update the corresponding mjsonrpc interface.

If you use JRPC and BRPC in the tmfe framework, you need to do nothing, the updated RPC handlers 
are already tested and merged, the only effect is that large data returned by HandleRpc() and 
HandleBinaryRpc() will no longer be truncated.

If you use your own handlers for JRPC and BRPC, please add the RPC handlers as shown at the end 
of this message. There is no need to delete/remove the old RPC handlers.

To avoid unexpected breakage, the new code is not yet enabled by default, but you can start 
using it immediately by replacing the mjsonrpc call:

mjsonrpc_call("jrpc", ...

with

mjsonrpc_call("jrpc_cxx", ...

ditto for "brpc", see resources/example.html for complete code.

After migration is completed, if you have some old frontends where you cannot add the new RPC 
handlers, you can still call them using the "jrpc_old" and "brpc_old" mjsonrpc calls.

I will cut-over the default "jrpc" and "brpc" calls to the new RPC_CXX in about a month or so.

If you need more time, please let me know.

K.O.

Register the new RPCs:

   cm_register_function(RPC_JRPC_CXX, rpc_cxx_callback);
   cm_register_function(RPC_BRPC_CXX, binary_rpc_cxx_callback);

and add the handler functions: (see tmfe.cxx for full example)

static INT rpc_cxx_callback(INT index, void *prpc_param[])
{
   const char* cmd  = CSTRING(0);
   const char* args = CSTRING(1);
   std::string* pstr = CPSTDSTRING(2);

   *pstr = "my return data";

   return RPC_SUCCESS;
}

static INT binary_rpc_cxx_callback(INT index, void *prpc_param[])
{
   const char* cmd  = CSTRING(0);
   const char* args = CSTRING(1);
   std::vector<char>* pbuf = CPSTDVECTOR(2);

   pbuf->clear();
   pbuf->push_back(my return data);

   return RPC_SUCCESS;
}

K.O.

> > This is moving slowly. I now have RPC caller side support for std::string and 
> > std::vector<char>. RPC server side is next. K.O.
> The RPC_CXX code is now merged into MIDAS branch feature/rpc_call_cxx.
> This code fully supports passing std::string and std::vector<char> through the MIDAS RPC is both directions.

The RPC_CXX in now merged into MIDAS develop. commit 34cd969fbbfecc82c290e6c2dfc7c6d53b6e0121.

There is a new RPC parameter encoder and decoder. To avoid unexpected breakage, it is only used for newly added RPC_CXX 
calls, but I expect to eventually switch all RPC calls to use the new encoder and decoder.

As examples of new code, see RPC_JRPC_CXX and RPC_BRPC_CXX, they return RPC data in an std::string and std::vector<char> 
respectively, amount of returned data is unlimited, mjsonrpc parameter "max_reply_length" is no longer needed/used.

Also included of RPC_BM_RECEIVE_EVENT_CXX, it receives event data as an std::vector<char> and maximum event size is no 
longer limited, ODB /Experiment/MAX_EVENT_SIZE is no longer needed/used. To avoid unexpected breakage, this new code is not 
enabled yet.

K.O.

> Now, mlogger just silently continues not writing to history. There is no ongoing error message, there is no 
> ongoing alarm, only sign of trouble is empty history plots and history files not growing.

I would recommend to use an "internal alarm". This is not an "ODB alarm" where values are compared limits, but it is directly triggered by C code. To do 
so, call 

  if (disk_full)
     al_trigger-alarm("Disk full", "Disk full, no history will be written", "Alarm", "Disk full", AT_INTERNAL);

This will cause an alarm to show up prominently on the status page and beep every few minutes.

Stefan

> 3b) mlogger used to rescan ODB each time a new run is started, this code was removed

One more kink turned out.

One of the computers ran out of disk space, mlogger dutifully recorded the "disk full" errors to midas.log and 
disabling writing to history (all history variables).

This was only noticed about one week later (it is not a busy computer).

In the past, when mlogger reopened the history at each begin of run, the "disk full" errors would have shown 
up in midas.log and somebody would have noticed. Or the problem would have gone away if disk space was cleared 
up.

Now, mlogger just silently continues not writing to history. There is no ongoing error message, there is no 
ongoing alarm, only sign of trouble is empty history plots and history files not growing.

Perhaps we should add an mlogger action to ask the history, "are you ok?" and report in midas.log or alarm if 
history is not happy.

Or have mlogger at the begin of run automatically reenable all disabled history variables. If these variables 
are still unhappy (error writing to disk or to mysql), there will be an error message in midas.log (and 
automatic self-disable).

All these solutions should be okey as long as they do not touch disk storage and so do not cause any long 
delay in run start.

K.O.

In the DL experiment (unknown version of midas, likely mid-summer 2025), we see artefacts in the 
history display where pieces of the data seem to be missing, there is gaps in the graphs. reloading the 
page restores correct display confirming that in fact there is no gaps in the data. This made history 
plots very painful to use.

This problem does not exist anymore in the latest midas, most likely it was fixed around September 4, 
2025. Most likely it was broken since at least February 2025 (previous changes to this file).

If you see this problem, updating mhistory.js to latest version is probably enough to fix it.

K.O.

Just to remember, MIDAS has cm_expand_env() to expand environment variables, in 
file paths, etc. It is used in several places in mhttpd, msequencer and mjsonrpc.

std::string my_secret_file = cm_expand_env("$HOME/.ssh/authorised_keys");

One could add it everywhere we open files, etc, except for the security 
consideration. We should not permit any/every web site to read any/every local 
file (directly by injecting malicious js code or by cross-site mjsonrpc call). 
Access should be limited to files in designated MIDAS experiment subdirectories. 
Places like $HOME/.ssh, $HOME/.cache/google-chrome, etc must be protected.

K.O.

> This is moving slowly. I now have RPC caller side support for std::string and 
> std::vector<char>. RPC server side is next. K.O.

The RPC_CXX code is now merged into MIDAS branch feature/rpc_call_cxx.

This code fully supports passing std::string and std::vector<char> through the MIDAS RPC is both directions.

For data passed from client to mserver, memory for string and vector data is allocated automatically as needed.

For data returned from mserver to client, memory to hold returned string and vector data is allocated automatically as 
need.

This means that RPC calls can return data of arbitrary size, the rpc caller does not need to know maximum data size.

Removing this limitation was the main motivation for this development.

I completed this code in June 2024, but could not merge it because I broke my git repository (oops). Now I am doing 
the merge manually. Changes are isolated to rpc_call_encode(), rpc_call_decode() and rpc_execute(). My intent right 
now is to use the new RPC code only for RPCs that pass std::string and std::vector<char>, existing RPCs will use the 
old code without any changes. This seems to be the safest way to move forward.

Included is test_rpc() which tests and probes most normal uses cases and some corner cases. When writing the test 
code, I found a few bugs in the old MIDAS RPC code. If I remember right, I committed fixes for those bugs to main 
MIDAS right then and there.

K.O.

I updated mvodb and test_mvodb. MIDAS ODB and JSON ODB now implement all API 
functions. ReadKey, ReadDir and ReadKeyLastWritten were previously missing from 
some implementations.

I do not remember any other bugs or problems in mvodb, if you want me to add, fix 
or change something, please speak up!

K.O.

Just in time, enter std::string_view.
https://stackoverflow.com/questions/40127965/how-exactly-is-stdstring-view-faster-than-const-stdstring

I was looking at https://root.cern/doc/v638/classROOT_1_1Experimental_1_1RFile.html and they use it everywhere instead of 
std::string and const char*.

(so now we have 4 string types to deal with, counting ROOT's TString).

P.S. For extra safety, this code compiles, then explodes:

std::string_view get_temporary_string() {
  std::string s = "temporary";
  return s; // DANGER! 's' is destroyed, view dangles.
}

K.O.

> > 2) "advanced" c++ code:
> > 
> > void foo(const std::string& xxx) { ... };
> > int main() { foo("bar"); }
> > 
> > copy-created 2nd string is avoided, but string object to hold "bar" is still must be 
> > made, 1 malloc(), 1 memcpy().
> 
> Are you sure about this? I always thought that foo only receives a pointer to xxx which it puts on the stack, so
> no additional malloc/free is involved.

Yes, "bar" is not an std::string, cannot be used to call foo(), and the c++ compiler has to automagically rewrite 
the function call

from: int main() { foo("bar"); }
to:   int main() { foo(std::string("bar"); }

the temporary std::string object may be on the stack, but storage for text "bar" is on the heap (unless std::string 
is optimized to store short strings internally).

one can put a printf() inside foo() to print the address of xxx (should be on the stack) and xxx.c_str() (should be 
on the heap). one could also try to print the address of "bar" (should be in the read-only-constant-strings memory 
area). (I am not sure if compiler-linker combines all instances of "bar" into one, this is also easy to check).

K.O.

> 2) "advanced" c++ code:
> 
> void foo(const std::string& xxx) { ... };
> int main() { foo("bar"); }
> 
> copy-created 2nd string is avoided, but string object to hold "bar" is still must be 
> made, 1 malloc(), 1 memcpy().

Are you sure about this? I always thought that foo only receives a pointer to xxx which it puts on the stack, so
no additional malloc/free is involved.

Have a look here: https://en.cppreference.com/w/cpp/language/reference.html

It says "References are not objects; they do not necessarily occupy storage".

Stefan

> 1) history is independent from "runs", we see a change, we apply it (even if it takes 10 sec or 2 minutes).
> 
> 2) "nothing should change during a run", we must process all changes before we start a run (starting a run takes forever),
>    and we must ignore changes during a run (i.e. updated frontend starts to write new data to history). (this is why
>    the trick to "start a new run twice" used to work).

"nothing should change during a run" violates the action when a user adds a new variable during a run. So if the user does that, they don't
care if things change during a run. Then we can also modify the history DB during the run. Note that some MIDAS installations are purely
slow control (kind of a replacement of LabView, have no runs at all). In those installations runs do not make sense at all, so keeping the
history independent of runs makes sense to me.

> It is "free" to rescan ODB every 10 second or so. Then we can output a midas message "please restart the logger",
> and set an ODB flag, then when user opens the history panel editor, it will see this flag
> and tell the user "please restart the logger to see the latest changes in history". It can even list
> the specific changes, if we want ot be verbose about it.

Sounds good to me.

> I say, let's take the low road for now and see if it's good enough:
> 
> a) have the history system report any changes in midas.log - "history event added", "new history variable added" (or "renamed"),
>    this will let user see that their changes to the equipment frontend "took" and flag any accidental/unwanted changes.
> 
> b) have mlogger periodically scan ODB and set a "please restart me" flag. observe this flag in the history editor
>    and tell the user "please restart the logger to see latest changes in the history".

Actually you don't have to actively "scan" the ODB. You have hotlinks to the logger anyway from the equipment variables. All we need 
in addition is a hotline to the settings array in the ODB. The logger receives the hotline update, checks if the names changed or got
extended, then flags this as a change.

Stefan