The latest version of mfe.c has a problem where poll_event() is called before
frontend_init() and this causes a crash because in poll_event() we try to access
VME before it is initialized in frontend_init(). K.O.

> The latest version of mfe.c has a problem where poll_event() is called before
> frontend_init() and this causes a crash because in poll_event() we try to access
> VME before it is initialized in frontend_init(). K.O.

Oops, that sneaked in when doing the last modification to display the frontend status. 
I refactored register_equipment() so that frontend_init() gets called before 
poll_event().

A little advertised feature of the modifications needed support the msysmon program is 
that MIDAS equipment names can support the injecting of the hostname of the system 
running the frontend at runtime (register_equipment(void)).

https://midas.triumf.ca/MidasWiki/index.php/Equipment_List_Parameters#Equipment_Name

A special string ${HOSTNAME} can be put in any position in the equipment name. It will 
be replaced with the hostname of the computer running the frontend at run-time. Note, 
the frontend_name string will be trimmed down to 32 characters.
Example usage: msysmon


EQUIPMENT equipment[] = {

  { "${HOSTNAME}_msysmon",   /* equipment name */    {
      EVID_MONITOR, 0,      /* event ID, trigger mask */
      "SYSTEM",             /* event buffer */
      EQ_PERIODIC,          /* equipment type */
      0,                    /* event source */
      "MIDAS",              /* format */
      TRUE,                 /* enabled */
      RO_ALWAYS,            /* Read when running */
      10000,                /* poll every so milliseconds */
      0,                    /* stop run after this event limit */
      0,                    /* number of sub events */
      1,                    /* history period */
      "", "", ""
    },
    read_system_load,/* readout routine */
  },
  { "" }
};

The JSON RPC branch has been merged into main MIDAS. Other than adding new functions, there are no changes to existing MIDAS functionality.

This is the current JSON RPC schema: (from the MIDAS Help page)

------------------------------------------------------------------------
Autogenerated schema for all MIDAS JSON-RPC methods
------------------------------------------------------------------------
cm_exist?      | calls MIDAS cm_exist() to check if given MIDAS program is running
               | -------------------------------------------------------
               | params   | name           | string         | name of the program, corresponding to ODB /Programs/name
               |          | unique?        | bool           | bUnique argument to cm_exist()
               | -------------------------------------------------------
               | result   | status         | integer        | return status of cm_exist()
------------------------------------------------------------------------
cm_shutdown?   | calls MIDAS cm_shutdown() to stop given MIDAS program
               | -------------------------------------------------------
               | params   | name           | string         | name of the program, corresponding to ODB /Programs/name
               |          | unique?        | bool           | bUnique argument to cm_shutdown()
               | -------------------------------------------------------
               | result   | status         | integer        | return status of cm_shutdown()
------------------------------------------------------------------------
db_copy?       | get copies of given ODB subtrees in the "save" json encoding
               | -------------------------------------------------------
               | params   | paths[]        | array of ODB subtree paths, see note on array indices
               |          |                | array of       | string 
               | -------------------------------------------------------
               | result   | data[]         | copy of ODB data for each path
               |          |                | array of       | object 
               |          | status[]       | return status of db_copy_json() for each path
               |          |                | array of       | integer
               |          | last_written[] | last_written value of the ODB subtree for each path
               |          |                | array of       | number 
------------------------------------------------------------------------
db_create?     | get copies of given ODB subtrees in the "save" json encoding
               | -------------------------------------------------------
               | params[] | array of ODB paths to be created
               |          | array of       | arguments to db_create() and db_resize()
               |          |                | path           | string  | ODB path
               |          |                | type           | integer | MIDAS TID_xxx type
               |          |                | array_length?  | integer | optional array length, default is 1
               |          |                | string_length? | integer | for TID_STRING, optional string length, default is NAME_LENGTH
               | -------------------------------------------------------
               | result   | status[]       | return status of db_create() for each path
               |          |                | array of       | integer
------------------------------------------------------------------------
db_get_values? | get values of ODB data from given subtrees
               | -------------------------------------------------------
               | params   | paths[]        | array of ODB subtree paths, see note on array indices
               |          |                | array of       | string 
               | -------------------------------------------------------
               | result   | data[]         | values of ODB data for each path, all key names are in lower case, all symlinks are followed
               |          |                | array of       | any    
               |          | status[]       | return status of db_copy_json() for each path
               |          |                | array of       | integer
               |          | last_written[] | last_written value of the ODB subtree for each path
               |          |                | array of       | number 
------------------------------------------------------------------------
db_paste?      | write data into ODB
               | -------------------------------------------------------
               | params   | paths[]        | array of ODB subtree paths, see note on array indices
               |          |                | array of       | string 
               |          | values[]       | data to be written using db_paste_json()
               |          |                | array of       | any    
               | -------------------------------------------------------
               | result   | status[]       | return status of db_paste_json() for each path
               |          |                | array of       | integer
------------------------------------------------------------------------
get_debug?     | get current value of mjsonrpc_debug
               | -------------------------------------------------------
               | params   | any            | there are no input parameters
               | -------------------------------------------------------
               | result   | integer        | current value of mjsonrpc_debug
------------------------------------------------------------------------
get_schema?    | Get the MIDAS JSON-RPC schema JSON object
               | -------------------------------------------------------
               | params   | any            | there are no input parameters
               | -------------------------------------------------------
               | result   | object         | returns the MIDAS JSON-RPC schema JSON object
------------------------------------------------------------------------
null?          | RPC method always returns null
               | -------------------------------------------------------
               | params   | any            | method parameters are ignored
               | -------------------------------------------------------
               | result   | null           | always returns null
------------------------------------------------------------------------
set_debug?     | set new value of mjsonrpc_debug
               | -------------------------------------------------------
               | params   | integer        | new value of mjsonrpc_debug
               | -------------------------------------------------------
               | result   | integer        | new value of mjsonrpc_debug
------------------------------------------------------------------------
start_program? | start MIDAS program defined in ODB /Programs/name
               | -------------------------------------------------------
               | params   | name           | string         | name of the program, corresponding to ODB /Programs/name
               | -------------------------------------------------------
               | result   | status         | integer        | return status of ss_system()
------------------------------------------------------------------------
user_example1? | example of user defined RPC method that returns up to 3 results
               | -------------------------------------------------------
               | params   | arg            | string         | example string argment
               |          | optional_arg?  | integer        | optional example integer argument
               | -------------------------------------------------------
               | result   | string         | string         | returns the value of "arg" parameter
               |          | integer        | integer        | returns the value of "optional_arg" parameter
------------------------------------------------------------------------
user_example2? | example of user defined RPC method that returns more than 3 results
               | -------------------------------------------------------
               | params   | arg            | string         | example string argment
               |          | optional_arg?  | integer        | optional example integer argument
               | -------------------------------------------------------
               | result   | string1        | string         | returns the value of "arg" parameter
               |          | string2        | string         | returns "hello"
               |          | string3        | string         | returns "world!"
               |          | value1         | integer        | returns the value of "optional_arg" parameter
               |          | value2         | number         | returns 3.14
------------------------------------------------------------------------
user_example3? | example of user defined RPC method that returns an error
               | -------------------------------------------------------
               | params   | arg            | integer        | integer value, if zero, throws a JSON-RPC error
               | -------------------------------------------------------
               | result   | status         | integer        | returns the value of "arg" parameter

K.O.

By request from Renee, I increased the MIDAS BANKLIST_MAX from 64 to 1024 and
after fixing a few buglets where YB_BANKLIST_MAX is used instead of (now bigger)
BANKLIST_MAX, I can do a full dump of ND280 FGD events (96 banks).

I also noticed that "mdump -b BANK" did not work, it turns out that it could not
handle 32bit-banks at all. This is now fixed, too.

svn rev 4624
K.O.

I converted mdump file i/o from older mdsupport library to newer midasio library 
and it can now read .mid, .mid.gz, .mid.lz4 and .mid.bz2 files. Output should be 
identical to what it printed before, if you see any differences, please report 
them here or on bitbucket. K.O.

> > > the update of mhttpd to mongoose version 6.16 was committed to the develop branch of midas.

current code looks for the mbedtls library in ../mbedtls (next to midas)

if cmake misdetects it, turn it off by setting NO_MBEDTLS (same as NO_ROOT & co)

if you do want to build mhttpd with mbedtls, do this:

cd .../midas
cd ../
git clone https://github.com/ARMmbed/mbedtls.git
cd mbedtls
git submodule update --init ### this will populate the "crypto" directory
make ### if "python2" is missing, building of test suite programs will fail, but the libraries needed for midas will be built

cd ../midas
make cmake...

K.O.

I had some quick look at the new mongoose code and didn't find anything I dislike. Did a quick test of the proxy which worked and is nice to have. 
Agree with all KO said about authentication.

So if there are no complaints, I would suggest that we move the summary of this thread into the official documentation.

Stefan

manalyzer was updated to latest version. mostly multi-threading improvements from 
Joseph and myself. K.O.

Hi all, I hope this is the right place to post two pull requests, if not, please let me know where I should be submitting them

Both are fairly small changes, please see them listed below (more details written on the PRs themselves)


- Enable ROOT's thread safety when running in multithreaded mode

This helps avoid users having to write their call to a global thread lock when calling ->Fill() on ROOT histograms and Trees
https://bitbucket.org/tmidas/manalyzer/pull-requests/5


- Add command argument to specify an IP of the root HTTP server to bind to

This was a problem I painted around when at ALPHA (quickly hardcoding the right external IP address into the local build. Obviously a bad habit)
https://bitbucket.org/tmidas/manalyzer/pull-requests/6

> - Enable ROOT's thread safety when running in multithreaded mode
> This helps avoid users having to write their call to a global thread lock when calling ->Fill() on ROOT histograms and Trees
> https://bitbucket.org/tmidas/manalyzer/pull-requests/5

merged by hand. (pull request shows a "rejected", bitbucket has no "merged manually" button).

also noted this change in the documentation: README.md

K.O.

Hi all,

Would it be possible to extend manalyzer to support custom .root file names that include the run number? 

As far as I understand, the current behavior is as follows:
The default filename is ./root_output_files/output%05d.root , which can be customized by the following two command line arguments.

-Doutputdirectory: Specify output root file directory
-Ooutputfile.root: Specify output root file filename

If an output file name is specified with -O, -D is ignored, so the full path should be provided to -O. 

I am aiming to write files where the filename contains sufficient information to be unique (e.g., experiment, year, and run number). However, if I specify it with -O, this would require restarting manalyzer after every run; a scenario that I would like to avoid if possible.

Please find a suggestion of how manalyzer could be extended to introduce this functionality through an additional command line argument at
https://bitbucket.org/krieger_j/manalyzer/commits/24f25bc8fe3f066ac1dc576349eabf04d174deec

Above code would allow the following call syntax: ' ./manalyzer.exe -O/data/experiment1_%06d.root --OutputNumbered '
But note that as is, it would fail if a user specifies an incompatible format such as -Ooutput%s.root . 

So a safer, but less flexible option might be to instead have the user provide only a prefix, and then attach %05d.root in the code.

Thank you for considering these suggestions!

Andrea Capra reported a problem with manalyzer module initialization order.

Original manalyzer design relies on the fact than module static constructors run in the same order as they 
appear on the linker command line. This has been true for a very long time, but now we have evidence that on 
Rocky Linux (unknown gcc version), this is no longer true.

The c++ standard does not define any specific order for static constructor in different source files. 
(unlike C, C++ tends to treat the linker as something magical and ask the linker to do magical things).

The tmfe c++ modular frontend was designed after manalyzer and one design change is to explicitly construct 
all objects from main(). (at the acceptable cost of extra boiler plate code).

One solution is to use GCC attribute "init_priority (priority)", see
https://stackoverflow.com/questions/211237/static-variables-initialisation-order
https://gcc.gnu.org/onlinedocs/gcc/C_002b_002b-Attributes.html

To use this, manalyzer module registration should be modified to read:

static TARegister tar __attribute__((init_priority(200))) (new ExampleCxxFactory);

A less magical solution is to change manalyzer design similar to tmfe where modules are constructed and 
registered explicitely in the order specified by source code.

K.O.

P.S. I now ran out of time to test this and commit it to the documentation. It also need to be tested with 
LLVM C++ compilers.

> Andrea Capra reported a problem with manalyzer module initialization order.

Permanent solution is now implemented.

In the analyzer module constructor (TARunObject), please set the value of fModuleOrder (default value is 0).

Modules with smaller fModuleOrder will run first, modules with bigger fModuleOrder will run last.

Please use the value range between -1 (built-in EventDump module) and 9999 (built-in InteractiveModule).

Modules with equal fModuleOrder will run in the same order as they were registered (std::stable_sort).

Example manalyzer modules and documentation README.md have been updated.

Linker command line and GCC attribute methods should also work, but I thought it better to provide explicit 
programmatic control of module ordering.

P.S. the c++ tmfe frontend module design is newer and the problem of module (equipment) ordering is solved by 
constructing them explicitly in main().

manalyzer commit:

commit 6ca130808cd05ead734450391bf6defc8335c04a (HEAD -> master, origin/master, origin/HEAD)
Author: Konstantin Olchanski <olchansk@daq00.triumf.ca>
Date:   Tue Mar 25 15:53:13 2025 -0700

    implement TARunObject::fModuleOrder to specify required ordering of analyzer modules

K.O.

updated manalyzer:

- similar to --jsroot switch, in online mode, the ROOT output file remains open after run is stopped. Previously, after run was 
stopped, all histograms & etc would disappear from JSROOT, making it hard to look at the full collected and analyzed data.

- there was a buglet in the multithreading code, if some module cannot analyze flow events as fast as we can read data from disk, 
the flow event queue of the first module thread would grow and grow and grow infinitely, potentially consume lots of RAM. This is 
because control of queue size for the first module thread was disabled to avoid a deadlock. I now added the queue size check to the 
main event loop (both offline mode and online mode) and this problem should now be fixed.

- also adjusted the default queue size from 100 to 1000 and queue-full wait sleep time from 100 us to 10 us.

- another buglet was in the flow event processing. per the README, module EndRun() should not generate flow events (instead, they 
should be generated in PreEndRun()). Previously this was not enforced, now there is an error message about this and the offending 
flow events are deleted. (they were not being processed anyway).

K.O.

When processing MIDAS files offline, JSROOT did not work, -Rxxx worked, http 
connection would open, but would not serve any histograms. This should now be 
fixed.

In addition, normally, after processing all input MIDAS files, manalyzer would 
exit, JSROOT would abruptly stop. To look at final results one had to open the 
ROOT files using some other method (roody, TBrowser, mjsroot, etc).

I now added a command line switch "--jsroot", if supplied, after processing all 
input MIDAS files, manalyzer will keep running in the JSROOT server mode (same as 
mjsroot).

"manalyzer -R8082 --jsroot run*.mid.lz4" now does something useful: open 
http://localhost:8082 (or ssh tunnel or mhttpd proxy per my mjsroot message) and 
watch histograms fill in real time, after analysis finishes, keep looking at the 
final results until bored. stop manalyzer using Ctrl-C. (we should add a "Stop 
JSROOT" botton to the JSROOT main page).

MIDAS commit 1d0d6448c3ec4ffd225b8d2030fe13e379fcd007

K.O.

We are considering to remove the analyzer framework mana.cxx from MIDAS. It 
currently has some compiler warnings and we wonder if we should fix them which 
would take some time or just remove the file. We have now to much more modern 
analyzer frameworks "manalyzer" and "ROOTANA" which should be used instead.

Is anybody still using the mana.cxx framework?

/Stefan

Ok, no relevant complains so far, so I removed mana and rmana from the CMake build 
process, but left the file mana.cxx still in the repository for educational 
purposes ;-)

Stefan

> Ok, no relevant complains so far, so I removed mana and rmana from the CMake build 
> process, but left the file mana.cxx still in the repository for educational 
> purposes ;-)

+1

K.O.

Stephan, why did you prohibit building mana.c without ROOT and HBOOK
support? I think such a configuration is valid and should be allowed.

Also, this prohibition broke the Midas Makefile, it now bombs building
mana.c. The Makefile is setup for building hmana.c with HBOOK support,
rmana.c with ROOT support (if ROOTSYS is set) and mana.c without HBOOK and
ROOT support (currently bombs on #error in mana.c).

K.O.