• A "Reload File" button
  
• Support for patterns in ODBSET, e.g.:
  

  • ODBSET "/Path/value[1,3,5]", 1

  • ODBSET "/Path/value[1-5,7-9]", 1

  • Arbitrary combinations of the above
    
• Support for variable substitution:
  

  • SET GOODCHANNELS, "1-5,7,9"; ODBSET "/Path/value[$GOODCHANNELS]", 1

  • SET BADCHANNELS, "6,8"; ODBSET "/Path/value[!$BADCHANNELS]", 1

  • ODBSET "/Path/value[0-100, except $BADCHANNELS]", 1

• Load a file
  
• Double-click it to edit it, acknowledge the "To edit the sequence it must be opened in an editor tab" dialog
  
• A new tab opens
  
• Edit something, click "Start", acknowledge the "Save and start?" dialog (which pops up even if no changes are made)
  
• Run the script
  
• Double-click to make more changes -> another tab opens
  

SET v, "54"
SET file, "MPPCHV_$v.odb"
ODBLOAD $file

SET v, "54.2"
SET file, "MPPCHV_$v.odb"
ODBLOAD $file

SET v, 54.2
SPRINTF file, "MPPCHV_%f.odb", $v
ODBLOAD $file

ODBLOAD f"MPPCHV_{v:.1f}.odb"

> ODBSET "/Path/value[1,3,5]"
> ODBSET "/Path/value[1-5,7-9]"

we support this array index syntax in several places,
specifically, in javascript odb get and set mjsonrpc RPCs.

> SET GOODCHANNELS, "1-5,7,9"; ODBSET "/Path/value[$GOODCHANNELS]"
> SET BADCHANNELS, "6,8"; ODBSET "/Path/value[!$BADCHANNELS]"
> ODBSET "/Path/value[0-100, except $BADCHANNELS]"

this is very clever syntax, but I have not seen any programming
language actually implement it (not even perl).

there must be a good reason why nobody does this. probably we should not do it either.

but as Stefan said (and my opinion), the route of extending MIDAS sequencer
language until it becomes a superset of python, perl, tcl, bash, javascript
and algol is not a sustainable approach. I once looked at using LUA for this,
but I think basing off an full featured programming language like python
is better.

K.O.

 I once looked at using LUA for this,
> but I think basing off an full featured programming language like python
> is better.

if it came to a vote, my vote would go to Lua: it would allow to do everything needed, 
with much less external dependencies and with much less motivation to over-use the interpreter. 
The CMS experience was very teaching in this respect... 

-- my 2c, regards, Pasha

> I once looked at using LUA for this
>
> > but I think basing off an full featured programming language like python
> > is better.
> 
> if it came to a vote, my vote would go to Lua: it would allow to do everything needed, 
> with much less external dependencies and with much less motivation to over-use the interpreter. 
> The CMS experience was very teaching in this respect... 

Unfortunately I am only slightly aware of Lua to say how nicve or how bad it is. And we are
not sure how well it supports the single-line-stepping that permits the nice graphical
visualization of Stefan's sequencer.

It looks like python has the single-line-stepping built-in as a standard feature
and python is a more popular and more versatile machine, so to me python looks
like a better choice compared to lua (obscure), perl ("nobody uses it anymore")
or bash (ugly syntax).

K.O.

And there is one more argument: 

We have a Python expert in our development team who wrote already the Python-to-C bindings. That means when running a Python 
script, we can already start/stop runs, write/read to the ODB etc. We only have to get the single stepping going which seems feasible to 
me, since there are some libraries like inspect.currentframe() and traceback.extract_stack(). For single-stepping there are debug APIs 
like debugpy. With Lua we really would have to start from scratch.

Stefan

Dear MIDAS experts,

I have a time series of slow control measurements in an ASCII format - 
data records in a format (run_number, time, temperature, voltage1, ..., voltageN), 
and, if possible, would like to convert them into a MIDAS history format. 

Making MIDAS events out of that data is easy, but is it possible to preserve 
the time stamps?  - Logically, this boils down to whether it is possible to have  
the event time set by a user frontend

-- as always - many thanks, regards, Pasha

I think I found an answer to my question: a user-controlled event header does have a time stamp: 

https://daq00.triumf.ca/MidasWiki/index.php/Event_Structure#Event_Header

-- apologies for the spam, regards, Pasha

> I have a time series of slow control measurements in an ASCII format - 
> data records in a format (run_number, time, temperature, voltage1, ..., voltageN), 
> and, if possible, would like to convert them into a MIDAS history format. 
> 
> Making MIDAS events out of that data is easy, but is it possible to preserve 
> the time stamps?  - Logically, this boils down to whether it is possible to have  
> the event time set by a user frontend

It looks that the original question was not as naive as I expected and may be pointing to a subtle bug. 
I have implemented a python frontend - essentially a clone of 

https://bitbucket.org/tmidas/midas/src/develop/python/midas/frontend.py

reading the old slow control data and setting the event.header.timestamp's to some dates from the year of 2022. 

When I run MIDAS and read the "old slow control events", one event in 10 seconds, 
the MIDAS Event Dump utility shows the data with the correct event timestamps, from the year of 2022. 

However the history plots show the event parameters with the timestamps from Feb 01 2025 and the adjacent 
data points separated by 10 sec.

Is it possible that the history system uses its own timestamp setting instead of using timestamps from the event headers? 
- Under normal circumstances, the two should be very close, and that could've kept the issue hidden... 

-- thanks, regards, Pasha

UPDATE:  I attached the frontend code and the input data file it is reading. The data file should reside in the local directory
- the frontend code doesn't have everything fully automated for the test, 
  -- an integer field "/Mu2e/Offline/Ops/LastTime" would need to be created manually
  -- the history plots would need to be declared manually

Dear MIDAS experts, 

I confirm that when writing out history files corresponding to the slow control event data, 
MIDAS history system timestamps the data not with the event time coming from the event data, 
but with the current time determined by the program - 

https://bitbucket.org/tmidas/midas/src/293d27fad0c87c80c4ed7b94b5c40ba1e150bea4/progs/mlogger.cxx#lines-5321

where 'now' is defined as  

time_t now = time(NULL);

I'm looking for a way to timestamp the history data with the event time - that is important 
for HEP applications outside the DAQ domain. Yes, MIDAS infrastructure is very well suited for that, 
there could have a number of such applications, and experiments could significantly benefit from that.

So I'm wondering whether the implementation is a design choice made or it could be changed. 

The change itself and especially its validation may require a non-negligible amount of work - I'd be happy to contribute.

Any insight much appreciated. 

-- thanks, regards, Pasha

> I confirm that when writing out history files corresponding to the slow control event data, 
> MIDAS history system timestamps the data not with the event time coming from the event data, 
> but with the current time determined by [mlogger].

This is correct. The timestamp in the history file is the mlogger timestamp.

In theory we could use the ODB "last_written" timestamp, but in practice,
timestamps are 1 second granularity and the difference between the two
timestamps normally would be less than 1 second. (time to react to db_watch()).

But ODB last_written also is not the data timestamp. For remote connected clients
it includes the mserver communication delay.

What is the data timestamp, only the user knows - for some FPGA based equipments,
I can see the data timestamp being read from an FPGA register together with the data.

But back to earth.

For making history plots, 1 second granularity with a small (a few seconds) delay should be okey,
and I think the mserver timestamp is good enough.

For data analysis, you are reading history data from a history data file and you are
not constrained to using the MIDAS timestamp.

You can always include your "true" data timestamp as the first value in your data.

We do this in felaview for writing labview data to midas history in the ALPHA antihydrogen experiment at CERN.

This also anticipates your next request, can we have millisecond, microsecond, nanosecond history timestamps:
since you define your "true" data timestamp, you an make it anything you want. (I use "double" time in seconds,
64-bit IEEE-754 "double" has enough precision for microsecond granularity. FPGA based devices can have timestamps
with 10 ns or 8 ns granularity, in this case a uint64_t clock counter could be more appropriate).

K.O.

> You can always include your "true" data timestamp as the first value in your data.

Are you saying that if the first data word of a history event were a timestamp, 
the MIDAS history system, when plotting the time dependencies, would use that timestamp 
instead of the mlogger timestamp?  

if that is true, what tells MIDAS that the first data word is the timestamp? 

I couldn't find a discussion of that on the page describing the history system - 

 https://daq00.triumf.ca/MidasWiki/index.php/History_System#Frontend_history_event

- perhaps I should be looking at a different page?

-- thanks again, regards, Pasha 

> > You can always include your "true" data timestamp as the first value in your data.
> 
> Are you saying that if the first data word of a history event were a timestamp, 
> the MIDAS history system, when plotting the time dependencies, would use that timestamp 
> instead of the mlogger timestamp?
>

you are correct, midas knows nothing about what you put in the history data.

what I suggested is: if you want your true data timestamp recorded in the history,
you can put it into the history data yourself, and I suggested using the 1st value,
but you can also make it the last value or the 10th value, it is up to you.

for making history plots, the history timestamp is used, as you wrote and I confirmed,
this timestamp is generated by mlogger.

what is not clear to me is why this is a problem? do you see a big difference between the 
true data timestamp and the mlogger data timestamp? bigger than 1 second? (this would change 
the shape of "last 10 minutes" plots (600 seconds). bigger than 1 minute? (this would change 
the shape of "last 1 hour plots" (60 minutes, 3600 seconds).

that said, note that we currently store the timestamp as a DWORD 32-bit UNIX time value 
which will overflow in 2038 and which is quickly becoming incompatible with the ongoing 
switch to 64-bit time_t. Ubuntu-24 already build a large number of system libraries with 64-
bit time_t and building MIDAS with 32-bit time_t may soon become as difficult as building 
32-bit MIDAS for 32-bit i686 VME processors. we have to move with the times.

what it means is that the history system data format will have to be updated to 64-bit 
time_t and at the same time, we may try to change the timestamp from mlogger-generated to 
frontend-generated.

but it is still not clear to me how that helps you, because the frontend-generated timestamp 
is not the true data timestamp that you wanted. (and only you know what the true data 
timestamp is and where it comes from and how to tell it to MIDAS).

K.O.

We see ODB corruption crashes in the DS20k vertical slice MIDAS instance.

Crash is memset() called by db_delete_key1() called by cm_connect_experiment().

I look at the source code and I see that ODB pkey and hkey validation is absent
from most iterators and it is possible for "bad" pkey to cause corruption. Many
other places in the ODB code use db_get_pkey() and db_validate_hkey() to prevent
invalid data from causing further corruption and breakage.

Also db_delete_key1() needs to be refactored and renamed db_delete_key_wlocked().

I will not do this immediately today, but hopefully next week or so.

Stack trace is attached, observe how free_data() was called on a completely invalid pkey,
bad pkey->type, bad pkey sizes, etc.

#0  __memset_avx512_unaligned_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:250
250	../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S: No such file or directory.
(gdb) bt
#0  __memset_avx512_unaligned_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:250
#1  0x00005ad4102b4217 in memset (__len=<optimized out>, __ch=0, __dest=<optimized out>) at /usr/include/x86_64-linux-
gnu/bits/string_fortified.h:59
#2  free_data (pheader=pheader@entry=0x75aaea4f4000, address=0x75aaed4cea50, size=<optimized out>, caller=caller@entry=0x5ad4102ffb6c 
"db_delete_key1") at /home/dsdaqdev/packages_common/midas/src/odb.cxx:513
#3  0x00005ad4102b6a5b in free_data (caller=0x5ad4102ffb6c "db_delete_key1", size=<optimized out>, address=<optimized out>, 
pheader=0x75aaea4f4000) at /home/dsdaqdev/packages_common/midas/src/odb.cxx:453
#4  db_delete_key1 (hDB=1, hKey=<optimized out>, level=<optimized out>, follow_links=0) at 
/home/dsdaqdev/packages_common/midas/src/odb.cxx:3789
#5  0x00005ad4102b6979 in db_delete_key1 (hDB=1, hKey=288672, level=0, follow_links=0) at 
/home/dsdaqdev/packages_common/midas/src/odb.cxx:3731
#6  0x00005ad4102cc923 in db_create_record (hDB=hDB@entry=1, hKey=hKey@entry=0, orig_key_name=orig_key_name@entry=0x7ffd75987280 
"/Programs/ODBEdit", init_str=<optimized out>) at /home/dsdaqdev/packages_common/midas/src/odb.cxx:12916
#7  0x00005ad4102cca73 in db_create_record (hDB=hDB@entry=1, hKey=hKey@entry=0, orig_key_name=orig_key_name@entry=0x7ffd75987280 
"/Programs/ODBEdit", init_str=<optimized out>) at /home/dsdaqdev/packages_common/midas/src/odb.cxx:12942
#8  0x00005ad4102a00ba in cm_set_client_info (hDB=1, hKeyClient=0x7ffd75987420, host_name=0x5ad412262ee0 "dsdaqgw.triumf.ca", 
client_name=0x7ffd759874c0 "ODBEdit", hw_type=<optimized out>, password=<optimized out>, watchdog_timeout=<optimized out>)
    at /usr/include/c++/11/bits/basic_string.h:194
#9  0x00005ad4102a902b in cm_connect_experiment1 (host_name=<optimized out>, host_name@entry=0x7ffd759876c0 "", 
default_exp_name=default_exp_name@entry=0x7ffd759876a0 "vslice", client_name=client_name@entry=0x5ad4102f28fa "ODBEdit", 
func=func@entry=0x0, 
    odb_size=odb_size@entry=1048576, watchdog_timeout=<optimized out>, watchdog_timeout@entry=10000) at 
/usr/include/c++/11/bits/basic_string.h:194
#10 0x00005ad41027e58d in main (argc=3, argv=0x7ffd759881e8) at /home/dsdaqdev/packages_common/midas/progs/odbedit.cxx:3025
(gdb) up
...
#4  db_delete_key1 (hDB=1, hKey=<optimized out>, level=<optimized out>, follow_links=0) at 
/home/dsdaqdev/packages_common/midas/src/odb.cxx:3789
3789	            free_data(pheader, (char *) pheader + pkey->data, pkey->total_size, "db_delete_key1");
(gdb) p pkey
$1 = (KEY *) 0x75aaea53b400
(gdb) p *pkey
$2 = {type = 1684370529, num_values = 0, name = '\000' <repeats 16 times>, "xQ\375\002\004\000\000\000\004\000\000\000\a\000\000", data = 0, 
total_size = 290944, item_size = 1743544378, access_mode = 0, notify_count = 0, next_key = 15, parent_keylist = 1, 
  last_written = 1953785965}
(gdb) 

K.O.

There is a long standing problem with ODB and event buffers. If they detect an 
internal data inconsistency and cannot continue running, they call abort() to 
dump core and stop.

Problem is in some code paths, they do this while holding the ODB or event 
buffer semaphore. (Linux kernel automatically releases SYSV semaphores after 
core dump is finished and program holding them is stopped).

If core dump takes longer than 10 seconds (for whatever reason, but we see this 
often enough), all other programs that wait for ODB or event buffer access, will 
also timeout and also crash (with core dump). Result is a core dump storm, at 
the end all MIDAS programs are crashed. (Luckily recovery is easy, simply 
restart everything).

Now I realize that in many situation, we do not need to hold the semaphore while 
dumping core - the content of ODB and event buffer shared memories is not 
important for debugging the crash - and it is safe to release the semaphore 
before calling abort().

This is now implemented for ODB and event buffers. Hopefully core dump storms 
will not happen again.

commit 96369c29deba1752fd3d25bed53e6594773d7e1a
release ODB semaphore before calling abort() to dump core. if core dump takes 
longer than 10 sec all other midas programs will timeout and crash.

commit 2506406813f1e7581572f0d5721d3761b7c8e8dd
unlock event buffer before calling abort() in bm_validate_client_index_locked(), 
refactor bm_get_my_client_locked()


K.O.

updated manalyzer:

- similar to --jsroot switch, in online mode, the ROOT output file remains open after run is stopped. Previously, after run was 
stopped, all histograms & etc would disappear from JSROOT, making it hard to look at the full collected and analyzed data.

- there was a buglet in the multithreading code, if some module cannot analyze flow events as fast as we can read data from disk, 
the flow event queue of the first module thread would grow and grow and grow infinitely, potentially consume lots of RAM. This is 
because control of queue size for the first module thread was disabled to avoid a deadlock. I now added the queue size check to the 
main event loop (both offline mode and online mode) and this problem should now be fixed.

- also adjusted the default queue size from 100 to 1000 and queue-full wait sleep time from 100 us to 10 us.

- another buglet was in the flow event processing. per the README, module EndRun() should not generate flow events (instead, they 
should be generated in PreEndRun()). Previously this was not enforced, now there is an error message about this and the offending 
flow events are deleted. (they were not being processed anyway).

K.O.

When processing MIDAS files offline, JSROOT did not work, -Rxxx worked, http 
connection would open, but would not serve any histograms. This should now be 
fixed.

In addition, normally, after processing all input MIDAS files, manalyzer would 
exit, JSROOT would abruptly stop. To look at final results one had to open the 
ROOT files using some other method (roody, TBrowser, mjsroot, etc).

I now added a command line switch "--jsroot", if supplied, after processing all 
input MIDAS files, manalyzer will keep running in the JSROOT server mode (same as 
mjsroot).

"manalyzer -R8082 --jsroot run*.mid.lz4" now does something useful: open 
http://localhost:8082 (or ssh tunnel or mhttpd proxy per my mjsroot message) and 
watch histograms fill in real time, after analysis finishes, keep looking at the 
final results until bored. stop manalyzer using Ctrl-C. (we should add a "Stop 
JSROOT" botton to the JSROOT main page).

MIDAS commit 1d0d6448c3ec4ffd225b8d2030fe13e379fcd007

K.O.

currently to link Midas to project one has to do several steps in cmake script:
- do `find_package`
- get Midas location from MIDASSYS, or from MIDAS_LIBRARY_DIRS
- set MIDAS_INCLUDE_DIRS, MIDAS_LIBRARY_DIRS and MIDAS_LIBRARIES to your target
- add sources from Midas for mfe, drivers, etc.

in general cmake already can to all of this automatically, and the only lines you would need are:
- do `find_package(Midas ... PATHS ~/midas_install_location)`
- and do `target_link_libraries(... midas::mfe)`
  (and all include dirs, libs, and deps are propagated automatically)

see PR https://bitbucket.org/tmidas/midas/pull-requests/48
- nothing should break with current setups
- if you want to try new `midas::` targets, try to link e.g. `midas::mfed` to your frontend

After some iterations, we merged the branch with the new build scheme. Now you can compile any midas program as described at

  https://bitbucket.org/tmidas/midas/pull-requests/48?link_source=email

A default CMakeLists.txt file can look like this:

cmake_minimum_required(VERSION 3.17)
project(example)
find_package(Midas REQUIRED PATHS $ENV{MIDASSYS})
add_executable(example example.cpp)
target_link_libraries(example midas::midas)


Which is much simpler than what we had before. The trick now is that the find_package() retrieves all include and link files automatically. 
There are different targets:

midas::midas              - normal midas program
midas::midas-shared       - normal midas programs using the shared midas library
midas::mfe                - old style mfe.cxx frontend
midas::mfed               - newer style frontend using mfed.cxx
midas::mscb               - programs using MSCB system
midas::drivers            - slow control program using any of the standard midas drivers

We are not absolutely sure that all midas installations will work that way, so far we have tested it on RH8, MacOSX with cmake version 
3.29.5.

Comments and bug reports are welcome as usual.

Alex and Stefan

> After some iterations, we merged the branch with the new build scheme.

the commit to implement this change in the manalyzer was not pushed, for reasons unknown.

fixed, commit f2b4dc87ca4830f6bed8667d6a4ee4afd6d242a1

K.O.

> currently to link Midas to project one has to do several steps ...

this information is incorrect. please read https://daq00.triumf.ca/elog-midas/Midas/2258

a very simple way to use link MIDAS using midas-targets.cmake has been implemented a long time ago.

before proposing a new way of doing things, it would be nice to hear about shortcomings
of the existing stuff. A simple "Konstantin's way sucks" or "this is not the cmake way!"
would have been sufficient.

K.O.

> > currently to link Midas to project one has to do several steps ...
> 
> this information is incorrect. please read https://daq00.triumf.ca/elog-midas/Midas/2258
> 
> a very simple way to use link MIDAS using midas-targets.cmake has been implemented a long time ago.

I admit that i did not see your post about targets import
via `include($ENV{MIDASSYS}/lib/midas-targets.cmake)`
before implementing changes to cmake scripts.
But in this respect the way you propose to do it via `include` should still work.
Note however that `include(...)` way is very unusual as one have to know exactly
where `...-targets.cmake` is located and standard way in cmake is via `find_package`
(similar to how e.g. ROOT, Geant4, etc. are found and linked).

The things that changed (and are incompatible with what was before)
is the naming of targets (in `midas-targets.cmake` with `midas::` namespace,
which is standard practice in cmake to distinguish cmake targets from bare library names
(e.g. when you do `link_libraries(midas)` it may be interpreted as linking with `-lmidas`
or if target is defined it does machinery to link actual cmake target; the namespace way
makes it unambiguous).
Though i again admit that maybe the namespace change was a bit too much as it may
have broken previous users of `include($ENV{MIDASSYS}/lib/midas-targets.cmake)`

> 
> before proposing a new way of doing things, it would be nice to hear about shortcomings
> of the existing stuff.

- shortcomings of what was before is usage of non-standard `include(...)`
- one shortcoming i see for new implementation is usage `midas::` namespace
  (mentioned above) that may have broken some setups

> A simple "Konstantin's way sucks" or "this is not the cmake way!"
> would have been sufficient.

- `find_package` is standard and recommended way of finding packages
    - note that `include($ENV{MIDASSYS}/lib/midas-targets.cmake)` should still work
      (but with usage of `midas::midas` instead of simply `midas`)
    - in the end `find_package` works by locating and loading `MidasConfig.cmake`,
      and it now actually does `include("${CMAKE_CURRENT_LIST_DIR}/../../midas-targets.cmake")`,
      so in this respect `find_package` is the same as `include(...)`,
      but it also preserves old behavior of exporting cmake vars for includes/libs
      such that prev uses are unaffected,
      and does a bit more checking such that it can be used for both in- and out-of-tree builds
    
- in addition `find_package` allows to handle components,
  e.g. now it is possible to do
  `find_package(Midas COMPONENTS manalyzer)`
  instead of also doing `include($ENV{MIDASSYS}/lib/manalyzer-targets.cmake)`

> 
> K.O.

Alex

> > > currently to link Midas to project one has to do several steps ...
> > this information is incorrect. please read https://daq00.triumf.ca/elog-midas/Midas/2258
> 
> I admit that i did not see your post about targets import
> via `include($ENV{MIDASSYS}/lib/midas-targets.cmake)`
> before implementing changes to cmake scripts.
>
> But in this respect the way you propose to do it via `include` should still work.
>

I proposed nothing, you did the proposing. I spent many hours trying to understand cmake (mission 
impossible!) and many more hours to implement the previously existing package scheme based
on the cmake "EXPORT" function.

> Note however that `include(...)` way is very unusual as one have to know exactly
> where `...-targets.cmake` is located and standard way in cmake is via `find_package`
> (similar to how e.g. ROOT, Geant4, etc. are found and linked).

Very difficult to cut-and-paste "include($ENV{MIDASSYS}/lib/midas-targets.cmake)".

You cannot simplify-out $ENV{MIDASSYS} because computer cannot read your mind, which of the 10 copies
of midas you want to use from which user account on which day.

Argument about "very unusual" I would buy, I am not a cmake expert and I do not know which package
finding method is in favour today.

>
> Though i again admit that maybe the namespace change was a bit too much as it may
> have broken previous users of `include($ENV{MIDASSYS}/lib/midas-targets.cmake)`
> 

I believe it did break at least one experiment, after updating MIDAS to latest version,
the analyzer would not build.

Speaking of which, did you implement your new scheme for the manalyzer so that it works
in standalone mode (without MIDAS)?

If you did not, now we have two schemes, your new scheme just for MIDAS and my old scheme
for manalyzer *and* MIDAS. xkcd 927.

> 
> - shortcomings of what was before is usage of non-standard `include(...)`
>

You should have started by posting a message spelling it out: Konstantin implemented
a scheme that uses the cmake "export" function to find midas, mfe and manalyzer,
it is very nice and works ok, but it is non-standard/obsoleted/obscure/frowned-upon/
unpopular/I-do-not-like-it/I-did-not-invent-it, and I propose implementing a new scheme
based on find_package().

>
> - one shortcoming i see for new implementation is usage `midas::` namespace
>   (mentioned above) that may have broken some setups
> 

If you think that your changes will break other people code, you should explicitely
say this in a message to this forum and hopefully provide instruction on fixing it,
i.e. in your makefile, please replace "midas" with "midas::midas".

> 
> - `find_package` is standard and recommended way of finding packages
>

Do you have a reference for this? When I look at cmake documentation, I do not see
any specific recommendation on creating packages and finding them. I do see
other people's code for finding packages and often spend hours fighting
them because said methods are designed to work only on the developer's laptop.

P.S. Did anybody ask Ben to update the MidasWiki documentation with the new find_package() information?

K.O.

> > > > currently to link Midas to project one has to do several steps ...
> > > this information is incorrect. please read https://daq00.triumf.ca/elog-midas/Midas/2258
> > 
> > I admit that i did not see your post about targets import
> > via `include($ENV{MIDASSYS}/lib/midas-targets.cmake)`
> > before implementing changes to cmake scripts.
> >
> > But in this respect the way you propose to do it via `include` should still work.
> >
> 
> I proposed nothing, you did the proposing. I spent many hours trying to understand cmake (mission 
> impossible!) and many more hours to implement the previously existing package scheme based
> on the cmake "EXPORT" function.
I agree that cmake is difficult, especially when it comes to creating cmake scripts for library
that should work for other people (as opposed to just using other libraries).
But that is why we should try to follow recommended way of using it.

> 
> > Note however that `include(...)` way is very unusual as one have to know exactly
> > where `...-targets.cmake` is located and standard way in cmake is via `find_package`
> > (similar to how e.g. ROOT, Geant4, etc. are found and linked).
> 
> Very difficult to cut-and-paste "include($ENV{MIDASSYS}/lib/midas-targets.cmake)".
> 
> You cannot simplify-out $ENV{MIDASSYS} because computer cannot read your mind, which of the 10 copies
> of midas you want to use from which user account on which day.
One can use `find_package(Midas PATH $ENV{MIDASSYS})` to set specific location of Midas
(this is mentioned in https://bitbucket.org/tmidas/midas/pull-requests/48)
and without `PATH` argument the default system/user locations are searched.

> 
> Argument about "very unusual" I would buy, I am not a cmake expert and I do not know which package
> finding method is in favour today.
> 
> >
> > Though i again admit that maybe the namespace change was a bit too much as it may
> > have broken previous users of `include($ENV{MIDASSYS}/lib/midas-targets.cmake)`
> > 
> 
> I believe it did break at least one experiment, after updating MIDAS to latest version,
> the analyzer would not build.
This unfortunately was not easy to avoid in this case, as both midas and manalyzer depend on each other:
midas should compile when manalyzer is enabled (as submodule)
and manalyzer should compile with midas as separate lib.
So in this case i would expect both midas and manalyzer be updated at same time to their matching versions.

> 
> Speaking of which, did you implement your new scheme for the manalyzer so that it works
> in standalone mode (without MIDAS)?
The only change on manalyzer i did was to use `find_package(Midas ...)` and `midas::midas` target
(see https://bitbucket.org/tmidas/manalyzer/commits/b219a916).
If there is interest to use same scheme in manalyzer as in Midas i can implement it.

> 
> If you did not, now we have two schemes, your new scheme just for MIDAS and my old scheme
> for manalyzer *and* MIDAS. xkcd 927.
> 
> > 
> > - shortcomings of what was before is usage of non-standard `include(...)`
> >
> 
> You should have started by posting a message spelling it out: Konstantin implemented
> a scheme that uses the cmake "export" function to find midas, mfe and manalyzer,
> it is very nice and works ok, but it is non-standard/obsoleted/obscure/frowned-upon/
> unpopular/I-do-not-like-it/I-did-not-invent-it, and I propose implementing a new scheme
> based on find_package().
As i mentioned i did not see you original post about usage of `include`
and otherwise i may have referenced it and though more about compatibility issues.

> 
> >
> > - one shortcoming i see for new implementation is usage `midas::` namespace
> >   (mentioned above) that may have broken some setups
> > 
> 
> If you think that your changes will break other people code, you should explicitely
> say this in a message to this forum and hopefully provide instruction on fixing it,
> i.e. in your makefile, please replace "midas" with "midas::midas".
In the original message to this thread i posted reference to PR
(https://bitbucket.org/tmidas/midas/pull-requests/48)
where it shows how to use `find_package` with this change.
As i did not expect the direct use of `include()` form
and assumed that manual linking was used (via specifying include/lib paths/names)
some scenarios where code for people broke were missed (not taken into account) by me.

> 
> > 
> > - `find_package` is standard and recommended way of finding packages
> >
> 
> Do you have a reference for this? When I look at cmake documentation, I do not see
> any specific recommendation on creating packages and finding them. I do see
> other people's code for finding packages and often spend hours fighting
> them because said methods are designed to work only on the developer's laptop.
see https://cmake.org/cmake/help/v3.27/guide/importing-exporting/index.html:
- about use of `find_package` see https://cmake.org/cmake/help/latest/guide/using-dependencies/index.html#guide:Using%20Dependencies%20Guide
- about double colon namespace for target see https://cmake.org/cmake/help/v3.27/guide/importing-exporting/index.html
  where it is mentioned "This convention of double-colons gives CMake a hint that the name is an IMPORTED target when it is used by downstream projects".

> 
> P.S. Did anybody ask Ben to update the MidasWiki documentation with the new find_package() information?
> 
> K.O.

> https://cmake.org/cmake/help/latest/guide/using-dependencies/index.html#guide:Using%20Dependencies%20Guide

thank you for providing a link to latest cmake find_package() guide.

I notice that this documentation was added in cmake 3.24 released circa Nov 2022
and does not exist in older versions. (It is easy to see in git history the last
time I touched any cmake stuff in midas).

I still see no documentation on "this is how you write a package that other people can import
using find_package()". Only see documentation on how to use find_package() on packages
that somebody who somehow knows how to do it already wrote.

> In the original message to this thread i posted reference to PR
> (https://bitbucket.org/tmidas/midas/pull-requests/48)

this pull request was rail-roaded through during the holidays without any
discussion on this forum. I was not given an opportunity to comment to it,
it was pushed and merged faster than I could blink.

bottom line. I voted against using cmake and was over-ruled. To me this cmake stuff
is only a source of wasted time and created bad feelings.

if midas is required to use cmake, we should have somebody on the team that at least
understands it and if not love it, at least does not hate it.

K.O.

I figured out the breakage, added a git tag to identify where the cmake incompatible change was made (roughly) 
and posted a note on how to fix it. Please reimburse me for the 2 hours I had to spend on this instead of doing 
useful work. K.O.

I reworked the midas cmake files:
- install via CMAKE_INSTALL_PREFIX should work correctly now:
- installed are bin, lib and include - everything needed to build against the midas library
- if built without CMAKE_INSTALL_PREFIX, a special mode "MIDAS_NO_INSTALL_INCLUDE_FILES" is activated, and the include path 
contains all the subdirectories need for compilation
- -I$MIDASSYS/include and -L$MIDASSYS/lib -lmidas work in both cases
- to "use" midas, I recommend: include($ENV{MIDASSYS}/lib/midas-targets.cmake)
- config files generated for find_package(midas) now have correct information (a manually constructed subset of information 
automatically exported by cmake's install(export))
- people who want to use "find_package(midas)" will have to contribute documentation on how to use it (explain the magic used to 
find the "right midas" in /usr/local/midas or in /midas or in ~/packages/midas or in ~/pacjages/new-midas) and contribute an 
example superproject that shows how to use it and that can be run from the bitpucket automatic build. (features that are not part 
of the automatic build we cannot insure against breakage).

On my side, here is an example of using include($ENV{MIDASSYS}/lib/midas-targets.cmake). I posted this before, it is used in 
midas/examples/experiment and I will ask ben to include it into the midas wiki documentation.

Below is the complete cmake file for building the alpha-g event bnuilder and main control frontend. When presented like this, I 
have to agree that cmake does provide positive value to the user. (the jury is still out whether it balances out against the 
negative value in the extra work to "just support find_package(midas) already!").

#
# CMakeLists.txt for alpha-g frontends
#

cmake_minimum_required(VERSION 3.12)
project(agdaq_frontends)

include($ENV{MIDASSYS}/lib/midas-targets.cmake)

add_compile_options("-O2")
add_compile_options("-g")
#add_compile_options("-std=c++11")
add_compile_options(-Wall -Wformat=2 -Wno-format-nonliteral -Wno-strict-aliasing -Wuninitialized -Wno-unused-function)
add_compile_options("-DTMFE_REV0")
add_compile_options("-DOS_LINUX")

add_executable(feevb feevb.cxx TsSync.cxx)
target_link_libraries(feevb midas)

add_executable(fectrl fectrl.cxx GrifComm.cxx EsperComm.cxx JsonTo.cxx KOtcp.cxx $ENV{MIDASSYS}/src/tmfe_rev0.cxx)
target_link_libraries(fectrl midas)

#end

MIDAS git tag midas-2025-01-a introduced an incompatible change to "include midas-targets.cmake". Instead of "midas" one now has to 
say "midas::midas", as updated below. K.O.

> 
> #
> # CMakeLists.txt for alpha-g frontends
> #
> 
> cmake_minimum_required(VERSION 3.12)
> project(agdaq_frontends)
> 
> include($ENV{MIDASSYS}/lib/midas-targets.cmake)
> 
> add_compile_options("-O2")
> add_compile_options("-g")
> #add_compile_options("-std=c++11")
> add_compile_options(-Wall -Wformat=2 -Wno-format-nonliteral -Wno-strict-aliasing -Wuninitialized -Wno-unused-function)
> add_compile_options("-DTMFE_REV0")
> add_compile_options("-DOS_LINUX")
> 
> add_executable(feevb feevb.cxx TsSync.cxx)
> target_link_libraries(feevb midas::midas)
> 
> add_executable(fectrl fectrl.cxx GrifComm.cxx EsperComm.cxx JsonTo.cxx KOtcp.cxx $ENV{MIDASSYS}/src/tmfe_rev0.cxx)
> target_link_libraries(fectrl midas::midas)
> 
> #end

I need to look at histograms inside a ROOT file, but all the old ways for doing this no longer work. (in theory I can scp the ROOT file to 
the computer I am sitting in front of, but this assumes I have a working ROOT there. anyhow it is pointless to fight this, all modern 
packages are written to only work on the developer's laptop).

- root new TBrowser starts a web server, tries to open firefox (and fails)
- root --web=off new TBrowser using ssh X11 tunnel no longer works, ROOT X11 graphics refresh is broken
- macos root binary kit is built without X11 support, root --web=off does not work at all
- root7 recommended "rootssh" prints an error message (and fails)

What does work well is JSROOT which we use to look at manalyzer live histograms (through apache and mhttpd web proxies).

So I wrote mjsroot.exe. It opens a ROOT file and starts JSROOT to look at it (plus a bit of dancing around to make it actually work):

mjsroot.exe -R8082 root_output_files/output00371.root

To actually see the histograms:

a) if you sitting in front of the same computer, open http://localhost:8082
b) if you are somewhere else, start an ssh tunnel: ssh daq13 -L8082:localhost:8082, open http://localhost:8082
c) if daq13 is running mhttpd, setup http proxy:
set ODB /webserver/proxy/mjsroot to http://localhost:8082
open https://daq13.triumf.ca/proxy/mjsroot/
also
set ODB /alias/mjsroot to "/proxy/mjsroot/"
reload MIDAS status page, observe "mjsroot" in listed in the left-hand side, open it.

K.O.
 

Hi,
I have a manalyzer that uses a derived class of TMFeRpcHandlerInterface to communicate information to 
Midas during online running.  At the end of each run it saves out custom data in the 
TMFeRpcHandlerInterface::HandleEndRun override. This works really well.
However, when I run offline on a Midas output file the HandleEndRun method is never called and my data is 
never saved.  Is this intentional?  I understand that there is no point for the HandleBinaryRpc method offline, 
but the other methods (HandleEndRun, HandleBeginRun etc) could serve a purpose.  Or is it a conscious 
choice to ignore all of TMFeRpcHandlerInterface when offline?

Thanks,

Mark.

Hi,

Sorry, we have been slammed with a couple projects on the TRIUMF side in the past weeks and haven't found time for a 
response.  I am hopeful that we will be able to answer this question (and the cache size question) within the next 10 
days.  

Again apologies,
Thomas

> Hi,
> I have a manalyzer that uses a derived class of TMFeRpcHandlerInterface to communicate information to 
> Midas during online running.  At the end of each run it saves out custom data in the 
> TMFeRpcHandlerInterface::HandleEndRun override. This works really well.
> However, when I run offline on a Midas output file the HandleEndRun method is never called and my data is 
> never saved.  Is this intentional?  I understand that there is no point for the HandleBinaryRpc method offline, 
> but the other methods (HandleEndRun, HandleBeginRun etc) could serve a purpose.  Or is it a conscious 
> choice to ignore all of TMFeRpcHandlerInterface when offline?
> 
> Thanks,
> 
> Mark.

> I have a manalyzer that uses a derived class of TMFeRpcHandlerInterface to communicate information to 
> Midas during online running.  At the end of each run it saves out custom data in the 
> TMFeRpcHandlerInterface::HandleEndRun override. This works really well.
> However, when I run offline on a Midas output file the HandleEndRun method is never called and my data is 
> never saved.  Is this intentional?  I understand that there is no point for the HandleBinaryRpc method offline, 
> but the other methods (HandleEndRun, HandleBeginRun etc) could serve a purpose.  Or is it a conscious 
> choice to ignore all of TMFeRpcHandlerInterface when offline?

apologies for delayed response.

I saw the question, completely did not understand it, only now got around to figure out what is going on.

according to manalyzer/README.md, section "manalyzer module and object life time", BeginRun() and EndRun() is called 
always. Offline and online. What you see would be a bug that we do not see in our environment. I confirm this by 
running manalyzer in a demo mode: ./bin/manalyzer_test.exe  --demo -e10 -t

no, wait, you say you use HandleBeginRun() and HandleEndRun(). this is not right, they are not part of the manalyzer 
API and they indeed are only used when running online.

correct solution would be to use BeginRun() and EndRun() instead of HandleBeginRun() and HandleEndRun().

you could also save your data in the module destructor (although good programming recommendation is to use 
destructor only for unavoidable things, like freeing memory, etc).

K.O.

Hi,
The question was about the TMFeRpcHandlerInterface, not the TARunObject interface.  Derived classes of TARunObject do indeed work as expected in our 
environment.  We have worked around the issue by using an implementation of TARunObject as well as the (separate) implementation of 
TMFeRpcHandlerInterface.

Thanks,

Mark.

> > I have a manalyzer that uses a derived class of TMFeRpcHandlerInterface to communicate information to 
> > Midas during online running.  At the end of each run it saves out custom data in the 
> > TMFeRpcHandlerInterface::HandleEndRun override. This works really well.
> > However, when I run offline on a Midas output file the HandleEndRun method is never called and my data is 
> > never saved.  Is this intentional?  I understand that there is no point for the HandleBinaryRpc method offline, 
> > but the other methods (HandleEndRun, HandleBeginRun etc) could serve a purpose.  Or is it a conscious 
> > choice to ignore all of TMFeRpcHandlerInterface when offline?
> 
> apologies for delayed response.
> 
> I saw the question, completely did not understand it, only now got around to figure out what is going on.
> 
> according to manalyzer/README.md, section "manalyzer module and object life time", BeginRun() and EndRun() is called 
> always. Offline and online. What you see would be a bug that we do not see in our environment. I confirm this by 
> running manalyzer in a demo mode: ./bin/manalyzer_test.exe  --demo -e10 -t
> 
> no, wait, you say you use HandleBeginRun() and HandleEndRun(). this is not right, they are not part of the manalyzer 
> API and they indeed are only used when running online.
> 
> correct solution would be to use BeginRun() and EndRun() instead of HandleBeginRun() and HandleEndRun().
> 
> you could also save your data in the module destructor (although good programming recommendation is to use 
> destructor only for unavoidable things, like freeing memory, etc).
> 
> K.O.

> The question was about the TMFeRpcHandlerInterface, not the TARunObject interface.  Derived classes of TARunObject do indeed work as expected in our 
> environment.  We have worked around the issue by using an implementation of TARunObject as well as the (separate) implementation of 
> TMFeRpcHandlerInterface.

then I do not understand the question. TMFeRpcHandlerInterface stuff is only used when running online and connected to MIDAS. How does it come into the 
picture when you analyze a data file offline? ProcessMidasOnlineTmfe() does not run, the RpcHandler object is not constructed.

maybe if you point me to your source code, I can see what you are doing?

K.O.

This was exactly the question, should I expect it to run?  There's no point in the HandleBinaryRpc method offline, but there's an argument that the HandleBeginRun/HandleEndRun methods have a use.
I have the answer and we have a workaround, thanks.

> then I do not understand the question. TMFeRpcHandlerInterface stuff is only used when running online and connected to MIDAS. How does it come into the 
> picture when you analyze a data file offline? ProcessMidasOnlineTmfe() does not run, the RpcHandler object is not constructed.
> 
> maybe if you point me to your source code, I can see what you are doing?
> 
> K.O.

I do not understand what you are doing. If you are offline, there is no TMFE singleton instance,
there is nothing TMFeRpcHandlerInterface to attach to, there is nobody to call TMFeRpcHandlerInterface methods.

Maybe what you are asking for is a mode where you analyze data from a file, but you want your analysis code
to think that it is online and that it is analyzing live data. This requires creating a fake TMFE singleton, attaching
TMFeRpcHandlerInterface to this fake TMFE singleton and using ProcessMidasOnlineTmfe(), driven by all this fake stuff:
a fake OpenBuffer() that actually opens a file, a fake ReceiveEvent() that actually reads from a file, fake callbacks
for begin and end run, etc.

That's a lot of work, but for what purpose? What is it about the existing offline and online modes that you do not like
and how all this fake stuff will make it better for you?

P.S. This is a 3rd version of my reply. Wrote and deleted 2 version. I think I completely do not understand
what you are doing and you completely do not understand what I am saying. Communication is not happening.

P.P.S. Simplest if you show me your code (email, elog), I am quite good at reading code and divining what
people are trying to do. You do not have to show me any of your secret secret stuff.

K.O.

> This was exactly the question, should I expect it to run?  There's no point in the HandleBinaryRpc method offline, but there's an argument that the HandleBeginRun/HandleEndRun methods have a use.
> I have the answer and we have a workaround, thanks.
> 
> > then I do not understand the question. TMFeRpcHandlerInterface stuff is only used when running online and connected to MIDAS. How does it come into the 
> > picture when you analyze a data file offline? ProcessMidasOnlineTmfe() does not run, the RpcHandler object is not constructed.
> > 
> > maybe if you point me to your source code, I can see what you are doing?
> > 
> > K.O.

12:51:20.154 2025/02/19 [SC Frontend,ERROR] [mfe.cxx:620:register_equipment,ERROR] Write cache size mismatch for buffer "SYSTEM": equipment "Environment" asked for 0, while eqiupment "LED" asked for 10000000
12:51:20.154 2025/02/19 [SC Frontend,ERROR] [mfe.cxx:620:register_equipment,ERROR] Write cache size mismatch for buffer "SYSTEM": equipment "LED" asked for 10000000, while eqiupment "Xenon" asked for 0

   {"Trigger",               /* equipment name */
      {1, 0,                 /* event ID, trigger mask */
         "SYSTEM",           /* event buffer */
         EQ_POLLED,          /* equipment type */
         0,                  /* event source */
         "MIDAS",            /* format */
         TRUE,               /* enabled */
         RO_RUNNING |        /* read only when running */
         RO_ODB,             /* and update ODB */
         100,                /* poll for 100ms */
         0,                  /* stop run after this event limit */
         0,                  /* number of sub events */
         0,                  /* don't log history */
         "", "", "", "", "", 0, 0},
      read_trigger_event,    /* readout routine */
      10000000,              /* write cache size */
   },

fe_set_write_case(int size);

I think I added the cache size correctly:

  {"Trigger",               /* equipment name */
      {1, 0,                 /* event ID, trigger mask */
         "SYSTEM",           /* event buffer */
         EQ_POLLED,          /* equipment type */
         0,                  /* event source */
         "MIDAS",            /* format */
         TRUE,               /* enabled */
         RO_RUNNING |        /* read only when running */
         RO_ODB,             /* and update ODB */
         100,                /* poll for 100ms */
         0,                  /* stop run after this event limit */
         0,                  /* number of sub events */
         0,                  /* don't log history */
         "", "", "", "", "", // frontend_host, name, file_name, status, status_color
         0, // hidden
         0  // write_cache_size <<--------------------- set this to zero -----------
      },
   }

K.O.

the main purpose of the event buffer write cache is to prevent high contention for the 
event buffer shared memory semaphore in the pathological case of very high rate of very 
small events.

there is a computation for this, I have posted it here several times, please search for 
it.

in the nutshell, you want the semaphore locking rate to be around 10/sec, 100/sec 
maximum. coupled with smallest event size and maximum practical rate (1 MHz), this 
yields the cache size.

for slow control events generated at 1 Hz, the write cache is not needed, 
write_cache_size value 0 is the correct setting.

for "typical" physics events generated at 1 kHz, write cache size should be set to fit 
10 events (100 Hz semaphore locking rate) to 100 events (10 Hz semaphore locking rate).

unfortunately, one cannot have two cache sizes for an event buffer, so typical frontends 
that generate physics data at 1 kHz and scalers and counters at 1 Hz must have a non-
zero write cache size (or semaphore locking rate will be too high).

the other consideration, we do not want data to sit in the cache "too long", so the 
cache is flushed every 1 second or so.

all this cache stuff could be completely removed, deleted. result would be MIDAS that 
works ok for small data sizes and rates, but completely falls down at 10 Gige speeds and 
rates.

P.S. why is high semaphore locking rate bad? it turns out that UNIX and Linux semaphores 
are not "fair", they do not give equal share to all users, and (for example) an event 
buffer writer can "capture" the semaphore so the buffer reader (mlogger) will never get 
it, a pathologic situation (to help with this, there is also a "read cache"). Read this 
discussion: https://stackoverflow.com/questions/17825508/fairness-setting-in-semaphore-
class

K.O.

> the main purpose of the event buffer write cache

how to control the write cache size:

1) in a frontend, all equipments should ask for the same write cache size, both mfe.c and 
tmfe frontends will complain about mismatch

2) tmfe c++ frontend, per tmfe.md, set fEqConfWriteCacheSize in the equipment constructor, in 
EqPreInitHandler() or EqInitHandler(), or set it in ODB. default value is 10 Mbytes or value 
of MIN_WRITE_CACHE_SIZE define. periodic cache flush period is 0.5 sec in 
fFeFlushWriteCachePeriodSec.

3) mfe.cxx frontend, set it in the equipment definition (number after "hidden"), set it in 
ODB, or change equipment[i].write_cache_size. Value 0 sets the cache size to 
MIN_WRITE_CACHE_SIZE, 10 Mbytes.

4) in bm_set_cache_size(), acceptable values are 0 (disable the cache), MIN_WRITE_CACHE_SIZE 
(10 Mbytes) or anything bigger. Attempt to set the cache smaller than 10 Mbytes will set it 
to 10 Mbytes and print an error message.

All this is kind of reasonable, as only two settings of write cache size are useful: 0 to 
disable it, and 10 Mbytes to limit semaphore locking rate to reasonable value for all event 
rate and size values practical on current computers.

In mfe.cxx it looks to be impossible to set the write cache size to 0 (disable it), but 
actually all you need is call "bm_set_cache_size(equipment[0].buffer_handle, 0, 0);" in 
frontend_init() (or is it in begin_of_run()?).

K.O.

> > the main purpose of the event buffer write cache
> how to control the write cache size:

OP provided insufficient information to say what went wrong for them, but do try this:

1) in ODB, for all equipments, set write_cache_size to 0
2) in the frontend equipment table, set write_cache_size to 0

That is how it is done in the example frontend: examples/experiment/frontend.cxx

If this configuration still produces an error, we may have a bug somewhere, so please let us know how it shakes out.

K.O.

 > All this is kind of reasonable, as only two settings of write cache size are useful: 0 to 
> disable it, and 10 Mbytes to limit semaphore locking rate to reasonable value for all event 
> rate and size values practical on current computers.

Indeed KO is correct that only 0 and 10MB make sense, and we cannot mix it. Having the cache setting in the equipment table is 
cumbersome. If you have 10 slow control equipment (cache size zero), you need to add many zeros at the end of 10 equipment 
definitions in the frontend. 

I would rather implement a function or variable similar to fEqConfWriteCacheSize in the tmfe framework also in the mfe.cxx 
framework, then we need only to add one line llike

gEqConfWriteCacheSize = 0;

in the frontend.cxx file and this will be used for all equipments of that frontend. If nobody complains, I will do that in April when I'm 
back from Japan.

Stefan

>  > All this is kind of reasonable, as only two settings of write cache size are useful: 0 to 
> > disable it, and 10 Mbytes to limit semaphore locking rate to reasonable value for all event 
> > rate and size values practical on current computers.
> 
> Indeed KO is correct that only 0 and 10MB make sense, and we cannot mix it. Having the cache setting in the equipment table is 
> cumbersome. If you have 10 slow control equipment (cache size zero), you need to add many zeros at the end of 10 equipment 
> definitions in the frontend. 
> 
> I would rather implement a function or variable similar to fEqConfWriteCacheSize in the tmfe framework also in the mfe.cxx 
> framework, then we need only to add one line llike
> 
> gEqConfWriteCacheSize = 0;
> 
> in the frontend.cxx file and this will be used for all equipments of that frontend. If nobody complains, I will do that in April when I'm 
> back from Japan.

Cache size is per-buffer. If different equipments write into different event buffers, should be possible to set different cache sizes.

Perhaps have:

set_write_cache_size("SYSTEM", 0);
set_write_cache_size("BUF1", bigsize);

with an internal std::map<std::string,size_t>; for write cache size for each named buffer

K.O.

I am getting a scary compiler warning from MSCB code. I generally avoid touching MSCB code because I have no MSCB hardware to test it, so I am 
asking the persons unnamed who wrote this code to fix it. Stock g++ on Ubuntu LTS 24.04. Thanks in advance!

In function ‘ssize_t read(int, void*, size_t)’,
    inlined from ‘int mscb_interprete_file(const char*, unsigned char**, unsigned int*, unsigned char**, unsigned int*, unsigned char*)’ at 
/home/olchansk/git/midas/mscb/src/mscb.cxx:2666:13:
/usr/include/x86_64-linux-gnu/bits/unistd.h:28:10: warning: ‘ssize_t __read_alias(int, void*, size_t)’ specified size 18446744073709551614 
exceeds maximum object size 9223372036854775807 [-Wstringop-overflow=]
   28 |   return __glibc_fortify (read, __nbytes, sizeof (char),
      |          ^~~~~~~~~~~~~~~
/usr/include/x86_64-linux-gnu/bits/unistd-decl.h: In function ‘int mscb_interprete_file(const char*, unsigned char**, unsigned int*, unsigned 
char**, unsigned int*, unsigned char*)’:
/usr/include/x86_64-linux-gnu/bits/unistd-decl.h:29:16: note: in a call to function ‘ssize_t __read_alias(int, void*, size_t)’ declared with 
attribute ‘access (write_only, 2, 3)’
   29 | extern ssize_t __REDIRECT_FORTIFY (__read_alias, (int __fd, void *__buf,
      |                ^~~~~~~~~~~~~~~~~~

K.O.

I hopefully fixed the waring (narrowing down from size_t to int). Please double check with your compiler.

Stefan

> I hopefully fixed the waring (narrowing down from size_t to int). Please double check with your compiler.

Nope, it turns out complain is about the read() size argument, they really really really want it to be 
size_t.

Looks like correct sequence is:

size_t size = file_size_somehow();
char* buf = (char*)malloc(size+1);
ssize_t rd = read(fd, buf, size);
if (rd < 0) { complain(); return; } // read() error
if ((size_t)rd != size) { complain(); return; } // cast to size_t is safe after we know rd >= 0
buf[rd] = 0; // make sure data is NUL terminated

What a mess. I want my UNIX V7 and K&R C back!

Perhaps I should add in system.cxx - ss_read_file(const char* filename, std::vector<char> &data);

Fixed it in mscb.cxx, odbedit.cxx, mhttpd.cxx and msequencer.cxx, commited, pushed.

U-24 builds without warnings. fwew...

K.O.

Unnamed person who added this clever bit of c++ coding, please fix this compiler warning. Stock g++ on Ubuntu LTS 24.04. Thanks in advance!

/home/olchansk/git/midas/src/system.cxx: In function ‘std::string ss_execs(const char*)’:
/home/olchansk/git/midas/src/system.cxx:2256:43: warning: ignoring attributes on template argument ‘int (*)(FILE*)’ [-Wignored-attributes]
 2256 |    std::unique_ptr<FILE, decltype(&pclose)> pipe(popen(cmd, "r"), pclose);

K.O.

> Unnamed person who added this clever bit of c++ coding, please fix this compiler warning. Stock g++ on Ubuntu LTS 24.04. Thanks in advance!
> 
> /home/olchansk/git/midas/src/system.cxx: In function ‘std::string ss_execs(const char*)’:
> /home/olchansk/git/midas/src/system.cxx:2256:43: warning: ignoring attributes on template argument ‘int (*)(FILE*)’ [-Wignored-attributes]
>  2256 |    std::unique_ptr<FILE, decltype(&pclose)> pipe(popen(cmd, "r"), pclose);
> 
> K.O.

Replace the code with:

   auto pclose_deleter = [](FILE* f) { pclose(f); };
   auto pipe = std::unique_ptr<FILE, decltype(pclose_deleter)>(
      popen(cmd, "r"),
      pclose_deleter
   );


Hope this is now warning-free.

Stefan

Confirming warning is fixed. Thanks! K.O.

> > Unnamed person who added this clever bit of c++ coding, please fix this compiler warning. Stock g++ on Ubuntu LTS 24.04. Thanks in advance!
> > 
> > /home/olchansk/git/midas/src/system.cxx: In function ‘std::string ss_execs(const char*)’:
> > /home/olchansk/git/midas/src/system.cxx:2256:43: warning: ignoring attributes on template argument ‘int (*)(FILE*)’ [-Wignored-attributes]
> >  2256 |    std::unique_ptr<FILE, decltype(&pclose)> pipe(popen(cmd, "r"), pclose);
> > 
> > K.O.
> 
> Replace the code with:
> 
>    auto pclose_deleter = [](FILE* f) { pclose(f); };
>    auto pipe = std::unique_ptr<FILE, decltype(pclose_deleter)>(
>       popen(cmd, "r"),
>       pclose_deleter
>    );
> 
> 
> Hope this is now warning-free.
> 
> Stefan

Andrea Capra reported a problem with manalyzer module initialization order.

Original manalyzer design relies on the fact than module static constructors run in the same order as they 
appear on the linker command line. This has been true for a very long time, but now we have evidence that on 
Rocky Linux (unknown gcc version), this is no longer true.

The c++ standard does not define any specific order for static constructor in different source files. 
(unlike C, C++ tends to treat the linker as something magical and ask the linker to do magical things).

The tmfe c++ modular frontend was designed after manalyzer and one design change is to explicitly construct 
all objects from main(). (at the acceptable cost of extra boiler plate code).

One solution is to use GCC attribute "init_priority (priority)", see
https://stackoverflow.com/questions/211237/static-variables-initialisation-order
https://gcc.gnu.org/onlinedocs/gcc/C_002b_002b-Attributes.html

To use this, manalyzer module registration should be modified to read:

static TARegister tar __attribute__((init_priority(200))) (new ExampleCxxFactory);

A less magical solution is to change manalyzer design similar to tmfe where modules are constructed and 
registered explicitely in the order specified by source code.

K.O.

P.S. I now ran out of time to test this and commit it to the documentation. It also need to be tested with 
LLVM C++ compilers.

> Andrea Capra reported a problem with manalyzer module initialization order.

Permanent solution is now implemented.

In the analyzer module constructor (TARunObject), please set the value of fModuleOrder (default value is 0).

Modules with smaller fModuleOrder will run first, modules with bigger fModuleOrder will run last.

Please use the value range between -1 (built-in EventDump module) and 9999 (built-in InteractiveModule).

Modules with equal fModuleOrder will run in the same order as they were registered (std::stable_sort).

Example manalyzer modules and documentation README.md have been updated.

Linker command line and GCC attribute methods should also work, but I thought it better to provide explicit 
programmatic control of module ordering.

P.S. the c++ tmfe frontend module design is newer and the problem of module (equipment) ordering is solved by 
constructing them explicitly in main().

manalyzer commit:

commit 6ca130808cd05ead734450391bf6defc8335c04a (HEAD -> master, origin/master, origin/HEAD)
Author: Konstantin Olchanski <olchansk@daq00.triumf.ca>
Date:   Tue Mar 25 15:53:13 2025 -0700

    implement TARunObject::fModuleOrder to specify required ordering of analyzer modules

K.O.

we are migrating the dragon experiment from an old mac to a new mac studio and we ran into a problem 
where one equipment format was set to "fixed" instead of "midas". lots of confusion, mdump crash, 
analyzer crash, etc. (mdump fixes for this are already committed).

it made us think whether equipment format is still needed. in the old days we had choice of MIDAS and 
YBOS formats, but YBOS was removed years ago, and I was surprised that format FIXED was permitted at 
all.

I did a midas source code review, this is what I found:

- remnants of YBOS support in a few places, commit to remove them pending.
- FORMAT_ROOT is used in mlogger for automatic conversion of MIDAS banks to ROOT trees
- FORMAT_FIXED is used in a few slow control drivers in drivers/class, instead of creating MIDAS 
banks, they copy raw data directly into an event (there is no bank header and no way to identify such 
events automatically)
- lots of code to support different formats in mdump (mostly dead code)
- the rest of the code does not care or use this format stuff

Current proposal is to remove support for all formats except FORMAT_MIDAS (and FORMAT_ROOT in 
mlogger).

- defines of FORMAT_XXX will be removed from midas.h
- "Format" will be removed from ODB Equipment/Common
- "Format" will be removed from ODB Logger/Channel
- to maintain binary compatibility, we can keep the "Format" ODB entries, but they will be ignored.

List of slow control drivers that support FORMAT_FIXED:

daq00:midas$ grep FORMAT_FIXED drivers/class/*
drivers/class/cd_fdg.cxx:   if (fgd_info->format == FORMAT_FIXED) {
drivers/class/cd_ivc32.cxx:   if (hv_info->format == XFORMAT_FIXED) {
drivers/class/cd_rf.cxx:	if (rf_info->format == XFORMAT_FIXED) 
drivers/class/generic.cxx:   if (gen_info->format == XFORMAT_FIXED) {
drivers/class/hv.cxx:   if (hv_info->format == XFORMAT_FIXED) {
drivers/class/multi.cxx:   if (m_info->format == XFORMAT_FIXED) {
drivers/class/slowdev.cxx:   if (gen_info->format == XFORMAT_FIXED) {
daq00:midas$ 

K.O.

Hi Konstantin,

In the PSI muSR laboratory, we are running about 140 slow control devices across six instruments using Format FIXED.

Could you please wait a bit with removing support for it so that we can assess if/how this will affect us?

Many thanks,
Jonas

> we are migrating the dragon experiment from an old mac to a new mac studio and we ran into a problem 
> where one equipment format was set to "fixed" instead of "midas". lots of confusion, mdump crash, 
> analyzer crash, etc. (mdump fixes for this are already committed).
> 
> it made us think whether equipment format is still needed. in the old days we had choice of MIDAS and 
> YBOS formats, but YBOS was removed years ago, and I was surprised that format FIXED was permitted at 
> all.
> 
> I did a midas source code review, this is what I found:
> 
> - remnants of YBOS support in a few places, commit to remove them pending.
> - FORMAT_ROOT is used in mlogger for automatic conversion of MIDAS banks to ROOT trees
> - FORMAT_FIXED is used in a few slow control drivers in drivers/class, instead of creating MIDAS 
> banks, they copy raw data directly into an event (there is no bank header and no way to identify such 
> events automatically)
> - lots of code to support different formats in mdump (mostly dead code)
> - the rest of the code does not care or use this format stuff
> 
> Current proposal is to remove support for all formats except FORMAT_MIDAS (and FORMAT_ROOT in 
> mlogger).
> 
> - defines of FORMAT_XXX will be removed from midas.h
> - "Format" will be removed from ODB Equipment/Common
> - "Format" will be removed from ODB Logger/Channel
> - to maintain binary compatibility, we can keep the "Format" ODB entries, but they will be ignored.
> 
> List of slow control drivers that support FORMAT_FIXED:
> 
> daq00:midas$ grep FORMAT_FIXED drivers/class/*
> drivers/class/cd_fdg.cxx:   if (fgd_info->format == FORMAT_FIXED) {
> drivers/class/cd_ivc32.cxx:   if (hv_info->format == XFORMAT_FIXED) {
> drivers/class/cd_rf.cxx:	if (rf_info->format == XFORMAT_FIXED) 
> drivers/class/generic.cxx:   if (gen_info->format == XFORMAT_FIXED) {
> drivers/class/hv.cxx:   if (hv_info->format == XFORMAT_FIXED) {
> drivers/class/multi.cxx:   if (m_info->format == XFORMAT_FIXED) {
> drivers/class/slowdev.cxx:   if (gen_info->format == XFORMAT_FIXED) {
> daq00:midas$ 
> 
> K.O.

> In the PSI muSR laboratory, we are running about 140 slow control devices across six instruments using Format FIXED.
> Could you please wait a bit with removing support for it so that we can assess if/how this will affect us?

Roger. I believe as long as these data do not go into the MIDAS event data stream, you should not see any difference from 
changing "FIXED" to "MIDAS", I hope you can test it and report how it works for you. If you see that things change in ODB 
or in history, perhaps we can implement some kind of workaround to make the transition transparent.

K.O.

Hello,

Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.

thanks,
Zaher

> Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.

Yes, in the ALPHA anti-hydrogen experiment at CERN we have been doing this since 2006.

Original system is very simple, labview side opens a TCP socket to the MIDAS felabview frontend
and sends the numeric data as an ASCII string. The first four chars of the data is the name
of the MIDAS data bank, second number is the data timestamp in seconds.

LCRY 1234567 1.1 2.2 3.3

A newer iteration is feGEM written by Joseph McKenna (member of this forum), it uses a more sophisticated
labview component. Please contact him directly for more information.

I can provide you with the source code for my original felabiew (pretty much unchanged from circa 2006).

K.O.

Thanks Konstantin. Please send me the felabview code or let me know where I can find it.

Zaher

> > Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.
> 
> Yes, in the ALPHA anti-hydrogen experiment at CERN we have been doing this since 2006.
> 
> Original system is very simple, labview side opens a TCP socket to the MIDAS felabview frontend
> and sends the numeric data as an ASCII string. The first four chars of the data is the name
> of the MIDAS data bank, second number is the data timestamp in seconds.
> 
> LCRY 1234567 1.1 2.2 3.3
> 
> A newer iteration is feGEM written by Joseph McKenna (member of this forum), it uses a more sophisticated
> labview component. Please contact him directly for more information.
> 
> I can provide you with the source code for my original felabiew (pretty much unchanged from circa 2006).
> 
> K.O.

> Thanks Konstantin. Please send me the felabview code or let me know where I can find it.

https://bitbucket.org/expalpha/a2daq/src/alpha/src/felabview.cxx

this is code circa 2006, there are now better ways to do some of that coding.

if you want bidirectional communication with labview, read/write odb, etc, simplest is probably
to write the "mjserver" that talks midas json-rpc over plain tcp, without all the http/https
gunk you need to go through mhttpd.

K.O.

> Hello,
> 
> Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.
> 
> thanks,
> Zaher

We do have a superconducting magnet from Cryogenic, UK, which comes with a LabView control program on a Windows PC. I did the only reasonable with this: trash it in the waste basket. Do NOT use Labveiw for anything which should run more than 24h in a row. Too many bad experiences with LabView control programs 
for separators at PSI and other devices. Instead of the Windows PC, we use MSCB devices and RasperryPis to communicate with the power supply directly, which has been proven to be much more stable (running for years without crashes). I'm happy to share our code with you.

Stefan

> > Hello,
> > 
> > Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.
> > 
> > thanks,
> > Zaher
> 
> We do have a superconducting magnet from Cryogenic, UK, which comes with a LabView control program on a Windows PC. I did the only reasonable with this: trash it in the waste basket. Do NOT use Labveiw for anything which should run more than 24h in a row. Too many bad experiences with LabView control programs 
> for separators at PSI and other devices. Instead of the Windows PC, we use MSCB devices and RasperryPis to communicate with the power supply directly, which has been proven to be much more stable (running for years without crashes). I'm happy to share our code with you.
>

Our parallel experience with the CERN ALPHA anti-hydrogen experiment: they have developed a whole labview empire
to control the cryogenics, the magnets, the positron source, the anti-proton trap, anti-hydrogen trap, etc.

At some point there was a wall of monitors in the counting room - each labview computer controlled one or two things -
so there is very many computers and each had to have a monitor (and mouse and keyboard).

All the data from this labview empire is logged to MIDAS history via felabview and feGEM, and they use
the MIDAS history to look and monitor almost everything. Control is done via Labview and Labview
based FPGA sequencers (National Instruments PXI hardware, $$$$$).

This works reasonably well to publish several papers in Nature.

But not 100%:

1) difficulties with labview source control (cannot be trivially managed by git, I guess)
2) unending fight against Microsoft and CERN IT trying to reboot the computers at the wrong time
3) more recently, forced Microsoft updates require trashing perfectly good machines and buy new ones

At TRIUMF there is very little Labview. All experiments use MIDAS and EPICS for most things.

Based on this experience, I agree with Stefan, today's sweet spot is RaspberryPi machines with USB attached
gizmos to control stuff. On the software side, drive the mess with MIDAS and custom web pages.

K.O.

Thanks Stefan, I would be very interested to see your code. At moment we have magnets and cryostats (3He and dilution) being delivered with Labview.



> > Hello,
> > 
> > Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.
> > 
> > thanks,
> > Zaher
> 
> We do have a superconducting magnet from Cryogenic, UK, which comes with a LabView control program on a Windows PC. I did the only reasonable with this: trash it in the waste basket. Do NOT use Labveiw for anything which should run more than 24h in a row. Too many bad experiences with LabView control programs 
> for separators at PSI and other devices. Instead of the Windows PC, we use MSCB devices and RasperryPis to communicate with the power supply directly, which has been proven to be much more stable (running for years without crashes). I'm happy to share our code with you.
> 
> Stefan

We had the case in MEG that some alarms were actually just warnings, not very severe. This happens for example if we calibrate our detector 
once every other day and modify the hardware which actually triggers the alarm for about an hour or so.

The problem with this is now that the alarm sounds every few minutes, and people get annoyed during that hour. They turn down the volume 
of their speakers, or even disable the alarm sound. If the detector gets back into the default mode again, they might forget to re-enable the 
alarm, which causes some risk. 

Turning down the volume is also not good, since during that hour we could have a "real" alarm on which people have to react quickly in order 
not destroy the detector.

The art is now to configure the alarm system in a way that "normal" changes do not annoy people or cover up really severe alarms. After long 
discussions we came to following conclusion: We need a special class of alarm (let's call it 'warning') which does not annoy people. The 
warning should be visible on the screen, but not ring the alarm bell. 

While we have different alarm classes in midas, which let us customize the frequency of alarms and the screen colors, all alarms or warnings 
ring the alarm sound right now. This can be changed in the browser under "Config/Alarm sound" but that switch affects ALL alarms, which is 
not what we want.

The idea we came up with was to add a flag "Alarm sound" to the alarm classes. For the 'warning' we can then turn off the alarm sound, so 
only the banner is shown on top of the screen, and the spoken message is generated every 15 mins to remind people, but not to annoy them.

I added this "Alarm sound" flag in the branch feature/alarm_sound so everybody can test it. The downside is that all "Alarm/Classs/xxx" need 
to be modified to add this flag. While the new code will add this flag automatically (with a default value of 'true'), the size of the alarm class 
record changes by four bytes (one bool). Therefore, all running midas programs will complain about the changed size, until they get 
recompiled. 

Therefore, to test the new feature, you have to checkout the branch and  re-compile all midas programs you use, otherwise you will get errors 
like 

  Fixing ODB "/Alarms/Classes/Alarm" struct size mismatch (expected 352, odb size 348)

I will keep the branch for a few days for people to try it out and report any issue, and later merge it to develop.

Stefan

Another modification has been done to the alarm system. 

We have often cases where an alarm is triggered on some readout noise. Like an analog voltage just over the alarm threshold for a very short period of time, triggered sometimes from environmental 
electromagnetic effects like turning on the light. 

To mitigate this problem, an "alarm trigger count" has been implemented. Every alarm has now a variable "Trigger count required". If this value is zero (the default), the alarm system works as before. If this 
value is nowever set to a non-zero value N, the alarm limit has to be met on N consecutive periods in order to trigger the alarm. Each alarm has a "Check interval" which determines the period of the alarm 
checking. If one has for example:

Check interval = 10
Trigger count required = 3

then the alarm condition has to be met for 3 consecutive periods of 10 seconds each, or for 30 seconds in total. 

The modification has been merged into the develop branch, and people have to be aware that the alarm structures in the ODB changed. The current code tries to fix this automatically, but it is important 
that ALL MIDAS CLIENTS GET RE-COMPILED after the new code is applied. Otherwise we could have "new" clients expecting the new ODB structure, and some "old" clients expecting the old structure, 
then both types of clients would fight against each other and change the ODB structure every few seconds, leading to tons of error messages. So if you pull the current develop branch, make sure to re-
compile ALL midas clients.

/Stefan

> ALL MIDAS CLIENTS GET RE-COMPILED after the new code is applied.

Usually we expect that it is "safe" to update to the latest version of MIDAS.

In the sense that we do not have to track down every single frontend
and rebuild it. We have several experiments where tracking down the source code
and rebuilding a frontend takes more than 5 seconds. In many cases these are
10 year old executables that worked just fine through many updates of MIDAS
without having to rebuild them.

So any binary incompatible change is best avoided and must be clearly announced.

The present binary-incompatible change is this commit:
https://bitbucket.org/tmidas/midas/commits/5899f1657ba31121c9f420a824b3c6c13b173988

I tagged the last commit before this change as: midas-2024-12-a

K.O.

> > ALL MIDAS CLIENTS GET RE-COMPILED after the new code is applied.
> 
> Usually we expect that it is "safe" to update to the latest version of MIDAS.
> 
> In the sense that we do not have to track down every single frontend
> and rebuild it. We have several experiments where tracking down the source code
> and rebuilding a frontend takes more than 5 seconds. In many cases these are
> 10 year old executables that worked just fine through many updates of MIDAS
> without having to rebuild them.
> 
> So any binary incompatible change is best avoided and must be clearly announced.
> 
> The present binary-incompatible change is this commit:
> https://bitbucket.org/tmidas/midas/commits/5899f1657ba31121c9f420a824b3c6c13b173988
> 
> I tagged the last commit before this change as: midas-2024-12-a

Here are my replies:

- this is not a binary incompatibility, but a incompatibility of the /Alarm record which got two more variables. Old 
frontends will complain during their structure check and remove the two variables, new frontend will then complain as 
well and add the two variables. This will go in circles, that why all frontends need to be recompiled if the new code is 
used

- I did clearly announce this change in the forum. Is there another location where I should communicate that?

- The extension is not there "just for fun" but needed by several experiments which experience spurious alarms 
triggered by some noisy signals. Requiring an value to be above a limit for a certain minimum time fixes many issues in 
many experiments here at PSI, this is why it has been implemented, even if it causes work for everybody with re-
compilation.

- If there are frontend programs which have not been re-compiled for ten years, I think it is very unlikely that these 
experiments need the latest cutting edge version of midas. The can safely stick to your tag midas-2024-12-a and not 
experience the issue of different /Alarm trees.

Stefan

> > > ALL MIDAS CLIENTS GET RE-COMPILED after the new code is applied.
>
> - I did clearly announce this change in the forum.
>

I missed the announcement and I was surprised to discover this change.

This is why I changed the title from "useful improvement" to "ACHTUNG!!!"

> - The extension is not there "just for fun" but needed by several experiments which experience spurious alarms 
> triggered by some noisy signals.

Agreed, a useful improvement.

To prevent this kind of trouble in the future, I think I should finish my crusade against db_get_record().

> - If there are frontend programs which have not been re-compiled for ten years, I think it is very unlikely that these 
> experiments need the latest cutting edge version of midas. The can safely stick to your tag midas-2024-12-a and not 
> experience the issue of different /Alarm trees.

Yes, now that I added the tag, posted a more visible notice and people who have to run frontends build against older midas 
(because reasons) can breathe out.

Also please note that "Konstantin != all experiments at TRIUMF".

K.O.

bitbucket automatic builds were broken after mfe.cxx started printing some additional messages added in commit
https://bitbucket.org/tmidas/midas/commits/0ae08cd3b96ebd8e4f57bfe00dd45527d82d7a38

this is now fixed. to check if your changes will break automatic builds, before final push, please do:

make clean
make mini -j
make cmake -j
make test

K.O.

> bitbucket automatic builds were broken after mfe.cxx started printing some additional messages added in commit
> https://bitbucket.org/tmidas/midas/commits/0ae08cd3b96ebd8e4f57bfe00dd45527d82d7a38
> 
> this is now fixed. to check if your changes will break automatic builds, before final push, please do:
> 
> make clean
> make mini -j
> make cmake -j
> make test

Unfortunately we will break the automatic build each time a program outputs one different character, which even might happen if we add a line of code and 
a cm_msg() gets produced with a different line number. Is there a standard way to update testexpt.example (like "make testexpt" or so). Should be trigger 
the update of testexpt.example before each commit via a hook? 

Stefan

> > bitbucket automatic builds
> 
> Unfortunately we will break the automatic build each time a program outputs one different character, which even might happen if we add a line of code and 
> a cm_msg() gets produced with a different line number. Is there a standard way to update testexpt.example (like "make testexpt" or so). Should be trigger 
> the update of testexpt.example before each commit via a hook? 
> 

Actually line numbers are not logged by messages printed from "make test", so moving code around does not break the test.

Changing what programs output does break the test and this is intentional - somebody must look at confirm
that program output was changed on purpose or because a bug was introduced (or fixed).

Most "make test" things work this way - run programs, compare output to what is expected. Discrepancies are flagged for human examination.

K.O.

to continue where we left off in 2019,
https://daq00.triumf.ca/elog-midas/Midas/1520

time to choose the next c++!

snapshot of 2019:

- Linux RHEL/SL/CentOS6 - gcc 4.4.7, no C++11.
- Linux RHEL/SL/CentOS7 - gcc 4.8.5, full C++11, no C++14, no C++17
- Ubuntu 18.04.2 LTS - gcc 7.3.0, full C++11, full C++14, "experimental" C++17.
- MacOS 10.13 - llvm 10.0.0 (clang-1000.11.45.5), full C++11, full C++14, full C++17

the world moved on:

- el6/SL6 is gone
- el7/CentOS-7 is out the door, only two experiments on my plate (EMMA and ALPHA-g)
- el8 was a still born child of RedHat
- el9 - gcc 11.5 with 12, 13, and 14 available.
- el10 - gcc 14.2

- U-18 - gcc  7.5
- U-20 - gcc  9.4 default, 10.5 available
- U-22 - gcc 11.4 default, 12.3 available
- U-24 - gcc 13.3 default, 14.2 available

- MacOS 15.2 - llvm/clang 16

Next we read C++ level support:

(see here for GCC C++ support: https://gcc.gnu.org/projects/cxx-status.html)
(see here for LLVM clang c++ support: https://clang.llvm.org/cxx_status.html)
(see here for GLIBC c++ support: https://gcc.gnu.org/onlinedocs/libstdc++/manual/status.html)

gcc:
4.4.7 - no C++11
4.8.5 - full C++11, no C++14, no C++17
7.3.0 - full C++11, full C++14, "experimental" C++17.
7.5.0 - c++17 and older
9.4.0 - c++17 and older
10.5 - no c++26, no c++23, mostly c++20, c++17 and older
11.4 - no c++26, no c++23, full c++20 and older
12.3 - no c++26, mostly c++23, full c++20 and older
13.3 - no c++26, mostly c++23, full c++20 and older
14.2 - limited c++26, mostly c++23, full c++20 and older

clang:
16 - no c++26, mostly c++23, mostly c++20, full c++17 and older

I think our preference is c++23, the number of useful improvements is quite big.

This choice will limit us to:
- el9 or older
- U-22 or older
- current MacOS 15.2 with Xcode 16.

It looks like gcc and llvm support for c++23 is only partial, so obviously we will use a subset of c++23 
supported by both.

Next step is to try to build midas with c++23 on el9 and U-22,24 and see what happens.

K.O.

> time to choose the next c++!

Ununtu-24.04, MIDAS builds with -std=c++23 without errors or any additional warnings. (but does it work? "make 
test" is ok).

K.O.

Some time ago Ben Smith added test coverage reports using GCC -fprofile-arcs -
ftest-coverage and lcov.

It reports code coverage after running "make test". Reported code coverage is 
surprisingly high for the very little code executed by "make test" - create ODB, 
start a frontend,  start run, stop run, check that mlogger created data files 
and history files.

(Of course coverage can never reach 100%, some obscure branches are unreachable 
without using an automated fuzzer, and some error paths are unreachable without 
also using a system call fault injector).

Simplest way to generate a coverage report:

make clean
make cmake YES_COVERAGE=1
make coverage
open cov_html/index.html

K.O.