Hi,

unfortunately I don't have a documentation link to the feature, I just know that it works on my machine ;-) The general idea is that you place a custom whatever.log file in Logger/Data Dir (where midas.log is stored). Then, in the Messages page, there will be a "midas" tab and a "whatever" tab - the latter showing the content of whatever.log. One problem here is that timestamping does not work automatically - you have to prepend every line with the same Hours:Minutes:Seconds.Milliseconds Year/Month/Day format that midas.log is using.

So you have a custom Programs page which does systemctl status on your systemd? Does the status then transfer over automatically to the Status page? Is there an example how to write such a custom page?

Best regards
Stefan

> Hi,
>  
> > Nick. Regarding the messages: Zaher showed me that it is possible to simply place
> > a custom log file generated by the systemd next to midas.log - then it shows up
> > next to the "midas" tab in "Messages".
> 
> Interesting. I'm not familiar with that feature. Do you have link to documentation?
> 
> > One follow-up question: Is it possible to use the systemctl status for the
> > "Running on host" column? Or does this even happen automatically?
> 
> On the programs page that column is populated by the odb key /System/Clients/<PID>/Host
> so no. However, there is nothing stopping you from writing your own version of
> programs.html to show whatever you want. For example I have a custom programs
> page the includes columns to enable/disable and to reset watchdog alarms.
> 
> Cheers,
> 
> Nick.

> I notice the cmake does not actually pass "-std=c++17" to the c++ compiler, and on U-20, it is likely 
> the default c++14 is used. cmake always does the wrong thing and this will need to be fixed later.

Does adding

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

Get it to do the right thing?

Cheers,

Nick..

Hi,
 
> Nick. Regarding the messages: Zaher showed me that it is possible to simply place
> a custom log file generated by the systemd next to midas.log - then it shows up
> next to the "midas" tab in "Messages".

Interesting. I'm not familiar with that feature. Do you have link to documentation?

> One follow-up question: Is it possible to use the systemctl status for the
> "Running on host" column? Or does this even happen automatically?

On the programs page that column is populated by the odb key /System/Clients/<PID>/Host
so no. However, there is nothing stopping you from writing your own version of
programs.html to show whatever you want. For example I have a custom programs
page the includes columns to enable/disable and to reset watchdog alarms.

Cheers,

Nick.

> > we should finally bite the bullet and remove ROOT support from MIDAS ...

as discussed, HAVE_ROOT and OBSOLETE were removed from mlogger. rmana and ROOT support in manalyzed remain, 
untouched.

last rmlogger is in MIDAS tagged midas-2025-11-a.

K.O.

> > Following discussions at the MIDAS workshop, we propose to move MIDAS from c++11 to c++17.
> We shall move forward with this change.

It is done. Last c++11 MIDAS is midas-2025-11-a (plus the db_delete_key merge).

I notice the cmake does not actually pass "-std=c++17" to the c++ compiler, and on U-20, it is likely 
the default c++14 is used. cmake always does the wrong thing and this will need to be fixed later.

K.O.

In the darkside vertical slice midas daq, we observed odb corruption which I 
traced to db_delete_key(). cause of corruption is not important. important is to 
have a robust odb where small corruption will stay localized and will not 
require erasing corrupt odb and reloading it from a backup file.

To help debug such corruption one can try to set ODB "/Experiment/Protect ODB" 
to "yes". This will make ODB shared memory read-only and user code scribbling 
into the wrong memory address will cause a seg fault and core dump instead of 
silent ODB corruption. This feature is not enabled by default because changing 
ODB shared memory mapping from "read-only" to "writable" (and back) is not very 
fast and it slows down MIDAS noticably.

MIDAS right before this merge was tagged "midas-2025-11-a", if you see this ODB 
update cause trouble, please report it here and revert to this tagged version.

Updates:
- harden db_delete_key() against internal corruption, if odb inconsistency is 
detected, do a clean crash instead of trying to delete stuff and corrupting odb 
to the point where it has to be erased and reloaded from a backup file.
- additional refactoring to separate read-locked and write-locked code.
- merge of missing patch to avoid odb corruption when key area becomes 100% full 
(or was it the data area? I forget now, I fixed one of them long time ago, now 
both are fixed).
- remove the "follow_links" argument from db_delete_key(), see separate 
discussion on this.
- add db_delete() to delete things by ODB path not by hkey (atomic fused 
together db_find_link() and db_delete_key()).
- fixes for incorrect use of db_find_key() and db_delete_key(), this 
unexpectedly follows symlinks and deletes the wrong ODB entry. (should have been 
db_find_link(), now replaced with atomic db_delete()).

K.O.

> > I would handle this actually like symbolic links are handled under linux. If you delete a symbolic link, the link gets 
> > detected and NOT the file the link is pointing to.
> > 
> > So I conclude that the "follow links" is a misconception and should be removed.
> 
> I am finally writing test code for the ODB API, partially to discover what db_delete_key(TRUE) actually does,
> because when I went ahead to remove it, I found that it's use is inconsistent. And indeed
> it does strange and unexpected things, I will proceed with removing it.
> 

this is now merged into develop. there will be deprecation warnings from mvodb and from midas_c_compat, I and Ben will clean 
them up, just not today.

for more detail, see separate message.

K.O.

Thanks a lot,

Nick. Regarding the messages: Zaher showed me that it is possible to simply place a custom log file generated by the systemd next to midas.log - then it shows up next to the "midas" tab in "Messages".

One follow-up question: Is it possible to use the systemctl status for the "Running on host" column? Or does this even happen automatically?

Best regards
Stefan

> Hi,
> 
> what you describe is exactly how I normally run mhttpd, mlogger, mserver and some other
> custom frontend programs. Eg:
> 
> [local:T2KGSC:Running]/>ls /programs/Logger/
> Required                        y
> Watchdog timeout                100000
> Check interval                  180000
> Start command                   systemctl --user start mlogger
> Auto start                      n
> Auto stop                       n
> Auto restart                    n
> Alarm class                     AlarmNotify
> First failed                    0
> 
> The only exception is your last point about stdout and stderr
> being midas messages. I use journalctl to see these.
> 
> Cheers,
> 
> Nick.
> 
> > I want to control (start / stop / monitor its stdout and stderr) an external process (systemd / EPICS IOC shell script) from within MIDAS.
> > 
> > In order to make this as convenient as possible for the user, I want the process to behave just like any other MIDAS client:
> > - I can start it from the ODB as a program
> > - The process gets regularly polled from MIDAS to see whether it is still running
> > - I can stop the process from the ODB like any other program
> > - Optional, but highly appreciated: Its stdout and stderr should be a MIDAS message.
> > 
> > Did anyone already solve a similar problem?
> > 
> > Best regards
> > Stefan

Hi,

what you describe is exactly how I normally run mhttpd, mlogger, mserver and some other
custom frontend programs. Eg:

[local:T2KGSC:Running]/>ls /programs/Logger/
Required                        y
Watchdog timeout                100000
Check interval                  180000
Start command                   systemctl --user start mlogger
Auto start                      n
Auto stop                       n
Auto restart                    n
Alarm class                     AlarmNotify
First failed                    0

The only exception is your last point about stdout and stderr
being midas messages. I use journalctl to see these.

Cheers,

Nick.

> I want to control (start / stop / monitor its stdout and stderr) an external process (systemd / EPICS IOC shell script) from within MIDAS.
> 
> In order to make this as convenient as possible for the user, I want the process to behave just like any other MIDAS client:
> - I can start it from the ODB as a program
> - The process gets regularly polled from MIDAS to see whether it is still running
> - I can stop the process from the ODB like any other program
> - Optional, but highly appreciated: Its stdout and stderr should be a MIDAS message.
> 
> Did anyone already solve a similar problem?
> 
> Best regards
> Stefan

Dear all,

I want to control (start / stop / monitor its stdout and stderr) an external process (systemd / EPICS IOC shell script) from within MIDAS.

In order to make this as convenient as possible for the user, I want the process to behave just like any other MIDAS client:
- I can start it from the ODB as a program
- The process gets regularly polled from MIDAS to see whether it is still running
- I can stop the process from the ODB like any other program
- Optional, but highly appreciated: Its stdout and stderr should be a MIDAS message.

Did anyone already solve a similar problem?

Best regards
Stefan

I'm constructing a TMFeEquipment with this constructor:

MagnetFe(const char *eqname, const char *eqfilename) // ctor
        : TMFeEquipment(eqname, eqfilename)
    {
        fEqConfEventID = 3;
        fEqConfBuffer = "SYSTEM";
        fEqConfPeriodMilliSec = 1000; // in milliseconds
        fEqConfLogHistory = 1;
        fEqConfReadOn = RO_ALWAYS;
    }


When I start with a fresh ODB, the directories are created correctly, and e.g. the 
event ID is set correctly, but "Read on" is set to 1 (i.e. RO_RUNNING) instead of 
0xFF.
Now when I set it to 0xFF manually and restart, it gets overwritten to 7 
(RO_NONTRANS), which I guess is a relatively recent change and doesn't affect me 
negatively.

> Sorry about that. I could not figure out what was the reason for doing this. This was during the time I was working on the file_picker. I removed these lines and see no effect on the file_picker. I'll continue checking it affect anything else.

I confirm reported problem seems to be fixed in commit:
https://bitbucket.org/tmidas/midas/commits/7f2690b478d6dfb16b48fc98955093e6369b04c1

Big thanks to Stefan and Zaher for figuring it out quickly.

K.O.

Sorry about that. I could not figure out what was the reason for doing this. This was during the time I was working on the file_picker. I removed these lines and see no effect on the file_picker. I'll continue checking it affect anything else.

Zaher

> This problem was introduced by ZS in March 2023 with these commits:
> 
> https://bitbucket.org/tmidas/midas/commits/25b13f875ff1f7e2f4e987273c81d6356dd2ff53
> https://bitbucket.org/tmidas/midas/commits/2a9e902e07156e12edecb5c2257e4dbd944f8377
> 
> by setting
> 
>  d.style.position = "fixed";
> 
> which prevents the scrolling. I have no idea why this change was made, so it should be fixed by the original 
> author.
> 
> Stefan

This problem was introduced by ZS in March 2023 with these commits:

https://bitbucket.org/tmidas/midas/commits/25b13f875ff1f7e2f4e987273c81d6356dd2ff53
https://bitbucket.org/tmidas/midas/commits/2a9e902e07156e12edecb5c2257e4dbd944f8377

by setting

 d.style.position = "fixed";

which prevents the scrolling. I have no idea why this change was made, so it should be fixed by the original 
author.

Stefan

> > We would like to write an sc_frontend in Python instead of C++. All our drivers 
> > work correctly, as well as the creation of the database in the ODB, including the 
> > creation of Commons, Statistics, Variables, and Settings.
> > However, we are unable to correctly create the database entries needed to manage 
> > History and Display.
> > 
> > As we understand it, in C++ this is handled by setting the EQ_SLOW flag, which 
> > doesn’t seem to be implemented in the Python libraries.
> > How can we manually create the necessary variables for Display and History?

I don't believe any of this is handled automatically by the EQ_SLOW flag in the C++ code. I think you always have to manually create the history plots, normally using the webpage interface.

There is also a function in the python code called "client.hist_create_plot(group_name, panel_name, variables, labels=[])" that can slightly automate this, though you do have to know what midas is internally calling your variables. 

You can find out what the variables are called either through the webpage interface when creating a plot, or via the python script at $MIDASSYS/python/examples/basic_hist_script.py

> We would like to write an sc_frontend in Python instead of C++. All our drivers 
> work correctly, as well as the creation of the database in the ODB, including the 
> creation of Commons, Statistics, Variables, and Settings.
> However, we are unable to correctly create the database entries needed to manage 
> History and Display.
> 
> As we understand it, in C++ this is handled by setting the EQ_SLOW flag, which 
> doesn’t seem to be implemented in the Python libraries.
> How can we manually create the necessary variables for Display and History?

I'm not an expert of the Python part of MIDAS (Ben Smith is), but I know that you have
functions to create keys and set values in the ODB, so you should be able to create
these things manually as you need them.

Stefan

I confirm the problem is there (at least under MacOSX Safari) and I will take care of it.

Stefan

Hi all,

Could you please get back to me about whether something like my earlier suggestion might be considered, or if I should set up some workaround to rename files at EOR for our experiments?  

https://daq00.triumf.ca/elog-midas/Midas/3042 :
-----------------------------------------------
> Hi all,
> 
> Would it be possible to extend manalyzer to support custom .root file names that include the run number? 
> 
> As far as I understand, the current behavior is as follows:
> The default filename is ./root_output_files/output%05d.root , which can be customized by the following two command line arguments.
> 
> -Doutputdirectory: Specify output root file directory
> -Ooutputfile.root: Specify output root file filename
> 
> If an output file name is specified with -O, -D is ignored, so the full path should be provided to -O. 
> 
> I am aiming to write files where the filename contains sufficient information to be unique (e.g., experiment, year, and run number). However, if I specify it with -O, this would require restarting manalyzer after every run; a scenario that I would like to avoid if possible.
> 
> Please find a suggestion of how manalyzer could be extended to introduce this functionality through an additional command line argument at
> https://bitbucket.org/krieger_j/manalyzer/commits/24f25bc8fe3f066ac1dc576349eabf04d174deec
> 
> Above code would allow the following call syntax: ' ./manalyzer.exe -O/data/experiment1_%06d.root --OutputNumbered '
> But note that as is, it would fail if a user specifies an incompatible format such as -Ooutput%s.root . 
> 
> So a safer, but less flexible option might be to instead have the user provide only a prefix, and then attach %05d.root in the code.
> 
> Thank you for considering these suggestions!

> I would handle this actually like symbolic links are handled under linux. If you delete a symbolic link, the link gets 
> detected and NOT the file the link is pointing to.
> 
> So I conclude that the "follow links" is a misconception and should be removed.

I am finally writing test code for the ODB API, partially to discover what db_delete_key(TRUE) actually does,
because when I went ahead to remove it, I found that it's use is inconsistent. And indeed
it does strange and unexpected things, I will proceed with removing it.

K.O.

I added Makefile rules for running MIDAS through the clang-tidy static code 
analyzer. (rules for running cppcheck have gone missing, I hope I find them).

Reports from clang-tidy are somewhat repetitive (complain about the same class of 
non-problem problems repeatedly), but several warnings and errors identified real 
bugs.

Specifically, detection of memory leaks in error paths and detection of NULL 
pointer dereference and use of uninitialized variables is pretty solid.

Fixes for the worst problems reported by cppcheck and clang-tidy are already 
committed to midas git.

K.O.