> > I would like to know how to retrieve the online data during the experiment so 
> > that I can create my own custom plot. I execute my own frontend.exe to start the 
> > experiment. I can get a midas file after the experiment, but I am not sure about 
> > how to retrieve the online data. I know that rootana can help us to get the 
> > online plots, but the instructions in rootana is not clear. Can anyone give me 
> > some suggestion?  Thank you.
> 
> The current package for analyzing MIDAS data is the "m" analyzer, usually in the manalyzer subdirectory of your midas package, 
> but it can also be used stand-alone without MIDAS.
> 
> There is several examples:
> 
> manalyzer_example_cxx.cxx - a simple "c++" example shows how to extra midas bank data
> manalyzer_example_root.cxx - how to create ROOT histograms (that you can see online using jsroot)
> manalyzer_example_root_graphics.cxx - how to create a ROOT graphical program (obsoleted by jsroot, but still possible)
> manalyzer_example_flow*.cxx - more advanced examples on using a flow analyzer
> 
> Documentation is in README.md
> 
> Unfortunately there is no tutorial or 5 min youtube explainer, each experiment needs are very different, there is no way to 
> write a one-size-fits-all recipe.
> 
> Please take a look at the existing examples first, then send me a PM with any additional questions (or ask here). If you can 
> explain what kind of data you have and how you want to look at it, I should be able to guide you through writing an appropriate 
> manalyzer module.
> 
> K.O.

Dear K.O.
I would like to create some online plots (also reading the mid) when the frontend code is running. I may try to break the task into small steps first.
In the frontend.cxx, I have a bank called "TDC0" which includes an array of integer. The code is as follow:

bk_create(pevent, "TDC0", TID_INT, (void**) &pdata);
for(int i =0; i<ch; ++i)
*pdata++ = data[i];
bk_close(pevent,pdata);

In the meantime, I want to run manalyzer to get some plots.

So I started our DAQ with an updated midas, after ca. 6 months+. 

No issues except all FEs complaining about the Alarm ODB structure.
* I adapted to the new structure ( trigger count & trigger count required )
* restarted fe's
* recompiled  

18:17:40.015 2025/09/30 [EPICS Frontend,INFO] Fixing ODB "/Alarms/Alarms/logger" 
struct size mismatch (expected 452, odb size 460)

18:17:40.009 2025/09/30 [SC Frontend,INFO] Fixing ODB "/Alarms/Alarms/logger" 
struct size mismatch (expected 460, odb size 452)

how do I get the FEs + ODB back in line here?

thanks

> So I started our DAQ with an updated midas, after ca. 6 months+.

Would be worthwhile mentioning the git commit hash or tag you are using.

> No issues except all FEs complaining about the Alarm ODB structure.
> * I adapted to the new structure ( trigger count & trigger count required )
> * restarted fe's
> * recompiled  
> 
> 18:17:40.015 2025/09/30 [EPICS Frontend,INFO] Fixing ODB "/Alarms/Alarms/logger" 
> struct size mismatch (expected 452, odb size 460)
> 
> 18:17:40.009 2025/09/30 [SC Frontend,INFO] Fixing ODB "/Alarms/Alarms/logger" 
> struct size mismatch (expected 460, odb size 452)

This seems to be https://daq00.triumf.ca/elog-midas/Midas/2980

> how do I get the FEs + ODB back in line here?

Recompile all frontends against new midas.

Nick.

Just to be clear, it seems that your "EPICS Frontend" was either not recompiled against the new midas yet or the old binary is being run, but "SC Frontend" is using the new midas.

> > So I started our DAQ with an updated midas, after ca. 6 months+.
> 
> Would be worthwhile mentioning the git commit hash or tag you are using.
> 
> > No issues except all FEs complaining about the Alarm ODB structure.
> > * I adapted to the new structure ( trigger count & trigger count required )
> > * restarted fe's
> > * recompiled
> > 
> > 18:17:40.015 2025/09/30 [EPICS Frontend,INFO] Fixing ODB "/Alarms/Alarms/logger" 
> > struct size mismatch (expected 452, odb size 460)
> > 
> > 18:17:40.009 2025/09/30 [SC Frontend,INFO] Fixing ODB "/Alarms/Alarms/logger" 
> > struct size mismatch (expected 460, odb size 452)
> 
> This seems to be https://daq00.triumf.ca/elog-midas/Midas/2980
> 
> > how do I get the FEs + ODB back in line here?
> 
> Recompile all frontends against new midas.
> 
> Nick.

Sorry to intervene there, but the FEs are usually compiled against libmidas.a . Therefore you have to compile midas, usually do a "make install" to update the libmidas.a/so, then recompile the FEs. You probably forgot the "make install".

Stefan

> Sorry to intervene there, but the FEs are usually compiled against libmidas.a . Therefore you have to compile midas, usually do a "make install" to update the libmidas.a/so, then recompile the FEs. You probably forgot the "make install".
> 
> Stefan

OK, solved, closed. Turned out I messed up rebuilding some of the FEs.

More generally, and with the "Documentation" discussion of the workshop in mind, ODB mismatch error messages of all kind are a recurring phenomena confusing users. And MIDASGPT gave complete wrong suggestions.  

Dear all,

I want to control (start / stop / monitor its stdout and stderr) an external process (systemd / EPICS IOC shell script) from within MIDAS.

In order to make this as convenient as possible for the user, I want the process to behave just like any other MIDAS client:
- I can start it from the ODB as a program
- The process gets regularly polled from MIDAS to see whether it is still running
- I can stop the process from the ODB like any other program
- Optional, but highly appreciated: Its stdout and stderr should be a MIDAS message.

Did anyone already solve a similar problem?

Best regards
Stefan

Hi,

what you describe is exactly how I normally run mhttpd, mlogger, mserver and some other
custom frontend programs. Eg:

[local:T2KGSC:Running]/>ls /programs/Logger/
Required                        y
Watchdog timeout                100000
Check interval                  180000
Start command                   systemctl --user start mlogger
Auto start                      n
Auto stop                       n
Auto restart                    n
Alarm class                     AlarmNotify
First failed                    0

The only exception is your last point about stdout and stderr
being midas messages. I use journalctl to see these.

Cheers,

Nick.

> I want to control (start / stop / monitor its stdout and stderr) an external process (systemd / EPICS IOC shell script) from within MIDAS.
> 
> In order to make this as convenient as possible for the user, I want the process to behave just like any other MIDAS client:
> - I can start it from the ODB as a program
> - The process gets regularly polled from MIDAS to see whether it is still running
> - I can stop the process from the ODB like any other program
> - Optional, but highly appreciated: Its stdout and stderr should be a MIDAS message.
> 
> Did anyone already solve a similar problem?
> 
> Best regards
> Stefan

Thanks a lot,

Nick. Regarding the messages: Zaher showed me that it is possible to simply place a custom log file generated by the systemd next to midas.log - then it shows up next to the "midas" tab in "Messages".

One follow-up question: Is it possible to use the systemctl status for the "Running on host" column? Or does this even happen automatically?

Best regards
Stefan

> Hi,
> 
> what you describe is exactly how I normally run mhttpd, mlogger, mserver and some other
> custom frontend programs. Eg:
> 
> [local:T2KGSC:Running]/>ls /programs/Logger/
> Required                        y
> Watchdog timeout                100000
> Check interval                  180000
> Start command                   systemctl --user start mlogger
> Auto start                      n
> Auto stop                       n
> Auto restart                    n
> Alarm class                     AlarmNotify
> First failed                    0
> 
> The only exception is your last point about stdout and stderr
> being midas messages. I use journalctl to see these.
> 
> Cheers,
> 
> Nick.
> 
> > I want to control (start / stop / monitor its stdout and stderr) an external process (systemd / EPICS IOC shell script) from within MIDAS.
> > 
> > In order to make this as convenient as possible for the user, I want the process to behave just like any other MIDAS client:
> > - I can start it from the ODB as a program
> > - The process gets regularly polled from MIDAS to see whether it is still running
> > - I can stop the process from the ODB like any other program
> > - Optional, but highly appreciated: Its stdout and stderr should be a MIDAS message.
> > 
> > Did anyone already solve a similar problem?
> > 
> > Best regards
> > Stefan

Hi,
 
> Nick. Regarding the messages: Zaher showed me that it is possible to simply place
> a custom log file generated by the systemd next to midas.log - then it shows up
> next to the "midas" tab in "Messages".

Interesting. I'm not familiar with that feature. Do you have link to documentation?

> One follow-up question: Is it possible to use the systemctl status for the
> "Running on host" column? Or does this even happen automatically?

On the programs page that column is populated by the odb key /System/Clients/<PID>/Host
so no. However, there is nothing stopping you from writing your own version of
programs.html to show whatever you want. For example I have a custom programs
page the includes columns to enable/disable and to reset watchdog alarms.

Cheers,

Nick.

Hi,

unfortunately I don't have a documentation link to the feature, I just know that it works on my machine ;-) The general idea is that you place a custom whatever.log file in Logger/Data Dir (where midas.log is stored). Then, in the Messages page, there will be a "midas" tab and a "whatever" tab - the latter showing the content of whatever.log. One problem here is that timestamping does not work automatically - you have to prepend every line with the same Hours:Minutes:Seconds.Milliseconds Year/Month/Day format that midas.log is using.

So you have a custom Programs page which does systemctl status on your systemd? Does the status then transfer over automatically to the Status page? Is there an example how to write such a custom page?

Best regards
Stefan

> Hi,
>  
> > Nick. Regarding the messages: Zaher showed me that it is possible to simply place
> > a custom log file generated by the systemd next to midas.log - then it shows up
> > next to the "midas" tab in "Messages".
> 
> Interesting. I'm not familiar with that feature. Do you have link to documentation?
> 
> > One follow-up question: Is it possible to use the systemctl status for the
> > "Running on host" column? Or does this even happen automatically?
> 
> On the programs page that column is populated by the odb key /System/Clients/<PID>/Host
> so no. However, there is nothing stopping you from writing your own version of
> programs.html to show whatever you want. For example I have a custom programs
> page the includes columns to enable/disable and to reset watchdog alarms.
> 
> Cheers,
> 
> Nick.

Dear all,

Stefan wants to run an external EPICS driver process as a detached process and somehow "glue" it to midas to control it. Actually a similar requirement led to the development of MIDAS in the '90s. We had too many configuration files lying around, to many process to control and interact together with each other and so on. With the development of MIDAS I wanted to integrate all that. There is one ODB to control an parasitize everything, one central process handling to see if processes are alive, raise an alarm if they die, automatically restart them if necessary and so on. Doing this now externally again is orthogonal to the original design concept of MIDAS and will cause many problems. I therefore strongly recommend to to juggle around with systemctl and syslog, but to make everything a MIDAS process. It's simply a "cm_connect_experiment()" and "cm_disconnect_experiment()" in the end. Then you set

/programs/requited = y

and

/programs/start command = <cmd>

You can set the "alarm class" to raise an alarm if the program crashes, and you will see all messages if you use "cm_msg()" inside the program rather than "printf()". Injecting a separate .log file into the system will show things on the message page, but these messages do not go through the SYSMSG buffer, and cannot received by other programs. Maybe you noticed that mhttpd on the status page always shows the last message it received, which can be very helpful. To see if a program is running, you only need a cm_exist() call, which also exists for custom web pages.

Rather than investing time to re-invent the wheel here, better try to modify your EPICS driver process to become a midas process.

If you have an external process which you absolutely cannot modify, I would rather write a wrapper midas program to start the external process, intercept it's output via a pipe, and put its output properly into the midas message system with cm_msg(). In the main loop of your wrapper function you check the external process via whatever you want, and if it dies trigger an alarm or restart it from your wrapper program. You can then set an alarm on your wrapper program to make sure this one is always running.

Best regards,
StefanR

> Rather than investing time to re-invent the wheel here, better try to modify your EPICS driver process to 
become a midas process.

I am with Stefan on this. Quite a bit of work went into the tmfe c++ framework to make it easy/easier to do 
this - take an existing standalone c/c++ program and midas-ize it: in main(), "just add" calls to connect to 
midas and to start the midas threads - rpc handler, watchdog, etc.

Alternatively, one can write a midas "stdout+stderr bridge", and start your standalone program
from the programs page like this:

myprogram |& cm_msg_bridge --name "myprogram" (redirect both stdout and stderr to cm_msg_bridge stdin)

cm_msg_bridge would read stdin and put them in cm_msg(). it will connect to midas using the name "myprogram" 
to make it show "green" on the status page and it will be stoppable from the programs page.

care will need to be taken for myprogram to die cleanly when stdout and stderr are closed after cm_msg_bridge 
exits.

K.O.

I am updating TWIST to the latest MIDAS and when I load a saved .odb file, I get
these messages. Their text ought to say where and what strings it does not like.
K.O.



[twistonl@midtwist ~/online]$ odbedit
Please define environment variable 'MIDASSYS'
pointing to the midas installation directory.
[local:twist:S]/>load /twist/data_onl/current/run17548.odb
[odb.c:5600:db_paste] found string exceeding MAX_STRING_LENGTH
[odb.c:5600:db_paste] found string exceeding MAX_STRING_LENGTH
[odb.c:5600:db_paste] found string exceeding MAX_STRING_LENGTH

Can you attach 

/twist/data_onl/current/run17548.odb

so I can reproduce the problem?

odbedit command "rename Display xxx/yyy" creates a key named "xxx/yyy" (yes,
with a slash in the name) and this key cannot be deleted or renamed...
K.O.

> odbedit command "rename Display xxx/yyy" creates a key named "xxx/yyy" (yes,
> with a slash in the name) and this key cannot be deleted or renamed...
> K.O.

"rename" is "rename", not "mv" under Unix. If you want this functionality, put it
in and don't complain!

The upgrade of TWIST to the latest midas has bombed- we see mevb and mlogger
crashes during shared memory data buffer accesses. I am looking into it and I
will add information as I figure things out. K.O.

> The upgrade of TWIST to the latest midas has bombed- we see mevb and mlogger
> crashes during shared memory data buffer accesses. I am looking into it and I
> will add information as I figure things out. K.O.

Since 1.9.5 the EventBuilder has been modified. Please consult the documentation
where the new mevb scheme is explained.
Test of the mevb with up to 16 frontends (15 different CPUs) has been tested
successfully. Data rate at the EventBuilder were measured about 50MB/s without the
logger and ~30MB/s with the logger.

> The upgrade of TWIST to the latest midas has bombed- we see mevb and mlogger
> crashes during shared memory data buffer accesses. I am looking into it and I
> will add information as I figure things out. K.O.

I traced buffer memory corruption to a logic error in system.c::ss_shm_open(). If
a .SHM file exists, it's size is used as the size of the sysv shared memory
segment, even if the requested shared memory size is bigger, but the caller of
ss_shm_open()  thinks it got all the requested memory. Eventually we try to use
the unallocated memory and crash. This is the proposed fix and I will commit it
after I retest the upgrade during the next few days.

[olchansk@send src]$ cvs diff -u system.c
olchansk@midas.psi.ch's password: 
Index: system.c
===================================================================
RCS file: /usr/local/cvsroot/midas/src/system.c,v
retrieving revision 1.83
diff -u -r1.83 system.c
--- system.c    4 Oct 2004 07:04:01 -0000       1.83
+++ system.c    14 Oct 2004 05:51:16 -0000
@@ -544,8 +544,14 @@
       } else {
          /* if file exists, retrieve its size */
          file_size = (INT) ss_file_size(file_name);
-         if (file_size > 0)
+         if (file_size > 0) {
+            if (file_size < size) {
+               cm_msg(MERROR, "ss_shm_open", "Shared memory segment \'%s\' size
%d is smaller than requested size %d. Please remove it and try
again",file_name,file_size,size);
+               return SS_NO_MEMORY;
+            }
+            
             size = file_size;
+         }
       }
 
       /* get the shared memory, create if not existing */

K.O.