Hi,
The question was about the TMFeRpcHandlerInterface, not the TARunObject interface.  Derived classes of TARunObject do indeed work as expected in our 
environment.  We have worked around the issue by using an implementation of TARunObject as well as the (separate) implementation of 
TMFeRpcHandlerInterface.

Thanks,

Mark.

> > I have a manalyzer that uses a derived class of TMFeRpcHandlerInterface to communicate information to 
> > Midas during online running.  At the end of each run it saves out custom data in the 
> > TMFeRpcHandlerInterface::HandleEndRun override. This works really well.
> > However, when I run offline on a Midas output file the HandleEndRun method is never called and my data is 
> > never saved.  Is this intentional?  I understand that there is no point for the HandleBinaryRpc method offline, 
> > but the other methods (HandleEndRun, HandleBeginRun etc) could serve a purpose.  Or is it a conscious 
> > choice to ignore all of TMFeRpcHandlerInterface when offline?
> 
> apologies for delayed response.
> 
> I saw the question, completely did not understand it, only now got around to figure out what is going on.
> 
> according to manalyzer/README.md, section "manalyzer module and object life time", BeginRun() and EndRun() is called 
> always. Offline and online. What you see would be a bug that we do not see in our environment. I confirm this by 
> running manalyzer in a demo mode: ./bin/manalyzer_test.exe  --demo -e10 -t
> 
> no, wait, you say you use HandleBeginRun() and HandleEndRun(). this is not right, they are not part of the manalyzer 
> API and they indeed are only used when running online.
> 
> correct solution would be to use BeginRun() and EndRun() instead of HandleBeginRun() and HandleEndRun().
> 
> you could also save your data in the module destructor (although good programming recommendation is to use 
> destructor only for unavoidable things, like freeing memory, etc).
> 
> K.O.

> The question was about the TMFeRpcHandlerInterface, not the TARunObject interface.  Derived classes of TARunObject do indeed work as expected in our 
> environment.  We have worked around the issue by using an implementation of TARunObject as well as the (separate) implementation of 
> TMFeRpcHandlerInterface.

then I do not understand the question. TMFeRpcHandlerInterface stuff is only used when running online and connected to MIDAS. How does it come into the 
picture when you analyze a data file offline? ProcessMidasOnlineTmfe() does not run, the RpcHandler object is not constructed.

maybe if you point me to your source code, I can see what you are doing?

K.O.

This was exactly the question, should I expect it to run?  There's no point in the HandleBinaryRpc method offline, but there's an argument that the HandleBeginRun/HandleEndRun methods have a use.
I have the answer and we have a workaround, thanks.

> then I do not understand the question. TMFeRpcHandlerInterface stuff is only used when running online and connected to MIDAS. How does it come into the 
> picture when you analyze a data file offline? ProcessMidasOnlineTmfe() does not run, the RpcHandler object is not constructed.
> 
> maybe if you point me to your source code, I can see what you are doing?
> 
> K.O.

I do not understand what you are doing. If you are offline, there is no TMFE singleton instance,
there is nothing TMFeRpcHandlerInterface to attach to, there is nobody to call TMFeRpcHandlerInterface methods.

Maybe what you are asking for is a mode where you analyze data from a file, but you want your analysis code
to think that it is online and that it is analyzing live data. This requires creating a fake TMFE singleton, attaching
TMFeRpcHandlerInterface to this fake TMFE singleton and using ProcessMidasOnlineTmfe(), driven by all this fake stuff:
a fake OpenBuffer() that actually opens a file, a fake ReceiveEvent() that actually reads from a file, fake callbacks
for begin and end run, etc.

That's a lot of work, but for what purpose? What is it about the existing offline and online modes that you do not like
and how all this fake stuff will make it better for you?

P.S. This is a 3rd version of my reply. Wrote and deleted 2 version. I think I completely do not understand
what you are doing and you completely do not understand what I am saying. Communication is not happening.

P.P.S. Simplest if you show me your code (email, elog), I am quite good at reading code and divining what
people are trying to do. You do not have to show me any of your secret secret stuff.

K.O.

> This was exactly the question, should I expect it to run?  There's no point in the HandleBinaryRpc method offline, but there's an argument that the HandleBeginRun/HandleEndRun methods have a use.
> I have the answer and we have a workaround, thanks.
> 
> > then I do not understand the question. TMFeRpcHandlerInterface stuff is only used when running online and connected to MIDAS. How does it come into the 
> > picture when you analyze a data file offline? ProcessMidasOnlineTmfe() does not run, the RpcHandler object is not constructed.
> > 
> > maybe if you point me to your source code, I can see what you are doing?
> > 
> > K.O.

Dear experts,
      in the CYGNO experiment, we readout CMOS cameras for optical readout of GEM-TPCs. So far, we only developed the readout for a single camera. In the future, we will have multiple cameras to read out (up to 18 in the next phase of the experiment), and we are investigating how to optimize the readout by exploiting parallelization.

One idea is to start parallel threads within a single equipment. Alternatively, one could associate different equipment with each camera and run an Event Builder. Perhaps other solutions did not come to mind. Which one would you regard as the most effective and elegant?

Thank you very much,
           Francesco

> Dear experts,
>       in the CYGNO experiment, we readout CMOS cameras for optical readout of GEM-TPCs. So far, we only developed the readout for a single camera. In the future, we will have multiple cameras to read out (up to 18 in the next phase of the experiment), and we are investigating how to optimize the readout by exploiting parallelization.
> 
> One idea is to start parallel threads within a single equipment. Alternatively, one could associate different equipment with each camera and run an Event Builder. Perhaps other solutions did not come to mind. Which one would you regard as the most effective and elegant?
> 
> Thank you very much,
>            Francesco

In principle both will work. It's kind of matter of taste. In the multi-threaded approach one has a single frontend to start and stop, and in the second case you have to start 18 individual frontends and make sure that they are running. 

For the multi-threaded frontend you have to ensure proper synchronization between the threads (like common run start/stop), and in the end you also have to do some event building, sending all 18 streams into a single buffer. As you know, multi-thread programming can be a bit of an art, using mutexes or semaphores, but it can be more flexible as the event builder which is a given piece of software.

Best,
Stefan

Dear experts,
I would like to know how to retrieve the online data during the experiment so 
that I can create my own custom plot. I execute my own frontend.exe to start the 
experiment. I can get a midas file after the experiment, but I am not sure about 
how to retrieve the online data. I know that rootana can help us to get the 
online plots, but the instructions in rootana is not clear. Can anyone give me 
some suggestion?  Thank you.

Best,
Terry

> I would like to know how to retrieve the online data during the experiment so 
> that I can create my own custom plot. I execute my own frontend.exe to start the 
> experiment. I can get a midas file after the experiment, but I am not sure about 
> how to retrieve the online data. I know that rootana can help us to get the 
> online plots, but the instructions in rootana is not clear. Can anyone give me 
> some suggestion?  Thank you.

The current package for analyzing MIDAS data is the "m" analyzer, usually in the manalyzer subdirectory of your midas package, 
but it can also be used stand-alone without MIDAS.

There is several examples:

manalyzer_example_cxx.cxx - a simple "c++" example shows how to extra midas bank data
manalyzer_example_root.cxx - how to create ROOT histograms (that you can see online using jsroot)
manalyzer_example_root_graphics.cxx - how to create a ROOT graphical program (obsoleted by jsroot, but still possible)
manalyzer_example_flow*.cxx - more advanced examples on using a flow analyzer

Documentation is in README.md

Unfortunately there is no tutorial or 5 min youtube explainer, each experiment needs are very different, there is no way to 
write a one-size-fits-all recipe.

Please take a look at the existing examples first, then send me a PM with any additional questions (or ask here). If you can 
explain what kind of data you have and how you want to look at it, I should be able to guide you through writing an appropriate 
manalyzer module.

K.O.

> > I would like to know how to retrieve the online data during the experiment so 
> > that I can create my own custom plot. I execute my own frontend.exe to start the 
> > experiment. I can get a midas file after the experiment, but I am not sure about 
> > how to retrieve the online data. I know that rootana can help us to get the 
> > online plots, but the instructions in rootana is not clear. Can anyone give me 
> > some suggestion?  Thank you.
> 
> The current package for analyzing MIDAS data is the "m" analyzer, usually in the manalyzer subdirectory of your midas package, 
> but it can also be used stand-alone without MIDAS.
> 
> There is several examples:
> 
> manalyzer_example_cxx.cxx - a simple "c++" example shows how to extra midas bank data
> manalyzer_example_root.cxx - how to create ROOT histograms (that you can see online using jsroot)
> manalyzer_example_root_graphics.cxx - how to create a ROOT graphical program (obsoleted by jsroot, but still possible)
> manalyzer_example_flow*.cxx - more advanced examples on using a flow analyzer
> 
> Documentation is in README.md
> 
> Unfortunately there is no tutorial or 5 min youtube explainer, each experiment needs are very different, there is no way to 
> write a one-size-fits-all recipe.
> 
> Please take a look at the existing examples first, then send me a PM with any additional questions (or ask here). If you can 
> explain what kind of data you have and how you want to look at it, I should be able to guide you through writing an appropriate 
> manalyzer module.
> 
> K.O.

> > I would like to know how to retrieve the online data during the experiment so 
> > that I can create my own custom plot. I execute my own frontend.exe to start the 
> > experiment. I can get a midas file after the experiment, but I am not sure about 
> > how to retrieve the online data. I know that rootana can help us to get the 
> > online plots, but the instructions in rootana is not clear. Can anyone give me 
> > some suggestion?  Thank you.
> 
> The current package for analyzing MIDAS data is the "m" analyzer, usually in the manalyzer subdirectory of your midas package, 
> but it can also be used stand-alone without MIDAS.
> 
> There is several examples:
> 
> manalyzer_example_cxx.cxx - a simple "c++" example shows how to extra midas bank data
> manalyzer_example_root.cxx - how to create ROOT histograms (that you can see online using jsroot)
> manalyzer_example_root_graphics.cxx - how to create a ROOT graphical program (obsoleted by jsroot, but still possible)
> manalyzer_example_flow*.cxx - more advanced examples on using a flow analyzer
> 
> Documentation is in README.md
> 
> Unfortunately there is no tutorial or 5 min youtube explainer, each experiment needs are very different, there is no way to 
> write a one-size-fits-all recipe.
> 
> Please take a look at the existing examples first, then send me a PM with any additional questions (or ask here). If you can 
> explain what kind of data you have and how you want to look at it, I should be able to guide you through writing an appropriate 
> manalyzer module.
> 
> K.O.

Dear K.O.
I would like to create some online plots (also reading the mid) when the frontend code is running. I may try to break the task into small steps first.
In the frontend.cxx, I have a bank called "TDC0" which includes an array of integer. The code is as follow:

bk_create(pevent, "TDC0", TID_INT, (void**) &pdata);
for(int i =0; i<ch; ++i)
*pdata++ = data[i];
bk_close(pevent,pdata);

In the meantime, I want to run manalyzer to get some plots.

So I started our DAQ with an updated midas, after ca. 6 months+. 

No issues except all FEs complaining about the Alarm ODB structure.
* I adapted to the new structure ( trigger count & trigger count required )
* restarted fe's
* recompiled  

18:17:40.015 2025/09/30 [EPICS Frontend,INFO] Fixing ODB "/Alarms/Alarms/logger" 
struct size mismatch (expected 452, odb size 460)

18:17:40.009 2025/09/30 [SC Frontend,INFO] Fixing ODB "/Alarms/Alarms/logger" 
struct size mismatch (expected 460, odb size 452)

how do I get the FEs + ODB back in line here?

thanks

> So I started our DAQ with an updated midas, after ca. 6 months+.

Would be worthwhile mentioning the git commit hash or tag you are using.

> No issues except all FEs complaining about the Alarm ODB structure.
> * I adapted to the new structure ( trigger count & trigger count required )
> * restarted fe's
> * recompiled  
> 
> 18:17:40.015 2025/09/30 [EPICS Frontend,INFO] Fixing ODB "/Alarms/Alarms/logger" 
> struct size mismatch (expected 452, odb size 460)
> 
> 18:17:40.009 2025/09/30 [SC Frontend,INFO] Fixing ODB "/Alarms/Alarms/logger" 
> struct size mismatch (expected 460, odb size 452)

This seems to be https://daq00.triumf.ca/elog-midas/Midas/2980

> how do I get the FEs + ODB back in line here?

Recompile all frontends against new midas.

Nick.

Just to be clear, it seems that your "EPICS Frontend" was either not recompiled against the new midas yet or the old binary is being run, but "SC Frontend" is using the new midas.

> > So I started our DAQ with an updated midas, after ca. 6 months+.
> 
> Would be worthwhile mentioning the git commit hash or tag you are using.
> 
> > No issues except all FEs complaining about the Alarm ODB structure.
> > * I adapted to the new structure ( trigger count & trigger count required )
> > * restarted fe's
> > * recompiled
> > 
> > 18:17:40.015 2025/09/30 [EPICS Frontend,INFO] Fixing ODB "/Alarms/Alarms/logger" 
> > struct size mismatch (expected 452, odb size 460)
> > 
> > 18:17:40.009 2025/09/30 [SC Frontend,INFO] Fixing ODB "/Alarms/Alarms/logger" 
> > struct size mismatch (expected 460, odb size 452)
> 
> This seems to be https://daq00.triumf.ca/elog-midas/Midas/2980
> 
> > how do I get the FEs + ODB back in line here?
> 
> Recompile all frontends against new midas.
> 
> Nick.

Sorry to intervene there, but the FEs are usually compiled against libmidas.a . Therefore you have to compile midas, usually do a "make install" to update the libmidas.a/so, then recompile the FEs. You probably forgot the "make install".

Stefan

> Sorry to intervene there, but the FEs are usually compiled against libmidas.a . Therefore you have to compile midas, usually do a "make install" to update the libmidas.a/so, then recompile the FEs. You probably forgot the "make install".
> 
> Stefan

OK, solved, closed. Turned out I messed up rebuilding some of the FEs.

More generally, and with the "Documentation" discussion of the workshop in mind, ODB mismatch error messages of all kind are a recurring phenomena confusing users. And MIDASGPT gave complete wrong suggestions.  

Dear all,

I want to control (start / stop / monitor its stdout and stderr) an external process (systemd / EPICS IOC shell script) from within MIDAS.

In order to make this as convenient as possible for the user, I want the process to behave just like any other MIDAS client:
- I can start it from the ODB as a program
- The process gets regularly polled from MIDAS to see whether it is still running
- I can stop the process from the ODB like any other program
- Optional, but highly appreciated: Its stdout and stderr should be a MIDAS message.

Did anyone already solve a similar problem?

Best regards
Stefan

Hi,

what you describe is exactly how I normally run mhttpd, mlogger, mserver and some other
custom frontend programs. Eg:

[local:T2KGSC:Running]/>ls /programs/Logger/
Required                        y
Watchdog timeout                100000
Check interval                  180000
Start command                   systemctl --user start mlogger
Auto start                      n
Auto stop                       n
Auto restart                    n
Alarm class                     AlarmNotify
First failed                    0

The only exception is your last point about stdout and stderr
being midas messages. I use journalctl to see these.

Cheers,

Nick.

> I want to control (start / stop / monitor its stdout and stderr) an external process (systemd / EPICS IOC shell script) from within MIDAS.
> 
> In order to make this as convenient as possible for the user, I want the process to behave just like any other MIDAS client:
> - I can start it from the ODB as a program
> - The process gets regularly polled from MIDAS to see whether it is still running
> - I can stop the process from the ODB like any other program
> - Optional, but highly appreciated: Its stdout and stderr should be a MIDAS message.
> 
> Did anyone already solve a similar problem?
> 
> Best regards
> Stefan

Thanks a lot,

Nick. Regarding the messages: Zaher showed me that it is possible to simply place a custom log file generated by the systemd next to midas.log - then it shows up next to the "midas" tab in "Messages".

One follow-up question: Is it possible to use the systemctl status for the "Running on host" column? Or does this even happen automatically?

Best regards
Stefan

> Hi,
> 
> what you describe is exactly how I normally run mhttpd, mlogger, mserver and some other
> custom frontend programs. Eg:
> 
> [local:T2KGSC:Running]/>ls /programs/Logger/
> Required                        y
> Watchdog timeout                100000
> Check interval                  180000
> Start command                   systemctl --user start mlogger
> Auto start                      n
> Auto stop                       n
> Auto restart                    n
> Alarm class                     AlarmNotify
> First failed                    0
> 
> The only exception is your last point about stdout and stderr
> being midas messages. I use journalctl to see these.
> 
> Cheers,
> 
> Nick.
> 
> > I want to control (start / stop / monitor its stdout and stderr) an external process (systemd / EPICS IOC shell script) from within MIDAS.
> > 
> > In order to make this as convenient as possible for the user, I want the process to behave just like any other MIDAS client:
> > - I can start it from the ODB as a program
> > - The process gets regularly polled from MIDAS to see whether it is still running
> > - I can stop the process from the ODB like any other program
> > - Optional, but highly appreciated: Its stdout and stderr should be a MIDAS message.
> > 
> > Did anyone already solve a similar problem?
> > 
> > Best regards
> > Stefan

Hi,
 
> Nick. Regarding the messages: Zaher showed me that it is possible to simply place
> a custom log file generated by the systemd next to midas.log - then it shows up
> next to the "midas" tab in "Messages".

Interesting. I'm not familiar with that feature. Do you have link to documentation?

> One follow-up question: Is it possible to use the systemctl status for the
> "Running on host" column? Or does this even happen automatically?

On the programs page that column is populated by the odb key /System/Clients/<PID>/Host
so no. However, there is nothing stopping you from writing your own version of
programs.html to show whatever you want. For example I have a custom programs
page the includes columns to enable/disable and to reset watchdog alarms.

Cheers,

Nick.

Hi,

unfortunately I don't have a documentation link to the feature, I just know that it works on my machine ;-) The general idea is that you place a custom whatever.log file in Logger/Data Dir (where midas.log is stored). Then, in the Messages page, there will be a "midas" tab and a "whatever" tab - the latter showing the content of whatever.log. One problem here is that timestamping does not work automatically - you have to prepend every line with the same Hours:Minutes:Seconds.Milliseconds Year/Month/Day format that midas.log is using.

So you have a custom Programs page which does systemctl status on your systemd? Does the status then transfer over automatically to the Status page? Is there an example how to write such a custom page?

Best regards
Stefan

> Hi,
>  
> > Nick. Regarding the messages: Zaher showed me that it is possible to simply place
> > a custom log file generated by the systemd next to midas.log - then it shows up
> > next to the "midas" tab in "Messages".
> 
> Interesting. I'm not familiar with that feature. Do you have link to documentation?
> 
> > One follow-up question: Is it possible to use the systemctl status for the
> > "Running on host" column? Or does this even happen automatically?
> 
> On the programs page that column is populated by the odb key /System/Clients/<PID>/Host
> so no. However, there is nothing stopping you from writing your own version of
> programs.html to show whatever you want. For example I have a custom programs
> page the includes columns to enable/disable and to reset watchdog alarms.
> 
> Cheers,
> 
> Nick.