This was exactly the question, should I expect it to run?  There's no point in the HandleBinaryRpc method offline, but there's an argument that the HandleBeginRun/HandleEndRun methods have a use.
I have the answer and we have a workaround, thanks.

> then I do not understand the question. TMFeRpcHandlerInterface stuff is only used when running online and connected to MIDAS. How does it come into the 
> picture when you analyze a data file offline? ProcessMidasOnlineTmfe() does not run, the RpcHandler object is not constructed.
> 
> maybe if you point me to your source code, I can see what you are doing?
> 
> K.O.

>  > All this is kind of reasonable, as only two settings of write cache size are useful: 0 to 
> > disable it, and 10 Mbytes to limit semaphore locking rate to reasonable value for all event 
> > rate and size values practical on current computers.
> 
> Indeed KO is correct that only 0 and 10MB make sense, and we cannot mix it. Having the cache setting in the equipment table is 
> cumbersome. If you have 10 slow control equipment (cache size zero), you need to add many zeros at the end of 10 equipment 
> definitions in the frontend. 
> 
> I would rather implement a function or variable similar to fEqConfWriteCacheSize in the tmfe framework also in the mfe.cxx 
> framework, then we need only to add one line llike
> 
> gEqConfWriteCacheSize = 0;
> 
> in the frontend.cxx file and this will be used for all equipments of that frontend. If nobody complains, I will do that in April when I'm 
> back from Japan.

Cache size is per-buffer. If different equipments write into different event buffers, should be possible to set different cache sizes.

Perhaps have:

set_write_cache_size("SYSTEM", 0);
set_write_cache_size("BUF1", bigsize);

with an internal std::map<std::string,size_t>; for write cache size for each named buffer

K.O.

> I hopefully fixed the waring (narrowing down from size_t to int). Please double check with your compiler.

Nope, it turns out complain is about the read() size argument, they really really really want it to be 
size_t.

Looks like correct sequence is:

size_t size = file_size_somehow();
char* buf = (char*)malloc(size+1);
ssize_t rd = read(fd, buf, size);
if (rd < 0) { complain(); return; } // read() error
if ((size_t)rd != size) { complain(); return; } // cast to size_t is safe after we know rd >= 0
buf[rd] = 0; // make sure data is NUL terminated

What a mess. I want my UNIX V7 and K&R C back!

Perhaps I should add in system.cxx - ss_read_file(const char* filename, std::vector<char> &data);

Fixed it in mscb.cxx, odbedit.cxx, mhttpd.cxx and msequencer.cxx, commited, pushed.

U-24 builds without warnings. fwew...

K.O.

Confirming warning is fixed. Thanks! K.O.

> > Unnamed person who added this clever bit of c++ coding, please fix this compiler warning. Stock g++ on Ubuntu LTS 24.04. Thanks in advance!
> > 
> > /home/olchansk/git/midas/src/system.cxx: In function ‘std::string ss_execs(const char*)’:
> > /home/olchansk/git/midas/src/system.cxx:2256:43: warning: ignoring attributes on template argument ‘int (*)(FILE*)’ [-Wignored-attributes]
> >  2256 |    std::unique_ptr<FILE, decltype(&pclose)> pipe(popen(cmd, "r"), pclose);
> > 
> > K.O.
> 
> Replace the code with:
> 
>    auto pclose_deleter = [](FILE* f) { pclose(f); };
>    auto pipe = std::unique_ptr<FILE, decltype(pclose_deleter)>(
>       popen(cmd, "r"),
>       pclose_deleter
>    );
> 
> 
> Hope this is now warning-free.
> 
> Stefan

> https://cmake.org/cmake/help/latest/guide/using-dependencies/index.html#guide:Using%20Dependencies%20Guide

thank you for providing a link to latest cmake find_package() guide.

I notice that this documentation was added in cmake 3.24 released circa Nov 2022
and does not exist in older versions. (It is easy to see in git history the last
time I touched any cmake stuff in midas).

I still see no documentation on "this is how you write a package that other people can import
using find_package()". Only see documentation on how to use find_package() on packages
that somebody who somehow knows how to do it already wrote.

> In the original message to this thread i posted reference to PR
> (https://bitbucket.org/tmidas/midas/pull-requests/48)

this pull request was rail-roaded through during the holidays without any
discussion on this forum. I was not given an opportunity to comment to it,
it was pushed and merged faster than I could blink.

bottom line. I voted against using cmake and was over-ruled. To me this cmake stuff
is only a source of wasted time and created bad feelings.

if midas is required to use cmake, we should have somebody on the team that at least
understands it and if not love it, at least does not hate it.

K.O.

> The question was about the TMFeRpcHandlerInterface, not the TARunObject interface.  Derived classes of TARunObject do indeed work as expected in our 
> environment.  We have worked around the issue by using an implementation of TARunObject as well as the (separate) implementation of 
> TMFeRpcHandlerInterface.

then I do not understand the question. TMFeRpcHandlerInterface stuff is only used when running online and connected to MIDAS. How does it come into the 
picture when you analyze a data file offline? ProcessMidasOnlineTmfe() does not run, the RpcHandler object is not constructed.

maybe if you point me to your source code, I can see what you are doing?

K.O.

> Andrea Capra reported a problem with manalyzer module initialization order.

Permanent solution is now implemented.

In the analyzer module constructor (TARunObject), please set the value of fModuleOrder (default value is 0).

Modules with smaller fModuleOrder will run first, modules with bigger fModuleOrder will run last.

Please use the value range between -1 (built-in EventDump module) and 9999 (built-in InteractiveModule).

Modules with equal fModuleOrder will run in the same order as they were registered (std::stable_sort).

Example manalyzer modules and documentation README.md have been updated.

Linker command line and GCC attribute methods should also work, but I thought it better to provide explicit 
programmatic control of module ordering.

P.S. the c++ tmfe frontend module design is newer and the problem of module (equipment) ordering is solved by 
constructing them explicitly in main().

manalyzer commit:

commit 6ca130808cd05ead734450391bf6defc8335c04a (HEAD -> master, origin/master, origin/HEAD)
Author: Konstantin Olchanski <olchansk@daq00.triumf.ca>
Date:   Tue Mar 25 15:53:13 2025 -0700

    implement TARunObject::fModuleOrder to specify required ordering of analyzer modules

K.O.

> In the PSI muSR laboratory, we are running about 140 slow control devices across six instruments using Format FIXED.
> Could you please wait a bit with removing support for it so that we can assess if/how this will affect us?

Roger. I believe as long as these data do not go into the MIDAS event data stream, you should not see any difference from 
changing "FIXED" to "MIDAS", I hope you can test it and report how it works for you. If you see that things change in ODB 
or in history, perhaps we can implement some kind of workaround to make the transition transparent.

K.O.

Hi,
The question was about the TMFeRpcHandlerInterface, not the TARunObject interface.  Derived classes of TARunObject do indeed work as expected in our 
environment.  We have worked around the issue by using an implementation of TARunObject as well as the (separate) implementation of 
TMFeRpcHandlerInterface.

Thanks,

Mark.

> > I have a manalyzer that uses a derived class of TMFeRpcHandlerInterface to communicate information to 
> > Midas during online running.  At the end of each run it saves out custom data in the 
> > TMFeRpcHandlerInterface::HandleEndRun override. This works really well.
> > However, when I run offline on a Midas output file the HandleEndRun method is never called and my data is 
> > never saved.  Is this intentional?  I understand that there is no point for the HandleBinaryRpc method offline, 
> > but the other methods (HandleEndRun, HandleBeginRun etc) could serve a purpose.  Or is it a conscious 
> > choice to ignore all of TMFeRpcHandlerInterface when offline?
> 
> apologies for delayed response.
> 
> I saw the question, completely did not understand it, only now got around to figure out what is going on.
> 
> according to manalyzer/README.md, section "manalyzer module and object life time", BeginRun() and EndRun() is called 
> always. Offline and online. What you see would be a bug that we do not see in our environment. I confirm this by 
> running manalyzer in a demo mode: ./bin/manalyzer_test.exe  --demo -e10 -t
> 
> no, wait, you say you use HandleBeginRun() and HandleEndRun(). this is not right, they are not part of the manalyzer 
> API and they indeed are only used when running online.
> 
> correct solution would be to use BeginRun() and EndRun() instead of HandleBeginRun() and HandleEndRun().
> 
> you could also save your data in the module destructor (although good programming recommendation is to use 
> destructor only for unavoidable things, like freeing memory, etc).
> 
> K.O.

Thanks Stefan, I would be very interested to see your code. At moment we have magnets and cryostats (3He and dilution) being delivered with Labview.



> > Hello,
> > 
> > Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.
> > 
> > thanks,
> > Zaher
> 
> We do have a superconducting magnet from Cryogenic, UK, which comes with a LabView control program on a Windows PC. I did the only reasonable with this: trash it in the waste basket. Do NOT use Labveiw for anything which should run more than 24h in a row. Too many bad experiences with LabView control programs 
> for separators at PSI and other devices. Instead of the Windows PC, we use MSCB devices and RasperryPis to communicate with the power supply directly, which has been proven to be much more stable (running for years without crashes). I'm happy to share our code with you.
> 
> Stefan

> > > > currently to link Midas to project one has to do several steps ...
> > > this information is incorrect. please read https://daq00.triumf.ca/elog-midas/Midas/2258
> > 
> > I admit that i did not see your post about targets import
> > via `include($ENV{MIDASSYS}/lib/midas-targets.cmake)`
> > before implementing changes to cmake scripts.
> >
> > But in this respect the way you propose to do it via `include` should still work.
> >
> 
> I proposed nothing, you did the proposing. I spent many hours trying to understand cmake (mission 
> impossible!) and many more hours to implement the previously existing package scheme based
> on the cmake "EXPORT" function.
I agree that cmake is difficult, especially when it comes to creating cmake scripts for library
that should work for other people (as opposed to just using other libraries).
But that is why we should try to follow recommended way of using it.

> 
> > Note however that `include(...)` way is very unusual as one have to know exactly
> > where `...-targets.cmake` is located and standard way in cmake is via `find_package`
> > (similar to how e.g. ROOT, Geant4, etc. are found and linked).
> 
> Very difficult to cut-and-paste "include($ENV{MIDASSYS}/lib/midas-targets.cmake)".
> 
> You cannot simplify-out $ENV{MIDASSYS} because computer cannot read your mind, which of the 10 copies
> of midas you want to use from which user account on which day.
One can use `find_package(Midas PATH $ENV{MIDASSYS})` to set specific location of Midas
(this is mentioned in https://bitbucket.org/tmidas/midas/pull-requests/48)
and without `PATH` argument the default system/user locations are searched.

> 
> Argument about "very unusual" I would buy, I am not a cmake expert and I do not know which package
> finding method is in favour today.
> 
> >
> > Though i again admit that maybe the namespace change was a bit too much as it may
> > have broken previous users of `include($ENV{MIDASSYS}/lib/midas-targets.cmake)`
> > 
> 
> I believe it did break at least one experiment, after updating MIDAS to latest version,
> the analyzer would not build.
This unfortunately was not easy to avoid in this case, as both midas and manalyzer depend on each other:
midas should compile when manalyzer is enabled (as submodule)
and manalyzer should compile with midas as separate lib.
So in this case i would expect both midas and manalyzer be updated at same time to their matching versions.

> 
> Speaking of which, did you implement your new scheme for the manalyzer so that it works
> in standalone mode (without MIDAS)?
The only change on manalyzer i did was to use `find_package(Midas ...)` and `midas::midas` target
(see https://bitbucket.org/tmidas/manalyzer/commits/b219a916).
If there is interest to use same scheme in manalyzer as in Midas i can implement it.

> 
> If you did not, now we have two schemes, your new scheme just for MIDAS and my old scheme
> for manalyzer *and* MIDAS. xkcd 927.
> 
> > 
> > - shortcomings of what was before is usage of non-standard `include(...)`
> >
> 
> You should have started by posting a message spelling it out: Konstantin implemented
> a scheme that uses the cmake "export" function to find midas, mfe and manalyzer,
> it is very nice and works ok, but it is non-standard/obsoleted/obscure/frowned-upon/
> unpopular/I-do-not-like-it/I-did-not-invent-it, and I propose implementing a new scheme
> based on find_package().
As i mentioned i did not see you original post about usage of `include`
and otherwise i may have referenced it and though more about compatibility issues.

> 
> >
> > - one shortcoming i see for new implementation is usage `midas::` namespace
> >   (mentioned above) that may have broken some setups
> > 
> 
> If you think that your changes will break other people code, you should explicitely
> say this in a message to this forum and hopefully provide instruction on fixing it,
> i.e. in your makefile, please replace "midas" with "midas::midas".
In the original message to this thread i posted reference to PR
(https://bitbucket.org/tmidas/midas/pull-requests/48)
where it shows how to use `find_package` with this change.
As i did not expect the direct use of `include()` form
and assumed that manual linking was used (via specifying include/lib paths/names)
some scenarios where code for people broke were missed (not taken into account) by me.

> 
> > 
> > - `find_package` is standard and recommended way of finding packages
> >
> 
> Do you have a reference for this? When I look at cmake documentation, I do not see
> any specific recommendation on creating packages and finding them. I do see
> other people's code for finding packages and often spend hours fighting
> them because said methods are designed to work only on the developer's laptop.
see https://cmake.org/cmake/help/v3.27/guide/importing-exporting/index.html:
- about use of `find_package` see https://cmake.org/cmake/help/latest/guide/using-dependencies/index.html#guide:Using%20Dependencies%20Guide
- about double colon namespace for target see https://cmake.org/cmake/help/v3.27/guide/importing-exporting/index.html
  where it is mentioned "This convention of double-colons gives CMake a hint that the name is an IMPORTED target when it is used by downstream projects".

> 
> P.S. Did anybody ask Ben to update the MidasWiki documentation with the new find_package() information?
> 
> K.O.

> > > ALL MIDAS CLIENTS GET RE-COMPILED after the new code is applied.
>
> - I did clearly announce this change in the forum.
>

I missed the announcement and I was surprised to discover this change.

This is why I changed the title from "useful improvement" to "ACHTUNG!!!"

> - The extension is not there "just for fun" but needed by several experiments which experience spurious alarms 
> triggered by some noisy signals.

Agreed, a useful improvement.

To prevent this kind of trouble in the future, I think I should finish my crusade against db_get_record().

> - If there are frontend programs which have not been re-compiled for ten years, I think it is very unlikely that these 
> experiments need the latest cutting edge version of midas. The can safely stick to your tag midas-2024-12-a and not 
> experience the issue of different /Alarm trees.

Yes, now that I added the tag, posted a more visible notice and people who have to run frontends build against older midas 
(because reasons) can breathe out.

Also please note that "Konstantin != all experiments at TRIUMF".

K.O.

> > > currently to link Midas to project one has to do several steps ...
> > this information is incorrect. please read https://daq00.triumf.ca/elog-midas/Midas/2258
> 
> I admit that i did not see your post about targets import
> via `include($ENV{MIDASSYS}/lib/midas-targets.cmake)`
> before implementing changes to cmake scripts.
>
> But in this respect the way you propose to do it via `include` should still work.
>

I proposed nothing, you did the proposing. I spent many hours trying to understand cmake (mission 
impossible!) and many more hours to implement the previously existing package scheme based
on the cmake "EXPORT" function.

> Note however that `include(...)` way is very unusual as one have to know exactly
> where `...-targets.cmake` is located and standard way in cmake is via `find_package`
> (similar to how e.g. ROOT, Geant4, etc. are found and linked).

Very difficult to cut-and-paste "include($ENV{MIDASSYS}/lib/midas-targets.cmake)".

You cannot simplify-out $ENV{MIDASSYS} because computer cannot read your mind, which of the 10 copies
of midas you want to use from which user account on which day.

Argument about "very unusual" I would buy, I am not a cmake expert and I do not know which package
finding method is in favour today.

>
> Though i again admit that maybe the namespace change was a bit too much as it may
> have broken previous users of `include($ENV{MIDASSYS}/lib/midas-targets.cmake)`
> 

I believe it did break at least one experiment, after updating MIDAS to latest version,
the analyzer would not build.

Speaking of which, did you implement your new scheme for the manalyzer so that it works
in standalone mode (without MIDAS)?

If you did not, now we have two schemes, your new scheme just for MIDAS and my old scheme
for manalyzer *and* MIDAS. xkcd 927.

> 
> - shortcomings of what was before is usage of non-standard `include(...)`
>

You should have started by posting a message spelling it out: Konstantin implemented
a scheme that uses the cmake "export" function to find midas, mfe and manalyzer,
it is very nice and works ok, but it is non-standard/obsoleted/obscure/frowned-upon/
unpopular/I-do-not-like-it/I-did-not-invent-it, and I propose implementing a new scheme
based on find_package().

>
> - one shortcoming i see for new implementation is usage `midas::` namespace
>   (mentioned above) that may have broken some setups
> 

If you think that your changes will break other people code, you should explicitely
say this in a message to this forum and hopefully provide instruction on fixing it,
i.e. in your makefile, please replace "midas" with "midas::midas".

> 
> - `find_package` is standard and recommended way of finding packages
>

Do you have a reference for this? When I look at cmake documentation, I do not see
any specific recommendation on creating packages and finding them. I do see
other people's code for finding packages and often spend hours fighting
them because said methods are designed to work only on the developer's laptop.

P.S. Did anybody ask Ben to update the MidasWiki documentation with the new find_package() information?

K.O.

> > bitbucket automatic builds
> 
> Unfortunately we will break the automatic build each time a program outputs one different character, which even might happen if we add a line of code and 
> a cm_msg() gets produced with a different line number. Is there a standard way to update testexpt.example (like "make testexpt" or so). Should be trigger 
> the update of testexpt.example before each commit via a hook? 
> 

Actually line numbers are not logged by messages printed from "make test", so moving code around does not break the test.

Changing what programs output does break the test and this is intentional - somebody must look at confirm
that program output was changed on purpose or because a bug was introduced (or fixed).

Most "make test" things work this way - run programs, compare output to what is expected. Discrepancies are flagged for human examination.

K.O.

> > Hello,
> > 
> > Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.
> > 
> > thanks,
> > Zaher
> 
> We do have a superconducting magnet from Cryogenic, UK, which comes with a LabView control program on a Windows PC. I did the only reasonable with this: trash it in the waste basket. Do NOT use Labveiw for anything which should run more than 24h in a row. Too many bad experiences with LabView control programs 
> for separators at PSI and other devices. Instead of the Windows PC, we use MSCB devices and RasperryPis to communicate with the power supply directly, which has been proven to be much more stable (running for years without crashes). I'm happy to share our code with you.
>

Our parallel experience with the CERN ALPHA anti-hydrogen experiment: they have developed a whole labview empire
to control the cryogenics, the magnets, the positron source, the anti-proton trap, anti-hydrogen trap, etc.

At some point there was a wall of monitors in the counting room - each labview computer controlled one or two things -
so there is very many computers and each had to have a monitor (and mouse and keyboard).

All the data from this labview empire is logged to MIDAS history via felabview and feGEM, and they use
the MIDAS history to look and monitor almost everything. Control is done via Labview and Labview
based FPGA sequencers (National Instruments PXI hardware, $$$$$).

This works reasonably well to publish several papers in Nature.

But not 100%:

1) difficulties with labview source control (cannot be trivially managed by git, I guess)
2) unending fight against Microsoft and CERN IT trying to reboot the computers at the wrong time
3) more recently, forced Microsoft updates require trashing perfectly good machines and buy new ones

At TRIUMF there is very little Labview. All experiments use MIDAS and EPICS for most things.

Based on this experience, I agree with Stefan, today's sweet spot is RaspberryPi machines with USB attached
gizmos to control stuff. On the software side, drive the mess with MIDAS and custom web pages.

K.O.

> Hello,
> 
> Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.
> 
> thanks,
> Zaher

We do have a superconducting magnet from Cryogenic, UK, which comes with a LabView control program on a Windows PC. I did the only reasonable with this: trash it in the waste basket. Do NOT use Labveiw for anything which should run more than 24h in a row. Too many bad experiences with LabView control programs 
for separators at PSI and other devices. Instead of the Windows PC, we use MSCB devices and RasperryPis to communicate with the power supply directly, which has been proven to be much more stable (running for years without crashes). I'm happy to share our code with you.

Stefan

I hopefully fixed the waring (narrowing down from size_t to int). Please double check with your compiler.

Stefan

> bitbucket automatic builds were broken after mfe.cxx started printing some additional messages added in commit
> https://bitbucket.org/tmidas/midas/commits/0ae08cd3b96ebd8e4f57bfe00dd45527d82d7a38
> 
> this is now fixed. to check if your changes will break automatic builds, before final push, please do:
> 
> make clean
> make mini -j
> make cmake -j
> make test

Unfortunately we will break the automatic build each time a program outputs one different character, which even might happen if we add a line of code and 
a cm_msg() gets produced with a different line number. Is there a standard way to update testexpt.example (like "make testexpt" or so). Should be trigger 
the update of testexpt.example before each commit via a hook? 

Stefan

> > currently to link Midas to project one has to do several steps ...
> 
> this information is incorrect. please read https://daq00.triumf.ca/elog-midas/Midas/2258
> 
> a very simple way to use link MIDAS using midas-targets.cmake has been implemented a long time ago.

I admit that i did not see your post about targets import
via `include($ENV{MIDASSYS}/lib/midas-targets.cmake)`
before implementing changes to cmake scripts.
But in this respect the way you propose to do it via `include` should still work.
Note however that `include(...)` way is very unusual as one have to know exactly
where `...-targets.cmake` is located and standard way in cmake is via `find_package`
(similar to how e.g. ROOT, Geant4, etc. are found and linked).

The things that changed (and are incompatible with what was before)
is the naming of targets (in `midas-targets.cmake` with `midas::` namespace,
which is standard practice in cmake to distinguish cmake targets from bare library names
(e.g. when you do `link_libraries(midas)` it may be interpreted as linking with `-lmidas`
or if target is defined it does machinery to link actual cmake target; the namespace way
makes it unambiguous).
Though i again admit that maybe the namespace change was a bit too much as it may
have broken previous users of `include($ENV{MIDASSYS}/lib/midas-targets.cmake)`

> 
> before proposing a new way of doing things, it would be nice to hear about shortcomings
> of the existing stuff.

- shortcomings of what was before is usage of non-standard `include(...)`
- one shortcoming i see for new implementation is usage `midas::` namespace
  (mentioned above) that may have broken some setups

> A simple "Konstantin's way sucks" or "this is not the cmake way!"
> would have been sufficient.

- `find_package` is standard and recommended way of finding packages
    - note that `include($ENV{MIDASSYS}/lib/midas-targets.cmake)` should still work
      (but with usage of `midas::midas` instead of simply `midas`)
    - in the end `find_package` works by locating and loading `MidasConfig.cmake`,
      and it now actually does `include("${CMAKE_CURRENT_LIST_DIR}/../../midas-targets.cmake")`,
      so in this respect `find_package` is the same as `include(...)`,
      but it also preserves old behavior of exporting cmake vars for includes/libs
      such that prev uses are unaffected,
      and does a bit more checking such that it can be used for both in- and out-of-tree builds
    
- in addition `find_package` allows to handle components,
  e.g. now it is possible to do
  `find_package(Midas COMPONENTS manalyzer)`
  instead of also doing `include($ENV{MIDASSYS}/lib/manalyzer-targets.cmake)`

> 
> K.O.

Alex

> Unnamed person who added this clever bit of c++ coding, please fix this compiler warning. Stock g++ on Ubuntu LTS 24.04. Thanks in advance!
> 
> /home/olchansk/git/midas/src/system.cxx: In function ‘std::string ss_execs(const char*)’:
> /home/olchansk/git/midas/src/system.cxx:2256:43: warning: ignoring attributes on template argument ‘int (*)(FILE*)’ [-Wignored-attributes]
>  2256 |    std::unique_ptr<FILE, decltype(&pclose)> pipe(popen(cmd, "r"), pclose);
> 
> K.O.

Replace the code with:

   auto pclose_deleter = [](FILE* f) { pclose(f); };
   auto pipe = std::unique_ptr<FILE, decltype(pclose_deleter)>(
      popen(cmd, "r"),
      pclose_deleter
   );


Hope this is now warning-free.

Stefan