> > ALL MIDAS CLIENTS GET RE-COMPILED after the new code is applied.
> 
> Usually we expect that it is "safe" to update to the latest version of MIDAS.
> 
> In the sense that we do not have to track down every single frontend
> and rebuild it. We have several experiments where tracking down the source code
> and rebuilding a frontend takes more than 5 seconds. In many cases these are
> 10 year old executables that worked just fine through many updates of MIDAS
> without having to rebuild them.
> 
> So any binary incompatible change is best avoided and must be clearly announced.
> 
> The present binary-incompatible change is this commit:
> https://bitbucket.org/tmidas/midas/commits/5899f1657ba31121c9f420a824b3c6c13b173988
> 
> I tagged the last commit before this change as: midas-2024-12-a

Here are my replies:

- this is not a binary incompatibility, but a incompatibility of the /Alarm record which got two more variables. Old 
frontends will complain during their structure check and remove the two variables, new frontend will then complain as 
well and add the two variables. This will go in circles, that why all frontends need to be recompiled if the new code is 
used

- I did clearly announce this change in the forum. Is there another location where I should communicate that?

- The extension is not there "just for fun" but needed by several experiments which experience spurious alarms 
triggered by some noisy signals. Requiring an value to be above a limit for a certain minimum time fixes many issues in 
many experiments here at PSI, this is why it has been implemented, even if it causes work for everybody with re-
compilation.

- If there are frontend programs which have not been re-compiled for ten years, I think it is very unlikely that these 
experiments need the latest cutting edge version of midas. The can safely stick to your tag midas-2024-12-a and not 
experience the issue of different /Alarm trees.

Stefan

 > All this is kind of reasonable, as only two settings of write cache size are useful: 0 to 
> disable it, and 10 Mbytes to limit semaphore locking rate to reasonable value for all event 
> rate and size values practical on current computers.

Indeed KO is correct that only 0 and 10MB make sense, and we cannot mix it. Having the cache setting in the equipment table is 
cumbersome. If you have 10 slow control equipment (cache size zero), you need to add many zeros at the end of 10 equipment 
definitions in the frontend. 

I would rather implement a function or variable similar to fEqConfWriteCacheSize in the tmfe framework also in the mfe.cxx 
framework, then we need only to add one line llike

gEqConfWriteCacheSize = 0;

in the frontend.cxx file and this will be used for all equipments of that frontend. If nobody complains, I will do that in April when I'm 
back from Japan.

Stefan

Hi Konstantin,

In the PSI muSR laboratory, we are running about 140 slow control devices across six instruments using Format FIXED.

Could you please wait a bit with removing support for it so that we can assess if/how this will affect us?

Many thanks,
Jonas

> we are migrating the dragon experiment from an old mac to a new mac studio and we ran into a problem 
> where one equipment format was set to "fixed" instead of "midas". lots of confusion, mdump crash, 
> analyzer crash, etc. (mdump fixes for this are already committed).
> 
> it made us think whether equipment format is still needed. in the old days we had choice of MIDAS and 
> YBOS formats, but YBOS was removed years ago, and I was surprised that format FIXED was permitted at 
> all.
> 
> I did a midas source code review, this is what I found:
> 
> - remnants of YBOS support in a few places, commit to remove them pending.
> - FORMAT_ROOT is used in mlogger for automatic conversion of MIDAS banks to ROOT trees
> - FORMAT_FIXED is used in a few slow control drivers in drivers/class, instead of creating MIDAS 
> banks, they copy raw data directly into an event (there is no bank header and no way to identify such 
> events automatically)
> - lots of code to support different formats in mdump (mostly dead code)
> - the rest of the code does not care or use this format stuff
> 
> Current proposal is to remove support for all formats except FORMAT_MIDAS (and FORMAT_ROOT in 
> mlogger).
> 
> - defines of FORMAT_XXX will be removed from midas.h
> - "Format" will be removed from ODB Equipment/Common
> - "Format" will be removed from ODB Logger/Channel
> - to maintain binary compatibility, we can keep the "Format" ODB entries, but they will be ignored.
> 
> List of slow control drivers that support FORMAT_FIXED:
> 
> daq00:midas$ grep FORMAT_FIXED drivers/class/*
> drivers/class/cd_fdg.cxx:   if (fgd_info->format == FORMAT_FIXED) {
> drivers/class/cd_ivc32.cxx:   if (hv_info->format == XFORMAT_FIXED) {
> drivers/class/cd_rf.cxx:	if (rf_info->format == XFORMAT_FIXED) 
> drivers/class/generic.cxx:   if (gen_info->format == XFORMAT_FIXED) {
> drivers/class/hv.cxx:   if (hv_info->format == XFORMAT_FIXED) {
> drivers/class/multi.cxx:   if (m_info->format == XFORMAT_FIXED) {
> drivers/class/slowdev.cxx:   if (gen_info->format == XFORMAT_FIXED) {
> daq00:midas$ 
> 
> K.O.

Andrea Capra reported a problem with manalyzer module initialization order.

Original manalyzer design relies on the fact than module static constructors run in the same order as they 
appear on the linker command line. This has been true for a very long time, but now we have evidence that on 
Rocky Linux (unknown gcc version), this is no longer true.

The c++ standard does not define any specific order for static constructor in different source files. 
(unlike C, C++ tends to treat the linker as something magical and ask the linker to do magical things).

The tmfe c++ modular frontend was designed after manalyzer and one design change is to explicitly construct 
all objects from main(). (at the acceptable cost of extra boiler plate code).

One solution is to use GCC attribute "init_priority (priority)", see
https://stackoverflow.com/questions/211237/static-variables-initialisation-order
https://gcc.gnu.org/onlinedocs/gcc/C_002b_002b-Attributes.html

To use this, manalyzer module registration should be modified to read:

static TARegister tar __attribute__((init_priority(200))) (new ExampleCxxFactory);

A less magical solution is to change manalyzer design similar to tmfe where modules are constructed and 
registered explicitely in the order specified by source code.

K.O.

P.S. I now ran out of time to test this and commit it to the documentation. It also need to be tested with 
LLVM C++ compilers.

> time to choose the next c++!

Ununtu-24.04, MIDAS builds with -std=c++23 without errors or any additional warnings. (but does it work? "make 
test" is ok).

K.O.

> > the main purpose of the event buffer write cache
> how to control the write cache size:

OP provided insufficient information to say what went wrong for them, but do try this:

1) in ODB, for all equipments, set write_cache_size to 0
2) in the frontend equipment table, set write_cache_size to 0

That is how it is done in the example frontend: examples/experiment/frontend.cxx

If this configuration still produces an error, we may have a bug somewhere, so please let us know how it shakes out.

K.O.

> the main purpose of the event buffer write cache

how to control the write cache size:

1) in a frontend, all equipments should ask for the same write cache size, both mfe.c and 
tmfe frontends will complain about mismatch

2) tmfe c++ frontend, per tmfe.md, set fEqConfWriteCacheSize in the equipment constructor, in 
EqPreInitHandler() or EqInitHandler(), or set it in ODB. default value is 10 Mbytes or value 
of MIN_WRITE_CACHE_SIZE define. periodic cache flush period is 0.5 sec in 
fFeFlushWriteCachePeriodSec.

3) mfe.cxx frontend, set it in the equipment definition (number after "hidden"), set it in 
ODB, or change equipment[i].write_cache_size. Value 0 sets the cache size to 
MIN_WRITE_CACHE_SIZE, 10 Mbytes.

4) in bm_set_cache_size(), acceptable values are 0 (disable the cache), MIN_WRITE_CACHE_SIZE 
(10 Mbytes) or anything bigger. Attempt to set the cache smaller than 10 Mbytes will set it 
to 10 Mbytes and print an error message.

All this is kind of reasonable, as only two settings of write cache size are useful: 0 to 
disable it, and 10 Mbytes to limit semaphore locking rate to reasonable value for all event 
rate and size values practical on current computers.

In mfe.cxx it looks to be impossible to set the write cache size to 0 (disable it), but 
actually all you need is call "bm_set_cache_size(equipment[0].buffer_handle, 0, 0);" in 
frontend_init() (or is it in begin_of_run()?).

K.O.

the main purpose of the event buffer write cache is to prevent high contention for the 
event buffer shared memory semaphore in the pathological case of very high rate of very 
small events.

there is a computation for this, I have posted it here several times, please search for 
it.

in the nutshell, you want the semaphore locking rate to be around 10/sec, 100/sec 
maximum. coupled with smallest event size and maximum practical rate (1 MHz), this 
yields the cache size.

for slow control events generated at 1 Hz, the write cache is not needed, 
write_cache_size value 0 is the correct setting.

for "typical" physics events generated at 1 kHz, write cache size should be set to fit 
10 events (100 Hz semaphore locking rate) to 100 events (10 Hz semaphore locking rate).

unfortunately, one cannot have two cache sizes for an event buffer, so typical frontends 
that generate physics data at 1 kHz and scalers and counters at 1 Hz must have a non-
zero write cache size (or semaphore locking rate will be too high).

the other consideration, we do not want data to sit in the cache "too long", so the 
cache is flushed every 1 second or so.

all this cache stuff could be completely removed, deleted. result would be MIDAS that 
works ok for small data sizes and rates, but completely falls down at 10 Gige speeds and 
rates.

P.S. why is high semaphore locking rate bad? it turns out that UNIX and Linux semaphores 
are not "fair", they do not give equal share to all users, and (for example) an event 
buffer writer can "capture" the semaphore so the buffer reader (mlogger) will never get 
it, a pathologic situation (to help with this, there is also a "read cache"). Read this 
discussion: https://stackoverflow.com/questions/17825508/fairness-setting-in-semaphore-
class

K.O.

I think I added the cache size correctly:

  {"Trigger",               /* equipment name */
      {1, 0,                 /* event ID, trigger mask */
         "SYSTEM",           /* event buffer */
         EQ_POLLED,          /* equipment type */
         0,                  /* event source */
         "MIDAS",            /* format */
         TRUE,               /* enabled */
         RO_RUNNING |        /* read only when running */
         RO_ODB,             /* and update ODB */
         100,                /* poll for 100ms */
         0,                  /* stop run after this event limit */
         0,                  /* number of sub events */
         0,                  /* don't log history */
         "", "", "", "", "", // frontend_host, name, file_name, status, status_color
         0, // hidden
         0  // write_cache_size <<--------------------- set this to zero -----------
      },
   }

K.O.

> I have a manalyzer that uses a derived class of TMFeRpcHandlerInterface to communicate information to 
> Midas during online running.  At the end of each run it saves out custom data in the 
> TMFeRpcHandlerInterface::HandleEndRun override. This works really well.
> However, when I run offline on a Midas output file the HandleEndRun method is never called and my data is 
> never saved.  Is this intentional?  I understand that there is no point for the HandleBinaryRpc method offline, 
> but the other methods (HandleEndRun, HandleBeginRun etc) could serve a purpose.  Or is it a conscious 
> choice to ignore all of TMFeRpcHandlerInterface when offline?

apologies for delayed response.

I saw the question, completely did not understand it, only now got around to figure out what is going on.

according to manalyzer/README.md, section "manalyzer module and object life time", BeginRun() and EndRun() is called 
always. Offline and online. What you see would be a bug that we do not see in our environment. I confirm this by 
running manalyzer in a demo mode: ./bin/manalyzer_test.exe  --demo -e10 -t

no, wait, you say you use HandleBeginRun() and HandleEndRun(). this is not right, they are not part of the manalyzer 
API and they indeed are only used when running online.

correct solution would be to use BeginRun() and EndRun() instead of HandleBeginRun() and HandleEndRun().

you could also save your data in the module destructor (although good programming recommendation is to use 
destructor only for unavoidable things, like freeing memory, etc).

K.O.

> After some iterations, we merged the branch with the new build scheme.

the commit to implement this change in the manalyzer was not pushed, for reasons unknown.

fixed, commit f2b4dc87ca4830f6bed8667d6a4ee4afd6d242a1

K.O.

> currently to link Midas to project one has to do several steps ...

this information is incorrect. please read https://daq00.triumf.ca/elog-midas/Midas/2258

a very simple way to use link MIDAS using midas-targets.cmake has been implemented a long time ago.

before proposing a new way of doing things, it would be nice to hear about shortcomings
of the existing stuff. A simple "Konstantin's way sucks" or "this is not the cmake way!"
would have been sufficient.

K.O.

Some time ago Ben Smith added test coverage reports using GCC -fprofile-arcs -
ftest-coverage and lcov.

It reports code coverage after running "make test". Reported code coverage is 
surprisingly high for the very little code executed by "make test" - create ODB, 
start a frontend,  start run, stop run, check that mlogger created data files 
and history files.

(Of course coverage can never reach 100%, some obscure branches are unreachable 
without using an automated fuzzer, and some error paths are unreachable without 
also using a system call fault injector).

Simplest way to generate a coverage report:

make clean
make cmake YES_COVERAGE=1
make coverage
open cov_html/index.html

K.O.

bitbucket automatic builds were broken after mfe.cxx started printing some additional messages added in commit
https://bitbucket.org/tmidas/midas/commits/0ae08cd3b96ebd8e4f57bfe00dd45527d82d7a38

this is now fixed. to check if your changes will break automatic builds, before final push, please do:

make clean
make mini -j
make cmake -j
make test

K.O.

I am getting a scary compiler warning from MSCB code. I generally avoid touching MSCB code because I have no MSCB hardware to test it, so I am 
asking the persons unnamed who wrote this code to fix it. Stock g++ on Ubuntu LTS 24.04. Thanks in advance!

In function ‘ssize_t read(int, void*, size_t)’,
    inlined from ‘int mscb_interprete_file(const char*, unsigned char**, unsigned int*, unsigned char**, unsigned int*, unsigned char*)’ at 
/home/olchansk/git/midas/mscb/src/mscb.cxx:2666:13:
/usr/include/x86_64-linux-gnu/bits/unistd.h:28:10: warning: ‘ssize_t __read_alias(int, void*, size_t)’ specified size 18446744073709551614 
exceeds maximum object size 9223372036854775807 [-Wstringop-overflow=]
   28 |   return __glibc_fortify (read, __nbytes, sizeof (char),
      |          ^~~~~~~~~~~~~~~
/usr/include/x86_64-linux-gnu/bits/unistd-decl.h: In function ‘int mscb_interprete_file(const char*, unsigned char**, unsigned int*, unsigned 
char**, unsigned int*, unsigned char*)’:
/usr/include/x86_64-linux-gnu/bits/unistd-decl.h:29:16: note: in a call to function ‘ssize_t __read_alias(int, void*, size_t)’ declared with 
attribute ‘access (write_only, 2, 3)’
   29 | extern ssize_t __REDIRECT_FORTIFY (__read_alias, (int __fd, void *__buf,
      |                ^~~~~~~~~~~~~~~~~~

K.O.

Unnamed person who added this clever bit of c++ coding, please fix this compiler warning. Stock g++ on Ubuntu LTS 24.04. Thanks in advance!

/home/olchansk/git/midas/src/system.cxx: In function ‘std::string ss_execs(const char*)’:
/home/olchansk/git/midas/src/system.cxx:2256:43: warning: ignoring attributes on template argument ‘int (*)(FILE*)’ [-Wignored-attributes]
 2256 |    std::unique_ptr<FILE, decltype(&pclose)> pipe(popen(cmd, "r"), pclose);

K.O.

> ALL MIDAS CLIENTS GET RE-COMPILED after the new code is applied.

Usually we expect that it is "safe" to update to the latest version of MIDAS.

In the sense that we do not have to track down every single frontend
and rebuild it. We have several experiments where tracking down the source code
and rebuilding a frontend takes more than 5 seconds. In many cases these are
10 year old executables that worked just fine through many updates of MIDAS
without having to rebuild them.

So any binary incompatible change is best avoided and must be clearly announced.

The present binary-incompatible change is this commit:
https://bitbucket.org/tmidas/midas/commits/5899f1657ba31121c9f420a824b3c6c13b173988

I tagged the last commit before this change as: midas-2024-12-a

K.O.

we are migrating the dragon experiment from an old mac to a new mac studio and we ran into a problem 
where one equipment format was set to "fixed" instead of "midas". lots of confusion, mdump crash, 
analyzer crash, etc. (mdump fixes for this are already committed).

it made us think whether equipment format is still needed. in the old days we had choice of MIDAS and 
YBOS formats, but YBOS was removed years ago, and I was surprised that format FIXED was permitted at 
all.

I did a midas source code review, this is what I found:

- remnants of YBOS support in a few places, commit to remove them pending.
- FORMAT_ROOT is used in mlogger for automatic conversion of MIDAS banks to ROOT trees
- FORMAT_FIXED is used in a few slow control drivers in drivers/class, instead of creating MIDAS 
banks, they copy raw data directly into an event (there is no bank header and no way to identify such 
events automatically)
- lots of code to support different formats in mdump (mostly dead code)
- the rest of the code does not care or use this format stuff

Current proposal is to remove support for all formats except FORMAT_MIDAS (and FORMAT_ROOT in 
mlogger).

- defines of FORMAT_XXX will be removed from midas.h
- "Format" will be removed from ODB Equipment/Common
- "Format" will be removed from ODB Logger/Channel
- to maintain binary compatibility, we can keep the "Format" ODB entries, but they will be ignored.

List of slow control drivers that support FORMAT_FIXED:

daq00:midas$ grep FORMAT_FIXED drivers/class/*
drivers/class/cd_fdg.cxx:   if (fgd_info->format == FORMAT_FIXED) {
drivers/class/cd_ivc32.cxx:   if (hv_info->format == XFORMAT_FIXED) {
drivers/class/cd_rf.cxx:	if (rf_info->format == XFORMAT_FIXED) 
drivers/class/generic.cxx:   if (gen_info->format == XFORMAT_FIXED) {
drivers/class/hv.cxx:   if (hv_info->format == XFORMAT_FIXED) {
drivers/class/multi.cxx:   if (m_info->format == XFORMAT_FIXED) {
drivers/class/slowdev.cxx:   if (gen_info->format == XFORMAT_FIXED) {
daq00:midas$ 

K.O.

> Thanks Konstantin. Please send me the felabview code or let me know where I can find it.

https://bitbucket.org/expalpha/a2daq/src/alpha/src/felabview.cxx

this is code circa 2006, there are now better ways to do some of that coding.

if you want bidirectional communication with labview, read/write odb, etc, simplest is probably
to write the "mjserver" that talks midas json-rpc over plain tcp, without all the http/https
gunk you need to go through mhttpd.

K.O.

Thanks Konstantin. Please send me the felabview code or let me know where I can find it.

Zaher

> > Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.
> 
> Yes, in the ALPHA anti-hydrogen experiment at CERN we have been doing this since 2006.
> 
> Original system is very simple, labview side opens a TCP socket to the MIDAS felabview frontend
> and sends the numeric data as an ASCII string. The first four chars of the data is the name
> of the MIDAS data bank, second number is the data timestamp in seconds.
> 
> LCRY 1234567 1.1 2.2 3.3
> 
> A newer iteration is feGEM written by Joseph McKenna (member of this forum), it uses a more sophisticated
> labview component. Please contact him directly for more information.
> 
> I can provide you with the source code for my original felabiew (pretty much unchanged from circa 2006).
> 
> K.O.