> I have a manalyzer that uses a derived class of TMFeRpcHandlerInterface to communicate information to 
> Midas during online running.  At the end of each run it saves out custom data in the 
> TMFeRpcHandlerInterface::HandleEndRun override. This works really well.
> However, when I run offline on a Midas output file the HandleEndRun method is never called and my data is 
> never saved.  Is this intentional?  I understand that there is no point for the HandleBinaryRpc method offline, 
> but the other methods (HandleEndRun, HandleBeginRun etc) could serve a purpose.  Or is it a conscious 
> choice to ignore all of TMFeRpcHandlerInterface when offline?

apologies for delayed response.

I saw the question, completely did not understand it, only now got around to figure out what is going on.

according to manalyzer/README.md, section "manalyzer module and object life time", BeginRun() and EndRun() is called 
always. Offline and online. What you see would be a bug that we do not see in our environment. I confirm this by 
running manalyzer in a demo mode: ./bin/manalyzer_test.exe  --demo -e10 -t

no, wait, you say you use HandleBeginRun() and HandleEndRun(). this is not right, they are not part of the manalyzer 
API and they indeed are only used when running online.

correct solution would be to use BeginRun() and EndRun() instead of HandleBeginRun() and HandleEndRun().

you could also save your data in the module destructor (although good programming recommendation is to use 
destructor only for unavoidable things, like freeing memory, etc).

K.O.

> After some iterations, we merged the branch with the new build scheme.

the commit to implement this change in the manalyzer was not pushed, for reasons unknown.

fixed, commit f2b4dc87ca4830f6bed8667d6a4ee4afd6d242a1

K.O.

> currently to link Midas to project one has to do several steps ...

this information is incorrect. please read https://daq00.triumf.ca/elog-midas/Midas/2258

a very simple way to use link MIDAS using midas-targets.cmake has been implemented a long time ago.

before proposing a new way of doing things, it would be nice to hear about shortcomings
of the existing stuff. A simple "Konstantin's way sucks" or "this is not the cmake way!"
would have been sufficient.

K.O.

Some time ago Ben Smith added test coverage reports using GCC -fprofile-arcs -
ftest-coverage and lcov.

It reports code coverage after running "make test". Reported code coverage is 
surprisingly high for the very little code executed by "make test" - create ODB, 
start a frontend,  start run, stop run, check that mlogger created data files 
and history files.

(Of course coverage can never reach 100%, some obscure branches are unreachable 
without using an automated fuzzer, and some error paths are unreachable without 
also using a system call fault injector).

Simplest way to generate a coverage report:

make clean
make cmake YES_COVERAGE=1
make coverage
open cov_html/index.html

K.O.

bitbucket automatic builds were broken after mfe.cxx started printing some additional messages added in commit
https://bitbucket.org/tmidas/midas/commits/0ae08cd3b96ebd8e4f57bfe00dd45527d82d7a38

this is now fixed. to check if your changes will break automatic builds, before final push, please do:

make clean
make mini -j
make cmake -j
make test

K.O.

I am getting a scary compiler warning from MSCB code. I generally avoid touching MSCB code because I have no MSCB hardware to test it, so I am 
asking the persons unnamed who wrote this code to fix it. Stock g++ on Ubuntu LTS 24.04. Thanks in advance!

In function ‘ssize_t read(int, void*, size_t)’,
    inlined from ‘int mscb_interprete_file(const char*, unsigned char**, unsigned int*, unsigned char**, unsigned int*, unsigned char*)’ at 
/home/olchansk/git/midas/mscb/src/mscb.cxx:2666:13:
/usr/include/x86_64-linux-gnu/bits/unistd.h:28:10: warning: ‘ssize_t __read_alias(int, void*, size_t)’ specified size 18446744073709551614 
exceeds maximum object size 9223372036854775807 [-Wstringop-overflow=]
   28 |   return __glibc_fortify (read, __nbytes, sizeof (char),
      |          ^~~~~~~~~~~~~~~
/usr/include/x86_64-linux-gnu/bits/unistd-decl.h: In function ‘int mscb_interprete_file(const char*, unsigned char**, unsigned int*, unsigned 
char**, unsigned int*, unsigned char*)’:
/usr/include/x86_64-linux-gnu/bits/unistd-decl.h:29:16: note: in a call to function ‘ssize_t __read_alias(int, void*, size_t)’ declared with 
attribute ‘access (write_only, 2, 3)’
   29 | extern ssize_t __REDIRECT_FORTIFY (__read_alias, (int __fd, void *__buf,
      |                ^~~~~~~~~~~~~~~~~~

K.O.

Unnamed person who added this clever bit of c++ coding, please fix this compiler warning. Stock g++ on Ubuntu LTS 24.04. Thanks in advance!

/home/olchansk/git/midas/src/system.cxx: In function ‘std::string ss_execs(const char*)’:
/home/olchansk/git/midas/src/system.cxx:2256:43: warning: ignoring attributes on template argument ‘int (*)(FILE*)’ [-Wignored-attributes]
 2256 |    std::unique_ptr<FILE, decltype(&pclose)> pipe(popen(cmd, "r"), pclose);

K.O.

> ALL MIDAS CLIENTS GET RE-COMPILED after the new code is applied.

Usually we expect that it is "safe" to update to the latest version of MIDAS.

In the sense that we do not have to track down every single frontend
and rebuild it. We have several experiments where tracking down the source code
and rebuilding a frontend takes more than 5 seconds. In many cases these are
10 year old executables that worked just fine through many updates of MIDAS
without having to rebuild them.

So any binary incompatible change is best avoided and must be clearly announced.

The present binary-incompatible change is this commit:
https://bitbucket.org/tmidas/midas/commits/5899f1657ba31121c9f420a824b3c6c13b173988

I tagged the last commit before this change as: midas-2024-12-a

K.O.

we are migrating the dragon experiment from an old mac to a new mac studio and we ran into a problem 
where one equipment format was set to "fixed" instead of "midas". lots of confusion, mdump crash, 
analyzer crash, etc. (mdump fixes for this are already committed).

it made us think whether equipment format is still needed. in the old days we had choice of MIDAS and 
YBOS formats, but YBOS was removed years ago, and I was surprised that format FIXED was permitted at 
all.

I did a midas source code review, this is what I found:

- remnants of YBOS support in a few places, commit to remove them pending.
- FORMAT_ROOT is used in mlogger for automatic conversion of MIDAS banks to ROOT trees
- FORMAT_FIXED is used in a few slow control drivers in drivers/class, instead of creating MIDAS 
banks, they copy raw data directly into an event (there is no bank header and no way to identify such 
events automatically)
- lots of code to support different formats in mdump (mostly dead code)
- the rest of the code does not care or use this format stuff

Current proposal is to remove support for all formats except FORMAT_MIDAS (and FORMAT_ROOT in 
mlogger).

- defines of FORMAT_XXX will be removed from midas.h
- "Format" will be removed from ODB Equipment/Common
- "Format" will be removed from ODB Logger/Channel
- to maintain binary compatibility, we can keep the "Format" ODB entries, but they will be ignored.

List of slow control drivers that support FORMAT_FIXED:

daq00:midas$ grep FORMAT_FIXED drivers/class/*
drivers/class/cd_fdg.cxx:   if (fgd_info->format == FORMAT_FIXED) {
drivers/class/cd_ivc32.cxx:   if (hv_info->format == XFORMAT_FIXED) {
drivers/class/cd_rf.cxx:	if (rf_info->format == XFORMAT_FIXED) 
drivers/class/generic.cxx:   if (gen_info->format == XFORMAT_FIXED) {
drivers/class/hv.cxx:   if (hv_info->format == XFORMAT_FIXED) {
drivers/class/multi.cxx:   if (m_info->format == XFORMAT_FIXED) {
drivers/class/slowdev.cxx:   if (gen_info->format == XFORMAT_FIXED) {
daq00:midas$ 

K.O.

> Thanks Konstantin. Please send me the felabview code or let me know where I can find it.

https://bitbucket.org/expalpha/a2daq/src/alpha/src/felabview.cxx

this is code circa 2006, there are now better ways to do some of that coding.

if you want bidirectional communication with labview, read/write odb, etc, simplest is probably
to write the "mjserver" that talks midas json-rpc over plain tcp, without all the http/https
gunk you need to go through mhttpd.

K.O.

Thanks Konstantin. Please send me the felabview code or let me know where I can find it.

Zaher

> > Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.
> 
> Yes, in the ALPHA anti-hydrogen experiment at CERN we have been doing this since 2006.
> 
> Original system is very simple, labview side opens a TCP socket to the MIDAS felabview frontend
> and sends the numeric data as an ASCII string. The first four chars of the data is the name
> of the MIDAS data bank, second number is the data timestamp in seconds.
> 
> LCRY 1234567 1.1 2.2 3.3
> 
> A newer iteration is feGEM written by Joseph McKenna (member of this forum), it uses a more sophisticated
> labview component. Please contact him directly for more information.
> 
> I can provide you with the source code for my original felabiew (pretty much unchanged from circa 2006).
> 
> K.O.

> beta version of macOS: Switching beta updates off fixed the issue.

I would be very surprised if that was the problem.

Bigger concern is that it fails without producing any useful error message.

Latest MacOS makes it extremely difficult to debug this kind of stuff, there
is several hoops to jump through to enable core dumps and to allow lldb
to attach and debug running programs.

K.O.

> Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.

Yes, in the ALPHA anti-hydrogen experiment at CERN we have been doing this since 2006.

Original system is very simple, labview side opens a TCP socket to the MIDAS felabview frontend
and sends the numeric data as an ASCII string. The first four chars of the data is the name
of the MIDAS data bank, second number is the data timestamp in seconds.

LCRY 1234567 1.1 2.2 3.3

A newer iteration is feGEM written by Joseph McKenna (member of this forum), it uses a more sophisticated
labview component. Please contact him directly for more information.

I can provide you with the source code for my original felabiew (pretty much unchanged from circa 2006).

K.O.

We have a problem with over aggressive AI bots. They are scanning everything and 
are putting a crazy load on the mediawiki mysql database. This makes all out 
wikis very slow and unresponsive, which is a big problem.

These AI scanners do not seem to know what they are doing and are trying to load 
all the special pages, like "what links here" and "recent changes" and complete 
revision histories for each page. I am not impressed. Old school google and bing 
scanner bots are much more respectful and do not cause problems.

AI bots are also scanning this elog, and to Stefan's credit elogd is holding the 
load just fine.

To reduce load on the MidasWiki mysql database, I now restricted access to most 
special pages to logged in users. It seems to have helped.

If this causes big difficulty or if you have suggestion on better ways to deal 
with aggressive AI scanner bots, please speak up.

One alternative is to put MidasWiki behind a generic login page with a well 
known password. Downside is it will block google and bing search engines, too.

K.O.

Hello,

Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.

thanks,
Zaher

>
> However right after running these commands I removed a .SHM_HOST.TXT file
>

Instead of deleting .SHM_HOSTS.TXT, please create it as an empty file. I thought the documentation is clear about it?

Also we recommend installing MIDAS to $HOME/packages/midas. There is a number of problems if installed at top level.

If you want to be compliant with the Linux LFS, /opt/midas is also a good place.

> 
> ... and it suddenly worked!
>

We still did not establish if mhist, mhdump and the other commands I sent you work correctly,
to confirm MIDAS is creating correct history files. (before you try to read them with python).

Also we did not establish that you have correct paths setup in ODB /Logger/History.

Many things can go wrong.

Next time python history malfunctions, please do all those other things and report to us. Thanks!

K.O.

> Unfortunately I again cannot reproduce this:
> 
> $ python ~/DAQ/midas_latest/python/examples/basic_hist_script.py
> Valid events are:
> * Run transitions
> * test_slow/data
> Enter event name: test_slow/data
> Valid tags for test_slow/data are:
> * data
> Enter tag name: data
> Event/tag test_slow/data/data has 1 elements
> How many hours: 1
> Interval in seconds: 1
> 78 entries found
> 2025/03/17 17:00:56 => 98.097391
> 2025/03/17 17:00:57 => 98.982151
> 2025/03/17 17:00:58 => 99.589187
> 2025/03/17 17:00:59 => 99.926821
> 2025/03/17 17:01:00 => 99.989878
> 2025/03/17 17:01:01 => 99.778216
> 2025/03/17 17:01:02 => 99.292485
> .......
> 
> 
> I want to narrow down whether the issue is in the basic_hist_script.py or the lower-level code. So there are a few steps of debugging to do.
> 
> 
> 
> 1) Run code directly in the python interpreter: 
> 
> Can you run the following and send the output please?
> 
> ```
> import midas.client
> c = midas.client.MidasClient("history_test")
> data = c.hist_get_recent_data(1,1,"test_slow/data","data")
> print(f"event_name='{data[0]['event_name']}', tag_name='{data[0]['tag_name']}', num_entries={data[0]['num_entries']}, status={data[0]['status']}, arrlen={len(data[0]['values'])}")
> ```
> 
> For me, I get:
> event_name='test_slow/data', tag_name='data', num_entries=441, status=1, arrlen=441
> 
> 
> 
> 2) If things look sensible for you (status=1, non-zero num_entries), then the problem is in the basic_hist_script.py. Can you add the same print() statement in basic_hist_script.py immediately after the call to hist_get_recent_data(), then run that script again and send the output of that?
> 
> 
> 
> 3) Debug the python/C conversions.
> 
> In midas/client.py add the following line to hist_get_data() immediately before the call to self.lib.c_hs_read():
> 
> ```
>         print(f"c_start_time={c_start_time.value}, c_end_time={c_end_time.value}, c_interval={c_interval.value}, c_event_name={c_event_name.value}, c_tag_name={c_tag_name.value}")
> ```
> 
> Then run the following and send the output:
> 
> ```
> import midas.client
> c = midas.client.MidasClient("history_test")
> data = c.hist_get_recent_data(1,1,"test_slow/data","data")
> ```
> 
> For me, I get:
> c_start_time=1742254428, c_end_time=1742258028, c_interval=1, c_event_name=b'test_slow/data', c_tag_name=b'data'
> 
> I want to check that the UNIX timestamps match what you expect for your server, and that nothing weird is going on with the python/C string conversions.
> 
> 
> Thanks,
> Ben

Hi, thank you for the support!

Running the commands on the Macbook pro leads to
1)
event_name='test_slow/data', tag_name='data', num_entries=0, status=1, arrlen=0
2)
The number of entries is zero
3)
I get
c_start_time=1742275653, c_end_time=1742279253, c_interval=1, c_event_name=b'test_slow/data', c_tag_name=b'data'

However right after running these commands I removed a .SHM_HOST.TXT file
(due to me working both at home and at PSI, my computer hostname changes when I switch the network, so I remove .SHM_HOST.TXT to be able to run experiments)

and reran the code, and it suddenly worked! This is good, but I do not know what fixed it... I had done more extensive tests yesterday and also had to delete .SHM_HOST.TXT multiple times, to no avail.
Do you have any ideas as to what could be happening? Similarly to my previous bug report, which was fixed by updating macOS to a more stable version, could this have been due to an automatic update?

If the problem still persists on the Windows machine I will post an update.

> Unfortunately I again cannot reproduce this:
> 
> $ python ~/DAQ/midas_latest/python/examples/basic_hist_script.py
> Valid events are:
> * Run transitions
> * test_slow/data
> Enter event name: test_slow/data
> Valid tags for test_slow/data are:
> * data
> Enter tag name: data
> Event/tag test_slow/data/data has 1 elements
> How many hours: 1
> Interval in seconds: 1
> 78 entries found
> 2025/03/17 17:00:56 => 98.097391
> 2025/03/17 17:00:57 => 98.982151
> 2025/03/17 17:00:58 => 99.589187
> 2025/03/17 17:00:59 => 99.926821
> 2025/03/17 17:01:00 => 99.989878
> 2025/03/17 17:01:01 => 99.778216
> 2025/03/17 17:01:02 => 99.292485
> .......
> 
> 
> I want to narrow down whether the issue is in the basic_hist_script.py or the lower-level code. So there are a few steps of debugging to do.
> 
> 
> 
> 1) Run code directly in the python interpreter: 
> 
> Can you run the following and send the output please?
> 
> ```
> import midas.client
> c = midas.client.MidasClient("history_test")
> data = c.hist_get_recent_data(1,1,"test_slow/data","data")
> print(f"event_name='{data[0]['event_name']}', tag_name='{data[0]['tag_name']}', num_entries={data[0]['num_entries']}, status={data[0]['status']}, arrlen={len(data[0]['values'])}")
> ```
> 
> For me, I get:
> event_name='test_slow/data', tag_name='data', num_entries=441, status=1, arrlen=441
> 
> 
> 
> 2) If things look sensible for you (status=1, non-zero num_entries), then the problem is in the basic_hist_script.py. Can you add the same print() statement in basic_hist_script.py immediately after the call to hist_get_recent_data(), then run that script again and send the output of that?
> 
> 
> 
> 3) Debug the python/C conversions.
> 
> In midas/client.py add the following line to hist_get_data() immediately before the call to self.lib.c_hs_read():
> 
> ```
>         print(f"c_start_time={c_start_time.value}, c_end_time={c_end_time.value}, c_interval={c_interval.value}, c_event_name={c_event_name.value}, c_tag_name={c_tag_name.value}")
> ```
> 
> Then run the following and send the output:
> 
> ```
> import midas.client
> c = midas.client.MidasClient("history_test")
> data = c.hist_get_recent_data(1,1,"test_slow/data","data")
> ```
> 
> For me, I get:
> c_start_time=1742254428, c_end_time=1742258028, c_interval=1, c_event_name=b'test_slow/data', c_tag_name=b'data'
> 
> I want to check that the UNIX timestamps match what you expect for your server, and that nothing weird is going on with the python/C string conversions.
> 
> 
> Thanks,
> Ben

Hi, thank you for the support!

1)
event_name='test_slow/data', tag_name='data', num_entries=0, status=1, arrlen=0
2)
The number of entries is zero
3)
I get
c_start_time=1742275653, c_end_time=1742279253, c_interval=1, c_event_name=b'test_slow/data', c_tag_name=b'data'

Unfortunately I again cannot reproduce this:

$ python ~/DAQ/midas_latest/python/examples/basic_hist_script.py
Valid events are:
* Run transitions
* test_slow/data
Enter event name: test_slow/data
Valid tags for test_slow/data are:
* data
Enter tag name: data
Event/tag test_slow/data/data has 1 elements
How many hours: 1
Interval in seconds: 1
78 entries found
2025/03/17 17:00:56 => 98.097391
2025/03/17 17:00:57 => 98.982151
2025/03/17 17:00:58 => 99.589187
2025/03/17 17:00:59 => 99.926821
2025/03/17 17:01:00 => 99.989878
2025/03/17 17:01:01 => 99.778216
2025/03/17 17:01:02 => 99.292485
.......


I want to narrow down whether the issue is in the basic_hist_script.py or the lower-level code. So there are a few steps of debugging to do.



1) Run code directly in the python interpreter: 

Can you run the following and send the output please?

```
import midas.client
c = midas.client.MidasClient("history_test")
data = c.hist_get_recent_data(1,1,"test_slow/data","data")
print(f"event_name='{data[0]['event_name']}', tag_name='{data[0]['tag_name']}', num_entries={data[0]['num_entries']}, status={data[0]['status']}, arrlen={len(data[0]['values'])}")
```

For me, I get:
event_name='test_slow/data', tag_name='data', num_entries=441, status=1, arrlen=441



2) If things look sensible for you (status=1, non-zero num_entries), then the problem is in the basic_hist_script.py. Can you add the same print() statement in basic_hist_script.py immediately after the call to hist_get_recent_data(), then run that script again and send the output of that?



3) Debug the python/C conversions.

In midas/client.py add the following line to hist_get_data() immediately before the call to self.lib.c_hs_read():

```
        print(f"c_start_time={c_start_time.value}, c_end_time={c_end_time.value}, c_interval={c_interval.value}, c_event_name={c_event_name.value}, c_tag_name={c_tag_name.value}")
```

Then run the following and send the output:

```
import midas.client
c = midas.client.MidasClient("history_test")
data = c.hist_get_recent_data(1,1,"test_slow/data","data")
```

For me, I get:
c_start_time=1742254428, c_end_time=1742258028, c_interval=1, c_event_name=b'test_slow/data', c_tag_name=b'data'

I want to check that the UNIX timestamps match what you expect for your server, and that nothing weird is going on with the python/C string conversions.


Thanks,
Ben

Setup:
setting up midas, starting mhttpd and mlogger and running fetest.

The History page and the javascript mjsonrpc client are both able to fetch historical data for test_slow/data. Javascript code used is included here:

mjsonrpc_call(
  "hs_read_arraybuffer",
  {
    start_time: Math.floor((new Date()).getTime() /1000) - 1000,
    end_time: Math.floor((new Date()).getTime() /1000),
    events: ["test_slow/data"],
    tags: ["data"],
    index: [0],
  },
  "arraybuffer"
).then(console.log)

However, the python client does not find any valid events:

Setup:
An exptab is created and the environment variables MIDAS_EXPTAB and MIDAS_EXPT_NAME and MIDASSYS are set (together with the correct PATH)

Running /midas/python/examples/basic_hist_script.py and typing in data:

Valid events are:
* Run transitions
* rrandom/SLOW
* test_slow/data
Enter event name: test_slow/data
Valid tags for test_slow/data are:
* data
Enter tag name: data
Event/tag test_slow/data/data has 1 elements
How many hours: 1
Interval in seconds: 1 # other values were also tested, without success
0 entries found

We expect entries to be found, however do not.

Tested setups:
Macbook Pro Sequoia 15.3 with Python 3.13.2, ROOT latest, midas bitbucket commit 84c7ef7
Windows 11 with Python 3.11, ROOT latest, midas latest commit (development branch)