> beta version of macOS: Switching beta updates off fixed the issue.

I would be very surprised if that was the problem.

Bigger concern is that it fails without producing any useful error message.

Latest MacOS makes it extremely difficult to debug this kind of stuff, there
is several hoops to jump through to enable core dumps and to allow lldb
to attach and debug running programs.

K.O.

> Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.

Yes, in the ALPHA anti-hydrogen experiment at CERN we have been doing this since 2006.

Original system is very simple, labview side opens a TCP socket to the MIDAS felabview frontend
and sends the numeric data as an ASCII string. The first four chars of the data is the name
of the MIDAS data bank, second number is the data timestamp in seconds.

LCRY 1234567 1.1 2.2 3.3

A newer iteration is feGEM written by Joseph McKenna (member of this forum), it uses a more sophisticated
labview component. Please contact him directly for more information.

I can provide you with the source code for my original felabiew (pretty much unchanged from circa 2006).

K.O.

We have a problem with over aggressive AI bots. They are scanning everything and 
are putting a crazy load on the mediawiki mysql database. This makes all out 
wikis very slow and unresponsive, which is a big problem.

These AI scanners do not seem to know what they are doing and are trying to load 
all the special pages, like "what links here" and "recent changes" and complete 
revision histories for each page. I am not impressed. Old school google and bing 
scanner bots are much more respectful and do not cause problems.

AI bots are also scanning this elog, and to Stefan's credit elogd is holding the 
load just fine.

To reduce load on the MidasWiki mysql database, I now restricted access to most 
special pages to logged in users. It seems to have helped.

If this causes big difficulty or if you have suggestion on better ways to deal 
with aggressive AI scanner bots, please speak up.

One alternative is to put MidasWiki behind a generic login page with a well 
known password. Downside is it will block google and bing search engines, too.

K.O.

Hello,

Does anyone have experience with writing a MIDAS frontends to communicate with a device that operates using LabView (e.g. superconducting magnets, cryostats etc.). Any information or experience regarding this would be highly appreciated.

thanks,
Zaher

>
> However right after running these commands I removed a .SHM_HOST.TXT file
>

Instead of deleting .SHM_HOSTS.TXT, please create it as an empty file. I thought the documentation is clear about it?

Also we recommend installing MIDAS to $HOME/packages/midas. There is a number of problems if installed at top level.

If you want to be compliant with the Linux LFS, /opt/midas is also a good place.

> 
> ... and it suddenly worked!
>

We still did not establish if mhist, mhdump and the other commands I sent you work correctly,
to confirm MIDAS is creating correct history files. (before you try to read them with python).

Also we did not establish that you have correct paths setup in ODB /Logger/History.

Many things can go wrong.

Next time python history malfunctions, please do all those other things and report to us. Thanks!

K.O.

> Unfortunately I again cannot reproduce this:
> 
> $ python ~/DAQ/midas_latest/python/examples/basic_hist_script.py
> Valid events are:
> * Run transitions
> * test_slow/data
> Enter event name: test_slow/data
> Valid tags for test_slow/data are:
> * data
> Enter tag name: data
> Event/tag test_slow/data/data has 1 elements
> How many hours: 1
> Interval in seconds: 1
> 78 entries found
> 2025/03/17 17:00:56 => 98.097391
> 2025/03/17 17:00:57 => 98.982151
> 2025/03/17 17:00:58 => 99.589187
> 2025/03/17 17:00:59 => 99.926821
> 2025/03/17 17:01:00 => 99.989878
> 2025/03/17 17:01:01 => 99.778216
> 2025/03/17 17:01:02 => 99.292485
> .......
> 
> 
> I want to narrow down whether the issue is in the basic_hist_script.py or the lower-level code. So there are a few steps of debugging to do.
> 
> 
> 
> 1) Run code directly in the python interpreter: 
> 
> Can you run the following and send the output please?
> 
> ```
> import midas.client
> c = midas.client.MidasClient("history_test")
> data = c.hist_get_recent_data(1,1,"test_slow/data","data")
> print(f"event_name='{data[0]['event_name']}', tag_name='{data[0]['tag_name']}', num_entries={data[0]['num_entries']}, status={data[0]['status']}, arrlen={len(data[0]['values'])}")
> ```
> 
> For me, I get:
> event_name='test_slow/data', tag_name='data', num_entries=441, status=1, arrlen=441
> 
> 
> 
> 2) If things look sensible for you (status=1, non-zero num_entries), then the problem is in the basic_hist_script.py. Can you add the same print() statement in basic_hist_script.py immediately after the call to hist_get_recent_data(), then run that script again and send the output of that?
> 
> 
> 
> 3) Debug the python/C conversions.
> 
> In midas/client.py add the following line to hist_get_data() immediately before the call to self.lib.c_hs_read():
> 
> ```
>         print(f"c_start_time={c_start_time.value}, c_end_time={c_end_time.value}, c_interval={c_interval.value}, c_event_name={c_event_name.value}, c_tag_name={c_tag_name.value}")
> ```
> 
> Then run the following and send the output:
> 
> ```
> import midas.client
> c = midas.client.MidasClient("history_test")
> data = c.hist_get_recent_data(1,1,"test_slow/data","data")
> ```
> 
> For me, I get:
> c_start_time=1742254428, c_end_time=1742258028, c_interval=1, c_event_name=b'test_slow/data', c_tag_name=b'data'
> 
> I want to check that the UNIX timestamps match what you expect for your server, and that nothing weird is going on with the python/C string conversions.
> 
> 
> Thanks,
> Ben

Hi, thank you for the support!

Running the commands on the Macbook pro leads to
1)
event_name='test_slow/data', tag_name='data', num_entries=0, status=1, arrlen=0
2)
The number of entries is zero
3)
I get
c_start_time=1742275653, c_end_time=1742279253, c_interval=1, c_event_name=b'test_slow/data', c_tag_name=b'data'

However right after running these commands I removed a .SHM_HOST.TXT file
(due to me working both at home and at PSI, my computer hostname changes when I switch the network, so I remove .SHM_HOST.TXT to be able to run experiments)

and reran the code, and it suddenly worked! This is good, but I do not know what fixed it... I had done more extensive tests yesterday and also had to delete .SHM_HOST.TXT multiple times, to no avail.
Do you have any ideas as to what could be happening? Similarly to my previous bug report, which was fixed by updating macOS to a more stable version, could this have been due to an automatic update?

If the problem still persists on the Windows machine I will post an update.

> Unfortunately I again cannot reproduce this:
> 
> $ python ~/DAQ/midas_latest/python/examples/basic_hist_script.py
> Valid events are:
> * Run transitions
> * test_slow/data
> Enter event name: test_slow/data
> Valid tags for test_slow/data are:
> * data
> Enter tag name: data
> Event/tag test_slow/data/data has 1 elements
> How many hours: 1
> Interval in seconds: 1
> 78 entries found
> 2025/03/17 17:00:56 => 98.097391
> 2025/03/17 17:00:57 => 98.982151
> 2025/03/17 17:00:58 => 99.589187
> 2025/03/17 17:00:59 => 99.926821
> 2025/03/17 17:01:00 => 99.989878
> 2025/03/17 17:01:01 => 99.778216
> 2025/03/17 17:01:02 => 99.292485
> .......
> 
> 
> I want to narrow down whether the issue is in the basic_hist_script.py or the lower-level code. So there are a few steps of debugging to do.
> 
> 
> 
> 1) Run code directly in the python interpreter: 
> 
> Can you run the following and send the output please?
> 
> ```
> import midas.client
> c = midas.client.MidasClient("history_test")
> data = c.hist_get_recent_data(1,1,"test_slow/data","data")
> print(f"event_name='{data[0]['event_name']}', tag_name='{data[0]['tag_name']}', num_entries={data[0]['num_entries']}, status={data[0]['status']}, arrlen={len(data[0]['values'])}")
> ```
> 
> For me, I get:
> event_name='test_slow/data', tag_name='data', num_entries=441, status=1, arrlen=441
> 
> 
> 
> 2) If things look sensible for you (status=1, non-zero num_entries), then the problem is in the basic_hist_script.py. Can you add the same print() statement in basic_hist_script.py immediately after the call to hist_get_recent_data(), then run that script again and send the output of that?
> 
> 
> 
> 3) Debug the python/C conversions.
> 
> In midas/client.py add the following line to hist_get_data() immediately before the call to self.lib.c_hs_read():
> 
> ```
>         print(f"c_start_time={c_start_time.value}, c_end_time={c_end_time.value}, c_interval={c_interval.value}, c_event_name={c_event_name.value}, c_tag_name={c_tag_name.value}")
> ```
> 
> Then run the following and send the output:
> 
> ```
> import midas.client
> c = midas.client.MidasClient("history_test")
> data = c.hist_get_recent_data(1,1,"test_slow/data","data")
> ```
> 
> For me, I get:
> c_start_time=1742254428, c_end_time=1742258028, c_interval=1, c_event_name=b'test_slow/data', c_tag_name=b'data'
> 
> I want to check that the UNIX timestamps match what you expect for your server, and that nothing weird is going on with the python/C string conversions.
> 
> 
> Thanks,
> Ben

Hi, thank you for the support!

1)
event_name='test_slow/data', tag_name='data', num_entries=0, status=1, arrlen=0
2)
The number of entries is zero
3)
I get
c_start_time=1742275653, c_end_time=1742279253, c_interval=1, c_event_name=b'test_slow/data', c_tag_name=b'data'

Unfortunately I again cannot reproduce this:

$ python ~/DAQ/midas_latest/python/examples/basic_hist_script.py
Valid events are:
* Run transitions
* test_slow/data
Enter event name: test_slow/data
Valid tags for test_slow/data are:
* data
Enter tag name: data
Event/tag test_slow/data/data has 1 elements
How many hours: 1
Interval in seconds: 1
78 entries found
2025/03/17 17:00:56 => 98.097391
2025/03/17 17:00:57 => 98.982151
2025/03/17 17:00:58 => 99.589187
2025/03/17 17:00:59 => 99.926821
2025/03/17 17:01:00 => 99.989878
2025/03/17 17:01:01 => 99.778216
2025/03/17 17:01:02 => 99.292485
.......


I want to narrow down whether the issue is in the basic_hist_script.py or the lower-level code. So there are a few steps of debugging to do.



1) Run code directly in the python interpreter: 

Can you run the following and send the output please?

```
import midas.client
c = midas.client.MidasClient("history_test")
data = c.hist_get_recent_data(1,1,"test_slow/data","data")
print(f"event_name='{data[0]['event_name']}', tag_name='{data[0]['tag_name']}', num_entries={data[0]['num_entries']}, status={data[0]['status']}, arrlen={len(data[0]['values'])}")
```

For me, I get:
event_name='test_slow/data', tag_name='data', num_entries=441, status=1, arrlen=441



2) If things look sensible for you (status=1, non-zero num_entries), then the problem is in the basic_hist_script.py. Can you add the same print() statement in basic_hist_script.py immediately after the call to hist_get_recent_data(), then run that script again and send the output of that?



3) Debug the python/C conversions.

In midas/client.py add the following line to hist_get_data() immediately before the call to self.lib.c_hs_read():

```
        print(f"c_start_time={c_start_time.value}, c_end_time={c_end_time.value}, c_interval={c_interval.value}, c_event_name={c_event_name.value}, c_tag_name={c_tag_name.value}")
```

Then run the following and send the output:

```
import midas.client
c = midas.client.MidasClient("history_test")
data = c.hist_get_recent_data(1,1,"test_slow/data","data")
```

For me, I get:
c_start_time=1742254428, c_end_time=1742258028, c_interval=1, c_event_name=b'test_slow/data', c_tag_name=b'data'

I want to check that the UNIX timestamps match what you expect for your server, and that nothing weird is going on with the python/C string conversions.


Thanks,
Ben

Setup:
setting up midas, starting mhttpd and mlogger and running fetest.

The History page and the javascript mjsonrpc client are both able to fetch historical data for test_slow/data. Javascript code used is included here:

mjsonrpc_call(
  "hs_read_arraybuffer",
  {
    start_time: Math.floor((new Date()).getTime() /1000) - 1000,
    end_time: Math.floor((new Date()).getTime() /1000),
    events: ["test_slow/data"],
    tags: ["data"],
    index: [0],
  },
  "arraybuffer"
).then(console.log)

However, the python client does not find any valid events:

Setup:
An exptab is created and the environment variables MIDAS_EXPTAB and MIDAS_EXPT_NAME and MIDASSYS are set (together with the correct PATH)

Running /midas/python/examples/basic_hist_script.py and typing in data:

Valid events are:
* Run transitions
* rrandom/SLOW
* test_slow/data
Enter event name: test_slow/data
Valid tags for test_slow/data are:
* data
Enter tag name: data
Event/tag test_slow/data/data has 1 elements
How many hours: 1
Interval in seconds: 1 # other values were also tested, without success
0 entries found

We expect entries to be found, however do not.

Tested setups:
Macbook Pro Sequoia 15.3 with Python 3.13.2, ROOT latest, midas bitbucket commit 84c7ef7
Windows 11 with Python 3.11, ROOT latest, midas latest commit (development branch)

> After starting midas (mhttpd &, and mlogger -D) and running the `fetest` frontend I went into the midas/python/examples directory and ran basic_hist_script.py, and, even though I could see the 'pytest' program in the Programs page,
> 
> Valid events are:
> Enter event name:
> 
> was printed out, which signified that no events were found. No errors were displayed.
> 
> Instead, when trying to do the same in javascript (using mjsonrpc_send_request( mjsonrpc_make_request("hs_get_events")).then(console.log)), I was able to get the expected events.
> 
> The History page also displayed the expected data and the plots worked correctly.
> 
> Device info: Chip: Apple M1 Pro, OS: Sequoia (15.3)
> 
> MIDAS version: bitbucket commit 84c7ef7
> 
> Python version: 3.13.2


I tested the command this morning and it worked. The most likely cause of the errors was that I was on a beta version of macOS: Switching beta updates off fixed the issue. Thanks for the help!

> After starting midas (mhttpd &, and mlogger -D) and running the `fetest` frontend I went into the midas/python/examples directory and ran basic_hist_script.py, and, even though I could see the 'pytest' program in the Programs page,
> 
> Valid events are:
> Enter event name:
> 
> was printed out, which signified that no events were found. No errors were displayed.

To check that MIDAS itself is built correctly, you can try "make test", this will create a sample experiment,
run fetest, start, stop a run and check that data file and history file is created with correct history events.

If "make test" fails, I can help debug it.

In your experiment, you can check that history files are created correctly:

1) "mhist -l" should show all available events
2) "mhdump -L *.hst" should show all events in the .hst history files
3) if you have the newer mhf*.dat files, you can "more mhf_1449770978_20151210_hv.dat" to see what data is inside

If all of that works as expected, there must be a problem with the python side and we will have to figure
out how to reproduce it.

This reminds me, "make test" does not test any of the python code, it should be added (and python should be added
to the bitbucket builds).

K.O.

> Valid events are:
> Enter event name:
> 
> was printed out, which signified that no events were found. No errors were displayed.

I can't reproduce this. I made a brand new experiment, started mlogger/mhttpd/fetest, then ran the same program. I get:

```
$ python basic_hist_script.py
Valid events are:
* Run transitions
* test_slow/data
Enter event name: 
```

Are you sure you ran the python program after running mlogger and not before? Can you try again after restarting mlogger? And can you verify that your python is connecting to the correct experiment if you have multiple experiments defined?

I tested with python 3.12.8 and 3.13.1, and am on MacOS 14.5, but I can't imagine those differences matter.

The python interface is a trivial wrapper around the C++ function, so the only python-specific thing that would result in an empty list is extracting an integer from a ctypes reference. If that's broken in your version then I don't think any of the midas python code would be working.

After starting midas (mhttpd &, and mlogger -D) and running the `fetest` frontend I went into the midas/python/examples directory and ran basic_hist_script.py, and, even though I could see the 'pytest' program in the Programs page,

Valid events are:
Enter event name:

was printed out, which signified that no events were found. No errors were displayed.

Instead, when trying to do the same in javascript (using mjsonrpc_send_request( mjsonrpc_make_request("hs_get_events")).then(console.log)), I was able to get the expected events.

The History page also displayed the expected data and the plots worked correctly.

Device info: Chip: Apple M1 Pro, OS: Sequoia (15.3)

MIDAS version: bitbucket commit 84c7ef7

Python version: 3.13.2

Hello,

I've implemented a very basic syntax validation in the sequencer GUI. Click the validation button to check the syntax in the current tab.

Please note that this does only a simple syntax validation, the correctness of the logic is still on you :)

Hi,

Sorry, we have been slammed with a couple projects on the TRIUMF side in the past weeks and haven't found time for a 
response.  I am hopeful that we will be able to answer this question (and the cache size question) within the next 10 
days.  

Again apologies,
Thomas

> Hi,
> I have a manalyzer that uses a derived class of TMFeRpcHandlerInterface to communicate information to 
> Midas during online running.  At the end of each run it saves out custom data in the 
> TMFeRpcHandlerInterface::HandleEndRun override. This works really well.
> However, when I run offline on a Midas output file the HandleEndRun method is never called and my data is 
> never saved.  Is this intentional?  I understand that there is no point for the HandleBinaryRpc method offline, 
> but the other methods (HandleEndRun, HandleBeginRun etc) could serve a purpose.  Or is it a conscious 
> choice to ignore all of TMFeRpcHandlerInterface when offline?
> 
> Thanks,
> 
> Mark.

   {"Trigger",               /* equipment name */
      {1, 0,                 /* event ID, trigger mask */
         "SYSTEM",           /* event buffer */
         EQ_POLLED,          /* equipment type */
         0,                  /* event source */
         "MIDAS",            /* format */
         TRUE,               /* enabled */
         RO_RUNNING |        /* read only when running */
         RO_ODB,             /* and update ODB */
         100,                /* poll for 100ms */
         0,                  /* stop run after this event limit */
         0,                  /* number of sub events */
         0,                  /* don't log history */
         "", "", "", "", "", 0, 0},
      read_trigger_event,    /* readout routine */
      10000000,              /* write cache size */
   },

fe_set_write_case(int size);

12:51:20.154 2025/02/19 [SC Frontend,ERROR] [mfe.cxx:620:register_equipment,ERROR] Write cache size mismatch for buffer "SYSTEM": equipment "Environment" asked for 0, while eqiupment "LED" asked for 10000000
12:51:20.154 2025/02/19 [SC Frontend,ERROR] [mfe.cxx:620:register_equipment,ERROR] Write cache size mismatch for buffer "SYSTEM": equipment "LED" asked for 10000000, while eqiupment "Xenon" asked for 0

Hi,
I have a manalyzer that uses a derived class of TMFeRpcHandlerInterface to communicate information to 
Midas during online running.  At the end of each run it saves out custom data in the 
TMFeRpcHandlerInterface::HandleEndRun override. This works really well.
However, when I run offline on a Midas output file the HandleEndRun method is never called and my data is 
never saved.  Is this intentional?  I understand that there is no point for the HandleBinaryRpc method offline, 
but the other methods (HandleEndRun, HandleBeginRun etc) could serve a purpose.  Or is it a conscious 
choice to ignore all of TMFeRpcHandlerInterface when offline?

Thanks,

Mark.

to continue where we left off in 2019,
https://daq00.triumf.ca/elog-midas/Midas/1520

time to choose the next c++!

snapshot of 2019:

- Linux RHEL/SL/CentOS6 - gcc 4.4.7, no C++11.
- Linux RHEL/SL/CentOS7 - gcc 4.8.5, full C++11, no C++14, no C++17
- Ubuntu 18.04.2 LTS - gcc 7.3.0, full C++11, full C++14, "experimental" C++17.
- MacOS 10.13 - llvm 10.0.0 (clang-1000.11.45.5), full C++11, full C++14, full C++17

the world moved on:

- el6/SL6 is gone
- el7/CentOS-7 is out the door, only two experiments on my plate (EMMA and ALPHA-g)
- el8 was a still born child of RedHat
- el9 - gcc 11.5 with 12, 13, and 14 available.
- el10 - gcc 14.2

- U-18 - gcc  7.5
- U-20 - gcc  9.4 default, 10.5 available
- U-22 - gcc 11.4 default, 12.3 available
- U-24 - gcc 13.3 default, 14.2 available

- MacOS 15.2 - llvm/clang 16

Next we read C++ level support:

(see here for GCC C++ support: https://gcc.gnu.org/projects/cxx-status.html)
(see here for LLVM clang c++ support: https://clang.llvm.org/cxx_status.html)
(see here for GLIBC c++ support: https://gcc.gnu.org/onlinedocs/libstdc++/manual/status.html)

gcc:
4.4.7 - no C++11
4.8.5 - full C++11, no C++14, no C++17
7.3.0 - full C++11, full C++14, "experimental" C++17.
7.5.0 - c++17 and older
9.4.0 - c++17 and older
10.5 - no c++26, no c++23, mostly c++20, c++17 and older
11.4 - no c++26, no c++23, full c++20 and older
12.3 - no c++26, mostly c++23, full c++20 and older
13.3 - no c++26, mostly c++23, full c++20 and older
14.2 - limited c++26, mostly c++23, full c++20 and older

clang:
16 - no c++26, mostly c++23, mostly c++20, full c++17 and older

I think our preference is c++23, the number of useful improvements is quite big.

This choice will limit us to:
- el9 or older
- U-22 or older
- current MacOS 15.2 with Xcode 16.

It looks like gcc and llvm support for c++23 is only partial, so obviously we will use a subset of c++23 
supported by both.

Next step is to try to build midas with c++23 on el9 and U-22,24 and see what happens.

K.O.

> > Is there a clear migration docu somewhere?

I can also give you links to the alpha-g analyzer (very complex) and the DarkLight trigger TDC analyzer (very simple),
there is also analyzer examples of in-between complexity.

K.O.