Dear experts,
       while trying to readout a MIDAS file from a python script. I get the error below at the very first event. Any hint?

Thank you very much,
            Francesco

  File "/home/cygno/DAQ/offline/file_reader.py", line 9, in <module>
    for event in mfile:
  File "/home/cygno/DAQ/python/midas/file_reader.py", line 159, in __next__
    ev = self.read_next_event()
  File "/home/cygno/DAQ/python/midas/file_reader.py", line 264, in read_next_event
    return self.read_this_event_body()
  File "/home/cygno/DAQ/python/midas/file_reader.py", line 307, in read_this_event_body
    self.event.unpack_body(body_data, 0, self.use_numpy)
  File "/home/cygno/DAQ/python/midas/event.py", line 648, in unpack_body
    bank.fill_header_from_bytes(bank_header_data, self.is_bank_32(), self.is_bank_data_64bit_aligned())
  File "/home/cygno/DAQ/python/midas/event.py", line 298, in fill_header_from_bytes
    self.name = "".join(x.decode('ascii') for x in unpacked[:4])
  File "/home/cygno/DAQ/python/midas/event.py", line 298, in <genexpr>
    self.name = "".join(x.decode('ascii') for x in unpacked[:4])
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc8 in position 0: ordinal not in range(128)

Dear all,
     I need to integrate in my MIDAS project the communication with an OPC UA 
server. My plan is to develop an OPC UA client as a "device" in 
midas/drivers/device.

Two questions:

1) Is anybody aware of some similar effort for some other project, so that I can 
get some example?

2) What could be the more appropriate driver's class to be used? generic.cxx? 
multi.cxx?

Thank you for your help,
           Francesco

Dear all,
      I am trying to make the MIDAS web server for my DAQ project accessible from other machines. In the ODB, I activated the necessary flags:

[local:CYGNUS_RD:S]/WebServer>ls
Enable localhost port           y
localhost port                  8080
localhost port passwords        n
Enable insecure port            y
insecure port                   8081
insecure port passwords         y
insecure port host list         y
Enable https port               y
https port                      8443
https port passwords            y
https port host list            n
Host list
                                localhost
Enable IPv6                     y
Proxy                           
mime.types

Following the instructions on the Wiki I enabled the SSL support. When running mhttpd, I get these messages:

Mongoose web server will use HTTP Digest authentication with realm "CYGNUS_RD" and password file "/home/cygno/DAQ/online/htpasswd.txt"
Mongoose web server will use the hostlist, connections will be accepted only from: localhost
Mongoose web server listening on http address "localhost:8080", passwords OFF, hostlist OFF
Mongoose web server listening on http address "[::1]:8080", passwords OFF, hostlist OFF
Mongoose web server listening on http address "8081", passwords enabled, hostlist enabled
[mhttpd,ERROR] [mhttpd.cxx:19166:mongoose_listen,ERROR] Cannot mg_bind address "[::0]:8081"
Mongoose web server will use https certificate file "/home/cygno/DAQ/online/ssl_cert.pem"
Mongoose web server listening on https address "8443", passwords enabled, hostlist OFF
[mhttpd,ERROR] [mhttpd.cxx:19166:mongoose_listen,ERROR] Cannot mg_bind address "[::0]:8443"

and the server is not accessible from other machines. Any suggestion to solve or better investigate this problem?

Thank you very much,
         Francesco

It worked, thank you very much!

Francesco

> > Enable IPv6                     y
> 
> Probably the IPv6 problem, see here elog:2269
> 
> I asked to turn off IPv6 by default, or at least mention this in the documentation,
> but unfortunately nothing happened.
> 
> Stefan

Hello,
     I'm experiencing crashes of the mlogger program on the time scale of a couple 
of days. The only messages from MIDAS are:

05:34:47.336 2021/10/24 [mhttpd,INFO] Client 'Logger' (PID 14281) on database 
'ODB' removed by db_cleanup called by cm_periodic_tasks (idle 10.2s,TO 10s)
05:34:47.335 2021/10/24 [mhttpd,INFO] Client 'Logger' on buffer 'SYSMSG' removed 
by cm_periodic_tasks (idle 10.2s, timeout 10s)

Any suggestion to further investigate this issue?

Thank you very much,
         Francesco

Dear all,
       I've a frontend writing a quite big bunch of data into a MIDAS bank (16bit output from a 4MP photo camera). 
I'm experiencing a writing speed problem that I don't understand. When the photo camera is triggered at a low rate (< 2 Hz) 
writing into the bank takes a very short time for each event (indeed, what I measure is the time to write and go back 
into the polling function). If I increase the rate to 4 Hz, I see that writing the first two events takes a sort time, 
but the third event takes a very long time (hundreds of ms), then again the fourth and fifth events are very fast, and 
the sixth is very slow. If I further increase the rate, every other event is very slow. The problem is not in the readout 
of the camera, because if I just remove the bank writing and keep the camera readout, the problem disappears. Can you 
explain this behavior? Is there any way to improve it?

Below you can also find the code I use to copy the data from the camera buffer into the bank. If you have any suggestion 
to improve it, it would be really appreciated.

Thank you very much,
          Francesco



  const char* pSrc = (const char*)bufframe.buf;

  for(int y = 0; y < bufframe.height; y++ ){

    //Copy one row
    const unsigned short* pDst = (const unsigned short*)pSrc;

    //go through the row
    for(int x = 0; x < bufframe.width; x++ ){

      WORD tmpData = *pDst++; 

      *pdata++ = tmpData;

    }

    pSrc += bufframe.rowbytes;

  }
 

Dear all,
        I finally succeeded to get a working driver for the communication with an OPC 
UA server. It is based on the open62541 library and I use it in combination with the 
generic.h driver class. This is still a crude implementation, but let me post it here, 
maybe it can be useful to somebody else.

BTW, if there is somebody more skilled than me with OPC UA and MIDAS drivers, who is 
willing to give suggestions for improving the implementation, it would be extremely 
appreciated.

Best Regards,
      Francesco



> Dear all,
>      I need to integrate in my MIDAS project the communication with an OPC UA 
> server. My plan is to develop an OPC UA client as a "device" in 
> midas/drivers/device.
> 
> Two questions:
> 
> 1) Is anybody aware of some similar effort for some other project, so that I can 
> get some example?
> 
> 2) What could be the more appropriate driver's class to be used? generic.cxx? 
> multi.cxx?
> 
> Thank you for your help,
>            Francesco

Dear all,
       I'm writing a very simple device driver starting from the nulldev.cxx 
example. 

I define an equipment as reported at the end of this message then, if all 
variables are Input variables, I define them with:

  mdevice device("myEquimpent", "Input", DF_INPUT | DF_MULTITHREAD, mydevice);
  device.define_var("Var1", 0.1);
  device.define_var("Var2", 0.1);
  ...

If all variables are output variables, I define them with:

  mdevice device("myEquipment", "Output", DF_OUTPUT | DF_MULTITHREAD, mydevice);
  device.define_var("Var1", 0.1);
  device.define_var("Var2", 0.1);

But I don't know what to do if I have mixed input and output variables in the same 
device. I think I can do:

  mdevice device_in("myEquipment", "Input", DF_INPUT | DF_MULTITHREAD, mydevice);
  device.define_var("Var1", 0.1);

  mdevice device_out("myEquipment", "Output", DF_OUTPUT | DF_MULTITHREAD, 
mydevice);
  device.define_var("Var2", 0.1);

but in this case, inside mydevice.cxx, I don't know how to distinguish Var1 and 
Var2, because they are both identified as channel 0.

Do you have any suggestion?

Thank you,
      Francesco




-------------------------------------------------------------------



   {"SourceMotor",                       /* equipment name */
    {7, 0,                       /* event ID, trigger mask */
     "SYSTEM",                  /* event buffer */
     EQ_SLOW,                   /* equipment type */
     0,                         /* event source */
     "MIDAS",                   /* format */
     TRUE,                      /* enabled */
     RO_ALWAYS,        /* read when running and on transitions */
     60000,                     /* read every 60 sec */
     0,                         /* stop run after this event limit */
     0,                         /* number of sub events */
     1,                         /* log history every event */
     "", "", ""} ,
    cd_multi_read,                 /* readout routine */
    cd_multi,                      /* class driver main routine */
   },

Dear experts,
         I would like to have some clarification about the meaning and use of the 
tags in the ODB under /History/Tags.

I noticed that, if a history plot is created, but the name of the corresponding 
variable is changed later and the plot is modified accordingly, the old name 
persists in the /History/Tags list along with the new one. So, it appears in the 
list of variables when a new history plot is created.

It seems not to compromise the functionalities of the history system, but it is 
prone to create confusion.

Is it the expected behavior? What is the correct procedure to follow if the name 
of a variable has to be changed?

Thank you,
     Francesco

Dear experts,
      in the CYGNO experiment, we readout CMOS cameras for optical readout of GEM-TPCs. So far, we only developed the readout for a single camera. In the future, we will have multiple cameras to read out (up to 18 in the next phase of the experiment), and we are investigating how to optimize the readout by exploiting parallelization.

One idea is to start parallel threads within a single equipment. Alternatively, one could associate different equipment with each camera and run an Event Builder. Perhaps other solutions did not come to mind. Which one would you regard as the most effective and elegant?

Thank you very much,
           Francesco

Hi,

I've been doing some preliminary work to use at least the MIDAS
SQL history component for a new CERN experiment (Aegis). I wonder
whether there is any plan to support 64-bit signed/unsigned integer data types
in MIDAS. time_t on 64-bit architectures is actually signed 64-bit
(the 'easy' way to work around the 2038 crisis), and this may be enough to
cause problems.

Thanks.
Francesco Prelz
INFN Milano

Hello,

When I  connect to analyzer on a x86_64 processor(with Roody),  
a analyzer break with segmentation violation in the root_server_thread  function.
Same code are working fine on a 32bit processor.
As I found the problem are in exchanging of pointers between analyzer and client.
Before to send a pointer, it is saved a pointer in int (size=4, instead of 8) at
this place:
Index: src/mana.c
===================================================================
--- src/mana.c  (revision 3498)
+++ src/mana.c  (working copy)
@@ -5386,7 +5386,7 @@

             //write pointer
             message->Reset(kMESS_ANY);
-            int p = (POINTER_T) obj;
+            POINTER_T p = (POINTER_T) obj;
             *message << p;
             sock->Send(*message);


Sincerely Yours,
Fedor Ignatov 

Yes right, Problem of a segmentation violation is solved with this patch. Now it works
fine on x86_64.

Fedor 

> Do I understand you right? With your patch it works even on 64 bit, right? Or do you
> mean there is still a segmentation violation? Anyhow I committed your patch since the
> "int" is clearly incorrect.
> 
> - Stefan

After starting midas (mhttpd &, and mlogger -D) and running the `fetest` frontend I went into the midas/python/examples directory and ran basic_hist_script.py, and, even though I could see the 'pytest' program in the Programs page,

Valid events are:
Enter event name:

was printed out, which signified that no events were found. No errors were displayed.

Instead, when trying to do the same in javascript (using mjsonrpc_send_request( mjsonrpc_make_request("hs_get_events")).then(console.log)), I was able to get the expected events.

The History page also displayed the expected data and the plots worked correctly.

Device info: Chip: Apple M1 Pro, OS: Sequoia (15.3)

MIDAS version: bitbucket commit 84c7ef7

Python version: 3.13.2

> After starting midas (mhttpd &, and mlogger -D) and running the `fetest` frontend I went into the midas/python/examples directory and ran basic_hist_script.py, and, even though I could see the 'pytest' program in the Programs page,
> 
> Valid events are:
> Enter event name:
> 
> was printed out, which signified that no events were found. No errors were displayed.
> 
> Instead, when trying to do the same in javascript (using mjsonrpc_send_request( mjsonrpc_make_request("hs_get_events")).then(console.log)), I was able to get the expected events.
> 
> The History page also displayed the expected data and the plots worked correctly.
> 
> Device info: Chip: Apple M1 Pro, OS: Sequoia (15.3)
> 
> MIDAS version: bitbucket commit 84c7ef7
> 
> Python version: 3.13.2


I tested the command this morning and it worked. The most likely cause of the errors was that I was on a beta version of macOS: Switching beta updates off fixed the issue. Thanks for the help!

Setup:
setting up midas, starting mhttpd and mlogger and running fetest.

The History page and the javascript mjsonrpc client are both able to fetch historical data for test_slow/data. Javascript code used is included here:

mjsonrpc_call(
  "hs_read_arraybuffer",
  {
    start_time: Math.floor((new Date()).getTime() /1000) - 1000,
    end_time: Math.floor((new Date()).getTime() /1000),
    events: ["test_slow/data"],
    tags: ["data"],
    index: [0],
  },
  "arraybuffer"
).then(console.log)

However, the python client does not find any valid events:

Setup:
An exptab is created and the environment variables MIDAS_EXPTAB and MIDAS_EXPT_NAME and MIDASSYS are set (together with the correct PATH)

Running /midas/python/examples/basic_hist_script.py and typing in data:

Valid events are:
* Run transitions
* rrandom/SLOW
* test_slow/data
Enter event name: test_slow/data
Valid tags for test_slow/data are:
* data
Enter tag name: data
Event/tag test_slow/data/data has 1 elements
How many hours: 1
Interval in seconds: 1 # other values were also tested, without success
0 entries found

We expect entries to be found, however do not.

Tested setups:
Macbook Pro Sequoia 15.3 with Python 3.13.2, ROOT latest, midas bitbucket commit 84c7ef7
Windows 11 with Python 3.11, ROOT latest, midas latest commit (development branch)

> Unfortunately I again cannot reproduce this:
> 
> $ python ~/DAQ/midas_latest/python/examples/basic_hist_script.py
> Valid events are:
> * Run transitions
> * test_slow/data
> Enter event name: test_slow/data
> Valid tags for test_slow/data are:
> * data
> Enter tag name: data
> Event/tag test_slow/data/data has 1 elements
> How many hours: 1
> Interval in seconds: 1
> 78 entries found
> 2025/03/17 17:00:56 => 98.097391
> 2025/03/17 17:00:57 => 98.982151
> 2025/03/17 17:00:58 => 99.589187
> 2025/03/17 17:00:59 => 99.926821
> 2025/03/17 17:01:00 => 99.989878
> 2025/03/17 17:01:01 => 99.778216
> 2025/03/17 17:01:02 => 99.292485
> .......
> 
> 
> I want to narrow down whether the issue is in the basic_hist_script.py or the lower-level code. So there are a few steps of debugging to do.
> 
> 
> 
> 1) Run code directly in the python interpreter: 
> 
> Can you run the following and send the output please?
> 
> ```
> import midas.client
> c = midas.client.MidasClient("history_test")
> data = c.hist_get_recent_data(1,1,"test_slow/data","data")
> print(f"event_name='{data[0]['event_name']}', tag_name='{data[0]['tag_name']}', num_entries={data[0]['num_entries']}, status={data[0]['status']}, arrlen={len(data[0]['values'])}")
> ```
> 
> For me, I get:
> event_name='test_slow/data', tag_name='data', num_entries=441, status=1, arrlen=441
> 
> 
> 
> 2) If things look sensible for you (status=1, non-zero num_entries), then the problem is in the basic_hist_script.py. Can you add the same print() statement in basic_hist_script.py immediately after the call to hist_get_recent_data(), then run that script again and send the output of that?
> 
> 
> 
> 3) Debug the python/C conversions.
> 
> In midas/client.py add the following line to hist_get_data() immediately before the call to self.lib.c_hs_read():
> 
> ```
>         print(f"c_start_time={c_start_time.value}, c_end_time={c_end_time.value}, c_interval={c_interval.value}, c_event_name={c_event_name.value}, c_tag_name={c_tag_name.value}")
> ```
> 
> Then run the following and send the output:
> 
> ```
> import midas.client
> c = midas.client.MidasClient("history_test")
> data = c.hist_get_recent_data(1,1,"test_slow/data","data")
> ```
> 
> For me, I get:
> c_start_time=1742254428, c_end_time=1742258028, c_interval=1, c_event_name=b'test_slow/data', c_tag_name=b'data'
> 
> I want to check that the UNIX timestamps match what you expect for your server, and that nothing weird is going on with the python/C string conversions.
> 
> 
> Thanks,
> Ben

Hi, thank you for the support!

1)
event_name='test_slow/data', tag_name='data', num_entries=0, status=1, arrlen=0
2)
The number of entries is zero
3)
I get
c_start_time=1742275653, c_end_time=1742279253, c_interval=1, c_event_name=b'test_slow/data', c_tag_name=b'data'

> Unfortunately I again cannot reproduce this:
> 
> $ python ~/DAQ/midas_latest/python/examples/basic_hist_script.py
> Valid events are:
> * Run transitions
> * test_slow/data
> Enter event name: test_slow/data
> Valid tags for test_slow/data are:
> * data
> Enter tag name: data
> Event/tag test_slow/data/data has 1 elements
> How many hours: 1
> Interval in seconds: 1
> 78 entries found
> 2025/03/17 17:00:56 => 98.097391
> 2025/03/17 17:00:57 => 98.982151
> 2025/03/17 17:00:58 => 99.589187
> 2025/03/17 17:00:59 => 99.926821
> 2025/03/17 17:01:00 => 99.989878
> 2025/03/17 17:01:01 => 99.778216
> 2025/03/17 17:01:02 => 99.292485
> .......
> 
> 
> I want to narrow down whether the issue is in the basic_hist_script.py or the lower-level code. So there are a few steps of debugging to do.
> 
> 
> 
> 1) Run code directly in the python interpreter: 
> 
> Can you run the following and send the output please?
> 
> ```
> import midas.client
> c = midas.client.MidasClient("history_test")
> data = c.hist_get_recent_data(1,1,"test_slow/data","data")
> print(f"event_name='{data[0]['event_name']}', tag_name='{data[0]['tag_name']}', num_entries={data[0]['num_entries']}, status={data[0]['status']}, arrlen={len(data[0]['values'])}")
> ```
> 
> For me, I get:
> event_name='test_slow/data', tag_name='data', num_entries=441, status=1, arrlen=441
> 
> 
> 
> 2) If things look sensible for you (status=1, non-zero num_entries), then the problem is in the basic_hist_script.py. Can you add the same print() statement in basic_hist_script.py immediately after the call to hist_get_recent_data(), then run that script again and send the output of that?
> 
> 
> 
> 3) Debug the python/C conversions.
> 
> In midas/client.py add the following line to hist_get_data() immediately before the call to self.lib.c_hs_read():
> 
> ```
>         print(f"c_start_time={c_start_time.value}, c_end_time={c_end_time.value}, c_interval={c_interval.value}, c_event_name={c_event_name.value}, c_tag_name={c_tag_name.value}")
> ```
> 
> Then run the following and send the output:
> 
> ```
> import midas.client
> c = midas.client.MidasClient("history_test")
> data = c.hist_get_recent_data(1,1,"test_slow/data","data")
> ```
> 
> For me, I get:
> c_start_time=1742254428, c_end_time=1742258028, c_interval=1, c_event_name=b'test_slow/data', c_tag_name=b'data'
> 
> I want to check that the UNIX timestamps match what you expect for your server, and that nothing weird is going on with the python/C string conversions.
> 
> 
> Thanks,
> Ben

Hi, thank you for the support!

Running the commands on the Macbook pro leads to
1)
event_name='test_slow/data', tag_name='data', num_entries=0, status=1, arrlen=0
2)
The number of entries is zero
3)
I get
c_start_time=1742275653, c_end_time=1742279253, c_interval=1, c_event_name=b'test_slow/data', c_tag_name=b'data'

However right after running these commands I removed a .SHM_HOST.TXT file
(due to me working both at home and at PSI, my computer hostname changes when I switch the network, so I remove .SHM_HOST.TXT to be able to run experiments)

and reran the code, and it suddenly worked! This is good, but I do not know what fixed it... I had done more extensive tests yesterday and also had to delete .SHM_HOST.TXT multiple times, to no avail.
Do you have any ideas as to what could be happening? Similarly to my previous bug report, which was fixed by updating macOS to a more stable version, could this have been due to an automatic update?

If the problem still persists on the Windows machine I will post an update.

I add the following to makefile and try to treat Darwin as FreeBSD/Linux.
But I failed.
============ 
#-----------------------
# This is for MacOS X
#
ifeq ($(OSTYPE), Darwin)
CC = gcc
OS_DIR = Darwin
OSFLAGS = -DOS_DARWIN -DOS_LINUX
LIBS = -lbsd -lcompat
SPECIFIC_OS_PRG =
endif
============

I got the following errors:
=============
gcc -c -g -O2 -Wall -Iinclude -Idrivers -LDarwin/lib -DINCLUDE_FTPLIB  
-DOS_DARWIN -DOS_FREEBSD -o Darwin/lib/midas.o src/midas.c
In file included from include/midasinc.h:45,
                 from include/msystem.h:114,
                 from src/midas.c:623:
/usr/include/string.h:112: error: conflicting types for `strlcat'
include/midas.h:1701: error: previous declaration of `strlcat'
/usr/include/string.h:113: error: conflicting types for `strlcpy'
include/midas.h:1700: error: previous declaration of `strlcpy'
In file included from include/msystem.h:114,
                 from src/midas.c:623:
include/midasinc.h:161:21: sys/vfs.h: No such file or directory
include/midasinc.h:164:17: pty.h: No such file or directory
src/midas.c:780: error: conflicting types for `dbg_malloc'
include/midas.h:1478: error: previous declaration of `dbg_malloc'
src/midas.c:817: error: conflicting types for `dbg_calloc'
include/midas.h:1479: error: previous declaration of `dbg_calloc'
src/midas.c:858: error: conflicting types for `strlcpy'
/usr/include/string.h:113: error: previous declaration of `strlcpy'
src/midas.c:892: error: conflicting types for `strlcat'
/usr/include/string.h:112: error: previous declaration of `strlcat'
gmake: *** [Darwin/lib/midas.o] Error 1
==========

Could anyone give me some hints. Thanks!