Dear MIDAS experts,

I gave a try to the MIDAS python interface and ran all tests available in midas/python/tests.
Two Int64 tests from test_odb.py had failed (see below), everthong else - succeeded

I'm using a ~ 2.5 weeks-old commit and python 3.9 on SL7 Linux platform.

commit c19b4e696400ee437d8790b7d3819051f66da62d (HEAD -> develop, origin/develop, origin/HEAD)
Author: Zaher Salman <zaher.salman@gmail.com>
Date:   Sun Jan 14 13:18:48 2024 +0100

The symptoms are consistent with a string --> int64 conversion not happening 
where it is needed. 

Perhaps the issue have already been fixed? 

-- many thanks, regards, Pasha
-------------------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mu2etrk/test_stand/pasha_020/midas/python/tests/test_odb.py", line 178, in testInt64
    self.set_and_readback_from_parent_dir("/pytest", "int64_2", [123, 40000000000000000], midas.TID_INT64, True)
  File "/home/mu2etrk/test_stand/pasha_020/midas/python/tests/test_odb.py", line 130, in set_and_readback_from_parent_dir
    self.validate_readback(value, retval[key_name], expected_key_type)
  File "/home/mu2etrk/test_stand/pasha_020/midas/python/tests/test_odb.py", line 87, in validate_readback
    self.assert_equal(val, retval[i], expected_key_type)
  File "/home/mu2etrk/test_stand/pasha_020/midas/python/tests/test_odb.py", line 60, in assert_equal
    self.assertEqual(val1, val2)
AssertionError: 123 != '123'

with the test on line 178 commented out, the test on the next line fails in a similar way:
        
Traceback (most recent call last):
  File "/home/mu2etrk/test_stand/pasha_020/midas/python/tests/test_odb.py", line 179, in testInt64
    self.set_and_readback_from_parent_dir("/pytest", "int64_2", 37, midas.TID_INT64, True)
  File "/home/mu2etrk/test_stand/pasha_020/midas/python/tests/test_odb.py", line 130, in set_and_readback_from_parent_dir
    self.validate_readback(value, retval[key_name], expected_key_type)
  File "/home/mu2etrk/test_stand/pasha_020/midas/python/tests/test_odb.py", line 102, in validate_readback
    self.assert_equal(value, retval, expected_key_type)
  File "/home/mu2etrk/test_stand/pasha_020/midas/python/tests/test_odb.py", line 60, in assert_equal
    self.assertEqual(val1, val2)
AssertionError: 37 != '37'
---------------------------------------------------------------------------

> The symptoms are consistent with a string --> int64 conversion not happening 
> where it is needed. 

Thanks for the report Pasha. Indeed I was missing a conversion in one place. Fixed now!

Ben

> > The symptoms are consistent with a string --> int64 conversion not happening 
> > where it is needed. 
> 
> Thanks for the report Pasha. Indeed I was missing a conversion in one place. Fixed now!
> 

Are we running these tests as part of the nightly build on bitbucket? They would be part of 
the "make test" target. Correct python dependancies may need to be added to the bitbucket OS 
image in bitbucket-pipelines.yml. (This is a PITA to get right).

K.O.

I am setting up a sqlite db to serve as a run database.

The easiest option is to use the history sqlite feature, and add run information 
as virtual history events

however:

Invalid tag 0 'Comment' in event 21 'Run Parameters': cannot do history for 
TID_STRING data, sorry!

I'd like to save e.g. the edit on start information , with shift crew checks. 
Would it be easy to allow for text, or is this inherent to the history system 
handling binary data?

> Invalid tag 0 'Comment' in event 21 'Run Parameters': cannot do history for 
> TID_STRING data, sorry!

The original MIDAS history API does not have provisions for storing TID_STRING data,
it is a very unfortunate limitation that has been with us for a very long time.

If I ever get around to rewrite the MIDAS history API, I will definitely add support for TID_STRING data.

But not today.

K.O.

P.S. Support for arbitrary binary blobs is also possible, but this will make the midas history
a kind of "a daq inside the daq" thing, probably we do not want to go this direction.

K.O.

I had to move strlcpy & strlcat into a separate file "strlcpy.c". A header file
"strlcpy.h" was added as well. This way one can omit the old HAVE_STRLCPY which
made life hard. The windows and linux makefiles were adjusted accordingly, but
for Max OS X there might be some fixes necessary which I could not test.

Hi all,
  I've started receiving the following error that I can't track down.  Does
anyone have a suggestion for where to start looking for the cause of this?

[Analyzer,ERROR] [odb.c:9460:db_open_record,ERROR] struct size mismatch for "/"
(expected size: 576, size in ODB: 0)

This error prevents me from running two runs in a row.  I have to close the DAQ
and restart to take multiple runs.  Also it prevents me from running the analyzer
in offline mode. 

I also noticed that several for the ODB directories no longer have the same html
format when viewed through the browser.  I've attached a screen print of the
"/Logger/Channels" page.

Thanks,
Robert

We have seen several times a problem with stuck data buffers. The symptoms are very confusing - 
frontends cannot start, instead hang forever in a state very hard to kill. Also "mdump -s -d -z 
BUF03" for the affected data buffers is stuck.

We have identified the source of this problem - the semaphore for the buffer is locked and nobody 
will ever unlock it - MIDAS relies on a feature of SYSV semaphores where they are automatically 
unlocked by the OS and cannot ever be stuck ever. (see man semop, SEM_UNDO function).

I think this SEM_UNDO function is broken in recent Linux kernels and sometimes the semaphore 
remains locked after the process that locked it has died. MIDAS is not programmed to deal with this 
situation and the stuck semaphore has to be cleared manually.

Here, "BUF3" is used as example, but we have seen "SYSTEM" and ODB with stuck semaphores, too.

Steps:
a) confirm that we are using SYSV semaphores: "ipcs" should show many semaphores
b) identify the stuck semaphore: "strace mdump -s -d -z BUF03".
c) here will be a large printout, but ultimately you will see repeated entries of 
"semtimedop(9633800, {{0, -1, SEM_UNDO}}, 1, {1, 0}^C <unfinished ...>"
d) erase the stuck semaphore "ipcrm -s 9633800", where the number comes from semtimedop() in 
the strace output.
e) try again: "mdump -s -d -z BUF03" should work now.

Ultimately, I think we should switch to POSIX semaphores - they are easier to manage (the strace 
and ipcrm dance becomes "rm /dev/shm/deap_BUF03.sem" - but they do not have the SEM_UNDO 
function, so detection of locked and stuck semaphores will have to be done by MIDAS. (Unless we 
can find some library of semaphore functions that already provides such advanced functionality).

K.O.

Hi,

I was wondering if there is a "mechanism" to run an executable
file after each subrun is closed...

I need to convert .mid.lz4 subrun files to ROOT (TTree) files;

Thanks,
Gennaro

There is no "mechanism" foreseen to be executed after each subrun. But you could 
run a shell script after each run which loops over all subruns and converts them 
one after the other.

Stefan

> Hi,
> 
> I was wondering if there is a "mechanism" to run an executable
> file after each subrun is closed...
> 
> I need to convert .mid.lz4 subrun files to ROOT (TTree) files;
> 
> Thanks,
> Gennaro

We use the lazylogger for something similar to this. You can specify the path to a custom script, and it will be run for each midas file that gets written:
https://midas.triumf.ca/MidasWiki/index.php/Lazylogger#Using_a_script
This means that you don't have to wait until the end of the run to start processing.

If the ROOT conversion is going to be slow, but you have a batch system available, you could use the lazylogger script to submit a job to the batch system for each file.

> 
> > Hi,
> > 
> > I was wondering if there is a "mechanism" to run an executable
> > file after each subrun is closed...
> > 
> > I need to convert .mid.lz4 subrun files to ROOT (TTree) files;
> > 
> > Thanks,
> > Gennaro

Please be aware of mlogger.c update rev 4566 on Sept 23rd 2009, when Stefan
fixed a buglet in the subrun file size computations. Before this fix, the first
subrun could be of a short length. If you use subruns, please update your
mlogger to at least rev 4566 (or newer, Stefan added the run and subrun time
limits just recently).
K.O.

> >
> > .../c++  ...  /opt/local/lib/libssl.dylib
> >
> 
> I get the same, libssl from /opt/local, so we are not using openssl shipped with mac os.
> 

I note that latest mongoose 6.16 finally has a virtual ssl layer and appears to support mbedtls (polarssl) in addition to openssl.

I now think I should see if it works - as it gives us a way to support https without relying on the user having
pre-installed working openssl packages - we consistently run into problems with openssl on macos, and even
on linux there was trouble with preinstalled openssl packages and libraries.

With mbedtls, one will have to "git pull" and "make" it, but historically this causes much less trouble.

Also, with luck, mbedtls has better support for certificate expiration (I would really love to have openssl report an error
or a warning or at least some hint if I feed it an expired certificate) and (gasp!) certbot integration.

K.O.

> ... support for certbot

The certbot tool to use instead of certbot is this: https://github.com/ndilieto/uacme

K.O.

> > ... support for certbot
> 
> The certbot tool to use instead of certbot is this: https://github.com/ndilieto/uacme
> 
> K.O.


HI Stefan and Konstantin, 

Thanks a lot for your messages. Sorry for my late reply, I only work on this project from Tuesday to Thursdays. I have 
run " make cmake" instead of "cd build; cmake" and this is the output regarding mhttpd:

/Library/Developer/CommandLineTools/usr/bin/c++  -O2 -g -DNDEBUG -Wl,-search_paths_first -Wl,-
headerpad_max_install_names  CMakeFiles/mhttpd.dir/mhttpd.cxx.o CMakeFiles/mhttpd.dir/mongoose6.cxx.o 
CMakeFiles/mhttpd.dir/mgd.cxx.o CMakeFiles/mhttpd.dir/__/mscb/src/mscb.cxx.o  -o mhttpd  -lsqlite3 ../libmidas.a 
/usr/lib/libssl.dylib /usr/lib/libcrypto.dylib -lz -lsqlite3 /usr/lib/libssl.dylib /usr/lib/libcrypto.dylib -lz 
Undefined symbols for architecture x86_64:
  "_OPENSSL_init_ssl", referenced from:
      _mg_mgr_init in mongoose6.cxx.o
  "_SSL_CTX_set_options", referenced from:
      _mg_set_ssl in mongoose6.cxx.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)


I see that in your outputs the openssl libs are in /opt/local/lib/ while mine are in /usr/lib/, and that is the only difference. 
I have checked that the libraries libssl.dylib and libcrypto.dylib are in my /usr/lib/, and indeed they are, so I don't 
understand the reason for the error, I will continue investigating. 

Thanks 

Berta 

> I only work on this project from Tuesday to Thursdays.

No problem. No hurry.

> I have run " make cmake" instead of "cd build; cmake" and this is the output regarding mhttpd: ...

There should be also a line where mhttpd.cxx is compiled into mhttpd.o, we need to see what compiler
flags are used - I suspect the compiler uses header files from /usr/local/include while the linker
is using libraries from /usr/lib, a mismatch.

To save time, please attach the full output of "make cmake". There may be something else I want to see there.

If you do not use the mhttpd built-in https support (for best security I recommend using https from the apache httpd password protected https proxy),
then it is perfectly fine to build midas with NO_SSL=1.

K.O.

Fuming, fuming, fuming.

The combination of "make indent" and "svn update" completely trashed my work copy of midas. Half of 
the files now show as status "M", half as status "C" ("in conflict"), even those I never edited myself (e.g. 
mscb firmware files).

I think what happened as that once I ran "make indent", the indent program did things to the source 
files (changed indentation, added spaces in "foo(a,b,c); --> foo(a, b, c);" etc, so now svn thinks that I 
edited the files and they are in conflict with later modifications.

I suggest that nobody ever ever ever should use "make indent", and if they do, they should better 
commit their "changes" made by indent very quickly, before their midas tree is trashed by the next "svn 
update".

And if they commit the changes made by "make indent", beware that "make indent" is not idempotent, 
running it multiple times, it keeps changing files (keeps moving some dox comments around).

Also beware of entering a tug-of-war with Stefan - at least on my machines, my "make indent" seems 
to produce different output from his.

Still fuming, even after some venting...
K.O.

 Hi,
I was trying to update the checkout of Midas, but it looks like something is not
working - maybe a component of the Savannah system:
[sergio@daq-pc midas-SVN]$ svn update
svn@savannah.psi.ch's password: svn
unix dgram connect: Connection refused at /bin/cvssh.pl line 32
no connection to syslog available at /bin/cvssh.pl line 32
svn: Connection closed unexpectedly

my .svn/entries says (amongst the rest)
 url="svn+ssh://svn@savannah.psi.ch/afs/psi.ch/project/meg/svn/midas/trunk"
and yes, it used to work well... 

Cheers,
  Sergio

>  Hi,
> I was trying to update the checkout of Midas, but it looks like something is not
> working - maybe a component of the Savannah system:
> [sergio@daq-pc midas-SVN]$ svn update
> svn@savannah.psi.ch's password: svn
> unix dgram connect: Connection refused at /bin/cvssh.pl line 32
> no connection to syslog available at /bin/cvssh.pl line 32
> svn: Connection closed unexpectedly
> 
> my .svn/entries says (amongst the rest)
>  url="svn+ssh://svn@savannah.psi.ch/afs/psi.ch/project/meg/svn/midas/trunk"
> and yes, it used to work well... 
> 
> Cheers,
>   Sergio

I just tried now and it seemed to work fine. Do you still have the problem?

- Stefan

> I just tried now and it seemed to work fine. Do you still have the problem?
> 
> - Stefan

 The problem was still there this morning, shortly after seeing your mail, but seems
to be fixed now.
 BTW, which is the best way to submit patches ? I have a version of khyt1331 for Linux
kernel 2.6 (we are running Scientific Linux 4.1), and a few smaller things, mostly in
the examples. 

 Thanks, Sergio