Back Midas Rome Roody Rootana
  Midas DAQ System, Page 118 of 122  Not logged in ELOG logo
New entries since:Wed Dec 31 16:00:00 1969
ID Date Author Topic Subjectup
  515   23 Oct 2008 Konstantin OlchanskiBug Reportstrange output from "odbedit cleanup"
When I run odbedit remotely (odbedit -h ladd09), the "cleanup" command unexpectedly produces the 
output of the "sor" command (sure enough, there is a call to db_get_open_records() there), but when I run 
it locally, I do not get this output (but db_get_open_records() is still called). Strange. K.O.
  518   28 Oct 2008 Stefan RittBug Reportstrange output from "odbedit cleanup"
> When I run odbedit remotely (odbedit -h ladd09), the "cleanup" command unexpectedly produces the 
> output of the "sor" command (sure enough, there is a call to db_get_open_records() there), but when I run 
> it locally, I do not get this output (but db_get_open_records() is still called). Strange. K.O.

The db_get_open_records() call was by mistake there, I removed it. What remains is that the notification 
message if a client is removed from the ODB goes through the system messages. When running locally, odbedit 
echoes it's own messages, but when running remotely, this is not the case. So the messages can be seen by 
everybody else (plus it ends up in the message file), but not by the remote odbedit where the cleanup is 
started. The quick fix for that is to say "old" in odbedit which shows the last few lines of the message 
file, so one can see any successful cleanup. 
  1370   05 Jun 2018 Frederik WautersForumstrings in sqlite
I am setting up a sqlite db to serve as a run database.

The easiest option is to use the history sqlite feature, and add run information 
as virtual history events

however:

Invalid tag 0 'Comment' in event 21 'Run Parameters': cannot do history for 
TID_STRING data, sorry!

I'd like to save e.g. the edit on start information , with shift crew checks. 
Would it be easy to allow for text, or is this inherent to the history system 
handling binary data?
  1377   20 Jul 2018 Konstantin OlchanskiForumstrings in sqlite
> Invalid tag 0 'Comment' in event 21 'Run Parameters': cannot do history for 
> TID_STRING data, sorry!

The original MIDAS history API does not have provisions for storing TID_STRING data,
it is a very unfortunate limitation that has been with us for a very long time.

If I ever get around to rewrite the MIDAS history API, I will definitely add support for TID_STRING data.

But not today.

K.O.

P.S. Support for arbitrary binary blobs is also possible, but this will make the midas history
a kind of "a daq inside the daq" thing, probably we do not want to go this direction.

K.O.
  210   02 May 2005 Stefan RittInfostrlcpy/strlcat moved into separate file
I had to move strlcpy & strlcat into a separate file "strlcpy.c". A header file
"strlcpy.h" was added as well. This way one can omit the old HAVE_STRLCPY which
made life hard. The windows and linux makefiles were adjusted accordingly, but
for Max OS X there might be some fixes necessary which I could not test.
  1038   12 Nov 2014 Robert PattieForumstruct mismatch
Hi all,
  I've started receiving the following error that I can't track down.  Does
anyone have a suggestion for where to start looking for the cause of this?

[Analyzer,ERROR] [odb.c:9460:db_open_record,ERROR] struct size mismatch for "/"
(expected size: 576, size in ODB: 0)

This error prevents me from running two runs in a row.  I have to close the DAQ
and restart to take multiple runs.  Also it prevents me from running the analyzer
in offline mode. 

I also noticed that several for the ODB directories no longer have the same html
format when viewed through the browser.  I've attached a screen print of the
"/Logger/Channels" page.

Thanks,
Robert
  938   15 Nov 2013 Konstantin OlchanskiBug Reportstuck data buffers
We have seen several times a problem with stuck data buffers. The symptoms are very confusing - 
frontends cannot start, instead hang forever in a state very hard to kill. Also "mdump -s -d -z 
BUF03" for the affected data buffers is stuck.

We have identified the source of this problem - the semaphore for the buffer is locked and nobody 
will ever unlock it - MIDAS relies on a feature of SYSV semaphores where they are automatically 
unlocked by the OS and cannot ever be stuck ever. (see man semop, SEM_UNDO function).

I think this SEM_UNDO function is broken in recent Linux kernels and sometimes the semaphore 
remains locked after the process that locked it has died. MIDAS is not programmed to deal with this 
situation and the stuck semaphore has to be cleared manually.

Here, "BUF3" is used as example, but we have seen "SYSTEM" and ODB with stuck semaphores, too.

Steps:
a) confirm that we are using SYSV semaphores: "ipcs" should show many semaphores
b) identify the stuck semaphore: "strace mdump -s -d -z BUF03".
c) here will be a large printout, but ultimately you will see repeated entries of 
"semtimedop(9633800, {{0, -1, SEM_UNDO}}, 1, {1, 0}^C <unfinished ...>"
d) erase the stuck semaphore "ipcrm -s 9633800", where the number comes from semtimedop() in 
the strace output.
e) try again: "mdump -s -d -z BUF03" should work now.

Ultimately, I think we should switch to POSIX semaphores - they are easier to manage (the strace 
and ipcrm dance becomes "rm /dev/shm/deap_BUF03.sem" - but they do not have the SEM_UNDO 
function, so detection of locked and stuck semaphores will have to be done by MIDAS. (Unless we 
can find some library of semaphore functions that already provides such advanced functionality).

K.O.
  1999   24 Sep 2020 Gennaro TortoneForumsubrun
Hi,

I was wondering if there is a "mechanism" to run an executable
file after each subrun is closed...

I need to convert .mid.lz4 subrun files to ROOT (TTree) files;

Thanks,
Gennaro
  2046   01 Dec 2020 Stefan RittForumsubrun
There is no "mechanism" foreseen to be executed after each subrun. But you could 
run a shell script after each run which loops over all subruns and converts them 
one after the other.

Stefan

> Hi,
> 
> I was wondering if there is a "mechanism" to run an executable
> file after each subrun is closed...
> 
> I need to convert .mid.lz4 subrun files to ROOT (TTree) files;
> 
> Thanks,
> Gennaro
  2047   01 Dec 2020 Ben SmithForumsubrun
We use the lazylogger for something similar to this. You can specify the path to a custom script, and it will be run for each midas file that gets written:
https://midas.triumf.ca/MidasWiki/index.php/Lazylogger#Using_a_script
This means that you don't have to wait until the end of the run to start processing.

If the ROOT conversion is going to be slow, but you have a batch system available, you could use the lazylogger script to submit a job to the batch system for each file.

> 
> > Hi,
> > 
> > I was wondering if there is a "mechanism" to run an executable
> > file after each subrun is closed...
> > 
> > I need to convert .mid.lz4 subrun files to ROOT (TTree) files;
> > 
> > Thanks,
> > Gennaro
  675   25 Nov 2009 Konstantin OlchanskiBug Fixsubrun file size
Please be aware of mlogger.c update rev 4566 on Sept 23rd 2009, when Stefan
fixed a buglet in the subrun file size computations. Before this fix, the first
subrun could be of a short length. If you use subruns, please update your
mlogger to at least rev 4566 (or newer, Stefan added the run and subrun time
limits just recently).
K.O.
  1793   26 Jan 2020 Konstantin OlchanskiBug Reportsupport for mbedtls - get an open ssl error while trying to compile Midas
> >
> > .../c++  ...  /opt/local/lib/libssl.dylib
> >
> 
> I get the same, libssl from /opt/local, so we are not using openssl shipped with mac os.
> 

I note that latest mongoose 6.16 finally has a virtual ssl layer and appears to support mbedtls (polarssl) in addition to openssl.

I now think I should see if it works - as it gives us a way to support https without relying on the user having
pre-installed working openssl packages - we consistently run into problems with openssl on macos, and even
on linux there was trouble with preinstalled openssl packages and libraries.

With mbedtls, one will have to "git pull" and "make" it, but historically this causes much less trouble.

Also, with luck, mbedtls has better support for certificate expiration (I would really love to have openssl report an error
or a warning or at least some hint if I feed it an expired certificate) and (gasp!) certbot integration.

K.O.
  1794   26 Jan 2020 Konstantin OlchanskiBug Reportsupport for mbedtls - get an open ssl error while trying to compile Midas
> ... support for certbot

The certbot tool to use instead of certbot is this: https://github.com/ndilieto/uacme

K.O.
  1795   28 Jan 2020 Berta BeltranBug Reportsupport for mbedtls - get an open ssl error while trying to compile Midas
> > ... support for certbot
> 
> The certbot tool to use instead of certbot is this: https://github.com/ndilieto/uacme
> 
> K.O.


HI Stefan and Konstantin, 

Thanks a lot for your messages. Sorry for my late reply, I only work on this project from Tuesday to Thursdays. I have 
run " make cmake" instead of "cd build; cmake" and this is the output regarding mhttpd:

/Library/Developer/CommandLineTools/usr/bin/c++  -O2 -g -DNDEBUG -Wl,-search_paths_first -Wl,-
headerpad_max_install_names  CMakeFiles/mhttpd.dir/mhttpd.cxx.o CMakeFiles/mhttpd.dir/mongoose6.cxx.o 
CMakeFiles/mhttpd.dir/mgd.cxx.o CMakeFiles/mhttpd.dir/__/mscb/src/mscb.cxx.o  -o mhttpd  -lsqlite3 ../libmidas.a 
/usr/lib/libssl.dylib /usr/lib/libcrypto.dylib -lz -lsqlite3 /usr/lib/libssl.dylib /usr/lib/libcrypto.dylib -lz 
Undefined symbols for architecture x86_64:
  "_OPENSSL_init_ssl", referenced from:
      _mg_mgr_init in mongoose6.cxx.o
  "_SSL_CTX_set_options", referenced from:
      _mg_set_ssl in mongoose6.cxx.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)


I see that in your outputs the openssl libs are in /opt/local/lib/ while mine are in /usr/lib/, and that is the only difference. 
I have checked that the libraries libssl.dylib and libcrypto.dylib are in my /usr/lib/, and indeed they are, so I don't 
understand the reason for the error, I will continue investigating. 

Thanks 

Berta 
  1804   02 Feb 2020 Konstantin OlchanskiBug Reportsupport for mbedtls - get an open ssl error while trying to compile Midas
> I only work on this project from Tuesday to Thursdays.

No problem. No hurry.

> I have run " make cmake" instead of "cd build; cmake" and this is the output regarding mhttpd: ...

There should be also a line where mhttpd.cxx is compiled into mhttpd.o, we need to see what compiler
flags are used - I suspect the compiler uses header files from /usr/local/include while the linker
is using libraries from /usr/lib, a mismatch.

To save time, please attach the full output of "make cmake". There may be something else I want to see there.

If you do not use the mhttpd built-in https support (for best security I recommend using https from the apache httpd password protected https proxy),
then it is perfectly fine to build midas with NO_SSL=1.

K.O.
  343   11 Feb 2007 Konstantin OlchanskiInfosvn and "make indent" trashed my svn checkout tree...
Fuming, fuming, fuming.

The combination of "make indent" and "svn update" completely trashed my work copy of midas. Half of 
the files now show as status "M", half as status "C" ("in conflict"), even those I never edited myself (e.g. 
mscb firmware files).

I think what happened as that once I ran "make indent", the indent program did things to the source 
files (changed indentation, added spaces in "foo(a,b,c); --> foo(a, b, c);" etc, so now svn thinks that I 
edited the files and they are in conflict with later modifications.

I suggest that nobody ever ever ever should use "make indent", and if they do, they should better 
commit their "changes" made by indent very quickly, before their midas tree is trashed by the next "svn 
update".

And if they commit the changes made by "make indent", beware that "make indent" is not idempotent, 
running it multiple times, it keeps changing files (keeps moving some dox comments around).

Also beware of entering a tug-of-war with Stefan - at least on my machines, my "make indent" seems 
to produce different output from his.

Still fuming, even after some venting...
K.O.
  249   23 Mar 2006 Sergio BallestreroInfosvn@savannah.psi.ch down ?
 Hi,
I was trying to update the checkout of Midas, but it looks like something is not
working - maybe a component of the Savannah system:
[sergio@daq-pc midas-SVN]$ svn update
svn@savannah.psi.ch's password: svn
unix dgram connect: Connection refused at /bin/cvssh.pl line 32
no connection to syslog available at /bin/cvssh.pl line 32
svn: Connection closed unexpectedly

my .svn/entries says (amongst the rest)
 url="svn+ssh://svn@savannah.psi.ch/afs/psi.ch/project/meg/svn/midas/trunk"
and yes, it used to work well... 

Cheers,
  Sergio
  250   26 Mar 2006 Stefan RittInfosvn@savannah.psi.ch down ?
>  Hi,
> I was trying to update the checkout of Midas, but it looks like something is not
> working - maybe a component of the Savannah system:
> [sergio@daq-pc midas-SVN]$ svn update
> svn@savannah.psi.ch's password: svn
> unix dgram connect: Connection refused at /bin/cvssh.pl line 32
> no connection to syslog available at /bin/cvssh.pl line 32
> svn: Connection closed unexpectedly
> 
> my .svn/entries says (amongst the rest)
>  url="svn+ssh://svn@savannah.psi.ch/afs/psi.ch/project/meg/svn/midas/trunk"
> and yes, it used to work well... 
> 
> Cheers,
>   Sergio

I just tried now and it seemed to work fine. Do you still have the problem?

- Stefan
  251   27 Mar 2006 Sergio BallestreroInfosvn@savannah.psi.ch down ?
> I just tried now and it seemed to work fine. Do you still have the problem?
> 
> - Stefan

 The problem was still there this morning, shortly after seeing your mail, but seems
to be fixed now.
 BTW, which is the best way to submit patches ? I have a version of khyt1331 for Linux
kernel 2.6 (we are running Scientific Linux 4.1), and a few smaller things, mostly in
the examples. 

 Thanks, Sergio
  1815   10 Feb 2020 Konstantin OlchanskiSuggestionswitch midas to c++ threads?
Hi, Stefan & co - now that midas is c++11 and c++11 comes with a threads library, should we 
switch midas to use the c++11 threads instead of pthreads? (Of course on Linux c++11 
threads are a layer on top of pthreads, the best I know).

This should remove the dependency on pthreads.h and use a more native implementation of 
threads on MacOS and Windows. (again, the best I can tell).

Of course this depends on c++11 threads having all the functions we need. Specifically, "lock 
with timeout" is useful to deal with "gah! everything stopped! what do I do!", a problem 
bedeviling midas in the early days (and still happens today!). Current midas kills everything 
after 5 minutes of deadlock - then the user knows how to restart everything and the developer 
has core dumps to look at. (to see which program/thread holds the lock and would not give it 
up).

Any thoughts on this?

K.O.
ELOG V3.1.4-2e1708b5