Index: Makefile
===================================================================
--- Makefile    (revision 4129)
+++ Makefile    (working copy)
@@ -270,7 +270,7 @@

 OBJS =  $(LIB_DIR)/midas.o $(LIB_DIR)/system.o $(LIB_DIR)/mrpc.o \
        $(LIB_DIR)/odb.o $(LIB_DIR)/ybos.o $(LIB_DIR)/ftplib.o \
-       $(LIB_DIR)/mxml.o $(LIB_DIR)/cnaf_callback.o \
+       $(LIB_DIR)/mxml.o \
        $(LIB_DIR)/history.o $(LIB_DIR)/alarm.o $(LIB_DIR)/elog.o

 ifdef NEED_STRLCPY

--- Makefile    (revision 4129)
+++ Makefile    (working copy)
-       $(LIB_DIR)/mxml.o $(LIB_DIR)/cnaf_callback.o \
+       $(LIB_DIR)/mxml.o \
> i.e. remove cnaf_callback.o which causes the link errors.


Hi, Denis - I confirm that cnaf_callback.c is only used by MIDAS frontends that implement CAMAC
functions and that it should not required for building the MIDAS library. I am now looking at removing 
it from libmidas.

> I propose that libmidas.so is built by default, so when something breaks it won't go unnoticed

We have been through this before and decided that shared libraries are bad and we do not want to use 
them. The option for building libmidas.so was preserved, though.

Not to refight old wars, on reason against using shared libraries was version skew - one could never be 
sure what version of midas is being used - depending on the PATH, LD_LIBRARY_PATH, rpath settings, 
etc. There were other reasons, perhaps practical, perhaps with the mserver.

The main problem with "just build it", is that then the rest of midas will link against it bringing back all 
the problems we solved by going away from using shared libraries.

So back to your proposal about building libmidas.so - can you look and see if you can do the Python 
bindings with a statically linked midas library?

I know it is possible with Perl bindings - perl creates it's own shared library containing perl api glue 
linked against a foreign static library libfoo.a , so in theory, the shared library is not needed.

But perhaps, Python do things differently...

K.O.

Having libmidas.so is absolutely necessary for pymidas to work. If there was no such
option in Makefile pymidas users would have to build it themselves.

What I proposed though is that you change Makefile so it builds libmidas.so in
addition to (not instead of) static library. So if someone prefer to build
binaries statically they
may continue to do so. On other hand when someone needs a shared library they won't 
discover that it can't be easily built.

Hello, 

The UCN group at TRIUMF has used the history system extensively with a MySQL backend. They see long wait times (up to 20 mins) to add new logged variables to MIDAS equipment. I did some testing and it seems to be a MySQL issue that is fixed in recent versions (UCN uses an older version of MySQL). 

Likely no update is needed to MIDAS source. 


Derek 

Hi,

On the DAQ system for T2K's ND280 near detector, we use deferred
transitions to make sure all triggered events were logged before issuing run
stops to frontends.

I've recently managed to update the frontends to use a
relatively modern version of MIDAS. I then noticed that run transitions are now
by default multithreaded, when issued from e.g. mhttpd, but deferred transitions
called by cm_check_deferred_transition are still performed synchronously.

It
would be nice to make run stops use multithreaded transitions as well. A naive
patch of adding the TR_MTHREAD flag does not work, since the client handling the
deferred transition attempts to communicate with itself instead of calling
cm_transition_call_direct.

After looking into the code a bit further, I noticed
that there is an intentional check against multithreaded transitions in the
logic for determining whether the client is the one calling the transition:

https://bitbucket.org/tmidas/midas/src/fd71f63c023b7e2d4a5c91e3121651b14bd9d27b/
src/midas.cxx#lines-5009

Was there a particular concern that lead to this
particular check?


Regards,
Yiwen.

Multithreaded transitions were introduced by KO in 2019. Please ask him to make deferred transitions work 
again or simply use non-multithreaded transitions. 

Stefan

> Multithreaded transitions were introduced by KO in 2019. Please ask him to make deferred transitions work 
> again or simply use non-multithreaded transitions. 

Deferred transition is the bane of MIDAS, I personally can never understand how they work
and what they do, and I studied them and understood them many times now.

I recommend against using them. Maybe you can explain what you do and I can suggest a way
to avoid using the deferred transition.

Some other people use deferred transitions, and it works for them,
in conjunction with normal transitions, which have been multithreaded
for years.

So unlikely this is a new bug.

P.S. I do not remember any use of deferred transition in the T2K/ND280 FGD, TPC and GSC frontends,
maybe it is in some other subsystem or was introduced after my time.

K.O.

> I recommend against using them. Maybe you can explain what you do and I can suggest a way
> to avoid using the
deferred transition.

I've only recently picked up the code and I'm not sure how it was envisioned when
initially designed, so here's my guess at how it's being used.

Deferred transition is registered by the FPN00
(main clock) process to make sure the logger has finished logging all events before continuing to stop readout
of other frontends. There is also a timeout, in case sometimes an event goes missing, to proceed with the run
stop without getting stuck waiting for the logger.

If there is a better way to implement this then I'm happy to
give it a go.

> P.S. I do not remember any use of deferred transition in the T2K/ND280 FGD, TPC and GSC
frontends,
> maybe it is in some other subsystem or was introduced after my time.

This is all from the global
DAQ code, so indeed none of the other subsystems' frontends use the feature directly. But if the clock module
frontend is started then run stops should be deferred.

Interestingly, I just checked and this feature is
disabled in the FGD ODB, but it is enabled in the TPC ODB.

Regards,
Yiwen.

> Deferred transition is registered by the FPN00

FPN00, yes, this rings a bell.

> (main clock) process to make sure the logger has finished logging all events before continuing to stop readout
> of other frontends. There is also a timeout, in case sometimes an event goes missing, to proceed with the run
> stop without getting stuck waiting for the logger.

yes, that's right, if trigger control and run control are in the same frontend, you run into this trouble:

user pushes run stop button
you get the run stop callback
you stop the trigger
but have to wait for all the data to flush down the pipes all the way to mlogger
but cannot wait, must return from the callback otherwise run transition is stuck

if trigger control and run control are in different frontends, things are simpler:

user pushes run stop botton
trigger control frontend disables the trigger, returns without waiting for anything
data frontend (i.e. FGD, TPC), flush all the hardware FIFOs, etc to MIDAS SYSTEM buffer (trigger is already 
disabled, there is no new data)
mlogger flushes SYSTEM buffer to disk
run control frontend reports successful run stop to the run database (and whatever else).

> Interestingly, I just checked and this feature is
> disabled in the FGD ODB, but it is enabled in the TPC ODB.

I do not have access to the current code, but I can check what I have,
and I am pretty sure it did not have any deferred transitions. Maybe
it was added after my time.

K.O.

Dear all,

I was wondering if one can use multiple pysequencers at the same time and if not
if this feature is planned in the future? Our experiment (mu3e) has one frontend
per detector and each sequencer would only access a subset of ODB entries.

Best,
Marius

> I was wondering if one can use multiple pysequencers at the same time and if not
> if this feature is planned in the future? Our experiment (mu3e) has one frontend
> per detector and each sequencer would only access a subset of ODB entries.

I believe sequencer design supports this, same as mlogger supports multiple output 
files and mhttpd supports multiple listener ports.

But how to do this with pysequencer, I am not sure. Simplest is to wait a few more 
weeks for Ben to return from vacation.

K.O.

[mhist,ERROR] [midas.cxx:5949:bm_validate_client_index,ERROR] My client index 10 in buffer 'SYSMSG' 
is invalid: client name '', pid 0 should be my pid 3773321
[mhist,ERROR] [midas.cxx:5952:bm_validate_client_index,ERROR] Maybe this client was removed by a 
timeout. See midas.log. Cannot continue, aborting...
Aborted (core dumped)

mhist -e "Xenon" -l

$ mhist -e "Xenon" -v "Det XeTmp 0-0" -t 100000 -s 200101 -p 250101 -f 
/data2/history/2022/mhf_1644698398_20220212_xenon.dat
ID 980316009, Aug 13 19:10:56, size 1851749486

• Bug: mhist -l crashes
  
• Bug: mhist -f does not work with "FILE" history format
  
• Feature request: mhist -e "Name" -l to only show variables of event "Name"
  
• Feature request: Set temporary history dir with a flag
  

I think I can offer some insight into your problems:

1) your mhist crash is due to the ODB timeout, it is probably set to 30 seconds in ODB /programs/mhist. you will 
have to make it biigger.

2) 1.5 years of files. yes. I have 10 years of files for ALPHA at CERN. and the number of files is a problem. 
But it should be better than the old system with 3 files per day (1000 files per year).

One solution you can try is symlinks. Assuming you have 10 years of history files in 10 per-year directory, you 
symlink as many of them as you need into the "current" directory, then remove the symlinks.

Why remove the symlinks? I use "ls" to read the list of history files and Unix/Linux does not have a syscall to 
"give me the 100 files with the newest mtime". I have to read the whole directory and that takes forever (if ZFS 
on HDD), it is quick with ZFS on SSD if ZFS cache is hot (you can have a cron job do "ls" every 5 minutes to 
keep the ZFS cache hot).

Now that I wrote the above, I think I see a way to make it "automatic", let me ponder this. (plus I always 
wanted to implement compressed history files (using "free" lz4)).

K.O.



I am having some trouble with mhist. I suppose that the problems are at least partially due to our specific 
needs which might exceed what has been tested. For context, in MEG II we have some 10^4 history variables in ~30 
different events. 

1. mhist -l crashes. After displaying around 7000 lines, I get the following error message:
[CODE]
[mhist,ERROR] [midas.cxx:5949:bm_validate_client_index,ERROR] My client index 10 in buffer 'SYSMSG' 
is invalid: client name '', pid 0 should be my pid 3773321
[mhist,ERROR] [midas.cxx:5952:bm_validate_client_index,ERROR] Maybe this client was removed by a 
timeout. See midas.log. Cannot continue, aborting...
Aborted (core dumped)
[/CODE]
Timing the execution shows around 33 seconds before the process is aborted.

I'm not sure if this would actually fix the problem, but while trying to circumvent the issue, I tried the 
following: [CODE]mhist -e "Xenon" -l[/CODE] This doesn't seem to be implemented. Listing only the variables of a 
single event would be nice 
regardless of our specific issue.

2. mhist and history files.
We have a directory directory with about 2500 history files (mhf_...dat) for the past 1.5 years. Older 
history files are archived in other directories with similar numbers of files. When trying to access them, I 
encountered two issues:
It seems like it is not possible to pass a "history directory" as an argument. To dump the history for a full 
year in the archive directory, I would need to run mhist many times with -f and then combine all the dumps.

If it really does not work, please consider this a feature request.
Also, even using single files does not work at the moment:
[CODE]
$ mhist -e "Xenon" -v "Det XeTmp 0-0" -t 100000 -s 200101 -p 250101 -f 
/data2/history/2022/mhf_1644698398_20220212_xenon.dat
ID 980316009, Aug 13 19:10:56, size 1851749486
[/CODE]
This command was supposed to show me the rough time frame covered in this particular history file. I was 
informed that the history files are in the new "FILE" format and mhist might not work with them properly.

tl;dr
[LIST]
[*] Bug: mhist -l crashes
[*] Bug: mhist -f does not work with "FILE" history format
[*] Feature request: mhist -e "Name" -l to only show variables of event "Name"
[*] Feature request: Set temporary history dir with a flag
[/LIST]

Lukas[/quote]

$ grep -n "class SqlODBC" src/*
src/history_odbc.cxx:282:class SqlODBC: public SqlBase
src/history_sql.cxx:293:class SqlODBC: public SqlBase

Linking CXX shared library lib/libmidas.so
/usr/bin/cmake -E cmake_link_script CMakeFiles/midas-shared.dir/link.txt --verbose=1
/usr/bin/c++  -fPIC   -shared -Wl,-soname,libmidas.so -o lib/libmidas.so CMakeFiles/midas-shared.dir/src/ftplib.c.o CMakeFiles/midas-shared.dir/src/midas.c.o CMakeFiles/midas-shared.dir/src/system.c.o CMakeFiles/midas-shared.dir/src/mrpc.c.o CMakeFiles/midas-shared.dir/src/odb.c.o CMakeFiles/midas-shared.dir/src/ybos.c.o CMakeFiles/midas-shared.dir/src/history.c.o CMakeFiles/midas-shared.dir/src/alarm.c.o CMakeFiles/midas-shared.dir/src/elog.c.o CMakeFiles/midas-shared.dir/opt/DAQ/repos/bot/mxml/mxml.c.o CMakeFiles/midas-shared.dir/opt/DAQ/repos/bot/mxml/strlcpy.c.o CMakeFiles/midas-shared.dir/src/history_odbc.cxx.o CMakeFiles/midas-shared.dir/src/history_sql.cxx.o 
CMakeFiles/midas-shared.dir/src/history_sql.cxx.o: In function `SqlODBC':
/opt/DAQ/repos/bot/midas/src/history_sql.cxx:200: multiple definition of `SqlODBC::SqlODBC()'
CMakeFiles/midas-shared.dir/src/history_odbc.cxx.o:/opt/DAQ/repos/bot/midas/src/history_odbc.cxx:315: first defined here
...
history_odbc.cxx:727: first defined here
collect2: ld returned 1 exit status
make[2]: *** [lib/libmidas.so] Error 1

> Linking CXX shared library lib/libmidas.so
/usr/bin/c++  ... -o lib/libmidas.so ... CMakeFiles/midas-shared.dir/src/history_odbc.cxx.o 
CMakeFiles/midas-shared.dir/src/history_sql.cxx.o 
CMakeFiles/midas-shared.dir/src/history_sql.cxx.o: In function `SqlODBC':
/opt/DAQ/repos/bot/midas/src/history_sql.cxx:200: multiple definition of `SqlODBC::SqlODBC()'
CMakeFiles/midas-
shared.dir/src/history_odbc.cxx.o:/opt/DAQ/repos/bot/midas/src/history_odbc.cxx:315: first defined 
here
> Why is the class "SqlODBC" duplicated?

This is interesting. I do not think my C++ book spells it out that I cannot have class A in foo.cxx
and a class A in bar.cxx. In fact I already discovered that the two classes A will collide (duplicate default 
constructor A::A, even if all other member functions are different) if I link an executable that uses both 
foo.o and bar.o.

Is there a way around this problem? In C, I can declare functions and variables "static" to "hide" them
from the linker.

What is the C++ equivalent for classes? (Any C++ experts here?)

In this specific case, the file history_odbc.cxx will disappear with the next commit of mhttpd, hopefully 
today. mlogger and mh2sql already do not use it.

And I will commit the Makefile change renaming libmidas.so to libmidas-shared.so, so we can build it 
by default but still link midas core executables to the normal midas library.

This will catch such a problem if it happens again.

K.O.

> > Why is the class "SqlODBC" duplicated?
> 
> This is interesting. I do not think my C++ book spells it out that I cannot have class A in foo.cxx
> and a class A in bar.cxx.

I guess nobody knows the answer to this C++ puzzle. In any case history_odbc.cxx is not used anymore removing duplication of class SqlODBC.

svn rev 4594
K.O.

Dear Forum,

I have this beautiful Intel Quadcore with fast disks, but MIDAS does obviously 
only make use of one CPU at a time. Has anyboy of you already done some work 
on making MIDAS parallel? Event-based data analysis should be the best 
candidate for this.

Has anybody done this with PVM? There is some PVM-related stuff in the MIDAS 
sources, but I got the impression this works only with HBOOK, not with ROOT. 
Or am I wrong?
But then PVM is probably also not the most efficient thing one ONE machine 
with multiple CPUs, right? And finally, with PVM we're back to 
adding .root-files efficiently (see my previous post).

Any thoughts?

Cheers,

Randolf

> I have this beautiful Intel Quadcore with fast disks, but MIDAS does obviously 
> only make use of one CPU at a time. Has anyboy of you already done some work 
> on making MIDAS parallel? Event-based data analysis should be the best 
> candidate for this.

There are ring buffer routines rb_xxx for distributed event analysis, but this is
currently only implemented in the front-end framework. These routines are pretty
simple, and their integration into the analyzer should not be very difficult.
Unfortunately I don't have time for that right now. We do our analysis such that we
analyze four different runs in parallel on a quadcore machine.

- Stefan

For the record, here's my report of difficulties getting mongoose to compile with macos.  This is a similar 
problem reported before, but with slightly different error messages.  So I put them here for posterity.

Setup: 
- macos: 10.13.6
- xcode: 9.2
- gcc: Thomass-MacBook-Pro-3:build lindner$ gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-
dir=/usr/include/c++/4.2.1
Apple LLVM version 9.0.0 (clang-900.0.39.2)
Target: x86_64-apple-darwin17.7.0
- midas: today's version

Start with the openssl version I had already installed.  cmake says 

-- Found OpenSSL: /usr/lib/libcrypto.dylib (found version "1.0.2s") 
-- MIDAS: Found OpenSSL version 1.0.2s

make install fails with this error:

[ 35%] Linking CXX executable mhttpd
cd /Users/lindner/packages/midas/build/progs && /usr/local/Cellar/cmake/3.15.0/bin/cmake -E 
cmake_link_script CMakeFiles/mhttpd.dir/link.txt --verbose=1
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++  -O2 -g -
DNDEBUG -Wl,-search_paths_first -Wl,-headerpad_max_install_names  CMakeFiles/mhttpd.dir/mhttpd.cxx.o 
CMakeFiles/mhttpd.dir/mongoose616.cxx.o CMakeFiles/mhttpd.dir/mgd.cxx.o 
CMakeFiles/mhttpd.dir/__/mscb/src/mscb.cxx.o  -o mhttpd ../libmidas.a /usr/lib/libssl.dylib 
/usr/lib/libcrypto.dylib -lz -lcurl -lsqlite3
Undefined symbols for architecture x86_64:
  "_SSL_CTX_set_psk_client_callback", referenced from:
      _mg_ssl_if_conn_init in mongoose616.cxx.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Use macports to upgrade to newer openssl

sudo port selfupdate
sudo port upgrade outdated

Now cmake says

-- Found OpenSSL: /usr/lib/libcrypto.dylib (found version "1.1.1g") 
-- MIDAS: Found OpenSSL version 1.1.1g

Error message is different now:

cd /Users/lindner/packages/midas/build/progs && /usr/local/Cellar/cmake/3.15.0/bin/cmake -E 
cmake_link_script CMakeFiles/mhttpd.dir/link.txt --verbose=1
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++  -O2 -g -
DNDEBUG -Wl,-search_paths_first -Wl,-headerpad_max_install_names  CMakeFiles/mhttpd.dir/mhttpd.cxx.o 
CMakeFiles/mhttpd.dir/mongoose616.cxx.o CMakeFiles/mhttpd.dir/mgd.cxx.o 
CMakeFiles/mhttpd.dir/__/mscb/src/mscb.cxx.o  -o mhttpd ../libmidas.a /usr/lib/libssl.dylib 
/usr/lib/libcrypto.dylib -lz -lcurl -lsqlite3
Undefined symbols for architecture x86_64:
  "_OPENSSL_init_ssl", referenced from:
      _mg_mgr_init_opt in mongoose616.cxx.o
      _mg_ssl_if_init in mongoose616.cxx.o
  "_SSL_CTX_set_options", referenced from:
      _mg_ssl_if_conn_init in mongoose616.cxx.o
  "_SSL_CTX_set_psk_client_callback", referenced from:
      _mg_ssl_if_conn_init in mongoose616.cxx.o
ld: symbol(s) not found for architecture x86_64

Fine.  Doing 'cmake -D NO_SSL=1 ..' to build still works fine; I will stick with that since I don't need SSL on my 
laptop.

Perhaps we should disable SSL by default on Macos?  People may only have ~50% chance of getting it to 
work.
  

> For the record, here's my report of difficulties getting mongoose to compile with macos. 
> -- MIDAS: Found OpenSSL version 1.0.2s
> -- MIDAS: Found OpenSSL version 1.1.1g
> ... [ all of them did not work ]

For the record, I get this on mac os 10.15.4 and it works.
-- MIDAS: Found OpenSSL version 1.1.1g

I think I am quite fed up with this openssl business, too.

What I will do in MIDAS is fix the mbedtls detection, add mbedtls instructions
in the documentation and remove openssl from mhttpd build.

Result will be:
- default build will have mhttpd without https support, and this works in 100% of our use cases at TRIUMF.
- if user do not want to use apache https proxy, they have to "git clone" mbedtls, build it, rebuild mhttpd, then
they get https support, but for https certificate management - getting them, renewing them, etc, they are
on their own.

Since mhttpd has no integration with certbot, automatic management of https certificates does not work,
so good luck again.

In theory, I can try to add certbot integration, but even the most basic tools are missing, for example, openssl
does not report certificate expiration dates (I guess I must write my own code to examine the certificate
and hope my idea of expiration matches their idea). Since I do not see certificate expiration dates, every day I could
blindly run "certbot renew" and restart openssl with the updated certificate (but I think openssl does
not have a "restart" function, so again, forget about it). Adding insult to injury, by default, certbot stores certificates
in a secret location in /etc where mhttpd cannot access them.

Bottom line is that powers-that-be messed up https certificate management and until that is sorted out and is easy
to integrate with custom web servers, I can only recommend that mhttpd must run behind the "OS support https proxy".

K.O.