Back Midas Rome Roody Rootana
  Midas DAQ System, Page 97 of 142  Not logged in ELOG logo
ID Date Authordown Topic Subject
  1563   18 Jun 2019 Konstantin OlchanskiBug Fixremoved modbset() from mhttpd.js
> Just to make this point clear: The "write-to-odb-read-via-hotlink" was never meant to guarantee the receiving side to see each change. If changes happen too often, updates might get lost. If one relies on the 
> sequence of updates, one should use direct RPC calls to the frontend or use a midas buffer and encode updates in events.

I recommend that people use the jrpc mechanism that does an RPC directly from javascript into the frontend.
It passes 2 strings as arguments (command and data value). Arbitrary objects can be passed by encoding
the data in json (use mjson.h to decode it in the frontend). A string is returned back to javascript (again, encode
arbitrary data as json, use the mjson.h library).

Call sequence:
javascript -> (http) -> mhttpd -> (MIDAS RPC call) -> frontend -> (write, read, frob hardware) -> frontend -> (MIDAS RPC reply) -> mhttpd -> (http reply) -> javascript

Example of all this is in example.html and fetest.cxx:

javascript side code: mjsonrpc_call("jrpc", { "client_name":"fetest", "cmd":"xxx", "args":"xxx" })

frontend side code:

INT rpc_callback(INT index, void *prpc_param[])
{
   const char* cmd  = CSTRING(0);
   const char* args = CSTRING(1);
   char* return_buf = CSTRING(2);
   int   return_max_length = CINT(3);
   cm_msg(MINFO, "rpc_callback", "--------> rpc_callback: index %d, max_length %d, cmd [%s], args [%s]", index, return_max_length, cmd, args);
   ... do stuff ... put result into string "tmp"
   strlcpy(return_buf, tmp, return_max_length);
   return RPC_SUCCESS;
}

... somewhere in frontend_init(), register the RPC:

#ifdef RPC_JRPC
   status = cm_register_function(RPC_JRPC, rpc_callback);
   assert(status == SUCCESS);
#endif

K.O.
  1564   19 Jun 2019 Konstantin OlchanskiReleasemidas-2019-06 with cmake and c++
We are happy to the midas release "midas-2019-06" with the build system implemented in cmake and the midas, mxml and mscb 
projects switched to C++.

Changes since midas-2019-03:

minor bug fixes
switch of midas build to c++ with c++ linkage (no "extern C")
switch of midas build to cmake
removal of $(OS_DIR) from the midas library and bin paths (use $MIDASSYS/lib instead of $MIDASSYS/linux/lib)
mxml and mscb are implemented as git submodules

Please review the following guide to update midas from previous release midas-2019-03 or older.

Update the code:

git checkout develop
git pull
git checkout feature/midas-2019-06
git pull
git submodule update --init # this will checkout correct versions of mxml and mscb
make clean
make cclean
rm -rf linux/bin
rm -rf linux/lib
rmdir linux
make cmake3 # or "make cmake" on ubuntu and macos
ls -l bin/odbedit bin/mlogger

Update experiment environment:

- change PATH from $MIDASSYS/linux/bin to $MIDASSYS/bin

Cleanup unneeded stuff:

- remove $HOME/packages/mxml (new location $MIDASSYS/mxml)
- remove $HOME/packages/mscb (new location $MIDASSYS/mscb)

Update experiment frontend build:

- change Makefile to remove $(OS_DIR) from library search path ($MIDASSYS/linux/lib becomes $MIDASSYS/lib)
- change Makefile to set mxml include path from $MIDASSYS/../mxml to $MIDASSYS/mxml (to avoid including the wrong 
version of mxml/strlcpy.h)
- update frontend code to use mfe.h and build as C++, see https://midas.triumf.ca/elog/Midas/1526

K.O.
  1569   25 Jun 2019 Konstantin OlchanskiBug ReportERROR INSTALLING 32BIT MIDAS LIBRARIES ON 64BIT HOST MACHINE
Yikes, the error is in the CRC library. The assembly-optimized crc32c function fails to build, and the 
error does not look familiar to me. I do not see this error here. What is your host system ("uname -
a") and what is your gcc ("gcc -v")?

BTW, "make linux32" will build an Intel 32-bit version (see "-m32" in "man gcc"). For ARM 32-bit 
you need a different switch, I think, also depending how you are cross-compiling it.

For straight cross-compilation, look at the Makefile target "make linuxarm" (you will need to change 
the location of your ARM gcc cross-compiler).

For running MIDAS frontend on the Raspberry Pi 3, I build MIDAS on the Pi3 itself, the machine is big 
enough to run CentOS7 linux and gcc to build the full MIDAS.

But if you have a different cross-compilation scheme, I am happy to help you and to add your 
scheme to the MIDAS Makefile. We can start by looking at "uname -a" and "gcc -v" and "lsb_release 
-a" (if you have it).

K.O.


> Hi, we are part of the Mu3e research based at University of Bristol. We have a 
> remote 32 bit frontend (raspberry pi) connected to a 64 bit Data Acquisition 
> system.we are following the instructions at installation/quickstart linux/Build 
> 32-bit MIDAS libraries. when we execute the commands: 
> [mhostpc] cd /home/packages/midas
> [mhostpc] make linux32             
> 
> we get an error:
> 
> make NO_ROOT=1 NO_MYSQL=1 NO_ODBC=1 NO_SQLITE=1 OS_DIR=linux-m32 USERFLAGS=-
m32
> make[1]: Entering directory `/home/hh19285/packages/midas'
> g++ -m32 -c -g -O2 -Wall -Wno-strict-aliasing -Wuninitialized -Iinclude -
> Idrivers -Imxml -Imscb/include -DHAVE_FTPLIB -D_LARGEFILE64_SOURCE -DHAVE_ZLIB -
> DHAVE_MSCB -DHAVE_MONGOOSE6 -DMG_ENABLE_THREADS -DMG_DISABLE_CGI -
DMG_ENABLE_SSL 
> -DOS_LINUX -fPIC -Wno-unused-function -o lib/crc32c.o src/crc32c.cxx
> src/crc32c.cxx: In function ‘uint32_t crc32c_hw(uint32_t, const void*, size_t)’:
> src/crc32c.cxx:283:66: error: ‘asm’ operand has impossible constraints
>                      : "r"(next), "0"(crc0), "1"(crc1), "2"(crc2));
>                                                                   ^
> src/crc32c.cxx:303:66: error: ‘asm’ operand has impossible constraints
>                      : "r"(next), "0"(crc0), "1"(crc1), "2"(crc2));
>                                                                   ^
> src/crc32c.cxx: In function ‘uint32_t crc32c(uint32_t, const void*, size_t)’:
> src/crc32c.cxx:348:34: error: PIC register clobbered by ‘%ebx’ in ‘asm’
>                  : "%ebx", "%edx"); \
>                                   ^
> src/crc32c.cxx:362:5: note: in expansion of macro ‘SSE42’
>      SSE42(sse42);
>      ^
> make[1]: *** [lib/crc32c.o] Error 1
> make[1]: Leaving directory `/home/hh19285/packages/midas'
> make: *** [linux32] Error 2
> 
> Could you please help with getting past this? otherwise we may need to change 
> our whole experimental setup.
> 
> Thank you in advance
  1570   25 Jun 2019 Konstantin OlchanskiForumStrange JS array creation
> --> In the midas back-end I never created an array. I created an INT in the ODB with db_create_key(hDB, 0,
> "Equipment/Switching/Variables/DATA_WRITE", TID_INT). By using modset in javascript and parsing the string
> "/Equipment/Switching/Variables/DATA_WRITE[" + String(i) + "]" I call it like an array and it shows up like an
> array in the ODB.

I think you are good. In ODB, a TID_INT is actually an array of size 1. Writing to an array index automatically
extends the array. I am not sure where this is written down, but this is how most ODB array index access functions
have always worked.

You do have a performance bug with your loop, though, by writing a[1], a[2], a[3], you cause
the array to grow from size 1 to size 2. Then grow it from size 2 to size 3, etc. Of course
this only happens the first time you run the thing. Afterwards, the array has the correct size
and does not need to be grown.

K.O.
  1573   26 Jun 2019 Konstantin OlchanskiForumProblem transferring fetest data from the remote frontend to the backend
> Hi again, we now have Midas installed on the Rpi (remote frontend machine) and
> have managed to run Fetest on it. Now we are at a stage where we want to send
> the Fetest data over to the Data Acquisition machine ...
> 
> Does the mserver have any role in this?
> 

Yes. mserver runs on your daq machine and handles connections from frontends running on frontend machines. It needs to be configured 
correctly before it will work: in odb on your daq machine, non-local rpc has to be enabled and the frontend machine has to be added to the 
midas rpc access control list.

Read this:
https://midas.triumf.ca/MidasWiki/index.php/Quickstart_Linux#Running_with_one_or_more_REMOTE_frontends

And this:
https://midas.triumf.ca/MidasWiki/index.php/Security#MIDAS_programs_on_remote_machines

K.O.
  1576   27 Jun 2019 Konstantin OlchanskiBug ReportGetting an error when trying to compile a frontend file
If the latest midas does not work, try the previous release versions. "git tags" and "git branch -
a" will show you what exists. Look for branch and tag names in the form "midas-YYYY-MM".

As shortcut, the latest release candidate is midas-2019-06, the latest release branch is midas-
2019-03, latest release tag midas-2019-03-h.

Read the messages in this thread for more information:
https://midas.triumf.ca/elog/Midas/1513

>
> When we run the following commands ...
> make[1]: *** [CMakeFiles/frontend.dir/all] Error 2
>

I do not understand cmake well enough to debug this. Falling back to midas-2019-03 may help 
you as it uses normal make and with luck you know how to debug normal Makefiles if you see 
the same problem.

K.O.
  1577   27 Jun 2019 Konstantin OlchanskiBug Reportmake linux32 bombs on el7 in crc32c.c, ERROR INSTALLING 32BIT MIDAS LIBRARIES ON 64BIT HOST MACHINE
Reproduced on el7 (CentOS7). Same thing works on el6 (SL6).

The error is in the SSE4.2-assembly-accelerated library for computing crc32c checksums. I do 
not understand this assembly stuff enough to tell what goes wrong.

In any case, "make linux32" is intended for VME processors that cannot run 64-bit code. These 
processors also happen to not have the SSE4.2 instructions needed for this code to actually 
work, so one solution would be to always disable crc32c SSE4.2-assembly-acceleration for the 
linux32 target.

Note that the original reported was running "make linux32" with the idea of generating code for 
the 32-bit ARM processor.

Here the situation is like this: the required CRC32C instructions are present on 64-bit capable 
ARM processors (RPi3, etc) and probably work in 32-bit mode, and I found the assembly-
language crc32c library that uses them. It needs to be added to MIDAS and tested.

For the older 32-bit-only ARM processors (Cyclone5 FPGA, Rpi2 and older) I do not see any 
hardware accelerated crc32c implementations, so it uses software computed CRC32 always. 
(P.S.: I see the current linux kernels have a hardware accelerated library for CRC32C, not sure if 
it can be imported into MIDAS easily).

K.O.


> $ make linux32 ...
> g++ -m32 -c -g -O2 -Wall -Wno-strict-aliasing -Wuninitialized -Iinclude -
> Idrivers -Imxml -Imscb/include -DHAVE_FTPLIB -D_LARGEFILE64_SOURCE -DHAVE_ZLIB -
> DHAVE_MSCB -DHAVE_MONGOOSE6 -DMG_ENABLE_THREADS -DMG_DISABLE_CGI -
DMG_ENABLE_SSL 
> -DOS_LINUX -fPIC -Wno-unused-function -o lib/crc32c.o src/crc32c.cxx
> src/crc32c.cxx: In function ‘uint32_t crc32c_hw(uint32_t, const void*, size_t)’:
> src/crc32c.cxx:283:66: error: ‘asm’ operand has impossible constraints
>                      : "r"(next), "0"(crc0), "1"(crc1), "2"(crc2));
>                                                                   ^
> src/crc32c.cxx:303:66: error: ‘asm’ operand has impossible constraints
>                      : "r"(next), "0"(crc0), "1"(crc1), "2"(crc2));
>                                                                   ^
> src/crc32c.cxx: In function ‘uint32_t crc32c(uint32_t, const void*, size_t)’:
> src/crc32c.cxx:348:34: error: PIC register clobbered by ‘%ebx’ in ‘asm’
>                  : "%ebx", "%edx"); \
>                                   ^
> src/crc32c.cxx:362:5: note: in expansion of macro ‘SSE42’
>      SSE42(sse42);
>      ^
> make[1]: *** [lib/crc32c.o] Error 1
> make[1]: Leaving directory `/home/hh19285/packages/midas'
> make: *** [linux32] Error 2
> 
> Could you please help with getting past this? otherwise we may need to change 
> our whole experimental setup.
> 
> Thank you in advance
  1580   28 Jun 2019 Konstantin OlchanskiBug Reportmake linux32 bombs on el7 in crc32c.c, ERROR INSTALLING 32BIT MIDAS LIBRARIES ON 64BIT HOST MACHINE
> Reproduced on el7 (CentOS7). Same thing works on el6 (SL6).

Fixed in commit dd937e6. Only enable SSE4.2 crc32c for 64-bit compilation. Still not sure why it worked for 32-bit 
compilation on el6 (SL6).

K.O.


> 
> The error is in the SSE4.2-assembly-accelerated library for computing crc32c checksums. I do 
> not understand this assembly stuff enough to tell what goes wrong.
> 
> In any case, "make linux32" is intended for VME processors that cannot run 64-bit code. These 
> processors also happen to not have the SSE4.2 instructions needed for this code to actually 
> work, so one solution would be to always disable crc32c SSE4.2-assembly-acceleration for the 
> linux32 target.
> 
> Note that the original reported was running "make linux32" with the idea of generating code for 
> the 32-bit ARM processor.
> 
> Here the situation is like this: the required CRC32C instructions are present on 64-bit capable 
> ARM processors (RPi3, etc) and probably work in 32-bit mode, and I found the assembly-
> language crc32c library that uses them. It needs to be added to MIDAS and tested.
> 
> For the older 32-bit-only ARM processors (Cyclone5 FPGA, Rpi2 and older) I do not see any 
> hardware accelerated crc32c implementations, so it uses software computed CRC32 always. 
> (P.S.: I see the current linux kernels have a hardware accelerated library for CRC32C, not sure if 
> it can be imported into MIDAS easily).
> 
> K.O.
> 
> 
> > $ make linux32 ...
> > g++ -m32 -c -g -O2 -Wall -Wno-strict-aliasing -Wuninitialized -Iinclude -
> > Idrivers -Imxml -Imscb/include -DHAVE_FTPLIB -D_LARGEFILE64_SOURCE -DHAVE_ZLIB -
> > DHAVE_MSCB -DHAVE_MONGOOSE6 -DMG_ENABLE_THREADS -DMG_DISABLE_CGI -
> DMG_ENABLE_SSL 
> > -DOS_LINUX -fPIC -Wno-unused-function -o lib/crc32c.o src/crc32c.cxx
> > src/crc32c.cxx: In function ‘uint32_t crc32c_hw(uint32_t, const void*, size_t)’:
> > src/crc32c.cxx:283:66: error: ‘asm’ operand has impossible constraints
> >                      : "r"(next), "0"(crc0), "1"(crc1), "2"(crc2));
> >                                                                   ^
> > src/crc32c.cxx:303:66: error: ‘asm’ operand has impossible constraints
> >                      : "r"(next), "0"(crc0), "1"(crc1), "2"(crc2));
> >                                                                   ^
> > src/crc32c.cxx: In function ‘uint32_t crc32c(uint32_t, const void*, size_t)’:
> > src/crc32c.cxx:348:34: error: PIC register clobbered by ‘%ebx’ in ‘asm’
> >                  : "%ebx", "%edx"); \
> >                                   ^
> > src/crc32c.cxx:362:5: note: in expansion of macro ‘SSE42’
> >      SSE42(sse42);
> >      ^
> > make[1]: *** [lib/crc32c.o] Error 1
> > make[1]: Leaving directory `/home/hh19285/packages/midas'
> > make: *** [linux32] Error 2
> > 
> > Could you please help with getting past this? otherwise we may need to change 
> > our whole experimental setup.
> > 
> > Thank you in advance
  1582   28 Jun 2019 Konstantin OlchanskiBug ReportStatus page reloads every second
> We observed a strange behavior, from our point of view:
> ... the Midas status page started to reload/refresh every second

What version of midas is this? Run the odbedit "ver" command please. Also which
browser on what OS is this? (chrome->about google chrome, firefox->about firefox).

The current versions of midas do not reload the status page ever, and I think
all the page-reload code has been removed and they cannot reload automatically.

Old versions of the midas status page were designed to reload every 60 seconds or so.
The reload interval is adjustable, but I do not think it was stored in ODB. It was
accessed from the status page "config" button and I think it stored the reload
period is a browser cookie.

This reload value may have gotten confused, and in this case, to fix it,
you can try to clear all the web cookies from the web page. Another test for this
would be to try an alternate web browser, which would presumable not have the bad cookie
and will not suffer from the reload problem.

You can also open the web page debugger (google chrome -> right click menu -> inspect ->> 
console & etc) and see if anything shows up there. I think you can set a break point
on the page reload function and catch the place that causes the reload.

K.O.
  1585   02 Jul 2019 Konstantin OlchanskiSuggestionmy_global.h not present in my linux distribution (needed)
Confirmed. my_global.h is removed in MySQL 8.0 (gives a compile error) and deprecated in 
MariaDB 10.2 (gives a #warning).

I removed include of my_global.h, it is not needed on el6, el7 and ubuntu.

Also added explicit support for mariadb via mariadb_config if it exists.

Note that the cmake build does not actually enable mysql, sqlite and odbc - it detects them, but 
does not do anything about it. We will fix this shortly.

K.O.




> Hey,
> 
> while trying to compile Midas under openSUSE 15.0 with mysql support, I was
> running into the problem that somehow the mysql header file my_global.h is not
> included in the packages. This might be a bug that concerns the suse developers
> more, but is it actually needed? Compilation worked fine with the include line
> commented out.
> 
> If it's not needed, I would like to suggest to remove line 735 of
> src/history_schema.cxx (where it's included)
> 
> Cheers
> Lukas
> 
> mysql  Ver 15.1 Distrib 10.2.22-MariaDB, for Linux (x86_64) using  EditLine wrapper
> Also, all mariadb development packages are installed
  1592   05 Jul 2019 Konstantin OlchanskiBug ReportHeader files missing when trying to compile rootana, roody and analyzer
> /home/hh19285/packages/rootana/include/TRootanaEventLoop.hxx:24:25: fatal error:
> THttpServer.h: No such file or directory
>  #include "THttpServer.h"
>
> include/TPeakFindPanel.h:46:23: fatal error: TSpectrum.h: No such file or directory
>  #include "TSpectrum.h"
>

Your ROOT is strange, missing some standard features. Also installed in a strange place, /usr/include/root.

Did you install ROOT from the EPEL RPM packages? In the last I have seen this ROOT built very strangely, with some standard features disabled for no obvious 
reason.

For this reason, I recommend that you install ROOT from the binary distribution at root.cern.ch or build it from source.

For more debugging, please post the output of:
which root-config
root-config --version
root-config --features
root-config --cflags

For reference, here is my output for a typical CentOS7 machine:
daq16:~$ which root-config
/daq/daqshare/olchansk/root/root_v6.12.04_el74_64/bin/root-config
daq16:~$ root-config --version
6.12/04
daq16:~$ root-config --features
asimage astiff builtin_afterimage builtin_ftgl builtin_gl2ps builtin_glew builtin_llvm builtin_lz4 builtin_unuran cling cxx11 exceptions explicitlink fftw3 gdmlgenvector 
http imt mathmore minuit2 opengl pch pgsql python roofit shared sqlite ssl thread tmva x11 xft xml
daq16:~$ root-config --cflags -pthread -std=c++11 -m64 -I/daq/daqshare/olchansk/root/root_v6.12.04_el74_64/include

The important one is the --features, see that "http" and "xml" are enabled. "spectrum" used to be an optional feature, I do not think it can be disabled these 
days, so your missing "TSpectrum.h" is strange. (But I just think the EPEL ROOT RPMs are built wrong).

K.O.
  1593   05 Jul 2019 Konstantin OlchanskiInfoLimitations of MSL
> I am missing a few features.

MSL did not start out as a fully featured programming language.

Rather than extending it until it becomes one, I think a better approach would be to take
one of the existing extensible scripting language libraries and extend it with midas functions.

For example, LUA (https://www.lua.org/about.html) seems to be popular.

There were also requests for a midas extension for PYTHON (actually this has been done before,
but never added to the midas repository. I still have that code and I suppose it could be resurrected).

K.O.


> Do any of the following exist and I have just
> overlooked them?
> 
> - Arithmetics:
>     SET foo, 2
>     SET bar, 2
>     SET FOOBAR, $foo + $bar
>     -> FOOBAR is 2, not 4
> 
> - Vectors/arrays
>     As far as I understand, "SET" only allows for single variables and no 
>     "scripting" in variable names, i. e. doing something like the following
>     scripted: 
>     SET var_1, 0
>     SET var_2, 0
>     ...
>     LOOP n, 10
>         ...
>         IF something
>             SET var_${n}, 1
>         ENDIF 
>     
>     The only way I could think of doing something similar is via the ODB.
>     I don't know if it's good practice to fill the ODB with variables like this.
> 
> - Loading scripts at run time (see other bug)
>     That would allow for script manipulation, even though it's kind of dirty, it
>     might be useful in some cases.
> 
> - Obtaining results from external scripts
>     A way to pass things from external scripts would be the solution to all
>     problems. Something that is not implemented could be done in a bash or
>     python script instead. 
> 
> Cheers
> Lukas
  1598   08 Jul 2019 Konstantin OlchanskiBug ReportFrontend killed at stop of run
> After a long overnight run to check that the frontend runs smoothly for a longer
> time, I stopped the run and the frontend was killed by midas.

run the frontend inside gdb and post the stack trace after the crash?

if there is no crash (the program is stopped by exit()), you may need
to set a breakpoint in exit() or _exit() (not sure what it's latest name is)
then with luck your stack trace will show who/what called it from where.

if it is hard to start the frontend inside gdb, you can start it normally,
and attach gdb later, using a "gdb frontend.exe pid" command.

K.O.



> I am not sure why
> this happened, as the end_of_run function returned successfully (at least the
> print statement right before "return SUCCESS;" appeared right away). So
> something else must have timed-out and caused it to be killed, I guess?
> 
> Any suggestions on where to look to find out what causes this?
> 
> Thanks in advance for your help!
  1599   08 Jul 2019 Konstantin OlchanskiInfoLimitations of MSL
> Konstantin, would you mind resurrecting and sharing the python code?

Not until September or later.

I encourage you to start an "issue" in the bitbucket repository (a "request for improvement") and contact
Thomas L. to see if we can/should discuss it at the incoming midas workshop at TRIUMF.

> So far, my workaround is to call odbedit, which is not ideal for many reasons.

Yes, this has been the way to do it for years...

Also I was thinking of writing an command line tool to invoke midas json-rpc methods (via mhttpd),
then you can do from command line or from a script everything that a web page can do. (of course
you can do it today, by using "curl" to send http requests to mhttpd directly).

K.O.
  1600   08 Jul 2019 Konstantin OlchanskiInfoLimitations of MSL
Hi, Stefan, on second thought, I agree, I do not know of any scripting language implementation (packaged as a library or not) that
can store it's state in a file ("checkpoint the execution") and that can execute it's program "one line at a time", like the midas
sequencer does now. In the most severe case, one invocation of msequencer executes one line of the msl script.

K.O.

> Sure some existing scripting languages can be used, but they fall short of a few important items in larger experiments:
> 
> - they are typically run from a local terminal in the counting house. A remote observer of the experiment has no idea which script is running and at which state it is.
> 
> - if DAQ crashes during a script or is aborted, it has to be restarted from the beginning. If you run a sequence of let's say 100 runs taking 8 hours, and on run #98 something goes wrong, you are screwed if you have to start at run #1 again.
> 
> This are the main reasons why I developed the midas sequencer. Having everything web-based, everybody can watch remotely how far the sequence progressed. If the whole DAQ crashes, the sequence resumes from the crash point, not from the beginning. This is by saving the current state into the ODB. So even if the sequencer itself is stopped and restarted, that still works.
> 
> I agree that the MSL is missing a few calculations, and I was just waiting to get a few specific requests. I will either add new functions such as basic calculations like adding and subtracting variables, or I will create a way to call an external shell like bash to do calculations. I put this high on my todo list.
> 
> Stefan
  1602   08 Jul 2019 Konstantin OlchanskiBug ReportFrontend killed at stop of run
> > run the frontend inside gdb and post the stack trace after the crash?
> > 
> > if there is no crash (the program is stopped by exit()), you may need
> > to set a breakpoint in exit() or _exit() (not sure what it's latest name is)
> > then with luck your stack trace will show who/what called it from where.
> > 
> 
> If I remember correctly from the last time I tried that, it doesn't use the exit
> function but gdb just reports that the program was terminated and no longer exists. I
> can't set a breakpoint on SIGKILL as the point of SIGKILL is to kill the program and
> gdb can't set a break at that point afaik.

For SIGKILL, my gdb reports "Program terminated with signal SIGKILL, Killed." and there is no stack 
trace. Is this what you see?

If your program stops "normally", not from receiving some signal, set breakpoints on "exit" and 
"_exit".

The normal stop sequence is to call exit(), which runs all the atexit() functions (the midas atexit() 
function prints the message about "cm_disconnect_experiment not called at end of program") and 
calls _exit() to stop the program.

So if you see the midas message "cm_disconnect_experiment not called at end of program", it is a 
good indication that somebody (not mfe.c) called exit() on you. A breakpoint on "exit" should catch 
who does it.

Good luck,
K.O.
  1604   08 Jul 2019 Konstantin OlchanskiBug ReportFrontend killed at stop of run
> > 
> > For SIGKILL, my gdb reports "Program terminated with signal SIGKILL, Killed." and there is no stack 
> > trace. Is this what you see?
> 
> Yes, that is exactly what I remember seeing.
> 

Where would a SIGKILL come from?!?

Look in the syslog (/var/log/messages). If the program was killed by the linux kernel, it would be logged there,
the usual cause is the machine runs out of memory and programs are killed by the OOM killer, this is logged
into the syslog, always.

MIDAS also can issue a SIGKILL sometimes, again this is always logged in midas.log. see src/midas.c, search for SIGKILL to see 
the exact messages printed before it is sent out.

K.O.
  1610   10 Jul 2019 Konstantin OlchanskiBug ReportFrontend killed at stop of run
> ... finding a current midas.log file

On the "help" page, see "midas.log".

Same information is in ODB, the midas log file name is concatenation of "/Logger/Data dir" and "message file".

K.O.
  1611   10 Jul 2019 Konstantin OlchanskiBug ReportHeader files missing when trying to compile rootana, roody and analyzer
>> [hh19285@it038146 ~]$ which root-config
> /software/root/v6.06.08/bin/root-config
> [hh19285@it038146 ~]$ root-config --cflags
> -pthread -std=c++11 -Wno-deprecated-declarations -m64 -I/software/root/v6.06.08/include
> 
> [hh19285@it038146 build]$ ./analyzer
> Warning in <TClassTable::Add>: class TApplication already in TClassTable
> ...
> ...
> #2  0x00007f7e911b21a4 in TUnixSystem::StackTrace() () from /usr/lib64/root/libCore.so.6.16

You have a mismatch. Your root-config thinks ROOT is installed in /software/..., but the crash
dump says your ROOT libraries are in /usr/lib64/root (not in /software/...).

You can confirm that you are linking against the correct ROOT by running cmake with VERBOSE=1
and examine the linker command line to see what library link path is specified for ROOT.

You can confirm which ROOT library is actually used when you run the analyzer
by running "ldd ./analyzer". You should see the same library paths as specified
to the linker (/software/.../lib*.so). A mismatch can be caused by the setting of LD_LIBRARY_PATH
and by 100 other reasons.

I suggest that you remove the "wrong" ROOT before you continue debugging this.

K.O.
  1614   11 Jul 2019 Konstantin OlchanskiBug ReportFrontend killed at stop of run
> Wed Jul 10 06:23:58 2019 [mhttpd,ERROR] [system.c:4580:ss_recv_net_command,ERROR] timeout receiving network  command header
> Wed Jul 10 06:23:58 2019 [mhttpd,ERROR] [midas.c:10322:rpc_client_call,ERROR] call to "fedescant" on  "grsmid00.triumf.ca" RPC "rc_transition": timeout waiting for reply

We should have started debugging from here. The error messages mean: your frontend is not responding to run transition (RPC timeout).

> problem in the communication via the A3818 card from CAEN.

Yes, this has been problematic before.

K.O.
ELOG V3.1.4-2e1708b5