> run the frontend inside gdb and post the stack trace after the crash?
> 
> if there is no crash (the program is stopped by exit()), you may need
> to set a breakpoint in exit() or _exit() (not sure what it's latest name is)
> then with luck your stack trace will show who/what called it from where.
> 

If I remember correctly from the last time I tried that, it doesn't use the exit
function but gdb just reports that the program was terminated and no longer exists. I
can't set a breakpoint on SIGKILL as the point of SIGKILL is to kill the program and
gdb can't set a break at that point afaik.

Hi, Stefan, on second thought, I agree, I do not know of any scripting language implementation (packaged as a library or not) that
can store it's state in a file ("checkpoint the execution") and that can execute it's program "one line at a time", like the midas
sequencer does now. In the most severe case, one invocation of msequencer executes one line of the msl script.

K.O.

> Sure some existing scripting languages can be used, but they fall short of a few important items in larger experiments:
> 
> - they are typically run from a local terminal in the counting house. A remote observer of the experiment has no idea which script is running and at which state it is.
> 
> - if DAQ crashes during a script or is aborted, it has to be restarted from the beginning. If you run a sequence of let's say 100 runs taking 8 hours, and on run #98 something goes wrong, you are screwed if you have to start at run #1 again.
> 
> This are the main reasons why I developed the midas sequencer. Having everything web-based, everybody can watch remotely how far the sequence progressed. If the whole DAQ crashes, the sequence resumes from the crash point, not from the beginning. This is by saving the current state into the ODB. So even if the sequencer itself is stopped and restarted, that still works.
> 
> I agree that the MSL is missing a few calculations, and I was just waiting to get a few specific requests. I will either add new functions such as basic calculations like adding and subtracting variables, or I will create a way to call an external shell like bash to do calculations. I put this high on my todo list.
> 
> Stefan

> Konstantin, would you mind resurrecting and sharing the python code?

Not until September or later.

I encourage you to start an "issue" in the bitbucket repository (a "request for improvement") and contact
Thomas L. to see if we can/should discuss it at the incoming midas workshop at TRIUMF.

> So far, my workaround is to call odbedit, which is not ideal for many reasons.

Yes, this has been the way to do it for years...

Also I was thinking of writing an command line tool to invoke midas json-rpc methods (via mhttpd),
then you can do from command line or from a script everything that a web page can do. (of course
you can do it today, by using "curl" to send http requests to mhttpd directly).

K.O.

> After a long overnight run to check that the frontend runs smoothly for a longer
> time, I stopped the run and the frontend was killed by midas.

run the frontend inside gdb and post the stack trace after the crash?

if there is no crash (the program is stopped by exit()), you may need
to set a breakpoint in exit() or _exit() (not sure what it's latest name is)
then with luck your stack trace will show who/what called it from where.

if it is hard to start the frontend inside gdb, you can start it normally,
and attach gdb later, using a "gdb frontend.exe pid" command.

K.O.



> I am not sure why
> this happened, as the end_of_run function returned successfully (at least the
> print statement right before "return SUCCESS;" appeared right away). So
> something else must have timed-out and caused it to be killed, I guess?
> 
> Any suggestions on where to look to find out what causes this?
> 
> Thanks in advance for your help!

I wrote a c++ frontend to read data from CAEN VX1730 digitizers which is used in
parallel with the GRIFFIN frontend to read out DESCANT.

After a long overnight run to check that the frontend runs smoothly for a longer
time, I stopped the run and the frontend was killed by midas. I am not sure why
this happened, as the end_of_run function returned successfully (at least the
print statement right before "return SUCCESS;" appeared right away). So
something else must have timed-out and caused it to be killed, I guess?

Any suggestions on where to look to find out what causes this?

Thanks in advance for your help!

Thank you two!

Actually, both solutions would allow me to fix my problem and I can see use cases for both. Having everything web-based is useful in bigger setups. However, in a testbeam environment, a python script is probably the more straight-forward solution. This gives access to the whole system of ones DAQ PC for whatever tricks one would like to perform.

Konstantin, would you mind resurrecting and sharing the python code? So far, my workaround is to call odbedit, which is not ideal for many reasons.

Cheers
Lukas

Sure some existing scripting languages can be used, but they fall short of a few important items in larger experiments:

- they are typically run from a local terminal in the counting house. A remote observer of the experiment has no idea which script is running and at which state it is.

- if DAQ crashes during a script or is aborted, it has to be restarted from the beginning. If you run a sequence of let's say 100 runs taking 8 hours, and on run #98 something goes wrong, you are screwed if you have to start at run #1 again.

This are the main reasons why I developed the midas sequencer. Having everything web-based, everybody can watch remotely how far the sequence progressed. If the whole DAQ crashes, the sequence resumes from the crash point, not from the beginning. This is by saving the current state into the ODB. So even if the sequencer itself is stopped and restarted, that still works.

I agree that the MSL is missing a few calculations, and I was just waiting to get a few specific requests. I will either add new functions such as basic calculations like adding and subtracting variables, or I will create a way to call an external shell like bash to do calculations. I put this high on my todo list.

Stefan

> I am missing a few features.

MSL did not start out as a fully featured programming language.

Rather than extending it until it becomes one, I think a better approach would be to take
one of the existing extensible scripting language libraries and extend it with midas functions.

For example, LUA (https://www.lua.org/about.html) seems to be popular.

There were also requests for a midas extension for PYTHON (actually this has been done before,
but never added to the midas repository. I still have that code and I suppose it could be resurrected).

K.O.


> Do any of the following exist and I have just
> overlooked them?
> 
> - Arithmetics:
>     SET foo, 2
>     SET bar, 2
>     SET FOOBAR, $foo + $bar
>     -> FOOBAR is 2, not 4
> 
> - Vectors/arrays
>     As far as I understand, "SET" only allows for single variables and no 
>     "scripting" in variable names, i. e. doing something like the following
>     scripted: 
>     SET var_1, 0
>     SET var_2, 0
>     ...
>     LOOP n, 10
>         ...
>         IF something
>             SET var_${n}, 1
>         ENDIF 
>     
>     The only way I could think of doing something similar is via the ODB.
>     I don't know if it's good practice to fill the ODB with variables like this.
> 
> - Loading scripts at run time (see other bug)
>     That would allow for script manipulation, even though it's kind of dirty, it
>     might be useful in some cases.
> 
> - Obtaining results from external scripts
>     A way to pass things from external scripts would be the solution to all
>     problems. Something that is not implemented could be done in a bash or
>     python script instead. 
> 
> Cheers
> Lukas

> /home/hh19285/packages/rootana/include/TRootanaEventLoop.hxx:24:25: fatal error:
> THttpServer.h: No such file or directory
>  #include "THttpServer.h"
>
> include/TPeakFindPanel.h:46:23: fatal error: TSpectrum.h: No such file or directory
>  #include "TSpectrum.h"
>

Your ROOT is strange, missing some standard features. Also installed in a strange place, /usr/include/root.

Did you install ROOT from the EPEL RPM packages? In the last I have seen this ROOT built very strangely, with some standard features disabled for no obvious 
reason.

For this reason, I recommend that you install ROOT from the binary distribution at root.cern.ch or build it from source.

For more debugging, please post the output of:
which root-config
root-config --version
root-config --features
root-config --cflags

For reference, here is my output for a typical CentOS7 machine:
daq16:~$ which root-config
/daq/daqshare/olchansk/root/root_v6.12.04_el74_64/bin/root-config
daq16:~$ root-config --version
6.12/04
daq16:~$ root-config --features
asimage astiff builtin_afterimage builtin_ftgl builtin_gl2ps builtin_glew builtin_llvm builtin_lz4 builtin_unuran cling cxx11 exceptions explicitlink fftw3 gdmlgenvector 
http imt mathmore minuit2 opengl pch pgsql python roofit shared sqlite ssl thread tmva x11 xft xml
daq16:~$ root-config --cflags -pthread -std=c++11 -m64 -I/daq/daqshare/olchansk/root/root_v6.12.04_el74_64/include

The important one is the --features, see that "http" and "xml" are enabled. "spectrum" used to be an optional feature, I do not think it can be disabled these 
days, so your missing "TSpectrum.h" is strange. (But I just think the EPEL ROOT RPMs are built wrong).

K.O.

First of all thank you for all the assistance provided so far, especially making
changes to the code in CMakeList file previously for our configuration.I am not
sure whether this is an appropriate Elog for this matter but we are getting the
following error when trying to make rootana, roody and analyzer on our 64 bit
DAQ machine.

At the bottom of this Elog entry I have provided information about the specifics
of our DAQ machine

Below are the 2 errors we are encountering:

=============================================================================
[hh19285@it038146 ~]$ cd packages/rootana/
[hh19285@it038146 rootana]$ ls
bitbucket-pipelines.yml  Dockerfile  Doxygen.cxx  include  libAnalyzer        
libMidasInterface  libNetDirectory  libXmlServer  Makefile.old  obj          
README.md        thisrootana.sh
doc                      Doxyfile    examples     lib      libAnalyzerDisplay 
libMidasServer     libUnpack        Makefile      manalyzer     old_analyzer 
thisrootana.csh
[hh19285@it038146 rootana]$ cd examples/
[hh19285@it038146 examples]$ make
g++ -o TV792Histogram.o -g -O2 -Wall -Wuninitialized -DHAVE_LIBZ
-I/home/hh19285/packages/rootana/include -DHAVE_ROOT -pthread -std=c++11 -m64
-I/usr/include/root -DHAVE_ROOT_XML -DHAVE_ROOT_HTTP -DHAVE_THTTP_SERVER
-DHAVE_MIDAS -DOS_LINUX -Dextname -I/home/hh19285/packages/midas/include -c
TV792Histogram.cxx
In file included from
/home/hh19285/packages/rootana/include/TRootanaDisplay.hxx:5:0,
                 from
/home/hh19285/packages/rootana/include/TCanvasHandleBase.hxx:13,
                 from
/home/hh19285/packages/rootana/include/THistogramArrayBase.h:9,
                 from TV792Histogram.h:5,
                 from TV792Histogram.cxx:1:
/home/hh19285/packages/rootana/include/TRootanaEventLoop.hxx:24:25: fatal error:
THttpServer.h: No such file or directory
 #include "THttpServer.h"
                         ^
compilation terminated.
make: *** [TV792Histogram.o] Error 1
[hh19285@it038146 examples]$ 

===============================================================================
[hh19285@it038146 analyzer]$ ls
ana.cxx         midas2root.cxx         TAgilentHistogram.h    
TCamacADCHistogram.h  TL2249Histogram.h    TV1190Histogram.h      
TV1720Waveform.h       TV1730RawWaveform.h
anaDisplay.cxx  README.txt             TAnaManager.cxx        
TDT724Waveform.cxx    TTRB3Histogram.cxx   TV1720Correlations.cxx 
TV1730DppWaveform.cxx  TV792Histogram.cxx
Makefile        root_server.cxx        TAnaManager.hxx         TDT724Waveform.h
     TTRB3Histogram.hxx   TV1720Correlations.h    TV1730DppWaveform.h   
TV792Histogram.h
Makefile.old    TAgilentHistogram.cxx  TCamacADCHistogram.cxx 
TL2249Histogram.cxx   TV1190Histogram.cxx  TV1720Waveform.cxx     
TV1730RawWaveform.cxx
[hh19285@it038146 analyzer]$ make
g++ -o TV792Histogram.o -g -O2 -Wall -Wuninitialized -DHAVE_LIBZ -I../include
-DHAVE_ROOT -pthread -std=c++11 -m64 -I/usr/include/root -DHAVE_ROOT_XML
-DHAVE_ROOT_HTTP -DHAVE_THTTP_SERVER -DHAVE_MIDAS -DOS_LINUX -Dextname
-I/home/hh19285/packages/midas/include -c TV792Histogram.cxx
In file included from TV792Histogram.cxx:1:0:
TV792Histogram.h:5:33: fatal error: THistogramArrayBase.h: No such file or directory
 #include "THistogramArrayBase.h"
                                 ^
compilation terminated.
make: *** [TV792Histogram.o] Error 1


===============================================================================

[hh19285@it038146 ~]$ cd $HOME/packages
[hh19285@it038146 packages]$ git clone https://bitbucket.org/tmidas/roody
Cloning into 'roody'...
remote: Counting objects: 1115, done.
remote: Compressing objects: 100% (470/470), done.
remote: Total 1115 (delta 662), reused 1063 (delta 631)
Receiving objects: 100% (1115/1115), 1.01 MiB | 2.12 MiB/s, done.
Resolving deltas: 100% (662/662), done.
[hh19285@it038146 packages]$ cd roody
[hh19285@it038146 roody]$ make
g++  -O2 -g -Wall -Wuninitialized -fPIC -pthread -std=c++11 -m64
-I/usr/include/root -DNEED_STRLCPY -I. -Iinclude -DHAVE_NETDIRECTORY
-I/home/hh19285/packages/rootana/include -c -MM src/*.cxx > Makefile.depends1
In file included from src/Roody.cxx:42:0:
include/TPeakFindPanel.h:46:23: fatal error: TSpectrum.h: No such file or directory
 #include "TSpectrum.h"
                       ^
compilation terminated.
In file included from src/TPeakFindPanel.cxx:12:0:
include/TPeakFindPanel.h:46:23: fatal error: TSpectrum.h: No such file or directory
 #include "TSpectrum.h"
                       ^
compilation terminated.
make: [depend] Error 1 (ignored)
sed 's#^#obj/#' Makefile.depends1 > Makefile.depends2
sed 's#^obj/ #   #' Makefile.depends2 > Makefile.depends
rm -f Makefile.depends1 Makefile.depends2
mkdir -p bin
mkdir -p obj
mkdir -p lib
cd doxfiles; doxygen roodydox.cfg
Warning: Tag `USE_WINDOWS_ENCODING' at line 11 of file roodydox.cfg has become
obsolete.
To avoid this warning please remove this line from your configuration file or
upgrade it using "doxygen -u"
Warning: Tag `DETAILS_AT_TOP' at line 23 of file roodydox.cfg has become obsolete.
To avoid this warning please remove this line from your configuration file or
upgrade it using "doxygen -u"
Warning: Tag `SHOW_DIRECTORIES' at line 58 of file roodydox.cfg has become obsolete.
To avoid this warning please remove this line from your configuration file or
upgrade it using "doxygen -u"
Warning: Tag `HTML_ALIGN_MEMBERS' at line 122 of file roodydox.cfg has become
obsolete.
To avoid this warning please remove this line from your configuration file or
upgrade it using "doxygen -u"
Warning: Tag `MAX_DOT_GRAPH_WIDTH' at line 220 of file roodydox.cfg has become
obsolete.
To avoid this warning please remove this line from your configuration file or
upgrade it using "doxygen -u"
Warning: Tag `MAX_DOT_GRAPH_HEIGHT' at line 221 of file roodydox.cfg has become
obsolete.
To avoid this warning please remove this line from your configuration file or
upgrade it using "doxygen -u"
Searching for include files...
Searching for example files...
Searching for images...
Searching for dot files...
Searching for msc files...
Searching for files to exclude
Searching for files to process...
Searching for files in directory /home/hh19285/packages/roody/doxfiles
Reading and parsing tag files
Parsing files
Preprocessing /home/hh19285/packages/roody/doxfiles/features.dox...
Parsing file /home/hh19285/packages/roody/doxfiles/features.dox...
Preprocessing /home/hh19285/packages/roody/doxfiles/quickstart.dox...
Parsing file /home/hh19285/packages/roody/doxfiles/quickstart.dox...
Preprocessing /home/hh19285/packages/roody/doxfiles/roody.dox...
Parsing file /home/hh19285/packages/roody/doxfiles/roody.dox...
Preprocessing /home/hh19285/packages/roody/include/MTGListTree.h...
Parsing file /home/hh19285/packages/roody/include/MTGListTree.h...
Preprocessing /home/hh19285/packages/roody/include/Roody.h...
Parsing file /home/hh19285/packages/roody/include/Roody.h...
Preprocessing /home/hh19285/packages/roody/include/RoodyXML.h...
Parsing file /home/hh19285/packages/roody/include/RoodyXML.h...
Preprocessing /home/hh19285/packages/roody/include/TGTextDialog.h...
Parsing file /home/hh19285/packages/roody/include/TGTextDialog.h...
Preprocessing /home/hh19285/packages/roody/include/TPeakFindPanel.h...
Parsing file /home/hh19285/packages/roody/include/TPeakFindPanel.h...
Preprocessing /home/hh19285/packages/roody/src/MTGListTree.cxx...
Parsing file /home/hh19285/packages/roody/src/MTGListTree.cxx...
Preprocessing /home/hh19285/packages/roody/src/Roody.cxx...
Parsing file /home/hh19285/packages/roody/src/Roody.cxx...
Preprocessing /home/hh19285/packages/roody/src/RoodyXML.cxx...
Parsing file /home/hh19285/packages/roody/src/RoodyXML.cxx...
Preprocessing /home/hh19285/packages/roody/src/TGTextDialog.cxx...
Parsing file /home/hh19285/packages/roody/src/TGTextDialog.cxx...
Building group list...
Building directory list...
Building namespace list...
Building file list...
Building class list...
Associating documentation with classes...
Computing nesting relations for classes...
Building example list...
Searching for enumerations...
Searching for documented typedefs...
Searching for members imported via using declarations...
Searching for included using directives...
Searching for documented variables...
Building interface member list...
Building member list...
Searching for friends...
Searching for documented defines...
Computing class inheritance relations...
Computing class usage relations...
Flushing cached template relations that have become invalid...
Creating members for template instances...
Computing class relations...
Add enum values to enums...
Searching for member function documentation...
Building page list...
Search for main page...
Computing page relations...
Determining the scope of groups...
Sorting lists...
Freeing entry tree
Determining which enums are documented
Computing member relations...
Building full member lists recursively...
Adding members to member groups.
Computing member references...
Inheriting documentation...
Generating disk names...
Adding source references...
Adding xrefitems...
Sorting member lists...
Computing dependencies between directories...
Generating citations page...
Counting data structures...
Resolving user defined references...
Finding anchors and sections in the documentation...
Transferring function references...
Combining using relations...
Adding members to index pages...
Generating style sheet...
Generating search indices...
Generating example documentation...
Generating file sources...
Parsing code for file features.dox...
Generating code for file MTGListTree.cxx...
Generating code for file MTGListTree.h...
Parsing code for file quickstart.dox...
Generating code for file Roody.cxx...
Parsing code for file roody.dox...
Generating code for file Roody.h...
Generating code for file RoodyXML.cxx...
Generating code for file RoodyXML.h...
Generating code for file TGTextDialog.cxx...
Generating code for file TGTextDialog.h...
Generating code for file TPeakFindPanel.h...
Generating file documentation...
Generating docs for file features.dox...
Generating docs for file MTGListTree.cxx...
Generating docs for file MTGListTree.h...
Generating docs for file quickstart.dox...
Generating docs for file Roody.cxx...
Generating docs for file roody.dox...
Generating docs for file Roody.h...
Generating docs for file RoodyXML.cxx...
Generating docs for file RoodyXML.h...
Generating docs for file TGTextDialog.cxx...
Generating docs for file TGTextDialog.h...
Generating docs for file TPeakFindPanel.h...
Generating page documentation...
Generating docs for page features...
Generating docs for page quickstart...
Generating group documentation...
Generating class documentation...
Generating docs for compound MemDebug...
Generating docs for compound MTGListTree...
Generating docs for compound OptStatMenu...
Generating docs for compound PadObject...
Generating docs for compound PadObjectVec...
Generating docs for compound Roody...
Generating docs for compound RoodyXML...
Generating docs for compound TGTextDialog...
Generating docs for compound TPeakFindPanel...
Generating namespace index...
Generating graph info page...
Generating directory documentation...
Generating index page...
Generating page index...
Generating module index...
Generating namespace index...
Generating namespace member index...
Generating annotated compound index...
Generating alphabetical compound index...
Generating hierarchical class index...
Generating member index...
Generating file index...
Generating file member index...
Generating example index...
finalizing index lists...
lookup cache used 433/65536 hits=3757 misses=436
finished...
g++  -O2 -g -Wall -Wuninitialized -fPIC -pthread -std=c++11 -m64
-I/usr/include/root -DNEED_STRLCPY -I. -Iinclude -DHAVE_NETDIRECTORY
-I/home/hh19285/packages/rootana/include -c -o obj/main.o src/main.cxx
g++  -O2 -g -Wall -Wuninitialized -fPIC -pthread -std=c++11 -m64
-I/usr/include/root -DNEED_STRLCPY -I. -Iinclude -DHAVE_NETDIRECTORY
-I/home/hh19285/packages/rootana/include -c -o obj/DataSourceTDirectory.o
src/DataSourceTDirectory.cxx
g++  -O2 -g -Wall -Wuninitialized -fPIC -pthread -std=c++11 -m64
-I/usr/include/root -DNEED_STRLCPY -I. -Iinclude -DHAVE_NETDIRECTORY
-I/home/hh19285/packages/rootana/include -c -o obj/Roody.o src/Roody.cxx
In file included from src/Roody.cxx:42:0:
include/TPeakFindPanel.h:46:23: fatal error: TSpectrum.h: No such file or directory
 #include "TSpectrum.h"
                       ^
compilation terminated.
make: *** [obj/Roody.o] Error 1
================================================================================


For your reference here is the info about our DAQ machine

[hh19285@it038146 bin]$ uname -a
Linux it038146.users.bris.ac.uk 3.10.0-957.21.2.el7.x86_64 #1 SMP Wed Jun 5
14:26:44 UTC 2019 x86_64 x86_64
x86_64 GNU/Linux
[hh19285@it038146 bin]$ uname -a
Linux it038146.users.bris.ac.uk 3.10.0-957.21.2.el7.x86_64 #1 SMP Wed Jun 5
14:26:44 UTC 2019 x86_64 x86_64
x86_64 GNU/Linux
[hh19285@it038146 bin]$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/4.8.5/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
--infodir=/usr/share/info
--with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap
--enable-shared --enable-threads=posix
--enable-checking=release --with-system-zlib --enable-__cxa_atexit
--disable-libunwind-exceptions
--enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu
--enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin
--enable-initfini-array
--disable-libgcj
--with-isl=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/isl-install
--with-cloog=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/cloog-install
--enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64
--build=x86_64-redhat-linux
Thread model: posix
gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) 
[hh19285@it038146 bin]$ lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch
Distributor ID:	CentOS
Description:	CentOS Linux release 7.6.1810 (Core) 
Release:	7.6.1810
Codename:	Core

Hi, 
I am missing a few features. Do any of the following exist and I have just
overlooked them?

- Arithmetics:
    SET foo, 2
    SET bar, 2
    SET FOOBAR, $foo + $bar
    -> FOOBAR is 2, not 4

- Vectors/arrays
    As far as I understand, "SET" only allows for single variables and no 
    "scripting" in variable names, i. e. doing something like the following
    scripted: 
    SET var_1, 0
    SET var_2, 0
    ...
    LOOP n, 10
        ...
        IF something
            SET var_${n}, 1
        ENDIF 
    
    The only way I could think of doing something similar is via the ODB.
    I don't know if it's good practice to fill the ODB with variables like this.

- Loading scripts at run time (see other bug)
    That would allow for script manipulation, even though it's kind of dirty, it
    might be useful in some cases.

- Obtaining results from external scripts
    A way to pass things from external scripts would be the solution to all
    problems. Something that is not implemented could be done in a bash or
    python script instead. 

Cheers
Lukas

Dear MIDAS users,

Here's further information on the third MIDAS workshop:

1) Workshop will take place from 1PM-5PM (Vancouver time) on Aug 7.

2) A mostly finalized agenda for the workshop is posted here:

https://indico.triumf.ca/conferenceDisplay.py?confId=2438

People are still welcome to email me if they want to present something.  We should be able to add it to the schedule.

3) For those who want to participate remotely, we will be using bluejeans.  The webpage for the bluejeans meeting is here:

https://bluejeans.com/462865444

4) For those at TRIUMF I will confirm the meeting room closer to the date.

Thomas


> Dear MIDAS users,
> 
> We would like to announce a third MIDAS workshop at TRIUMF on Aug 7, 2019.
> Stefan Ritt will again be visiting TRIUMF at this time.
> 
> The overall goal of the workshop is to present new features in MIDAS, to discuss
> future changes and to hear experiences from different experiments.
> 
> We expect that some participants will connect remotely to the workshop; we will
> setup a video-conferencing option.  The exact time of the workshop will be
> decided later and will be optimized based on the geographic distribution of
> remote attendees. So please let us know if you want to attend remotely.  We are
> also happy for people to come in person to TRIUMF.  
> 
> A (very) preliminary agenda includes
> 
> - New default mhttpd pages and new APIs for custom pages
> - Conversion of MIDAS to C++
> - new C++ based frontend framework (tmfe and mvodb from ALPHA-g)
> - MySQL/Postgres database for storing ODB configurations
> - Plans for updating history plotting
> - Using MIDAS with an online trigger farm
> - new C++ multithreaded flow analyzer (manalyzer from ALPHA-g)
> 
> But please suggest other topics; we also hope to hear reports from particular
> experiments.
> 
> Sincerely,
> Thomas (on behalf of MIDAS developers)

Hi,
the subject line describes the project already
Suppose you have a file foo.msl. Somewhere in the file, you have the line 
INCLUDE bar.msl

Once you click save in the sequencer page, mhttpd crashes:
$ mhttpd
free(): double free detected in tcache 2
[1]    27590 abort (core dumped)  mhttpd


GDB helps shed some light on the problem:

#0  0x00007ffff76b057f in raise () from /lib64/libc.so.6
#1  0x00007ffff769a895 in abort () from /lib64/libc.so.6
#2  0x00007ffff76f39d7 in __libc_message () from /lib64/libc.so.6
#3  0x00007ffff76fa2ec in malloc_printerr () from /lib64/libc.so.6
#4  0x00007ffff76fbdf5 in _int_free () from /lib64/libc.so.6
#5  0x00000000004b8b41 in mxml_parse_entity (buf=buf@entry=0x7fffffffc2c8,
file_name=file_name@entry=0x7fffffffc710
"/home/luk/packages/mutrig_daq/online/foo.xml",
            error=error@entry=0x7fffffffcd24 "XML read error in file
\"/home/luk/packages/mutrig_daq/online/foo.xml\", line 2: bar.msl.xml is
missing", error_size=error_size@entry=256,
                error_line=error_line@entry=0x7fffffffce24) at ../mxml/mxml.c:1996
#6  0x00000000004b966d in mxml_parse_file
(file_name=file_name@entry=0x7fffffffc710
"/home/luk/packages/mutrig_daq/online/foo.xml",
            error=error@entry=0x7fffffffcd24 "XML read error in file
\"/home/luk/packages/mutrig_daq/online/foo.xml\", line 2: bar.msl.xml is
missing", error_size=error_size@entry=256,
                error_line=error_line@entry=0x7fffffffce24) at ../mxml/mxml.c:2041
#7  0x000000000041d9c2 in init_sequencer () at src/mhttpd.cxx:14321
#8  0x000000000040c2b6 in main (argc=<optimized out>, argv=<optimized out>) at
src/mhttpd.cxx:18028

Cheers
Lukas

P. S. This problem reminds me of the old joke: A man goes to his doctor and says
"Doc, it hurts when I do this" to which the doctor replies "Then don't do that".
However, I think, mhttpd should not crash even if you're not supposed to include
non-existent scripts in msequencer.

Thanks!

Confirmed. my_global.h is removed in MySQL 8.0 (gives a compile error) and deprecated in 
MariaDB 10.2 (gives a #warning).

I removed include of my_global.h, it is not needed on el6, el7 and ubuntu.

Also added explicit support for mariadb via mariadb_config if it exists.

Note that the cmake build does not actually enable mysql, sqlite and odbc - it detects them, but 
does not do anything about it. We will fix this shortly.

K.O.




> Hey,
> 
> while trying to compile Midas under openSUSE 15.0 with mysql support, I was
> running into the problem that somehow the mysql header file my_global.h is not
> included in the packages. This might be a bug that concerns the suse developers
> more, but is it actually needed? Compilation worked fine with the include line
> commented out.
> 
> If it's not needed, I would like to suggest to remove line 735 of
> src/history_schema.cxx (where it's included)
> 
> Cheers
> Lukas
> 
> mysql  Ver 15.1 Distrib 10.2.22-MariaDB, for Linux (x86_64) using  EditLine wrapper
> Also, all mariadb development packages are installed

Hey,

while trying to compile Midas under openSUSE 15.0 with mysql support, I was
running into the problem that somehow the mysql header file my_global.h is not
included in the packages. This might be a bug that concerns the suse developers
more, but is it actually needed? Compilation worked fine with the include line
commented out.

If it's not needed, I would like to suggest to remove line 735 of
src/history_schema.cxx (where it's included)

Cheers
Lukas

mysql  Ver 15.1 Distrib 10.2.22-MariaDB, for Linux (x86_64) using  EditLine wrapper
Also, all mariadb development packages are installed

I am sorry, yesterday evening I must have been a bit tired after a long day with a lot of 
problems and error messages, so that I did not realize that yes, the frontend was finally 
starting well again but by recovering the odb file from an old one, it was stuck in the 
transition "stopping run" and this caused the continuous reloading of the status page. 

A "obdedit -C stop" solved the problem. 

Sorry for this! 





 > We observed a strange behavior, from our point of view:
> > ... the Midas status page started to reload/refresh every second
> 
> What version of midas is this? Run the odbedit "ver" command please. Also which
> browser on what OS is this? (chrome->about google chrome, firefox->about firefox).
> 
> The current versions of midas do not reload the status page ever, and I think
> all the page-reload code has been removed and they cannot reload automatically.
> 
> Old versions of the midas status page were designed to reload every 60 seconds or so.
> The reload interval is adjustable, but I do not think it was stored in ODB. It was
> accessed from the status page "config" button and I think it stored the reload
> period is a browser cookie.
> 
> This reload value may have gotten confused, and in this case, to fix it,
> you can try to clear all the web cookies from the web page. Another test for this
> would be to try an alternate web browser, which would presumable not have the bad cookie
> and will not suffer from the reload problem.
> 
> You can also open the web page debugger (google chrome -> right click menu -> inspect ->> 
> console & etc) and see if anything shows up there. I think you can set a break point
> on the page reload function and catch the place that causes the reload.
> 
> K.O.

> We observed a strange behavior, from our point of view:
> ... the Midas status page started to reload/refresh every second

What version of midas is this? Run the odbedit "ver" command please. Also which
browser on what OS is this? (chrome->about google chrome, firefox->about firefox).

The current versions of midas do not reload the status page ever, and I think
all the page-reload code has been removed and they cannot reload automatically.

Old versions of the midas status page were designed to reload every 60 seconds or so.
The reload interval is adjustable, but I do not think it was stored in ODB. It was
accessed from the status page "config" button and I think it stored the reload
period is a browser cookie.

This reload value may have gotten confused, and in this case, to fix it,
you can try to clear all the web cookies from the web page. Another test for this
would be to try an alternate web browser, which would presumable not have the bad cookie
and will not suffer from the reload problem.

You can also open the web page debugger (google chrome -> right click menu -> inspect ->> 
console & etc) and see if anything shows up there. I think you can set a break point
on the page reload function and catch the place that causes the reload.

K.O.

Hello,

We observed a strange behavior, from our point of view:
After some issues with with a 100% full database and recovering from this by 
creating a new odb file from a previous copy, the Midas status page started to 
reload/refresh every second while on the other hand it solved the 100% full 
issue.
There are no error messages and the pages looks like normal but it is impossible 
to start a new run due to the permanent reloading.
It is possible to click for example on odb and check the settings. 

Any idea what could be the problem and what the solution? 

Thanks 

Thorsten 

> Reproduced on el7 (CentOS7). Same thing works on el6 (SL6).

Fixed in commit dd937e6. Only enable SSE4.2 crc32c for 64-bit compilation. Still not sure why it worked for 32-bit 
compilation on el6 (SL6).

K.O.


> 
> The error is in the SSE4.2-assembly-accelerated library for computing crc32c checksums. I do 
> not understand this assembly stuff enough to tell what goes wrong.
> 
> In any case, "make linux32" is intended for VME processors that cannot run 64-bit code. These 
> processors also happen to not have the SSE4.2 instructions needed for this code to actually 
> work, so one solution would be to always disable crc32c SSE4.2-assembly-acceleration for the 
> linux32 target.
> 
> Note that the original reported was running "make linux32" with the idea of generating code for 
> the 32-bit ARM processor.
> 
> Here the situation is like this: the required CRC32C instructions are present on 64-bit capable 
> ARM processors (RPi3, etc) and probably work in 32-bit mode, and I found the assembly-
> language crc32c library that uses them. It needs to be added to MIDAS and tested.
> 
> For the older 32-bit-only ARM processors (Cyclone5 FPGA, Rpi2 and older) I do not see any 
> hardware accelerated crc32c implementations, so it uses software computed CRC32 always. 
> (P.S.: I see the current linux kernels have a hardware accelerated library for CRC32C, not sure if 
> it can be imported into MIDAS easily).
> 
> K.O.
> 
> 
> > $ make linux32 ...
> > g++ -m32 -c -g -O2 -Wall -Wno-strict-aliasing -Wuninitialized -Iinclude -
> > Idrivers -Imxml -Imscb/include -DHAVE_FTPLIB -D_LARGEFILE64_SOURCE -DHAVE_ZLIB -
> > DHAVE_MSCB -DHAVE_MONGOOSE6 -DMG_ENABLE_THREADS -DMG_DISABLE_CGI -
> DMG_ENABLE_SSL 
> > -DOS_LINUX -fPIC -Wno-unused-function -o lib/crc32c.o src/crc32c.cxx
> > src/crc32c.cxx: In function ‘uint32_t crc32c_hw(uint32_t, const void*, size_t)’:
> > src/crc32c.cxx:283:66: error: ‘asm’ operand has impossible constraints
> >                      : "r"(next), "0"(crc0), "1"(crc1), "2"(crc2));
> >                                                                   ^
> > src/crc32c.cxx:303:66: error: ‘asm’ operand has impossible constraints
> >                      : "r"(next), "0"(crc0), "1"(crc1), "2"(crc2));
> >                                                                   ^
> > src/crc32c.cxx: In function ‘uint32_t crc32c(uint32_t, const void*, size_t)’:
> > src/crc32c.cxx:348:34: error: PIC register clobbered by ‘%ebx’ in ‘asm’
> >                  : "%ebx", "%edx"); \
> >                                   ^
> > src/crc32c.cxx:362:5: note: in expansion of macro ‘SSE42’
> >      SSE42(sse42);
> >      ^
> > make[1]: *** [lib/crc32c.o] Error 1
> > make[1]: Leaving directory `/home/hh19285/packages/midas'
> > make: *** [linux32] Error 2
> > 
> > Could you please help with getting past this? otherwise we may need to change 
> > our whole experimental setup.
> > 
> > Thank you in advance