> -  ss_suspend_set_dispatch_ipc(NULL);
> +  //  ss_suspend_set_dispatch_ipc(NULL);
> 
> This compiles and at least runs for me; so maybe that is helpful for you.  But Konstantin will provide a longer term solution.

I now understand why this fix worked. Around December 2018 timeframe, I reworked the MIDAS event buffer code
and one improvement was to only send UDP buffer notifications if somebody is waiting for them. This probably
reduced to zero the probability of recursive calls to the user event handler - the problem originally fixed by the monkey
work against the midas ipc handler.

After looking at it, I now understand that the correct solution is to call ss_suspend(MSG_BM), but it turns out
inside MIDAS, handling of MSG_BM was incomplete and recursive calls to the user event handler were still
possible. (but most likely not actually happening anymore because of those changes to the event buffer code).

So.

a) ss_suspend(MSG_BM) inside midas now works correctly, recursive call to the user event handler will not happen.
b) TMidasOnline::sleep() now calls ss_suspend(MSG_BM), monkey business with ss_suspend_set_dispatch_ipc() is removed.

The problem of recursive call to the analyzer event handler is now fixed, both rootana and manalyzer (both use the same TMidasOnline code).

Read more about this here:
https://midas.triumf.ca/elog/Midas/1663

K.O.

I'm not sure if these issues are related or not, but I'm getting an error
message when I want to access the root of the ODB via the webserver:
[mhttpd,ERROR] [mhttpd.cxx:563:rread,ERROR] Cannot read file '/root', read of
4096 returned -1, errno 21 (Is a directory)

I also tried turning the re-direct from http to https off, but this does not
seem to work. I also noticed that the redirect changes the localhost into a
hostname. Where does mongoose take this hostname from?

EDIT: Seems that the change of the hostname is due to a setting in /etc/hosts,
i.e. all my fault ...

EDIT: I think there was some issue with the mhttpd. When I checked the output (I
used screen to run it), it was full of these messages:

ss_semaphore_wait_for: semop/semtimedop(2588679) returned -1, errno 43
(Identifier removed)
al_check: Something is wrong with our semaphore, ss_semaphore_wait_for()
returned 408, aborting.
al_check: Cannot abort - this will lock you out of odb. From this point, MIDAS
will not work correctly. Please read the discussion at
https://midas.triumf.ca/elog/Midas/945

Restarted it and it stopped redirecting. So accessing the root of the ODB via
the webserver is the only issue now.

> I'm not sure if these issues are related or not, but I'm getting an error
> message when I want to access the root of the ODB via the webserver:
> [mhttpd,ERROR] [mhttpd.cxx:563:rread,ERROR] Cannot read file '/root', read of
> 4096 returned -1, errno 21 (Is a directory)

This is an old bug. It was part of the "custom path" confusion. Fixed (I think) in all midas-2019 
releases.

To confirm, which version are you using (run "odbedit ver" or look on the mhttpd "help" page)?

If you have an older version, I recommend that you update to midas-2019-03 (cd midas; git pull; 
git checkout midas-2019-03; make clean; make).

If you feel adventurous, you can also update to the head of the development version
and see all the new features (cmake, c++11, new history pages).

If you do not feel adventurous, wait until we have midas-2019-09 ready, use midas-2019-03 
until then.

K.O.

compiling:
----------

The compile option -DHAVE_FTPLIB checked in mdsupport.cxx disappeared if you 
compile with cmake.

I added:

  add_compile_options(-DHAVE_FTPLIB)

to CMakeLists.txt to fix this. Can probably be done in a more elegant way 
(checking if the right libraries exist?).

running:
-------

Our MAX_EVENT_SIZE is set in the odb to 805306368. This number is also used in 

  INT lazy_copy(char *outfile, char *infile, int max_event_size)

this is to big when copying with ftp, causing a crash. Reducing it here with a 
factor 10 solves our problems.

> The compile option -DHAVE_FTPLIB checked in mdsupport.cxx disappeared if you 
> compile with cmake.

Hi, Stefan - do we still need to support FTP in the logger? In the lazylogger, special support for 
FTP is not needed, they can you the "script" method and do FTP without our help.

I move to remove FTP support from MIDAS. (second? other opinions?)

> Our MAX_EVENT_SIZE is set in the odb to 805306368. This number is also used in 
> this is to big when copying with ftp, causing a crash. Reducing it here with a 
> factor 10 solves our problems.

I am surprised that changing MAX_EVENT_SIZE (to a "too big" value) causes lazylogger to 
crash. More usually MAX_EVENT_SIZE has no effect until you try to write an event that is 
somehow "too big", then there is a crash. Perhaps there is a bug specifically in the FTP code.

Anyhow, I recommend the solution of using the "script" method. We have example lazylogger 
scripts in midas/progs/lazy*.perl (the scripts do not have to be in perl, python is ok). We do
not have any example that uses FTP because we do not use FTP for data storage. But you can
easily adapt lazy_test.perl and lazy_copy.perl to use scp and sftp, the secure versions of FTP.

K.O.

Logging a list of variables to the history via links in the history ODB subtree,
we get messages as follows at every run start:

19:43:24.009 2019/10/06 [Logger,ERROR] [history_schema.cxx:2676:hs_write_event,ERROR] Event 'System' data size mismatch: expected 412 bytes, got 416 bytes

19:43:24.008 2019/10/06 [Logger,ERROR] [history_schema.cxx:2676:hs_write_event,ERROR] Event 'System' data size mismatch: expected 412 bytes, got 416 bytes

19:43:23.850 2019/10/06 [Logger,ERROR] [history_schema.cxx:455:hs_write_event,ERROR] Event 'System' data size mismatch count: 25, expected 412 bytes, hs_write_event() called with as much as 416 bytes

19:43:23.850 2019/10/06 [Logger,ERROR] [history_schema.cxx:455:hs_write_event,ERROR] Event 'System' data size mismatch count: 25, expected 412 bytes, hs_write_event() called with as much as 416 bytes

The history calculates the size of a record from the size of the individual variables, (history_schema.cxx, L2666 ff), whereas the ODB delivers the data aligned/padded to the size of the largest value in the record.
In our history, a long list of doubles (64 Bit) fas followed by three floats (32 bit), leading to a padded response from the ODB, 4 byte longer than the history expects.
Quick fix: Add another 32 bit dummy variable to the history. Gets rid of the error messages...
Should probably be fixed at a deeper level...

I wonder why do you this via ODB links. The "standard" way of writing to the history should be to create events for an equipment and flag this equipment as being written to the
history. All variables under /Equipment/<name>/Variables then automatically go into the history and you don't have to worry about ODB links. Only variables not fitting the
equipment/variables scheme should be dealt with via ODB links, like variables under equipment/statistics or parameters in another ODB tree. In a typical midas experiment, only
very few variables typically go into the 'System' event. This is however probably not a solution to your problem. If you have a similar structure (doubles plus an odd number of floats)
under 'variables', you might get the same error. I'n in contact with KO to fix this problem at the root level.

Stefan

> Logging a list of variables to the history via links in the history ODB subtree,
> we get messages as follows at every run start:
> 
> 19:43:24.009 2019/10/06 [Logger,ERROR] [history_schema.cxx:2676:hs_write_event,ERROR] Event 'System' data size mismatch: expected 412 bytes, got 416 bytes
> 
> 19:43:24.008 2019/10/06 [Logger,ERROR] [history_schema.cxx:2676:hs_write_event,ERROR] Event 'System' data size mismatch: expected 412 bytes, got 416 bytes
> 
> 19:43:23.850 2019/10/06 [Logger,ERROR] [history_schema.cxx:455:hs_write_event,ERROR] Event 'System' data size mismatch count: 25, expected 412 bytes, hs_write_event() called with as much as 416 bytes
> 
> 19:43:23.850 2019/10/06 [Logger,ERROR] [history_schema.cxx:455:hs_write_event,ERROR] Event 'System' data size mismatch count: 25, expected 412 bytes, hs_write_event() called with as much as 416 bytes
> 
> The history calculates the size of a record from the size of the individual variables, (history_schema.cxx, L2666 ff), whereas the ODB delivers the data aligned/padded to the size of the largest value in the record.
> In our history, a long list of doubles (64 Bit) fas followed by three floats (32 bit), leading to a padded response from the ODB, 4 byte longer than the history expects.
> Quick fix: Add another 32 bit dummy variable to the history. Gets rid of the error messages...
> Should probably be fixed at a deeper level...

>
> In our history, a long list of doubles (64 Bit) fas followed by three floats (32 bit)
>

Padding trouble, mixing "double" and "float" trouble. Ouch.

Best wisdom I received on this: never use "float", always use "double".

I was burned by "float" with following code, which produced the same result from
analyzing 100 files as from analyzing 1000 files. (why did we take data for 10 weeks
instead of 1 week?). Hint: "float" overflows way too quickly, after overflow sum+=1 does not change
the value of "sum". The actual code used ROOT TH1F. Lesson: always use TH1D.

float sum = 0; // should always be "double" !!!
foreach data_file {
    foreach data from current data file {
        sum += data;
    }   
}
print sum;

K.O.

>I wonder why do you this via ODB links. The "standard" way of writing to the history should be to create events for an equipment and flag this equipment as being written to the
>history. All variables under /Equipment/<name>/Variables then automatically go into the history and you don't have to worry about ODB links. Only variables not fitting the
>equipment/variables scheme should be dealt with via ODB links, like variables under equipment/statistics or parameters in another ODB tree. In a typical midas experiment, only
>very few variables typically go into the 'System' event. This is however probably not a solution to your problem. If you have a similar structure (doubles plus an odd number of floats)
>under 'variables', you might get the same error.

>
> In our history, a long list of doubles (64 Bit) fas followed by three floats (32 bit)
>

We do this in the MuX DAQ and mix things that come directly from MIDAS (the MIDAS trigger rate) and things from the
analyzer (rates in the self-triggering detectors) and some temperatures from yet somewhere else. Yes, we could have
kept that apart, yes, in this case a double would also work (and not break things), but a bug is a bug...
I could think of senisble use cases where doubles and ints are mixed and I also know quite a few areas where it makes
sense to use floats...

Nik

> Yes, we could have
> kept that apart, yes, in this case a double would also work (and not break things), but a bug is a bug...
> I could think of senisble use cases where doubles and ints are mixed and I also know quite a few areas where it makes
> sense to use floats...

I agree with Nik that we should fix this on the midas level. Since it happens in history_schema.cxx which was written by KO, maybe he can have a look.

Stefan

> > The compile option -DHAVE_FTPLIB checked in mdsupport.cxx disappeared if you 
> > compile with cmake.
> 
> Hi, Stefan - do we still need to support FTP in the logger? In the lazylogger, special support for 
> FTP is not needed, they can you the "script" method and do FTP without our help.
> 
> I move to remove FTP support from MIDAS. (second? other opinions?)

I oppose to remove FTP support from lazylogger. We still use it heavily at PSI. In comparison to the "script" method, it 
shows the current speed in MB/s which helps us to diagnose some network problem by writing this number into the 
history. The "script" method only give you an integral transfer speed after a file has be completely written.

I'm however not sure who FTP is used in lazylogger. It goes into mdsupport.cxx and I seem to remember that Pierre 
wrote the FTP code by hand, so no external library is necessary.

Stefan

> > > The compile option -DHAVE_FTPLIB checked in mdsupport.cxx disappeared if you 
> > > compile with cmake.
> > 
> > Hi, Stefan - do we still need to support FTP in the logger? In the lazylogger, special support for 
> > FTP is not needed, they can you the "script" method and do FTP without our help.
> > 
> > I move to remove FTP support from MIDAS. (second? other opinions?)
> 
> I oppose to remove FTP support from lazylogger.

Confirmed. FTP support in lazylogger stays.

K.O.

Context: out of the box  MIDAS (using cmake) on OSX Mojave. 

Running with mongoose/opensslm installation following instruction here:
https://midas.triumf.ca/MidasWiki/index.php/Quickstart_Linux

mhttpd crashing when midas webpage opened with Safari (12.1.2). Usually when opening the "chat" tab but sometimes also with the "message" tab.
mhttpd(11109,0x70000827a000) malloc: *** error for object 0x7f8669501ef0: pointer being freed was not allocated
mhttpd(11109,0x70000827a000) malloc: *** set a breakpoint in malloc_error_break to debug

No crash if using firefox (70.0.1 (64-bit))

> Context: out of the box  MIDAS (using cmake) on OSX Mojave. 
> 
> Running with mongoose/opensslm installation following instruction here:
> https://midas.triumf.ca/MidasWiki/index.php/Quickstart_Linux
> 
> mhttpd crashing when midas webpage opened with Safari (12.1.2). Usually when opening the "chat" tab but sometimes also with the "message" tab.
> mhttpd(11109,0x70000827a000) malloc: *** error for object 0x7f8669501ef0: pointer being freed was not allocated
> mhttpd(11109,0x70000827a000) malloc: *** set a breakpoint in malloc_error_break to debug
> 
> No crash if using firefox (70.0.1 (64-bit))

I think we also have reports of mhttpd crash on macos with safari from the Dragon experiment,
but cannot reproduce the problem.

If you can reproduce this, can you capture the crash stack trace?

One way to do this is to enable core dumps in odb "/expt/enable core dumps" set to "y", restart mhttpd,
wait for the crash. I think macos writes core dumps into /cores/... Or you can run mhttpd inside lldb
and wait for the crash. the lldb command to show the stack trace is "bt", but you may need
to switch to different threads to see which one actually crashed. I forget what the command
for that is.

BTW, the mhttpd networking code has not changed in a long time, but an update
of mongoose web server library is overdue (to fix a memory leak, at least).

K.O.

It is reproducible alright.
Here are the core dump and the backtrace (I think  the former is more informative).



> > Context: out of the box  MIDAS (using cmake) on OSX Mojave. 
> > 
> > Running with mongoose/opensslm installation following instruction here:
> > https://midas.triumf.ca/MidasWiki/index.php/Quickstart_Linux
> > 
> > mhttpd crashing when midas webpage opened with Safari (12.1.2). Usually when opening the "chat" tab but sometimes also with the "message" tab.
> > mhttpd(11109,0x70000827a000) malloc: *** error for object 0x7f8669501ef0: pointer being freed was not allocated
> > mhttpd(11109,0x70000827a000) malloc: *** set a breakpoint in malloc_error_break to debug
> > 
> > No crash if using firefox (70.0.1 (64-bit))
> 
> I think we also have reports of mhttpd crash on macos with safari from the Dragon experiment,
> but cannot reproduce the problem.
> 
> If you can reproduce this, can you capture the crash stack trace?
> 
> One way to do this is to enable core dumps in odb "/expt/enable core dumps" set to "y", restart mhttpd,
> wait for the crash. I think macos writes core dumps into /cores/... Or you can run mhttpd inside lldb
> and wait for the crash. the lldb command to show the stack trace is "bt", but you may need
> to switch to different threads to see which one actually crashed. I forget what the command
> for that is.
> 
> BTW, the mhttpd networking code has not changed in a long time, but an update
> of mongoose web server library is overdue (to fix a memory leak, at least).
> 
> K.O.

> It is reproducible alright.

Thanks. At first blush, a guess, read_passwords() is not thread-safe and is called from multiple threads, not protected by semaphore. Crash report shows 2 active threads 
(one made it is far as processing the mjson rpc, the other one crashed in read_passwords()).

K.O.


> Here are the core dump and the backtrace (I think  the former is more informative).
> 
> 
> 
> > > Context: out of the box  MIDAS (using cmake) on OSX Mojave. 
> > > 
> > > Running with mongoose/opensslm installation following instruction here:
> > > https://midas.triumf.ca/MidasWiki/index.php/Quickstart_Linux
> > > 
> > > mhttpd crashing when midas webpage opened with Safari (12.1.2). Usually when opening the "chat" tab but sometimes also with the "message" tab.
> > > mhttpd(11109,0x70000827a000) malloc: *** error for object 0x7f8669501ef0: pointer being freed was not allocated
> > > mhttpd(11109,0x70000827a000) malloc: *** set a breakpoint in malloc_error_break to debug
> > > 
> > > No crash if using firefox (70.0.1 (64-bit))
> > 
> > I think we also have reports of mhttpd crash on macos with safari from the Dragon experiment,
> > but cannot reproduce the problem.
> > 
> > If you can reproduce this, can you capture the crash stack trace?
> > 
> > One way to do this is to enable core dumps in odb "/expt/enable core dumps" set to "y", restart mhttpd,
> > wait for the crash. I think macos writes core dumps into /cores/... Or you can run mhttpd inside lldb
> > and wait for the crash. the lldb command to show the stack trace is "bt", but you may need
> > to switch to different threads to see which one actually crashed. I forget what the command
> > for that is.
> > 
> > BTW, the mhttpd networking code has not changed in a long time, but an update
> > of mongoose web server library is overdue (to fix a memory leak, at least).
> > 
> > K.O.

I accidentally discovered a problem with the alarm sounds played by midas.

The javascript code is very simple: var audio=new Audio("alarm.mp3"); audio.play();

In the past, this reliably played a sound.

More recently, I started seeing javascript log messages about "unhandled exception" from audio.play(). (for me, they often
interrupt my javascript debugger sessions, in a very annoying way).

Adding an exception handler to audio.play() was not effective: it turns out, these days, audio.play() returns a Promise
and one must handle the Promise rejection case. (also it turns out that in the rejection handler
one *must* clear audio.src to avoid problems with excessive memory and cpu use).

But why does audio.play() throw a rejected promise?!?

This is the error message provided: "play() failed because the user didn't interact with the document first. https://goo.gl/xX8pDD"
name: "NotAllowedError".

The link takes us to https://developers.google.com/web/updates/2017/09/autoplay-policy-changes. This web page seems to explain
everything neatly, but most of the information turns out to be wrong and unhelpful:

a) "Muted autoplay is always allowed" - wrong - I see audio.play() rejection from muted tabs
b) "User has interacted with the domain (click, tap, etc.)." - wrong - I see rejection in the javascript debugger even as I debug the web page
c) "user's Media Engagement Index threshold has been crossed" - broken - playback count and MEI is always 0 (zero) for midas - per chrome://media-engagement/
d) "user has added the site to their home screen on mobile or installed the PWA on desktop" - what ?!?
e) chrome://flags/#autoplay-policy does not exist (was removed)

There may be a way to start google-chrome with special flags to globally enable autoplay, but it seems
to be aimed at "kiosk" type applications, not for general use browsers:
https://stackoverflow.com/questions/57455849/chrome-autoplay-policy-chrome-76

The bottom line:

there is no way to ensure that the alarm sound will always play

K.O.

P.S. I am tracking this problem here:
https://bitbucket.org/tmidas/midas/issues/191/exception-on-audioplay

The document

https://docs.google.com/document/d/1_278v_plodvgtXSgnEJ0yjZJLg14Ogf-ekAFNymAJoU/edit

says that the "... playback must be at lest 7 seconds long" to signal significant playback.

I attached a modified alarm sound file with 3 seconds of sound followed by 5 seconds of silence. Try that one and let me know if this changes anything. If successful, I can modify the 
other sound files as well.

Stefan

Hi all, 

I have a Mac with OS 10.13.6 and Xcode 10.1. I am following the instructions in the wiki to install Midas. 
I have installed openssl via MacPorts as per the instructions. But then I get an error related to open ssl 
when I try to compile Midas. See the results of cmake .. and make install 

Darrens-Mac-mini:build betacage$ cmake ..
-- MIDAS: cmake version: 3.16.3
-- MIDAS: CMAKE_INSTALL_PREFIX: /Users/betacage/packages/midas
-- MIDAS: Found ROOT version 6.18/04
-- MIDAS: Found ZLIB version 1.2.11
-- MIDAS: Found OpenSSL version 1.1.1d
-- MIDAS: MySQL not found
-- MIDAS: ODBC not found
-- MIDAS: Found SQLITE /usr/include/sqlite3.h
-- MIDAS: nvidia-smi not found
-- MIDAS example/experiment: MIDAS in-tree-build
-- MIDAS: Found ZLIB version 1.2.11
-- MIDAS example/experiment: Found ROOT version 6.18/04
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/betacage/packages/midas/build
Darrens-Mac-mini:build betacage$ make install 
[  1%] Built target mfeo
[  3%] Built target mfe
[  4%] Built target rmana
[  5%] Built target manao
[  6%] Built target mana
[  6%] Built target git_revision_h
[ 30%] Built target objlib
[ 31%] Built target midas
[ 32%] Built target rmanao
[ 32%] Built target objlib-c-compat
[ 32%] Built target midas-c-compat
[ 33%] Built target midas-shared
[ 35%] Built target rmlogger
[ 36%] Linking CXX executable mhttpd
Undefined symbols for architecture x86_64:
  "_OPENSSL_init_ssl", referenced from:
      _mg_mgr_init in mongoose6.cxx.o
  "_SSL_CTX_set_options", referenced from:
      _mg_set_ssl in mongoose6.cxx.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [progs/mhttpd] Error 1
make[1]: *** [progs/CMakeFiles/mhttpd.dir/all] Error 2
make: *** [all] Error 2


I hope that this is the right forum to submit this kind of reports.
Any idea what do I have to do to continue ? Thanks is advance !

Berta 

Hi, yes, this is the right place to report problems and to ask questions about midas.

As for your trouble, I have the same macos 10.13.6, so we should be able to figure out what goes wrong.

Unfortunately, the posted instructions (to run cmake directly) hide all the information needed to debug build problems.

Instead of "cd build; cmake ..", please run "make cmake" and post the output. Printed should be all the compiler
and linker commands executed by the build. You can examine the mhttpd linker command to see if somehow
the wrong openssl library is being used.

If you do not need built-in https support in mhttpd, you can run "make cmake NO_SSL=1".

K.O.

> Hi all, 
> 
> I have a Mac with OS 10.13.6 and Xcode 10.1. I am following the instructions in the wiki to install Midas. 
> I have installed openssl via MacPorts as per the instructions. But then I get an error related to open ssl 
> when I try to compile Midas. See the results of cmake .. and make install 
> 
> Darrens-Mac-mini:build betacage$ cmake ..
> -- MIDAS: cmake version: 3.16.3
> -- MIDAS: CMAKE_INSTALL_PREFIX: /Users/betacage/packages/midas
> -- MIDAS: Found ROOT version 6.18/04
> -- MIDAS: Found ZLIB version 1.2.11
> -- MIDAS: Found OpenSSL version 1.1.1d
> -- MIDAS: MySQL not found
> -- MIDAS: ODBC not found
> -- MIDAS: Found SQLITE /usr/include/sqlite3.h
> -- MIDAS: nvidia-smi not found
> -- MIDAS example/experiment: MIDAS in-tree-build
> -- MIDAS: Found ZLIB version 1.2.11
> -- MIDAS example/experiment: Found ROOT version 6.18/04
> -- Configuring done
> -- Generating done
> -- Build files have been written to: /Users/betacage/packages/midas/build
> Darrens-Mac-mini:build betacage$ make install 
> [  1%] Built target mfeo
> [  3%] Built target mfe
> [  4%] Built target rmana
> [  5%] Built target manao
> [  6%] Built target mana
> [  6%] Built target git_revision_h
> [ 30%] Built target objlib
> [ 31%] Built target midas
> [ 32%] Built target rmanao
> [ 32%] Built target objlib-c-compat
> [ 32%] Built target midas-c-compat
> [ 33%] Built target midas-shared
> [ 35%] Built target rmlogger
> [ 36%] Linking CXX executable mhttpd
> Undefined symbols for architecture x86_64:
>   "_OPENSSL_init_ssl", referenced from:
>       _mg_mgr_init in mongoose6.cxx.o
>   "_SSL_CTX_set_options", referenced from:
>       _mg_set_ssl in mongoose6.cxx.o
> ld: symbol(s) not found for architecture x86_64
> clang: error: linker command failed with exit code 1 (use -v to see invocation)
> make[2]: *** [progs/mhttpd] Error 1
> make[1]: *** [progs/CMakeFiles/mhttpd.dir/all] Error 2
> make: *** [all] Error 2
> 
> 
> I hope that this is the right forum to submit this kind of reports.
> Any idea what do I have to do to continue ? Thanks is advance !
> 
> Berta