Context: out of the box  MIDAS (using cmake) on OSX Mojave. 

Running with mongoose/opensslm installation following instruction here:
https://midas.triumf.ca/MidasWiki/index.php/Quickstart_Linux

mhttpd crashing when midas webpage opened with Safari (12.1.2). Usually when opening the "chat" tab but sometimes also with the "message" tab.
mhttpd(11109,0x70000827a000) malloc: *** error for object 0x7f8669501ef0: pointer being freed was not allocated
mhttpd(11109,0x70000827a000) malloc: *** set a breakpoint in malloc_error_break to debug

No crash if using firefox (70.0.1 (64-bit))

> Context: out of the box  MIDAS (using cmake) on OSX Mojave. 
> 
> Running with mongoose/opensslm installation following instruction here:
> https://midas.triumf.ca/MidasWiki/index.php/Quickstart_Linux
> 
> mhttpd crashing when midas webpage opened with Safari (12.1.2). Usually when opening the "chat" tab but sometimes also with the "message" tab.
> mhttpd(11109,0x70000827a000) malloc: *** error for object 0x7f8669501ef0: pointer being freed was not allocated
> mhttpd(11109,0x70000827a000) malloc: *** set a breakpoint in malloc_error_break to debug
> 
> No crash if using firefox (70.0.1 (64-bit))

I think we also have reports of mhttpd crash on macos with safari from the Dragon experiment,
but cannot reproduce the problem.

If you can reproduce this, can you capture the crash stack trace?

One way to do this is to enable core dumps in odb "/expt/enable core dumps" set to "y", restart mhttpd,
wait for the crash. I think macos writes core dumps into /cores/... Or you can run mhttpd inside lldb
and wait for the crash. the lldb command to show the stack trace is "bt", but you may need
to switch to different threads to see which one actually crashed. I forget what the command
for that is.

BTW, the mhttpd networking code has not changed in a long time, but an update
of mongoose web server library is overdue (to fix a memory leak, at least).

K.O.

It is reproducible alright.
Here are the core dump and the backtrace (I think  the former is more informative).



> > Context: out of the box  MIDAS (using cmake) on OSX Mojave. 
> > 
> > Running with mongoose/opensslm installation following instruction here:
> > https://midas.triumf.ca/MidasWiki/index.php/Quickstart_Linux
> > 
> > mhttpd crashing when midas webpage opened with Safari (12.1.2). Usually when opening the "chat" tab but sometimes also with the "message" tab.
> > mhttpd(11109,0x70000827a000) malloc: *** error for object 0x7f8669501ef0: pointer being freed was not allocated
> > mhttpd(11109,0x70000827a000) malloc: *** set a breakpoint in malloc_error_break to debug
> > 
> > No crash if using firefox (70.0.1 (64-bit))
> 
> I think we also have reports of mhttpd crash on macos with safari from the Dragon experiment,
> but cannot reproduce the problem.
> 
> If you can reproduce this, can you capture the crash stack trace?
> 
> One way to do this is to enable core dumps in odb "/expt/enable core dumps" set to "y", restart mhttpd,
> wait for the crash. I think macos writes core dumps into /cores/... Or you can run mhttpd inside lldb
> and wait for the crash. the lldb command to show the stack trace is "bt", but you may need
> to switch to different threads to see which one actually crashed. I forget what the command
> for that is.
> 
> BTW, the mhttpd networking code has not changed in a long time, but an update
> of mongoose web server library is overdue (to fix a memory leak, at least).
> 
> K.O.

> It is reproducible alright.

Thanks. At first blush, a guess, read_passwords() is not thread-safe and is called from multiple threads, not protected by semaphore. Crash report shows 2 active threads 
(one made it is far as processing the mjson rpc, the other one crashed in read_passwords()).

K.O.


> Here are the core dump and the backtrace (I think  the former is more informative).
> 
> 
> 
> > > Context: out of the box  MIDAS (using cmake) on OSX Mojave. 
> > > 
> > > Running with mongoose/opensslm installation following instruction here:
> > > https://midas.triumf.ca/MidasWiki/index.php/Quickstart_Linux
> > > 
> > > mhttpd crashing when midas webpage opened with Safari (12.1.2). Usually when opening the "chat" tab but sometimes also with the "message" tab.
> > > mhttpd(11109,0x70000827a000) malloc: *** error for object 0x7f8669501ef0: pointer being freed was not allocated
> > > mhttpd(11109,0x70000827a000) malloc: *** set a breakpoint in malloc_error_break to debug
> > > 
> > > No crash if using firefox (70.0.1 (64-bit))
> > 
> > I think we also have reports of mhttpd crash on macos with safari from the Dragon experiment,
> > but cannot reproduce the problem.
> > 
> > If you can reproduce this, can you capture the crash stack trace?
> > 
> > One way to do this is to enable core dumps in odb "/expt/enable core dumps" set to "y", restart mhttpd,
> > wait for the crash. I think macos writes core dumps into /cores/... Or you can run mhttpd inside lldb
> > and wait for the crash. the lldb command to show the stack trace is "bt", but you may need
> > to switch to different threads to see which one actually crashed. I forget what the command
> > for that is.
> > 
> > BTW, the mhttpd networking code has not changed in a long time, but an update
> > of mongoose web server library is overdue (to fix a memory leak, at least).
> > 
> > K.O.

compiling:
----------

The compile option -DHAVE_FTPLIB checked in mdsupport.cxx disappeared if you 
compile with cmake.

I added:

  add_compile_options(-DHAVE_FTPLIB)

to CMakeLists.txt to fix this. Can probably be done in a more elegant way 
(checking if the right libraries exist?).

running:
-------

Our MAX_EVENT_SIZE is set in the odb to 805306368. This number is also used in 

  INT lazy_copy(char *outfile, char *infile, int max_event_size)

this is to big when copying with ftp, causing a crash. Reducing it here with a 
factor 10 solves our problems.

> The compile option -DHAVE_FTPLIB checked in mdsupport.cxx disappeared if you 
> compile with cmake.

Hi, Stefan - do we still need to support FTP in the logger? In the lazylogger, special support for 
FTP is not needed, they can you the "script" method and do FTP without our help.

I move to remove FTP support from MIDAS. (second? other opinions?)

> Our MAX_EVENT_SIZE is set in the odb to 805306368. This number is also used in 
> this is to big when copying with ftp, causing a crash. Reducing it here with a 
> factor 10 solves our problems.

I am surprised that changing MAX_EVENT_SIZE (to a "too big" value) causes lazylogger to 
crash. More usually MAX_EVENT_SIZE has no effect until you try to write an event that is 
somehow "too big", then there is a crash. Perhaps there is a bug specifically in the FTP code.

Anyhow, I recommend the solution of using the "script" method. We have example lazylogger 
scripts in midas/progs/lazy*.perl (the scripts do not have to be in perl, python is ok). We do
not have any example that uses FTP because we do not use FTP for data storage. But you can
easily adapt lazy_test.perl and lazy_copy.perl to use scp and sftp, the secure versions of FTP.

K.O.

> > The compile option -DHAVE_FTPLIB checked in mdsupport.cxx disappeared if you 
> > compile with cmake.
> 
> Hi, Stefan - do we still need to support FTP in the logger? In the lazylogger, special support for 
> FTP is not needed, they can you the "script" method and do FTP without our help.
> 
> I move to remove FTP support from MIDAS. (second? other opinions?)

I oppose to remove FTP support from lazylogger. We still use it heavily at PSI. In comparison to the "script" method, it 
shows the current speed in MB/s which helps us to diagnose some network problem by writing this number into the 
history. The "script" method only give you an integral transfer speed after a file has be completely written.

I'm however not sure who FTP is used in lazylogger. It goes into mdsupport.cxx and I seem to remember that Pierre 
wrote the FTP code by hand, so no external library is necessary.

Stefan

> > > The compile option -DHAVE_FTPLIB checked in mdsupport.cxx disappeared if you 
> > > compile with cmake.
> > 
> > Hi, Stefan - do we still need to support FTP in the logger? In the lazylogger, special support for 
> > FTP is not needed, they can you the "script" method and do FTP without our help.
> > 
> > I move to remove FTP support from MIDAS. (second? other opinions?)
> 
> I oppose to remove FTP support from lazylogger.

Confirmed. FTP support in lazylogger stays.

K.O.

I keep on getting messages like this:

16:25:35 [fecaen,ERROR] [odb.c:4567:db_get_data,ERROR] data for key
"/DAQ/params/VX1730/custom/Board 0/Channel 0/Input range" truncated

whenever I start my frontend. Input range is defined to be a BOOL and using
odbedit to read it shows:

Key name                        Type    #Val  Size  Last Opn Mode Value
---------------------------------------------------------------------------
Input range                     BOOL    1     4     75h  0   RWD  y

without any error message. The entry is read using

         size = sizeof(fInputRange);
         db_get_data(hDb, hSubKey, &fInputRange, &size, TID_BOOL);

where fInputRange is a bool.

Where does this message come from and how can I resolve this?

> I keep on getting messages like this:
> 16:25:35 [fecaen,ERROR] [odb.c:4567:db_get_data,ERROR] data for key
> "/DAQ/params/VX1730/custom/Board 0/Channel 0/Input range" truncated
>
>  [  bool fInputRange... ]
>          size = sizeof(fInputRange);
>          db_get_data(hDb, hSubKey, &fInputRange, &size, TID_BOOL);
>

The error is correct. size of TID_BOOL is 4 byte (uint32_t) and you give is sizeof(bool) instead which is probably not 4.

Note that sizeof(bool) is not well defined, sometimes it is 1 (you need 4), sometimes something else, see
https://stackoverflow.com/questions/4897844/is-sizeofbool-defined-in-the-c-language-standard

A good fix would be to change fInputRange from bool to uint32_t (which is always 4 byte size).

#include <stdint.h>
...
uint32_t fInputRange;

K.O.

We have encountered a safety issue with our HPGe HV and it's midas frontend. Turning off or changing HV unknowingly has to be avoided at all costs.

Current safety protection

We use the DF_REPORT_STATUS flag to give the hardware settings precedence over odb settings. This all takes place in the init.

DAQ recovery Issue?

In the setup / development state, we sometimes have to remove the SHM files and reload an odb dump to recover the DAQ. When the FE is running, this can modify hardware settings. E.g. change a voltage

Question

Is there a way one can let the frontend know the "load"  function is called in odbedit? Or other suggestions to build in this safety.

> We have encountered a safety issue with our HPGe HV and it's midas frontend.

At TRIUMF and other labs the words "safety issue" have very specific meaning and
we tend to follow this guidance: MIDAS is not certified for and is not intended for use with 
safety critical applications as defined here:
https://en.wikipedia.org/wiki/Safety-critical_system

> A safety-critical system ... malfunction may result in ... following outcomes:
> death or serious injury to people
> loss or severe damage to equipment/property
> environmental harm

If this is your case, you should use properly certified software *and hardware*. Safety 
officers at most institutions require certified hardware interlocks and other protections to 
prevent such undesirable outcomes. Use of certified PLCs is sometimes permitted.

But I suspect in your case, there is no "safety issue", you only want to protect some 
valuable but not critical equipment against accidental damage.

In this case, you can probably use midas, but if midas malfunction may result in destroying 
your experiment (i.e. accidentally set wrong voltage on 3000 phototubes), you should also 
have hardware based protections (hardware limits on max/min high voltage). Most HV 
power supplies implement such protections (screw-driver actuated max voltage limits).

If there is danger of destroying your experiment you should also have an independent 
review of your control system to avoid avoidable mistakes and obvious problems.

> Turning off or changing HV unknowingly has to be avoided at all costs

The function of changing high-voltage is implemented in your frontend program. Right in 
the place in this program where you transmit the voltage setting from ODB to the hardware 
is where you implement your protections (validate the voltage range, check that changing 
the voltage is permitted, etc). This protects you against unexpected/incorrect/erroneous
changes in ODB (wrong ODB is loaded, wrong values in ODB, ODB is corrupted, etc).

In addition, it is wise to set software based limits in the HV power supply (software 
controlled max high voltage, software controlled max current, etc). Most HV power supplies 
implement such functions.

To ensure high voltage cannot be changed at the wrong times, you can also implement 
procedural and hardware protections, such as unplug the power supply control connection 
(usually ethernet or serial or usb cable). This will prevent you from monitoring the high 
voltage currents and the only solution is to use a  power supply with a hardware "write 
protect" function (a key needs to be inserted and turned to allow changing anything).

All of this is generic and applies to any controls software, not just MIDAS.

Without at least some of these protections (especially protections in your frontend 
program), the questions you asked about loading ODB are insufficient.

K.O.

Dear Konstantin,

So let me retract the term "safety issue" then, it was more a request/question for this type of 
info between the fe and the odb.

We have most of what you mention:
* The HV hardware has current limits
* The Hardware has fixed ramping limits.

same for the software. 

The issue occurs when e.g. one channel can not be turned on and ramp for some temp/specific 
reason, and someone else is working on the daq and reloads the odb for e.g. 1h ago.  

> > We have encountered a safety issue with our HPGe HV and it's midas frontend.
> 
> At TRIUMF and other labs the words "safety issue" have very specific meaning and
> we tend to follow this guidance: MIDAS is not certified for and is not intended for use with 
> safety critical applications as defined here:
> https://en.wikipedia.org/wiki/Safety-critical_system
> 
> > A safety-critical system ... malfunction may result in ... following outcomes:
> > death or serious injury to people
> > loss or severe damage to equipment/property
> > environmental harm
> 
> If this is your case, you should use properly certified software *and hardware*. Safety 
> officers at most institutions require certified hardware interlocks and other protections to 
> prevent such undesirable outcomes. Use of certified PLCs is sometimes permitted.
> 
> But I suspect in your case, there is no "safety issue", you only want to protect some 
> valuable but not critical equipment against accidental damage.
> 
> In this case, you can probably use midas, but if midas malfunction may result in destroying 
> your experiment (i.e. accidentally set wrong voltage on 3000 phototubes), you should also 
> have hardware based protections (hardware limits on max/min high voltage). Most HV 
> power supplies implement such protections (screw-driver actuated max voltage limits).
> 
> If there is danger of destroying your experiment you should also have an independent 
> review of your control system to avoid avoidable mistakes and obvious problems.
> 
> > Turning off or changing HV unknowingly has to be avoided at all costs
> 
> The function of changing high-voltage is implemented in your frontend program. Right in 
> the place in this program where you transmit the voltage setting from ODB to the hardware 
> is where you implement your protections (validate the voltage range, check that changing 
> the voltage is permitted, etc). This protects you against unexpected/incorrect/erroneous
> changes in ODB (wrong ODB is loaded, wrong values in ODB, ODB is corrupted, etc).
> 
> In addition, it is wise to set software based limits in the HV power supply (software 
> controlled max high voltage, software controlled max current, etc). Most HV power supplies 
> implement such functions.
> 
> To ensure high voltage cannot be changed at the wrong times, you can also implement 
> procedural and hardware protections, such as unplug the power supply control connection 
> (usually ethernet or serial or usb cable). This will prevent you from monitoring the high 
> voltage currents and the only solution is to use a  power supply with a hardware "write 
> protect" function (a key needs to be inserted and turned to allow changing anything).
> 
> All of this is generic and applies to any controls software, not just MIDAS.
> 
> Without at least some of these protections (especially protections in your frontend 
> program), the questions you asked about loading ODB are insufficient.
> 
> K.O.

> 
> The issue occurs when e.g. one channel can not be turned on and ramp for some temp/specific 
> reason, and someone else is working on the daq and reloads the odb for e.g. 1h ago.  
> 

So you want to ensure that some HV channels are turned off and stay turned off. Yes?

Most effective solution will depend on the consequences of unwanted turning-on of your channels:

- if hardware is destroyed if turned on - I think you should have a hardware lock-out. (unplug the HV cable)
- if hardware malfunctions and will degrade if left turned on for long time (i.e. a hot phototube or sparking wire chamber) - your data 
monitoring software should detect the anomaly (it will show up as a hot channel, dead channel, etc) and the people running the 
experiment will realize the mistake and turn the channel back off. also hardware monitoring (HV currents, etc) should detect this, with 
same effect.
- if collected data becomes useless (the turned-off channel make big noise in all other channels), then same thing, your data 
monitoring should catch it.

The next consideration is what are you protecting against:

a) one person flags channel defective, turns it off, next person knows nothing, turns it back on - you need to work on documentation, 
shift hand-off and other human-level procedures
b) people running experiment load random odb files - same thing, from human-level procedures and documentation it should be made 
clear which odb files are correct and which should not be used
c) software malfunction (not human person) causes data change in odb causes turned-off channel to turn back on
d) hardware malfunction causes turned-off channel to turn back on (HV power supply hardware or firmware malfunctions and decides 
that all channels should be turned on at maximum high voltage)

In the experiments I am most familiar with, problem (b) is avoided by never loading/reloading odb files directly, most/all interaction
with the experiment is done through web pages, and these web pages are carefully coded to be safe against most user mistakes.

Cases (a), (b) and (c) you can protect against by changing the frontend code to refuse to turn on some channels:

int set_hv(int channel, int voltage) {
   if (channel == 35) return COMMAND_REFUSED;
   write_to_hardware(channel, voltage);
   return COMMAND_SUCCESS;
}

But in reality this solution only creates problem (e):

e) people running the experiment start random versions of the frontend program, make random changes to the frontend source code, 
multiple people working on the frontend have their own personal versions/copies of the source code, etc.

This is the worst-case scenario, meaning the experiment lost control of software configuration, and even basic software version 
control tools (like svn or git) are not being used. If your experiment gets that chaotic, all protections are likely to be ineffective - 
documentation will not work (people will ignore post-it notes "do not turn on!"), hardware protections will not work (unplugged cable 
labeled "do not plug in!" will be plugged back in and powered), etc. good luck, then.

K.O.

There is a not-so-well-known function in the ODB to write protect some keys. You can do

odbedit> chmod 1 /Equipment/HV/Demand

which will write protect your Demand values. You see that by doing

odbedit> ls -ls

where you only see a "R" at the right end instead a "RWD". I haven't tried it yet (so better do a dry run yourself), but that should prevent an odb load to overwrite your demand values. To change the values, put some logic on a custom page to unprotect the 
values, change them, and then protect them again.

Stefan

Hi,

I have been looking at the 2019 workshop slides, I am interested in the C++ future of MIDAS. 

I am quite interested in using the object oriented 


ALPHA will start data taking in 2021

Logging a list of variables to the history via links in the history ODB subtree,
we get messages as follows at every run start:

19:43:24.009 2019/10/06 [Logger,ERROR] [history_schema.cxx:2676:hs_write_event,ERROR] Event 'System' data size mismatch: expected 412 bytes, got 416 bytes

19:43:24.008 2019/10/06 [Logger,ERROR] [history_schema.cxx:2676:hs_write_event,ERROR] Event 'System' data size mismatch: expected 412 bytes, got 416 bytes

19:43:23.850 2019/10/06 [Logger,ERROR] [history_schema.cxx:455:hs_write_event,ERROR] Event 'System' data size mismatch count: 25, expected 412 bytes, hs_write_event() called with as much as 416 bytes

19:43:23.850 2019/10/06 [Logger,ERROR] [history_schema.cxx:455:hs_write_event,ERROR] Event 'System' data size mismatch count: 25, expected 412 bytes, hs_write_event() called with as much as 416 bytes

The history calculates the size of a record from the size of the individual variables, (history_schema.cxx, L2666 ff), whereas the ODB delivers the data aligned/padded to the size of the largest value in the record.
In our history, a long list of doubles (64 Bit) fas followed by three floats (32 bit), leading to a padded response from the ODB, 4 byte longer than the history expects.
Quick fix: Add another 32 bit dummy variable to the history. Gets rid of the error messages...
Should probably be fixed at a deeper level...

I wonder why do you this via ODB links. The "standard" way of writing to the history should be to create events for an equipment and flag this equipment as being written to the
history. All variables under /Equipment/<name>/Variables then automatically go into the history and you don't have to worry about ODB links. Only variables not fitting the
equipment/variables scheme should be dealt with via ODB links, like variables under equipment/statistics or parameters in another ODB tree. In a typical midas experiment, only
very few variables typically go into the 'System' event. This is however probably not a solution to your problem. If you have a similar structure (doubles plus an odd number of floats)
under 'variables', you might get the same error. I'n in contact with KO to fix this problem at the root level.

Stefan

> Logging a list of variables to the history via links in the history ODB subtree,
> we get messages as follows at every run start:
> 
> 19:43:24.009 2019/10/06 [Logger,ERROR] [history_schema.cxx:2676:hs_write_event,ERROR] Event 'System' data size mismatch: expected 412 bytes, got 416 bytes
> 
> 19:43:24.008 2019/10/06 [Logger,ERROR] [history_schema.cxx:2676:hs_write_event,ERROR] Event 'System' data size mismatch: expected 412 bytes, got 416 bytes
> 
> 19:43:23.850 2019/10/06 [Logger,ERROR] [history_schema.cxx:455:hs_write_event,ERROR] Event 'System' data size mismatch count: 25, expected 412 bytes, hs_write_event() called with as much as 416 bytes
> 
> 19:43:23.850 2019/10/06 [Logger,ERROR] [history_schema.cxx:455:hs_write_event,ERROR] Event 'System' data size mismatch count: 25, expected 412 bytes, hs_write_event() called with as much as 416 bytes
> 
> The history calculates the size of a record from the size of the individual variables, (history_schema.cxx, L2666 ff), whereas the ODB delivers the data aligned/padded to the size of the largest value in the record.
> In our history, a long list of doubles (64 Bit) fas followed by three floats (32 bit), leading to a padded response from the ODB, 4 byte longer than the history expects.
> Quick fix: Add another 32 bit dummy variable to the history. Gets rid of the error messages...
> Should probably be fixed at a deeper level...

>
> In our history, a long list of doubles (64 Bit) fas followed by three floats (32 bit)
>

Padding trouble, mixing "double" and "float" trouble. Ouch.

Best wisdom I received on this: never use "float", always use "double".

I was burned by "float" with following code, which produced the same result from
analyzing 100 files as from analyzing 1000 files. (why did we take data for 10 weeks
instead of 1 week?). Hint: "float" overflows way too quickly, after overflow sum+=1 does not change
the value of "sum". The actual code used ROOT TH1F. Lesson: always use TH1D.

float sum = 0; // should always be "double" !!!
foreach data_file {
    foreach data from current data file {
        sum += data;
    }   
}
print sum;

K.O.

>I wonder why do you this via ODB links. The "standard" way of writing to the history should be to create events for an equipment and flag this equipment as being written to the
>history. All variables under /Equipment/<name>/Variables then automatically go into the history and you don't have to worry about ODB links. Only variables not fitting the
>equipment/variables scheme should be dealt with via ODB links, like variables under equipment/statistics or parameters in another ODB tree. In a typical midas experiment, only
>very few variables typically go into the 'System' event. This is however probably not a solution to your problem. If you have a similar structure (doubles plus an odd number of floats)
>under 'variables', you might get the same error.

>
> In our history, a long list of doubles (64 Bit) fas followed by three floats (32 bit)
>

We do this in the MuX DAQ and mix things that come directly from MIDAS (the MIDAS trigger rate) and things from the
analyzer (rates in the self-triggering detectors) and some temperatures from yet somewhere else. Yes, we could have
kept that apart, yes, in this case a double would also work (and not break things), but a bug is a bug...
I could think of senisble use cases where doubles and ints are mixed and I also know quite a few areas where it makes
sense to use floats...

Nik

> Yes, we could have
> kept that apart, yes, in this case a double would also work (and not break things), but a bug is a bug...
> I could think of senisble use cases where doubles and ints are mixed and I also know quite a few areas where it makes
> sense to use floats...

I agree with Nik that we should fix this on the midas level. Since it happens in history_schema.cxx which was written by KO, maybe he can have a look.

Stefan

> online monitor would show (almost in real-time) the 
> gain, the dark noise, and the pedestal for all the channels, the 2D tracks 
> inside the detectors for each spill and so on.

Hmm... I now realize that the midas distribution does not include an example web page that 
integrates all these elements into one easy to understand html file.

I think an example page that would answer your questions and the questions from the other 
thread about starting/stopping runs, should include following elements:

- the general midas web page framework (the midas left-hand side menu, the top side 
status display, Stefan's new odb tags)
- buttons to start and stop runs (javascript code to call the run transition RPCs)
- embedded images for history display (old style gif and new style canvas)
- embedded images for ROOT histograms (via the ROOT http server and jsroot)
- code to live-update all these elements independantly from each other (to allow history 
plots and ROOT histograms to update at different frequencies).

As for the web page code for showing a mini-event-display, I think we do not know yet how 
to do - the event data lives inside the analyzer as C++ data structures, so somehow it 
needs to be encoded as json (this code is missing - but one can use the ROOT C++ to json 
encoder/streamer), needs to be transported to the web browser (we know how to do this) 
and at the end, plotting the json data on a canvas is the easy part.

I know some experiments have done all of this, and I think we should have such a pipeline 
available as part of the ROOTANA package. Maybe some day...

K.O.

Dear Thomas and Konstantin,

thank you very much for the feedback. I found the ROOTANA javascript display a good source of 
information and references.

As Thomas said, maybe the simplest thing would be to use the ROOT THttpServer. Honestly, I do 
not think that ROOT was ever meant to act as an online monitor due to its wacky memory 
management and abysmal multithread support. In other words, I think that by using ROOT we would 
inevitably lose some performance. 

Perhaps there are better ways of achieving the same goal. For example, I was leaning towards a 
plotly.js based approach where I would encode a series of vectors in base64 strings (for better 
transmission performance), send them to the client through the MJSONRPC mechanism, decode them 
and then feed them to plotly.js. But in this case, I should study many new libraries 
(plotly.js, the library for the base64 encoding, the Gaussian fitting, etc...) and I do not 
have the time to do that now: "beam is coming".

So ROOT it is. I will use the ROOTANA javascript display as a reference. Do you happen to know 
who wrote that part? 

In that example, you have some "static" histograms that you keep always in memory, while in our 
case the number of channels is so big that we have to dynamically generate the histograms only 
when needed (when the user select a single channel).

Best regards
Giorgio

> 
> As Thomas said, maybe the simplest thing would be to use the ROOT THttpServer. Honestly, I do 
> not think that ROOT was ever meant to act as an online monitor due to its wacky memory 
> management and abysmal multithread support. In other words, I think that by using ROOT we would 
> inevitably lose some performance.
>

Yes. The previous data analysis frameworks - PAW/PAW++/CERNlib (CERN), NOVA (TRIUMF) - certainly had
support for online monitoring. In CERNlib/PAW the histograms were stored in shared memory,
the analyzer running in the background was filling them, the PAW/PAW++ display was displaying
them "live". I was very surprised to find this function removed/not implemented in ROOT, given
that the same people were behind both projects (Rene Brun & co).

We tried to roll our own implementation of this in ROOTANA/ROODY, with mixed success.

I am glad the JSROOT project finally gained traction and web based "I can see the data" is now
available in ROOT.

> 
> Perhaps there are better ways of achieving the same goal. For example, I was leaning towards a 
> plotly.js based approach where 
>

There are many web/javascript graphics libraries out there, all have the weak spot - how do you
get your data into them?

Going forward, I see us standardizing on JSROOT: https://root.cern.ch/js/

>
> ... send them to the client through the MJSONRPC mechanism ...
>

I am not sure JSROOT have any support for interacting from the web page to the back-end analyzer. Perhaps
we can use the MIDAS MJSONRPC library for this. Hmm... (Note that the ROOT HTTP server is a derivative
of the mongoose web server library, which we use in mhttpd, so I already know how to work it)

>
> I would encode a series of vectors in base64 strings (for better transmission performance)
>

We looked into this when deciding on the data encoding for the midas history data. There is a tradeoff
between network use and cpu use - to save on the network, you try to reduce the data size by using
compressed binary data - to save on the CPU you try to minimize data encoding.

For history data, we gave up on binary json (extra decoding needed), gave up on text json (extra decoding 
needed), gave up on compression (extra cpu use for decompression) and use javascript native binary processing
("arraybuffer").

Our thinking is that network bandwidth is usually quite big and is getting bigger, but cpu resource is limited
and is expensive. (mobile devices seems to be stuck with ~2 GHz CPUs; cpu use means battery use and
battery capacity is limited, not improving quickly)

> 
> So ROOT it is. I will use the ROOTANA javascript display as a reference. Do you happen to know 
> who wrote that part? 
> 

Yes. See "Contact" at https://root.cern.ch/js/

>
> In that example, you have some "static" histograms that you keep always in memory, while in our 
> case the number of channels is so big that we have to dynamically generate the histograms only 
> when needed (when the user select a single channel).
> 

This requires interaction with the analyzer, requires some kind of RPC mechanism. I am now curious what jsroot 
have, also it would not be too hard to add the mjsonrpc library to rootana. Cooperation from ROOT multithreading
is not required: I can queue the RPC requests in a separate (thread safe, non-ROOT) buffer, then process
them in the ROOT main event loop (this is how the ROOTANA histogram server worked in the days when
ROOT had no multithread support at all).

K.O.

I recently upgraded to MIDAS version midas-2019-06-b. I had to make a few changes 
to get our custom page running again, but am a little confused on starting and 
stopping runs. When I click on my "Start" button, it now redirects to a 
Transition page rather than reloading the status page. The standard MIDAS status 
page does the same. Could someone explain the reasoning for the current behavior?

Furthermore my "Stop" button is now broken with the following error:
Error: Invalid URL "CS/EngeRun&" or query "cmd=Stop&redir=EngeRun%26" or command 
"Stop"

I looked through the mhttpd.js code and managed to get the start button to load 
the status page, at least, but the stop button seems to be written differently. 
For example, start calls:
location.search = "cmd=Transition";
whereas stop does:
mhttpd_goto_page("Transition"); // DOES NOT RETURN

Can anyone offer any insights or advice? I can change the former to "cmd=Status", but 
the latter doesn't allow it.

> I recently upgraded to MIDAS version midas-2019-06-b. I had to make a few changes 
> to get our custom page running again, but am a little confused on starting and 
> stopping runs.

So far so good.

> When I click on my "Start" button, it now redirects to a 
> Transition page rather than reloading the status page.

Are you sure? The "start" button redirects to the "start" page (start.html) which redirects
to the "transition" page (transition.html), which does not redirect anywhere so you can see
the result of the transition.

> Could someone explain the reasoning for the current behavior?

It's been like this for years now. Stefan suggest that we implement the "start" page
and the "transition" page as overlays on top of the status page, but it did not happen yet.

> Furthermore my "Stop" button is now broken with the following error:
> Error: Invalid URL "CS/EngeRun&" or query "cmd=Stop&redir=EngeRun%26" or command  "Stop"

I grep for "EngeRun" and I do not see it anywhere in the midas sources. Can you grep for it
to see if it is coming from one of your pages?

If you want to start/stop runs from your custom page, look at start.html and transition.html - you will
need to make the run transition RPC calls (cut-and-paste the code to your page) and (obviously)
you will not have any redirects to some strange pages.

> For example, start calls:
> location.search = "cmd=Transition";
> whereas stop does:
> mhttpd_goto_page("Transition"); // DOES NOT RETURN

It's the same thing, look at mhttpd_goto_page().

> Can anyone offer any insights or advice? I can change the former to "cmd=Status", but 
> the latter doesn't allow it.

I am not sure what you are trying to do. If you need the "start" button on the status page
to do something different from what it does now, just hack status.html until it does so.
If you need some specific help with that, I am happy to help. I think I answered all questions
you asked so far.

K.O.

    for (INT i = 0; i < hv_info->num_channels; i++) {
      char element[HKEY_STRING_LENGTH];
      snprintf(element, HKEY_STRING_LENGTH, "%s[%d]", path, i);
      if (db_find_key(hDB, hv_info->hKeyRoot, element, &hKey) == DB_SUCCESS) {
        if ((hv_info->driver[i]->flags & flag) == 0) 
          db_open_record(hDB, hKey, &array[i], sizeof(double), MODE_READ, callback, pequipment);
        else
          db_open_record(hDB, hKey, &array[i], sizeof(double), MODE_READ, NULL, NULL);
      } else {
        cm_msg(MERROR, __func__, "Key %s not found", element);
      }
    }

> Is it possible to open a hotlink to a single element in an ODB array?

Not possible.

> sprintf(element, "%s[%d]", path, i);
> db_find_key(hDB, hv_info->hKeyRoot, element, &hKey);

There is some confusion and inconsistency between db_xxx() functions,
some of them accept the array index "a[10]" syntax, some do not.

db_find_key() and db_watch()/db_open_record() do not operate on array elements
and do not accept the "a[10]" array index syntax.

K.O.

> I will try to use the db_watch function in the future.

Note that db_watch() and db_open_record() work exactly the same way, both only allow 
watching "whole" odb entries, you cannot watch individual array elements.

The db_watch() callback function gives you the array index of the array element that was 
changed and that fired the notification. 

*but*

If you change many array elements quickly you will not necessary receive notifications for 
all and each of of them (underlying transport is UDP allows notification packet loss).

If you are watching 1 array element change at a slow rate (1/sec), db_watch() will work well.

Otherwise, you can watch the whole array, in the db_watch() callback, read the new array 
contents, compare it with your saved copy of pervious array contents, identify which array 
elements have changed and dance from here. (this method does not work if you do not 
actually change the array element values: change from "1" to "1", this is an old weakness in 
the midas hot link mechanism).

If you are not sure how to use db_watch(), look inside midas/progs/odbedit.cxx search for 
db_watch() and search for the db_watch() callback function.

K.O.

During my visit at TRIUMF we rewrote the history plotting functionality of midas. Instead of 
static GIF images, we have now interactive JavaScript panels where we can scroll, zoom, 
inspect values and much more (example is attached). We are now in a state where this is still 
work in progress, but already at this stage it might be useful for others to report any 
feedback.

Simply upgrade the the newest develop branch of midas, and you will see two menu items 
"OldHistory" which is the old system and "History" which is the new system. In the new 
system, you can drag with the mouse to scroll, use the mouse wheel to zoom in and out the 
time axis, and hover with your mouse over data points to see its value. If you zoom out, 
old data is loaded automatically in the background.

Following items are planned, but not yet implemented:

- Printing of run markers as in the old history

- Delete old data in the buffer to limit memory consumption if the browser window is 
   open for very long (weeks)

- Implement time interval selector (clock icon, select "last day", "last 8 hours" etc.)

- New settings dialog as a floating dialog box. At the moment, the setting page of the 
   old history system is used

- Export / Printing / Sending to ELOG any history plot

- Implement a formula for plotting data, such as "y = 12 * (x-14) +32". This will replace 
   the old "offset" and "factor" and is more flexible. The formula can be passed directly 
   to the JavaScript engine and will be executed on the web page. It should be also 
   possible to combine different channels, like the difference of two history values.

- Determine the number of digits for variable display from the axis limits. Like if a value 
   changes between 520001 and 520002 only, we need more digits than the usual 6.

Many of these things will be implemented in the next weeks. If you have any more idea 
or find some bugs, please report back to me.

Best,
Stefan for the midas team

I like the new history system very much, but I stumbled over a couple of issues.
I used the version "Thu Aug 29 08:24:29 2019 +0200 -
midas-2019-06-b-244-gdd6585bb on branch develop":

1) it would be nice to have an option to format the label output (see attachment 1)

2) the background of a history plot is very handy if you only show one measure.
If you have multiple ones (see attachment 2), this is not the case anymore. It
would be nice if the background could be enabled/disabled.

> During my visit at TRIUMF we rewrote the history plotting functionality of
midas. Instead of 
> static GIF images, we have now interactive JavaScript panels where we can
scroll, zoom, 
> inspect values and much more (example is attached). We are now in a state
where this is still 
> work in progress, but already at this stage it might be useful for others to
report any 
> feedback.
> 
> Simply upgrade the the newest develop branch of midas, and you will see two
menu items 
> "OldHistory" which is the old system and "History" which is the new system. In
the new 
> system, you can drag with the mouse to scroll, use the mouse wheel to zoom in
and out the 
> time axis, and hover with your mouse over data points to see its value. If you
zoom out, 
> old data is loaded automatically in the background.
> 
> Following items are planned, but not yet implemented:
> 
> - Printing of run markers as in the old history
> 
> - Delete old data in the buffer to limit memory consumption if the browser
window is 
>    open for very long (weeks)
> 
> - Implement time interval selector (clock icon, select "last day", "last 8
hours" etc.)
> 
> - New settings dialog as a floating dialog box. At the moment, the setting
page of the 
>    old history system is used
> 
> - Export / Printing / Sending to ELOG any history plot
> 
> - Implement a formula for plotting data, such as "y = 12 * (x-14) +32". This
will replace 
>    the old "offset" and "factor" and is more flexible. The formula can be
passed directly 
>    to the JavaScript engine and will be executed on the web page. It should be
also 
>    possible to combine different channels, like the difference of two history
values.
> 
> - Determine the number of digits for variable display from the axis limits.
Like if a value 
>    changes between 520001 and 520002 only, we need more digits than the usual 6.
> 
> Many of these things will be implemented in the next weeks. If you have any
more idea 
> or find some bugs, please report back to me.
> 
> Best,
> Stefan for the midas team

> 1) it would be nice to have an option to format the label output (see attachment 1)

That's clearly a bug, I will fix it.

 
> 2) the background of a history plot is very handy if you only show one measure.
> If you have multiple ones (see attachment 2), this is not the case anymore. It
> would be nice if the background could be enabled/disabled.

Looking at your plot, even without the background things look messy. Please note
that you can display only a single curve by double clicking on it (and back with Escape).
If all curves are on top of each other, you can get them apart a bit by zooming
in to the vertical axis, then double click. Let ma know if that works for you.

Best regards,
Stefan

> > 2) the background of a history plot is very handy if you only show one measure.
> > If you have multiple ones (see attachment 2), this is not the case anymore. It
> > would be nice if the background could be enabled/disabled.
> 
> Looking at your plot, even without the background things look messy. Please note
> that you can display only a single curve by double clicking on it (and back with Escape).
> If all curves are on top of each other, you can get them apart a bit by zooming
> in to the vertical axis, then double click. Let ma know if that works for you.

This I found out, yet the attachment here shows another case where it would be useful to be
able to disable the background, namely if you have positive and negative measures in one
plot. Somehow it suggests that CH1 and CH2 show very different values, whereas it is only a
difference in the sign of this variables.

It's not all the important but I would like to mention this is the early stage before
everything is fully frozen.

> This I found out, yet the attachment here shows another case where it would be useful to be
> able to disable the background, namely if you have positive and negative measures in one
> plot. Somehow it suggests that CH1 and CH2 show very different values, whereas it is only a
> difference in the sign of this variables.

Ok, I added an option which lets you switch off the background. 

I also changed the background drawing such that it only goes to the y=0 axis, not the bottom of the screen. 
That should help displaying negative values.

Stefan

> 1) it would be nice to have an option to format the label output (see attachment 1)

I fixed that in the current version.

Stefan

Our typical use case is that a lot of people are connected to the experiment
having some history tabs open most of the time. Hence, I setup a test system and
connect to it from all kind of systems/browsers. What I see currently quite
often is the error hs_read_arraybuffer (see the attachement).

For firefox 60.8.0esr this can result into a full freeze of the tab and only
closing it is possible.

For chromium based browsers you eventually get a popup informing that it is not
responsive anymore.

The worst though is safari 12.1.2 which not only freezes the tab, but
reproducibly crashes the mhttpd on the server side.

Are there ways to get a log which would document where the problems start?  

> I see currently quite often is the error hs_read_arraybuffer (see the 
attachement).
> Are there ways to get a log which would document where the problems 
start?
> [also crash of mhttpd]

We can debug it from both ends, javascript and mhttpd:

On the web page, the error message says "see javascript console", do you see 
anything there?

Or the tab is so hung-up that you cannot even access the console? In this 
case, can you open the console before running your test?

In some browsers (firefox, google-chrome) this will also activate the javascript 
debugger and as likely as not will make the bug go away (ouch!)

On the mhttpd side, please capture the stack trace from the crash: enable 
core dumps (ODB "/experiment/enable core dumps" set to "y", after the crash, 
run "ls -l core.*; gdb mhttpd core.9999") or run mhttpd inside gdb or attach 
gdb to a running mhttpd (gdb -p 9999). Once in gdb, run "info thr" to list all 
threads, "thr 0; bt", "thr 1; bt", etc to get stack traces from all threads, only 
one of them contains the crash (tedious!).

Email me the stack trace (or post here), in case we want to look at values
of any variables from the crash, keep the core dump and do not rebuild 
mhttpd.

K.O.

> On the mhttpd side, please capture the stack trace from the crash: enable 
> core dumps (ODB "/experiment/enable core dumps" set to "y", after the crash, 
> run "ls -l core.*; gdb mhttpd core.9999") or run mhttpd inside gdb or attach 
> gdb to a running mhttpd (gdb -p 9999). Once in gdb, run "info thr" to list all 
> threads, "thr 0; bt", "thr 1; bt", etc to get stack traces from all threads, only 
> one of them contains the crash (tedious!).
> 
> Email me the stack trace (or post here), in case we want to look at values
> of any variables from the crash, keep the core dump and do not rebuild 
> mhttpd.
> 
> K.O.

here comes the stack trace (only happens when using safari 12.1.2 macOS 10.14.6):

(gdb) thr 1
[Switching to thread 1 (Thread 0x7f57ceffd700 (LWP 3538))]
#0  0x00007f57f29fe377 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007f57f29fe377 in raise () from /lib64/libc.so.6
#1  0x00007f57f29ffa68 in abort () from /lib64/libc.so.6
#2  0x00007f57f330e7d5 in __gnu_cxx::__verbose_terminate_handler() () from
/lib64/libstdc++.so.6
#3  0x00007f57f330c746 in ?? () from /lib64/libstdc++.so.6
#4  0x00007f57f330c773 in std::terminate() () from /lib64/libstdc++.so.6
#5  0x00007f57f330c993 in __cxa_throw () from /lib64/libstdc++.so.6
#6  0x00007f57f330cf2d in operator new(unsigned long) () from /lib64/libstdc++.so.6
#7  0x00007f57f336ba19 in std::string::_Rep::_S_create(unsigned long, unsigned long,
std::allocator<char> const&)
    () from /lib64/libstdc++.so.6
#8  0x00007f57f336c62b in std::string::_Rep::_M_clone(std::allocator<char> const&,
unsigned long) ()
   from /lib64/libstdc++.so.6
#9  0x00007f57f336ccfc in std::basic_string<char, std::char_traits<char>,
std::allocator<char> >::basic_string(std::string const&) () from /lib64/libstdc++.so.6
#10 0x000000000041ce0f in check_digest_auth (hm=hm@entry=0x7f57ceffc520, auth=0x74b060
<auth_mg>)
    at /home/nemu/nemu/tmidas/midas/progs/mhttpd.cxx:17143
#11 0x0000000000452a61 in handle_http_message (msg=0x7f57ceffc520, nc=0x2019ca0,
this=<optimized out>, 
    this=<optimized out>, this=<optimized out>) at
/home/nemu/nemu/tmidas/midas/progs/mhttpd.cxx:17703
#12 handle_http_event_mg (nc=nc@entry=0x2019ca0, ev=ev@entry=100,
ev_data=ev_data@entry=0x7f57ceffc520)
    at /home/nemu/nemu/tmidas/midas/progs/mhttpd.cxx:17753
#13 0x0000000000464c4b in mg_call (nc=nc@entry=0x2019ca0, 
    ev_handler=0x4521f0 <handle_http_event_mg(mg_connection*, int, void*)>, ev=100, 
    ev_data=ev_data@entry=0x7f57ceffc520) at
/home/nemu/nemu/tmidas/midas/progs/mongoose6.cxx:2120
#14 0x000000000046790e in mg_http_call_endpoint_handler (nc=nc@entry=0x2019ca0,
ev=<optimized out>, 
    hm=hm@entry=0x7f57ceffc520) at /home/nemu/nemu/tmidas/midas/progs/mongoose6.cxx:4946
#15 0x0000000000467e3f in mg_http_handler (nc=nc@entry=0x2019ca0, ev=ev@entry=3, 
    ev_data=ev_data@entry=0x7f57ceffcb2c) at
/home/nemu/nemu/tmidas/midas/progs/mongoose6.cxx:5139
#16 0x0000000000464c4b in mg_call (nc=nc@entry=0x2019ca0, 
    ev_handler=0x467a20 <mg_http_handler(mg_connection*, int, void*)>,
ev_handler@entry=0x0, ev=ev@entry=3, 
    ev_data=ev_data@entry=0x7f57ceffcb2c) at
/home/nemu/nemu/tmidas/midas/progs/mongoose6.cxx:2120
#17 0x0000000000464fb7 in mg_recv_common (nc=nc@entry=0x2019ca0,
buf=buf@entry=0x7f57c0000cd0, len=len@entry=279)
    at /home/nemu/nemu/tmidas/midas/progs/mongoose6.cxx:2676
#18 0x00000000004659c8 in mg_if_recv_tcp_cb (len=279, buf=0x7f57c0000cd0, nc=0x2019ca0)
    at /home/nemu/nemu/tmidas/midas/progs/mongoose6.cxx:2680
#19 mg_read_from_socket (conn=0x2019ca0) at
/home/nemu/nemu/tmidas/midas/progs/mongoose6.cxx:3378
#20 mg_mgr_handle_conn (nc=0x2019ca0, fd_flags=1, now=now@entry=1568705761.3290441)
    at /home/nemu/nemu/tmidas/midas/progs/mongoose6.cxx:3511
#21 0x0000000000465ee0 in mg_mgr_poll (mgr=mgr@entry=0x7f57ceffcda0,
timeout_ms=timeout_ms@entry=1000)
    at /home/nemu/nemu/tmidas/midas/progs/mongoose6.cxx:3687
#22 0x0000000000466085 in per_connection_thread_function (param=0x2019ca0)
    at /home/nemu/nemu/tmidas/midas/progs/mongoose6.cxx:3805
#23 0x00007f57f39c7ea5 in start_thread () from /lib64/libpthread.so.0
#24 0x00007f57f2ac68cd in clone () from /lib64/libc.so.6

> > On the mhttpd side, please capture the stack trace from the crash
> 
> here comes the stack trace (only happens when using safari 12.1.2 macOS 10.14.6):
> 
> #10 0x000000000041ce0f in check_digest_auth ...
>

The crash is in check_digest_auth() which checks the mongoose web server password (if not using 
password protection from the https proxy i.e. apache httpd).

If so you should see this crash on all pages, not just when you access history pages, yes?

Ok, I just checked, my safari is "Version 12.1.2 (13607.3.10)" and I see no immediate crash, even on 
history pages.

But I am macos 10.13.6, maybe that makes a difference.

If you see the safari crash on all pages, then it is not history-specific.

In this case, I would like you to file a bug report on bitbucket "mhttpd crash with safari" and we follow up 
on it there.

K.O.

> During my visit at TRIUMF we rewrote the history plotting functionality of midas.

This is a most amazing achievement. We wanted to do this "for years" and I think we have
benefitted greatly from the delay - tools available for building interactive web graphics
have improved so much so recently.

For example, delivering binary data from mhttpd to javascript (avoiding json encoding and decoding
saves tons of CPU cycles) went from "how do I do this?!?" to "I did it in only 3 hours!".

> We are now in a state where this is still work in progress, but already at this stage it might
> be useful for others to report any feedback.

The old gif-based history plots took a lot of effort and a long time to get where they work well
for most experiments and where we are happy with them.

From the TRIUMF side of things, lots of polishing of the graphics and of the user interface came
through use at our bigger experiments - TWIST (TRIUMF), ALPHA (CERN), T2K/ND280 (Japan).

So, much improvement and polishing of the new graphics is still ahead for us.

> Simply upgrade the the newest develop branch of midas, and you will see two menu items 
> "OldHistory" which is the old system and "History" which is the new system.

I hope to start the new release branch for midas-2019-09 soon. For the release, we will try
to have both the old and the new history graphics to integrate smoothly. The old graphics
still has to work well, as some users may prefer the old graphics and the old user interface.

Also the new system is still incomplete, i.e. there is no trivial way to save a history plot into a file:

> Following items are planned, but not yet implemented:
> - Printing of run markers as in the old history
> - Export / Printing / Sending to ELOG any history plot

K.O.

>  Also the new system is still incomplete, i.e. there is no trivial way to save a history plot into a file:

That has been implemented in meantime. Just click on the download arrow and you can save the current window in CSV or PNG format.

Stefan

I'm not sure if these issues are related or not, but I'm getting an error
message when I want to access the root of the ODB via the webserver:
[mhttpd,ERROR] [mhttpd.cxx:563:rread,ERROR] Cannot read file '/root', read of
4096 returned -1, errno 21 (Is a directory)

I also tried turning the re-direct from http to https off, but this does not
seem to work. I also noticed that the redirect changes the localhost into a
hostname. Where does mongoose take this hostname from?

EDIT: Seems that the change of the hostname is due to a setting in /etc/hosts,
i.e. all my fault ...

EDIT: I think there was some issue with the mhttpd. When I checked the output (I
used screen to run it), it was full of these messages:

ss_semaphore_wait_for: semop/semtimedop(2588679) returned -1, errno 43
(Identifier removed)
al_check: Something is wrong with our semaphore, ss_semaphore_wait_for()
returned 408, aborting.
al_check: Cannot abort - this will lock you out of odb. From this point, MIDAS
will not work correctly. Please read the discussion at
https://midas.triumf.ca/elog/Midas/945

Restarted it and it stopped redirecting. So accessing the root of the ODB via
the webserver is the only issue now.

> I'm not sure if these issues are related or not, but I'm getting an error
> message when I want to access the root of the ODB via the webserver:
> [mhttpd,ERROR] [mhttpd.cxx:563:rread,ERROR] Cannot read file '/root', read of
> 4096 returned -1, errno 21 (Is a directory)

This is an old bug. It was part of the "custom path" confusion. Fixed (I think) in all midas-2019 
releases.

To confirm, which version are you using (run "odbedit ver" or look on the mhttpd "help" page)?

If you have an older version, I recommend that you update to midas-2019-03 (cd midas; git pull; 
git checkout midas-2019-03; make clean; make).

If you feel adventurous, you can also update to the head of the development version
and see all the new features (cmake, c++11, new history pages).

If you do not feel adventurous, wait until we have midas-2019-09 ready, use midas-2019-03 
until then.

K.O.

A new tag has been implemented to display history panels in custom pages, integrated in the 
new custom page design from 2017. The full documentation can be found at 

https://midas.triumf.ca/MidasWiki/index.php/New_Custom_Pages_(2017)#mhistory

Attached is a simple example of such a panel and the result. You have to rename triggerrate.txt to triggerrate.html if you want to use it (can't do it here since otherwise the browser wants to render it which does not work outside of midas).

Stefan

> A new tag has been implemented to display history panels in custom pages, integrated in the 
> new custom page design from 2017. The full documentation can be found at 
> 

As part of consolidating/cleaning the MIDAS Wiki documentation, the "New Custom Pages" was folded into the main "Custom Page".  So to see a
description of Stefan's new functionality please go to 

https://midas.triumf.ca/MidasWiki/index.php/Custom_Page#mhistory

<html>
<head>
   <title>Test</title>
   <link rel="stylesheet" href="midas.css">
   <script src="controls.js"></script>
   <script src="midas.js"></script>
   <script src="mhttpd.js"></script>
</head>
<body class="mcss" onload="mhttpd_init('Test');">

<div id="mheader"></div>
<div id="msidenav"></div>

<div id="mmain">
  <div name="mhistory" data-group="Test" data-panel="Test" data-scale="1m" style="width:600px;border:1px solid black;"></div>
</div>
</body>
</html>

Hi,

I posted on the ROOTANA elog but there seems to be little activity there...

Could someone confirm if this is a bug?
https://midas.triumf.ca/elog/Rootana/14

Another user replied that they are encountering the same issue, so I think it is unlikely it is just our installation.

While ROOTANA is unusable for us, I tried to use the example Frontend and Analyzer (under the Experiment source folder). The analyzer does not seem to do much though. A root file is produced but nothing is placed into it. Is that normal?

Any help would be welcome.

Hi Paolo,

Sorry for the slow response.  We were discussing this with Konstantin yesterday.  He is aware of the problem now and will be working on a solution soon.

In the short term I found that it works if you just comment out the offending line:

indnerlt:rootana lindner$ git diff libMidasInterface/TMidasOnline.cxx
diff --git a/libMidasInterface/TMidasOnline.cxx b/libMidasInterface/TMidasOnline.cxx
index 92eb3e9..67da613 100644
--- a/libMidasInterface/TMidasOnline.cxx
+++ b/libMidasInterface/TMidasOnline.cxx
@@ -191,7 +191,7 @@ bool TMidasOnline::sleep(int mdelay)
   #ifdef CH_IPC
   ss_suspend_set_dispatch(CH_IPC, 0, NULL);
   #else
-  ss_suspend_set_dispatch_ipc(NULL);
+  //  ss_suspend_set_dispatch_ipc(NULL);
   #endif
  int status = ss_suspend(mdelay, 0);
   if (status == SS_SUCCESS)

This compiles and at least runs for me; so maybe that is helpful for you.  But Konstantin will provide a longer term solution.



> Hi,
> 
> I posted on the ROOTANA elog but there seems to be little activity there...
> 
> Could someone confirm if this is a bug?
> https://midas.triumf.ca/elog/Rootana/14
> 
> Another user replied that they are encountering the same issue, so I think it is unlikely it is just our installation.
> 
> While ROOTANA is unusable for us, I tried to use the example Frontend and Analyzer (under the Experiment source folder). The analyzer does not seem to do much though. A root file is produced but nothing is placed into it. Is that normal?
> 
> Any help would be welcome.

Hi,

Thank you, commenting out the line worked and we can now compile the code. However, when we try to run ana.exe or anaDisplay.exe, we get the following errors:

Error in <TCling::RegisterModule>: cannot find dictionary module TMainDisplayWindowDict_rdict.pcm
Error in <TCling::RegisterModule>: cannot find dictionary module TRootanaDisplayDict_rdict.pcm
Error in <TCling::RegisterModule>: cannot find dictionary module TFancyHistogramCanvasDict_rdict.pcm
 

We see that the files are in /rootana/obj but we cannot find a way to point the compiler to them.

Could you please advise how to proceed,

Many thanks

> Hi Paolo,
> 
> Sorry for the slow response.  We were discussing this with Konstantin yesterday.  He is aware of the problem now and will be working on a solution soon.
> 
> In the short term I found that it works if you just comment out the offending line:
> 
> indnerlt:rootana lindner$ git diff libMidasInterface/TMidasOnline.cxx
> diff --git a/libMidasInterface/TMidasOnline.cxx b/libMidasInterface/TMidasOnline.cxx
> index 92eb3e9..67da613 100644
> --- a/libMidasInterface/TMidasOnline.cxx
> +++ b/libMidasInterface/TMidasOnline.cxx
> @@ -191,7 +191,7 @@ bool TMidasOnline::sleep(int mdelay)
>    #ifdef CH_IPC
>    ss_suspend_set_dispatch(CH_IPC, 0, NULL);
>    #else
> -  ss_suspend_set_dispatch_ipc(NULL);
> +  //  ss_suspend_set_dispatch_ipc(NULL);
>    #endif
>   int status = ss_suspend(mdelay, 0);
>    if (status == SS_SUCCESS)
> 
> This compiles and at least runs for me; so maybe that is helpful for you.  But Konstantin will provide a longer term solution.
> 
> 
> 
> > Hi,
> > 
> > I posted on the ROOTANA elog but there seems to be little activity there...
> > 
> > Could someone confirm if this is a bug?
> > https://midas.triumf.ca/elog/Rootana/14
> > 
> > Another user replied that they are encountering the same issue, so I think it is unlikely it is just our installation.
> > 
> > While ROOTANA is unusable for us, I tried to use the example Frontend and Analyzer (under the Experiment source folder). The analyzer does not seem to do much though. A root file is produced but nothing is placed into it. Is that normal?
> > 
> > Any help would be welcome.

> indnerlt:rootana lindner$ git diff libMidasInterface/TMidasOnline.cxx
> diff --git a/libMidasInterface/TMidasOnline.cxx b/libMidasInterface/TMidasOnline.cxx
> index 92eb3e9..67da613 100644
> --- a/libMidasInterface/TMidasOnline.cxx
> +++ b/libMidasInterface/TMidasOnline.cxx
> @@ -191,7 +191,7 @@ bool TMidasOnline::sleep(int mdelay)
>    #ifdef CH_IPC
>    ss_suspend_set_dispatch(CH_IPC, 0, NULL);
>    #else
> -  ss_suspend_set_dispatch_ipc(NULL);
> +  //  ss_suspend_set_dispatch_ipc(NULL);
>    #endif
>   int status = ss_suspend(mdelay, 0);
>    if (status == SS_SUCCESS)
> 
> This compiles and at least runs for me; so maybe that is helpful for you.  But Konstantin will provide a longer term solution.


This is a problem with the latest development version of MIDAS. ss_suspend overrides have been removed from system.cxx
and there is no way for rootana to avoid the problem of recursive call to the event handler.

I recommend that instead of using the latest development version of MIDAS you use one of the recent released versions (use "git tag -l", midas-2019-06-b is the latest release).

All the released versions of midas have the ss_suspend overrides implemented and rootana will work correctly. For the next release of midas
I will restore the ss_suspend override and update the rootana code.


K.O.

> -  ss_suspend_set_dispatch_ipc(NULL);
> +  //  ss_suspend_set_dispatch_ipc(NULL);
> 
> This compiles and at least runs for me; so maybe that is helpful for you.  But Konstantin will provide a longer term solution.

I now understand why this fix worked. Around December 2018 timeframe, I reworked the MIDAS event buffer code
and one improvement was to only send UDP buffer notifications if somebody is waiting for them. This probably
reduced to zero the probability of recursive calls to the user event handler - the problem originally fixed by the monkey
work against the midas ipc handler.

After looking at it, I now understand that the correct solution is to call ss_suspend(MSG_BM), but it turns out
inside MIDAS, handling of MSG_BM was incomplete and recursive calls to the user event handler were still
possible. (but most likely not actually happening anymore because of those changes to the event buffer code).

So.

a) ss_suspend(MSG_BM) inside midas now works correctly, recursive call to the user event handler will not happen.
b) TMidasOnline::sleep() now calls ss_suspend(MSG_BM), monkey business with ss_suspend_set_dispatch_ipc() is removed.

The problem of recursive call to the analyzer event handler is now fixed, both rootana and manalyzer (both use the same TMidasOnline code).

Read more about this here:
https://midas.triumf.ca/elog/Midas/1663

K.O.

The ROOTANA midas analyzer uncovered a problem with recursive use of ss_suspend().

When running in graphical mode, the ROOT graphics main event loop was calling 
ss_suspend(0) (no MSG_BM) which would sometimes call the user event handler (if a new 
event arrives into the midas event buffer). Because this loop was already running in the 
user event handler, there was a crash.

This is the calling sequence leading to the incorrect recursion: (from system.cxx comments 
to ss_suspend())

analyzer ->
     -> cm_yield() in the main loop
     -> ss_suspend(0)
     -> MSG_BM message arrives in the UDP socket
     -> ss_suspend_process_ipc(0)
     -> cm_dispatch_ipc()
     -> bm_push_event()
     -> bm_push_buffer()
     -> bm_read_buffer()
     -> bm_dispatch_event()
     -> user event handler
     -> user event handler ROOT graphics main loop needs to sleep
     -> ss_suspend(0) <--- should be ss_suspend(MSG_BM)!!!     
     -> MSG_BM message arrives in the UDP socket
     -> ss_suspend_process_ipc(0) <- should be ss_suspend_process_ipc(MSG_BM)!!!
     -> cm_dispatch_ipc() <- without MSG_BM, calling cm_dispatch_ipc() again
     -> bm_push_event()
     -> bm_push_buffer()
     -> bm_read_buffer()
     -> bm_dispatch_event()
     -> user event handler <---- called recursively, very bad!

The proper fix for this is to always call ss_suspend(MSG_BM) from the user event handler 
and ss_suspend(MSG_ODB) from the user db_watch handler.

In this second case, ss_suspend(MSG_OBM) will lose/ignore/discard db_watch notifications, 
if you do not want that, call ss_suspend(0) and be ready for recursive calls to your 
db_watch handler. (this can happen in a frontend program that acts on changes in ODB and 
where some of these actions may require sleeping via ss_suspend()).

ss_suspend(MSG_BM) will discard MSG_BM messages, which is not a problem as 
bm_wait_for_events() and cm_yield() will immediately poll the event buffer and there will be 
no delay in receiving new events.

Until commit 99d6e90 there were problems in ss_suspend(MSG_BM) and recursive calls to 
the user event handler were still possible. It is now fixed in this and the previous commits.

K.O.

Today I fixed a long-annoying problem. We have in each front-end an equipment structure 
which defined the event id, event type, readout frequency etc. This is mapped to the ODB 
subtree

/Equipment/<name>/Common

In the past, the ODB setting took precedence over the frontend structure. We defined this 
like 25 years ago and I forgot what the exact reason was. It causes however many people 
(including myself) to fall into this trap: You change something in the front-end EQUIPMENT 
structure, you restart the front-end, but the new setting does not take effect since the 
(old) ODB value took precedence. After some debugging you find out that you have to both 
change the EQUIPMENT structure (which defines the default value for a fresh ODB) and the 
ODB value itself.

So I changed it in the current develop tree that the front-end structure takes precedence. 
You still have a hot-link, so if you want to change anything while the front-end is running 
(like the readout period), you can do that in the ODB and it takes effect immediately. But 
when you start the front-end the next time, the value from the EQUIPMENT structure is 
taken again. So please be aware of this new feature.

Happy BC day,
Stefan

Hi Stefan,

This change does not sound like a good idea to me.  I think that this change will cause just as much confusion as before; probably more since you are changing established behaviour.

It is common that MIDAS frontends usually have a Settings directory in the ODB where details about the frontend behaviour are set.  The Settings directory might get initialized from strings in the frontend code, but after initialization the Settings in the ODB have precedence and define how the frontend will behave.  Indeed, most of my custom webpages are designed to control my frontend programs through their Settings ODB tree.

So you have created a situation where Frontend/Settings in the ODB has precedence and is the main place for changing frontend behaviour; but Frontend/Common in the ODB is essentially meaningless and will get overwritten the next time the frontend restarts.  That seems likely to confuse people. 

If you really want to make this change I suggest that you delete the Frontend/Common directory entirely; or make it read-only so that people aren't fooled into changing it.

Thomas 



> Today I fixed a long-annoying problem. We have in each front-end an equipment structure 
> which defined the event id, event type, readout frequency etc. This is mapped to the ODB 
> subtree
> 
> /Equipment/<name>/Common
> 
> In the past, the ODB setting took precedence over the frontend structure. We defined this 
> like 25 years ago and I forgot what the exact reason was. It causes however many people 
> (including myself) to fall into this trap: You change something in the front-end EQUIPMENT 
> structure, you restart the front-end, but the new setting does not take effect since the 
> (old) ODB value took precedence. After some debugging you find out that you have to both 
> change the EQUIPMENT structure (which defines the default value for a fresh ODB) and the 
> ODB value itself.
> 
> So I changed it in the current develop tree that the front-end structure takes precedence. 
> You still have a hot-link, so if you want to change anything while the front-end is running 
> (like the readout period), you can do that in the ODB and it takes effect immediately. But 
> when you start the front-end the next time, the value from the EQUIPMENT structure is 
> taken again. So please be aware of this new feature.
> 
> Happy BC day,
> Stefan

Hi Thomas,

the change only affects Eqipment/<name>/common not the Equipment/<name>/Settings. 

The Common subtree is still hot-linked into the frontend, so when running things can be changed if needed. This mainly concerns the readout period of periodic events. 
Sometimes you want to change this quickly without restarting the frontend. Changing the other settings are kind of dangerous. If you change the ID of an event on the fly
you won't be able to analyze your data. So having this read-only in the ODB might be a good idea (you still need it in the ODB for the status page), except for the values
you want to change (like the readout period). 

Let's see what other people have to say.

Stefan

> Hi Stefan,
> 
> This change does not sound like a good idea to me.  I think that this change will cause just as much confusion as before; probably more since you are changing established behaviour.
> 
> It is common that MIDAS frontends usually have a Settings directory in the ODB where details about the frontend behaviour are set.  The Settings directory might get initialized from strings in the frontend code, but after initialization the Settings in the ODB have precedence and define how the frontend will behave.  Indeed, most of my custom webpages are designed to control my frontend programs through their Settings ODB tree.
> 
> So you have created a situation where Frontend/Settings in the ODB has precedence and is the main place for changing frontend behaviour; but Frontend/Common in the ODB is essentially meaningless and will get overwritten the next time the frontend restarts.  That seems likely to confuse people. 
> 
> If you really want to make this change I suggest that you delete the Frontend/Common directory entirely; or make it read-only so that people aren't fooled into changing it.
> 
> Thomas 
> 
> 
> 
> > Today I fixed a long-annoying problem. We have in each front-end an equipment structure 
> > which defined the event id, event type, readout frequency etc. This is mapped to the ODB 
> > subtree
> > 
> > /Equipment/<name>/Common
> > 
> > In the past, the ODB setting took precedence over the frontend structure. We defined this 
> > like 25 years ago and I forgot what the exact reason was. It causes however many people 
> > (including myself) to fall into this trap: You change something in the front-end EQUIPMENT 
> > structure, you restart the front-end, but the new setting does not take effect since the 
> > (old) ODB value took precedence. After some debugging you find out that you have to both 
> > change the EQUIPMENT structure (which defines the default value for a fresh ODB) and the 
> > ODB value itself.
> > 
> > So I changed it in the current develop tree that the front-end structure takes precedence. 
> > You still have a hot-link, so if you want to change anything while the front-end is running 
> > (like the readout period), you can do that in the ODB and it takes effect immediately. But 
> > when you start the front-end the next time, the value from the EQUIPMENT structure is 
> > taken again. So please be aware of this new feature.
> > 
> > Happy BC day,
> > Stefan

After some internal discussion, I decided to undo my previous change again, in order not to break existing habits. Instead, I created a new function

set_odb_equipment_common(equipment, name);

which should be called from frontend_init() which explicitly copies all data from the equipment structure in the front-end into the ODB.

Stefan

> Today I fixed a long-annoying problem. ...
> /Equipment/<name>/Common
> In the past, the ODB setting took precedence over the frontend structure...
> We defined this like 25 years ago and I forgot what the exact reason was.
> It causes however many people (including myself) to fall into this trap: ...

There is good number of confusions regarding entries in /eq/xxx/common:

- for some of them, the frontend code settings take precedence and overwrite settings in odb ("frontend file name")
- for some of them, ODB takes precedence and frontend code values are ignored ("read on" and "period")
- for some of them, changes in ODB take effect immediately (via db_watch) ("period")
- for some of them, frontend restart is required for changes to take effect (output event buffer name "buffer")
- some of them continuously update the odb values ("status", "status color")

I do not think there is a simple way to improve on this.

(One solution would replace the single "common" with several subdirectories, "per function",
one would have items where the code takes precedence, one would have items where odb takes
precedence (in effect, "standard settings"), one will have items that the frontend always updates
and that should not be changes via odb ("frontend name", etc). I am not sure this one solution
is necessarily an "improvement").

Lacking any ideas for improvements, I vote for the status quo. (plus a review of the documentation to ensure we have clearly
written up what each entry in "common" does and whether the user is permitted to edit it in odb).

K.O.

> Lacking any ideas for improvements, I vote for the status quo. (plus a review of the documentation to ensure we have clearly
> written up what each entry in "common" does and whether the user is permitted to edit it in odb).

I agree with that.

Stefan

Hi,

We've been trying to run Fetest in the attempt of plotting the sine wave data on
the history page on the web server. However each time we've tried running a new
plot we have come across the error of 'no data' from the variables. In the
status page we are clearly obtaining data from the frontend and it is updating
the variable as expected in SLOW.

When setting up MIDAS we managed to produce a history plot from Fetest but are
unable to do so any longer. We did have a go at modifying the Fetest code but
created a backup before doing so and are now running the original backup.

What could be causing the Fetest data not to be showing in the history plot?

> Hi,
> 
> We've been trying to run Fetest in the attempt of plotting the sine wave data on
> the history page on the web server. However each time we've tried running a new
> plot we have come across the error of 'no data' from the variables. In the
> status page we are clearly obtaining data from the frontend and it is updating
> the variable as expected in SLOW.
> 
> When setting up MIDAS we managed to produce a history plot from Fetest but are
> unable to do so any longer. We did have a go at modifying the Fetest code but
> created a backup before doing so and are now running the original backup.
> 
> What could be causing the Fetest data not to be showing in the history plot?

Is the logger running? (this application is handling the history data).
If yes: Did you change the variable names? If yes: restart the logger.

Hi, our logger was running. I have tried restarting mlogger (even though we haven't
changed variable names). We ran the following commands one after another and still no
luck with history plot. Is there anything else that could be causing these problems?

Kind regards,
Hassan 

==================================================================================

[lm17773@it038146 ~]$ cd /opt/midas_software/midas/bin/
[lm17773@it038146 bin]$ mhttpd
[mhttpd,ERROR] [odb.cxx:1646:db_open_database,ERROR] Removed ODB client 'mhttpd',
index 0 because process pid 20094 does not exists
[mhttpd,ERROR] [odb.cxx:1646:db_open_database,ERROR] Removed ODB client 'Logger',
index 1 because process pid 20214 does not exists
[mhttpd,INFO] Removed open record flag from "/Experiment/Security/RPC hosts/Allowed hosts"
[mhttpd,INFO] Removed exclusive access mode from "/Experiment/Security/RPC
hosts/Allowed hosts"
[mhttpd,INFO] Removed open record flag from "/Experiment/Security/mhttpd hosts/Allowed
hosts"
[mhttpd,INFO] Removed exclusive access mode from "/Experiment/Security/mhttpd
hosts/Allowed hosts"
[mhttpd,INFO] Removed open record flag from "/Logger/History"
[mhttpd,INFO] Removed exclusive access mode from "/Logger/History"
[mhttpd,INFO] Removed open record flag from "/Sequencer/State"
[mhttpd,INFO] Removed exclusive access mode from "/Sequencer/State"
[mhttpd,INFO] Removed open record flag from "/History/LoggerHistoryChannel"
[mhttpd,INFO] Removed exclusive access mode from "/History/LoggerHistoryChannel"
[mhttpd,INFO] Removed open record flag from "/Equipment/slow/Variables"
[mhttpd,INFO] Removed exclusive access mode from "/Equipment/slow/Variables"
[mhttpd,INFO] Removed open record flag from "/Equipment/Trigger/Statistics/Events per
sec."
[mhttpd,INFO] Removed exclusive access mode from "/Equipment/Trigger/Statistics/Events
per sec."
[mhttpd,INFO] Removed open record flag from "/Equipment/Trigger/Statistics/kBytes per
sec."
[mhttpd,INFO] Removed exclusive access mode from "/Equipment/Trigger/Statistics/kBytes
per sec."
[mhttpd,INFO] Removed open record flag from "/Equipment/Periodic/Variables"
[mhttpd,INFO] Removed exclusive access mode from "/Equipment/Periodic/Variables"
[mhttpd,INFO] Removed open record flag from "/Equipment/Scaler/Variables"
[mhttpd,INFO] Removed exclusive access mode from "/Equipment/Scaler/Variables"
[mhttpd,INFO] Corrected 10 ODB entries
[mhttpd,INFO] Deleted entry '/System/Clients/20094' for client 'mhttpd' because it is
not connected to ODB
Mongoose web server will use SSL certificate file "/home/lm17773/online/ssl_cert.pem"
Mongoose web server will use authentication realm "sampleexpt", password file
"/home/lm17773/online/htpasswd.txt"
mongoose web server is redirecting HTTP port 8080 to
https://it038146.users.bris.ac.uk:8443
mongoose web server is listening on the HTTP port 8080
mongoose web server is listening on the HTTPS port 8443
====================================================================================

[lm17773@it038146 bin]$ mlogger
[Logger,INFO] Deleted entry '/System/Clients/20214' for client 'Logger' because it is
not connected to ODB
Log     directory is /home/lm17773/online/
Data    directory is same as Log unless specified in /Logger/channels/
History directory is same as Log unless specified in /Logger/history/
ELog    directory is same as Log
SQL     database is localhost/sampleexpt/Runlog
MIDAS logger started. Stop with "!"
====================================================================================
[lm17773@it038146 bin]$ fetest
Frontend name          :     fetest
Event buffer size      :     10485760
User max event size    :     4194304
User max frag. size    :     4194304
# of events per buffer :     2

Connect to experiment sampleexpt...
OK
Init hardware...frontend_init!
Event size set to 10240 bytes
Ring buffer wait sleep 1 ms
OK
time 1564131394, data 97.814758
time 1564131395, data 96.592583
time 1564131396, data 95.105652
time 1564131397, data 93.358040
time 1564131398, data 91.354546
time 1564131399, data 89.100655
time 1564131400, data 86.602539
time 1564131401, data 83.867058
time 1564131402, data 80.901703
time 1564131403, data 77.714592
Warning: bank RND4 has zero size
time 1564131404, data 74.314484
time 1564131405, data 70.710678
time 1564131406, data 66.913063
time 1564131407, data 62.932041
====================================================================================






> > Hi,
> > 
> > We've been trying to run Fetest in the attempt of plotting the sine wave data on
> > the history page on the web server. However each time we've tried running a new
> > plot we have come across the error of 'no data' from the variables. In the
> > status page we are clearly obtaining data from the frontend and it is updating
> > the variable as expected in SLOW.
> > 
> > When setting up MIDAS we managed to produce a history plot from Fetest but are
> > unable to do so any longer. We did have a go at modifying the Fetest code but
> > created a backup before doing so and are now running the original backup.
> > 
> > What could be causing the Fetest data not to be showing in the history plot?
> 
> Is the logger running? (this application is handling the history data).
> If yes: Did you change the variable names? If yes: restart the logger.

> Hi, our logger was running.


One more thing is to check that the history files are actually being written to. Be default
the history files are written into the same directory where you have ODB, etc and
the file names are "*.hst".

Second thing, you can run "mlogger -v" and it will report that it is writing history events
into the history.

If all these things are happening,

Third, you can run "mhist" to see the data directly from the data files.

If all of these work, but you still get nothing in mhttpd, that would be very strange. If I remember
right, to see the sine wave, the history variable to plot is equipment "slow", variable "slow".


K.O.



> I have tried restarting mlogger (even though we haven't
> changed variable names). We ran the following commands one after another and still no
> luck with history plot. Is there anything else that could be causing these problems?
> 
> Kind regards,
> Hassan 
> 
> ==================================================================================
> 
> [lm17773@it038146 ~]$ cd /opt/midas_software/midas/bin/
> [lm17773@it038146 bin]$ mhttpd
> [mhttpd,ERROR] [odb.cxx:1646:db_open_database,ERROR] Removed ODB client 'mhttpd',
> index 0 because process pid 20094 does not exists
> [mhttpd,ERROR] [odb.cxx:1646:db_open_database,ERROR] Removed ODB client 'Logger',
> index 1 because process pid 20214 does not exists
> [mhttpd,INFO] Removed open record flag from "/Experiment/Security/RPC hosts/Allowed hosts"
> [mhttpd,INFO] Removed exclusive access mode from "/Experiment/Security/RPC
> hosts/Allowed hosts"
> [mhttpd,INFO] Removed open record flag from "/Experiment/Security/mhttpd hosts/Allowed
> hosts"
> [mhttpd,INFO] Removed exclusive access mode from "/Experiment/Security/mhttpd
> hosts/Allowed hosts"
> [mhttpd,INFO] Removed open record flag from "/Logger/History"
> [mhttpd,INFO] Removed exclusive access mode from "/Logger/History"
> [mhttpd,INFO] Removed open record flag from "/Sequencer/State"
> [mhttpd,INFO] Removed exclusive access mode from "/Sequencer/State"
> [mhttpd,INFO] Removed open record flag from "/History/LoggerHistoryChannel"
> [mhttpd,INFO] Removed exclusive access mode from "/History/LoggerHistoryChannel"
> [mhttpd,INFO] Removed open record flag from "/Equipment/slow/Variables"
> [mhttpd,INFO] Removed exclusive access mode from "/Equipment/slow/Variables"
> [mhttpd,INFO] Removed open record flag from "/Equipment/Trigger/Statistics/Events per
> sec."
> [mhttpd,INFO] Removed exclusive access mode from "/Equipment/Trigger/Statistics/Events
> per sec."
> [mhttpd,INFO] Removed open record flag from "/Equipment/Trigger/Statistics/kBytes per
> sec."
> [mhttpd,INFO] Removed exclusive access mode from "/Equipment/Trigger/Statistics/kBytes
> per sec."
> [mhttpd,INFO] Removed open record flag from "/Equipment/Periodic/Variables"
> [mhttpd,INFO] Removed exclusive access mode from "/Equipment/Periodic/Variables"
> [mhttpd,INFO] Removed open record flag from "/Equipment/Scaler/Variables"
> [mhttpd,INFO] Removed exclusive access mode from "/Equipment/Scaler/Variables"
> [mhttpd,INFO] Corrected 10 ODB entries
> [mhttpd,INFO] Deleted entry '/System/Clients/20094' for client 'mhttpd' because it is
> not connected to ODB
> Mongoose web server will use SSL certificate file "/home/lm17773/online/ssl_cert.pem"
> Mongoose web server will use authentication realm "sampleexpt", password file
> "/home/lm17773/online/htpasswd.txt"
> mongoose web server is redirecting HTTP port 8080 to
> https://it038146.users.bris.ac.uk:8443
> mongoose web server is listening on the HTTP port 8080
> mongoose web server is listening on the HTTPS port 8443
> 
====================================================================================
> 
> [lm17773@it038146 bin]$ mlogger
> [Logger,INFO] Deleted entry '/System/Clients/20214' for client 'Logger' because it is
> not connected to ODB
> Log     directory is /home/lm17773/online/
> Data    directory is same as Log unless specified in /Logger/channels/
> History directory is same as Log unless specified in /Logger/history/
> ELog    directory is same as Log
> SQL     database is localhost/sampleexpt/Runlog
> MIDAS logger started. Stop with "!"
> 
====================================================================================
> [lm17773@it038146 bin]$ fetest
> Frontend name          :     fetest
> Event buffer size      :     10485760
> User max event size    :     4194304
> User max frag. size    :     4194304
> # of events per buffer :     2
> 
> Connect to experiment sampleexpt...
> OK
> Init hardware...frontend_init!
> Event size set to 10240 bytes
> Ring buffer wait sleep 1 ms
> OK
> time 1564131394, data 97.814758
> time 1564131395, data 96.592583
> time 1564131396, data 95.105652
> time 1564131397, data 93.358040
> time 1564131398, data 91.354546
> time 1564131399, data 89.100655
> time 1564131400, data 86.602539
> time 1564131401, data 83.867058
> time 1564131402, data 80.901703
> time 1564131403, data 77.714592
> Warning: bank RND4 has zero size
> time 1564131404, data 74.314484
> time 1564131405, data 70.710678
> time 1564131406, data 66.913063
> time 1564131407, data 62.932041
> 
====================================================================================
> 
> 
> 
> 
> 
> 
> > > Hi,
> > > 
> > > We've been trying to run Fetest in the attempt of plotting the sine wave data on
> > > the history page on the web server. However each time we've tried running a new
> > > plot we have come across the error of 'no data' from the variables. In the
> > > status page we are clearly obtaining data from the frontend and it is updating
> > > the variable as expected in SLOW.
> > > 
> > > When setting up MIDAS we managed to produce a history plot from Fetest but are
> > > unable to do so any longer. We did have a go at modifying the Fetest code but
> > > created a backup before doing so and are now running the original backup.
> > > 
> > > What could be causing the Fetest data not to be showing in the history plot?
> > 
> > Is the logger running? (this application is handling the history data).
> > If yes: Did you change the variable names? If yes: restart the logger.

> Hi, our logger was running.

Please do these simple tests:

- run "mlogger -v", it should report that it is writing slow/slow data into the history with rate 1 Hz (fetest 
should be running at this point, yes?)
- normally the history files are written into the experiment directory (where ODB is, etc) and have file names 
"*.hst". Observe that the files are growing. Use "ls -ltr". (mlogger and fetest should be running at this point, 
yes?)
- if all if this is happening, you can try to run "mhist" to see the history data

If all of the above works, but you still get nothing from the history plots in mhttpd, then we probably have a 
bug in midas and we would like very much to fix it. For this we will need some more information from you. I 
hope you have some time available to help us with this.

Hmm... the fetest history plots are not defined automatically, you have to create the history plot manually,
maybe this is where the problem happens. One thing to check here, the correct variable to plot
is "slow/slow", if I remember right.


K.O.


> I have tried restarting mlogger (even though we haven't
> changed variable names). We ran the following commands one after another and still no
> luck with history plot. Is there anything else that could be causing these problems?
> 
> Kind regards,
> Hassan 
> 
> 
================================================================================
==
> 
> [lm17773@it038146 ~]$ cd /opt/midas_software/midas/bin/
> [lm17773@it038146 bin]$ mhttpd
> [mhttpd,ERROR] [odb.cxx:1646:db_open_database,ERROR] Removed ODB client 'mhttpd',
> index 0 because process pid 20094 does not exists
> [mhttpd,ERROR] [odb.cxx:1646:db_open_database,ERROR] Removed ODB client 'Logger',
> index 1 because process pid 20214 does not exists
> [mhttpd,INFO] Removed open record flag from "/Experiment/Security/RPC hosts/Allowed hosts"
> [mhttpd,INFO] Removed exclusive access mode from "/Experiment/Security/RPC
> hosts/Allowed hosts"
> [mhttpd,INFO] Removed open record flag from "/Experiment/Security/mhttpd hosts/Allowed
> hosts"
> [mhttpd,INFO] Removed exclusive access mode from "/Experiment/Security/mhttpd
> hosts/Allowed hosts"
> [mhttpd,INFO] Removed open record flag from "/Logger/History"
> [mhttpd,INFO] Removed exclusive access mode from "/Logger/History"
> [mhttpd,INFO] Removed open record flag from "/Sequencer/State"
> [mhttpd,INFO] Removed exclusive access mode from "/Sequencer/State"
> [mhttpd,INFO] Removed open record flag from "/History/LoggerHistoryChannel"
> [mhttpd,INFO] Removed exclusive access mode from "/History/LoggerHistoryChannel"
> [mhttpd,INFO] Removed open record flag from "/Equipment/slow/Variables"
> [mhttpd,INFO] Removed exclusive access mode from "/Equipment/slow/Variables"
> [mhttpd,INFO] Removed open record flag from "/Equipment/Trigger/Statistics/Events per
> sec."
> [mhttpd,INFO] Removed exclusive access mode from "/Equipment/Trigger/Statistics/Events
> per sec."
> [mhttpd,INFO] Removed open record flag from "/Equipment/Trigger/Statistics/kBytes per
> sec."
> [mhttpd,INFO] Removed exclusive access mode from "/Equipment/Trigger/Statistics/kBytes
> per sec."
> [mhttpd,INFO] Removed open record flag from "/Equipment/Periodic/Variables"
> [mhttpd,INFO] Removed exclusive access mode from "/Equipment/Periodic/Variables"
> [mhttpd,INFO] Removed open record flag from "/Equipment/Scaler/Variables"
> [mhttpd,INFO] Removed exclusive access mode from "/Equipment/Scaler/Variables"
> [mhttpd,INFO] Corrected 10 ODB entries
> [mhttpd,INFO] Deleted entry '/System/Clients/20094' for client 'mhttpd' because it is
> not connected to ODB
> Mongoose web server will use SSL certificate file "/home/lm17773/online/ssl_cert.pem"
> Mongoose web server will use authentication realm "sampleexpt", password file
> "/home/lm17773/online/htpasswd.txt"
> mongoose web server is redirecting HTTP port 8080 to
> https://it038146.users.bris.ac.uk:8443
> mongoose web server is listening on the HTTP port 8080
> mongoose web server is listening on the HTTPS port 8443
> 
================================================================================
====
> 
> [lm17773@it038146 bin]$ mlogger
> [Logger,INFO] Deleted entry '/System/Clients/20214' for client 'Logger' because it is
> not connected to ODB
> Log     directory is /home/lm17773/online/
> Data    directory is same as Log unless specified in /Logger/channels/
> History directory is same as Log unless specified in /Logger/history/
> ELog    directory is same as Log
> SQL     database is localhost/sampleexpt/Runlog
> MIDAS logger started. Stop with "!"
> 
================================================================================
====
> [lm17773@it038146 bin]$ fetest
> Frontend name          :     fetest
> Event buffer size      :     10485760
> User max event size    :     4194304
> User max frag. size    :     4194304
> # of events per buffer :     2
> 
> Connect to experiment sampleexpt...
> OK
> Init hardware...frontend_init!
> Event size set to 10240 bytes
> Ring buffer wait sleep 1 ms
> OK
> time 1564131394, data 97.814758
> time 1564131395, data 96.592583
> time 1564131396, data 95.105652
> time 1564131397, data 93.358040
> time 1564131398, data 91.354546
> time 1564131399, data 89.100655
> time 1564131400, data 86.602539
> time 1564131401, data 83.867058
> time 1564131402, data 80.901703
> time 1564131403, data 77.714592
> Warning: bank RND4 has zero size
> time 1564131404, data 74.314484
> time 1564131405, data 70.710678
> time 1564131406, data 66.913063
> time 1564131407, data 62.932041
> 
================================================================================
====
> 
> 
> 
> 
> 
> 
> > > Hi,
> > > 
> > > We've been trying to run Fetest in the attempt of plotting the sine wave data on
> > > the history page on the web server. However each time we've tried running a new
> > > plot we have come across the error of 'no data' from the variables. In the
> > > status page we are clearly obtaining data from the frontend and it is updating
> > > the variable as expected in SLOW.
> > > 
> > > When setting up MIDAS we managed to produce a history plot from Fetest but are
> > > unable to do so any longer. We did have a go at modifying the Fetest code but
> > > created a backup before doing so and are now running the original backup.
> > > 
> > > What could be causing the Fetest data not to be showing in the history plot?
> > 
> > Is the logger running? (this application is handling the history data).
> > If yes: Did you change the variable names? If yes: restart the logger.

I want to report a bug in the ROOT build process that might be relevant to the midas implementation. I had an annoying failure to build root 6.18 (current pro version) with a misleading error message about a fault in the root code. It turned out this was a cmake problem, and the error was from my cmake version being older than 3.14, which is quite recent. Took a bit of searching to find this.

I recommend when the cmake version is distributed that the instructions include the required cmake version. Developers are generally working well ahead of what is available in the older OS's.

Each CMakeLists.txt should specify which version of CMake it requires. The MIDAS CMakeLists.txt requires CMake 3.1 or later. 
We deliberately stayed away from fancy cutting edge CMake features in order to make midas easier to compile. On top of that,
midas is a much simpler package compared to root, so things are not so complicated.

Stefan

> Each CMakeLists.txt should specify which version of CMake it requires. The MIDAS CMakeLists.txt requires CMake 3.1 or later. 
> We deliberately stayed away from fancy cutting edge CMake features in order to make midas easier to compile. On top of that,
> midas is a much simpler package compared to root, so things are not so complicated.

The oldest cmake I actually used is 3.6.1 (on SL6), so I do not know if cmake versions between 3.1 and 3.6 actually work for us. Perhaps we should set 
the CMakefile requirement to 3.6.1 to match the oldest version we know works. If somebody has an older cmake, they have a choice of updating it or 
trying it as as and reporting success/failure to the midas forum here.

K.O.

I just tried CMake 3.1.0 and it worked with midas. So I believe all versions between 3.1.0 and 3.6.1 are ok.

Actually playing around with different versions I realized that 3.0.0 is also ok, so I changed the requirement of midas down to 3.0

Stefan

Hello!
I am creating this thread to comment on an issue raised today during the MIDAS 
workshop.

It was said that ROOT doesn't play well with multithreading ... and it is 
definitely true. But since last year, many improvements have been done in ROOT 
multi-threading support and now even the ROOT fitter can be made thread-safe.

I know this because recently I had to completely rewrite the calibration 
software for the WAGASCI experiment and I wanted to use many ROOT analyzers in 
parallel.

Getting ROOT to work in a multi-thread environment is not painless. It took me 
many weeks to get to the end of it. But if using the latest ROOT version (from 
about ROOT 6.12.00 onwards), it should be possible. Basically, you have to 
compile ROOT with the Minuit2 minimizer support and then select it.

This is the thread on the ROOT forum where I have asked about and solved my 
issues (I am LastStarDust): https://root-forum.cern.ch/t/root-crashes-in-multi-
threaded-environment/35407

You can also refer to this bug report: https://sft.its.cern.ch/jira/browse/ROOT-
7173
and this documentation page: https://root.cern.ch/how/how-express-parallelism-
many-cores

Hope it may help.
Giorgio

Dear MIDAS users,

We would like to announce a third MIDAS workshop at TRIUMF on Aug 7, 2019.
Stefan Ritt will again be visiting TRIUMF at this time.

The overall goal of the workshop is to present new features in MIDAS, to discuss
future changes and to hear experiences from different experiments.

We expect that some participants will connect remotely to the workshop; we will
setup a video-conferencing option.  The exact time of the workshop will be
decided later and will be optimized based on the geographic distribution of
remote attendees. So please let us know if you want to attend remotely.  We are
also happy for people to come in person to TRIUMF.  

A (very) preliminary agenda includes

- New default mhttpd pages and new APIs for custom pages
- Conversion of MIDAS to C++
- new C++ based frontend framework (tmfe and mvodb from ALPHA-g)
- MySQL/Postgres database for storing ODB configurations
- Plans for updating history plotting
- Using MIDAS with an online trigger farm
- new C++ multithreaded flow analyzer (manalyzer from ALPHA-g)

But please suggest other topics; we also hope to hear reports from particular
experiments.

Sincerely,
Thomas (on behalf of MIDAS developers)

Dear MIDAS users,

Here's further information on the third MIDAS workshop:

1) Workshop will take place from 1PM-5PM (Vancouver time) on Aug 7.

2) A mostly finalized agenda for the workshop is posted here:

https://indico.triumf.ca/conferenceDisplay.py?confId=2438

People are still welcome to email me if they want to present something.  We should be able to add it to the schedule.

3) For those who want to participate remotely, we will be using bluejeans.  The webpage for the bluejeans meeting is here:

https://bluejeans.com/462865444

4) For those at TRIUMF I will confirm the meeting room closer to the date.

Thomas


> Dear MIDAS users,
> 
> We would like to announce a third MIDAS workshop at TRIUMF on Aug 7, 2019.
> Stefan Ritt will again be visiting TRIUMF at this time.
> 
> The overall goal of the workshop is to present new features in MIDAS, to discuss
> future changes and to hear experiences from different experiments.
> 
> We expect that some participants will connect remotely to the workshop; we will
> setup a video-conferencing option.  The exact time of the workshop will be
> decided later and will be optimized based on the geographic distribution of
> remote attendees. So please let us know if you want to attend remotely.  We are
> also happy for people to come in person to TRIUMF.  
> 
> A (very) preliminary agenda includes
> 
> - New default mhttpd pages and new APIs for custom pages
> - Conversion of MIDAS to C++
> - new C++ based frontend framework (tmfe and mvodb from ALPHA-g)
> - MySQL/Postgres database for storing ODB configurations
> - Plans for updating history plotting
> - Using MIDAS with an online trigger farm
> - new C++ multithreaded flow analyzer (manalyzer from ALPHA-g)
> 
> But please suggest other topics; we also hope to hear reports from particular
> experiments.
> 
> Sincerely,
> Thomas (on behalf of MIDAS developers)

Dear MIDAS users,

A final reminder about the MIDAS workshop tomorrow.  A couple reminders/notes: 

1) Workshop will take place from 1PM-5:30PM (Vancouver time) on Aug 7.

2) A finalized agenda for the workshop is posted here:

https://indico.triumf.ca/conferenceDisplay.py?confId=2438

Speakers should email me their talks beforehand and I will post them.

3) For those who want to participate remotely, we will be using bluejeans.  The webpage for the bluejeans meeting is here:
 
https://bluejeans.com/462865444

I will start the connection ~20min before the start of the session; people calling in remotely might want to call early, so we can test the sound quality.

4) For those at TRIUMF, the meeting will be in the ISAC-II conference room.

Thomas
 

> Dear MIDAS users,
> 
> Here's further information on the third MIDAS workshop:
> 
> 1) Workshop will take place from 1PM-5PM (Vancouver time) on Aug 7.
> 
> 2) A mostly finalized agenda for the workshop is posted here:
> 
> https://indico.triumf.ca/conferenceDisplay.py?confId=2438
> 
> People are still welcome to email me if they want to present something.  We should be able to add it to the schedule.
> 
> 3) For those who want to participate remotely, we will be using bluejeans.  The webpage for the bluejeans meeting is here:
> 
> https://bluejeans.com/462865444
> 
> 4) For those at TRIUMF I will confirm the meeting room closer to the date.
> 
> Thomas
> 
> 
> > Dear MIDAS users,
> > 
> > We would like to announce a third MIDAS workshop at TRIUMF on Aug 7, 2019.
> > Stefan Ritt will again be visiting TRIUMF at this time.
> > 
> > The overall goal of the workshop is to present new features in MIDAS, to discuss
> > future changes and to hear experiences from different experiments.
> > 
> > We expect that some participants will connect remotely to the workshop; we will
> > setup a video-conferencing option.  The exact time of the workshop will be
> > decided later and will be optimized based on the geographic distribution of
> > remote attendees. So please let us know if you want to attend remotely.  We are
> > also happy for people to come in person to TRIUMF.  
> > 
> > A (very) preliminary agenda includes
> > 
> > - New default mhttpd pages and new APIs for custom pages
> > - Conversion of MIDAS to C++
> > - new C++ based frontend framework (tmfe and mvodb from ALPHA-g)
> > - MySQL/Postgres database for storing ODB configurations
> > - Plans for updating history plotting
> > - Using MIDAS with an online trigger farm
> > - new C++ multithreaded flow analyzer (manalyzer from ALPHA-g)
> > 
> > But please suggest other topics; we also hope to hear reports from particular
> > experiments.
> > 
> > Sincerely,
> > Thomas (on behalf of MIDAS developers)

Hi, 
I am missing a few features. Do any of the following exist and I have just
overlooked them?

- Arithmetics:
    SET foo, 2
    SET bar, 2
    SET FOOBAR, $foo + $bar
    -> FOOBAR is 2, not 4

- Vectors/arrays
    As far as I understand, "SET" only allows for single variables and no 
    "scripting" in variable names, i. e. doing something like the following
    scripted: 
    SET var_1, 0
    SET var_2, 0
    ...
    LOOP n, 10
        ...
        IF something
            SET var_${n}, 1
        ENDIF 
    
    The only way I could think of doing something similar is via the ODB.
    I don't know if it's good practice to fill the ODB with variables like this.

- Loading scripts at run time (see other bug)
    That would allow for script manipulation, even though it's kind of dirty, it
    might be useful in some cases.

- Obtaining results from external scripts
    A way to pass things from external scripts would be the solution to all
    problems. Something that is not implemented could be done in a bash or
    python script instead. 

Cheers
Lukas

> I am missing a few features.

MSL did not start out as a fully featured programming language.

Rather than extending it until it becomes one, I think a better approach would be to take
one of the existing extensible scripting language libraries and extend it with midas functions.

For example, LUA (https://www.lua.org/about.html) seems to be popular.

There were also requests for a midas extension for PYTHON (actually this has been done before,
but never added to the midas repository. I still have that code and I suppose it could be resurrected).

K.O.


> Do any of the following exist and I have just
> overlooked them?
> 
> - Arithmetics:
>     SET foo, 2
>     SET bar, 2
>     SET FOOBAR, $foo + $bar
>     -> FOOBAR is 2, not 4
> 
> - Vectors/arrays
>     As far as I understand, "SET" only allows for single variables and no 
>     "scripting" in variable names, i. e. doing something like the following
>     scripted: 
>     SET var_1, 0
>     SET var_2, 0
>     ...
>     LOOP n, 10
>         ...
>         IF something
>             SET var_${n}, 1
>         ENDIF 
>     
>     The only way I could think of doing something similar is via the ODB.
>     I don't know if it's good practice to fill the ODB with variables like this.
> 
> - Loading scripts at run time (see other bug)
>     That would allow for script manipulation, even though it's kind of dirty, it
>     might be useful in some cases.
> 
> - Obtaining results from external scripts
>     A way to pass things from external scripts would be the solution to all
>     problems. Something that is not implemented could be done in a bash or
>     python script instead. 
> 
> Cheers
> Lukas

Sure some existing scripting languages can be used, but they fall short of a few important items in larger experiments:

- they are typically run from a local terminal in the counting house. A remote observer of the experiment has no idea which script is running and at which state it is.

- if DAQ crashes during a script or is aborted, it has to be restarted from the beginning. If you run a sequence of let's say 100 runs taking 8 hours, and on run #98 something goes wrong, you are screwed if you have to start at run #1 again.

This are the main reasons why I developed the midas sequencer. Having everything web-based, everybody can watch remotely how far the sequence progressed. If the whole DAQ crashes, the sequence resumes from the crash point, not from the beginning. This is by saving the current state into the ODB. So even if the sequencer itself is stopped and restarted, that still works.

I agree that the MSL is missing a few calculations, and I was just waiting to get a few specific requests. I will either add new functions such as basic calculations like adding and subtracting variables, or I will create a way to call an external shell like bash to do calculations. I put this high on my todo list.

Stefan

Thank you two!

Actually, both solutions would allow me to fix my problem and I can see use cases for both. Having everything web-based is useful in bigger setups. However, in a testbeam environment, a python script is probably the more straight-forward solution. This gives access to the whole system of ones DAQ PC for whatever tricks one would like to perform.

Konstantin, would you mind resurrecting and sharing the python code? So far, my workaround is to call odbedit, which is not ideal for many reasons.

Cheers
Lukas

> Konstantin, would you mind resurrecting and sharing the python code?

Not until September or later.

I encourage you to start an "issue" in the bitbucket repository (a "request for improvement") and contact
Thomas L. to see if we can/should discuss it at the incoming midas workshop at TRIUMF.

> So far, my workaround is to call odbedit, which is not ideal for many reasons.

Yes, this has been the way to do it for years...

Also I was thinking of writing an command line tool to invoke midas json-rpc methods (via mhttpd),
then you can do from command line or from a script everything that a web page can do. (of course
you can do it today, by using "curl" to send http requests to mhttpd directly).

K.O.

> Yes, this has been the way to do it for years...

Calling odbedit -c ... ist certainly not the most effective way, but it works. I just tried on my Mac and found that I can call odbedit about 150 times per second. So not so bad if you have a limited operations to perform.

Stefan

Hi, Stefan, on second thought, I agree, I do not know of any scripting language implementation (packaged as a library or not) that
can store it's state in a file ("checkpoint the execution") and that can execute it's program "one line at a time", like the midas
sequencer does now. In the most severe case, one invocation of msequencer executes one line of the msl script.

K.O.

> Sure some existing scripting languages can be used, but they fall short of a few important items in larger experiments:
> 
> - they are typically run from a local terminal in the counting house. A remote observer of the experiment has no idea which script is running and at which state it is.
> 
> - if DAQ crashes during a script or is aborted, it has to be restarted from the beginning. If you run a sequence of let's say 100 runs taking 8 hours, and on run #98 something goes wrong, you are screwed if you have to start at run #1 again.
> 
> This are the main reasons why I developed the midas sequencer. Having everything web-based, everybody can watch remotely how far the sequence progressed. If the whole DAQ crashes, the sequence resumes from the crash point, not from the beginning. This is by saving the current state into the ODB. So even if the sequencer itself is stopped and restarted, that still works.
> 
> I agree that the MSL is missing a few calculations, and I was just waiting to get a few specific requests. I will either add new functions such as basic calculations like adding and subtracting variables, or I will create a way to call an external shell like bash to do calculations. I put this high on my todo list.
> 
> Stefan

Dear Stefan,

another thing which does not work is the comparison of floating point numbers.

The script:

IF 1.1 > 1.0 
    MESSAGE "foo" 
ENDIF 

Produces an error "Invalid number in comparison". This is due to isdigit() being used to find the numerical values in the condition at progs/msequencer.cxx:343.

Would it be possible to add something like the following?
 343 if (!isdigit(value1_var[i]) && value1_var[i] != '.')
 344          break;
Which would only leave open a problem with some string like "2.3.4"

Cheers
Lukas

> Would it be possible to add something like the following?
>  343 if (!isdigit(value1_var[i]) && value1_var[i] != '.')
>  344          break;

Actually isdigit() is completely wrong here, because it also fails the minus sign and the 'E' exponent, like in -1.2E-3 

So I changed it to strchr("0123456789.+-Ee", var[i]) which should cover this case. If you put 1.2.3, it takes it as 1.2. 

Stefan