> Lacking any ideas for improvements, I vote for the status quo. (plus a review of the documentation to ensure we have clearly
> written up what each entry in "common" does and whether the user is permitted to edit it in odb).

I agree with that.

Stefan

During my visit at TRIUMF we rewrote the history plotting functionality of midas. Instead of 
static GIF images, we have now interactive JavaScript panels where we can scroll, zoom, 
inspect values and much more (example is attached). We are now in a state where this is still 
work in progress, but already at this stage it might be useful for others to report any 
feedback.

Simply upgrade the the newest develop branch of midas, and you will see two menu items 
"OldHistory" which is the old system and "History" which is the new system. In the new 
system, you can drag with the mouse to scroll, use the mouse wheel to zoom in and out the 
time axis, and hover with your mouse over data points to see its value. If you zoom out, 
old data is loaded automatically in the background.

Following items are planned, but not yet implemented:

- Printing of run markers as in the old history

- Delete old data in the buffer to limit memory consumption if the browser window is 
   open for very long (weeks)

- Implement time interval selector (clock icon, select "last day", "last 8 hours" etc.)

- New settings dialog as a floating dialog box. At the moment, the setting page of the 
   old history system is used

- Export / Printing / Sending to ELOG any history plot

- Implement a formula for plotting data, such as "y = 12 * (x-14) +32". This will replace 
   the old "offset" and "factor" and is more flexible. The formula can be passed directly 
   to the JavaScript engine and will be executed on the web page. It should be also 
   possible to combine different channels, like the difference of two history values.

- Determine the number of digits for variable display from the axis limits. Like if a value 
   changes between 520001 and 520002 only, we need more digits than the usual 6.

Many of these things will be implemented in the next weeks. If you have any more idea 
or find some bugs, please report back to me.

Best,
Stefan for the midas team

My first question would be why are you using several font-ends at all? That makes things more 
complicated than needed. In the normal FE framework, you can define either several equipment 
served by one frontend, or even one equipment linked to several devices. In the MEG experiment 
we have one slow control frontend controlling ~100 devices without problem. In the old days there 
was a problem that some slow devices could throttle the readout, but since the invention of multi-
threaded slow control equipment, each device gets its own thread so they don't block each other.

Stefan

> 1) it would be nice to have an option to format the label output (see attachment 1)

That's clearly a bug, I will fix it.

 
> 2) the background of a history plot is very handy if you only show one measure.
> If you have multiple ones (see attachment 2), this is not the case anymore. It
> would be nice if the background could be enabled/disabled.

Looking at your plot, even without the background things look messy. Please note
that you can display only a single curve by double clicking on it (and back with Escape).
If all curves are on top of each other, you can get them apart a bit by zooming
in to the vertical axis, then double click. Let ma know if that works for you.

Best regards,
Stefan

> This I found out, yet the attachment here shows another case where it would be useful to be
> able to disable the background, namely if you have positive and negative measures in one
> plot. Somehow it suggests that CH1 and CH2 show very different values, whereas it is only a
> difference in the sign of this variables.

Ok, I added 

- a correction which does the fill not to the bottom of the window, but only to the y=0 axis.
- a flag "Show graph fille" which lets you turn on and off the filling for each plot

Best,
Stefan

> This I found out, yet the attachment here shows another case where it would be useful to be
> able to disable the background, namely if you have positive and negative measures in one
> plot. Somehow it suggests that CH1 and CH2 show very different values, whereas it is only a
> difference in the sign of this variables.

Ok, I added an option which lets you switch off the background. 

I also changed the background drawing such that it only goes to the y=0 axis, not the bottom of the screen. 
That should help displaying negative values.

Stefan

> 1) it would be nice to have an option to format the label output (see attachment 1)

I fixed that in the current version.

Stefan

>  Also the new system is still incomplete, i.e. there is no trivial way to save a history plot into a file:

That has been implemented in meantime. Just click on the download arrow and you can save the current window in CSV or PNG format.

Stefan

I wonder why do you this via ODB links. The "standard" way of writing to the history should be to create events for an equipment and flag this equipment as being written to the
history. All variables under /Equipment/<name>/Variables then automatically go into the history and you don't have to worry about ODB links. Only variables not fitting the
equipment/variables scheme should be dealt with via ODB links, like variables under equipment/statistics or parameters in another ODB tree. In a typical midas experiment, only
very few variables typically go into the 'System' event. This is however probably not a solution to your problem. If you have a similar structure (doubles plus an odd number of floats)
under 'variables', you might get the same error. I'n in contact with KO to fix this problem at the root level.

Stefan

> Logging a list of variables to the history via links in the history ODB subtree,
> we get messages as follows at every run start:
> 
> 19:43:24.009 2019/10/06 [Logger,ERROR] [history_schema.cxx:2676:hs_write_event,ERROR] Event 'System' data size mismatch: expected 412 bytes, got 416 bytes
> 
> 19:43:24.008 2019/10/06 [Logger,ERROR] [history_schema.cxx:2676:hs_write_event,ERROR] Event 'System' data size mismatch: expected 412 bytes, got 416 bytes
> 
> 19:43:23.850 2019/10/06 [Logger,ERROR] [history_schema.cxx:455:hs_write_event,ERROR] Event 'System' data size mismatch count: 25, expected 412 bytes, hs_write_event() called with as much as 416 bytes
> 
> 19:43:23.850 2019/10/06 [Logger,ERROR] [history_schema.cxx:455:hs_write_event,ERROR] Event 'System' data size mismatch count: 25, expected 412 bytes, hs_write_event() called with as much as 416 bytes
> 
> The history calculates the size of a record from the size of the individual variables, (history_schema.cxx, L2666 ff), whereas the ODB delivers the data aligned/padded to the size of the largest value in the record.
> In our history, a long list of doubles (64 Bit) fas followed by three floats (32 bit), leading to a padded response from the ODB, 4 byte longer than the history expects.
> Quick fix: Add another 32 bit dummy variable to the history. Gets rid of the error messages...
> Should probably be fixed at a deeper level...

> Yes, we could have
> kept that apart, yes, in this case a double would also work (and not break things), but a bug is a bug...
> I could think of senisble use cases where doubles and ints are mixed and I also know quite a few areas where it makes
> sense to use floats...

I agree with Nik that we should fix this on the midas level. Since it happens in history_schema.cxx which was written by KO, maybe he can have a look.

Stefan

> > The compile option -DHAVE_FTPLIB checked in mdsupport.cxx disappeared if you 
> > compile with cmake.
> 
> Hi, Stefan - do we still need to support FTP in the logger? In the lazylogger, special support for 
> FTP is not needed, they can you the "script" method and do FTP without our help.
> 
> I move to remove FTP support from MIDAS. (second? other opinions?)

I oppose to remove FTP support from lazylogger. We still use it heavily at PSI. In comparison to the "script" method, it 
shows the current speed in MB/s which helps us to diagnose some network problem by writing this number into the 
history. The "script" method only give you an integral transfer speed after a file has be completely written.

I'm however not sure who FTP is used in lazylogger. It goes into mdsupport.cxx and I seem to remember that Pierre 
wrote the FTP code by hand, so no external library is necessary.

Stefan

There is a not-so-well-known function in the ODB to write protect some keys. You can do

odbedit> chmod 1 /Equipment/HV/Demand

which will write protect your Demand values. You see that by doing

odbedit> ls -ls

where you only see a "R" at the right end instead a "RWD". I haven't tried it yet (so better do a dry run yourself), but that should prevent an odb load to overwrite your demand values. To change the values, put some logic on a custom page to unprotect the 
values, change them, and then protect them again.

Stefan

Very good idea. And thanks for finding the document.hidden solution. I put it in, so give it a try.

Best,
Stefan


> I am currently testing the new history system on the mhttpd side and stumbled over the following issue: typically our user open a lot of midas web-page tabs and keep them open. With the current version this leads after a night typically to a state where the browser is busy with itself and not reacting anymore.
> 
> One important reason seems to be that ALL tabs trying to communicate all the time which is totally unnecessary, since I think a hidden tab should stay in a sleeping mode. 
> 
> I was browsing if there is a way to find out if a tab is active or not, and found the following API which exactly does this:
> 
> https://developer.mozilla.org/en-US/docs/Web/API/Page_Visibility_API
> 
> Furthermore, the simple
> 
> document.hidden 
> 
> tag, could be used to find out if the page is currently active.
> 
> Wouldn't it a good idea to send all midas tabs which are not active into a sleep mode and only reactivate them if they come into focus?
> 
> I had a quick look at the JavaScript libs of midas, but I am not quite certain where to best inject this. 

> Hi, Stefan - I did not look at your code, if all midas tabs are inactive, will the alarm sound still play?

Nope. All updates are done in mhhtpd_refresh(), and I changed it such that nothing is updated if hidden.

I agree however that this is bad. You want to hear alarms always. So I added some code to do ONLY alarm updates if in the background. 
No GUI changes, only audio playing. Checking is reduced from 1 Hz to 0.1 Hz (once every 10 seconds). I don't have however a solution 
to your problem "not enough interaction with page" which I have never seen so far.

I wonder if we should try push notifications: https://developers.google.com/web/fundamentals/codelabs/push-notifications
which seems a bit complicated to me. It might also be too subtle if someone is sleeping in front of the computer.

Stefan

> a) google chrome slows down the execution of javascript in inactive tabs, leading to trouble
> with memory management - midas pages poll at 1/sec, each poll allocates memory for processing RPC messages,
> and (until recently) allocates memory for new DOM objects to update the web page - but the garbage collector
> gets slowed down and does not keep up - leading to huge memory use (up to 200 Mbytes) for inactive midas pages
> that normally consume 50-ish Mbytes.

Try my latest change, which drops any update if hidden, except the alarm sound. At the moment this only works on the main "status" page.

> most midas web pages poll in two places - mhttpd_refresh() updates the current date timestamp, alarms, currently active midas.log message;
> and each page has it's own loop for updating it's own data (i.e. "alarms" page, "programs" page).

That's correct. I changed mhttpd_refresh() now and the main loop for the "status" page. If that works for everybody, we can do that also for "programs" and other pages. The code is here:

      if (document.hidden) {
         // don't update page if hidden
         setTimeout(update_page, 500);
         return;
      }

where "update_page" has to be replaced with the proper function.

Stefan

The document

https://docs.google.com/document/d/1_278v_plodvgtXSgnEJ0yjZJLg14Ogf-ekAFNymAJoU/edit

says that the "... playback must be at lest 7 seconds long" to signal significant playback.

I attached a modified alarm sound file with 3 seconds of sound followed by 5 seconds of silence. Try that one and let me know if this changes anything. If successful, I can modify the 
other sound files as well.

Stefan

We updated midas/examples/experiment/frontend.cxx to correctly contain

/*-- Interrupt configuration ---------------------------------------*/

INT interrupt_configure(INT cmd, INT source, POINTER_T adr)
{
   switch (cmd) {
   case CMD_INTERRUPT_ENABLE:
      break;
   case CMD_INTERRUPT_DISABLE:
      break;
   case CMD_INTERRUPT_ATTACH:
      break;
   case CMD_INTERRUPT_DETACH:
      break;
   }
   return SUCCESS;
}

but if you upgrade from C to C++ from your own old frontend code you might be hit by that issue. 

Maybe Konstantin can update elog:1526 to contain a hint about "INT source".

Stefan

Actually now I see that 

a4) poll_event() and interrupt_configure() have "source" as "int[]" instead of "int" (why did this work before?)

mention this already, but maybe it's not completely clear that one has to change "int[] source" to "int source"

Stefan


> We updated midas/examples/experiment/frontend.cxx to correctly contain
> 
> /*-- Interrupt configuration ---------------------------------------*/
> 
> INT interrupt_configure(INT cmd, INT source, POINTER_T adr)
> {
>    switch (cmd) {
>    case CMD_INTERRUPT_ENABLE:
>       break;
>    case CMD_INTERRUPT_DISABLE:
>       break;
>    case CMD_INTERRUPT_ATTACH:
>       break;
>    case CMD_INTERRUPT_DETACH:
>       break;
>    }
>    return SUCCESS;
> }
> 
> but if you upgrade from C to C++ from your own old frontend code you might be hit by that issue. 
> 
> Maybe Konstantin can update elog:1526 to contain a hint about "INT source".
> 
> Stefan