Back Midas Rome Roody Rootana
  Midas DAQ System  Not logged in ELOG logo
Entry  19 Nov 2020, Joseph McKenna, Forum, History plot consuming too much memory 
    Reply  19 Nov 2020, Stefan Ritt, Forum, History plot consuming too much memory 
       Reply  20 Nov 2020, Joseph McKenna, Forum, History plot consuming too much memory 
          Reply  20 Nov 2020, Stefan Ritt, Forum, History plot consuming too much memory 
          Reply  27 Nov 2020, Konstantin Olchanski, Forum, History plot consuming too much memory 
       Reply  27 Nov 2020, Konstantin Olchanski, Forum, History plot consuming too much memory 
    Reply  27 Nov 2020, Konstantin Olchanski, Forum, History plot consuming too much memory 
       Reply  27 Nov 2020, Konstantin Olchanski, Forum, History plot consuming too much memory 
Message ID: 2028     Entry time: 27 Nov 2020     In reply to: 2015     Reply to this: 2029
Author: Konstantin Olchanski 
Topic: Forum 
Subject: History plot consuming too much memory 
> 
> Tested with midas-2020-08-a up until the HEAD of develop
> 

Just so you know, it took myself and Stefan quite a bit of effort
to improve memory and data handling in the new history plots
to be able to plot 1 year of data without bogging down too much. I got
to learn the google-chrome javascript cpu profiler, memory profiler
and the intricacies of javascript shift() and unshift() operators.

Before midas-2020-08-a, pressing the zoom-out button you would never
reach the javascript memory limit, the code would go into "100% cpu use"
and the browser tab will become progressively unresponsive well before
running out of memory. With the original code, our alpha-g history plots
could go back a few weeks at most, with the current code, we can go back
about 11 months. Compared to the old "C" history plots that can
do "last 10 years", no problem.

Loading all the history data into the browser is a design choice.

It has benefits and downsides.

The main benefit is that looking at immediate live data is much easier.

The main downside is that "plot last 10 years" becomes impossible.

As they say "appetite comes during eating", we have learned about these
downsides as we developed the new system. When we started, we did not
know much about javascript memory limits, cpu limits, etc. We did learn
a lot, though.

With the current code, we are limited to loading history data up to 50% of
the javascript memory limit. I know how to change the code to get up to 100%,
but I think it is not worth it, it still does not get as to plot "last 10 year".

We think the solution to recovering "last 10 years" capability is to use
binned data (which the history system can already deliver to javascript).
With binned data, the data volume in Mbytes remains constant, javascript
memory use has an upper-bound (we never use more memory than X Mbytes)
and data movement over the network is reduced.

Another way to look at this - typical display has only 1000-4000 vertical pixels,
it cannot physically display a bigger number of data points (no more
then 1 data point per pixel). So why load 1000000 data points when we only
can plot 1000-4000 of them?

So all the infrastructure for plotting binned data is already there,
but the javascript code still needs to be written. I think the biggest
challenge will be in blending or combining binned and unbinned data
on the same plot or in seamlessly switching the plot between binned and
unbinned data.

K.O.
ELOG V3.1.4-2e1708b5