I noticed that midas web pages consume unexpectedly large amount of resources, as observed by the chrome browser
"task manager" and by other tools.
For example, size of the "status" page was observe to reach 200, 600 and even 900 Mbytes. The "programs" page (which
does not have nearly as much stuff as the status page), was observed to reach 200-600 Mbytes. This is comparable to the
New York Times front page, which has much more stuff, but usually runs at about 200 Mbytes. (they do force a periodic full
page reload, to deal with exactly this same type of trouble, I suspect).
Also I observed the midas web pages consume an unusual amount of CPU - 5-10-15% - all in inactive tabs in minimized
windows.
All this was quite noticeable in my oldish mac laptop with only 8 GBytes of RAM.
Using the google-chrome performance analyzer I was able to identify the reason of high memory use - our 1/sec periodic
updates leak "too many" DOM "nodes" and I suspect that due to throttling of inactive tabs, the garbage collector simply
does not keep up with us.
(Note that javascript features automatic memory management with garbage collection. In practice in means that where in
C/C++ we have malloc() and free(), in javascript we only have malloc() and no free(), and cannot explicitly release memory
we know we no longer need. In the C/C++ sense, all memory allocations are leaked, and one relies on a janitor to "clean it all
up" eventually, later).
The source of node leakage was unexpected (unexpected to me). It turns out that each assignment to e.innerHTML creates
a new node, even if the new contents is the same as the old contents. (also the html parser has to run, consuming extra cpu
cycles).
Obvious solution is to write code like this:
if (v !== e.innerHTML) { e.innerHTML = v };
This helped quite a bit on the "programs" page, but not as much as expected, and hardly at all on the "status" page.
It turns out, that read of innerHTML does not necessarily return the same string as it was written into it.
For example, if "v" is "a&b", e.innerHTML will return "a&b" and the comparison will misfire.
There is more cases like this, see the section "Test set and get e.innerHTML" on the "example" midas page.
To help dealing with this, I suggest that instead of "inline" comparison (as above), one writes this:
mhttpd_set_if_changed(e, v);
Then to check that the comparison is effective, go to mhttpd.js and uncomment the console.log() call in
mhttpd_set_if_changed(), reload the page and look at the javascript console to see all calls that result
in assignment of innerHTML (and leakage of DOM nodes).
This done, after replacing many "&" with "&" and many "\'" with "\"", node leakage on the "programs" page was reduced
to 1 node per 1/sec update: the unavoidable change to the timestamp on the top-right of the page.
Luckily, Stefan pointed me to the solution for this: use of e.firstChild.data instead of e.innerHTML. The only quirk is that the
node should not be empty, which was easy to arrange by setting the initial value of the timestamp to a dummy value.
With these changes, the "programs" page (and most other pages) now leak 0 nodes (from the 1/sec periodic updates).
There is still some small memory leakage from making the RPC requests and from receiving the RPC replies, but the
garbage collector seems to have no trouble with them.
Typical memory use for all midas pages is now 50-60 Mbytes (down from 100-200 Mbytes).
The "status" page took a bit more work to fix due to it's curious coding, but it, too now uses 50-60 Mbytes as well. It still
leaks quite a few nodes (to be fixed!), but the garbage collector seems to keep up with the allocations.
K.O. |