Midas has a nice alarm system. I am wondering whether it is easily possible to
get the Alarm/Warning banner also on top of custom pages?!

I think there sneaked in a little bug in the mhttpd: when starting an experiment
from scratch and starting the mhttpd, the Menu Buttons are missing and,
correctly, I get periodic error messages. I expected that the default ODB entry
for the Menu Buttons is create if it doesn't exist. As far as I see this happens
now since the default creation of the 'Menu Buttons' is now tag as an obsolete
feature. In case this is not a bug but a feature, it should documented.

We use the customHeader to display some useful information. Currently I do not
like its style. What about to make it more alike the footer?

I just changed in resources/mhttpd.css

diff --git a/resources/mhttpd.css b/resources/mhttpd.css
index fb0070d..f3264c8 100644
--- a/resources/mhttpd.css
+++ b/resources/mhttpd.css
@@ -280,6 +280,15 @@ table.headerTable td{
        border: none;
 }
 
+div.headerDiv{
+       background-color: #6F6F6F;
+       text-align: center;
+       padding:1em;
+       color:#EEEEEE;
+       border-bottom:1px solid #000000;
+       height:3em;
+}
+
 div.footerDiv{
        background-color: #808080;
        text-align: center;

and

diff --git a/resources/mhttpd.js b/resources/mhttpd.js
index de8bc6c..972c261 100644
--- a/resources/mhttpd.js
+++ b/resources/mhttpd.js
@@ -172,7 +172,7 @@ function mhttpd_goto_page(page) {
 
 function mhttpd_navigation_bar(current_page, path)
 {
-   document.write("<div id=\"customHeader\">\n");
+   document.write("<div class=\"headerDiv\" id=\"customHeader\">\n");
    document.write("</div>\n");
 
    document.write("<div class=\"mnavcss\">\n");

What do you think?

I'd liked very much the possibility to hide away Equipment on the main page. It
is also nice to have the '+' to get it quickly back when needed. However, this
seems not to work anymore (git c9d9d604803). Is this a feature or something went
wrong?

> > I'd liked very much the possibility to hide away Equipment on the main page. It
> > is also nice to have the '+' to get it quickly back when needed. However, this
> > seems not to work anymore (git c9d9d604803). Is this a feature or something went
> > wrong?
> 
> The expansion of the equipment list is handled by a Cookie ("expeq" being 1 or 0). When Konstantin 
> implemented the mongoose server instead of the internal mhttp server, he neglected to evaluate 
> this cookie. I fixed this now (also renamed the cookie to "midas_expeq") in the current development 
> branch. Please check if it's working.
> 
> Stefan

Tested it on two machines and expansion is back and working! Thanks a lot!

Andreas

when I try to stop a run from odbedit I get a core dump.

[ODBEdit1,INFO] Run #31 stopped odbedit: src/system.c:1223: ss_shm_flush:
Assertion `size == mmap_size[handle]' failed. Aborted (core dumped)

midas commit 53af92a5d0...

-----

I checked what happens if I try to stop a run via the mhttpd web-page: this
works! So what is different?

-----

I placed a issue (# 47) on bitbucket as well.

What is the preferred channel to report potential bugs (elog / bitbucket issues)? 

> when I try to stop a run from odbedit I get a core dump.
> 
> [ODBEdit1,INFO] Run #31 stopped odbedit: src/system.c:1223: ss_shm_flush:
> Assertion `size == mmap_size[handle]' failed. Aborted (core dumped)
> 
> midas commit 53af92a5d0...
> 
> -----
> 
> I checked what happens if I try to stop a run via the mhttpd web-page: this
> works! So what is different?
> 
> -----
> 
> I placed a issue (# 47) on bitbucket as well.
> 
> What is the preferred channel to report potential bugs (elog / bitbucket issues)? 

I think I found the problem. Some ODB String values which are **automatically**
generated:

CSS File = STRING : [1024] mhttpd.css
Sqlite dir = STRING : [1024]
History dir = STRING : [1024]
Sound = STRING : [1000] alarm.mp3

are exceeding the MAX_STRING_LENGTH 256 (defined in msystem.h)

It looks as if this screws up quite a bit of the system! When deleting .ODB.SHM and
afterwards try to reload the ODB via a dump I previously made with odbedit, the
following is happening:

1) I get the error message that some strings are too long (exceeding
MAX_STRING_LENGTH). Unfortunately the underlying routine doesn't tell which ODB
variables this is.

2) After this reload, essentially nothing is working anymore. Any client I tried to
start just crashed.

Since it seems that the string length of MAX_STRING_LENGTH is very crucial I would
suggest that db_create_record (or whatever routine is dealing with it) checks for
STRING variables and ensures that they cannot exceed MAX_STRING_LENGTH.

When I shortened in my dump the above variables to MAX_STRING_LENGTH, regenerated the
ODB, also the 'stop' Problem in odbedit is gone.

I stumbled over an oddity which I would like to understand better. Here the
boundaries:
- Enable non-localhost RPC -> y
- Disable RPC hosts check  -> y

1) I am starting a run from ODBedit (start now -v):

07:13:11.272 2017/04/19 [ODBEdit,INFO] Run #26 started

07:13:25.516 2017/04/19 [Logger,LOG] File '/data/max/dlog/lem17_0026.root'
CRC32C checksum: 0x05ca4e7e, 1523383 bytes

On this little test experiment there is not much running, but it already shows
the effect I wanted to understand.

2) I am stopping the run from ODBedit (stop -v):

07:13:25.519 2017/04/19 [ODBEdit,INFO] Run #26 stopped

So, everything looks perfectly fine up to this point.

3) Now the 'strange' thing happens. To any point in time after this, I will stop
ODBEdit which results in the following messages:

07:13:32.335 2017/04/19 [ODBEdit,INFO] Program ODBEdit on host pc7962 stopped

07:13:32.335 2017/04/19 [Logger,ERROR] [midas.c:14079:rpc_server_receive,ERROR]
rpc check ok, abort canceled

This I do NOT understand! It looks as if the Logger (or any other client which
gets the run state transition) thinks that some Client (here ODBEdit) has a
broken connection. At least this is how I understand the comment in midas.c /
rpc_server_receive(). Is something broken in the de-registration from the RPC
server? By the way, all clients where running on the localhost, i.e. no remote
connection used here.

All this only happens if a run transition took place.

Unfortunately I do not understand the system well enough to suggest any fix to
this :-( and hence would appreciate any help. 

A handy feature of the history of the mhttpd is to export the data. However, this 
seems to be broken. It currently only works if the run marker flag is activated by 
fails otherwise.

Description of the problem (starting with 61be7a1):

When starting a new experiment, creating a fresh ODB and than adding the 
directory '/Custom', the mhttpd runs into a problem on RHEL/Fedora, but not on 
Ubuntu and macOS. When trying to open the ODB from within whatever browser I get 
the following error message in the midas message queque:

[mhttpd,ERROR] [mhttpd.cxx:563:rread,ERROR] Cannot read file '/root', read of 
4096 returned -1, errno 21 (Is a directory)

and in the browser I get a popup which tries to save a file called 'root'.

I track this down to the following: in mhttpd, interprete (line 18046) it is 
check if a custom page file exists (ss_file_exist) and if yes, it tries to 'load' 
it. Now, at this stage the variable dec_path contains '/root'.

Here now what goes wrong: ss_file_exist tries to open the given path, and if a 
valid file descriptor is returned it assumes the file exists. This is not 
perfectly correct since it also will get a valid file descriptor is path is an 
accessible directory!

Now for whatever reason, on RHEL/Fedora '/root' will return a valid file 
descriptor, but not on macOS and Ubuntu. Others I haven't tested. A possible fix 
would be to check explicitly if path is a directory and if yes return 0 in 
ss_file_exist (see attached diff).

Perhaps there is cleaner way to deal with this issue?! 

> > In any case, IMO, mhttpd has no business serving the contents of /root,
> > or serving any files outside of the mhttpd user $HOME directory. (but also
> > should not serve files from ~user/.ssh, or any other "secret" files, good
> > luck making a complete axhuastive list of all secret files that should not be
> > served).
> 
> I fully agree with Konstantin. mhttpd should only serve files under certain directories. One is the 
> midas/resources directory, another is the one defined in the ODB under /Custom/Path. I plan to modify 
> mhttpd to only serve these files (and also prevent tricks like putting "../../../" into the URL). This will then 
> also fix Andreas' problem.
> 
> Stefan

Yes, I think this is a good idea as well. But just to make sure that you get the problem: it's not the custom page call by itself! If you currently type e.g. <midas-server>/root it will open the ODB html-page.

There is a bug in the string handling when changing ODB string entries via the
mhttpd (git sha 07dfb83). It truncates the string length in the ODB.

For instance I create a string with length 32 and set it with odbedit to 'a'.
Then the string length stays 32, as expected. If the same is done through the
web-interface, the string length will be truncated to 2.

This can lead to problems if some frontend has a hotlink to a structure
containing this string since it will complain about structure size mismatch. 

> > There is a bug in the string handling when changing ODB string entries via the
> > mhttpd (git sha 07dfb83). It truncates the string length in the ODB.
> > 
> > For instance I create a string with length 32 and set it with odbedit to 'a'.
> > Then the string length stays 32, as expected. If the same is done through the
> > web-interface, the string length will be truncated to 2.
> > 
> > This can lead to problems if some frontend has a hotlink to a structure
> > containing this string since it will complain about structure size mismatch. 
> 
> I know about this problem since last summer. I mentioned it to KO, since it's deep down in his 
> JSONRPC code. We had a long discussion, where he kind of insisted that this is not a bug but a 
> feature. The ODB should store strings with variable lengths, and thus adapt it according to the 
> current string length. This makes some sense, since in the future we plan to put C++ string 
> support for the ODB, where strings have dynamically varying lengths. But this will take a while, so 
> I asked KO to change the truncation of the strings though the web interface, because this breaks 
> many experiments. He did not react so far. Several people complained. Maybe your request will 
> help now.
> 
> Stefan

Well I appreciate the direction towards more C++ string handling, yet it must not break the hotlink
functionality which is very important at many places.

Is there a simple way from the javascript side to check if a fontend is running?
Currently one would need to go through the /System/Client list to find out if a
frontend/client is running. Wouldn't it be nice to have this centralized, either
in the mhttpd.cxx or mhttpd.js part? 

> > Is there a simple way from the javascript side to check if a fontend is running?
> > Currently one would need to go through the /System/Client list to find out if a
> > frontend/client is running. Wouldn't it be nice to have this centralized, either
> > in the mhttpd.cxx or mhttpd.js part? 
> 
> Hi,
> 
> I think that this option already exists with the cm_exist method for the mjsonrpc calls.  For instance, you can use a 
> call like 
> 
> curl -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","id":null,"method":"cm_exist","params":
> {"name":"Logger"}}' 'http://localhost:8081?mjsonrpc'
> 
> to get the status of the logger program.  There is a description of the cm_exist parameters on this page:
> 
> https://midas.triumf.ca/MidasWiki/index.php/Mjsonrpc

Thanks a lot for the info. I just simply missed it :-|

A feature I always missed (or just missed to find in the docu) is the following:
1) It would be nice to have a command in odbedit which allows to check how full
the ODB currently is.
2) Even more important: I would like to have an ODB routine which allows me to
check the fill level of the ODB, and/or a routine which tells me if I would
create a structure of given size that it still fits in the current ODB or not. 
The use case is that some clients create on the fly ODB entries and I would like
to make sure before hand the ODB's remaining space in order not to crash things
by overfilling the ODB. 

> > A feature I always missed (or just missed to find in the docu) is the following:
> > 1) It would be nice to have a command in odbedit which allows to check how full
> > the ODB currently is.
> > 2) Even more important: I would like to have an ODB routine which allows me to
> > check the fill level of the ODB, and/or a routine which tells me if I would
> > create a structure of given size that it still fits in the current ODB or not. 
> > The use case is that some clients create on the fly ODB entries and I would like
> > to make sure before hand the ODB's remaining space in order not to crash things
> > by overfilling the ODB. 
> 
> If you do "mem" in odbedit, you see the currently free areas, one for the keys themselves, one for 
> the data of the keys. The corresponding C function is db_show_mem. At the moment it outputs the 
> list of free blocks as a long ASCII string, but if necessary I can write a variant which returns the 
> number of free bytes.
> 
> Stefan

Thanks for the info, and yes a variant of db_show_mem returning the number of free bytes would be just
prefect!

For us it would be a handy feature if history data could be requested directly
from a custom page (time range or run based intervals) . Here I am not talking
about history plots but I am talking about recorded time series data. This way
we could easily generate useful graphs. For instance, if we measure the voltage
(constant current) while cooling, we could instantly get the resistance versus
temperature. Often we would like to 'correlate' recorded slow control data.

Is it already possible to extract history data the way suggested?