Back Midas Rome Roody Rootana
  Midas DAQ System  Not logged in ELOG logo
Entry  01 Mar 2019, Konstantin Olchanski, Info, Gyrations of custom pages and ODB /Custom/Path 
    Reply  04 Mar 2019, Stefan Ritt, Info, Gyrations of custom pages and ODB /Custom/Path 
       Reply  04 Mar 2019, Thomas Lindner, Info, Gyrations of custom pages and ODB /Custom/Path 
          Reply  04 Mar 2019, Stefan Ritt, Info, Gyrations of custom pages and ODB /Custom/Path 
             Reply  04 Mar 2019, Suzannah Daviel, Info, Gyrations of custom pages and ODB /Custom/Path 
                Reply  04 Mar 2019, Konstantin Olchanski, Info, Gyrations of custom pages and ODB /Custom/Path 
                   Reply  05 Mar 2019, Stefan Ritt, Info, Gyrations of custom pages and ODB /Custom/Path 
                      Reply  05 Mar 2019, Thomas Lindner, Info, Gyrations of custom pages and ODB /Custom/Path 
                         Reply  05 Mar 2019, Konstantin Olchanski, Info, Gyrations of custom pages and ODB /Custom/Path 
                      Reply  05 Mar 2019, Konstantin Olchanski, Info, Gyrations of custom pages and ODB /Custom/Path 
                         Reply  05 Mar 2019, Stefan Ritt, Info, Gyrations of custom pages and ODB /Custom/Path 
                            Reply  05 Mar 2019, Konstantin Olchanski, Info, Gyrations of custom pages and ODB /Custom/Path 
                               Reply  05 Mar 2019, Stefan Ritt, Info, Gyrations of custom pages and ODB /Custom/Path 
                                  Reply  06 Mar 2019, Konstantin Olchanski, Info, Gyrations of custom pages and ODB /Custom/Path 
                   Reply  05 Mar 2019, Stefan Ritt, Info, Gyrations of custom pages and ODB /Custom/Path 
                      Reply  05 Mar 2019, Konstantin Olchanski, Info, Gyrations of custom pages and ODB /Custom/Path 
                         Reply  05 Mar 2019, Stefan Ritt, Info, Gyrations of custom pages and ODB /Custom/Path 
                            Reply  05 Mar 2019, Konstantin Olchanski, Info, Gyrations of custom pages and ODB /Custom/Path 
       Reply  14 Mar 2019, Konstantin Olchanski, Info, Gyrations of custom pages and ODB /Custom/Path 
          Reply  14 Mar 2019, Konstantin Olchanski, Info, Gyrations of custom pages and ODB /Custom/Path 
             Reply  14 Mar 2019, Konstantin Olchanski, Info, Gyrations of custom pages and ODB /Custom/Path 
          Reply  21 Mar 2019, Konstantin Olchanski, Info, Gyrations of custom pages and ODB /Custom/Path 
    Reply  21 Mar 2019, Konstantin Olchanski, Info, Gyrations of custom pages and ODB /Custom/Path 
Message ID: 1455     Entry time: 01 Mar 2019     Reply to this: 1457   1504
Author: Konstantin Olchanski 
Topic: Info 
Subject: Gyrations of custom pages and ODB /Custom/Path 
Before the days of javascript and ajax and web 2.0, MIDAS introduced "custom pages" for
building graphical display that could show "live" data from MIDAS and that could
have buttons and controls to operate slow controls equipment, etc.

This is how it works:
- entries from ODB /Custom are shown on the MIDAS menu -
an odb entry /Custom/Foo generate a link labeled "Foo"
to a special mhttpd url /CS/Foo.
- access to mhttpd url /CS/Foo invokes show_custom_page()
- show_custom_page() reads the custom page file name from ODB /Custom/Foo
- content of this file is served the web browser (after substituting the <odb> tags with values from ODB).
- in addition, it is possible to store the contents of the custom page in the ODB variable /Custom/Foo itself,
making it easy to edit the custom pages through the web browser (using the mhttpd odb editor).
- (if the value of /Custom/Foo has no "\n", then it's a file name, otherwise it is the page contents).
- if /Custom/Path exists, it is prepended to all file names.
- read more about this here:
https://midas.triumf.ca/MidasWiki/index.php//Custom_ODB_tree
https://midas.triumf.ca/MidasWiki/index.php/Internal_Custom_Page

This method required each custom web page served by mhttpd to have a corresponding
entry in ODB /Custom. Quite tedious for big experiments with a large number
of web pages (in T2K/ND280/FGD, for 1 page per frontend board, these entries
had to be created using a script, no practical to create them manually).

To fix this, in 2015, a modification was made to the code for ODB /Custom/Path. Instead of reading the file name
from ODB /Custom/Foo, the filename was made by adding /Custom/Path and the URL itself:
URL /CS/Foo.html will serve the file ODBValue["/Custom/Path"]/Foo.html. In this scheme,
the ODB entry /Custom/Foo.html is optional and is only needed to create a link in the MIDAS menu.

Commit: https://bitbucket.org/tmidas/midas/commits/5a2ef7d66df353684c4b40882a391b64a068f61f made on 2015-03-19.
Corresponding midas forum entry: https://midas.triumf.ca/elog/Midas/1109 made on 2015-09-09.

This change had the effect of creating 2 different operating modes for custom pages:
- if ODB /Custom/Path was absent (the default case), file names are taken from ODB /Custom/Foo.
- if ODB /Custom/Path is present (it has be created manually), file names are taken from the URLs.

These two modes could not be mixed, some experiments (TRIUMF MUSR/BNMR, CERN ALPHA, etc)
continued to use the old scheme (no /Custom/Path), some experiments switched
to the new scheme (DEAP, etc - TBC?).

(The best I can tell, this also create a security problem were using URLs containing "..", one can
force mhttpd to escape the /Custom/Path filename jail to cause it to serve arbitrary files from
the filesystem. Note that many web browsers remove the ".." entries from URLs, special tricks
are required to exploit this).

Then, in 2017, something called "new custom pages" was added to mhttpd interprete().
commit https://bitbucket.org/tmidas/midas/commits/c574a45c6da1290430f5a85a348fb629596a0de0 made on 2017-07-24
(I cannot find the corresponding explanation on the midas forum).

The best I can tell this code is intended to do the same thing that was done before:
map URLs like http://localhost:8080/Foo.html to file name ODBValue["/Custom/Path"]/Foo.html
and if this file exists, serve it to the browser.

However, the code has a bug - there is no check for absence of ODB /Custom/Path. If it does
not exist (as in experiments that do not use custom pages or use the old-style custom pages without
/Custom/Path), an empty string is used and URLs like http://localhost:8080/Foo.html becomes
mapped to /Foo.html (in the root of the file system, "/"!!!).

This introduced 3 problems:
- http://localhost:8080/etc/hosts serve the system password file (a security problem)
- because this code is before the odb code, it intercepts (and breaks) valid ODB URLs:
- on Linux, the ODB editor URL http://localhost:8080/root instead of showing the ODB root, attempts to serve the local subdirectory "/root" as a file
(fails, /root is a directory on most Linuxes). This is reported here: https://midas.triumf.ca/elog/Midas/1335 (2018-01-10)
- on MacOS, ODB editor URL http://localhost:8080/System instead of showing ODB /System, attempts to serve the local directory "/System" is a file (fails, 
/System is a subdirectory in MacOS). This is reported here: https://bitbucket.org/tmidas/midas/issues/156/mhttpd-odb-editor-cannot-open-system-on 
(2018-12-20)

The problem with broken access to the ODB editor URLs is fixed by commit:
https://bitbucket.org/tmidas/midas/commits/f071bc1daa4dc7ec582586659dd2339f1cd1fa21 (2018-12-21)
https://midas.triumf.ca/elog/Midas/1416

The fix unconditionally creates /Custom/Path and sets it to the value of $MIDAS_DIR or $MIDASSYS.

This breaks experiments that use custom pages with no /Custom/Path (ALPHA, BNMR/BNQR), see
https://bitbucket.org/tmidas/midas/issues/157/odb-custom-is-broken, and this is impossible
to fix by deleting /Custom/Path as it will be created again.

Also the problem with serving the system password file remains, see
https://midas.triumf.ca/elog/Midas/1425

So, a https://en.wikipedia.org/wiki/SNAFU

Since then, we have changed the mhttpd URL scheme: we no longer use URL subdirectories to navigate the ODB editor (we now use &odb_path=xxx)
and to refer to custom files (use now use &cmd=custom&page=xxx).

This liberates the mhttpd URL space for serving custom files without colliding with mhttpd internals and allows us to repair the current situation.

I will propose a solution in a follow-up message.

K.O.

P.S. The actual series of malfunctions is a bit more complicated than I described in this message,
but there is no need for getting to the very bottom bottom bottom of this as we can now do
a fairly clean solution.
ELOG V3.1.4-2e1708b5