Hi, guys, as I was exploring the code and the commit history on Thursday (git rules!) and
as I worked on getting the old custom files to work with Suzannah on Friday, I think
I know how I want this code to work. I think there is no need to break with the old
way of doing things and force every experiment to move things around if they want
to use the latest midas.

I will write more about this, but in the nutshell, I am happy with the current code:
- the old custom pages work (filename is taken from ODB, with /Custom/Path prepended if it exists; this is what we had for a very long time)
- the serving of new custom pages also works - via Stefan's code in interprete() where /Custom/File is added to the URL
  and if the file exists, we serve it. The only problem in that code was the missing check for absent or empty ODB /Custom/Path.
- the serving of new custom pages through the normal resource path (show_resource()) also works now,
   this serves files from $MIDASSYS/resource, from ODB /Experiment/Resource and a few other locations. (this code was there
   for a long time, but disabled because for a number of reasons, things like http://localhost:8080/Status.html did not work right,
   and after fixing a few buglets, they do now).

The serving of /etc/passwd I killed by forbidding "/" (the directory separator) in resource file names. I think this is safer
for enforcing a file jail compared to checking for "..".

I think the current code fixes all the reported problems (in conjunction with the change of the URL scheme) -
- /Custom/Path set to "" now works and provides the old way of doing custom pages
- /Custom/Path set to a directory name works and all Thomas's experiments should be good. the old custom files way *also* works, as long as the filenames in ODB are adjusted.
- URL /root and /System no longer try to serve system directories, plus /Custom/Path set to "/" is explicitly forbidden.
- mhttpd cannot serve /etc/passwd by default as "/" is forbidden in file names added to /Custom/Path. (I discount the case where mhttpd is running in /etc or /Custom/Path is set to /etc or symlinked to /etc)

But not all is good. The change of custom page URL scheme from /CS/... to top level or ?cmd=custom&page=...
cannot be swept under the rug - the user will have to make changes to their custom pages to adjust for it,
I see no way to avoid that. The current code catches and redirects/serves/helps with some of that,
i.e. pages loading custom files like "bnmr.css" still work even though bnmr.css is no longer under /CS/bnmr.css).

Again, apologies that things are moving faster than I can write them up. I am trying.


K.O.


> I see two separate issues here. 
> 
> One is restricting the custom pages to ONE directory such as 
> <exptab>/resources -> /home/users/exp/resources
> and its subdirectories which seems like a good solution for all the
> reasons you've mentioned. 
> 
> The other issue is the use of the "Path" key in /Custom, which is used to differentiate
> between the "new" way (all resources served from the Path directory) 
> and the original way where all the custom keys are specified with their full directories.
> 
> Recent versions of Midas had broken the original behaviour by insisting on the presence of the
> "Path" key.  Konstantin fixed this by allowing the "Path" key to take the value "".  It is true
> that some experiments currently may be serving resources from more than one directory tree, but changing
> to storage of all custom pages in one directory (and its subdirectories) does not necessarily mean that
> the original way of serving resources must be made obsolete.
>  
> I actually like the original way of specifying the custom keys for the pages and resources under /Custom, which
> is presently selected  without the /Custom/Path key present at all (older versions) or with the
> /Custom/Path key set to "" (latest versions). I like it for debugging, and I like to be able to see 
> at a glance what resource files are in use from /Custom. 
> 
> I have a suggestion:
> 
> The resources could still be served from the /Custom directory if desired, except now mhttpd will ALWAYS add the 
> fixed path in front of the given paths in /Custom. This would mean a fixed path and a minimal disruption to older pages 
> (the <script> and <link> statements in the HTML code to include the resources would not need to be changed).
> The "/Path" key is no longer be useful, since the resource path is now fixed. Instead a key e.g. "FlagRS"  could 
> be used to select the desired behaviour,  with the default being the "new" (no key present).
> 
> For example, the full directory paths in /custom
> ScanParams&                     /home/users/online/custom/scan/scan_select_popup.html
> mpet.css!                       /home/users/online/custom/rs/mpet.css
> scanvoltages!                   /home/users/online/custom/scan/scan_voltages.js
> 
> would become subdirectory path(s)
> ScanParams&                     custom/scan/scan_select_popup.html
> mpet.css!                       custom/rs/mpet.css
> scanvoltages!                   custom/scan/scan_voltages.js
> FlagRS                          y
> 
> The pages would be served from /home/users/exp/resources/custom/...
> 
> Suzannah
> 
>  
> > > Hi Stefan and Konstantin,
> > > 
> > > I think that this proposal sounds fairly reasonable.  I agree that we might as well move to a secure final solution at this point.  
> > > 
> > > One comment: since this change would break almost every experiment I have worked on for the last 4 years, it would be nice to add a command-line option to mhttpd that preserves the old /Custom/Path behavior.  This would allow experiments a transition 
> > > period, so that they didn't immediately need to fix their setup.  The command-line option could be clearly marked as obsolete behaviour and could be removed within a year.
> > > 
> > > Cheers,
> > > Thomas
> > > 
> > > 
> > > 
> > > > Parsing all URL in mhttpd to prevent /etc/passwd etc. to be returned is tricky, because people can use escape sequences etc. Therefore I think it is much better to restrict file access 
> > > > on the file system level when opening a file. The only escape there one could have is "..", which can be tested easily. 
> > > > 
> > > > Therefore, I propose to restrict file access to two well-defined directories, which is one system directory and one user directory. The system directory should be defined via 
> > > > $MIDASSYS/resources, and the user directory should be the experiment directory (as defined in exptab) followed by "resources". So if MIDASSYS equals to /usr/local/midas and the 
> > > > experiment directory equals to /home/users/exp for example, we would only have these two directories (and of course the subdirectories within these) served by mhttpd:
> > > > 
> > > > $MIDASSYS/resource -> /usr/local/midas/resources
> > > > <exptab>/resources -> /home/users/exp/resources
> > > > 
> > > > These directories should be hard-wired into mhttpd, and not go through and ODB entry, since otherwise one could manipulate the ODB entries (knowingly or unknowingly) and open a 
> > > > back-door. 
> > > > 
> > > > If users need a more complex structure, they can put soft links into these directories.
> > > > 
> > > > The code which opens a resource file should then first evaluate $MIDASSYS, then add "/resources/", then add the requested file name, make sure that there is no ".." in the file name, 
> > > > then open the file. If not existing, do the same for the <exptab>/resources/ directory.
> > > > 
> > > > This change will break most experiments, and forces people to move their custom pages to different directories, but I think it's the only clean solution and we just have to bite the 
> > > > bullet.
> > > > 
> > > > Comments are welcome.
> > > > 
> > > > Stefan