Back Midas Rome Roody Rootana
  Midas DAQ System, Page 34 of 142  Not logged in ELOG logo
ID Date Author Topic Subjectdown
  1103   27 Aug 2015 Konstantin OlchanskiInfomhttpd HTTPS/SSL server updated
> 
> I find that I don't understand this recommendation to use secure mongoose 
> instead of putting mhttpd behind an apache proxy.
>

This is a very valid question.

I think for a small operation that does not require root access to the host computer, mhttpd+mongoose is a good light weight solution.

For a more elaborate setup with private networks, etc, apache https proxy is probably better - for big experiments, resources like webcams,
ganglia, couchdb, etc also need password protection and apache https is the one stone to kill all birds. (one bird to kill all pigs).

Which one is easier to setup?

mhttpd+mongoose I tried to make simple - you have to create a password file and (optional) a properly signed https certificate.
apache httpd is fairly straightforward if you follow well written instructions (such as we provide for using it with midas). but you do need root access
and you do have to edit a good number of config files.

Which one is secure?

By one definition - will it pass muster with central IT - only apache httpd is secure.

At CERN all we have to say "we use password protected apache httpd HTTPS proxy" and they say "ok!".

If we were to say "we use custom web server based on some strange version of mongoose, customized", they will probably raise some eyebrows.

And keeping central IT happy is important if you want holes in their firewall for off-site access to MIDAS.

Now, which one is secure?

The default distribution of apache httpd in SL6 is insecure. period. some steps to secure it are non-controversial - disable SSLv2, SSLv3, disable RC4 ciphers. This is not enough to pass 
muster with the SSLlabs scanner. One should also disable some obsolete and known-weak ciphers. This will disable some old web browsers and is a more controversial step. (see the SSLlabs 
reports).

The default distribution of mhttpd+mongoose passes muster with SSLlabs and on the strength of that I deem it "secure out of the box". One can suggest alternative security tools and one 
can/should run the SSLlabs scanner against mhttpd after each update, report problems as a bug to midas.

Now, which one is *secure*?

Both apache httpd and mongoose are based on OpenSSL which has been recently demonstrated to be severely insecure. (look into the OpenBSD fork of OpenSSL).

There are alternative HTTPS libraries, such as PolarSSL, which are intended for embedding into other applications and devices - such as into mhttpd or into MSCB-ethernet boxes -
and I hope mongoose/fossa will make the switch by the end of the year. (a compatibility layer for using mongoose with PolarSSL already exists).

So, which one to use?

- for maximum security, use httpd apache (but remember to restrict access to mhttpd web port to be "only from the proxy")
- for light-weight cases, or when root access is not available use built-in https in mhttpd.

The midas wiki documentation should probably be updated to explain all of this.

K.O.
  1104   31 Aug 2015 Konstantin OlchanskiInfomhttpd HTTPS/SSL server updated
Configuration of web server completely changed (merge of branch feature/mongoose-config2). Hopefully for the last time.

mhttpd is now controlled by these ODB variables:

/experiment/
.../midas http port	-> 8080 (0x1F90)
.../midas https port	-> 8443 (0x20FB)
.../http redirect to https -> "y"

the names are self obvious (hopefully)

access control is done by the "-a" command line arguments
and by the access control list in ODB, which works the same way
as the RPC ports access control list. An empty list means free access
from everywhere:

/experiment/security/mhttpd hosts/allowed hosts

the access control list is watched by httpd, there is no need to restarted it after updating the list.

after changing the port number settings, mhttpd should be restarted.

other web control options to mhttpd are:

daq06:midas$ ./linux/bin/mhttpd -h
usage: ./linux/bin/mhttpd [-h Hostname[:port]] [-e Experiment] [-v] [-D] [-a Hostname]
       -a only allow access for specific host(s), several [-a Hostname] statements might be given (default list is ODB "/Experiment/security/mhttpd hosts/allowed hosts")
       --http port - bind to specified HTTP port (default is ODB "/Experiment/midas http port")
       --https port - bind to specified HTTP port (default is ODB "/Experiment/midas https port")
       --nomg use the old mhttpd web server
       --oldserver [port] - use the old web server on given port
       --nooldserver - do not use the old mhttpd web server

To run mhttpd "the old way" (mhttpd -D -p 8080), say "mhttpd --oldserver 8080 --nomg".

The normal way to run mhttpd is: "mhttpd -v" to get debug information and "mhttpd -D" to run in the background.

K.O.
  1107   09 Sep 2015 Thomas LindnerInfomhttpd HTTPS/SSL server updated
> > 
> > I find that I don't understand this recommendation to use secure mongoose 
> > instead of putting mhttpd behind an apache proxy.
> >
> 
> This is a very valid question.
> 
> I think for a small operation that does not require root access to the host computer, mhttpd+mongoose is a good light weight solution.
> ...
> So, which one to use?
> 
> - for maximum security, use httpd apache (but remember to restrict access to mhttpd web port to be "only from the proxy")
> - for light-weight cases, or when root access is not available use built-in https in mhttpd.
> 
> The midas wiki documentation should probably be updated to explain all of this.


Thanks for the detailed explanation.  I agree with your recommendations.  I was mostly interested in having both options treated equally in the documentation.

My only small complaint is that since the default mhttpd comes with mongoose security turned on, you need to explicitly disable the mhttpd+mongoose security first before you can start setting up apache.  I guess that the motivation is 
that we should force people to disable security, rather than hoping that they will enable it.  That's a convincing argument; so all I really need is that this procedure be well documented.
  1110   11 Sep 2015 Konstantin OlchanskiInfomhttpd HTTPS/SSL server updated
> 
> Thanks for the detailed explanation.  I agree with your recommendations.  I was mostly interested in having both options treated equally in the documentation.
> 

I did not review the documentation yet, so it is most likely completely wrong.

But in the nutshell, we should document 2 configurations:

1) standalone mhttpd - with built-in https and password protection
2) mhttpd behind a password-protected https proxy (apache httpd) - mhttpd will have https and built-in passwords disabled, http access restricted to localhost (or the host of the httpd, if they are not the same - as at CERN/ALPHA).

K.O.
  1343   01 Mar 2018 Andreas SuterBug Reportmhttpd / odb set strings -> truncates odb entry
There is a bug in the string handling when changing ODB string entries via the
mhttpd (git sha 07dfb83). It truncates the string length in the ODB.

For instance I create a string with length 32 and set it with odbedit to 'a'.
Then the string length stays 32, as expected. If the same is done through the
web-interface, the string length will be truncated to 2.

This can lead to problems if some frontend has a hotlink to a structure
containing this string since it will complain about structure size mismatch. 
  1344   02 Mar 2018 Stefan RittBug Reportmhttpd / odb set strings -> truncates odb entry
> There is a bug in the string handling when changing ODB string entries via the
> mhttpd (git sha 07dfb83). It truncates the string length in the ODB.
> 
> For instance I create a string with length 32 and set it with odbedit to 'a'.
> Then the string length stays 32, as expected. If the same is done through the
> web-interface, the string length will be truncated to 2.
> 
> This can lead to problems if some frontend has a hotlink to a structure
> containing this string since it will complain about structure size mismatch. 

I know about this problem since last summer. I mentioned it to KO, since it's deep down in his 
JSONRPC code. We had a long discussion, where he kind of insisted that this is not a bug but a 
feature. The ODB should store strings with variable lengths, and thus adapt it according to the 
current string length. This makes some sense, since in the future we plan to put C++ string 
support for the ODB, where strings have dynamically varying lengths. But this will take a while, so 
I asked KO to change the truncation of the strings though the web interface, because this breaks 
many experiments. He did not react so far. Several people complained. Maybe your request will 
help now.

Stefan
  1345   05 Mar 2018 Andreas SuterBug Reportmhttpd / odb set strings -> truncates odb entry
> > There is a bug in the string handling when changing ODB string entries via the
> > mhttpd (git sha 07dfb83). It truncates the string length in the ODB.
> > 
> > For instance I create a string with length 32 and set it with odbedit to 'a'.
> > Then the string length stays 32, as expected. If the same is done through the
> > web-interface, the string length will be truncated to 2.
> > 
> > This can lead to problems if some frontend has a hotlink to a structure
> > containing this string since it will complain about structure size mismatch. 
> 
> I know about this problem since last summer. I mentioned it to KO, since it's deep down in his 
> JSONRPC code. We had a long discussion, where he kind of insisted that this is not a bug but a 
> feature. The ODB should store strings with variable lengths, and thus adapt it according to the 
> current string length. This makes some sense, since in the future we plan to put C++ string 
> support for the ODB, where strings have dynamically varying lengths. But this will take a while, so 
> I asked KO to change the truncation of the strings though the web interface, because this breaks 
> many experiments. He did not react so far. Several people complained. Maybe your request will 
> help now.
> 
> Stefan

Well I appreciate the direction towards more C++ string handling, yet it must not break the hotlink
functionality which is very important at many places.
  1364   18 Apr 2018 Thomas LindnerBug Fixmhttpd / odb set strings -> truncates odb entry
I wanted to try to summarize the current situation with regards to the handling of strings through the MIDAS web interface.

During the last year, we started the switch-over to using the new mjson-rpc functions in the MIDAS webpages.  As part of this work, changes were made that allowed for 
resizing strings through the MIDAS web interface (specifically through the MJSON-RPC calls). This reflected desires from some users that strings could be allowed to 
grow larger than the default 256 size that is usually used for MIDAS strings (for instance, see [1]).  In this first set of changes the ODB strings would always be resized to 
the exact size of the string that it was set to.

The problem with this change was that it breaks the behaviour of db_get_record(), which typically expects strings to be a fixed length of 32 or 256 (for instance, see 
[2,3]).  Konstantin notes that we can work around this problem by using db_get_record1() function, which automatically resizes strings to expected values; this method 
has already been used in the MIDAS core code.  But the problem would still remain for some user code that uses db_get_record.

As a short-term compromise solution, Stefan implemented a change where the MJSON-RPC string setting will now expand the size of strings, but not truncate them; ie a 
string that is size 256 will stay 256 bytes, unless you set it to something larger.  So this should fix most cases of user code that call db_get_record() and hence expects 
fixed string lengths. Users who call that db_get_record with strings that exceed the expected length will still have problems; but now you will at least get an explicit 
MIDAS error message, rather than just silent string truncation (as was the case before).

In the longer run we still want to develop a new set of ODB methods that more naturally support C++-style variable-length strings. But that's a discussion for the longer 
term.

[1] https://midas.triumf.ca/elog/Midas/1150
[2] https://bitbucket.org/tmidas/midas/issues/121/odb-inline-editor-truncates-stings
[3] https://midas.triumf.ca/elog/Midas/1343
  1351   12 Mar 2018 Andreas SuterForummhttpd / javascript - simple check if a client is running
Is there a simple way from the javascript side to check if a fontend is running?
Currently one would need to go through the /System/Client list to find out if a
frontend/client is running. Wouldn't it be nice to have this centralized, either
in the mhttpd.cxx or mhttpd.js part? 
  1353   13 Mar 2018 Thomas LindnerForummhttpd / javascript - simple check if a client is running
> Is there a simple way from the javascript side to check if a fontend is running?
> Currently one would need to go through the /System/Client list to find out if a
> frontend/client is running. Wouldn't it be nice to have this centralized, either
> in the mhttpd.cxx or mhttpd.js part? 

Hi,

I think that this option already exists with the cm_exist method for the mjsonrpc calls.  For instance, you can use a 
call like 

curl -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","id":null,"method":"cm_exist","params":
{"name":"Logger"}}' 'http://localhost:8081?mjsonrpc'

to get the status of the logger program.  There is a description of the cm_exist parameters on this page:

https://midas.triumf.ca/MidasWiki/index.php/Mjsonrpc
  1354   13 Mar 2018 Andreas SuterForummhttpd / javascript - simple check if a client is running
> > Is there a simple way from the javascript side to check if a fontend is running?
> > Currently one would need to go through the /System/Client list to find out if a
> > frontend/client is running. Wouldn't it be nice to have this centralized, either
> > in the mhttpd.cxx or mhttpd.js part? 
> 
> Hi,
> 
> I think that this option already exists with the cm_exist method for the mjsonrpc calls.  For instance, you can use a 
> call like 
> 
> curl -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","id":null,"method":"cm_exist","params":
> {"name":"Logger"}}' 'http://localhost:8081?mjsonrpc'
> 
> to get the status of the logger program.  There is a description of the cm_exist parameters on this page:
> 
> https://midas.triumf.ca/MidasWiki/index.php/Mjsonrpc

Thanks a lot for the info. I just simply missed it :-|
  1291   09 May 2017 Andreas SuterBug Reportmhttpd / history / export data
A handy feature of the history of the mhttpd is to export the data. However, this 
seems to be broken. It currently only works if the run marker flag is activated by 
fails otherwise.
  1292   16 May 2017 Konstantin OlchanskiBug Reportmhttpd / history / export data
> A handy feature of the history of the mhttpd is to export the data. However, this 
> seems to be broken. It currently only works if the run marker flag is activated by 
> fails otherwise.

imo, it never worked properly. I think the best hope for working "export" button
is an "export as json" which gives you basically the output of hs_read_buffer() in the json
format. With options for "raw data" or "binned, with mean, rms, min, max for each bin".

K.O.
  1499   18 Mar 2019 Andreas SuterBug Reportmhttpd - slowcontrol frontend - multi class driver
When using a slowcontrol frontend which operates a device using the multi class
driver the current midas version (ec3225902d6) has the following issue:

There is a row labeled with: All Input Output

So far by clicking e.g. on Input, only the Input related part was displayed, etc.

Currently this leads to the following error message: 

Error: cannot find key LS336/Variables/Input

where LS336 is my DD.
  1501   18 Mar 2019 Konstantin OlchanskiBug Reportmhttpd - slowcontrol frontend - multi class driver
> When using a slowcontrol frontend which operates a device using the multi class
> driver the current midas version (ec3225902d6) has the following issue:
> 
> There is a row labeled with: All Input Output
> 

This is the "slow control" page you enter by clicking on the equipment name
on the midas status page, yes?

There is some kid of logic behind that link that send one either to the ODB page
for /Eq/XXX/Variables or to the "slow control" page that displays
/Eq/XXX/Variables in a table.

I just tried it and I cannot get to this page in my test experiment, so it will take
a few minutes for me to reproduce your problem. But I think I know where the breakage
is - with the new URL scheme, the links for selecting which variable to show either
was not converted to the new URL scheme or there is a bug and it send us to the wrong place.

I do not remember testing that code, so I will take a look at it definitely.

> Currently this leads to the following error message: 
> Error: cannot find key LS336/Variables/Input

Looks like the /Equipment part is missing from the ODB path...

K.O.
  1506   25 Mar 2019 Konstantin OlchanskiBug Reportmhttpd - slowcontrol frontend - multi class driver
Fixed in https://bitbucket.org/tmidas/midas/commits/e2c4871026121ed1cc44a69b9e3e2d428a6c84d1

The link was pointing to the wrong place - going to ODB instead of staying on the same page.

K.O.



> > When using a slowcontrol frontend which operates a device using the multi class
> > driver the current midas version (ec3225902d6) has the following issue:
> > 
> > There is a row labeled with: All Input Output
> > 
> 
> This is the "slow control" page you enter by clicking on the equipment name
> on the midas status page, yes?
> 
> There is some kid of logic behind that link that send one either to the ODB page
> for /Eq/XXX/Variables or to the "slow control" page that displays
> /Eq/XXX/Variables in a table.
> 
> I just tried it and I cannot get to this page in my test experiment, so it will take
> a few minutes for me to reproduce your problem. But I think I know where the breakage
> is - with the new URL scheme, the links for selecting which variable to show either
> was not converted to the new URL scheme or there is a bug and it send us to the wrong place.
> 
> I do not remember testing that code, so I will take a look at it definitely.
> 
> > Currently this leads to the following error message: 
> > Error: cannot find key LS336/Variables/Input
> 
> Looks like the /Equipment part is missing from the ODB path...
> 
> K.O.
  1335   10 Jan 2018 Andreas SuterBug Reportmhttpd - custom page - RHEL/Fedora
Description of the problem (starting with 61be7a1):

When starting a new experiment, creating a fresh ODB and than adding the 
directory '/Custom', the mhttpd runs into a problem on RHEL/Fedora, but not on 
Ubuntu and macOS. When trying to open the ODB from within whatever browser I get 
the following error message in the midas message queque:

[mhttpd,ERROR] [mhttpd.cxx:563:rread,ERROR] Cannot read file '/root', read of 
4096 returned -1, errno 21 (Is a directory)

and in the browser I get a popup which tries to save a file called 'root'.

I track this down to the following: in mhttpd, interprete (line 18046) it is 
check if a custom page file exists (ss_file_exist) and if yes, it tries to 'load' 
it. Now, at this stage the variable dec_path contains '/root'.

Here now what goes wrong: ss_file_exist tries to open the given path, and if a 
valid file descriptor is returned it assumes the file exists. This is not 
perfectly correct since it also will get a valid file descriptor is path is an 
accessible directory!

Now for whatever reason, on RHEL/Fedora '/root' will return a valid file 
descriptor, but not on macOS and Ubuntu. Others I haven't tested. A possible fix 
would be to check explicitly if path is a directory and if yes return 0 in 
ss_file_exist (see attached diff).

Perhaps there is cleaner way to deal with this issue?! 
Attachment 1: system.c.diff
diff --git a/src/system.c b/src/system.c
index bdeb847..2581247 100755
--- a/src/system.c
+++ b/src/system.c
@@ -6274,6 +6274,13 @@ int ss_file_exist(const char *path)
  
  \********************************************************************/
 {
+   // first check if path is a directory and if yes, return 0
+   struct stat buf;
+   stat(path, &buf);
+   if (S_ISDIR(buf.st_mode))
+      return 0;
+
+   // check if the file exists
    int fd = open(path, O_RDONLY, 0);
    if (fd < 0)
       return 0;
  1336   11 Jan 2018 Konstantin OlchanskiBug Reportmhttpd - custom page - RHEL/Fedora
> [mhttpd,ERROR] [mhttpd.cxx:563:rread,ERROR] Cannot read file '/root', read of 
> 4096 returned -1, errno 21 (Is a directory)

On some linux systems, "/root" exists, it is a directory used as the home directory 
of user "root" (~root is /root; traditional UNIX has ~root as /).

open() of a directory succeeds on some UNIX systems, on some of them,
read() also works, but on other systems one is supposed
to use opendir() and readdir().

MacOS is of course a BSD system (not SysV like Solaris, not Linux), so things
are different yet again. I think MacOS does not have a /root.

In any case, IMO, mhttpd has no business serving the contents of /root,
or serving any files outside of the mhttpd user $HOME directory. (but also
should not serve files from ~user/.ssh, or any other "secret" files, good
luck making a complete axhuastive list of all secret files that should not be
served).

K.O.


> 
> and in the browser I get a popup which tries to save a file called 'root'.
> 
> I track this down to the following: in mhttpd, interprete (line 18046) it is 
> check if a custom page file exists (ss_file_exist) and if yes, it tries to 'load' 
> it. Now, at this stage the variable dec_path contains '/root'.
> 
> Here now what goes wrong: ss_file_exist tries to open the given path, and if a 
> valid file descriptor is returned it assumes the file exists. This is not 
> perfectly correct since it also will get a valid file descriptor is path is an 
> accessible directory!
> 
> Now for whatever reason, on RHEL/Fedora '/root' will return a valid file 
> descriptor, but not on macOS and Ubuntu. Others I haven't tested. A possible fix 
> would be to check explicitly if path is a directory and if yes return 0 in 
> ss_file_exist (see attached diff).
> 
> Perhaps there is cleaner way to deal with this issue?! 
  1337   12 Jan 2018 Stefan RittBug Reportmhttpd - custom page - RHEL/Fedora
> In any case, IMO, mhttpd has no business serving the contents of /root,
> or serving any files outside of the mhttpd user $HOME directory. (but also
> should not serve files from ~user/.ssh, or any other "secret" files, good
> luck making a complete axhuastive list of all secret files that should not be
> served).

I fully agree with Konstantin. mhttpd should only serve files under certain directories. One is the 
midas/resources directory, another is the one defined in the ODB under /Custom/Path. I plan to modify 
mhttpd to only serve these files (and also prevent tricks like putting "../../../" into the URL). This will then 
also fix Andreas' problem.

Stefan
  Draft   15 Jan 2018 Andreas SuterBug Reportmhttpd - custom page - RHEL/Fedora
> > In any case, IMO, mhttpd has no business serving the contents of /root,
> > or serving any files outside of the mhttpd user $HOME directory. (but also
> > should not serve files from ~user/.ssh, or any other "secret" files, good
> > luck making a complete axhuastive list of all secret files that should not be
> > served).
> 
> I fully agree with Konstantin. mhttpd should only serve files under certain directories. One is the 
> midas/resources directory, another is the one defined in the ODB under /Custom/Path. I plan to modify 
> mhttpd to only serve these files (and also prevent tricks like putting "../../../" into the URL). This will then 
> also fix Andreas' problem.
> 
> Stefan

Yes, I think this is a good idea as well. But just to make sure that you get the problem: it's not the custom page call by itself! If you currently type e.g. <midas-server>/root it will open the ODB html-page.
ELOG V3.1.4-2e1708b5