For password protection of midas web pages, https is required, good old http 
with passwords transmitted in-the-clear is no longer considered secure. Latest 
recommendation is to run mhttpd behind an industry-standard https proxy, for 
example apache httpd. These proxies provide built-in password protection and 
have integration with certbot to provide automatic renewal of https 
certificates.

That said, for a long time now mhttpd provides native https support through the 
mongoose web server library and the openssl cryptography library.

Unfortunately, for years now, we have been running into trouble with the midas 
build process bombing out due to inconsistent versions and locations of system-
provided and user-installed openssl libraries. Despite our best efforts (and 
through the switch to cmake!) these problems keep coming back and coming back.

Luckily, latest versions of mongoose support the mbedtls cryptography library. I 
have tested it and it works well enough for me to switch the MIDAS default build 
from "openssl if found" to "mbedtls if-asked-for-by-user".

Starting with commit e7b02f9, cmake builds do not look for and do not try to use 
openssl. mhttpd is built without support for https. This is consistent with the 
recommendation to run it behind an apache httpd password protected https proxy.

To enable https support using mbedtls, run "make mbedtls". This will "git clone" 
the mbedtls library and add it to the midas build. mhttpd will be built with 
https support enabled.

To disable mbedtls support, use "make cmake NO_MBEDTLS=1" or run "make 
clean_mbedtls" (this will remove the mbedtls sources from the midas build).

To restore previous use of openssl, set the cmake variable "USE_OPENSSL".

In my test, mhttpd with https through mbedtls and a letsencrypt certificate gain 
a score of "A" from SSLlabs. (very good).

(you have to use progs/mtcproxy to run this test - SSLlabs only probe port 443 
and mtcproxy will forward it to mhttpd port 8443. to build, run "make 
mtcpproxy").

References:
https://github.com/cesanta/mongoose
https://github.com/ARMmbed/mbedtls

K.O.

To add. Using https with either openssl or mbedtls requires obtaining an https certificate. This can be self-
signed, or signed by a higher authority, or issued by the "let's encrypt" project.

mhttpd is looking for this certificate in the file ssl_cert.pem.

If this file does not exist, mhttpd will print the instructions for creating it using openssl (self-signed) or 
using certbot (instantaneously and automatically issued let's encrypt certificate).

The certbot route is recommended:

1) (as root) setup certbot (i.e. see my CentOS and Ubuntu instructions on DAQWiki)
2) (as root) copy /etc/letsencrypt/live/$HOME/fullchain.pem and privkey.pem to $MIDASSYS
3) cat fullchain.pem privkey.pem > ssl_cert.pem
4) start mhttpd, watch the first few lines it prints to confirm it found the right certificate file.

The only missing piece for using this in production is lack of integration
with certbot automatic certificate renewal:

- a script has to run for steps (2) and (3) above
- mhttpd has to tell openssl/mbedtls to reload the certificate file (alternative is to automatically restart 
mhttpd, bad!).

As an alternative, we can wait for the mongoose web server library and for the mbedtls crypto library to "grow" 
certbot-style automatic certificate renewal features. (unavoidable, in my view).

K.O.

I now merged almost all the mhttpd changes from CERN AD-5/ALPHA. Only the code
for mhttpd HTTP:// access control list remains unmerged.

Changes to the history display code merged from ALPHA: 
- add option to show latest values of history variables: "show values of
variables" check box on the history config panel 
- add custom labels for each variables: instead of midas variable name, history
plots would show the text entered into the "label" text area 
- show history errors on the plot: before, if one out of 10 history variables
could not be plotted, nothing was shown at all, now all variables are \
show, those that could not be read with hs_read() show the error code 
- the selection of which variables to plot is alphanumerically sorted (adc11 >
adc9) [this code is not active for standard midas because mlogger sup\
port has not yet been committed] 
- the selection of which variables to plot shows the last variable selected, not
the first one - useful when entering variables from a long list [th\
is code is not active for standard midas because mlogger support has not yet
been committed] 
 
These changes have been extensively tested since last Summer at the AD-5 ALPHA
expt at CERN. 
I could only do minimal testing for this merged code, so if there are any errors, 
they would most likely be merge errors. This new code will be heavily used at
TRIUMF, 
so if any errors got any, we hope to flush them out quickly. 
 
As noted, mlogger support for some of the mhttpd functions is not in standard
midas yet. It will be committed shortly. 

K.O.

You misspelled one ODB entry:

Line 9014:
            sprintf(str, "/History/Display/%s/Label", path);

Line 9028:

            sprintf(str, "/History/Display/%s/Labels", path);
                                                ---^

I wonder how you could have tested that code for 1/2 year without noticing this error.
I fixed and committed it.
 

> You misspelled one ODB entry:
> Line 9014:
>             sprintf(str, "/History/Display/%s/Label", path);
> 
> Line 9028:
>             sprintf(str, "/History/Display/%s/Labels", path);
>                                                 ---^
> 
> I wonder how you could have tested that code for 1/2 year without noticing this error.
> I fixed and committed it.


It turns out that the program was tested as originally committed. With the above
modification, it corrupts ODB - originally, it used the wrong array element size to create
the wrong array. Corrected, it creates the right array with the wrong size, then
subsequent db_set_data_index() happily corrupts ODB.

Fix for mhttpd committed as svn revision 4128.
Fix for ODB corruption committed at svn revision 4129 (also fixes extract_key())


K.O.

Dear all,
      I am trying to make the MIDAS web server for my DAQ project accessible from other machines. In the ODB, I activated the necessary flags:

[local:CYGNUS_RD:S]/WebServer>ls
Enable localhost port           y
localhost port                  8080
localhost port passwords        n
Enable insecure port            y
insecure port                   8081
insecure port passwords         y
insecure port host list         y
Enable https port               y
https port                      8443
https port passwords            y
https port host list            n
Host list
                                localhost
Enable IPv6                     y
Proxy                           
mime.types

Following the instructions on the Wiki I enabled the SSL support. When running mhttpd, I get these messages:

Mongoose web server will use HTTP Digest authentication with realm "CYGNUS_RD" and password file "/home/cygno/DAQ/online/htpasswd.txt"
Mongoose web server will use the hostlist, connections will be accepted only from: localhost
Mongoose web server listening on http address "localhost:8080", passwords OFF, hostlist OFF
Mongoose web server listening on http address "[::1]:8080", passwords OFF, hostlist OFF
Mongoose web server listening on http address "8081", passwords enabled, hostlist enabled
[mhttpd,ERROR] [mhttpd.cxx:19166:mongoose_listen,ERROR] Cannot mg_bind address "[::0]:8081"
Mongoose web server will use https certificate file "/home/cygno/DAQ/online/ssl_cert.pem"
Mongoose web server listening on https address "8443", passwords enabled, hostlist OFF
[mhttpd,ERROR] [mhttpd.cxx:19166:mongoose_listen,ERROR] Cannot mg_bind address "[::0]:8443"

and the server is not accessible from other machines. Any suggestion to solve or better investigate this problem?

Thank you very much,
         Francesco

> Enable IPv6                     y

Probably the IPv6 problem, see here elog:2269

I asked to turn off IPv6 by default, or at least mention this in the documentation,
but unfortunately nothing happened.

Stefan

It worked, thank you very much!

Francesco

> > Enable IPv6                     y
> 
> Probably the IPv6 problem, see here elog:2269
> 
> I asked to turn off IPv6 by default, or at least mention this in the documentation,
> but unfortunately nothing happened.
> 
> Stefan

> > Enable IPv6                     y
> 
> Probably the IPv6 problem, see here elog:2269
> 
> I asked to turn off IPv6 by default, or at least mention this in the documentation,
> but unfortunately nothing happened.

But IPv4 and IPv6 code is completely separate, if IPv6 bind fails, IPv4 should still 
work.

This is all very strange.

It does not help that the OP does not say in which way things do not work,
"the server is not accessible from other machines" is not an error message
reported by any browser, and we do not know what URL he is using
to access mhttpd - http: or https:

Also he is enabling the "insecure" port 8081, I am pretty sure the documentation
is pretty clear, either use the secure https port or the insecure port,
but not both at the same time.

In any case, I see current version of mongoose have removed support
for password files, so all this stuff will likely become reworked
and at the end mhttpd will only listen to localhost ports. To make it "accessible
to other machines", one will have to use the apache https proxy. (or mtpcproxy from 
midas).

K.O.

Hi,

I like the new eqtable, but stumbled over some issues.

1) In the attached snapshot you see that the values shown from our vacuum Pirani and Penning cells are all zero, which of course is not true.
It would be nice to have under the equipment settings some formatting options, like the possibility to add units.

2) If one of the number evaluates to Infinity, the table is not shown properly anymore.

Best,
  Andreas

> 1) In the attached snapshot you see that the values shown from our vacuum Pirani and Penning cells are all zero, which of course is not true.
> It would be nice to have under the equipment settings some formatting options, like the possibility to add units.

You have a 

  /Equipment/LEMVAC/Settings/Format Input

array where you can specify the format for every value. Default is "%f2" for two digits after the period. For vacuum levels you might want to 
consider "%e3" which give you exponential format with three significant digits. The "format" setting is described at

  https://daq00.triumf.ca/MidasWiki/index.php//Equipment_ODB_tree#Format_%3Cvariable%3E

and the details are at

  https://daq00.triumf.ca/MidasWiki/index.php/Custom_Page#Formatting

The was a bug with the format handling, so please pull the current develop branch.


> 2) If one of the number evaluates to Infinity, the table is not shown properly anymore.

I fixed that as well in the current version.

Best,
Stefan

Great! This is it.
Sorry that I missed it in the docu.

Best,
  Andreas

I have two more questions related to Units, Format for Equipment/Settings:

1) It looks as if I can have units per channel only for the Input/Output channels but not for Demand/Measured channels. 
For instance we do have HV FE which collect devices with kV and V demand settings. It looks like this is not possible (see attachment) to have per channel units.
Is this right, or do I miss something here?

2) This new functionality needs entries under /Equipment/<eq-name>/Settings. The class driver generates the necessary structures if they are missing at the startup
of the scfe. It would be nice that the new, additional entries would be generate as well: Editable, Unit Input, Unit Format, etc. Perhaps optionally, if a DD is providing it?

Best,
  Andreas 

I fixed both in the current version, so please give it a try.

Stefan

Aparently the mhttpd elog will corrupt the elog files if two (or more?) elog entries are being edited at the 
same time. K.O.

[quote="Stefan Ritt"][Quote="K.O.]Aparently the mhttpd elog will corrupt the
elog files if two (or more\?) elog entries are being edited at the same time.
K.O.[/quote]

The corruption is very simple. mhttpd elog indexes the elog entries by the elog
file and offset inside the file, i.e. "http://ladd00:8088/EL/060927.318",
"060927" corresponds to log file "060927.log", "318" is the offset inside the
file where the message is located.

During "edit", the code "remembers" the offset of the original message and in
el_submit() blindly writes the edited message into the file at the remembered
offset.

If another message was edited before the edit of the first message is submitted,
the remembered offset becomes invalid (messages have shifted inside the file)
and el_submit() writes the edited text into the wrong place in the file,
corrupting it.

I have now added a check for this and we crash instead of corrupting the elog
file (midas.c rev 3340).

I do not know how to "properly" fix this bug without changing the indexing
scheme to something similar to what is used by elogd- message numbers instead of
file indices. In the existing scheme, message editing also breaks URLs shown in
the email notifications (they contain file indices that point to the wrong
places after messages are moved around by editing) and "reply threading" links.

Here is how I reproduce this bug:

1) start with an empty elog
2) create two messages
3) "edit" the second message, but do not submit it yet.
4) "edit" the first message, change the text to make sure the message size
becomes different; submit this change.
5) submit the "edit" of the first message. !!BOOM!!

K.O.

> I do not know how to "properly" fix this bug without changing the indexing
> scheme to something similar to what is used by elogd- message numbers instead of
> file indices. In the existing scheme, message editing also breaks URLs shown in
> the email notifications (they contain file indices that point to the wrong
> places after messages are moved around by editing) and "reply threading" links.

Well, the development of elogd with it's message numbers was actually stimulated by
the problem you mentioned. After that all those problems went away. Another
incarnation of that problem is if you edit an mhttpd log file manually. Afterwards
the file offsets are different and the system gets corrupted. To fix this properly,
one would have to backport the el_xxx functions from elogd to mhttpd, or, even
simpler, remove the elog functionality in mhttpd and "force" everybody to use elogd
(after doing elconv to convert the files into the new format).

Hi,
the subject line describes the project already
Suppose you have a file foo.msl. Somewhere in the file, you have the line 
INCLUDE bar.msl

Once you click save in the sequencer page, mhttpd crashes:
$ mhttpd
free(): double free detected in tcache 2
[1]    27590 abort (core dumped)  mhttpd


GDB helps shed some light on the problem:

#0  0x00007ffff76b057f in raise () from /lib64/libc.so.6
#1  0x00007ffff769a895 in abort () from /lib64/libc.so.6
#2  0x00007ffff76f39d7 in __libc_message () from /lib64/libc.so.6
#3  0x00007ffff76fa2ec in malloc_printerr () from /lib64/libc.so.6
#4  0x00007ffff76fbdf5 in _int_free () from /lib64/libc.so.6
#5  0x00000000004b8b41 in mxml_parse_entity (buf=buf@entry=0x7fffffffc2c8,
file_name=file_name@entry=0x7fffffffc710
"/home/luk/packages/mutrig_daq/online/foo.xml",
            error=error@entry=0x7fffffffcd24 "XML read error in file
\"/home/luk/packages/mutrig_daq/online/foo.xml\", line 2: bar.msl.xml is
missing", error_size=error_size@entry=256,
                error_line=error_line@entry=0x7fffffffce24) at ../mxml/mxml.c:1996
#6  0x00000000004b966d in mxml_parse_file
(file_name=file_name@entry=0x7fffffffc710
"/home/luk/packages/mutrig_daq/online/foo.xml",
            error=error@entry=0x7fffffffcd24 "XML read error in file
\"/home/luk/packages/mutrig_daq/online/foo.xml\", line 2: bar.msl.xml is
missing", error_size=error_size@entry=256,
                error_line=error_line@entry=0x7fffffffce24) at ../mxml/mxml.c:2041
#7  0x000000000041d9c2 in init_sequencer () at src/mhttpd.cxx:14321
#8  0x000000000040c2b6 in main (argc=<optimized out>, argv=<optimized out>) at
src/mhttpd.cxx:18028

Cheers
Lukas

P. S. This problem reminds me of the old joke: A man goes to his doctor and says
"Doc, it hurts when I do this" to which the doctor replies "Then don't do that".
However, I think, mhttpd should not crash even if you're not supposed to include
non-existent scripts in msequencer.

The bug has been fixed. It was actually in the mxml library. So you have to go to the midas/mxml 
subdirectory and update that one via "git pull origin master".

Stefan

> Hi,
> the subject line describes the project already
> Suppose you have a file foo.msl. Somewhere in the file, you have the line 
> INCLUDE bar.msl
> 
> Once you click save in the sequencer page, mhttpd crashes:
> $ mhttpd
> free(): double free detected in tcache 2
> [1]    27590 abort (core dumped)  mhttpd
> 
> 
> GDB helps shed some light on the problem:
> 
> #0  0x00007ffff76b057f in raise () from /lib64/libc.so.6
> #1  0x00007ffff769a895 in abort () from /lib64/libc.so.6
> #2  0x00007ffff76f39d7 in __libc_message () from /lib64/libc.so.6
> #3  0x00007ffff76fa2ec in malloc_printerr () from /lib64/libc.so.6
> #4  0x00007ffff76fbdf5 in _int_free () from /lib64/libc.so.6
> #5  0x00000000004b8b41 in mxml_parse_entity (buf=buf@entry=0x7fffffffc2c8,
> file_name=file_name@entry=0x7fffffffc710
> "/home/luk/packages/mutrig_daq/online/foo.xml",
>             error=error@entry=0x7fffffffcd24 "XML read error in file
> \"/home/luk/packages/mutrig_daq/online/foo.xml\", line 2: bar.msl.xml is
> missing", error_size=error_size@entry=256,
>                 error_line=error_line@entry=0x7fffffffce24) at ../mxml/mxml.c:1996
> #6  0x00000000004b966d in mxml_parse_file
> (file_name=file_name@entry=0x7fffffffc710
> "/home/luk/packages/mutrig_daq/online/foo.xml",
>             error=error@entry=0x7fffffffcd24 "XML read error in file
> \"/home/luk/packages/mutrig_daq/online/foo.xml\", line 2: bar.msl.xml is
> missing", error_size=error_size@entry=256,
>                 error_line=error_line@entry=0x7fffffffce24) at ../mxml/mxml.c:2041
> #7  0x000000000041d9c2 in init_sequencer () at src/mhttpd.cxx:14321
> #8  0x000000000040c2b6 in main (argc=<optimized out>, argv=<optimized out>) at
> src/mhttpd.cxx:18028
> 
> Cheers
> Lukas
> 
> P. S. This problem reminds me of the old joke: A man goes to his doctor and says
> "Doc, it hurts when I do this" to which the doctor replies "Then don't do that".
> However, I think, mhttpd should not crash even if you're not supposed to include
> non-existent scripts in msequencer.