>    r->rsprintf("Expires: %s\r\n", str);

The best I can tell, none of this works in current browsers. with google-chrome,
I see it cache pretty much everything regardless of "expires", "no cache", etc
and anything else I tried.

Things like shift-<reload>, etc used to work to refresh the cache, but not any more.

So, I too, see confusing side-effects of caching, where I change something in ODB,
but "nothing happens". Then I scratch my head for 30 minutes until I remember
to open the javascript debugger where shift-<reload> (or is it ctrl-<reload>) actually works.

It seems that the only reliable way to bypass the browser cache is to add
a tag with a random number to the URL ("&ts=currenttime").

This is for HTTP GET requests. HTTP POST does not seem to be cached, so I do not worry
about this nonsense for json-rpc requests.

Perhaps we should do this random number trick for all user actions. User can
press buttons only so fast, we should be able to sustain the rate. Anything
loaded automatically or from a timer, we should allow caching.

BTW, things like midas.js are also cached, and it is common to see problems
after updating midas, where status.html is newly loaded, but midas.js is an old
stale version from cache.

Messy.

K.O.

> It seems that the only reliable way to bypass the browser cache is to add
> a tag with a random number to the URL ("&ts=currenttime").

Indeed that's the only reliable way to avoid caching across browsers. An alternative is

("&r=" + Math.random())

to add a random number.


> BTW, things like midas.js are also cached, and it is common to see problems
> after updating midas, where status.html is newly loaded, but midas.js is an old
> stale version from cache.

Reloading JavaScript file NOT from the cache is really tricky these days. I added a
special Google Chrome extension to clear my browser cache, which works reliably:

https://chrome.google.com/webstore/detail/clear-cache/cppjkneekbjaeellbfkmgnhonkkjfpdn

Stefan

Hi,

Using midas version 12-2020,  I am trying to run mhttpd from within a docker container using docker-compose.
Starting from an empty ODB, I simply run `mhttpd` and this is the output I have:
midas_hatfe_1  | <Warning> Starting mhttpd...
midas_hatfe_1  | [mhttpd,INFO] ODB subtree /Runinfo corrected successfully
midas_hatfe_1  | MVOdb::SetMidasStatus: Error: MIDAS db_find_key() at ODB path "/WebServer/Host list" returned status 312
midas_hatfe_1  | Mongoose web server will not use password protection
midas_hatfe_1  | Mongoose web server will not use the hostlist, connections from anywhere will be accepted
midas_hatfe_1  | Mongoose web server listening on http address "localhost:8080", passwords OFF, hostlist OFF
midas_hatfe_1  | [mhttpd,ERROR] [mhttpd.cxx:19160:mongoose_listen,ERROR] Cannot mg_bind address "[::1]:8080"

According to the documentation, the WebServer tree should be created automatically when starting the mhttpd; but it seems not as it doesn't find the entry "/WebServer/Host list".
If I create it by end (using "create STRING /WebServer/Host list"), I still get the error message that mhttpd didn't bind properly to the local port 8080.
I am not sure what it wrong, as mhttpd is working perfectly well in this exact container for midas 03-2020.

Any idea what difference makes it not possible anymore to run into these container?

Thanks very much for your help.
Cheers
Mathieu

> midas_hatfe_1  | Mongoose web server listening on http address "localhost:8080", passwords OFF, hostlist OFF
> midas_hatfe_1  | [mhttpd,ERROR] [mhttpd.cxx:19160:mongoose_listen,ERROR] Cannot mg_bind address "[::1]:8080"

It looks like mhttpd managed to bind to the IPv4 address (localhost), but not the IPv6 address (::1). If you don't need it, try setting "/Webserver/Enable IPv6" to false.

> It looks like mhttpd managed to bind to the IPv4 address (localhost), but not the IPv6 address (::1). If you don't need it, try setting "/Webserver/Enable IPv6" to false.

We had this issue already several times. This info should be put into the documentation at a prominent location.

Stefan

> > It looks like mhttpd managed to bind to the IPv4 address (localhost), but not the IPv6 address (::1). If you don't need it, try setting "/Webserver/Enable IPv6" to false.
> 
> We had this issue already several times. This info should be put into the documentation at a prominent location.
> 
> Stefan

Thanks a lot, this solved my issue!

> Thanks a lot, this solved my issue!

... or we should turn IPv6 off by default, since not many people use this right now.

> > Thanks a lot, this solved my issue!
> 
> ... or we should turn IPv6 off by default, since not many people use this right now.

IPv6 certainly works and is used at CERN.

But I am not sure why people see this message. I do not see it on any machines at 
TRIUMF, even those with IPv6 turned off.

K.O.

Well, we all see it here at PSI, so this is enough reason to turn this off by default. Shall 
I do it?

I guess this might be related to the changes in the last elog conversation; but
I'll break it out as a separate problem.

The new mhttpd ODB editor seems to resize all strings (not just strings that are
greater than 256 characters).  So, when I change some string with the mhttpd ODB
editor to 'ffffff', then I find that the string size is now ~7 characters.

This might be fine in general; but it seems to cause a problem when dealing with
alarms.  In particular, I find that if I try to set (through mhttpd) the
"execute command" for an alarm class or the "condition" for an alarm, then I get
into lots of trouble.  For instance, I changed the "execute command" for my
alarm class through mhttpd; when associated alarms were triggered, I got errors

21:58:12 [feSourceEpics,ERROR] [odb.c:9133:db_get_record,ERROR] struct size
mismatch for "/Alarms/Classes/Alarm" (expected size: 348, size in ODB: 100)
21:58:12 [feSourceEpics,ERROR] [alarm.c:379:al_trigger_class,ERROR] Cannot get
alarm class record

This makes sense, since ALARM_CLASS has a fixed size

typedef struct {
   BOOL write_system_message;
   ...
   char execute_command[256];
   ...
   char display_fgcolor[32];
} ALARM_CLASS;

so problems will clearly occur when I change the size and try to grab it:

   ALARM_CLASS ac;
   status = db_get_record1(hDB, hkeyclass, &ac, &size, 0, strcomb(alarm_class_str));
 
I guess that similar problems also occur if you edit the string for ALARM or
PROGRAM_INFO instances.  These problems do not occur when I change my strings
with odbedit, which doesn't resize strings below 256.

I'm not sure what the proper solution is.  A temporary solution is that the
mhttpd ODB editor shouldn't resize strings if the new size is less than 256
characters; in that case the size should be left as 256 characters.

This test was done with MIDAS git repository as of today:
commit 45a90dc329554f528485da121501daf6ecde100d

To follow up; with some help from Konstantin and Stefan, we realized that this
particular problem should already be fixed.  While I was using the most recent version
of MIDAS, I hadn't rebuild the EPICS frontend programs when I was doing this test.  Once
I did that the error no longer occurred.  This is because the most recent version of
MIDAS includes a check that will resize these particular string variables before using
them (technically, this is included in db_get_record1()); this resizing only happens for
these couple strings that must have a fixed size.

We are still having a separate discussion about whether this treatment of string lengths
that need to have a fixed size can be further improved.  Will update once discussion
converges.


> I guess this might be related to the changes in the last elog conversation; but
> I'll break it out as a separate problem.
> 
> The new mhttpd ODB editor seems to resize all strings (not just strings that are
> greater than 256 characters).  So, when I change some string with the mhttpd ODB
> editor to 'ffffff', then I find that the string size is now ~7 characters.
> 
> This might be fine in general; but it seems to cause a problem when dealing with
> alarms.  In particular, I find that if I try to set (through mhttpd) the
> "execute command" for an alarm class or the "condition" for an alarm, then I get
> into lots of trouble.  For instance, I changed the "execute command" for my
> alarm class through mhttpd; when associated alarms were triggered, I got errors
> 
> 21:58:12 [feSourceEpics,ERROR] [odb.c:9133:db_get_record,ERROR] struct size
> mismatch for "/Alarms/Classes/Alarm" (expected size: 348, size in ODB: 100)
> 21:58:12 [feSourceEpics,ERROR] [alarm.c:379:al_trigger_class,ERROR] Cannot get
> alarm class record
> 
> This makes sense, since ALARM_CLASS has a fixed size
> 
> typedef struct {
>    BOOL write_system_message;
>    ...
>    char execute_command[256];
>    ...
>    char display_fgcolor[32];
> } ALARM_CLASS;
> 
> so problems will clearly occur when I change the size and try to grab it:
> 
>    ALARM_CLASS ac;
>    status = db_get_record1(hDB, hkeyclass, &ac, &size, 0, strcomb(alarm_class_str));
>  
> I guess that similar problems also occur if you edit the string for ALARM or
> PROGRAM_INFO instances.  These problems do not occur when I change my strings
> with odbedit, which doesn't resize strings below 256.
> 
> I'm not sure what the proper solution is.  A temporary solution is that the
> mhttpd ODB editor shouldn't resize strings if the new size is less than 256
> characters; in that case the size should be left as 256 characters.
> 
> This test was done with MIDAS git repository as of today:
> commit 45a90dc329554f528485da121501daf6ecde100d

> 
> Added JSON encoding format to Javascript ODBCopy(path,format) ("jcopy"). Use format="json", Javascript example updated with an example example.
> 

More ODBCopy() expansion: format="json-p" returns data suitable for JSON-P ("script tag") messaging.

Also implemented multiple-paths for "jcopy" (similar to "jget"/ODBMGet()). An example ODBMCopy(paths,callback,format) is present in example.html (will move to mhttpd.js).

Added JSON encoding options:
- format="json-nokeys" will omit all KEY information except for "last_written"
- "json-nokeys-nolastwritten" will also omit "last_written"
- "json-nofollowlinks" will return ODB symlink KEYs instead of following them (ODBGet/ODBMGet always follows symlinks)
- "json-p" adds JSON-P encapsulation
All these JSON format options can be used at the same time, i.e. format="json-p-nofollowlinks"

To see how it all works, please look at examples/javascript1/example.html.

The new code seems to be functional enough, but it is still work in progress and there are a few problems:
- ODBMCopy() using the "xml" format returns gibberish (the MIDAS XML encoder has to be told to omit the <?xml> header)
- example.html does not actually parse any of the XML data, so we do not know if XML encoding is okey
- JSON encoding has an extra layer of objects (variables.Variables.foo instead of variables.foo)
- ODBRpc() with JSON/JSON-P encoding not done yet.

mhttpd.cxx, example.html
svn rev 5364
K.O.

> To see how it all works, please look at examples/javascript1/example.html.
> 
> - JSON encoding has an extra layer of objects (variables.Variables.foo instead of variables.foo)
>

This is now fixed. See updated example.html. Current encoding looks like this:

{
  "System" : {
    "Clients" : {
      "24885" : {
        "Name/key" : { "type" : 12, "item_size" : 32, "last_written" : 1370024816 },
        "Name" : "ODBEdit",
        "Host/key" : { "type" : 12, "item_size" : 256, "last_written" : 1370024816 },
        "Host" : "ladd03.triumf.ca",
        "Hardware type/key" : { "type" : 7, "last_written" : 1370024816 },
        "Hardware type" : 44,
        "Server Port/key" : { "type" : 7, "last_written" : 1370024816 },
        "Server Port" : 52539
      }
    },
    "Tmp" : {
...

odb.c, example.html
svn rev 5368
K.O.

> odbedit can now save ODB in JSON-formatted files.
> Added functions:
>    INT EXPRT db_save_json(HNDLE hDB, HNDLE hKey, const char *file_name);
>    INT EXPRT db_copy_json(HNDLE hDB, HNDLE hKey, char **buffer, int *buffer_size, int *buffer_end, int  save_keys, int follow_links);
> 

Added JSON encoding format to Javascript ODBCopy() ("jcopy"). Use format="json", Javascript example updated with an example example.

Also updated db_copy_json():
- always return NUL-terminated string
- "save_keys" values: 0 - do not save any KEY data, 1 - save all KEY data, 2 - save only KEY.last_written

odb.c, mhttpd.cxx, example.html
svn rev 5362
K.O.

I updated the mhttpd HTTPS/SSL server (mongoose) and https://www.ssllabs.com/ssltest/index.html is 
now more or less happy with it. google chrome connects using "modern cryptography".

The HTTPS/SSL server is activated using "mhttpd --mg" (instead of -p) and it listens on port 8443.

The example SSL certificate provided in midas git is self-signed, for instructions on generating your own 
signed certificate, remove it and run "mhttpd --mg" - it will print the correct instructions.

List of corrected problems:
a) SSL certificate was generated with key length 1024 and SHA1 signature - should be 2048 and SHA256.
b) SSLv2, SSLv3 were not disabled per latest recommendations
c) RC4 and other weak ciphers were not disabled per latest recommendations
d) "modern cryptography" and "forward secrecy" were not available because they require special fondling of 
openssl.
e) on MacOS 10.9 *again* a whole bunch of openssl functions are listed as deprecated with no suggested 
replacement, there is a mismatch between system openssl and macports openssl and "modern 
cryptography" ECDH ciphers are not available.

Also to remember, mhttpd uses the latest release of mongoose 4.2 which is no longer supported by 
author. Latest version of mongoose is 5.x which has a severely improved API, but removed automatic 
multithreading.

I recommend that you use "mhttpd --mg" as the alternative for running "mhttpd -p" behind an apache 
proxy. Using "mhttpd -p" (no HTTPS/SSL) on an internet-connected machine is insecure and should not be 
done. (private network such as 192.168.x.y addresses is okey for now, I guess).

https://bitbucket.org/tmidas/midas/commits/d85ba733573f1fca9946804eeb71d6fdc23bea22

K.O.

> mhttpd uses the latest release of mongoose 4.2 which is no longer supported by 
> author. Latest version of mongoose is 5.x which has a severely improved API, but removed automatic 
> multithreading.

The exact version of mongoose 4.2 included with MIDAS is git revision 607651a3ffce43ef424530b22c7b1d22381de02d from 11 
November 2013.

https://github.com/cesanta/mongoose/commit/607651a3ffce43ef424530b22c7b1d22381de02d.

Documentation for this version of mongoose is committed to midas git repository .../midas/doc/mongoose.

K.O.

> > mhttpd uses the latest release of mongoose 4.2

mhttpd is now explicitly linked with OpenSSL to provide secure https connections via the mongoose web server.

a) google chrome reports "Your connection to ... is encrypted with modern cryptography." via TLS 1.2 and ECDHE ciphers
b) we believe there are no Linux systems that require running mhttpd and lack OpenSSL, but building mhttpd without OpenSSL is supported, see the cross 
compilation section in the Makefile.
c) MacOS comes with a very old version of OpenSSL. mhttpd will build, https will work, but with a complaint about "obsolete cryptography". Please install an up-to-
date OpenSSL package via macports.
d) security of OpenSSL itself is quite problematic, please keep an eye open on OpenSSL security advisories, update OpenSSL and restart/rebuild mhttpd promptly. I 
expect the mongoose project to eventually switch from OpenSSL to one of the new-generation TLS libraries, such as PolarSSL (embed_tls) and we will follow their 
lead.

K.O.

> > > mhttpd uses the latest release of mongoose 4.2

HTTPS support is completely broken in mongoose.c between July 28th (1bc9d8eae48f51ceb73ffd918046cfe74d286909)
and August 12th 2015 (fdc5a80a0a9ca54cba794d7c1131add7f55f112f).

I accidentally broke it by a wrong check against absence of EC_KEY in prehistoric openssl shipped with SL4.

As result, the ECDHE ciphers were enabled but did not work - google chrome complained about "obsolete cryptography",
firefox failed to connect at all.

Please update src/mongoose.c to the latest version if you are using https in mhttpd. (as you should)

Sorry about this problem.

K.O.

> 
> I recommend that you use "mhttpd --mg" as the alternative for running "mhttpd -p" behind an apache 
> proxy. Using "mhttpd -p" (no HTTPS/SSL) on an internet-connected machine is insecure and should not be 
> done. (private network such as 192.168.x.y addresses is okey for now, I guess).

Finally reading through your documentation in detail [1,2].  I find that I don't understand this recommendation to use secure mongoose 
instead of putting mhttpd behind an apache proxy.  I think that it is nice to have secure mhttpd with mongoose as an option, but your 
documentation seems to imply that mhttpd-mongoose is much better than mhttpd-behind-apache and that the latter solution is strongly 
deprecated.

Perhaps I am not understanding the benefits of the new system.  In reference [2] you say "If this is not possible, somewhat better security 
for HTTP is gained by using a password protected SSL (https) proxy."  This seems to imply that the security of mhttpd-mongoose is better 
than the security of mhttpd-behind-apache.  Is that correct?  I thought that they provided similar security (assuming you follow 
recommended configurations for APACHE).

Setting up apache is trivial and it seems that mhttpd-behind-apache has other advantages, like being able to put other web resources 
(ganglia, cameras, elog, etc) behind the same secure server.  Also you can start to build complicated custom pages that are served directly 
from apache and just use MIDAS AJAX calls.  I was imagining slowly moving away from using mhttpd at all and just having html/js/css 
resources served up by apache.

So, unless I'm missing something, at this point I would continue to recommend people use mhttpd-behind-apache and I'd suggest this be 
presented as an equally valid option in the documentation.

[1] https://midas.triumf.ca/MidasWiki/index.php/Mhttpd
[2] https://midas.triumf.ca/MidasWiki/index.php/Setup_MIDAS_experiment#Install_SSL_proxy

Stefan identified a serious multi-thread locking bug in mhttpd that affects the operation of the sequencer (a race condition between db_set_record() and 
db_get_record() inside the hotlink code). This is now fixed. If you use the sequencer, please update mhttpd.cxx to the latest (or to this) version.

https://bitbucket.org/tmidas/midas/commits/9d79218a125a4427d0cc2f2b5e4e56d585655c88
K.O.