Back Midas Rome Roody Rootana
  Midas DAQ System, Page 31 of 139  Not logged in ELOG logo
New entries since:Wed Dec 31 16:00:00 1969
ID Date Author Topicdown Subject
  1077   24 Jul 2015 Konstantin OlchanskiInfoMAX_EVENT_SIZE removed
The define for MAX_EVENT_SIZE was removed from midas.h.

Replacing it is DEFAULT_MAX_EVENT_SIZE set to 4 MiBytes and DEFAULT_BUFFER_SIZE 
set to 32 MiBytes.

For a long time now MIDAS does not have hardcoded maximum event size and buffer size 
and this change merely renames the define to reflect it's current function.

The actual maximum event size is set by ODB /Experiment/MAX_EVENT_SIZE.

The actual event buffer sizes are set by ODB "/Experiment/Buffer sizes/SYSTEM" & co

K.O.
  Draft   24 Jul 2015 Konstantin OlchanskiInfoPlans for improving midas network security
There is a number of problems with network security in midas. (as separate from web/http/https security).

1) too many network sockets are unnecessarily bound to the external network interface instead of localhost (UDP ports are already bound to localhost on MacOS).
2) by default the RPC ports of each midas program accept connections and RPC commands from anywhere in the world (an access control list is already implemented via /Experiment/Security/Rpc Hosts, but not active by default)
3) mserver also has an access control list but it is not integrated with the access control list for the RPC ports.
4) it is difficult to run midas in the presence of firewalls (midas programs listen on random network ports - cannot be easily added to firewall rules)

There is a new git branch "feature/rpcsecurity" where I am addressing some of these problems:

1) UDP sockets are only used for internal communications (hotlinks & etc) within one machine, so they should be bound to the localhost address and become invisible to external machines. This change breaks binary compatibility from old clients - they are have to be relinked with the new midas library or hotlinks & etc will stop working. If some clients cannot be rebuild (I have one like this), I am preserving the old way by checking for a special file in the experiment directory (same place as ODB.SHM). (done)

2) if one runs on a single machine, does not use the mserver and does not have clients running on other machines, then all the RPC ports can be bound to localhost. (this kills the MacOS popups about "odbedit wants to connect to the Internet"). (partially done)

This (2) will become the new default - out of the box, midas will not listen to any external network connections - making it very secure.

To use the mserver, one will have to change the ODB setting "/Experiment/Security/Enable external RPC connections" and restart all midas programs (I am looking for a better name for this odb setting).

3) the out-of-the-box default access control list for RPC connections will be set to "localhost", which will reject all external connections, even when they are enabled by (2). One will be required to enter the names of all machines that will run midas clients in "/Experiment/Security/Rpc hosts". (already implemented in main midas, but default access control list is empty meaning everybody is permitted)

4) the mserver will be required to attach to some experiment and will use this same access control list to restrict access to the main mserver listener port. Right now the mserver listens on this port without attaching to any experiment and accepts the access control list via command line arguments. I think after this change a single mserver will still be able to service multiple experiments (TBD).

5) I am adding an option to fix TCP port numbers for MIDAS programs via "/Experiment/Security/Rpc ports/fename = (int)5555". Once a remote frontend is bound to a fixed port, appropriate openings can be made in the firewall, etc. Default port number value will be 0 meaning "use random port", same as now.

One problem remains with initial connecting to the mserver. The client connects to the main mserver listener port (easy to firewall), but then the mserver connects back to the client - this reverse connection is difficult to firewall and this handshaking is difficult to fix in the midas sources. It will probably remain unresolved for now.

K.O.
  1079   24 Jul 2015 Konstantin OlchanskiInfoPlans for improving midas network security
There is a number of problems with network security in midas. (as separate from 
web/http/https security).

1) too many network sockets are unnecessarily bound to the external network interface 
instead of localhost (UDP ports are already bound to localhost on MacOS).
2) by default the RPC ports of each midas program accept connections and RPC commands 
from anywhere in the world (an access control list is already implemented via 
/Experiment/Security/Rpc Hosts, but not active by default)
3) mserver also has an access control list but it is not integrated with the access control list 
for the RPC ports.
4) it is difficult to run midas in the presence of firewalls (midas programs listen on random 
network ports - cannot be easily added to firewall rules)

There is a new git branch "feature/rpcsecurity" where I am addressing some of these 
problems:

1) UDP sockets are only used for internal communications (hotlinks & etc) within one 
machine, so they should be bound to the localhost address and become invisible to external 
machines. This change breaks binary compatibility from old clients - they are have to be 
relinked with the new midas library or hotlinks & etc will stop working. If some clients cannot 
be rebuild (I have one like this), I am preserving the old way by checking for a special file in 
the experiment directory (same place as ODB.SHM). (done)

2) if one runs on a single machine, does not use the mserver and does not have clients 
running on other machines, then all the RPC ports can be bound to localhost. (this kills the 
MacOS popups about "odbedit wants to connect to the Internet"). (partially done)

This (2) will become the new default - out of the box, midas will not listen to any external 
network connections - making it very secure.

To use the mserver, one will have to change the ODB setting "/Experiment/Security/Enable 
external RPC connections" and restart all midas programs (I am looking for a better name for 
this odb setting).

3) the out-of-the-box default access control list for RPC connections will be set to 
"localhost", which will reject all external connections, even when they are enabled by (2). One 
will be required to enter the names of all machines that will run midas clients in 
"/Experiment/Security/Rpc hosts". (already implemented in main midas, but default access 
control list is empty meaning everybody is permitted)

4) the mserver will be required to attach to some experiment and will use this same access 
control list to restrict access to the main mserver listener port. Right now the mserver listens 
on this port without attaching to any experiment and accepts the access control list via 
command line arguments. I think after this change a single mserver will still be able to service 
multiple experiments (TBD).

5) I am adding an option to fix TCP port numbers for MIDAS programs via 
"/Experiment/Security/Rpc ports/fename = (int)5555". Once a remote frontend is bound to a 
fixed port, appropriate openings can be made in the firewall, etc. Default port number value 
will be 0 meaning "use random port", same as now.

One problem remains with initial connecting to the mserver. The client connects to the main 
mserver listener port (easy to firewall), but then the mserver connects back to the client - this 
reverse connection is difficult to firewall and this handshaking is difficult to fix in the midas 
sources. It will probably remain unresolved for now.

K.O.
  1080   28 Jul 2015 Konstantin OlchanskiInfoPlans for improving midas network security
New git branch "feature/rpcsecurity" implements  these security features:

- all UDP ports are bound to the localhost interface - connections from outside are not possible
- by default out of the box MIDAS RPC TCP ports are bound to the localhost interface - connections from the outside are not possible.

This configuration is suitable for testing MIDAS on a laptop and for running a simple experiment where all programs run on one machine.

This configuration is secure (connections from the outside are not possible).

This configuration makes corporate security people happy - MIDAS ports do not show up on network port scans (nmap & etc). (except for the mhttpd 
web ports).

The change in binding UDP ports is incompatible with previous versions of MIDAS (except on MacOS, where UDP ports were always bound to localhost). 
All MIDAS programs should be rebuild and restarted, otherwise ODB hotlinks and some other stuff will not work. If rebuilding all MIDAS programs is 
impossible (for example I have one magic MIDAS frontend that cannot be rebuilt), one can force the old (insecure) behavior by creating a file 
.UDP_BIND_HOSTNAME in the experiment directory (next to .ODB.SHM).

The mserver will still work in this localhost-restricted configuration - one should use "odbedit -h localhost" to connect. Multiple mserver instances on 
the same machine - using different TCP ports via "-p" and ODB "/Experiment/midas server port" - are still supported.

To run MIDAS programs on remote machines, one should change the ODB setting "/Experiment/Security/Enable non-localhost RPC" to "yes" and 
add the hostnames of all remote machines that will run MIDAS programs to the MIDAS RPC access control list in ODB "/Experiment/Security/RPC hosts".

To avoid "guessing" the host names expected by MIDAS, do this: set "enable non-localhost rpc" to "yes" and restart the mserver. Then go to the remote 
machine and try to start the MIDAS program, i.e. "odbedit -h daq06". This will bomb and there will be a message in the midas log file saying - rejecting 
connection from unallowed host 'ladd21.triumf.ca'. Add this host to "/Experiment/Security/RPC hosts". After you add this hostname to "RPC hosts" and 
restart the mserver, the connection should be successful. When "RPC hosts" is fully populated, one should restart all midas programs - the access 
control list is only loaded at program startup.

If MIDAS clients have to connect from random hosts (i.e. dynamically assigned random DHCP addresses), one can disable the host name checks by 
setting ODB "/experiment/security/Disable RPC hosts check" to "yes". This configuration is insecure and should only be done on a private network 
behind a firewall.

After some more testing this branch will be merged into the main midas.

K.O.
  1085   29 Jul 2015 Konstantin OlchanskiInfomlogger improvements - CRC32C, SHA-2
> A set of improvements to mlogger is in:

Preliminary support for CRC32-zlib, CRC32C, SHA-256 and SHA-512 is in. Checksums are computed correctly, but plumbing configuration is 
preliminary. Good enough for testing and benchmarking.

To enable checksums, set channel compression:
100 - no checksum (LZ4 compression)
11100 - CRC32-zlib checksum
22100 - CRC32C
33100 - SHA-256
44100 - SHA-512
checksums for both uncompressed and compressed data will be computed and reported into midas.log.

To compare:

CRC32-zlib is for compatibility with gzip and zlib tools
CRC32C is for maximum speed
SHA-256 and SHA-512 is for maximum data security

To remember:

- CRC32-zlib is the CRC32 computation from gzip/png/zlib library. I believe the technical name of the algorithm is "adler32".
- CRC32C is the most recently improved version of CRC32 family of checksums. Implementation is from Mark Adler (same Adler as adler32) uses 
hardware acceleration on recent Intel CPUs.
- SHA-256 and SHA-512 are checksums currently accepted as cryptographically secure. One of them is supposed to be faster on 64-bit 
machines. I implement both for benchmarking.

"Cryptographically secure" means "nobody has a practical way to construct two different files with the same checksum".

In simpler words, the file contents cannot change without breaking the checksum - by software bug, by hardware fault, by benign or malicious 
intent.

The CRC family of checksum functions were never cryptographically secure.
MD5 and SHA-1 used to be secure but are no longer considered to be so. MD5 was definitely broken as different files with the same checksum 
have been discovered or constructed.

K.O.
  1086   29 Jul 2015 Konstantin OlchanskiInfoROOT support in flux
The preliminary version of the .bashrc blurb looks like this
(a couple of flaws:
1) identification of CentOS7 is incomplete - please send me a patch
fixed -> 2) there should be a check for root-config already in the PATH, as on Ubuntu, the ROOT package 
may be installed in /usr and root-
config may be already in the path - please send me patch).

if [ -x $(which root-config) ]; then 
    # root already in the PATH 
    true 
elif [ `uname -i` == "i386" ]; then 
    . /daq/daqshare/olchansk/root/root_v5.34.01_SL62_32/bin/thisroot.sh 
    true 
elif [ `lsb_release -r -s` == "7.1.1503" ]; then 
    #. /daq/daqshare/olchansk/root/root_v5.34.32_SL66_64/bin/thisroot.sh 
    true 
else 
     . /daq/daqshare/olchansk/root/root_v5.34.32_SL66_64/bin/thisroot.sh 
     true 
fi 
  1090   12 Aug 2015 Konstantin OlchanskiInfoMerged - improved midas network security
> New git branch "feature/rpcsecurity" implements  these security features:

Branch was merged into main midas with a few minor changes:

1) RPC access control list is now an array of strings - "/Experiment/Security/RPC hosts/Allowed hosts" - this fixes the previous limit of 32 bytes for host name length.
1a) the access control list array is self-growing - it will have at least 10 empty entries at the end at all times.

2) All clients db_watch() the access control list and automatically reload it when it is changed - no need to restart clients. (suggested by Stefan)

3) "mserver -a hostname" switches for manually expanding the mserver access control list had to be removed because it stopped working with reloading from db_watch() - the ODB list will always overwrite 
anything manually added by "-a".

The text below is corrected for these changes:

> 
> - all UDP ports are bound to the localhost interface - connections from outside are not possible
> - by default out of the box MIDAS RPC TCP ports are bound to the localhost interface - connections from the outside are not possible.
> 
> This configuration is suitable for testing MIDAS on a laptop and for running a simple experiment where all programs run on one machine.
> 
> This configuration is secure (connections from the outside are not possible).
> 
> This configuration makes corporate security people happy - MIDAS ports do not show up on network port scans (nmap & etc). (except for the mhttpd 
> web ports).
> 
> The change in binding UDP ports is incompatible with previous versions of MIDAS (except on MacOS, where UDP ports were always bound to localhost). 
> All MIDAS programs should be rebuild and restarted, otherwise ODB hotlinks and some other stuff will not work. If rebuilding all MIDAS programs is 
> impossible (for example I have one magic MIDAS frontend that cannot be rebuilt), one can force the old (insecure) behavior by creating a file 
> .UDP_BIND_HOSTNAME in the experiment directory (next to .ODB.SHM).
> 
> The mserver will still work in this localhost-restricted configuration - one should use "odbedit -h localhost" to connect. Multiple mserver instances on 
> the same machine - using different TCP ports via "-p" and ODB "/Experiment/midas server port" - are still supported.
> 
> To run MIDAS programs on remote machines, one should change the ODB setting "/Experiment/Security/Enable non-localhost RPC" to "yes" and 
> add the hostnames of all remote machines that will run MIDAS programs to the MIDAS RPC access control list in ODB "/Experiment/Security/RPC hosts/Allowed hosts".
> 
> To avoid "guessing" the host names expected by MIDAS, do this: set "enable non-localhost rpc" to "yes" and restart the mserver. Then go to the remote 
> machine and try to start the MIDAS program, i.e. "odbedit -h daq06". This will bomb and there will be a message in the midas log file saying - rejecting 
> connection from unallowed host 'ladd21.triumf.ca'. Add this host to "/Experiment/Security/RPC hosts/Allowed hosts". After you add this hostname to "RPC hosts"
> you should see messages in midas.log about reloading the access control list, try connecting again, it should work now.
> 
> If MIDAS clients have to connect from random hosts (i.e. dynamically assigned random DHCP addresses), one can disable the host name checks by 
> setting ODB "/experiment/security/Disable RPC hosts check" to "yes". This configuration is insecure and should only be done on a private network 
> behind a firewall.
> 
  1091   12 Aug 2015 Konstantin OlchanskiInfomhttpd HTTPS/SSL server updated
> > > mhttpd uses the latest release of mongoose 4.2

HTTPS support is completely broken in mongoose.c between July 28th (1bc9d8eae48f51ceb73ffd918046cfe74d286909)
and August 12th 2015 (fdc5a80a0a9ca54cba794d7c1131add7f55f112f).

I accidentally broke it by a wrong check against absence of EC_KEY in prehistoric openssl shipped with SL4.

As result, the ECDHE ciphers were enabled but did not work - google chrome complained about "obsolete cryptography",
firefox failed to connect at all.

Please update src/mongoose.c to the latest version if you are using https in mhttpd. (as you should)

Sorry about this problem.

K.O.
  1092   14 Aug 2015 Stefan RittInfoMerged - improved midas network security
I tested the new scheme and am quite happy with. Just a minor thing. When I change the ACL, I get messages from all attached programs, like:

[local:Online:S]RPC hosts>set "Allowed hosts[1]" "host.psi.ch"
[ODBEdit,INFO] Reloading RPC hosts access control list via hotlink callback
09:05:11 [mserver,INFO] Reloading RPC hosts access control list via hotlink callback
09:05:11 [ODBEdit,INFO] Reloading RPC hosts access control list via hotlink callback
[ODBEdit,INFO] Reloading RPC hosts access control list via hotlink callback
[ODBEdit,INFO] Reloading RPC hosts access control list via hotlink callback
09:05:11 [mserver,INFO] Reloading RPC hosts access control list via hotlink callback
09:05:11 [ODBEdit,INFO] Reloading RPC hosts access control list via hotlink callback
09:05:11 [mserver,INFO] Reloading RPC hosts access control list via hotlink callback
09:05:11 [ODBEdit,INFO] Reloading RPC hosts access control list via hotlink callback


While this is good for debugging, I would remove it not to confuse the average user.

The trick not to have to "guess" a remote name is quite useful. I'm happy it even made it into the documentation. There is however an
important shortcoming in the documentation. The old documentation had a "quick start" section, which allowed people quickly to
set-up and run a midas system. This is still missing. And it should now contain a link to the "Security" page, so that people can set-up
quickly remote programs.
  1093   14 Aug 2015 Konstantin OlchanskiInfoMerged - improved midas network security
> [local:Online:S]RPC hosts>set "Allowed hosts[1]" "host.psi.ch"
> [ODBEdit,INFO] Reloading RPC hosts access control list via hotlink callback

Yes, this debug message can be removed after (say) one or two weeks. I find it useful at the moment.

> The trick not to have to "guess" a remote name is quite useful. I'm happy it even made it into the documentation.

Yes, this came from CERN/ALPHA felabview where we did not have a full list of labview machines that were sending us labview data.

> There is however an
> important shortcoming in the documentation. The old documentation had a "quick start" section, which allowed people quickly to
> set-up and run a midas system. This is still missing. And it should now contain a link to the "Security" page, so that people can set-up
> quickly remote programs.

Yes, Suzannah was just asking me about the same - we will need to review the quick start guide and the documentation about the mserver and midas rpc.

K.O.
  1100   21 Aug 2015 Thomas LindnerInfomhttpd HTTPS/SSL server updated
> 
> I recommend that you use "mhttpd --mg" as the alternative for running "mhttpd -p" behind an apache 
> proxy. Using "mhttpd -p" (no HTTPS/SSL) on an internet-connected machine is insecure and should not be 
> done. (private network such as 192.168.x.y addresses is okey for now, I guess).

Finally reading through your documentation in detail [1,2].  I find that I don't understand this recommendation to use secure mongoose 
instead of putting mhttpd behind an apache proxy.  I think that it is nice to have secure mhttpd with mongoose as an option, but your 
documentation seems to imply that mhttpd-mongoose is much better than mhttpd-behind-apache and that the latter solution is strongly 
deprecated.

Perhaps I am not understanding the benefits of the new system.  In reference [2] you say "If this is not possible, somewhat better security 
for HTTP is gained by using a password protected SSL (https) proxy."  This seems to imply that the security of mhttpd-mongoose is better 
than the security of mhttpd-behind-apache.  Is that correct?  I thought that they provided similar security (assuming you follow 
recommended configurations for APACHE).

Setting up apache is trivial and it seems that mhttpd-behind-apache has other advantages, like being able to put other web resources 
(ganglia, cameras, elog, etc) behind the same secure server.  Also you can start to build complicated custom pages that are served directly 
from apache and just use MIDAS AJAX calls.  I was imagining slowly moving away from using mhttpd at all and just having html/js/css 
resources served up by apache.

So, unless I'm missing something, at this point I would continue to recommend people use mhttpd-behind-apache and I'd suggest this be 
presented as an equally valid option in the documentation.

[1] https://midas.triumf.ca/MidasWiki/index.php/Mhttpd
[2] https://midas.triumf.ca/MidasWiki/index.php/Setup_MIDAS_experiment#Install_SSL_proxy
  1101   24 Aug 2015 Konstantin OlchanskiInfonote on midas history
> 
> *
  1102   27 Aug 2015 Konstantin OlchanskiInfomhttpd HTTPS/SSL server updated
Stefan identified a serious multi-thread locking bug in mhttpd that affects the operation of the sequencer (a race condition between db_set_record() and 
db_get_record() inside the hotlink code). This is now fixed. If you use the sequencer, please update mhttpd.cxx to the latest (or to this) version.

https://bitbucket.org/tmidas/midas/commits/9d79218a125a4427d0cc2f2b5e4e56d585655c88
K.O.
  1103   27 Aug 2015 Konstantin OlchanskiInfomhttpd HTTPS/SSL server updated
> 
> I find that I don't understand this recommendation to use secure mongoose 
> instead of putting mhttpd behind an apache proxy.
>

This is a very valid question.

I think for a small operation that does not require root access to the host computer, mhttpd+mongoose is a good light weight solution.

For a more elaborate setup with private networks, etc, apache https proxy is probably better - for big experiments, resources like webcams,
ganglia, couchdb, etc also need password protection and apache https is the one stone to kill all birds. (one bird to kill all pigs).

Which one is easier to setup?

mhttpd+mongoose I tried to make simple - you have to create a password file and (optional) a properly signed https certificate.
apache httpd is fairly straightforward if you follow well written instructions (such as we provide for using it with midas). but you do need root access
and you do have to edit a good number of config files.

Which one is secure?

By one definition - will it pass muster with central IT - only apache httpd is secure.

At CERN all we have to say "we use password protected apache httpd HTTPS proxy" and they say "ok!".

If we were to say "we use custom web server based on some strange version of mongoose, customized", they will probably raise some eyebrows.

And keeping central IT happy is important if you want holes in their firewall for off-site access to MIDAS.

Now, which one is secure?

The default distribution of apache httpd in SL6 is insecure. period. some steps to secure it are non-controversial - disable SSLv2, SSLv3, disable RC4 ciphers. This is not enough to pass 
muster with the SSLlabs scanner. One should also disable some obsolete and known-weak ciphers. This will disable some old web browsers and is a more controversial step. (see the SSLlabs 
reports).

The default distribution of mhttpd+mongoose passes muster with SSLlabs and on the strength of that I deem it "secure out of the box". One can suggest alternative security tools and one 
can/should run the SSLlabs scanner against mhttpd after each update, report problems as a bug to midas.

Now, which one is *secure*?

Both apache httpd and mongoose are based on OpenSSL which has been recently demonstrated to be severely insecure. (look into the OpenBSD fork of OpenSSL).

There are alternative HTTPS libraries, such as PolarSSL, which are intended for embedding into other applications and devices - such as into mhttpd or into MSCB-ethernet boxes -
and I hope mongoose/fossa will make the switch by the end of the year. (a compatibility layer for using mongoose with PolarSSL already exists).

So, which one to use?

- for maximum security, use httpd apache (but remember to restrict access to mhttpd web port to be "only from the proxy")
- for light-weight cases, or when root access is not available use built-in https in mhttpd.

The midas wiki documentation should probably be updated to explain all of this.

K.O.
  1104   31 Aug 2015 Konstantin OlchanskiInfomhttpd HTTPS/SSL server updated
Configuration of web server completely changed (merge of branch feature/mongoose-config2). Hopefully for the last time.

mhttpd is now controlled by these ODB variables:

/experiment/
.../midas http port	-> 8080 (0x1F90)
.../midas https port	-> 8443 (0x20FB)
.../http redirect to https -> "y"

the names are self obvious (hopefully)

access control is done by the "-a" command line arguments
and by the access control list in ODB, which works the same way
as the RPC ports access control list. An empty list means free access
from everywhere:

/experiment/security/mhttpd hosts/allowed hosts

the access control list is watched by httpd, there is no need to restarted it after updating the list.

after changing the port number settings, mhttpd should be restarted.

other web control options to mhttpd are:

daq06:midas$ ./linux/bin/mhttpd -h
usage: ./linux/bin/mhttpd [-h Hostname[:port]] [-e Experiment] [-v] [-D] [-a Hostname]
       -a only allow access for specific host(s), several [-a Hostname] statements might be given (default list is ODB "/Experiment/security/mhttpd hosts/allowed hosts")
       --http port - bind to specified HTTP port (default is ODB "/Experiment/midas http port")
       --https port - bind to specified HTTP port (default is ODB "/Experiment/midas https port")
       --nomg use the old mhttpd web server
       --oldserver [port] - use the old web server on given port
       --nooldserver - do not use the old mhttpd web server

To run mhttpd "the old way" (mhttpd -D -p 8080), say "mhttpd --oldserver 8080 --nomg".

The normal way to run mhttpd is: "mhttpd -v" to get debug information and "mhttpd -D" to run in the background.

K.O.
  1105   01 Sep 2015 Konstantin OlchanskiInfonote on midas history
Sorting
  1106   02 Sep 2015 Konstantin OlchanskiInfomlogger history changes
The git branch feature/logger_db_watch is getting ready for merging into main midas.

The main change in the logger is the switch from db_open_record() to db_watch() as the 
method of listening to ODB variables. The new db_watch() function makes it cheap (in the 
number of hotlinks used) to implement "per-variable" history as the new default. In the new 
code, the old "per-equipment" history is no longer available.

In other words:

old per-equipment history: 1 hotlink per equipment
old per-variable history: 1 hotlink per "ls" entry in /eq/xxx/variables (big experiments can 
easily exceed the maximum number of hotlinks!)
new (per-variable) history: (back to) 1 hotlink per equipment

Notable changes from old history:

- works as described in my recent notes - the new code will complain about all incorrect use 
of history - where the old code sometimes silently malfunctioned (i.e. symlinks in unexpected 
places) or bombed (i.e. infinite loop reloading the history).
- all references to "PerVariableHistory" in ODB are removed (this is the new default)
- the "structured bank" records (subdirectories under variables, as in /eq/xxx/var/struct/value) 
are now broken up into individual items. This change is forced by the difference between 
db_open_record() and db_watch() for structured banks written using db_set_record(). The old 
per-variable history kept these items together in one event.

This change is also inline with Stefan's suggestion that all compound items, including arrays, 
should be broken up into separate history events. Keeping with this suggestion, right now 
only arrays are not broken up - because of limitations in the history storage level. As history 
storage is improved, arrays will also be broken up into individual elements.

The new code is functionally complete and all are welcome to try it (but beware as it may eat 
your odb or your history storage - make a backup!).

git checkout feature/logger_db_watch

K.O.
  1107   09 Sep 2015 Thomas LindnerInfomhttpd HTTPS/SSL server updated
> > 
> > I find that I don't understand this recommendation to use secure mongoose 
> > instead of putting mhttpd behind an apache proxy.
> >
> 
> This is a very valid question.
> 
> I think for a small operation that does not require root access to the host computer, mhttpd+mongoose is a good light weight solution.
> ...
> So, which one to use?
> 
> - for maximum security, use httpd apache (but remember to restrict access to mhttpd web port to be "only from the proxy")
> - for light-weight cases, or when root access is not available use built-in https in mhttpd.
> 
> The midas wiki documentation should probably be updated to explain all of this.


Thanks for the detailed explanation.  I agree with your recommendations.  I was mostly interested in having both options treated equally in the documentation.

My only small complaint is that since the default mhttpd comes with mongoose security turned on, you need to explicitly disable the mhttpd+mongoose security first before you can start setting up apache.  I guess that the motivation is 
that we should force people to disable security, rather than hoping that they will enable it.  That's a convincing argument; so all I really need is that this procedure be well documented.
  1108   09 Sep 2015 Thomas LindnerInfomhttpd/SSL error message on MacOS
On my macbook (OS X 10.10.3) I get this error message when starting mhttpd with mongoose-SSL:

[mhttpd,ERROR] [mhttpd.cxx:17092:mongoose,ERROR] mongoose web server error: set_ssl_option: 
openssl "modern cryptography" ECDH ciphers not available

mhttpd seems to start fine anyway and safari connects to the secure midas page without complaining 
about the SSL (it complains about the certificate of course).  So maybe this error message is 
relatively harmless?

I don't get this error message with Scientific Linux 6.7.
  1109   09 Sep 2015 Thomas LindnerInfoDocumentation regarding specifying custom pages
Hi,

We have recently been changing the code in mhttpd that maps custom web pages and resources to 
particular files on the server file system.  Ie, changing the code that uses the ODB keys in /Custom to 
map a web address like 

http://myhost:8081/CS/MyCustomPage 

to some file like

/home/user/resource/mypage.html

This mapping gets complicated when you use the /Custom/Path key to specify a location for web 
resources like images.  We have tried to summarize how the current system works on the wiki

https://midas.triumf.ca/MidasWiki/index.php//Custom_ODB_tree

Please provide any suggestions on how either the documentation or the actual algorithm can be 
improved.
ELOG V3.1.4-2e1708b5