> I have a problem with using analyzer offline.
...
> [Analyzer,INFO] Set run number 20 in ODB
> analyzer: src/odb.c:6631: db_paste_node: Assertion `status == 1' failed.

The problem has been traced to unexpected interaction between code in mana.c and 
aggressive error checking in the odb xml decoder (used to load odb.xml dump from the midas 
data file into odb).

We are working on a solution.

K.O.

Good news, on the new experimental branch feature/jsonrpc, I have implemented all 3 program management functions - start program, 
stop program and "is running?" as JSON-RPC methods. On that branch, the midas "programs" page is done completely using 
javascript.

You can try this new code right now or you can wait until the branch is merged into main midas - I am still ironing some last minute kinks 
in JSON encoding of ODB. But all the programs management should work (and all previously existing stuff should work). Look in 
mhttpd.js for the mjsonrpc_start_program() & etc.

K.O.



> Just in case anyone needs this in the future I am adding a comment about this issue. After a few months of working
> with this solution we noticed that when using
> 
> request.open('GET', url, false);
> 
> on firefox it could cause the javascript to stop at that point, possibly due to a delayed response from the
> server. A simple solution is to send an asynchronous requests
> 
> request.open('GET', url, true);
> 
> Zaher
> 
> > Thank you very much, this is exactly what I need and it works.
> > 
> > Zaher
> > 
> > > All functions in midas are controlled through special URLs. So the URL
> > > 
> > > http://<host:port>/?cmd=Start&value=10
> > > 
> > > will start run #10. Similarly with ?cmd=Stop. Now all you need is to set up a custom button, and use the 
> > > OnClick="" JavaScript method to fire off an Ajax request with the above URL. 
> > > 
> > > To send an Ajax request, you can use the function XMLHttpRequestGeneric which ships as part of midas in the 
> > > mhttpd.js file. Then the code would be
> > > 
> > > <input type="button" onclick="start()">
> > > 
> > > and in your JavaScript code:
> > > 
> > > ...
> > > function start()
> > > {
> > >    var request = XMLHttpRequestGeneric();
> > > 
> > >    url = '?cmd=Start&value=10';
> > >    request.open('GET', url, false);
> > >    request.send(null);
> > > }
> > > ...
> > > 
> > > 
> > > Cheers,
> > > Stefan

I have a problem with using analyzer offline.
I'm trying to do it this way:

[lkst@pklinux online]$ kill_daq.sh
[lkst@pklinux online]$ odbedit -c "load data/run00020.odb"
[ODBEdit,INFO] Reloading RPC hosts access control list via hotlink callback
[lkst@pklinux online]$ analyzer -i data/run00020.mid -o test20.root
Root server listening on port 9090...
Running analyzer offline. Stop with "!"
[Analyzer,INFO] Set run number 20 in ODB
analyzer: src/odb.c:6631: db_paste_node: Assertion `status == 1' failed.
Load ODB from run 20...Aborted
[lkst@pklinux online]$

I always get this "Assertion `status == 1' failed." error even
if I try it in the examples/experiment enclosed in MIDAS distribution.
After this try I can not run any midas program, even odbedit reports an error,
until I delete the ODB file in /dev/shm/ and load last ODB dump with odbedit.
What do I do wrong and how this can be fixed?

Just in case anyone needs this in the future I am adding a comment about this issue. After a few months of working
with this solution we noticed that when using

request.open('GET', url, false);

on firefox it could cause the javascript to stop at that point, possibly due to a delayed response from the
server. A simple solution is to send an asynchronous requests

request.open('GET', url, true);

Zaher

> Thank you very much, this is exactly what I need and it works.
> 
> Zaher
> 
> > All functions in midas are controlled through special URLs. So the URL
> > 
> > http://<host:port>/?cmd=Start&value=10
> > 
> > will start run #10. Similarly with ?cmd=Stop. Now all you need is to set up a custom button, and use the 
> > OnClick="" JavaScript method to fire off an Ajax request with the above URL. 
> > 
> > To send an Ajax request, you can use the function XMLHttpRequestGeneric which ships as part of midas in the 
> > mhttpd.js file. Then the code would be
> > 
> > <input type="button" onclick="start()">
> > 
> > and in your JavaScript code:
> > 
> > ...
> > function start()
> > {
> >    var request = XMLHttpRequestGeneric();
> > 
> >    url = '?cmd=Start&value=10';
> >    request.open('GET', url, false);
> >    request.send(null);
> > }
> > ...
> > 
> > 
> > Cheers,
> > Stefan

The midas wiki at https://midas.triumf.ca has been upgraded to mediawiki version 1.25.2 (current 
production version). If you see any problems, please report them on this forum. K.O.

the midas forum elog is updated to latest version from Stefan - ELOG V3.1.1-b4d2a37 built from git sources. K.O.

> On my macbook (OS X 10.10.3) I get this error message when starting mhttpd with mongoose-SSL:
> 
> [mhttpd,ERROR] [mhttpd.cxx:17092:mongoose,ERROR] mongoose web server error: set_ssl_option: 
> openssl "modern cryptography" ECDH ciphers not available
> 

It means what it says - "modern cryptography" is not available (in google-chrome terms), different browsers report this 
differently, same (apple safari) do not seem to care.

In practice if ECDH ciphers are not available, the https connection uses "obsolete cryptography" and (depending) it 
probably not actually secure (might even be using RC4 ciphers).

The reason you get this error is the obsolete OpenSSL library shipped with MacOS (all version). (same on SL4 and SL5).

Reasonably up-to-date OpenSSL library that has ECDH support can be installed using MacPorts, this step should be 
added to the MIDAS documentation.

>
> mhttpd seems to start fine anyway and safari connects to the secure midas page without complaining 
> about the SSL (it complains about the certificate of course).  So maybe this error message is 
> relatively harmless?
> 

Some browsers do not care about the quality of the connection - google-chrome seems to be the most conservative 
and flags anything that is less than "most state of the art encryption".

Some browsers seem to be happy even if the connection is SSLv2 with RC4 encryption, even though it is not secure at 
all by current thinking.

Is that harmless? (browser says "secure" when it is not?)

> I don't get this error message with Scientific Linux 6.7.

el6 has a reasonably recent OpenSSL library which supports "modern cryptography".

The best guide to this is to run the SSLlabs scanner and read through it's report.


K.O.

P.S. All this said, I hope my rationale to switching away from OpenSSL makes a bit more sense. If we use something 
like PolarSSL, at least we get the same behaviour on all OSes.

K.O.

> 
> Thanks for the detailed explanation.  I agree with your recommendations.  I was mostly interested in having both options treated equally in the documentation.
> 

I did not review the documentation yet, so it is most likely completely wrong.

But in the nutshell, we should document 2 configurations:

1) standalone mhttpd - with built-in https and password protection
2) mhttpd behind a password-protected https proxy (apache httpd) - mhttpd will have https and built-in passwords disabled, http access restricted to localhost (or the host of the httpd, if they are not the same - as at CERN/ALPHA).

K.O.

Hi,

We have recently been changing the code in mhttpd that maps custom web pages and resources to 
particular files on the server file system.  Ie, changing the code that uses the ODB keys in /Custom to 
map a web address like 

http://myhost:8081/CS/MyCustomPage 

to some file like

/home/user/resource/mypage.html

This mapping gets complicated when you use the /Custom/Path key to specify a location for web 
resources like images.  We have tried to summarize how the current system works on the wiki

https://midas.triumf.ca/MidasWiki/index.php//Custom_ODB_tree

Please provide any suggestions on how either the documentation or the actual algorithm can be 
improved.

On my macbook (OS X 10.10.3) I get this error message when starting mhttpd with mongoose-SSL:

[mhttpd,ERROR] [mhttpd.cxx:17092:mongoose,ERROR] mongoose web server error: set_ssl_option: 
openssl "modern cryptography" ECDH ciphers not available

mhttpd seems to start fine anyway and safari connects to the secure midas page without complaining 
about the SSL (it complains about the certificate of course).  So maybe this error message is 
relatively harmless?

I don't get this error message with Scientific Linux 6.7.

> > 
> > I find that I don't understand this recommendation to use secure mongoose 
> > instead of putting mhttpd behind an apache proxy.
> >
> 
> This is a very valid question.
> 
> I think for a small operation that does not require root access to the host computer, mhttpd+mongoose is a good light weight solution.
> ...
> So, which one to use?
> 
> - for maximum security, use httpd apache (but remember to restrict access to mhttpd web port to be "only from the proxy")
> - for light-weight cases, or when root access is not available use built-in https in mhttpd.
> 
> The midas wiki documentation should probably be updated to explain all of this.


Thanks for the detailed explanation.  I agree with your recommendations.  I was mostly interested in having both options treated equally in the documentation.

My only small complaint is that since the default mhttpd comes with mongoose security turned on, you need to explicitly disable the mhttpd+mongoose security first before you can start setting up apache.  I guess that the motivation is 
that we should force people to disable security, rather than hoping that they will enable it.  That's a convincing argument; so all I really need is that this procedure be well documented.

The git branch feature/logger_db_watch is getting ready for merging into main midas.

The main change in the logger is the switch from db_open_record() to db_watch() as the 
method of listening to ODB variables. The new db_watch() function makes it cheap (in the 
number of hotlinks used) to implement "per-variable" history as the new default. In the new 
code, the old "per-equipment" history is no longer available.

In other words:

old per-equipment history: 1 hotlink per equipment
old per-variable history: 1 hotlink per "ls" entry in /eq/xxx/variables (big experiments can 
easily exceed the maximum number of hotlinks!)
new (per-variable) history: (back to) 1 hotlink per equipment

Notable changes from old history:

- works as described in my recent notes - the new code will complain about all incorrect use 
of history - where the old code sometimes silently malfunctioned (i.e. symlinks in unexpected 
places) or bombed (i.e. infinite loop reloading the history).
- all references to "PerVariableHistory" in ODB are removed (this is the new default)
- the "structured bank" records (subdirectories under variables, as in /eq/xxx/var/struct/value) 
are now broken up into individual items. This change is forced by the difference between 
db_open_record() and db_watch() for structured banks written using db_set_record(). The old 
per-variable history kept these items together in one event.

This change is also inline with Stefan's suggestion that all compound items, including arrays, 
should be broken up into separate history events. Keeping with this suggestion, right now 
only arrays are not broken up - because of limitations in the history storage level. As history 
storage is improved, arrays will also be broken up into individual elements.

The new code is functionally complete and all are welcome to try it (but beware as it may eat 
your odb or your history storage - make a backup!).

git checkout feature/logger_db_watch

K.O.

Sorting

Configuration of web server completely changed (merge of branch feature/mongoose-config2). Hopefully for the last time.

mhttpd is now controlled by these ODB variables:

/experiment/
.../midas http port	-> 8080 (0x1F90)
.../midas https port	-> 8443 (0x20FB)
.../http redirect to https -> "y"

the names are self obvious (hopefully)

access control is done by the "-a" command line arguments
and by the access control list in ODB, which works the same way
as the RPC ports access control list. An empty list means free access
from everywhere:

/experiment/security/mhttpd hosts/allowed hosts

the access control list is watched by httpd, there is no need to restarted it after updating the list.

after changing the port number settings, mhttpd should be restarted.

other web control options to mhttpd are:

daq06:midas$ ./linux/bin/mhttpd -h
usage: ./linux/bin/mhttpd [-h Hostname[:port]] [-e Experiment] [-v] [-D] [-a Hostname]
       -a only allow access for specific host(s), several [-a Hostname] statements might be given (default list is ODB "/Experiment/security/mhttpd hosts/allowed hosts")
       --http port - bind to specified HTTP port (default is ODB "/Experiment/midas http port")
       --https port - bind to specified HTTP port (default is ODB "/Experiment/midas https port")
       --nomg use the old mhttpd web server
       --oldserver [port] - use the old web server on given port
       --nooldserver - do not use the old mhttpd web server

To run mhttpd "the old way" (mhttpd -D -p 8080), say "mhttpd --oldserver 8080 --nomg".

The normal way to run mhttpd is: "mhttpd -v" to get debug information and "mhttpd -D" to run in the background.

K.O.

> 
> I find that I don't understand this recommendation to use secure mongoose 
> instead of putting mhttpd behind an apache proxy.
>

This is a very valid question.

I think for a small operation that does not require root access to the host computer, mhttpd+mongoose is a good light weight solution.

For a more elaborate setup with private networks, etc, apache https proxy is probably better - for big experiments, resources like webcams,
ganglia, couchdb, etc also need password protection and apache https is the one stone to kill all birds. (one bird to kill all pigs).

Which one is easier to setup?

mhttpd+mongoose I tried to make simple - you have to create a password file and (optional) a properly signed https certificate.
apache httpd is fairly straightforward if you follow well written instructions (such as we provide for using it with midas). but you do need root access
and you do have to edit a good number of config files.

Which one is secure?

By one definition - will it pass muster with central IT - only apache httpd is secure.

At CERN all we have to say "we use password protected apache httpd HTTPS proxy" and they say "ok!".

If we were to say "we use custom web server based on some strange version of mongoose, customized", they will probably raise some eyebrows.

And keeping central IT happy is important if you want holes in their firewall for off-site access to MIDAS.

Now, which one is secure?

The default distribution of apache httpd in SL6 is insecure. period. some steps to secure it are non-controversial - disable SSLv2, SSLv3, disable RC4 ciphers. This is not enough to pass 
muster with the SSLlabs scanner. One should also disable some obsolete and known-weak ciphers. This will disable some old web browsers and is a more controversial step. (see the SSLlabs 
reports).

The default distribution of mhttpd+mongoose passes muster with SSLlabs and on the strength of that I deem it "secure out of the box". One can suggest alternative security tools and one 
can/should run the SSLlabs scanner against mhttpd after each update, report problems as a bug to midas.

Now, which one is *secure*?

Both apache httpd and mongoose are based on OpenSSL which has been recently demonstrated to be severely insecure. (look into the OpenBSD fork of OpenSSL).

There are alternative HTTPS libraries, such as PolarSSL, which are intended for embedding into other applications and devices - such as into mhttpd or into MSCB-ethernet boxes -
and I hope mongoose/fossa will make the switch by the end of the year. (a compatibility layer for using mongoose with PolarSSL already exists).

So, which one to use?

- for maximum security, use httpd apache (but remember to restrict access to mhttpd web port to be "only from the proxy")
- for light-weight cases, or when root access is not available use built-in https in mhttpd.

The midas wiki documentation should probably be updated to explain all of this.

K.O.

Stefan identified a serious multi-thread locking bug in mhttpd that affects the operation of the sequencer (a race condition between db_set_record() and 
db_get_record() inside the hotlink code). This is now fixed. If you use the sequencer, please update mhttpd.cxx to the latest (or to this) version.

https://bitbucket.org/tmidas/midas/commits/9d79218a125a4427d0cc2f2b5e4e56d585655c88
K.O.

> 
> *

> 
> I recommend that you use "mhttpd --mg" as the alternative for running "mhttpd -p" behind an apache 
> proxy. Using "mhttpd -p" (no HTTPS/SSL) on an internet-connected machine is insecure and should not be 
> done. (private network such as 192.168.x.y addresses is okey for now, I guess).

Finally reading through your documentation in detail [1,2].  I find that I don't understand this recommendation to use secure mongoose 
instead of putting mhttpd behind an apache proxy.  I think that it is nice to have secure mhttpd with mongoose as an option, but your 
documentation seems to imply that mhttpd-mongoose is much better than mhttpd-behind-apache and that the latter solution is strongly 
deprecated.

Perhaps I am not understanding the benefits of the new system.  In reference [2] you say "If this is not possible, somewhat better security 
for HTTP is gained by using a password protected SSL (https) proxy."  This seems to imply that the security of mhttpd-mongoose is better 
than the security of mhttpd-behind-apache.  Is that correct?  I thought that they provided similar security (assuming you follow 
recommended configurations for APACHE).

Setting up apache is trivial and it seems that mhttpd-behind-apache has other advantages, like being able to put other web resources 
(ganglia, cameras, elog, etc) behind the same secure server.  Also you can start to build complicated custom pages that are served directly 
from apache and just use MIDAS AJAX calls.  I was imagining slowly moving away from using mhttpd at all and just having html/js/css 
resources served up by apache.

So, unless I'm missing something, at this point I would continue to recommend people use mhttpd-behind-apache and I'd suggest this be 
presented as an equally valid option in the documentation.

[1] https://midas.triumf.ca/MidasWiki/index.php/Mhttpd
[2] https://midas.triumf.ca/MidasWiki/index.php/Setup_MIDAS_experiment#Install_SSL_proxy

> New version of MIDAS has a feature where it seems to automatically resize the
> message page horizontally in order to fix each MIDAS message into one line. 
> Some of my MIDAS messages (in particular error messages, where I need details)
> are very long.  The result is that the MIDAS page automatically becomes very
> wide and I have to scroll a lot left/right in order to read my messages.  This
> is annoying.
> 
> I would vote to roll-back this new feature.

Issues should be reported in the bitbucket issue tracker. I moved this issue over there.

/Stefan

New version of MIDAS has a feature where it seems to automatically resize the
message page horizontally in order to fix each MIDAS message into one line. 
Some of my MIDAS messages (in particular error messages, where I need details)
are very long.  The result is that the MIDAS page automatically becomes very
wide and I have to scroll a lot left/right in order to read my messages.  This
is annoying.

I would vote to roll-back this new feature.