> It would be useful to have it be specified for each script.  Reason is that it's simpler, some scripts might only 
> change a few sensitive settings, then on abort it only has to set back to "normal" what it touched to begin with.  
> Also, the "normal" values are usually stored in local variables, so it's important to have those similarly accessible 
> to the "Abort" portion of the script.

Agree. So I will put a special optional label, which will be accessed upon abort.

Stefan

> > it would be useful to have a kind of scripted "abort" for when something goes wrong ...
> 
> How about having the sequencer switching from the aborted sequence file to the special "abort" sequence file? That 
> should be simple to implement if it is not already there.
> 
> K.O.

It's about the same effort if we jump to a specific label in a script or to a separate script. I just have to find some time to implement it.

Stefan

> it would be useful to have a kind of scripted "abort" for when something goes wrong ...

How about having the sequencer switching from the aborted sequence file to the special "abort" sequence file? That 
should be simple to implement if it is not already there.

K.O.

To improve on the existing HTTP "GET" based AJAX interface to MIDAS, I have been looking at other possible RPCs.

The JSON-RPC standard looks to be the most interesting and my experimental implementation now reached the point where other midas users are welcome to try it:

1. Please checkout the git branch "feature/json_rpc", build and run midas as per normal instructions.
2. Look at the MIDAS "Programs" page, you will "see double", the top is the normal midas programs page, the bottom is the new JSON-RPC based page that updates 
every 1 second.
3. Look at example.html page in examples/javascript1, run that experiment push the buttons.
4. Look at mhttpd.js functions mjsonrpc_xxx() to see how the RPC works.
5. Look at jsonrpc_user.cxx in .../src for examples of adding custom rpc functions to midas.

The main improvement is the use of HTTP POST request which allows unlimited data to be sent to midas (permitting proper implementation of ODB "paste" or "mset") 
and use of JSON encoding for all data, including error responses (removing previous ambiguity and poor documentability of some old AJAX functions).

Cross-origin AJAX requests continue to be fully supported (thanks to Bill Mills) - web pages loaded from local file or from some other web server can make AJAX 
requests into mhttpd. (this trivial functionality is normally prohibited by browser security).

My implementation follows these internet standards:

// https://tools.ietf.org/html/rfc4627 - JSON RFC
// http://www.jsonrpc.org/specification - specification of JSON-RPC 2.0
// http://www.simple-is-better.org/json-rpc/transport_http.html

With following variances:
- JSON encoding for NAN and Inf is Javascript-compatible strings "NaN", "Infinity" and "-Infinity"
- HTTP GET is not supported (not recommended by standard)
- batched JSON-RPC requests not supported yet

K.O.

We're using the lazylogger to trigger a script that copies files, and the lag 
between a completed file appearing and the lazylogger trigger occasionally feels 
uncomfortably long.  It's not too bad - at most, around five or so minutes.  But 
of course it's tough to be patient when you're waiting to look at data.

The settings for our lazylogger specify a 'Stay behind' of zero and a 'Period' of 
zero.

Is there a way to make the lazylogger less lazy?  What determines the time 
between the appearance of a file and the lazylogger trigger?

I am working for the SuperCDMS collaboration on some DAQ issues for our upcoming
SNOLAB installation.  So far, the MIDAS sequencer seems to be a good paradigm
for us to do procedural tasks for our detectors and data running interspersed
with other protocols.  

In our testing we've found that the sequencer works very well for this kind of
activity, although it would be useful to have a kind of scripted "abort" for
when something goes wrong -- especially if the user selects to abort a run
sequence. 

Because the sequencer is setting various detector parameters to a certain value
before performing the tasks, the values will never be restored if the user
aborts the sequence.  Instead, perhaps there can be a portion of a MIDAS
sequence script which is instructed to happen on an abort.  Perhaps something
like all commands after a given tag like:

ON ABORT:

get run on a user-initiated abort?  

> MIDAS does not seem to be consistent (or at least convenient) with how it
> handles booleans in AJAX functions.

MIDAS documentation does not say that arguments to "jset" (ODBSet) should be JSON-encoded:
https://midas.triumf.ca/MidasWiki/index.php/AJAX#jset

You found undocumented behavior.

As solution, we plan to replace "jset/ODBSet" with a JSON-RPC function, where all parameters will be JSON-encoded 
(obviously).

K.O.


> 
> When you request an ODB value that is a boolean with AJAX call like
> 
> http://neut14.triumf.ca:8081/?cmd=jcopy&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys
> 
> then you get
> 
> { "Hidden/last_written" : 1437065425, "Hidden" : false }
> 
> This seems correct, since the JSON convention has booleans encoded as true/false.
> 
> But this convention does not work when trying to set the boolean value. For instance
> 
> http://neut14.triumf.ca:8081/?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=true
> 
> does not set the variable to true. To make this work you need to use the
> characters y/n
> 
> http://neut14.triumf.ca:8081/?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=y
> 
> I tested this with ajax/jset, but the same problem seems to occur when using the
> javascript function ODBSet. The documentation doesn't say what sort of encoding
> to use when using these functions, so I guess the idea is that these functions
> use MIDAS encoding for booleans. But it seems to me that it would be more
> convenient if jset/ODBSet allowed the option to use json/javascript encoding for
> boolean values; or at least had that as a format option for jset/ODBSet.  That
> way my javascript could look like
> 
> var mybool = true;
> URI_command =
> "?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=" + mybool;
> 
> instead of
> 
> var mybool = true;
> URI_command = ""
> if(mybool){
>   URI_command =
> "?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=y";
> else
>   URI_command =
> "?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=n";
> 
> 
> __________________________________________________________
> Cross-posting from bitbucket issue tracker:
> 
> https://bitbucket.org/tmidas/midas/issues/29/jset-odbset-using-true-false-for-booleans

> I have a problem with using analyzer offline.
...
> [Analyzer,INFO] Set run number 20 in ODB
> analyzer: src/odb.c:6631: db_paste_node: Assertion `status == 1' failed.

The problem has been traced to unexpected interaction between code in mana.c and 
aggressive error checking in the odb xml decoder (used to load odb.xml dump from the midas 
data file into odb).

We are working on a solution.

K.O.

Good news, on the new experimental branch feature/jsonrpc, I have implemented all 3 program management functions - start program, 
stop program and "is running?" as JSON-RPC methods. On that branch, the midas "programs" page is done completely using 
javascript.

You can try this new code right now or you can wait until the branch is merged into main midas - I am still ironing some last minute kinks 
in JSON encoding of ODB. But all the programs management should work (and all previously existing stuff should work). Look in 
mhttpd.js for the mjsonrpc_start_program() & etc.

K.O.



> Just in case anyone needs this in the future I am adding a comment about this issue. After a few months of working
> with this solution we noticed that when using
> 
> request.open('GET', url, false);
> 
> on firefox it could cause the javascript to stop at that point, possibly due to a delayed response from the
> server. A simple solution is to send an asynchronous requests
> 
> request.open('GET', url, true);
> 
> Zaher
> 
> > Thank you very much, this is exactly what I need and it works.
> > 
> > Zaher
> > 
> > > All functions in midas are controlled through special URLs. So the URL
> > > 
> > > http://<host:port>/?cmd=Start&value=10
> > > 
> > > will start run #10. Similarly with ?cmd=Stop. Now all you need is to set up a custom button, and use the 
> > > OnClick="" JavaScript method to fire off an Ajax request with the above URL. 
> > > 
> > > To send an Ajax request, you can use the function XMLHttpRequestGeneric which ships as part of midas in the 
> > > mhttpd.js file. Then the code would be
> > > 
> > > <input type="button" onclick="start()">
> > > 
> > > and in your JavaScript code:
> > > 
> > > ...
> > > function start()
> > > {
> > >    var request = XMLHttpRequestGeneric();
> > > 
> > >    url = '?cmd=Start&value=10';
> > >    request.open('GET', url, false);
> > >    request.send(null);
> > > }
> > > ...
> > > 
> > > 
> > > Cheers,
> > > Stefan

I have a problem with using analyzer offline.
I'm trying to do it this way:

[lkst@pklinux online]$ kill_daq.sh
[lkst@pklinux online]$ odbedit -c "load data/run00020.odb"
[ODBEdit,INFO] Reloading RPC hosts access control list via hotlink callback
[lkst@pklinux online]$ analyzer -i data/run00020.mid -o test20.root
Root server listening on port 9090...
Running analyzer offline. Stop with "!"
[Analyzer,INFO] Set run number 20 in ODB
analyzer: src/odb.c:6631: db_paste_node: Assertion `status == 1' failed.
Load ODB from run 20...Aborted
[lkst@pklinux online]$

I always get this "Assertion `status == 1' failed." error even
if I try it in the examples/experiment enclosed in MIDAS distribution.
After this try I can not run any midas program, even odbedit reports an error,
until I delete the ODB file in /dev/shm/ and load last ODB dump with odbedit.
What do I do wrong and how this can be fixed?

Just in case anyone needs this in the future I am adding a comment about this issue. After a few months of working
with this solution we noticed that when using

request.open('GET', url, false);

on firefox it could cause the javascript to stop at that point, possibly due to a delayed response from the
server. A simple solution is to send an asynchronous requests

request.open('GET', url, true);

Zaher

> Thank you very much, this is exactly what I need and it works.
> 
> Zaher
> 
> > All functions in midas are controlled through special URLs. So the URL
> > 
> > http://<host:port>/?cmd=Start&value=10
> > 
> > will start run #10. Similarly with ?cmd=Stop. Now all you need is to set up a custom button, and use the 
> > OnClick="" JavaScript method to fire off an Ajax request with the above URL. 
> > 
> > To send an Ajax request, you can use the function XMLHttpRequestGeneric which ships as part of midas in the 
> > mhttpd.js file. Then the code would be
> > 
> > <input type="button" onclick="start()">
> > 
> > and in your JavaScript code:
> > 
> > ...
> > function start()
> > {
> >    var request = XMLHttpRequestGeneric();
> > 
> >    url = '?cmd=Start&value=10';
> >    request.open('GET', url, false);
> >    request.send(null);
> > }
> > ...
> > 
> > 
> > Cheers,
> > Stefan

The midas wiki at https://midas.triumf.ca has been upgraded to mediawiki version 1.25.2 (current 
production version). If you see any problems, please report them on this forum. K.O.

the midas forum elog is updated to latest version from Stefan - ELOG V3.1.1-b4d2a37 built from git sources. K.O.

> On my macbook (OS X 10.10.3) I get this error message when starting mhttpd with mongoose-SSL:
> 
> [mhttpd,ERROR] [mhttpd.cxx:17092:mongoose,ERROR] mongoose web server error: set_ssl_option: 
> openssl "modern cryptography" ECDH ciphers not available
> 

It means what it says - "modern cryptography" is not available (in google-chrome terms), different browsers report this 
differently, same (apple safari) do not seem to care.

In practice if ECDH ciphers are not available, the https connection uses "obsolete cryptography" and (depending) it 
probably not actually secure (might even be using RC4 ciphers).

The reason you get this error is the obsolete OpenSSL library shipped with MacOS (all version). (same on SL4 and SL5).

Reasonably up-to-date OpenSSL library that has ECDH support can be installed using MacPorts, this step should be 
added to the MIDAS documentation.

>
> mhttpd seems to start fine anyway and safari connects to the secure midas page without complaining 
> about the SSL (it complains about the certificate of course).  So maybe this error message is 
> relatively harmless?
> 

Some browsers do not care about the quality of the connection - google-chrome seems to be the most conservative 
and flags anything that is less than "most state of the art encryption".

Some browsers seem to be happy even if the connection is SSLv2 with RC4 encryption, even though it is not secure at 
all by current thinking.

Is that harmless? (browser says "secure" when it is not?)

> I don't get this error message with Scientific Linux 6.7.

el6 has a reasonably recent OpenSSL library which supports "modern cryptography".

The best guide to this is to run the SSLlabs scanner and read through it's report.


K.O.

P.S. All this said, I hope my rationale to switching away from OpenSSL makes a bit more sense. If we use something 
like PolarSSL, at least we get the same behaviour on all OSes.

K.O.

> 
> Thanks for the detailed explanation.  I agree with your recommendations.  I was mostly interested in having both options treated equally in the documentation.
> 

I did not review the documentation yet, so it is most likely completely wrong.

But in the nutshell, we should document 2 configurations:

1) standalone mhttpd - with built-in https and password protection
2) mhttpd behind a password-protected https proxy (apache httpd) - mhttpd will have https and built-in passwords disabled, http access restricted to localhost (or the host of the httpd, if they are not the same - as at CERN/ALPHA).

K.O.

Hi,

We have recently been changing the code in mhttpd that maps custom web pages and resources to 
particular files on the server file system.  Ie, changing the code that uses the ODB keys in /Custom to 
map a web address like 

http://myhost:8081/CS/MyCustomPage 

to some file like

/home/user/resource/mypage.html

This mapping gets complicated when you use the /Custom/Path key to specify a location for web 
resources like images.  We have tried to summarize how the current system works on the wiki

https://midas.triumf.ca/MidasWiki/index.php//Custom_ODB_tree

Please provide any suggestions on how either the documentation or the actual algorithm can be 
improved.

On my macbook (OS X 10.10.3) I get this error message when starting mhttpd with mongoose-SSL:

[mhttpd,ERROR] [mhttpd.cxx:17092:mongoose,ERROR] mongoose web server error: set_ssl_option: 
openssl "modern cryptography" ECDH ciphers not available

mhttpd seems to start fine anyway and safari connects to the secure midas page without complaining 
about the SSL (it complains about the certificate of course).  So maybe this error message is 
relatively harmless?

I don't get this error message with Scientific Linux 6.7.

> > 
> > I find that I don't understand this recommendation to use secure mongoose 
> > instead of putting mhttpd behind an apache proxy.
> >
> 
> This is a very valid question.
> 
> I think for a small operation that does not require root access to the host computer, mhttpd+mongoose is a good light weight solution.
> ...
> So, which one to use?
> 
> - for maximum security, use httpd apache (but remember to restrict access to mhttpd web port to be "only from the proxy")
> - for light-weight cases, or when root access is not available use built-in https in mhttpd.
> 
> The midas wiki documentation should probably be updated to explain all of this.


Thanks for the detailed explanation.  I agree with your recommendations.  I was mostly interested in having both options treated equally in the documentation.

My only small complaint is that since the default mhttpd comes with mongoose security turned on, you need to explicitly disable the mhttpd+mongoose security first before you can start setting up apache.  I guess that the motivation is 
that we should force people to disable security, rather than hoping that they will enable it.  That's a convincing argument; so all I really need is that this procedure be well documented.

The git branch feature/logger_db_watch is getting ready for merging into main midas.

The main change in the logger is the switch from db_open_record() to db_watch() as the 
method of listening to ODB variables. The new db_watch() function makes it cheap (in the 
number of hotlinks used) to implement "per-variable" history as the new default. In the new 
code, the old "per-equipment" history is no longer available.

In other words:

old per-equipment history: 1 hotlink per equipment
old per-variable history: 1 hotlink per "ls" entry in /eq/xxx/variables (big experiments can 
easily exceed the maximum number of hotlinks!)
new (per-variable) history: (back to) 1 hotlink per equipment

Notable changes from old history:

- works as described in my recent notes - the new code will complain about all incorrect use 
of history - where the old code sometimes silently malfunctioned (i.e. symlinks in unexpected 
places) or bombed (i.e. infinite loop reloading the history).
- all references to "PerVariableHistory" in ODB are removed (this is the new default)
- the "structured bank" records (subdirectories under variables, as in /eq/xxx/var/struct/value) 
are now broken up into individual items. This change is forced by the difference between 
db_open_record() and db_watch() for structured banks written using db_set_record(). The old 
per-variable history kept these items together in one event.

This change is also inline with Stefan's suggestion that all compound items, including arrays, 
should be broken up into separate history events. Keeping with this suggestion, right now 
only arrays are not broken up - because of limitations in the history storage level. As history 
storage is improved, arrays will also be broken up into individual elements.

The new code is functionally complete and all are welcome to try it (but beware as it may eat 
your odb or your history storage - make a backup!).

git checkout feature/logger_db_watch

K.O.

Sorting