> 
> Thanks for the detailed explanation.  I agree with your recommendations.  I was mostly interested in having both options treated equally in the documentation.
> 

I did not review the documentation yet, so it is most likely completely wrong.

But in the nutshell, we should document 2 configurations:

1) standalone mhttpd - with built-in https and password protection
2) mhttpd behind a password-protected https proxy (apache httpd) - mhttpd will have https and built-in passwords disabled, http access restricted to localhost (or the host of the httpd, if they are not the same - as at CERN/ALPHA).

K.O.

> On my macbook (OS X 10.10.3) I get this error message when starting mhttpd with mongoose-SSL:
> 
> [mhttpd,ERROR] [mhttpd.cxx:17092:mongoose,ERROR] mongoose web server error: set_ssl_option: 
> openssl "modern cryptography" ECDH ciphers not available
> 

It means what it says - "modern cryptography" is not available (in google-chrome terms), different browsers report this 
differently, same (apple safari) do not seem to care.

In practice if ECDH ciphers are not available, the https connection uses "obsolete cryptography" and (depending) it 
probably not actually secure (might even be using RC4 ciphers).

The reason you get this error is the obsolete OpenSSL library shipped with MacOS (all version). (same on SL4 and SL5).

Reasonably up-to-date OpenSSL library that has ECDH support can be installed using MacPorts, this step should be 
added to the MIDAS documentation.

>
> mhttpd seems to start fine anyway and safari connects to the secure midas page without complaining 
> about the SSL (it complains about the certificate of course).  So maybe this error message is 
> relatively harmless?
> 

Some browsers do not care about the quality of the connection - google-chrome seems to be the most conservative 
and flags anything that is less than "most state of the art encryption".

Some browsers seem to be happy even if the connection is SSLv2 with RC4 encryption, even though it is not secure at 
all by current thinking.

Is that harmless? (browser says "secure" when it is not?)

> I don't get this error message with Scientific Linux 6.7.

el6 has a reasonably recent OpenSSL library which supports "modern cryptography".

The best guide to this is to run the SSLlabs scanner and read through it's report.


K.O.

P.S. All this said, I hope my rationale to switching away from OpenSSL makes a bit more sense. If we use something 
like PolarSSL, at least we get the same behaviour on all OSes.

K.O.

The midas wiki at https://midas.triumf.ca has been upgraded to mediawiki version 1.25.2 (current 
production version). If you see any problems, please report them on this forum. K.O.

To improve on the existing HTTP "GET" based AJAX interface to MIDAS, I have been looking at other possible RPCs.

The JSON-RPC standard looks to be the most interesting and my experimental implementation now reached the point where other midas users are welcome to try it:

1. Please checkout the git branch "feature/json_rpc", build and run midas as per normal instructions.
2. Look at the MIDAS "Programs" page, you will "see double", the top is the normal midas programs page, the bottom is the new JSON-RPC based page that updates 
every 1 second.
3. Look at example.html page in examples/javascript1, run that experiment push the buttons.
4. Look at mhttpd.js functions mjsonrpc_xxx() to see how the RPC works.
5. Look at jsonrpc_user.cxx in .../src for examples of adding custom rpc functions to midas.

The main improvement is the use of HTTP POST request which allows unlimited data to be sent to midas (permitting proper implementation of ODB "paste" or "mset") 
and use of JSON encoding for all data, including error responses (removing previous ambiguity and poor documentability of some old AJAX functions).

Cross-origin AJAX requests continue to be fully supported (thanks to Bill Mills) - web pages loaded from local file or from some other web server can make AJAX 
requests into mhttpd. (this trivial functionality is normally prohibited by browser security).

My implementation follows these internet standards:

// https://tools.ietf.org/html/rfc4627 - JSON RFC
// http://www.jsonrpc.org/specification - specification of JSON-RPC 2.0
// http://www.simple-is-better.org/json-rpc/transport_http.html

With following variances:
- JSON encoding for NAN and Inf is Javascript-compatible strings "NaN", "Infinity" and "-Infinity"
- HTTP GET is not supported (not recommended by standard)
- batched JSON-RPC requests not supported yet

K.O.

> 
> My implementation follows these internet standards:
> 
> // https://tools.ietf.org/html/rfc4627 - JSON RFC
> // http://www.jsonrpc.org/specification - specification of JSON-RPC 2.0
> // http://www.simple-is-better.org/json-rpc/transport_http.html
> 
> With following variances:
> - JSON encoding for NAN and Inf is Javascript-compatible strings "NaN", "Infinity" and "-Infinity"
> - HTTP GET is not supported (not recommended by standard)
> - batched JSON-RPC requests not supported yet
> 

The last missing piece is now committed - the JSON-RPC interface is now self-documenting via an automatically
generated JSON Schema that lists all RPC methods with their parameters and return values. This documentation
schema is created from simple to use documentation code in each rpc server function, see mjsonrpc.cxx.

To kick the tires, checkout the feature/json_rpc branch, build mhttpd, setup the examples/javascript1 experiment,
run mhttpd in the terminal, from the "status" page, go to the "example" custom page, press "push me" in the mjsonrpc_db_get_values() section,
mhttpd will print the schema file on the terminal. Use any json schema visualization tool to look at it. In the future I hope
to link this schema to the midas "help" page.

The impatient can go directly here: (it is safe to press all buttons) (elog is making a dog's breakfast of my url)
http://ladd00.triumf.ca/~olchansk/test/docson/#../test.json
docson is here:
https://github.com/lbovet/docson

For more informantion about JSON Schema stuff, go here:

https://tools.ietf.org/html/draft-zyp-json-schema-04
http://spacetelescope.github.io/understanding-json-schema/
http://json-schema.org/

JSON Schema examples:
http://json-schema.org/examples.html
http://json-schema.org/example1.html

JSON Schema visualization: (schema file has to have a .json extension)
https://github.com/lbovet/docson

(there is also an interesting discussion on why there is no RFC for JSON schema - the draft expired several years ago)

K.O.

> JSON-RPC interface

For interfacing to MIDAS just from browser javascript, the user does not need to know anything about JSON-RPC, all the user-level mjsonrpc_xxx() functions provided by 
mhttpd.js work the same as the old ODBxxx() functions.

As usual, the functions are documented using Doxygen, so here there is no difference between old and new interfaces.

To generate the documentation, run "make dox" (doxygen and graphviz "dot" packages should be installed). (it will take some time to generate everything), then open 
html/index.html and navigate to "files" to "mhttpd.js" and you will see the list of all RPC funcrions (old functions are ODBxxx, new functions are mjsonrpc_xxx).

There was a possibility to implement the mjsonrpc javascript client interface as a javascript class, but older versions of doxygen seem to work incorrectly for such code making it 
impossible to document the code. So it remains implemented as traditional functions with a few globals, but the design an implementation are done with a view to convert the 
code to a javascript class some time in the future.

As ever, the examples/javascript1 experiment provides examples of using all available javascript functions supported by midas. (except for functions that are hard to example or 
hard to document).

K.O.

If using a synchronous AJAX call, such as "foo=ODBGet("/runinfo/state");", google chrome will prints this to the javascript console:

"Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user's experience. For more help, check http://xhr.spec.whatwg.org/."

The referenced URL has this text:

"Synchronous XMLHttpRequest outside of workers is in the process of being removed from the web platform as it has detrimental effects to the end user's experience. (This is a long 
process that takes many years.) Developers must not pass false for the async argument when the JavaScript global environment is a document environment. User agents are strongly 
encouraged to warn about such usage in developer tools and may experiment with throwing an InvalidAccessError exception when it occurs."

Then jQuery say this: http://api.jquery.com/jquery.ajax/

"As of jQuery 1.8, the use of async: false with jqXHR ($.Deferred) is deprecated; you must use the success/error/complete callback options instead of the corresponding methods of the 
jqXHR object such as jqXHR.done() or the deprecated jqXHR.success()."

This sounds rather severe but one must flow with the flow. Synchronous RPC is out, async is in.

Many of the old MIDAS AJAX functions are fully synchronous (i.e. "foo=ODBGet("/blah");"), some more recent ones support both sync and async use (i.e. ODBMCopy()).

All the newly added functions *must* by async-only. For example, all the new JSON-RPC functions are async-only and require the use of callbacks to get at the data.

Converting existing javascript custom pages from sync AJAX (hah! it's SJAX, not AJAX) will require some work, and one might as well start today.

Personally, I think this excessive use of callbacks for all javascript web page programming is an unnecessary PITA, but I also do understand the motivation
of people who write web browsers and javascript engines - removal of support for synchronous RPC makes many things much simpler -
and even small speedup of javascript execution and better browser efficiency is welcome improvements (but not free improvements - as old web pages need to be converted).

K.O.

We're using mhttpd for calls that end up working better with asynchronous requests, and we've built up sort of a parallel, asynchronous library using javascript Promises.

The Promises (which are in the ES6 spec) have worked incredibly well for building well-behaved, sequential calls to mhttpd.  Personally, I also find their syntax much easier to wrap my
head around, especially compared to callbacks.

I'd be happy to add these functions to midas.js if there's general interest. 

> If using a synchronous AJAX call, such as "foo=ODBGet("/runinfo/state");", google chrome will prints this to the javascript console:
> 
> "Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user's experience. For more help, check http://xhr.spec.whatwg.org/."
> 
> The referenced URL has this text:
> 
> "Synchronous XMLHttpRequest outside of workers is in the process of being removed from the web platform as it has detrimental effects to the end user's experience. (This is a long 
> process that takes many years.) Developers must not pass false for the async argument when the JavaScript global environment is a document environment. User agents are strongly 
> encouraged to warn about such usage in developer tools and may experiment with throwing an InvalidAccessError exception when it occurs."
> 
> Then jQuery say this: http://api.jquery.com/jquery.ajax/
> 
> "As of jQuery 1.8, the use of async: false with jqXHR ($.Deferred) is deprecated; you must use the success/error/complete callback options instead of the corresponding methods of the 
> jqXHR object such as jqXHR.done() or the deprecated jqXHR.success()."
> 
> This sounds rather severe but one must flow with the flow. Synchronous RPC is out, async is in.
> 
> Many of the old MIDAS AJAX functions are fully synchronous (i.e. "foo=ODBGet("/blah");"), some more recent ones support both sync and async use (i.e. ODBMCopy()).
> 
> All the newly added functions *must* by async-only. For example, all the new JSON-RPC functions are async-only and require the use of callbacks to get at the data.
> 
> Converting existing javascript custom pages from sync AJAX (hah! it's SJAX, not AJAX) will require some work, and one might as well start today.
> 
> Personally, I think this excessive use of callbacks for all javascript web page programming is an unnecessary PITA, but I also do understand the motivation
> of people who write web browsers and javascript engines - removal of support for synchronous RPC makes many things much simpler -
> and even small speedup of javascript execution and better browser efficiency is welcome improvements (but not free improvements - as old web pages need to be converted).
> 
> K.O.

> We're using mhttpd for calls that end up working better with asynchronous requests, and we've built up sort of a parallel, asynchronous library using javascript Promises.
> 
> The Promises (which are in the ES6 spec) have worked incredibly well for building well-behaved, sequential calls to mhttpd.  Personally, I also find their syntax much easier to wrap my
> head around, especially compared to callbacks.
> 
> I'd be happy to add these functions to midas.js if there's general interest. 

Why don't you post the functions here so that we can have a look? They don't have to be incorporated into mhttpd.js necessarily, but could live in a separate file, the people can choose which one to use.

Stefan

> We're using mhttpd for calls that end up working better with asynchronous requests, and we've built up sort of a parallel, asynchronous library using javascript Promises.
> 
> The Promises (which are in the ES6 spec) have worked incredibly well for building well-behaved, sequential calls to mhttpd.  Personally, I also find their syntax much easier to wrap my
> head around, especially compared to callbacks.
> 

Yes, the javascript wrappers for the json-rpc interface follow the Promise pattern - an RPC call is provided with two user functions,
one is called on success (and provides the rpc reply), the other on failure (and provides all rpc call information - the xhr object, any exception context, etc).

Use of the Promise class itself seems to be problematic - apparently it does not exist in google chrome 28 (the last version for RHEL/CentOS/SL6).

SL6 is still our main workhorse and it is good to have a choice of 2 browsers (old google chrome vs old firefox).

(All SL5 web browsers are already unusable with the modern web and current mhttpd.)

(Also the RPC calls have more than 1 item of data permitted by the javascript Promise class - of course it can be wrapped
be a container object - just an extra complication to document and to understand).

K.O.

> > 
> > JSON-RPC My implementation follows these internet standards:
> > 
> > // https://tools.ietf.org/html/rfc4627 - JSON RFC
> > // http://www.jsonrpc.org/specification - specification of JSON-RPC 2.0
> > // http://www.simple-is-better.org/json-rpc/transport_http.html
>
> JSON Schema
>
> https://github.com/lbovet/docson
> http://spacetelescope.github.io/understanding-json-schema/
> http://json-schema.org/
> 

Without figuring out how to run docson one can see the JSON-RPC Schema linked from the mhttpd "Help" page
follow link "JSON RPC schema" -> "text format" you will see it pretty printed like this:

---------------------------------------------------------------------
Autogenerated schema for all MIDAS JSON-RPC methods
---------------------------------------------------------------------
cm_exist      | calls MIDAS cm_exist() to check if given MIDAS program is running
              | -----------------------------------------------------
              | params | name           | string         | name of the program, corresponding to ODB /Programs/name
              |        | unique?        | bool           | bUnique argument to cm_exist()
              | -----------------------------------------------------
              | result | status         | integer        | return status of cm_exist()
---------------------------------------------------------------------
cm_shutdown   | calls MIDAS cm_shutdown() to stop given MIDAS program
              | -----------------------------------------------------
              | params | name           | string         | name of the program, corresponding to ODB /Programs/name
              |        | unique?        | bool           | bUnique argument to cm_shutdown()
              | -----------------------------------------------------
              | result | status         | integer        | return status of cm_shutdown()
---------------------------------------------------------------------
db_copy       | get copies of given ODB subtrees in the "save" json encoding
              | -----------------------------------------------------
              | params | paths[]        | array of ODB subtree paths, see note on array indices
              |        |                | array of       | string 
              | -----------------------------------------------------
              | result | data[]         | copy of ODB data for each path
              |        |                | array of       | object 
              |        | status[]       | return status of db_copy_json() for each path
              |        |                | array of       | integer
              |        | last_written[] | last_written value of the ODB subtree for each path
              |        |                | array of       | number 
---------------------------------------------------------------------
db_create     | get copies of given ODB subtrees in the "save" json encoding
              | -----------------------------------------------------
              | params | array of ODB paths to be created
              |        | array of       | arguments to db_create() and db_resize()
              |        |                | path           | string  | ODB path
              |        |                | type           | integer | MIDAS TID_xxx type
              |        |                | array_length?  | integer | optional array length, default is 1
              |        |                | string_length? | integer | for TID_STRING, optional string length, default is NAME_LENGTH
              | -----------------------------------------------------
              | result | status[]       | return status of db_create() for each path
              |        |                | array of       | integer
---------------------------------------------------------------------
db_get_values | get values of ODB data from given subtrees
              | -----------------------------------------------------
              | params | paths[]        | array of ODB subtree paths, see note on array indices
              |        |                | array of       | string 
              | -----------------------------------------------------
              | result | data[]         | values of ODB data for each path, all key names are in lower case, all symlinks are followed
              |        |                | array of       | any    
              |        | status[]       | return status of db_copy_json() for each path
              |        |                | array of       | integer
              |        | last_written[] | last_written value of the ODB subtree for each path
              |        |                | array of       | number 
---------------------------------------------------------------------
db_paste      | write data into ODB
              | -----------------------------------------------------
              | params | paths[]        | array of ODB subtree paths, see note on array indices
              |        |                | array of       | string 
              |        | values[]       | data to be written using db_paste_json()
              |        |                | array of       | any    
              | -----------------------------------------------------
              | result | status[]       | return status of db_paste_json() for each path
              |        |                | array of       | integer
---------------------------------------------------------------------
get_debug     | get current value of mjsonrpc_debug
              | -----------------------------------------------------
              | params | any            | there are no input parameters
              | -----------------------------------------------------
              | result | integer        | current value of mjsonrpc_debug
---------------------------------------------------------------------
get_schema    | Get the MIDAS JSON-RPC schema JSON object
              | -----------------------------------------------------
              | params | any            | there are no input parameters
              | -----------------------------------------------------
              | result | object         | returns the MIDAS JSON-RPC schema JSON object
---------------------------------------------------------------------
null          | RPC method always returns null
              | -----------------------------------------------------
              | params | any            | method parameters are ignored
              | -----------------------------------------------------
              | result | null           | always returns null
---------------------------------------------------------------------
set_debug     | set new value of mjsonrpc_debug
              | -----------------------------------------------------
              | params | integer        | new value of mjsonrpc_debug
              | -----------------------------------------------------
              | result | integer        | new value of mjsonrpc_debug
---------------------------------------------------------------------
start_program | start MIDAS program defined in ODB /Programs/name
              | -----------------------------------------------------
              | params | name           | string         | name of the program, corresponding to ODB /Programs/name
              | -----------------------------------------------------
              | result | status         | integer        | return status of ss_system()
---------------------------------------------------------------------
user_example1 | any   
---------------------------------------------------------------------
user_example2 | any   
---------------------------------------------------------------------
user_example3 | any   

The JSON RPC branch has been merged into main MIDAS. Other than adding new functions, there are no changes to existing MIDAS functionality.

This is the current JSON RPC schema: (from the MIDAS Help page)

------------------------------------------------------------------------
Autogenerated schema for all MIDAS JSON-RPC methods
------------------------------------------------------------------------
cm_exist?      | calls MIDAS cm_exist() to check if given MIDAS program is running
               | -------------------------------------------------------
               | params   | name           | string         | name of the program, corresponding to ODB /Programs/name
               |          | unique?        | bool           | bUnique argument to cm_exist()
               | -------------------------------------------------------
               | result   | status         | integer        | return status of cm_exist()
------------------------------------------------------------------------
cm_shutdown?   | calls MIDAS cm_shutdown() to stop given MIDAS program
               | -------------------------------------------------------
               | params   | name           | string         | name of the program, corresponding to ODB /Programs/name
               |          | unique?        | bool           | bUnique argument to cm_shutdown()
               | -------------------------------------------------------
               | result   | status         | integer        | return status of cm_shutdown()
------------------------------------------------------------------------
db_copy?       | get copies of given ODB subtrees in the "save" json encoding
               | -------------------------------------------------------
               | params   | paths[]        | array of ODB subtree paths, see note on array indices
               |          |                | array of       | string 
               | -------------------------------------------------------
               | result   | data[]         | copy of ODB data for each path
               |          |                | array of       | object 
               |          | status[]       | return status of db_copy_json() for each path
               |          |                | array of       | integer
               |          | last_written[] | last_written value of the ODB subtree for each path
               |          |                | array of       | number 
------------------------------------------------------------------------
db_create?     | get copies of given ODB subtrees in the "save" json encoding
               | -------------------------------------------------------
               | params[] | array of ODB paths to be created
               |          | array of       | arguments to db_create() and db_resize()
               |          |                | path           | string  | ODB path
               |          |                | type           | integer | MIDAS TID_xxx type
               |          |                | array_length?  | integer | optional array length, default is 1
               |          |                | string_length? | integer | for TID_STRING, optional string length, default is NAME_LENGTH
               | -------------------------------------------------------
               | result   | status[]       | return status of db_create() for each path
               |          |                | array of       | integer
------------------------------------------------------------------------
db_get_values? | get values of ODB data from given subtrees
               | -------------------------------------------------------
               | params   | paths[]        | array of ODB subtree paths, see note on array indices
               |          |                | array of       | string 
               | -------------------------------------------------------
               | result   | data[]         | values of ODB data for each path, all key names are in lower case, all symlinks are followed
               |          |                | array of       | any    
               |          | status[]       | return status of db_copy_json() for each path
               |          |                | array of       | integer
               |          | last_written[] | last_written value of the ODB subtree for each path
               |          |                | array of       | number 
------------------------------------------------------------------------
db_paste?      | write data into ODB
               | -------------------------------------------------------
               | params   | paths[]        | array of ODB subtree paths, see note on array indices
               |          |                | array of       | string 
               |          | values[]       | data to be written using db_paste_json()
               |          |                | array of       | any    
               | -------------------------------------------------------
               | result   | status[]       | return status of db_paste_json() for each path
               |          |                | array of       | integer
------------------------------------------------------------------------
get_debug?     | get current value of mjsonrpc_debug
               | -------------------------------------------------------
               | params   | any            | there are no input parameters
               | -------------------------------------------------------
               | result   | integer        | current value of mjsonrpc_debug
------------------------------------------------------------------------
get_schema?    | Get the MIDAS JSON-RPC schema JSON object
               | -------------------------------------------------------
               | params   | any            | there are no input parameters
               | -------------------------------------------------------
               | result   | object         | returns the MIDAS JSON-RPC schema JSON object
------------------------------------------------------------------------
null?          | RPC method always returns null
               | -------------------------------------------------------
               | params   | any            | method parameters are ignored
               | -------------------------------------------------------
               | result   | null           | always returns null
------------------------------------------------------------------------
set_debug?     | set new value of mjsonrpc_debug
               | -------------------------------------------------------
               | params   | integer        | new value of mjsonrpc_debug
               | -------------------------------------------------------
               | result   | integer        | new value of mjsonrpc_debug
------------------------------------------------------------------------
start_program? | start MIDAS program defined in ODB /Programs/name
               | -------------------------------------------------------
               | params   | name           | string         | name of the program, corresponding to ODB /Programs/name
               | -------------------------------------------------------
               | result   | status         | integer        | return status of ss_system()
------------------------------------------------------------------------
user_example1? | example of user defined RPC method that returns up to 3 results
               | -------------------------------------------------------
               | params   | arg            | string         | example string argment
               |          | optional_arg?  | integer        | optional example integer argument
               | -------------------------------------------------------
               | result   | string         | string         | returns the value of "arg" parameter
               |          | integer        | integer        | returns the value of "optional_arg" parameter
------------------------------------------------------------------------
user_example2? | example of user defined RPC method that returns more than 3 results
               | -------------------------------------------------------
               | params   | arg            | string         | example string argment
               |          | optional_arg?  | integer        | optional example integer argument
               | -------------------------------------------------------
               | result   | string1        | string         | returns the value of "arg" parameter
               |          | string2        | string         | returns "hello"
               |          | string3        | string         | returns "world!"
               |          | value1         | integer        | returns the value of "optional_arg" parameter
               |          | value2         | number         | returns 3.14
------------------------------------------------------------------------
user_example3? | example of user defined RPC method that returns an error
               | -------------------------------------------------------
               | params   | arg            | integer        | integer value, if zero, throws a JSON-RPC error
               | -------------------------------------------------------
               | result   | status         | integer        | returns the value of "arg" parameter

K.O.

> > We're using mhttpd for calls that end up working better with asynchronous requests, and we've built up sort of a parallel, asynchronous library using javascript Promises.

I checked again on browser compatibility:

el6: firefox 38 - ok, google-chrome 27 - no
el7: firefox 38 - ok, google-chrome 46 - ok
ubuntu: firefox 42 - ok

mac os, windows - we say "latest firefox or google-chrome is required", then - ok

So we are probably okey with using javascript Promises with MIDAS...

I shall try to convert the json-rpc client library to promises, see how it shakes out.

K.O.

> Why don't you post the functions here so that we can have a look? 

Here is (1) my promisified HTTP request function and (2) a function that uses the returned promises to build an asynchronous, sequential chain of requests to Midas.

Note that if something seems ugly, it's likely because I didn't take the time to clean it up, and not because it particularly *needs* to be ugly.

###### promisified HTTP request ######
In addition to promisifying HTTP requests to Midas, I wanted the Promise.resolve from this function to always return valid JSON.  I also wanted the promise to reject if the response from mhttpd indicated
failure - so that we wouldn't have to rewrite this error checking throughout the code.  The function is so long becuase we make many different calls to mhttpd, and most of them need custom error checking
and, if successful, response packaging.

// begin cdms.daq.utilities.get()
cdms.daq.utilities.get = function(url) {
  return new Promise(function(resolve, reject) {
    // XHR request
    var req = new XMLHttpRequest();
    req.open('GET',url);
 
    req.onload = function() {
      //console.log('done with ', url);
      if(req.status == 200 || req.status == 302) {
        // 'http://dcrc01.triumf.ca:8081/?cmd=jcopy&odb=CustomScript/nonexistent&encoding=json'
        // 'http://dcrc01.triumf.ca:8081/?cmd=jkey&odb=CustomScript/nonexistent'
        // both return <DB_NO_KEY>
        if(/DB_NO_KEY/gi.test(req.response)) {
          reject(req.response);
      
        // 'http://dcrc01.triumf.ca:8081/?cmd=jkey&odb=CustomScript/nonexistent&encoding=json' 
        // returns {"/error": 312}
        } else if(/error.+:\s*\d+/.test(req.response)) {
          reject('<DB_NO_KEY>');
 
        // attempting to start (or stop) a run when a run is already started (or stopped) with
        // 'http://dcrc01.triumf.ca:8081/?cmd=stop' 
        // returns ...<title>MIDAS error</title></head><body><H1>Run is not running</H1>...
        } else if(/MIDAS error/i.test(req.response)) {
          var error_str = /<H1>(.*)<\/H1>/i.exec(req.response);
          reject(error_str[0]);
 
        // 'http://dcrc01.triumf.ca:8081/?cmd=jcreate&odb=/test/foo2&type=7&encoding=json'
        // returns either 1 (creation successful) or 311 (already exists)
        } else if(/jcreate/i.test(url) && (req.response=='1' || req.response=='311')) {
          resolve({'success': true});
 
        // 'http://dcrc01.triumf.ca:8081/?cmd=jset&odb=/test/foo2&val=9&encoding=json'
        // returns OK
        } else if(/jset/i.test(url) && (req.response=='OK')) {
          resolve({'success': true});
 
        // http://dcrc01.triumf.ca:8081/SEQ/?cmd=Load+Script
        // returns an html page showing all available sequencer files
        } else if(/SEQ.*cmd=Load.+Script/i.test(url)) {
          console.log('match Seq load script command url');
          fileList = req.response.match(/\w+.msl/gi);
          resolve({'success': true, 'files': fileList});
 
        // http://dcrc01.triumf.ca:8081/SEQ/?cmd=jmsg&n=10
        // returns 10 most recent messages
        } else if(/.*cmd=jmsg/i.test(url)) {
          console.log('match command url to get messages from mlogger');
          //console.log(req.response);
          resolve({'success': true, 'messages': req.response});
 
        // http://dcrc01.triumf.ca:8081/SEQ/?fs=testflash.msl&dir=&cmd=Load
        // returns <html>redir</html>
        } else if(/SEQ.+fs=\w+.msl&.*cmd=Load/i.test(url)) {
          resolve({'success': true});
 
        // http://dcrc01.triumf.ca:8081/SEQ/?cmd=Start+Script&params=1
        // returns a status of 302 but no response
        } else if(/SEQ.+\?cmd=Start.+Script/i.test(url)) {
          resolve({'success': true});
 
        // http://dcrc01.triumf.ca:8081/?customscript=XYZ&redir=.
        // returns a status of 302 but no response
        // although the redir causes a network call to
        // (in this case) the Midas home page,
        // and that seems to be the req.response
        } else if(/redir/i.test(url)) {
          resolve({'success': true});
 
        // all other responses should be valid JSON
        } else {
          try {
            var json_obj = JSON.parse(req.response);
 
            if(json_obj && typeof json_obj === "object" && json_obj !== null){
              resolve(json_obj);
            }
          } catch(err) {
            console.log('url is ',url);
            console.log('response is ',req.response);
 
            alert(err);
            reject(err);
          }
      }
    } 
    };
     
    req.onerror = function() {
      reject(Error("Network Error"));
    };
 
    req.send();
    //console.log('request to ', url);
  });
}; // end cdms.daq.utilities.get()


###### using promisified HTTP request ######
This is an excerpt that attempts to 
(1) run a script on the DAQ computer, producing a sequencer file
(2) check that the script is completed
(3) load the sequencer file
(4) run the sequencer

Failures on any step jump to the catch, which prints the error on screen.

These are HTTP calls, and given my buggy network should be asynchronous.  At the same time, each of these steps should happen only after its predecessor is completed.  To force sequential execution,
functions within a .then() clause return a Promise object. 

// set the argument string for {{scriptname}}
// then run {{scriptname}}
// check the message log until the success of the script is verified
// check that the sequencer sees the expected file (not implemented!)
// then load the resulting sequencer file
// then start the sequencer
cdms.daq.utilities.get(url).then(function() {
  // call customscript
  var cmd_str = '?customscript=' + scriptName_str + '&redir=.';
  return cdms.daq.utilities.get(baseURL_str + cmd_str);
}).then(function() {
  return checkScriptLoop(uid);
}).then(function() {
  var load_str = baseURL_str + '/SEQ/?cmd=Load+Script';
  console.log('execute promise for url ', load_str);
  return cdms.daq.utilities.get(load_str);
}).then(function() {
  var load_str = '/SEQ/?fs=' + seqFile_str + '&dir=&cmd=Load';
  console.log('execute promise for url ', load_str);
  return cdms.daq.utilities.get(baseURL_str + load_str);
}).then(function() {
  var startSeq_str = '/SEQ/?cmd=Start+Script&params=1';
  console.log('execute promise for url ', startSeq_str);
  return cdms.daq.utilities.get(baseURL_str + startSeq_str);
}).catch(function(error) {
  alert(error);
}).then(function() {
  window.frames['flash-frame'].location = baseURL_str + '/SEQ/';
});

> I checked again on browser compatibility:
> 
> el6: firefox 38 - ok, google-chrome 27 - no
> el7: firefox 38 - ok, google-chrome 46 - ok
> ubuntu: firefox 42 - ok
> 
> mac os, windows - we say "latest firefox or google-chrome is required", then - ok
> 
> So we are probably okey with using javascript Promises with MIDAS...

It looks like this does mean that people using RHEL6 won't have the option of chrome - can they update chrome?

One option is to include a polyfill library like Lie (https://github.com/calvinmetcalf/lie).

> > Why don't you post the functions here so that we can have a look? 
> Here is (1) my promisified HTTP request function and (2) a function that uses the returned promises to build an asynchronous, sequential chain of requests to Midas.
> 
> In addition to promisifying HTTP requests to Midas, I wanted the Promise.resolve from this function to always return valid JSON.

Thank you very much for posting this code. It is very similar to what I just wrote this morning for the JSON-RPC client library. In my case, the JSON-RPC responses
are much more regular so error handling is simple: a) check HTTP response 200, b) check JSON-RPC reply parses into valid JSON (catch exception), c) check JSON-RPC error status.

> I also wanted the promise to reject if the response from mhttpd indicated
> failure - so that we wouldn't have to rewrite this error checking throughout the code.

Right now the JSON-RPC client library does not check the return status of MIDAS calls themselves, i.e. ODBGet("/nonexistant") will go to Promise.resolve() with
the MIDAS db_find_key() status DB_NO_KEY instead of Promise.reject(). So some error handling in Promise.resolve() is still required.

I am thinking how to make these calls go to the error handler automatically, but there is no obvious solution for ODBMGet(["/runinfo", "/nonexistant"]) - the first path
is a success, the second is a failure, do I fail the entire transaction (i.e. with a JSON-RPC error)? Same for JSON-RPC batch transactions - if one transaction
in the batch fails, do I Promise.reject() all of them?

I guess I could "split hairs" and create a separate Promise for each "atomic" transaction, the Promise mechanism does seem to support that,
but this will create more complexity than I feel comfortable with.

Please take a look at the branch feature/js_promise - mjsonrpc_call() is Promisified (resources/mhttpd.js) and the db_copy() example is Promisified (examples/javascript1/example.html)

K.O.

> Right now the JSON-RPC client library does not check the return status of MIDAS calls themselves, i.e. ODBGet("/nonexistant") will go to Promise.resolve() with
> the MIDAS db_find_key() status DB_NO_KEY instead of Promise.reject(). So some error handling in Promise.resolve() is still required.

> I am thinking how to make these calls go to the error handler automatically, but there is no obvious solution for ODBMGet(["/runinfo", "/nonexistant"]) - the first path
> is a success, the second is a failure, do I fail the entire transaction (i.e. with a JSON-RPC error)? Same for JSON-RPC batch transactions - if one transaction
> in the batch fails, do I Promise.reject() all of them?


Generally, I'd prefer a grouped-request like ODBMGet to return an array of
Promises.  This way, I get to decide how to handle the request responses.  While
I do have cases where I use Promise.all(...), most of my current code would want
to treat each response individually.

But as you point out, my approach differs from the Midas approach significantly.
While I've set up my queries to reject on responses like DB_NO_KEY, the function
mjsonrpc_send_request in mhttpd.js tests purely for a successful http
request.

Since ODBMGet makes a *single* http call, I'd naively lean toward having it
return a single Promise.  Presumably, one that resolves if the http request
"goes through" and rejects if the http request fails. 

My perspective may not be the useful one to consider, though.  If other users
expect an array of promises returned from ODBMGet, definitely feel free to
ignore my thoughts on the matter.

If people really want ODBMGet to return an array of promises, one way to do it
would be to have a 'get' function that only cares about the success of the http
request, and a separate 'response checking' function that validates the
response.  ODBMGet can use these two functions together to return an array of
Promises:

#########
mhttpd.js
#########
function get(url) { 
  // return a promise that resolves if the http request returns status=200
  // reject if the http request does anything else
}

function checkResponse(response) {
  // return a promise that resolves for "good" responses
  // and rejects for "bad response"
}

function ODBMGet(path_arr) {
  var url = // build url from path_arr
  
  // syntax get().then(A).catch(B) means
  // if the http request goes through, A is executed
  // if the http request fails, B is executed
  // ;
  // for get, failure means no successful reply at all
  // so ODBMGet should return an array of rejected Promises 
  get(url).then(function(response) {
    response_arr = // split the response
    return response_arr.map(checkResponse)
  }).catch(function(err) {
    return path_arr.forEach(function() {
      return Promise.reject(err)
    })
  })
}

#########
user code
#########
// here the Promises are treated individually
var response_arr = ODBMGet([path1, path2, path3])

response_arr.forEach(function(thisResponse) {
  thisResponse.then( /* do something */ )
              .catch( /* do something else */ )
})

// and here the failure of a single promise in the array
// determines the code that's executed next
var required_arr = ODBMGet([pathA, pathB, pathC])

Promise.all(required_arr).then( /* do something */ )
                         .catch( /* do something else */ )

> The JSON RPC branch has been merged into main MIDAS.

The interface is now mostly documented, go here: https://midas.triumf.ca/MidasWiki/index.php/Mjsonrpc

Documentation for individual javascript functions in mhttpd.js not merged into the MIDAS documentation yet, because the API is being converted to the Javascript Promise 
pattern (git branch feature/js_promise).

The functions available from mhttpd.js are documented via doxygen, also linked from the mjsonrpc wiki page.

K.O.

I updated the links on the midas wiki to the doxygen-generated documentation for MIDAS that you 
get after running "git clone midas; cd midas; make dox; firefox html/index.html".

Correct link is:
https://daq.triumf.ca/~daqweb/doc/midas-devel/html/

This takes you to a daily/nightly generated snapshot of the midas develop branch and the 
generated documentation with full call graphs.

Previous links were deficient is different ways:
- referred to http://ladd00 instead of https://daq
- referred to wrong path ~daqweb/doc/midas instead of ~daqweb/doc/midas-devel
- referred to the obsolete doxygen generator in midas/doc/html instead of midas/html.

If wrong links are still present on the midas wiki, please let us know and we will fix them.
K.O.

> > I checked again on browser compatibility:
> > 
> > el6: firefox 38 - ok, google-chrome 27 - no
> > el7: firefox 38 - ok, google-chrome 46 - ok
> > ubuntu: firefox 42 - ok
> > 
> > mac os, windows - we say "latest firefox or google-chrome is required", then - ok
> > 
> > So we are probably okey with using javascript Promises with MIDAS...
> 
> [too bad about chrome on SL6] ... include a polyfill library like Lie (https://github.com/calvinmetcalf/lie).

Results of cross-browser testing.

MacOS 10.10.5:

google-chrome 46: Promise ok, Programs page ok, Overlay ok
firefox 42: Promise ok, Programs page ok, Overlay ok.
safari 9.0.1: Promise ok, Programs page ok, Overlay ok.

Linux SL6.7:

google-chrome 27: "Promise not defined"
firefox 38.4.0: Promise ok, Programs page ok, Overlay ok.
konqueror 4.3.4: no go "Can't find variable: JSON"
chromium/google-chrome 38: ok

Linux SL7.1:

google-crome 46: ok
firefox 38.4.0: ok
konqueror 4.10.5: no go, mhttpd.js parse error

Conclusion:

1) firefox is good everywhere
2) google-chrome is good on Mac, Windows and el7 Linux
3) chromium/google-chrome 38 is good on el6 Linux (SL6/CentOS6).

We are good to proceed with adopting the Promise API for MIDAS.

K.O.