The git branch feature/logger_db_watch is getting ready for merging into main midas.

The main change in the logger is the switch from db_open_record() to db_watch() as the 
method of listening to ODB variables. The new db_watch() function makes it cheap (in the 
number of hotlinks used) to implement "per-variable" history as the new default. In the new 
code, the old "per-equipment" history is no longer available.

In other words:

old per-equipment history: 1 hotlink per equipment
old per-variable history: 1 hotlink per "ls" entry in /eq/xxx/variables (big experiments can 
easily exceed the maximum number of hotlinks!)
new (per-variable) history: (back to) 1 hotlink per equipment

Notable changes from old history:

- works as described in my recent notes - the new code will complain about all incorrect use 
of history - where the old code sometimes silently malfunctioned (i.e. symlinks in unexpected 
places) or bombed (i.e. infinite loop reloading the history).
- all references to "PerVariableHistory" in ODB are removed (this is the new default)
- the "structured bank" records (subdirectories under variables, as in /eq/xxx/var/struct/value) 
are now broken up into individual items. This change is forced by the difference between 
db_open_record() and db_watch() for structured banks written using db_set_record(). The old 
per-variable history kept these items together in one event.

This change is also inline with Stefan's suggestion that all compound items, including arrays, 
should be broken up into separate history events. Keeping with this suggestion, right now 
only arrays are not broken up - because of limitations in the history storage level. As history 
storage is improved, arrays will also be broken up into individual elements.

The new code is functionally complete and all are welcome to try it (but beware as it may eat 
your odb or your history storage - make a backup!).

git checkout feature/logger_db_watch

K.O.

> > 
> > I find that I don't understand this recommendation to use secure mongoose 
> > instead of putting mhttpd behind an apache proxy.
> >
> 
> This is a very valid question.
> 
> I think for a small operation that does not require root access to the host computer, mhttpd+mongoose is a good light weight solution.
> ...
> So, which one to use?
> 
> - for maximum security, use httpd apache (but remember to restrict access to mhttpd web port to be "only from the proxy")
> - for light-weight cases, or when root access is not available use built-in https in mhttpd.
> 
> The midas wiki documentation should probably be updated to explain all of this.


Thanks for the detailed explanation.  I agree with your recommendations.  I was mostly interested in having both options treated equally in the documentation.

My only small complaint is that since the default mhttpd comes with mongoose security turned on, you need to explicitly disable the mhttpd+mongoose security first before you can start setting up apache.  I guess that the motivation is 
that we should force people to disable security, rather than hoping that they will enable it.  That's a convincing argument; so all I really need is that this procedure be well documented.

On my macbook (OS X 10.10.3) I get this error message when starting mhttpd with mongoose-SSL:

[mhttpd,ERROR] [mhttpd.cxx:17092:mongoose,ERROR] mongoose web server error: set_ssl_option: 
openssl "modern cryptography" ECDH ciphers not available

mhttpd seems to start fine anyway and safari connects to the secure midas page without complaining 
about the SSL (it complains about the certificate of course).  So maybe this error message is 
relatively harmless?

I don't get this error message with Scientific Linux 6.7.

Hi,

We have recently been changing the code in mhttpd that maps custom web pages and resources to 
particular files on the server file system.  Ie, changing the code that uses the ODB keys in /Custom to 
map a web address like 

http://myhost:8081/CS/MyCustomPage 

to some file like

/home/user/resource/mypage.html

This mapping gets complicated when you use the /Custom/Path key to specify a location for web 
resources like images.  We have tried to summarize how the current system works on the wiki

https://midas.triumf.ca/MidasWiki/index.php//Custom_ODB_tree

Please provide any suggestions on how either the documentation or the actual algorithm can be 
improved.

> 
> Thanks for the detailed explanation.  I agree with your recommendations.  I was mostly interested in having both options treated equally in the documentation.
> 

I did not review the documentation yet, so it is most likely completely wrong.

But in the nutshell, we should document 2 configurations:

1) standalone mhttpd - with built-in https and password protection
2) mhttpd behind a password-protected https proxy (apache httpd) - mhttpd will have https and built-in passwords disabled, http access restricted to localhost (or the host of the httpd, if they are not the same - as at CERN/ALPHA).

K.O.

> On my macbook (OS X 10.10.3) I get this error message when starting mhttpd with mongoose-SSL:
> 
> [mhttpd,ERROR] [mhttpd.cxx:17092:mongoose,ERROR] mongoose web server error: set_ssl_option: 
> openssl "modern cryptography" ECDH ciphers not available
> 

It means what it says - "modern cryptography" is not available (in google-chrome terms), different browsers report this 
differently, same (apple safari) do not seem to care.

In practice if ECDH ciphers are not available, the https connection uses "obsolete cryptography" and (depending) it 
probably not actually secure (might even be using RC4 ciphers).

The reason you get this error is the obsolete OpenSSL library shipped with MacOS (all version). (same on SL4 and SL5).

Reasonably up-to-date OpenSSL library that has ECDH support can be installed using MacPorts, this step should be 
added to the MIDAS documentation.

>
> mhttpd seems to start fine anyway and safari connects to the secure midas page without complaining 
> about the SSL (it complains about the certificate of course).  So maybe this error message is 
> relatively harmless?
> 

Some browsers do not care about the quality of the connection - google-chrome seems to be the most conservative 
and flags anything that is less than "most state of the art encryption".

Some browsers seem to be happy even if the connection is SSLv2 with RC4 encryption, even though it is not secure at 
all by current thinking.

Is that harmless? (browser says "secure" when it is not?)

> I don't get this error message with Scientific Linux 6.7.

el6 has a reasonably recent OpenSSL library which supports "modern cryptography".

The best guide to this is to run the SSLlabs scanner and read through it's report.


K.O.

P.S. All this said, I hope my rationale to switching away from OpenSSL makes a bit more sense. If we use something 
like PolarSSL, at least we get the same behaviour on all OSes.

K.O.

The midas wiki at https://midas.triumf.ca has been upgraded to mediawiki version 1.25.2 (current 
production version). If you see any problems, please report them on this forum. K.O.

To improve on the existing HTTP "GET" based AJAX interface to MIDAS, I have been looking at other possible RPCs.

The JSON-RPC standard looks to be the most interesting and my experimental implementation now reached the point where other midas users are welcome to try it:

1. Please checkout the git branch "feature/json_rpc", build and run midas as per normal instructions.
2. Look at the MIDAS "Programs" page, you will "see double", the top is the normal midas programs page, the bottom is the new JSON-RPC based page that updates 
every 1 second.
3. Look at example.html page in examples/javascript1, run that experiment push the buttons.
4. Look at mhttpd.js functions mjsonrpc_xxx() to see how the RPC works.
5. Look at jsonrpc_user.cxx in .../src for examples of adding custom rpc functions to midas.

The main improvement is the use of HTTP POST request which allows unlimited data to be sent to midas (permitting proper implementation of ODB "paste" or "mset") 
and use of JSON encoding for all data, including error responses (removing previous ambiguity and poor documentability of some old AJAX functions).

Cross-origin AJAX requests continue to be fully supported (thanks to Bill Mills) - web pages loaded from local file or from some other web server can make AJAX 
requests into mhttpd. (this trivial functionality is normally prohibited by browser security).

My implementation follows these internet standards:

// https://tools.ietf.org/html/rfc4627 - JSON RFC
// http://www.jsonrpc.org/specification - specification of JSON-RPC 2.0
// http://www.simple-is-better.org/json-rpc/transport_http.html

With following variances:
- JSON encoding for NAN and Inf is Javascript-compatible strings "NaN", "Infinity" and "-Infinity"
- HTTP GET is not supported (not recommended by standard)
- batched JSON-RPC requests not supported yet

K.O.

> 
> My implementation follows these internet standards:
> 
> // https://tools.ietf.org/html/rfc4627 - JSON RFC
> // http://www.jsonrpc.org/specification - specification of JSON-RPC 2.0
> // http://www.simple-is-better.org/json-rpc/transport_http.html
> 
> With following variances:
> - JSON encoding for NAN and Inf is Javascript-compatible strings "NaN", "Infinity" and "-Infinity"
> - HTTP GET is not supported (not recommended by standard)
> - batched JSON-RPC requests not supported yet
> 

The last missing piece is now committed - the JSON-RPC interface is now self-documenting via an automatically
generated JSON Schema that lists all RPC methods with their parameters and return values. This documentation
schema is created from simple to use documentation code in each rpc server function, see mjsonrpc.cxx.

To kick the tires, checkout the feature/json_rpc branch, build mhttpd, setup the examples/javascript1 experiment,
run mhttpd in the terminal, from the "status" page, go to the "example" custom page, press "push me" in the mjsonrpc_db_get_values() section,
mhttpd will print the schema file on the terminal. Use any json schema visualization tool to look at it. In the future I hope
to link this schema to the midas "help" page.

The impatient can go directly here: (it is safe to press all buttons) (elog is making a dog's breakfast of my url)
http://ladd00.triumf.ca/~olchansk/test/docson/#../test.json
docson is here:
https://github.com/lbovet/docson

For more informantion about JSON Schema stuff, go here:

https://tools.ietf.org/html/draft-zyp-json-schema-04
http://spacetelescope.github.io/understanding-json-schema/
http://json-schema.org/

JSON Schema examples:
http://json-schema.org/examples.html
http://json-schema.org/example1.html

JSON Schema visualization: (schema file has to have a .json extension)
https://github.com/lbovet/docson

(there is also an interesting discussion on why there is no RFC for JSON schema - the draft expired several years ago)

K.O.

> JSON-RPC interface

For interfacing to MIDAS just from browser javascript, the user does not need to know anything about JSON-RPC, all the user-level mjsonrpc_xxx() functions provided by 
mhttpd.js work the same as the old ODBxxx() functions.

As usual, the functions are documented using Doxygen, so here there is no difference between old and new interfaces.

To generate the documentation, run "make dox" (doxygen and graphviz "dot" packages should be installed). (it will take some time to generate everything), then open 
html/index.html and navigate to "files" to "mhttpd.js" and you will see the list of all RPC funcrions (old functions are ODBxxx, new functions are mjsonrpc_xxx).

There was a possibility to implement the mjsonrpc javascript client interface as a javascript class, but older versions of doxygen seem to work incorrectly for such code making it 
impossible to document the code. So it remains implemented as traditional functions with a few globals, but the design an implementation are done with a view to convert the 
code to a javascript class some time in the future.

As ever, the examples/javascript1 experiment provides examples of using all available javascript functions supported by midas. (except for functions that are hard to example or 
hard to document).

K.O.

If using a synchronous AJAX call, such as "foo=ODBGet("/runinfo/state");", google chrome will prints this to the javascript console:

"Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user's experience. For more help, check http://xhr.spec.whatwg.org/."

The referenced URL has this text:

"Synchronous XMLHttpRequest outside of workers is in the process of being removed from the web platform as it has detrimental effects to the end user's experience. (This is a long 
process that takes many years.) Developers must not pass false for the async argument when the JavaScript global environment is a document environment. User agents are strongly 
encouraged to warn about such usage in developer tools and may experiment with throwing an InvalidAccessError exception when it occurs."

Then jQuery say this: http://api.jquery.com/jquery.ajax/

"As of jQuery 1.8, the use of async: false with jqXHR ($.Deferred) is deprecated; you must use the success/error/complete callback options instead of the corresponding methods of the 
jqXHR object such as jqXHR.done() or the deprecated jqXHR.success()."

This sounds rather severe but one must flow with the flow. Synchronous RPC is out, async is in.

Many of the old MIDAS AJAX functions are fully synchronous (i.e. "foo=ODBGet("/blah");"), some more recent ones support both sync and async use (i.e. ODBMCopy()).

All the newly added functions *must* by async-only. For example, all the new JSON-RPC functions are async-only and require the use of callbacks to get at the data.

Converting existing javascript custom pages from sync AJAX (hah! it's SJAX, not AJAX) will require some work, and one might as well start today.

Personally, I think this excessive use of callbacks for all javascript web page programming is an unnecessary PITA, but I also do understand the motivation
of people who write web browsers and javascript engines - removal of support for synchronous RPC makes many things much simpler -
and even small speedup of javascript execution and better browser efficiency is welcome improvements (but not free improvements - as old web pages need to be converted).

K.O.

We're using mhttpd for calls that end up working better with asynchronous requests, and we've built up sort of a parallel, asynchronous library using javascript Promises.

The Promises (which are in the ES6 spec) have worked incredibly well for building well-behaved, sequential calls to mhttpd.  Personally, I also find their syntax much easier to wrap my
head around, especially compared to callbacks.

I'd be happy to add these functions to midas.js if there's general interest. 

> If using a synchronous AJAX call, such as "foo=ODBGet("/runinfo/state");", google chrome will prints this to the javascript console:
> 
> "Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user's experience. For more help, check http://xhr.spec.whatwg.org/."
> 
> The referenced URL has this text:
> 
> "Synchronous XMLHttpRequest outside of workers is in the process of being removed from the web platform as it has detrimental effects to the end user's experience. (This is a long 
> process that takes many years.) Developers must not pass false for the async argument when the JavaScript global environment is a document environment. User agents are strongly 
> encouraged to warn about such usage in developer tools and may experiment with throwing an InvalidAccessError exception when it occurs."
> 
> Then jQuery say this: http://api.jquery.com/jquery.ajax/
> 
> "As of jQuery 1.8, the use of async: false with jqXHR ($.Deferred) is deprecated; you must use the success/error/complete callback options instead of the corresponding methods of the 
> jqXHR object such as jqXHR.done() or the deprecated jqXHR.success()."
> 
> This sounds rather severe but one must flow with the flow. Synchronous RPC is out, async is in.
> 
> Many of the old MIDAS AJAX functions are fully synchronous (i.e. "foo=ODBGet("/blah");"), some more recent ones support both sync and async use (i.e. ODBMCopy()).
> 
> All the newly added functions *must* by async-only. For example, all the new JSON-RPC functions are async-only and require the use of callbacks to get at the data.
> 
> Converting existing javascript custom pages from sync AJAX (hah! it's SJAX, not AJAX) will require some work, and one might as well start today.
> 
> Personally, I think this excessive use of callbacks for all javascript web page programming is an unnecessary PITA, but I also do understand the motivation
> of people who write web browsers and javascript engines - removal of support for synchronous RPC makes many things much simpler -
> and even small speedup of javascript execution and better browser efficiency is welcome improvements (but not free improvements - as old web pages need to be converted).
> 
> K.O.

> We're using mhttpd for calls that end up working better with asynchronous requests, and we've built up sort of a parallel, asynchronous library using javascript Promises.
> 
> The Promises (which are in the ES6 spec) have worked incredibly well for building well-behaved, sequential calls to mhttpd.  Personally, I also find their syntax much easier to wrap my
> head around, especially compared to callbacks.
> 
> I'd be happy to add these functions to midas.js if there's general interest. 

Why don't you post the functions here so that we can have a look? They don't have to be incorporated into mhttpd.js necessarily, but could live in a separate file, the people can choose which one to use.

Stefan

> We're using mhttpd for calls that end up working better with asynchronous requests, and we've built up sort of a parallel, asynchronous library using javascript Promises.
> 
> The Promises (which are in the ES6 spec) have worked incredibly well for building well-behaved, sequential calls to mhttpd.  Personally, I also find their syntax much easier to wrap my
> head around, especially compared to callbacks.
> 

Yes, the javascript wrappers for the json-rpc interface follow the Promise pattern - an RPC call is provided with two user functions,
one is called on success (and provides the rpc reply), the other on failure (and provides all rpc call information - the xhr object, any exception context, etc).

Use of the Promise class itself seems to be problematic - apparently it does not exist in google chrome 28 (the last version for RHEL/CentOS/SL6).

SL6 is still our main workhorse and it is good to have a choice of 2 browsers (old google chrome vs old firefox).

(All SL5 web browsers are already unusable with the modern web and current mhttpd.)

(Also the RPC calls have more than 1 item of data permitted by the javascript Promise class - of course it can be wrapped
be a container object - just an extra complication to document and to understand).

K.O.

> > 
> > JSON-RPC My implementation follows these internet standards:
> > 
> > // https://tools.ietf.org/html/rfc4627 - JSON RFC
> > // http://www.jsonrpc.org/specification - specification of JSON-RPC 2.0
> > // http://www.simple-is-better.org/json-rpc/transport_http.html
>
> JSON Schema
>
> https://github.com/lbovet/docson
> http://spacetelescope.github.io/understanding-json-schema/
> http://json-schema.org/
> 

Without figuring out how to run docson one can see the JSON-RPC Schema linked from the mhttpd "Help" page
follow link "JSON RPC schema" -> "text format" you will see it pretty printed like this:

---------------------------------------------------------------------
Autogenerated schema for all MIDAS JSON-RPC methods
---------------------------------------------------------------------
cm_exist      | calls MIDAS cm_exist() to check if given MIDAS program is running
              | -----------------------------------------------------
              | params | name           | string         | name of the program, corresponding to ODB /Programs/name
              |        | unique?        | bool           | bUnique argument to cm_exist()
              | -----------------------------------------------------
              | result | status         | integer        | return status of cm_exist()
---------------------------------------------------------------------
cm_shutdown   | calls MIDAS cm_shutdown() to stop given MIDAS program
              | -----------------------------------------------------
              | params | name           | string         | name of the program, corresponding to ODB /Programs/name
              |        | unique?        | bool           | bUnique argument to cm_shutdown()
              | -----------------------------------------------------
              | result | status         | integer        | return status of cm_shutdown()
---------------------------------------------------------------------
db_copy       | get copies of given ODB subtrees in the "save" json encoding
              | -----------------------------------------------------
              | params | paths[]        | array of ODB subtree paths, see note on array indices
              |        |                | array of       | string 
              | -----------------------------------------------------
              | result | data[]         | copy of ODB data for each path
              |        |                | array of       | object 
              |        | status[]       | return status of db_copy_json() for each path
              |        |                | array of       | integer
              |        | last_written[] | last_written value of the ODB subtree for each path
              |        |                | array of       | number 
---------------------------------------------------------------------
db_create     | get copies of given ODB subtrees in the "save" json encoding
              | -----------------------------------------------------
              | params | array of ODB paths to be created
              |        | array of       | arguments to db_create() and db_resize()
              |        |                | path           | string  | ODB path
              |        |                | type           | integer | MIDAS TID_xxx type
              |        |                | array_length?  | integer | optional array length, default is 1
              |        |                | string_length? | integer | for TID_STRING, optional string length, default is NAME_LENGTH
              | -----------------------------------------------------
              | result | status[]       | return status of db_create() for each path
              |        |                | array of       | integer
---------------------------------------------------------------------
db_get_values | get values of ODB data from given subtrees
              | -----------------------------------------------------
              | params | paths[]        | array of ODB subtree paths, see note on array indices
              |        |                | array of       | string 
              | -----------------------------------------------------
              | result | data[]         | values of ODB data for each path, all key names are in lower case, all symlinks are followed
              |        |                | array of       | any    
              |        | status[]       | return status of db_copy_json() for each path
              |        |                | array of       | integer
              |        | last_written[] | last_written value of the ODB subtree for each path
              |        |                | array of       | number 
---------------------------------------------------------------------
db_paste      | write data into ODB
              | -----------------------------------------------------
              | params | paths[]        | array of ODB subtree paths, see note on array indices
              |        |                | array of       | string 
              |        | values[]       | data to be written using db_paste_json()
              |        |                | array of       | any    
              | -----------------------------------------------------
              | result | status[]       | return status of db_paste_json() for each path
              |        |                | array of       | integer
---------------------------------------------------------------------
get_debug     | get current value of mjsonrpc_debug
              | -----------------------------------------------------
              | params | any            | there are no input parameters
              | -----------------------------------------------------
              | result | integer        | current value of mjsonrpc_debug
---------------------------------------------------------------------
get_schema    | Get the MIDAS JSON-RPC schema JSON object
              | -----------------------------------------------------
              | params | any            | there are no input parameters
              | -----------------------------------------------------
              | result | object         | returns the MIDAS JSON-RPC schema JSON object
---------------------------------------------------------------------
null          | RPC method always returns null
              | -----------------------------------------------------
              | params | any            | method parameters are ignored
              | -----------------------------------------------------
              | result | null           | always returns null
---------------------------------------------------------------------
set_debug     | set new value of mjsonrpc_debug
              | -----------------------------------------------------
              | params | integer        | new value of mjsonrpc_debug
              | -----------------------------------------------------
              | result | integer        | new value of mjsonrpc_debug
---------------------------------------------------------------------
start_program | start MIDAS program defined in ODB /Programs/name
              | -----------------------------------------------------
              | params | name           | string         | name of the program, corresponding to ODB /Programs/name
              | -----------------------------------------------------
              | result | status         | integer        | return status of ss_system()
---------------------------------------------------------------------
user_example1 | any   
---------------------------------------------------------------------
user_example2 | any   
---------------------------------------------------------------------
user_example3 | any   

The JSON RPC branch has been merged into main MIDAS. Other than adding new functions, there are no changes to existing MIDAS functionality.

This is the current JSON RPC schema: (from the MIDAS Help page)

------------------------------------------------------------------------
Autogenerated schema for all MIDAS JSON-RPC methods
------------------------------------------------------------------------
cm_exist?      | calls MIDAS cm_exist() to check if given MIDAS program is running
               | -------------------------------------------------------
               | params   | name           | string         | name of the program, corresponding to ODB /Programs/name
               |          | unique?        | bool           | bUnique argument to cm_exist()
               | -------------------------------------------------------
               | result   | status         | integer        | return status of cm_exist()
------------------------------------------------------------------------
cm_shutdown?   | calls MIDAS cm_shutdown() to stop given MIDAS program
               | -------------------------------------------------------
               | params   | name           | string         | name of the program, corresponding to ODB /Programs/name
               |          | unique?        | bool           | bUnique argument to cm_shutdown()
               | -------------------------------------------------------
               | result   | status         | integer        | return status of cm_shutdown()
------------------------------------------------------------------------
db_copy?       | get copies of given ODB subtrees in the "save" json encoding
               | -------------------------------------------------------
               | params   | paths[]        | array of ODB subtree paths, see note on array indices
               |          |                | array of       | string 
               | -------------------------------------------------------
               | result   | data[]         | copy of ODB data for each path
               |          |                | array of       | object 
               |          | status[]       | return status of db_copy_json() for each path
               |          |                | array of       | integer
               |          | last_written[] | last_written value of the ODB subtree for each path
               |          |                | array of       | number 
------------------------------------------------------------------------
db_create?     | get copies of given ODB subtrees in the "save" json encoding
               | -------------------------------------------------------
               | params[] | array of ODB paths to be created
               |          | array of       | arguments to db_create() and db_resize()
               |          |                | path           | string  | ODB path
               |          |                | type           | integer | MIDAS TID_xxx type
               |          |                | array_length?  | integer | optional array length, default is 1
               |          |                | string_length? | integer | for TID_STRING, optional string length, default is NAME_LENGTH
               | -------------------------------------------------------
               | result   | status[]       | return status of db_create() for each path
               |          |                | array of       | integer
------------------------------------------------------------------------
db_get_values? | get values of ODB data from given subtrees
               | -------------------------------------------------------
               | params   | paths[]        | array of ODB subtree paths, see note on array indices
               |          |                | array of       | string 
               | -------------------------------------------------------
               | result   | data[]         | values of ODB data for each path, all key names are in lower case, all symlinks are followed
               |          |                | array of       | any    
               |          | status[]       | return status of db_copy_json() for each path
               |          |                | array of       | integer
               |          | last_written[] | last_written value of the ODB subtree for each path
               |          |                | array of       | number 
------------------------------------------------------------------------
db_paste?      | write data into ODB
               | -------------------------------------------------------
               | params   | paths[]        | array of ODB subtree paths, see note on array indices
               |          |                | array of       | string 
               |          | values[]       | data to be written using db_paste_json()
               |          |                | array of       | any    
               | -------------------------------------------------------
               | result   | status[]       | return status of db_paste_json() for each path
               |          |                | array of       | integer
------------------------------------------------------------------------
get_debug?     | get current value of mjsonrpc_debug
               | -------------------------------------------------------
               | params   | any            | there are no input parameters
               | -------------------------------------------------------
               | result   | integer        | current value of mjsonrpc_debug
------------------------------------------------------------------------
get_schema?    | Get the MIDAS JSON-RPC schema JSON object
               | -------------------------------------------------------
               | params   | any            | there are no input parameters
               | -------------------------------------------------------
               | result   | object         | returns the MIDAS JSON-RPC schema JSON object
------------------------------------------------------------------------
null?          | RPC method always returns null
               | -------------------------------------------------------
               | params   | any            | method parameters are ignored
               | -------------------------------------------------------
               | result   | null           | always returns null
------------------------------------------------------------------------
set_debug?     | set new value of mjsonrpc_debug
               | -------------------------------------------------------
               | params   | integer        | new value of mjsonrpc_debug
               | -------------------------------------------------------
               | result   | integer        | new value of mjsonrpc_debug
------------------------------------------------------------------------
start_program? | start MIDAS program defined in ODB /Programs/name
               | -------------------------------------------------------
               | params   | name           | string         | name of the program, corresponding to ODB /Programs/name
               | -------------------------------------------------------
               | result   | status         | integer        | return status of ss_system()
------------------------------------------------------------------------
user_example1? | example of user defined RPC method that returns up to 3 results
               | -------------------------------------------------------
               | params   | arg            | string         | example string argment
               |          | optional_arg?  | integer        | optional example integer argument
               | -------------------------------------------------------
               | result   | string         | string         | returns the value of "arg" parameter
               |          | integer        | integer        | returns the value of "optional_arg" parameter
------------------------------------------------------------------------
user_example2? | example of user defined RPC method that returns more than 3 results
               | -------------------------------------------------------
               | params   | arg            | string         | example string argment
               |          | optional_arg?  | integer        | optional example integer argument
               | -------------------------------------------------------
               | result   | string1        | string         | returns the value of "arg" parameter
               |          | string2        | string         | returns "hello"
               |          | string3        | string         | returns "world!"
               |          | value1         | integer        | returns the value of "optional_arg" parameter
               |          | value2         | number         | returns 3.14
------------------------------------------------------------------------
user_example3? | example of user defined RPC method that returns an error
               | -------------------------------------------------------
               | params   | arg            | integer        | integer value, if zero, throws a JSON-RPC error
               | -------------------------------------------------------
               | result   | status         | integer        | returns the value of "arg" parameter

K.O.

> > We're using mhttpd for calls that end up working better with asynchronous requests, and we've built up sort of a parallel, asynchronous library using javascript Promises.

I checked again on browser compatibility:

el6: firefox 38 - ok, google-chrome 27 - no
el7: firefox 38 - ok, google-chrome 46 - ok
ubuntu: firefox 42 - ok

mac os, windows - we say "latest firefox or google-chrome is required", then - ok

So we are probably okey with using javascript Promises with MIDAS...

I shall try to convert the json-rpc client library to promises, see how it shakes out.

K.O.

> Why don't you post the functions here so that we can have a look? 

Here is (1) my promisified HTTP request function and (2) a function that uses the returned promises to build an asynchronous, sequential chain of requests to Midas.

Note that if something seems ugly, it's likely because I didn't take the time to clean it up, and not because it particularly *needs* to be ugly.

###### promisified HTTP request ######
In addition to promisifying HTTP requests to Midas, I wanted the Promise.resolve from this function to always return valid JSON.  I also wanted the promise to reject if the response from mhttpd indicated
failure - so that we wouldn't have to rewrite this error checking throughout the code.  The function is so long becuase we make many different calls to mhttpd, and most of them need custom error checking
and, if successful, response packaging.

// begin cdms.daq.utilities.get()
cdms.daq.utilities.get = function(url) {
  return new Promise(function(resolve, reject) {
    // XHR request
    var req = new XMLHttpRequest();
    req.open('GET',url);
 
    req.onload = function() {
      //console.log('done with ', url);
      if(req.status == 200 || req.status == 302) {
        // 'http://dcrc01.triumf.ca:8081/?cmd=jcopy&odb=CustomScript/nonexistent&encoding=json'
        // 'http://dcrc01.triumf.ca:8081/?cmd=jkey&odb=CustomScript/nonexistent'
        // both return <DB_NO_KEY>
        if(/DB_NO_KEY/gi.test(req.response)) {
          reject(req.response);
      
        // 'http://dcrc01.triumf.ca:8081/?cmd=jkey&odb=CustomScript/nonexistent&encoding=json' 
        // returns {"/error": 312}
        } else if(/error.+:\s*\d+/.test(req.response)) {
          reject('<DB_NO_KEY>');
 
        // attempting to start (or stop) a run when a run is already started (or stopped) with
        // 'http://dcrc01.triumf.ca:8081/?cmd=stop' 
        // returns ...<title>MIDAS error</title></head><body><H1>Run is not running</H1>...
        } else if(/MIDAS error/i.test(req.response)) {
          var error_str = /<H1>(.*)<\/H1>/i.exec(req.response);
          reject(error_str[0]);
 
        // 'http://dcrc01.triumf.ca:8081/?cmd=jcreate&odb=/test/foo2&type=7&encoding=json'
        // returns either 1 (creation successful) or 311 (already exists)
        } else if(/jcreate/i.test(url) && (req.response=='1' || req.response=='311')) {
          resolve({'success': true});
 
        // 'http://dcrc01.triumf.ca:8081/?cmd=jset&odb=/test/foo2&val=9&encoding=json'
        // returns OK
        } else if(/jset/i.test(url) && (req.response=='OK')) {
          resolve({'success': true});
 
        // http://dcrc01.triumf.ca:8081/SEQ/?cmd=Load+Script
        // returns an html page showing all available sequencer files
        } else if(/SEQ.*cmd=Load.+Script/i.test(url)) {
          console.log('match Seq load script command url');
          fileList = req.response.match(/\w+.msl/gi);
          resolve({'success': true, 'files': fileList});
 
        // http://dcrc01.triumf.ca:8081/SEQ/?cmd=jmsg&n=10
        // returns 10 most recent messages
        } else if(/.*cmd=jmsg/i.test(url)) {
          console.log('match command url to get messages from mlogger');
          //console.log(req.response);
          resolve({'success': true, 'messages': req.response});
 
        // http://dcrc01.triumf.ca:8081/SEQ/?fs=testflash.msl&dir=&cmd=Load
        // returns <html>redir</html>
        } else if(/SEQ.+fs=\w+.msl&.*cmd=Load/i.test(url)) {
          resolve({'success': true});
 
        // http://dcrc01.triumf.ca:8081/SEQ/?cmd=Start+Script&params=1
        // returns a status of 302 but no response
        } else if(/SEQ.+\?cmd=Start.+Script/i.test(url)) {
          resolve({'success': true});
 
        // http://dcrc01.triumf.ca:8081/?customscript=XYZ&redir=.
        // returns a status of 302 but no response
        // although the redir causes a network call to
        // (in this case) the Midas home page,
        // and that seems to be the req.response
        } else if(/redir/i.test(url)) {
          resolve({'success': true});
 
        // all other responses should be valid JSON
        } else {
          try {
            var json_obj = JSON.parse(req.response);
 
            if(json_obj && typeof json_obj === "object" && json_obj !== null){
              resolve(json_obj);
            }
          } catch(err) {
            console.log('url is ',url);
            console.log('response is ',req.response);
 
            alert(err);
            reject(err);
          }
      }
    } 
    };
     
    req.onerror = function() {
      reject(Error("Network Error"));
    };
 
    req.send();
    //console.log('request to ', url);
  });
}; // end cdms.daq.utilities.get()


###### using promisified HTTP request ######
This is an excerpt that attempts to 
(1) run a script on the DAQ computer, producing a sequencer file
(2) check that the script is completed
(3) load the sequencer file
(4) run the sequencer

Failures on any step jump to the catch, which prints the error on screen.

These are HTTP calls, and given my buggy network should be asynchronous.  At the same time, each of these steps should happen only after its predecessor is completed.  To force sequential execution,
functions within a .then() clause return a Promise object. 

// set the argument string for {{scriptname}}
// then run {{scriptname}}
// check the message log until the success of the script is verified
// check that the sequencer sees the expected file (not implemented!)
// then load the resulting sequencer file
// then start the sequencer
cdms.daq.utilities.get(url).then(function() {
  // call customscript
  var cmd_str = '?customscript=' + scriptName_str + '&redir=.';
  return cdms.daq.utilities.get(baseURL_str + cmd_str);
}).then(function() {
  return checkScriptLoop(uid);
}).then(function() {
  var load_str = baseURL_str + '/SEQ/?cmd=Load+Script';
  console.log('execute promise for url ', load_str);
  return cdms.daq.utilities.get(load_str);
}).then(function() {
  var load_str = '/SEQ/?fs=' + seqFile_str + '&dir=&cmd=Load';
  console.log('execute promise for url ', load_str);
  return cdms.daq.utilities.get(baseURL_str + load_str);
}).then(function() {
  var startSeq_str = '/SEQ/?cmd=Start+Script&params=1';
  console.log('execute promise for url ', startSeq_str);
  return cdms.daq.utilities.get(baseURL_str + startSeq_str);
}).catch(function(error) {
  alert(error);
}).then(function() {
  window.frames['flash-frame'].location = baseURL_str + '/SEQ/';
});

> I checked again on browser compatibility:
> 
> el6: firefox 38 - ok, google-chrome 27 - no
> el7: firefox 38 - ok, google-chrome 46 - ok
> ubuntu: firefox 42 - ok
> 
> mac os, windows - we say "latest firefox or google-chrome is required", then - ok
> 
> So we are probably okey with using javascript Promises with MIDAS...

It looks like this does mean that people using RHEL6 won't have the option of chrome - can they update chrome?

One option is to include a polyfill library like Lie (https://github.com/calvinmetcalf/lie).

> > Why don't you post the functions here so that we can have a look? 
> Here is (1) my promisified HTTP request function and (2) a function that uses the returned promises to build an asynchronous, sequential chain of requests to Midas.
> 
> In addition to promisifying HTTP requests to Midas, I wanted the Promise.resolve from this function to always return valid JSON.

Thank you very much for posting this code. It is very similar to what I just wrote this morning for the JSON-RPC client library. In my case, the JSON-RPC responses
are much more regular so error handling is simple: a) check HTTP response 200, b) check JSON-RPC reply parses into valid JSON (catch exception), c) check JSON-RPC error status.

> I also wanted the promise to reject if the response from mhttpd indicated
> failure - so that we wouldn't have to rewrite this error checking throughout the code.

Right now the JSON-RPC client library does not check the return status of MIDAS calls themselves, i.e. ODBGet("/nonexistant") will go to Promise.resolve() with
the MIDAS db_find_key() status DB_NO_KEY instead of Promise.reject(). So some error handling in Promise.resolve() is still required.

I am thinking how to make these calls go to the error handler automatically, but there is no obvious solution for ODBMGet(["/runinfo", "/nonexistant"]) - the first path
is a success, the second is a failure, do I fail the entire transaction (i.e. with a JSON-RPC error)? Same for JSON-RPC batch transactions - if one transaction
in the batch fails, do I Promise.reject() all of them?

I guess I could "split hairs" and create a separate Promise for each "atomic" transaction, the Promise mechanism does seem to support that,
but this will create more complexity than I feel comfortable with.

Please take a look at the branch feature/js_promise - mjsonrpc_call() is Promisified (resources/mhttpd.js) and the db_copy() example is Promisified (examples/javascript1/example.html)

K.O.