 02 Nov 2015, Konstantin Olchanski, Info, midas JSON-RPC interface
|
> >
> > JSON-RPC My implementation follows these internet standards:
> >
> > // https://tools.ietf.org/html/rfc4627 - JSON RFC
> > // http://www.jsonrpc.org/specification - specification of JSON-RPC 2.0
> > // http://www.simple-is-better.org/json-rpc/transport_http.html
>
> JSON Schema
>
> https://github.com/lbovet/docson
> http://spacetelescope.github.io/understanding-json-schema/
> http://json-schema.org/
>
Without figuring out how to run docson one can see the JSON-RPC Schema linked from the mhttpd "Help" page
follow link "JSON RPC schema" -> "text format" you will see it pretty printed like this:
---------------------------------------------------------------------
Autogenerated schema for all MIDAS JSON-RPC methods
---------------------------------------------------------------------
cm_exist | calls MIDAS cm_exist() to check if given MIDAS program is running
| -----------------------------------------------------
| params | name | string | name of the program, corresponding to ODB /Programs/name
| | unique? | bool | bUnique argument to cm_exist()
| -----------------------------------------------------
| result | status | integer | return status of cm_exist()
---------------------------------------------------------------------
cm_shutdown | calls MIDAS cm_shutdown() to stop given MIDAS program
| -----------------------------------------------------
| params | name | string | name of the program, corresponding to ODB /Programs/name
| | unique? | bool | bUnique argument to cm_shutdown()
| -----------------------------------------------------
| result | status | integer | return status of cm_shutdown()
---------------------------------------------------------------------
db_copy | get copies of given ODB subtrees in the "save" json encoding
| -----------------------------------------------------
| params | paths[] | array of ODB subtree paths, see note on array indices
| | | array of | string
| -----------------------------------------------------
| result | data[] | copy of ODB data for each path
| | | array of | object
| | status[] | return status of db_copy_json() for each path
| | | array of | integer
| | last_written[] | last_written value of the ODB subtree for each path
| | | array of | number
---------------------------------------------------------------------
db_create | get copies of given ODB subtrees in the "save" json encoding
| -----------------------------------------------------
| params | array of ODB paths to be created
| | array of | arguments to db_create() and db_resize()
| | | path | string | ODB path
| | | type | integer | MIDAS TID_xxx type
| | | array_length? | integer | optional array length, default is 1
| | | string_length? | integer | for TID_STRING, optional string length, default is NAME_LENGTH
| -----------------------------------------------------
| result | status[] | return status of db_create() for each path
| | | array of | integer
---------------------------------------------------------------------
db_get_values | get values of ODB data from given subtrees
| -----------------------------------------------------
| params | paths[] | array of ODB subtree paths, see note on array indices
| | | array of | string
| -----------------------------------------------------
| result | data[] | values of ODB data for each path, all key names are in lower case, all symlinks are followed
| | | array of | any
| | status[] | return status of db_copy_json() for each path
| | | array of | integer
| | last_written[] | last_written value of the ODB subtree for each path
| | | array of | number
---------------------------------------------------------------------
db_paste | write data into ODB
| -----------------------------------------------------
| params | paths[] | array of ODB subtree paths, see note on array indices
| | | array of | string
| | values[] | data to be written using db_paste_json()
| | | array of | any
| -----------------------------------------------------
| result | status[] | return status of db_paste_json() for each path
| | | array of | integer
---------------------------------------------------------------------
get_debug | get current value of mjsonrpc_debug
| -----------------------------------------------------
| params | any | there are no input parameters
| -----------------------------------------------------
| result | integer | current value of mjsonrpc_debug
---------------------------------------------------------------------
get_schema | Get the MIDAS JSON-RPC schema JSON object
| -----------------------------------------------------
| params | any | there are no input parameters
| -----------------------------------------------------
| result | object | returns the MIDAS JSON-RPC schema JSON object
---------------------------------------------------------------------
null | RPC method always returns null
| -----------------------------------------------------
| params | any | method parameters are ignored
| -----------------------------------------------------
| result | null | always returns null
---------------------------------------------------------------------
set_debug | set new value of mjsonrpc_debug
| -----------------------------------------------------
| params | integer | new value of mjsonrpc_debug
| -----------------------------------------------------
| result | integer | new value of mjsonrpc_debug
---------------------------------------------------------------------
start_program | start MIDAS program defined in ODB /Programs/name
| -----------------------------------------------------
| params | name | string | name of the program, corresponding to ODB /Programs/name
| -----------------------------------------------------
| result | status | integer | return status of ss_system()
---------------------------------------------------------------------
user_example1 | any
---------------------------------------------------------------------
user_example2 | any
---------------------------------------------------------------------
user_example3 | any |
|
11 Nov 2015, Konstantin Olchanski, Info, merged: midas JSON-RPC interface
|
The JSON RPC branch has been merged into main MIDAS. Other than adding new functions, there are no changes to existing MIDAS functionality.
This is the current JSON RPC schema: (from the MIDAS Help page)
------------------------------------------------------------------------
Autogenerated schema for all MIDAS JSON-RPC methods
------------------------------------------------------------------------
cm_exist? | calls MIDAS cm_exist() to check if given MIDAS program is running
| -------------------------------------------------------
| params | name | string | name of the program, corresponding to ODB /Programs/name
| | unique? | bool | bUnique argument to cm_exist()
| -------------------------------------------------------
| result | status | integer | return status of cm_exist()
------------------------------------------------------------------------
cm_shutdown? | calls MIDAS cm_shutdown() to stop given MIDAS program
| -------------------------------------------------------
| params | name | string | name of the program, corresponding to ODB /Programs/name
| | unique? | bool | bUnique argument to cm_shutdown()
| -------------------------------------------------------
| result | status | integer | return status of cm_shutdown()
------------------------------------------------------------------------
db_copy? | get copies of given ODB subtrees in the "save" json encoding
| -------------------------------------------------------
| params | paths[] | array of ODB subtree paths, see note on array indices
| | | array of | string
| -------------------------------------------------------
| result | data[] | copy of ODB data for each path
| | | array of | object
| | status[] | return status of db_copy_json() for each path
| | | array of | integer
| | last_written[] | last_written value of the ODB subtree for each path
| | | array of | number
------------------------------------------------------------------------
db_create? | get copies of given ODB subtrees in the "save" json encoding
| -------------------------------------------------------
| params[] | array of ODB paths to be created
| | array of | arguments to db_create() and db_resize()
| | | path | string | ODB path
| | | type | integer | MIDAS TID_xxx type
| | | array_length? | integer | optional array length, default is 1
| | | string_length? | integer | for TID_STRING, optional string length, default is NAME_LENGTH
| -------------------------------------------------------
| result | status[] | return status of db_create() for each path
| | | array of | integer
------------------------------------------------------------------------
db_get_values? | get values of ODB data from given subtrees
| -------------------------------------------------------
| params | paths[] | array of ODB subtree paths, see note on array indices
| | | array of | string
| -------------------------------------------------------
| result | data[] | values of ODB data for each path, all key names are in lower case, all symlinks are followed
| | | array of | any
| | status[] | return status of db_copy_json() for each path
| | | array of | integer
| | last_written[] | last_written value of the ODB subtree for each path
| | | array of | number
------------------------------------------------------------------------
db_paste? | write data into ODB
| -------------------------------------------------------
| params | paths[] | array of ODB subtree paths, see note on array indices
| | | array of | string
| | values[] | data to be written using db_paste_json()
| | | array of | any
| -------------------------------------------------------
| result | status[] | return status of db_paste_json() for each path
| | | array of | integer
------------------------------------------------------------------------
get_debug? | get current value of mjsonrpc_debug
| -------------------------------------------------------
| params | any | there are no input parameters
| -------------------------------------------------------
| result | integer | current value of mjsonrpc_debug
------------------------------------------------------------------------
get_schema? | Get the MIDAS JSON-RPC schema JSON object
| -------------------------------------------------------
| params | any | there are no input parameters
| -------------------------------------------------------
| result | object | returns the MIDAS JSON-RPC schema JSON object
------------------------------------------------------------------------
null? | RPC method always returns null
| -------------------------------------------------------
| params | any | method parameters are ignored
| -------------------------------------------------------
| result | null | always returns null
------------------------------------------------------------------------
set_debug? | set new value of mjsonrpc_debug
| -------------------------------------------------------
| params | integer | new value of mjsonrpc_debug
| -------------------------------------------------------
| result | integer | new value of mjsonrpc_debug
------------------------------------------------------------------------
start_program? | start MIDAS program defined in ODB /Programs/name
| -------------------------------------------------------
| params | name | string | name of the program, corresponding to ODB /Programs/name
| -------------------------------------------------------
| result | status | integer | return status of ss_system()
------------------------------------------------------------------------
user_example1? | example of user defined RPC method that returns up to 3 results
| -------------------------------------------------------
| params | arg | string | example string argment
| | optional_arg? | integer | optional example integer argument
| -------------------------------------------------------
| result | string | string | returns the value of "arg" parameter
| | integer | integer | returns the value of "optional_arg" parameter
------------------------------------------------------------------------
user_example2? | example of user defined RPC method that returns more than 3 results
| -------------------------------------------------------
| params | arg | string | example string argment
| | optional_arg? | integer | optional example integer argument
| -------------------------------------------------------
| result | string1 | string | returns the value of "arg" parameter
| | string2 | string | returns "hello"
| | string3 | string | returns "world!"
| | value1 | integer | returns the value of "optional_arg" parameter
| | value2 | number | returns 3.14
------------------------------------------------------------------------
user_example3? | example of user defined RPC method that returns an error
| -------------------------------------------------------
| params | arg | integer | integer value, if zero, throws a JSON-RPC error
| -------------------------------------------------------
| result | status | integer | returns the value of "arg" parameter
K.O. |
|
17 Nov 2015, Konstantin Olchanski, Info, synchronous ajax deprecated
|
> > We're using mhttpd for calls that end up working better with asynchronous requests, and we've built up sort of a parallel, asynchronous library using javascript Promises.
I checked again on browser compatibility:
el6: firefox 38 - ok, google-chrome 27 - no
el7: firefox 38 - ok, google-chrome 46 - ok
ubuntu: firefox 42 - ok
mac os, windows - we say "latest firefox or google-chrome is required", then - ok
So we are probably okey with using javascript Promises with MIDAS...
I shall try to convert the json-rpc client library to promises, see how it shakes out.
K.O. |
|
18 Nov 2015, Amy Roberts, Info, synchronous ajax deprecated
|
> Why don't you post the functions here so that we can have a look?
Here is (1) my promisified HTTP request function and (2) a function that uses the returned promises to build an asynchronous, sequential chain of requests to Midas.
Note that if something seems ugly, it's likely because I didn't take the time to clean it up, and not because it particularly *needs* to be ugly.
###### promisified HTTP request ######
In addition to promisifying HTTP requests to Midas, I wanted the Promise.resolve from this function to always return valid JSON. I also wanted the promise to reject if the response from mhttpd indicated
failure - so that we wouldn't have to rewrite this error checking throughout the code. The function is so long becuase we make many different calls to mhttpd, and most of them need custom error checking
and, if successful, response packaging.
// begin cdms.daq.utilities.get()
cdms.daq.utilities.get = function(url) {
return new Promise(function(resolve, reject) {
// XHR request
var req = new XMLHttpRequest();
req.open('GET',url);
req.onload = function() {
//console.log('done with ', url);
if(req.status == 200 || req.status == 302) {
// 'http://dcrc01.triumf.ca:8081/?cmd=jcopy&odb=CustomScript/nonexistent&encoding=json'
// 'http://dcrc01.triumf.ca:8081/?cmd=jkey&odb=CustomScript/nonexistent'
// both return <DB_NO_KEY>
if(/DB_NO_KEY/gi.test(req.response)) {
reject(req.response);
// 'http://dcrc01.triumf.ca:8081/?cmd=jkey&odb=CustomScript/nonexistent&encoding=json'
// returns {"/error": 312}
} else if(/error.+:\s*\d+/.test(req.response)) {
reject('<DB_NO_KEY>');
// attempting to start (or stop) a run when a run is already started (or stopped) with
// 'http://dcrc01.triumf.ca:8081/?cmd=stop'
// returns ...<title>MIDAS error</title></head><body><H1>Run is not running</H1>...
} else if(/MIDAS error/i.test(req.response)) {
var error_str = /<H1>(.*)<\/H1>/i.exec(req.response);
reject(error_str[0]);
// 'http://dcrc01.triumf.ca:8081/?cmd=jcreate&odb=/test/foo2&type=7&encoding=json'
// returns either 1 (creation successful) or 311 (already exists)
} else if(/jcreate/i.test(url) && (req.response=='1' || req.response=='311')) {
resolve({'success': true});
// 'http://dcrc01.triumf.ca:8081/?cmd=jset&odb=/test/foo2&val=9&encoding=json'
// returns OK
} else if(/jset/i.test(url) && (req.response=='OK')) {
resolve({'success': true});
// http://dcrc01.triumf.ca:8081/SEQ/?cmd=Load+Script
// returns an html page showing all available sequencer files
} else if(/SEQ.*cmd=Load.+Script/i.test(url)) {
console.log('match Seq load script command url');
fileList = req.response.match(/\w+.msl/gi);
resolve({'success': true, 'files': fileList});
// http://dcrc01.triumf.ca:8081/SEQ/?cmd=jmsg&n=10
// returns 10 most recent messages
} else if(/.*cmd=jmsg/i.test(url)) {
console.log('match command url to get messages from mlogger');
//console.log(req.response);
resolve({'success': true, 'messages': req.response});
// http://dcrc01.triumf.ca:8081/SEQ/?fs=testflash.msl&dir=&cmd=Load
// returns <html>redir</html>
} else if(/SEQ.+fs=\w+.msl&.*cmd=Load/i.test(url)) {
resolve({'success': true});
// http://dcrc01.triumf.ca:8081/SEQ/?cmd=Start+Script¶ms=1
// returns a status of 302 but no response
} else if(/SEQ.+\?cmd=Start.+Script/i.test(url)) {
resolve({'success': true});
// http://dcrc01.triumf.ca:8081/?customscript=XYZ&redir=.
// returns a status of 302 but no response
// although the redir causes a network call to
// (in this case) the Midas home page,
// and that seems to be the req.response
} else if(/redir/i.test(url)) {
resolve({'success': true});
// all other responses should be valid JSON
} else {
try {
var json_obj = JSON.parse(req.response);
if(json_obj && typeof json_obj === "object" && json_obj !== null){
resolve(json_obj);
}
} catch(err) {
console.log('url is ',url);
console.log('response is ',req.response);
alert(err);
reject(err);
}
}
}
};
req.onerror = function() {
reject(Error("Network Error"));
};
req.send();
//console.log('request to ', url);
});
}; // end cdms.daq.utilities.get()
###### using promisified HTTP request ######
This is an excerpt that attempts to
(1) run a script on the DAQ computer, producing a sequencer file
(2) check that the script is completed
(3) load the sequencer file
(4) run the sequencer
Failures on any step jump to the catch, which prints the error on screen.
These are HTTP calls, and given my buggy network should be asynchronous. At the same time, each of these steps should happen only after its predecessor is completed. To force sequential execution,
functions within a .then() clause return a Promise object.
// set the argument string for {{scriptname}}
// then run {{scriptname}}
// check the message log until the success of the script is verified
// check that the sequencer sees the expected file (not implemented!)
// then load the resulting sequencer file
// then start the sequencer
cdms.daq.utilities.get(url).then(function() {
// call customscript
var cmd_str = '?customscript=' + scriptName_str + '&redir=.';
return cdms.daq.utilities.get(baseURL_str + cmd_str);
}).then(function() {
return checkScriptLoop(uid);
}).then(function() {
var load_str = baseURL_str + '/SEQ/?cmd=Load+Script';
console.log('execute promise for url ', load_str);
return cdms.daq.utilities.get(load_str);
}).then(function() {
var load_str = '/SEQ/?fs=' + seqFile_str + '&dir=&cmd=Load';
console.log('execute promise for url ', load_str);
return cdms.daq.utilities.get(baseURL_str + load_str);
}).then(function() {
var startSeq_str = '/SEQ/?cmd=Start+Script¶ms=1';
console.log('execute promise for url ', startSeq_str);
return cdms.daq.utilities.get(baseURL_str + startSeq_str);
}).catch(function(error) {
alert(error);
}).then(function() {
window.frames['flash-frame'].location = baseURL_str + '/SEQ/';
}); |
|
18 Nov 2015, Amy Roberts, Info, synchronous ajax deprecated
|
> I checked again on browser compatibility:
>
> el6: firefox 38 - ok, google-chrome 27 - no
> el7: firefox 38 - ok, google-chrome 46 - ok
> ubuntu: firefox 42 - ok
>
> mac os, windows - we say "latest firefox or google-chrome is required", then - ok
>
> So we are probably okey with using javascript Promises with MIDAS...
It looks like this does mean that people using RHEL6 won't have the option of chrome - can they update chrome?
One option is to include a polyfill library like Lie (https://github.com/calvinmetcalf/lie). |
|
18 Nov 2015, Konstantin Olchanski, Info, synchronous ajax deprecated
|
> > Why don't you post the functions here so that we can have a look?
> Here is (1) my promisified HTTP request function and (2) a function that uses the returned promises to build an asynchronous, sequential chain of requests to Midas.
>
> In addition to promisifying HTTP requests to Midas, I wanted the Promise.resolve from this function to always return valid JSON.
Thank you very much for posting this code. It is very similar to what I just wrote this morning for the JSON-RPC client library. In my case, the JSON-RPC responses
are much more regular so error handling is simple: a) check HTTP response 200, b) check JSON-RPC reply parses into valid JSON (catch exception), c) check JSON-RPC error status.
> I also wanted the promise to reject if the response from mhttpd indicated
> failure - so that we wouldn't have to rewrite this error checking throughout the code.
Right now the JSON-RPC client library does not check the return status of MIDAS calls themselves, i.e. ODBGet("/nonexistant") will go to Promise.resolve() with
the MIDAS db_find_key() status DB_NO_KEY instead of Promise.reject(). So some error handling in Promise.resolve() is still required.
I am thinking how to make these calls go to the error handler automatically, but there is no obvious solution for ODBMGet(["/runinfo", "/nonexistant"]) - the first path
is a success, the second is a failure, do I fail the entire transaction (i.e. with a JSON-RPC error)? Same for JSON-RPC batch transactions - if one transaction
in the batch fails, do I Promise.reject() all of them?
I guess I could "split hairs" and create a separate Promise for each "atomic" transaction, the Promise mechanism does seem to support that,
but this will create more complexity than I feel comfortable with.
Please take a look at the branch feature/js_promise - mjsonrpc_call() is Promisified (resources/mhttpd.js) and the db_copy() example is Promisified (examples/javascript1/example.html)
K.O. |
|
19 Nov 2015, Amy Roberts, Info, synchronous ajax deprecated
|
> Right now the JSON-RPC client library does not check the return status of MIDAS calls themselves, i.e. ODBGet("/nonexistant") will go to Promise.resolve() with
> the MIDAS db_find_key() status DB_NO_KEY instead of Promise.reject(). So some error handling in Promise.resolve() is still required.
> I am thinking how to make these calls go to the error handler automatically, but there is no obvious solution for ODBMGet(["/runinfo", "/nonexistant"]) - the first path
> is a success, the second is a failure, do I fail the entire transaction (i.e. with a JSON-RPC error)? Same for JSON-RPC batch transactions - if one transaction
> in the batch fails, do I Promise.reject() all of them?
Generally, I'd prefer a grouped-request like ODBMGet to return an array of
Promises. This way, I get to decide how to handle the request responses. While
I do have cases where I use Promise.all(...), most of my current code would want
to treat each response individually.
But as you point out, my approach differs from the Midas approach significantly.
While I've set up my queries to reject on responses like DB_NO_KEY, the function
mjsonrpc_send_request in mhttpd.js tests purely for a successful http
request.
Since ODBMGet makes a *single* http call, I'd naively lean toward having it
return a single Promise. Presumably, one that resolves if the http request
"goes through" and rejects if the http request fails.
My perspective may not be the useful one to consider, though. If other users
expect an array of promises returned from ODBMGet, definitely feel free to
ignore my thoughts on the matter.
If people really want ODBMGet to return an array of promises, one way to do it
would be to have a 'get' function that only cares about the success of the http
request, and a separate 'response checking' function that validates the
response. ODBMGet can use these two functions together to return an array of
Promises:
#########
mhttpd.js
#########
function get(url) {
// return a promise that resolves if the http request returns status=200
// reject if the http request does anything else
}
function checkResponse(response) {
// return a promise that resolves for "good" responses
// and rejects for "bad response"
}
function ODBMGet(path_arr) {
var url = // build url from path_arr
// syntax get().then(A).catch(B) means
// if the http request goes through, A is executed
// if the http request fails, B is executed
// ;
// for get, failure means no successful reply at all
// so ODBMGet should return an array of rejected Promises
get(url).then(function(response) {
response_arr = // split the response
return response_arr.map(checkResponse)
}).catch(function(err) {
return path_arr.forEach(function() {
return Promise.reject(err)
})
})
}
#########
user code
#########
// here the Promises are treated individually
var response_arr = ODBMGet([path1, path2, path3])
response_arr.forEach(function(thisResponse) {
thisResponse.then( /* do something */ )
.catch( /* do something else */ )
})
// and here the failure of a single promise in the array
// determines the code that's executed next
var required_arr = ODBMGet([pathA, pathB, pathC])
Promise.all(required_arr).then( /* do something */ )
.catch( /* do something else */ ) |
|
20 Nov 2015, Konstantin Olchanski, Info, documented, merged: midas JSON-RPC interface
|
> The JSON RPC branch has been merged into main MIDAS.
The interface is now mostly documented, go here: https://midas.triumf.ca/MidasWiki/index.php/Mjsonrpc
Documentation for individual javascript functions in mhttpd.js not merged into the MIDAS documentation yet, because the API is being converted to the Javascript Promise
pattern (git branch feature/js_promise).
The functions available from mhttpd.js are documented via doxygen, also linked from the mjsonrpc wiki page.
K.O. |
 20 Nov 2015, Konstantin Olchanski, Info, midas wiki doxygen documentation links
|
I updated the links on the midas wiki to the doxygen-generated documentation for MIDAS that you
get after running "git clone midas; cd midas; make dox; firefox html/index.html".
Correct link is:
https://daq.triumf.ca/~daqweb/doc/midas-devel/html/
This takes you to a daily/nightly generated snapshot of the midas develop branch and the
generated documentation with full call graphs.
Previous links were deficient is different ways:
- referred to http://ladd00 instead of https://daq
- referred to wrong path ~daqweb/doc/midas instead of ~daqweb/doc/midas-devel
- referred to the obsolete doxygen generator in midas/doc/html instead of midas/html.
If wrong links are still present on the midas wiki, please let us know and we will fix them.
K.O. |
|
26 Nov 2015, Konstantin Olchanski, Info, browser compatibility test: synchronous ajax deprecated
|
> > I checked again on browser compatibility:
> >
> > el6: firefox 38 - ok, google-chrome 27 - no
> > el7: firefox 38 - ok, google-chrome 46 - ok
> > ubuntu: firefox 42 - ok
> >
> > mac os, windows - we say "latest firefox or google-chrome is required", then - ok
> >
> > So we are probably okey with using javascript Promises with MIDAS...
>
> [too bad about chrome on SL6] ... include a polyfill library like Lie (https://github.com/calvinmetcalf/lie).
Results of cross-browser testing.
MacOS 10.10.5:
google-chrome 46: Promise ok, Programs page ok, Overlay ok
firefox 42: Promise ok, Programs page ok, Overlay ok.
safari 9.0.1: Promise ok, Programs page ok, Overlay ok.
Linux SL6.7:
google-chrome 27: "Promise not defined"
firefox 38.4.0: Promise ok, Programs page ok, Overlay ok.
konqueror 4.3.4: no go "Can't find variable: JSON"
chromium/google-chrome 38: ok
Linux SL7.1:
google-crome 46: ok
firefox 38.4.0: ok
konqueror 4.10.5: no go, mhttpd.js parse error
Conclusion:
1) firefox is good everywhere
2) google-chrome is good on Mac, Windows and el7 Linux
3) chromium/google-chrome 38 is good on el6 Linux (SL6/CentOS6).
We are good to proceed with adopting the Promise API for MIDAS.
K.O. |
|
27 Nov 2015, Konstantin Olchanski, Info, synchronous ajax deprecated
|
> > I checked again on browser compatibility:
> >
> > el6: firefox 38 - ok, google-chrome 27 - no
>
> It looks like this does mean that people using RHEL6 won't have the option of chrome - can they update chrome?
>
It turns out that google-chrome 38 is available for RHEL6/SL6 via an old chromium build. Promises are supported (passes my tests).
See here:
http://www.if-not-true-then-false.com/2013/install-chromium-on-centos-red-hat-rhel
This is where I got the working chromium 38 (no explanation of why there are no newer builds)
http://people.centos.org/hughesjr/chromium/6/x86_64/RPMS/
There appear to be newer builds here: (but I will not test them)
http://install.linux.ncsu.edu/pub/yum/itecs/public/chromium-dev/rhel6/x86_64/
My SL6 google-chrome and chromium instructions:
https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Install_Google_Chrome_web_browser_.2864-bit_SL6.29
K.O. |
|
27 Nov 2015, Konstantin Olchanski, Info, updated: note on midas history
|
(update: resolve all FIXMEs, document the breakup of "structured banks")
This note documents the workings of the midas history.
There is 2 separate history sections: equipment history and links history.
* is equipment history enabled?
For each equipment, history is controlled by the value of /eq/xxx/common/period:
0 = history disabled
1 = history is enabled
>1 = history is enabled, throttled down
The throttling is implemented in log_history()/watch_history() by this algorithm:
the very first history event is recorded, then all changed to the data are ignored until
"period" seconds has elapsed. Then the next history event will be recorded, and following
changes will be ignored until "period" second elapses, and so forth. Period value "1" has
special meaning - there is no throttling, all history events are logged.
If equipment history is enabled, history events are created by parsing the content of /eq/xxx/variables.
* what is history events?
A "history event" is a history atomic unit of data. Associated with each history event is a timestamp (unix time),
a name (limited to NAME_LENGTH in the old history) and a list of history tags that describe the individual data
values inside the history event.
When making history plots in mhttpd, for each curve on the plot, one selects a history event (from the list
of currently active events, recently active events or the list of all events that ever existed), then from the list of tags
inside the history event one selects the particular variable that will be plotted.
In the old MIDAS history, all history events are written into one history file (.hst file + optional .def and .idx event definition and time index files
which can be/are regenerated automatically from the .hst file). History events are identified by 16-bit history event IDs, the persistent mapping
from history event names and the 16-bit history event IDs is stored in ODB /History/Events. In addition the list of all known history event tags is
stored in ODB /History/Tags. For per-equipment history, the 16-bit history event ID is the value of ODB "/eq/xxx/common/event id".
In the SQL history (MySQL, SQLITE, etc), each history event is an SQL table. The history event tags are the SQL table columns.
In the new FILE history, each history event is written into a separate file, tag definition are recorded in text formal in the file header, history event
data is appended to the file in binary format (fixed record size). If the history event definition is changed, a new file will be started.
* how are history events constructed?
The mlogger creates history events in open_history() by parsing ODB /eq/xxx/variables. Each ODB entry under "variables" is referred to as a "variable".
Each variable can be a single ODB value, an array of ODB values, or a subdirectory (corresponding to TID_STRUCT structured data banks). As each variable
is processed, one or more tags are created to describe it. Single ODB values will generally produce a single tag, while arrays can produce
one single tag - describing the whole array - or multiple tags - one per array element - depending whether the array is "named" or not.
The code can generate two types of history:
- "per-equipment" history will have the tags for all variables concatenated together into one single history event
- "per-variable" history will have one history event defined for each variable. Inside could be one tag - for single odb values and unnamed arrays - or multiple tags - for named arrays and structured data
banks.
Per-equipment history is the original MIDAS history implementation.
Per-variable history was added to permit efficient data storage in SQL tables. It's initial implementation used 1 ODB hotlink for each variable and it was easy to exceed the maximum permitted number of
ODB hotlinks (db_open_record()).
To reduce consumption of hotlinks, db_watch() has been implemented and now per-variable history only uses 1 ODB hotlink per equipment.
With db_watch, per-equipment history is no longer available. per-variable history is the new default (and the only option).
* how are the history event tags constructed?
(quirk - single odb values are treated as arrays of length "1")
FIXME: single odb values should be treated as such, /eq/xxx/settings/names should not be applied
(quirk - "string" ODB entries are not permitted)
FIXME - single odb values of type TID_STRING should be possible with SQL, FILE and MIDAS history. arrays of strings is impossible "struct TAG" does not have a data field for string length - only n_data and
item length implied through it's TID.
History event tags are constructed in the mlogger add_equipment().
For variables of type TID_KEY (subdirectories, corresponding to TID_STRUCT structured banks), one tag is generated for each subdirectory entry. Tag names for /eq/xxx/var/aaa/bbb will be "aaa_bbb".
(with an underscore).
FIXME: subdirectory entries of type TID_KEY and TID_LINK should be explicitly forbidden.
FIXME: TID_KEY could be supported by replacing db_get_data() with db_get_record() in watch_history().
FIXME: TID_LINK could be supported by adding db_watch() on the link target.
For named arrays, individual tags are generated for each array element. Tag names are taken from the names array. For empty tag names (empty names array), tags are "aaa_0", "aaa_1", etc (for
/eq/xxx/var/aaa). For "single names" arrays, tag names have the variable name appended (with a space), for /eq/xxx/var/aaa and an empty names array, tags will be "aaa_0 aaa", "aaa_1 aaa", etc. For
populated names array, the tags will be "name0 aaa", "name1 aaa", etc.
For unnamed arrays and single odb variables (in ODB, single odb variables are arrays of length 1), a single tag is generated.
For TID_LINK variables what happens? FIXME!
FIXME: support TID_LINK variables by correctly parsing the link target and setting a db_watch() on the link target.
Named arrays have a "Names" entry in /eq/xxx/settings. For example, to add names to /eq/xxx/var/aaa, create a string array "/eq/xxx/settings/names aaa". The names array should be at least as long as
the corresponding data array. Individual entries in the names array can be left blank (tag names will be "aaa_0", "aaa_1", etc). Duplicate tag names are not permitted.
A single "Names" entry can be created to name all arrays in variables with the same names ("single names"). Create /eq/xxx/settings/names" and arrays /eq/xxx/var/aaa and /eq/xxx/var/bbb will have
history tags "name0 aaa", "name1 aaa", "name0 bbb", "name1 bbb", etc. If "names" are left blank, tag names will be "aaa_0 aaa", "aaa_1 aaa", "bbb_0 bbb", "bbb_1 bbb", etc.
In the mhttpd variables viewer, "single name" arrays are displayed in a 2D table.
* /history/links history
History events are created for each entry under /history/links.
Two types of links are permitted:
/history/links/aaa is a link to a subdirectory: db_watch() is setup to watch this subdirectory, tags are created for each subdirectory entry (1 tag per entry). There is no possibility for naming array elements, so 1 tag per array, regardless of the number of elements.
/history/links/bbb is a subdirectory with links to odb values: db_watch is setup to watch each link target, tags are created for each link (1 tag per link). tag name is the link name (NOT the target name). There is no possibility for naming array elements.
FIXME: Mixing links and subdirectories is not permitted, but could be done - additional db_watch() will need to be done on any links.
Update period history events created for /history/links is controlled by entries in "/history/links periods". Numeric values of periods are same as for equipment histories. Numeric value 0 disables the history for a particular event.
K.O. |
|
10 Dec 2015, Stefan Ritt, Info, Small change in loading .odb files
|
A small change in loading .odb files has been implemented. When you load an array from a .odb file, the indices in each line were not evaluated, only the complete array was loaded. In our experiment we need however to load only a few values, like some HV values for some channels but leaving the other values as they are. I changed slightly the code of db_paste() to correctly evaluate the index in each line of the .odb file. This way one can write for example following .odb file:
[/Equipment/HV/Variables]
Demand = FLOAT[256] :
[10] 100.1
[11] 100.2
[12] 100.3
[13] 100.4
[14] 100.5
[15] 100.6
then load it in odbedit via the "load" command, and then only change channels 10-15.
Stefan |
|
09 Mar 2016, Konstantin Olchanski, Info, /Experiment/Edit on start/Edit Run number
|
The MIDAS documentation here:
https://midas.triumf.ca/MidasWiki/index.php/Edit-on-start_Parameters
is missing informaiton about this ODB entry:
/Experiment/Edit on start/Edit Run number (TID_BOOL)
This is what it does in mhttpd:
a) if it exists and is of type TID_BOOL and set to "n", run number is not editable
b) "Edit run number" itself is hidden, will not show up on the web page
This is what it does in odbedit:
a) it is hidden, will not show up in the list of run parameters
b) it's value has no effect, run number is always editable.
K.O. |
|
10 Mar 2016, Thomas Lindner, Info, New rootana forum | rootana web display tools
|
We have started a new elog for discussions of the ROOTANA MIDAS analyzer package
[1], which is used at TRIUMF and elsewhere for quick displays of MIDAS data.
The forum is available here
https://midas.triumf.ca/elog/Rootana
I would note that we have recently finished implementing a system in rootana for
easy web displays of MIDAS data, using ROOT's THttpServer to post histograms.
Details on this new scheme are here
https://midas.triumf.ca/elog/Rootana/1
and
https://midas.triumf.ca/MidasWiki/index.php/Rootana_javascript_displays
Please sign up for the forum if you are interested in getting ROOTANA-related
discussions.
Thomas
[1] https://midas.triumf.ca/MidasWiki/index.php/ROOTANA |
|
22 Mar 2016, Konstantin Olchanski, Info, emacs web-mode.el
|
For those who use emacs to edit web pages - the built-in CSS and Javascript modes seem to work
just fine for editing files.css and files.js, but the built-in html modes fall flat on modern web pages
which contain a mix of html, javascript inside <script> tags and javascript inside button "onclick"
attributes.
So I looked at several emacs "html5 modes" and web-mode.el works well for me - html is indented
correctly (default indent level is easy to change), javascript inside <script> tags is indented
correctly (default indent level is easy to change), javascript inside "onclick" attributes has to be
indented manually.
web-mode code repository and instructions are here, the author is very responsive and fixed my
one request (permit manual intentation of javascript inside html attributes):
https://github.com/fxbois/web-mode
I now edit the html files in the MIDAS repository using these emacs settings:
8s-macbook-pro:web-mode 8ss$ more ~/.emacs
(setq-default indent-tabs-mode nil)
(setq-default tab-width 3)
(add-to-list 'load-path "~/git/web-mode")
(require 'web-mode)
(add-to-list 'auto-mode-alist '("\\.html\\'" . web-mode))
(setq web-mode-markup-indent-offset 2)
(setq web-mode-css-indent-offset 2)
(setq web-mode-code-indent-offset 3)
(setq web-mode-script-padding 0)
(setq web-mode-attr-indent-offset 2)
8s-macbook-pro:web-mode 8ss$
K.O. |
|
11 May 2016, Thomas Lindner, Info, MacOS 10.11 (El Capitan) openssl compilation errors
|
I recently upgraded my macbook to MacOS 10.11. The compilation of MIDAS failed after the upgrade,
complaining about
gcc -c -g -O2 -Wall <snip> src/mongoose.c
src/mongoose.c:322:10: fatal error: 'openssl/ssl.h' file not found
It seems that MacOS has now fully removed openssl header files (they were deprecated for a while). There
seems to be some notes on that here
http://lists.apple.com/archives/macnetworkprog/2015/Jun/msg00025.html
Konstantin suggested installing open-source builds of openssl using MacPorts. I did that and MIDAS
compiled fine. I documented the procedure here:
https://midas.triumf.ca/MidasWiki/index.php/Installation/Compilation_problems#MacOS_10.11_.28El_Capitan.2
9_openssl_errors |
|
12 May 2016, Stefan Ritt, Info, MacOS 10.11 (El Capitan) openssl compilation errors
|
> I recently upgraded my macbook to MacOS 10.11. The compilation of MIDAS failed after the upgrade,
> complaining about
>
> gcc -c -g -O2 -Wall <snip> src/mongoose.c
> src/mongoose.c:322:10: fatal error: 'openssl/ssl.h' file not found
>
> It seems that MacOS has now fully removed openssl header files (they were deprecated for a while). There
> seems to be some notes on that here
>
> http://lists.apple.com/archives/macnetworkprog/2015/Jun/msg00025.html
>
> Konstantin suggested installing open-source builds of openssl using MacPorts. I did that and MIDAS
> compiled fine. I documented the procedure here:
>
> https://midas.triumf.ca/MidasWiki/index.php/Installation/Compilation_problems#MacOS_10.11_.28El_Capitan.2
> 9_openssl_errors
The MIDAS Wiki page points to https://guide.macports.org/ which covers OSX up to 10.9. Installers for 10.10 and the current 10.11
(El Captain) can be found here: https://www.macports.org/install.php
Stefan |
|
17 May 2016, Konstantin Olchanski, Info, openssl situation, MacOS 10.11 (El Capitan) openssl compilation errors
|
> I recently upgraded my macbook to MacOS 10.11.
> [ and midas would not compile ]
> It seems that MacOS has now fully removed openssl ...
My read of tea leaves - the macos version of openssl was so old it was almost useless, did not support any of the modern HTTPS
features. So to use mhttpd with https you pretty much had to install openssl from macports anyway. For macos 10.11 maybe they
looked at upgrading to newer version, but since the openssl kerfuffle last year, there is several forks of openssl (the OpenBSD fork
named libressl is the best, IMO), so rather than picking and choosing, they deleted the whole thing.
Now back to MIDAS.
We use the mongoose web server module and I have expected by now for them to make a move on improving HTTPS support, but no
move happened.
Right now mongoose support OpenSSL only (I would expect the OpenBSD LibreSSL fork to work to of the box, too). Other then that,
they have:
a) their own mickey-mouse https library (krypton) which does not support any modern cryptography (RC4 only - when RC4 is known to
be useless).
b) an adapter library (polar) for interfacing with PolarSSL (mbedtls)
At this point I would rather abandon the implicit dependency on the system-provided openssl and have an explicit dependancy on a
modern https crypto library.
Option (b) would work for us -
1) add "git clone mbedtls; cd mbedtls; make" to midas build instructions
2) add polarssl_compat.c to midas git (from cessanta/polar repo)
3) retest mhttpd against ssllabs https scanner, retest against all web browsers.
The downside of this route is loss of automatic nightly updates to the https crypto library (for better or for worse).
K.O.
P.S. Because on MacOS use of openssl from macports is pretty much required, it should be moved from the "tricks" page to the
standard midas installation instructions ("install required packages"). |
|
13 Jun 2016, Konstantin Olchanski, Info, running mhttpd on port 443
|
mhttpd running as non-root cannot bind to standard https port 443. By default, mhttpd uses port
8443 and it works just fine, but some applications such as the SSLlabs https tester insist on using
port 443.
To connect mhttpd with port 443, I use the tcpproxy package from
git://git.spreadspace.org/tcpproxy.git
./tcpproxy -D -U -p 443 -r localhost4 -o 8443
(you can run this from rc.local)
(to remember, for best security one should run mhttpd behind an industry-standard https proxy)
K.O. |
|