Thank you for reporting this problem:

a) ODB key *names* are restricted to 31 characters (32 bytes, last byte is a NUL), not 256 characters.
b) ODB string length is unlimited (32-bit length field)
c) ODB C API "db_get_value" & co require fixed length buffer and most users of this API provide a 256-byte fixed buffer for strings, some of them also do not 
check the status code, resulting in silent truncation. (I think the ODB functions themselves report truncation to midas.log, so not completely silent).

We try to fix this where we must - but it is cumbersome with the current ODB API - as in your fix on has to:
- get the ODB key, extract size
- allocate buffer
- call db_get_value() & co
- use the data
- remember to free the buffer on each and every return path

The first three steps could become one if we had an ODB "get_data" function that automatically allocated the data buffer.

But the main source of bugs will be the last step - remember to free the buffer, always.

P.S.

We are not alone in pondering how to do this best. If you want to see it "done right",
read the fresh-off-the-presses book "Go Programming Language" by Alan Donovan and Brian Kernighan,
http://www.gopl.io/

Brian Kernighan is the "K" in K&R "C programming language", still around and kicking, now at Google.
Sadly the "R" passed away in 2011 - http://www.nytimes.com/2011/10/14/technology/dennis-ritchie-programming-trailblazer-dies-at-70.html

K.O.

> Both the /Script and /CustomScript trees in the ODB allow users to trigger a 
> script via Midas - which silently truncates command strings longer than 
> 256 characters.
> 
> I'd prefer that Midas place no limit on string length.  Failing that, it would be
> helpful to have character limits called out in the documentation 
> (https://midas.triumf.ca/MidasWiki/index.php//Script_ODB_tree#.3Cscript-name.3E_key_or_subtree,
> https://midas.triumf.ca/MidasWiki/index.php//Customscript_ODB_tree).
> 
> As far as I can tell, odb.c allows arbitrarily large strings in the ODB data.  
> (Although key *names* are restricted to 256 characters.)  I've submitted one 
> possible version of an arbitrary-length exec_script() as a pull request 
> (https://bitbucket.org/tmidas/midas/pull-requests/).
> 
> Am I misunderstanding any critical pieces?  Does Midas intentionally treat 
> strings in the ODB as limited to 256 characters?

> > The final bits of the JSON-RPC API to MIDAS are committed.

JSON-RPC methods are now provided for all old ODBxxx() javascript functions, except ODBGetMsg().

The currently present RPC methods are sufficient to write the MIDAS "programs" and "alarms" pages
purely in HTML+Javascript (see the git branch feature/mhttpd_js). These pages can be served i.e. by apache httpd
with midas mhttpd only required to service the RPC requests.

Please see .../examples/javascript1/example.html on how to use the new RPC methods.

K.O.

P.S. Note how many examples use the generic mjsonrpc_call() because I did not write the corresponding
javascript functions - I wore out the cut-and-paste button on my keyboard. All are welcome to contribute
the missing functions, post them here or email them to me, I will commit them to midas git.

> > I've seen that a similar question has been asked in 2011 but I'll ask again in 
> > case there are any updates. Is there any way to write 64-bit data words to MIDAS 
> > banks (other than breaking them up in to two 32-bit words, such as 2 DWORDs) 
> > currently? And if not, is there any plan to introduce this feature in the future?
> 
> There is no "breaking them up" as such, you can treat a midas bank as a char* array
> and store arbitrary data inside. In this sense, "there is no need" for a special 64-bit bank type.
> 
> For endian-ness conversion (if such things still matter, big-endian PPC CPUs still exist), single 64-bit 
> word converts the same as two 32-bit words, so here also "there is no need", once can use banks of 
> DWORD with equal effect.
> 
> The above applies equally to 64-bit integers and 64-bit double-precision IEEE-754 floating point 
> numbers.
> 
> But specifically for 64-bit values, such as float64, there is a big gotcha.
> 
> The MIDAS banks structure goes to great lengths to make sure each data type is correctly aligned,
> and gets it exactly wrong for 64-bit quantities - all because the bank header is three 32-bit words.
> 
> bankhheader1
> bh2
> bh3
> bankdata1 <--- misaligned
> ...
> bankdataN
> bh1
> bh2
> bh3
> banddata1 <--- aligned
> ... etc
> 
> So we could introduce QWORD banks today, but inside the midas file, they will be misaligned defeating 
> the only purpose of adding them.
> 
> I guess the misalignement could be cured by adding dummy words, dummy banks, dummy bank 
> headers, etc.
> 
> I figure this problem dates all the way bank where alignement to 16-bits was just getting important. 
> Today, in the VME word, I have to align things on 128-bit boundaries (for 2eSST 2x2 DWORD transfers).
> 
> So back to your question, what advantage do you see in using a QWORD bank instead of putting the 
> same data in a DWORD bank?
> 
> K.O.


Thanks very much for your reply. I have implemented your suggestion of treating the 64-bit array as a 32-bit 
array for the bank write/read and this solution is working for me.

Thanks again for your help.

> > I've seen that a similar question has been asked in 2011 but I'll ask again in 
> > case there are any updates. Is there any way to write 64-bit data words to MIDAS 
> > banks (other than breaking them up in to two 32-bit words, such as 2 DWORDs) 
> > currently? And if not, is there any plan to introduce this feature in the future?
> 
> There is no "breaking them up" as such, you can treat a midas bank as a char* array
> and store arbitrary data inside. In this sense, "there is no need" for a special 64-bit bank type.
> 
> For endian-ness conversion (if such things still matter, big-endian PPC CPUs still exist), single 64-bit 
> word converts the same as two 32-bit words, so here also "there is no need", once can use banks of 
> DWORD with equal effect.
> 
> The above applies equally to 64-bit integers and 64-bit double-precision IEEE-754 floating point 
> numbers.
> 
> But specifically for 64-bit values, such as float64, there is a big gotcha.
> 
> The MIDAS banks structure goes to great lengths to make sure each data type is correctly aligned,
> and gets it exactly wrong for 64-bit quantities - all because the bank header is three 32-bit words.
> 
> bankhheader1
> bh2
> bh3
> bankdata1 <--- misaligned
> ...
> bankdataN
> bh1
> bh2
> bh3
> banddata1 <--- aligned
> ... etc
> 
> So we could introduce QWORD banks today, but inside the midas file, they will be misaligned defeating 
> the only purpose of adding them.
> 
> I guess the misalignement could be cured by adding dummy words, dummy banks, dummy bank 
> headers, etc.
> 
> I figure this problem dates all the way bank where alignement to 16-bits was just getting important. 
> Today, in the VME word, I have to align things on 128-bit boundaries (for 2eSST 2x2 DWORD transfers).
> 
> So back to your question, what advantage do you see in using a QWORD bank instead of putting the 
> same data in a DWORD bank?
> 
> K.O.

> I've seen that a similar question has been asked in 2011 but I'll ask again in 
> case there are any updates. Is there any way to write 64-bit data words to MIDAS 
> banks (other than breaking them up in to two 32-bit words, such as 2 DWORDs) 
> currently? And if not, is there any plan to introduce this feature in the future?

There is no "breaking them up" as such, you can treat a midas bank as a char* array
and store arbitrary data inside. In this sense, "there is no need" for a special 64-bit bank type.

For endian-ness conversion (if such things still matter, big-endian PPC CPUs still exist), single 64-bit 
word converts the same as two 32-bit words, so here also "there is no need", once can use banks of 
DWORD with equal effect.

The above applies equally to 64-bit integers and 64-bit double-precision IEEE-754 floating point 
numbers.

But specifically for 64-bit values, such as float64, there is a big gotcha.

The MIDAS banks structure goes to great lengths to make sure each data type is correctly aligned,
and gets it exactly wrong for 64-bit quantities - all because the bank header is three 32-bit words.

bankhheader1
bh2
bh3
bankdata1 <--- misaligned
...
bankdataN
bh1
bh2
bh3
banddata1 <--- aligned
... etc

So we could introduce QWORD banks today, but inside the midas file, they will be misaligned defeating 
the only purpose of adding them.

I guess the misalignement could be cured by adding dummy words, dummy banks, dummy bank 
headers, etc.

I figure this problem dates all the way bank where alignement to 16-bits was just getting important. 
Today, in the VME word, I have to align things on 128-bit boundaries (for 2eSST 2x2 DWORD transfers).

So back to your question, what advantage do you see in using a QWORD bank instead of putting the 
same data in a DWORD bank?

K.O.

I've seen that a similar question has been asked in 2011 but I'll ask again in 
case there are any updates. Is there any way to write 64-bit data words to MIDAS 
banks (other than breaking them up in to two 32-bit words, such as 2 DWORDs) 
currently? And if not, is there any plan to introduce this feature in the future?

Many thanks,
Tom

A small change in loading .odb files has been implemented. When you load an array from a .odb file, the indices in each line were not evaluated, only the complete array was loaded. In our experiment we need however to load only a few values, like some HV values for some channels but leaving the other values as they are. I changed slightly the code of db_paste() to correctly evaluate the index in each line of the .odb file. This way one can write for example following .odb file:

[/Equipment/HV/Variables]
Demand = FLOAT[256] :
[10] 100.1
[11] 100.2
[12] 100.3
[13] 100.4
[14] 100.5
[15] 100.6

then load it in odbedit via the "load" command, and then only change channels 10-15.

Stefan

Both the /Script and /CustomScript trees in the ODB allow users to trigger a 
script via Midas - which silently truncates command strings longer than 
256 characters.

I'd prefer that Midas place no limit on string length.  Failing that, it would be
helpful to have character limits called out in the documentation 
(https://midas.triumf.ca/MidasWiki/index.php//Script_ODB_tree#.3Cscript-name.3E_key_or_subtree,
https://midas.triumf.ca/MidasWiki/index.php//Customscript_ODB_tree).

As far as I can tell, odb.c allows arbitrarily large strings in the ODB data.  
(Although key *names* are restricted to 256 characters.)  I've submitted one 
possible version of an arbitrary-length exec_script() as a pull request 
(https://bitbucket.org/tmidas/midas/pull-requests/).

Am I misunderstanding any critical pieces?  Does Midas intentionally treat 
strings in the ODB as limited to 256 characters?

> The final bits of the JSON-RPC API to MIDAS are committed.

Here is example conversion of the function "generate midas message" from old-style AJAX to JSON-RPC:

before (mhttpd.cxx):

   /* process "jgenmsg" command */
   if (equal_ustring(getparam("cmd"), "jgenmsg")) {

      if (getparam("facility") && *getparam("facility"))
         strlcpy(facility, getparam("facility"), sizeof(facility));
      else
         strlcpy(facility, "midas", sizeof(facility));
      
      if (getparam("user") && *getparam("user"))
         strlcpy(user, getparam("user"), sizeof(user));
      else
         strlcpy(user, "javascript_commands", sizeof(user));
      
      if (getparam("type") && *getparam("type"))
         type = atoi(getparam("type"));
      else
         type = MT_INFO;

      if (getparam("msg") && *getparam("msg")) {
         cm_msg1(type, __FILE__, __LINE__, facility, user, "%s", getparam("msg"));
      }

      show_text_header();
      rsputs("Message successfully created\n");
      return;
   }

after: (mjsonrpc.cxx)

static MJsonNode* js_cm_msg1(const MJsonNode* params)
{
   if (!params) {
      MJSO *doc = MJSO::I();
      doc->D("Generate a midas message using cm_msg1()");
      doc->P("facility?", MJSON_STRING, "message facility, default is \"midas\"");
      doc->P("user?", MJSON_STRING, "message user, default is \"javascript_commands\"");
      doc->P("type?", MJSON_INT, "message type, MT_xxx from midas.h, default is MT_INFO");
      doc->P("message", MJSON_STRING, "message text");
      doc->R("status", MJSON_INT, "return status of cm_msg1()");
      return doc;
   }

   MJsonNode* error = NULL;

   const char* facility = mjsonrpc_get_param(params, "facility", &error)->GetString().c_str();
   const char* user = mjsonrpc_get_param(params, "user", &error)->GetString().c_str();
   int type = mjsonrpc_get_param(params, "type", &error)->GetInt();
   const char* message = mjsonrpc_get_param(params, "message", &error)->GetString().c_str(); if (error) return error;

   if (strlen(facility)<1)
      facility = "midas";
   if (strlen(user)<1)
      user = "javascript_commands";
   if (type == 0)
      type = MT_INFO;

   int status = cm_msg1(type, __FILE__, __LINE__, facility, user, "%s", message);

   return mjsonrpc_make_result("status", MJsonNode::MakeInt(status));
}

With the corresponding javascript-side stabs:

before:

function ODBGenerateMsg(type,facility,user,msg)
{
   var request = XMLHttpRequestGeneric();

   var url = ODBUrlBase + '?cmd=jgenmsg';
   url += '&type='+type;
   url += '&facility='+facility;
   url += '&user='+user;
   url += '&msg=' + encodeURIComponent(msg);
   request.open('GET', url, false);
   request.send(null);
   return request.responseText;
}

after:

function mjsonrpc_cm_msg(message, type, id) {
   /// \ingroup mjsonrpc_js
   /// Get values of ODB variables
   ///
   /// RPC method: "cm_msg1"
   ///
   /// \code
   /// mjsonrpc_cm_msg("this is a new message").then(function(rpc) {
   ///    var req    = rpc.request; // reference to the rpc request
   ///    var id     = rpc.id;      // rpc response id (should be same as req.id)
   ///    var status = rpc.result.status;  // return status of MIDAS cm_msg1()
   ///    ...
   /// }).catch(function(error) {
   ///    mjsonrpc_error_alert(error);
   /// });
   /// \endcode
   /// @param[in] message Text of midas message (string)
   /// @param[in] type optional message type, one of MT_xxx. Default is MT_INFO (integer)
   /// @param[in] id optional request id (see JSON-RPC specs) (object)
   /// @returns new Promise
   ///
   var req = new Object();
   req.message = message;
   if (type)
      req.type = type;
   return mjsonrpc_call("cm_msg1", req, id);
}

K.O

The final bits of the JSON-RPC API to MIDAS are committed. The API uses the Javascript Promise mechanism (supported on all 
supported platforms - MacOS, Windows, Linux Ubuntu, el5, el6, el7).

Simple example for pasting the current run number into an html element:

mjsonrpc_db_get_values(["/runinfo/run number"]).then(function(rpc) {
   document.getElementById("run_number").innerHTML = rpc.response.data[0];
}).catch(function(error) {
   mjsonrpc_error_alert(error);
});

The documentation for the JSON-RPC API, including special quirks in JSON encoding of ODB data is here:
https://midas.triumf.ca/MidasWiki/index.php/Mjsonrpc

Documentation (with examples) for the related Javascript functions in mhttpd.js is here (via Doxygen):
https://daq.triumf.ca/~daqweb/doc/midas-devel/html/group__mjsonrpc__js.html

Examples of using all mhttpd.js functions is in .../examples/javascript1/example.html

The experimental git branch feature/mhttpd_js implements the MIDAS "programs" page purely in html and javascript,
go there to see all this new JSON and RPC stuff in action. See .../resources/programs.html.

K.O.

(update: resolve all FIXMEs, document the breakup of "structured banks")

This note documents the workings of the midas history.

There is 2 separate history sections: equipment history and links history.

* is equipment history enabled?

For each equipment, history is controlled by the value of /eq/xxx/common/period:

0 = history disabled
1 = history is enabled
>1 = history is enabled, throttled down

The throttling is implemented in log_history()/watch_history() by this algorithm:
the very first history event is recorded, then all changed to the data are ignored until
"period" seconds has elapsed. Then the next history event will be recorded, and following
changes will be ignored until "period" second elapses, and so forth. Period value "1" has
special meaning - there is no throttling, all history events are logged.

If equipment history is enabled, history events are created by parsing the content of /eq/xxx/variables.

* what is history events?

A "history event" is a history atomic unit of data. Associated with each history event is a timestamp (unix time),
a name (limited to NAME_LENGTH in the old history) and a list of history tags that describe the individual data
values inside the history event.

When making history plots in mhttpd, for each curve on the plot, one selects a history event (from the list
of currently active events, recently active events or the list of all events that ever existed), then from the list of tags
inside the history event one selects the particular variable that will be plotted.

In the old MIDAS history, all history events are written into one history file (.hst file + optional .def and .idx event definition and time index files
which can be/are regenerated automatically from the .hst file). History events are identified by 16-bit history event IDs, the persistent mapping
from history event names and the 16-bit history event IDs is stored in ODB /History/Events. In addition the list of all known history event tags is
stored in ODB /History/Tags. For per-equipment history, the 16-bit history event ID is the value of ODB "/eq/xxx/common/event id".

In the SQL history (MySQL, SQLITE, etc), each history event is an SQL table. The history event tags are the SQL table columns.

In the new FILE history, each history event is written into a separate file, tag definition are recorded in text formal in the file header, history event
data is appended to the file in binary format (fixed record size). If the history event definition is changed, a new file will be started.

* how are history events constructed?

The mlogger creates history events in open_history() by parsing ODB /eq/xxx/variables. Each ODB entry under "variables" is referred to as a "variable".

Each variable can be a single ODB value, an array of ODB values, or a subdirectory (corresponding to TID_STRUCT structured data banks). As each variable
is processed, one or more tags are created to describe it. Single ODB values will generally produce a single tag, while arrays can produce
one single tag - describing the whole array - or multiple tags - one per array element - depending whether the array is "named" or not.

The code can generate two types of history:
- "per-equipment" history will have the tags for all variables concatenated together into one single history event
- "per-variable" history will have one history event defined for each variable. Inside could be one tag - for single odb values and unnamed arrays - or multiple tags - for named arrays and structured data 
banks.

Per-equipment history is the original MIDAS history implementation.

Per-variable history was added to permit efficient data storage in SQL tables. It's initial implementation used 1 ODB hotlink for each variable and it was easy to exceed the maximum permitted number of 
ODB hotlinks (db_open_record()).

To reduce consumption of hotlinks, db_watch() has been implemented and now per-variable history only uses 1 ODB hotlink per equipment.

With db_watch, per-equipment history is no longer available. per-variable history is the new default (and the only option).

* how are the history event tags constructed?

(quirk - single odb values are treated as arrays of length "1")

FIXME: single odb values should be treated as such, /eq/xxx/settings/names should not be applied

(quirk - "string" ODB entries are not permitted)

FIXME - single odb values of type TID_STRING should be possible with SQL, FILE and MIDAS history. arrays of strings is impossible "struct TAG" does not have a data field for string length - only n_data and 
item length implied through it's TID.

History event tags are constructed in the mlogger add_equipment().

For variables of type TID_KEY (subdirectories, corresponding to TID_STRUCT structured banks), one tag is generated for each subdirectory entry. Tag names for /eq/xxx/var/aaa/bbb will be "aaa_bbb". 
(with an underscore).

FIXME: subdirectory entries of type TID_KEY and TID_LINK should be explicitly forbidden.
FIXME: TID_KEY could be supported by replacing db_get_data() with db_get_record() in watch_history().
FIXME: TID_LINK could be supported by adding db_watch() on the link target.

For named arrays, individual tags are generated for each array element. Tag names are taken from the names array. For empty tag names (empty names array), tags are "aaa_0", "aaa_1", etc (for 
/eq/xxx/var/aaa). For "single names" arrays, tag names have the variable name appended (with a space), for /eq/xxx/var/aaa and an empty names array, tags will be "aaa_0 aaa", "aaa_1 aaa", etc. For 
populated names array, the tags will be "name0 aaa", "name1 aaa", etc.

For unnamed arrays and single odb variables (in ODB, single odb variables are arrays of length 1), a single tag is generated.

For TID_LINK variables what happens? FIXME!

FIXME: support TID_LINK variables by correctly parsing the link target and setting a db_watch() on the link target.

Named arrays have a "Names" entry in /eq/xxx/settings. For example, to add names to /eq/xxx/var/aaa, create a string array "/eq/xxx/settings/names aaa". The names array should be at least as long as 
the corresponding data array. Individual entries in the names array can be left blank (tag names will be "aaa_0", "aaa_1", etc). Duplicate tag names are not permitted.

A single "Names" entry can be created to name all arrays in variables with the same names ("single names"). Create /eq/xxx/settings/names" and arrays /eq/xxx/var/aaa and /eq/xxx/var/bbb will have 
history tags "name0 aaa", "name1 aaa", "name0 bbb", "name1 bbb", etc. If "names" are left blank, tag names will be "aaa_0 aaa", "aaa_1 aaa", "bbb_0 bbb", "bbb_1 bbb", etc.

In the mhttpd variables viewer, "single name" arrays are displayed in a 2D table.

* /history/links history

History events are created for each entry under /history/links.

Two types of links are permitted:

/history/links/aaa is a link to a subdirectory: db_watch() is setup to watch this subdirectory, tags are created for each subdirectory entry (1 tag per entry). There is no possibility for naming array elements, so 1 tag per array, regardless of the number of elements.

/history/links/bbb is a subdirectory with links to odb values: db_watch is setup to watch each link target, tags are created for each link (1 tag per link). tag name is the link name (NOT the target name). There is no possibility for naming array elements.

FIXME: Mixing links and subdirectories is not permitted, but could be done - additional db_watch() will need to be done on any links.

Update period history events created for /history/links is controlled by entries in "/history/links periods". Numeric values of periods are same as for equipment histories. Numeric value 0 disables the history for a particular event.

K.O.

> > I checked again on browser compatibility:
> > 
> > el6: firefox 38 - ok, google-chrome 27 - no
>
> It looks like this does mean that people using RHEL6 won't have the option of chrome - can they update chrome?
>

It turns out that google-chrome 38 is available for RHEL6/SL6 via an old chromium build. Promises are supported (passes my tests).

See here:
http://www.if-not-true-then-false.com/2013/install-chromium-on-centos-red-hat-rhel

This is where I got the working chromium 38 (no explanation of why there are no newer builds)
http://people.centos.org/hughesjr/chromium/6/x86_64/RPMS/

There appear to be newer builds here: (but I will not test them)
http://install.linux.ncsu.edu/pub/yum/itecs/public/chromium-dev/rhel6/x86_64/

My SL6 google-chrome and chromium instructions:
https://www.triumf.info/wiki/DAQwiki/index.php/SLinstall#Install_Google_Chrome_web_browser_.2864-bit_SL6.29

K.O.

> > I checked again on browser compatibility:
> > 
> > el6: firefox 38 - ok, google-chrome 27 - no
> > el7: firefox 38 - ok, google-chrome 46 - ok
> > ubuntu: firefox 42 - ok
> > 
> > mac os, windows - we say "latest firefox or google-chrome is required", then - ok
> > 
> > So we are probably okey with using javascript Promises with MIDAS...
> 
> [too bad about chrome on SL6] ... include a polyfill library like Lie (https://github.com/calvinmetcalf/lie).

Results of cross-browser testing.

MacOS 10.10.5:

google-chrome 46: Promise ok, Programs page ok, Overlay ok
firefox 42: Promise ok, Programs page ok, Overlay ok.
safari 9.0.1: Promise ok, Programs page ok, Overlay ok.

Linux SL6.7:

google-chrome 27: "Promise not defined"
firefox 38.4.0: Promise ok, Programs page ok, Overlay ok.
konqueror 4.3.4: no go "Can't find variable: JSON"
chromium/google-chrome 38: ok

Linux SL7.1:

google-crome 46: ok
firefox 38.4.0: ok
konqueror 4.10.5: no go, mhttpd.js parse error

Conclusion:

1) firefox is good everywhere
2) google-chrome is good on Mac, Windows and el7 Linux
3) chromium/google-chrome 38 is good on el6 Linux (SL6/CentOS6).

We are good to proceed with adopting the Promise API for MIDAS.

K.O.

Not clear on what you are doing, so here is the brief description:

- you have two machines - say "midas" and "frontend"
- you run mttpd, mlogger, etc on "midas"
- you want to run some frontend on "frontend"

Do this:
- on "midas", open a new terminal, run "mserver -p 7071"
- on "frontend", open a new terminal, run "odbedit -h midas:7071"

If you did follow all the online instruction correctly, at this point, your odbedit on "frontend" would have 
connected and all commands would work same as if run locally on "midas".

If I understand you correctly, you got this far.

Next do this:
- on "frontend", open a new terminal, run "your_frontend.exe -h midas:7071"

If all is good, the frontend would start, would connect to midas, you would see
it in odbedit "scl" and on the midas status page and you would be able to stop
it from the midas "programs" page. (this last bit is important).

I guess this is where things go wrong and you do not get anything working.

Do this:
a) cut and paste all the output from the terminal window where you are running the mserver (including the 
command you used to start the mserver)
b) cut and paste all the output from the terminal window where you are starting the frontemd (again, 
including the command you used to start the frontend)
c) cut  and paste the contents of midas.log (in the experiment directory) from the time you started the 
mserver until the very end.

Paste all this as reply to this message or email it to me at olchansk@triumf.ca

With this additional information we should be able to get you going (and hopefully improve the 
documentation so the next person does not run into the same problem - whatever the problem turns out to 
be).

K.O.

> I'm trying to set up an experiment with 2 frontends for the first time.  When I
> start the remote frontend I get the following errors:
> 
> first time (odd attempt):
> 
> [MCS_Frontend_203,ERROR] [midas.c:9678:rpc_server_connect,ERROR] mserver
> subprocess could not be started (check path)
> [MCS_Frontend_203,ERROR] [mfe.c:2696:mainFE,ERROR] Cannot connect to experiment
> 'Default' on host 'ucntau-daq.lanl.gov', status 503
> 
> second time (even attempt):
> 
> MCS6A_frontend: src/midas.c:9085: rpc_client_dispatch: Assertion `n ==
> sizeof(NET_COMMAND_HEADER) + 4 * sizeof(INT)' failed.
> 
> On the local host I'm running : mlogger, a frontend, an analzer, mhttp, and
> mserver.  I followed the instructions for adding the remote computer to the
> RPC_ALLOWED list and I do see that the remote frontend was able to edit the
> local odb equipment list.  At present I'm not running an event builder I just
> wanted to get the frontends connected to start.  Do I need to have the mserver
> running on both computers?  Any suggestions on where to start troubleshooting this?
> 
> Thanks 

I'm trying to set up an experiment with 2 frontends for the first time.  When I
start the remote frontend I get the following errors:

first time (odd attempt):

[MCS_Frontend_203,ERROR] [midas.c:9678:rpc_server_connect,ERROR] mserver
subprocess could not be started (check path)
[MCS_Frontend_203,ERROR] [mfe.c:2696:mainFE,ERROR] Cannot connect to experiment
'Default' on host 'ucntau-daq.lanl.gov', status 503

second time (even attempt):

MCS6A_frontend: src/midas.c:9085: rpc_client_dispatch: Assertion `n ==
sizeof(NET_COMMAND_HEADER) + 4 * sizeof(INT)' failed.

On the local host I'm running : mlogger, a frontend, an analzer, mhttp, and
mserver.  I followed the instructions for adding the remote computer to the
RPC_ALLOWED list and I do see that the remote frontend was able to edit the
local odb equipment list.  At present I'm not running an event builder I just
wanted to get the frontends connected to start.  Do I need to have the mserver
running on both computers?  Any suggestions on where to start troubleshooting this?

Thanks 

I updated the links on the midas wiki to the doxygen-generated documentation for MIDAS that you 
get after running "git clone midas; cd midas; make dox; firefox html/index.html".

Correct link is:
https://daq.triumf.ca/~daqweb/doc/midas-devel/html/

This takes you to a daily/nightly generated snapshot of the midas develop branch and the 
generated documentation with full call graphs.

Previous links were deficient is different ways:
- referred to http://ladd00 instead of https://daq
- referred to wrong path ~daqweb/doc/midas instead of ~daqweb/doc/midas-devel
- referred to the obsolete doxygen generator in midas/doc/html instead of midas/html.

If wrong links are still present on the midas wiki, please let us know and we will fix them.
K.O.

> The JSON RPC branch has been merged into main MIDAS.

The interface is now mostly documented, go here: https://midas.triumf.ca/MidasWiki/index.php/Mjsonrpc

Documentation for individual javascript functions in mhttpd.js not merged into the MIDAS documentation yet, because the API is being converted to the Javascript Promise 
pattern (git branch feature/js_promise).

The functions available from mhttpd.js are documented via doxygen, also linked from the mjsonrpc wiki page.

K.O.

> Right now the JSON-RPC client library does not check the return status of MIDAS calls themselves, i.e. ODBGet("/nonexistant") will go to Promise.resolve() with
> the MIDAS db_find_key() status DB_NO_KEY instead of Promise.reject(). So some error handling in Promise.resolve() is still required.

> I am thinking how to make these calls go to the error handler automatically, but there is no obvious solution for ODBMGet(["/runinfo", "/nonexistant"]) - the first path
> is a success, the second is a failure, do I fail the entire transaction (i.e. with a JSON-RPC error)? Same for JSON-RPC batch transactions - if one transaction
> in the batch fails, do I Promise.reject() all of them?


Generally, I'd prefer a grouped-request like ODBMGet to return an array of
Promises.  This way, I get to decide how to handle the request responses.  While
I do have cases where I use Promise.all(...), most of my current code would want
to treat each response individually.

But as you point out, my approach differs from the Midas approach significantly.
While I've set up my queries to reject on responses like DB_NO_KEY, the function
mjsonrpc_send_request in mhttpd.js tests purely for a successful http
request.

Since ODBMGet makes a *single* http call, I'd naively lean toward having it
return a single Promise.  Presumably, one that resolves if the http request
"goes through" and rejects if the http request fails. 

My perspective may not be the useful one to consider, though.  If other users
expect an array of promises returned from ODBMGet, definitely feel free to
ignore my thoughts on the matter.

If people really want ODBMGet to return an array of promises, one way to do it
would be to have a 'get' function that only cares about the success of the http
request, and a separate 'response checking' function that validates the
response.  ODBMGet can use these two functions together to return an array of
Promises:

#########
mhttpd.js
#########
function get(url) { 
  // return a promise that resolves if the http request returns status=200
  // reject if the http request does anything else
}

function checkResponse(response) {
  // return a promise that resolves for "good" responses
  // and rejects for "bad response"
}

function ODBMGet(path_arr) {
  var url = // build url from path_arr
  
  // syntax get().then(A).catch(B) means
  // if the http request goes through, A is executed
  // if the http request fails, B is executed
  // ;
  // for get, failure means no successful reply at all
  // so ODBMGet should return an array of rejected Promises 
  get(url).then(function(response) {
    response_arr = // split the response
    return response_arr.map(checkResponse)
  }).catch(function(err) {
    return path_arr.forEach(function() {
      return Promise.reject(err)
    })
  })
}

#########
user code
#########
// here the Promises are treated individually
var response_arr = ODBMGet([path1, path2, path3])

response_arr.forEach(function(thisResponse) {
  thisResponse.then( /* do something */ )
              .catch( /* do something else */ )
})

// and here the failure of a single promise in the array
// determines the code that's executed next
var required_arr = ODBMGet([pathA, pathB, pathC])

Promise.all(required_arr).then( /* do something */ )
                         .catch( /* do something else */ )

> > Why don't you post the functions here so that we can have a look? 
> Here is (1) my promisified HTTP request function and (2) a function that uses the returned promises to build an asynchronous, sequential chain of requests to Midas.
> 
> In addition to promisifying HTTP requests to Midas, I wanted the Promise.resolve from this function to always return valid JSON.

Thank you very much for posting this code. It is very similar to what I just wrote this morning for the JSON-RPC client library. In my case, the JSON-RPC responses
are much more regular so error handling is simple: a) check HTTP response 200, b) check JSON-RPC reply parses into valid JSON (catch exception), c) check JSON-RPC error status.

> I also wanted the promise to reject if the response from mhttpd indicated
> failure - so that we wouldn't have to rewrite this error checking throughout the code.

Right now the JSON-RPC client library does not check the return status of MIDAS calls themselves, i.e. ODBGet("/nonexistant") will go to Promise.resolve() with
the MIDAS db_find_key() status DB_NO_KEY instead of Promise.reject(). So some error handling in Promise.resolve() is still required.

I am thinking how to make these calls go to the error handler automatically, but there is no obvious solution for ODBMGet(["/runinfo", "/nonexistant"]) - the first path
is a success, the second is a failure, do I fail the entire transaction (i.e. with a JSON-RPC error)? Same for JSON-RPC batch transactions - if one transaction
in the batch fails, do I Promise.reject() all of them?

I guess I could "split hairs" and create a separate Promise for each "atomic" transaction, the Promise mechanism does seem to support that,
but this will create more complexity than I feel comfortable with.

Please take a look at the branch feature/js_promise - mjsonrpc_call() is Promisified (resources/mhttpd.js) and the db_copy() example is Promisified (examples/javascript1/example.html)

K.O.

> I checked again on browser compatibility:
> 
> el6: firefox 38 - ok, google-chrome 27 - no
> el7: firefox 38 - ok, google-chrome 46 - ok
> ubuntu: firefox 42 - ok
> 
> mac os, windows - we say "latest firefox or google-chrome is required", then - ok
> 
> So we are probably okey with using javascript Promises with MIDAS...

It looks like this does mean that people using RHEL6 won't have the option of chrome - can they update chrome?

One option is to include a polyfill library like Lie (https://github.com/calvinmetcalf/lie).