I updated the MIDAS javascript examples in examples/javascript1. All existing mhttpd.js functions are 
now exampled. (yes).

Here is the full list of functions, with notes:

ODBSet(path, value, pwdname);
ODBGet(path, format, defval, len, type);
ODBMGet(paths, callback, formats); --- doc incomplete - no example of callback() use
ODBGetRecord(path);
ODBExtractRecord(record, key);
new ODBKey(path); --- doc incomplete, wrong - one has to use "new ODBKey" - last_used was added.
ODBCopy(path, format); -- no doc
ODBRpc_rev0(name, rpc, args); --- doc refer to example
ODBRpc_rev1(name, rpc, max_reply_length, args); --- same
ODBGetMsg(n);
ODBGenerateMsg(m);
ODBGetAlarms(); --- no doc
ODBEdit(path); --- undoc - forces page reload

As annotated, the main documentation is partially incomplete and partially wrong (i.e. ODBKey() has to be 
invoked as "new ODBKey()"). I hope this will be corrected soon. In the mean time, I recommend that 
everybody uses this example as best documentation available.
http://ladd00.triumf.ca/~daqweb/doc/midas/html/RC_mhttpd_custom_js_lib.html

svn rev 5360
K.O.

> odbedit can now save ODB in JSON-formatted files.
> Added functions:
>    INT EXPRT db_save_json(HNDLE hDB, HNDLE hKey, const char *file_name);
>    INT EXPRT db_copy_json(HNDLE hDB, HNDLE hKey, char **buffer, int *buffer_size, int *buffer_end, int  save_keys, int follow_links);
> 

Added JSON encoding format to Javascript ODBCopy() ("jcopy"). Use format="json", Javascript example updated with an example example.

Also updated db_copy_json():
- always return NUL-terminated string
- "save_keys" values: 0 - do not save any KEY data, 1 - save all KEY data, 2 - save only KEY.last_written

odb.c, mhttpd.cxx, example.html
svn rev 5362
K.O.

> ODBCopy(path, format); -- no doc

Updated example of ODBCopy:

format="" returns data in traditional ODB save format
format="xml" returns data in XML encoding
format="json" returns data in JSON encoding.

K.O.

> 
> Added JSON encoding format to Javascript ODBCopy(path,format) ("jcopy"). Use format="json", Javascript example updated with an example example.
> 

More ODBCopy() expansion: format="json-p" returns data suitable for JSON-P ("script tag") messaging.

Also implemented multiple-paths for "jcopy" (similar to "jget"/ODBMGet()). An example ODBMCopy(paths,callback,format) is present in example.html (will move to mhttpd.js).

Added JSON encoding options:
- format="json-nokeys" will omit all KEY information except for "last_written"
- "json-nokeys-nolastwritten" will also omit "last_written"
- "json-nofollowlinks" will return ODB symlink KEYs instead of following them (ODBGet/ODBMGet always follows symlinks)
- "json-p" adds JSON-P encapsulation
All these JSON format options can be used at the same time, i.e. format="json-p-nofollowlinks"

To see how it all works, please look at examples/javascript1/example.html.

The new code seems to be functional enough, but it is still work in progress and there are a few problems:
- ODBMCopy() using the "xml" format returns gibberish (the MIDAS XML encoder has to be told to omit the <?xml> header)
- example.html does not actually parse any of the XML data, so we do not know if XML encoding is okey
- JSON encoding has an extra layer of objects (variables.Variables.foo instead of variables.foo)
- ODBRpc() with JSON/JSON-P encoding not done yet.

mhttpd.cxx, example.html
svn rev 5364
K.O.

> To see how it all works, please look at examples/javascript1/example.html.
> 
> - JSON encoding has an extra layer of objects (variables.Variables.foo instead of variables.foo)
>

This is now fixed. See updated example.html. Current encoding looks like this:

{
  "System" : {
    "Clients" : {
      "24885" : {
        "Name/key" : { "type" : 12, "item_size" : 32, "last_written" : 1370024816 },
        "Name" : "ODBEdit",
        "Host/key" : { "type" : 12, "item_size" : 256, "last_written" : 1370024816 },
        "Host" : "ladd03.triumf.ca",
        "Hardware type/key" : { "type" : 7, "last_written" : 1370024816 },
        "Hardware type" : 44,
        "Server Port/key" : { "type" : 7, "last_written" : 1370024816 },
        "Server Port" : 52539
      }
    },
    "Tmp" : {
...

odb.c, example.html
svn rev 5368
K.O.

We are happy to announce the creation of the MidasWiki at TRIUMF (https://midas.triumf.ca) as the 
new location of MIDAS documentation, user instructions, examples, etc.

https://midas.triumf.ca
K.O.

The MIDAS source code repository was converted from SVN to GIT, hosted as bitbucket: 
https://bitbucket.org/tmidas.

A clonable copy of the repository is located at TRIUMF: git clone 
http://daq.triumf.ca/~daqweb/git/midas.git (and mxml.git).

The documentation is being slowly updated with GIT instructions (git clone) instead of SVN (svn 
checkout).

The MIDAS code history goes all the way to CVS/SVN rev 1 dated Thu Oct 8 13:46:02 1998.

K.O.

Konstantin forgot to tell people outside of TRIUMF how to get the newest version of MIDAS. Here it is:

$ git clone https://bitbucket.org/tmidas/midas.git

Not that you can also browse the repository at

https://bitbucket.org/tmidas/midas

On some (older) systems, you might have to install git (http://git-scm.com/downloads).

/Stefan

> We are happy to announce the creation of the MidasWiki at TRIUMF
> https://midas.triumf.ca

We are running MediaWiki in the world-readable, authenticated-user-writable mode.

New user registration is done by the "confirm new user" extension (https://www.mediawiki.org/wiki/Extension:ConfirmAccount):
- go to https://midas.triumf.ca/MidasWiki/index.php/Special:RequestAccount
- fill the form, submit request - request goes to wiki administrator for confirmation
- wait for email about email address confirmation, follow instructions to confirm your email address
- wait for email about account confirmation
- try to login to the wiki.

K.O.

The bulk of the MIDAS AJAX and Javascript functions is now documented on the MIDAS Wiki:

https://midas.triumf.ca/MidasWiki/index.php/Mhttpd.js
https://midas.triumf.ca/MidasWiki/index.php/AJAX

Enjoy,
K.O.

> The bulk of the MIDAS AJAX and Javascript functions is now documented on the MIDAS Wiki:
> 
> https://midas.triumf.ca/MidasWiki/index.php/Mhttpd.js
> https://midas.triumf.ca/MidasWiki/index.php/AJAX
> 

The documentation was updated again.

All functions and AJAX methods except jset, jget, jrpc (ODBGet, ODBSet, ODBRpc) and inline edit are now fully 
documented. AJAX methods jset/jget and their javascript wrappers ODBSet/ODBGet/ODBMGet/ODBGetRecord() are 
partially documented. Inline edit will have to be documented by Stefan.

When using these functions please read the "BUG" sections carefully.

When using the Javascript functions (ODBGet, ODBSet, ODBMCopy, etc) please pay special attention to the rules for URI-
encoding function arguments - some functions require that some arguments be pre-encoded by 
"encodeURIComponent()", some functions require some arguments to NOT be pre-encoded. The examples in 
examples/javascript1/example.html are mostly correct.

Special confusion is created by special handling in mhttpd of URI-encoding of parameters named "format".

Special confusion is created by ODBSet(path, value), where "path" should be pre-encoded, while "value" is now encoded 
internally, which is a recent change introduced with the inline edit function. Older versions of mhttpd.js still require that 
"value" be URI-encoded.

Going forward, I hope to resolve most of the confusion by providing a cleaner interface for reading ODB
- ODBMCopy() already looks good with full async JSON/JSONP support (already implemented)
- ODBMKey() to read just the keys, with async JSON/JSONP support (to be added)
- ODBMCreate() to create ODB keys (RPC for db_create()) (to be added)
- ODBMSet() to write into ODB. (to be added)

K.O.

I would like to share with you a silly problem with mktime() and daylight savings time (Summer 
time/Winter time) that I have run into while working on the mhttpd history query page.

I am implementing 1 hour granularity for the queries (was 1 day granularity) and somehow all my queries 
were off by 1 hour.

It turns out that the mktime() and localtime() functions for converting between time_t and normal time 
units (days, hours) are not exact inverses of each other.

Daylight savings time (DST) is to blame.

While localtime() always applies the current DST,  mktime() will return the wrong answer unless tm_isdst is 
set correctly.

For tm_isdst, the default value 0 is wrong 50% of the time in most locations as it means "DST off" (whether 
that's Summer time or Winter time depends on your location).

Today in Vancouver, BC, DST is in effect, and localtime(mktime()) is off by 1 hour.

If I were doing this in January, I would not see this problem.

"man mktime" talks about "tm_isdst" special value "-1" that is supposed to fix this. But the wording of 
"man mktime" on Linux and on MacOS is different (I am amused by the talk about "attempting to divine 
the DST setting"). Wording at http://pubs.opengroup.org/onlinepubs/007908799/xsh/mktime.html is 
different again. MS Windows (Visual Studio) documentation says different things for different versions.

So for mhttpd I use the following code. First mktime() gets the approximate time, a call to localtime() 
returns the DST setting in effect for that date, a second mktime() with the correct DST setting returns the 
correct time. (By "correct" I mean that localtime(mktime(t)) == t).

time_t mktime_with_dst(const struct tm* ptms)
{
   // this silly stuff is required to correctly handle daylight savings time (Summer time/Winter time)
   // when we fill "struct tm" from user input, we cannot know if daylight savings time is in effect
   // and we do not know how to initialize the value of tms->tm_isdst.
   // This can cause the output of mktime() to be off by one hour.
   // (Rules for daylight savings time are set by national and local govt and in some locations, changes 
yearly)
   // (There are no locations with 2 hour or half-hour daylight savings that I know of)
   // (Yes, "man mktime" talks about using "tms->tm_isdst = -1")
   //
   // We assume the user is using local time and we convert in two steps:
   //
   // first we convert "struct tm" to "time_t" using mktime() with unknown tm_isdst
   // second we convert "time_t" back to "struct tm" using localtime_r()
   // this fills "tm_isdst" with correct value from the system time zone database
   // then we reset all the time fields (except for sub-minute fields not affected by daylight savings)
   // and call mktime() again, now with the correct value of "tm_isdst".
   // K.O. 2013-09-14

   struct tm tms = *ptms;
   struct tm tms2;
   time_t t1 = mktime(&tms);
   localtime_r(&t1, &tms2);
   tms2.tm_year = ptms->tm_year;
   tms2.tm_mon  = ptms->tm_mon;
   tms2.tm_mday = ptms->tm_mday;
   tms2.tm_hour = ptms->tm_hour;
   tms2.tm_min  = ptms->tm_min;
   time_t t2 = mktime(&tms2);
   //printf("t1 %.0f, t2 %.0f, diff %d\n", (double)t1, (double)t2, (int)(t1-t2));
   return t2;
}

K.O.

Due to popular request, I implemented a custom header for mhttpd. This allows to inject some HTML code 
to be shown on top of the menu bar on all mhttpd pages. One possible application is to bring back the old 
status line with the name of the current experiment, the actual time and the refresh interval. 

To use this feature, one can put a new entry into the ODB under

/Custom/Header

which can be either a string (to show some short HTML code directly) or the name of a file containing some 
HTML code. If /Custom/Path is present, that path is used to locate the header file. A simple header file to 
recreate the GOT look (good-old-times) is here:

<div id="footerDiv" class="footerDiv">
<div style="display:inline; float:left;">MIDAS experiment "Test"</div>
<div id="refr" style="display:inline; float:right;"></div>
</div>
<script type="text/javascript">
var r = document.getElementById('refr');
var now	= new Date();
var c =	document.cookie.split('midas_refr=');
r.innerHTML = now.toString() + '&nbsp;&nbsp;&nbsp;' + 'Refr:' + c.pop().split(';').shift();
</script>

The JavaScript code is used to retrieve the midas_refr cookie which stores the refresh interval and displays 
it together with the current time.

Another application of this feature might be to check certain values in the ODB (via the ODBGet function) 
and some some important status or error condition.

/Stefan

I vaguely remember that I had a similar problem with ELOG. The solution was to call tzset() at the beginning of the program. The man page says that 
this function is called automatically by programs using time zones, but apparently it is not. Can you try that? There is also the TZ environment 
variable and /etc/localtime. I never understood the details, but playing with these things can influence mktime() and localtime().

/Stefan

> I vaguely remember that I had a similar problem with ELOG. The solution was to call tzset() at the beginning of the program. The man page says that 
> this function is called automatically by programs using time zones, but apparently it is not. Can you try that? There is also the TZ environment 
> variable and /etc/localtime. I never understood the details, but playing with these things can influence mktime() and localtime().

I confirm that the timezone is set correctly - I do get the correct time eventually - so there is no missing call to tzet().

K.O.

> > I vaguely remember that I had a similar problem with ELOG. The solution was to call tzset() at the beginning of the program. The man page says that 
> > this function is called automatically by programs using time zones, but apparently it is not. Can you try that? There is also the TZ environment 
> > variable and /etc/localtime. I never understood the details, but playing with these things can influence mktime() and localtime().
> 
> I confirm that the timezone is set correctly - I do get the correct time eventually - so there is no missing call to tzet().
> 
> K.O.

tzset() not only sets the time zone, but also DST.

> > > I vaguely remember that I had a similar problem with ELOG. The solution was to call tzset() at the beginning of the program. The man page says that 
> > > this function is called automatically by programs using time zones, but apparently it is not. Can you try that? There is also the TZ environment 
> > > variable and /etc/localtime. I never understood the details, but playing with these things can influence mktime() and localtime().
> > 
> > I confirm that the timezone is set correctly - I do get the correct time eventually - so there is no missing call to tzet().
> > 
> > K.O.
> 
> tzset() not only sets the time zone, but also DST.

I found following code in elogd.c, maybe it helps:

/* workaround for wong timezone under MAX OSX */
long my_timezone()
{
#if defined(OS_MACOSX) || defined(__FreeBSD__) || defined(__OpenBSD__)
   time_t tp;
   time(&tp);
   return -localtime(&tp)->tm_gmtoff;
#else
   return timezone;
#endif
}



void get_rfc2822_date(char *date, int size, time_t ltime)
{
   time_t now;
   char buf[256];
   int offset;
   struct tm *ts;

   /* switch locale temporarily back to english to comply with RFC2822 date format */
   setlocale(LC_ALL, "C");

   if (ltime == 0)
      time(&now);
   else
      now = ltime;
   ts = localtime(&now);
   assert(ts);
   strftime(buf, sizeof(buf), "%a, %d %b %Y %H:%M:%S", ts);
   offset = (-(int) my_timezone());
   if (ts->tm_isdst)
      offset += 3600;
   snprintf(date, size - 1, "%s %+03d%02d", buf, (int) (offset / 3600),
            (int) ((abs((int) offset) / 60) % 60));
}

> > The bulk of the MIDAS AJAX and Javascript functions is now documented on the MIDAS Wiki:
> > 
> > https://midas.triumf.ca/MidasWiki/index.php/Mhttpd.js
> > https://midas.triumf.ca/MidasWiki/index.php/AJAX
> > 
> 
> The documentation was updated again.
> 

Newly documented are the additional Javascript and AJAX functions present in the GIT branch "feature/ajax":

ODBMCreate(paths, types);
ODBMCreate(paths, types, arraylengths, stringlengths, callback);  
ODBMResize(paths, arraylengths, stringlengths, callback);  
ODBMRename(paths, names, callback);  
ODBMLink(paths, links, callback);  
ODBMReorder(paths, indices, callback);
ODBMKey(paths, callback);
ODBMDelete(paths, callback);

All these functions permit asynchronous use (with callback on completion) and the underlying AJAX functions permit JSON-P encoding.

ODBSetUrl("http://mhttpd.somewhere.com:8080") : this new function removes the restriction that custom scripts had to be loaded from the same mhttpd that they will 
access. Together with the newly added CORS support in mhttpd, allows loading custom scripts from any web server, including local file, and having then access any one (or 
any several) mhttpd data sources.

I think these new functions are now stable (I still had to make some changes to ODBMCreate() recently) and after some more testing this branch will be merged into 
"develop".

To use this branch, do either:
a) git clone midas; git pull; git checkout feature/ajax
b) git clone midas; git checkout develop; git pull; git checkout -b ajaxtest; git merge feature/ajax;

(Option (b) creates a local branch with the latest "develop" and "feature/ajax" merged together).

K.O.

> odbedit can now save ODB in JSON-formatted files.
> 
> JSON encoding implementation follows specifications at:
> http://json.org/
> 
> The result passes validation by:
> http://jsonlint.com/
> 

A bug was reported in my JSON ODB encoder: NaN values are not encoded correctly. A quick review found this:

1) the authors of JSON smoked some bad mushrooms and specifically disallowed NaN and Inf values for floating point numbers: 
http://tools.ietf.org/html/rfc4627
2) most JSON encoders and decoders do reasonable and unreasonable things with NaN and Inf values. The worst ones encode them as zero. More bad 
mushrooms.

There is a quick survey at: http://lavag.org/topic/16217-cr-json-labview/?p=99058

<pre>
Some Javascript engines allow it since it is valid Javascript but not valid Json however there is no concensus.
cmj-JSON4Lua: raw tostring() output (invalid JSON).
dkjson: 'null' (like in the original JSON-implementation).
Fleece: NaN is 0.0000, freezes on +/-Inf.
jf-JSON: NaN is 'null', Inf is 1e+9999 (the encode_pretty function still outputs raw tostring()).
Lua-Yajl: NaN is -0, Inf is 1e+666.
mp-CJSON: raises invalid JSON error by default, but runtime configurable ('null' or Nan/Inf).
nm-luajsonlib: 'null' (like in the original JSON-implementation).
sb-Json: raw tostring() output (invalid JSON).
th-LuaJSON: JavaScript? constants: NaN is 'NaN', Inf is 'Infinity' (this is valid JavaScript?, but invalid JSON).
</pre>

For the MIDAS JSON encoder (and decoder) I have several choices:
a) encode NaN and Inf using the printf("%f") encoding (as strings, making it valid JSON)
b) encode NaN and Inf as strings using the Javascript special values: "NaN", "Infinity" and "-Infinity", see 
http://www.w3schools.com/jsref/jsref_positive_infinity.asp

I note that the Python JSON encoder does (b), see section 18.2.3.3 at http://docs.python.org/2/library/json.html

In either case, behaviour of the JSON decoder on the Javascript side needs to be tested. (Silent conversion to value of zero is not acceptable).

If anybody has an suggestion on this, please let me know.



P.S. If you do not know all about NaN, Inf, "-0" and other floating point funnies, please read:  https://www.ualberta.ca/~kbeach/phys420_580_2010/docs/ACM-Goldberg.pdf

P.P.S. If you ever used the type "float" or "double", used the "/" operator or the function "sqrt()" you also should read that reference.

K.O.

The following code found in mhttpd does not work on MacOS (BSD UNIX).

On Linux, the do-loop will finish after 2 seconds as expected. On MacOS (and other BSD systems), it will 
loop forever.

The cause is the MIDAS watchdog alarm() signal that fires every 1 second and always interrupts the 2 
second sleep of select(). The Linux select() updates it's timeout argument to reflect time already slept, so 
eventually we finish. The MacOS (BSD) select() does not update the timeout argument and select goes back 
to sleep for another 2 seconds (to be again interrupted half-way through).

The POSIX standard (specification for select() & co) permits either behaviour. Compare "man select" on 
MacOS and on Linux.

If the select() timeout were not 2 seconds, but 0.9 seconds; or if the MIDAS watchdog alarm fired every 
2.1 seconds, this problem would also not exist.

I think there are several places in MIDAS with code like this. An audit is required.

{
                  FD_ZERO(&readfds);
                  FD_SET(_sock, &readfds);

                  timeout.tv_sec = 2;
                  timeout.tv_usec = 0;

                  do {
                     status = select(FD_SETSIZE, &readfds, NULL, NULL, &timeout);
                     /* if an alarm signal was cought, restart with reduced timeout */
                  } while (status == -1 && errno == EINTR);
}

K.O.