> This is the jset code. The best I can tell it truncates string variables to the existing size in ODB:
> 
> db_find_key(hDB, 0, str, &hkey)
> db_get_key(hDB, hkey, &key);
> memset(data, 0, sizeof(data));
> size = sizeof(data);
> db_sscanf(getparam("value"), data, &size, 0, key.type);
> db_set_data_index(hDB, hkey, data, key.item_size, index, key.type);


Correct. So I added some code which extends strings if necessary (NOT string arrays, they are more complicated to handle).

> > > I vaguely remember that I had a similar problem with ELOG. The solution was to call tzset() at the beginning of the program. The man page says that 
> > > this function is called automatically by programs using time zones, but apparently it is not. Can you try that? There is also the TZ environment 
> > > variable and /etc/localtime. I never understood the details, but playing with these things can influence mktime() and localtime().
> > 
> > I confirm that the timezone is set correctly - I do get the correct time eventually - so there is no missing call to tzet().
> > 
> > K.O.
> 
> tzset() not only sets the time zone, but also DST.

I found following code in elogd.c, maybe it helps:

/* workaround for wong timezone under MAX OSX */
long my_timezone()
{
#if defined(OS_MACOSX) || defined(__FreeBSD__) || defined(__OpenBSD__)
   time_t tp;
   time(&tp);
   return -localtime(&tp)->tm_gmtoff;
#else
   return timezone;
#endif
}



void get_rfc2822_date(char *date, int size, time_t ltime)
{
   time_t now;
   char buf[256];
   int offset;
   struct tm *ts;

   /* switch locale temporarily back to english to comply with RFC2822 date format */
   setlocale(LC_ALL, "C");

   if (ltime == 0)
      time(&now);
   else
      now = ltime;
   ts = localtime(&now);
   assert(ts);
   strftime(buf, sizeof(buf), "%a, %d %b %Y %H:%M:%S", ts);
   offset = (-(int) my_timezone());
   if (ts->tm_isdst)
      offset += 3600;
   snprintf(date, size - 1, "%s %+03d%02d", buf, (int) (offset / 3600),
            (int) ((abs((int) offset) / 60) % 60));
}

> > I vaguely remember that I had a similar problem with ELOG. The solution was to call tzset() at the beginning of the program. The man page says that 
> > this function is called automatically by programs using time zones, but apparently it is not. Can you try that? There is also the TZ environment 
> > variable and /etc/localtime. I never understood the details, but playing with these things can influence mktime() and localtime().
> 
> I confirm that the timezone is set correctly - I do get the correct time eventually - so there is no missing call to tzet().
> 
> K.O.

tzset() not only sets the time zone, but also DST.

> I vaguely remember that I had a similar problem with ELOG. The solution was to call tzset() at the beginning of the program. The man page says that 
> this function is called automatically by programs using time zones, but apparently it is not. Can you try that? There is also the TZ environment 
> variable and /etc/localtime. I never understood the details, but playing with these things can influence mktime() and localtime().

I confirm that the timezone is set correctly - I do get the correct time eventually - so there is no missing call to tzet().

K.O.

Actually this was no bug, but a missing feature. Strings were never meant to be extended via the web interface. 
Now I added that feature to the current version. Please check it.

/Stefan

I vaguely remember that I had a similar problem with ELOG. The solution was to call tzset() at the beginning of the program. The man page says that 
this function is called automatically by programs using time zones, but apparently it is not. Can you try that? There is also the TZ environment 
variable and /etc/localtime. I never understood the details, but playing with these things can influence mktime() and localtime().

/Stefan

Due to popular request, I implemented a custom header for mhttpd. This allows to inject some HTML code 
to be shown on top of the menu bar on all mhttpd pages. One possible application is to bring back the old 
status line with the name of the current experiment, the actual time and the refresh interval. 

To use this feature, one can put a new entry into the ODB under

/Custom/Header

which can be either a string (to show some short HTML code directly) or the name of a file containing some 
HTML code. If /Custom/Path is present, that path is used to locate the header file. A simple header file to 
recreate the GOT look (good-old-times) is here:

<div id="footerDiv" class="footerDiv">
<div style="display:inline; float:left;">MIDAS experiment "Test"</div>
<div id="refr" style="display:inline; float:right;"></div>
</div>
<script type="text/javascript">
var r = document.getElementById('refr');
var now	= new Date();
var c =	document.cookie.split('midas_refr=');
r.innerHTML = now.toString() + '&nbsp;&nbsp;&nbsp;' + 'Refr:' + c.pop().split(';').shift();
</script>

The JavaScript code is used to retrieve the midas_refr cookie which stores the refresh interval and displays 
it together with the current time.

Another application of this feature might be to check certain values in the ODB (via the ODBGet function) 
and some some important status or error condition.

/Stefan

I confirm the second part of the problem.

Inline edit uses ODBSet(), which uses the "jset" AJAX call to mhttpd which does not extend string variables.

This is the jset code. The best I can tell it truncates string variables to the existing size in ODB:

db_find_key(hDB, 0, str, &hkey)
db_get_key(hDB, hkey, &key);
memset(data, 0, sizeof(data));
size = sizeof(data);
db_sscanf(getparam("value"), data, &size, 0, key.type);
db_set_data_index(hDB, hkey, data, key.item_size, index, key.type);

These original jset/jget functions are a little bit too complicated and there is no documentation (what exists is done by me trying to read the existing code).

We now have a jcopy/ODBMCopy() as a sane replacement for jget, but nothing comparable for jset, yet.

I think this quirk of inline edit cannot be fixed in javascript - the mhttpd code for "jset" has to change.

K.O.


> 
> I can confirm part of the problem - the new inline-edit function - after you finish editing - shows you what you 
> have typed, not what's actually in ODB - at the very end it should do an ODBGet() to load the actual ODB 
> contents and show *that* to the user.
> 
> The truncation to 32 characters - most likely it is a failure to resize the ODB string - is probably in mhttpd and 
> I can take a quick look into it.
> 
> There is a 3rd problem - the mhttpd ODB editor "create" function does not ask for the string length to create.
> 
> Actually, in ODB, "create" and "set string size" are two separate functions - db_create_key(TID_STRING) creates 
> a string of length zero, then db_set_data() creates an empty string of desired length.
> 
> In the new AJAX interface these two functions are separate (ODBCreate just calls db_create_key()).
> 
> In the present ODBSet() function the two are mixed together - and the ODB inline edit function uses ODBSet().
> 
> K.O.
> 
> 
> 
> > I find that new mhttpd has strange behaviour for ODB strings.
> > 
> > - I create a new STRING variable in ODB through mhttpd.  It defaults to size 32.
> > 
> > - I then edit the STRING variable through mhttpd, writing a new string larger
> > than 32 characters.
> > 
> > - Initially everything looks fine; it seems as if the new string value has been
> > accepted.
> > 
> > - But if you reload the page, or navigate back to the page, you realize that
> > mhttpd has silently truncated the string back to 32 characters.
> > 
> > You can reproduce this problem on a test page here:
> > 
> > http://midptf01.triumf.ca:8081/AnnMessage
> > 
> > Older versions of mhttpd (I'm testing one from 2 years ago) don't have this
> > 'feature'.  For older mhttpd the string variable would get resized when a larger
> > string was inputted.  That definitely seems like the right behavior to me.
> > 
> > I am using fresh copy of midas from bitbucket as of this morning. (How do I get
> > a particular tag/hash of the version of midas that I am using?)

I would like to share with you a silly problem with mktime() and daylight savings time (Summer 
time/Winter time) that I have run into while working on the mhttpd history query page.

I am implementing 1 hour granularity for the queries (was 1 day granularity) and somehow all my queries 
were off by 1 hour.

It turns out that the mktime() and localtime() functions for converting between time_t and normal time 
units (days, hours) are not exact inverses of each other.

Daylight savings time (DST) is to blame.

While localtime() always applies the current DST,  mktime() will return the wrong answer unless tm_isdst is 
set correctly.

For tm_isdst, the default value 0 is wrong 50% of the time in most locations as it means "DST off" (whether 
that's Summer time or Winter time depends on your location).

Today in Vancouver, BC, DST is in effect, and localtime(mktime()) is off by 1 hour.

If I were doing this in January, I would not see this problem.

"man mktime" talks about "tm_isdst" special value "-1" that is supposed to fix this. But the wording of 
"man mktime" on Linux and on MacOS is different (I am amused by the talk about "attempting to divine 
the DST setting"). Wording at http://pubs.opengroup.org/onlinepubs/007908799/xsh/mktime.html is 
different again. MS Windows (Visual Studio) documentation says different things for different versions.

So for mhttpd I use the following code. First mktime() gets the approximate time, a call to localtime() 
returns the DST setting in effect for that date, a second mktime() with the correct DST setting returns the 
correct time. (By "correct" I mean that localtime(mktime(t)) == t).

time_t mktime_with_dst(const struct tm* ptms)
{
   // this silly stuff is required to correctly handle daylight savings time (Summer time/Winter time)
   // when we fill "struct tm" from user input, we cannot know if daylight savings time is in effect
   // and we do not know how to initialize the value of tms->tm_isdst.
   // This can cause the output of mktime() to be off by one hour.
   // (Rules for daylight savings time are set by national and local govt and in some locations, changes 
yearly)
   // (There are no locations with 2 hour or half-hour daylight savings that I know of)
   // (Yes, "man mktime" talks about using "tms->tm_isdst = -1")
   //
   // We assume the user is using local time and we convert in two steps:
   //
   // first we convert "struct tm" to "time_t" using mktime() with unknown tm_isdst
   // second we convert "time_t" back to "struct tm" using localtime_r()
   // this fills "tm_isdst" with correct value from the system time zone database
   // then we reset all the time fields (except for sub-minute fields not affected by daylight savings)
   // and call mktime() again, now with the correct value of "tm_isdst".
   // K.O. 2013-09-14

   struct tm tms = *ptms;
   struct tm tms2;
   time_t t1 = mktime(&tms);
   localtime_r(&t1, &tms2);
   tms2.tm_year = ptms->tm_year;
   tms2.tm_mon  = ptms->tm_mon;
   tms2.tm_mday = ptms->tm_mday;
   tms2.tm_hour = ptms->tm_hour;
   tms2.tm_min  = ptms->tm_min;
   time_t t2 = mktime(&tms2);
   //printf("t1 %.0f, t2 %.0f, diff %d\n", (double)t1, (double)t2, (int)(t1-t2));
   return t2;
}

K.O.

I can confirm part of the problem - the new inline-edit function - after you finish editing - shows you what you 
have typed, not what's actually in ODB - at the very end it should do an ODBGet() to load the actual ODB 
contents and show *that* to the user.

The truncation to 32 characters - most likely it is a failure to resize the ODB string - is probably in mhttpd and 
I can take a quick look into it.

There is a 3rd problem - the mhttpd ODB editor "create" function does not ask for the string length to create.

Actually, in ODB, "create" and "set string size" are two separate functions - db_create_key(TID_STRING) creates 
a string of length zero, then db_set_data() creates an empty string of desired length.

In the new AJAX interface these two functions are separate (ODBCreate just calls db_create_key()).

In the present ODBSet() function the two are mixed together - and the ODB inline edit function uses ODBSet().

K.O.



> I find that new mhttpd has strange behaviour for ODB strings.
> 
> - I create a new STRING variable in ODB through mhttpd.  It defaults to size 32.
> 
> - I then edit the STRING variable through mhttpd, writing a new string larger
> than 32 characters.
> 
> - Initially everything looks fine; it seems as if the new string value has been
> accepted.
> 
> - But if you reload the page, or navigate back to the page, you realize that
> mhttpd has silently truncated the string back to 32 characters.
> 
> You can reproduce this problem on a test page here:
> 
> http://midptf01.triumf.ca:8081/AnnMessage
> 
> Older versions of mhttpd (I'm testing one from 2 years ago) don't have this
> 'feature'.  For older mhttpd the string variable would get resized when a larger
> string was inputted.  That definitely seems like the right behavior to me.
> 
> I am using fresh copy of midas from bitbucket as of this morning. (How do I get
> a particular tag/hash of the version of midas that I am using?)

I find that new mhttpd has strange behaviour for ODB strings.

- I create a new STRING variable in ODB through mhttpd.  It defaults to size 32.

- I then edit the STRING variable through mhttpd, writing a new string larger
than 32 characters.

- Initially everything looks fine; it seems as if the new string value has been
accepted.

- But if you reload the page, or navigate back to the page, you realize that
mhttpd has silently truncated the string back to 32 characters.

You can reproduce this problem on a test page here:

http://midptf01.triumf.ca:8081/AnnMessage

Older versions of mhttpd (I'm testing one from 2 years ago) don't have this
'feature'.  For older mhttpd the string variable would get resized when a larger
string was inputted.  That definitely seems like the right behavior to me.

I am using fresh copy of midas from bitbucket as of this morning. (How do I get
a particular tag/hash of the version of midas that I am using?)

> Dear MIDAS programmers,
> 
> I have been using your software in my lab (APC, Paris)
> to run our data acqusition system. It is very robust and flexible.s
> 
> I would like to give you the large amount of credit which you are due.
> How should I cite both MIDAS and ROODY? I have not been able to find any
> information in the usual places.
> 
> Cheers, and thanks for the great program!
> -Carl

The standard citation for midas is a link to

http://midas.psi.ch

At the moment this points automatically to http://midas.triumf.ca, so both institutes are credited.

/Stefan

> 
> I have been using your software in my lab (APC, Paris)
> to run our data acqusition system. It is very robust and flexible.s
> 
> I would like to give you the large amount of credit which you are due.
> How should I cite both MIDAS and ROODY? I have not been able to find any
> information in the usual places.
> 

Good to hear from a happy user.

I think the best way to give us credit is to recommend MIDAS to 10 of your friends.

For MIDAS citations, I think Pierre and Stefan have a standard one somewhere, we should have it linked from 
midas.triumf.ca.

For ROODY citations, I am not sure we have one. The idea behind it was to make a ROOT version of PAW++. 
Main authors are Greg King (UBC COOP student, where is he now?), Joe Chuma (also author of PHYSICA, 
R.I.P.), Pierre Amaudruz, myself and a few others.

K.O.

Dear MIDAS programmers,

I have been using your software in my lab (APC, Paris)
to run our data acqusition system. It is very robust and flexible.s

I would like to give you the large amount of credit which you are due.
How should I cite both MIDAS and ROODY? I have not been able to find any
information in the usual places.

Cheers, and thanks for the great program!
-Carl

Javascript "AJAX" functions (and their MIDAS wrappers - ODBGet/ODBSet) are subject to something called 
"same origin policy" intended to prevent something called "cross-site scripting attacks", i.e. see
http://en.wikipedia.org/wiki/Same-origin_policy

In practice it means that if you load the MIDAS custom web page from test.foo.com and try to access 
mhttpd at midas.foo.com, ODBSet/ODBGet will not work.

I always thought that this meant that the requests are blocked by the browser and are a form of 
protection of the web server - only scripts loaded from mhttpd can do AJAX (ODBGet/ODBSet) to mhttpd.

It turns out that I was wrong. This is what actually happens: the "cross-site" requests are still sent to the 
server (mhttpd), the response it received, parsed and discarded if "same origin" conditions are not met.

This means that the "same origin" policy does not protect mhttpd at all - any script from any page 
anywhere can issue AJAX requests into any mhttpd, these requests will be successfully sent, received
and processed by mhttpd, including requests for writing into ODB ("jset" command using the HTTP GET 
method).

So for the case of MIDAS, "same origin" does not prevent malicious (or buggy) scripts from writing into the 
wrong mhttpd of the wrong experiment.

All it does is prevent desired and intentional access to mhttpd (ODBGet) from scripts that happen to have 
been loaded outside of mhttpd (i.e. from a developer own test page).

Then it turns out that there is an "official" way to disable this unwanted protection policy, called CORS, see
http://www.w3.org/TR/cors/

I have now implemented this in mhttpd and added an mhttpd.js function ODBSetURL() to explicitly set the 
URL of mhttpd that we want to talk to.

This work is on the feature/ajax branch, to be merged soon. For the impatient, here is what you need to 
do in mhttpd:

diff --git a/src/mhttpd.cxx b/src/mhttpd.cxx
index 1d9d1cc..0460cec 100755
--- a/src/mhttpd.cxx
+++ b/src/mhttpd.cxx
@@ -1070,6 +1070,7 @@ void show_text_header()
 {
    rsprintf("HTTP/1.0 200 Document follows\r\n");
    rsprintf("Server: MIDAS HTTP %d\r\n", mhttpd_revision());
+   rsprintf("Access-Control-Allow-Origin: *\r\n");
    rsprintf("Pragma: no-cache\r\n");
    rsprintf("Expires: Fri, 01 Jan 1983 00:00:00 GMT\r\n");
    rsprintf("Content-Type: text/plain; charset=iso-8859-1\r\n\r\n");

K.O.

> The bulk of the MIDAS AJAX and Javascript functions is now documented on the MIDAS Wiki:
> 
> https://midas.triumf.ca/MidasWiki/index.php/Mhttpd.js
> https://midas.triumf.ca/MidasWiki/index.php/AJAX
> 

The documentation was updated again.

All functions and AJAX methods except jset, jget, jrpc (ODBGet, ODBSet, ODBRpc) and inline edit are now fully 
documented. AJAX methods jset/jget and their javascript wrappers ODBSet/ODBGet/ODBMGet/ODBGetRecord() are 
partially documented. Inline edit will have to be documented by Stefan.

When using these functions please read the "BUG" sections carefully.

When using the Javascript functions (ODBGet, ODBSet, ODBMCopy, etc) please pay special attention to the rules for URI-
encoding function arguments - some functions require that some arguments be pre-encoded by 
"encodeURIComponent()", some functions require some arguments to NOT be pre-encoded. The examples in 
examples/javascript1/example.html are mostly correct.

Special confusion is created by special handling in mhttpd of URI-encoding of parameters named "format".

Special confusion is created by ODBSet(path, value), where "path" should be pre-encoded, while "value" is now encoded 
internally, which is a recent change introduced with the inline edit function. Older versions of mhttpd.js still require that 
"value" be URI-encoded.

Going forward, I hope to resolve most of the confusion by providing a cleaner interface for reading ODB
- ODBMCopy() already looks good with full async JSON/JSONP support (already implemented)
- ODBMKey() to read just the keys, with async JSON/JSONP support (to be added)
- ODBMCreate() to create ODB keys (RPC for db_create()) (to be added)
- ODBMSet() to write into ODB. (to be added)

K.O.

The bulk of the MIDAS AJAX and Javascript functions is now documented on the MIDAS Wiki:

https://midas.triumf.ca/MidasWiki/index.php/Mhttpd.js
https://midas.triumf.ca/MidasWiki/index.php/AJAX

Enjoy,
K.O.

As of commit
https://bitbucket.org/tmidas/midas/commits/dfa5fb1a93cae11a2960d441044c7fd277e1f0ec
(we are now liberated from the tyranny of SVN IDs),
multithreaded run transitions seem to work reliably and are now the default in mhttpd.

In odbedit and mtransition, the default is the old sequential transitions. "-m" and "-a" flags activate 
the new multithread run transitions. mhttpd now uses the equivalent of "mtransition -a" 
(mutithreaded asynchronous).

This is one of the new features implemented by Stefan while at TRIUMF.

K.O.

(We hope to write up all the recent changes soon).

odbedit uses a fixed size buffer for ODB data. If an array in ODB is bugger than this size, 
db_get_data() correctly returns DB_TRUNCATED and there is no memory overwrite, but the following 
code for printing the data does not know about this truncation and proceeds printing memory 
values contained after the end of the fixed size data buffer - in the current case, this memory 
somehow had the contents of the shell history - this very confused the MIDAS users as they though 
that the funny printout was actually in ODB. This is in the print_key() function. If db_get_data() 
returns DB_TRUNCATED, it should allocate a buffer of bigger size. This (and the previous) bug found 
by the TIGRESS experiment. K.O.

The odb.c function merge_records() has a fixed size array of 10000 bytes to handle the data and it 
aborts with an assert() if passed data bigger than that. It is called from db_create_record() which 
already allocates a data buffer of correct size for it's operations.
K.O.