> > > I vaguely remember that I had a similar problem with ELOG. The solution was to call tzset() at the beginning of the program. The man page says that 
> > > this function is called automatically by programs using time zones, but apparently it is not. Can you try that? There is also the TZ environment 
> > > variable and /etc/localtime. I never understood the details, but playing with these things can influence mktime() and localtime().
> > 
> > I confirm that the timezone is set correctly - I do get the correct time eventually - so there is no missing call to tzet().
> > 
> > K.O.
> 
> tzset() not only sets the time zone, but also DST.

I found following code in elogd.c, maybe it helps:

/* workaround for wong timezone under MAX OSX */
long my_timezone()
{
#if defined(OS_MACOSX) || defined(__FreeBSD__) || defined(__OpenBSD__)
   time_t tp;
   time(&tp);
   return -localtime(&tp)->tm_gmtoff;
#else
   return timezone;
#endif
}



void get_rfc2822_date(char *date, int size, time_t ltime)
{
   time_t now;
   char buf[256];
   int offset;
   struct tm *ts;

   /* switch locale temporarily back to english to comply with RFC2822 date format */
   setlocale(LC_ALL, "C");

   if (ltime == 0)
      time(&now);
   else
      now = ltime;
   ts = localtime(&now);
   assert(ts);
   strftime(buf, sizeof(buf), "%a, %d %b %Y %H:%M:%S", ts);
   offset = (-(int) my_timezone());
   if (ts->tm_isdst)
      offset += 3600;
   snprintf(date, size - 1, "%s %+03d%02d", buf, (int) (offset / 3600),
            (int) ((abs((int) offset) / 60) % 60));
}

> > The bulk of the MIDAS AJAX and Javascript functions is now documented on the MIDAS Wiki:
> > 
> > https://midas.triumf.ca/MidasWiki/index.php/Mhttpd.js
> > https://midas.triumf.ca/MidasWiki/index.php/AJAX
> > 
> 
> The documentation was updated again.
> 

Newly documented are the additional Javascript and AJAX functions present in the GIT branch "feature/ajax":

ODBMCreate(paths, types);
ODBMCreate(paths, types, arraylengths, stringlengths, callback);  
ODBMResize(paths, arraylengths, stringlengths, callback);  
ODBMRename(paths, names, callback);  
ODBMLink(paths, links, callback);  
ODBMReorder(paths, indices, callback);
ODBMKey(paths, callback);
ODBMDelete(paths, callback);

All these functions permit asynchronous use (with callback on completion) and the underlying AJAX functions permit JSON-P encoding.

ODBSetUrl("http://mhttpd.somewhere.com:8080") : this new function removes the restriction that custom scripts had to be loaded from the same mhttpd that they will 
access. Together with the newly added CORS support in mhttpd, allows loading custom scripts from any web server, including local file, and having then access any one (or 
any several) mhttpd data sources.

I think these new functions are now stable (I still had to make some changes to ODBMCreate() recently) and after some more testing this branch will be merged into 
"develop".

To use this branch, do either:
a) git clone midas; git pull; git checkout feature/ajax
b) git clone midas; git checkout develop; git pull; git checkout -b ajaxtest; git merge feature/ajax;

(Option (b) creates a local branch with the latest "develop" and "feature/ajax" merged together).

K.O.

> odbedit can now save ODB in JSON-formatted files.
> 
> JSON encoding implementation follows specifications at:
> http://json.org/
> 
> The result passes validation by:
> http://jsonlint.com/
> 

A bug was reported in my JSON ODB encoder: NaN values are not encoded correctly. A quick review found this:

1) the authors of JSON smoked some bad mushrooms and specifically disallowed NaN and Inf values for floating point numbers: 
http://tools.ietf.org/html/rfc4627
2) most JSON encoders and decoders do reasonable and unreasonable things with NaN and Inf values. The worst ones encode them as zero. More bad 
mushrooms.

There is a quick survey at: http://lavag.org/topic/16217-cr-json-labview/?p=99058

<pre>
Some Javascript engines allow it since it is valid Javascript but not valid Json however there is no concensus.
cmj-JSON4Lua: raw tostring() output (invalid JSON).
dkjson: 'null' (like in the original JSON-implementation).
Fleece: NaN is 0.0000, freezes on +/-Inf.
jf-JSON: NaN is 'null', Inf is 1e+9999 (the encode_pretty function still outputs raw tostring()).
Lua-Yajl: NaN is -0, Inf is 1e+666.
mp-CJSON: raises invalid JSON error by default, but runtime configurable ('null' or Nan/Inf).
nm-luajsonlib: 'null' (like in the original JSON-implementation).
sb-Json: raw tostring() output (invalid JSON).
th-LuaJSON: JavaScript? constants: NaN is 'NaN', Inf is 'Infinity' (this is valid JavaScript?, but invalid JSON).
</pre>

For the MIDAS JSON encoder (and decoder) I have several choices:
a) encode NaN and Inf using the printf("%f") encoding (as strings, making it valid JSON)
b) encode NaN and Inf as strings using the Javascript special values: "NaN", "Infinity" and "-Infinity", see 
http://www.w3schools.com/jsref/jsref_positive_infinity.asp

I note that the Python JSON encoder does (b), see section 18.2.3.3 at http://docs.python.org/2/library/json.html

In either case, behaviour of the JSON decoder on the Javascript side needs to be tested. (Silent conversion to value of zero is not acceptable).

If anybody has an suggestion on this, please let me know.



P.S. If you do not know all about NaN, Inf, "-0" and other floating point funnies, please read:  https://www.ualberta.ca/~kbeach/phys420_580_2010/docs/ACM-Goldberg.pdf

P.P.S. If you ever used the type "float" or "double", used the "/" operator or the function "sqrt()" you also should read that reference.

K.O.

The following code found in mhttpd does not work on MacOS (BSD UNIX).

On Linux, the do-loop will finish after 2 seconds as expected. On MacOS (and other BSD systems), it will 
loop forever.

The cause is the MIDAS watchdog alarm() signal that fires every 1 second and always interrupts the 2 
second sleep of select(). The Linux select() updates it's timeout argument to reflect time already slept, so 
eventually we finish. The MacOS (BSD) select() does not update the timeout argument and select goes back 
to sleep for another 2 seconds (to be again interrupted half-way through).

The POSIX standard (specification for select() & co) permits either behaviour. Compare "man select" on 
MacOS and on Linux.

If the select() timeout were not 2 seconds, but 0.9 seconds; or if the MIDAS watchdog alarm fired every 
2.1 seconds, this problem would also not exist.

I think there are several places in MIDAS with code like this. An audit is required.

{
                  FD_ZERO(&readfds);
                  FD_SET(_sock, &readfds);

                  timeout.tv_sec = 2;
                  timeout.tv_usec = 0;

                  do {
                     status = select(FD_SETSIZE, &readfds, NULL, NULL, &timeout);
                     /* if an alarm signal was cought, restart with reduced timeout */
                  } while (status == -1 && errno == EINTR);
}

K.O.

> > odbedit can now save ODB in JSON-formatted files.
> A bug was reported in my JSON ODB encoder: NaN values are not encoded correctly.

Tested the browser-builtin JSON.stringify() function in google-chrome, firefox, safari, opera:
everybody encodes numeric values NaN and Inf as JSON value [null].

To me, this clearly demonstrates a severe defect in the JSON standard and in it's Javascript implementation:
a) NaN, Inf and -Inf are valid, useful and commonly used numeric values defined by the IEEE754/854 standard (as opposed to the special value "-0", which is also defined by the standard, but is not nearly as useful)
b) they are all distinct numeric values, encoding them all into the same JSON value [null] is the same as encoding all even numbers into the JSON value [42].
c) on the decoding end, JSON value [null] is decoded into Javascript value [null], which works as 0 for numeric computation, so effectively NaN, Inf and -Inf are made equal to zero. A neat trick.

Note that (c) - NaN, Inf is same as 0 - eventually produces incorrect numerical results by breaking the IEEE754/854 standard specification that number+NaN->NaN, number+infinity->infinity, etc.

In MIDAS we have a requirement that results be numerically correct: if an ODB value is "infinity", the corresponding web page should not show "0".

In addition we have a requirement that JSON encoding should be lossess: i.e. ODB contents encoded by JSON should decode back into the same ODB contents.

To satisfy both requirements, I now encode NaN, Inf and -Inf as JSON string values "NaN", "Infinity" and "-Infinity". (Corresponding to the respective Javascript values).

Notes:
1) this is valid JSON
2) it survives decode/encode in the browser (ODBMCopy()/JSON.parse/modify some values/JSON.stringify/ODBMPaste() does not destroy these special values)
3) it is numerically correct for "NaN" values (Javascript [1+"NaN"] -> NaN)
4) it fails in an obvious way for Inf and -Inf values (Javascript [1+"Infinity"] is NaN instead of Infinity).

https://bitbucket.org/tmidas/midas/commits/82dd203cc95dacb6ec9c0a24bc97ffd45bb58427
K.O.

MIDAS "programs" settings include: /programs/xxx/"auto start", "auto restart" and "auto stop". What do 
they do?

"auto start":

if set to "y", the program's "start command" will be unconditionally executed at the beginning of the run 
start transition.

Because there are no checks or tests, the "start command" will be executed even if the program is already 
running. It means that this function cannot be used to start frontend programs - a new copy will be 
started each time, and a previously running copy will be killed.

Also the timing of the program startup and run transition is wrong - in my tests, the program starts too 
late to see the run transition. If the program is a frontend, it will never see the begin-of-run transition.

1st conclusion: "auto start" should be "n" for frontend programs and for any other programs that are 
supposed to be continuously running (mlogger, lazylogger, etc).

2nd conclusion: "auto start" does the same thing as "/programs/execute on start run".

"auto stop":

if set to "y", the program will be stopped after the end of run. (using cm_shutdown).

"auto restart":

this has nothing to do with starting and stopping runs. Instead, it works in conjunction with the alarm 
system and the "program is not running" alarm.

The alarm system periodically calls al_check(). al_check() checks all programs defined under /Programs to 
see if they are running (using cm_exist()). If a program is not running and an alarm is defined, the alarm is 
raised ("program is not running" alarm). If there is a start command and "auto restart" is set to "y", the 
start command is executed.

When using these "auto start" and "auto restart" functions, one needs to be careful about the context 
where the start command will be executed: midas clients may be running from different directories, under 
different user names and on different computers.

In "auto start", the start command is executed from cm_transition. For remote clients, this will happen on 
the remote computer. (against the expectation that the program will be started on the main computer).

In "auto restart", the start command is executed by al_check() which always runs locally (for remote 
clients, it runs inside the mserver). So the started program will always run on the main computer, but 
maybe not in the same directory as when started from the mhttpd "programs -> start" button.

Conclusion:

"programs auto start" : works but has strange interactions and side effects, do not use it.
"programs auto stop" : works, can be used to stop programs at the end of run (but what for?)
"programs auto restart" : works, seems to work correctly, can be used to auto restart mlogger, frontends, 
etc.

K.O.

Record-oriented ODB functions db_create_record(), db_get_record(), db_check_record() and 
db_set_record() require special attention to the consistency between their "C struct"s (usually defined in 
midas.h), their initialization strings (usually defined in midas.h) and the contents of ODB.

When these 3 items become inconsistent, the corresponding midas functions tend to break.

Unlike ODB internal structures and event buffer internal structures, these record-oriented functions are 
not part of the midas binary-compatibility abi and they are not protected by db_validate_sizes().

From time to time, new items are added to some of these data structures. Usually this does not cause 
problems, but recently we had some difficulty with the runinfo and equipment structures, prompting this 
audit.

db_check_record: note: (C) means that this record is created there

alarm.c: alarm_odb_str(C)
mana.c: skipped
mfe.c: equipment_common_str, equipment_statistics_str(C), event_descrip(C), bank_list(C)
mhttpd.cxx: cgif_label_str(C), cgif_bar_str(C), runinfo_str(C), equipment_common_str(C)
mlogger.cxx: ch_settings_str(C)
sequencer.cxx: sequencer_str(C)

db_create_record:

alarm.c: alarm_odb_str, alarm_periodic_str, alarm_class_str
fal.c: skipped
mfe.c: equipment_common_str
midas.c: program_info_str (maybe)
odb.c: (maybe)
lazylogger.cxx: lazy_settings, lazy_statistics
mhttpd.cxx: runinfo_str
mlogger.cxx: chn_settings_str

db_get_record: (hard to do with grep, will have to check every db_get_record by hand)

alarm.c: alarm, class, program_info
fal.c: skipped
mana.c: skipped
midas.c: program_info
odb.c: (maybe)
lazylogger.cxx: lazyst
mhttpd.cxx: runinfo, equipment, ?hkeytemp?, chn_settings, chn_stats, ?label?, ?bar?
mlogger.cxx: ?, ?, chn_stats, chn, settings
sequencer.cxx: hkeyseq

db_set_record:

alarm.c: hkeyalarm, hkeyclass, ???, program_info
fal.c: skipped
mana.c: skipped
mfe.c: equipment_info, ?event structure?
odb.c: (maybe)
lazelogger.cxx: lazyst
mlogger.cxx: chn_stat
sequencer.cxx: seq

db_open_record: note: (W) means MODE_WRITE

fal.c: skipped
mana.c: skipped
mfe.c: equipment_info, equipment_stats(W)
midas.c: requested_transition
odbedit.c: key_update - generic test of hotlink
lazylogger.cxx: runstate, lazyst(W), lazy?
mlogger.cxx: history, chn_statistics, chn_settings
sequencer.cxx: seq

K.O.

> The following code found in mhttpd does not work on MacOS (BSD UNIX). ...

Because of this problem, on MacOS, run transitions can get stuck forever - most timeouts do not work. (Specifically, recv_string() never times out)

K.O.

> "programs auto start" : works but has strange interactions and side effects, do not use it.
> "programs auto stop" : works, can be used to stop programs at the end of run (but what for?)
> "programs auto restart" : works, seems to work correctly, can be used to auto restart mlogger, frontends, 

auto start and auto stop have been requested by PAA loooong time ago. Maybe he remembers if/where this has been used at all. I never used it. So if 
this is the case for others, we can easily change it and won't break anything. Like auto start can be executed before the run transition happens, check 
for a previous version of the program, and only continue when the program is actually running. Should be only a few lines of code. Auto restart is used 
successfully here at PSI, for example for the lazy logger.

/Stefan

In one of the experiments, we hit a long time bug in mdump - there was an array of 32 equipments and if 
there were more than 32 entries under /equipment, it would overrun and corrupt memory. Somehow this 
only showed up after mdump was switched to c++. The solution was to use std::vector instead of fixed 
size array.

Just in case, I checked other midas programs for fixed size arrays (other than fixed size strings) and found 
none. (in midas.c, there is a fixed size array of TR_FIFO[10], but code inspection shows that it cannot 
overrun).

I used this script. It can be modified to also identify any strange sized string arrays.

K.O.

#!/usr/bin/perl -w

while (1) {
  my $in = <STDIN>;
  last unless $in;
  #print $in;

  $in =~ s/^\s+//;

  next if $in =~ /^char/;
  next if $in =~ /^static char/;

  my $a = $in =~ /(.*)[(\d+)\]/;

  next unless $a;

  my $a1 = $1;
  my $a2 = $2;

  next if $a2 == 0;
  next if $a2 == 1;
  next if $a2 == 2;
  next if $a2 == 3;

  #print "[$a] [$a1] [$a2]\n";
  print "-> $a1[$a2]\n";
}

# end

I added MIDAS Makefile rules for building ARM binaries: "make linuxarm" and "make cleanarm" will create 
(and clean) object files, libraries and executables under "linux-arm" using the TI Sitara ARM SDK or the 
Yocto SDK ARM cross-compilers (GCC 4.7.x and 4.8.x respectively). (Makefile rules for building PPC 
binaries have existed for years).

The hardware we have at TRIUMF are "ARMv7" machines - TI Sitara 335x CPUs (google mityarm) and Altera 
Cyclone 5 FPGA ARM (google sockit). (as opposed to the ARMv5 CPU on the RaspberryPi). The software 
binary API standard settled by Fedora Linux is "hard float" (as opposed to "soft float" used by older SDKs).

So "ARMv7 hard float" is what we intend to use at TRIUMF, but ARMv5 and soft-float should also work ok, 
so please report successes and/or problems to this forum.

K.O.

Hello,

I'm co-organizing the upcoming Real Time Conference, which covers also the field of data acquisition, so it might be interesting for people working 
with MIDAS. If you have something to report, you could also consider to send an abstract to this conference. It will be located in Nara, Japan. The conference
site is now open at http://rt2014.rcnp.osaka-u.ac.jp/

Best regards,
Stefan Ritt

I made some tests of MIDAS support for "international characters" and we seem to be in a reasonable 
shape.

The standard standard is UTF-8 encoding of Unicode and the MIDAS core is believed to be UTF-8 clean - 
one can use "international characters" in ODB names, in ODB values, in filenames, etc.

The web interface had some problems with percent-encoding of ODB URLs, but as of current git version, 
everything seems to work okey, as long as the web browser is in the UTF-8 encoding mode. The default 
mode is "Western ISO-8859-1" and javascript encodeURIComponent() is mangling some stuff making the 
ODB editor not work. Switching to UTF-8 mode seems to fix that.

Perhaps we should make the UTF-8 encoding the default for mhttpd-generated web pages. This should be 
okey for TRIUMF - we use English language almost exclusively, but need to check with other labs before 
making such a change. I especially worry about PSI because I am not sure if and how they any of the special 
German-language characters.

On the minus side, odbedit does not seem to accept non-English characters at all. Maybe it is easy to fix.

K.O.

Since several projects at PSI need MSCB but not MIDAS, I decided to separate the two repositories. So if you 
need MIDAS with MSCB support inside mhttpd, you have to clone MIDAS, MXML and MSCB from bitbucket 
(or the local clone at TRIUMF) as described in

https://midas.triumf.ca/MidasWiki/index.php/Main_Page#Download

I tried to fix all Makefiles to link to the new locations, but I'm not sure if I got all. So if something does not 
compile please let me know.

-Stefan

As reported in the bug tracker, the proposed header does not work if no specific (= different from the default 60 sec.) update period is specified, 
since then no cookie is present. Here is the updated code which works for all cases:



<div id="footerDiv" class="footerDiv">
<div style="display:inline; float:left;">MIDAS experiment "Test"</div>
<div id="refr" style="display:inline; float:right;"></div>
</div>
<script type="text/javascript">
var r = document.getElementById('refr');
var now = new Date();
var refr;
if (document.cookie.search('midas_refr') == -1)
   refr = 60;
else {
   var c = document.cookie.split('midas_refr=');
   refr = c.pop().split(';').shift();
}
r.innerHTML = now.toString() + '&nbsp;&nbsp;&nbsp;' + 'Refr:' + refr;
</script>



/Stefan

> Since several projects at PSI need MSCB but not MIDAS, I decided to separate the two repositories. So if you 
> need MIDAS with MSCB support inside mhttpd, you have to clone MIDAS, MXML and MSCB from bitbucket 
> (or the local clone at TRIUMF) as described in
> 
> https://midas.triumf.ca/MidasWiki/index.php/Main_Page#Download
> 
> I tried to fix all Makefiles to link to the new locations, but I'm not sure if I got all. So if something does not 
> compile please let me know.
> 
> -Stefan

After this split, Makefiles used to build experiment frontends need to be modified for the new location of the mscb tree:

replace
$(MIDASSYS)/mscb
with
$(MIDASSYS)/../mscb

K.O.

I am not sure what to do with the javascript snippet - I understand it should be somehow connected to /Custom/Header, but if I create the /Custom/Header string, I cannot put this snippet 
into this string using odbedit - if I try to cut&paste it into odbedit, it is truncated to the first line - nor using the mhttpd odb editor - when I cut&paste it into the odb editor text entry box, it 
is truncated to the first 519 bytes (must be a hard limit somewhere). K.O.

> As reported in the bug tracker, the proposed header does not work if no specific (= different from the default 60 sec.) update period is specified, 
> since then no cookie is present. Here is the updated code which works for all cases:
> 
> 
> 
> <div id="footerDiv" class="footerDiv">
> <div style="display:inline; float:left;">MIDAS experiment "Test"</div>
> <div id="refr" style="display:inline; float:right;"></div>
> </div>
> <script type="text/javascript">
> var r = document.getElementById('refr');
> var now = new Date();
> var refr;
> if (document.cookie.search('midas_refr') == -1)
>    refr = 60;
> else {
>    var c = document.cookie.split('midas_refr=');
>    refr = c.pop().split(';').shift();
> }
> r.innerHTML = now.toString() + '&nbsp;&nbsp;&nbsp;' + 'Refr:' + refr;
> </script>
> 
> 
> 
> /Stefan

> I am not sure what to do with the javascript snippet 

Just read elog:908, it tells you to put this into a file, name it header.html for example, and put into the ODB:

/Custom/Header [string32] = header.html

make sure that you put the file into the directory indicated by /Custom/Path.

Cheers,
Stefan

I made a few minor modifications to the ODB JSON encoder and implemented a javascript "ls" function to 
report full ODB directory information as available from odbedit "ls -l" and the mhttpd odb editor page.

Using the new ODBMLs(), the existing ODBMCreate(), ODBMDelete() & etc a complete ODB editor can be 
written in Javascript (or in any other AJAX-capable language).

While implementing this function, I found some problems in the ODB JSON encoder when handling 
symlinks, also some problems with handling symlinks in odbedit and in the mhttpd ODB editor - these are 
now fixed.

Changes to the ODB JSON encoder:
- added the missing information to the ODB KEY (access_mode, notify_count)
- added symlink target information ("link")
- changed encoding of simple variable (i.e. jcopy of /experiment/name) - when possible (i.e. ODB KEY 
information is omitted), they are encoded as bare values (before, they were always encoded as structures 
with variable names, etc). This change makes it possible to implement ODBGet() and ODBMGet() using the 
AJAX jcopy method with JSON data encoding. Bare value encoding in ODBMCopy()/AJAX jcopy is enabled by 
using the "json-nokeys-nolastwritten" encoding option.

All these changes are supposed to be backward compatible (encoding used by ODBMCopy() for simple 
values and "-nokeys-nolastwritten" was previously not documented).

Documentation was updated:
https://midas.triumf.ca/MidasWiki/index.php/Mhttpd.js

K.O.

In the old SVN midas world it was typically such that the Windows dll's and
exe's were ready to be used when checking out. I am not so sure this is the case
for the current version, since when I use the packed dll's  and exe's (e.g.
odbedit.exe) I get the warning that this is running midas 2.0.0 but the current
version (on the linux server) is 2.1.

What does this mean?

1) A little bug in the packed windows part, but up-to-date dll's and exe's?
2) The dll's and exe's are not bundled any more to up-to-date version?

If 2) is the case, I would like to get a hint how to build midas under Windows
(Windows 7), since we still have some few Windows clients.