> > odbedit can now save ODB in JSON-formatted files.
> A bug was reported in my JSON ODB encoder: NaN values are not encoded correctly.

Tested the browser-builtin JSON.stringify() function in google-chrome, firefox, safari, opera:
everybody encodes numeric values NaN and Inf as JSON value [null].

To me, this clearly demonstrates a severe defect in the JSON standard and in it's Javascript implementation:
a) NaN, Inf and -Inf are valid, useful and commonly used numeric values defined by the IEEE754/854 standard (as opposed to the special value "-0", which is also defined by the standard, but is not nearly as useful)
b) they are all distinct numeric values, encoding them all into the same JSON value [null] is the same as encoding all even numbers into the JSON value [42].
c) on the decoding end, JSON value [null] is decoded into Javascript value [null], which works as 0 for numeric computation, so effectively NaN, Inf and -Inf are made equal to zero. A neat trick.

Note that (c) - NaN, Inf is same as 0 - eventually produces incorrect numerical results by breaking the IEEE754/854 standard specification that number+NaN->NaN, number+infinity->infinity, etc.

In MIDAS we have a requirement that results be numerically correct: if an ODB value is "infinity", the corresponding web page should not show "0".

In addition we have a requirement that JSON encoding should be lossess: i.e. ODB contents encoded by JSON should decode back into the same ODB contents.

To satisfy both requirements, I now encode NaN, Inf and -Inf as JSON string values "NaN", "Infinity" and "-Infinity". (Corresponding to the respective Javascript values).

Notes:
1) this is valid JSON
2) it survives decode/encode in the browser (ODBMCopy()/JSON.parse/modify some values/JSON.stringify/ODBMPaste() does not destroy these special values)
3) it is numerically correct for "NaN" values (Javascript [1+"NaN"] -> NaN)
4) it fails in an obvious way for Inf and -Inf values (Javascript [1+"Infinity"] is NaN instead of Infinity).

https://bitbucket.org/tmidas/midas/commits/82dd203cc95dacb6ec9c0a24bc97ffd45bb58427
K.O.

MIDAS "programs" settings include: /programs/xxx/"auto start", "auto restart" and "auto stop". What do 
they do?

"auto start":

if set to "y", the program's "start command" will be unconditionally executed at the beginning of the run 
start transition.

Because there are no checks or tests, the "start command" will be executed even if the program is already 
running. It means that this function cannot be used to start frontend programs - a new copy will be 
started each time, and a previously running copy will be killed.

Also the timing of the program startup and run transition is wrong - in my tests, the program starts too 
late to see the run transition. If the program is a frontend, it will never see the begin-of-run transition.

1st conclusion: "auto start" should be "n" for frontend programs and for any other programs that are 
supposed to be continuously running (mlogger, lazylogger, etc).

2nd conclusion: "auto start" does the same thing as "/programs/execute on start run".

"auto stop":

if set to "y", the program will be stopped after the end of run. (using cm_shutdown).

"auto restart":

this has nothing to do with starting and stopping runs. Instead, it works in conjunction with the alarm 
system and the "program is not running" alarm.

The alarm system periodically calls al_check(). al_check() checks all programs defined under /Programs to 
see if they are running (using cm_exist()). If a program is not running and an alarm is defined, the alarm is 
raised ("program is not running" alarm). If there is a start command and "auto restart" is set to "y", the 
start command is executed.

When using these "auto start" and "auto restart" functions, one needs to be careful about the context 
where the start command will be executed: midas clients may be running from different directories, under 
different user names and on different computers.

In "auto start", the start command is executed from cm_transition. For remote clients, this will happen on 
the remote computer. (against the expectation that the program will be started on the main computer).

In "auto restart", the start command is executed by al_check() which always runs locally (for remote 
clients, it runs inside the mserver). So the started program will always run on the main computer, but 
maybe not in the same directory as when started from the mhttpd "programs -> start" button.

Conclusion:

"programs auto start" : works but has strange interactions and side effects, do not use it.
"programs auto stop" : works, can be used to stop programs at the end of run (but what for?)
"programs auto restart" : works, seems to work correctly, can be used to auto restart mlogger, frontends, 
etc.

K.O.

Record-oriented ODB functions db_create_record(), db_get_record(), db_check_record() and 
db_set_record() require special attention to the consistency between their "C struct"s (usually defined in 
midas.h), their initialization strings (usually defined in midas.h) and the contents of ODB.

When these 3 items become inconsistent, the corresponding midas functions tend to break.

Unlike ODB internal structures and event buffer internal structures, these record-oriented functions are 
not part of the midas binary-compatibility abi and they are not protected by db_validate_sizes().

From time to time, new items are added to some of these data structures. Usually this does not cause 
problems, but recently we had some difficulty with the runinfo and equipment structures, prompting this 
audit.

db_check_record: note: (C) means that this record is created there

alarm.c: alarm_odb_str(C)
mana.c: skipped
mfe.c: equipment_common_str, equipment_statistics_str(C), event_descrip(C), bank_list(C)
mhttpd.cxx: cgif_label_str(C), cgif_bar_str(C), runinfo_str(C), equipment_common_str(C)
mlogger.cxx: ch_settings_str(C)
sequencer.cxx: sequencer_str(C)

db_create_record:

alarm.c: alarm_odb_str, alarm_periodic_str, alarm_class_str
fal.c: skipped
mfe.c: equipment_common_str
midas.c: program_info_str (maybe)
odb.c: (maybe)
lazylogger.cxx: lazy_settings, lazy_statistics
mhttpd.cxx: runinfo_str
mlogger.cxx: chn_settings_str

db_get_record: (hard to do with grep, will have to check every db_get_record by hand)

alarm.c: alarm, class, program_info
fal.c: skipped
mana.c: skipped
midas.c: program_info
odb.c: (maybe)
lazylogger.cxx: lazyst
mhttpd.cxx: runinfo, equipment, ?hkeytemp?, chn_settings, chn_stats, ?label?, ?bar?
mlogger.cxx: ?, ?, chn_stats, chn, settings
sequencer.cxx: hkeyseq

db_set_record:

alarm.c: hkeyalarm, hkeyclass, ???, program_info
fal.c: skipped
mana.c: skipped
mfe.c: equipment_info, ?event structure?
odb.c: (maybe)
lazelogger.cxx: lazyst
mlogger.cxx: chn_stat
sequencer.cxx: seq

db_open_record: note: (W) means MODE_WRITE

fal.c: skipped
mana.c: skipped
mfe.c: equipment_info, equipment_stats(W)
midas.c: requested_transition
odbedit.c: key_update - generic test of hotlink
lazylogger.cxx: runstate, lazyst(W), lazy?
mlogger.cxx: history, chn_statistics, chn_settings
sequencer.cxx: seq

K.O.

> The following code found in mhttpd does not work on MacOS (BSD UNIX). ...

Because of this problem, on MacOS, run transitions can get stuck forever - most timeouts do not work. (Specifically, recv_string() never times out)

K.O.

> "programs auto start" : works but has strange interactions and side effects, do not use it.
> "programs auto stop" : works, can be used to stop programs at the end of run (but what for?)
> "programs auto restart" : works, seems to work correctly, can be used to auto restart mlogger, frontends, 

auto start and auto stop have been requested by PAA loooong time ago. Maybe he remembers if/where this has been used at all. I never used it. So if 
this is the case for others, we can easily change it and won't break anything. Like auto start can be executed before the run transition happens, check 
for a previous version of the program, and only continue when the program is actually running. Should be only a few lines of code. Auto restart is used 
successfully here at PSI, for example for the lazy logger.

/Stefan

In one of the experiments, we hit a long time bug in mdump - there was an array of 32 equipments and if 
there were more than 32 entries under /equipment, it would overrun and corrupt memory. Somehow this 
only showed up after mdump was switched to c++. The solution was to use std::vector instead of fixed 
size array.

Just in case, I checked other midas programs for fixed size arrays (other than fixed size strings) and found 
none. (in midas.c, there is a fixed size array of TR_FIFO[10], but code inspection shows that it cannot 
overrun).

I used this script. It can be modified to also identify any strange sized string arrays.

K.O.

#!/usr/bin/perl -w

while (1) {
  my $in = <STDIN>;
  last unless $in;
  #print $in;

  $in =~ s/^\s+//;

  next if $in =~ /^char/;
  next if $in =~ /^static char/;

  my $a = $in =~ /(.*)[(\d+)\]/;

  next unless $a;

  my $a1 = $1;
  my $a2 = $2;

  next if $a2 == 0;
  next if $a2 == 1;
  next if $a2 == 2;
  next if $a2 == 3;

  #print "[$a] [$a1] [$a2]\n";
  print "-> $a1[$a2]\n";
}

# end

I added MIDAS Makefile rules for building ARM binaries: "make linuxarm" and "make cleanarm" will create 
(and clean) object files, libraries and executables under "linux-arm" using the TI Sitara ARM SDK or the 
Yocto SDK ARM cross-compilers (GCC 4.7.x and 4.8.x respectively). (Makefile rules for building PPC 
binaries have existed for years).

The hardware we have at TRIUMF are "ARMv7" machines - TI Sitara 335x CPUs (google mityarm) and Altera 
Cyclone 5 FPGA ARM (google sockit). (as opposed to the ARMv5 CPU on the RaspberryPi). The software 
binary API standard settled by Fedora Linux is "hard float" (as opposed to "soft float" used by older SDKs).

So "ARMv7 hard float" is what we intend to use at TRIUMF, but ARMv5 and soft-float should also work ok, 
so please report successes and/or problems to this forum.

K.O.

Hello,

I'm co-organizing the upcoming Real Time Conference, which covers also the field of data acquisition, so it might be interesting for people working 
with MIDAS. If you have something to report, you could also consider to send an abstract to this conference. It will be located in Nara, Japan. The conference
site is now open at http://rt2014.rcnp.osaka-u.ac.jp/

Best regards,
Stefan Ritt

I made some tests of MIDAS support for "international characters" and we seem to be in a reasonable 
shape.

The standard standard is UTF-8 encoding of Unicode and the MIDAS core is believed to be UTF-8 clean - 
one can use "international characters" in ODB names, in ODB values, in filenames, etc.

The web interface had some problems with percent-encoding of ODB URLs, but as of current git version, 
everything seems to work okey, as long as the web browser is in the UTF-8 encoding mode. The default 
mode is "Western ISO-8859-1" and javascript encodeURIComponent() is mangling some stuff making the 
ODB editor not work. Switching to UTF-8 mode seems to fix that.

Perhaps we should make the UTF-8 encoding the default for mhttpd-generated web pages. This should be 
okey for TRIUMF - we use English language almost exclusively, but need to check with other labs before 
making such a change. I especially worry about PSI because I am not sure if and how they any of the special 
German-language characters.

On the minus side, odbedit does not seem to accept non-English characters at all. Maybe it is easy to fix.

K.O.

Since several projects at PSI need MSCB but not MIDAS, I decided to separate the two repositories. So if you 
need MIDAS with MSCB support inside mhttpd, you have to clone MIDAS, MXML and MSCB from bitbucket 
(or the local clone at TRIUMF) as described in

https://midas.triumf.ca/MidasWiki/index.php/Main_Page#Download

I tried to fix all Makefiles to link to the new locations, but I'm not sure if I got all. So if something does not 
compile please let me know.

-Stefan

As reported in the bug tracker, the proposed header does not work if no specific (= different from the default 60 sec.) update period is specified, 
since then no cookie is present. Here is the updated code which works for all cases:



<div id="footerDiv" class="footerDiv">
<div style="display:inline; float:left;">MIDAS experiment "Test"</div>
<div id="refr" style="display:inline; float:right;"></div>
</div>
<script type="text/javascript">
var r = document.getElementById('refr');
var now = new Date();
var refr;
if (document.cookie.search('midas_refr') == -1)
   refr = 60;
else {
   var c = document.cookie.split('midas_refr=');
   refr = c.pop().split(';').shift();
}
r.innerHTML = now.toString() + '&nbsp;&nbsp;&nbsp;' + 'Refr:' + refr;
</script>



/Stefan

> Since several projects at PSI need MSCB but not MIDAS, I decided to separate the two repositories. So if you 
> need MIDAS with MSCB support inside mhttpd, you have to clone MIDAS, MXML and MSCB from bitbucket 
> (or the local clone at TRIUMF) as described in
> 
> https://midas.triumf.ca/MidasWiki/index.php/Main_Page#Download
> 
> I tried to fix all Makefiles to link to the new locations, but I'm not sure if I got all. So if something does not 
> compile please let me know.
> 
> -Stefan

After this split, Makefiles used to build experiment frontends need to be modified for the new location of the mscb tree:

replace
$(MIDASSYS)/mscb
with
$(MIDASSYS)/../mscb

K.O.

I am not sure what to do with the javascript snippet - I understand it should be somehow connected to /Custom/Header, but if I create the /Custom/Header string, I cannot put this snippet 
into this string using odbedit - if I try to cut&paste it into odbedit, it is truncated to the first line - nor using the mhttpd odb editor - when I cut&paste it into the odb editor text entry box, it 
is truncated to the first 519 bytes (must be a hard limit somewhere). K.O.

> As reported in the bug tracker, the proposed header does not work if no specific (= different from the default 60 sec.) update period is specified, 
> since then no cookie is present. Here is the updated code which works for all cases:
> 
> 
> 
> <div id="footerDiv" class="footerDiv">
> <div style="display:inline; float:left;">MIDAS experiment "Test"</div>
> <div id="refr" style="display:inline; float:right;"></div>
> </div>
> <script type="text/javascript">
> var r = document.getElementById('refr');
> var now = new Date();
> var refr;
> if (document.cookie.search('midas_refr') == -1)
>    refr = 60;
> else {
>    var c = document.cookie.split('midas_refr=');
>    refr = c.pop().split(';').shift();
> }
> r.innerHTML = now.toString() + '&nbsp;&nbsp;&nbsp;' + 'Refr:' + refr;
> </script>
> 
> 
> 
> /Stefan

> I am not sure what to do with the javascript snippet 

Just read elog:908, it tells you to put this into a file, name it header.html for example, and put into the ODB:

/Custom/Header [string32] = header.html

make sure that you put the file into the directory indicated by /Custom/Path.

Cheers,
Stefan

I made a few minor modifications to the ODB JSON encoder and implemented a javascript "ls" function to 
report full ODB directory information as available from odbedit "ls -l" and the mhttpd odb editor page.

Using the new ODBMLs(), the existing ODBMCreate(), ODBMDelete() & etc a complete ODB editor can be 
written in Javascript (or in any other AJAX-capable language).

While implementing this function, I found some problems in the ODB JSON encoder when handling 
symlinks, also some problems with handling symlinks in odbedit and in the mhttpd ODB editor - these are 
now fixed.

Changes to the ODB JSON encoder:
- added the missing information to the ODB KEY (access_mode, notify_count)
- added symlink target information ("link")
- changed encoding of simple variable (i.e. jcopy of /experiment/name) - when possible (i.e. ODB KEY 
information is omitted), they are encoded as bare values (before, they were always encoded as structures 
with variable names, etc). This change makes it possible to implement ODBGet() and ODBMGet() using the 
AJAX jcopy method with JSON data encoding. Bare value encoding in ODBMCopy()/AJAX jcopy is enabled by 
using the "json-nokeys-nolastwritten" encoding option.

All these changes are supposed to be backward compatible (encoding used by ODBMCopy() for simple 
values and "-nokeys-nolastwritten" was previously not documented).

Documentation was updated:
https://midas.triumf.ca/MidasWiki/index.php/Mhttpd.js

K.O.

In the old SVN midas world it was typically such that the Windows dll's and
exe's were ready to be used when checking out. I am not so sure this is the case
for the current version, since when I use the packed dll's  and exe's (e.g.
odbedit.exe) I get the warning that this is running midas 2.0.0 but the current
version (on the linux server) is 2.1.

What does this mean?

1) A little bug in the packed windows part, but up-to-date dll's and exe's?
2) The dll's and exe's are not bundled any more to up-to-date version?

If 2) is the case, I would like to get a hint how to build midas under Windows
(Windows 7), since we still have some few Windows clients.  

The midas wiki at https://midas.triumf.ca was updated to mediawiki 1.22.4 - the latest production version. 
If you see any problems, please report them to this elog. K.O.

> In the old SVN midas world it was typically such that the Windows dll's and
> exe's were ready to be used when checking out.

The Windows executables are no longer included in the midas git repository. Old versions are still available in 
the git repository - they got pulled in during conversion from svn.

One reason for removing them is that neither myself, nor Pierre, nor Stefan have ready access to a Windows 
development environment and we cannot keep Windows binaries up to date. Theoretically we can setup a 
Windows machine just for compiling MIDAS, but then there is a question of which Windows we should use and 
how much priority we should put into it. I do not think there is any demand for MIDAS on Windows at TRIUMF.

(Personally, I think Windows is no longer a viable platform for any business use - with Microsoft focusing on 
"experiences", "tiles", touch screens, portable devices, and other gimmicks - rather than on providing a solid OS 
to get work done)

> I am not so sure this is the case
> for the current version, since when I use the packed dll's  and exe's (e.g.
> odbedit.exe) I get the warning that this is running midas 2.0.0 but the current
> version (on the linux server) is 2.1. What does this mean?

You can ignore this message. Stefan incremented the MIDAS version when we migrated to git, but
there are no changes to the MIDAS RPC mechanism and we are still fully compatible with old versions,
at least in the MIDAS RPC and in the mserver.

So tools like odbedit.exe should still work okey when connecting from Windows to MIDAS running on Linux or 
MacOS.

But old frontend programs may cause some trouble because the ODB layout changed somewhat with new things 
added to /eq/xxx/common. Simplest is to try, if it works, it works.

> 1) A little bug in the packed windows part, but up-to-date dll's and exe's?
> 2) The dll's and exe's are not bundled any more to up-to-date version?

Case (2) is the case. Personally I do not have any capability to build Windows binaries. Same for Pierre and I think 
for Stefan.

> If 2) is the case, I would like to get a hint how to build midas under Windows
> (Windows 7), since we still have some few Windows clients.  

I do not think pre-built executables will ever return - the new way of things is to "cut-and-paste" the "git clone" 
command from a web page, type "make", and be done with it. If your OS does not have "git", "make" & etc, you 
should switch to a real OS.

On the MIDAS software side, we have no problem with supporting Windows - same as on any other platform, 
please try to build and run it, report any problems, fixes, patches and improvements - we will commit them into 
the midas repository.

K.O.

> The Windows executables are no longer included in the midas git repository. Old versions are still available in 
> the git repository - they got pulled in during conversion from svn.
> 
> One reason for removing them is that neither myself, nor Pierre, nor Stefan have ready access to a Windows 
> development environment and we cannot keep Windows binaries up to date. Theoretically we can setup a 
> Windows machine just for compiling MIDAS, but then there is a question of which Windows we should use and 
> how much priority we should put into it. I do not think there is any demand for MIDAS on Windows at TRIUMF.

I double checked and can confirm that the executables in GIT are very old. So I tried to compile the current version for Windows. I found that I had to change lots 
of places (basically all the new files written by KO) to make it work again, so it took me half a day, but now should be fine.

I'm not sure if it's a good idea to keep .exe files in GIT, maybe we should remove it some day, but for the moment I updated the executables to the current 
version. Feedback welcome.

/Stefan

> > > odbedit can now save ODB in JSON-formatted files.
> encode NaN, Inf and -Inf as JSON string values "NaN", "Infinity" and "-Infinity". (Corresponding to the respective Javascript values).

A new standard just came out - Oasis OData JSON format 4.0 - 
http://docs.oasis-open.org/odata/odata-json-format/v4.0/os/odata-json-format-v4.0-os.html

Section 7.1 reads:

> Values of types [...] Edm.Single, Edm.Double, and Edm.Decimal are represented as JSON numbers, except for NaN, INF, and –INF which are represented as strings.

This is consistent with what we do in MIDAS - encode special numbers as strings. For now I think we stay with Javascript-standard "Infinity", "-Infinity",
but if more standards start using "INF", "-INF", maybe we will switch. It is easy enough to support both encodings in the JSON parser and in the ODB decoder.

https://xkcd.com/927/
K.O.