> while (1)
>    wait 10 ms for an event
>    process event, loop back
>    if there is no event, exit
> }


This code has been rewritten now and should work for event rates >100 Hz.

/Stefan

> Doing this through the ODB seems ok to me. If the ODB cannot be accessed, you can fall back to no protection.
> 
> At PSI we fortunately do not have these network scans because PSI uses a institute-wide firewall.
>

Same here at TRIUMF, no problems with hostile network activity. Only see this trouble at CERN. Nominally CERN also have
everything behind the CERN firewall, that is why I tend to think that I am seeing network scans done by CERN security people,
or some badniks on the CERN local network (PC malware, etc).

> So you can connect from outside PSI to inside PSI only 
> on certain well-defined ports (like SSH to certain machines). You can do the same in Alpha. Use one computer as a router with two network cards, where 
> the DAQ network runs on the second card as a private network. Then program the routing tables in that gateway such that only certain ports can be 
> accessed from outside, like port 8080 to mhttpd. This way you block all except the things which are needed.

Yes, this is how we did it for DEAP at SNOLAB. No network trouble there.

But generically for MIDAS, I think we should have built-in capability for MIDAS to protect itself without reliance on OS-level means (local firewall)
or network-level means ("site firewalls").

Sometimes we have very small MIDAS installations, i.e. just one machine by itself, and such setups should be secure/secured easily -
too much work to setup an external firewall box just for one machine and OS-level firewall rules sometimes conflict
with some OS services (i.e. NIS) (I am still waiting for the "NIS to LDAP migration for dummies" guide).

K.O.

> Sometimes we have very small MIDAS installations, i.e. just one machine by itself, and such setups should be secure/secured easily -
> too much work to setup an external firewall box just for one machine and OS-level firewall rules sometimes conflict
> with some OS services (i.e. NIS) (I am still waiting for the "NIS to LDAP migration for dummies" guide).

I fully agree with you. So if you find time to implement this, I will be more than happy.

/Stefan

To support IPV6, we need to migrate MIDAS from gethostbyname() to getaddrinfo(). (Thanks to 
http://www.openwall.com/lists/oss-security/2015/01/27/9). K.O.

MIDAS does not seem to be consistent (or at least convenient) with how it
handles booleans in AJAX functions.

When you request an ODB value that is a boolean with AJAX call like

http://neut14.triumf.ca:8081/?cmd=jcopy&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys

then you get

{ "Hidden/last_written" : 1437065425, "Hidden" : false }

This seems correct, since the JSON convention has booleans encoded as true/false.

But this convention does not work when trying to set the boolean value. For instance

http://neut14.triumf.ca:8081/?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=true

does not set the variable to true. To make this work you need to use the
characters y/n

http://neut14.triumf.ca:8081/?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=y

I tested this with ajax/jset, but the same problem seems to occur when using the
javascript function ODBSet. The documentation doesn't say what sort of encoding
to use when using these functions, so I guess the idea is that these functions
use MIDAS encoding for booleans. But it seems to me that it would be more
convenient if jset/ODBSet allowed the option to use json/javascript encoding for
boolean values; or at least had that as a format option for jset/ODBSet.  That
way my javascript could look like

var mybool = true;
URI_command =
"?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=" + mybool;

instead of

var mybool = true;
URI_command = ""
if(mybool){
  URI_command =
"?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=y";
else
  URI_command =
"?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=n";


__________________________________________________________
Cross-posting from bitbucket issue tracker:

https://bitbucket.org/tmidas/midas/issues/29/jset-odbset-using-true-false-for-booleans

See bitbucket for the solution.

https://bitbucket.org/tmidas/midas/issues/29/jset-odbset-using-true-false-for-booleans#comment-20550474

While I know some of those limits/problems have been already been reported from
DEAP (and maybe corrected in the last version), I am recording them here:

Bugs (not working as it should): 
- "SCRIPT" does not seem to take the parameters into account
- The operators for WAIT are incorrectly set:
the default ">=" and ">" are correct, but "<=", "<", "==" and "!=" are all using
">=" for the test. 

Possible improvements:
- in LOOP, how can I get the index of the LOOP? I used an extra variable that I
increment, but it there a better way?
- PARAM is giving "string" (or a bool) whose size is set by the user input. The
side effect is that if I am making a loop starting at "1", the incrementation
will loop at "9" -> "1". If I start at "01", the incrementation will give "2.",
"3.",... "9.", "10"... The later is probably what most people would use.
- ODBGet (and ODBSet?) does seem to be able to take a variable as a path... I
was trying to use an array whose index would be incremented.

These issues have been addressed by Stefan during his visit at Triumf last month.
The latest git has those fixes.

> While I know some of those limits/problems have been already been reported from
> DEAP (and maybe corrected in the last version), I am recording them here:
> 
> Bugs (not working as it should): 
> - "SCRIPT" does not seem to take the parameters into account

Fixed

> - The operators for WAIT are incorrectly set:
> the default ">=" and ">" are correct, but "<=", "<", "==" and "!=" are all using
> ">=" for the test. 

Fixed

> 
> Possible improvements:
> - in LOOP, how can I get the index of the LOOP? I used an extra variable that I
> increment, but it there a better way?

See LOOP doc
 LOOP cnt, 10
   ODBGET /foo/bflag, bb 
   IF $bb==1 THEN
     SET cnt, 10
   ELSE 
     ... 

> - PARAM is giving "string" (or a bool) whose size is set by the user input. The
> side effect is that if I am making a loop starting at "1", the incrementation
> will loop at "9" -> "1". If I start at "01", the incrementation will give "2.",
> "3.",... "9.", "10"... The later is probably what most people would use.

Fixed

> - ODBGet (and ODBSet?) does seem to be able to take a variable as a path... I
> was trying to use an array whose index would be incremented.

To be checked.

> 
> See LOOP doc
>  LOOP cnt, 10
>    ODBGET /foo/bflag, bb 
>    IF $bb==1 THEN
>      SET cnt, 10
>    ELSE 
>      ... 
> 

Looks like we have PE

> > - ODBGet (and ODBSet?) does seem to be able to take a variable as a path... I
> > was trying to use an array whose index would be incremented.
> 
> To be checked.

It does not take a variable as a path, but as an index. So you can do

LOOP i, 5

  WAIT seconds, 3
  ODBSET /System/Tmp/Test[$i], $i

ENDLOOP


And you will get

/System/Tmp/Test
[1] 1
[2] 2
[3] 3
[4] 4
[5] 5


/Stefan 

New version of MIDAS has a feature where it seems to automatically resize the
message page horizontally in order to fix each MIDAS message into one line. 
Some of my MIDAS messages (in particular error messages, where I need details)
are very long.  The result is that the MIDAS page automatically becomes very
wide and I have to scroll a lot left/right in order to read my messages.  This
is annoying.

I would vote to roll-back this new feature.

> New version of MIDAS has a feature where it seems to automatically resize the
> message page horizontally in order to fix each MIDAS message into one line. 
> Some of my MIDAS messages (in particular error messages, where I need details)
> are very long.  The result is that the MIDAS page automatically becomes very
> wide and I have to scroll a lot left/right in order to read my messages.  This
> is annoying.
> 
> I would vote to roll-back this new feature.

Issues should be reported in the bitbucket issue tracker. I moved this issue over there.

/Stefan

> MIDAS does not seem to be consistent (or at least convenient) with how it
> handles booleans in AJAX functions.

MIDAS documentation does not say that arguments to "jset" (ODBSet) should be JSON-encoded:
https://midas.triumf.ca/MidasWiki/index.php/AJAX#jset

You found undocumented behavior.

As solution, we plan to replace "jset/ODBSet" with a JSON-RPC function, where all parameters will be JSON-encoded 
(obviously).

K.O.


> 
> When you request an ODB value that is a boolean with AJAX call like
> 
> http://neut14.triumf.ca:8081/?cmd=jcopy&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys
> 
> then you get
> 
> { "Hidden/last_written" : 1437065425, "Hidden" : false }
> 
> This seems correct, since the JSON convention has booleans encoded as true/false.
> 
> But this convention does not work when trying to set the boolean value. For instance
> 
> http://neut14.triumf.ca:8081/?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=true
> 
> does not set the variable to true. To make this work you need to use the
> characters y/n
> 
> http://neut14.triumf.ca:8081/?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=y
> 
> I tested this with ajax/jset, but the same problem seems to occur when using the
> javascript function ODBSet. The documentation doesn't say what sort of encoding
> to use when using these functions, so I guess the idea is that these functions
> use MIDAS encoding for booleans. But it seems to me that it would be more
> convenient if jset/ODBSet allowed the option to use json/javascript encoding for
> boolean values; or at least had that as a format option for jset/ODBSet.  That
> way my javascript could look like
> 
> var mybool = true;
> URI_command =
> "?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=" + mybool;
> 
> instead of
> 
> var mybool = true;
> URI_command = ""
> if(mybool){
>   URI_command =
> "?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=y";
> else
>   URI_command =
> "?cmd=jset&odb=/Equipment/DCRC/Common/Hidden&format=json-nokeys&value=n";
> 
> 
> __________________________________________________________
> Cross-posting from bitbucket issue tracker:
> 
> https://bitbucket.org/tmidas/midas/issues/29/jset-odbset-using-true-false-for-booleans

When using the sequencer to start and stop runs which use a deferred transition,
the sequencer fails with a "Cannot stop run: ..." error.

Checking for the "CM_DEFERRED_TRANSITION" case in the first-pass block of the
'Stop' code in sequencer.cxx is one way to solve the problem - though there may
well be better solutions. 

My edited portion of sequencer.cxx is below.  Is this an acceptable solution
that could be introduced to the master branch?

} else if (equal_ustring(mxml_get_value(pn), "Stop")) {
    if (!seq.transition_request) {
       seq.transition_request = TRUE;
       size = sizeof(state);
       db_get_value(hDB, 0, "/Runinfo/State", &state, &size, TID_INT, FALSE);
       if (state != STATE_STOPPED) {
          status = cm_transition(TR_STOP, 0, str, sizeof(str), TR_MTHREAD |
TR_SYNC, TRUE);
          if (status == CM_DEFERRED_TRANSITION) {
             // do nothing
          } else if (status != CM_SUCCESS) {
             sprintf(str, "Cannot stop run: %s", str);
             seq_error(str);
          }
       }
    } else {
       // Wait until transition has finished
       size = sizeof(state);
       db_get_value(hDB, 0, "/Runinfo/State", &state, &size, TID_INT, FALSE);
       if (state == STATE_STOPPED) {
          seq.transition_request = FALSE;

          if (seq.stop_after_run) {
             seq.stop_after_run = FALSE;
             seq.running = FALSE;
             seq.finished = TRUE;
             cm_msg(MTALK, "sequencer", "Sequencer is finished.");
          } else
             seq.current_line_number++;

          db_set_record(hDB, hKeySeq, &seq, sizeof(seq), 0);
       } else {
         // do nothing
       }
    }
}

> > MIDAS does not seem to be consistent (or at least convenient) with how it
> > handles booleans in AJAX functions.

The JSON-RPC functions have been merged into main midas and you can now use the new function mjsonrpc_db_paste(paths, values, id, 
callback, error_callback);

For example:

mjsonrpc_db_paste(["/foo","/bar","/baz"],[1,2,3]);

the target items should already exist (for this example, not in general).

All data is JSON encoded, success/failure is returned via callbacks.

K.O.

Hi,

Attempting to copy a subtree to a new location in the ODB using odbedit with "copy <src> <dest>" is 
occasionally not copying the entire <src> subtree.

I am experiencing this issue consistently when trying to copy subtrees from the "/Equipment" ODB tree to 
a new location.  The first 2-3 variables/directories of the <src> subtree will be copied to <dest> but the 
full subtree will not be copied over.

> Hi,
> 
> Attempting to copy a subtree to a new location in the ODB using odbedit with "copy <src> <dest>" is 
> occasionally not copying the entire <src> subtree.
> 
> I am experiencing this issue consistently when trying to copy subtrees from the "/Equipment" ODB tree to 
> a new location.  The first 2-3 variables/directories of the <src> subtree will be copied to <dest> but the 
> full subtree will not be copied over.

I just tried myself and could successfully copy of even large trees in the /Equipment subtree. Need to reproduce the problem to fix 
it. Maybe close-to-full ODB?

Stefan

Can the MIDAS Sequencer call an external script? It seems that it should be able to. I have a simple 
test script to do so. It claims to execute, but the bash script never appears to be executed. Any 
suggestions?

1 COMMENT "This is a MSL test file"
  2 RUNDESCRIPTION "Test run"
  3 
  4 LOOP setting, 1,2, 3
  5      SCRIPT test_wheel.sh ,$setting 
  6      TRANSITION START
  7      WAIT Seconds 10
  8      TRANSITION STOP
  9 ENDLOOP

I've also tried using an xml script with <Script params="1">test_wheel.sh</Script>, but with the same 
result.

Thanks!

Nevermind. I just had to give it a path to my script. Now it's fine. 

> Can the MIDAS Sequencer call an external script? It seems that it should be able to. I have a simple 
> test script to do so. It claims to execute, but the bash script never appears to be executed. Any 
> suggestions?
> 
> 1 COMMENT "This is a MSL test file"
>   2 RUNDESCRIPTION "Test run"
>   3 
>   4 LOOP setting, 1,2, 3
>   5      SCRIPT test_wheel.sh ,$setting 
>   6      TRANSITION START
>   7      WAIT Seconds 10
>   8      TRANSITION STOP
>   9 ENDLOOP
> 
> I've also tried using an xml script with <Script params="1">test_wheel.sh</Script>, but with the same 
> result.
> 
> Thanks!

I've recently run into issues when using JSON.parse on ODB keys containing 
8-bit data.

For JSON.parse to successfully parse a string, (A) the string must be valid 
UTF-8, (B) several whitespace characters, control characters, and the 
characters " and \ must be escaped, and (C) you've got to follow the key-
value rules laid out in http://www.json.org/.

The web browser takes care of (A), and I verified that for this key Midas 
handled (C) correctly.  In principle, the function json_write in odb.c 
handles (B) - but json_write does not escape control characters.

To manage this problem, I modified json_write (in odb.c) to replace any 
control character with the more-inocuous character, 'C'.  My default case 
now looks like:

default:
         {
           // if a char is a control character,
           // print 'C' in its place
           // note that this loses data:
           // a more-correct method would be to print
           // \uXXXX, where XXXX is the character in hex
           if(iscntrl(*s)){
             (*buffer)[(*buffer_end)++] = 'C';
             s++;
           } else {
             (*buffer)[(*buffer_end)++] = *s++;
           }
         }
      
Where the call to iscntrl(*s) requires the addition of the ctype.h header 
file.

I'm guessing a blanket replacement of control characters with 'C' isn't 
something all Midas users would want to do.  Replacing the control character 
with its hex value seems like a good choice - but not without adding bounds 
checking!

An alternative to changing odb.c could be to add a regex to Midas response 
text which removes all control characters (U+0000 - U+001F): 

var resp_lint = req.response.replace(/[\u{0000}-\u{001F}]/gmu, '');
var json_obj = JSON.parse(resp_lint);

Unfortunately, the 'u' regex flax doesn't work on the Firefox version 
included in Scientific Linux 6.8.