See bitbucket for the solution.

https://bitbucket.org/tmidas/midas/issues/29/jset-odbset-using-true-false-for-booleans#comment-20550474

I tested the new scheme and am quite happy with. Just a minor thing. When I change the ACL, I get messages from all attached programs, like:

[local:Online:S]RPC hosts>set "Allowed hosts[1]" "host.psi.ch"
[ODBEdit,INFO] Reloading RPC hosts access control list via hotlink callback
09:05:11 [mserver,INFO] Reloading RPC hosts access control list via hotlink callback
09:05:11 [ODBEdit,INFO] Reloading RPC hosts access control list via hotlink callback
[ODBEdit,INFO] Reloading RPC hosts access control list via hotlink callback
[ODBEdit,INFO] Reloading RPC hosts access control list via hotlink callback
09:05:11 [mserver,INFO] Reloading RPC hosts access control list via hotlink callback
09:05:11 [ODBEdit,INFO] Reloading RPC hosts access control list via hotlink callback
09:05:11 [mserver,INFO] Reloading RPC hosts access control list via hotlink callback
09:05:11 [ODBEdit,INFO] Reloading RPC hosts access control list via hotlink callback


While this is good for debugging, I would remove it not to confuse the average user.

The trick not to have to "guess" a remote name is quite useful. I'm happy it even made it into the documentation. There is however an
important shortcoming in the documentation. The old documentation had a "quick start" section, which allowed people quickly to
set-up and run a midas system. This is still missing. And it should now contain a link to the "Security" page, so that people can set-up
quickly remote programs.

> > - ODBGet (and ODBSet?) does seem to be able to take a variable as a path... I
> > was trying to use an array whose index would be incremented.
> 
> To be checked.

It does not take a variable as a path, but as an index. So you can do

LOOP i, 5

  WAIT seconds, 3
  ODBSET /System/Tmp/Test[$i], $i

ENDLOOP


And you will get

/System/Tmp/Test
[1] 1
[2] 2
[3] 3
[4] 4
[5] 5


/Stefan 

> New version of MIDAS has a feature where it seems to automatically resize the
> message page horizontally in order to fix each MIDAS message into one line. 
> Some of my MIDAS messages (in particular error messages, where I need details)
> are very long.  The result is that the MIDAS page automatically becomes very
> wide and I have to scroll a lot left/right in order to read my messages.  This
> is annoying.
> 
> I would vote to roll-back this new feature.

Issues should be reported in the bitbucket issue tracker. I moved this issue over there.

/Stefan

> > it would be useful to have a kind of scripted "abort" for when something goes wrong ...
> 
> How about having the sequencer switching from the aborted sequence file to the special "abort" sequence file? That 
> should be simple to implement if it is not already there.
> 
> K.O.

It's about the same effort if we jump to a specific label in a script or to a separate script. I just have to find some time to implement it.

Stefan

> It would be useful to have it be specified for each script.  Reason is that it's simpler, some scripts might only 
> change a few sensitive settings, then on abort it only has to set back to "normal" what it touched to begin with.  
> Also, the "normal" values are usually stored in local variables, so it's important to have those similarly accessible 
> to the "Abort" portion of the script.

Agree. So I will put a special optional label, which will be accessed upon abort.

Stefan

> We're using mhttpd for calls that end up working better with asynchronous requests, and we've built up sort of a parallel, asynchronous library using javascript Promises.
> 
> The Promises (which are in the ES6 spec) have worked incredibly well for building well-behaved, sequential calls to mhttpd.  Personally, I also find their syntax much easier to wrap my
> head around, especially compared to callbacks.
> 
> I'd be happy to add these functions to midas.js if there's general interest. 

Why don't you post the functions here so that we can have a look? They don't have to be incorporated into mhttpd.js necessarily, but could live in a separate file, the people can choose which one to use.

Stefan

A small change in loading .odb files has been implemented. When you load an array from a .odb file, the indices in each line were not evaluated, only the complete array was loaded. In our experiment we need however to load only a few values, like some HV values for some channels but leaving the other values as they are. I changed slightly the code of db_paste() to correctly evaluate the index in each line of the .odb file. This way one can write for example following .odb file:

[/Equipment/HV/Variables]
Demand = FLOAT[256] :
[10] 100.1
[11] 100.2
[12] 100.3
[13] 100.4
[14] 100.5
[15] 100.6

then load it in odbedit via the "load" command, and then only change channels 10-15.

Stefan

> There were some complaints that the MIDAS sequencer was slow.  Specifically, the
> complaint was that even lines in the sequence that didn't do any (like COMMENT
> commands) tooks > 100ms to execute.  These slow sequencer steps could be a
> little annoying if a script had to change a large number of ODB variables before
> starting.
> 
> I tested this a little using a trivial sequence; note that I did all tests using
> mhttpd with mongoose enabled on a newer macbook pro.  I found that with the
> mongoose server each line in a sequencer script was taking ~100ms.  This is
> consistent with the loop in the main thread, which is only doing a cm_yield and
> a sleep:
> 
>    while (!_abort) {
>       status = ss_mutex_wait_for(request_mutex, 0);
>       status = cm_yield(0);
>       if (status == RPC_SHUTDOWN)
>          break;
>       sequencer();
>       status = ss_mutex_release(request_mutex);
>       ss_sleep(100);
>    }
> 
> I tested reducing the sleep to 20ms.  As expected, this made the sequencer more
> zippy, able to execute ~50 commands per second.
> 
> I tried to think what would be downsides to making this change.  I think that
> the main web communication should not be affected, because that communication is
> all handled by the separate mongoose thread.
> 
> I checked how much extra CPU was used if the sleep was reduced from 100ms to
> 20ms.  I found that when a sequence was not running the CPU increased from 0% to
> 0.2% with my change.  When a sequence was running the CPU increased from 0.8% to
> 4% with my change.  4% is a little high, though I'd say still reasonable.  I
> found that most of the CPU usage was occuring because every call to
> 'sequencer()' resulted in a call to db_set_record("/Sequencer/State"...).  I
> guess that making that call 50 times causes the somewhat heavy CPU usage.
> 
> I would argue that it would still be worth making that change, so that the
> sequencer can be more zippy.

The minimal time slice on most systems is 10 ms, and nothing prevents us from switching to
that. The original 100 ms was more for the fact that you can see the sequencer statements
executed one after the other (with the color bar). But this is more a "debugging" feature which 
we not really need. 

To do it "right" the sequencer would have to _return_ a sleep time. Like if it is in a wait loop (as
most of the time), the sleep time could be close to 1 second, to correctly update the wait
progress bar. If the sequencer executes ODB set statements, the wait time could be zero, so
thousands of statements can be executed in one second. The problem we will then have of course
that the sequencer will block the "request_mutex" almost always, which would prevent the
mongoose server from serving anything. So this should be carefully tested. It could be (on most OS)
that releasing the mutex by the main loop immediately switches to the mongoose thread, which would
make the web server still quite responsive, but I'm not sure about that. So as a first change making
the sleep time 10ms should be fine.

Stefan

> Hmm, yeah, I'm not sure about how to handle reducing the wait time to zero after ODB set commands.
> 
> But it does seem like it would be straight-forward to increase the sleep time for waits; I'll look into
> a clean way of doing that.

Let's see how your 10 ms work in real life. If we need variable wait times, I can implement this for your without much effort.

Stefan

> Hi,
> 
> Attempting to copy a subtree to a new location in the ODB using odbedit with "copy <src> <dest>" is 
> occasionally not copying the entire <src> subtree.
> 
> I am experiencing this issue consistently when trying to copy subtrees from the "/Equipment" ODB tree to 
> a new location.  The first 2-3 variables/directories of the <src> subtree will be copied to <dest> but the 
> full subtree will not be copied over.

I just tried myself and could successfully copy of even large trees in the /Equipment subtree. Need to reproduce the problem to fix 
it. Maybe close-to-full ODB?

Stefan

> I recently upgraded my macbook to MacOS 10.11.  The compilation of MIDAS failed after the upgrade, 
> complaining about  
> 
> gcc  -c -g -O2 -Wall <snip> src/mongoose.c
> src/mongoose.c:322:10: fatal error: 'openssl/ssl.h' file not found
> 
> It seems that MacOS has now fully removed openssl header files (they were deprecated for a while).  There 
> seems to be some notes on that here
> 
> http://lists.apple.com/archives/macnetworkprog/2015/Jun/msg00025.html
> 
> Konstantin suggested installing open-source builds of openssl using MacPorts.  I did that and MIDAS 
> compiled fine.  I documented the procedure here:
> 
> https://midas.triumf.ca/MidasWiki/index.php/Installation/Compilation_problems#MacOS_10.11_.28El_Capitan.2
> 9_openssl_errors

The MIDAS Wiki page points to https://guide.macports.org/  which covers OSX up to 10.9. Installers for 10.10 and the current 10.11 
(El Captain) can be found here: https://www.macports.org/install.php

Stefan

> Midas has a nice alarm system. I am wondering whether it is easily possible to
> get the Alarm/Warning banner also on top of custom pages?!

K.O. made nice JavaScript routines to access the alarm status. The new alarm page is completely 
made dynamically from JavaScript code (mhttpd does not supply any HTML code any more, only 
functions to obtain ODB values etc). Part of this new dynamic page must be some code to display 
the alarm status. You just need to copy this to your custom page. K.O. can tell you details.

Stefan

> Hi. Is it currently possible to automatically save the MIDAS ODB as a JSON file?
> I can do it manually in odbedit, but it looks like the only option for the
> automatic ODB save for each run is the standard .ODB format. Is there a way to
> change this?

You mean you like an ODB dump at the end of every run in JSON format?

Sure this can be implemented. But I wonder for what purpose you need that. Can you elaborate a 
bit, maybe it's a useful feature also other people should be aware of. I'm also thinking if we should 
offer a CouchDB interface, so ODB data is written directly to that database.

Stefan

> creating a frontend on MAC Sierra OSX 10
> include the midas.h file and when compiling with XCode I get an error based on
> this entry in the midas.h include
> 
> #if !defined(OS_IRIX) && !defined(OS_VMS) && !defined(OS_MSDOS) &&
> !defined(OS_UNIX) && !defined(OS_VXWORKS) && !defined(OS_WINNT)
> #error MIDAS cannot be used on this operating system
> #endif
> 
> 
> Perhaps I should not use Xcode?
> Perhaps I won't need Midas.h?
> 
> The MIDAS system is running on my MAC but I need to add a very simple front end
> for testing and I encounted this error.

If you compile with the included Makefile, you will see a 

-DOS_LINUX -DOS_DARWIN

flag which tells the compiler that we are on a mac. If you do this with XCode, you have to do it via "Build Settings" (see 
attached picture).

Stefan

> 
> > At Konstantin's suggestion, I committed the function I found for checking if a string was UTF-8 compatible to
> > odb.c.  The function is currently not used; I commented out a proposed use in db_create_key.  Experts can decide
> > if the code was good enough to use.
> 
> After more discussion, I have enabled the parts of the ODB code that check that key names are UTF-8 compliant. 
> 
> This check will show up in (at least) two ways:
> 
> 1) Attempts to create a new ODB variable if the ODB key is not UTF-8 compliant.  You will see error messages like
> 
> [fesimdaq,ERROR] [odb.c:572:db_validate_name,ERROR] Invalid name "Eur€" passed to db_create_key: UTF-8 incompatible
> string
> 
> 2) When a program first connects to the ODB, it runs a check to ensure that the ODB is valid.  This will now include
> a check that all key names are UTF-8 compliant. Any non-UTF8 compliant key names will be replaced by a string of the
> pointer to the key.  You will see error messages like:
> 
> [fesimdaq,ERROR] [odb.c:572:db_validate_name,ERROR] Invalid name "Eur€" passed to db_validate_key: UTF-8
> incompatible string
> [fesimdaq,ERROR] [odb.c:647:db_validate_key,ERROR] Warning: corrected key "/Equipment/SIMDAQ/Eur€": invalid name
> "Eur€" replaced with "0x7f74be63f970"
> 
> This behaviour (checking UTF-8 compatibility and automatically fixing ODB names) can be disabled by setting an
> environment variable
> 
> MIDAS_INVALID_STRING_IS_OK
> 
> It doesn't matter what the environment variable is set to; it just needs to be set.  Note also that this variable is
> only checked once, when a program starts.



I see you put some switches into the environment ("MIDAS_INVALID_STRING_IS_OK"). Do you think this is a good idea? Most variables are 
sitting in the ODB (/experiment/xxx), except those which cannot be in the ODB because we need it before we open the ODB, like MIDAS_DIR. 
Having them in the ODB has the advantage that everything is in one place, and we see a "list" of things we can change. From an empty 
environment it is not clear that such a thing like "MIDAS_INVALID_STRING_IS_OK" does exist, while if it would be an ODB key it would be 
obvious. Can I convince you to move this flag into the ODB?

> > > I switched mhttpd to use the new javascript based run start and stop pages.
> > 
> > One initial complaint: the transition.html page doesn't seem to deal well with a frontend program using
> > a deferred transition.
> >
> 
> We now have a test frontend for deferred transitions, and this problem will likely be fixed.
> 
> > 
> > I separately still think that the transition page should automatically go away after 5 seconds
> >
> 
> This is a user-interface philosophy issue.
> 
> Instead of using personal preferences one should follow established design principles
> (there is research done and books written about this).
> 
> I did not recently look at current recommendations for this type of interaction, but generally
> one expects web pages to "do things" (such as switch to a different page) only when directed
> by user input (press a button).
> 
> My personal opinion is that half the users will find 5 sec delay too slow, the other half will
> find 5 sec too fast and the 3rd half will wonder "what happened, the web page flashed and disappeared,
> did I miss something important, how do I get back to whatever is was?!?".
> 
> One idea is to implement the transition page as a implant on the state page - after the "start" page
> you go back to the status page where you can see the progress of the transition. After the transition
> completes, it's progress window "collapses" into a "success/failure" display with a link to the full
> transition page to see any details of what happened. Any volunteers? (I would html-ize the status page first).
> 
> K.O.

I agree with Konstantin's plans and volunteer for the "collapsable" display. We will address this during my next visit to TRIUMF.

> > 
> > If you compile with the included Makefile, you will see a 
> > 
> > -DOS_LINUX -DOS_DARWIN
> > 
> 
> Moving forward, it looks like I can define these variables in midas.h and remove the need to define them on the compiler command line.
> 
> This would be part of the Makefile and header files cleanup to get things working on Windows10.
> 
> K.O.

Will you detect the underlying OS automatically in midas.h? Note that you have several compilers in MacOS (llvm and gcc), and they might use different 
predefined symbols. I appreciate however getting rid of these flags in the Makefile.

Stefan

> Some additional explanation.
> 
> Time passed, the world turned, and the current web-compatible standard for text strings is UTF-8 encoded Unicode, see 
> https://en.wikipedia.org/wiki/UTF-8
> (ObCanadianContent, UTF-8 was invented the Canadian Rob Pike https://en.wikipedia.org/wiki/Rob_Pike)
> (and by some other guy https://en.wikipedia.org/wiki/Ken_Thompson).
> 
> It turns out that not every combination of 8-bit characters (char*) is valid UTF-8 Unicode.
> 
> In the MIDAS world we run into this when MIDAS ODB strings are exported to Javascript running inside web
> browsers ("custom pages", etc). ODB strings (TID_STRING) and ODB key names that are not valid UTF-8
> make such web pages malfunction and do not work right.
> 
> One solution to this is to declare that ODB strings (TID_STRING) and ODB key names *must* be valid UTF-8 Unicode.
> 
> The present commits implemented this solution. Invalid UTF-8 is rejected by db_create() & co and by the ODB integrity validator.
> 
> This means some existing running experiment may suddenly break because somehow they have "old-style" ODB entries
> or they mistakenly use TID_STRING to store arbitrary binary data (use array of TID_CHAR instead).
> 
> To permit such experiments to use current releases of MIDAS, we include a "defeat" device - to disable UTF-8 checks
> until they figure out where non-UTF-8 strings come from and correct the problem.
> 
> Why is this defeat device non an ODB entry? Because it is not a normal mode of operation - there is no use-case where
> an experiment will continue to use non-UTF-8 compatible ODB indefinitely, in the long term. For example, as the MIDAS user
> interface moves to more and more to HTML+Javascript+"AJAX", such experiments will see that non-UTF-8 compatible ODB entries
> cause all sorts of problems and will have to convert.
> 
> 
> K.O.

Ok, I agree.

Stefan

In my opinion this makes sense. If KO agrees, you should commit your change.

Stefan

> We use the customHeader to display some useful information. Currently I do not
> like its style. What about to make it more alike the footer?
> 
> I just changed in resources/mhttpd.css
> 
> diff --git a/resources/mhttpd.css b/resources/mhttpd.css
> index fb0070d..f3264c8 100644
> --- a/resources/mhttpd.css
> +++ b/resources/mhttpd.css
> @@ -280,6 +280,15 @@ table.headerTable td{
>         border: none;
>  }
>  
> +div.headerDiv{
> +       background-color: #6F6F6F;
> +       text-align: center;
> +       padding:1em;
> +       color:#EEEEEE;
> +       border-bottom:1px solid #000000;
> +       height:3em;
> +}
> +
>  div.footerDiv{
>         background-color: #808080;
>         text-align: center;
> 
> and
> 
> diff --git a/resources/mhttpd.js b/resources/mhttpd.js
> index de8bc6c..972c261 100644
> --- a/resources/mhttpd.js
> +++ b/resources/mhttpd.js
> @@ -172,7 +172,7 @@ function mhttpd_goto_page(page) {
>  
>  function mhttpd_navigation_bar(current_page, path)
>  {
> -   document.write("<div id=\"customHeader\">\n");
> +   document.write("<div class=\"headerDiv\" id=\"customHeader\">\n");
>     document.write("</div>\n");
>  
>     document.write("<div class=\"mnavcss\">\n");
> 
> What do you think?