> Hi Stefan, thanks for the quick reply. I guess my question was not clear enough.
> 
> My aim is to create a button which mimics the "Start/Stop" button functionality in the
> "Programs" page where we start all the front-ends for the various equipment. The idea is that
> the user will use a simple interface in a custom page (not the status page) which sets up the
> equipment needed for a specific type of measurement.

All functions in midas are controlled through special URLs. So the URL

http://<host:port>/?cmd=Start&value=10

will start run #10. Similarly with ?cmd=Stop. Now all you need is to set up a custom button, and use the 
OnClick="" JavaScript method to fire off an Ajax request with the above URL. 

To send an Ajax request, you can use the function XMLHttpRequestGeneric which ships as part of midas in the 
mhttpd.js file. Then the code would be

<input type="button" onclick="start()">

and in your JavaScript code:

...
function start()
{
   var request = XMLHttpRequestGeneric();

   url = '?cmd=Start&value=10';
   request.open('GET', url, false);
   request.send(null);
}
...


Cheers,
Stefan

> Any thoughts on this?

We use binary midas files now for ~20 years and never felt the necessity to put any checksums or even encryption on these files. The reason for that is the following: Data on 
modern hard disks is already protected by CRC code or even ERC on the lower level, so it's very unlikely that single bytes change. If something happens, then it's a 
corruption of the file system, so a few sectors of a file are missing or wrong. In that case a CRC won't help you much, just tells you that the files are corrupt. But you see that 
also in the midas event structure. Each event has a header with the size of the event, so you can follow the file event by event. If something is missing, the next event header 
is no event header but something in the middle of the date, and you recognise this immediately since the header does not make any send (date is off by many years, event ID 
is arbitrary, event size is very different). So this redundancy in the midas event structure helps you to identify any corrupt files as good in my opinion as a CRC code will. I 
would not want to waste a single CPU cycle on lengthy CRC or even SHA algorithms, unless I see single bytes change inside events. But in this case this can even happen at 
the network level between frontend and backends. So we should add the CRC/SHA code at the frontend level. This could increase the dead time of the experiment which is 
bad. And what about VME transfer? While hard disks and Ethernet networks have already built-in CRC checks, VME transfer doesn't. So how can you be sure that no bits 
get corrupt between your ADC and your frontend computer?

If people insist of having CRC or SHA protection/encryption for some reason I do not understand yet, we should make this optional, so that I can turn it off, since I don't 
need it.

/Stefan

> If anyone has any ideas or has had previous work with the SCS2000 and knows how to read back the 
> internal values please let me know. 

The current MIDAS distribution contains a file /midas/examples/slowcont/mscb_fe.c which contains example code of how to read some MSCB devices.

/Stefan

See bitbucket for the solution.

https://bitbucket.org/tmidas/midas/issues/29/jset-odbset-using-true-false-for-booleans#comment-20550474

I tested the new scheme and am quite happy with. Just a minor thing. When I change the ACL, I get messages from all attached programs, like:

[local:Online:S]RPC hosts>set "Allowed hosts[1]" "host.psi.ch"
[ODBEdit,INFO] Reloading RPC hosts access control list via hotlink callback
09:05:11 [mserver,INFO] Reloading RPC hosts access control list via hotlink callback
09:05:11 [ODBEdit,INFO] Reloading RPC hosts access control list via hotlink callback
[ODBEdit,INFO] Reloading RPC hosts access control list via hotlink callback
[ODBEdit,INFO] Reloading RPC hosts access control list via hotlink callback
09:05:11 [mserver,INFO] Reloading RPC hosts access control list via hotlink callback
09:05:11 [ODBEdit,INFO] Reloading RPC hosts access control list via hotlink callback
09:05:11 [mserver,INFO] Reloading RPC hosts access control list via hotlink callback
09:05:11 [ODBEdit,INFO] Reloading RPC hosts access control list via hotlink callback


While this is good for debugging, I would remove it not to confuse the average user.

The trick not to have to "guess" a remote name is quite useful. I'm happy it even made it into the documentation. There is however an
important shortcoming in the documentation. The old documentation had a "quick start" section, which allowed people quickly to
set-up and run a midas system. This is still missing. And it should now contain a link to the "Security" page, so that people can set-up
quickly remote programs.

> > - ODBGet (and ODBSet?) does seem to be able to take a variable as a path... I
> > was trying to use an array whose index would be incremented.
> 
> To be checked.

It does not take a variable as a path, but as an index. So you can do

LOOP i, 5

  WAIT seconds, 3
  ODBSET /System/Tmp/Test[$i], $i

ENDLOOP


And you will get

/System/Tmp/Test
[1] 1
[2] 2
[3] 3
[4] 4
[5] 5


/Stefan 

> New version of MIDAS has a feature where it seems to automatically resize the
> message page horizontally in order to fix each MIDAS message into one line. 
> Some of my MIDAS messages (in particular error messages, where I need details)
> are very long.  The result is that the MIDAS page automatically becomes very
> wide and I have to scroll a lot left/right in order to read my messages.  This
> is annoying.
> 
> I would vote to roll-back this new feature.

Issues should be reported in the bitbucket issue tracker. I moved this issue over there.

/Stefan

> > it would be useful to have a kind of scripted "abort" for when something goes wrong ...
> 
> How about having the sequencer switching from the aborted sequence file to the special "abort" sequence file? That 
> should be simple to implement if it is not already there.
> 
> K.O.

It's about the same effort if we jump to a specific label in a script or to a separate script. I just have to find some time to implement it.

Stefan

> It would be useful to have it be specified for each script.  Reason is that it's simpler, some scripts might only 
> change a few sensitive settings, then on abort it only has to set back to "normal" what it touched to begin with.  
> Also, the "normal" values are usually stored in local variables, so it's important to have those similarly accessible 
> to the "Abort" portion of the script.

Agree. So I will put a special optional label, which will be accessed upon abort.

Stefan

> We're using mhttpd for calls that end up working better with asynchronous requests, and we've built up sort of a parallel, asynchronous library using javascript Promises.
> 
> The Promises (which are in the ES6 spec) have worked incredibly well for building well-behaved, sequential calls to mhttpd.  Personally, I also find their syntax much easier to wrap my
> head around, especially compared to callbacks.
> 
> I'd be happy to add these functions to midas.js if there's general interest. 

Why don't you post the functions here so that we can have a look? They don't have to be incorporated into mhttpd.js necessarily, but could live in a separate file, the people can choose which one to use.

Stefan

A small change in loading .odb files has been implemented. When you load an array from a .odb file, the indices in each line were not evaluated, only the complete array was loaded. In our experiment we need however to load only a few values, like some HV values for some channels but leaving the other values as they are. I changed slightly the code of db_paste() to correctly evaluate the index in each line of the .odb file. This way one can write for example following .odb file:

[/Equipment/HV/Variables]
Demand = FLOAT[256] :
[10] 100.1
[11] 100.2
[12] 100.3
[13] 100.4
[14] 100.5
[15] 100.6

then load it in odbedit via the "load" command, and then only change channels 10-15.

Stefan

> There were some complaints that the MIDAS sequencer was slow.  Specifically, the
> complaint was that even lines in the sequence that didn't do any (like COMMENT
> commands) tooks > 100ms to execute.  These slow sequencer steps could be a
> little annoying if a script had to change a large number of ODB variables before
> starting.
> 
> I tested this a little using a trivial sequence; note that I did all tests using
> mhttpd with mongoose enabled on a newer macbook pro.  I found that with the
> mongoose server each line in a sequencer script was taking ~100ms.  This is
> consistent with the loop in the main thread, which is only doing a cm_yield and
> a sleep:
> 
>    while (!_abort) {
>       status = ss_mutex_wait_for(request_mutex, 0);
>       status = cm_yield(0);
>       if (status == RPC_SHUTDOWN)
>          break;
>       sequencer();
>       status = ss_mutex_release(request_mutex);
>       ss_sleep(100);
>    }
> 
> I tested reducing the sleep to 20ms.  As expected, this made the sequencer more
> zippy, able to execute ~50 commands per second.
> 
> I tried to think what would be downsides to making this change.  I think that
> the main web communication should not be affected, because that communication is
> all handled by the separate mongoose thread.
> 
> I checked how much extra CPU was used if the sleep was reduced from 100ms to
> 20ms.  I found that when a sequence was not running the CPU increased from 0% to
> 0.2% with my change.  When a sequence was running the CPU increased from 0.8% to
> 4% with my change.  4% is a little high, though I'd say still reasonable.  I
> found that most of the CPU usage was occuring because every call to
> 'sequencer()' resulted in a call to db_set_record("/Sequencer/State"...).  I
> guess that making that call 50 times causes the somewhat heavy CPU usage.
> 
> I would argue that it would still be worth making that change, so that the
> sequencer can be more zippy.

The minimal time slice on most systems is 10 ms, and nothing prevents us from switching to
that. The original 100 ms was more for the fact that you can see the sequencer statements
executed one after the other (with the color bar). But this is more a "debugging" feature which 
we not really need. 

To do it "right" the sequencer would have to _return_ a sleep time. Like if it is in a wait loop (as
most of the time), the sleep time could be close to 1 second, to correctly update the wait
progress bar. If the sequencer executes ODB set statements, the wait time could be zero, so
thousands of statements can be executed in one second. The problem we will then have of course
that the sequencer will block the "request_mutex" almost always, which would prevent the
mongoose server from serving anything. So this should be carefully tested. It could be (on most OS)
that releasing the mutex by the main loop immediately switches to the mongoose thread, which would
make the web server still quite responsive, but I'm not sure about that. So as a first change making
the sleep time 10ms should be fine.

Stefan

> Hmm, yeah, I'm not sure about how to handle reducing the wait time to zero after ODB set commands.
> 
> But it does seem like it would be straight-forward to increase the sleep time for waits; I'll look into
> a clean way of doing that.

Let's see how your 10 ms work in real life. If we need variable wait times, I can implement this for your without much effort.

Stefan

> Hi,
> 
> Attempting to copy a subtree to a new location in the ODB using odbedit with "copy <src> <dest>" is 
> occasionally not copying the entire <src> subtree.
> 
> I am experiencing this issue consistently when trying to copy subtrees from the "/Equipment" ODB tree to 
> a new location.  The first 2-3 variables/directories of the <src> subtree will be copied to <dest> but the 
> full subtree will not be copied over.

I just tried myself and could successfully copy of even large trees in the /Equipment subtree. Need to reproduce the problem to fix 
it. Maybe close-to-full ODB?

Stefan

> I recently upgraded my macbook to MacOS 10.11.  The compilation of MIDAS failed after the upgrade, 
> complaining about  
> 
> gcc  -c -g -O2 -Wall <snip> src/mongoose.c
> src/mongoose.c:322:10: fatal error: 'openssl/ssl.h' file not found
> 
> It seems that MacOS has now fully removed openssl header files (they were deprecated for a while).  There 
> seems to be some notes on that here
> 
> http://lists.apple.com/archives/macnetworkprog/2015/Jun/msg00025.html
> 
> Konstantin suggested installing open-source builds of openssl using MacPorts.  I did that and MIDAS 
> compiled fine.  I documented the procedure here:
> 
> https://midas.triumf.ca/MidasWiki/index.php/Installation/Compilation_problems#MacOS_10.11_.28El_Capitan.2
> 9_openssl_errors

The MIDAS Wiki page points to https://guide.macports.org/  which covers OSX up to 10.9. Installers for 10.10 and the current 10.11 
(El Captain) can be found here: https://www.macports.org/install.php

Stefan

> Midas has a nice alarm system. I am wondering whether it is easily possible to
> get the Alarm/Warning banner also on top of custom pages?!

K.O. made nice JavaScript routines to access the alarm status. The new alarm page is completely 
made dynamically from JavaScript code (mhttpd does not supply any HTML code any more, only 
functions to obtain ODB values etc). Part of this new dynamic page must be some code to display 
the alarm status. You just need to copy this to your custom page. K.O. can tell you details.

Stefan

> Hi. Is it currently possible to automatically save the MIDAS ODB as a JSON file?
> I can do it manually in odbedit, but it looks like the only option for the
> automatic ODB save for each run is the standard .ODB format. Is there a way to
> change this?

You mean you like an ODB dump at the end of every run in JSON format?

Sure this can be implemented. But I wonder for what purpose you need that. Can you elaborate a 
bit, maybe it's a useful feature also other people should be aware of. I'm also thinking if we should 
offer a CouchDB interface, so ODB data is written directly to that database.

Stefan

> creating a frontend on MAC Sierra OSX 10
> include the midas.h file and when compiling with XCode I get an error based on
> this entry in the midas.h include
> 
> #if !defined(OS_IRIX) && !defined(OS_VMS) && !defined(OS_MSDOS) &&
> !defined(OS_UNIX) && !defined(OS_VXWORKS) && !defined(OS_WINNT)
> #error MIDAS cannot be used on this operating system
> #endif
> 
> 
> Perhaps I should not use Xcode?
> Perhaps I won't need Midas.h?
> 
> The MIDAS system is running on my MAC but I need to add a very simple front end
> for testing and I encounted this error.

If you compile with the included Makefile, you will see a 

-DOS_LINUX -DOS_DARWIN

flag which tells the compiler that we are on a mac. If you do this with XCode, you have to do it via "Build Settings" (see 
attached picture).

Stefan

> 
> > At Konstantin's suggestion, I committed the function I found for checking if a string was UTF-8 compatible to
> > odb.c.  The function is currently not used; I commented out a proposed use in db_create_key.  Experts can decide
> > if the code was good enough to use.
> 
> After more discussion, I have enabled the parts of the ODB code that check that key names are UTF-8 compliant. 
> 
> This check will show up in (at least) two ways:
> 
> 1) Attempts to create a new ODB variable if the ODB key is not UTF-8 compliant.  You will see error messages like
> 
> [fesimdaq,ERROR] [odb.c:572:db_validate_name,ERROR] Invalid name "Eur€" passed to db_create_key: UTF-8 incompatible
> string
> 
> 2) When a program first connects to the ODB, it runs a check to ensure that the ODB is valid.  This will now include
> a check that all key names are UTF-8 compliant. Any non-UTF8 compliant key names will be replaced by a string of the
> pointer to the key.  You will see error messages like:
> 
> [fesimdaq,ERROR] [odb.c:572:db_validate_name,ERROR] Invalid name "Eur€" passed to db_validate_key: UTF-8
> incompatible string
> [fesimdaq,ERROR] [odb.c:647:db_validate_key,ERROR] Warning: corrected key "/Equipment/SIMDAQ/Eur€": invalid name
> "Eur€" replaced with "0x7f74be63f970"
> 
> This behaviour (checking UTF-8 compatibility and automatically fixing ODB names) can be disabled by setting an
> environment variable
> 
> MIDAS_INVALID_STRING_IS_OK
> 
> It doesn't matter what the environment variable is set to; it just needs to be set.  Note also that this variable is
> only checked once, when a program starts.



I see you put some switches into the environment ("MIDAS_INVALID_STRING_IS_OK"). Do you think this is a good idea? Most variables are 
sitting in the ODB (/experiment/xxx), except those which cannot be in the ODB because we need it before we open the ODB, like MIDAS_DIR. 
Having them in the ODB has the advantage that everything is in one place, and we see a "list" of things we can change. From an empty 
environment it is not clear that such a thing like "MIDAS_INVALID_STRING_IS_OK" does exist, while if it would be an ODB key it would be 
obvious. Can I convince you to move this flag into the ODB?