The document

https://docs.google.com/document/d/1_278v_plodvgtXSgnEJ0yjZJLg14Ogf-ekAFNymAJoU/edit

says that the "... playback must be at lest 7 seconds long" to signal significant playback.

I attached a modified alarm sound file with 3 seconds of sound followed by 5 seconds of silence. Try that one and let me know if this changes anything. If successful, I can modify the 
other sound files as well.

Stefan

I am updating the TRIUMF IRIS experiment to the latest version of MIDAS. I see following error messages in midas.log:

19:06:16.806 2024/05/16 [mhttpd,ERROR] [odb.cxx:6967:db_get_data_index,ERROR] index (29) exceeds array length (20) for key 
"/Equipment/Beamline/Variables/Measured"

19:06:15.806 2024/05/16 [mhttpd,ERROR] [odb.cxx:6967:db_get_data_index,ERROR] index (30) exceeds array length (20) for key 
"/Equipment/Beamline/Variables/Measured"

The errors are correct, there is only 20 elements in that array. The errors are coming every few seconds, they spam midas.log. How do I fix 
them? Where do they come from? There is no additional diagnostics or information to go from.

In the worst case, they come from some custom web page that reads wrong index variables from ODB. mhttpd currently provides no diagnostics to 
find out which web page could be causing this.

But maybe it's internal to MIDAS? I save odb to odb.json, "grep Measured odb.json" yields:
iris00:~> grep Measured odb.json 
        "Condition" : "/Equipment/Beamline/Variables/Measured[29] > 1e-5",
        "Condition" : "/Equipment/Beamline/Variables/Measured[30] < 0.5",

So wrong index errors is coming from evaluated alarms.
ODB "/Alarms/Alarm system active" is set to "no" (alarm system is disabled), the errors are coming.
ODB "/Alarms/Alarms/TP4/Active" is set to "no" (specific alarm is disabled), the errors are coming.
WTF? (and this is recentish borkage, old IRIS MIDAS had the same wrong index alarms and did not generate these errors).

Breakage:
- where is the error message "evaluated alarm XXX cannot be computed because YYY cannot be read from ODB!"
- disabled alarm should not be computed
- alarm system is disabled, alarms should not be computed

K.O.

P.S. I am filing bug reports here, I cannot be bothered with the 25-factor authentication to access bitbucket.

This is a common problem I also encountered in the past. You get a low-level ODB access error (could also be a read of a non-existing key) and you 
have no idea where this comes from. Could be the alarm system, a mhttpd web page, even some user code in a front-end over which the midas library 
has no control.

One option would be to add a complete stack dump to each of these error (ROOT does something like that), but I hear already people shouting "my 
midas.log is flooded with stack dumps!". So what I do in this case is I run a midas program in the debugger and set a breakpoint (in your case at 
odb.cxx:6967). If the breakpoint triggers, I inspect the stack and find out where this comes from.

Not that I print a stack dump for such error in the odbxx API. This goes to stdout, not the midas log, and it helped me in the past. Unfortunately 
stack dumps work only under linux (not MacOSX), and they do not contain all information a debugger can show you.

It is not true that alarm conditions are evaluated when the alarm system is off. I just tried and it works fine. The code is here:

alarm.cxx:591

      /* check global alarm flag */
      flag = TRUE;
      size = sizeof(flag);
      db_get_value(hDB, 0, "/Alarms/Alarm system active", &flag, &size, TID_BOOL, TRUE);
      if (!flag)
         return AL_SUCCESS;

so no idea why you see this error if you correctly st "Alarm system active" to false.

Stefan

> This is a common problem I also encountered in the past. You get a low-level ODB access error (could also be a read of a non-existing key) and you 
> have no idea where this comes from. Could be the alarm system, a mhttpd web page, even some user code in a front-end over which the midas library 
> has no control.

committed a partial fix, added an error message in alarm condition evaluation code to report alarm name and odb paths when we cannot get something from 
ODB. Now at least midas.log gives some clue that ODB errors are coming from alarms.

and the errors are actually coming from the alarms web page.

the alarms web page shows all the alarms even if alarms are disabled and it shows evaluated alarm conditions and current values even for alarms that 
are disabled.

I could change it to show "disabled" for disabled alarms, but I think it would not be an improvement,
right now it is quite convenient to see the alarm values for disabled/inactive alarms,
it is easy to see if they will immediately trip if I enable them. Hiding the values would make
them blind.

And I think I know what caused the original problem in IRIS experiment, I think the list of EPICS variables got truncated from 30 to 20 and EPICS 
values 29 and 30 used in the alarm conditions have become lost.

So the next step is to fix feepics to not truncate the list of variables (right now it is hardwired to 20 variables) and restore
the lost variable definition from a saved odb dump.

K.O.






> 
> One option would be to add a complete stack dump to each of these error (ROOT does something like that), but I hear already people shouting "my 
> midas.log is flooded with stack dumps!". So what I do in this case is I run a midas program in the debugger and set a breakpoint (in your case at 
> odb.cxx:6967). If the breakpoint triggers, I inspect the stack and find out where this comes from.
> 
> Not that I print a stack dump for such error in the odbxx API. This goes to stdout, not the midas log, and it helped me in the past. Unfortunately 
> stack dumps work only under linux (not MacOSX), and they do not contain all information a debugger can show you.
> 
> It is not true that alarm conditions are evaluated when the alarm system is off. I just tried and it works fine. The code is here:
> 
> alarm.cxx:591
> 
>       /* check global alarm flag */
>       flag = TRUE;
>       size = sizeof(flag);
>       db_get_value(hDB, 0, "/Alarms/Alarm system active", &flag, &size, TID_BOOL, TRUE);
>       if (!flag)
>          return AL_SUCCESS;
> 
> so no idea why you see this error if you correctly st "Alarm system active" to false.
> 
> Stefan

> 
> And I think I know what caused the original problem in IRIS experiment, I think the list of EPICS variables got truncated from 30 to 20 and EPICS 
> values 29 and 30 used in the alarm conditions have become lost.
> 
> So the next step is to fix feepics to not truncate the list of variables (right now it is hardwired to 20 variables) and restore
> the lost variable definition from a saved odb dump.

for the record, I restored the old ODB settings from feepics, my EPICS variables now have the correct size and the alarm now works correctly.

I also updated the example feepics to read the number of EPICS variables from ODB instead of always truncating them to 20 (IRIS MIDAS had a local change 
setting number of variables to 40).

I think I will make no more changes to the alarms, leave well enough alone.

K.O.

For everybody using EPICS: There is now a new system called MSetPoint (Midas Set Point) to control whole beamlines via EPICS.
It's under midas/msetpoint and the documentation is here:

   https://bitbucket.org/tmidas/midas/wiki/MSetPoint

It is basically an EPICS frontend and two custom pages. The special thing is that the EPICS elements are not hardcoded in
the frontend but come from the ODB. There is even an editor for the beamline elements (second custom page). By loading different
ODB settings, one can switch easily between completely different beamlines without having to recompile the frontend. The system
can be operated standalone (all other MIDAS pages do not appear), or as a custom page in a normal midas setup. At PSI, this
system is now used as the standard editor for our beamlines.

Attached and example screen.

Stefan

For everybody using EPICS: There is now a new system called MSetPoint (Midas Set Point) to control whole beamlines via EPICS.
It's under midas/msetpoint and the documentation is here:

   https://bitbucket.org/tmidas/midas/wiki/MSetPoint

It is basically an EPICS frontend and two custom pages. The special thing is that the EPICS elements are not hardcoded in
the frontend but come from the ODB. There is even an editor for the beamline elements (second custom page). By loading different
ODB settings, one can switch easily between completely different beamlines without having to recompile the frontend. The system
can be operated standalone (all other MIDAS pages do not appear), or as a custom page in a normal midas setup. At PSI, this
system is now used as the standard editor for our beamlines.

Attached and example screen.

Stefan

Minor Makefile changes:
- add "-m32" gcc flag to force 32-bit compilation on 64-bit Linux.
- do not link ybos.o into lazylogger and mdump.
K.O.

To improve on the existing HTTP "GET" based AJAX interface to MIDAS, I have been looking at other possible RPCs.

The JSON-RPC standard looks to be the most interesting and my experimental implementation now reached the point where other midas users are welcome to try it:

1. Please checkout the git branch "feature/json_rpc", build and run midas as per normal instructions.
2. Look at the MIDAS "Programs" page, you will "see double", the top is the normal midas programs page, the bottom is the new JSON-RPC based page that updates 
every 1 second.
3. Look at example.html page in examples/javascript1, run that experiment push the buttons.
4. Look at mhttpd.js functions mjsonrpc_xxx() to see how the RPC works.
5. Look at jsonrpc_user.cxx in .../src for examples of adding custom rpc functions to midas.

The main improvement is the use of HTTP POST request which allows unlimited data to be sent to midas (permitting proper implementation of ODB "paste" or "mset") 
and use of JSON encoding for all data, including error responses (removing previous ambiguity and poor documentability of some old AJAX functions).

Cross-origin AJAX requests continue to be fully supported (thanks to Bill Mills) - web pages loaded from local file or from some other web server can make AJAX 
requests into mhttpd. (this trivial functionality is normally prohibited by browser security).

My implementation follows these internet standards:

// https://tools.ietf.org/html/rfc4627 - JSON RFC
// http://www.jsonrpc.org/specification - specification of JSON-RPC 2.0
// http://www.simple-is-better.org/json-rpc/transport_http.html

With following variances:
- JSON encoding for NAN and Inf is Javascript-compatible strings "NaN", "Infinity" and "-Infinity"
- HTTP GET is not supported (not recommended by standard)
- batched JSON-RPC requests not supported yet

K.O.

> 
> My implementation follows these internet standards:
> 
> // https://tools.ietf.org/html/rfc4627 - JSON RFC
> // http://www.jsonrpc.org/specification - specification of JSON-RPC 2.0
> // http://www.simple-is-better.org/json-rpc/transport_http.html
> 
> With following variances:
> - JSON encoding for NAN and Inf is Javascript-compatible strings "NaN", "Infinity" and "-Infinity"
> - HTTP GET is not supported (not recommended by standard)
> - batched JSON-RPC requests not supported yet
> 

The last missing piece is now committed - the JSON-RPC interface is now self-documenting via an automatically
generated JSON Schema that lists all RPC methods with their parameters and return values. This documentation
schema is created from simple to use documentation code in each rpc server function, see mjsonrpc.cxx.

To kick the tires, checkout the feature/json_rpc branch, build mhttpd, setup the examples/javascript1 experiment,
run mhttpd in the terminal, from the "status" page, go to the "example" custom page, press "push me" in the mjsonrpc_db_get_values() section,
mhttpd will print the schema file on the terminal. Use any json schema visualization tool to look at it. In the future I hope
to link this schema to the midas "help" page.

The impatient can go directly here: (it is safe to press all buttons) (elog is making a dog's breakfast of my url)
http://ladd00.triumf.ca/~olchansk/test/docson/#../test.json
docson is here:
https://github.com/lbovet/docson

For more informantion about JSON Schema stuff, go here:

https://tools.ietf.org/html/draft-zyp-json-schema-04
http://spacetelescope.github.io/understanding-json-schema/
http://json-schema.org/

JSON Schema examples:
http://json-schema.org/examples.html
http://json-schema.org/example1.html

JSON Schema visualization: (schema file has to have a .json extension)
https://github.com/lbovet/docson

(there is also an interesting discussion on why there is no RFC for JSON schema - the draft expired several years ago)

K.O.

> > 
> > JSON-RPC My implementation follows these internet standards:
> > 
> > // https://tools.ietf.org/html/rfc4627 - JSON RFC
> > // http://www.jsonrpc.org/specification - specification of JSON-RPC 2.0
> > // http://www.simple-is-better.org/json-rpc/transport_http.html
>
> JSON Schema
>
> https://github.com/lbovet/docson
> http://spacetelescope.github.io/understanding-json-schema/
> http://json-schema.org/
> 

Without figuring out how to run docson one can see the JSON-RPC Schema linked from the mhttpd "Help" page
follow link "JSON RPC schema" -> "text format" you will see it pretty printed like this:

---------------------------------------------------------------------
Autogenerated schema for all MIDAS JSON-RPC methods
---------------------------------------------------------------------
cm_exist      | calls MIDAS cm_exist() to check if given MIDAS program is running
              | -----------------------------------------------------
              | params | name           | string         | name of the program, corresponding to ODB /Programs/name
              |        | unique?        | bool           | bUnique argument to cm_exist()
              | -----------------------------------------------------
              | result | status         | integer        | return status of cm_exist()
---------------------------------------------------------------------
cm_shutdown   | calls MIDAS cm_shutdown() to stop given MIDAS program
              | -----------------------------------------------------
              | params | name           | string         | name of the program, corresponding to ODB /Programs/name
              |        | unique?        | bool           | bUnique argument to cm_shutdown()
              | -----------------------------------------------------
              | result | status         | integer        | return status of cm_shutdown()
---------------------------------------------------------------------
db_copy       | get copies of given ODB subtrees in the "save" json encoding
              | -----------------------------------------------------
              | params | paths[]        | array of ODB subtree paths, see note on array indices
              |        |                | array of       | string 
              | -----------------------------------------------------
              | result | data[]         | copy of ODB data for each path
              |        |                | array of       | object 
              |        | status[]       | return status of db_copy_json() for each path
              |        |                | array of       | integer
              |        | last_written[] | last_written value of the ODB subtree for each path
              |        |                | array of       | number 
---------------------------------------------------------------------
db_create     | get copies of given ODB subtrees in the "save" json encoding
              | -----------------------------------------------------
              | params | array of ODB paths to be created
              |        | array of       | arguments to db_create() and db_resize()
              |        |                | path           | string  | ODB path
              |        |                | type           | integer | MIDAS TID_xxx type
              |        |                | array_length?  | integer | optional array length, default is 1
              |        |                | string_length? | integer | for TID_STRING, optional string length, default is NAME_LENGTH
              | -----------------------------------------------------
              | result | status[]       | return status of db_create() for each path
              |        |                | array of       | integer
---------------------------------------------------------------------
db_get_values | get values of ODB data from given subtrees
              | -----------------------------------------------------
              | params | paths[]        | array of ODB subtree paths, see note on array indices
              |        |                | array of       | string 
              | -----------------------------------------------------
              | result | data[]         | values of ODB data for each path, all key names are in lower case, all symlinks are followed
              |        |                | array of       | any    
              |        | status[]       | return status of db_copy_json() for each path
              |        |                | array of       | integer
              |        | last_written[] | last_written value of the ODB subtree for each path
              |        |                | array of       | number 
---------------------------------------------------------------------
db_paste      | write data into ODB
              | -----------------------------------------------------
              | params | paths[]        | array of ODB subtree paths, see note on array indices
              |        |                | array of       | string 
              |        | values[]       | data to be written using db_paste_json()
              |        |                | array of       | any    
              | -----------------------------------------------------
              | result | status[]       | return status of db_paste_json() for each path
              |        |                | array of       | integer
---------------------------------------------------------------------
get_debug     | get current value of mjsonrpc_debug
              | -----------------------------------------------------
              | params | any            | there are no input parameters
              | -----------------------------------------------------
              | result | integer        | current value of mjsonrpc_debug
---------------------------------------------------------------------
get_schema    | Get the MIDAS JSON-RPC schema JSON object
              | -----------------------------------------------------
              | params | any            | there are no input parameters
              | -----------------------------------------------------
              | result | object         | returns the MIDAS JSON-RPC schema JSON object
---------------------------------------------------------------------
null          | RPC method always returns null
              | -----------------------------------------------------
              | params | any            | method parameters are ignored
              | -----------------------------------------------------
              | result | null           | always returns null
---------------------------------------------------------------------
set_debug     | set new value of mjsonrpc_debug
              | -----------------------------------------------------
              | params | integer        | new value of mjsonrpc_debug
              | -----------------------------------------------------
              | result | integer        | new value of mjsonrpc_debug
---------------------------------------------------------------------
start_program | start MIDAS program defined in ODB /Programs/name
              | -----------------------------------------------------
              | params | name           | string         | name of the program, corresponding to ODB /Programs/name
              | -----------------------------------------------------
              | result | status         | integer        | return status of ss_system()
---------------------------------------------------------------------
user_example1 | any   
---------------------------------------------------------------------
user_example2 | any   
---------------------------------------------------------------------
user_example3 | any   

While investigating the mhttpd password error with the MacOS Safari browser, I
found that it was caused by an strcpy() buffer overflow. With Stefan's blessing,
I now converted most uses of strcpy() and strcat() to strlcpy() and strlcat().

This fixes the Safari password problem (it was memory corruption in mhttpd).

While validating these changes, I also found an incorrect use of sizeof() in the
mhttpd history code for plotting run markers. I fixed that as well.

P.S. The remaining strcpy() calls look safe wrt buffer overflows. There are no
strcat() calls left. But there is still a large number of unsafe-looking
sprintf() uses.

K.O.

For some time now, boolean arrays did not work correctly in "/experiment/edit on start". This is now fixed 
in rev 3680. K.O.

From "history" pages, the "create elog" button stopped working - it takes us to the elog entry form, but 
then, the "submit" button does not create any elog entries, instead dumps us into an invalid history 
display. This is using the internal elog.

This change in mhttpd.c::show_elog_new() makes it work again:

-       ("<body><form method=\"POST\" action=\"./\" enctype=\"multipart/form-data\">\n");
+       ("<body><form method=\"POST\" action=\"/EL/\" enctype=\"multipart/form-data\">\n");

Problem and fix confirmed with Linux/firefox and MacOS/firefox and Safari.

K.O.

> From "history" pages, the "create elog" button stopped working - it takes us to the elog entry form, but 
> then, the "submit" button does not create any elog entries, instead dumps us into an invalid history 
> display. This is using the internal elog.
> 
> This change in mhttpd.c::show_elog_new() makes it work again:
> -       ("<body><form method=\"POST\" action=\"./\" enctype=\"multipart/form-data\">\n");
> +       ("<body><form method=\"POST\" action=\"/EL/\" enctype=\"multipart/form-data\">\n");

This was a problem with relative URLs and it is now fixed. Svn revision 4131, fixes: delete elog, make elog from odb, make elog from history.

K.O.

This commit adds the elog message text to the outgoing email message. This
functionality has been requested a logn time ago, but I guess nobody got
around to implement it, until now. I also added assert() traps for the most
common array overruns in the Elog code.

Here is the cvs diff:

Index: src/mhttpd.c
===================================================================
RCS file: /usr/local/cvsroot/midas/src/mhttpd.c,v
retrieving revision 1.252
diff -r1.252 mhttpd.c
768a769
> #include <assert.h>
3740c3741
< char   mail_to[256], mail_from[256], mail_text[256], mail_list[256],
---
> char   mail_to[256], mail_from[256], mail_text[10000], mail_list[256],
3921a3923,3925
>         // zero out the array. needed because later strncat() does not
always add the trailing '\0'
>         memset(mail_text,0,sizeof(mail_text));
> 
3931a3936,3945
> 
>         assert(strlen(mail_text) + 100 < sizeof(mail_text)); // bomb out
on array overrun.
> 
>         strcat(mail_text+strlen(mail_text),"\n");
>         // this strncat() depends on the mail_text array being zeroed out:
>         // strncat() does not always add the trailing '\0'
>        
strncat(mail_text+strlen(mail_text),getparam("text"),sizeof(mail_text)-strlen(mail_text)-50);
>         strcat(mail_text+strlen(mail_text),"\n");
> 
>         assert(strlen(mail_text) < sizeof(mail_text)); // bomb out on
array overrun.
Index: src/midas.c
===================================================================
RCS file: /usr/local/cvsroot/midas/src/midas.c,v
retrieving revision 1.192
diff -r1.192 midas.c
604a605
> #include <assert.h>
16267a16269,16270
> 
>   assert(strlen(message) < sizeof(message)); // bomb out on array overrun.

K.O.

> around to implement it, until now. I also added assert() traps for the most
> common array overruns in the Elog code.

In addition to the assert() one should use strlcat() and strlcpy() all over 
the code to avoid buffer overruns. The ELOG standalone code does that already 
properly.

- Stefan

> > around to implement it, until now. I also added assert() traps for the most
> > common array overruns in the Elog code.
> 
> In addition to the assert() one should use strlcat() and strlcpy() all over 
> the code to avoid buffer overruns. The ELOG standalone code does that already 
> properly.
> 
> - Stefan

Yes, the original authors should have used strlcat(). Now that I uncovered this source of mhttpd 
memory corruption, maybe some volunteer will fix it up properly.

K.O.

> > > around to implement it, until now. I also added assert() traps for the 
most
> > > common array overruns in the Elog code.
> > 
> > In addition to the assert() one should use strlcat() and strlcpy() all 
over 
> > the code to avoid buffer overruns. The ELOG standalone code does that 
already 
> > properly.
> > 
> > - Stefan
> 
> Yes, the original authors should have used strlcat(). Now that I uncovered 
this source of mhttpd 
> memory corruption, maybe some volunteer will fix it up properly.
> 
> K.O.

I am the original author and will fix all that once I merged mhttpd and elog. 
Due to my current task list, this will happen probably in November.

- Stefan

   A couple of experiments at triumf wanted certain important odb variables
displayed on their status page.  (There was already the possibility to show the
run comment)
   A new folder "/Experiment/Status Items" was created containing links to the
variables of interest, these items are show on the status page, under the run
comment (if any), in 3 columns.

the code from mhttpd.c:show_status_page()
between
   /*---- run comment ----*/
and
   /*---- Equipment list ----*/
is attached