we are migrating the dragon experiment from an old mac to a new mac studio and we ran into a problem 
where one equipment format was set to "fixed" instead of "midas". lots of confusion, mdump crash, 
analyzer crash, etc. (mdump fixes for this are already committed).

it made us think whether equipment format is still needed. in the old days we had choice of MIDAS and 
YBOS formats, but YBOS was removed years ago, and I was surprised that format FIXED was permitted at 
all.

I did a midas source code review, this is what I found:

- remnants of YBOS support in a few places, commit to remove them pending.
- FORMAT_ROOT is used in mlogger for automatic conversion of MIDAS banks to ROOT trees
- FORMAT_FIXED is used in a few slow control drivers in drivers/class, instead of creating MIDAS 
banks, they copy raw data directly into an event (there is no bank header and no way to identify such 
events automatically)
- lots of code to support different formats in mdump (mostly dead code)
- the rest of the code does not care or use this format stuff

Current proposal is to remove support for all formats except FORMAT_MIDAS (and FORMAT_ROOT in 
mlogger).

- defines of FORMAT_XXX will be removed from midas.h
- "Format" will be removed from ODB Equipment/Common
- "Format" will be removed from ODB Logger/Channel
- to maintain binary compatibility, we can keep the "Format" ODB entries, but they will be ignored.

List of slow control drivers that support FORMAT_FIXED:

daq00:midas$ grep FORMAT_FIXED drivers/class/*
drivers/class/cd_fdg.cxx:   if (fgd_info->format == FORMAT_FIXED) {
drivers/class/cd_ivc32.cxx:   if (hv_info->format == XFORMAT_FIXED) {
drivers/class/cd_rf.cxx:	if (rf_info->format == XFORMAT_FIXED) 
drivers/class/generic.cxx:   if (gen_info->format == XFORMAT_FIXED) {
drivers/class/hv.cxx:   if (hv_info->format == XFORMAT_FIXED) {
drivers/class/multi.cxx:   if (m_info->format == XFORMAT_FIXED) {
drivers/class/slowdev.cxx:   if (gen_info->format == XFORMAT_FIXED) {
daq00:midas$ 

K.O.

Hi Konstantin,

In the PSI muSR laboratory, we are running about 140 slow control devices across six instruments using Format FIXED.

Could you please wait a bit with removing support for it so that we can assess if/how this will affect us?

Many thanks,
Jonas

> we are migrating the dragon experiment from an old mac to a new mac studio and we ran into a problem 
> where one equipment format was set to "fixed" instead of "midas". lots of confusion, mdump crash, 
> analyzer crash, etc. (mdump fixes for this are already committed).
> 
> it made us think whether equipment format is still needed. in the old days we had choice of MIDAS and 
> YBOS formats, but YBOS was removed years ago, and I was surprised that format FIXED was permitted at 
> all.
> 
> I did a midas source code review, this is what I found:
> 
> - remnants of YBOS support in a few places, commit to remove them pending.
> - FORMAT_ROOT is used in mlogger for automatic conversion of MIDAS banks to ROOT trees
> - FORMAT_FIXED is used in a few slow control drivers in drivers/class, instead of creating MIDAS 
> banks, they copy raw data directly into an event (there is no bank header and no way to identify such 
> events automatically)
> - lots of code to support different formats in mdump (mostly dead code)
> - the rest of the code does not care or use this format stuff
> 
> Current proposal is to remove support for all formats except FORMAT_MIDAS (and FORMAT_ROOT in 
> mlogger).
> 
> - defines of FORMAT_XXX will be removed from midas.h
> - "Format" will be removed from ODB Equipment/Common
> - "Format" will be removed from ODB Logger/Channel
> - to maintain binary compatibility, we can keep the "Format" ODB entries, but they will be ignored.
> 
> List of slow control drivers that support FORMAT_FIXED:
> 
> daq00:midas$ grep FORMAT_FIXED drivers/class/*
> drivers/class/cd_fdg.cxx:   if (fgd_info->format == FORMAT_FIXED) {
> drivers/class/cd_ivc32.cxx:   if (hv_info->format == XFORMAT_FIXED) {
> drivers/class/cd_rf.cxx:	if (rf_info->format == XFORMAT_FIXED) 
> drivers/class/generic.cxx:   if (gen_info->format == XFORMAT_FIXED) {
> drivers/class/hv.cxx:   if (hv_info->format == XFORMAT_FIXED) {
> drivers/class/multi.cxx:   if (m_info->format == XFORMAT_FIXED) {
> drivers/class/slowdev.cxx:   if (gen_info->format == XFORMAT_FIXED) {
> daq00:midas$ 
> 
> K.O.

> In the PSI muSR laboratory, we are running about 140 slow control devices across six instruments using Format FIXED.
> Could you please wait a bit with removing support for it so that we can assess if/how this will affect us?

Roger. I believe as long as these data do not go into the MIDAS event data stream, you should not see any difference from 
changing "FIXED" to "MIDAS", I hope you can test it and report how it works for you. If you see that things change in ODB 
or in history, perhaps we can implement some kind of workaround to make the transition transparent.

K.O.

Around Wednesday Noon, there was a power outage at triumf (loss of ups power in the triumf 
computing center) and after rebooting ladd00, https/ssl access stopped working with a complaint 
about mismatching server name and ssl certificate name. This configuration used to work, so one of the 
system updated must have broke it. This problem is now fixed and access to midas elog is restored. 
K.O.

I reworked the midas cmake files:
- install via CMAKE_INSTALL_PREFIX should work correctly now:
- installed are bin, lib and include - everything needed to build against the midas library
- if built without CMAKE_INSTALL_PREFIX, a special mode "MIDAS_NO_INSTALL_INCLUDE_FILES" is activated, and the include path 
contains all the subdirectories need for compilation
- -I$MIDASSYS/include and -L$MIDASSYS/lib -lmidas work in both cases
- to "use" midas, I recommend: include($ENV{MIDASSYS}/lib/midas-targets.cmake)
- config files generated for find_package(midas) now have correct information (a manually constructed subset of information 
automatically exported by cmake's install(export))
- people who want to use "find_package(midas)" will have to contribute documentation on how to use it (explain the magic used to 
find the "right midas" in /usr/local/midas or in /midas or in ~/packages/midas or in ~/pacjages/new-midas) and contribute an 
example superproject that shows how to use it and that can be run from the bitpucket automatic build. (features that are not part 
of the automatic build we cannot insure against breakage).

On my side, here is an example of using include($ENV{MIDASSYS}/lib/midas-targets.cmake). I posted this before, it is used in 
midas/examples/experiment and I will ask ben to include it into the midas wiki documentation.

Below is the complete cmake file for building the alpha-g event bnuilder and main control frontend. When presented like this, I 
have to agree that cmake does provide positive value to the user. (the jury is still out whether it balances out against the 
negative value in the extra work to "just support find_package(midas) already!").

#
# CMakeLists.txt for alpha-g frontends
#

cmake_minimum_required(VERSION 3.12)
project(agdaq_frontends)

include($ENV{MIDASSYS}/lib/midas-targets.cmake)

add_compile_options("-O2")
add_compile_options("-g")
#add_compile_options("-std=c++11")
add_compile_options(-Wall -Wformat=2 -Wno-format-nonliteral -Wno-strict-aliasing -Wuninitialized -Wno-unused-function)
add_compile_options("-DTMFE_REV0")
add_compile_options("-DOS_LINUX")

add_executable(feevb feevb.cxx TsSync.cxx)
target_link_libraries(feevb midas)

add_executable(fectrl fectrl.cxx GrifComm.cxx EsperComm.cxx JsonTo.cxx KOtcp.cxx $ENV{MIDASSYS}/src/tmfe_rev0.cxx)
target_link_libraries(fectrl midas)

#end

MIDAS git tag midas-2025-01-a introduced an incompatible change to "include midas-targets.cmake". Instead of "midas" one now has to 
say "midas::midas", as updated below. K.O.

> 
> #
> # CMakeLists.txt for alpha-g frontends
> #
> 
> cmake_minimum_required(VERSION 3.12)
> project(agdaq_frontends)
> 
> include($ENV{MIDASSYS}/lib/midas-targets.cmake)
> 
> add_compile_options("-O2")
> add_compile_options("-g")
> #add_compile_options("-std=c++11")
> add_compile_options(-Wall -Wformat=2 -Wno-format-nonliteral -Wno-strict-aliasing -Wuninitialized -Wno-unused-function)
> add_compile_options("-DTMFE_REV0")
> add_compile_options("-DOS_LINUX")
> 
> add_executable(feevb feevb.cxx TsSync.cxx)
> target_link_libraries(feevb midas::midas)
> 
> add_executable(fectrl fectrl.cxx GrifComm.cxx EsperComm.cxx JsonTo.cxx KOtcp.cxx $ENV{MIDASSYS}/src/tmfe_rev0.cxx)
> target_link_libraries(fectrl midas::midas)
> 
> #end

I want to report a bug in the ROOT build process that might be relevant to the midas implementation. I had an annoying failure to build root 6.18 (current pro version) with a misleading error message about a fault in the root code. It turned out this was a cmake problem, and the error was from my cmake version being older than 3.14, which is quite recent. Took a bit of searching to find this.

I recommend when the cmake version is distributed that the instructions include the required cmake version. Developers are generally working well ahead of what is available in the older OS's.

Each CMakeLists.txt should specify which version of CMake it requires. The MIDAS CMakeLists.txt requires CMake 3.1 or later. 
We deliberately stayed away from fancy cutting edge CMake features in order to make midas easier to compile. On top of that,
midas is a much simpler package compared to root, so things are not so complicated.

Stefan

> Each CMakeLists.txt should specify which version of CMake it requires. The MIDAS CMakeLists.txt requires CMake 3.1 or later. 
> We deliberately stayed away from fancy cutting edge CMake features in order to make midas easier to compile. On top of that,
> midas is a much simpler package compared to root, so things are not so complicated.

The oldest cmake I actually used is 3.6.1 (on SL6), so I do not know if cmake versions between 3.1 and 3.6 actually work for us. Perhaps we should set 
the CMakefile requirement to 3.6.1 to match the oldest version we know works. If somebody has an older cmake, they have a choice of updating it or 
trying it as as and reporting success/failure to the midas forum here.

K.O.

I just tried CMake 3.1.0 and it worked with midas. So I believe all versions between 3.1.0 and 3.6.1 are ok.

Actually playing around with different versions I realized that 3.0.0 is also ok, so I changed the requirement of midas down to 3.0

Stefan

I accidentally discovered a problem with the alarm sounds played by midas.

The javascript code is very simple: var audio=new Audio("alarm.mp3"); audio.play();

In the past, this reliably played a sound.

More recently, I started seeing javascript log messages about "unhandled exception" from audio.play(). (for me, they often
interrupt my javascript debugger sessions, in a very annoying way).

Adding an exception handler to audio.play() was not effective: it turns out, these days, audio.play() returns a Promise
and one must handle the Promise rejection case. (also it turns out that in the rejection handler
one *must* clear audio.src to avoid problems with excessive memory and cpu use).

But why does audio.play() throw a rejected promise?!?

This is the error message provided: "play() failed because the user didn't interact with the document first. https://goo.gl/xX8pDD"
name: "NotAllowedError".

The link takes us to https://developers.google.com/web/updates/2017/09/autoplay-policy-changes. This web page seems to explain
everything neatly, but most of the information turns out to be wrong and unhelpful:

a) "Muted autoplay is always allowed" - wrong - I see audio.play() rejection from muted tabs
b) "User has interacted with the domain (click, tap, etc.)." - wrong - I see rejection in the javascript debugger even as I debug the web page
c) "user's Media Engagement Index threshold has been crossed" - broken - playback count and MEI is always 0 (zero) for midas - per chrome://media-engagement/
d) "user has added the site to their home screen on mobile or installed the PWA on desktop" - what ?!?
e) chrome://flags/#autoplay-policy does not exist (was removed)

There may be a way to start google-chrome with special flags to globally enable autoplay, but it seems
to be aimed at "kiosk" type applications, not for general use browsers:
https://stackoverflow.com/questions/57455849/chrome-autoplay-policy-chrome-76

The bottom line:

there is no way to ensure that the alarm sound will always play

K.O.

P.S. I am tracking this problem here:
https://bitbucket.org/tmidas/midas/issues/191/exception-on-audioplay

The document

https://docs.google.com/document/d/1_278v_plodvgtXSgnEJ0yjZJLg14Ogf-ekAFNymAJoU/edit

says that the "... playback must be at lest 7 seconds long" to signal significant playback.

I attached a modified alarm sound file with 3 seconds of sound followed by 5 seconds of silence. Try that one and let me know if this changes anything. If successful, I can modify the 
other sound files as well.

Stefan

I am updating the TRIUMF IRIS experiment to the latest version of MIDAS. I see following error messages in midas.log:

19:06:16.806 2024/05/16 [mhttpd,ERROR] [odb.cxx:6967:db_get_data_index,ERROR] index (29) exceeds array length (20) for key 
"/Equipment/Beamline/Variables/Measured"

19:06:15.806 2024/05/16 [mhttpd,ERROR] [odb.cxx:6967:db_get_data_index,ERROR] index (30) exceeds array length (20) for key 
"/Equipment/Beamline/Variables/Measured"

The errors are correct, there is only 20 elements in that array. The errors are coming every few seconds, they spam midas.log. How do I fix 
them? Where do they come from? There is no additional diagnostics or information to go from.

In the worst case, they come from some custom web page that reads wrong index variables from ODB. mhttpd currently provides no diagnostics to 
find out which web page could be causing this.

But maybe it's internal to MIDAS? I save odb to odb.json, "grep Measured odb.json" yields:
iris00:~> grep Measured odb.json 
        "Condition" : "/Equipment/Beamline/Variables/Measured[29] > 1e-5",
        "Condition" : "/Equipment/Beamline/Variables/Measured[30] < 0.5",

So wrong index errors is coming from evaluated alarms.
ODB "/Alarms/Alarm system active" is set to "no" (alarm system is disabled), the errors are coming.
ODB "/Alarms/Alarms/TP4/Active" is set to "no" (specific alarm is disabled), the errors are coming.
WTF? (and this is recentish borkage, old IRIS MIDAS had the same wrong index alarms and did not generate these errors).

Breakage:
- where is the error message "evaluated alarm XXX cannot be computed because YYY cannot be read from ODB!"
- disabled alarm should not be computed
- alarm system is disabled, alarms should not be computed

K.O.

P.S. I am filing bug reports here, I cannot be bothered with the 25-factor authentication to access bitbucket.

This is a common problem I also encountered in the past. You get a low-level ODB access error (could also be a read of a non-existing key) and you 
have no idea where this comes from. Could be the alarm system, a mhttpd web page, even some user code in a front-end over which the midas library 
has no control.

One option would be to add a complete stack dump to each of these error (ROOT does something like that), but I hear already people shouting "my 
midas.log is flooded with stack dumps!". So what I do in this case is I run a midas program in the debugger and set a breakpoint (in your case at 
odb.cxx:6967). If the breakpoint triggers, I inspect the stack and find out where this comes from.

Not that I print a stack dump for such error in the odbxx API. This goes to stdout, not the midas log, and it helped me in the past. Unfortunately 
stack dumps work only under linux (not MacOSX), and they do not contain all information a debugger can show you.

It is not true that alarm conditions are evaluated when the alarm system is off. I just tried and it works fine. The code is here:

alarm.cxx:591

      /* check global alarm flag */
      flag = TRUE;
      size = sizeof(flag);
      db_get_value(hDB, 0, "/Alarms/Alarm system active", &flag, &size, TID_BOOL, TRUE);
      if (!flag)
         return AL_SUCCESS;

so no idea why you see this error if you correctly st "Alarm system active" to false.

Stefan

> This is a common problem I also encountered in the past. You get a low-level ODB access error (could also be a read of a non-existing key) and you 
> have no idea where this comes from. Could be the alarm system, a mhttpd web page, even some user code in a front-end over which the midas library 
> has no control.

committed a partial fix, added an error message in alarm condition evaluation code to report alarm name and odb paths when we cannot get something from 
ODB. Now at least midas.log gives some clue that ODB errors are coming from alarms.

and the errors are actually coming from the alarms web page.

the alarms web page shows all the alarms even if alarms are disabled and it shows evaluated alarm conditions and current values even for alarms that 
are disabled.

I could change it to show "disabled" for disabled alarms, but I think it would not be an improvement,
right now it is quite convenient to see the alarm values for disabled/inactive alarms,
it is easy to see if they will immediately trip if I enable them. Hiding the values would make
them blind.

And I think I know what caused the original problem in IRIS experiment, I think the list of EPICS variables got truncated from 30 to 20 and EPICS 
values 29 and 30 used in the alarm conditions have become lost.

So the next step is to fix feepics to not truncate the list of variables (right now it is hardwired to 20 variables) and restore
the lost variable definition from a saved odb dump.

K.O.






> 
> One option would be to add a complete stack dump to each of these error (ROOT does something like that), but I hear already people shouting "my 
> midas.log is flooded with stack dumps!". So what I do in this case is I run a midas program in the debugger and set a breakpoint (in your case at 
> odb.cxx:6967). If the breakpoint triggers, I inspect the stack and find out where this comes from.
> 
> Not that I print a stack dump for such error in the odbxx API. This goes to stdout, not the midas log, and it helped me in the past. Unfortunately 
> stack dumps work only under linux (not MacOSX), and they do not contain all information a debugger can show you.
> 
> It is not true that alarm conditions are evaluated when the alarm system is off. I just tried and it works fine. The code is here:
> 
> alarm.cxx:591
> 
>       /* check global alarm flag */
>       flag = TRUE;
>       size = sizeof(flag);
>       db_get_value(hDB, 0, "/Alarms/Alarm system active", &flag, &size, TID_BOOL, TRUE);
>       if (!flag)
>          return AL_SUCCESS;
> 
> so no idea why you see this error if you correctly st "Alarm system active" to false.
> 
> Stefan

> 
> And I think I know what caused the original problem in IRIS experiment, I think the list of EPICS variables got truncated from 30 to 20 and EPICS 
> values 29 and 30 used in the alarm conditions have become lost.
> 
> So the next step is to fix feepics to not truncate the list of variables (right now it is hardwired to 20 variables) and restore
> the lost variable definition from a saved odb dump.

for the record, I restored the old ODB settings from feepics, my EPICS variables now have the correct size and the alarm now works correctly.

I also updated the example feepics to read the number of EPICS variables from ODB instead of always truncating them to 20 (IRIS MIDAS had a local change 
setting number of variables to 40).

I think I will make no more changes to the alarms, leave well enough alone.

K.O.

For everybody using EPICS: There is now a new system called MSetPoint (Midas Set Point) to control whole beamlines via EPICS.
It's under midas/msetpoint and the documentation is here:

   https://bitbucket.org/tmidas/midas/wiki/MSetPoint

It is basically an EPICS frontend and two custom pages. The special thing is that the EPICS elements are not hardcoded in
the frontend but come from the ODB. There is even an editor for the beamline elements (second custom page). By loading different
ODB settings, one can switch easily between completely different beamlines without having to recompile the frontend. The system
can be operated standalone (all other MIDAS pages do not appear), or as a custom page in a normal midas setup. At PSI, this
system is now used as the standard editor for our beamlines.

Attached and example screen.

Stefan

For everybody using EPICS: There is now a new system called MSetPoint (Midas Set Point) to control whole beamlines via EPICS.
It's under midas/msetpoint and the documentation is here:

   https://bitbucket.org/tmidas/midas/wiki/MSetPoint

It is basically an EPICS frontend and two custom pages. The special thing is that the EPICS elements are not hardcoded in
the frontend but come from the ODB. There is even an editor for the beamline elements (second custom page). By loading different
ODB settings, one can switch easily between completely different beamlines without having to recompile the frontend. The system
can be operated standalone (all other MIDAS pages do not appear), or as a custom page in a normal midas setup. At PSI, this
system is now used as the standard editor for our beamlines.

Attached and example screen.

Stefan

Minor Makefile changes:
- add "-m32" gcc flag to force 32-bit compilation on 64-bit Linux.
- do not link ybos.o into lazylogger and mdump.
K.O.

To improve on the existing HTTP "GET" based AJAX interface to MIDAS, I have been looking at other possible RPCs.

The JSON-RPC standard looks to be the most interesting and my experimental implementation now reached the point where other midas users are welcome to try it:

1. Please checkout the git branch "feature/json_rpc", build and run midas as per normal instructions.
2. Look at the MIDAS "Programs" page, you will "see double", the top is the normal midas programs page, the bottom is the new JSON-RPC based page that updates 
every 1 second.
3. Look at example.html page in examples/javascript1, run that experiment push the buttons.
4. Look at mhttpd.js functions mjsonrpc_xxx() to see how the RPC works.
5. Look at jsonrpc_user.cxx in .../src for examples of adding custom rpc functions to midas.

The main improvement is the use of HTTP POST request which allows unlimited data to be sent to midas (permitting proper implementation of ODB "paste" or "mset") 
and use of JSON encoding for all data, including error responses (removing previous ambiguity and poor documentability of some old AJAX functions).

Cross-origin AJAX requests continue to be fully supported (thanks to Bill Mills) - web pages loaded from local file or from some other web server can make AJAX 
requests into mhttpd. (this trivial functionality is normally prohibited by browser security).

My implementation follows these internet standards:

// https://tools.ietf.org/html/rfc4627 - JSON RFC
// http://www.jsonrpc.org/specification - specification of JSON-RPC 2.0
// http://www.simple-is-better.org/json-rpc/transport_http.html

With following variances:
- JSON encoding for NAN and Inf is Javascript-compatible strings "NaN", "Infinity" and "-Infinity"
- HTTP GET is not supported (not recommended by standard)
- batched JSON-RPC requests not supported yet

K.O.