Dear Yu Chen,

in my opinion, you can follow two strategies:

1) Follow the EQ_SLOW example from the distribution, and write some device driver for you hardware to control. There are dozens of experiments worldwide which use that scheme and it works ok for lots of 
different devices. By doing that, you get automatically things like write cache (only write a value to the actual device if it really has chande) and dead band (only write measured data to the ODB if it changes more 
than a certain value to suppress noise). Furthermore, you get events created automatically and history for all your measured variables. This scheme might look complicated on the first look, and it's quite old-
fashioned (no C++ yet) but it has proven stable over the last ~20 years or so. Maybe the biggest advantage of this system is that each device gets its own readout and control thread, so one slow device cannot 
block other devices. Once this has been introduced more than 10 years ago, we saw a big improvement in all experiments.

2) Do everything by yourself. This is the way you describe below. Of course you are free to do whatever you like, you will find "special" solutions for all your problems. But then you move away from the "standard 
scheme" and you loose all the benefits I described under 1) and of course you are on your own. If something does not work, it will be in your code and nobody from the community can help you.

So choose carefully between 1) and 2).

Best regards,
Stefan

> Dear MIDAS developers and colleagues,
> 
>     This is Yu CHEN of School of Physics, Sun Yat-sen University, China, working in the PandaX-III collaboration, an experiment under development to search the neutrinoless double 
> beta decay. We are working on the DAQ and slow control systems and would like to use Midas framework to communicate with the custom hardware systems, generally via Ethernet 
> interfaces. So currently we are focusing on the development of the FRONTEND program of Midas and have some questions and discussions on its ODB structure. Since I’m still not 
> experienced in the framework, it would be precious that you can provide some suggestions on the topic.
>     The current structure of the frontend ODB tree we have designed, together with our understanding on them, is as follows:
>       /Equipment/<frontend_name>/
>         ->  /Common/: Basic controls and left unchanged.
>         ->  /Variables/: (ODB records with MODE_READ) Monitored values that are AUTOMATICALLY updated from the bank data within each packed event. It is done by the update_odb() 
> function in mfe.cxx.
>         ->  /Statistics/: (ODB records with MODE_WRITE) Default status values that are AUTOMATICALLY updated by the calling of db_send_changed_records() within the mfe.cxx.
>         ->  /Settings/: All the user defined data can be located here. 
>             -> /stat/: (ODB records with MODE_WRITE) All the monitored values as well as program internal status. The update operation is done simultaneously when 
> db_send_changed_records() is called within the mfe.cxx.
>             -> /set/: (ODB records with MODE_READ) All the “Control” data used to configure the custom program and custom hardware.
> 
>     For our application, some of the our detector equipment outputs small amount of status and monitored data together with the event data, so we currently choose not to use EQ_SLOW 
> and 3-layer drivers for the readout. Our solution is to create two ODB sub-trees in the /Settings/ similar to what the device_driver does. However, this could introduce two 
> troubles:
>     1) For /Settings/stat/: To prevent the potential destroy on the hot-links of /Variables/ and /Statistics/ sub-trees, all our status and monitored data are stored separately in 
> the /Settings/stat/ sub-tree. Another consideration is that some monitored data are not directly from the raw data, so packaging them into the Bank for later showing in /Variables/ 
> could somehow lead to a complicated readout() function. However, this solution may be complicated for history loggings. I have find that the ANALYZER modules could provide some 
> processes for writing to the /Variables/ sub-tree, so I would like to know whether an analyzer should be used in this case.
>     2) For /Settings/set/: The “control” data (similar to the “demand” data in the EQ_SLOW equipment) are currently put in several /Settings/set/ sub-trees where each key in 
> them is hot-linked to a pre-defined hardware configuration function. However, some control operations are not related to a certain ODB key or related to several ODB keys (e.g. 
> configuration the Ethernet sockets), so the dispatcher function should be assigned to the whole sub-tree, which I think can slow the response speed of the software. What we are 
> currently using is to setup a dedicated “control key”, and then the input of different value means different operations (e.g. 1 means socket opening, 2 means sending the UDP 
> packets to the target hardware, et al.). This “control key” is also used to develop the buttons to be shown on the Status/Custom webpage. However, we would like to have your 
> suggestions or better solutions on that, considering the stability and fast response of the control.
> 
>     We are not sure whether the above understanding and troubles on the Midas framework are correct or they are just due to our limits on the knowledge of the framework, so we 
> really appreciate your knowledge and help for a better using on Midas. Thank you so much!

Dear MIDAS developers and colleagues,

    This is Yu CHEN of School of Physics, Sun Yat-sen University, China, working in the PandaX-III collaboration, an experiment under development to search the neutrinoless double 
beta decay. We are working on the DAQ and slow control systems and would like to use Midas framework to communicate with the custom hardware systems, generally via Ethernet 
interfaces. So currently we are focusing on the development of the FRONTEND program of Midas and have some questions and discussions on its ODB structure. Since I’m still not 
experienced in the framework, it would be precious that you can provide some suggestions on the topic.
    The current structure of the frontend ODB tree we have designed, together with our understanding on them, is as follows:
      /Equipment/<frontend_name>/
        ->  /Common/: Basic controls and left unchanged.
        ->  /Variables/: (ODB records with MODE_READ) Monitored values that are AUTOMATICALLY updated from the bank data within each packed event. It is done by the update_odb() 
function in mfe.cxx.
        ->  /Statistics/: (ODB records with MODE_WRITE) Default status values that are AUTOMATICALLY updated by the calling of db_send_changed_records() within the mfe.cxx.
        ->  /Settings/: All the user defined data can be located here. 
            -> /stat/: (ODB records with MODE_WRITE) All the monitored values as well as program internal status. The update operation is done simultaneously when 
db_send_changed_records() is called within the mfe.cxx.
            -> /set/: (ODB records with MODE_READ) All the “Control” data used to configure the custom program and custom hardware.

    For our application, some of the our detector equipment outputs small amount of status and monitored data together with the event data, so we currently choose not to use EQ_SLOW 
and 3-layer drivers for the readout. Our solution is to create two ODB sub-trees in the /Settings/ similar to what the device_driver does. However, this could introduce two 
troubles:
    1) For /Settings/stat/: To prevent the potential destroy on the hot-links of /Variables/ and /Statistics/ sub-trees, all our status and monitored data are stored separately in 
the /Settings/stat/ sub-tree. Another consideration is that some monitored data are not directly from the raw data, so packaging them into the Bank for later showing in /Variables/ 
could somehow lead to a complicated readout() function. However, this solution may be complicated for history loggings. I have find that the ANALYZER modules could provide some 
processes for writing to the /Variables/ sub-tree, so I would like to know whether an analyzer should be used in this case.
    2) For /Settings/set/: The “control” data (similar to the “demand” data in the EQ_SLOW equipment) are currently put in several /Settings/set/ sub-trees where each key in 
them is hot-linked to a pre-defined hardware configuration function. However, some control operations are not related to a certain ODB key or related to several ODB keys (e.g. 
configuration the Ethernet sockets), so the dispatcher function should be assigned to the whole sub-tree, which I think can slow the response speed of the software. What we are 
currently using is to setup a dedicated “control key”, and then the input of different value means different operations (e.g. 1 means socket opening, 2 means sending the UDP 
packets to the target hardware, et al.). This “control key” is also used to develop the buttons to be shown on the Status/Custom webpage. However, we would like to have your 
suggestions or better solutions on that, considering the stability and fast response of the control.

    We are not sure whether the above understanding and troubles on the Midas framework are correct or they are just due to our limits on the knowledge of the framework, so we 
really appreciate your knowledge and help for a better using on Midas. Thank you so much!

I received my new 2020 mac book air, so between Stefan and myself, MacOS support for 
MIDAS is assured for 5 more years at the least. K.O.

> > /Users/francesco/MIDAS/midas/src/system.cxx:3187:18: error: use of undeclared identifier 
> > 'CLOCK_REALTIME'
> >    clock_settime(CLOCK_REALTIME, &ltm);
> > 
> > Is it related to my (old) version of MacOS? Can I fix it somehow?

I think the "set clock" function is a holdover from embedded operating systems
that did not keep track of clock time, i.e. VxWorks, and similar. Here a midas program
will get the time from the mserver and set it on the local system. Poor man's ntp,
poor man's ntpd/chronyd.

We should check if this function is called by anything, and if nothing calls it, maybe remove it?

K.O.

> LOOP runs, 5
> ...
> ENDLOOP

Classical loop constructs usually include "break" (exit the loop) and "continue" (loop again!) 
constructs. I would say it is an unfortunate omission if they are not present in the midas sequencer.

But Stefan is right, of course, both commands are just funny names for "goto".

K.O.

<p>[quote=&quot;Pintaudi Giorgio&quot;]Dear MIDAS people, I need to borrow your 
wisdom for a bit. I am developing a piece of software that should read the history data 
stored in a [FONT=Times New Roman].midas[/FONT] file (MIDAS format) and integrate it 
into the WAGASCI data quality output. In other words, I need to read some temperature 
values stored in a [FONT=Times New Roman].midas[/FONT] file and compare them with 
the MPPC gains and check for temperature/gain dependence. I see three possibilities: 
[LIST] [*] write a custom parser in C++ using the instructions contained in the 
[URL=https://midas.triumf.ca/MidasWiki/index.php/Mhformat]Mhformat page[/URL]; [*] call 
the mhist program from within my application; [*] call the mhdump program from within my 
application; [/LIST] [B]Which solution do you think is the best?[/B] Because there is no 
need for raw performance, if possible, I would like to write my application in Python3 but 
C++ is also an option. [/quote]</p>

(Please write messages in plain text format, thank you)

The format of .hst midas history files is pretty simple and mhdump.cxx is an easy to read 
illustration on how to read it from basic principles (without going through the midas library, 
which can be somewhat complicated). The newer "FILE" format for history is even simpler 
to read because it is just fixed-record-size binary data prepended by a text header.

You can also use the mh2sql program to import history data into an sql database (mysql 
and sqlite should work) or to convert .hst files to "FILE" format files. This works well
for "archiving" history data, because the "FILE" format works better for looking at old data,
and for looking at data in "months" or "years" timescale.

Back to your question, you can certainly use "mhdump" as is, using a pipe (popen()), or 
you can package mhdump.cxx as a c++ class and use it in your application. If you go this 
route, your contribution of such a c++ class back to midas would be very welcome.

You can also use mhist, but the mhist code cannot be trivially packaged as a c++ class
to use in your application.

You can also suggest that we write an easier to use history utility, we are always open to 
suggested improvements.

Let us know how it works out for you. Good luck!

K.O.

• write a custom parser in C++ using the instructions contained in the Mhformat page;
  
• call the mhist program from within my application;
  
• call the mhdump program from within my application;
  

> > You can do that with the "GOTO" statement, jumping to the first line after the loop.
> > 
> > Here is a working example:
> > 
> > 
> > LOOP runs, 5
> >      WAIT Seconds 3
> >      IF $runs > 2
> >          GOTO 7
> >      ENDIF
> > ENDLOOP
> > MESSAGE "Finished", 1
> > 
> > Best,
> > Stefan
> 
> Hoi Stefan
> 
> Thanks for your answer. As I understand it, this has to be in the sequence script before 
> running. So, in the end, it is not different than just saying "LOOP runs, 2" and 
> therefore the number of runs has do be known in advance as well. Or is there an option to 
> change the script on runtime? What I would like, is to start a sequence with "LOOP runs, 
> infinite" and when I come back to the experiment after falling asleep being able to break 
> the loop after the next iteration, but still execute everything after ENDLOOP, i.e. the 
> MESSAGE statement in your example. Because if I do a "Stop after current run", this seems 
> not to happen. 
> 
> Best, Ivo

First, you have the sequencer button "Stop after current run", but that does of course ot
execute anything after ENDLOOP. 

Second, you can put anything in the IF statement. Like create a variable on the ODB like
/Experiment/Run parameters/Stop loop and set this to zero. Then put in your script:

...
ODBGET /Experiment/Run parameters/Stop loop, flag
IF $flag == 1
   GOTO 7
...

So once you want to stop the loop, set the flag in the ODB to one.

Best,
Stefan

> You can do that with the "GOTO" statement, jumping to the first line after the loop.
> 
> Here is a working example:
> 
> 
> LOOP runs, 5
>      WAIT Seconds 3
>      IF $runs > 2
>          GOTO 7
>      ENDIF
> ENDLOOP
> MESSAGE "Finished", 1
> 
> Best,
> Stefan

Hoi Stefan

Thanks for your answer. As I understand it, this has to be in the sequence script before 
running. So, in the end, it is not different than just saying "LOOP runs, 2" and 
therefore the number of runs has do be known in advance as well. Or is there an option to 
change the script on runtime? What I would like, is to start a sequence with "LOOP runs, 
infinite" and when I come back to the experiment after falling asleep being able to break 
the loop after the next iteration, but still execute everything after ENDLOOP, i.e. the 
MESSAGE statement in your example. Because if I do a "Stop after current run", this seems 
not to happen. 

Best, Ivo

> I am using the Midas sequencer to run subsequent measurements in a loop, without 
> knowing how many iterations in advance. Therefore, I am using the "infinity" 
> option. Since I have other commands after the loop, it would be nice to have the 
> possibility to break the loop, but let the sequencer then finish the rest of the 
> commands. 
> Cheers,
> Ivo

You can do that with the "GOTO" statement, jumping to the first line after the loop.

Here is a working example:


LOOP runs, 5
     WAIT Seconds 3
     IF $runs > 2
         GOTO 7
     ENDIF
ENDLOOP
MESSAGE "Finished", 1

Best,
Stefan

I am using the Midas sequencer to run subsequent measurements in a loop, without 
knowing how many iterations in advance. Therefore, I am using the "infinity" 
option. Since I have other commands after the loop, it would be nice to have the 
possibility to break the loop, but let the sequencer then finish the rest of the 
commands. 
Cheers,
Ivo

> Dear all,
>        I'm trying to compile MIDAS on MacOS 10.10 and I get this error:
> 
> /Users/francesco/MIDAS/midas/src/system.cxx:3187:18: error: use of undeclared identifier 
> 'CLOCK_REALTIME'
>    clock_settime(CLOCK_REALTIME, &ltm);
> 
> Is it related to my (old) version of MacOS? Can I fix it somehow?
> 
> Thank you,
>       Francesco

If I see this correctly, you need at least MacOSX 10.12. If you can't upgrade, you can just remove line 3187 
from system.cxx. This function is only used in an online environment, where you would run a frontend on your 
Mac, which you probably don't do. So removing it does not hurt you.

Stefan

Dear all,
       I'm trying to compile MIDAS on MacOS 10.10 and I get this error:

/Users/francesco/MIDAS/midas/src/system.cxx:3187:18: error: use of undeclared identifier 
'CLOCK_REALTIME'
   clock_settime(CLOCK_REALTIME, &ltm);

Is it related to my (old) version of MacOS? Can I fix it somehow?

Thank you,
      Francesco

We have to request of a 64-bit integer data type to be included in MIDAS banks.
Since 64-bit integers are on some systems "long" and on other systems "long long",
I decided to create the two new data types

TID_INT64
TID_UINT64

which follows more the standard C++ tradition:

https://en.cppreference.com/w/cpp/types/integer

To be consistent, I renamed the old types:

TID_BYTE       -> TID_UINT8
TID_SBYTE      -> TID_INT8
TID_WORD       -> TID_UINT16
TID_SHORT      -> TID_INT16
TID_DWORD      -> TID_UINT32
TID_INT        -> TID_INT32

I left the old definitions in midas.h, so old code will still compile fine and be binary
compatible. But if you write new code, the new types are recommended.

If you save the ODB in ASCII format, the new types are used as stings as well, like

[/Experiment]
ODB timeout = INT32 : 10000

but the old types are still understood when you load an old ODB file.

I hope I didn't break anything, please report if you have any issue.

Stefan

I had some quick look at the new mongoose code and didn't find anything I dislike. Did a quick test of the proxy which worked and is nice to have. 
Agree with all KO said about authentication.

So if there are no complaints, I would suggest that we move the summary of this thread into the official documentation.

Stefan