> > - multiple subdetectors are taking test data during early commissioning 
> > - a postgres db is a single sorce of run numbers.
> > - test runs taken by different subsystems are assigned different [unique] run numbers and 
> >   the data taken by the subsystem are identified not by the run number/dataset name , but 
> >   by the run type, different for different susbsystems.
> 
> For that purpose I would not "mis-use" run numbers. Run number are meant to be incremented 
> sequentially, like if you have a time-stamp in seconds since 1.1.1970 (Unix time). Intead, I 
> would add additional attributes under /Experiment/Run Parameters like "Subsystem type", "Run 
> mode (production/commissioning)" etc. You have much more freedom in choosing any number of 
> attributes there. Then, send this attributes to your postgred db via "/Logger/Runlog/SQL/Links 
> BOR". Then you can query your database to give you all runs of a certain subtype or mode.
> 
> See https://daq00.triumf.ca/MidasWiki/index.php/Logging_to_a_mySQL_database
> 
> Stefan

Ben, Stefan - thanks much for your suggestions!(and apologies for the thanks being delayed) 

Stefan, I don't think we're talking 'mis-use' - rather different subdetectors being commisisoned 
at different locations, on an uncorrelated schedule, using independent run control (RC) instances. 
At this point in time, we can't use a common RC instance. 
The collected data, however, are written back into a common storage, and we need to avoid two 
subdetectors  using the same run number. As all RC instances can connect to the same database and request a 
run number from there, an external DB serving run numbers to multiple clients looks as a reasonable solution, 
which provides unique run numbers for everyone. Of course, the run number gets incremented (although on the DB 
server side), and of course different susbystems are assigned different subsystem types. 

So, in essense, it is about _where_ the run number is incremented - the RC vs the DB. 
If there were a good strategy to implement a DB-based solution that w/o violating 
first principles of Midas:), I'd be happy to contribute. It looks like a legitimate use case.

-- let me know, regards, Pasha 

I have one more question, though I understand that it could be somewhat border-line.

The MIDAS state machine doesn't seem to have a state in between 'initialized' and 
'running'.
In a larger detectors with multiple subsystems, the DAQ systems often have one more state:
after ending a previous run and before starting a new one from the 'stopped' state, 
one needs to make sure that all subdetectors are ready, or 'configured' for the new run. 
So theat calls for a 'configure' step during which the detector (all subsystems in 
parallel, to save the time) transitions from 'initialized'/'stopped' to 'configured' state, 
from which it transitions to the 'running' state.

If one of the subdetectors fails to get configured, it could be excluded from the run 
configuration and another attempt to reconfigure the system could be made without 
starting a new run. Or an attempt could be made to troubleshoot and configure the 
failed subsytem individually , with the rest subsystems waiting in a 'configured' state.

How does the logic of configuring the detector for the new run is implemented in MIDAS? 
- it is a fairly common operational procedure, so I'm sure there should be a way 
of doing that.

-- thanks again, Pasha

> Stefan, I don't think we're talking 'mis-use' - rather different subdetectors being commisisoned 
> at different locations, on an uncorrelated schedule, using independent run control (RC) instances. 
> At this point in time, we can't use a common RC instance. 
> The collected data, however, are written back into a common storage, and we need to avoid two 
> subdetectors  using the same run number. As all RC instances can connect to the same database and request a 
> run number from there, an external DB serving run numbers to multiple clients looks as a reasonable solution, 
> which provides unique run numbers for everyone. Of course, the run number gets incremented (although on the DB 
> server side), and of course different susbystems are assigned different subsystem types. 
> 
> So, in essense, it is about _where_ the run number is incremented - the RC vs the DB. 
> If there were a good strategy to implement a DB-based solution that w/o violating 
> first principles of Midas:), I'd be happy to contribute. It looks like a legitimate use case.

Ok, maybe attitude comes from the fact that I never used such a scheme in the last 30 years with midas.

If you go in this direction, there is an alternative to what Ben wrote: Use the sequencer to start a run.
The sequencer script can obtain a new run number from a central instance (e.g. by calling a shell script 
like 'curl ...' to obtain the new run number, then put it into /Runinfo/Run number as Ben wrote. This has
the advantage that the run is _started_ already with the correct number, so the history system is fine.

The script can then wait for n events, then stop the run etc. A sequencer script will also be necessary if
you want to configure your electronics (see next answer...)

Stefan

> The MIDAS state machine doesn't seem to have a state in between 'initialized' and 
> 'running'.
> In a larger detectors with multiple subsystems, the DAQ systems often have one more state:
> after ending a previous run and before starting a new one from the 'stopped' state, 
> one needs to make sure that all subdetectors are ready, or 'configured' for the new run. 
> So theat calls for a 'configure' step during which the detector (all subsystems in 
> parallel, to save the time) transitions from 'initialized'/'stopped' to 'configured' state, 
> from which it transitions to the 'running' state.
> 
> If one of the subdetectors fails to get configured, it could be excluded from the run 
> configuration and another attempt to reconfigure the system could be made without 
> starting a new run. Or an attempt could be made to troubleshoot and configure the 
> failed subsytem individually , with the rest subsystems waiting in a 'configured' state.
> 
> How does the logic of configuring the detector for the new run is implemented in MIDAS? 
> - it is a fairly common operational procedure, so I'm sure there should be a way 
> of doing that.

We have a similar requirement in our MEG experiment. Configuring your subdetectors can
be quite complex and therefore it's almost impossible to define a 'configure' step in
the run transition system to accommodate all corner cases.

Instead of a new state, we do everything through the sequencer:

- To start a run, we start a special sequencer script. We have different scripts for
calibration runs, data runs, special runs.

- When the user starts the script, they are asked for certain parameters, like number
of events, number of runs to take, how to configure the subdetectors, which subdetectors
to read out etc.

- The script then configures the whole experiment by setting everything in the ODB for
each equipment.

- The frontends connected to their equipment get a hotlink from the ODB and start the
configuration of the trigger etc. based on the parameters from the ODB

- The progress of the configuration is indicated by the frontend by writing back the
progress (like 0...100) into the ODB

- The script now waits for the progress to reach 100. It shows the current progress
on the sequencer page, so you see exactly where we are.

- If we have several subdetectors, each of them can publish a progress, and the script
can wait for an AND of all progress, or exclude one if it fails etc. Any logic is
possible there.

- Once all progresses are at 100, the run is finally started.

- If the mechanics of configuration become more elaborate, one can 'hide' it in
sub-modules of the script.

This scheme allows us to configure very different run modes, we use it in MEG since
many years (about 0.5M runs) and it works very nice.

Attached is our main script to start a full data run. You don't have to understand
all details, but it can give you a glimpse of what it's possible with the sequencer.
The function "ApplySettings" is the one waiting for the configuration flag in the ODB
(we simply use a boolean flag there). The code is:

SUBROUTINE ApplySettings
  ODBSET "/Equipment/Trigger/Settings/Reload all", y, 1
  WAIT seconds, 2
  WAIT ODBValue, "/Equipment/Trigger/Variables/Config busy", ==, y
ENDSUBROUTINE

Best,
Stefan

> 
> If you go in this direction, there is an alternative to what Ben wrote: Use the sequencer to start a run.
> The sequencer script can obtain a new run number from a central instance (e.g. by calling a shell script 
> like 'curl ...' to obtain the new run number, then put it into /Runinfo/Run number as Ben wrote. This has
> the advantage that the run is _started_ already with the correct number, so the history system is fine.
> 

Hi Stefan, this sounds like a perfect solution - thanks! - and leads to another, more technical, question:

- how does one communicate with an external shell script from MSL ? I looked at the MIDAS Sequencer page 
  
  https://daq00.triumf.ca/MidasWiki/index.php/Sequencer

  and didn't find an immediately obvious candidate among the MSL commands. 
  The closest seems to be 

  'SCRIPT script [, a, b, c, ...]' 

  but I couldn't easily figure how to propagate the output of the script back to MIDAS. 
  Let say, the script creates an ASCII file with the next run number. What is the easiest 
  way to import the run number into ODB? - Should an external script spawn a [short-lived] 
  MIDAS client ?  - That would work, but I'm almost sure there is a more straightforward solution. 

  Of course, the assumption that the 'SCRIPT' command provides the solution could be wrong.

-- thanks again, regards, Pasha

> - To start a run, we start a special sequencer script. We have different scripts for
> calibration runs, data runs, special runs.
> 
a sequencer-based way sounds like a very good solution, which provides all needed functionality 
and even more flexibility than a state machine transition. Will give it a try.

-- thanks again, regards, Pasha

> - how does one communicate with an external shell script from MSL ? 

trying to answer my own question, as I didn't find a clear answer in the forum archive :

1. one could have a MSL script with a 'SCRIPT ./myscript.sh' command in it - 
   that would run a shell script named 'myscript.sh'
   [that was not obvious from the documentation on MIDAS wiki, and adding a couple of clarifying 
    sentences there would go long ways]

2. if a script produces an ascii file with a known name, for example, 'a.odb', with the following two lines:

--------------------------------------- a.odb
[/Runinfo]
Run number  = INT32 : 105
--------------------------------------- end a.odb

one can use the 'odbload' MSL command :

odbload a.odb

and get the run number set to 105. It works, but I'm curious if that is the right (envisaged) 
way of interacting with the shell scripts, or one could do better than that.

-- thanks, regards, Pasha

> - how does one communicate with an external shell script from MSL ? I looked at the MIDAS Sequencer page 
>   
>   https://daq00.triumf.ca/MidasWiki/index.php/Sequencer
> 
>   and didn't find an immediately obvious candidate among the MSL commands. 
>   The closest seems to be 
> 
>   'SCRIPT script [, a, b, c, ...]' 
> 
>   but I couldn't easily figure how to propagate the output of the script back to MIDAS. 
>   Let say, the script creates an ASCII file with the next run number. What is the easiest 
>   way to import the run number into ODB? - Should an external script spawn a [short-lived] 
>   MIDAS client ?  - That would work, but I'm almost sure there is a more straightforward solution. 

The output of the SCRTIP command is stored in the variable $SCRIPT_RESULT. Please pull midas to get this
new functionality.

Stefan

>    [that was not obvious from the documentation on MIDAS wiki, and adding a couple of clarifying 
>     sentences there would go long ways]

I added a sentence there. Please have a look. If you like more info, please write it yourself and send it to me. 
It's always better if that comes from users than from me.

 
> 2. if a script produces an ascii file with a known name, for example, 'a.odb', with the following two lines:

Use $SCRIPT_RESULT as described before.

Best,
Stefan

[Dear All, I apologize in advance for spamming.]

1) I tried to login into the forum from the lab computer and realized 
   that I forgot my password
2) I tried to reset the password and found that when registering 
   I mistyped my email address, having typed '.giv' instead of '.gov'
   in the domain name, so the recovery email went into nowhere 

(still have one session open on the laptop so can post this question)

- how do I get my email address fixed so I'd be able to reset the password?

-- many thanks, Pasha

to plot time dependencies of the monitored detector parameters, say, voltages or temperatures,
one needs to convert the coresponging ADC readings into floats.

One could think of two ways of doing that: 

- one can perform the ADC-->T or ADC-->V conversion in the MIDAS frontend, 
  store their [float] values in the data bank, and plot precalculated parameters vs time

- one can also store in the data bank the ADC readings which typically are short's 
  and convert them into floats (V's or T's) at the plotting time 

The first approach doubles the storage space requirements, and I couldn't find the place where 
one would do the conversion, if stored were the 16-bit ADC readings.

I'm sure this issue has been thought about, so what is the "recommended MIDAS way" of performing 
the ADC -> monitored_number conversion when making MIDAS history plots ?

-- many thanks, regards, Pasha

> to plot time dependencies of the monitored detector parameters, say, voltages or temperatures,
> one needs to convert the coresponging ADC readings into floats.
> 
> One could think of two ways of doing that: 
> 
> - one can perform the ADC-->T or ADC-->V conversion in the MIDAS frontend, 
>   store their [float] values in the data bank, and plot precalculated parameters vs time
> 
> - one can also store in the data bank the ADC readings which typically are short's 
>   and convert them into floats (V's or T's) at the plotting time 
> 
> The first approach doubles the storage space requirements, and I couldn't find the place where 
> one would do the conversion, if stored were the 16-bit ADC readings.
> 
> I'm sure this issue has been thought about, so what is the "recommended MIDAS way" of performing 
> the ADC -> monitored_number conversion when making MIDAS history plots ?

Most experiment go with the second method. The front-end program converts all ADC reading into physicsl 
units, i.e. not only Volt, but even Degrees Centigrade or Tesla or whatever. The slow control part of 
midas then puts these number into /Equipment/<name>/Variables as "float", and the history system picks 
them up from there. This way your history is shown in physical units and not ADC count. Actually the 
recommended slow control framework (check the examples direcotory) does not rely on data banks, but 
puts values directly into the ODB. This is typically done faster, like once per second if a value 
changes, rather than slow control events which are generated maybe once per 10 seconds or once per 
minute. Usually the slow control values are only few compared with trigger data, so a factor of two 
there does not really matter. In the MEG experiment, we have like 400 GB of slow control data per year, 
but 400 TB of trigger data per year.

Best,
Stefan

Dear MIDAS developers,

I'm trying to understand handling of the history (slow control) variables in MIDAS,
and it seems that the behavior I'm observing is somewhat counterintuitive. 
Most likely, I just do not understand the implemented logic.

As it it rather difficult to report on the behavior of the interactive program,
I'll describe what I'm doing and illustrate the report with the series of attached 
screenshots showing the history plots and the status of the run control at different 
consecutive points in time.

Starting with the landscape:

- I'm running MIDAS, git commit=30a03c4c (the latest, as of today).

- I have built the midas/examples/slowcont frontend with the following modifications.
  (the diffs are enclosed below):
  
  1) the frequency of the history updates is increased from 60sec/10sec to 6sec/1sec
     and, in hope to have updates continuos, I replaced (RO_RUNNING | RO_TRANSITIONS)
     with RO_ALWAYS.

  2) for convenience of debugging, midas/drivers/nulldrv.cxx is replaced with its clone,
     which instead of returning zeroes in each channel, generates a sine curve:

                  V(t) = 100*sin(t/60)+10*channel

- an active channel in /Logger/History is chosen to be FILE

- /History/LoggerHistoryChannel is also set to FILE 

- I'm running mlogger and modified, as described, 'scfe' frontend from midas/examples/slowcont

- the attached history plots include three (0,4 and 7) HV:MEASURED channels


Now, the observations:

1) the history plots are updated only when a new run starts, no matter how hard
   I'm trying to update them by clicking on various buttons.

   The attached screenshots show the timing sequence of the run control states
   (with the times printed) and the corresponding history plots. 

   The "measured voltages" change only when the next run starts - the voltage graphs 
   break only at the times corresponding to the vertical green lines.

2) No matter for how long I wait within the run, the history updates are not happening.

3) if the time difference between the two run starts gets too large,
   the plotted time dependence starts getting discontinuities

4) finally, if I switch the logging channel from FILE to MIDAS (activate the MIDAS
   channel in /Logger/History and set /History/LoggerHistoryChannel to MIDAS),
   the updates of the history plots simply stop.

MIDAS feels as a great DAQ framework, so I would appreciate any suggestion on 
what I could be doing wrong. I'd also be happy to give a demo in real time 
(via ZOOM/SKYPE etc).

-- much appreciate your time, thanks, regards, Pasha
    
------------------------------------------------------------------------------
diff --git a/examples/slowcont/scfe.cxx b/examples/slowcont/scfe.cxx
index 11f09042..c98d37e8 100644
--- a/examples/slowcont/scfe.cxx
+++ b/examples/slowcont/scfe.cxx
@@ -24,9 +24,10 @@
 #include "mfe.h"
 #include "class/hv.h"
 #include "class/multi.h"
-#include "device/nulldev.h"
 #include "bus/null.h"
 
+#include "nulldev.h"
+
 /*-- Globals -------------------------------------------------------*/
 
 /* The frontend name (client name) as seen by other MIDAS clients   */
@@ -74,11 +75,11 @@ EQUIPMENT equipment[] = {
      0,                         /* event source */
      "FIXED",                   /* format */
      TRUE,                      /* enabled */
-     RO_RUNNING | RO_TRANSITIONS,        /* read when running and on transitions */
-     60000,                     /* read every 60 sec */
+     RO_ALWAYS,        /* read when running and on transitions */
+     6000,                     /* read every 6 sec */
      0,                         /* stop run after this event limit */
      0,                         /* number of sub events */
-     10000,                     /* log history at most every ten seconds */
+     1000,                     /* log history at most every one second */
      "", "", ""} ,
     cd_hv_read,                 /* readout routine */
     cd_hv,                      /* class driver main routine */
@@ -93,8 +94,8 @@ EQUIPMENT equipment[] = {
      0,                         /* event source */
      "FIXED",                   /* format */
      TRUE,                      /* enabled */
-     RO_RUNNING | RO_TRANSITIONS,        /* read when running and on transitions */
-     60000,                     /* read every 60 sec */
+     RO_ALWAYS,        /* read when running and on transitions */
+     6000,                     /* read every 6 sec */
      0,                         /* stop run after this event limit */
      0,                         /* number of sub events */
      1,                         /* log history every event as often as it changes (max 1 Hz) */
------------------------------------------------------------------------------
[test_001]$ diff ../midas/examples/slowcont/nulldev.cxx ../midas/drivers/device/nulldev.cxx 
13d12
< #include <math.h>
150,154c149,150
<    if (channel < info->num_channels) {
<      // *pvalue = info->array[channel];
<      time_t t = time(NULL);;
<      *pvalue = 100*sin(M_PI*t/60)+10*channel;
<    }
---
>    if (channel < info->num_channels)
>       *pvalue = info->array[channel];
------------------------------------------------------------------------------

First of all it's important to understand that the slow control system has nothing to do
with events. So if you look at event statistics, these are the events with the slow control
data sent to the midas data file, not the history database. So the logging period (the one you 
decreased from 60s to 10s to 6s) only affect the generation of events. 

What is important in your case is the number of events sent to the ODB. You see these in the
screen output of the slow control frontend (see attachment). This number show increase every
second.

I tried your modification (change nulldev with a sine function), and left the sc_fe.cxx
otherwise untouched. I then started with a fresh ODB ("rm /"). Started logger, mhttpd, sc_fe
and started a run. In the attachments is what I see. So I don't understand what your problem
is.

Hi Sfefan, thanks a lot for taking time to reproduce the issue! 

Here comes the resolution, and of course, it was something deeply trivial :  

the definition of the HV equipment in midas/examples/slowcont/scfe.cxx has 
the history logging time in seconds, however the comment suggests milliseconds (see below), 
and for a few days I believed to the comment (:smile:)

Easy to fix. 

Also, I think that having a sine wave displayed by midas/examples/slowcont/scfe.cxx 
would make this example even more helpful.

-- thanks again, regards, Pasha

--------------------------------------------------------------------------------------------------------
EQUIPMENT equipment[] = {

   {"HV",                       /* equipment name */
    {3, 0,                      /* event ID, trigger mask */
     "SYSTEM",                  /* event buffer */
     EQ_SLOW,                   /* equipment type */
     0,                         /* event source */
     "FIXED",                   /* format */
     TRUE,                      /* enabled */
     RO_RUNNING | RO_TRANSITIONS,        /* read when running and on transitions */
     60000,                     /* read every 60 sec */
     0,                         /* stop run after this event limit */
     0,                         /* number of sub events */
     10000,                     /* log history at most every ten seconds */ // <------------ this is 10^4 seconds, not 10 seconds
     "", "", ""} ,
    cd_hv_read,                 /* readout routine */
    cd_hv,                      /* class driver main routine */
    hv_driver,                  /* device driver list */
    NULL,                       /* init string */
    },


https://bitbucket.org/tmidas/midas/src/7f0147eb7bc7395f262b3ae90dd0d2af0625af39/examples/slowcont/scfe.cxx#lines-81

> Also, I think that having a sine wave displayed by midas/examples/slowcont/scfe.cxx 
> would make this example even more helpful.

Indeed. I reworked the example to have a out-of-the-box sine wave plotter, including the
automatic creation of a history panel. Thanks for the hint.

Best,
Stefan

MidasWiki was updated to current mediawiki LTS 1.39.6 supported until Nov 2025, 
see https://www.mediawiki.org/wiki/Version_lifecycle

as downside, after this update, I see large amounts of "account request" spam, 
something that did not exist before. I suspect new mediawiki phones home to 
subscribe itself to some "please spam me" list.

if you want a user account on MidasWiki, please email me or Stefan directly, we 
will make it happen.

K.O.

the DNS alias for midas.triumf.ca moved from old ladd00.triumf.ca to new 
daq00.triumf.ca. same as before it redirects to the MidasWiki and to the midas 
forum (elog) that moved from ladd00 to daq00 quite some time ago. if you see any 
anomalies in accessing them (broken links, bad https certificates), please report 
them to this forum or to me directly at olchansk@triumf.ca. K.O.

> the DNS alias for midas.triumf.ca moved from old ladd00.triumf.ca to new 
> daq00.triumf.ca. same as before it redirects to the MidasWiki and to the midas 
> forum (elog) that moved from ladd00 to daq00 quite some time ago. if you see any 
> anomalies in accessing them (broken links, bad https certificates), please report 
> them to this forum or to me directly at olchansk@triumf.ca. K.O.

I found the first issue: The link to

https://midas.triumf.ca/MidasWiki/index.php/Custom_plots_with_mplot

does not work any more. The link

https://daq00.triumf.ca/MidasWiki/index.php/Custom_plots_with_mplot

however does work. Same with

https://midas.triumf.ca/MidasWiki/index.php/Sequencer

and

https://daq00.triumf.ca/MidasWiki/index.php/Sequencer

I have a few cases in mhttpd where I link directly to our documentation. I prefer
to have those link with "midas.triumf.ca" instead of "daq00.triumf.ca" in case you
change the machine again in the future.

Best,
Stefan

> I found the first issue: The link to
> https://midas.triumf.ca/MidasWiki/index.php/Custom_plots_with_mplot

fixed.

https://midas.triumf.ca/Custom_plots_with_mplot

also works.

I tried to get rid of redirect to daq00 completely and make the whole MidasWiki show up 
under midas.triumf.ca, but discovered/remembered that I cannot do this without changing 
MidasWiki config [$wgServer = "https://daq00.triumf.ca";] which causes mediawiki to 
redirect everything to daq00 (using the 301 "moved permanently" reply, ouch!). In theory, 
if I change it to "https://midas.triumf.ca" it will redirect everything there instead, 
but I am hesitant to make this change. It has been like this since forever and I have no 
idea what else will break if I change it.

K.O.