I have a problem with using analyzer offline.
I'm trying to do it this way:

[lkst@pklinux online]$ kill_daq.sh
[lkst@pklinux online]$ odbedit -c "load data/run00020.odb"
[ODBEdit,INFO] Reloading RPC hosts access control list via hotlink callback
[lkst@pklinux online]$ analyzer -i data/run00020.mid -o test20.root
Root server listening on port 9090...
Running analyzer offline. Stop with "!"
[Analyzer,INFO] Set run number 20 in ODB
analyzer: src/odb.c:6631: db_paste_node: Assertion `status == 1' failed.
Load ODB from run 20...Aborted
[lkst@pklinux online]$

I always get this "Assertion `status == 1' failed." error even
if I try it in the examples/experiment enclosed in MIDAS distribution.
After this try I can not run any midas program, even odbedit reports an error,
until I delete the ODB file in /dev/shm/ and load last ODB dump with odbedit.
What do I do wrong and how this can be fixed?

Dear MIDAQ developers,

I wonder if there is a non-intrusive way to have an external (wrt MIDAS)*SQL database 
serving as a primary source of the run number information for a MIDAS-based DAQ system? 
- like a plugin with a getNextRunNumber() function, for example, or a special client?

Here is the use case: 

- multiple subdetectors are taking test data during early commissioning 
- a postgres db is a single sorce of run numbers.
- test runs taken by different subsystems are assigned different [unique] run numbers and 
  the data taken by the subsystem are identified not by the run number/dataset name , but 
  by the run type, different for different susbsystems.

-- many thanks, regards, Pasha

> > - multiple subdetectors are taking test data during early commissioning 
> > - a postgres db is a single sorce of run numbers.
> > - test runs taken by different subsystems are assigned different [unique] run numbers and 
> >   the data taken by the subsystem are identified not by the run number/dataset name , but 
> >   by the run type, different for different susbsystems.
> 
> For that purpose I would not "mis-use" run numbers. Run number are meant to be incremented 
> sequentially, like if you have a time-stamp in seconds since 1.1.1970 (Unix time). Intead, I 
> would add additional attributes under /Experiment/Run Parameters like "Subsystem type", "Run 
> mode (production/commissioning)" etc. You have much more freedom in choosing any number of 
> attributes there. Then, send this attributes to your postgred db via "/Logger/Runlog/SQL/Links 
> BOR". Then you can query your database to give you all runs of a certain subtype or mode.
> 
> See https://daq00.triumf.ca/MidasWiki/index.php/Logging_to_a_mySQL_database
> 
> Stefan

Ben, Stefan - thanks much for your suggestions!(and apologies for the thanks being delayed) 

Stefan, I don't think we're talking 'mis-use' - rather different subdetectors being commisisoned 
at different locations, on an uncorrelated schedule, using independent run control (RC) instances. 
At this point in time, we can't use a common RC instance. 
The collected data, however, are written back into a common storage, and we need to avoid two 
subdetectors  using the same run number. As all RC instances can connect to the same database and request a 
run number from there, an external DB serving run numbers to multiple clients looks as a reasonable solution, 
which provides unique run numbers for everyone. Of course, the run number gets incremented (although on the DB 
server side), and of course different susbystems are assigned different subsystem types. 

So, in essense, it is about _where_ the run number is incremented - the RC vs the DB. 
If there were a good strategy to implement a DB-based solution that w/o violating 
first principles of Midas:), I'd be happy to contribute. It looks like a legitimate use case.

-- let me know, regards, Pasha 

I have one more question, though I understand that it could be somewhat border-line.

The MIDAS state machine doesn't seem to have a state in between 'initialized' and 
'running'.
In a larger detectors with multiple subsystems, the DAQ systems often have one more state:
after ending a previous run and before starting a new one from the 'stopped' state, 
one needs to make sure that all subdetectors are ready, or 'configured' for the new run. 
So theat calls for a 'configure' step during which the detector (all subsystems in 
parallel, to save the time) transitions from 'initialized'/'stopped' to 'configured' state, 
from which it transitions to the 'running' state.

If one of the subdetectors fails to get configured, it could be excluded from the run 
configuration and another attempt to reconfigure the system could be made without 
starting a new run. Or an attempt could be made to troubleshoot and configure the 
failed subsytem individually , with the rest subsystems waiting in a 'configured' state.

How does the logic of configuring the detector for the new run is implemented in MIDAS? 
- it is a fairly common operational procedure, so I'm sure there should be a way 
of doing that.

-- thanks again, Pasha

> 
> If you go in this direction, there is an alternative to what Ben wrote: Use the sequencer to start a run.
> The sequencer script can obtain a new run number from a central instance (e.g. by calling a shell script 
> like 'curl ...' to obtain the new run number, then put it into /Runinfo/Run number as Ben wrote. This has
> the advantage that the run is _started_ already with the correct number, so the history system is fine.
> 

Hi Stefan, this sounds like a perfect solution - thanks! - and leads to another, more technical, question:

- how does one communicate with an external shell script from MSL ? I looked at the MIDAS Sequencer page 
  
  https://daq00.triumf.ca/MidasWiki/index.php/Sequencer

  and didn't find an immediately obvious candidate among the MSL commands. 
  The closest seems to be 

  'SCRIPT script [, a, b, c, ...]' 

  but I couldn't easily figure how to propagate the output of the script back to MIDAS. 
  Let say, the script creates an ASCII file with the next run number. What is the easiest 
  way to import the run number into ODB? - Should an external script spawn a [short-lived] 
  MIDAS client ?  - That would work, but I'm almost sure there is a more straightforward solution. 

  Of course, the assumption that the 'SCRIPT' command provides the solution could be wrong.

-- thanks again, regards, Pasha

> - To start a run, we start a special sequencer script. We have different scripts for
> calibration runs, data runs, special runs.
> 
a sequencer-based way sounds like a very good solution, which provides all needed functionality 
and even more flexibility than a state machine transition. Will give it a try.

-- thanks again, regards, Pasha

> - how does one communicate with an external shell script from MSL ? 

trying to answer my own question, as I didn't find a clear answer in the forum archive :

1. one could have a MSL script with a 'SCRIPT ./myscript.sh' command in it - 
   that would run a shell script named 'myscript.sh'
   [that was not obvious from the documentation on MIDAS wiki, and adding a couple of clarifying 
    sentences there would go long ways]

2. if a script produces an ascii file with a known name, for example, 'a.odb', with the following two lines:

--------------------------------------- a.odb
[/Runinfo]
Run number  = INT32 : 105
--------------------------------------- end a.odb

one can use the 'odbload' MSL command :

odbload a.odb

and get the run number set to 105. It works, but I'm curious if that is the right (envisaged) 
way of interacting with the shell scripts, or one could do better than that.

-- thanks, regards, Pasha

[Dear All, I apologize in advance for spamming.]

1) I tried to login into the forum from the lab computer and realized 
   that I forgot my password
2) I tried to reset the password and found that when registering 
   I mistyped my email address, having typed '.giv' instead of '.gov'
   in the domain name, so the recovery email went into nowhere 

(still have one session open on the laptop so can post this question)

- how do I get my email address fixed so I'd be able to reset the password?

-- many thanks, Pasha

to plot time dependencies of the monitored detector parameters, say, voltages or temperatures,
one needs to convert the coresponging ADC readings into floats.

One could think of two ways of doing that: 

- one can perform the ADC-->T or ADC-->V conversion in the MIDAS frontend, 
  store their [float] values in the data bank, and plot precalculated parameters vs time

- one can also store in the data bank the ADC readings which typically are short's 
  and convert them into floats (V's or T's) at the plotting time 

The first approach doubles the storage space requirements, and I couldn't find the place where 
one would do the conversion, if stored were the 16-bit ADC readings.

I'm sure this issue has been thought about, so what is the "recommended MIDAS way" of performing 
the ADC -> monitored_number conversion when making MIDAS history plots ?

-- many thanks, regards, Pasha

Dear MIDAS developers,

I'm trying to understand handling of the history (slow control) variables in MIDAS,
and it seems that the behavior I'm observing is somewhat counterintuitive. 
Most likely, I just do not understand the implemented logic.

As it it rather difficult to report on the behavior of the interactive program,
I'll describe what I'm doing and illustrate the report with the series of attached 
screenshots showing the history plots and the status of the run control at different 
consecutive points in time.

Starting with the landscape:

- I'm running MIDAS, git commit=30a03c4c (the latest, as of today).

- I have built the midas/examples/slowcont frontend with the following modifications.
  (the diffs are enclosed below):
  
  1) the frequency of the history updates is increased from 60sec/10sec to 6sec/1sec
     and, in hope to have updates continuos, I replaced (RO_RUNNING | RO_TRANSITIONS)
     with RO_ALWAYS.

  2) for convenience of debugging, midas/drivers/nulldrv.cxx is replaced with its clone,
     which instead of returning zeroes in each channel, generates a sine curve:

                  V(t) = 100*sin(t/60)+10*channel

- an active channel in /Logger/History is chosen to be FILE

- /History/LoggerHistoryChannel is also set to FILE 

- I'm running mlogger and modified, as described, 'scfe' frontend from midas/examples/slowcont

- the attached history plots include three (0,4 and 7) HV:MEASURED channels


Now, the observations:

1) the history plots are updated only when a new run starts, no matter how hard
   I'm trying to update them by clicking on various buttons.

   The attached screenshots show the timing sequence of the run control states
   (with the times printed) and the corresponding history plots. 

   The "measured voltages" change only when the next run starts - the voltage graphs 
   break only at the times corresponding to the vertical green lines.

2) No matter for how long I wait within the run, the history updates are not happening.

3) if the time difference between the two run starts gets too large,
   the plotted time dependence starts getting discontinuities

4) finally, if I switch the logging channel from FILE to MIDAS (activate the MIDAS
   channel in /Logger/History and set /History/LoggerHistoryChannel to MIDAS),
   the updates of the history plots simply stop.

MIDAS feels as a great DAQ framework, so I would appreciate any suggestion on 
what I could be doing wrong. I'd also be happy to give a demo in real time 
(via ZOOM/SKYPE etc).

-- much appreciate your time, thanks, regards, Pasha
    
------------------------------------------------------------------------------
diff --git a/examples/slowcont/scfe.cxx b/examples/slowcont/scfe.cxx
index 11f09042..c98d37e8 100644
--- a/examples/slowcont/scfe.cxx
+++ b/examples/slowcont/scfe.cxx
@@ -24,9 +24,10 @@
 #include "mfe.h"
 #include "class/hv.h"
 #include "class/multi.h"
-#include "device/nulldev.h"
 #include "bus/null.h"
 
+#include "nulldev.h"
+
 /*-- Globals -------------------------------------------------------*/
 
 /* The frontend name (client name) as seen by other MIDAS clients   */
@@ -74,11 +75,11 @@ EQUIPMENT equipment[] = {
      0,                         /* event source */
      "FIXED",                   /* format */
      TRUE,                      /* enabled */
-     RO_RUNNING | RO_TRANSITIONS,        /* read when running and on transitions */
-     60000,                     /* read every 60 sec */
+     RO_ALWAYS,        /* read when running and on transitions */
+     6000,                     /* read every 6 sec */
      0,                         /* stop run after this event limit */
      0,                         /* number of sub events */
-     10000,                     /* log history at most every ten seconds */
+     1000,                     /* log history at most every one second */
      "", "", ""} ,
     cd_hv_read,                 /* readout routine */
     cd_hv,                      /* class driver main routine */
@@ -93,8 +94,8 @@ EQUIPMENT equipment[] = {
      0,                         /* event source */
      "FIXED",                   /* format */
      TRUE,                      /* enabled */
-     RO_RUNNING | RO_TRANSITIONS,        /* read when running and on transitions */
-     60000,                     /* read every 60 sec */
+     RO_ALWAYS,        /* read when running and on transitions */
+     6000,                     /* read every 6 sec */
      0,                         /* stop run after this event limit */
      0,                         /* number of sub events */
      1,                         /* log history every event as often as it changes (max 1 Hz) */
------------------------------------------------------------------------------
[test_001]$ diff ../midas/examples/slowcont/nulldev.cxx ../midas/drivers/device/nulldev.cxx 
13d12
< #include <math.h>
150,154c149,150
<    if (channel < info->num_channels) {
<      // *pvalue = info->array[channel];
<      time_t t = time(NULL);;
<      *pvalue = 100*sin(M_PI*t/60)+10*channel;
<    }
---
>    if (channel < info->num_channels)
>       *pvalue = info->array[channel];
------------------------------------------------------------------------------

Hi Sfefan, thanks a lot for taking time to reproduce the issue! 

Here comes the resolution, and of course, it was something deeply trivial :  

the definition of the HV equipment in midas/examples/slowcont/scfe.cxx has 
the history logging time in seconds, however the comment suggests milliseconds (see below), 
and for a few days I believed to the comment (:smile:)

Easy to fix. 

Also, I think that having a sine wave displayed by midas/examples/slowcont/scfe.cxx 
would make this example even more helpful.

-- thanks again, regards, Pasha

--------------------------------------------------------------------------------------------------------
EQUIPMENT equipment[] = {

   {"HV",                       /* equipment name */
    {3, 0,                      /* event ID, trigger mask */
     "SYSTEM",                  /* event buffer */
     EQ_SLOW,                   /* equipment type */
     0,                         /* event source */
     "FIXED",                   /* format */
     TRUE,                      /* enabled */
     RO_RUNNING | RO_TRANSITIONS,        /* read when running and on transitions */
     60000,                     /* read every 60 sec */
     0,                         /* stop run after this event limit */
     0,                         /* number of sub events */
     10000,                     /* log history at most every ten seconds */ // <------------ this is 10^4 seconds, not 10 seconds
     "", "", ""} ,
    cd_hv_read,                 /* readout routine */
    cd_hv,                      /* class driver main routine */
    hv_driver,                  /* device driver list */
    NULL,                       /* init string */
    },


https://bitbucket.org/tmidas/midas/src/7f0147eb7bc7395f262b3ae90dd0d2af0625af39/examples/slowcont/scfe.cxx#lines-81

Dear all,

I have implemented a number of slow control frontends which are directed to update the 
history once in every 10 sec, and they do just that. 

I expected that such frontends would be spending most of the time sleeping and waking up 
once in ten seconds to call their respective drivers and send the data to the server. 

However I observe that each frontend process consumes almost 100% of a single core CPU time 
and the frontend driver is called many times per second. 

Is that the expected behavior ?

So far, I couldn't find the place in the system part of the frontend code (is that the right 
place to look for?) which regulates the frequency of the frontend driver calls, so I'd greatly 
appreciate if someone could point me to that place.

I'm using the following commit:

commit 30a03c4c develop origin/develop Make sure line numbers and sequence lines are aligned.

-- many thanks, regards, Pasha

Hi Stefan, thanks a lot !

I just thought that for the EQ_SLOW type equipment calls to sleep() could be hidden in mfe.cxx 
and handled based on the requested frequency of the history updates.

Doing the same in the user side is straighforward - the important part is to know where the 
responsibility line goes (: smile :) 

-- regards, Pasha

Dear all,

I have a "slow control" frontend which reads out 100 slow control parameters.
When I'm interactively adding a parameter to a history plot, 
a nice "Add history variable" pops up .. , but with 100 parameters in the list, 
it doesn't fit within the screen... 

The browser becomes passive, and I didn't find any easy way of scrolling.

In the attached example, adding a channel 32 variable becomes rather cumbersome, 
not speaking about channel 99.

Two questions:

a) how do people get around this "no-scrolling" issue? - perhaps there is a workaround

b) how big of a deal is it to add a scroll bar to the "Add history variables" popup ? 
   - I do not know javascript myself, but could find help to contribute..

-- many thanks, regards, Pasha

> Have you updated to the current midas version? This issue has been fixed a while ago. 

Hi Stefan, thanks a lot! I pulled from the head, and the scrolling works now. -- regards, Pasha

I don't immediately see a reason for saying that if a DB key is older than 10 yrs, it may not be valid.

However, it would be worth learning what was the logic behind choosing 10 yrs as a threshold. 
If 10 is just a more or less arbitrary number, changing 10 --> 100 seems to be the way to go.

-- regards, Pasha 

Dear MIDAS experts, 

- I have a detector configuration with a variable number of hardware components - FPGA's receiving data 
  from the detector. They are described in ODB using a set of keys ranging 
  from "/Detector/FPGAs/FPGA00" .... to "/Detector/FPGAs/FPGA68".
  Each of "FPGAxx" corresponds to an ODB subdirectory containing parameters of a given FPGA. 

  The number of FPGAs in the detector configuration is variable - [independent] commissioning 
  of different detector subsystems involves different number of FPGAs.

  In the beginning of the data taking one needs to loop over all of "FPGAxx", 
  parse the information there and initialize the corresponding FPGAs.

The actual question sounds rather trivial - what is the best way to implement a loop over them? 

- it is certainly possible to have the number of FPGAs introduced as an additional configuration parameter, 
  say, "/Detector/Number_of_FPGAs", and this is what I have resorted to right now.

  However, not only that loooks ugly, but it also opens a way to make a mistake 
  and have the Number_of_FPGAs, introduced separately, different from the actual number 
  of FPGA's in the detector configuration.
 
I therefore wonder if there could be a function, smth like 

    int db_get_n_keys(HNDLE hdb, HNDLE hKeyParent)

returning the number of ODB keys with a common parent, or, to put it simpler, 
a number of ODB entries in a given subdirectory.

And if there were a better solution to the problem I'm dealing with, knowing it might be helpful 
for more than one person - configuring detector readout may require to deal with a variable number 
of very different parameters.

-- many thanks, regards, Pasha

Hi Stefan, Konstantin, 

thanks a lot for your responses - they are very teaching and it is good to have them archived in the forum.
 
Konstantin, as Stefan already noticed, in this particular case the race condition is not really a concern.

Stefan, the ChatGPT-generated code snippet is awesome! (teach a man how to fish ...)

-- regards, Pasha

> If you have some ideas on how to better present 100500 history variables, please shout out!

let me share some thoughts. In a particular case which lead to the original posting, 
I was using a multi-threaded driver and monitoring several pieces of equipment with different device drivers.  
In fact, it was not even hardware, but processes running on different nodes of a distributed computer farm.
To reduce the number of frontends, I was combining together the output of what could've been implemented 
as multiple slow control drivers and got 100+ variables in the list - hence the scrolling experience.

At the same time, a list of control variables per driver could've been kept relatively short.

So if a list of control variables of a slow control frontend were split in a History GUI not only by the 
equipment piece, but within the equipment "folder", also by the driver, that might help improving 
the scalability of the graphical interface. 

May be that is already implemented and it is just a matter of me not finding the right base class / example 
in the MIDAS code

-- regards, Pasha

Konstantin is right: KEY.num_values is not the same as the number of subkeys (should it be ?)
For those looking for an example in the future, I attach a working piece of code converted 
from the ChatGPT example, together with its printout.

-- regards, Pasha