Back Midas Rome Roody Rootana
  Midas DAQ System, Page 4 of 136  Not logged in ELOG logo
ID Date Author Topic Subject
  2669   11 Jan 2024 Pavel MuratForumslow control frontends - how much do they sleep and how often their drivers are called?
Hi Stefan, thanks a lot !

I just thought that for the EQ_SLOW type equipment calls to sleep() could be hidden in mfe.cxx 
and handled based on the requested frequency of the history updates.

Doing the same in the user side is straighforward - the important part is to know where the 
responsibility line goes (: smile :) 

-- regards, Pasha
  2668   11 Jan 2024 Stefan RittForumslow control frontends - how much do they sleep and how often their drivers are called?
Put a 

  ss_sleep(10);

into your frontend_loop(), then you should be fine.

The event loop runs as fast as possible in order not to miss any (triggered) event, so no seep in the 
event loop, because this would limit the (triggered) event rate to 100 Hz (minimum sleep is 10 ms). 
Therefore, you have to slow down the event loop manually with the method described above.

Best,
Stefan
  2667   10 Jan 2024 Pavel MuratForumslow control frontends - how much do they sleep and how often their drivers are called?
Dear all,

I have implemented a number of slow control frontends which are directed to update the 
history once in every 10 sec, and they do just that. 

I expected that such frontends would be spending most of the time sleeping and waking up 
once in ten seconds to call their respective drivers and send the data to the server. 

However I observe that each frontend process consumes almost 100% of a single core CPU time 
and the frontend driver is called many times per second. 

Is that the expected behavior ?

So far, I couldn't find the place in the system part of the frontend code (is that the right 
place to look for?) which regulates the frequency of the frontend driver calls, so I'd greatly 
appreciate if someone could point me to that place.

I'm using the following commit:

commit 30a03c4c develop origin/develop Make sure line numbers and sequence lines are aligned.

-- many thanks, regards, Pasha
  2666   03 Jan 2024 Konstantin OlchanskiForummidas.triumf.ca alias moved to daq00.triumf.ca
> I found the first issue: The link to
> https://midas.triumf.ca/MidasWiki/index.php/Custom_plots_with_mplot

fixed.

https://midas.triumf.ca/Custom_plots_with_mplot

also works.

I tried to get rid of redirect to daq00 completely and make the whole MidasWiki show up 
under midas.triumf.ca, but discovered/remembered that I cannot do this without changing 
MidasWiki config [$wgServer = "https://daq00.triumf.ca";] which causes mediawiki to 
redirect everything to daq00 (using the 301 "moved permanently" reply, ouch!). In theory, 
if I change it to "https://midas.triumf.ca" it will redirect everything there instead, 
but I am hesitant to make this change. It has been like this since forever and I have no 
idea what else will break if I change it.

K.O.
  2665   03 Jan 2024 Stefan RittBug ReportCompilation error on RPi
> > git pull
> > git submodule update
> 
> confirmed. just run into this myself. I think "make" should warn about out of 
> date git modules. Also check that the build git version is tagged with "-dirty".
> 
> K.O.

The submodule business becomes kind of annoying. I updated the documentation at

https://daq00.triumf.ca/MidasWiki/index.php/Quickstart_Linux#MIDAS_Package_Installatio
n

to tell people to use 

1) "git clone ... --recurse-submodules" during the first clone
2) "git submodule update --init --recursive" in case they forgot 1)
3) "git pull --recurse-submodules" for each update or to use
4) "git config submodule.recurse true" to make the --recurse-submodules the default

I use 4) since a while and it works nicely, so one does not have to remember to pull 
recursively each time.

Stefan
  2664   03 Jan 2024 Stefan RittForummidas.triumf.ca alias moved to daq00.triumf.ca
> the DNS alias for midas.triumf.ca moved from old ladd00.triumf.ca to new 
> daq00.triumf.ca. same as before it redirects to the MidasWiki and to the midas 
> forum (elog) that moved from ladd00 to daq00 quite some time ago. if you see any 
> anomalies in accessing them (broken links, bad https certificates), please report 
> them to this forum or to me directly at olchansk@triumf.ca. K.O.

I found the first issue: The link to

https://midas.triumf.ca/MidasWiki/index.php/Custom_plots_with_mplot

does not work any more. The link

https://daq00.triumf.ca/MidasWiki/index.php/Custom_plots_with_mplot

however does work. Same with

https://midas.triumf.ca/MidasWiki/index.php/Sequencer

and

https://daq00.triumf.ca/MidasWiki/index.php/Sequencer

I have a few cases in mhttpd where I link directly to our documentation. I prefer
to have those link with "midas.triumf.ca" instead of "daq00.triumf.ca" in case you
change the machine again in the future.

Best,
Stefan
  2663   02 Jan 2024 Konstantin OlchanskiForummidas.triumf.ca alias moved to daq00.triumf.ca
the DNS alias for midas.triumf.ca moved from old ladd00.triumf.ca to new 
daq00.triumf.ca. same as before it redirects to the MidasWiki and to the midas 
forum (elog) that moved from ladd00 to daq00 quite some time ago. if you see any 
anomalies in accessing them (broken links, bad https certificates), please report 
them to this forum or to me directly at olchansk@triumf.ca. K.O.
  2662   29 Dec 2023 Konstantin OlchanskiBug ReportCompilation error on RPi
> git pull
> git submodule update

confirmed. just run into this myself. I think "make" should warn about out of 
date git modules. Also check that the build git version is tagged with "-dirty".

K.O.
  2661   27 Dec 2023 Konstantin OlchanskiForumMidasWiki updated to 1.39.6
MidasWiki was updated to current mediawiki LTS 1.39.6 supported until Nov 2025, 
see https://www.mediawiki.org/wiki/Version_lifecycle

as downside, after this update, I see large amounts of "account request" spam, 
something that did not exist before. I suspect new mediawiki phones home to 
subscribe itself to some "please spam me" list.

if you want a user account on MidasWiki, please email me or Stefan directly, we 
will make it happen.

K.O.
  2660   15 Dec 2023 Stefan RittInfoImplementation of custom scatter, histogram and color map plots
Custom plots including scatter, histogram and color map plots have been 
implemented. This lets you plot graphs of X/Y data or histogram data stored in the 
ODB on a custom page. For some examples and documentation please go to

https://daq00.triumf.ca/MidasWiki/index.php/Custom_plots_with_mplot

Enjoy!
Stefan
Attachment 1: plots.png
plots.png
  2658   14 Dec 2023 Zaher SalmanBug ReportCompilation error on RPi
This issue was resolved thanks to Konstantin and Stefan. I simply had to update submodules:

git submodule update

and then recompile.

Zaher
  2657   13 Dec 2023 Stefan RittForumthe logic of handling history variables ?
> Also, I think that having a sine wave displayed by midas/examples/slowcont/scfe.cxx 
> would make this example even more helpful.

Indeed. I reworked the example to have a out-of-the-box sine wave plotter, including the
automatic creation of a history panel. Thanks for the hint.

Best,
Stefan
  2656   12 Dec 2023 Pavel MuratForumthe logic of handling history variables ?
Hi Sfefan, thanks a lot for taking time to reproduce the issue! 

Here comes the resolution, and of course, it was something deeply trivial :  

the definition of the HV equipment in midas/examples/slowcont/scfe.cxx has 
the history logging time in seconds, however the comment suggests milliseconds (see below), 
and for a few days I believed to the comment (:smile:)

Easy to fix. 

Also, I think that having a sine wave displayed by midas/examples/slowcont/scfe.cxx 
would make this example even more helpful.

-- thanks again, regards, Pasha

--------------------------------------------------------------------------------------------------------
EQUIPMENT equipment[] = {

   {"HV",                       /* equipment name */
    {3, 0,                      /* event ID, trigger mask */
     "SYSTEM",                  /* event buffer */
     EQ_SLOW,                   /* equipment type */
     0,                         /* event source */
     "FIXED",                   /* format */
     TRUE,                      /* enabled */
     RO_RUNNING | RO_TRANSITIONS,        /* read when running and on transitions */
     60000,                     /* read every 60 sec */
     0,                         /* stop run after this event limit */
     0,                         /* number of sub events */
     10000,                     /* log history at most every ten seconds */ // <------------ this is 10^4 seconds, not 10 seconds
     "", "", ""} ,
    cd_hv_read,                 /* readout routine */
    cd_hv,                      /* class driver main routine */
    hv_driver,                  /* device driver list */
    NULL,                       /* init string */
    },


https://bitbucket.org/tmidas/midas/src/7f0147eb7bc7395f262b3ae90dd0d2af0625af39/examples/slowcont/scfe.cxx#lines-81
  2655   12 Dec 2023 Zaher SalmanBug ReportCompilation error on RPi

Hello,

Since commit bc227a8a34def271a598c0200ca30d73223c3373 I've been getting the compilation error below (on a Raspberry Pi 3 Model B Plus Rev 1.3).

The fix is obvious from the reported error, but I am wondering whether this should be fixed in the main git??

Thanks,

Zaher

 

[  7%] Building CXX object CMakeFiles/objlib.dir/src/json_paste.cxx.o
/home/nemu/nemu/tmidas/midas/src/json_paste.cxx: In function ‘int GetQWORD(const MJsonNode*
, const char*, UINT64*)’:
/home/nemu/nemu/tmidas/midas/src/json_paste.cxx:324:19: error: const class MJsonNode’ has
no member named ‘GetLL’; did you mean ‘GetInt’?
      *qw = node->GetLL();
                  ^~~~~
                  GetInt
make[2]: *** [CMakeFiles/objlib.dir/build.make:271: CMakeFiles/objlib.dir/src/json_paste.cx

  2654   12 Dec 2023 Stefan RittInfoMidas Holiday Update
> > 3) Make /Sequencer/State/Path relative to <exp>/userfiles. Like if /Sequencer/State/Path=test would then result to a final directory <exp>/userfiles/sequencer/test
> > 
> > I'm kind of tempted to go with 3), since this allows the experiment to define different subdirectories under <exp>/userfiles/sequencer/... depending on the situation of the experiment.
> > 
> > Best,
> > Stefan
> 
> For me the option 3) seems the most coherent one.
> Andreas

Ok, I implemented option 3) above. This means everybody using the midas sequencer has to change /Sequencer/State/Path to an empty string and move the 
sequencer files under <exp>/userfiles/sequencer as a starting point. I tested most thing, including the INCLUDE statements, but there could still be
a bug here or there, so please give it a try and report any issue to me.

Best,
Stefan
  2653   11 Dec 2023 Stefan RittForumthe logic of handling history variables ?
First of all it's important to understand that the slow control system has nothing to do
with events. So if you look at event statistics, these are the events with the slow control
data sent to the midas data file, not the history database. So the logging period (the one you 
decreased from 60s to 10s to 6s) only affect the generation of events. 

What is important in your case is the number of events sent to the ODB. You see these in the
screen output of the slow control frontend (see attachment). This number show increase every
second.

I tried your modification (change nulldev with a sine function), and left the sc_fe.cxx
otherwise untouched. I then started with a fresh ODB ("rm /"). Started logger, mhttpd, sc_fe
and started a run. In the attachments is what I see. So I don't understand what your problem
is.
Attachment 1: Screenshot_2023-12-12_at_09.03.13.png
Screenshot_2023-12-12_at_09.03.13.png
Attachment 2: Screenshot_2023-12-12_at_08.57.45.png
Screenshot_2023-12-12_at_08.57.45.png
Attachment 3: Screenshot_2023-12-12_at_08.55.14.png
Screenshot_2023-12-12_at_08.55.14.png
Attachment 4: Screenshot_2023-12-12_at_08.57.37.png
Screenshot_2023-12-12_at_08.57.37.png
  2652   11 Dec 2023 Pavel MuratForumthe logic of handling history variables ?
Dear MIDAS developers,

I'm trying to understand handling of the history (slow control) variables in MIDAS,
and it seems that the behavior I'm observing is somewhat counterintuitive. 
Most likely, I just do not understand the implemented logic.

As it it rather difficult to report on the behavior of the interactive program,
I'll describe what I'm doing and illustrate the report with the series of attached 
screenshots showing the history plots and the status of the run control at different 
consecutive points in time.

Starting with the landscape:

- I'm running MIDAS, git commit=30a03c4c (the latest, as of today).

- I have built the midas/examples/slowcont frontend with the following modifications.
  (the diffs are enclosed below):
  
  1) the frequency of the history updates is increased from 60sec/10sec to 6sec/1sec
     and, in hope to have updates continuos, I replaced (RO_RUNNING | RO_TRANSITIONS)
     with RO_ALWAYS.

  2) for convenience of debugging, midas/drivers/nulldrv.cxx is replaced with its clone,
     which instead of returning zeroes in each channel, generates a sine curve:

                  V(t) = 100*sin(t/60)+10*channel

- an active channel in /Logger/History is chosen to be FILE

- /History/LoggerHistoryChannel is also set to FILE 

- I'm running mlogger and modified, as described, 'scfe' frontend from midas/examples/slowcont

- the attached history plots include three (0,4 and 7) HV:MEASURED channels


Now, the observations:

1) the history plots are updated only when a new run starts, no matter how hard
   I'm trying to update them by clicking on various buttons.

   The attached screenshots show the timing sequence of the run control states
   (with the times printed) and the corresponding history plots. 

   The "measured voltages" change only when the next run starts - the voltage graphs 
   break only at the times corresponding to the vertical green lines.

2) No matter for how long I wait within the run, the history updates are not happening.

3) if the time difference between the two run starts gets too large,
   the plotted time dependence starts getting discontinuities

4) finally, if I switch the logging channel from FILE to MIDAS (activate the MIDAS
   channel in /Logger/History and set /History/LoggerHistoryChannel to MIDAS),
   the updates of the history plots simply stop.

MIDAS feels as a great DAQ framework, so I would appreciate any suggestion on 
what I could be doing wrong. I'd also be happy to give a demo in real time 
(via ZOOM/SKYPE etc).

-- much appreciate your time, thanks, regards, Pasha
    
------------------------------------------------------------------------------
diff --git a/examples/slowcont/scfe.cxx b/examples/slowcont/scfe.cxx
index 11f09042..c98d37e8 100644
--- a/examples/slowcont/scfe.cxx
+++ b/examples/slowcont/scfe.cxx
@@ -24,9 +24,10 @@
 #include "mfe.h"
 #include "class/hv.h"
 #include "class/multi.h"
-#include "device/nulldev.h"
 #include "bus/null.h"
 
+#include "nulldev.h"
+
 /*-- Globals -------------------------------------------------------*/
 
 /* The frontend name (client name) as seen by other MIDAS clients   */
@@ -74,11 +75,11 @@ EQUIPMENT equipment[] = {
      0,                         /* event source */
      "FIXED",                   /* format */
      TRUE,                      /* enabled */
-     RO_RUNNING | RO_TRANSITIONS,        /* read when running and on transitions */
-     60000,                     /* read every 60 sec */
+     RO_ALWAYS,        /* read when running and on transitions */
+     6000,                     /* read every 6 sec */
      0,                         /* stop run after this event limit */
      0,                         /* number of sub events */
-     10000,                     /* log history at most every ten seconds */
+     1000,                     /* log history at most every one second */
      "", "", ""} ,
     cd_hv_read,                 /* readout routine */
     cd_hv,                      /* class driver main routine */
@@ -93,8 +94,8 @@ EQUIPMENT equipment[] = {
      0,                         /* event source */
      "FIXED",                   /* format */
      TRUE,                      /* enabled */
-     RO_RUNNING | RO_TRANSITIONS,        /* read when running and on transitions */
-     60000,                     /* read every 60 sec */
+     RO_ALWAYS,        /* read when running and on transitions */
+     6000,                     /* read every 6 sec */
      0,                         /* stop run after this event limit */
      0,                         /* number of sub events */
      1,                         /* log history every event as often as it changes (max 1 Hz) */
------------------------------------------------------------------------------
[test_001]$ diff ../midas/examples/slowcont/nulldev.cxx ../midas/drivers/device/nulldev.cxx 
13d12
< #include <math.h>
150,154c149,150
<    if (channel < info->num_channels) {
<      // *pvalue = info->array[channel];
<      time_t t = time(NULL);;
<      *pvalue = 100*sin(M_PI*t/60)+10*channel;
<    }
---
>    if (channel < info->num_channels)
>       *pvalue = info->array[channel];
------------------------------------------------------------------------------
Attachment 1: Screenshot_20231211_213608.png
Screenshot_20231211_213608.png
Attachment 2: Screenshot_20231211_213736.png
Screenshot_20231211_213736.png
Attachment 3: Screenshot_20231211_220347.png
Screenshot_20231211_220347.png
Attachment 4: Screenshot_20231211_220508.png
Screenshot_20231211_220508.png
Attachment 5: Screenshot_20231211_221041.png
Screenshot_20231211_221041.png
Attachment 6: Screenshot_20231211_221252.png
Screenshot_20231211_221252.png
Attachment 7: Screenshot_20231211_221406.png
Screenshot_20231211_221406.png
Attachment 8: Screenshot_20231211_222042.png
Screenshot_20231211_222042.png
Attachment 9: Screenshot_20231211_222114.png
Screenshot_20231211_222114.png
Attachment 10: Screenshot_20231211_224016.png
Screenshot_20231211_224016.png
Attachment 11: Screenshot_20231211_224141.png
Screenshot_20231211_224141.png
  2651   10 Dec 2023 Andreas SuterInfoMidas Holiday Update
> > If I understand the msequencer code correctly:
> > Under '/Sequencer/State/Path' the path can be defined from where the msequencer gets the files, generates the xml, etc.
> > However, the new javascript code reads/writes the files to '<exp>/userfiles/sequencer/'
> > 
> > If the path in the ODB is different to '<exp>/userfiles/sequencer/', it leads to quite some unexpected behavior. If '<exp>/userfiles/sequencer/' is the place where things should go, the ODB entry of the msequencer and the internal handling should probably adopted, no?
> 
> Indeed there is a change in philosophy. Previously, /Sequencer/State/Path could point anywhere in the file system. This was considered a security problem, since one could access system files under /etc for example via the midas interface. When the new file API was 
> introduced recently, it has therefor been decided that all files accessible remotely should reside under <exp>/userfiles. If an experiment needs some files outside of that directory, the experiment could define some symbolic link, but that's then in the responsibility of 
> the experiment. 
> 
> To resolve now the issue between the sequencer path and the userfiles, we have different options, and I would like to get some feedback from the community, since *all experiments* have to do that change.
> 
> 1) Leave thins as they are, but explain that everybody should modify /Sequencer/Stat/Path to some subdirectory of <exp>/userfiles/sequencer
> 
> 2) Drop /Sequencer/State/Path completely and "hard-wire" it to <exp>/usefiles/sequencer
> 
> 3) Make /Sequencer/State/Path relative to <exp>/userfiles. Like if /Sequencer/State/Path=test would then result to a final directory <exp>/userfiles/sequencer/test
> 
> I'm kind of tempted to go with 3), since this allows the experiment to define different subdirectories under <exp>/userfiles/sequencer/... depending on the situation of the experiment.
> 
> Best,
> Stefan

For me the option 3) seems the most coherent one.
Andreas
  2650   10 Dec 2023 Stefan RittForumhistory plotting: where to convert the ADC readings into temps/voltages?
> to plot time dependencies of the monitored detector parameters, say, voltages or temperatures,
> one needs to convert the coresponging ADC readings into floats.
> 
> One could think of two ways of doing that: 
> 
> - one can perform the ADC-->T or ADC-->V conversion in the MIDAS frontend, 
>   store their [float] values in the data bank, and plot precalculated parameters vs time
> 
> - one can also store in the data bank the ADC readings which typically are short's 
>   and convert them into floats (V's or T's) at the plotting time 
> 
> The first approach doubles the storage space requirements, and I couldn't find the place where 
> one would do the conversion, if stored were the 16-bit ADC readings.
> 
> I'm sure this issue has been thought about, so what is the "recommended MIDAS way" of performing 
> the ADC -> monitored_number conversion when making MIDAS history plots ?

Most experiment go with the second method. The front-end program converts all ADC reading into physicsl 
units, i.e. not only Volt, but even Degrees Centigrade or Tesla or whatever. The slow control part of 
midas then puts these number into /Equipment/<name>/Variables as "float", and the history system picks 
them up from there. This way your history is shown in physical units and not ADC count. Actually the 
recommended slow control framework (check the examples direcotory) does not rely on data banks, but 
puts values directly into the ODB. This is typically done faster, like once per second if a value 
changes, rather than slow control events which are generated maybe once per 10 seconds or once per 
minute. Usually the slow control values are only few compared with trigger data, so a factor of two 
there does not really matter. In the MEG experiment, we have like 400 GB of slow control data per year, 
but 400 TB of trigger data per year.

Best,
Stefan
  2649   10 Dec 2023 Stefan RittInfoMidas Holiday Update
> If I understand the msequencer code correctly:
> Under '/Sequencer/State/Path' the path can be defined from where the msequencer gets the files, generates the xml, etc.
> However, the new javascript code reads/writes the files to '<exp>/userfiles/sequencer/'
> 
> If the path in the ODB is different to '<exp>/userfiles/sequencer/', it leads to quite some unexpected behavior. If '<exp>/userfiles/sequencer/' is the place where things should go, the ODB entry of the msequencer and the internal handling should probably adopted, no?

Indeed there is a change in philosophy. Previously, /Sequencer/State/Path could point anywhere in the file system. This was considered a security problem, since one could access system files under /etc for example via the midas interface. When the new file API was 
introduced recently, it has therefor been decided that all files accessible remotely should reside under <exp>/userfiles. If an experiment needs some files outside of that directory, the experiment could define some symbolic link, but that's then in the responsibility of 
the experiment. 

To resolve now the issue between the sequencer path and the userfiles, we have different options, and I would like to get some feedback from the community, since *all experiments* have to do that change.

1) Leave thins as they are, but explain that everybody should modify /Sequencer/Stat/Path to some subdirectory of <exp>/userfiles/sequencer

2) Drop /Sequencer/State/Path completely and "hard-wire" it to <exp>/usefiles/sequencer

3) Make /Sequencer/State/Path relative to <exp>/userfiles. Like if /Sequencer/State/Path=test would then result to a final directory <exp>/userfiles/sequencer/test

I'm kind of tempted to go with 3), since this allows the experiment to define different subdirectories under <exp>/userfiles/sequencer/... depending on the situation of the experiment.

Best,
Stefan
ELOG V3.1.4-2e1708b5