ID |
Date |
Author |
Topic |
Subject |
2667
|
10 Jan 2024 |
Pavel Murat | Forum | slow control frontends - how much do they sleep and how often their drivers are called? | Dear all,
I have implemented a number of slow control frontends which are directed to update the
history once in every 10 sec, and they do just that.
I expected that such frontends would be spending most of the time sleeping and waking up
once in ten seconds to call their respective drivers and send the data to the server.
However I observe that each frontend process consumes almost 100% of a single core CPU time
and the frontend driver is called many times per second.
Is that the expected behavior ?
So far, I couldn't find the place in the system part of the frontend code (is that the right
place to look for?) which regulates the frequency of the frontend driver calls, so I'd greatly
appreciate if someone could point me to that place.
I'm using the following commit:
commit 30a03c4c develop origin/develop Make sure line numbers and sequence lines are aligned.
-- many thanks, regards, Pasha |
2668
|
11 Jan 2024 |
Stefan Ritt | Forum | slow control frontends - how much do they sleep and how often their drivers are called? | Put a
ss_sleep(10);
into your frontend_loop(), then you should be fine.
The event loop runs as fast as possible in order not to miss any (triggered) event, so no seep in the
event loop, because this would limit the (triggered) event rate to 100 Hz (minimum sleep is 10 ms).
Therefore, you have to slow down the event loop manually with the method described above.
Best,
Stefan |
2669
|
11 Jan 2024 |
Pavel Murat | Forum | slow control frontends - how much do they sleep and how often their drivers are called? | Hi Stefan, thanks a lot !
I just thought that for the EQ_SLOW type equipment calls to sleep() could be hidden in mfe.cxx
and handled based on the requested frequency of the history updates.
Doing the same in the user side is straighforward - the important part is to know where the
responsibility line goes (: smile :)
-- regards, Pasha |
2670
|
12 Jan 2024 |
Stefan Ritt | Forum | slow control frontends - how much do they sleep and how often their drivers are called? | > Hi Stefan, thanks a lot !
>
> I just thought that for the EQ_SLOW type equipment calls to sleep() could be hidden in mfe.cxx
> and handled based on the requested frequency of the history updates.
Most people combine EQ_SLOW with EQ_POLLED, so they want to read out as quickly as possible. Since
the framework cannot "guess" what the users want there, I removed all sleep() in the framework.
> Doing the same in the user side is straighforward - the important part is to know where the
> responsibility line goes (: smile :)
Pushing this to the user gives you more freedom. Like you can add sleep() for some frontends, but not
for others, only when the run is stopped and more.
Stefan |
2671
|
15 Jan 2024 |
Frederik Wauters | Forum | dump history FILE files | We switched from the history files from MIDAS to FILE, so we have *.dat files now (per variable), instead of the old *.hst.
How shoul
d one now extract data from these data files? With the old *,hst files I can e.g. mhdump -E 102 231010.hst
but with the new *.dat files I get
...2023/history$ mhdump -E 0 -T "Run number" mhf_1697445335_20231016_run_transitions.dat | head -n 15
event name: [Run transitions], time [1697445335]
tag: tag: /DWORD 1 4 /timestamp
tag: tag: UINT32 1 4 State
tag: tag: UINT32 1 4 Run number
record size: 12, data offset: 1024
record 0, time 1697557722, incr 112387
record 1, time 1697557783, incr 61
record 2, time 1697557804, incr 21
record 3, time 1697557834, incr 30
record 4, time 1697557888, incr 54
record 5, time 1697558318, incr 430
record 6, time 1697558323, incr 5
record 7, time 1697558659, incr 336
record 8, time 1697558668, incr 9
record 9, time 1697558753, incr 85
not very intelligible
Yes, I can do csv export on the webpage. But it would be nice to be able to extract from just the files. Also, the webpage export only saves the data shown ( range limited and/or downsampled) |
2672
|
16 Jan 2024 |
Pavel Murat | Forum | a scroll option for "add history variables" window? | Dear all,
I have a "slow control" frontend which reads out 100 slow control parameters.
When I'm interactively adding a parameter to a history plot,
a nice "Add history variable" pops up .. , but with 100 parameters in the list,
it doesn't fit within the screen...
The browser becomes passive, and I didn't find any easy way of scrolling.
In the attached example, adding a channel 32 variable becomes rather cumbersome,
not speaking about channel 99.
Two questions:
a) how do people get around this "no-scrolling" issue? - perhaps there is a workaround
b) how big of a deal is it to add a scroll bar to the "Add history variables" popup ?
- I do not know javascript myself, but could find help to contribute..
-- many thanks, regards, Pasha |
Attachment 1: adding_a_variable_to_the_history_plot.png
|
|
2673
|
16 Jan 2024 |
Stefan Ritt | Forum | a scroll option for "add history variables" window? | Have you updated to the current midas version? This issue has been fixed a while ago. Below
you see a screenshot of a long list scrolled all the way to the bottom.
Revision: Thu Dec 7 14:26:37 2023 +0100 - midas-2022-05-c-762-g1eb9f627-dirty on branch
develop
Chrome on MacOSX 14.2.1
The fix is actually in "controls.js", so make sure your browser does not cache an old
version of that file. I usually have to clear my browser history to get the new file from
mhttpd.
Best regards,
Stefan |
Attachment 1: Screenshot_2024-01-17_at_08.17.30.png
|
|
2676
|
17 Jan 2024 |
Francesco Renga | Forum | History tags | Dear experts,
I would like to have some clarification about the meaning and use of the
tags in the ODB under /History/Tags.
I noticed that, if a history plot is created, but the name of the corresponding
variable is changed later and the plot is modified accordingly, the old name
persists in the /History/Tags list along with the new one. So, it appears in the
list of variables when a new history plot is created.
It seems not to compromise the functionalities of the history system, but it is
prone to create confusion.
Is it the expected behavior? What is the correct procedure to follow if the name
of a variable has to be changed?
Thank you,
Francesco |
2677
|
17 Jan 2024 |
Pavel Murat | Forum | a scroll option for "add history variables" window? | > Have you updated to the current midas version? This issue has been fixed a while ago.
Hi Stefan, thanks a lot! I pulled from the head, and the scrolling works now. -- regards, Pasha |
2679
|
18 Jan 2024 |
Andreas Suter | Forum | mhttpd eqtable | I have two more questions related to Units, Format for Equipment/Settings:
1) It looks as if I can have units per channel only for the Input/Output channels but not for Demand/Measured channels.
For instance we do have HV FE which collect devices with kV and V demand settings. It looks like this is not possible (see attachment) to have per channel units.
Is this right, or do I miss something here?
2) This new functionality needs entries under /Equipment/<eq-name>/Settings. The class driver generates the necessary structures if they are missing at the startup
of the scfe. It would be nice that the new, additional entries would be generate as well: Editable, Unit Input, Unit Format, etc. Perhaps optionally, if a DD is providing it?
Best,
Andreas |
Attachment 1: hv-web.png
|
|
2680
|
18 Jan 2024 |
Stefan Ritt | Forum | History tags | This part of the system has been designed by KO, so he should reply here.
Stefan |
2681
|
18 Jan 2024 |
Stefan Ritt | Forum | mhttpd eqtable | I fixed both in the current version, so please give it a try.
Stefan |
2686
|
28 Jan 2024 |
Pavel Murat | Forum | number of entries in a given ODB subdirectory ? | Dear MIDAS experts,
- I have a detector configuration with a variable number of hardware components - FPGA's receiving data
from the detector. They are described in ODB using a set of keys ranging
from "/Detector/FPGAs/FPGA00" .... to "/Detector/FPGAs/FPGA68".
Each of "FPGAxx" corresponds to an ODB subdirectory containing parameters of a given FPGA.
The number of FPGAs in the detector configuration is variable - [independent] commissioning
of different detector subsystems involves different number of FPGAs.
In the beginning of the data taking one needs to loop over all of "FPGAxx",
parse the information there and initialize the corresponding FPGAs.
The actual question sounds rather trivial - what is the best way to implement a loop over them?
- it is certainly possible to have the number of FPGAs introduced as an additional configuration parameter,
say, "/Detector/Number_of_FPGAs", and this is what I have resorted to right now.
However, not only that loooks ugly, but it also opens a way to make a mistake
and have the Number_of_FPGAs, introduced separately, different from the actual number
of FPGA's in the detector configuration.
I therefore wonder if there could be a function, smth like
int db_get_n_keys(HNDLE hdb, HNDLE hKeyParent)
returning the number of ODB keys with a common parent, or, to put it simpler,
a number of ODB entries in a given subdirectory.
And if there were a better solution to the problem I'm dealing with, knowing it might be helpful
for more than one person - configuring detector readout may require to deal with a variable number
of very different parameters.
-- many thanks, regards, Pasha |
2687
|
28 Jan 2024 |
Konstantin Olchanski | Forum | number of entries in a given ODB subdirectory ? | Very good question. It exposes a very nasty problem, the race condition between "ls" and "rm". While you are
looping over directory entries, somebody else is completely permitted to remove one of the files (or add more
files), making the output of "ls" incorrect (contains non-existant/removed files, does not contain newly added
files). even the simple count of number of files can be wrong.
Exactly the same problem exists in ODB. As you loop over directory entries, some other ODB client can remove or
add new entries.
To help with this, I considered adding an db_ls() function that would take the odb lock, atomically iterate over
a directory and return an std::vector<std::string> with names of all entries. (current odb iterator returns ODB
handles that may be invalid if corresponding entry was removed while we were iterating). Unfortunately the
delete/add race condition remains, some returned entries may be invalid or missing.
For your specific application, you can swear that you will never add/delete files "at the wrong time", and you
will not see this problem until one of your users writes a script that uses odbedit to add/remove subdirectory
entries exactly at the wrong time. (you run your "ls" in the BeginRun() handler of your frontend, they run their
"rm" from their's, so both run at the same time, a race condition.
Closer to your question, I think it is simplest to always iterate over the subdirectory, collect names of all
entries, then work with them:
std::vector<std::string> names;
iterate over odb {
names.push_back(name);
}
foreach (name in names)
work_on(name);
instead of:
size_t n = db_get_num_entries();
for (size_t i=0; i<n; i++) {
std::string name = sprintf("FPGA%d", i);
work_on(name);
}
K.O.
> Dear MIDAS experts,
>
> - I have a detector configuration with a variable number of hardware components - FPGA's receiving data
> from the detector. They are described in ODB using a set of keys ranging
> from "/Detector/FPGAs/FPGA00" .... to "/Detector/FPGAs/FPGA68".
> Each of "FPGAxx" corresponds to an ODB subdirectory containing parameters of a given FPGA.
>
> The number of FPGAs in the detector configuration is variable - [independent] commissioning
> of different detector subsystems involves different number of FPGAs.
>
> In the beginning of the data taking one needs to loop over all of "FPGAxx",
> parse the information there and initialize the corresponding FPGAs.
>
> The actual question sounds rather trivial - what is the best way to implement a loop over them?
>
> - it is certainly possible to have the number of FPGAs introduced as an additional configuration parameter,
> say, "/Detector/Number_of_FPGAs", and this is what I have resorted to right now.
>
> However, not only that loooks ugly, but it also opens a way to make a mistake
> and have the Number_of_FPGAs, introduced separately, different from the actual number
> of FPGA's in the detector configuration.
>
> I therefore wonder if there could be a function, smth like
>
> int db_get_n_keys(HNDLE hdb, HNDLE hKeyParent)
>
> returning the number of ODB keys with a common parent, or, to put it simpler,
> a number of ODB entries in a given subdirectory.
>
> And if there were a better solution to the problem I'm dealing with, knowing it might be helpful
> for more than one person - configuring detector readout may require to deal with a variable number
> of very different parameters.
>
> -- many thanks, regards, Pasha |
2689
|
28 Jan 2024 |
Konstantin Olchanski | Forum | History tags | > This part of the system has been designed by KO, so he should reply here.
That's right. Some of this stuff is historical gibberish that is no longer needed
for FILE and SQL histories.
/History/Events is needed to create persistent mapping between history event names
and history event id's (at some point history event id was same equipment event
id, with the obvious problems when equipment event ids are duplicated, reused,
renamed, deleted).
/History/Tags was used by the history editor to speed up "give me all tag names
for this history event name". With the "MIDAS" history storage this required
reading a lot of data from disk. With the "FILE" history and cached ZFS SSD, disk
access is much cheaper and caching history event names and tags in odb is no
longer necessary.
/History/Tags should probably be removed (be check that nobody uses it first).
/History/Events has to remain as long as "MIDAS" history storage is still used.
K.O. |
2690
|
28 Jan 2024 |
Konstantin Olchanski | Forum | a scroll option for "add history variables" window? | > > Have you updated to the current midas version? This issue has been fixed a while ago.
>
> Hi Stefan, thanks a lot! I pulled from the head, and the scrolling works now. -- regards, Pasha
Right, I remember running into this problem, too.
If you have some ideas on how to better present 100500 history variables, please shout out!
K.O. |
2691
|
28 Jan 2024 |
Konstantin Olchanski | Forum | dump history FILE files | $ cat mhf_1697445335_20231016_run_transitions.dat
event name: [Run transitions], time [1697445335]
tag: tag: /DWORD 1 4 /timestamp
tag: tag: UINT32 1 4 State
tag: tag: UINT32 1 4 Run number
record size: 12, data offset: 1024
...
data is in fixed-length record format. from the file header, you read "record size" is 12 and data starts at offset 1024.
the 12 bytes of the data record are described by the tags:
4 bytes of timestamp (DWORD, unix time)
4 bytes of State (UINT32)
4 bytes of "Run number" (UINT32)
endianess is "local endian", which means "little endian" as we have no big-endian hardware anymore to test endian conversions.
file format is designed for reading using read() or mmap().
and you are right mhdump, does not work on these files, I guess I can write another utility that does what I just described and spews the numbers to stdout.
K.O. |
2692
|
28 Jan 2024 |
Konstantin Olchanski | Forum | slow control frontends - how much do they sleep and how often their drivers are called? | > I have implemented a number of slow control frontends which are directed to update the
> history once in every 10 sec, and they do just that.
I suggest that you switch from the old mfe.c frontend framework to the new tmfe framework that was
designed to solve exactly this type of problems.
Look at .../midas/progs/tmfe_example*.cxx
You have a choice of:
- single threaded frontend, most robust, no race conditions, but readout is interrupted during
begin/end of run.
- two-threaded frontend, your periodic equipments run in one thread, midas loop and rpc run in a
different thread, you have to handle locking yourself.
- you can run each of your equipments in it's own thread without help from the framework, it is
obvious how to do it if you can program c++ threads, "new std::thread" to create/start a thread,
stop threads using a binary flag, thread->join() to reap them at the end (or thread sanitizer will
complain).
K.O. |
2693
|
28 Jan 2024 |
Konstantin Olchanski | Forum | the logic of handling history variables ? | MIDAS history is very simple:
from your frontend, your write your history data to ODB /eq/xxx/variables (see below)
mlogger has a hotlink to all /eq/*/variables and it will "see" the new data, write it to history file (see below)
you should see the history file grow using "ls"
history web page in your browser sends a "give me more data" JSON-RPC request to mhttpd
mhttpd looks at the history file, if there is new data (file got bigger) it send it to the web page
web page shows the new data.
where things usually go wrong:
- mlogger only looks for new history variables on startup and on begin-of-run. if you add new stuff in your frontend, you
will not see it until you restart mlogger or start a new run.
- mlogger only looks at history data if corresponding "/eq/xxx/common/log history" is non-zero. for best effect, set it to
"1". (or "0" to turn history off).
- history file is not growing, likely mlogger does not "see" your new data
- timestamps of stuff in /eq/xxx/variables are not getting updated, likely frontend is not writing them, and there is no
new data for mlogger to "see" and write to file.
Frontend has several ways of writing to /eq/xxx/variables:
- write to ODB directly using ODB API db_set_data(), mvodb->Wx(), etc. this is the most foolproof method. use in
conjunction with a printf() statement to make sure you actually do write to ODB. Sometimes your frontend event loop fails
to run, a bug/failure that has nothing to do with midas history.
- generate a midas event and set the per-equipment "write event to ODB" flag (RO_ODB for mfe.c frontends), the mfe/tmfe
framework will write event data to ODB, each data bank will be written to /eq/xxx/variables/BANKNAME, data type is taken
from the event data bank definition.
This second method sometimes malfunctions, typical problems are missing RO_ODB in the equipment table, equipment table in
ODB overwriting the value in source code (this is confusing in mfe.c frontends).
Least likely failure is "/eq/xxx/common/log history" set to bogus value. Normal values are 0=history disabled, 1=history
enabled, other values are only needed if you do not want mlogger to record history as often as you generate it, i.e. you
update /eq/xxx/variables every 1/sec, but you want mlogger to only record it 1/minute.
I hope this helps.
P.S. I notice your equipment tables do not have RO_ODB, so if you use the 2nd method, write history via event data banks,
it will not work.
K.O.
> Dear MIDAS developers,
>
> I'm trying to understand handling of the history (slow control) variables in MIDAS,
> and it seems that the behavior I'm observing is somewhat counterintuitive.
> Most likely, I just do not understand the implemented logic.
>
> As it it rather difficult to report on the behavior of the interactive program,
> I'll describe what I'm doing and illustrate the report with the series of attached
> screenshots showing the history plots and the status of the run control at different
> consecutive points in time.
>
> Starting with the landscape:
>
> - I'm running MIDAS, git commit=30a03c4c (the latest, as of today).
>
> - I have built the midas/examples/slowcont frontend with the following modifications.
> (the diffs are enclosed below):
>
> 1) the frequency of the history updates is increased from 60sec/10sec to 6sec/1sec
> and, in hope to have updates continuos, I replaced (RO_RUNNING | RO_TRANSITIONS)
> with RO_ALWAYS.
>
> 2) for convenience of debugging, midas/drivers/nulldrv.cxx is replaced with its clone,
> which instead of returning zeroes in each channel, generates a sine curve:
>
> V(t) = 100*sin(t/60)+10*channel
>
> - an active channel in /Logger/History is chosen to be FILE
>
> - /History/LoggerHistoryChannel is also set to FILE
>
> - I'm running mlogger and modified, as described, 'scfe' frontend from midas/examples/slowcont
>
> - the attached history plots include three (0,4 and 7) HV:MEASURED channels
>
>
> Now, the observations:
>
> 1) the history plots are updated only when a new run starts, no matter how hard
> I'm trying to update them by clicking on various buttons.
>
> The attached screenshots show the timing sequence of the run control states
> (with the times printed) and the corresponding history plots.
>
> The "measured voltages" change only when the next run starts - the voltage graphs
> break only at the times corresponding to the vertical green lines.
>
> 2) No matter for how long I wait within the run, the history updates are not happening.
>
> 3) if the time difference between the two run starts gets too large,
> the plotted time dependence starts getting discontinuities
>
> 4) finally, if I switch the logging channel from FILE to MIDAS (activate the MIDAS
> channel in /Logger/History and set /History/LoggerHistoryChannel to MIDAS),
> the updates of the history plots simply stop.
>
> MIDAS feels as a great DAQ framework, so I would appreciate any suggestion on
> what I could be doing wrong. I'd also be happy to give a demo in real time
> (via ZOOM/SKYPE etc).
>
> -- much appreciate your time, thanks, regards, Pasha
>
> ------------------------------------------------------------------------------
> diff --git a/examples/slowcont/scfe.cxx b/examples/slowcont/scfe.cxx
> index 11f09042..c98d37e8 100644
> --- a/examples/slowcont/scfe.cxx
> +++ b/examples/slowcont/scfe.cxx
> @@ -24,9 +24,10 @@
> #include "mfe.h"
> #include "class/hv.h"
> #include "class/multi.h"
> -#include "device/nulldev.h"
> #include "bus/null.h"
>
> +#include "nulldev.h"
> +
> /*-- Globals -------------------------------------------------------*/
>
> /* The frontend name (client name) as seen by other MIDAS clients */
> @@ -74,11 +75,11 @@ EQUIPMENT equipment[] = {
> 0, /* event source */
> "FIXED", /* format */
> TRUE, /* enabled */
> - RO_RUNNING | RO_TRANSITIONS, /* read when running and on transitions */
> - 60000, /* read every 60 sec */
> + RO_ALWAYS, /* read when running and on transitions */
> + 6000, /* read every 6 sec */
> 0, /* stop run after this event limit */
> 0, /* number of sub events */
> - 10000, /* log history at most every ten seconds */
> + 1000, /* log history at most every one second */
> "", "", ""} ,
> cd_hv_read, /* readout routine */
> cd_hv, /* class driver main routine */
> @@ -93,8 +94,8 @@ EQUIPMENT equipment[] = {
> 0, /* event source */
> "FIXED", /* format */
> TRUE, /* enabled */
> - RO_RUNNING | RO_TRANSITIONS, /* read when running and on transitions */
> - 60000, /* read every 60 sec */
> + RO_ALWAYS, /* read when running and on transitions */
> + 6000, /* read every 6 sec */
> 0, /* stop run after this event limit */
> 0, /* number of sub events */
> 1, /* log history every event as often as it changes (max 1 Hz) */
> ------------------------------------------------------------------------------
> [test_001]$ diff ../midas/examples/slowcont/nulldev.cxx ../midas/drivers/device/nulldev.cxx
> 13d12
> < #include <math.h>
> 150,154c149,150
> < if (channel < info->num_channels) {
> < // *pvalue = info->array[channel];
> < time_t t = time(NULL);;
> < *pvalue = 100*sin(M_PI*t/60)+10*channel;
> < }
> ---
> > if (channel < info->num_channels)
> > *pvalue = info->array[channel];
> ------------------------------------------------------------------------------ |
2694
|
28 Jan 2024 |
Stefan Ritt | Forum | number of entries in a given ODB subdirectory ? | I guess you won't change your FPGA configuration just when you start a run, so I don't consider the race
condition very crucial (although KO is correct, it it there).
I guess rather than any pseudo code you want to see real working code (db_get_num_entries() does not exist!), right?
The easiest these day is to ask ChatGPT. MIDAS has been open source since a long time, so it has been used
to train modern Large Language Models. Attached is the result. Here is the direct link from where you can
copy the code:
https://chat.openai.com/share/d927c78d-9914-4413-ab5e-3b0e5d173132
Please note that you never can be 100% sure that the code from a LLM is correct, so always compile and debug it.
But nevertheless, it's always easier to start from some existing code, even if there is a danger that it's not perfect.
Best,
Stefan |
Attachment 1: Screenshot_2024-01-29_at_07.20.50.png
|
|
|