ID |
Date![down down](down.png) |
Author |
Topic |
Subject |
2045
|
30 Nov 2020 |
Konstantin Olchanski | Info | more wisdom from linux kernel people | As you may know, I am a big fan of two software projects - the linux kernel and ROOT. The linux kernel is one of
the few software projects "done right". ROOT is where normal people try to "get it right" with real-world level
of success. I use both softwares daily and I try to apply their ways and methods to MIDAS as much as I can.
So just in time for our discussion of array indexes, a talk by gregkh shows
up on slashdot. The title is "how to keep your users happy". (Nobody
ever wants to be nasty to their users, but do read his talk).
https://git.sr.ht/~gregkh/presentation-application_summit/tree/main/keep_users_happy.pdf
The talk refers to some older stuff, still relevant, of course, in case you miss the links
in the pdf file, here they are:
https://ozlabs.org/~rusty/index.cgi/tech/2008-03-30.html
https://ozlabs.org/~rusty/index.cgi/tech/2008-04-01.html
https://ozlabs.org/~rusty/ols-2003-keynote/img0.html (click on "continue" to see next page)
K.O. |
2044
|
30 Nov 2020 |
Konstantin Olchanski | Suggestion | ODBSET wildcards with array keys in Sequencer files | >
> we are considering to make the range selection uniform among json, sequencer and
> odbedit "set" command. Having multiple ranges like [1,4-5] will be quite some work, so
> my question is did you just implement it on the json side because it was easy, or are
> there experiments who really need it? Wouldn't it be enough to have
>
> [*]
> [n]
> [n-m]
>
It has been a long time, but most likely I designed the interface to work this
way to permit maximum flexibility for writing into an array using just one
rpc operation.
The generalized form is:
[range,range,range...]
where range is:
array index or
index1-index2 or
index2-index1 (write in reverse order)
This is all documented here:
https://midas.triumf.ca/MidasWiki/index.php/Mjsonrpc#Supported_array_index_syntax
I think it is too late to change it.
>
> This way we always have only one db_set_data() value behind that. Any set of indices
> we have to split into several db_set_data()
>
Sounds good. I think it is easy to have a common implementation:
One would need following functions:
parse range selection from string into std::vector<int> of array indices (we already have it)
call db_set_data_range() (this is easy to add).
>
> trouble by triggering a hot link on each access.
>
There is no escaping this trouble. Half the experiments want notification
"per array", the other half, "per array element". We cannot choose one or the other for them,
we have to provide a way for the user to say how they want it.
P.S. With existing ODB calls, you cannot do [n-m] ranges. You can do whole array or you
can do one-element-at-a-time.
K.O. |
2043
|
30 Nov 2020 |
Konstantin Olchanski | Suggestion | ODBSET wildcards with array keys in Sequencer files | > I totally agree that we should have a consistent formatting for array index expansion.
> I had a look to the mjsonrpc code and I found the function parse_array_index_list(...) which does this job.
Yes, it is good to review this stuff. I think the json-rpc call should accept more array index patterns:
a[*] - whole array (even though it is unnatural use in javascript, we do not say "let a[*] = b[*]", we say "let a = b".
a[5-] - from 5th element to the end (in the case we do not know the length)
a[-10] - from first element to element 10 (this is same as a[0-10], but needed for consistency with previous case).
K.O.
> I have a similar function (adapted form previous code) in odb.cxx called strarrayindex(...) that is designed for the same "consistency" purposes between odbedit and sequencer.
>
> Let me put few points that I noticed:
> - mjsonrpc has a very different way to write the full array (no indexes given) while currently sequencer requires "[*]" to do the same (otherwise it only changes the first value of the array)
> - currently the sequencer and the underlying ODB calls use two indexes (that are the same if you want to write only one key) so we will need a serious rewriting to allow something like "ODBSET array[1,3,5]"
> - if I correctly understood the code, mjsonrpc instead generates a list of indices and then calls an ODB write on each of them. That's not always a good thing, for example if you are writing an array of n parameters on a DAQ
> board you will call the hotlink on that key n times
> - in addition to that the sequencer will also have to cope with variable-based indexes like "ODBSET array[$val]", but then how it should parse something like "[$a,1]" or "[$a*]"?
>
> For the very first point I do not see a clean way to do this without breaking the compatibility of existing sequencers or having a difference between the two implementations.
> For the others I guess we can find a way out, however that's a major modification so I will put it on my todo list when I can find some free time.
> In any case I would propose to merge the two functions, so we have only to maintain a single implementation of the parsing.
>
> I guess it's a good moment to brainstorm about that, let me know what you think
>
> Marco
>
>
> > > The following all fail with "Cannot find ODB key "<key>""
> > >
> > > ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)[*]" 0
> > > ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)[0-9]" 0
> > > ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)[1]" 0
> > > ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)*" 0
> > > ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)" 0
> > >
> >
> > It would be cool if ODB pattern matching in the sequencer
> > were consistent with the ODB pattern matching in the json-rpc
> > interface for web pages:
> >
> > https://midas.triumf.ca/MidasWiki/index.php/Mjsonrpc#Supported_array_index_syntax
> >
> > K.O. |
2042
|
30 Nov 2020 |
Konstantin Olchanski | Info | Equipment "common" settings in ODB | > One more change:
>
> After using the new code for some hours, we realized that the "enabled" flag should not come from the frontend code,
> but always be defined by the ODB. So if you quickly have to disable some equipment because the associated hardware is
> off, you want to change this flag only in the ODB and not have to recompile the frontend. So we exclude that flag from
> being set by the frontend. It is anyhow special, because one sees all disable equipment in the main midas status page,
> so one knows what's on and what's off.
>
> Please comment here if you think that change causes problem. Anyhow it's working now for the enabled flag as before
> all these changes.
>
Good catch. I still think this is fundamentally impossible to "get right". But good, you
are now in the same boat with me. The documentation will read: "if flag is TRUE, these data fields
are read from ODB, if flag is FALSE, those other fields are read from ODB". I will have to check
how this will work out for the TMFE C++ frontend (I think both mfe.c and TMFE frontends should
work "the same").
I think we have at least one month to play with this, I do not think we can do the next release
of midas until January.
K.O. |
2041
|
30 Nov 2020 |
Stefan Ritt | Info | Equipment "common" settings in ODB | One more change:
After using the new code for some hours, we realized that the "enabled" flag should not come from the frontend code,
but always be defined by the ODB. So if you quickly have to disable some equipment because the associated hardware is
off, you want to change this flag only in the ODB and not have to recompile the frontend. So we exclude that flag from
being set by the frontend. It is anyhow special, because one sees all disable equipment in the main midas status page,
so one knows what's on and what's off.
Please comment here if you think that change causes problem. Anyhow it's working now for the enabled flag as before
all these changes.
Stefan |
2040
|
30 Nov 2020 |
Stefan Ritt | Suggestion | ODBSET wildcards with array keys in Sequencer files | Hi Konstantin,
we are considering to make the range selection uniform among json, sequencer and
odbedit "set" command. Having multiple ranges like [1,4-5] will be quite some work, so
my question is did you just implement it on the json side because it was easy, or are
there experiments who really need it? Wouldn't it be enough to have
[*]
[n]
[n-m]
This way we always have only one db_set_data() value behind that. Any set of indices
we have to split into several db_set_data(), which especially for the front-end
configuration can cause trouble by triggering a hot link on each access.
Stefan |
2039
|
30 Nov 2020 |
Stefan Ritt | Info | Equipment "common" settings in ODB | Ok, I implemented it the following way:
- Added a boolean flag "equipment_common_overwrite", which must be contained in EACH frontend, preferably just
before the EQUIPMENT structure, such as:
BOOL equipment_common_overwrite = TRUE;
EQUIPMENT equipment[] = {
...
};
- If that flag is TRUE, then the contents of the "equipment" structure is copied to the ODB on each start of the
front-end
- If the flag is FALSE, then the ODB values are kept on the start of the front-end
The setting of the flag depends now on the philosophy of the experiment. Some experiments say that everything
needed should be in the front-end code, so when it starts everything gets set correctly. They don't change the
values in the ODB, but in the frontend code, which then goes into their repository. Other experiments just need
some default values from the frontend code, and the fine-tune things by changing values in the ODB. These
experiments should set this flag to FALSE.
*****
Please note that EVERY frontend now needs this flag, so all of you have to add it to all of your front-ends,
otherwise the front-end will not compile! I could not figure out how to this could be done without this
requirement, since you can define a global variable only once.
*****
Stefan |
2038
|
30 Nov 2020 |
Marco Francesconi | Suggestion | ODBSET wildcards with array keys in Sequencer files | I totally agree that we should have a consistent formatting for array index expansion.
I had a look to the mjsonrpc code and I found the function parse_array_index_list(...) which does this job.
I have a similar function (adapted form previous code) in odb.cxx called strarrayindex(...) that is designed for the same "consistency" purposes between odbedit and sequencer.
Let me put few points that I noticed:
- mjsonrpc has a very different way to write the full array (no indexes given) while currently sequencer requires "[*]" to do the same (otherwise it only changes the first value of the array)
- currently the sequencer and the underlying ODB calls use two indexes (that are the same if you want to write only one key) so we will need a serious rewriting to allow something like "ODBSET array[1,3,5]"
- if I correctly understood the code, mjsonrpc instead generates a list of indices and then calls an ODB write on each of them. That's not always a good thing, for example if you are writing an array of n parameters on a DAQ
board you will call the hotlink on that key n times
- in addition to that the sequencer will also have to cope with variable-based indexes like "ODBSET array[$val]", but then how it should parse something like "[$a,1]" or "[$a*]"?
For the very first point I do not see a clean way to do this without breaking the compatibility of existing sequencers or having a difference between the two implementations.
For the others I guess we can find a way out, however that's a major modification so I will put it on my todo list when I can find some free time.
In any case I would propose to merge the two functions, so we have only to maintain a single implementation of the parsing.
I guess it's a good moment to brainstorm about that, let me know what you think
Marco
> > The following all fail with "Cannot find ODB key "<key>""
> >
> > ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)[*]" 0
> > ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)[0-9]" 0
> > ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)[1]" 0
> > ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)*" 0
> > ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)" 0
> >
>
> It would be cool if ODB pattern matching in the sequencer
> were consistent with the ODB pattern matching in the json-rpc
> interface for web pages:
>
> https://midas.triumf.ca/MidasWiki/index.php/Mjsonrpc#Supported_array_index_syntax
>
> K.O. |
2037
|
27 Nov 2020 |
Konstantin Olchanski | Info | Equipment "common" settings in ODB | Yes, I think this will work.
For old mfe.c frontends, global variable set to "do it the new way" should be okey,
new experiments will have it the new way. Old experiments, will be forced to add a one-line definition
of this global variable (otherwise mfe.o will not link), at that time they get to chose "new way" or "old way".
For the new TMFE c++ frontend, this will work naturally when they create the Equipment Common object,
in the object constructor, you can see how it explicitly honors or overwrites the ODB common entries.
The TMFE frontend does not do a live "period", so there should be no issue with that.
Should I open a bitbucket issue "update TMFE frontend to new Equipment/Common scheme", to make sure
I do not forget about it?
K.O. |
2036
|
27 Nov 2020 |
Stefan Ritt | Info | Equipment "common" settings in ODB | Ok, so what about the following proposal:
- I change back the mfe.cxx code to behave like before (ODB has precedence and does not get overwritten when the
front-end restarts)
- I add a global flag
BOOL equipment_common_overwrite;
and pre-set it to FALSE;
- So if nothing is changed the flag stays false and ODB keeps precedence
- If a frontend wants to overwrite equipment/common on each start, the user sets
BOOL equipment_common_overwrite = TRUE;
near the equipment[] structure in the front-end code.
- If the flag is true, the mfe.cxx init code copies the equipment[] structure to the ODB on each frontend start
I believe this way we can keep backward compatibility, and add the new way with minimal effort. The only downside
is that all frontends on this plane have to add at least "BOOL equipment_common_overwrite = FALSE;" in their
code.
I know global variables are evil, but this way the user can just add the line above to the equipment[] array, so
one sees this when one edits the equipment[] array, giving motivation to change as needed. So the code would be
BOOL equipment_common_overwrite = TRUE;
EQUIPMENT equipment[] = {
....
}
An alternative way would be to add a function
set_equipment_common_overwrite(TRUE);
into the frontend_init() code. That's somehow cleaner (still needs an internal global variable), but it has to go
into frontend_init() so won't be at the same place as the EQUIPMENT list in the frontend.
Thoughts?
Best,
Stefan |
2035
|
27 Nov 2020 |
Konstantin Olchanski | Forum | Invalid name "Analyzer/Tests" | https://bitbucket.org/tmidas/midas/issues/298/invalid-odb-names-in-example-midas
K.O. |
2034
|
27 Nov 2020 |
Konstantin Olchanski | Suggestion | cmake build fixes |
Hi, Alexandr, thank you for making improvements to MIDAS. I have some question
about your suggestions:
> there are several problems with current cmake build files in midas:
> - not all systems have cuda libs in /usr/local/cuda
> - not all cmake version like when redefining vars
we do not see these problems with the normal cmake on our current linux systems,
centos-7 and -8, Ubuntu LTS 18.04, 20.04.
so you have something different? can you be a bit more specific,
which version of cmake and which OS you have so see these troubles?
> - c++ standard not matching the one used to build ROOT
> - ROOTSYS is not needed to find ROOT (it is enough to have root in PATH)
Again ROOT tangles with the build of MIDAS.
MIDAS does not use ROOT. As a convenience to the users, we have a "ROOT output" driver
in mlogger and we build a special executable rmlogger with ROOT. Only this special
executable should be linked with ROOT and compiled with ROOT-specific flags.
The rest of the MIDAS build should not be affected by presence or absence of ROOT.
One would have to read old messages on this forum to understand this situation.
>
> I have posted pull request 'https://bitbucket.org/tmidas/midas/pull-requests/17'
> which tries to fix some of the problems.
> Tests and comments are welcome.
>
I look at the diffs:
- CUDA detection is changed to "find_package(CUDA)". This code was added by Joseph and Ben, and there
must be a reason why they did not use find_package(CUDA). They will have to sign-off on this change.
- ROOT related logic assumes that all of MIDAS will be built "the ROOT way". CFLAGS are changed, the C++
standard is changed, etc. this assumption is wrong. only rmlogger and rmana should be built "with ROOT".
If you want to follow through on this, I suggest that you split the pull request into two,
one pull request for the CUDA changes and one pull request for the ROOT changes. Also rework
your ROOT changes as I explained above (but also read all ROOT-related messages on this forum).
K.O. |
2033
|
27 Nov 2020 |
Konstantin Olchanski | Forum | Building an experiment using CAEN VME interface - unknown type name 'VARIANT_BOOL' | >
> The header file used to defined the CAEN types (CAENVMEtypes.h) defines 'CAEN_BOOL' like this:
>
>
> #ifdef LINUX
> #define CAEN_BYTE unsigned char
> #define CAEN_BOOL int
> #else
> #define CAEN_BYTE byte
> #define CAEN_BOOL VARIANT_BOOL
> #endif
>
Complain to CAEN.
The year is 2020 and they should use standard C/C++ data types from stdint.h (uint32_t, etc).
K.O. |
2032
|
27 Nov 2020 |
Konstantin Olchanski | Info | Equipment "common" settings in ODB | > Today I addressed a topic which bugged me since long time.
Right. No easy subject. For me, too, this has been a problem in MIDAS for a long time.
> Now, on each start of the frontend program, the values from equipment[] are written to
> the ODB. They are still "live". If one changes them when the frontend is
> running, that change takes effect immediately. But on the next restart of the
> frontend, the old values from equipment[] is put back there.
There is a downside from this behaviour.
If some values in equipment/common are "live" and the user is expected to change them,
the user will be unpleasantly surprised when their changes magically disappear (after reboot,
after frontend crash, after run restart if experiment requires restarting some frontends
before starting a new run).
This change will also break some experiments that rely in things like specifying
event buffer names through ODB. But experiments can adapt and specify buffer names
through command line switch instead of ODB.
This new way also it makes the "live" Common/Period unusable. Sure I can speed up or slow
down a frontend even during the run, but if my change does not "stick", what good is it?
Personally, I think there is no easy solution for all these troubles.
I would advocate the following approach:
- think of MIDAS as a "mature" system,
- treasure backward compatibility
- (if we must break backward compatibility to introduce a new "must have" improvement, so be it)
- document how things work. if it is clearly written down what different fields in "common" do, fewer people
"get burned" by unexpected or illogical things. (and any non-trivial system has plenty of those).
Going back to ODB equipment/common, my experience with midas and odb tells me
that one should avoid mixing together ODB entries set by user and ODB entries set by code.
For example, separating them as equipment/settings and equipment/variables works well. Mixing
them as in equipment/common and sequencer/state causes trouble.
So perhaps we should split Equipment/common into two pieces, user settable fields like
"Period" and "event buffer name" would move to equipment/settings or whatever.
This will open the discussion of which items in equipment/common should be user settable,
and some people would want event buffer specified in the code to prevail, while other
people would want the name from odb to prevail, and both are valid but conflicting preferences.
Or we could bite the bullet and say, equipment/common is controlled by the frontend code,
the user should not change it. (and mark it read-only in ODB).
For all the pain this may cause, at least this will make it self-consistent.
Per this proposal, in addition to Stefan's change, the hotlink on equipment/common goes away,
"period" is no longer "live" and the whole subdirectory is made "read-only".
K.O. |
2031
|
27 Nov 2020 |
Konstantin Olchanski | Forum | History plot consuming too much memory | >
> Taking this down a tangent, I have a mild concern that a user could temporarily
> flood our gigabit network if we do have faster disks to read the history data.
>
By my measurements, right now our javascript code can reach 30-50-70% of Gige ethernet
bandwidth, so, no, we cannot flood the network just by making history plots.
(we cannot reach 100% because javascript code is not multithreaded,
it cycles through "request new data" and "decode javascript, make plot" states,
and the network is idle in this second state).
>
> Have there been any plans or thoughts on limiting the bandwidth users can pull from
> mhttpd?
>
10gige networking is here (and 5 and 2.5 Gige, too). I would not worry too much
about saturating 1gige network interfaces.
>
> I do not see this as a critical item as I can plan the future network
> infrastructure at the same time as the next system upgrade (putting critical data
> taking traffic on a separate physical network).
>
10gige network between all computers, everything on SSD ZFS arrays, except
bulk data on ZFS HDD arrays (only for cost reasons $$$/TB).
K.O. |
2030
|
27 Nov 2020 |
Konstantin Olchanski | Forum | History plot consuming too much memory | >
> Are you sure that the delay comes from the browser or actually from mhttpd
> digging through GBytes of history data?
>
I think we will need to address this question "head-on". The history plot
will need to display the following information:
"time to load data from disk: N seconds, time to transfer data to javascript: M
seconds, time to make the plot: Q seconds".
The second and third items are already available, the first one will need
to be computed in mhttpd and passed to javascript.
K.O. |
2029
|
27 Nov 2020 |
Konstantin Olchanski | Forum | History plot consuming too much memory | >
> With the current code, we are limited to loading history data up to 50% of
> the javascript memory limit.
>
The javascript memory limit itself seems to be a moving target. (google javascript
memory limit, and good luck!).
Historically, javascript did not have any memory or cpu use limits, but with
the raise of abusive web sites, bitcoin miners, etc, I see browsers clamp down
on allowed/allocated CPU use (inactive tabs are throttled down). memory use
is already clamped down severely, on a 64 GB computer, a browser tab
can only allocate a handful of GBs.
This throttling of browser tabs is already intrusive enough that we need
to be careful in programming midas web pages. for examples throttled events
are not firing at the same rate or in the same order as in active tabs.
One logical conclusion of these restrictions could be that, eventually,
google-chrome permits only just enough cpu and memory to run gmail.
K.O. |
2028
|
27 Nov 2020 |
Konstantin Olchanski | Forum | History plot consuming too much memory | >
> Tested with midas-2020-08-a up until the HEAD of develop
>
Just so you know, it took myself and Stefan quite a bit of effort
to improve memory and data handling in the new history plots
to be able to plot 1 year of data without bogging down too much. I got
to learn the google-chrome javascript cpu profiler, memory profiler
and the intricacies of javascript shift() and unshift() operators.
Before midas-2020-08-a, pressing the zoom-out button you would never
reach the javascript memory limit, the code would go into "100% cpu use"
and the browser tab will become progressively unresponsive well before
running out of memory. With the original code, our alpha-g history plots
could go back a few weeks at most, with the current code, we can go back
about 11 months. Compared to the old "C" history plots that can
do "last 10 years", no problem.
Loading all the history data into the browser is a design choice.
It has benefits and downsides.
The main benefit is that looking at immediate live data is much easier.
The main downside is that "plot last 10 years" becomes impossible.
As they say "appetite comes during eating", we have learned about these
downsides as we developed the new system. When we started, we did not
know much about javascript memory limits, cpu limits, etc. We did learn
a lot, though.
With the current code, we are limited to loading history data up to 50% of
the javascript memory limit. I know how to change the code to get up to 100%,
but I think it is not worth it, it still does not get as to plot "last 10 year".
We think the solution to recovering "last 10 years" capability is to use
binned data (which the history system can already deliver to javascript).
With binned data, the data volume in Mbytes remains constant, javascript
memory use has an upper-bound (we never use more memory than X Mbytes)
and data movement over the network is reduced.
Another way to look at this - typical display has only 1000-4000 vertical pixels,
it cannot physically display a bigger number of data points (no more
then 1 data point per pixel). So why load 1000000 data points when we only
can plot 1000-4000 of them?
So all the infrastructure for plotting binned data is already there,
but the javascript code still needs to be written. I think the biggest
challenge will be in blending or combining binned and unbinned data
on the same plot or in seamlessly switching the plot between binned and
unbinned data.
K.O. |
2027
|
27 Nov 2020 |
Konstantin Olchanski | Forum | Invalid name "Analyzer/Tests" | > I've recently took the analyzer template from $MIDASSYS/examples/experiment and
> modified it to be able to use Roody on a very simple frontend setup.
Hmm... the old midas analyzer framework is very old and I do not recommend
to use it for new experiments.
A newer analyzer system is ROOTANA and an even newer is the "m" analyzer (manalyzer). These
analyzers progressively introduce improved c++-style programming environments amongst other
improvements. If starting from scratch, I recommend that you use manalyzer (currently from the rootana
git repository).
> The analyzer works fine and I am able to view the online histograms but my console
> prints out this error:
>
> [Analyzer,ERROR] [odb.cxx:919:db_validate_name,ERROR] Invalid name
> "/Analyzer/Tests/Always true/Rate [Hz]" passed to db_create_key_wlocked: should
> not contain "["
The error says what it means. "[" is not a permitted character in odb names. It is used
by many odb functions to access array elements.
The midas analyzer example should be updated to change "[Hz]" to "(Hz)" or something similar.
K.O. |
2026
|
27 Nov 2020 |
Konstantin Olchanski | Suggestion | ODBSET wildcards with array keys in Sequencer files | > The following all fail with "Cannot find ODB key "<key>""
>
> ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)[*]" 0
> ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)[0-9]" 0
> ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)[1]" 0
> ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)*" 0
> ODBSET "/Detectors/Det*/Settings/Charge/Bias (V)" 0
>
It would be cool if ODB pattern matching in the sequencer
were consistent with the ODB pattern matching in the json-rpc
interface for web pages:
https://midas.triumf.ca/MidasWiki/index.php/Mjsonrpc#Supported_array_index_syntax
K.O. |
|