| ID |
Date |
Author |
Topic |
Subject |
|
1816
|
10 Feb 2020 |
Konstantin Olchanski | Info | Force triggering of idle routine of a frontend | > We had a case where a detector produced electrostatic discharges which only lasted for a second or so
> and we were happy to detect this in spikes in the HV current. With measurements of only one per minute
> we would not have realized that so quicky.
For the T2K/ND280 TPC we implemented something similar. The TPC uses MicroMegas detector which sparks during
normal operation. We asked Wiener/ISEG to implement a "spark counting mode" for us (and they did). In this mode,
high voltage over-current (a micromegas spark) sets a special flag (does not trip the high voltage). Our midas frontend
reads this flag at rate about 1/min, if flag is set, clears it, increments the software spark counter, reads the flag again,
if the flag is still set (failed to clear), it means this was not a normal spark but a high voltage breakdown
and the offending channel is shut down. I believe this mode is still part of the ISEG normal firmware.
Because the Wiener/ISEG interface uses SNMP to "read all data in one operation", the MIDAS "device driver" structure
was not useful, the readout was a simple loop, the readout frequency was easy to control, and indeed,
we read the high voltage with increased frequency during ramping. This was easy to implement because we
did not have to fight the MIDAS "device driver" framework.
If you want a similar solution, talk to the device, interpret the data, record values to odb and history, generate
midas events - all without hand holding from (arm wrestling with the rest of) midas - I recommend
the new tmfe.h/tmfe.cxx c++ frontend - see the two examples in midas/progs/fetest_tmfe.cxx
and fetest_tmfe_thread.cxx (single-threaded and multi-threaded).
K.O. |
|
1821
|
12 Feb 2020 |
Stefan Ritt | Info | Force triggering of idle routine of a frontend | I had a look again at the issue. If you sett the event limit to zero in the EQUIPMENT list, then the idle() routine of your class driver is called as often as possible. Typically with 100 Hz. It's then up to you what to do in the class driver. The hv_idle() routine of the HV class driver shipped in the distribution for example read a channel more often if it has been changed recently. Look at the lines
/* additionally read channel recently updated if not multithreaded */
if (!(hv_info->driver[hv_info->last_channel]->flags & DF_MULTITHREAD)) {
act_time = ss_millitime();
act = (hv_info->last_channel_updated + 1) % hv_info->num_channels;
while (!(act_time - hv_info->last_change[act] < 10000)) {
act = (act + 1) % hv_info->num_channels;
if (act == hv_info->last_channel_updated) {
/* non found, so return */
return status;
}
}
/* updated channel found, so read it additionally */
status = hv_read(pequipment, act);
hv_info->last_channel_updated = act;
}
You can do similar things there like if you are ramping. On an end-of-run, the class drivers cd_xx_read() routine is called from the framework, which in turn sends a full midas event down the stream, but getting the current slow control values from its local cache, not from the actual device (otherwise stopping a run could be very slow). So if you want all values at the end of the run with good precision, you have to read them DURING the run as fast as possible. That's why I posted my comment about fixing dropped serial connections automatically and reading as fast as possible.
Stefan
| Pintaudi Giorgio wrote: | Dear Stefan,
Thank you for the advice. I will try to modify the driver as you say. As for the dynamical change of readout rate, basically you are telling me that is not achievable without dirty hacks like mine and it is better to find a way to avoid it.
|
|
|
1851
|
10 Mar 2020 |
Konstantin Olchanski | Info | MIDAS vs JSROOT web pages | Just FYI, I am looking at the ROOT web programming component JSROOT and I notice that the RPC mechanism quite different from the JSON-
RPC I implemented for MIDAS.
https://github.com/root-project/jsroot/blob/master/docs/HttpServer.md (explanation of JSROOT RPC and server side machinery)
https://github.com/root-project/jsroot/blob/master/docs/JSROOT.md (explanation of JSROOT javascript library)
Then I looked at the dates:
MIDAS mjsonrpc was done at the end of 2013
JSROOT main development started at the end of 2014.
The web server component in both projects is (almost) the same - vanilla mongoose in mhttpd
and civetweb, a fork of an older version of mongoose, in ROOT/JSROOT.
The web server in both projects is partially multithreaded:
- ROOT THttpServer/TCivetWeb uses multiple threads to handle the network connections and some file access,
but interaction with ROOT is done in the main thread of ROOT. (The main thread must periodically call ProcessRequests()).
- mhttpd uses a single thread to multiplex the network connections (it is a change from old mongoose/civetweb to current mongoose 6.16),
but all requests are farmed to a pool of threads and execute in parallel (unless not thread-safe, i.e. accessing history files).
Both implementations suffer from "head of queue" blocking, a "slow" request i.e. a slow file read, will
delay subsequent quick requests, see https://en.wikipedia.org/wiki/Head-of-line_blocking#In_HTTP
Solution for this problem is to use HTTP/2 when it becomes supported in mongoose/civetweb/apache httpd (in el7).
It will be interesting to see which on of the two systems works better for building "user facing" web pages... especially
hybrid pages that have to pull data both from midas (using mjsonrpc) and from online ROOT analyzers (using jsroot).
K.O. |
|
1852
|
16 Mar 2020 |
Konstantin Olchanski | Info | MIDAS will use C++11 | > After much discussion, and following the MIDAS workshop at TRIUMF, we made the decision to use C++11 in MIDAS.
>
> There are many benefits, and only one drawback - no c++11 compilers in the default OS install on older computers (i.e.
> RHEL/SL/CentOS before el7). (the same applies to our use of cmake).
>
It turns out that support for the c++11 "regex" feature is missing on el7 (CentOS-7, our most common platform at TRIUMF).
According to https://stackoverflow.com/questions/12530406/is-gcc-4-8-or-earlier-buggy-about-regular-expressions
gcc 4.9.0 is the first one to implement c++11 regular expressions. el7 comes with gcc-4.8.5 and I confirm
that examples of using std::regex_replace() do not compile. I was looking to use std::regex_replace to implement URL rewriting
in the reverse proxy code in mhttpd.
I do not need this feature immediately, but I am surprised that such a thing can happen, thought others should know.
K.O. |
|
1853
|
16 Mar 2020 |
Pintaudi Giorgio | Info | MIDAS will use C++11 | About the boost library, that is exactly
what I did for a project of mine (the
calibration software for the WAGASCI
experiment). It turned out not so easy to
mantain because different Linux distros
package different versions of boost.
The reason I went down the "c++11 plus
boost" road is that the official T2K OS
is CentOS7 as well.
Looking back I think that using c++17 and
requiring a more recent version of the
compiler is much easier to maintain than
the combo c++11 + boost. In CentOS is
just a matter of installing a recent
devtool package ...
Another solution might be too repackage
boost into MIDAS so you have full control
of the environment.
> > After much discussion, and following
the MIDAS workshop at TRIUMF, we made the
decision to use C++11 in MIDAS.
> >
> > There are many benefits, and only one
drawback - no c++11 compilers in the
default OS install on older computers
(i.e.
> > RHEL/SL/CentOS before el7). (the same
applies to our use of cmake).
> >
>
> It turns out that support for the c++11
"regex" feature is missing on el7
(CentOS-7, our most common platform at
TRIUMF).
>
> According to
https://stackoverflow.com/questions/12530
406/is-gcc-4-8-or-earlier-buggy-about-
regular-expressions
> gcc 4.9.0 is the first one to implement
c++11 regular expressions. el7 comes with
gcc-4.8.5 and I confirm
> that examples of using
std::regex_replace() do not compile. I
was looking to use std::regex_replace to
implement URL rewriting
> in the reverse proxy code in mhttpd.
>
> I do not need this feature immediately,
but I am surprised that such a thing can
happen, thought others should know.
>
> K.O. |
|
1855
|
16 Mar 2020 |
Konstantin Olchanski | Info | mhttpd mongoose 6.16 update | the update of mhttpd to mongoose version 6.16 was committed to the develop branch of midas. If you do not want to use this
updated code or if it causes problems, please use the mhttpd6 executable or midas from the midas-2020-03 release branch.
new features:
- IPv6 support
- built-in http proxy
- fine grain locking - serving "resource" files (html, css, etc) and serving json-rpc requests no longer takes the global lock
- reduced number of DNS queries when checking host list access (DNS replies are cached)
- (I decided to not implement caching of password requests and dynamic reload of password file - it is too hard).
internal changes:
Recent versions of the mongoose web server library have removed all their internal multithreading,
leaving the library fully single-threaded. This resulted in major simplification of many things. An improvement.
(the civetweb fork of mongoose retains the old multithreading code, that model seems to work better
which used inside ROOT). As implemented in mhttpd, all network connections are handled by the main thread,
all midas http requests are handled by worker threads that are started on the as-needed basis.
The old mongoose 6.4 based mhttpd code survived almost without changes - as a compile-time
option - so now I build 2 mhttpd executables: mhttpd with the new code and mhttpd6 with the old code
so people have something to run in case the new code bombs.
http proxy:
Experiments that use private networks usually configure the apache httpd as a web proxy to allow
access from the outside to the web-controlled devices on the private network. Making changes
to this proxy requires root access, requires restarting httpd, etc. To make things simpler, mhttpd now
includes a web proxy (almost the complete implementation is provided by the mongoose library). Configuration
is done from ODB, restarting mhttpd is not needed.
improved multithreading:
Since most of the MIDAS library is now thread-safe, mhttpd no longer needs to take the "big midas lock"
to service most web requests. Access to files, access to ODB, etc is now fully threaded. Some parts
of MIDAS are not thread-safe, i.e. access to history and log files, so a flag was added to the mjsonrpc library
to mark which RPC methods are not thread-safe.
Note that despite these improvements, mhttpd still suffers from "http head-of-queue blocking"
https://en.wikipedia.org/wiki/Head-of-line_blocking
because (i.e. the google chrome web browser) tends to use just 1 TCP connection for all JSONRPC requests,
after a request for a history read (can take a long time), all subsequent requests for web page updates, etc
will have to wait until it completes, causing unresponsive user experience. (it looks as if mhttpd is single-threaded!).
A solution for this problem is HTTP/2, which is not yet implemented by mongoose and is not quite yet available
for apache httpd.
More later...
K.O. |
|
1856
|
16 Mar 2020 |
Konstantin Olchanski | Info | mhttpd mongoose 6.16 update | > the update of mhttpd to mongoose version 6.16 was committed to the develop branch of midas.
The new code implements 3 http ports:
- localhost port 8080 - enabled by default - suitable for "I want to test midas on my laptop" and for connecting from the apache httpd
https password protected gateway.
- insecure http port 8081 - disabled by default - with optional password protection (HTTP Digest auth), and optional hostlist access
control - for the case when the https gateway is running on a different computer (i.e. ALPHA at CERN).
(My reading of "internet opinions" about HTTP Digest authentication over unencrypted HTTP is
that while considered very obsolete, there are no specific security problems and exploits
against it - other than the usual - man-in-the-middle and "steal the password file" attacks.
So while I do not recommend using it, I do not feel justified to remove/disable it on security grounds.
It provides an alternative password protection when use of SSL/HTTPS is too difficult).
- https port 8443 - disabled by default - also with optional password protection (HTTP Digest auth), and optional hostlist access
control. HTTP Digest password protection over HTTPS is deemed as secure at "HTTP Basic" password protection over HTTPS and
that is what is used by apache httpd password protection.
(The main problem with mhttpd support of HTTPS is obtaining an https certificate. Right now mhttpd
instructs the user to generate a self-signed certificate. But there is 2 problems: modern browsers dislike self-signed
certificates (even when explicitely marked "trust it!") and there is no check for certificate expiration.
I guess one could try to integrate mhttpd with certbot and the let's-encrypt system, but there
is problems, i.e. the certificate files live in readable-only-by-root directories, etc. I would rather
wait until mongoose implement certbot integration in their code).
More later...
K.O. |
|
1857
|
16 Mar 2020 |
Konstantin Olchanski | Info | mhttpd mongoose 6.16 update | > > the update of mhttpd to mongoose version 6.16 was committed to the develop branch of midas.
Configuration is done by ODB /WebServer:
---------------------------------------------------------------------------
[local:javascript1:S]/WebServer>ls -l
Key name Type #Val Size Last Opn Mode Value
---------------------------------------------------------------------------
mime.types DIR
Enable localhost port BOOL 1 4 2h 0 RWD y
localhost port INT 1 4 2h 0 RWD 8080
localhost port passwords BOOL 1 4 2h 0 RWD n
Enable insecure port BOOL 1 4 12h 0 RWD n
insecure port INT 1 4 2h 0 RWD 8081
insecure port passwords BOOL 1 4 2h 0 RWD y
insecure port host list BOOL 1 4 2h 0 RWD y
Enable https port BOOL 1 4 12h 0 RWD n
https port INT 1 4 2h 0 RWD 8443
https port passwords BOOL 1 4 2h 0 RWD y
https port host list BOOL 1 4 2h 0 RWD y
Host list STRING 10 32 2h 0 RWD
[0] localhost
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
Enable IPv6 BOOL 1 4 2h 0 RWD y
Proxy DIR
---------------------------------------------------------------------------
Most entries are self-obvious, but note:
- mime.types contains the mapping of file extensions of file content-type telling browser what to do:
---------------------------------------------------------------------------
[local:javascript1:S]/WebServer>ls -l mime.types/
Key name Type #Val Size Last Opn Mode Value
---------------------------------------------------------------------------
.HTML STRING 1 10 2h 0 RWD text/html
.HTM STRING 1 10 2h 0 RWD text/html
.CSS STRING 1 9 2h 0 RWD text/css
---------------------------------------------------------------------------
- Proxy directory configures the http proxy (as implemented by mongoose, I am
not sure if I understand all limitations):
---------------------------------------------------------------------------
[local:javascript1:S]/WebServer>ls -l Proxy/
Key name Type #Val Size Last Opn Mode Value
---------------------------------------------------------------------------
example STRING 1 27 17h 0 RWD #http://localhost:8080
---------------------------------------------------------------------------
("#" means - commented-out)
http://localhost:8080/proxy/example/foo/bar/baz proxies to http://localhost:8080/foo/bar/baz
- "Enable IPv6" tells mhttpd to also listen on the IPv6 ports. The best I can tell IPv6 works on the Mac,
and with luck will get some testing at CERN where IPv6 is in use.
Documentation on the midas wiki still needs to be updated for this.
K.O. |
|
1858
|
17 Mar 2020 |
Konstantin Olchanski | Info | mbedtls, mhttpd mongoose 6.16 update | > > > the update of mhttpd to mongoose version 6.16 was committed to the develop branch of midas.
current code looks for the mbedtls library in ../mbedtls (next to midas)
if cmake misdetects it, turn it off by setting NO_MBEDTLS (same as NO_ROOT & co)
if you do want to build mhttpd with mbedtls, do this:
cd .../midas
cd ../
git clone https://github.com/ARMmbed/mbedtls.git
cd mbedtls
git submodule update --init ### this will populate the "crypto" directory
make ### if "python2" is missing, building of test suite programs will fail, but the libraries needed for midas will be built
cd ../midas
make cmake...
K.O. |
|
1870
|
30 Mar 2020 |
Stefan Ritt | Info | mbedtls, mhttpd mongoose 6.16 update | I had some quick look at the new mongoose code and didn't find anything I dislike. Did a quick test of the proxy which worked and is nice to have.
Agree with all KO said about authentication.
So if there are no complaints, I would suggest that we move the summary of this thread into the official documentation.
Stefan |
|
1871
|
03 Apr 2020 |
Stefan Ritt | Info | Change of TID_xxx data types | We have to request of a 64-bit integer data type to be included in MIDAS banks.
Since 64-bit integers are on some systems "long" and on other systems "long long",
I decided to create the two new data types
TID_INT64
TID_UINT64
which follows more the standard C++ tradition:
https://en.cppreference.com/w/cpp/types/integer
To be consistent, I renamed the old types:
TID_BYTE -> TID_UINT8
TID_SBYTE -> TID_INT8
TID_WORD -> TID_UINT16
TID_SHORT -> TID_INT16
TID_DWORD -> TID_UINT32
TID_INT -> TID_INT32
I left the old definitions in midas.h, so old code will still compile fine and be binary
compatible. But if you write new code, the new types are recommended.
If you save the ODB in ASCII format, the new types are used as stings as well, like
[/Experiment]
ODB timeout = INT32 : 10000
but the old types are still understood when you load an old ODB file.
I hope I didn't break anything, please report if you have any issue.
Stefan |
|
1872
|
03 Apr 2020 |
Francesco Renga | Info | CLOCK_REALTIME on MacOS | Dear all,
I'm trying to compile MIDAS on MacOS 10.10 and I get this error:
/Users/francesco/MIDAS/midas/src/system.cxx:3187:18: error: use of undeclared identifier
'CLOCK_REALTIME'
clock_settime(CLOCK_REALTIME, <m);
Is it related to my (old) version of MacOS? Can I fix it somehow?
Thank you,
Francesco |
|
1873
|
03 Apr 2020 |
Stefan Ritt | Info | CLOCK_REALTIME on MacOS | > Dear all,
> I'm trying to compile MIDAS on MacOS 10.10 and I get this error:
>
> /Users/francesco/MIDAS/midas/src/system.cxx:3187:18: error: use of undeclared identifier
> 'CLOCK_REALTIME'
> clock_settime(CLOCK_REALTIME, <m);
>
> Is it related to my (old) version of MacOS? Can I fix it somehow?
>
> Thank you,
> Francesco
If I see this correctly, you need at least MacOSX 10.12. If you can't upgrade, you can just remove line 3187
from system.cxx. This function is only used in an online environment, where you would run a frontend on your
Mac, which you probably don't do. So removing it does not hurt you.
Stefan |
|
1886
|
25 Apr 2020 |
Konstantin Olchanski | Info | CLOCK_REALTIME on MacOS | > > /Users/francesco/MIDAS/midas/src/system.cxx:3187:18: error: use of undeclared identifier
> > 'CLOCK_REALTIME'
> > clock_settime(CLOCK_REALTIME, <m);
> >
> > Is it related to my (old) version of MacOS? Can I fix it somehow?
I think the "set clock" function is a holdover from embedded operating systems
that did not keep track of clock time, i.e. VxWorks, and similar. Here a midas program
will get the time from the mserver and set it on the local system. Poor man's ntp,
poor man's ntpd/chronyd.
We should check if this function is called by anything, and if nothing calls it, maybe remove it?
K.O. |
|
1887
|
25 Apr 2020 |
Konstantin Olchanski | Info | new mac! | I received my new 2020 mac book air, so between Stefan and myself, MacOS support for
MIDAS is assured for 5 more years at the least. K.O. |
|
1890
|
26 Apr 2020 |
Stefan Ritt | Info | CLOCK_REALTIME on MacOS | > > > /Users/francesco/MIDAS/midas/src/system.cxx:3187:18: error: use of undeclared identifier
> > > 'CLOCK_REALTIME'
> > > clock_settime(CLOCK_REALTIME, <m);
> > >
> > > Is it related to my (old) version of MacOS? Can I fix it somehow?
>
> I think the "set clock" function is a holdover from embedded operating systems
> that did not keep track of clock time, i.e. VxWorks, and similar. Here a midas program
> will get the time from the mserver and set it on the local system. Poor man's ntp,
> poor man's ntpd/chronyd.
>
> We should check if this function is called by anything, and if nothing calls it, maybe remove it?
>
> K.O.
It's called in mfe.cxx via cm_synchronize:
/* set time from server */
#ifdef OS_VXWORKS
cm_synchronize(NULL);
#endif
This was for old VxWorks systems which had no ntp/crond. Was asked for by Pierre long time ago. I don't use it
(have no VxWorks). We can either remove it completely, or remove just the MacOSX part and just exit the program
if called with an error message "not implemented on this OS".
Stefan |
|
1906
|
12 May 2020 |
Stefan Ritt | Info | New ODB++ API | Since the beginning of the lockdown I have been working hard on a new object-oriented interface to the online database ODB. I have the code now in an initial state where it is ready for
testing and commenting. The basic idea is that there is an object midas::odb, which represents a value or a sub-tree in the ODB. Reading, writing and watching is done through this
object. To get started, the new API has to be included with
#include <odbxx.hxx>
To create ODB values under a certain sub-directory, you can either create one key at a time like:
midas::odb o;
o.connect("/Test/Settings", true); // this creates /Test/Settings
o.set_auto_create(true); // this turns on auto-creation
o["Int32 Key"] = 1; // create all these keys with different types
o["Double Key"] = 1.23;
o["String Key"] = "Hello";
or you can create a whole sub-tree at once like:
midas::odb o = {
{"Int32 Key", 1},
{"Double Key", 1.23},
{"String Key", "Hello"},
{"Subdir", {
{"Another value", 1.2f}
}
};
o.connect("/Test/Settings");
To read and write to the ODB, just read and write to the odb object
int i = o["Int32 Key];
o["Int32 Key"] = 42;
std::cout << o << std::endl;
This works with basic types, strings, std::array and std::vector. Each read access to this object triggers an underlying read from the ODB, and each write access triggers a write to the
ODB. To watch a value for change in the odb (the old db_watch() function), you can use now c++ lambdas like:
o.watch([](midas::odb &o) {
std::cout << "Value of key \"" + o.get_full_path() + "\" changed to " << o << std::endl;
});
Attached is a full running example, which is now also part of the midas repository. I have tested most things, but would not yet use it in a production environment. Not 100% sure if there
are any memory leaks. If someone could valgrind the test program, I would appreciate (currently does not work on my Mac).
Have fun!
Stefan
|
| Attachment 1: odbxx_test.cxx
|
/********************************************************************\
Name: odbxx_test.cxx
Created by: Stefan Ritt
Contents: Test and Demo of Object oriented interface to ODB
\********************************************************************/
#include <string>
#include <iostream>
#include <array>
#include <functional>
#include "odbxx.hxx"
#include "midas.h"
/*------------------------------------------------------------------*/
int main() {
cm_connect_experiment(NULL, NULL, "test", NULL);
midas::odb::set_debug(true);
// create ODB structure...
midas::odb o = {
{"Int32 Key", 42},
{"Bool Key", true},
{"Subdir", {
{"Int32 key", 123 },
{"Double Key", 1.2},
{"Subsub", {
{"Float key", 1.2f}, // floats must be explicitly specified
{"String Key", "Hello"},
}}
}},
{"Int Array", {1, 2, 3}},
{"Double Array", {1.2, 2.3, 3.4}},
{"String Array", {"Hello1", "Hello2", "Hello3"}},
{"Large Array", std::array<int, 10>{} }, // array with explicit size
{"Large String", std::string(63, '\0') }, // string with explicit size
};
// ...and push it to ODB. If keys are present in the
// ODB, their value is kept. If not, the default values
// from above are copied to the ODB
o.connect("/Test/Settings", true);
// alternatively, a structure can be created from an existing ODB subtree
midas::odb o2("/Test/Settings/Subdir");
std::cout << o2 << std::endl;
// retrieve, set, and change ODB value
int i = o["Int32 Key"];
o["Int32 Key"] = i+1;
o["Int32 Key"]++;
o["Int32 Key"] *= 1.3;
std::cout << "Should be 57: " << o["Int32 Key"] << std::endl;
// test with bool
o["Bool Key"] = !o["Bool Key"];
// test with std::string
std::string s = o["Subdir"]["Subsub"]["String Key"];
s += " world!";
o["Subdir"]["Subsub"]["String Key"] = s;
// test with a vector
std::vector<int> v = o["Int Array"];
v[1] = 10;
o["Int Array"] = v; // assign vector to ODB object
o["Int Array"][1] = 2; // modify ODB object directly
i = o["Int Array"][1]; // read from ODB object
o["Int Array"].resize(5); // resize array
o["Int Array"]++; // increment all values of array
// test with a string vector
std::vector<std::string> sv;
sv = o["String Array"];
sv[1] = "New String";
o["String Array"] = sv;
o["String Array"][2] = "Another String";
// iterate over array
int sum = 0;
for (int e : o["Int Array"])
sum += e;
std::cout << "Sum should be 11: " << sum << std::endl;
// creat key from other key
midas::odb oi(o["Int32 Key"]);
oi = 123;
// test auto refresh
std::cout << oi << std::endl; // each read access reads value from ODB
oi.set_auto_refresh_read(false); // turn off auto refresh
std::cout << oi << std::endl; // this does not read value from ODB
oi.read(); // this does manual read
std::cout << oi << std::endl;
midas::odb ox("/Test/Settings/OTF");
ox.delete_key();
// create ODB entries on-the-fly
midas::odb ot;
ot.connect("/Test/Settings/OTF", true); // this forces /Test/OTF to be created if not already there
ot.set_auto_create(true); // this turns on auto-creation
ot["Int32 Key"] = 1; // create all these keys with different types
ot["Double Key"] = 1.23;
ot["String Key"] = "Hello";
ot["Int Array"] = std::array<int, 10>{};
ot["Subdir"]["Int32 Key"] = 42;
ot["String Array"] = std::vector<std::string>{"S1", "S2", "S3"};
std::cout << ot << std::endl;
o.read(); // re-read the underlying ODB tree which got changed by above OTF code
std::cout << o.print() << std::endl;
// iterate over sub-keys
for (auto& oit : o)
std::cout << oit.get_odb()->get_name() << std::endl;
// print whole sub-tree
std::cout << o.print() << std::endl;
// dump whole subtree
std::cout << o.dump() << std::endl;
// delete test key from ODB
o.delete_key();
// watch ODB key for any change with lambda function
midas::odb ow("/Experiment");
ow.watch([](midas::odb &o) {
std::cout << "Value of key \"" + o.get_full_path() + "\" changed to " << o << std::endl;
});
do {
int status = cm_yield(100);
if (status == SS_ABORT || status == RPC_SHUTDOWN)
break;
} while (!ss_kbhit());
cm_disconnect_experiment();
return 1;
}
|
|
1913
|
20 May 2020 |
Konstantin Olchanski | Info | New ODB++ API | > midas::odb o;
> o["foo"] = 1;
This is an excellent development.
ODB is a tree-structured database, JSON is a tree-structured data format,
and they seem to fit together like hand and glove. For programming
web pages, Javascript and JSON-style access to ODB seems to work really well.
And now with modern C++ we can have a similar API for working with ODB tree data,
as if it were Javascript JSON tree data.
Let's see how well it works in practice!
K.O. |
|
1914
|
20 May 2020 |
Stefan Ritt | Info | New ODB++ API | In meanwhile, there have been minor changes and improvements to the API:
Previously, we had:
> midas::odb o;
> o.connect("/Test/Settings", true); // this creates /Test/Settings
> o.set_auto_create(true); // this turns on auto-creation
> o["Int32 Key"] = 1; // create all these keys with different types
> o["Double Key"] = 1.23;
> o["String Key"] = "Hello";
Now, we only need:
o.connect("/Test/Settings");
o["Int32 Key"] = 1; // create all these keys with different types
...
no "true" needed any more. If the ODB tree does not exist, it gets created. Similarly, set_auto_create() can be dropped, it's on by default (thought this makes more sense). Also the iteration over subkeys has
been changed slightly.
The full example attached has been updated accordingly.
Best,
Stefan |
| Attachment 1: odbxx_test.cxx
|
/********************************************************************\
Name: odbxx_test.cxx
Created by: Stefan Ritt
Contents: Test and Demo of Object oriented interface to ODB
\********************************************************************/
#include <string>
#include <iostream>
#include <array>
#include <functional>
#include "midas.h"
#include "odbxx.hxx"
/*------------------------------------------------------------------*/
int main() {
cm_connect_experiment(NULL, NULL, "test", NULL);
midas::odb::set_debug(true);
// create ODB structure...
midas::odb o = {
{"Int32 Key", 42},
{"Bool Key", true},
{"Subdir", {
{"Int32 key", 123 },
{"Double Key", 1.2},
{"Subsub", {
{"Float key", 1.2f}, // floats must be explicitly specified
{"String Key", "Hello"},
}}
}},
{"Int Array", {1, 2, 3}},
{"Double Array", {1.2, 2.3, 3.4}},
{"String Array", {"Hello1", "Hello2", "Hello3"}},
{"Large Array", std::array<int, 10>{} }, // array with explicit size
{"Large String", std::string(63, '\0') }, // string with explicit size
};
// ...and push it to ODB. If keys are present in the
// ODB, their value is kept. If not, the default values
// from above are copied to the ODB
o.connect("/Test/Settings");
// alternatively, a structure can be created from an existing ODB subtree
midas::odb o2("/Test/Settings/Subdir");
std::cout << o2 << std::endl;
// set, retrieve, and change ODB value
o["Int32 Key"] = 42;
int i = o["Int32 Key"];
o["Int32 Key"] = i+1;
o["Int32 Key"]++;
o["Int32 Key"] *= 1.3;
std::cout << "Should be 57: " << o["Int32 Key"] << std::endl;
// test with bool
o["Bool Key"] = false;
o["Bool Key"] = !o["Bool Key"];
// test with std::string
o["Subdir"]["Subsub"]["String Key"] = "Hello";
std::string s = o["Subdir"]["Subsub"]["String Key"];
s += " world!";
o["Subdir"]["Subsub"]["String Key"] = s;
// test with a vector
std::vector<int> v = o["Int Array"]; // read vector
std::fill(v.begin(), v.end(), 10);
o["Int Array"] = v; // assign vector to ODB array
o["Int Array"][1] = 2; // modify array element
i = o["Int Array"][1]; // read from array element
o["Int Array"].resize(5); // resize array
o["Int Array"]++; // increment all values of array
// test with a string vector
std::vector<std::string> sv;
sv = o["String Array"];
sv[1] = "New String";
o["String Array"] = sv;
o["String Array"][2] = "Another String";
// iterate over array
int sum = 0;
for (int e : o["Int Array"])
sum += e;
std::cout << "Sum should be 47: " << sum << std::endl;
// creat key from other key
midas::odb oi(o["Int32 Key"]);
oi = 123;
// test auto refresh
std::cout << oi << std::endl; // each read access reads value from ODB
oi.set_auto_refresh_read(false); // turn off auto refresh
std::cout << oi << std::endl; // this does not read value from ODB
oi.read(); // this forces a manual read
std::cout << oi << std::endl;
// create ODB entries on-the-fly
midas::odb ot;
ot.connect("/Test/Settings/OTF");// this forces /Test/OTF to be created if not already there
ot["Int32 Key"] = 1; // create all these keys with different types
ot["Double Key"] = 1.23;
ot["String Key"] = "Hello";
ot["Int Array"] = std::array<int, 10>{};
ot["Subdir"]["Int32 Key"] = 42;
ot["String Array"] = std::vector<std::string>{"S1", "S2", "S3"};
std::cout << ot << std::endl;
o.read(); // re-read the underlying ODB tree which got changed by above OTF code
std::cout << o.print() << std::endl;
// iterate over sub-keys
for (midas::odb& oit : o)
std::cout << oit.get_name() << std::endl;
// print whole sub-tree
std::cout << o.print() << std::endl;
// dump whole subtree
std::cout << o.dump() << std::endl;
// delete test key from ODB
o.delete_key();
// watch ODB key for any change with lambda function
midas::odb ow("/Experiment");
ow.watch([](midas::odb &o) {
std::cout << "Value of key \"" + o.get_full_path() + "\" changed to " << o << std::endl;
});
do {
int status = cm_yield(100);
if (status == SS_ABORT || status == RPC_SHUTDOWN)
break;
} while (!ss_kbhit());
cm_disconnect_experiment();
return 1;
}
|
|
1915
|
20 May 2020 |
Pintaudi Giorgio | Info | New ODB++ API | All this is very good news. I really wish this were available some months ago: it would have helped me immensely. The old C API was clunky at best.
I really like the idea and looking forward to using it (even if at the moment I do not have the need to) ... |
|