ID |
Date |
Author |
Topic |
Subject |
2476
|
27 Apr 2023 |
Marius Koeppel | Suggestion | Maximum ODB size | Hi all,
> I agree, I think we can safely bump the limit from 100 Mbytes to 1 Gbyte, maybe 1.5 or
> 1.99 Gbytes. Above that we run into 32-bit/31-bit cleanliness problems.
We just went in and changed: int odb_size_limit = INT_MAX;//100*1000*1000; in odb.cxx. And we could create ODBs with 1GB and 1.5 GB.
Since the DecodeSize function in odbinit has also foreseen yottabytes ;) (const char units[] = {'k', 'M', 'G', 'T', 'P', 'E', 'Z', 'Y'};) we think going to GB for the maximum ODB size would be create.
> there is a bug in odbinit, if initial odbinit fails, ODB with default size is creates,
> and original rejected ODB size is written to .ODB_SIZE.TXT (an inconsistency).
Can#t we go with the maximum size here if the user inputs a larger size? So just below printf("Checking ODB size...\n"); one could check for the odb_size_limit. In general one could move the odb_size_limit to midas.h so its not only available in odb.cxx.
Best,
Marius |
2475
|
26 Apr 2023 |
Martin Mueller | Forum | Problem with running midas odbxx frontends on a remote machine using the -h option | Hi
We have a problem with running midas frontends when they should connect to a experiment on a different machine using the -h option. Starting them locally works fine. Firewall is off on both systems, Enable non-localhost RPC and Disable RPC hosts check are set to 'yes', mserver is running on the machine that we want to connect to.
Error message looks like this:
...
Connect to experiment Mu3e on host 10.32.113.210...
OK
Init hardware...
terminate called after throwing an instance of 'mexception'
what():
/home/mu3e/midas/include/odbxx.h:1102: Wrong key type in XML file
Stack trace:
1 0x00000000000042D828 (null) + 4380712
2 0x00000000000048ED4D midas::odb::odb_from_xml(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 605
3 0x0000000000004999BD midas::odb::odb(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 317
4 0x000000000000495383 midas::odb::read_key(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 1459
5 0x0000000000004971E3 midas::odb::connect(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool) + 259
6 0x000000000000497636 midas::odb::connect(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, bool) + 502
7 0x00000000000049883B midas::odb::connect_and_fix_structure(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 171
8 0x0000000000004385EF setup_odb() + 8351
9 0x00000000000043B2E6 frontend_init() + 22
10 0x000000000000433304 main + 1540
11 0x0000007F8C6FE3724D __libc_start_main + 239
12 0x000000000000433F7A _start + 42
Aborted (core dumped)
We have the same problem for all our frontends. When we want to start them locally they work. Starting them locally with ./frontend -h localhost also reproduces the error above.
The error can also be reproduced with the odbxx_test.cxx example in the midas repo by replacing line 22 in midas/examples/odbxx/odbxx_test.cxx (cm_connect_experiment(NULL, NULL, "test", NULL);) with cm_connect_experiment("localhost", "Mu3e", "test", NULL); (Put the name of the experiment instead of "Mu3e")
running odbxx_test locally gives us then the same error as our other frontend.
Thanks in advance,
Martin |
2474
|
21 Apr 2023 |
Konstantin Olchanski | Forum | Setup Midas with Caen vx2740 - ask for help | > I'm trying to setup Midas with the Caen vx2740 VME digitizer board.
welcome to the world of daq and midas! Ben already answered and he will help you with this specific hardware. (we work together)
> #0 0x00005555555c2ee1 in rpc_register_function (id=id@entry=18000, func=func@entry=0x5555555a2790 <jrpc_helper(int, void**)>)
> at /home/astrocent/workspace/packages/midas/src/midas.cxx:11947
I look at this line in midas and I do not see any problems other than all functions that touch rpc_list are not thread safe,
and calling them at the same time as rpc calls are active will cause memory corruption and crash. This is not a problem
in most programs because rpc_register_function() is usually called once at the beginning of everything, before any RPCs
are received, sent or processed. I filed a bug against this problem. https://bitbucket.org/tmidas/midas/issues/362/rpc_list-is-not-thread-safe
> with is "jrpc"?
I implemented it years ago to allow web pages to call mhttpd (XHR/HTTP) to call user frontends (MIDAS RPC) to perform real-time actions,
i.e. to turn power supplied on or off. "j" stands for "json", but most experiments send very simple commands
and do not use json encoding.
K.O. |
2473
|
21 Apr 2023 |
Ben Smith | Forum | Setup Midas with Caen vx2740 - ask for help | > I'm not able to find documentation what is purpose of the RPC? Could someone give any indicators how I can start debug this behavior? Or there is some documentation about the RPC?
The RPC system allows midas clients to issue commands to each other. In the case of the VX2740 code we use it so the midas webserver (mhttpd) can tell the frontend to perform some actions when a user clicks a button on a webpage.
I've been writing most of the code for the VX2740 for Darkside, so will contact you directly to help debug the issue. |
2472
|
21 Apr 2023 |
Grzegorz Nieradka | Forum | Setup Midas with Caen vx2740 - ask for help | I'm trying to setup Midas with the Caen vx2740 VME digitizer board.
As the backend driver I used the software from Darkside located here:
https://bitbucket.org/ttriumfdaq/dsproto_vx2740/src/develop/
They implemented some helpers program and one from them should diagnose correct running of digitizer. But when I'm trying to run example program "vx2740_readout_test" I have segmentation fault:
Thread 1 "vx2740_readout_" received signal SIGSEGV, Segmentation fault.
0x00005555555c2ee1 in rpc_register_function (id=id@entry=18000, func=func@entry=0x5555555a2790 <jrpc_helper(int, void**)>) at /home/astrocent/workspace/packages/midas/src/midas.cxx:11947
During the calling this program I have running mhttpd, mlogger and the backend for vx2740 from the repository.
I'm not able to find documentation what is purpose of the RPC? Could someone give any indicators how I can start debug this behavior? Or there is some documentation about the RPC?
I'm freshman in the Midas world, so at this moment everything seems for me very complicated - and I'm learning by doing.
Regards,
Grzegorz
The backtrace from gdb which indicates the function in Midas package:
#0 0x00005555555c2ee1 in rpc_register_function (id=id@entry=18000, func=func@entry=0x5555555a2790 <jrpc_helper(int, void**)>)
at /home/astrocent/workspace/packages/midas/src/midas.cxx:11947
#1 0x00005555555c2f12 in cm_register_function (id=id@entry=18000, func=func@entry=0x5555555a2790 <jrpc_helper(int, void**)>)
at /home/astrocent/workspace/packages/midas/src/midas.cxx:5840
#2 0x00005555555a26f6 in VX2740GroupFrontend::init (this=this@entry=0x7fffffffcba0, group_idx=group_idx@entry=-1, hDB=hDB@entry=0, enable_jrpc=enable_jrpc@entry=true)
at /home/astrocent/workspace/packages/ttriumfdaq-dsproto_vx2740-8122058cacd1/vx2740_fe_class.cxx:134
#3 0x000055555557e492 in do_fe (board_name=..., is_scope=<optimized out>) at /home/astrocent/workspace/packages/ttriumfdaq-dsproto_vx2740-8122058cacd1/vx2740_readout_test.cxx:185
#4 0x000055555557adc9 in main (argc=<optimized out>, argv=0x7fffffffd1f8) at /home/astrocent/workspace/packages/ttriumfdaq-dsproto_vx2740-8122058cacd1/vx2740_readout_test.cxx:253 |
2471
|
20 Mar 2023 |
Gennaro Tortone | Forum | pull request for PostgreSQL support |
Hi,
I have updated the PR with a new one that includes TimescaleDB support and some
changes to mhistory.js to support downsampling queries...
Cheers,
Gennaro
> > some minutes ago I published a PR for PostgreSQL support I developed
> > at INFN-Napoli for Darkside experiment...
> >
> > I don't know if you receive a notification about this PR and in doubt
> > I wrote this message...
>
> Hi, Gennaro, thank you for the very useful contribution. I saw the previous version
> of your pull request and everything looked quite good. But that pull request was
> for an older version of midas and it would not have applied cleanly to the current
> version. I will take a look at your updated pull request. In theory it should only
> add the Postgres class and modify a few other places in history_schema.cxx and have
> no changes to anything else. (if you need those changes, it should be a separate
> pull request).
>
> Also I am curious what benefits and drawbacks of Postgres vs mysql/mariadb you have
> observed for storing and using midas history data.
>
> K.O. |
2470
|
17 Mar 2023 |
Konstantin Olchanski | Info | T2K/ND280 - Many warning from ten year old variables in ODB | Forwarded from the T2K/ND280 elog:
Author : Nick Hastings
Subject : Many warning from ten year old variables in ODB
Logbook URL : http://elog.nd280.org/elog/FGD/2553
Midas does period checks that the variables in the ODB are ok.
One of these is a check to see if each variable was set with +/- 10
years. Since this experiment has been running for longer than 10
years there are *many* variables that fail this check.
As a result the midas.log and messages in mhttpd are spammed
with many warnings. Eg
Mon Feb 13 14:49:18 2023 [ODBEdit,ERROR] [odb.c:548:db_validate_key,ERROR] Warning: invalid access time, key
"/System/Prompt", time 1288763123
These can be removed by simply setting the variable again with its current value.
So I wonder if it would be best to just do a full odbdump and then load all the values
back in. Comments from MIDAS experts would be appreciated. Eg:
odbedit -c 'save fgddaq.odb'
odbedit -c 'load fgddaq.odb'
Note this problem is currently seen on both the FGD DAQ and the global slow control MIDAS instances.
It may also be a problem on the INGRID GSC and the DAQs of other ND280 systems but I did not check. |
2469
|
16 Mar 2023 |
Konstantin Olchanski | Forum | bitbucket issue spam cleaned | > > midas bitbucket repository had a spam attack, about 40 spam messages were posted
> > into the issues. I was able to delete them manually. No idea how they got past
> > bitbucket spam filters and if they are spam or an attack against automated issue
> > tracker tools or an attack against the repo owner (who is vulnerable as they rush
> > in to deal with the spam). if this happens again, "anonymous issues" may have to
> > be disabled, bitbucket login required. K.O.
>
> Two more spam messages, deleted. "Anonymous users can create issues" is now turned off.
Also, same for: rootana.
Also, empty issue trackers disabled: mvodb, midasio, mscb.
K.O. |
2468
|
16 Mar 2023 |
Konstantin Olchanski | Forum | bitbucket issue spam cleaned | > midas bitbucket repository had a spam attack, about 40 spam messages were posted
> into the issues. I was able to delete them manually. No idea how they got past
> bitbucket spam filters and if they are spam or an attack against automated issue
> tracker tools or an attack against the repo owner (who is vulnerable as they rush
> in to deal with the spam). if this happens again, "anonymous issues" may have to
> be disabled, bitbucket login required. K.O.
Two more spam messages, deleted. "Anonymous users can create issues" is now turned off.
K.O. |
2467
|
16 Mar 2023 |
Konstantin Olchanski | Forum | Having trouble with MIDAS setup | > I'm not sure if this is the right forum for this query
this might be the right place, depending.
> I'm having a little bit of trouble with the setup of a Midas system ...
> inherited after the previous guy ...
a rather major problem, but a typical situation.
> There was a point at which it was working.
this is very good. if it worked before, there is good chance it will work again.
> And then there was an unrelated issue in the electrical system which, as a side effect,
> meant that the building lost power for a time, and the entire system had to be rebooted.
> No problem, I thought. I'll just reset and restart all of the software...
> ...and I can't seem to get it to work.
this happens often enough. several things are likely to happen:
- unexpected software updates, i.e. new linux kernel was installed but inactive, waiting for a reboot
- hardware failures, i.e. we usually see blown up power supplies. check that all VME crate voltages are okey. (ask me how).
- firmware corruption, i.e. we have seen VME modules lose their firmware after power outage, had to be reloaded by jtag
> I keep getting the error message "mvme_read_value: Could not perform read!: Bad address".
this is a generic error, it does not mean that software suddenly is trying to read from wrong address.
> I imagine that there is something that needs to be set, twiddled, tweaked, or turned on in the driver. The output of 'lsmod | grep vme' is:
> vmedriver 117742 0
this is not the vme driver we use at TRIUMF, so I am not familiar with it's errors. we use the vme_tsi148 driver and the vme_universe driver. (ask me about them).
> so presumably the driver is at least *present*, even if I have no idea how to twiddle anything on it.
could be the wrong version of the driver or the wrong version of the linux kernel. worth checking log files
to see if kernel and driver version numbers are the same.
> Could anyone perhaps suggest a way forward?
Yes. You will have to tell me much more about your system. You can do this publicly here or privately by email to olchansk@triumf.ca
To start, I need to know your VME setup, what is the crate, what is the VME processor, what OS you run, what VME driver you use, what VME modules you have installed.
K.O. |
2466
|
16 Mar 2023 |
Konstantin Olchanski | Forum | bitbucket issue spam cleaned | midas bitbucket repository had a spam attack, about 40 spam messages were posted
into the issues. I was able to delete them manually. No idea how they got past
bitbucket spam filters and if they are spam or an attack against automated issue
tracker tools or an attack against the repo owner (who is vulnerable as they rush
in to deal with the spam). if this happens again, "anonymous issues" may have to
be disabled, bitbucket login required. K.O. |
2465
|
15 Mar 2023 |
Casey | Forum | Having trouble with MIDAS setup | Hi
I'm not sure if this is the right forum for this query (if it is not, I would truly appreciate it if someone could point me at the right forum). I'm having a little bit of trouble with the setup of a Midas system.
Now, this is a system that I inherited after the previous guy who was looking after it went to other employment. There was a point at which it was working. And then there was an unrelated issue in the electrical system which, as a side effect, meant that the building lost power for a time, and the entire system had to be rebooted.
No problem, I thought. I'll just reset and restart all of the software...
...and I can't seem to get it to work. I keep getting the error message "mvme_read_value: Could not perform read!: Bad address". So far as I can tell, this seems to be related to the idea that the base address being used to read from the boards is incorrect. The base addresses are hardcoded in the software (not autogenerated) and, aside from the power going down and up again, the hardware hasn't been touched since the system was working.
I imagine that there is something that needs to be set, twiddled, tweaked, or turned on in the driver. The output of 'lsmod | grep vme' is:
vmedriver 117742 0
so presumably the driver is at least *present*, even if I have no idea how to twiddle anything on it.
Could anyone perhaps suggest a way forward? Is there some way to gather the information that I need, perhaps, or some way to twiddle anything twiddle-able on the driver?
Casey |
2464
|
07 Mar 2023 |
Stefano Piacentini | Info | connection to a MySQL server: retry procedure in the Logger | > > Dear all,
> >
> > we are experiencing a connection problem to the MySQL server that we use to log informations. Is there an
> > option to retry multiple times the I/O on the MySQL?
> >
> > The error we are experiencing is the following (hiding the IP address):
> >
> > [Logger,ERROR] [mlogger.cxx:2455:write_runlog_sql,ERROR] Failed to connect to database: Error: Can't
> > connect to MySQL server on 'xxx.xxx.xxx.xxx:6033' (110)
> >
> > Then the logger stops, and must be restarted. This eventually happens only during the BOR or the EOR.
>
> What would you propose? If the connection does not work, most likely the server is down or busy. If we retry,
> the connection still might not work. If we retry many times, people will complain that the run start or stop
> takes very long. If we then just continue (without stopping the logger), the MySQL database will miss important
> information and the runs probably cannot be analyzed later. So I believe it's better to really stop the logger
> so that people get aware that there is a problem and fix the source, rather than curing the symptoms.
>
> In the MEG experiment at PSI we run the logger with a MySQL database and we never see any connection issue,
> except when the MySQL server gets in maintenance (once a year), but usually we don't take data then. Since we
> use the same logger code, it cannot be a problem there. So I would try to fix the problem on the MySQL side.
>
> Best,
> Stefan
Dear Stefan,
a possible solution could be to define the number of times to retry as a parameter that is 0 by default, as well as a wait time between two subsequent tries. This
would leave the decision on how to handle a possible failed connection to the user. In our case, for example, we would prefer to not stop the acquisition in case
of a failed connection to the external SQL. In addition, we have other software that, with a retry procedure, doesn’t fail: with 1 re-try and a sleep time of 0.5 s
we already recover 100% of the faults.
Anyway, we implemented a local database, which is a mirror of the external one, and the problems disappeared.
Thanks,
Stefano. |
2463
|
06 Mar 2023 |
Gennaro Tortone | Forum | pull request for PostgreSQL support |
Hi Konstantin,
thanks for this update |
My main interest for PostgreSQL is usage of TimescaleDB
(https://github.com/timescale/timescaledb) a PostgreSQL extension that
makes possible usage of downsampling functions on time-series...
here at INFN-Napoli we have a large history dataset that we manage
with MIDAS history and MySQL tables. We have a lot of issues
(wait time, browser hangs, crashes) when we use MIDAS history plot
pages on large time period because the Javascript web page try to
download million of records in order to display them on a plot of
(max) 2000 pixel width...
with native downsampling we can reduce a large dataset keeping the
"shape" of the curve using only the points needed by the plot area;
in TimescaleDB there is "lttb" ( Largest Triangle Three Bucket) a very
nice and impressive downsampling function that preserve very well the
shape of the series.
If you are interested to see a lttb at work on some data you can open this page:
https://www.base.is/flot
In next days I will work to add TimescaleDB backend to MIDAS history (it will be
similar to PostgreSQL backend) and we can discuss on how to add these
downsampling features to history plot web pages, I already developed some
solutions and I will be happy to share them with MIDAS community;
Cheers,
Gennaro
> > some minutes ago I published a PR for PostgreSQL support I developed
> > at INFN-Napoli for Darkside experiment...
> >
> > I don't know if you receive a notification about this PR and in doubt
> > I wrote this message...
>
> Hi, Gennaro, thank you for the very useful contribution. I saw the previous version
> of your pull request and everything looked quite good. But that pull request was
> for an older version of midas and it would not have applied cleanly to the current
> version. I will take a look at your updated pull request. In theory it should only
> add the Postgres class and modify a few other places in history_schema.cxx and have
> no changes to anything else. (if you need those changes, it should be a separate
> pull request).
>
> Also I am curious what benefits and drawbacks of Postgres vs mysql/mariadb you have
> observed for storing and using midas history data.
>
> K.O. |
Draft
|
06 Mar 2023 |
Gennaro Tortone | Forum | pull request for PostgreSQL support |
Hi
> > some minutes ago I published a PR for PostgreSQL support I developed
> > at INFN-Napoli for Darkside experiment...
> >
> > I don't know if you receive a notification about this PR and in doubt
> > I wrote this message...
>
> Hi, Gennaro, thank you for the very useful contribution. I saw the previous version
> of your pull request and everything looked quite good. But that pull request was
> for an older version of midas and it would not have applied cleanly to the current
> version. I will take a look at your updated pull request. In theory it should only
> add the Postgres class and modify a few other places in history_schema.cxx and have
> no changes to anything else. (if you need those changes, it should be a separate
> pull request).
>
> Also I am curious what benefits and drawbacks of Postgres vs mysql/mariadb you have
> observed for storing and using midas history data.
>
> K.O. |
2461
|
06 Mar 2023 |
Konstantin Olchanski | Forum | pull request for PostgreSQL support | > some minutes ago I published a PR for PostgreSQL support I developed
> at INFN-Napoli for Darkside experiment...
>
> I don't know if you receive a notification about this PR and in doubt
> I wrote this message...
Hi, Gennaro, thank you for the very useful contribution. I saw the previous version
of your pull request and everything looked quite good. But that pull request was
for an older version of midas and it would not have applied cleanly to the current
version. I will take a look at your updated pull request. In theory it should only
add the Postgres class and modify a few other places in history_schema.cxx and have
no changes to anything else. (if you need those changes, it should be a separate
pull request).
Also I am curious what benefits and drawbacks of Postgres vs mysql/mariadb you have
observed for storing and using midas history data.
K.O. |
2460
|
06 Mar 2023 |
Gennaro Tortone | Forum | pull request for PostgreSQL support |
Hi,
some minutes ago I published a PR for PostgreSQL support I developed
at INFN-Napoli for Darkside experiment...
I don't know if you receive a notification about this PR and in doubt
I wrote this message...
Thanks in advance,
Gennaro |
2459
|
22 Feb 2023 |
Stefan Ritt | Info | connection to a MySQL server: retry procedure in the Logger | > Dear all,
>
> we are experiencing a connection problem to the MySQL server that we use to log informations. Is there an
> option to retry multiple times the I/O on the MySQL?
>
> The error we are experiencing is the following (hiding the IP address):
>
> [Logger,ERROR] [mlogger.cxx:2455:write_runlog_sql,ERROR] Failed to connect to database: Error: Can't
> connect to MySQL server on 'xxx.xxx.xxx.xxx:6033' (110)
>
> Then the logger stops, and must be restarted. This eventually happens only during the BOR or the EOR.
What would you propose? If the connection does not work, most likely the server is down or busy. If we retry,
the connection still might not work. If we retry many times, people will complain that the run start or stop
takes very long. If we then just continue (without stopping the logger), the MySQL database will miss important
information and the runs probably cannot be analyzed later. So I believe it's better to really stop the logger
so that people get aware that there is a problem and fix the source, rather than curing the symptoms.
In the MEG experiment at PSI we run the logger with a MySQL database and we never see any connection issue,
except when the MySQL server gets in maintenance (once a year), but usually we don't take data then. Since we
use the same logger code, it cannot be a problem there. So I would try to fix the problem on the MySQL side.
Best,
Stefan |
2458
|
22 Feb 2023 |
Stefano Piacentini | Info | connection to a MySQL server: retry procedure in the Logger | Dear all,
we are experiencing a connection problem to the MySQL server that we use to log informations. Is there an
option to retry multiple times the I/O on the MySQL?
The error we are experiencing is the following (hiding the IP address):
[Logger,ERROR] [mlogger.cxx:2455:write_runlog_sql,ERROR] Failed to connect to database: Error: Can't
connect to MySQL server on 'xxx.xxx.xxx.xxx:6033' (110)
Then the logger stops, and must be restarted. This eventually happens only during the BOR or the EOR.
Best,
Stefano. |
2457
|
09 Feb 2023 |
Stefan Ritt | Suggestion | histories capture 'ruy' | > I agree with you, option 2 is better and works well.
> The only problem is that if you are showing multiple histograms in the same window the keyDown even will affect all of the histories in the window.
> This may be the intended behaviour, but I think that if we can find a way to have the event affecting only the intended history (focused element for example) it would be better.
The history panels are simple pictures from the perspective of the browser and have no concept of a focus. This would have to be tweaked somehow. If you want to reset a single history,
just click on its reset button (circle arrow at the right top).
Stefan |
|