 24 Apr 2020, Pintaudi Giorgio, Forum, API to read MIDAS format file
|
Dear MIDAS people,
I need to borrow your wisdom for a bit.
I am developing a piece of software that should read the history data stored in a
|
 24 Apr 2020, Stefan Ritt, Forum, API to read MIDAS format file
|
I guess all three options would work. I just tried mhist and it still works with the "FILE" history
mhist -e <equipment name> -v <variable name> -h 10
|
|
24 Apr 2020, Pintaudi Giorgio, Forum, API to read MIDAS format file
|
[quote="Stefan Ritt"]I guess all three options would work. I just tried mhist and it still works with the "FILE" history
mhist -e <equipment name> -v <variable name> -h 10
|
|
24 Apr 2020, Stefan Ritt, Forum, API to read MIDAS format file
|
[quote="Pintaudi Giorgio"]
Hypothetically which one between the two lends itself the better to being "batched"? I mean to be read and controlled by a program/routine. For example,
some programs give the option to have the output formatted in json, etc...[/quote]
|
|
25 Apr 2020, Konstantin Olchanski, Forum, API to read MIDAS format file
|
<p>[quote="Pintaudi Giorgio"]Dear MIDAS people, I need to borrow your
wisdom for a bit. I am developing a piece of software that should read the history data
stored in a [FONT=Times New Roman].midas[/FONT] file (MIDAS format) and integrate it
|
|
03 May 2020, Pintaudi Giorgio, Forum, API to read MIDAS format file
|
> The format of .hst midas history files is pretty simple and mhdump.cxx is an easy to read
> illustration on how to read it from basic principles (without going through the midas library,
> which can be somewhat complicated). The newer "FILE" format for history is even simpler
|
|
03 May 2020, Konstantin Olchanski, Forum, API to read MIDAS format file
|
>
> - One is to convert to SQL format and then use a SQLite library to import the data in my
> application.
|
|
04 May 2020, Pintaudi Giorgio, Forum, API to read MIDAS format file
|
> (But note that back when I implemented the SQLITE history writer, sqlite database corruption
> recovery instructions were "delete the file, restore from backup". And indeed in every test
> experiment I tried, the sqlite history databases eventually corrupted themselves. You see
|
03 May 2020, Stefan Ritt, Forum, API to read MIDAS format file 
|
> PS some time ago, I don't remember if you or Stefan, recommended CLion as C++ IDE. I have tried it
> (together with PyCharm) and I must admit that it is really good. It took me years to configure Emacs
|
|
26 May 2020, Pintaudi Giorgio, Forum, API to read MIDAS format file
|
Eventually, I have settled for the SQLite format.
I could convert the MIDAS history files .hst to SQLite
database .sqlite3 using the utility mh2sql.
|
|
22 May 2020, Thomas Lindner, Bug Report, More trouble with openssl on macos
|
For the record, here's my report of difficulties getting mongoose to compile with macos. This is a similar
problem reported before, but with slightly different error messages. So I put them here for posterity.
|
|
22 May 2020, Konstantin Olchanski, Bug Report, More trouble with openssl on macos
|
> For the record, here's my report of difficulties getting mongoose to compile with macos.
> -- MIDAS: Found OpenSSL version 1.0.2s
> -- MIDAS: Found OpenSSL version 1.1.1g
|
|
16 Mar 2020, Konstantin Olchanski, Release, midas-2020-03-a
|
midas-2020-03-a is here.
Accumulated changes and bug fixes since last tag midas-2019-09-i.
|
|
22 May 2020, Konstantin Olchanski, Release, midas-2020-03-a
|
> midas-2020-03-a is here.
> checkout the top of branch release/midas-2020-03 (recommended) or
> checkout the tag midas-2020-03-a.
|
|
12 May 2020, Stefan Ritt, Info, New ODB++ API
|
Since the beginning of the lockdown I have been working hard on a new object-oriented interface to the online database ODB. I have the code now in an initial
state where it is ready for
testing and commenting. The basic idea is that there is an object midas::odb, which represents a value or a sub-tree in the ODB. Reading, writing and watching |
|
20 May 2020, Konstantin Olchanski, Info, New ODB++ API
|
> midas::odb o;
> o["foo"] = 1;
|
|
20 May 2020, Stefan Ritt, Info, New ODB++ API
|
In meanwhile, there have been minor changes and improvements to the API:
Previously, we had:
|
|
20 May 2020, Pintaudi Giorgio, Info, New ODB++ API
|
All this is very good news. I really wish this were available some months ago: it would have helped me immensely. The old C API was clunky at best.
I really like the idea and looking forward to using it (even if at the moment I do not have the need to) ... |
|
20 May 2020, Konstantin Olchanski, Info, New ODB++ API
|
> All this is very good news. I really wish this were available some months ago: it would have helped me immensely. The old C API was clunky at best.
> I really like the idea and looking forward to using it (even if at the moment I do not have the need to) ...
|
|
12 May 2020, Ruslan Podviianiuk, Forum, List of sequencer files
|
Hello,
We are going to implement a list of sequencer files to allow users to select one
|
|
13 May 2020, Stefan Ritt, Forum, List of sequencer files
|
If you load a file into the sequencer from the web interface, you get a list of all files in that directory.
This basically gives you a list of possible sequencer files. It's even more powerful, since you can
create subdirectories and thus group the sequencer files. Attached an example from our
|
|
18 May 2020, Ruslan Podviianiuk, Forum, List of sequencer files
|
> If you load a file into the sequencer from the web interface, you get a list of all files in that directory.
> This basically gives you a list of possible sequencer files. It's even more powerful, since you can
> create subdirectories and thus group the sequencer files. Attached an example from our
|
|
19 May 2020, Ruslan Podviianiuk, Forum, List of sequencer files
|
> If you load a file into the sequencer from the web interface, you get a list of all files in that directory.
> This basically gives you a list of possible sequencer files. It's even more powerful, since you can
> create subdirectories and thus group the sequencer files. Attached an example from our
|
|
20 May 2020, Konstantin Olchanski, Forum, List of sequencer files
|
>
> We have a custom webpage and trying to get list of files from the custom webpage and need jrpc command to show it
> in custom page. Is there a jrpc command to get this file list?
|
|
07 May 2020, Estelle, Bug Report, Conflic between Rootana and midas about the redefinition of TID_xxx data types
|
Dear Midas and Rootana people,
We have tried to update our midas DAQ with the new TID definitions describe in https://midas.triumf.ca/elog/Midas/1871
|
|
20 May 2020, Konstantin Olchanski, Bug Report, Conflic between Rootana and midas about the redefinition of TID_xxx data types
|
> Dear Midas and Rootana people,
>
> We have tried to update our midas DAQ with the new TID definitions describe in https://midas.triumf.ca/elog/Midas/1871
|
|
01 May 2020, Joseph McKenna, Forum, Taking MIDAS beyond 64 clients
|
Hi all,
|
|
01 May 2020, Stefan Ritt, Forum, Taking MIDAS beyond 64 clients
|
Hi Joseph,
here some thoughts from my side:
|
|
01 May 2020, Pierre Gorel, Forum, Taking MIDAS beyond 64 clients
|
> - On the other hand, if we have to break compatibility, now is maybe a good time since most accelerators worldwide are off. But before doing so, I would
like to get feedback from the main experiments
> around the world (MEG, T2K, g-2, DEAP besides ALPHA).
|
|
02 May 2020, Stefan Ritt, Forum, Taking MIDAS beyond 64 clients
|
TRIUMF stayed quiet, probably they have other things to do.
I allowed myself to move the maximum number of clients back to its original value, in order not to break running experiments.
|
|
02 May 2020, Joseph McKenna, Forum, Taking MIDAS beyond 64 clients
|
Thank you very much for feedback.
|
|
02 May 2020, Stefan Ritt, Forum, Taking MIDAS beyond 64 clients
|
> Perhaps a item for future discussion would be for the odbinit program to be able to 'upgrade' the ODB and enable some backwards
> compatibility.
|
|
02 May 2020, Konstantin Olchanski, Forum, Taking MIDAS beyond 64 clients
|
>
> Does the community here have strong opinions about increasing the
> MAX_CLIENTS and MAX_RPC_CONNECTION limits?
|
|
02 May 2020, Konstantin Olchanski, Forum, Taking MIDAS beyond 64 clients
|
> >
> > Does the community here have strong opinions about increasing the
> > MAX_CLIENTS and MAX_RPC_CONNECTION limits?
|
|
02 May 2020, Konstantin Olchanski, Forum, Taking MIDAS beyond 64 clients
|
> > >
> > > Does the community here have strong opinions about increasing the
> > > MAX_CLIENTS and MAX_RPC_CONNECTION limits?
|
|
03 Apr 2020, Francesco Renga, Info, CLOCK_REALTIME on MacOS
|
Dear all,
I'm trying to compile MIDAS on MacOS 10.10 and I get this error:
|
|
03 Apr 2020, Stefan Ritt, Info, CLOCK_REALTIME on MacOS
|
> Dear all,
> I'm trying to compile MIDAS on MacOS 10.10 and I get this error:
>
|
|
25 Apr 2020, Konstantin Olchanski, Info, CLOCK_REALTIME on MacOS
|
> > /Users/francesco/MIDAS/midas/src/system.cxx:3187:18: error: use of undeclared identifier
> > 'CLOCK_REALTIME'
> > clock_settime(CLOCK_REALTIME, <m);
|
|
26 Apr 2020, Stefan Ritt, Info, CLOCK_REALTIME on MacOS
|
> > > /Users/francesco/MIDAS/midas/src/system.cxx:3187:18: error: use of undeclared identifier
> > > 'CLOCK_REALTIME'
> > > clock_settime(CLOCK_REALTIME, <m);
|
|
26 Apr 2020, Yu Chen (SYSU), Forum, Questions and discussions on the Frontend ODB tree structure.
|
Dear MIDAS developers and colleagues,
This is Yu CHEN of School of Physics, Sun Yat-sen University, China, working in the PandaX-III collaboration, an experiment under development to search |
|
26 Apr 2020, Stefan Ritt, Forum, Questions and discussions on the Frontend ODB tree structure.
|
Dear Yu Chen,
in my opinion, you can follow two strategies:
|
|
25 Apr 2020, Konstantin Olchanski, Info, new mac!
|
I received my new 2020 mac book air, so between Stefan and myself, MacOS support for
MIDAS is assured for 5 more years at the least. K.O. |
|
07 Apr 2020, Ivo Schulthess, Suggestion, Sequencer loop break
|
I am using the Midas sequencer to run subsequent measurements in a loop, without
knowing how many iterations in advance. Therefore, I am using the "infinity"
option. Since I have other commands after the loop, it would be nice to have the
|
|
21 Apr 2020, Stefan Ritt, Suggestion, Sequencer loop break
|
> I am using the Midas sequencer to run subsequent measurements in a loop, without
> knowing how many iterations in advance. Therefore, I am using the "infinity"
> option. Since I have other commands after the loop, it would be nice to have the
|
|
23 Apr 2020, Ivo Schulthess, Suggestion, Sequencer loop break
|
> You can do that with the "GOTO" statement, jumping to the first line after the loop.
>
> Here is a working example:
|
|
23 Apr 2020, Stefan Ritt, Suggestion, Sequencer loop break
|
> > You can do that with the "GOTO" statement, jumping to the first line after the loop.
> >
> > Here is a working example:
|
|
25 Apr 2020, Konstantin Olchanski, Suggestion, Sequencer loop break
|
> LOOP runs, 5
> ...
> ENDLOOP
|
|
16 Mar 2020, Konstantin Olchanski, Info, mhttpd mongoose 6.16 update
|
the update of mhttpd to mongoose version 6.16 was committed to the develop branch of midas. If you do not want to use this
updated code or if it causes problems, please use the mhttpd6 executable or midas from the midas-2020-03 release branch.
|
|
16 Mar 2020, Konstantin Olchanski, Info, mhttpd mongoose 6.16 update
|
> the update of mhttpd to mongoose version 6.16 was committed to the develop branch of midas.
The new code implements 3 http ports:
|
|
16 Mar 2020, Konstantin Olchanski, Info, mhttpd mongoose 6.16 update
|
> > the update of mhttpd to mongoose version 6.16 was committed to the develop branch of midas.
Configuration is done by ODB /WebServer:
|
|
17 Mar 2020, Konstantin Olchanski, Info, mbedtls, mhttpd mongoose 6.16 update
|
> > > the update of mhttpd to mongoose version 6.16 was committed to the develop branch of midas.
current code looks for the mbedtls library in ../mbedtls (next to midas)
|
|
30 Mar 2020, Stefan Ritt, Info, mbedtls, mhttpd mongoose 6.16 update
|
I had some quick look at the new mongoose code and didn't find anything I dislike. Did a quick test of the proxy which worked and is nice to have.
Agree with all KO said about authentication.
|
|
25 Mar 2020, Andreas Suter, Forum, mlogger: misleading error messages for ROOT
|
Dear All,
At our experiment we write ROOT files. When starting/stopping runs we get the following error messages:
|
|
25 Mar 2020, Konstantin Olchanski, Forum, mlogger: misleading error messages for ROOT
|
> [Logger,ERROR] [mlogger.cxx:3358:root_write,ERROR] Cannot write system event into ROOT file, event_id 0xffff8000
Hi, Andreas, please open a bug report for this problem on bitbucket, there is now at least 2 bugs against
|
|
27 Mar 2020, Stefan Ritt, Forum, mlogger: misleading error messages for ROOT
|
Dear simplest solution seems to me to just remove the error message generation and silently ignore the BOE EOR events.
Committed that change.
|
|
27 Mar 2020, Andreas Suter, Forum, mlogger: misleading error messages for ROOT
|
Hi Stefan,
I think this only partially resolves the issue, in log_write:
|
|
27 Mar 2020, Stefan Ritt, Forum, mlogger: misleading error messages for ROOT
|
Ok, changed.
Stefan |
|
23 Mar 2020, Ivo Schulthess, Forum, Save data to FTP
|
Dear all
I try to save data to an FTP server but don't get any data on the server. Midas does not complain or message any error but also nothing gets saved. Does
somebody have experience with this? I use the following settings for the ODB mlogger channel settings: Type: FTP, Filename: server.com, 21, user, pw, ., |
|
23 Mar 2020, Konstantin Olchanski, Forum, Save data to FTP
|
> I try to save data to an FTP server but don't get any data on the server. Midas does not complain or message any error but also nothing gets saved. Does
somebody have experience with this? I use the following settings for the ODB mlogger channel settings: Type: FTP, Filename: server.com, 21, user, pw, .,
run%06d.mid, Format: MIDAS, Output: FILE. What would be the Output: FTP setting for? I tried this but it does not work at all.
|
|
24 Mar 2020, Ivo Schulthess, Forum, Save data to FTP
|
> > I try to save data to an FTP server but don't get any data on the server. Midas does not complain or message any error but also nothing gets saved.
Does somebody have experience with this? I use the following settings for the ODB mlogger channel settings: Type: FTP, Filename: server.com, 21, user,
pw, ., run%06d.mid, Format: MIDAS, Output: FILE. What would be the Output: FTP setting for? I tried this but it does not work at all.
|
|
24 Mar 2020, Stefan Ritt, Forum, Save data to FTP
|
Logging directly from the midas logger to FTP is a bit cumbersome. In case of delays during login etc. this can throttle the whole DAQ chain.
What we use in our lab is to write to local disk, then use the lazylogger (https://midas.triumf.ca/MidasWiki/index.php/Lazylogger) to copy the
local files to a remote FTP server. This way we de-couple data taking from backup, making the system much more swift.
|
|
24 Mar 2020, Ivo Schulthess, Forum, Save data to FTP
|
> Logging directly from the midas logger to FTP is a bit cumbersome. In case of delays during login etc. this can throttle the whole DAQ chain.
> What we use in our lab is to write to local disk, then use the lazylogger (https://midas.triumf.ca/MidasWiki/index.php/Lazylogger) to copy the
> local files to a remote FTP server. This way we de-couple data taking from backup, making the system much more swift.
|
|
24 Mar 2020, Konstantin Olchanski, Forum, Save data to FTP
|
>
> Since ILL only provides access via SFTP and everything else is not existent or blocked (not even ssh is possible),
> this is the only thing we can work with by now.
|
|
08 Aug 2019, Konstantin Olchanski, Info, MIDAS will use C++11
|
After much discussion, and following the MIDAS workshop at TRIUMF, we made the decision to use C++11 in MIDAS.
There are many benefits, and only one drawback - no c++11 compilers in the default OS install on older computers (i.e.
|
|
16 Mar 2020, Konstantin Olchanski, Info, MIDAS will use C++11
|
> After much discussion, and following the MIDAS workshop at TRIUMF, we made the decision to use C++11 in MIDAS.
>
> There are many benefits, and only one drawback - no c++11 compilers in the default OS install on older computers (i.e.
|
|
16 Mar 2020, Pintaudi Giorgio, Info, MIDAS will use C++11
|
About the boost library, that is exactly
what I did for a project of mine (the
calibration software for the WAGASCI
|
|
10 Mar 2020, Konstantin Olchanski, Info, MIDAS vs JSROOT web pages
|
Just FYI, I am looking at the ROOT web programming component JSROOT and I notice that the RPC mechanism quite different from the JSON-
RPC I implemented for MIDAS.
|
|
06 Mar 2020, Lars Martin, Forum, RPC error
|
I ported a bunch of frontends to C++ and now I'm occasionally getting this RPC
error message:
|
|
08 Mar 2020, Konstantin Olchanski, Forum, RPC error
|
I do not see this error, but there was one more report (they did not clearly say what http errors
they see) https://bitbucket.org/tmidas/midas/issues/209/get-rid-of-mjsonrpc-dialogs-put-it-to-
the
|
|
29 Jan 2020, Berta Beltran, Bug Report, Compiling Midas in OS 10.15 Catalina
|
Hi all,
I have updated our daq computer to the latest OS 10.15 with the idea that then I will get all our daq
|
|
02 Feb 2020, Konstantin Olchanski, Bug Report, Compiling Midas in OS 10.15 Catalina
|
> I have updated our daq computer to the latest OS 10.15 ...
FWIW, I do not have macos 10.15. I have 10.13 at home and 10.14 in the office. Maybe Stefan has it?
|
|
03 Feb 2020, Stefan Ritt, Bug Report, Compiling Midas in OS 10.15 Catalina
|
> > I have updated our daq computer to the latest OS 10.15 ...
>
> FWIW, I do not have macos 10.15. I have 10.13 at home and 10.14 in the office. Maybe Stefan has it?
|
|
04 Feb 2020, Konstantin Olchanski, Bug Report, Compiling Midas in OS 10.15 Catalina
|
> > > I have updated our daq computer to the latest OS 10.15 ...
> >
> > FWIW, I do not have macos 10.15. I have 10.13 at home and 10.14 in the office. Maybe Stefan has it?
|
|
06 Feb 2020, Berta Beltran, Bug Report, Compiling Midas in OS 10.15 Catalina
|
>
> Ok, in this case, I will update my office mac mini to 10.15.
|
|
10 Feb 2020, Konstantin Olchanski, Bug Report, Compiling Midas in OS 10.15 Catalina
|
> Any luck with Midas in OS 10.15?
Best I can tell, the problem is not in midas: pthread.h should be there, somewhere.
|
|
11 Feb 2020, Berta Beltran, Bug Report, Compiling Midas in OS 10.15 Catalina
|
> > Any luck with Midas in OS 10.15?
>
> Best I can tell, the problem is not in midas: pthread.h should be there, somewhere.
|
|
11 Feb 2020, Stefan Ritt, Bug Report, Compiling Midas in OS 10.15 Catalina
|
For your reference, here on my MacOSX 10.14.6 with XCode 11.3.1 the pthread.h file is present in locations listed below.
Did you execute "xcode-select --install" ?
|
|
12 Feb 2020, Berta Beltran, Bug Report, Compiling Midas in OS 10.15 Catalina
|
> For your reference, here on my MacOSX 10.14.6 with XCode 11.3.1 the pthread.h file is present in locations listed below.
>
> Did you execute "xcode-select --install" ?
|
|
12 Feb 2020, Stefan Ritt, Bug Report, Compiling Midas in OS 10.15 Catalina
|
Another thought: Can you delete the midas build directory and run cmake again? Like
$ cd midas/build
|
|
12 Feb 2020, Berta Beltran, Bug Report, Compiling Midas in OS 10.15 Catalina
|
> Another thought: Can you delete the midas build directory and run cmake again? Like
>
> $ cd midas/build
|
|
13 Feb 2020, Stefan Ritt, Bug Report, Compiling Midas in OS 10.15 Catalina
|
Now you are stuck with openssl, which is optional for mhttpd. If you only use mhttpd locally, you maybe don't need SSL support. In that case you can jus
do
|
|
13 Feb 2020, Berta Beltran, Bug Report, Compiling Midas in OS 10.15 Catalina
|
> Now you are stuck with openssl, which is optional for mhttpd. If you only use mhttpd locally, you maybe don't need SSL support. In that case you can jus
do
>
|
|
28 Feb 2020, Konstantin Olchanski, Bug Report, Compiling Midas in OS 10.15 Catalina
|
> > [midas/build] $ cmake -D NO_SSL=1 ..
> If I run the compilation with the flag NO_SSL it works just fine. ...
|
|
03 Mar 2020, Berta Beltran, Bug Report, Compiling Midas in OS 10.15 Catalina
|
Thanks Konstantin,
I will keep an eye for the next release so that I can update my Midas to include ssl libraries.
|
|
13 Feb 2020, Marius Koeppel, Forum, Writting Midas Events via FPGAs
|
Dear all,
we creating Midas events directly inside a FPGA and send them off via DMA into the PC RAM. For reading out this RAM via Midas the FPGA sends as a pointer |
|
13 Feb 2020, Stefan Ritt, Forum, Writting Midas Events via FPGAs
|
The rb_xxx function are (thoroughly tested!) robust against high data rate given that you use them as intended:
1) Once you create the ring buffer via rb_create(), specify the maximum event size (overall event size, not bank size!). Later there is no protection any |
|
14 Feb 2020, Konrad Briggl, Forum, Writting Midas Events via FPGAs
|
Hello Stefan,
is there a difference for the later data processing (after writing the ring buffer blocks)
if we write single events or multiple in one rb_get_wp - memcopy - rb_increment_wp cycle?
|
|
14 Feb 2020, Stefan Ritt, Forum, Writting Midas Events via FPGAs
|
rb_xxx functions are midas event agnostic. The receiving side in mfe.cxx (lines 1418 in receive_trigger_event) however pulls one event at a time. If you
have some inconsistency I would put some debugging code there.
|
|
20 Feb 2020, Konstantin Olchanski, Forum, Writting Midas Events via FPGAs
|
> rb_xxx functions are midas event agnostic. The receiving side in mfe.cxx (lines 1418 in receive_trigger_event) however pulls one event at a time. If you
> have some inconsistency I would put some debugging code there.
|
|
20 Feb 2020, Marius Koeppel, Forum, Writting Midas Events via FPGAs
|
We also agree and found the problem now. Since we build everything (MIDAS Event Header, Bank Header, Banks etc.) in the FPGA we had some struggle with
the MIDAS data format (http://lmu.web.psi.ch/docu/manuals/bulk_manuals/software/midas195/html/AppendixA.html). We thought that only the MIDAS Event needs |
|
20 Feb 2020, Stefan Ritt, Forum, Writting Midas Events via FPGAs
|
Actually the cause of all of the is a real bug in the midas functions. We want each bank 8-byte aligned, so there is code in bk_close like:
midas.cxx:14788:
|
|
21 Feb 2020, Konstantin Olchanski, Forum, Writting Midas Events via FPGAs
|
Hi, Stefan - is this our famous 64-bit misalignement? Where we have each alternating bank aligned and misaligned at 64 bits? Without changing the data
format, one can always store data in 64-bit aligned banks by inserting a dummy banks between real banks:
|
|
21 Feb 2020, Stefan Ritt, Forum, Writting Midas Events via FPGAs
|
> Hi, Stefan - is this our famous 64-bit misalignement? Where we have each alternating bank aligned and misaligned at 64 bits? Without changing the data
> format, one can always store data in 64-bit aligned banks by inserting a dummy banks between real banks:
>
|
|
21 Feb 2020, Konstantin Olchanski, Forum, Writting Midas Events via FPGAs
|
> We also agree and found the problem now.
Good. what was wrong?
|
|
20 Feb 2020, Marius Koeppel, ,
|
We also agree and found the problem now. Since we build everything (MIDAS Event Header, Bank Header, Banks etc.) in the FPGA we had some struggle with
the MIDAS data format (http://lmu.web.psi.ch/docu/manuals/bulk_manuals/software/midas195/html/AppendixA.html). We thought that only the MIDAS Event needs |