ID |
Date |
Author |
Topic |
Subject |
1882
|
24 Apr 2020 |
Pintaudi Giorgio | Forum | API to read MIDAS format file |
Stefan Ritt wrote: | I guess all three options would work. I just tried mhist and it still works with the "FILE" history
mhist -e <equipment name> -v <variable name> -h 10
for dumping a variable for the last 10 hours.
I could not get mhdump to work with current history files, maybe it only works with "MIDAS" history and not "FILE" history (see https://midas.triumf.ca/MidasWiki/index.php/History_System#History_drivers). Maybe Konstantin who wrote mhdump has some idea.
Writing your own parser is certainly possible (even in Python), but of course more work.
Stefan |
Dear Stefan,
thank you very much for the quick reply. Sorry if my message was not very clear, actually we are using the "MIDAS" history format and not the "FILE" one. So both mhist and mhdump should be ok (however I have only tested mhist).
Hypothetically which one between the two lends itself the better to being "batched"? I mean to be read and controlled by a program/routine. For example, some programs give the option to have the output formatted in json, etc... |
Draft
|
24 Apr 2020 |
Pintaudi Giorgio | Forum | API to read MIDAS format file |
Stefan Ritt wrote: | I guess all three options would work. I just tried mhist and it still works with the "FILE" historymhist -e <equipment name> -v <variable name> -h 10for dumping a variable for the last 10 hours.I could not get mhdump to work with current history files, maybe it only works with "MIDAS" history and not "FILE" history (see https://midas.triumf.ca/MidasWiki/index.php/History_System#History_drivers). Maybe Konstantin who wrote mhdump has some idea.Writing your own parser is certainly possible (even in Python), but of course more work.Stefan |
Thank you for the quick reply. Do notice that we have "MIDAS" history files and not "FILE", so both mhist and mhdump should be fine (however I have only tested mhist). Hipotetically, which one between mhist and mhdump do you think is better suited to be "batched"? I mean to be controlled and read by a routine? |
1880
|
24 Apr 2020 |
Stefan Ritt | Forum | API to read MIDAS format file | I guess all three options would work. I just tried mhist and it still works with the "FILE" history
mhist -e <equipment name> -v <variable name> -h 10
for dumping a variable for the last 10 hours.
I could not get mhdump to work with current history files, maybe it only works with "MIDAS" history and not "FILE" history (see https://midas.triumf.ca/MidasWiki/index.php/History_System#History_drivers). Maybe Konstantin who wrote mhdump has some idea.
Writing your own parser is certainly possible (even in Python), but of course more work.
Stefan |
Draft
|
24 Apr 2020 |
Stefan Ritt | Forum | API to read MIDAS format file | |
1878
|
24 Apr 2020 |
Pintaudi Giorgio | Forum | API to read MIDAS format file | Dear MIDAS people,
I need to borrow your wisdom for a bit.
I am developing a piece of software that should read the history data stored in a
.midas file (MIDAS format) and integrate it into the WAGASCI data quality output.
In other words, I need to read some temperature values stored in a .midas file and
compare them with the MPPC gains and check for temperature/gain dependence.
I see three possibilities:
- write a custom parser in C++ using the instructions contained in the Mhformat page;
- call the mhist program from within my application;
- call the mhdump program from within my application;
Which solution do you think is the best?
Because there is no need for raw performance, if possible, I would like to write my application in Python3 but C++ is also an option. |
1877
|
23 Apr 2020 |
Stefan Ritt | Suggestion | Sequencer loop break | > > You can do that with the "GOTO" statement, jumping to the first line after the loop.
> >
> > Here is a working example:
> >
> >
> > LOOP runs, 5
> > WAIT Seconds 3
> > IF $runs > 2
> > GOTO 7
> > ENDIF
> > ENDLOOP
> > MESSAGE "Finished", 1
> >
> > Best,
> > Stefan
>
> Hoi Stefan
>
> Thanks for your answer. As I understand it, this has to be in the sequence script before
> running. So, in the end, it is not different than just saying "LOOP runs, 2" and
> therefore the number of runs has do be known in advance as well. Or is there an option to
> change the script on runtime? What I would like, is to start a sequence with "LOOP runs,
> infinite" and when I come back to the experiment after falling asleep being able to break
> the loop after the next iteration, but still execute everything after ENDLOOP, i.e. the
> MESSAGE statement in your example. Because if I do a "Stop after current run", this seems
> not to happen.
>
> Best, Ivo
First, you have the sequencer button "Stop after current run", but that does of course ot
execute anything after ENDLOOP.
Second, you can put anything in the IF statement. Like create a variable on the ODB like
/Experiment/Run parameters/Stop loop and set this to zero. Then put in your script:
...
ODBGET /Experiment/Run parameters/Stop loop, flag
IF $flag == 1
GOTO 7
...
So once you want to stop the loop, set the flag in the ODB to one.
Best,
Stefan |
1876
|
23 Apr 2020 |
Ivo Schulthess | Suggestion | Sequencer loop break | > You can do that with the "GOTO" statement, jumping to the first line after the loop.
>
> Here is a working example:
>
>
> LOOP runs, 5
> WAIT Seconds 3
> IF $runs > 2
> GOTO 7
> ENDIF
> ENDLOOP
> MESSAGE "Finished", 1
>
> Best,
> Stefan
Hoi Stefan
Thanks for your answer. As I understand it, this has to be in the sequence script before
running. So, in the end, it is not different than just saying "LOOP runs, 2" and
therefore the number of runs has do be known in advance as well. Or is there an option to
change the script on runtime? What I would like, is to start a sequence with "LOOP runs,
infinite" and when I come back to the experiment after falling asleep being able to break
the loop after the next iteration, but still execute everything after ENDLOOP, i.e. the
MESSAGE statement in your example. Because if I do a "Stop after current run", this seems
not to happen.
Best, Ivo |
1875
|
21 Apr 2020 |
Stefan Ritt | Suggestion | Sequencer loop break | > I am using the Midas sequencer to run subsequent measurements in a loop, without
> knowing how many iterations in advance. Therefore, I am using the "infinity"
> option. Since I have other commands after the loop, it would be nice to have the
> possibility to break the loop, but let the sequencer then finish the rest of the
> commands.
> Cheers,
> Ivo
You can do that with the "GOTO" statement, jumping to the first line after the loop.
Here is a working example:
LOOP runs, 5
WAIT Seconds 3
IF $runs > 2
GOTO 7
ENDIF
ENDLOOP
MESSAGE "Finished", 1
Best,
Stefan |
1874
|
07 Apr 2020 |
Ivo Schulthess | Suggestion | Sequencer loop break | I am using the Midas sequencer to run subsequent measurements in a loop, without
knowing how many iterations in advance. Therefore, I am using the "infinity"
option. Since I have other commands after the loop, it would be nice to have the
possibility to break the loop, but let the sequencer then finish the rest of the
commands.
Cheers,
Ivo |
1873
|
03 Apr 2020 |
Stefan Ritt | Info | CLOCK_REALTIME on MacOS | > Dear all,
> I'm trying to compile MIDAS on MacOS 10.10 and I get this error:
>
> /Users/francesco/MIDAS/midas/src/system.cxx:3187:18: error: use of undeclared identifier
> 'CLOCK_REALTIME'
> clock_settime(CLOCK_REALTIME, <m);
>
> Is it related to my (old) version of MacOS? Can I fix it somehow?
>
> Thank you,
> Francesco
If I see this correctly, you need at least MacOSX 10.12. If you can't upgrade, you can just remove line 3187
from system.cxx. This function is only used in an online environment, where you would run a frontend on your
Mac, which you probably don't do. So removing it does not hurt you.
Stefan |
1872
|
03 Apr 2020 |
Francesco Renga | Info | CLOCK_REALTIME on MacOS | Dear all,
I'm trying to compile MIDAS on MacOS 10.10 and I get this error:
/Users/francesco/MIDAS/midas/src/system.cxx:3187:18: error: use of undeclared identifier
'CLOCK_REALTIME'
clock_settime(CLOCK_REALTIME, <m);
Is it related to my (old) version of MacOS? Can I fix it somehow?
Thank you,
Francesco |
1871
|
03 Apr 2020 |
Stefan Ritt | Info | Change of TID_xxx data types | We have to request of a 64-bit integer data type to be included in MIDAS banks.
Since 64-bit integers are on some systems "long" and on other systems "long long",
I decided to create the two new data types
TID_INT64
TID_UINT64
which follows more the standard C++ tradition:
https://en.cppreference.com/w/cpp/types/integer
To be consistent, I renamed the old types:
TID_BYTE -> TID_UINT8
TID_SBYTE -> TID_INT8
TID_WORD -> TID_UINT16
TID_SHORT -> TID_INT16
TID_DWORD -> TID_UINT32
TID_INT -> TID_INT32
I left the old definitions in midas.h, so old code will still compile fine and be binary
compatible. But if you write new code, the new types are recommended.
If you save the ODB in ASCII format, the new types are used as stings as well, like
[/Experiment]
ODB timeout = INT32 : 10000
but the old types are still understood when you load an old ODB file.
I hope I didn't break anything, please report if you have any issue.
Stefan |
1870
|
30 Mar 2020 |
Stefan Ritt | Info | mbedtls, mhttpd mongoose 6.16 update | I had some quick look at the new mongoose code and didn't find anything I dislike. Did a quick test of the proxy which worked and is nice to have.
Agree with all KO said about authentication.
So if there are no complaints, I would suggest that we move the summary of this thread into the official documentation.
Stefan |
1869
|
27 Mar 2020 |
Stefan Ritt | Forum | mlogger: misleading error messages for ROOT | Ok, changed.
Stefan |
1868
|
27 Mar 2020 |
Andreas Suter | Forum | mlogger: misleading error messages for ROOT | Hi Stefan,
I think this only partially resolves the issue, in log_write:
#ifdef HAVE_ROOT
} else if (log_chn->format == FORMAT_ROOT) {
status = root_write(log_chn, pevent, pevent->data_size + sizeof(EVENT_HEADER));
#endif
}
actual_time = ss_millitime();
if ((int) actual_time - (int) start_time > 3000)
cm_msg(MINFO, "log_write", "Write operation on \'%s\' took %d ms", log_chn->path.c_str(), actual_time - start_time);
if (status != SS_SUCCESS && !stop_requested) {
cm_msg(MTALK, "log_write", "Error writing output file, stopping run");
cm_msg(MERROR, "log_write", "Cannot write \'%s\', error %d, stopping run", log_chn->path.c_str(), status);
stop_the_run(0);
return status;
}
In your solution root_write returns quietly but status == SS_INVALID_FORMAT (not SS_SUCCESS) and hence I get another misleading error message "Error writing output file, stopping run".
In order to prevent this you also would need to change the return value to SS_SUCCESS. |
1867
|
27 Mar 2020 |
Stefan Ritt | Forum | mlogger: misleading error messages for ROOT | Dear simplest solution seems to me to just remove the error message generation and silently ignore the BOE EOR events.
Committed that change.
Stefan |
1866
|
25 Mar 2020 |
Konstantin Olchanski | Forum | mlogger: misleading error messages for ROOT | > [Logger,ERROR] [mlogger.cxx:3358:root_write,ERROR] Cannot write system event into ROOT file, event_id 0xffff8000
Hi, Andreas, please open a bug report for this problem on bitbucket, there is now at least 2 bugs against
the ROOT writer (some events are written in duplicate sometimes), and I hope to fix this next time i review
the mlogger (RSN!). Biggest problem is that I do not use the ROOT output myself, so I have no way
to know if ROOT files produced by mlogger are correct or make sense. (without setting up some kind
of test environment with a ROOT file reader.
Thank you for reporting this problem here, so more people know about it.
If somebody has a patch to fix this, please send it in!
K.O. |
1865
|
25 Mar 2020 |
Andreas Suter | Forum | mlogger: misleading error messages for ROOT | Dear All,
At our experiment we write ROOT files. When starting/stopping runs we get the following error messages:
[Logger,ERROR] [mlogger.cxx:3358:root_write,ERROR] Cannot write system event into ROOT file, event_id 0xffff8000
[Logger,ERROR] [mlogger.cxx:3358:root_write,ERROR] Cannot write system event into ROOT file, event_id 0xffff8001
Looking into the source code I found that log_write (line 4248) sends these Midas System Events (BOR,EOR) to root_write without filtering them. root_write() checks in a first step if it gets such Midas System Events and if yes, moans.
Wouldn't it be better just to filter these events in log_write, before calling root_write, avoiding unnecessary error messages?
Is there something I miss?
Thanks,
Andreas |
1864
|
24 Mar 2020 |
Konstantin Olchanski | Forum | Save data to FTP | >
> Since ILL only provides access via SFTP and everything else is not existent or blocked (not even ssh is possible),
> this is the only thing we can work with by now.
>
Oops. SFTP != FTP.
SFTP uses SSH for data transport, so we cannot do it directly from C++ code in MIDAS. (we could use libssh, etc, but...)
I suggest you use lazylogger with the lazy_dache script, replace "dccp" with "sftp", replace "nsls" with an sftp "ls" command.
If you get it working, please consider contributing your lazylogger script to midas. (and does not have to be written in perl, python should work equally well).
For setting up lazylogger with the script method, I am pretty sure I posted the instructions to the forum (ages ago),
let me know if you cannot find them.
Good luck.
K.O. |
1863
|
24 Mar 2020 |
Ivo Schulthess | Forum | Save data to FTP | > Logging directly from the midas logger to FTP is a bit cumbersome. In case of delays during login etc. this can throttle the whole DAQ chain.
> What we use in our lab is to write to local disk, then use the lazylogger (https://midas.triumf.ca/MidasWiki/index.php/Lazylogger) to copy the
> local files to a remote FTP server. This way we de-couple data taking from backup, making the system much more swift.
>
> Best,
> Stefan
Yes, see this now too. I will, therefore, try to set up the lazylogger properly. |
|