ID |
Date |
Author |
Topic |
Subject |
1980
|
11 Aug 2020 |
Ivo Schulthess | Bug Report | data missing in runXXXXXX.mid |
> It would be good to pin point there the data is lost. This is the sequence:
>
> frontend user code -> mfe.c code -> SYSTEM buffer -> mlogger -> disk
>
> To see if correct data arrives to the SYSTEM buffer, run:
> mdump -z SYSTEM
>
> To see if mlogger is receiving events from the SYSTEM buffer, run:
> mlogger -v ### mlogger should report all events, history and data
>
> To see if mlogger writes events to disk, examine the disk file (in this case, you already did, data is not there).
>
> I would guess that your data does not make it out from the frontend (mdump shows "nothing"),
> if data were to arrive into the SYSTEM buffer, it would make it to disk, unless
> mlogger is misconfigured (but you already checked that).
>
> If you have trouble with the frontend framework code, you can try to switch from the mfe.c frontend
> to the newer c++ tmfe frontend (see progs/fetest_tmfe.cxx and progs/fetest_tmfe_thread.cxx).
>
> K.O.
Good evening
I tried to reproduce the behavior in a very simple FE but it did not work out. The next thing for me would be to take the FE that is producing this behavior, replace all the device communication and data with dummies. If the problem is still there I would start to simplify as much as possible.
Following the inputs of KO, I pin-pointed the data loss. The system buffer still gets the data but the mlogger does not write the data event. Then of course the data is also not anymore present in the data file. Therefore, I checked the logger settings again, Event ID and Trigger Mask still -1. Nothing else, at least from my point of view, that is misconfigured. Nevertheless, if it helps I can send my ODB settings.
When doing the tests just before I found something else that probably can give a hint to the problem. The data is only lost if the time between two runs is long (a few seconds). As an example: If I run a sequence with a loop and after the FE stops the run the loop ends and the next run is started automatically, then only the first run has no data, which is the one after a longer time of no data taking. When I add a "WAIT Seconds 5" after the run before starting the next, not data is written to the disk for any run. I also found this once when adding a sleep(1) at the end of the FE readout function but back then did not think about it any further.
Best, Ivo |
1979
|
11 Aug 2020 |
Konstantin Olchanski | Bug Report | data missing in runXXXXXX.mid |
> I have to reproduce the problem to fix it. Why don't you go and modify midas/examples/experiment/frontend.cxx in such a way that
> it creates exactly the banks you have, just with random data. If you see the same problem, send me your frontend file so that I
> can reproduce it.
It would be good to pin point there the data is lost. This is the sequence:
frontend user code -> mfe.c code -> SYSTEM buffer -> mlogger -> disk
To see if correct data arrives to the SYSTEM buffer, run:
mdump -z SYSTEM
To see if mlogger is receiving events from the SYSTEM buffer, run:
mlogger -v ### mlogger should report all events, history and data
To see if mlogger writes events to disk, examine the disk file (in this case, you already did, data is not there).
I would guess that your data does not make it out from the frontend (mdump shows "nothing"),
if data were to arrive into the SYSTEM buffer, it would make it to disk, unless
mlogger is misconfigured (but you already checked that).
If you have trouble with the frontend framework code, you can try to switch from the mfe.c frontend
to the newer c++ tmfe frontend (see progs/fetest_tmfe.cxx and progs/fetest_tmfe_thread.cxx).
K.O. |
1978
|
10 Aug 2020 |
Stefan Ritt | Bug Report | data missing in runXXXXXX.mid |
I have to reproduce the problem to fix it. Why don't you go and modify midas/examples/experiment/frontend.cxx in such a way that
it creates exactly the banks you have, just with random data. If you see the same problem, send me your frontend file so that I
can reproduce it. |
Draft
|
10 Aug 2020 |
Pierre-Andre Amaudruz | Bug Report | data missing in runXXXXXX.mid |
> Have you tried longer files? Maybe a few 100 MB or so. Maybe a buffer is not flushed correctly at the end of a run.
Hi,
If the online mdump is correct (by default using the -1, -1 filter), the data are in the main SYSTEM buffer.
by the way, similar to the dump - mdump -x file.mid -m raw -d x will show you the events.
The fact that the analyzer doesn't see the banks would indicate a buffer handling issue as mentioned by Stefan.
To confirm, I would check at the end of a run that the sum of the equipment "events sent" matches the logger "Events written". |
Draft
|
10 Aug 2020 |
Pierre-Andre Amaudruz | Bug Report | data missing in runXXXXXX.mid |
> > > Dear all
> > >
> > > We just started our beam time at ILL and just found yesterday that for certain
> > > settings of our detector the data is not saved into the .mid files. Running "mdump
> > > -l 10" online we see the data coming in as they should. Nevertheless, if we run
> > > "mdump -x runXXXXXX.mid" offline, the data file has no events and the banks are
> > > missing. Any ideas where the data could go lost?
> > >
> > > Thanks in advance,
> > > Ivo
> >
> > Have you checked
> >
> > /Logger/Channels/0/Settings/Event ID = -1
> > /Logger/Channels/0/Settings/Trigger mask = -1
> >
> > If these settings are not -1, they filter the data stream for certain events and trigger
> > masks.
> >
> > Stefan
>
> Good morning Stefan
>
> Both set to -1. We only have one logging channel. If we run a sequence with a few runs and the
> same settings, sometimes data is in the .mid file and sometimes it is not.
>
> Best,
> Ivo
Hi,
If the online mdump is correct (by default using the -1, -1 filter), the data are in the main SYSTEM buffer.
Similar to the dump -
The fact that the analyzer doesn't see the banks would indicate a buffer handling issue as mentioned by Stefan.
To confirm, I would check at the end of a run that the sum of the equipment "events sent" matches the logger "Events written". |
1975
|
10 Aug 2020 |
Ivo Schulthess | Bug Report | data missing in runXXXXXX.mid |
> Have you tried longer files? Maybe a few 100 MB or so. Maybe a buffer is not flushed correctly at the end of a run.
Yes, I did. This 7 KB of the data bank is about the limit. If we go only 1 KB higher it seems that we save all data. In
our specific case, this is the number of time bins (256 pixels with 7 time bins results in data loss, with 8 time bins it
seems to be okay, data type is DWORD).
Of course, a workaround for us is to save at least 8 time bins and throw 7 of them away later on. Nevertheless, since we
are only in the commissioning phase now this is okay, I would just like to avoid data loss in the data taking phase of the
experiment so knowing where the problem origins could help.
I did another test with another FE running that produces a lot of data. The behavior is the same though. If the bank size
is less than about 8 KB, the bank is not saved anymore. But probably this is anyway the expected behavior since it is a
different FE that produces the data.
So if it is coming from the buffer, is there something I could change to test or solve the problem?
Best, Ivo |
1974
|
10 Aug 2020 |
Mathieu Guigue | Info | MidasConfig.cmake usage |
As the Midas software is installed using CMake, it can be easily integrated into
other CMake projects using the MidasConfig.cmake file produced during the Midas
installation.
This file points to the location of the include and libraries of Midas using three
variables:
- MIDAS_INCLUDE_DIRS
- MIDAS_LIBRARY_DIRS
- MIDAS_LIBRARIES
Then the CMakeLists file of the new project can use the CMake find_package
functionalities like:
```
find_package (Midas REQUIRED)
if (MIDAS_FOUND)
MESSAGE(STATUS "Found midas: libraries ${MIDAS_LIBRARIES}")
pbuilder_add_ext_libraries (${MIDAS_LIBRARIES})
else (MIDAS_FOUND)
message(FATAL "Unable to find midas")
endif (MIDAS_FOUND)
include_directories (${MIDAS_INCLUDE_DIR})
```
pbuilder_add_ext_libraries is a CMake macro allowing to automatically add the
libraries into the project: this macro can be found here:
https://github.com/project8/scarab/blob/master/cmake/PackageBuilder.cmake
If such macro doesn't exist, the linkage to each executable/library can be done
similarly to https://midas.triumf.ca/elog/Midas/1964 using:
```
target_link_libraries(crfe ${MIDAS_LIBARIES} ${LIBS})
```
The current version of the MidasConfig.cmake is minimal and could for example
include a version number: this would allow to define a e.g. minimal version of
Midas needed by the new project. |
1973
|
10 Aug 2020 |
Stefan Ritt | Bug Report | data missing in runXXXXXX.mid |
Have you tried longer files? Maybe a few 100 MB or so. Maybe a buffer is not flushed correctly at the end of a run. |
1972
|
10 Aug 2020 |
Ivo Schulthess | Bug Report | data missing in runXXXXXX.mid |
> with "dump" I meant a true object dump like "hexdump -C run000001.mid". I produced a file with ADC0 and TDC0
> banks (that's the example from the distribution under exampels/experiments/frontend.cxx), and I get
>
> ....
> 00024220 01 00 00 00 41 44 43 30 04 00 08 00 eb 06 35 04 |....ADC0......5.|
> 00024230 31 09 4f 06 54 44 43 30 04 00 08 00 93 04 fb 07 |1.O.TDC0........|
> 00024240 5c 09 88 0b 01 00 00 00 01 00 00 00 2a 0b 31 5f |\...........*.1_|
> 00024250 28 00 00 00 20 00 00 00 01 00 00 00 41 44 43 30 |(... .......ADC0|
> 00024260 04 00 08 00 c3 09 24 05 85 05 f3 06 54 44 43 30 |......$.....TDC0|
> 00024270 04 00 08 00 88 08 2d 03 3b 0d d6 02 01 00 00 00 |......-.;.......|
> 00024280 02 00 00 00 2a 0b 31 5f 28 00 00 00 20 00 00 00 |....*.1_(... ...|
> 00024290 01 00 00 00 41 44 43 30 04 00 08 00 a5 0a 69 09 |....ADC0......i.|
>
> where you clearly see the ADC0 and TDC0 banks.
>
> Stefan
So at least I learned something new. I tried it with the hexdump and the banks are not existent in the .mid file. I
only have the ODB inside the file. The 7K difference in size is actually just about what I expect to be the data
(1792 x 4 bytes)
Best, Ivo |
1971
|
10 Aug 2020 |
Stefan Ritt | Bug Report | data missing in runXXXXXX.mid |
> So I did a quick check. The file size is about the same (322K and 329K). When I dump the .mid I don't see
> the banks. It only prints two lines with "------ Event# 0 ------" and "------ Event# 1 ------" whereas for
> the file with data I get the two banks with all the data. Our online analyzer also fails to see the banks.
> Is there another way to check what is in the .mid file?
with "dump" I meant a true object dump like "hexdump -C run000001.mid". I produced a file with ADC0 and TDC0
banks (that's the example from the distribution under exampels/experiments/frontend.cxx), and I get
....
00024220 01 00 00 00 41 44 43 30 04 00 08 00 eb 06 35 04 |....ADC0......5.|
00024230 31 09 4f 06 54 44 43 30 04 00 08 00 93 04 fb 07 |1.O.TDC0........|
00024240 5c 09 88 0b 01 00 00 00 01 00 00 00 2a 0b 31 5f |\...........*.1_|
00024250 28 00 00 00 20 00 00 00 01 00 00 00 41 44 43 30 |(... .......ADC0|
00024260 04 00 08 00 c3 09 24 05 85 05 f3 06 54 44 43 30 |......$.....TDC0|
00024270 04 00 08 00 88 08 2d 03 3b 0d d6 02 01 00 00 00 |......-.;.......|
00024280 02 00 00 00 2a 0b 31 5f 28 00 00 00 20 00 00 00 |....*.1_(... ...|
00024290 01 00 00 00 41 44 43 30 04 00 08 00 a5 0a 69 09 |....ADC0......i.|
where you clearly see the ADC0 and TDC0 banks.
Stefan |
1970
|
10 Aug 2020 |
Ivo Schulthess | Bug Report | data missing in runXXXXXX.mid |
> Then I'm running out of ideas. Things I would check:
>
> - Are the file sizes about the same?
>
> - When you dump the .mid file, you do you see your bank names?
>
> This would tell you if the events are really missing or if mdump would just not find them.
>
> But I guess without being able to debug the system at ILL I cannot be of any more help. You are the
> first one reporting such a problem, so it must have to do with your local setup.
>
> Stefan
So I did a quick check. The file size is about the same (322K and 329K). When I dump the .mid I don't see
the banks. It only prints two lines with "------ Event# 0 ------" and "------ Event# 1 ------" whereas for
the file with data I get the two banks with all the data. Our online analyzer also fails to see the banks.
Is there another way to check what is in the .mid file?
Best,
Ivo |
1969
|
10 Aug 2020 |
Stefan Ritt | Bug Report | data missing in runXXXXXX.mid |
> Both set to -1. We only have one logging channel. If we run a sequence with a few runs and the
> same settings, sometimes data is in the .mid file and sometimes it is not.
Then I'm running out of ideas. Things I would check:
- Are the file sizes about the same?
- When you dump the .mid file, you do you see your bank names?
This would tell you if the events are really missing or if mdump would just not find them.
But I guess without being able to debug the system at ILL I cannot be of any more help. You are the
first one reporting such a problem, so it must have to do with your local setup.
Stefan |
1968
|
10 Aug 2020 |
Ivo Schulthess | Bug Report | data missing in runXXXXXX.mid |
> > Dear all
> >
> > We just started our beam time at ILL and just found yesterday that for certain
> > settings of our detector the data is not saved into the .mid files. Running "mdump
> > -l 10" online we see the data coming in as they should. Nevertheless, if we run
> > "mdump -x runXXXXXX.mid" offline, the data file has no events and the banks are
> > missing. Any ideas where the data could go lost?
> >
> > Thanks in advance,
> > Ivo
>
> Have you checked
>
> /Logger/Channels/0/Settings/Event ID = -1
> /Logger/Channels/0/Settings/Trigger mask = -1
>
> If these settings are not -1, they filter the data stream for certain events and trigger
> masks.
>
> Stefan
Good morning Stefan
Both set to -1. We only have one logging channel. If we run a sequence with a few runs and the
same settings, sometimes data is in the .mid file and sometimes it is not.
Best,
Ivo |
1967
|
10 Aug 2020 |
Stefan Ritt | Bug Report | data missing in runXXXXXX.mid |
> Dear all
>
> We just started our beam time at ILL and just found yesterday that for certain
> settings of our detector the data is not saved into the .mid files. Running "mdump
> -l 10" online we see the data coming in as they should. Nevertheless, if we run
> "mdump -x runXXXXXX.mid" offline, the data file has no events and the banks are
> missing. Any ideas where the data could go lost?
>
> Thanks in advance,
> Ivo
Have you checked
/Logger/Channels/0/Settings/Event ID = -1
/Logger/Channels/0/Settings/Trigger mask = -1
If these settings are not -1, they filter the data stream for certain events and trigger
masks.
Stefan |
1966
|
10 Aug 2020 |
Ivo Schulthess | Bug Report | data missing in runXXXXXX.mid |
Dear all
We just started our beam time at ILL and just found yesterday that for certain
settings of our detector the data is not saved into the .mid files. Running "mdump
-l 10" online we see the data coming in as they should. Nevertheless, if we run
"mdump -x runXXXXXX.mid" offline, the data file has no events and the banks are
missing. Any ideas where the data could go lost?
Thanks in advance,
Ivo |
1965
|
07 Aug 2020 |
Konstantin Olchanski | Info | update of MYSQL history documentation |
I updated the documentation for setting up a MYSQL (MariaDB) database for
recording MIDAS history: https://midas.triumf.ca/MidasWiki/index.php/History_System#Write_MYSQL-history_events
One thing to note: the "writer" user must have the "INDEX" permission, otherwise
many things will not work correctly.
Included are the instructions for importing exiting *.hst history files into the
SQL database: mh2sql --mysql mysql_writer.txt *.hst
Let me know if there is interest in adding support for writing into Postgres SQL
database. We used to support both MySQL and Postgres through the ODBC library,
but in the new code, each database has to be supported through it's native API.
There is code for SQLITE, MYSQL, but no code for Postgres, although it is not too
hard to add.
K.O. |
1964
|
15 Jul 2020 |
Stefan Ritt | Info | Minimal CMakeLists.txt for your midas front-end |
Since a few people asked me, here is a "minimal" CMakeLists.txt file for a user-written front-end
program "myfe":
---------------------------
cmake_minimum_required(VERSION 3.0)
project(myfe)
# Check for MIDASSYS environment variable
if (NOT DEFINED ENV{MIDASSYS})
message(SEND_ERROR "MIDASSYS environment variable not defined.")
endif()
set(CMAKE_CXX_STANDARD 11)
set(MIDASSYS $ENV{MIDASSYS})
if (${CMAKE_SYSTEM_NAME} MATCHES Linux)
set(LIBS -lpthread -lutil -lrt)
endif()
add_executable(myfe myfe.cxx)
target_include_directories(myfe PRIVATE ${MIDASSYS}/include)
target_link_libraries(crfe ${MIDASSYS}/lib/libmfe.a ${MIDASSYS}/lib/libmidas.a ${LIBS}) |
1963
|
15 Jul 2020 |
Stefan Ritt | Info | Makefile update |
Please note that you can also compile midas in the standard cmake way with
$ mkdir build
$ cd build
$ cmake ..
$ make install
in the root midas directory. You might have to use "cmake3" on some systems.
Stefan |
1961
|
28 Jun 2020 |
Konstantin Olchanski | Info | Makefile update |
I reworked the MIDAS Makefile to simplify things and to remove redundancy with functions
provided by cmake.
When you say "make", the list of options is printed.
The first and main options are "make cmake" and "make cclean" to run the cmake build.
This is my recommended way to build midas - the output of "make cmake" was tuned to provide
the information need to debug build problems (all compiler commands, command line switches
and file paths are reported). (normal "cmake VERBOSE=1" is tuned for debugging of cmake and
for maximum obfuscation of problems building the actual project).
Build options are implemented through cmake variables:
options that can be added to "make cmake":
NO_LOCAL_ROUTINES=1 NO_CURL=1
NO_ROOT=1 NO_ODBC=1 NO_SQLITE=1 NO_MYSQL=1 NO_SSL=1 NO_MBEDTLS=1
NO_EXPORT_COMPILE_COMMANDS=1
for example "make cmake NO_ROOT=1" to disable auto-detection of ROOT.
Two more make targets create reduced builds of midas:
"make mini" builds a subset of midas suitable for building frontend programs. Big programs
like mlogger and mhttpd are excluded, optional components like CURL or SQLITE are not needed.
"make remoteonly" builds a subset of midas suitable for building remotely connected
frontends. Big parts of midas are excluded, many system-dependent functions are excluded,
etc. This is intended for embedded applications, such as fpga, uclinux, etc.
But wait, there is more. Here is the full list:
daqubuntu:midas$ make
Usage:
make cmake --- full build of midas
make cclean --- remove everything build by make cmake
options that can be added to "make cmake":
NO_LOCAL_ROUTINES=1 NO_CURL=1
NO_ROOT=1 NO_ODBC=1 NO_SQLITE=1 NO_MYSQL=1 NO_SSL=1 NO_MBEDTLS=1
NO_EXPORT_COMPILE_COMMANDS=1
make dox --- run doxygen, results are in ./html/index.html
make cleandox --- remove doxygen output
make htmllint --- run html check on resources/*.html
make test --- run midas self test
make mbedtls --- enable mhttpd support for https via the mbedtls https library
make update_mbedtls --- update mbedtls to latest version
make clean_mbedtls --- remove mbedtls from this midas build
make mtcpproxy --- build the https proxy to forward root-only port 443 to mhttpd https
port 8443
make mini --- minimal build, results are in linux/{bin,lib}
make cleanmini --- remove everything build by make mini
make remoteonly --- minimal build, remote connetion only, results are in linux-
remoteonly/{bin,lib}
make cleanremoteonly --- remove everything build by make remoteonly
make linux32 --- minimal x86 -m32 build, results are in linux-m32/{bin,lib}
make clean32 --- remove everything built by make linux32
make linux64 --- minimal x86 -m64 build, results are in linux-m64/{bin,lib}
make clean64 --- remove everything built by make linux64
make linuxarm --- minimal ARM cross-build, results are in linux-arm/{bin,lib}
make cleanarm --- remove everything built by make linuxarm
make clean --- run all 'clean' commands
daqubuntu:midas$
K.O. |
1960
|
28 Jun 2020 |
Konstantin Olchanski | Info | mhttpd https support openssl -> mbedtls |
To add. Using https with either openssl or mbedtls requires obtaining an https certificate. This can be self-
signed, or signed by a higher authority, or issued by the "let's encrypt" project.
mhttpd is looking for this certificate in the file ssl_cert.pem.
If this file does not exist, mhttpd will print the instructions for creating it using openssl (self-signed) or
using certbot (instantaneously and automatically issued let's encrypt certificate).
The certbot route is recommended:
1) (as root) setup certbot (i.e. see my CentOS and Ubuntu instructions on DAQWiki)
2) (as root) copy /etc/letsencrypt/live/$HOME/fullchain.pem and privkey.pem to $MIDASSYS
3) cat fullchain.pem privkey.pem > ssl_cert.pem
4) start mhttpd, watch the first few lines it prints to confirm it found the right certificate file.
The only missing piece for using this in production is lack of integration
with certbot automatic certificate renewal:
- a script has to run for steps (2) and (3) above
- mhttpd has to tell openssl/mbedtls to reload the certificate file (alternative is to automatically restart
mhttpd, bad!).
As an alternative, we can wait for the mongoose web server library and for the mbedtls crypto library to "grow"
certbot-style automatic certificate renewal features. (unavoidable, in my view).
K.O. |