ID |
Date |
Author |
Topic![up up](up.png) |
Subject |
1980
|
11 Aug 2020 |
Ivo Schulthess | Bug Report | data missing in runXXXXXX.mid |
> It would be good to pin point there the data is lost. This is the sequence:
>
> frontend user code -> mfe.c code -> SYSTEM buffer -> mlogger -> disk
>
> To see if correct data arrives to the SYSTEM buffer, run:
> mdump -z SYSTEM
>
> To see if mlogger is receiving events from the SYSTEM buffer, run:
> mlogger -v ### mlogger should report all events, history and data
>
> To see if mlogger writes events to disk, examine the disk file (in this case, you already did, data is not there).
>
> I would guess that your data does not make it out from the frontend (mdump shows "nothing"),
> if data were to arrive into the SYSTEM buffer, it would make it to disk, unless
> mlogger is misconfigured (but you already checked that).
>
> If you have trouble with the frontend framework code, you can try to switch from the mfe.c frontend
> to the newer c++ tmfe frontend (see progs/fetest_tmfe.cxx and progs/fetest_tmfe_thread.cxx).
>
> K.O.
Good evening
I tried to reproduce the behavior in a very simple FE but it did not work out. The next thing for me would be to take the FE that is producing this behavior, replace all the device communication and data with dummies. If the problem is still there I would start to simplify as much as possible.
Following the inputs of KO, I pin-pointed the data loss. The system buffer still gets the data but the mlogger does not write the data event. Then of course the data is also not anymore present in the data file. Therefore, I checked the logger settings again, Event ID and Trigger Mask still -1. Nothing else, at least from my point of view, that is misconfigured. Nevertheless, if it helps I can send my ODB settings.
When doing the tests just before I found something else that probably can give a hint to the problem. The data is only lost if the time between two runs is long (a few seconds). As an example: If I run a sequence with a loop and after the FE stops the run the loop ends and the next run is started automatically, then only the first run has no data, which is the one after a longer time of no data taking. When I add a "WAIT Seconds 5" after the run before starting the next, not data is written to the disk for any run. I also found this once when adding a sleep(1) at the end of the FE readout function but back then did not think about it any further.
Best, Ivo |
2064
|
05 Jan 2021 |
Isaac Labrie Boulay | Bug Report | Logger: Disk nearly full. |
Hi all,
I've ran into a problem where my experiment gets interrupted with a message from
the logger saying that my disk is nearly full. This does not make sense to me
because I have deleted almost all the data files from my data directory. I'm
guessing that somewhere the ODB perceives that the directory is full when in
reality its not.
Here is the exact message:
[ODBEdit,INFO] Run #252 stopped
09:22:19 [Logger,TALK] disk nearly full, stopping the run
09:22:19 [Logger,ERROR] [mlogger.cxx:4475:log_write,ERROR] Disk '/home/caendaq/A
NIS/data/run00252.mid.lz4' is almost full: 81 MiBytes free out of 922497 MiBytes
, stopping the run
Does any body have a solution for this? Thanks so much.
Isaac |
2066
|
06 Jan 2021 |
Stefan Ritt | Bug Report | Logger: Disk nearly full. |
The logger simple requests the disk free space level from the operating system in the same
way as the "df" command does. Can you do a "df" on your system? I have seen that some file
systems free up space not immediately if you delete files, but some times later (like 24h).
Stefan |
2068
|
06 Jan 2021 |
Isaac Labrie Boulay | Bug Report | Logger: Disk nearly full. |
> The logger simple requests the disk free space level from the operating system in the same
> way as the "df" command does. Can you do a "df" on your system? I have seen that some file
> systems free up space not immediately if you delete files, but some times later (like 24h).
>
> Stefan
Thanks Stefan. Yes the files were still held open by some processes. It's solved now.
Cheers.
Isaac |
2091
|
12 Feb 2021 |
Konstantin Olchanski | Bug Report | mlogger history snafu |
there is a problem with mlogger between commits xxx (17 Nov 2020) and a762bb8 (12 feb 2021). because of
confusion between seconds and milliseconds, FILE (mhf*.dat files) and SQL history are recording with
incorrect timestamps.
- traditional MIDAS history (*.hst files) does not have this problem (because of a buglet)
- midas-2020-12 release does not have this problem (it has mlogger from midas-2020-08 release)
there are some additional changes in mlogger that we are sorting out, when ready, we will make a new
release of midas.
K.O. |
2094
|
18 Feb 2021 |
Pintaudi Giorgio | Bug Report | Unexpected end-of-file |
Hello!
Sometimes when I mess around with the history plots I get the following error:
[mhttpd,ERROR] [history.cxx:97:xread,ERROR] Error: Unexpected end-of-file when
reading file "/home/wagasci-ana/Data/online/210219.hst"
I have tried the following without success:
- Remove the MIDAS history files
- Restart mhttpd and mlogger
I do not know what triggers the error but when it triggers the above message is
printed hundres of times a second, completely spamming the message log.
It happened again today after I set the label of a frontend too long making
mlogger crash. After fixing the label length, the above message appeared and it
does not seem to go away. |
2095
|
18 Feb 2021 |
Pintaudi Giorgio | Bug Report | Unexpected end-of-file |
It appears that the issue is trigger by a nonexisting Event and Variable as shown
in the attached picture. This issue can arise when restoring the ODB from a
previous version or importing ODB values from other MIDAS instances.
It might be useful if the error message were more clear about the source of the
problem.
> Hello!
> Sometimes when I mess around with the history plots I get the following error:
>
> [mhttpd,ERROR] [history.cxx:97:xread,ERROR] Error: Unexpected end-of-file when
> reading file "/home/wagasci-ana/Data/online/210219.hst"
>
> I have tried the following without success:
>
> - Remove the MIDAS history files
> - Restart mhttpd and mlogger
>
> I do not know what triggers the error but when it triggers the above message is
> printed hundres of times a second, completely spamming the message log.
>
> It happened again today after I set the label of a frontend too long making
> mlogger crash. After fixing the label length, the above message appeared and it
> does not seem to go away. |
2096
|
24 Feb 2021 |
Zaher Salman | Bug Report | history reload |
I have a history that is embedded in a custom page using
<div class="mjshistory" data-group="SampleCryo" data-panel="SampleTemp" data-scale="30m" style="'+size+' position: relative;left: 640px;top: -205px;"></div>
this works fine when I load the page but seems to cause a timeout when reloading (F5) the page. It used to work fine last year but since a midas update this year it does not work.
When I manually stop the script when firefox reports that it is slowing down the browse I get the following in the console:
Script terminated by timeout at:
binarySearch@http://xxx.psi.ch:8081/mhistory.js:1051:11
MhistoryGraph.prototype.findMinMax@http://xxx.psi.ch:8081/mhistory.js:1583:28
MhistoryGraph.prototype.loadInitialData/<@http://xxx.psi.ch:8081/mhistory.js:780:15
any ideas what may be causing this?
thanks.
Another hint to the problem, the custom page is accessible via
http://xxx.psi.ch:8081/?cmd=custom&page=SampleCryo&
once loaded the address changes to where A and B change values as time passes (I guess B-A=30m).
http://xxx.psi.ch:8081/?cmd=custom&page=SampleCryo&&A=1614173369&B=1614175169 |
2097
|
25 Feb 2021 |
Stefan Ritt | Bug Report | history reload |
I have to reproduce the problem. Can you please send me the full link by direct email. As you know, I'm also at PSI.
Stefan |
2098
|
25 Feb 2021 |
Isaac Labrie Boulay | Bug Report | Undefined client causing issues in transition. |
Hi all,
I'm currently experiencing an issue during run transitions. It comes in the form
of an alert saying "TypeError: Cannot read property 'length' of undefined"
whenever I'm in the "transition" window on mhttpd. I have attached an image of
what the transition window looks like when this happens.
By the looks of it and by peering at the lines in transition.html where the
error occurs, it's pretty obvious that there is some strange undefined client
that the web page tries to access.
I don't know how to find what this client is. Is there a way to see it in the
ODB?
The issues happens in show_client() of transition.html (called by callback()).
Here's the trace:
Uncaught (in promise) TypeError: Cannot read property 'length' of undefined
at show_client (?cmd=Transition:227)
at callback (?cmd=Transition:420)
at ?cmd=Transition:430
Any help would be very appreciated!
Thanks so much.
Isaac |
2099
|
25 Feb 2021 |
Lars Martin | Bug Report | tmfe_main.cxx missing include <signal.h> |
The most recent commit (b43aef648c2f8a7e710a327d0b322751ae44afea) throws this
compiler error:
src/tmfe_main.cxx:39:11: error: 'SIGPIPE' was not declared in this scope
signal(SIGPIPE, SIG_IGN);
It's fixed by adding #include <signal.h> to that file. |
2101
|
25 Feb 2021 |
Konstantin Olchanski | Bug Report | tmfe_main.cxx missing include <signal.h> |
> The most recent commit (b43aef648c2f8a7e710a327d0b322751ae44afea) throws this
> compiler error:
> src/tmfe_main.cxx:39:11: error: 'SIGPIPE' was not declared in this scope
> signal(SIGPIPE, SIG_IGN);
>
> It's fixed by adding #include <signal.h> to that file.
"but it works just fine on my mac!"
anyhow, thank you for reporting this problem, it already fixed. the bitbucket auto-
build also caught it. I also boogered up "make remoteonly", also fixed now.
BTW, for production use I recommend midas from the "release" branches, unless one
needs a bug fix or new feature from the development branch.
K.O. |
2103
|
25 Feb 2021 |
Konstantin Olchanski | Bug Report | Undefined client causing issues in transition. |
Clearly something goes wrong with the STARTABORT transition. Actually from your
sceenshot, it is not clear why the STARTABORT transition was initiated.
Usually it is called after some client fails the "start run" transition to inform
other clients that the run did not start after all. (mlogger uses this to close the
output file, etc).
But in the screenshot, we do not see any client fail the transition (only rootana1
was called, and it returned "green").
So, a puzzle. One possibility is that the transition code gets so confused
that it does not record correct transition data to ODB, then the web page
gets even more confused.
One way to see what happens, is to run the odbedit command "start now -v".
Can you try that? And attach all it's output here?
K.O. |
2104
|
25 Feb 2021 |
Konstantin Olchanski | Bug Report | Unexpected end-of-file |
> > [mhttpd,ERROR] [history.cxx:97:xread,ERROR] Error: Unexpected end-of-file when
> > reading file "/home/wagasci-ana/Data/online/210219.hst"
I am puzzled. We can try two things:
a) look inside the "bad" hst file, maybe we can see something. run "mhdump -L
/home/wagasci-ana/Data/online/210219.hst". If there is anything wrong with the file, it
will be probably at the end. You can also try to run it without "-L".
b) switch from "midas" history (.hst files) to "FILE" history (mh*.dat files), the
"FILE" history code is newer and the file format is more robust, with luck it may
survive whatever trouble is happening in your experiment. This is controlled in ODB
/Logger/History/XXX/Active (set to "y/n").
c) the output of "mlogger -v" may give us some clue, it usually complains if something
is not right with definitions of history data.
K.O. |
2109
|
26 Feb 2021 |
Isaac Labrie Boulay | Bug Report | Undefined client causing issues in transition. |
> Clearly something goes wrong with the STARTABORT transition. Actually from your
> sceenshot, it is not clear why the STARTABORT transition was initiated.
>
> Usually it is called after some client fails the "start run" transition to inform
> other clients that the run did not start after all. (mlogger uses this to close the
> output file, etc).
>
> But in the screenshot, we do not see any client fail the transition (only rootana1
> was called, and it returned "green").
>
> So, a puzzle. One possibility is that the transition code gets so confused
> that it does not record correct transition data to ODB, then the web page
> gets even more confused.
>
> One way to see what happens, is to run the odbedit command "start now -v".
>
> Can you try that? And attach all its output here?
>
> K.O.
Thanks for getting back to me right away. I've attached two screenshots. The first one
is the output after running "start now -v" (everything seemed to work nicely there), the
second output is after using odbedit to stop the run with "stop". Notice that the DAQ
never stops because it gets stuck in between transitions (You can see the run status
being "stopping run" with the cancel transition button).
Thanks.
Isaac |
2111
|
26 Feb 2021 |
Konstantin Olchanski | Bug Report | Undefined client causing issues in transition. |
So there is no error on run start anymore? To debug the stuck run stop, please use "stop -v"
to see where it got stuck. You can also play with the RPC timeouts (the connect timeout and
the response timeout), to make it get "unstuck" quicker. Definitely it should not be stuck
forever, it should timeout at maximum of "rpc timeout * number of clients". K.O. |
2112
|
26 Feb 2021 |
Lars Martin | Bug Report | tmfe_main.cxx missing include <signal.h> |
> BTW, for production use I recommend midas from the "release" branches, unless one
> needs a bug fix or new feature from the development branch.
Fair point. I would suggest adding that recommendation to the wiki instructions. I
forget to add that step otherwise. |
2113
|
26 Feb 2021 |
Isaac Labrie Boulay | Bug Report | Undefined client causing issues in transition. |
> So there is no error on run start anymore? To debug the stuck run stop, please use "stop -v"
> to see where it got stuck. You can also play with the RPC timeouts (the connect timeout and
> the response timeout), to make it get "unstuck" quicker. Definitely it should not be stuck
> forever, it should timeout at maximum of "rpc timeout * number of clients". K.O.
You're right it does not stay stuck forever, it eventually gets unstuck. I forgot to mention this.
I will try to play with these timeout parameters. It does not get stuck if I run my DAQ using the
odbedit commands (start/stop). I don't know if this is relevant information that could help us
identify the problem.
Thanks for all your help as always!
Isaac |
2119
|
03 Mar 2021 |
Konstantin Olchanski | Bug Report | Undefined client causing issues in transition. |
> It does not get stuck if I run my DAQ using the odbedit commands (start/stop).
Interesting. Run start/stop from odbedit works but from mhttpd gets stuck.
I think they do not run the transition quite the same way. mhttpd uses the multithreaded transition.
So we can debug this using the "mtransition" program. Try:
- mtransition -v -d 1 START/STOP -- this should be same as odbedit
- mtransition -m -v -d 1 START/STOP -- this should be same as mhttpd
The "-v" and "-d 1" flags should cause lots of output, for failed transitions,
cut-and-paste it all into this elog here, should give us plenty of meat to debug.
K.O. |
2125
|
05 Mar 2021 |
Svetlana Chesnevskaya | Bug Report | New MIDAS old frontend incompatibility |
Hello!
Could you help me solve the problem of compatibility between our frontend (created in 2017) and the fresh MIDAS? The old MIDAS (2017) worked well, then we did not use it.
While compiling the frontend, I get a lot of warnings and a few compilation errors.
Any help will be greatly appreciated.
Thanks in advance.
With the best regards,
Svetlana |