ID |
Date |
Author |
Topic |
Subject |
2493
|
01 May 2023 |
Giovanni Mazzitelli | Bug Report | python issue with mathplot lib vs odb query | Ciao,
we have a very strange issue with python lib with client.odb_get("/") function
when running as midas process and matplotlib is used.
we are developing a remote console by means of sending via kafka producer the odb,
camera image and pmt waveforms, in the INFN cloud where grafana make available
data for non expert shifters, as well as sending midas events for online
reconstruction to the htcondr queue on cloud. The process work perfectly and allow
use to parallelise to standard midas pipeline for file production, ecc the online
monitoring and data processing where we have computing resources (our DAQ is
underground at LNGS). Part of the work will be presented next weak at CHEP
the full code is available at https://github.com/CYGNUS-
RD/middleware/blob/master/dev/event_producer_s3.py
but to get the strange behaviour I report here a test script:
----
def main(verbose=False):
from matplotlib import pyplot as plt
import time
import midas
import midas.client
client = midas.client.MidasClient("middleware")
buffer_handle = client.open_event_buffer("SYSTEM",None,1000000000)
request_id = client.register_event_request(buffer_handle, sampling_type = 2)
fpath = os.path.dirname(os.path.realpath(sys.argv[0]))
while True:
#
odb = client.odb_get("/")
if verbose:
print(odb)
start1 = time.time()
client.communicate(10)
time.sleep(1)
client.deregister_event_request(buffer_handle, request_id)
client.disconnect()
----
if I run it as cli interactivity including or not matplotlib the everything si ok.
As I run it as midas "program" I get:
-----
Traceback (most recent call last):
File "/home/standard/daq/middleware/dev/test_midas_error.py", line 48, in
<module>
main(verbose=options.verbose)
File "/home/standard/daq/middleware/dev/test_midas_error.py", line 29, in main
odb = client.odb_get("/")
File "/home/standard/packages/midas/python/midas/client.py", line 354, in
odb_get
retval = midas.safe_to_json(buf.value, use_ordered_dict=True)
File "/home/standard/packages/midas/python/midas/__init__.py", line 552, in
safe_to_json
return json.loads(decoded, strict=False,
object_pairs_hook=collections.OrderedDict)
File "/usr/lib/python3.8/json/__init__.py", line 370, in loads
return cls(**kw).decode(s)
File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.8/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes:
line 300 column 26 (char 17535)
----
if I comment out the import of matplotlib every think works perfectly again also
as midas program.
it seams that there is a difference between the to way of use the code, and that
is sufficient the call to matplotlib to corrupt in some way the odb. any ideas? |
2494
|
01 May 2023 |
Ben Smith | Bug Report | python issue with mathplot lib vs odb query | > it seams that there is a difference between the to way of use the code, and that
> is sufficient the call to matplotlib to corrupt in some way the odb. any ideas?
I can't reproduce this on my machines, so this is going to be fun to debug!
Can you try running the program below please? It takes the important bits from odb_get() but prints out the string before we try to parse it as JSON. Feel free to send me the output via email (bsmith@triumf.ca) if you don't want to post your entire ODB dump in the elog.
import sys
import os
import time
import midas
import midas.client
import ctypes
def debug_get(client):
c_path = ctypes.create_string_buffer(b"/")
hKey = ctypes.c_int()
client.lib.c_db_find_key(client.hDB, 0, c_path, ctypes.byref(hKey))
buf = ctypes.c_char_p()
bufsize = ctypes.c_int()
bufend = ctypes.c_int()
client.lib.c_db_copy_json_save(client.hDB, hKey, ctypes.byref(buf), ctypes.byref(bufsize), ctypes.byref(bufend))
print("-" * 80)
print("FULL DUMP")
print("-" * 80)
print(buf.value)
print("-" * 80)
print("Chars 17000-18000")
print("-" * 80)
print(buf.value[17000:18000])
print("-" * 80)
as_dict = midas.safe_to_json(buf.value, use_ordered_dict=True)
client.lib.c_free(buf)
return as_dict
def main(verbose=False):
client = midas.client.MidasClient("middleware")
buffer_handle = client.open_event_buffer("SYSTEM",None,1000000000)
request_id = client.register_event_request(buffer_handle, sampling_type = 2)
fpath = os.path.dirname(os.path.realpath(sys.argv[0]))
while True:
# odb = client.odb_get("/")
odb = debug_get(client)
if verbose:
print(odb)
start1 = time.time()
client.communicate(10)
time.sleep(1)
client.deregister_event_request(buffer_handle, request_id)
client.disconnect()
if __name__ == "__main__":
main() |
Draft
|
01 May 2023 |
Giovanni Mazzitelli | Bug Report | python issue with mathplot lib vs odb query | > > it seams that there is a difference between the to way of use the code, and that
> > is sufficient the call to matplotlib to corrupt in some way the odb. any ideas?
>
> I can't reproduce this on my machines, so this is going to be fun to debug!
>
> Can you try running the program below please? It takes the important bits from odb_get() but prints out the string before we try to parse it as JSON. Feel free to send me the output via email (bsmith@triumf.ca) if you don't want to post your entire ODB dump in the elog.
>
>
>
>
> import sys
> import os
> import time
> import midas
> import midas.client
> import ctypes
>
> def debug_get(client):
> c_path = ctypes.create_string_buffer(b"/")
> hKey = ctypes.c_int()
> client.lib.c_db_find_key(client.hDB, 0, c_path, ctypes.byref(hKey))
>
> buf = ctypes.c_char_p()
> bufsize = ctypes.c_int()
> bufend = ctypes.c_int()
>
> client.lib.c_db_copy_json_save(client.hDB, hKey, ctypes.byref(buf), ctypes.byref(bufsize), ctypes.byref(bufend))
>
> print("-" * 80)
> print("FULL DUMP")
> print("-" * 80)
> print(buf.value)
> print("-" * 80)
> print("Chars 17000-18000")
> print("-" * 80)
> print(buf.value[17000:18000])
> print("-" * 80)
>
> as_dict = midas.safe_to_json(buf.value, use_ordered_dict=True)
>
> client.lib.c_free(buf)
>
> return as_dict
>
> def main(verbose=False):
> client = midas.client.MidasClient("middleware")
> buffer_handle = client.open_event_buffer("SYSTEM",None,1000000000)
> request_id = client.register_event_request(buffer_handle, sampling_type = 2)
>
> fpath = os.path.dirname(os.path.realpath(sys.argv[0]))
>
> while True:
> # odb = client.odb_get("/")
> odb = debug_get(client)
>
> if verbose:
> print(odb)
> start1 = time.time()
>
> client.communicate(10)
> time.sleep(1)
>
>
> client.deregister_event_request(buffer_handle, request_id)
>
> client.disconnect()
>
> if __name__ == "__main__":
> main()
Thank you!
if I added the mat |
2496
|
01 May 2023 |
Giovanni Mazzitelli | Bug Report | python issue with mathplot lib vs odb query | > > it seams that there is a difference between the to way of use the code, and that
> > is sufficient the call to matplotlib to corrupt in some way the odb. any ideas?
>
> I can't reproduce this on my machines, so this is going to be fun to debug!
>
> Can you try running the program below please? It takes the important bits from odb_get() but prints out the string before we try to parse it as JSON. Feel free to send me the output via email (bsmith@triumf.ca) if you don't want to post your entire ODB dump in the elog.
Thank you!
if I added the matplotlib as follow:
#!/usr/bin/env python3
import sys
import os
import time
import midas
import midas.client
import ctypes
from matplotlib import pyplot as plt
def debug_get(client):
c_path = ctypes.create_string_buffer(b"/")
hKey = ctypes.c_int()
client.lib.c_db_find_key(client.hDB, 0, c_path, ctypes.byref(hKey))
buf = ctypes.c_char_p()
bufsize = ctypes.c_int()
bufend = ctypes.c_int()
client.lib.c_db_copy_json_save(client.hDB, hKey, ctypes.byref(buf), ctypes.byref(bufsize), ctypes.byref(bufend))
print("-" * 80)
print("FULL DUMP")
print("-" * 80)
print(buf.value)
print("-" * 80)
print("Chars 17000-18000")
print("-" * 80)
print(buf.value[17000:18000])
print("-" * 80)
as_dict = midas.safe_to_json(buf.value, use_ordered_dict=True)
client.lib.c_free(buf)
return as_dict
def main(verbose=False):
client = midas.client.MidasClient("middleware")
buffer_handle = client.open_event_buffer("SYSTEM",None,1000000000)
request_id = client.register_event_request(buffer_handle, sampling_type = 2)
fpath = os.path.dirname(os.path.realpath(sys.argv[0]))
while True:
# odb = client.odb_get("/")
odb = debug_get(client)
if verbose:
print(odb)
start1 = time.time()
client.communicate(10)
time.sleep(1)
client.deregister_event_request(buffer_handle, request_id)
client.disconnect()
if __name__ == "__main__":
from optparse import OptionParser
parser = OptionParser(usage='usage: %prog\t ')
parser.add_option('-v','--verbose', dest='verbose', action="store_true", default=False, help='verbose output;');
(options, args) = parser.parse_args()
main(verbose=options.verbose)
then tested the code in interactive mode without any error. as soon as I submit as midas "Program" I get the attached output.
thank you again, Giovanni |
2497
|
01 May 2023 |
Ben Smith | Bug Report | python issue with mathplot lib vs odb query | Looks like a localisation issue. Your floats are formatted as "6,6584e+01", whereas the JSON decoder expects "6.6584e+01".
Can you run the following few lines please? Then I'll be able to write a test using the same setup as you:
import locale
print(locale.getlocale())
from matplotlib import pyplot as plt
print(locale.getlocale())
|
2498
|
01 May 2023 |
Ben Smith | Bug Report | python issue with mathplot lib vs odb query | > Looks like a localisation issue. Your floats are formatted as "6,6584e+01", whereas the JSON decoder expects "6.6584e+01".
This should be fixed in the latest commit to the midas develop branch. The JSON specification requires a dot for the decimal separator, so we must ignore the user's locale when formatting floats/doubles for JSON.
I've tested the fix on my machine by manually changing the locale, and also added an automated test in the python directory. |
2499
|
01 May 2023 |
Giovanni Mazzitelli | Bug Report | python issue with mathplot lib vs odb query | > > Looks like a localisation issue. Your floats are formatted as "6,6584e+01", whereas the JSON decoder expects "6.6584e+01".
>
> This should be fixed in the latest commit to the midas develop branch. The JSON specification requires a dot for the decimal separator, so we must ignore the user's locale when formatting floats/doubles for JSON.
>
> I've tested the fix on my machine by manually changing the locale, and also added an automated test in the python directory.
Thanks very macth Ben,
so if I understand correctly we have to update MIDAS to latest develop branch available? can you sand me the link to be sure of install the right update.
can you also tell me how you fix manually? we are restarting and then well be difficult install and makes updete.
thank you again, regards, Giovanni |
2516
|
16 May 2023 |
Konstantin Olchanski | Bug Report | excessive logging of http requests | Our default configuration of apache httpd logs every request. MIDAS custom web pages can easily make a huge number of RPC calls creating a
huge log file and filling system disk to 100% capacity
this example has around 100 RPC requests per second. reasonable/unreasonable? available hardware can handle it (web browser, network
httpd, mhttpd, etc), so we should try to get this to work. perhaps filter the apache httpd logs to exclude mjsonrpc requests? of course we
can ask the affected experiment why they make so many RPC calls, is there a bug?
[14/May/2023:03:49:01 -0700] 142.90.111.176 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "POST /?mjsonrpc HTTP/1.1" 299
[14/May/2023:03:49:01 -0700] 142.90.111.176 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "POST /?mjsonrpc HTTP/1.1" 299
[14/May/2023:03:49:01 -0700] 142.90.111.176 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "POST /?mjsonrpc HTTP/1.1" 299
[14/May/2023:03:49:01 -0700] 142.90.111.176 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "POST /?mjsonrpc HTTP/1.1" 299
[14/May/2023:03:49:01 -0700] 142.90.111.176 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "POST /?mjsonrpc HTTP/1.1" 299
K.O. |
2517
|
16 May 2023 |
Konstantin Olchanski | Bug Report | excessive logging of http requests | > Our default configuration of apache httpd logs every request. MIDAS custom web pages can easily make a huge number of RPC calls creating a
> huge log file and filling system disk to 100% capacity
perhaps use existing logrotate, add limit on file size (size) and limit of 2 old log files (rotate).
/etc/logrotate.d/httpd
/var/log/httpd/*log {
size 100M
rotate 2
missingok
notifempty
sharedscripts
delaycompress
postrotate
/bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true
endscript
}
K.O. |
2518
|
16 May 2023 |
Stefan Ritt | Bug Report | excessive logging of http requests | Maybe you remember the problems we had with a custom page in Japan loading it from TRIUMF. It took almost one minute since each RPC request took
about 1s round-trip. This got fixed by the modb* scheme where the framework actually collects all ODB variables in a custom page and puts them
into ONE rpc request (making the path an actual array of paths). That reduced the requests from 100 to 1 in the above example. Maybe the same
could be done in your current case. Pulling one ODB variable at a time is not very efficient.
Stefan |
2519
|
23 May 2023 |
Kou Oishi | Bug Report | Event builder fails at every 10 runs | Dear MIDAS experts,
Greetings!
I am currently utilizing MIDAS for our experiment and I have encountered an issue with our event builder, which was developed based on the example code 'eventbuilder/mevb.cxx'. I'm uncertain whether this is a genuine bug or an inherent feature of MIDAS.
The event builder fails to initiate the 10th run since its startup, requiring us to relaunch it. Upon investigating the code, I have identified that this issue stems from line 8404 of mfe.cxx (the version's hash is db94df6fa79772c49888da9374e143067a1fff3a). According to the code, the 10-run limit is imposed by the variable MAX_EVENT_REQUESTS in midas.h. While I can increase this value as the code suggests, it does not provide a complete solution, as the same problem will inevitably resurface. This complication unnecessarily hampers our data collection during long observation periods.
Despite the code indicating 'BM_NO_MEMORY: too many requests,' this explanation does not seem logical to me. In fact, other standard frontends do not encounter this problem and can start new runs as required without requiring a frontend relaunch.
I apologize for not yet fully grasping the intricate implementation of midas.cxx and mfe.cxx. However, I would greatly appreciate any suggestions or insights you can offer to help resolve this issue.
Thank you in advance for your kind assistance. |
2521
|
31 May 2023 |
Ben Smith | Bug Report | Event builder fails at every 10 runs | > The event builder fails to initiate the 10th run since its startup,
> 'BM_NO_MEMORY: too many requests,'
Hi Kou,
It sounds like you might be calling bm_request_event() when starting a run, but not calling bm_delete_request() when the run stops. So you end up "leaking" event requests and eventually reach the limit of 10 open requests.
In examples/eventbuilder/mevb.c the request deletion happens in source_unbooking(), which is called as part of the "run stopping" logic. I've just updated the midas repository so the example compiles correctly, and was able to start/stop 15 runs without crashing.
Can you check the end-of-run logic in your version to ensure you're calling bm_delete_request()? |
Draft
|
02 Jun 2023 |
Kou Oishi | Bug Report | Event builder fails at every 10 runs | Dear
> > The event builder fails to initiate the 10th run since its startup,
> > 'BM_NO_MEMORY: too many requests,'
>
> Hi Kou,
>
> It sounds like you might be calling bm_request_event() when starting a run, but not calling bm_delete_request() when the run stops. So you end up "leaking" event requests and eventually reach the limit of 10 open requests.
>
> In examples/eventbuilder/mevb.c the request deletion happens in source_unbooking(), which is called as part of the "run stopping" logic. I've just updated the midas repository so the example compiles correctly, and was able to start/stop 15 runs without crashing.
>
> Can you check the end-of-run logic in your version to ensure you're calling bm_delete_request()? |
2524
|
02 Jun 2023 |
Kou Oishi | Bug Report | Event builder fails at every 10 runs | Dear Ben,
Hello. Thank you for your attention to this problem!
> It sounds like you might be calling bm_request_event() when starting a run, but not calling bm_delete_request() when the run stops. So you end up "leaking" event requests and eventually reach the limit of 10 open requests.
I understand. Thanks for the description.
> In examples/eventbuilder/mevb.c the request deletion happens in source_unbooking(), which is called as part of the "run stopping" logic. I've just updated the midas repository so the example compiles correctly, and was able to start/stop 15 runs without crashing.
>
> Can you check the end-of-run logic in your version to ensure you're calling bm_delete_request()?
I really appreciate your update.
Although I am away at the moment from the DAQ development, I will test it and report the result here as soon as possible.
Best regards,
Kou |
2543
|
21 Jun 2023 |
Gennaro Tortone | Bug Report | mserver and script execution |
Hi,
I have the following setup:
- MIDAS release: release/midas-2022-05-c
- host with MIDAS frontend (mclient)
- host with MIDAS server (mhttpd / mserver)
On mclient I run a frontend with:
./feodt5751 -h mserver -e develop -i 0
On mserver I see frontend ready and ODB variables in place;
I noticed a strange behavior with "/Programs/Execute on start run" and
"/Programs/Execute on stop run". In details the script to execute at start of run
is executed on "mserver" host but the script to execute at stop of run is executed on
"mclient" host (!)
Is this a bug or I'm missing some documentation links ?
Thanks in advance,
Gennaro |
2544
|
23 Jun 2023 |
Gennaro Tortone | Bug Report | deferred stop transition | Hi,
I'm facing some issues with 'stop' deferred transition and I suspect of
a MIDAS bug regarding this...
to reproduce the issue I use the 'deferredfe' MIDAS example (develop branch),
changing only the equipment name from 'Deferred' to 'Deferred%02d' in order
to be able to run multiple 'deferredfe' instances;
I run *three* 'deferredfe' frontends using:
./deferredfe -i 0
./deferredfe -i 2
./deferredfe -i 3
Everything goes fine on MIDAS web page and 'deferredfe' frontends are initialized
and ready to run; issues occour after 'start' when I stop the frontends: sometimes
at first shot and sometimes at next 'start'/'stop' the deferred 'stop' transition
seems to be handled in wrong way... and often one frontend goes in 'segmentation fault'
The odd thing is when I run *two* instances: in this case no issues are reported...
Thanks in advance,
Gennaro |
2545
|
23 Jun 2023 |
Stefan Ritt | Bug Report | deferred stop transition | Deferred transitions were only implemented with a single instance of a program deferring the
transition. To have several instances, MIDAS probably needs to be extended. Certainly this
was never tested, so it's not a surprise that we get a segmentation fault.
Stefan |
2546
|
23 Jun 2023 |
Gennaro Tortone | Bug Report | deferred stop transition | Hi Stefan,
so if I have two different frontends (feov1725 and feodt5751) connected on the same 'mserver'
I'm in the same situation ?
Cheers,
Gennaro
> Deferred transitions were only implemented with a single instance of a program deferring the
> transition. To have several instances, MIDAS probably needs to be extended. Certainly this
> was never tested, so it's not a surprise that we get a segmentation fault.
>
> Stefan |
2547
|
23 Jun 2023 |
Stefan Ritt | Bug Report | deferred stop transition | > I'm in the same situation ?
Yepp. |
2548
|
26 Jun 2023 |
Gennaro Tortone | Bug Report | deferred stop transition | > Deferred transitions were only implemented with a single instance of a program deferring the
> transition. To have several instances, MIDAS probably needs to be extended. Certainly this
> was never tested, so it's not a surprise that we get a segmentation fault.
>
> Stefan
Hi Stefan,
I copied deferredfe.cxx to mydeferredfe.cxx and I changed mydeferred.cxx to be a different frontend:
const char *frontend_name = "mydeferredfe";
If I start two "different" frontends:
./deferredfe
./mydeferredfe
and try to start/stop a run... the result is the same: frontend status messing up on next 'start':
---
deferredfe:
Started run 332
Event ID:4 - Event#: 0
Event ID:4 - Event#: 1
Event ID:4 - Event#: 2
Event ID:4 - Event#: 3
Event ID:4 - Event#: 4
Event ID:4 - Event#: 5
Event ID:4 - Event#: 6
mydeferredfe:
Started run 332
Transition ignored, Event ID:2 - Event#: 0
Transition ignored, Event ID:2 - Event#: 1
Transition ignored, Event ID:2 - Event#: 2
Transition ignored, Event ID:2 - Event#: 3
Transition ignored, Event ID:2 - Event#: 4
End of cycle... perform transition
Event ID:2 - Event#: 5
End of cycle... perform transition
Event ID:2 - Event#: 6
---
so, it seems that the issue is not related to different 'instances' of same frontend but
that *at most* one frontend on whole MIDAS server can handle deferred transitions...
is this the case ?
Cheers,
Gennaro |
|