|
Back
Midas
Rome
Roody
Rootana
|
Midas DAQ System |
Not logged in |
|
|
05 Sep 2024, Jack Carlton, Forum, Python frontend rate limitations?
|
05 Sep 2024, Ben Smith, Forum, Python frontend rate limitations?
|
05 Sep 2024, Stefan Ritt, Forum, Python frontend rate limitations?
|
06 Sep 2024, Jack Carlton, Forum, Python frontend rate limitations?
|
11 Sep 2024, Konstantin Olchanski, Forum, Python frontend rate limitations?
|
27 Sep 2024, Ben Smith, Forum, Python frontend rate limitations?
|
11 Sep 2024, Konstantin Olchanski, Forum, Python frontend rate limitations?
|
11 Sep 2024, Konstantin Olchanski, Forum, Python frontend rate limitations?
|
|
Message ID: 2860
Entry time: 27 Sep 2024
In reply to: 2826
|
Author: |
Ben Smith |
Topic: |
Forum |
Subject: |
Python frontend rate limitations? |
|
|
> in your case the limitation is likely that you're sending 1.25MB per event and we have a lot of data marshalling to do between the python and C++ layer.
>
> I'll add it to my to-do list to investigate improving the performance of medium-to-large events in the python code.
I've now added better support for numpy arrays in the python code that encodes a `midas.event.Event` object. If you use the "correct" numpy data type then you can get vastly improved performance as numpy already stores the data in memory in the format that we need.
In your example, if you change
self.zero_buffer = [0] * self.total_data_size
to
self.zero_buffer = np.ndarray(self.total_data_size, np.int16)
then the max data rate of the frontend goes from 330MB/s to 7600MB/s on my laptop (a factor 20 improvement from one line of code!)
To ensure you're using the optimal numpy dtype for your bank, you can reference a dict called `midas.tid_np_formats`. For example `midas.tid_np_formats[midas.TID_SHORT]` is equivalent to `np.int16`. If you use an int16 array and write it as a TID_SHORT bank, then we'll use the fast path. If there is a mismatch, we'll have to do type conversions and will end up on the slow path. |