Back Midas Rome Roody Rootana
  Midas DAQ System  Not logged in ELOG logo
Entry  05 Sep 2024, Jack Carlton, Forum, Python frontend rate limitations? frontend.pyfrontend.cxx
    Reply  05 Sep 2024, Ben Smith, Forum, Python frontend rate limitations? 
       Reply  05 Sep 2024, Stefan Ritt, Forum, Python frontend rate limitations? 
          Reply  06 Sep 2024, Jack Carlton, Forum, Python frontend rate limitations? 
          Reply  11 Sep 2024, Konstantin Olchanski, Forum, Python frontend rate limitations? 
       Reply  27 Sep 2024, Ben Smith, Forum, Python frontend rate limitations? 
    Reply  11 Sep 2024, Konstantin Olchanski, Forum, Python frontend rate limitations? 
       Reply  11 Sep 2024, Konstantin Olchanski, Forum, Python frontend rate limitations? 
Message ID: 2860     Entry time: 27 Sep 2024     In reply to: 2826
Author: Ben Smith 
Topic: Forum 
Subject: Python frontend rate limitations? 
> in your case the limitation is likely that you're sending 1.25MB per event and we have a lot of data marshalling to do between the python and C++ layer.
> 
> I'll add it to my to-do list to investigate improving the performance of medium-to-large events in the python code.

I've now added better support for numpy arrays in the python code that encodes a `midas.event.Event` object. If you use the "correct" numpy data type then you can get vastly improved performance as numpy already stores the data in memory in the format that we need.

In your example, if you change
        self.zero_buffer = [0] * self.total_data_size
to 
        self.zero_buffer = np.ndarray(self.total_data_size, np.int16)

then the max data rate of the frontend goes from 330MB/s to 7600MB/s on my laptop (a factor 20 improvement from one line of code!) 

To ensure you're using the optimal numpy dtype for your bank, you can reference a dict called `midas.tid_np_formats`. For example `midas.tid_np_formats[midas.TID_SHORT]` is equivalent to `np.int16`. If you use an int16 array and write it as a TID_SHORT bank, then we'll use the fast path. If there is a mismatch, we'll have to do type conversions and will end up on the slow path.
ELOG V3.1.4-2e1708b5