Back Midas Rome Roody Rootana
  Midas DAQ System  Not logged in ELOG logo
Entry  05 Sep 2024, Jack Carlton, Forum, Python frontend rate limitations? frontend.pyfrontend.cxx
    Reply  05 Sep 2024, Ben Smith, Forum, Python frontend rate limitations? 
       Reply  05 Sep 2024, Stefan Ritt, Forum, Python frontend rate limitations? 
          Reply  06 Sep 2024, Jack Carlton, Forum, Python frontend rate limitations? 
          Reply  11 Sep 2024, Konstantin Olchanski, Forum, Python frontend rate limitations? 
       Reply  27 Sep 2024, Ben Smith, Forum, Python frontend rate limitations? 
    Reply  11 Sep 2024, Konstantin Olchanski, Forum, Python frontend rate limitations? 
       Reply  11 Sep 2024, Konstantin Olchanski, Forum, Python frontend rate limitations? 
Message ID: 2831     Entry time: 11 Sep 2024     In reply to: 2825     Reply to this: 2833
Author: Konstantin Olchanski 
Topic: Forum 
Subject: Python frontend rate limitations? 
> I'm trying to get a sense of the rate limitations of a python frontend.

1) python is single-threaded, for ultimate performance, a MIDAS frontend (or any DAQ 
application) has to be multithreaded:
a) thread with busy loop read the data and place it into a FIFO
b) thread to read data from FIFO and send it to SYSTEM buffer shared memory or to 
mserver
c) thread to respond to begin-run, end-run, etc RPCs
d) probably a thread to recycle memory from thread (b) back to thread (a) if per-event 
malloc()/free() adds too much overhead

2) data readout. C++ AXI bus access is compiled into 1 instruction and results in 1 AXI 
bus operation. comparable for python likely has much more overhead, slows you down.

3) event bank filling. C++ for() loop is compiled into very compact machine code, 
python loop cannot because each array element can be random data type, shows you down.

bottom line, there is a reason high speed data acquisitions are written in C/C++, not 
in shell, perl, tcl/tk, or (today's favourite) python.

> The C++ frontend is about 100 times faster in both data and event rates.

This is as expected. You can probably improve python code to get closer to 10 times 
slower than C++. But consider:

a) will it be "fast enough" for the task?
b) learning C++ and optimizing python to within "2-3-10x slower than C++" may involve a 
similar amount of time and effort.

And you have not looked at the real-time properties of your frontend. You may discover 
that it's actually faster than you think, but occasionally stops for a millisecond (or 
two or hundred). some applications a notorious for running memory garbage collection 
just at the wrong time.

I am working right now on exactly this problem, I have a 1 GHz ARM CPU (Cyclone-V FPGA) 
and I need to push data out at 100 Mbytes/sec while avoiding and bad-real-time dropouts  
that cause the FPGA data FIFO to overflow. And I only have 2 CPU cores, 1 to read the 
FPGA FIFO, 1 to run the TCP/IP stack and the ethernet driver. No this can be done with 
python.

K.O.
ELOG V3.1.4-2e1708b5