Starting from SVN revision 3642, sending events from the front-end has been revised.
Since long time ago, there is a special TCP socket established between any front-end and the mserver which can be used to bypass the midas RPC layer completely and purely send events. There was a #define USE_EVENT_CHANNEL but to my knowledge nobody used it.
While optimizing data throughput for the MEG experiment, I revisited this mechanism and got it finally working. Here are some benchmark tests made with the produce program on two dual-CPU machines running on Gigabit Ethernet:
Using normal RPC socket:
event size speed [MB/sec] CPU usage front-end CPU usage server
==================================================================
40 3 22 100
1000 44 25 100
100000 101 14 50
Using new event socket:
event size speed [MB/sec] CPU usage front-end CPU usage server
==================================================================
40 12 100 34
1000 99 58 59
100000 101 14 43
As can be seen, the CPU load on the server drops significantly for smaller events since the processing time per event is reduced. If the transfer was limited by the server, the throughput goes up significantly. For large events the bottleneck on the server side is the memcpy of events, so no big improvement is visible. The saved CPU time however can be used to analyze more events for example.
The event socket is now enabled by default in the front-end by setting
rpc_mode = 1
in mfe.c and should be checked carefully in various experiments. There is a small chance that events get stuck in the buffer cache on the server side at the end of the run, in which case they would show up as the first events of the next run. I know that this problem happened in some experiment before, but that must have been unrelated to the rpc_mode. So please check again and report any problem with the new rpc_mode. |