Hello,
We have a small experiment with MIDAS based DAQ.
Status page shows :
ES ESFrontend@192.168.0.37 207 0.2 0.000
Trigger06 Sample Frontend06@192.168.0.37 1.297M 0.3 0.000
Trigger01 Sample Frontend01@192.168.0.37 1.297M 0.3 0.000
Trigger16 Sample Frontend16@192.168.0.37 1.297M 0.3 0.000
Trigger38 Sample Frontend38@192.168.0.37 1.297M 0.3 0.000
Trigger37 Sample Frontend37@192.168.0.37 1.297M 0.3 0.000
Trigger03 Sample Frontend03@192.168.0.38 1.297M 0.3 0.000
Trigger07 Sample Frontend07@192.168.0.38 1.297M 0.3 0.000
Trigger04 Sample Frontend04@192.168.0.38 59898 0.0 0.000
Trigger08 Sample Frontend08@192.168.0.38 59898 0.0 0.000
Trigger17 Sample Frontend17@192.168.0.38 59898 0.0 0.000
And SYSTEM buffers page shows:
ESFrontend 1968 198 47520 0 0x00000000 0
193 ms
Sample Frontend06 1332547 1330826 379729872 0 0x00000000
0 1.1 sec
Sample Frontend16 1332542 1330839 361988208 0 0x00000000
0 94 ms
Sample Frontend37 1332530 1330841 337798408 0 0x00000000
0 1.1 sec
Sample Frontend01 1332543 1330829 467136688 0 0x00000000
0 34 ms
Sample Frontend38 1332528 1330830 291453608 0 0x00000000
0 1.1 sec
Sample Frontend04 63254 61467 20882584 0 0x00000000
0 208 ms
Sample Frontend08 63262 61476 27904056 0 0x00000000
0 205 ms
Sample Frontend17 63271 61473 20433840 0 0x00000000
0 213 ms
Sample Frontend03 1332549 1330818 386821728 0 0x00000000
0 82 ms
Sample Frontend07 1332554 1330821 462210896 0 0x00000000
0 37 ms
Logger 968742 0w+9500418r 0w+2718405736r 0 0x00000000 0
GET_ALL Used 0 bytes 0.0% 303 ms
rootana 254561 0w+29856958r 0w+8718288352r 0 0x00000000 0
762 ms
The problem is that eventually some of frontend closed with message
:19:22:31.834 2021/12/02 [rootana,INFO] Client 'Sample Frontend38' on buffer
'SYSMSG' removed by cm_periodic_tasks because process pid 9789 does not exist
in the meantime mserver loggging :
mserver started interactively
mserver will listen on TCP port 1175
double free or corruption (!prev)
double free or corruption (!prev)
free(): invalid next size (normal)
double free or corruption (!prev)
I can find some correlation between number of events/event size produced by
frontend, cause its failed when its become big enough.
frontend scheme is like this:
poll event time set to 0;
poll_event{
//if buffer not transferred return (continue cutting the main buffer)
//read main buffer from hardware
//buffer not transfered
}
read event{
// cut the main buffer to subevents (cut one event from main buffer) return;
//if (last subevent) {buffer transfered ;return}
}
What is strange to me that 2 frontends (1 per remote pc) causing this.
Also, I'm executing one FEcode with -i # flag , put setting eventid in
frontend_init , and using SYSTEM buffer for all.
Is there something I'm missing?
Thanks.
A. |