ID |
Date |
Author |
Topic |
Subject |
590
|
04 Jun 2009 |
bazinski | Bug Report | mhttpd command line experiment specifying |
Hi
Not sure how the rest of you specify mhttpd to work with multiple experiments on
one machine, but it would seem not the same as me ;-)
when executing mhttpd with
mhttpd -e "experimentname" -p "experimentport" -D
that experiment name is not transfered to transitions as cm_transition never
specifies the experiment in the call to "transition STOP" etc.
the only flag it sends is a -d for debug if selected.
The result is that the stop and start button of the webinterface does not work,
and transitions sit endlessly doing nothing but consuming all the processor,
odbedit works fine though.
Does everyone else use an apache reverse proxy and or explicit experiment choice
in the url ?
As an aside in mhttpd.c in the reply to -? it states 2 -h options the second
should be a -e. line 13378.
Thanks
Sean |
589
|
04 Jun 2009 |
Stefan Ritt | Info | RPC.SHM gyration |
> Right now, MIDAS does not have an abstraction for "local multi-thread mutex" (i.e. pthread_mutex & co) and mostly uses global semaphores
> for this task (with interesting coding results, i.e. for multithreaded locking of ODB). Perhaps such an abstraction should be introduced?
Yes. In the old days when I designed the inter-process communication (~1993), there was no such thing like pthread_mutex (only under Windows).
Now it would be time to implement this thing, since it then will work under Posix and Windows (don't know about VxWorks). But that will at least
allow multi-threaded client applications, which can safely call midas functions through the RPC layer. For local thread-safeness, all midas
functions have to be checked an modified if necessary, which is a major work right now, but for remote clients it's rather simple. |
588
|
04 Jun 2009 |
Stefan Ritt | Bug Report | odbedit bad ctrl-C |
> When using "/bin/bash" shell, if I exit odbedit (and other midas programs) using ctrl-C, the terminal
> enters a funny state, "echo" is turned off (I cannot see what I type), "delete" key does not work (echoes
> ^H instead).
>
> This problem does not happen if I exit using the "exit" command or if I use the "/bin/tcsh" shell.
>
> When this happens, the terminal can be restored to close to normal state using "stty sane", and "stty
> erase ^H".
>
> The terminal is set into this funny state by system.c::getchar() and normal settings are never restored
> unless the midas program calls getchar(1) at the end. If the program does not finish normally, original
> terminal settings are never restored and the terminal is left in a funny state.
>
> It is not clear why the problem does not happen with /bin/tcsh - perhaps they restore sane terminal
> settings automatically for us.
> K.O.
Who uses bash ??? And who keeps baning on Ctrl-C, when there is a nice "exit" command ;-)
Well, I implemented a simple CTRL-C handler in odbedit (Rev. 4503) which resets the terminal before exiting.
Give it a try. Of course this cannot catch a hard kill (-9), but CTRL-C works now correctly under bash at
least. |
587
|
03 Jun 2009 |
Konstantin Olchanski | Bug Fix | Fix db_open_record() error return |
The odb hot-link function db_open_record() did not return an error when the system limit for hotlinks is
exceeded and no more hot links could be added (silent failure). This is now fixed.
odb.c svn rev 4500
K.O. |
586
|
02 Jun 2009 |
Konstantin Olchanski | Info | RPC.SHM gyration |
> When using remote midas clients with mserver, you may have noticed the zero-size .RPC.SHM files
> these clients create in the directory where you run them. These files are associated with the semaphore
> created by the midas rpc layer (rpc_call) to synchronize rpc calls between multiple threads. This
> semaphore is always created, even for single-threaded midas applications. Also normally midas
> semaphore files are created in the midas experiment directory specified in exptab (same place as
> .ODB.SHM), but for remote clients, we do not know that location until we start making rpc calls, so the
> semaphore file is created in the current directory (and it is on a remote machine anyway, so this
> location may not be visible locally).
>
> There are 2 problems with these semaphores:
A 3rd problem surfaced - on SL5 Linux, the global limit is 128 or so semaphores and on at least one heavily used machine that hosts multiple
experiments we simply run out of semaphores.
For "normal" semaphores, their number is fixed to about 5 per experiment (one for each shared memory buffer), but the number of RPC
semaphores is not bounded by the number of experiments or even by the number of user accounts - they are created (and never deleted) for
each experiment, for each user that connects to each experiment, for each subdirectory where the each user happened to try to start a
program that connects to the each experiment. (to reuse the old children's rhyme).
Right now, MIDAS does not have an abstraction for "local multi-thread mutex" (i.e. pthread_mutex & co) and mostly uses global semaphores
for this task (with interesting coding results, i.e. for multithreaded locking of ODB). Perhaps such an abstraction should be introduced?
K.O. |
585
|
02 Jun 2009 |
Konstantin Olchanski | Bug Report | mlogger duplicate event problem |
> We have seen on several daq systems this problem: we start a run and observe that the number of
> events written by mlogger to the output file is double the number of events actually collected. Upon
> inspection of the output file, we see that every event is written twice. Restarting the run usually fixes
> this problem.
mlogger.c fixed svn rev 4497. (from tr_start(), call tr_stop() if somehow it was not called already by end-run transition).
K.O. |
584
|
02 Jun 2009 |
Konstantin Olchanski | Info | mhttpd now uses mtransition |
> > mhttpd function for starting and stopping runs now uses cm_transition(DETACH) which spawns an
> > external helper program called mtransition to handle the transition sequencing.
>
> ... the GLIBC "system()" function breaks if the user application
> installs a SIGCHLD handler that "steals" wait() notifications. Such a handler is installed by the MIDAS ss_exec()
> function in system.c.
>
> I am now testing an implementation using MIDAS ss_spawnvp().
cm_transition() starting mtransition helper using ss_spawnvp() committed svn rev 4495.
K.O. |
583
|
21 May 2009 |
Konstantin Olchanski | Info | mhttpd now uses mtransition |
> mhttpd function for starting and stopping runs now uses cm_transition(DETACH) which spawns an
> external helper program called mtransition to handle the transition sequencing.
>
> P.S. In one of our experiments, I sometimes see mhttpd getting "stuck" when starting or stopping a run
> using this feature. strace shows it is stuck in repeated calls to wait(), but I am unable to reproduce this
> problem in a test system and it happens only sometimes in the experiment. When it does, mhttpd has to
> be restarted. Replacing system("mtransition ...") with ss_sysem("mtransition ...") seems to fix this problem,
> but there are downsides to this (mtransition debug output vanishes) so I am not committing this yet.
> K.O.
Found the problem. As observed on SL5 systems, the GLIBC "system()" function breaks if the user application
installs a SIGCHLD handler that "steals" wait() notifications. Such a handler is installed by the MIDAS ss_exec()
function in system.c.
I would count this as a GLIBC bug - their "system()" function should survive in the presence of non-default signal
handlers installed by the user, and in fact my copy of "man signal" talks about the "system()" doing something
special about SIGCHLD. Obviously whatever they do is broken, at least in the SL5 GLIBC.
I am now testing an implementation using MIDAS ss_spawnvp().
The simplest way to reproduce the problem: start mhttpd; start/stop runs - mtransition works perfectly; start some
program from the MIDAS "programs" page (this calls "ss_exec()"), try to start a run - mhttpd will hang inside the
system() GLIBC function, every time. mhttpd has to be killed with "kill -KILL" to recover.
K.O. |
582
|
20 May 2009 |
Exaos Lee | Suggestion | Question about using mvmestd.h |
> > The problem is: I renamed my SIS1100 devices as /dev/sis1100/xxxxx. So I have to hack the
> "sis3100.c".
>
> As in the old joke, "Doctor, it hurts when I do *this*; Doctor answers: then don't do it!"
>
> But I am curious why you want to change the "manufacturer-default" device names. For the vmivme.c and
> gefvme.c drivers that we use at TRIUMF, there is no obvious reason or gain from changing device names.
>
> K.O.
I used the old V2.04 driver for SIS1100/SIS3100. The old package contains a script which creates devices
as /tmp/sis1100_XXXX. So I created another script and installed it into /etc/init.d/. That script can be
invoked by using standard rc.d tools. In order to make the /dev directory tidy, it creates device files
into just one directory as /dev/sis1100/. That's the story.
Now, I found, the new sis1100.ko of version 2.12 can create devices automatically as /dev/sis1100_xxxx.
So, my script can be retired now. And also, I needn't to hack the "sis3100.c" anymore. |
581
|
19 May 2009 |
Konstantin Olchanski | Suggestion | Question about using mvmestd.h |
> The problem is: I renamed my SIS1100 devices as /dev/sis1100/xxxxx. So I have to hack the
"sis3100.c".
As in the old joke, "Doctor, it hurts when I do *this*; Doctor answers: then don't do it!"
But I am curious why you want to change the "manufacturer-default" device names. For the vmivme.c and
gefvme.c drivers that we use at TRIUMF, there is no obvious reason or gain from changing device names.
K.O. |
580
|
18 May 2009 |
Stefan Ritt | Suggestion | Question about using mvmestd.h |
Exaos Lee wrote: | The "mvmestd.h" uses the following function to open a VME device:int mvme_open(MVME_INTERFACE **vme, int idx) I found that the "driver/vme/sis3100/sis3100.c" uses the implementation as:
/* open VME */
sprintf(str, "/dev/sis1100_%02dremote", idx);
(*vme)->handle = open(str, O_RDWR, 0);
if ((*vme)->handle < 0)
return MVME_NO_INTERFACE;
}
The problem is: I renamed my SIS1100 devices as /dev/sis1100/xxxxx. So I have to hack the "sis3100.c".
Shall we have some smart way? |
In principle one could pass the device name to the user level. But I would like to keep the same code for Windows and Linux, and Windows does not need a device name. So you can either hack the file (I'm pretty sure it won't change in the next few years) or what I do is to make a symbolic link
/dev/sis1100/xxxx -> /dev/sis1100_00remote
Best regards,
Stefan |
579
|
18 May 2009 |
Exaos Lee | Suggestion | Question about using mvmestd.h |
The "mvmestd.h" uses the following function to open a VME device:int mvme_open(MVME_INTERFACE **vme, int idx) I found that the "driver/vme/sis3100/sis3100.c" uses the implementation as:
/* open VME */
sprintf(str, "/dev/sis1100_%02dremote", idx);
(*vme)->handle = open(str, O_RDWR, 0);
if ((*vme)->handle < 0)
return MVME_NO_INTERFACE;
}
The problem is: I renamed my SIS1100 devices as /dev/sis1100/xxxxx. So I have to hack the "sis3100.c".
Shall we have some smart way? |
578
|
15 May 2009 |
Konstantin Olchanski | Info | midas misc timeout fixes |
> - cm_transition() timeouts now settable from ODB (/experiment/transition timeout, transition connect timeout). Rev 4479
transition connect timeout was actually only half of that specified because of an error in computing timeout arguments to the select() system
call in recv_string() in system.c. This is now fixed.
rev 4488
K.O. |
577
|
15 May 2009 |
Konstantin Olchanski | Info | midas misc timeout fixes |
> - cm_transition() timeouts now settable from ODB (/experiment/transition timeout, transition connect timeout). Rev 4479
transition connect timeout was actually only half of that specified because of an error in computing timeout arguments to the select() system
call in recv_string() in system.c. This is now fixed.
rev 4488
K.O. |
576
|
07 May 2009 |
Konstantin Olchanski | Bug Report | mlogger duplicate event problem |
We have seen on several daq systems this problem: we start a run and observe that the number of
events written by mlogger to the output file is double the number of events actually collected. Upon
inspection of the output file, we see that every event is written twice. Restarting the run usually fixes
this problem.
We now traced this to an error in mlogger.c. If we start a run and the run transition fails in some
frontend, mlogger does not disconnect from the SYSTEM buffer (it does not know the transition failed
and the run did not really start). The SYSTEM buffer connection and the associated event request
remain active. Then we start the next run and mlogger connects to the SYSTEM buffer again, creates a
second (third, etc) event request. Eventually mlogger reaches the maximum permitted number of event
requests and no more runs can be started unless mlogger is restarted.
If at some point a run actually starts successfully, there are multiple event requests present from
mlogger and theoretically, each event should be written to the output file many times. This was a
puzzle until we got a good laugh from looking at mlogger.c::receive_event() callback - in retrospect it
is obvious why events are only written in duplicate.
Then, after the run is ended, mlogger disconnects from the SYSTEM buffer, all multiple event requests
are automatically deleted and the problem is not present during the next run.
I am not yet sure how to best fix this, but I see that other midas programs (i.e. mevb) suffer form the
same problem - multiple connections to the event buffer - in presence of failed run starts. I think we
have seen "event duplication" from mevb, as well.
K.O. |
575
|
07 May 2009 |
Konstantin Olchanski | Bug Report | odbedit bad ctrl-C |
When using "/bin/bash" shell, if I exit odbedit (and other midas programs) using ctrl-C, the terminal
enters a funny state, "echo" is turned off (I cannot see what I type), "delete" key does not work (echoes
^H instead).
This problem does not happen if I exit using the "exit" command or if I use the "/bin/tcsh" shell.
When this happens, the terminal can be restored to close to normal state using "stty sane", and "stty
erase ^H".
The terminal is set into this funny state by system.c::getchar() and normal settings are never restored
unless the midas program calls getchar(1) at the end. If the program does not finish normally, original
terminal settings are never restored and the terminal is left in a funny state.
It is not clear why the problem does not happen with /bin/tcsh - perhaps they restore sane terminal
settings automatically for us.
K.O. |
574
|
07 May 2009 |
Konstantin Olchanski | Info | SQL history documentation |
Documentation for writing midas history data to SQL (mysql) is now documented in midas doxygen files
(make dox; firefox doxfiles/html/index.html). The corresponding logger and mhttpd code has been
committed for some time now and it is used in production environment by the t2k/nd280 slow controls
daq system at TRIUMF.
svn rev 4487
K.O. |
573
|
07 May 2009 |
Konstantin Olchanski | Bug Fix | Fixed mlogger run start and stop |
Fixed problems with mlogger starting and stopping runs.
Basic difficulty was with the mlogger using ASYNC transitions, which did not implement proper
transition sequencing according to transition sequence numbers. Basically all clients were called at the
same time, regardless of how long they took to process the transitions.
Switching from ASYNC to SYNC transitions introduces a deadlock between mlogger (not reading data
from SYSTEM buffer while inside cm_transition) and any program trying to write into the SYSTEM buffer
(buffer is full, does not listen for transition requests while waiting for mlogger which tries to call it's
transition handler).
Then we invented the mtransition helper program. In the original implemtation for t2k it was spawned
directly from the mlogger to stop the run (avoiding the deadlock). Then cm_transition(DETACHED) was
introduced, but the mlogger start/stop/restart run logic became broken. One problem was with when
auto restart delay is zero, mtransition tries to restart the run before previous run is stopped (instead,
mlogger should restart the run from it's tr_stop() handler). Another problem was with the auto restart
delay counting from the time when we start stopping the run - because stopping the run can take an
unpredictable time, depending on when various frontends have to do - it is impossible to have a
predictable delay between runs (again this is fixed by restarting the run from mlogger.c::tr_stop()).
All this has been straightened out by svn revision 4484. Basically the old run stop/restart logic was
restored in mlogger.c, using cm_transition(DETACH) to avoid the deadlocks.
To remind all, these are the present controls for transitions initiated by mlogger:
/experiment/transition debug flag - set to "2" to capture transition sequences into midas.log
/experiment/transition timeout and transition connect timeout - one can change default timeouts as
needed to accommodate non cooperative frontends.
/logger/async transitions - do not use mtransition - do ASYNC transitions, as before.
/logger/auto restart delay - delay between stopping the run (mlogger.c::tr_stop) and starting the next
run.
svn rev 4484
K.O. |
572
|
07 May 2009 |
Konstantin Olchanski | Bug Fix | mhttpd "Names" length |
mhttpd did not like it when the equipment "Names" arrays had different length compared to the
corresponding "Variables" arrays. These limitations are now removed.
svn rev 4469
K.O. |
571
|
07 May 2009 |
Konstantin Olchanski | Info | mhttpd now uses mtransition |
mhttpd function for starting and stopping runs now uses cm_transition(DETACH) which spawns an
external helper program called mtransition to handle the transition sequencing. This helps with the old
problem of looking at a blank screen for a long time if some frontends take a long time to process run
transitions. Now mhttpd returns right back and shows start "starting run", "stopping run", etc as
appropriate.
svn rev 4484 (some bits of this feature are present in rev 4473 and later).
K.O.
P.S. In one of our experiments, I sometimes see mhttpd getting "stuck" when starting or stopping a run
using this feature. strace shows it is stuck in repeated calls to wait(), but I am unable to reproduce this
problem in a test system and it happens only sometimes in the experiment. When it does, mhttpd has to
be restarted. Replacing system("mtransition ...") with ss_sysem("mtransition ...") seems to fix this problem,
but there are downsides to this (mtransition debug output vanishes) so I am not committing this yet.
K.O. |