ID |
Date |
Author |
Topic |
Subject |
680
|
27 Nov 2009 |
Stefan Ritt | Bug Report | "mserver -s" is broken | > I notice that "mserver -s" (a non-default mode of operation) does not work right
> - if I connect odbedit for the first time, all is okey, if I connect the second
> time, mserver crashes - because after the first connection closed,
> rpc_deregister_functions() was called, rpc_list is deleted and causes a crash
> later on. Because everybody uses the default "mserver -m" mode, I am not sure
> how important it is to fix this.
> K.O.
"mserver -s" is there for historical reasons and for debugging. I started originally
with a single process server back in the 90's, and only afterwards developed the multi
process scheme. The single process server now only works for one connection and then
crashes, as you described. But it can be used for debugging any server connection,
since you don't have to follow the creation of a subprocess with your debugger, and
therefore it's much easier. But after the first connection has been closed, you have
to restart that single server process. Maybe one could add some warning about that, or
even fix it, but it's nowhere used in production mode. |
679
|
26 Nov 2009 |
Konstantin Olchanski | Bug Fix | mserver network routing fix | mserver update svn rev 4625 fixes an anomaly in the MIDAS RPC network code where
in some network configurations MIDAS mserver connections work, but some RPC
transactions, such as starting and stopping runs, do not (use the wrong network
names or are routed over the wrong network).
The problem is a possible discrepancy between network addresses used to
establish the mserver connection and the value of "/System/Clients/xxx/Host"
which is ultimately set to the value of "hostname" of the remote client. This
ODB setting is then used to establish additional network connections, for
example to start or stop runs.
Use the client "hostname" setting works well for standard configurations, when
there is only one network interface in the machine, with only one IP address,
and with "hostname" set to the value that this IP address resolves to using DNS.
However, if there are private networks, multiple network interfaces, or multiple
network routes between machines, "/System/Clients/xxx/Host" may become set to an
undesirable value resulting in asymmetrical network routing or complete failure
to establish RPC connections.
Svn rev 4625 updates mserver.c to automatically set "/System/clients/xxx/Host"
to the same network name as was used to establish the original mserver connection.
As always with networking, any fix always breaks something somewhere for
somebody, in which case the old behavior can be restored by "setenv
MIDAS_MSERVER_DO_NOT_USE_CALLBACK_ADDR 1" before starting mserver.
The specific problem fixed by this change is when the MIDAS client and server
are on machines connected by 2 separate networks ("client.triumf.ca" and
"client.daq"; "server.triumf.ca" and "server.daq"). The ".triumf.ca" network
carries the normal SSH, NFS, etc traffic, and the ".daq" network carries MIDAS
data traffic.
The client would use the "server.daq" name to connect to the server and this
traffic would go over the data network (good).
However, previously, the client "/System/Clients/xxx/Host" would be set to
"client.triumf.ca" and any reverse connections (i.e. RPC to start/stop runs)
would go over the normal ".triumf.ca" network (bad).
With this modification, mserver will set "/System/Clients/xxx/Host" to
"client.daq" (the IP address of the interface on the ".daq" network) and all
reverse connections would also go over the ".daq" network (good).
P.S. This modification definitely works only for the default "mserver -m" mode,
but I do not think this is a problem as using "-s" and "-t" modes is not
recommended, and the "-s" mode is definitely broken (see my previous message).
svn rev 4625
K.O. |
678
|
26 Nov 2009 |
Konstantin Olchanski | Bug Report | "mserver -s" is broken | I notice that "mserver -s" (a non-default mode of operation) does not work right
- if I connect odbedit for the first time, all is okey, if I connect the second
time, mserver crashes - because after the first connection closed,
rpc_deregister_functions() was called, rpc_list is deleted and causes a crash
later on. Because everybody uses the default "mserver -m" mode, I am not sure
how important it is to fix this.
K.O. |
677
|
26 Nov 2009 |
Konstantin Olchanski | Bug Fix | mdump max number of banks and dump of 32-bit banks | By request from Renee, I increased the MIDAS BANKLIST_MAX from 64 to 1024 and
after fixing a few buglets where YB_BANKLIST_MAX is used instead of (now bigger)
BANKLIST_MAX, I can do a full dump of ND280 FGD events (96 banks).
I also noticed that "mdump -b BANK" did not work, it turns out that it could not
handle 32bit-banks at all. This is now fixed, too.
svn rev 4624
K.O. |
676
|
25 Nov 2009 |
Konstantin Olchanski | Bug Report | once in 100 years midas shared memory bug | We were debugging a strange problem in the event builder, where out of 14
fragments, two fragments were always getting serial number mismatches and the
serial numbers were not sequentially increasing (the other 12 fragments were
just fine).
Then we noticed in the event builder debug output that these 2 fragments were
getting assigned the same buffer handle number, despite having different names -
BUF09 and BUFTPC.
Then we looked at "ipcs", counted the buffers, and there are only 13 buffers for
14 frontends.
Aha, we went, maybe we have unlucky buffer names, renamed BUFTPC to BUFAAA and
everything started to work just fine.
It turns out that the MIDAS ss_shm_open() function uses "ftok" to convert buffer
names to SystemV shared memory keys. This "ftok" function promises to create
unique keys, but I guess, just not today.
Using a short test program, I confirmed that indeed we have unlucky buffer names
and ftok() returns duplicate keys, see below.
Apparently ftok() uses the low 16 bits of the file inode number, but in our
case, the files are NFS mounted and inode numbers are faked inside NFS. When I
run my test program on a different computer, I get non-duplicate keys. So I
guess we are double unlucky.
Test program:
#include <stdio.h>
#include <sys/types.h>
#include <sys/ipc.h>
int main(int argc, char* argv[])
{
//key_t ftok(const char *pathname, int proj_id);
int k1 = ftok("/home/t2kdaq/midas/nd280/backend/.BUF09.SHM", 'M');
int k2 = ftok("/home/t2kdaq/midas/nd280/backend/.BUFTPC.SHM", 'M');
int k3 = ftok("/home/t2kdaq/midas/nd280/backend/.BUFFGD.SHM", 'M');
printf("key1: 0x%08x, key2: 0x%08x, key3: 0x%08x\n", k1, k2, k3);
return 0;
}
[t2kfgd@t2knd280logger ~/xxx]$ g++ -o ftok -Wall ftok.cxx
[t2kfgd@t2knd280logger ~/xxx]$ ./ftok
key1: 0x4d138154, key2: 0x4d138154, key3: 0x4d138152
Also:
[t2kfgd@t2knd280logger ~/xxx]$ ls -li ...
14385492 -rw-r--r-- 1 t2kdaq t2kdaq 8405052 Nov 24 17:42
/home/t2kdaq/midas/nd280/backend/.BUF09.SHM
36077906 -rw-r--r-- 1 t2kdaq t2kdaq 67125308 Nov 26 10:19
/home/t2kdaq/midas/nd280/backend/.BUFFGD.SHM
36077908 -rw-r--r-- 1 t2kdaq t2kdaq 8405052 Nov 25 15:53
/home/t2kdaq/midas/nd280/backend/.BUFTPC.SHM
(hint: print the inode numbers in hex and compare to shm keys printed by the
program)
K.O. |
675
|
25 Nov 2009 |
Konstantin Olchanski | Bug Fix | subrun file size | Please be aware of mlogger.c update rev 4566 on Sept 23rd 2009, when Stefan
fixed a buglet in the subrun file size computations. Before this fix, the first
subrun could be of a short length. If you use subruns, please update your
mlogger to at least rev 4566 (or newer, Stefan added the run and subrun time
limits just recently).
K.O. |
674
|
23 Nov 2009 |
Exaos Lee | Suggestion | Scripts for "midas-config" | Supposing you have installed MIDAS to some directory such as "/opt/MIDAS/r4621", you have to write some Makefile as the following while building some applications based on the version installed:
Quote: |
CFLAGS += -I/opt/MIDAS/r4621/include -DOS_LINUX -g -O2 -Wall -fPIC
LIBS += -lutil -lpthread -lodbc -lz
....
|
Why not use a script to record your MIDAS building options? When you want to build something based on it, just type something such as
Quote: |
M_CFLAGS := `midas-config --cflags`
M_LIBS := `midas-config --libs`
|
You needn't to check your installed options each time when you build something against it. Each time you install a new version of MIDAS, you only need to update the script called 'midas-config'. I wrote a sample script named "genconf.sh" in the first zipped attachment. The 2nd "midas-config" is a sampled generated by it. Also a diff of Makefile is included. I hope it may help. |
Attachment 1: genc.zip
|
Attachment 2: midas-config
|
#!/bin/sh
prefix=/usr/local
cflags="-I/usr/local/include -g -O2 -Wall -Wuninitialized -DINCLUDE_FTPLIB -DOS_LINUX -fPIC -Wno-unused-function -DHAVE_ODBC -DHAVE_ZLIB"
st_libs="-lutil -lpthread -lodbc -lz /usr/local/lib/libmidas.a"
dyn_libs="-lutil -lpthread -lodbc -lz -L/usr/local/lib -lmidas"
incdir=/usr/local/include
libdir=/usr/local/lib
svnver=r4621
platform=linux
drvdir=/usr/local/share/drivers
camacdir=/usr/local/share/drivers/camac
vmedir=/usr/local/share/drivers/vme
srcdir=/usr/local/share/src
exampledir=/usr/local/share/examples
out=""
while test $# -gt 0; do
case "$1" in
-*=*) optarg=/usr/local ;;
*) optarg= ;;
esac
case $1 in
--prefix)
out="$out ${prefix}"
;;
--cflags|-c)
out="$out ${cflags}"
;;
--incdir|-dI)
out="$out ${incdir}"
;;
--libdir|-dL)
out="$out ${libdir}"
;;
--libs|--static-libs|-l)
out="$out ${st_libs}"
;;
--dyn-libs)
out="$out ${dyn_libs}"
;;
--platform)
out="$out ${platform}"
;;
--drvdir|-dD)
out="$out ${drvdir}"
;;
--camacdir|-dC)
out="$out ${camacdir}"
;;
--vmedir|-dV)
out="$out ${vmedir}"
;;
--srcdir|-dS)
out="$out ${srcdir}"
;;
--exampledir|-dE)
out="$out ${exampledir}"
;;
--version|-v)
out="$out ${svnver}"
;;
--help|-h)
echo "Usage: midas-config [options]"
echo "Options:"
echo " --prefix Display the installation prefix"
echo " --cflags,-c Display C/C++ flags using for compilation"
echo " --incdir,-dI Display midas header directory installed"
echo " --libdir,-dL Display midas library directory installed"
echo " --libs, -l"
echo " --static-libs Display link flags while linking with static midas library"
echo " --dyn-libs Display link flags while linking with dynamic library"
echo " --platform Display installed platform"
echo " --drvdir,-dD Display the driver directory"
echo " --camacdir,-dC Display the directory including CAMAC APIs"
echo " --vmedir,-dV Display the directory including VME APIs"
echo " --srcdir,-dS Display the source directory"
echo " --exampledir,-dE Display the examples directory"
echo " --version, -v Show the subversion number"
echo " --help, -h Display this usage"
echo
exit 0
;;
*)
### Give an error
echo "Unknown argument \"/usr/local\"!" 1>&2
echo "${usage}" 1>&2
exit 1
;;
esac
shift
done
echo $out
|
673
|
20 Nov 2009 |
Konstantin Olchanski | Info | RPC.SHM gyration | > When using remote midas clients with mserver, you may have noticed the zero-size .RPC.SHM files
> these clients create in the directory where you run them.
Well, RPC.SHM bites. Please reread the parent message for full details, but in the nutshell, it is a global
semaphore that permits only one midas rpc client to talk to midas at a time (it was intended for local
locking between threads inside one midas application).
I have about 10 remote midas frontends started by ssh all in the same directory, so they all share the same
.RPC.SHM semaphore and do not live through the night - die from ODB timeouts because of RPC semaphore contention.
In a test version of MIDAS, I disabled the RPC.SHM semaphore, and now my clients live through the night, very
good.
Long term, we should fix this by using application-local mutexes (i.e. pthread_mutex, also works on MacOS, do
Windows have pthreads yet?).
This will also cleanup some of the ODB locking, which currently confuses pid's, tid's etc and is completely
broken on MacOS because some of these values are 64-bit and do not fit into the 32-bit data fields in MIDAS
shared memories.
Short term, I can add a flag for enabling and disabling the RPC semaphore from the user application: enabled
by default, but user can disable it if they do not use threads.
Alternatively, I can disable it by default, then enable it automatically if multiple threads are detected or
if ss_thread_create() is called.
Could also make it an environment variable.
Any preferences?
K.O. |
672
|
20 Nov 2009 |
Konstantin Olchanski | Bug Fix | disallow client names with slash '/' characters | > odb.c rev 4622 fixes ODB corruption by db_connect_database() if client_name is
> too long. Also fixed is potential ODB corruption by too long key names in
> db_create_key(). Problem kindly reported by Tim Nichols of T2K/ND280 experiment.
Related bug fix - db_connect_database() should not permit client names that contain
the slash (/) character. Names like "aaa/bbb" create entries /Programs/aaa/bbb (aaa
is a subdirectory) and names like "../aaa" create entries in the ODB root directory.
svn rev 4623.
K.O. |
671
|
20 Nov 2009 |
Konstantin Olchanski | Bug Fix | fix odb corruption from too long client names | odb.c rev 4622 fixes ODB corruption by db_connect_database() if client_name is
too long. Also fixed is potential ODB corruption by too long key names in
db_create_key(). Problem kindly reported by Tim Nichols of T2K/ND280 experiment.
K.O. |
670
|
10 Nov 2009 |
Stefan Ritt | Forum | It' better to fix this warnings | > This will cause "type conversion" warnings. I hope that "odbedit" can generate codes like this:
>
> #define EXPCVADC_COMMON_STR(_name) const char *_name[] = {\
> "[.]",\
>...
Ok, I changed that in odb.c rev. 4620, should be fine now. |
669
|
06 Nov 2009 |
Jimmy Ngai | Forum | Run multiple frontend on the same host | Dear All,
I want to run two frontend programs (one for trigger and one for slow control)
concurrently on the same computer, but I failed. The second frontend said:
Semaphore already present
There is another process using the semaphore.
Or a process using the semaphore exited abnormally.
In That case try to manually release the semaphore with:
ipcrm sem XXX.
The two frontends are connected to the same experiment. Is there any way I can
overcome this problem?
Thanks!
Jimmy |
668
|
02 Nov 2009 |
Exaos Lee | Suggestion | New cmake files | Though ended with ".c", "lazylogger.c" has to be build with C++ compiler. I have
to modify my CMakeLists.txt.
Please see the attachment if you need it. It works with svn-r4616. |
Attachment 1: cmake4midas.zip
|
667
|
02 Nov 2009 |
Exaos Lee | Bug Fix | Build error due to missing header | I encountered a build error as "sort undefined...". It is caused by missing C++ header <algorithm> in which "sort" is defined. It can be fixed as the attachment.
Environment:
G++: 4.3.4
Platform: Debian Linux testing
> I committed an updated lazylogger with updated documentation. The new version supports subruns and
> can save to external storage arbitrary files (i.e. odb dump files). It also moves most book keeping out of
> odb to permit handling more files on bigger storage disks.
>
> Example lazylogger scripts for castor (CERN) and dcache (TRIUMF) are in the directory "utils".
>
> The lazylogger documentation was updated to remove obsolete information and to describe the new
> functions. As usual "make dox; cd doxfiles/html; firefox index.html" or see my copy at:
>
> http://ladd00.triumf.ca/~olchansk/midas/Utilities.html#lazylogger_task
>
> svn rev 4615, 4616.
> K.O. |
Attachment 1: lazylogger.diff
|
diff --git a/src/lazylogger.c b/src/lazylogger.c
index 34be17a..a7ba4a3 100644
--- a/src/lazylogger.c
+++ b/src/lazylogger.c
@@ -17,6 +17,7 @@ $Id$
#include <vector>
#include <string>
+#include <algorithm>
#define NOTHING_TODO 0
#define FORCE_EXIT 1
|
666
|
30 Oct 2009 |
Konstantin Olchanski | Release | new lazylogger release | I committed an updated lazylogger with updated documentation. The new version supports subruns and
can save to external storage arbitrary files (i.e. odb dump files). It also moves most book keeping out of
odb to permit handling more files on bigger storage disks.
Example lazylogger scripts for castor (CERN) and dcache (TRIUMF) are in the directory "utils".
The lazylogger documentation was updated to remove obsolete information and to describe the new
functions. As usual "make dox; cd doxfiles/html; firefox index.html" or see my copy at:
http://ladd00.triumf.ca/~olchansk/midas/Utilities.html#lazylogger_task
svn rev 4615, 4616.
K.O. |
665
|
20 Oct 2009 |
Peter Simpson | Forum | Midas in linux | Hi,
I'm new to both Linux and Midas and having trouble installing the programme -
the install file suggeats that I should have a directory:
midas/unix
but this doesn't appear.
Also, when running the "make" for the library (step 3 of the installation), I
don't see the directory "zlib.a", but there is a "libz.a" and I note the
original .tar file has no libz.a file in it. Is this a typo on the
installation?
The terminal window at this point displays:
undefined reference to `errno'
make: ***[example] Error 1
Further help will no doubt be required as I've used windows throughout my
research and now looking to learn how to use linux. Any help greatly
appreciated. Thanks! |
664
|
19 Oct 2009 |
Exaos Lee | Forum | It' better to fix this warnings | > There is no "type conversions". The compiler is whining about code like this:
>
> /* data type names */
> static char *tid_name[] = {
> "NULL",
> "BYTE",
> ...
>
> I guess we should keep the compiler happy and make them "static const char*".
>
> BTW, my compiler is SL5.2 gcc-4.1.2 and it does not complain. What's your compiler?
>
> K.O.
When I use "make" with odbedit, the generated "experim.h" also contains such codes as:
#define EXPCVADC_COMMON_STR(_name) char *_name[] = {\
"[.]",\
...
This will cause "type conversion" warnings. I hope that "odbedit" can generate codes like this:
#define EXPCVADC_COMMON_STR(_name) const char *_name[] = {\
"[.]",\
...
In fact, "const char*" is enough to suppress the warnings. Using "const char* foo=\"blarblar\"" is a good habit because it can avoid the following bugs:
char *s = "whatever";
...
strcpy(s, "Hello, this is a string longer than the initial one.");
The above code can successfully generate an executable, but it will encounter segmentation fault while executing. "const char*" means we should not change the variable in other place, which is important to suppress bugs.
Another problem, if I change the "experim.h" as I wished, new warnings appeared when the compiler parsing the following code:
status = db_create_record(hDB, 0, set_str, strcomb(expcvadc_settings_str));
Warning:
cvadcfe.c: In function ‘frontend_init’:
cvadcfe.c:144: warning: passing argument 1 of ‘strcomb’ from incompatible pointer type
|
663
|
15 Oct 2009 |
Exaos Lee | Suggestion | Building MIDAS using CMake | The attached zip file is the updated configurations for building MIDAS using CMake. It works with svn-r4604.
If you want to use it, please follow the steps here:
Quote: |
- Unzip the attachment, copy CMakeLists.txt and directory "cmake" into the midas source tree.
$ cp -rp CMakeLists.txt cmake/ <Path-to-MIDAS-tree>/ - Make a build dir, and change to it.
- Execute cmake as this
$ cmake -DCMAKE_INSTALL_PREFIX=<path-to-install> <path-to-MIDAS-tree> - Make and install
|
You may use 'cmake -G <generator-name>' to generate building files for Unix Makefiles, Eclipse CDT4, KDevelop3 or Xcode, etc. I didn't test with other platforms. It now works with Unix Makefiles under Linux system. Please feedback any bugs to me: Exaos.Lee(AT)gmail.com . |
Attachment 1: cmake4midas.zip
|
662
|
11 Oct 2009 |
Konstantin Olchanski | Info | SQL history documentation | > Documentation for writing midas history data to SQL (mysql) is now documented in midas doxygen files
> (make dox; firefox doxfiles/html/index.html). The corresponding logger and mhttpd code has been
> committed for some time now and it is used in production environment by the t2k/nd280 slow controls
> daq system at TRIUMF.
> svn rev 4487
An updated version of the SQL history code is now committed to midas svn.
The new code is in history_sql.cxx. It implements a C++ interface to the MIDAS history (history.h),
and improves on the old code history_odbc.cxx by adding:
- an index table for remembering MIDAS names of SQL tables and columns (our midas users like to use funny characters in history
names that are not permitted in SQL table and column names),
- caching of database schema (event names, etc) with a noticeable speedup of mhttpd (there is a new button on the history panel editor
"clear history cache" to make mhttpd reload the database schema.
The updated documentation for using SQL history is committed to midas svn doxfiles/internal.dox (svn up; make dox; firefox
doxfiles/html/index.html), or see my copy on the web at
http://ladd00.triumf.ca/~olchansk/midas/Internal.html#History_sql_internal
svn rev 4595
K.O. |
661
|
11 Oct 2009 |
Konstantin Olchanski | Bug Report | Multiple definition of `SqlODBC::SqlODBC() | > > Why is the class "SqlODBC" duplicated?
>
> This is interesting. I do not think my C++ book spells it out that I cannot have class A in foo.cxx
> and a class A in bar.cxx.
I guess nobody knows the answer to this C++ puzzle. In any case history_odbc.cxx is not used anymore removing duplication of class SqlODBC.
svn rev 4594
K.O. |
|