Back Midas Rome Roody Rootana
  Midas DAQ System, Page 110 of 143  Not logged in ELOG logo
New entries since:Wed Dec 31 16:00:00 1969
ID Date Author Topic Subject
  680   27 Nov 2009 Stefan RittBug Report"mserver -s" is broken
> I notice that "mserver -s" (a non-default mode of operation) does not work right
> - if I connect odbedit for the first time, all is okey, if I connect the second
> time, mserver crashes - because after the first connection closed,
> rpc_deregister_functions() was called, rpc_list is deleted and causes a crash
> later on. Because everybody uses the default "mserver -m" mode, I am not sure
> how important it is to fix this.
> K.O.

"mserver -s" is there for historical reasons and for debugging. I started originally 
with a single process server back in the 90's, and only afterwards developed the multi 
process scheme. The single process server now only works for one connection and then 
crashes, as you described. But it can be used for debugging any server connection, 
since you don't have to follow the creation of a subprocess with your debugger, and 
therefore it's much easier. But after the first connection has been closed, you have 
to restart that single server process. Maybe one could add some warning about that, or 
even fix it, but it's nowhere used in production mode.
  679   26 Nov 2009 Konstantin OlchanskiBug Fixmserver network routing fix
mserver update svn rev 4625 fixes an anomaly in the MIDAS RPC network code where
in some network configurations MIDAS mserver connections work, but some RPC
transactions, such as starting and stopping runs, do not (use the wrong network
names or are routed over the wrong network).

The problem is a possible discrepancy between network addresses used to
establish the mserver connection and the value of "/System/Clients/xxx/Host"
which is ultimately set to the value of "hostname" of the remote client. This
ODB setting is then used to establish additional network connections, for
example to start or stop runs.

Use the client "hostname" setting works well for standard configurations, when
there is only one network interface in the machine, with only one IP address,
and with "hostname" set to the value that this IP address resolves to using DNS.

However, if there are private networks, multiple network interfaces, or multiple
network routes between machines, "/System/Clients/xxx/Host" may become set to an
undesirable value resulting in asymmetrical network routing or complete failure
to establish RPC connections.

Svn rev 4625 updates mserver.c to automatically set "/System/clients/xxx/Host"
to the same network name as was used to establish the original mserver connection.

As always with networking, any fix always breaks something somewhere for
somebody, in which case the old behavior can be restored by "setenv
MIDAS_MSERVER_DO_NOT_USE_CALLBACK_ADDR 1" before starting mserver.

The specific problem fixed by this change is when the MIDAS client and server
are on machines connected by 2 separate networks ("client.triumf.ca" and
"client.daq"; "server.triumf.ca" and "server.daq"). The ".triumf.ca" network
carries the normal SSH, NFS, etc traffic, and the ".daq" network carries MIDAS
data traffic.

The client would use the "server.daq" name to connect to the server and this
traffic would go over the data network (good).

However, previously, the client "/System/Clients/xxx/Host" would be set to
"client.triumf.ca" and any reverse connections (i.e. RPC to start/stop runs)
would go over the normal ".triumf.ca" network (bad).

With this modification, mserver will set "/System/Clients/xxx/Host" to
"client.daq" (the IP address of the interface on the ".daq" network) and all
reverse connections would also go over the ".daq" network (good).

P.S. This modification definitely works only for the default "mserver -m" mode,
but I do not think this is a problem as using "-s" and "-t" modes is not
recommended, and the "-s" mode is definitely broken (see my previous message).

svn rev 4625
K.O.
  678   26 Nov 2009 Konstantin OlchanskiBug Report"mserver -s" is broken
I notice that "mserver -s" (a non-default mode of operation) does not work right
- if I connect odbedit for the first time, all is okey, if I connect the second
time, mserver crashes - because after the first connection closed,
rpc_deregister_functions() was called, rpc_list is deleted and causes a crash
later on. Because everybody uses the default "mserver -m" mode, I am not sure
how important it is to fix this.
K.O.
  677   26 Nov 2009 Konstantin OlchanskiBug Fixmdump max number of banks and dump of 32-bit banks
By request from Renee, I increased the MIDAS BANKLIST_MAX from 64 to 1024 and
after fixing a few buglets where YB_BANKLIST_MAX is used instead of (now bigger)
BANKLIST_MAX, I can do a full dump of ND280 FGD events (96 banks).

I also noticed that "mdump -b BANK" did not work, it turns out that it could not
handle 32bit-banks at all. This is now fixed, too.

svn rev 4624
K.O.
  676   25 Nov 2009 Konstantin OlchanskiBug Reportonce in 100 years midas shared memory bug
We were debugging a strange problem in the event builder, where out of 14
fragments, two fragments were always getting serial number mismatches and the
serial numbers were not sequentially increasing (the other 12 fragments were
just fine).

Then we noticed in the event builder debug output that these 2 fragments were
getting assigned the same buffer handle number, despite having different names -
BUF09 and BUFTPC.

Then we looked at "ipcs", counted the buffers, and there are only 13 buffers for
14 frontends.

Aha, we went, maybe we have unlucky buffer names, renamed BUFTPC to BUFAAA and
everything started to work just fine.

It turns out that the MIDAS ss_shm_open() function uses "ftok" to convert buffer
names to SystemV shared memory keys. This "ftok" function promises to create
unique keys, but I guess, just not today.

Using a short test program, I confirmed that indeed we have unlucky buffer names
and ftok() returns duplicate keys, see below.

Apparently ftok() uses the low 16 bits of the file inode number, but in our
case, the files are NFS mounted and inode numbers are faked inside NFS. When I
run my test program on a different computer, I get non-duplicate keys. So I
guess we are double unlucky.

Test program:

#include <stdio.h>
#include <sys/types.h>
#include <sys/ipc.h>

int main(int argc, char* argv[])
{
  //key_t ftok(const char *pathname, int proj_id);
  
  int k1 = ftok("/home/t2kdaq/midas/nd280/backend/.BUF09.SHM", 'M');
  int k2 = ftok("/home/t2kdaq/midas/nd280/backend/.BUFTPC.SHM", 'M');
  int k3 = ftok("/home/t2kdaq/midas/nd280/backend/.BUFFGD.SHM", 'M');

  printf("key1: 0x%08x, key2: 0x%08x, key3: 0x%08x\n", k1, k2, k3);
  return 0;
}

[t2kfgd@t2knd280logger ~/xxx]$ g++ -o ftok -Wall ftok.cxx
[t2kfgd@t2knd280logger ~/xxx]$ ./ftok
key1: 0x4d138154, key2: 0x4d138154, key3: 0x4d138152

Also:

[t2kfgd@t2knd280logger ~/xxx]$ ls -li ...
14385492 -rw-r--r-- 1 t2kdaq t2kdaq  8405052 Nov 24 17:42
/home/t2kdaq/midas/nd280/backend/.BUF09.SHM
36077906 -rw-r--r-- 1 t2kdaq t2kdaq 67125308 Nov 26 10:19
/home/t2kdaq/midas/nd280/backend/.BUFFGD.SHM
36077908 -rw-r--r-- 1 t2kdaq t2kdaq  8405052 Nov 25 15:53
/home/t2kdaq/midas/nd280/backend/.BUFTPC.SHM

(hint: print the inode numbers in hex and compare to shm keys printed by the
program)

K.O.
  675   25 Nov 2009 Konstantin OlchanskiBug Fixsubrun file size
Please be aware of mlogger.c update rev 4566 on Sept 23rd 2009, when Stefan
fixed a buglet in the subrun file size computations. Before this fix, the first
subrun could be of a short length. If you use subruns, please update your
mlogger to at least rev 4566 (or newer, Stefan added the run and subrun time
limits just recently).
K.O.
  674   23 Nov 2009 Exaos LeeSuggestionScripts for "midas-config"
Supposing you have installed MIDAS to some directory such as "/opt/MIDAS/r4621", you have to write some Makefile as the following while building some applications based on the version installed:

Quote:
CFLAGS += -I/opt/MIDAS/r4621/include -DOS_LINUX -g -O2 -Wall -fPIC
LIBS += -lutil -lpthread -lodbc -lz
....

Why not use a script to record your MIDAS building options? When you want to build something based on it, just type something such as

Quote:
M_CFLAGS := `midas-config --cflags`
M_LIBS := `midas-config --libs`

You needn't to check your installed options each time when you build something against it. Each time you install a new version of MIDAS, you only need to update the script called 'midas-config'. I wrote a sample script named "genconf.sh" in the first zipped attachment. The 2nd "midas-config" is a sampled generated by it. Also a diff of Makefile is included. I hope it may help. Smile
Attachment 1: genc.zip
Attachment 2: midas-config
#!/bin/sh
prefix=/usr/local
cflags="-I/usr/local/include -g -O2 -Wall -Wuninitialized -DINCLUDE_FTPLIB  -DOS_LINUX -fPIC -Wno-unused-function -DHAVE_ODBC -DHAVE_ZLIB"
st_libs="-lutil -lpthread -lodbc -lz /usr/local/lib/libmidas.a"
dyn_libs="-lutil -lpthread -lodbc -lz -L/usr/local/lib -lmidas"
incdir=/usr/local/include
libdir=/usr/local/lib
svnver=r4621
platform=linux

drvdir=/usr/local/share/drivers
camacdir=/usr/local/share/drivers/camac
vmedir=/usr/local/share/drivers/vme

srcdir=/usr/local/share/src
exampledir=/usr/local/share/examples

out=""
while test $# -gt 0; do
  case "$1" in
  -*=*) optarg=/usr/local ;;
  *)    optarg= ;;
  esac

  case $1 in
      --prefix)
	  out="$out ${prefix}"
	  ;;
      --cflags|-c)
	  out="$out ${cflags}"
	  ;;
      --incdir|-dI)
	  out="$out ${incdir}"
	  ;;
      --libdir|-dL)
	  out="$out ${libdir}"
	  ;;
      --libs|--static-libs|-l)
	  out="$out ${st_libs}"
	  ;;
      --dyn-libs)
	  out="$out ${dyn_libs}"
	  ;;
      --platform)
	  out="$out ${platform}"
	  ;;
      --drvdir|-dD)
	  out="$out ${drvdir}"
	  ;;
      --camacdir|-dC)
	  out="$out ${camacdir}"
	  ;;
      --vmedir|-dV)
	  out="$out ${vmedir}"
	  ;;
      --srcdir|-dS)
	  out="$out ${srcdir}"
	  ;;
      --exampledir|-dE)
	  out="$out ${exampledir}"
	  ;;
      --version|-v)
          out="$out ${svnver}"
          ;;
      --help|-h)
	  echo "Usage: midas-config [options]"
	  echo "Options:"
	  echo "   --prefix        Display the installation prefix"
	  echo "   --cflags,-c     Display C/C++ flags using for compilation"
	  echo "   --incdir,-dI    Display midas header directory installed"
	  echo "   --libdir,-dL    Display midas library directory installed"
          echo "   --libs, -l"
	  echo "   --static-libs   Display link flags while linking with static midas library"
	  echo "   --dyn-libs      Display link flags while linking with dynamic library"
	  echo "   --platform      Display installed platform"
	  echo "   --drvdir,-dD    Display the driver directory"
	  echo "   --camacdir,-dC    Display the directory including CAMAC APIs"
	  echo "   --vmedir,-dV      Display the directory including VME APIs"
	  echo "   --srcdir,-dS      Display the source directory"
	  echo "   --exampledir,-dE  Display the examples directory"
          echo "   --version, -v  Show the subversion number"
	  echo "   --help, -h     Display this usage"
	  echo
          exit 0
	  ;;
      *)
      ### Give an error
	  echo "Unknown argument \"/usr/local\"!" 1>&2
	  echo "${usage}" 1>&2
	  exit 1
	  ;;
  esac
  shift
done

echo $out

  673   20 Nov 2009 Konstantin OlchanskiInfoRPC.SHM gyration
> When using remote midas clients with mserver, you may have noticed the zero-size .RPC.SHM files 
> these clients create in the directory where you run them.

Well, RPC.SHM bites. Please reread the parent message for full details, but in the nutshell, it is a global
semaphore that permits only one midas rpc client to talk to midas at a time (it was intended for local
locking between threads inside one midas application).

I have about 10 remote midas frontends started by ssh all in the same directory, so they all share the same
.RPC.SHM semaphore and do not live through the night - die from ODB timeouts because of RPC semaphore contention.

In a test version of MIDAS, I disabled the RPC.SHM semaphore, and now my clients live through the night, very
good.

Long term, we should fix this by using application-local mutexes (i.e. pthread_mutex, also works on MacOS, do
Windows have pthreads yet?).

This will also cleanup some of the ODB locking, which currently confuses pid's, tid's etc and is completely
broken on MacOS because some of these values are 64-bit and do not fit into the 32-bit data fields in MIDAS
shared memories.

Short term, I can add a flag for enabling and disabling the RPC semaphore from the user application: enabled
by default, but user can disable it if they do not use threads.

Alternatively, I can disable it by default, then enable it automatically if multiple threads are detected or
if ss_thread_create() is called.

Could also make it an environment variable.

Any preferences?

K.O.
  672   20 Nov 2009 Konstantin OlchanskiBug Fixdisallow client names with slash '/' characters
> odb.c rev 4622 fixes ODB corruption by db_connect_database() if client_name is
> too long. Also fixed is potential ODB corruption by too long key names in
> db_create_key(). Problem kindly reported by Tim Nichols of T2K/ND280 experiment.


Related bug fix - db_connect_database() should not permit client names that contain
the slash (/) character. Names like "aaa/bbb" create entries /Programs/aaa/bbb (aaa
is a subdirectory) and names like "../aaa" create entries in the ODB root directory.

svn rev 4623.
K.O.
  671   20 Nov 2009 Konstantin OlchanskiBug Fixfix odb corruption from too long client names
odb.c rev 4622 fixes ODB corruption by db_connect_database() if client_name is
too long. Also fixed is potential ODB corruption by too long key names in
db_create_key(). Problem kindly reported by Tim Nichols of T2K/ND280 experiment.
K.O.
  670   10 Nov 2009 Stefan RittForumIt' better to fix this warnings
> This will cause "type conversion" warnings. I hope that "odbedit" can generate codes like this:
>
> #define EXPCVADC_COMMON_STR(_name) const char *_name[] = {\
> "[.]",\
>...

Ok, I changed that in odb.c rev. 4620, should be fine now.
  669   06 Nov 2009 Jimmy NgaiForumRun multiple frontend on the same host
Dear All,

I want to run two frontend programs (one for trigger and one for slow control)
concurrently on the same computer, but I failed. The second frontend said: 

Semaphore already present
 There is another process using the semaphore.
 Or a process using the semaphore exited abnormally.
 In That case try to manually release the semaphore with:
   ipcrm sem XXX.

The two frontends are connected to the same experiment. Is there any way I can
overcome this problem?

Thanks!

Jimmy
  668   02 Nov 2009 Exaos LeeSuggestionNew cmake files
Though ended with ".c", "lazylogger.c" has to be build with C++ compiler. I have 
to modify my CMakeLists.txt. 
Please see the attachment if you need it. It works with svn-r4616.
Attachment 1: cmake4midas.zip
  667   02 Nov 2009 Exaos LeeBug FixBuild error due to missing header
I encountered a build error as "sort undefined...". It is caused by missing C++ header <algorithm> in which "sort" is defined. It can be fixed as the attachment.

Environment:
G++: 4.3.4
Platform: Debian Linux testing

> I committed an updated lazylogger with updated documentation. The new version supports subruns and
> can save to external storage arbitrary files (i.e. odb dump files). It also moves most book keeping out of
> odb to permit handling more files on bigger storage disks.
>
> Example lazylogger scripts for castor (CERN) and dcache (TRIUMF) are in the directory "utils".
>
> The lazylogger documentation was updated to remove obsolete information and to describe the new
> functions. As usual "make dox; cd doxfiles/html; firefox index.html" or see my copy at:
>
> http://ladd00.triumf.ca/~olchansk/midas/Utilities.html#lazylogger_task
>
> svn rev 4615, 4616.
> K.O.
Attachment 1: lazylogger.diff
diff --git a/src/lazylogger.c b/src/lazylogger.c
index 34be17a..a7ba4a3 100644
--- a/src/lazylogger.c
+++ b/src/lazylogger.c
@@ -17,6 +17,7 @@ $Id$
 
 #include <vector>
 #include <string>
+#include <algorithm>
 
 #define NOTHING_TODO  0
 #define FORCE_EXIT    1
  666   30 Oct 2009 Konstantin OlchanskiReleasenew lazylogger release
I committed an updated lazylogger with updated documentation. The new version supports subruns and 
can save to external storage arbitrary files (i.e. odb dump files). It also moves most book keeping out of 
odb to permit handling more files on bigger storage disks.

Example lazylogger scripts for castor (CERN) and dcache (TRIUMF) are in the directory "utils".

The lazylogger documentation was updated to remove obsolete information and to describe the new 
functions. As usual "make dox; cd doxfiles/html; firefox index.html" or see my copy at:

http://ladd00.triumf.ca/~olchansk/midas/Utilities.html#lazylogger_task

svn rev 4615, 4616.
K.O.
  665   20 Oct 2009 Peter SimpsonForumMidas in linux
Hi,
I'm new to both Linux and Midas and having trouble installing the programme - 
the install file suggeats that I should have a directory:

midas/unix

but this doesn't appear.

Also, when running the "make" for the library (step 3 of the installation), I 
don't see the directory "zlib.a", but there is a "libz.a" and I note the 
original .tar file has no libz.a file in it. Is this a typo on the 
installation?
The terminal window at this point displays:

undefined reference to `errno'
make: ***[example] Error 1

Further help will no doubt be required as I've used windows throughout my 
research and now looking to learn how to use linux. Any help greatly 
appreciated. Thanks!
  664   19 Oct 2009 Exaos LeeForumIt' better to fix this warnings
> There is no "type conversions". The compiler is whining about code like this:
>
> /* data type names */
> static char *tid_name[] = {
> "NULL",
> "BYTE",
> ...
>
> I guess we should keep the compiler happy and make them "static const char*".
>
> BTW, my compiler is SL5.2 gcc-4.1.2 and it does not complain. What's your compiler?
>
> K.O.

When I use "make" with odbedit, the generated "experim.h" also contains such codes as:
#define EXPCVADC_COMMON_STR(_name) char *_name[] = {\
"[.]",\
...
This will cause "type conversion" warnings. I hope that "odbedit" can generate codes like this:
#define EXPCVADC_COMMON_STR(_name) const char *_name[] = {\
"[.]",\
...
In fact, "const char*" is enough to suppress the warnings. Using "const char* foo=\"blarblar\"" is a good habit because it can avoid the following bugs:
char *s = "whatever";
...
strcpy(s, "Hello, this is a string longer than the initial one.");
The above code can successfully generate an executable, but it will encounter segmentation fault while executing. "const char*" means we should not change the variable in other place, which is important to suppress bugs.

Another problem, if I change the "experim.h" as I wished, new warnings appeared when the compiler parsing the following code:
status = db_create_record(hDB, 0, set_str, strcomb(expcvadc_settings_str));
Warning:
cvadcfe.c: In function ‘frontend_init’:
cvadcfe.c:144: warning: passing argument 1 of ‘strcomb’ from incompatible pointer type
  663   15 Oct 2009 Exaos LeeSuggestionBuilding MIDAS using CMake
The attached zip file is the updated configurations for building MIDAS using CMake. It works with svn-r4604.
If you want to use it, please follow the steps here:

Quote:


  1. Unzip the attachment, copy CMakeLists.txt and directory "cmake" into the midas source tree.
    $ cp -rp CMakeLists.txt cmake/ <Path-to-MIDAS-tree>/
  2. Make a build dir, and change to it.
  3. Execute cmake as this
    $ cmake -DCMAKE_INSTALL_PREFIX=<path-to-install> <path-to-MIDAS-tree>
  4. Make and install

You may use 'cmake -G <generator-name>' to generate building files for Unix Makefiles, Eclipse CDT4, KDevelop3 or Xcode, etc. I didn't test with other platforms. It now works with Unix Makefiles under Linux system. Please feedback any bugs to me: Exaos.Lee(AT)gmail.com .
Attachment 1: cmake4midas.zip
  662   11 Oct 2009 Konstantin OlchanskiInfoSQL history documentation
> Documentation for writing midas history data to SQL (mysql) is now documented in midas doxygen files 
> (make dox; firefox doxfiles/html/index.html). The corresponding logger and mhttpd code has been 
> committed for some time now and it is used in production environment by the t2k/nd280 slow controls 
> daq system at TRIUMF.
> svn rev 4487


An updated version of the SQL history code is now committed to midas svn.

The new code is in history_sql.cxx. It implements a C++ interface to the MIDAS history (history.h),
and improves on the old code history_odbc.cxx by adding:
- an index table for remembering MIDAS names of SQL tables and columns (our midas users like to use funny characters in history 
names that are not permitted in SQL table and column names),
- caching of database schema (event names, etc) with a noticeable speedup of mhttpd (there is a new button on the history panel editor 
"clear history cache" to make mhttpd reload the database schema.

The updated documentation for using SQL history is committed to midas svn doxfiles/internal.dox (svn up; make dox; firefox 
doxfiles/html/index.html), or see my copy on the web at
http://ladd00.triumf.ca/~olchansk/midas/Internal.html#History_sql_internal

svn rev 4595
K.O.
  661   11 Oct 2009 Konstantin OlchanskiBug ReportMultiple definition of `SqlODBC::SqlODBC()
> > Why is the class "SqlODBC" duplicated?
> 
> This is interesting. I do not think my C++ book spells it out that I cannot have class A in foo.cxx
> and a class A in bar.cxx.

I guess nobody knows the answer to this C++ puzzle. In any case history_odbc.cxx is not used anymore removing duplication of class SqlODBC.

svn rev 4594
K.O.
ELOG V3.1.4-2e1708b5