FYI - in conjunction with replacement of ladd00.triumf.ca, this MIDAS ELOG has been updated to the latest 
version 2.7.3-2058. Please report any problems or anomalies. K.O.

Now that we use more and more C++, lack of "const" attribute on most midas functions is causing some 
problems. I am now ready to commit changes to midas.h and odb.c that add the const attributes to ODB 
access functions db_xxx(), i.e.
INT db_rename_key(HNDLE hDB, HNDLE hKey, char *name)
becomes
INT db_rename_key(HNDLE hDB, HNDLE hKey, const char *name)

If we proceed with this conversion, and it does not cause major havoc, I can continue and "const"ify the 
rest of midas.h. I note that the mxml functions appear to already have the correct "const" declarations.

P.S. Adding the "const" attribute caught a few places where we were modifying a "char*" string passed by 
the caller. This is undesirable if we are passed a string literal, i.e. db_rename_key(...,"foo"), and it is a 
complete disaster in conjunction with C++ strings, i.e. db_rename_key(...,foo.c_str())

K.O.

> Now that we use more and more C++, lack of "const" attribute on most midas functions is causing some 
> problems. I am now ready to commit changes to midas.h and odb.c that add the const attributes to ODB 
> access functions db_xxx(), i.e.
> INT db_rename_key(HNDLE hDB, HNDLE hKey, char *name)
> becomes
> INT db_rename_key(HNDLE hDB, HNDLE hKey, const char *name)
> 
> If we proceed with this conversion, and it does not cause major havoc, I can continue and "const"ify the 
> rest of midas.h. I note that the mxml functions appear to already have the correct "const" declarations.
> 
> P.S. Adding the "const" attribute caught a few places where we were modifying a "char*" string passed by 
> the caller. This is undesirable if we are passed a string literal, i.e. db_rename_key(...,"foo"), and it is a 
> complete disaster in conjunction with C++ strings, i.e. db_rename_key(...,foo.c_str())

I fully approve your idea. You are absolutely right that it also will help to prevent errors such as modifying
fixed strings. I was just too lazy to do that, because it requires some additional code like:

func(const char *p)
{
char str[256];

  strlcpy(str, p, sizeof(str));
  strlcat(str, ...)
} 

So if you do it, it's great!

> > I am now ready to commit changes to midas.h and odb.c that add the const attributes to ODB 
> > access functions db_xxx(), i.e.
> > INT db_rename_key(HNDLE hDB, HNDLE hKey, char *name)
> > becomes
> > INT db_rename_key(HNDLE hDB, HNDLE hKey, const char *name)
>
> I fully approve your idea.

Committed revision 4172.

K.O.

Now that we use more and more C++, lack of "const" attribute on most midas functions is causing some 
problems. I am now ready to commit changes to midas.h and odb.c that add the const attributes to ODB 
access functions db_xxx(), i.e.
INT db_rename_key(HNDLE hDB, HNDLE hKey, char *name)
becomes
INT db_rename_key(HNDLE hDB, HNDLE hKey, const char *name)

If we proceed with this conversion, and it does not cause major havoc, I can continue and "const"ify the 
rest of midas.h. I note that the mxml functions appear to already have the correct "const" declarations.

P.S. Adding the "const" attribute caught a few places where we were modifying a "char*" string passed by 
the caller. This is undesirable if we are passed a string literal, i.e. db_rename_key(...,"foo"), and it is a 
complete disaster in conjunction with C++ strings, i.e. db_rename_key(...,foo.c_str())

K.O.

I would like to hear from potential users on which SQL database would be
preferable for storage of MIDAS history data.

My current preference is to use the ODBC interface, leaving the choice of
database engine to the user. While ODBC is not pretty, it appears to be adequate
for the job, permits "funny" databases (i.e. flat files) and I already have
prototype implementations for reading (mhttpd) and writing (mhdump/mlogger)
history data using ODBC.

In practice, MySQL and PgSQL are the main two viable choices for using with the
MIDAS history system. We tested both (no change in code - just tell ODBC which
driver to use) and both provide comparable performance and disk space use. We
were glad to see that the disk space use by both SQL databases is very
efficient, only slightly worse than uncompressed MIDAS history files.

At TRIUMF, for T2K/ND280, we now decided to use MySQL - it provides a better
match to MIDAS data types (has 1-byte and 2-byte integers, etc) and appears to
have working database replication (required for our use).

With mlogger already including support for MySQL, and MySQL being a better match
for MIDAS data, this gives them a slight edge and I think it would be reasonable
choice to only implement support for MySQL.

So I see 3 alternatives:

1) use ODBC (my preference)
2) use MySQL exclusively
3) implement a "midas odbc layer" supporting either MySQL or PgSQL.

Before jumping either way, I would like to hear from you folks.

K.O.

I rewrote the Makefile for MIDAS in order to make it tidy. I tested it on my box
and it works here.
1. The full file is seperated to several parts
  a. initialized setup
  b. environment setup
  c. specify OS-specific flags
  d. processing environment for building flags
  e. targets
2. The file is less than 400 lines now. The original one is more than 500 lines.
3. The modified one is easy for debuging.

I tried to learn "autoconf" and "automake" in order to make building MIDAS more
compatible for various platforms. But I havn't enough time now. Hope somebody
can help it. The attached file is original named "Makefile.in" for using "autoconf".

:-)

> I rewrote the Makefile for MIDAS in order to make it tidy. I tested it on my box
> and it works here.
> 1. The full file is seperated to several parts
>   a. initialized setup
>   b. environment setup
>   c. specify OS-specific flags
>   d. processing environment for building flags
>   e. targets
> 2. The file is less than 400 lines now. The original one is more than 500 lines.
> 3. The modified one is easy for debuging.
> 
> I tried to learn "autoconf" and "automake" in order to make building MIDAS more
> compatible for various platforms. But I havn't enough time now. Hope somebody
> can help it. The attached file is original named "Makefile.in" for using "autoconf".

I think it is a good idea to cleanup the Makefile. It grew over many years and certainly
had some inconsistencies. We did however not use "autoconf" since it is not of much use.
It is meant for systems where small differences between different Unix flavors are
covered by this system, but the midas source code is supposed not only to run on Unix,
but also on vxWorks and Windows. As you can imagine, the differences are much more
severe and a simple makefile generator cannot cover the details. Furthermore, under
Windows there is no such thing like autoconf. So all the work to make the source code
compile on all systems has been put into system.c using conditional compiling. So
putting another abstraction layer on this would maybe more complicate things than
simplify it. I will test your Makefile, and I also ask the guys at TRIUMF to do so. Once
we conclude that it works fine, we can replace the original Makefile from the distribution.

> I rewrote the Makefile for MIDAS in order to make it tidy. I tested it on my box
> and it works here.
> 1. The full file is seperated to several parts
>   a. initialized setup
>   b. environment setup
>   c. specify OS-specific flags
>   d. processing environment for building flags
>   e. targets
> 2. The file is less than 400 lines now. The original one is more than 500 lines.
> 3. The modified one is easy for debuging.
> 
> I tried to learn "autoconf" and "automake" in order to make building MIDAS more
> compatible for various platforms. But I havn't enough time now. Hope somebody
> can help it. The attached file is original named "Makefile.in" for using "autoconf".

The Makefile is missing -lzip:

[ritt@pc5082 ~/midas]$ make -f Makefile-by-EL
Making directory linux ...
Making directory linux/lib ...
Making directory linux/bin ...
g++ -g -O3 -Wall -Wuninitialized -DINCLUDE_FTPLIB -D_LARGEFILE64_SOURCE -DOS_LINUX -fPIC
-Wno-unused-function   -DHAVE_ZLIB -Iinclude -Idrivers -I../mxml -Llinux/lib -o
linux/bin/mlogger src/mlogger.c linux/lib/libmidas.a -lutil -lpthread
/tmp/cceHnAKe.o(.text+0x83c): In function `midas_flush_buffer(LOG_CHN*)':
src/mlogger.c:984: undefined reference to `gzwrite'
/tmp/cceHnAKe.o(.text+0xb24): In function `midas_log_open(LOG_CHN*, int)':
src/mlogger.c:1132: undefined reference to `gzopen'
/tmp/cceHnAKe.o(.text+0xb46):src/mlogger.c:1140: undefined reference to `gzsetparams'
/tmp/cceHnAKe.o(.text+0xe2a): In function `midas_log_close(LOG_CHN*, int)':
src/mlogger.c:1208: undefined reference to `gzflush'
/tmp/cceHnAKe.o(.text+0xe40):src/mlogger.c:1210: undefined reference to `gzclose'
collect2: ld returned 1 exit status
make: *** [linux/bin/mlogger] Error 1
[ritt@pc5082 ~/midas]$

> The Makefile is missing -lzip:

Sorry, spelling error.
The "LIBS +=" should be replaced by "LDFLAGS +="

> I rewrote the Makefile for MIDAS in order to make it tidy.

Not that the current Makefile is too pretty (I have seen worse), but it works and it is fairly compact for a project of 
this complexity, it handles a large number of operating systems and build options very efficiently.

I think you found that out with your rewriting exercise - your version of the Makefile contains all the same code, 
just rearranged to suite your taste, with existing bugs preserved and new bugs added.

> I tested it on my box and it works here.

As they say, the devil is in the details. I notice some subtle changes in your Makefile that make me go "what?":

1) the command for building the midas shared library used to be "ld -shared", in your version, "-shared" is gone. 
But check with the GCC manual, today's recommended command is probably "gcc -shared".
2) mhdump is now linked with ROOT, but I wrote it recently enough to remember that it does not use ROOT
3) hand-crafted dependancies have been replaced with generic "almost every .o depends on every .h", which is 
incorrect. The "almost every .o" part bothers me.
4) "make clean" runs "rm -rf" - plain scary.
5) "$(shell ...)" is overused

I think by the end all these little details are sorted out and all the quirks are put back in, your Makefile will look no 
better than the current Makefile.

> 2. The file is less than 400 lines now. The original one is more than 500 lines.

It looks like your savings came from removing comments, removing hand-crafted dependancy lists and replacing 
fairly verbose "make install" targets (which we do not use anyway) with your own much simpler scripts.

All the juicy bits needed to actually build all the code appear to take about as much space as before.

Also the original mistake of recompiling programs when they only need relinking was not fixed. (For example, 
when libmidas is updated, to update mhttpd, the current Makefile needlessly recompiles mhttpd.c. Better use 
would be to compile mhttpd.c into mhttpd.o, then only a relink is needed).

> I tried to learn "autoconf" and "automake" in order to make building MIDAS more
> compatible for various platforms. But I havn't enough time now. Hope somebody
> can help it. The attached file is original named "Makefile.in" for using "autoconf".

Most experience with autoconf/automake is all negative. The promise was "never debug your Makefile ever 
again!", delivered was "debug the configure script instead!". In practice, with autoconf/automake, you try to run 
configure, kludge it until it stops crashing, then tweak the incomprehensible Makefiles it produces until the code 
compiles.

K.O.

> Most experience with autoconf/automake is all negative. The promise was "never debug your Makefile ever 
> again!", delivered was "debug the configure script instead!". In practice, with autoconf/automake, you try to run 
> configure, kludge it until it stops crashing, then tweak the incomprehensible Makefiles it produces until the code 
> compiles.
> 
> K.O.

I admit that the new one is fit to my flavor. For a common user, I think, a simple procedure of configure/make/install
is better than changing the Makefile manually because many users are lack of knowledges about Makefile. That's why 
I want to learn autotools. The configure script is generated automatically by "autoconf", so you needn't to debug it. 
For the developer, you need to debug the configure.ac/in files for generating the configure script. For a common user, 
he/she only needs to run it. In fact, some more complex projects like ROOT use AUTOTOOLS and they don't include 
the original files which are needed for generating the "configure" script.  I prefer the MIDAS project includes such a 
script to make the compiling simpler and easier instead of changing the Makefile manually. 

> > Most experience with autoconf/automake is all negative. The promise was "never debug your Makefile ever 
> > again!", delivered was "debug the configure script instead!". 
> 
> I admit that the new one is fit to my flavor. For a common user, I think, a simple procedure of configure/make/install
> is better than changing the Makefile manually because many users are lack of knowledges about Makefile. That's why 
> I want to learn autotools.

The reality is that you will never deliver a Makefile/Configure script that works for everybody in every case - users will always have a need to tweak the build 
process to suit their needs. In this situation, "Makefile" is a much better language and "make" is a much better tool for users to deal with - much simpler, better 
documented and better understood compared to autotools (*nobody* understands autotools; also compare the size of the midas Makefile with the size of a 
typical configure script).

Anyhow, we will be cross-compiling midas to run on a PowerPC processor inside a Virtex4 FPGA. Go handle that with configure scripts.

K.O.

> Not that the current Makefile is too pretty (I have seen worse), but it 
works and it is fairly compact for a project of 
> this complexity, it handles a large number of operating systems and build 
options very efficiently.
> 
> I think you found that out with your rewriting exercise - your version of 
the Makefile contains all the same code, 
> just rearranged to suite your taste, with existing bugs preserved and new 
bugs added.

I derived the new Makefile from the original one so that feathers and bugs are 
also included. 
I havn't experiences on platforms other than Linux and MacOS, so I cannot 
recognize bugs on 
other platforms if they exists in the original one. And if there are bugs, 
hope users can figure
them out.

>
> As they say, the devil is in the details. I notice some subtle changes in 
your Makefile that make me go "what?":
> 
> 1) the command for building the midas shared library used to be "ld -
shared", in your version, "-shared" is gone. 
> But check with the GCC manual, today's recommended command is probably "gcc -
shared".

Fixed.

> 2) mhdump is now linked with ROOT, but I wrote it recently enough to 
remember that it does not use ROOT

The building dependence on ROOT of mhdump may be eliminated by changing the 
specific target.

> 3) hand-crafted dependancies have been replaced with generic "almost 
every .o depends on every .h", which is 
> incorrect. The "almost every .o" part bothers me.

Fixed now.

> 4) "make clean" runs "rm -rf" - plain scary.

Fixed.

> 5) "$(shell ...)" is overused

Replaced with GNU make internal methods.


> I think by the end all these little details are sorted out and all the 
quirks are put back in, your Makefile will look no 
> better than the current Makefile.

I realized it now. But anyway, it looks tidy to me now. I still hope to use 
AUTOTOOLS with MIDAS.
 
> > 2. The file is less than 400 lines now. The original one is more than 500 
lines.

The new one is about 430 lines. Hmmm, it reaches the original one which is 
more than 600 lines.

> 
> It looks like your savings came from removing comments, removing hand-
crafted dependancy lists and replacing 
> fairly verbose "make install" targets (which we do not use anyway) with your 
own much simpler scripts.
> 
> All the juicy bits needed to actually build all the code appear to take 
about as much space as before.
> 
> Also the original mistake of recompiling programs when they only need 
relinking was not fixed. (For example, 
> when libmidas is updated, to update mhttpd, the current Makefile needlessly 
recompiles mhttpd.c. Better use 
> would be to compile mhttpd.c into mhttpd.o, then only a relink is needed).

Fixed.

> 
> Most experience with autoconf/automake is all negative. The promise 
was "never debug your Makefile ever 
> again!", delivered was "debug the configure script instead!". In practice, 
with autoconf/automake, you try to run 
> configure, kludge it until it stops crashing, then tweak the 
incomprehensible Makefiles it produces until the code 
> compiles.
> 
> K.O.

====================================================
Maybe BUGS or FEATURES in this new one:
1. The shared libmidas.so and the static libmidas.a are built sperately. 
The "libmidas.a" is 
always built whether "NEED_SHLIB" is set or not. And all executables are built 
staticly default. 
I commented this in the Makefile-by-EL. Hence, if you want to use libmidas.so 
with PyMIDAS and
do not want to encounter "Segmentation fault" while executing the utilies 
linked dynamicly, you
may try this one.
2. I found that "minife" is failed to be built, so I remove it from the 
example list.
3. Some bugs while building on MacOS Tiger 10.4.11 PPC are commented out in 
the Makefile. These
bugs are still exists in the original one.
4. Using "VPATH" instead of adding pathnames.
5. Using "UTILS_SUID" to handle utilities which need SUID mode. And 
the "UTILS_SUID_NEED" may be
defined in the OS-specific field, so you need not to use OS-specific commands 
in the "install"
target.
6. Using "tr" with "uname" in order to delete some extra "ifeq 
($(OSTYPE),...)".
7. Other things, please see the file.

Anyway, easier building is my purpose. :-)

EXECS += $(EXAMPLES:%/$(BIN_DIR)/%)

EXECS += $(EXAMPLES:%=$(BIN_DIR)/%)

The linking of mhttpd misses a "-lm":

cc -g -O3 -Wall -Wuninitialized -DINCLUDE_FTPLIB -D_LARGEFILE64_SOURCE -DOS_LINUX
-fPIC -Wno-unused-function   -DHAVE_ZLIB -Iinclude -Idrivers -I../mxml -o
linux/bin/mhttpd linux/lib/mhttpd.o linux/lib/mgd.o linux/lib/libmidas.a -lutil
-lpthread -lz
linux/lib/mhttpd.o(.text+0xe08f): In function `show_custom_gif':
src/mhttpd.c:5058: undefined reference to `log'
linux/lib/mhttpd.o(.text+0xe0a8):src/mhttpd.c:5058: undefined reference to `log'

The header of the makefile should contain a short description, the author(s), an
$Id:$ tag for SVN, some explanation what "icc", "ifort" means, a note about the
CFLAGS and a clear statement what can be modified by the user and why and what not.

> The linking of mhttpd misses a "-lm":
> 
> cc -g -O3 -Wall -Wuninitialized -DINCLUDE_FTPLIB -D_LARGEFILE64_SOURCE -DOS_LINUX
> -fPIC -Wno-unused-function   -DHAVE_ZLIB -Iinclude -Idrivers -I../mxml -o
> linux/bin/mhttpd linux/lib/mhttpd.o linux/lib/mgd.o linux/lib/libmidas.a -lutil
> -lpthread -lz
> linux/lib/mhttpd.o(.text+0xe08f): In function `show_custom_gif':
> src/mhttpd.c:5058: undefined reference to `log'
> linux/lib/mhttpd.o(.text+0xe0a8):src/mhttpd.c:5058: undefined reference to `log'
> 

Strange. I tested it on Debian Linux 4.0r2 AMD64 with gcc 4.1.2, MIDAS SVN 4124. It worked fine.
Anyway, it can be fixed by addling "-lm" to the initial"LDFLAGS".

> The header of the makefile should contain a short description, the author(s), an
> $Id:$ tag for SVN, some explanation what "icc", "ifort" means, a note about the
> CFLAGS and a clear statement what can be modified by the user and why and what not.

OK. I will comment it in detail.

Hi,

I have just compiled MIDAS svn 4132 on a fresh SuSE 10.3 x86_64 system and gcc 
found a bunch of bugs, I guess.

> uname -a
Linux pc 2.6.22.17-0.1-default #1 SMP 2008/02/10 20:01:04 UTC x86_64 x86_64 
x86_64 GNU/Linux

> gcc --version
gcc (GCC) 4.2.1 (SUSE Linux)

I see loads of warnings during compile, most of which I know from earlier 
compiles:
* warning: dereferencing type-punned pointer will break strict-aliasing rules
* warning: pointer targets in passing argument 3 of 'getsockname' differ in
           signedness

But then there is a new one (in fact lots of this one), and brief
inspection suggests this is a true bug with the possibility of truly
nasty consequences.

(1)=========================
src/midas.c:7398: warning: array subscript is above array bounds
Inspection of midas.c:

   if (i == MAX_DEFRAG_EVENTS) {
      /* no buffer available -> no first fragment received */
7398: free(defrag_buffer[i].pevent);
      memset(&defrag_buffer[i].event_id, 0, sizeof(EVENT_DEFRAG_BUFFER));
      cm_msg(MERROR, "bm_defragement_event",
             "Received fragment without first fragment (ID %d) Ser#:%d",
             pevent->event_id & 0x0FFF, pevent->serial_number);
      return;
   }

And midas.c line 7297:
EVENT_DEFRAG_BUFFER defrag_buffer[MAX_DEFRAG_EVENTS];

So, if(i==MAX_DEFRAG_EVENTS) free(defrag_buffer[i]).
I guess this is an off-by-one bug.

(2)==========================
src/midas.c:2958: warning: array subscript is above array bounds

   for (i = 0; i < 13; i++)
2958  if (trans_name[i].transition == transition)
         break;

Holy cow, hard-coded "13" in the code! Should be a #define, shouldn't it?

Now look at midas.c lines 94ff:
struct {
   int transition;
   char name[32];
} trans_name[] = {
   {
   TR_START, "START",}, {
   TR_STOP, "STOP",}, {
   TR_PAUSE, "PAUSE",}, {
   TR_RESUME, "RESUME",}, {
   TR_DEFERRED, "DEFERRED",}, {
0, "",},};

There is no trans_name[12].

The trans_name[12] shows up in line 2894 and 2790, too.

(3)=============================
mfe.c:
src/mfe.c:412: warning: array subscript is above array bounds
src/mfe.c:311: warning: array subscript is above array bounds
src/mfe.c:340: warning: array subscript is above array bounds

412: device_drv->dd(CMD_GET_DEMAND, device_drv->dd_info, i, 
          &device_drv->mt_buffer->channel[i].array[CMD_GET_DEMAND]);


   for (cmd = CMD_GET_FIRST; cmd <= CMD_GET_LAST; cmd++) {
 (..)
311:  device_drv->mt_buffer->channel[current_channel].array[cmd] = value;

   for (cmd = CMD_GET_FIRST; cmd <= CMD_GET_LAST; cmd++) {
 (..)
340:  device_drv->mt_buffer->channel[i].array[cmd] = value;


CMD_GET_DEMAND is in include/midas.h:
#define CMD_GET_DEMAND               CMD_GET_DIRECT  // = 20

I haven't even tried to understand mfe.c, nor did I read it. 
But I suspect the thing should always be something like
....array[cmd-CMD_GET_FIRST]
so array[] is indexed from [0], not from an arbitrary number that
depends on the number of commands you insert before line 698 in
midas.h. But please could the author of this check this very carefully?


(4)=========================
src/lazylogger.c:1957: warning: array subscript is below array bounds

if ((channel < 0) && (lazyinfo[channel].hKey != 0))

That is lazyinfo[something below zero].


(5)=============================
More warnings an expert might want to have a look at:

* warning: deprecated conversion from string constant to 'char*'

* src/fal.c:106: warning: non-local variable '<anonymous struct> out_info'
                 uses anonymous type
* src/fal.c:3064: warning: non-local variable '<anonymous struct> eb' uses
                  anonymous type

I attach the full output of make.
Could someone knowledgeable please have a look at these warnings and fix them?

They make me a bit nervous when thinking about data integrity, and
there are now so many that they actually start to hide serious stuff
like the ones I presented.

Oh, I got rid of the "dereferencing type-punned pointer" thing by adding
"-fno-strict-aliasing" as a compiler flag. Was suggested on the Web. Seemed to 
have worked during data taking (the data look reasonable :-). Is that a 
possible fix/workaround?


Cheers,

Randolf

> I have just compiled MIDAS svn 4132 on a fresh SuSE 10.3 x86_64 system and gcc 
> found a bunch of bugs, I guess.

Ahh, great! gcc is getting more and more clever. Each time gcc is updated, it finds
a few new issues.

Indeed some are real bugs, and I will work down the list as time permits. I see
however no immediate thread (you are not using fragmented events, a transition 12
never occurs, etc.). Issue #4 from your list has to be checked by Pierre-Andre. 

There were some trivial and some non-trivial issues. Glad the compiled picked up on
this!

> I see loads of warnings during compile, most of which I know from earlier 
> compiles:
> * warning: dereferencing type-punned pointer will break strict-aliasing rules
> * warning: pointer targets in passing argument 3 of 'getsockname' differ in
>            signedness

I ignore these for the moment until I have a gcc 4.2 myself (we use Scientific
Linux 5 which has gcc 4.1 for the moment). As Randolph pointed out correctly you
can make gcc shut up by a proper flag there. The warnings have no influence on the
stability of midas.

> (1)=========================
> src/midas.c:7398: warning: array subscript is above array bounds
> Inspection of midas.c:
> 
>    if (i == MAX_DEFRAG_EVENTS) {
>       /* no buffer available -> no first fragment received */
> 7398: free(defrag_buffer[i].pevent);
>       memset(&defrag_buffer[i].event_id, 0, sizeof(EVENT_DEFRAG_BUFFER));
>       cm_msg(MERROR, "bm_defragement_event",
>              "Received fragment without first fragment (ID %d) Ser#:%d",
>              pevent->event_id & 0x0FFF, pevent->serial_number);
>       return;
>    }

The free() was just wrong at that place, I removed it.

> (2)==========================
> src/midas.c:2958: warning: array subscript is above array bounds
> 
>    for (i = 0; i < 13; i++)
> 2958  if (trans_name[i].transition == transition)
>          break;

Fixed that by
     
     for (i=0 ;; i++)
        if (trans_name[i].name[0] == 0 || trans_name[i].transition == transition)
           break;

Since trans_name[i].name = "" indicates the end of the list.

> (3)=============================
> mfe.c:
> src/mfe.c:412: warning: array subscript is above array bounds
> src/mfe.c:311: warning: array subscript is above array bounds
> src/mfe.c:340: warning: array subscript is above array bounds
> 
> 412: device_drv->dd(CMD_GET_DEMAND, device_drv->dd_info, i, 
>           &device_drv->mt_buffer->channel[i].array[CMD_GET_DEMAND]);

The code at 412 was wrong there, the demand value is queried later by the device
driver directly. For the other two occurences (311 and 340) I had to really
increase the array size by one. This issue can cause segfaults if you have a slow
control front-end which uses multithreading (not many people use it except me).

> (4)=========================
> src/lazylogger.c:1957: warning: array subscript is below array bounds
> 
> if ((channel < 0) && (lazyinfo[channel].hKey != 0))
> 
> That is lazyinfo[something below zero].

This has to be fixed by Pierre. I guess an or instead of an and would do it, but
I'm not 100% sure.

> (5)=============================
> More warnings an expert might want to have a look at:
> 
> * warning: deprecated conversion from string constant to 'char*'
> 
> * src/fal.c:106: warning: non-local variable '<anonymous struct> out_info'
>                  uses anonymous type
> * src/fal.c:3064: warning: non-local variable '<anonymous struct> eb' uses
>                   anonymous type
> 
> I attach the full output of make.
> Could someone knowledgeable please have a look at these warnings and fix them?

Uahhh. Especially the "const char*" vs. "char*" is in principle right, but will
cause a major rework. Probably hundreds of occations have to be fixed. Many strings
must be declared const, others not. It will help the programmer to find some errors
during compile which would later show up only during runtime (like writing into a
fixed string), but I only will go through that when I have gcc 4.2 installed
myself, and have two free days to work on this ;-)

> They make me a bit nervous when thinking about data integrity, and
> there are now so many that they actually start to hide serious stuff
> like the ones I presented.

Except the slow control stuff (which only is an issue for multithreaded frontends)
none of the above things will have an influence on the data integrity. But I agree
that they should be fixed.

- Stefan

Running MIDAS at CERN is proving more challenging than I expected. The network environement is not 
as benign as I am used to (i.e. at TRIUMF) and our machines are being constantly probed by something/
somebody.

This already caused failures in the mserver (fixed in midas svn) and I would like to resolve this problem 
once and for all. The age of "nice networks" is over.

The case of the mserver and for the midas rpc servers (every midas applications listens for midas rpc 
requests, i.e. run transitions) is simple. The list of machines running midas applications is known ahead 
of time, so we can put them all into a list of permitted machines and deny rpc connections to anybody 
else. I propose we keep this list of permitted mserver clients in "/experiment/security/mserver hosts".

(The already existing "/experiment/security/allowed hosts" mechanism is insufficient: it does not 
prevent the mserver from accepting connections from hostile machines, and talking to them, for 
example giving them the list of available experiments. There is a fair amount of code involved and I do 
not presume to certify any of it as hack-proof or even as crash-proof.)

For mhttpd http:// access control, I thought of using tcp_wrappers, but C-API documentation does not 
exist (I looked), the example code in tcpd.c is way too complicated, editing the ACL /etc/hosts.allow 
unnecessarily requires root privileges and non of it would work on Windows.

So I am favouring a home-made hostname or ip-address filter, similar to /etc/hosts.allow, with ACL 
stored, for example, in "/experiment/security/mhttpd hosts".

Any thoughts?

K.O.

I am in favor of tcp_wrappers.

tcp_wrappers is well understood.

It works well in combination with a firewall.

mhttpd hangs when our security folks scan us.  We are not allowed to block them
with a firewall, but we can use tcpwrappers.

Would it make sense to put the same mechanism on mserver?

the man page for libtcpwrappers.a (taken from the tcpwrappers7.6 tar ball) is
attached. And the output after running it through nroff -man.

The odb is too fragile for security.  It is not understood well enough by many
experimenters.

As you can see I am in favor of tcp_wrappers.  This is mainly because it is part
of an existing and tested security model.  I don't know about the windows
world, but as you can also see, I vote for using something that is already part
of the windows security model.  Here's an example of how well the integrated
security model works:

    if an person is part of an experiment I make sure they can ssh to the
    experiment's computer

    the same rules could provide them with web access

Second is that when a change is needed to the security model then it is easy to
keep it current.  What if somebody restores an old ODB?  What if they setup a
small test with a new ODB?

If mhttpd used tcp_wrappers, then all our machines here at LANL would already be
configured!  No need for
users to do any root access (though those that need it have it anyway).

John.

First I have a general question: mserver is started through xinetd, and xinetd has
the options "only_from" and "no_access". This is equivalent to the tcp_wrapper
functionality. Why not using this? It's possible without changing anything in midas.
Or am I missing anything?

If that does not work for some reason, here are some thought from my side:

- We don't have much of a problem with malicious hackers, but with institute-wide
security checking. Hackers are only interested in mechanisms where they can obtain
control over thousands of machines (like breaking ssh etc.). The few midas machines
are not a good target for them. But even at PSI there are security scans, which try
to connect to various ports and can crash systems, so I agree that something needs
to be done.

- Whatever we do, it should be consistent on linux and windows and should not rely
on external packages, since I don't want to get into dependencies there.

- I see that both having the security information in the ODB or having them in
external files can be advantageous. There is certainly the aspect of restoring old
ODBs, or keeping several experiments (ODB) on one machine consistent. On the other
hand storing data in the ODB might me liked by people who are familiar with this
concept, and want to change things though mhttpd for example.

- Having said all that, it would make sense to me to write a simple central routine
access_allowed(), which takes the IP address of a remote client wanting to connect,
and return true or false. This routine should read /etc/hosts.allow, /etc/hosts.deny
and interprete it, but only the section for midas, and maybe only a subset of the
functionality there (we probably don't need NIS netgroup names, external files and
spawn commands there). If the files /etc/hosts.x do not contain anything about midas
or are not preset (Windows!), the routine should look in the ODB under
/experiment/security/mserver/hosts.allow and /experiment/security/mserver/hosts.deny
and use that information instead of the files.

- We probably need different mechanisms for mserver and for mhttpd. The mserver
clients are usually only a few programs like the front-ends, while one may want to
control an experiment over mhttpd from much more machines. So we should establish a
second ACL for mhttpd. The already present "/experiment/security/allowed hosts" for
mhttpd should be converted into "/Experiment/Security/mhttpd/hosts.allow" and the
function access_allowed() should be used to interprete that, so that we only need to
write it once.

The mhttpd host-based access control list as used by ALPHA at CERN is now committed to
SVN (revision 4135).

When accepting connection from a remote host, the remote IP address is converted to a
hostname using gethostbyaddr(). If ODB directory "/experiment/security/mhttpd hosts",
exists, access is permitted if there is an entry for the this hostname. "localhost" is
always permitted.

In other words:

1) To enable the mhttpd access control list, create an ODB directory
"/experiment/security/mhttpd hosts".

2) From this moment, only access from "localhost" is permitted.

3) All connections from remote hosts are rejected with an error written into the midas
log file: Rejecting http connection from 'ladd05.triumf.ca'.

4) To permit access from remote hosts, take the hostname from this error message and
create an entry in "mhttpd hosts": odbedit -> cd "/Experiment/Security/mhttpd hosts" ->
create INT ladd05.triumf.ca

The idea behind this is that mhttpd is running behind an SSL proxy (or an SSH tunnel)
and only accepts connections from this proxy and perhaps from selected machines in the
experiment counting room.

P.S. I considered using tcp_wrappers, but this package does not seem to contain any
simple-to-use function "bool areTheyPermitted(const char* remoteHostname);".

P.P.S. The ODB path name is in variance from Stefan's email. I committed this code
before rereading it, please let me know if I should change the ODB paths.

P.P.P.S. I will now proceed with implementing similar code for the mserver/midas rpc.
Again, the use case is very simple: all machines permitted access to the mserver are
known in advance and can be listed in the access list. All unknown machines should be
rejected.

K.O.

> When accepting connection from a remote host, the remote IP address is converted to a
> hostname using gethostbyaddr(). If ODB directory "/experiment/security/mhttpd hosts",
> exists, access is permitted if there is an entry for the this hostname. "localhost" is
> always permitted.

While your "positive list" will certainly work, it is much more inflexible than a more
general hosts.allow/hosts.deny with wildcards. Assume some experiment decides it wants to
be controlled from all inside CERN. With hosts.allow/deny you could do

host.deny *
host.allow *.cern.ch

to have everything ending with "cern.ch" allowed. Otherwise it would be a nightmare finding
all possible terminals at CERN and add them manually. If you are considering modifying your
committed code to this scheme, you could have a look at my elog package, where exactly this
is implemented. You could copy/paste it from there.

After you finished, also talk to Pierre about documenting this in doxygen (or do it yourself).

> While your "positive list" will certainly work, it is much more inflexible than a more
> general hosts.allow/hosts.deny with wildcards. Assume some experiment decides it wants to
> be controlled from all inside CERN. With hosts.allow/deny you could do

I was going to bring this up later, but since mhttpd does not pass security audits, I believe
the only way it should be run in the modern computing environement is behind
a password-protected SSL proxy. In this case, the allow/deny list is very simple: deny all,
allow localhost (assuming httpd runs on the same host as mhttpd).

Speaking about CERN, "deny all; allow *.cern.ch" is the "default" setting, enforced by the CERN firewall. Our problem is with 
random "*.cern.ch" computers poking at our DAQ and crashing the mserver. Plus we do not want our competition to access our 
DAQ system, so "allow *.cern.ch" is a no go.

But since hosts.allow/hosts.deny is a superset of what I want, and since we can reuse existing code from elogd, I guess I have 
no ground to object your suggestion.

I will do the mserver/mrpc this way, then retrofit it into mhttpd. (But have to commit mlogger history changes first!!!).

K.O.

> I was going to bring this up later, but since mhttpd does not pass security audits, I believe
> the only way it should be run in the modern computing environement is behind
> a password-protected SSL proxy.

I recently built in native SSL into elogd.c and found it was very simple. We could do the same for mhttpd.

> Speaking about CERN, "deny all; allow *.cern.ch" is the "default" setting, enforced by the CERN firewall. Our problem is with 
> random "*.cern.ch" computers poking at our DAQ and crashing the mserver. Plus we do not want our competition to access our 
> DAQ system, so "allow *.cern.ch" is a no go.

I understand your point. But I want to tell you that there are other experiments, which want domain based access. For example at
PSI some experiments want allowed access from the experimental hall, which is the subdomain 129.129.140.* (there is not so much
competition here ;-) but not from other PSI subdomains. So you would need "deny all; allow 129.129.140.*; allow 129.129.228.*" for
example.

> I will do the mserver/mrpc this way, then retrofit it into mhttpd. (But have to commit mlogger history changes first!!!).

Agree.

#!/bin/sh

[ ! "$MIDASSYS" ] && MIDASSYS=/opt/MIDAS.PSI/Version/Current
[ ! "$HTTPPORT" ] && HTTPPORT=8080
[ ! "$SRVHOST" ]  && SRVHOST=localhost
LOGGER=${MIDASSYS}/bin/mlogger

EXPPATH=/home/das/online/test
CODEPATH=${EXPPATH}/code
LOGGER=${MIDASSYS}/bin/mlogger

PROG_FE=${CODEPATH}/frontend
PROG_ANA=${CODEPATH}/analyzer

if [ ! "$MIDAS_EXPTAB" ]; then
	MIDAS_DIR=${EXPPATH}
else
	MIDAS_EXPT_NAME="test"
fi

Hi!

I have released pymidas - Python binding to Midas.
It includes support for Online Database, Buffer, event
construction and parsing. 

We have used it for a couple years now here at CMD. (http://cmd.inp.nsk.su)
One of principal DAQ applications here (Slow Control Frontend) is
written in Python using pymidas.

http://cmd.inp.nsk.su/~bilenko/projects/pymidas/pymidas.html

Traceback (most recent call last):
  File "runtest.py", line 42, in <module>
    import midas
  File "/opt/MIDAS.PSI/Resources/PyMIDAS/pymidas/midas/__init__.py", line 140, in
<module>
    cmidas = ctypes.cdll.LoadLibrary('libmidas.so')
  File "/usr/lib/python2.5/site-packages/PIL/__init__.py", line 431, in LoadLibrary
    
  File "/usr/lib/python2.5/site-packages/PIL/__init__.py", line 348, in __init__
    
OSError: /opt/MIDAS.PSI/Versions/Current/lib/libmidas.so: undefined symbol:
cam16i_rq

Index: Makefile
===================================================================
--- Makefile    (revision 4129)
+++ Makefile    (working copy)
@@ -270,7 +270,7 @@

 OBJS =  $(LIB_DIR)/midas.o $(LIB_DIR)/system.o $(LIB_DIR)/mrpc.o \
        $(LIB_DIR)/odb.o $(LIB_DIR)/ybos.o $(LIB_DIR)/ftplib.o \
-       $(LIB_DIR)/mxml.o $(LIB_DIR)/cnaf_callback.o \
+       $(LIB_DIR)/mxml.o \
        $(LIB_DIR)/history.o $(LIB_DIR)/alarm.o $(LIB_DIR)/elog.o

 ifdef NEED_STRLCPY

--- Makefile    (revision 4129)
+++ Makefile    (working copy)
-       $(LIB_DIR)/mxml.o $(LIB_DIR)/cnaf_callback.o \
+       $(LIB_DIR)/mxml.o \
> i.e. remove cnaf_callback.o which causes the link errors.


Hi, Denis - I confirm that cnaf_callback.c is only used by MIDAS frontends that implement CAMAC
functions and that it should not required for building the MIDAS library. I am now looking at removing 
it from libmidas.

> I propose that libmidas.so is built by default, so when something breaks it won't go unnoticed

We have been through this before and decided that shared libraries are bad and we do not want to use 
them. The option for building libmidas.so was preserved, though.

Not to refight old wars, on reason against using shared libraries was version skew - one could never be 
sure what version of midas is being used - depending on the PATH, LD_LIBRARY_PATH, rpath settings, 
etc. There were other reasons, perhaps practical, perhaps with the mserver.

The main problem with "just build it", is that then the rest of midas will link against it bringing back all 
the problems we solved by going away from using shared libraries.

So back to your proposal about building libmidas.so - can you look and see if you can do the Python 
bindings with a statically linked midas library?

I know it is possible with Perl bindings - perl creates it's own shared library containing perl api glue 
linked against a foreign static library libfoo.a , so in theory, the shared library is not needed.

But perhaps, Python do things differently...

K.O.

Having libmidas.so is absolutely necessary for pymidas to work. If there was no such
option in Makefile pymidas users would have to build it themselves.

What I proposed though is that you change Makefile so it builds libmidas.so in
addition to (not instead of) static library. So if someone prefer to build
binaries statically they
may continue to do so. On other hand when someone needs a shared library they won't 
discover that it can't be easily built.

From "history" pages, the "create elog" button stopped working - it takes us to the elog entry form, but 
then, the "submit" button does not create any elog entries, instead dumps us into an invalid history 
display. This is using the internal elog.

This change in mhttpd.c::show_elog_new() makes it work again:

-       ("<body><form method=\"POST\" action=\"./\" enctype=\"multipart/form-data\">\n");
+       ("<body><form method=\"POST\" action=\"/EL/\" enctype=\"multipart/form-data\">\n");

Problem and fix confirmed with Linux/firefox and MacOS/firefox and Safari.

K.O.

> From "history" pages, the "create elog" button stopped working - it takes us to the elog entry form, but 
> then, the "submit" button does not create any elog entries, instead dumps us into an invalid history 
> display. This is using the internal elog.
> 
> This change in mhttpd.c::show_elog_new() makes it work again:
> -       ("<body><form method=\"POST\" action=\"./\" enctype=\"multipart/form-data\">\n");
> +       ("<body><form method=\"POST\" action=\"/EL/\" enctype=\"multipart/form-data\">\n");

This was a problem with relative URLs and it is now fixed. Svn revision 4131, fixes: delete elog, make elog from odb, make elog from history.

K.O.

Some time ago, the "remote CAMAC" functionality in mfe.c was made conditional on
HAVE_CAMAC. This flag is not set by default so remote camac calls silently do
not work, unless midas is compiled in a special way. I am too lazy to compile
midas differently depending on what hardware I use, so I split
register_cnaf_callback() into a separate file and made it easy to call directly
from the user front end.

I left the HAVE_CAMAC bits in mfe.c so people who use that would see no change.

Affected files:
Makefile (add cnaf_callback.o)
midas.h (add void register_cnaf_callback(int debug);
mfe.c (move the rpc code to cnaf_callback.c, call register_cnaf_callback())
cnaf_callback.c (new file)

K.O.

> Some time ago, the "remote CAMAC" functionality in mfe.c was made conditional on
> HAVE_CAMAC. This flag is not set by default so remote camac calls silently do
> not work, unless midas is compiled in a special way. I am too lazy to compile
> midas differently depending on what hardware I use, so I split
> register_cnaf_callback() into a separate file and made it easy to call directly
> from the user front end.
> 
> I left the HAVE_CAMAC bits in mfe.c so people who use that would see no change.
> 
> Affected files:
> Makefile (add cnaf_callback.o)
> midas.h (add void register_cnaf_callback(int debug);
> mfe.c (move the rpc code to cnaf_callback.c, call register_cnaf_callback())
> cnaf_callback.c (new file)
> 
> K.O.

That's a good idea. The frontend framework should be independent of the used
hardware (CAMAC or VME or whatever). I event went further and removed the HAVE_CAMAC
completely. This means that people have to add the call to register_cnaf_callback()
explicitly into the frontend user init routine. I think this inconvenience is not a
big deal because even before that people had to add the cnaf_callback.c file
explicitely into their Makefile. So they have to be aware of that change, and then
it's not a big deal to modify the init routine as well. But this way we have mfe.c
completely independen of the DAQ hardware which is how it should be.

To make things a bit easier, I modified the midas\examples\experiment\fronted.c to
contain this call, so people should be guided by that. I also added cnaf_callback.c
to the Makefile of the example frontend.

> > Affected files:
> > Makefile (add cnaf_callback.o)
> That's a good idea.
> To make things a bit easier, I modified the midas\examples\experiment\fronted.c to
> contain this call, so people should be guided by that. I also added cnaf_callback.c
> to the Makefile of the example frontend.

A request was made to remove cnaf_callback.o from libmidas as it creates a unwanted dependency on the CAMAC 
hardware driver when libmidas.so is used in programs that do not use CAMAC.

After looking around, it appears that removing cnaf_callback.o from libmidas would not break anything critical, 
other than CAMAC frontends that would fail to link with an obvious and easy to fix error.

I am leaving cnaf_callback.o in the Makefile - so it will be built and placed in linux/lib/cnaf_callback.o for anybody 
who wants to use it.

svn revision 4130.

K.O.

It looks like ODB function extract_key() will overwrite the array pointed to by "key_name" if given an odb 
path with very long names (as seems to happen when redirection explodes in the Safari web browser, via 
db_get_value(TRUE) via mhttpd "start program" button). All  callers of this function seem to provide 256 
byte strings, so the problem would not show up in normal use - only when abnormal odb paths are being 
parsed. Proposed solution is to add a "length" argument to this function. (Actually ODB path elements 
should be restricted to NAME_LENGTH (32 bytes), right?). K.O.

> It looks like ODB function extract_key() will overwrite the array pointed to by "key_name" if given an odb 
> path with very long names (as seems to happen when redirection explodes in the Safari web browser, via 
> db_get_value(TRUE) via mhttpd "start program" button). All  callers of this function seem to provide 256 
> byte strings, so the problem would not show up in normal use - only when abnormal odb paths are being 
> parsed. Proposed solution is to add a "length" argument to this function. (Actually ODB path elements 
> should be restricted to NAME_LENGTH (32 bytes), right?). K.O.

This is fixed in svn revision 4129.

K.O.

I now merged almost all the mhttpd changes from CERN AD-5/ALPHA. Only the code
for mhttpd HTTP:// access control list remains unmerged.

Changes to the history display code merged from ALPHA: 
- add option to show latest values of history variables: "show values of
variables" check box on the history config panel 
- add custom labels for each variables: instead of midas variable name, history
plots would show the text entered into the "label" text area 
- show history errors on the plot: before, if one out of 10 history variables
could not be plotted, nothing was shown at all, now all variables are \
show, those that could not be read with hs_read() show the error code 
- the selection of which variables to plot is alphanumerically sorted (adc11 >
adc9) [this code is not active for standard midas because mlogger sup\
port has not yet been committed] 
- the selection of which variables to plot shows the last variable selected, not
the first one - useful when entering variables from a long list [th\
is code is not active for standard midas because mlogger support has not yet
been committed] 
 
These changes have been extensively tested since last Summer at the AD-5 ALPHA
expt at CERN. 
I could only do minimal testing for this merged code, so if there are any errors, 
they would most likely be merge errors. This new code will be heavily used at
TRIUMF, 
so if any errors got any, we hope to flush them out quickly. 
 
As noted, mlogger support for some of the mhttpd functions is not in standard
midas yet. It will be committed shortly. 

K.O.

You misspelled one ODB entry:

Line 9014:
            sprintf(str, "/History/Display/%s/Label", path);

Line 9028:

            sprintf(str, "/History/Display/%s/Labels", path);
                                                ---^

I wonder how you could have tested that code for 1/2 year without noticing this error.
I fixed and committed it.
 

> You misspelled one ODB entry:
> Line 9014:
>             sprintf(str, "/History/Display/%s/Label", path);
> 
> Line 9028:
>             sprintf(str, "/History/Display/%s/Labels", path);
>                                                 ---^
> 
> I wonder how you could have tested that code for 1/2 year without noticing this error.
> I fixed and committed it.


It turns out that the program was tested as originally committed. With the above
modification, it corrupts ODB - originally, it used the wrong array element size to create
the wrong array. Corrected, it creates the right array with the wrong size, then
subsequent db_set_data_index() happily corrupts ODB.

Fix for mhttpd committed as svn revision 4128.
Fix for ODB corruption committed at svn revision 4129 (also fixes extract_key())


K.O.

I now encountered a new problem with mhttpd - I connect using the Safari 3.0.4 browser, go to the 
"Programs" page, press the button "Start feplc" (or any other "start" button) and instead of starting this 
program, I get an error in the browser, funny entries in ODB in "/Programs", corrupted ODB and a spew 
of messages in the midas log file about the mess. ODB has to be reloaded from backup to recover.

Investigation shows that the culprit is odd bahaviour of the "redirect()" function:

    /* start command */
    if (*getparam("Start")) {
       /* for NT: close reply socket before starting subprocess */
-      redirect2("?cmd=programs");
+      redirect2("/?cmd=programs");

The version without "/" makes Safari explode - it appends the "?cmd..." stuff to the existing URL, which 
already has the "?cmd..." tags, making a mess.

Firefox accepts either version.

ODB corruption happens here:
 
       sprintf(str, "/Programs/%s/Start command", name);
-      db_get_value(hDB, 0, str, command, &size, TID_STRING, TRUE);
+      db_get_value(hDB, 0, str, command, &size, TID_STRING, FALSE);
       if (command[0]) {
          ss_system(command);

It looks like db_get_value() would corrupt ODB if given funny "str". When Safari explodes,
funny strings are generated.

The simple fix is to replace "TRUE" with "FALSE", then at least db_get_value() does not try to make bogus 
entries in ODB.

The "Stop" command has the same problem, but does not currupt ODB - there is no db_get_value() in 
that code path.

I am reporting this "fresh" as I made one of our daq systems work again.

I did not investigate the history of changes to this "redirect" command (perhaps it was broken in the 
recent reorganisation of midas urls?), what versions of Safari work or not.

K.O.

>     /* start command */
>     if (*getparam("Start")) {
>        /* for NT: close reply socket before starting subprocess */
> -      redirect2("?cmd=programs");
> +      redirect2("/?cmd=programs");

The second version won't work if mhttpd is run under an Apache proxy. Assume the proxy redirects

http://proxy.ca/midas

to

http://daq.ca:8080

If you now do a redirect to "/?cmd=programs", you will end up at

http://proxy.ca/?cmd=programs

which is now what you want. I tried to put a "./?cmd=programs", and that bings you to

http://proxy.ca/midas/./?cmd=programs

which is correctly redirected to

http://daq.ca:8080/?cmd=programs

I tried with the windows version (ughhh) of Safari and it worked for me. So give it a try, the change is committed.

> ODB corruption happens here:
>  
>        sprintf(str, "/Programs/%s/Start command", name);
> -      db_get_value(hDB, 0, str, command, &size, TID_STRING, TRUE);
> +      db_get_value(hDB, 0, str, command, &size, TID_STRING, FALSE);
>        if (command[0]) {
>           ss_system(command);
> 
> It looks like db_get_value() would corrupt ODB if given funny "str". When Safari explodes,
> funny strings are generated.

What happes is an endless redirect from xxxx -> xxxx?cmd=Programs. So in the end you have

http://url.ca?cmd=programs?cmd=programs?cmd=programs?cmd=programs....

and in the end you get a stack overflow, which busts all.

> The simple fix is to replace "TRUE" with "FALSE", then at least db_get_value() does not try to make bogus 
> entries in ODB.

I changed both butting FALSE there and adding

   if (strchr(name, '?'))
      *strchr(name, '?') = 0;

which keeps the URL short.

So for me it looks fine at the moment, but I cannot guarantee that everything works, so keep an eye open on that.

While executing "make install" under MacOS 10.4.7, you may encounter errors about "dio". It is the 
problem of "Makefile". I did some change to it and attach the diff file here.

> While executing "make install" under MacOS 10.4.7, you may encounter errors about "dio". It is the 
> problem of "Makefile". I did some change to it and attach the diff file here.

I committed your patch. Thank you.

> While executing "make install" under MacOS 10.4.7, you may encounter errors about "dio". It is the 
> problem of "Makefile". I did some change to it and attach the diff file here.

Thank you very much for your instructions for installing Midas on MacOSX.
I followed your instructions to change the Makefile but I still get the following error message:

... 
... Installing programs and utilities to /usr/local/bin
... 
install: darwin/bin/lazylogger exists but is not a directory
install: darwin/bin/mchart exists but is not a directory
install: darwin/bin/mcnaf exists but is not a directory
install: darwin/bin/mdump exists but is not a directory
install: darwin/bin/melog exists but is not a directory
install: darwin/bin/mhdump exists but is not a directory
install: darwin/bin/mhist exists but is not a directory
install: darwin/bin/mhttpd exists but is not a directory
install: darwin/bin/mlogger exists but is not a directory
install: darwin/bin/mlxspeaker exists but is not a directory
install: darwin/bin/mserver exists but is not a directory
install: darwin/bin/mstat exists but is not a directory
install: darwin/bin/mtape exists but is not a directory
install: darwin/bin/odbedit exists but is not a directory
install: darwin/bin/odbhist exists but is not a directory
install: darwin/bin/stripchart.tcl exists but is not a directory
install: darwin/bin/webpaw exists but is not a directory
make: *** [install] Error 71

Could you help me solve this problem? Thank you in advance =)

I forgot to mention that, the following (and similar) lines:
           install -v -D -m 755 $$file $(SYSBIN_DIR)/`basename $$file` ; \
are changed into
           install -v -d -m 755 $$file $(SYSBIN_DIR)/`basename $$file` ; \

since -D is an illegal option for install. I am not sure whether -D in Linux means the same thing for -d in MacOSX install. 


> > While executing "make install" under MacOS 10.4.7, you may encounter errors about "dio". It is the 
> > problem of "Makefile". I did some change to it and attach the diff file here.
> 
> Thank you very much for your instructions for installing Midas on MacOSX.
> I followed your instructions to change the Makefile but I still get the following error message:
> 
> ... 
> ... Installing programs and utilities to /usr/local/bin
> ... 
> install: darwin/bin/lazylogger exists but is not a directory
> install: darwin/bin/mchart exists but is not a directory
> install: darwin/bin/mcnaf exists but is not a directory
> install: darwin/bin/mdump exists but is not a directory
> install: darwin/bin/melog exists but is not a directory
> install: darwin/bin/mhdump exists but is not a directory
> install: darwin/bin/mhist exists but is not a directory
> install: darwin/bin/mhttpd exists but is not a directory
> install: darwin/bin/mlogger exists but is not a directory
> install: darwin/bin/mlxspeaker exists but is not a directory
> install: darwin/bin/mserver exists but is not a directory
> install: darwin/bin/mstat exists but is not a directory
> install: darwin/bin/mtape exists but is not a directory
> install: darwin/bin/odbedit exists but is not a directory
> install: darwin/bin/odbhist exists but is not a directory
> install: darwin/bin/stripchart.tcl exists but is not a directory
> install: darwin/bin/webpaw exists but is not a directory
> make: *** [install] Error 71
> 
> Could you help me solve this problem? Thank you in advance =)

> I forgot to mention that, the following (and similar) lines:
>            install -v -D -m 755 $$file $(SYSBIN_DIR)/`basename $$file` ; \
> are changed into
>            install -v -d -m 755 $$file $(SYSBIN_DIR)/`basename $$file` ; \
> 
> since -D is an illegal option for install. I am not sure whether -D in Linux means the same thing for -d in MacOSX install. 

-D under linux means:

       -D     create all leading components of DEST except the last, then
              copy SOURCE to DEST; useful in the 1st format

This means if you install the first time, and eithe SYSBIN_DIR or `basename is not existing, it will be created on-the-fly from
the install program. If OSX does not support this, you somehow have to crate these subdirectories manually.

Thank you for your help =)

Since SYSBIN_DIR is defined as /usr/local/bin in the Makefile and it exists in my computer, so I deleted the -D in the Makefile and tried to "make install" again and the 
error message becomes:

... 
... Installing programs and utilities to /usr/local/bin
... 
/bin/sh: -c: line 2: syntax error: unexpected end of file
make: *** [install] Error 2

Can anyone help me solve this problem? 


> > I forgot to mention that, the following (and similar) lines:
> >            install -v -D -m 755 $$file $(SYSBIN_DIR)/`basename $$file` ; \
> > are changed into
> >            install -v -d -m 755 $$file $(SYSBIN_DIR)/`basename $$file` ; \
> > 
> > since -D is an illegal option for install. I am not sure whether -D in Linux means the same thing for -d in MacOSX install. 
> 
> -D under linux means:
> 
>        -D     create all leading components of DEST except the last, then
>               copy SOURCE to DEST; useful in the 1st format
> 
> This means if you install the first time, and eithe SYSBIN_DIR or `basename is not existing, it will be created on-the-fly from
> the install program. If OSX does not support this, you somehow have to crate these subdirectories manually.

Backend computer with SLC4.4 Linux did'not work as mserver because some security
protection under iptables service (could not connect with frontend computers).
The connection established if to make ( under root ) iptables disable
by command: service iptables stop, or much more gently
just to accept mserver port with command (under root):
iptables -I INPUT -p tcp --dport 1175 -j ACCEPT
( in /etc/service 
  midas    1175/tcp       #Midas server)
To check which ports is open
it is possible to use the command: "netstat -n" to see digital numbers of ports.

Hi All,

I'm testing MIDAS SVN rev-4113 on Scientific Linux 5.1 (i386) and the Analyzer 
can't start as Daemon. What I mean "can't" is that it stops running 
immediately without leaving any error messages. However, it can run offline or 
without becoming a Daemon. I have tested with ROOT 5.14e/5.16/5.18 and 
the "Experiment" example coming with MIDAS and this problem always happens. 
Any ideas?

Best Regards,
Jimmy

A major change was made to mhttpd, changing all internal URLs to relative paths.
This allows proxy access to mhttpd via an apache server for example, which might
be needed to securely access an experiment from outside the lab through a
firewall. Following setting can be places into the Apache configuration,
assuming the experiment runs on machine "online1.your.domain", and apache on a
publically available machine "www.your.domain":

Redirect permanent /online1 http://www.your.domain/online1
ProxyPass /online1/ http://online1.your.domain/

<Location "/online1">
  AuthType Basic
  AuthName ...
  AuthUserFile ...
  Require user ...
</Location>

If the the URL http://www.your.domain/online1 is accessed, it gets redirected
(after optional authentication) to http://online1.your.domain. If you click on
the mhttpd history page for example, mhttpd would normally redirect this to 

http://online1.your.domain/HS/

but this is not correct since you want to go through the proxy www.your.domain.
The new relative redirection inside mhttpd now redirects the history page
correctly to

http://www.your.domain/onlin1/HS/

I had to change many places inside mhttpd to make this work, and I'm not 100%
sure if I covered all occurrences. So if you upgrade to mhttpd revision 4115 and
observe some error accessing some pages, please report it to me.

- Stefan

> A major change was made to mhttpd, changing all internal URLs to relative paths.
> This allows proxy access to mhttpd via an apache server for example, which might
> be needed to securely access an experiment from outside the lab through a
> firewall.

It is good to see improvements to the MIDAS URLs. We have been successfully running
mhttpd behind an apache SSL/HTTPS proxy without these changes, but our case was very
limited to one experiment, one mhttpd behind one proxy. I hope to test these changes
in the near future at CERN, I guess we will hear if things broke. I am especialloy
worried about the function for "split mhttpd history generator" via "/History/URL".
I remember it was hard to get it right and I hope if this function did not survive
this update, it will be easy to resurrect.
K.O.

I'm using midas to read data from a waveform digitizer at event rates of
10-30kHz. To accomplish this the digitizer is read via Block transfers and the
raw data put into a single MIDAS event.  Thus a MIDAS event could contain upto
250 physical events and at maximum 350kBytes.  In the analyzer modules I had
been analyzing the first physics event contained in a MIDAS event with no
problem.  Recently I tried to analyze all the physical events.  At low rates,
100hz-1khz, this was no problem, 1-5 physical events in a MIDAS event.  At
higher rates 10-20kHz, where there are about 40physical events per MIDAS event,
 the analyzer keeps up for a  few seconds then seg faults with " 'shared object
read from target memory' has disappear; keeping it symbols".  Any suggestions as
to why the analyzer is crashing would be very helpful.

Thanks,

Robert     

> I'm using midas to read data from a waveform digitizer at event rates of
> 10-30kHz. To accomplish this the digitizer is read via Block transfers and the
> raw data put into a single MIDAS event.  Thus a MIDAS event could contain upto
> 250 physical events and at maximum 350kBytes.  In the analyzer modules I had
> been analyzing the first physics event contained in a MIDAS event with no
> problem.  Recently I tried to analyze all the physical events.  At low rates,
> 100hz-1khz, this was no problem, 1-5 physical events in a MIDAS event.  At
> higher rates 10-20kHz, where there are about 40physical events per MIDAS event,
>  the analyzer keeps up for a  few seconds then seg faults with " 'shared object
> read from target memory' has disappear; keeping it symbols".  Any suggestions as
> to why the analyzer is crashing would be very helpful.

I personally have never seen this error message. The analyzer is designed such that
it produces "back pressure" if the data rate is higher than the analysis rate and
you have "request all events" on. The only thing I can image are the following two
issues:

- At higher rate where you have more than 40 physical events per MIDAS event, there
is some bug in your analysis code which gets exploited only in that case. Maybe some
temporary array which is only 35 entries long or something like this.

- The back pressure mentioned above will slow down the frontend. If your computer
busy logic is not working correctly, you might get more triggers than you can
acquire. Maybe then the data gets screwed up and the analyzer chokes on it.

Finding the exact reason is not simple. For sure you have to run the analyzer inside
the debugger, to see exactly where the segfault happens. You then maybe have to
produce some dummy data in the frontend (like always sending the same event) to
disentangle some possible trigger problems from other problems.

Best regards,

  Stefan