Back Midas Rome Roody Rootana
  Midas DAQ System, Page 113 of 136  Not logged in ELOG logo
IDdown Date Author Topic Subject
  473   23 Mar 2008 Konstantin OlchanskiInfoHistory SQL database poll: MySQL, PgSQL, ODBC?
I would like to hear from potential users on which SQL database would be
preferable for storage of MIDAS history data.

My current preference is to use the ODBC interface, leaving the choice of
database engine to the user. While ODBC is not pretty, it appears to be adequate
for the job, permits "funny" databases (i.e. flat files) and I already have
prototype implementations for reading (mhttpd) and writing (mhdump/mlogger)
history data using ODBC.

In practice, MySQL and PgSQL are the main two viable choices for using with the
MIDAS history system. We tested both (no change in code - just tell ODBC which
driver to use) and both provide comparable performance and disk space use. We
were glad to see that the disk space use by both SQL databases is very
efficient, only slightly worse than uncompressed MIDAS history files.

At TRIUMF, for T2K/ND280, we now decided to use MySQL - it provides a better
match to MIDAS data types (has 1-byte and 2-byte integers, etc) and appears to
have working database replication (required for our use).

With mlogger already including support for MySQL, and MySQL being a better match
for MIDAS data, this gives them a slight edge and I think it would be reasonable
choice to only implement support for MySQL.

So I see 3 alternatives:

1) use ODBC (my preference)
2) use MySQL exclusively
3) implement a "midas odbc layer" supporting either MySQL or PgSQL.

Before jumping either way, I would like to hear from you folks.

K.O.
  472   23 Mar 2008 Konstantin OlchanskiInfoPer-variable history implementation in the mlogger
> The changes to mlogger implementing per-variable history have been committed to
> svn. Revision 4145.

To make code changes more clear, the commit was done in 3 stages:

revision 4142+4143 are minor fixes, refactoring (switch the code to use helper
functions) and implementation of history for structured banks
revision 4144 implements the per-variable history
revision 4145 is minor cleanup.

K.O.
  471   23 Mar 2008 Konstantin OlchanskiInfoPer-variable history implementation in the mlogger
The changes to mlogger implementing per-variable history have been committed to
svn. Revision 4145.

The rationale for these changes is roughly described in
https://ladd00.triumf.ca/elog/Midas/347

The main user-visible effect is reduction of data volume written to history
files and better integration with the history plot system in mhttpd.

The new functionality is disabled by default, pending review by Stefan (Except
for /history/tags stuff, which will be created by mlogger and used by mhttpd).
To enable it, set "/equipment/xxx/Common/PerVariableHistory" to 1 (type TID_INT).

In the "per-variable" mode, each entry in /equipment/xxx/variables is assigned
it's own event id and creates it's own events in the history file. In the
"classical" (or per-equipment) mode, all variables are assigned the same event
id (equal to the equipment id) and are written to disk at the same time.

In other words, in per-equipment mode, if there are 100 variables and 1 of them
is updated, all 100 numbers are written to disk. In per-variable mode, only the
one updated variable is written out.

The one point for review in this implementation is the assignment of event id's.
Committed code uses the formula "1000*eq_id + n" (i.e. variables in equipment id
2 get 2001, 2002, etc..., equipment id 3 get 3001, 3002, ...). This formula
works for most experiments, but as I understand is no good for some experiments
at PSI. Other than inventing a better formula that would work for everybody in
every case, one can also assign event id's manually by creating appropriate
entries in "/history/events".

This code has been used at CERN for running ALPHA since last Summer and it will
be used extensively at TRIUMF for T2K/ND280 slow controls. Per-variable history
is also required for the pending implementation of "history logged directly to
an SQL database", to be used at T2K/ND280.

If history (ahem) is any guide, we will now have a brief period of fixing merge
errors and "works for me" mistakes.

K.O.
  470   12 Mar 2008 Konstantin OlchanskiSuggestionNew Makefile for building MIDAS
> > Most experience with autoconf/automake is all negative. The promise was "never debug your Makefile ever 
> > again!", delivered was "debug the configure script instead!". 
> 
> I admit that the new one is fit to my flavor. For a common user, I think, a simple procedure of configure/make/install
> is better than changing the Makefile manually because many users are lack of knowledges about Makefile. That's why 
> I want to learn autotools.

The reality is that you will never deliver a Makefile/Configure script that works for everybody in every case - users will always have a need to tweak the build 
process to suit their needs. In this situation, "Makefile" is a much better language and "make" is a much better tool for users to deal with - much simpler, better 
documented and better understood compared to autotools (*nobody* understands autotools; also compare the size of the midas Makefile with the size of a 
typical configure script).

Anyhow, we will be cross-compiling midas to run on a PowerPC processor inside a Virtex4 FPGA. Go handle that with configure scripts.

K.O.
  469   11 Mar 2008 Exaos LeeSuggestion"Makefile-by-EL" updated
> The linking of mhttpd misses a "-lm":
> 
> cc -g -O3 -Wall -Wuninitialized -DINCLUDE_FTPLIB -D_LARGEFILE64_SOURCE -DOS_LINUX
> -fPIC -Wno-unused-function   -DHAVE_ZLIB -Iinclude -Idrivers -I../mxml -o
> linux/bin/mhttpd linux/lib/mhttpd.o linux/lib/mgd.o linux/lib/libmidas.a -lutil
> -lpthread -lz
> linux/lib/mhttpd.o(.text+0xe08f): In function `show_custom_gif':
> src/mhttpd.c:5058: undefined reference to `log'
> linux/lib/mhttpd.o(.text+0xe0a8):src/mhttpd.c:5058: undefined reference to `log'
> 

Strange. I tested it on Debian Linux 4.0r2 AMD64 with gcc 4.1.2, MIDAS SVN 4124. It worked fine.
Anyway, it can be fixed by addling "-lm" to the initial"LDFLAGS".

> The header of the makefile should contain a short description, the author(s), an
> $Id:$ tag for SVN, some explanation what "icc", "ifort" means, a note about the
> CFLAGS and a clear statement what can be modified by the user and why and what not.

OK. I will comment it in detail.
  468   11 Mar 2008 Stefan RittSuggestion"Makefile-by-EL" updated
The linking of mhttpd misses a "-lm":

cc -g -O3 -Wall -Wuninitialized -DINCLUDE_FTPLIB -D_LARGEFILE64_SOURCE -DOS_LINUX
-fPIC -Wno-unused-function   -DHAVE_ZLIB -Iinclude -Idrivers -I../mxml -o
linux/bin/mhttpd linux/lib/mhttpd.o linux/lib/mgd.o linux/lib/libmidas.a -lutil
-lpthread -lz
linux/lib/mhttpd.o(.text+0xe08f): In function `show_custom_gif':
src/mhttpd.c:5058: undefined reference to `log'
linux/lib/mhttpd.o(.text+0xe0a8):src/mhttpd.c:5058: undefined reference to `log'

The header of the makefile should contain a short description, the author(s), an
$Id:$ tag for SVN, some explanation what "icc", "ifort" means, a note about the
CFLAGS and a clear statement what can be modified by the user and why and what not.
  467   10 Mar 2008 Exaos LeeSuggestion"Makefile-by-EL" updated
Sorry, this line:
EXECS += $(EXAMPLES:%/$(BIN_DIR)/%)
should be replaced by
EXECS += $(EXAMPLES:%=$(BIN_DIR)/%)
  466   10 Mar 2008 Exaos LeeSuggestion"Makefile-by-EL" updated
> Not that the current Makefile is too pretty (I have seen worse), but it 
works and it is fairly compact for a project of 
> this complexity, it handles a large number of operating systems and build 
options very efficiently.
> 
> I think you found that out with your rewriting exercise - your version of 
the Makefile contains all the same code, 
> just rearranged to suite your taste, with existing bugs preserved and new 
bugs added.

I derived the new Makefile from the original one so that feathers and bugs are 
also included. 
I havn't experiences on platforms other than Linux and MacOS, so I cannot 
recognize bugs on 
other platforms if they exists in the original one. And if there are bugs, 
hope users can figure
them out.

>
> As they say, the devil is in the details. I notice some subtle changes in 
your Makefile that make me go "what?":
> 
> 1) the command for building the midas shared library used to be "ld -
shared", in your version, "-shared" is gone. 
> But check with the GCC manual, today's recommended command is probably "gcc -
shared".

Fixed.

> 2) mhdump is now linked with ROOT, but I wrote it recently enough to 
remember that it does not use ROOT

The building dependence on ROOT of mhdump may be eliminated by changing the 
specific target.

> 3) hand-crafted dependancies have been replaced with generic "almost 
every .o depends on every .h", which is 
> incorrect. The "almost every .o" part bothers me.

Fixed now.

> 4) "make clean" runs "rm -rf" - plain scary.

Fixed.

> 5) "$(shell ...)" is overused

Replaced with GNU make internal methods.


> I think by the end all these little details are sorted out and all the 
quirks are put back in, your Makefile will look no 
> better than the current Makefile.

I realized it now. But anyway, it looks tidy to me now. I still hope to use 
AUTOTOOLS with MIDAS.
 
> > 2. The file is less than 400 lines now. The original one is more than 500 
lines.

The new one is about 430 lines. Hmmm, it reaches the original one which is 
more than 600 lines.

> 
> It looks like your savings came from removing comments, removing hand-
crafted dependancy lists and replacing 
> fairly verbose "make install" targets (which we do not use anyway) with your 
own much simpler scripts.
> 
> All the juicy bits needed to actually build all the code appear to take 
about as much space as before.
> 
> Also the original mistake of recompiling programs when they only need 
relinking was not fixed. (For example, 
> when libmidas is updated, to update mhttpd, the current Makefile needlessly 
recompiles mhttpd.c. Better use 
> would be to compile mhttpd.c into mhttpd.o, then only a relink is needed).

Fixed.

> 
> Most experience with autoconf/automake is all negative. The promise 
was "never debug your Makefile ever 
> again!", delivered was "debug the configure script instead!". In practice, 
with autoconf/automake, you try to run 
> configure, kludge it until it stops crashing, then tweak the 
incomprehensible Makefiles it produces until the code 
> compiles.
> 
> K.O.

====================================================
Maybe BUGS or FEATURES in this new one:
1. The shared libmidas.so and the static libmidas.a are built sperately. 
The "libmidas.a" is 
always built whether "NEED_SHLIB" is set or not. And all executables are built 
staticly default. 
I commented this in the Makefile-by-EL. Hence, if you want to use libmidas.so 
with PyMIDAS and
do not want to encounter "Segmentation fault" while executing the utilies 
linked dynamicly, you
may try this one.
2. I found that "minife" is failed to be built, so I remove it from the 
example list.
3. Some bugs while building on MacOS Tiger 10.4.11 PPC are commented out in 
the Makefile. These
bugs are still exists in the original one.
4. Using "VPATH" instead of adding pathnames.
5. Using "UTILS_SUID" to handle utilities which need SUID mode. And 
the "UTILS_SUID_NEED" may be
defined in the OS-specific field, so you need not to use OS-specific commands 
in the "install"
target.
6. Using "tr" with "uname" in order to delete some extra "ifeq 
($(OSTYPE),...)".
7. Other things, please see the file.

Anyway, easier building is my purpose. :-)
Attachment 1: Makefile-by-EL
#-include MIDAS-env.mk --- split the environment setup in a seperate file?

# initialized flags

#CC = icc
#FC = ifort
#LD = ld
CFLAGS   = -g -O3 -Wall -Wuninitialized -DINCLUDE_FTPLIB

###########################################################################
## Environments setup for building MIDAS/PSI

## Prefix
ifndef PREFIX
PREFIX = /usr/local
endif

# User defined flags for compiler
USERFLAGS     =

# Do you need to compile the shared library? If so, set "1" here.
NEED_SHLIB    =

# If RPATH needed ...
NEED_RPATH    =

# If static ROOT libaries needed ...
NEED_LIBROOTA =

# If strlcpy.o needed ...
NEED_STRLCPY  = 1

# If zlib needed ...
NEED_ZLIB     = 1

# MIDAS max event size
MIDAS_MAX_EVENT_SIZE =

# OS specific programs included
SPECIFIC_OS_PRG      =

# ..? Why it is needed?
MIDAS_PREF_FLAGS     =

# Build examples or not? If so, set "1" here ...
EXAMPLES_NEED = 1

## installer
INSTALL  = $(shell which install)

# **** CAUTION ****
# **** On MacOS Tiger, "/usr/bin/install" doesn't accept "-D" option!
# **** NEED to be fixed ...
ifndef INSTALL
INSTALL  = ./install.sh
else
INS_OPTS = -v -D
endif

# MXML dir
ifndef MXML_DIR
MXML_DIR = ../mxml
endif

# MySQL
# **** CAUTION ****
# **** On MacOS Tiger (10.4.11), the "MYSQL_CONFIG" will be always defined because
# **** "which" will always return something whether it finds the exec or not!
# **** NEED to be fixed!
MYSQL_CONFIG := $(shell which mysql_config 2> /dev/null)

ifdef MYSQL_CONFIG
MYSQLINCDIR := $(shell mysql_config --cflags | sed -e 's,^.*-I\([^ ]*\).*$$,\1,' -e s/\'//g)
NEED_MYSQL  := $(shell if test -e $(MYSQLINCDIR)/mysql.h ; then echo 1; fi)
endif

###########################################################################
## Check OS Type for OS-specified flags

# *** CAUTION *** Does "tr" exist on all POSIX platform?
OSTYPE = $(shell uname | tr '[A-Z]' '[a-z]')

ifeq ($(OSTYPE), linux)
# > 2GB file support
CFLAGS  += -D_LARGEFILE64_SOURCE
CFLAGS  += -DOS_LINUX -fPIC -Wno-unused-function
LDFLAGS += -lutil -lpthread
NEED_ZLIB        = 1
SPECIFIC_OS_PRG += mlxspeaker dio
endif

ifeq ($(OSTYPE), osf1)
CFLAGS  += -DOS_OSF1
FFLAGS  += -nofor_main -D 40000000 -T 20000000
LDFLAGS += -lc -lbsd
endif

ifeq ($(OSTYPE), ultrix)
CFLAGS  += -DOS_ULTRIX -DNO_PTY
endif

ifeq ($(OSTYPE), freebsd)
CFLAGS  += -DOS_FREEBSD
LDFLAGS += -lbsd -lcompat
endif

ifeq ($(OSTYPE), darwin)
CFLAGS  += -DOS_LINUX -DOS_DARWIN -DHAVE_STRLCPY -fPIC -Wno-unused-function
LDFLAGS += -lpthread
NEED_RANLIB      = 1
SPECIFIC_OS_PRG += mlxspeaker
endif

ifeq ($(OSTYPE), cygwin_nt-5.1)
OSTYPE = cygwin
endif

ifeq ($(OSTYPE), cygwin)
CFLAGS  += -DOS_LINUX -DOS_CYGWIN -Wno-unused-function
LDFLAGS += -lutil -lpthread
endif

ifeq ($(OSTYPE), solaris)
CFLAGS  += -DOS_SOLARIS
LDFLAGS += -lsocket -lnsl
endif

###########################################################################
## Processing environment for building flags

CFLAGS += $(MIDAS_PREF_FLAGS) $(USERFLAGS)

ifdef MIDAS_MAX_EVENT_SIZE
CFLAGS   += -DMAX_EVENT_SIZE=$(MIDAS_MAX_EVENT_SIZE)
endif # MIDAS_MAX_EVENT_SIZE

ifdef NEED_MYSQL
CFLAGS     += -DHAVE_MYSQL $(shell $(MYSQL_CONFIG) --include)
LDFLAGS    += $(shell $(MYSQL_CONFIG) --libs)
NEED_ZLIB   = 1
endif # NEED_MYSQL

ifdef ROOTSYS
ROOTLIBS   := $(shell $(ROOTSYS)/bin/root-config --libs)
ROOTCFLAGS := $(shell $(ROOTSYS)/bin/root-config --cflags)

ifdef NEED_RPATH
ROOTLIBS   += -Wl,-rpath,$(ROOTSYS)/lib
endif

ifdef NEED_LIBROOTA
ROOTLIBS   := $(ROOTSYS)/lib/libRoot.a -lssl -ldl -lcrypt
endif

CFLAGS     += $(ROOTCFLAGS)
endif # ROOTSYS

ifdef NEED_ZLIB
CFLAGS     += -DHAVE_ZLIB
LDFLAGS    += -lz
endif # ZLIB

###########################################################################
## Processing building targets

## Directories needed
INC_DIR = include
SRC_DIR = src
UTL_DIR = utils
DRV_DIR = drivers
EXM_DIR = examples

## Driectories which will be created
OS_DIR  = $(OSTYPE)
LIB_DIR = $(OS_DIR)/lib
BIN_DIR = $(OS_DIR)/bin
DIRS    = $(OS_DIR) $(LIB_DIR) $(BIN_DIR)

CFLAGS += -I$(INC_DIR) -I$(DRV_DIR) -I$(MXML_DIR)

## Directories for install
INSTDIR_BIN = $(PREFIX)/bin
INSTDIR_INC = $(PREFIX)/include
INSTDIR_LIB = $(PREFIX)/lib
INSTDIR_DRV = $(PREFIX)/driver

## Utilities
UTILS = mserver mhttpd mlogger odbedit mtape  mhist    \
	mstat   mcnaf  mdump   mhdump  mchart odbhist  \
	melog   webpaw lazylogger   stripchart.tcl  \
	$(SPECIFIC_OS_PRG)

# Utilities need to change SUID mode
UTILS_SUID_NEED = dio mhttpd webpaw
UTILS_SUID = $(UTILS_SUID_NEED:%=$(BIN_DIR)/%)

## Analyzer objects
ANALYZER  = $(LIB_DIR)/mana.o

ifdef CERNLIB
ANALYZER += $(LIB_DIR)/hmana.o
endif
ifdef ROOTSYS
ANALYZER += $(LIB_DIR)/rmana.o
endif

## Objects for building $(LIBMIDAS_A) and $(LIBMIDAS_SO)
OBJS4LIB_ALL = midas.o system.o mrpc.o odb.o ybos.o ftplib.o history.o alarm.o elog.o
OBJS4LIB_A   = $(OBJS4LIB_ALL:%=$(LIB_DIR)/%) $(LIB_DIR)/cnaf_callback.o
OBJS4LIB_SO  = $(OBJS4LIB_ALL:%=$(LIB_DIR)/%)

## MXML OBJS
MXML_OBJS_NEED  = mxml.o
ifdef NEED_STRLCPY
MXML_OBJS_NEED += strlcpy.o
endif

MXML_OBJS = $(MXML_OBJS_NEED:%=$(LIB_DIR)/%)

## Objects for install
OBJS_INS_NEED = mfe.o fal.o
OBJS_INS = $(OBJS_INS_NEED:%=$(LIB_DIR)/%)

## library
LIBMIDAS_A  = $(LIB_DIR)/libmidas.a

ifdef NEED_SHLIB
LIBMIDAS_SO = $(LIB_DIR)/libmidas.so

CFLAGS     += -shared -fPIC
LDFLAGS    += -shared
# if need to link utilities dynamically with libmidas.so, add this flag...
LDFLAGS_SH += -shared -L$(LIB_DIR) -lmidas
endif # NEED_SHLIB

## Examples
EXAMPLES = consume produce rpc_test msgdump minirc odb_test #minife - failed building!

## All executables
EXECS  = $(UTILS:%=$(BIN_DIR)/%)
ifdef EXAMPLES_NEED
EXECS += $(EXAMPLES:%/$(BIN_DIR)/%)
endif

###########################################################################

CXXFLAGS  = $(CFLAGS)

ifdef ROOTSYS
CXXFLAGS += -DHAVE_ROOT
endif

all: check-mxml $(DIRS) $(LIBMIDAS_A) $(LIBMIDAS_SO) \
     $(ANALYZER) $(OBJS_INS) $(EXECS)

.PHONY: clean

examples: $(LIBMIDAS_A) $(EXAMPLES:%=$(BIN_DIR)/%)

## Static
static:
	@rm -f $(EXECS)
	@$(MAKE) USERFLAGS=-static

## Install
install: install-utils install-includes install-libs \
	 install-drivers install-src install-examples
	@echo "..."
	@echo "... Happy working with MIDAS! "
	@echo "..."

###########################################################################

$(DIRS):
	@echo "Making directory $@ ..."
	@mkdir -p $@

## Libraries
$(LIBMIDAS_A): $(OBJS4LIB_A) $(MXML_OBJS)
	@rm -f $@
	@echo "Creating $@ ..."
	@ar -crv $@ $^
ifdef NEED_RANLIB
	@ranlib $@
endif

$(LIBMIDAS_SO): $(OBJS4LIB_SO) $(MXML_OBJS)
	@rm -f $@
	@echo "Creating $@ ..."
	@$(CC) $(LDFLAGS) -o $@ $^

## MXML-check
check-mxml:
ifdef NEED_STRLCPY
	@if test ! -e $(MXML_DIR)/strlcpy.h ; then \
		echo "Please download mxml from: " ; \
		echo "http://midas.psi.ch/htmldoc/quickstart.html" ;\
		exit 1; \
	fi
... 135 more lines ...
  465   10 Mar 2008 Exaos LeeSuggestionNew Makefile for building MIDAS
> The Makefile is missing -lzip:

Sorry, spelling error.
The "LIBS +=" should be replaced by "LDFLAGS +="
  464   10 Mar 2008 Exaos LeeSuggestionNew Makefile for building MIDAS
> Most experience with autoconf/automake is all negative. The promise was "never debug your Makefile ever 
> again!", delivered was "debug the configure script instead!". In practice, with autoconf/automake, you try to run 
> configure, kludge it until it stops crashing, then tweak the incomprehensible Makefiles it produces until the code 
> compiles.
> 
> K.O.

I admit that the new one is fit to my flavor. For a common user, I think, a simple procedure of configure/make/install
is better than changing the Makefile manually because many users are lack of knowledges about Makefile. That's why 
I want to learn autotools. The configure script is generated automatically by "autoconf", so you needn't to debug it. 
For the developer, you need to debug the configure.ac/in files for generating the configure script. For a common user, 
he/she only needs to run it. In fact, some more complex projects like ROOT use AUTOTOOLS and they don't include 
the original files which are needed for generating the "configure" script.  I prefer the MIDAS project includes such a 
script to make the compiling simpler and easier instead of changing the Makefile manually. 
  462   10 Mar 2008 Stefan RittBug Reportarray overflows and other bugs
There were some trivial and some non-trivial issues. Glad the compiled picked up on
this!

> I see loads of warnings during compile, most of which I know from earlier 
> compiles:
> * warning: dereferencing type-punned pointer will break strict-aliasing rules
> * warning: pointer targets in passing argument 3 of 'getsockname' differ in
>            signedness

I ignore these for the moment until I have a gcc 4.2 myself (we use Scientific
Linux 5 which has gcc 4.1 for the moment). As Randolph pointed out correctly you
can make gcc shut up by a proper flag there. The warnings have no influence on the
stability of midas.

> (1)=========================
> src/midas.c:7398: warning: array subscript is above array bounds
> Inspection of midas.c:
> 
>    if (i == MAX_DEFRAG_EVENTS) {
>       /* no buffer available -> no first fragment received */
> 7398: free(defrag_buffer[i].pevent);
>       memset(&defrag_buffer[i].event_id, 0, sizeof(EVENT_DEFRAG_BUFFER));
>       cm_msg(MERROR, "bm_defragement_event",
>              "Received fragment without first fragment (ID %d) Ser#:%d",
>              pevent->event_id & 0x0FFF, pevent->serial_number);
>       return;
>    }

The free() was just wrong at that place, I removed it.

> (2)==========================
> src/midas.c:2958: warning: array subscript is above array bounds
> 
>    for (i = 0; i < 13; i++)
> 2958  if (trans_name[i].transition == transition)
>          break;

Fixed that by
     
     for (i=0 ;; i++)
        if (trans_name[i].name[0] == 0 || trans_name[i].transition == transition)
           break;

Since trans_name[i].name = "" indicates the end of the list.

> (3)=============================
> mfe.c:
> src/mfe.c:412: warning: array subscript is above array bounds
> src/mfe.c:311: warning: array subscript is above array bounds
> src/mfe.c:340: warning: array subscript is above array bounds
> 
> 412: device_drv->dd(CMD_GET_DEMAND, device_drv->dd_info, i, 
>           &device_drv->mt_buffer->channel[i].array[CMD_GET_DEMAND]);

The code at 412 was wrong there, the demand value is queried later by the device
driver directly. For the other two occurences (311 and 340) I had to really
increase the array size by one. This issue can cause segfaults if you have a slow
control front-end which uses multithreading (not many people use it except me).

> (4)=========================
> src/lazylogger.c:1957: warning: array subscript is below array bounds
> 
> if ((channel < 0) && (lazyinfo[channel].hKey != 0))
> 
> That is lazyinfo[something below zero].

This has to be fixed by Pierre. I guess an or instead of an and would do it, but
I'm not 100% sure.

> (5)=============================
> More warnings an expert might want to have a look at:
> 
> * warning: deprecated conversion from string constant to 'char*'
> 
> * src/fal.c:106: warning: non-local variable '<anonymous struct> out_info'
>                  uses anonymous type
> * src/fal.c:3064: warning: non-local variable '<anonymous struct> eb' uses
>                   anonymous type
> 
> I attach the full output of make.
> Could someone knowledgeable please have a look at these warnings and fix them?

Uahhh. Especially the "const char*" vs. "char*" is in principle right, but will
cause a major rework. Probably hundreds of occations have to be fixed. Many strings
must be declared const, others not. It will help the programmer to find some errors
during compile which would later show up only during runtime (like writing into a
fixed string), but I only will go through that when I have gcc 4.2 installed
myself, and have two free days to work on this ;-)

> They make me a bit nervous when thinking about data integrity, and
> there are now so many that they actually start to hide serious stuff
> like the ones I presented.

Except the slow control stuff (which only is an issue for multithreaded frontends)
none of the above things will have an influence on the data integrity. But I agree
that they should be fixed.

- Stefan
  461   10 Mar 2008 Stefan RittSuggestionRFC- ACLs for midas rpc, mserver, mhttpd access
> I was going to bring this up later, but since mhttpd does not pass security audits, I believe
> the only way it should be run in the modern computing environement is behind
> a password-protected SSL proxy.

I recently built in native SSL into elogd.c and found it was very simple. We could do the same for mhttpd.

> Speaking about CERN, "deny all; allow *.cern.ch" is the "default" setting, enforced by the CERN firewall. Our problem is with 
> random "*.cern.ch" computers poking at our DAQ and crashing the mserver. Plus we do not want our competition to access our 
> DAQ system, so "allow *.cern.ch" is a no go.

I understand your point. But I want to tell you that there are other experiments, which want domain based access. For example at
PSI some experiments want allowed access from the experimental hall, which is the subdomain 129.129.140.* (there is not so much
competition here ;-) but not from other PSI subdomains. So you would need "deny all; allow 129.129.140.*; allow 129.129.228.*" for
example.

> I will do the mserver/mrpc this way, then retrofit it into mhttpd. (But have to commit mlogger history changes first!!!).

Agree.
  460   10 Mar 2008 Konstantin OlchanskiSuggestionRFC- ACLs for midas rpc, mserver, mhttpd access
> While your "positive list" will certainly work, it is much more inflexible than a more
> general hosts.allow/hosts.deny with wildcards. Assume some experiment decides it wants to
> be controlled from all inside CERN. With hosts.allow/deny you could do

I was going to bring this up later, but since mhttpd does not pass security audits, I believe
the only way it should be run in the modern computing environement is behind
a password-protected SSL proxy. In this case, the allow/deny list is very simple: deny all,
allow localhost (assuming httpd runs on the same host as mhttpd).

Speaking about CERN, "deny all; allow *.cern.ch" is the "default" setting, enforced by the CERN firewall. Our problem is with 
random "*.cern.ch" computers poking at our DAQ and crashing the mserver. Plus we do not want our competition to access our 
DAQ system, so "allow *.cern.ch" is a no go.

But since hosts.allow/hosts.deny is a superset of what I want, and since we can reuse existing code from elogd, I guess I have 
no ground to object your suggestion.

I will do the mserver/mrpc this way, then retrofit it into mhttpd. (But have to commit mlogger history changes first!!!).

K.O.
  459   10 Mar 2008 Konstantin OlchanskiSuggestionNew Makefile for building MIDAS
> I rewrote the Makefile for MIDAS in order to make it tidy.

Not that the current Makefile is too pretty (I have seen worse), but it works and it is fairly compact for a project of 
this complexity, it handles a large number of operating systems and build options very efficiently.

I think you found that out with your rewriting exercise - your version of the Makefile contains all the same code, 
just rearranged to suite your taste, with existing bugs preserved and new bugs added.

> I tested it on my box and it works here.

As they say, the devil is in the details. I notice some subtle changes in your Makefile that make me go "what?":

1) the command for building the midas shared library used to be "ld -shared", in your version, "-shared" is gone. 
But check with the GCC manual, today's recommended command is probably "gcc -shared".
2) mhdump is now linked with ROOT, but I wrote it recently enough to remember that it does not use ROOT
3) hand-crafted dependancies have been replaced with generic "almost every .o depends on every .h", which is 
incorrect. The "almost every .o" part bothers me.
4) "make clean" runs "rm -rf" - plain scary.
5) "$(shell ...)" is overused

I think by the end all these little details are sorted out and all the quirks are put back in, your Makefile will look no 
better than the current Makefile.

> 2. The file is less than 400 lines now. The original one is more than 500 lines.

It looks like your savings came from removing comments, removing hand-crafted dependancy lists and replacing 
fairly verbose "make install" targets (which we do not use anyway) with your own much simpler scripts.

All the juicy bits needed to actually build all the code appear to take about as much space as before.

Also the original mistake of recompiling programs when they only need relinking was not fixed. (For example, 
when libmidas is updated, to update mhttpd, the current Makefile needlessly recompiles mhttpd.c. Better use 
would be to compile mhttpd.c into mhttpd.o, then only a relink is needed).

> I tried to learn "autoconf" and "automake" in order to make building MIDAS more
> compatible for various platforms. But I havn't enough time now. Hope somebody
> can help it. The attached file is original named "Makefile.in" for using "autoconf".

Most experience with autoconf/automake is all negative. The promise was "never debug your Makefile ever 
again!", delivered was "debug the configure script instead!". In practice, with autoconf/automake, you try to run 
configure, kludge it until it stops crashing, then tweak the incomprehensible Makefiles it produces until the code 
compiles.

K.O.
  458   10 Mar 2008 Stefan RittSuggestionNew Makefile for building MIDAS
> I rewrote the Makefile for MIDAS in order to make it tidy. I tested it on my box
> and it works here.
> 1. The full file is seperated to several parts
>   a. initialized setup
>   b. environment setup
>   c. specify OS-specific flags
>   d. processing environment for building flags
>   e. targets
> 2. The file is less than 400 lines now. The original one is more than 500 lines.
> 3. The modified one is easy for debuging.
> 
> I tried to learn "autoconf" and "automake" in order to make building MIDAS more
> compatible for various platforms. But I havn't enough time now. Hope somebody
> can help it. The attached file is original named "Makefile.in" for using "autoconf".

The Makefile is missing -lzip:

[ritt@pc5082 ~/midas]$ make -f Makefile-by-EL
Making directory linux ...
Making directory linux/lib ...
Making directory linux/bin ...
g++ -g -O3 -Wall -Wuninitialized -DINCLUDE_FTPLIB -D_LARGEFILE64_SOURCE -DOS_LINUX -fPIC
-Wno-unused-function   -DHAVE_ZLIB -Iinclude -Idrivers -I../mxml -Llinux/lib -o
linux/bin/mlogger src/mlogger.c linux/lib/libmidas.a -lutil -lpthread
/tmp/cceHnAKe.o(.text+0x83c): In function `midas_flush_buffer(LOG_CHN*)':
src/mlogger.c:984: undefined reference to `gzwrite'
/tmp/cceHnAKe.o(.text+0xb24): In function `midas_log_open(LOG_CHN*, int)':
src/mlogger.c:1132: undefined reference to `gzopen'
/tmp/cceHnAKe.o(.text+0xb46):src/mlogger.c:1140: undefined reference to `gzsetparams'
/tmp/cceHnAKe.o(.text+0xe2a): In function `midas_log_close(LOG_CHN*, int)':
src/mlogger.c:1208: undefined reference to `gzflush'
/tmp/cceHnAKe.o(.text+0xe40):src/mlogger.c:1210: undefined reference to `gzclose'
collect2: ld returned 1 exit status
make: *** [linux/bin/mlogger] Error 1
[ritt@pc5082 ~/midas]$
  457   10 Mar 2008 Stefan RittSuggestionRFC- ACLs for midas rpc, mserver, mhttpd access
> When accepting connection from a remote host, the remote IP address is converted to a
> hostname using gethostbyaddr(). If ODB directory "/experiment/security/mhttpd hosts",
> exists, access is permitted if there is an entry for the this hostname. "localhost" is
> always permitted.

While your "positive list" will certainly work, it is much more inflexible than a more
general hosts.allow/hosts.deny with wildcards. Assume some experiment decides it wants to
be controlled from all inside CERN. With hosts.allow/deny you could do

host.deny *
host.allow *.cern.ch

to have everything ending with "cern.ch" allowed. Otherwise it would be a nightmare finding
all possible terminals at CERN and add them manually. If you are considering modifying your
committed code to this scheme, you could have a look at my elog package, where exactly this
is implemented. You could copy/paste it from there.

After you finished, also talk to Pierre about documenting this in doxygen (or do it yourself).
  456   09 Mar 2008 Stefan RittSuggestionNew Makefile for building MIDAS
> I rewrote the Makefile for MIDAS in order to make it tidy. I tested it on my box
> and it works here.
> 1. The full file is seperated to several parts
>   a. initialized setup
>   b. environment setup
>   c. specify OS-specific flags
>   d. processing environment for building flags
>   e. targets
> 2. The file is less than 400 lines now. The original one is more than 500 lines.
> 3. The modified one is easy for debuging.
> 
> I tried to learn "autoconf" and "automake" in order to make building MIDAS more
> compatible for various platforms. But I havn't enough time now. Hope somebody
> can help it. The attached file is original named "Makefile.in" for using "autoconf".

I think it is a good idea to cleanup the Makefile. It grew over many years and certainly
had some inconsistencies. We did however not use "autoconf" since it is not of much use.
It is meant for systems where small differences between different Unix flavors are
covered by this system, but the midas source code is supposed not only to run on Unix,
but also on vxWorks and Windows. As you can imagine, the differences are much more
severe and a simple makefile generator cannot cover the details. Furthermore, under
Windows there is no such thing like autoconf. So all the work to make the source code
compile on all systems has been put into system.c using conditional compiling. So
putting another abstraction layer on this would maybe more complicate things than
simplify it. I will test your Makefile, and I also ask the guys at TRIUMF to do so. Once
we conclude that it works fine, we can replace the original Makefile from the distribution.
  455   09 Mar 2008 Exaos LeeSuggestionNew Makefile for building MIDAS
I rewrote the Makefile for MIDAS in order to make it tidy. I tested it on my box
and it works here.
1. The full file is seperated to several parts
  a. initialized setup
  b. environment setup
  c. specify OS-specific flags
  d. processing environment for building flags
  e. targets
2. The file is less than 400 lines now. The original one is more than 500 lines.
3. The modified one is easy for debuging.

I tried to learn "autoconf" and "automake" in order to make building MIDAS more
compatible for various platforms. But I havn't enough time now. Hope somebody
can help it. The attached file is original named "Makefile.in" for using "autoconf".

:-)
Attachment 1: Makefile-by-EL
# initialized flags

#CC =
#FC = 
#LD =
CFLAGS   = -g -O3 -Wall -Wuninitialized -DINCLUDE_FTPLIB

###########################################################################
## Environments setup

## Prefix
ifndef PREFIX
PREFIX = /usr/local
endif

# User defined flags for compiler
USERFLAGS     =

# Do you need to compile the shared library? If so, set "1" here.
NEED_SHLIB    =

# If RPATH needed ...
NEED_RPATH    =

# If static ROOT libaries needed ...
NEED_LIBROOTA =

# If strlcpy.o needed ...
NEED_STRLCPY  = 1

# If zlib needed ...
NEED_ZLIB     =

# MIDAS max event size
MIDAS_MAX_EVENT_SIZE =

# OS specific programs included
SPECIFIC_OS_PRG      =

# ..? Why it is needed?
MIDAS_PREF_FLAGS     =

## installer

INSTALL  = $(shell which install)
ifndef INSTALL
INSTALL  = ./install.sh
else
INS_OPTS = -v -D
endif

# MXML dir
ifndef MXML_DIR
MXML_DIR = ../mxml
endif

# MySQL
MYSQL_CONFIG := $(shell which mysql_config 2> /dev/null)
ifdef MYSQL_CONFIG
MYSQLINCDIR := $(shell mysql_config --cflags | sed -e 's,^.*-I\([^ ]*\).*$$,\1,' -e s/\'//g)
NEED_MYSQL  := $(shell if test -e $(MYSQLINCDIR)/mysql.h ; then echo 1; fi) 
endif

###########################################################################
## Check OS Type for OS-specified flags

OSTYPE = $(shell uname | tr '[A-Z]' '[a-z]')

ifeq ($(OSTYPE), linux)
# > 2GB file support
CFLAGS  += -D_LARGEFILE64_SOURCE
CFLAGS  += -DOS_LINUX -fPIC -Wno-unused-function
LDFLAGS += -lutil -lpthread
NEED_ZLIB        =1
SPECIFIC_OS_PRG += mlxspeaker dio
endif

ifeq ($(OSTYPE), osf1)
CFLAGS  += -DOS_OSF1
FFLAGS  += -nofor_main -D 40000000 -T 20000000
LDFLAGS += -lc -lbsd
endif

ifeq ($(OSTYPE), ultrix)
CFLAGS  += -DOS_ULTRIX -DNO_PTY
endif

ifeq ($(OSTYPE), freebsd)
CFLAGS  += -DOS_FREEBSD
LDFLAGS += -lbsd -lcompat
endif

ifeq ($(OSTYPE), darwin)
CFLAGS  += -DOS_LINUX -DOS_DARWIN -DHAVE_STRLCPY -fPIC -Wno-unused-function
LDFLAGS += -lpthread
NEED_RANLIB      = 1
SPECIFIC_OS_PRG += mlxspeaker
endif

ifeq ($(OSTYPE), cygwin_nt-5.1)
OSTYPE = cygwin
endif

ifeq ($(OSTYPE), cygwin)
CFLAGS  += -DOS_LINUX -DOS_CYGWIN -Wno-unused-function
LDFLAGS += -lutil -lpthread
endif

ifeq ($(OSTYPE), solaris)
CFLAGS  += -DOS_SOLARIS
LDFLAGS += -lsocket -lnsl
endif

###########################################################################
## Processing environment for building flags

CFLAGS += $(MIDAS_PREF_FLAGS) $(USERFLAGS)

ifdef MIDAS_MAX_EVENT_SIZE
CFLAGS += -DMAX_EVENT_SIZE=$(MIDAS_MAX_EVENT_SIZE)
endif # MIDAS_MAX_EVENT_SIZE

ifdef NEED_MYSQL
CFLAGS  += -DHAVE_MYSQL $(shell $(MYSQL_CONFIG) --include)
LDFLAGS += $(shell $(MYSQL_CONFIG) --libs)
NEED_ZLIB =1
endif # NEED_MYSQL

ifdef ROOTSYS
ROOTLIBS    := $(shell $(ROOTSYS)/bin/root-config --libs)
ROOTCFLAGS  := $(shell $(ROOTSYS)/bin/root-config --cflags)

ifdef NEED_RPATH
ROOTLIBS   += -Wl,-rpath,$(ROOTSYS)/lib
endif

ifdef NEED_LIBROOTA
ROOTLIBS    := $(ROOTSYS)/lib/libRoot.a -lssl -ldl -lcrypt
endif

CFLAGS += $(ROOTCFLAGS)
endif # ROOTSYS

ifdef NEED_ZLIB
CFLAGS     += -DHAVE_ZLIB
LIBS       += -lz
endif # ZLIB

ifdef NEED_SHLIB
CFLAGS  += -shared -fPIC
endif # SHLIB

###########################################################################
## Processing building targets

## Directories needed

INC_DIR = include
SRC_DIR = src
UTL_DIR = utils
DRV_DIR = drivers
EXM_DIR = examples

## Driectories which will be created

OS_DIR  = $(OSTYPE)
LIB_DIR = $(OS_DIR)/lib
BIN_DIR = $(OS_DIR)/bin
DIRS    = $(OS_DIR)  $(LIB_DIR)  $(BIN_DIR)

CFLAGS += -I$(INC_DIR) -I$(DRV_DIR) -I$(MXML_DIR) -L$(LIB_DIR)

## Directories for install

INSTDIR_BIN = $(PREFIX)/bin
INSTDIR_INC = $(PREFIX)/include
INSTDIR_LIB = $(PREFIX)/lib
INSTDIR_DRV = $(PREFIX)/driver

## Utilities

PROGS = mserver  mhttpd  mlogger  odbedit  mtape   mhist    \
	mstat    mcnaf   mdump    mhdump    mchart  odbhist  \
	melog    webpaw  lazylogger   stripchart.tcl  \
	$(SPECIFIC_OS_PRG)

UTILS = $(shell for i in $(PROGS); do echo $(BIN_DIR)/$$i ; done)

## Analyzer objects

ANALYZER  = $(LIB_DIR)/mana.o
ifdef CERNLIB
ANALYZER += $(LIB_DIR)/hmana.o
endif
ifdef ROOTSYS
ANALYZER += $(LIB_DIR)/rmana.o
endif

## Objects for building $(LIBMIDAS_A) and $(LIBMIDAS_SO)

OBJS_LIB_NEED = midas.o  system.o  mrpc.o  odb.o  ybos.o  ftplib.o  history.o  alarm.o  elog.o

ifndef NEED_SHLIB
OBJS_LIB_NEED += cnaf_callback.o
endif

OBJS_LIB = $(shell for i in $(OBJS_LIB_NEED) ; do echo $(LIB_DIR)/$$i ; done)

## MXML OBJS

MXML_OBJS_NEED = mxml.o
ifdef NEED_STRLCPY
MXML_OBJS_NEED += strlcpy.o
endif

MXML_OBJS = $(shell for i in $(MXML_OBJS_NEED) ; do echo $(LIB_DIR)/$$i ; done)

## Objects for install

OBJS_INS_NEED = mfe.o fal.o
OBJS_INS = $(shell for i in $(OBJS_INS_NEED) ; do echo $(LIB_DIR)/$$i ; done)

## library

LIBMIDAS_A  = $(LIB_DIR)/libmidas.a
ifdef NEED_SHLIB
LIBMIDAS_SO  = $(LIB_DIR)/libmidas.so
LDFLAGS     += -shared -L$(LIB_DIR) -lmidas
endif

## Examples

EXAMPLES_NEED = consume produce rpc_test msgdump minife minirc odb_test
EXAMPLES = $(shell for i in $(EXAMPLES_NEED); do echo examples/$$i ; done)

## Includes

INCLUDES = $(wildcard $(INC_DIR)/*.h)

###########################################################################

CXXFLAGS  = $(CFLAGS)
ifdef ROOTSYS
CXXFLAGS += -DHAVE_ROOT
endif

all: check-mxml mkdirs $(LIBMIDAS_A) $(LIBMIDAS_SO) $(ANALYZER) $(OBJS_INS) utilities

.PHONY: clean

###########################################################################

## Library objects

$(LIB_DIR)/%.o: $(SRC_DIR)/%.c $(INCLUDES)
	$(CC) $(CFLAGS) -c -o $@ $<

## MXML_OBJS

$(LIB_DIR)/mxml.o: $(MXML_DIR)/mxml.c
	$(CC) $(CFLAGS) -c -o $@ $<

$(LIB_DIR)/strlcpy.o: $(MXML_DIR)/strlcpy.c
	$(CC) $(CFLAGS) -c -o $@ $<

## Frontend and backend framework

$(LIB_DIR)/fal.o: $(SRC_DIR)/fal.c $(INCLUDES)
	$(CXX) -Dextname -DMANA_LITE -c -o $@ $< $(CFLAGS)

$(LIB_DIR)/hmana.o: $(SRC_DIR)/mana.c $(INCLUDES)
	$(CC) -Dextname -DHAVE_HBOOK $(CFLAGS) -c -o $@ $<

$(LIB_DIR)/rmana.o: $(SRC_DIR)/mana.c $(INCLUDES)
	$(CXX) -DUSE_ROOT $(CXXFLAGS) -c -o $@ $<

## Utilities

$(BIN_DIR)/%: $(UTL_DIR)/%.c $(INCLUDES)
	$(CC) $(CFLAGS) -o $@ $< $(LIBMIDAS_A) $(LDFLAGS)

$(BIN_DIR)/mlogger: $(SRC_DIR)/mlogger.c $(LIBMIDAS_A)
	$(CXX) $(CXXFLAGS) -o $@ $^ $(LDFLAGS) $(ROOTLIBS)
$(BIN_DIR)/mserver: $(SRC_DIR)/mserver.c $(INCLUDES)
$(BIN_DIR)/mhttpd:  $(SRC_DIR)/mhttpd.c  $(SRC_DIR)/mgd.c
$(BIN_DIR)/odbedit: $(SRC_DIR)/odbedit.c $(SRC_DIR)/cmdedit.c
$(BIN_DIR)/mcnaf:   $(UTL_DIR)/mcnaf.c   $(DRV_DIR)/camac/camacrpc.c
$(BIN_DIR)/mhdump:  $(UTL_DIR)/mhdump.cxx
	$(CXX) $(CXXFLAGS) -o $@ $< $(LDFLAGS) $(ROOTLIBS)
$(BIN_DIR)/lazylogger: $(SRC_DIR)/lazylogger.c
$(BIN_DIR)/stripchart.tcl: $(UTL_DIR)/stripchart.tcl
	@cp -f $< $(BIN_DIR)/$(shell basename $@)

###########################################################################

mkdirs:
	@for i in $(DIRS) ; do \
		echo "Making directory $$i ..." ; \
		mkdir -p $$i ; \
	done
... 108 more lines ...
  454   07 Mar 2008 Konstantin OlchanskiSuggestionRFC- ACLs for midas rpc, mserver, mhttpd access
The mhttpd host-based access control list as used by ALPHA at CERN is now committed to
SVN (revision 4135).

When accepting connection from a remote host, the remote IP address is converted to a
hostname using gethostbyaddr(). If ODB directory "/experiment/security/mhttpd hosts",
exists, access is permitted if there is an entry for the this hostname. "localhost" is
always permitted.

In other words:

1) To enable the mhttpd access control list, create an ODB directory
"/experiment/security/mhttpd hosts".

2) From this moment, only access from "localhost" is permitted.

3) All connections from remote hosts are rejected with an error written into the midas
log file: Rejecting http connection from 'ladd05.triumf.ca'.

4) To permit access from remote hosts, take the hostname from this error message and
create an entry in "mhttpd hosts": odbedit -> cd "/Experiment/Security/mhttpd hosts" ->
create INT ladd05.triumf.ca

The idea behind this is that mhttpd is running behind an SSL proxy (or an SSH tunnel)
and only accepts connections from this proxy and perhaps from selected machines in the
experiment counting room.

P.S. I considered using tcp_wrappers, but this package does not seem to contain any
simple-to-use function "bool areTheyPermitted(const char* remoteHostname);".

P.P.S. The ODB path name is in variance from Stefan's email. I committed this code
before rereading it, please let me know if I should change the ODB paths.

P.P.P.S. I will now proceed with implementing similar code for the mserver/midas rpc.
Again, the use case is very simple: all machines permitted access to the mserver are
known in advance and can be listed in the access list. All unknown machines should be
rejected.

K.O.
  453   07 Mar 2008 Stefan RittBug Reportarray overflows and other bugs
> I have just compiled MIDAS svn 4132 on a fresh SuSE 10.3 x86_64 system and gcc 
> found a bunch of bugs, I guess.

Ahh, great! gcc is getting more and more clever. Each time gcc is updated, it finds
a few new issues.

Indeed some are real bugs, and I will work down the list as time permits. I see
however no immediate thread (you are not using fragmented events, a transition 12
never occurs, etc.). Issue #4 from your list has to be checked by Pierre-Andre. 
ELOG V3.1.4-2e1708b5