I changed out_info into a global structure definition ANA_OUTPUT_INFO and put it into
midas.h, so it can be accessed easily from the user analyzer source code.

> Would it be relevant to transform out_info into a *non-static* variable of a type
> defined by a *named* struct?
> Currently,  programs that  try to access out_info cannot do it anymore; and they
> typically copy the struct definition from mana.c, which is not robust against future
> changes in mana.c.
> 
> If mana.c could be changed in the way described above, that would be great . 
> Otherwise, is it safe to patch it myself for local use?  or is there a better way of
> accessing out_info from mana.c?
> 
> As always, any help would be much appreciated :)
> 
> EOL
> 
> > Hello,
> > 
> > Is it possible to access out_info (defined in mana.c) from another program?
> > 
> > In fact, out_info is now defined as an (anonymous) "static struct" in mana.c,
> > which it seems to me precludes any direct use in another program.  Is there an
> > indirect way of getting ahold of out_info?  or of the information it contains?
> > 
> > out_info used to be defined as a *non-static* struct, and the code I'm currently
> > modifying used to compile seamlessly: it now stops the compilation during
> > linking time, as out_info is now static and the program I have to compile
> > contains an "extern struct {} out_info".
> > 
> > Any help would be much appreciated!  I searched in vain in this forum for
> > details about out_info and I really need to access the information it contains!
> > 
> > EOL (a pure MIDAS novice)

Running MIDAS at CERN is proving more challenging than I expected. The network environement is not 
as benign as I am used to (i.e. at TRIUMF) and our machines are being constantly probed by something/
somebody.

This already caused failures in the mserver (fixed in midas svn) and I would like to resolve this problem 
once and for all. The age of "nice networks" is over.

The case of the mserver and for the midas rpc servers (every midas applications listens for midas rpc 
requests, i.e. run transitions) is simple. The list of machines running midas applications is known ahead 
of time, so we can put them all into a list of permitted machines and deny rpc connections to anybody 
else. I propose we keep this list of permitted mserver clients in "/experiment/security/mserver hosts".

(The already existing "/experiment/security/allowed hosts" mechanism is insufficient: it does not 
prevent the mserver from accepting connections from hostile machines, and talking to them, for 
example giving them the list of available experiments. There is a fair amount of code involved and I do 
not presume to certify any of it as hack-proof or even as crash-proof.)

For mhttpd http:// access control, I thought of using tcp_wrappers, but C-API documentation does not 
exist (I looked), the example code in tcpd.c is way too complicated, editing the ACL /etc/hosts.allow 
unnecessarily requires root privileges and non of it would work on Windows.

So I am favouring a home-made hostname or ip-address filter, similar to /etc/hosts.allow, with ACL 
stored, for example, in "/experiment/security/mhttpd hosts".

Any thoughts?

K.O.

I am in favor of tcp_wrappers.

tcp_wrappers is well understood.

It works well in combination with a firewall.

mhttpd hangs when our security folks scan us.  We are not allowed to block them
with a firewall, but we can use tcpwrappers.

Would it make sense to put the same mechanism on mserver?

the man page for libtcpwrappers.a (taken from the tcpwrappers7.6 tar ball) is
attached. And the output after running it through nroff -man.

The odb is too fragile for security.  It is not understood well enough by many
experimenters.

As you can see I am in favor of tcp_wrappers.  This is mainly because it is part
of an existing and tested security model.  I don't know about the windows
world, but as you can also see, I vote for using something that is already part
of the windows security model.  Here's an example of how well the integrated
security model works:

    if an person is part of an experiment I make sure they can ssh to the
    experiment's computer

    the same rules could provide them with web access

Second is that when a change is needed to the security model then it is easy to
keep it current.  What if somebody restores an old ODB?  What if they setup a
small test with a new ODB?

If mhttpd used tcp_wrappers, then all our machines here at LANL would already be
configured!  No need for
users to do any root access (though those that need it have it anyway).

John.

First I have a general question: mserver is started through xinetd, and xinetd has
the options "only_from" and "no_access". This is equivalent to the tcp_wrapper
functionality. Why not using this? It's possible without changing anything in midas.
Or am I missing anything?

If that does not work for some reason, here are some thought from my side:

- We don't have much of a problem with malicious hackers, but with institute-wide
security checking. Hackers are only interested in mechanisms where they can obtain
control over thousands of machines (like breaking ssh etc.). The few midas machines
are not a good target for them. But even at PSI there are security scans, which try
to connect to various ports and can crash systems, so I agree that something needs
to be done.

- Whatever we do, it should be consistent on linux and windows and should not rely
on external packages, since I don't want to get into dependencies there.

- I see that both having the security information in the ODB or having them in
external files can be advantageous. There is certainly the aspect of restoring old
ODBs, or keeping several experiments (ODB) on one machine consistent. On the other
hand storing data in the ODB might me liked by people who are familiar with this
concept, and want to change things though mhttpd for example.

- Having said all that, it would make sense to me to write a simple central routine
access_allowed(), which takes the IP address of a remote client wanting to connect,
and return true or false. This routine should read /etc/hosts.allow, /etc/hosts.deny
and interprete it, but only the section for midas, and maybe only a subset of the
functionality there (we probably don't need NIS netgroup names, external files and
spawn commands there). If the files /etc/hosts.x do not contain anything about midas
or are not preset (Windows!), the routine should look in the ODB under
/experiment/security/mserver/hosts.allow and /experiment/security/mserver/hosts.deny
and use that information instead of the files.

- We probably need different mechanisms for mserver and for mhttpd. The mserver
clients are usually only a few programs like the front-ends, while one may want to
control an experiment over mhttpd from much more machines. So we should establish a
second ACL for mhttpd. The already present "/experiment/security/allowed hosts" for
mhttpd should be converted into "/Experiment/Security/mhttpd/hosts.allow" and the
function access_allowed() should be used to interprete that, so that we only need to
write it once.

#!/bin/sh

[ ! "$MIDASSYS" ] && MIDASSYS=/opt/MIDAS.PSI/Version/Current
[ ! "$HTTPPORT" ] && HTTPPORT=8080
[ ! "$SRVHOST" ]  && SRVHOST=localhost
LOGGER=${MIDASSYS}/bin/mlogger

EXPPATH=/home/das/online/test
CODEPATH=${EXPPATH}/code
LOGGER=${MIDASSYS}/bin/mlogger

PROG_FE=${CODEPATH}/frontend
PROG_ANA=${CODEPATH}/analyzer

if [ ! "$MIDAS_EXPTAB" ]; then
	MIDAS_DIR=${EXPPATH}
else
	MIDAS_EXPT_NAME="test"
fi

The mhttpd host-based access control list as used by ALPHA at CERN is now committed to
SVN (revision 4135).

When accepting connection from a remote host, the remote IP address is converted to a
hostname using gethostbyaddr(). If ODB directory "/experiment/security/mhttpd hosts",
exists, access is permitted if there is an entry for the this hostname. "localhost" is
always permitted.

In other words:

1) To enable the mhttpd access control list, create an ODB directory
"/experiment/security/mhttpd hosts".

2) From this moment, only access from "localhost" is permitted.

3) All connections from remote hosts are rejected with an error written into the midas
log file: Rejecting http connection from 'ladd05.triumf.ca'.

4) To permit access from remote hosts, take the hostname from this error message and
create an entry in "mhttpd hosts": odbedit -> cd "/Experiment/Security/mhttpd hosts" ->
create INT ladd05.triumf.ca

The idea behind this is that mhttpd is running behind an SSL proxy (or an SSH tunnel)
and only accepts connections from this proxy and perhaps from selected machines in the
experiment counting room.

P.S. I considered using tcp_wrappers, but this package does not seem to contain any
simple-to-use function "bool areTheyPermitted(const char* remoteHostname);".

P.P.S. The ODB path name is in variance from Stefan's email. I committed this code
before rereading it, please let me know if I should change the ODB paths.

P.P.P.S. I will now proceed with implementing similar code for the mserver/midas rpc.
Again, the use case is very simple: all machines permitted access to the mserver are
known in advance and can be listed in the access list. All unknown machines should be
rejected.

K.O.

I rewrote the Makefile for MIDAS in order to make it tidy. I tested it on my box
and it works here.
1. The full file is seperated to several parts
  a. initialized setup
  b. environment setup
  c. specify OS-specific flags
  d. processing environment for building flags
  e. targets
2. The file is less than 400 lines now. The original one is more than 500 lines.
3. The modified one is easy for debuging.

I tried to learn "autoconf" and "automake" in order to make building MIDAS more
compatible for various platforms. But I havn't enough time now. Hope somebody
can help it. The attached file is original named "Makefile.in" for using "autoconf".

:-)

> I rewrote the Makefile for MIDAS in order to make it tidy. I tested it on my box
> and it works here.
> 1. The full file is seperated to several parts
>   a. initialized setup
>   b. environment setup
>   c. specify OS-specific flags
>   d. processing environment for building flags
>   e. targets
> 2. The file is less than 400 lines now. The original one is more than 500 lines.
> 3. The modified one is easy for debuging.
> 
> I tried to learn "autoconf" and "automake" in order to make building MIDAS more
> compatible for various platforms. But I havn't enough time now. Hope somebody
> can help it. The attached file is original named "Makefile.in" for using "autoconf".

I think it is a good idea to cleanup the Makefile. It grew over many years and certainly
had some inconsistencies. We did however not use "autoconf" since it is not of much use.
It is meant for systems where small differences between different Unix flavors are
covered by this system, but the midas source code is supposed not only to run on Unix,
but also on vxWorks and Windows. As you can imagine, the differences are much more
severe and a simple makefile generator cannot cover the details. Furthermore, under
Windows there is no such thing like autoconf. So all the work to make the source code
compile on all systems has been put into system.c using conditional compiling. So
putting another abstraction layer on this would maybe more complicate things than
simplify it. I will test your Makefile, and I also ask the guys at TRIUMF to do so. Once
we conclude that it works fine, we can replace the original Makefile from the distribution.

> When accepting connection from a remote host, the remote IP address is converted to a
> hostname using gethostbyaddr(). If ODB directory "/experiment/security/mhttpd hosts",
> exists, access is permitted if there is an entry for the this hostname. "localhost" is
> always permitted.

While your "positive list" will certainly work, it is much more inflexible than a more
general hosts.allow/hosts.deny with wildcards. Assume some experiment decides it wants to
be controlled from all inside CERN. With hosts.allow/deny you could do

host.deny *
host.allow *.cern.ch

to have everything ending with "cern.ch" allowed. Otherwise it would be a nightmare finding
all possible terminals at CERN and add them manually. If you are considering modifying your
committed code to this scheme, you could have a look at my elog package, where exactly this
is implemented. You could copy/paste it from there.

After you finished, also talk to Pierre about documenting this in doxygen (or do it yourself).

> I rewrote the Makefile for MIDAS in order to make it tidy. I tested it on my box
> and it works here.
> 1. The full file is seperated to several parts
>   a. initialized setup
>   b. environment setup
>   c. specify OS-specific flags
>   d. processing environment for building flags
>   e. targets
> 2. The file is less than 400 lines now. The original one is more than 500 lines.
> 3. The modified one is easy for debuging.
> 
> I tried to learn "autoconf" and "automake" in order to make building MIDAS more
> compatible for various platforms. But I havn't enough time now. Hope somebody
> can help it. The attached file is original named "Makefile.in" for using "autoconf".

The Makefile is missing -lzip:

[ritt@pc5082 ~/midas]$ make -f Makefile-by-EL
Making directory linux ...
Making directory linux/lib ...
Making directory linux/bin ...
g++ -g -O3 -Wall -Wuninitialized -DINCLUDE_FTPLIB -D_LARGEFILE64_SOURCE -DOS_LINUX -fPIC
-Wno-unused-function   -DHAVE_ZLIB -Iinclude -Idrivers -I../mxml -Llinux/lib -o
linux/bin/mlogger src/mlogger.c linux/lib/libmidas.a -lutil -lpthread
/tmp/cceHnAKe.o(.text+0x83c): In function `midas_flush_buffer(LOG_CHN*)':
src/mlogger.c:984: undefined reference to `gzwrite'
/tmp/cceHnAKe.o(.text+0xb24): In function `midas_log_open(LOG_CHN*, int)':
src/mlogger.c:1132: undefined reference to `gzopen'
/tmp/cceHnAKe.o(.text+0xb46):src/mlogger.c:1140: undefined reference to `gzsetparams'
/tmp/cceHnAKe.o(.text+0xe2a): In function `midas_log_close(LOG_CHN*, int)':
src/mlogger.c:1208: undefined reference to `gzflush'
/tmp/cceHnAKe.o(.text+0xe40):src/mlogger.c:1210: undefined reference to `gzclose'
collect2: ld returned 1 exit status
make: *** [linux/bin/mlogger] Error 1
[ritt@pc5082 ~/midas]$

> I rewrote the Makefile for MIDAS in order to make it tidy.

Not that the current Makefile is too pretty (I have seen worse), but it works and it is fairly compact for a project of 
this complexity, it handles a large number of operating systems and build options very efficiently.

I think you found that out with your rewriting exercise - your version of the Makefile contains all the same code, 
just rearranged to suite your taste, with existing bugs preserved and new bugs added.

> I tested it on my box and it works here.

As they say, the devil is in the details. I notice some subtle changes in your Makefile that make me go "what?":

1) the command for building the midas shared library used to be "ld -shared", in your version, "-shared" is gone. 
But check with the GCC manual, today's recommended command is probably "gcc -shared".
2) mhdump is now linked with ROOT, but I wrote it recently enough to remember that it does not use ROOT
3) hand-crafted dependancies have been replaced with generic "almost every .o depends on every .h", which is 
incorrect. The "almost every .o" part bothers me.
4) "make clean" runs "rm -rf" - plain scary.
5) "$(shell ...)" is overused

I think by the end all these little details are sorted out and all the quirks are put back in, your Makefile will look no 
better than the current Makefile.

> 2. The file is less than 400 lines now. The original one is more than 500 lines.

It looks like your savings came from removing comments, removing hand-crafted dependancy lists and replacing 
fairly verbose "make install" targets (which we do not use anyway) with your own much simpler scripts.

All the juicy bits needed to actually build all the code appear to take about as much space as before.

Also the original mistake of recompiling programs when they only need relinking was not fixed. (For example, 
when libmidas is updated, to update mhttpd, the current Makefile needlessly recompiles mhttpd.c. Better use 
would be to compile mhttpd.c into mhttpd.o, then only a relink is needed).

> I tried to learn "autoconf" and "automake" in order to make building MIDAS more
> compatible for various platforms. But I havn't enough time now. Hope somebody
> can help it. The attached file is original named "Makefile.in" for using "autoconf".

Most experience with autoconf/automake is all negative. The promise was "never debug your Makefile ever 
again!", delivered was "debug the configure script instead!". In practice, with autoconf/automake, you try to run 
configure, kludge it until it stops crashing, then tweak the incomprehensible Makefiles it produces until the code 
compiles.

K.O.

> While your "positive list" will certainly work, it is much more inflexible than a more
> general hosts.allow/hosts.deny with wildcards. Assume some experiment decides it wants to
> be controlled from all inside CERN. With hosts.allow/deny you could do

I was going to bring this up later, but since mhttpd does not pass security audits, I believe
the only way it should be run in the modern computing environement is behind
a password-protected SSL proxy. In this case, the allow/deny list is very simple: deny all,
allow localhost (assuming httpd runs on the same host as mhttpd).

Speaking about CERN, "deny all; allow *.cern.ch" is the "default" setting, enforced by the CERN firewall. Our problem is with 
random "*.cern.ch" computers poking at our DAQ and crashing the mserver. Plus we do not want our competition to access our 
DAQ system, so "allow *.cern.ch" is a no go.

But since hosts.allow/hosts.deny is a superset of what I want, and since we can reuse existing code from elogd, I guess I have 
no ground to object your suggestion.

I will do the mserver/mrpc this way, then retrofit it into mhttpd. (But have to commit mlogger history changes first!!!).

K.O.

> I was going to bring this up later, but since mhttpd does not pass security audits, I believe
> the only way it should be run in the modern computing environement is behind
> a password-protected SSL proxy.

I recently built in native SSL into elogd.c and found it was very simple. We could do the same for mhttpd.

> Speaking about CERN, "deny all; allow *.cern.ch" is the "default" setting, enforced by the CERN firewall. Our problem is with 
> random "*.cern.ch" computers poking at our DAQ and crashing the mserver. Plus we do not want our competition to access our 
> DAQ system, so "allow *.cern.ch" is a no go.

I understand your point. But I want to tell you that there are other experiments, which want domain based access. For example at
PSI some experiments want allowed access from the experimental hall, which is the subdomain 129.129.140.* (there is not so much
competition here ;-) but not from other PSI subdomains. So you would need "deny all; allow 129.129.140.*; allow 129.129.228.*" for
example.

> I will do the mserver/mrpc this way, then retrofit it into mhttpd. (But have to commit mlogger history changes first!!!).

Agree.

> Most experience with autoconf/automake is all negative. The promise was "never debug your Makefile ever 
> again!", delivered was "debug the configure script instead!". In practice, with autoconf/automake, you try to run 
> configure, kludge it until it stops crashing, then tweak the incomprehensible Makefiles it produces until the code 
> compiles.
> 
> K.O.

I admit that the new one is fit to my flavor. For a common user, I think, a simple procedure of configure/make/install
is better than changing the Makefile manually because many users are lack of knowledges about Makefile. That's why 
I want to learn autotools. The configure script is generated automatically by "autoconf", so you needn't to debug it. 
For the developer, you need to debug the configure.ac/in files for generating the configure script. For a common user, 
he/she only needs to run it. In fact, some more complex projects like ROOT use AUTOTOOLS and they don't include 
the original files which are needed for generating the "configure" script.  I prefer the MIDAS project includes such a 
script to make the compiling simpler and easier instead of changing the Makefile manually. 

> The Makefile is missing -lzip:

Sorry, spelling error.
The "LIBS +=" should be replaced by "LDFLAGS +="

> Not that the current Makefile is too pretty (I have seen worse), but it 
works and it is fairly compact for a project of 
> this complexity, it handles a large number of operating systems and build 
options very efficiently.
> 
> I think you found that out with your rewriting exercise - your version of 
the Makefile contains all the same code, 
> just rearranged to suite your taste, with existing bugs preserved and new 
bugs added.

I derived the new Makefile from the original one so that feathers and bugs are 
also included. 
I havn't experiences on platforms other than Linux and MacOS, so I cannot 
recognize bugs on 
other platforms if they exists in the original one. And if there are bugs, 
hope users can figure
them out.

>
> As they say, the devil is in the details. I notice some subtle changes in 
your Makefile that make me go "what?":
> 
> 1) the command for building the midas shared library used to be "ld -
shared", in your version, "-shared" is gone. 
> But check with the GCC manual, today's recommended command is probably "gcc -
shared".

Fixed.

> 2) mhdump is now linked with ROOT, but I wrote it recently enough to 
remember that it does not use ROOT

The building dependence on ROOT of mhdump may be eliminated by changing the 
specific target.

> 3) hand-crafted dependancies have been replaced with generic "almost 
every .o depends on every .h", which is 
> incorrect. The "almost every .o" part bothers me.

Fixed now.

> 4) "make clean" runs "rm -rf" - plain scary.

Fixed.

> 5) "$(shell ...)" is overused

Replaced with GNU make internal methods.


> I think by the end all these little details are sorted out and all the 
quirks are put back in, your Makefile will look no 
> better than the current Makefile.

I realized it now. But anyway, it looks tidy to me now. I still hope to use 
AUTOTOOLS with MIDAS.
 
> > 2. The file is less than 400 lines now. The original one is more than 500 
lines.

The new one is about 430 lines. Hmmm, it reaches the original one which is 
more than 600 lines.

> 
> It looks like your savings came from removing comments, removing hand-
crafted dependancy lists and replacing 
> fairly verbose "make install" targets (which we do not use anyway) with your 
own much simpler scripts.
> 
> All the juicy bits needed to actually build all the code appear to take 
about as much space as before.
> 
> Also the original mistake of recompiling programs when they only need 
relinking was not fixed. (For example, 
> when libmidas is updated, to update mhttpd, the current Makefile needlessly 
recompiles mhttpd.c. Better use 
> would be to compile mhttpd.c into mhttpd.o, then only a relink is needed).

Fixed.

> 
> Most experience with autoconf/automake is all negative. The promise 
was "never debug your Makefile ever 
> again!", delivered was "debug the configure script instead!". In practice, 
with autoconf/automake, you try to run 
> configure, kludge it until it stops crashing, then tweak the 
incomprehensible Makefiles it produces until the code 
> compiles.
> 
> K.O.

====================================================
Maybe BUGS or FEATURES in this new one:
1. The shared libmidas.so and the static libmidas.a are built sperately. 
The "libmidas.a" is 
always built whether "NEED_SHLIB" is set or not. And all executables are built 
staticly default. 
I commented this in the Makefile-by-EL. Hence, if you want to use libmidas.so 
with PyMIDAS and
do not want to encounter "Segmentation fault" while executing the utilies 
linked dynamicly, you
may try this one.
2. I found that "minife" is failed to be built, so I remove it from the 
example list.
3. Some bugs while building on MacOS Tiger 10.4.11 PPC are commented out in 
the Makefile. These
bugs are still exists in the original one.
4. Using "VPATH" instead of adding pathnames.
5. Using "UTILS_SUID" to handle utilities which need SUID mode. And 
the "UTILS_SUID_NEED" may be
defined in the OS-specific field, so you need not to use OS-specific commands 
in the "install"
target.
6. Using "tr" with "uname" in order to delete some extra "ifeq 
($(OSTYPE),...)".
7. Other things, please see the file.

Anyway, easier building is my purpose. :-)

EXECS += $(EXAMPLES:%/$(BIN_DIR)/%)

EXECS += $(EXAMPLES:%=$(BIN_DIR)/%)

The linking of mhttpd misses a "-lm":

cc -g -O3 -Wall -Wuninitialized -DINCLUDE_FTPLIB -D_LARGEFILE64_SOURCE -DOS_LINUX
-fPIC -Wno-unused-function   -DHAVE_ZLIB -Iinclude -Idrivers -I../mxml -o
linux/bin/mhttpd linux/lib/mhttpd.o linux/lib/mgd.o linux/lib/libmidas.a -lutil
-lpthread -lz
linux/lib/mhttpd.o(.text+0xe08f): In function `show_custom_gif':
src/mhttpd.c:5058: undefined reference to `log'
linux/lib/mhttpd.o(.text+0xe0a8):src/mhttpd.c:5058: undefined reference to `log'

The header of the makefile should contain a short description, the author(s), an
$Id:$ tag for SVN, some explanation what "icc", "ifort" means, a note about the
CFLAGS and a clear statement what can be modified by the user and why and what not.

> The linking of mhttpd misses a "-lm":
> 
> cc -g -O3 -Wall -Wuninitialized -DINCLUDE_FTPLIB -D_LARGEFILE64_SOURCE -DOS_LINUX
> -fPIC -Wno-unused-function   -DHAVE_ZLIB -Iinclude -Idrivers -I../mxml -o
> linux/bin/mhttpd linux/lib/mhttpd.o linux/lib/mgd.o linux/lib/libmidas.a -lutil
> -lpthread -lz
> linux/lib/mhttpd.o(.text+0xe08f): In function `show_custom_gif':
> src/mhttpd.c:5058: undefined reference to `log'
> linux/lib/mhttpd.o(.text+0xe0a8):src/mhttpd.c:5058: undefined reference to `log'
> 

Strange. I tested it on Debian Linux 4.0r2 AMD64 with gcc 4.1.2, MIDAS SVN 4124. It worked fine.
Anyway, it can be fixed by addling "-lm" to the initial"LDFLAGS".

> The header of the makefile should contain a short description, the author(s), an
> $Id:$ tag for SVN, some explanation what "icc", "ifort" means, a note about the
> CFLAGS and a clear statement what can be modified by the user and why and what not.

OK. I will comment it in detail.

> > Most experience with autoconf/automake is all negative. The promise was "never debug your Makefile ever 
> > again!", delivered was "debug the configure script instead!". 
> 
> I admit that the new one is fit to my flavor. For a common user, I think, a simple procedure of configure/make/install
> is better than changing the Makefile manually because many users are lack of knowledges about Makefile. That's why 
> I want to learn autotools.

The reality is that you will never deliver a Makefile/Configure script that works for everybody in every case - users will always have a need to tweak the build 
process to suit their needs. In this situation, "Makefile" is a much better language and "make" is a much better tool for users to deal with - much simpler, better 
documented and better understood compared to autotools (*nobody* understands autotools; also compare the size of the midas Makefile with the size of a 
typical configure script).

Anyhow, we will be cross-compiling midas to run on a PowerPC processor inside a Virtex4 FPGA. Go handle that with configure scripts.

K.O.