Back Midas Rome Roody Rootana
  Midas DAQ System, Page 31 of 142  Not logged in ELOG logo
ID Date Author Topicdown Subject
  982   14 Mar 2014 Konstantin OlchanskiInfomidas wiki updated to mediawiki 1.22.4
The midas wiki at https://midas.triumf.ca was updated to mediawiki 1.22.4 - the latest production version. 
If you see any problems, please report them to this elog. K.O.
  983   14 Mar 2014 Konstantin OlchanskiInfoWindows support droped?
> In the old SVN midas world it was typically such that the Windows dll's and
> exe's were ready to be used when checking out.

The Windows executables are no longer included in the midas git repository. Old versions are still available in 
the git repository - they got pulled in during conversion from svn.

One reason for removing them is that neither myself, nor Pierre, nor Stefan have ready access to a Windows 
development environment and we cannot keep Windows binaries up to date. Theoretically we can setup a 
Windows machine just for compiling MIDAS, but then there is a question of which Windows we should use and 
how much priority we should put into it. I do not think there is any demand for MIDAS on Windows at TRIUMF.

(Personally, I think Windows is no longer a viable platform for any business use - with Microsoft focusing on 
"experiences", "tiles", touch screens, portable devices, and other gimmicks - rather than on providing a solid OS 
to get work done)

> I am not so sure this is the case
> for the current version, since when I use the packed dll's  and exe's (e.g.
> odbedit.exe) I get the warning that this is running midas 2.0.0 but the current
> version (on the linux server) is 2.1. What does this mean?

You can ignore this message. Stefan incremented the MIDAS version when we migrated to git, but
there are no changes to the MIDAS RPC mechanism and we are still fully compatible with old versions,
at least in the MIDAS RPC and in the mserver.

So tools like odbedit.exe should still work okey when connecting from Windows to MIDAS running on Linux or 
MacOS.

But old frontend programs may cause some trouble because the ODB layout changed somewhat with new things 
added to /eq/xxx/common. Simplest is to try, if it works, it works.

> 1) A little bug in the packed windows part, but up-to-date dll's and exe's?
> 2) The dll's and exe's are not bundled any more to up-to-date version?

Case (2) is the case. Personally I do not have any capability to build Windows binaries. Same for Pierre and I think 
for Stefan.

> If 2) is the case, I would like to get a hint how to build midas under Windows
> (Windows 7), since we still have some few Windows clients.  

I do not think pre-built executables will ever return - the new way of things is to "cut-and-paste" the "git clone" 
command from a web page, type "make", and be done with it. If your OS does not have "git", "make" & etc, you 
should switch to a real OS.

On the MIDAS software side, we have no problem with supporting Windows - same as on any other platform, 
please try to build and run it, report any problems, fixes, patches and improvements - we will commit them into 
the midas repository.

K.O.
  986   17 Mar 2014 Stefan RittInfoWindows support droped?
> The Windows executables are no longer included in the midas git repository. Old versions are still available in 
> the git repository - they got pulled in during conversion from svn.
> 
> One reason for removing them is that neither myself, nor Pierre, nor Stefan have ready access to a Windows 
> development environment and we cannot keep Windows binaries up to date. Theoretically we can setup a 
> Windows machine just for compiling MIDAS, but then there is a question of which Windows we should use and 
> how much priority we should put into it. I do not think there is any demand for MIDAS on Windows at TRIUMF.

I double checked and can confirm that the executables in GIT are very old. So I tried to compile the current version for Windows. I found that I had to change lots 
of places (basically all the new files written by KO) to make it work again, so it took me half a day, but now should be fine.

I'm not sure if it's a good idea to keep .exe files in GIT, maybe we should remove it some day, but for the moment I updated the executables to the current 
version. Feedback welcome.

/Stefan
  988   17 Mar 2014 Konstantin OlchanskiInfoODB JSON support
> > > odbedit can now save ODB in JSON-formatted files.
> encode NaN, Inf and -Inf as JSON string values "NaN", "Infinity" and "-Infinity". (Corresponding to the respective Javascript values).

A new standard just came out - Oasis OData JSON format 4.0 - 
http://docs.oasis-open.org/odata/odata-json-format/v4.0/os/odata-json-format-v4.0-os.html

Section 7.1 reads:

> Values of types [...] Edm.Single, Edm.Double, and Edm.Decimal are represented as JSON numbers, except for NaN, INF, and –INF which are represented as strings.

This is consistent with what we do in MIDAS - encode special numbers as strings. For now I think we stay with Javascript-standard "Infinity", "-Infinity",
but if more standards start using "INF", "-INF", maybe we will switch. It is easy enough to support both encodings in the JSON parser and in the ODB decoder.

https://xkcd.com/927/
K.O.
  1014   11 Jul 2014 Konstantin OlchanskiInfoMIDAS high speed test
We have tested operation of MIDAS using a 10GigE network connection. Using a dummy frontend 
generating fake data, we can record MIDAS data to disk at at least 700 Mbytes/sec as reported by 
the MIDAS status page.

Two configurations were tested, both run at at least 700Mbytes/sec sustained:

1) MIDAS mhttpd, mserver, mlogger running on the disk server machine (mlogger writes to local 
disk), frontend running on remote machine (10GigE mserver connection).
2) MIDAS mhttpd, mserver, mlogger, frontend running on remote machine (mlogger writes data to 
an NFS-mounted disk over a 10GigE connection).

In addition, for configuration (2), I simulated online analysis reading fresh midas files at the same 
time as MIDAS writes new data. The resulting observation is that Linux seems to be giving main 
priority to disk write traffic (700 Mbytes/sec) with the remaining disk bandwidth given to read traffic 
(50-100Mbytes/sec). In other words, when running online data analysis on fresh data files, mlogger 
continues to run at full speed (analysis does not slow down data taking).

A few problems with MIDAS were observed during this test:

a) mlogger data compression using gzip-1 has to be turned off (limits data rate to about 
200Mbytes/sec). We plan to implement high speed LZO/LZ4 data compression that we expect to 
keep up with a 10GigE network interface.
b) CPU use by mserver and mlogger is rather high (about 40% CPU)
c) when writing to the NFS disk, mlogger has a pause of 1-2 seconds when closing and reopening 
subrun data files. To avoid a interruption in data taking, the SYSTEM event buffer has to be big 
enough to ride through this pause, but stock MIDAS limits the maximum size of event buffer to 1GB 
(too small), this can be easily increased to 2GB (almost big enough) and with some more work it can 
be increased to 4GB, but no more because the buffer length is a 32-bit integer.
d) when writing to the NFS disk, we also see periodic 3-5 second interruptions ("write operation took 
5123 ms") and we had one death of mlogger by a timeout of 60 sec.

Details of the hardware:

1) the disk server machine CPU is 3.4GHz Intel i7-4770, mobo is ASUS Z87 WS (10 SATA, 2xGigE), 
RAM is 32GB DDR3-1600.
2) disk array is 8x4TB Seagate ST4000VN000-1H4168 NAS disks RAID0 (striped) configuration, raw 
data read/write rate is around 1 GByte/sec, disks are directly attached to mobo (no raid card), linux 
software raid.
3) the frontend machine CPU is 3.7GHz Intel i7-4820, mobo is ASUS P9X79 WS, RAM is 32GB DDR3-
1600.
4) 10GigE network is Solarflare Communications SFC9120 (both machines) with a cross-over fiber 
cable (direct connection,no switches)
5) OS is up-to-date SL6.5 (both machines)

K.O.
  1019   06 Aug 2014 Konstantin OlchanskiInfoMIDAS high speed test
> We have tested operation of MIDAS using a 10GigE network connection. Using a dummy frontend 
> generating fake data, we can record MIDAS data to disk at at least 700 Mbytes/sec as reported by 
> the MIDAS status page.
>
> Details of the hardware:
> 
> 1) the disk server machine CPU is 3.4GHz Intel i7-4770, mobo is ASUS Z87 WS (10 SATA, 2xGigE), 
> RAM is 32GB DDR3-1600.
> 2) disk array is 8x4TB Seagate ST4000VN000-1H4168 NAS disks RAID0 (striped) configuration, raw 
> data read/write rate is around 1 GByte/sec, disks are directly attached to mobo (no raid card), linux 
> software raid.
>

These tests were done using a raid0 array (striped), which is not suitable for production use.

For production use, RAID5 and RAID6 is recommended. But their default configuration has severely reduced performance (50% of 
RAID0) this is because internally the raid driver issues disk read operations that compete against and severely slow down the disk write 
requests. This is easy to see with "iostat -x 1" - when writing to the raid array, there should be no reads from the disks. Following 
changes are required to achieve maximum performance:

echo 32000 > /sys/block/md6/md/stripe_cache_size # increase internal memory buffers - because "raid write" is always "read-
modify-write", bigger buffers ensure that the reads are done from cache, not from phsyical disk
mdadm --grow --bitmap=/md6bitmap /dev/md6 # use external bitmap - if bitmap is internal, there is a large number of disk reads 
competing against writes. external bitmap seems to help quite a bit.

With these settings, my RAID6 array can read and write at about 700-900 Mbytes/sec - this is comparable to RAID0 (minus 2 disks).

With this, I repeated the MIDAS performance tests - (but without 10GigE) - MIDAS can write 700 Mbytes/sec of fake data to a local 
RAID6 data array. (hardware configuration is listed above).

K.O.
  1051   07 May 2015 Konstantin OlchanskiInfomidas.triumf.ca https ssl certificate update
The SSL certificate for https://midas.triumf.ca has been resigned with SHA256 to fix the complaint from google-chrome about SHA1-signed certificate - 
SHA1 signatures are now considered to be insufficiently secure, have to be replaced by SHA256.

The fingerprints for the new certificate are:
SHA256: 44:03:EA:FB:C5:83:24:01:23:7F:B6:4A:B3:87:A1:0C:98:6F:9F:1D:20:F4:3C:38:45:38:09:A4:6C:30:B9:4B
SHA1: 34:FB:6A:42:0D:92:D7:69:48:75:AD:FE:C8:1C:F7:B6:0B:07:1E:2F
MD5: C1 3D 99 50 13 81 19 FA 7E 65 60 4F F0 FC 99 EA

K.O.
  1062   22 May 2015 Konstantin OlchanskiInfomhttpd HTTPS/SSL server updated
I updated the mhttpd HTTPS/SSL server (mongoose) and https://www.ssllabs.com/ssltest/index.html is 
now more or less happy with it. google chrome connects using "modern cryptography".

The HTTPS/SSL server is activated using "mhttpd --mg" (instead of -p) and it listens on port 8443.

The example SSL certificate provided in midas git is self-signed, for instructions on generating your own 
signed certificate, remove it and run "mhttpd --mg" - it will print the correct instructions.

List of corrected problems:
a) SSL certificate was generated with key length 1024 and SHA1 signature - should be 2048 and SHA256.
b) SSLv2, SSLv3 were not disabled per latest recommendations
c) RC4 and other weak ciphers were not disabled per latest recommendations
d) "modern cryptography" and "forward secrecy" were not available because they require special fondling of 
openssl.
e) on MacOS 10.9 *again* a whole bunch of openssl functions are listed as deprecated with no suggested 
replacement, there is a mismatch between system openssl and macports openssl and "modern 
cryptography" ECDH ciphers are not available.

Also to remember, mhttpd uses the latest release of mongoose 4.2 which is no longer supported by 
author. Latest version of mongoose is 5.x which has a severely improved API, but removed automatic 
multithreading.

I recommend that you use "mhttpd --mg" as the alternative for running "mhttpd -p" behind an apache 
proxy. Using "mhttpd -p" (no HTTPS/SSL) on an internet-connected machine is insecure and should not be 
done. (private network such as 192.168.x.y addresses is okey for now, I guess).

https://bitbucket.org/tmidas/midas/commits/d85ba733573f1fca9946804eeb71d6fdc23bea22

K.O.
  1066   07 Jul 2015 Konstantin OlchanskiInfomhttpd HTTPS/SSL server updated
> mhttpd uses the latest release of mongoose 4.2 which is no longer supported by 
> author. Latest version of mongoose is 5.x which has a severely improved API, but removed automatic 
> multithreading.

The exact version of mongoose 4.2 included with MIDAS is git revision 607651a3ffce43ef424530b22c7b1d22381de02d from 11 
November 2013.

https://github.com/cesanta/mongoose/commit/607651a3ffce43ef424530b22c7b1d22381de02d.

Documentation for this version of mongoose is committed to midas git repository .../midas/doc/mongoose.

K.O.
  1068   15 Jul 2015 Konstantin OlchanskiInfomlogger improvements
A set of improvements to mlogger is in:

a) event buffer (SYSTEM) size up to 2GB
b) test version of LZ4 high speed compression, support for bzip2 and pbzip2

Details:

a) previously contents of shared memory buffers (SYSTEM, SYSMSG, etc) were periodically saved to disk 
files SYSTEM.SHM, SYSMSG.SHM, etc. This was not workable for large event buffers - reading/writing 2GB 
of data takes quite some time. We have decided that saving buffer contents to disk is no longer necessary 
and ss_shm_close() no longer writes SYSTEM.SHM, SYSMSG.SHM, etc. From now on you will still see these 
files created, but size will be 0. The file ODB.SHM is not affected by this - ODB contents is saved to 
ODB.SHM via ss_shm_flush().

b) as a rework of mlogger file output drivers (using chainable c++ classes), test versions of new 
compression algorithms have been added. In the present test version, they are controlled by the value of 
"compression".

The plan is to ultimately have following outputs from the mlogger:

- ROOT output - save as before, but you have to use rmlogger executable
- FTP output - for high speed write over the network
- .mid output for uncompressed data
- .mid.gz - gzip1 compressed data - best compromise between compression ratio and speed - will be the 
new default
- .mid.bz2 via pbzip2 (parallel bzip2) - maximum compression ratio
- .mid.lz4 - lz4 compression for high speed data taking - maximum compression speed

The current test version implements the following selections of "compression":

80 - ROOT output through the new driver (use rmlogger executable)
98 - null output (no file written)
99 - uncompressed disk output
100 - lz4 comression
200 - piped bzip2 compression
201 - piped pbzip2 compression
300 - gzip compression
301 - gzip1 compression
309 - gzip9 compression

in addition the old selections are still available:
0 - uncompressed output
1 - gzip1 compression
9 - gzip9 compression

The final implementation will include a better way to configure the mlogger output channels.

K.O.
  1069   15 Jul 2015 Konstantin OlchanskiInfomhttpd HTTPS/SSL server updated
> > mhttpd uses the latest release of mongoose 4.2

mhttpd is now explicitly linked with OpenSSL to provide secure https connections via the mongoose web server.

a) google chrome reports "Your connection to ... is encrypted with modern cryptography." via TLS 1.2 and ECDHE ciphers
b) we believe there are no Linux systems that require running mhttpd and lack OpenSSL, but building mhttpd without OpenSSL is supported, see the cross 
compilation section in the Makefile.
c) MacOS comes with a very old version of OpenSSL. mhttpd will build, https will work, but with a complaint about "obsolete cryptography". Please install an up-to-
date OpenSSL package via macports.
d) security of OpenSSL itself is quite problematic, please keep an eye open on OpenSSL security advisories, update OpenSSL and restart/rebuild mhttpd promptly. I 
expect the mongoose project to eventually switch from OpenSSL to one of the new-generation TLS libraries, such as PolarSSL (embed_tls) and we will follow their 
lead.

K.O.
  1070   15 Jul 2015 Konstantin OlchanskiInfoROOT support in flux
ROOT support in MIDAS is being reworked:

a) ROOT support moved from midas.h to rmidas.h
b) default mlogger is built without ROOT support, use rmlogger if you need ROOT output
c) Makefile inconsistency between use of ROOTSYS and use of root-config has been identified, but not yet 
fixed. the plan is to use root-config to detect and use the ROOT package.
d) cross compilation will not support ROOT (same as now. "make linux32", "make linux64", "make 
linuxarm" disable most optional packages. To build full featured midas with ROOT & etc please compile 
natively on the ARM machine).
e) histogram servers in MIDAS and ROOTANA will be switched to use the new ROOT Web server classes 
(based on "civet", a fork of the mongoose web server).

K.O.
  1073   22 Jul 2015 Konstantin OlchanskiInfoROOT support in flux
> ROOT support in MIDAS is being reworked:
>
> c) Makefile inconsistency between use of ROOTSYS and use of root-config has been identified,

MIDAS Makefile was corrected to use root-config exclusively to find and use ROOT. This makes us more consistent
with the ROOT-recommended use of the thisroot.{sh,csh} scripts.

In other words, if root-config is in the PATH, ROOT support will be enabled, rmlogger and rmana.o will be built.

To explicitly disable ROOT, say "make NO_ROOT=1"

K.O.
  1074   22 Jul 2015 Konstantin OlchanskiInfoROOT support in flux
> > ROOT support in MIDAS is being reworked:
> >
> > c) Makefile inconsistency between use of ROOTSYS and use of root-config has been identified,
> 
> MIDAS Makefile was corrected to use root-config exclusively to find and use ROOT. This makes us more consistent
> with the ROOT-recommended use of the thisroot.{sh,csh} scripts.
> 

The updated ROOT instructions on DAQwiki
https://www.triumf.info/wiki/DAQwiki/index.php/ROOT
now explain how to use "thisroot" to select the right version of the package.

The preliminary version of the .bashrc blurb looks like this
(a couple of flaws:
1) identification of CentOS7 is incomplete - please send me a patch
2) there should be a check for root-config already in the PATH, as on Ubuntu, the ROOT package may be installed in /usr and root-
config may be already in the path - please send me patch).

if [ `uname -i` == "i386" ]; then 
    . /daq/daqshare/olchansk/root/root_v5.34.01_SL62_32/bin/thisroot.sh 
    true 
elif [ `lsb_release -r -s` == "7.1.1503" ]; then 
    #. /daq/daqshare/olchansk/root/root_v5.34.32_SL66_64/bin/thisroot.sh 
    true 
else 
     . /daq/daqshare/olchansk/root/root_v5.34.32_SL66_64/bin/thisroot.sh 
     true 
fi 

K.O.
  1075   23 Jul 2015 Konstantin OlchanskiInforootana lz4 support, mlogger improvements
> A set of improvements to mlogger is in:
> b) test version of LZ4 high speed compression, support for bzip2 and pbzip2

rootana TMidasFile now supports reading .mid.lz4 compressed files via pipe through the "lz4" utility ("yum install lz4", "apt-get install liblz4-tool").

In MIDAS, the lz4 libraries are included with the MIDAS distribution, we are considering the same for ROOTANA.

(Support for reading mid.bz2 files via pipe through the bzip2 utility existed for a very long time).

https://bitbucket.org/tmidas/rootana/commits/e06bb7296a466b4178c7768bbc2470be361b2c72

K.O.
  1076   23 Jul 2015 Konstantin OlchanskiInfomlogger improvements
> A set of improvements to mlogger is in:
> The current test version implements the following selections of "compression":
> 
> 80 - ROOT output through the new driver (use rmlogger executable)
> ...

Additional output modes through the new output drivers:

81 - FTP output
82 - FTP output with LZ4 compression

The format of the "Channels/xxx/Settings/Filename" for FTP output is like this:
"/localhost, 5555, ftpuser, ftppwd, ., run%05dsub%05d.mid"

- the leading slash is required (for now)
- localhost is the FTP server hostname
- 5555 is the FTP server port number
- ftpuser and ftppwd are the FTP login. password is stored and transmitted in clear text for extra security
- "." is the output directory on the FTP server
- the rest is the file name in the usual format.

For testing this driver, I run the ftp server like this:

# vsftpd -olisten=YES -obackground=no -olisten_port=5555 -olisten_address=127.0.0.1 -oport_promiscuous=yes -oconnect_from_port_20=no -oftp_data_port=6666

K.O.
  1077   24 Jul 2015 Konstantin OlchanskiInfoMAX_EVENT_SIZE removed
The define for MAX_EVENT_SIZE was removed from midas.h.

Replacing it is DEFAULT_MAX_EVENT_SIZE set to 4 MiBytes and DEFAULT_BUFFER_SIZE 
set to 32 MiBytes.

For a long time now MIDAS does not have hardcoded maximum event size and buffer size 
and this change merely renames the define to reflect it's current function.

The actual maximum event size is set by ODB /Experiment/MAX_EVENT_SIZE.

The actual event buffer sizes are set by ODB "/Experiment/Buffer sizes/SYSTEM" & co

K.O.
  Draft   24 Jul 2015 Konstantin OlchanskiInfoPlans for improving midas network security
There is a number of problems with network security in midas. (as separate from web/http/https security).

1) too many network sockets are unnecessarily bound to the external network interface instead of localhost (UDP ports are already bound to localhost on MacOS).
2) by default the RPC ports of each midas program accept connections and RPC commands from anywhere in the world (an access control list is already implemented via /Experiment/Security/Rpc Hosts, but not active by default)
3) mserver also has an access control list but it is not integrated with the access control list for the RPC ports.
4) it is difficult to run midas in the presence of firewalls (midas programs listen on random network ports - cannot be easily added to firewall rules)

There is a new git branch "feature/rpcsecurity" where I am addressing some of these problems:

1) UDP sockets are only used for internal communications (hotlinks & etc) within one machine, so they should be bound to the localhost address and become invisible to external machines. This change breaks binary compatibility from old clients - they are have to be relinked with the new midas library or hotlinks & etc will stop working. If some clients cannot be rebuild (I have one like this), I am preserving the old way by checking for a special file in the experiment directory (same place as ODB.SHM). (done)

2) if one runs on a single machine, does not use the mserver and does not have clients running on other machines, then all the RPC ports can be bound to localhost. (this kills the MacOS popups about "odbedit wants to connect to the Internet"). (partially done)

This (2) will become the new default - out of the box, midas will not listen to any external network connections - making it very secure.

To use the mserver, one will have to change the ODB setting "/Experiment/Security/Enable external RPC connections" and restart all midas programs (I am looking for a better name for this odb setting).

3) the out-of-the-box default access control list for RPC connections will be set to "localhost", which will reject all external connections, even when they are enabled by (2). One will be required to enter the names of all machines that will run midas clients in "/Experiment/Security/Rpc hosts". (already implemented in main midas, but default access control list is empty meaning everybody is permitted)

4) the mserver will be required to attach to some experiment and will use this same access control list to restrict access to the main mserver listener port. Right now the mserver listens on this port without attaching to any experiment and accepts the access control list via command line arguments. I think after this change a single mserver will still be able to service multiple experiments (TBD).

5) I am adding an option to fix TCP port numbers for MIDAS programs via "/Experiment/Security/Rpc ports/fename = (int)5555". Once a remote frontend is bound to a fixed port, appropriate openings can be made in the firewall, etc. Default port number value will be 0 meaning "use random port", same as now.

One problem remains with initial connecting to the mserver. The client connects to the main mserver listener port (easy to firewall), but then the mserver connects back to the client - this reverse connection is difficult to firewall and this handshaking is difficult to fix in the midas sources. It will probably remain unresolved for now.

K.O.
  1079   24 Jul 2015 Konstantin OlchanskiInfoPlans for improving midas network security
There is a number of problems with network security in midas. (as separate from 
web/http/https security).

1) too many network sockets are unnecessarily bound to the external network interface 
instead of localhost (UDP ports are already bound to localhost on MacOS).
2) by default the RPC ports of each midas program accept connections and RPC commands 
from anywhere in the world (an access control list is already implemented via 
/Experiment/Security/Rpc Hosts, but not active by default)
3) mserver also has an access control list but it is not integrated with the access control list 
for the RPC ports.
4) it is difficult to run midas in the presence of firewalls (midas programs listen on random 
network ports - cannot be easily added to firewall rules)

There is a new git branch "feature/rpcsecurity" where I am addressing some of these 
problems:

1) UDP sockets are only used for internal communications (hotlinks & etc) within one 
machine, so they should be bound to the localhost address and become invisible to external 
machines. This change breaks binary compatibility from old clients - they are have to be 
relinked with the new midas library or hotlinks & etc will stop working. If some clients cannot 
be rebuild (I have one like this), I am preserving the old way by checking for a special file in 
the experiment directory (same place as ODB.SHM). (done)

2) if one runs on a single machine, does not use the mserver and does not have clients 
running on other machines, then all the RPC ports can be bound to localhost. (this kills the 
MacOS popups about "odbedit wants to connect to the Internet"). (partially done)

This (2) will become the new default - out of the box, midas will not listen to any external 
network connections - making it very secure.

To use the mserver, one will have to change the ODB setting "/Experiment/Security/Enable 
external RPC connections" and restart all midas programs (I am looking for a better name for 
this odb setting).

3) the out-of-the-box default access control list for RPC connections will be set to 
"localhost", which will reject all external connections, even when they are enabled by (2). One 
will be required to enter the names of all machines that will run midas clients in 
"/Experiment/Security/Rpc hosts". (already implemented in main midas, but default access 
control list is empty meaning everybody is permitted)

4) the mserver will be required to attach to some experiment and will use this same access 
control list to restrict access to the main mserver listener port. Right now the mserver listens 
on this port without attaching to any experiment and accepts the access control list via 
command line arguments. I think after this change a single mserver will still be able to service 
multiple experiments (TBD).

5) I am adding an option to fix TCP port numbers for MIDAS programs via 
"/Experiment/Security/Rpc ports/fename = (int)5555". Once a remote frontend is bound to a 
fixed port, appropriate openings can be made in the firewall, etc. Default port number value 
will be 0 meaning "use random port", same as now.

One problem remains with initial connecting to the mserver. The client connects to the main 
mserver listener port (easy to firewall), but then the mserver connects back to the client - this 
reverse connection is difficult to firewall and this handshaking is difficult to fix in the midas 
sources. It will probably remain unresolved for now.

K.O.
  1080   28 Jul 2015 Konstantin OlchanskiInfoPlans for improving midas network security
New git branch "feature/rpcsecurity" implements  these security features:

- all UDP ports are bound to the localhost interface - connections from outside are not possible
- by default out of the box MIDAS RPC TCP ports are bound to the localhost interface - connections from the outside are not possible.

This configuration is suitable for testing MIDAS on a laptop and for running a simple experiment where all programs run on one machine.

This configuration is secure (connections from the outside are not possible).

This configuration makes corporate security people happy - MIDAS ports do not show up on network port scans (nmap & etc). (except for the mhttpd 
web ports).

The change in binding UDP ports is incompatible with previous versions of MIDAS (except on MacOS, where UDP ports were always bound to localhost). 
All MIDAS programs should be rebuild and restarted, otherwise ODB hotlinks and some other stuff will not work. If rebuilding all MIDAS programs is 
impossible (for example I have one magic MIDAS frontend that cannot be rebuilt), one can force the old (insecure) behavior by creating a file 
.UDP_BIND_HOSTNAME in the experiment directory (next to .ODB.SHM).

The mserver will still work in this localhost-restricted configuration - one should use "odbedit -h localhost" to connect. Multiple mserver instances on 
the same machine - using different TCP ports via "-p" and ODB "/Experiment/midas server port" - are still supported.

To run MIDAS programs on remote machines, one should change the ODB setting "/Experiment/Security/Enable non-localhost RPC" to "yes" and 
add the hostnames of all remote machines that will run MIDAS programs to the MIDAS RPC access control list in ODB "/Experiment/Security/RPC hosts".

To avoid "guessing" the host names expected by MIDAS, do this: set "enable non-localhost rpc" to "yes" and restart the mserver. Then go to the remote 
machine and try to start the MIDAS program, i.e. "odbedit -h daq06". This will bomb and there will be a message in the midas log file saying - rejecting 
connection from unallowed host 'ladd21.triumf.ca'. Add this host to "/Experiment/Security/RPC hosts". After you add this hostname to "RPC hosts" and 
restart the mserver, the connection should be successful. When "RPC hosts" is fully populated, one should restart all midas programs - the access 
control list is only loaded at program startup.

If MIDAS clients have to connect from random hosts (i.e. dynamically assigned random DHCP addresses), one can disable the host name checks by 
setting ODB "/experiment/security/Disable RPC hosts check" to "yes". This configuration is insecure and should only be done on a private network 
behind a firewall.

After some more testing this branch will be merged into the main midas.

K.O.
ELOG V3.1.4-2e1708b5