> > > This list of responsible being attached to alarm message strings ...
> > 
> > This is a great idea. But I think we do not need to artificially limit ourselves
> > to string and array lengths.
> > 
> > The code in alarm.c should be changes to use std::string and std::vector<std::string> (STRING_LIST 
> > #define), db_get_record() should be replaced with individual ODB reads (that's what it does behind 
> > the scenes, but in a non-type and -size safe way).
> > 
> > I think the web page code will work correctly, it does not care about string lengths.
> > 
> > K.O.
> 
> Auto growing lists is an excellent plan. I am making decent progress and should have something to 
> report soon

This has sent me down a little rabbit hole, and I'd like to check in with efforts to improve the efficiency and simplicity of the alarm code.

I can keep with the current 'C' style of the alarm.cxx code, replace struct read and writes to the odb with individual odb entries... put functions in alarm.cxx to create, read and write to the odb... 

If we go this 'C' style route, then I'll have duplication of the 'users responsible' setters and getter functions for structs ALARM and PROGRAM_INFO

What would the MIDAS developers thing of creating classes for ALARM and PROGRAM_INFO (I am thinking for binary compatibilities of not touching ALARM and PROGRAM_INFO structs, and inheriting from them:

class UsersResponsible
{
   public:
      STRING_LIST fUsersResponsible;
}

class Alarm: public ALARM, public UsersResponsible
{
}

class ProgramInfo: public PROGRAM_INFO, public UsersResponsible
{
}

Each of these three classes would have member to functions to Create, Read and Write to the ODB. We could get rid of the PROGRAM_INFO_STR precompiler macro and instead have a SetToDefault member function. 

It seems clear we should set the ODB path in the constructor of Alarm and ProgramInfo

mlogger (history_image) now supports rtsp cameras, in ALPHA we have 
acquisitioned several new network connected cameras. Unfortunately they dont 
have a way of just capturing a single frame using libcurl


========================================
Motivation to link to OpenCV libraries
========================================

After looking at the ffmpeg libraries, it seemed non trivial to use them to 
listen to a rtsp stream and write a series of jpgs.

OpenCV became an obvious choice (it is itself linked to ffmpeg and 
gstreamer), its a popular, multiplatform, open source library that's easy to 
use. It is available in the default package managers in centos 7 and ubuntu 
(an is installed by default on lxplus).

========================================
How it works:
========================================

The framework laid out in history_image.cxx is great. A separate thread is 
dedicated for each camera. This is continued with the rtsp support, using 
the same periodicity:

if (ss_time() >= o["Last fetch"] + o["Period"]) {
An rtsp camera is detected by its URL, if the URL starts with ‘rtsp://’ its 
obvious its using the rtsp protocol and the cv::VideoCapture object is 
created (line 147).

If the connection fails, it will continue to retry, but only send an error 
message on the first 10 attempts (line 150). This counter is reset on 
successful connection
If MIDAS has been built without OpenCV, mlogger will send an error message 
that OpenCV is required if a rtsp URL is given (line 166)
The VideoCapture ‘stays live' and will grab frames from the camera based on 
the sleep, saving to file based on the Period set in the ODB.

If the VideoCapture object is unable to grab a frame, it will release() the 
camera, send an error message to MIDAS, then destroy itself, and create a 
new version (this destroy and create fully resets the connection to a 
camera, required if its on flaky wifi)
If the VideoCapture gets an empty frame, it also follows the same reset 
steps.
If the VideoCaption fills a cv::Frame object successfully, the image is 
saved to disk in the same way as the curl tools.

========================================
Concerns for the future:
========================================

VideoCapture is decoding the video stream in the background, allowing us to 
grab frames at will. This is nice as we can be pretty agnostic to the video 
format in the stream (I tested with h264 from a TP-LINK TAPO C100, but the 
CPU usage is not negligible.

I noticed that this used ~2% of the CPU time on an intel i7-4770 CPU, given 
enough cameras this is considerable. In ALPHA, I have been testing with 10 
cameras:
elog:2220/1 

My suggestion / request would be to move the camera management out of 
mlogger and into a new program (mcamera?), so that users can choose to off 
load the CPU load to another system (I understand the OpenCV will use GPU 
decoders if available also, which can also lighten the CPU load).

In ALPHA, I get RPC timeouts running a (reasonably heavy) analyzer on a remote machine (connected directly via a ~30 meter 10Gbe Ethernet cable) after ~5 minutes of running. If I run the analyser locally, I dont not see a timeout...

gdb trace:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff5d35859 in __GI_abort () at abort.c:79
#2  0x00005555555a2a22 in rpc_call (routine_id=11111) at /home/alpha/packages/midas/src/midas.cxx:13866
#3  0x000055555562699d in bm_receive_event_rpc (buffer_handle=buffer_handle@entry=2, buf=buf@entry=0x0, buf_size=buf_size@entry=0x0, ppevent=ppevent@entry=0x0, pvec=pvec@entry=0x7fffffffd700,
    timeout_msec=timeout_msec@entry=100) at /home/alpha/packages/midas/src/midas.cxx:10510
#4  0x0000555555631082 in bm_receive_event_vec (buffer_handle=2, pvec=pvec@entry=0x7fffffffd700, timeout_msec=timeout_msec@entry=100) at /home/alpha/packages/midas/src/midas.cxx:10794
#5  0x0000555555673dbb in TMEventBuffer::ReceiveEvent (this=this@entry=0x555557388b30, e=e@entry=0x7fffffffd700, timeout_msec=timeout_msec@entry=100) at /home/alpha/packages/midas/src/tmfe.cxx:312
#6  0x0000555555607b56 in ReceiveEvent (b=0x555557388b30, e=0x7fffffffd6c0, timeout_msec=100) at /home/alpha/packages/midas/manalyzer/manalyzer.cxx:1411
#7  0x000055555560d8dc in ProcessMidasOnlineTmfe (args=..., progname=<optimized out>, hostname=<optimized out>, exptname=<optimized out>, bufname=<optimized out>, event_id=<optimized out>,
    trigger_mask=<optimized out>, sampling_type_string=<optimized out>, num_analyze=0, writer=<optimized out>, multithread=<optimized out>, profiler=<optimized out>,
    queue_interval_check=<optimized out>) at /home/alpha/packages/midas/manalyzer/manalyzer.cxx:1534
#8  0x000055555560f93b in manalyzer_main (argc=<optimized out>, argv=<optimized out>) at /usr/include/c++/9/bits/basic_string.h:2304
#9  0x00007ffff5d37083 in __libc_start_main (main=0x5555555b1130 <main(int, char**)>, argc=8, argv=0x7fffffffdda8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>,
    stack_end=0x7fffffffdd98) at ../csu/libc-start.c:308
#10 0x00005555555b184e in _start () at /usr/include/c++/9/bits/stl_vector.h:94

Any suggestions? Many thanks

I want to read data from the ODB with the mvodb interface in one of my frontends, it's useful to know how old that data is, so I prototyped functionality in a pull request to mvodb:

https://bitbucket.org/tmidas/mvodb/pull-requests/2/add-readkeylastwritten-function-to-extract

I'm brand new to MIDAS, and C system programming in general, so please be
gentle. I've compiled and installed MIDAS 2.3.0 on Ubuntu 10.04 LTS. I've built
the kernel driver, khyt1331.ko and installed it. It appears to be working, since
the camactest and esonetest programs included with the driver work just fine.

So I attempted to build the example experiment distributed with MIDAS, with the
following changes to the Makefile:

DRV_DIR   = $(MIDASSYS)/drivers/kernel/khyt1331_26

and

DRIVER = camac

The programs build without error but when I try to start the frontend, I get:

$ ./frontend
Frontend name          :     CSM-Nuclear Portable DAQ Frontend
Event buffer size      :     1000000
User max event size    :     10000
User max frag. size    :     5242880
# of events per buffer :     100

Connect to experiment...
*** buffer overflow detected ***: ./frontend terminated
======= Backtrace: =========
/lib/tls/i686/cmov/libc.so.6(__fortify_fail+0x50)[0x6de390]
/lib/tls/i686/cmov/libc.so.6(+0xe12ca)[0x6dd2ca]
/lib/tls/i686/cmov/libc.so.6(__strcpy_chk+0x44)[0x6dc644]
./frontend[0x805611f]
./frontend[0x806f656]
./frontend[0x8053d82]
/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x612bd6]
./frontend[0x804bb81]
======= Memory map: ========
00110000-0012d000 r-xp 00000000 08:05 7471187    /lib/libgcc_s.so.1
0012d000-0012e000 r--p 0001c000 08:05 7471187    /lib/libgcc_s.so.1
0012e000-0012f000 rw-p 0001d000 08:05 7471187    /lib/libgcc_s.so.1
00264000-00277000 r-xp 00000000 08:05 7603242    /lib/tls/i686/cmov/libnsl-2.11.1.so
00277000-00278000 r--p 00012000 08:05 7603242    /lib/tls/i686/cmov/libnsl-2.11.1.so
00278000-00279000 rw-p 00013000 08:05 7603242    /lib/tls/i686/cmov/libnsl-2.11.1.so
00279000-0027b000 rw-p 00000000 00:00 0 
002db000-002dd000 r-xp 00000000 08:05 7603265   
/lib/tls/i686/cmov/libutil-2.11.1.so
002dd000-002de000 r--p 00001000 08:05 7603265   
/lib/tls/i686/cmov/libutil-2.11.1.so
002de000-002df000 rw-p 00002000 08:05 7603265   
/lib/tls/i686/cmov/libutil-2.11.1.so
003b1000-003c6000 r-xp 00000000 08:05 7603257   
/lib/tls/i686/cmov/libpthread-2.11.1.so
003c6000-003c7000 r--p 00014000 08:05 7603257   
/lib/tls/i686/cmov/libpthread-2.11.1.so
003c7000-003c8000 rw-p 00015000 08:05 7603257   
/lib/tls/i686/cmov/libpthread-2.11.1.so
003c8000-003ca000 rw-p 00000000 00:00 0 
004ea000-004f1000 r-xp 00000000 08:05 7603261    /lib/tls/i686/cmov/librt-2.11.1.so
004f1000-004f2000 r--p 00006000 08:05 7603261    /lib/tls/i686/cmov/librt-2.11.1.so
004f2000-004f3000 rw-p 00007000 08:05 7603261    /lib/tls/i686/cmov/librt-2.11.1.so
005fb000-005fc000 r-xp 00000000 00:00 0          [vdso]
005fc000-0074f000 r-xp 00000000 08:05 7603231    /lib/tls/i686/cmov/libc-2.11.1.so
0074f000-00750000 ---p 00153000 08:05 7603231    /lib/tls/i686/cmov/libc-2.11.1.so
00750000-00752000 r--p 00153000 08:05 7603231    /lib/tls/i686/cmov/libc-2.11.1.so
00752000-00753000 rw-p 00155000 08:05 7603231    /lib/tls/i686/cmov/libc-2.11.1.so
00753000-00756000 rw-p 00000000 00:00 0 
00783000-00796000 r-xp 00000000 08:05 7471302    /lib/libz.so.1.2.3.3
00796000-00797000 r--p 00012000 08:05 7471302    /lib/libz.so.1.2.3.3
00797000-00798000 rw-p 00013000 08:05 7471302    /lib/libz.so.1.2.3.3
008ab000-008c6000 r-xp 00000000 08:05 7471129    /lib/ld-2.11.1.so
008c6000-008c7000 r--p 0001a000 08:05 7471129    /lib/ld-2.11.1.so
008c7000-008c8000 rw-p 0001b000 08:05 7471129    /lib/ld-2.11.1.so
008e4000-00908000 r-xp 00000000 08:05 7603239    /lib/tls/i686/cmov/libm-2.11.1.so
00908000-00909000 r--p 00023000 08:05 7603239    /lib/tls/i686/cmov/libm-2.11.1.so
00909000-0090a000 rw-p 00024000 08:05 7603239    /lib/tls/i686/cmov/libm-2.11.1.so
08048000-0809d000 r-xp 00000000 08:11 20318114   /home/midas/online/test/frontend
0809d000-0809e000 r--p 00055000 08:11 20318114   /home/midas/online/test/frontend
0809e000-080a3000 rw-p 00056000 08:11 20318114   /home/midas/online/test/frontend
080a3000-080c5000 rw-p 00000000 00:00 0 
0835f000-08380000 rw-p 00000000 00:00 0          [heap]
b7881000-b7884000 rw-p 00000000 00:00 0 
b7893000-b7895000 rw-p 00000000 00:00 0 
bf938000-bf94d000 rw-p 00000000 00:00 0          [stack]
Aborted

Please help me figure out what's going wrong!

Thank you,
Jon

At DANCE we have a similar issue.  We are still doing "software
handshaking" between multiple frontends (15 which read data, and 16th
with direct accessto the trigger logic), and we apply a time stamp
using gettimeofday().  We use the regular mevb, sorting on serial number.

In the analyzer (MIDAS or ROME) we then keep a big circular buffer of
event fragments, which are rebuilt into new events based on the time stamp
obtained from gettimeofday().  We keep the system clocks synchronized
(often to within about 1ms) using ntp (need to average over several
ntp servers to avoid issues with network noise).  ntp can take a while
to stabilize, so we never reboot our computers... (well almost never).
We have a slow control frontend which monitors the ntp time offsets and
put's them in the history system for easy visualization.

Occasionally we seem to get in a mess, but somehow this fixes itself on
the next run, so it has been a useable system.  Maybe one day we will
get hardware handshaking between the frontend computers and the trigger
logic, but in the meantime we are taking data.

John.

Hello,
      I'm trying to build khyt1331 under scientific linux 5.5, kernel
2.6.18-238.9.1el5.  Has anyone succeeded with this.  So far, I've
managed to compile by hacking all the references to man9 pages out
of the makefile.  I've then hand installed the kernel driver with 
insmod.  cat /proc/khyt1331 produces 
Hytec 5331 card found at address 0xE800, using interrupt 10
Device not in use
CAMAC crate 0: responding
CAMAC crate 1: not responding
CAMAC crate 2: not responding
CAMAC crate 3: not responding 
and the "addr" LED blinks - so progress of some sort.  
There's no sign of /dev/camac.

Next up I'm going to compile stuff like camactest.c - though the 
makefiles in the drivers folder don't mention these, so I'll have to
work through what is needed by hand.   

At some point I'll have to rewrite a bit so that it all load automatically.

Any hints or tips greatfully received.

John McMillan

For a long time the "de facto" standard was to spread a package around in many
directories under /usr/local.  This proved to be a bad idea, as removing the
package
became very difficult.

With POSIX there is a written standard, which says that each pacakge goes in
it's own
directory under /opt. eg. /opt/midas.  Each package gets to define it's own
structure
within that directory.  One could imagine several versions installed at the
same time
/opt/midas/v1.9.2 and /opt/midas/v1.9.4 each with a bin, lib include etc. 
Following the
ROOT example, you could make a link from /opt/midas/pro to
/opt/midas/v1.9.4, so that
system files and login files are easy to maintain etc.  The basic idea is

MIDASSYS=/opt/midas/pro
PATH=$PATH:$MIDASSYS/bin

though a more sophisticated approach is

MIDASSYS=/opt/midas/pro
echo $PATH | grep -q $MIDASSYS || PATH=$PATH:$MIDASSYS/bin

where the assignment line (Bourne shell, and BASH shell) ensures
that multiple entries are not added on the PATH even if the script is more
than once.

POSIX also goes on to say that links from /opt/bin can be made if desired. 
I find this
usefull if a package has only one or two executables, and I don't to make
multiple
versions available.

I hope that the POSIX ideas are usefull,

John.

> > Here's my suggestion
> > MIDASSYS=/opt/midas-1.9.4 (for example)   
> 
> I guess we should follow the "standard" as much as possible. MIDASSYS was
inspired by
> ROOTSYS. Now where do people usually install ROOT? Is it /opt/root-x.x.x
or something
> else. Some years ago (when I did the last time some linux administration)
optional
> packages were put into /usr/local by default. I guess you have more
experience with
> today's tradition, so do whatever you thing is standard.
> 
> > so the Linux binaries would go to 
> > MIDASHOST=i386-pc-linux-gnu
> > $MIDASSYS/$MIDASHOST/bin
> > $MIDASSYS/$MIDASHOST/lib
> 
> Does that mean that the path has to be modified to include
$MIDASSYS/$MIDASHOST/bin?
> If we put a link to /usr/local/bin, the path does not have to be modified.
What about
> shared libraries? Does ldconfig know about /usr/local/lib, or
$MIDASYS/$MIDASHOST/lib?
> 
> > and the shared stuff would go to 
> > $MIDASSYS/include
> > $MIDASSYS/share/drivers
> > $MIDASSYS/share/examples
> 
> What about /usr/share? Is that a common place for documentatino etc?
> 
> Thanks for your advice.
> 
> - Stefan

> The current h1_book() macros (and the previous example analyzer code) have an
> odd persistency problem: for example, the user wants to change some histogram
> limits, edits the h1_book() calls, rebuilds and restarts the analyzer, starts a
> new run, and observes that all histograms are filled using the old limits, his
> changes "did not take". The user panics, I get paged during the Holy Lunch Hour,
> everybody is unhappy.
> 
> This is what I think happens:
> 
> 1) analyzer starts
> 2) LoadRootHistgrams() loads old histograms from file

I can't get onto cvs@midas.psi.ch right now
(cvs update
cvs@midas.psi.ch's password: 
Permission denied, please try again.)

but when I changed LoadRootHistograms a few days ago I left it as:

    } else if (obj->InheritsFrom( "TH1")) {

      // still don't know how to do TH1s

so h1_book() is creating the first and only copy of the histograms.
I am able to create new histogram limits.
I don't get the memory leak problems.

However I have seen the memory leak problems before, and they are real.
They must be dealt with either by (1) first deleteing the old histogram
or (2) ensuring that histogram names are unique in the whole application
(different modules/folders can not use the same histogram names).

I will return to this once I can do a cvs update for midas.

John.

> 3) user code calls h1_book()
> 4) h1_book template in midas.h does this (roughly):
>       hist = (TH1X *) gManaHistosFolder->FindObjectAny(name);
>       if (hist == NULL) {
>          hist = new TH1X(name, title, bins, min, max);
> 5) since the histogram already exists (loaded from the file, with the old
> limits), the TH1X constructor is not called at all, new histogram limits are
> utterly ignored.
> 
> A possible solution is to unconditionally create the ROOT objects, like I do in
> the example code posted at <a
href="http://dasdevpc.triumf.ca:9080/Midas/191">http://dasdevpc.triumf.ca:9080/Midas/191</a>.
That code
> produces an annoying warning from ROOT about possible memory leaks. This could
> be fixed by adding a two liner to "find and delete" the object before it is
> created, trippling the number of user code lines per histogram (find & delete,
> then create). Highly ugly.
> 
> midas.h macros (h1_book & co) can be fixed by adding checks for histogram limits
> and such, but I would much prefer a generic solution/convention that would work
> for arbitrary ROOT objects without MIDAS-specific wrappers (think TProfile,
> TGraph, etc...).
> 
> Any suggestions?
> 
> K.O.

So now that cvs is reachable again I have confirmed that
the code segment
 
     } else if (obj->InheritsFrom( "TH1")) {
 
       // still don't know how to do TH1s

is indeed still present.
If you want me to look at this some more, you need to provide some code to exhibit the problem.

John.

> > The current h1_book() macros (and the previous example analyzer code) have an
> > odd persistency problem: for example, the user wants to change some histogram
> > limits, edits the h1_book() calls, rebuilds and restarts the analyzer, starts a
> > new run, and observes that all histograms are filled using the old limits, his
> > changes "did not take". The user panics, I get paged during the Holy Lunch Hour,
> > everybody is unhappy.
> > 
> > This is what I think happens:
> > 
> > 1) analyzer starts
> > 2) LoadRootHistgrams() loads old histograms from file
> 
> I can't get onto cvs@midas.psi.ch right now
> (cvs update
> cvs@midas.psi.ch's password: 
> Permission denied, please try again.)
> 
> but when I changed LoadRootHistograms a few days ago I left it as:
> 
>     } else if (obj->InheritsFrom( "TH1")) {
> 
>       // still don't know how to do TH1s
> 
> so h1_book() is creating the first and only copy of the histograms.
> I am able to create new histogram limits.
> I don't get the memory leak problems.
> 
> However I have seen the memory leak problems before, and they are real.
> They must be dealt with either by (1) first deleteing the old histogram
> or (2) ensuring that histogram names are unique in the whole application
> (different modules/folders can not use the same histogram names).
> 
> I will return to this once I can do a cvs update for midas.
> 
> John.
> 
> > 3) user code calls h1_book()
> > 4) h1_book template in midas.h does this (roughly):
> >       hist = (TH1X *) gManaHistosFolder->FindObjectAny(name);
> >       if (hist == NULL) {
> >          hist = new TH1X(name, title, bins, min, max);
> > 5) since the histogram already exists (loaded from the file, with the old
> > limits), the TH1X constructor is not called at all, new histogram limits are
> > utterly ignored.
> > 
> > A possible solution is to unconditionally create the ROOT objects, like I do in
> > the example code posted at <a
> href="<a
href="http://dasdevpc.triumf.ca:9080/Midas/191">http://dasdevpc.triumf.ca:9080/Midas/191</a>">http://dasdevpc.triumf.ca:9080/Midas/191"><a
href="http://dasdevpc.triumf.ca:9080/Midas/191</a>">http://dasdevpc.triumf.ca:9080/Midas/191</a></a></a>.
> That code
> > produces an annoying warning from ROOT about possible memory leaks. This could
> > be fixed by adding a two liner to "find and delete" the object before it is
> > created, trippling the number of user code lines per histogram (find & delete,
> > then create). Highly ugly.
> > 
> > midas.h macros (h1_book & co) can be fixed by adding checks for histogram limits
> > and such, but I would much prefer a generic solution/convention that would work
> > for arbitrary ROOT objects without MIDAS-specific wrappers (think TProfile,
> > TGraph, etc...).
> > 
> > Any suggestions?
> > 
> > K.O.

> (preliminary, untested. I will keep this updated as I get testing feedback)
> 
> With recent changes to mana.c, creation of user ROOT objects in the MIDAS
> analyser has changed. Here is the new example code for creating ROOT objects
> that are visible in ROODY and are saved into the histogram file.
> 
> 1) in the "global" context (outside of any function)
> 
> #include <TH1D.h>
> #include <TProfile.h>
> 
> static TH1D* gMyHist1 = 0;
> static TProfile* gMyHist2 = 0;
> 
> 2) In the analyzer "init" or "begin run" method, create the histogram:
> 
> //extern TFolder *gManaHistosFolder; // from midas.h
> gMyHist1 = new TH1D("gMyHist1",...);
> gMyHist2 = new TProfile("gMyHist2",...);
> gManaHistosFolder->Add(gMyHist1);
> gManaHistosFolder->Add(gMyHist2);
> 
> (note: this will produce an warning about "possible memory leak")
> 
> 3) In the per-event method, fill the histograms
> 
> gMyHist1->Fill(x);
> gMyHist2->Fill(x,y);
> 
> K.O.


the book functions provide a convenient place to check against object duplication
and memory leaks etc., and a place to ensure that consistent subfolders are being
used.  eg. a while back we decided that TCutGs should be in a "cuts" subfolder.

To extend the booking to TProfile is fairly easy.  In fact if you want to
use the simple constructor TProfile::TProfile (const char *, const char *, Int_t,
Axis_t, Axis_t), then you could infact just use h1_book<TProfile>.

It now seems to me that the names h1_book, h2_book, cut_book are all too long
and even more upsetting are inconsistent.  Some of them are templates (most) and
some are not.  Perhaps they should all be templates, and all have the same name.
The attached patch accomplishes this (without deleting the old names).  With this
patch you can now do

gMyHist1 = book<TProfile>( "gMyHist2",...);

New book templates are needed when you (1) wish to change the subfolder, or (2)
need to use a different argument list in the constructor.  If you need help with
this for the TProfile constructors which are different from TH1D constructors then
let me know.  They should be easy to do.

For TGraph at lot depends on how you want to initialise the data points.

is there a reason why histograms are not saved after a replay?

   /* save histos if requested */
   if (out_info.histo_dump && clp.online) {
                              ^^^^^^^^^^

perhaps the && should be ||?

We stopped using tapes at Los Alamos a while ago.  The model we use is:

write data with mlogger to a local RAID system.  This is NFS mounted read only on teh analysis machines, and
becomes the working copy for most tasks.  Copy data to external hardrives.  We have been using USB.  The USB
system is sometime a little flaky (lnux 2.4.21-7, so we have a computer dedicated to this task.  The USB driver
can be reloaded, or if the user is not so knowledgeable, the copmuter can be rebooted.  users on this computer
have sudo privs, so they can format hard drives.  The disks are inserted into boxes while in use, and stored on
a shelf for data archival, so we don't have a lot of enclosures.

I use the automounter to mount and unmount the drives.  With a 10 second timeout, the user needs only to wait a
few seconds before unplugging the disk.  (cat /proc/mounts allows them to check if they want.) dmesg allows
them to find the drive letter.  This works for any device which appears later as a SCSI disk.  The automounter
manages /mnt/usb for vfat formatted devices, and /mnt/usbl for ext3 formatted devices (preferred for data
archiving).

autofs config files are:

/etc/auto.usb

# This is an automounter map and it has the following format
# key [ -mount-options-separated-by-comma ] location
# Details may be found in the autofs(5) manpage
 
*       -fstype=auto,nosuid,nodev,umask=0000,noatime    :/dev/&

/etc/auto.usbl

# This is an automounter map and it has the following format
# key [ -mount-options-separated-by-comma ] location
# Details may be found in the autofs(5) manpage
 
*       -fstype=auto,nosuid,nodev       :/dev/&

/etc/auto.master contains

/mnt/usb                /etc/auto.usb  --timeout=10
/mnt/usbl               /etc/auto.usbl --timeout=10


John.

> At triumf, we are developing a system to use removable hard drives to store data collected by midas 
> daq stations. The basic idea is to replace storage on 300 GB DLT tapes with storage on removable 
> esata, usb2 or firewire 750 GB hard drives.
> 
> To minimize culture shock, we stay as close as possible to the "tape" paradigm. Two removable disks 
> are used in tandem. Data is written to the first removable disk until it is full. Then midas automatically 
> switches to the second disk and asks the operator to replace the full disk with a blank disk. Similar to 
> handling tapes, the operator takes the full disk and stores it on the shelf (offline); takes a blank disk 
> and connects it to the computer. To read data from one of the disks, the operator takes the disk from 
> the shelf and connects it to the daq computer or to some other computer equipped with a compatible 
> removable storage bay. The full data disks are mounted read-only to prevent accidental data 
> modifications.
> 
> Two pieces of software are needed to implement this system:
> 
> 1) midas support for switching to alternate output disks as they become full. Data could be written to 
> the removable disk directly by the mlogger (no extra data copy on local disks) or by the lazylogger 
> (mlogger writes the data to the local disk, then the lazylogger copies it to the removable disk). Writing 
> directly to the removable disk is more efficient as it avoids the one extra data copy operation by the 
> lazylogger.
> 
> 2) a user interface utility for mounting and dismounting removable disks. Handling of removable disks 
> cannot be fully automatic: before unplugging a removable disk, the user has to inform the system; after 
> connecting a removable disk, the user has to tell the system to mount it read-only (for existing data), 
> read-write (to add more data) or to initialize a blank disk (fdisk+mkfs). (Also, some SATA interfaces do 
> not implement automatic hot-plug: they have to be manually told "please look for new disks").
> 
> We are presently evaluating various internal SATA hot-plug enclosures. We evaluated external eSATA 
> and USB2 enclosures and decided not to use them: while the performance is adequate, presence of 
> extra bulky components (eSATA and USB cables, non-standardized power bricks) and the extra cost of 
> eSATA and USB hard drive enclosures makes them unattractive.
> 
> I am open to suggestions and comments. I am most interested in hearing which data path (mlogger or 
> the lazylogger) would be most useful for other users.
> 
> K.O.

At Los Alamos, we have 15+1 frontends - the 15 between them read about 2 or 3
TB/hour and reduce it to 1 to 5 GB/hour which is then sent to the mevb on a 17th
computer.  The 16th frontend handles deadtime issues and scalers (small data rate).

frontends are 1GHz pentium 3, and backend is 2.8GHz dual CPU with hyperthreading.
Interconnect is 100Mb ethernet from frontends to switch, and 1Gb ethernet from
switch to backend.

Our bottle neck is (a) compactPCI backplane reading data from waveform digitizers
to the frontend CPUs and (b) CPU power on the frontend CPUs to analyzer the waveforms.

John

I am in favor of tcp_wrappers.

tcp_wrappers is well understood.

It works well in combination with a firewall.

mhttpd hangs when our security folks scan us.  We are not allowed to block them
with a firewall, but we can use tcpwrappers.

Would it make sense to put the same mechanism on mserver?

the man page for libtcpwrappers.a (taken from the tcpwrappers7.6 tar ball) is
attached. And the output after running it through nroff -man.

The odb is too fragile for security.  It is not understood well enough by many
experimenters.

As you can see I am in favor of tcp_wrappers.  This is mainly because it is part
of an existing and tested security model.  I don't know about the windows
world, but as you can also see, I vote for using something that is already part
of the windows security model.  Here's an example of how well the integrated
security model works:

    if an person is part of an experiment I make sure they can ssh to the
    experiment's computer

    the same rules could provide them with web access

Second is that when a change is needed to the security model then it is easy to
keep it current.  What if somebody restores an old ODB?  What if they setup a
small test with a new ODB?

If mhttpd used tcp_wrappers, then all our machines here at LANL would already be
configured!  No need for
users to do any root access (though those that need it have it anyway).

John.

The following program handles regular directories in a file, or folders (ugh).
Most histograms are added bin by bin.

For scaler events it is convenient to see the counts as a function of time (ala
sclaer history plots in mhttpd).  If the histogram looks like a scaler plot versus
time, then new bins are added on to the end (or into the middle!) of the histogram.

All different versions of cuts are kept.

TTrees are not explicitly supported, so probably don't do the right thing...

John.

> Dear MIDAS users,
> 
> I want to add several .root-files produced by the MIDAS analyzer, in a fast 
> and convenient way. ROOT's hadd fails because it does not know how to treat 
> TFolders. I guess this problem is not unique to me, so I hope that somebody of 
> you might already have found a solution.
> 
> Why don't I just run "analyzer -r 1 10000"?
> We have taken lots of runs under (rapidly) varying conditions, so it would be 
> lots of "-r". And the analysis is quite involved, so rerunning all data takes 
> about one hour on a fast PC making this quite painful.
> Therefore, I would like to rerun all data only once, and then add the result 
> files depending on different criteria.
> 
> Of course, I tried to write a script that does the adding. But somehow it is 
> incredibly slow. And I am not the Master Of C++, too.
> 
> Is there any deeper reason for MIDAS using TFolders, not TDirectorys? ROOT's 
> hadd can treat TDirectory. Can I simply patch "my" MIDAS? Is there general 
> interest in a change like this? (Does anyone have experience with the speed of 
> hadd?)
> 
> Looking forward to comments from the Forum.
> 
> Cheers,
> 
> Randolf

some semi-random thoughts:

no templates strictly means you can't use std::string, std::vector etc.

printf is in any case part of C++ (#include <cstdio>), but std::ostreams can be faster (for std::cout, endl line causes buffer flushing, whereas "\n" does not flush the buffer but printf
always flushes the buffer), and formatting is possible (though very long winded).  printf does not allow to print things other than simple data, e.g. BANK_HEADER* bh; printf( "%?", *bh);

I've been writing all our DAQ code in C++ for a while now.

> > >
> > > With the removal of the requirement to make it possible to write MIDAS frontends in C, we can switch the MIDAS 
> > > default build to C++ and start using C++ features in the MIDAS API (std::string, std::vector, etc).
> > > 
> 
> C++ is a big animal. Obviously we want to use std::string, std::vector and similar improvements over plain C (we already use "//" for comments).
> 
> But in keeping with the Camel's nose fable (https://en.wikipedia.org/wiki/Camel%27s_nose), there are some parts of C++ we definitely do not want to use in MIDAS. Even the C++ FAQ talks 
> about "evil features", see https://isocpp.org/wiki/faq/big-picture#use-evil-things-sometimes
> 
> Here is my list of things to use and to avoid. Comments on this are very welcome - as everybody's experience with C++ is different (and everybody's experience is very valuable and very 
> welcome).
> 
> - std::string, std:vector, etc are in. I am already using them in the MIDAS API (midas.h)
> - extern "C" is out, everything has to be C++, will remove "extern "C"" from all midas header files.
> - exceptions are out, see https://stackoverflow.com/questions/1736146/why-is-exception-handling-bad
> - std::thread and std::mutex are in, at least for writing new frontends, but see discussion of "cannot use c++11". (maybe replace ss_mutex_xxx() with out own std::mutex look-alike).
> - heavy use of templates and heavy use of argument overloading is out - just by looking at the code, impossible to tell what function will be called
> - "auto" is on probation. I need to know if "auto v=f()" is an integer or a double when I write "auto w=v/2" or "auto w=v/2.0". see 
> https://softwareengineering.stackexchange.com/questions/180216/does-auto-make-c-code-harder-to-understand
> - unreadable gibberish is out (lambdas, etc)
> - C-style malloc()/free() is in. C++ new and delete are okey, but "delete[]" confuses me.
> - C-style printf() is in. C++ cout and "<<" gunk provide no way to easily format the output for easy reading.
> 
> K.O.

Hi All,

I'm testing MIDAS SVN rev-4113 on Scientific Linux 5.1 (i386) and the Analyzer 
can't start as Daemon. What I mean "can't" is that it stops running 
immediately without leaving any error messages. However, it can run offline or 
without becoming a Daemon. I have tested with ROOT 5.14e/5.16/5.18 and 
the "Experiment" example coming with MIDAS and this problem always happens. 
Any ideas?

Best Regards,
Jimmy

Hi All,

Is there any example code for using MIDAS with the CAEN VME-USB Bridge V1718? 
Thanks.

Regards,
Jimmy

Hi All,

I am testing the libraries provided by CAEN with the sample softwares in the 
bundle CD. The Windows sample program works fine, but I cannot get started with 
the Linux sample program. When I run CAENVMEDemo in Scientific Linux 5.1, it 
gives me a message "Error opening the device". I have followed the instructions 
in CAENVMElibReadme.txt: 
- compile and load the device driver v1718.ko
- install the library libCAENVME.so

Does anyone have any experience of using V1718 in Scientific Linux? Thanks.

Regards,
Jimmy


> Hi All,
> 
> Is there any example code for using MIDAS with the CAEN VME-USB Bridge V1718? 
> Thanks.
> 
> Regards,
> Jimmy