> I deployed several Debian Linux boxes as the DAQ systems in our lab. But I
feel it's boring to build and install midas and its related softwares (such as
root) on each box.


Our solution at TRIUMF is to install such packages on a shared NFS filesystem
visible to all client computers. This works well for ROOT and but MIDAS we found
it nearly impossible to keep MIDAS versions in sync between different projects
and expiments, so each experiment uses it's own copy of MIDAS, usually located
in the experiment home directory ($HOME/packages/midas). Because we often need
to make local modifications to MIDAS sources (Makefile, etc), we do not
"install" MIDAS into non-user-writable /usr/local & etc.


> I use autoconf/automake


The promise (premise) of autoconf/automake is to "hide" system dependencies. The
scripts are supposed to automatically probe the build environment and construct
an appropriate Makefile.

In practice, the autotool scripts always have bugs and incorrect assumptions
about the build environment and only work well for a few standardized systems
(RHEL and Debian derivatives) where the differences are so trivial that
autotools is an overkill and a normal Makefile is adequate for the job.

In my experience, as soon as I try to build an autotool-ized package on anything
that does not look like RHEL or Debian, autotool scripts explode and have to be
debugged and kludged by hand. Anybody who has ever done that would agree with me
that one would rather hack the ugliest Makefile than any of the  autotool
generated gibberish.

And of course autotools have never handled cross-compilation in any reasonable
way. Since we do cross-compile MIDAS (for VxWorks and embedded Linux, see "make
crosscompile") a Makefile is required and it so happens that the same Makefile
also works for normal Linux and MacOS, thank you very much.



> Here are the installation:
> [*] executalbes -- /usr/lib/daq-midas/bin
> [*] library and objs -- /usr/lib/daq-midas/lib


Is this in violation of the LSB (or LFS)? I though they mandate that files
controlled by package manager should be /usr/bin/odbedit, /usr/lib64/libmidas.a,
etc (/usr/bin/midas/odbedit no permitted).


> gcc `mdaq-config --cflags` -c -o myfe.o myfe.c


Please check if your config scripts correctly handle the "-m32" and "-m64" flags
- we frequently cross-compile 32-bit MIDAS executables on 64-bit machines.


K.O.

1. Untar daq-midas_deb.tar.gz somewhere, assumming ~/Temp.
2. cd ~/Temp/daq-midas
3. svn co -r 5065 svn+ssh://svn@savannah.psi.ch/repos/meg/midas/trunk midas
4. svn co -r 68 svn+ssh://svn@savannah.psi.ch/repos/meg/mxml/trunk mxml
5. cp -rvp debian/autoconf/* ./
6. ./configure --help
7. ./configure <--options>
8. make && make install

1. untar daq-midas_deb.tar.gz somewhere, assuming ~/Temp.
2. cd ~/Temp/daq-midas
3. svn co -r 5065 svn+ssh://svn@savannah.psi.ch/repos/meg/midas/trunk midas
4. svn co -r 68 svn+ssh://svn@savannah.psi.ch/repos/meg/mxml/trunk mxml
5. dpkg-buildpackage -b -us -uc

• daq-midas-doc -- The documents and references
  
• daq-midas-root -- the midas runtime library and utilities built with root
  
• daq-midas-noroot -- the midas runtime library and utilities built without root
  
• daq-midas-dev-root -- the midas devel files (headers, objects, drivers, examples) built with root
  
• daq-midas-dev-noroot -- the midas devel files (headers, objects, drivers, examples) built without root
  

• executalbes -- /usr/lib/daq-midas/bin
  
• library and objs -- /usr/lib/daq-midas/lib
  
• headers -- /usr/lib/daq-midas/include
  
• sources and drivers -- /usr/share/daq-midas/
  
• docs and examples -- /usr/share/doc/daq-midas
  
• mdaq-config -- /usr/bin/mdaq-config
  

gcc `mdaq-config --cflags` -c -o myfe.o myfe.c
gcc `mdaq-config --libs` -o myfe myfe.o `mdaq-config --libdir`/mfe.o

• each midas experiment will be configured in a file named "mdaq.yaml"
  
• mdaq.py reads the configure file and prepare the daq environment, just like "examples/experiment/start_daq.sh"
  
• mdaq.py will handle "start/stop/restart/info" about the daq codes.
  

• <Comment>comment</Comment>
  a comment for this XML file, for information only
  
  
• <ODBSet path="path">value</ODBSet>
  to set a value in the ODB
  
  
• <ODBInc path="path">delta</ODBInc>
  to increment a value in the ODB
  
  
• <RunDescription>Description</RunDescription>
  a run description which is stored under /Experiment/Run Parameters/Run Description.
  
  
• <Transition>Start | Stop</Transition>
  to start or stop a run
  
  
• <Loop n="n"> ... </Loop>
  to execute a loop n times. For infinite loops, "infinit" can be specified as n
  
  
• <Wait for="events | ODBvalue | seconds" [path="ODB path"]>x</Wait>
  wait until a number of events is acquired (testing /Equipment/Trigger/Statistics/Events sent), or until a value in the ODB exceeds x, or wait for x seconds.
  
  
• <Script [loop_counter="1"]>Script</Script>
  to call a script on the server side. Optionally, the loop counter(s) are passed to the script
  

• ODBGet(path[, format]): This functions works now also with subdirectories in the ODB. The command ODBGet('/Runinfo') returns for example:

  1
  1
  1024
  0
  0
  0
  Mon Jun 20 09:40:14 2011
  1308588014
  Mon Jun 20 09:40:46 2011
  1308588046
  

  
  
• ODBGetRecord(path), ODBExtractRecord(key): While ODBGet can be used for subdirectories, an easier way is to use ODBGetRecord and ODBExtractRecord. The first function retrieves the subtree (record), while the second one can be used to extract individual items. Here is an example:

  result = ODBGetRecord('/Runinfo');
  run_number = ODBExtractRecord(result, 'Run number');
  start_time = ODBExtractRecord(result, 'Start time');
  

  
  
• ODBMGet(paths[, callback, formats]): This function ("Multi-Get") can be used to obtain ODB values from different paths in one call. The ODB paths have to be supplied in an array, the result is again an array. An optional callback routine might be supplied for asynchronous operation. Optional formats might be supplied if the resulting number should be formatted in a specific way. Here is an example:

     var req = new Array();
     req[0] = "/Runinfo/Run number";
     req[1] = "/Equipment/Trigger/Statistics/Events sent";
     var result = ODBMGet(req);
     run_number = result[0];
     events_sent = result[1];
  

  
  

Dear All,

Thanks Konstantin Olchanski for providing me a hint. The file can be opened now after I 
changed the line: 

file->gzfile = gzopen(file_name, "rb");

in function ma_open() in mana.c to the followings: 

INT fd = open(file_name, O_RDONLY | O_LARGEFILE);
if (fd <= 0)
   return NULL;

file->gzfile = gzdopen(fd, "rb");

No modifications to the Makefile is needed in this case.

Best Regards,
Jimmy


> Dear All,
> 
> I got a "Cannot open input file" error when I tried to analyze a .mid.gz file with 
> size over 5 GB on a 32-bit Linux. The error traced back to gzopen() in mana.c 
> where it returned NULL when opening the file. I understand that 32-bit Linux may 
> not be able to handle files with size over 2 GB. I tried to add -
> D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 to CFLAGS in the Makefile of MIDAS and 
> the analyzer, but I still got the same error. Is there any workarounds that enable 
> me to analyze large files on 32-bit systems?
> 
> p.s. The data file was also produced on a 32-bit Linux.
> 
> Thanks & Best Regards,
> 
> Jimmy

The HTTPS SSL certificate on ladd00.triumf.ca has been updated. Same as the old
certificate, the new one is self-signed and your web browser may complain about
that and ask you to "save a security exception".

When you save the new certificate, you can verify that you are connected to the
real ladd00.triumf.ca by comparing the "SHA1 fingerprint" reported by your web
browser to the one given below (as reported by "svn update"):

Certificate information:
 - Hostname: ladd00.triumf.ca
 - Valid: from Jun 17 23:36:35 2011 GMT until Jun 16 23:36:35 2012 GMT
 - Issuer: DAQ, TRIUMF, Vancouver, BC, CA
 - Fingerprint: 2a:be:9f:9f:70:d4:dc:72:9f:63:bf:4f:fe:c0:2c:8f:a8:29:f2:f1

K.O.

Dear All,

I got a "Cannot open input file" error when I tried to analyze a .mid.gz file with 
size over 5 GB on a 32-bit Linux. The error traced back to gzopen() in mana.c 
where it returned NULL when opening the file. I understand that 32-bit Linux may 
not be able to handle files with size over 2 GB. I tried to add -
D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 to CFLAGS in the Makefile of MIDAS and 
the analyzer, but I still got the same error. Is there any workarounds that enable 
me to analyze large files on 32-bit systems?

p.s. The data file was also produced on a 32-bit Linux.

Thanks & Best Regards,

Jimmy

> Reading an FADC buffer often needs large buffer size, especially while several
> FADCs work together. I want to know how large a bank32 can support.

Limitations in order:

- bank32 size is limited to a 32 bit integer size (about 4000 Gbytes)
- bank size is limited by event size
- event size in a midas mfe.c based frontend is limited to the value of
max_event_size set by the user
- maximum event size that can go through the MIDAS event buffer system is limited
to ODB value /Experiment/MAX_EVENT_SIZE (MAX_EVENT_SIZE in midas.h does not do
anything now)
- maximum event size is limited to *half* the size of the SYSTEM shared memory
event buffer (or any other buffers that the event has to go through)
- default size of the SYSTEM buffer is 8 Mbytes set by ODB /Experiment/Buffer
sizes/SYSTEM. This limits maximum event size to about 4 Mbytes.
- size of SYSTEM buffer can be increased to arbitrary size, but in practice no
bigger than the amount of computer physical memory minus space needed for running
the frontend program and the mlogger, which also allocate buffer space to hold 1
event of maximum size.

So for a computer with 8 Gbytes of RAM, you can make the SYSTEM buffer size 4
GBytes, set ODB MAX_EVENT_SIZE to 1 Gbyte, and in theory, you should be able to
write 1 Gbyte events from your frontend to disk.

In practice, I think the biggest events I have seen go through a MIDAS system are
non-compressed waveforms in the T2K/ND280 FGD and TPC detectors, about 4 Mbytes of
data per event.

Other considerations (rules of thumb):

1) the SYSTEM event buffer should be big enough to hold 10-100 events.
2) the SYSTEM event buffer should be big enough to hold about 5-10 seconds of data
- i.e. if your event size is 1 Mbyte and data rate is 1 Hz, 10 seconds of data
will be 1Mbyte*1Hz*10sec = 10 Mbytes.

This is because the SYSTEM buffer decouples the real-time activity of the frontend
program from non-real-time activity of writing data to storage devices.

K.O.

I'm brand new to MIDAS, and C system programming in general, so please be
gentle. I've compiled and installed MIDAS 2.3.0 on Ubuntu 10.04 LTS. I've built
the kernel driver, khyt1331.ko and installed it. It appears to be working, since
the camactest and esonetest programs included with the driver work just fine.

So I attempted to build the example experiment distributed with MIDAS, with the
following changes to the Makefile:

DRV_DIR   = $(MIDASSYS)/drivers/kernel/khyt1331_26

and

DRIVER = camac

The programs build without error but when I try to start the frontend, I get:

$ ./frontend
Frontend name          :     CSM-Nuclear Portable DAQ Frontend
Event buffer size      :     1000000
User max event size    :     10000
User max frag. size    :     5242880
# of events per buffer :     100

Connect to experiment...
*** buffer overflow detected ***: ./frontend terminated
======= Backtrace: =========
/lib/tls/i686/cmov/libc.so.6(__fortify_fail+0x50)[0x6de390]
/lib/tls/i686/cmov/libc.so.6(+0xe12ca)[0x6dd2ca]
/lib/tls/i686/cmov/libc.so.6(__strcpy_chk+0x44)[0x6dc644]
./frontend[0x805611f]
./frontend[0x806f656]
./frontend[0x8053d82]
/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x612bd6]
./frontend[0x804bb81]
======= Memory map: ========
00110000-0012d000 r-xp 00000000 08:05 7471187    /lib/libgcc_s.so.1
0012d000-0012e000 r--p 0001c000 08:05 7471187    /lib/libgcc_s.so.1
0012e000-0012f000 rw-p 0001d000 08:05 7471187    /lib/libgcc_s.so.1
00264000-00277000 r-xp 00000000 08:05 7603242    /lib/tls/i686/cmov/libnsl-2.11.1.so
00277000-00278000 r--p 00012000 08:05 7603242    /lib/tls/i686/cmov/libnsl-2.11.1.so
00278000-00279000 rw-p 00013000 08:05 7603242    /lib/tls/i686/cmov/libnsl-2.11.1.so
00279000-0027b000 rw-p 00000000 00:00 0 
002db000-002dd000 r-xp 00000000 08:05 7603265   
/lib/tls/i686/cmov/libutil-2.11.1.so
002dd000-002de000 r--p 00001000 08:05 7603265   
/lib/tls/i686/cmov/libutil-2.11.1.so
002de000-002df000 rw-p 00002000 08:05 7603265   
/lib/tls/i686/cmov/libutil-2.11.1.so
003b1000-003c6000 r-xp 00000000 08:05 7603257   
/lib/tls/i686/cmov/libpthread-2.11.1.so
003c6000-003c7000 r--p 00014000 08:05 7603257   
/lib/tls/i686/cmov/libpthread-2.11.1.so
003c7000-003c8000 rw-p 00015000 08:05 7603257   
/lib/tls/i686/cmov/libpthread-2.11.1.so
003c8000-003ca000 rw-p 00000000 00:00 0 
004ea000-004f1000 r-xp 00000000 08:05 7603261    /lib/tls/i686/cmov/librt-2.11.1.so
004f1000-004f2000 r--p 00006000 08:05 7603261    /lib/tls/i686/cmov/librt-2.11.1.so
004f2000-004f3000 rw-p 00007000 08:05 7603261    /lib/tls/i686/cmov/librt-2.11.1.so
005fb000-005fc000 r-xp 00000000 00:00 0          [vdso]
005fc000-0074f000 r-xp 00000000 08:05 7603231    /lib/tls/i686/cmov/libc-2.11.1.so
0074f000-00750000 ---p 00153000 08:05 7603231    /lib/tls/i686/cmov/libc-2.11.1.so
00750000-00752000 r--p 00153000 08:05 7603231    /lib/tls/i686/cmov/libc-2.11.1.so
00752000-00753000 rw-p 00155000 08:05 7603231    /lib/tls/i686/cmov/libc-2.11.1.so
00753000-00756000 rw-p 00000000 00:00 0 
00783000-00796000 r-xp 00000000 08:05 7471302    /lib/libz.so.1.2.3.3
00796000-00797000 r--p 00012000 08:05 7471302    /lib/libz.so.1.2.3.3
00797000-00798000 rw-p 00013000 08:05 7471302    /lib/libz.so.1.2.3.3
008ab000-008c6000 r-xp 00000000 08:05 7471129    /lib/ld-2.11.1.so
008c6000-008c7000 r--p 0001a000 08:05 7471129    /lib/ld-2.11.1.so
008c7000-008c8000 rw-p 0001b000 08:05 7471129    /lib/ld-2.11.1.so
008e4000-00908000 r-xp 00000000 08:05 7603239    /lib/tls/i686/cmov/libm-2.11.1.so
00908000-00909000 r--p 00023000 08:05 7603239    /lib/tls/i686/cmov/libm-2.11.1.so
00909000-0090a000 rw-p 00024000 08:05 7603239    /lib/tls/i686/cmov/libm-2.11.1.so
08048000-0809d000 r-xp 00000000 08:11 20318114   /home/midas/online/test/frontend
0809d000-0809e000 r--p 00055000 08:11 20318114   /home/midas/online/test/frontend
0809e000-080a3000 rw-p 00056000 08:11 20318114   /home/midas/online/test/frontend
080a3000-080c5000 rw-p 00000000 00:00 0 
0835f000-08380000 rw-p 00000000 00:00 0          [heap]
b7881000-b7884000 rw-p 00000000 00:00 0 
b7893000-b7895000 rw-p 00000000 00:00 0 
bf938000-bf94d000 rw-p 00000000 00:00 0          [stack]
Aborted

Please help me figure out what's going wrong!

Thank you,
Jon

> Reading an FADC buffer often needs large buffer size, especially while several
> FADCs work together. I want to know how large a bank32 can support.

The "32" in bank32 means 32-bits, so the bank holds 2^32=4 GBytes, hope that is enough for your ADC. 
The "normal" bank has only a 16-bit header, so it can hold only 64 kBytes. But for small banks, the overhead 
is therefore smaller.

Reading an FADC buffer often needs large buffer size, especially while several
FADCs work together. I want to know how large a bank32 can support.

I just committed a MIDAS example for using most mhttpd html and javascript functions: ODBGet(), 
ODBSet(), ODBRpc() & co.

Please checkout .../midas/examples/javascript1, and follow instructions in the README file.

For your enjoyment, the example html file is attached to this message.

(to use the function ODBRpc_rev1 - the midas rpc call that returns data to the web page - you need 
mhttpd svn rev 4994 committed today. Other functions should work with any revision of mhttpd)

svn rev 4992.
K.O.

> uhm, mine might be completely unrelated to this, but it just so happened that the rev.
> 4936 was one that was used in a recent experiment, in which there was complaints about
> the responsiveness of the history plots. The history plots would take up to 30 seconds
> to respond, if the run was about 30-40 minutes old. When the run is about  < 10 minutes
> old , the history plot was responsive to within 1-2 seconds.

Ah, that rings a bell. How big are your history files on disk? How much RAM do you have?

What I see in our experiment is that linux buffers everything I write to disk in a cache 
located in RAM. But this cache is limited, so after a certain time it's overwritten. Now 
this is handled by the OS, the application (mlogger in this case) has no influence on it. 
Let's say you write 5 MB/minute of history, and your cache is 50 MB large. Then after 10 
minutes you can still read the history data from the RAM cache which is ~10x faster than 
your disk. But your older history data (30-40 min) is flushed out of the cache and has to be 
re-read from disk. A typical symptom of this is that the first time you display this it 
takes maybe 30 seconds, but if you do a "reload" of your page it goes much faster. In that 
case the contents is cached again in RAM. If you observe this, you can almost be certain 
that you see th "too small RAM cache" problem. In that case just add RAM and things should 
run better (I use 16 GByte in my machine).

Best regards,

  Stefan

> > 
> > Do you mind giving little more detail? We might have the same issue, where we got
> > complaints that midas history stops working after a certain time.
> > 
> 
> 
> Yes, please do supply more information. What problems do *you* see?
> 
> 
> K.O.

Hi.

uhm, mine might be completely unrelated to this, but it just so happened that the rev.
4936 was one that was used in a recent experiment, in which there was complaints about
the responsiveness of the history plots. The history plots would take up to 30 seconds
to respond, if the run was about 30-40 minutes old. When the run is about  < 10 minutes
old , the history plot was responsive to within 1-2 seconds.

I received rather limited information regarding this problem. So hence my apprehension
on stating it as a *problem* or bug. It could be something related to hardware/beam etc.

Lee

> My torture test runs okey in my mac now, one remaining problem is spurious client removal caused
> by semaphore starvation...

My torture test runs okey on Linux and I do not see any problems with spurious client removal - actually
I do not see any strange longs waits for semaphores that I was seeing on MacOS. Must be another
proof that MacOS is years behind Linux in kernel technology (but parsecs ahead in user experience)

K.O.