> Reading an FADC buffer often needs large buffer size, especially while several
> FADCs work together. I want to know how large a bank32 can support.

The "32" in bank32 means 32-bits, so the bank holds 2^32=4 GBytes, hope that is enough for your ADC. 
The "normal" bank has only a 16-bit header, so it can hold only 64 kBytes. But for small banks, the overhead 
is therefore smaller.

• ODBGet(path[, format]): This functions works now also with subdirectories in the ODB. The command ODBGet('/Runinfo') returns for example:

  1
  1
  1024
  0
  0
  0
  Mon Jun 20 09:40:14 2011
  1308588014
  Mon Jun 20 09:40:46 2011
  1308588046
  

  
  
• ODBGetRecord(path), ODBExtractRecord(key): While ODBGet can be used for subdirectories, an easier way is to use ODBGetRecord and ODBExtractRecord. The first function retrieves the subtree (record), while the second one can be used to extract individual items. Here is an example:

  result = ODBGetRecord('/Runinfo');
  run_number = ODBExtractRecord(result, 'Run number');
  start_time = ODBExtractRecord(result, 'Start time');
  

  
  
• ODBMGet(paths[, callback, formats]): This function ("Multi-Get") can be used to obtain ODB values from different paths in one call. The ODB paths have to be supplied in an array, the result is again an array. An optional callback routine might be supplied for asynchronous operation. Optional formats might be supplied if the resulting number should be formatted in a specific way. Here is an example:

     var req = new Array();
     req[0] = "/Runinfo/Run number";
     req[1] = "/Equipment/Trigger/Statistics/Events sent";
     var result = ODBMGet(req);
     run_number = result[0];
     events_sent = result[1];
  

  
  

• <Comment>comment</Comment>
  a comment for this XML file, for information only
  
  
• <ODBSet path="path">value</ODBSet>
  to set a value in the ODB
  
  
• <ODBInc path="path">delta</ODBInc>
  to increment a value in the ODB
  
  
• <RunDescription>Description</RunDescription>
  a run description which is stored under /Experiment/Run Parameters/Run Description.
  
  
• <Transition>Start | Stop</Transition>
  to start or stop a run
  
  
• <Loop n="n"> ... </Loop>
  to execute a loop n times. For infinite loops, "infinit" can be specified as n
  
  
• <Wait for="events | ODBvalue | seconds" [path="ODB path"]>x</Wait>
  wait until a number of events is acquired (testing /Equipment/Trigger/Statistics/Events sent), or until a value in the ODB exceeds x, or wait for x seconds.
  
  
• <Script [loop_counter="1"]>Script</Script>
  to call a script on the server side. Optionally, the loop counter(s) are passed to the script
  

> > In midas.c, the bk_delete function removes a bank by decrementing the total
> > event size and then copying the remaining banks into the location of the first
> > using memcpy from string.h.
> 
> 
> I confirm the documented difference between memcpy() and memmove() and I confirm the 
> questionable use of memcpy() in bk_delete(). I think it should be memmove(). I made it so in my copy 
> of midas, so this change will not be lost.
> 
> But I am not sure how to test it - I do not think I ever used bk_delete(). I will probably ponder upon 
> this and do a blind commit.
> 
> 
> K.O.

It cannot hurt to use memmove(), so please go ahead to commit the changes.

- Stefan

Hello,

I'm co-organizing the upcoming Real Time Conference, which covers also the field of data acquisition, so it might be interesting for people working 
with MIDAS. If you have something to report, you could also consider to send an abstract to this conference.  It will be nicely located in Berkeley, 
California. We plan excursions to San Francisco and to Napa Valley.

Best regards,
Stefan Ritt

---------------------------

18th Real Time Conference
June 11 – 15, 2012
Berkeley, CA

We invite you to the Hotel Shattuck Plaza in downtown Berkeley, California for
the 2012 Real-Time Conference (RT2012).   It will take place Monday, June 11
through Friday, June 15, 2012, with optional pre-conference tutorials Saturday
and Sunday, June 9-10.

Like the previous editions, RT2012 will be a multidisciplinary conference
devoted to the latest developments on realtime techniques in the fields of
plasma and nuclear fusion, particle physics, nuclear physics and astrophysics,
space science, accelerators, medical physics, nuclear power instrumentation and
other radiation instrumentation.

Abstract submission is open as of 18 January (deadline 2 March). Please visit

http://www.npss-confs.org/rtc/welcome.asp?flag=44675.77&Retry=1 to submit an

abstract.

Call for Abstracts 

RT 2012 is an interdisciplinary conference on realtime data acquisition and
computing applications in the physical sciences. These applications include:

* High energy physics 
* Nuclear physics 
* Astrophysics and astroparticle physics 
* Nuclear fusion 
* Medical physics 
* Space instrumentation 
* Nuclear power instrumentation 
* Realtime security and safety 
* General Radiation Instrumentation 

Specific topics include (but are certainly not limited to) the list shown below.
We welcome correspondence to see how your research fits our venue.   

Key Dates

* Abstract submission opened:  January 18, 2012 
* Abstract deadline:  March 2, 2012 
* Program available: April 2 

Suggested Topics

* Realtime system architectures 
* Intelligent signal processing 
* Programmable devices 
* Fast data transfer links and networks 
* Trigger systems 
* Data acquisition 
* Processing farms 
* Control, monitoring, and test systems 
* Upgrades 
* Emerging realtime technologies 
* New standards 
* Realtime safety and security 
* Feedback on experiences 

Contact Information

If you have a question or wish to opt in for occasional e-mail updates about
RT2012, send us a message at RT2012@lbl.gov. To view full conference
information, visit http://rt2012.lbl.gov/index.html

> Anyhow, if anybody can suggest such an advanced locking library it would be great. Will save me the 
> effort of writing one.

Hi Konstantin,

yes there is a good way, which I used during development of the buffer manager function. Put in each sm_xxx function a cm_msg(M_DEBUG, ...) to 
generate a debug system message. They go only into the SYSMSG ring buffer and thus are light weight and don't influence the timing much. You can 
keep odbedit open to see these messages, but there is also another way. You can write a little program which dumps the whole SYSMSG buffer, which 
you can call when the lock happens. You then look "backwards" in time and get all messages stored there, depending of the size of the SYSMSG buffer of 
course. Of course this only works if the lock does not happen on the SYSMSB buffer itself. In that case you have to produce M_LOG messages which are 
written to the logging file. This will influence the timing slightly (the file might grow rapidly) but you are independent of semaphores.

The interesting thing is that in the MEG experiment (9 Front-ends, Event Builder, Logger, Lazylogger, ....) we run for months without any lock up. So I 
might suspect it's caused in your case from a program only you are using.

Best regards,
Stefan

> I found the problem only appears when I run mhttpd in scripts, whether bash or python.
> And I'm quite sure that the MIDAS environments (e.g. PATH, MIDAS_EXPTAB, MIDASSYS, etc.)
> are set in such scripts. If I start mhttpd in an xterm with or without "-D", it works
> fine. So, what's the difference between invoking mhttpd directly and through a script?

When you start it with "-D", then mhttpd become a daemon. According to linux rules, it has to "cd /", so it lives in the 
root directory, in order not to block any NFS mount/unmount. If something with the path is not correct then, mhttpd 
cannot find mtransition then. Once I fixed that problem my moving mtransition to /usr/bin.

Stefan

Just for completeness: Attached is the VME transfer speed I get with the SIS3100/SIS1100 interface using 
2eVME transfer. This curve can be explained exactly with an overhead of 125 us per DMA transfer and a 
continuous link speed of 83 MB/sec.

> I agree. Somehow mhttpd cannot run mtransition. I am not super happy with this dependance on user $PATH settings and the inability to capture error messages 
> from attempts to start mtransition. I am now thinking in the direction of running mtransition code by forking. But remember that mlogger and the event builder also
> have to use mtransition to stop runs (otherwise they can dead-lock). So an mhttpd-only solution is not good enough...

The way to go is to make cm_transition multi-threaded. Like on thread for each client to be contacted. This way the transition can go in parallel when there are many frontend computers for example, which will speed up 
transitions significantly. In addition, cm_transition should execute a callback whenever a client succeeded or failed, so to give immediate feedback to the user. I think of something like implementing WebSockets in mhttpd for that (http://en.wikipedia.org/wiki/WebSocket).

I have this in mind since many years, but did not have time to implement it yet. Maybe on my next visit to TRIUMF?

Stefan

> > Just for completeness: Attached is the VME transfer speed I get with the SIS3100/SIS1100 interface using 
> > 2eVME transfer. This curve can be explained exactly with an overhead of 125 us per DMA transfer and a 
> > continuous link speed of 83 MB/sec.
> 
> What VME module is on the other end?
> 
> K.O.

The PSI-built DRS4 board, where we implemented the 2eVME protocol in the Virtex II FPGA. The same speed can be obtained with the commercial 
VME memory module CI-VME64 from Chrislin Industries (see http://www.controlled.com/vme/chinp1.html).

Stefan

> P.S. Observe the ever present unexplained event rate fluctuations between 130-140 event/sec.

An important aspect of optimizing your system is to keep the network traffic under control. I use GBit Ethernet between FE and BE, and make sure the switch 
can accomodate all accumulated network traffic through its backplane. This way I do not have any TCP retransmits which kill you. Like if a single low-level 
ethernet packet is lost due to collision, the TCP stack retransmits it. Depending on the local settings, this can be after a timeout of one (!) second, which 
punches already a hole in your data rate. On the MSCB system actually I use UDP packets, where I schedule the retransmit myself. For a LAN, 10-100ms timeout 
is there enough. The one second is optimized for a WAN (like between two continents) where this is fine, but it is not what you want on a LAN system. Also 
make sure that the outgoing traffic (lazylogger) uses a different network card than the incoming traffic. I found that this also helps a lot.

- Stefan

> Then I realized that I see a recursive call to rpc_execute(): rpc_execute() calls tr_stop() calls cm_yield() calls 
> ss_suspend() calls rpc_execute(). The second rpc_execute successfully completes, but leave corrupted 
> data for the original rpc_execute(), which happily crashes. At the moment of the crash, recursive call to 
> rpc_execute() is no longer visible.

This is really strange. I did not protect rpc_execute against recursive calls since this should not happen. rpc_server_receive() is linked to rpc_call() on the client side. So there cannot be 
several rpc_call() since there I do the recursive checking (also multi-thread checking) via a mutex. See line 10142 in midas.c. So there CANNOT be recursive calls to rpc_execute() because 
there cannot be recursive calls to rpc_server_receive(). But apparently there are, according to your stack trace.

So even if your patch works fine, I would like to know where the recursive calls to rpc_server_receive() come from. Since we have one subproces of mserver for each client, there should only 
be one client connected to each mserver process, and the client is protected via the mutex in rpc_call(). Can you please debug this? I would like to understand what is going on there. Maybe 
there is a deeper underlying problem, which we better solve, otherwise it might fall back on use in the future.

For debugging, you have to see what commands rpc_call() send and what rpc_server_receive() gets, maybe by writing this into a common file together with a time stamp.

SR

A new pipe compression has been implemented in mlogger thanks to Fedor Ignatov from BINP 
Novosibirsk. The way it works that the logger write into a pipe instead directly into a file. The pipe can 
then be connected to any compression program without the need to copile against any additional C 
library.

To use is, enter as the filename for example

|bzip2>run%05d.mid     (note the pipe '|' in front of the bzip2)

This way the data stream is run through the bzip2 program, which is known to have better compression 
ratio than gzip. Furthermore, the parallel version of bzip2 can be used, which spreads over all available 
CPU cures and speeds up compression almost linearly with the number of cores. This parallel version 
called pbzip2 can be found here:

http://compression.ca/pbzip2/

It can be easily compiled and installed. Using this method in the MEG experiment at PSI, we can compress 
our waveform data to 37% or it's original size (49% with gzip), and on 8 cores we get a compression rate 
of about 40 MBytes/sec (23 MBytes with gzip on a single core).

The disadvantage of that method is that one cannot see the compression ratio online, but this is not a big 
deal I guess. The new version has been committed as rev. 5324. 

/Stefan

> Hi,
> 
> I've written a python script that reads some data from a file and generates a
> .png image. I want to have a button on my MIDAS status page that:
> 
> - executes the script and waits for it to finish,
> - then displays the image
> 
> How can I do that? I tried using the sequencer to just execute the script every
> 30 seconds, but I can't get it to work, and it would be better to only execute
> the script on demand anyway. 
> 
> I also am having trouble getting image display to work. I have the ODB keys set:
> 
> [local:oven1:S]/Custom>ls
> Temperature Map&                /home/deap/ovendaq/online/index.html
> Images
> 
> [local:oven1:S]/Custom>ls Images/temps.png/           
> Background                      /home/deap/ovendaq/online/temps.png
> 
> And the HTML file is just this:
> <img src="temps.png">
> 
> But the image won't display. It shows a "broken" picture, and when I try to view
> it directly it says: Invalid custom page: Page not found in ODB.
> 
> Any help would be appreciated...
> 
> Thanks
> Shaun


If you use the "custom" image system, you need to use GIF images. mhttpd can dynamically create GIF 
images, 
with a background image and overlaid labels, bar graphs etc. But mhttpd just contains a GIF library to do 
that 
in memory, but no PNG library.

Actually I would recommend you not to use a script to create an image, but use the custom image system 
to 
display temperatures. In the attachment you see an page from our experiment which contains a 
background image (the greyish boxes), labels (white temperature boxes), bar graphs (blue level boxes) 
and history pages (left side). This is all dynamically created inside mhttpd using the custom page system 
without any external script. All you have to do is to get the temperatures and levels inside the ODB via the 
slow control system. If you want, I can send you the full code for that page.

Cheers,
Stefan

> Hi,
> 
> I had to apply the attached patch to convince SuSE Linux 12.2 to compile mana.c
> gcc version is "(SUSE Linux) 4.6.2"
> 
> Problem is that gz{write,close, etc.} expect a 1st argument of type gzFile (see
> zlib.h), whereas out_file is FILE*. In fact, out_file is a cast to FILE*, even
> in the case when we work on a gzfile (HAVE_ZLIB).
> 
> Could you please confirm that the patch is correct, and possibly apply it to trunk?
> 
> I haven't checked if mana works as advertised now.
> 
> Cheers,
> 
> 
> Randolf

I applied your patch to the trunk.

Best,
Stefan

The Linux thread functionality was introduced by Konstantin, so he might have a better idea about that.

What I usually do is a graceful thread shutdown just by a flag. Like

int stop_thread = 0;

INT f(void *param)
{
  for (int x = 0; x < 100; x++) {
    sleep(1);
    if (stop_thread) {
      // clean up things here...
      return 0;
    }
  }
  return 0;
}

int main()
{
 printf("creating thread\n");
 midas_thread_t thr = ss_thread_create(f, NULL);
 sleep(2);
 printf("killing thread\n");
 stop_thread = 1;
 sleep(2);
 printf("success\n");
 return 0;
}


This way I have a chance to clean up things in the thread, which otherwise I would not be able to.

> I've been trying to set up midas to create an automatic elog entry at the end of
> each run and I've run into a problem. I've setup an elog on our server which
> uses SSL and it seems that the melog provided by midas to create logbook entries
> doesn't know any SSL.
> 
> My solution to this was to copy the crypt.c from the elog package to the
> computer running midas and changed melog.c and the makefile to use SSL if a flag
> -s is used. Does this seem like a sensible solution or did I oversee the obvious
> and/or right way to do this?

Indeed melog.c is an old version of the elog.c utility in the elog package, which has not been maintained since a 
long time. Can't you just use the recent elog.c utility from the elog package?