Just some idea:

The ultimate solution to that would be to do that completely JavaScript driven. You load ONCE the list of all 
variables into a local array, then sort this into your history panel LOCALLY. When I did the original mhttpd 
history config page, there was not much JavaScript around, but today this would be the ultimate option. It 
even supports drag-and-drop. So let's keep that in mind for the future.

- Stefan

> We had javascript ODBGet() and ODBSet() functions for some time now, permitting 
implementation of 
> "page-reload-free" "self-updating" web pages. I finally got around to put all 
the javascript bits together 
> to actually implement such a page.

Unfortunately the page has tons of JavaScript errors, probably happened during 
copy-and-paste to elog. Note that 
such files are better places as attachment. I attached a screen dump from the JS 
debugger inside Chrome which I 
use normally to debug JS.

Please find attached another example of a JavaScript (JS) page using the 
ODBGet/Set functions. 

In contrast to the previous posting, the page is not constructed via the 
document.writeln() function, but written directly in HTML and modified through the 
"innerHTML = ..." functionality. 

It is a control page for our beamline, which gets updated in the background. In 
addition, the user can set the beamline to three predefined settings which are 
stored in an array at the top of the page. As an little extra there is a progress 
bar, which is updated locally via JS since changing the beamline takes a while. 
The progress bar is implemented as a table with variable width, and dynamically 
updated by the JS program. The second attachment is a screen dump from such a 
switching process. Since only values in the ODB are changed, you can try it 
yourself without actually modifying a PSI beam line ;-)

> The attached errors all seem to be from cut-and-paste line breaks in the long "document.writeln()" statements. 
> When the page runs, there are no errors from Firefox and Safari.

Then it would be good if you re-submit the file as an attachment so that other people can use it.

> This example uses "document.writeln()" because the number of PostAmp devices displayed in the table is not 
> known in advance and is potentially read from ODB at page load time.

This was not a criticism but just to show that there are different ways of constructing such a page, depending on the 
needs. So people have the choice. Anyhow I think it's very good to have some working examples for people to start 
with.

> very recent mhttpd mangles spaces in URL encoding-decoding and I cannot create or delete entries in for 
> example "/experiment/edit on start". For example attempt to delete "/experiment/Pedestals Run" 
> produces:
> <h1>Cannot find key Experiment/edit%20on%20start/Pedestals run</h1>
> (notice "%20" instead of spaces. further navigation sometimes replaces the "%" sign with "%25" making it 
> even more mangled)
> 
> this used to work. looks like a call to URL unmangling went missing somewhere.
> K.O.

Thanks for reporting. Fixed in SVN revision 4882. Actually I outcommented the fix some time ago and forgot to 
put it back. Now I hope that this does not blow anything else...

- Stefan

midas\mscb\embedded
  \include                <- place for framework include file mscbemb.h
  \lib                    <- precompiled TCP/IP library for SCS-260 submaster
  \src                    <- framework sources mscbmain.c and mscbutil.c
  \<project1>             <- separate folder for project1
      config.h            <- config file for project1
  \<project2>             <- separate folder for project2
      config.h            <- config file for project2
  ...
  \experiment
     \<experiment1>
     \<experiment2>

• The framework files mscbmain.c mscbutil.c and mscbemb.h have moved and therefore they have to be re-added to the projects in the Keil MicroVision Development Environment.
  
• The name of the device should not be defined under compiler settings (Project Options/C51/Preprocessor Symbols), but put directly into the config.h file associated with the project.
  
• The include paths in the compile have to be changed and point to \midas\mscb\embedded\include
  
• The file config.h has to be copied from a similar project and adjusted to fit the new project.
  

> uhm, mine might be completely unrelated to this, but it just so happened that the rev.
> 4936 was one that was used in a recent experiment, in which there was complaints about
> the responsiveness of the history plots. The history plots would take up to 30 seconds
> to respond, if the run was about 30-40 minutes old. When the run is about  < 10 minutes
> old , the history plot was responsive to within 1-2 seconds.

Ah, that rings a bell. How big are your history files on disk? How much RAM do you have?

What I see in our experiment is that linux buffers everything I write to disk in a cache 
located in RAM. But this cache is limited, so after a certain time it's overwritten. Now 
this is handled by the OS, the application (mlogger in this case) has no influence on it. 
Let's say you write 5 MB/minute of history, and your cache is 50 MB large. Then after 10 
minutes you can still read the history data from the RAM cache which is ~10x faster than 
your disk. But your older history data (30-40 min) is flushed out of the cache and has to be 
re-read from disk. A typical symptom of this is that the first time you display this it 
takes maybe 30 seconds, but if you do a "reload" of your page it goes much faster. In that 
case the contents is cached again in RAM. If you observe this, you can almost be certain 
that you see th "too small RAM cache" problem. In that case just add RAM and things should 
run better (I use 16 GByte in my machine).

Best regards,

  Stefan

> Reading an FADC buffer often needs large buffer size, especially while several
> FADCs work together. I want to know how large a bank32 can support.

The "32" in bank32 means 32-bits, so the bank holds 2^32=4 GBytes, hope that is enough for your ADC. 
The "normal" bank has only a 16-bit header, so it can hold only 64 kBytes. But for small banks, the overhead 
is therefore smaller.

• ODBGet(path[, format]): This functions works now also with subdirectories in the ODB. The command ODBGet('/Runinfo') returns for example:

  1
  1
  1024
  0
  0
  0
  Mon Jun 20 09:40:14 2011
  1308588014
  Mon Jun 20 09:40:46 2011
  1308588046
  

  
  
• ODBGetRecord(path), ODBExtractRecord(key): While ODBGet can be used for subdirectories, an easier way is to use ODBGetRecord and ODBExtractRecord. The first function retrieves the subtree (record), while the second one can be used to extract individual items. Here is an example:

  result = ODBGetRecord('/Runinfo');
  run_number = ODBExtractRecord(result, 'Run number');
  start_time = ODBExtractRecord(result, 'Start time');
  

  
  
• ODBMGet(paths[, callback, formats]): This function ("Multi-Get") can be used to obtain ODB values from different paths in one call. The ODB paths have to be supplied in an array, the result is again an array. An optional callback routine might be supplied for asynchronous operation. Optional formats might be supplied if the resulting number should be formatted in a specific way. Here is an example:

     var req = new Array();
     req[0] = "/Runinfo/Run number";
     req[1] = "/Equipment/Trigger/Statistics/Events sent";
     var result = ODBMGet(req);
     run_number = result[0];
     events_sent = result[1];
  

  
  

• <Comment>comment</Comment>
  a comment for this XML file, for information only
  
  
• <ODBSet path="path">value</ODBSet>
  to set a value in the ODB
  
  
• <ODBInc path="path">delta</ODBInc>
  to increment a value in the ODB
  
  
• <RunDescription>Description</RunDescription>
  a run description which is stored under /Experiment/Run Parameters/Run Description.
  
  
• <Transition>Start | Stop</Transition>
  to start or stop a run
  
  
• <Loop n="n"> ... </Loop>
  to execute a loop n times. For infinite loops, "infinit" can be specified as n
  
  
• <Wait for="events | ODBvalue | seconds" [path="ODB path"]>x</Wait>
  wait until a number of events is acquired (testing /Equipment/Trigger/Statistics/Events sent), or until a value in the ODB exceeds x, or wait for x seconds.
  
  
• <Script [loop_counter="1"]>Script</Script>
  to call a script on the server side. Optionally, the loop counter(s) are passed to the script
  

> > In midas.c, the bk_delete function removes a bank by decrementing the total
> > event size and then copying the remaining banks into the location of the first
> > using memcpy from string.h.
> 
> 
> I confirm the documented difference between memcpy() and memmove() and I confirm the 
> questionable use of memcpy() in bk_delete(). I think it should be memmove(). I made it so in my copy 
> of midas, so this change will not be lost.
> 
> But I am not sure how to test it - I do not think I ever used bk_delete(). I will probably ponder upon 
> this and do a blind commit.
> 
> 
> K.O.

It cannot hurt to use memmove(), so please go ahead to commit the changes.

- Stefan

Hello,

I'm co-organizing the upcoming Real Time Conference, which covers also the field of data acquisition, so it might be interesting for people working 
with MIDAS. If you have something to report, you could also consider to send an abstract to this conference.  It will be nicely located in Berkeley, 
California. We plan excursions to San Francisco and to Napa Valley.

Best regards,
Stefan Ritt

---------------------------

18th Real Time Conference
June 11 – 15, 2012
Berkeley, CA

We invite you to the Hotel Shattuck Plaza in downtown Berkeley, California for
the 2012 Real-Time Conference (RT2012).   It will take place Monday, June 11
through Friday, June 15, 2012, with optional pre-conference tutorials Saturday
and Sunday, June 9-10.

Like the previous editions, RT2012 will be a multidisciplinary conference
devoted to the latest developments on realtime techniques in the fields of
plasma and nuclear fusion, particle physics, nuclear physics and astrophysics,
space science, accelerators, medical physics, nuclear power instrumentation and
other radiation instrumentation.

Abstract submission is open as of 18 January (deadline 2 March). Please visit

http://www.npss-confs.org/rtc/welcome.asp?flag=44675.77&Retry=1 to submit an

abstract.

Call for Abstracts 

RT 2012 is an interdisciplinary conference on realtime data acquisition and
computing applications in the physical sciences. These applications include:

* High energy physics 
* Nuclear physics 
* Astrophysics and astroparticle physics 
* Nuclear fusion 
* Medical physics 
* Space instrumentation 
* Nuclear power instrumentation 
* Realtime security and safety 
* General Radiation Instrumentation 

Specific topics include (but are certainly not limited to) the list shown below.
We welcome correspondence to see how your research fits our venue.   

Key Dates

* Abstract submission opened:  January 18, 2012 
* Abstract deadline:  March 2, 2012 
* Program available: April 2 

Suggested Topics

* Realtime system architectures 
* Intelligent signal processing 
* Programmable devices 
* Fast data transfer links and networks 
* Trigger systems 
* Data acquisition 
* Processing farms 
* Control, monitoring, and test systems 
* Upgrades 
* Emerging realtime technologies 
* New standards 
* Realtime safety and security 
* Feedback on experiences 

Contact Information

If you have a question or wish to opt in for occasional e-mail updates about
RT2012, send us a message at RT2012@lbl.gov. To view full conference
information, visit http://rt2012.lbl.gov/index.html

> Anyhow, if anybody can suggest such an advanced locking library it would be great. Will save me the 
> effort of writing one.

Hi Konstantin,

yes there is a good way, which I used during development of the buffer manager function. Put in each sm_xxx function a cm_msg(M_DEBUG, ...) to 
generate a debug system message. They go only into the SYSMSG ring buffer and thus are light weight and don't influence the timing much. You can 
keep odbedit open to see these messages, but there is also another way. You can write a little program which dumps the whole SYSMSG buffer, which 
you can call when the lock happens. You then look "backwards" in time and get all messages stored there, depending of the size of the SYSMSG buffer of 
course. Of course this only works if the lock does not happen on the SYSMSB buffer itself. In that case you have to produce M_LOG messages which are 
written to the logging file. This will influence the timing slightly (the file might grow rapidly) but you are independent of semaphores.

The interesting thing is that in the MEG experiment (9 Front-ends, Event Builder, Logger, Lazylogger, ....) we run for months without any lock up. So I 
might suspect it's caused in your case from a program only you are using.

Best regards,
Stefan

> I found the problem only appears when I run mhttpd in scripts, whether bash or python.
> And I'm quite sure that the MIDAS environments (e.g. PATH, MIDAS_EXPTAB, MIDASSYS, etc.)
> are set in such scripts. If I start mhttpd in an xterm with or without "-D", it works
> fine. So, what's the difference between invoking mhttpd directly and through a script?

When you start it with "-D", then mhttpd become a daemon. According to linux rules, it has to "cd /", so it lives in the 
root directory, in order not to block any NFS mount/unmount. If something with the path is not correct then, mhttpd 
cannot find mtransition then. Once I fixed that problem my moving mtransition to /usr/bin.

Stefan

Just for completeness: Attached is the VME transfer speed I get with the SIS3100/SIS1100 interface using 
2eVME transfer. This curve can be explained exactly with an overhead of 125 us per DMA transfer and a 
continuous link speed of 83 MB/sec.

> I agree. Somehow mhttpd cannot run mtransition. I am not super happy with this dependance on user $PATH settings and the inability to capture error messages 
> from attempts to start mtransition. I am now thinking in the direction of running mtransition code by forking. But remember that mlogger and the event builder also
> have to use mtransition to stop runs (otherwise they can dead-lock). So an mhttpd-only solution is not good enough...

The way to go is to make cm_transition multi-threaded. Like on thread for each client to be contacted. This way the transition can go in parallel when there are many frontend computers for example, which will speed up 
transitions significantly. In addition, cm_transition should execute a callback whenever a client succeeded or failed, so to give immediate feedback to the user. I think of something like implementing WebSockets in mhttpd for that (http://en.wikipedia.org/wiki/WebSocket).

I have this in mind since many years, but did not have time to implement it yet. Maybe on my next visit to TRIUMF?

Stefan

> > Just for completeness: Attached is the VME transfer speed I get with the SIS3100/SIS1100 interface using 
> > 2eVME transfer. This curve can be explained exactly with an overhead of 125 us per DMA transfer and a 
> > continuous link speed of 83 MB/sec.
> 
> What VME module is on the other end?
> 
> K.O.

The PSI-built DRS4 board, where we implemented the 2eVME protocol in the Virtex II FPGA. The same speed can be obtained with the commercial 
VME memory module CI-VME64 from Chrislin Industries (see http://www.controlled.com/vme/chinp1.html).

Stefan