 20 Jun 2012, Konstantin Olchanski, Info, lazylogger write to HADOOP HDFS
|
I tried using the lazylogger "Disk" method to write into a HADOOP HDFS clustered filesystem and found a
number of problems. I ended up replacing the lazylogger lazy_copy() function that still uses former YBOS
code with a new lazy_disk_copy() function that uses generic fread/fwrite. Also fixed the situation where
|
20 Jun 2012, Konstantin Olchanski, Info, midas vme benchmarks 
|
I am recording here the results from a test VME system using two VF48 waveform digitizers and a 64-bit
dual-core VME processor (V7865). VF48 data suppression is off, VF48 modules set to read 48 channels,
1000 ADC samples each. mlogger data compression is enabled (gzip -1).
|
 20 Jun 2012, Konstantin Olchanski, Info, midas vme benchmarks
|
> I am recording here the results from a test VME system using two VF48 waveform digitizers
Note 1: data compression is about 89% (hence "data to disk" rate is much smaller than the "data from VME" rate)
|
|
21 Jun 2012, Konstantin Olchanski, Info, midas vme benchmarks
|
> Just for completeness: Attached is the VME transfer speed I get with the SIS3100/SIS1100 interface using
> 2eVME transfer. This curve can be explained exactly with an overhead of 125 us per DMA transfer and a
> continuous link speed of 83 MB/sec.
|
|
24 Jun 2012, Konstantin Olchanski, Info, midas vme benchmarks
|
> > > Just for completeness: Attached is the VME transfer speed I get with the SIS3100/SIS1100 interface using
> > > 2eVME transfer. This curve can be explained exactly with an overhead of 125 us per DMA transfer and a
> > > continuous link speed of 83 MB/sec.
|
|
24 Jun 2012, Konstantin Olchanski, Info, midas vme benchmarks
|
> > I am recording here the results from a test VME system using two VF48 waveform digitizers
(I now have 4 VF48 waveform digitizers, so the event rates are half of those reported before. Date rate
|
|
25 Jun 2012, Konstantin Olchanski, Info, midas vme benchmarks
|
> > P.S. Observe the ever present unexplained event rate fluctuations between 130-140 event/sec.
>
> An important aspect of optimizing your system is to keep the network traffic under control. I use GBit Ethernet between FE and BE, and make sure the |
|
26 Jun 2012, Konstantin Olchanski, Info, midas vme benchmarks
|
> > > I am recording here the results from a test VME system using four VF48 waveform digitizers
Now we look at the detail of the event readout, or if you want, the real-time properties of the MIDAS
|
|
26 Jun 2012, Konstantin Olchanski, Info, midas vme benchmarks
|
> > > > I am recording here the results from a test VME system using four VF48
waveform digitizers
|
|
29 Jun 2012, Konstantin Olchanski, Info, lazylogger write to HADOOP HDFS
|
> Anyhow, the new lazylogger writes into HDFS just fine and I expect that it would also work for writing into
> DCACHE using PNFS (if ever we get the SL6 PNFS working with our DCACHE servers).
>
|
|
04 Jul 2012, Konstantin Olchanski, Bug Report, Crash after recursive use of rpc_execute()
|
I am looking at a MIDAS kaboom when running out of space on the data disk - everything was freezing
up, even the VME frontend crashed sometimes.
|
|
04 Jul 2012, Konstantin Olchanski, Bug Report, Crash after recursive use of rpc_execute()
|
> ... I see a recursive call to rpc_execute(): rpc_execute() calls tr_stop() calls cm_yield() calls
> ss_suspend() calls rpc_execute()
> ... rpc_execute() cannot be called recursively - it is not re-entrant as it uses a global buffer
|
|
17 Aug 2012, Konstantin Olchanski, Bug Report, launching roody kills the analyzer
|
> I've installed midas (Rev:5294) on SLC6.3 (64bit), along with recent trunk versions of rootana and roody.
>
> #6 root_server_thread (arg=ox7f54fc001150) at src/mana.c:5154
|
|
26 Sep 2012, Konstantin Olchanski, Bug Report, launching roody kills the analyzer
|
> >
> > I guess you can also debug the old midas server code inside mana.c...
> >
|
|
13 Dec 2012, Konstantin Olchanski, Bug Report, ss_thread_kill() kills entire program
|
> Hi, I'm having some trouble getting ss_thread_kill() to work properly. It seems
> to kill the entire program instead of just the thread.
|
|
24 Jan 2013, Konstantin Olchanski, Info, Compression benchmarks
|
In the DEAP experiment, the normal MIDAS mlogger gzip compression is not fast enough for some data
taking modes, so I am doing tests of other compression programs. Here is the results.
|
|
13 Feb 2013, Konstantin Olchanski, Info, Review of github and bitbucket
|
I have done a review of github and bitbucket as candidates for hosting GIT repositories for collaborative
DAQ-type projects. Here is my impressions.
|
|
08 Mar 2013, Konstantin Olchanski, Info, ODB /Experiment/MAX_EVENT_SIZE
|
Somebody pointed out an error in the MIDAS documentation regarding maximum event size
supported by MIDAS and the MAX_EVENT_SIZE #define in midas.h.
|
|
02 Apr 2013, Konstantin Olchanski, Info, Review of github and bitbucket
|
Hi, thanks for your positive feedback. I have been using git for small private projects for a few years now
and I like it. It is similar to the old SCCS days - good version control without having to setup servers,
accounts, doodads, etc.
|
|
05 Apr 2013, Konstantin Olchanski, Info, ODB JSON support
|
odbedit can now save ODB in JSON-formatted files. (JSON is a popular data encoding standard associated
with Javascript). The intent is to eventually use the ODB JSON encoder in mhttpd to simplify passing of
ODB data to custom web pages. In mhttpd I also intend to support the JSON-P variation of JSON (via the
|