Back Midas Rome Roody Rootana
  Midas DAQ System  Not logged in ELOG logo
Entry  20 Jun 2012, Konstantin Olchanski, Info, lazylogger write to HADOOP HDFS 
    Reply  29 Jun 2012, Konstantin Olchanski, Info, lazylogger write to HADOOP HDFS 
Message ID: 804     Entry time: 20 Jun 2012     Reply to this: 818
Author: Konstantin Olchanski 
Topic: Info 
Subject: lazylogger write to HADOOP HDFS 
I tried using the lazylogger "Disk" method to write into a HADOOP HDFS clustered filesystem and found a 
number of problems. I ended up replacing the lazylogger lazy_copy() function that still uses former YBOS 
code with a new lazy_disk_copy() function that uses generic fread/fwrite. Also fixed the situation where 
lazylogger cannot cleanly stop from the mhttpd "programs/stop" button while it is busy writing (the fix 
works only for the "Disk" method).

(Note that one can also use the "Script" method for writing into HDFS)

Anyhow, the new lazylogger writes into HDFS just fine and I expect that it would also work for writing into 
DCACHE using PNFS (if ever we get the SL6 PNFS working with our DCACHE servers).

Writing into our test HDFS cluster runs at about 20 MiBytes/sec for 1GB files with replication set to 3.

svn rev 5295
K.O.
ELOG V3.1.4-2e1708b5