Lazylogger

• Midas Doc
• Feature Listing
• Application Listing
• Online Database
• FAQ
• Index

Links

• Data Logger
• /Lazy ODB tree

Purpose

The lazylogger utility decouples the data acquisition from the data logging mechanism. The need for such an application has been dictated by the slow response time of some of the media logging devices (i.e. tape devices). Delay due to tape mounting, re-tension and re-positioning implies that the data acquisition has to be held until operation completion. By using mlogger to log data to disk in a first stage and then using lazylogger to copy or move the stored files to the "slow device", the data acquisition can be kept running without interruption.

• Multiple lazyloggers can be running simultaneously on the same computer, each one taking care of a particular channel.
• Each lazylogger channel will have a dedicated ODB tree (see ODB /Lazy Tree ) containing its own information.
• All the lazylogger channels will be in the ODB under /Lazy/<channel-name>/....
• Each channel tree is composed of three sub-trees Settings,Statistics and List.
• Dynamic directory destination based on run number or date.
• Script copy support.
• Compression copy.

The Settings and Statistics trees are self-explanatory, and contain the operational parameters of the channel(s). The List key will have a dynamic list of run numbers which have been successfully dealt with by the lazylogger channel. This list won't exist until the first successful operation of the channel is completed. While the lazylogger was developed specifically for tape devices, it also supports data file transfer to an FTP repository system. Improvements towards more generic support (including scripting and pipe compression) have been added.

Arguments

[-h ] : help.
[-h hostname ]        : see common parameters
[-e experiment_name ] : see common parameters
[-D ]                 : start program as a daemon.
[-c channel ]         : logging channel <channel-name>. Specify the lazylogger to activate.
[-z ]                 : zap statistics. Clear the statistics tree of all the defined lazylogger channels.

Setting up the Lazylogger

The Lazylogger requires to be set up before the data file can be moved. This setup consists of the following steps:

1. Invoke the lazylogger once for setting up the appropriate ODB <channel-name> tree and exit. In this example, the <channel-name> is "Pion".

 >lazylogger -c pion

2. Edit the newly created ODB tree. Set the Settings field(s) to match your requirements.

The following example is shown using odbedit :

> odbedit -e midas
[local:midas:Stopped]/>cd /Lazy/pion/
[local:midas:Stopped]pion>ls
[local:midas:Stopped]pion>ls -lr
Key name                        Type    #Val  Size  Last Opn Mode Value
---------------------------------------------------------------------------
pion                           DIR
   Settings                    DIR
       Maintain free space(%)  INT     1     4     3m   0   RWD  0
       Stay behind             INT     1     4     3m   0   RWD  -3
       Alarm Class             STRING  1     32    3m   0   RWD  
       Running condition       STRING  1     128   3m   0   RWD  ALWAYS
       Data dir                STRING  1     256   3m   0   RWD  /home/midas/online
       Data format             STRING  1     8     3m   0   RWD  MIDAS
       Filename format         STRING  1     128   3m   0   RWD  run%05d.mid
       Backup type             STRING  1     8     3m   0   RWD  Pion
       Execute after rewind    STRING  1     64    3m   0   RWD  
       Path                    STRING  1     128   3m   0   RWD  
       Capacity (Bytes)        FLOAT   1     4     3m   0   RWD  5e+09
       List label              STRING  1     128   3m   0   RWD  
   Statistics                  DIR
       Backup file             STRING  1     128   3m   0   RWD  none 
       File size [Bytes]       DOUBLE  1     4     3m   0   RWD  0
       KBytes copied           DOUBLE  1     4     3m   0   RWD  0
       Total Bytes copied      DOUBLE  1     4     3m   0   RWD  0
       Copy progress [%]       DOUBLE  1     4     3m   0   RWD  0
       Copy Rate [bytes per s] DOUBLE  1     4     3m   0   RWD  0
       Backup status [%]       DOUBLE  1     4     3m   0   RWD  0
       Number of Files         INT     1     4     3m   0   RWD  0
       Current Lazy run        INT     1     4     3m   0   RWD  0
[local:midas:Stopped]pion>cd Settings/
[local:midas:Stopped]Settings>set "Data dir" /data
[local:midas:Stopped]Settings>set "Capacity (Bytes)" 15e9

3. Start lazylogger in the background

>lazylogger -c Pion -D

4. Define List label key

At this point, the lazylogger is running and waiting for the List label key to be defined before starting the copy procedure.

> odbedit -e midas
[local:midas:Stopped]/>cd /Lazy/pion/Settings
[local:midas:Stopped]Settings>set "List label" cni-043

NOTE: The mstat utility will display information regarding the status of the lazylogger.

Operation of the Lazylogger

For every major operation of the lazylogger a message is sent to the Message buffer and will be appended to a default Lazy log file (lazy.log). These messages are the only means of finding out What/When/Where/How the lazylogger has operated on a data file. A fragment of the lazy.log for the iris experiment is shown below. In this case, the Maintain free space field was enabled which produces the cleanup of the data files and the entry in the List tree after copy. In this particular case, the lazylogger uses a perl script to push the data file to a data storage system using dCache.

[local:iris:Stopped]/Lazy> Key name Type #Val Size Last Opn Mode Value

---

Lazy DIR

   Dcache                      DIR
       Settings                DIR
           Period              INT     1     4     51h  0   RWD  2
           Maintain free space INT     1     4     51h  0   RWD  15
           Stay behind         INT     1     4     51h  0   RWD  1
           Alarm Class         STRING  1     32    51h  0   RWD  
           Running condition   STRING  1     128   51h  0   RWD  ALWAYS
           Data dir            STRING  1     256   51h  0   RWD  /iris_data0/iris/current
           Data format         STRING  1     8     51h  0   RWD  MIDAS
           Filename format     STRING  1     128   51h  0   RWD  iris_%08d%04d.mid.gz
           Backup type         STRING  1     8     51h  0   RWD  Script
           Execute after rewindSTRING  1     64    51h  0   RWD  
           Path                STRING  1     128   51h  0   RWD  /home/iris/bin/lazy_dcache.perl
           Capacity (Bytes)    FLOAT   1     4     51h  0   RWD  5e+09
           List label          STRING  1     128   51h  0   RWD  Dcache
           Execute before writiSTRING  1     64    51h  0   RWD  
           Execute after writinSTRING  1     64    51h  0   RWD  
           Modulo.Position     STRING  1     8     51h  0   RWD  
           Tape Data Append    BOOL    1     4     51h  0   RWD  y
       Statistics              DIR
           Backup file         STRING  1     128   15h  0   RWDE iris_000035270000.mid.gz
           File size (Bytes)   DOUBLE  1     8     15h  0   RWDE 326345493
           KBytes copied       DOUBLE  1     8     15h  0   RWDE 0
           Total Bytes copied  DOUBLE  1     8     15h  0   RWDE 0
           Copy progress (%)   DOUBLE  1     8     15h  0   RWDE 0
           Copy Rate (Bytes perDOUBLE  1     8     15h  0   RWDE 0
           Backup status (%)   DOUBLE  1     8     15h  0   RWDE 0
           Number of Files     INT     1     4     15h  0   RWDE 247
           Current Lazy run    INT     1     4     15h  0   RWDE 35270000

Wed May 6 17:26:00 2015 [Lazy_Dcache] Dcache[30] (cp:1.0s) iris_000032140000.mid.gz 6.146MB file NEW Wed May 6 17:26:02 2015 [Lazy_Dcache] Dcache[31] (cp:0.1s) iris_000032150000.mid.gz 22.894MB file NEW Wed May 6 17:26:04 2015 [Lazy_Dcache] Dcache[32] (cp:0.1s) iris_000032160000.mid.gz 116.110MB file NEW Wed May 6 17:26:07 2015 [Lazy_Dcache] Dcache[33] (cp:0.1s) iris_000032170000.mid.gz 110.228MB file NEW Wed May 6 17:26:09 2015 [Lazy_Dcache] Dcache[34] (cp:0.0s) iris_000032180000.mid.gz 101.333MB file NEW Wed May 6 17:26:12 2015 [Lazy_Dcache] Dcache[35] (cp:0.1s) iris_000032190000.mid.gz 90.928MB file NEW Wed May 6 17:26:14 2015 [Lazy_Dcache] Dcache[36] (cp:0.1s) iris_000032200000.mid.gz 88.461MB file NEW Thu May 7 15:47:02 2015 [Lazy_Dcache] Dcache[1] (cp:3.2s) iris_000032210000.mid.gz 87.862MB file NEW Thu May 7 15:47:06 2015 [Lazy_Dcache] Dcache[2] (cp:2.2s) iris_000032220000.mid.gz 74.789MB file NEW Thu May 7 15:47:10 2015 [Lazy_Dcache] Dcache[3] (cp:1.2s) iris_000032230000.mid.gz 0.204MB file NEW Thu May 7 15:47:16 2015 [Lazy_Dcache] Dcache[4] (cp:4.1s) iris_000032240000.mid.gz 105.346MB file NEW Thu May 7 15:47:20 2015 [Lazy_Dcache] Dcache[5] (cp:1.2s) iris_000032250000.mid.gz 14.048MB file NEW Thu May 7 15:47:23 2015 [Lazy_Dcache] Dcache[6] (cp:1.2s) iris_000032260000.mid.gz 5.011MB file NEW Thu May 7 15:47:30 2015 [Lazy_Dcache] Dcache[7] (cp:4.2s) iris_000032270000.mid.gz 94.028MB file NEW Thu May 7 15:47:36 2015 [Lazy_Dcache] Dcache[8] (cp:4.2s) iris_000032280000.mid.gz 110.765MB file NEW Thu May 7 15:47:42 2015 [Lazy_Dcache] Dcache[9] (cp:3.0s) iris_000032290000.mid.gz 88.982MB file NEW Thu May 7 15:47:46 2015 [Lazy_Dcache] Dcache[10] (cp:2.0s) iris_000032300000.mid.gz 51.735MB file NEW

... Once the lazylogger has started a job on a data file, trying to terminate the application will result in producing a log message informing the user of the actual percentage of the backup completed so far. This message will repeat itself until completion of the backup, and only then the lazylogger application will terminate. If an interruption of the lazylogger is forced (kill...), the state of the backup device is undetermined. Recovery is not possible, and the full backup set has to be redone. In order to do this, you need:

1. To rewind the backup device.
2. Delete the /Lazy/<channel_name>/List/<list label> array.
3. Restart the lazylogger with the -z switch which will "zap" the statistics entries.

In order to facilitate the recovery procedure, lazylogger produces an ODB ASCII file of the lazy channel tree after completion of successful operation. This file (Tape_recover.odb) stored in Data dir can be used for ODB as well as lazylogger recovery.